Activate 2018

Search is an integral part of Box. It enables millions of users, across thousands of enterprises, to find relevant content. At its core, this is powered by Solr clusters across multiple data-centers, hosting hundreds of terabytes of sharded inverted index. Each day we are ingesting millions of files which causes this index to grow at a staggering pace. At this rate we will soon reach a petabyte scale search index that still needs to support near realtime indexing, low latency queries, high-availability and multi-tenancy.

In this session we will talk about some of the key scalability challenges we have been facing at Box and how we have addressed them by implementing a dynamic sharding scheme using Key Range Partitioning and Bin-packing. We will discuss the high level architecture of this new system, including some key aspects of it such as load-balancing, fault tolerance and failure-recovery. Finally we will share some lessons we have learnt in the process of building this system on top of Solr, which will hopefully help others who intend to undertake similar endeavors.