Activate 2018 has ended
Back To Schedule
Thursday, October 18 • 1:15pm - 1:55pm
Scaling Box-Search: Gearing up for Petabyte Scale

Log in to save this to your schedule, view media, leave feedback and see who's attending!

Search is an integral part of Box. It enables millions of users, across thousands of enterprises, to find relevant content. At its core, this is powered by Solr clusters across multiple data-centers, hosting hundreds of terabytes of sharded inverted index. Each day we are ingesting millions of files which causes this index to grow at a staggering pace. At this rate we will soon reach a petabyte scale search index that still needs to support near realtime indexing, low latency queries, high-availability and multi-tenancy.

In this session we will talk about some of the key scalability challenges we have been facing at Box and how we have addressed them by implementing a dynamic sharding scheme using Key Range Partitioning and Bin-packing. We will discuss the high level architecture of this new system, including some key aspects of it such as load-balancing, fault tolerance and failure-recovery. Finally we will share some lessons we have learnt in the process of building this system on top of Solr, which will hopefully help others who intend to undertake similar endeavors.

avatar for Shubhro Roy

Shubhro Roy

Senior Software Engineer, Box
Shubhro enjoy working with data, be it indexing, mining or analyzing it. Currently he is part of the Search team at Box, building infrastructure components that enable millions of users to find relevant content. Prior to Box, Shubhro worked on full text database search at Oracle... Read More →
avatar for Anthony Urbanowicz

Anthony Urbanowicz

Staff Engineer, Box
Anthony Urbanowicz is a staff engineer at Box. Before Box, Anthony worked at multiple startups to multinationals, including Microsoft, Uber and a multi winter stint teaching snowboarding. As a relevance engineer at Microsoft, Anthony worked on Bing's speller, search results ranking... Read More →

Thursday October 18, 2018 1:15pm - 1:55pm EDT
Salon 6&7