Activate 2018

Salesforce implemented and runs Solr clusters serving customers whose data volumes and query characteristics are hard to predict. These clusters automatically handle index growth and data rebalancing.

But to more efficiently use hardware, adapt to variable total and core-specific loads, and run in cloud infrastructures, changes were required. We introduced a new architecture in which cores are persisted in a Blob Store (S3) with Solr servers’ local SSD storage used only as a cache.

Cores are then loaded by servers as needed, and popular cores are replicated and maintained. Inactive cores are removed (but kept in Blob Store!). Servers can be shut down when overall load is low, because the remaining servers can serve any core. And performance is globally unaffected, because querying and indexing use a local core copy after the core is initially loaded from the Blob Store.

Core segment updates between Solr servers and the Blob Store build on Solr Replication logic, replacing the dialogue between two servers with Blob Store persisted metadata.

This architecture provides an elastic, highly available and scalable search cluster with a relatively simple implementation.