Over the past year and a half, I’ve watched ElasticSearch [1] grow from a seemingly part-time code experiment into a thriving ecosystem. Not only has the number of client libraries [2] grown from 1 to over 25 (and counting!); it’s now a commercially sponsored project [3] to the tune of $34 million [4] (a $10M series A [5] and a $24M series B [6]) with 200,000 downloads a month.
Search is the touchstone of the Internet; without search the Internet wouldn’t be all that useful. Google’s meteoric rise and its resultant eponymous name for search is clear evidence as to the importance of search. Search, however, hasn’t always been that easy nor affordable to implement.
Before the likes of open source projects like Lucene [7], implementing search in an application involved an expensive commercial product or a series of SQL like statements that were never really good enough. Lucene, however, isn’t a simple hobbyist’s pursuit. Lucene requires a lot of expertise; what’s more, successful projects within the Lucene ecosystem, like Solr [8] lack a key feature that is defining modern architectures: distributed [9].
The continued low cost of storage combined with cheap rent-able infrastructures like AWS [10] has made it convenient to store a plethora of data, thus bringing to bear the importance that applications must support searching vast amounts of data, do it quickly, and affordably. This is where ElasticSearch shines [11].
ElasticSearch sits on top of Lucene [12] and adds not only a simple API for adding and searching content, but does it in a distributed manner. With infinitesimal arm grease, you can set up a search cluster that smears your data and resultant queries across a series of nodes. Not only is this resultant architecture fast, but it’s easy to set up and extremely affordable as search nodes can run on commodity hardware. In essence, ElasticSearch brings search to the masses.
Google and its resultant ease of search has changed the mindset of application users. Searching content is a presumed feature and if you don’t provide it, you’re already a few steps behind your competition. ElasticSearch is clearly a path to constructing a viable, easy to install, affordable, distributed search infrastructure for any application. If you don’t believe me, take a look at some of the innovative companies with substantial amounts of users using it: Github [13], foursquare [14], and Stack Overflow [15] are just a few.