Welcome!

Apache Authors: Pat Romanski, Liz McMillan, Elizabeth White, Christopher Harrold, Janakiram MSV

Related Topics: @DevOpsSummit, Java IoT, Linux Containers, @CloudExpo, Apache

@DevOpsSummit: Blog Feed Post

Solr vs. Elasticsearch — How to Decide? By @Sematext | @DevOpsSummit [#DevOps]

Which one is better, Solr or Elasticsearch? Which one is faster? Which one scales better?

Solr vs. Elasticsearch — How to Decide?

By Otis Gospodnetić

[Otis is a Lucene, Solr, and Elasticsearch expert and co-author of “Lucene in Action” (1st and 2nd editions).  He is also the founder and CEO of Sematext. See full bio below.]

“Solr or Elasticsearch?”…well, at least that is the common question I hear from Sematext’s consulting services clients and prospects.  Which one is better, Solr or Elasticsearch?  Which one is faster?  Which one scales better?  Which one can do X, and Y, and Z?  Which one is easier to manage?  Which one should we use?  Which one do you recommend? etc., etc.

These are all great questions, though not always with clear and definite, universally applicable answers. So which one do we recommend you use? How do you choose in the end?  Well, let me share how I see Solr and Elasticsearch past, present, and future, let’s do a bit of comparing and contrasting, and hopefully help you make the right choice for your particular needs.

Early Days: Youth vs. Experience

Apache Solr is a mature project with a large and active development and user community behind it, as well as the Apache brand.  First released to open-source in 2006, Solr has long dominated the search engine space and was the go-to engine for anyone needing search functionality.  Its maturity translates to rich functionality beyond vanilla text indexing and searching; such as faceting, grouping (aka field collapsing), powerful filtering, pluggable document processing, pluggable search chain components, language detection, etc.

Solr dominated the search scene for several years.  Then, around 2010, Elasticsearch appeared as another option on the market.  Back then it was nowhere near as stable as Solr, did not have Solr’s feature depth, did not have the mindshare, brand, and so on.  But it had a few other things going for it: Elasticsearch was young and built on more modern principles, aimed at more modern use cases, and was built to make handling of large indices and high query rates easier.  Moreover, because it was so young and without a community to work with, it had the freedom to move forward in leaps and bounds, without requiring any sort of consensus or cooperation with others (users or developers), backwards compatibility, or anything else that more mature software typically has to handle.  As such it exposed certain highly sought-after functionality (e.g., Near Real-Time Search) before Solr did.  Technically speaking, the ability to have NRT Search really came from Lucene, the underlying search library to both Solr and Elasticsearch use.  The irony is that because Elasticsearch exposed NRT Search first, people associated NRT Search with Elasticsearch, even though Solr and Lucene are both part of the same Apache project and, as such, one would expect Solr to have such highly demanded functionality first.

Elasticsearch, being more modern, appealed to several groups of people and organizations:

  • those who didn’t yet have a search engine and hadn’t invested a lot of time, money, and energy in its adoption, integration, etc.
  • those who had to deal with large volumes of data and needed to more easily shard and replicate data (search indices) and shrink or grow their search cluster

Of course, let’s admit it, there will always be those who like jumping on new shiny objects, too.

Evening the Search Playing Field

Fast forward to 2014 and now 2015.  Elasticsearch is no longer new, but it’s still shiny.  It closed the feature gap with Solr and, in some cases, surpassed it.  It certainly has more buzz around it.  At this point both projects are very mature.  Both have lots of features.  Both are stable.  I have to say though, that I do see more Elasticsearch clusters with issues, but I think that is primarily because of a few reasons:

  • Elasticsearch, traditionally being easier to get started with, made it possible for anyone to start using it out of the box, without too much understanding of how things work.  That’s great to get started, but dangerous when data/cluster grows.
  • Elasticsearch, lending itself to easier scaling, attracts use cases demanding larger clusters with more data and more nodes.
  • Elasticsearch is more dynamic – data can easily move around the cluster as its nodes come and go, and this can impact stability and performance of the cluster.
  • While Solr has traditionally been more geared toward text search, Elasticsearch is aiming to handle analytical types of queries, too, and such queries come at a price.

Although this may sound scary, let me put it this way — Elasticsearch exposes a ton of control knobs one can play with to control the beast.  Of course, the key bit is that one has to be aware of all possible knobs, know what they do, and make use of that.  For example, despite what you just read about Elasticsearch, we rely on it in our organization for several different products, even though we know Solr just as well as we know Elasticsearch.

Solr: Not Totally Eclipsed

What about Solr?  Solr hasn’t exactly stood still.  The appearance of Elasticsearch was actually great for Solr and its community of developers and users.  Despite being almost 10 years old, Solr development is going faster than ever.  It, too, has a friendly API now.  It, too, has the ability to more easily grow and shrink clusters, create indices more dynamically, shard them on the fly, route documents and queries, etc., etc. Note: when people refer to SolrCloud they specifically mean this form of very distributed, Elasticsearch-like Solr deployment.

I recently attended a Lucene/Solr Revolution conference in Washington, DC and was pleasantly surprised by what I saw: a strong community, healthy project, lots of big name companies not only using Solr, but investing in it through adoption, contribution through development/engineering time, etc.  If you follow just the news you’d be led to believe Solr is dead and everyone is just flocking to Elasticsearch.  That is actually not the case.  Elasticsearch being newer, is naturally more interesting to write about.  Solr was news 5+ years ago.  And of course there were some people going from Solr to Elasticsearch when Elasticsearch appeared — in the beginning there were simply no Elasticsearch users.

So which is better?  Which one should you use?  Where do Solr and Elasticsearch differ?  What does the future hold?

Here are some other things you should keep in mind:

  • Both are released under the Apache Software License
  • Solr is truly open-source — community over code.  Anyone can contribute to Solr and new Solr developers (aka committers) are elected based on merit.  Elasticsearch is technically open-source, but less so in spirit.  Anyone can see the source, anyone can change it and offer a contribution, but only employees of Elasticsearch can actually make changes to Elasticsearch.
  • Solr contributors and committers come from a number of different organizations, while Elasticsearch committers are from a single company.
  • A number of organizations have chosen Solr over Elasticsearch as their horses in the search race (e.g. Cloudera, Hortonworks, MapR, etc.) even though they’ve also partnered with Elasticsearch.
  • Both Solr and Elasticsearch have lively user and developer communities and are rapidly being developed.
  • If you need to add certain missing functionality to either Solr or Elasticsearch, you may have more luck with Solr.  True, there are ancient Solr JIRA issues that are still open, but at least they are still open and not closed.  In Solr world the community has a bit more say even though at the end of the day it’s one of the Solr developers who has to accept and handle the contribution.
  • Both have good commercial support (consulting, production support, integration, etc.)
  • Both have good operational tools around it, although Elasticsearch has, because of its easier-to-work-with API, attracted the DevOps crowd a lot more, thus enabling a livelier ecosystem of tools around it.
  • Elasticsearch dominates the open-source log management use case — lots of organizations index their logs in Elasticsearch to make them searchable.  While Solr can now be used for this, too (see Solr for Indexing and Searching Logs and Tuning Solr for Logs), it just missed the mindshare boat on this one.
  • Solr is still much more text-search-oriented.  On the other hand, Elasticsearch is often for filtering and grouping – the analytical query workload – and not necessarily text search.  Elasticsearch developers are putting a lot of effort into making such queries more efficient (lowering of the memory footprint and CPU usage) at both Lucene and Elasticsearch level.  As such, at this point in time, Elasticsearch is a better choice for applications that need to do not just text search, but also complex search-time aggregations.
  • Elasticsearch is a bit easier to get started – a single download and a single command to get everything started.  Solr has traditionally required a bit more work and knowledge, but Solr has recently made great strides to eliminate this and now just has to work on changing its reputation.
  • Performance-wise, they are roughly the same.  I say “roughly”, because nobody has ever done comprehensive and non-biased benchmarks.  For 95% of use cases either choice will be just fine in terms of performance, and the remaining 5% need to test both solutions with their particular data and their particular access patterns.
  • Operationally speaking, Elasticsearch is a bit simpler to work with – it has just a single process.  Solr, in its Elasticsearch-like fully distributed deployment mode known as SolrCloud, depends on Apache ZooKeeper.  ZooKeeper is super mature, super widely used, etc. etc., but it’s still another moving part.  That said, if you are using Hadoop, HBase, Spark, Kafka, or a number of other newer distributed software, you are likely already running ZooKeeper somewhere in your organization.
  • While Elasticsearch has built-in ZooKeeper-like component called Xen, ZooKeeper is better at preventing the dreaded split-brain problem sometimes seen in Elasticsearch clusters.  To be fair, Elasticsearch developers are aware of this problem and are working on improving this aspect of Elasticsearch.
  • If you love monitoring and metrics, with Elasticsearch you’ll be in heaven.  The thing has more metrics than people you can squeeze in Times Square on New Year’s Eve!  Solr exposes the key metrics, but nowhere near as many as Elasticsearch.  Regardless, having comprehensive monitoring and centralized logging tools like Sematext’s SPM Performance Monitoring and Logsene Log Management and Analytics — especially when they work seamlessly together like these two do — is essential if you want to have a handle on metrics and other operational data.

Here are a few charts to demonstrate what I mean:

Elasticsearch user mailing list traffic: 36,127 (source: Search-Lucene)

Solr user mailing list traffic is about two thirds of the Elasticsearch mailing list traffic.  Of course, this could be because there are more Elasticsearch users, or because there are more problems with Elasticsearch and users are in need of more help, or perhaps they are just a chattier bunch.

Elasticsearch vs. Solr Contributors (source: Open Hub)   click to enlarge

As you can see, Elasticsearch numbers are trending sharply upward, and now more than double Solr with regard to Commit activity.  This is not a very precise or absolutely correct way to compare open-source projects, but it gives us some data points.  For example, Elasticsearch is developed on Github, which makes it very easy to merge others’ Pull Requests, while Solr contributors tend to create patches, upload them to JIRA, where they get reviewed by Solr committers before being applied — a less streamlined process.  Moreover, Elasticsearch repository contains documentation, not just code, while Solr keeps its documentation in a Wiki.  This contributes to higher numbers for both commits and contributors for Elasticsearch.

Boil It Down For Me

In conclusion, here are the bits that I think make the most difference for anyone having to make a choice:

  • If you’ve already invested a lot of time in Solr, stick with it, unless there are specific use cases that it just doesn’t handle well.  If you think that is the case, speak to somebody close to both Solr and Elasticsearch projects to save you time, guessing, research, and avoid mistakes.
  • If you are a strong believer in true open-source, Solr is closer to that than Elasticsearch, and having one company control Elasticsearch may be a turn-off.
  • If you need a data store that can handle analytical queries in addition to text searching, Elasticsearch is a better choice for that today.

If you expected a single definitive winner, I’m sorry to disappoint.  We don’t have one here.  However, I hope this quick comparison of the two leading open-source search engines provides enough information and guidance to help you make the right choice for your organization

About the author: in addition to being a Lucene, Solr, and Elasticsearch expert and author, Otis Gospodnetić is the founder and CEO of Sematext. Sematext is a globally distributed organization that builds innovative Cloud and On Premises solutions for performance monitoring, alerting and anomaly detection of Solr, Elasticsearch, Hadoop, Spark and many other applications (SPM), log management and analytics (Logsene), site search analytics (SSA), and search enhancement. The company also provides Search and Big Data consulting services and offers 24/7 production support for Solr and Elasticsearch to clients worldwide.

[Note: the original version of this article appeared at Datanami.com

Read the original blog entry...

More Stories By Sematext Blog

Sematext is a globally distributed organization that builds innovative Cloud and On Premises solutions for performance monitoring, alerting and anomaly detection (SPM), log management and analytics (Logsene), and search analytics (SSA). We also provide Search and Big Data consulting services and offer 24/7 production support for Solr and Elasticsearch.

@ThingsExpo Stories
Nordstrom is transforming the way that they do business and the cloud is the key to enabling speed and hyper personalized customer experiences. In his session at 21st Cloud Expo, Ken Schow, VP of Engineering at Nordstrom, discussed some of the key learnings and common pitfalls of large enterprises moving to the cloud. This includes strategies around choosing a cloud provider(s), architecture, and lessons learned. In addition, he covered some of the best practices for structured team migration an...
Recently, REAN Cloud built a digital concierge for a North Carolina hospital that had observed that most patient call button questions were repetitive. In addition, the paper-based process used to measure patient health metrics was laborious, not in real-time and sometimes error-prone. In their session at 21st Cloud Expo, Sean Finnerty, Executive Director, Practice Lead, Health Care & Life Science at REAN Cloud, and Dr. S.P.T. Krishnan, Principal Architect at REAN Cloud, discussed how they built...
In his session at 21st Cloud Expo, Raju Shreewastava, founder of Big Data Trunk, provided a fun and simple way to introduce Machine Leaning to anyone and everyone. He solved a machine learning problem and demonstrated an easy way to be able to do machine learning without even coding. Raju Shreewastava is the founder of Big Data Trunk (www.BigDataTrunk.com), a Big Data Training and consulting firm with offices in the United States. He previously led the data warehouse/business intelligence and B...
In his Opening Keynote at 21st Cloud Expo, John Considine, General Manager of IBM Cloud Infrastructure, led attendees through the exciting evolution of the cloud. He looked at this major disruption from the perspective of technology, business models, and what this means for enterprises of all sizes. John Considine is General Manager of Cloud Infrastructure Services at IBM. In that role he is responsible for leading IBM’s public cloud infrastructure including strategy, development, and offering m...
With tough new regulations coming to Europe on data privacy in May 2018, Calligo will explain why in reality the effect is global and transforms how you consider critical data. EU GDPR fundamentally rewrites the rules for cloud, Big Data and IoT. In his session at 21st Cloud Expo, Adam Ryan, Vice President and General Manager EMEA at Calligo, examined the regulations and provided insight on how it affects technology, challenges the established rules and will usher in new levels of diligence arou...
The 22nd International Cloud Expo | 1st DXWorld Expo has announced that its Call for Papers is open. Cloud Expo | DXWorld Expo, to be held June 5-7, 2018, at the Javits Center in New York, NY, brings together Cloud Computing, Digital Transformation, Big Data, Internet of Things, DevOps, Machine Learning and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding busin...
Smart cities have the potential to change our lives at so many levels for citizens: less pollution, reduced parking obstacles, better health, education and more energy savings. Real-time data streaming and the Internet of Things (IoT) possess the power to turn this vision into a reality. However, most organizations today are building their data infrastructure to focus solely on addressing immediate business needs vs. a platform capable of quickly adapting emerging technologies to address future ...
No hype cycles or predictions of a gazillion things here. IoT is here. You get it. You know your business and have great ideas for a business transformation strategy. What comes next? Time to make it happen. In his session at @ThingsExpo, Jay Mason, an Associate Partner of Analytics, IoT & Cybersecurity at M&S Consulting, presented a step-by-step plan to develop your technology implementation strategy. He also discussed the evaluation of communication standards and IoT messaging protocols, data...
22nd International Cloud Expo, taking place June 5-7, 2018, at the Javits Center in New York City, NY, and co-located with the 1st DXWorld Expo will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud ...
22nd International Cloud Expo, taking place June 5-7, 2018, at the Javits Center in New York City, NY, and co-located with the 1st DXWorld Expo will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud ...
DevOps at Cloud Expo – being held June 5-7, 2018, at the Javits Center in New York, NY – announces that its Call for Papers is open. Born out of proven success in agile development, cloud computing, and process automation, DevOps is a macro trend you cannot afford to miss. From showcase success stories from early adopters and web-scale businesses, DevOps is expanding to organizations of all sizes, including the world's largest enterprises – and delivering real results. Among the proven benefits,...
@DevOpsSummit at Cloud Expo, taking place June 5-7, 2018, at the Javits Center in New York City, NY, is co-located with 22nd Cloud Expo | 1st DXWorld Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait...
Cloud Expo | DXWorld Expo have announced the conference tracks for Cloud Expo 2018. Cloud Expo will be held June 5-7, 2018, at the Javits Center in New York City, and November 6-8, 2018, at the Santa Clara Convention Center, Santa Clara, CA. Digital Transformation (DX) is a major focus with the introduction of DX Expo within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive ov...
SYS-CON Events announced today that T-Mobile exhibited at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. As America's Un-carrier, T-Mobile US, Inc., is redefining the way consumers and businesses buy wireless services through leading product and service innovation. The Company's advanced nationwide 4G LTE network delivers outstanding wireless experiences to 67.4 million customers who are unwilling to compromise on qua...
SYS-CON Events announced today that Cedexis will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Cedexis is the leader in data-driven enterprise global traffic management. Whether optimizing traffic through datacenters, clouds, CDNs, or any combination, Cedexis solutions drive quality and cost-effectiveness. For more information, please visit https://www.cedexis.com.
SYS-CON Events announced today that Google Cloud has been named “Keynote Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Companies come to Google Cloud to transform their businesses. Google Cloud’s comprehensive portfolio – from infrastructure to apps to devices – helps enterprises innovate faster, scale smarter, stay secure, and do more with data than ever before.
SYS-CON Events announced today that Vivint to exhibit at SYS-CON's 21st Cloud Expo, which will take place on October 31 through November 2nd 2017 at the Santa Clara Convention Center in Santa Clara, California. As a leading smart home technology provider, Vivint offers home security, energy management, home automation, local cloud storage, and high-speed Internet solutions to more than one million customers throughout the United States and Canada. The end result is a smart home solution that sav...
SYS-CON Events announced today that Opsani will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Opsani is the leading provider of deployment automation systems for running and scaling traditional enterprise applications on container infrastructure.
SYS-CON Events announced today that Nirmata will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Nirmata provides a comprehensive platform, for deploying, operating, and optimizing containerized applications across clouds, powered by Kubernetes. Nirmata empowers enterprise DevOps teams by fully automating the complex operations and management of application containers and its underlying ...
SYS-CON Events announced today that Opsani to exhibit at SYS-CON's 21st Cloud Expo, which will take place on October 31 through November 2nd 2017 at the Santa Clara Convention Center in Santa Clara, California. Opsani is creating the next generation of automated continuous deployment tools designed specifically for containers. How is continuous deployment different from continuous integration and continuous delivery? CI/CD tools provide build and test. Continuous Deployment is the means by which...