#elasticsearch

Elasticsearch is a distributed search engine. One can store “documents” within “indices”, which are collections of documents. One common pattern for storing time based data is to use one index per day. E.g.: tweets-2017-01-01, tweets-2017-02-01 and so on. This makes having many indices decently common, and it also makes searching across many indices common in a single search.

Elasticsearch has a query type called IndicesQuery, which lets you (among other things) optimise your query path by knowing which index to search in. For example, say that you have one index pattern for tweets (tweets-YYYY-MM-DD) and one for Reddit posts (reddit-YYYY-MM-DD). Say that someone searches for “hashtag: kittens OR subreddit: awww”. You know that you don’t have to execute the hashtag query in the Reddit indices and that you do not have to execute the subreddit query in the tweet indices. This is done by specifying to only execute the hashtag in indices that match tweets-*.

When executing the query, Elasticsearch will consider each shard (of an index) separately. Usually one has 2-5 shards per index. For each shard, it will check if the index name matches the pattern given by the IndicesQuery. This is where we find our performance bug.

The algorithm in the 1.X and 2.X versions of ES work by first expanding a pattern into the list of all matching indices – tweets-* becomes ["tweets-2017-01-01", "tweets-2017-01-02", …]. Then, for each index being considered, it checks for membership in that list:

protected boolean matchesIndices(String currentIndex, String... indices) { final String[] concreteIndices = indexNameExpressionResolver.concreteIndices(clusterService.state(), IndicesOptions.lenientExpandOpen(), indices); for (String index : concreteIndices) { if (Regex.simpleMatch(index, currentIndex)) { return true; } } return false; }

(Here, indices is a list of patterns, and the concreteIndices() method applies the wildcard internally to the list of all indices)

This means that the time complexity per shard is O(n), where n is the number of indices in your cluster. Since we consider at least one shard per index, the overall search complexity then goes to O(n^2). Furthermore, this expansion (and Elasticsearch’s Regex.simpleMatch pattern matcher) generate a lot of garbage, stressing the garbage collector and making the process even slower.

(For those of you arriving here via Russ’s blog, you’ll be chagrined to note that ES implements an exponential-time backtracking glob matcher, although that fact isn’t implicated in the bug in question.)

We used this query in our not-too-shabby production cluster. At the time, it had a total of 1152 cores and we searched over roughly 150 TB of data in about 8500 indices. We discovered that we spent almost half of our CPU time in the Regex.simpleMatch method. We patched the algorithm to instead directly check if the current index matches any of the specified indices, making it O(m), where m is the number of index patterns specified in the query (we usually had 2-3). On top of the saved CPU, this also had the benefit of making us spent about 1/3 as much time in Garbage Collection pauses, due to the fewer allocations of Strings.

This seems to have been fixed in the ‘Great Query Rewrite’ for Elasticsearch 5. I do not know if this was explicitly targeted as a performance fix, if it was by accident or if someone just thought 'oh this might be slow’.

OpenSearch Gains Momentum as the Open Source Standard for Search and Analytics

With downloads reaching 1.4 billion, OpenSearch is no longer just a fork of Elasticsearch. We examine the licensing, maintenance, and feature trade-offs driving this massive adoption shift.

In modern digital infrastructures, monitoring and analyzing data in real time is critical — and the ELK Stack has become a powerful solution for observability and operational intelligence.

The ELK Stack — Elasticsearch, Logstash, and Kibana — enables organizations to centralize, search, analyze, and visualize massive streams of logs and machine data efficiently.

From application monitoring and cybersecurity analysis to infrastructure observability and real-time troubleshooting, ELK helps businesses gain actionable insights from complex data environments.

By transforming raw logs into meaningful dashboards and analytics, the ELK Stack empowers teams to improve performance, detect anomalies, and maintain system reliability at scale.

In a data-driven world, visibility and observability are the foundation of resilient systems.

Read more:

Elasticsearch & Inverted Indices — The Death of SQL ILIKE (2026)

Rethinking Search: From SQL to Elasticsearch

When tasked with adding a search bar to an application, many developers instinctively turn to their trusty SQL database. However, this approach can lead to performance issues and scalability problems. The reason lies in how SQL databases are designed to handle queries.

The Limitations of SQL

SQL databases utilize B-Trees for indexing, which excel at finding specific values, such as IDs or dates. However, when it comes to searching for text patterns, especially with wildcards at the beginning of a string, B-Trees become inefficient. This leads to a full table scan, where the database must read every row, resulting in significant performance degradation.

Introducing Elasticsearch

Elasticsearch is a distributed, NoSQL search engine built on top of Apache Lucene. It's designed specifically for full-text search and can handle massive amounts of data with ease. By pushing JSON documents into Elasticsearch, it creates an inverted index, mapping each word to a list of documents that contain it. This allows for fast and efficient searching, even with complex queries.

Real-World Applications

Elasticsearch is particularly useful in scenarios where text search is critical, such as:

E-commerce catalogs, where users may search for products with typos or variations in spelling

Log aggregation, where developers need to find specific log entries among millions of lines

Autocomplete and search bars, where users expect instant results as they type

Implementing Elasticsearch

In a production environment, it's recommended to use an existing Elasticsearch cluster or a cloud-based service. The official Python library provides a simple way to interact with the cluster, allowing developers to query the data using a domain-specific language.

from elasticsearch import Elasticsearch es = Elasticsearch("https://my-es-cluster.internal:9200", basic_auth=("admin", "secret")) search_body = { "query": { "multi_match": { "query": "python backend architecture", "fields": ["title^3", "description"], "fuzziness": "AUTO" } } } response = es.search(index="technical_blogs", body=search_body) for hit in response["hits"]["hits"]: print(f"Found: {hit['_source']['title']} (Score: {hit['_score']})")

The Power of Inverted Indices

Elasticsearch's inverted index allows it to search billions of documents in milliseconds. By mapping each word to a list of documents, the engine can quickly find the intersection of multiple sets, resulting in fast and accurate search results. This approach is akin to using a glossary to find specific pages in a book, rather than reading the entire book from cover to cover.

The key to this efficiency lies in the way the index is structured. Instead of mapping documents to their words, an inverted index maps words to their documents. This simple flip in perspective enables Elasticsearch to handle complex searches with ease, making it an essential tool for any application that requires robust text search capabilities.

Optimizing Retrieval Pipelines: A Practical Guide to Hybrid Search Architectures in 2026

Pure vector search often fails at specific keyword recall. This guide breaks down how to architect hybrid search systems that balance semantic depth with lexical precision using RRF and cross-encoders.