In modern software engineering, we take the easy way out when designing data intensive systems: we use our search engines and databases as simple storage. We tend to extract a lot of data (SELECT * or match_all) and then process it in our application layer using complex logic. If you are building an MVP with 100 users, it can work. However, when you are designing a system to scale, this strategy is the perfect recipe to have red bottlenecks, memory spikes, and a slow user experience. Today, we are going to talk about a fundamental paradigm for scalability: move compute, not data. Specifically, we will explore how the Open Search Aggregations let us do complex analysis and operations in milliseconds.
Let’s imagine we work in an E-commerce platform and we need to make an operation to know what is the average sale price per product category within the last 30 days.
The solution should not be to request more RAM. The solution is Aggregations.
In order to understand why aggregations are much faster than the application processing, we must look into the motor: Apache Lucene.
OpenSearch uses two principal data structures:
SQL databases are used to be orientated to rows (save all the user data together), whereas OpenSearch uses storage in columns for the fields we will aggregate.
When you request an aggregation, OpenSearch does not load the entire document (JSON _sorce), but it directly accesses the columnar structure in disk (or in the SO cache system).
Analogy: A library.
This allows us to scan millions of values per second with predictable memory usage.
An aggregation query has an interesting structure. The most important thing is the size: 0. This explicitly tells the motor: "Do not look for documents, I just want the answers".
There are different type of aggregations that cover most of the use cases:
Calculates numeric values
Group documents into containers based on criterias.
It is when we take the result of an aggregation and use it as input for another.
Let's go back to our initial problem. We want the total of sales per day and, inside each day, the breakdown per category.
Query en OpenSearch (DSL):
Result (Simplified):
The motor returns us a compact JSON of barely a few kilobytes, independently of if it processed 100 registers or 10 millions.
Our backend now acts simply as a light proxy or a minimal transformation layer, freeing resources to attend more concurrent HTTP petitions.
To illustrate the architectural impact with empirical data, we can look at official benchmarks and community case studies comparing traditional data extraction versus engine-side aggregations.
We know that not everything is exactly perfect. Although aggregations are potent, they have their risks if they are not managed correctly.
If you nest many aggregations of terms type (e.g. Group by Country > then by City > then by User), the number of buckets grows exponentially. This can saturate the Heap memory of the OpenSearch nodes.
Making aggregations over fields with millions of unique values (like user IDs or UUIDs) is expensive.
OpenSearch has security mechanisms (indices.breaker.request.limit). If an aggregation tries to reserve too much memory, the cluster will abort the petition to protect itself.
DevOps Advice: Monitor the logs in search of CircuitBreakingException. It is an indicator that your queries need optimization, not necessarily that you need more hardware.
Aggregations in OpenSearch are not only a functionality to make pretty graphics in Kibana or OpenSearch Dashboards. They are a fundamental tool of backend architecture.
By adopting aggregations, we principally achieve:
Infrastructure Cost Reduction: By eliminating the need to transfer and process massive JSON payloads in memory, our application layer requires significantly less CPU and RAM. This allows us to deploy smaller, cheaper containers or microservices, directly reducing our cloud hosting bills while avoiding network bottlenecks.
