[timeseries] Add initial support for elasticsearch #99

[nepython]

Checks:

• I have manually tested the proposed changes
• I have written new test cases to avoid regressions (if necessary)
• I have updated the documentation (e.g. README.rst)

Closes #99, #68

Need Help
There is one test (test_get_device_metrics_csv) failing on travis (it's probably caused by cardinality aggregation counting None from date_histogram as a value). I am unable to find a solution to this. In real case this should be rare as can be confirmed by the passing of test_wifi_hostapd.

PS: Query posted on https://discuss.elastic.co/t/cardinality-aggregation-always-returning-one-extra-count/243462

[nemesifier]

Good progress @nepython, see my comments below.

[nemesifier]

why don't you change the code so that calling this functon from the configuration is not necessary?
You can loop over the data structure and call it when it's initialized, this way we make things easy for users and we avoid them come to complain to us in the support channels 😂

[nepython]

Yes that can be done and I have done the same for built-in charts. _make_query is just a utility function which will update the aggregation for a default_chart_query defined in openwisp_monitoring.db.backends.elasticsearch.queries. So queries returned via _make_query will always retain the structure of default_chart_query.

I wanted to leave the option of directly using a dsl query with timeseries_db.query, exactly like how we can query InfluxDB directly using the same function.

A full dsl query will look like this,

{'query': {'nested': {'path': 'tags',
   'query': {'bool': {'must': [{'match': {'tags.object_id': {'query': '9a39a5ae-146b-4a50-b113-f9381b8c1721'}}},
      {'match': {'tags.content_type': {'query': 'config.device'}}}]}}}},
 '_source': False,
 'size': 0,
 'aggs': {'GroupByTime': {'nested': {'path': 'points',
    'aggs': {'set_range': {'filter': {'range': {'points.time': {'from': 'now-1d/d',
         'to': 'now/d'}}},
      'aggs': {'time': {'date_histogram': {'field': 'points.time',
         'fixed_interval': '10m',
         'format': 'date_time_no_millis',
         'order': {'_key': 'desc'},
         'time_zone': 'Asia/Kolkata'},
        'aggs': {'nest': {'nested': {'path': 'points.fields',
           'aggs': {'CPU_load': {'avg': {'field': 'points.fields.cpu_usage'}}}}}}}}}}}}}}

So, if we make _make_query as a compulsion, we might be cutting down a user's freedom to query via DSL. Personally, I would like to give user this freedom (this would enable him to just put a query like above in chart configuration and it will work) 😄.

[PabloCastellano]

We don't need to run Elasticsearch in a High Available environment. Testing HA capabilities is ElasticSearch's job 😃 . We can simply make sure that setting up a multi-nodes cluster works but IMHO it is enough for us to run tests in only one instance.

WDYT?

[nepython]

I agree though there are some problems that I was facing with elasticsearch docker due to which too I am using two nodes 😅. Can you please check out if it's possible to run elasticsearch on a single port (I am not sure about this as I could not :/ ) and then I can adapt. Thanks!

[nepython]

Currently, things to do:~

• Fix cardinality aggregation returning an extra value for wifi_clients causing test_get_device_metrics_csv to fail.
• Elasticsearch tests are currently consuming ~120s to run. This can be reduced to 75s by retaining indices. There are other optimizations too which shall be implemented together in [timeseries] Optimize elasticsearch #168.

[nemesifier]

Currently, things to do:~

• Fix cardinality aggregation returning an extra value for wifi_clients causing test_get_device_metrics_csv to fail.

Please resolve this asap.

• Elasticsearch tests are currently consuming ~120s to run. This can be reduced to 75s by retaining indices. There are other optimizations too which shall be implemented together in #168.

We may do this in a second step. We make it work first, then we optimize.

[nepython]

Currently, things to do:~

• Fix cardinality aggregation returning an extra value for wifi_clients causing test_get_device_metrics_csv to fail.

Please resolve this asap.

I need help, mentioned in opening comment 😬. I am still trying to resolve but couldn't find any fix, will question this on elasticsearch discussion forums if I am unable to in the next 1-2 days.

• Elasticsearch tests are currently consuming ~120s to run. This can be reduced to 75s by retaining indices. There are other optimizations too which shall be implemented together in [timeseries] Optimize elasticsearch #168.

We may do this in a second step. We make it work first, then we optimize.

Ok

[nemesifier]

Currently, things to do:~

• Fix cardinality aggregation returning an extra value for wifi_clients causing test_get_device_metrics_csv to fail.

Please resolve this asap.

I need help, mentioned in opening comment . I am still trying to resolve but couldn't find any fix, will question this on elasticsearch discussion forums if I am unable to in the next 1-2 days.

I see that you mentioned it, can you please write a detailed explanation of what is going on and provide links to the external info you mention?

[nemesifier]

@nepython how's it going with fixing the build here?
Please focus on completing this ⚠️

[nepython]

@nepython how's it going with fixing the build here?
Please focus on completing this ⚠️

@nemesisdesign, definitely I am working to fix the build, working mostly on https://github.com/nepython/openwisp-monitoring/tree/issues/99-add-initial-support-second-timeseries_db to try out minor changes but so far no good. I asked a query on S/O in the morning (due to no response on elasticsearch discussion forum 😶), it can be found at https://stackoverflow.com/questions/63590693/elasticsearch-cardinality-aggregation-returning-one-extra-count.

As far as I can visualize, the clients Chart sometimes returns an extra count and sometimes it doesn't. In the current travis build one test is failing due to this. Locally, that one test is always passing now (but it was failing earlier 5-6 days ago just don't know why it is passing now :/).

What I am trying to figure out right now is why sometimes the clients charts returns an extra count and correct count the other times. Is this even an issue on our end or on elasticsearch's (unsure)? I posted a detailed status on the PR and a request too to provide me with some JSON data or access to any instance in which the user is ready to experiment with elasticsearch (as you had mentioned in our last call) few days back on the IM channel https://gitter.im/openwisp/openwisp-monitoring?at=5f3f709f49148b41c9619185 and reminded the same personally but didn't receive any response 😓.

Right now if we are in a hurry to merge this PR, we can do two things:~

1. Disable clients chart for elasticsearch and state it in README as Work in Progress.
2. We can include the code related to clients chart in dev with a short note related to this problem of clients chart in a section say Known Issues and open up an issue for the same, while skipping that one test for now and adding a comment above it that this needs to be fixed.

This is a big add-on and something I have worked really hard upon so would really like to see this getting merged though even after a month I am not able to solve this one single issue (which is blocking everything else) :(. I am determined to fix this issue even after GSoC and see this module getting released and be helpful to others 😃.

[cbeaujoin-stellar]

Proposal: Modernizing Elasticsearch Integration for OpenWISP Monitoring - GSoC 2026

To properly reopen the subject of Elasticsearch integration, we should align the implementation with modern Elastic Stack standards and ensure seamless deployment across the OpenWISP ecosystem.

The following requirements are proposed for the updated implementation:

1. Versioning & Library Updates

Elasticsearch 9.x: The integration must target Elasticsearch version 9.x to ensure long-term compatibility.

Client Library: The elasticsearch Python client must be updated, and the codebase refactored to handle the API changes inherent in the 9.x series.

2. Data Architecture: ECS & Data Streams

ECS Compliance: Data must follow the Elastic Common Schema (ECS) format. This is critical for leveraging Kibana’s native capabilities and ensuring interoperability with the wider Elastic ecosystem.

Bidirectional Mapping: To maintain compatibility with OpenWISP's internal logic and other time-series backends (like InfluxDB), a mapping layer must be implemented to translate between OpenWISP metrics and ECS fields bidirectionally.

Data Streams: We must move away from traditional indices in favor of Elasticsearch Data Streams. This modern approach better handles time-series data and simplifies rollover management.

4. Deployment & Automation

OpenWISP Docker: Support for Elasticsearch must be integrated into the openwisp-docker project for both development and production-ready stacks.

Ansible Integration: * Elasticsearch deployment should be included as an optional component in ansible-openwisp2.
To follow best practices, we should leverage proven third-party collections (e.g., NETWAYS/ansible-collection-elasticstack) rather than maintaining custom deployment logic.

5. Lifecycle & Configuration Management

Templates & ILM: Default Index Templates and Index Lifecycle Management (ILM) policies must be provided out-of-the-box.

Configurability: Users must be able to override these templates and policies via Django settings or Ansible variables to suit their specific retention and hardware requirements.

6. Visualization (Bonus)

Kibana Dashboards: The contribution should include pre-configured Kibana dashboards and visualizations that leverage the ECS data to provide immediate insights into network health and performance.

[855princekumar]

@cbeaujoin-stellar Thanks for pointing me here earlier.

I’ve started going through this thread and the Elasticsearch PR (#164) to understand the current state better.

I also read through the proposed changes around ECS, data streams, and updated client versions. For now, I’m planning to begin by trying to get the existing implementation working again with the current codebase (rebasing, fixing tests, updating dependencies) so I have a clearer base to build on. Once I have that in a better state, I’ll try to move step by step into the improvements mentioned here.

If there’s any specific part in the current PR that would be a good starting point, please let me know.