[monitoring] Adding influxDB 2.x version support #274 by praptisharma28 · Pull Request #584 · openwisp/openwisp-monitoring

praptisharma28 · 2024-05-31T17:06:21Z

Fixes #274

Checks:

I have manually tested the proposed changes
I have written new test cases to avoid regressions (if necessary)
I have updated the documentation (e.g. README.rst)

Fixes #274

praptisharma28 · 2024-06-18T16:11:59Z

June 18th weekly call summary:

Need to get ping check in influxdb 2.0 at the server locally, so that health status like "CRITICAL", "OK" are visible too and not just "UNKNOWN".
Need to get charts which right now says an error when I check for charts.
Note: Celery beat and worker will be running all the time along with the server, when I am working on 8000 port locally.

Fixes #274

nemesifier

There's too many if statements in the code (eg: if influxdb1 or influxdb2).
This is not the way to go. The logic for influxdb1 or influxdb2 must be encapsulated in each respective timeseries DB backend, the rest of the code should just be adapted to call that logic.

Please have a look at the work which was done for the elasticsearch backend to get an idea:
#164

Fixes #274

pandafy

In our call today, @praptisharma28 walked me through the code and helped me understand the changes. In this process, we found some areas for improvement.

pandafy · 2024-07-01T12:04:32Z

+        timezone=settings.TIME_ZONE
+    ):
+        bucket = self.bucket
+        measurement = params.get('measurement')


params.get('measurement') is equal to params.get('key'). Let's avoid duplicating the values that are already present to the method.

pandafy · 2024-07-01T12:09:47Z

+            'start_date': start_date, 
+            'end_date': end_date,
+            'measurement': self.config_dict.get('measurement', self.metric.key),
+            'field_name': fields or self.config_dict.get('field_name'),


If you are not using the field_name in the query, then please remove it.

I used pdb, got:

(Pdb) params {'field_name': None, 'key': 'test_metric', 'time': '2024-06-25', 'days': '7d', 'content_type': 'openwisp_users.user', 'object_id': '4f5f77d3-3b30-4eca-97c6-1301217a4edc', 'start_date': None, 'end_date': None, 'measurement': 'test_metric'}

But if I remove it, I lose on the summary of uptime chart. Charts like mostly reachable, unreachable, partially reachable are lost.

There should be only one summary in the uptime chart.

pandafy · 2024-07-01T12:10:09Z

+        params.update({
+            'start_date': start_date, 
+            'end_date': end_date,
+            'measurement': self.config_dict.get('measurement', self.metric.key),


This is duplicate to key, let's remove it.

pandafy · 2024-07-01T12:24:14Z

        return timeseries_db._get_top_fields(
-            query=q,
+            default_query=self._default_query,
+            query=self.get_query(),
            chart_type=self.type,
            group_map=self._get_group_map(params['days']),
            number=number,


The InfluxDB 2 client does not accept default_query keyword argument. We need to ensure that this works at par with InfluxDB 1 and add any required tests for this method.

pandafy · 2024-07-01T12:29:00Z

+                points = summary = timeseries_db._get_top_fields(
+                    default_query=self._default_query,
+                    chart_type=self.type,
+                    group_map=self.GROUP_MAP,
+                    number=self.top_fields,
+                    params=self._get_query_params(self.DEFAULT_TIME),
+                    time=time,
+                    query=self.query,
+                    get_fields=False,


This change is wrong. Please revert them to what was here before.

pandafy · 2024-07-01T12:42:06Z

-            summary = timeseries_db.get_list_query(summary_query)
+                points = timeseries_db.get_list_query(data_query, key=self.metric.key)
+                summary = timeseries_db.get_list_query(
+                    summary_query, key=self.metric.key


Suggested change

summary_query, key=self.metric.key

summary_query,

I don't see the key argument being used in the timeseries_db.get_list_query method. Remove it if it is not needed.

pandafy · 2024-07-01T12:45:55Z

+            raise
+
        for point in points:
+            time_value = point.get('time') or point.get('_time')


Let's avoid doing this and rename the time field in the flux query.

pandafy · 2024-07-01T12:48:11Z

+            if not time_value:
+                logger.warning(f"Point missing time value: {point}")
+                continue


Will this code ever get executed? Isn't every point in timeseries db mapped to a timestamp.

pandafy · 2024-07-01T12:54:19Z

-                if decimal_places and isinstance(value, (int, float)):
+                if decimal_places is not None and value is not None:


Can you please verify the error you were getting here?

pandafy · 2024-07-01T13:09:30Z

+    def get_ping_data_query(self, bucket, start, stop, device_ids):
+        device_filter = ' or '.join([f'r["object_id"] == "{id}"' for id in device_ids])
+        query = f'''
+        from(bucket: "{bucket}")
+          |> range(start: {start}, stop: {stop})
+          |> filter(fn: (r) => r["_measurement"] == "ping")
+          |> filter(fn: (r) => r["_field"] == "loss" or r["_field"] == "reachable" or r["_field"] == "rtt_avg" or r["_field"] == "rtt_max" or r["_field"] == "rtt_min")
+          |> filter(fn: (r) => r["content_type"] == "config.device")
+          |> filter(fn: (r) => {device_filter})
+          |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
+          |> yield(name: "mean")
+        '''
+        return query


Please move this query to queries.py and use get_query() method to generate the final query.

pandafy · 2024-07-04T11:50:35Z

+        bucket = self.bucket
+
+        # Start building the Flux query
+        flux_query = f'from(bucket:"{bucket}")'


~~@praptisharma28 why are you building the query like this here? Why aren't you using the queries defined in queries.py?~~

~~This looks wrong. Charts for different metrics are handled differently. That's why there are individual queries for each chart.~~

My bad, I got confused between the get_list_queries and read method. The current implementation is okay.

Fixes #274

pushpitkamboj · 2026-04-01T18:54:14Z

Hi, i was going through previous GSOC projects and saw someone did worked on this same gsoc project 2 years ago, was not completed? How much of the project was completed back then?
getting context from this feels relevant for gsoc'26 monitoring project
@pandafy @nemesifier

praptisharma28 force-pushed the influxdb2.x branch from e2b2446 to 295446d Compare May 31, 2024 17:39

[monitoring] Adding influxDB 2.x version support #274

4a6ae39

Fixes #274

praptisharma28 force-pushed the influxdb2.x branch 2 times, most recently from 83694c5 to cfe8da0 Compare June 1, 2024 15:30

[fix] Workflow and test updated #274

d02db9c

Fixes #274

praptisharma28 force-pushed the influxdb2.x branch from 87f36b9 to d02db9c Compare June 1, 2024 20:10

[monitoring] Updated files and corrected failing qa checks #274

0aa3c6d

Fixes #274

praptisharma28 force-pushed the influxdb2.x branch from 5647cd4 to 0aa3c6d Compare June 11, 2024 10:09

pandafy reviewed Jun 11, 2024

View reviewed changes

Comment thread README.rst

pandafy reviewed Jun 11, 2024

View reviewed changes

Comment thread openwisp_monitoring/db/backends/__init__.py Outdated

[monitoring] Changes #274

cfc9b58

Fixes #274

nemesifier requested changes Jun 25, 2024

View reviewed changes

praptisharma28 added 2 commits June 29, 2024 17:40

[monitoring] Avoided if statements #274

61300a6

Fixes #274

[monitoring] Chart generation code modified #274

bff44f1

Fixes #274

pandafy requested changes Jul 1, 2024

View reviewed changes

pandafy reviewed Jul 4, 2024

View reviewed changes

[monitoring] Some changes updated #274

3859434

Fixes #274

Baboux98 mentioned this pull request Apr 18, 2026

Influxdb2.x port on master #775

Draft

4 tasks

		if decimal_places and isinstance(value, (int, float)):
		if decimal_places is not None and value is not None:

Uh oh!

Conversation

praptisharma28 commented May 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

praptisharma28 commented Jun 18, 2024

Uh oh!

nemesifier left a comment

Choose a reason for hiding this comment

Uh oh!

pandafy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pandafy Jul 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pushpitkamboj commented Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

praptisharma28 commented May 31, 2024 •

edited

Loading

pandafy Jul 4, 2024 •

edited

Loading

pushpitkamboj commented Apr 1, 2026 •

edited

Loading