FAIR Metadata Enriching Processor (User Guide)

Table of Contents

• Overview
• Functionality
  • Temporal Coverage
  • Spatial Coverage
• Requirements
  • Expected input entities
  • Expected input attributes
• How it works in NiFi
  • Metadata enrichment for distributions
  • Metadata enrichment for datasets
• Future improvements

Overview

The FAIR Metadata Enriching Processor is a NiFi processor designed to improve the metadata quality of Dataset and Distribution entities by calculating spatial coverage and temporal coverage. Enhancing these attributes supports better FAIR (Findable, Accessible, Interoperable, Reusable) data compliance.

Functionality

Temporal Coverage

Distribution

It is determined by the earliest and the latest dates for that distribution.

Dataset

It is determined by the earliest and the latest dates from all the distributions belonging to that dataset.

Output format

"temporal": {
  "type": "JsonProperty",
  "json": {
    "status": "active",
    "startDate": "YYYY-MM-DDTHH:MM:SSZ",
    "endDate": "YYYY-MM-DDTHH:MM:SSZ"
  }
} 

It has a status sub-attribute equal to:

• active: for entities with temporal values
• inactive: for entities without temporal evolution
• pending: for entities expected to have temporal evolution in the future

Spatial Coverage

Distribution

It is determined by the value of the location attribute.

Dataset

It is determined by the geographical area enclosing the locations of all the distributions belonging to that dataset.

Output format

"spatial": {
  "type": "GeoProperty",
  "value": {
    "type": "Polygon",
    "coordinates": [...]
  }
}

Requirements

Expected input entities

The processor supports entities of type Dataset (dcat:Dataset) and Distribution (dcat:Distribution) in alignment with W3C DCAT v3 specification.

Expected input attributes

Distribution

• A default temporal attribute with an initial pending or inactive status
• A location NGSI-LD GeoProperty

"location": {
  "type": "GeoProperty",
  "value": {
    "type": "Point",
    "coordinates": [longitude, latitude]
  }
}

Dataset

• Distributions with temporal coverage
• Distributions with a location NGSI-LD GeoProperty

How it works in NiFi

The Metadata Enriching Processor is divided into two NiFi process groups:

• Metadata enrichment for distributions
• Metadata enrichment for datasets

Metadata enrichment for distributions

The distribution process group is scheduled to run daily at 00:00 UTC, but this schedule can be configured as needed. It must always be executed before the dataset process group, since the temporal coverage of distributions is required for calculating the temporal coverage of datasets.

By default, all distributions must have a temporal attribute initialized with the status pending or inactive. This status will be later updated in the NiFi flow based on different cases:

• Distributions with the inactive status are discarded immediately in the flow.
• Distributions with the pending status are routed to the initial calculation branch.
• Distributions with the active status are routed to the daily calculation branch. In this case, the temporal coverage is recalculated only when a newly retrieved observedAt for any attribute extends the existing coverage period.

When launching this flow for the first time, most of the distributions are in pending status and the calculation can take some time if there is a lot of history in the distributions. Thus it is recommended to manually run the flow for the first initialisation of the temporal coverage in the distributions

Metadata enrichment for datasets

The dataset process group is scheduled to run daily at 01:00 UTC. It uses the previously calculated temporal coverage of distributions to determine the dataset’s temporal coverage.

It uses the GeoTools library to calculate the dataset’s spatial coverage based on the locations of its distributions.

Future improvements

• Better handling of datasets with only two distributions: Currently, if a dataset contains only two distributions, the spatial coverage is represented as a LineString between the two points. In the future, this could be improved by calculating a bounding rectangle derived from the two locations.
• The temporal resolution, which describes the minimum spacing of items within the dataset, could also be calculated. It is given in dcat:temporalResolution using the standard datatype xsd:duration.

Acknowledgments

This project has been funded by the WATERVERSE project of the European Union’s Horizon Europe programme under Grant Agreement no 101070262.

WATERVERSE is a project that promotes the use of FAIR (Findable, Accessible, Interoperable, and Reusable) data principles to improve water sector data management and sharing.

As part of this effort, the FAIR Metadata Enriching Processor project enables to automatically enrich Datasets and Distributions with FAIR metrics, making them more open, interoperable, and reusable.