Data quality: rules & ideas

[raphodn]

We're starting to see new github issues related to data quality, and users asking for more flexibility on price addition.

So I was thinking we should start a list of possible rules that can be used in the future by moderators (nutripatrol) or automated (robotoff) to help flag errors or low quality prices

[raphodn]

Per-model rules

Model	Rule	Description	Related issue(s)
Price	if with a product_code, then the category-related fields should be empty (category_tag, labels_tags, origins_tags, price_per)
Price	if with a category_tag, then the product-related fields should be empty (product_code)
Price	if with a category_tag, then price_per should be set
Price	the price cannot be negative	there is already a check in the backend on save i believe
Price	if price_without_discount is set, then price_is_discounted must be True
Price	if both price_without_discount & price are set, then price_without_discount > price
Price	location_osm_type should be set if location_osm_id is set (and vice-versa)
Price	a price date cannot be in the future
Product	check if the barcode is valid (EAN8, EAN13)		openfoodfacts/open-prices-frontend#533
Proof	a RECEIPT proof should have all it's prices with the same location (as the proof)		#244
Proof	a PRICE_TAG proof should have all it's prices with the same location (as the proof)		#244
Proof	a RECEIPT proof should have all it's prices with the same date (as the proof)		#314
Proof	a PRICE_TAG proof should have all it's prices with the same date (as the proof)		#314
Proof	a RECEIPT proof should have all it's prices with the same currency (as the proof)
Proof	a PRICE_TAG proof should have all it's prices with the same currency (as the proof)
Price	location_osm_type should be set if location_osm_id is set (and vice-versa)
Proof	a proof date cannot be in the future

Other messages

• info: if the price owner & proof owner are different

Other scripts

Model	Description	Frequency	Live ?
Proof	Some proofs uploaded via the mobile app (old version) lack location/data/currency fields. Fix this using the proof's prices	Daily	✅
Product	Some products with a short EAN (8 for instance) are created with 0s padding, but should be merged with their shorter counterpart instead (ex: 59032823 & 0000059032823). The opposite most likely exists !
Product	Some prices are created with product EAN longer than 13, we should remove these products & prices (delete ONLY if they come from the validation page)		✅ #700
Price	Some shops don't display the barcode on their price tags. For instance Aldi, Lidl, Delhaize. So prices coming from these shops + validated automatically are most likely wrong...
Price	For receipts, some products appear multiple times because bought in multiple quantities, when they could be added once + put the info in the receipt_quantity field

Other UIs

Model	Description	Solution
Proof	Predicted type is different from actual type	A UI for moderators to fix
Proof	Uploaded multiple times, for multiple prices, when it could have been uploaded just once
Price	Currency AED	A UI to fix to the correct currency
Price	Multiple identical prices uploaded on a receipt, when it could be uploaded once + set the receipt_quantity (sometimes the case in some GDPR imports)
Product	Product unknown	A UI to add the product to OxF
Product	Product missing some fields (weight, brand...)	A UI to update the product

[teolemon]

@raphodn

• there's the notion of error and warning (eg all prices on a receipt are the same, probably a warning, not obligatory an error)
• Proof: send to Google Cloud Vision, and store the matching JSON output #320

[serpico]

Close to add a price for 3450970023295 which was in fact 3450970028295, I guess the first one verify the checksum since I didn't get any warning. I wish we could send a clue to the user when only one digit is wrong similar to "Do you mean" and show the product's pic of a near match...

[baslia]

We could also use the interquartile range to detect if a price has a typo or is some type of outlier.
We would need a certain amount of prices per product to have this method effective.

[raphodn]

How OFF manages data quality

General info

• https://wiki.openfoodfacts.org/Data_quality
• https://github.com/openfoodfacts/openfoodfacts-server/blob/main/taxonomies/data_quality.txt

API example

https://world.openfoodfacts.org/api/v2/product/2000000152012

"data_quality_bugs_tags": [],
"data_quality_errors_tags": [],
"data_quality_info_tags": [
    "en:no-packaging-data",
    "en:ingredients-percent-analysis-ok",
    "en:no-nutrition-data",
    "en:missing-specific-ingredient-for-this-category",
    "en:environmental-score-extended-data-not-computed",
    "en:food-groups-1-unknown",
    "en:food-groups-2-unknown",
    "en:food-groups-3-unknown"
],
"data_quality_tags": [
    "en:no-packaging-data",
    "en:ingredients-percent-analysis-ok",
    "en:no-nutrition-data",
    "en:missing-specific-ingredient-for-this-category",
    "en:environmental-score-extended-data-not-computed",
    "en:food-groups-1-unknown",
    "en:food-groups-2-unknown",
    "en:food-groups-3-unknown",
    "en:environmental-score-origins-of-ingredients-origins-are-100-percent-unknown",
    "en:environmental-score-packaging-packaging-data-missing"
],
"data_quality_warnings_tags": [
    "en:environmental-score-origins-of-ingredients-origins-are-100-percent-unknown",
    "en:environmental-score-packaging-packaging-data-missing"
],

[raphodn]

Current cleanup commands

Command	Details
match_price_tags_with_existing_prices
match_receipt_items_with_existing_prices