Search returns duplicates when more than one field matched

[avillar]

The issue was first reported here: opengeospatial/NamingAuthority#330

See for example this URL: https://defs.opengis.net/prez/search?q=unix+time+datu ; Prez returns a single result twice, because both prefLabel and label match.

Additionally, there's something weird going on with the RDF Prez returns, where 1,696 prefixes are defined for just 49 triples.

[recalcitrantsupplant]

Hi Alejandro,

The current query groups by the predicate. Some options are

1. FE does a final grouping - probably not great and as it only has the first page it can't group over the whole result set
2. Remove the grouping by predicate, and use some function (sum, max) by weight
3. ?? other

This is the current query generated by prez:

CONSTRUCT {
  ?focus_node <https://prez.dev/type> <https://prez.dev/FocusNode>.
  ?focus_node <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> ?prof_1_node_1.
  ?focus_node <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> ?default_class_var.
  ?hashID <https://prez.dev/searchResultWeight> ?weight.
  ?hashID <https://prez.dev/searchResultPredicate> ?pred.
  ?hashID <https://prez.dev/searchResultMatch> ?match.
  ?hashID <https://prez.dev/searchResultURI> ?focus_node.
  ?hashID <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://prez.dev/SearchResult>.
}
WHERE {
  {
    SELECT DISTINCT ?focus_node ?pred ?match ?weight (URI(CONCAT("urn:hash:", SHA256(CONCAT(STR(?focus_node), STR(?pred), STR(?match), STR(?weight))))) AS ?hashID) WHERE {
      ?focus_node <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> ?default_class_var.
      {
        SELECT ?focus_node ?pred ?match (SUM(?w) AS ?weight) WHERE {
          VALUES ?pred {
            <http://www.w3.org/2000/01/rdf-schema#label>
            <http://www.w3.org/2004/02/skos/core#prefLabel>
            <http://www.w3.org/2004/02/skos/core#altLabel>
            <http://www.w3.org/2004/02/skos/core#hiddenLabel>
            <https://schema.org/name>
            <http://purl.org/dc/terms/title>
          }
          {
            ?focus_node ?pred ?match.
            BIND(100  AS ?w)
            FILTER((LCASE(?match)) = "unix time datu")
          }
          UNION
          {
            ?focus_node ?pred ?match.
            BIND(20  AS ?w)
            FILTER(REGEX(?match, "^unix time datu", "i"))
          }
          UNION
          {
            ?focus_node ?pred ?match.
            BIND(10  AS ?w)
            FILTER(REGEX(?match, "unix time datu", "i"))
          }
        }
        GROUP BY ?focus_node ?pred ?match
      }
    }
    ORDER BY DESC (?weight)
    OFFSET 0
    LIMIT 11
  }
  ?focus_node <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> ?prof_1_node_1.
}

Without a group by on pred, switching to group concat:

  SELECT
    ?focus_node
    (MAX(?weight) AS ?max_weight)
    (GROUP_CONCAT(DISTINCT STR(?pred);separator="|") as ?preds)
  WHERE {
    {
      SELECT ?focus_node ?pred ?match (SUM(?w) AS ?weight)
      WHERE {
        VALUES ?pred {
          <http://www.w3.org/2000/01/rdf-schema#label>
          <http://www.w3.org/2004/02/skos/core#prefLabel>
          <http://www.w3.org/2004/02/skos/core#altLabel>
          <http://www.w3.org/2004/02/skos/core#hiddenLabel>
          <https://schema.org/name>
          <http://purl.org/dc/terms/title>
        }
        {
          ?focus_node ?pred ?match.
          BIND(100 AS ?w)
          FILTER(LCASE(?match) = "unix time datu")
        }
        UNION {
          ?focus_node ?pred ?match.
          BIND(20 AS ?w)
          FILTER(REGEX(?match, "^unix time datu", "i"))
        }
        UNION {
          ?focus_node ?pred ?match.
          BIND(10 AS ?w)
          FILTER(REGEX(?match, "unix time datu", "i"))
        }
      }
      GROUP BY ?focus_node ?pred ?match
    }
  }
  GROUP BY ?focus_node

NB in this version we don't know what the ?match of the max weight is, I don't think there's a simple way of getting it back in SPARQL, we could group concat it as well (and probably it comes out in the same order as the concated preds).

So... perhaps some microformat like this:

SELECT
  ?focus_node
  (MAX(?weight) AS ?max_weight)
  (GROUP_CONCAT(
     DISTINCT CONCAT(STR(?pred), "@", STR(?weight));
     separator="|"
   ) AS ?pred_weight_trace)
WHERE {
  {
    SELECT ?focus_node ?pred ?match (SUM(?w) AS ?weight)
    WHERE {
      VALUES ?pred {
        <http://www.w3.org/2000/01/rdf-schema#label>
        <http://www.w3.org/2004/02/skos/core#prefLabel>
        <http://www.w3.org/2004/02/skos/core#altLabel>
        <http://www.w3.org/2004/02/skos/core#hiddenLabel>
        <https://schema.org/name>
        <http://purl.org/dc/terms/title>
      }
      {
        ?focus_node ?pred ?match.
        BIND(100 AS ?w)
        FILTER(LCASE(?match) = "unix time datu")
      }
      UNION {
        ?focus_node ?pred ?match.
        BIND(20 AS ?w)
        FILTER(REGEX(?match, "^unix time datu", "i"))
      }
      UNION {
        ?focus_node ?pred ?match.
        BIND(10 AS ?w)
        FILTER(REGEX(?match, "unix time datu", "i"))
      }
    }
    GROUP BY ?focus_node ?pred ?match
  }
}
GROUP BY ?focus_node

Which produces:

{
  "head": {
    "vars": [
      "focus_node",
      "max_weight",
      "pred_weight_trace"
    ]
  },
  "results": {
    "bindings": [
      {
        "focus_node": {
          "type": "uri",
          "value": "http://www.opengis.net/def/datum/OGC/0/UnixTimeDatum"
        },
        "max_weight": {
          "type": "literal",
          "datatype": "http://www.w3.org/2001/XMLSchema#integer",
          "value": "30"
        },
        "pred_weight_trace": {
          "type": "literal",
          "value": "http://www.w3.org/2000/01/rdf-schema#label@30|http://www.w3.org/2004/02/skos/core#prefLabel@30"
        }
      }
    ]
  }
}

Are you able to let me know what you think is best, any of the above or a new version entirely.

With the prefixes, this is due to oxigraph/oxigraph#1344

[avillar]

Thanks, David. I like the microformat approach, because I think it lets us have our cake and eat it too (is the weight even used in the FE right now?). We can also merge that in the FE to show multiple predicate hits for the same result.

[recalcitrantsupplant]

I don't believe the weight is displayed in the FE at present, but it is of course used for ordering the results.

What sort of time frame are you working to?

[avillar]

Well, it's not mission-critical, but it does cause confusion for some users (e.g., wondering what the difference is between two identical results, or thinking we have duplicate data on our system...).

[avillar]

Hi, David, any news on this? If there's no plans to implement it in the near future, I could go ahead and at least do some deduplication on the FE (even if the pagination ends up being a bit inconsistent). Thanks.