[Hyrax v5] inconsistent label-fetching on import

[briesenberg07]

Descriptive summary

label-fetching for Bulkrax imports to v5 staging was inconsistent, with details below by importer

Expected behavior

all labels are fetched

Related work

I think there are probably lots of label-fetching issues and I don't think this is near a complete list

• unable to fetch LOC uris unable to fetch LOC uris #3383
• [v4] Controlled vocab metadata fetch issues [v4] Controlled vocab metadata fetch issues #3445
• Wikidata vocabulary endpoint is slow #416
• New Wikidata URIs in Subject Not Saved and Label Not Displayed on Existing Works New Wikidata URIs in Subject Not Saved and Label Not Displayed on Existing Works #2996

Accessibility Concerns

seems to have at least indirect relation to accessibility

[briesenberg07]

@CGillen this is a very small sample size -- would it be better for me to test again and make sure this is a repeatable issue before anything else?
Or, this doesn't need a ticket b/c it's just the same known issues?

[CGillen]

@briesenberg07 I think it's good to have this ticket. Sometimes updates like this can knock things loose or make it easier to find/fix issues.
Do you have any URIs or works that this failed on particularly for v5? Those would be most helpful

[briesenberg07]

Hi @CGillen

URIs or works that this failed on particularly for v5

PIDs/failed labels from the very small importer mentioned above:

• 9g54xk45v : all labels fetched for this work ✅ (AAT and Wikidata)
• rv042v74b : missed one Wikidata label -- http://www.wikidata.org/entity/Q483453
• 8w32r729v : missed one LCSH label -- http://id.loc.gov/authorities/subjects/sh85036604

@CGillen if it would be helpful to have a larger test import with more works/URIs/URI sources let me know and I will put something together -- won't be able to do it today 5/26 but could do in next couple days -- thank you!

[briesenberg07]

another test importer 5/27

@CGillen thank you for picking this up!
Details from another test importer here, another test importer still to be reviewed

IMPORTER 3179 label-fetch totals

• ULAN -- 1/1 fetched
• AAT -- 1/1 fetched
• GeoNames - 12/12 fetched (not unique)
• ITIS -- 6/6 fetched (not unique)
• LCNAF -- 0/4 fetched (not unique)
• LCSH -- 1/4 fetched (not unique)
• Wikidata -- 6/6 fetched (not unique)

IMPORTER 3179 failures details

• images/2f75r955m
  • LCNAF (2) no label fetched
    • http://id.loc.gov/authorities/names/n78086404
    • http://id.loc.gov/authorities/names/n85802437
  • LCSH (1) no label fetched
    • http://id.loc.gov/authorities/subjects/sh2010012822
  • LCSH (1) label fetched
  • GeoNames (1) label fetched
• images/6m311q81m
  • LCNAF (2) no label fetched
    • http://id.loc.gov/authorities/names/n85802437
    • http://id.loc.gov/authorities/names/n78086404
  • LCSH (2) no label fetched
    • http://id.loc.gov/authorities/subjects/sh85076820
    • http://id.loc.gov/authorities/subjects/sh2010012822
  • GeoNames (1) label fetched

[CGillen]

This was an issue introduced with #3606. If a vocab term fails to fetch for any reason it'll get cached as a fail for a week. This would be fine except for infrastructural and other out-of-our control issues crop up.
I've updated stagings to decrease this time to 1 minute for testing purposes

The wikidata labels are an issue referenced here: #2996 (comment)
True https support for wikidata is something that can be looked at, but by default they do some out-of-spec stuff for their linked data.

This should be working now, so give QA another go

[briesenberg07]

Improved but still not 100% -- is this a pass or a fail?

QA results for importer 3182

• Wikidata URIs -- fetched 4/4 (not unique)
• AAT URIs -- fetched 4/4 (not unique)
• ULAN URIs -- fetched 3/4 (not unique)
  • not fetched http://vocab.getty.edu/ulan/500091068
• GeoNames URIs -- fetched 9/9 (not unique)
• LCNAF URIs -- fetched 4/4 (not unique)
• LCSH URIs -- fetched 3/4 (not unique)
  • not fetched http://id.loc.gov/authorities/subjects/sh85129595

[briesenberg07]

🔬 importer 3183 results
Ran another importer with better, but not complete, coverage for URI sources. Very good results! Not sure how good we expect this to get before closing the ticket -- please let me know if you'd like more testing

URIs from each source not unique
Interesting (?) note -- URI entered as value more than once in different fields shows more than once in Data Sources

• wikidata -- fetched ✅
• LCNAF -- fetched ✅
• LCSH -- fetched ✅
• LC TGM -- not fetched ❌
  • http://id.loc.gov/vocabulary/graphicMaterials/tgm011493 (recorded 6 times)
• LC cultural heritage orgs -- fetched ✅
• LC AFS ethnographic terms -- fetched ✅
• ONS publisher -- fetched ✅
• ONS creator -- fetched ✅
• BNE -- fetched ✅
• ONS OSU buildings -- fetched ✅
• ONS subject -- fetched ✅
• ONS people -- fetched ✅
• ONS OSU academic units -- fetched ✅
• ONS TFDD river basins -- fetched ✅
• ITIS -- fetched ✅
• AAT -- fetched ✅
• ULAN -- fetched ✅
• Homosaurus -- fetched ✅
• GeoNames -- fetched ✅

[briesenberg07]

closing ticket b/c

• remaining problems are likely an environment issue (prod vs v4-staging), and
• performance in last test is on par with current performance in OD prod