GeoBlacklight Sidecar Images

Store local copies of remote imagery in GeoBlacklight.

• Requirements
• Installation
• Rake Tasks
• View Customization
• Development

Description

This GeoBlacklight plugin captures remote images from geographic web services and saves them locally. It borrows the concept of a SolrDocumentSidecar from Spotlight, to have an ActiveRecord-based "sidecar" to match each non-AR SolrDocument. This allows us to use ActiveStorage to attach images to our solr documents.

Example Screenshot

Requirements

• Ruby >= 3.3 (CI also runs on 3.4 and 4.0)
• Rails >= 7.2, < 9 (tested on 7.2 with GeoBlacklight 4 and 8.1 with GeoBlacklight 5/6)
• GeoBlacklight 4.x, 5.x, or 6.x
• libvips (Rails default) or ImageMagick

Suggested

• Background job processor — Solid Queue (Rails 8) or Sidekiq

Installation

Existing GeoBlacklight instance

gem "geoblacklight_sidecar_images", "~> 2.0"

GeoBlacklight v3 with GBL 1.0 metadata still uses the 0.9.x series:

gem "geoblacklight_sidecar_images", "~> 0.9.1", "< 1.0"

Run the generator.

$ bin/rails generate geoblacklight_sidecar_images:install

Use --skip-views on GeoBlacklight 5/6 apps that render results with ViewComponents rather than the GBL 4 split catalog partial. Use --skip-assets when the host is not using Sprockets.

Run the database migration.

$ bin/rails db:migrate

Complete any necessary Active Storage setup steps, for example:

1. Add a config/storage.yml file

local:
  service: Disk
  root: <%= Rails.root.join("storage") %>

2. Add config/environments declarations, development.rb for example:

# Store uploaded files on the local file system (see config/storage.yml for options)
config.active_storage.service = :local

The install generator appends Sidecar Images settings to config/settings.yml (GBLSI_THUMBNAIL_FIELD, optional GeoServer proxy keys, and optional GBLSI_OGM_API_URL). Leave the GeoServer URLs blank unless you need authenticated local WMS harvesting. Leave GBLSI_OGM_API_URL blank unless you want to skip harvest and render OpenGeoMetadata API thumbnails instead.

SolrDocument#sidecar is included by the engine. You do not need to copy a method into app/models/solr_document.rb. If you are upgrading from 1.x, you can remove the generator-injected sidecar method from that file.

New GeoBlacklight instance

$ rails new app-name -m https://raw.githubusercontent.com/geoblacklight/geoblacklight_sidecar_images/develop/template.rb

Ingest test documents

  # Run your GBL instance
  bundle exec rake geoblacklight:server

  # Index the GBL test fixtures
bundle exec rake gblsci:sample_data:seed

Rake tasks

Harvest images

Harvest is only needed when you store thumbnails in Active Storage. If GBLSI_OGM_API_URL is set, skip these tasks and see OpenGeoMetadata API thumbnails.

Harvest all images

Spawns background jobs to harvest images for all documents in your Solr index (paginated with Solr cursorMarks).

bundle exec rake gblsci:images:harvest_all

Harvest an individual image

Allows you to add images one document id at a time. Pass a DOC_ID env var.

DOC_ID='stanford-cz128vq0535' bundle exec rake gblsci:images:harvest_doc_id

Harvest all incomplete states

Reattempt image harvesting for all non-successful state objects.

bundle exec rake gblsci:images:harvest_retry

Check image states

bundle exec rake gblsci:images:harvest_states

We use a state machine library to track success/failure of our harvest tasks. The states we track are:

• initialized - SolrDocumentSidecar created, no harvest attempt run
• queued - Harvest attempt queued as background job
• processing - Harvest attempt at work
• succeeded - Harvest was successful, image attached
• failed - Harvest failed, no image attached, error logged
• placeheld - Harvest was not successful, placeholder imagery will be used

SolrDocumentSidecar.in_state(:succeeded) => [#<SolrDocumentSidecar:0x0000000170697960 ... ]
SolrDocumentSidecar.image.attached? => false
SolrDocumentSidecar.image_state.current_state => "placeheld"
SolrDocumentSidecar.image_state.last_transition => #<SidecarImageTransition id: 207, to_state: "placeheld", metadata: {"solr_doc_id"=>"stanford-cg357zz0321", ...>

Destroy images

Destructive tasks require CONFIRM=1.

Remove everything

CONFIRM=1 bundle exec rake gblsci:images:harvest_purge_all

Remove orphaned AR objects

CONFIRM=1 bundle exec rake gblsci:images:harvest_purge_orphans

Remove a batch

Remove sidecar objects and attached images via a CSV file of document ids at tmp/destroy_batch.csv.

CONFIRM=1 bundle exec rake gblsci:images:harvest_destroy_batch

Troubleshooting

Harvest report

Generate a CSV file of sidecar objects and associated image state under tmp/.

bundle exec rake gblsci:images:harvest_report

Failed state inspect

Prints details for failed state harvest objects to stdout

bundle exec rake gblsci:images:harvest_failed_state_inspect

Prioritize Solr Thumbnail Field URIs

If you add a thumbnail uri to your geoblacklight solr documents...

Example Doc

{
  ...
  "dct_format_s": "TIFF",
  "dct_creator_sm": ["Minnesota. Department of Highways."],
  "thumbnail_path_ss": "https://umedia.lib.umn.edu/sites/default/files/imagecache/square300/reference/562/image/jpeg/1089695.jpg",
  "gbl_resourceClass_sm": ["Imagery"],
  ...
}

Then you can edit your GeoBlacklight settings.yml file to point at that solr field (Settings.GBLSI_THUMBNAIL_FIELD). Any docs in your index that have a value for that field will harvest the image at that URI instead of trying to retrieve an image via IIIF or the other web services.

OpenGeoMetadata API thumbnails

To skip harvest, Active Storage, and the sidecar state machine entirely, point views at the OpenGeoMetadata API. Thumbnails are loaded in the browser from:

https://ogm.geo4lib.app/api/v1/resources/{id}/thumbnail

Set the API root in config/settings.yml:

GBLSI_OGM_API_URL: 'https://ogm.geo4lib.app/api/v1'

sidecar_thumbnail_tag document then renders an img for that URL (for example stanford-mp692kw6192 → /resources/stanford-mp692kw6192/thumbnail). No sidecar row is created. Harvest rake tasks are unused in this mode.

The harvest/Active Storage path remains the default when GBLSI_OGM_API_URL is blank.

View customization

Use basic Active Storage patterns, or the engine helper, to display imagery in your application.

Example methods

# Helper (available in host views). Uses the OGM API when GBLSI_OGM_API_URL is
# set; otherwise uses a harvested Active Storage attachment.
<%= sidecar_thumbnail_tag document, size: [200, 200] %>

# Harvested Active Storage (only when OGM API thumbnails are not enabled)
document.sidecar.image.attached?
document.sidecar.image.variable?
<%= image_tag document.sidecar.image.variant(resize_to_fit: [100, 100]), {class: 'media-object'} %>

Search results

On GeoBlacklight 4, the install generator can copy a catalog _index_split_default.html.erb partial. GeoBlacklight 5/6 apps should call sidecar_thumbnail_tag (or document.sidecar.image) from their result component instead of replacing Blacklight helpers.

Show pages

Example for adding a thumbnail to the show page sidebar.

<%= sidecar_thumbnail_tag @document, size: [200, 200], class: "mr-3" %>

Development

# Run test suite
bundle exec rake ci

# Launch test app server
cd .internal_test_app/
bundle exec rake geoblacklight:server

# Load test fixtures
bundle exec rake gblsci:sample_data:seed

# Run harvest
bundle exec rake gblsci:images:harvest_all

# Tail image service log file
tail -f log/image_service_development.log

Test against a specific stack with environment variables:

RAILS_VERSION=8.1.3 GEOBLACKLIGHT_VERSION="~> 5.3" bundle exec rake ci

See Localhost Results