News outside of mapme.biodiversity that can be of interest

[fBedecarrats]

I propose to open this thread so all contributors can post links to news that can be inspiring for the development of this package and the underlying framework.

[fBedecarrats]

@cboettig recently posted about the Source Cooperative in #186

[fBedecarrats]

This video showcases the use of wind data from the ERA5 dataset using the ecmwfr package

[cboettig]

Thanks! In this spirit, have you seen the duckdb spatial extension ? A lot of remote assets are in tabular form with lat/lon columns in csv etc. I've been exploring how we can use duckdb to access and perform spatial operations on those data without downloading them.

Here's a trivial example:

library(duckdbfs)
library(dplyr)
spatial_csv <- paste0("https://raw.githubusercontent.com/cboettig/duckdbfs/",
                     "7189ed7e0456927a55a877acf77aaadc074ba01b/",
                     "inst/extdata/spatial-test.csv") |>
  open_dataset(format = "csv") 

spatial_csv |>
  mutate(geometry = ST_Point(longitude, latitude)) |>
  to_sf()

For an ambitious example, here I try and do a spatial polygon filter on the GBIF data on AWS. This takes about 18min on my machine, but even a non-spatial filter, e.g. by bounding box, takes about 8min since GBIF is quite large. (This uses my thin wrapper around duckdb to streamline some of the syntax).

library(dplyr)
library(sf)
library(spData)
library(duckdbfs) #  remotes::install_github("cboettig/duckdbfs") 

# let's say we have some polygon to filter by:
costa_rica <- world |> filter(grepl("Costa Rica", name_long)) |> pull(geom) |> st_as_text()

# connect to the dataset
gbif <- duckdbfs::open_dataset("s3://gbif-open-data-us-east-1/occurrence/2022-12-01/occurrence.parquet", s3_region="us-east-1")

# Create a geometry column from the point data, then use it in filters
cr_species <- 
  gbif |> 
  mutate(geometry = ST_Point(decimallongitude, decimallatitude)) |>  
  filter(class == "Mammalia") |>
  filter(ST_Within(geometry, ST_GeomFromText({costa_rica}))) |>
  to_sf()

Haven't tested this that extensively, but curious what you think and if it could be relevant here. The docs for the spatial extension have richer examples and notes on all the supported spatial functions

[goergen95]

Iconnect is an R package to analyse landscape connectivity that also works with vector data.

[goergen95]

Chelsea is a high-resolution (~1km) climatology data set with a very high number of pre-computed variables.

[goergen95]

Added in d2986fe

[goergen95]

WorldCereal, a dynamic, 10m crop and irrigation product developed by ESA. Might be interesting for agricultural use-cases.

[fBedecarrats]

That looks great! For now it seems that they only have 2021, right? I am a bit surprised by the agro-ecological zones that they used to stratify the analysis: they seem really coarse to me (eg. only one zone for Madagascar when there are at least 3 different climates with very disting croping calendars in Madagascar...).

[goergen95]

Clay is a foundational model of Earth. It uses a Vision Transformer architecture adapted to 
understand geospatial and temporal relations on Earth Observation data. The model is 
trained via Self-supervised learning (SSL) using a Masked Autoencoder (MAE) method.

The Clay model can be used in three main ways:

   - Generate semantic embeddings for any location and time.

   - Fine-tune the model for downstream tasks such as classification, regression, and generative tasks.

   - Use the model as a backbone for other models.

https://clay-foundation.github.io/model/

[goergen95]

NAIP embeddings over continental US on source.coop

[goergen95]

ACLED released a conflict exposure tool based on WorldPop. Could be an inspiration for a new indicator with UCDP GED. https://www.worldpop.org/blog/conflict-exposure-a-new-tool-to-measure-the-impact-of-conflict/

[goergen95]

First draft added here. Feedback to this new indicators is very welcomed!

[goergen95]

Copernicus Marine Data Store

[goergen95]

New portal with WAPOR V3 released.

[goergen95]

There is a new OGC Standard Working Group currently in the process of formation on the topic of Geospatial Indicator Reporting in the context of SDGs.

[goergen95]

Upcoming GDAL 3.9 will have a feature implemented that sorts geoparquet files by bounding-box for faster reads when spatially filtering (see here). Could be useful when evaluating performant formats/file organizations to translate vector resources to in the next months.

[goergen95]

New coastal mid-res digital terrain model with low errors here.

[goergen95]

JRC's technical report on global forest map for 2020.

[goergen95]

Near-real-time estimation of carbon dioxide and water flux from geostationary satellites from the ALIVE project.

[goergen95]

1m resolution with global coverage of tree heights from Meta and WRI -> here... Absolutely crazy!

[fBedecarrats]

I understand from you posts on Linkedin and other comments on bluesky that there are still questions about the reliability of this dataset. Do you have an opinion on this now?

[goergen95]

I have not validated the data myself, but there is an interesting analysis by Iain Woodhouse available here.

There are two main take-aways from my POV:

• its a global dataset which comes with the usual caveats, i.e. the reported global accuracy does not uphold to ones expectations when comparing it with local data sets derived from e.g. airborne LiDAR or similar. However, it can provide valuable information for areas where no higher quality measurements are available or for continental/global comparisons.

• the canopy height layer was generated from image mosaics with very different time stamps between 2009 (!) - 2020. In the first release the temporal information was missing but in my understanding it is now available. However, the temporal distribution is quite patchy, especially in the tropics due to persistent cloud cover, so that properly harmonizing this data with e.g. GFW will be really challenging in my view.

[cboettig]

Hi friends, have you seen the beautiful work of Gilberto Camara and colleagues in the sits package and accompanying text? https://e-sensing.github.io/sitsbook/ . I haven't delved too much into it yet but they appear to have wrapped around rstac and gdalcubes rather nicely.

[fBedecarrats]

Thanks @cboettig ! I understand that this as some kind of extension of the {stars} approach on data cubes, applied to landcover classification and with a strong focus on accessing raster data that sits on the cloud.

[edzer]

We're working on an openEO interface around the sits package, and will try to deploy it on the CDSE.

[fBedecarrats]

This paper is not that new but I found it interesting regarding the appropriate scale of analysis to use for spatial impact evaluation: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0167945
Briefly, it says that smaller area of analysis is not always better and one should take into account the scale at which the process of interest is hapenning (eg. deforestation, land use change...) and the resolution of the covariates used for analysis.

[goergen95]

Hi,

this simulation study is gold, thanks for sharing!

I have a few thoughts in my head with some question marks of how to relate
this study to what we are trying to do here:

1. The simulation study assumes a grid-based analysis which corresponds
   well with the honeycomb procedure of the conducted impact assessments.
   I wonder if there are similar studies when the unit of analysis is vector
   based?

2. Depending on the scale of analysis, significant bias can be introduced when
   treatment is represented as a discrete variable rather than on a continuous scale.
   I wonder if this holds with our exactextract approach with which for most
   variables we extract area-weighted values based on the overlap between a
   vector with individual cells.

3. The main recommendation, in my understanding, goes something like
   think about the scale of the process you are modelling to decide on an
   appropriate scale of analysis. This does not translate well to
   global-portfolio based analysis, does it? When we are looking at globally
   distributed PAs, I assume that there are very different variables and
   relationships between those driving the underlying processes. Selecting
   a scale would assume that a) the process can be observed globally and b)
   the scale of this process does not change with space. What kind of processes
   are we interested in where we can reasonably assume this?

[goergen95]

Access to zarr data cubes for weather data via dynamical.org

[edzer]

Thanks for dragging me in this interesting issue! @cboettig brought some of this up here: r-spatial/stars#659 . Yes, the read_mdim() in stars is currently pretty minimal, reflecting what the underlying GDAL function can do: read an array, possibly starting from a certain offset, with a certain size, possibly skipping every n cells (is this called strided read?) It would be nice if it read nothing, then only do everything you wish in a cleverly chunked way, like some of the methods for stars_proxy do. That problem is however pretty complicated, and will take some time to implement. Any help is welcome!

[cboettig]

Thanks @edzer! Yes I think this mapme.biodiversity team has been doing some really cool stuff, and obviously leaning heavily on your work.

Maybe should move this over to the other thread but anyway:

Yeah, I suspected read_mdim was rather constrained by what the gdal mdim interface does and doesn't do. As nice as the idea sounds in theory, it probably doesn't make sense to try and implement everything stars can do with the standard gdal interface over on the mdim side, given that even the OSGeo team hasn't attempted feature parity there.

I do think @mdsumner 's instincts are right though in that we don't really have anything equivalent to xarray in R, i.e. a very intuitive, R-native or "tidy" implementation of (not-necessarily-spatial) n-dimensional array data. Yeah, we have the various bindings to netcdf and hdf5 C libraries, and Mike's tidync might be closest, but nothing quite like xarray syntax like in the https://dynamical.org examples (or many others) up top. On the flip side, I do think GDAL has optimized the hell out the virtual filesystem bindings, and I think at least some xarray pipelines pay a real overhead by trying to do file virtualization in fsspec etc (i.e. in python, rather than C). Though I do kind of admire the way xarray syntax is agnostic (or nearly so) across some 5 or so different backend drivers, (including gdal if we go with rioxarray, though that mechanism seems remarkably un-loved by pythonistas).

[mdsumner]

VirtualiZarr (the natural successor of kerchunk) is the wildest thing I've found, and I think it has potential and I'm already using it to transcend from the old raster package stuff we built, it's essentially the same as VRT-for-xarray that I was long looking for. It uses references to store where array blobs live in file formats. I think it's a bit silly to use for tiff, but when you think like xarray it's natural to want to avoid GDAL and any actual formal library support. (Doesn't yet support HDF4 but HDF5, netcdf, tiff, and grib are). It obviously can't support some compression encodings without being a lot more complex, but it works great for those enormous global holdings of netcdf, hdf, grib. 👌

There's pizarrr for R, I don't know if there's a real C lib for Zarr but GDAL's inbuilt is independent.

GDAL multidim is a bit undercooked but it's really a matter of having more than one person actually contribute to it (community discourse could help here), and all of this is in that uncomfortable space where matlab-think meets geospatial. If xarray (and hence also Zarr) would support extent georeferencing and not just degenerate rectilinear coords it would be better.

Python communities are pretty GDAL agnostic and that's a huge mistake. They only see it via rasterio which is a major problem. I guess if 'wheels' (binary package builds) were available for osgeo.gdal that would help, so that's something the R community could obviously help with. R now has good GDAL raster and vector API support in {gdalraster}. Get behind it!

[goergen95]

stars recently gained initial support for lazy-evaluation of multidim data sets, however, I tend to agree that GDAL's mdim-functionality currently feels undercooked and I am not sure how much this restricts what stars can/will be able to do.

I think I do see the path for a POC off of something like a virtual Zarr in R (though I am surely terribly underestimating the effort required to achieve this). I am unsure as to where is the best place to put something like this, too. Could stars (maybe eventually even GDAL) consume a virtual Zarr store? If so, we could draw on the methods already implemented there for actually getting things done with the underlying data. If not, one could build upon gdalraster, but substantial effort would be required to re-implement methods (such as subset, concatenation, aggregation, ...) on multidimensional, labelled arrays.

Naively sketching out the components I think would be required to get to a first POC:

• methods for ingesting multidim data (e.g. netcdf and other formats) into a virtual Zarr store:
  I imagine this to look something like a multidim GTI that references the chunks of multidim arrays.
  This potentially should be serialized to Parquet to allow for efficient queries when data actually needs to be read.

• call-stack mechanism:
  This mechanism should allow for the usual operations, similar to what stars already provides, e.g. cropping, sub-setting, aggregating, applying functions etc. This would also require to update dimensions/labels on-the-fly, like stars does, so you always know the shape of your arrays. I think this is actually the part that requires the most effort to get it right, but stars already has most of this, so it would make sense to build up on this.

• compute-stage:
  Data I/O via GDAL to read the required chunks as R arrays. Then apply the operations in the stack to those chunks. This could be parallelized via the future framework, allowing to scale from multi-core on a single machine to distributed compute nodes relatively cheap. I think this would also require an efficient mechanism to serialize (intermediate) results of chunks to disk.

[mdsumner]

I'm not across it enough where it does already work, but definitely interested to discuss the ideas.

It's also not always as compelling as I've been led to believe, it's faster to open 15000 NetCDFs and pull out one pixel with terra+furrr than it is to open a kerchunk json version of the entire collection with dask (instant), sel() or isel() to that pixel and variable (also instant), and then call .values (takes ~5x longer than terra start to finish). so not enough to say that dask isn't parallelizing, but I don't know how to isolate it to make a fair comparison yet. I mostly want some real discussion and detailed examples, but switching languages still largely means not talking about the other alternatives in context.

GDAL is always lazy - the only interface that's faithful is osgeo.gdal, and fwiw that is really the best way to learn the API, it's how most features are tested too. xarry already handles multiple engines that way, but it's not strong on leveraging GDAL either (rioxararay puts a particular lens on it, much like odc does too). but gdalraster is quickly becoming very rich in features, entirely suitable for extension and building tools with.

[karpfen]

I just stumbled upon https://overturemaps.org/. Right now, it looks a bit like a mirror for OSM data to me, but maybe this is something to keep an eye on, since it may evolve into a cool data source for non-remote sensing based indicators.

[goergen95]

Interesting! I assume most of the data form there is vector based, right? That would require the package to take next steps into handling vector data more efficiently, e.g. by going through duckdb.

[cboettig]

That is quite cool! Yup, looks like they are using geoparquet, and docs include duckdb examples as well as utilities for pmtiles for efficient visualization.

[goergen95]

Paper on the technical implementation of the DOPA processing back-end.

[goergen95]

Annual global total estimated wildfire carbon emissions from 2003 to 2023, Source: CAMS

[goergen95]

R package post to streamline vector datacubes for polygons that change over space and time.

[fBedecarrats]

Interesting!

[fBedecarrats]

3ie has launched a new inventory of remote-sensing indicators for development, with this blog providing some background. “If you are interested in using geospatial data, or are interpreting geospatial data, you can navigate here, to your sector of interest, and then be presented with a list of RS indicators and which social science variables they are used as a proxy for. You can find descriptions and pros and cons for each indicator, and then can explore the paper repository to find methodological and empirical papers related to that indicator” “The largest categories were agriculture (37 papers), firms (30), education (29), social protection (24), and macroeconomics (20).”. He then provides one sentence summaries of the education papers.
Source

[fBedecarrats]

Interesting resource here: rspatialdata is a collection of data sources and tutorials on visualising spatial data using R

[zivankaraman]

@fBedecarrats, thank you for this interesting info.