Image Update - March 2026 by Zeitsperre · Pull Request #159 · Ouranosinc/PAVICS-e2e-workflow-tests

Zeitsperre · 2026-03-17T20:39:46Z

Overview

This PR prepares the next major update of the Jupyter image of PAVICS (2026-03).

The goal of this update is to update the Ouranos library offerings as well as remove several older workarounds and lift several dependency pins. It also updates the base Python from 3.11 to 3.12.

The most significant changes are that the base image is now mambaorg/micromamba, which is the official micromamba image, and jupyterlab is now v4.x (previously 3.x). For the base image, this saves us the work of needing to install micromamba and also saves us the hassle of needing to set channel priority. It's also much slimmer than the previous base image. For the new jupyterlab, many workarounds and the functionality from older/deprecated plugins are now built in. The image not only builds faster (GitHub CI: 17 mins → 15 mins), but it is also slightly smaller (~150 MB from previous).

Building on #157, several system packages have been replaced with conda-forge equivalents. As it stands, the only package installed via apt is the XKCD font (fonts-humor-sans).

launchcontainer and launchnotebook no longer overwrite UID and GID (fails in zsh) and instead now use USER_ID and GROUP_ID for call substitutions. They also no longer use back-tick commands:

VARIABLE = `some-command -flag`

is now:

VARIABLE = "$(some-command -flag)"

I've removed the x11-utils` as I'm not sure if it is actually required (what are we using it for?).

Changes

Replaces base image continuumio/miniconda3 with mambaorg/micromamba.
birdy now uses Python 3.12.
Replaces all conda calls with micromamba ones.
- Installs mamba as a requirement for gator only
Simplifies the two-stage micromamba create for initializing the birdy environment.
Migrates system installation of the following libraries to the environment.yml file:
- firefox (no longer using firefox-esr)
- git
- mercurial
- patch
- unzip
Updates jupyterlab from v3.x to v4.x (current)
Removes the jupyter lab calls for installing/building extensions in Dockerfile.
Adds three new jupyterlab extensions:
- jupyterlab-geojson (geojson file visualizer)
- jupyterlab-logout (dedicated logout button in top-right) and jupyterlab-theme-toggler (dedicated light/dark theme toggle in top-right):
Jupyter env changes:
- Change 1
  - Related PR URL
- Change 2
  - Related PR URL
- Relevant changes (alphabetical order):

<   - somelib=0.8.1=pyh6c4a22f_1
>   - somelib=0.8.4=pyh1a96a4e_0

(...)

Testing Checklist

Deployed as "beta" image in production for bokeh visualization performance regression testing.
Manually tested notebook https://github.com/Ouranosinc/PAVICS-landing/blob/master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb for bokeh visualization performance.
Committed the Jenkins build log to this Pull Request: https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/NEW-release-py###-######/docker/saved_buildout/jenkins-buildlogs-default.txt

Related Issue / Discussion

Matching notebook fixes:
- pavics-sdi: PR URL
- finch: PR URL
- PAVICS-landing: PR URL
- RavenPy: PR URL
- (...)
Deployment to PAVICS: PR URL
Jenkins-config changes for new notebooks: PR URL
Other issues found while working on this one
- Issue 1 URL
- Issue 2 URL
- (...)
Previous release: PR URL

Additional Information

Full diff of the conda env export:
https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/compare/PREVIOUS-release-py###-######...NEW-release-py###-######

Full new conda env export:
https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/NEW-release-py###-######/docker/saved_buildout/conda-env-export.yml

DockerHub build log:
https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/NEW-release-py###-######/docker/saved_buildout/docker-buildlogs.txt

…otlib and pymetalink higher, pin proj<=9.7.1, unpin intake-xarray and jupyter-archive, add vcs tools Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

…mamba Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

Zeitsperre · 2026-03-19T19:24:18Z

I'm currently being held up by a weird error in the latest gator: mamba-org/gator#398

Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

… frameworks, update jupyterlab extensions Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

Zeitsperre · 2026-03-19T21:14:18Z

Note: For the final release, the tag should be py312!

Zeitsperre · 2026-03-20T19:28:49Z

FYI @tlvu

This PR has become a total rewrite; Full details of my approach will follow. I've removed a lot of brittle workarounds and need to speak to you about design and what can/should be broken. Building the image is not only now faster, but also a lot cleaner.

…ion for image operations

tlvu

First pass review, clean up looks good.

…ck, activate birdy environment Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

…ult repo for users who use conda (already set for mamba and micromamba) Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

tlvu · 2026-03-24T23:45:50Z

Do not force push, this breaks all tags that point to older commit hash.

tlvu · 2026-03-24T23:54:27Z

+ENV DEBIAN_FRONTEND=noninteractive
+
+# Install conda and mamba in base for mamba_gator
+RUN micromamba install --yes --name base "conda>=4.7.12" "mamba>=2.0" && \


We have to install conda and mamba? Isn't micromamba a replacement for both of them?

I only have micromamba installed on my machine and I can do everything, just have to remember to type micromamba instead of conda or mamba.

If mamba and conda aren't present, I find that mamba-gator doesn't recognize the base kernel properly. I can try removing it though.

Removing mamba and conda from base. Package resolution is really slow (~3 mins vs 20 seconds):

The birdy environment is also no longer listed.

... [I 2026-03-25 13:37:56.188 ServerApp.mamba_gator] No available packages list in cache. [I 2026-03-25 13:37:56.342 LabApp] Build is up to date [I 2026-03-25 13:38:35.283 ServerApp.mamba_gator] No available packages list in cache. ...

Adding conda and mamba back in:

Oh weird ! So then it was more or less useless to use micromamba as base image ! We could have used the mamba base image instead !

For future reference, we can drop conda according to the docs of mamba_gator. This should be done once we come back to redoing the base image.

tlvu · 2026-03-24T23:57:11Z

+ARG MAMBA_DOCKERFILE_ACTIVATE=1
+
+# Set birdy as the default kernel
+RUN echo "c.MultiKernelManager.default_kernel_name = 'birdy'" >> /etc/jupyter/jupyter_server_config.py


This do not work, remove or comment out to not give false idea.

Are you sure? I am seeing a difference when it comes to the kernel order.

Removing this line:

Default kernel when opening a notebook is ipykernel.

Adding it back in:

Default kernel is now birdy.

Oh, I though, via our discussions, that this kernel setting is to activate the kernel by default. So this is only to change the display ordering of the kernel?

Can you update the comment to clarify it is for default kernel in the selection list display ordering, not for automated activation?

This sets the default ipython kernel in Jupyterlab environment, not the same as activating the birdy environment; These are two different things! I think the comment is clear enough.

tlvu · 2026-03-25T00:04:43Z

+  - mercurial
+  # openssh
+  # https://anaconda.org/channels/conda-forge/packages/openssh/overview
+  - openssh


openssh ?

I installed it for git+ssh support, but I realize now that when people use jupyterlab-git, they aren't necessarily pushing changes back to GitHub.

Do you think users might want to do that though? Would require them pushing ssh credentials into their writeable workspace.

Oh I though (never tried) user could already push using the jupyterlab-git plugin !

You are adding openssh to support git push on the cli? Or because git push using jupyterlab-git plugin do not work?

I think supporting the git cli is a good idea.

Zeitsperre · 2026-03-25T13:29:28Z

Do not force push, this breaks all tags that point to older commit hash.

I don't force push to remove any commits when a tag is involved. Commits with a wip/test and no tag afterward are safe (in my mind) to be destroyed. This is why we do things in branches.

…nment activation Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

tlvu · 2026-03-25T14:33:30Z

Do not force push, this breaks all tags that point to older commit hash.

I don't force push to remove any commits when a tag is involved. Commits with a wip/test and no tag afterward are safe (in my mind) to be destroyed. This is why we do things in branches.

The compare link on github did not show what commits are affected, just the change between the force push.

I was simply reminding you the danger of force push. Agreed that force push that only affect commits after the last tag is safe.

However, in the heat of action, it is easy to forget when is the last tag so best to not use force push at all on a big PR with multiple existing tags. In a small PR, without existing tags, it is OK because the risk of error is much smaller.

…he conda installed in opt/conda Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

Zeitsperre · 2026-03-26T17:53:47Z

Update 2026-03-25: This new base image exposes a few major issues with the spawn commands of our Jupyterhub instance. Once #161 is merged and deployed as the new stable, we should come right back to this.

Problems to resolve:

The continuumio/minconda3 needs to be replaced with something that uses explicitly conda-forge (mambaorg/micromamba probably).
The spawn commands for our Jupyterhub instance need to be tracked down and examined to figure out why the docker run arguments aren't working as expected.
We can probably stand to remove a few dependencies for tools/utilities that aren't required.

It might be worthwhile to reach out to the https://github.com/bird-house/birdhouse-deploy maintainers to see if we can find a solution to some of these problems.

Zeitsperre added 4 commits March 17, 2026 15:43

core library version updates, remove nested-dict, pin numpy and matpl…

2340926

…otlib and pymetalink higher, pin proj<=9.7.1, unpin intake-xarray and jupyter-archive, add vcs tools Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

use micromamba image, move VCS tools to conda env

565bc1a

Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

remove jupyter lab build command, update comments, CMD now uses micro…

4a1bb81

…mamba Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

do not install nodejs, DEBIAN_FRONTEND=noninteractive, pin intake to 2.0

5be5022

Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

Zeitsperre self-assigned this Mar 17, 2026

Zeitsperre added 6 commits March 17, 2026 17:00

use micromamba

20b4ba4

Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

release: update to use image pavics/workflow-tests:py311-260317

eb52112

remove plotly-dash, add exactextract and mamba, raise birdy pin

9ea491e

Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

typo fix, add requests

7377969

Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

you can do this

3fbc508

Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

release: update to use image pavics/workflow-tests:py311-260318

985b001

Zeitsperre added 4 commits March 19, 2026 15:50

use modern $ replacement for better shell support

2d8033a

Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

update jupyterlab version, use python3.12, use conda-provided browser…

29196bc

… frameworks, update jupyterlab extensions Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

let ravenpy manage raven-hydro

b16946c

Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

set workdir to /

ece26fd

Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

Zeitsperre marked this pull request as ready for review March 19, 2026 20:54

release: update to use image pavics/workflow-tests:py311-260319

669fd9b

rewrite Dockerfile to use mambaorg recommendations

3d7cc98

Zeitsperre added 4 commits March 20, 2026 15:41

release: update to use image pavics/workflow-tests:py312-260320

2d80777

TEST try forcing env name

d14e1a0

TEST use args

9b2708a

split jupyter dependencies into base, no longer rely on birdy activat…

f63d414

…ion for image operations

tlvu reviewed Mar 23, 2026

View reviewed changes

WIP - use mamba env vars, update docker-stacks scripts, add healthche…

18b4ebe

…ck, activate birdy environment Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

tlvu reviewed Mar 23, 2026

View reviewed changes

Comment thread docker/Dockerfile Outdated

Zeitsperre added 2 commits March 23, 2026 15:49

drop workdir, copy env files to root dir

3a49639

Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

recombine environment files, add openssh and jupyterlab-unfold

0eb3c9d

Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

Zeitsperre added 4 commits March 24, 2026 14:36

save build artifacts in build-src, clean tmp, set conda-forge as defa…

acb1bb9

…ult repo for users who use conda (already set for mamba and micromamba) Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

add hyperlinks to projects

5b9ec72

Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

release: update to use image pavics/workflow-tests:py312-260324

7068909

re-add shell scripts

db20d18

Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

Zeitsperre force-pushed the update-2026-03 branch from 3a3fc3b to db20d18 Compare March 24, 2026 20:13

Zeitsperre added 2 commits March 24, 2026 16:43

re-add PATH overrides

0bd1dba

Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

release: update to use image pavics/workflow-tests:py312-260324-1

84c3d03

tlvu mentioned this pull request Mar 24, 2026

Add some xHydro notebooks to Jenkins, enabled by default #160

Closed

tlvu reviewed Mar 24, 2026

View reviewed changes

tlvu reviewed Mar 25, 2026

View reviewed changes

undo PATH modification in order to try a different approach to enviro…

4047e0f

…nment activation Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

do not install conda in birdy environment as it takes priority over t…

4d62b0c

…he conda installed in opt/conda Signed-off-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>

Base automatically changed from add-build-tools to master April 15, 2026 17:14

Conversation

Zeitsperre commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Changes

Testing Checklist

Related Issue / Discussion

Additional Information

Uh oh!

Zeitsperre commented Mar 19, 2026

Uh oh!

Zeitsperre commented Mar 19, 2026

Uh oh!

Zeitsperre commented Mar 20, 2026

Uh oh!

tlvu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tlvu commented Mar 24, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Zeitsperre commented Mar 25, 2026

Uh oh!

tlvu commented Mar 25, 2026

Uh oh!

Zeitsperre commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Zeitsperre commented Mar 17, 2026 •

edited

Loading

Zeitsperre commented Mar 26, 2026 •

edited

Loading