Floating point exception when running MIRACL Ace workflow

[clemencehurni]

I am testing the MIRACL Ace workflow and encounter a Fatal Python error: Floating point exception during processing.
This happens both with the provided sample dataset and with my own dataset.

Error log:

...
Slice: 126240_089000_038960.tif
Slice: 126240_089000_038980.tif

Saving patches for Z-dim to 'output/final_ctn_down_10_rca_voxel_size_25/seg_final/generated_patches/'...

In total, 256 patches have been saved to 'output/final_ctn_down_10_rca_voxel_size_25/seg_final/generated_patches/'!
Patches folder path is: output/final_ctn_down_10_rca_voxel_size_25/seg_final/generated_patches
preparing dataset!
data dicts are ready!
some samples from data dicts: [{'image': 'output/.../patch_0_0.tiff'}, {'image': 'output/.../patch_0_1.tiff'}, {'image': 'output/.../patch_0_10.tiff'}]
in total we have 253 data dicts
model(s) are initialized!
number of available gpus: 1
Trained model(s) are loaded!
Data # 1 , {'image': 'output/.../patch_0_0.tiff'}
Fatal Python error: Floating point exception

The system configuration is the following:

OS: Ubuntu 24.04.2 LTS
Workstation: HP Z840, Intel® Xeon® E5-2687W v3 × 40
GPU: NVIDIA GeForce RTX 5060 Ti
Driver: 575.57.08
CUDA: 12.9

Command used, as in tutorial:

miracl flow ace
--single Ex_561_Em_600_stitched
--sa_output_folder output
--sa_model_type unet
--rca_orient_code ARI
--sa_resolution 3.5 3.5 4.0
--ctn_down 10
--rca_voxel_size 25
--ctn_channame Signal
--sa_batch_size 1

Thanks in advance for your help,
Clémence

[jono3030]

Hi @clemencehurni , thanks a lot for using MIRACL/ACE and for raising a GitHub issue about the error you are experiencing!

We unsuccessfully tried to re-create the floating point exception error on our end. ACE runs successfully on the 25GB example dataset that you were using.

[...]
In total, 256 patches have been saved to 'output_dir/final_ctn_down_10_rca_voxel_size_25/seg_final/generated_patches/'!
Patches folder path is: output_dir/final_ctn_down_10_rca_voxel_size_25/seg_final/generated_patches
preparing dataset!
data dicts are ready!
some samples from data dicts: [{'image': 'output_dir/final_ctn_down_10_rca_voxel_size_25/seg_final/generated_patches/patch_0_0.tiff'}, {'image': 'output_dir/final_ctn_down_10_rca_voxel_size_25/seg_final/generated_patches/patch_0_1.tiff'}, {'image': 'output_dir/final_ctn_down_10_rca_voxel_size_25/seg_final/generated_patches/patch_0_10.tiff'}]
in total we have 253 data dicts
model(s) are initialized!
number of available gpus: 2
Trained model(s) are loaded!
Data # 1 ,  {'image': 'output_dir/final_ctn_down_10_rca_voxel_size_25/seg_final/generated_patches/patch_0_0.tiff'}
Data # 3 ,  {'image': 'output_dir/final_ctn_down_10_rca_voxel_size_25/seg_final/generated_patches/patch_0_10.tiff'}
Data # 5 ,  {'image': 'output_dir/final_ctn_down_10_rca_voxel_size_25/seg_final/generated_patches/patch_0_12.tiff'}
Data # 7 ,  {'image': 'output_dir/final_ctn_down_10_rca_voxel_size_25/seg_final/generated_patches/patch_0_14.tiff'}
Data # 9 ,  {'image': 'output_dir/final_ctn_down_10_rca_voxel_size_25/seg_final/generated_patches/patch_0_16.tiff'}
Data # 11 ,  {'image': 'output_dir/final_ctn_down_10_rca_voxel_size_25/seg_final/generated_patches/patch_0_18.tiff'}
Data # 13 ,  {'image': 'output_dir/final_ctn_down_10_rca_voxel_size_25/seg_final/generated_patches/patch_0_2.tiff'}
Data # 15 ,  {'image': 'output_dir/final_ctn_down_10_rca_voxel_size_25/seg_final/generated_patches/patch_0_21.tiff'}
Data # 17 ,  {'image': 'output_dir/final_ctn_down_10_rca_voxel_size_25/seg_final/generated_patches/patch_0_23.tiff'}
Data # 19 ,  {'image': 'output_dir/final_ctn_down_10_rca_voxel_size_25/seg_final/generated_patches/patch_0_25.tiff'}
[...]

We used an Ubuntu 20.04, Ryzen Threadripper 32 cores, 1TB RAM, 96GB conbined GPU memory (same Nvidia drivers version) processing server.

Would you be able to provide us with some more information about your system:

1. Are you running MIRACL in the WSL2?
2. Which version of MIRACL are you using?
3. Did you use the ./install.sh script to install MIRACL?
4. Could you re-run the analysis and pipe the output into tee (e.g. <miracl_command> | tee output.log) and link to the created file here?

Maybe this is a low level memory issue so it might also make sense to play around with the batch size and the downsample factor. Could you try a batch size of 2 and a downsampling factor of 20?

In the meantime we will try to re-create the error on another system on our end.

Thanks!

[clemencehurni]

Thanks for your quick reply! Here’s the info you requested:

1. No, I’m running a native Linux installation, not WSL2
2. Miracl version: v2.5.0
3. Yes, installed via ./install.sh script from the official MIRACL repository, reinstalled fresh but the issue persists.
4. I re-ran the analysis with batch size 2 and downsampling factor 20 as suggested. Please find the output.log attached.

20250813_MiracleAce_output.log

[jono3030]

Thanks for providing your error log!

The stack trace in the log shows a deadlock occurring during ACE workflow execution, specifically in the deep learning model inference phase. The deadlock is happening in the generate_output_single function during PyTorch's softmax operation. The main thread is stuck at the softmax computation while multiple background threads are waiting in multiprocessing queue operations with this callchain:

• miracl flow ace -> run_ace
• ACEWorkflows.execute_workflow -> _execute_single_workflow
• ACESegmentation.segment -> ace_interface.main
• ace_deploy_model.deploy_functions -> generate_output_single

According to the PyTorch IRC channel this is a known issue with PyTorch's DataLoader when using multiprocessing in certain environments, particularly Docker containers.

So in our case the issue could occur when PyTorch's sliding window inference uses multiple worker processes for data loading and the multiprocessing queues become deadlocked during tensor operations.

I haven't been able to replicate the issue on our end but while I keep trying, could you separately try each of the following so we can further isolate possible causes for the issue:

1. Reduce worker processes: Set num_workers_var=0 in the deploy_functions call to disable multiprocessing. This is done by setting the -saw SA_NR_WORKERS,--sa_nr_workers SA_NR_WORKERS flag to 0 when calling the ACE workflow.
2. Adjust batch size: Lower the batch_size_var parameter to reduce memory pressure. This can be done by setting the -sab SA_BATCH_SIZE, --sa_batch_size SA_BATCH_SIZE flag to 0 when calling the ACE workflow.
3. Use single-threaded mode: Set OMP_NUM_THREADS=1 before running the ACE workflow by adding an environment variable to the existing environment section in your docker-compose.yml:

environment:  
  - DISPLAY  
  - OMP_NUM_THREADS=1

Also, and this might not make a difference, but you should upgrade MIRACL to version 2.5.1.

Let me know if any of these worked. In particular if disabling multiprocessing data loading did the trick.

Thanks!

[nikewindows113225-commits]

Hello,

I'm experiencing the same Fatal Python error: Floating point exception during GPU-accelerated segmentation. The error occurs after the ensemble model is successfully loaded. My system uses an NVIDIA GeForce RTX 5070, and I suspect this may be a compatibility issue between MIRACL's CUDA version and the GPU architecture:

• MIRACL Docker environment: CUDA 11.7 (MIRACL ver 2.5.1)
• RTX 50 series requirement: CUDA 12+

If this is indeed the root cause, could you kindly suggest a solution or workaround?

I have already attempted all the troubleshooting steps mentioned in the previous comments (--sa_nr_workers 0, OMP_NUM_THREADS=1, both model types, etc.), but unfortunately, all attempts resulted in the same floating point exception error.

Any guidance would be greatly appreciated. Thank you!