Skip to content

Generalise table preprocessing#2791

Merged
kosack merged 36 commits into
mainfrom
generalise_table_preprocessing
Oct 10, 2025
Merged

Generalise table preprocessing#2791
kosack merged 36 commits into
mainfrom
generalise_table_preprocessing

Conversation

@Voutsi
Copy link
Copy Markdown

@Voutsi Voutsi commented Jun 30, 2025

Closes #2790

Refactoring the DL2 processing for producing IRFs. The motivation is to use the same code for more usecases that process DL2 data, e.g. the telescope cross calibration.

The refactored module is the irf/preprocessing.py

The default behaviour of the preprocessing remains the same as before.

@Voutsi Voutsi self-assigned this Jun 30, 2025
@Voutsi Voutsi added the module:irf issues related to ctapipe.irf label Jun 30, 2025
@Voutsi Voutsi marked this pull request as draft June 30, 2025 08:34
@mexanick
Copy link
Copy Markdown
Contributor

Overall looks good, however I'm wondering if this preprocessing module should go under ctapipe.io instead of staying under ctapipe.irf as it will be generic now and will be used beyond IRF production. @maxnoe what do you think?

@ctao-dpps-sonarqube

This comment has been minimized.

Copy link
Copy Markdown
Contributor

@LukasBeiske LukasBeiske left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Besides that, looks good to me!
Kind of related #2789

Comment thread src/ctapipe/irf/preprocessing.py Outdated
Comment thread src/ctapipe/irf/preprocessing.py
Comment thread src/ctapipe/irf/preprocessing.py Outdated
Comment thread src/ctapipe/irf/preprocessing.py Outdated
@ctao-dpps-sonarqube

This comment has been minimized.

@Voutsi Voutsi marked this pull request as ready for review July 1, 2025 13:07
Comment thread src/ctapipe/irf/preprocessing.py Outdated
@ctao-dpps-sonarqube

This comment has been minimized.

Comment thread docs/changes/2791.optimization.rst Outdated
Comment thread src/ctapipe/irf/preprocessing.py Outdated
Comment thread src/ctapipe/irf/preprocessing.py Outdated
@ctao-dpps-sonarqube

This comment has been minimized.

Comment thread docs/changes/2791.optimization.rst
@mdebony
Copy link
Copy Markdown
Contributor

mdebony commented Sep 19, 2025

It's ok for me.

calculate_event_weights,
)
from pyirf.utils import calculate_source_fov_offset, calculate_theta
except ModuleNotFoundError:
Copy link
Copy Markdown
Member

@maxnoe maxnoe Sep 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think with this many imports, it's simpler to do something like this:

try:
    from ...
    has_pyirf = True
except ModuleNotFounderror:
    has_pyirf = False

and then raise on has_pyirf is False.

@maxnoe
Copy link
Copy Markdown
Member

maxnoe commented Sep 19, 2025

@mdebony @maxnoe is there anything else we should do to proceed?

Most of the review comments by @kosack were not addressed yet, so the ball is in your park

@Voutsi
Copy link
Copy Markdown
Author

Voutsi commented Sep 19, 2025

@mdebony @maxnoe is there anything else we should do to proceed?

Most of the review comments by @kosack were not addressed yet, so the ball is in your park

Hi @maxnoe , @kosack , I had commented above, after Karl's comments, that some of these comments are out of the scope of what this PR aims. I didn't get a reply, if you agree or not. So what do you think?

@maxnoe
Copy link
Copy Markdown
Member

maxnoe commented Sep 19, 2025

At least the request for issuing a warning and for proper variable names should be addressed.

@Voutsi Voutsi requested review from maxnoe and mexanick September 21, 2025 16:24
mexanick
mexanick previously approved these changes Sep 22, 2025
Copy link
Copy Markdown
Contributor

@mexanick mexanick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks to me the urgent requests has been addressed. I'd prefer to implement the rest in the separate PRs to have them easier to digest and review.


if SimulatedEventsInfo is None:
raise OptionalDependencyMissing("pyirf")
raise ImportError("pyirf is required for this functionality")
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you replace the correct usage of the OptionalDependencyMissing exception here?

maxnoe
maxnoe previously approved these changes Sep 22, 2025
mexanick
mexanick previously approved these changes Sep 22, 2025
@maxnoe
Copy link
Copy Markdown
Member

maxnoe commented Sep 22, 2025

@kosack please have another look here

@mexanick
Copy link
Copy Markdown
Contributor

@kosack please have another look here

also, shouldn't it be bumped in priority in DataPipe Followup to be merged before DL3 production?

@maxnoe maxnoe dismissed stale reviews from mexanick and themself via e94ac42 October 9, 2025 10:03
@mexanick
Copy link
Copy Markdown
Contributor

mexanick commented Oct 9, 2025

Looks like sonar couldn't find the coverage report:

An error occurred while trying to import the coverage report: '/home/runner/work/ctapipe/ctapipe/coverage/coverage.xml'

@maxnoe
Copy link
Copy Markdown
Member

maxnoe commented Oct 9, 2025

I restarted the sonar job, the artifact exists

@maxnoe
Copy link
Copy Markdown
Member

maxnoe commented Oct 9, 2025

The error is that a merge happened between the CI run here creating the coverage report and the running of the sonar job, resulting in a mismatch of line numbers.

I merged main here again, that should fix it.

@ctao-sonarqube
Copy link
Copy Markdown

ctao-sonarqube Bot commented Oct 9, 2025

Copy link
Copy Markdown
Member

@kosack kosack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good now, thanks for addressing my comments.

@kosack kosack merged commit 7590019 into main Oct 10, 2025
13 checks passed
@kosack kosack deleted the generalise_table_preprocessing branch October 10, 2025 12:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module:irf issues related to ctapipe.irf

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Generalise DL2 tables loading and filtering

6 participants