ref: get proper base manager typing

[asottile-sentry]

an absolute ton of work went into enabling this -- but after this PR model types / create / update / etc. should mostly be typechecked whereas they were entirely unchecked (everything Any) prior to this.

most of the core problem was that mypy and django-stubs did not understand our custom BaseManager and BaseQuerySet since there's a significant amount of django stuff that goes into making those classes "real"

fix that involved these issues:

• get_dynamic_class_hook does not handle defer properly python/mypy#17402 (worked around)
• add PlaceholderNode when deferring manager class hook typeddjango/django-stubs#2228 (the workaround)

with that fixed, it exposed a handful issues in django-stubs itself:

• don't copy magic or private methods in from_queryset managers typeddjango/django-stubs#2240
• fix handling of explicit objects annotation typeddjango/django-stubs#2241
• walk mro for Field set type annotations typeddjango/django-stubs#2244
• use field annotations for values_list types typeddjango/django-stubs#2248
• lookup manager type via mro typeddjango/django-stubs#2276
• fill QuerySet generics using the manager's model type typeddjango/django-stubs#2281

fixing all of those together exposed around 1k issues in sentry code itself which was fixed over a number of PRs:

• ref: fix reassignment of key_errors #75186
• ref: fix override of get_or_create #75108
• ref: make create or update typesafe #75149
• ref: avoid filtering on id=None #75162
• ref: fix QuerySet soundness of event_frequency #75150
• ref: culprit is nullable #75154
• ref: fix reassignment of incorrect QuerySet types #75158
• ref: upgrade sentry-forked-django-stubs #75148
• ref: add non-null assertion after guard against query params #75157
• ref: fix lookups by None keys #75156
• ref: fix re-assignment from 4-ary QuerySet to 2-ary list of tuples #75152
• ref: GroupValues.priority is nullable #75153
• ref: allow SnubaParams.teams to take a QuerySet #75151
• ref: use list[Project] for all branches in search.events.builder.base #75113
• ref: use tuples for loops over disparate model classes #75112
• ref: fix erroneous project= passed to ProjectTemplate #75111
• ref: fix num_comments typing in migrations again #75110
• ref: fix reassignment to release (Release vs str) #75109
• ref: upgrade sentry-forked-django-stubs #75013
• ref: add typing-only hints to dict fields for django-stubs #74641
• ref: fix subscription_id None check #74640
• ref: fix Activity nullability issues in sentry.rules.conditions #73783
• ref: organization cannot be None for Dashboard #73787
• ref: fix more cases where HC data must be dict #73788
• ref: explicit None check to resolve nullability properly #73793
• ref: fix nullability in alert_rule serializer #73791
• ref: revert Activity.data assertions #73786
• ref: upgrade sentry-forked-django-stubs #73761
• ref: fix some typing issues in tests when models are checked #73742
• ref: use concrete dict in HC payloads #73744
• ref: improve annotations for Outbox Managers #73602
• ref: fix types for create_or_update to match the requirements of BaseManager #73596
• ref: annotate query type for ReleaseQuerySet #73595
• ref: fix various types for sentry.monitors #72892
• ref: add row type to our custom QuerySet #73589
• ref: fix accessing user_id from Member in tests #73587
• ref: fix more types in tasks #73581
• ref: fix more typing issues in HC modules #73580
• ref: make HybridCloudForeignKey null=True reflected in field types #73213
• ref: use a TypedDict for a few test case dicts #73207
• ref: fix some type issues in tests once models are checked #73206
• ref: rewrite bounded tests to use fields directly #73205
• ref: delete dead test #73202
• ref: annotate another create_project #73198
• ref: fix passing slug=None to HC slug check #73121
• ref: annotate test helper create_project as returning Project #73109
• ref: remove .query star import from sentry.db.models.__init__ #73073
• ref: remove paranoia star import from sentry.db.models.__init__ #73061
• getsentry/getsentry#14370
• ref: use absolute imports for BaseManager #72965
• ref: improve typing of sentry.db.models #72963
• ref: have models.User extend Model instead of BaseModel #72967
• ref: add mypy plugin to fix referencing .objects through a TypeVar #72877 (eventually was able to remove this when upgrading to mypy 1.11)
• ref: reduce the surface area of sentry.db.models.manager.__init__ #72948
• ref: remove dead code in models.eventattachment #72799
• ref: fix types for integrations tasks #72898
• ref: fix types for various tasks #72899
• ref: fix types in org provisioning #72896
• ref: fix types in sentry.rules.processing #72895
• ref: fix more types for TSDB #72903
• ref: fix some types in sentry.replays #72894
• ref: fix types for EventUser #72897
• ref: use connection name directly in receivers.core #72893
• ref: fix type for Organization's option_manager #72889
• ref: properly genericize AlertRuleSerializer #72887
• ref: fix return type for TeamKeyTransactionSerializer #72888
• ref: fix type errors in sentry.incidents #72811
• ref: improve typing for sentry.db.models.query #72872
• ref: fix some types for sentry.issues #72813
• ref: reset_options only supports project #72815
• ref: fix types for sentry.integrations when BaseManager is typechecked #72812
• ref: fix types for sentry.auth.access #72808
• ref: fix types in several models #72802
• ref: fix typing for sentry.web.frontend.accounts #72807
• ref: fix typing for sentry.buffer.base #72809
• ref: fix typing for sentry.debug_files.artifact_bundles #72810
• ref: fix some types for sentry.notifications #72814
• ref: fix typing for sentry.profiles #72816
• ref: fix types for relay.config.metric_extraction #72818
• ref: fix a few type errors in sentry testutils #72806
• ref: remove unused UserRole.permissions_for_user #72801
• ref: remove OptionMixin #72797
• ref: fix typing for models.debugfile #72800
• ref: improve typing for sentry.models.integrations.integration_feature #72798
• ref: fix typing for db.models.manager.base #72771
• ref: fix typing for event_id (it is str) #72718
• ref: improve typing of base_query_set #72719
• ref: fix typing in sentry.migrations #72710
• ref: fix types for releases dicts in event_manager #72709
• ref: reuse type definition for Job from event_manager #72706
• ref: pass event data instead of event #72693
• ref: fix some types for sentry.api #72641
• ref: fix typing of sentry.models.groupsubscription #72591
• ref: fix some typing in hybrid cloud modules #72635

mypy 1.11 also included some important improvements with typechecking for model_cls in (M1, M2, M2): ... which also exposed some problems in our code. unfortunately upgrading mypy involved fixing a lot of things as well:

• ref: upgrade mypy to 1.11.0 #75020
• ref: remove CsvMixin #74982
• ref: work around partial missing overload support #74976
• ref: match signature of authenticate_credentials #74974
• ref: unify signature of get_projects_list #74972
• ref: replace CsvMixin with a typesafe alternative #74967
• ref: match signatures of Manager post_save #74954
• getsentry/getsentry#14739
• getsentry/getsentry#14734
• ref: match signature of Signal.connect #74959
• ref: fix clobbered signatures of base deletion task #74958
• ref: fix local reassignment of model class union #74956
• ref: make get_api_context compatible with baseclass #74953
• ref: match signature of Model.delete #74955
• ref: match signature of Func.as_sql #74952
• ref: unify signature of notify_users #74895
• ref: unify signatures of get_issue_label / get_issue_url #74892
• ref: unify signature of get_release_hook #74897
• ref: match base class signature of render(...) for widgets #74896
• ref: unify signature of is_configured #74893
• ref: unify signature of plugin get_config #74880
• ref: use BaseApiClient's implementation of get_cached #74900
• ref: unify signature of create_issue #74902
• ref: unify signature of notify(...) #74903
• ref: unify signature of Webhook.__call__ #74904
• ref: copy signature of _request to plugin api client #74899
• ref: unify signature of validate_config #74894
• ref: use base factories create_environment #74882
• ref: avoid clobbering base class do_request #74881
• ref: unify signature of get_configure_plugin_fields #74871
• ref: unify signature of get_annotations #74870
• ref: match signature of run_test in test_slack #74855
• ref: match signature of feature handler has #74856
• ref: fix some various test mypy errors in mypy 1.11 #74853
• ref: fix serialize method in serializers.test_base #74857
• ref: fix incorrect signature of setUp for SCIMTestCase #74858
• ref: adjust typing so rate_limits can be a callable #74807
• ref: match signatures in mocked bigtable implementation #74805
• ref: unify get_actions signature #74803
• ref: use baseclass implementation of monitor creation #74806
• ref: unify the signature of Interface.to_string #74798
• ref: prevent clobbering create_integration in base class #74809
• ref: fix shadowing of make_request in tests #74801
• ref: match signature of create_event in test_post_process #74804
• ref: remove erroneously overridden do_request #74800
• ref: generated Enum type variables must match name #74799
• ref: improve signatures of tsdb #74725
• ref: fix Future types for sentry_metrics routing_producer #74682
• ref: fix signature overrides for most Serializers #74677
• ref: fix signatures of contribute_to_class #74680
• ref: fix missing Generic in spans.consumer strategy #74683
• ref: unify the signature of get_tag_values #74681

needless to say this is probably the largest stacked PR I've ever done -- and I'm pretty happy with how I was able to split this up into digestable chunks

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.08%. Comparing base (3dc95d7) to head (0d84170).
Report is 5 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #72221      +/-   ##
==========================================
- Coverage   78.09%   78.08%   -0.01%     
==========================================
  Files        6758     6758              
  Lines      301731   301732       +1     
  Branches    51925    51925              
==========================================
- Hits       235624   235617       -7     
- Misses      59761    59765       +4     
- Partials     6346     6350       +4     

Files	Coverage Δ
src/sentry/backup/imports.py	90.35% <100.00%> (ø)
src/sentry/db/models/manager/base.py	75.07% <100.00%> (+0.07%)	⬆️
src/sentry/models/files/abstractfile.py	90.36% <100.00%> (ø)
src/sentry/models/files/abstractfileblob.py	86.66% <100.00%> (ø)
...sentry/organizations/services/organization/impl.py	85.44% <100.00%> (ø)

... and 5 files with indirect coverage changes

[getsantry]

This pull request has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you add the label WIP, I will leave it alone unless WIP is removed ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

[getsantry]

This pull request has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you add the label WIP, I will leave it alone unless WIP is removed ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

[asottile-sentry]

talked a bit with @markstory about these particular models -- we believe there's a way to convert these to free functions to capture the dependent-types inheritance hierarchy with the type system and make these sound -- but as a followup

[asottile-sentry]

the issue here is HC converts the bitfield to an integer over the wire and the assignment then confuses our workaround for bitfields

I plan to follow up by making bitfield an actual mypy plugin which understands the assignments and values properly here

[asottile-sentry]

note that there are a number of "newly identified errors" in ignored files (~50-100) -- I'm deferring fixing those for now and will fix those as they get opted into type checking

[joshuarli]

damnnnnnnnnnnnn