Merge pull request #9 from DanCardin/dc/view

Author: DanCardin

Makefile

@@ -13,12 +13,10 @@ test:
 	coverage xml
 
 lint:
-	flake8 --ignore=E203,W503 src tests
-	isort --check-only src tests
-	pydocstyle src tests
+	ruff src tests
 	mypy src tests
 	black --check src tests
 
 format:
-	isort src tests
+	ruff src tests --fix
 	black src tests

README.md

@@ -126,43 +126,56 @@ include database specific objects.
 
 ## Alembic-utils
 
-Currently, the set of supported declarative objects is essentially non-overlapping with
+Currently, the set of supported declarative objects is largely non-overlapping with
 [Alembic-utils](https://github.com/olirice/alembic_utils). However in principle, there's
-no reason that objects supported by this library couldn't begin to overlap (views, functions,
+no reason that objects supported by this library couldn't begin to overlap (functions,
 triggers); and one might begin to question when to use which library.
 
-First, it's likely that this library can/should grow handlers for objects already supported by
-alembic-utils. In particular, it's likely that any future support in this library for something
-like a view could easily accept an `alembic_utils.pg_view.PGView` definition and handle it directly.
-The two libraries are likely fairly complementary in that way, although it's important to note
-some of the differences.
+Note that where possible this library tries to support alembic-utils native objects
+as stand-ins for the objects defined in this library. For example, `alembic_utils.pg_view.PGView`
+can be declared instead of a `sqlalchemy_declarative_extensions.View`, and we will internally
+coerce it into the appropriate type. Hopefully this eases any transitional costs, or
+issues using one or the other library.
 
 Alembic utils:
 
-- Is more directly tied to Alembic and specifically provides functionality for autogenerating
-  DDL for alembic, as the name might imply. It does **not** register into sqlalchemy's event
-  system.
-- Requires one to explicitly find/include the objects one wants to track with alembic.
-- It provides direct translation of individual entities (like a single, specific `PGGrantTable`).
-- In most cases, it appears to define a very "literal" interface (for example, `PGView` accepts
-  the whole view definition as a raw literal string), rather than an abstracted one.
+1. Is more directly tied to Alembic and specifically provides functionality for autogenerating
+   DDL for alembic, as the name might imply. It does **not** register into sqlalchemy's event
+   system.
+
+2. Requires one to explicitly find/include the objects one wants to track with alembic.
+
+3. Declares single, specific object instances (like a single, specific `PGGrantTable`). This
+   has the side-effect that it can only track included objects. It cannot, for example,
+   remove objects which should not exist due to their omission.
+
+4. In most cases, it appears to define a very "literal" interface (for example, `PGView` accepts
+   the whole view definition as a raw literal string), rather than attempting to either abstract
+   the objects or accept abstracted (like a `select` object) definition.
+
+5. Appears to only be interested in supporting PostgreSQL.
 
 By contrast, this library:
 
-- SqlAlchemy is the main dependency and registration point. The primary function of the library
-  is to register into sqlalchemy's event system to ensure that a `metadata.create_all` performs
-  the requisite statements to ensure the state of the database matches the declaration.
-
-  This library does **not** require alembic, but it does (optionally) perform a similar function
-  by way of enabling autogeneration support for non-native objects.
-
-- Perhaps a technical detail, but this library registers the declaratively stated objects directly
-  on the metadata/declarative-base. This allows the library to automatically know the intended
-  state of the world, rather than needing to discover objects.
-- The intended purpose of the supported objects is to declare what the state of the world **should**
-  look like. Therefore the function of this library includes the (optional) **removal** of objects
-  detected to exist which are not declared (much like alembic does for tables). Whereas alembic-utils
-  only operates on objects you create entities for.
-- As much as possible, this library provides more abstracted interfaces for defining objects.
-  This is particularly important for objects like roles/grants where not every operation is a create
-  or delete (in contrast to something like a view).
+1. SqlAlchemy is the main dependency and registration point (Alembic is, in fact, an optional dependency).
+   The primary function of the library is to declare the underlying objects. And then registration into
+   sqlalchemy's event system, or registration into alembic's detection system are both optional features.
+
+2. Perhaps a technical detail, but this library registers the declaratively stated objects directly
+   on the metadata/declarative-base. This allows the library to automatically know the intended
+   state of the world, rather than needing to discover objects.
+
+3. The intended purpose of the supported objects is to declare what the state of the world **should**
+   look like. Therefore the function of this library includes the (optional) **removal** of objects
+   detected to exist which are not declared (much like alembic does for tables).
+
+4. As much as possible, this library provides more abstracted interfaces for defining objects.
+   This is particularly important for objects like roles/grants where not every operation is a create
+   or delete (in contrast to something like a view), where a raw SQL string makes it impossible to
+   diff two different a-like objects.
+
+5. Tries to define functionality in cross-dialect terms and only where required farm details out to
+   dialect-specific handlers. Not to claim that all dialects are treated equally (currently only
+   PostgreSQL has first-class support), but technically, there should be no reason we wouldn't support
+   any supportable dialect. Today SQLite (for whatever that's worth), and MySQL have **some** level
+   of support.

docs/source/api.md

@@ -1,14 +1,14 @@
 # API
 
-## Schemas
-
 ```{eval-rst}
 .. autoapimodule:: sqlalchemy_declarative_extensions.schema
-   :members:
-   :noindex:
+   :members: Schema, Schemas
 ```
 
-## Roles
+```{eval-rst}
+.. autoapimodule:: sqlalchemy_declarative_extensions.view
+   :members: Views, View, view, register_view
+```
 
 ```{eval-rst}
 .. autoapimodule:: sqlalchemy_declarative_extensions.role.base
@@ -20,16 +20,19 @@
    :members: Role
 ```
 
-## Grants
-
 ```{eval-rst}
 .. autoapimodule:: sqlalchemy_declarative_extensions.grant
    :members: Grants
 ```
 
-## Rows
-
 ```{eval-rst}
 .. autoapimodule:: sqlalchemy_declarative_extensions.row
+   :members: Row, Rows
+```
+
+## Alembic
+
+```{eval-rst}
+.. autoapimodule:: sqlalchemy_declarative_extensions.alembic
    :members:
 ```

docs/source/conf.py

@@ -41,7 +41,10 @@
     "tasklist",
 ]
 
-intersphinx_mapping = {"python": ("https://docs.python.org/3", None)}
+intersphinx_mapping = {
+    "python": ("https://docs.python.org/3", None),
+    "sqlalchemy": ("https://docs.sqlalchemy.org/en/14/", None),
+}
 
 autoapi_type = "python"
 autoapi_dirs = ["../../src/sqlalchemy_declarative_extensions"]

docs/source/index.md

@@ -16,6 +16,7 @@ API <api>
 :hidden:
 
 Schemas <schemas>
+Views <views>
 Roles <roles>
 Grants <grants>
 Rows <rows>

docs/source/views.md

@@ -0,0 +1,161 @@
+# Views
+
+Views definition and registration can be performed exactly as it is done with other object
+types, by defining the set of views on the `MetaData` or declarative base, like so:
+
+```python
+from sqlalchemy.ext.declarative import declarative_base
+from sqlalchemy_declarative_extensions import declarative_database, View, Views
+
+_Base = declarative_base()
+
+
+@declarative_database()
+class Base(_Base):
+    __abstract__ = True
+
+    views = Views().are(
+        View("foo", "select * from bar where id > 10", schema="baz"),
+    )
+```
+
+And if you want to define views using raw strings, or otherwise not reference the tables
+produced off the `MetaData`, then this is absolutely a valid way to organize.
+
+## The `view` decorator
+
+However views differ from most of the other object types, in that they are convenient to
+define **in terms of** the tables they reference (i.e. your existing set of models/tables).
+In fact personally, all of my views are produced from [select](sqlalchemy.sql.expression.select) expressions
+referencing the underlying [Table](sqlalchemy.schema.Table) object.
+
+This commonly introduce a circular reference problem wherein your tables/models are defined
+through subclassing the declarative base, which means your declarative base cannot then
+have the views statically defined **on** the base (while simultaneously referencing those models).
+
+```{note}
+There are ways of working around this in SQLAlchemy-land. For example by creating a ``MetaData``
+ahead of time and defining all models in terms of their underlying ``Table``.
+
+Or perhaps by using SQLAlchemy's mapper apis such that you're not subclassing the declarative base
+for models.
+
+In any case, these options are more complex and probably atypical. As such, we cannot assume
+you will adopt them.
+```
+
+For everyone else, the [view](sqlalchemy_declarative_extensions.view) decorator is meant to be the
+solution to that problem.
+
+This strategy allows one to organize their views alongside the models/tables those
+views happen to be referencing, without requiring the view be importable at MetaData/model base
+definition time.
+
+### Option 1
+
+```python
+from sqlalchemy import Column, types, select
+from sqlalchemy.ext.declarative import declarative_base
+from sqlalchemy_declarative_extensions import view
+
+Base = declarative_base()
+
+
+class Foo(Base):
+    __tablename__ = 'foo'
+
+    id = Column(types.Integer, primary_key=True)
+
+
+@view()
+class Bar1(Base):
+    __tablename__ = 'bar'
+    __view__ = select(Foo.__table__).where(Foo.__table__.id > 10)
+
+    id = Column(types.Integer, primary_key=True)
+```
+
+The primary difference between Options 1 and 2 in the above example is of how the
+resulting classes are seen by SQLAlchemy/Alembic natively.
+
+In the case of `Bar1`, SQLAlchemy/Alembic actually think that class is a normal table.
+Therefore querying the view looks identical to a real table: `session.query(Bar1).all()`
+
+For alembic, this means that alembic thinks you defined a table and will attempt to
+autogenerate it (while this library will also notice it and attempt to autogenerate
+a conflicting view.
+
+In order to use this option, we suggest you use one or both of some utility functions provided
+under the `sqlalchemy_declarative_extensions.alembic`: [ignore_view_tables](sqlalchemy_declarative_extensions.alembic.ignore_view_tables)
+and [compose_include_object_callbacks](sqlalchemy_declarative_extensions.alembic.compose_include_object_callbacks).
+
+Somewhere in your Alembic `env.py`, you will have a block which looks like this:
+
+```python
+with connectable.connect() as connection:
+    context.configure(
+        connection=connection,
+        target_metadata=target_metadata,
+        ...
+    )
+```
+
+The above call to `configure` accepts an `include_object`, which tells alembic to include or ignore
+all detected objects.
+
+```python
+from sqlalchemy_declarative_extensions.alembic import ignore_view_tables
+...
+context.configure(..., include_object=ignore_view_tables)
+```
+
+If you happen to already be using `include_object` to perform filtering, we provide an additional
+utility to more easily compose our version with your own. Although you can certainly manually call
+`ignore_view_tables` directly, yourself.
+
+```python
+from sqlalchemy_declarative_extensions.alembic import ignore_view_tables, compose_include_object_callbacks
+...
+def my_include_object(object, *_):
+    if object.name != 'foo':
+        return True
+    return False
+
+context.configure(..., include_object=compose_include_object_callbacks(my_include_object, ignore_view_tables))
+```
+
+## Option 2
+
+```python
+from sqlalchemy import Column, types, select
+from sqlalchemy.ext.declarative import declarative_base
+from sqlalchemy_declarative_extensions import view
+
+Base = declarative_base()
+
+
+class Foo(Base):
+    __tablename__ = 'foo'
+
+    id = Column(types.Integer, primary_key=True)
+
+
+@view(Base)  # or `@view(Base.metadata)`
+class Bar2:
+    __tablename__ = 'bar'
+    __view__ = select(Foo.__table__).where(Foo.__table__.id > 10)
+```
+
+By contrast, with Option 2, your class is not subclassing `Base`, therefore it's
+not registered as a real table by SQLAlchemy or Alembic. There's no additional
+work required to get them to ignore the table, because it's not one.
+
+Unfortunately, that means you cannot **invisibly** treat it as though it's a normal model,
+largely because it doesn't have the columns enumerated out in the same way.
+
+However we can provide some basic support for treating it as a table as far as the ORM is concerned.
+For example, you can still `session.query(Bar2).all()` directly.
+
+However, in most cases views primarily benefit non-code consumers of the database, because there's
+no practical difference between querying a literal view, versus executing the underlying query
+of that view, through something like `session.execute(Bar2.__view__)`.