[GH] (hamilton/2609/richer-sql-metadata): Workflow run "Build Sphinx Documentation" is working again!

[GitBox <gi...@apache.org>]

The GitHub Actions job "Build Sphinx Documentation" on hamilton.git/2609/richer-sql-metadata has succeeded.
Run started by GitHub user Dev-iL (triggered by Dev-iL).

Head commit for run:
a0e4f3f7b01db70008712e3c691e42dcd9adb076 / Dev-iL <65...@users.noreply.github.com>
feat: automatic SQL datasource metadata and OpenLineage table lineage

Hamilton's built-in SQL loaders/savers (`@load_from.sql`, `@save_to.sql`,
`from_.sql`/`to.sql`, and the underlying `PandasSqlReader`/`PandasSqlWriter`)
now capture enough datasource context on their own for table-level lineage,
without any custom loader or hand-maintained mapping.

Motivating use case: a graph reads a query joining `orders` and `customers`
from a sales database, aggregates daily revenue in Python, and writes
`daily_revenue` to a separate warehouse database. Previously the emitted SQL
metadata had no table name for a query read, and the OpenLineage adapter
filed writes under the job namespace with a bare table name, so a report
written by one job and read by another could not be connected, and same-named
tables in different databases were indistinguishable. Users had to write a
custom loader and duplicate connection details to get real lineage.

Core (`hamilton/io/utils.py`):
- `get_sql_metadata()` gains optional keyword `db_connection`/`schema`
  params and a version-1.1.0 `sql_metadata` shape: `schema`, `operation`
  (read/write), `source` (dialect, host, port, database, default schema —
  scalars only, never credentials or live objects), and `notes` explaining
  an unresolved source. The legacy two-argument call keeps its keys and
  row-count semantics.
- New `get_sql_source()` inspects SQLAlchemy URL strings, Engines and
  Connections, and raw `sqlite3` connections, read-only: no new connection,
  no transaction, no session-setting query. In-memory SQLite is deliberately
  left unidentified rather than given a shared/fabricated identity.

Pandas SQL I/O (`hamilton/plugins/pandas_extensions.py`):
- `PandasSqlReader`/`PandasSqlWriter` pass their connection and (for the
  writer) explicit schema into the new metadata; the reader's connection
  type is widened to `Any` to match what it already accepted in practice
  (SQLAlchemy Engines/Connections, not just `str | sqlite3.Connection`).

OpenLineage integration (`hamilton/plugins/h_openlineage.py`):
- New reusable `sql_datasets(sql_metadata, operation=None) -> SqlDatasets`
  converts Hamilton's SQL metadata into OpenLineage datasets using the
  `openlineage-sql` parser: every physical table a query reads/writes is
  named per the OpenLineage naming convention (`postgres://host:port` +
  `database.schema.table` for PostgreSQL, `sqlite://path` + `table` for
  SQLite), aliases and CTEs excluded, quoting/case-folding respected.
  Emits nothing, opens nothing, so a future Airflow provider (or anything
  else) can call it directly.
- `OpenLineageAdapter` routes SQL nodes through it, so dataset identity
  comes from the datasource rather than the job namespace — a read of a
  report now resolves to the same identity it was written under. A
  conversion failure is logged (exception type only, never a message that
  might quote a connection string) and the run event is still emitted
  without datasets; the underlying SQL I/O is never affected.

Packaging/CI:
- `openlineage` extra gains `openlineage-sql` (no Windows wheel; marked
  `sys_platform != 'win32'` so the base extra still installs there).
- `.github/workflows/hamilton-main.yml` adds a disposable `postgres:16`
  service and a "Test openlineage" step across the Python matrix.

Tests: `tests/io/test_utils.py` and `tests/plugins/test_pandas_extensions.py`
extended; new `tests/plugins/test_h_openlineage.py` covers dataset naming
(SQLite and PostgreSQL, env-gated via `HAMILTON_TEST_POSTGRES_URL`), schema
precedence, credential-leak negatives, and that metadata/parser failures
never turn a successful SQL read/write into a failed node. A disposable
per-test PostgreSQL schema fixture is added to `tests/conftest.py`.

Docs: `docs/concepts/materialization.rst` gains a "SQL metadata and
lineage" section (fields, supported connections, precedence, failure
behavior); `docs/reference/lifecycle-hooks/OpenLineageAdapter.rst` documents
dataset naming, the identity change from job-scoped SQL datasets, and the
reusable `sql_datasets()` entry point. The `examples/openlineage` example
is rewritten around the revenue-reporting story on two local SQLite files,
runnable with no external server.

Co-Authored-By: Claude Sonnet 5 <no...@anthropic.com>

Report URL: https://github.com/apache/hamilton/actions/runs/35639680558

With regards,
GitHub Actions via GitBox