[GH] (hamilton/main): Workflow run "npm_and_yarn in /contrib/docs for postcss-selector-parser - Update #1637988274" failed!

[GitBox <gi...@apache.org>]

The GitHub Actions job "npm_and_yarn in /contrib/docs for postcss-selector-parser - Update #1637988274" on hamilton.git/main has failed.
Run started by GitHub user dependabot[bot] (triggered by dependabot[bot]).

Head commit for run:
fa786eae13145af0961238b823b3aa8112199f43 / Dev-iL <65...@users.noreply.github.com>
OpenLineage: Automatic SQL datasource metadata and table lineage (#1720)

* OpenLineage: automatic SQL datasource metadata and table lineage

Hamilton's built-in SQL loaders/savers (`@load_from.sql`, `@save_to.sql`,
`from_.sql`/`to.sql`, and the underlying `PandasSqlReader`/`PandasSqlWriter`)
now capture enough datasource context on their own for table-level lineage,
without any custom loader or hand-maintained mapping.

Motivating use case: a graph reads a query joining `orders` and `customers`
from a sales database, aggregates daily revenue in Python, and writes
`daily_revenue` to a separate warehouse database. Previously the emitted SQL
metadata had no table name for a query read, and the OpenLineage adapter
filed writes under the job namespace with a bare table name, so a report
written by one job and read by another could not be connected, and same-named
tables in different databases were indistinguishable. Users had to write a
custom loader and duplicate connection details to get real lineage.

Core (`hamilton/io/utils.py`):
- `get_sql_metadata()` gains optional keyword `db_connection`/`schema`
  params and a version-1.1.0 `sql_metadata` shape: `schema`, `operation`
  (read/write), `source` (dialect, host, port, database, default schema —
  scalars only, never credentials or live objects), and `notes` explaining
  an unresolved source. The legacy two-argument call keeps its keys and
  row-count semantics.
- New `get_sql_source()` inspects SQLAlchemy URL strings, Engines and
  Connections, and raw `sqlite3` connections, read-only: no new connection,
  no transaction, no session-setting query. In-memory SQLite is deliberately
  left unidentified rather than given a shared/fabricated identity.

Pandas SQL I/O (`hamilton/plugins/pandas_extensions.py`):
- `PandasSqlReader`/`PandasSqlWriter` pass their connection and (for the
  writer) explicit schema into the new metadata; the reader's connection
  type is widened to `Any` to match what it already accepted in practice
  (SQLAlchemy Engines/Connections, not just `str | sqlite3.Connection`).

OpenLineage integration (`hamilton/plugins/h_openlineage.py`):
- New reusable `sql_datasets(sql_metadata, operation=None) -> SqlDatasets`
  converts Hamilton's SQL metadata into OpenLineage datasets using the
  `openlineage-sql` parser: every physical table a query reads/writes is
  named per the OpenLineage naming convention (`postgres://host:port` +
  `database.schema.table` for PostgreSQL, `sqlite://path` + `table` for
  SQLite), aliases and CTEs excluded, quoting/case-folding respected.
  Emits nothing, opens nothing, so a future Airflow provider (or anything
  else) can call it directly.
- `OpenLineageAdapter` routes SQL nodes through it, so dataset identity
  comes from the datasource rather than the job namespace — a read of a
  report now resolves to the same identity it was written under. A
  conversion failure is logged (exception type only, never a message that
  might quote a connection string) and the run event is still emitted
  without datasets; the underlying SQL I/O is never affected.

Packaging/CI:
- `openlineage` extra gains `openlineage-sql` (no Windows wheel; marked
  `sys_platform != 'win32'` so the base extra still installs there).
- `.github/workflows/hamilton-main.yml` adds a disposable `postgres:16`
  service and a "Test openlineage" step across the Python matrix.

Tests: `tests/io/test_utils.py` and `tests/plugins/test_pandas_extensions.py`
extended; new `tests/plugins/test_h_openlineage.py` covers dataset naming
(SQLite and PostgreSQL, env-gated via `HAMILTON_TEST_POSTGRES_URL`), schema
precedence, credential-leak negatives, and that metadata/parser failures
never turn a successful SQL read/write into a failed node. A disposable
per-test PostgreSQL schema fixture is added to `tests/conftest.py`.

Docs: `docs/concepts/materialization.rst` gains a "SQL metadata and
lineage" section (fields, supported connections, precedence, failure
behavior); `docs/reference/lifecycle-hooks/OpenLineageAdapter.rst` documents
dataset naming, the identity change from job-scoped SQL datasets, and the
reusable `sql_datasets()` entry point. The `examples/openlineage` example
is rewritten around the revenue-reporting story on two local SQLite files,
runnable with no external server.

Co-Authored-By: Claude Sonnet 5 <no...@anthropic.com>

* Keep OpenLineage SQL dataset identity backward compatible

Follow-up to the maintainer's review. Each decision below, with its context
and the alternatives rejected, is recorded in writeups/adr/:

- 2609-01: OpenLineageAdapter(sql_dataset_identity="legacy" | "datasource").
  The default stays "legacy", which emits exactly what earlier releases
  emitted and needs no openlineage-sql. Leaving it unset warns once
  (FutureWarning) with migration steps. Lineage never fails a node.
  create_input_dataset/create_output_dataset keep their SQL handling.
- 2609-02: SQL metadata keeps the 1.0.0 query/table_name filing, and
  get_sql_metadata(operation=) carries read/write intent. Reads also
  recognise select/with statements in any case (a linear scan).
- 2609-03: datasource mode resolves what a write names by parsing: a plain
  name is the table, a statement contributes its written tables, and
  nothing is guessed without a parse. The job's sql facet comes from the
  statement actually resolved.
- 2609-04: SQLite tables in attached databases are named by their own file.

Docs, examples and tests are updated to match.

Co-Authored-By: Claude Opus 5.5 (1M context) <no...@anthropic.com>

* Emit OpenLineage inputs for reads by non-plain table names

A PandasSqlReader("daily revenue", engine) read succeeds, but datasource
mode re-parsed the name as SQL and emitted no input dataset. A read string
filed under table_name is now the table read when openlineage-sql reports
that no statement starts at its first character; the read succeeded, so
pandas found it as a table.

Anything else stays SQL, so read statements that metadata files under
table_name ("(select * from t)", "values (1)", "pragma ...") are still
never named as datasets. A string starting with "table" is never a name,
since Postgres' TABLE t is a statement the parser rejects at its first
word. An unexpected parser message or no parser leaves the read out with a
note, as before.

Addresses review feedback on apache/hamilton#1720.

Co-Authored-By: Claude Opus 5.5 (1M context) <no...@anthropic.com>

* Install openlineage-sql with the OpenLineage extra on Windows too

openlineage-sql publishes no Windows wheel, but its sdist builds there with
a Rust toolchain. Drop the sys_platform marker so the extra no longer skips
the parser on Windows, and reword the adapter docs and ADRs to describe the
source build instead of assuming the parser is unavailable.

Co-Authored-By: Claude Opus 5.5 (1M context) <no...@anthropic.com>

* Record SQLite file paths with forward slashes on every platform

On Windows the SQLite datasource path, and so the OpenLineage namespace,
used native backslashes (sqlite://C:\Users\...\warehouse.db), which is
unusual in a URI-like namespace. get_sql_source now records absolute SQLite
paths, including attached databases, via Path.as_posix(), giving
sqlite://C:/Users/.../warehouse.db. The conversion happens where the
metadata is produced, the only place that knows which platform the path is
from; Windows accepts forward-slash paths, so the metadata stays usable.

Paths on Linux and macOS are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <no...@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <no...@anthropic.com>

Report URL: https://github.com/apache/hamilton/actions/runs/39320235840

With regards,
GitHub Actions via GitBox