You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@hamilton.apache.org by "Dev-iL (via GitHub)" <gi...@apache.org> on 2026/09/27 12:35:50 UTC
[I] Polars database reader/writer record file metadata, so SQL lineage is missing [hamilton]
Dev-iL opened a new issue, #1740:
URL: https://github.com/apache/hamilton/issues/1740
Related: #1720
`PolarsDatabaseReader` and `PolarsDatabaseWriter` (in both `polars_post_1_0_0_extensions.py` and `polars_pre_1_0_0_extension.py`) describe database I/O with `get_file_and_dataframe_metadata`. They pass the SQL query or table name in where a file path is expected:
```python
# PolarsDatabaseReader.load_data
metadata = utils.get_file_and_dataframe_metadata(self.query, df)
# PolarsDatabaseWriter.save_data
return utils.get_file_and_dataframe_metadata(self.table_name, data)
```
This happens whatever connection is passed, including a SQLAlchemy `Engine`.
**Effects**
- The metadata has no `sql_metadata` block. So the `OpenLineageAdapter` can't identify the tables, and `sql_dataset_identity="datasource"` (from #1720) resolves nothing: `sql_datasets()` returns `SQL datasource is unknown; no dataset emitted`.
- Lineage treats database I/O as filesystem I/O:
- a read becomes an input dataset named after the node (e.g. `orders_df`), with `storageLayer="FileSystem"` and the SQL text as its `dataSource` URI;
- a write becomes an output dataset named after the table, also marked `FileSystem`;
- no `sql` job facet is emitted.
- `file_metadata` carries meaningless values, e.g. `path: "SELECT * FROM orders"`, `scheme: ""`, and a note that the file does not exist.
**Reproduction**
```python
import polars as pl
from sqlalchemy import create_engine
from hamilton.plugins.polars_post_1_0_0_extensions import PolarsDatabaseReader
engine = create_engine("sqlite:///a.db") # with a table `orders`
df, md = PolarsDatabaseReader(query="SELECT * FROM orders", connection=engine).load_data(pl.DataFrame)
print(sorted(md)) # ['dataframe_metadata', 'file_metadata']
print(md["file_metadata"]["path"]) # 'SELECT * FROM orders'
```
**Proposed fix**
1. In both Polars database classes, record `get_sql_metadata(query_or_table, results, db_connection=self.connection, operation="read" | "write")` plus `get_dataframe_metadata`, as `PandasSqlReader` and `PandasSqlWriter` do. SQLAlchemy URLs, Engines and Connections, and `sqlite3` connections, then get full datasource identity.
2. Extend `get_sql_source` to recognise the other connections Polars accepts (ADBC connections, connectorx URI strings, cursors). Until then these are left out of lineage with a note, which is safe but gives no lineage.
3. Add Polars read and write tests against SQLite, mirroring the pandas SQL lineage tests.
**Compatibility**
Metadata for Polars database I/O changes from `file_metadata` to `sql_metadata`. Default-mode (`"legacy"`) OpenLineage output changes with it: the fake filesystem datasets are replaced by the legacy SQL datasets. Anyone consuming `file_metadata` from these classes would be affected, so this probably deserves a changelog entry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@hamilton.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Polars database reader/writer record file metadata, so SQL lineage is missing [hamilton]
Posted by "skrawcz (via GitHub) via notifications" <no...@hamilton.apache.org>.
skrawcz closed issue #1740: Polars database reader/writer record file metadata, so SQL lineage is missing
URL: https://github.com/apache/hamilton/issues/1740
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@hamilton.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Polars database reader/writer record file metadata, so SQL lineage is missing [hamilton]
Posted by "namantiwari26 (via GitHub)" <gi...@apache.org>.
namantiwari26 commented on issue #1740:
URL: https://github.com/apache/hamilton/issues/1740#issuecomment-6079285385
hey! i would like to work on this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@hamilton.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Polars database reader/writer record file metadata, so SQL lineage is missing [hamilton]
Posted by "Dev-iL (via GitHub) via notifications" <no...@hamilton.apache.org>.
Dev-iL commented on issue #1740:
URL: https://github.com/apache/hamilton/issues/1740#issuecomment-6713661831
#1743 notes it doesn't fully fix this, since
> more connection types in get_sql_source (ADBC connections, Sessions, cursors) is left for a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@hamilton.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Polars database reader/writer record file metadata, so SQL lineage is missing [hamilton]
Posted by "Dev-iL (via GitHub)" <gi...@apache.org>.
Dev-iL commented on issue #1740:
URL: https://github.com/apache/hamilton/issues/1740#issuecomment-6083363496
@namantiwari26 There's already a PR up for some time addressing this. The best way to help resolve this issue is review the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@hamilton.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org