You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2025/01/29 17:51:35 UTC
[PR] Support marking columns as system columns via metadata [datafusion]
adriangb opened a new pull request, #14362:
URL: https://github.com/apache/datafusion/pull/14362
Closes #14057
Closes #13975
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "chenkovsky (via GitHub)" <gi...@apache.org>.
chenkovsky commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2623096983
something like this.
```rust
#[tokio::test]
async fn test_name_conflict() {
let batch = record_batch!(
("_rowid", UInt32, [0, 1, 2]),
("_rowid", Utf8, ["file-0", "file-1", "file-2"])
)
.unwrap();
let batch = batch
.with_schema(Arc::new(Schema::new(vec![
Field::new("_rowid", DataType::UInt32, true).to_system_column(),
Field::new("_rowid", DataType::Utf8, true),
])))
.unwrap();
let ctx = SessionContext::new_with_config(
SessionConfig::new().with_information_schema(true),
);
let _ = ctx.register_batch("test", batch);
let select = "SELECT _rowid FROM test";
let df = ctx.sql(select).await.unwrap();
let batchs = df.collect().await.unwrap();
let expected = [
"+--------+",
"| _rowid |",
"+--------+",
"| file-0 |",
"| file-1 |",
"| file-2 |",
"+--------+",
];
assert_batches_sorted_eq!(expected, &batchs);
}
```
Currently DuplicateQualifiedField error is thrown. But it's valid in spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2625954444
See also https://github.com/apache/datafusion/issues/12736
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "chenkovsky (via GitHub)" <gi...@apache.org>.
chenkovsky commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2625266767
> > Another thing I haven't checked for this approach is that system column doesn't only apply on scan. it can also be applied on Projection, Join, SubqueryAlias
>
> Could you give example test cases that you think are relevant? I have a hunch that the current implementation may already handle these correctly.
sorry I have no test. I haven't implemented these features now. because my implementation changes some APIs, so I think it's better to do it when APIs are stable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2624730749
@chenkovsky does aa5abb94774faa9a607361232c6e5784b5ac1744 satisfy the use case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2622510720
@chenkovsky @jayzhan211 could you please review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "chenkovsky (via GitHub)" <gi...@apache.org>.
chenkovsky commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2646049486
could you please also test
```rust
let tb = ctx.table("test").await.unwrap();
tb.write_csv("xxxx.csv", DataFrameWriteOptions::new(), None).await.unwrap();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2819872465
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2626122404
One thing I didn't understand from you PR @chenkovsky: you got it to work only modifying TableScan? I had trouble understanding when I was grokking your PR. If you have a better way to implement this than modifying projection, join, etc. I'm all ears.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2623125556
Is this that important to support? The example seems a bit contrived, I think it'd be more reasonable if it occurred naturally as part of a join or something where a user could unexpectedly run into it. Otherwise it seems to me like something very spark specific.
Postgres for example doesn't let you duplicate system columns:
```sql
create table t (ctid int);
ERROR: column name "ctid" conflicts with a system column name
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2622993100
@chenkovsky could you maybe translate that into a test that we can add? I'm having trouble imagining in what sorts of situations this would apply. Generally in SQL if you have two columns with the same name it's an error (and this is enforced by `DFSchema`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via metadata [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2622449713
I don't know if there's any other "known" metadata, but I feel like it would be good to have an extension trait along the lines of:
```rust
/// Extension of [`Field`] to manage DataFusion specific field metadata.
pub trait DataFusionFieldInfo {
/// A nice docstring!
fn is_system_column(&self) -> bool;
/// Another nice docstring :)
fn as_system_column(mut self) -> Self;
}
impl DataFusionFieldInfo for Field { ... }
```
And then we can call `field.is_system_column()` from anywhere that needs to check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2623148014
I will still give a shot at adding that feature tomorrow. But I'm not sold on the behavior being ideal even if that's what Spark does. Besides if it's an error now we can always make it not error in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "chenkovsky (via GitHub)" <gi...@apache.org>.
chenkovsky commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2622977789
as described in document, https://spark.apache.org/docs/3.5.1/api/java/org/apache/spark/sql/connector/catalog/SupportsMetadataColumns.html
If a table column and a metadata column have the same name, the conflict is resolved by either renaming or suppressing the metadata column. I think we should also guarantee suppressing the metadata column in case of arbitrary fields order.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #14362: Support marking columns as system columns via Field's metadata
URL: https://github.com/apache/datafusion/pull/14362
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2625004316
> Another thing I haven't checked for this approach is that system column doesn't only apply on scan. it can also be applied on Projection, Join, SubqueryAlias
Could you give example test cases that you think are relevant? I have a hunch that the current implementation may already handle these correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2625987325
I see. That makes sense for something like row_id: once you write it to a new file it may not be valid anymore. But for other metadata you might want to round trip it as a system column. I think the argument could be made both ways.
But I do think there is a stronger argument for doing something fancier:
```sql
WITH cte AS (
SELECT id, _row_id FROM t
)
SELECT * FROM cte
```
IMO _row_id should be included.
So should we say that a system column stops being a system column once it's projected?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#discussion_r1946315799
##########
datafusion/common/src/dfschema.rs:
##########
@@ -1056,6 +1079,107 @@ pub fn qualified_name(qualifier: Option<&TableReference>, name: &str) -> String
}
}
+/// Extension trait to manage DataFusion specific metadata on Arrow fields.
+pub trait FieldExt {
+ /// Check if this field is a system columns.
+ ///
+ /// System columns are columns which meant to be semi-public stores of the internal details of the table.
+ /// For example, `ctid` in Postgres would be considered a metadata column
+ /// (Postgres calls these "system columns", see [the Postgres docs](https://www.postgresql.org/docs/current/ddl-system-columns.html) for more information and examples.
+ /// Spark has a `_metadata` column that it uses to include details about each file read in a query (see [Spark's docs](https://docs.databricks.com/en/ingestion/file-metadata-column.html)).
+ ///
+ /// DataFusion allows fields to be declared as metadata columns by setting the `datafusion.system_column` key in the field's metadata
+ /// to `true`.
+ ///
+ /// As an example of how this works in practice, if you have the following Postgres table:
+ ///
+ /// ```sql
+ /// CREATE TABLE t (x int);
+ /// INSERT INTO t VALUES (1);
+ /// ```
+ ///
+ /// And you do a `SELECT * FROM t`, you would get the following schema:
+ ///
+ /// ```text
+ /// +---+
+ /// | x |
+ /// +---+
+ /// | 1 |
+ /// +---+
+ /// ```
+ ///
+ /// But if you do `SELECT ctid, * FROM t`, you would get the following schema (ignore the meaning of the value of `ctid`, this is just an example):
+ ///
+ /// ```text
+ /// +-----+---+
+ /// | ctid| x |
+ /// +-----+---+
+ /// | 0 | 1 |
+ /// +-----+---+
+ /// ```
+ fn is_system_column(&self) -> bool;
+
+ /// Mark this field as a system column.
+ ///
+ /// See [`FieldExt::is_system_column`] for more information on what a system column is.
+ fn to_system_column(self) -> Self;
+
+ /// Mark this field as a non system column by removing the `datafusion.system_column` key from the field's metadata.
+ ///
+ /// See [`FieldExt::is_system_column`] for more information on what a system column is.
+ fn to_non_system_column(self) -> Self;
+}
+
+/// See [`FieldExt`].
+impl FieldExt for Field {
+ /// Check if this field is a system column.
+ /// See [`FieldExt::is_system_column`] for more information on what a system column is.
+ fn is_system_column(&self) -> bool {
+ self.metadata()
+ .get("datafusion.system_column")
Review Comment:
Make it a const?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2845809583
See discussion on https://github.com/apache/datafusion/issues/15173
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2623202562
Was that working in #14057? I didn't see a test for it.
Hypothetically speaking we could do something in DFSchema to deduplicate but I worry that won't make it work e.g. we'll end up with mismatched fields and arrays.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "chenkovsky (via GitHub)" <gi...@apache.org>.
chenkovsky commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2625942249
> I don't think (2) or (3) are real concerns. Things manipulating metadata should already be careful about not clobbering existing metadata and I really, really don't think the performance of inserting into a small HashMap will ever be an issue.
for(3) give an example. many file formats contain field metadata, e.g. parquet. if input file contains the magic metadata in this approach, normal field will also be treated as system field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "chenkovsky (via GitHub)" <gi...@apache.org>.
chenkovsky commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2623174593
> Is this that important to support? The example seems a bit contrived, I think it'd be more reasonable if it occurred naturally as part of a join or something where a user could unexpectedly run into it. Otherwise it seems to me like something very spark specific.
>
> Postgres for example doesn't let you duplicate system columns:
>
> ```sql
> create table t (ctid int);
> ERROR: column name "ctid" conflicts with a system column name
> ```
because spark is not a database, it's a compute engine. For databases such as postgres, it can manipulate schema and data by itself. so it's ok to disable system column and normal column name conflict. but for spark, the data comes from other systems. it cannot guarantee data doesn't contain these conflict fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2624991055
I don't think (2) or (3) are real concerns. Things manipulating metadata should already be careful about not clobbering existing metadata and I really, really don't think the performance of inserting into a small HashMap will ever be an issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "chenkovsky (via GitHub)" <gi...@apache.org>.
chenkovsky commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2646050299
because in this PR system columns are propagated by default. I guess we have to write a lot of monkey patch to prevent system column leak.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "chenkovsky (via GitHub)" <gi...@apache.org>.
chenkovsky commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2624080673
Yes, it supports. it's my fault. I have added the missing ut.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "chenkovsky (via GitHub)" <gi...@apache.org>.
chenkovsky commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2625969618
> > > So you're worried that if a Parquet file has this special metadata on a field we will wrongly interpret it as a system column? Or are you saying that's a good thing / the goal?
> >
> >
> > I'm worried about it.
>
> You're worried that someone by accident (eg because it was written by another system) would add field metadata with the key `datafusion.system_column` and the value `true` and that their intention would not be for DataFusion to interpret that as a system column?
event not by accident. sometimes we select _rowid and save it in parquet. the parquet will contain magic metadata.
then we read this parquet. we get two system columns _rowid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "chenkovsky (via GitHub)" <gi...@apache.org>.
chenkovsky commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2625954231
> So you're worried that if a Parquet file has this special metadata on a field we will wrongly interpret it as a system column? Or are you saying that's a good thing / the goal?
I'm worried about it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2626145613
For now I pushed https://github.com/apache/datafusion/pull/14362/commits/c2b58ee1632b7090188f4f7d5af6c78fbf40462e which has a test that I think covers all of the cases discussed.
So our job now is to get those tests passing, using either the approach in this PR or in #14057.
It seems to me the main difference is how to identify and track metadata columns. But the hooks needed into Projection, Join, etc. should be the same. Do you agree @chenkovsky ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#discussion_r1964480019
##########
datafusion/common/src/dfschema.rs:
##########
@@ -1056,6 +1079,107 @@ pub fn qualified_name(qualifier: Option<&TableReference>, name: &str) -> String
}
}
+/// Extension trait to manage DataFusion specific metadata on Arrow fields.
+pub trait FieldExt {
+ /// Check if this field is a system columns.
+ ///
+ /// System columns are columns which meant to be semi-public stores of the internal details of the table.
+ /// For example, `ctid` in Postgres would be considered a metadata column
+ /// (Postgres calls these "system columns", see [the Postgres docs](https://www.postgresql.org/docs/current/ddl-system-columns.html) for more information and examples.
+ /// Spark has a `_metadata` column that it uses to include details about each file read in a query (see [Spark's docs](https://docs.databricks.com/en/ingestion/file-metadata-column.html)).
+ ///
+ /// DataFusion allows fields to be declared as metadata columns by setting the `datafusion.system_column` key in the field's metadata
+ /// to `true`.
+ ///
+ /// As an example of how this works in practice, if you have the following Postgres table:
+ ///
+ /// ```sql
+ /// CREATE TABLE t (x int);
+ /// INSERT INTO t VALUES (1);
+ /// ```
+ ///
+ /// And you do a `SELECT * FROM t`, you would get the following schema:
+ ///
+ /// ```text
+ /// +---+
+ /// | x |
+ /// +---+
+ /// | 1 |
+ /// +---+
+ /// ```
+ ///
+ /// But if you do `SELECT ctid, * FROM t`, you would get the following schema (ignore the meaning of the value of `ctid`, this is just an example):
+ ///
+ /// ```text
+ /// +-----+---+
+ /// | ctid| x |
+ /// +-----+---+
+ /// | 0 | 1 |
+ /// +-----+---+
+ /// ```
+ fn is_system_column(&self) -> bool;
+
+ /// Mark this field as a system column.
+ ///
+ /// See [`FieldExt::is_system_column`] for more information on what a system column is.
+ fn to_system_column(self) -> Self;
+
+ /// Mark this field as a non system column by removing the `datafusion.system_column` key from the field's metadata.
+ ///
+ /// See [`FieldExt::is_system_column`] for more information on what a system column is.
+ fn to_non_system_column(self) -> Self;
+}
+
+/// See [`FieldExt`].
+impl FieldExt for Field {
+ /// Check if this field is a system column.
+ /// See [`FieldExt::is_system_column`] for more information on what a system column is.
+ fn is_system_column(&self) -> bool {
+ self.metadata()
+ .get("datafusion.system_column")
+ .map(|v| v.to_lowercase().starts_with("t"))
+ .unwrap_or(false)
+ }
+
+ /// Mark this field as a system column.
+ /// See [`FieldExt::to_system_column`] for more information on what a system column is.
+ fn to_system_column(mut self) -> Self {
+ let mut metadata = self.metadata().clone();
+ metadata.insert("datafusion.system_column".to_string(), "true".to_string());
+ self.set_metadata(metadata);
+ self
+ }
+
+ /// Mark this field as a non system column by removing the `datafusion.system_column` key from the field's metadata.
+ /// See [`FieldExt::to_non_system_column`] for more information on what a system column is.
+ fn to_non_system_column(mut self) -> Self {
+ let mut metadata = self.metadata().clone();
+ metadata.remove("datafusion.system_column");
+ self.set_metadata(metadata);
+ self
+ }
+}
+
+impl FieldExt for Arc<Field> {
+ /// Check if this field is a system column.
+ /// See [`FieldExt::is_system_column`] for more information on what a system column is.
+ fn is_system_column(&self) -> bool {
+ FieldExt::is_system_column(self.as_ref())
+ }
+
+ /// Mark this field as a system column.
+ /// See [`FieldExt::to_system_column`] for more information on what a system column is.
+ fn to_system_column(self) -> Self {
+ Arc::new(FieldExt::to_system_column(Arc::unwrap_or_clone(self)))
+ }
+
+ /// Mark this field as a non system column by removing the `datafusion.system_column` key from the field's metadata.
+ /// See [`FieldExt::to_non_system_column`] for more information on what a system column is.
+ fn to_non_system_column(self) -> Self {
+ Arc::new(FieldExt::to_non_system_column(Arc::unwrap_or_clone(self)))
+ }
Review Comment:
Can probably make these no op / zero cost by calling `is_system_column()` first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "chenkovsky (via GitHub)" <gi...@apache.org>.
chenkovsky commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2626887571
> One thing I didn't understand from you PR @chenkovsky: you got it to work only modifying TableScan? I had trouble understanding when I was grokking your PR. If you have a better way to implement this than modifying projection, join, etc. I'm all ears.
No, I have no better way. because for different logical plans, the system column propagation strategies are different. Currently in this PR, system column will be propagated by default. it's also not correct. you need to modify some logical plans to stop propagation. In my implementation, system columns are not propagated by default. I need to modify logical plan to do propagation. so I think logical plan should always be changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "chenkovsky (via GitHub)" <gi...@apache.org>.
chenkovsky commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2624764005
> @chenkovsky does 3cfce67b035c82f3c442c7e12fb50c2cfb836a80 satisfy the use case?
yes,it's the case. but here you decided whether this is system field. it should be decided by table provider automatically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#discussion_r1934364024
##########
datafusion/expr/src/utils.rs:
##########
@@ -736,11 +802,18 @@ pub fn exprlist_to_fields<'a>(
.into_iter()
.map(|c| c.flat_name())
.collect();
+ excluded.extend(
+ get_system_columns(wildcard_schema, None)?
+ .into_iter()
+ .map(|c| c.flat_name()),
+ );
Ok(wildcard_schema
.fields_with_qualified(qualifier)
.into_iter()
.filter_map(|field| {
- let flat_name = format!("{}.{}", qualifier, field.name());
+ let flat_name =
+ Column::new(Some(qualifier.clone()), field.name())
+ .flat_name();
Review Comment:
Small nit of the existing code. I'd rather re-use the logic for creating flat names at a trivial cost just to avoid having arbitrary string formatting going on here.
##########
datafusion/expr/src/utils.rs:
##########
@@ -706,6 +770,7 @@ pub fn exprlist_to_fields<'a>(
// Look for exact match in plan's output schema
let wildcard_schema = find_base_plan(plan).schema();
let input_schema = plan.schema();
+ let exprs = exprs.into_iter().collect::<Vec<_>>();
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2624775441
> it should be decided by table provider automatically
I'm not sure I follow. At some point you have to mark a field as a system column. I used explicit batches / schemas just to make the tests easier, but of course you can do it via a `TableProvider` and register that, you just have to return the right schema from [TableProvider::schema](https://docs.rs/datafusion/latest/datafusion/datasource/trait.TableProvider.html#tymethod.schema). I can add cook up an example of a TableProvider with system columns if you'd like.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "chenkovsky (via GitHub)" <gi...@apache.org>.
chenkovsky commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2624968962
i dont mean that this way cannot reach the target. but when i use spark style api i dont need to take care it. it's battery included. no need to call extra api to do it.
please correct me if I'm wrong.
the benefit of this approch is
1. dont need to change api
the drawbacks are
1. table provider developers have to take care of that and call an api to merge system field and normal field.
2. hashmap performance is not good.
3. hashmap need more effort to take care. for example, maybe we have another method that can also change hashmap later. we have to make sure that method wont change this special key value.
Another thing I haven't checked for this approach is that system column doesn't only apply on scan. it can also be applied on
[Projection](https://github.com/apache/spark/blob/19fbeaa04abe71529a707e07a36adf14098e7741/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala#L95)
[Join](https://github.com/apache/spark/blob/19fbeaa04abe71529a707e07a36adf14098e7741/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala#L648)
[SubqueryAlias](https://github.com/apache/spark/blob/19fbeaa04abe71529a707e07a36adf14098e7741/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala#L1657)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "chenkovsky (via GitHub)" <gi...@apache.org>.
chenkovsky commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2645665768
1. could you please add more dataframe tests?
e.g.
```
assert!(ctx.table("test").await.unwrap().union(ctx.table("test").await.unwrap()).unwrap().select(vec![col("_rowid")]).is_err());
```
2. could you please add metadata dict check to file reader, it will make your PR more robust.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2625952339
So you're worried that if a Parquet file has this special metadata on a field we will wrongly interpret it as a system column? Or are you saying that's a good thing / the goal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2624999006
For (1) yes implementers of TableProvider will have to decide what fields they return, but isn't that already the case? I don't know how DataFusion would automatically determine what is a system column or not. It really depends on the data source.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2628399078
Makes sense.
I *think* I've gotten all of the important cases here working, could you take a look at the tests and see if you are happy with them?
I will note that if we drop just *one* of the requirements from Spark, namely the resolution of name conflicts in favor of non-system columns, the whole thing becomes a lot simpler: hiding system columns from `SELECT *` is not that difficult, but overriding how name collisions are resolved in the many places they can happen is a lot more involved. Although I think I did it correctly in this PR I fear that the extent of the modification to the rest of the codebase and risk of things falling through the cracks in the future (e.g. if someone adds a new join type or otherwise does more invasive work of their own) is higher than the other part of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2625961004
> > So you're worried that if a Parquet file has this special metadata on a field we will wrongly interpret it as a system column? Or are you saying that's a good thing / the goal?
>
>
>
> I'm worried about it.
You're worried that someone by accident (eg because it was written by another system) would add field metadata with the key `datafusion.system_column` and the value `true` and that their intention would not be for DataFusion to interpret that as a system column?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "chenkovsky (via GitHub)" <gi...@apache.org>.
chenkovsky commented on PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2624083257
by the way, I also encapsulate METADATA_OFFSET.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #14362:
URL: https://github.com/apache/datafusion/pull/14362#discussion_r1964478571
##########
datafusion/common/src/dfschema.rs:
##########
@@ -1056,6 +1079,107 @@ pub fn qualified_name(qualifier: Option<&TableReference>, name: &str) -> String
}
}
+/// Extension trait to manage DataFusion specific metadata on Arrow fields.
+pub trait FieldExt {
+ /// Check if this field is a system columns.
+ ///
+ /// System columns are columns which meant to be semi-public stores of the internal details of the table.
+ /// For example, `ctid` in Postgres would be considered a metadata column
+ /// (Postgres calls these "system columns", see [the Postgres docs](https://www.postgresql.org/docs/current/ddl-system-columns.html) for more information and examples.
+ /// Spark has a `_metadata` column that it uses to include details about each file read in a query (see [Spark's docs](https://docs.databricks.com/en/ingestion/file-metadata-column.html)).
+ ///
+ /// DataFusion allows fields to be declared as metadata columns by setting the `datafusion.system_column` key in the field's metadata
+ /// to `true`.
+ ///
+ /// As an example of how this works in practice, if you have the following Postgres table:
+ ///
+ /// ```sql
+ /// CREATE TABLE t (x int);
+ /// INSERT INTO t VALUES (1);
+ /// ```
+ ///
+ /// And you do a `SELECT * FROM t`, you would get the following schema:
+ ///
+ /// ```text
+ /// +---+
+ /// | x |
+ /// +---+
+ /// | 1 |
+ /// +---+
+ /// ```
+ ///
+ /// But if you do `SELECT ctid, * FROM t`, you would get the following schema (ignore the meaning of the value of `ctid`, this is just an example):
+ ///
+ /// ```text
+ /// +-----+---+
+ /// | ctid| x |
+ /// +-----+---+
+ /// | 0 | 1 |
+ /// +-----+---+
+ /// ```
+ fn is_system_column(&self) -> bool;
+
+ /// Mark this field as a system column.
+ ///
+ /// See [`FieldExt::is_system_column`] for more information on what a system column is.
+ fn to_system_column(self) -> Self;
+
+ /// Mark this field as a non system column by removing the `datafusion.system_column` key from the field's metadata.
+ ///
+ /// See [`FieldExt::is_system_column`] for more information on what a system column is.
+ fn to_non_system_column(self) -> Self;
+}
+
+/// See [`FieldExt`].
+impl FieldExt for Field {
+ /// Check if this field is a system column.
+ /// See [`FieldExt::is_system_column`] for more information on what a system column is.
+ fn is_system_column(&self) -> bool {
+ self.metadata()
+ .get("datafusion.system_column")
Review Comment:
👍🏻 will do next iteration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org