You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "luoyuxia (via GitHub)" <gi...@apache.org> on 2023/03/15 02:01:50 UTC
[GitHub] [iceberg] luoyuxia opened a new issue, #7110: Support to read iceberg table's metadata with Flink
luoyuxia opened a new issue, #7110:
URL: https://github.com/apache/iceberg/issues/7110
### Feature Request / Improvement
Since Flink 1.12, Flink supports to read metadata of table source with [FLIP-107](https://cwiki.apache.org/confluence/display/FLINK/FLIP-107%3A+Handling+of+metadata+in+SQL+connectors), and Flink has an implemententaion for [flink-connnector-kafka](https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/table/KafkaDynamicSource.java) & [flink-connector-files](https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/table/FileSystemTableSource.java).
It'll be valuable for Flink can also read the metadata of iceberg table.
If there is anyone would like to take it, I'm glad to help review.
### Query engine
Flink
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] xuzhiwen1255 commented on issue #7110: Support to read iceberg table's metadata with Flink
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on issue #7110:
URL: https://github.com/apache/iceberg/issues/7110#issuecomment-1475517216
@luoyuxia What does file_path refer to, the path of the table, or the file location
If it's a file location you can look at this https://iceberg.apache.org/docs/latest/spark-queries/#metadata-log-entries
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on issue #7110: Support to read iceberg table's metadata with Flink
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on issue #7110:
URL: https://github.com/apache/iceberg/issues/7110#issuecomment-1475526294
this is already added by @hililiwei in PR #6222
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support to read iceberg table's metadata column with Flink [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #7110: Support to read iceberg table's metadata column with Flink
URL: https://github.com/apache/iceberg/issues/7110
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] xuzhiwen1255 commented on issue #7110: Support to read iceberg table's metadata with Flink
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on issue #7110:
URL: https://github.com/apache/iceberg/issues/7110#issuecomment-1475517817
If flink implements metadata tables, it should also be able to query them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hililiwei commented on issue #7110: Support to read iceberg table's metadata column with Flink
Posted by "hililiwei (via GitHub)" <gi...@apache.org>.
hililiwei commented on issue #7110:
URL: https://github.com/apache/iceberg/issues/7110#issuecomment-1489933400
This is useful, and we can try to implement it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] luoyuxia commented on issue #7110: Support to read iceberg table's metadata column with Flink
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on issue #7110:
URL: https://github.com/apache/iceberg/issues/7110#issuecomment-1475532560
Thanks all for the information. Now I got it. Maybe the issue's title is not clear. Now I change the title from `Support to read iceberg table's metadata with Flink` to `Support icberge table's metadata column`.
What I mean is that we can implemet `SupportsReadingMetadata` for `IcebergTableSource` just like `SparkTable` implements `SupportsMetadataColumns`. So that user can use such Flink SQL to know which file the row comes from by the column `file_path`.
```
CREATE TABLE icberg_t(
id STRING,
file_path STRING METADATA
) with
('connector' = 'iceberge', xxx);
SELECT id, file_path FROM icberg_t;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] luoyuxia commented on issue #7110: Support to read iceberg table's metadata with Flink
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on issue #7110:
URL: https://github.com/apache/iceberg/issues/7110#issuecomment-1475482371
Thanks @xuzhiwen1255 for your informatin. But I expect read metadata with Flink SQL, like:
```
CREATE TABLE icberg_t(
id STRING,
file_path STRING METADATA
) with
('connector' = 'iceberge', xxx);
SELECT file_path FROM icberg_t;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] xuzhiwen1255 commented on issue #7110: Support to read iceberg table's metadata with Flink
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on issue #7110:
URL: https://github.com/apache/iceberg/issues/7110#issuecomment-1469249741
This is implemented in #6222
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support to read iceberg table's metadata column with Flink [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7110:
URL: https://github.com/apache/iceberg/issues/7110#issuecomment-1873064317
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on issue #7110: Support to read iceberg table's metadata column with Flink
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on issue #7110:
URL: https://github.com/apache/iceberg/issues/7110#issuecomment-1475550801
ah. I see now. for kafka source, the metadata columns could be topic and offset
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support to read iceberg table's metadata column with Flink [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7110:
URL: https://github.com/apache/iceberg/issues/7110#issuecomment-1859338977
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org