You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "steFaiz (via GitHub)" <gi...@apache.org> on 2026/01/08 11:29:01 UTC
[PR] [core] add data-evolution info to Files System Table. [paimon]
steFaiz opened a new pull request, #6986:
URL: https://github.com/apache/paimon/pull/6986
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
Add firstRowId, writeCols and IsBlob columns to Files System Table.
<!-- Linking this pull request to the issue -->
<!-- What is the purpose of the change -->
### Tests
None
<!-- List UT and IT cases to verify this change -->
### API and Format
None
<!-- Does this change affect API or storage format -->
### Documentation
None
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] add data-evolution info to Files System Table. [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #6986:
URL: https://github.com/apache/paimon/pull/6986
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] add data-evolution info to Files System Table. [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #6986:
URL: https://github.com/apache/paimon/pull/6986#issuecomment-3726946557
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] add data-evolution info to Files System Table. [paimon]
Posted by "steFaiz (via GitHub)" <gi...@apache.org>.
steFaiz commented on code in PR #6986:
URL: https://github.com/apache/paimon/pull/6986#discussion_r2674562146
##########
paimon-core/src/main/java/org/apache/paimon/table/system/FilesTable.java:
##########
@@ -117,7 +119,10 @@ public class FilesTable implements ReadonlyTable {
new DataField(14, "max_sequence_number", new BigIntType(true)),
new DataField(15, "creation_time", DataTypes.TIMESTAMP_MILLIS()),
new DataField(16, "deleteRowCount", DataTypes.BIGINT()),
- new DataField(17, "file_source", DataTypes.STRING())));
+ new DataField(17, "file_source", DataTypes.STRING()),
+ new DataField(18, "first_row_id", DataTypes.BIGINT()),
+ new DataField(19, "write_cols", DataTypes.ARRAY(DataTypes.STRING())),
+ new DataField(20, "blob", DataTypes.BOOLEAN())));
Review Comment:
Thanks for your explaination! I've adapted the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] add data-evolution info to Files System Table. [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #6986:
URL: https://github.com/apache/paimon/pull/6986#discussion_r2672376488
##########
paimon-core/src/main/java/org/apache/paimon/table/system/FilesTable.java:
##########
@@ -117,7 +119,10 @@ public class FilesTable implements ReadonlyTable {
new DataField(14, "max_sequence_number", new BigIntType(true)),
new DataField(15, "creation_time", DataTypes.TIMESTAMP_MILLIS()),
new DataField(16, "deleteRowCount", DataTypes.BIGINT()),
- new DataField(17, "file_source", DataTypes.STRING())));
+ new DataField(17, "file_source", DataTypes.STRING()),
+ new DataField(18, "first_row_id", DataTypes.BIGINT()),
+ new DataField(19, "write_cols", DataTypes.ARRAY(DataTypes.STRING())),
+ new DataField(20, "blob", DataTypes.BOOLEAN())));
Review Comment:
Remove this, just get info from file name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] add data-evolution info to Files System Table. [paimon]
Posted by "steFaiz (via GitHub)" <gi...@apache.org>.
steFaiz commented on code in PR #6986:
URL: https://github.com/apache/paimon/pull/6986#discussion_r2672452552
##########
paimon-core/src/main/java/org/apache/paimon/table/system/FilesTable.java:
##########
@@ -117,7 +119,10 @@ public class FilesTable implements ReadonlyTable {
new DataField(14, "max_sequence_number", new BigIntType(true)),
new DataField(15, "creation_time", DataTypes.TIMESTAMP_MILLIS()),
new DataField(16, "deleteRowCount", DataTypes.BIGINT()),
- new DataField(17, "file_source", DataTypes.STRING())));
+ new DataField(17, "file_source", DataTypes.STRING()),
+ new DataField(18, "first_row_id", DataTypes.BIGINT()),
+ new DataField(19, "write_cols", DataTypes.ARRAY(DataTypes.STRING())),
+ new DataField(20, "blob", DataTypes.BOOLEAN())));
Review Comment:
Thanks for your advise! I was thinking that this is convenient for sql. For example, if we want to calculate all blob's storage, we just need to write `where T.blob = true` instead of `where T.fileName LIKE '%s.blob'`.
Would you be okay with keeping this boolean field, or do you prefer avoid persisting redundant metadata?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] add data-evolution info to Files System Table. [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #6986:
URL: https://github.com/apache/paimon/pull/6986#discussion_r2672816081
##########
paimon-core/src/main/java/org/apache/paimon/table/system/FilesTable.java:
##########
@@ -117,7 +119,10 @@ public class FilesTable implements ReadonlyTable {
new DataField(14, "max_sequence_number", new BigIntType(true)),
new DataField(15, "creation_time", DataTypes.TIMESTAMP_MILLIS()),
new DataField(16, "deleteRowCount", DataTypes.BIGINT()),
- new DataField(17, "file_source", DataTypes.STRING())));
+ new DataField(17, "file_source", DataTypes.STRING()),
+ new DataField(18, "first_row_id", DataTypes.BIGINT()),
+ new DataField(19, "write_cols", DataTypes.ARRAY(DataTypes.STRING())),
+ new DataField(20, "blob", DataTypes.BOOLEAN())));
Review Comment:
This system table should just reflect the schema of data file meta, so I think it is better to remove this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org