[GitHub] [iceberg] wangmiao1981 opened a new pull request #2290: [WIP]: Add Index File Metadata into Iceberg

[GitBox <gi...@apache.org>]

wangmiao1981 opened a new pull request #2290:
URL: https://github.com/apache/iceberg/pull/2290


   As we discussed in the sync up, we plan to define Iceberg index interfaces to enable plugin different Skip Indexes. In general, there should be three areas of work:
   
   1). Adding Index File Metadata into Iceberg: Data File and Index File should have one to one Mapping. Index File Metadata includes file location (always <DataFile Parent Folder Path>/Index/x.index.data), file size and recordcount (its corresponding Data File records, maybe redundant). Since Data File and Index File are closely coupled, Index File doesn't need to include Partition information. Then, we need to update ManifestWriter to write Index File metadata;
   
   2). Index Data content: As we discussed in the sync up, the Index Content should be Blob format, including a header (index type, column name, version, file Pos of index) and contents (e.g., index binaries);
   
   3). Load and use the index while filter push down;
   
   This PR mainly focus on 1). I submit this WIP PR to get some early feedback to make sure that I am on the right track. It is a bit more clear to submit a PR than to update the document. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] wangmiao1981 commented on pull request #2290: [WIP] Core: Add Index File Metadata into Iceberg

[GitBox <gi...@apache.org>]

wangmiao1981 commented on pull request #2290:
URL: https://github.com/apache/iceberg/pull/2290#issuecomment-789512587


   made a mistake here. Don't Read. Will update soon


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org