You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Fokko (via GitHub)" <gi...@apache.org> on 2023/12/13 10:17:34 UTC

[I] Support `initial-default` when reading Avro [iceberg-rust]

Fokko opened a new issue, #119:
URL: https://github.com/apache/iceberg-rust/issues/119

   I would suggest adding support for `initial-default` to read V1 Avro files into V2 objects in memory.
   
   For example, V1 does not have sequence numbers, and then need to be set to `0`. This is important when you have a V1 table, and then you upgrade it to V2, the underlying Avro files are still V1 until you rewrite everything. See comment here: https://github.com/apache/iceberg-rust/pull/79#discussion_r1425117160
   
   With the initial-default you could just read a `0` if the field is not present.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Support `initial-default` when reading Avro [iceberg-rust]

Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on issue #119:
URL: https://github.com/apache/iceberg-rust/issues/119#issuecomment-1855134729

   If we read a V1 Avro file as a V2 object, when we write back, we also need to set [format-version in manifest](https://iceberg.apache.org/spec/#manifests:~:text=required-,format%2Dversion,-Table%20format%20version) as 2, right?
   
   I realized that I miss the case that read the v1 avro file as v2 object and write it back. I will complete them and solve this issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Support `initial-default` when reading Avro [iceberg-rust]

Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on issue #119:
URL: https://github.com/apache/iceberg-rust/issues/119#issuecomment-1855364719

   > > If we read a V1 Avro file as a V2 object, when we write back, we also need to set [format-version in manifest](https://iceberg.apache.org/spec/#manifests:~:text=required-,format%2Dversion,-Table%20format%20version) as 2, right?
   > 
   > Yes, this is correct, then the version of the table will be 2. There are two situation then:
   > 
   > * If you do a full overwrite, then manifest will be rewritten and everything will be V2.
   > * If you do an append, there can still be V1 manifests in the metadata tree.
   
   Thanks for your explanation!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Support `initial-default` when reading Avro [iceberg-rust]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #119:
URL: https://github.com/apache/iceberg-rust/issues/119#issuecomment-1855355460

   > If we read a V1 Avro file as a V2 object, when we write back, we also need to set [format-version in manifest](https://iceberg.apache.org/spec/#manifests:~:text=required-,format%2Dversion,-Table%20format%20version) as 2, right?
   
   Yes, this is correct, then the version of the table will be 2. There are two situation then:
   
   - If you do a full overwrite, then manifest will be rewritten and everything will be V2.
   - If you do an append, there can still be V1 manifests in the metadata tree.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Support `initial-default` when reading Avro [iceberg-rust]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko closed issue #119: Support `initial-default` when reading Avro
URL: https://github.com/apache/iceberg-rust/issues/119


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org