You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "tmnd1991 (via GitHub)" <gi...@apache.org> on 2024/03/29 16:45:32 UTC
[I] merge-on-read without full row rewrite [iceberg]
tmnd1991 opened a new issue, #10066:
URL: https://github.com/apache/iceberg/issues/10066
### Proposed Change
Prior to the open sourcing of Iceberg I have implemented a process to mirror large operational database tables to plain parquet files.
Given that at the time the only option was only copy on write and I could not afford to live with it, what we implemented was a variation of merge-on-read. The gist is that we did not need to "look back" to the target table in order to create the "delta" files because we defined the concept of primary key and "squashed" older rows into the new ones at read time.
This lead to very efficient "delta" writes. On the other side I see that on the read path we might be less optimised than the current implementation, because we need to join delta writes with snapshot writes, while Iceberg current implementation can probably simply perform a lookup on delete files (if they are small enough to be broadcasted). Probably the same can be done with delta files though. Right now we're trying to port our use case to Iceberg and we are dealing with performance issues due to this choice that Is built-into the table format specification. I see this different way of dealing with MoR can be beneficial to use cases where updates are partial (w/r/t columns) and/or it's ok to have a less optimised read path in order to have a faster write path. This can be captured by a new kind of files (among data files and delete files we could have "update" files).
If the community is open to discuss such topic on a potential v3 spec, I'm open to write an in-depth proposal document.
### Proposal document
_No response_
### Specifications
- [X] Table
- [ ] View
- [ ] REST
- [ ] Puffin
- [ ] Encryption
- [ ] Other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] merge-on-read without full row rewrite [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10066:
URL: https://github.com/apache/iceberg/issues/10066#issuecomment-2433857662
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] merge-on-read without full row rewrite [iceberg]
Posted by "tmnd1991 (via GitHub)" <gi...@apache.org>.
tmnd1991 commented on issue #10066:
URL: https://github.com/apache/iceberg/issues/10066#issuecomment-2042335006
Related to: https://github.com/apache/iceberg/pull/6043
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] merge-on-read without full row rewrite [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #10066: merge-on-read without full row rewrite
URL: https://github.com/apache/iceberg/issues/10066
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] merge-on-read without full row rewrite [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10066:
URL: https://github.com/apache/iceberg/issues/10066#issuecomment-2465928084
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org