You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2019/07/26 04:16:38 UTC
[GitHub] [incubator-iceberg] rdsr opened a new pull request #315: [WIP]
Incremental processing prototype
rdsr opened a new pull request #315: [WIP] Incremental processing prototype
URL: https://github.com/apache/incubator-iceberg/pull/315
WIP branch to feedback on the overall approach.
The general idea is we support incremental processing as the delta of new files added between two snapshots.
An `IncrScan` interface is added to support this, which is also exposed as part of the `Table` API
`IncrDataTableScan` [ the impl of `IncrScan` ] is where the actual diff snapshot is constructed but is not committed to the table. To make sure older manifests are not part of the diff snapshot, we create a new `IncrAppend` class which does not add older snapshots.
`IncAppend` api can be cleaned up more and maybe merge with one of the existing Append apis possibly.
The question of how the incremental snapshot is cleaned up is still open.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org