You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/06/22 19:14:21 UTC
[GitHub] [iceberg] SreeramGarlapati opened a new issue #2723: Support for low latency writes to iceberg table
SreeramGarlapati opened a new issue #2723:
URL: https://github.com/apache/iceberg/issues/2723
Am trying to understand how to configure an iceberg table for `low-latency` writes. Here are my thoughts on how to configure it. Truly appreciate any inputs.
### Supporting low latency writes to iceberg table entails the below sub-problems:
1. **Optimizing the data payload:** optimizing the data payload to be written to the table.
- **There is an Optimization that is specific to the write pattern**
1. **Appends:** in case of Appends - this is already solved - as iceberg always writes the new new inserts as new files.
2. **Deletes (/Upserts):** in case of Deletes (or Upserts - which are broken down into Insert + Delete in 2.0) - this problem is solved as well.
- **File Format**: There is another optimization knob useful at file format level. It might not make sense to generate data in the columnar format here - al the time and space spent for encoding, storing stats etc can be saved. So, thankfully, for these low-latency writes with iceberg table - `AVRO` file format can be used.
2. **Locating the records that need to be updated with low-latency:** In case of Upserts - locating the records that need to be updated is the key problem to be solved.
- One popular solution for this is to maintain indexes to support the **equality filters** used for upserts. **Do you know if there is any ongoing effort for this!?**
3. **Optimizing the Metadata payload**: for every write to Iceberg table - the **schema file** & **manifest list** file are rewritten. To further push the payload down - we can potentially write the "change set" here. Is this the current direction of thought? If so, pointers to any any work stream in this regard is truly appreciated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on issue #2723: Support for low latency writes to iceberg table
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on issue #2723:
URL: https://github.com/apache/iceberg/issues/2723#issuecomment-867133182
Adding responses from Slack:
@SreeramGarlapati
in our streaming write tests - we are seeing that metadata.json file grows to 3mb - with in couple of hours of streaming writes (at 100 events per sec rate). At this point - each write is taking several seconds
@jackye1995
I think I replied in the dev mailing list about this, please let me know if that is missing anything. For streaming, my understanding is that Iceberg’s expected streaming commit rate is at one per few seconds, 100 events per second sounds like too much for it usage. But maybe I am just underestimating its potential. Plus if you commit more often, you should also expire old snapshots more often, especially in the streaming use case.
@SreeramGarlapati
One concern with - expiring snapshots more often is that - we will lose the ability to stream incrementally (like we do here). We want to be able to stream incrementally thru the snapshots - atleast for ~1 day.
& to your suggestion on commit rate - the commit rate we were using is 1 commit a sec (and as the time progresses - as each commit is taking a bit longer - 1 commit after multiple seconds) - just that - it has several events in them - which could span across couple of partitions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on issue #2723: Support for low latency writes to iceberg table
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on issue #2723:
URL: https://github.com/apache/iceberg/issues/2723#issuecomment-867132431
Adding responses from mailing list:
@flyrain
For 3, it may not be worth adding extra complexity by introducing a "change set", unless we get solid data that shows writing a "change set" is faster than a complete rewrite.
@jackye1995
I think the summary looks mostly correct, where we already can do "low latency" append and delete by primary key, or delete with any predicate that resolves to finitely many equality constraints without the need to scan all rows.
Secondary index would be useful for scan, which is subsequently used by the generation of delete files for complex delete predicates. This is still in design, we should probably push progress in this domain a bit more because I haven't heard any update since a few months ago:
- https://docs.google.com/document/d/1E1ofBQoKRnX04bWT3utgyHQGaHZoelgXosk_UNsTUuQ
- https://docs.google.com/document/d/11o3T7XQVITY_5F9Vbri9lF9oJjDZKjHIso7K8tEaFfY
For metadata, I agree with Yufei that a change set approach sounds like quite a lot of extra complexity just to keep the commit phase a little bit faster. I understand that your single commit time is in the end bounded by the time you rewrite that file, but this almost sounds like we want to do merge-on-read even for metadata, which would be cool but likely an overkill for most users. In my mind, the metadata layer should be managed by the admin constantly, and it is much simpler to just periodically optimize the number of manifests in order to control the size of the manifest list file during each commit, which would also benefit scan planning performance at the same time. I am curious about your "low latency" requirement, do you have any actual numbers you need to hit, if you could share them here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on issue #2723: Support for low latency writes to iceberg table
Posted by GitBox <gi...@apache.org>.
pvary commented on issue #2723:
URL: https://github.com/apache/iceberg/issues/2723#issuecomment-867438599
I am really interested everyone's thoughts on this.
My main concern with Iceberg is that we have an inherent bottleneck for writes, and as it is mentioned before it contains 2 steps:
1. Write the snapshot json - the size increases with every commit
2. Commit the write using the catalog
Even though the we can write the snapshot json-s concurrently, if there are concurrent writes we still have to throw away and rewrite the snapshot json for the 2nd commit, so basically the commit is bottlenecked by the 2 steps above.
I like Iceberg a lot, but based on this Hive ACID tables are still superior in use-cases where there are multiple concurrent writes. Having the option to use "change set approach" might alleviate this bottleneck.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org