Updates/Deletes/Upserts in Iceberg

[Anton Okolnychyi <ao...@apple.com.INVALID>]

Hi folks,

Miguel (cc) and I have spent some time thinking about how to perform updates/deletes/upserts on top of Iceberg tables. This functionality is essential for many modern use cases. We've summarized our ideas in a doc [1], which, hopefully, will trigger a discussion in the community. The document presents different conceptual approaches alongside their trade-offs. We will be glad to consider any other ideas as well.

Thanks,
Anton

[1] - https://docs.google.com/document/d/1Pk34C3diOfVCRc-sfxfhXZfzvxwum1Odo-6Jj9mwK38/ <https://docs.google.com/document/d/1Pk34C3diOfVCRc-sfxfhXZfzvxwum1Odo-6Jj9mwK38/>

Re: Updates/Deletes/Upserts in Iceberg

[Owen O'Malley <ow...@gmail.com>]

> On May 15, 2019, at 12:54 PM, Ryan Blue <rb...@netflix.com.INVALID> wrote:
> 
> 2. Iceberg diff files should use synthetic keys
> 
> A lot of the discussion on the doc is about whether natural keys are practical or what assumptions we can make or trade about them. In my opinion, Iceberg tables will absolutely need natural keys for reasonable use cases. And those natural keys will need to be unique. And Iceberg will need to rely on engines to enforce that uniqueness.
> 
Agreed. One restriction that we should probably adopt is not allowing mutations of the natural keys or the partition/bucketing. Allowing such mutations while enforcing uniqueness would typically be expensive. Without those mutations, uniqueness is relatively cheap to enforce.

> But, there is a difference between table behavior and implementation. We can use synthetic keys to implement the requirements of natural keys. Each row should be identified by its file and position in a file. When deleting by a natural key, we just need to find out what the synthetic key is and encode that in the delete diff.
> 
+1

> With the physical representation using synthetic keys, we should also define how to communicate a natural key constraint for a table. That way, writers can fail if a write may violate the key constraints of a table.
> 
> 3. Synthetic keys should be based on filename and position
> 
> I think identifying the file in a synthetic key makes a lot of sense. This would allow for delta file reuse as individual files are rewritten by a “major” compaction and provides nice flexibility that fits with the format. We will need to think through all the impacts, like how file relocation works (e.g., move between regions) and the requirements for rewrites (must apply the delta when rewriting).
> 
I’d recommend using the file and row number within the file. I believe Avro, ORC, and Parquet all track row numbers and thus they are a basically free synthetic id within each file. The critical feature is that each input split needs to know how many rows are above it in the file, so that delete files can be read efficiently.

.. Owen

> Open questions
> 
> There are also quite a few remaining questions for a design:
> 
> Should Iceberg use insert diff files? (My initial answer is no)
> Should Iceberg require diff compaction? Iceberg could require one delete diff per partition, for example. (My answer: no)
> Should data files store synthetic key position? If so, why?
> Should there be a dense format for deletes, or just a sparse format?
> What is the scope of a delete diff? At a minimum, partition. But does it make sense to build ways to restrict scope further?
> 
> On Fri, May 10, 2019 at 11:27 AM Anton Okolnychyi <ao...@apple.com.invalid> wrote:
> We did take a look at Hudi. The overall design seems to be pretty complicated and, unfortunately, I didn’t have time to explore every detail.
> 
> Here is my understanding (correct me if I am wrong):
> 
> - Hudi has RECORD_KEY, which is expected to be unique.
> - Hudi has PRECOMBINED_KEY, which is used to pick only one row in the incoming batch if there are multiple rows with the same key. As I understand, this isn't used on reads. It is used on writes to deduplicate rows with identical keys within one incoming batch. For example, if we are inserting 10 records and two rows have the same key, PRECOMBINED_KEY will be used to pick up only one row.
> - Once Hudi ensures the uniqueness of RECORD_KEY within the incoming batch, it loads the Bloom filter index from all existing Parquet files in the involved partitions (meaning, partitions spread from the input batch) and tags each record as either an update or insert by mapping the incoming keys to existing files for updates. At this point, it seems to rely on join.
> 
> Is my understanding correct? If so, do we want to consider joins on write? We mentioned this technique as one way to ensure the uniqueness of natural keys but we were concerned about the performance. Also, does Hudi support record-level updates? 
> 
> Thanks,
> Anton
> 
>> On 10 May 2019, at 18:22, Erik Wright <erik.wright@shopify.com.INVALID <ma...@shopify.com.INVALID>> wrote:
>> 
>> Thanks for putting this forward.
>> 
>> Another term for the "lazy" approach would be "merge on read".
>> 
>> My team has built something internally that uses merge-on-read internally but uses an "Eager" materialization for publication to Presto. Roughly, we maintain a table metadata file that looks a bit like Iceberg's and tracks the "live" version of each partition as it is updated over time. We are looking into a solution that will allow us to push the merge-on-read all the way to Presto (and other consumers), and adding Merge-On-Read to Iceberg is one of the approaches we are considering.
>> 
>> It's worth noting that Hudi does have support for upserts/deletes as well, so that's another model to consider.
>> 
>> On Fri, May 10, 2019 at 8:30 AM Miguel Miranda <miguelnmiranda@apple.com.invalid <ma...@apple.com.invalid>> wrote:
>> Hi,
>> 
>> As Anton said, we purposely avoided making a "decision" on which approach should be implemented in order to allow for a meaningful discussion with the community.
>> 
>> The document starts with an eager approach as it is straightforward and easy to understand: steps resemble the usual file level operations/manipulations frequently used by engineers when implementing Update/Delete/Upsert behaviour themselves, hopefully creating a conceptual bridge to the more involved designs. Right now, Iceberg has almost everything to implement the "eager" approach as we simply need to adjust the retry mechanism. For example, I have implemented a prototype of the eager solution with Spark and Iceberg.
>> 
>> We looked into many existing solutions for inspiration, but when there isn't a paper or code in the public domain it becomes hard to assess the underlying design, although some of it can be inferred from the API or documentation.
>> 
>> Best,
>> Miguel
>> 
>>> On 10 May 2019, at 11:57, Anton Okolnychyi <aokolnychyi@apple.com <ma...@apple.com>> wrote:
>>> 
>>> Thanks for the feedback, Jacques!
>>> 
>>> You are correct, we kept the question of the best approach as open :) The idea was to have a discussion in the community. Hopefully, we can reach a consensus.
>>> 
>>> While the proposed “lazy” approaches certainly offer significant benefits, they require more changes in Iceberg as well as in readers/query engines (depending on how we want to merge base and diff files). For us, it is important to understand whether the Iceberg community would even consider such changes. 
>>> 
>>> Hive ACID 3 is one the projects we looked at. In fact, we spoke to Owen, the original creator of updates/deletes/upserts in Hive. I believe the “lazy” approaches are close to what Hive 3 does but with their own distinctions that Iceberg allows us to have. It would be great to have Owen’s feedback.
>>> 
>>> We don’t know the internals of Delta as updates/deletes/upserts are not open source. My personal guess, yes, it might be similar to the “eager” approach in our doc.
>>> 
>>> Jacques, could you share some insights how you implement the merge of diffs? Is it done by readers?
>>> 
>>> Thanks,
>>> Anton
>>> 
>>>> On 10 May 2019, at 06:24, Jacques Nadeau <jacques@dremio.com <ma...@dremio.com>> wrote:
>>>> 
>>>> This is a nice doc and it covers many different options. Upon first skim, I don't see a strong argument for particular approach. D
>>>> 
>>>> In our own development, we've been leaning heavily towards what you describe in the document as "lazy with SRI". I believe this is consistent with what the Hive community did on top of Orc. It's interesting because my (maybe incorrect) understanding of the Databricks Delta approach is they chose what you title "eager" in their approach to upserts. They may also have a lazy approach for other types of mutations but I don't think they do.
>>>> 
>>>> Thanks again for putting this together!
>>>> Jacques
>>>> --
>>>> Jacques Nadeau
>>>> CTO and Co-Founder, Dremio
>>>> 
>>>> 
>>>> On Wed, May 8, 2019 at 3:42 AM Anton Okolnychyi <aokolnychyi@apple.com.invalid <ma...@apple.com.invalid>> wrote:
>>>> Hi folks,
>>>> 
>>>> Miguel (cc) and I have spent some time thinking about how to perform updates/deletes/upserts on top of Iceberg tables. This functionality is essential for many modern use cases. We've summarized our ideas in a doc [1], which, hopefully, will trigger a discussion in the community. The document presents different conceptual approaches alongside their trade-offs. We will be glad to consider any other ideas as well.
>>>> 
>>>> Thanks,
>>>> Anton
>>>> 
>>>> [1] - https://docs.google.com/document/d/1Pk34C3diOfVCRc-sfxfhXZfzvxwum1Odo-6Jj9mwK38/ <https://docs.google.com/document/d/1Pk34C3diOfVCRc-sfxfhXZfzvxwum1Odo-6Jj9mwK38/>
>>>> 
>>>> 
>>> 
>> 
> 
> 
> 
> -- 
> Ryan Blue
> Software Engineer
> Netflix

Re: Updates/Deletes/Upserts in Iceberg

[Owen O'Malley <ow...@gmail.com>]

> On May 21, 2019, at 1:31 PM, Jacques Nadeau <ja...@dremio.com> wrote:
> 
> The main thing I'm talking about is how you target a deletion across time. If you have a file A, and you want to delete record X in A, you define delete A.X. At the same time, another process may be compacting A into A'. In so doing, the position of A.X in A' is something other than X.

I would argue that this is backwards. This argues that compactions need a lock so that the delete either happens before or after the compaction. If it happens before, the delete is incorporated into the new version of the file. If it happens afterwards, the delete is using the new version of the file.

> At this point, the deletion needs to be rerun against A' so that we can ensure that the deletion is propagated forward. If the only thing you have is A.X, you need to have way from of getting to the same location in A'. You should be able to take the delta file that lists the delete of A.2 and apply it directly to A' without having to also consult A. If you didn't need to solve this number, then you could simply use A.X as opposed to the key of A.X in your delta files.

I’d much prefer using file/row# as the reference for the synthetic key for the deletes. Thus, it would be file 1: rows 100, 200, and 300. That makes it clear that the delta can only be applied to a given version of the file. This has the added advantage that you know how many rows are left in file.

.. Owen

> 
> Synthetic seems relative. If the synthetic key is client-supplied, in what way is it relevant to Iceberg whether it is synthetic vs. natural? By calling it synthetic within Iceberg there is a strong implication that it is the implementation that generates it (the filename/position key suggests that). If it's the client that supplies it, it _may_ be synthetic (from the point of view of the overall data model; i.e. a customer key in a database vs. a customer ID that shows up on a bill) but from Iceberg's case that doesn't matter. Only the unicity constraint does.
> 
> I agree with the main statement here: the only real requirement is keys need to be unique across all existing snapshots. There could be two generators: one that uses an iceberg internal behavior to generate keys and one that is user definable. While there could be a third which uses an existing field (or set of fields) to define the key I think we probably should avoid implementing this as it has a whole other sets of problems that are best left outside of Iceberg's area of concern.
> 

Re: Updates/Deletes/Upserts in Iceberg

[Anton Okolnychyi <ao...@apple.com.INVALID>]

I would propose to have a series of sessions over hangouts to clarify all pending points. We can start this week if there is a timeslot that works for everyone.

Potential topics (feel free to suggest yours):
- Use cases

I believe it is critical that everyone is on the same page when it comes to our target use cases. There were some comments in the doc and in this thread related to this topic. I think we should start with this.

- Should Iceberg support both eager and lazy approaches?

Iceberg is a table format and it looks reasonable to support both. One or another might be more beneficial depending on a particular use case.

- Synthetic vs natural keys in diff files

This is a fundamental design decision and we need to ensure that it works for every target use case.

I would like to emphasize the difference between having a natural key constraint and using a natural key in diff files.

It seems we agree that it is not feasible to ensure the uniqueness of natural keys in Iceberg itself. As a table format, Iceberg can and in fact should have a notion of a natural key but query engines would need to respect that while writing. If Iceberg cannot guarantee the uniqueness of natural keys, this raises a question if we can rely on that uniqueness in our diff files. For example, if a query engine violates the natural key constraint and inserts two rows with the same natural key, we might be in a bad situation while doing updates/deletes. Consider an example below.

We have two rows with the same natural key and we use that natural key in diff files:
nk | col1 | col2
1 | 1 | 1
1 | 2 | 2

Then we have a delete statement:
DELETE FROM t WHERE col1 = 1

If we use the natural key in diff files, then we will delete both records even though only one of them matched our delete predicate.

So, the only way to make this reliable is either by enforcing the uniqueness of the natural keys or by limiting operations that are supported (i.e. we can use predicated on the natural key columns ONLY in our update/delete/upserts).

Synthetic keys don't have that problem as we can ensure their uniqueness in Iceberg itself. So, I would disagree with Cristian that we don't see any benefits of synthetic keys. My personal opinion is that we should keep this question open and properly discuss it during our session. The same goes for having multiple writers and conflict resolution. Iceberg already has some support for this and we shouldn’t break it by introducing updates/deletes/upserts.

Thanks,
Anton

> On 21 May 2019, at 22:57, Jacques Nadeau <ja...@dremio.com> wrote:
> 
> 
> That's my point, truly independent writers (two Spark jobs, or a Spark job and Dremio job) means a distributed transaction. It would need yet another external transaction coordinator on top of both Spark and Dremio, Iceberg by itself 
> cannot solve this.
>  
> I'm not ready to accept this. Iceberg already supports a set of semantics around multiple writers committing simultaneously and how conflict resolution is done. The same can be done here.
>  
> By single writer, I don't mean single process, I mean multiple coordinated processes like Spark executors coordinated by Spark driver. The coordinator ensures that the data is pre-partitioned on 
> each executor, and the coordinator commits the snapshot. 
> 
> Note however that single writer job/multiple concurrent reader jobs is perfectly feasible, i.e. it shouldn't be a problem to write from a Spark job and read from multiple Dremio queries concurrently (for example)
> 
> :D This is still "single process" from my perspective. That process may be coordinating other processes to do distributed work but ultimately it is a single process. 
>  
> I'm not sure what you mean exactly. If we can't enforce uniqueness we shouldn't assume it.
>  
> I disagree. We can specify that as a requirement and state that you'll get unintended consequences if you provide your own keys and don't maintain this.
>  
> We do expect that most of the time the natural key is unique, but the eager and lazy with natural key designs can handle duplicates 
> consistently. Basically it's not a problem to have duplicate natural keys, everything works fine.
> 
> That heavily depends on how things are implemented. For example, we may write a bunch of code that generates internal data structures based on this expectation. If we have to support duplicate matches, all of sudden we can no longer size various data structures to improve performance and may be unable to preallocate memory associated with a guaranteed completion.
> 
> Let me try and clarify each point:
> 
> - lookup for query or update on a non-(partition/bucket/sort) key predicate implies scanning large amounts of data - because these are the only data structures that can narrow down the lookup, right ? One could argue that the min/max index (file skipping) can be applied to any column, but in reality if that column is not sorted the min/max intervals can have huge overlaps so it may be next to useless.
> - remote storage - this is a critical architecture decision - implementations on local storage imply a vastly different design for the entire system, storage and compute. 
> - deleting single records per snapshot is unfeasible in eager but also particularly in the lazy design: each deletion creates a very small snapshot. Deleting 1 million records one at a time would create 1 million small files, and 1 million RPC calls.
> 
> Why is this unfeasible? If I have a dataset of 100mm files including 1mm small files, is that a major problem? It seems like your usecase isn't one where you want to support single record deletes but it is definitely something important to many people.
>  
> Eager is conceptually just lazy + compaction done, well, eagerly. The logic for both is exactly the same, the trade-off is just that with eager you implicitly compact every time so that you don't do any work on read, while with lazy 
> you want to amortize the cost of compaction over multiple snapshots.
> 
> Basically there should be no difference between the two conceptually, or with regard to keys, etc. The only difference is some mechanics in implementation.
> 
> I think you have deconstruct the problem too much to say these are the same (or at least that is what I'm starting to think given this thread). It seems like real world implementation decisions (per our discussion here) are in conflict. For example, you just argued against having a 1mm arbitrary mutations but I think that is because you aren't thinking about things over time with a delta implementation. Having 10,000 mutations a day where we do delta compaction once a week and local file mappings (key to offset sparse bitmaps) seems like it could result in very good performance in a case where we're mutating small amounts of data. In this scenario, you may not do major compaction ever unless you get to a high enough percentage of records that have been deleted in the original dataset. That drives a very different set of implementation decisions from a situation where you're trying to restate an entire partition at once.

Re: Updates/Deletes/Upserts in Iceberg

[Erik Wright <er...@shopify.com.INVALID>]

On Thu, Jun 20, 2019 at 3:26 PM Erik Wright <er...@shopify.com> wrote:

>
>
> On Thu, Jun 20, 2019 at 12:57 PM Ryan Blue <rb...@netflix.com> wrote:
>
>> Sounds like we’re in agreement on the direction! Let’s have a sync up
>> sometime next week to make sure we are agreed and plan some of this work.
>> What days work best for everyone?
>>
> Tuesday until 2PM PT
> Wednesday 11AM -> 2PM PT
> Thursday 9AM -> 10:30AM / 12PM -> 2PM PT
> Friday until 2PM PT
>
>
>> Here’s a quick summary of the approach from my perspective:
>>
>>    - Will define two options for writing delete files that identify rows
>>    by:
>>       1. File and offset for MERGE INTO or similar operations, and
>>       2. Column equality for encoding deletes without reading data
>>    - Will add a delete file type to manifests
>>    - Will add sequence numbers to existing metadata to track when to
>>    apply delete files to data files (in time), used by both delete file options
>>
>> I don't see that we need these for file/offset-deletes, since they apply
> to a specific file. They're not harmful, but the don't seem relevant.
>
>>
>>    - Will add a tracking sequence number to manifest file metadata in
>>    the manifest list, as well as a min sequence number for each manifest (min
>>    data seq as well?)
>>
>> I don't understand the purpose of the min sequence number, nor what the
> "min data seq" is.
>
>>
>>    - For column equality deletes, the partition of the file in the
>>    manifest will track the partition that it applies to (stats filtering to be
>>    done at read time)
>>
>> I’d also like to have a way to encode deletes across the entire dataset,
>> for GDPR deletes (e.g. delete a user from all sessions partitioned by
>> time). Any ideas about how to encode this?
>>
> Off the top of my head this requires adding additional information to the
> manifest file, indicating the columns that are used for the deletion. Only
> equality would be supported; if multiple columns were used, they would be
> combined with boolean-and. I don't see anything too tricky about it.
>

If we add this on a per-deletion file basis it is not clear if there is any
relevance in preserving the concept of a unique row ID. For example, if it
is possible to say "Delete all orders for user ID 17" (and we don't enforce
any uniqueness constraints, nor require that there _actually_ be any orders
for user 17) it is not clear why we would go to the trouble of doing
something more rigorous for "Delete the order with ID 1001". Remember that
upserts must necessarily be represented as a delete and an insert
_anyways_, so we are not relying on the unique row ID to replace a previous
row with the updated value.


>
> More replies to specific points:
>>
>> If there is a monotonically increasing value per snapshot, should we
>> simply update the requirement for the snapshot ID to require it to be
>> monotonically increasing?
>>
>> I was thinking about this and I don’t think it’s a good idea, but I could
>> be convinced otherwise.
>>
>> When compacting data and delete files, the sequence number of the
>> compacted result is set to the sequence of the latest delete file applied
>> to the data, but the compacted file is actually added in the replace
>> snapshot. So we wouldn’t be able to identify the files that were added by a
>> compaction without running a diff with another snapshot. I think that we
>> don’t want to lose that information because we want to be able to recover
>> what operation created any given file, if the snapshot is still around with
>> metadata.
>>
> You've convinced me, below, that we need to be able to commit new files in
> a new snapshot with a past sequence number.
>
> So far, we will have the following information per file:
>
>    - The snapshot that added it.
>    - The snapshot that deleted it.
>    - Its sequence number (which is, indirectly, a snapshot ID, but not
>    necessarily the one in which the file was added)
>
> Regardless of whether the sequence number and the snapshot are the same or
> simply have a 1:1 mapping, these three properties need to be tracked. It's
> true that using different names (and different value spaces) for them
> reduces some potential confusion.
>
> I would assume that the maximum sequence number should appear in the table
> metadata, as the alternative requires scanning manifests. Furthermore, I
> could imagine some cases where the number goes backwards according to the
> manifests (because the deletes from 13 are applied, but not all of the
> deletes from 12, so the compacted files are given sequence number 12 and
> there are no remaining files from number 13).
>
> Given that, would you make it optional to assign a sequence number to a
> snapshot? "Replace" snapshots would not need one.
>
>> The sequence number assigned to the resulting files is irrelevant if
>> *all* relevant deletes have been applied.
>>
>> I agree, but we can also use the sequence number to avoid commit
>> conflicts. If a delete is committed just before a compaction, we don’t want
>> to re-run the compaction. Using the highest sequence number from the delete
>> files applied to the data, we can confidently commit the compaction,
>> knowing that any new deltas will be applied.
>>
>> Ultimately, all files would appear to be eligible for compaction in any
>> rewrite snapshot.
>>
>> Yes, which I think is a benefit, right?
>>
> Absolutely.
>
>> I don’t think that the other approach inhibits aging off data, it just
>> represents the deletion of the data differently.
>>
>> True, but if we can preserve the existing functionality then this is a
>> quick operation. Usually, these file-level deletes align with partitioning
>> so they are quick to run. And, we can prune any delete files that no longer
>> have data files after it happens.
>>
>> On Thu, Jun 20, 2019 at 8:06 AM Erik Wright <er...@shopify.com>
>> wrote:
>>
>>>
>>>
>>> On Wed, Jun 19, 2019 at 6:39 PM Ryan Blue <rb...@netflix.com.invalid>
>>> wrote:
>>>
>>>> Replies inline.
>>>>
>>>> On Mon, Jun 17, 2019 at 10:59 AM Erik Wright
>>>> erik.wright@shopify.com.invalid
>>>> <ht...@shopify.com.invalid> wrote:
>>>>
>>>> Because snapshots and versions are basically the same idea, we don’t
>>>>>> need both. If we were to add versions, they should replace snapshots.
>>>>>>
>>>>>
>>>>> I'm a little confused by this. You mention sequence numbers quite a
>>>>> bit, but then say that we should not introduce versions. From my
>>>>> understanding, sequence numbers and versions are essentially identical.
>>>>> What has changed is where they are encoded (in file metadata vs. in
>>>>> snapshot metadata). It would also seem necessary to keep them distinct from
>>>>> snapshots in order to be able to collapse/compact files older than N
>>>>> (oldest valid version) without losing the ability to incrementally apply
>>>>> the changes from N + 1.
>>>>>
>>>> Snapshots are versions of the table, but Iceberg does not use a
>>>> monotonically increasing identifier to track them. Your proposal added
>>>> table versions in addition to snapshots to get a monotonically increasing
>>>> ID, but my point is that we don’t need to duplicate the “table state at
>>>> some time” idea by adding a layer of versions inside snapshots. We just
>>>> need a monotonically increasing ID associated with each snapshot. That ID
>>>> what I’m calling a sequence number. Each snapshot will have a new sequence
>>>> number used to track the order of files for applying changes that are
>>>> introduced by that snapshot.
>>>>
>>> If there is a monotonically increasing value per snapshot, should we
>>> simply update the requirement for the snapshot ID to require it to be
>>> monotonically increasing?
>>>
>>>
>>>> With sequence numbers embedded in metadata, you can still compact. If
>>>> you have a file with sequence number N, then updated by sequence numbers
>>>> N+2 and N+3, you can compact the changes from N and N+2 by writing the
>>>> compacted file with sequence N+2. Rewriting files in N+2 is still possible,
>>>> just like with the scheme you suggested, where the version N+2 would be
>>>> rewritten.
>>>>
>>>> A key difference is that because the approach using sequence numbers
>>>> doesn’t require keeping state around in the root metadata file, tables are
>>>> not required to rewrite or compact data files just to keep the table
>>>> functioning. If the data won’t be read, then why bother changing it?
>>>>
>>>> Because I am having a really hard time seeing how things would work
>>>>> without these sequence numbers (versions), and given their prominence in
>>>>> your reply, my remaining comments assume they are present, distinct from
>>>>> snapshots, and monotonically increasing.
>>>>>
>>>> Yes, a sequence number would be added to the existing metadata. Each
>>>> snapshot would produce a new sequence number.
>>>>
>>>> Another way of thinking about it: if versions are stored as data/delete
>>>>>> file metadata and not in the table metadata file, then the complete history
>>>>>> can be kept.
>>>>>
>>>>>
>>>>> Sure. Although version expiry would still be an important process to
>>>>> improve read performance and reduce storage overhead.
>>>>>
>>>> Maybe. If it is unlikely that the data will be read, the best option
>>>> may be to leave the original data file and associated delete file.
>>>>
>>> It's safe to say that is optional for any given use-case, but that it is
>>> a required capability.
>>>
>>>> Based on your proposal, it would seem that we can basically collapse
>>>>> any given data file with its corresponding deletions. It's important to
>>>>> consider how this will affect incremental readers. The following make sense
>>>>> to me:
>>>>>
>>>>>    1. We should track an oldest-valid-version in the table metadata.
>>>>>    No changes newer than this version should be collapsed, as a reader who has
>>>>>    consumed up to that version must be able to continue reading.
>>>>>
>>>>> Table versions are tracked by snapshots, which are kept for a window
>>>> of time. The oldest valid version is the oldest snapshot in the table. New
>>>> snapshots can always compact or rewrite older data and incremental
>>>> consumers ignore those changes by skipping snapshots where the operation
>>>> was replace because the data has not changed.
>>>>
>>> Incremental consumers operate on append, delete, and overwrite snapshots
>>>> and consume data files that are added or deleted. With delete files, these
>>>> would also consume delete files that were added in a snapshot.
>>>>
>>> Sure, when a whole file is deleted there is already a column for
>>> indicating that, so we do not need to represent it a second time.
>>>
>>>
>>>> It is unlikely that sequence number would be used by incremental
>>>> consumers because the changes are already available for each snapshot. This
>>>> distinction is the main reason why I’m calling these sequence numbers and
>>>> not “version”. The sequence number for a file indicates what delete files
>>>> need to be applied; files can be rewritten the new copy gets a different
>>>> sequence number as you describe just below.
>>>>
>>> Interesting. This explains the difference in the way we are looking at
>>> this. Yes, I agree that this ought to work and seems more consistent with
>>> the way that Iceberg works currently.
>>>
>>>>
>>>>>    1. Any data file strictly older than the oldest-valid-version is
>>>>>    eligible for delete collapsing. During delete collapsing, all deletions up
>>>>>    to version N (any version <= the oldest-valid-version) are applied to the
>>>>>    file. *The newly created file should be assigned a sequence number
>>>>>    of N.*
>>>>>
>>>>> First, a data file is kept until the last snapshot that references it
>>>> is expired. That enable incremental consumers without a restriction on
>>>> rewriting the file in a later replace snapshot.
>>>>
>>>> Second, I agree with your description of rewriting the file. That’s
>>>> what I was trying to say above.
>>>>
>>> Given that incremental consumers will skip "replace" snapshots and will
>>> not attempt to incrementally read the changes introduced in version "N"
>>> using the manifest list file for version "N+1", there would appear to be no
>>> restrictions on how much data can be collapsed in any rewrite operation.
>>> The sequence number assigned to the resulting files is irrelevant if _all_
>>> relevant deletes have been applied.
>>>
>>>>
>>>>>    1. Any delete file that has been completely applied is eligible
>>>>>    for expiry. A delete file is completely applied if its sequence number is
>>>>>    <= all data files to which it may apply (based on partitioning data).
>>>>>
>>>>> Yes, I agree.
>>>>
>>>>
>>>>>    1. Any data file equal to or older than the oldest-valid-version
>>>>>    is eligible for compaction. During compaction, some or all data files up to
>>>>>    version N (any version <= the oldest-valid version) are selected. There may
>>>>>    not be any deletions applicable to any of the selected data files. The
>>>>>    selected data files are read, rewritten as desired (partitioning,
>>>>>    compaction, sorting, etc.). The new files are included in the new snapshot (*with
>>>>>    sequence number N*) while the old files are dropped.
>>>>>
>>>>> Yes, I agree.
>>>>
>>> Ultimately, all files would appear to be eligible for compaction in any
>>> rewrite snapshot.
>>>
>>>>
>>>>>    1. Delete collapsing, delete expiry, and compaction may be
>>>>>    performed in a single commit. The oldest-valid-version may be advanced
>>>>>    during this process as well. The outcome must logically be consistent with
>>>>>    applying these steps independently.
>>>>>
>>>>> Because snapshots (table versions) are independent, they can always be
>>>> expired. Incremental consumers must process a new snapshot before it is old
>>>> enough to expire. The time period after which snapshots are expired is up
>>>> to the process running ExpireSnapshots.
>>>>
>>> Agreed.
>>>
>>>> Important to note in the above is that, during collapsing/compaction,
>>>>> new files may be emitted with a version number N that are older than the
>>>>> most recent version. This is important to ensure that all deltas newer than
>>>>> N are appropriately applied, and that incremental readers are able to
>>>>> continue processing the dataset.
>>>>>
>>>> Because incremental consumers operate using snapshots and not sequence
>>>> numbers, I think this is decoupled in the approach I’m proposing.
>>>>
>>> Agreed.
>>>
>>>> It is also compatible with file-level deletes used to age off old data
>>>>>> (e.g. delete where day(ts) < '2019-01-01').
>>>>>
>>>>>
>>>>> This particular operation is not compatible with incremental
>>>>> consumption.
>>>>>
>>>> I disagree. This just encodes deletes for an entire file worth of rows
>>>> in the manifest file instead of in a delete file. Current metadata tracks
>>>> when files are deleted and we would also associate a sequence number with
>>>> those changes, if we needed to. Incremental consumers will work like they
>>>> do today: they will consume snapshots that contain changes
>>>> (append/delete/overwrite) and will get the files that were changed in that
>>>> snapshot.
>>>>
>>> Agreed.
>>>
>>>
>>>> Also, being able to cleanly age off data is a hard requirement for
>>>> Iceberg. We all need to comply with data policies with age limits and we
>>>> need to ensure that we can cleanly apply those changes.
>>>>
>>> I don't think that the other approach inhibits aging off data, it just
>>> represents the deletion of the data differently. In any case, I agree that
>>> we can reuse the existing "deleted in snapshot" mechanism.
>>>
>>>> One must still generate a deletion that is associated with a new
>>>>> sequence number. I could see a reasonable workaround where, in order to
>>>>> delete an entire file (insertion file `x.dat`, sequence number X) one would
>>>>> reference the same file a second time (deletion file `x.dat, sequence
>>>>> number Y > X). Since an insertion file and a deletion file have compatible
>>>>> formats, there is no need to rewrite this file simply to mark each row in
>>>>> it as deleted. A savvy consumer would be able to tell that this is a
>>>>> whole-file deletion while a naïve consumer would be able to apply them
>>>>> using the basic algorithm.
>>>>>
>>>> With file-level deletes, the files are no longer processed by readers,
>>>> unless an incremental reader needs to read all of the deletes from the file.
>>>>
>>>> If it seems like we are roughly on the same page I will take a stab at
>>>>> updating that document to go to the same level of detail that it does now
>>>>> but use the sequence-number approach.
>>>>>
>>>> Sure, that sounds good.
>>>>
>>> Phew. I think we are nearly there!
>>>
>>>> --
>>>> Ryan Blue
>>>> Software Engineer
>>>> Netflix
>>>>
>>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>

Re: Updates/Deletes/Upserts in Iceberg

[Erik Wright <er...@shopify.com.INVALID>]

I have a new version
<https://docs.google.com/document/d/1FMKh_SQ6xSUUmoCA8LerTkzIxDUN5JbStQp5Hzot4eo/edit?usp=sharing>
of the proposal, updated to reflect our discussion.

I have misgivings about the elimination of the unique row ID. It makes
reading the dataset potentially much more expensive. We eliminated it in
order to support the idea of "global" deletes but might want to revisit
whether we need to handle that use case now. If we do, we may want to
handle it in a way that is distinct from the hopefully more common case of
deleting by unique row ID.

On Wed, Jul 3, 2019 at 2:44 PM Owen O'Malley <ow...@gmail.com> wrote:

> It works for me too.
>
> .. Owen
>
> On Jul 3, 2019, at 11:27, Anton Okolnychyi <ao...@apple.com.invalid>
> wrote:
>
> Works for me too.
>
> On 3 Jul 2019, at 19:09, Erik Wright <er...@shopify.com.INVALID>
> wrote:
>
> That works for me.
>
> On Wed, Jul 3, 2019 at 2:01 PM Ryan Blue <rb...@netflix.com.invalid>
> wrote:
>
>> How about 9AM PDT on Friday, 5 July then?
>>
>> On Wed, Jul 3, 2019 at 10:55 AM Owen O'Malley <ow...@gmail.com>
>> wrote:
>>
>>> I'd like to call in, but I'm out Thursday. Friday would work except 11am
>>> to 1pm pdt.
>>>
>>> .. Owen
>>>
>>> On Wed, Jul 3, 2019 at 10:42 AM Ryan Blue <rb...@netflix.com.invalid>
>>> wrote:
>>>
>>>> I'm available Thursday and Friday this week as well, but it's a holiday
>>>> in the US so some people may be out. If there are no objections from anyone
>>>> that would like to attend, then I'm up for that.
>>>>
>>>> On Wed, Jul 3, 2019 at 10:40 AM Anton Okolnychyi <ao...@apple.com>
>>>> wrote:
>>>>
>>>>> I apologize for the delay on my side. I’ll still have to go through
>>>>> the last emails. I am available on Thursday/Friday this week and would be
>>>>> great to sync.
>>>>>
>>>>> Thanks,
>>>>> Anton
>>>>>
>>>>> On 3 Jul 2019, at 01:29, Ryan Blue <rb...@netflix.com.INVALID> wrote:
>>>>>
>>>>> Sorry I didn't get back to this thread last week. Let's try to have a
>>>>> video call to sync up on this next week. What days would work for everyone?
>>>>>
>>>>> rb
>>>>>
>>>>> On Fri, Jun 21, 2019 at 9:06 AM Erik Wright <er...@shopify.com>
>>>>> wrote:
>>>>>
>>>>>> With regards to operation values. Currently they are:
>>>>>>
>>>>>>    - append: data files were added and no files were removed.
>>>>>>    - replace: data files were rewritten with the same data; i.e.,
>>>>>>    compaction, changing the data file format, or relocating data files.
>>>>>>    - overwrite: data files were deleted and added in a logical
>>>>>>    overwrite operation.
>>>>>>    - delete: data files were removed and their contents logically
>>>>>>    deleted.
>>>>>>
>>>>>> If deletion files (with or without data files) are appended to the
>>>>>> dataset, will we consider that an `append` operation? If so, if deletion
>>>>>> and/or data files are appended, and whole files are also deleted, will we
>>>>>> consider that an `overwrite`?
>>>>>>
>>>>>> Given that the only apparent purpose of the operation field is to
>>>>>> optimize snapshot expiration the above seems to meet its needs. An
>>>>>> incremental reader can also skip `replace` snapshots but no others. Once it
>>>>>> decides to read a snapshot I don't think there's any difference in how it
>>>>>> processes the data for append/overwrite/delete cases.
>>>>>>
>>>>>> On Thu, Jun 20, 2019 at 8:55 PM Ryan Blue <rb...@netflix.com> wrote:
>>>>>>
>>>>>>> I don’t see that we need [sequence numbers] for file/offset-deletes,
>>>>>>> since they apply to a specific file. They’re not harmful, but the don’t
>>>>>>> seem relevant.
>>>>>>>
>>>>>>> These delete files will probably contain a path and an offset and
>>>>>>> could contain deletes for multiple files. In that case, the sequence number
>>>>>>> can be used to eliminate delete files that don’t need to be applied to a
>>>>>>> particular data file, just like the column equality deletes. Likewise, it
>>>>>>> can be used to drop the delete files when there are no data files with an
>>>>>>> older sequence number.
>>>>>>>
>>>>>>> I don’t understand the purpose of the min sequence number, nor what
>>>>>>> the “min data seq” is.
>>>>>>>
>>>>>>> Min sequence number would be used for pruning delete files without
>>>>>>> reading all the manifests to find out if there are old data files. If no
>>>>>>> manifest with data for a partition contains a file older than some sequence
>>>>>>> number N, then any delete file with a sequence number < N can be removed.
>>>>>>>
>>>>>> OK, so the minimum sequence number is an attribute of manifest files.
>>>>>> Sounds good. It can likely permit us to optimize compaction operations as
>>>>>> well (i.e., you can easily limit the operation to a subset of manifest
>>>>>> files as long as they are the oldest ones).
>>>>>>
>>>>>>
>>>>>>> The “min data seq” is the minimum sequence number of a data file.
>>>>>>> That seems like what we actually want for the pruning I described above.
>>>>>>>
>>>>>> I would expect a data file (appended rows or deletions by column
>>>>>> value) to have a single sequence number that applies to the whole file.
>>>>>> Even a delete-by-file-and-offset file can do with only a single sequence
>>>>>> number (which must be larger than the sequence numbers of all deleted
>>>>>> files). Why do we need a "minimum" data sequence per file?
>>>>>>
>>>>>>> Off the top of my head [supporting non-key delete] requires adding
>>>>>>> additional information to the manifest file, indicating the columns that
>>>>>>> are used for the deletion. Only equality would be supported; if multiple
>>>>>>> columns were used, they would be combined with boolean-and. I don’t see
>>>>>>> anything too tricky about it.
>>>>>>>
>>>>>>> Yes, exactly. I actually phrased it wrong initially. I think it
>>>>>>> would be simple to extend the equality deletes to do this. We just need a
>>>>>>> way to have global scope, not just partition scope.
>>>>>>>
>>>>>> I don't think anything special needs to be done with regards to
>>>>>> scoping/partitioning of delete files. When scanning one or more data files,
>>>>>> one must also consider any and all deletion files that could apply to them.
>>>>>> The only way to prune deletion files from consideration is:
>>>>>>
>>>>>>    1. All of your data files have at least one partition column in
>>>>>>    common.
>>>>>>    2. The deletion file is also partitioned on that column (at
>>>>>>    least).
>>>>>>    3. The value sets of the data files do not overlap the value sets
>>>>>>    of the deletion files in that column.
>>>>>>
>>>>>>  So given a dataset of sessions that is partitioned by device form
>>>>>> factor and date, for example, you could have a delete (user_id=9876) in a
>>>>>> deletion file that is not partitioned. And it would be "in scope" for all
>>>>>> of those data files.
>>>>>>
>>>>>> If you had the same dataset partitioned by hash(user_id) and your
>>>>>> deletes were _also_ partitioned by hash(user_id) you would be able to prune
>>>>>> those deletes while scanning the sessions.
>>>>>>
>>>>>>> If we add this on a per-deletion file basis it is not clear if there
>>>>>>> is any relevance in preserving the concept of a unique row ID.
>>>>>>>
>>>>>>> Agreed. That’s why I’ve been steering us away from the debate about
>>>>>>> whether keys are unique or not. Either way, a natural key delete must
>>>>>>> delete all of the records it matches.
>>>>>>>
>>>>>>> I would assume that the maximum sequence number should appear in the
>>>>>>> table metadata
>>>>>>>
>>>>>>> Agreed.
>>>>>>>
>>>>>>> [W]ould you make it optional to assign a sequence number to a
>>>>>>> snapshot? “Replace” snapshots would not need one.
>>>>>>>
>>>>>>> The only requirement is that it is monotonically increasing. If one
>>>>>>> isn’t used, we don’t have to increment. I’d say it is up to the
>>>>>>> implementation to decide. I would probably increment it every time to avoid
>>>>>>> errors.
>>>>>>> --
>>>>>>> Ryan Blue
>>>>>>> Software Engineer
>>>>>>> Netflix
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Ryan Blue
>>>>> Software Engineer
>>>>> Netflix
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Ryan Blue
>>>> Software Engineer
>>>> Netflix
>>>>
>>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>
>

Re: Updates/Deletes/Upserts in Iceberg

[Owen O'Malley <ow...@gmail.com>]

It works for me too. 

.. Owen

> On Jul 3, 2019, at 11:27, Anton Okolnychyi <ao...@apple.com.invalid> wrote:
> 
> Works for me too.
> 
>> On 3 Jul 2019, at 19:09, Erik Wright <er...@shopify.com.INVALID> wrote:
>> 
>> That works for me.
>> 
>> On Wed, Jul 3, 2019 at 2:01 PM Ryan Blue <rb...@netflix.com.invalid> wrote:
>>> How about 9AM PDT on Friday, 5 July then?
>>> 
>>>> On Wed, Jul 3, 2019 at 10:55 AM Owen O'Malley <ow...@gmail.com> wrote:
>>>> I'd like to call in, but I'm out Thursday. Friday would work except 11am to 1pm pdt.
>>>> 
>>>> .. Owen
>>>> 
>>>>> On Wed, Jul 3, 2019 at 10:42 AM Ryan Blue <rb...@netflix.com.invalid> wrote:
>>>>> I'm available Thursday and Friday this week as well, but it's a holiday in the US so some people may be out. If there are no objections from anyone that would like to attend, then I'm up for that.
>>>>> 
>>>>>> On Wed, Jul 3, 2019 at 10:40 AM Anton Okolnychyi <ao...@apple.com> wrote:
>>>>>> I apologize for the delay on my side. I’ll still have to go through the last emails. I am available on Thursday/Friday this week and would be great to sync.
>>>>>> 
>>>>>> Thanks,
>>>>>> Anton
>>>>>> 
>>>>>>> On 3 Jul 2019, at 01:29, Ryan Blue <rb...@netflix.com.INVALID> wrote:
>>>>>>> 
>>>>>>> Sorry I didn't get back to this thread last week. Let's try to have a video call to sync up on this next week. What days would work for everyone?
>>>>>>> 
>>>>>>> rb
>>>>>>> 
>>>>>>>> On Fri, Jun 21, 2019 at 9:06 AM Erik Wright <er...@shopify.com> wrote:
>>>>>>>> With regards to operation values. Currently they are:
>>>>>>>> append: data files were added and no files were removed.
>>>>>>>> replace: data files were rewritten with the same data; i.e., compaction, changing the data file format, or relocating data files.
>>>>>>>> overwrite: data files were deleted and added in a logical overwrite operation.
>>>>>>>> delete: data files were removed and their contents logically deleted.
>>>>>>>> If deletion files (with or without data files) are appended to the dataset, will we consider that an `append` operation? If so, if deletion and/or data files are appended, and whole files are also deleted, will we consider that an `overwrite`?
>>>>>>>> 
>>>>>>>> Given that the only apparent purpose of the operation field is to optimize snapshot expiration the above seems to meet its needs. An incremental reader can also skip `replace` snapshots but no others. Once it decides to read a snapshot I don't think there's any difference in how it processes the data for append/overwrite/delete cases.
>>>>>>>> 
>>>>>>>>> On Thu, Jun 20, 2019 at 8:55 PM Ryan Blue <rb...@netflix.com> wrote:
>>>>>>>>> I don’t see that we need [sequence numbers] for file/offset-deletes, since they apply to a specific file. They’re not harmful, but the don’t seem relevant.
>>>>>>>>> 
>>>>>>>>> These delete files will probably contain a path and an offset and could contain deletes for multiple files. In that case, the sequence number can be used to eliminate delete files that don’t need to be applied to a particular data file, just like the column equality deletes. Likewise, it can be used to drop the delete files when there are no data files with an older sequence number.
>>>>>>>>> 
>>>>>>>>> I don’t understand the purpose of the min sequence number, nor what the “min data seq” is.
>>>>>>>>> 
>>>>>>>>> Min sequence number would be used for pruning delete files without reading all the manifests to find out if there are old data files. If no manifest with data for a partition contains a file older than some sequence number N, then any delete file with a sequence number < N can be removed.
>>>>>>>>> 
>>>>>>>> OK, so the minimum sequence number is an attribute of manifest files. Sounds good. It can likely permit us to optimize compaction operations as well (i.e., you can easily limit the operation to a subset of manifest files as long as they are the oldest ones).
>>>>>>>>  
>>>>>>>>> The “min data seq” is the minimum sequence number of a data file. That seems like what we actually want for the pruning I described above.
>>>>>>>>> 
>>>>>>>> I would expect a data file (appended rows or deletions by column value) to have a single sequence number that applies to the whole file. Even a delete-by-file-and-offset file can do with only a single sequence number (which must be larger than the sequence numbers of all deleted files). Why do we need a "minimum" data sequence per file?
>>>>>>>>> Off the top of my head [supporting non-key delete] requires adding additional information to the manifest file, indicating the columns that are used for the deletion. Only equality would be supported; if multiple columns were used, they would be combined with boolean-and. I don’t see anything too tricky about it.
>>>>>>>>> 
>>>>>>>>> Yes, exactly. I actually phrased it wrong initially. I think it would be simple to extend the equality deletes to do this. We just need a way to have global scope, not just partition scope.
>>>>>>>>> 
>>>>>>>> I don't think anything special needs to be done with regards to scoping/partitioning of delete files. When scanning one or more data files, one must also consider any and all deletion files that could apply to them. The only way to prune deletion files from consideration is:
>>>>>>>> All of your data files have at least one partition column in common.
>>>>>>>> The deletion file is also partitioned on that column (at least).
>>>>>>>> The value sets of the data files do not overlap the value sets of the deletion files in that column.
>>>>>>>>  So given a dataset of sessions that is partitioned by device form factor and date, for example, you could have a delete (user_id=9876) in a deletion file that is not partitioned. And it would be "in scope" for all of those data files.
>>>>>>>> 
>>>>>>>> If you had the same dataset partitioned by hash(user_id) and your deletes were _also_ partitioned by hash(user_id) you would be able to prune those deletes while scanning the sessions.
>>>>>>>>> If we add this on a per-deletion file basis it is not clear if there is any relevance in preserving the concept of a unique row ID.
>>>>>>>>> 
>>>>>>>>> Agreed. That’s why I’ve been steering us away from the debate about whether keys are unique or not. Either way, a natural key delete must delete all of the records it matches.
>>>>>>>>> 
>>>>>>>>> I would assume that the maximum sequence number should appear in the table metadata
>>>>>>>>> 
>>>>>>>>> Agreed.
>>>>>>>>> 
>>>>>>>>> [W]ould you make it optional to assign a sequence number to a snapshot? “Replace” snapshots would not need one.
>>>>>>>>> 
>>>>>>>>> The only requirement is that it is monotonically increasing. If one isn’t used, we don’t have to increment. I’d say it is up to the implementation to decide. I would probably increment it every time to avoid errors.
>>>>>>>>> 
>>>>>>>>> -- 
>>>>>>>>> Ryan Blue
>>>>>>>>> Software Engineer
>>>>>>>>> Netflix
>>>>>>> 
>>>>>>> 
>>>>>>> -- 
>>>>>>> Ryan Blue
>>>>>>> Software Engineer
>>>>>>> Netflix
>>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>> Ryan Blue
>>>>> Software Engineer
>>>>> Netflix
>>> 
>>> 
>>> -- 
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
> 

Re: Updates/Deletes/Upserts in Iceberg

[Anton Okolnychyi <ao...@apple.com.INVALID>]

Works for me too.

> On 3 Jul 2019, at 19:09, Erik Wright <er...@shopify.com.INVALID> wrote:
> 
> That works for me.
> 
> On Wed, Jul 3, 2019 at 2:01 PM Ryan Blue <rb...@netflix.com.invalid> wrote:
> How about 9AM PDT on Friday, 5 July then?
> 
> On Wed, Jul 3, 2019 at 10:55 AM Owen O'Malley <owen.omalley@gmail.com <ma...@gmail.com>> wrote:
> I'd like to call in, but I'm out Thursday. Friday would work except 11am to 1pm pdt.
> 
> .. Owen
> 
> On Wed, Jul 3, 2019 at 10:42 AM Ryan Blue <rb...@netflix.com.invalid> wrote:
> I'm available Thursday and Friday this week as well, but it's a holiday in the US so some people may be out. If there are no objections from anyone that would like to attend, then I'm up for that.
> 
> On Wed, Jul 3, 2019 at 10:40 AM Anton Okolnychyi <aokolnychyi@apple.com <ma...@apple.com>> wrote:
> I apologize for the delay on my side. I’ll still have to go through the last emails. I am available on Thursday/Friday this week and would be great to sync.
> 
> Thanks,
> Anton
> 
>> On 3 Jul 2019, at 01:29, Ryan Blue <rblue@netflix.com.INVALID <ma...@netflix.com.INVALID>> wrote:
>> 
>> Sorry I didn't get back to this thread last week. Let's try to have a video call to sync up on this next week. What days would work for everyone?
>> 
>> rb
>> 
>> On Fri, Jun 21, 2019 at 9:06 AM Erik Wright <erik.wright@shopify.com <ma...@shopify.com>> wrote:
>> With regards to operation values. Currently they are:
>> append: data files were added and no files were removed.
>> replace: data files were rewritten with the same data; i.e., compaction, changing the data file format, or relocating data files.
>> overwrite: data files were deleted and added in a logical overwrite operation.
>> delete: data files were removed and their contents logically deleted.
>> If deletion files (with or without data files) are appended to the dataset, will we consider that an `append` operation? If so, if deletion and/or data files are appended, and whole files are also deleted, will we consider that an `overwrite`?
>> 
>> Given that the only apparent purpose of the operation field is to optimize snapshot expiration the above seems to meet its needs. An incremental reader can also skip `replace` snapshots but no others. Once it decides to read a snapshot I don't think there's any difference in how it processes the data for append/overwrite/delete cases.
>> 
>> On Thu, Jun 20, 2019 at 8:55 PM Ryan Blue <rblue@netflix.com <ma...@netflix.com>> wrote:
>> I don’t see that we need [sequence numbers] for file/offset-deletes, since they apply to a specific file. They’re not harmful, but the don’t seem relevant.
>> 
>> These delete files will probably contain a path and an offset and could contain deletes for multiple files. In that case, the sequence number can be used to eliminate delete files that don’t need to be applied to a particular data file, just like the column equality deletes. Likewise, it can be used to drop the delete files when there are no data files with an older sequence number.
>> 
>> I don’t understand the purpose of the min sequence number, nor what the “min data seq” is.
>> 
>> Min sequence number would be used for pruning delete files without reading all the manifests to find out if there are old data files. If no manifest with data for a partition contains a file older than some sequence number N, then any delete file with a sequence number < N can be removed.
>> 
>> OK, so the minimum sequence number is an attribute of manifest files. Sounds good. It can likely permit us to optimize compaction operations as well (i.e., you can easily limit the operation to a subset of manifest files as long as they are the oldest ones).
>>  
>> The “min data seq” is the minimum sequence number of a data file. That seems like what we actually want for the pruning I described above.
>> 
>> I would expect a data file (appended rows or deletions by column value) to have a single sequence number that applies to the whole file. Even a delete-by-file-and-offset file can do with only a single sequence number (which must be larger than the sequence numbers of all deleted files). Why do we need a "minimum" data sequence per file?
>> Off the top of my head [supporting non-key delete] requires adding additional information to the manifest file, indicating the columns that are used for the deletion. Only equality would be supported; if multiple columns were used, they would be combined with boolean-and. I don’t see anything too tricky about it.
>> 
>> Yes, exactly. I actually phrased it wrong initially. I think it would be simple to extend the equality deletes to do this. We just need a way to have global scope, not just partition scope.
>> 
>> I don't think anything special needs to be done with regards to scoping/partitioning of delete files. When scanning one or more data files, one must also consider any and all deletion files that could apply to them. The only way to prune deletion files from consideration is:
>> All of your data files have at least one partition column in common.
>> The deletion file is also partitioned on that column (at least).
>> The value sets of the data files do not overlap the value sets of the deletion files in that column.
>>  So given a dataset of sessions that is partitioned by device form factor and date, for example, you could have a delete (user_id=9876) in a deletion file that is not partitioned. And it would be "in scope" for all of those data files.
>> 
>> If you had the same dataset partitioned by hash(user_id) and your deletes were _also_ partitioned by hash(user_id) you would be able to prune those deletes while scanning the sessions.
>> If we add this on a per-deletion file basis it is not clear if there is any relevance in preserving the concept of a unique row ID.
>> 
>> Agreed. That’s why I’ve been steering us away from the debate about whether keys are unique or not. Either way, a natural key delete must delete all of the records it matches.
>> 
>> I would assume that the maximum sequence number should appear in the table metadata
>> 
>> Agreed.
>> 
>> [W]ould you make it optional to assign a sequence number to a snapshot? “Replace” snapshots would not need one.
>> 
>> The only requirement is that it is monotonically increasing. If one isn’t used, we don’t have to increment. I’d say it is up to the implementation to decide. I would probably increment it every time to avoid errors.
>> 
>> -- 
>> Ryan Blue
>> Software Engineer
>> Netflix
>> 
>> 
>> -- 
>> Ryan Blue
>> Software Engineer
>> Netflix
> 
> 
> 
> -- 
> Ryan Blue
> Software Engineer
> Netflix
> 
> 
> -- 
> Ryan Blue
> Software Engineer
> Netflix

Re: Updates/Deletes/Upserts in Iceberg

[Erik Wright <er...@shopify.com.INVALID>]

That works for me.

On Wed, Jul 3, 2019 at 2:01 PM Ryan Blue <rb...@netflix.com.invalid> wrote:

> How about 9AM PDT on Friday, 5 July then?
>
> On Wed, Jul 3, 2019 at 10:55 AM Owen O'Malley <ow...@gmail.com>
> wrote:
>
>> I'd like to call in, but I'm out Thursday. Friday would work except 11am
>> to 1pm pdt.
>>
>> .. Owen
>>
>> On Wed, Jul 3, 2019 at 10:42 AM Ryan Blue <rb...@netflix.com.invalid>
>> wrote:
>>
>>> I'm available Thursday and Friday this week as well, but it's a holiday
>>> in the US so some people may be out. If there are no objections from anyone
>>> that would like to attend, then I'm up for that.
>>>
>>> On Wed, Jul 3, 2019 at 10:40 AM Anton Okolnychyi <ao...@apple.com>
>>> wrote:
>>>
>>>> I apologize for the delay on my side. I’ll still have to go through the
>>>> last emails. I am available on Thursday/Friday this week and would be great
>>>> to sync.
>>>>
>>>> Thanks,
>>>> Anton
>>>>
>>>> On 3 Jul 2019, at 01:29, Ryan Blue <rb...@netflix.com.INVALID> wrote:
>>>>
>>>> Sorry I didn't get back to this thread last week. Let's try to have a
>>>> video call to sync up on this next week. What days would work for everyone?
>>>>
>>>> rb
>>>>
>>>> On Fri, Jun 21, 2019 at 9:06 AM Erik Wright <er...@shopify.com>
>>>> wrote:
>>>>
>>>>> With regards to operation values. Currently they are:
>>>>>
>>>>>    - append: data files were added and no files were removed.
>>>>>    - replace: data files were rewritten with the same data; i.e.,
>>>>>    compaction, changing the data file format, or relocating data files.
>>>>>    - overwrite: data files were deleted and added in a logical
>>>>>    overwrite operation.
>>>>>    - delete: data files were removed and their contents logically
>>>>>    deleted.
>>>>>
>>>>> If deletion files (with or without data files) are appended to the
>>>>> dataset, will we consider that an `append` operation? If so, if deletion
>>>>> and/or data files are appended, and whole files are also deleted, will we
>>>>> consider that an `overwrite`?
>>>>>
>>>>> Given that the only apparent purpose of the operation field is to
>>>>> optimize snapshot expiration the above seems to meet its needs. An
>>>>> incremental reader can also skip `replace` snapshots but no others. Once it
>>>>> decides to read a snapshot I don't think there's any difference in how it
>>>>> processes the data for append/overwrite/delete cases.
>>>>>
>>>>> On Thu, Jun 20, 2019 at 8:55 PM Ryan Blue <rb...@netflix.com> wrote:
>>>>>
>>>>>> I don’t see that we need [sequence numbers] for file/offset-deletes,
>>>>>> since they apply to a specific file. They’re not harmful, but the don’t
>>>>>> seem relevant.
>>>>>>
>>>>>> These delete files will probably contain a path and an offset and
>>>>>> could contain deletes for multiple files. In that case, the sequence number
>>>>>> can be used to eliminate delete files that don’t need to be applied to a
>>>>>> particular data file, just like the column equality deletes. Likewise, it
>>>>>> can be used to drop the delete files when there are no data files with an
>>>>>> older sequence number.
>>>>>>
>>>>>> I don’t understand the purpose of the min sequence number, nor what
>>>>>> the “min data seq” is.
>>>>>>
>>>>>> Min sequence number would be used for pruning delete files without
>>>>>> reading all the manifests to find out if there are old data files. If no
>>>>>> manifest with data for a partition contains a file older than some sequence
>>>>>> number N, then any delete file with a sequence number < N can be removed.
>>>>>>
>>>>> OK, so the minimum sequence number is an attribute of manifest files.
>>>>> Sounds good. It can likely permit us to optimize compaction operations as
>>>>> well (i.e., you can easily limit the operation to a subset of manifest
>>>>> files as long as they are the oldest ones).
>>>>>
>>>>>
>>>>>> The “min data seq” is the minimum sequence number of a data file.
>>>>>> That seems like what we actually want for the pruning I described above.
>>>>>>
>>>>> I would expect a data file (appended rows or deletions by column
>>>>> value) to have a single sequence number that applies to the whole file.
>>>>> Even a delete-by-file-and-offset file can do with only a single sequence
>>>>> number (which must be larger than the sequence numbers of all deleted
>>>>> files). Why do we need a "minimum" data sequence per file?
>>>>>
>>>>>> Off the top of my head [supporting non-key delete] requires adding
>>>>>> additional information to the manifest file, indicating the columns that
>>>>>> are used for the deletion. Only equality would be supported; if multiple
>>>>>> columns were used, they would be combined with boolean-and. I don’t see
>>>>>> anything too tricky about it.
>>>>>>
>>>>>> Yes, exactly. I actually phrased it wrong initially. I think it would
>>>>>> be simple to extend the equality deletes to do this. We just need a way to
>>>>>> have global scope, not just partition scope.
>>>>>>
>>>>> I don't think anything special needs to be done with regards to
>>>>> scoping/partitioning of delete files. When scanning one or more data files,
>>>>> one must also consider any and all deletion files that could apply to them.
>>>>> The only way to prune deletion files from consideration is:
>>>>>
>>>>>    1. All of your data files have at least one partition column in
>>>>>    common.
>>>>>    2. The deletion file is also partitioned on that column (at least).
>>>>>    3. The value sets of the data files do not overlap the value sets
>>>>>    of the deletion files in that column.
>>>>>
>>>>>  So given a dataset of sessions that is partitioned by device form
>>>>> factor and date, for example, you could have a delete (user_id=9876) in a
>>>>> deletion file that is not partitioned. And it would be "in scope" for all
>>>>> of those data files.
>>>>>
>>>>> If you had the same dataset partitioned by hash(user_id) and your
>>>>> deletes were _also_ partitioned by hash(user_id) you would be able to prune
>>>>> those deletes while scanning the sessions.
>>>>>
>>>>>> If we add this on a per-deletion file basis it is not clear if there
>>>>>> is any relevance in preserving the concept of a unique row ID.
>>>>>>
>>>>>> Agreed. That’s why I’ve been steering us away from the debate about
>>>>>> whether keys are unique or not. Either way, a natural key delete must
>>>>>> delete all of the records it matches.
>>>>>>
>>>>>> I would assume that the maximum sequence number should appear in the
>>>>>> table metadata
>>>>>>
>>>>>> Agreed.
>>>>>>
>>>>>> [W]ould you make it optional to assign a sequence number to a
>>>>>> snapshot? “Replace” snapshots would not need one.
>>>>>>
>>>>>> The only requirement is that it is monotonically increasing. If one
>>>>>> isn’t used, we don’t have to increment. I’d say it is up to the
>>>>>> implementation to decide. I would probably increment it every time to avoid
>>>>>> errors.
>>>>>> --
>>>>>> Ryan Blue
>>>>>> Software Engineer
>>>>>> Netflix
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Ryan Blue
>>>> Software Engineer
>>>> Netflix
>>>>
>>>>
>>>>
>>>
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>>
>>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: Updates/Deletes/Upserts in Iceberg

[Ryan Blue <rb...@netflix.com.INVALID>]

How about 9AM PDT on Friday, 5 July then?

On Wed, Jul 3, 2019 at 10:55 AM Owen O'Malley <ow...@gmail.com>
wrote:

> I'd like to call in, but I'm out Thursday. Friday would work except 11am
> to 1pm pdt.
>
> .. Owen
>
> On Wed, Jul 3, 2019 at 10:42 AM Ryan Blue <rb...@netflix.com.invalid>
> wrote:
>
>> I'm available Thursday and Friday this week as well, but it's a holiday
>> in the US so some people may be out. If there are no objections from anyone
>> that would like to attend, then I'm up for that.
>>
>> On Wed, Jul 3, 2019 at 10:40 AM Anton Okolnychyi <ao...@apple.com>
>> wrote:
>>
>>> I apologize for the delay on my side. I’ll still have to go through the
>>> last emails. I am available on Thursday/Friday this week and would be great
>>> to sync.
>>>
>>> Thanks,
>>> Anton
>>>
>>> On 3 Jul 2019, at 01:29, Ryan Blue <rb...@netflix.com.INVALID> wrote:
>>>
>>> Sorry I didn't get back to this thread last week. Let's try to have a
>>> video call to sync up on this next week. What days would work for everyone?
>>>
>>> rb
>>>
>>> On Fri, Jun 21, 2019 at 9:06 AM Erik Wright <er...@shopify.com>
>>> wrote:
>>>
>>>> With regards to operation values. Currently they are:
>>>>
>>>>    - append: data files were added and no files were removed.
>>>>    - replace: data files were rewritten with the same data; i.e.,
>>>>    compaction, changing the data file format, or relocating data files.
>>>>    - overwrite: data files were deleted and added in a logical
>>>>    overwrite operation.
>>>>    - delete: data files were removed and their contents logically
>>>>    deleted.
>>>>
>>>> If deletion files (with or without data files) are appended to the
>>>> dataset, will we consider that an `append` operation? If so, if deletion
>>>> and/or data files are appended, and whole files are also deleted, will we
>>>> consider that an `overwrite`?
>>>>
>>>> Given that the only apparent purpose of the operation field is to
>>>> optimize snapshot expiration the above seems to meet its needs. An
>>>> incremental reader can also skip `replace` snapshots but no others. Once it
>>>> decides to read a snapshot I don't think there's any difference in how it
>>>> processes the data for append/overwrite/delete cases.
>>>>
>>>> On Thu, Jun 20, 2019 at 8:55 PM Ryan Blue <rb...@netflix.com> wrote:
>>>>
>>>>> I don’t see that we need [sequence numbers] for file/offset-deletes,
>>>>> since they apply to a specific file. They’re not harmful, but the don’t
>>>>> seem relevant.
>>>>>
>>>>> These delete files will probably contain a path and an offset and
>>>>> could contain deletes for multiple files. In that case, the sequence number
>>>>> can be used to eliminate delete files that don’t need to be applied to a
>>>>> particular data file, just like the column equality deletes. Likewise, it
>>>>> can be used to drop the delete files when there are no data files with an
>>>>> older sequence number.
>>>>>
>>>>> I don’t understand the purpose of the min sequence number, nor what
>>>>> the “min data seq” is.
>>>>>
>>>>> Min sequence number would be used for pruning delete files without
>>>>> reading all the manifests to find out if there are old data files. If no
>>>>> manifest with data for a partition contains a file older than some sequence
>>>>> number N, then any delete file with a sequence number < N can be removed.
>>>>>
>>>> OK, so the minimum sequence number is an attribute of manifest files.
>>>> Sounds good. It can likely permit us to optimize compaction operations as
>>>> well (i.e., you can easily limit the operation to a subset of manifest
>>>> files as long as they are the oldest ones).
>>>>
>>>>
>>>>> The “min data seq” is the minimum sequence number of a data file. That
>>>>> seems like what we actually want for the pruning I described above.
>>>>>
>>>> I would expect a data file (appended rows or deletions by column value)
>>>> to have a single sequence number that applies to the whole file. Even a
>>>> delete-by-file-and-offset file can do with only a single sequence number
>>>> (which must be larger than the sequence numbers of all deleted files). Why
>>>> do we need a "minimum" data sequence per file?
>>>>
>>>>> Off the top of my head [supporting non-key delete] requires adding
>>>>> additional information to the manifest file, indicating the columns that
>>>>> are used for the deletion. Only equality would be supported; if multiple
>>>>> columns were used, they would be combined with boolean-and. I don’t see
>>>>> anything too tricky about it.
>>>>>
>>>>> Yes, exactly. I actually phrased it wrong initially. I think it would
>>>>> be simple to extend the equality deletes to do this. We just need a way to
>>>>> have global scope, not just partition scope.
>>>>>
>>>> I don't think anything special needs to be done with regards to
>>>> scoping/partitioning of delete files. When scanning one or more data files,
>>>> one must also consider any and all deletion files that could apply to them.
>>>> The only way to prune deletion files from consideration is:
>>>>
>>>>    1. All of your data files have at least one partition column in
>>>>    common.
>>>>    2. The deletion file is also partitioned on that column (at least).
>>>>    3. The value sets of the data files do not overlap the value sets
>>>>    of the deletion files in that column.
>>>>
>>>>  So given a dataset of sessions that is partitioned by device form
>>>> factor and date, for example, you could have a delete (user_id=9876) in a
>>>> deletion file that is not partitioned. And it would be "in scope" for all
>>>> of those data files.
>>>>
>>>> If you had the same dataset partitioned by hash(user_id) and your
>>>> deletes were _also_ partitioned by hash(user_id) you would be able to prune
>>>> those deletes while scanning the sessions.
>>>>
>>>>> If we add this on a per-deletion file basis it is not clear if there
>>>>> is any relevance in preserving the concept of a unique row ID.
>>>>>
>>>>> Agreed. That’s why I’ve been steering us away from the debate about
>>>>> whether keys are unique or not. Either way, a natural key delete must
>>>>> delete all of the records it matches.
>>>>>
>>>>> I would assume that the maximum sequence number should appear in the
>>>>> table metadata
>>>>>
>>>>> Agreed.
>>>>>
>>>>> [W]ould you make it optional to assign a sequence number to a
>>>>> snapshot? “Replace” snapshots would not need one.
>>>>>
>>>>> The only requirement is that it is monotonically increasing. If one
>>>>> isn’t used, we don’t have to increment. I’d say it is up to the
>>>>> implementation to decide. I would probably increment it every time to avoid
>>>>> errors.
>>>>> --
>>>>> Ryan Blue
>>>>> Software Engineer
>>>>> Netflix
>>>>>
>>>>
>>>
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>>
>>>
>>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>

-- 
Ryan Blue
Software Engineer
Netflix

Re: Updates/Deletes/Upserts in Iceberg

[Owen O'Malley <ow...@gmail.com>]

I'd like to call in, but I'm out Thursday. Friday would work except 11am to
1pm pdt.

.. Owen

On Wed, Jul 3, 2019 at 10:42 AM Ryan Blue <rb...@netflix.com.invalid> wrote:

> I'm available Thursday and Friday this week as well, but it's a holiday in
> the US so some people may be out. If there are no objections from anyone
> that would like to attend, then I'm up for that.
>
> On Wed, Jul 3, 2019 at 10:40 AM Anton Okolnychyi <ao...@apple.com>
> wrote:
>
>> I apologize for the delay on my side. I’ll still have to go through the
>> last emails. I am available on Thursday/Friday this week and would be great
>> to sync.
>>
>> Thanks,
>> Anton
>>
>> On 3 Jul 2019, at 01:29, Ryan Blue <rb...@netflix.com.INVALID> wrote:
>>
>> Sorry I didn't get back to this thread last week. Let's try to have a
>> video call to sync up on this next week. What days would work for everyone?
>>
>> rb
>>
>> On Fri, Jun 21, 2019 at 9:06 AM Erik Wright <er...@shopify.com>
>> wrote:
>>
>>> With regards to operation values. Currently they are:
>>>
>>>    - append: data files were added and no files were removed.
>>>    - replace: data files were rewritten with the same data; i.e.,
>>>    compaction, changing the data file format, or relocating data files.
>>>    - overwrite: data files were deleted and added in a logical
>>>    overwrite operation.
>>>    - delete: data files were removed and their contents logically
>>>    deleted.
>>>
>>> If deletion files (with or without data files) are appended to the
>>> dataset, will we consider that an `append` operation? If so, if deletion
>>> and/or data files are appended, and whole files are also deleted, will we
>>> consider that an `overwrite`?
>>>
>>> Given that the only apparent purpose of the operation field is to
>>> optimize snapshot expiration the above seems to meet its needs. An
>>> incremental reader can also skip `replace` snapshots but no others. Once it
>>> decides to read a snapshot I don't think there's any difference in how it
>>> processes the data for append/overwrite/delete cases.
>>>
>>> On Thu, Jun 20, 2019 at 8:55 PM Ryan Blue <rb...@netflix.com> wrote:
>>>
>>>> I don’t see that we need [sequence numbers] for file/offset-deletes,
>>>> since they apply to a specific file. They’re not harmful, but the don’t
>>>> seem relevant.
>>>>
>>>> These delete files will probably contain a path and an offset and could
>>>> contain deletes for multiple files. In that case, the sequence number can
>>>> be used to eliminate delete files that don’t need to be applied to a
>>>> particular data file, just like the column equality deletes. Likewise, it
>>>> can be used to drop the delete files when there are no data files with an
>>>> older sequence number.
>>>>
>>>> I don’t understand the purpose of the min sequence number, nor what the
>>>> “min data seq” is.
>>>>
>>>> Min sequence number would be used for pruning delete files without
>>>> reading all the manifests to find out if there are old data files. If no
>>>> manifest with data for a partition contains a file older than some sequence
>>>> number N, then any delete file with a sequence number < N can be removed.
>>>>
>>> OK, so the minimum sequence number is an attribute of manifest files.
>>> Sounds good. It can likely permit us to optimize compaction operations as
>>> well (i.e., you can easily limit the operation to a subset of manifest
>>> files as long as they are the oldest ones).
>>>
>>>
>>>> The “min data seq” is the minimum sequence number of a data file. That
>>>> seems like what we actually want for the pruning I described above.
>>>>
>>> I would expect a data file (appended rows or deletions by column value)
>>> to have a single sequence number that applies to the whole file. Even a
>>> delete-by-file-and-offset file can do with only a single sequence number
>>> (which must be larger than the sequence numbers of all deleted files). Why
>>> do we need a "minimum" data sequence per file?
>>>
>>>> Off the top of my head [supporting non-key delete] requires adding
>>>> additional information to the manifest file, indicating the columns that
>>>> are used for the deletion. Only equality would be supported; if multiple
>>>> columns were used, they would be combined with boolean-and. I don’t see
>>>> anything too tricky about it.
>>>>
>>>> Yes, exactly. I actually phrased it wrong initially. I think it would
>>>> be simple to extend the equality deletes to do this. We just need a way to
>>>> have global scope, not just partition scope.
>>>>
>>> I don't think anything special needs to be done with regards to
>>> scoping/partitioning of delete files. When scanning one or more data files,
>>> one must also consider any and all deletion files that could apply to them.
>>> The only way to prune deletion files from consideration is:
>>>
>>>    1. All of your data files have at least one partition column in
>>>    common.
>>>    2. The deletion file is also partitioned on that column (at least).
>>>    3. The value sets of the data files do not overlap the value sets of
>>>    the deletion files in that column.
>>>
>>>  So given a dataset of sessions that is partitioned by device form
>>> factor and date, for example, you could have a delete (user_id=9876) in a
>>> deletion file that is not partitioned. And it would be "in scope" for all
>>> of those data files.
>>>
>>> If you had the same dataset partitioned by hash(user_id) and your
>>> deletes were _also_ partitioned by hash(user_id) you would be able to prune
>>> those deletes while scanning the sessions.
>>>
>>>> If we add this on a per-deletion file basis it is not clear if there is
>>>> any relevance in preserving the concept of a unique row ID.
>>>>
>>>> Agreed. That’s why I’ve been steering us away from the debate about
>>>> whether keys are unique or not. Either way, a natural key delete must
>>>> delete all of the records it matches.
>>>>
>>>> I would assume that the maximum sequence number should appear in the
>>>> table metadata
>>>>
>>>> Agreed.
>>>>
>>>> [W]ould you make it optional to assign a sequence number to a snapshot?
>>>> “Replace” snapshots would not need one.
>>>>
>>>> The only requirement is that it is monotonically increasing. If one
>>>> isn’t used, we don’t have to increment. I’d say it is up to the
>>>> implementation to decide. I would probably increment it every time to avoid
>>>> errors.
>>>> --
>>>> Ryan Blue
>>>> Software Engineer
>>>> Netflix
>>>>
>>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>>
>>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: Updates/Deletes/Upserts in Iceberg

[Ryan Blue <rb...@netflix.com.INVALID>]

I'm available Thursday and Friday this week as well, but it's a holiday in
the US so some people may be out. If there are no objections from anyone
that would like to attend, then I'm up for that.

On Wed, Jul 3, 2019 at 10:40 AM Anton Okolnychyi <ao...@apple.com>
wrote:

> I apologize for the delay on my side. I’ll still have to go through the
> last emails. I am available on Thursday/Friday this week and would be great
> to sync.
>
> Thanks,
> Anton
>
> On 3 Jul 2019, at 01:29, Ryan Blue <rb...@netflix.com.INVALID> wrote:
>
> Sorry I didn't get back to this thread last week. Let's try to have a
> video call to sync up on this next week. What days would work for everyone?
>
> rb
>
> On Fri, Jun 21, 2019 at 9:06 AM Erik Wright <er...@shopify.com>
> wrote:
>
>> With regards to operation values. Currently they are:
>>
>>    - append: data files were added and no files were removed.
>>    - replace: data files were rewritten with the same data; i.e.,
>>    compaction, changing the data file format, or relocating data files.
>>    - overwrite: data files were deleted and added in a logical overwrite
>>    operation.
>>    - delete: data files were removed and their contents logically
>>    deleted.
>>
>> If deletion files (with or without data files) are appended to the
>> dataset, will we consider that an `append` operation? If so, if deletion
>> and/or data files are appended, and whole files are also deleted, will we
>> consider that an `overwrite`?
>>
>> Given that the only apparent purpose of the operation field is to
>> optimize snapshot expiration the above seems to meet its needs. An
>> incremental reader can also skip `replace` snapshots but no others. Once it
>> decides to read a snapshot I don't think there's any difference in how it
>> processes the data for append/overwrite/delete cases.
>>
>> On Thu, Jun 20, 2019 at 8:55 PM Ryan Blue <rb...@netflix.com> wrote:
>>
>>> I don’t see that we need [sequence numbers] for file/offset-deletes,
>>> since they apply to a specific file. They’re not harmful, but the don’t
>>> seem relevant.
>>>
>>> These delete files will probably contain a path and an offset and could
>>> contain deletes for multiple files. In that case, the sequence number can
>>> be used to eliminate delete files that don’t need to be applied to a
>>> particular data file, just like the column equality deletes. Likewise, it
>>> can be used to drop the delete files when there are no data files with an
>>> older sequence number.
>>>
>>> I don’t understand the purpose of the min sequence number, nor what the
>>> “min data seq” is.
>>>
>>> Min sequence number would be used for pruning delete files without
>>> reading all the manifests to find out if there are old data files. If no
>>> manifest with data for a partition contains a file older than some sequence
>>> number N, then any delete file with a sequence number < N can be removed.
>>>
>> OK, so the minimum sequence number is an attribute of manifest files.
>> Sounds good. It can likely permit us to optimize compaction operations as
>> well (i.e., you can easily limit the operation to a subset of manifest
>> files as long as they are the oldest ones).
>>
>>
>>> The “min data seq” is the minimum sequence number of a data file. That
>>> seems like what we actually want for the pruning I described above.
>>>
>> I would expect a data file (appended rows or deletions by column value)
>> to have a single sequence number that applies to the whole file. Even a
>> delete-by-file-and-offset file can do with only a single sequence number
>> (which must be larger than the sequence numbers of all deleted files). Why
>> do we need a "minimum" data sequence per file?
>>
>>> Off the top of my head [supporting non-key delete] requires adding
>>> additional information to the manifest file, indicating the columns that
>>> are used for the deletion. Only equality would be supported; if multiple
>>> columns were used, they would be combined with boolean-and. I don’t see
>>> anything too tricky about it.
>>>
>>> Yes, exactly. I actually phrased it wrong initially. I think it would be
>>> simple to extend the equality deletes to do this. We just need a way to
>>> have global scope, not just partition scope.
>>>
>> I don't think anything special needs to be done with regards to
>> scoping/partitioning of delete files. When scanning one or more data files,
>> one must also consider any and all deletion files that could apply to them.
>> The only way to prune deletion files from consideration is:
>>
>>    1. All of your data files have at least one partition column in
>>    common.
>>    2. The deletion file is also partitioned on that column (at least).
>>    3. The value sets of the data files do not overlap the value sets of
>>    the deletion files in that column.
>>
>>  So given a dataset of sessions that is partitioned by device form factor
>> and date, for example, you could have a delete (user_id=9876) in a deletion
>> file that is not partitioned. And it would be "in scope" for all of those
>> data files.
>>
>> If you had the same dataset partitioned by hash(user_id) and your deletes
>> were _also_ partitioned by hash(user_id) you would be able to prune those
>> deletes while scanning the sessions.
>>
>>> If we add this on a per-deletion file basis it is not clear if there is
>>> any relevance in preserving the concept of a unique row ID.
>>>
>>> Agreed. That’s why I’ve been steering us away from the debate about
>>> whether keys are unique or not. Either way, a natural key delete must
>>> delete all of the records it matches.
>>>
>>> I would assume that the maximum sequence number should appear in the
>>> table metadata
>>>
>>> Agreed.
>>>
>>> [W]ould you make it optional to assign a sequence number to a snapshot?
>>> “Replace” snapshots would not need one.
>>>
>>> The only requirement is that it is monotonically increasing. If one
>>> isn’t used, we don’t have to increment. I’d say it is up to the
>>> implementation to decide. I would probably increment it every time to avoid
>>> errors.
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>>
>>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>
>
>

-- 
Ryan Blue
Software Engineer
Netflix

Re: Updates/Deletes/Upserts in Iceberg

[Anton Okolnychyi <ao...@apple.com.INVALID>]

I apologize for the delay on my side. I’ll still have to go through the last emails. I am available on Thursday/Friday this week and would be great to sync.

Thanks,
Anton

> On 3 Jul 2019, at 01:29, Ryan Blue <rb...@netflix.com.INVALID> wrote:
> 
> Sorry I didn't get back to this thread last week. Let's try to have a video call to sync up on this next week. What days would work for everyone?
> 
> rb
> 
> On Fri, Jun 21, 2019 at 9:06 AM Erik Wright <erik.wright@shopify.com <ma...@shopify.com>> wrote:
> With regards to operation values. Currently they are:
> append: data files were added and no files were removed.
> replace: data files were rewritten with the same data; i.e., compaction, changing the data file format, or relocating data files.
> overwrite: data files were deleted and added in a logical overwrite operation.
> delete: data files were removed and their contents logically deleted.
> If deletion files (with or without data files) are appended to the dataset, will we consider that an `append` operation? If so, if deletion and/or data files are appended, and whole files are also deleted, will we consider that an `overwrite`?
> 
> Given that the only apparent purpose of the operation field is to optimize snapshot expiration the above seems to meet its needs. An incremental reader can also skip `replace` snapshots but no others. Once it decides to read a snapshot I don't think there's any difference in how it processes the data for append/overwrite/delete cases.
> 
> On Thu, Jun 20, 2019 at 8:55 PM Ryan Blue <rblue@netflix.com <ma...@netflix.com>> wrote:
> I don’t see that we need [sequence numbers] for file/offset-deletes, since they apply to a specific file. They’re not harmful, but the don’t seem relevant.
> 
> These delete files will probably contain a path and an offset and could contain deletes for multiple files. In that case, the sequence number can be used to eliminate delete files that don’t need to be applied to a particular data file, just like the column equality deletes. Likewise, it can be used to drop the delete files when there are no data files with an older sequence number.
> 
> I don’t understand the purpose of the min sequence number, nor what the “min data seq” is.
> 
> Min sequence number would be used for pruning delete files without reading all the manifests to find out if there are old data files. If no manifest with data for a partition contains a file older than some sequence number N, then any delete file with a sequence number < N can be removed.
> 
> OK, so the minimum sequence number is an attribute of manifest files. Sounds good. It can likely permit us to optimize compaction operations as well (i.e., you can easily limit the operation to a subset of manifest files as long as they are the oldest ones).
>  
> The “min data seq” is the minimum sequence number of a data file. That seems like what we actually want for the pruning I described above.
> 
> I would expect a data file (appended rows or deletions by column value) to have a single sequence number that applies to the whole file. Even a delete-by-file-and-offset file can do with only a single sequence number (which must be larger than the sequence numbers of all deleted files). Why do we need a "minimum" data sequence per file?
> Off the top of my head [supporting non-key delete] requires adding additional information to the manifest file, indicating the columns that are used for the deletion. Only equality would be supported; if multiple columns were used, they would be combined with boolean-and. I don’t see anything too tricky about it.
> 
> Yes, exactly. I actually phrased it wrong initially. I think it would be simple to extend the equality deletes to do this. We just need a way to have global scope, not just partition scope.
> 
> I don't think anything special needs to be done with regards to scoping/partitioning of delete files. When scanning one or more data files, one must also consider any and all deletion files that could apply to them. The only way to prune deletion files from consideration is:
> All of your data files have at least one partition column in common.
> The deletion file is also partitioned on that column (at least).
> The value sets of the data files do not overlap the value sets of the deletion files in that column.
>  So given a dataset of sessions that is partitioned by device form factor and date, for example, you could have a delete (user_id=9876) in a deletion file that is not partitioned. And it would be "in scope" for all of those data files.
> 
> If you had the same dataset partitioned by hash(user_id) and your deletes were _also_ partitioned by hash(user_id) you would be able to prune those deletes while scanning the sessions.
> If we add this on a per-deletion file basis it is not clear if there is any relevance in preserving the concept of a unique row ID.
> 
> Agreed. That’s why I’ve been steering us away from the debate about whether keys are unique or not. Either way, a natural key delete must delete all of the records it matches.
> 
> I would assume that the maximum sequence number should appear in the table metadata
> 
> Agreed.
> 
> [W]ould you make it optional to assign a sequence number to a snapshot? “Replace” snapshots would not need one.
> 
> The only requirement is that it is monotonically increasing. If one isn’t used, we don’t have to increment. I’d say it is up to the implementation to decide. I would probably increment it every time to avoid errors.
> 
> -- 
> Ryan Blue
> Software Engineer
> Netflix
> 
> 
> -- 
> Ryan Blue
> Software Engineer
> Netflix

Re: Updates/Deletes/Upserts in Iceberg

[Ryan Blue <rb...@netflix.com.INVALID>]

Sorry I didn't get back to this thread last week. Let's try to have a video
call to sync up on this next week. What days would work for everyone?

rb

On Fri, Jun 21, 2019 at 9:06 AM Erik Wright <er...@shopify.com> wrote:

> With regards to operation values. Currently they are:
>
>    - append: data files were added and no files were removed.
>    - replace: data files were rewritten with the same data; i.e.,
>    compaction, changing the data file format, or relocating data files.
>    - overwrite: data files were deleted and added in a logical overwrite
>    operation.
>    - delete: data files were removed and their contents logically deleted.
>
> If deletion files (with or without data files) are appended to the
> dataset, will we consider that an `append` operation? If so, if deletion
> and/or data files are appended, and whole files are also deleted, will we
> consider that an `overwrite`?
>
> Given that the only apparent purpose of the operation field is to optimize
> snapshot expiration the above seems to meet its needs. An incremental
> reader can also skip `replace` snapshots but no others. Once it decides to
> read a snapshot I don't think there's any difference in how it processes
> the data for append/overwrite/delete cases.
>
> On Thu, Jun 20, 2019 at 8:55 PM Ryan Blue <rb...@netflix.com> wrote:
>
>> I don’t see that we need [sequence numbers] for file/offset-deletes,
>> since they apply to a specific file. They’re not harmful, but the don’t
>> seem relevant.
>>
>> These delete files will probably contain a path and an offset and could
>> contain deletes for multiple files. In that case, the sequence number can
>> be used to eliminate delete files that don’t need to be applied to a
>> particular data file, just like the column equality deletes. Likewise, it
>> can be used to drop the delete files when there are no data files with an
>> older sequence number.
>>
>> I don’t understand the purpose of the min sequence number, nor what the
>> “min data seq” is.
>>
>> Min sequence number would be used for pruning delete files without
>> reading all the manifests to find out if there are old data files. If no
>> manifest with data for a partition contains a file older than some sequence
>> number N, then any delete file with a sequence number < N can be removed.
>>
> OK, so the minimum sequence number is an attribute of manifest files.
> Sounds good. It can likely permit us to optimize compaction operations as
> well (i.e., you can easily limit the operation to a subset of manifest
> files as long as they are the oldest ones).
>
>
>> The “min data seq” is the minimum sequence number of a data file. That
>> seems like what we actually want for the pruning I described above.
>>
> I would expect a data file (appended rows or deletions by column value) to
> have a single sequence number that applies to the whole file. Even a
> delete-by-file-and-offset file can do with only a single sequence number
> (which must be larger than the sequence numbers of all deleted files). Why
> do we need a "minimum" data sequence per file?
>
>> Off the top of my head [supporting non-key delete] requires adding
>> additional information to the manifest file, indicating the columns that
>> are used for the deletion. Only equality would be supported; if multiple
>> columns were used, they would be combined with boolean-and. I don’t see
>> anything too tricky about it.
>>
>> Yes, exactly. I actually phrased it wrong initially. I think it would be
>> simple to extend the equality deletes to do this. We just need a way to
>> have global scope, not just partition scope.
>>
> I don't think anything special needs to be done with regards to
> scoping/partitioning of delete files. When scanning one or more data files,
> one must also consider any and all deletion files that could apply to them.
> The only way to prune deletion files from consideration is:
>
>    1. All of your data files have at least one partition column in common.
>    2. The deletion file is also partitioned on that column (at least).
>    3. The value sets of the data files do not overlap the value sets of
>    the deletion files in that column.
>
>  So given a dataset of sessions that is partitioned by device form factor
> and date, for example, you could have a delete (user_id=9876) in a deletion
> file that is not partitioned. And it would be "in scope" for all of those
> data files.
>
> If you had the same dataset partitioned by hash(user_id) and your deletes
> were _also_ partitioned by hash(user_id) you would be able to prune those
> deletes while scanning the sessions.
>
>> If we add this on a per-deletion file basis it is not clear if there is
>> any relevance in preserving the concept of a unique row ID.
>>
>> Agreed. That’s why I’ve been steering us away from the debate about
>> whether keys are unique or not. Either way, a natural key delete must
>> delete all of the records it matches.
>>
>> I would assume that the maximum sequence number should appear in the
>> table metadata
>>
>> Agreed.
>>
>> [W]ould you make it optional to assign a sequence number to a snapshot?
>> “Replace” snapshots would not need one.
>>
>> The only requirement is that it is monotonically increasing. If one isn’t
>> used, we don’t have to increment. I’d say it is up to the implementation to
>> decide. I would probably increment it every time to avoid errors.
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>

-- 
Ryan Blue
Software Engineer
Netflix

Re: Updates/Deletes/Upserts in Iceberg

[Erik Wright <er...@shopify.com.INVALID>]

With regards to operation values. Currently they are:

   - append: data files were added and no files were removed.
   - replace: data files were rewritten with the same data; i.e.,
   compaction, changing the data file format, or relocating data files.
   - overwrite: data files were deleted and added in a logical overwrite
   operation.
   - delete: data files were removed and their contents logically deleted.

If deletion files (with or without data files) are appended to the dataset,
will we consider that an `append` operation? If so, if deletion and/or data
files are appended, and whole files are also deleted, will we consider that
an `overwrite`?

Given that the only apparent purpose of the operation field is to optimize
snapshot expiration the above seems to meet its needs. An incremental
reader can also skip `replace` snapshots but no others. Once it decides to
read a snapshot I don't think there's any difference in how it processes
the data for append/overwrite/delete cases.

On Thu, Jun 20, 2019 at 8:55 PM Ryan Blue <rb...@netflix.com> wrote:

> I don’t see that we need [sequence numbers] for file/offset-deletes, since
> they apply to a specific file. They’re not harmful, but the don’t seem
> relevant.
>
> These delete files will probably contain a path and an offset and could
> contain deletes for multiple files. In that case, the sequence number can
> be used to eliminate delete files that don’t need to be applied to a
> particular data file, just like the column equality deletes. Likewise, it
> can be used to drop the delete files when there are no data files with an
> older sequence number.
>
> I don’t understand the purpose of the min sequence number, nor what the
> “min data seq” is.
>
> Min sequence number would be used for pruning delete files without reading
> all the manifests to find out if there are old data files. If no manifest
> with data for a partition contains a file older than some sequence number
> N, then any delete file with a sequence number < N can be removed.
>
OK, so the minimum sequence number is an attribute of manifest files.
Sounds good. It can likely permit us to optimize compaction operations as
well (i.e., you can easily limit the operation to a subset of manifest
files as long as they are the oldest ones).


> The “min data seq” is the minimum sequence number of a data file. That
> seems like what we actually want for the pruning I described above.
>
I would expect a data file (appended rows or deletions by column value) to
have a single sequence number that applies to the whole file. Even a
delete-by-file-and-offset file can do with only a single sequence number
(which must be larger than the sequence numbers of all deleted files). Why
do we need a "minimum" data sequence per file?

> Off the top of my head [supporting non-key delete] requires adding
> additional information to the manifest file, indicating the columns that
> are used for the deletion. Only equality would be supported; if multiple
> columns were used, they would be combined with boolean-and. I don’t see
> anything too tricky about it.
>
> Yes, exactly. I actually phrased it wrong initially. I think it would be
> simple to extend the equality deletes to do this. We just need a way to
> have global scope, not just partition scope.
>
I don't think anything special needs to be done with regards to
scoping/partitioning of delete files. When scanning one or more data files,
one must also consider any and all deletion files that could apply to them.
The only way to prune deletion files from consideration is:

   1. All of your data files have at least one partition column in common.
   2. The deletion file is also partitioned on that column (at least).
   3. The value sets of the data files do not overlap the value sets of the
   deletion files in that column.

 So given a dataset of sessions that is partitioned by device form factor
and date, for example, you could have a delete (user_id=9876) in a deletion
file that is not partitioned. And it would be "in scope" for all of those
data files.

If you had the same dataset partitioned by hash(user_id) and your deletes
were _also_ partitioned by hash(user_id) you would be able to prune those
deletes while scanning the sessions.

> If we add this on a per-deletion file basis it is not clear if there is
> any relevance in preserving the concept of a unique row ID.
>
> Agreed. That’s why I’ve been steering us away from the debate about
> whether keys are unique or not. Either way, a natural key delete must
> delete all of the records it matches.
>
> I would assume that the maximum sequence number should appear in the table
> metadata
>
> Agreed.
>
> [W]ould you make it optional to assign a sequence number to a snapshot?
> “Replace” snapshots would not need one.
>
> The only requirement is that it is monotonically increasing. If one isn’t
> used, we don’t have to increment. I’d say it is up to the implementation to
> decide. I would probably increment it every time to avoid errors.
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: Updates/Deletes/Upserts in Iceberg

[Ryan Blue <rb...@netflix.com.INVALID>]

I don’t see that we need [sequence numbers] for file/offset-deletes, since
they apply to a specific file. They’re not harmful, but the don’t seem
relevant.

These delete files will probably contain a path and an offset and could
contain deletes for multiple files. In that case, the sequence number can
be used to eliminate delete files that don’t need to be applied to a
particular data file, just like the column equality deletes. Likewise, it
can be used to drop the delete files when there are no data files with an
older sequence number.

I don’t understand the purpose of the min sequence number, nor what the
“min data seq” is.

Min sequence number would be used for pruning delete files without reading
all the manifests to find out if there are old data files. If no manifest
with data for a partition contains a file older than some sequence number
N, then any delete file with a sequence number < N can be removed.

The “min data seq” is the minimum sequence number of a data file. That
seems like what we actually want for the pruning I described above.

Off the top of my head [supporting non-key delete] requires adding
additional information to the manifest file, indicating the columns that
are used for the deletion. Only equality would be supported; if multiple
columns were used, they would be combined with boolean-and. I don’t see
anything too tricky about it.

Yes, exactly. I actually phrased it wrong initially. I think it would be
simple to extend the equality deletes to do this. We just need a way to
have global scope, not just partition scope.

If we add this on a per-deletion file basis it is not clear if there is any
relevance in preserving the concept of a unique row ID.

Agreed. That’s why I’ve been steering us away from the debate about whether
keys are unique or not. Either way, a natural key delete must delete all of
the records it matches.

I would assume that the maximum sequence number should appear in the table
metadata

Agreed.

[W]ould you make it optional to assign a sequence number to a snapshot?
“Replace” snapshots would not need one.

The only requirement is that it is monotonically increasing. If one isn’t
used, we don’t have to increment. I’d say it is up to the implementation to
decide. I would probably increment it every time to avoid errors.
-- 
Ryan Blue
Software Engineer
Netflix

Re: Updates/Deletes/Upserts in Iceberg

[Erik Wright <er...@shopify.com.INVALID>]

On Thu, Jun 20, 2019 at 12:57 PM Ryan Blue <rb...@netflix.com> wrote:

> Sounds like we’re in agreement on the direction! Let’s have a sync up
> sometime next week to make sure we are agreed and plan some of this work.
> What days work best for everyone?
>
Tuesday until 2PM PT
Wednesday 11AM -> 2PM PT
Thursday 9AM -> 10:30AM / 12PM -> 2PM PT
Friday until 2PM PT


> Here’s a quick summary of the approach from my perspective:
>
>    - Will define two options for writing delete files that identify rows
>    by:
>       1. File and offset for MERGE INTO or similar operations, and
>       2. Column equality for encoding deletes without reading data
>    - Will add a delete file type to manifests
>    - Will add sequence numbers to existing metadata to track when to
>    apply delete files to data files (in time), used by both delete file options
>
> I don't see that we need these for file/offset-deletes, since they apply
to a specific file. They're not harmful, but the don't seem relevant.

>
>    - Will add a tracking sequence number to manifest file metadata in the
>    manifest list, as well as a min sequence number for each manifest (min data
>    seq as well?)
>
> I don't understand the purpose of the min sequence number, nor what the
"min data seq" is.

>
>    - For column equality deletes, the partition of the file in the
>    manifest will track the partition that it applies to (stats filtering to be
>    done at read time)
>
> I’d also like to have a way to encode deletes across the entire dataset,
> for GDPR deletes (e.g. delete a user from all sessions partitioned by
> time). Any ideas about how to encode this?
>
Off the top of my head this requires adding additional information to the
manifest file, indicating the columns that are used for the deletion. Only
equality would be supported; if multiple columns were used, they would be
combined with boolean-and. I don't see anything too tricky about it.

More replies to specific points:
>
> If there is a monotonically increasing value per snapshot, should we
> simply update the requirement for the snapshot ID to require it to be
> monotonically increasing?
>
> I was thinking about this and I don’t think it’s a good idea, but I could
> be convinced otherwise.
>
> When compacting data and delete files, the sequence number of the
> compacted result is set to the sequence of the latest delete file applied
> to the data, but the compacted file is actually added in the replace
> snapshot. So we wouldn’t be able to identify the files that were added by a
> compaction without running a diff with another snapshot. I think that we
> don’t want to lose that information because we want to be able to recover
> what operation created any given file, if the snapshot is still around with
> metadata.
>
You've convinced me, below, that we need to be able to commit new files in
a new snapshot with a past sequence number.

So far, we will have the following information per file:

   - The snapshot that added it.
   - The snapshot that deleted it.
   - Its sequence number (which is, indirectly, a snapshot ID, but not
   necessarily the one in which the file was added)

Regardless of whether the sequence number and the snapshot are the same or
simply have a 1:1 mapping, these three properties need to be tracked. It's
true that using different names (and different value spaces) for them
reduces some potential confusion.

I would assume that the maximum sequence number should appear in the table
metadata, as the alternative requires scanning manifests. Furthermore, I
could imagine some cases where the number goes backwards according to the
manifests (because the deletes from 13 are applied, but not all of the
deletes from 12, so the compacted files are given sequence number 12 and
there are no remaining files from number 13).

Given that, would you make it optional to assign a sequence number to a
snapshot? "Replace" snapshots would not need one.

> The sequence number assigned to the resulting files is irrelevant if *all*
> relevant deletes have been applied.
>
> I agree, but we can also use the sequence number to avoid commit
> conflicts. If a delete is committed just before a compaction, we don’t want
> to re-run the compaction. Using the highest sequence number from the delete
> files applied to the data, we can confidently commit the compaction,
> knowing that any new deltas will be applied.
>
> Ultimately, all files would appear to be eligible for compaction in any
> rewrite snapshot.
>
> Yes, which I think is a benefit, right?
>
Absolutely.

> I don’t think that the other approach inhibits aging off data, it just
> represents the deletion of the data differently.
>
> True, but if we can preserve the existing functionality then this is a
> quick operation. Usually, these file-level deletes align with partitioning
> so they are quick to run. And, we can prune any delete files that no longer
> have data files after it happens.
>
> On Thu, Jun 20, 2019 at 8:06 AM Erik Wright <er...@shopify.com>
> wrote:
>
>>
>>
>> On Wed, Jun 19, 2019 at 6:39 PM Ryan Blue <rb...@netflix.com.invalid>
>> wrote:
>>
>>> Replies inline.
>>>
>>> On Mon, Jun 17, 2019 at 10:59 AM Erik Wright
>>> erik.wright@shopify.com.invalid
>>> <ht...@shopify.com.invalid> wrote:
>>>
>>> Because snapshots and versions are basically the same idea, we don’t
>>>>> need both. If we were to add versions, they should replace snapshots.
>>>>>
>>>>
>>>> I'm a little confused by this. You mention sequence numbers quite a
>>>> bit, but then say that we should not introduce versions. From my
>>>> understanding, sequence numbers and versions are essentially identical.
>>>> What has changed is where they are encoded (in file metadata vs. in
>>>> snapshot metadata). It would also seem necessary to keep them distinct from
>>>> snapshots in order to be able to collapse/compact files older than N
>>>> (oldest valid version) without losing the ability to incrementally apply
>>>> the changes from N + 1.
>>>>
>>> Snapshots are versions of the table, but Iceberg does not use a
>>> monotonically increasing identifier to track them. Your proposal added
>>> table versions in addition to snapshots to get a monotonically increasing
>>> ID, but my point is that we don’t need to duplicate the “table state at
>>> some time” idea by adding a layer of versions inside snapshots. We just
>>> need a monotonically increasing ID associated with each snapshot. That ID
>>> what I’m calling a sequence number. Each snapshot will have a new sequence
>>> number used to track the order of files for applying changes that are
>>> introduced by that snapshot.
>>>
>> If there is a monotonically increasing value per snapshot, should we
>> simply update the requirement for the snapshot ID to require it to be
>> monotonically increasing?
>>
>>
>>> With sequence numbers embedded in metadata, you can still compact. If
>>> you have a file with sequence number N, then updated by sequence numbers
>>> N+2 and N+3, you can compact the changes from N and N+2 by writing the
>>> compacted file with sequence N+2. Rewriting files in N+2 is still possible,
>>> just like with the scheme you suggested, where the version N+2 would be
>>> rewritten.
>>>
>>> A key difference is that because the approach using sequence numbers
>>> doesn’t require keeping state around in the root metadata file, tables are
>>> not required to rewrite or compact data files just to keep the table
>>> functioning. If the data won’t be read, then why bother changing it?
>>>
>>> Because I am having a really hard time seeing how things would work
>>>> without these sequence numbers (versions), and given their prominence in
>>>> your reply, my remaining comments assume they are present, distinct from
>>>> snapshots, and monotonically increasing.
>>>>
>>> Yes, a sequence number would be added to the existing metadata. Each
>>> snapshot would produce a new sequence number.
>>>
>>> Another way of thinking about it: if versions are stored as data/delete
>>>>> file metadata and not in the table metadata file, then the complete history
>>>>> can be kept.
>>>>
>>>>
>>>> Sure. Although version expiry would still be an important process to
>>>> improve read performance and reduce storage overhead.
>>>>
>>> Maybe. If it is unlikely that the data will be read, the best option may
>>> be to leave the original data file and associated delete file.
>>>
>> It's safe to say that is optional for any given use-case, but that it is
>> a required capability.
>>
>>> Based on your proposal, it would seem that we can basically collapse any
>>>> given data file with its corresponding deletions. It's important to
>>>> consider how this will affect incremental readers. The following make sense
>>>> to me:
>>>>
>>>>    1. We should track an oldest-valid-version in the table metadata.
>>>>    No changes newer than this version should be collapsed, as a reader who has
>>>>    consumed up to that version must be able to continue reading.
>>>>
>>>> Table versions are tracked by snapshots, which are kept for a window of
>>> time. The oldest valid version is the oldest snapshot in the table. New
>>> snapshots can always compact or rewrite older data and incremental
>>> consumers ignore those changes by skipping snapshots where the operation
>>> was replace because the data has not changed.
>>>
>> Incremental consumers operate on append, delete, and overwrite snapshots
>>> and consume data files that are added or deleted. With delete files, these
>>> would also consume delete files that were added in a snapshot.
>>>
>> Sure, when a whole file is deleted there is already a column for
>> indicating that, so we do not need to represent it a second time.
>>
>>
>>> It is unlikely that sequence number would be used by incremental
>>> consumers because the changes are already available for each snapshot. This
>>> distinction is the main reason why I’m calling these sequence numbers and
>>> not “version”. The sequence number for a file indicates what delete files
>>> need to be applied; files can be rewritten the new copy gets a different
>>> sequence number as you describe just below.
>>>
>> Interesting. This explains the difference in the way we are looking at
>> this. Yes, I agree that this ought to work and seems more consistent with
>> the way that Iceberg works currently.
>>
>>>
>>>>    1. Any data file strictly older than the oldest-valid-version is
>>>>    eligible for delete collapsing. During delete collapsing, all deletions up
>>>>    to version N (any version <= the oldest-valid-version) are applied to the
>>>>    file. *The newly created file should be assigned a sequence number
>>>>    of N.*
>>>>
>>>> First, a data file is kept until the last snapshot that references it
>>> is expired. That enable incremental consumers without a restriction on
>>> rewriting the file in a later replace snapshot.
>>>
>>> Second, I agree with your description of rewriting the file. That’s what
>>> I was trying to say above.
>>>
>> Given that incremental consumers will skip "replace" snapshots and will
>> not attempt to incrementally read the changes introduced in version "N"
>> using the manifest list file for version "N+1", there would appear to be no
>> restrictions on how much data can be collapsed in any rewrite operation.
>> The sequence number assigned to the resulting files is irrelevant if _all_
>> relevant deletes have been applied.
>>
>>>
>>>>    1. Any delete file that has been completely applied is eligible for
>>>>    expiry. A delete file is completely applied if its sequence number is <=
>>>>    all data files to which it may apply (based on partitioning data).
>>>>
>>>> Yes, I agree.
>>>
>>>
>>>>    1. Any data file equal to or older than the oldest-valid-version is
>>>>    eligible for compaction. During compaction, some or all data files up to
>>>>    version N (any version <= the oldest-valid version) are selected. There may
>>>>    not be any deletions applicable to any of the selected data files. The
>>>>    selected data files are read, rewritten as desired (partitioning,
>>>>    compaction, sorting, etc.). The new files are included in the new snapshot (*with
>>>>    sequence number N*) while the old files are dropped.
>>>>
>>>> Yes, I agree.
>>>
>> Ultimately, all files would appear to be eligible for compaction in any
>> rewrite snapshot.
>>
>>>
>>>>    1. Delete collapsing, delete expiry, and compaction may be
>>>>    performed in a single commit. The oldest-valid-version may be advanced
>>>>    during this process as well. The outcome must logically be consistent with
>>>>    applying these steps independently.
>>>>
>>>> Because snapshots (table versions) are independent, they can always be
>>> expired. Incremental consumers must process a new snapshot before it is old
>>> enough to expire. The time period after which snapshots are expired is up
>>> to the process running ExpireSnapshots.
>>>
>> Agreed.
>>
>>> Important to note in the above is that, during collapsing/compaction,
>>>> new files may be emitted with a version number N that are older than the
>>>> most recent version. This is important to ensure that all deltas newer than
>>>> N are appropriately applied, and that incremental readers are able to
>>>> continue processing the dataset.
>>>>
>>> Because incremental consumers operate using snapshots and not sequence
>>> numbers, I think this is decoupled in the approach I’m proposing.
>>>
>> Agreed.
>>
>>> It is also compatible with file-level deletes used to age off old data
>>>>> (e.g. delete where day(ts) < '2019-01-01').
>>>>
>>>>
>>>> This particular operation is not compatible with incremental
>>>> consumption.
>>>>
>>> I disagree. This just encodes deletes for an entire file worth of rows
>>> in the manifest file instead of in a delete file. Current metadata tracks
>>> when files are deleted and we would also associate a sequence number with
>>> those changes, if we needed to. Incremental consumers will work like they
>>> do today: they will consume snapshots that contain changes
>>> (append/delete/overwrite) and will get the files that were changed in that
>>> snapshot.
>>>
>> Agreed.
>>
>>
>>> Also, being able to cleanly age off data is a hard requirement for
>>> Iceberg. We all need to comply with data policies with age limits and we
>>> need to ensure that we can cleanly apply those changes.
>>>
>> I don't think that the other approach inhibits aging off data, it just
>> represents the deletion of the data differently. In any case, I agree that
>> we can reuse the existing "deleted in snapshot" mechanism.
>>
>>> One must still generate a deletion that is associated with a new
>>>> sequence number. I could see a reasonable workaround where, in order to
>>>> delete an entire file (insertion file `x.dat`, sequence number X) one would
>>>> reference the same file a second time (deletion file `x.dat, sequence
>>>> number Y > X). Since an insertion file and a deletion file have compatible
>>>> formats, there is no need to rewrite this file simply to mark each row in
>>>> it as deleted. A savvy consumer would be able to tell that this is a
>>>> whole-file deletion while a naïve consumer would be able to apply them
>>>> using the basic algorithm.
>>>>
>>> With file-level deletes, the files are no longer processed by readers,
>>> unless an incremental reader needs to read all of the deletes from the file.
>>>
>>> If it seems like we are roughly on the same page I will take a stab at
>>>> updating that document to go to the same level of detail that it does now
>>>> but use the sequence-number approach.
>>>>
>>> Sure, that sounds good.
>>>
>> Phew. I think we are nearly there!
>>
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>>
>>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: Updates/Deletes/Upserts in Iceberg

[Ryan Blue <rb...@netflix.com.INVALID>]

Sounds like we’re in agreement on the direction! Let’s have a sync up
sometime next week to make sure we are agreed and plan some of this work.
What days work best for everyone?

Here’s a quick summary of the approach from my perspective:

   - Will define two options for writing delete files that identify rows by:
      1. File and offset for MERGE INTO or similar operations, and
      2. Column equality for encoding deletes without reading data
   - Will add a delete file type to manifests
   - Will add sequence numbers to existing metadata to track when to apply
   delete files to data files (in time), used by both delete file options
   - Will add a tracking sequence number to manifest file metadata in the
   manifest list, as well as a min sequence number for each manifest (min data
   seq as well?)
   - For column equality deletes, the partition of the file in the manifest
   will track the partition that it applies to (stats filtering to be done at
   read time)

I’d also like to have a way to encode deletes across the entire dataset,
for GDPR deletes (e.g. delete a user from all sessions partitioned by
time). Any ideas about how to encode this?

More replies to specific points:

If there is a monotonically increasing value per snapshot, should we simply
update the requirement for the snapshot ID to require it to be
monotonically increasing?

I was thinking about this and I don’t think it’s a good idea, but I could
be convinced otherwise.

When compacting data and delete files, the sequence number of the compacted
result is set to the sequence of the latest delete file applied to the
data, but the compacted file is actually added in the replace snapshot. So
we wouldn’t be able to identify the files that were added by a compaction
without running a diff with another snapshot. I think that we don’t want to
lose that information because we want to be able to recover what operation
created any given file, if the snapshot is still around with metadata.

The sequence number assigned to the resulting files is irrelevant if *all*
relevant deletes have been applied.

I agree, but we can also use the sequence number to avoid commit conflicts.
If a delete is committed just before a compaction, we don’t want to re-run
the compaction. Using the highest sequence number from the delete files
applied to the data, we can confidently commit the compaction, knowing that
any new deltas will be applied.

Ultimately, all files would appear to be eligible for compaction in any
rewrite snapshot.

Yes, which I think is a benefit, right?

I don’t think that the other approach inhibits aging off data, it just
represents the deletion of the data differently.

True, but if we can preserve the existing functionality then this is a
quick operation. Usually, these file-level deletes align with partitioning
so they are quick to run. And, we can prune any delete files that no longer
have data files after it happens.

On Thu, Jun 20, 2019 at 8:06 AM Erik Wright <er...@shopify.com> wrote:

>
>
> On Wed, Jun 19, 2019 at 6:39 PM Ryan Blue <rb...@netflix.com.invalid>
> wrote:
>
>> Replies inline.
>>
>> On Mon, Jun 17, 2019 at 10:59 AM Erik Wright
>> erik.wright@shopify.com.invalid
>> <ht...@shopify.com.invalid> wrote:
>>
>> Because snapshots and versions are basically the same idea, we don’t need
>>>> both. If we were to add versions, they should replace snapshots.
>>>>
>>>
>>> I'm a little confused by this. You mention sequence numbers quite a bit,
>>> but then say that we should not introduce versions. From my understanding,
>>> sequence numbers and versions are essentially identical. What has changed
>>> is where they are encoded (in file metadata vs. in snapshot metadata). It
>>> would also seem necessary to keep them distinct from snapshots in order to
>>> be able to collapse/compact files older than N (oldest valid version)
>>> without losing the ability to incrementally apply the changes from N + 1.
>>>
>> Snapshots are versions of the table, but Iceberg does not use a
>> monotonically increasing identifier to track them. Your proposal added
>> table versions in addition to snapshots to get a monotonically increasing
>> ID, but my point is that we don’t need to duplicate the “table state at
>> some time” idea by adding a layer of versions inside snapshots. We just
>> need a monotonically increasing ID associated with each snapshot. That ID
>> what I’m calling a sequence number. Each snapshot will have a new sequence
>> number used to track the order of files for applying changes that are
>> introduced by that snapshot.
>>
> If there is a monotonically increasing value per snapshot, should we
> simply update the requirement for the snapshot ID to require it to be
> monotonically increasing?
>
>
>> With sequence numbers embedded in metadata, you can still compact. If you
>> have a file with sequence number N, then updated by sequence numbers N+2
>> and N+3, you can compact the changes from N and N+2 by writing the
>> compacted file with sequence N+2. Rewriting files in N+2 is still possible,
>> just like with the scheme you suggested, where the version N+2 would be
>> rewritten.
>>
>> A key difference is that because the approach using sequence numbers
>> doesn’t require keeping state around in the root metadata file, tables are
>> not required to rewrite or compact data files just to keep the table
>> functioning. If the data won’t be read, then why bother changing it?
>>
>> Because I am having a really hard time seeing how things would work
>>> without these sequence numbers (versions), and given their prominence in
>>> your reply, my remaining comments assume they are present, distinct from
>>> snapshots, and monotonically increasing.
>>>
>> Yes, a sequence number would be added to the existing metadata. Each
>> snapshot would produce a new sequence number.
>>
>> Another way of thinking about it: if versions are stored as data/delete
>>>> file metadata and not in the table metadata file, then the complete history
>>>> can be kept.
>>>
>>>
>>> Sure. Although version expiry would still be an important process to
>>> improve read performance and reduce storage overhead.
>>>
>> Maybe. If it is unlikely that the data will be read, the best option may
>> be to leave the original data file and associated delete file.
>>
> It's safe to say that is optional for any given use-case, but that it is a
> required capability.
>
>> Based on your proposal, it would seem that we can basically collapse any
>>> given data file with its corresponding deletions. It's important to
>>> consider how this will affect incremental readers. The following make sense
>>> to me:
>>>
>>>    1. We should track an oldest-valid-version in the table metadata. No
>>>    changes newer than this version should be collapsed, as a reader who has
>>>    consumed up to that version must be able to continue reading.
>>>
>>> Table versions are tracked by snapshots, which are kept for a window of
>> time. The oldest valid version is the oldest snapshot in the table. New
>> snapshots can always compact or rewrite older data and incremental
>> consumers ignore those changes by skipping snapshots where the operation
>> was replace because the data has not changed.
>>
> Incremental consumers operate on append, delete, and overwrite snapshots
>> and consume data files that are added or deleted. With delete files, these
>> would also consume delete files that were added in a snapshot.
>>
> Sure, when a whole file is deleted there is already a column for
> indicating that, so we do not need to represent it a second time.
>
>
>> It is unlikely that sequence number would be used by incremental
>> consumers because the changes are already available for each snapshot. This
>> distinction is the main reason why I’m calling these sequence numbers and
>> not “version”. The sequence number for a file indicates what delete files
>> need to be applied; files can be rewritten the new copy gets a different
>> sequence number as you describe just below.
>>
> Interesting. This explains the difference in the way we are looking at
> this. Yes, I agree that this ought to work and seems more consistent with
> the way that Iceberg works currently.
>
>>
>>>    1. Any data file strictly older than the oldest-valid-version is
>>>    eligible for delete collapsing. During delete collapsing, all deletions up
>>>    to version N (any version <= the oldest-valid-version) are applied to the
>>>    file. *The newly created file should be assigned a sequence number
>>>    of N.*
>>>
>>> First, a data file is kept until the last snapshot that references it is
>> expired. That enable incremental consumers without a restriction on
>> rewriting the file in a later replace snapshot.
>>
>> Second, I agree with your description of rewriting the file. That’s what
>> I was trying to say above.
>>
> Given that incremental consumers will skip "replace" snapshots and will
> not attempt to incrementally read the changes introduced in version "N"
> using the manifest list file for version "N+1", there would appear to be no
> restrictions on how much data can be collapsed in any rewrite operation.
> The sequence number assigned to the resulting files is irrelevant if _all_
> relevant deletes have been applied.
>
>>
>>>    1. Any delete file that has been completely applied is eligible for
>>>    expiry. A delete file is completely applied if its sequence number is <=
>>>    all data files to which it may apply (based on partitioning data).
>>>
>>> Yes, I agree.
>>
>>
>>>    1. Any data file equal to or older than the oldest-valid-version is
>>>    eligible for compaction. During compaction, some or all data files up to
>>>    version N (any version <= the oldest-valid version) are selected. There may
>>>    not be any deletions applicable to any of the selected data files. The
>>>    selected data files are read, rewritten as desired (partitioning,
>>>    compaction, sorting, etc.). The new files are included in the new snapshot (*with
>>>    sequence number N*) while the old files are dropped.
>>>
>>> Yes, I agree.
>>
> Ultimately, all files would appear to be eligible for compaction in any
> rewrite snapshot.
>
>>
>>>    1. Delete collapsing, delete expiry, and compaction may be performed
>>>    in a single commit. The oldest-valid-version may be advanced during this
>>>    process as well. The outcome must logically be consistent with applying
>>>    these steps independently.
>>>
>>> Because snapshots (table versions) are independent, they can always be
>> expired. Incremental consumers must process a new snapshot before it is old
>> enough to expire. The time period after which snapshots are expired is up
>> to the process running ExpireSnapshots.
>>
> Agreed.
>
>> Important to note in the above is that, during collapsing/compaction, new
>>> files may be emitted with a version number N that are older than the most
>>> recent version. This is important to ensure that all deltas newer than N
>>> are appropriately applied, and that incremental readers are able to
>>> continue processing the dataset.
>>>
>> Because incremental consumers operate using snapshots and not sequence
>> numbers, I think this is decoupled in the approach I’m proposing.
>>
> Agreed.
>
>> It is also compatible with file-level deletes used to age off old data
>>>> (e.g. delete where day(ts) < '2019-01-01').
>>>
>>>
>>> This particular operation is not compatible with incremental consumption.
>>>
>> I disagree. This just encodes deletes for an entire file worth of rows in
>> the manifest file instead of in a delete file. Current metadata tracks when
>> files are deleted and we would also associate a sequence number with those
>> changes, if we needed to. Incremental consumers will work like they do
>> today: they will consume snapshots that contain changes
>> (append/delete/overwrite) and will get the files that were changed in that
>> snapshot.
>>
> Agreed.
>
>
>> Also, being able to cleanly age off data is a hard requirement for
>> Iceberg. We all need to comply with data policies with age limits and we
>> need to ensure that we can cleanly apply those changes.
>>
> I don't think that the other approach inhibits aging off data, it just
> represents the deletion of the data differently. In any case, I agree that
> we can reuse the existing "deleted in snapshot" mechanism.
>
>> One must still generate a deletion that is associated with a new sequence
>>> number. I could see a reasonable workaround where, in order to delete an
>>> entire file (insertion file `x.dat`, sequence number X) one would reference
>>> the same file a second time (deletion file `x.dat, sequence number Y > X).
>>> Since an insertion file and a deletion file have compatible formats, there
>>> is no need to rewrite this file simply to mark each row in it as deleted. A
>>> savvy consumer would be able to tell that this is a whole-file deletion
>>> while a naïve consumer would be able to apply them using the basic
>>> algorithm.
>>>
>> With file-level deletes, the files are no longer processed by readers,
>> unless an incremental reader needs to read all of the deletes from the file.
>>
>> If it seems like we are roughly on the same page I will take a stab at
>>> updating that document to go to the same level of detail that it does now
>>> but use the sequence-number approach.
>>>
>> Sure, that sounds good.
>>
> Phew. I think we are nearly there!
>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>

-- 
Ryan Blue
Software Engineer
Netflix

Re: Updates/Deletes/Upserts in Iceberg

[Erik Wright <er...@shopify.com.INVALID>]

On Wed, Jun 19, 2019 at 6:39 PM Ryan Blue <rb...@netflix.com.invalid> wrote:

> Replies inline.
>
> On Mon, Jun 17, 2019 at 10:59 AM Erik Wright
> erik.wright@shopify.com.invalid
> <ht...@shopify.com.invalid> wrote:
>
> Because snapshots and versions are basically the same idea, we don’t need
>>> both. If we were to add versions, they should replace snapshots.
>>>
>>
>> I'm a little confused by this. You mention sequence numbers quite a bit,
>> but then say that we should not introduce versions. From my understanding,
>> sequence numbers and versions are essentially identical. What has changed
>> is where they are encoded (in file metadata vs. in snapshot metadata). It
>> would also seem necessary to keep them distinct from snapshots in order to
>> be able to collapse/compact files older than N (oldest valid version)
>> without losing the ability to incrementally apply the changes from N + 1.
>>
> Snapshots are versions of the table, but Iceberg does not use a
> monotonically increasing identifier to track them. Your proposal added
> table versions in addition to snapshots to get a monotonically increasing
> ID, but my point is that we don’t need to duplicate the “table state at
> some time” idea by adding a layer of versions inside snapshots. We just
> need a monotonically increasing ID associated with each snapshot. That ID
> what I’m calling a sequence number. Each snapshot will have a new sequence
> number used to track the order of files for applying changes that are
> introduced by that snapshot.
>
If there is a monotonically increasing value per snapshot, should we simply
update the requirement for the snapshot ID to require it to be
monotonically increasing?


> With sequence numbers embedded in metadata, you can still compact. If you
> have a file with sequence number N, then updated by sequence numbers N+2
> and N+3, you can compact the changes from N and N+2 by writing the
> compacted file with sequence N+2. Rewriting files in N+2 is still possible,
> just like with the scheme you suggested, where the version N+2 would be
> rewritten.
>
> A key difference is that because the approach using sequence numbers
> doesn’t require keeping state around in the root metadata file, tables are
> not required to rewrite or compact data files just to keep the table
> functioning. If the data won’t be read, then why bother changing it?
>
> Because I am having a really hard time seeing how things would work
>> without these sequence numbers (versions), and given their prominence in
>> your reply, my remaining comments assume they are present, distinct from
>> snapshots, and monotonically increasing.
>>
> Yes, a sequence number would be added to the existing metadata. Each
> snapshot would produce a new sequence number.
>
> Another way of thinking about it: if versions are stored as data/delete
>>> file metadata and not in the table metadata file, then the complete history
>>> can be kept.
>>
>>
>> Sure. Although version expiry would still be an important process to
>> improve read performance and reduce storage overhead.
>>
> Maybe. If it is unlikely that the data will be read, the best option may
> be to leave the original data file and associated delete file.
>
It's safe to say that is optional for any given use-case, but that it is a
required capability.

> Based on your proposal, it would seem that we can basically collapse any
>> given data file with its corresponding deletions. It's important to
>> consider how this will affect incremental readers. The following make sense
>> to me:
>>
>>    1. We should track an oldest-valid-version in the table metadata. No
>>    changes newer than this version should be collapsed, as a reader who has
>>    consumed up to that version must be able to continue reading.
>>
>> Table versions are tracked by snapshots, which are kept for a window of
> time. The oldest valid version is the oldest snapshot in the table. New
> snapshots can always compact or rewrite older data and incremental
> consumers ignore those changes by skipping snapshots where the operation
> was replace because the data has not changed.
>
Incremental consumers operate on append, delete, and overwrite snapshots
> and consume data files that are added or deleted. With delete files, these
> would also consume delete files that were added in a snapshot.
>
Sure, when a whole file is deleted there is already a column for indicating
that, so we do not need to represent it a second time.


> It is unlikely that sequence number would be used by incremental consumers
> because the changes are already available for each snapshot. This
> distinction is the main reason why I’m calling these sequence numbers and
> not “version”. The sequence number for a file indicates what delete files
> need to be applied; files can be rewritten the new copy gets a different
> sequence number as you describe just below.
>
Interesting. This explains the difference in the way we are looking at
this. Yes, I agree that this ought to work and seems more consistent with
the way that Iceberg works currently.

>
>>    1. Any data file strictly older than the oldest-valid-version is
>>    eligible for delete collapsing. During delete collapsing, all deletions up
>>    to version N (any version <= the oldest-valid-version) are applied to the
>>    file. *The newly created file should be assigned a sequence number of
>>    N.*
>>
>> First, a data file is kept until the last snapshot that references it is
> expired. That enable incremental consumers without a restriction on
> rewriting the file in a later replace snapshot.
>
> Second, I agree with your description of rewriting the file. That’s what I
> was trying to say above.
>
Given that incremental consumers will skip "replace" snapshots and will not
attempt to incrementally read the changes introduced in version "N" using
the manifest list file for version "N+1", there would appear to be no
restrictions on how much data can be collapsed in any rewrite operation.
The sequence number assigned to the resulting files is irrelevant if _all_
relevant deletes have been applied.

>
>>    1. Any delete file that has been completely applied is eligible for
>>    expiry. A delete file is completely applied if its sequence number is <=
>>    all data files to which it may apply (based on partitioning data).
>>
>> Yes, I agree.
>
>
>>    1. Any data file equal to or older than the oldest-valid-version is
>>    eligible for compaction. During compaction, some or all data files up to
>>    version N (any version <= the oldest-valid version) are selected. There may
>>    not be any deletions applicable to any of the selected data files. The
>>    selected data files are read, rewritten as desired (partitioning,
>>    compaction, sorting, etc.). The new files are included in the new snapshot (*with
>>    sequence number N*) while the old files are dropped.
>>
>> Yes, I agree.
>
Ultimately, all files would appear to be eligible for compaction in any
rewrite snapshot.

>
>>    1. Delete collapsing, delete expiry, and compaction may be performed
>>    in a single commit. The oldest-valid-version may be advanced during this
>>    process as well. The outcome must logically be consistent with applying
>>    these steps independently.
>>
>> Because snapshots (table versions) are independent, they can always be
> expired. Incremental consumers must process a new snapshot before it is old
> enough to expire. The time period after which snapshots are expired is up
> to the process running ExpireSnapshots.
>
Agreed.

> Important to note in the above is that, during collapsing/compaction, new
>> files may be emitted with a version number N that are older than the most
>> recent version. This is important to ensure that all deltas newer than N
>> are appropriately applied, and that incremental readers are able to
>> continue processing the dataset.
>>
> Because incremental consumers operate using snapshots and not sequence
> numbers, I think this is decoupled in the approach I’m proposing.
>
Agreed.

> It is also compatible with file-level deletes used to age off old data
>>> (e.g. delete where day(ts) < '2019-01-01').
>>
>>
>> This particular operation is not compatible with incremental consumption.
>>
> I disagree. This just encodes deletes for an entire file worth of rows in
> the manifest file instead of in a delete file. Current metadata tracks when
> files are deleted and we would also associate a sequence number with those
> changes, if we needed to. Incremental consumers will work like they do
> today: they will consume snapshots that contain changes
> (append/delete/overwrite) and will get the files that were changed in that
> snapshot.
>
Agreed.


> Also, being able to cleanly age off data is a hard requirement for
> Iceberg. We all need to comply with data policies with age limits and we
> need to ensure that we can cleanly apply those changes.
>
I don't think that the other approach inhibits aging off data, it just
represents the deletion of the data differently. In any case, I agree that
we can reuse the existing "deleted in snapshot" mechanism.

> One must still generate a deletion that is associated with a new sequence
>> number. I could see a reasonable workaround where, in order to delete an
>> entire file (insertion file `x.dat`, sequence number X) one would reference
>> the same file a second time (deletion file `x.dat, sequence number Y > X).
>> Since an insertion file and a deletion file have compatible formats, there
>> is no need to rewrite this file simply to mark each row in it as deleted. A
>> savvy consumer would be able to tell that this is a whole-file deletion
>> while a naïve consumer would be able to apply them using the basic
>> algorithm.
>>
> With file-level deletes, the files are no longer processed by readers,
> unless an incremental reader needs to read all of the deletes from the file.
>
> If it seems like we are roughly on the same page I will take a stab at
>> updating that document to go to the same level of detail that it does now
>> but use the sequence-number approach.
>>
> Sure, that sounds good.
>
Phew. I think we are nearly there!

> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: Updates/Deletes/Upserts in Iceberg

[Ryan Blue <rb...@netflix.com.INVALID>]

Replies inline.

On Mon, Jun 17, 2019 at 10:59 AM Erik Wright erik.wright@shopify.com.invalid
<ht...@shopify.com.invalid> wrote:

Because snapshots and versions are basically the same idea, we don’t need
>> both. If we were to add versions, they should replace snapshots.
>>
>
> I'm a little confused by this. You mention sequence numbers quite a bit,
> but then say that we should not introduce versions. From my understanding,
> sequence numbers and versions are essentially identical. What has changed
> is where they are encoded (in file metadata vs. in snapshot metadata). It
> would also seem necessary to keep them distinct from snapshots in order to
> be able to collapse/compact files older than N (oldest valid version)
> without losing the ability to incrementally apply the changes from N + 1.
>
Snapshots are versions of the table, but Iceberg does not use a
monotonically increasing identifier to track them. Your proposal added
table versions in addition to snapshots to get a monotonically increasing
ID, but my point is that we don’t need to duplicate the “table state at
some time” idea by adding a layer of versions inside snapshots. We just
need a monotonically increasing ID associated with each snapshot. That ID
what I’m calling a sequence number. Each snapshot will have a new sequence
number used to track the order of files for applying changes that are
introduced by that snapshot.

With sequence numbers embedded in metadata, you can still compact. If you
have a file with sequence number N, then updated by sequence numbers N+2
and N+3, you can compact the changes from N and N+2 by writing the
compacted file with sequence N+2. Rewriting files in N+2 is still possible,
just like with the scheme you suggested, where the version N+2 would be
rewritten.

A key difference is that because the approach using sequence numbers
doesn’t require keeping state around in the root metadata file, tables are
not required to rewrite or compact data files just to keep the table
functioning. If the data won’t be read, then why bother changing it?

Because I am having a really hard time seeing how things would work without
> these sequence numbers (versions), and given their prominence in your
> reply, my remaining comments assume they are present, distinct from
> snapshots, and monotonically increasing.
>
Yes, a sequence number would be added to the existing metadata. Each
snapshot would produce a new sequence number.

Another way of thinking about it: if versions are stored as data/delete
>> file metadata and not in the table metadata file, then the complete history
>> can be kept.
>
>
> Sure. Although version expiry would still be an important process to
> improve read performance and reduce storage overhead.
>
Maybe. If it is unlikely that the data will be read, the best option may be
to leave the original data file and associated delete file.

Based on your proposal, it would seem that we can basically collapse any
> given data file with its corresponding deletions. It's important to
> consider how this will affect incremental readers. The following make sense
> to me:
>
>    1. We should track an oldest-valid-version in the table metadata. No
>    changes newer than this version should be collapsed, as a reader who has
>    consumed up to that version must be able to continue reading.
>
> Table versions are tracked by snapshots, which are kept for a window of
time. The oldest valid version is the oldest snapshot in the table. New
snapshots can always compact or rewrite older data and incremental
consumers ignore those changes by skipping snapshots where the operation
was replace because the data has not changed.

Incremental consumers operate on append, delete, and overwrite snapshots
and consume data files that are added or deleted. With delete files, these
would also consume delete files that were added in a snapshot.

It is unlikely that sequence number would be used by incremental consumers
because the changes are already available for each snapshot. This
distinction is the main reason why I’m calling these sequence numbers and
not “version”. The sequence number for a file indicates what delete files
need to be applied; files can be rewritten the new copy gets a different
sequence number as you describe just below.


>    1. Any data file strictly older than the oldest-valid-version is
>    eligible for delete collapsing. During delete collapsing, all deletions up
>    to version N (any version <= the oldest-valid-version) are applied to the
>    file. *The newly created file should be assigned a sequence number of
>    N.*
>
> First, a data file is kept until the last snapshot that references it is
expired. That enable incremental consumers without a restriction on
rewriting the file in a later replace snapshot.

Second, I agree with your description of rewriting the file. That’s what I
was trying to say above.


>    1. Any delete file that has been completely applied is eligible for
>    expiry. A delete file is completely applied if its sequence number is <=
>    all data files to which it may apply (based on partitioning data).
>
> Yes, I agree.


>    1. Any data file equal to or older than the oldest-valid-version is
>    eligible for compaction. During compaction, some or all data files up to
>    version N (any version <= the oldest-valid version) are selected. There may
>    not be any deletions applicable to any of the selected data files. The
>    selected data files are read, rewritten as desired (partitioning,
>    compaction, sorting, etc.). The new files are included in the new snapshot (*with
>    sequence number N*) while the old files are dropped.
>
> Yes, I agree.


>    1. Delete collapsing, delete expiry, and compaction may be performed
>    in a single commit. The oldest-valid-version may be advanced during this
>    process as well. The outcome must logically be consistent with applying
>    these steps independently.
>
> Because snapshots (table versions) are independent, they can always be
expired. Incremental consumers must process a new snapshot before it is old
enough to expire. The time period after which snapshots are expired is up
to the process running ExpireSnapshots.

Important to note in the above is that, during collapsing/compaction, new
> files may be emitted with a version number N that are older than the most
> recent version. This is important to ensure that all deltas newer than N
> are appropriately applied, and that incremental readers are able to
> continue processing the dataset.
>
Because incremental consumers operate using snapshots and not sequence
numbers, I think this is decoupled in the approach I’m proposing.

It is also compatible with file-level deletes used to age off old data
>> (e.g. delete where day(ts) < '2019-01-01').
>
>
> This particular operation is not compatible with incremental consumption.
>
I disagree. This just encodes deletes for an entire file worth of rows in
the manifest file instead of in a delete file. Current metadata tracks when
files are deleted and we would also associate a sequence number with those
changes, if we needed to. Incremental consumers will work like they do
today: they will consume snapshots that contain changes
(append/delete/overwrite) and will get the files that were changed in that
snapshot.

Also, being able to cleanly age off data is a hard requirement for Iceberg.
We all need to comply with data policies with age limits and we need to
ensure that we can cleanly apply those changes.

One must still generate a deletion that is associated with a new sequence
> number. I could see a reasonable workaround where, in order to delete an
> entire file (insertion file `x.dat`, sequence number X) one would reference
> the same file a second time (deletion file `x.dat, sequence number Y > X).
> Since an insertion file and a deletion file have compatible formats, there
> is no need to rewrite this file simply to mark each row in it as deleted. A
> savvy consumer would be able to tell that this is a whole-file deletion
> while a naïve consumer would be able to apply them using the basic
> algorithm.
>
With file-level deletes, the files are no longer processed by readers,
unless an incremental reader needs to read all of the deletes from the file.

If it seems like we are roughly on the same page I will take a stab at
> updating that document to go to the same level of detail that it does now
> but use the sequence-number approach.
>
Sure, that sounds good.
-- 
Ryan Blue
Software Engineer
Netflix

Re: Updates/Deletes/Upserts in Iceberg

[Erik Wright <er...@shopify.com.INVALID>]

On Mon, Jun 17, 2019 at 5:51 PM Erik Wright <er...@shopify.com> wrote:

> That's a really insightful summary of the different proposals that have
> been made. My reading of Ryan's suggestion pointed to ways that both
> approaches can be supported. The precise mechanisms available
> for representing deltas will certainly differ depending on use-case, but it
> would seem ideal (and, I believe, quite feasible) to unify those into a
> single model.
>
> On Mon, Jun 17, 2019 at 4:46 PM Anton Okolnychyi <ao...@apple.com>
> wrote:
>
>> I am really glad to see progress on this! I was off for a couple of days,
>> so could take a detailed look at Erik’s proposal just recently.
>>
>> I would highlight the following features of Erik’s proposal:
>> *- *Performing certain natural key operations without reading existing
>> data.
>> *- *Cheap retries if an update/delete modifies rows in files that are
>> being compacted (it is about compaction only as two concurrent
>> updates/deletes/inserts will require a row-based conflict resolution
>> mechanism, so it is the same as in our proposal).
>>
>> These two features could not be achieved in our approaches since we
>> relied on ‘fileId’ to scope deletes. The cost on read in Erik’s approach
>> seems to be higher as we basically sort the dataset (or at least a part we
>> query) not only by the natural key but also by version + delete flag and
>> then iterate through those records. It is a bit hard to predict how that
>> will behave from the performance perspective.
>>
>
> Performance is best evaluated in real life, but I am confident that the
> "optimized" algorithm I described, with proper partitioning and sorting,
> offers read complexity of O(N), where N is the total number of insertion
> and deletion rows (which will exceed the number of rows in an effective
> dataset whenever there are deletions). I also believe that it permits
> parallelization of reads without shuffles. As to what we can get for free
> from Spark / Presto vs. what Iceberg will need to implement, I don't know
> enough about the internals of these things to guess at that.
>

I should clarify, that O(N) is the baseline, when one actually needs to
scan an entire effective dataset. Partitioning and sorting preserve their
utility, in that any given datafile (or deletion file) can be pruned based
on query predicates. Looking up the effective value of an individual row or
range of rows when query predicates correspond to sort columns is also fast
as expected.

That being said, the file ID approach seems evidently faster to read.
>
>
>> Maybe, having more details will clarify this for me (e.g. whether we can
>> avoid shuffles, what is done by query engines/Iceberg, etc). In our
>> approaches, we were reading datasets during updates/deletes and then
>> generating diffs that would point to specific rows in specific files, so
>> the read path was cheap and straightforward but we had to read the data
>> while generating diffs and we could face unresolvable conflicts if a
>> compaction was performed concurrently to updates/deletes. Potentially, the
>> latter can be circumvented. Also, the cost to read data on write might not
>> be that high because the data is sorted by natural keys and we can leverage
>> file stats. If we are to update 1 natural key, we will need to read 1 data
>> file or even a part of it if we are dealing with columnar data.
>>
>
> That assumes that data is partitioned by natural key. If it is not, you
> will likely need to look at many data files. In my experiences, when a
> dataset is materialized for analytical purposes, natural key partitioning
> is rarely useful for reading.
>
> Overall, Erik seems to focus on use cases when we already know a set of
>> natural keys to update/delete instead of supporting UPDATE/DELETE/MERGE
>> INTO statements in a broader sense, where reading data before generating
>> diffs is inevitable in certain scenarios. Both use cases are perfectly
>> valid but they are substantially different.
>>
>
> That's all true. One thing I'm curious about is whether there is a
> specific need for SQL DML or whether this is simply the way that you chose
> to approach the problem initially. Naturally, not knowing your use cases it
> is hard to say whether they could be reconsidered in a natural-key centric
> world.
>
>
>> I think we can consider changes to the proposed merge-on-read logic to
>> cover more use cases with natural keys. For example, it seems impossible to
>> perform an upsert by natural key without reading data in the currently
>> proposed logic. One simple way to support this is to always issue a delete
>> before an insert but this will result in an unresolved delete error right
>> now. Alternatively, we can always pick the last version of each row so that
>> every insert is basically an upsert. In theory, we can even have
>> general-purpose updates/deletes using that design. However, UPDATE/DELETE/MERGE
>> INTO statements might require reading data to generate diffs, killing
>> the main benefit and potentially making the overall solution inefficient as
>> we would need to do a lot during write and read. As it stands today, this
>> approach seems relevant only if we can perform all operations without
>> reading data.
>>
>
> Assuming that data is partitioned by columns that can change over time
> (for a given natural key) it is essential to express upserts as a
> combination of a delete followed by an insertion. The reason is that one
> wishes to be able to prune and parallelize the reading of data files. The
> upsert may end up being in a different partition – either pruned, or
> handled by a different worker – than the row that it replaces. Hence the
> need to have the deletion. And, ideally, all of the (pre-upsert/pre-delete
> valued) columns that are used to partition the base dataset are also
> included in the deletion files (and used to partition the deletion files).
>
> Also, Ryan’s suggestion with sequence numbers looks promising to me as it
>> allows us to achieve the same features with minimum changes to the existing
>> metadata.
>>
>
> Agreed.
>
> To sum up, it would be great to support both use cases but I am not sure
>> it can be done with one approach. So, supporting multiple ways and
>> letting users decide which one they need can be a good way out.
>>
>
> In Ryan's latest message he suggested allowing both file-and-row-level
> deletions (synthetic key) _and_ natural-key-based deletions. Some users
> would only produce one or the other, but those based on natural-key could
> (for example) be periodically optimized to synthetic-key deletions with the
> same effect. By using sequence numbers to represent the series of changes
> that have been applied to the effective dataset, a consumer can read the
> entire dataset as of any version or incrementally observe changes to the
> dataset from version to version. And the same API can be used to consume
> the dataset regardless of which method was used to produce it.
>
> Thanks,
>> Anton
>>
>> On 17 Jun 2019, at 18:53, Erik Wright <er...@shopify.com> wrote:
>>
>> Thanks so much for this in-depth response.
>>
>> Analytical and incremental reads are the two use cases that I had
>> primarily been considering. I don't think synthetic keys are necessary for
>> those scenarios, although perhaps they offer a level of performance
>> optimization.
>>
>> The MERGE INTO scenarios are quite different. I agree with the premise
>> that they already require reading data and, therefore, the resulting output
>> should capture as much value from that effort as possible.
>>
>> Because snapshots and versions are basically the same idea, we don’t need
>>> both. If we were to add versions, they should replace snapshots.
>>>
>>
>> I'm a little confused by this. You mention sequence numbers quite a bit,
>> but then say that we should not introduce versions. From my understanding,
>> sequence numbers and versions are essentially identical. What has changed
>> is where they are encoded (in file metadata vs. in snapshot metadata). It
>> would also seem necessary to keep them distinct from snapshots in order to
>> be able to collapse/compact files older than N (oldest valid version)
>> without losing the ability to incrementally apply the changes from N + 1.
>>
>> Because I am having a really hard time seeing how things would work
>> without these sequence numbers (versions), and given their prominence in
>> your reply, my remaining comments assume they are present, distinct from
>> snapshots, and monotonically increasing.
>>
>> Another way of thinking about it: if versions are stored as data/delete
>>> file metadata and not in the table metadata file, then the complete history
>>> can be kept.
>>
>>
>> Sure. Although version expiry would still be an important process to
>> improve read performance and reduce storage overhead.
>>
>> Based on your proposal, it would seem that we can basically collapse any
>> given data file with its corresponding deletions. It's important to
>> consider how this will affect incremental readers. The following make sense
>> to me:
>>
>>    1. We should track an oldest-valid-version in the table metadata. No
>>    changes newer than this version should be collapsed, as a reader who has
>>    consumed up to that version must be able to continue reading.
>>    2. Any data file strictly older than the oldest-valid-version is
>>    eligible for delete collapsing. During delete collapsing, all deletions up
>>    to version N (any version <= the oldest-valid-version) are applied to the
>>    file. *The newly created file should be assigned a sequence number of
>>    N.*
>>    3. Any delete file that has been completely applied is eligible for
>>    expiry. A delete file is completely applied if its sequence number is <=
>>    all data files to which it may apply (based on partitioning data).
>>    4. Any data file equal to or older than the oldest-valid-version is
>>    eligible for compaction. During compaction, some or all data files up to
>>    version N (any version <= the oldest-valid version) are selected. There may
>>    not be any deletions applicable to any of the selected data files. The
>>    selected data files are read, rewritten as desired (partitioning,
>>    compaction, sorting, etc.). The new files are included in the new snapshot (*with
>>    sequence number N*) while the old files are dropped.
>>    5. Delete collapsing, delete expiry, and compaction may be performed
>>    in a single commit. The oldest-valid-version may be advanced during this
>>    process as well. The outcome must logically be consistent with applying
>>    these steps independently.
>>
>> Important to note in the above is that, during collapsing/compaction, new
>> files may be emitted with a version number N that are older than the most
>> recent version. This is important to ensure that all deltas newer than N
>> are appropriately applied, and that incremental readers are able to
>> continue processing the dataset.
>>
>> It is also compatible with file-level deletes used to age off old data
>>> (e.g. delete where day(ts) < '2019-01-01').
>>
>>
>> This particular operation is not compatible with incremental consumption.
>> One must still generate a deletion that is associated with a new sequence
>> number. I could see a reasonable workaround where, in order to delete an
>> entire file (insertion file `x.dat`, sequence number X) one would reference
>> the same file a second time (deletion file `x.dat, sequence number Y > X).
>> Since an insertion file and a deletion file have compatible formats, there
>> is no need to rewrite this file simply to mark each row in it as deleted. A
>> savvy consumer would be able to tell that this is a whole-file deletion
>> while a naïve consumer would be able to apply them using the basic
>> algorithm.
>>
>> If it seems like we are roughly on the same page I will take a stab at
>> updating that document to go to the same level of detail that it does now
>> but use the sequence-number approach.
>>
>> On Fri, Jun 14, 2019 at 6:30 PM Ryan Blue <rb...@netflix.com> wrote:
>>
>>> First, I want to clear up some of the confusion with a bit of background
>>> on how Iceberg works, but I have inline replies below as well.
>>>
>>> Conceptually, a snapshot is a table version. Each snapshot is the
>>> complete table state at some point in time and each snapshot is
>>> independent. Independence is a key property of snapshots because it allows
>>> us to maintain table data and metadata.
>>>
>>> Snapshots track additions and deletions using per-file status (ADDED/
>>> EXISTING/DELETED) and the snapshot ID when each file was added or
>>> deleted. This can be efficiently loaded and used to process changes
>>> incrementally. Snapshots are also annotated with the operation that created
>>> each snapshot. This is used to identify snapshots with no logical changes
>>> (replace) and snapshots with no deletions (append) to speed up incremental
>>> consumption and operations like snapshot expiration.
>>>
>>> Readers use the current snapshot for isolation from writes. Because
>>> readers may be using an old snapshot, Iceberg keeps snapshots around and
>>> only physically deletes files when snapshots expire — data files referenced
>>> in a valid snapshot cannot be deleted. Independence is important so that
>>> snapshots can expire and get cleaned up without any other actions on the
>>> table data.
>>>
>>> Consider the alternative: if each snapshot depended on the previous one,
>>> then Iceberg cannot expire them and physically delete data. Here’s an
>>> example snapshot history:
>>>
>>> S1
>>> - add file_a.parquet
>>> - add file_b.parquet
>>> S2
>>> - delete file_a.parquet
>>> S3
>>> - add file_c.parquet
>>>
>>> To delete file_a.parquet, S1 and S2 need to be rewritten into S2’ that
>>> contains only file_b.parquet. This would work, but takes more time than
>>> dropping references and deleting files that are no longer used. An
>>> expensive metadata rewrite to delete data isn’t ideal, and not taking the
>>> time to rewrite makes metadata stack up in the table metadata file, that
>>> gets rewritten often.
>>>
>>> Finally, Iceberg relies on regular metadata maintenance — manifest
>>> compaction — to reduce both write volume and the number of file reads
>>> needed to plan a scan. Snapshots also reuse metadata files from previous
>>> snapshots to reduce write volume.
>>>
>>> On Wed, Jun 12, 2019 at 9:12 AM Erik Wright erik.wright@shopify.com
>>> <http://mailto:erik.wright@shopify.com/> wrote:
>>>
>>>
>>>
>>>>>    - Maintaining separate manifest lists for each version requires
>>>>>    keeping versions in separate metadata trees, so we can’t merge manifest
>>>>>    files together across versions. We initially had this problem when we used
>>>>>    only fast appends for our streaming writes. We ended up with way too many
>>>>>    metadata files and job planning took a really long time.
>>>>>
>>>>> If I understand correctly, the main concern here is for scenarios
>>>> where versions are essentially micro-batches. They accumulate rapidly, each
>>>> one is relatively small, and there may be many valid versions at a time
>>>> even if expiry happens regularly. The negative impact is many small
>>>> versions cannot be combined together. Even the manifest list files are
>>>> many, and all of them need to be read.
>>>>
>>>
>>>
>>> I’m not sure how you define micro-batch, but this happens when commits
>>> are on the order of every 10 minutes and consist of large data files.
>>> Manifest compaction is required to avoid thousands of small metadata files
>>> impacting performance.
>>>
>>>
>>>
>>>>>    - Although snapshots are independent and can be aged off, this
>>>>>    requires rewriting the base version to expire versions. That means we can’t
>>>>>    delete old data files until versions are rewritten, so the problem affects
>>>>>    versions instead of snapshots. We would still have the problem of not being
>>>>>    able to delete old data until after a compaction.
>>>>>
>>>>> I don't quite understand this point, so forgive me if the following is
>>>> irrelevant or obvious.
>>>>
>>>> You are correct that snapshots become less useful as a tool, compared
>>>> to versions, for expiring data. The process now goes through (1) expire a
>>>> version and then (2) invalidate the last snapshot containing that version.
>>>> Of course, fundamentally you need to keep version X around until you are
>>>> willing to compact delta Y (or more) into it. And this is (I believe) our
>>>> common purpose in this discussion: permit a trade-off between write
>>>> amplification and storage/read costs.
>>>>
>>>
>>>
>>> Hopefully the background above is useful. Requiring expensive data-file
>>> compaction in order to compact version metadata or delete files is a big
>>> problem.
>>>
>>> And you’re right that snapshots are less useful. Because snapshots and
>>> versions are basically the same idea, we don’t need both. If we were to add
>>> versions, they should replace snapshots. But I don’t think it is a good
>>> idea to give up the independence.
>>>
>>> The good news is that we can use snapshots exactly like versions and
>>> avoid these problems by adding sequence numbers.
>>>
>>>
>>> The value of decoupling versions from snapshots is:
>>>>
>>>> 1. It makes the process of correctly applying multiple deltas easy to
>>>> reason about.
>>>> 2. It allows us to issue updates that don't introduce new versions
>>>> (i.e., those that merely expire old versions, or that perform other
>>>> optimizations).
>>>> 3. It enables clients to cleanly consume new deltas when those clients
>>>> are sophisticated enough to incrementally update their own outputs in
>>>> function of those changes.
>>>>
>>>
>>>
>>> Points 2 and 3 are available already, and 1 can be done with sequence
>>> numbers in snapshot metadata.
>>>
>>>
>>> Here are some benefits to this approach:
>>>>>
>>>>>    - There is no explicit version that must be expired so compaction
>>>>>    is only required to optimize reads — partition or file-level delete can
>>>>>    coexist with deltas
>>>>>
>>>>> Can you expand on this?
>>>>
>>>
>>>
>>> By adding a sequence number instead of versions, the essential version
>>> information is embedded in the existing metadata files. Those files are
>>> reused across snapshots so there is no need to compact deltas into data
>>> files to reduce the write volume.
>>>
>>> Another way of thinking about it: if versions are stored as data/delete
>>> file metadata and not in the table metadata file, then the complete history
>>> can be kept.
>>>
>>> It is also compatible with file-level deletes used to age off old data
>>> (e.g. delete where day(ts) < '2019-01-01'). When files match a delete,
>>> they are removed from metadata and physically deleted when the delete
>>> snapshot is expired. This is independent of the delete files that get
>>> applied on read. In fact, if a delete file’s sequence number is less than
>>> all data files in a partition, the delete itself can be removed and never
>>> compacted.
>>>
>>>
>>>
>>>>>    - When a delete’s sequence number is lower than all data files, it
>>>>>    can be pruned
>>>>>
>>>>> In combination with the previous comment, I have the impression that
>>>> we might be losing the distinction between snapshots and versions. In
>>>> particular, what I'm concerned about is that, from a consumer's point of
>>>> view, it may be difficult to figure out, from one snapshot to the next,
>>>> what new deltas I need to process to update my view.
>>>>
>>>
>>>
>>> We don’t need both and should go with either snapshots (independent) or
>>> versions (dependent). I don’t think it is possible to make dependent
>>> versions work.
>>>
>>> But as I noted above, the changes that occurred in a snapshot are
>>> available as data file additions and data file deletions. Delete file
>>> additions and deletions would be tracked the same way. Consumers would have
>>> full access to the changes.
>>>
>>>
>>>
>>>>>    - Compaction can translate a natural key delete to a synthetic key
>>>>>    delete by adding a synthetic key delete file and updating the data file’s
>>>>>    sequence number
>>>>>
>>>>> I don't understand what this means.
>>>>
>>>
>>>
>>> Synthetic keys are better than natural keys in several ways. They are
>>> cheaper to apply at read time, they are scoped to a single data file, they
>>> compress well, etc. If both are supported, compaction could rewrite a
>>> natural key delete as a synthetic key delete to avoid rewriting the entire
>>> data file to delete just a few rows.
>>>
>>> In addition, if a natural key delete is applied to an entire table, it
>>> would be possible to change sequence numbers to apply this process
>>> incrementally. But, this would change history just like rewriting the base
>>> version of a table loses the changes in that version. So we would need to
>>> be careful about when changing a sequence number is allowed.
>>>
>>>
>>>
>>>>>    - Synthetic and natural key approaches can coexist
>>>>>
>>>>> Can you elaborate on what the synthetic key approach would imply, and
>>>> what use-cases it would serve?
>>>>
>>>
>>>
>>> Some use cases, like MERGE INTO, already require reading data so we can
>>> write synthetic key delete files without additional work. That means
>>> Iceberg should support not just natural key deletes, but also synthetic.
>>>
>>>
>>> I agree we are getting closer. I am confused by some things, and I
>>>> suspect that the difference lies in the use-cases I wish to support for
>>>> incremental consumers of these datasets. I have the impression that most of
>>>> you folks are primarily considering incremental writes that end up serving
>>>> analytical use cases (that want to see the full effective dataset as of the
>>>> most recent snapshot).
>>>>
>>>
>>>
>>> We have both use cases and I want to make sure Iceberg can be used for
>>> both.
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>>
>>
>>

Re: Updates/Deletes/Upserts in Iceberg

[Erik Wright <er...@shopify.com.INVALID>]

That's a really insightful summary of the different proposals that have
been made. My reading of Ryan's suggestion pointed to ways that both
approaches can be supported. The precise mechanisms available
for representing deltas will certainly differ depending on use-case, but it
would seem ideal (and, I believe, quite feasible) to unify those into a
single model.

On Mon, Jun 17, 2019 at 4:46 PM Anton Okolnychyi <ao...@apple.com>
wrote:

> I am really glad to see progress on this! I was off for a couple of days,
> so could take a detailed look at Erik’s proposal just recently.
>
> I would highlight the following features of Erik’s proposal:
> *- *Performing certain natural key operations without reading existing
> data.
> *- *Cheap retries if an update/delete modifies rows in files that are
> being compacted (it is about compaction only as two concurrent
> updates/deletes/inserts will require a row-based conflict resolution
> mechanism, so it is the same as in our proposal).
>
> These two features could not be achieved in our approaches since we relied
> on ‘fileId’ to scope deletes. The cost on read in Erik’s approach seems to
> be higher as we basically sort the dataset (or at least a part we query)
> not only by the natural key but also by version + delete flag and then
> iterate through those records. It is a bit hard to predict how that will
> behave from the performance perspective.
>

Performance is best evaluated in real life, but I am confident that the
"optimized" algorithm I described, with proper partitioning and sorting,
offers read complexity of O(N), where N is the total number of insertion
and deletion rows (which will exceed the number of rows in an effective
dataset whenever there are deletions). I also believe that it permits
parallelization of reads without shuffles. As to what we can get for free
from Spark / Presto vs. what Iceberg will need to implement, I don't know
enough about the internals of these things to guess at that.

That being said, the file ID approach seems evidently faster to read.


> Maybe, having more details will clarify this for me (e.g. whether we can
> avoid shuffles, what is done by query engines/Iceberg, etc). In our
> approaches, we were reading datasets during updates/deletes and then
> generating diffs that would point to specific rows in specific files, so
> the read path was cheap and straightforward but we had to read the data
> while generating diffs and we could face unresolvable conflicts if a
> compaction was performed concurrently to updates/deletes. Potentially, the
> latter can be circumvented. Also, the cost to read data on write might not
> be that high because the data is sorted by natural keys and we can leverage
> file stats. If we are to update 1 natural key, we will need to read 1 data
> file or even a part of it if we are dealing with columnar data.
>

That assumes that data is partitioned by natural key. If it is not, you
will likely need to look at many data files. In my experiences, when a
dataset is materialized for analytical purposes, natural key partitioning
is rarely useful for reading.

Overall, Erik seems to focus on use cases when we already know a set of
> natural keys to update/delete instead of supporting UPDATE/DELETE/MERGE
> INTO statements in a broader sense, where reading data before generating
> diffs is inevitable in certain scenarios. Both use cases are perfectly
> valid but they are substantially different.
>

That's all true. One thing I'm curious about is whether there is a specific
need for SQL DML or whether this is simply the way that you chose to
approach the problem initially. Naturally, not knowing your use cases it is
hard to say whether they could be reconsidered in a natural-key centric
world.


> I think we can consider changes to the proposed merge-on-read logic to
> cover more use cases with natural keys. For example, it seems impossible to
> perform an upsert by natural key without reading data in the currently
> proposed logic. One simple way to support this is to always issue a delete
> before an insert but this will result in an unresolved delete error right
> now. Alternatively, we can always pick the last version of each row so that
> every insert is basically an upsert. In theory, we can even have
> general-purpose updates/deletes using that design. However, UPDATE/DELETE/MERGE
> INTO statements might require reading data to generate diffs, killing the
> main benefit and potentially making the overall solution inefficient as we
> would need to do a lot during write and read. As it stands today, this
> approach seems relevant only if we can perform all operations without
> reading data.
>

Assuming that data is partitioned by columns that can change over time (for
a given natural key) it is essential to express upserts as a combination of
a delete followed by an insertion. The reason is that one wishes to be able
to prune and parallelize the reading of data files. The upsert may end up
being in a different partition – either pruned, or handled by a different
worker – than the row that it replaces. Hence the need to have the
deletion. And, ideally, all of the (pre-upsert/pre-delete valued) columns
that are used to partition the base dataset are also included in the
deletion files (and used to partition the deletion files).

Also, Ryan’s suggestion with sequence numbers looks promising to me as it
> allows us to achieve the same features with minimum changes to the existing
> metadata.
>

Agreed.

To sum up, it would be great to support both use cases but I am not sure it
> can be done with one approach. So, supporting multiple ways and letting
> users decide which one they need can be a good way out.
>

In Ryan's latest message he suggested allowing both file-and-row-level
deletions (synthetic key) _and_ natural-key-based deletions. Some users
would only produce one or the other, but those based on natural-key could
(for example) be periodically optimized to synthetic-key deletions with the
same effect. By using sequence numbers to represent the series of changes
that have been applied to the effective dataset, a consumer can read the
entire dataset as of any version or incrementally observe changes to the
dataset from version to version. And the same API can be used to consume
the dataset regardless of which method was used to produce it.

Thanks,
> Anton
>
> On 17 Jun 2019, at 18:53, Erik Wright <er...@shopify.com> wrote:
>
> Thanks so much for this in-depth response.
>
> Analytical and incremental reads are the two use cases that I had
> primarily been considering. I don't think synthetic keys are necessary for
> those scenarios, although perhaps they offer a level of performance
> optimization.
>
> The MERGE INTO scenarios are quite different. I agree with the premise
> that they already require reading data and, therefore, the resulting output
> should capture as much value from that effort as possible.
>
> Because snapshots and versions are basically the same idea, we don’t need
>> both. If we were to add versions, they should replace snapshots.
>>
>
> I'm a little confused by this. You mention sequence numbers quite a bit,
> but then say that we should not introduce versions. From my understanding,
> sequence numbers and versions are essentially identical. What has changed
> is where they are encoded (in file metadata vs. in snapshot metadata). It
> would also seem necessary to keep them distinct from snapshots in order to
> be able to collapse/compact files older than N (oldest valid version)
> without losing the ability to incrementally apply the changes from N + 1.
>
> Because I am having a really hard time seeing how things would work
> without these sequence numbers (versions), and given their prominence in
> your reply, my remaining comments assume they are present, distinct from
> snapshots, and monotonically increasing.
>
> Another way of thinking about it: if versions are stored as data/delete
>> file metadata and not in the table metadata file, then the complete history
>> can be kept.
>
>
> Sure. Although version expiry would still be an important process to
> improve read performance and reduce storage overhead.
>
> Based on your proposal, it would seem that we can basically collapse any
> given data file with its corresponding deletions. It's important to
> consider how this will affect incremental readers. The following make sense
> to me:
>
>    1. We should track an oldest-valid-version in the table metadata. No
>    changes newer than this version should be collapsed, as a reader who has
>    consumed up to that version must be able to continue reading.
>    2. Any data file strictly older than the oldest-valid-version is
>    eligible for delete collapsing. During delete collapsing, all deletions up
>    to version N (any version <= the oldest-valid-version) are applied to the
>    file. *The newly created file should be assigned a sequence number of
>    N.*
>    3. Any delete file that has been completely applied is eligible for
>    expiry. A delete file is completely applied if its sequence number is <=
>    all data files to which it may apply (based on partitioning data).
>    4. Any data file equal to or older than the oldest-valid-version is
>    eligible for compaction. During compaction, some or all data files up to
>    version N (any version <= the oldest-valid version) are selected. There may
>    not be any deletions applicable to any of the selected data files. The
>    selected data files are read, rewritten as desired (partitioning,
>    compaction, sorting, etc.). The new files are included in the new snapshot (*with
>    sequence number N*) while the old files are dropped.
>    5. Delete collapsing, delete expiry, and compaction may be performed
>    in a single commit. The oldest-valid-version may be advanced during this
>    process as well. The outcome must logically be consistent with applying
>    these steps independently.
>
> Important to note in the above is that, during collapsing/compaction, new
> files may be emitted with a version number N that are older than the most
> recent version. This is important to ensure that all deltas newer than N
> are appropriately applied, and that incremental readers are able to
> continue processing the dataset.
>
> It is also compatible with file-level deletes used to age off old data
>> (e.g. delete where day(ts) < '2019-01-01').
>
>
> This particular operation is not compatible with incremental consumption.
> One must still generate a deletion that is associated with a new sequence
> number. I could see a reasonable workaround where, in order to delete an
> entire file (insertion file `x.dat`, sequence number X) one would reference
> the same file a second time (deletion file `x.dat, sequence number Y > X).
> Since an insertion file and a deletion file have compatible formats, there
> is no need to rewrite this file simply to mark each row in it as deleted. A
> savvy consumer would be able to tell that this is a whole-file deletion
> while a naïve consumer would be able to apply them using the basic
> algorithm.
>
> If it seems like we are roughly on the same page I will take a stab at
> updating that document to go to the same level of detail that it does now
> but use the sequence-number approach.
>
> On Fri, Jun 14, 2019 at 6:30 PM Ryan Blue <rb...@netflix.com> wrote:
>
>> First, I want to clear up some of the confusion with a bit of background
>> on how Iceberg works, but I have inline replies below as well.
>>
>> Conceptually, a snapshot is a table version. Each snapshot is the
>> complete table state at some point in time and each snapshot is
>> independent. Independence is a key property of snapshots because it allows
>> us to maintain table data and metadata.
>>
>> Snapshots track additions and deletions using per-file status (ADDED/
>> EXISTING/DELETED) and the snapshot ID when each file was added or
>> deleted. This can be efficiently loaded and used to process changes
>> incrementally. Snapshots are also annotated with the operation that created
>> each snapshot. This is used to identify snapshots with no logical changes
>> (replace) and snapshots with no deletions (append) to speed up incremental
>> consumption and operations like snapshot expiration.
>>
>> Readers use the current snapshot for isolation from writes. Because
>> readers may be using an old snapshot, Iceberg keeps snapshots around and
>> only physically deletes files when snapshots expire — data files referenced
>> in a valid snapshot cannot be deleted. Independence is important so that
>> snapshots can expire and get cleaned up without any other actions on the
>> table data.
>>
>> Consider the alternative: if each snapshot depended on the previous one,
>> then Iceberg cannot expire them and physically delete data. Here’s an
>> example snapshot history:
>>
>> S1
>> - add file_a.parquet
>> - add file_b.parquet
>> S2
>> - delete file_a.parquet
>> S3
>> - add file_c.parquet
>>
>> To delete file_a.parquet, S1 and S2 need to be rewritten into S2’ that
>> contains only file_b.parquet. This would work, but takes more time than
>> dropping references and deleting files that are no longer used. An
>> expensive metadata rewrite to delete data isn’t ideal, and not taking the
>> time to rewrite makes metadata stack up in the table metadata file, that
>> gets rewritten often.
>>
>> Finally, Iceberg relies on regular metadata maintenance — manifest
>> compaction — to reduce both write volume and the number of file reads
>> needed to plan a scan. Snapshots also reuse metadata files from previous
>> snapshots to reduce write volume.
>>
>> On Wed, Jun 12, 2019 at 9:12 AM Erik Wright erik.wright@shopify.com
>> <http://mailto:erik.wright@shopify.com/> wrote:
>>
>>
>>
>>>>    - Maintaining separate manifest lists for each version requires
>>>>    keeping versions in separate metadata trees, so we can’t merge manifest
>>>>    files together across versions. We initially had this problem when we used
>>>>    only fast appends for our streaming writes. We ended up with way too many
>>>>    metadata files and job planning took a really long time.
>>>>
>>>> If I understand correctly, the main concern here is for scenarios where
>>> versions are essentially micro-batches. They accumulate rapidly, each one
>>> is relatively small, and there may be many valid versions at a time even if
>>> expiry happens regularly. The negative impact is many small versions cannot
>>> be combined together. Even the manifest list files are many, and all of
>>> them need to be read.
>>>
>>
>>
>> I’m not sure how you define micro-batch, but this happens when commits
>> are on the order of every 10 minutes and consist of large data files.
>> Manifest compaction is required to avoid thousands of small metadata files
>> impacting performance.
>>
>>
>>
>>>>    - Although snapshots are independent and can be aged off, this
>>>>    requires rewriting the base version to expire versions. That means we can’t
>>>>    delete old data files until versions are rewritten, so the problem affects
>>>>    versions instead of snapshots. We would still have the problem of not being
>>>>    able to delete old data until after a compaction.
>>>>
>>>> I don't quite understand this point, so forgive me if the following is
>>> irrelevant or obvious.
>>>
>>> You are correct that snapshots become less useful as a tool, compared to
>>> versions, for expiring data. The process now goes through (1) expire a
>>> version and then (2) invalidate the last snapshot containing that version.
>>> Of course, fundamentally you need to keep version X around until you are
>>> willing to compact delta Y (or more) into it. And this is (I believe) our
>>> common purpose in this discussion: permit a trade-off between write
>>> amplification and storage/read costs.
>>>
>>
>>
>> Hopefully the background above is useful. Requiring expensive data-file
>> compaction in order to compact version metadata or delete files is a big
>> problem.
>>
>> And you’re right that snapshots are less useful. Because snapshots and
>> versions are basically the same idea, we don’t need both. If we were to add
>> versions, they should replace snapshots. But I don’t think it is a good
>> idea to give up the independence.
>>
>> The good news is that we can use snapshots exactly like versions and
>> avoid these problems by adding sequence numbers.
>>
>>
>> The value of decoupling versions from snapshots is:
>>>
>>> 1. It makes the process of correctly applying multiple deltas easy to
>>> reason about.
>>> 2. It allows us to issue updates that don't introduce new versions
>>> (i.e., those that merely expire old versions, or that perform other
>>> optimizations).
>>> 3. It enables clients to cleanly consume new deltas when those clients
>>> are sophisticated enough to incrementally update their own outputs in
>>> function of those changes.
>>>
>>
>>
>> Points 2 and 3 are available already, and 1 can be done with sequence
>> numbers in snapshot metadata.
>>
>>
>> Here are some benefits to this approach:
>>>>
>>>>    - There is no explicit version that must be expired so compaction
>>>>    is only required to optimize reads — partition or file-level delete can
>>>>    coexist with deltas
>>>>
>>>> Can you expand on this?
>>>
>>
>>
>> By adding a sequence number instead of versions, the essential version
>> information is embedded in the existing metadata files. Those files are
>> reused across snapshots so there is no need to compact deltas into data
>> files to reduce the write volume.
>>
>> Another way of thinking about it: if versions are stored as data/delete
>> file metadata and not in the table metadata file, then the complete history
>> can be kept.
>>
>> It is also compatible with file-level deletes used to age off old data
>> (e.g. delete where day(ts) < '2019-01-01'). When files match a delete,
>> they are removed from metadata and physically deleted when the delete
>> snapshot is expired. This is independent of the delete files that get
>> applied on read. In fact, if a delete file’s sequence number is less than
>> all data files in a partition, the delete itself can be removed and never
>> compacted.
>>
>>
>>
>>>>    - When a delete’s sequence number is lower than all data files, it
>>>>    can be pruned
>>>>
>>>> In combination with the previous comment, I have the impression that we
>>> might be losing the distinction between snapshots and versions. In
>>> particular, what I'm concerned about is that, from a consumer's point of
>>> view, it may be difficult to figure out, from one snapshot to the next,
>>> what new deltas I need to process to update my view.
>>>
>>
>>
>> We don’t need both and should go with either snapshots (independent) or
>> versions (dependent). I don’t think it is possible to make dependent
>> versions work.
>>
>> But as I noted above, the changes that occurred in a snapshot are
>> available as data file additions and data file deletions. Delete file
>> additions and deletions would be tracked the same way. Consumers would have
>> full access to the changes.
>>
>>
>>
>>>>    - Compaction can translate a natural key delete to a synthetic key
>>>>    delete by adding a synthetic key delete file and updating the data file’s
>>>>    sequence number
>>>>
>>>> I don't understand what this means.
>>>
>>
>>
>> Synthetic keys are better than natural keys in several ways. They are
>> cheaper to apply at read time, they are scoped to a single data file, they
>> compress well, etc. If both are supported, compaction could rewrite a
>> natural key delete as a synthetic key delete to avoid rewriting the entire
>> data file to delete just a few rows.
>>
>> In addition, if a natural key delete is applied to an entire table, it
>> would be possible to change sequence numbers to apply this process
>> incrementally. But, this would change history just like rewriting the base
>> version of a table loses the changes in that version. So we would need to
>> be careful about when changing a sequence number is allowed.
>>
>>
>>
>>>>    - Synthetic and natural key approaches can coexist
>>>>
>>>> Can you elaborate on what the synthetic key approach would imply, and
>>> what use-cases it would serve?
>>>
>>
>>
>> Some use cases, like MERGE INTO, already require reading data so we can
>> write synthetic key delete files without additional work. That means
>> Iceberg should support not just natural key deletes, but also synthetic.
>>
>>
>> I agree we are getting closer. I am confused by some things, and I
>>> suspect that the difference lies in the use-cases I wish to support for
>>> incremental consumers of these datasets. I have the impression that most of
>>> you folks are primarily considering incremental writes that end up serving
>>> analytical use cases (that want to see the full effective dataset as of the
>>> most recent snapshot).
>>>
>>
>>
>> We have both use cases and I want to make sure Iceberg can be used for
>> both.
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>
>

Re: Updates/Deletes/Upserts in Iceberg

[Anton Okolnychyi <ao...@apple.com.INVALID>]

I am really glad to see progress on this! I was off for a couple of days, so could take a detailed look at Erik’s proposal just recently.

I would highlight the following features of Erik’s proposal:
- Performing certain natural key operations without reading existing data.
- Cheap retries if an update/delete modifies rows in files that are being compacted (it is about compaction only as two concurrent updates/deletes/inserts will require a row-based conflict resolution mechanism, so it is the same as in our proposal).

These two features could not be achieved in our approaches since we relied on ‘fileId’ to scope deletes. The cost on read in Erik’s approach seems to be higher as we basically sort the dataset (or at least a part we query) not only by the natural key but also by version + delete flag and then iterate through those records. It is a bit hard to predict how that will behave from the performance perspective. Maybe, having more details will clarify this for me (e.g. whether we can avoid shuffles, what is done by query engines/Iceberg, etc). In our approaches, we were reading datasets during updates/deletes and then generating diffs that would point to specific rows in specific files, so the read path was cheap and straightforward but we had to read the data while generating diffs and we could face unresolvable conflicts if a compaction was performed concurrently to updates/deletes. Potentially, the latter can be circumvented. Also, the cost to read data on write might not be that high because the data is sorted by natural keys and we can leverage file stats. If we are to update 1 natural key, we will need to read 1 data file or even a part of it if we are dealing with columnar data.

Overall, Erik seems to focus on use cases when we already know a set of natural keys to update/delete instead of supporting UPDATE/DELETE/MERGE INTO statements in a broader sense, where reading data before generating diffs is inevitable in certain scenarios. Both use cases are perfectly valid but they are substantially different. I think we can consider changes to the proposed merge-on-read logic to cover more use cases with natural keys. For example, it seems impossible to perform an upsert by natural key without reading data in the currently proposed logic. One simple way to support this is to always issue a delete before an insert but this will result in an unresolved delete error right now. Alternatively, we can always pick the last version of each row so that every insert is basically an upsert. In theory, we can even have general-purpose updates/deletes using that design. However, UPDATE/DELETE/MERGE INTO statements might require reading data to generate diffs, killing the main benefit and potentially making the overall solution inefficient as we would need to do a lot during write and read. As it stands today, this approach seems relevant only if we can perform all operations without reading data.

Also, Ryan’s suggestion with sequence numbers looks promising to me as it allows us to achieve the same features with minimum changes to the existing metadata.

To sum up, it would be great to support both use cases but I am not sure it can be done with one approach. So, supporting multiple ways and letting users decide which one they need can be a good way out.

Thanks,
Anton

> On 17 Jun 2019, at 18:53, Erik Wright <er...@shopify.com> wrote:
> 
> Thanks so much for this in-depth response.
> 
> Analytical and incremental reads are the two use cases that I had primarily been considering. I don't think synthetic keys are necessary for those scenarios, although perhaps they offer a level of performance optimization.
> 
> The MERGE INTO scenarios are quite different. I agree with the premise that they already require reading data and, therefore, the resulting output should capture as much value from that effort as possible.
> 
> Because snapshots and versions are basically the same idea, we don’t need both. If we were to add versions, they should replace snapshots.
> 
> I'm a little confused by this. You mention sequence numbers quite a bit, but then say that we should not introduce versions. From my understanding, sequence numbers and versions are essentially identical. What has changed is where they are encoded (in file metadata vs. in snapshot metadata). It would also seem necessary to keep them distinct from snapshots in order to be able to collapse/compact files older than N (oldest valid version) without losing the ability to incrementally apply the changes from N + 1.
> 
> Because I am having a really hard time seeing how things would work without these sequence numbers (versions), and given their prominence in your reply, my remaining comments assume they are present, distinct from snapshots, and monotonically increasing.
> 
> Another way of thinking about it: if versions are stored as data/delete file metadata and not in the table metadata file, then the complete history can be kept.
> 
> Sure. Although version expiry would still be an important process to improve read performance and reduce storage overhead.
> 
> Based on your proposal, it would seem that we can basically collapse any given data file with its corresponding deletions. It's important to consider how this will affect incremental readers. The following make sense to me:
> We should track an oldest-valid-version in the table metadata. No changes newer than this version should be collapsed, as a reader who has consumed up to that version must be able to continue reading.
> Any data file strictly older than the oldest-valid-version is eligible for delete collapsing. During delete collapsing, all deletions up to version N (any version <= the oldest-valid-version) are applied to the file. The newly created file should be assigned a sequence number of N.
> Any delete file that has been completely applied is eligible for expiry. A delete file is completely applied if its sequence number is <= all data files to which it may apply (based on partitioning data).
> Any data file equal to or older than the oldest-valid-version is eligible for compaction. During compaction, some or all data files up to version N (any version <= the oldest-valid version) are selected. There may not be any deletions applicable to any of the selected data files. The selected data files are read, rewritten as desired (partitioning, compaction, sorting, etc.). The new files are included in the new snapshot (with sequence number N) while the old files are dropped.
> Delete collapsing, delete expiry, and compaction may be performed in a single commit. The oldest-valid-version may be advanced during this process as well. The outcome must logically be consistent with applying these steps independently.
> Important to note in the above is that, during collapsing/compaction, new files may be emitted with a version number N that are older than the most recent version. This is important to ensure that all deltas newer than N are appropriately applied, and that incremental readers are able to continue processing the dataset.
> 
> It is also compatible with file-level deletes used to age off old data (e.g. delete where day(ts) < '2019-01-01').
> 
> This particular operation is not compatible with incremental consumption. One must still generate a deletion that is associated with a new sequence number. I could see a reasonable workaround where, in order to delete an entire file (insertion file `x.dat`, sequence number X) one would reference the same file a second time (deletion file `x.dat, sequence number Y > X). Since an insertion file and a deletion file have compatible formats, there is no need to rewrite this file simply to mark each row in it as deleted. A savvy consumer would be able to tell that this is a whole-file deletion while a naïve consumer would be able to apply them using the basic algorithm.
> 
> If it seems like we are roughly on the same page I will take a stab at updating that document to go to the same level of detail that it does now but use the sequence-number approach.
> 
> On Fri, Jun 14, 2019 at 6:30 PM Ryan Blue <rblue@netflix.com <ma...@netflix.com>> wrote:
> First, I want to clear up some of the confusion with a bit of background on how Iceberg works, but I have inline replies below as well.
> 
> Conceptually, a snapshot is a table version. Each snapshot is the complete table state at some point in time and each snapshot is independent. Independence is a key property of snapshots because it allows us to maintain table data and metadata.
> 
> Snapshots track additions and deletions using per-file status (ADDED/EXISTING/DELETED) and the snapshot ID when each file was added or deleted. This can be efficiently loaded and used to process changes incrementally. Snapshots are also annotated with the operation that created each snapshot. This is used to identify snapshots with no logical changes (replace) and snapshots with no deletions (append) to speed up incremental consumption and operations like snapshot expiration.
> 
> Readers use the current snapshot for isolation from writes. Because readers may be using an old snapshot, Iceberg keeps snapshots around and only physically deletes files when snapshots expire — data files referenced in a valid snapshot cannot be deleted. Independence is important so that snapshots can expire and get cleaned up without any other actions on the table data.
> 
> Consider the alternative: if each snapshot depended on the previous one, then Iceberg cannot expire them and physically delete data. Here’s an example snapshot history:
> 
> S1
> - add file_a.parquet
> - add file_b.parquet
> S2
> - delete file_a.parquet
> S3
> - add file_c.parquet
> To delete file_a.parquet, S1 and S2 need to be rewritten into S2’ that contains only file_b.parquet. This would work, but takes more time than dropping references and deleting files that are no longer used. An expensive metadata rewrite to delete data isn’t ideal, and not taking the time to rewrite makes metadata stack up in the table metadata file, that gets rewritten often.
> 
> Finally, Iceberg relies on regular metadata maintenance — manifest compaction — to reduce both write volume and the number of file reads needed to plan a scan. Snapshots also reuse metadata files from previous snapshots to reduce write volume.
> 
> On Wed, Jun 12, 2019 at 9:12 AM Erik Wright erik.wright@shopify.com <http://mailto:erik.wright@shopify.com/> wrote:
> 
> 
> 
> Maintaining separate manifest lists for each version requires keeping versions in separate metadata trees, so we can’t merge manifest files together across versions. We initially had this problem when we used only fast appends for our streaming writes. We ended up with way too many metadata files and job planning took a really long time.
> If I understand correctly, the main concern here is for scenarios where versions are essentially micro-batches. They accumulate rapidly, each one is relatively small, and there may be many valid versions at a time even if expiry happens regularly. The negative impact is many small versions cannot be combined together. Even the manifest list files are many, and all of them need to be read.
> 
> 
> I’m not sure how you define micro-batch, but this happens when commits are on the order of every 10 minutes and consist of large data files. Manifest compaction is required to avoid thousands of small metadata files impacting performance.
> 
> 
> 
> Although snapshots are independent and can be aged off, this requires rewriting the base version to expire versions. That means we can’t delete old data files until versions are rewritten, so the problem affects versions instead of snapshots. We would still have the problem of not being able to delete old data until after a compaction.
> I don't quite understand this point, so forgive me if the following is irrelevant or obvious.
> 
> You are correct that snapshots become less useful as a tool, compared to versions, for expiring data. The process now goes through (1) expire a version and then (2) invalidate the last snapshot containing that version. Of course, fundamentally you need to keep version X around until you are willing to compact delta Y (or more) into it. And this is (I believe) our common purpose in this discussion: permit a trade-off between write amplification and storage/read costs.
> 
> 
> Hopefully the background above is useful. Requiring expensive data-file compaction in order to compact version metadata or delete files is a big problem.
> 
> And you’re right that snapshots are less useful. Because snapshots and versions are basically the same idea, we don’t need both. If we were to add versions, they should replace snapshots. But I don’t think it is a good idea to give up the independence.
> 
> The good news is that we can use snapshots exactly like versions and avoid these problems by adding sequence numbers.
> 
> 
> 
> The value of decoupling versions from snapshots is:
> 
> 1. It makes the process of correctly applying multiple deltas easy to reason about.
> 2. It allows us to issue updates that don't introduce new versions (i.e., those that merely expire old versions, or that perform other optimizations).
> 3. It enables clients to cleanly consume new deltas when those clients are sophisticated enough to incrementally update their own outputs in function of those changes.
> 
> 
> Points 2 and 3 are available already, and 1 can be done with sequence numbers in snapshot metadata.
> 
> 
> 
> Here are some benefits to this approach:
> 
> There is no explicit version that must be expired so compaction is only required to optimize reads — partition or file-level delete can coexist with deltas
> Can you expand on this?
> 
> 
> By adding a sequence number instead of versions, the essential version information is embedded in the existing metadata files. Those files are reused across snapshots so there is no need to compact deltas into data files to reduce the write volume.
> 
> Another way of thinking about it: if versions are stored as data/delete file metadata and not in the table metadata file, then the complete history can be kept.
> 
> It is also compatible with file-level deletes used to age off old data (e.g. delete where day(ts) < '2019-01-01'). When files match a delete, they are removed from metadata and physically deleted when the delete snapshot is expired. This is independent of the delete files that get applied on read. In fact, if a delete file’s sequence number is less than all data files in a partition, the delete itself can be removed and never compacted.
> 
> 
> 
> When a delete’s sequence number is lower than all data files, it can be pruned
> In combination with the previous comment, I have the impression that we might be losing the distinction between snapshots and versions. In particular, what I'm concerned about is that, from a consumer's point of view, it may be difficult to figure out, from one snapshot to the next, what new deltas I need to process to update my view.
> 
> 
> We don’t need both and should go with either snapshots (independent) or versions (dependent). I don’t think it is possible to make dependent versions work.
> 
> But as I noted above, the changes that occurred in a snapshot are available as data file additions and data file deletions. Delete file additions and deletions would be tracked the same way. Consumers would have full access to the changes.
> 
> 
> 
> Compaction can translate a natural key delete to a synthetic key delete by adding a synthetic key delete file and updating the data file’s sequence number
> I don't understand what this means.
> 
> 
> Synthetic keys are better than natural keys in several ways. They are cheaper to apply at read time, they are scoped to a single data file, they compress well, etc. If both are supported, compaction could rewrite a natural key delete as a synthetic key delete to avoid rewriting the entire data file to delete just a few rows.
> 
> In addition, if a natural key delete is applied to an entire table, it would be possible to change sequence numbers to apply this process incrementally. But, this would change history just like rewriting the base version of a table loses the changes in that version. So we would need to be careful about when changing a sequence number is allowed.
> 
> 
> 
> Synthetic and natural key approaches can coexist
> Can you elaborate on what the synthetic key approach would imply, and what use-cases it would serve? 
> 
> 
> Some use cases, like MERGE INTO, already require reading data so we can write synthetic key delete files without additional work. That means Iceberg should support not just natural key deletes, but also synthetic.
> 
> 
> 
> I agree we are getting closer. I am confused by some things, and I suspect that the difference lies in the use-cases I wish to support for incremental consumers of these datasets. I have the impression that most of you folks are primarily considering incremental writes that end up serving analytical use cases (that want to see the full effective dataset as of the most recent snapshot).
> 
> 
> We have both use cases and I want to make sure Iceberg can be used for both.
> 
> -- 
> Ryan Blue
> Software Engineer
> Netflix

Re: Updates/Deletes/Upserts in Iceberg

[Erik Wright <er...@shopify.com.INVALID>]

Thanks so much for this in-depth response.

Analytical and incremental reads are the two use cases that I had primarily
been considering. I don't think synthetic keys are necessary for those
scenarios, although perhaps they offer a level of performance optimization.

The MERGE INTO scenarios are quite different. I agree with the premise that
they already require reading data and, therefore, the resulting output
should capture as much value from that effort as possible.

Because snapshots and versions are basically the same idea, we don’t need
> both. If we were to add versions, they should replace snapshots.
>

I'm a little confused by this. You mention sequence numbers quite a bit,
but then say that we should not introduce versions. From my understanding,
sequence numbers and versions are essentially identical. What has changed
is where they are encoded (in file metadata vs. in snapshot metadata). It
would also seem necessary to keep them distinct from snapshots in order to
be able to collapse/compact files older than N (oldest valid version)
without losing the ability to incrementally apply the changes from N + 1.

Because I am having a really hard time seeing how things would work without
these sequence numbers (versions), and given their prominence in your
reply, my remaining comments assume they are present, distinct from
snapshots, and monotonically increasing.

Another way of thinking about it: if versions are stored as data/delete
> file metadata and not in the table metadata file, then the complete history
> can be kept.


Sure. Although version expiry would still be an important process to
improve read performance and reduce storage overhead.

Based on your proposal, it would seem that we can basically collapse any
given data file with its corresponding deletions. It's important to
consider how this will affect incremental readers. The following make sense
to me:

   1. We should track an oldest-valid-version in the table metadata. No
   changes newer than this version should be collapsed, as a reader who has
   consumed up to that version must be able to continue reading.
   2. Any data file strictly older than the oldest-valid-version is
   eligible for delete collapsing. During delete collapsing, all deletions up
   to version N (any version <= the oldest-valid-version) are applied to the
   file. *The newly created file should be assigned a sequence number of N.*
   3. Any delete file that has been completely applied is eligible for
   expiry. A delete file is completely applied if its sequence number is <=
   all data files to which it may apply (based on partitioning data).
   4. Any data file equal to or older than the oldest-valid-version is
   eligible for compaction. During compaction, some or all data files up to
   version N (any version <= the oldest-valid version) are selected. There may
   not be any deletions applicable to any of the selected data files. The
   selected data files are read, rewritten as desired (partitioning,
   compaction, sorting, etc.). The new files are included in the new
snapshot (*with
   sequence number N*) while the old files are dropped.
   5. Delete collapsing, delete expiry, and compaction may be performed in
   a single commit. The oldest-valid-version may be advanced during this
   process as well. The outcome must logically be consistent with applying
   these steps independently.

Important to note in the above is that, during collapsing/compaction, new
files may be emitted with a version number N that are older than the most
recent version. This is important to ensure that all deltas newer than N
are appropriately applied, and that incremental readers are able to
continue processing the dataset.

It is also compatible with file-level deletes used to age off old data
> (e.g. delete where day(ts) < '2019-01-01').


This particular operation is not compatible with incremental consumption.
One must still generate a deletion that is associated with a new sequence
number. I could see a reasonable workaround where, in order to delete an
entire file (insertion file `x.dat`, sequence number X) one would reference
the same file a second time (deletion file `x.dat, sequence number Y > X).
Since an insertion file and a deletion file have compatible formats, there
is no need to rewrite this file simply to mark each row in it as deleted. A
savvy consumer would be able to tell that this is a whole-file deletion
while a naïve consumer would be able to apply them using the basic
algorithm.

If it seems like we are roughly on the same page I will take a stab at
updating that document to go to the same level of detail that it does now
but use the sequence-number approach.

On Fri, Jun 14, 2019 at 6:30 PM Ryan Blue <rb...@netflix.com> wrote:

> First, I want to clear up some of the confusion with a bit of background
> on how Iceberg works, but I have inline replies below as well.
>
> Conceptually, a snapshot is a table version. Each snapshot is the complete
> table state at some point in time and each snapshot is independent.
> Independence is a key property of snapshots because it allows us to
> maintain table data and metadata.
>
> Snapshots track additions and deletions using per-file status (ADDED/
> EXISTING/DELETED) and the snapshot ID when each file was added or
> deleted. This can be efficiently loaded and used to process changes
> incrementally. Snapshots are also annotated with the operation that created
> each snapshot. This is used to identify snapshots with no logical changes
> (replace) and snapshots with no deletions (append) to speed up incremental
> consumption and operations like snapshot expiration.
>
> Readers use the current snapshot for isolation from writes. Because
> readers may be using an old snapshot, Iceberg keeps snapshots around and
> only physically deletes files when snapshots expire — data files referenced
> in a valid snapshot cannot be deleted. Independence is important so that
> snapshots can expire and get cleaned up without any other actions on the
> table data.
>
> Consider the alternative: if each snapshot depended on the previous one,
> then Iceberg cannot expire them and physically delete data. Here’s an
> example snapshot history:
>
> S1
> - add file_a.parquet
> - add file_b.parquet
> S2
> - delete file_a.parquet
> S3
> - add file_c.parquet
>
> To delete file_a.parquet, S1 and S2 need to be rewritten into S2’ that
> contains only file_b.parquet. This would work, but takes more time than
> dropping references and deleting files that are no longer used. An
> expensive metadata rewrite to delete data isn’t ideal, and not taking the
> time to rewrite makes metadata stack up in the table metadata file, that
> gets rewritten often.
>
> Finally, Iceberg relies on regular metadata maintenance — manifest
> compaction — to reduce both write volume and the number of file reads
> needed to plan a scan. Snapshots also reuse metadata files from previous
> snapshots to reduce write volume.
>
> On Wed, Jun 12, 2019 at 9:12 AM Erik Wright erik.wright@shopify.com
> <ht...@shopify.com> wrote:
>
>
>>>    - Maintaining separate manifest lists for each version requires
>>>    keeping versions in separate metadata trees, so we can’t merge manifest
>>>    files together across versions. We initially had this problem when we used
>>>    only fast appends for our streaming writes. We ended up with way too many
>>>    metadata files and job planning took a really long time.
>>>
>>> If I understand correctly, the main concern here is for scenarios where
>> versions are essentially micro-batches. They accumulate rapidly, each one
>> is relatively small, and there may be many valid versions at a time even if
>> expiry happens regularly. The negative impact is many small versions cannot
>> be combined together. Even the manifest list files are many, and all of
>> them need to be read.
>>
> I’m not sure how you define micro-batch, but this happens when commits are
> on the order of every 10 minutes and consist of large data files. Manifest
> compaction is required to avoid thousands of small metadata files impacting
> performance.
>
>
>>>    - Although snapshots are independent and can be aged off, this
>>>    requires rewriting the base version to expire versions. That means we can’t
>>>    delete old data files until versions are rewritten, so the problem affects
>>>    versions instead of snapshots. We would still have the problem of not being
>>>    able to delete old data until after a compaction.
>>>
>>> I don't quite understand this point, so forgive me if the following is
>> irrelevant or obvious.
>>
>> You are correct that snapshots become less useful as a tool, compared to
>> versions, for expiring data. The process now goes through (1) expire a
>> version and then (2) invalidate the last snapshot containing that version.
>> Of course, fundamentally you need to keep version X around until you are
>> willing to compact delta Y (or more) into it. And this is (I believe) our
>> common purpose in this discussion: permit a trade-off between write
>> amplification and storage/read costs.
>>
> Hopefully the background above is useful. Requiring expensive data-file
> compaction in order to compact version metadata or delete files is a big
> problem.
>
> And you’re right that snapshots are less useful. Because snapshots and
> versions are basically the same idea, we don’t need both. If we were to add
> versions, they should replace snapshots. But I don’t think it is a good
> idea to give up the independence.
>
> The good news is that we can use snapshots exactly like versions and avoid
> these problems by adding sequence numbers.
>
> The value of decoupling versions from snapshots is:
>>
>> 1. It makes the process of correctly applying multiple deltas easy to
>> reason about.
>> 2. It allows us to issue updates that don't introduce new versions (i.e.,
>> those that merely expire old versions, or that perform other optimizations).
>> 3. It enables clients to cleanly consume new deltas when those clients
>> are sophisticated enough to incrementally update their own outputs in
>> function of those changes.
>>
> Points 2 and 3 are available already, and 1 can be done with sequence
> numbers in snapshot metadata.
>
> Here are some benefits to this approach:
>>>
>>>    - There is no explicit version that must be expired so compaction is
>>>    only required to optimize reads — partition or file-level delete can
>>>    coexist with deltas
>>>
>>> Can you expand on this?
>>
> By adding a sequence number instead of versions, the essential version
> information is embedded in the existing metadata files. Those files are
> reused across snapshots so there is no need to compact deltas into data
> files to reduce the write volume.
>
> Another way of thinking about it: if versions are stored as data/delete
> file metadata and not in the table metadata file, then the complete history
> can be kept.
>
> It is also compatible with file-level deletes used to age off old data
> (e.g. delete where day(ts) < '2019-01-01'). When files match a delete,
> they are removed from metadata and physically deleted when the delete
> snapshot is expired. This is independent of the delete files that get
> applied on read. In fact, if a delete file’s sequence number is less than
> all data files in a partition, the delete itself can be removed and never
> compacted.
>
>
>>>    - When a delete’s sequence number is lower than all data files, it
>>>    can be pruned
>>>
>>> In combination with the previous comment, I have the impression that we
>> might be losing the distinction between snapshots and versions. In
>> particular, what I'm concerned about is that, from a consumer's point of
>> view, it may be difficult to figure out, from one snapshot to the next,
>> what new deltas I need to process to update my view.
>>
> We don’t need both and should go with either snapshots (independent) or
> versions (dependent). I don’t think it is possible to make dependent
> versions work.
>
> But as I noted above, the changes that occurred in a snapshot are
> available as data file additions and data file deletions. Delete file
> additions and deletions would be tracked the same way. Consumers would have
> full access to the changes.
>
>
>>>    - Compaction can translate a natural key delete to a synthetic key
>>>    delete by adding a synthetic key delete file and updating the data file’s
>>>    sequence number
>>>
>>> I don't understand what this means.
>>
> Synthetic keys are better than natural keys in several ways. They are
> cheaper to apply at read time, they are scoped to a single data file, they
> compress well, etc. If both are supported, compaction could rewrite a
> natural key delete as a synthetic key delete to avoid rewriting the entire
> data file to delete just a few rows.
>
> In addition, if a natural key delete is applied to an entire table, it
> would be possible to change sequence numbers to apply this process
> incrementally. But, this would change history just like rewriting the base
> version of a table loses the changes in that version. So we would need to
> be careful about when changing a sequence number is allowed.
>
>
>>>    - Synthetic and natural key approaches can coexist
>>>
>>> Can you elaborate on what the synthetic key approach would imply, and
>> what use-cases it would serve?
>>
> Some use cases, like MERGE INTO, already require reading data so we can
> write synthetic key delete files without additional work. That means
> Iceberg should support not just natural key deletes, but also synthetic.
>
> I agree we are getting closer. I am confused by some things, and I suspect
>> that the difference lies in the use-cases I wish to support for incremental
>> consumers of these datasets. I have the impression that most of you folks
>> are primarily considering incremental writes that end up serving analytical
>> use cases (that want to see the full effective dataset as of the most
>> recent snapshot).
>>
> We have both use cases and I want to make sure Iceberg can be used for
> both.
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: Updates/Deletes/Upserts in Iceberg

[Cristian Opris <co...@apple.com.INVALID>]

Hi Ryan,

Thanks for providing more clarification on the current workings of the metadata system in Iceberg.

I must say that having read the Table Spec document, it's still quite difficult to understand in detail how everything works conceptually. 

Based on your description below, and particularly the "independence" property of snapshots that you describe, it feels to me that the abstract data structure that this is attempting
to model is some kind of persistent tree (persistent in the sense of functional, immutable: https://en.wikipedia.org/wiki/Persistent_data_structure#Trees <https://en.wikipedia.org/wiki/Persistent_data_structure#Trees>)

This kind of data structures seems to align well with the requirements of not being able to support in place updates (since we use write-once storage for metadata) and not changing unaffected parts (maximizing data sharing).
  
I wonder if it would be possible to formalize the metadata system with such an abstract data structure. I think this will go a long way towards making sure that we understand the kind of operations that can be applied to it and
the complexity bounds of those operations.

Or, at the very least, it would be extremely helpful for everyone looking to understand Iceberg to have available a diagram of the current metadata design, as a tree with the nodes and edges annotated with the essential information they contain, and then present (perhaps also visually) the main operations that can be currently applied to this data structure ?

I think this would help reduce the confusion around numbering systems, snapshot vs. version, etc. where I admit at least myself I'm a bit lost in understanding. 

It would also help in gaining the confidence that operations are all consistent, and there are no edge cases that may leave metadata in a weird state.


Also, from what I can gather from a few email exchanges, the main reason for the current design is to allow uncoordinated concurrent writers to consistently modify the metadata structures with the lowest possible write latency ?  Would be interesting if you can provide more detail on this particular use case, to better understand the rationale for the current design ?


Thanks,
Cristian



> On 14 Jun 2019, at 23:29, Ryan Blue <rb...@netflix.com.INVALID> wrote:
> 
> First, I want to clear up some of the confusion with a bit of background on how Iceberg works, but I have inline replies below as well.
> 
> Conceptually, a snapshot is a table version. Each snapshot is the complete table state at some point in time and each snapshot is independent. Independence is a key property of snapshots because it allows us to maintain table data and metadata.
> 
> Snapshots track additions and deletions using per-file status (ADDED/EXISTING/DELETED) and the snapshot ID when each file was added or deleted. This can be efficiently loaded and used to process changes incrementally. Snapshots are also annotated with the operation that created each snapshot. This is used to identify snapshots with no logical changes (replace) and snapshots with no deletions (append) to speed up incremental consumption and operations like snapshot expiration.
> 
> Readers use the current snapshot for isolation from writes. Because readers may be using an old snapshot, Iceberg keeps snapshots around and only physically deletes files when snapshots expire — data files referenced in a valid snapshot cannot be deleted. Independence is important so that snapshots can expire and get cleaned up without any other actions on the table data.
> 
> Consider the alternative: if each snapshot depended on the previous one, then Iceberg cannot expire them and physically delete data. Here’s an example snapshot history:
> 
> S1
> - add file_a.parquet
> - add file_b.parquet
> S2
> - delete file_a.parquet
> S3
> - add file_c.parquet
> To delete file_a.parquet, S1 and S2 need to be rewritten into S2’ that contains only file_b.parquet. This would work, but takes more time than dropping references and deleting files that are no longer used. An expensive metadata rewrite to delete data isn’t ideal, and not taking the time to rewrite makes metadata stack up in the table metadata file, that gets rewritten often.
> 
> Finally, Iceberg relies on regular metadata maintenance — manifest compaction — to reduce both write volume and the number of file reads needed to plan a scan. Snapshots also reuse metadata files from previous snapshots to reduce write volume.
> 
> On Wed, Jun 12, 2019 at 9:12 AM Erik Wright erik.wright@shopify.com <http://mailto:erik.wright@shopify.com/> wrote:
> 
> 
> 
> Maintaining separate manifest lists for each version requires keeping versions in separate metadata trees, so we can’t merge manifest files together across versions. We initially had this problem when we used only fast appends for our streaming writes. We ended up with way too many metadata files and job planning took a really long time.
> If I understand correctly, the main concern here is for scenarios where versions are essentially micro-batches. They accumulate rapidly, each one is relatively small, and there may be many valid versions at a time even if expiry happens regularly. The negative impact is many small versions cannot be combined together. Even the manifest list files are many, and all of them need to be read.
> 
> 
> I’m not sure how you define micro-batch, but this happens when commits are on the order of every 10 minutes and consist of large data files. Manifest compaction is required to avoid thousands of small metadata files impacting performance.
> 
> 
> 
> Although snapshots are independent and can be aged off, this requires rewriting the base version to expire versions. That means we can’t delete old data files until versions are rewritten, so the problem affects versions instead of snapshots. We would still have the problem of not being able to delete old data until after a compaction.
> I don't quite understand this point, so forgive me if the following is irrelevant or obvious.
> 
> You are correct that snapshots become less useful as a tool, compared to versions, for expiring data. The process now goes through (1) expire a version and then (2) invalidate the last snapshot containing that version. Of course, fundamentally you need to keep version X around until you are willing to compact delta Y (or more) into it. And this is (I believe) our common purpose in this discussion: permit a trade-off between write amplification and storage/read costs.
> 
> 
> Hopefully the background above is useful. Requiring expensive data-file compaction in order to compact version metadata or delete files is a big problem.
> 
> And you’re right that snapshots are less useful. Because snapshots and versions are basically the same idea, we don’t need both. If we were to add versions, they should replace snapshots. But I don’t think it is a good idea to give up the independence.
> 
> The good news is that we can use snapshots exactly like versions and avoid these problems by adding sequence numbers.
> 
> 
> 
> The value of decoupling versions from snapshots is:
> 
> 1. It makes the process of correctly applying multiple deltas easy to reason about.
> 2. It allows us to issue updates that don't introduce new versions (i.e., those that merely expire old versions, or that perform other optimizations).
> 3. It enables clients to cleanly consume new deltas when those clients are sophisticated enough to incrementally update their own outputs in function of those changes.
> 
> 
> Points 2 and 3 are available already, and 1 can be done with sequence numbers in snapshot metadata.
> 
> 
> 
> Here are some benefits to this approach:
> 
> There is no explicit version that must be expired so compaction is only required to optimize reads — partition or file-level delete can coexist with deltas
> Can you expand on this?
> 
> 
> By adding a sequence number instead of versions, the essential version information is embedded in the existing metadata files. Those files are reused across snapshots so there is no need to compact deltas into data files to reduce the write volume.
> 
> Another way of thinking about it: if versions are stored as data/delete file metadata and not in the table metadata file, then the complete history can be kept.
> 
> It is also compatible with file-level deletes used to age off old data (e.g. delete where day(ts) < '2019-01-01'). When files match a delete, they are removed from metadata and physically deleted when the delete snapshot is expired. This is independent of the delete files that get applied on read. In fact, if a delete file’s sequence number is less than all data files in a partition, the delete itself can be removed and never compacted.
> 
> 
> 
> When a delete’s sequence number is lower than all data files, it can be pruned
> In combination with the previous comment, I have the impression that we might be losing the distinction between snapshots and versions. In particular, what I'm concerned about is that, from a consumer's point of view, it may be difficult to figure out, from one snapshot to the next, what new deltas I need to process to update my view.
> 
> 
> We don’t need both and should go with either snapshots (independent) or versions (dependent). I don’t think it is possible to make dependent versions work.
> 
> But as I noted above, the changes that occurred in a snapshot are available as data file additions and data file deletions. Delete file additions and deletions would be tracked the same way. Consumers would have full access to the changes.
> 
> 
> 
> Compaction can translate a natural key delete to a synthetic key delete by adding a synthetic key delete file and updating the data file’s sequence number
> I don't understand what this means.
> 
> 
> Synthetic keys are better than natural keys in several ways. They are cheaper to apply at read time, they are scoped to a single data file, they compress well, etc. If both are supported, compaction could rewrite a natural key delete as a synthetic key delete to avoid rewriting the entire data file to delete just a few rows.
> 
> In addition, if a natural key delete is applied to an entire table, it would be possible to change sequence numbers to apply this process incrementally. But, this would change history just like rewriting the base version of a table loses the changes in that version. So we would need to be careful about when changing a sequence number is allowed.
> 
> 
> 
> Synthetic and natural key approaches can coexist
> Can you elaborate on what the synthetic key approach would imply, and what use-cases it would serve? 
> 
> 
> Some use cases, like MERGE INTO, already require reading data so we can write synthetic key delete files without additional work. That means Iceberg should support not just natural key deletes, but also synthetic.
> 
> 
> 
> I agree we are getting closer. I am confused by some things, and I suspect that the difference lies in the use-cases I wish to support for incremental consumers of these datasets. I have the impression that most of you folks are primarily considering incremental writes that end up serving analytical use cases (that want to see the full effective dataset as of the most recent snapshot).
> 
> 
> We have both use cases and I want to make sure Iceberg can be used for both.
> 
> -- 
> Ryan Blue
> Software Engineer
> Netflix

Re: Updates/Deletes/Upserts in Iceberg

[Ryan Blue <rb...@netflix.com.INVALID>]

First, I want to clear up some of the confusion with a bit of background on
how Iceberg works, but I have inline replies below as well.

Conceptually, a snapshot is a table version. Each snapshot is the complete
table state at some point in time and each snapshot is independent.
Independence is a key property of snapshots because it allows us to
maintain table data and metadata.

Snapshots track additions and deletions using per-file status (ADDED/
EXISTING/DELETED) and the snapshot ID when each file was added or deleted.
This can be efficiently loaded and used to process changes incrementally.
Snapshots are also annotated with the operation that created each snapshot.
This is used to identify snapshots with no logical changes (replace) and
snapshots with no deletions (append) to speed up incremental consumption
and operations like snapshot expiration.

Readers use the current snapshot for isolation from writes. Because readers
may be using an old snapshot, Iceberg keeps snapshots around and only
physically deletes files when snapshots expire — data files referenced in a
valid snapshot cannot be deleted. Independence is important so that
snapshots can expire and get cleaned up without any other actions on the
table data.

Consider the alternative: if each snapshot depended on the previous one,
then Iceberg cannot expire them and physically delete data. Here’s an
example snapshot history:

S1
- add file_a.parquet
- add file_b.parquet
S2
- delete file_a.parquet
S3
- add file_c.parquet

To delete file_a.parquet, S1 and S2 need to be rewritten into S2’ that
contains only file_b.parquet. This would work, but takes more time than
dropping references and deleting files that are no longer used. An
expensive metadata rewrite to delete data isn’t ideal, and not taking the
time to rewrite makes metadata stack up in the table metadata file, that
gets rewritten often.

Finally, Iceberg relies on regular metadata maintenance — manifest
compaction — to reduce both write volume and the number of file reads
needed to plan a scan. Snapshots also reuse metadata files from previous
snapshots to reduce write volume.

On Wed, Jun 12, 2019 at 9:12 AM Erik Wright erik.wright@shopify.com
<ht...@shopify.com> wrote:


>>    - Maintaining separate manifest lists for each version requires
>>    keeping versions in separate metadata trees, so we can’t merge manifest
>>    files together across versions. We initially had this problem when we used
>>    only fast appends for our streaming writes. We ended up with way too many
>>    metadata files and job planning took a really long time.
>>
>> If I understand correctly, the main concern here is for scenarios where
> versions are essentially micro-batches. They accumulate rapidly, each one
> is relatively small, and there may be many valid versions at a time even if
> expiry happens regularly. The negative impact is many small versions cannot
> be combined together. Even the manifest list files are many, and all of
> them need to be read.
>
I’m not sure how you define micro-batch, but this happens when commits are
on the order of every 10 minutes and consist of large data files. Manifest
compaction is required to avoid thousands of small metadata files impacting
performance.


>>    - Although snapshots are independent and can be aged off, this
>>    requires rewriting the base version to expire versions. That means we can’t
>>    delete old data files until versions are rewritten, so the problem affects
>>    versions instead of snapshots. We would still have the problem of not being
>>    able to delete old data until after a compaction.
>>
>> I don't quite understand this point, so forgive me if the following is
> irrelevant or obvious.
>
> You are correct that snapshots become less useful as a tool, compared to
> versions, for expiring data. The process now goes through (1) expire a
> version and then (2) invalidate the last snapshot containing that version.
> Of course, fundamentally you need to keep version X around until you are
> willing to compact delta Y (or more) into it. And this is (I believe) our
> common purpose in this discussion: permit a trade-off between write
> amplification and storage/read costs.
>
Hopefully the background above is useful. Requiring expensive data-file
compaction in order to compact version metadata or delete files is a big
problem.

And you’re right that snapshots are less useful. Because snapshots and
versions are basically the same idea, we don’t need both. If we were to add
versions, they should replace snapshots. But I don’t think it is a good
idea to give up the independence.

The good news is that we can use snapshots exactly like versions and avoid
these problems by adding sequence numbers.

The value of decoupling versions from snapshots is:
>
> 1. It makes the process of correctly applying multiple deltas easy to
> reason about.
> 2. It allows us to issue updates that don't introduce new versions (i.e.,
> those that merely expire old versions, or that perform other optimizations).
> 3. It enables clients to cleanly consume new deltas when those clients are
> sophisticated enough to incrementally update their own outputs in function
> of those changes.
>
Points 2 and 3 are available already, and 1 can be done with sequence
numbers in snapshot metadata.

Here are some benefits to this approach:
>>
>>    - There is no explicit version that must be expired so compaction is
>>    only required to optimize reads — partition or file-level delete can
>>    coexist with deltas
>>
>> Can you expand on this?
>
By adding a sequence number instead of versions, the essential version
information is embedded in the existing metadata files. Those files are
reused across snapshots so there is no need to compact deltas into data
files to reduce the write volume.

Another way of thinking about it: if versions are stored as data/delete
file metadata and not in the table metadata file, then the complete history
can be kept.

It is also compatible with file-level deletes used to age off old data
(e.g. delete where day(ts) < '2019-01-01'). When files match a delete, they
are removed from metadata and physically deleted when the delete snapshot
is expired. This is independent of the delete files that get applied on
read. In fact, if a delete file’s sequence number is less than all data
files in a partition, the delete itself can be removed and never compacted.


>>    - When a delete’s sequence number is lower than all data files, it
>>    can be pruned
>>
>> In combination with the previous comment, I have the impression that we
> might be losing the distinction between snapshots and versions. In
> particular, what I'm concerned about is that, from a consumer's point of
> view, it may be difficult to figure out, from one snapshot to the next,
> what new deltas I need to process to update my view.
>
We don’t need both and should go with either snapshots (independent) or
versions (dependent). I don’t think it is possible to make dependent
versions work.

But as I noted above, the changes that occurred in a snapshot are available
as data file additions and data file deletions. Delete file additions and
deletions would be tracked the same way. Consumers would have full access
to the changes.


>>    - Compaction can translate a natural key delete to a synthetic key
>>    delete by adding a synthetic key delete file and updating the data file’s
>>    sequence number
>>
>> I don't understand what this means.
>
Synthetic keys are better than natural keys in several ways. They are
cheaper to apply at read time, they are scoped to a single data file, they
compress well, etc. If both are supported, compaction could rewrite a
natural key delete as a synthetic key delete to avoid rewriting the entire
data file to delete just a few rows.

In addition, if a natural key delete is applied to an entire table, it
would be possible to change sequence numbers to apply this process
incrementally. But, this would change history just like rewriting the base
version of a table loses the changes in that version. So we would need to
be careful about when changing a sequence number is allowed.


>>    - Synthetic and natural key approaches can coexist
>>
>> Can you elaborate on what the synthetic key approach would imply, and
> what use-cases it would serve?
>
Some use cases, like MERGE INTO, already require reading data so we can
write synthetic key delete files without additional work. That means
Iceberg should support not just natural key deletes, but also synthetic.

I agree we are getting closer. I am confused by some things, and I suspect
> that the difference lies in the use-cases I wish to support for incremental
> consumers of these datasets. I have the impression that most of you folks
> are primarily considering incremental writes that end up serving analytical
> use cases (that want to see the full effective dataset as of the most
> recent snapshot).
>
We have both use cases and I want to make sure Iceberg can be used for both.
-- 
Ryan Blue
Software Engineer
Netflix

Re: Updates/Deletes/Upserts in Iceberg

[Erik Wright <er...@shopify.com.INVALID>]

Thanks for taking the time to read through this and give your feedback. I
agree that we are closing in on something here.

On Tue, Jun 11, 2019 at 12:36 PM Ryan Blue <rb...@netflix.com> wrote:

> Erik, thanks for working on this doc. It’s a good detailed write-up of the
> approach using natural keys and I found the section about efficiently
> merging changes together to be helpful.
>
> For the specific updates to Iceberg metadata, I think we will need to
> update it. Dan and I were thinking about the sequencing problem last week
> and came up with a solution to augment what’s in the proposal.
> Conceptually, what is in the doc would definitely work, but it hits a few
> pitfalls that we’ve already had work around:
>
>    - In early versions Iceberg kept the list of manifests in the table
>    metadata (JSON) file that are equivalent to the manifest list for each
>    version. That ended up taking too much space, causing high write latency.
>    It also required opening too many files at planning time.
>    - Maintaining separate manifest lists for each version requires
>    keeping versions in separate metadata trees, so we can’t merge manifest
>    files together across versions. We initially had this problem when we used
>    only fast appends for our streaming writes. We ended up with way too many
>    metadata files and job planning took a really long time.
>
> If I understand correctly, the main concern here is for scenarios where
versions are essentially micro-batches. They accumulate rapidly, each one
is relatively small, and there may be many valid versions at a time even if
expiry happens regularly. The negative impact is many small versions cannot
be combined together. Even the manifest list files are many, and all of
them need to be read.

I totally agree with that. One thing worth noting is that Iceberg _already_
handles cases where updates are pure inserts. It's worth considering
whether the cases that we are now looking at (where there are also updates
and deletes) will likely exhibit the same update patterns that you have
mentioned (many small writes). It may very well be that they do.

>
>    - Although snapshots are independent and can be aged off, this
>    requires rewriting the base version to expire versions. That means we can’t
>    delete old data files until versions are rewritten, so the problem affects
>    versions instead of snapshots. We would still have the problem of not being
>    able to delete old data until after a compaction.
>
> I don't quite understand this point, so forgive me if the following is
irrelevant or obvious.

You are correct that snapshots become less useful as a tool, compared to
versions, for expiring data. The process now goes through (1) expire a
version and then (2) invalidate the last snapshot containing that version.
Of course, fundamentally you need to keep version X around until you are
willing to compact delta Y (or more) into it. And this is (I believe) our
common purpose in this discussion: permit a trade-off between write
amplification and storage/read costs.

The value of decoupling versions from snapshots is:

1. It makes the process of correctly applying multiple deltas easy to
reason about.
2. It allows us to issue updates that don't introduce new versions (i.e.,
those that merely expire old versions, or that perform other optimizations).
3. It enables clients to cleanly consume new deltas when those clients are
sophisticated enough to incrementally update their own outputs in function
of those changes.

>
>
> The solution we came up with is to assign each data or delete file a
> sequence number. This is basically the same thing as the version numbers in
> your doc, so what you described for reading would work from the point where
> you have files that are associated with a version number. That requires a
> way to assign sequence numbers without needing to rewrite metadata when
> commits conflict.
>
> The proposal included sequence numbers in the table metadata file as
> versions and avoided rewrites by adding more manifest lists. Building on
> that idea, we suggest adding those sequence numbers to each manifest file
> in a manifest list. That way, we only have one manifest list file per
> snapshot just like the current format. This works because the manifest list
> is always rewritten when there is a commit conflict. The proposal avoids
> rewriting manifest lists by embedding this information in the table
> metadata file, but then each conflict causes you to rewrite the version
> list for all snapshots: instead of rewriting 2 files, it rewrites one much
> larger file — as I mentioned above, that cost is part of why we introduced
> manifest lists.
>
> Because the manifest list is rewritten when commits conflict, the sequence
> number of manifests added in that snapshot can be updated. That solves one
> problem, but then we would still have the problem that the actual manifests
> do not contain sequence numbers so we can’t merge manifests. To solve that,
> we propose optionally writing a sequence number with each data file in a
> manifest. If a data or delete file doesn’t have a sequence number, it
> inherits the sequence number of the manifest on read. Then compacting
> manifests just works because data files are guaranteed to have a correct
> sequence number when they are read. Also, manifests don’t need to be
> rewritten during a conflict because new manifests are written without a
> sequence number for each data file: the data files inherit the sequence
> number that is set for the manifest when it is committed.
>
Up to here, I think I am with you. You are OK rewriting manifest list files
in a conflict, and by having the flexibility to assign the sequence number
either to a manifest file or to individual data files you make it easy to
write those small manifest files, easy to deal with conflicts, but also
easy to combine manifest files (even though you cannot compact data files
across versions).


> Using this strategy, we can store all of the version information from the
> proposal as sequence numbers in the existing metadata structures, with
> minimal changes to Iceberg’s existing metadata structure. Here are some
> benefits to this approach:
>
>    - There is no explicit version that must be expired so compaction is
>    only required to optimize reads — partition or file-level delete can
>    coexist with deltas
>
> Can you expand on this?

>
>    - When a delete’s sequence number is lower than all data files, it can
>    be pruned
>
> In combination with the previous comment, I have the impression that we
might be losing the distinction between snapshots and versions. In
particular, what I'm concerned about is that, from a consumer's point of
view, it may be difficult to figure out, from one snapshot to the next,
what new deltas I need to process to update my view.

>
>    - Compaction can translate a natural key delete to a synthetic key
>    delete by adding a synthetic key delete file and updating the data file’s
>    sequence number
>
> I don't understand what this means.

>
>    - Synthetic and natural key approaches can coexist
>
> Can you elaborate on what the synthetic key approach would imply, and what
use-cases it would serve?

>
>    - More of the existing Iceberg operations and structure are reused —
>    less code is required to get this working
>
> I agree that this model would minimize, at least, changes to the table
structure. I'm not sure that it changes much with respect to the algorithm
for reading data or producing it, and I would imagine that the reading side
is where most of the changes will lie.

>
>
> I think by combining your ideas with the sequence number approach, we’re
> fairly close to something we can start building and I’m confident that it
> would perform well.
>
> I’m happy to reply with updates if what I’m proposing isn’t clear enough
> to understand. We can also set up another sync-up to talk this through if
> everyone is interested.
>
I agree we are getting closer. I am confused by some things, and I suspect
that the difference lies in the use-cases I wish to support for incremental
consumers of these datasets. I have the impression that most of you folks
are primarily considering incremental writes that end up serving analytical
use cases (that want to see the full effective dataset as of the most
recent snapshot).

Can you take a stab at responding to the areas of confusion I highlighted
above? Let me know if I need to do some more writing/drawing to describe
the incremental read concerns. Yes, after another round or two like this it
would likely make sense to have another video conference. At the moment,
next week's afternoons (west coast mornings) are good for me apart from
Tuesday.

rb
>
> On Fri, Jun 7, 2019 at 11:47 AM Ryan Blue <rb...@netflix.com> wrote:
>
>> Thanks, Erik! Great to see progress here. I'll set aside some time to
>> look this over in detail.
>>
>> On Fri, Jun 7, 2019 at 11:46 AM Erik Wright <er...@shopify.com>
>> wrote:
>>
>>> I apologize for the delay, but I have finally put together a document
>>> describing an alternative approach to supporting updates in Iceberg while
>>> minimizing write amplification.
>>>
>>> Proposal: Iceberg Merge-on-Read
>>> <https://docs.google.com/document/d/1KuOMeS8Hw_yuE5IXtII8EClJlEMtqFECfWbg-gN5lsQ/edit?usp=sharing>
>>>
>>> Thank you, Anton and Miguel, for starting this conversation, and
>>> everyone else as well for the ongoing dialogue. I'm looking forward to
>>> continuing to discuss this and hopefully finding an approach that can meet
>>> our different needs and that we can work together on.
>>>
>>> Cheers,
>>>
>>> Erik
>>>
>>> On Wed, May 29, 2019 at 7:26 PM Jacques Nadeau <ja...@dremio.com>
>>> wrote:
>>>
>>>> Yeah, I totally forgot to record our discussion. Will do so next time,
>>>> sorry.
>>>> --
>>>> Jacques Nadeau
>>>> CTO and Co-Founder, Dremio
>>>>
>>>>
>>>> On Wed, May 29, 2019 at 4:24 PM Ryan Blue <rb...@netflix.com> wrote:
>>>>
>>>>> It wasn't recorded, but I can summarize what we talked about. Sorry I
>>>>> haven't sent this out earlier.
>>>>>
>>>>> We talked about the options and some of the background in Iceberg --
>>>>> basically that it isn't possible to determine the order of commits before
>>>>> you commit so you can't rely on some monotonically increasing value from a
>>>>> snapshot to know which deltas to apply to a file. The result is that we
>>>>> can't apply diffs to data files using a rule like "files older than X"
>>>>> because we can't identify those files without the snapshot history.
>>>>>
>>>>> That gives us basically 2 options for scoping delete diffs: either
>>>>> identify the files to apply a diff to when writing the diff, or log changes
>>>>> applied to a snapshot and keep the snapshot history around (which is how we
>>>>> know the order of snapshots). The first option is not good if you want to
>>>>> write without reading data to determine where the deleted records are. The
>>>>> second prevents cleaning up snapshot history.
>>>>>
>>>>> We also talked about whether we should encode IDs in data files.
>>>>> Jacques pointed out that retrying a commit is easier if you don't need to
>>>>> re-read the original data to reconcile changes. For example, if a data file
>>>>> was compacted in a concurrent write, how do we reconcile a delete for it?
>>>>> We discussed other options, like rolling back the compaction for delete
>>>>> events. I think that's a promising option.
>>>>>
>>>>> For action items, Jacques was going to think about whether we need to
>>>>> encode IDs in data files or if we could use positions to identify rows and
>>>>> write up a summary/proposal. Erik was going to take on planning how
>>>>> identifying rows without reading data would work and similarly write up a
>>>>> summary/proposal.
>>>>>
>>>>> That's from memory, so if I've missed anything, I hope that other
>>>>> attendees will fill in the details!
>>>>>
>>>>> rb
>>>>>
>>>>> On Wed, May 29, 2019 at 3:34 PM Venkatakrishnan Sowrirajan <
>>>>> vsowrira@asu.edu> wrote:
>>>>>
>>>>>> Hi Ryan,
>>>>>>
>>>>>> I couldn't attend the meeting. Just curious, if this is recorded by
>>>>>> any chance.
>>>>>>
>>>>>> Regards
>>>>>> Venkata krishnan
>>>>>>
>>>>>>
>>>>>> On Fri, May 24, 2019 at 8:49 AM Ryan Blue <rb...@netflix.com.invalid>
>>>>>> wrote:
>>>>>>
>>>>>>> Yes, I agree. I'll talk a little about a couple of the constraints
>>>>>>> of this as well.
>>>>>>>
>>>>>>> On Fri, May 24, 2019 at 5:52 AM Anton Okolnychyi <
>>>>>>> aokolnychyi@apple.com> wrote:
>>>>>>>
>>>>>>>> The agenda looks good to me. I think it would also make sense to
>>>>>>>> clarify the responsibilities of query engines and Iceberg. Not only in
>>>>>>>> terms of uniqueness, but also in terms of applying diffs on read, for
>>>>>>>> example.
>>>>>>>>
>>>>>>>> On 23 May 2019, at 01:59, Ryan Blue <rb...@netflix.com.INVALID>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Here’s a rough agenda:
>>>>>>>>
>>>>>>>>    - Use cases: everyone come with a use case that you’d like to
>>>>>>>>    have supported. We’ll go around and introduce ourselves and our use cases.
>>>>>>>>    - Main topic: How should Iceberg identify rows that are deleted?
>>>>>>>>    - Side topics from my initial email, if we have time: should we
>>>>>>>>    use insert diffs, should we support dense and sparse formats, etc.
>>>>>>>>
>>>>>>>> The main topic I think we should discuss is: *How should Iceberg
>>>>>>>> identify rows that are deleted?*
>>>>>>>>
>>>>>>>> I’m phrasing it this way to avoid where I think we’re talking past
>>>>>>>> one another because we are making assumptions. The important thing is that
>>>>>>>> there are two main options:
>>>>>>>>
>>>>>>>>    - Filename and position, vs
>>>>>>>>    - Specific values of (few) columns in the data
>>>>>>>>
>>>>>>>> This phrasing also avoids discussing uniqueness constraints. Once
>>>>>>>> we get down to behavior, I think we agree. For example, I think we all
>>>>>>>> agree that uniqueness cannot be enforced in Iceberg.
>>>>>>>>
>>>>>>>> If uniqueness can’t be enforced in Iceberg, the main choice comes
>>>>>>>> down to how we identify rows that are deleted. If we use (filename,
>>>>>>>> position) then we know that there is only one row. On the other hand, if we
>>>>>>>> use data values to identify rows then a delete may identify more than one
>>>>>>>> row because there are no uniqueness guarantees. I think we also agree that
>>>>>>>> if there is more than one row identified, all of them should be deleted.
>>>>>>>>
>>>>>>>> At that point, there are trade-offs between the approaches:
>>>>>>>>
>>>>>>>>    - When identifying deleted rows by data values, situations like
>>>>>>>>    the one that Anton pointed out are possible.
>>>>>>>>    - Jacques also had a good point about concurrency. If at all
>>>>>>>>    possible, we want to be able to reconcile changes between concurrent
>>>>>>>>    commits without re-running an operation.
>>>>>>>>
>>>>>>>> Sound like a reasonable amount to talk through?
>>>>>>>>
>>>>>>>> rb
>>>>>>>>
>>>>>>>> On Wed, May 22, 2019 at 1:17 PM Erik Wright <
>>>>>>>> erik.wright@shopify.com> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, May 22, 2019 at 4:04 PM Cristian Opris <
>>>>>>>>> copris@apple.com.invalid> wrote:
>>>>>>>>>
>>>>>>>>>> Agreed with Erik here, we're certainly not looking to build the
>>>>>>>>>> equivalent of a relational database, and for that matter not even that of a
>>>>>>>>>> local disk storage analytics database (like Vertica). Those are very
>>>>>>>>>> different designs with very different trade-offs and
>>>>>>>>>> optimizations.
>>>>>>>>>>
>>>>>>>>>> We're looking to automate and optimize specific types of file
>>>>>>>>>> manipulation for large files on remote storage, while presenting that to
>>>>>>>>>> the user under the common SQL API for *bulk* data manipulation
>>>>>>>>>> (MERGE INTO)
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> What I would encourage is to decouple the storage model from the
>>>>>>>>> implementation of that API. If Iceberg has support for merge-on-read of
>>>>>>>>> upserts and deletes, in addition to its powerful support for partitioning,
>>>>>>>>> it will be easy for a higher-level application to implement those APIs
>>>>>>>>> given certain other constraints (that might not be appropriate to all
>>>>>>>>> applications).
>>>>>>>>>
>>>>>>>>> Myself and Miguel are out on Friday, but Anton should be able to
>>>>>>>>>> handle the discussion on our side.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Cristian
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 22 May 2019, at 17:51, Erik Wright <
>>>>>>>>>> erik.wright@shopify.com.INVALID> wrote:
>>>>>>>>>>
>>>>>>>>>> We have two rows with the same natural key and we use that
>>>>>>>>>>> natural key in diff files:
>>>>>>>>>>> nk | col1 | col2
>>>>>>>>>>> 1 | 1 | 1
>>>>>>>>>>> 1 | 2 | 2
>>>>>>>>>>> Then we have a delete statement:
>>>>>>>>>>> DELETE FROM t WHERE col1 = 1
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I think this example cuts to the point of the differences of
>>>>>>>>>> understanding. Does Iceberg want to be approaching the utility of a
>>>>>>>>>> relational database, against which I can execute complex update queries?
>>>>>>>>>> This is not what I would have imagined.
>>>>>>>>>>
>>>>>>>>>> I would have, instead, imagined that it was up to the client to
>>>>>>>>>> identify, through whatever means, that they want to update or delete a row
>>>>>>>>>> with a given ID. If there are multiple (distinct) rows with the same ID,
>>>>>>>>>> _too bad_. Any user should _expect_ that they could potentially see any one
>>>>>>>>>> or more of those rows at read time. And that an upsert/delete would affect
>>>>>>>>>> any/all of them (I would argue for all).
>>>>>>>>>>
>>>>>>>>>> *In summary:* Instead of trying to come up with a consistent,
>>>>>>>>>> logical handling for complex queries that are best suited for a relational
>>>>>>>>>> database, leave such handling up to the client and concentrate on problems
>>>>>>>>>> that can be solved simply and more generally.
>>>>>>>>>>
>>>>>>>>>> On Wed, May 22, 2019 at 12:11 PM Ryan Blue <
>>>>>>>>>> rblue@netflix.com.invalid> wrote:
>>>>>>>>>>
>>>>>>>>>>> Yes, I think we should. I was going to propose one after
>>>>>>>>>>> catching up on the rest of this thread today.
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 22, 2019 at 9:08 AM Anton Okolnychyi <
>>>>>>>>>>> aokolnychyi@apple.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks! Would it make sense to discuss the agenda in advance?
>>>>>>>>>>>>
>>>>>>>>>>>> On 22 May 2019, at 17:04, Ryan Blue <rb...@netflix.com.INVALID>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I sent out an invite and included everyone on this thread. If
>>>>>>>>>>>> anyone else would like to join, please join the Zoom meeting. If you'd like
>>>>>>>>>>>> to be added to the calendar invite, just let me know and I'll add you.
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, May 22, 2019 at 8:57 AM Jacques Nadeau <
>>>>>>>>>>>> jacques@dremio.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> works for me.
>>>>>>>>>>>>>
>>>>>>>>>>>>> To make things easier, we can use my zoom meeting if people
>>>>>>>>>>>>> like:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Join Zoom Meeting
>>>>>>>>>>>>> https://zoom.us/j/4157302092
>>>>>>>>>>>>>
>>>>>>>>>>>>> One tap mobile
>>>>>>>>>>>>> +16465588656,,4157302092# US (New York)
>>>>>>>>>>>>> +16699006833,,4157302092# US (San Jose)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Dial by your location
>>>>>>>>>>>>>         +1 646 558 8656 US (New York)
>>>>>>>>>>>>>         +1 669 900 6833 US (San Jose)
>>>>>>>>>>>>>         877 853 5257 US Toll-free
>>>>>>>>>>>>>         888 475 4499 US Toll-free
>>>>>>>>>>>>> Meeting ID: 415 730 2092
>>>>>>>>>>>>> Find your local number: https://zoom.us/u/aH9XYBfm
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Jacques Nadeau
>>>>>>>>>>>>> CTO and Co-Founder, Dremio
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, May 22, 2019 at 8:54 AM Ryan Blue <
>>>>>>>>>>>>> rblue@netflix.com.invalid> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> 9AM on Friday works best for me. How about then?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, May 22, 2019 at 5:05 AM Anton Okolnychyi <
>>>>>>>>>>>>>> aokolnychyi@apple.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> What about this Friday? One hour slot from 9:00 to 10:00 am
>>>>>>>>>>>>>>> or 10:00 to 11:00 am PST? Some folks are based in London, so meeting later
>>>>>>>>>>>>>>> than this is hard. If Friday doesn’t work, we can consider Tuesday or
>>>>>>>>>>>>>>> Wednesday next week.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 22 May 2019, at 00:54, Jacques Nadeau <ja...@dremio.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I agree with Anton that we should probably spend some time
>>>>>>>>>>>>>>> on hangouts further discussing things. Definitely differing expectations
>>>>>>>>>>>>>>> here and we seem to be talking a bit past each other.
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Jacques Nadeau
>>>>>>>>>>>>>>> CTO and Co-Founder, Dremio
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, May 21, 2019 at 3:44 PM Cristian Opris <
>>>>>>>>>>>>>>> copris@apple.com.invalid> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I love a good flame war :P
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 21 May 2019, at 22:57, Jacques Nadeau <
>>>>>>>>>>>>>>>> jacques@dremio.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> That's my point, truly independent writers (two Spark jobs,
>>>>>>>>>>>>>>>>> or a Spark job and Dremio job) means a distributed transaction. It would
>>>>>>>>>>>>>>>>> need yet another external transaction coordinator on top of both Spark and
>>>>>>>>>>>>>>>>> Dremio, Iceberg by itself
>>>>>>>>>>>>>>>>> cannot solve this.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm not ready to accept this. Iceberg already supports a
>>>>>>>>>>>>>>>> set of semantics around multiple writers committing simultaneously and how
>>>>>>>>>>>>>>>> conflict resolution is done. The same can be done here.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> MVCC (which is what Iceberg tries to implement) requires a
>>>>>>>>>>>>>>>> total ordering of snapshots. Also the snapshots need to be non-conflicting.
>>>>>>>>>>>>>>>> I really don't see how any metadata data structures can solve this without
>>>>>>>>>>>>>>>> an outside coordinator.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Consider this:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Snapshot 0: (K,A) = 1
>>>>>>>>>>>>>>>> Job X: UPDATE K SET A=A+1
>>>>>>>>>>>>>>>> Job Y: UPDATE K SET A=10
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> What should the final value of A be and who decides ?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> By single writer, I don't mean single process, I mean
>>>>>>>>>>>>>>>>> multiple coordinated processes like Spark executors coordinated by Spark
>>>>>>>>>>>>>>>>> driver. The coordinator ensures that the data is pre-partitioned on
>>>>>>>>>>>>>>>>> each executor, and the coordinator commits the snapshot.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Note however that single writer job/multiple concurrent
>>>>>>>>>>>>>>>>> reader jobs is perfectly feasible, i.e. it shouldn't be a problem to write
>>>>>>>>>>>>>>>>> from a Spark job and read from multiple Dremio queries concurrently (for
>>>>>>>>>>>>>>>>> example)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> :D This is still "single process" from my perspective. That
>>>>>>>>>>>>>>>> process may be coordinating other processes to do distributed work but
>>>>>>>>>>>>>>>> ultimately it is a single process.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Fair enough
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'm not sure what you mean exactly. If we can't enforce
>>>>>>>>>>>>>>>>> uniqueness we shouldn't assume it.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I disagree. We can specify that as a requirement and state
>>>>>>>>>>>>>>>> that you'll get unintended consequences if you provide your own keys and
>>>>>>>>>>>>>>>> don't maintain this.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> There's no need for unintended consequences, we can specify
>>>>>>>>>>>>>>>> consistent behaviour (and I believe the document says what that is)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We do expect that most of the time the natural key is
>>>>>>>>>>>>>>>>> unique, but the eager and lazy with natural key designs can handle
>>>>>>>>>>>>>>>>> duplicates
>>>>>>>>>>>>>>>>> consistently. Basically it's not a problem to have
>>>>>>>>>>>>>>>>> duplicate natural keys, everything works fine.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> That heavily depends on how things are implemented. For
>>>>>>>>>>>>>>>> example, we may write a bunch of code that generates internal data
>>>>>>>>>>>>>>>> structures based on this expectation. If we have to support duplicate
>>>>>>>>>>>>>>>> matches, all of sudden we can no longer size various data structures to
>>>>>>>>>>>>>>>> improve performance and may be unable to preallocate memory associated with
>>>>>>>>>>>>>>>> a guaranteed completion.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Again we need to operate on the assumption that this is a
>>>>>>>>>>>>>>>> large scale distributed compute/remote storage scenario. Key matching is
>>>>>>>>>>>>>>>> done with shuffles with data movement across the network, such
>>>>>>>>>>>>>>>> optimizations would really have little impact on overall performance. Not
>>>>>>>>>>>>>>>> to mention that most query engines would already optimize the shuffle
>>>>>>>>>>>>>>>> already as much as it can be optimized.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It is true that if actual duplicate keys would make the key
>>>>>>>>>>>>>>>> matching join (anti-join) somewhat more expensive, however it can be done
>>>>>>>>>>>>>>>> in such a way that if the keys are in practice unique the join is as
>>>>>>>>>>>>>>>> efficient as it can be.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Let me try and clarify each point:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - lookup for query or update on a
>>>>>>>>>>>>>>>>> non-(partition/bucket/sort) key predicate implies scanning large amounts of
>>>>>>>>>>>>>>>>> data - because these are the only data structures that can narrow down the
>>>>>>>>>>>>>>>>> lookup, right ? One could argue that the min/max index (file skipping) can
>>>>>>>>>>>>>>>>> be applied to any column, but in reality if that column is not sorted the
>>>>>>>>>>>>>>>>> min/max intervals can have huge overlaps so it may be next to useless.
>>>>>>>>>>>>>>>>> - remote storage - this is a critical architecture
>>>>>>>>>>>>>>>>> decision - implementations on local storage imply a vastly different design
>>>>>>>>>>>>>>>>> for the entire system, storage and compute.
>>>>>>>>>>>>>>>>> - deleting single records per snapshot is unfeasible in
>>>>>>>>>>>>>>>>> eager but also particularly in the lazy design: each deletion creates a
>>>>>>>>>>>>>>>>> very small snapshot. Deleting 1 million records one at a time would create
>>>>>>>>>>>>>>>>> 1 million small files, and 1 million RPC calls.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Why is this unfeasible? If I have a dataset of 100mm files
>>>>>>>>>>>>>>>> including 1mm small files, is that a major problem? It seems like your
>>>>>>>>>>>>>>>> usecase isn't one where you want to support single record deletes but it is
>>>>>>>>>>>>>>>> definitely something important to many people.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 100 mm total files or 1 mm files per dataset is definitely
>>>>>>>>>>>>>>>> a problem on HDFS, and I believe on S3 too. Single key delete would work
>>>>>>>>>>>>>>>> just fine, but it's simply not optimal to do that on remote storage. This
>>>>>>>>>>>>>>>> is a very well known problem with HDFS, and one of the very reasons to have
>>>>>>>>>>>>>>>> something like Iceberg in the first place.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Basically the users would be able to do single key
>>>>>>>>>>>>>>>> mutation, but it's not the use case we should be optimizing for, but it's
>>>>>>>>>>>>>>>> really not advisable.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Eager is conceptually just lazy + compaction done, well,
>>>>>>>>>>>>>>>>> eagerly. The logic for both is exactly the same, the trade-off is just that
>>>>>>>>>>>>>>>>> with eager you implicitly compact every time so that you don't do any work
>>>>>>>>>>>>>>>>> on read, while with lazy
>>>>>>>>>>>>>>>>> you want to amortize the cost of compaction over multiple
>>>>>>>>>>>>>>>>> snapshots.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Basically there should be no difference between the two
>>>>>>>>>>>>>>>>> conceptually, or with regard to keys, etc. The only difference is some
>>>>>>>>>>>>>>>>> mechanics in implementation.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I think you have deconstruct the problem too much to say
>>>>>>>>>>>>>>>> these are the same (or at least that is what I'm starting to think given
>>>>>>>>>>>>>>>> this thread). It seems like real world implementation decisions (per our
>>>>>>>>>>>>>>>> discussion here) are in conflict. For example, you just argued against
>>>>>>>>>>>>>>>> having a 1mm arbitrary mutations but I think that is because you aren't
>>>>>>>>>>>>>>>> thinking about things over time with a delta implementation. Having 10,000
>>>>>>>>>>>>>>>> mutations a day where we do delta compaction once a week
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> and local file mappings (key to offset sparse bitmaps)
>>>>>>>>>>>>>>>> seems like it could result in very good performance in a case where we're
>>>>>>>>>>>>>>>> mutating small amounts of data. In this scenario, you may not do major
>>>>>>>>>>>>>>>> compaction ever unless you get to a high enough percentage of records that
>>>>>>>>>>>>>>>> have been deleted in the original dataset. That drives a very different set
>>>>>>>>>>>>>>>> of implementation decisions from a situation where you're trying to restate
>>>>>>>>>>>>>>>> an entire partition at once.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We operate on 1 billion mutations per day at least. This is
>>>>>>>>>>>>>>>> the problem Iceberg wants to solve, I believe it's stated upfront.
>>>>>>>>>>>>>>>> 10000/day is not a big data problem. It can be done fairly trivially and it
>>>>>>>>>>>>>>>> would be supported, but there's not much point in extra optimizing for this
>>>>>>>>>>>>>>>> use case I believe.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>> Netflix
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>> Netflix
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Ryan Blue
>>>>>>>>>>> Software Engineer
>>>>>>>>>>> Netflix
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Ryan Blue
>>>>>>>> Software Engineer
>>>>>>>> Netflix
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Ryan Blue
>>>>>>> Software Engineer
>>>>>>> Netflix
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Ryan Blue
>>>>> Software Engineer
>>>>> Netflix
>>>>>
>>>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: Updates/Deletes/Upserts in Iceberg

[Ryan Blue <rb...@netflix.com.INVALID>]

Erik, thanks for working on this doc. It’s a good detailed write-up of the
approach using natural keys and I found the section about efficiently
merging changes together to be helpful.

For the specific updates to Iceberg metadata, I think we will need to
update it. Dan and I were thinking about the sequencing problem last week
and came up with a solution to augment what’s in the proposal.
Conceptually, what is in the doc would definitely work, but it hits a few
pitfalls that we’ve already had work around:

   - In early versions Iceberg kept the list of manifests in the table
   metadata (JSON) file that are equivalent to the manifest list for each
   version. That ended up taking too much space, causing high write latency.
   It also required opening too many files at planning time.
   - Maintaining separate manifest lists for each version requires keeping
   versions in separate metadata trees, so we can’t merge manifest files
   together across versions. We initially had this problem when we used only
   fast appends for our streaming writes. We ended up with way too many
   metadata files and job planning took a really long time.
   - Although snapshots are independent and can be aged off, this requires
   rewriting the base version to expire versions. That means we can’t delete
   old data files until versions are rewritten, so the problem affects
   versions instead of snapshots. We would still have the problem of not being
   able to delete old data until after a compaction.

The solution we came up with is to assign each data or delete file a
sequence number. This is basically the same thing as the version numbers in
your doc, so what you described for reading would work from the point where
you have files that are associated with a version number. That requires a
way to assign sequence numbers without needing to rewrite metadata when
commits conflict.

The proposal included sequence numbers in the table metadata file as
versions and avoided rewrites by adding more manifest lists. Building on
that idea, we suggest adding those sequence numbers to each manifest file
in a manifest list. That way, we only have one manifest list file per
snapshot just like the current format. This works because the manifest list
is always rewritten when there is a commit conflict. The proposal avoids
rewriting manifest lists by embedding this information in the table
metadata file, but then each conflict causes you to rewrite the version
list for all snapshots: instead of rewriting 2 files, it rewrites one much
larger file — as I mentioned above, that cost is part of why we introduced
manifest lists.

Because the manifest list is rewritten when commits conflict, the sequence
number of manifests added in that snapshot can be updated. That solves one
problem, but then we would still have the problem that the actual manifests
do not contain sequence numbers so we can’t merge manifests. To solve that,
we propose optionally writing a sequence number with each data file in a
manifest. If a data or delete file doesn’t have a sequence number, it
inherits the sequence number of the manifest on read. Then compacting
manifests just works because data files are guaranteed to have a correct
sequence number when they are read. Also, manifests don’t need to be
rewritten during a conflict because new manifests are written without a
sequence number for each data file: the data files inherit the sequence
number that is set for the manifest when it is committed.

Using this strategy, we can store all of the version information from the
proposal as sequence numbers in the existing metadata structures, with
minimal changes to Iceberg’s existing metadata structure. Here are some
benefits to this approach:

   - There is no explicit version that must be expired so compaction is
   only required to optimize reads — partition or file-level delete can
   coexist with deltas
   - When a delete’s sequence number is lower than all data files, it can
   be pruned
   - Compaction can translate a natural key delete to a synthetic key
   delete by adding a synthetic key delete file and updating the data file’s
   sequence number
   - Synthetic and natural key approaches can coexist
   - More of the existing Iceberg operations and structure are reused —
   less code is required to get this working

I think by combining your ideas with the sequence number approach, we’re
fairly close to something we can start building and I’m confident that it
would perform well.

I’m happy to reply with updates if what I’m proposing isn’t clear enough to
understand. We can also set up another sync-up to talk this through if
everyone is interested.

rb

On Fri, Jun 7, 2019 at 11:47 AM Ryan Blue <rb...@netflix.com> wrote:

> Thanks, Erik! Great to see progress here. I'll set aside some time to look
> this over in detail.
>
> On Fri, Jun 7, 2019 at 11:46 AM Erik Wright <er...@shopify.com>
> wrote:
>
>> I apologize for the delay, but I have finally put together a document
>> describing an alternative approach to supporting updates in Iceberg while
>> minimizing write amplification.
>>
>> Proposal: Iceberg Merge-on-Read
>> <https://docs.google.com/document/d/1KuOMeS8Hw_yuE5IXtII8EClJlEMtqFECfWbg-gN5lsQ/edit?usp=sharing>
>>
>> Thank you, Anton and Miguel, for starting this conversation, and everyone
>> else as well for the ongoing dialogue. I'm looking forward to continuing to
>> discuss this and hopefully finding an approach that can meet our different
>> needs and that we can work together on.
>>
>> Cheers,
>>
>> Erik
>>
>> On Wed, May 29, 2019 at 7:26 PM Jacques Nadeau <ja...@dremio.com>
>> wrote:
>>
>>> Yeah, I totally forgot to record our discussion. Will do so next time,
>>> sorry.
>>> --
>>> Jacques Nadeau
>>> CTO and Co-Founder, Dremio
>>>
>>>
>>> On Wed, May 29, 2019 at 4:24 PM Ryan Blue <rb...@netflix.com> wrote:
>>>
>>>> It wasn't recorded, but I can summarize what we talked about. Sorry I
>>>> haven't sent this out earlier.
>>>>
>>>> We talked about the options and some of the background in Iceberg --
>>>> basically that it isn't possible to determine the order of commits before
>>>> you commit so you can't rely on some monotonically increasing value from a
>>>> snapshot to know which deltas to apply to a file. The result is that we
>>>> can't apply diffs to data files using a rule like "files older than X"
>>>> because we can't identify those files without the snapshot history.
>>>>
>>>> That gives us basically 2 options for scoping delete diffs: either
>>>> identify the files to apply a diff to when writing the diff, or log changes
>>>> applied to a snapshot and keep the snapshot history around (which is how we
>>>> know the order of snapshots). The first option is not good if you want to
>>>> write without reading data to determine where the deleted records are. The
>>>> second prevents cleaning up snapshot history.
>>>>
>>>> We also talked about whether we should encode IDs in data files.
>>>> Jacques pointed out that retrying a commit is easier if you don't need to
>>>> re-read the original data to reconcile changes. For example, if a data file
>>>> was compacted in a concurrent write, how do we reconcile a delete for it?
>>>> We discussed other options, like rolling back the compaction for delete
>>>> events. I think that's a promising option.
>>>>
>>>> For action items, Jacques was going to think about whether we need to
>>>> encode IDs in data files or if we could use positions to identify rows and
>>>> write up a summary/proposal. Erik was going to take on planning how
>>>> identifying rows without reading data would work and similarly write up a
>>>> summary/proposal.
>>>>
>>>> That's from memory, so if I've missed anything, I hope that other
>>>> attendees will fill in the details!
>>>>
>>>> rb
>>>>
>>>> On Wed, May 29, 2019 at 3:34 PM Venkatakrishnan Sowrirajan <
>>>> vsowrira@asu.edu> wrote:
>>>>
>>>>> Hi Ryan,
>>>>>
>>>>> I couldn't attend the meeting. Just curious, if this is recorded by
>>>>> any chance.
>>>>>
>>>>> Regards
>>>>> Venkata krishnan
>>>>>
>>>>>
>>>>> On Fri, May 24, 2019 at 8:49 AM Ryan Blue <rb...@netflix.com.invalid>
>>>>> wrote:
>>>>>
>>>>>> Yes, I agree. I'll talk a little about a couple of the constraints of
>>>>>> this as well.
>>>>>>
>>>>>> On Fri, May 24, 2019 at 5:52 AM Anton Okolnychyi <
>>>>>> aokolnychyi@apple.com> wrote:
>>>>>>
>>>>>>> The agenda looks good to me. I think it would also make sense to
>>>>>>> clarify the responsibilities of query engines and Iceberg. Not only in
>>>>>>> terms of uniqueness, but also in terms of applying diffs on read, for
>>>>>>> example.
>>>>>>>
>>>>>>> On 23 May 2019, at 01:59, Ryan Blue <rb...@netflix.com.INVALID>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Here’s a rough agenda:
>>>>>>>
>>>>>>>    - Use cases: everyone come with a use case that you’d like to
>>>>>>>    have supported. We’ll go around and introduce ourselves and our use cases.
>>>>>>>    - Main topic: How should Iceberg identify rows that are deleted?
>>>>>>>    - Side topics from my initial email, if we have time: should we
>>>>>>>    use insert diffs, should we support dense and sparse formats, etc.
>>>>>>>
>>>>>>> The main topic I think we should discuss is: *How should Iceberg
>>>>>>> identify rows that are deleted?*
>>>>>>>
>>>>>>> I’m phrasing it this way to avoid where I think we’re talking past
>>>>>>> one another because we are making assumptions. The important thing is that
>>>>>>> there are two main options:
>>>>>>>
>>>>>>>    - Filename and position, vs
>>>>>>>    - Specific values of (few) columns in the data
>>>>>>>
>>>>>>> This phrasing also avoids discussing uniqueness constraints. Once we
>>>>>>> get down to behavior, I think we agree. For example, I think we all agree
>>>>>>> that uniqueness cannot be enforced in Iceberg.
>>>>>>>
>>>>>>> If uniqueness can’t be enforced in Iceberg, the main choice comes
>>>>>>> down to how we identify rows that are deleted. If we use (filename,
>>>>>>> position) then we know that there is only one row. On the other hand, if we
>>>>>>> use data values to identify rows then a delete may identify more than one
>>>>>>> row because there are no uniqueness guarantees. I think we also agree that
>>>>>>> if there is more than one row identified, all of them should be deleted.
>>>>>>>
>>>>>>> At that point, there are trade-offs between the approaches:
>>>>>>>
>>>>>>>    - When identifying deleted rows by data values, situations like
>>>>>>>    the one that Anton pointed out are possible.
>>>>>>>    - Jacques also had a good point about concurrency. If at all
>>>>>>>    possible, we want to be able to reconcile changes between concurrent
>>>>>>>    commits without re-running an operation.
>>>>>>>
>>>>>>> Sound like a reasonable amount to talk through?
>>>>>>>
>>>>>>> rb
>>>>>>>
>>>>>>> On Wed, May 22, 2019 at 1:17 PM Erik Wright <er...@shopify.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, May 22, 2019 at 4:04 PM Cristian Opris <
>>>>>>>> copris@apple.com.invalid> wrote:
>>>>>>>>
>>>>>>>>> Agreed with Erik here, we're certainly not looking to build the
>>>>>>>>> equivalent of a relational database, and for that matter not even that of a
>>>>>>>>> local disk storage analytics database (like Vertica). Those are very
>>>>>>>>> different designs with very different trade-offs and optimizations.
>>>>>>>>>
>>>>>>>>> We're looking to automate and optimize specific types of file
>>>>>>>>> manipulation for large files on remote storage, while presenting that to
>>>>>>>>> the user under the common SQL API for *bulk* data manipulation
>>>>>>>>> (MERGE INTO)
>>>>>>>>>
>>>>>>>>
>>>>>>>> What I would encourage is to decouple the storage model from the
>>>>>>>> implementation of that API. If Iceberg has support for merge-on-read of
>>>>>>>> upserts and deletes, in addition to its powerful support for partitioning,
>>>>>>>> it will be easy for a higher-level application to implement those APIs
>>>>>>>> given certain other constraints (that might not be appropriate to all
>>>>>>>> applications).
>>>>>>>>
>>>>>>>> Myself and Miguel are out on Friday, but Anton should be able to
>>>>>>>>> handle the discussion on our side.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Cristian
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 22 May 2019, at 17:51, Erik Wright <
>>>>>>>>> erik.wright@shopify.com.INVALID> wrote:
>>>>>>>>>
>>>>>>>>> We have two rows with the same natural key and we use that natural
>>>>>>>>>> key in diff files:
>>>>>>>>>> nk | col1 | col2
>>>>>>>>>> 1 | 1 | 1
>>>>>>>>>> 1 | 2 | 2
>>>>>>>>>> Then we have a delete statement:
>>>>>>>>>> DELETE FROM t WHERE col1 = 1
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I think this example cuts to the point of the differences of
>>>>>>>>> understanding. Does Iceberg want to be approaching the utility of a
>>>>>>>>> relational database, against which I can execute complex update queries?
>>>>>>>>> This is not what I would have imagined.
>>>>>>>>>
>>>>>>>>> I would have, instead, imagined that it was up to the client to
>>>>>>>>> identify, through whatever means, that they want to update or delete a row
>>>>>>>>> with a given ID. If there are multiple (distinct) rows with the same ID,
>>>>>>>>> _too bad_. Any user should _expect_ that they could potentially see any one
>>>>>>>>> or more of those rows at read time. And that an upsert/delete would affect
>>>>>>>>> any/all of them (I would argue for all).
>>>>>>>>>
>>>>>>>>> *In summary:* Instead of trying to come up with a consistent,
>>>>>>>>> logical handling for complex queries that are best suited for a relational
>>>>>>>>> database, leave such handling up to the client and concentrate on problems
>>>>>>>>> that can be solved simply and more generally.
>>>>>>>>>
>>>>>>>>> On Wed, May 22, 2019 at 12:11 PM Ryan Blue <
>>>>>>>>> rblue@netflix.com.invalid> wrote:
>>>>>>>>>
>>>>>>>>>> Yes, I think we should. I was going to propose one after catching
>>>>>>>>>> up on the rest of this thread today.
>>>>>>>>>>
>>>>>>>>>> On Wed, May 22, 2019 at 9:08 AM Anton Okolnychyi <
>>>>>>>>>> aokolnychyi@apple.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks! Would it make sense to discuss the agenda in advance?
>>>>>>>>>>>
>>>>>>>>>>> On 22 May 2019, at 17:04, Ryan Blue <rb...@netflix.com.INVALID>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> I sent out an invite and included everyone on this thread. If
>>>>>>>>>>> anyone else would like to join, please join the Zoom meeting. If you'd like
>>>>>>>>>>> to be added to the calendar invite, just let me know and I'll add you.
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 22, 2019 at 8:57 AM Jacques Nadeau <
>>>>>>>>>>> jacques@dremio.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> works for me.
>>>>>>>>>>>>
>>>>>>>>>>>> To make things easier, we can use my zoom meeting if people
>>>>>>>>>>>> like:
>>>>>>>>>>>>
>>>>>>>>>>>> Join Zoom Meeting
>>>>>>>>>>>> https://zoom.us/j/4157302092
>>>>>>>>>>>>
>>>>>>>>>>>> One tap mobile
>>>>>>>>>>>> +16465588656,,4157302092# US (New York)
>>>>>>>>>>>> +16699006833,,4157302092# US (San Jose)
>>>>>>>>>>>>
>>>>>>>>>>>> Dial by your location
>>>>>>>>>>>>         +1 646 558 8656 US (New York)
>>>>>>>>>>>>         +1 669 900 6833 US (San Jose)
>>>>>>>>>>>>         877 853 5257 US Toll-free
>>>>>>>>>>>>         888 475 4499 US Toll-free
>>>>>>>>>>>> Meeting ID: 415 730 2092
>>>>>>>>>>>> Find your local number: https://zoom.us/u/aH9XYBfm
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Jacques Nadeau
>>>>>>>>>>>> CTO and Co-Founder, Dremio
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, May 22, 2019 at 8:54 AM Ryan Blue <
>>>>>>>>>>>> rblue@netflix.com.invalid> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> 9AM on Friday works best for me. How about then?
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, May 22, 2019 at 5:05 AM Anton Okolnychyi <
>>>>>>>>>>>>> aokolnychyi@apple.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> What about this Friday? One hour slot from 9:00 to 10:00 am
>>>>>>>>>>>>>> or 10:00 to 11:00 am PST? Some folks are based in London, so meeting later
>>>>>>>>>>>>>> than this is hard. If Friday doesn’t work, we can consider Tuesday or
>>>>>>>>>>>>>> Wednesday next week.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 22 May 2019, at 00:54, Jacques Nadeau <ja...@dremio.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I agree with Anton that we should probably spend some time on
>>>>>>>>>>>>>> hangouts further discussing things. Definitely differing expectations here
>>>>>>>>>>>>>> and we seem to be talking a bit past each other.
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Jacques Nadeau
>>>>>>>>>>>>>> CTO and Co-Founder, Dremio
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, May 21, 2019 at 3:44 PM Cristian Opris <
>>>>>>>>>>>>>> copris@apple.com.invalid> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I love a good flame war :P
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 21 May 2019, at 22:57, Jacques Nadeau <ja...@dremio.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> That's my point, truly independent writers (two Spark jobs,
>>>>>>>>>>>>>>>> or a Spark job and Dremio job) means a distributed transaction. It would
>>>>>>>>>>>>>>>> need yet another external transaction coordinator on top of both Spark and
>>>>>>>>>>>>>>>> Dremio, Iceberg by itself
>>>>>>>>>>>>>>>> cannot solve this.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm not ready to accept this. Iceberg already supports a set
>>>>>>>>>>>>>>> of semantics around multiple writers committing simultaneously and how
>>>>>>>>>>>>>>> conflict resolution is done. The same can be done here.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> MVCC (which is what Iceberg tries to implement) requires a
>>>>>>>>>>>>>>> total ordering of snapshots. Also the snapshots need to be non-conflicting.
>>>>>>>>>>>>>>> I really don't see how any metadata data structures can solve this without
>>>>>>>>>>>>>>> an outside coordinator.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Consider this:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Snapshot 0: (K,A) = 1
>>>>>>>>>>>>>>> Job X: UPDATE K SET A=A+1
>>>>>>>>>>>>>>> Job Y: UPDATE K SET A=10
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> What should the final value of A be and who decides ?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> By single writer, I don't mean single process, I mean
>>>>>>>>>>>>>>>> multiple coordinated processes like Spark executors coordinated by Spark
>>>>>>>>>>>>>>>> driver. The coordinator ensures that the data is pre-partitioned on
>>>>>>>>>>>>>>>> each executor, and the coordinator commits the snapshot.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Note however that single writer job/multiple concurrent
>>>>>>>>>>>>>>>> reader jobs is perfectly feasible, i.e. it shouldn't be a problem to write
>>>>>>>>>>>>>>>> from a Spark job and read from multiple Dremio queries concurrently (for
>>>>>>>>>>>>>>>> example)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> :D This is still "single process" from my perspective. That
>>>>>>>>>>>>>>> process may be coordinating other processes to do distributed work but
>>>>>>>>>>>>>>> ultimately it is a single process.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Fair enough
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm not sure what you mean exactly. If we can't enforce
>>>>>>>>>>>>>>>> uniqueness we shouldn't assume it.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I disagree. We can specify that as a requirement and state
>>>>>>>>>>>>>>> that you'll get unintended consequences if you provide your own keys and
>>>>>>>>>>>>>>> don't maintain this.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> There's no need for unintended consequences, we can specify
>>>>>>>>>>>>>>> consistent behaviour (and I believe the document says what that is)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We do expect that most of the time the natural key is
>>>>>>>>>>>>>>>> unique, but the eager and lazy with natural key designs can handle
>>>>>>>>>>>>>>>> duplicates
>>>>>>>>>>>>>>>> consistently. Basically it's not a problem to have
>>>>>>>>>>>>>>>> duplicate natural keys, everything works fine.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> That heavily depends on how things are implemented. For
>>>>>>>>>>>>>>> example, we may write a bunch of code that generates internal data
>>>>>>>>>>>>>>> structures based on this expectation. If we have to support duplicate
>>>>>>>>>>>>>>> matches, all of sudden we can no longer size various data structures to
>>>>>>>>>>>>>>> improve performance and may be unable to preallocate memory associated with
>>>>>>>>>>>>>>> a guaranteed completion.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Again we need to operate on the assumption that this is a
>>>>>>>>>>>>>>> large scale distributed compute/remote storage scenario. Key matching is
>>>>>>>>>>>>>>> done with shuffles with data movement across the network, such
>>>>>>>>>>>>>>> optimizations would really have little impact on overall performance. Not
>>>>>>>>>>>>>>> to mention that most query engines would already optimize the shuffle
>>>>>>>>>>>>>>> already as much as it can be optimized.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It is true that if actual duplicate keys would make the key
>>>>>>>>>>>>>>> matching join (anti-join) somewhat more expensive, however it can be done
>>>>>>>>>>>>>>> in such a way that if the keys are in practice unique the join is as
>>>>>>>>>>>>>>> efficient as it can be.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Let me try and clarify each point:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - lookup for query or update on a
>>>>>>>>>>>>>>>> non-(partition/bucket/sort) key predicate implies scanning large amounts of
>>>>>>>>>>>>>>>> data - because these are the only data structures that can narrow down the
>>>>>>>>>>>>>>>> lookup, right ? One could argue that the min/max index (file skipping) can
>>>>>>>>>>>>>>>> be applied to any column, but in reality if that column is not sorted the
>>>>>>>>>>>>>>>> min/max intervals can have huge overlaps so it may be next to useless.
>>>>>>>>>>>>>>>> - remote storage - this is a critical architecture decision
>>>>>>>>>>>>>>>> - implementations on local storage imply a vastly different design for the
>>>>>>>>>>>>>>>> entire system, storage and compute.
>>>>>>>>>>>>>>>> - deleting single records per snapshot is unfeasible in
>>>>>>>>>>>>>>>> eager but also particularly in the lazy design: each deletion creates a
>>>>>>>>>>>>>>>> very small snapshot. Deleting 1 million records one at a time would create
>>>>>>>>>>>>>>>> 1 million small files, and 1 million RPC calls.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Why is this unfeasible? If I have a dataset of 100mm files
>>>>>>>>>>>>>>> including 1mm small files, is that a major problem? It seems like your
>>>>>>>>>>>>>>> usecase isn't one where you want to support single record deletes but it is
>>>>>>>>>>>>>>> definitely something important to many people.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 100 mm total files or 1 mm files per dataset is definitely a
>>>>>>>>>>>>>>> problem on HDFS, and I believe on S3 too. Single key delete would work just
>>>>>>>>>>>>>>> fine, but it's simply not optimal to do that on remote storage. This is a
>>>>>>>>>>>>>>> very well known problem with HDFS, and one of the very reasons to have
>>>>>>>>>>>>>>> something like Iceberg in the first place.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Basically the users would be able to do single key mutation,
>>>>>>>>>>>>>>> but it's not the use case we should be optimizing for, but it's really not
>>>>>>>>>>>>>>> advisable.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Eager is conceptually just lazy + compaction done, well,
>>>>>>>>>>>>>>>> eagerly. The logic for both is exactly the same, the trade-off is just that
>>>>>>>>>>>>>>>> with eager you implicitly compact every time so that you don't do any work
>>>>>>>>>>>>>>>> on read, while with lazy
>>>>>>>>>>>>>>>> you want to amortize the cost of compaction over multiple
>>>>>>>>>>>>>>>> snapshots.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Basically there should be no difference between the two
>>>>>>>>>>>>>>>> conceptually, or with regard to keys, etc. The only difference is some
>>>>>>>>>>>>>>>> mechanics in implementation.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think you have deconstruct the problem too much to say
>>>>>>>>>>>>>>> these are the same (or at least that is what I'm starting to think given
>>>>>>>>>>>>>>> this thread). It seems like real world implementation decisions (per our
>>>>>>>>>>>>>>> discussion here) are in conflict. For example, you just argued against
>>>>>>>>>>>>>>> having a 1mm arbitrary mutations but I think that is because you aren't
>>>>>>>>>>>>>>> thinking about things over time with a delta implementation. Having 10,000
>>>>>>>>>>>>>>> mutations a day where we do delta compaction once a week
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> and local file mappings (key to offset sparse bitmaps) seems
>>>>>>>>>>>>>>> like it could result in very good performance in a case where we're
>>>>>>>>>>>>>>> mutating small amounts of data. In this scenario, you may not do major
>>>>>>>>>>>>>>> compaction ever unless you get to a high enough percentage of records that
>>>>>>>>>>>>>>> have been deleted in the original dataset. That drives a very different set
>>>>>>>>>>>>>>> of implementation decisions from a situation where you're trying to restate
>>>>>>>>>>>>>>> an entire partition at once.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We operate on 1 billion mutations per day at least. This is
>>>>>>>>>>>>>>> the problem Iceberg wants to solve, I believe it's stated upfront.
>>>>>>>>>>>>>>> 10000/day is not a big data problem. It can be done fairly trivially and it
>>>>>>>>>>>>>>> would be supported, but there's not much point in extra optimizing for this
>>>>>>>>>>>>>>> use case I believe.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>> Netflix
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Ryan Blue
>>>>>>>>>>> Software Engineer
>>>>>>>>>>> Netflix
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Ryan Blue
>>>>>>>>>> Software Engineer
>>>>>>>>>> Netflix
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Ryan Blue
>>>>>>> Software Engineer
>>>>>>> Netflix
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Ryan Blue
>>>>>> Software Engineer
>>>>>> Netflix
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Ryan Blue
>>>> Software Engineer
>>>> Netflix
>>>>
>>>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>


-- 
Ryan Blue
Software Engineer
Netflix

Re: Updates/Deletes/Upserts in Iceberg

[Erik Wright <er...@shopify.com.INVALID>]

Hi Cristian,

I'm glad to hear that this feels like alignment to you. I agree that it
should closely correspond to the option that you described. My intent in
writing this out was to clarify some of the details because they were a
sticking point in the live discussion.

One thing though: guaranteeing uniques doesn't seem possible with large
> datasets. In fact, no real big data system does that (We posted some
> examples in the comment thread on our proposal doc).


Clearly it is "possible" to guarantee uniques by comparing deltas to a base
version during the commit process. But, as you alluded to, and as I mention
in the document, the cost of doing so is highly dependent on the nature of
the dataset and the updates to it. Furthermore, the likelihood of
duplicates is highly variable, as is the cost of violations.

With all of those things in mind, I have recommended making this
enforcement optional (if available at all). Instead I have chosen to
describe the potential impact of errors in the dataset production. I
identified two possible errors and illustrated how those errors might be
observed by a consumer.

The good news however, is we don't need uniqueness for all of this to work.
> It works just the same even if there are duplicates, however with a bit
> more care in implementation to handle
> some corner cases consistently.


Agreed, that under certain conditions, deterministic behaviour can be
achieved in the face of unintended duplicates in input. And this would be
my recommended approach.

Thanks for taking the time to read my document.

Erik

On Fri, Jun 7, 2019 at 3:27 PM Cristian Opris <co...@apple.com> wrote:

> Hey Erik,
>
> I was able to parse your proposal quite quickly since it seems to be
> pretty much exactly the "lazy with natural key" in our proposed approach.
>
>
> I must say it is however much concretely expressed.
>
> This natural key approach is in fact our preferred option too, even though
> it wasn't explicitly said in the document. I guess we wanted to get more
> feedback just in case we're missing something. Turns out we weren't :)
>
> In particular, I fully agree with this:
>
> "The effective dataset of any other version is the equivalent of the
> previous version, anti-join the deletes in the current version (on the
> unique row identifier), union the insertions in the current version.
> Optimal performance will be achieved when consistent partition
> specifications and sorting are used from one version to the next
> (minimizing the number of deletes that need to be compared to each
> insertion)."
>
> Yes, nothing more than sorted natural keys is needed to achieve a good
> implementation that covers all use cases.
>
> One thing though: guaranteeing uniques doesn't seem possible with large
> datasets. In fact, no real big data system does that (We posted some
> examples in the comment thread on our proposal doc).
>
> It's not clear from your doc how enforcing uniqueness would work ?
>
> The good news however, is we don't need uniqueness for all of this to
> work. It works just the same even if there are duplicates, however with a
> bit more care in implementation to handle
> some corner cases consistently.
>
> Let me know if I got the gist of your proposal right ?
>
> Thanks,
> Cristian
>
>
> On 7 Jun 2019, at 19:47, Ryan Blue <rb...@netflix.com.INVALID> wrote:
>
> Thanks, Erik! Great to see progress here. I'll set aside some time to look
> this over in detail.
>
> On Fri, Jun 7, 2019 at 11:46 AM Erik Wright <er...@shopify.com>
> wrote:
>
>> I apologize for the delay, but I have finally put together a document
>> describing an alternative approach to supporting updates in Iceberg while
>> minimizing write amplification.
>>
>> Proposal: Iceberg Merge-on-Read
>> <https://docs.google.com/document/d/1KuOMeS8Hw_yuE5IXtII8EClJlEMtqFECfWbg-gN5lsQ/edit?usp=sharing>
>>
>> Thank you, Anton and Miguel, for starting this conversation, and everyone
>> else as well for the ongoing dialogue. I'm looking forward to continuing to
>> discuss this and hopefully finding an approach that can meet our different
>> needs and that we can work together on.
>>
>> Cheers,
>>
>> Erik
>>
>> On Wed, May 29, 2019 at 7:26 PM Jacques Nadeau <ja...@dremio.com>
>> wrote:
>>
>>> Yeah, I totally forgot to record our discussion. Will do so next time,
>>> sorry.
>>> --
>>> Jacques Nadeau
>>> CTO and Co-Founder, Dremio
>>>
>>>
>>> On Wed, May 29, 2019 at 4:24 PM Ryan Blue <rb...@netflix.com> wrote:
>>>
>>>> It wasn't recorded, but I can summarize what we talked about. Sorry I
>>>> haven't sent this out earlier.
>>>>
>>>> We talked about the options and some of the background in Iceberg --
>>>> basically that it isn't possible to determine the order of commits before
>>>> you commit so you can't rely on some monotonically increasing value from a
>>>> snapshot to know which deltas to apply to a file. The result is that we
>>>> can't apply diffs to data files using a rule like "files older than X"
>>>> because we can't identify those files without the snapshot history.
>>>>
>>>> That gives us basically 2 options for scoping delete diffs: either
>>>> identify the files to apply a diff to when writing the diff, or log changes
>>>> applied to a snapshot and keep the snapshot history around (which is how we
>>>> know the order of snapshots). The first option is not good if you want to
>>>> write without reading data to determine where the deleted records are. The
>>>> second prevents cleaning up snapshot history.
>>>>
>>>> We also talked about whether we should encode IDs in data files.
>>>> Jacques pointed out that retrying a commit is easier if you don't need to
>>>> re-read the original data to reconcile changes. For example, if a data file
>>>> was compacted in a concurrent write, how do we reconcile a delete for it?
>>>> We discussed other options, like rolling back the compaction for delete
>>>> events. I think that's a promising option.
>>>>
>>>> For action items, Jacques was going to think about whether we need to
>>>> encode IDs in data files or if we could use positions to identify rows and
>>>> write up a summary/proposal. Erik was going to take on planning how
>>>> identifying rows without reading data would work and similarly write up a
>>>> summary/proposal.
>>>>
>>>> That's from memory, so if I've missed anything, I hope that other
>>>> attendees will fill in the details!
>>>>
>>>> rb
>>>>
>>>> On Wed, May 29, 2019 at 3:34 PM Venkatakrishnan Sowrirajan <
>>>> vsowrira@asu.edu> wrote:
>>>>
>>>>> Hi Ryan,
>>>>>
>>>>> I couldn't attend the meeting. Just curious, if this is recorded by
>>>>> any chance.
>>>>>
>>>>> Regards
>>>>> Venkata krishnan
>>>>>
>>>>>
>>>>> On Fri, May 24, 2019 at 8:49 AM Ryan Blue <rb...@netflix.com.invalid>
>>>>> wrote:
>>>>>
>>>>>> Yes, I agree. I'll talk a little about a couple of the constraints of
>>>>>> this as well.
>>>>>>
>>>>>> On Fri, May 24, 2019 at 5:52 AM Anton Okolnychyi <
>>>>>> aokolnychyi@apple.com> wrote:
>>>>>>
>>>>>>> The agenda looks good to me. I think it would also make sense to
>>>>>>> clarify the responsibilities of query engines and Iceberg. Not only in
>>>>>>> terms of uniqueness, but also in terms of applying diffs on read, for
>>>>>>> example.
>>>>>>>
>>>>>>> On 23 May 2019, at 01:59, Ryan Blue <rb...@netflix.com.INVALID>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Here’s a rough agenda:
>>>>>>>
>>>>>>>    - Use cases: everyone come with a use case that you’d like to
>>>>>>>    have supported. We’ll go around and introduce ourselves and our use cases.
>>>>>>>    - Main topic: How should Iceberg identify rows that are deleted?
>>>>>>>    - Side topics from my initial email, if we have time: should we
>>>>>>>    use insert diffs, should we support dense and sparse formats, etc.
>>>>>>>
>>>>>>> The main topic I think we should discuss is: *How should Iceberg
>>>>>>> identify rows that are deleted?*
>>>>>>>
>>>>>>> I’m phrasing it this way to avoid where I think we’re talking past
>>>>>>> one another because we are making assumptions. The important thing is that
>>>>>>> there are two main options:
>>>>>>>
>>>>>>>    - Filename and position, vs
>>>>>>>    - Specific values of (few) columns in the data
>>>>>>>
>>>>>>> This phrasing also avoids discussing uniqueness constraints. Once we
>>>>>>> get down to behavior, I think we agree. For example, I think we all agree
>>>>>>> that uniqueness cannot be enforced in Iceberg.
>>>>>>>
>>>>>>> If uniqueness can’t be enforced in Iceberg, the main choice comes
>>>>>>> down to how we identify rows that are deleted. If we use (filename,
>>>>>>> position) then we know that there is only one row. On the other hand, if we
>>>>>>> use data values to identify rows then a delete may identify more than one
>>>>>>> row because there are no uniqueness guarantees. I think we also agree that
>>>>>>> if there is more than one row identified, all of them should be deleted.
>>>>>>>
>>>>>>> At that point, there are trade-offs between the approaches:
>>>>>>>
>>>>>>>    - When identifying deleted rows by data values, situations like
>>>>>>>    the one that Anton pointed out are possible.
>>>>>>>    - Jacques also had a good point about concurrency. If at all
>>>>>>>    possible, we want to be able to reconcile changes between concurrent
>>>>>>>    commits without re-running an operation.
>>>>>>>
>>>>>>> Sound like a reasonable amount to talk through?
>>>>>>>
>>>>>>> rb
>>>>>>>
>>>>>>> On Wed, May 22, 2019 at 1:17 PM Erik Wright <er...@shopify.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, May 22, 2019 at 4:04 PM Cristian Opris <
>>>>>>>> copris@apple.com.invalid> wrote:
>>>>>>>>
>>>>>>>>> Agreed with Erik here, we're certainly not looking to build the
>>>>>>>>> equivalent of a relational database, and for that matter not even that of a
>>>>>>>>> local disk storage analytics database (like Vertica). Those are very
>>>>>>>>> different designs with very different trade-offs and optimizations.
>>>>>>>>>
>>>>>>>>> We're looking to automate and optimize specific types of file
>>>>>>>>> manipulation for large files on remote storage, while presenting that to
>>>>>>>>> the user under the common SQL API for *bulk* data manipulation
>>>>>>>>> (MERGE INTO)
>>>>>>>>>
>>>>>>>>
>>>>>>>> What I would encourage is to decouple the storage model from the
>>>>>>>> implementation of that API. If Iceberg has support for merge-on-read of
>>>>>>>> upserts and deletes, in addition to its powerful support for partitioning,
>>>>>>>> it will be easy for a higher-level application to implement those APIs
>>>>>>>> given certain other constraints (that might not be appropriate to all
>>>>>>>> applications).
>>>>>>>>
>>>>>>>> Myself and Miguel are out on Friday, but Anton should be able to
>>>>>>>>> handle the discussion on our side.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Cristian
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 22 May 2019, at 17:51, Erik Wright <
>>>>>>>>> erik.wright@shopify.com.INVALID> wrote:
>>>>>>>>>
>>>>>>>>> We have two rows with the same natural key and we use that natural
>>>>>>>>>> key in diff files:
>>>>>>>>>> nk | col1 | col2
>>>>>>>>>> 1 | 1 | 1
>>>>>>>>>> 1 | 2 | 2
>>>>>>>>>> Then we have a delete statement:
>>>>>>>>>> DELETE FROM t WHERE col1 = 1
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I think this example cuts to the point of the differences of
>>>>>>>>> understanding. Does Iceberg want to be approaching the utility of a
>>>>>>>>> relational database, against which I can execute complex update queries?
>>>>>>>>> This is not what I would have imagined.
>>>>>>>>>
>>>>>>>>> I would have, instead, imagined that it was up to the client to
>>>>>>>>> identify, through whatever means, that they want to update or delete a row
>>>>>>>>> with a given ID. If there are multiple (distinct) rows with the same ID,
>>>>>>>>> _too bad_. Any user should _expect_ that they could potentially see any one
>>>>>>>>> or more of those rows at read time. And that an upsert/delete would affect
>>>>>>>>> any/all of them (I would argue for all).
>>>>>>>>>
>>>>>>>>> *In summary:* Instead of trying to come up with a consistent,
>>>>>>>>> logical handling for complex queries that are best suited for a relational
>>>>>>>>> database, leave such handling up to the client and concentrate on problems
>>>>>>>>> that can be solved simply and more generally.
>>>>>>>>>
>>>>>>>>> On Wed, May 22, 2019 at 12:11 PM Ryan Blue <
>>>>>>>>> rblue@netflix.com.invalid> wrote:
>>>>>>>>>
>>>>>>>>>> Yes, I think we should. I was going to propose one after catching
>>>>>>>>>> up on the rest of this thread today.
>>>>>>>>>>
>>>>>>>>>> On Wed, May 22, 2019 at 9:08 AM Anton Okolnychyi <
>>>>>>>>>> aokolnychyi@apple.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks! Would it make sense to discuss the agenda in advance?
>>>>>>>>>>>
>>>>>>>>>>> On 22 May 2019, at 17:04, Ryan Blue <rb...@netflix.com.INVALID>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> I sent out an invite and included everyone on this thread. If
>>>>>>>>>>> anyone else would like to join, please join the Zoom meeting. If you'd like
>>>>>>>>>>> to be added to the calendar invite, just let me know and I'll add you.
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 22, 2019 at 8:57 AM Jacques Nadeau <
>>>>>>>>>>> jacques@dremio.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> works for me.
>>>>>>>>>>>>
>>>>>>>>>>>> To make things easier, we can use my zoom meeting if people
>>>>>>>>>>>> like:
>>>>>>>>>>>>
>>>>>>>>>>>> Join Zoom Meeting
>>>>>>>>>>>> https://zoom.us/j/4157302092
>>>>>>>>>>>>
>>>>>>>>>>>> One tap mobile
>>>>>>>>>>>> +16465588656,,4157302092# US (New York)
>>>>>>>>>>>> +16699006833,,4157302092# US (San Jose)
>>>>>>>>>>>>
>>>>>>>>>>>> Dial by your location
>>>>>>>>>>>>         +1 646 558 8656 US (New York)
>>>>>>>>>>>>         +1 669 900 6833 US (San Jose)
>>>>>>>>>>>>         877 853 5257 US Toll-free
>>>>>>>>>>>>         888 475 4499 US Toll-free
>>>>>>>>>>>> Meeting ID: 415 730 2092
>>>>>>>>>>>> Find your local number: https://zoom.us/u/aH9XYBfm
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Jacques Nadeau
>>>>>>>>>>>> CTO and Co-Founder, Dremio
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, May 22, 2019 at 8:54 AM Ryan Blue <
>>>>>>>>>>>> rblue@netflix.com.invalid> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> 9AM on Friday works best for me. How about then?
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, May 22, 2019 at 5:05 AM Anton Okolnychyi <
>>>>>>>>>>>>> aokolnychyi@apple.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> What about this Friday? One hour slot from 9:00 to 10:00 am
>>>>>>>>>>>>>> or 10:00 to 11:00 am PST? Some folks are based in London, so meeting later
>>>>>>>>>>>>>> than this is hard. If Friday doesn’t work, we can consider Tuesday or
>>>>>>>>>>>>>> Wednesday next week.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 22 May 2019, at 00:54, Jacques Nadeau <ja...@dremio.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I agree with Anton that we should probably spend some time on
>>>>>>>>>>>>>> hangouts further discussing things. Definitely differing expectations here
>>>>>>>>>>>>>> and we seem to be talking a bit past each other.
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Jacques Nadeau
>>>>>>>>>>>>>> CTO and Co-Founder, Dremio
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, May 21, 2019 at 3:44 PM Cristian Opris <
>>>>>>>>>>>>>> copris@apple.com.invalid> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I love a good flame war :P
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 21 May 2019, at 22:57, Jacques Nadeau <ja...@dremio.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> That's my point, truly independent writers (two Spark jobs,
>>>>>>>>>>>>>>>> or a Spark job and Dremio job) means a distributed transaction. It would
>>>>>>>>>>>>>>>> need yet another external transaction coordinator on top of both Spark and
>>>>>>>>>>>>>>>> Dremio, Iceberg by itself
>>>>>>>>>>>>>>>> cannot solve this.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm not ready to accept this. Iceberg already supports a set
>>>>>>>>>>>>>>> of semantics around multiple writers committing simultaneously and how
>>>>>>>>>>>>>>> conflict resolution is done. The same can be done here.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> MVCC (which is what Iceberg tries to implement) requires a
>>>>>>>>>>>>>>> total ordering of snapshots. Also the snapshots need to be non-conflicting.
>>>>>>>>>>>>>>> I really don't see how any metadata data structures can solve this without
>>>>>>>>>>>>>>> an outside coordinator.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Consider this:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Snapshot 0: (K,A) = 1
>>>>>>>>>>>>>>> Job X: UPDATE K SET A=A+1
>>>>>>>>>>>>>>> Job Y: UPDATE K SET A=10
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> What should the final value of A be and who decides ?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> By single writer, I don't mean single process, I mean
>>>>>>>>>>>>>>>> multiple coordinated processes like Spark executors coordinated by Spark
>>>>>>>>>>>>>>>> driver. The coordinator ensures that the data is pre-partitioned on
>>>>>>>>>>>>>>>> each executor, and the coordinator commits the snapshot.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Note however that single writer job/multiple concurrent
>>>>>>>>>>>>>>>> reader jobs is perfectly feasible, i.e. it shouldn't be a problem to write
>>>>>>>>>>>>>>>> from a Spark job and read from multiple Dremio queries concurrently (for
>>>>>>>>>>>>>>>> example)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> :D This is still "single process" from my perspective. That
>>>>>>>>>>>>>>> process may be coordinating other processes to do distributed work but
>>>>>>>>>>>>>>> ultimately it is a single process.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Fair enough
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm not sure what you mean exactly. If we can't enforce
>>>>>>>>>>>>>>>> uniqueness we shouldn't assume it.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I disagree. We can specify that as a requirement and state
>>>>>>>>>>>>>>> that you'll get unintended consequences if you provide your own keys and
>>>>>>>>>>>>>>> don't maintain this.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> There's no need for unintended consequences, we can specify
>>>>>>>>>>>>>>> consistent behaviour (and I believe the document says what that is)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We do expect that most of the time the natural key is
>>>>>>>>>>>>>>>> unique, but the eager and lazy with natural key designs can handle
>>>>>>>>>>>>>>>> duplicates
>>>>>>>>>>>>>>>> consistently. Basically it's not a problem to have
>>>>>>>>>>>>>>>> duplicate natural keys, everything works fine.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> That heavily depends on how things are implemented. For
>>>>>>>>>>>>>>> example, we may write a bunch of code that generates internal data
>>>>>>>>>>>>>>> structures based on this expectation. If we have to support duplicate
>>>>>>>>>>>>>>> matches, all of sudden we can no longer size various data structures to
>>>>>>>>>>>>>>> improve performance and may be unable to preallocate memory associated with
>>>>>>>>>>>>>>> a guaranteed completion.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Again we need to operate on the assumption that this is a
>>>>>>>>>>>>>>> large scale distributed compute/remote storage scenario. Key matching is
>>>>>>>>>>>>>>> done with shuffles with data movement across the network, such
>>>>>>>>>>>>>>> optimizations would really have little impact on overall performance. Not
>>>>>>>>>>>>>>> to mention that most query engines would already optimize the shuffle
>>>>>>>>>>>>>>> already as much as it can be optimized.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It is true that if actual duplicate keys would make the key
>>>>>>>>>>>>>>> matching join (anti-join) somewhat more expensive, however it can be done
>>>>>>>>>>>>>>> in such a way that if the keys are in practice unique the join is as
>>>>>>>>>>>>>>> efficient as it can be.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Let me try and clarify each point:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - lookup for query or update on a
>>>>>>>>>>>>>>>> non-(partition/bucket/sort) key predicate implies scanning large amounts of
>>>>>>>>>>>>>>>> data - because these are the only data structures that can narrow down the
>>>>>>>>>>>>>>>> lookup, right ? One could argue that the min/max index (file skipping) can
>>>>>>>>>>>>>>>> be applied to any column, but in reality if that column is not sorted the
>>>>>>>>>>>>>>>> min/max intervals can have huge overlaps so it may be next to useless.
>>>>>>>>>>>>>>>> - remote storage - this is a critical architecture decision
>>>>>>>>>>>>>>>> - implementations on local storage imply a vastly different design for the
>>>>>>>>>>>>>>>> entire system, storage and compute.
>>>>>>>>>>>>>>>> - deleting single records per snapshot is unfeasible in
>>>>>>>>>>>>>>>> eager but also particularly in the lazy design: each deletion creates a
>>>>>>>>>>>>>>>> very small snapshot. Deleting 1 million records one at a time would create
>>>>>>>>>>>>>>>> 1 million small files, and 1 million RPC calls.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Why is this unfeasible? If I have a dataset of 100mm files
>>>>>>>>>>>>>>> including 1mm small files, is that a major problem? It seems like your
>>>>>>>>>>>>>>> usecase isn't one where you want to support single record deletes but it is
>>>>>>>>>>>>>>> definitely something important to many people.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 100 mm total files or 1 mm files per dataset is definitely a
>>>>>>>>>>>>>>> problem on HDFS, and I believe on S3 too. Single key delete would work just
>>>>>>>>>>>>>>> fine, but it's simply not optimal to do that on remote storage. This is a
>>>>>>>>>>>>>>> very well known problem with HDFS, and one of the very reasons to have
>>>>>>>>>>>>>>> something like Iceberg in the first place.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Basically the users would be able to do single key mutation,
>>>>>>>>>>>>>>> but it's not the use case we should be optimizing for, but it's really not
>>>>>>>>>>>>>>> advisable.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Eager is conceptually just lazy + compaction done, well,
>>>>>>>>>>>>>>>> eagerly. The logic for both is exactly the same, the trade-off is just that
>>>>>>>>>>>>>>>> with eager you implicitly compact every time so that you don't do any work
>>>>>>>>>>>>>>>> on read, while with lazy
>>>>>>>>>>>>>>>> you want to amortize the cost of compaction over multiple
>>>>>>>>>>>>>>>> snapshots.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Basically there should be no difference between the two
>>>>>>>>>>>>>>>> conceptually, or with regard to keys, etc. The only difference is some
>>>>>>>>>>>>>>>> mechanics in implementation.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think you have deconstruct the problem too much to say
>>>>>>>>>>>>>>> these are the same (or at least that is what I'm starting to think given
>>>>>>>>>>>>>>> this thread). It seems like real world implementation decisions (per our
>>>>>>>>>>>>>>> discussion here) are in conflict. For example, you just argued against
>>>>>>>>>>>>>>> having a 1mm arbitrary mutations but I think that is because you aren't
>>>>>>>>>>>>>>> thinking about things over time with a delta implementation. Having 10,000
>>>>>>>>>>>>>>> mutations a day where we do delta compaction once a week
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> and local file mappings (key to offset sparse bitmaps) seems
>>>>>>>>>>>>>>> like it could result in very good performance in a case where we're
>>>>>>>>>>>>>>> mutating small amounts of data. In this scenario, you may not do major
>>>>>>>>>>>>>>> compaction ever unless you get to a high enough percentage of records that
>>>>>>>>>>>>>>> have been deleted in the original dataset. That drives a very different set
>>>>>>>>>>>>>>> of implementation decisions from a situation where you're trying to restate
>>>>>>>>>>>>>>> an entire partition at once.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We operate on 1 billion mutations per day at least. This is
>>>>>>>>>>>>>>> the problem Iceberg wants to solve, I believe it's stated upfront.
>>>>>>>>>>>>>>> 10000/day is not a big data problem. It can be done fairly trivially and it
>>>>>>>>>>>>>>> would be supported, but there's not much point in extra optimizing for this
>>>>>>>>>>>>>>> use case I believe.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>> Netflix
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Ryan Blue
>>>>>>>>>>> Software Engineer
>>>>>>>>>>> Netflix
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Ryan Blue
>>>>>>>>>> Software Engineer
>>>>>>>>>> Netflix
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Ryan Blue
>>>>>>> Software Engineer
>>>>>>> Netflix
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Ryan Blue
>>>>>> Software Engineer
>>>>>> Netflix
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Ryan Blue
>>>> Software Engineer
>>>> Netflix
>>>>
>>>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>
>
>

Re: Updates/Deletes/Upserts in Iceberg

[Cristian Opris <co...@apple.com.INVALID>]

Hey Erik,

I was able to parse your proposal quite quickly since it seems to be pretty much exactly the "lazy with natural key" in our proposed approach.


I must say it is however much concretely expressed.

This natural key approach is in fact our preferred option too, even though it wasn't explicitly said in the document. I guess we wanted to get more feedback just in case we're missing something. Turns out we weren't :)

In particular, I fully agree with this:

"The effective dataset of any other version is the equivalent of the previous version, anti-join the deletes in the current version (on the unique row identifier), union the insertions in the current version. Optimal performance will be achieved when consistent partition specifications and sorting are used from one version to the next (minimizing the number of deletes that need to be compared to each insertion)."

Yes, nothing more than sorted natural keys is needed to achieve a good implementation that covers all use cases.

One thing though: guaranteeing uniques doesn't seem possible with large datasets. In fact, no real big data system does that (We posted some examples in the comment thread on our proposal doc).

It's not clear from your doc how enforcing uniqueness would work ?

The good news however, is we don't need uniqueness for all of this to work. It works just the same even if there are duplicates, however with a bit more care in implementation to handle 
some corner cases consistently.

 Let me know if I got the gist of your proposal right ?

Thanks,
Cristian


> On 7 Jun 2019, at 19:47, Ryan Blue <rb...@netflix.com.INVALID> wrote:
> 
> Thanks, Erik! Great to see progress here. I'll set aside some time to look this over in detail.
> 
> On Fri, Jun 7, 2019 at 11:46 AM Erik Wright <erik.wright@shopify.com <ma...@shopify.com>> wrote:
> I apologize for the delay, but I have finally put together a document describing an alternative approach to supporting updates in Iceberg while minimizing write amplification.
> 
> Proposal: Iceberg Merge-on-Read <https://docs.google.com/document/d/1KuOMeS8Hw_yuE5IXtII8EClJlEMtqFECfWbg-gN5lsQ/edit?usp=sharing>
> 
> Thank you, Anton and Miguel, for starting this conversation, and everyone else as well for the ongoing dialogue. I'm looking forward to continuing to discuss this and hopefully finding an approach that can meet our different needs and that we can work together on.
> 
> Cheers,
> 
> Erik
> 
> On Wed, May 29, 2019 at 7:26 PM Jacques Nadeau <jacques@dremio.com <ma...@dremio.com>> wrote:
> Yeah, I totally forgot to record our discussion. Will do so next time, sorry.
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
> 
> 
> On Wed, May 29, 2019 at 4:24 PM Ryan Blue <rblue@netflix.com <ma...@netflix.com>> wrote:
> It wasn't recorded, but I can summarize what we talked about. Sorry I haven't sent this out earlier.
> 
> We talked about the options and some of the background in Iceberg -- basically that it isn't possible to determine the order of commits before you commit so you can't rely on some monotonically increasing value from a snapshot to know which deltas to apply to a file. The result is that we can't apply diffs to data files using a rule like "files older than X" because we can't identify those files without the snapshot history.
> 
> That gives us basically 2 options for scoping delete diffs: either identify the files to apply a diff to when writing the diff, or log changes applied to a snapshot and keep the snapshot history around (which is how we know the order of snapshots). The first option is not good if you want to write without reading data to determine where the deleted records are. The second prevents cleaning up snapshot history.
> 
> We also talked about whether we should encode IDs in data files. Jacques pointed out that retrying a commit is easier if you don't need to re-read the original data to reconcile changes. For example, if a data file was compacted in a concurrent write, how do we reconcile a delete for it? We discussed other options, like rolling back the compaction for delete events. I think that's a promising option.
> 
> For action items, Jacques was going to think about whether we need to encode IDs in data files or if we could use positions to identify rows and write up a summary/proposal. Erik was going to take on planning how identifying rows without reading data would work and similarly write up a summary/proposal.
> 
> That's from memory, so if I've missed anything, I hope that other attendees will fill in the details!
> 
> rb
> 
> On Wed, May 29, 2019 at 3:34 PM Venkatakrishnan Sowrirajan <vsowrira@asu.edu <ma...@asu.edu>> wrote:
> Hi Ryan,
> 
> I couldn't attend the meeting. Just curious, if this is recorded by any chance.
> 
> Regards
> Venkata krishnan
> 
> 
> On Fri, May 24, 2019 at 8:49 AM Ryan Blue <rb...@netflix.com.invalid> wrote:
> Yes, I agree. I'll talk a little about a couple of the constraints of this as well.
> 
> On Fri, May 24, 2019 at 5:52 AM Anton Okolnychyi <aokolnychyi@apple.com <ma...@apple.com>> wrote:
> The agenda looks good to me. I think it would also make sense to clarify the responsibilities of query engines and Iceberg. Not only in terms of uniqueness, but also in terms of applying diffs on read, for example. 
> 
>> On 23 May 2019, at 01:59, Ryan Blue <rblue@netflix.com.INVALID <ma...@netflix.com.INVALID>> wrote:
>> 
>> Here’s a rough agenda:
>> 
>> Use cases: everyone come with a use case that you’d like to have supported. We’ll go around and introduce ourselves and our use cases.
>> Main topic: How should Iceberg identify rows that are deleted?
>> Side topics from my initial email, if we have time: should we use insert diffs, should we support dense and sparse formats, etc.
>> The main topic I think we should discuss is: How should Iceberg identify rows that are deleted?
>> 
>> I’m phrasing it this way to avoid where I think we’re talking past one another because we are making assumptions. The important thing is that there are two main options:
>> 
>> Filename and position, vs
>> Specific values of (few) columns in the data
>> This phrasing also avoids discussing uniqueness constraints. Once we get down to behavior, I think we agree. For example, I think we all agree that uniqueness cannot be enforced in Iceberg.
>> 
>> If uniqueness can’t be enforced in Iceberg, the main choice comes down to how we identify rows that are deleted. If we use (filename, position) then we know that there is only one row. On the other hand, if we use data values to identify rows then a delete may identify more than one row because there are no uniqueness guarantees. I think we also agree that if there is more than one row identified, all of them should be deleted.
>> 
>> At that point, there are trade-offs between the approaches:
>> 
>> When identifying deleted rows by data values, situations like the one that Anton pointed out are possible.
>> Jacques also had a good point about concurrency. If at all possible, we want to be able to reconcile changes between concurrent commits without re-running an operation.
>> Sound like a reasonable amount to talk through?
>> 
>> rb
>> 
>> 
>> On Wed, May 22, 2019 at 1:17 PM Erik Wright <erik.wright@shopify.com <ma...@shopify.com>> wrote:
>> 
>> 
>> On Wed, May 22, 2019 at 4:04 PM Cristian Opris <copris@apple.com.invalid <ma...@apple.com.invalid>> wrote:
>> Agreed with Erik here, we're certainly not looking to build the equivalent of a relational database, and for that matter not even that of a local disk storage analytics database (like Vertica). Those are very 
>> different designs with very different trade-offs and optimizations.
>> 
>> We're looking to automate and optimize specific types of file manipulation for large files on remote storage, while presenting that to the user under the common SQL API for bulk data manipulation (MERGE INTO)
>> 
>> What I would encourage is to decouple the storage model from the implementation of that API. If Iceberg has support for merge-on-read of upserts and deletes, in addition to its powerful support for partitioning, it will be easy for a higher-level application to implement those APIs given certain other constraints (that might not be appropriate to all applications).
>> 
>> Myself and Miguel are out on Friday, but Anton should be able to handle the discussion on our side.
>> 
>> 
>> Thanks,
>> Cristian
>> 
>> 
>>> On 22 May 2019, at 17:51, Erik Wright <erik.wright@shopify.com.INVALID <ma...@shopify.com.INVALID>> wrote:
>>> 
>>> We have two rows with the same natural key and we use that natural key in diff files:
>>> nk | col1 | col2
>>> 1 | 1 | 1
>>> 1 | 2 | 2
>>> Then we have a delete statement:
>>> DELETE FROM t WHERE col1 = 1
>>> 
>>> I think this example cuts to the point of the differences of understanding. Does Iceberg want to be approaching the utility of a relational database, against which I can execute complex update queries? This is not what I would have imagined.
>>> 
>>> I would have, instead, imagined that it was up to the client to identify, through whatever means, that they want to update or delete a row with a given ID. If there are multiple (distinct) rows with the same ID, _too bad_. Any user should _expect_ that they could potentially see any one or more of those rows at read time. And that an upsert/delete would affect any/all of them (I would argue for all).
>>> 
>>> In summary: Instead of trying to come up with a consistent, logical handling for complex queries that are best suited for a relational database, leave such handling up to the client and concentrate on problems that can be solved simply and more generally.
>>> 
>>> On Wed, May 22, 2019 at 12:11 PM Ryan Blue <rblue@netflix.com.invalid <ma...@netflix.com.invalid>> wrote:
>>> Yes, I think we should. I was going to propose one after catching up on the rest of this thread today.
>>> 
>>> On Wed, May 22, 2019 at 9:08 AM Anton Okolnychyi <aokolnychyi@apple.com <ma...@apple.com>> wrote:
>>> Thanks! Would it make sense to discuss the agenda in advance?
>>> 
>>>> On 22 May 2019, at 17:04, Ryan Blue <rblue@netflix.com.INVALID <ma...@netflix.com.INVALID>> wrote:
>>>> 
>>>> I sent out an invite and included everyone on this thread. If anyone else would like to join, please join the Zoom meeting. If you'd like to be added to the calendar invite, just let me know and I'll add you.
>>>> 
>>>> On Wed, May 22, 2019 at 8:57 AM Jacques Nadeau <jacques@dremio.com <ma...@dremio.com>> wrote:
>>>> works for me.
>>>> 
>>>> To make things easier, we can use my zoom meeting if people like:
>>>> 
>>>> Join Zoom Meeting
>>>> https://zoom.us/j/4157302092 <https://zoom.us/j/4157302092>
>>>> 
>>>> One tap mobile
>>>> +16465588656,,4157302092# US (New York)
>>>> +16699006833,,4157302092# US (San Jose)
>>>> 
>>>> Dial by your location
>>>>         +1 646 558 8656 US (New York)
>>>>         +1 669 900 6833 US (San Jose)
>>>>         877 853 5257 US Toll-free
>>>>         888 475 4499 US Toll-free
>>>> Meeting ID: 415 730 2092
>>>> Find your local number: https://zoom.us/u/aH9XYBfm <https://zoom.us/u/aH9XYBfm>
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Jacques Nadeau
>>>> CTO and Co-Founder, Dremio
>>>> 
>>>> 
>>>> On Wed, May 22, 2019 at 8:54 AM Ryan Blue <rblue@netflix.com.invalid <ma...@netflix.com.invalid>> wrote:
>>>> 9AM on Friday works best for me. How about then?
>>>> 
>>>> On Wed, May 22, 2019 at 5:05 AM Anton Okolnychyi <aokolnychyi@apple.com <ma...@apple.com>> wrote:
>>>> What about this Friday? One hour slot from 9:00 to 10:00 am or 10:00 to 11:00 am PST? Some folks are based in London, so meeting later than this is hard. If Friday doesn’t work, we can consider Tuesday or Wednesday next week.
>>>> 
>>>>> On 22 May 2019, at 00:54, Jacques Nadeau <jacques@dremio.com <ma...@dremio.com>> wrote:
>>>>> 
>>>>> I agree with Anton that we should probably spend some time on hangouts further discussing things. Definitely differing expectations here and we seem to be talking a bit past each other.
>>>>> --
>>>>> Jacques Nadeau
>>>>> CTO and Co-Founder, Dremio
>>>>> 
>>>>> 
>>>>> On Tue, May 21, 2019 at 3:44 PM Cristian Opris <copris@apple.com.invalid <ma...@apple.com.invalid>> wrote:
>>>>> I love a good flame war :P
>>>>> 
>>>>>> On 21 May 2019, at 22:57, Jacques Nadeau <jacques@dremio.com <ma...@dremio.com>> wrote:
>>>>>> 
>>>>>> 
>>>>>> That's my point, truly independent writers (two Spark jobs, or a Spark job and Dremio job) means a distributed transaction. It would need yet another external transaction coordinator on top of both Spark and Dremio, Iceberg by itself 
>>>>>> cannot solve this.
>>>>>>  
>>>>>> I'm not ready to accept this. Iceberg already supports a set of semantics around multiple writers committing simultaneously and how conflict resolution is done. The same can be done here.
>>>>> 
>>>>> 
>>>>> 
>>>>> MVCC (which is what Iceberg tries to implement) requires a total ordering of snapshots. Also the snapshots need to be non-conflicting. I really don't see how any metadata data structures can solve this without an outside coordinator.
>>>>> 
>>>>> Consider this:
>>>>> 
>>>>> Snapshot 0: (K,A) = 1
>>>>> Job X: UPDATE K SET A=A+1
>>>>> Job Y: UPDATE K SET A=10
>>>>> 
>>>>> What should the final value of A be and who decides ?
>>>>> 
>>>>>>  
>>>>>> By single writer, I don't mean single process, I mean multiple coordinated processes like Spark executors coordinated by Spark driver. The coordinator ensures that the data is pre-partitioned on 
>>>>>> each executor, and the coordinator commits the snapshot. 
>>>>>> 
>>>>>> Note however that single writer job/multiple concurrent reader jobs is perfectly feasible, i.e. it shouldn't be a problem to write from a Spark job and read from multiple Dremio queries concurrently (for example)
>>>>>> 
>>>>>> :D This is still "single process" from my perspective. That process may be coordinating other processes to do distributed work but ultimately it is a single process. 
>>>>> 
>>>>> Fair enough
>>>>> 
>>>>>>  
>>>>>> I'm not sure what you mean exactly. If we can't enforce uniqueness we shouldn't assume it.
>>>>>>  
>>>>>> I disagree. We can specify that as a requirement and state that you'll get unintended consequences if you provide your own keys and don't maintain this.
>>>>> 
>>>>> There's no need for unintended consequences, we can specify consistent behaviour (and I believe the document says what that is)
>>>>> 
>>>>> 
>>>>>>  
>>>>>> We do expect that most of the time the natural key is unique, but the eager and lazy with natural key designs can handle duplicates 
>>>>>> consistently. Basically it's not a problem to have duplicate natural keys, everything works fine.
>>>>>> 
>>>>>> That heavily depends on how things are implemented. For example, we may write a bunch of code that generates internal data structures based on this expectation. If we have to support duplicate matches, all of sudden we can no longer size various data structures to improve performance and may be unable to preallocate memory associated with a guaranteed completion.
>>>>> 
>>>>> Again we need to operate on the assumption that this is a large scale distributed compute/remote storage scenario. Key matching is done with shuffles with data movement across the network, such optimizations would really have little impact on overall performance. Not to mention that most query engines would already optimize the shuffle already as much as it can be optimized.
>>>>> 
>>>>> It is true that if actual duplicate keys would make the key matching join (anti-join) somewhat more expensive, however it can be done in such a way that if the keys are in practice unique the join is as efficient as it can be.
>>>>> 
>>>>> 
>>>>>> 
>>>>>> Let me try and clarify each point:
>>>>>> 
>>>>>> - lookup for query or update on a non-(partition/bucket/sort) key predicate implies scanning large amounts of data - because these are the only data structures that can narrow down the lookup, right ? One could argue that the min/max index (file skipping) can be applied to any column, but in reality if that column is not sorted the min/max intervals can have huge overlaps so it may be next to useless.
>>>>>> - remote storage - this is a critical architecture decision - implementations on local storage imply a vastly different design for the entire system, storage and compute. 
>>>>>> - deleting single records per snapshot is unfeasible in eager but also particularly in the lazy design: each deletion creates a very small snapshot. Deleting 1 million records one at a time would create 1 million small files, and 1 million RPC calls.
>>>>>> 
>>>>>> Why is this unfeasible? If I have a dataset of 100mm files including 1mm small files, is that a major problem? It seems like your usecase isn't one where you want to support single record deletes but it is definitely something important to many people.
>>>>> 
>>>>> 100 mm total files or 1 mm files per dataset is definitely a problem on HDFS, and I believe on S3 too. Single key delete would work just fine, but it's simply not optimal to do that on remote storage. This is a very well known problem with HDFS, and one of the very reasons to have something like Iceberg in the first place.
>>>>> 
>>>>> Basically the users would be able to do single key mutation, but it's not the use case we should be optimizing for, but it's really not advisable.
>>>>> 
>>>>> 
>>>>>>  
>>>>>> Eager is conceptually just lazy + compaction done, well, eagerly. The logic for both is exactly the same, the trade-off is just that with eager you implicitly compact every time so that you don't do any work on read, while with lazy 
>>>>>> you want to amortize the cost of compaction over multiple snapshots.
>>>>>> 
>>>>>> Basically there should be no difference between the two conceptually, or with regard to keys, etc. The only difference is some mechanics in implementation.
>>>>>> 
>>>>>> I think you have deconstruct the problem too much to say these are the same (or at least that is what I'm starting to think given this thread). It seems like real world implementation decisions (per our discussion here) are in conflict. For example, you just argued against having a 1mm arbitrary mutations but I think that is because you aren't thinking about things over time with a delta implementation. Having 10,000 mutations a day where we do delta compaction once a week
>>>>>> and local file mappings (key to offset sparse bitmaps) seems like it could result in very good performance in a case where we're mutating small amounts of data. In this scenario, you may not do major compaction ever unless you get to a high enough percentage of records that have been deleted in the original dataset. That drives a very different set of implementation decisions from a situation where you're trying to restate an entire partition at once.
>>>>> 
>>>>> 
>>>>> We operate on 1 billion mutations per day at least. This is the problem Iceberg wants to solve, I believe it's stated upfront. 10000/day is not a big data problem. It can be done fairly trivially and it would be supported, but there's not much point in extra optimizing for this use case I believe.
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> Ryan Blue
>>>> Software Engineer
>>>> Netflix
>>>> 
>>>> 
>>>> -- 
>>>> Ryan Blue
>>>> Software Engineer
>>>> Netflix
>>> 
>>> 
>>> 
>>> -- 
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>> 
>> 
>> 
>> -- 
>> Ryan Blue
>> Software Engineer
>> Netflix
> 
> 
> 
> -- 
> Ryan Blue
> Software Engineer
> Netflix
> 
> 
> -- 
> Ryan Blue
> Software Engineer
> Netflix
> 
> 
> -- 
> Ryan Blue
> Software Engineer
> Netflix

Re: Updates/Deletes/Upserts in Iceberg

[Ryan Blue <rb...@netflix.com.INVALID>]

Thanks, Erik! Great to see progress here. I'll set aside some time to look
this over in detail.

On Fri, Jun 7, 2019 at 11:46 AM Erik Wright <er...@shopify.com> wrote:

> I apologize for the delay, but I have finally put together a document
> describing an alternative approach to supporting updates in Iceberg while
> minimizing write amplification.
>
> Proposal: Iceberg Merge-on-Read
> <https://docs.google.com/document/d/1KuOMeS8Hw_yuE5IXtII8EClJlEMtqFECfWbg-gN5lsQ/edit?usp=sharing>
>
> Thank you, Anton and Miguel, for starting this conversation, and everyone
> else as well for the ongoing dialogue. I'm looking forward to continuing to
> discuss this and hopefully finding an approach that can meet our different
> needs and that we can work together on.
>
> Cheers,
>
> Erik
>
> On Wed, May 29, 2019 at 7:26 PM Jacques Nadeau <ja...@dremio.com> wrote:
>
>> Yeah, I totally forgot to record our discussion. Will do so next time,
>> sorry.
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>>
>>
>> On Wed, May 29, 2019 at 4:24 PM Ryan Blue <rb...@netflix.com> wrote:
>>
>>> It wasn't recorded, but I can summarize what we talked about. Sorry I
>>> haven't sent this out earlier.
>>>
>>> We talked about the options and some of the background in Iceberg --
>>> basically that it isn't possible to determine the order of commits before
>>> you commit so you can't rely on some monotonically increasing value from a
>>> snapshot to know which deltas to apply to a file. The result is that we
>>> can't apply diffs to data files using a rule like "files older than X"
>>> because we can't identify those files without the snapshot history.
>>>
>>> That gives us basically 2 options for scoping delete diffs: either
>>> identify the files to apply a diff to when writing the diff, or log changes
>>> applied to a snapshot and keep the snapshot history around (which is how we
>>> know the order of snapshots). The first option is not good if you want to
>>> write without reading data to determine where the deleted records are. The
>>> second prevents cleaning up snapshot history.
>>>
>>> We also talked about whether we should encode IDs in data files. Jacques
>>> pointed out that retrying a commit is easier if you don't need to re-read
>>> the original data to reconcile changes. For example, if a data file was
>>> compacted in a concurrent write, how do we reconcile a delete for it? We
>>> discussed other options, like rolling back the compaction for delete
>>> events. I think that's a promising option.
>>>
>>> For action items, Jacques was going to think about whether we need to
>>> encode IDs in data files or if we could use positions to identify rows and
>>> write up a summary/proposal. Erik was going to take on planning how
>>> identifying rows without reading data would work and similarly write up a
>>> summary/proposal.
>>>
>>> That's from memory, so if I've missed anything, I hope that other
>>> attendees will fill in the details!
>>>
>>> rb
>>>
>>> On Wed, May 29, 2019 at 3:34 PM Venkatakrishnan Sowrirajan <
>>> vsowrira@asu.edu> wrote:
>>>
>>>> Hi Ryan,
>>>>
>>>> I couldn't attend the meeting. Just curious, if this is recorded by any
>>>> chance.
>>>>
>>>> Regards
>>>> Venkata krishnan
>>>>
>>>>
>>>> On Fri, May 24, 2019 at 8:49 AM Ryan Blue <rb...@netflix.com.invalid>
>>>> wrote:
>>>>
>>>>> Yes, I agree. I'll talk a little about a couple of the constraints of
>>>>> this as well.
>>>>>
>>>>> On Fri, May 24, 2019 at 5:52 AM Anton Okolnychyi <
>>>>> aokolnychyi@apple.com> wrote:
>>>>>
>>>>>> The agenda looks good to me. I think it would also make sense to
>>>>>> clarify the responsibilities of query engines and Iceberg. Not only in
>>>>>> terms of uniqueness, but also in terms of applying diffs on read, for
>>>>>> example.
>>>>>>
>>>>>> On 23 May 2019, at 01:59, Ryan Blue <rb...@netflix.com.INVALID>
>>>>>> wrote:
>>>>>>
>>>>>> Here’s a rough agenda:
>>>>>>
>>>>>>    - Use cases: everyone come with a use case that you’d like to
>>>>>>    have supported. We’ll go around and introduce ourselves and our use cases.
>>>>>>    - Main topic: How should Iceberg identify rows that are deleted?
>>>>>>    - Side topics from my initial email, if we have time: should we
>>>>>>    use insert diffs, should we support dense and sparse formats, etc.
>>>>>>
>>>>>> The main topic I think we should discuss is: *How should Iceberg
>>>>>> identify rows that are deleted?*
>>>>>>
>>>>>> I’m phrasing it this way to avoid where I think we’re talking past
>>>>>> one another because we are making assumptions. The important thing is that
>>>>>> there are two main options:
>>>>>>
>>>>>>    - Filename and position, vs
>>>>>>    - Specific values of (few) columns in the data
>>>>>>
>>>>>> This phrasing also avoids discussing uniqueness constraints. Once we
>>>>>> get down to behavior, I think we agree. For example, I think we all agree
>>>>>> that uniqueness cannot be enforced in Iceberg.
>>>>>>
>>>>>> If uniqueness can’t be enforced in Iceberg, the main choice comes
>>>>>> down to how we identify rows that are deleted. If we use (filename,
>>>>>> position) then we know that there is only one row. On the other hand, if we
>>>>>> use data values to identify rows then a delete may identify more than one
>>>>>> row because there are no uniqueness guarantees. I think we also agree that
>>>>>> if there is more than one row identified, all of them should be deleted.
>>>>>>
>>>>>> At that point, there are trade-offs between the approaches:
>>>>>>
>>>>>>    - When identifying deleted rows by data values, situations like
>>>>>>    the one that Anton pointed out are possible.
>>>>>>    - Jacques also had a good point about concurrency. If at all
>>>>>>    possible, we want to be able to reconcile changes between concurrent
>>>>>>    commits without re-running an operation.
>>>>>>
>>>>>> Sound like a reasonable amount to talk through?
>>>>>>
>>>>>> rb
>>>>>>
>>>>>> On Wed, May 22, 2019 at 1:17 PM Erik Wright <er...@shopify.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, May 22, 2019 at 4:04 PM Cristian Opris <
>>>>>>> copris@apple.com.invalid> wrote:
>>>>>>>
>>>>>>>> Agreed with Erik here, we're certainly not looking to build the
>>>>>>>> equivalent of a relational database, and for that matter not even that of a
>>>>>>>> local disk storage analytics database (like Vertica). Those are very
>>>>>>>> different designs with very different trade-offs and optimizations.
>>>>>>>>
>>>>>>>> We're looking to automate and optimize specific types of file
>>>>>>>> manipulation for large files on remote storage, while presenting that to
>>>>>>>> the user under the common SQL API for *bulk* data manipulation
>>>>>>>> (MERGE INTO)
>>>>>>>>
>>>>>>>
>>>>>>> What I would encourage is to decouple the storage model from the
>>>>>>> implementation of that API. If Iceberg has support for merge-on-read of
>>>>>>> upserts and deletes, in addition to its powerful support for partitioning,
>>>>>>> it will be easy for a higher-level application to implement those APIs
>>>>>>> given certain other constraints (that might not be appropriate to all
>>>>>>> applications).
>>>>>>>
>>>>>>> Myself and Miguel are out on Friday, but Anton should be able to
>>>>>>>> handle the discussion on our side.
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Cristian
>>>>>>>>
>>>>>>>>
>>>>>>>> On 22 May 2019, at 17:51, Erik Wright <
>>>>>>>> erik.wright@shopify.com.INVALID> wrote:
>>>>>>>>
>>>>>>>> We have two rows with the same natural key and we use that natural
>>>>>>>>> key in diff files:
>>>>>>>>> nk | col1 | col2
>>>>>>>>> 1 | 1 | 1
>>>>>>>>> 1 | 2 | 2
>>>>>>>>> Then we have a delete statement:
>>>>>>>>> DELETE FROM t WHERE col1 = 1
>>>>>>>>
>>>>>>>>
>>>>>>>> I think this example cuts to the point of the differences of
>>>>>>>> understanding. Does Iceberg want to be approaching the utility of a
>>>>>>>> relational database, against which I can execute complex update queries?
>>>>>>>> This is not what I would have imagined.
>>>>>>>>
>>>>>>>> I would have, instead, imagined that it was up to the client to
>>>>>>>> identify, through whatever means, that they want to update or delete a row
>>>>>>>> with a given ID. If there are multiple (distinct) rows with the same ID,
>>>>>>>> _too bad_. Any user should _expect_ that they could potentially see any one
>>>>>>>> or more of those rows at read time. And that an upsert/delete would affect
>>>>>>>> any/all of them (I would argue for all).
>>>>>>>>
>>>>>>>> *In summary:* Instead of trying to come up with a consistent,
>>>>>>>> logical handling for complex queries that are best suited for a relational
>>>>>>>> database, leave such handling up to the client and concentrate on problems
>>>>>>>> that can be solved simply and more generally.
>>>>>>>>
>>>>>>>> On Wed, May 22, 2019 at 12:11 PM Ryan Blue <
>>>>>>>> rblue@netflix.com.invalid> wrote:
>>>>>>>>
>>>>>>>>> Yes, I think we should. I was going to propose one after catching
>>>>>>>>> up on the rest of this thread today.
>>>>>>>>>
>>>>>>>>> On Wed, May 22, 2019 at 9:08 AM Anton Okolnychyi <
>>>>>>>>> aokolnychyi@apple.com> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks! Would it make sense to discuss the agenda in advance?
>>>>>>>>>>
>>>>>>>>>> On 22 May 2019, at 17:04, Ryan Blue <rb...@netflix.com.INVALID>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> I sent out an invite and included everyone on this thread. If
>>>>>>>>>> anyone else would like to join, please join the Zoom meeting. If you'd like
>>>>>>>>>> to be added to the calendar invite, just let me know and I'll add you.
>>>>>>>>>>
>>>>>>>>>> On Wed, May 22, 2019 at 8:57 AM Jacques Nadeau <
>>>>>>>>>> jacques@dremio.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> works for me.
>>>>>>>>>>>
>>>>>>>>>>> To make things easier, we can use my zoom meeting if people like:
>>>>>>>>>>>
>>>>>>>>>>> Join Zoom Meeting
>>>>>>>>>>> https://zoom.us/j/4157302092
>>>>>>>>>>>
>>>>>>>>>>> One tap mobile
>>>>>>>>>>> +16465588656,,4157302092# US (New York)
>>>>>>>>>>> +16699006833,,4157302092# US (San Jose)
>>>>>>>>>>>
>>>>>>>>>>> Dial by your location
>>>>>>>>>>>         +1 646 558 8656 US (New York)
>>>>>>>>>>>         +1 669 900 6833 US (San Jose)
>>>>>>>>>>>         877 853 5257 US Toll-free
>>>>>>>>>>>         888 475 4499 US Toll-free
>>>>>>>>>>> Meeting ID: 415 730 2092
>>>>>>>>>>> Find your local number: https://zoom.us/u/aH9XYBfm
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Jacques Nadeau
>>>>>>>>>>> CTO and Co-Founder, Dremio
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 22, 2019 at 8:54 AM Ryan Blue <
>>>>>>>>>>> rblue@netflix.com.invalid> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> 9AM on Friday works best for me. How about then?
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, May 22, 2019 at 5:05 AM Anton Okolnychyi <
>>>>>>>>>>>> aokolnychyi@apple.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> What about this Friday? One hour slot from 9:00 to 10:00 am or
>>>>>>>>>>>>> 10:00 to 11:00 am PST? Some folks are based in London, so meeting later
>>>>>>>>>>>>> than this is hard. If Friday doesn’t work, we can consider Tuesday or
>>>>>>>>>>>>> Wednesday next week.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 22 May 2019, at 00:54, Jacques Nadeau <ja...@dremio.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> I agree with Anton that we should probably spend some time on
>>>>>>>>>>>>> hangouts further discussing things. Definitely differing expectations here
>>>>>>>>>>>>> and we seem to be talking a bit past each other.
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Jacques Nadeau
>>>>>>>>>>>>> CTO and Co-Founder, Dremio
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, May 21, 2019 at 3:44 PM Cristian Opris <
>>>>>>>>>>>>> copris@apple.com.invalid> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I love a good flame war :P
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 21 May 2019, at 22:57, Jacques Nadeau <ja...@dremio.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> That's my point, truly independent writers (two Spark jobs,
>>>>>>>>>>>>>>> or a Spark job and Dremio job) means a distributed transaction. It would
>>>>>>>>>>>>>>> need yet another external transaction coordinator on top of both Spark and
>>>>>>>>>>>>>>> Dremio, Iceberg by itself
>>>>>>>>>>>>>>> cannot solve this.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm not ready to accept this. Iceberg already supports a set
>>>>>>>>>>>>>> of semantics around multiple writers committing simultaneously and how
>>>>>>>>>>>>>> conflict resolution is done. The same can be done here.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> MVCC (which is what Iceberg tries to implement) requires a
>>>>>>>>>>>>>> total ordering of snapshots. Also the snapshots need to be non-conflicting.
>>>>>>>>>>>>>> I really don't see how any metadata data structures can solve this without
>>>>>>>>>>>>>> an outside coordinator.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Consider this:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Snapshot 0: (K,A) = 1
>>>>>>>>>>>>>> Job X: UPDATE K SET A=A+1
>>>>>>>>>>>>>> Job Y: UPDATE K SET A=10
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What should the final value of A be and who decides ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> By single writer, I don't mean single process, I mean
>>>>>>>>>>>>>>> multiple coordinated processes like Spark executors coordinated by Spark
>>>>>>>>>>>>>>> driver. The coordinator ensures that the data is pre-partitioned on
>>>>>>>>>>>>>>> each executor, and the coordinator commits the snapshot.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Note however that single writer job/multiple concurrent
>>>>>>>>>>>>>>> reader jobs is perfectly feasible, i.e. it shouldn't be a problem to write
>>>>>>>>>>>>>>> from a Spark job and read from multiple Dremio queries concurrently (for
>>>>>>>>>>>>>>> example)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> :D This is still "single process" from my perspective. That
>>>>>>>>>>>>>> process may be coordinating other processes to do distributed work but
>>>>>>>>>>>>>> ultimately it is a single process.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Fair enough
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm not sure what you mean exactly. If we can't enforce
>>>>>>>>>>>>>>> uniqueness we shouldn't assume it.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I disagree. We can specify that as a requirement and state
>>>>>>>>>>>>>> that you'll get unintended consequences if you provide your own keys and
>>>>>>>>>>>>>> don't maintain this.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> There's no need for unintended consequences, we can specify
>>>>>>>>>>>>>> consistent behaviour (and I believe the document says what that is)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We do expect that most of the time the natural key is
>>>>>>>>>>>>>>> unique, but the eager and lazy with natural key designs can handle
>>>>>>>>>>>>>>> duplicates
>>>>>>>>>>>>>>> consistently. Basically it's not a problem to have duplicate
>>>>>>>>>>>>>>> natural keys, everything works fine.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> That heavily depends on how things are implemented. For
>>>>>>>>>>>>>> example, we may write a bunch of code that generates internal data
>>>>>>>>>>>>>> structures based on this expectation. If we have to support duplicate
>>>>>>>>>>>>>> matches, all of sudden we can no longer size various data structures to
>>>>>>>>>>>>>> improve performance and may be unable to preallocate memory associated with
>>>>>>>>>>>>>> a guaranteed completion.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Again we need to operate on the assumption that this is a
>>>>>>>>>>>>>> large scale distributed compute/remote storage scenario. Key matching is
>>>>>>>>>>>>>> done with shuffles with data movement across the network, such
>>>>>>>>>>>>>> optimizations would really have little impact on overall performance. Not
>>>>>>>>>>>>>> to mention that most query engines would already optimize the shuffle
>>>>>>>>>>>>>> already as much as it can be optimized.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It is true that if actual duplicate keys would make the key
>>>>>>>>>>>>>> matching join (anti-join) somewhat more expensive, however it can be done
>>>>>>>>>>>>>> in such a way that if the keys are in practice unique the join is as
>>>>>>>>>>>>>> efficient as it can be.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Let me try and clarify each point:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - lookup for query or update on a
>>>>>>>>>>>>>>> non-(partition/bucket/sort) key predicate implies scanning large amounts of
>>>>>>>>>>>>>>> data - because these are the only data structures that can narrow down the
>>>>>>>>>>>>>>> lookup, right ? One could argue that the min/max index (file skipping) can
>>>>>>>>>>>>>>> be applied to any column, but in reality if that column is not sorted the
>>>>>>>>>>>>>>> min/max intervals can have huge overlaps so it may be next to useless.
>>>>>>>>>>>>>>> - remote storage - this is a critical architecture decision
>>>>>>>>>>>>>>> - implementations on local storage imply a vastly different design for the
>>>>>>>>>>>>>>> entire system, storage and compute.
>>>>>>>>>>>>>>> - deleting single records per snapshot is unfeasible in
>>>>>>>>>>>>>>> eager but also particularly in the lazy design: each deletion creates a
>>>>>>>>>>>>>>> very small snapshot. Deleting 1 million records one at a time would create
>>>>>>>>>>>>>>> 1 million small files, and 1 million RPC calls.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Why is this unfeasible? If I have a dataset of 100mm files
>>>>>>>>>>>>>> including 1mm small files, is that a major problem? It seems like your
>>>>>>>>>>>>>> usecase isn't one where you want to support single record deletes but it is
>>>>>>>>>>>>>> definitely something important to many people.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 100 mm total files or 1 mm files per dataset is definitely a
>>>>>>>>>>>>>> problem on HDFS, and I believe on S3 too. Single key delete would work just
>>>>>>>>>>>>>> fine, but it's simply not optimal to do that on remote storage. This is a
>>>>>>>>>>>>>> very well known problem with HDFS, and one of the very reasons to have
>>>>>>>>>>>>>> something like Iceberg in the first place.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Basically the users would be able to do single key mutation,
>>>>>>>>>>>>>> but it's not the use case we should be optimizing for, but it's really not
>>>>>>>>>>>>>> advisable.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Eager is conceptually just lazy + compaction done, well,
>>>>>>>>>>>>>>> eagerly. The logic for both is exactly the same, the trade-off is just that
>>>>>>>>>>>>>>> with eager you implicitly compact every time so that you don't do any work
>>>>>>>>>>>>>>> on read, while with lazy
>>>>>>>>>>>>>>> you want to amortize the cost of compaction over multiple
>>>>>>>>>>>>>>> snapshots.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Basically there should be no difference between the two
>>>>>>>>>>>>>>> conceptually, or with regard to keys, etc. The only difference is some
>>>>>>>>>>>>>>> mechanics in implementation.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think you have deconstruct the problem too much to say
>>>>>>>>>>>>>> these are the same (or at least that is what I'm starting to think given
>>>>>>>>>>>>>> this thread). It seems like real world implementation decisions (per our
>>>>>>>>>>>>>> discussion here) are in conflict. For example, you just argued against
>>>>>>>>>>>>>> having a 1mm arbitrary mutations but I think that is because you aren't
>>>>>>>>>>>>>> thinking about things over time with a delta implementation. Having 10,000
>>>>>>>>>>>>>> mutations a day where we do delta compaction once a week
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> and local file mappings (key to offset sparse bitmaps) seems
>>>>>>>>>>>>>> like it could result in very good performance in a case where we're
>>>>>>>>>>>>>> mutating small amounts of data. In this scenario, you may not do major
>>>>>>>>>>>>>> compaction ever unless you get to a high enough percentage of records that
>>>>>>>>>>>>>> have been deleted in the original dataset. That drives a very different set
>>>>>>>>>>>>>> of implementation decisions from a situation where you're trying to restate
>>>>>>>>>>>>>> an entire partition at once.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We operate on 1 billion mutations per day at least. This is
>>>>>>>>>>>>>> the problem Iceberg wants to solve, I believe it's stated upfront.
>>>>>>>>>>>>>> 10000/day is not a big data problem. It can be done fairly trivially and it
>>>>>>>>>>>>>> would be supported, but there's not much point in extra optimizing for this
>>>>>>>>>>>>>> use case I believe.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>> Netflix
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Ryan Blue
>>>>>>>>>> Software Engineer
>>>>>>>>>> Netflix
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Ryan Blue
>>>>>>>>> Software Engineer
>>>>>>>>> Netflix
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Ryan Blue
>>>>>> Software Engineer
>>>>>> Netflix
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Ryan Blue
>>>>> Software Engineer
>>>>> Netflix
>>>>>
>>>>
>>>
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>>
>>

-- 
Ryan Blue
Software Engineer
Netflix

Re: Updates/Deletes/Upserts in Iceberg

[Erik Wright <er...@shopify.com.INVALID>]

I apologize for the delay, but I have finally put together a document
describing an alternative approach to supporting updates in Iceberg while
minimizing write amplification.

Proposal: Iceberg Merge-on-Read
<https://docs.google.com/document/d/1KuOMeS8Hw_yuE5IXtII8EClJlEMtqFECfWbg-gN5lsQ/edit?usp=sharing>

Thank you, Anton and Miguel, for starting this conversation, and everyone
else as well for the ongoing dialogue. I'm looking forward to continuing to
discuss this and hopefully finding an approach that can meet our different
needs and that we can work together on.

Cheers,

Erik

On Wed, May 29, 2019 at 7:26 PM Jacques Nadeau <ja...@dremio.com> wrote:

> Yeah, I totally forgot to record our discussion. Will do so next time,
> sorry.
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
>
> On Wed, May 29, 2019 at 4:24 PM Ryan Blue <rb...@netflix.com> wrote:
>
>> It wasn't recorded, but I can summarize what we talked about. Sorry I
>> haven't sent this out earlier.
>>
>> We talked about the options and some of the background in Iceberg --
>> basically that it isn't possible to determine the order of commits before
>> you commit so you can't rely on some monotonically increasing value from a
>> snapshot to know which deltas to apply to a file. The result is that we
>> can't apply diffs to data files using a rule like "files older than X"
>> because we can't identify those files without the snapshot history.
>>
>> That gives us basically 2 options for scoping delete diffs: either
>> identify the files to apply a diff to when writing the diff, or log changes
>> applied to a snapshot and keep the snapshot history around (which is how we
>> know the order of snapshots). The first option is not good if you want to
>> write without reading data to determine where the deleted records are. The
>> second prevents cleaning up snapshot history.
>>
>> We also talked about whether we should encode IDs in data files. Jacques
>> pointed out that retrying a commit is easier if you don't need to re-read
>> the original data to reconcile changes. For example, if a data file was
>> compacted in a concurrent write, how do we reconcile a delete for it? We
>> discussed other options, like rolling back the compaction for delete
>> events. I think that's a promising option.
>>
>> For action items, Jacques was going to think about whether we need to
>> encode IDs in data files or if we could use positions to identify rows and
>> write up a summary/proposal. Erik was going to take on planning how
>> identifying rows without reading data would work and similarly write up a
>> summary/proposal.
>>
>> That's from memory, so if I've missed anything, I hope that other
>> attendees will fill in the details!
>>
>> rb
>>
>> On Wed, May 29, 2019 at 3:34 PM Venkatakrishnan Sowrirajan <
>> vsowrira@asu.edu> wrote:
>>
>>> Hi Ryan,
>>>
>>> I couldn't attend the meeting. Just curious, if this is recorded by any
>>> chance.
>>>
>>> Regards
>>> Venkata krishnan
>>>
>>>
>>> On Fri, May 24, 2019 at 8:49 AM Ryan Blue <rb...@netflix.com.invalid>
>>> wrote:
>>>
>>>> Yes, I agree. I'll talk a little about a couple of the constraints of
>>>> this as well.
>>>>
>>>> On Fri, May 24, 2019 at 5:52 AM Anton Okolnychyi <ao...@apple.com>
>>>> wrote:
>>>>
>>>>> The agenda looks good to me. I think it would also make sense to
>>>>> clarify the responsibilities of query engines and Iceberg. Not only in
>>>>> terms of uniqueness, but also in terms of applying diffs on read, for
>>>>> example.
>>>>>
>>>>> On 23 May 2019, at 01:59, Ryan Blue <rb...@netflix.com.INVALID> wrote:
>>>>>
>>>>> Here’s a rough agenda:
>>>>>
>>>>>    - Use cases: everyone come with a use case that you’d like to have
>>>>>    supported. We’ll go around and introduce ourselves and our use cases.
>>>>>    - Main topic: How should Iceberg identify rows that are deleted?
>>>>>    - Side topics from my initial email, if we have time: should we
>>>>>    use insert diffs, should we support dense and sparse formats, etc.
>>>>>
>>>>> The main topic I think we should discuss is: *How should Iceberg
>>>>> identify rows that are deleted?*
>>>>>
>>>>> I’m phrasing it this way to avoid where I think we’re talking past one
>>>>> another because we are making assumptions. The important thing is that
>>>>> there are two main options:
>>>>>
>>>>>    - Filename and position, vs
>>>>>    - Specific values of (few) columns in the data
>>>>>
>>>>> This phrasing also avoids discussing uniqueness constraints. Once we
>>>>> get down to behavior, I think we agree. For example, I think we all agree
>>>>> that uniqueness cannot be enforced in Iceberg.
>>>>>
>>>>> If uniqueness can’t be enforced in Iceberg, the main choice comes down
>>>>> to how we identify rows that are deleted. If we use (filename, position)
>>>>> then we know that there is only one row. On the other hand, if we use data
>>>>> values to identify rows then a delete may identify more than one row
>>>>> because there are no uniqueness guarantees. I think we also agree that if
>>>>> there is more than one row identified, all of them should be deleted.
>>>>>
>>>>> At that point, there are trade-offs between the approaches:
>>>>>
>>>>>    - When identifying deleted rows by data values, situations like
>>>>>    the one that Anton pointed out are possible.
>>>>>    - Jacques also had a good point about concurrency. If at all
>>>>>    possible, we want to be able to reconcile changes between concurrent
>>>>>    commits without re-running an operation.
>>>>>
>>>>> Sound like a reasonable amount to talk through?
>>>>>
>>>>> rb
>>>>>
>>>>> On Wed, May 22, 2019 at 1:17 PM Erik Wright <er...@shopify.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, May 22, 2019 at 4:04 PM Cristian Opris <
>>>>>> copris@apple.com.invalid> wrote:
>>>>>>
>>>>>>> Agreed with Erik here, we're certainly not looking to build the
>>>>>>> equivalent of a relational database, and for that matter not even that of a
>>>>>>> local disk storage analytics database (like Vertica). Those are very
>>>>>>> different designs with very different trade-offs and optimizations.
>>>>>>>
>>>>>>> We're looking to automate and optimize specific types of file
>>>>>>> manipulation for large files on remote storage, while presenting that to
>>>>>>> the user under the common SQL API for *bulk* data manipulation
>>>>>>> (MERGE INTO)
>>>>>>>
>>>>>>
>>>>>> What I would encourage is to decouple the storage model from the
>>>>>> implementation of that API. If Iceberg has support for merge-on-read of
>>>>>> upserts and deletes, in addition to its powerful support for partitioning,
>>>>>> it will be easy for a higher-level application to implement those APIs
>>>>>> given certain other constraints (that might not be appropriate to all
>>>>>> applications).
>>>>>>
>>>>>> Myself and Miguel are out on Friday, but Anton should be able to
>>>>>>> handle the discussion on our side.
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Cristian
>>>>>>>
>>>>>>>
>>>>>>> On 22 May 2019, at 17:51, Erik Wright <
>>>>>>> erik.wright@shopify.com.INVALID> wrote:
>>>>>>>
>>>>>>> We have two rows with the same natural key and we use that natural
>>>>>>>> key in diff files:
>>>>>>>> nk | col1 | col2
>>>>>>>> 1 | 1 | 1
>>>>>>>> 1 | 2 | 2
>>>>>>>> Then we have a delete statement:
>>>>>>>> DELETE FROM t WHERE col1 = 1
>>>>>>>
>>>>>>>
>>>>>>> I think this example cuts to the point of the differences of
>>>>>>> understanding. Does Iceberg want to be approaching the utility of a
>>>>>>> relational database, against which I can execute complex update queries?
>>>>>>> This is not what I would have imagined.
>>>>>>>
>>>>>>> I would have, instead, imagined that it was up to the client to
>>>>>>> identify, through whatever means, that they want to update or delete a row
>>>>>>> with a given ID. If there are multiple (distinct) rows with the same ID,
>>>>>>> _too bad_. Any user should _expect_ that they could potentially see any one
>>>>>>> or more of those rows at read time. And that an upsert/delete would affect
>>>>>>> any/all of them (I would argue for all).
>>>>>>>
>>>>>>> *In summary:* Instead of trying to come up with a consistent,
>>>>>>> logical handling for complex queries that are best suited for a relational
>>>>>>> database, leave such handling up to the client and concentrate on problems
>>>>>>> that can be solved simply and more generally.
>>>>>>>
>>>>>>> On Wed, May 22, 2019 at 12:11 PM Ryan Blue <
>>>>>>> rblue@netflix.com.invalid> wrote:
>>>>>>>
>>>>>>>> Yes, I think we should. I was going to propose one after catching
>>>>>>>> up on the rest of this thread today.
>>>>>>>>
>>>>>>>> On Wed, May 22, 2019 at 9:08 AM Anton Okolnychyi <
>>>>>>>> aokolnychyi@apple.com> wrote:
>>>>>>>>
>>>>>>>>> Thanks! Would it make sense to discuss the agenda in advance?
>>>>>>>>>
>>>>>>>>> On 22 May 2019, at 17:04, Ryan Blue <rb...@netflix.com.INVALID>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> I sent out an invite and included everyone on this thread. If
>>>>>>>>> anyone else would like to join, please join the Zoom meeting. If you'd like
>>>>>>>>> to be added to the calendar invite, just let me know and I'll add you.
>>>>>>>>>
>>>>>>>>> On Wed, May 22, 2019 at 8:57 AM Jacques Nadeau <ja...@dremio.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> works for me.
>>>>>>>>>>
>>>>>>>>>> To make things easier, we can use my zoom meeting if people like:
>>>>>>>>>>
>>>>>>>>>> Join Zoom Meeting
>>>>>>>>>> https://zoom.us/j/4157302092
>>>>>>>>>>
>>>>>>>>>> One tap mobile
>>>>>>>>>> +16465588656,,4157302092# US (New York)
>>>>>>>>>> +16699006833,,4157302092# US (San Jose)
>>>>>>>>>>
>>>>>>>>>> Dial by your location
>>>>>>>>>>         +1 646 558 8656 US (New York)
>>>>>>>>>>         +1 669 900 6833 US (San Jose)
>>>>>>>>>>         877 853 5257 US Toll-free
>>>>>>>>>>         888 475 4499 US Toll-free
>>>>>>>>>> Meeting ID: 415 730 2092
>>>>>>>>>> Find your local number: https://zoom.us/u/aH9XYBfm
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Jacques Nadeau
>>>>>>>>>> CTO and Co-Founder, Dremio
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, May 22, 2019 at 8:54 AM Ryan Blue <
>>>>>>>>>> rblue@netflix.com.invalid> wrote:
>>>>>>>>>>
>>>>>>>>>>> 9AM on Friday works best for me. How about then?
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 22, 2019 at 5:05 AM Anton Okolnychyi <
>>>>>>>>>>> aokolnychyi@apple.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> What about this Friday? One hour slot from 9:00 to 10:00 am or
>>>>>>>>>>>> 10:00 to 11:00 am PST? Some folks are based in London, so meeting later
>>>>>>>>>>>> than this is hard. If Friday doesn’t work, we can consider Tuesday or
>>>>>>>>>>>> Wednesday next week.
>>>>>>>>>>>>
>>>>>>>>>>>> On 22 May 2019, at 00:54, Jacques Nadeau <ja...@dremio.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I agree with Anton that we should probably spend some time on
>>>>>>>>>>>> hangouts further discussing things. Definitely differing expectations here
>>>>>>>>>>>> and we seem to be talking a bit past each other.
>>>>>>>>>>>> --
>>>>>>>>>>>> Jacques Nadeau
>>>>>>>>>>>> CTO and Co-Founder, Dremio
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, May 21, 2019 at 3:44 PM Cristian Opris <
>>>>>>>>>>>> copris@apple.com.invalid> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I love a good flame war :P
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 21 May 2019, at 22:57, Jacques Nadeau <ja...@dremio.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> That's my point, truly independent writers (two Spark jobs, or
>>>>>>>>>>>>>> a Spark job and Dremio job) means a distributed transaction. It would need
>>>>>>>>>>>>>> yet another external transaction coordinator on top of both Spark and
>>>>>>>>>>>>>> Dremio, Iceberg by itself
>>>>>>>>>>>>>> cannot solve this.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm not ready to accept this. Iceberg already supports a set
>>>>>>>>>>>>> of semantics around multiple writers committing simultaneously and how
>>>>>>>>>>>>> conflict resolution is done. The same can be done here.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> MVCC (which is what Iceberg tries to implement) requires a
>>>>>>>>>>>>> total ordering of snapshots. Also the snapshots need to be non-conflicting.
>>>>>>>>>>>>> I really don't see how any metadata data structures can solve this without
>>>>>>>>>>>>> an outside coordinator.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Consider this:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Snapshot 0: (K,A) = 1
>>>>>>>>>>>>> Job X: UPDATE K SET A=A+1
>>>>>>>>>>>>> Job Y: UPDATE K SET A=10
>>>>>>>>>>>>>
>>>>>>>>>>>>> What should the final value of A be and who decides ?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> By single writer, I don't mean single process, I mean
>>>>>>>>>>>>>> multiple coordinated processes like Spark executors coordinated by Spark
>>>>>>>>>>>>>> driver. The coordinator ensures that the data is pre-partitioned on
>>>>>>>>>>>>>> each executor, and the coordinator commits the snapshot.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Note however that single writer job/multiple concurrent
>>>>>>>>>>>>>> reader jobs is perfectly feasible, i.e. it shouldn't be a problem to write
>>>>>>>>>>>>>> from a Spark job and read from multiple Dremio queries concurrently (for
>>>>>>>>>>>>>> example)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> :D This is still "single process" from my perspective. That
>>>>>>>>>>>>> process may be coordinating other processes to do distributed work but
>>>>>>>>>>>>> ultimately it is a single process.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Fair enough
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm not sure what you mean exactly. If we can't enforce
>>>>>>>>>>>>>> uniqueness we shouldn't assume it.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I disagree. We can specify that as a requirement and state
>>>>>>>>>>>>> that you'll get unintended consequences if you provide your own keys and
>>>>>>>>>>>>> don't maintain this.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> There's no need for unintended consequences, we can specify
>>>>>>>>>>>>> consistent behaviour (and I believe the document says what that is)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> We do expect that most of the time the natural key is unique,
>>>>>>>>>>>>>> but the eager and lazy with natural key designs can handle duplicates
>>>>>>>>>>>>>> consistently. Basically it's not a problem to have duplicate
>>>>>>>>>>>>>> natural keys, everything works fine.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> That heavily depends on how things are implemented. For
>>>>>>>>>>>>> example, we may write a bunch of code that generates internal data
>>>>>>>>>>>>> structures based on this expectation. If we have to support duplicate
>>>>>>>>>>>>> matches, all of sudden we can no longer size various data structures to
>>>>>>>>>>>>> improve performance and may be unable to preallocate memory associated with
>>>>>>>>>>>>> a guaranteed completion.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Again we need to operate on the assumption that this is a
>>>>>>>>>>>>> large scale distributed compute/remote storage scenario. Key matching is
>>>>>>>>>>>>> done with shuffles with data movement across the network, such
>>>>>>>>>>>>> optimizations would really have little impact on overall performance. Not
>>>>>>>>>>>>> to mention that most query engines would already optimize the shuffle
>>>>>>>>>>>>> already as much as it can be optimized.
>>>>>>>>>>>>>
>>>>>>>>>>>>> It is true that if actual duplicate keys would make the key
>>>>>>>>>>>>> matching join (anti-join) somewhat more expensive, however it can be done
>>>>>>>>>>>>> in such a way that if the keys are in practice unique the join is as
>>>>>>>>>>>>> efficient as it can be.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Let me try and clarify each point:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - lookup for query or update on a non-(partition/bucket/sort)
>>>>>>>>>>>>>> key predicate implies scanning large amounts of data - because these are
>>>>>>>>>>>>>> the only data structures that can narrow down the lookup, right ? One could
>>>>>>>>>>>>>> argue that the min/max index (file skipping) can be applied to any column,
>>>>>>>>>>>>>> but in reality if that column is not sorted the min/max intervals can have
>>>>>>>>>>>>>> huge overlaps so it may be next to useless.
>>>>>>>>>>>>>> - remote storage - this is a critical architecture decision -
>>>>>>>>>>>>>> implementations on local storage imply a vastly different design for the
>>>>>>>>>>>>>> entire system, storage and compute.
>>>>>>>>>>>>>> - deleting single records per snapshot is unfeasible in eager
>>>>>>>>>>>>>> but also particularly in the lazy design: each deletion creates a very
>>>>>>>>>>>>>> small snapshot. Deleting 1 million records one at a time would create 1
>>>>>>>>>>>>>> million small files, and 1 million RPC calls.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Why is this unfeasible? If I have a dataset of 100mm files
>>>>>>>>>>>>> including 1mm small files, is that a major problem? It seems like your
>>>>>>>>>>>>> usecase isn't one where you want to support single record deletes but it is
>>>>>>>>>>>>> definitely something important to many people.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 100 mm total files or 1 mm files per dataset is definitely a
>>>>>>>>>>>>> problem on HDFS, and I believe on S3 too. Single key delete would work just
>>>>>>>>>>>>> fine, but it's simply not optimal to do that on remote storage. This is a
>>>>>>>>>>>>> very well known problem with HDFS, and one of the very reasons to have
>>>>>>>>>>>>> something like Iceberg in the first place.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Basically the users would be able to do single key mutation,
>>>>>>>>>>>>> but it's not the use case we should be optimizing for, but it's really not
>>>>>>>>>>>>> advisable.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Eager is conceptually just lazy + compaction done, well,
>>>>>>>>>>>>>> eagerly. The logic for both is exactly the same, the trade-off is just that
>>>>>>>>>>>>>> with eager you implicitly compact every time so that you don't do any work
>>>>>>>>>>>>>> on read, while with lazy
>>>>>>>>>>>>>> you want to amortize the cost of compaction over multiple
>>>>>>>>>>>>>> snapshots.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Basically there should be no difference between the two
>>>>>>>>>>>>>> conceptually, or with regard to keys, etc. The only difference is some
>>>>>>>>>>>>>> mechanics in implementation.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think you have deconstruct the problem too much to say these
>>>>>>>>>>>>> are the same (or at least that is what I'm starting to think given this
>>>>>>>>>>>>> thread). It seems like real world implementation decisions (per our
>>>>>>>>>>>>> discussion here) are in conflict. For example, you just argued against
>>>>>>>>>>>>> having a 1mm arbitrary mutations but I think that is because you aren't
>>>>>>>>>>>>> thinking about things over time with a delta implementation. Having 10,000
>>>>>>>>>>>>> mutations a day where we do delta compaction once a week
>>>>>>>>>>>>>
>>>>>>>>>>>>> and local file mappings (key to offset sparse bitmaps) seems
>>>>>>>>>>>>> like it could result in very good performance in a case where we're
>>>>>>>>>>>>> mutating small amounts of data. In this scenario, you may not do major
>>>>>>>>>>>>> compaction ever unless you get to a high enough percentage of records that
>>>>>>>>>>>>> have been deleted in the original dataset. That drives a very different set
>>>>>>>>>>>>> of implementation decisions from a situation where you're trying to restate
>>>>>>>>>>>>> an entire partition at once.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> We operate on 1 billion mutations per day at least. This is
>>>>>>>>>>>>> the problem Iceberg wants to solve, I believe it's stated upfront.
>>>>>>>>>>>>> 10000/day is not a big data problem. It can be done fairly trivially and it
>>>>>>>>>>>>> would be supported, but there's not much point in extra optimizing for this
>>>>>>>>>>>>> use case I believe.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Ryan Blue
>>>>>>>>>>> Software Engineer
>>>>>>>>>>> Netflix
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Ryan Blue
>>>>>>>>> Software Engineer
>>>>>>>>> Netflix
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Ryan Blue
>>>>>>>> Software Engineer
>>>>>>>> Netflix
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>>> Ryan Blue
>>>>> Software Engineer
>>>>> Netflix
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Ryan Blue
>>>> Software Engineer
>>>> Netflix
>>>>
>>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>

Re: Updates/Deletes/Upserts in Iceberg

[Jacques Nadeau <ja...@dremio.com>]

Yeah, I totally forgot to record our discussion. Will do so next time,
sorry.
--
Jacques Nadeau
CTO and Co-Founder, Dremio


On Wed, May 29, 2019 at 4:24 PM Ryan Blue <rb...@netflix.com> wrote:

> It wasn't recorded, but I can summarize what we talked about. Sorry I
> haven't sent this out earlier.
>
> We talked about the options and some of the background in Iceberg --
> basically that it isn't possible to determine the order of commits before
> you commit so you can't rely on some monotonically increasing value from a
> snapshot to know which deltas to apply to a file. The result is that we
> can't apply diffs to data files using a rule like "files older than X"
> because we can't identify those files without the snapshot history.
>
> That gives us basically 2 options for scoping delete diffs: either
> identify the files to apply a diff to when writing the diff, or log changes
> applied to a snapshot and keep the snapshot history around (which is how we
> know the order of snapshots). The first option is not good if you want to
> write without reading data to determine where the deleted records are. The
> second prevents cleaning up snapshot history.
>
> We also talked about whether we should encode IDs in data files. Jacques
> pointed out that retrying a commit is easier if you don't need to re-read
> the original data to reconcile changes. For example, if a data file was
> compacted in a concurrent write, how do we reconcile a delete for it? We
> discussed other options, like rolling back the compaction for delete
> events. I think that's a promising option.
>
> For action items, Jacques was going to think about whether we need to
> encode IDs in data files or if we could use positions to identify rows and
> write up a summary/proposal. Erik was going to take on planning how
> identifying rows without reading data would work and similarly write up a
> summary/proposal.
>
> That's from memory, so if I've missed anything, I hope that other
> attendees will fill in the details!
>
> rb
>
> On Wed, May 29, 2019 at 3:34 PM Venkatakrishnan Sowrirajan <
> vsowrira@asu.edu> wrote:
>
>> Hi Ryan,
>>
>> I couldn't attend the meeting. Just curious, if this is recorded by any
>> chance.
>>
>> Regards
>> Venkata krishnan
>>
>>
>> On Fri, May 24, 2019 at 8:49 AM Ryan Blue <rb...@netflix.com.invalid>
>> wrote:
>>
>>> Yes, I agree. I'll talk a little about a couple of the constraints of
>>> this as well.
>>>
>>> On Fri, May 24, 2019 at 5:52 AM Anton Okolnychyi <ao...@apple.com>
>>> wrote:
>>>
>>>> The agenda looks good to me. I think it would also make sense to
>>>> clarify the responsibilities of query engines and Iceberg. Not only in
>>>> terms of uniqueness, but also in terms of applying diffs on read, for
>>>> example.
>>>>
>>>> On 23 May 2019, at 01:59, Ryan Blue <rb...@netflix.com.INVALID> wrote:
>>>>
>>>> Here’s a rough agenda:
>>>>
>>>>    - Use cases: everyone come with a use case that you’d like to have
>>>>    supported. We’ll go around and introduce ourselves and our use cases.
>>>>    - Main topic: How should Iceberg identify rows that are deleted?
>>>>    - Side topics from my initial email, if we have time: should we use
>>>>    insert diffs, should we support dense and sparse formats, etc.
>>>>
>>>> The main topic I think we should discuss is: *How should Iceberg
>>>> identify rows that are deleted?*
>>>>
>>>> I’m phrasing it this way to avoid where I think we’re talking past one
>>>> another because we are making assumptions. The important thing is that
>>>> there are two main options:
>>>>
>>>>    - Filename and position, vs
>>>>    - Specific values of (few) columns in the data
>>>>
>>>> This phrasing also avoids discussing uniqueness constraints. Once we
>>>> get down to behavior, I think we agree. For example, I think we all agree
>>>> that uniqueness cannot be enforced in Iceberg.
>>>>
>>>> If uniqueness can’t be enforced in Iceberg, the main choice comes down
>>>> to how we identify rows that are deleted. If we use (filename, position)
>>>> then we know that there is only one row. On the other hand, if we use data
>>>> values to identify rows then a delete may identify more than one row
>>>> because there are no uniqueness guarantees. I think we also agree that if
>>>> there is more than one row identified, all of them should be deleted.
>>>>
>>>> At that point, there are trade-offs between the approaches:
>>>>
>>>>    - When identifying deleted rows by data values, situations like the
>>>>    one that Anton pointed out are possible.
>>>>    - Jacques also had a good point about concurrency. If at all
>>>>    possible, we want to be able to reconcile changes between concurrent
>>>>    commits without re-running an operation.
>>>>
>>>> Sound like a reasonable amount to talk through?
>>>>
>>>> rb
>>>>
>>>> On Wed, May 22, 2019 at 1:17 PM Erik Wright <er...@shopify.com>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Wed, May 22, 2019 at 4:04 PM Cristian Opris <
>>>>> copris@apple.com.invalid> wrote:
>>>>>
>>>>>> Agreed with Erik here, we're certainly not looking to build the
>>>>>> equivalent of a relational database, and for that matter not even that of a
>>>>>> local disk storage analytics database (like Vertica). Those are very
>>>>>> different designs with very different trade-offs and optimizations.
>>>>>>
>>>>>> We're looking to automate and optimize specific types of file
>>>>>> manipulation for large files on remote storage, while presenting that to
>>>>>> the user under the common SQL API for *bulk* data manipulation
>>>>>> (MERGE INTO)
>>>>>>
>>>>>
>>>>> What I would encourage is to decouple the storage model from the
>>>>> implementation of that API. If Iceberg has support for merge-on-read of
>>>>> upserts and deletes, in addition to its powerful support for partitioning,
>>>>> it will be easy for a higher-level application to implement those APIs
>>>>> given certain other constraints (that might not be appropriate to all
>>>>> applications).
>>>>>
>>>>> Myself and Miguel are out on Friday, but Anton should be able to
>>>>>> handle the discussion on our side.
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Cristian
>>>>>>
>>>>>>
>>>>>> On 22 May 2019, at 17:51, Erik Wright <
>>>>>> erik.wright@shopify.com.INVALID> wrote:
>>>>>>
>>>>>> We have two rows with the same natural key and we use that natural
>>>>>>> key in diff files:
>>>>>>> nk | col1 | col2
>>>>>>> 1 | 1 | 1
>>>>>>> 1 | 2 | 2
>>>>>>> Then we have a delete statement:
>>>>>>> DELETE FROM t WHERE col1 = 1
>>>>>>
>>>>>>
>>>>>> I think this example cuts to the point of the differences of
>>>>>> understanding. Does Iceberg want to be approaching the utility of a
>>>>>> relational database, against which I can execute complex update queries?
>>>>>> This is not what I would have imagined.
>>>>>>
>>>>>> I would have, instead, imagined that it was up to the client to
>>>>>> identify, through whatever means, that they want to update or delete a row
>>>>>> with a given ID. If there are multiple (distinct) rows with the same ID,
>>>>>> _too bad_. Any user should _expect_ that they could potentially see any one
>>>>>> or more of those rows at read time. And that an upsert/delete would affect
>>>>>> any/all of them (I would argue for all).
>>>>>>
>>>>>> *In summary:* Instead of trying to come up with a consistent,
>>>>>> logical handling for complex queries that are best suited for a relational
>>>>>> database, leave such handling up to the client and concentrate on problems
>>>>>> that can be solved simply and more generally.
>>>>>>
>>>>>> On Wed, May 22, 2019 at 12:11 PM Ryan Blue <rb...@netflix.com.invalid>
>>>>>> wrote:
>>>>>>
>>>>>>> Yes, I think we should. I was going to propose one after catching up
>>>>>>> on the rest of this thread today.
>>>>>>>
>>>>>>> On Wed, May 22, 2019 at 9:08 AM Anton Okolnychyi <
>>>>>>> aokolnychyi@apple.com> wrote:
>>>>>>>
>>>>>>>> Thanks! Would it make sense to discuss the agenda in advance?
>>>>>>>>
>>>>>>>> On 22 May 2019, at 17:04, Ryan Blue <rb...@netflix.com.INVALID>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> I sent out an invite and included everyone on this thread. If
>>>>>>>> anyone else would like to join, please join the Zoom meeting. If you'd like
>>>>>>>> to be added to the calendar invite, just let me know and I'll add you.
>>>>>>>>
>>>>>>>> On Wed, May 22, 2019 at 8:57 AM Jacques Nadeau <ja...@dremio.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> works for me.
>>>>>>>>>
>>>>>>>>> To make things easier, we can use my zoom meeting if people like:
>>>>>>>>>
>>>>>>>>> Join Zoom Meeting
>>>>>>>>> https://zoom.us/j/4157302092
>>>>>>>>>
>>>>>>>>> One tap mobile
>>>>>>>>> +16465588656,,4157302092# US (New York)
>>>>>>>>> +16699006833,,4157302092# US (San Jose)
>>>>>>>>>
>>>>>>>>> Dial by your location
>>>>>>>>>         +1 646 558 8656 US (New York)
>>>>>>>>>         +1 669 900 6833 US (San Jose)
>>>>>>>>>         877 853 5257 US Toll-free
>>>>>>>>>         888 475 4499 US Toll-free
>>>>>>>>> Meeting ID: 415 730 2092
>>>>>>>>> Find your local number: https://zoom.us/u/aH9XYBfm
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Jacques Nadeau
>>>>>>>>> CTO and Co-Founder, Dremio
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, May 22, 2019 at 8:54 AM Ryan Blue <
>>>>>>>>> rblue@netflix.com.invalid> wrote:
>>>>>>>>>
>>>>>>>>>> 9AM on Friday works best for me. How about then?
>>>>>>>>>>
>>>>>>>>>> On Wed, May 22, 2019 at 5:05 AM Anton Okolnychyi <
>>>>>>>>>> aokolnychyi@apple.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> What about this Friday? One hour slot from 9:00 to 10:00 am or
>>>>>>>>>>> 10:00 to 11:00 am PST? Some folks are based in London, so meeting later
>>>>>>>>>>> than this is hard. If Friday doesn’t work, we can consider Tuesday or
>>>>>>>>>>> Wednesday next week.
>>>>>>>>>>>
>>>>>>>>>>> On 22 May 2019, at 00:54, Jacques Nadeau <ja...@dremio.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> I agree with Anton that we should probably spend some time on
>>>>>>>>>>> hangouts further discussing things. Definitely differing expectations here
>>>>>>>>>>> and we seem to be talking a bit past each other.
>>>>>>>>>>> --
>>>>>>>>>>> Jacques Nadeau
>>>>>>>>>>> CTO and Co-Founder, Dremio
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, May 21, 2019 at 3:44 PM Cristian Opris <
>>>>>>>>>>> copris@apple.com.invalid> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I love a good flame war :P
>>>>>>>>>>>>
>>>>>>>>>>>> On 21 May 2019, at 22:57, Jacques Nadeau <ja...@dremio.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> That's my point, truly independent writers (two Spark jobs, or
>>>>>>>>>>>>> a Spark job and Dremio job) means a distributed transaction. It would need
>>>>>>>>>>>>> yet another external transaction coordinator on top of both Spark and
>>>>>>>>>>>>> Dremio, Iceberg by itself
>>>>>>>>>>>>> cannot solve this.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I'm not ready to accept this. Iceberg already supports a set of
>>>>>>>>>>>> semantics around multiple writers committing simultaneously and how
>>>>>>>>>>>> conflict resolution is done. The same can be done here.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> MVCC (which is what Iceberg tries to implement) requires a
>>>>>>>>>>>> total ordering of snapshots. Also the snapshots need to be non-conflicting.
>>>>>>>>>>>> I really don't see how any metadata data structures can solve this without
>>>>>>>>>>>> an outside coordinator.
>>>>>>>>>>>>
>>>>>>>>>>>> Consider this:
>>>>>>>>>>>>
>>>>>>>>>>>> Snapshot 0: (K,A) = 1
>>>>>>>>>>>> Job X: UPDATE K SET A=A+1
>>>>>>>>>>>> Job Y: UPDATE K SET A=10
>>>>>>>>>>>>
>>>>>>>>>>>> What should the final value of A be and who decides ?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> By single writer, I don't mean single process, I mean multiple
>>>>>>>>>>>>> coordinated processes like Spark executors coordinated by Spark driver. The
>>>>>>>>>>>>> coordinator ensures that the data is pre-partitioned on
>>>>>>>>>>>>> each executor, and the coordinator commits the snapshot.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Note however that single writer job/multiple concurrent reader
>>>>>>>>>>>>> jobs is perfectly feasible, i.e. it shouldn't be a problem to write from a
>>>>>>>>>>>>> Spark job and read from multiple Dremio queries concurrently (for example)
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> :D This is still "single process" from my perspective. That
>>>>>>>>>>>> process may be coordinating other processes to do distributed work but
>>>>>>>>>>>> ultimately it is a single process.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Fair enough
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> I'm not sure what you mean exactly. If we can't enforce
>>>>>>>>>>>>> uniqueness we shouldn't assume it.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I disagree. We can specify that as a requirement and state that
>>>>>>>>>>>> you'll get unintended consequences if you provide your own keys and don't
>>>>>>>>>>>> maintain this.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> There's no need for unintended consequences, we can specify
>>>>>>>>>>>> consistent behaviour (and I believe the document says what that is)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> We do expect that most of the time the natural key is unique,
>>>>>>>>>>>>> but the eager and lazy with natural key designs can handle duplicates
>>>>>>>>>>>>> consistently. Basically it's not a problem to have duplicate
>>>>>>>>>>>>> natural keys, everything works fine.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> That heavily depends on how things are implemented. For
>>>>>>>>>>>> example, we may write a bunch of code that generates internal data
>>>>>>>>>>>> structures based on this expectation. If we have to support duplicate
>>>>>>>>>>>> matches, all of sudden we can no longer size various data structures to
>>>>>>>>>>>> improve performance and may be unable to preallocate memory associated with
>>>>>>>>>>>> a guaranteed completion.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Again we need to operate on the assumption that this is a large
>>>>>>>>>>>> scale distributed compute/remote storage scenario. Key matching is done
>>>>>>>>>>>> with shuffles with data movement across the network, such optimizations
>>>>>>>>>>>> would really have little impact on overall performance. Not to mention that
>>>>>>>>>>>> most query engines would already optimize the shuffle already as much as it
>>>>>>>>>>>> can be optimized.
>>>>>>>>>>>>
>>>>>>>>>>>> It is true that if actual duplicate keys would make the key
>>>>>>>>>>>> matching join (anti-join) somewhat more expensive, however it can be done
>>>>>>>>>>>> in such a way that if the keys are in practice unique the join is as
>>>>>>>>>>>> efficient as it can be.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Let me try and clarify each point:
>>>>>>>>>>>>>
>>>>>>>>>>>>> - lookup for query or update on a non-(partition/bucket/sort)
>>>>>>>>>>>>> key predicate implies scanning large amounts of data - because these are
>>>>>>>>>>>>> the only data structures that can narrow down the lookup, right ? One could
>>>>>>>>>>>>> argue that the min/max index (file skipping) can be applied to any column,
>>>>>>>>>>>>> but in reality if that column is not sorted the min/max intervals can have
>>>>>>>>>>>>> huge overlaps so it may be next to useless.
>>>>>>>>>>>>> - remote storage - this is a critical architecture decision -
>>>>>>>>>>>>> implementations on local storage imply a vastly different design for the
>>>>>>>>>>>>> entire system, storage and compute.
>>>>>>>>>>>>> - deleting single records per snapshot is unfeasible in eager
>>>>>>>>>>>>> but also particularly in the lazy design: each deletion creates a very
>>>>>>>>>>>>> small snapshot. Deleting 1 million records one at a time would create 1
>>>>>>>>>>>>> million small files, and 1 million RPC calls.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Why is this unfeasible? If I have a dataset of 100mm files
>>>>>>>>>>>> including 1mm small files, is that a major problem? It seems like your
>>>>>>>>>>>> usecase isn't one where you want to support single record deletes but it is
>>>>>>>>>>>> definitely something important to many people.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 100 mm total files or 1 mm files per dataset is definitely a
>>>>>>>>>>>> problem on HDFS, and I believe on S3 too. Single key delete would work just
>>>>>>>>>>>> fine, but it's simply not optimal to do that on remote storage. This is a
>>>>>>>>>>>> very well known problem with HDFS, and one of the very reasons to have
>>>>>>>>>>>> something like Iceberg in the first place.
>>>>>>>>>>>>
>>>>>>>>>>>> Basically the users would be able to do single key mutation,
>>>>>>>>>>>> but it's not the use case we should be optimizing for, but it's really not
>>>>>>>>>>>> advisable.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Eager is conceptually just lazy + compaction done, well,
>>>>>>>>>>>>> eagerly. The logic for both is exactly the same, the trade-off is just that
>>>>>>>>>>>>> with eager you implicitly compact every time so that you don't do any work
>>>>>>>>>>>>> on read, while with lazy
>>>>>>>>>>>>> you want to amortize the cost of compaction over multiple
>>>>>>>>>>>>> snapshots.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Basically there should be no difference between the two
>>>>>>>>>>>>> conceptually, or with regard to keys, etc. The only difference is some
>>>>>>>>>>>>> mechanics in implementation.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I think you have deconstruct the problem too much to say these
>>>>>>>>>>>> are the same (or at least that is what I'm starting to think given this
>>>>>>>>>>>> thread). It seems like real world implementation decisions (per our
>>>>>>>>>>>> discussion here) are in conflict. For example, you just argued against
>>>>>>>>>>>> having a 1mm arbitrary mutations but I think that is because you aren't
>>>>>>>>>>>> thinking about things over time with a delta implementation. Having 10,000
>>>>>>>>>>>> mutations a day where we do delta compaction once a week
>>>>>>>>>>>>
>>>>>>>>>>>> and local file mappings (key to offset sparse bitmaps) seems
>>>>>>>>>>>> like it could result in very good performance in a case where we're
>>>>>>>>>>>> mutating small amounts of data. In this scenario, you may not do major
>>>>>>>>>>>> compaction ever unless you get to a high enough percentage of records that
>>>>>>>>>>>> have been deleted in the original dataset. That drives a very different set
>>>>>>>>>>>> of implementation decisions from a situation where you're trying to restate
>>>>>>>>>>>> an entire partition at once.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> We operate on 1 billion mutations per day at least. This is the
>>>>>>>>>>>> problem Iceberg wants to solve, I believe it's stated upfront. 10000/day is
>>>>>>>>>>>> not a big data problem. It can be done fairly trivially and it would be
>>>>>>>>>>>> supported, but there's not much point in extra optimizing for this use case
>>>>>>>>>>>> I believe.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Ryan Blue
>>>>>>>>>> Software Engineer
>>>>>>>>>> Netflix
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Ryan Blue
>>>>>>>> Software Engineer
>>>>>>>> Netflix
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Ryan Blue
>>>>>>> Software Engineer
>>>>>>> Netflix
>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Ryan Blue
>>>> Software Engineer
>>>> Netflix
>>>>
>>>>
>>>>
>>>
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>>
>>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: Updates/Deletes/Upserts in Iceberg

[Ryan Blue <rb...@netflix.com.INVALID>]

It wasn't recorded, but I can summarize what we talked about. Sorry I
haven't sent this out earlier.

We talked about the options and some of the background in Iceberg --
basically that it isn't possible to determine the order of commits before
you commit so you can't rely on some monotonically increasing value from a
snapshot to know which deltas to apply to a file. The result is that we
can't apply diffs to data files using a rule like "files older than X"
because we can't identify those files without the snapshot history.

That gives us basically 2 options for scoping delete diffs: either identify
the files to apply a diff to when writing the diff, or log changes applied
to a snapshot and keep the snapshot history around (which is how we know
the order of snapshots). The first option is not good if you want to write
without reading data to determine where the deleted records are. The second
prevents cleaning up snapshot history.

We also talked about whether we should encode IDs in data files. Jacques
pointed out that retrying a commit is easier if you don't need to re-read
the original data to reconcile changes. For example, if a data file was
compacted in a concurrent write, how do we reconcile a delete for it? We
discussed other options, like rolling back the compaction for delete
events. I think that's a promising option.

For action items, Jacques was going to think about whether we need to
encode IDs in data files or if we could use positions to identify rows and
write up a summary/proposal. Erik was going to take on planning how
identifying rows without reading data would work and similarly write up a
summary/proposal.

That's from memory, so if I've missed anything, I hope that other attendees
will fill in the details!

rb

On Wed, May 29, 2019 at 3:34 PM Venkatakrishnan Sowrirajan <vs...@asu.edu>
wrote:

> Hi Ryan,
>
> I couldn't attend the meeting. Just curious, if this is recorded by any
> chance.
>
> Regards
> Venkata krishnan
>
>
> On Fri, May 24, 2019 at 8:49 AM Ryan Blue <rb...@netflix.com.invalid>
> wrote:
>
>> Yes, I agree. I'll talk a little about a couple of the constraints of
>> this as well.
>>
>> On Fri, May 24, 2019 at 5:52 AM Anton Okolnychyi <ao...@apple.com>
>> wrote:
>>
>>> The agenda looks good to me. I think it would also make sense to clarify
>>> the responsibilities of query engines and Iceberg. Not only in terms of
>>> uniqueness, but also in terms of applying diffs on read, for example.
>>>
>>> On 23 May 2019, at 01:59, Ryan Blue <rb...@netflix.com.INVALID> wrote:
>>>
>>> Here’s a rough agenda:
>>>
>>>    - Use cases: everyone come with a use case that you’d like to have
>>>    supported. We’ll go around and introduce ourselves and our use cases.
>>>    - Main topic: How should Iceberg identify rows that are deleted?
>>>    - Side topics from my initial email, if we have time: should we use
>>>    insert diffs, should we support dense and sparse formats, etc.
>>>
>>> The main topic I think we should discuss is: *How should Iceberg
>>> identify rows that are deleted?*
>>>
>>> I’m phrasing it this way to avoid where I think we’re talking past one
>>> another because we are making assumptions. The important thing is that
>>> there are two main options:
>>>
>>>    - Filename and position, vs
>>>    - Specific values of (few) columns in the data
>>>
>>> This phrasing also avoids discussing uniqueness constraints. Once we get
>>> down to behavior, I think we agree. For example, I think we all agree that
>>> uniqueness cannot be enforced in Iceberg.
>>>
>>> If uniqueness can’t be enforced in Iceberg, the main choice comes down
>>> to how we identify rows that are deleted. If we use (filename, position)
>>> then we know that there is only one row. On the other hand, if we use data
>>> values to identify rows then a delete may identify more than one row
>>> because there are no uniqueness guarantees. I think we also agree that if
>>> there is more than one row identified, all of them should be deleted.
>>>
>>> At that point, there are trade-offs between the approaches:
>>>
>>>    - When identifying deleted rows by data values, situations like the
>>>    one that Anton pointed out are possible.
>>>    - Jacques also had a good point about concurrency. If at all
>>>    possible, we want to be able to reconcile changes between concurrent
>>>    commits without re-running an operation.
>>>
>>> Sound like a reasonable amount to talk through?
>>>
>>> rb
>>>
>>> On Wed, May 22, 2019 at 1:17 PM Erik Wright <er...@shopify.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Wed, May 22, 2019 at 4:04 PM Cristian Opris <
>>>> copris@apple.com.invalid> wrote:
>>>>
>>>>> Agreed with Erik here, we're certainly not looking to build the
>>>>> equivalent of a relational database, and for that matter not even that of a
>>>>> local disk storage analytics database (like Vertica). Those are very
>>>>> different designs with very different trade-offs and optimizations.
>>>>>
>>>>> We're looking to automate and optimize specific types of file
>>>>> manipulation for large files on remote storage, while presenting that to
>>>>> the user under the common SQL API for *bulk* data manipulation (MERGE
>>>>> INTO)
>>>>>
>>>>
>>>> What I would encourage is to decouple the storage model from the
>>>> implementation of that API. If Iceberg has support for merge-on-read of
>>>> upserts and deletes, in addition to its powerful support for partitioning,
>>>> it will be easy for a higher-level application to implement those APIs
>>>> given certain other constraints (that might not be appropriate to all
>>>> applications).
>>>>
>>>> Myself and Miguel are out on Friday, but Anton should be able to handle
>>>>> the discussion on our side.
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Cristian
>>>>>
>>>>>
>>>>> On 22 May 2019, at 17:51, Erik Wright <er...@shopify.com.INVALID>
>>>>> wrote:
>>>>>
>>>>> We have two rows with the same natural key and we use that natural key
>>>>>> in diff files:
>>>>>> nk | col1 | col2
>>>>>> 1 | 1 | 1
>>>>>> 1 | 2 | 2
>>>>>> Then we have a delete statement:
>>>>>> DELETE FROM t WHERE col1 = 1
>>>>>
>>>>>
>>>>> I think this example cuts to the point of the differences of
>>>>> understanding. Does Iceberg want to be approaching the utility of a
>>>>> relational database, against which I can execute complex update queries?
>>>>> This is not what I would have imagined.
>>>>>
>>>>> I would have, instead, imagined that it was up to the client to
>>>>> identify, through whatever means, that they want to update or delete a row
>>>>> with a given ID. If there are multiple (distinct) rows with the same ID,
>>>>> _too bad_. Any user should _expect_ that they could potentially see any one
>>>>> or more of those rows at read time. And that an upsert/delete would affect
>>>>> any/all of them (I would argue for all).
>>>>>
>>>>> *In summary:* Instead of trying to come up with a consistent, logical
>>>>> handling for complex queries that are best suited for a relational
>>>>> database, leave such handling up to the client and concentrate on problems
>>>>> that can be solved simply and more generally.
>>>>>
>>>>> On Wed, May 22, 2019 at 12:11 PM Ryan Blue <rb...@netflix.com.invalid>
>>>>> wrote:
>>>>>
>>>>>> Yes, I think we should. I was going to propose one after catching up
>>>>>> on the rest of this thread today.
>>>>>>
>>>>>> On Wed, May 22, 2019 at 9:08 AM Anton Okolnychyi <
>>>>>> aokolnychyi@apple.com> wrote:
>>>>>>
>>>>>>> Thanks! Would it make sense to discuss the agenda in advance?
>>>>>>>
>>>>>>> On 22 May 2019, at 17:04, Ryan Blue <rb...@netflix.com.INVALID>
>>>>>>> wrote:
>>>>>>>
>>>>>>> I sent out an invite and included everyone on this thread. If anyone
>>>>>>> else would like to join, please join the Zoom meeting. If you'd like to be
>>>>>>> added to the calendar invite, just let me know and I'll add you.
>>>>>>>
>>>>>>> On Wed, May 22, 2019 at 8:57 AM Jacques Nadeau <ja...@dremio.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> works for me.
>>>>>>>>
>>>>>>>> To make things easier, we can use my zoom meeting if people like:
>>>>>>>>
>>>>>>>> Join Zoom Meeting
>>>>>>>> https://zoom.us/j/4157302092
>>>>>>>>
>>>>>>>> One tap mobile
>>>>>>>> +16465588656,,4157302092# US (New York)
>>>>>>>> +16699006833,,4157302092# US (San Jose)
>>>>>>>>
>>>>>>>> Dial by your location
>>>>>>>>         +1 646 558 8656 US (New York)
>>>>>>>>         +1 669 900 6833 US (San Jose)
>>>>>>>>         877 853 5257 US Toll-free
>>>>>>>>         888 475 4499 US Toll-free
>>>>>>>> Meeting ID: 415 730 2092
>>>>>>>> Find your local number: https://zoom.us/u/aH9XYBfm
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Jacques Nadeau
>>>>>>>> CTO and Co-Founder, Dremio
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, May 22, 2019 at 8:54 AM Ryan Blue <
>>>>>>>> rblue@netflix.com.invalid> wrote:
>>>>>>>>
>>>>>>>>> 9AM on Friday works best for me. How about then?
>>>>>>>>>
>>>>>>>>> On Wed, May 22, 2019 at 5:05 AM Anton Okolnychyi <
>>>>>>>>> aokolnychyi@apple.com> wrote:
>>>>>>>>>
>>>>>>>>>> What about this Friday? One hour slot from 9:00 to 10:00 am or
>>>>>>>>>> 10:00 to 11:00 am PST? Some folks are based in London, so meeting later
>>>>>>>>>> than this is hard. If Friday doesn’t work, we can consider Tuesday or
>>>>>>>>>> Wednesday next week.
>>>>>>>>>>
>>>>>>>>>> On 22 May 2019, at 00:54, Jacques Nadeau <ja...@dremio.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> I agree with Anton that we should probably spend some time on
>>>>>>>>>> hangouts further discussing things. Definitely differing expectations here
>>>>>>>>>> and we seem to be talking a bit past each other.
>>>>>>>>>> --
>>>>>>>>>> Jacques Nadeau
>>>>>>>>>> CTO and Co-Founder, Dremio
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, May 21, 2019 at 3:44 PM Cristian Opris <
>>>>>>>>>> copris@apple.com.invalid> wrote:
>>>>>>>>>>
>>>>>>>>>>> I love a good flame war :P
>>>>>>>>>>>
>>>>>>>>>>> On 21 May 2019, at 22:57, Jacques Nadeau <ja...@dremio.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> That's my point, truly independent writers (two Spark jobs, or a
>>>>>>>>>>>> Spark job and Dremio job) means a distributed transaction. It would need
>>>>>>>>>>>> yet another external transaction coordinator on top of both Spark and
>>>>>>>>>>>> Dremio, Iceberg by itself
>>>>>>>>>>>> cannot solve this.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I'm not ready to accept this. Iceberg already supports a set of
>>>>>>>>>>> semantics around multiple writers committing simultaneously and how
>>>>>>>>>>> conflict resolution is done. The same can be done here.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> MVCC (which is what Iceberg tries to implement) requires a total
>>>>>>>>>>> ordering of snapshots. Also the snapshots need to be non-conflicting. I
>>>>>>>>>>> really don't see how any metadata data structures can solve this without an
>>>>>>>>>>> outside coordinator.
>>>>>>>>>>>
>>>>>>>>>>> Consider this:
>>>>>>>>>>>
>>>>>>>>>>> Snapshot 0: (K,A) = 1
>>>>>>>>>>> Job X: UPDATE K SET A=A+1
>>>>>>>>>>> Job Y: UPDATE K SET A=10
>>>>>>>>>>>
>>>>>>>>>>> What should the final value of A be and who decides ?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> By single writer, I don't mean single process, I mean multiple
>>>>>>>>>>>> coordinated processes like Spark executors coordinated by Spark driver. The
>>>>>>>>>>>> coordinator ensures that the data is pre-partitioned on
>>>>>>>>>>>> each executor, and the coordinator commits the snapshot.
>>>>>>>>>>>>
>>>>>>>>>>>> Note however that single writer job/multiple concurrent reader
>>>>>>>>>>>> jobs is perfectly feasible, i.e. it shouldn't be a problem to write from a
>>>>>>>>>>>> Spark job and read from multiple Dremio queries concurrently (for example)
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> :D This is still "single process" from my perspective. That
>>>>>>>>>>> process may be coordinating other processes to do distributed work but
>>>>>>>>>>> ultimately it is a single process.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Fair enough
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> I'm not sure what you mean exactly. If we can't enforce
>>>>>>>>>>>> uniqueness we shouldn't assume it.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I disagree. We can specify that as a requirement and state that
>>>>>>>>>>> you'll get unintended consequences if you provide your own keys and don't
>>>>>>>>>>> maintain this.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> There's no need for unintended consequences, we can specify
>>>>>>>>>>> consistent behaviour (and I believe the document says what that is)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> We do expect that most of the time the natural key is unique,
>>>>>>>>>>>> but the eager and lazy with natural key designs can handle duplicates
>>>>>>>>>>>> consistently. Basically it's not a problem to have duplicate
>>>>>>>>>>>> natural keys, everything works fine.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> That heavily depends on how things are implemented. For example,
>>>>>>>>>>> we may write a bunch of code that generates internal data structures based
>>>>>>>>>>> on this expectation. If we have to support duplicate matches, all of sudden
>>>>>>>>>>> we can no longer size various data structures to improve performance and
>>>>>>>>>>> may be unable to preallocate memory associated with a guaranteed completion.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Again we need to operate on the assumption that this is a large
>>>>>>>>>>> scale distributed compute/remote storage scenario. Key matching is done
>>>>>>>>>>> with shuffles with data movement across the network, such optimizations
>>>>>>>>>>> would really have little impact on overall performance. Not to mention that
>>>>>>>>>>> most query engines would already optimize the shuffle already as much as it
>>>>>>>>>>> can be optimized.
>>>>>>>>>>>
>>>>>>>>>>> It is true that if actual duplicate keys would make the key
>>>>>>>>>>> matching join (anti-join) somewhat more expensive, however it can be done
>>>>>>>>>>> in such a way that if the keys are in practice unique the join is as
>>>>>>>>>>> efficient as it can be.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Let me try and clarify each point:
>>>>>>>>>>>>
>>>>>>>>>>>> - lookup for query or update on a non-(partition/bucket/sort)
>>>>>>>>>>>> key predicate implies scanning large amounts of data - because these are
>>>>>>>>>>>> the only data structures that can narrow down the lookup, right ? One could
>>>>>>>>>>>> argue that the min/max index (file skipping) can be applied to any column,
>>>>>>>>>>>> but in reality if that column is not sorted the min/max intervals can have
>>>>>>>>>>>> huge overlaps so it may be next to useless.
>>>>>>>>>>>> - remote storage - this is a critical architecture decision -
>>>>>>>>>>>> implementations on local storage imply a vastly different design for the
>>>>>>>>>>>> entire system, storage and compute.
>>>>>>>>>>>> - deleting single records per snapshot is unfeasible in eager
>>>>>>>>>>>> but also particularly in the lazy design: each deletion creates a very
>>>>>>>>>>>> small snapshot. Deleting 1 million records one at a time would create 1
>>>>>>>>>>>> million small files, and 1 million RPC calls.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Why is this unfeasible? If I have a dataset of 100mm files
>>>>>>>>>>> including 1mm small files, is that a major problem? It seems like your
>>>>>>>>>>> usecase isn't one where you want to support single record deletes but it is
>>>>>>>>>>> definitely something important to many people.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 100 mm total files or 1 mm files per dataset is definitely a
>>>>>>>>>>> problem on HDFS, and I believe on S3 too. Single key delete would work just
>>>>>>>>>>> fine, but it's simply not optimal to do that on remote storage. This is a
>>>>>>>>>>> very well known problem with HDFS, and one of the very reasons to have
>>>>>>>>>>> something like Iceberg in the first place.
>>>>>>>>>>>
>>>>>>>>>>> Basically the users would be able to do single key mutation, but
>>>>>>>>>>> it's not the use case we should be optimizing for, but it's really not
>>>>>>>>>>> advisable.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Eager is conceptually just lazy + compaction done, well,
>>>>>>>>>>>> eagerly. The logic for both is exactly the same, the trade-off is just that
>>>>>>>>>>>> with eager you implicitly compact every time so that you don't do any work
>>>>>>>>>>>> on read, while with lazy
>>>>>>>>>>>> you want to amortize the cost of compaction over multiple
>>>>>>>>>>>> snapshots.
>>>>>>>>>>>>
>>>>>>>>>>>> Basically there should be no difference between the two
>>>>>>>>>>>> conceptually, or with regard to keys, etc. The only difference is some
>>>>>>>>>>>> mechanics in implementation.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I think you have deconstruct the problem too much to say these
>>>>>>>>>>> are the same (or at least that is what I'm starting to think given this
>>>>>>>>>>> thread). It seems like real world implementation decisions (per our
>>>>>>>>>>> discussion here) are in conflict. For example, you just argued against
>>>>>>>>>>> having a 1mm arbitrary mutations but I think that is because you aren't
>>>>>>>>>>> thinking about things over time with a delta implementation. Having 10,000
>>>>>>>>>>> mutations a day where we do delta compaction once a week
>>>>>>>>>>>
>>>>>>>>>>> and local file mappings (key to offset sparse bitmaps) seems
>>>>>>>>>>> like it could result in very good performance in a case where we're
>>>>>>>>>>> mutating small amounts of data. In this scenario, you may not do major
>>>>>>>>>>> compaction ever unless you get to a high enough percentage of records that
>>>>>>>>>>> have been deleted in the original dataset. That drives a very different set
>>>>>>>>>>> of implementation decisions from a situation where you're trying to restate
>>>>>>>>>>> an entire partition at once.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> We operate on 1 billion mutations per day at least. This is the
>>>>>>>>>>> problem Iceberg wants to solve, I believe it's stated upfront. 10000/day is
>>>>>>>>>>> not a big data problem. It can be done fairly trivially and it would be
>>>>>>>>>>> supported, but there's not much point in extra optimizing for this use case
>>>>>>>>>>> I believe.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Ryan Blue
>>>>>>>>> Software Engineer
>>>>>>>>> Netflix
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Ryan Blue
>>>>>>> Software Engineer
>>>>>>> Netflix
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Ryan Blue
>>>>>> Software Engineer
>>>>>> Netflix
>>>>>>
>>>>>
>>>>>
>>>
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>>
>>>
>>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>

-- 
Ryan Blue
Software Engineer
Netflix

Re: Updates/Deletes/Upserts in Iceberg

[Venkatakrishnan Sowrirajan <vs...@asu.edu>]

Hi Ryan,

I couldn't attend the meeting. Just curious, if this is recorded by any
chance.

Regards
Venkata krishnan


On Fri, May 24, 2019 at 8:49 AM Ryan Blue <rb...@netflix.com.invalid> wrote:

> Yes, I agree. I'll talk a little about a couple of the constraints of this
> as well.
>
> On Fri, May 24, 2019 at 5:52 AM Anton Okolnychyi <ao...@apple.com>
> wrote:
>
>> The agenda looks good to me. I think it would also make sense to clarify
>> the responsibilities of query engines and Iceberg. Not only in terms of
>> uniqueness, but also in terms of applying diffs on read, for example.
>>
>> On 23 May 2019, at 01:59, Ryan Blue <rb...@netflix.com.INVALID> wrote:
>>
>> Here’s a rough agenda:
>>
>>    - Use cases: everyone come with a use case that you’d like to have
>>    supported. We’ll go around and introduce ourselves and our use cases.
>>    - Main topic: How should Iceberg identify rows that are deleted?
>>    - Side topics from my initial email, if we have time: should we use
>>    insert diffs, should we support dense and sparse formats, etc.
>>
>> The main topic I think we should discuss is: *How should Iceberg
>> identify rows that are deleted?*
>>
>> I’m phrasing it this way to avoid where I think we’re talking past one
>> another because we are making assumptions. The important thing is that
>> there are two main options:
>>
>>    - Filename and position, vs
>>    - Specific values of (few) columns in the data
>>
>> This phrasing also avoids discussing uniqueness constraints. Once we get
>> down to behavior, I think we agree. For example, I think we all agree that
>> uniqueness cannot be enforced in Iceberg.
>>
>> If uniqueness can’t be enforced in Iceberg, the main choice comes down to
>> how we identify rows that are deleted. If we use (filename, position) then
>> we know that there is only one row. On the other hand, if we use data
>> values to identify rows then a delete may identify more than one row
>> because there are no uniqueness guarantees. I think we also agree that if
>> there is more than one row identified, all of them should be deleted.
>>
>> At that point, there are trade-offs between the approaches:
>>
>>    - When identifying deleted rows by data values, situations like the
>>    one that Anton pointed out are possible.
>>    - Jacques also had a good point about concurrency. If at all
>>    possible, we want to be able to reconcile changes between concurrent
>>    commits without re-running an operation.
>>
>> Sound like a reasonable amount to talk through?
>>
>> rb
>>
>> On Wed, May 22, 2019 at 1:17 PM Erik Wright <er...@shopify.com>
>> wrote:
>>
>>>
>>>
>>> On Wed, May 22, 2019 at 4:04 PM Cristian Opris <co...@apple.com.invalid>
>>> wrote:
>>>
>>>> Agreed with Erik here, we're certainly not looking to build the
>>>> equivalent of a relational database, and for that matter not even that of a
>>>> local disk storage analytics database (like Vertica). Those are very
>>>> different designs with very different trade-offs and optimizations.
>>>>
>>>> We're looking to automate and optimize specific types of file
>>>> manipulation for large files on remote storage, while presenting that to
>>>> the user under the common SQL API for *bulk* data manipulation (MERGE
>>>> INTO)
>>>>
>>>
>>> What I would encourage is to decouple the storage model from the
>>> implementation of that API. If Iceberg has support for merge-on-read of
>>> upserts and deletes, in addition to its powerful support for partitioning,
>>> it will be easy for a higher-level application to implement those APIs
>>> given certain other constraints (that might not be appropriate to all
>>> applications).
>>>
>>> Myself and Miguel are out on Friday, but Anton should be able to handle
>>>> the discussion on our side.
>>>>
>>>>
>>>> Thanks,
>>>> Cristian
>>>>
>>>>
>>>> On 22 May 2019, at 17:51, Erik Wright <er...@shopify.com.INVALID>
>>>> wrote:
>>>>
>>>> We have two rows with the same natural key and we use that natural key
>>>>> in diff files:
>>>>> nk | col1 | col2
>>>>> 1 | 1 | 1
>>>>> 1 | 2 | 2
>>>>> Then we have a delete statement:
>>>>> DELETE FROM t WHERE col1 = 1
>>>>
>>>>
>>>> I think this example cuts to the point of the differences of
>>>> understanding. Does Iceberg want to be approaching the utility of a
>>>> relational database, against which I can execute complex update queries?
>>>> This is not what I would have imagined.
>>>>
>>>> I would have, instead, imagined that it was up to the client to
>>>> identify, through whatever means, that they want to update or delete a row
>>>> with a given ID. If there are multiple (distinct) rows with the same ID,
>>>> _too bad_. Any user should _expect_ that they could potentially see any one
>>>> or more of those rows at read time. And that an upsert/delete would affect
>>>> any/all of them (I would argue for all).
>>>>
>>>> *In summary:* Instead of trying to come up with a consistent, logical
>>>> handling for complex queries that are best suited for a relational
>>>> database, leave such handling up to the client and concentrate on problems
>>>> that can be solved simply and more generally.
>>>>
>>>> On Wed, May 22, 2019 at 12:11 PM Ryan Blue <rb...@netflix.com.invalid>
>>>> wrote:
>>>>
>>>>> Yes, I think we should. I was going to propose one after catching up
>>>>> on the rest of this thread today.
>>>>>
>>>>> On Wed, May 22, 2019 at 9:08 AM Anton Okolnychyi <
>>>>> aokolnychyi@apple.com> wrote:
>>>>>
>>>>>> Thanks! Would it make sense to discuss the agenda in advance?
>>>>>>
>>>>>> On 22 May 2019, at 17:04, Ryan Blue <rb...@netflix.com.INVALID>
>>>>>> wrote:
>>>>>>
>>>>>> I sent out an invite and included everyone on this thread. If anyone
>>>>>> else would like to join, please join the Zoom meeting. If you'd like to be
>>>>>> added to the calendar invite, just let me know and I'll add you.
>>>>>>
>>>>>> On Wed, May 22, 2019 at 8:57 AM Jacques Nadeau <ja...@dremio.com>
>>>>>> wrote:
>>>>>>
>>>>>>> works for me.
>>>>>>>
>>>>>>> To make things easier, we can use my zoom meeting if people like:
>>>>>>>
>>>>>>> Join Zoom Meeting
>>>>>>> https://zoom.us/j/4157302092
>>>>>>>
>>>>>>> One tap mobile
>>>>>>> +16465588656,,4157302092# US (New York)
>>>>>>> +16699006833,,4157302092# US (San Jose)
>>>>>>>
>>>>>>> Dial by your location
>>>>>>>         +1 646 558 8656 US (New York)
>>>>>>>         +1 669 900 6833 US (San Jose)
>>>>>>>         877 853 5257 US Toll-free
>>>>>>>         888 475 4499 US Toll-free
>>>>>>> Meeting ID: 415 730 2092
>>>>>>> Find your local number: https://zoom.us/u/aH9XYBfm
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Jacques Nadeau
>>>>>>> CTO and Co-Founder, Dremio
>>>>>>>
>>>>>>>
>>>>>>> On Wed, May 22, 2019 at 8:54 AM Ryan Blue <rb...@netflix.com.invalid>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> 9AM on Friday works best for me. How about then?
>>>>>>>>
>>>>>>>> On Wed, May 22, 2019 at 5:05 AM Anton Okolnychyi <
>>>>>>>> aokolnychyi@apple.com> wrote:
>>>>>>>>
>>>>>>>>> What about this Friday? One hour slot from 9:00 to 10:00 am or
>>>>>>>>> 10:00 to 11:00 am PST? Some folks are based in London, so meeting later
>>>>>>>>> than this is hard. If Friday doesn’t work, we can consider Tuesday or
>>>>>>>>> Wednesday next week.
>>>>>>>>>
>>>>>>>>> On 22 May 2019, at 00:54, Jacques Nadeau <ja...@dremio.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> I agree with Anton that we should probably spend some time on
>>>>>>>>> hangouts further discussing things. Definitely differing expectations here
>>>>>>>>> and we seem to be talking a bit past each other.
>>>>>>>>> --
>>>>>>>>> Jacques Nadeau
>>>>>>>>> CTO and Co-Founder, Dremio
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, May 21, 2019 at 3:44 PM Cristian Opris <
>>>>>>>>> copris@apple.com.invalid> wrote:
>>>>>>>>>
>>>>>>>>>> I love a good flame war :P
>>>>>>>>>>
>>>>>>>>>> On 21 May 2019, at 22:57, Jacques Nadeau <ja...@dremio.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> That's my point, truly independent writers (two Spark jobs, or a
>>>>>>>>>>> Spark job and Dremio job) means a distributed transaction. It would need
>>>>>>>>>>> yet another external transaction coordinator on top of both Spark and
>>>>>>>>>>> Dremio, Iceberg by itself
>>>>>>>>>>> cannot solve this.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I'm not ready to accept this. Iceberg already supports a set of
>>>>>>>>>> semantics around multiple writers committing simultaneously and how
>>>>>>>>>> conflict resolution is done. The same can be done here.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> MVCC (which is what Iceberg tries to implement) requires a total
>>>>>>>>>> ordering of snapshots. Also the snapshots need to be non-conflicting. I
>>>>>>>>>> really don't see how any metadata data structures can solve this without an
>>>>>>>>>> outside coordinator.
>>>>>>>>>>
>>>>>>>>>> Consider this:
>>>>>>>>>>
>>>>>>>>>> Snapshot 0: (K,A) = 1
>>>>>>>>>> Job X: UPDATE K SET A=A+1
>>>>>>>>>> Job Y: UPDATE K SET A=10
>>>>>>>>>>
>>>>>>>>>> What should the final value of A be and who decides ?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> By single writer, I don't mean single process, I mean multiple
>>>>>>>>>>> coordinated processes like Spark executors coordinated by Spark driver. The
>>>>>>>>>>> coordinator ensures that the data is pre-partitioned on
>>>>>>>>>>> each executor, and the coordinator commits the snapshot.
>>>>>>>>>>>
>>>>>>>>>>> Note however that single writer job/multiple concurrent reader
>>>>>>>>>>> jobs is perfectly feasible, i.e. it shouldn't be a problem to write from a
>>>>>>>>>>> Spark job and read from multiple Dremio queries concurrently (for example)
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> :D This is still "single process" from my perspective. That
>>>>>>>>>> process may be coordinating other processes to do distributed work but
>>>>>>>>>> ultimately it is a single process.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Fair enough
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> I'm not sure what you mean exactly. If we can't enforce
>>>>>>>>>>> uniqueness we shouldn't assume it.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I disagree. We can specify that as a requirement and state that
>>>>>>>>>> you'll get unintended consequences if you provide your own keys and don't
>>>>>>>>>> maintain this.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> There's no need for unintended consequences, we can specify
>>>>>>>>>> consistent behaviour (and I believe the document says what that is)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> We do expect that most of the time the natural key is unique,
>>>>>>>>>>> but the eager and lazy with natural key designs can handle duplicates
>>>>>>>>>>> consistently. Basically it's not a problem to have duplicate
>>>>>>>>>>> natural keys, everything works fine.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> That heavily depends on how things are implemented. For example,
>>>>>>>>>> we may write a bunch of code that generates internal data structures based
>>>>>>>>>> on this expectation. If we have to support duplicate matches, all of sudden
>>>>>>>>>> we can no longer size various data structures to improve performance and
>>>>>>>>>> may be unable to preallocate memory associated with a guaranteed completion.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Again we need to operate on the assumption that this is a large
>>>>>>>>>> scale distributed compute/remote storage scenario. Key matching is done
>>>>>>>>>> with shuffles with data movement across the network, such optimizations
>>>>>>>>>> would really have little impact on overall performance. Not to mention that
>>>>>>>>>> most query engines would already optimize the shuffle already as much as it
>>>>>>>>>> can be optimized.
>>>>>>>>>>
>>>>>>>>>> It is true that if actual duplicate keys would make the key
>>>>>>>>>> matching join (anti-join) somewhat more expensive, however it can be done
>>>>>>>>>> in such a way that if the keys are in practice unique the join is as
>>>>>>>>>> efficient as it can be.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Let me try and clarify each point:
>>>>>>>>>>>
>>>>>>>>>>> - lookup for query or update on a non-(partition/bucket/sort)
>>>>>>>>>>> key predicate implies scanning large amounts of data - because these are
>>>>>>>>>>> the only data structures that can narrow down the lookup, right ? One could
>>>>>>>>>>> argue that the min/max index (file skipping) can be applied to any column,
>>>>>>>>>>> but in reality if that column is not sorted the min/max intervals can have
>>>>>>>>>>> huge overlaps so it may be next to useless.
>>>>>>>>>>> - remote storage - this is a critical architecture decision -
>>>>>>>>>>> implementations on local storage imply a vastly different design for the
>>>>>>>>>>> entire system, storage and compute.
>>>>>>>>>>> - deleting single records per snapshot is unfeasible in eager
>>>>>>>>>>> but also particularly in the lazy design: each deletion creates a very
>>>>>>>>>>> small snapshot. Deleting 1 million records one at a time would create 1
>>>>>>>>>>> million small files, and 1 million RPC calls.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Why is this unfeasible? If I have a dataset of 100mm files
>>>>>>>>>> including 1mm small files, is that a major problem? It seems like your
>>>>>>>>>> usecase isn't one where you want to support single record deletes but it is
>>>>>>>>>> definitely something important to many people.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 100 mm total files or 1 mm files per dataset is definitely a
>>>>>>>>>> problem on HDFS, and I believe on S3 too. Single key delete would work just
>>>>>>>>>> fine, but it's simply not optimal to do that on remote storage. This is a
>>>>>>>>>> very well known problem with HDFS, and one of the very reasons to have
>>>>>>>>>> something like Iceberg in the first place.
>>>>>>>>>>
>>>>>>>>>> Basically the users would be able to do single key mutation, but
>>>>>>>>>> it's not the use case we should be optimizing for, but it's really not
>>>>>>>>>> advisable.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Eager is conceptually just lazy + compaction done, well,
>>>>>>>>>>> eagerly. The logic for both is exactly the same, the trade-off is just that
>>>>>>>>>>> with eager you implicitly compact every time so that you don't do any work
>>>>>>>>>>> on read, while with lazy
>>>>>>>>>>> you want to amortize the cost of compaction over multiple
>>>>>>>>>>> snapshots.
>>>>>>>>>>>
>>>>>>>>>>> Basically there should be no difference between the two
>>>>>>>>>>> conceptually, or with regard to keys, etc. The only difference is some
>>>>>>>>>>> mechanics in implementation.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I think you have deconstruct the problem too much to say these
>>>>>>>>>> are the same (or at least that is what I'm starting to think given this
>>>>>>>>>> thread). It seems like real world implementation decisions (per our
>>>>>>>>>> discussion here) are in conflict. For example, you just argued against
>>>>>>>>>> having a 1mm arbitrary mutations but I think that is because you aren't
>>>>>>>>>> thinking about things over time with a delta implementation. Having 10,000
>>>>>>>>>> mutations a day where we do delta compaction once a week
>>>>>>>>>>
>>>>>>>>>> and local file mappings (key to offset sparse bitmaps) seems like
>>>>>>>>>> it could result in very good performance in a case where we're mutating
>>>>>>>>>> small amounts of data. In this scenario, you may not do major compaction
>>>>>>>>>> ever unless you get to a high enough percentage of records that have been
>>>>>>>>>> deleted in the original dataset. That drives a very different set of
>>>>>>>>>> implementation decisions from a situation where you're trying to restate an
>>>>>>>>>> entire partition at once.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> We operate on 1 billion mutations per day at least. This is the
>>>>>>>>>> problem Iceberg wants to solve, I believe it's stated upfront. 10000/day is
>>>>>>>>>> not a big data problem. It can be done fairly trivially and it would be
>>>>>>>>>> supported, but there's not much point in extra optimizing for this use case
>>>>>>>>>> I believe.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Ryan Blue
>>>>>>>> Software Engineer
>>>>>>>> Netflix
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Ryan Blue
>>>>>> Software Engineer
>>>>>> Netflix
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Ryan Blue
>>>>> Software Engineer
>>>>> Netflix
>>>>>
>>>>
>>>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>>
>>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: Updates/Deletes/Upserts in Iceberg

[Ryan Blue <rb...@netflix.com.INVALID>]

Yes, I agree. I'll talk a little about a couple of the constraints of this
as well.

On Fri, May 24, 2019 at 5:52 AM Anton Okolnychyi <ao...@apple.com>
wrote:

> The agenda looks good to me. I think it would also make sense to clarify
> the responsibilities of query engines and Iceberg. Not only in terms of
> uniqueness, but also in terms of applying diffs on read, for example.
>
> On 23 May 2019, at 01:59, Ryan Blue <rb...@netflix.com.INVALID> wrote:
>
> Here’s a rough agenda:
>
>    - Use cases: everyone come with a use case that you’d like to have
>    supported. We’ll go around and introduce ourselves and our use cases.
>    - Main topic: How should Iceberg identify rows that are deleted?
>    - Side topics from my initial email, if we have time: should we use
>    insert diffs, should we support dense and sparse formats, etc.
>
> The main topic I think we should discuss is: *How should Iceberg identify
> rows that are deleted?*
>
> I’m phrasing it this way to avoid where I think we’re talking past one
> another because we are making assumptions. The important thing is that
> there are two main options:
>
>    - Filename and position, vs
>    - Specific values of (few) columns in the data
>
> This phrasing also avoids discussing uniqueness constraints. Once we get
> down to behavior, I think we agree. For example, I think we all agree that
> uniqueness cannot be enforced in Iceberg.
>
> If uniqueness can’t be enforced in Iceberg, the main choice comes down to
> how we identify rows that are deleted. If we use (filename, position) then
> we know that there is only one row. On the other hand, if we use data
> values to identify rows then a delete may identify more than one row
> because there are no uniqueness guarantees. I think we also agree that if
> there is more than one row identified, all of them should be deleted.
>
> At that point, there are trade-offs between the approaches:
>
>    - When identifying deleted rows by data values, situations like the
>    one that Anton pointed out are possible.
>    - Jacques also had a good point about concurrency. If at all possible,
>    we want to be able to reconcile changes between concurrent commits without
>    re-running an operation.
>
> Sound like a reasonable amount to talk through?
>
> rb
>
> On Wed, May 22, 2019 at 1:17 PM Erik Wright <er...@shopify.com>
> wrote:
>
>>
>>
>> On Wed, May 22, 2019 at 4:04 PM Cristian Opris <co...@apple.com.invalid>
>> wrote:
>>
>>> Agreed with Erik here, we're certainly not looking to build the
>>> equivalent of a relational database, and for that matter not even that of a
>>> local disk storage analytics database (like Vertica). Those are very
>>> different designs with very different trade-offs and optimizations.
>>>
>>> We're looking to automate and optimize specific types of file
>>> manipulation for large files on remote storage, while presenting that to
>>> the user under the common SQL API for *bulk* data manipulation (MERGE
>>> INTO)
>>>
>>
>> What I would encourage is to decouple the storage model from the
>> implementation of that API. If Iceberg has support for merge-on-read of
>> upserts and deletes, in addition to its powerful support for partitioning,
>> it will be easy for a higher-level application to implement those APIs
>> given certain other constraints (that might not be appropriate to all
>> applications).
>>
>> Myself and Miguel are out on Friday, but Anton should be able to handle
>>> the discussion on our side.
>>>
>>>
>>> Thanks,
>>> Cristian
>>>
>>>
>>> On 22 May 2019, at 17:51, Erik Wright <er...@shopify.com.INVALID>
>>> wrote:
>>>
>>> We have two rows with the same natural key and we use that natural key
>>>> in diff files:
>>>> nk | col1 | col2
>>>> 1 | 1 | 1
>>>> 1 | 2 | 2
>>>> Then we have a delete statement:
>>>> DELETE FROM t WHERE col1 = 1
>>>
>>>
>>> I think this example cuts to the point of the differences of
>>> understanding. Does Iceberg want to be approaching the utility of a
>>> relational database, against which I can execute complex update queries?
>>> This is not what I would have imagined.
>>>
>>> I would have, instead, imagined that it was up to the client to
>>> identify, through whatever means, that they want to update or delete a row
>>> with a given ID. If there are multiple (distinct) rows with the same ID,
>>> _too bad_. Any user should _expect_ that they could potentially see any one
>>> or more of those rows at read time. And that an upsert/delete would affect
>>> any/all of them (I would argue for all).
>>>
>>> *In summary:* Instead of trying to come up with a consistent, logical
>>> handling for complex queries that are best suited for a relational
>>> database, leave such handling up to the client and concentrate on problems
>>> that can be solved simply and more generally.
>>>
>>> On Wed, May 22, 2019 at 12:11 PM Ryan Blue <rb...@netflix.com.invalid>
>>> wrote:
>>>
>>>> Yes, I think we should. I was going to propose one after catching up on
>>>> the rest of this thread today.
>>>>
>>>> On Wed, May 22, 2019 at 9:08 AM Anton Okolnychyi <ao...@apple.com>
>>>> wrote:
>>>>
>>>>> Thanks! Would it make sense to discuss the agenda in advance?
>>>>>
>>>>> On 22 May 2019, at 17:04, Ryan Blue <rb...@netflix.com.INVALID> wrote:
>>>>>
>>>>> I sent out an invite and included everyone on this thread. If anyone
>>>>> else would like to join, please join the Zoom meeting. If you'd like to be
>>>>> added to the calendar invite, just let me know and I'll add you.
>>>>>
>>>>> On Wed, May 22, 2019 at 8:57 AM Jacques Nadeau <ja...@dremio.com>
>>>>> wrote:
>>>>>
>>>>>> works for me.
>>>>>>
>>>>>> To make things easier, we can use my zoom meeting if people like:
>>>>>>
>>>>>> Join Zoom Meeting
>>>>>> https://zoom.us/j/4157302092
>>>>>>
>>>>>> One tap mobile
>>>>>> +16465588656,,4157302092# US (New York)
>>>>>> +16699006833,,4157302092# US (San Jose)
>>>>>>
>>>>>> Dial by your location
>>>>>>         +1 646 558 8656 US (New York)
>>>>>>         +1 669 900 6833 US (San Jose)
>>>>>>         877 853 5257 US Toll-free
>>>>>>         888 475 4499 US Toll-free
>>>>>> Meeting ID: 415 730 2092
>>>>>> Find your local number: https://zoom.us/u/aH9XYBfm
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Jacques Nadeau
>>>>>> CTO and Co-Founder, Dremio
>>>>>>
>>>>>>
>>>>>> On Wed, May 22, 2019 at 8:54 AM Ryan Blue <rb...@netflix.com.invalid>
>>>>>> wrote:
>>>>>>
>>>>>>> 9AM on Friday works best for me. How about then?
>>>>>>>
>>>>>>> On Wed, May 22, 2019 at 5:05 AM Anton Okolnychyi <
>>>>>>> aokolnychyi@apple.com> wrote:
>>>>>>>
>>>>>>>> What about this Friday? One hour slot from 9:00 to 10:00 am or
>>>>>>>> 10:00 to 11:00 am PST? Some folks are based in London, so meeting later
>>>>>>>> than this is hard. If Friday doesn’t work, we can consider Tuesday or
>>>>>>>> Wednesday next week.
>>>>>>>>
>>>>>>>> On 22 May 2019, at 00:54, Jacques Nadeau <ja...@dremio.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> I agree with Anton that we should probably spend some time on
>>>>>>>> hangouts further discussing things. Definitely differing expectations here
>>>>>>>> and we seem to be talking a bit past each other.
>>>>>>>> --
>>>>>>>> Jacques Nadeau
>>>>>>>> CTO and Co-Founder, Dremio
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, May 21, 2019 at 3:44 PM Cristian Opris <
>>>>>>>> copris@apple.com.invalid> wrote:
>>>>>>>>
>>>>>>>>> I love a good flame war :P
>>>>>>>>>
>>>>>>>>> On 21 May 2019, at 22:57, Jacques Nadeau <ja...@dremio.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> That's my point, truly independent writers (two Spark jobs, or a
>>>>>>>>>> Spark job and Dremio job) means a distributed transaction. It would need
>>>>>>>>>> yet another external transaction coordinator on top of both Spark and
>>>>>>>>>> Dremio, Iceberg by itself
>>>>>>>>>> cannot solve this.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'm not ready to accept this. Iceberg already supports a set of
>>>>>>>>> semantics around multiple writers committing simultaneously and how
>>>>>>>>> conflict resolution is done. The same can be done here.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> MVCC (which is what Iceberg tries to implement) requires a total
>>>>>>>>> ordering of snapshots. Also the snapshots need to be non-conflicting. I
>>>>>>>>> really don't see how any metadata data structures can solve this without an
>>>>>>>>> outside coordinator.
>>>>>>>>>
>>>>>>>>> Consider this:
>>>>>>>>>
>>>>>>>>> Snapshot 0: (K,A) = 1
>>>>>>>>> Job X: UPDATE K SET A=A+1
>>>>>>>>> Job Y: UPDATE K SET A=10
>>>>>>>>>
>>>>>>>>> What should the final value of A be and who decides ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> By single writer, I don't mean single process, I mean multiple
>>>>>>>>>> coordinated processes like Spark executors coordinated by Spark driver. The
>>>>>>>>>> coordinator ensures that the data is pre-partitioned on
>>>>>>>>>> each executor, and the coordinator commits the snapshot.
>>>>>>>>>>
>>>>>>>>>> Note however that single writer job/multiple concurrent reader
>>>>>>>>>> jobs is perfectly feasible, i.e. it shouldn't be a problem to write from a
>>>>>>>>>> Spark job and read from multiple Dremio queries concurrently (for example)
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> :D This is still "single process" from my perspective. That
>>>>>>>>> process may be coordinating other processes to do distributed work but
>>>>>>>>> ultimately it is a single process.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Fair enough
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> I'm not sure what you mean exactly. If we can't enforce
>>>>>>>>>> uniqueness we shouldn't assume it.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I disagree. We can specify that as a requirement and state that
>>>>>>>>> you'll get unintended consequences if you provide your own keys and don't
>>>>>>>>> maintain this.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> There's no need for unintended consequences, we can specify
>>>>>>>>> consistent behaviour (and I believe the document says what that is)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> We do expect that most of the time the natural key is unique, but
>>>>>>>>>> the eager and lazy with natural key designs can handle duplicates
>>>>>>>>>> consistently. Basically it's not a problem to have duplicate
>>>>>>>>>> natural keys, everything works fine.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> That heavily depends on how things are implemented. For example,
>>>>>>>>> we may write a bunch of code that generates internal data structures based
>>>>>>>>> on this expectation. If we have to support duplicate matches, all of sudden
>>>>>>>>> we can no longer size various data structures to improve performance and
>>>>>>>>> may be unable to preallocate memory associated with a guaranteed completion.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Again we need to operate on the assumption that this is a large
>>>>>>>>> scale distributed compute/remote storage scenario. Key matching is done
>>>>>>>>> with shuffles with data movement across the network, such optimizations
>>>>>>>>> would really have little impact on overall performance. Not to mention that
>>>>>>>>> most query engines would already optimize the shuffle already as much as it
>>>>>>>>> can be optimized.
>>>>>>>>>
>>>>>>>>> It is true that if actual duplicate keys would make the key
>>>>>>>>> matching join (anti-join) somewhat more expensive, however it can be done
>>>>>>>>> in such a way that if the keys are in practice unique the join is as
>>>>>>>>> efficient as it can be.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Let me try and clarify each point:
>>>>>>>>>>
>>>>>>>>>> - lookup for query or update on a non-(partition/bucket/sort) key
>>>>>>>>>> predicate implies scanning large amounts of data - because these are the
>>>>>>>>>> only data structures that can narrow down the lookup, right ? One could
>>>>>>>>>> argue that the min/max index (file skipping) can be applied to any column,
>>>>>>>>>> but in reality if that column is not sorted the min/max intervals can have
>>>>>>>>>> huge overlaps so it may be next to useless.
>>>>>>>>>> - remote storage - this is a critical architecture decision -
>>>>>>>>>> implementations on local storage imply a vastly different design for the
>>>>>>>>>> entire system, storage and compute.
>>>>>>>>>> - deleting single records per snapshot is unfeasible in eager but
>>>>>>>>>> also particularly in the lazy design: each deletion creates a very small
>>>>>>>>>> snapshot. Deleting 1 million records one at a time would create 1 million
>>>>>>>>>> small files, and 1 million RPC calls.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Why is this unfeasible? If I have a dataset of 100mm files
>>>>>>>>> including 1mm small files, is that a major problem? It seems like your
>>>>>>>>> usecase isn't one where you want to support single record deletes but it is
>>>>>>>>> definitely something important to many people.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 100 mm total files or 1 mm files per dataset is definitely a
>>>>>>>>> problem on HDFS, and I believe on S3 too. Single key delete would work just
>>>>>>>>> fine, but it's simply not optimal to do that on remote storage. This is a
>>>>>>>>> very well known problem with HDFS, and one of the very reasons to have
>>>>>>>>> something like Iceberg in the first place.
>>>>>>>>>
>>>>>>>>> Basically the users would be able to do single key mutation, but
>>>>>>>>> it's not the use case we should be optimizing for, but it's really not
>>>>>>>>> advisable.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Eager is conceptually just lazy + compaction done, well, eagerly.
>>>>>>>>>> The logic for both is exactly the same, the trade-off is just that with
>>>>>>>>>> eager you implicitly compact every time so that you don't do any work on
>>>>>>>>>> read, while with lazy
>>>>>>>>>> you want to amortize the cost of compaction over multiple
>>>>>>>>>> snapshots.
>>>>>>>>>>
>>>>>>>>>> Basically there should be no difference between the two
>>>>>>>>>> conceptually, or with regard to keys, etc. The only difference is some
>>>>>>>>>> mechanics in implementation.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I think you have deconstruct the problem too much to say these are
>>>>>>>>> the same (or at least that is what I'm starting to think given this
>>>>>>>>> thread). It seems like real world implementation decisions (per our
>>>>>>>>> discussion here) are in conflict. For example, you just argued against
>>>>>>>>> having a 1mm arbitrary mutations but I think that is because you aren't
>>>>>>>>> thinking about things over time with a delta implementation. Having 10,000
>>>>>>>>> mutations a day where we do delta compaction once a week
>>>>>>>>>
>>>>>>>>> and local file mappings (key to offset sparse bitmaps) seems like
>>>>>>>>> it could result in very good performance in a case where we're mutating
>>>>>>>>> small amounts of data. In this scenario, you may not do major compaction
>>>>>>>>> ever unless you get to a high enough percentage of records that have been
>>>>>>>>> deleted in the original dataset. That drives a very different set of
>>>>>>>>> implementation decisions from a situation where you're trying to restate an
>>>>>>>>> entire partition at once.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> We operate on 1 billion mutations per day at least. This is the
>>>>>>>>> problem Iceberg wants to solve, I believe it's stated upfront. 10000/day is
>>>>>>>>> not a big data problem. It can be done fairly trivially and it would be
>>>>>>>>> supported, but there's not much point in extra optimizing for this use case
>>>>>>>>> I believe.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Ryan Blue
>>>>>>> Software Engineer
>>>>>>> Netflix
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Ryan Blue
>>>>> Software Engineer
>>>>> Netflix
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Ryan Blue
>>>> Software Engineer
>>>> Netflix
>>>>
>>>
>>>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>
>
>

-- 
Ryan Blue
Software Engineer
Netflix

Re: Updates/Deletes/Upserts in Iceberg

[Anton Okolnychyi <ao...@apple.com.INVALID>]

The agenda looks good to me. I think it would also make sense to clarify the responsibilities of query engines and Iceberg. Not only in terms of uniqueness, but also in terms of applying diffs on read, for example. 

> On 23 May 2019, at 01:59, Ryan Blue <rb...@netflix.com.INVALID> wrote:
> 
> Here’s a rough agenda:
> 
> Use cases: everyone come with a use case that you’d like to have supported. We’ll go around and introduce ourselves and our use cases.
> Main topic: How should Iceberg identify rows that are deleted?
> Side topics from my initial email, if we have time: should we use insert diffs, should we support dense and sparse formats, etc.
> The main topic I think we should discuss is: How should Iceberg identify rows that are deleted?
> 
> I’m phrasing it this way to avoid where I think we’re talking past one another because we are making assumptions. The important thing is that there are two main options:
> 
> Filename and position, vs
> Specific values of (few) columns in the data
> This phrasing also avoids discussing uniqueness constraints. Once we get down to behavior, I think we agree. For example, I think we all agree that uniqueness cannot be enforced in Iceberg.
> 
> If uniqueness can’t be enforced in Iceberg, the main choice comes down to how we identify rows that are deleted. If we use (filename, position) then we know that there is only one row. On the other hand, if we use data values to identify rows then a delete may identify more than one row because there are no uniqueness guarantees. I think we also agree that if there is more than one row identified, all of them should be deleted.
> 
> At that point, there are trade-offs between the approaches:
> 
> When identifying deleted rows by data values, situations like the one that Anton pointed out are possible.
> Jacques also had a good point about concurrency. If at all possible, we want to be able to reconcile changes between concurrent commits without re-running an operation.
> Sound like a reasonable amount to talk through?
> 
> rb
> 
> 
> On Wed, May 22, 2019 at 1:17 PM Erik Wright <erik.wright@shopify.com <ma...@shopify.com>> wrote:
> 
> 
> On Wed, May 22, 2019 at 4:04 PM Cristian Opris <co...@apple.com.invalid> wrote:
> Agreed with Erik here, we're certainly not looking to build the equivalent of a relational database, and for that matter not even that of a local disk storage analytics database (like Vertica). Those are very 
> different designs with very different trade-offs and optimizations.
> 
> We're looking to automate and optimize specific types of file manipulation for large files on remote storage, while presenting that to the user under the common SQL API for bulk data manipulation (MERGE INTO)
> 
> What I would encourage is to decouple the storage model from the implementation of that API. If Iceberg has support for merge-on-read of upserts and deletes, in addition to its powerful support for partitioning, it will be easy for a higher-level application to implement those APIs given certain other constraints (that might not be appropriate to all applications).
> 
> Myself and Miguel are out on Friday, but Anton should be able to handle the discussion on our side.
> 
> 
> Thanks,
> Cristian
> 
> 
>> On 22 May 2019, at 17:51, Erik Wright <erik.wright@shopify.com.INVALID <ma...@shopify.com.INVALID>> wrote:
>> 
>> We have two rows with the same natural key and we use that natural key in diff files:
>> nk | col1 | col2
>> 1 | 1 | 1
>> 1 | 2 | 2
>> Then we have a delete statement:
>> DELETE FROM t WHERE col1 = 1
>> 
>> I think this example cuts to the point of the differences of understanding. Does Iceberg want to be approaching the utility of a relational database, against which I can execute complex update queries? This is not what I would have imagined.
>> 
>> I would have, instead, imagined that it was up to the client to identify, through whatever means, that they want to update or delete a row with a given ID. If there are multiple (distinct) rows with the same ID, _too bad_. Any user should _expect_ that they could potentially see any one or more of those rows at read time. And that an upsert/delete would affect any/all of them (I would argue for all).
>> 
>> In summary: Instead of trying to come up with a consistent, logical handling for complex queries that are best suited for a relational database, leave such handling up to the client and concentrate on problems that can be solved simply and more generally.
>> 
>> On Wed, May 22, 2019 at 12:11 PM Ryan Blue <rblue@netflix.com.invalid <ma...@netflix.com.invalid>> wrote:
>> Yes, I think we should. I was going to propose one after catching up on the rest of this thread today.
>> 
>> On Wed, May 22, 2019 at 9:08 AM Anton Okolnychyi <aokolnychyi@apple.com <ma...@apple.com>> wrote:
>> Thanks! Would it make sense to discuss the agenda in advance?
>> 
>>> On 22 May 2019, at 17:04, Ryan Blue <rblue@netflix.com.INVALID <ma...@netflix.com.INVALID>> wrote:
>>> 
>>> I sent out an invite and included everyone on this thread. If anyone else would like to join, please join the Zoom meeting. If you'd like to be added to the calendar invite, just let me know and I'll add you.
>>> 
>>> On Wed, May 22, 2019 at 8:57 AM Jacques Nadeau <jacques@dremio.com <ma...@dremio.com>> wrote:
>>> works for me.
>>> 
>>> To make things easier, we can use my zoom meeting if people like:
>>> 
>>> Join Zoom Meeting
>>> https://zoom.us/j/4157302092 <https://zoom.us/j/4157302092>
>>> 
>>> One tap mobile
>>> +16465588656,,4157302092# US (New York)
>>> +16699006833,,4157302092# US (San Jose)
>>> 
>>> Dial by your location
>>>         +1 646 558 8656 US (New York)
>>>         +1 669 900 6833 US (San Jose)
>>>         877 853 5257 US Toll-free
>>>         888 475 4499 US Toll-free
>>> Meeting ID: 415 730 2092
>>> Find your local number: https://zoom.us/u/aH9XYBfm <https://zoom.us/u/aH9XYBfm>
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> --
>>> Jacques Nadeau
>>> CTO and Co-Founder, Dremio
>>> 
>>> 
>>> On Wed, May 22, 2019 at 8:54 AM Ryan Blue <rblue@netflix.com.invalid <ma...@netflix.com.invalid>> wrote:
>>> 9AM on Friday works best for me. How about then?
>>> 
>>> On Wed, May 22, 2019 at 5:05 AM Anton Okolnychyi <aokolnychyi@apple.com <ma...@apple.com>> wrote:
>>> What about this Friday? One hour slot from 9:00 to 10:00 am or 10:00 to 11:00 am PST? Some folks are based in London, so meeting later than this is hard. If Friday doesn’t work, we can consider Tuesday or Wednesday next week.
>>> 
>>>> On 22 May 2019, at 00:54, Jacques Nadeau <jacques@dremio.com <ma...@dremio.com>> wrote:
>>>> 
>>>> I agree with Anton that we should probably spend some time on hangouts further discussing things. Definitely differing expectations here and we seem to be talking a bit past each other.
>>>> --
>>>> Jacques Nadeau
>>>> CTO and Co-Founder, Dremio
>>>> 
>>>> 
>>>> On Tue, May 21, 2019 at 3:44 PM Cristian Opris <copris@apple.com.invalid <ma...@apple.com.invalid>> wrote:
>>>> I love a good flame war :P
>>>> 
>>>>> On 21 May 2019, at 22:57, Jacques Nadeau <jacques@dremio.com <ma...@dremio.com>> wrote:
>>>>> 
>>>>> 
>>>>> That's my point, truly independent writers (two Spark jobs, or a Spark job and Dremio job) means a distributed transaction. It would need yet another external transaction coordinator on top of both Spark and Dremio, Iceberg by itself 
>>>>> cannot solve this.
>>>>>  
>>>>> I'm not ready to accept this. Iceberg already supports a set of semantics around multiple writers committing simultaneously and how conflict resolution is done. The same can be done here.
>>>> 
>>>> 
>>>> 
>>>> MVCC (which is what Iceberg tries to implement) requires a total ordering of snapshots. Also the snapshots need to be non-conflicting. I really don't see how any metadata data structures can solve this without an outside coordinator.
>>>> 
>>>> Consider this:
>>>> 
>>>> Snapshot 0: (K,A) = 1
>>>> Job X: UPDATE K SET A=A+1
>>>> Job Y: UPDATE K SET A=10
>>>> 
>>>> What should the final value of A be and who decides ?
>>>> 
>>>>>  
>>>>> By single writer, I don't mean single process, I mean multiple coordinated processes like Spark executors coordinated by Spark driver. The coordinator ensures that the data is pre-partitioned on 
>>>>> each executor, and the coordinator commits the snapshot. 
>>>>> 
>>>>> Note however that single writer job/multiple concurrent reader jobs is perfectly feasible, i.e. it shouldn't be a problem to write from a Spark job and read from multiple Dremio queries concurrently (for example)
>>>>> 
>>>>> :D This is still "single process" from my perspective. That process may be coordinating other processes to do distributed work but ultimately it is a single process. 
>>>> 
>>>> Fair enough
>>>> 
>>>>>  
>>>>> I'm not sure what you mean exactly. If we can't enforce uniqueness we shouldn't assume it.
>>>>>  
>>>>> I disagree. We can specify that as a requirement and state that you'll get unintended consequences if you provide your own keys and don't maintain this.
>>>> 
>>>> There's no need for unintended consequences, we can specify consistent behaviour (and I believe the document says what that is)
>>>> 
>>>> 
>>>>>  
>>>>> We do expect that most of the time the natural key is unique, but the eager and lazy with natural key designs can handle duplicates 
>>>>> consistently. Basically it's not a problem to have duplicate natural keys, everything works fine.
>>>>> 
>>>>> That heavily depends on how things are implemented. For example, we may write a bunch of code that generates internal data structures based on this expectation. If we have to support duplicate matches, all of sudden we can no longer size various data structures to improve performance and may be unable to preallocate memory associated with a guaranteed completion.
>>>> 
>>>> Again we need to operate on the assumption that this is a large scale distributed compute/remote storage scenario. Key matching is done with shuffles with data movement across the network, such optimizations would really have little impact on overall performance. Not to mention that most query engines would already optimize the shuffle already as much as it can be optimized.
>>>> 
>>>> It is true that if actual duplicate keys would make the key matching join (anti-join) somewhat more expensive, however it can be done in such a way that if the keys are in practice unique the join is as efficient as it can be.
>>>> 
>>>> 
>>>>> 
>>>>> Let me try and clarify each point:
>>>>> 
>>>>> - lookup for query or update on a non-(partition/bucket/sort) key predicate implies scanning large amounts of data - because these are the only data structures that can narrow down the lookup, right ? One could argue that the min/max index (file skipping) can be applied to any column, but in reality if that column is not sorted the min/max intervals can have huge overlaps so it may be next to useless.
>>>>> - remote storage - this is a critical architecture decision - implementations on local storage imply a vastly different design for the entire system, storage and compute. 
>>>>> - deleting single records per snapshot is unfeasible in eager but also particularly in the lazy design: each deletion creates a very small snapshot. Deleting 1 million records one at a time would create 1 million small files, and 1 million RPC calls.
>>>>> 
>>>>> Why is this unfeasible? If I have a dataset of 100mm files including 1mm small files, is that a major problem? It seems like your usecase isn't one where you want to support single record deletes but it is definitely something important to many people.
>>>> 
>>>> 100 mm total files or 1 mm files per dataset is definitely a problem on HDFS, and I believe on S3 too. Single key delete would work just fine, but it's simply not optimal to do that on remote storage. This is a very well known problem with HDFS, and one of the very reasons to have something like Iceberg in the first place.
>>>> 
>>>> Basically the users would be able to do single key mutation, but it's not the use case we should be optimizing for, but it's really not advisable.
>>>> 
>>>> 
>>>>>  
>>>>> Eager is conceptually just lazy + compaction done, well, eagerly. The logic for both is exactly the same, the trade-off is just that with eager you implicitly compact every time so that you don't do any work on read, while with lazy 
>>>>> you want to amortize the cost of compaction over multiple snapshots.
>>>>> 
>>>>> Basically there should be no difference between the two conceptually, or with regard to keys, etc. The only difference is some mechanics in implementation.
>>>>> 
>>>>> I think you have deconstruct the problem too much to say these are the same (or at least that is what I'm starting to think given this thread). It seems like real world implementation decisions (per our discussion here) are in conflict. For example, you just argued against having a 1mm arbitrary mutations but I think that is because you aren't thinking about things over time with a delta implementation. Having 10,000 mutations a day where we do delta compaction once a week
>>>>> and local file mappings (key to offset sparse bitmaps) seems like it could result in very good performance in a case where we're mutating small amounts of data. In this scenario, you may not do major compaction ever unless you get to a high enough percentage of records that have been deleted in the original dataset. That drives a very different set of implementation decisions from a situation where you're trying to restate an entire partition at once.
>>>> 
>>>> 
>>>> We operate on 1 billion mutations per day at least. This is the problem Iceberg wants to solve, I believe it's stated upfront. 10000/day is not a big data problem. It can be done fairly trivially and it would be supported, but there's not much point in extra optimizing for this use case I believe.
>>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>> 
>>> 
>>> -- 
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>> 
>> 
>> 
>> -- 
>> Ryan Blue
>> Software Engineer
>> Netflix
> 
> 
> 
> -- 
> Ryan Blue
> Software Engineer
> Netflix

Re: Updates/Deletes/Upserts in Iceberg

[Ryan Blue <rb...@netflix.com.INVALID>]

Here’s a rough agenda:

   - Use cases: everyone come with a use case that you’d like to have
   supported. We’ll go around and introduce ourselves and our use cases.
   - Main topic: How should Iceberg identify rows that are deleted?
   - Side topics from my initial email, if we have time: should we use
   insert diffs, should we support dense and sparse formats, etc.

The main topic I think we should discuss is: *How should Iceberg identify
rows that are deleted?*

I’m phrasing it this way to avoid where I think we’re talking past one
another because we are making assumptions. The important thing is that
there are two main options:

   - Filename and position, vs
   - Specific values of (few) columns in the data

This phrasing also avoids discussing uniqueness constraints. Once we get
down to behavior, I think we agree. For example, I think we all agree that
uniqueness cannot be enforced in Iceberg.

If uniqueness can’t be enforced in Iceberg, the main choice comes down to
how we identify rows that are deleted. If we use (filename, position) then
we know that there is only one row. On the other hand, if we use data
values to identify rows then a delete may identify more than one row
because there are no uniqueness guarantees. I think we also agree that if
there is more than one row identified, all of them should be deleted.

At that point, there are trade-offs between the approaches:

   - When identifying deleted rows by data values, situations like the one
   that Anton pointed out are possible.
   - Jacques also had a good point about concurrency. If at all possible,
   we want to be able to reconcile changes between concurrent commits without
   re-running an operation.

Sound like a reasonable amount to talk through?

rb

On Wed, May 22, 2019 at 1:17 PM Erik Wright <er...@shopify.com> wrote:

>
>
> On Wed, May 22, 2019 at 4:04 PM Cristian Opris <co...@apple.com.invalid>
> wrote:
>
>> Agreed with Erik here, we're certainly not looking to build the
>> equivalent of a relational database, and for that matter not even that of a
>> local disk storage analytics database (like Vertica). Those are very
>> different designs with very different trade-offs and optimizations.
>>
>> We're looking to automate and optimize specific types of file
>> manipulation for large files on remote storage, while presenting that to
>> the user under the common SQL API for *bulk* data manipulation (MERGE
>> INTO)
>>
>
> What I would encourage is to decouple the storage model from the
> implementation of that API. If Iceberg has support for merge-on-read of
> upserts and deletes, in addition to its powerful support for partitioning,
> it will be easy for a higher-level application to implement those APIs
> given certain other constraints (that might not be appropriate to all
> applications).
>
> Myself and Miguel are out on Friday, but Anton should be able to handle
>> the discussion on our side.
>>
>>
>> Thanks,
>> Cristian
>>
>>
>> On 22 May 2019, at 17:51, Erik Wright <er...@shopify.com.INVALID>
>> wrote:
>>
>> We have two rows with the same natural key and we use that natural key in
>>> diff files:
>>> nk | col1 | col2
>>> 1 | 1 | 1
>>> 1 | 2 | 2
>>> Then we have a delete statement:
>>> DELETE FROM t WHERE col1 = 1
>>
>>
>> I think this example cuts to the point of the differences of
>> understanding. Does Iceberg want to be approaching the utility of a
>> relational database, against which I can execute complex update queries?
>> This is not what I would have imagined.
>>
>> I would have, instead, imagined that it was up to the client to identify,
>> through whatever means, that they want to update or delete a row with a
>> given ID. If there are multiple (distinct) rows with the same ID, _too
>> bad_. Any user should _expect_ that they could potentially see any one or
>> more of those rows at read time. And that an upsert/delete would affect
>> any/all of them (I would argue for all).
>>
>> *In summary:* Instead of trying to come up with a consistent, logical
>> handling for complex queries that are best suited for a relational
>> database, leave such handling up to the client and concentrate on problems
>> that can be solved simply and more generally.
>>
>> On Wed, May 22, 2019 at 12:11 PM Ryan Blue <rb...@netflix.com.invalid>
>> wrote:
>>
>>> Yes, I think we should. I was going to propose one after catching up on
>>> the rest of this thread today.
>>>
>>> On Wed, May 22, 2019 at 9:08 AM Anton Okolnychyi <ao...@apple.com>
>>> wrote:
>>>
>>>> Thanks! Would it make sense to discuss the agenda in advance?
>>>>
>>>> On 22 May 2019, at 17:04, Ryan Blue <rb...@netflix.com.INVALID> wrote:
>>>>
>>>> I sent out an invite and included everyone on this thread. If anyone
>>>> else would like to join, please join the Zoom meeting. If you'd like to be
>>>> added to the calendar invite, just let me know and I'll add you.
>>>>
>>>> On Wed, May 22, 2019 at 8:57 AM Jacques Nadeau <ja...@dremio.com>
>>>> wrote:
>>>>
>>>>> works for me.
>>>>>
>>>>> To make things easier, we can use my zoom meeting if people like:
>>>>>
>>>>> Join Zoom Meeting
>>>>> https://zoom.us/j/4157302092
>>>>>
>>>>> One tap mobile
>>>>> +16465588656,,4157302092# US (New York)
>>>>> +16699006833,,4157302092# US (San Jose)
>>>>>
>>>>> Dial by your location
>>>>>         +1 646 558 8656 US (New York)
>>>>>         +1 669 900 6833 US (San Jose)
>>>>>         877 853 5257 US Toll-free
>>>>>         888 475 4499 US Toll-free
>>>>> Meeting ID: 415 730 2092
>>>>> Find your local number: https://zoom.us/u/aH9XYBfm
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Jacques Nadeau
>>>>> CTO and Co-Founder, Dremio
>>>>>
>>>>>
>>>>> On Wed, May 22, 2019 at 8:54 AM Ryan Blue <rb...@netflix.com.invalid>
>>>>> wrote:
>>>>>
>>>>>> 9AM on Friday works best for me. How about then?
>>>>>>
>>>>>> On Wed, May 22, 2019 at 5:05 AM Anton Okolnychyi <
>>>>>> aokolnychyi@apple.com> wrote:
>>>>>>
>>>>>>> What about this Friday? One hour slot from 9:00 to 10:00 am or 10:00
>>>>>>> to 11:00 am PST? Some folks are based in London, so meeting later than this
>>>>>>> is hard. If Friday doesn’t work, we can consider Tuesday or Wednesday next
>>>>>>> week.
>>>>>>>
>>>>>>> On 22 May 2019, at 00:54, Jacques Nadeau <ja...@dremio.com> wrote:
>>>>>>>
>>>>>>> I agree with Anton that we should probably spend some time on
>>>>>>> hangouts further discussing things. Definitely differing expectations here
>>>>>>> and we seem to be talking a bit past each other.
>>>>>>> --
>>>>>>> Jacques Nadeau
>>>>>>> CTO and Co-Founder, Dremio
>>>>>>>
>>>>>>>
>>>>>>> On Tue, May 21, 2019 at 3:44 PM Cristian Opris <
>>>>>>> copris@apple.com.invalid> wrote:
>>>>>>>
>>>>>>>> I love a good flame war :P
>>>>>>>>
>>>>>>>> On 21 May 2019, at 22:57, Jacques Nadeau <ja...@dremio.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> That's my point, truly independent writers (two Spark jobs, or a
>>>>>>>>> Spark job and Dremio job) means a distributed transaction. It would need
>>>>>>>>> yet another external transaction coordinator on top of both Spark and
>>>>>>>>> Dremio, Iceberg by itself
>>>>>>>>> cannot solve this.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I'm not ready to accept this. Iceberg already supports a set of
>>>>>>>> semantics around multiple writers committing simultaneously and how
>>>>>>>> conflict resolution is done. The same can be done here.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> MVCC (which is what Iceberg tries to implement) requires a total
>>>>>>>> ordering of snapshots. Also the snapshots need to be non-conflicting. I
>>>>>>>> really don't see how any metadata data structures can solve this without an
>>>>>>>> outside coordinator.
>>>>>>>>
>>>>>>>> Consider this:
>>>>>>>>
>>>>>>>> Snapshot 0: (K,A) = 1
>>>>>>>> Job X: UPDATE K SET A=A+1
>>>>>>>> Job Y: UPDATE K SET A=10
>>>>>>>>
>>>>>>>> What should the final value of A be and who decides ?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> By single writer, I don't mean single process, I mean multiple
>>>>>>>>> coordinated processes like Spark executors coordinated by Spark driver. The
>>>>>>>>> coordinator ensures that the data is pre-partitioned on
>>>>>>>>> each executor, and the coordinator commits the snapshot.
>>>>>>>>>
>>>>>>>>> Note however that single writer job/multiple concurrent reader
>>>>>>>>> jobs is perfectly feasible, i.e. it shouldn't be a problem to write from a
>>>>>>>>> Spark job and read from multiple Dremio queries concurrently (for example)
>>>>>>>>>
>>>>>>>>
>>>>>>>> :D This is still "single process" from my perspective. That process
>>>>>>>> may be coordinating other processes to do distributed work but ultimately
>>>>>>>> it is a single process.
>>>>>>>>
>>>>>>>>
>>>>>>>> Fair enough
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> I'm not sure what you mean exactly. If we can't enforce uniqueness
>>>>>>>>> we shouldn't assume it.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I disagree. We can specify that as a requirement and state that
>>>>>>>> you'll get unintended consequences if you provide your own keys and don't
>>>>>>>> maintain this.
>>>>>>>>
>>>>>>>>
>>>>>>>> There's no need for unintended consequences, we can specify
>>>>>>>> consistent behaviour (and I believe the document says what that is)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> We do expect that most of the time the natural key is unique, but
>>>>>>>>> the eager and lazy with natural key designs can handle duplicates
>>>>>>>>> consistently. Basically it's not a problem to have duplicate
>>>>>>>>> natural keys, everything works fine.
>>>>>>>>>
>>>>>>>>
>>>>>>>> That heavily depends on how things are implemented. For example, we
>>>>>>>> may write a bunch of code that generates internal data structures based on
>>>>>>>> this expectation. If we have to support duplicate matches, all of sudden we
>>>>>>>> can no longer size various data structures to improve performance and may
>>>>>>>> be unable to preallocate memory associated with a guaranteed completion.
>>>>>>>>
>>>>>>>>
>>>>>>>> Again we need to operate on the assumption that this is a large
>>>>>>>> scale distributed compute/remote storage scenario. Key matching is done
>>>>>>>> with shuffles with data movement across the network, such optimizations
>>>>>>>> would really have little impact on overall performance. Not to mention that
>>>>>>>> most query engines would already optimize the shuffle already as much as it
>>>>>>>> can be optimized.
>>>>>>>>
>>>>>>>> It is true that if actual duplicate keys would make the key
>>>>>>>> matching join (anti-join) somewhat more expensive, however it can be done
>>>>>>>> in such a way that if the keys are in practice unique the join is as
>>>>>>>> efficient as it can be.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Let me try and clarify each point:
>>>>>>>>>
>>>>>>>>> - lookup for query or update on a non-(partition/bucket/sort) key
>>>>>>>>> predicate implies scanning large amounts of data - because these are the
>>>>>>>>> only data structures that can narrow down the lookup, right ? One could
>>>>>>>>> argue that the min/max index (file skipping) can be applied to any column,
>>>>>>>>> but in reality if that column is not sorted the min/max intervals can have
>>>>>>>>> huge overlaps so it may be next to useless.
>>>>>>>>> - remote storage - this is a critical architecture decision -
>>>>>>>>> implementations on local storage imply a vastly different design for the
>>>>>>>>> entire system, storage and compute.
>>>>>>>>> - deleting single records per snapshot is unfeasible in eager but
>>>>>>>>> also particularly in the lazy design: each deletion creates a very small
>>>>>>>>> snapshot. Deleting 1 million records one at a time would create 1 million
>>>>>>>>> small files, and 1 million RPC calls.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Why is this unfeasible? If I have a dataset of 100mm files
>>>>>>>> including 1mm small files, is that a major problem? It seems like your
>>>>>>>> usecase isn't one where you want to support single record deletes but it is
>>>>>>>> definitely something important to many people.
>>>>>>>>
>>>>>>>>
>>>>>>>> 100 mm total files or 1 mm files per dataset is definitely a
>>>>>>>> problem on HDFS, and I believe on S3 too. Single key delete would work just
>>>>>>>> fine, but it's simply not optimal to do that on remote storage. This is a
>>>>>>>> very well known problem with HDFS, and one of the very reasons to have
>>>>>>>> something like Iceberg in the first place.
>>>>>>>>
>>>>>>>> Basically the users would be able to do single key mutation, but
>>>>>>>> it's not the use case we should be optimizing for, but it's really not
>>>>>>>> advisable.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Eager is conceptually just lazy + compaction done, well, eagerly.
>>>>>>>>> The logic for both is exactly the same, the trade-off is just that with
>>>>>>>>> eager you implicitly compact every time so that you don't do any work on
>>>>>>>>> read, while with lazy
>>>>>>>>> you want to amortize the cost of compaction over multiple
>>>>>>>>> snapshots.
>>>>>>>>>
>>>>>>>>> Basically there should be no difference between the two
>>>>>>>>> conceptually, or with regard to keys, etc. The only difference is some
>>>>>>>>> mechanics in implementation.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I think you have deconstruct the problem too much to say these are
>>>>>>>> the same (or at least that is what I'm starting to think given this
>>>>>>>> thread). It seems like real world implementation decisions (per our
>>>>>>>> discussion here) are in conflict. For example, you just argued against
>>>>>>>> having a 1mm arbitrary mutations but I think that is because you aren't
>>>>>>>> thinking about things over time with a delta implementation. Having 10,000
>>>>>>>> mutations a day where we do delta compaction once a week
>>>>>>>>
>>>>>>>> and local file mappings (key to offset sparse bitmaps) seems like
>>>>>>>> it could result in very good performance in a case where we're mutating
>>>>>>>> small amounts of data. In this scenario, you may not do major compaction
>>>>>>>> ever unless you get to a high enough percentage of records that have been
>>>>>>>> deleted in the original dataset. That drives a very different set of
>>>>>>>> implementation decisions from a situation where you're trying to restate an
>>>>>>>> entire partition at once.
>>>>>>>>
>>>>>>>>
>>>>>>>> We operate on 1 billion mutations per day at least. This is the
>>>>>>>> problem Iceberg wants to solve, I believe it's stated upfront. 10000/day is
>>>>>>>> not a big data problem. It can be done fairly trivially and it would be
>>>>>>>> supported, but there's not much point in extra optimizing for this use case
>>>>>>>> I believe.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Ryan Blue
>>>>>> Software Engineer
>>>>>> Netflix
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Ryan Blue
>>>> Software Engineer
>>>> Netflix
>>>>
>>>>
>>>>
>>>
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>>
>>
>>

-- 
Ryan Blue
Software Engineer
Netflix

Re: Updates/Deletes/Upserts in Iceberg

[Erik Wright <er...@shopify.com.INVALID>]

On Wed, May 22, 2019 at 4:04 PM Cristian Opris <co...@apple.com.invalid>
wrote:

> Agreed with Erik here, we're certainly not looking to build the equivalent
> of a relational database, and for that matter not even that of a local disk
> storage analytics database (like Vertica). Those are very
> different designs with very different trade-offs and optimizations.
>
> We're looking to automate and optimize specific types of file manipulation
> for large files on remote storage, while presenting that to the user under
> the common SQL API for *bulk* data manipulation (MERGE INTO)
>

What I would encourage is to decouple the storage model from the
implementation of that API. If Iceberg has support for merge-on-read of
upserts and deletes, in addition to its powerful support for partitioning,
it will be easy for a higher-level application to implement those APIs
given certain other constraints (that might not be appropriate to all
applications).

Myself and Miguel are out on Friday, but Anton should be able to handle the
> discussion on our side.
>
>
> Thanks,
> Cristian
>
>
> On 22 May 2019, at 17:51, Erik Wright <er...@shopify.com.INVALID>
> wrote:
>
> We have two rows with the same natural key and we use that natural key in
>> diff files:
>> nk | col1 | col2
>> 1 | 1 | 1
>> 1 | 2 | 2
>> Then we have a delete statement:
>> DELETE FROM t WHERE col1 = 1
>
>
> I think this example cuts to the point of the differences of
> understanding. Does Iceberg want to be approaching the utility of a
> relational database, against which I can execute complex update queries?
> This is not what I would have imagined.
>
> I would have, instead, imagined that it was up to the client to identify,
> through whatever means, that they want to update or delete a row with a
> given ID. If there are multiple (distinct) rows with the same ID, _too
> bad_. Any user should _expect_ that they could potentially see any one or
> more of those rows at read time. And that an upsert/delete would affect
> any/all of them (I would argue for all).
>
> *In summary:* Instead of trying to come up with a consistent, logical
> handling for complex queries that are best suited for a relational
> database, leave such handling up to the client and concentrate on problems
> that can be solved simply and more generally.
>
> On Wed, May 22, 2019 at 12:11 PM Ryan Blue <rb...@netflix.com.invalid>
> wrote:
>
>> Yes, I think we should. I was going to propose one after catching up on
>> the rest of this thread today.
>>
>> On Wed, May 22, 2019 at 9:08 AM Anton Okolnychyi <ao...@apple.com>
>> wrote:
>>
>>> Thanks! Would it make sense to discuss the agenda in advance?
>>>
>>> On 22 May 2019, at 17:04, Ryan Blue <rb...@netflix.com.INVALID> wrote:
>>>
>>> I sent out an invite and included everyone on this thread. If anyone
>>> else would like to join, please join the Zoom meeting. If you'd like to be
>>> added to the calendar invite, just let me know and I'll add you.
>>>
>>> On Wed, May 22, 2019 at 8:57 AM Jacques Nadeau <ja...@dremio.com>
>>> wrote:
>>>
>>>> works for me.
>>>>
>>>> To make things easier, we can use my zoom meeting if people like:
>>>>
>>>> Join Zoom Meeting
>>>> https://zoom.us/j/4157302092
>>>>
>>>> One tap mobile
>>>> +16465588656,,4157302092# US (New York)
>>>> +16699006833,,4157302092# US (San Jose)
>>>>
>>>> Dial by your location
>>>>         +1 646 558 8656 US (New York)
>>>>         +1 669 900 6833 US (San Jose)
>>>>         877 853 5257 US Toll-free
>>>>         888 475 4499 US Toll-free
>>>> Meeting ID: 415 730 2092
>>>> Find your local number: https://zoom.us/u/aH9XYBfm
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Jacques Nadeau
>>>> CTO and Co-Founder, Dremio
>>>>
>>>>
>>>> On Wed, May 22, 2019 at 8:54 AM Ryan Blue <rb...@netflix.com.invalid>
>>>> wrote:
>>>>
>>>>> 9AM on Friday works best for me. How about then?
>>>>>
>>>>> On Wed, May 22, 2019 at 5:05 AM Anton Okolnychyi <
>>>>> aokolnychyi@apple.com> wrote:
>>>>>
>>>>>> What about this Friday? One hour slot from 9:00 to 10:00 am or 10:00
>>>>>> to 11:00 am PST? Some folks are based in London, so meeting later than this
>>>>>> is hard. If Friday doesn’t work, we can consider Tuesday or Wednesday next
>>>>>> week.
>>>>>>
>>>>>> On 22 May 2019, at 00:54, Jacques Nadeau <ja...@dremio.com> wrote:
>>>>>>
>>>>>> I agree with Anton that we should probably spend some time on
>>>>>> hangouts further discussing things. Definitely differing expectations here
>>>>>> and we seem to be talking a bit past each other.
>>>>>> --
>>>>>> Jacques Nadeau
>>>>>> CTO and Co-Founder, Dremio
>>>>>>
>>>>>>
>>>>>> On Tue, May 21, 2019 at 3:44 PM Cristian Opris <
>>>>>> copris@apple.com.invalid> wrote:
>>>>>>
>>>>>>> I love a good flame war :P
>>>>>>>
>>>>>>> On 21 May 2019, at 22:57, Jacques Nadeau <ja...@dremio.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>> That's my point, truly independent writers (two Spark jobs, or a
>>>>>>>> Spark job and Dremio job) means a distributed transaction. It would need
>>>>>>>> yet another external transaction coordinator on top of both Spark and
>>>>>>>> Dremio, Iceberg by itself
>>>>>>>> cannot solve this.
>>>>>>>>
>>>>>>>
>>>>>>> I'm not ready to accept this. Iceberg already supports a set of
>>>>>>> semantics around multiple writers committing simultaneously and how
>>>>>>> conflict resolution is done. The same can be done here.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> MVCC (which is what Iceberg tries to implement) requires a total
>>>>>>> ordering of snapshots. Also the snapshots need to be non-conflicting. I
>>>>>>> really don't see how any metadata data structures can solve this without an
>>>>>>> outside coordinator.
>>>>>>>
>>>>>>> Consider this:
>>>>>>>
>>>>>>> Snapshot 0: (K,A) = 1
>>>>>>> Job X: UPDATE K SET A=A+1
>>>>>>> Job Y: UPDATE K SET A=10
>>>>>>>
>>>>>>> What should the final value of A be and who decides ?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> By single writer, I don't mean single process, I mean multiple
>>>>>>>> coordinated processes like Spark executors coordinated by Spark driver. The
>>>>>>>> coordinator ensures that the data is pre-partitioned on
>>>>>>>> each executor, and the coordinator commits the snapshot.
>>>>>>>>
>>>>>>>> Note however that single writer job/multiple concurrent reader jobs
>>>>>>>> is perfectly feasible, i.e. it shouldn't be a problem to write from a Spark
>>>>>>>> job and read from multiple Dremio queries concurrently (for example)
>>>>>>>>
>>>>>>>
>>>>>>> :D This is still "single process" from my perspective. That process
>>>>>>> may be coordinating other processes to do distributed work but ultimately
>>>>>>> it is a single process.
>>>>>>>
>>>>>>>
>>>>>>> Fair enough
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> I'm not sure what you mean exactly. If we can't enforce uniqueness
>>>>>>>> we shouldn't assume it.
>>>>>>>>
>>>>>>>
>>>>>>> I disagree. We can specify that as a requirement and state that
>>>>>>> you'll get unintended consequences if you provide your own keys and don't
>>>>>>> maintain this.
>>>>>>>
>>>>>>>
>>>>>>> There's no need for unintended consequences, we can specify
>>>>>>> consistent behaviour (and I believe the document says what that is)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> We do expect that most of the time the natural key is unique, but
>>>>>>>> the eager and lazy with natural key designs can handle duplicates
>>>>>>>> consistently. Basically it's not a problem to have duplicate
>>>>>>>> natural keys, everything works fine.
>>>>>>>>
>>>>>>>
>>>>>>> That heavily depends on how things are implemented. For example, we
>>>>>>> may write a bunch of code that generates internal data structures based on
>>>>>>> this expectation. If we have to support duplicate matches, all of sudden we
>>>>>>> can no longer size various data structures to improve performance and may
>>>>>>> be unable to preallocate memory associated with a guaranteed completion.
>>>>>>>
>>>>>>>
>>>>>>> Again we need to operate on the assumption that this is a large
>>>>>>> scale distributed compute/remote storage scenario. Key matching is done
>>>>>>> with shuffles with data movement across the network, such optimizations
>>>>>>> would really have little impact on overall performance. Not to mention that
>>>>>>> most query engines would already optimize the shuffle already as much as it
>>>>>>> can be optimized.
>>>>>>>
>>>>>>> It is true that if actual duplicate keys would make the key matching
>>>>>>> join (anti-join) somewhat more expensive, however it can be done in such a
>>>>>>> way that if the keys are in practice unique the join is as efficient as it
>>>>>>> can be.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Let me try and clarify each point:
>>>>>>>>
>>>>>>>> - lookup for query or update on a non-(partition/bucket/sort) key
>>>>>>>> predicate implies scanning large amounts of data - because these are the
>>>>>>>> only data structures that can narrow down the lookup, right ? One could
>>>>>>>> argue that the min/max index (file skipping) can be applied to any column,
>>>>>>>> but in reality if that column is not sorted the min/max intervals can have
>>>>>>>> huge overlaps so it may be next to useless.
>>>>>>>> - remote storage - this is a critical architecture decision -
>>>>>>>> implementations on local storage imply a vastly different design for the
>>>>>>>> entire system, storage and compute.
>>>>>>>> - deleting single records per snapshot is unfeasible in eager but
>>>>>>>> also particularly in the lazy design: each deletion creates a very small
>>>>>>>> snapshot. Deleting 1 million records one at a time would create 1 million
>>>>>>>> small files, and 1 million RPC calls.
>>>>>>>>
>>>>>>>
>>>>>>> Why is this unfeasible? If I have a dataset of 100mm files including
>>>>>>> 1mm small files, is that a major problem? It seems like your usecase isn't
>>>>>>> one where you want to support single record deletes but it is definitely
>>>>>>> something important to many people.
>>>>>>>
>>>>>>>
>>>>>>> 100 mm total files or 1 mm files per dataset is definitely a problem
>>>>>>> on HDFS, and I believe on S3 too. Single key delete would work just fine,
>>>>>>> but it's simply not optimal to do that on remote storage. This is a very
>>>>>>> well known problem with HDFS, and one of the very reasons to have something
>>>>>>> like Iceberg in the first place.
>>>>>>>
>>>>>>> Basically the users would be able to do single key mutation, but
>>>>>>> it's not the use case we should be optimizing for, but it's really not
>>>>>>> advisable.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Eager is conceptually just lazy + compaction done, well, eagerly.
>>>>>>>> The logic for both is exactly the same, the trade-off is just that with
>>>>>>>> eager you implicitly compact every time so that you don't do any work on
>>>>>>>> read, while with lazy
>>>>>>>> you want to amortize the cost of compaction over multiple snapshots.
>>>>>>>>
>>>>>>>> Basically there should be no difference between the two
>>>>>>>> conceptually, or with regard to keys, etc. The only difference is some
>>>>>>>> mechanics in implementation.
>>>>>>>>
>>>>>>>
>>>>>>> I think you have deconstruct the problem too much to say these are
>>>>>>> the same (or at least that is what I'm starting to think given this
>>>>>>> thread). It seems like real world implementation decisions (per our
>>>>>>> discussion here) are in conflict. For example, you just argued against
>>>>>>> having a 1mm arbitrary mutations but I think that is because you aren't
>>>>>>> thinking about things over time with a delta implementation. Having 10,000
>>>>>>> mutations a day where we do delta compaction once a week
>>>>>>>
>>>>>>> and local file mappings (key to offset sparse bitmaps) seems like it
>>>>>>> could result in very good performance in a case where we're mutating small
>>>>>>> amounts of data. In this scenario, you may not do major compaction ever
>>>>>>> unless you get to a high enough percentage of records that have been
>>>>>>> deleted in the original dataset. That drives a very different set of
>>>>>>> implementation decisions from a situation where you're trying to restate an
>>>>>>> entire partition at once.
>>>>>>>
>>>>>>>
>>>>>>> We operate on 1 billion mutations per day at least. This is the
>>>>>>> problem Iceberg wants to solve, I believe it's stated upfront. 10000/day is
>>>>>>> not a big data problem. It can be done fairly trivially and it would be
>>>>>>> supported, but there's not much point in extra optimizing for this use case
>>>>>>> I believe.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Ryan Blue
>>>>> Software Engineer
>>>>> Netflix
>>>>>
>>>>
>>>
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>>
>>>
>>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>
>

Re: Updates/Deletes/Upserts in Iceberg

[Cristian Opris <co...@apple.com.INVALID>]

Agreed with Erik here, we're certainly not looking to build the equivalent of a relational database, and for that matter not even that of a local disk storage analytics database (like Vertica). Those are very 
different designs with very different trade-offs and optimizations.

We're looking to automate and optimize specific types of file manipulation for large files on remote storage, while presenting that to the user under the common SQL API for bulk data manipulation (MERGE INTO)



Myself and Miguel are out on Friday, but Anton should be able to handle the discussion on our side.


Thanks,
Cristian


> On 22 May 2019, at 17:51, Erik Wright <er...@shopify.com.INVALID> wrote:
> 
> We have two rows with the same natural key and we use that natural key in diff files:
> nk | col1 | col2
> 1 | 1 | 1
> 1 | 2 | 2
> Then we have a delete statement:
> DELETE FROM t WHERE col1 = 1
> 
> I think this example cuts to the point of the differences of understanding. Does Iceberg want to be approaching the utility of a relational database, against which I can execute complex update queries? This is not what I would have imagined.
> 
> I would have, instead, imagined that it was up to the client to identify, through whatever means, that they want to update or delete a row with a given ID. If there are multiple (distinct) rows with the same ID, _too bad_. Any user should _expect_ that they could potentially see any one or more of those rows at read time. And that an upsert/delete would affect any/all of them (I would argue for all).
> 
> In summary: Instead of trying to come up with a consistent, logical handling for complex queries that are best suited for a relational database, leave such handling up to the client and concentrate on problems that can be solved simply and more generally.
> 
> On Wed, May 22, 2019 at 12:11 PM Ryan Blue <rb...@netflix.com.invalid> wrote:
> Yes, I think we should. I was going to propose one after catching up on the rest of this thread today.
> 
> On Wed, May 22, 2019 at 9:08 AM Anton Okolnychyi <aokolnychyi@apple.com <ma...@apple.com>> wrote:
> Thanks! Would it make sense to discuss the agenda in advance?
> 
>> On 22 May 2019, at 17:04, Ryan Blue <rblue@netflix.com.INVALID <ma...@netflix.com.INVALID>> wrote:
>> 
>> I sent out an invite and included everyone on this thread. If anyone else would like to join, please join the Zoom meeting. If you'd like to be added to the calendar invite, just let me know and I'll add you.
>> 
>> On Wed, May 22, 2019 at 8:57 AM Jacques Nadeau <jacques@dremio.com <ma...@dremio.com>> wrote:
>> works for me.
>> 
>> To make things easier, we can use my zoom meeting if people like:
>> 
>> Join Zoom Meeting
>> https://zoom.us/j/4157302092 <https://zoom.us/j/4157302092>
>> 
>> One tap mobile
>> +16465588656,,4157302092# US (New York)
>> +16699006833,,4157302092# US (San Jose)
>> 
>> Dial by your location
>>         +1 646 558 8656 US (New York)
>>         +1 669 900 6833 US (San Jose)
>>         877 853 5257 US Toll-free
>>         888 475 4499 US Toll-free
>> Meeting ID: 415 730 2092
>> Find your local number: https://zoom.us/u/aH9XYBfm <https://zoom.us/u/aH9XYBfm>
>> 
>> 
>> 
>> 
>> 
>> 
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>> 
>> 
>> On Wed, May 22, 2019 at 8:54 AM Ryan Blue <rblue@netflix.com.invalid <ma...@netflix.com.invalid>> wrote:
>> 9AM on Friday works best for me. How about then?
>> 
>> On Wed, May 22, 2019 at 5:05 AM Anton Okolnychyi <aokolnychyi@apple.com <ma...@apple.com>> wrote:
>> What about this Friday? One hour slot from 9:00 to 10:00 am or 10:00 to 11:00 am PST? Some folks are based in London, so meeting later than this is hard. If Friday doesn’t work, we can consider Tuesday or Wednesday next week.
>> 
>>> On 22 May 2019, at 00:54, Jacques Nadeau <jacques@dremio.com <ma...@dremio.com>> wrote:
>>> 
>>> I agree with Anton that we should probably spend some time on hangouts further discussing things. Definitely differing expectations here and we seem to be talking a bit past each other.
>>> --
>>> Jacques Nadeau
>>> CTO and Co-Founder, Dremio
>>> 
>>> 
>>> On Tue, May 21, 2019 at 3:44 PM Cristian Opris <copris@apple.com.invalid <ma...@apple.com.invalid>> wrote:
>>> I love a good flame war :P
>>> 
>>>> On 21 May 2019, at 22:57, Jacques Nadeau <jacques@dremio.com <ma...@dremio.com>> wrote:
>>>> 
>>>> 
>>>> That's my point, truly independent writers (two Spark jobs, or a Spark job and Dremio job) means a distributed transaction. It would need yet another external transaction coordinator on top of both Spark and Dremio, Iceberg by itself 
>>>> cannot solve this.
>>>>  
>>>> I'm not ready to accept this. Iceberg already supports a set of semantics around multiple writers committing simultaneously and how conflict resolution is done. The same can be done here.
>>> 
>>> 
>>> 
>>> MVCC (which is what Iceberg tries to implement) requires a total ordering of snapshots. Also the snapshots need to be non-conflicting. I really don't see how any metadata data structures can solve this without an outside coordinator.
>>> 
>>> Consider this:
>>> 
>>> Snapshot 0: (K,A) = 1
>>> Job X: UPDATE K SET A=A+1
>>> Job Y: UPDATE K SET A=10
>>> 
>>> What should the final value of A be and who decides ?
>>> 
>>>>  
>>>> By single writer, I don't mean single process, I mean multiple coordinated processes like Spark executors coordinated by Spark driver. The coordinator ensures that the data is pre-partitioned on 
>>>> each executor, and the coordinator commits the snapshot. 
>>>> 
>>>> Note however that single writer job/multiple concurrent reader jobs is perfectly feasible, i.e. it shouldn't be a problem to write from a Spark job and read from multiple Dremio queries concurrently (for example)
>>>> 
>>>> :D This is still "single process" from my perspective. That process may be coordinating other processes to do distributed work but ultimately it is a single process. 
>>> 
>>> Fair enough
>>> 
>>>>  
>>>> I'm not sure what you mean exactly. If we can't enforce uniqueness we shouldn't assume it.
>>>>  
>>>> I disagree. We can specify that as a requirement and state that you'll get unintended consequences if you provide your own keys and don't maintain this.
>>> 
>>> There's no need for unintended consequences, we can specify consistent behaviour (and I believe the document says what that is)
>>> 
>>> 
>>>>  
>>>> We do expect that most of the time the natural key is unique, but the eager and lazy with natural key designs can handle duplicates 
>>>> consistently. Basically it's not a problem to have duplicate natural keys, everything works fine.
>>>> 
>>>> That heavily depends on how things are implemented. For example, we may write a bunch of code that generates internal data structures based on this expectation. If we have to support duplicate matches, all of sudden we can no longer size various data structures to improve performance and may be unable to preallocate memory associated with a guaranteed completion.
>>> 
>>> Again we need to operate on the assumption that this is a large scale distributed compute/remote storage scenario. Key matching is done with shuffles with data movement across the network, such optimizations would really have little impact on overall performance. Not to mention that most query engines would already optimize the shuffle already as much as it can be optimized.
>>> 
>>> It is true that if actual duplicate keys would make the key matching join (anti-join) somewhat more expensive, however it can be done in such a way that if the keys are in practice unique the join is as efficient as it can be.
>>> 
>>> 
>>>> 
>>>> Let me try and clarify each point:
>>>> 
>>>> - lookup for query or update on a non-(partition/bucket/sort) key predicate implies scanning large amounts of data - because these are the only data structures that can narrow down the lookup, right ? One could argue that the min/max index (file skipping) can be applied to any column, but in reality if that column is not sorted the min/max intervals can have huge overlaps so it may be next to useless.
>>>> - remote storage - this is a critical architecture decision - implementations on local storage imply a vastly different design for the entire system, storage and compute. 
>>>> - deleting single records per snapshot is unfeasible in eager but also particularly in the lazy design: each deletion creates a very small snapshot. Deleting 1 million records one at a time would create 1 million small files, and 1 million RPC calls.
>>>> 
>>>> Why is this unfeasible? If I have a dataset of 100mm files including 1mm small files, is that a major problem? It seems like your usecase isn't one where you want to support single record deletes but it is definitely something important to many people.
>>> 
>>> 100 mm total files or 1 mm files per dataset is definitely a problem on HDFS, and I believe on S3 too. Single key delete would work just fine, but it's simply not optimal to do that on remote storage. This is a very well known problem with HDFS, and one of the very reasons to have something like Iceberg in the first place.
>>> 
>>> Basically the users would be able to do single key mutation, but it's not the use case we should be optimizing for, but it's really not advisable.
>>> 
>>> 
>>>>  
>>>> Eager is conceptually just lazy + compaction done, well, eagerly. The logic for both is exactly the same, the trade-off is just that with eager you implicitly compact every time so that you don't do any work on read, while with lazy 
>>>> you want to amortize the cost of compaction over multiple snapshots.
>>>> 
>>>> Basically there should be no difference between the two conceptually, or with regard to keys, etc. The only difference is some mechanics in implementation.
>>>> 
>>>> I think you have deconstruct the problem too much to say these are the same (or at least that is what I'm starting to think given this thread). It seems like real world implementation decisions (per our discussion here) are in conflict. For example, you just argued against having a 1mm arbitrary mutations but I think that is because you aren't thinking about things over time with a delta implementation. Having 10,000 mutations a day where we do delta compaction once a week
>>>> and local file mappings (key to offset sparse bitmaps) seems like it could result in very good performance in a case where we're mutating small amounts of data. In this scenario, you may not do major compaction ever unless you get to a high enough percentage of records that have been deleted in the original dataset. That drives a very different set of implementation decisions from a situation where you're trying to restate an entire partition at once.
>>> 
>>> 
>>> We operate on 1 billion mutations per day at least. This is the problem Iceberg wants to solve, I believe it's stated upfront. 10000/day is not a big data problem. It can be done fairly trivially and it would be supported, but there's not much point in extra optimizing for this use case I believe.
>>> 
>> 
>> 
>> 
>> -- 
>> Ryan Blue
>> Software Engineer
>> Netflix
>> 
>> 
>> -- 
>> Ryan Blue
>> Software Engineer
>> Netflix
> 
> 
> 
> -- 
> Ryan Blue
> Software Engineer
> Netflix

Re: Updates/Deletes/Upserts in Iceberg

[Erik Wright <er...@shopify.com.INVALID>]

>
> We have two rows with the same natural key and we use that natural key in
> diff files:
> nk | col1 | col2
> 1 | 1 | 1
> 1 | 2 | 2
> Then we have a delete statement:
> DELETE FROM t WHERE col1 = 1


I think this example cuts to the point of the differences of understanding.
Does Iceberg want to be approaching the utility of a relational database,
against which I can execute complex update queries? This is not what I
would have imagined.

I would have, instead, imagined that it was up to the client to identify,
through whatever means, that they want to update or delete a row with a
given ID. If there are multiple (distinct) rows with the same ID, _too
bad_. Any user should _expect_ that they could potentially see any one or
more of those rows at read time. And that an upsert/delete would affect
any/all of them (I would argue for all).

*In summary:* Instead of trying to come up with a consistent, logical
handling for complex queries that are best suited for a relational
database, leave such handling up to the client and concentrate on problems
that can be solved simply and more generally.

On Wed, May 22, 2019 at 12:11 PM Ryan Blue <rb...@netflix.com.invalid>
wrote:

> Yes, I think we should. I was going to propose one after catching up on
> the rest of this thread today.
>
> On Wed, May 22, 2019 at 9:08 AM Anton Okolnychyi <ao...@apple.com>
> wrote:
>
>> Thanks! Would it make sense to discuss the agenda in advance?
>>
>> On 22 May 2019, at 17:04, Ryan Blue <rb...@netflix.com.INVALID> wrote:
>>
>> I sent out an invite and included everyone on this thread. If anyone else
>> would like to join, please join the Zoom meeting. If you'd like to be added
>> to the calendar invite, just let me know and I'll add you.
>>
>> On Wed, May 22, 2019 at 8:57 AM Jacques Nadeau <ja...@dremio.com>
>> wrote:
>>
>>> works for me.
>>>
>>> To make things easier, we can use my zoom meeting if people like:
>>>
>>> Join Zoom Meeting
>>> https://zoom.us/j/4157302092
>>>
>>> One tap mobile
>>> +16465588656,,4157302092# US (New York)
>>> +16699006833,,4157302092# US (San Jose)
>>>
>>> Dial by your location
>>>         +1 646 558 8656 US (New York)
>>>         +1 669 900 6833 US (San Jose)
>>>         877 853 5257 US Toll-free
>>>         888 475 4499 US Toll-free
>>> Meeting ID: 415 730 2092
>>> Find your local number: https://zoom.us/u/aH9XYBfm
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Jacques Nadeau
>>> CTO and Co-Founder, Dremio
>>>
>>>
>>> On Wed, May 22, 2019 at 8:54 AM Ryan Blue <rb...@netflix.com.invalid>
>>> wrote:
>>>
>>>> 9AM on Friday works best for me. How about then?
>>>>
>>>> On Wed, May 22, 2019 at 5:05 AM Anton Okolnychyi <ao...@apple.com>
>>>> wrote:
>>>>
>>>>> What about this Friday? One hour slot from 9:00 to 10:00 am or 10:00
>>>>> to 11:00 am PST? Some folks are based in London, so meeting later than this
>>>>> is hard. If Friday doesn’t work, we can consider Tuesday or Wednesday next
>>>>> week.
>>>>>
>>>>> On 22 May 2019, at 00:54, Jacques Nadeau <ja...@dremio.com> wrote:
>>>>>
>>>>> I agree with Anton that we should probably spend some time on hangouts
>>>>> further discussing things. Definitely differing expectations here and we
>>>>> seem to be talking a bit past each other.
>>>>> --
>>>>> Jacques Nadeau
>>>>> CTO and Co-Founder, Dremio
>>>>>
>>>>>
>>>>> On Tue, May 21, 2019 at 3:44 PM Cristian Opris <
>>>>> copris@apple.com.invalid> wrote:
>>>>>
>>>>>> I love a good flame war :P
>>>>>>
>>>>>> On 21 May 2019, at 22:57, Jacques Nadeau <ja...@dremio.com> wrote:
>>>>>>
>>>>>>
>>>>>> That's my point, truly independent writers (two Spark jobs, or a
>>>>>>> Spark job and Dremio job) means a distributed transaction. It would need
>>>>>>> yet another external transaction coordinator on top of both Spark and
>>>>>>> Dremio, Iceberg by itself
>>>>>>> cannot solve this.
>>>>>>>
>>>>>>
>>>>>> I'm not ready to accept this. Iceberg already supports a set of
>>>>>> semantics around multiple writers committing simultaneously and how
>>>>>> conflict resolution is done. The same can be done here.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> MVCC (which is what Iceberg tries to implement) requires a total
>>>>>> ordering of snapshots. Also the snapshots need to be non-conflicting. I
>>>>>> really don't see how any metadata data structures can solve this without an
>>>>>> outside coordinator.
>>>>>>
>>>>>> Consider this:
>>>>>>
>>>>>> Snapshot 0: (K,A) = 1
>>>>>> Job X: UPDATE K SET A=A+1
>>>>>> Job Y: UPDATE K SET A=10
>>>>>>
>>>>>> What should the final value of A be and who decides ?
>>>>>>
>>>>>>
>>>>>>
>>>>>>> By single writer, I don't mean single process, I mean multiple
>>>>>>> coordinated processes like Spark executors coordinated by Spark driver. The
>>>>>>> coordinator ensures that the data is pre-partitioned on
>>>>>>> each executor, and the coordinator commits the snapshot.
>>>>>>>
>>>>>>> Note however that single writer job/multiple concurrent reader jobs
>>>>>>> is perfectly feasible, i.e. it shouldn't be a problem to write from a Spark
>>>>>>> job and read from multiple Dremio queries concurrently (for example)
>>>>>>>
>>>>>>
>>>>>> :D This is still "single process" from my perspective. That process
>>>>>> may be coordinating other processes to do distributed work but ultimately
>>>>>> it is a single process.
>>>>>>
>>>>>>
>>>>>> Fair enough
>>>>>>
>>>>>>
>>>>>>
>>>>>>> I'm not sure what you mean exactly. If we can't enforce uniqueness
>>>>>>> we shouldn't assume it.
>>>>>>>
>>>>>>
>>>>>> I disagree. We can specify that as a requirement and state that
>>>>>> you'll get unintended consequences if you provide your own keys and don't
>>>>>> maintain this.
>>>>>>
>>>>>>
>>>>>> There's no need for unintended consequences, we can specify
>>>>>> consistent behaviour (and I believe the document says what that is)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> We do expect that most of the time the natural key is unique, but
>>>>>>> the eager and lazy with natural key designs can handle duplicates
>>>>>>> consistently. Basically it's not a problem to have duplicate natural
>>>>>>> keys, everything works fine.
>>>>>>>
>>>>>>
>>>>>> That heavily depends on how things are implemented. For example, we
>>>>>> may write a bunch of code that generates internal data structures based on
>>>>>> this expectation. If we have to support duplicate matches, all of sudden we
>>>>>> can no longer size various data structures to improve performance and may
>>>>>> be unable to preallocate memory associated with a guaranteed completion.
>>>>>>
>>>>>>
>>>>>> Again we need to operate on the assumption that this is a large scale
>>>>>> distributed compute/remote storage scenario. Key matching is done with
>>>>>> shuffles with data movement across the network, such optimizations would
>>>>>> really have little impact on overall performance. Not to mention that most
>>>>>> query engines would already optimize the shuffle already as much as it can
>>>>>> be optimized.
>>>>>>
>>>>>> It is true that if actual duplicate keys would make the key matching
>>>>>> join (anti-join) somewhat more expensive, however it can be done in such a
>>>>>> way that if the keys are in practice unique the join is as efficient as it
>>>>>> can be.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Let me try and clarify each point:
>>>>>>>
>>>>>>> - lookup for query or update on a non-(partition/bucket/sort) key
>>>>>>> predicate implies scanning large amounts of data - because these are the
>>>>>>> only data structures that can narrow down the lookup, right ? One could
>>>>>>> argue that the min/max index (file skipping) can be applied to any column,
>>>>>>> but in reality if that column is not sorted the min/max intervals can have
>>>>>>> huge overlaps so it may be next to useless.
>>>>>>> - remote storage - this is a critical architecture decision -
>>>>>>> implementations on local storage imply a vastly different design for the
>>>>>>> entire system, storage and compute.
>>>>>>> - deleting single records per snapshot is unfeasible in eager but
>>>>>>> also particularly in the lazy design: each deletion creates a very small
>>>>>>> snapshot. Deleting 1 million records one at a time would create 1 million
>>>>>>> small files, and 1 million RPC calls.
>>>>>>>
>>>>>>
>>>>>> Why is this unfeasible? If I have a dataset of 100mm files including
>>>>>> 1mm small files, is that a major problem? It seems like your usecase isn't
>>>>>> one where you want to support single record deletes but it is definitely
>>>>>> something important to many people.
>>>>>>
>>>>>>
>>>>>> 100 mm total files or 1 mm files per dataset is definitely a problem
>>>>>> on HDFS, and I believe on S3 too. Single key delete would work just fine,
>>>>>> but it's simply not optimal to do that on remote storage. This is a very
>>>>>> well known problem with HDFS, and one of the very reasons to have something
>>>>>> like Iceberg in the first place.
>>>>>>
>>>>>> Basically the users would be able to do single key mutation, but it's
>>>>>> not the use case we should be optimizing for, but it's really not advisable.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Eager is conceptually just lazy + compaction done, well, eagerly.
>>>>>>> The logic for both is exactly the same, the trade-off is just that with
>>>>>>> eager you implicitly compact every time so that you don't do any work on
>>>>>>> read, while with lazy
>>>>>>> you want to amortize the cost of compaction over multiple snapshots.
>>>>>>>
>>>>>>> Basically there should be no difference between the two
>>>>>>> conceptually, or with regard to keys, etc. The only difference is some
>>>>>>> mechanics in implementation.
>>>>>>>
>>>>>>
>>>>>> I think you have deconstruct the problem too much to say these are
>>>>>> the same (or at least that is what I'm starting to think given this
>>>>>> thread). It seems like real world implementation decisions (per our
>>>>>> discussion here) are in conflict. For example, you just argued against
>>>>>> having a 1mm arbitrary mutations but I think that is because you aren't
>>>>>> thinking about things over time with a delta implementation. Having 10,000
>>>>>> mutations a day where we do delta compaction once a week
>>>>>>
>>>>>> and local file mappings (key to offset sparse bitmaps) seems like it
>>>>>> could result in very good performance in a case where we're mutating small
>>>>>> amounts of data. In this scenario, you may not do major compaction ever
>>>>>> unless you get to a high enough percentage of records that have been
>>>>>> deleted in the original dataset. That drives a very different set of
>>>>>> implementation decisions from a situation where you're trying to restate an
>>>>>> entire partition at once.
>>>>>>
>>>>>>
>>>>>> We operate on 1 billion mutations per day at least. This is the
>>>>>> problem Iceberg wants to solve, I believe it's stated upfront. 10000/day is
>>>>>> not a big data problem. It can be done fairly trivially and it would be
>>>>>> supported, but there's not much point in extra optimizing for this use case
>>>>>> I believe.
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Ryan Blue
>>>> Software Engineer
>>>> Netflix
>>>>
>>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>>
>>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: Updates/Deletes/Upserts in Iceberg

[Ryan Blue <rb...@netflix.com.INVALID>]

Yes, I think we should. I was going to propose one after catching up on the
rest of this thread today.

On Wed, May 22, 2019 at 9:08 AM Anton Okolnychyi <ao...@apple.com>
wrote:

> Thanks! Would it make sense to discuss the agenda in advance?
>
> On 22 May 2019, at 17:04, Ryan Blue <rb...@netflix.com.INVALID> wrote:
>
> I sent out an invite and included everyone on this thread. If anyone else
> would like to join, please join the Zoom meeting. If you'd like to be added
> to the calendar invite, just let me know and I'll add you.
>
> On Wed, May 22, 2019 at 8:57 AM Jacques Nadeau <ja...@dremio.com> wrote:
>
>> works for me.
>>
>> To make things easier, we can use my zoom meeting if people like:
>>
>> Join Zoom Meeting
>> https://zoom.us/j/4157302092
>>
>> One tap mobile
>> +16465588656,,4157302092# US (New York)
>> +16699006833,,4157302092# US (San Jose)
>>
>> Dial by your location
>>         +1 646 558 8656 US (New York)
>>         +1 669 900 6833 US (San Jose)
>>         877 853 5257 US Toll-free
>>         888 475 4499 US Toll-free
>> Meeting ID: 415 730 2092
>> Find your local number: https://zoom.us/u/aH9XYBfm
>>
>>
>>
>>
>>
>>
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>>
>>
>> On Wed, May 22, 2019 at 8:54 AM Ryan Blue <rb...@netflix.com.invalid>
>> wrote:
>>
>>> 9AM on Friday works best for me. How about then?
>>>
>>> On Wed, May 22, 2019 at 5:05 AM Anton Okolnychyi <ao...@apple.com>
>>> wrote:
>>>
>>>> What about this Friday? One hour slot from 9:00 to 10:00 am or 10:00 to
>>>> 11:00 am PST? Some folks are based in London, so meeting later than this is
>>>> hard. If Friday doesn’t work, we can consider Tuesday or Wednesday next
>>>> week.
>>>>
>>>> On 22 May 2019, at 00:54, Jacques Nadeau <ja...@dremio.com> wrote:
>>>>
>>>> I agree with Anton that we should probably spend some time on hangouts
>>>> further discussing things. Definitely differing expectations here and we
>>>> seem to be talking a bit past each other.
>>>> --
>>>> Jacques Nadeau
>>>> CTO and Co-Founder, Dremio
>>>>
>>>>
>>>> On Tue, May 21, 2019 at 3:44 PM Cristian Opris <
>>>> copris@apple.com.invalid> wrote:
>>>>
>>>>> I love a good flame war :P
>>>>>
>>>>> On 21 May 2019, at 22:57, Jacques Nadeau <ja...@dremio.com> wrote:
>>>>>
>>>>>
>>>>> That's my point, truly independent writers (two Spark jobs, or a Spark
>>>>>> job and Dremio job) means a distributed transaction. It would need yet
>>>>>> another external transaction coordinator on top of both Spark and Dremio,
>>>>>> Iceberg by itself
>>>>>> cannot solve this.
>>>>>>
>>>>>
>>>>> I'm not ready to accept this. Iceberg already supports a set of
>>>>> semantics around multiple writers committing simultaneously and how
>>>>> conflict resolution is done. The same can be done here.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> MVCC (which is what Iceberg tries to implement) requires a total
>>>>> ordering of snapshots. Also the snapshots need to be non-conflicting. I
>>>>> really don't see how any metadata data structures can solve this without an
>>>>> outside coordinator.
>>>>>
>>>>> Consider this:
>>>>>
>>>>> Snapshot 0: (K,A) = 1
>>>>> Job X: UPDATE K SET A=A+1
>>>>> Job Y: UPDATE K SET A=10
>>>>>
>>>>> What should the final value of A be and who decides ?
>>>>>
>>>>>
>>>>>
>>>>>> By single writer, I don't mean single process, I mean multiple
>>>>>> coordinated processes like Spark executors coordinated by Spark driver. The
>>>>>> coordinator ensures that the data is pre-partitioned on
>>>>>> each executor, and the coordinator commits the snapshot.
>>>>>>
>>>>>> Note however that single writer job/multiple concurrent reader jobs
>>>>>> is perfectly feasible, i.e. it shouldn't be a problem to write from a Spark
>>>>>> job and read from multiple Dremio queries concurrently (for example)
>>>>>>
>>>>>
>>>>> :D This is still "single process" from my perspective. That process
>>>>> may be coordinating other processes to do distributed work but ultimately
>>>>> it is a single process.
>>>>>
>>>>>
>>>>> Fair enough
>>>>>
>>>>>
>>>>>
>>>>>> I'm not sure what you mean exactly. If we can't enforce uniqueness we
>>>>>> shouldn't assume it.
>>>>>>
>>>>>
>>>>> I disagree. We can specify that as a requirement and state that you'll
>>>>> get unintended consequences if you provide your own keys and don't maintain
>>>>> this.
>>>>>
>>>>>
>>>>> There's no need for unintended consequences, we can specify consistent
>>>>> behaviour (and I believe the document says what that is)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> We do expect that most of the time the natural key is unique, but the
>>>>>> eager and lazy with natural key designs can handle duplicates
>>>>>> consistently. Basically it's not a problem to have duplicate natural
>>>>>> keys, everything works fine.
>>>>>>
>>>>>
>>>>> That heavily depends on how things are implemented. For example, we
>>>>> may write a bunch of code that generates internal data structures based on
>>>>> this expectation. If we have to support duplicate matches, all of sudden we
>>>>> can no longer size various data structures to improve performance and may
>>>>> be unable to preallocate memory associated with a guaranteed completion.
>>>>>
>>>>>
>>>>> Again we need to operate on the assumption that this is a large scale
>>>>> distributed compute/remote storage scenario. Key matching is done with
>>>>> shuffles with data movement across the network, such optimizations would
>>>>> really have little impact on overall performance. Not to mention that most
>>>>> query engines would already optimize the shuffle already as much as it can
>>>>> be optimized.
>>>>>
>>>>> It is true that if actual duplicate keys would make the key matching
>>>>> join (anti-join) somewhat more expensive, however it can be done in such a
>>>>> way that if the keys are in practice unique the join is as efficient as it
>>>>> can be.
>>>>>
>>>>>
>>>>>
>>>>> Let me try and clarify each point:
>>>>>>
>>>>>> - lookup for query or update on a non-(partition/bucket/sort) key
>>>>>> predicate implies scanning large amounts of data - because these are the
>>>>>> only data structures that can narrow down the lookup, right ? One could
>>>>>> argue that the min/max index (file skipping) can be applied to any column,
>>>>>> but in reality if that column is not sorted the min/max intervals can have
>>>>>> huge overlaps so it may be next to useless.
>>>>>> - remote storage - this is a critical architecture decision -
>>>>>> implementations on local storage imply a vastly different design for the
>>>>>> entire system, storage and compute.
>>>>>> - deleting single records per snapshot is unfeasible in eager but
>>>>>> also particularly in the lazy design: each deletion creates a very small
>>>>>> snapshot. Deleting 1 million records one at a time would create 1 million
>>>>>> small files, and 1 million RPC calls.
>>>>>>
>>>>>
>>>>> Why is this unfeasible? If I have a dataset of 100mm files including
>>>>> 1mm small files, is that a major problem? It seems like your usecase isn't
>>>>> one where you want to support single record deletes but it is definitely
>>>>> something important to many people.
>>>>>
>>>>>
>>>>> 100 mm total files or 1 mm files per dataset is definitely a problem
>>>>> on HDFS, and I believe on S3 too. Single key delete would work just fine,
>>>>> but it's simply not optimal to do that on remote storage. This is a very
>>>>> well known problem with HDFS, and one of the very reasons to have something
>>>>> like Iceberg in the first place.
>>>>>
>>>>> Basically the users would be able to do single key mutation, but it's
>>>>> not the use case we should be optimizing for, but it's really not advisable.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Eager is conceptually just lazy + compaction done, well, eagerly. The
>>>>>> logic for both is exactly the same, the trade-off is just that with eager
>>>>>> you implicitly compact every time so that you don't do any work on read,
>>>>>> while with lazy
>>>>>> you want to amortize the cost of compaction over multiple snapshots.
>>>>>>
>>>>>> Basically there should be no difference between the two conceptually,
>>>>>> or with regard to keys, etc. The only difference is some mechanics in
>>>>>> implementation.
>>>>>>
>>>>>
>>>>> I think you have deconstruct the problem too much to say these are the
>>>>> same (or at least that is what I'm starting to think given this thread). It
>>>>> seems like real world implementation decisions (per our discussion here)
>>>>> are in conflict. For example, you just argued against having a 1mm
>>>>> arbitrary mutations but I think that is because you aren't thinking about
>>>>> things over time with a delta implementation. Having 10,000 mutations a day
>>>>> where we do delta compaction once a week
>>>>>
>>>>> and local file mappings (key to offset sparse bitmaps) seems like it
>>>>> could result in very good performance in a case where we're mutating small
>>>>> amounts of data. In this scenario, you may not do major compaction ever
>>>>> unless you get to a high enough percentage of records that have been
>>>>> deleted in the original dataset. That drives a very different set of
>>>>> implementation decisions from a situation where you're trying to restate an
>>>>> entire partition at once.
>>>>>
>>>>>
>>>>> We operate on 1 billion mutations per day at least. This is the
>>>>> problem Iceberg wants to solve, I believe it's stated upfront. 10000/day is
>>>>> not a big data problem. It can be done fairly trivially and it would be
>>>>> supported, but there's not much point in extra optimizing for this use case
>>>>> I believe.
>>>>>
>>>>>
>>>>
>>>
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>>
>>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>
>
>

-- 
Ryan Blue
Software Engineer
Netflix

Re: Updates/Deletes/Upserts in Iceberg

[Anton Okolnychyi <ao...@apple.com.INVALID>]

Thanks! Would it make sense to discuss the agenda in advance?

> On 22 May 2019, at 17:04, Ryan Blue <rb...@netflix.com.INVALID> wrote:
> 
> I sent out an invite and included everyone on this thread. If anyone else would like to join, please join the Zoom meeting. If you'd like to be added to the calendar invite, just let me know and I'll add you.
> 
> On Wed, May 22, 2019 at 8:57 AM Jacques Nadeau <jacques@dremio.com <ma...@dremio.com>> wrote:
> works for me.
> 
> To make things easier, we can use my zoom meeting if people like:
> 
> Join Zoom Meeting
> https://zoom.us/j/4157302092 <https://zoom.us/j/4157302092>
> 
> One tap mobile
> +16465588656,,4157302092# US (New York)
> +16699006833,,4157302092# US (San Jose)
> 
> Dial by your location
>         +1 646 558 8656 US (New York)
>         +1 669 900 6833 US (San Jose)
>         877 853 5257 US Toll-free
>         888 475 4499 US Toll-free
> Meeting ID: 415 730 2092
> Find your local number: https://zoom.us/u/aH9XYBfm <https://zoom.us/u/aH9XYBfm>
> 
> 
> 
> 
> 
> 
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
> 
> 
> On Wed, May 22, 2019 at 8:54 AM Ryan Blue <rb...@netflix.com.invalid> wrote:
> 9AM on Friday works best for me. How about then?
> 
> On Wed, May 22, 2019 at 5:05 AM Anton Okolnychyi <aokolnychyi@apple.com <ma...@apple.com>> wrote:
> What about this Friday? One hour slot from 9:00 to 10:00 am or 10:00 to 11:00 am PST? Some folks are based in London, so meeting later than this is hard. If Friday doesn’t work, we can consider Tuesday or Wednesday next week.
> 
>> On 22 May 2019, at 00:54, Jacques Nadeau <jacques@dremio.com <ma...@dremio.com>> wrote:
>> 
>> I agree with Anton that we should probably spend some time on hangouts further discussing things. Definitely differing expectations here and we seem to be talking a bit past each other.
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>> 
>> 
>> On Tue, May 21, 2019 at 3:44 PM Cristian Opris <copris@apple.com.invalid <ma...@apple.com.invalid>> wrote:
>> I love a good flame war :P
>> 
>>> On 21 May 2019, at 22:57, Jacques Nadeau <jacques@dremio.com <ma...@dremio.com>> wrote:
>>> 
>>> 
>>> That's my point, truly independent writers (two Spark jobs, or a Spark job and Dremio job) means a distributed transaction. It would need yet another external transaction coordinator on top of both Spark and Dremio, Iceberg by itself 
>>> cannot solve this.
>>>  
>>> I'm not ready to accept this. Iceberg already supports a set of semantics around multiple writers committing simultaneously and how conflict resolution is done. The same can be done here.
>> 
>> 
>> 
>> MVCC (which is what Iceberg tries to implement) requires a total ordering of snapshots. Also the snapshots need to be non-conflicting. I really don't see how any metadata data structures can solve this without an outside coordinator.
>> 
>> Consider this:
>> 
>> Snapshot 0: (K,A) = 1
>> Job X: UPDATE K SET A=A+1
>> Job Y: UPDATE K SET A=10
>> 
>> What should the final value of A be and who decides ?
>> 
>>>  
>>> By single writer, I don't mean single process, I mean multiple coordinated processes like Spark executors coordinated by Spark driver. The coordinator ensures that the data is pre-partitioned on 
>>> each executor, and the coordinator commits the snapshot. 
>>> 
>>> Note however that single writer job/multiple concurrent reader jobs is perfectly feasible, i.e. it shouldn't be a problem to write from a Spark job and read from multiple Dremio queries concurrently (for example)
>>> 
>>> :D This is still "single process" from my perspective. That process may be coordinating other processes to do distributed work but ultimately it is a single process. 
>> 
>> Fair enough
>> 
>>>  
>>> I'm not sure what you mean exactly. If we can't enforce uniqueness we shouldn't assume it.
>>>  
>>> I disagree. We can specify that as a requirement and state that you'll get unintended consequences if you provide your own keys and don't maintain this.
>> 
>> There's no need for unintended consequences, we can specify consistent behaviour (and I believe the document says what that is)
>> 
>> 
>>>  
>>> We do expect that most of the time the natural key is unique, but the eager and lazy with natural key designs can handle duplicates 
>>> consistently. Basically it's not a problem to have duplicate natural keys, everything works fine.
>>> 
>>> That heavily depends on how things are implemented. For example, we may write a bunch of code that generates internal data structures based on this expectation. If we have to support duplicate matches, all of sudden we can no longer size various data structures to improve performance and may be unable to preallocate memory associated with a guaranteed completion.
>> 
>> Again we need to operate on the assumption that this is a large scale distributed compute/remote storage scenario. Key matching is done with shuffles with data movement across the network, such optimizations would really have little impact on overall performance. Not to mention that most query engines would already optimize the shuffle already as much as it can be optimized.
>> 
>> It is true that if actual duplicate keys would make the key matching join (anti-join) somewhat more expensive, however it can be done in such a way that if the keys are in practice unique the join is as efficient as it can be.
>> 
>> 
>>> 
>>> Let me try and clarify each point:
>>> 
>>> - lookup for query or update on a non-(partition/bucket/sort) key predicate implies scanning large amounts of data - because these are the only data structures that can narrow down the lookup, right ? One could argue that the min/max index (file skipping) can be applied to any column, but in reality if that column is not sorted the min/max intervals can have huge overlaps so it may be next to useless.
>>> - remote storage - this is a critical architecture decision - implementations on local storage imply a vastly different design for the entire system, storage and compute. 
>>> - deleting single records per snapshot is unfeasible in eager but also particularly in the lazy design: each deletion creates a very small snapshot. Deleting 1 million records one at a time would create 1 million small files, and 1 million RPC calls.
>>> 
>>> Why is this unfeasible? If I have a dataset of 100mm files including 1mm small files, is that a major problem? It seems like your usecase isn't one where you want to support single record deletes but it is definitely something important to many people.
>> 
>> 100 mm total files or 1 mm files per dataset is definitely a problem on HDFS, and I believe on S3 too. Single key delete would work just fine, but it's simply not optimal to do that on remote storage. This is a very well known problem with HDFS, and one of the very reasons to have something like Iceberg in the first place.
>> 
>> Basically the users would be able to do single key mutation, but it's not the use case we should be optimizing for, but it's really not advisable.
>> 
>> 
>>>  
>>> Eager is conceptually just lazy + compaction done, well, eagerly. The logic for both is exactly the same, the trade-off is just that with eager you implicitly compact every time so that you don't do any work on read, while with lazy 
>>> you want to amortize the cost of compaction over multiple snapshots.
>>> 
>>> Basically there should be no difference between the two conceptually, or with regard to keys, etc. The only difference is some mechanics in implementation.
>>> 
>>> I think you have deconstruct the problem too much to say these are the same (or at least that is what I'm starting to think given this thread). It seems like real world implementation decisions (per our discussion here) are in conflict. For example, you just argued against having a 1mm arbitrary mutations but I think that is because you aren't thinking about things over time with a delta implementation. Having 10,000 mutations a day where we do delta compaction once a week
>>> and local file mappings (key to offset sparse bitmaps) seems like it could result in very good performance in a case where we're mutating small amounts of data. In this scenario, you may not do major compaction ever unless you get to a high enough percentage of records that have been deleted in the original dataset. That drives a very different set of implementation decisions from a situation where you're trying to restate an entire partition at once.
>> 
>> 
>> We operate on 1 billion mutations per day at least. This is the problem Iceberg wants to solve, I believe it's stated upfront. 10000/day is not a big data problem. It can be done fairly trivially and it would be supported, but there's not much point in extra optimizing for this use case I believe.
>> 
> 
> 
> 
> -- 
> Ryan Blue
> Software Engineer
> Netflix
> 
> 
> -- 
> Ryan Blue
> Software Engineer
> Netflix

Re: Updates/Deletes/Upserts in Iceberg

[Ryan Blue <rb...@netflix.com.INVALID>]

I sent out an invite and included everyone on this thread. If anyone else
would like to join, please join the Zoom meeting. If you'd like to be added
to the calendar invite, just let me know and I'll add you.

On Wed, May 22, 2019 at 8:57 AM Jacques Nadeau <ja...@dremio.com> wrote:

> works for me.
>
> To make things easier, we can use my zoom meeting if people like:
>
> Join Zoom Meeting
> https://zoom.us/j/4157302092
>
> One tap mobile
> +16465588656,,4157302092# US (New York)
> +16699006833,,4157302092# US (San Jose)
>
> Dial by your location
>         +1 646 558 8656 US (New York)
>         +1 669 900 6833 US (San Jose)
>         877 853 5257 US Toll-free
>         888 475 4499 US Toll-free
> Meeting ID: 415 730 2092
> Find your local number: https://zoom.us/u/aH9XYBfm
>
>
>
>
>
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
>
> On Wed, May 22, 2019 at 8:54 AM Ryan Blue <rb...@netflix.com.invalid>
> wrote:
>
>> 9AM on Friday works best for me. How about then?
>>
>> On Wed, May 22, 2019 at 5:05 AM Anton Okolnychyi <ao...@apple.com>
>> wrote:
>>
>>> What about this Friday? One hour slot from 9:00 to 10:00 am or 10:00 to
>>> 11:00 am PST? Some folks are based in London, so meeting later than this is
>>> hard. If Friday doesn’t work, we can consider Tuesday or Wednesday next
>>> week.
>>>
>>> On 22 May 2019, at 00:54, Jacques Nadeau <ja...@dremio.com> wrote:
>>>
>>> I agree with Anton that we should probably spend some time on hangouts
>>> further discussing things. Definitely differing expectations here and we
>>> seem to be talking a bit past each other.
>>> --
>>> Jacques Nadeau
>>> CTO and Co-Founder, Dremio
>>>
>>>
>>> On Tue, May 21, 2019 at 3:44 PM Cristian Opris <co...@apple.com.invalid>
>>> wrote:
>>>
>>>> I love a good flame war :P
>>>>
>>>> On 21 May 2019, at 22:57, Jacques Nadeau <ja...@dremio.com> wrote:
>>>>
>>>>
>>>> That's my point, truly independent writers (two Spark jobs, or a Spark
>>>>> job and Dremio job) means a distributed transaction. It would need yet
>>>>> another external transaction coordinator on top of both Spark and Dremio,
>>>>> Iceberg by itself
>>>>> cannot solve this.
>>>>>
>>>>
>>>> I'm not ready to accept this. Iceberg already supports a set of
>>>> semantics around multiple writers committing simultaneously and how
>>>> conflict resolution is done. The same can be done here.
>>>>
>>>>
>>>>
>>>>
>>>> MVCC (which is what Iceberg tries to implement) requires a total
>>>> ordering of snapshots. Also the snapshots need to be non-conflicting. I
>>>> really don't see how any metadata data structures can solve this without an
>>>> outside coordinator.
>>>>
>>>> Consider this:
>>>>
>>>> Snapshot 0: (K,A) = 1
>>>> Job X: UPDATE K SET A=A+1
>>>> Job Y: UPDATE K SET A=10
>>>>
>>>> What should the final value of A be and who decides ?
>>>>
>>>>
>>>>
>>>>> By single writer, I don't mean single process, I mean multiple
>>>>> coordinated processes like Spark executors coordinated by Spark driver. The
>>>>> coordinator ensures that the data is pre-partitioned on
>>>>> each executor, and the coordinator commits the snapshot.
>>>>>
>>>>> Note however that single writer job/multiple concurrent reader jobs is
>>>>> perfectly feasible, i.e. it shouldn't be a problem to write from a Spark
>>>>> job and read from multiple Dremio queries concurrently (for example)
>>>>>
>>>>
>>>> :D This is still "single process" from my perspective. That process may
>>>> be coordinating other processes to do distributed work but ultimately it is
>>>> a single process.
>>>>
>>>>
>>>> Fair enough
>>>>
>>>>
>>>>
>>>>> I'm not sure what you mean exactly. If we can't enforce uniqueness we
>>>>> shouldn't assume it.
>>>>>
>>>>
>>>> I disagree. We can specify that as a requirement and state that you'll
>>>> get unintended consequences if you provide your own keys and don't maintain
>>>> this.
>>>>
>>>>
>>>> There's no need for unintended consequences, we can specify consistent
>>>> behaviour (and I believe the document says what that is)
>>>>
>>>>
>>>>
>>>>
>>>>> We do expect that most of the time the natural key is unique, but the
>>>>> eager and lazy with natural key designs can handle duplicates
>>>>> consistently. Basically it's not a problem to have duplicate natural
>>>>> keys, everything works fine.
>>>>>
>>>>
>>>> That heavily depends on how things are implemented. For example, we may
>>>> write a bunch of code that generates internal data structures based on this
>>>> expectation. If we have to support duplicate matches, all of sudden we can
>>>> no longer size various data structures to improve performance and may be
>>>> unable to preallocate memory associated with a guaranteed completion.
>>>>
>>>>
>>>> Again we need to operate on the assumption that this is a large scale
>>>> distributed compute/remote storage scenario. Key matching is done with
>>>> shuffles with data movement across the network, such optimizations would
>>>> really have little impact on overall performance. Not to mention that most
>>>> query engines would already optimize the shuffle already as much as it can
>>>> be optimized.
>>>>
>>>> It is true that if actual duplicate keys would make the key matching
>>>> join (anti-join) somewhat more expensive, however it can be done in such a
>>>> way that if the keys are in practice unique the join is as efficient as it
>>>> can be.
>>>>
>>>>
>>>>
>>>> Let me try and clarify each point:
>>>>>
>>>>> - lookup for query or update on a non-(partition/bucket/sort) key
>>>>> predicate implies scanning large amounts of data - because these are the
>>>>> only data structures that can narrow down the lookup, right ? One could
>>>>> argue that the min/max index (file skipping) can be applied to any column,
>>>>> but in reality if that column is not sorted the min/max intervals can have
>>>>> huge overlaps so it may be next to useless.
>>>>> - remote storage - this is a critical architecture decision -
>>>>> implementations on local storage imply a vastly different design for the
>>>>> entire system, storage and compute.
>>>>> - deleting single records per snapshot is unfeasible in eager but also
>>>>> particularly in the lazy design: each deletion creates a very small
>>>>> snapshot. Deleting 1 million records one at a time would create 1 million
>>>>> small files, and 1 million RPC calls.
>>>>>
>>>>
>>>> Why is this unfeasible? If I have a dataset of 100mm files including
>>>> 1mm small files, is that a major problem? It seems like your usecase isn't
>>>> one where you want to support single record deletes but it is definitely
>>>> something important to many people.
>>>>
>>>>
>>>> 100 mm total files or 1 mm files per dataset is definitely a problem on
>>>> HDFS, and I believe on S3 too. Single key delete would work just fine, but
>>>> it's simply not optimal to do that on remote storage. This is a very well
>>>> known problem with HDFS, and one of the very reasons to have something like
>>>> Iceberg in the first place.
>>>>
>>>> Basically the users would be able to do single key mutation, but it's
>>>> not the use case we should be optimizing for, but it's really not advisable.
>>>>
>>>>
>>>>
>>>>
>>>>> Eager is conceptually just lazy + compaction done, well, eagerly. The
>>>>> logic for both is exactly the same, the trade-off is just that with eager
>>>>> you implicitly compact every time so that you don't do any work on read,
>>>>> while with lazy
>>>>> you want to amortize the cost of compaction over multiple snapshots.
>>>>>
>>>>> Basically there should be no difference between the two conceptually,
>>>>> or with regard to keys, etc. The only difference is some mechanics in
>>>>> implementation.
>>>>>
>>>>
>>>> I think you have deconstruct the problem too much to say these are the
>>>> same (or at least that is what I'm starting to think given this thread). It
>>>> seems like real world implementation decisions (per our discussion here)
>>>> are in conflict. For example, you just argued against having a 1mm
>>>> arbitrary mutations but I think that is because you aren't thinking about
>>>> things over time with a delta implementation. Having 10,000 mutations a day
>>>> where we do delta compaction once a week
>>>>
>>>> and local file mappings (key to offset sparse bitmaps) seems like it
>>>> could result in very good performance in a case where we're mutating small
>>>> amounts of data. In this scenario, you may not do major compaction ever
>>>> unless you get to a high enough percentage of records that have been
>>>> deleted in the original dataset. That drives a very different set of
>>>> implementation decisions from a situation where you're trying to restate an
>>>> entire partition at once.
>>>>
>>>>
>>>> We operate on 1 billion mutations per day at least. This is the problem
>>>> Iceberg wants to solve, I believe it's stated upfront. 10000/day is not a
>>>> big data problem. It can be done fairly trivially and it would be
>>>> supported, but there's not much point in extra optimizing for this use case
>>>> I believe.
>>>>
>>>>
>>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>

-- 
Ryan Blue
Software Engineer
Netflix

Re: Updates/Deletes/Upserts in Iceberg

[Jacques Nadeau <ja...@dremio.com>]

works for me.

To make things easier, we can use my zoom meeting if people like:

Join Zoom Meeting
https://zoom.us/j/4157302092

One tap mobile
+16465588656,,4157302092# US (New York)
+16699006833,,4157302092# US (San Jose)

Dial by your location
        +1 646 558 8656 US (New York)
        +1 669 900 6833 US (San Jose)
        877 853 5257 US Toll-free
        888 475 4499 US Toll-free
Meeting ID: 415 730 2092
Find your local number: https://zoom.us/u/aH9XYBfm






--
Jacques Nadeau
CTO and Co-Founder, Dremio


On Wed, May 22, 2019 at 8:54 AM Ryan Blue <rb...@netflix.com.invalid> wrote:

> 9AM on Friday works best for me. How about then?
>
> On Wed, May 22, 2019 at 5:05 AM Anton Okolnychyi <ao...@apple.com>
> wrote:
>
>> What about this Friday? One hour slot from 9:00 to 10:00 am or 10:00 to
>> 11:00 am PST? Some folks are based in London, so meeting later than this is
>> hard. If Friday doesn’t work, we can consider Tuesday or Wednesday next
>> week.
>>
>> On 22 May 2019, at 00:54, Jacques Nadeau <ja...@dremio.com> wrote:
>>
>> I agree with Anton that we should probably spend some time on hangouts
>> further discussing things. Definitely differing expectations here and we
>> seem to be talking a bit past each other.
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>>
>>
>> On Tue, May 21, 2019 at 3:44 PM Cristian Opris <co...@apple.com.invalid>
>> wrote:
>>
>>> I love a good flame war :P
>>>
>>> On 21 May 2019, at 22:57, Jacques Nadeau <ja...@dremio.com> wrote:
>>>
>>>
>>> That's my point, truly independent writers (two Spark jobs, or a Spark
>>>> job and Dremio job) means a distributed transaction. It would need yet
>>>> another external transaction coordinator on top of both Spark and Dremio,
>>>> Iceberg by itself
>>>> cannot solve this.
>>>>
>>>
>>> I'm not ready to accept this. Iceberg already supports a set of
>>> semantics around multiple writers committing simultaneously and how
>>> conflict resolution is done. The same can be done here.
>>>
>>>
>>>
>>>
>>> MVCC (which is what Iceberg tries to implement) requires a total
>>> ordering of snapshots. Also the snapshots need to be non-conflicting. I
>>> really don't see how any metadata data structures can solve this without an
>>> outside coordinator.
>>>
>>> Consider this:
>>>
>>> Snapshot 0: (K,A) = 1
>>> Job X: UPDATE K SET A=A+1
>>> Job Y: UPDATE K SET A=10
>>>
>>> What should the final value of A be and who decides ?
>>>
>>>
>>>
>>>> By single writer, I don't mean single process, I mean multiple
>>>> coordinated processes like Spark executors coordinated by Spark driver. The
>>>> coordinator ensures that the data is pre-partitioned on
>>>> each executor, and the coordinator commits the snapshot.
>>>>
>>>> Note however that single writer job/multiple concurrent reader jobs is
>>>> perfectly feasible, i.e. it shouldn't be a problem to write from a Spark
>>>> job and read from multiple Dremio queries concurrently (for example)
>>>>
>>>
>>> :D This is still "single process" from my perspective. That process may
>>> be coordinating other processes to do distributed work but ultimately it is
>>> a single process.
>>>
>>>
>>> Fair enough
>>>
>>>
>>>
>>>> I'm not sure what you mean exactly. If we can't enforce uniqueness we
>>>> shouldn't assume it.
>>>>
>>>
>>> I disagree. We can specify that as a requirement and state that you'll
>>> get unintended consequences if you provide your own keys and don't maintain
>>> this.
>>>
>>>
>>> There's no need for unintended consequences, we can specify consistent
>>> behaviour (and I believe the document says what that is)
>>>
>>>
>>>
>>>
>>>> We do expect that most of the time the natural key is unique, but the
>>>> eager and lazy with natural key designs can handle duplicates
>>>> consistently. Basically it's not a problem to have duplicate natural
>>>> keys, everything works fine.
>>>>
>>>
>>> That heavily depends on how things are implemented. For example, we may
>>> write a bunch of code that generates internal data structures based on this
>>> expectation. If we have to support duplicate matches, all of sudden we can
>>> no longer size various data structures to improve performance and may be
>>> unable to preallocate memory associated with a guaranteed completion.
>>>
>>>
>>> Again we need to operate on the assumption that this is a large scale
>>> distributed compute/remote storage scenario. Key matching is done with
>>> shuffles with data movement across the network, such optimizations would
>>> really have little impact on overall performance. Not to mention that most
>>> query engines would already optimize the shuffle already as much as it can
>>> be optimized.
>>>
>>> It is true that if actual duplicate keys would make the key matching
>>> join (anti-join) somewhat more expensive, however it can be done in such a
>>> way that if the keys are in practice unique the join is as efficient as it
>>> can be.
>>>
>>>
>>>
>>> Let me try and clarify each point:
>>>>
>>>> - lookup for query or update on a non-(partition/bucket/sort) key
>>>> predicate implies scanning large amounts of data - because these are the
>>>> only data structures that can narrow down the lookup, right ? One could
>>>> argue that the min/max index (file skipping) can be applied to any column,
>>>> but in reality if that column is not sorted the min/max intervals can have
>>>> huge overlaps so it may be next to useless.
>>>> - remote storage - this is a critical architecture decision -
>>>> implementations on local storage imply a vastly different design for the
>>>> entire system, storage and compute.
>>>> - deleting single records per snapshot is unfeasible in eager but also
>>>> particularly in the lazy design: each deletion creates a very small
>>>> snapshot. Deleting 1 million records one at a time would create 1 million
>>>> small files, and 1 million RPC calls.
>>>>
>>>
>>> Why is this unfeasible? If I have a dataset of 100mm files including 1mm
>>> small files, is that a major problem? It seems like your usecase isn't one
>>> where you want to support single record deletes but it is definitely
>>> something important to many people.
>>>
>>>
>>> 100 mm total files or 1 mm files per dataset is definitely a problem on
>>> HDFS, and I believe on S3 too. Single key delete would work just fine, but
>>> it's simply not optimal to do that on remote storage. This is a very well
>>> known problem with HDFS, and one of the very reasons to have something like
>>> Iceberg in the first place.
>>>
>>> Basically the users would be able to do single key mutation, but it's
>>> not the use case we should be optimizing for, but it's really not advisable.
>>>
>>>
>>>
>>>
>>>> Eager is conceptually just lazy + compaction done, well, eagerly. The
>>>> logic for both is exactly the same, the trade-off is just that with eager
>>>> you implicitly compact every time so that you don't do any work on read,
>>>> while with lazy
>>>> you want to amortize the cost of compaction over multiple snapshots.
>>>>
>>>> Basically there should be no difference between the two conceptually,
>>>> or with regard to keys, etc. The only difference is some mechanics in
>>>> implementation.
>>>>
>>>
>>> I think you have deconstruct the problem too much to say these are the
>>> same (or at least that is what I'm starting to think given this thread). It
>>> seems like real world implementation decisions (per our discussion here)
>>> are in conflict. For example, you just argued against having a 1mm
>>> arbitrary mutations but I think that is because you aren't thinking about
>>> things over time with a delta implementation. Having 10,000 mutations a day
>>> where we do delta compaction once a week
>>>
>>> and local file mappings (key to offset sparse bitmaps) seems like it
>>> could result in very good performance in a case where we're mutating small
>>> amounts of data. In this scenario, you may not do major compaction ever
>>> unless you get to a high enough percentage of records that have been
>>> deleted in the original dataset. That drives a very different set of
>>> implementation decisions from a situation where you're trying to restate an
>>> entire partition at once.
>>>
>>>
>>> We operate on 1 billion mutations per day at least. This is the problem
>>> Iceberg wants to solve, I believe it's stated upfront. 10000/day is not a
>>> big data problem. It can be done fairly trivially and it would be
>>> supported, but there's not much point in extra optimizing for this use case
>>> I believe.
>>>
>>>
>>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: Updates/Deletes/Upserts in Iceberg

[Ryan Blue <rb...@netflix.com.INVALID>]

9AM on Friday works best for me. How about then?

On Wed, May 22, 2019 at 5:05 AM Anton Okolnychyi <ao...@apple.com>
wrote:

> What about this Friday? One hour slot from 9:00 to 10:00 am or 10:00 to
> 11:00 am PST? Some folks are based in London, so meeting later than this is
> hard. If Friday doesn’t work, we can consider Tuesday or Wednesday next
> week.
>
> On 22 May 2019, at 00:54, Jacques Nadeau <ja...@dremio.com> wrote:
>
> I agree with Anton that we should probably spend some time on hangouts
> further discussing things. Definitely differing expectations here and we
> seem to be talking a bit past each other.
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
>
> On Tue, May 21, 2019 at 3:44 PM Cristian Opris <co...@apple.com.invalid>
> wrote:
>
>> I love a good flame war :P
>>
>> On 21 May 2019, at 22:57, Jacques Nadeau <ja...@dremio.com> wrote:
>>
>>
>> That's my point, truly independent writers (two Spark jobs, or a Spark
>>> job and Dremio job) means a distributed transaction. It would need yet
>>> another external transaction coordinator on top of both Spark and Dremio,
>>> Iceberg by itself
>>> cannot solve this.
>>>
>>
>> I'm not ready to accept this. Iceberg already supports a set of semantics
>> around multiple writers committing simultaneously and how conflict
>> resolution is done. The same can be done here.
>>
>>
>>
>>
>> MVCC (which is what Iceberg tries to implement) requires a total ordering
>> of snapshots. Also the snapshots need to be non-conflicting. I really don't
>> see how any metadata data structures can solve this without an outside
>> coordinator.
>>
>> Consider this:
>>
>> Snapshot 0: (K,A) = 1
>> Job X: UPDATE K SET A=A+1
>> Job Y: UPDATE K SET A=10
>>
>> What should the final value of A be and who decides ?
>>
>>
>>
>>> By single writer, I don't mean single process, I mean multiple
>>> coordinated processes like Spark executors coordinated by Spark driver. The
>>> coordinator ensures that the data is pre-partitioned on
>>> each executor, and the coordinator commits the snapshot.
>>>
>>> Note however that single writer job/multiple concurrent reader jobs is
>>> perfectly feasible, i.e. it shouldn't be a problem to write from a Spark
>>> job and read from multiple Dremio queries concurrently (for example)
>>>
>>
>> :D This is still "single process" from my perspective. That process may
>> be coordinating other processes to do distributed work but ultimately it is
>> a single process.
>>
>>
>> Fair enough
>>
>>
>>
>>> I'm not sure what you mean exactly. If we can't enforce uniqueness we
>>> shouldn't assume it.
>>>
>>
>> I disagree. We can specify that as a requirement and state that you'll
>> get unintended consequences if you provide your own keys and don't maintain
>> this.
>>
>>
>> There's no need for unintended consequences, we can specify consistent
>> behaviour (and I believe the document says what that is)
>>
>>
>>
>>
>>> We do expect that most of the time the natural key is unique, but the
>>> eager and lazy with natural key designs can handle duplicates
>>> consistently. Basically it's not a problem to have duplicate natural
>>> keys, everything works fine.
>>>
>>
>> That heavily depends on how things are implemented. For example, we may
>> write a bunch of code that generates internal data structures based on this
>> expectation. If we have to support duplicate matches, all of sudden we can
>> no longer size various data structures to improve performance and may be
>> unable to preallocate memory associated with a guaranteed completion.
>>
>>
>> Again we need to operate on the assumption that this is a large scale
>> distributed compute/remote storage scenario. Key matching is done with
>> shuffles with data movement across the network, such optimizations would
>> really have little impact on overall performance. Not to mention that most
>> query engines would already optimize the shuffle already as much as it can
>> be optimized.
>>
>> It is true that if actual duplicate keys would make the key matching join
>> (anti-join) somewhat more expensive, however it can be done in such a way
>> that if the keys are in practice unique the join is as efficient as it can
>> be.
>>
>>
>>
>> Let me try and clarify each point:
>>>
>>> - lookup for query or update on a non-(partition/bucket/sort) key
>>> predicate implies scanning large amounts of data - because these are the
>>> only data structures that can narrow down the lookup, right ? One could
>>> argue that the min/max index (file skipping) can be applied to any column,
>>> but in reality if that column is not sorted the min/max intervals can have
>>> huge overlaps so it may be next to useless.
>>> - remote storage - this is a critical architecture decision -
>>> implementations on local storage imply a vastly different design for the
>>> entire system, storage and compute.
>>> - deleting single records per snapshot is unfeasible in eager but also
>>> particularly in the lazy design: each deletion creates a very small
>>> snapshot. Deleting 1 million records one at a time would create 1 million
>>> small files, and 1 million RPC calls.
>>>
>>
>> Why is this unfeasible? If I have a dataset of 100mm files including 1mm
>> small files, is that a major problem? It seems like your usecase isn't one
>> where you want to support single record deletes but it is definitely
>> something important to many people.
>>
>>
>> 100 mm total files or 1 mm files per dataset is definitely a problem on
>> HDFS, and I believe on S3 too. Single key delete would work just fine, but
>> it's simply not optimal to do that on remote storage. This is a very well
>> known problem with HDFS, and one of the very reasons to have something like
>> Iceberg in the first place.
>>
>> Basically the users would be able to do single key mutation, but it's not
>> the use case we should be optimizing for, but it's really not advisable.
>>
>>
>>
>>
>>> Eager is conceptually just lazy + compaction done, well, eagerly. The
>>> logic for both is exactly the same, the trade-off is just that with eager
>>> you implicitly compact every time so that you don't do any work on read,
>>> while with lazy
>>> you want to amortize the cost of compaction over multiple snapshots.
>>>
>>> Basically there should be no difference between the two conceptually, or
>>> with regard to keys, etc. The only difference is some mechanics in
>>> implementation.
>>>
>>
>> I think you have deconstruct the problem too much to say these are the
>> same (or at least that is what I'm starting to think given this thread). It
>> seems like real world implementation decisions (per our discussion here)
>> are in conflict. For example, you just argued against having a 1mm
>> arbitrary mutations but I think that is because you aren't thinking about
>> things over time with a delta implementation. Having 10,000 mutations a day
>> where we do delta compaction once a week
>>
>> and local file mappings (key to offset sparse bitmaps) seems like it
>> could result in very good performance in a case where we're mutating small
>> amounts of data. In this scenario, you may not do major compaction ever
>> unless you get to a high enough percentage of records that have been
>> deleted in the original dataset. That drives a very different set of
>> implementation decisions from a situation where you're trying to restate an
>> entire partition at once.
>>
>>
>> We operate on 1 billion mutations per day at least. This is the problem
>> Iceberg wants to solve, I believe it's stated upfront. 10000/day is not a
>> big data problem. It can be done fairly trivially and it would be
>> supported, but there's not much point in extra optimizing for this use case
>> I believe.
>>
>>
>

-- 
Ryan Blue
Software Engineer
Netflix

Re: Updates/Deletes/Upserts in Iceberg

[Anton Okolnychyi <ao...@apple.com.INVALID>]

What about this Friday? One hour slot from 9:00 to 10:00 am or 10:00 to 11:00 am PST? Some folks are based in London, so meeting later than this is hard. If Friday doesn’t work, we can consider Tuesday or Wednesday next week.

> On 22 May 2019, at 00:54, Jacques Nadeau <ja...@dremio.com> wrote:
> 
> I agree with Anton that we should probably spend some time on hangouts further discussing things. Definitely differing expectations here and we seem to be talking a bit past each other.
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
> 
> 
> On Tue, May 21, 2019 at 3:44 PM Cristian Opris <co...@apple.com.invalid> wrote:
> I love a good flame war :P
> 
>> On 21 May 2019, at 22:57, Jacques Nadeau <jacques@dremio.com <ma...@dremio.com>> wrote:
>> 
>> 
>> That's my point, truly independent writers (two Spark jobs, or a Spark job and Dremio job) means a distributed transaction. It would need yet another external transaction coordinator on top of both Spark and Dremio, Iceberg by itself 
>> cannot solve this.
>>  
>> I'm not ready to accept this. Iceberg already supports a set of semantics around multiple writers committing simultaneously and how conflict resolution is done. The same can be done here.
> 
> 
> 
> MVCC (which is what Iceberg tries to implement) requires a total ordering of snapshots. Also the snapshots need to be non-conflicting. I really don't see how any metadata data structures can solve this without an outside coordinator.
> 
> Consider this:
> 
> Snapshot 0: (K,A) = 1
> Job X: UPDATE K SET A=A+1
> Job Y: UPDATE K SET A=10
> 
> What should the final value of A be and who decides ?
> 
>>  
>> By single writer, I don't mean single process, I mean multiple coordinated processes like Spark executors coordinated by Spark driver. The coordinator ensures that the data is pre-partitioned on 
>> each executor, and the coordinator commits the snapshot. 
>> 
>> Note however that single writer job/multiple concurrent reader jobs is perfectly feasible, i.e. it shouldn't be a problem to write from a Spark job and read from multiple Dremio queries concurrently (for example)
>> 
>> :D This is still "single process" from my perspective. That process may be coordinating other processes to do distributed work but ultimately it is a single process. 
> 
> Fair enough
> 
>>  
>> I'm not sure what you mean exactly. If we can't enforce uniqueness we shouldn't assume it.
>>  
>> I disagree. We can specify that as a requirement and state that you'll get unintended consequences if you provide your own keys and don't maintain this.
> 
> There's no need for unintended consequences, we can specify consistent behaviour (and I believe the document says what that is)
> 
> 
>>  
>> We do expect that most of the time the natural key is unique, but the eager and lazy with natural key designs can handle duplicates 
>> consistently. Basically it's not a problem to have duplicate natural keys, everything works fine.
>> 
>> That heavily depends on how things are implemented. For example, we may write a bunch of code that generates internal data structures based on this expectation. If we have to support duplicate matches, all of sudden we can no longer size various data structures to improve performance and may be unable to preallocate memory associated with a guaranteed completion.
> 
> Again we need to operate on the assumption that this is a large scale distributed compute/remote storage scenario. Key matching is done with shuffles with data movement across the network, such optimizations would really have little impact on overall performance. Not to mention that most query engines would already optimize the shuffle already as much as it can be optimized.
> 
> It is true that if actual duplicate keys would make the key matching join (anti-join) somewhat more expensive, however it can be done in such a way that if the keys are in practice unique the join is as efficient as it can be.
> 
> 
>> 
>> Let me try and clarify each point:
>> 
>> - lookup for query or update on a non-(partition/bucket/sort) key predicate implies scanning large amounts of data - because these are the only data structures that can narrow down the lookup, right ? One could argue that the min/max index (file skipping) can be applied to any column, but in reality if that column is not sorted the min/max intervals can have huge overlaps so it may be next to useless.
>> - remote storage - this is a critical architecture decision - implementations on local storage imply a vastly different design for the entire system, storage and compute. 
>> - deleting single records per snapshot is unfeasible in eager but also particularly in the lazy design: each deletion creates a very small snapshot. Deleting 1 million records one at a time would create 1 million small files, and 1 million RPC calls.
>> 
>> Why is this unfeasible? If I have a dataset of 100mm files including 1mm small files, is that a major problem? It seems like your usecase isn't one where you want to support single record deletes but it is definitely something important to many people.
> 
> 100 mm total files or 1 mm files per dataset is definitely a problem on HDFS, and I believe on S3 too. Single key delete would work just fine, but it's simply not optimal to do that on remote storage. This is a very well known problem with HDFS, and one of the very reasons to have something like Iceberg in the first place.
> 
> Basically the users would be able to do single key mutation, but it's not the use case we should be optimizing for, but it's really not advisable.
> 
> 
>>  
>> Eager is conceptually just lazy + compaction done, well, eagerly. The logic for both is exactly the same, the trade-off is just that with eager you implicitly compact every time so that you don't do any work on read, while with lazy 
>> you want to amortize the cost of compaction over multiple snapshots.
>> 
>> Basically there should be no difference between the two conceptually, or with regard to keys, etc. The only difference is some mechanics in implementation.
>> 
>> I think you have deconstruct the problem too much to say these are the same (or at least that is what I'm starting to think given this thread). It seems like real world implementation decisions (per our discussion here) are in conflict. For example, you just argued against having a 1mm arbitrary mutations but I think that is because you aren't thinking about things over time with a delta implementation. Having 10,000 mutations a day where we do delta compaction once a week
>> and local file mappings (key to offset sparse bitmaps) seems like it could result in very good performance in a case where we're mutating small amounts of data. In this scenario, you may not do major compaction ever unless you get to a high enough percentage of records that have been deleted in the original dataset. That drives a very different set of implementation decisions from a situation where you're trying to restate an entire partition at once.
> 
> 
> We operate on 1 billion mutations per day at least. This is the problem Iceberg wants to solve, I believe it's stated upfront. 10000/day is not a big data problem. It can be done fairly trivially and it would be supported, but there's not much point in extra optimizing for this use case I believe.
> 

Re: Updates/Deletes/Upserts in Iceberg

[Jacques Nadeau <ja...@dremio.com>]

I agree with Anton that we should probably spend some time on hangouts
further discussing things. Definitely differing expectations here and we
seem to be talking a bit past each other.
--
Jacques Nadeau
CTO and Co-Founder, Dremio


On Tue, May 21, 2019 at 3:44 PM Cristian Opris <co...@apple.com.invalid>
wrote:

> I love a good flame war :P
>
> On 21 May 2019, at 22:57, Jacques Nadeau <ja...@dremio.com> wrote:
>
>
> That's my point, truly independent writers (two Spark jobs, or a Spark job
>> and Dremio job) means a distributed transaction. It would need yet another
>> external transaction coordinator on top of both Spark and Dremio, Iceberg
>> by itself
>> cannot solve this.
>>
>
> I'm not ready to accept this. Iceberg already supports a set of semantics
> around multiple writers committing simultaneously and how conflict
> resolution is done. The same can be done here.
>
>
>
>
> MVCC (which is what Iceberg tries to implement) requires a total ordering
> of snapshots. Also the snapshots need to be non-conflicting. I really don't
> see how any metadata data structures can solve this without an outside
> coordinator.
>
> Consider this:
>
> Snapshot 0: (K,A) = 1
> Job X: UPDATE K SET A=A+1
> Job Y: UPDATE K SET A=10
>
> What should the final value of A be and who decides ?
>
>
>
>> By single writer, I don't mean single process, I mean multiple
>> coordinated processes like Spark executors coordinated by Spark driver. The
>> coordinator ensures that the data is pre-partitioned on
>> each executor, and the coordinator commits the snapshot.
>>
>> Note however that single writer job/multiple concurrent reader jobs is
>> perfectly feasible, i.e. it shouldn't be a problem to write from a Spark
>> job and read from multiple Dremio queries concurrently (for example)
>>
>
> :D This is still "single process" from my perspective. That process may be
> coordinating other processes to do distributed work but ultimately it is a
> single process.
>
>
> Fair enough
>
>
>
>> I'm not sure what you mean exactly. If we can't enforce uniqueness we
>> shouldn't assume it.
>>
>
> I disagree. We can specify that as a requirement and state that you'll get
> unintended consequences if you provide your own keys and don't maintain
> this.
>
>
> There's no need for unintended consequences, we can specify consistent
> behaviour (and I believe the document says what that is)
>
>
>
>
>> We do expect that most of the time the natural key is unique, but the
>> eager and lazy with natural key designs can handle duplicates
>> consistently. Basically it's not a problem to have duplicate natural
>> keys, everything works fine.
>>
>
> That heavily depends on how things are implemented. For example, we may
> write a bunch of code that generates internal data structures based on this
> expectation. If we have to support duplicate matches, all of sudden we can
> no longer size various data structures to improve performance and may be
> unable to preallocate memory associated with a guaranteed completion.
>
>
> Again we need to operate on the assumption that this is a large scale
> distributed compute/remote storage scenario. Key matching is done with
> shuffles with data movement across the network, such optimizations would
> really have little impact on overall performance. Not to mention that most
> query engines would already optimize the shuffle already as much as it can
> be optimized.
>
> It is true that if actual duplicate keys would make the key matching join
> (anti-join) somewhat more expensive, however it can be done in such a way
> that if the keys are in practice unique the join is as efficient as it can
> be.
>
>
>
> Let me try and clarify each point:
>>
>> - lookup for query or update on a non-(partition/bucket/sort) key
>> predicate implies scanning large amounts of data - because these are the
>> only data structures that can narrow down the lookup, right ? One could
>> argue that the min/max index (file skipping) can be applied to any column,
>> but in reality if that column is not sorted the min/max intervals can have
>> huge overlaps so it may be next to useless.
>> - remote storage - this is a critical architecture decision -
>> implementations on local storage imply a vastly different design for the
>> entire system, storage and compute.
>> - deleting single records per snapshot is unfeasible in eager but also
>> particularly in the lazy design: each deletion creates a very small
>> snapshot. Deleting 1 million records one at a time would create 1 million
>> small files, and 1 million RPC calls.
>>
>
> Why is this unfeasible? If I have a dataset of 100mm files including 1mm
> small files, is that a major problem? It seems like your usecase isn't one
> where you want to support single record deletes but it is definitely
> something important to many people.
>
>
> 100 mm total files or 1 mm files per dataset is definitely a problem on
> HDFS, and I believe on S3 too. Single key delete would work just fine, but
> it's simply not optimal to do that on remote storage. This is a very well
> known problem with HDFS, and one of the very reasons to have something like
> Iceberg in the first place.
>
> Basically the users would be able to do single key mutation, but it's not
> the use case we should be optimizing for, but it's really not advisable.
>
>
>
>
>> Eager is conceptually just lazy + compaction done, well, eagerly. The
>> logic for both is exactly the same, the trade-off is just that with eager
>> you implicitly compact every time so that you don't do any work on read,
>> while with lazy
>> you want to amortize the cost of compaction over multiple snapshots.
>>
>> Basically there should be no difference between the two conceptually, or
>> with regard to keys, etc. The only difference is some mechanics in
>> implementation.
>>
>
> I think you have deconstruct the problem too much to say these are the
> same (or at least that is what I'm starting to think given this thread). It
> seems like real world implementation decisions (per our discussion here)
> are in conflict. For example, you just argued against having a 1mm
> arbitrary mutations but I think that is because you aren't thinking about
> things over time with a delta implementation. Having 10,000 mutations a day
> where we do delta compaction once a week
>
> and local file mappings (key to offset sparse bitmaps) seems like it could
> result in very good performance in a case where we're mutating small
> amounts of data. In this scenario, you may not do major compaction ever
> unless you get to a high enough percentage of records that have been
> deleted in the original dataset. That drives a very different set of
> implementation decisions from a situation where you're trying to restate an
> entire partition at once.
>
>
> We operate on 1 billion mutations per day at least. This is the problem
> Iceberg wants to solve, I believe it's stated upfront. 10000/day is not a
> big data problem. It can be done fairly trivially and it would be
> supported, but there's not much point in extra optimizing for this use case
> I believe.
>
>

Re: Updates/Deletes/Upserts in Iceberg

[Cristian Opris <co...@apple.com.INVALID>]

I love a good flame war :P

> On 21 May 2019, at 22:57, Jacques Nadeau <ja...@dremio.com> wrote:
> 
> 
> That's my point, truly independent writers (two Spark jobs, or a Spark job and Dremio job) means a distributed transaction. It would need yet another external transaction coordinator on top of both Spark and Dremio, Iceberg by itself 
> cannot solve this.
>  
> I'm not ready to accept this. Iceberg already supports a set of semantics around multiple writers committing simultaneously and how conflict resolution is done. The same can be done here.



MVCC (which is what Iceberg tries to implement) requires a total ordering of snapshots. Also the snapshots need to be non-conflicting. I really don't see how any metadata data structures can solve this without an outside coordinator.

Consider this:

Snapshot 0: (K,A) = 1
Job X: UPDATE K SET A=A+1
Job Y: UPDATE K SET A=10

What should the final value of A be and who decides ?

>  
> By single writer, I don't mean single process, I mean multiple coordinated processes like Spark executors coordinated by Spark driver. The coordinator ensures that the data is pre-partitioned on 
> each executor, and the coordinator commits the snapshot. 
> 
> Note however that single writer job/multiple concurrent reader jobs is perfectly feasible, i.e. it shouldn't be a problem to write from a Spark job and read from multiple Dremio queries concurrently (for example)
> 
> :D This is still "single process" from my perspective. That process may be coordinating other processes to do distributed work but ultimately it is a single process. 

Fair enough

>  
> I'm not sure what you mean exactly. If we can't enforce uniqueness we shouldn't assume it.
>  
> I disagree. We can specify that as a requirement and state that you'll get unintended consequences if you provide your own keys and don't maintain this.

There's no need for unintended consequences, we can specify consistent behaviour (and I believe the document says what that is)


>  
> We do expect that most of the time the natural key is unique, but the eager and lazy with natural key designs can handle duplicates 
> consistently. Basically it's not a problem to have duplicate natural keys, everything works fine.
> 
> That heavily depends on how things are implemented. For example, we may write a bunch of code that generates internal data structures based on this expectation. If we have to support duplicate matches, all of sudden we can no longer size various data structures to improve performance and may be unable to preallocate memory associated with a guaranteed completion.

Again we need to operate on the assumption that this is a large scale distributed compute/remote storage scenario. Key matching is done with shuffles with data movement across the network, such optimizations would really have little impact on overall performance. Not to mention that most query engines would already optimize the shuffle already as much as it can be optimized.

It is true that if actual duplicate keys would make the key matching join (anti-join) somewhat more expensive, however it can be done in such a way that if the keys are in practice unique the join is as efficient as it can be.


> 
> Let me try and clarify each point:
> 
> - lookup for query or update on a non-(partition/bucket/sort) key predicate implies scanning large amounts of data - because these are the only data structures that can narrow down the lookup, right ? One could argue that the min/max index (file skipping) can be applied to any column, but in reality if that column is not sorted the min/max intervals can have huge overlaps so it may be next to useless.
> - remote storage - this is a critical architecture decision - implementations on local storage imply a vastly different design for the entire system, storage and compute. 
> - deleting single records per snapshot is unfeasible in eager but also particularly in the lazy design: each deletion creates a very small snapshot. Deleting 1 million records one at a time would create 1 million small files, and 1 million RPC calls.
> 
> Why is this unfeasible? If I have a dataset of 100mm files including 1mm small files, is that a major problem? It seems like your usecase isn't one where you want to support single record deletes but it is definitely something important to many people.

100 mm total files or 1 mm files per dataset is definitely a problem on HDFS, and I believe on S3 too. Single key delete would work just fine, but it's simply not optimal to do that on remote storage. This is a very well known problem with HDFS, and one of the very reasons to have something like Iceberg in the first place.

Basically the users would be able to do single key mutation, but it's not the use case we should be optimizing for, but it's really not advisable.


>  
> Eager is conceptually just lazy + compaction done, well, eagerly. The logic for both is exactly the same, the trade-off is just that with eager you implicitly compact every time so that you don't do any work on read, while with lazy 
> you want to amortize the cost of compaction over multiple snapshots.
> 
> Basically there should be no difference between the two conceptually, or with regard to keys, etc. The only difference is some mechanics in implementation.
> 
> I think you have deconstruct the problem too much to say these are the same (or at least that is what I'm starting to think given this thread). It seems like real world implementation decisions (per our discussion here) are in conflict. For example, you just argued against having a 1mm arbitrary mutations but I think that is because you aren't thinking about things over time with a delta implementation. Having 10,000 mutations a day where we do delta compaction once a week
> and local file mappings (key to offset sparse bitmaps) seems like it could result in very good performance in a case where we're mutating small amounts of data. In this scenario, you may not do major compaction ever unless you get to a high enough percentage of records that have been deleted in the original dataset. That drives a very different set of implementation decisions from a situation where you're trying to restate an entire partition at once.


We operate on 1 billion mutations per day at least. This is the problem Iceberg wants to solve, I believe it's stated upfront. 10000/day is not a big data problem. It can be done fairly trivially and it would be supported, but there's not much point in extra optimizing for this use case I believe.

Re: Updates/Deletes/Upserts in Iceberg

[Jacques Nadeau <ja...@dremio.com>]

> That's my point, truly independent writers (two Spark jobs, or a Spark job
> and Dremio job) means a distributed transaction. It would need yet another
> external transaction coordinator on top of both Spark and Dremio, Iceberg
> by itself
> cannot solve this.
>

I'm not ready to accept this. Iceberg already supports a set of semantics
around multiple writers committing simultaneously and how conflict
resolution is done. The same can be done here.


> By single writer, I don't mean single process, I mean multiple coordinated
> processes like Spark executors coordinated by Spark driver. The coordinator
> ensures that the data is pre-partitioned on
> each executor, and the coordinator commits the snapshot.
>
> Note however that single writer job/multiple concurrent reader jobs is
> perfectly feasible, i.e. it shouldn't be a problem to write from a Spark
> job and read from multiple Dremio queries concurrently (for example)
>

:D This is still "single process" from my perspective. That process may be
coordinating other processes to do distributed work but ultimately it is a
single process.


> I'm not sure what you mean exactly. If we can't enforce uniqueness we
> shouldn't assume it.
>

I disagree. We can specify that as a requirement and state that you'll get
unintended consequences if you provide your own keys and don't maintain
this.


> We do expect that most of the time the natural key is unique, but the
> eager and lazy with natural key designs can handle duplicates
> consistently. Basically it's not a problem to have duplicate natural keys,
> everything works fine.
>

That heavily depends on how things are implemented. For example, we may
write a bunch of code that generates internal data structures based on this
expectation. If we have to support duplicate matches, all of sudden we can
no longer size various data structures to improve performance and may be
unable to preallocate memory associated with a guaranteed completion.

Let me try and clarify each point:
>
> - lookup for query or update on a non-(partition/bucket/sort) key
> predicate implies scanning large amounts of data - because these are the
> only data structures that can narrow down the lookup, right ? One could
> argue that the min/max index (file skipping) can be applied to any column,
> but in reality if that column is not sorted the min/max intervals can have
> huge overlaps so it may be next to useless.
> - remote storage - this is a critical architecture decision -
> implementations on local storage imply a vastly different design for the
> entire system, storage and compute.
> - deleting single records per snapshot is unfeasible in eager but also
> particularly in the lazy design: each deletion creates a very small
> snapshot. Deleting 1 million records one at a time would create 1 million
> small files, and 1 million RPC calls.
>

Why is this unfeasible? If I have a dataset of 100mm files including 1mm
small files, is that a major problem? It seems like your usecase isn't one
where you want to support single record deletes but it is definitely
something important to many people.


> Eager is conceptually just lazy + compaction done, well, eagerly. The
> logic for both is exactly the same, the trade-off is just that with eager
> you implicitly compact every time so that you don't do any work on read,
> while with lazy
> you want to amortize the cost of compaction over multiple snapshots.
>
> Basically there should be no difference between the two conceptually, or
> with regard to keys, etc. The only difference is some mechanics in
> implementation.
>

I think you have deconstruct the problem too much to say these are the same
(or at least that is what I'm starting to think given this thread). It
seems like real world implementation decisions (per our discussion here)
are in conflict. For example, you just argued against having a 1mm
arbitrary mutations but I think that is because you aren't thinking about
things over time with a delta implementation. Having 10,000 mutations a day
where we do delta compaction once a week and local file mappings (key to
offset sparse bitmaps) seems like it could result in very good performance
in a case where we're mutating small amounts of data. In this scenario, you
may not do major compaction ever unless you get to a high enough percentage
of records that have been deleted in the original dataset. That drives a
very different set of implementation decisions from a situation where
you're trying to restate an entire partition at once.

Re: Updates/Deletes/Upserts in Iceberg

[Cristian Opris <co...@apple.com.INVALID>]

Another point to add to my list of clarifications, which I hope will help understand what the document is proposing better:

The delete diff files can be simply list of keys to delete from previous snapshots. Natural keys or synthetic keys, it really doesn't matter. This is simple and elegant, in Parquet it can be a Parquet
file with just the key columns for example. 

In the discussions we had internally, we couldn't come up with a single compelling benefit that synthetic keys have over natural keys. The drawback of synthetic key is you have to generate them and in general
is one more thing to add to complexity. So there's no obvious reason to have them, but we thought let's put it out there, and see if there might be something we're missing ?

The document is not outright proposing a solution out of the three, but the implied conclusion when carefully considering the designs should appear to be:

- both eager and lazy are useful. 
- both should use natural keys



> On 21 May 2019, at 22:33, Cristian Opris <co...@apple.com.INVALID> wrote:
> 
> Hi Jacques,
> 
> 
> 
>> On 21 May 2019, at 22:11, Jacques Nadeau <jacques@dremio.com <ma...@dremio.com>> wrote:
>> 
>> It’s not at all clear why unique keys would be needed at all. 
>> 
>> If we turn your questions around, you answer yourself. If you have independent writers, you need unique keys.
>> 
>> Also truly independent writers (like a job writing while a job compacts), means effectively a distributed transaction, and I believe it’s clearly out of scope for Iceberg to solve that ?
>> 
>> Assuming a single process is writing seems severely limiting in design and scale. I'm also surprised that you would think this is outside of Iceberg's scope. A table format that can only be modified by a single process basically locks that format into a single tool for a particular deployment.
> 
> 
> That's my point, truly independent writers (two Spark jobs, or a Spark job and Dremio job) means a distributed transaction. It would need yet another external transaction coordinator on top of both Spark and Dremio, Iceberg by itself 
> cannot solve this.
> 
> By single writer, I don't mean single process, I mean multiple coordinated processes like Spark executors coordinated by Spark driver. The coordinator ensures that the data is pre-partitioned on 
> each executor, and the coordinator commits the snapshot. 
> 
> Note however that single writer job/multiple concurrent reader jobs is perfectly feasible, i.e. it shouldn't be a problem to write from a Spark job and read from multiple Dremio queries concurrently (for example)
> 
> 
> 
>> 
>> Uniqueness - enforcing uniqueness at scale is not feasible (proovably so).
>> 
>> Expecting uniqueness is different than enforcing it. If you're saying it is impossible to enforce, I understand that. If your we can't define a system where it is expected and there are ramifications if it is not maintained.
> 
> 
> I'm not sure what you mean exactly. If we can't enforce uniqueness we shouldn't assume it. We do expect that most of the time the natural key is unique, but the eager and lazy with natural key designs can handle duplicates 
> consistently. Basically it's not a problem to have duplicate natural keys, everything works fine.
> 
> 
> 
>> 
>> Also, at scale, it’s really only feasible to do query and update/upsert on
>> the partition/bucket/sort key, any other access is likely a full scan of terabytes of data, on remote storage.
>> 
>> I'm not sure why you would say unless you assume a particular implementation. Single record deletion is definitely an important use case. There is no need to do a full table scan to accomplish that unless you're assuming an eager approach to deletion.
> 
> Let me try and clarify each point:
> 
> - lookup for query or update on a non-(partition/bucket/sort) key predicate implies scanning large amounts of data - because these are the only data structures that can narrow down the lookup, right ? One could argue that the min/max index (file skipping) can be applied to any column, but in reality if that column is not sorted the min/max intervals can have huge overlaps so it may be next to useless.
> - remote storage - this is a critical architecture decision - implementations on local storage imply a vastly different design for the entire system, storage and compute. 
> - deleting single records per snapshot is unfeasible in eager but also particularly in the lazy design: each deletion creates a very small snapshot. Deleting 1 million records one at a time would create 1 million small files, and 1 million RPC calls.
> 
> 
> 
>> 
>> I do continue to wonder how much of this back and forth is the mixing of thinking around restatement (eager) versus delta (lazy) implementations. Maybe we should separate them out as two different conversations?
>> 
> 
> 
> Eager is conceptually just lazy + compaction done, well, eagerly. The logic for both is exactly the same, the trade-off is just that with eager you implicitly compact every time so that you don't do any work on read, while with lazy 
> you want to amortize the cost of compaction over multiple snapshots.
> 
> Basically there should be no difference between the two conceptually, or with regard to keys, etc. The only difference is some mechanics in implementation.
> 
> 
> 

Re: Updates/Deletes/Upserts in Iceberg

[Cristian Opris <co...@apple.com.INVALID>]

Hi Jacques,



> On 21 May 2019, at 22:11, Jacques Nadeau <ja...@dremio.com> wrote:
> 
> It’s not at all clear why unique keys would be needed at all. 
> 
> If we turn your questions around, you answer yourself. If you have independent writers, you need unique keys.
> 
> Also truly independent writers (like a job writing while a job compacts), means effectively a distributed transaction, and I believe it’s clearly out of scope for Iceberg to solve that ?
> 
> Assuming a single process is writing seems severely limiting in design and scale. I'm also surprised that you would think this is outside of Iceberg's scope. A table format that can only be modified by a single process basically locks that format into a single tool for a particular deployment.


That's my point, truly independent writers (two Spark jobs, or a Spark job and Dremio job) means a distributed transaction. It would need yet another external transaction coordinator on top of both Spark and Dremio, Iceberg by itself 
cannot solve this.

By single writer, I don't mean single process, I mean multiple coordinated processes like Spark executors coordinated by Spark driver. The coordinator ensures that the data is pre-partitioned on 
each executor, and the coordinator commits the snapshot. 

Note however that single writer job/multiple concurrent reader jobs is perfectly feasible, i.e. it shouldn't be a problem to write from a Spark job and read from multiple Dremio queries concurrently (for example)



> 
> Uniqueness - enforcing uniqueness at scale is not feasible (proovably so).
> 
> Expecting uniqueness is different than enforcing it. If you're saying it is impossible to enforce, I understand that. If your we can't define a system where it is expected and there are ramifications if it is not maintained.


I'm not sure what you mean exactly. If we can't enforce uniqueness we shouldn't assume it. We do expect that most of the time the natural key is unique, but the eager and lazy with natural key designs can handle duplicates 
consistently. Basically it's not a problem to have duplicate natural keys, everything works fine.



> 
> Also, at scale, it’s really only feasible to do query and update/upsert on
> the partition/bucket/sort key, any other access is likely a full scan of terabytes of data, on remote storage.
> 
> I'm not sure why you would say unless you assume a particular implementation. Single record deletion is definitely an important use case. There is no need to do a full table scan to accomplish that unless you're assuming an eager approach to deletion.

Let me try and clarify each point:

- lookup for query or update on a non-(partition/bucket/sort) key predicate implies scanning large amounts of data - because these are the only data structures that can narrow down the lookup, right ? One could argue that the min/max index (file skipping) can be applied to any column, but in reality if that column is not sorted the min/max intervals can have huge overlaps so it may be next to useless.
- remote storage - this is a critical architecture decision - implementations on local storage imply a vastly different design for the entire system, storage and compute. 
- deleting single records per snapshot is unfeasible in eager but also particularly in the lazy design: each deletion creates a very small snapshot. Deleting 1 million records one at a time would create 1 million small files, and 1 million RPC calls.



> 
> I do continue to wonder how much of this back and forth is the mixing of thinking around restatement (eager) versus delta (lazy) implementations. Maybe we should separate them out as two different conversations?
> 


Eager is conceptually just lazy + compaction done, well, eagerly. The logic for both is exactly the same, the trade-off is just that with eager you implicitly compact every time so that you don't do any work on read, while with lazy 
you want to amortize the cost of compaction over multiple snapshots.

Basically there should be no difference between the two conceptually, or with regard to keys, etc. The only difference is some mechanics in implementation.

Re: Updates/Deletes/Upserts in Iceberg

[Jacques Nadeau <ja...@dremio.com>]

>
> It’s not at all clear why unique keys would be needed at all.


If we turn your questions around, you answer yourself. If you have
independent writers, you need unique keys.

Also truly independent writers (like a job writing while a job compacts),
> means effectively a distributed transaction, and I believe it’s clearly out
> of scope for Iceberg to solve that ?
>

Assuming a single process is writing seems severely limiting in design and
scale. I'm also surprised that you would think this is outside of Iceberg's
scope. A table format that can only be modified by a single process
basically locks that format into a single tool for a particular deployment.

Uniqueness - enforcing uniqueness at scale is not feasible (proovably so).


Expecting uniqueness is different than enforcing it. If you're saying it is
impossible to enforce, I understand that. If your we can't define a system
where it is expected and there are ramifications if it is not maintained.

Also, at scale, it’s really only feasible to do query and update/upsert on
> the partition/bucket/sort key, any other access is likely a full scan of
> terabytes of data, on remote storage.


I'm not sure why you would say unless you assume a particular
implementation. Single record deletion is definitely an important use case.
There is no need to do a full table scan to accomplish that unless you're
assuming an eager approach to deletion.

I do continue to wonder how much of this back and forth is the mixing of
thinking around restatement (eager) versus delta (lazy) implementations.
Maybe we should separate them out as two different conversations?

Re: Updates/Deletes/Upserts in Iceberg

[Cristian Opris <co...@apple.com.INVALID>]

Yep, sounds good.

Thank you for putting this together, it certainly adds a lot of clarification to the discussion, and opens a clear path forward.

+1 from me (or +3 if we add Miguel and Anton) on going ahead with this approach

Sent from my iPhone

> On 7 Jun 2019, at 21:23, Erik Wright <er...@shopify.com> wrote:
> 
> Hi Cristian,
> 
> I'm glad to hear that this feels like alignment to you. I agree that it should closely correspond to the option that you described. My intent in writing this out was to clarify some of the details because they were a sticking point in the live discussion.
> 
>> One thing though: guaranteeing uniques doesn't seem possible with large datasets. In fact, no real big data system does that (We posted some examples in the comment thread on our proposal doc).
> 
> Clearly it is "possible" to guarantee uniques by comparing deltas to a base version during the commit process. But, as you alluded to, and as I mention in the document, the cost of doing so is highly dependent on the nature of the dataset and the updates to it. Furthermore, the likelihood of duplicates is highly variable, as is the cost of violations.
> 
> With all of those things in mind, I have recommended making this enforcement optional (if available at all). Instead I have chosen to describe the potential impact of errors in the dataset production. I identified two possible errors and illustrated how those errors might be observed by a consumer.
> 
>> The good news however, is we don't need uniqueness for all of this to work. It works just the same even if there are duplicates, however with a bit more care in implementation to handle
>> some corner cases consistently.
> 
> Agreed, that under certain conditions, deterministic behaviour can be achieved in the face of unintended duplicates in input. And this would be my recommended approach.
> 
> Thanks for taking the time to read my document.
> 
> Erik 
> 
>> On Fri, Jun 7, 2019 at 3:27 PM Cristian Opris <co...@apple.com> wrote:
>> Hey Erik,
>> 
>> I was able to parse your proposal quite quickly since it seems to be pretty much exactly the "lazy with natural key" in our proposed approach.
>> 
>> 
>> I must say it is however much concretely expressed.
>> 
>> This natural key approach is in fact our preferred option too, even though it wasn't explicitly said in the document. I guess we wanted to get more feedback just in case we're missing something. Turns out we weren't :)
>> 
>> In particular, I fully agree with this:
>> 
>> "The effective dataset of any other version is the equivalent of the previous version, anti-join the deletes in the current version (on the unique row identifier), union the insertions in the current version. Optimal performance will be achieved when consistent partition specifications and sorting are used from one version to the next (minimizing the number of deletes that need to be compared to each insertion)."
>> 
>> Yes, nothing more than sorted natural keys is needed to achieve a good implementation that covers all use cases.
>> 
>> One thing though: guaranteeing uniques doesn't seem possible with large datasets. In fact, no real big data system does that (We posted some examples in the comment thread on our proposal doc).
>> 
>> It's not clear from your doc how enforcing uniqueness would work ?
>> 
>> The good news however, is we don't need uniqueness for all of this to work. It works just the same even if there are duplicates, however with a bit more care in implementation to handle 
>> some corner cases consistently.
>> 
>>  Let me know if I got the gist of your proposal right ?
>> 
>> Thanks,
>> Cristian
>> 
>> 
>>> On 7 Jun 2019, at 19:47, Ryan Blue <rb...@netflix.com.INVALID> wrote:
>>> 
>>> Thanks, Erik! Great to see progress here. I'll set aside some time to look this over in detail.
>>> 
>>> On Fri, Jun 7, 2019 at 11:46 AM Erik Wright <er...@shopify.com> wrote:
>>>> I apologize for the delay, but I have finally put together a document describing an alternative approach to supporting updates in Iceberg while minimizing write amplification.
>>>> 
>>>> Proposal: Iceberg Merge-on-Read
>>>> 
>>>> Thank you, Anton and Miguel, for starting this conversation, and everyone else as well for the ongoing dialogue. I'm looking forward to continuing to discuss this and hopefully finding an approach that can meet our different needs and that we can work together on.
>>>> 
>>>> Cheers,
>>>> 
>>>> Erik
>>>> 
>>>>> On Wed, May 29, 2019 at 7:26 PM Jacques Nadeau <ja...@dremio.com> wrote:
>>>>> Yeah, I totally forgot to record our discussion. Will do so next time, sorry.
>>>>> --
>>>>> Jacques Nadeau
>>>>> CTO and Co-Founder, Dremio
>>>>> 
>>>>> 
>>>>>> On Wed, May 29, 2019 at 4:24 PM Ryan Blue <rb...@netflix.com> wrote:
>>>>>> It wasn't recorded, but I can summarize what we talked about. Sorry I haven't sent this out earlier.
>>>>>> 
>>>>>> We talked about the options and some of the background in Iceberg -- basically that it isn't possible to determine the order of commits before you commit so you can't rely on some monotonically increasing value from a snapshot to know which deltas to apply to a file. The result is that we can't apply diffs to data files using a rule like "files older than X" because we can't identify those files without the snapshot history.
>>>>>> 
>>>>>> That gives us basically 2 options for scoping delete diffs: either identify the files to apply a diff to when writing the diff, or log changes applied to a snapshot and keep the snapshot history around (which is how we know the order of snapshots). The first option is not good if you want to write without reading data to determine where the deleted records are. The second prevents cleaning up snapshot history.
>>>>>> 
>>>>>> We also talked about whether we should encode IDs in data files. Jacques pointed out that retrying a commit is easier if you don't need to re-read the original data to reconcile changes. For example, if a data file was compacted in a concurrent write, how do we reconcile a delete for it? We discussed other options, like rolling back the compaction for delete events. I think that's a promising option.
>>>>>> 
>>>>>> For action items, Jacques was going to think about whether we need to encode IDs in data files or if we could use positions to identify rows and write up a summary/proposal. Erik was going to take on planning how identifying rows without reading data would work and similarly write up a summary/proposal.
>>>>>> 
>>>>>> That's from memory, so if I've missed anything, I hope that other attendees will fill in the details!
>>>>>> 
>>>>>> rb
>>>>>> 
>>>>>>> On Wed, May 29, 2019 at 3:34 PM Venkatakrishnan Sowrirajan <vs...@asu.edu> wrote:
>>>>>>> Hi Ryan,
>>>>>>> 
>>>>>>> I couldn't attend the meeting. Just curious, if this is recorded by any chance.
>>>>>>> 
>>>>>>> Regards
>>>>>>> Venkata krishnan
>>>>>>> 
>>>>>>> 
>>>>>>>> On Fri, May 24, 2019 at 8:49 AM Ryan Blue <rb...@netflix.com.invalid> wrote:
>>>>>>>> Yes, I agree. I'll talk a little about a couple of the constraints of this as well.
>>>>>>>> 
>>>>>>>>> On Fri, May 24, 2019 at 5:52 AM Anton Okolnychyi <ao...@apple.com> wrote:
>>>>>>>>> The agenda looks good to me. I think it would also make sense to clarify the responsibilities of query engines and Iceberg. Not only in terms of uniqueness, but also in terms of applying diffs on read, for example. 
>>>>>>>>> 
>>>>>>>>>> On 23 May 2019, at 01:59, Ryan Blue <rb...@netflix.com.INVALID> wrote:
>>>>>>>>>> 
>>>>>>>>>> Here’s a rough agenda:
>>>>>>>>>> 
>>>>>>>>>> Use cases: everyone come with a use case that you’d like to have supported. We’ll go around and introduce ourselves and our use cases.
>>>>>>>>>> Main topic: How should Iceberg identify rows that are deleted?
>>>>>>>>>> Side topics from my initial email, if we have time: should we use insert diffs, should we support dense and sparse formats, etc.
>>>>>>>>>> The main topic I think we should discuss is: How should Iceberg identify rows that are deleted?
>>>>>>>>>> 
>>>>>>>>>> I’m phrasing it this way to avoid where I think we’re talking past one another because we are making assumptions. The important thing is that there are two main options:
>>>>>>>>>> 
>>>>>>>>>> Filename and position, vs
>>>>>>>>>> Specific values of (few) columns in the data
>>>>>>>>>> This phrasing also avoids discussing uniqueness constraints. Once we get down to behavior, I think we agree. For example, I think we all agree that uniqueness cannot be enforced in Iceberg.
>>>>>>>>>> 
>>>>>>>>>> If uniqueness can’t be enforced in Iceberg, the main choice comes down to how we identify rows that are deleted. If we use (filename, position) then we know that there is only one row. On the other hand, if we use data values to identify rows then a delete may identify more than one row because there are no uniqueness guarantees. I think we also agree that if there is more than one row identified, all of them should be deleted.
>>>>>>>>>> 
>>>>>>>>>> At that point, there are trade-offs between the approaches:
>>>>>>>>>> 
>>>>>>>>>> When identifying deleted rows by data values, situations like the one that Anton pointed out are possible.
>>>>>>>>>> Jacques also had a good point about concurrency. If at all possible, we want to be able to reconcile changes between concurrent commits without re-running an operation.
>>>>>>>>>> Sound like a reasonable amount to talk through?
>>>>>>>>>> 
>>>>>>>>>> rb
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> On Wed, May 22, 2019 at 1:17 PM Erik Wright <er...@shopify.com> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> On Wed, May 22, 2019 at 4:04 PM Cristian Opris <co...@apple.com.invalid> wrote:
>>>>>>>>>>>> Agreed with Erik here, we're certainly not looking to build the equivalent of a relational database, and for that matter not even that of a local disk storage analytics database (like Vertica). Those are very 
>>>>>>>>>>>> different designs with very different trade-offs and optimizations.
>>>>>>>>>>>> 
>>>>>>>>>>>> We're looking to automate and optimize specific types of file manipulation for large files on remote storage, while presenting that to the user under the common SQL API for bulk data manipulation (MERGE INTO)
>>>>>>>>>>> 
>>>>>>>>>>> What I would encourage is to decouple the storage model from the implementation of that API. If Iceberg has support for merge-on-read of upserts and deletes, in addition to its powerful support for partitioning, it will be easy for a higher-level application to implement those APIs given certain other constraints (that might not be appropriate to all applications).
>>>>>>>>>>> 
>>>>>>>>>>>> Myself and Miguel are out on Friday, but Anton should be able to handle the discussion on our side.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Cristian
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 22 May 2019, at 17:51, Erik Wright <er...@shopify.com.INVALID> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> We have two rows with the same natural key and we use that natural key in diff files:
>>>>>>>>>>>>>> nk | col1 | col2
>>>>>>>>>>>>>> 1 | 1 | 1
>>>>>>>>>>>>>> 1 | 2 | 2
>>>>>>>>>>>>>> Then we have a delete statement:
>>>>>>>>>>>>>> DELETE FROM t WHERE col1 = 1
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I think this example cuts to the point of the differences of understanding. Does Iceberg want to be approaching the utility of a relational database, against which I can execute complex update queries? This is not what I would have imagined.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I would have, instead, imagined that it was up to the client to identify, through whatever means, that they want to update or delete a row with a given ID. If there are multiple (distinct) rows with the same ID, _too bad_. Any user should _expect_ that they could potentially see any one or more of those rows at read time. And that an upsert/delete would affect any/all of them (I would argue for all).
>>>>>>>>>>>>> 
>>>>>>>>>>>>> In summary: Instead of trying to come up with a consistent, logical handling for complex queries that are best suited for a relational database, leave such handling up to the client and concentrate on problems that can be solved simply and more generally.
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Wed, May 22, 2019 at 12:11 PM Ryan Blue <rb...@netflix.com.invalid> wrote:
>>>>>>>>>>>>>> Yes, I think we should. I was going to propose one after catching up on the rest of this thread today.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Wed, May 22, 2019 at 9:08 AM Anton Okolnychyi <ao...@apple.com> wrote:
>>>>>>>>>>>>>>> Thanks! Would it make sense to discuss the agenda in advance?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On 22 May 2019, at 17:04, Ryan Blue <rb...@netflix.com.INVALID> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I sent out an invite and included everyone on this thread. If anyone else would like to join, please join the Zoom meeting. If you'd like to be added to the calendar invite, just let me know and I'll add you.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Wed, May 22, 2019 at 8:57 AM Jacques Nadeau <ja...@dremio.com> wrote:
>>>>>>>>>>>>>>>>> works for me.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> To make things easier, we can use my zoom meeting if people like:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Join Zoom Meeting
>>>>>>>>>>>>>>>>> https://zoom.us/j/4157302092
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> One tap mobile
>>>>>>>>>>>>>>>>> +16465588656,,4157302092# US (New York)
>>>>>>>>>>>>>>>>> +16699006833,,4157302092# US (San Jose)
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Dial by your location
>>>>>>>>>>>>>>>>>         +1 646 558 8656 US (New York)
>>>>>>>>>>>>>>>>>         +1 669 900 6833 US (San Jose)
>>>>>>>>>>>>>>>>>         877 853 5257 US Toll-free
>>>>>>>>>>>>>>>>>         888 475 4499 US Toll-free
>>>>>>>>>>>>>>>>> Meeting ID: 415 730 2092
>>>>>>>>>>>>>>>>> Find your local number: https://zoom.us/u/aH9XYBfm
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Jacques Nadeau
>>>>>>>>>>>>>>>>> CTO and Co-Founder, Dremio
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Wed, May 22, 2019 at 8:54 AM Ryan Blue <rb...@netflix.com.invalid> wrote:
>>>>>>>>>>>>>>>>>> 9AM on Friday works best for me. How about then?
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Wed, May 22, 2019 at 5:05 AM Anton Okolnychyi <ao...@apple.com> wrote:
>>>>>>>>>>>>>>>>>>> What about this Friday? One hour slot from 9:00 to 10:00 am or 10:00 to 11:00 am PST? Some folks are based in London, so meeting later than this is hard. If Friday doesn’t work, we can consider Tuesday or Wednesday next week.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On 22 May 2019, at 00:54, Jacques Nadeau <ja...@dremio.com> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I agree with Anton that we should probably spend some time on hangouts further discussing things. Definitely differing expectations here and we seem to be talking a bit past each other.
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> Jacques Nadeau
>>>>>>>>>>>>>>>>>>>> CTO and Co-Founder, Dremio
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Tue, May 21, 2019 at 3:44 PM Cristian Opris <co...@apple.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>> I love a good flame war :P
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On 21 May 2019, at 22:57, Jacques Nadeau <ja...@dremio.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> That's my point, truly independent writers (two Spark jobs, or a Spark job and Dremio job) means a distributed transaction. It would need yet another external transaction coordinator on top of both Spark and Dremio, Iceberg by itself 
>>>>>>>>>>>>>>>>>>>>>>> cannot solve this.
>>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>>> I'm not ready to accept this. Iceberg already supports a set of semantics around multiple writers committing simultaneously and how conflict resolution is done. The same can be done here.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> MVCC (which is what Iceberg tries to implement) requires a total ordering of snapshots. Also the snapshots need to be non-conflicting. I really don't see how any metadata data structures can solve this without an outside coordinator.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Consider this:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Snapshot 0: (K,A) = 1
>>>>>>>>>>>>>>>>>>>>> Job X: UPDATE K SET A=A+1
>>>>>>>>>>>>>>>>>>>>> Job Y: UPDATE K SET A=10
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> What should the final value of A be and who decides ?
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>>>> By single writer, I don't mean single process, I mean multiple coordinated processes like Spark executors coordinated by Spark driver. The coordinator ensures that the data is pre-partitioned on 
>>>>>>>>>>>>>>>>>>>>>>> each executor, and the coordinator commits the snapshot. 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Note however that single writer job/multiple concurrent reader jobs is perfectly feasible, i.e. it shouldn't be a problem to write from a Spark job and read from multiple Dremio queries concurrently (for example)
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> :D This is still "single process" from my perspective. That process may be coordinating other processes to do distributed work but ultimately it is a single process. 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Fair enough
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>>>> I'm not sure what you mean exactly. If we can't enforce uniqueness we shouldn't assume it.
>>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>>> I disagree. We can specify that as a requirement and state that you'll get unintended consequences if you provide your own keys and don't maintain this.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> There's no need for unintended consequences, we can specify consistent behaviour (and I believe the document says what that is)
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>>>> We do expect that most of the time the natural key is unique, but the eager and lazy with natural key designs can handle duplicates 
>>>>>>>>>>>>>>>>>>>>>>> consistently. Basically it's not a problem to have duplicate natural keys, everything works fine.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> That heavily depends on how things are implemented. For example, we may write a bunch of code that generates internal data structures based on this expectation. If we have to support duplicate matches, all of sudden we can no longer size various data structures to improve performance and may be unable to preallocate memory associated with a guaranteed completion.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Again we need to operate on the assumption that this is a large scale distributed compute/remote storage scenario. Key matching is done with shuffles with data movement across the network, such optimizations would really have little impact on overall performance. Not to mention that most query engines would already optimize the shuffle already as much as it can be optimized.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> It is true that if actual duplicate keys would make the key matching join (anti-join) somewhat more expensive, however it can be done in such a way that if the keys are in practice unique the join is as efficient as it can be.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Let me try and clarify each point:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> - lookup for query or update on a non-(partition/bucket/sort) key predicate implies scanning large amounts of data - because these are the only data structures that can narrow down the lookup, right ? One could argue that the min/max index (file skipping) can be applied to any column, but in reality if that column is not sorted the min/max intervals can have huge overlaps so it may be next to useless.
>>>>>>>>>>>>>>>>>>>>>>> - remote storage - this is a critical architecture decision - implementations on local storage imply a vastly different design for the entire system, storage and compute. 
>>>>>>>>>>>>>>>>>>>>>>> - deleting single records per snapshot is unfeasible in eager but also particularly in the lazy design: each deletion creates a very small snapshot. Deleting 1 million records one at a time would create 1 million small files, and 1 million RPC calls.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Why is this unfeasible? If I have a dataset of 100mm files including 1mm small files, is that a major problem? It seems like your usecase isn't one where you want to support single record deletes but it is definitely something important to many people.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 100 mm total files or 1 mm files per dataset is definitely a problem on HDFS, and I believe on S3 too. Single key delete would work just fine, but it's simply not optimal to do that on remote storage. This is a very well known problem with HDFS, and one of the very reasons to have something like Iceberg in the first place.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Basically the users would be able to do single key mutation, but it's not the use case we should be optimizing for, but it's really not advisable.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>>>> Eager is conceptually just lazy + compaction done, well, eagerly. The logic for both is exactly the same, the trade-off is just that with eager you implicitly compact every time so that you don't do any work on read, while with lazy 
>>>>>>>>>>>>>>>>>>>>>>> you want to amortize the cost of compaction over multiple snapshots.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Basically there should be no difference between the two conceptually, or with regard to keys, etc. The only difference is some mechanics in implementation.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> I think you have deconstruct the problem too much to say these are the same (or at least that is what I'm starting to think given this thread). It seems like real world implementation decisions (per our discussion here) are in conflict. For example, you just argued against having a 1mm arbitrary mutations but I think that is because you aren't thinking about things over time with a delta implementation. Having 10,000 mutations a day where we do delta compaction once a week
>>>>>>>>>>>>>>>>>>>>>> and local file mappings (key to offset sparse bitmaps) seems like it could result in very good performance in a case where we're mutating small amounts of data. In this scenario, you may not do major compaction ever unless you get to a high enough percentage of records that have been deleted in the original dataset. That drives a very different set of implementation decisions from a situation where you're trying to restate an entire partition at once.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> We operate on 1 billion mutations per day at least. This is the problem Iceberg wants to solve, I believe it's stated upfront. 10000/day is not a big data problem. It can be done fairly trivially and it would be supported, but there's not much point in extra optimizing for this use case I believe.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>> Netflix
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>> Netflix
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>> Netflix
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> -- 
>>>>>>>>>> Ryan Blue
>>>>>>>>>> Software Engineer
>>>>>>>>>> Netflix
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> -- 
>>>>>>>> Ryan Blue
>>>>>>>> Software Engineer
>>>>>>>> Netflix
>>>>>> 
>>>>>> 
>>>>>> -- 
>>>>>> Ryan Blue
>>>>>> Software Engineer
>>>>>> Netflix
>>> 
>>> 
>>> -- 
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>> 

Re: Updates/Deletes/Upserts in Iceberg

[Cristian Opris <co...@apple.com.INVALID>]

Hi Jacques,

It’s not at all clear why unique keys would be needed at all. 

Also truly independent writers (like a job writing while a job compacts), means effectively a distributed transaction, and I believe it’s clearly out of scope for Iceberg to solve that ?

> On 21 May 2019, at 21:31, Jacques Nadeau <ja...@dremio.com> wrote:
> 
> 
>> It would be useful to describe the types of concurrent operations that would be supported (i.e., failed snapshotting could easily be recovered, vs. the whole operation needing to be re-executed) vs. those that wouldn't. Solving for unlimited concurrency cases may create way more complexity than is necessary.
> 
> I'd like to restate my comment a little bit. We need unique keys to make things work. They can be synthetic or not but they should not have any retrievable iceberg related data in them.
> 
> The main thing I'm talking about is how you target a deletion across time. If you have a file A, and you want to delete record X in A, you define delete A.X. At the same time, another process may be compacting A into A'. In so doing, the position of A.X in A' is something other than X. At this point, the deletion needs to be rerun against A' so that we can ensure that the deletion is propagated forward. If the only thing you have is A.X, you need to have way from of getting to the same location in A'. You should be able to take the delta file that lists the delete of A.2 and apply it directly to A' without having to also consult A. If you didn't need to solve this number, then you could simply use A.X as opposed to the key of A.X in your delta files.
> 
>> Synthetic seems relative. If the synthetic key is client-supplied, in what way is it relevant to Iceberg whether it is synthetic vs. natural? By calling it synthetic within Iceberg there is a strong implication that it is the implementation that generates it (the filename/position key suggests that). If it's the client that supplies it, it _may_ be synthetic (from the point of view of the overall data model; i.e. a customer key in a database vs. a customer ID that shows up on a bill) but from Iceberg's case that doesn't matter. Only the unicity constraint does.
> 
> I agree with the main statement here: the only real requirement is keys need to be unique across all existing snapshots. There could be two generators: one that uses an iceberg internal behavior to generate keys and one that is user definable. While there could be a third which uses an existing field (or set of fields) to define the key I think we probably should avoid implementing this as it has a whole other sets of problems that are best left outside of Iceberg's area of concern.
> 

Re: Updates/Deletes/Upserts in Iceberg

[Jacques Nadeau <ja...@dremio.com>]

It would be useful to describe the types of concurrent operations that
> would be supported (i.e., failed snapshotting could easily be recovered,
> vs. the whole operation needing to be re-executed) vs. those that wouldn't.
> Solving for unlimited concurrency cases may create way more complexity than
> is necessary.
>

I'd like to restate my comment a little bit. We need unique keys to make
things work. They can be synthetic or not but they should not have any
retrievable iceberg related data in them.

The main thing I'm talking about is how you target a deletion across time.
If you have a file A, and you want to delete record X in A, you define
delete A.X. At the same time, another process may be compacting A into A'.
In so doing, the position of A.X in A' is something other than X. At this
point, the deletion needs to be rerun against A' so that we can ensure that
the deletion is propagated forward. If the only thing you have is A.X, you
need to have way from of getting to the same location in A'. You should be
able to take the delta file that lists the delete of A.2 and apply it
directly to A' without having to also consult A. If you didn't need to
solve this number, then you could simply use A.X as opposed to the key of
A.X in your delta files.

Synthetic seems relative. If the synthetic key is client-supplied, in what
> way is it relevant to Iceberg whether it is synthetic vs. natural? By
> calling it synthetic within Iceberg there is a strong implication that it
> is the implementation that generates it (the filename/position key suggests
> that). If it's the client that supplies it, it _may_ be synthetic (from the
> point of view of the overall data model; i.e. a customer key in a database
> vs. a customer ID that shows up on a bill) but from Iceberg's case that
> doesn't matter. Only the unicity constraint does.
>

I agree with the main statement here: the only real requirement is keys
need to be unique across all existing snapshots. There could be two
generators: one that uses an iceberg internal behavior to generate keys and
one that is user definable. While there could be a third which uses an
existing field (or set of fields) to define the key I think we probably
should avoid implementing this as it has a whole other sets of problems
that are best left outside of Iceberg's area of concern.

Re: Updates/Deletes/Upserts in Iceberg

[Cristian Opris <co...@apple.com.INVALID>]

Synthetic vs natural keys generated a lot of discussion internally when coming
up with the proposal :)

There is lots of detail and lots of subtelty but 
a few things that may help explain our thinking:

Scale - Iceberg is for remote storage at scale, not local disk. This means mainly operating on large
batches of data, not one row at a time. One row at a time would generate RPC calls for each row, and 
many small files in HDFS/object stores. 

Concurrency - while there are multiple writers, they are usually externally coordinated
(think Spark Driver for Spark executors). It’s very unlikely we would need to support
completely independent writers, at least we couldn’t come up with a real world use case
where that would be needed.

Uniqueness - enforcing uniqueness at scale is not feasible (proovably so). If we cannot enforce it,
we shouldn’t assume it. This is fine since we can have consistent behaviour even in the presence of
duplicates (the document explains this in some detail)

Natural keys - the key is called ‘natural’ because it’s a key that’s naturally unique in the dataset. For example 
combination (timestamp, eventId). Because it’s naturally unique, it’s the most selective index for the dataset, so it’s what
naturally people would partition/bucket/sort by. So almost by definition, the natural key is the
composite key made up of partition/bucket/sort key. It wouldn’t make logical sense for users to define a ‘natural key’ 
separate from that in their table definition. This key is usually naturally unique in the dataset, but even if it isn’t the behaviour 
is consistent (please convince yourself of this from the document)

Also, at scale, it’s really only feasible to do query and update/upsert on
the partition/bucket/sort key, any other access is likely a full scan of terabytes of data, on remote storage.
So we mostly only really care about query/upsert on the natural key predicate. Update/upsert
on non-natural key would be possible, and consitent, but it’s intrinsically very expensive (full scan), so
we shouldn’t worry too much about optimizing that.





> On 21 May 2019, at 19:26, Erik Wright <er...@shopify.com.invalid> wrote:
> 
> 
> 
>> On Tue, May 21, 2019 at 2:01 PM Jacques Nadeau <ja...@dremio.com> wrote:
>> I think we just need to have further discussion about keys. Ryan said:
>> 
>>> 3. Synthetic keys should be based on filename and position
>> 
>> But I'm not clear there is consensus around that. I'm also not sure whether he means lossless inclusion, simply derived-from or something else. My thinking before is you must have synthetic keys in many cases since solving concurrency becomes challenging otherwise.
> 
> It would be useful to describe the types of concurrent operations that would be supported (i.e., failed snapshotting could easily be recovered, vs. the whole operation needing to be re-executed) vs. those that wouldn't. Solving for unlimited concurrency cases may create way more complexity than is necessary.
> 
>> Erik said:
>>> C might not be partitioned or sorted by the natural key, either, which means that finding the synthetic key can be expensive.
>> 
>> But I'm not sure why this needs to be true. If we give control over the creation of synthetic key, wouldn't that resolve this issue?
> 
> Synthetic seems relative. If the synthetic key is client-supplied, in what way is it relevant to Iceberg whether it is synthetic vs. natural? By calling it synthetic within Iceberg there is a strong implication that it is the implementation that generates it (the filename/position key suggests that). If it's the client that supplies it, it _may_ be synthetic (from the point of view of the overall data model; i.e. a customer key in a database vs. a customer ID that shows up on a bill) but from Iceberg's case that doesn't matter. Only the unicity constraint does.
> 
> On the other hand, everything _but_ the thing about filename/position can probably be resolved. One solution could be to allow either user-specified _or_ filename/position-based keys in delta-files. If you use the latter, there are constraints on the types of operations you can do. If you use the former, there are constraints on the types of concurrency that are supported (I'm not sure about that, really, just going off of your comment above).
> 
>> 
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>> 
>> 
>>> On Tue, May 21, 2019 at 7:54 AM Erik Wright <er...@shopify.com.invalid> wrote:
>>>> On Thu, May 16, 2019 at 4:13 PM Ryan Blue <rb...@netflix.com> wrote:
>>>> Replies inline.
>>>> 
>>>>> On Thu, May 16, 2019 at 10:07 AM Erik Wright <er...@shopify.com> wrote:
>>>>> I would be happy to participate. Iceberg with merge-on-read capabilities is a technology choice that my team is actively considering. It appears that our scenario differs meaningfully from the one that Anton and Miguel are considering. It would be great to take the time to compare the two and see if there is a single implementation that can meet the needs of each scenario.
>>>> 
>>>> Can you be more specific about where the use cases differ meaningfully? I think that if we agree that operations on natural keys can be implemented using synthetic keys to encode deletes (#2), then everyone is aligned on the core parts of a design. We can figure out the implications of how synthetic keys are encoded, but I don't see that issue (#3) having a huge impact on use cases. So is #2 the main disagreement?
>>> 
>>> We are mainly interested in upserts and deletes by natural key. We are not interested in more powerful queries of the types mentioned in the doc.
>>> 
>>> On the other hand, in our cases, the upserts and deletes are generated upstream. In other words, I have an incremental transformation job J1 with inputs A and B, producing an output C. I can generate the upserts and deletes directly from my inputs, without referring to the current state of C.
>>> 
>>> C might not be partitioned or sorted by the natural key, either, which means that finding the synthetic key can be expensive.
>>> 
>>> In our bespoke model, we are able to generate a new delta (and append it to the dataset) without referring to the existing base or existing deltas.
>>> 
>>> Another apparent divergence is that my jobs are incremental and want to be able to track the state they have seen so far in order to consume only new deltas in their next execution. So if job J2 consumes C, it needs to know that the deltas since its last read will not have been compacted into the base. This implies support for multiple generations of deltas, as different consumers could be at different points in the stream of deltas.
>>>  
>>>>>> On Wed, May 15, 2019 at 3:55 PM Ryan Blue <rb...@netflix.com.invalid> wrote:
>>>>>> 2. Iceberg diff files should use synthetic keys
>>>>>> 
>>>>>> A lot of the discussion on the doc is about whether natural keys are practical or what assumptions we can make or trade about them. In my opinion, Iceberg tables will absolutely need natural keys for reasonable use cases. And those natural keys will need to be unique. And Iceberg will need to rely on engines to enforce that uniqueness.
>>>>>> 
>>>>>> But, there is a difference between table behavior and implementation. We can use synthetic keys to implement the requirements of natural keys. Each row should be identified by its file and position in a file. When deleting by a natural key, we just need to find out what the synthetic key is and encode that in the delete diff.
>>>>>> 
>>>>> This comment has important implications for the effort required to generate delete diff files. I've tried to cover why in comments I added today to the doc, but it could also be a topic of the hangout.
>>>> 
>>>> Do you mean that you can't encode a delete without reading data to locate the affected rows?
>>> 
>>> Yes.
>>>  
>>>> 
>>>>>> 3. Synthetic keys should be based on filename and position
>>>>>> 
>>>>>> I think identifying the file in a synthetic key makes a lot of sense. This would allow for delta file reuse as individual files are rewritten by a “major” compaction and provides nice flexibility that fits with the format. We will need to think through all the impacts, like how file relocation works (e.g., move between regions) and the requirements for rewrites (must apply the delta when rewriting).
>>>>>> 
>>>>> I'm confused. I feel like specifying the filename has the opposite effect. One of the biggest advantages of Iceberg is the decoupling of a dataset from physical location of the constituent files. If a delta file encodes the filename of the row that it updates/deletes you are putting a significant constraint on the way that an implementation can manipulate those files later.
>>>> 
>>>> If I understand your concern, it is that we are encoding a file location in the delete diff. We could solve this with a level of indirection like an ID for data files in table metadata. So there are options to make sure we can still move files and data around. 
>>>> 
>>>> What I like about using a filename or file-specific identifier is that the deltas are tied to a particular file. When that file is deleted, the delta no longer needs to be carried forward. So if I have a maintenance job that is compacting small files, it must apply deletes when rewriting all of those files. But we don't have to rewrite or replace the delete diff file because its deletes were scoped to a file that no longer exists.
>>> 
>>> My intuition is that you overestimate the complexity of maintaining a sequence of deltas and underestimate the constraints that would be imposed by the "level of indirection like an ID for data files in table metadata". Prior to this proposal, there were many legitimate and useful things one could do to the files making up a dataset between two snapshots. For example, one could split a file into multiple files using a new partition scheme. How does the level of indirection work when what was previously one file becomes five?
>>> 
>>>> This is a good way around an ugly problem of knowing when to apply a particular delete diff. Say we are using a UUID column for a natural key. Then deletes just need to encode a set of UUIDs. But when a row is upserted, it gets deleted and then re-appended with the same (natural) UUID key. So we have to scope the delete to just the file that contained the original row and not the inserted data file. It is tempting to use snapshot ordering for this (delete diff applies to versions < V) but we eventually lose the order of snapshots when they expire. We could also use insert diff files, but then replacing the same row more than once has the same problem (how does a delete apply to an insert diff?). Then to correctly encode state, we would have to either solve the order problem or require a minor compaction.
>>> 
>>> > lose the order of snapshots when they expire
>>> 
>>> Generations of data can be considered distinct from snapshots. Within a given snapshot, the files would be grouped by generation. The rules for expiring snapshots would be orthogonal to the rules for compacting generations. And the rules for GC'ing individual files would be unaffected.
>>> 
>>>> Scoping the delete diffs using a file solves this problem tying the lifecycle of deltas to the lifecycle of files. If a file is live in the table, then all deletes for that file must be applied when reading or compacting it. When compacting, you can either write a delete diff for the new file or remove the deleted records. That requirement seems pretty reasonable to me.
>>> 
>>> Iceberg has always had to consider which files need to be accessed in order to find all data relevant to a query. Applying deltas to a base (whether for merge-on-read or for compaction) is not different. One first translates filter predicates to partition predicates, picks up all partitions that might contain relevant data, applies remaining filter predicates, and then one collapses base data with subsequent deltas on the fly.

Re: Updates/Deletes/Upserts in Iceberg

[Erik Wright <er...@shopify.com.INVALID>]

On Tue, May 21, 2019 at 2:01 PM Jacques Nadeau <ja...@dremio.com> wrote:

> I think we just need to have further discussion about keys. Ryan said:
>
> 3. Synthetic keys should be based on filename and position
>
>
> But I'm not clear there is consensus around that. I'm also not sure
> whether he means lossless inclusion, simply derived-from or something else.
> My thinking before is you must have synthetic keys in many cases since
> solving concurrency becomes challenging otherwise.
>

It would be useful to describe the types of concurrent operations that
would be supported (i.e., failed snapshotting could easily be recovered,
vs. the whole operation needing to be re-executed) vs. those that wouldn't.
Solving for unlimited concurrency cases may create way more complexity than
is necessary.

Erik said:
>
>> C might not be partitioned or sorted by the natural key, either, which
>> means that finding the synthetic key can be expensive.
>
>
> But I'm not sure why this needs to be true. If we give control over the
> creation of synthetic key, wouldn't that resolve this issue?
>

Synthetic seems relative. If the synthetic key is client-supplied, in what
way is it relevant to Iceberg whether it is synthetic vs. natural? By
calling it synthetic within Iceberg there is a strong implication that it
is the implementation that generates it (the filename/position key suggests
that). If it's the client that supplies it, it _may_ be synthetic (from the
point of view of the overall data model; i.e. a customer key in a database
vs. a customer ID that shows up on a bill) but from Iceberg's case that
doesn't matter. Only the unicity constraint does.

On the other hand, everything _but_ the thing about filename/position can
probably be resolved. One solution could be to allow either user-specified
_or_ filename/position-based keys in delta-files. If you use the latter,
there are constraints on the types of operations you can do. If you use the
former, there are constraints on the types of concurrency that are
supported (I'm not sure about that, really, just going off of your comment
above).


> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
>
> On Tue, May 21, 2019 at 7:54 AM Erik Wright
> <er...@shopify.com.invalid> wrote:
>
>> On Thu, May 16, 2019 at 4:13 PM Ryan Blue <rb...@netflix.com> wrote:
>>
>>> Replies inline.
>>>
>>> On Thu, May 16, 2019 at 10:07 AM Erik Wright <er...@shopify.com>
>>> wrote:
>>>
>>>> I would be happy to participate. Iceberg with merge-on-read
>>>> capabilities is a technology choice that my team is actively considering.
>>>> It appears that our scenario differs meaningfully from the one that Anton
>>>> and Miguel are considering. It would be great to take the time to compare
>>>> the two and see if there is a single implementation that can meet the needs
>>>> of each scenario.
>>>>
>>>
>>> Can you be more specific about where the use cases differ meaningfully?
>>> I think that if we agree that operations on natural keys can be implemented
>>> using synthetic keys to encode deletes (#2), then everyone is aligned on
>>> the core parts of a design. We can figure out the implications of how
>>> synthetic keys are encoded, but I don't see that issue (#3) having a huge
>>> impact on use cases. So is #2 the main disagreement?
>>>
>>
>> We are mainly interested in upserts and deletes by natural key. We are
>> not interested in more powerful queries of the types mentioned in the doc.
>>
>> On the other hand, in our cases, the upserts and deletes are generated
>> upstream. In other words, I have an incremental transformation job J1 with
>> inputs A and B, producing an output C. I can generate the upserts and
>> deletes directly from my inputs, without referring to the current state of
>> C.
>>
>> C might not be partitioned or sorted by the natural key, either, which
>> means that finding the synthetic key can be expensive.
>>
>> In our bespoke model, we are able to generate a new delta (and append it
>> to the dataset) without referring to the existing base or existing deltas.
>>
>> Another apparent divergence is that my jobs are incremental and want to
>> be able to track the state they have seen so far in order to consume only
>> new deltas in their next execution. So if job J2 consumes C, it needs to
>> know that the deltas since its last read will not have been compacted into
>> the base. This implies support for multiple generations of deltas, as
>> different consumers could be at different points in the stream of deltas.
>>
>>
>>> On Wed, May 15, 2019 at 3:55 PM Ryan Blue <rb...@netflix.com.invalid>
>>>> wrote:
>>>>
>>>>> *2. Iceberg diff files should use synthetic keys*
>>>>>
>>>>> A lot of the discussion on the doc is about whether natural keys are
>>>>> practical or what assumptions we can make or trade about them. In my
>>>>> opinion, Iceberg tables will absolutely need natural keys for reasonable
>>>>> use cases. And those natural keys will need to be unique. And Iceberg will
>>>>> need to rely on engines to enforce that uniqueness.
>>>>>
>>>>> But, there is a difference between table behavior and implementation.
>>>>> We can use synthetic keys to implement the requirements of natural keys.
>>>>> Each row should be identified by its file and position in a file. When
>>>>> deleting by a natural key, we just need to find out what the synthetic key
>>>>> is and encode that in the delete diff.
>>>>>
>>>> This comment has important implications for the effort required to
>>>> generate delete diff files. I've tried to cover why in comments I added
>>>> today to the doc, but it could also be a topic of the hangout.
>>>>
>>>
>>> Do you mean that you can't encode a delete without reading data to
>>> locate the affected rows?
>>>
>>
>> Yes.
>>
>>
>>>
>>> *3. Synthetic keys should be based on filename and position*
>>>>>
>>>>> I think identifying the file in a synthetic key makes a lot of sense.
>>>>> This would allow for delta file reuse as individual files are rewritten by
>>>>> a “major” compaction and provides nice flexibility that fits with the
>>>>> format. We will need to think through all the impacts, like how file
>>>>> relocation works (e.g., move between regions) and the requirements for
>>>>> rewrites (must apply the delta when rewriting).
>>>>>
>>>> I'm confused. I feel like specifying the filename has the opposite
>>>> effect. One of the biggest advantages of Iceberg is the decoupling of a
>>>> dataset from physical location of the constituent files. If a delta file
>>>> encodes the filename of the row that it updates/deletes you are putting a
>>>> significant constraint on the way that an implementation can manipulate
>>>> those files later.
>>>>
>>>
>>> If I understand your concern, it is that we are encoding a file location
>>> in the delete diff. We could solve this with a level of indirection like an
>>> ID for data files in table metadata. So there are options to make sure we
>>> can still move files and data around.
>>>
>>
>>> What I like about using a filename or file-specific identifier is that
>>> the deltas are tied to a particular file. When that file is deleted, the
>>> delta no longer needs to be carried forward. So if I have a maintenance job
>>> that is compacting small files, it must apply deletes when rewriting all of
>>> those files. But we don't have to rewrite or replace the delete diff file
>>> because its deletes were scoped to a file that no longer exists.
>>>
>>
>> My intuition is that you overestimate the complexity of maintaining a
>> sequence of deltas and underestimate the constraints that would be imposed
>> by the "level of indirection like an ID for data files in table metadata".
>> Prior to this proposal, there were many legitimate and useful things one
>> could do to the files making up a dataset between two snapshots. For
>> example, one could split a file into multiple files using a new partition
>> scheme. How does the level of indirection work when what was previously one
>> file becomes five?
>>
>> This is a good way around an ugly problem of knowing when to apply a
>>> particular delete diff. Say we are using a UUID column for a natural key.
>>> Then deletes just need to encode a set of UUIDs. But when a row is
>>> upserted, it gets deleted and then re-appended with the same (natural) UUID
>>> key. So we have to scope the delete to just the file that contained the
>>> original row and not the inserted data file. It is tempting to use snapshot
>>> ordering for this (delete diff applies to versions < V) but we eventually
>>> lose the order of snapshots when they expire. We could also use insert diff
>>> files, but then replacing the same row more than once has the same problem
>>> (how does a delete apply to an insert diff?). Then to correctly encode
>>> state, we would have to either solve the order problem or require a minor
>>> compaction.
>>>
>>
>> > lose the order of snapshots when they expire
>>
>> Generations of data can be considered distinct from snapshots. Within a
>> given snapshot, the files would be grouped by generation. The rules for
>> expiring snapshots would be orthogonal to the rules for compacting
>> generations. And the rules for GC'ing individual files would be unaffected.
>>
>> Scoping the delete diffs using a file solves this problem tying the
>>> lifecycle of deltas to the lifecycle of files. If a file is live in the
>>> table, then all deletes for that file must be applied when reading or
>>> compacting it. When compacting, you can either write a delete diff for the
>>> new file or remove the deleted records. That requirement seems pretty
>>> reasonable to me.
>>>
>>
>> Iceberg has always had to consider which files need to be accessed in
>> order to find all data relevant to a query. Applying deltas to a base
>> (whether for merge-on-read or for compaction) is not different. One first
>> translates filter predicates to partition predicates, picks up all
>> partitions that might contain relevant data, applies remaining filter
>> predicates, and then one collapses base data with subsequent deltas on the
>> fly.
>>
>

Re: Updates/Deletes/Upserts in Iceberg

[Jacques Nadeau <ja...@dremio.com>]

I think we just need to have further discussion about keys. Ryan said:

3. Synthetic keys should be based on filename and position


But I'm not clear there is consensus around that. I'm also not sure whether
he means lossless inclusion, simply derived-from or something else. My
thinking before is you must have synthetic keys in many cases since solving
concurrency becomes challenging otherwise.

Erik said:

> C might not be partitioned or sorted by the natural key, either, which
> means that finding the synthetic key can be expensive.


But I'm not sure why this needs to be true. If we give control over the
creation of synthetic key, wouldn't that resolve this issue?

--
Jacques Nadeau
CTO and Co-Founder, Dremio


On Tue, May 21, 2019 at 7:54 AM Erik Wright <er...@shopify.com.invalid>
wrote:

> On Thu, May 16, 2019 at 4:13 PM Ryan Blue <rb...@netflix.com> wrote:
>
>> Replies inline.
>>
>> On Thu, May 16, 2019 at 10:07 AM Erik Wright <er...@shopify.com>
>> wrote:
>>
>>> I would be happy to participate. Iceberg with merge-on-read capabilities
>>> is a technology choice that my team is actively considering. It appears
>>> that our scenario differs meaningfully from the one that Anton and Miguel
>>> are considering. It would be great to take the time to compare the two and
>>> see if there is a single implementation that can meet the needs of each
>>> scenario.
>>>
>>
>> Can you be more specific about where the use cases differ meaningfully? I
>> think that if we agree that operations on natural keys can be implemented
>> using synthetic keys to encode deletes (#2), then everyone is aligned on
>> the core parts of a design. We can figure out the implications of how
>> synthetic keys are encoded, but I don't see that issue (#3) having a huge
>> impact on use cases. So is #2 the main disagreement?
>>
>
> We are mainly interested in upserts and deletes by natural key. We are not
> interested in more powerful queries of the types mentioned in the doc.
>
> On the other hand, in our cases, the upserts and deletes are generated
> upstream. In other words, I have an incremental transformation job J1 with
> inputs A and B, producing an output C. I can generate the upserts and
> deletes directly from my inputs, without referring to the current state of
> C.
>
> C might not be partitioned or sorted by the natural key, either, which
> means that finding the synthetic key can be expensive.
>
> In our bespoke model, we are able to generate a new delta (and append it
> to the dataset) without referring to the existing base or existing deltas.
>
> Another apparent divergence is that my jobs are incremental and want to be
> able to track the state they have seen so far in order to consume only new
> deltas in their next execution. So if job J2 consumes C, it needs to know
> that the deltas since its last read will not have been compacted into the
> base. This implies support for multiple generations of deltas, as different
> consumers could be at different points in the stream of deltas.
>
>
>> On Wed, May 15, 2019 at 3:55 PM Ryan Blue <rb...@netflix.com.invalid>
>>> wrote:
>>>
>>>> *2. Iceberg diff files should use synthetic keys*
>>>>
>>>> A lot of the discussion on the doc is about whether natural keys are
>>>> practical or what assumptions we can make or trade about them. In my
>>>> opinion, Iceberg tables will absolutely need natural keys for reasonable
>>>> use cases. And those natural keys will need to be unique. And Iceberg will
>>>> need to rely on engines to enforce that uniqueness.
>>>>
>>>> But, there is a difference between table behavior and implementation.
>>>> We can use synthetic keys to implement the requirements of natural keys.
>>>> Each row should be identified by its file and position in a file. When
>>>> deleting by a natural key, we just need to find out what the synthetic key
>>>> is and encode that in the delete diff.
>>>>
>>> This comment has important implications for the effort required to
>>> generate delete diff files. I've tried to cover why in comments I added
>>> today to the doc, but it could also be a topic of the hangout.
>>>
>>
>> Do you mean that you can't encode a delete without reading data to locate
>> the affected rows?
>>
>
> Yes.
>
>
>>
>> *3. Synthetic keys should be based on filename and position*
>>>>
>>>> I think identifying the file in a synthetic key makes a lot of sense.
>>>> This would allow for delta file reuse as individual files are rewritten by
>>>> a “major” compaction and provides nice flexibility that fits with the
>>>> format. We will need to think through all the impacts, like how file
>>>> relocation works (e.g., move between regions) and the requirements for
>>>> rewrites (must apply the delta when rewriting).
>>>>
>>> I'm confused. I feel like specifying the filename has the opposite
>>> effect. One of the biggest advantages of Iceberg is the decoupling of a
>>> dataset from physical location of the constituent files. If a delta file
>>> encodes the filename of the row that it updates/deletes you are putting a
>>> significant constraint on the way that an implementation can manipulate
>>> those files later.
>>>
>>
>> If I understand your concern, it is that we are encoding a file location
>> in the delete diff. We could solve this with a level of indirection like an
>> ID for data files in table metadata. So there are options to make sure we
>> can still move files and data around.
>>
>
>> What I like about using a filename or file-specific identifier is that
>> the deltas are tied to a particular file. When that file is deleted, the
>> delta no longer needs to be carried forward. So if I have a maintenance job
>> that is compacting small files, it must apply deletes when rewriting all of
>> those files. But we don't have to rewrite or replace the delete diff file
>> because its deletes were scoped to a file that no longer exists.
>>
>
> My intuition is that you overestimate the complexity of maintaining a
> sequence of deltas and underestimate the constraints that would be imposed
> by the "level of indirection like an ID for data files in table metadata".
> Prior to this proposal, there were many legitimate and useful things one
> could do to the files making up a dataset between two snapshots. For
> example, one could split a file into multiple files using a new partition
> scheme. How does the level of indirection work when what was previously one
> file becomes five?
>
> This is a good way around an ugly problem of knowing when to apply a
>> particular delete diff. Say we are using a UUID column for a natural key.
>> Then deletes just need to encode a set of UUIDs. But when a row is
>> upserted, it gets deleted and then re-appended with the same (natural) UUID
>> key. So we have to scope the delete to just the file that contained the
>> original row and not the inserted data file. It is tempting to use snapshot
>> ordering for this (delete diff applies to versions < V) but we eventually
>> lose the order of snapshots when they expire. We could also use insert diff
>> files, but then replacing the same row more than once has the same problem
>> (how does a delete apply to an insert diff?). Then to correctly encode
>> state, we would have to either solve the order problem or require a minor
>> compaction.
>>
>
> > lose the order of snapshots when they expire
>
> Generations of data can be considered distinct from snapshots. Within a
> given snapshot, the files would be grouped by generation. The rules for
> expiring snapshots would be orthogonal to the rules for compacting
> generations. And the rules for GC'ing individual files would be unaffected.
>
> Scoping the delete diffs using a file solves this problem tying the
>> lifecycle of deltas to the lifecycle of files. If a file is live in the
>> table, then all deletes for that file must be applied when reading or
>> compacting it. When compacting, you can either write a delete diff for the
>> new file or remove the deleted records. That requirement seems pretty
>> reasonable to me.
>>
>
> Iceberg has always had to consider which files need to be accessed in
> order to find all data relevant to a query. Applying deltas to a base
> (whether for merge-on-read or for compaction) is not different. One first
> translates filter predicates to partition predicates, picks up all
> partitions that might contain relevant data, applies remaining filter
> predicates, and then one collapses base data with subsequent deltas on the
> fly.
>

Re: Updates/Deletes/Upserts in Iceberg

[Erik Wright <er...@shopify.com.INVALID>]

On Thu, May 16, 2019 at 4:13 PM Ryan Blue <rb...@netflix.com> wrote:

> Replies inline.
>
> On Thu, May 16, 2019 at 10:07 AM Erik Wright <er...@shopify.com>
> wrote:
>
>> I would be happy to participate. Iceberg with merge-on-read capabilities
>> is a technology choice that my team is actively considering. It appears
>> that our scenario differs meaningfully from the one that Anton and Miguel
>> are considering. It would be great to take the time to compare the two and
>> see if there is a single implementation that can meet the needs of each
>> scenario.
>>
>
> Can you be more specific about where the use cases differ meaningfully? I
> think that if we agree that operations on natural keys can be implemented
> using synthetic keys to encode deletes (#2), then everyone is aligned on
> the core parts of a design. We can figure out the implications of how
> synthetic keys are encoded, but I don't see that issue (#3) having a huge
> impact on use cases. So is #2 the main disagreement?
>

We are mainly interested in upserts and deletes by natural key. We are not
interested in more powerful queries of the types mentioned in the doc.

On the other hand, in our cases, the upserts and deletes are generated
upstream. In other words, I have an incremental transformation job J1 with
inputs A and B, producing an output C. I can generate the upserts and
deletes directly from my inputs, without referring to the current state of
C.

C might not be partitioned or sorted by the natural key, either, which
means that finding the synthetic key can be expensive.

In our bespoke model, we are able to generate a new delta (and append it to
the dataset) without referring to the existing base or existing deltas.

Another apparent divergence is that my jobs are incremental and want to be
able to track the state they have seen so far in order to consume only new
deltas in their next execution. So if job J2 consumes C, it needs to know
that the deltas since its last read will not have been compacted into the
base. This implies support for multiple generations of deltas, as different
consumers could be at different points in the stream of deltas.


> On Wed, May 15, 2019 at 3:55 PM Ryan Blue <rb...@netflix.com.invalid>
>> wrote:
>>
>>> *2. Iceberg diff files should use synthetic keys*
>>>
>>> A lot of the discussion on the doc is about whether natural keys are
>>> practical or what assumptions we can make or trade about them. In my
>>> opinion, Iceberg tables will absolutely need natural keys for reasonable
>>> use cases. And those natural keys will need to be unique. And Iceberg will
>>> need to rely on engines to enforce that uniqueness.
>>>
>>> But, there is a difference between table behavior and implementation. We
>>> can use synthetic keys to implement the requirements of natural keys. Each
>>> row should be identified by its file and position in a file. When deleting
>>> by a natural key, we just need to find out what the synthetic key is and
>>> encode that in the delete diff.
>>>
>> This comment has important implications for the effort required to
>> generate delete diff files. I've tried to cover why in comments I added
>> today to the doc, but it could also be a topic of the hangout.
>>
>
> Do you mean that you can't encode a delete without reading data to locate
> the affected rows?
>

Yes.


>
> *3. Synthetic keys should be based on filename and position*
>>>
>>> I think identifying the file in a synthetic key makes a lot of sense.
>>> This would allow for delta file reuse as individual files are rewritten by
>>> a “major” compaction and provides nice flexibility that fits with the
>>> format. We will need to think through all the impacts, like how file
>>> relocation works (e.g., move between regions) and the requirements for
>>> rewrites (must apply the delta when rewriting).
>>>
>> I'm confused. I feel like specifying the filename has the opposite
>> effect. One of the biggest advantages of Iceberg is the decoupling of a
>> dataset from physical location of the constituent files. If a delta file
>> encodes the filename of the row that it updates/deletes you are putting a
>> significant constraint on the way that an implementation can manipulate
>> those files later.
>>
>
> If I understand your concern, it is that we are encoding a file location
> in the delete diff. We could solve this with a level of indirection like an
> ID for data files in table metadata. So there are options to make sure we
> can still move files and data around.
>

> What I like about using a filename or file-specific identifier is that the
> deltas are tied to a particular file. When that file is deleted, the delta
> no longer needs to be carried forward. So if I have a maintenance job that
> is compacting small files, it must apply deletes when rewriting all of
> those files. But we don't have to rewrite or replace the delete diff file
> because its deletes were scoped to a file that no longer exists.
>

My intuition is that you overestimate the complexity of maintaining a
sequence of deltas and underestimate the constraints that would be imposed
by the "level of indirection like an ID for data files in table metadata".
Prior to this proposal, there were many legitimate and useful things one
could do to the files making up a dataset between two snapshots. For
example, one could split a file into multiple files using a new partition
scheme. How does the level of indirection work when what was previously one
file becomes five?

This is a good way around an ugly problem of knowing when to apply a
> particular delete diff. Say we are using a UUID column for a natural key.
> Then deletes just need to encode a set of UUIDs. But when a row is
> upserted, it gets deleted and then re-appended with the same (natural) UUID
> key. So we have to scope the delete to just the file that contained the
> original row and not the inserted data file. It is tempting to use snapshot
> ordering for this (delete diff applies to versions < V) but we eventually
> lose the order of snapshots when they expire. We could also use insert diff
> files, but then replacing the same row more than once has the same problem
> (how does a delete apply to an insert diff?). Then to correctly encode
> state, we would have to either solve the order problem or require a minor
> compaction.
>

> lose the order of snapshots when they expire

Generations of data can be considered distinct from snapshots. Within a
given snapshot, the files would be grouped by generation. The rules for
expiring snapshots would be orthogonal to the rules for compacting
generations. And the rules for GC'ing individual files would be unaffected.

Scoping the delete diffs using a file solves this problem tying the
> lifecycle of deltas to the lifecycle of files. If a file is live in the
> table, then all deletes for that file must be applied when reading or
> compacting it. When compacting, you can either write a delete diff for the
> new file or remove the deleted records. That requirement seems pretty
> reasonable to me.
>

Iceberg has always had to consider which files need to be accessed in order
to find all data relevant to a query. Applying deltas to a base (whether
for merge-on-read or for compaction) is not different. One first translates
filter predicates to partition predicates, picks up all partitions that
might contain relevant data, applies remaining filter predicates, and then
one collapses base data with subsequent deltas on the fly.

Fwd: Updates/Deletes/Upserts in Iceberg

[Ryan Blue <rb...@netflix.com.INVALID>]

Sorry, I didn't mean to reply to only Erik. Here's my response from
yesterday.

---------- Forwarded message ---------
From: Ryan Blue <rb...@netflix.com>
Date: Thu, May 16, 2019 at 1:13 PM
Subject: Re: Updates/Deletes/Upserts in Iceberg
To: Erik Wright <er...@shopify.com>


Replies inline.

On Thu, May 16, 2019 at 10:07 AM Erik Wright <er...@shopify.com>
wrote:

> I would be happy to participate. Iceberg with merge-on-read capabilities
> is a technology choice that my team is actively considering. It appears
> that our scenario differs meaningfully from the one that Anton and Miguel
> are considering. It would be great to take the time to compare the two and
> see if there is a single implementation that can meet the needs of each
> scenario.
>

Can you be more specific about where the use cases differ meaningfully? I
think that if we agree that operations on natural keys can be implemented
using synthetic keys to encode deletes (#2), then everyone is aligned on
the core parts of a design. We can figure out the implications of how
synthetic keys are encoded, but I don't see that issue (#3) having a huge
impact on use cases. So is #2 the main disagreement?


> On Wed, May 15, 2019 at 3:55 PM Ryan Blue <rb...@netflix.com.invalid>
> wrote:
>
>> *2. Iceberg diff files should use synthetic keys*
>>
>> A lot of the discussion on the doc is about whether natural keys are
>> practical or what assumptions we can make or trade about them. In my
>> opinion, Iceberg tables will absolutely need natural keys for reasonable
>> use cases. And those natural keys will need to be unique. And Iceberg will
>> need to rely on engines to enforce that uniqueness.
>>
>> But, there is a difference between table behavior and implementation. We
>> can use synthetic keys to implement the requirements of natural keys. Each
>> row should be identified by its file and position in a file. When deleting
>> by a natural key, we just need to find out what the synthetic key is and
>> encode that in the delete diff.
>>
> This comment has important implications for the effort required to
> generate delete diff files. I've tried to cover why in comments I added
> today to the doc, but it could also be a topic of the hangout.
>

Do you mean that you can't encode a delete without reading data to locate
the affected rows?

*3. Synthetic keys should be based on filename and position*
>>
>> I think identifying the file in a synthetic key makes a lot of sense.
>> This would allow for delta file reuse as individual files are rewritten by
>> a “major” compaction and provides nice flexibility that fits with the
>> format. We will need to think through all the impacts, like how file
>> relocation works (e.g., move between regions) and the requirements for
>> rewrites (must apply the delta when rewriting).
>>
> I'm confused. I feel like specifying the filename has the opposite effect.
> One of the biggest advantages of Iceberg is the decoupling of a dataset
> from physical location of the constituent files. If a delta file encodes
> the filename of the row that it updates/deletes you are putting a
> significant constraint on the way that an implementation can manipulate
> those files later.
>

If I understand your concern, it is that we are encoding a file location in
the delete diff. We could solve this with a level of indirection like an ID
for data files in table metadata. So there are options to make sure we can
still move files and data around.

What I like about using a filename or file-specific identifier is that the
deltas are tied to a particular file. When that file is deleted, the delta
no longer needs to be carried forward. So if I have a maintenance job that
is compacting small files, it must apply deletes when rewriting all of
those files. But we don't have to rewrite or replace the delete diff file
because its deletes were scoped to a file that no longer exists.

This is a good way around an ugly problem of knowing when to apply a
particular delete diff. Say we are using a UUID column for a natural key.
Then deletes just need to encode a set of UUIDs. But when a row is
upserted, it gets deleted and then re-appended with the same (natural) UUID
key. So we have to scope the delete to just the file that contained the
original row and not the inserted data file. It is tempting to use snapshot
ordering for this (delete diff applies to versions < V) but we eventually
lose the order of snapshots when they expire. We could also use insert diff
files, but then replacing the same row more than once has the same problem
(how does a delete apply to an insert diff?). Then to correctly encode
state, we would have to either solve the order problem or require a minor
compaction.

Scoping the delete diffs using a file solves this problem tying the
lifecycle of deltas to the lifecycle of files. If a file is live in the
table, then all deletes for that file must be applied when reading or
compacting it. When compacting, you can either write a delete diff for the
new file or remove the deleted records. That requirement seems pretty
reasonable to me.


-- 
Ryan Blue
Software Engineer
Netflix

Re: Updates/Deletes/Upserts in Iceberg

[Erik Wright <er...@shopify.com.INVALID>]

I would be happy to participate. Iceberg with merge-on-read capabilities is
a technology choice that my team is actively considering. It appears that
our scenario differs meaningfully from the one that Anton and Miguel are
considering. It would be great to take the time to compare the two and see
if there is a single implementation that can meet the needs of each
scenario.

On Wed, May 15, 2019 at 3:55 PM Ryan Blue <rb...@netflix.com.invalid> wrote:

> Thanks for working on this, Anton and Miguel!
>
> Would anyone be interested in scheduling a hangout to talk about next
> steps and tentative design choices?
>
> The doc is a great start and does a good job laying out the trade-offs
> between different approaches. I appreciate the idea to get a discussion
> started and not to pick one particular approach, but I think that it does
> make a few choices clear:
>
> *1. Iceberg should support lazy (read-side) merging using diff files*
>
> The eager approach doesn’t require much beyond Iceberg’s existing support.
> Adding diff files is the next step for engines that need to implement lazy
> merging for merge/upsert/delete. I support adding these structures to the
> spec (as a new format version).
>
> *2. Iceberg diff files should use synthetic keys*
>
> A lot of the discussion on the doc is about whether natural keys are
> practical or what assumptions we can make or trade about them. In my
> opinion, Iceberg tables will absolutely need natural keys for reasonable
> use cases. And those natural keys will need to be unique. And Iceberg will
> need to rely on engines to enforce that uniqueness.
>
> But, there is a difference between table behavior and implementation. We
> can use synthetic keys to implement the requirements of natural keys. Each
> row should be identified by its file and position in a file. When deleting
> by a natural key, we just need to find out what the synthetic key is and
> encode that in the delete diff.
>
This comment has important implications for the effort required to generate
delete diff files. I've tried to cover why in comments I added today to the
doc, but it could also be a topic of the hangout.


> With the physical representation using synthetic keys, we should also
> define how to communicate a natural key constraint for a table. That way,
> writers can fail if a write may violate the key constraints of a table.
>
> *3. Synthetic keys should be based on filename and position*
>
> I think identifying the file in a synthetic key makes a lot of sense. This
> would allow for delta file reuse as individual files are rewritten by a
> “major” compaction and provides nice flexibility that fits with the format.
> We will need to think through all the impacts, like how file relocation
> works (e.g., move between regions) and the requirements for rewrites (must
> apply the delta when rewriting).
>
I'm confused. I feel like specifying the filename has the opposite effect.
One of the biggest advantages of Iceberg is the decoupling of a dataset
from physical location of the constituent files. If a delta file encodes
the filename of the row that it updates/deletes you are putting a
significant constraint on the way that an implementation can manipulate
those files later.


> *Open questions*
>
> There are also quite a few remaining questions for a design:
>
>    - Should Iceberg use insert diff files? (My initial answer is no)
>    - Should Iceberg require diff compaction? Iceberg could require one
>    delete diff per partition, for example. (My answer: no)
>    - Should data files store synthetic key position? If so, why?
>    - Should there be a dense format for deletes, or just a sparse format?
>    - What is the scope of a delete diff? At a minimum, partition. But
>    does it make sense to build ways to restrict scope further?
>
>
> On Fri, May 10, 2019 at 11:27 AM Anton Okolnychyi
> <ao...@apple.com.invalid> wrote:
>
>> We did take a look at Hudi. The overall design seems to be pretty
>> complicated and, unfortunately, I didn’t have time to explore every detail.
>>
>> Here is my understanding (correct me if I am wrong):
>>
>> - Hudi has RECORD_KEY, which is expected to be unique.
>> - Hudi has PRECOMBINED_KEY, which is used to pick only one row in the
>> incoming batch if there are multiple rows with the same key. As I
>> understand, this isn't used on reads. It is used on writes to deduplicate
>> rows with identical keys within one incoming batch. For example, if we are
>> inserting 10 records and two rows have the same key, PRECOMBINED_KEY will
>> be used to pick up only one row.
>> - Once Hudi ensures the uniqueness of RECORD_KEY within the incoming
>> batch, it loads the Bloom filter index from all existing Parquet files in
>> the involved partitions (meaning, partitions spread from the input batch)
>> and tags each record as either an update or insert by mapping the incoming
>> keys to existing files for updates. At this point, it seems to rely on join.
>>
>> Is my understanding correct? If so, do we want to consider joins on
>> write? We mentioned this technique as one way to ensure the uniqueness of
>> natural keys but we were concerned about the performance. Also, does Hudi
>> support record-level updates?
>>
>> Thanks,
>> Anton
>>
>> On 10 May 2019, at 18:22, Erik Wright <er...@shopify.com.INVALID>
>> wrote:
>>
>> Thanks for putting this forward.
>>
>> Another term for the "lazy" approach would be "merge on read".
>>
>> My team has built something internally that uses merge-on-read internally
>> but uses an "Eager" materialization for publication to Presto. Roughly, we
>> maintain a table metadata file that looks a bit like Iceberg's and tracks
>> the "live" version of each partition as it is updated over time. We are
>> looking into a solution that will allow us to push the merge-on-read all
>> the way to Presto (and other consumers), and adding Merge-On-Read to
>> Iceberg is one of the approaches we are considering.
>>
>> It's worth noting that Hudi does have support for upserts/deletes as
>> well, so that's another model to consider.
>>
>> On Fri, May 10, 2019 at 8:30 AM Miguel Miranda <
>> miguelnmiranda@apple.com.invalid> wrote:
>>
>>> Hi,
>>>
>>> As Anton said, we purposely avoided making a "decision" on which
>>> approach should be implemented in order to allow for a meaningful
>>> discussion with the community.
>>>
>>> The document starts with an eager approach as it is straightforward and
>>> easy to understand: steps resemble the usual file level
>>> operations/manipulations frequently used by engineers when implementing
>>> Update/Delete/Upsert behaviour themselves, hopefully creating a conceptual
>>> bridge to the more involved designs. Right now, Iceberg has almost
>>> everything to implement the "eager" approach as we simply need to adjust
>>> the retry mechanism. For example, I have implemented a prototype of the
>>> eager solution with Spark and Iceberg.
>>>
>>> We looked into many existing solutions for inspiration, but when there
>>> isn't a paper or code in the public domain it becomes hard to assess the
>>> underlying design, although some of it can be inferred from the API or
>>> documentation.
>>>
>>> Best,
>>> Miguel
>>>
>>> On 10 May 2019, at 11:57, Anton Okolnychyi <ao...@apple.com>
>>> wrote:
>>>
>>> Thanks for the feedback, Jacques!
>>>
>>> You are correct, we kept the question of the best approach as open :)
>>> The idea was to have a discussion in the community. Hopefully, we can reach
>>> a consensus.
>>>
>>> While the proposed “lazy” approaches certainly offer significant
>>> benefits, they require more changes in Iceberg as well as in readers/query
>>> engines (depending on how we want to merge base and diff files). For us, it
>>> is important to understand whether the Iceberg community would even
>>> consider such changes.
>>>
>>> Hive ACID 3 is one the projects we looked at. In fact, we spoke to Owen,
>>> the original creator of updates/deletes/upserts in Hive. I believe the
>>> “lazy” approaches are close to what Hive 3 does but with their own
>>> distinctions that Iceberg allows us to have. It would be great to have
>>> Owen’s feedback.
>>>
>>> We don’t know the internals of Delta as updates/deletes/upserts are not
>>> open source. My personal guess, yes, it might be similar to the “eager”
>>> approach in our doc.
>>>
>>> Jacques, could you share some insights how you implement the merge of
>>> diffs? Is it done by readers?
>>>
>>> Thanks,
>>> Anton
>>>
>>> On 10 May 2019, at 06:24, Jacques Nadeau <ja...@dremio.com> wrote:
>>>
>>> This is a nice doc and it covers many different options. Upon first
>>> skim, I don't see a strong argument for particular approach. D
>>>
>>> In our own development, we've been leaning heavily towards what you
>>> describe in the document as "lazy with SRI". I believe this is consistent
>>> with what the Hive community did on top of Orc. It's interesting because my
>>> (maybe incorrect) understanding of the Databricks Delta approach is they
>>> chose what you title "eager" in their approach to upserts. They may also
>>> have a lazy approach for other types of mutations but I don't think they do.
>>>
>>> Thanks again for putting this together!
>>> Jacques
>>> --
>>> Jacques Nadeau
>>> CTO and Co-Founder, Dremio
>>>
>>>
>>> On Wed, May 8, 2019 at 3:42 AM Anton Okolnychyi <
>>> aokolnychyi@apple.com.invalid> wrote:
>>>
>>>> Hi folks,
>>>>
>>>> Miguel (cc) and I have spent some time thinking about how to perform
>>>> updates/deletes/upserts on top of Iceberg tables. This functionality is
>>>> essential for many modern use cases. We've summarized our ideas in a doc
>>>> [1], which, hopefully, will trigger a discussion in the community. The
>>>> document presents different conceptual approaches alongside their
>>>> trade-offs. We will be glad to consider any other ideas as well.
>>>>
>>>> Thanks,
>>>> Anton
>>>>
>>>> [1] -
>>>> https://docs.google.com/document/d/1Pk34C3diOfVCRc-sfxfhXZfzvxwum1Odo-6Jj9mwK38/
>>>>
>>>>
>>>>
>>>
>>>
>>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: Updates/Deletes/Upserts in Iceberg

[Ryan Blue <rb...@netflix.com.INVALID>]

Thanks for working on this, Anton and Miguel!

Would anyone be interested in scheduling a hangout to talk about next steps
and tentative design choices?

The doc is a great start and does a good job laying out the trade-offs
between different approaches. I appreciate the idea to get a discussion
started and not to pick one particular approach, but I think that it does
make a few choices clear:

*1. Iceberg should support lazy (read-side) merging using diff files*

The eager approach doesn’t require much beyond Iceberg’s existing support.
Adding diff files is the next step for engines that need to implement lazy
merging for merge/upsert/delete. I support adding these structures to the
spec (as a new format version).

*2. Iceberg diff files should use synthetic keys*

A lot of the discussion on the doc is about whether natural keys are
practical or what assumptions we can make or trade about them. In my
opinion, Iceberg tables will absolutely need natural keys for reasonable
use cases. And those natural keys will need to be unique. And Iceberg will
need to rely on engines to enforce that uniqueness.

But, there is a difference between table behavior and implementation. We
can use synthetic keys to implement the requirements of natural keys. Each
row should be identified by its file and position in a file. When deleting
by a natural key, we just need to find out what the synthetic key is and
encode that in the delete diff.

With the physical representation using synthetic keys, we should also
define how to communicate a natural key constraint for a table. That way,
writers can fail if a write may violate the key constraints of a table.

*3. Synthetic keys should be based on filename and position*

I think identifying the file in a synthetic key makes a lot of sense. This
would allow for delta file reuse as individual files are rewritten by a
“major” compaction and provides nice flexibility that fits with the format.
We will need to think through all the impacts, like how file relocation
works (e.g., move between regions) and the requirements for rewrites (must
apply the delta when rewriting).

*Open questions*

There are also quite a few remaining questions for a design:

   - Should Iceberg use insert diff files? (My initial answer is no)
   - Should Iceberg require diff compaction? Iceberg could require one
   delete diff per partition, for example. (My answer: no)
   - Should data files store synthetic key position? If so, why?
   - Should there be a dense format for deletes, or just a sparse format?
   - What is the scope of a delete diff? At a minimum, partition. But does
   it make sense to build ways to restrict scope further?


On Fri, May 10, 2019 at 11:27 AM Anton Okolnychyi
<ao...@apple.com.invalid> wrote:

> We did take a look at Hudi. The overall design seems to be pretty
> complicated and, unfortunately, I didn’t have time to explore every detail.
>
> Here is my understanding (correct me if I am wrong):
>
> - Hudi has RECORD_KEY, which is expected to be unique.
> - Hudi has PRECOMBINED_KEY, which is used to pick only one row in the
> incoming batch if there are multiple rows with the same key. As I
> understand, this isn't used on reads. It is used on writes to deduplicate
> rows with identical keys within one incoming batch. For example, if we are
> inserting 10 records and two rows have the same key, PRECOMBINED_KEY will
> be used to pick up only one row.
> - Once Hudi ensures the uniqueness of RECORD_KEY within the incoming
> batch, it loads the Bloom filter index from all existing Parquet files in
> the involved partitions (meaning, partitions spread from the input batch)
> and tags each record as either an update or insert by mapping the incoming
> keys to existing files for updates. At this point, it seems to rely on join.
>
> Is my understanding correct? If so, do we want to consider joins on write?
> We mentioned this technique as one way to ensure the uniqueness of natural
> keys but we were concerned about the performance. Also, does Hudi support
> record-level updates?
>
> Thanks,
> Anton
>
> On 10 May 2019, at 18:22, Erik Wright <er...@shopify.com.INVALID>
> wrote:
>
> Thanks for putting this forward.
>
> Another term for the "lazy" approach would be "merge on read".
>
> My team has built something internally that uses merge-on-read internally
> but uses an "Eager" materialization for publication to Presto. Roughly, we
> maintain a table metadata file that looks a bit like Iceberg's and tracks
> the "live" version of each partition as it is updated over time. We are
> looking into a solution that will allow us to push the merge-on-read all
> the way to Presto (and other consumers), and adding Merge-On-Read to
> Iceberg is one of the approaches we are considering.
>
> It's worth noting that Hudi does have support for upserts/deletes as well,
> so that's another model to consider.
>
> On Fri, May 10, 2019 at 8:30 AM Miguel Miranda <
> miguelnmiranda@apple.com.invalid> wrote:
>
>> Hi,
>>
>> As Anton said, we purposely avoided making a "decision" on which approach
>> should be implemented in order to allow for a meaningful discussion with
>> the community.
>>
>> The document starts with an eager approach as it is straightforward and
>> easy to understand: steps resemble the usual file level
>> operations/manipulations frequently used by engineers when implementing
>> Update/Delete/Upsert behaviour themselves, hopefully creating a conceptual
>> bridge to the more involved designs. Right now, Iceberg has almost
>> everything to implement the "eager" approach as we simply need to adjust
>> the retry mechanism. For example, I have implemented a prototype of the
>> eager solution with Spark and Iceberg.
>>
>> We looked into many existing solutions for inspiration, but when there
>> isn't a paper or code in the public domain it becomes hard to assess the
>> underlying design, although some of it can be inferred from the API or
>> documentation.
>>
>> Best,
>> Miguel
>>
>> On 10 May 2019, at 11:57, Anton Okolnychyi <ao...@apple.com> wrote:
>>
>> Thanks for the feedback, Jacques!
>>
>> You are correct, we kept the question of the best approach as open :) The
>> idea was to have a discussion in the community. Hopefully, we can reach a
>> consensus.
>>
>> While the proposed “lazy” approaches certainly offer significant
>> benefits, they require more changes in Iceberg as well as in readers/query
>> engines (depending on how we want to merge base and diff files). For us, it
>> is important to understand whether the Iceberg community would even
>> consider such changes.
>>
>> Hive ACID 3 is one the projects we looked at. In fact, we spoke to Owen,
>> the original creator of updates/deletes/upserts in Hive. I believe the
>> “lazy” approaches are close to what Hive 3 does but with their own
>> distinctions that Iceberg allows us to have. It would be great to have
>> Owen’s feedback.
>>
>> We don’t know the internals of Delta as updates/deletes/upserts are not
>> open source. My personal guess, yes, it might be similar to the “eager”
>> approach in our doc.
>>
>> Jacques, could you share some insights how you implement the merge of
>> diffs? Is it done by readers?
>>
>> Thanks,
>> Anton
>>
>> On 10 May 2019, at 06:24, Jacques Nadeau <ja...@dremio.com> wrote:
>>
>> This is a nice doc and it covers many different options. Upon first skim,
>> I don't see a strong argument for particular approach. D
>>
>> In our own development, we've been leaning heavily towards what you
>> describe in the document as "lazy with SRI". I believe this is consistent
>> with what the Hive community did on top of Orc. It's interesting because my
>> (maybe incorrect) understanding of the Databricks Delta approach is they
>> chose what you title "eager" in their approach to upserts. They may also
>> have a lazy approach for other types of mutations but I don't think they do.
>>
>> Thanks again for putting this together!
>> Jacques
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>>
>>
>> On Wed, May 8, 2019 at 3:42 AM Anton Okolnychyi <
>> aokolnychyi@apple.com.invalid> wrote:
>>
>>> Hi folks,
>>>
>>> Miguel (cc) and I have spent some time thinking about how to perform
>>> updates/deletes/upserts on top of Iceberg tables. This functionality is
>>> essential for many modern use cases. We've summarized our ideas in a doc
>>> [1], which, hopefully, will trigger a discussion in the community. The
>>> document presents different conceptual approaches alongside their
>>> trade-offs. We will be glad to consider any other ideas as well.
>>>
>>> Thanks,
>>> Anton
>>>
>>> [1] -
>>> https://docs.google.com/document/d/1Pk34C3diOfVCRc-sfxfhXZfzvxwum1Odo-6Jj9mwK38/
>>>
>>>
>>>
>>
>>
>

-- 
Ryan Blue
Software Engineer
Netflix

Re: Updates/Deletes/Upserts in Iceberg

[Anton Okolnychyi <ao...@apple.com.INVALID>]

We did take a look at Hudi. The overall design seems to be pretty complicated and, unfortunately, I didn’t have time to explore every detail.

Here is my understanding (correct me if I am wrong):

- Hudi has RECORD_KEY, which is expected to be unique.
- Hudi has PRECOMBINED_KEY, which is used to pick only one row in the incoming batch if there are multiple rows with the same key. As I understand, this isn't used on reads. It is used on writes to deduplicate rows with identical keys within one incoming batch. For example, if we are inserting 10 records and two rows have the same key, PRECOMBINED_KEY will be used to pick up only one row.
- Once Hudi ensures the uniqueness of RECORD_KEY within the incoming batch, it loads the Bloom filter index from all existing Parquet files in the involved partitions (meaning, partitions spread from the input batch) and tags each record as either an update or insert by mapping the incoming keys to existing files for updates. At this point, it seems to rely on join.

Is my understanding correct? If so, do we want to consider joins on write? We mentioned this technique as one way to ensure the uniqueness of natural keys but we were concerned about the performance. Also, does Hudi support record-level updates? 

Thanks,
Anton

> On 10 May 2019, at 18:22, Erik Wright <er...@shopify.com.INVALID> wrote:
> 
> Thanks for putting this forward.
> 
> Another term for the "lazy" approach would be "merge on read".
> 
> My team has built something internally that uses merge-on-read internally but uses an "Eager" materialization for publication to Presto. Roughly, we maintain a table metadata file that looks a bit like Iceberg's and tracks the "live" version of each partition as it is updated over time. We are looking into a solution that will allow us to push the merge-on-read all the way to Presto (and other consumers), and adding Merge-On-Read to Iceberg is one of the approaches we are considering.
> 
> It's worth noting that Hudi does have support for upserts/deletes as well, so that's another model to consider.
> 
> On Fri, May 10, 2019 at 8:30 AM Miguel Miranda <mi...@apple.com.invalid> wrote:
> Hi,
> 
> As Anton said, we purposely avoided making a "decision" on which approach should be implemented in order to allow for a meaningful discussion with the community.
> 
> The document starts with an eager approach as it is straightforward and easy to understand: steps resemble the usual file level operations/manipulations frequently used by engineers when implementing Update/Delete/Upsert behaviour themselves, hopefully creating a conceptual bridge to the more involved designs. Right now, Iceberg has almost everything to implement the "eager" approach as we simply need to adjust the retry mechanism. For example, I have implemented a prototype of the eager solution with Spark and Iceberg.
> 
> We looked into many existing solutions for inspiration, but when there isn't a paper or code in the public domain it becomes hard to assess the underlying design, although some of it can be inferred from the API or documentation.
> 
> Best,
> Miguel
> 
>> On 10 May 2019, at 11:57, Anton Okolnychyi <aokolnychyi@apple.com <ma...@apple.com>> wrote:
>> 
>> Thanks for the feedback, Jacques!
>> 
>> You are correct, we kept the question of the best approach as open :) The idea was to have a discussion in the community. Hopefully, we can reach a consensus.
>> 
>> While the proposed “lazy” approaches certainly offer significant benefits, they require more changes in Iceberg as well as in readers/query engines (depending on how we want to merge base and diff files). For us, it is important to understand whether the Iceberg community would even consider such changes. 
>> 
>> Hive ACID 3 is one the projects we looked at. In fact, we spoke to Owen, the original creator of updates/deletes/upserts in Hive. I believe the “lazy” approaches are close to what Hive 3 does but with their own distinctions that Iceberg allows us to have. It would be great to have Owen’s feedback.
>> 
>> We don’t know the internals of Delta as updates/deletes/upserts are not open source. My personal guess, yes, it might be similar to the “eager” approach in our doc.
>> 
>> Jacques, could you share some insights how you implement the merge of diffs? Is it done by readers?
>> 
>> Thanks,
>> Anton
>> 
>>> On 10 May 2019, at 06:24, Jacques Nadeau <jacques@dremio.com <ma...@dremio.com>> wrote:
>>> 
>>> This is a nice doc and it covers many different options. Upon first skim, I don't see a strong argument for particular approach. D
>>> 
>>> In our own development, we've been leaning heavily towards what you describe in the document as "lazy with SRI". I believe this is consistent with what the Hive community did on top of Orc. It's interesting because my (maybe incorrect) understanding of the Databricks Delta approach is they chose what you title "eager" in their approach to upserts. They may also have a lazy approach for other types of mutations but I don't think they do.
>>> 
>>> Thanks again for putting this together!
>>> Jacques
>>> --
>>> Jacques Nadeau
>>> CTO and Co-Founder, Dremio
>>> 
>>> 
>>> On Wed, May 8, 2019 at 3:42 AM Anton Okolnychyi <aokolnychyi@apple.com.invalid <ma...@apple.com.invalid>> wrote:
>>> Hi folks,
>>> 
>>> Miguel (cc) and I have spent some time thinking about how to perform updates/deletes/upserts on top of Iceberg tables. This functionality is essential for many modern use cases. We've summarized our ideas in a doc [1], which, hopefully, will trigger a discussion in the community. The document presents different conceptual approaches alongside their trade-offs. We will be glad to consider any other ideas as well.
>>> 
>>> Thanks,
>>> Anton
>>> 
>>> [1] - https://docs.google.com/document/d/1Pk34C3diOfVCRc-sfxfhXZfzvxwum1Odo-6Jj9mwK38/ <https://docs.google.com/document/d/1Pk34C3diOfVCRc-sfxfhXZfzvxwum1Odo-6Jj9mwK38/>
>>> 
>>> 
>> 
> 

Re: Updates/Deletes/Upserts in Iceberg

[Erik Wright <er...@shopify.com.INVALID>]

Thanks for putting this forward.

Another term for the "lazy" approach would be "merge on read".

My team has built something internally that uses merge-on-read internally
but uses an "Eager" materialization for publication to Presto. Roughly, we
maintain a table metadata file that looks a bit like Iceberg's and tracks
the "live" version of each partition as it is updated over time. We are
looking into a solution that will allow us to push the merge-on-read all
the way to Presto (and other consumers), and adding Merge-On-Read to
Iceberg is one of the approaches we are considering.

It's worth noting that Hudi does have support for upserts/deletes as well,
so that's another model to consider.

On Fri, May 10, 2019 at 8:30 AM Miguel Miranda
<mi...@apple.com.invalid> wrote:

> Hi,
>
> As Anton said, we purposely avoided making a "decision" on which approach
> should be implemented in order to allow for a meaningful discussion with
> the community.
>
> The document starts with an eager approach as it is straightforward and
> easy to understand: steps resemble the usual file level
> operations/manipulations frequently used by engineers when implementing
> Update/Delete/Upsert behaviour themselves, hopefully creating a conceptual
> bridge to the more involved designs. Right now, Iceberg has almost
> everything to implement the "eager" approach as we simply need to adjust
> the retry mechanism. For example, I have implemented a prototype of the
> eager solution with Spark and Iceberg.
>
> We looked into many existing solutions for inspiration, but when there
> isn't a paper or code in the public domain it becomes hard to assess the
> underlying design, although some of it can be inferred from the API or
> documentation.
>
> Best,
> Miguel
>
> On 10 May 2019, at 11:57, Anton Okolnychyi <ao...@apple.com> wrote:
>
> Thanks for the feedback, Jacques!
>
> You are correct, we kept the question of the best approach as open :) The
> idea was to have a discussion in the community. Hopefully, we can reach a
> consensus.
>
> While the proposed “lazy” approaches certainly offer significant benefits,
> they require more changes in Iceberg as well as in readers/query engines
> (depending on how we want to merge base and diff files). For us, it is
> important to understand whether the Iceberg community would even consider
> such changes.
>
> Hive ACID 3 is one the projects we looked at. In fact, we spoke to Owen,
> the original creator of updates/deletes/upserts in Hive. I believe the
> “lazy” approaches are close to what Hive 3 does but with their own
> distinctions that Iceberg allows us to have. It would be great to have
> Owen’s feedback.
>
> We don’t know the internals of Delta as updates/deletes/upserts are not
> open source. My personal guess, yes, it might be similar to the “eager”
> approach in our doc.
>
> Jacques, could you share some insights how you implement the merge of
> diffs? Is it done by readers?
>
> Thanks,
> Anton
>
> On 10 May 2019, at 06:24, Jacques Nadeau <ja...@dremio.com> wrote:
>
> This is a nice doc and it covers many different options. Upon first skim,
> I don't see a strong argument for particular approach. D
>
> In our own development, we've been leaning heavily towards what you
> describe in the document as "lazy with SRI". I believe this is consistent
> with what the Hive community did on top of Orc. It's interesting because my
> (maybe incorrect) understanding of the Databricks Delta approach is they
> chose what you title "eager" in their approach to upserts. They may also
> have a lazy approach for other types of mutations but I don't think they do.
>
> Thanks again for putting this together!
> Jacques
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
>
> On Wed, May 8, 2019 at 3:42 AM Anton Okolnychyi <
> aokolnychyi@apple.com.invalid> wrote:
>
>> Hi folks,
>>
>> Miguel (cc) and I have spent some time thinking about how to perform
>> updates/deletes/upserts on top of Iceberg tables. This functionality is
>> essential for many modern use cases. We've summarized our ideas in a doc
>> [1], which, hopefully, will trigger a discussion in the community. The
>> document presents different conceptual approaches alongside their
>> trade-offs. We will be glad to consider any other ideas as well.
>>
>> Thanks,
>> Anton
>>
>> [1] -
>> https://docs.google.com/document/d/1Pk34C3diOfVCRc-sfxfhXZfzvxwum1Odo-6Jj9mwK38/
>>
>>
>>
>
>

Re: Updates/Deletes/Upserts in Iceberg

[Miguel Miranda <mi...@apple.com.INVALID>]

Hi,

As Anton said, we purposely avoided making a "decision" on which approach should be implemented in order to allow for a meaningful discussion with the community.

The document starts with an eager approach as it is straightforward and easy to understand: steps resemble the usual file level operations/manipulations frequently used by engineers when implementing Update/Delete/Upsert behaviour themselves, hopefully creating a conceptual bridge to the more involved designs. Right now, Iceberg has almost everything to implement the "eager" approach as we simply need to adjust the retry mechanism. For example, I have implemented a prototype of the eager solution with Spark and Iceberg.

We looked into many existing solutions for inspiration, but when there isn't a paper or code in the public domain it becomes hard to assess the underlying design, although some of it can be inferred from the API or documentation.

Best,
Miguel

> On 10 May 2019, at 11:57, Anton Okolnychyi <ao...@apple.com> wrote:
> 
> Thanks for the feedback, Jacques!
> 
> You are correct, we kept the question of the best approach as open :) The idea was to have a discussion in the community. Hopefully, we can reach a consensus.
> 
> While the proposed “lazy” approaches certainly offer significant benefits, they require more changes in Iceberg as well as in readers/query engines (depending on how we want to merge base and diff files). For us, it is important to understand whether the Iceberg community would even consider such changes. 
> 
> Hive ACID 3 is one the projects we looked at. In fact, we spoke to Owen, the original creator of updates/deletes/upserts in Hive. I believe the “lazy” approaches are close to what Hive 3 does but with their own distinctions that Iceberg allows us to have. It would be great to have Owen’s feedback.
> 
> We don’t know the internals of Delta as updates/deletes/upserts are not open source. My personal guess, yes, it might be similar to the “eager” approach in our doc.
> 
> Jacques, could you share some insights how you implement the merge of diffs? Is it done by readers?
> 
> Thanks,
> Anton
> 
>> On 10 May 2019, at 06:24, Jacques Nadeau <jacques@dremio.com <ma...@dremio.com>> wrote:
>> 
>> This is a nice doc and it covers many different options. Upon first skim, I don't see a strong argument for particular approach. D
>> 
>> In our own development, we've been leaning heavily towards what you describe in the document as "lazy with SRI". I believe this is consistent with what the Hive community did on top of Orc. It's interesting because my (maybe incorrect) understanding of the Databricks Delta approach is they chose what you title "eager" in their approach to upserts. They may also have a lazy approach for other types of mutations but I don't think they do.
>> 
>> Thanks again for putting this together!
>> Jacques
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>> 
>> 
>> On Wed, May 8, 2019 at 3:42 AM Anton Okolnychyi <aokolnychyi@apple.com.invalid <ma...@apple.com.invalid>> wrote:
>> Hi folks,
>> 
>> Miguel (cc) and I have spent some time thinking about how to perform updates/deletes/upserts on top of Iceberg tables. This functionality is essential for many modern use cases. We've summarized our ideas in a doc [1], which, hopefully, will trigger a discussion in the community. The document presents different conceptual approaches alongside their trade-offs. We will be glad to consider any other ideas as well.
>> 
>> Thanks,
>> Anton
>> 
>> [1] - https://docs.google.com/document/d/1Pk34C3diOfVCRc-sfxfhXZfzvxwum1Odo-6Jj9mwK38/ <https://docs.google.com/document/d/1Pk34C3diOfVCRc-sfxfhXZfzvxwum1Odo-6Jj9mwK38/>
>> 
>> 
> 

Re: Updates/Deletes/Upserts in Iceberg

[Anton Okolnychyi <ao...@apple.com.INVALID>]

Thanks for the feedback, Jacques!

You are correct, we kept the question of the best approach as open :) The idea was to have a discussion in the community. Hopefully, we can reach a consensus.

While the proposed “lazy” approaches certainly offer significant benefits, they require more changes in Iceberg as well as in readers/query engines (depending on how we want to merge base and diff files). For us, it is important to understand whether the Iceberg community would even consider such changes. 

Hive ACID 3 is one the projects we looked at. In fact, we spoke to Owen, the original creator of updates/deletes/upserts in Hive. I believe the “lazy” approaches are close to what Hive 3 does but with their own distinctions that Iceberg allows us to have. It would be great to have Owen’s feedback.

We don’t know the internals of Delta as updates/deletes/upserts are not open source. My personal guess, yes, it might be similar to the “eager” approach in our doc.

Jacques, could you share some insights how you implement the merge of diffs? Is it done by readers?

Thanks,
Anton

> On 10 May 2019, at 06:24, Jacques Nadeau <ja...@dremio.com> wrote:
> 
> This is a nice doc and it covers many different options. Upon first skim, I don't see a strong argument for particular approach. D
> 
> In our own development, we've been leaning heavily towards what you describe in the document as "lazy with SRI". I believe this is consistent with what the Hive community did on top of Orc. It's interesting because my (maybe incorrect) understanding of the Databricks Delta approach is they chose what you title "eager" in their approach to upserts. They may also have a lazy approach for other types of mutations but I don't think they do.
> 
> Thanks again for putting this together!
> Jacques
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
> 
> 
> On Wed, May 8, 2019 at 3:42 AM Anton Okolnychyi <ao...@apple.com.invalid> wrote:
> Hi folks,
> 
> Miguel (cc) and I have spent some time thinking about how to perform updates/deletes/upserts on top of Iceberg tables. This functionality is essential for many modern use cases. We've summarized our ideas in a doc [1], which, hopefully, will trigger a discussion in the community. The document presents different conceptual approaches alongside their trade-offs. We will be glad to consider any other ideas as well.
> 
> Thanks,
> Anton
> 
> [1] - https://docs.google.com/document/d/1Pk34C3diOfVCRc-sfxfhXZfzvxwum1Odo-6Jj9mwK38/ <https://docs.google.com/document/d/1Pk34C3diOfVCRc-sfxfhXZfzvxwum1Odo-6Jj9mwK38/>
> 
> 

Re: Updates/Deletes/Upserts in Iceberg

[Jacques Nadeau <ja...@dremio.com>]

This is a nice doc and it covers many different options. Upon first skim, I
don't see a strong argument for particular approach. D

In our own development, we've been leaning heavily towards what you
describe in the document as "lazy with SRI". I believe this is consistent
with what the Hive community did on top of Orc. It's interesting because my
(maybe incorrect) understanding of the Databricks Delta approach is they
chose what you title "eager" in their approach to upserts. They may also
have a lazy approach for other types of mutations but I don't think they do.

Thanks again for putting this together!
Jacques
--
Jacques Nadeau
CTO and Co-Founder, Dremio


On Wed, May 8, 2019 at 3:42 AM Anton Okolnychyi
<ao...@apple.com.invalid> wrote:

> Hi folks,
>
> Miguel (cc) and I have spent some time thinking about how to perform
> updates/deletes/upserts on top of Iceberg tables. This functionality is
> essential for many modern use cases. We've summarized our ideas in a doc
> [1], which, hopefully, will trigger a discussion in the community. The
> document presents different conceptual approaches alongside their
> trade-offs. We will be glad to consider any other ideas as well.
>
> Thanks,
> Anton
>
> [1] -
> https://docs.google.com/document/d/1Pk34C3diOfVCRc-sfxfhXZfzvxwum1Odo-6Jj9mwK38/
>
>
>

Re: Updates/Deletes/Upserts in Iceberg

[Jacques Nadeau <ja...@dremio.com>]

Awesome. This was on my list for some time. Glad you got it started.

On Wed, May 8, 2019, 3:42 AM Anton Okolnychyi <ao...@apple.com.invalid>
wrote:

> Hi folks,
>
> Miguel (cc) and I have spent some time thinking about how to perform
> updates/deletes/upserts on top of Iceberg tables. This functionality is
> essential for many modern use cases. We've summarized our ideas in a doc
> [1], which, hopefully, will trigger a discussion in the community. The
> document presents different conceptual approaches alongside their
> trade-offs. We will be glad to consider any other ideas as well.
>
> Thanks,
> Anton
>
> [1] -
> https://docs.google.com/document/d/1Pk34C3diOfVCRc-sfxfhXZfzvxwum1Odo-6Jj9mwK38/
>
>
>