[Specification] Uniqueness of Data file names and cardinality in a snapshot?

[Micah Kornfield <em...@gmail.com>]

Hi Iceberg Dev,
I tried searching for it in the specification but couldn't find anything
explicit:

1.  Is it assumed that all data files and delete files will always have
globally unique names in a table?
2.  Is it expected that the pairwise intersection of all manifest files in
a snapshot is empty (i.e. For any given data file it has exactly zero or 1
entries across all manifest files in a snapshot)?

I think the uniqueness of both can maybe be inferred by this sentence (but
I'm not 100% sure):

> When a file is replaced or deleted from the dataset, it’s manifest entry
> fields store the snapshot ID in which the file was deleted and status 2
> (deleted). The file may be deleted from the file system when the snapshot
> in which it was deleted is garbage collected, assuming that older snapshots
> have also been garbage collected [1].


Thanks,
Micah


P.S. Is there a way to add a table of contents to the specification.  I
might be missing it but I don't see one rendered at:
https://iceberg.apache.org/spec/

Re: [Specification] Uniqueness of Data file names and cardinality in a snapshot?

[Daniel Weeks <da...@gmail.com>]

Yeah, I follow what you're saying now and it's an excellent observation.  I
agree that a clarification would help because there are a number of places
where the behavior is somewhat implied, but not clearly stated.

I can't think of a case where you would want to have the same file either
duplicated in a manifest or across manifests with the same status (and I
can think of some problematic situations that can arise from that).

The cases where we have seen duplication have typically been due to
implementation issues that result in duplicates, which is why there are
some checks in place.

I'll follow up on this,

Thanks Micah!



On Fri, Mar 4, 2022 at 10:28 PM Micah Kornfield <em...@gmail.com>
wrote:

> Hi Dan,
>>
>> The paths are not unique across snapshots 1/2, but within each snapshot
>> they are.
>
>
> This is exactly the contract I was trying to determine.  It seems like it
> could potentially be clarified in the specification (again apologies if I
> missed it). I opened a PR to try to add language around these requirements,
> hopefully this is a helpful contribution.
>
> Thanks,
> Micah
>
> [1] https://github.com/apache/iceberg/pull/4272
>
> On Fri, Mar 4, 2022 at 4:41 PM Daniel Weeks <da...@gmail.com>
> wrote:
>
>> I think in the situation you're demonstrating, the manifests are
>> separated across two separate snapshots.
>>
>> Here's an example:
>>
>> create table t1 (s string);
>> insert into t1 values ('foo');  -- snapshot 0, manifest-list with 1
>> manifest pointing to file A (ADDED)
>> insert into t1 values ('bar'); -- snapshot 1, manifest-list with 2
>> manifests pointing to file A (ADDED),  file B (ADDED)
>> delete from t1 where s = 'foo'; -- snapshot 2, manifest-list with 2
>> manifests pointing to file A (DELETED),  file B (ADDED)
>>
>> The paths are not unique across snapshots 1/2, but within each snapshot
>> they are.
>>
>> Now in the same case if the data was in the same file, you would have a
>> rewrite of the datafile like this (assuming no row-level deletes):
>>
>> create table t1 (s string);
>> insert into t1 values ('foo'), ('bar');  -- snapshot 0, manifest-list
>> with 1 manifest pointing to file A (ADDED)
>> delete from t1 where s = 'foo'; -- snapshot 1, manifest-list with 1
>> manifests pointing to file A (DELETED) + file B (ADDED)
>>
>> I hope I'm understanding your example correctly, but let me know if I'm
>> off track here.
>>
>> Thanks,
>> Dan
>>
>>
>>
>> On Fri, Mar 4, 2022 at 2:23 PM Micah Kornfield <em...@gmail.com>
>> wrote:
>>
>>> Hi Dan,
>>> Thanks for the quick reply.
>>>
>>>
>>>> For #2, the answer follows mostly because if the answer to #1 holds,
>>>> then yes the pairwise intersection of entries in the manifest files of a
>>>> given snapshot is empty.
>>>
>>>
>>> Just to be pedantic, even with unique file names.  It seems one could
>>> construct a snapshots as:
>>> Manifest 1: Add File A
>>> Manifest 2: Delete File A
>>>
>>> From your answer it sounds like this is unexpected and readers generally
>>> don't try to reconcile Deletes add Adds?
>>>
>>> Thanks,
>>> Micah
>>>
>>> On Fri, Mar 4, 2022 at 2:10 PM Daniel Weeks <dw...@apache.org> wrote:
>>>
>>>> Hey Micah,
>>>>
>>>> For #1, I don't believe spec clearly calls out that all data/delete
>>>> files must be unique, but the requirements for cleanup would be violated in
>>>> certain cases if you had the same file referenced in multiple manifests.
>>>> In practice, the best way to ensure data correctness and metadata
>>>> consistency is to ensure that all referenced files have unique locations
>>>> and that those locations do not get overwritten.
>>>>
>>>> For #2, the answer follows mostly because if the answer to #1 holds,
>>>> then yes the pairwise intersection of entries in the manifest files of a
>>>> given snapshot is empty.
>>>>
>>>> The java library does perform some checks to prevent a file from being
>>>> added to the same manifest multiple times, but I don't think that
>>>> extends to all possible ways of adding files.  So it may be possible, but
>>>> not a good idea.
>>>>
>>>> Sam might know if there's a way to add a nav for the format page (it is
>>>> a little difficult to navigate at the moment).
>>>>
>>>> -Dan
>>>>
>>>> On Thu, Mar 3, 2022 at 4:49 PM Micah Kornfield <em...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Iceberg Dev,
>>>>> I tried searching for it in the specification but couldn't find
>>>>> anything explicit:
>>>>>
>>>>> 1.  Is it assumed that all data files and delete files will always
>>>>> have globally unique names in a table?
>>>>> 2.  Is it expected that the pairwise intersection of all manifest
>>>>> files in a snapshot is empty (i.e. For any given data file it has exactly
>>>>> zero or 1 entries across all manifest files in a snapshot)?
>>>>>
>>>>> I think the uniqueness of both can maybe be inferred by this sentence
>>>>> (but I'm not 100% sure):
>>>>>
>>>>>> When a file is replaced or deleted from the dataset, it’s manifest
>>>>>> entry fields store the snapshot ID in which the file was deleted and status
>>>>>> 2 (deleted). The file may be deleted from the file system when the snapshot
>>>>>> in which it was deleted is garbage collected, assuming that older snapshots
>>>>>> have also been garbage collected [1].
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Micah
>>>>>
>>>>>
>>>>> P.S. Is there a way to add a table of contents to the specification.
>>>>> I might be missing it but I don't see one rendered at:
>>>>> https://iceberg.apache.org/spec/
>>>>>
>>>>

Re: [Specification] Uniqueness of Data file names and cardinality in a snapshot?

[Micah Kornfield <em...@gmail.com>]

Hi Dan,
>
> The paths are not unique across snapshots 1/2, but within each snapshot
> they are.


This is exactly the contract I was trying to determine.  It seems like it
could potentially be clarified in the specification (again apologies if I
missed it). I opened a PR to try to add language around these requirements,
hopefully this is a helpful contribution.

Thanks,
Micah

[1] https://github.com/apache/iceberg/pull/4272

On Fri, Mar 4, 2022 at 4:41 PM Daniel Weeks <da...@gmail.com>
wrote:

> I think in the situation you're demonstrating, the manifests are separated
> across two separate snapshots.
>
> Here's an example:
>
> create table t1 (s string);
> insert into t1 values ('foo');  -- snapshot 0, manifest-list with 1
> manifest pointing to file A (ADDED)
> insert into t1 values ('bar'); -- snapshot 1, manifest-list with 2
> manifests pointing to file A (ADDED),  file B (ADDED)
> delete from t1 where s = 'foo'; -- snapshot 2, manifest-list with 2
> manifests pointing to file A (DELETED),  file B (ADDED)
>
> The paths are not unique across snapshots 1/2, but within each snapshot
> they are.
>
> Now in the same case if the data was in the same file, you would have a
> rewrite of the datafile like this (assuming no row-level deletes):
>
> create table t1 (s string);
> insert into t1 values ('foo'), ('bar');  -- snapshot 0, manifest-list with
> 1 manifest pointing to file A (ADDED)
> delete from t1 where s = 'foo'; -- snapshot 1, manifest-list with 1
> manifests pointing to file A (DELETED) + file B (ADDED)
>
> I hope I'm understanding your example correctly, but let me know if I'm
> off track here.
>
> Thanks,
> Dan
>
>
>
> On Fri, Mar 4, 2022 at 2:23 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
>> Hi Dan,
>> Thanks for the quick reply.
>>
>>
>>> For #2, the answer follows mostly because if the answer to #1 holds,
>>> then yes the pairwise intersection of entries in the manifest files of a
>>> given snapshot is empty.
>>
>>
>> Just to be pedantic, even with unique file names.  It seems one could
>> construct a snapshots as:
>> Manifest 1: Add File A
>> Manifest 2: Delete File A
>>
>> From your answer it sounds like this is unexpected and readers generally
>> don't try to reconcile Deletes add Adds?
>>
>> Thanks,
>> Micah
>>
>> On Fri, Mar 4, 2022 at 2:10 PM Daniel Weeks <dw...@apache.org> wrote:
>>
>>> Hey Micah,
>>>
>>> For #1, I don't believe spec clearly calls out that all data/delete
>>> files must be unique, but the requirements for cleanup would be violated in
>>> certain cases if you had the same file referenced in multiple manifests.
>>> In practice, the best way to ensure data correctness and metadata
>>> consistency is to ensure that all referenced files have unique locations
>>> and that those locations do not get overwritten.
>>>
>>> For #2, the answer follows mostly because if the answer to #1 holds,
>>> then yes the pairwise intersection of entries in the manifest files of a
>>> given snapshot is empty.
>>>
>>> The java library does perform some checks to prevent a file from being
>>> added to the same manifest multiple times, but I don't think that
>>> extends to all possible ways of adding files.  So it may be possible, but
>>> not a good idea.
>>>
>>> Sam might know if there's a way to add a nav for the format page (it is
>>> a little difficult to navigate at the moment).
>>>
>>> -Dan
>>>
>>> On Thu, Mar 3, 2022 at 4:49 PM Micah Kornfield <em...@gmail.com>
>>> wrote:
>>>
>>>> Hi Iceberg Dev,
>>>> I tried searching for it in the specification but couldn't find
>>>> anything explicit:
>>>>
>>>> 1.  Is it assumed that all data files and delete files will always have
>>>> globally unique names in a table?
>>>> 2.  Is it expected that the pairwise intersection of all manifest files
>>>> in a snapshot is empty (i.e. For any given data file it has exactly zero or
>>>> 1 entries across all manifest files in a snapshot)?
>>>>
>>>> I think the uniqueness of both can maybe be inferred by this sentence
>>>> (but I'm not 100% sure):
>>>>
>>>>> When a file is replaced or deleted from the dataset, it’s manifest
>>>>> entry fields store the snapshot ID in which the file was deleted and status
>>>>> 2 (deleted). The file may be deleted from the file system when the snapshot
>>>>> in which it was deleted is garbage collected, assuming that older snapshots
>>>>> have also been garbage collected [1].
>>>>
>>>>
>>>> Thanks,
>>>> Micah
>>>>
>>>>
>>>> P.S. Is there a way to add a table of contents to the specification.  I
>>>> might be missing it but I don't see one rendered at:
>>>> https://iceberg.apache.org/spec/
>>>>
>>>

Re: [Specification] Uniqueness of Data file names and cardinality in a snapshot?

[Daniel Weeks <da...@gmail.com>]

I think in the situation you're demonstrating, the manifests are separated
across two separate snapshots.

Here's an example:

create table t1 (s string);
insert into t1 values ('foo');  -- snapshot 0, manifest-list with 1
manifest pointing to file A (ADDED)
insert into t1 values ('bar'); -- snapshot 1, manifest-list with 2
manifests pointing to file A (ADDED),  file B (ADDED)
delete from t1 where s = 'foo'; -- snapshot 2, manifest-list with 2
manifests pointing to file A (DELETED),  file B (ADDED)

The paths are not unique across snapshots 1/2, but within each snapshot
they are.

Now in the same case if the data was in the same file, you would have a
rewrite of the datafile like this (assuming no row-level deletes):

create table t1 (s string);
insert into t1 values ('foo'), ('bar');  -- snapshot 0, manifest-list with
1 manifest pointing to file A (ADDED)
delete from t1 where s = 'foo'; -- snapshot 1, manifest-list with 1
manifests pointing to file A (DELETED) + file B (ADDED)

I hope I'm understanding your example correctly, but let me know if I'm off
track here.

Thanks,
Dan



On Fri, Mar 4, 2022 at 2:23 PM Micah Kornfield <em...@gmail.com>
wrote:

> Hi Dan,
> Thanks for the quick reply.
>
>
>> For #2, the answer follows mostly because if the answer to #1 holds, then
>> yes the pairwise intersection of entries in the manifest files of a given
>> snapshot is empty.
>
>
> Just to be pedantic, even with unique file names.  It seems one could
> construct a snapshots as:
> Manifest 1: Add File A
> Manifest 2: Delete File A
>
> From your answer it sounds like this is unexpected and readers generally
> don't try to reconcile Deletes add Adds?
>
> Thanks,
> Micah
>
> On Fri, Mar 4, 2022 at 2:10 PM Daniel Weeks <dw...@apache.org> wrote:
>
>> Hey Micah,
>>
>> For #1, I don't believe spec clearly calls out that all data/delete files
>> must be unique, but the requirements for cleanup would be violated in
>> certain cases if you had the same file referenced in multiple manifests.
>> In practice, the best way to ensure data correctness and metadata
>> consistency is to ensure that all referenced files have unique locations
>> and that those locations do not get overwritten.
>>
>> For #2, the answer follows mostly because if the answer to #1 holds, then
>> yes the pairwise intersection of entries in the manifest files of a given
>> snapshot is empty.
>>
>> The java library does perform some checks to prevent a file from being
>> added to the same manifest multiple times, but I don't think that
>> extends to all possible ways of adding files.  So it may be possible, but
>> not a good idea.
>>
>> Sam might know if there's a way to add a nav for the format page (it is a
>> little difficult to navigate at the moment).
>>
>> -Dan
>>
>> On Thu, Mar 3, 2022 at 4:49 PM Micah Kornfield <em...@gmail.com>
>> wrote:
>>
>>> Hi Iceberg Dev,
>>> I tried searching for it in the specification but couldn't find anything
>>> explicit:
>>>
>>> 1.  Is it assumed that all data files and delete files will always have
>>> globally unique names in a table?
>>> 2.  Is it expected that the pairwise intersection of all manifest files
>>> in a snapshot is empty (i.e. For any given data file it has exactly zero or
>>> 1 entries across all manifest files in a snapshot)?
>>>
>>> I think the uniqueness of both can maybe be inferred by this sentence
>>> (but I'm not 100% sure):
>>>
>>>> When a file is replaced or deleted from the dataset, it’s manifest
>>>> entry fields store the snapshot ID in which the file was deleted and status
>>>> 2 (deleted). The file may be deleted from the file system when the snapshot
>>>> in which it was deleted is garbage collected, assuming that older snapshots
>>>> have also been garbage collected [1].
>>>
>>>
>>> Thanks,
>>> Micah
>>>
>>>
>>> P.S. Is there a way to add a table of contents to the specification.  I
>>> might be missing it but I don't see one rendered at:
>>> https://iceberg.apache.org/spec/
>>>
>>

Re: [Specification] Uniqueness of Data file names and cardinality in a snapshot?

[Micah Kornfield <em...@gmail.com>]

Hi Dan,
Thanks for the quick reply.


> For #2, the answer follows mostly because if the answer to #1 holds, then
> yes the pairwise intersection of entries in the manifest files of a given
> snapshot is empty.


Just to be pedantic, even with unique file names.  It seems one could
construct a snapshots as:
Manifest 1: Add File A
Manifest 2: Delete File A

From your answer it sounds like this is unexpected and readers generally
don't try to reconcile Deletes add Adds?

Thanks,
Micah

On Fri, Mar 4, 2022 at 2:10 PM Daniel Weeks <dw...@apache.org> wrote:

> Hey Micah,
>
> For #1, I don't believe spec clearly calls out that all data/delete files
> must be unique, but the requirements for cleanup would be violated in
> certain cases if you had the same file referenced in multiple manifests.
> In practice, the best way to ensure data correctness and metadata
> consistency is to ensure that all referenced files have unique locations
> and that those locations do not get overwritten.
>
> For #2, the answer follows mostly because if the answer to #1 holds, then
> yes the pairwise intersection of entries in the manifest files of a given
> snapshot is empty.
>
> The java library does perform some checks to prevent a file from being
> added to the same manifest multiple times, but I don't think that
> extends to all possible ways of adding files.  So it may be possible, but
> not a good idea.
>
> Sam might know if there's a way to add a nav for the format page (it is a
> little difficult to navigate at the moment).
>
> -Dan
>
> On Thu, Mar 3, 2022 at 4:49 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
>> Hi Iceberg Dev,
>> I tried searching for it in the specification but couldn't find anything
>> explicit:
>>
>> 1.  Is it assumed that all data files and delete files will always have
>> globally unique names in a table?
>> 2.  Is it expected that the pairwise intersection of all manifest files
>> in a snapshot is empty (i.e. For any given data file it has exactly zero or
>> 1 entries across all manifest files in a snapshot)?
>>
>> I think the uniqueness of both can maybe be inferred by this sentence
>> (but I'm not 100% sure):
>>
>>> When a file is replaced or deleted from the dataset, it’s manifest entry
>>> fields store the snapshot ID in which the file was deleted and status 2
>>> (deleted). The file may be deleted from the file system when the snapshot
>>> in which it was deleted is garbage collected, assuming that older snapshots
>>> have also been garbage collected [1].
>>
>>
>> Thanks,
>> Micah
>>
>>
>> P.S. Is there a way to add a table of contents to the specification.  I
>> might be missing it but I don't see one rendered at:
>> https://iceberg.apache.org/spec/
>>
>

Re: [Specification] Uniqueness of Data file names and cardinality in a snapshot?

[Daniel Weeks <dw...@apache.org>]

Hey Micah,

For #1, I don't believe spec clearly calls out that all data/delete files
must be unique, but the requirements for cleanup would be violated in
certain cases if you had the same file referenced in multiple manifests.
In practice, the best way to ensure data correctness and metadata
consistency is to ensure that all referenced files have unique locations
and that those locations do not get overwritten.

For #2, the answer follows mostly because if the answer to #1 holds, then
yes the pairwise intersection of entries in the manifest files of a given
snapshot is empty.

The java library does perform some checks to prevent a file from being
added to the same manifest multiple times, but I don't think that
extends to all possible ways of adding files.  So it may be possible, but
not a good idea.

Sam might know if there's a way to add a nav for the format page (it is a
little difficult to navigate at the moment).

-Dan

On Thu, Mar 3, 2022 at 4:49 PM Micah Kornfield <em...@gmail.com>
wrote:

> Hi Iceberg Dev,
> I tried searching for it in the specification but couldn't find anything
> explicit:
>
> 1.  Is it assumed that all data files and delete files will always have
> globally unique names in a table?
> 2.  Is it expected that the pairwise intersection of all manifest files in
> a snapshot is empty (i.e. For any given data file it has exactly zero or 1
> entries across all manifest files in a snapshot)?
>
> I think the uniqueness of both can maybe be inferred by this sentence (but
> I'm not 100% sure):
>
>> When a file is replaced or deleted from the dataset, it’s manifest entry
>> fields store the snapshot ID in which the file was deleted and status 2
>> (deleted). The file may be deleted from the file system when the snapshot
>> in which it was deleted is garbage collected, assuming that older snapshots
>> have also been garbage collected [1].
>
>
> Thanks,
> Micah
>
>
> P.S. Is there a way to add a table of contents to the specification.  I
> might be missing it but I don't see one rendered at:
> https://iceberg.apache.org/spec/
>