You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Wing Yew Poon <wy...@cloudera.com.INVALID> on 2024/04/12 18:15:32 UTC
spec question on equality deletes
Hi,
I have some questions on the current Iceberg spec regarding equality
deletes:
https://iceberg.apache.org/spec/#equality-delete-files
The spec says that for "a table with the following data:
<https://iceberg.apache.org/spec/#__codelineno-1-1> 1: id | 2:
category | 3: name
<https://iceberg.apache.org/spec/#__codelineno-1-2>-------|-------------|---------
<https://iceberg.apache.org/spec/#__codelineno-1-3> 1 | marsupial
| Koala <https://iceberg.apache.org/spec/#__codelineno-1-4> 2 |
toy | Teddy
<https://iceberg.apache.org/spec/#__codelineno-1-5> 3 | NULL
| Grizzly <https://iceberg.apache.org/spec/#__codelineno-1-6> 4 |
NULL | Polar
The delete id = 3 could be written as either of the following equality
delete files:
<https://iceberg.apache.org/spec/#__codelineno-2-1>equality_ids=[1]
<https://iceberg.apache.org/spec/#__codelineno-2-2>
<https://iceberg.apache.org/spec/#__codelineno-2-3> 1: id
<https://iceberg.apache.org/spec/#__codelineno-2-4>-------
<https://iceberg.apache.org/spec/#__codelineno-2-5> 3
equality_ids=[1] <https://iceberg.apache.org/spec/#__codelineno-3-2>
<https://iceberg.apache.org/spec/#__codelineno-3-3> 1: id | 2:
category | 3: name
<https://iceberg.apache.org/spec/#__codelineno-3-4>-------|-------------|---------
<https://iceberg.apache.org/spec/#__codelineno-3-5> 3 | NULL
| Grizzly
"
1. Are the options either (a) write only the column(s) listed in
equality_ids or (b) write all the columns? i.e, no in between.
2. If we write all the columns, are only columns listed in equality_ids
considered? What happens if a non-equality_id column does not match? e.g.,
equality_ids=[1] <https://iceberg.apache.org/spec/#__codelineno-3-2>
<https://iceberg.apache.org/spec/#__codelineno-3-3> 1: id | 2: category |
3: name <https://iceberg.apache.org/spec/#__codelineno-3-4>-------|-------------|---------
<https://iceberg.apache.org/spec/#__codelineno-3-5> 3 | NULL | Polar
Is that (a) invalid, or does that (b) still result in deleting id = 3, or
(c) result in deleting no rows?
The spec says "Each row of the delete file produces one equality predicate
that matches any row where the delete columns are equal. Multiple columns
can be thought of as an AND of equality predicates." That could be
interpreted to mean (c).
Thanks,
Wing Yew
Re: spec question on equality deletes
Posted by Manu Zhang <ow...@gmail.com>.
+1 on defining it clearly in the spec. Note the “spec doc” is the spec
itself, which requires more accurate description than doc. We may also need
spec test to check whether compute engine conforms to spec, not the other
way around.
Yufei Gu <fl...@gmail.com>于2024年4月17日 周三01:08写道：
> For me, (b) is the right behavior, we may just be clearer in the spec doc,
> but open for suggestions in case I missed something.
>
> Yufei
>
>
> On Mon, Apr 15, 2024 at 11:02 PM Renjie Liu <li...@gmail.com>
> wrote:
>
>> Hi, Wing:
>>
>> I totally agree that we should clearly define the expected behavior in
>> spec. I lean towards a), e.g. the row should be completed ignored or
>> completed same as original row, intermediate state should be defined as
>> invalid.
>>
>> On Tue, Apr 16, 2024 at 8:40 AM Wing Yew Poon <wy...@cloudera.com.invalid>
>> wrote:
>>
>>> Hi Yufei,
>>> Thank you for your response.
>>> It sounds like on 2, your thinking is that (b) is the correct behavior.
>>> Indeed, I have tried it out with Spark and afaict, it does (b). However,
>>> that does not mean that it is the correct behavior. The spec should clearly
>>> define it.
>>> - Wing Yew
>>>
>>>
>>> On Mon, Apr 15, 2024 at 5:25 PM Yufei Gu <fl...@gmail.com> wrote:
>>>
>>>> Hi Wing Yew Poon,
>>>>
>>>> Here is my understanding, but not necessarily how an engine implements
>>>> it.
>>>> It should only consider the columns in equality_ids when we apply eq
>>>> deletes. Also the engine should ignore the unrelated columns.
>>>> It will still delete the row with id 3 in the following case you
>>>> described even if the name doesn't match.
>>>> equality_ids=[1] <https://iceberg.apache.org/spec/#__codelineno-3-2>
>>>> <https://iceberg.apache.org/spec/#__codelineno-3-3> 1: id | 2:
>>>> category | 3: name <https://iceberg.apache.org/spec/#__codelineno-3-4>-------|-------------|---------
>>>> <https://iceberg.apache.org/spec/#__codelineno-3-5> 3 | NULL | Polar
>>>>
>>>> To verify the behavior, we can check the test case
>>>> like TestSparkReaderDeletes::testReadEqualityDeleteRows.
>>>>
>>>> Yufei
>>>>
>>>>
>>>> On Fri, Apr 12, 2024 at 11:16 AM Wing Yew Poon
>>>> <wy...@cloudera.com.invalid> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have some questions on the current Iceberg spec regarding equality
>>>>> deletes:
>>>>> https://iceberg.apache.org/spec/#equality-delete-files
>>>>> The spec says that for "a table with the following data:
>>>>>
>>>>> <https://iceberg.apache.org/spec/#__codelineno-1-1> 1: id | 2: category | 3: name <https://iceberg.apache.org/spec/#__codelineno-1-2>-------|-------------|--------- <https://iceberg.apache.org/spec/#__codelineno-1-3> 1 | marsupial | Koala <https://iceberg.apache.org/spec/#__codelineno-1-4> 2 | toy | Teddy <https://iceberg.apache.org/spec/#__codelineno-1-5> 3 | NULL | Grizzly <https://iceberg.apache.org/spec/#__codelineno-1-6> 4 | NULL | Polar
>>>>>
>>>>> The delete id = 3 could be written as either of the following
>>>>> equality delete files:
>>>>>
>>>>> <https://iceberg.apache.org/spec/#__codelineno-2-1>equality_ids=[1] <https://iceberg.apache.org/spec/#__codelineno-2-2> <https://iceberg.apache.org/spec/#__codelineno-2-3> 1: id <https://iceberg.apache.org/spec/#__codelineno-2-4>------- <https://iceberg.apache.org/spec/#__codelineno-2-5> 3
>>>>>
>>>>> equality_ids=[1] <https://iceberg.apache.org/spec/#__codelineno-3-2> <https://iceberg.apache.org/spec/#__codelineno-3-3> 1: id | 2: category | 3: name <https://iceberg.apache.org/spec/#__codelineno-3-4>-------|-------------|--------- <https://iceberg.apache.org/spec/#__codelineno-3-5> 3 | NULL | Grizzly
>>>>>
>>>>> "
>>>>>
>>>>> 1. Are the options either (a) write only the column(s) listed in
>>>>> equality_ids or (b) write all the columns? i.e, no in between.
>>>>> 2. If we write all the columns, are only columns listed in
>>>>> equality_ids considered? What happens if a non-equality_id column does not
>>>>> match? e.g.,
>>>>>
>>>>
>>>>> equality_ids=[1] <https://iceberg.apache.org/spec/#__codelineno-3-2>
>>>>> <https://iceberg.apache.org/spec/#__codelineno-3-3> 1: id | 2:
>>>>> category | 3: name <https://iceberg.apache.org/spec/#__codelineno-3-4>-------|-------------|---------
>>>>> <https://iceberg.apache.org/spec/#__codelineno-3-5> 3 | NULL | Polar
>>>>>
>>>>> Is that (a) invalid, or does that (b) still result in deleting id = 3,
>>>>> or (c) result in deleting no rows?
>>>>>
>>>>> The spec says "Each row of the delete file produces one equality
>>>>> predicate that matches any row where the delete columns are equal. Multiple
>>>>> columns can be thought of as an AND of equality predicates." That
>>>>> could be interpreted to mean (c).
>>>>>
>>>>> Thanks,
>>>>> Wing Yew
>>>>>
>>>>>
Re: spec question on equality deletes
Posted by Yufei Gu <fl...@gmail.com>.
For me, (b) is the right behavior, we may just be clearer in the spec doc,
but open for suggestions in case I missed something.
Yufei
On Mon, Apr 15, 2024 at 11:02 PM Renjie Liu <li...@gmail.com> wrote:
> Hi, Wing:
>
> I totally agree that we should clearly define the expected behavior in
> spec. I lean towards a), e.g. the row should be completed ignored or
> completed same as original row, intermediate state should be defined as
> invalid.
>
> On Tue, Apr 16, 2024 at 8:40 AM Wing Yew Poon <wy...@cloudera.com.invalid>
> wrote:
>
>> Hi Yufei,
>> Thank you for your response.
>> It sounds like on 2, your thinking is that (b) is the correct behavior.
>> Indeed, I have tried it out with Spark and afaict, it does (b). However,
>> that does not mean that it is the correct behavior. The spec should clearly
>> define it.
>> - Wing Yew
>>
>>
>> On Mon, Apr 15, 2024 at 5:25 PM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> Hi Wing Yew Poon,
>>>
>>> Here is my understanding, but not necessarily how an engine implements
>>> it.
>>> It should only consider the columns in equality_ids when we apply eq
>>> deletes. Also the engine should ignore the unrelated columns.
>>> It will still delete the row with id 3 in the following case you
>>> described even if the name doesn't match.
>>> equality_ids=[1] <https://iceberg.apache.org/spec/#__codelineno-3-2>
>>> <https://iceberg.apache.org/spec/#__codelineno-3-3> 1: id | 2: category
>>> | 3: name <https://iceberg.apache.org/spec/#__codelineno-3-4>-------|-------------|---------
>>> <https://iceberg.apache.org/spec/#__codelineno-3-5> 3 | NULL | Polar
>>>
>>> To verify the behavior, we can check the test case
>>> like TestSparkReaderDeletes::testReadEqualityDeleteRows.
>>>
>>> Yufei
>>>
>>>
>>> On Fri, Apr 12, 2024 at 11:16 AM Wing Yew Poon
>>> <wy...@cloudera.com.invalid> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have some questions on the current Iceberg spec regarding equality
>>>> deletes:
>>>> https://iceberg.apache.org/spec/#equality-delete-files
>>>> The spec says that for "a table with the following data:
>>>>
>>>> <https://iceberg.apache.org/spec/#__codelineno-1-1> 1: id | 2: category | 3: name <https://iceberg.apache.org/spec/#__codelineno-1-2>-------|-------------|--------- <https://iceberg.apache.org/spec/#__codelineno-1-3> 1 | marsupial | Koala <https://iceberg.apache.org/spec/#__codelineno-1-4> 2 | toy | Teddy <https://iceberg.apache.org/spec/#__codelineno-1-5> 3 | NULL | Grizzly <https://iceberg.apache.org/spec/#__codelineno-1-6> 4 | NULL | Polar
>>>>
>>>> The delete id = 3 could be written as either of the following equality
>>>> delete files:
>>>>
>>>> <https://iceberg.apache.org/spec/#__codelineno-2-1>equality_ids=[1] <https://iceberg.apache.org/spec/#__codelineno-2-2> <https://iceberg.apache.org/spec/#__codelineno-2-3> 1: id <https://iceberg.apache.org/spec/#__codelineno-2-4>------- <https://iceberg.apache.org/spec/#__codelineno-2-5> 3
>>>>
>>>> equality_ids=[1] <https://iceberg.apache.org/spec/#__codelineno-3-2> <https://iceberg.apache.org/spec/#__codelineno-3-3> 1: id | 2: category | 3: name <https://iceberg.apache.org/spec/#__codelineno-3-4>-------|-------------|--------- <https://iceberg.apache.org/spec/#__codelineno-3-5> 3 | NULL | Grizzly
>>>>
>>>> "
>>>>
>>>> 1. Are the options either (a) write only the column(s) listed in
>>>> equality_ids or (b) write all the columns? i.e, no in between.
>>>> 2. If we write all the columns, are only columns listed in equality_ids
>>>> considered? What happens if a non-equality_id column does not match? e.g.,
>>>>
>>>
>>>> equality_ids=[1] <https://iceberg.apache.org/spec/#__codelineno-3-2>
>>>> <https://iceberg.apache.org/spec/#__codelineno-3-3> 1: id | 2:
>>>> category | 3: name <https://iceberg.apache.org/spec/#__codelineno-3-4>-------|-------------|---------
>>>> <https://iceberg.apache.org/spec/#__codelineno-3-5> 3 | NULL | Polar
>>>>
>>>> Is that (a) invalid, or does that (b) still result in deleting id = 3,
>>>> or (c) result in deleting no rows?
>>>>
>>>> The spec says "Each row of the delete file produces one equality
>>>> predicate that matches any row where the delete columns are equal. Multiple
>>>> columns can be thought of as an AND of equality predicates." That
>>>> could be interpreted to mean (c).
>>>>
>>>> Thanks,
>>>> Wing Yew
>>>>
>>>>
Re: spec question on equality deletes
Posted by Renjie Liu <li...@gmail.com>.
Hi, Wing:
I totally agree that we should clearly define the expected behavior in
spec. I lean towards a), e.g. the row should be completed ignored or
completed same as original row, intermediate state should be defined as
invalid.
On Tue, Apr 16, 2024 at 8:40 AM Wing Yew Poon <wy...@cloudera.com.invalid>
wrote:
> Hi Yufei,
> Thank you for your response.
> It sounds like on 2, your thinking is that (b) is the correct behavior.
> Indeed, I have tried it out with Spark and afaict, it does (b). However,
> that does not mean that it is the correct behavior. The spec should clearly
> define it.
> - Wing Yew
>
>
> On Mon, Apr 15, 2024 at 5:25 PM Yufei Gu <fl...@gmail.com> wrote:
>
>> Hi Wing Yew Poon,
>>
>> Here is my understanding, but not necessarily how an engine implements it.
>> It should only consider the columns in equality_ids when we apply eq
>> deletes. Also the engine should ignore the unrelated columns.
>> It will still delete the row with id 3 in the following case you
>> described even if the name doesn't match.
>> equality_ids=[1] <https://iceberg.apache.org/spec/#__codelineno-3-2>
>> <https://iceberg.apache.org/spec/#__codelineno-3-3> 1: id | 2: category
>> | 3: name <https://iceberg.apache.org/spec/#__codelineno-3-4>-------|-------------|---------
>> <https://iceberg.apache.org/spec/#__codelineno-3-5> 3 | NULL | Polar
>>
>> To verify the behavior, we can check the test case
>> like TestSparkReaderDeletes::testReadEqualityDeleteRows.
>>
>> Yufei
>>
>>
>> On Fri, Apr 12, 2024 at 11:16 AM Wing Yew Poon
>> <wy...@cloudera.com.invalid> wrote:
>>
>>> Hi,
>>>
>>> I have some questions on the current Iceberg spec regarding equality
>>> deletes:
>>> https://iceberg.apache.org/spec/#equality-delete-files
>>> The spec says that for "a table with the following data:
>>>
>>> <https://iceberg.apache.org/spec/#__codelineno-1-1> 1: id | 2: category | 3: name <https://iceberg.apache.org/spec/#__codelineno-1-2>-------|-------------|--------- <https://iceberg.apache.org/spec/#__codelineno-1-3> 1 | marsupial | Koala <https://iceberg.apache.org/spec/#__codelineno-1-4> 2 | toy | Teddy <https://iceberg.apache.org/spec/#__codelineno-1-5> 3 | NULL | Grizzly <https://iceberg.apache.org/spec/#__codelineno-1-6> 4 | NULL | Polar
>>>
>>> The delete id = 3 could be written as either of the following equality
>>> delete files:
>>>
>>> <https://iceberg.apache.org/spec/#__codelineno-2-1>equality_ids=[1] <https://iceberg.apache.org/spec/#__codelineno-2-2> <https://iceberg.apache.org/spec/#__codelineno-2-3> 1: id <https://iceberg.apache.org/spec/#__codelineno-2-4>------- <https://iceberg.apache.org/spec/#__codelineno-2-5> 3
>>>
>>> equality_ids=[1] <https://iceberg.apache.org/spec/#__codelineno-3-2> <https://iceberg.apache.org/spec/#__codelineno-3-3> 1: id | 2: category | 3: name <https://iceberg.apache.org/spec/#__codelineno-3-4>-------|-------------|--------- <https://iceberg.apache.org/spec/#__codelineno-3-5> 3 | NULL | Grizzly
>>>
>>> "
>>>
>>> 1. Are the options either (a) write only the column(s) listed in
>>> equality_ids or (b) write all the columns? i.e, no in between.
>>> 2. If we write all the columns, are only columns listed in equality_ids
>>> considered? What happens if a non-equality_id column does not match? e.g.,
>>>
>>
>>> equality_ids=[1] <https://iceberg.apache.org/spec/#__codelineno-3-2>
>>> <https://iceberg.apache.org/spec/#__codelineno-3-3> 1: id | 2: category
>>> | 3: name <https://iceberg.apache.org/spec/#__codelineno-3-4>-------|-------------|---------
>>> <https://iceberg.apache.org/spec/#__codelineno-3-5> 3 | NULL | Polar
>>>
>>> Is that (a) invalid, or does that (b) still result in deleting id = 3,
>>> or (c) result in deleting no rows?
>>>
>>> The spec says "Each row of the delete file produces one equality
>>> predicate that matches any row where the delete columns are equal. Multiple
>>> columns can be thought of as an AND of equality predicates." That could
>>> be interpreted to mean (c).
>>>
>>> Thanks,
>>> Wing Yew
>>>
>>>
Re: spec question on equality deletes
Posted by Wing Yew Poon <wy...@cloudera.com.INVALID>.
Hi Yufei,
Thank you for your response.
It sounds like on 2, your thinking is that (b) is the correct behavior.
Indeed, I have tried it out with Spark and afaict, it does (b). However,
that does not mean that it is the correct behavior. The spec should clearly
define it.
- Wing Yew
On Mon, Apr 15, 2024 at 5:25 PM Yufei Gu <fl...@gmail.com> wrote:
> Hi Wing Yew Poon,
>
> Here is my understanding, but not necessarily how an engine implements it.
> It should only consider the columns in equality_ids when we apply eq
> deletes. Also the engine should ignore the unrelated columns.
> It will still delete the row with id 3 in the following case you described
> even if the name doesn't match.
> equality_ids=[1] <https://iceberg.apache.org/spec/#__codelineno-3-2>
> <https://iceberg.apache.org/spec/#__codelineno-3-3> 1: id | 2: category |
> 3: name <https://iceberg.apache.org/spec/#__codelineno-3-4>-------|-------------|---------
> <https://iceberg.apache.org/spec/#__codelineno-3-5> 3 | NULL | Polar
>
> To verify the behavior, we can check the test case
> like TestSparkReaderDeletes::testReadEqualityDeleteRows.
>
> Yufei
>
>
> On Fri, Apr 12, 2024 at 11:16 AM Wing Yew Poon <wy...@cloudera.com.invalid>
> wrote:
>
>> Hi,
>>
>> I have some questions on the current Iceberg spec regarding equality
>> deletes:
>> https://iceberg.apache.org/spec/#equality-delete-files
>> The spec says that for "a table with the following data:
>>
>> <https://iceberg.apache.org/spec/#__codelineno-1-1> 1: id | 2: category | 3: name <https://iceberg.apache.org/spec/#__codelineno-1-2>-------|-------------|--------- <https://iceberg.apache.org/spec/#__codelineno-1-3> 1 | marsupial | Koala <https://iceberg.apache.org/spec/#__codelineno-1-4> 2 | toy | Teddy <https://iceberg.apache.org/spec/#__codelineno-1-5> 3 | NULL | Grizzly <https://iceberg.apache.org/spec/#__codelineno-1-6> 4 | NULL | Polar
>>
>> The delete id = 3 could be written as either of the following equality
>> delete files:
>>
>> <https://iceberg.apache.org/spec/#__codelineno-2-1>equality_ids=[1] <https://iceberg.apache.org/spec/#__codelineno-2-2> <https://iceberg.apache.org/spec/#__codelineno-2-3> 1: id <https://iceberg.apache.org/spec/#__codelineno-2-4>------- <https://iceberg.apache.org/spec/#__codelineno-2-5> 3
>>
>> equality_ids=[1] <https://iceberg.apache.org/spec/#__codelineno-3-2> <https://iceberg.apache.org/spec/#__codelineno-3-3> 1: id | 2: category | 3: name <https://iceberg.apache.org/spec/#__codelineno-3-4>-------|-------------|--------- <https://iceberg.apache.org/spec/#__codelineno-3-5> 3 | NULL | Grizzly
>>
>> "
>>
>> 1. Are the options either (a) write only the column(s) listed in
>> equality_ids or (b) write all the columns? i.e, no in between.
>> 2. If we write all the columns, are only columns listed in equality_ids
>> considered? What happens if a non-equality_id column does not match? e.g.,
>>
>
>> equality_ids=[1] <https://iceberg.apache.org/spec/#__codelineno-3-2>
>> <https://iceberg.apache.org/spec/#__codelineno-3-3> 1: id | 2: category
>> | 3: name <https://iceberg.apache.org/spec/#__codelineno-3-4>-------|-------------|---------
>> <https://iceberg.apache.org/spec/#__codelineno-3-5> 3 | NULL | Polar
>>
>> Is that (a) invalid, or does that (b) still result in deleting id = 3,
>> or (c) result in deleting no rows?
>>
>> The spec says "Each row of the delete file produces one equality
>> predicate that matches any row where the delete columns are equal. Multiple
>> columns can be thought of as an AND of equality predicates." That could
>> be interpreted to mean (c).
>>
>> Thanks,
>> Wing Yew
>>
>>
Re: spec question on equality deletes
Posted by Yufei Gu <fl...@gmail.com>.
Hi Wing Yew Poon,
Here is my understanding, but not necessarily how an engine implements it.
It should only consider the columns in equality_ids when we apply eq
deletes. Also the engine should ignore the unrelated columns.
It will still delete the row with id 3 in the following case you described
even if the name doesn't match.
equality_ids=[1] <https://iceberg.apache.org/spec/#__codelineno-3-2>
<https://iceberg.apache.org/spec/#__codelineno-3-3> 1: id | 2: category |
3: name <https://iceberg.apache.org/spec/#__codelineno-3-4>-------|-------------|---------
<https://iceberg.apache.org/spec/#__codelineno-3-5> 3 | NULL | Polar
To verify the behavior, we can check the test case
like TestSparkReaderDeletes::testReadEqualityDeleteRows.
Yufei
On Fri, Apr 12, 2024 at 11:16 AM Wing Yew Poon <wy...@cloudera.com.invalid>
wrote:
> Hi,
>
> I have some questions on the current Iceberg spec regarding equality
> deletes:
> https://iceberg.apache.org/spec/#equality-delete-files
> The spec says that for "a table with the following data:
>
> <https://iceberg.apache.org/spec/#__codelineno-1-1> 1: id | 2: category | 3: name <https://iceberg.apache.org/spec/#__codelineno-1-2>-------|-------------|--------- <https://iceberg.apache.org/spec/#__codelineno-1-3> 1 | marsupial | Koala <https://iceberg.apache.org/spec/#__codelineno-1-4> 2 | toy | Teddy <https://iceberg.apache.org/spec/#__codelineno-1-5> 3 | NULL | Grizzly <https://iceberg.apache.org/spec/#__codelineno-1-6> 4 | NULL | Polar
>
> The delete id = 3 could be written as either of the following equality
> delete files:
>
> <https://iceberg.apache.org/spec/#__codelineno-2-1>equality_ids=[1] <https://iceberg.apache.org/spec/#__codelineno-2-2> <https://iceberg.apache.org/spec/#__codelineno-2-3> 1: id <https://iceberg.apache.org/spec/#__codelineno-2-4>------- <https://iceberg.apache.org/spec/#__codelineno-2-5> 3
>
> equality_ids=[1] <https://iceberg.apache.org/spec/#__codelineno-3-2> <https://iceberg.apache.org/spec/#__codelineno-3-3> 1: id | 2: category | 3: name <https://iceberg.apache.org/spec/#__codelineno-3-4>-------|-------------|--------- <https://iceberg.apache.org/spec/#__codelineno-3-5> 3 | NULL | Grizzly
>
> "
>
> 1. Are the options either (a) write only the column(s) listed in
> equality_ids or (b) write all the columns? i.e, no in between.
> 2. If we write all the columns, are only columns listed in equality_ids
> considered? What happens if a non-equality_id column does not match? e.g.,
>
> equality_ids=[1] <https://iceberg.apache.org/spec/#__codelineno-3-2>
> <https://iceberg.apache.org/spec/#__codelineno-3-3> 1: id | 2: category |
> 3: name <https://iceberg.apache.org/spec/#__codelineno-3-4>-------|-------------|---------
> <https://iceberg.apache.org/spec/#__codelineno-3-5> 3 | NULL | Polar
>
> Is that (a) invalid, or does that (b) still result in deleting id = 3, or
> (c) result in deleting no rows?
>
> The spec says "Each row of the delete file produces one equality
> predicate that matches any row where the delete columns are equal. Multiple
> columns can be thought of as an AND of equality predicates." That could
> be interpreted to mean (c).
>
> Thanks,
> Wing Yew
>
>
Re: spec question on equality deletes
Posted by Wing Yew Poon <wy...@cloudera.com.INVALID>.
Hi Renjie,
Thank you for your perspective.
On 1, I am inclined to the same view as you.
On 2, I feel that the spec should clearly define the expected behavior; it
should not be left to engines. At worst, the spec can say, e.g., that the
correct behavior is (b) but it is acceptable for an engine to throw an
error (a); or that the correct behavior is (c). We cannot have some engines
doing (b) and some doing (c), as (b) and (c) are basically opposite.
I'm interested in other perspectives.
- Wing Yew
Btw, I go by Wing Yew, not Wing.
On Sat, Apr 13, 2024 at 6:12 AM Renjie Liu <li...@gmail.com> wrote:
> Hi, Wing:
>
>
>
> 1. Are the options either (a) write only the column(s) listed in
> equality_ids or (b) write all the columns? i.e, no in between.
>
>
>
> Yes, I think so.
>
>
>
> 2. If we write all the columns, are only columns listed in equality_ids
> considered? What happens if a non-equality_id column does not match? e.g.,
>
>
>
> equality_ids=[1] 1: id | 2: category | 3: name
> -------|-------------|--------- 3 | NULL | Polar
>
>
>
> Is that (a) invalid, or does that (b) still result in deleting id = 3, or
> (c) result in deleting no rows?
>
>
>
> What columns are considered are depent:
>
> - Only columns listed in eqality_ids are considered when applying
> deletions.
> - If other columns are filled, they are considered during planning,
> e.g. helps to prune equal deletion files that should be applied to data
> file.
>
>
>
> I think it’s considered as invalid since it may produce wrong results,
> e.g. pruning extra deletion file.
>
>
>
> The spec says "Each row of the delete file produces one equality
> predicate that matches any row where the delete columns are equal. Multiple
> columns can be thought of as an AND of equality predicates." That could
> be interpreted to mean (c).
>
>
>
> Whether it’s incorrect depends on how the compute engine works. If the
> compute engine doesn’t try to prune deletion files, then inconsistent
> column data may not affect the result. But in general it should be
> considered as incorrect data.
>
>
>
> *From: *Wing Yew Poon <wy...@cloudera.com.INVALID>
> *Date: *Saturday, April 13, 2024 at 02:16
> *To: *dev@iceberg.apache.org <de...@iceberg.apache.org>
> *Subject: *spec question on equality deletes
>
> Hi,
>
>
>
> I have some questions on the current Iceberg spec regarding equality
> deletes:
>
> https://iceberg.apache.org/spec/#equality-delete-files
>
> The spec says that for "a table with the following data:
>
> 1: id | 2: category | 3: name
>
> -------|-------------|---------
>
> 1 | marsupial | Koala
>
> 2 | toy | Teddy
>
> 3 | NULL | Grizzly
>
> 4 | NULL | Polar
>
> The delete id = 3 could be written as either of the following equality
> delete files:
>
> equality_ids=[1]
>
>
>
> 1: id
>
> -------
>
> 3
>
> equality_ids=[1]
>
>
>
> 1: id | 2: category | 3: name
>
> -------|-------------|---------
>
> 3 | NULL | Grizzly
>
> "
>
>
>
> 1. Are the options either (a) write only the column(s) listed in
> equality_ids or (b) write all the columns? i.e, no in between.
>
> 2. If we write all the columns, are only columns listed in equality_ids
> considered? What happens if a non-equality_id column does not match? e.g.,
>
>
>
> equality_ids=[1] 1: id | 2: category | 3: name -------|-------------|---------
> 3 | NULL | Polar
>
>
>
> Is that (a) invalid, or does that (b) still result in deleting id = 3, or
> (c) result in deleting no rows?
>
>
>
> The spec says "Each row of the delete file produces one equality
> predicate that matches any row where the delete columns are equal. Multiple
> columns can be thought of as an AND of equality predicates." That could
> be interpreted to mean (c).
>
>
>
> Thanks,
>
> Wing Yew
>
>
>
Re: spec question on equality deletes
Posted by Renjie Liu <li...@gmail.com>.
Hi, Wing:
1. Are the options either (a) write only the column(s) listed in equality_ids or (b) write all the columns? i.e, no in between.
Yes, I think so.
2. If we write all the columns, are only columns listed in equality_ids considered? What happens if a non-equality_id column does not match? e.g.,
equality_ids=[1] 1: id | 2: category | 3: name -------|-------------|--------- 3 | NULL | Polar
Is that (a) invalid, or does that (b) still result in deleting id = 3, or (c) result in deleting no rows?
What columns are considered are depent:
* Only columns listed in eqality_ids are considered when applying deletions.
* If other columns are filled, they are considered during planning, e.g. helps to prune equal deletion files that should be applied to data file.
I think it’s considered as invalid since it may produce wrong results, e.g. pruning extra deletion file.
The spec says "Each row of the delete file produces one equality predicate that matches any row where the delete columns are equal. Multiple columns can be thought of as an AND of equality predicates." That could be interpreted to mean (c).
Whether it’s incorrect depends on how the compute engine works. If the compute engine doesn’t try to prune deletion files, then inconsistent column data may not affect the result. But in general it should be considered as incorrect data.
From: Wing Yew Poon <wy...@cloudera.com.INVALID>
Date: Saturday, April 13, 2024 at 02:16
To: dev@iceberg.apache.org <de...@iceberg.apache.org>
Subject: spec question on equality deletes
Hi,
I have some questions on the current Iceberg spec regarding equality deletes:
https://iceberg.apache.org/spec/#equality-delete-files
The spec says that for "a table with the following data:
1: id | 2: category | 3: name
-------|-------------|---------
1 | marsupial | Koala
2 | toy | Teddy
3 | NULL | Grizzly
4 | NULL | Polar
The delete id = 3 could be written as either of the following equality delete files:
equality_ids=[1]
1: id
-------
3
equality_ids=[1]
1: id | 2: category | 3: name
-------|-------------|---------
3 | NULL | Grizzly
"
1. Are the options either (a) write only the column(s) listed in equality_ids or (b) write all the columns? i.e, no in between.
2. If we write all the columns, are only columns listed in equality_ids considered? What happens if a non-equality_id column does not match? e.g.,
equality_ids=[1] 1: id | 2: category | 3: name -------|-------------|--------- 3 | NULL | Polar
Is that (a) invalid, or does that (b) still result in deleting id = 3, or (c) result in deleting no rows?
The spec says "Each row of the delete file produces one equality predicate that matches any row where the delete columns are equal. Multiple columns can be thought of as an AND of equality predicates." That could be interpreted to mean (c).
Thanks,
Wing Yew