You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Anton Okolnychyi <ao...@gmail.com> on 2025/02/01 02:13:24 UTC
[DISCUSS] Clarify delete counts handling in partition stats
Hi all,
I propose to clarify our delete counts handling in partition stats. We have
the following metrics that are marked as optional:
- position_delete_record_count
- position_delete_file_count
- equality_delete_record_count
- equality_delete_file_count
If I remember correctly, the reasoning behind this was that tables may have
no deletes, hence the counts are optional. The problem is that it creates
confusion for readers. Does null mean unknown or absent? I propose we
clarify that no counts means 0 for V1/V2 tables in the spec (this is the
current behavior in the Java implementation) and make the counts required
in V3.
What does everybody think?
- Anton
Re: [DISCUSS] Clarify delete counts handling in partition stats
Posted by Anton Okolnychyi <ao...@gmail.com>.
Thanks a lot, Ajantha!
пн, 3 лют. 2025 р. о 20:29 Ajantha Bhat <aj...@gmail.com> пише:
> Thanks, I will have a PR to update the implementation.
>
> On Tue, Feb 4, 2025 at 8:26 AM Anton Okolnychyi <ao...@gmail.com>
> wrote:
>
>> Oh, I think we actually missed to handle that particular count correctly.
>> The total number of records must stay optional as it is expensive to
>> compute with V2 delete files. In V3, however, we can reliably use DV
>> cardinalities to populate that field without scanning data if there are no
>> equality deletes and no existing V2 delete files. That said, it should
>> still be optional and NULL should mean unknown.
>>
>> пн, 3 лют. 2025 р. о 04:52 Ajantha Bhat <aj...@gmail.com> пише:
>>
>>> SGTM,
>>>
>>> Along with those counters, we should also update the
>>> `total_record_count` as during implementation we decided to keep all
>>> counter behavior the same if not computed.
>>>
>>> - Ajantha
>>>
>>>
>>> On Sat, Feb 1, 2025 at 2:38 PM Russell Spitzer <
>>> russell.spitzer@gmail.com> wrote:
>>>
>>>> Sounds reasonable, I think the intent was that N/A is different then 0
>>>> but that only makes sense for V1. For V2/V3 0 makes sense
>>>>
>>>> On Sat, Feb 1, 2025 at 3:15 AM Anton Okolnychyi <ao...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I propose to clarify our delete counts handling in partition stats. We
>>>>> have the following metrics that are marked as optional:
>>>>>
>>>>> - position_delete_record_count
>>>>> - position_delete_file_count
>>>>> - equality_delete_record_count
>>>>> - equality_delete_file_count
>>>>>
>>>>> If I remember correctly, the reasoning behind this was that tables may
>>>>> have no deletes, hence the counts are optional. The problem is that it
>>>>> creates confusion for readers. Does null mean unknown or absent? I propose
>>>>> we clarify that no counts means 0 for V1/V2 tables in the spec (this is the
>>>>> current behavior in the Java implementation) and make the counts required
>>>>> in V3.
>>>>>
>>>>> What does everybody think?
>>>>>
>>>>> - Anton
>>>>>
>>>>
Re: [DISCUSS] Clarify delete counts handling in partition stats
Posted by Ajantha Bhat <aj...@gmail.com>.
Thanks, I will have a PR to update the implementation.
On Tue, Feb 4, 2025 at 8:26 AM Anton Okolnychyi <ao...@gmail.com>
wrote:
> Oh, I think we actually missed to handle that particular count correctly.
> The total number of records must stay optional as it is expensive to
> compute with V2 delete files. In V3, however, we can reliably use DV
> cardinalities to populate that field without scanning data if there are no
> equality deletes and no existing V2 delete files. That said, it should
> still be optional and NULL should mean unknown.
>
> пн, 3 лют. 2025 р. о 04:52 Ajantha Bhat <aj...@gmail.com> пише:
>
>> SGTM,
>>
>> Along with those counters, we should also update the `total_record_count`
>> as during implementation we decided to keep all counter behavior the same
>> if not computed.
>>
>> - Ajantha
>>
>>
>> On Sat, Feb 1, 2025 at 2:38 PM Russell Spitzer <ru...@gmail.com>
>> wrote:
>>
>>> Sounds reasonable, I think the intent was that N/A is different then 0
>>> but that only makes sense for V1. For V2/V3 0 makes sense
>>>
>>> On Sat, Feb 1, 2025 at 3:15 AM Anton Okolnychyi <ao...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I propose to clarify our delete counts handling in partition stats. We
>>>> have the following metrics that are marked as optional:
>>>>
>>>> - position_delete_record_count
>>>> - position_delete_file_count
>>>> - equality_delete_record_count
>>>> - equality_delete_file_count
>>>>
>>>> If I remember correctly, the reasoning behind this was that tables may
>>>> have no deletes, hence the counts are optional. The problem is that it
>>>> creates confusion for readers. Does null mean unknown or absent? I propose
>>>> we clarify that no counts means 0 for V1/V2 tables in the spec (this is the
>>>> current behavior in the Java implementation) and make the counts required
>>>> in V3.
>>>>
>>>> What does everybody think?
>>>>
>>>> - Anton
>>>>
>>>
Re: [DISCUSS] Clarify delete counts handling in partition stats
Posted by Anton Okolnychyi <ao...@gmail.com>.
Oh, I think we actually missed to handle that particular count correctly.
The total number of records must stay optional as it is expensive to
compute with V2 delete files. In V3, however, we can reliably use DV
cardinalities to populate that field without scanning data if there are no
equality deletes and no existing V2 delete files. That said, it should
still be optional and NULL should mean unknown.
пн, 3 лют. 2025 р. о 04:52 Ajantha Bhat <aj...@gmail.com> пише:
> SGTM,
>
> Along with those counters, we should also update the `total_record_count`
> as during implementation we decided to keep all counter behavior the same
> if not computed.
>
> - Ajantha
>
>
> On Sat, Feb 1, 2025 at 2:38 PM Russell Spitzer <ru...@gmail.com>
> wrote:
>
>> Sounds reasonable, I think the intent was that N/A is different then 0
>> but that only makes sense for V1. For V2/V3 0 makes sense
>>
>> On Sat, Feb 1, 2025 at 3:15 AM Anton Okolnychyi <ao...@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> I propose to clarify our delete counts handling in partition stats. We
>>> have the following metrics that are marked as optional:
>>>
>>> - position_delete_record_count
>>> - position_delete_file_count
>>> - equality_delete_record_count
>>> - equality_delete_file_count
>>>
>>> If I remember correctly, the reasoning behind this was that tables may
>>> have no deletes, hence the counts are optional. The problem is that it
>>> creates confusion for readers. Does null mean unknown or absent? I propose
>>> we clarify that no counts means 0 for V1/V2 tables in the spec (this is the
>>> current behavior in the Java implementation) and make the counts required
>>> in V3.
>>>
>>> What does everybody think?
>>>
>>> - Anton
>>>
>>
Re: [DISCUSS] Clarify delete counts handling in partition stats
Posted by Ajantha Bhat <aj...@gmail.com>.
SGTM,
Along with those counters, we should also update the `total_record_count`
as during implementation we decided to keep all counter behavior the same
if not computed.
- Ajantha
On Sat, Feb 1, 2025 at 2:38 PM Russell Spitzer <ru...@gmail.com>
wrote:
> Sounds reasonable, I think the intent was that N/A is different then 0 but
> that only makes sense for V1. For V2/V3 0 makes sense
>
> On Sat, Feb 1, 2025 at 3:15 AM Anton Okolnychyi <ao...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> I propose to clarify our delete counts handling in partition stats. We
>> have the following metrics that are marked as optional:
>>
>> - position_delete_record_count
>> - position_delete_file_count
>> - equality_delete_record_count
>> - equality_delete_file_count
>>
>> If I remember correctly, the reasoning behind this was that tables may
>> have no deletes, hence the counts are optional. The problem is that it
>> creates confusion for readers. Does null mean unknown or absent? I propose
>> we clarify that no counts means 0 for V1/V2 tables in the spec (this is the
>> current behavior in the Java implementation) and make the counts required
>> in V3.
>>
>> What does everybody think?
>>
>> - Anton
>>
>
Re: [DISCUSS] Clarify delete counts handling in partition stats
Posted by Russell Spitzer <ru...@gmail.com>.
Sounds reasonable, I think the intent was that N/A is different then 0 but
that only makes sense for V1. For V2/V3 0 makes sense
On Sat, Feb 1, 2025 at 3:15 AM Anton Okolnychyi <ao...@gmail.com>
wrote:
> Hi all,
>
> I propose to clarify our delete counts handling in partition stats. We
> have the following metrics that are marked as optional:
>
> - position_delete_record_count
> - position_delete_file_count
> - equality_delete_record_count
> - equality_delete_file_count
>
> If I remember correctly, the reasoning behind this was that tables may
> have no deletes, hence the counts are optional. The problem is that it
> creates confusion for readers. Does null mean unknown or absent? I propose
> we clarify that no counts means 0 for V1/V2 tables in the spec (this is the
> current behavior in the Java implementation) and make the counts required
> in V3.
>
> What does everybody think?
>
> - Anton
>