An edge-case on snapshot expiration, incremental reads and very slow consecutive writes

[Filip <fi...@gmail.com>]

Hi team,

We've recently bumped into an issue with a particular edge case that messes
with our implementation of leveraging the incremental read and the expire
snapshot features combined.

With incremental read we're relying on the client to preserve the snapshot
that was last used for reading data as a checkpoint. Every time the client
does an incremental read it gets new data (if available) along with the
current snapshot that the client will store along as its new checkpoint.

Expire snapshot is scheduled to kick in and wipe snapshots based on recency
(say older than N days).
But in the edge-case of two consecutive write operations happening less
often than the expiration interval (*)  if the incremental read process
doesn't run before the snapshot expiration then the client will be left in
an inconsistent state since the snapshot it has stored as checkpoint is not
going to work anymore.

So we were looking at either extending the snapshot expiration feature or
extending the implementation of incremental read.

I'll just drop-in some details on exploring the solution to extend
incremental read - extend it by adding a fallback logic when the provided
snapshot is missing and try to locate the snapshot parented by that
particular snapshot instead.
This would change the logic of the incremental read with respect to
inclusiveness of loading the snapshots, if it currently considers the
provided "from" snapshot as exclusive, in the case of the fallback logic to
using the child snapshot as "from" it would have to be inclusive.

Let me know if you think this edge-case should be supported by Iceberg and
if this idea of extending the incremental read logic makes sense or if
folks in the community have a better solution for this.

(*) We expire snapshots older than 10 days but we observe two
consecutive write operations 11 days apart.

-- 
Filip Bocse

Re: An edge-case on snapshot expiration, incremental reads and very slow consecutive writes

[Filip <fi...@gmail.com>]

Really appreciate the insights, thank you.

So one takeaway is to keep around a comfortable buffer of snapshots
to reduce the probability of intersection of snapshot expiration and
incremental reads.

The other takeaway is the point of having snapshot expiration and
incremental read work by ranges of snapshots instead of ranges of time.
I think this provides a more deterministic understanding of when snapshot
expiration and incremental reading will intersect since it removes
the variance of write operations from the equation, right?
This simplification should help reduce even further the probability of
intersection even more.

Thank you,
/Filip


On Tue, Jan 12, 2021 at 11:34 PM John Clara <jo...@gmail.com>
wrote:

> To get around this, my team expires snapshots based on the number of
> snapshots rather than by time. For example, if the reader jobs is scheduled
> to consume 2k snapshot increments, we have a cron to retain the last 10k
> snapshots.
>
> That gives enough time to unclog the pipeline if the read job gets stuck.
>
> On Jan 12, 2021, at 9:44 AM, Ryan Blue <rb...@netflix.com.invalid> wrote:
>
> 
> I agree with Russell. The right thing to do is to keep snapshots around
> long enough that you don't lose the history. I don't think that changing
> the read logic would work because you need all the history to ensure you
> aren't skipping a snapshot.
>
> On Mon, Jan 11, 2021 at 11:24 AM Russell Spitzer <
> russell.spitzer@gmail.com> wrote:
>
>> I would probably try to just extend my expiration interval if that was
>> possible to fix the issue since it's basically functioning as a watermark
>> for state at the moment.
>>
>> Is our underlying issue here that we cannot determine the lineage of a
>> Snapshot that has been expired? IE: We know all the files and which
>> snapshots added them, but we cannot determine where our "From" snapshot
>> exists in history since we did the expiration?
>>
>> On Mon, Jan 11, 2021 at 11:07 AM Filip <fi...@gmail.com> wrote:
>>
>>> Hi team,
>>>
>>> We've recently bumped into an issue with a particular edge case that
>>> messes with our implementation of leveraging the incremental read and the
>>> expire snapshot features combined.
>>>
>>> With incremental read we're relying on the client to preserve the
>>> snapshot that was last used for reading data as a checkpoint. Every time
>>> the client does an incremental read it gets new data (if available) along
>>> with the current snapshot that the client will store along as its new
>>> checkpoint.
>>>
>>> Expire snapshot is scheduled to kick in and wipe snapshots based on
>>> recency (say older than N days).
>>> But in the edge-case of two consecutive write operations happening less
>>> often than the expiration interval (*)  if the incremental read process
>>> doesn't run before the snapshot expiration then the client will be left in
>>> an inconsistent state since the snapshot it has stored as checkpoint is not
>>> going to work anymore.
>>>
>>> So we were looking at either extending the snapshot expiration feature
>>> or extending the implementation of incremental read.
>>>
>>> I'll just drop-in some details on exploring the solution to extend
>>> incremental read - extend it by adding a fallback logic when the provided
>>> snapshot is missing and try to locate the snapshot parented by that
>>> particular snapshot instead.
>>> This would change the logic of the incremental read with respect to
>>> inclusiveness of loading the snapshots, if it currently considers the
>>> provided "from" snapshot as exclusive, in the case of the fallback logic to
>>> using the child snapshot as "from" it would have to be inclusive.
>>>
>>> Let me know if you think this edge-case should be supported by Iceberg
>>> and if this idea of extending the incremental read logic makes sense or if
>>> folks in the community have a better solution for this.
>>>
>>> (*) We expire snapshots older than 10 days but we observe two
>>> consecutive write operations 11 days apart.
>>>
>>> --
>>> Filip Bocse
>>>
>>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>
>

-- 
Filip Bocse

Re: An edge-case on snapshot expiration, incremental reads and very slow consecutive writes

[John Clara <jo...@gmail.com>]

To get around this, my team expires snapshots based on the number of snapshots rather than by time. For example, if the reader jobs is scheduled to consume 2k snapshot increments, we have a cron to retain the last 10k snapshots. 

That gives enough time to unclog the pipeline if the read job gets stuck.

> On Jan 12, 2021, at 9:44 AM, Ryan Blue <rb...@netflix.com.invalid> wrote:
> 
> 
> I agree with Russell. The right thing to do is to keep snapshots around long enough that you don't lose the history. I don't think that changing the read logic would work because you need all the history to ensure you aren't skipping a snapshot.
> 
>> On Mon, Jan 11, 2021 at 11:24 AM Russell Spitzer <ru...@gmail.com> wrote:
>> I would probably try to just extend my expiration interval if that was possible to fix the issue since it's basically functioning as a watermark for state at the moment. 
>> 
>> Is our underlying issue here that we cannot determine the lineage of a Snapshot that has been expired? IE: We know all the files and which snapshots added them, but we cannot determine where our "From" snapshot exists in history since we did the expiration?
>> 
>>> On Mon, Jan 11, 2021 at 11:07 AM Filip <fi...@gmail.com> wrote:
>>> Hi team,
>>> 
>>> We've recently bumped into an issue with a particular edge case that messes with our implementation of leveraging the incremental read and the expire snapshot features combined.
>>> 
>>> With incremental read we're relying on the client to preserve the snapshot that was last used for reading data as a checkpoint. Every time the client does an incremental read it gets new data (if available) along with the current snapshot that the client will store along as its new checkpoint.
>>> 
>>> Expire snapshot is scheduled to kick in and wipe snapshots based on recency (say older than N days).
>>> But in the edge-case of two consecutive write operations happening less often than the expiration interval (*)  if the incremental read process doesn't run before the snapshot expiration then the client will be left in an inconsistent state since the snapshot it has stored as checkpoint is not going to work anymore.
>>> 
>>> So we were looking at either extending the snapshot expiration feature or extending the implementation of incremental read.
>>> 
>>> I'll just drop-in some details on exploring the solution to extend incremental read - extend it by adding a fallback logic when the provided snapshot is missing and try to locate the snapshot parented by that particular snapshot instead. 
>>> This would change the logic of the incremental read with respect to inclusiveness of loading the snapshots, if it currently considers the provided "from" snapshot as exclusive, in the case of the fallback logic to using the child snapshot as "from" it would have to be inclusive.
>>> 
>>> Let me know if you think this edge-case should be supported by Iceberg and if this idea of extending the incremental read logic makes sense or if folks in the community have a better solution for this.
>>> 
>>> (*) We expire snapshots older than 10 days but we observe two consecutive write operations 11 days apart.
>>> 
>>> -- 
>>> Filip Bocse
> 
> 
> -- 
> Ryan Blue
> Software Engineer
> Netflix

Re: An edge-case on snapshot expiration, incremental reads and very slow consecutive writes

[Ryan Blue <rb...@netflix.com.INVALID>]

I agree with Russell. The right thing to do is to keep snapshots around
long enough that you don't lose the history. I don't think that changing
the read logic would work because you need all the history to ensure you
aren't skipping a snapshot.

On Mon, Jan 11, 2021 at 11:24 AM Russell Spitzer <ru...@gmail.com>
wrote:

> I would probably try to just extend my expiration interval if that was
> possible to fix the issue since it's basically functioning as a watermark
> for state at the moment.
>
> Is our underlying issue here that we cannot determine the lineage of a
> Snapshot that has been expired? IE: We know all the files and which
> snapshots added them, but we cannot determine where our "From" snapshot
> exists in history since we did the expiration?
>
> On Mon, Jan 11, 2021 at 11:07 AM Filip <fi...@gmail.com> wrote:
>
>> Hi team,
>>
>> We've recently bumped into an issue with a particular edge case that
>> messes with our implementation of leveraging the incremental read and the
>> expire snapshot features combined.
>>
>> With incremental read we're relying on the client to preserve the
>> snapshot that was last used for reading data as a checkpoint. Every time
>> the client does an incremental read it gets new data (if available) along
>> with the current snapshot that the client will store along as its new
>> checkpoint.
>>
>> Expire snapshot is scheduled to kick in and wipe snapshots based on
>> recency (say older than N days).
>> But in the edge-case of two consecutive write operations happening less
>> often than the expiration interval (*)  if the incremental read process
>> doesn't run before the snapshot expiration then the client will be left in
>> an inconsistent state since the snapshot it has stored as checkpoint is not
>> going to work anymore.
>>
>> So we were looking at either extending the snapshot expiration feature or
>> extending the implementation of incremental read.
>>
>> I'll just drop-in some details on exploring the solution to extend
>> incremental read - extend it by adding a fallback logic when the provided
>> snapshot is missing and try to locate the snapshot parented by that
>> particular snapshot instead.
>> This would change the logic of the incremental read with respect to
>> inclusiveness of loading the snapshots, if it currently considers the
>> provided "from" snapshot as exclusive, in the case of the fallback logic to
>> using the child snapshot as "from" it would have to be inclusive.
>>
>> Let me know if you think this edge-case should be supported by Iceberg
>> and if this idea of extending the incremental read logic makes sense or if
>> folks in the community have a better solution for this.
>>
>> (*) We expire snapshots older than 10 days but we observe two
>> consecutive write operations 11 days apart.
>>
>> --
>> Filip Bocse
>>
>

-- 
Ryan Blue
Software Engineer
Netflix

Re: An edge-case on snapshot expiration, incremental reads and very slow consecutive writes

[Russell Spitzer <ru...@gmail.com>]

I would probably try to just extend my expiration interval if that was
possible to fix the issue since it's basically functioning as a watermark
for state at the moment.

Is our underlying issue here that we cannot determine the lineage of a
Snapshot that has been expired? IE: We know all the files and which
snapshots added them, but we cannot determine where our "From" snapshot
exists in history since we did the expiration?

On Mon, Jan 11, 2021 at 11:07 AM Filip <fi...@gmail.com> wrote:

> Hi team,
>
> We've recently bumped into an issue with a particular edge case that
> messes with our implementation of leveraging the incremental read and the
> expire snapshot features combined.
>
> With incremental read we're relying on the client to preserve the snapshot
> that was last used for reading data as a checkpoint. Every time the client
> does an incremental read it gets new data (if available) along with the
> current snapshot that the client will store along as its new checkpoint.
>
> Expire snapshot is scheduled to kick in and wipe snapshots based on
> recency (say older than N days).
> But in the edge-case of two consecutive write operations happening less
> often than the expiration interval (*)  if the incremental read process
> doesn't run before the snapshot expiration then the client will be left in
> an inconsistent state since the snapshot it has stored as checkpoint is not
> going to work anymore.
>
> So we were looking at either extending the snapshot expiration feature or
> extending the implementation of incremental read.
>
> I'll just drop-in some details on exploring the solution to extend
> incremental read - extend it by adding a fallback logic when the provided
> snapshot is missing and try to locate the snapshot parented by that
> particular snapshot instead.
> This would change the logic of the incremental read with respect to
> inclusiveness of loading the snapshots, if it currently considers the
> provided "from" snapshot as exclusive, in the case of the fallback logic to
> using the child snapshot as "from" it would have to be inclusive.
>
> Let me know if you think this edge-case should be supported by Iceberg and
> if this idea of extending the incremental read logic makes sense or if
> folks in the community have a better solution for this.
>
> (*) We expire snapshots older than 10 days but we observe two
> consecutive write operations 11 days apart.
>
> --
> Filip Bocse
>