You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Szehon Ho <sz...@gmail.com> on 2021/07/20 23:20:24 UTC

Serializable isolation for insert overwrites?

Hi,

Does anyone know if its feasible to consider making Spark's "insert
overwrite" implement serializable transaction, like delete, update, merge?

Maybe at least for "overwrite by filter", then it can narrow down the
conflict checks needed on the commitWithSerializableTransaction side.  I
don't have the full context on the Spark side if its feasible to do the
rewrite as Delete/Merge/Update does, to use this mechanism.

Its for a use-case like "insert overwrite into table foo partition
(date=...) select ... from foo", which I understand is not the common use
case for insert overwrite, as its usually select from another table.

Thanks in advance,
Szehon

Re: Serializable isolation for insert overwrites?

Posted by Szehon Ho <sz...@gmail.com>.
Thanks Ryan for the confirmation, I'm definitely interested to take a
look.  if it can be done, the serializable isolation level could probably
be an option as for the other operations.  I will look a bit and ping you
when I get a chance.

Szehon

On Tue, Jul 20, 2021 at 5:11 PM Ryan Blue <bl...@tabular.io> wrote:

> Szehon,
>
> We implemented the current behavior because that’s what was expected for INSERT
> OVERWRITE. But the ReplacePartitions operation uses the same base class
> as the expression overwrite, so you could add more validation, including
> the conflict checks that you’re talking about by calling the
> validateAddedDataFiles helper method
> <https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java#L249-L250>
> on the base class.
>
> If you want to implement this, ping me on Slack and I can point you in the
> right direction.
>
> Ryan
>
> On Tue, Jul 20, 2021 at 4:20 PM Szehon Ho <sz...@gmail.com> wrote:
>
>> Hi,
>>
>> Does anyone know if its feasible to consider making Spark's "insert
>> overwrite" implement serializable transaction, like delete, update, merge?
>>
>> Maybe at least for "overwrite by filter", then it can narrow down the
>> conflict checks needed on the commitWithSerializableTransaction side.  I
>> don't have the full context on the Spark side if its feasible to do the
>> rewrite as Delete/Merge/Update does, to use this mechanism.
>>
>> Its for a use-case like "insert overwrite into table foo partition
>> (date=...) select ... from foo", which I understand is not the common use
>> case for insert overwrite, as its usually select from another table.
>>
>> Thanks in advance,
>> Szehon
>>
>>
>>
>
> --
> Ryan Blue
> Tabular
>

Re: Serializable isolation for insert overwrites?

Posted by Ryan Blue <bl...@tabular.io>.
Szehon,

We implemented the current behavior because that’s what was expected for INSERT
OVERWRITE. But the ReplacePartitions operation uses the same base class as
the expression overwrite, so you could add more validation, including the
conflict checks that you’re talking about by calling the
validateAddedDataFiles helper method
<https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java#L249-L250>
on the base class.

If you want to implement this, ping me on Slack and I can point you in the
right direction.

Ryan

On Tue, Jul 20, 2021 at 4:20 PM Szehon Ho <sz...@gmail.com> wrote:

> Hi,
>
> Does anyone know if its feasible to consider making Spark's "insert
> overwrite" implement serializable transaction, like delete, update, merge?
>
> Maybe at least for "overwrite by filter", then it can narrow down the
> conflict checks needed on the commitWithSerializableTransaction side.  I
> don't have the full context on the Spark side if its feasible to do the
> rewrite as Delete/Merge/Update does, to use this mechanism.
>
> Its for a use-case like "insert overwrite into table foo partition
> (date=...) select ... from foo", which I understand is not the common use
> case for insert overwrite, as its usually select from another table.
>
> Thanks in advance,
> Szehon
>
>
>

-- 
Ryan Blue
Tabular