Append file and update properties in a single transaction

[Omar Aloraini <al...@gmail.com>]

Hello everyone,

I'm trying to append new files and update the table properties in single
transaction, but it seems from the source code that I can't. Is their a
workaround for this?

Regards

Re: Append file and update properties in a single transaction

[Omar Aloraini <al...@gmail.com>]

Wow, Thanks a lot. I will try it tomorrow. 🙏🙏

On Mon, Dec 7, 2020, 9:25 PM Ryan Blue <rb...@netflix.com> wrote:

> Transactions do support property updates:
> https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/BaseTransaction.java#L106-L111
>
> The commit check is to ensure that the last operation was complete before
> adding a new operation. You need to call commit on each operation created
> from the transaction or transaction table, but it doesn't modify the table
> until the entire transaction is committed.
>
> On Mon, Dec 7, 2020 at 10:18 AM Omar Aloraini <al...@gmail.com>
> wrote:
>
>> Hi Ryan, thanks for the reply,
>>
>> I can't recall the class name (will update you once I check), I think it
>> was TransactionalTable, every method that modifies the table has a check
>> weather the last part of a transaction (belonging to the same object
>> created by table.newTransaction) was committed, i.e. newAppend,
>> updateProperties and so on.
>>
>> I'm not familiar with how iceberg performs a transaction at a low level,
>> be it through hdfs rename or else, I'll look into it tomorrow and if it's
>> not too difficult, I would like to work on it.
>>
>> I will update you with details once I am at work tomorrow.
>>
>> Regards
>>
>> On Mon, Dec 7, 2020, 9:05 PM Ryan Blue <rb...@netflix.com.invalid> wrote:
>>
>>> Omar,
>>>
>>> You can append files and update properties. You just need to create a
>>> transaction using `newTransaction` in the `Table` API.
>>>
>>> rb
>>>
>>> On Sun, Dec 6, 2020 at 7:16 AM Omar Aloraini <al...@gmail.com>
>>> wrote:
>>>
>>>> Hello everyone,
>>>>
>>>> I'm trying to append new files and update the table properties in
>>>> single transaction, but it seems from the source code that I can't. Is
>>>> their a workaround for this?
>>>>
>>>> Regards
>>>>
>>>
>>>
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>>
>>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: Append file and update properties in a single transaction

[Ryan Blue <rb...@netflix.com.INVALID>]

Transactions do support property updates:
https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/BaseTransaction.java#L106-L111

The commit check is to ensure that the last operation was complete before
adding a new operation. You need to call commit on each operation created
from the transaction or transaction table, but it doesn't modify the table
until the entire transaction is committed.

On Mon, Dec 7, 2020 at 10:18 AM Omar Aloraini <al...@gmail.com>
wrote:

> Hi Ryan, thanks for the reply,
>
> I can't recall the class name (will update you once I check), I think it
> was TransactionalTable, every method that modifies the table has a check
> weather the last part of a transaction (belonging to the same object
> created by table.newTransaction) was committed, i.e. newAppend,
> updateProperties and so on.
>
> I'm not familiar with how iceberg performs a transaction at a low level,
> be it through hdfs rename or else, I'll look into it tomorrow and if it's
> not too difficult, I would like to work on it.
>
> I will update you with details once I am at work tomorrow.
>
> Regards
>
> On Mon, Dec 7, 2020, 9:05 PM Ryan Blue <rb...@netflix.com.invalid> wrote:
>
>> Omar,
>>
>> You can append files and update properties. You just need to create a
>> transaction using `newTransaction` in the `Table` API.
>>
>> rb
>>
>> On Sun, Dec 6, 2020 at 7:16 AM Omar Aloraini <al...@gmail.com>
>> wrote:
>>
>>> Hello everyone,
>>>
>>> I'm trying to append new files and update the table properties in single
>>> transaction, but it seems from the source code that I can't. Is their a
>>> workaround for this?
>>>
>>> Regards
>>>
>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>

-- 
Ryan Blue
Software Engineer
Netflix

Re: Append file and update properties in a single transaction

[Omar Aloraini <al...@gmail.com>]

Hi Ryan, thanks for the reply,

I can't recall the class name (will update you once I check), I think it
was TransactionalTable, every method that modifies the table has a check
weather the last part of a transaction (belonging to the same object
created by table.newTransaction) was committed, i.e. newAppend,
updateProperties and so on.

I'm not familiar with how iceberg performs a transaction at a low level, be
it through hdfs rename or else, I'll look into it tomorrow and if it's not
too difficult, I would like to work on it.

I will update you with details once I am at work tomorrow.

Regards

On Mon, Dec 7, 2020, 9:05 PM Ryan Blue <rb...@netflix.com.invalid> wrote:

> Omar,
>
> You can append files and update properties. You just need to create a
> transaction using `newTransaction` in the `Table` API.
>
> rb
>
> On Sun, Dec 6, 2020 at 7:16 AM Omar Aloraini <al...@gmail.com>
> wrote:
>
>> Hello everyone,
>>
>> I'm trying to append new files and update the table properties in single
>> transaction, but it seems from the source code that I can't. Is their a
>> workaround for this?
>>
>> Regards
>>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: Append file and update properties in a single transaction

[Ryan Blue <rb...@netflix.com.INVALID>]

Omar,

You can append files and update properties. You just need to create a
transaction using `newTransaction` in the `Table` API.

rb

On Sun, Dec 6, 2020 at 7:16 AM Omar Aloraini <al...@gmail.com>
wrote:

> Hello everyone,
>
> I'm trying to append new files and update the table properties in single
> transaction, but it seems from the source code that I can't. Is their a
> workaround for this?
>
> Regards
>


-- 
Ryan Blue
Software Engineer
Netflix