You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by OpenInx <op...@gmail.com> on 2020/10/20 02:14:44 UTC
Several flink pull requests need to get merged before the next
release 0.10.0
Hi
As we know that we next release 0.10.0 is coming, there are several issues
which should be merged as soon as possible in my mind:
1. https://github.com/apache/iceberg/pull/1477
It will change the flink state design to maintain the complete data files
into manifest before checkpoint finished, it good for minimal the flink
state size and improve the state compatibility (Before that we will
serialize the DataFile into flink state backend, while DataFile class have
depended some java serializable classes, the means if we change the
dependencies classes, it may fail to deserialize the state). Currently,
I gained a +1 from Steven Zhen Wu, thanks for his patient reviewing.
According to the apache rule, I need another +1 from iceberg committers,
anyone have time to get the review finished ?
2. https://github.com/apache/iceberg/pull/1586
This will introduce options to load the external hive-site.xml for flink
hive catalog, which is really helpful for production environment, not a
hard change. But will still need a review from iceberg members. Thanks.
3. https://github.com/apache/iceberg/pull/1619
We introduced another write parallelism for iceberg flink stream writers.
Thanks kbendick and Stevenzwu for the reviewing, gain two +1 now. Should
I merge this ?
Besides the flink PRs, it is very beneficial to put forward other related
issues which is blocking the release 0.10.0 . I am happy to help resolve
these issues.
Thanks.
Re: Several flink pull requests need to get merged before the next
release 0.10.0
Posted by Ryan Blue <rb...@netflix.com.INVALID>.
Yes, we've merged the PRs and I think we're good to go for the 0.10.0
release. Great to have Flink support in!
On Thu, Oct 29, 2020 at 12:33 AM Kyle Bendickson <kj...@gmail.com>
wrote:
> Oops spoke too soon. Looks like they've all been merged.and I approved the
> last one /shrug. Sorry for the late night email response everyone.
>
> - Kyle
>
> On Thu, Oct 29, 2020 at 12:22 AM Kyle Bendickson <kj...@gmail.com>
> wrote:
>
>> I will go through and re-review all of these PRs over the next two days
>> Zheng to help get these merged asap.
>>
>> - Kyle
>> @kbendick
>>
>> On Tue, Oct 27, 2020 at 1:30 AM OpenInx <op...@gmail.com> wrote:
>>
>>> Hi Ryan
>>>
>>> Is it the correct time once we get the PR 1477 merged ? Do we have any
>>> other blockers for the coming release 0.10.0 ?
>>>
>>> Thanks.
>>>
>>> On Wed, Oct 21, 2020 at 9:13 AM Ryan Blue <rb...@netflix.com.invalid>
>>> wrote:
>>>
>>>> Hey, thanks for bringing these up. I'm planning on spending some time
>>>> reviewing tomorrow and I can take a look at the first two.
>>>>
>>>> I just merged the first one since it was small, thanks for the fix!
>>>> Feel free to ping me or other committers to review these. I do think it is
>>>> important to have a committer review, even if the community also has
>>>> positive reviews.
>>>>
>>>> rb
>>>>
>>>> On Mon, Oct 19, 2020 at 7:15 PM OpenInx <op...@gmail.com> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> As we know that we next release 0.10.0 is coming, there are several
>>>>> issues which should be merged as soon as possible in my mind:
>>>>>
>>>>> 1. https://github.com/apache/iceberg/pull/1477
>>>>>
>>>>> It will change the flink state design to maintain the complete data
>>>>> files into manifest before checkpoint finished, it good for minimal the
>>>>> flink state size and improve the state compatibility (Before that we will
>>>>> serialize the DataFile into flink state backend, while DataFile class have
>>>>> depended some java serializable classes, the means if we change the
>>>>> dependencies classes, it may fail to deserialize the state). Currently,
>>>>> I gained a +1 from Steven Zhen Wu, thanks for his patient reviewing.
>>>>> According to the apache rule, I need another +1 from iceberg committers,
>>>>> anyone have time to get the review finished ?
>>>>>
>>>>> 2. https://github.com/apache/iceberg/pull/1586
>>>>>
>>>>> This will introduce options to load the external hive-site.xml for
>>>>> flink hive catalog, which is really helpful for production environment, not
>>>>> a hard change. But will still need a review from iceberg members. Thanks.
>>>>>
>>>>> 3. https://github.com/apache/iceberg/pull/1619
>>>>>
>>>>> We introduced another write parallelism for iceberg flink stream
>>>>> writers. Thanks kbendick and Stevenzwu for the reviewing, gain two +1
>>>>> now. Should I merge this ?
>>>>>
>>>>>
>>>>> Besides the flink PRs, it is very beneficial to put forward other
>>>>> related issues which is blocking the release 0.10.0 . I am happy to help
>>>>> resolve these issues.
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Ryan Blue
>>>> Software Engineer
>>>> Netflix
>>>>
>>>
--
Ryan Blue
Software Engineer
Netflix
Re: Several flink pull requests need to get merged before the next
release 0.10.0
Posted by Kyle Bendickson <kj...@gmail.com>.
Oops spoke too soon. Looks like they've all been merged.and I approved the
last one /shrug. Sorry for the late night email response everyone.
- Kyle
On Thu, Oct 29, 2020 at 12:22 AM Kyle Bendickson <kj...@gmail.com>
wrote:
> I will go through and re-review all of these PRs over the next two days
> Zheng to help get these merged asap.
>
> - Kyle
> @kbendick
>
> On Tue, Oct 27, 2020 at 1:30 AM OpenInx <op...@gmail.com> wrote:
>
>> Hi Ryan
>>
>> Is it the correct time once we get the PR 1477 merged ? Do we have any
>> other blockers for the coming release 0.10.0 ?
>>
>> Thanks.
>>
>> On Wed, Oct 21, 2020 at 9:13 AM Ryan Blue <rb...@netflix.com.invalid>
>> wrote:
>>
>>> Hey, thanks for bringing these up. I'm planning on spending some time
>>> reviewing tomorrow and I can take a look at the first two.
>>>
>>> I just merged the first one since it was small, thanks for the fix! Feel
>>> free to ping me or other committers to review these. I do think it is
>>> important to have a committer review, even if the community also has
>>> positive reviews.
>>>
>>> rb
>>>
>>> On Mon, Oct 19, 2020 at 7:15 PM OpenInx <op...@gmail.com> wrote:
>>>
>>>> Hi
>>>>
>>>> As we know that we next release 0.10.0 is coming, there are several
>>>> issues which should be merged as soon as possible in my mind:
>>>>
>>>> 1. https://github.com/apache/iceberg/pull/1477
>>>>
>>>> It will change the flink state design to maintain the complete data
>>>> files into manifest before checkpoint finished, it good for minimal the
>>>> flink state size and improve the state compatibility (Before that we will
>>>> serialize the DataFile into flink state backend, while DataFile class have
>>>> depended some java serializable classes, the means if we change the
>>>> dependencies classes, it may fail to deserialize the state). Currently,
>>>> I gained a +1 from Steven Zhen Wu, thanks for his patient reviewing.
>>>> According to the apache rule, I need another +1 from iceberg committers,
>>>> anyone have time to get the review finished ?
>>>>
>>>> 2. https://github.com/apache/iceberg/pull/1586
>>>>
>>>> This will introduce options to load the external hive-site.xml for
>>>> flink hive catalog, which is really helpful for production environment, not
>>>> a hard change. But will still need a review from iceberg members. Thanks.
>>>>
>>>> 3. https://github.com/apache/iceberg/pull/1619
>>>>
>>>> We introduced another write parallelism for iceberg flink stream
>>>> writers. Thanks kbendick and Stevenzwu for the reviewing, gain two +1
>>>> now. Should I merge this ?
>>>>
>>>>
>>>> Besides the flink PRs, it is very beneficial to put forward other
>>>> related issues which is blocking the release 0.10.0 . I am happy to help
>>>> resolve these issues.
>>>>
>>>> Thanks.
>>>>
>>>>
>>>
>>>
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>>
>>
Re: Several flink pull requests need to get merged before the next
release 0.10.0
Posted by Kyle Bendickson <kj...@gmail.com>.
I will go through and re-review all of these PRs over the next two days
Zheng to help get these merged asap.
- Kyle
@kbendick
On Tue, Oct 27, 2020 at 1:30 AM OpenInx <op...@gmail.com> wrote:
> Hi Ryan
>
> Is it the correct time once we get the PR 1477 merged ? Do we have any
> other blockers for the coming release 0.10.0 ?
>
> Thanks.
>
> On Wed, Oct 21, 2020 at 9:13 AM Ryan Blue <rb...@netflix.com.invalid>
> wrote:
>
>> Hey, thanks for bringing these up. I'm planning on spending some time
>> reviewing tomorrow and I can take a look at the first two.
>>
>> I just merged the first one since it was small, thanks for the fix! Feel
>> free to ping me or other committers to review these. I do think it is
>> important to have a committer review, even if the community also has
>> positive reviews.
>>
>> rb
>>
>> On Mon, Oct 19, 2020 at 7:15 PM OpenInx <op...@gmail.com> wrote:
>>
>>> Hi
>>>
>>> As we know that we next release 0.10.0 is coming, there are several
>>> issues which should be merged as soon as possible in my mind:
>>>
>>> 1. https://github.com/apache/iceberg/pull/1477
>>>
>>> It will change the flink state design to maintain the complete data
>>> files into manifest before checkpoint finished, it good for minimal the
>>> flink state size and improve the state compatibility (Before that we will
>>> serialize the DataFile into flink state backend, while DataFile class have
>>> depended some java serializable classes, the means if we change the
>>> dependencies classes, it may fail to deserialize the state). Currently,
>>> I gained a +1 from Steven Zhen Wu, thanks for his patient reviewing.
>>> According to the apache rule, I need another +1 from iceberg committers,
>>> anyone have time to get the review finished ?
>>>
>>> 2. https://github.com/apache/iceberg/pull/1586
>>>
>>> This will introduce options to load the external hive-site.xml for flink
>>> hive catalog, which is really helpful for production environment, not a
>>> hard change. But will still need a review from iceberg members. Thanks.
>>>
>>> 3. https://github.com/apache/iceberg/pull/1619
>>>
>>> We introduced another write parallelism for iceberg flink stream
>>> writers. Thanks kbendick and Stevenzwu for the reviewing, gain two +1
>>> now. Should I merge this ?
>>>
>>>
>>> Besides the flink PRs, it is very beneficial to put forward other
>>> related issues which is blocking the release 0.10.0 . I am happy to help
>>> resolve these issues.
>>>
>>> Thanks.
>>>
>>>
>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>
Re: Several flink pull requests need to get merged before the next
release 0.10.0
Posted by OpenInx <op...@gmail.com>.
Hi Ryan
Is it the correct time once we get the PR 1477 merged ? Do we have any
other blockers for the coming release 0.10.0 ?
Thanks.
On Wed, Oct 21, 2020 at 9:13 AM Ryan Blue <rb...@netflix.com.invalid> wrote:
> Hey, thanks for bringing these up. I'm planning on spending some time
> reviewing tomorrow and I can take a look at the first two.
>
> I just merged the first one since it was small, thanks for the fix! Feel
> free to ping me or other committers to review these. I do think it is
> important to have a committer review, even if the community also has
> positive reviews.
>
> rb
>
> On Mon, Oct 19, 2020 at 7:15 PM OpenInx <op...@gmail.com> wrote:
>
>> Hi
>>
>> As we know that we next release 0.10.0 is coming, there are several
>> issues which should be merged as soon as possible in my mind:
>>
>> 1. https://github.com/apache/iceberg/pull/1477
>>
>> It will change the flink state design to maintain the complete data files
>> into manifest before checkpoint finished, it good for minimal the flink
>> state size and improve the state compatibility (Before that we will
>> serialize the DataFile into flink state backend, while DataFile class have
>> depended some java serializable classes, the means if we change the
>> dependencies classes, it may fail to deserialize the state). Currently,
>> I gained a +1 from Steven Zhen Wu, thanks for his patient reviewing.
>> According to the apache rule, I need another +1 from iceberg committers,
>> anyone have time to get the review finished ?
>>
>> 2. https://github.com/apache/iceberg/pull/1586
>>
>> This will introduce options to load the external hive-site.xml for flink
>> hive catalog, which is really helpful for production environment, not a
>> hard change. But will still need a review from iceberg members. Thanks.
>>
>> 3. https://github.com/apache/iceberg/pull/1619
>>
>> We introduced another write parallelism for iceberg flink stream writers.
>> Thanks kbendick and Stevenzwu for the reviewing, gain two +1 now. Should
>> I merge this ?
>>
>>
>> Besides the flink PRs, it is very beneficial to put forward other
>> related issues which is blocking the release 0.10.0 . I am happy to help
>> resolve these issues.
>>
>> Thanks.
>>
>>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>
Re: Several flink pull requests need to get merged before the next
release 0.10.0
Posted by Ryan Blue <rb...@netflix.com.INVALID>.
Hey, thanks for bringing these up. I'm planning on spending some time
reviewing tomorrow and I can take a look at the first two.
I just merged the first one since it was small, thanks for the fix! Feel
free to ping me or other committers to review these. I do think it is
important to have a committer review, even if the community also has
positive reviews.
rb
On Mon, Oct 19, 2020 at 7:15 PM OpenInx <op...@gmail.com> wrote:
> Hi
>
> As we know that we next release 0.10.0 is coming, there are several issues
> which should be merged as soon as possible in my mind:
>
> 1. https://github.com/apache/iceberg/pull/1477
>
> It will change the flink state design to maintain the complete data files
> into manifest before checkpoint finished, it good for minimal the flink
> state size and improve the state compatibility (Before that we will
> serialize the DataFile into flink state backend, while DataFile class have
> depended some java serializable classes, the means if we change the
> dependencies classes, it may fail to deserialize the state). Currently,
> I gained a +1 from Steven Zhen Wu, thanks for his patient reviewing.
> According to the apache rule, I need another +1 from iceberg committers,
> anyone have time to get the review finished ?
>
> 2. https://github.com/apache/iceberg/pull/1586
>
> This will introduce options to load the external hive-site.xml for flink
> hive catalog, which is really helpful for production environment, not a
> hard change. But will still need a review from iceberg members. Thanks.
>
> 3. https://github.com/apache/iceberg/pull/1619
>
> We introduced another write parallelism for iceberg flink stream writers.
> Thanks kbendick and Stevenzwu for the reviewing, gain two +1 now. Should
> I merge this ?
>
>
> Besides the flink PRs, it is very beneficial to put forward other related
> issues which is blocking the release 0.10.0 . I am happy to help resolve
> these issues.
>
> Thanks.
>
>
--
Ryan Blue
Software Engineer
Netflix