You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Udi Meiri <eh...@google.com> on 2021/09/10 22:50:38 UTC
Re: [PROPOSAL] Preparing for Beam 2.33.0 Release
The release branch is now passing Jenkins tests:
https://github.com/apache/beam/pull/15431
Thank you all so far for help with release blockers. There are 5 left
<https://issues.apache.org/jira/projects/BEAM/versions/12350404>.
On Thu, Aug 26, 2021 at 7:54 AM Alexey Romanenko <ar...@gmail.com>
wrote:
> See my comment below.
>
> On 26 Aug 2021, at 03:44, Ahmet Altay <al...@google.com> wrote:
>
> On Wed, Aug 25, 2021 at 1:50 PM Udi Meiri <eh...@google.com> wrote:
>
>> I've found 2 performance regressions in IO. Could someone triage these?
>>
>
>
>> https://issues.apache.org/jira/browse/BEAM-12801
>>
>
> @Alexey Romanenko <ar...@gmail.com> @Tim Robertson
> <ti...@gmail.com> - Could you comment on this? It is related
> to HadoopFormatIO.
>
>
>
> I don’t see any code changes for “sdks/java/io/hadoop-format” package
> since June 2nd while the Write performance dropped at Aug 10th for the
> first time.
> I’ll try to reproduce it locally with a direct runner but seems it was
> caused by something else. Any other ideas about this?
>
>
>
>> https://issues.apache.org/jira/browse/BEAM-12800
>>
>
> @Chamikara Jayalath <ch...@google.com> @Yifan Mai
> <yi...@google.com> - Do you know who could comment on this?
>
>
>>
>>
>> +14 other blockers currently
>>
>
> I have not looked at the other blockers. It might help to check with the
> assignees. They may not know that they are release blockers.
>
>
>>
>> On Wed, Aug 25, 2021 at 12:22 PM Udi Meiri <eh...@google.com> wrote:
>>
>>> I'll try cutting the release today. From what I can tell [1], the only
>>> perma-reds are python postcommits.
>>>
>>
> I believe this is fixed now. (Thanks to @Kyle Weaver <kc...@google.com>
> )
>
>
>>
>>> [1]
>>> https://github.com/apache/beam/blob/master/.github/PULL_REQUEST_TEMPLATE.md
>>>
>>> On Tue, Aug 17, 2021 at 10:12 AM Udi Meiri <eh...@google.com> wrote:
>>>
>>>> Please mark 2.33.0 release blockers by setting the "Fix version" field
>>>> to 2.33.0. Otherwise, it doesn't appear in the dashboard
>>>> <https://issues.apache.org/jira/projects/BEAM/versions/12350404>.
>>>>
>>>> On Tue, Aug 17, 2021 at 9:32 AM Luke Cwik <lc...@google.com> wrote:
>>>>
>>>>> +Rui Wang <ru...@google.com> as the current build monitor.
>>>>>
>>>>> At the time I did not mean to add any additional tests as it was a
>>>>> copy paste error.
>>>>>
>>>>> I do have another fix for the failing Python PreCommit which I fixed
>>>>> since it seemed like a trivial one:
>>>>> https://issues.apache.org/jira/browse/BEAM-12768
>>>>> - test_df_agg_method_invalid_kwarg_raises too strict on error checking. fix
>>>>> in https://github.com/apache/beam/pull/15341
>>>>>
>>>>> https://issues.apache.org/jira/browse/BEAM-12733 (RecommendationAIIT
>>>>> Java Version)
>>>>> +Matthias Baetens <ba...@gmail.com> for the above issue.
>>>>>
>>>>> On Mon, Aug 16, 2021 at 6:54 PM Ahmet Altay <al...@google.com> wrote:
>>>>>
>>>>>> Thank you Udi and +1 to the release cut on schedule.
>>>>>>
>>>>>> On Mon, Aug 16, 2021 at 3:53 PM Luke Cwik <lc...@google.com> wrote:
>>>>>>
>>>>>>> It seems as though several tests were broken due to commits during
>>>>>>> the issue with some existing flakes. It looks like the Python PostCommit
>>>>>>> tests are perma-red due to these failures. So far I'm tracking:
>>>>>>>
>>>>>>
>>>>>> Thank you very much Luke. Adding bug owners here to explicitly call
>>>>>> out release blocking issues.
>>>>>>
>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/BEAM-12764 (new_block method
>>>>>>> missing in pandas)
>>>>>>>
>>>>>>
>>>>>> +Sam Rohde <sr...@google.com> for the above issue.
>>>>>>
>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/BEAM-12683
>>>>>>> (RecommendationAIIT)
>>>>>>>
>>>>>>
>>>>>> +Matthias Baetens <ba...@gmail.com> for the above issue.
>>>>>>
>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/BEAM-12765 (cannot access
>>>>>>> field fruit)
>>>>>>> https://issues.apache.org/jira/browse/BEAM-12766 (Already Exists:
>>>>>>> Dataset apache-beam-testing:python_bq_file_loads_)
>>>>>>>
>>>>>>
>>>>>> +Pablo Estrada <pa...@google.com> for the two issues above.
>>>>>>
>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/BEAM-12694
>>>>>>> (DICOMIoIntegrationTest)
>>>>>>>
>>>>>>
>>>>>> +Frank Yang <xi...@google.com> +Dave Anderson
>>>>>> <dr...@google.com> for the DICOMIO. Could one of you look at
>>>>>> this? Looks like it might be a simple fix.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/BEAM-12766 (BQ Already exists
>>>>>>> Dataset)
>>>>>>>
>>>>>>
>>>>>> Luke, did you mean link to another issue? This one is a duplicate.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Aug 16, 2021 at 10:45 AM Luke Cwik <lc...@google.com> wrote:
>>>>>>>
>>>>>>>> There are the ones that I believe went perma-red due to the issue:
>>>>>>>> beam_PostCommit_Python36
>>>>>>>> beam_PostCommit_Python37
>>>>>>>> beam_PostCommit_Python38
>>>>>>>> beam_PreCommit_Python_Cron
>>>>>>>> beam_PostCommit_Java
>>>>>>>> beam_PostCommit_Java_DataflowV2
>>>>>>>> beam_PreCommit_Java_Examples_Dataflow_Cron
>>>>>>>> beam_PostRelease_NightlySnapshot
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Aug 16, 2021 at 10:42 AM Udi Meiri <eh...@google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Sure, I'll verify the status of the tests in our PR template. Any
>>>>>>>>> others I should be aware of?
>>>>>>>>>
>>>>>>>>> On Mon, Aug 16, 2021 at 10:15 AM Andrew Pilloud <
>>>>>>>>> apilloud@google.com> wrote:
>>>>>>>>>
>>>>>>>>>> That sounds reasonable to me. A week should give everyone enough
>>>>>>>>>> time to get their changes in.
>>>>>>>>>>
>>>>>>>>>> On Mon, Aug 16, 2021 at 9:57 AM Luke Cwik <lc...@google.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> The issue impacting Dataflow should be resolved. We can see if
>>>>>>>>>>> the precommits stabilize by Wednesday. Udi before cutting the first release
>>>>>>>>>>> can you post what is the status of the testing to the group and if it looks
>>>>>>>>>>> green then we can go ahead (assuming no blockers)?
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Aug 13, 2021 at 5:47 PM Andrew Pilloud <
>>>>>>>>>>> apilloud@google.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks for volunteering!
>>>>>>>>>>>>
>>>>>>>>>>>> I'm going to object to the 2.33.0 release being cut until we
>>>>>>>>>>>> stabilize our precommits.
>>>>>>>>>>>>
>>>>>>>>>>>> Both java and python precommits are reporting the last
>>>>>>>>>>>> successful run being in July.
>>>>>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/
>>>>>>>>>>>>
>>>>>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Aug 11, 2021 at 10:36 AM Udi Meiri <eh...@google.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi everyone!
>>>>>>>>>>>>>
>>>>>>>>>>>>> Beam 2.33.0 release is scheduled to be cut on 2021-08-25
>>>>>>>>>>>>> (Wednesday) and released by 2021-09-29 according to the release calendar
>>>>>>>>>>>>> [1].
>>>>>>>>>>>>>
>>>>>>>>>>>>> If there are no objections then I would like to nominate
>>>>>>>>>>>>> myself to do this release.
>>>>>>>>>>>>> Please review your bug list and mark the blockers as blocker
>>>>>>>>>>>>> for the release and move non blockers to the next release by 2021-08-25.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please let me know if you have any questions.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Udi
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1]
>>>>>>>>>>>>> https://calendar.google.com/calendar/u/0/embed?src=0p73sl034k80oob7seouanigd0@group.calendar.google.com&ctz=America/Los_Angeles
>>>>>>>>>>>>>
>>>>>>>>>>>>
>
Re: [PROPOSAL] Preparing for Beam 2.33.0 Release
Posted by Ahmet Altay <al...@google.com>.
On Tue, Sep 14, 2021 at 12:52 PM Udi Meiri <eh...@google.com> wrote:
> I'll consider the 2 remaining issues as closed unless told otherwise, and
> start the RC process.
>
+1 to starting the RC process now.
> If I don't hear any objections I'll mark these issues as fixed in a week
> (2021-09-21).
>
> On Mon, Sep 13, 2021 at 4:57 PM Ahmet Altay <al...@google.com> wrote:
>
>> Nice and thank you for the update Udi!
>>
>> It looks like there are only 2 issues left, adding the assignees here
>> explicitly in case they miss the jira notifications:
>> https://issues.apache.org/jira/browse/BEAM-12628 - +Claire, could this
>> be closed? IIRC it was fixed (https://github.com/apache/beam/pull/15292)
>> and the issue was open to cherry pick into the previous release but that
>> did not happen; and for this release (2.33.0) it should be already fixed.
>> https://issues.apache.org/jira/browse/BEAM-12781 - +Pablo Estrada
>> <pa...@google.com> should this be a blocker for 2.33.0 release? It
>> looks like it is already fixed (https://github.com/apache/beam/pull/15357
>> )
>>
>> On Fri, Sep 10, 2021 at 3:51 PM Udi Meiri <eh...@google.com> wrote:
>>
>>> The release branch is now passing Jenkins tests:
>>> https://github.com/apache/beam/pull/15431
>>>
>>> Thank you all so far for help with release blockers. There are 5 left
>>> <https://issues.apache.org/jira/projects/BEAM/versions/12350404>.
>>>
>>> On Thu, Aug 26, 2021 at 7:54 AM Alexey Romanenko <
>>> aromanenko.dev@gmail.com> wrote:
>>>
>>>> See my comment below.
>>>>
>>>> On 26 Aug 2021, at 03:44, Ahmet Altay <al...@google.com> wrote:
>>>>
>>>> On Wed, Aug 25, 2021 at 1:50 PM Udi Meiri <eh...@google.com> wrote:
>>>>
>>>>> I've found 2 performance regressions in IO. Could someone triage these?
>>>>>
>>>>
>>>>
>>>>> https://issues.apache.org/jira/browse/BEAM-12801
>>>>>
>>>>
>>>> @Alexey Romanenko <ar...@gmail.com> @Tim Robertson
>>>> <ti...@gmail.com> - Could you comment on this? It is related
>>>> to HadoopFormatIO.
>>>>
>>>>
>>>>
>>>> I don’t see any code changes for “sdks/java/io/hadoop-format” package
>>>> since June 2nd while the Write performance dropped at Aug 10th for the
>>>> first time.
>>>> I’ll try to reproduce it locally with a direct runner but seems it was
>>>> caused by something else. Any other ideas about this?
>>>>
>>>>
>>>>
>>>>> https://issues.apache.org/jira/browse/BEAM-12800
>>>>>
>>>>
>>>> @Chamikara Jayalath <ch...@google.com> @Yifan Mai
>>>> <yi...@google.com> - Do you know who could comment on this?
>>>>
>>>>
>>>>>
>>>>>
>>>>> +14 other blockers currently
>>>>>
>>>>
>>>> I have not looked at the other blockers. It might help to check with
>>>> the assignees. They may not know that they are release blockers.
>>>>
>>>>
>>>>>
>>>>> On Wed, Aug 25, 2021 at 12:22 PM Udi Meiri <eh...@google.com> wrote:
>>>>>
>>>>>> I'll try cutting the release today. From what I can tell [1], the
>>>>>> only perma-reds are python postcommits.
>>>>>>
>>>>>
>>>> I believe this is fixed now. (Thanks to @Kyle Weaver
>>>> <kc...@google.com>)
>>>>
>>>>
>>>>>
>>>>>> [1]
>>>>>> https://github.com/apache/beam/blob/master/.github/PULL_REQUEST_TEMPLATE.md
>>>>>>
>>>>>> On Tue, Aug 17, 2021 at 10:12 AM Udi Meiri <eh...@google.com> wrote:
>>>>>>
>>>>>>> Please mark 2.33.0 release blockers by setting the "Fix version"
>>>>>>> field to 2.33.0. Otherwise, it doesn't appear in the dashboard
>>>>>>> <https://issues.apache.org/jira/projects/BEAM/versions/12350404>.
>>>>>>>
>>>>>>> On Tue, Aug 17, 2021 at 9:32 AM Luke Cwik <lc...@google.com> wrote:
>>>>>>>
>>>>>>>> +Rui Wang <ru...@google.com> as the current build monitor.
>>>>>>>>
>>>>>>>> At the time I did not mean to add any additional tests as it was a
>>>>>>>> copy paste error.
>>>>>>>>
>>>>>>>> I do have another fix for the failing Python PreCommit which I
>>>>>>>> fixed since it seemed like a trivial one:
>>>>>>>> https://issues.apache.org/jira/browse/BEAM-12768
>>>>>>>> - test_df_agg_method_invalid_kwarg_raises too strict on error checking. fix
>>>>>>>> in https://github.com/apache/beam/pull/15341
>>>>>>>>
>>>>>>>> https://issues.apache.org/jira/browse/BEAM-12733 (RecommendationAIIT
>>>>>>>> Java Version)
>>>>>>>> +Matthias Baetens <ba...@gmail.com> for the above issue.
>>>>>>>>
>>>>>>>> On Mon, Aug 16, 2021 at 6:54 PM Ahmet Altay <al...@google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thank you Udi and +1 to the release cut on schedule.
>>>>>>>>>
>>>>>>>>> On Mon, Aug 16, 2021 at 3:53 PM Luke Cwik <lc...@google.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> It seems as though several tests were broken due to commits
>>>>>>>>>> during the issue with some existing flakes. It looks like the Python
>>>>>>>>>> PostCommit tests are perma-red due to these failures. So far I'm tracking:
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thank you very much Luke. Adding bug owners here to
>>>>>>>>> explicitly call out release blocking issues.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-12764 (new_block
>>>>>>>>>> method missing in pandas)
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> +Sam Rohde <sr...@google.com> for the above issue.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-12683
>>>>>>>>>> (RecommendationAIIT)
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> +Matthias Baetens <ba...@gmail.com> for the above issue.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-12765 (cannot access
>>>>>>>>>> field fruit)
>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-12766 (Already
>>>>>>>>>> Exists: Dataset apache-beam-testing:python_bq_file_loads_)
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> +Pablo Estrada <pa...@google.com> for the two issues above.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-12694
>>>>>>>>>> (DICOMIoIntegrationTest)
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> +Frank Yang <xi...@google.com> +Dave Anderson
>>>>>>>>> <dr...@google.com> for the DICOMIO. Could one of you look at
>>>>>>>>> this? Looks like it might be a simple fix.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-12766 (BQ Already
>>>>>>>>>> exists Dataset)
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Luke, did you mean link to another issue? This one is a duplicate.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Aug 16, 2021 at 10:45 AM Luke Cwik <lc...@google.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> There are the ones that I believe went perma-red due to the
>>>>>>>>>>> issue:
>>>>>>>>>>> beam_PostCommit_Python36
>>>>>>>>>>> beam_PostCommit_Python37
>>>>>>>>>>> beam_PostCommit_Python38
>>>>>>>>>>> beam_PreCommit_Python_Cron
>>>>>>>>>>> beam_PostCommit_Java
>>>>>>>>>>> beam_PostCommit_Java_DataflowV2
>>>>>>>>>>> beam_PreCommit_Java_Examples_Dataflow_Cron
>>>>>>>>>>> beam_PostRelease_NightlySnapshot
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Aug 16, 2021 at 10:42 AM Udi Meiri <eh...@google.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Sure, I'll verify the status of the tests in our PR template.
>>>>>>>>>>>> Any others I should be aware of?
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Aug 16, 2021 at 10:15 AM Andrew Pilloud <
>>>>>>>>>>>> apilloud@google.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> That sounds reasonable to me. A week should give everyone
>>>>>>>>>>>>> enough time to get their changes in.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Aug 16, 2021 at 9:57 AM Luke Cwik <lc...@google.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> The issue impacting Dataflow should be resolved. We can see
>>>>>>>>>>>>>> if the precommits stabilize by Wednesday. Udi before cutting the first
>>>>>>>>>>>>>> release can you post what is the status of the testing to the group and if
>>>>>>>>>>>>>> it looks green then we can go ahead (assuming no blockers)?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Aug 13, 2021 at 5:47 PM Andrew Pilloud <
>>>>>>>>>>>>>> apilloud@google.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for volunteering!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm going to object to the 2.33.0 release being cut until we
>>>>>>>>>>>>>>> stabilize our precommits.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Both java and python precommits are reporting the last
>>>>>>>>>>>>>>> successful run being in July.
>>>>>>>>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Aug 11, 2021 at 10:36 AM Udi Meiri <eh...@google.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi everyone!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Beam 2.33.0 release is scheduled to be cut on 2021-08-25
>>>>>>>>>>>>>>>> (Wednesday) and released by 2021-09-29 according to the release calendar
>>>>>>>>>>>>>>>> [1].
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If there are no objections then I would like to nominate
>>>>>>>>>>>>>>>> myself to do this release.
>>>>>>>>>>>>>>>> Please review your bug list and mark the blockers as
>>>>>>>>>>>>>>>> blocker for the release and move non blockers to the next release by
>>>>>>>>>>>>>>>> 2021-08-25.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Please let me know if you have any questions.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Udi
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>> https://calendar.google.com/calendar/u/0/embed?src=0p73sl034k80oob7seouanigd0@group.calendar.google.com&ctz=America/Los_Angeles
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>
Re: [PROPOSAL] Preparing for Beam 2.33.0 Release
Posted by Udi Meiri <eh...@google.com>.
I'll consider the 2 remaining issues as closed unless told otherwise, and
start the RC process.
If I don't hear any objections I'll mark these issues as fixed in a week
(2021-09-21).
On Mon, Sep 13, 2021 at 4:57 PM Ahmet Altay <al...@google.com> wrote:
> Nice and thank you for the update Udi!
>
> It looks like there are only 2 issues left, adding the assignees here
> explicitly in case they miss the jira notifications:
> https://issues.apache.org/jira/browse/BEAM-12628 - +Claire, could this be
> closed? IIRC it was fixed (https://github.com/apache/beam/pull/15292) and
> the issue was open to cherry pick into the previous release but that did
> not happen; and for this release (2.33.0) it should be already fixed.
> https://issues.apache.org/jira/browse/BEAM-12781 - +Pablo Estrada
> <pa...@google.com> should this be a blocker for 2.33.0 release? It
> looks like it is already fixed (https://github.com/apache/beam/pull/15357)
>
> On Fri, Sep 10, 2021 at 3:51 PM Udi Meiri <eh...@google.com> wrote:
>
>> The release branch is now passing Jenkins tests:
>> https://github.com/apache/beam/pull/15431
>>
>> Thank you all so far for help with release blockers. There are 5 left
>> <https://issues.apache.org/jira/projects/BEAM/versions/12350404>.
>>
>> On Thu, Aug 26, 2021 at 7:54 AM Alexey Romanenko <
>> aromanenko.dev@gmail.com> wrote:
>>
>>> See my comment below.
>>>
>>> On 26 Aug 2021, at 03:44, Ahmet Altay <al...@google.com> wrote:
>>>
>>> On Wed, Aug 25, 2021 at 1:50 PM Udi Meiri <eh...@google.com> wrote:
>>>
>>>> I've found 2 performance regressions in IO. Could someone triage these?
>>>>
>>>
>>>
>>>> https://issues.apache.org/jira/browse/BEAM-12801
>>>>
>>>
>>> @Alexey Romanenko <ar...@gmail.com> @Tim Robertson
>>> <ti...@gmail.com> - Could you comment on this? It is related
>>> to HadoopFormatIO.
>>>
>>>
>>>
>>> I don’t see any code changes for “sdks/java/io/hadoop-format” package
>>> since June 2nd while the Write performance dropped at Aug 10th for the
>>> first time.
>>> I’ll try to reproduce it locally with a direct runner but seems it was
>>> caused by something else. Any other ideas about this?
>>>
>>>
>>>
>>>> https://issues.apache.org/jira/browse/BEAM-12800
>>>>
>>>
>>> @Chamikara Jayalath <ch...@google.com> @Yifan Mai
>>> <yi...@google.com> - Do you know who could comment on this?
>>>
>>>
>>>>
>>>>
>>>> +14 other blockers currently
>>>>
>>>
>>> I have not looked at the other blockers. It might help to check with the
>>> assignees. They may not know that they are release blockers.
>>>
>>>
>>>>
>>>> On Wed, Aug 25, 2021 at 12:22 PM Udi Meiri <eh...@google.com> wrote:
>>>>
>>>>> I'll try cutting the release today. From what I can tell [1], the only
>>>>> perma-reds are python postcommits.
>>>>>
>>>>
>>> I believe this is fixed now. (Thanks to @Kyle Weaver
>>> <kc...@google.com>)
>>>
>>>
>>>>
>>>>> [1]
>>>>> https://github.com/apache/beam/blob/master/.github/PULL_REQUEST_TEMPLATE.md
>>>>>
>>>>> On Tue, Aug 17, 2021 at 10:12 AM Udi Meiri <eh...@google.com> wrote:
>>>>>
>>>>>> Please mark 2.33.0 release blockers by setting the "Fix version"
>>>>>> field to 2.33.0. Otherwise, it doesn't appear in the dashboard
>>>>>> <https://issues.apache.org/jira/projects/BEAM/versions/12350404>.
>>>>>>
>>>>>> On Tue, Aug 17, 2021 at 9:32 AM Luke Cwik <lc...@google.com> wrote:
>>>>>>
>>>>>>> +Rui Wang <ru...@google.com> as the current build monitor.
>>>>>>>
>>>>>>> At the time I did not mean to add any additional tests as it was a
>>>>>>> copy paste error.
>>>>>>>
>>>>>>> I do have another fix for the failing Python PreCommit which I fixed
>>>>>>> since it seemed like a trivial one:
>>>>>>> https://issues.apache.org/jira/browse/BEAM-12768
>>>>>>> - test_df_agg_method_invalid_kwarg_raises too strict on error checking. fix
>>>>>>> in https://github.com/apache/beam/pull/15341
>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/BEAM-12733 (RecommendationAIIT
>>>>>>> Java Version)
>>>>>>> +Matthias Baetens <ba...@gmail.com> for the above issue.
>>>>>>>
>>>>>>> On Mon, Aug 16, 2021 at 6:54 PM Ahmet Altay <al...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thank you Udi and +1 to the release cut on schedule.
>>>>>>>>
>>>>>>>> On Mon, Aug 16, 2021 at 3:53 PM Luke Cwik <lc...@google.com> wrote:
>>>>>>>>
>>>>>>>>> It seems as though several tests were broken due to commits during
>>>>>>>>> the issue with some existing flakes. It looks like the Python PostCommit
>>>>>>>>> tests are perma-red due to these failures. So far I'm tracking:
>>>>>>>>>
>>>>>>>>
>>>>>>>> Thank you very much Luke. Adding bug owners here to explicitly call
>>>>>>>> out release blocking issues.
>>>>>>>>
>>>>>>>>
>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-12764 (new_block
>>>>>>>>> method missing in pandas)
>>>>>>>>>
>>>>>>>>
>>>>>>>> +Sam Rohde <sr...@google.com> for the above issue.
>>>>>>>>
>>>>>>>>
>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-12683
>>>>>>>>> (RecommendationAIIT)
>>>>>>>>>
>>>>>>>>
>>>>>>>> +Matthias Baetens <ba...@gmail.com> for the above issue.
>>>>>>>>
>>>>>>>>
>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-12765 (cannot access
>>>>>>>>> field fruit)
>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-12766 (Already Exists:
>>>>>>>>> Dataset apache-beam-testing:python_bq_file_loads_)
>>>>>>>>>
>>>>>>>>
>>>>>>>> +Pablo Estrada <pa...@google.com> for the two issues above.
>>>>>>>>
>>>>>>>>
>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-12694
>>>>>>>>> (DICOMIoIntegrationTest)
>>>>>>>>>
>>>>>>>>
>>>>>>>> +Frank Yang <xi...@google.com> +Dave Anderson
>>>>>>>> <dr...@google.com> for the DICOMIO. Could one of you look at
>>>>>>>> this? Looks like it might be a simple fix.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-12766 (BQ Already
>>>>>>>>> exists Dataset)
>>>>>>>>>
>>>>>>>>
>>>>>>>> Luke, did you mean link to another issue? This one is a duplicate.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Aug 16, 2021 at 10:45 AM Luke Cwik <lc...@google.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> There are the ones that I believe went perma-red due to the issue:
>>>>>>>>>> beam_PostCommit_Python36
>>>>>>>>>> beam_PostCommit_Python37
>>>>>>>>>> beam_PostCommit_Python38
>>>>>>>>>> beam_PreCommit_Python_Cron
>>>>>>>>>> beam_PostCommit_Java
>>>>>>>>>> beam_PostCommit_Java_DataflowV2
>>>>>>>>>> beam_PreCommit_Java_Examples_Dataflow_Cron
>>>>>>>>>> beam_PostRelease_NightlySnapshot
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Aug 16, 2021 at 10:42 AM Udi Meiri <eh...@google.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Sure, I'll verify the status of the tests in our PR template.
>>>>>>>>>>> Any others I should be aware of?
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Aug 16, 2021 at 10:15 AM Andrew Pilloud <
>>>>>>>>>>> apilloud@google.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> That sounds reasonable to me. A week should give everyone
>>>>>>>>>>>> enough time to get their changes in.
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Aug 16, 2021 at 9:57 AM Luke Cwik <lc...@google.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> The issue impacting Dataflow should be resolved. We can see if
>>>>>>>>>>>>> the precommits stabilize by Wednesday. Udi before cutting the first release
>>>>>>>>>>>>> can you post what is the status of the testing to the group and if it looks
>>>>>>>>>>>>> green then we can go ahead (assuming no blockers)?
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Aug 13, 2021 at 5:47 PM Andrew Pilloud <
>>>>>>>>>>>>> apilloud@google.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for volunteering!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm going to object to the 2.33.0 release being cut until we
>>>>>>>>>>>>>> stabilize our precommits.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Both java and python precommits are reporting the last
>>>>>>>>>>>>>> successful run being in July.
>>>>>>>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Aug 11, 2021 at 10:36 AM Udi Meiri <eh...@google.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi everyone!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Beam 2.33.0 release is scheduled to be cut on 2021-08-25
>>>>>>>>>>>>>>> (Wednesday) and released by 2021-09-29 according to the release calendar
>>>>>>>>>>>>>>> [1].
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If there are no objections then I would like to nominate
>>>>>>>>>>>>>>> myself to do this release.
>>>>>>>>>>>>>>> Please review your bug list and mark the blockers as blocker
>>>>>>>>>>>>>>> for the release and move non blockers to the next release by 2021-08-25.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Please let me know if you have any questions.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Udi
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>> https://calendar.google.com/calendar/u/0/embed?src=0p73sl034k80oob7seouanigd0@group.calendar.google.com&ctz=America/Los_Angeles
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>
Re: [PROPOSAL] Preparing for Beam 2.33.0 Release
Posted by Ahmet Altay <al...@google.com>.
Nice and thank you for the update Udi!
It looks like there are only 2 issues left, adding the assignees here
explicitly in case they miss the jira notifications:
https://issues.apache.org/jira/browse/BEAM-12628 - +Claire, could this be
closed? IIRC it was fixed (https://github.com/apache/beam/pull/15292) and
the issue was open to cherry pick into the previous release but that did
not happen; and for this release (2.33.0) it should be already fixed.
https://issues.apache.org/jira/browse/BEAM-12781 - +Pablo Estrada
<pa...@google.com> should this be a blocker for 2.33.0 release? It looks
like it is already fixed (https://github.com/apache/beam/pull/15357)
On Fri, Sep 10, 2021 at 3:51 PM Udi Meiri <eh...@google.com> wrote:
> The release branch is now passing Jenkins tests:
> https://github.com/apache/beam/pull/15431
>
> Thank you all so far for help with release blockers. There are 5 left
> <https://issues.apache.org/jira/projects/BEAM/versions/12350404>.
>
> On Thu, Aug 26, 2021 at 7:54 AM Alexey Romanenko <ar...@gmail.com>
> wrote:
>
>> See my comment below.
>>
>> On 26 Aug 2021, at 03:44, Ahmet Altay <al...@google.com> wrote:
>>
>> On Wed, Aug 25, 2021 at 1:50 PM Udi Meiri <eh...@google.com> wrote:
>>
>>> I've found 2 performance regressions in IO. Could someone triage these?
>>>
>>
>>
>>> https://issues.apache.org/jira/browse/BEAM-12801
>>>
>>
>> @Alexey Romanenko <ar...@gmail.com> @Tim Robertson
>> <ti...@gmail.com> - Could you comment on this? It is related
>> to HadoopFormatIO.
>>
>>
>>
>> I don’t see any code changes for “sdks/java/io/hadoop-format” package
>> since June 2nd while the Write performance dropped at Aug 10th for the
>> first time.
>> I’ll try to reproduce it locally with a direct runner but seems it was
>> caused by something else. Any other ideas about this?
>>
>>
>>
>>> https://issues.apache.org/jira/browse/BEAM-12800
>>>
>>
>> @Chamikara Jayalath <ch...@google.com> @Yifan Mai
>> <yi...@google.com> - Do you know who could comment on this?
>>
>>
>>>
>>>
>>> +14 other blockers currently
>>>
>>
>> I have not looked at the other blockers. It might help to check with the
>> assignees. They may not know that they are release blockers.
>>
>>
>>>
>>> On Wed, Aug 25, 2021 at 12:22 PM Udi Meiri <eh...@google.com> wrote:
>>>
>>>> I'll try cutting the release today. From what I can tell [1], the only
>>>> perma-reds are python postcommits.
>>>>
>>>
>> I believe this is fixed now. (Thanks to @Kyle Weaver
>> <kc...@google.com>)
>>
>>
>>>
>>>> [1]
>>>> https://github.com/apache/beam/blob/master/.github/PULL_REQUEST_TEMPLATE.md
>>>>
>>>> On Tue, Aug 17, 2021 at 10:12 AM Udi Meiri <eh...@google.com> wrote:
>>>>
>>>>> Please mark 2.33.0 release blockers by setting the "Fix version" field
>>>>> to 2.33.0. Otherwise, it doesn't appear in the dashboard
>>>>> <https://issues.apache.org/jira/projects/BEAM/versions/12350404>.
>>>>>
>>>>> On Tue, Aug 17, 2021 at 9:32 AM Luke Cwik <lc...@google.com> wrote:
>>>>>
>>>>>> +Rui Wang <ru...@google.com> as the current build monitor.
>>>>>>
>>>>>> At the time I did not mean to add any additional tests as it was a
>>>>>> copy paste error.
>>>>>>
>>>>>> I do have another fix for the failing Python PreCommit which I fixed
>>>>>> since it seemed like a trivial one:
>>>>>> https://issues.apache.org/jira/browse/BEAM-12768
>>>>>> - test_df_agg_method_invalid_kwarg_raises too strict on error checking. fix
>>>>>> in https://github.com/apache/beam/pull/15341
>>>>>>
>>>>>> https://issues.apache.org/jira/browse/BEAM-12733 (RecommendationAIIT
>>>>>> Java Version)
>>>>>> +Matthias Baetens <ba...@gmail.com> for the above issue.
>>>>>>
>>>>>> On Mon, Aug 16, 2021 at 6:54 PM Ahmet Altay <al...@google.com> wrote:
>>>>>>
>>>>>>> Thank you Udi and +1 to the release cut on schedule.
>>>>>>>
>>>>>>> On Mon, Aug 16, 2021 at 3:53 PM Luke Cwik <lc...@google.com> wrote:
>>>>>>>
>>>>>>>> It seems as though several tests were broken due to commits during
>>>>>>>> the issue with some existing flakes. It looks like the Python PostCommit
>>>>>>>> tests are perma-red due to these failures. So far I'm tracking:
>>>>>>>>
>>>>>>>
>>>>>>> Thank you very much Luke. Adding bug owners here to explicitly call
>>>>>>> out release blocking issues.
>>>>>>>
>>>>>>>
>>>>>>>> https://issues.apache.org/jira/browse/BEAM-12764 (new_block method
>>>>>>>> missing in pandas)
>>>>>>>>
>>>>>>>
>>>>>>> +Sam Rohde <sr...@google.com> for the above issue.
>>>>>>>
>>>>>>>
>>>>>>>> https://issues.apache.org/jira/browse/BEAM-12683
>>>>>>>> (RecommendationAIIT)
>>>>>>>>
>>>>>>>
>>>>>>> +Matthias Baetens <ba...@gmail.com> for the above issue.
>>>>>>>
>>>>>>>
>>>>>>>> https://issues.apache.org/jira/browse/BEAM-12765 (cannot access
>>>>>>>> field fruit)
>>>>>>>> https://issues.apache.org/jira/browse/BEAM-12766 (Already Exists:
>>>>>>>> Dataset apache-beam-testing:python_bq_file_loads_)
>>>>>>>>
>>>>>>>
>>>>>>> +Pablo Estrada <pa...@google.com> for the two issues above.
>>>>>>>
>>>>>>>
>>>>>>>> https://issues.apache.org/jira/browse/BEAM-12694
>>>>>>>> (DICOMIoIntegrationTest)
>>>>>>>>
>>>>>>>
>>>>>>> +Frank Yang <xi...@google.com> +Dave Anderson
>>>>>>> <dr...@google.com> for the DICOMIO. Could one of you look at
>>>>>>> this? Looks like it might be a simple fix.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> https://issues.apache.org/jira/browse/BEAM-12766 (BQ Already
>>>>>>>> exists Dataset)
>>>>>>>>
>>>>>>>
>>>>>>> Luke, did you mean link to another issue? This one is a duplicate.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Aug 16, 2021 at 10:45 AM Luke Cwik <lc...@google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> There are the ones that I believe went perma-red due to the issue:
>>>>>>>>> beam_PostCommit_Python36
>>>>>>>>> beam_PostCommit_Python37
>>>>>>>>> beam_PostCommit_Python38
>>>>>>>>> beam_PreCommit_Python_Cron
>>>>>>>>> beam_PostCommit_Java
>>>>>>>>> beam_PostCommit_Java_DataflowV2
>>>>>>>>> beam_PreCommit_Java_Examples_Dataflow_Cron
>>>>>>>>> beam_PostRelease_NightlySnapshot
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Aug 16, 2021 at 10:42 AM Udi Meiri <eh...@google.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Sure, I'll verify the status of the tests in our PR template. Any
>>>>>>>>>> others I should be aware of?
>>>>>>>>>>
>>>>>>>>>> On Mon, Aug 16, 2021 at 10:15 AM Andrew Pilloud <
>>>>>>>>>> apilloud@google.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> That sounds reasonable to me. A week should give everyone enough
>>>>>>>>>>> time to get their changes in.
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Aug 16, 2021 at 9:57 AM Luke Cwik <lc...@google.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> The issue impacting Dataflow should be resolved. We can see if
>>>>>>>>>>>> the precommits stabilize by Wednesday. Udi before cutting the first release
>>>>>>>>>>>> can you post what is the status of the testing to the group and if it looks
>>>>>>>>>>>> green then we can go ahead (assuming no blockers)?
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Aug 13, 2021 at 5:47 PM Andrew Pilloud <
>>>>>>>>>>>> apilloud@google.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for volunteering!
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm going to object to the 2.33.0 release being cut until we
>>>>>>>>>>>>> stabilize our precommits.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Both java and python precommits are reporting the last
>>>>>>>>>>>>> successful run being in July.
>>>>>>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Aug 11, 2021 at 10:36 AM Udi Meiri <eh...@google.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi everyone!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Beam 2.33.0 release is scheduled to be cut on 2021-08-25
>>>>>>>>>>>>>> (Wednesday) and released by 2021-09-29 according to the release calendar
>>>>>>>>>>>>>> [1].
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If there are no objections then I would like to nominate
>>>>>>>>>>>>>> myself to do this release.
>>>>>>>>>>>>>> Please review your bug list and mark the blockers as blocker
>>>>>>>>>>>>>> for the release and move non blockers to the next release by 2021-08-25.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Please let me know if you have any questions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Udi
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>> https://calendar.google.com/calendar/u/0/embed?src=0p73sl034k80oob7seouanigd0@group.calendar.google.com&ctz=America/Los_Angeles
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>