You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Mark Liu <ma...@google.com> on 2018/11/29 16:36:22 UTC
What is Jenkins job "Portable_Python" in PreCommit?
Hi guys,
I made some changes <https://github.com/apache/beam/pull/7146> to Python
PreCommit Gradle and then Portable_Python is invoked as a PrecCommit test
and failed. However, I can't find where it's defined / generated in Gradle
or Jenkins groovy. Does anyone know? My branch is synced to master
yesterday.
Thanks!
Mark
Re: What is Jenkins job "Portable_Python" in PreCommit?
Posted by Ankur Goenka <go...@google.com>.
Makes sense.
I would like to keep this a separate precommit for a few weeks to make sure
that it gets enough bake time and its not flaky.
We should merge it with ":pythonPreCommit" in a few weeks. This takes 5 min
on avg so it should not add to the overall time for precommit results.
On Fri, Nov 30, 2018 at 2:09 PM Mark Liu <ma...@google.com> wrote:
> Thank you for improving the test coverage. That's very helpful!
>
> On the other hand, I have a PR <https://github.com/apache/beam/pull/7163> in
> review to make time-consuming integration tests to run parallel in
> PreCommit. This could benefits tests like portable_python to be enabled in
> `:pythonPreCommit` without creating a Jenkins branch and run seed job for
> testing. However, if we have other reasons, then it may not applied.
>
> Would love to hear thoughts on this.
>
> Mark
>
> On Fri, Nov 30, 2018 at 12:27 PM Thomas Weise <th...@apache.org> wrote:
>
>> This is a very valuable addition! Given the execution times of Python and
>> Java pre-commit, having this run in parallel should not be an issue.
>>
>> Thanks,
>> Thomas
>>
>>
>> On Fri, Nov 30, 2018 at 11:01 AM Ankur Goenka <go...@google.com> wrote:
>>
>>> We added new precommit which tests a wordcount pipeline on portable
>>> flink runner.
>>> In recent time we missed catching some obvious issues which broke
>>> potability and could have been caught by this test.
>>> The current test is fairly light weight and executes in ~5min which
>>> seems to be reasonable for a precommit test.
>>> This precommit is added just yesterday so we don't actively track it
>>> yet. However please let me know if you see any issues with this precommit.
>>>
>>> On Fri, Nov 30, 2018 at 3:49 AM Maximilian Michels <mx...@apache.org>
>>> wrote:
>>>
>>>> This was merged with https://github.com/apache/beam/pull/6954.
>>>>
>>>> Eventually we want to run a portable WordCount on PreCommit. We will do
>>>> some more testing on Jenkins before it becomes an official PreCommit
>>>> task.
>>>>
>>>> Thanks,
>>>> Max
>>>>
>>>> On 29.11.18 19:03, Mark Liu wrote:
>>>> > ah, thanks Boyuan! Probably I created the PR in a bad timing. Looks
>>>> like
>>>> > a new PR will fix it.
>>>> >
>>>> > On Thu, Nov 29, 2018 at 9:50 AM Boyuan Zhang <boyuanz@google.com
>>>> > <ma...@google.com>> wrote:
>>>> >
>>>> > I believe it's in this pending PR:
>>>> > https://github.com/apache/beam/pull/7157.
>>>> >
>>>> > On Thu, Nov 29, 2018 at 8:36 AM Mark Liu <markliu@google.com
>>>> > <ma...@google.com>> wrote:
>>>> >
>>>> > Hi guys,
>>>> >
>>>> > I made some changes <https://github.com/apache/beam/pull/7146
>>>> >
>>>> > to Python PreCommit Gradle and then Portable_Python is invoked
>>>> > as a PrecCommit test and failed. However, I can't find where
>>>> > it's defined / generated in Gradle or Jenkins groovy. Does
>>>> > anyone know? My branch is synced to master yesterday.
>>>> >
>>>> > Thanks!
>>>> > Mark
>>>> >
>>>>
>>>
Re: What is Jenkins job "Portable_Python" in PreCommit?
Posted by Mark Liu <ma...@google.com>.
Thank you for improving the test coverage. That's very helpful!
On the other hand, I have a PR <https://github.com/apache/beam/pull/7163> in
review to make time-consuming integration tests to run parallel in
PreCommit. This could benefits tests like portable_python to be enabled in
`:pythonPreCommit` without creating a Jenkins branch and run seed job for
testing. However, if we have other reasons, then it may not applied.
Would love to hear thoughts on this.
Mark
On Fri, Nov 30, 2018 at 12:27 PM Thomas Weise <th...@apache.org> wrote:
> This is a very valuable addition! Given the execution times of Python and
> Java pre-commit, having this run in parallel should not be an issue.
>
> Thanks,
> Thomas
>
>
> On Fri, Nov 30, 2018 at 11:01 AM Ankur Goenka <go...@google.com> wrote:
>
>> We added new precommit which tests a wordcount pipeline on portable flink
>> runner.
>> In recent time we missed catching some obvious issues which broke
>> potability and could have been caught by this test.
>> The current test is fairly light weight and executes in ~5min which seems
>> to be reasonable for a precommit test.
>> This precommit is added just yesterday so we don't actively track it yet.
>> However please let me know if you see any issues with this precommit.
>>
>> On Fri, Nov 30, 2018 at 3:49 AM Maximilian Michels <mx...@apache.org>
>> wrote:
>>
>>> This was merged with https://github.com/apache/beam/pull/6954.
>>>
>>> Eventually we want to run a portable WordCount on PreCommit. We will do
>>> some more testing on Jenkins before it becomes an official PreCommit
>>> task.
>>>
>>> Thanks,
>>> Max
>>>
>>> On 29.11.18 19:03, Mark Liu wrote:
>>> > ah, thanks Boyuan! Probably I created the PR in a bad timing. Looks
>>> like
>>> > a new PR will fix it.
>>> >
>>> > On Thu, Nov 29, 2018 at 9:50 AM Boyuan Zhang <boyuanz@google.com
>>> > <ma...@google.com>> wrote:
>>> >
>>> > I believe it's in this pending PR:
>>> > https://github.com/apache/beam/pull/7157.
>>> >
>>> > On Thu, Nov 29, 2018 at 8:36 AM Mark Liu <markliu@google.com
>>> > <ma...@google.com>> wrote:
>>> >
>>> > Hi guys,
>>> >
>>> > I made some changes <https://github.com/apache/beam/pull/7146>
>>> > to Python PreCommit Gradle and then Portable_Python is invoked
>>> > as a PrecCommit test and failed. However, I can't find where
>>> > it's defined / generated in Gradle or Jenkins groovy. Does
>>> > anyone know? My branch is synced to master yesterday.
>>> >
>>> > Thanks!
>>> > Mark
>>> >
>>>
>>
Re: What is Jenkins job "Portable_Python" in PreCommit?
Posted by Thomas Weise <th...@apache.org>.
This is a very valuable addition! Given the execution times of Python and
Java pre-commit, having this run in parallel should not be an issue.
Thanks,
Thomas
On Fri, Nov 30, 2018 at 11:01 AM Ankur Goenka <go...@google.com> wrote:
> We added new precommit which tests a wordcount pipeline on portable flink
> runner.
> In recent time we missed catching some obvious issues which broke
> potability and could have been caught by this test.
> The current test is fairly light weight and executes in ~5min which seems
> to be reasonable for a precommit test.
> This precommit is added just yesterday so we don't actively track it yet.
> However please let me know if you see any issues with this precommit.
>
> On Fri, Nov 30, 2018 at 3:49 AM Maximilian Michels <mx...@apache.org> wrote:
>
>> This was merged with https://github.com/apache/beam/pull/6954.
>>
>> Eventually we want to run a portable WordCount on PreCommit. We will do
>> some more testing on Jenkins before it becomes an official PreCommit task.
>>
>> Thanks,
>> Max
>>
>> On 29.11.18 19:03, Mark Liu wrote:
>> > ah, thanks Boyuan! Probably I created the PR in a bad timing. Looks
>> like
>> > a new PR will fix it.
>> >
>> > On Thu, Nov 29, 2018 at 9:50 AM Boyuan Zhang <boyuanz@google.com
>> > <ma...@google.com>> wrote:
>> >
>> > I believe it's in this pending PR:
>> > https://github.com/apache/beam/pull/7157.
>> >
>> > On Thu, Nov 29, 2018 at 8:36 AM Mark Liu <markliu@google.com
>> > <ma...@google.com>> wrote:
>> >
>> > Hi guys,
>> >
>> > I made some changes <https://github.com/apache/beam/pull/7146>
>> > to Python PreCommit Gradle and then Portable_Python is invoked
>> > as a PrecCommit test and failed. However, I can't find where
>> > it's defined / generated in Gradle or Jenkins groovy. Does
>> > anyone know? My branch is synced to master yesterday.
>> >
>> > Thanks!
>> > Mark
>> >
>>
>
Re: What is Jenkins job "Portable_Python" in PreCommit?
Posted by Ankur Goenka <go...@google.com>.
We added new precommit which tests a wordcount pipeline on portable flink
runner.
In recent time we missed catching some obvious issues which broke
potability and could have been caught by this test.
The current test is fairly light weight and executes in ~5min which seems
to be reasonable for a precommit test.
This precommit is added just yesterday so we don't actively track it yet.
However please let me know if you see any issues with this precommit.
On Fri, Nov 30, 2018 at 3:49 AM Maximilian Michels <mx...@apache.org> wrote:
> This was merged with https://github.com/apache/beam/pull/6954.
>
> Eventually we want to run a portable WordCount on PreCommit. We will do
> some more testing on Jenkins before it becomes an official PreCommit task.
>
> Thanks,
> Max
>
> On 29.11.18 19:03, Mark Liu wrote:
> > ah, thanks Boyuan! Probably I created the PR in a bad timing. Looks like
> > a new PR will fix it.
> >
> > On Thu, Nov 29, 2018 at 9:50 AM Boyuan Zhang <boyuanz@google.com
> > <ma...@google.com>> wrote:
> >
> > I believe it's in this pending PR:
> > https://github.com/apache/beam/pull/7157.
> >
> > On Thu, Nov 29, 2018 at 8:36 AM Mark Liu <markliu@google.com
> > <ma...@google.com>> wrote:
> >
> > Hi guys,
> >
> > I made some changes <https://github.com/apache/beam/pull/7146>
> > to Python PreCommit Gradle and then Portable_Python is invoked
> > as a PrecCommit test and failed. However, I can't find where
> > it's defined / generated in Gradle or Jenkins groovy. Does
> > anyone know? My branch is synced to master yesterday.
> >
> > Thanks!
> > Mark
> >
>
Re: What is Jenkins job "Portable_Python" in PreCommit?
Posted by Maximilian Michels <mx...@apache.org>.
This was merged with https://github.com/apache/beam/pull/6954.
Eventually we want to run a portable WordCount on PreCommit. We will do
some more testing on Jenkins before it becomes an official PreCommit task.
Thanks,
Max
On 29.11.18 19:03, Mark Liu wrote:
> ah, thanks Boyuan! Probably I created the PR in a bad timing. Looks like
> a new PR will fix it.
>
> On Thu, Nov 29, 2018 at 9:50 AM Boyuan Zhang <boyuanz@google.com
> <ma...@google.com>> wrote:
>
> I believe it's in this pending PR:
> https://github.com/apache/beam/pull/7157.
>
> On Thu, Nov 29, 2018 at 8:36 AM Mark Liu <markliu@google.com
> <ma...@google.com>> wrote:
>
> Hi guys,
>
> I made some changes <https://github.com/apache/beam/pull/7146>
> to Python PreCommit Gradle and then Portable_Python is invoked
> as a PrecCommit test and failed. However, I can't find where
> it's defined / generated in Gradle or Jenkins groovy. Does
> anyone know? My branch is synced to master yesterday.
>
> Thanks!
> Mark
>
Re: What is Jenkins job "Portable_Python" in PreCommit?
Posted by Mark Liu <ma...@google.com>.
ah, thanks Boyuan! Probably I created the PR in a bad timing. Looks like a
new PR will fix it.
On Thu, Nov 29, 2018 at 9:50 AM Boyuan Zhang <bo...@google.com> wrote:
> I believe it's in this pending PR:
> https://github.com/apache/beam/pull/7157.
>
> On Thu, Nov 29, 2018 at 8:36 AM Mark Liu <ma...@google.com> wrote:
>
>> Hi guys,
>>
>> I made some changes <https://github.com/apache/beam/pull/7146> to Python
>> PreCommit Gradle and then Portable_Python is invoked as a PrecCommit test
>> and failed. However, I can't find where it's defined / generated in Gradle
>> or Jenkins groovy. Does anyone know? My branch is synced to master
>> yesterday.
>>
>> Thanks!
>> Mark
>>
>
Re: What is Jenkins job "Portable_Python" in PreCommit?
Posted by Boyuan Zhang <bo...@google.com>.
I believe it's in this pending PR: https://github.com/apache/beam/pull/7157.
On Thu, Nov 29, 2018 at 8:36 AM Mark Liu <ma...@google.com> wrote:
> Hi guys,
>
> I made some changes <https://github.com/apache/beam/pull/7146> to Python
> PreCommit Gradle and then Portable_Python is invoked as a PrecCommit test
> and failed. However, I can't find where it's defined / generated in Gradle
> or Jenkins groovy. Does anyone know? My branch is synced to master
> yesterday.
>
> Thanks!
> Mark
>