You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Udi Meiri <eh...@google.com> on 2019/07/12 00:12:07 UTC
python precommits failing at head
This is due to
https://github.com/apache/beam/pull/8969
and
https://github.com/apache/beam/pull/8934
being merged today.
Fix is here: https://github.com/apache/beam/pull/9044
Re: python precommits failing at head
Posted by Tanay Tummalapalli <tt...@gmail.com>.
Thanks @Robert! Python PreCommit is green now!
On Mon, Jul 22, 2019 at 5:57 PM Robert Bradshaw <ro...@google.com> wrote:
> This was due to a bad release artifact push. This has now been fixed
> upstream.
>
> On Mon, Jul 22, 2019 at 11:00 AM Robert Bradshaw <ro...@google.com>
> wrote:
> >
> > Looks like https://sourceforge.net/p/docutils/bugs/365/
> >
> > On Sun, Jul 21, 2019 at 11:56 PM Tanay Tummalapalli <tt...@gmail.com>
> wrote:
> >>
> >> Hi everyone,
> >>
> >> The Python PreCommit from the Jenkins job "beam_PreCommit_Python_Cron"
> is failing[1]. The task :sdks:python:docs is failing with this traceback:
> >>
> >> Traceback (most recent call last):
> >> File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/bin/sphinx-apidoc",
> line 8, in <module>
> >> from sphinx.apidoc import main
> >> File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/local/lib/python2.7/site-packages/sphinx/apidoc.py",
> line 27, in <module>
> >> from sphinx.quickstart import EXTENSIONS
> >> File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/local/lib/python2.7/site-packages/sphinx/quickstart.py",
> line 35, in <module>
> >> from docutils.utils import column_width
> >> File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/local/lib/python2.7/site-packages/docutils/utils/__init__.py",
> line 21, in <module>
> >> import docutils.io
> >> File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/local/lib/python2.7/site-packages/docutils/io.py",
> line 348
> >> (self.destination.mode, mode)), file=self._stderr)
> >> ^
> >> SyntaxError: invalid syntax
> >>
> >> It seems to be due to a bug in docutils which is a dependency of sphinx.
> >> Unfortunately, I don't have much context about this. I'd love to help
> with fixing this.
> >>
> >> There is another failure in test_wordcount_it of task
> :sdks:python:test-suites:dataflow:preCommitIT, but it seems to be a flaky
> test.
> >>
> >> [1]
> https://scans.gradle.com/s/ygctjiqxcz6qa/console-log?task=:sdks:python:docs
> >>
> >> On Mon, Jul 15, 2019 at 11:16 PM Yifan Zou <yi...@google.com> wrote:
> >>>
> >>> We saw similar timeouts of the python precommit and it usually break
> the Jenkins build workers. I've run the precommit manually several times.
> It sometimes stuck at :sdks:python:docs and consumes 80G+ memory. Our build
> VMs eventually ran out of memory (104G memory available in total) then
> disconnected. Not sure what happened during that job.
> >>>
> >>>
> >>>
> >>> On Sat, Jul 13, 2019 at 10:27 PM Tanay Tummalapalli <
> ttanay100@gmail.com> wrote:
> >>>>
> >>>> Yes. It passed on the second attempt.
> >>>>
> >>>> But, I'm yet to figure out why it hangs for ~1.5 hours.
> >>>>
> >>>> On Sun, Jul 14, 2019 at 10:36 AM Rakesh Kumar <ra...@lyft.com>
> wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>> Even I am running into the same issue. Though my test passed but
> somehow the task didn't terminate, eventually the task was aborted. I have
> already tried a couple of times to retrigger tye python precommit but it
> failed every time.
> >>>>>
> >>>>> @Tanay did it pass it for you?
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Fri, Jul 12, 2019 at 2:22 PM Tanay Tummalapalli <
> ttanay100@gmail.com> wrote:
> >>>>>>
> >>>>>> Thank You Valentyn!
> >>>>>>
> >>>>>> I'll retest it.
> >>>>>> Hopefully, it's a transient issue.
> >>>>>>
> >>>>>> Regards,
> >>>>>> - Tanay Tummalapalli
> >>>>>>
> >>>>>> On Sat, Jul 13, 2019 at 2:39 AM Valentyn Tymofieiev <
> valentyn@google.com> wrote:
> >>>>>>>
> >>>>>>> No, we did not reduce the timeout recently. Looking at console
> logs, nothing happened for an hour or so,
> >>>>>>>
> >>>>>>> 06:57:50 py27-cython: commands succeeded
> >>>>>>> 06:57:50 congratulations :)
> >>>>>>> 06:57:50
> >>>>>>>
> >>>>>>> 06:57:50 > Task :sdks:python:preCommitPy2
> >>>>>>> 08:22:33 Build timed out (after 120 minutes). Marking the build as
> aborted.
> >>>>>>>
> >>>>>>>
> >>>>>>> However, we can also see in the logs that py36-cython suite never
> started, not sure way. I assume gradle waited for this suite to finish.
> >>>>>>> Try "retest this please", hopefully this is a transient gradle
> issue. I did not observe it before.
> >>>>>>>
> >>>>>>> On Fri, Jul 12, 2019 at 1:22 PM Tanay Tummalapalli <
> ttanay100@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>> Hi Udi,
> >>>>>>>>
> >>>>>>>> I rebased another PR[1] onto the fix mentioned above. The lint
> error is fixed, but, the "beam_PreCommit_Python_Commit" Jenkins job is
> failing because of a timeout at 120 minutes[2].
> >>>>>>>> The log says "Build timed out (after 120 minutes). Marking the
> build as aborted."
> >>>>>>>> Another PR's Python PreCommit job aborted with the same error[3].
> >>>>>>>>
> >>>>>>>> I found this issue - "[BEAM-3040] Python precommit timed out
> after 150 minutes"[4].
> >>>>>>>> Was the timeout reduced recently?
> >>>>>>>>
> >>>>>>>> Regards,
> >>>>>>>> - Tanay Tummalapalli
> >>>>>>>>
> >>>>>>>> [1] https://github.com/apache/beam/pull/8871
> >>>>>>>> [2]
> https://builds.apache.org/job/beam_PreCommit_Python_Commit/7412/consoleFull
> >>>>>>>> [3] https://github.com/apache/beam/pull/9050
> >>>>>>>> [4] https://issues.apache.org/jira/browse/BEAM-3040
> >>>>>>>>
> >>>>>>>> On Fri, Jul 12, 2019 at 5:42 AM Udi Meiri <eh...@google.com>
> wrote:
> >>>>>>>>>
> >>>>>>>>> This is due to
> >>>>>>>>> https://github.com/apache/beam/pull/8969
> >>>>>>>>> and
> >>>>>>>>> https://github.com/apache/beam/pull/8934
> >>>>>>>>> being merged today.
> >>>>>>>>>
> >>>>>>>>> Fix is here: https://github.com/apache/beam/pull/9044
>
Re: python precommits failing at head
Posted by Robert Bradshaw <ro...@google.com>.
This was due to a bad release artifact push. This has now been fixed upstream.
On Mon, Jul 22, 2019 at 11:00 AM Robert Bradshaw <ro...@google.com> wrote:
>
> Looks like https://sourceforge.net/p/docutils/bugs/365/
>
> On Sun, Jul 21, 2019 at 11:56 PM Tanay Tummalapalli <tt...@gmail.com> wrote:
>>
>> Hi everyone,
>>
>> The Python PreCommit from the Jenkins job "beam_PreCommit_Python_Cron" is failing[1]. The task :sdks:python:docs is failing with this traceback:
>>
>> Traceback (most recent call last):
>> File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/bin/sphinx-apidoc", line 8, in <module>
>> from sphinx.apidoc import main
>> File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/local/lib/python2.7/site-packages/sphinx/apidoc.py", line 27, in <module>
>> from sphinx.quickstart import EXTENSIONS
>> File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/local/lib/python2.7/site-packages/sphinx/quickstart.py", line 35, in <module>
>> from docutils.utils import column_width
>> File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/local/lib/python2.7/site-packages/docutils/utils/__init__.py", line 21, in <module>
>> import docutils.io
>> File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/local/lib/python2.7/site-packages/docutils/io.py", line 348
>> (self.destination.mode, mode)), file=self._stderr)
>> ^
>> SyntaxError: invalid syntax
>>
>> It seems to be due to a bug in docutils which is a dependency of sphinx.
>> Unfortunately, I don't have much context about this. I'd love to help with fixing this.
>>
>> There is another failure in test_wordcount_it of task :sdks:python:test-suites:dataflow:preCommitIT, but it seems to be a flaky test.
>>
>> [1] https://scans.gradle.com/s/ygctjiqxcz6qa/console-log?task=:sdks:python:docs
>>
>> On Mon, Jul 15, 2019 at 11:16 PM Yifan Zou <yi...@google.com> wrote:
>>>
>>> We saw similar timeouts of the python precommit and it usually break the Jenkins build workers. I've run the precommit manually several times. It sometimes stuck at :sdks:python:docs and consumes 80G+ memory. Our build VMs eventually ran out of memory (104G memory available in total) then disconnected. Not sure what happened during that job.
>>>
>>>
>>>
>>> On Sat, Jul 13, 2019 at 10:27 PM Tanay Tummalapalli <tt...@gmail.com> wrote:
>>>>
>>>> Yes. It passed on the second attempt.
>>>>
>>>> But, I'm yet to figure out why it hangs for ~1.5 hours.
>>>>
>>>> On Sun, Jul 14, 2019 at 10:36 AM Rakesh Kumar <ra...@lyft.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> Even I am running into the same issue. Though my test passed but somehow the task didn't terminate, eventually the task was aborted. I have already tried a couple of times to retrigger tye python precommit but it failed every time.
>>>>>
>>>>> @Tanay did it pass it for you?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Jul 12, 2019 at 2:22 PM Tanay Tummalapalli <tt...@gmail.com> wrote:
>>>>>>
>>>>>> Thank You Valentyn!
>>>>>>
>>>>>> I'll retest it.
>>>>>> Hopefully, it's a transient issue.
>>>>>>
>>>>>> Regards,
>>>>>> - Tanay Tummalapalli
>>>>>>
>>>>>> On Sat, Jul 13, 2019 at 2:39 AM Valentyn Tymofieiev <va...@google.com> wrote:
>>>>>>>
>>>>>>> No, we did not reduce the timeout recently. Looking at console logs, nothing happened for an hour or so,
>>>>>>>
>>>>>>> 06:57:50 py27-cython: commands succeeded
>>>>>>> 06:57:50 congratulations :)
>>>>>>> 06:57:50
>>>>>>>
>>>>>>> 06:57:50 > Task :sdks:python:preCommitPy2
>>>>>>> 08:22:33 Build timed out (after 120 minutes). Marking the build as aborted.
>>>>>>>
>>>>>>>
>>>>>>> However, we can also see in the logs that py36-cython suite never started, not sure way. I assume gradle waited for this suite to finish.
>>>>>>> Try "retest this please", hopefully this is a transient gradle issue. I did not observe it before.
>>>>>>>
>>>>>>> On Fri, Jul 12, 2019 at 1:22 PM Tanay Tummalapalli <tt...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Hi Udi,
>>>>>>>>
>>>>>>>> I rebased another PR[1] onto the fix mentioned above. The lint error is fixed, but, the "beam_PreCommit_Python_Commit" Jenkins job is failing because of a timeout at 120 minutes[2].
>>>>>>>> The log says "Build timed out (after 120 minutes). Marking the build as aborted."
>>>>>>>> Another PR's Python PreCommit job aborted with the same error[3].
>>>>>>>>
>>>>>>>> I found this issue - "[BEAM-3040] Python precommit timed out after 150 minutes"[4].
>>>>>>>> Was the timeout reduced recently?
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> - Tanay Tummalapalli
>>>>>>>>
>>>>>>>> [1] https://github.com/apache/beam/pull/8871
>>>>>>>> [2] https://builds.apache.org/job/beam_PreCommit_Python_Commit/7412/consoleFull
>>>>>>>> [3] https://github.com/apache/beam/pull/9050
>>>>>>>> [4] https://issues.apache.org/jira/browse/BEAM-3040
>>>>>>>>
>>>>>>>> On Fri, Jul 12, 2019 at 5:42 AM Udi Meiri <eh...@google.com> wrote:
>>>>>>>>>
>>>>>>>>> This is due to
>>>>>>>>> https://github.com/apache/beam/pull/8969
>>>>>>>>> and
>>>>>>>>> https://github.com/apache/beam/pull/8934
>>>>>>>>> being merged today.
>>>>>>>>>
>>>>>>>>> Fix is here: https://github.com/apache/beam/pull/9044
Re: python precommits failing at head
Posted by Robert Bradshaw <ro...@google.com>.
Looks like https://sourceforge.net/p/docutils/bugs/365/
On Sun, Jul 21, 2019 at 11:56 PM Tanay Tummalapalli <tt...@gmail.com>
wrote:
> Hi everyone,
>
> The Python PreCommit from the Jenkins job "beam_PreCommit_Python_Cron" is
> failing[1]. The task :sdks:python:docs is failing with this traceback:
>
> Traceback (most recent call last):
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/bin/sphinx-apidoc",
> line 8, in <module>
> from sphinx.apidoc import main
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/local/lib/python2.7/site-packages/sphinx/apidoc.py",
> line 27, in <module>
> from sphinx.quickstart import EXTENSIONS
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/local/lib/python2.7/site-packages/sphinx/quickstart.py",
> line 35, in <module>
> from docutils.utils import column_width
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/local/lib/python2.7/site-packages/docutils/utils/__init__.py",
> line 21, in <module>
> import docutils.io
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/local/lib/python2.7/site-packages/docutils/io.py",
> line 348
> (self.destination.mode, mode)), file=self._stderr)
> ^
> SyntaxError: invalid syntax
>
> It seems to be due to a bug in docutils which is a dependency of sphinx.
> Unfortunately, I don't have much context about this. I'd love to help with
> fixing this.
>
> There is another failure in test_wordcount_it of task :sdks:python:test-suites:dataflow:preCommitIT,
> but it seems to be a flaky test.
>
> [1]
> https://scans.gradle.com/s/ygctjiqxcz6qa/console-log?task=:sdks:python:docs
>
> On Mon, Jul 15, 2019 at 11:16 PM Yifan Zou <yi...@google.com> wrote:
>
>> We saw similar timeouts of the python precommit and it usually break the
>> Jenkins build workers. I've run the precommit manually several times. It
>> sometimes stuck at :sdks:python:docs and consumes 80G+ memory. Our build
>> VMs eventually ran out of memory (104G memory available in total) then
>> disconnected. Not sure what happened during that job.
>>
>>
>>
>> On Sat, Jul 13, 2019 at 10:27 PM Tanay Tummalapalli <tt...@gmail.com>
>> wrote:
>>
>>> Yes. It passed on the second attempt.
>>>
>>> But, I'm yet to figure out why it hangs for ~1.5 hours.
>>>
>>> On Sun, Jul 14, 2019 at 10:36 AM Rakesh Kumar <ra...@lyft.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> Even I am running into the same issue. Though my test passed but
>>>> <https://builds.apache.org/job/beam_PreCommit_Python_Phrase/656/console>somehow
>>>> the task didn't terminate, eventually the task was aborted. I have already
>>>> tried a couple of times to retrigger tye python precommit but it failed
>>>> every time.
>>>>
>>>> @Tanay did it pass it for you?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Jul 12, 2019 at 2:22 PM Tanay Tummalapalli <tt...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thank You Valentyn!
>>>>>
>>>>> I'll retest it.
>>>>> Hopefully, it's a transient issue.
>>>>>
>>>>> Regards,
>>>>> - Tanay Tummalapalli
>>>>>
>>>>> On Sat, Jul 13, 2019 at 2:39 AM Valentyn Tymofieiev <
>>>>> valentyn@google.com> wrote:
>>>>>
>>>>>> No, we did not reduce the timeout recently. Looking at console logs,
>>>>>> nothing happened for an hour or so,
>>>>>>
>>>>>> *06:57:50 py27-cython: commands succeeded 06:57:50 congratulations :)
>>>>>> 06:57:50 *
>>>>>>
>>>>>> *06:57:50* >* Task :sdks:python:preCommitPy2**08:22:33* Build timed out (after 120 minutes). Marking the build as aborted.
>>>>>>
>>>>>>
>>>>>> However, we can also see in the logs that py36-cython suite never
>>>>>> started, not sure way. I assume gradle waited for this suite to finish.
>>>>>> Try "retest this please", hopefully this is a transient gradle issue.
>>>>>> I did not observe it before.
>>>>>>
>>>>>> On Fri, Jul 12, 2019 at 1:22 PM Tanay Tummalapalli <
>>>>>> ttanay100@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Udi,
>>>>>>>
>>>>>>> I rebased another PR[1] onto the fix mentioned above. The lint error
>>>>>>> is fixed, but, the "beam_PreCommit_Python_Commit" Jenkins job is failing
>>>>>>> because of a timeout at 120 minutes[2].
>>>>>>> The log says "Build timed out (after 120 minutes). Marking the
>>>>>>> build as aborted."
>>>>>>> Another PR's Python PreCommit job aborted with the same error[3].
>>>>>>>
>>>>>>> I found this issue - "[BEAM-3040] Python precommit timed out after
>>>>>>> 150 minutes"[4].
>>>>>>> Was the timeout reduced recently?
>>>>>>>
>>>>>>> Regards,
>>>>>>> - Tanay Tummalapalli
>>>>>>>
>>>>>>> [1] https://github.com/apache/beam/pull/8871
>>>>>>> [2]
>>>>>>> https://builds.apache.org/job/beam_PreCommit_Python_Commit/7412/consoleFull
>>>>>>>
>>>>>>> [3] https://github.com/apache/beam/pull/9050
>>>>>>> [4] https://issues.apache.org/jira/browse/BEAM-3040
>>>>>>>
>>>>>>> On Fri, Jul 12, 2019 at 5:42 AM Udi Meiri <eh...@google.com> wrote:
>>>>>>>
>>>>>>>> This is due to
>>>>>>>> https://github.com/apache/beam/pull/8969
>>>>>>>> and
>>>>>>>> https://github.com/apache/beam/pull/8934
>>>>>>>> being merged today.
>>>>>>>>
>>>>>>>> Fix is here: https://github.com/apache/beam/pull/9044
>>>>>>>>
>>>>>>>
Re: python precommits failing at head
Posted by Tanay Tummalapalli <tt...@gmail.com>.
Hi everyone,
The Python PreCommit from the Jenkins job "beam_PreCommit_Python_Cron" is
failing[1]. The task :sdks:python:docs is failing with this traceback:
Traceback (most recent call last):
File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/bin/sphinx-apidoc",
line 8, in <module>
from sphinx.apidoc import main
File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/local/lib/python2.7/site-packages/sphinx/apidoc.py",
line 27, in <module>
from sphinx.quickstart import EXTENSIONS
File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/local/lib/python2.7/site-packages/sphinx/quickstart.py",
line 35, in <module>
from docutils.utils import column_width
File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/local/lib/python2.7/site-packages/docutils/utils/__init__.py",
line 21, in <module>
import docutils.io
File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/local/lib/python2.7/site-packages/docutils/io.py",
line 348
(self.destination.mode, mode)), file=self._stderr)
^
SyntaxError: invalid syntax
It seems to be due to a bug in docutils which is a dependency of sphinx.
Unfortunately, I don't have much context about this. I'd love to help with
fixing this.
There is another failure in test_wordcount_it of task
:sdks:python:test-suites:dataflow:preCommitIT,
but it seems to be a flaky test.
[1]
https://scans.gradle.com/s/ygctjiqxcz6qa/console-log?task=:sdks:python:docs
On Mon, Jul 15, 2019 at 11:16 PM Yifan Zou <yi...@google.com> wrote:
> We saw similar timeouts of the python precommit and it usually break the
> Jenkins build workers. I've run the precommit manually several times. It
> sometimes stuck at :sdks:python:docs and consumes 80G+ memory. Our build
> VMs eventually ran out of memory (104G memory available in total) then
> disconnected. Not sure what happened during that job.
>
>
>
> On Sat, Jul 13, 2019 at 10:27 PM Tanay Tummalapalli <tt...@gmail.com>
> wrote:
>
>> Yes. It passed on the second attempt.
>>
>> But, I'm yet to figure out why it hangs for ~1.5 hours.
>>
>> On Sun, Jul 14, 2019 at 10:36 AM Rakesh Kumar <ra...@lyft.com>
>> wrote:
>>
>>>
>>>
>>> Even I am running into the same issue. Though my test passed but
>>> <https://builds.apache.org/job/beam_PreCommit_Python_Phrase/656/console>somehow
>>> the task didn't terminate, eventually the task was aborted. I have already
>>> tried a couple of times to retrigger tye python precommit but it failed
>>> every time.
>>>
>>> @Tanay did it pass it for you?
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Jul 12, 2019 at 2:22 PM Tanay Tummalapalli <tt...@gmail.com>
>>> wrote:
>>>
>>>> Thank You Valentyn!
>>>>
>>>> I'll retest it.
>>>> Hopefully, it's a transient issue.
>>>>
>>>> Regards,
>>>> - Tanay Tummalapalli
>>>>
>>>> On Sat, Jul 13, 2019 at 2:39 AM Valentyn Tymofieiev <
>>>> valentyn@google.com> wrote:
>>>>
>>>>> No, we did not reduce the timeout recently. Looking at console logs,
>>>>> nothing happened for an hour or so,
>>>>>
>>>>> *06:57:50 py27-cython: commands succeeded 06:57:50 congratulations :)
>>>>> 06:57:50 *
>>>>>
>>>>> *06:57:50* >* Task :sdks:python:preCommitPy2**08:22:33* Build timed out (after 120 minutes). Marking the build as aborted.
>>>>>
>>>>>
>>>>> However, we can also see in the logs that py36-cython suite never
>>>>> started, not sure way. I assume gradle waited for this suite to finish.
>>>>> Try "retest this please", hopefully this is a transient gradle issue.
>>>>> I did not observe it before.
>>>>>
>>>>> On Fri, Jul 12, 2019 at 1:22 PM Tanay Tummalapalli <
>>>>> ttanay100@gmail.com> wrote:
>>>>>
>>>>>> Hi Udi,
>>>>>>
>>>>>> I rebased another PR[1] onto the fix mentioned above. The lint error
>>>>>> is fixed, but, the "beam_PreCommit_Python_Commit" Jenkins job is failing
>>>>>> because of a timeout at 120 minutes[2].
>>>>>> The log says "Build timed out (after 120 minutes). Marking the build
>>>>>> as aborted."
>>>>>> Another PR's Python PreCommit job aborted with the same error[3].
>>>>>>
>>>>>> I found this issue - "[BEAM-3040] Python precommit timed out after
>>>>>> 150 minutes"[4].
>>>>>> Was the timeout reduced recently?
>>>>>>
>>>>>> Regards,
>>>>>> - Tanay Tummalapalli
>>>>>>
>>>>>> [1] https://github.com/apache/beam/pull/8871
>>>>>> [2]
>>>>>> https://builds.apache.org/job/beam_PreCommit_Python_Commit/7412/consoleFull
>>>>>>
>>>>>> [3] https://github.com/apache/beam/pull/9050
>>>>>> [4] https://issues.apache.org/jira/browse/BEAM-3040
>>>>>>
>>>>>> On Fri, Jul 12, 2019 at 5:42 AM Udi Meiri <eh...@google.com> wrote:
>>>>>>
>>>>>>> This is due to
>>>>>>> https://github.com/apache/beam/pull/8969
>>>>>>> and
>>>>>>> https://github.com/apache/beam/pull/8934
>>>>>>> being merged today.
>>>>>>>
>>>>>>> Fix is here: https://github.com/apache/beam/pull/9044
>>>>>>>
>>>>>>
Re: python precommits failing at head
Posted by Yifan Zou <yi...@google.com>.
We saw similar timeouts of the python precommit and it usually break the
Jenkins build workers. I've run the precommit manually several times. It
sometimes stuck at :sdks:python:docs and consumes 80G+ memory. Our build
VMs eventually ran out of memory (104G memory available in total) then
disconnected. Not sure what happened during that job.
On Sat, Jul 13, 2019 at 10:27 PM Tanay Tummalapalli <tt...@gmail.com>
wrote:
> Yes. It passed on the second attempt.
>
> But, I'm yet to figure out why it hangs for ~1.5 hours.
>
> On Sun, Jul 14, 2019 at 10:36 AM Rakesh Kumar <ra...@lyft.com>
> wrote:
>
>>
>>
>> Even I am running into the same issue. Though my test passed but
>> <https://builds.apache.org/job/beam_PreCommit_Python_Phrase/656/console>somehow
>> the task didn't terminate, eventually the task was aborted. I have already
>> tried a couple of times to retrigger tye python precommit but it failed
>> every time.
>>
>> @Tanay did it pass it for you?
>>
>>
>>
>>
>>
>> On Fri, Jul 12, 2019 at 2:22 PM Tanay Tummalapalli <tt...@gmail.com>
>> wrote:
>>
>>> Thank You Valentyn!
>>>
>>> I'll retest it.
>>> Hopefully, it's a transient issue.
>>>
>>> Regards,
>>> - Tanay Tummalapalli
>>>
>>> On Sat, Jul 13, 2019 at 2:39 AM Valentyn Tymofieiev <va...@google.com>
>>> wrote:
>>>
>>>> No, we did not reduce the timeout recently. Looking at console logs,
>>>> nothing happened for an hour or so,
>>>>
>>>> *06:57:50 py27-cython: commands succeeded 06:57:50 congratulations :)
>>>> 06:57:50 *
>>>>
>>>> *06:57:50* >* Task :sdks:python:preCommitPy2**08:22:33* Build timed out (after 120 minutes). Marking the build as aborted.
>>>>
>>>>
>>>> However, we can also see in the logs that py36-cython suite never
>>>> started, not sure way. I assume gradle waited for this suite to finish.
>>>> Try "retest this please", hopefully this is a transient gradle issue. I
>>>> did not observe it before.
>>>>
>>>> On Fri, Jul 12, 2019 at 1:22 PM Tanay Tummalapalli <tt...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Udi,
>>>>>
>>>>> I rebased another PR[1] onto the fix mentioned above. The lint error
>>>>> is fixed, but, the "beam_PreCommit_Python_Commit" Jenkins job is failing
>>>>> because of a timeout at 120 minutes[2].
>>>>> The log says "Build timed out (after 120 minutes). Marking the build
>>>>> as aborted."
>>>>> Another PR's Python PreCommit job aborted with the same error[3].
>>>>>
>>>>> I found this issue - "[BEAM-3040] Python precommit timed out after 150
>>>>> minutes"[4].
>>>>> Was the timeout reduced recently?
>>>>>
>>>>> Regards,
>>>>> - Tanay Tummalapalli
>>>>>
>>>>> [1] https://github.com/apache/beam/pull/8871
>>>>> [2]
>>>>> https://builds.apache.org/job/beam_PreCommit_Python_Commit/7412/consoleFull
>>>>>
>>>>> [3] https://github.com/apache/beam/pull/9050
>>>>> [4] https://issues.apache.org/jira/browse/BEAM-3040
>>>>>
>>>>> On Fri, Jul 12, 2019 at 5:42 AM Udi Meiri <eh...@google.com> wrote:
>>>>>
>>>>>> This is due to
>>>>>> https://github.com/apache/beam/pull/8969
>>>>>> and
>>>>>> https://github.com/apache/beam/pull/8934
>>>>>> being merged today.
>>>>>>
>>>>>> Fix is here: https://github.com/apache/beam/pull/9044
>>>>>>
>>>>>
Re: python precommits failing at head
Posted by Tanay Tummalapalli <tt...@gmail.com>.
Yes. It passed on the second attempt.
But, I'm yet to figure out why it hangs for ~1.5 hours.
On Sun, Jul 14, 2019 at 10:36 AM Rakesh Kumar <ra...@lyft.com> wrote:
>
>
> Even I am running into the same issue. Though my test passed but
> <https://builds.apache.org/job/beam_PreCommit_Python_Phrase/656/console>somehow
> the task didn't terminate, eventually the task was aborted. I have already
> tried a couple of times to retrigger tye python precommit but it failed
> every time.
>
> @Tanay did it pass it for you?
>
>
>
>
>
> On Fri, Jul 12, 2019 at 2:22 PM Tanay Tummalapalli <tt...@gmail.com>
> wrote:
>
>> Thank You Valentyn!
>>
>> I'll retest it.
>> Hopefully, it's a transient issue.
>>
>> Regards,
>> - Tanay Tummalapalli
>>
>> On Sat, Jul 13, 2019 at 2:39 AM Valentyn Tymofieiev <va...@google.com>
>> wrote:
>>
>>> No, we did not reduce the timeout recently. Looking at console logs,
>>> nothing happened for an hour or so,
>>>
>>> *06:57:50 py27-cython: commands succeeded 06:57:50 congratulations :)
>>> 06:57:50 *
>>>
>>> *06:57:50* >* Task :sdks:python:preCommitPy2**08:22:33* Build timed out (after 120 minutes). Marking the build as aborted.
>>>
>>>
>>> However, we can also see in the logs that py36-cython suite never
>>> started, not sure way. I assume gradle waited for this suite to finish.
>>> Try "retest this please", hopefully this is a transient gradle issue. I
>>> did not observe it before.
>>>
>>> On Fri, Jul 12, 2019 at 1:22 PM Tanay Tummalapalli <tt...@gmail.com>
>>> wrote:
>>>
>>>> Hi Udi,
>>>>
>>>> I rebased another PR[1] onto the fix mentioned above. The lint error is
>>>> fixed, but, the "beam_PreCommit_Python_Commit" Jenkins job is failing
>>>> because of a timeout at 120 minutes[2].
>>>> The log says "Build timed out (after 120 minutes). Marking the build
>>>> as aborted."
>>>> Another PR's Python PreCommit job aborted with the same error[3].
>>>>
>>>> I found this issue - "[BEAM-3040] Python precommit timed out after 150
>>>> minutes"[4].
>>>> Was the timeout reduced recently?
>>>>
>>>> Regards,
>>>> - Tanay Tummalapalli
>>>>
>>>> [1] https://github.com/apache/beam/pull/8871
>>>> [2]
>>>> https://builds.apache.org/job/beam_PreCommit_Python_Commit/7412/consoleFull
>>>>
>>>> [3] https://github.com/apache/beam/pull/9050
>>>> [4] https://issues.apache.org/jira/browse/BEAM-3040
>>>>
>>>> On Fri, Jul 12, 2019 at 5:42 AM Udi Meiri <eh...@google.com> wrote:
>>>>
>>>>> This is due to
>>>>> https://github.com/apache/beam/pull/8969
>>>>> and
>>>>> https://github.com/apache/beam/pull/8934
>>>>> being merged today.
>>>>>
>>>>> Fix is here: https://github.com/apache/beam/pull/9044
>>>>>
>>>>
Re: python precommits failing at head
Posted by Rakesh Kumar <ra...@lyft.com>.
Even I am running into the same issue. Though my test passed but
<https://builds.apache.org/job/beam_PreCommit_Python_Phrase/656/console>somehow
the task didn't terminate, eventually the task was aborted. I have already
tried a couple of times to retrigger tye python precommit but it failed
every time.
@Tanay did it pass it for you?
On Fri, Jul 12, 2019 at 2:22 PM Tanay Tummalapalli <tt...@gmail.com>
wrote:
> Thank You Valentyn!
>
> I'll retest it.
> Hopefully, it's a transient issue.
>
> Regards,
> - Tanay Tummalapalli
>
> On Sat, Jul 13, 2019 at 2:39 AM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> No, we did not reduce the timeout recently. Looking at console logs,
>> nothing happened for an hour or so,
>>
>> *06:57:50 py27-cython: commands succeeded 06:57:50 congratulations :)
>> 06:57:50 *
>>
>> *06:57:50* >* Task :sdks:python:preCommitPy2**08:22:33* Build timed out (after 120 minutes). Marking the build as aborted.
>>
>>
>> However, we can also see in the logs that py36-cython suite never
>> started, not sure way. I assume gradle waited for this suite to finish.
>> Try "retest this please", hopefully this is a transient gradle issue. I
>> did not observe it before.
>>
>> On Fri, Jul 12, 2019 at 1:22 PM Tanay Tummalapalli <tt...@gmail.com>
>> wrote:
>>
>>> Hi Udi,
>>>
>>> I rebased another PR[1] onto the fix mentioned above. The lint error is
>>> fixed, but, the "beam_PreCommit_Python_Commit" Jenkins job is failing
>>> because of a timeout at 120 minutes[2].
>>> The log says "Build timed out (after 120 minutes). Marking the build as
>>> aborted."
>>> Another PR's Python PreCommit job aborted with the same error[3].
>>>
>>> I found this issue - "[BEAM-3040] Python precommit timed out after 150
>>> minutes"[4].
>>> Was the timeout reduced recently?
>>>
>>> Regards,
>>> - Tanay Tummalapalli
>>>
>>> [1] https://github.com/apache/beam/pull/8871
>>> [2]
>>> https://builds.apache.org/job/beam_PreCommit_Python_Commit/7412/consoleFull
>>>
>>> [3] https://github.com/apache/beam/pull/9050
>>> [4] https://issues.apache.org/jira/browse/BEAM-3040
>>>
>>> On Fri, Jul 12, 2019 at 5:42 AM Udi Meiri <eh...@google.com> wrote:
>>>
>>>> This is due to
>>>> https://github.com/apache/beam/pull/8969
>>>> and
>>>> https://github.com/apache/beam/pull/8934
>>>> being merged today.
>>>>
>>>> Fix is here: https://github.com/apache/beam/pull/9044
>>>>
>>>
Re: python precommits failing at head
Posted by Tanay Tummalapalli <tt...@gmail.com>.
Thank You Valentyn!
I'll retest it.
Hopefully, it's a transient issue.
Regards,
- Tanay Tummalapalli
On Sat, Jul 13, 2019 at 2:39 AM Valentyn Tymofieiev <va...@google.com>
wrote:
> No, we did not reduce the timeout recently. Looking at console logs,
> nothing happened for an hour or so,
>
> *06:57:50 py27-cython: commands succeeded 06:57:50 congratulations :)
> 06:57:50 *
>
> *06:57:50* >* Task :sdks:python:preCommitPy2**08:22:33* Build timed out (after 120 minutes). Marking the build as aborted.
>
>
> However, we can also see in the logs that py36-cython suite never started,
> not sure way. I assume gradle waited for this suite to finish.
> Try "retest this please", hopefully this is a transient gradle issue. I
> did not observe it before.
>
> On Fri, Jul 12, 2019 at 1:22 PM Tanay Tummalapalli <tt...@gmail.com>
> wrote:
>
>> Hi Udi,
>>
>> I rebased another PR[1] onto the fix mentioned above. The lint error is
>> fixed, but, the "beam_PreCommit_Python_Commit" Jenkins job is failing
>> because of a timeout at 120 minutes[2].
>> The log says "Build timed out (after 120 minutes). Marking the build as
>> aborted."
>> Another PR's Python PreCommit job aborted with the same error[3].
>>
>> I found this issue - "[BEAM-3040] Python precommit timed out after 150
>> minutes"[4].
>> Was the timeout reduced recently?
>>
>> Regards,
>> - Tanay Tummalapalli
>>
>> [1] https://github.com/apache/beam/pull/8871
>> [2]
>> https://builds.apache.org/job/beam_PreCommit_Python_Commit/7412/consoleFull
>>
>> [3] https://github.com/apache/beam/pull/9050
>> [4] https://issues.apache.org/jira/browse/BEAM-3040
>>
>> On Fri, Jul 12, 2019 at 5:42 AM Udi Meiri <eh...@google.com> wrote:
>>
>>> This is due to
>>> https://github.com/apache/beam/pull/8969
>>> and
>>> https://github.com/apache/beam/pull/8934
>>> being merged today.
>>>
>>> Fix is here: https://github.com/apache/beam/pull/9044
>>>
>>
Re: python precommits failing at head
Posted by Valentyn Tymofieiev <va...@google.com>.
No, we did not reduce the timeout recently. Looking at console logs,
nothing happened for an hour or so,
*06:57:50 py27-cython: commands succeeded 06:57:50 congratulations :)
06:57:50 *
*06:57:50* >* Task :sdks:python:preCommitPy2**08:22:33* Build timed
out (after 120 minutes). Marking the build as aborted.
However, we can also see in the logs that py36-cython suite never started,
not sure way. I assume gradle waited for this suite to finish.
Try "retest this please", hopefully this is a transient gradle issue. I did
not observe it before.
On Fri, Jul 12, 2019 at 1:22 PM Tanay Tummalapalli <tt...@gmail.com>
wrote:
> Hi Udi,
>
> I rebased another PR[1] onto the fix mentioned above. The lint error is
> fixed, but, the "beam_PreCommit_Python_Commit" Jenkins job is failing
> because of a timeout at 120 minutes[2].
> The log says "Build timed out (after 120 minutes). Marking the build as
> aborted."
> Another PR's Python PreCommit job aborted with the same error[3].
>
> I found this issue - "[BEAM-3040] Python precommit timed out after 150
> minutes"[4].
> Was the timeout reduced recently?
>
> Regards,
> - Tanay Tummalapalli
>
> [1] https://github.com/apache/beam/pull/8871
> [2]
> https://builds.apache.org/job/beam_PreCommit_Python_Commit/7412/consoleFull
>
> [3] https://github.com/apache/beam/pull/9050
> [4] https://issues.apache.org/jira/browse/BEAM-3040
>
> On Fri, Jul 12, 2019 at 5:42 AM Udi Meiri <eh...@google.com> wrote:
>
>> This is due to
>> https://github.com/apache/beam/pull/8969
>> and
>> https://github.com/apache/beam/pull/8934
>> being merged today.
>>
>> Fix is here: https://github.com/apache/beam/pull/9044
>>
>
Re: python precommits failing at head
Posted by Tanay Tummalapalli <tt...@gmail.com>.
Hi Udi,
I rebased another PR[1] onto the fix mentioned above. The lint error is
fixed, but, the "beam_PreCommit_Python_Commit" Jenkins job is failing
because of a timeout at 120 minutes[2].
The log says "Build timed out (after 120 minutes). Marking the build as
aborted."
Another PR's Python PreCommit job aborted with the same error[3].
I found this issue - "[BEAM-3040] Python precommit timed out after 150
minutes"[4].
Was the timeout reduced recently?
Regards,
- Tanay Tummalapalli
[1] https://github.com/apache/beam/pull/8871
[2]
https://builds.apache.org/job/beam_PreCommit_Python_Commit/7412/consoleFull
[3] https://github.com/apache/beam/pull/9050
[4] https://issues.apache.org/jira/browse/BEAM-3040
On Fri, Jul 12, 2019 at 5:42 AM Udi Meiri <eh...@google.com> wrote:
> This is due to
> https://github.com/apache/beam/pull/8969
> and
> https://github.com/apache/beam/pull/8934
> being merged today.
>
> Fix is here: https://github.com/apache/beam/pull/9044
>