You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Alex Amato <aj...@google.com> on 2019/01/23 21:00:42 UTC
ContainerLaunchException in precommit [BEAM-6497]
1.
See: BEAM-6497 <https://jira.apache.org/jira/browse/BEAM-6497>
1. This is also causing issues blocking precommits.
2.
Seems to be caused by this failure to locate the image. Are these
stored somewhere or built by the build process? Any idea why these are
failing?
Caused by:
com.github.dockerjava.api.exception.NotFoundException: {"message":"No
such image: yandex/clickhouse-server:18.10.3"}
Re: ContainerLaunchException in precommit [BEAM-6497]
Posted by Gleb Kanterov <gl...@spotify.com>.
It seems ClickHouse tests aren't flaky anymore, please reopen JIRA issue if
you find it flaky again.
On Thu, Jan 31, 2019 at 4:08 PM Gleb Kanterov <gl...@spotify.com> wrote:
> There are two tests using testcontainers. I've noticed that in one of the
> failed builds
> <https://builds.apache.org/job/beam_PreCommit_Java_Commit/3869/> [1] only
> one of them has failed to pull docker image. I suspect that adding retries
> to container pull step can mitigate this issue. I've submitted a pull
> request apache/beam#7689 <https://github.com/apache/beam/pull/7689> [2].
>
> [1] https://builds.apache.org/job/beam_PreCommit_Java_Commit/3869/
> [2] https://github.com/apache/beam/pull/7689
>
> On Wed, Jan 30, 2019 at 12:15 AM Kenneth Knowles <kl...@google.com> wrote:
>
>> I retract my statement. I failed at web browsing.
>>
>> On Tue, Jan 29, 2019 at 3:14 PM Kenneth Knowles <kl...@google.com> wrote:
>>
>>> Version 18.10.3 no longer appears on the linked page.
>>>
>>> On Tue, Jan 29, 2019 at 3:08 PM David Rieber <dr...@google.com> wrote:
>>>
>>>> I am consistently hitting that error on this PR:
>>>> https://github.com/apache/beam/pull/7631
>>>>
>>>>
>>>> On Thu, Jan 24, 2019 at 9:14 AM Alex Amato <aj...@google.com> wrote:
>>>>
>>>>> I have just seen it randomly occur on presubmits. So I don't have a
>>>>> reliable repro, unfortunately.
>>>>> It may be a specific environmental issue to the beam1 machine the
>>>>> tests ran on?
>>>>> https://builds.apache.org/job/beam_PreCommit_Java_Commit/3722/
>>>>>
>>>>>
>>>>> On Thu, Jan 24, 2019 at 8:16 AM Gleb Kanterov <gl...@spotify.com>
>>>>> wrote:
>>>>>
>>>>>> I'm wondering if anybody can reproduce this issue. The build has
>>>>>> failed once because testcontainers didn't pull docker image. If we use
>>>>>> caching proxy for docker, it could be a reason for that. I didn't find any
>>>>>> similar known issue in testcontainers fixed recently, but just in case, I
>>>>>> bumped testcontainers to use never docker-java.
>>>>>>
>>>>>> https://github.com/apache/beam/pull/7610
>>>>>>
>>>>>> On Thu, Jan 24, 2019 at 12:27 AM Alex Amato <aj...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thank you Gleb, appreciate it.
>>>>>>>
>>>>>>> On Wed, Jan 23, 2019 at 2:40 PM Gleb Kanterov <gl...@spotify.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I'm looking into it. This image exists in docker hub [1], but for
>>>>>>>> some reason, it wasn't picked up.
>>>>>>>>
>>>>>>>> [1] https://hub.docker.com/r/yandex/clickhouse-server/tags
>>>>>>>>
>>>>>>>> On Wed, Jan 23, 2019 at 10:01 PM Alex Amato <aj...@google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> 1.
>>>>>>>>> See: BEAM-6497
>>>>>>>>> <https://jira.apache.org/jira/browse/BEAM-6497>
>>>>>>>>> 1. This is also causing issues blocking precommits.
>>>>>>>>> 2.
>>>>>>>>> Seems to be caused by this failure to locate the image. Are
>>>>>>>>> these stored somewhere or built by the build process? Any idea why these
>>>>>>>>> are failing?
>>>>>>>>>
>>>>>>>>> Caused by: com.github.dockerjava.api.exception.NotFoundException: {"message":"No such image: yandex/clickhouse-server:18.10.3"}
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Cheers,
>>>>>>>> Gleb
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Cheers,
>>>>>> Gleb
>>>>>>
>>>>>
>
> --
> Cheers,
> Gleb
>
--
Cheers,
Gleb
Re: ContainerLaunchException in precommit [BEAM-6497]
Posted by Gleb Kanterov <gl...@spotify.com>.
There are two tests using testcontainers. I've noticed that in one of the
failed builds
<https://builds.apache.org/job/beam_PreCommit_Java_Commit/3869/> [1] only
one of them has failed to pull docker image. I suspect that adding retries
to container pull step can mitigate this issue. I've submitted a pull
request apache/beam#7689 <https://github.com/apache/beam/pull/7689> [2].
[1] https://builds.apache.org/job/beam_PreCommit_Java_Commit/3869/
[2] https://github.com/apache/beam/pull/7689
On Wed, Jan 30, 2019 at 12:15 AM Kenneth Knowles <kl...@google.com> wrote:
> I retract my statement. I failed at web browsing.
>
> On Tue, Jan 29, 2019 at 3:14 PM Kenneth Knowles <kl...@google.com> wrote:
>
>> Version 18.10.3 no longer appears on the linked page.
>>
>> On Tue, Jan 29, 2019 at 3:08 PM David Rieber <dr...@google.com> wrote:
>>
>>> I am consistently hitting that error on this PR:
>>> https://github.com/apache/beam/pull/7631
>>>
>>>
>>> On Thu, Jan 24, 2019 at 9:14 AM Alex Amato <aj...@google.com> wrote:
>>>
>>>> I have just seen it randomly occur on presubmits. So I don't have a
>>>> reliable repro, unfortunately.
>>>> It may be a specific environmental issue to the beam1 machine the tests
>>>> ran on?
>>>> https://builds.apache.org/job/beam_PreCommit_Java_Commit/3722/
>>>>
>>>>
>>>> On Thu, Jan 24, 2019 at 8:16 AM Gleb Kanterov <gl...@spotify.com> wrote:
>>>>
>>>>> I'm wondering if anybody can reproduce this issue. The build has
>>>>> failed once because testcontainers didn't pull docker image. If we use
>>>>> caching proxy for docker, it could be a reason for that. I didn't find any
>>>>> similar known issue in testcontainers fixed recently, but just in case, I
>>>>> bumped testcontainers to use never docker-java.
>>>>>
>>>>> https://github.com/apache/beam/pull/7610
>>>>>
>>>>> On Thu, Jan 24, 2019 at 12:27 AM Alex Amato <aj...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Thank you Gleb, appreciate it.
>>>>>>
>>>>>> On Wed, Jan 23, 2019 at 2:40 PM Gleb Kanterov <gl...@spotify.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I'm looking into it. This image exists in docker hub [1], but for
>>>>>>> some reason, it wasn't picked up.
>>>>>>>
>>>>>>> [1] https://hub.docker.com/r/yandex/clickhouse-server/tags
>>>>>>>
>>>>>>> On Wed, Jan 23, 2019 at 10:01 PM Alex Amato <aj...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> 1.
>>>>>>>> See: BEAM-6497
>>>>>>>> <https://jira.apache.org/jira/browse/BEAM-6497>
>>>>>>>> 1. This is also causing issues blocking precommits.
>>>>>>>> 2.
>>>>>>>> Seems to be caused by this failure to locate the image. Are
>>>>>>>> these stored somewhere or built by the build process? Any idea why these
>>>>>>>> are failing?
>>>>>>>>
>>>>>>>> Caused by: com.github.dockerjava.api.exception.NotFoundException: {"message":"No such image: yandex/clickhouse-server:18.10.3"}
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Cheers,
>>>>>>> Gleb
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Cheers,
>>>>> Gleb
>>>>>
>>>>
--
Cheers,
Gleb
Re: ContainerLaunchException in precommit [BEAM-6497]
Posted by Kenneth Knowles <kl...@google.com>.
I retract my statement. I failed at web browsing.
On Tue, Jan 29, 2019 at 3:14 PM Kenneth Knowles <kl...@google.com> wrote:
> Version 18.10.3 no longer appears on the linked page.
>
> On Tue, Jan 29, 2019 at 3:08 PM David Rieber <dr...@google.com> wrote:
>
>> I am consistently hitting that error on this PR:
>> https://github.com/apache/beam/pull/7631
>>
>>
>> On Thu, Jan 24, 2019 at 9:14 AM Alex Amato <aj...@google.com> wrote:
>>
>>> I have just seen it randomly occur on presubmits. So I don't have a
>>> reliable repro, unfortunately.
>>> It may be a specific environmental issue to the beam1 machine the tests
>>> ran on?
>>> https://builds.apache.org/job/beam_PreCommit_Java_Commit/3722/
>>>
>>>
>>> On Thu, Jan 24, 2019 at 8:16 AM Gleb Kanterov <gl...@spotify.com> wrote:
>>>
>>>> I'm wondering if anybody can reproduce this issue. The build has failed
>>>> once because testcontainers didn't pull docker image. If we use caching
>>>> proxy for docker, it could be a reason for that. I didn't find any similar
>>>> known issue in testcontainers fixed recently, but just in case, I bumped
>>>> testcontainers to use never docker-java.
>>>>
>>>> https://github.com/apache/beam/pull/7610
>>>>
>>>> On Thu, Jan 24, 2019 at 12:27 AM Alex Amato <aj...@google.com> wrote:
>>>>
>>>>> Thank you Gleb, appreciate it.
>>>>>
>>>>> On Wed, Jan 23, 2019 at 2:40 PM Gleb Kanterov <gl...@spotify.com>
>>>>> wrote:
>>>>>
>>>>>> I'm looking into it. This image exists in docker hub [1], but for
>>>>>> some reason, it wasn't picked up.
>>>>>>
>>>>>> [1] https://hub.docker.com/r/yandex/clickhouse-server/tags
>>>>>>
>>>>>> On Wed, Jan 23, 2019 at 10:01 PM Alex Amato <aj...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> 1.
>>>>>>> See: BEAM-6497 <https://jira.apache.org/jira/browse/BEAM-6497>
>>>>>>> 1. This is also causing issues blocking precommits.
>>>>>>> 2.
>>>>>>> Seems to be caused by this failure to locate the image. Are
>>>>>>> these stored somewhere or built by the build process? Any idea why these
>>>>>>> are failing?
>>>>>>>
>>>>>>> Caused by: com.github.dockerjava.api.exception.NotFoundException: {"message":"No such image: yandex/clickhouse-server:18.10.3"}
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Cheers,
>>>>>> Gleb
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Cheers,
>>>> Gleb
>>>>
>>>
Re: ContainerLaunchException in precommit [BEAM-6497]
Posted by Kenneth Knowles <kl...@google.com>.
Version 18.10.3 no longer appears on the linked page.
On Tue, Jan 29, 2019 at 3:08 PM David Rieber <dr...@google.com> wrote:
> I am consistently hitting that error on this PR:
> https://github.com/apache/beam/pull/7631
>
>
> On Thu, Jan 24, 2019 at 9:14 AM Alex Amato <aj...@google.com> wrote:
>
>> I have just seen it randomly occur on presubmits. So I don't have a
>> reliable repro, unfortunately.
>> It may be a specific environmental issue to the beam1 machine the tests
>> ran on?
>> https://builds.apache.org/job/beam_PreCommit_Java_Commit/3722/
>>
>>
>> On Thu, Jan 24, 2019 at 8:16 AM Gleb Kanterov <gl...@spotify.com> wrote:
>>
>>> I'm wondering if anybody can reproduce this issue. The build has failed
>>> once because testcontainers didn't pull docker image. If we use caching
>>> proxy for docker, it could be a reason for that. I didn't find any similar
>>> known issue in testcontainers fixed recently, but just in case, I bumped
>>> testcontainers to use never docker-java.
>>>
>>> https://github.com/apache/beam/pull/7610
>>>
>>> On Thu, Jan 24, 2019 at 12:27 AM Alex Amato <aj...@google.com> wrote:
>>>
>>>> Thank you Gleb, appreciate it.
>>>>
>>>> On Wed, Jan 23, 2019 at 2:40 PM Gleb Kanterov <gl...@spotify.com> wrote:
>>>>
>>>>> I'm looking into it. This image exists in docker hub [1], but for some
>>>>> reason, it wasn't picked up.
>>>>>
>>>>> [1] https://hub.docker.com/r/yandex/clickhouse-server/tags
>>>>>
>>>>> On Wed, Jan 23, 2019 at 10:01 PM Alex Amato <aj...@google.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> 1.
>>>>>> See: BEAM-6497 <https://jira.apache.org/jira/browse/BEAM-6497>
>>>>>> 1. This is also causing issues blocking precommits.
>>>>>> 2.
>>>>>> Seems to be caused by this failure to locate the image. Are
>>>>>> these stored somewhere or built by the build process? Any idea why these
>>>>>> are failing?
>>>>>>
>>>>>> Caused by: com.github.dockerjava.api.exception.NotFoundException: {"message":"No such image: yandex/clickhouse-server:18.10.3"}
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Cheers,
>>>>> Gleb
>>>>>
>>>>
>>>
>>> --
>>> Cheers,
>>> Gleb
>>>
>>
Re: ContainerLaunchException in precommit [BEAM-6497]
Posted by David Rieber <dr...@google.com>.
I am consistently hitting that error on this PR:
https://github.com/apache/beam/pull/7631
On Thu, Jan 24, 2019 at 9:14 AM Alex Amato <aj...@google.com> wrote:
> I have just seen it randomly occur on presubmits. So I don't have a
> reliable repro, unfortunately.
> It may be a specific environmental issue to the beam1 machine the tests
> ran on?
> https://builds.apache.org/job/beam_PreCommit_Java_Commit/3722/
>
>
> On Thu, Jan 24, 2019 at 8:16 AM Gleb Kanterov <gl...@spotify.com> wrote:
>
>> I'm wondering if anybody can reproduce this issue. The build has failed
>> once because testcontainers didn't pull docker image. If we use caching
>> proxy for docker, it could be a reason for that. I didn't find any similar
>> known issue in testcontainers fixed recently, but just in case, I bumped
>> testcontainers to use never docker-java.
>>
>> https://github.com/apache/beam/pull/7610
>>
>> On Thu, Jan 24, 2019 at 12:27 AM Alex Amato <aj...@google.com> wrote:
>>
>>> Thank you Gleb, appreciate it.
>>>
>>> On Wed, Jan 23, 2019 at 2:40 PM Gleb Kanterov <gl...@spotify.com> wrote:
>>>
>>>> I'm looking into it. This image exists in docker hub [1], but for some
>>>> reason, it wasn't picked up.
>>>>
>>>> [1] https://hub.docker.com/r/yandex/clickhouse-server/tags
>>>>
>>>> On Wed, Jan 23, 2019 at 10:01 PM Alex Amato <aj...@google.com> wrote:
>>>>
>>>>>
>>>>> 1.
>>>>> See: BEAM-6497 <https://jira.apache.org/jira/browse/BEAM-6497>
>>>>> 1. This is also causing issues blocking precommits.
>>>>> 2.
>>>>> Seems to be caused by this failure to locate the image. Are
>>>>> these stored somewhere or built by the build process? Any idea why these
>>>>> are failing?
>>>>>
>>>>> Caused by: com.github.dockerjava.api.exception.NotFoundException: {"message":"No such image: yandex/clickhouse-server:18.10.3"}
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Cheers,
>>>> Gleb
>>>>
>>>
>>
>> --
>> Cheers,
>> Gleb
>>
>
Re: ContainerLaunchException in precommit [BEAM-6497]
Posted by Alex Amato <aj...@google.com>.
I have just seen it randomly occur on presubmits. So I don't have a
reliable repro, unfortunately.
It may be a specific environmental issue to the beam1 machine the tests ran
on?
https://builds.apache.org/job/beam_PreCommit_Java_Commit/3722/
On Thu, Jan 24, 2019 at 8:16 AM Gleb Kanterov <gl...@spotify.com> wrote:
> I'm wondering if anybody can reproduce this issue. The build has failed
> once because testcontainers didn't pull docker image. If we use caching
> proxy for docker, it could be a reason for that. I didn't find any similar
> known issue in testcontainers fixed recently, but just in case, I bumped
> testcontainers to use never docker-java.
>
> https://github.com/apache/beam/pull/7610
>
> On Thu, Jan 24, 2019 at 12:27 AM Alex Amato <aj...@google.com> wrote:
>
>> Thank you Gleb, appreciate it.
>>
>> On Wed, Jan 23, 2019 at 2:40 PM Gleb Kanterov <gl...@spotify.com> wrote:
>>
>>> I'm looking into it. This image exists in docker hub [1], but for some
>>> reason, it wasn't picked up.
>>>
>>> [1] https://hub.docker.com/r/yandex/clickhouse-server/tags
>>>
>>> On Wed, Jan 23, 2019 at 10:01 PM Alex Amato <aj...@google.com> wrote:
>>>
>>>>
>>>> 1.
>>>> See: BEAM-6497 <https://jira.apache.org/jira/browse/BEAM-6497>
>>>> 1. This is also causing issues blocking precommits.
>>>> 2.
>>>> Seems to be caused by this failure to locate the image. Are
>>>> these stored somewhere or built by the build process? Any idea why these
>>>> are failing?
>>>>
>>>> Caused by: com.github.dockerjava.api.exception.NotFoundException: {"message":"No such image: yandex/clickhouse-server:18.10.3"}
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> Cheers,
>>> Gleb
>>>
>>
>
> --
> Cheers,
> Gleb
>
Re: ContainerLaunchException in precommit [BEAM-6497]
Posted by Gleb Kanterov <gl...@spotify.com>.
I'm wondering if anybody can reproduce this issue. The build has failed
once because testcontainers didn't pull docker image. If we use caching
proxy for docker, it could be a reason for that. I didn't find any similar
known issue in testcontainers fixed recently, but just in case, I bumped
testcontainers to use never docker-java.
https://github.com/apache/beam/pull/7610
On Thu, Jan 24, 2019 at 12:27 AM Alex Amato <aj...@google.com> wrote:
> Thank you Gleb, appreciate it.
>
> On Wed, Jan 23, 2019 at 2:40 PM Gleb Kanterov <gl...@spotify.com> wrote:
>
>> I'm looking into it. This image exists in docker hub [1], but for some
>> reason, it wasn't picked up.
>>
>> [1] https://hub.docker.com/r/yandex/clickhouse-server/tags
>>
>> On Wed, Jan 23, 2019 at 10:01 PM Alex Amato <aj...@google.com> wrote:
>>
>>>
>>> 1.
>>> See: BEAM-6497 <https://jira.apache.org/jira/browse/BEAM-6497>
>>> 1. This is also causing issues blocking precommits.
>>> 2.
>>> Seems to be caused by this failure to locate the image. Are these
>>> stored somewhere or built by the build process? Any idea why these are
>>> failing?
>>>
>>> Caused by: com.github.dockerjava.api.exception.NotFoundException: {"message":"No such image: yandex/clickhouse-server:18.10.3"}
>>>
>>>
>>>
>>>
>>
>> --
>> Cheers,
>> Gleb
>>
>
--
Cheers,
Gleb
Re: ContainerLaunchException in precommit [BEAM-6497]
Posted by Alex Amato <aj...@google.com>.
Thank you Gleb, appreciate it.
On Wed, Jan 23, 2019 at 2:40 PM Gleb Kanterov <gl...@spotify.com> wrote:
> I'm looking into it. This image exists in docker hub [1], but for some
> reason, it wasn't picked up.
>
> [1] https://hub.docker.com/r/yandex/clickhouse-server/tags
>
> On Wed, Jan 23, 2019 at 10:01 PM Alex Amato <aj...@google.com> wrote:
>
>>
>> 1.
>> See: BEAM-6497 <https://jira.apache.org/jira/browse/BEAM-6497>
>> 1. This is also causing issues blocking precommits.
>> 2.
>> Seems to be caused by this failure to locate the image. Are these
>> stored somewhere or built by the build process? Any idea why these are
>> failing?
>>
>> Caused by: com.github.dockerjava.api.exception.NotFoundException: {"message":"No such image: yandex/clickhouse-server:18.10.3"}
>>
>>
>>
>>
>
> --
> Cheers,
> Gleb
>
Re: ContainerLaunchException in precommit [BEAM-6497]
Posted by Gleb Kanterov <gl...@spotify.com>.
I'm looking into it. This image exists in docker hub [1], but for some
reason, it wasn't picked up.
[1] https://hub.docker.com/r/yandex/clickhouse-server/tags
On Wed, Jan 23, 2019 at 10:01 PM Alex Amato <aj...@google.com> wrote:
>
> 1.
> See: BEAM-6497 <https://jira.apache.org/jira/browse/BEAM-6497>
> 1. This is also causing issues blocking precommits.
> 2.
> Seems to be caused by this failure to locate the image. Are these
> stored somewhere or built by the build process? Any idea why these are
> failing?
>
> Caused by: com.github.dockerjava.api.exception.NotFoundException: {"message":"No such image: yandex/clickhouse-server:18.10.3"}
>
>
>
>
--
Cheers,
Gleb