docker-compose run does not return on Jenkins

[Francis Chuang <fr...@apache.org>]

Hey everyone,

I am currently setting up automated website builds for Apache Calcite. 
We are currently using docker-compose and the jekyll image to generate 
the assets on our machines locally: 
https://github.com/apache/calcite/blob/test-site/site/docker-compose.yml

This works well and `docker-compose run ` command returns after the 
generation is completed.

The problem is that when we use `docker-compose run ...` to generate the 
assets on Jenkins, it never returns. We are using this Jenkinsfile: 
https://github.com/apache/calcite/blob/test-site/Jenkinsfile

The logs showing the generating being successful, but not terminating 
and manually aborted are here: 
https://builds.apache.org/job/Calcite-Site/9/console

I have tried using the bash and sh interpreters when running 
`docker-compose run`, but it does not make any difference.

Does anyone know what might be the cause?

Francis

Re: docker-compose run does not return on Jenkins

[Francis Chuang <fr...@apache.org>]

Hi Karl,

The `docker-compose run` command should terminate and return once the 
assets are generated. We are not using docker-compose to spin up a bunch 
of long running processes, but rather a one-off build/command. I can 
confirm that on my local machine, the job terminates once the command 
finishes. The exit code is also 0.

This problem only seems occur on the Jenkins nodes. As a test, I got rid 
of docker-compose and ran `docker run --rm alpine whoami` and it never 
exits. I also tried disabling the pseudo-tty in docker-compose using 
`docker-compose run -T ...`, but it also does not return when the 
command in the container finishes.

The version of docker-compose installed on the websites1 node is the 
latest version, but the version of docker is a bit old, although I don't 
think this is the cause.

I am not sure what's the cause. We could fall back of using `-d`: 
`docker-compose run -d ...`, and then run an infinite loop to check if 
the command in the docker-container has finished, but this is quite a 
hack and I'd prefer to avoid this.

Francis

On 10/06/2019 11:53 pm, Karl Heinz Marbaise wrote:
> Hi,
> 
> On 10.06.19 01:45, Francis Chuang wrote:
>> Hey everyone,
>>
>> I am currently setting up automated website builds for Apache Calcite.
>> We are currently using docker-compose and the jekyll image to generate
>> the assets on our machines locally:
>> https://github.com/apache/calcite/blob/test-site/site/docker-compose.yml
>>
>> This works well and `docker-compose run ` command returns after the
>> generation is completed.
> 
> A docker-compose run ... or docker run will run not in daemon mode which
> means the calling process is blocked until you press ctrl-c ...
> 
> The first thing I would suggest to use is:
> 
> docker-compose run -d ...
> 
> -d => Daemon mode which run all processes in background.
> 
> This will not block the whole pipeline from running...
> 
> But be aware that your consuming services must be handling that the
> starting process may take time to be up and running
> correctly...(resilence..)
> 
> 
> Kidn regards
> Karl Heinz Marbaise
>>
>> The problem is that when we use `docker-compose run ...` to generate the
>> assets on Jenkins, it never returns. We are using this Jenkinsfile:
>> https://github.com/apache/calcite/blob/test-site/Jenkinsfile
>>
>> The logs showing the generating being successful, but not terminating
>> and manually aborted are here:
>> https://builds.apache.org/job/Calcite-Site/9/console
>>
>> I have tried using the bash and sh interpreters when running
>> `docker-compose run`, but it does not make any difference.
>>
>> Does anyone know what might be the cause?
>>
>> Francis
>>

Re: docker-compose run does not return on Jenkins

[Karl Heinz Marbaise <kh...@gmx.de>]

Hi,

On 10.06.19 01:45, Francis Chuang wrote:
> Hey everyone,
>
> I am currently setting up automated website builds for Apache Calcite.
> We are currently using docker-compose and the jekyll image to generate
> the assets on our machines locally:
> https://github.com/apache/calcite/blob/test-site/site/docker-compose.yml
>
> This works well and `docker-compose run ` command returns after the
> generation is completed.

A docker-compose run ... or docker run will run not in daemon mode which
means the calling process is blocked until you press ctrl-c ...

The first thing I would suggest to use is:

docker-compose run -d ...

-d => Daemon mode which run all processes in background.

This will not block the whole pipeline from running...

But be aware that your consuming services must be handling that the
starting process may take time to be up and running
correctly...(resilence..)


Kidn regards
Karl Heinz Marbaise
>
> The problem is that when we use `docker-compose run ...` to generate the
> assets on Jenkins, it never returns. We are using this Jenkinsfile:
> https://github.com/apache/calcite/blob/test-site/Jenkinsfile
>
> The logs showing the generating being successful, but not terminating
> and manually aborted are here:
> https://builds.apache.org/job/Calcite-Site/9/console
>
> I have tried using the bash and sh interpreters when running
> `docker-compose run`, but it does not make any difference.
>
> Does anyone know what might be the cause?
>
> Francis
>