You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/06/15 14:06:13 UTC

[GitHub] [airflow] CodingJonas opened a new issue #9310: Delay until Docker Swarm recognizes finished service

CodingJonas opened a new issue #9310:
URL: https://github.com/apache/airflow/issues/9310


   <!--
   
   **Apache Airflow version**: 2.0.0dev
   - **OS** (e.g. from /etc/os-release): Ubuntu 18.04
   - **Install tools**: pip
   
   **What happened**:
   When running a DockerSwarmTask the service finishes, but Airflow will detect that finished service only 60 seconds after it has finished.
   
   The issue lies in this line:
   https://github.com/apache/airflow/blob/832593a9fc80f4b56605f3cbf656375bd94cc136/airflow/providers/docker/operators/docker_swarm.py#L171
   ```
       def _stream_logs_to_output(self):
           logs = self.cli.service_logs(
               self.service['ID'], follow=True, stdout=True, stderr=True, is_tty=self.tty
           )
           line = ''
           while True:
               try:
                   log = next(logs)
   ```
   When removing this it works as expected. `next(logs)` is a blocking call, and for some reason the docker-py library, which is behind this call, does not recognize the finished service.
   
   **How to reproduce it**:
   Any dag using the DockerSwarmOperator works, e.g.:
   ```
       task1 = DockerSwarmOperator(
           task_id='docker_swarm_validator',
           image='alpine:3.11.5',
           api_version='auto',
           command='echo X',
           tty=True,
       )
   ```
   
   **Anything else we need to know**:
   
   A workaround I found was to execute the logging in a separate process, while checking for the current status of the service in the main process. Once the service has finished, the logging process can simply be terminated.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] CodingJonas edited a comment on issue #9310: Delay until Docker Swarm recognizes finished service

Posted by GitBox <gi...@apache.org>.
CodingJonas edited a comment on issue #9310:
URL: https://github.com/apache/airflow/issues/9310#issuecomment-913696551


   Sorry for not following up on my workaround, we moved to Kubernetes, so we did not finish polishing the workaround properly. I can still add the main code parts we used to try to fix it. Perhaps it helps you!
   
   ```python
       def _run_service(self):
           ...
           if self.enable_logging:
               service_log_stream = self.cli.service_logs(self.service['ID'], follow=True, stdout=True, stderr=True, is_tty=self.tty)
               _start_logging_async(self.log, service_log_stream)
           ...
   
   def _start_logging_async(logger, service_log_stream):
       """
       The logging task is blocking and thus stops the operator from recognizing in time when the service finishes.
       Since the logging thread is demonized, it will automatically be terminated once the main script finishes.
       """
       p = Thread(target=_stream_logs_to_output,
                  args=(logger, service_log_stream),
                  daemon=True)
       p.start()
   
   def _stream_logs_to_output(logger, logs):
       line = ''
       while True:
           try:
               log = next(logs)
           # TODO: Remove this clause once https://github.com/docker/docker-py/issues/931 is fixed
           except requests.exceptions.ConnectionError:
               logger.info("Connection Error while fetching log")
               # If the service log stream stopped sending messages, check if it the service has
               # terminated.
               break
           except StopIteration:
               logger.info("StopIteration while fetching log")
               # If the service log stream terminated, stop fetching logs further.
               break
           else:
               try:
                   log = log.decode()
               except UnicodeDecodeError:
                   continue
               if log == '\n':
                   logger.info(line)
                   line = ''
               else:
                   line += log
       # flush any remaining log stream
       if line:
           logger.info(line)
   ```
   
   The only addition we did is wrapping the `_stream_logs_to_output` method in a Thread.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] nullhack commented on issue #9310: Delay until Docker Swarm recognizes finished service

Posted by GitBox <gi...@apache.org>.
nullhack commented on issue #9310:
URL: https://github.com/apache/airflow/issues/9310#issuecomment-647134923


   Sure, I'll put on my list, after reviewing the other PR


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] CodingJonas commented on issue #9310: Delay until Docker Swarm recognizes finished service

Posted by GitBox <gi...@apache.org>.
CodingJonas commented on issue #9310:
URL: https://github.com/apache/airflow/issues/9310#issuecomment-662872273


   That's a good point, I see that I open an issue on their side today and come around to polish my changes for a at least WIP merge request to see if my current solution is feasible :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] mik-laj commented on issue #9310: Delay until Docker Swarm recognizes finished service

Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #9310:
URL: https://github.com/apache/airflow/issues/9310#issuecomment-644266039


   @nullhack @retornam @CodingJonas @CaptainCuddleCube @nalepae @akki Can you look at it? It may be interesting for you.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] CodingJonas commented on issue #9310: Delay until Docker Swarm recognizes finished service

Posted by GitBox <gi...@apache.org>.
CodingJonas commented on issue #9310:
URL: https://github.com/apache/airflow/issues/9310#issuecomment-647367141


   I currently using the workaround with a subprocess without issues. Not saying it is perfect, but I can make a pull request out of if, since it seems to me like a sufficient solution.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] eladkal commented on issue #9310: Delay until Docker Swarm recognizes finished service

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #9310:
URL: https://github.com/apache/airflow/issues/9310#issuecomment-833507495


   I'm not quite sure if this is an Airflow issue?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] retornam commented on issue #9310: Delay until Docker Swarm recognizes finished service

Posted by GitBox <gi...@apache.org>.
retornam commented on issue #9310:
URL: https://github.com/apache/airflow/issues/9310#issuecomment-663269105


   @CodingJonas you are hitting a known bug in `docker-py` where is `is_tty` is set and there isn't any streaming output for about 60 seconds, the connection is closed  see https://github.com/docker/docker-py/issues/931#issuecomment-545336374, there is a fix for it  in https://github.com/docker/docker-py/pull/1959


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] CodingJonas commented on issue #9310: Delay until Docker Swarm recognizes finished service

Posted by GitBox <gi...@apache.org>.
CodingJonas commented on issue #9310:
URL: https://github.com/apache/airflow/issues/9310#issuecomment-913696551


   Sorry for not following up on my workaround, we moved to Kubernetes, so we did not finish polishing the workaround properly. I can still add the main code parts we used to try to fix it. Perhaps it helps you!
   
   ```python
       def _run_service(self):
           ...
           if self.enable_logging:
               service_log_stream = self.cli.service_logs(self.service['ID'], follow=True, stdout=True, stderr=True, is_tty=self.tty)
               _start_logging_async(self.log, service_log_stream)
           ...
   
   def _start_logging_async(logger, service_log_stream):
       """
       The logging task is blocking and thus stops the operator from recognizing in time when the service finishes.
       Since the logging thread is demonized, it will automatically be terminated once the main script finishes.
       """
       p = Thread(target=_stream_logs_to_output,
                  args=(logger, service_log_stream),
                  daemon=True)
       p.start()
   
   def _stream_logs_to_output(logger, logs):
       line = ''
       while True:
           try:
               log = next(logs)
           # TODO: Remove this clause once https://github.com/docker/docker-py/issues/931 is fixed
           except requests.exceptions.ConnectionError:
               logger.info("Connection Error while fetching log")
               # If the service log stream stopped sending messages, check if it the service has
               # terminated.
               break
           except StopIteration:
               logger.info("StopIteration while fetching log")
               # If the service log stream terminated, stop fetching logs further.
               break
           else:
               try:
                   log = log.decode()
               except UnicodeDecodeError:
                   continue
               if log == '\n':
                   logger.info(line)
                   line = ''
               else:
                   line += log
       # flush any remaining log stream
       if line:
           logger.info(line)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] eladkal closed issue #9310: Delay until Docker Swarm recognizes finished service

Posted by GitBox <gi...@apache.org>.
eladkal closed issue #9310:
URL: https://github.com/apache/airflow/issues/9310


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] akki edited a comment on issue #9310: Delay until Docker Swarm recognizes finished service

Posted by GitBox <gi...@apache.org>.
akki edited a comment on issue #9310:
URL: https://github.com/apache/airflow/issues/9310#issuecomment-662863371


   @CodingJonas Just curious if you asked upstream why `docker-py` behaves that way?
   It might be worth asking them to know if they can fix the root cause itself.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] akki commented on issue #9310: Delay until Docker Swarm recognizes finished service

Posted by GitBox <gi...@apache.org>.
akki commented on issue #9310:
URL: https://github.com/apache/airflow/issues/9310#issuecomment-662863371


   @CodingJonas Just curious if you asked upstream why `docker-py` behaves that way?
   It might be worth asking them to know if they can fix it.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] enima2684 commented on issue #9310: Delay until Docker Swarm recognizes finished service

Posted by GitBox <gi...@apache.org>.
enima2684 commented on issue #9310:
URL: https://github.com/apache/airflow/issues/9310#issuecomment-895893308


   The issue https://github.com/docker/docker-py/issues/931  is still not yet fixed and the DockerSwarmOperator still have the 60s delay before finishing a completed service.
   
   Any updated workaround you are using to prevent this delay ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org