You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Aizhamal Nurmamat kyzy (JIRA)" <ji...@apache.org> on 2019/05/18 02:33:00 UTC
[jira] [Updated] (AIRFLOW-4150) Modify the docker operator
implementation
[ https://issues.apache.org/jira/browse/AIRFLOW-4150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aizhamal Nurmamat kyzy updated AIRFLOW-4150:
--------------------------------------------
Labels: docker (was: )
> Modify the docker operator implementation
> -----------------------------------------
>
> Key: AIRFLOW-4150
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4150
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators
> Reporter: aaronluo
> Priority: Major
> Labels: docker
>
> 1. I create a test python script testpython.py for docker
> {quote}import time
> time.sleep(1000)
> {quote}
>
> 2. I create a DAG with a task that calls the script through a docker
> {quote}docker_ls = DockerOperator(
> task_id='docker_ls',
> image='python',
> working_dir = '/data/wf/',
> command='python testpython.py',
> docker_url='http://192.168.1.215:2375',
> start_date=datetime(2015, 6, 1),
> volumes = ['/data/wf:/data/wf/'],
> dag=dag
> )
> {quote}
>
> 3. When I run this DAG, obviously, celery worker will be working for a very long time,
> In addition, docker container will also run for a long time。
> {quote}for line in self.cli.logs(container=self.container['Id'], stream=True):
> line = line.strip()
> if hasattr(line, 'decode'):
> line = line.decode('utf-8')
> self.log.info(line)
> result = self.cli.wait(self.container['Id'])
> if result['StatusCode'] != 0:
> raise AirflowException('docker container failed: ' + repr(result)){quote}
>
> My suggestion is that after submitting the task to docker, celery's corresponding worker should end up monitoring the docker's events, rather than blocking them all the time, because the events take a long time to execute.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)