You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "David (JIRA)" <ji...@apache.org> on 2019/05/16 18:42:00 UTC
[jira] [Comment Edited] (AIRFLOW-4077) Using gevent worker class
breaks reading remote log from s3
[ https://issues.apache.org/jira/browse/AIRFLOW-4077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16841610#comment-16841610 ]
David edited comment on AIRFLOW-4077 at 5/16/19 6:41 PM:
---------------------------------------------------------
I can confirm this behavior, though I'm switching the worker class on all my containers. I'm using celery on 1.10.2 and only changing the worker_class back to sync from gevent fixed my s3 log reading.
was (Author: caddac):
I can confirm this behavior. I'm using celery on 1.10.2 and only changing the worker_class back to sync from gevent fixed my s3 log reading.
> Using gevent worker class breaks reading remote log from s3
> -----------------------------------------------------------
>
> Key: AIRFLOW-4077
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4077
> Project: Apache Airflow
> Issue Type: Bug
> Components: logging, webapp, worker
> Affects Versions: 1.10.2
> Environment: Local - Ubuntu
> Production - AWS Elastic Beanstalk
> Reporter: Sanjay Hallan
> Assignee: Tom
> Priority: Major
> Labels: S3, gevent, logging, worker
>
> We have been writing our airflow logs to S3 for quite sometime now successfully, using the worker class 'sync' which is the default. Viewing logs via the airflow UI works perfectly, in the sense that logs are read from the remote s3 location.
> When switching the worker class to gevent, the airflow UI no longer attempts to read the logs from the remote s3 location, rather it always attempts to read the logs from the file and as we are using a containerized environment reading log files in this way fails.
> Switching only the worker class back to sync and leaving all other airflow configuration untouched resolves this issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)