You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Kamil Bregula (Jira)" <ji...@apache.org> on 2019/09/10 09:42:00 UTC

[jira] [Comment Edited] (AIRFLOW-1754) Add GCP logging download for Dataflow operator

    [ https://issues.apache.org/jira/browse/AIRFLOW-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16926476#comment-16926476 ] 

Kamil Bregula edited comment on AIRFLOW-1754 at 9/10/19 9:41 AM:
-----------------------------------------------------------------

Hi.

This will be copying a large amount of data that is already available to the user. In special cases, this will involve double billing, because these logs will be available in two places if the user uses Cloud Composer. What do you think about the introduction of extra link mechanisms that will allow you to debug the problem easily from the GCP console?  
 [https://airflow.readthedocs.io/en/latest/howto/define_extra_link.html]

Cheers


was (Author: kamil.bregula):
Hi.

 

This will be copying a large amount of data that is already available to the user. In special cases, this will involve double billing, because these logs will be available in two places if the user uses Cloud Composer. What do you think about the introduction of extra link mechanisms that will allow you to debug the problem easily from the GCP console?  
[https://airflow.readthedocs.io/en/latest/howto/define_extra_link.html]

> Add GCP logging download for Dataflow operator
> ----------------------------------------------
>
>                 Key: AIRFLOW-1754
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1754
>             Project: Apache Airflow
>          Issue Type: New Feature
>          Components: gcp
>            Reporter: Chris Riccomini
>            Priority: Major
>             Fix For: 1.10.0
>
>
> Based on conversation in AIRFLOW-1732 and https://github.com/apache/incubator-airflow/pull/2702, there is useful logging that occurs for Dataflow on the server-side (i.e. it's not visible simply by piping client logs to the Airflow log file).
> We should add a method to fetch logs from GCP logging (stack driver?), so we can spool server side logging into the Dataflow operator for debugging purposes.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)