You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "jack (JIRA)" <ji...@apache.org> on 2019/08/04 11:29:00 UTC

[jira] [Commented] (AIRFLOW-1713) Return results optionally from spark_sql_hook

    [ https://issues.apache.org/jira/browse/AIRFLOW-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16899615#comment-16899615 ] 

jack commented on AIRFLOW-1713:
-------------------------------

Is there something to do here? 

> Return results optionally from spark_sql_hook
> ---------------------------------------------
>
>                 Key: AIRFLOW-1713
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1713
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: contrib
>    Affects Versions: 1.8.1
>            Reporter: Boris Tyukin
>            Assignee: Boris Tyukin
>            Priority: Minor
>
> spark_sql_hook is very useful to execute Spark SQL queries without much of boilerplate with spark-submit but right now it is not possible to capture execution results. One example if one wants to run "select count(1) from table" and get that count back to airflow dag. To address that, optional argument with regex expression is added to spark_sql_hook that will capture all spark log output matching regex and return back to airflow.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)