You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/05/08 10:01:04 UTC
[GitHub] [airflow] JavierLopezT opened a new pull request #15736: Log info about query execution SnowflakeHook
JavierLopezT opened a new pull request #15736:
URL: https://github.com/apache/airflow/pull/15736
This modification will log useful info about the execution. Some examples are:
```
Statement execution info - {'status': 'Statement executed successfully.'}
Statement Execution Info - {'status': 'Table STG_GOOGLE_ADS_HOURLY_CAMPAIGN successfully created.'}
Statement execution info - {'status': 'Copy executed with 0 files processed.'}
Statement Execution Info - {'number of rows deleted': 0}
Statement Execution Info - {'number of rows inserted': 0}
```
Also, these values are returned so that we can further implement behavior modifications in the Operators. For instance, in S3ToSnowflakeOperator, if the status is `Copy executed with 0 files processed we could raise an error`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #15736: Logging and returning info about query execution SnowflakeHook
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #15736:
URL: https://github.com/apache/airflow/pull/15736#issuecomment-835339434
I guess we can also add it as value returned by SnowflakeOperator's execute method ? It's backwards compatible, and with do_xcom_push = True (default) it will automatically create XCom with the output. Might be really useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #15736: Logging and returning info about query execution SnowflakeHook
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #15736:
URL: https://github.com/apache/airflow/pull/15736#issuecomment-872092759
> > Seems that execution_info are "rows" returned by Cursor which is not json-serializable.
> > This PyMSQL shows an example how you can do it: https://programmersought.com/article/12771376188/
>
> I made a workaround eventually. The real output is a list of dicts, so it is serializable with real data. The error was due to the mocking I think.
I see. Yeah . Plausible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on pull request #15736: Logging and returning info about query execution SnowflakeHook
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on pull request #15736:
URL: https://github.com/apache/airflow/pull/15736#issuecomment-871243050
> I guess we can also add it as value returned by SnowflakeOperator's execute method ? It's backwards compatible, and with do_xcom_push = True (default) it will automatically create XCom with the output. Might be really useful.
Done :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #15736: Log info about query execution SnowflakeHook
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #15736:
URL: https://github.com/apache/airflow/pull/15736#issuecomment-835296424
[The Workflow run](https://github.com/apache/airflow/actions/runs/822880621) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on pull request #15736: Logging and returning info about query execution SnowflakeHook
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on pull request #15736:
URL: https://github.com/apache/airflow/pull/15736#issuecomment-872066459
> Seems that execution_info are "rows" returned by Cursor which is not json-serializable.
>
> This PyMSQL shows an example how you can do it: https://programmersought.com/article/12771376188/
I made a workaround eventually. The real output is a list of dicts, so it is serializable with real data. The error was due to the mocking I think.
Everything green now, I hope it can be merged soon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #15736: Logging and returning info about query execution SnowflakeHook
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #15736:
URL: https://github.com/apache/airflow/pull/15736
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #15736: Logging and returning info about query execution SnowflakeHook
Posted by GitBox <gi...@apache.org>.
uranusjr commented on pull request #15736:
URL: https://github.com/apache/airflow/pull/15736#issuecomment-871407511
It means the value you want to push cannot be serialised into JSON. What is the value that caused the failure?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #15736: Logging and returning info about query execution SnowflakeHook
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #15736:
URL: https://github.com/apache/airflow/pull/15736#issuecomment-871509066
Seems that execution_info are "rows" returned by Cursor which is not json-serializable.
This PyMSQL shows an example how you can do it: https://programmersought.com/article/12771376188/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT edited a comment on pull request #15736: Logging and returning info about query execution SnowflakeHook
Posted by GitBox <gi...@apache.org>.
JavierLopezT edited a comment on pull request #15736:
URL: https://github.com/apache/airflow/pull/15736#issuecomment-871243050
> I guess we can also add it as value returned by SnowflakeOperator's execute method ? It's backwards compatible, and with do_xcom_push = True (default) it will automatically create XCom with the output. Might be really useful.
Done :)
Edit: it seems that the test of the XCom is failing
`Could not serialize the XCom value into JSON. If you are using pickle instead of JSON for XCom, then you need to enable pickle support for XCom in your airflow config.`
Any idea how to solve it? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org