You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/04/22 10:24:37 UTC
[GitHub] [airflow] tanuj-ahuja opened a new issue #15485: Support for View/Export third party artifacts from Airflow
tanuj-ahuja opened a new issue #15485:
URL: https://github.com/apache/airflow/issues/15485
**Description**
Enable an option to download or view artifacts like log files, reports, execution detail files, etc. generated by third party tool people run on Airflow.
**Use case / motivation**
We encountered this use-case while using Airflow to run our [Data Build Tool](https://www.getdbt.com/) jobs via the [BashOperator](https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/bash.html). For more information [refer](https://www.astronomer.io/blog/airflow-dbt-1).
Now when these dbt commands are run they generate [dbt-artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts) inside the project repository. We wish to be able view or download these artifacts from the Airflow UI.
Why we need dbt artifacts?
1. Calculate project-level test coverage
2. Perform longitudinal analysis of run timing
3. Identify historical changes in table structure
4. Do much, much more
This was just one example with dbt but it can be extended to any other third party tool that generates its own artifacts. So the reach of this feature could be to a wider set of Airflow users in my opinion.
-->
**Are you willing to submit a PR?**
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tanuj-ahuja commented on issue #15485: Support for View/Export third party artifacts from Airflow
Posted by GitBox <gi...@apache.org>.
tanuj-ahuja commented on issue #15485:
URL: https://github.com/apache/airflow/issues/15485#issuecomment-843089047
@potiuk These artifacts are usually log files or some other files which were generated during the DAG run.
The idea to store these files names as XCom sounds good, since it will allow us to tie them up with the DAG runs + the "download as file" option would be nice to have as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15485: Support for View/Export third party artifacts from Airflow
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15485:
URL: https://github.com/apache/airflow/issues/15485#issuecomment-843069999
A good feature might be to have a convenience method to store set of files as XCom (automatically reading the files and using file names as keys). Also adding a "download" as file feature in XCom UI might provide a convenient way to interact with those via UI.
That might be a nice, small PR for Airflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tanuj-ahuja commented on issue #15485: Support for View/Export third party artifacts from Airflow
Posted by GitBox <gi...@apache.org>.
tanuj-ahuja commented on issue #15485:
URL: https://github.com/apache/airflow/issues/15485#issuecomment-825386780
@ashb Thank you for your response.
The above shared link shows how to add an extra link to Airflow but we want to be able to view or download specific files from Airflow UI.
Just to be clear these files are generated within the host machine/container running Airflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #15485: Support for View/Export third party artifacts from Airflow
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15485:
URL: https://github.com/apache/airflow/issues/15485#issuecomment-843069999
A good feature might be to have a convenience method to store set of files as XCom (automatically reading the files and using file names as keys). Also adding a "download as file" feature in XCom UI might provide a convenient way to interact with those via UI.
That might be a nice, small PR for Airflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tanuj-ahuja commented on issue #15485: Support for View/Export third party artifacts from Airflow
Posted by GitBox <gi...@apache.org>.
tanuj-ahuja commented on issue #15485:
URL: https://github.com/apache/airflow/issues/15485#issuecomment-843099831
Yeah, totally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #15485: Support for View/Export third party artifacts from Airflow
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15485:
URL: https://github.com/apache/airflow/issues/15485#issuecomment-843067448
I guess those artifacts are not huge, those are just meta-data. Can their content be read and stored as XCom values? Sounds like good way to persistently store them and link them with the actual runs.
You can then query the XCom and retrieve the manfest and other artifacts content via the XCom API (on top of the existing ability of viewing them via UI):
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_xcom_entry
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tanuj-ahuja commented on issue #15485: Support for View/Export third party artifacts from Airflow
Posted by GitBox <gi...@apache.org>.
tanuj-ahuja commented on issue #15485:
URL: https://github.com/apache/airflow/issues/15485#issuecomment-843061895
@mik-laj Yeah, but can we have it as new feature in Airflow?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #15485: Support for View/Export third party artifacts from Airflow
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #15485:
URL: https://github.com/apache/airflow/issues/15485#issuecomment-826116935
@tanuj-ahuja I am afraid that Airflow does not have any generic mechanism for storing artifacts and we rely on each integration to upload the relevant artifacts by itself, eg Dataflow operators save artifacts in GCS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15485: Support for View/Export third party artifacts from Airflow
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15485:
URL: https://github.com/apache/airflow/issues/15485#issuecomment-843067448
I guess those artifacts are not huge, those are just meta-data. Can they be read and stored as XCom parameters? Sounds like good way to persistently store them and link them with the actual runs.
You can then query the XCom and retrieve the manfest and other artifacts content via the XCom API:
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_xcom_entry
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed issue #15485: Support for View/Export third party artifacts from Airflow
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #15485:
URL: https://github.com/apache/airflow/issues/15485
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #15485: Support for View/Export third party artifacts from Airflow
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #15485:
URL: https://github.com/apache/airflow/issues/15485#issuecomment-863636513
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #15485: Support for View/Export third party artifacts from Airflow
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #15485:
URL: https://github.com/apache/airflow/issues/15485#issuecomment-824870724
Check out https://airflow.apache.org/docs/apache-airflow/stable/howto/define_extra_link.html?highlight=operator%20links -- does that give you the hook points you are after?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #15485: Support for View/Export third party artifacts from Airflow
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #15485:
URL: https://github.com/apache/airflow/issues/15485#issuecomment-868890614
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #15485: Support for View/Export third party artifacts from Airflow
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #15485:
URL: https://github.com/apache/airflow/issues/15485#issuecomment-824722496
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #15485: Support for View/Export third party artifacts from Airflow
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #15485:
URL: https://github.com/apache/airflow/issues/15485#issuecomment-843067448
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15485: Support for View/Export third party artifacts from Airflow
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15485:
URL: https://github.com/apache/airflow/issues/15485#issuecomment-843095236
I think it really depends on how big those files are. When using default DB XCom backend, there are some limits, but when you use custom XCom backends - https://www.astronomer.io/guides/custom-xcom-backends - you can store as much as you want.
Maybe you would like to work on a PR for that @tanuj-ahuja ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #15485: Support for View/Export third party artifacts from Airflow
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #15485:
URL: https://github.com/apache/airflow/issues/15485#issuecomment-826116935
@tanuj-ahuja I am afraid that Airflow does not have any generic mechanism for storing artifacts and we rely on each integration to upload the relevant artifacts by itself, eg Dataflow operators save artifacts in GCS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org