You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/10/18 10:45:55 UTC
[GitHub] [airflow] dinigo opened a new pull request, #27108: Create path for downloaded dbt artefact (#27107)
dinigo opened a new pull request, #27108:
URL: https://github.com/apache/airflow/pull/27108
closes: #27107
related: #27107
As mentioned in the Issue #27107, when downloading a dbt cloud artefact to a path with parents not yet created it (logically) fails, as this is the default behaviour. However integration with mounted fs like s3 and gcs benefit from descriptive path names, and further exploration benefits from having those names split in keys (folders). This is a default use case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27108: Create path for downloaded dbt artefact (#27107)
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27108:
URL: https://github.com/apache/airflow/pull/27108#discussion_r1009000377
##########
airflow/providers/dbt/cloud/operators/dbt.py:
##########
@@ -212,7 +214,7 @@ def execute(self, context: Context) -> None:
response = hook.get_job_run_artifact(
run_id=self.run_id, path=self.path, account_id=self.account_id, step=self.step
)
-
+ Path(self.output_file_name).parent.mkdir(parents=True, exist_ok=True)
Review Comment:
yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed pull request #27108: Create path for downloaded dbt artefact (#27107)
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #27108: Create path for downloaded dbt artefact (#27107)
URL: https://github.com/apache/airflow/pull/27108
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #27108: Create path for downloaded dbt artefact (#27107)
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #27108:
URL: https://github.com/apache/airflow/pull/27108#issuecomment-1364759352
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27108: Create path for downloaded dbt artefact (#27107)
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #27108:
URL: https://github.com/apache/airflow/pull/27108#issuecomment-1289734647
Needs statuc checks fixing (and rebase ideally)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27108: Create path for downloaded dbt artefact (#27107)
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #27108:
URL: https://github.com/apache/airflow/pull/27108#issuecomment-1296487928
Also a test for this would be necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #27108: Create path for downloaded dbt artefact (#27107)
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #27108:
URL: https://github.com/apache/airflow/pull/27108#discussion_r1004165706
##########
airflow/providers/dbt/cloud/operators/dbt.py:
##########
@@ -212,7 +214,7 @@ def execute(self, context: Context) -> None:
response = hook.get_job_run_artifact(
run_id=self.run_id, path=self.path, account_id=self.account_id, step=self.step
)
-
+ Path(self.output_file_name).parent.mkdir(parents=True, exist_ok=True)
Review Comment:
If we’re using pathlib, it’d be a good idea to also rewrite the code below to reuse the same Path object instead of throwing it away.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org