You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/18 09:07:10 UTC
[GitHub] [airflow] swiftomkar opened a new pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query
swiftomkar opened a new pull request #7688:
URL: https://github.com/apache/airflow/pull/7688
---
Issue link: [AIRFLOW-6794](https://issues.apache.org/jira/browse/AIRFLOW-6794)
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN = JIRA ID<sup>*</sup>
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
<sup>*</sup> For document-only changes commit message can start with `[AIRFLOW-XXXX]`.
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle commented on pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query
Posted by GitBox <gi...@apache.org>.
feluelle commented on pull request #7688:
URL: https://github.com/apache/airflow/pull/7688#issuecomment-747960377
Please go ahead @swiftomkar :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle commented on pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query
Posted by GitBox <gi...@apache.org>.
feluelle commented on pull request #7688:
URL: https://github.com/apache/airflow/pull/7688#issuecomment-753846074
> One thing I am not sure is which way is more straightforward/intuitive for a dev using this operator? Thoughts?
Exactly. I am in favour of keeping it as it is now - having separate args **and making sure to document it**. Can you please add a note about it in the docs?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on pull request #7688:
URL: https://github.com/apache/airflow/pull/7688#issuecomment-749212949
I would combine schema, table and custom_query_select in one argument: table_or_query, so that if you pass a table you would have to specify the schema also like schema.table in the argument. WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query
Posted by GitBox <gi...@apache.org>.
eladkal commented on pull request #7688:
URL: https://github.com/apache/airflow/pull/7688#issuecomment-788251164
Closing in favour of https://github.com/apache/airflow/pull/14177
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7688:
URL: https://github.com/apache/airflow/pull/7688#issuecomment-749570855
[The Workflow run](https://github.com/apache/airflow/actions/runs/438185564) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] swiftomkar edited a comment on pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query
Posted by GitBox <gi...@apache.org>.
swiftomkar edited a comment on pull request #7688:
URL: https://github.com/apache/airflow/pull/7688#issuecomment-747764156
Hi, Can this PR be reopened again? I would like to finish this @feluelle
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on pull request #7688:
URL: https://github.com/apache/airflow/pull/7688#issuecomment-629126259
I would add at the beggining:
template_fields = ('s3_key', 'select_custom_query',)
template_ext = ('.sql',)
So that you can place the custom query in a file and also use s3_key and custom query with templates
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal closed pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query
Posted by GitBox <gi...@apache.org>.
eladkal closed pull request #7688:
URL: https://github.com/apache/airflow/pull/7688
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] swiftomkar commented on pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query
Posted by GitBox <gi...@apache.org>.
swiftomkar commented on pull request #7688:
URL: https://github.com/apache/airflow/pull/7688#issuecomment-749235656
@JavierLopezT I like the idea. If the parameter is formatted as `schema.table`, it is not a custom query else it is interpreted as a custom query.
One thing I am not sure is which way is more straightforward/intuitive for a dev using this operator? Thoughts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] swiftomkar commented on pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query
Posted by GitBox <gi...@apache.org>.
swiftomkar commented on pull request #7688:
URL: https://github.com/apache/airflow/pull/7688#issuecomment-747764156
Hi, Can this PR be reopened again? I would like to finish this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] stale[bot] closed pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query
Posted by GitBox <gi...@apache.org>.
stale[bot] closed pull request #7688:
URL: https://github.com/apache/airflow/pull/7688
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7688:
URL: https://github.com/apache/airflow/pull/7688#issuecomment-749296225
[The Workflow run](https://github.com/apache/airflow/actions/runs/437041931) is cancelling this PR. Building images for the PR has failed. Follow the the workflow link to check the reason.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] stale[bot] commented on pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query
Posted by GitBox <gi...@apache.org>.
stale[bot] commented on pull request #7688:
URL: https://github.com/apache/airflow/pull/7688#issuecomment-652838412
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org