You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/22 08:28:36 UTC
[GitHub] [airflow] tuanchris opened a new pull request #13246: Add OracleToGCS Transfer
tuanchris opened a new pull request #13246:
URL: https://github.com/apache/airflow/pull/13246
This PR add the OracleToGCS transfer to airflow.providers.google.cloud.transfers. This is my first PR, please let me know if any aspect could be improved.
Thank you!
Tuan
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tuanchris commented on pull request #13246: Add OracleToGCS Transfer
Posted by GitBox <gi...@apache.org>.
tuanchris commented on pull request #13246:
URL: https://github.com/apache/airflow/pull/13246#issuecomment-751142226
@potiuk Thank you for your help! Do let me know if there's any issue/feature I can help with!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #13246: Add OracleToGCS Transfer
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #13246:
URL: https://github.com/apache/airflow/pull/13246#issuecomment-751305166
Cool @tuanchris ! Glad you liked it :)
You can look at the [good-first-issue](https://github.com/apache/airflow/labels/good%20first%20issue) label - but also you can take a look at the "Airflow 2.0" cleanup [milestone](https://github.com/apache/airflow/milestones/Airflow%202.0%20clean-up) - those are things we always wanted to do and are important but not urgent (and most of them require some but not very deep knowledge of Airflow). Happy to help with any of those !.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #13246: Add OracleToGCS Transfer
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #13246:
URL: https://github.com/apache/airflow/pull/13246#issuecomment-750881004
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest master or amend the last commit of the PR, and push it with --force-with-lease.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #13246: Add OracleToGCS Transfer
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #13246:
URL: https://github.com/apache/airflow/pull/13246#issuecomment-749945362
First quick view looks good. The tests will have to be fixed (both static checks - I heartily recommend to install pre-commit loclly and run it see https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst) as well as the unit tests.
What could be really useful as well is adding an example_dag and HowTo Guide for the operator. You can see plenty of examples in Google Provider, the nice thing is that you can use pieces of the example dag in the HowToGuide.
Finally (though this is not a requirement) the example dag can be turned into a "system test" https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-system-tests - we have not yet automated them, but soon we will. The example dag could be simply made into a runnable example that could perform end-2-end test of the operator. This would require connection to the actual Oracle DB though (credentials might be stored in environment variable).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #13246: Add OracleToGCS Transfer
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #13246:
URL: https://github.com/apache/airflow/pull/13246#issuecomment-750881081
Awesome work, congrats on your first merged pull request!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #13246: Add OracleToGCS Transfer
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #13246:
URL: https://github.com/apache/airflow/pull/13246#issuecomment-749945362
First quick view looks good. The tests will have to be fixed (both static checks - I heartily recommend to install pre-commit loclly and run it see https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst) as well as the tests. What could be really useful as well is adding an example_dag and HowTo Guide for the operator. You can see plenty of examples in Google Provider, the nice thing is that you can use pieces of the example dag in the HowToGuide.
Finally (though this is not a requirement) the example dag can be turned into a "system test" https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-system-tests - we have not yet automated them, but soon we will. The example dag could be simply made into a runnable example that could perform end-2-end test of the operator. This would require connection to the actual Oracle DB though (credentials might be stored in environment variable).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #13246: Add OracleToGCS Transfer
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #13246:
URL: https://github.com/apache/airflow/pull/13246#issuecomment-749945362
First quick view looks good. The tests will have to be fixed (both static checks - I heartily recommend to install pre-commit locally and run it automatically before commits. See https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst) as well as the unit tests.
What could be really useful as well is adding an example_dag and HowTo Guide for the operator. You can see plenty of examples in Google Provider, the nice thing is that you can use pieces of the example dag in the HowToGuide.
Finally (though this is not a requirement) the example dag can be turned into a "system test" https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-system-tests - we have not yet automated them, but soon we will. The example dag could be simply made into a runnable example that could perform end-2-end test of the operator. This would require connection to the actual Oracle DB though (credentials might be stored in environment variable).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #13246: Add OracleToGCS Transfer
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #13246:
URL: https://github.com/apache/airflow/pull/13246#issuecomment-749416023
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing locally, itโs a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better ๐.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #13246: Add OracleToGCS Transfer
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #13246:
URL: https://github.com/apache/airflow/pull/13246
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tuanchris edited a comment on pull request #13246: Add OracleToGCS Transfer
Posted by GitBox <gi...@apache.org>.
tuanchris edited a comment on pull request #13246:
URL: https://github.com/apache/airflow/pull/13246#issuecomment-750097302
Hi @potiuk, thank you for the guidance. I have been able to:
- fix the unit test
- add howto doc
- add and example dag
I'm still struggling to get the static test to pass, though. Running pre-commit test locally is taking too long for me (stuck on pylint). Any tips on how to speed up the tests locally?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tuanchris commented on pull request #13246: Add OracleToGCS Transfer
Posted by GitBox <gi...@apache.org>.
tuanchris commented on pull request #13246:
URL: https://github.com/apache/airflow/pull/13246#issuecomment-750097302
Hi @potiuk, thank you for the guidance. I have been able to:
- fix the unit test
- add howto doc
- add and example dag
I'm still struggling to get the static test to pass, though. Running pre-commit test locally is taking too long for me (stuck on pylint). Any tips on how to speed up the tests locally?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #13246: Add OracleToGCS Transfer
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #13246:
URL: https://github.com/apache/airflow/pull/13246#issuecomment-750105942
It will take a while but you can run pre-commit for all files.
By default when you install pre-commit with pre-commit install, it will only run the tests on staged files. But if you already have the commit you can run it only on the files changed in last commit by running `pre-commit run --from-ref HEAD^ --to-ref HEAD`
This will limit the checks to only files that were modified in the last commit (yours) .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org