You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/26 20:53:41 UTC
[GitHub] [airflow] millertracy opened a new pull request #10591: Added Plexus as an Airflow provider
millertracy opened a new pull request #10591:
URL: https://github.com/apache/airflow/pull/10591
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #10591: Added Plexus as an Airflow provider
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #10591:
URL: https://github.com/apache/airflow/pull/10591#issuecomment-689730993
@potiuk thank you! I will make those changes. will i also need to add backward compatibility for airflow 1.10* by adding plexus hook/operator to the airflow.contrib package?
No need. By adding it to the setup properly, backport provider is automatically created and tested, so no need to do anything.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #10591: Added Plexus as an Airflow provider
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #10591:
URL: https://github.com/apache/airflow/pull/10591#discussion_r483921549
##########
File path: setup.py
##########
@@ -687,6 +687,7 @@ def is_package_excluded(package: str, exclusion_list: List[str]):
INSTALL_REQUIREMENTS = [
'alembic>=1.2, <2.0',
'argcomplete~=1.10',
+ 'arrow==0.16.0',
Review comment:
You should create a "plexus" extra and add "arrow" to the extra , rather than to install requirements.
##########
File path: airflow/providers/plexus/hooks/plexus.py
##########
@@ -0,0 +1,73 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import jwt
+import arrow
+import requests
+from airflow.hooks.base_hook import BaseHook
+from airflow.models import Variable
+from airflow.exceptions import AirflowException
+
+
+class PlexusHook(BaseHook):
+ """
+ Used for jwt token generation and storage to
+ make Plexus API calls. Requires email and password
+ Airflow variables be created.
+
+ Example:
+ - export AIRFLOW_VAR_EMAIL = user@corescientific.com
+ - export AIRFLOW_VAR_PASSWORD = *******
+
+ """
+
+ def __init__(self) -> None:
+ super().__init__()
+ self.__token = None
+ self.__token_exp = None
+ self.host = "https://apiplexus.corescientific.com/"
+ self.user_id = None
+
+ def _generate_token(self):
+ login = Variable.get("email")
+ pwd = Variable.get("password")
+ if login is None or pwd is None:
+ raise AirflowException("No valid email/password supplied.")
+ token_endpoint = self.host + "sso/jwt-token/"
+ response = requests.post(token_endpoint, data={"email": login, "password": pwd}, timeout=5)
+ if not response.ok:
+ raise AirflowException(
+ "Could not retrieve JWT Token. Status Code: [{}]. "
+ "Reason: {} - {}".format(response.status_code, response.reason, response.text)
+ )
+ token = response.json()["access"]
+ payload = jwt.decode(token, verify=False)
+ self.user_id = payload["user_id"]
+ self.__token_exp = payload["exp"]
+
+ return token
+
+ @property
Review comment:
Nice!
##########
File path: tests/providers/plexus/operators/test_plexus.py
##########
@@ -0,0 +1,189 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import mock
+import pytest
+from mock import Mock
+from requests.exceptions import Timeout
+from airflow.exceptions import AirflowException
+from airflow.providers.plexus.operators.job import PlexusJobOperator
+
+
+class TestPlexusOperator:
+ @pytest.fixture
Review comment:
Nice! We do not use pytest fixtures too much (we allow both unit tests style and pytest ones). But I think it's good to have more examples of nice pytest fixtures.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #10591: Added Plexus as an Airflow provider
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #10591:
URL: https://github.com/apache/airflow/pull/10591#issuecomment-690581454
Awesome work, congrats on your first merged pull request!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] millertracy edited a comment on pull request #10591: Added Plexus as an Airflow provider
Posted by GitBox <gi...@apache.org>.
millertracy edited a comment on pull request #10591:
URL: https://github.com/apache/airflow/pull/10591#issuecomment-689725958
> Hey @millertracy -> reviewed it. Nice piece of provider :).
>
> The main comment is about the settup.py change (plexus needs 'plexus' extra and arrow should be added there - also then plexus extra should be added to "all_devs/all" extras similarly to other extras.
>
> Also then you will need to mark it as "provider" dependency in PROVIDERS_REQUIREMENTS.
>
> This should likely fix backport package problem that you have (airflow 1.10.10 does not have arrow as requirement, neither the generated backport package).
>
> There is also some work to make the docs work - some spellcheck problems from what I see.
>
> One more NIT - It would also be great if you create a HowTo out of your example. You can see how other operators do that, but basically it is a very short "Howto" doc that extracts part of the "example_dag" and adds some description on what the operator does.
>
> Another part (but this is something that we have no full CI automation yet) is to turn the example dag into a runnable "system test". This is described in https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-system-tests - and it's not required, but it seems to be easy to do in your case as this is rather simple example you have.
>
> We plan to automate system tests later this year and it would be great to have such easy-to-run system test in place. But this one is nice-to-havee.
>
> This is actually the most powerful part of the exmple_dags
@potiuk thank you! I will make those changes. will i also need to add backward compatibility for airflow 1.10* by adding plexus hook/operator to the airflow.contrib package?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #10591: Added Plexus as an Airflow provider
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #10591:
URL: https://github.com/apache/airflow/pull/10591#issuecomment-681118065
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing locally, itโs a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better ๐.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://apache-airflow-slack.herokuapp.com/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] millertracy commented on pull request #10591: Added Plexus as an Airflow provider
Posted by GitBox <gi...@apache.org>.
millertracy commented on pull request #10591:
URL: https://github.com/apache/airflow/pull/10591#issuecomment-689833072
@potiuk finished changes. is there anything else that needs to be done?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] millertracy commented on pull request #10591: Added Plexus as an Airflow provider
Posted by GitBox <gi...@apache.org>.
millertracy commented on pull request #10591:
URL: https://github.com/apache/airflow/pull/10591#issuecomment-689737309
> > @potiuk thank you! I will make those changes. will i also need to add backward compatibility for airflow 1.10* by adding plexus hook/operator to the airflow.contrib package?
>
> No need. By adding it to the setup properly, backport provider is automatically created and tested, so no need to do anything.
That's great. Double checking though - i will need to reference the 1.10* compatibility in the README.md like the other providers correct? By referencing "Moved Hooks and Operators"?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] millertracy commented on pull request #10591: Added Plexus as an Airflow provider
Posted by GitBox <gi...@apache.org>.
millertracy commented on pull request #10591:
URL: https://github.com/apache/airflow/pull/10591#issuecomment-689725958
> Hey @millertracy -> reviewed it. Nice piece of provider :).
>
> The main comment is about the settup.py change (plexus needs 'plexus' extra and arrow should be added there - also then plexus extra should be added to "all_devs/all" extras similarly to other extras.
>
> Also then you will need to mark it as "provider" dependency in PROVIDERS_REQUIREMENTS.
>
> This should likely fix backport package problem that you have (airflow 1.10.10 does not have arrow as requirement, neither the generated backport package).
>
> There is also some work to make the docs work - some spellcheck problems from what I see.
>
> One more NIT - It would also be great if you create a HowTo out of your example. You can see how other operators do that, but basically it is a very short "Howto" doc that extracts part of the "example_dag" and adds some description on what the operator does.
>
> Another part (but this is something that we have no full CI automation yet) is to turn the example dag into a runnable "system test". This is described in https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-system-tests - and it's not required, but it seems to be easy to do in your case as this is rather simple example you have.
>
> We plan to automate system tests later this year and it would be great to have such easy-to-run system test in place. But this one is nice-to-havee.
>
> This is actually the most powerful part of the exmple_dags
@potiuk will i also need to add backward compatibility for airflow 1.10* by adding plexus hook/operator to the airflow.contrib package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] millertracy commented on pull request #10591: Added Plexus as an Airflow provider
Posted by GitBox <gi...@apache.org>.
millertracy commented on pull request #10591:
URL: https://github.com/apache/airflow/pull/10591#issuecomment-690531779
@potiuk thanks for approving changes. 2 checks were cancelled, but it doesn't look related to my code. When should i expect a merge into the base branch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #10591: Added Plexus as an Airflow provider
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #10591:
URL: https://github.com/apache/airflow/pull/10591#issuecomment-690031637
Almost perfect! Just remove the README file. We are going to generate one automatically when we release next wave of backports.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #10591: Added Plexus as an Airflow provider
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #10591:
URL: https://github.com/apache/airflow/pull/10591
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] millertracy edited a comment on pull request #10591: Added Plexus as an Airflow provider
Posted by GitBox <gi...@apache.org>.
millertracy edited a comment on pull request #10591:
URL: https://github.com/apache/airflow/pull/10591#issuecomment-689737309
> > @potiuk thank you! I will make those changes. will i also need to add backward compatibility for airflow 1.10* by adding plexus hook/operator to the airflow.contrib package?
>
> No need. By adding it to the setup properly, backport provider is automatically created and tested, so no need to do anything.
@potiuk That's great. Double checking though - i will need to reference the 1.10* compatibility in the README.md like the other providers correct? By referencing "Moved Hooks and Operators"?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #10591: Added Plexus as an Airflow provider
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #10591:
URL: https://github.com/apache/airflow/pull/10591#issuecomment-689730993
> @potiuk thank you! I will make those changes. will i also need to add backward compatibility for airflow 1.10* by adding plexus hook/operator to the airflow.contrib package?
No need. By adding it to the setup properly, backport provider is automatically created and tested, so no need to do anything.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] millertracy closed pull request #10591: Added Plexus as an Airflow provider
Posted by GitBox <gi...@apache.org>.
millertracy closed pull request #10591:
URL: https://github.com/apache/airflow/pull/10591
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] millertracy closed pull request #10591: Added Plexus as an Airflow provider
Posted by GitBox <gi...@apache.org>.
millertracy closed pull request #10591:
URL: https://github.com/apache/airflow/pull/10591
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org