You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/27 11:32:09 UTC
[GitHub] [airflow] lordozb opened a new pull request #10599: Added jupytercmd and fixed task failure when notify set as true in qubole operator
lordozb opened a new pull request #10599:
URL: https://github.com/apache/airflow/pull/10599
Closes [#9252](https://github.com/apache/airflow/issues/9252)
**Description**
This PR adds jupytercmd in Qubole Operator which fires a JupyterNotebookCommand to the jupyter notebooks running on user's QDS account. Along with this, we have fixed a minor bug which caused the tasks to fail with _--notify_ is set in Qubole Operator.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lordozb commented on pull request #10599: Added jupytercmd and fixed task failure when notify set as true in qubole operator
Posted by GitBox <gi...@apache.org>.
lordozb commented on pull request #10599:
URL: https://github.com/apache/airflow/pull/10599#issuecomment-683269151
@chattarajoy
Master branch already has 1.16.0 version of qds-sdk
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] msumit merged pull request #10599: Added jupytercmd and fixed task failure when notify set as true in qubole operator
Posted by GitBox <gi...@apache.org>.
msumit merged pull request #10599:
URL: https://github.com/apache/airflow/pull/10599
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #10599: Added jupytercmd and fixed task failure when notify set as true in qubole operator
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #10599:
URL: https://github.com/apache/airflow/pull/10599#issuecomment-686370727
Awesome work, congrats on your first merged pull request!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] chattarajoy commented on pull request #10599: Added jupytercmd and fixed task failure when notify set as true in qubole operator
Posted by GitBox <gi...@apache.org>.
chattarajoy commented on pull request #10599:
URL: https://github.com/apache/airflow/pull/10599#issuecomment-682991015
Upgrade the version of qds sdk to latest in the requirements.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #10599: Added jupytercmd and fixed task failure when notify set as true in qubole operator
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #10599:
URL: https://github.com/apache/airflow/pull/10599#issuecomment-681891745
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing locally, itโs a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better ๐.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://apache-airflow-slack.herokuapp.com/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] chattarajoy commented on a change in pull request #10599: Added jupytercmd and fixed task failure when notify set as true in qubole operator
Posted by GitBox <gi...@apache.org>.
chattarajoy commented on a change in pull request #10599:
URL: https://github.com/apache/airflow/pull/10599#discussion_r479453510
##########
File path: airflow/providers/qubole/hooks/qubole.py
##########
@@ -239,12 +241,12 @@ def create_cmd_args(self, context):
inplace_args = value
elif key == 'tags':
self._add_tags(tags, value)
+ elif key == 'notify':
+ if value is True: # pylint: disable=too-many-nested-blocks
Review comment:
Add a test case for this?
##########
File path: airflow/providers/qubole/operators/qubole.py
##########
@@ -106,13 +106,15 @@ class QuboleOperator(BaseOperator):
script_location is supplied
sparkcmd:
:program: the complete Spark Program in Scala, R, or Python
- :cmdline: spark-submit command line, all required information must be specify
+ :cmdline: spark-submit command line, all required arguments must be specify
in cmdline itself.
:sql: inline sql query
:script_location: s3 location containing query statement
:language: language of the program, Scala, R, or Python
:app_id: ID of an Spark job server app
- :arguments: spark-submit command line arguments
+ :arguments: spark-submit command line arguments.
+ If `cmdline` is selected, this should not be used because all
+ required arguments and configurations are to be passed in the `cmdline` itself.
:user_program_arguments: arguments that the user program takes in
:macros: macro values which were used in query
Review comment:
Add parameters for jupytercmd as well
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org