You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/08/04 10:32:05 UTC

[GitHub] [airflow] vijayakumarpattanashetti opened a new pull request #17408: Fix issue #17371

vijayakumarpattanashetti opened a new pull request #17408:
URL: https://github.com/apache/airflow/pull/17408


   closes: #17371
   
   Fixed the bug with SparkKubernetesOperator in loading the file.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] vijayakumarpattanashetti commented on pull request #17408: Fix the issue #17371 to load the file contents

Posted by GitBox <gi...@apache.org>.

vijayakumarpattanashetti commented on pull request #17408:
URL: https://github.com/apache/airflow/pull/17408#issuecomment-896521385


   > Wait no, I don’t think this is right. `application_file` can be either a path _or a JSON string_, so just passing it to `open` is wrong.
   
   No, `application_file` should only take the path to the files with the extensions `template_ext = ('.yaml', '.yml', '.json')`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] uranusjr commented on a change in pull request #17408: Fix the issue #17371 to load the file contents

Posted by GitBox <gi...@apache.org>.

uranusjr commented on a change in pull request #17408:
URL: https://github.com/apache/airflow/pull/17408#discussion_r682504788



##########
File path: airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py
##########
@@ -71,7 +71,7 @@ def execute(self, context):
             group=self.api_group,
             version=self.api_version,
             plural="sparkapplications",
-            body=self.application_file,
+            body=open(self.application_file).read(),

Review comment:
       Don’t do this. Use either `with open(...)` or `pathlib.Path(...).read_text()` instead.
   
   Also, please specify `encoding="utf8")`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #17408: SparkKubernetesOperator fix bug with loading application_file

Posted by GitBox <gi...@apache.org>.

github-actions[bot] commented on pull request #17408:
URL: https://github.com/apache/airflow/pull/17408#issuecomment-968395993


   This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] github-actions[bot] closed pull request #17408: SparkKubernetesOperator fix bug with loading application_file

Posted by GitBox <gi...@apache.org>.

github-actions[bot] closed pull request #17408:
URL: https://github.com/apache/airflow/pull/17408


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #17408: Fix issue #17371

Posted by GitBox <gi...@apache.org>.

boring-cyborg[bot] commented on pull request #17408:
URL: https://github.com/apache/airflow/pull/17408#issuecomment-892549274

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on pull request #17408: SparkKubernetesOperator fix bug with loading application_file

Posted by GitBox <gi...@apache.org>.

eladkal commented on pull request #17408:
URL: https://github.com/apache/airflow/pull/17408#issuecomment-931320020


   @vijayakumarpattanashetti static checks and tests are still failing. can you please address it and rebase?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #17408: Fix the issue #17371 to load the file contents

Posted by GitBox <gi...@apache.org>.

github-actions[bot] commented on pull request #17408:
URL: https://github.com/apache/airflow/pull/17408#issuecomment-896517214


   The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] vijayakumarpattanashetti commented on pull request #17408: Fix the issue #17371 to load the file contents

Posted by GitBox <gi...@apache.org>.

vijayakumarpattanashetti commented on pull request #17408:
URL: https://github.com/apache/airflow/pull/17408#issuecomment-895457498


   Hi @uranusjr , modified the code. Kindly review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] uranusjr commented on a change in pull request #17408: Fix the issue #17371 to load the file contents

Posted by GitBox <gi...@apache.org>.

uranusjr commented on a change in pull request #17408:
URL: https://github.com/apache/airflow/pull/17408#discussion_r686507785



##########
File path: airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py
##########
@@ -67,11 +67,12 @@ def __init__(
     def execute(self, context):
         self.log.info("Creating sparkApplication")
         hook = KubernetesHook(conn_id=self.kubernetes_conn_id)
-        response = hook.create_custom_object(
-            group=self.api_group,
-            version=self.api_version,
-            plural="sparkapplications",
-            body=self.application_file,
-            namespace=self.namespace,
-        )
-        return response
+        with open(self.application_file, 'r', encoding='utf-8') as app_file:

Review comment:
       The `r` part is not technically needed, but this is also fine.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] uranusjr commented on a change in pull request #17408: Fix the issue #17371 to load the file contents

Posted by GitBox <gi...@apache.org>.

uranusjr commented on a change in pull request #17408:
URL: https://github.com/apache/airflow/pull/17408#discussion_r682504788



##########
File path: airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py
##########
@@ -71,7 +71,7 @@ def execute(self, context):
             group=self.api_group,
             version=self.api_version,
             plural="sparkapplications",
-            body=self.application_file,
+            body=open(self.application_file).read(),

Review comment:
       Don’t do this. Use either `with open(...)` or `pathlib.Path(...).read_text()` instead.
   
   Also, please specify `encoding="utf8")`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on pull request #17408: SparkKubernetesOperator fix bug with loading application_file

Posted by GitBox <gi...@apache.org>.

eladkal commented on pull request #17408:
URL: https://github.com/apache/airflow/pull/17408#issuecomment-905329007


   @vijayakumarpattanashetti static checks are failing. can you please fix it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] uranusjr commented on pull request #17408: Fix the issue #17371 to load the file contents

Posted by GitBox <gi...@apache.org>.

uranusjr commented on pull request #17408:
URL: https://github.com/apache/airflow/pull/17408#issuecomment-896554782


   That’s not what the documentation says.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #17408: Fix the issue #17371 to load the file contents

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #17408:
URL: https://github.com/apache/airflow/pull/17408#issuecomment-898959984


   You need to rebase to latest master


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #17408: Fix issue #17371

Posted by GitBox <gi...@apache.org>.

boring-cyborg[bot] commented on pull request #17408:
URL: https://github.com/apache/airflow/pull/17408#issuecomment-892549274

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org