You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/02/06 20:57:41 UTC
[GitHub] [airflow] blcksrx opened a new pull request #21327: [WIP] SparkKubernetesOperator
blcksrx opened a new pull request #21327:
URL: https://github.com/apache/airflow/pull/21327
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
related: #21325
There are two issues on `SparkKubernetesOperator` listed here:
1. based on this [question](https://stackoverflow.com/questions/70902485/airflow-sparkkubernetesoperator-logging) on stackoverflow, the operator does not watch the whole process of the job unlike other operator/transformations that monitors the job and change status of the task based on that.
2. It needs to write `on_kill` method on the operator in order to kill spark driver and executors in case of receiving `SIGTERM` signal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #21327: [WIP] SparkKubernetesOperator
Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #21327:
URL: https://github.com/apache/airflow/pull/21327
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21327: [WIP] SparkKubernetesOperator
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21327:
URL: https://github.com/apache/airflow/pull/21327#issuecomment-1040172316
Is still in progress?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21327: [WIP] SparkKubernetesOperator
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21327:
URL: https://github.com/apache/airflow/pull/21327#issuecomment-1040172316
Is still in progress?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] blcksrx closed pull request #21327: [WIP] SparkKubernetesOperator
Posted by GitBox <gi...@apache.org>.
blcksrx closed pull request #21327:
URL: https://github.com/apache/airflow/pull/21327
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] blcksrx commented on pull request #21327: [WIP] SparkKubernetesOperator
Posted by GitBox <gi...@apache.org>.
blcksrx commented on pull request #21327:
URL: https://github.com/apache/airflow/pull/21327#issuecomment-1040296936
Actually I can't test it with MiniKube, I got 400 and 404 error on it with K8s version 1.20.7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] blcksrx commented on pull request #21327: [WIP] SparkKubernetesOperator
Posted by GitBox <gi...@apache.org>.
blcksrx commented on pull request #21327:
URL: https://github.com/apache/airflow/pull/21327#issuecomment-1040296936
Actually I can't test it with MiniKube, I got 400 and 404 error on it with K8s version 1.20.7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] blcksrx commented on pull request #21327: override on_kill method on SparkKubernetesOperator
Posted by GitBox <gi...@apache.org>.
blcksrx commented on pull request #21327:
URL: https://github.com/apache/airflow/pull/21327#issuecomment-1030255623
Thanks for the review. actually I checked the test files there were not unittest for similar methods on the hook side such as `create_custom_object` and `get_custom_object`. Im going to write tests for all of them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] blcksrx removed a comment on pull request #21327: WIP: SparkKubernetesOperator
Posted by GitBox <gi...@apache.org>.
blcksrx removed a comment on pull request #21327:
URL: https://github.com/apache/airflow/pull/21327#issuecomment-1030860838
@potiuk there are several problems in the `SparkKubernetesOperator` that I'm going to fix it and update the issue for that to make clarifications
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] blcksrx commented on pull request #21327: WIP: SparkKubernetesOperator
Posted by GitBox <gi...@apache.org>.
blcksrx commented on pull request #21327:
URL: https://github.com/apache/airflow/pull/21327#issuecomment-1030860838
@potiuk there are several problems in the `SparkKubernetesOperator` that I'm going to fix it and update the issue for that to make clarifications
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21327: override on_kill method on SparkKubernetesOperator
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21327:
URL: https://github.com/apache/airflow/pull/21327#issuecomment-1030249083
It looks cool. but is it possible to add a unit test for it ?
BTW. One small comment - unlike Unit tests, the Issues are not really needed with PRs like that :). PR is more than enough actually. We are not beauraucrats :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org