You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dirrao (via GitHub)" <gi...@apache.org> on 2024/08/10 10:23:06 UTC
[PR] spark kubernetes operator arguments description reordering [airflow]
dirrao opened a new pull request, #41372:
URL: https://github.com/apache/airflow/pull/41372
Right now the spark kubernetes operator arguments description order is not same as the args order. This PR aligns the args description order same as args order.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] spark kubernetes operator arguments description reordering [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal merged PR #41372:
URL: https://github.com/apache/airflow/pull/41372
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] spark kubernetes operator arguments description reordering [airflow]
Posted by "romsharon98 (via GitHub)" <gi...@apache.org>.
romsharon98 commented on code in PR #41372:
URL: https://github.com/apache/airflow/pull/41372#discussion_r1712641418
##########
airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py:
##########
@@ -48,24 +48,24 @@ class SparkKubernetesOperator(KubernetesPodOperator):
For more detail about Spark Application Object have a look at the reference:
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/v1beta2-1.3.3-3.1.1/docs/api-docs.md#sparkapplication
- :param application_file: filepath to kubernetes custom_resource_definition of sparkApplication
- :param kubernetes_conn_id: the connection to Kubernetes cluster
:param image: Docker image you wish to launch. Defaults to hub.docker.com,
:param code_path: path to the spark code in image,
:param namespace: kubernetes namespace to put sparkApplication
- :param cluster_context: context of the cluster
- :param application_file: yaml file if passed
+ :param application_file: filepath to kubernetes custom_resource_definition of sparkApplication
+ :param template_spec: kubernetes sparkApplication specification
:param get_logs: get the stdout of the container as logs of the tasks.
:param do_xcom_push: If True, the content of the file
/airflow/xcom/return.json in the container will also be pushed to an
XCom when the container completes.
:param success_run_history_limit: Number of past successful runs of the application to keep.
- :param delete_on_termination: What to do when the pod reaches its final
- state, or the execution is interrupted. If True (default), delete the
- pod; if False, leave the pod.
:param startup_timeout_seconds: timeout in seconds to startup the pod.
:param log_events_on_failure: Log the pod's events if a failure occurs
:param reattach_on_restart: if the scheduler dies while the pod is running, reattach and monitor
+ :param delete_on_termination: What to do when the pod reaches its final
+ state, or the execution is interrupted. If True (default), delete the
+ pod; if False, leave the pod.
+ :param kubernetes_conn_id: the connection to Kubernetes cluster
+ :param cluster_context: context of the cluster
Review Comment:
I think `cluster_context` is part of kwargs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] spark kubernetes operator arguments description reordering [airflow]
Posted by "romsharon98 (via GitHub)" <gi...@apache.org>.
romsharon98 commented on code in PR #41372:
URL: https://github.com/apache/airflow/pull/41372#discussion_r1712642058
##########
airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py:
##########
@@ -48,24 +48,24 @@ class SparkKubernetesOperator(KubernetesPodOperator):
For more detail about Spark Application Object have a look at the reference:
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/v1beta2-1.3.3-3.1.1/docs/api-docs.md#sparkapplication
- :param application_file: filepath to kubernetes custom_resource_definition of sparkApplication
- :param kubernetes_conn_id: the connection to Kubernetes cluster
:param image: Docker image you wish to launch. Defaults to hub.docker.com,
:param code_path: path to the spark code in image,
:param namespace: kubernetes namespace to put sparkApplication
Review Comment:
can we add also `name` to docstring?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] spark kubernetes operator arguments description reordering [airflow]
Posted by "dirrao (via GitHub)" <gi...@apache.org>.
dirrao commented on code in PR #41372:
URL: https://github.com/apache/airflow/pull/41372#discussion_r1712910794
##########
airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py:
##########
@@ -48,24 +48,24 @@ class SparkKubernetesOperator(KubernetesPodOperator):
For more detail about Spark Application Object have a look at the reference:
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/v1beta2-1.3.3-3.1.1/docs/api-docs.md#sparkapplication
- :param application_file: filepath to kubernetes custom_resource_definition of sparkApplication
- :param kubernetes_conn_id: the connection to Kubernetes cluster
:param image: Docker image you wish to launch. Defaults to hub.docker.com,
:param code_path: path to the spark code in image,
:param namespace: kubernetes namespace to put sparkApplication
Review Comment:
I have updated it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] spark kubernetes operator arguments description reordering [airflow]
Posted by "romsharon98 (via GitHub)" <gi...@apache.org>.
romsharon98 commented on code in PR #41372:
URL: https://github.com/apache/airflow/pull/41372#discussion_r1712641418
##########
airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py:
##########
@@ -48,24 +48,24 @@ class SparkKubernetesOperator(KubernetesPodOperator):
For more detail about Spark Application Object have a look at the reference:
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/v1beta2-1.3.3-3.1.1/docs/api-docs.md#sparkapplication
- :param application_file: filepath to kubernetes custom_resource_definition of sparkApplication
- :param kubernetes_conn_id: the connection to Kubernetes cluster
:param image: Docker image you wish to launch. Defaults to hub.docker.com,
:param code_path: path to the spark code in image,
:param namespace: kubernetes namespace to put sparkApplication
- :param cluster_context: context of the cluster
- :param application_file: yaml file if passed
+ :param application_file: filepath to kubernetes custom_resource_definition of sparkApplication
+ :param template_spec: kubernetes sparkApplication specification
:param get_logs: get the stdout of the container as logs of the tasks.
:param do_xcom_push: If True, the content of the file
/airflow/xcom/return.json in the container will also be pushed to an
XCom when the container completes.
:param success_run_history_limit: Number of past successful runs of the application to keep.
- :param delete_on_termination: What to do when the pod reaches its final
- state, or the execution is interrupted. If True (default), delete the
- pod; if False, leave the pod.
:param startup_timeout_seconds: timeout in seconds to startup the pod.
:param log_events_on_failure: Log the pod's events if a failure occurs
:param reattach_on_restart: if the scheduler dies while the pod is running, reattach and monitor
+ :param delete_on_termination: What to do when the pod reaches its final
+ state, or the execution is interrupted. If True (default), delete the
+ pod; if False, leave the pod.
+ :param kubernetes_conn_id: the connection to Kubernetes cluster
+ :param cluster_context: context of the cluster
Review Comment:
I think it part of kwargs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] spark kubernetes operator arguments description reordering [airflow]
Posted by "dirrao (via GitHub)" <gi...@apache.org>.
dirrao commented on code in PR #41372:
URL: https://github.com/apache/airflow/pull/41372#discussion_r1712910742
##########
airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py:
##########
@@ -48,24 +48,24 @@ class SparkKubernetesOperator(KubernetesPodOperator):
For more detail about Spark Application Object have a look at the reference:
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/v1beta2-1.3.3-3.1.1/docs/api-docs.md#sparkapplication
- :param application_file: filepath to kubernetes custom_resource_definition of sparkApplication
- :param kubernetes_conn_id: the connection to Kubernetes cluster
:param image: Docker image you wish to launch. Defaults to hub.docker.com,
:param code_path: path to the spark code in image,
:param namespace: kubernetes namespace to put sparkApplication
- :param cluster_context: context of the cluster
- :param application_file: yaml file if passed
+ :param application_file: filepath to kubernetes custom_resource_definition of sparkApplication
+ :param template_spec: kubernetes sparkApplication specification
:param get_logs: get the stdout of the container as logs of the tasks.
:param do_xcom_push: If True, the content of the file
/airflow/xcom/return.json in the container will also be pushed to an
XCom when the container completes.
:param success_run_history_limit: Number of past successful runs of the application to keep.
- :param delete_on_termination: What to do when the pod reaches its final
- state, or the execution is interrupted. If True (default), delete the
- pod; if False, leave the pod.
:param startup_timeout_seconds: timeout in seconds to startup the pod.
:param log_events_on_failure: Log the pod's events if a failure occurs
:param reattach_on_restart: if the scheduler dies while the pod is running, reattach and monitor
+ :param delete_on_termination: What to do when the pod reaches its final
+ state, or the execution is interrupted. If True (default), delete the
+ pod; if False, leave the pod.
+ :param kubernetes_conn_id: the connection to Kubernetes cluster
+ :param cluster_context: context of the cluster
Review Comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org