You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "shohamy7 (via GitHub)" <gi...@apache.org> on 2023/12/25 20:23:15 UTC
[PR] change spark connection form and add spark connections docs [airflow]
shohamy7 opened a new pull request, #36419:
URL: https://github.com/apache/airflow/pull/36419
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
Related: #28790
This PR aims to add docs about each Spark connection in a separate page.
In addition, adding a more user-friendly connection form for Spark and Spark SQL connection
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] change spark connection form and add spark connections docs [airflow]
Posted by "shohamy7 (via GitHub)" <gi...@apache.org>.
shohamy7 commented on code in PR #36419:
URL: https://github.com/apache/airflow/pull/36419#discussion_r1436359527
##########
docs/apache-airflow-providers-apache-spark/connections/spark-sql.rst:
##########
@@ -0,0 +1,48 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+.. _howto/connection:spark-sql:
+
+Apache Spark Sql Connection
Review Comment:
fix it :)
##########
docs/apache-airflow-providers-apache-spark/connections/spark-sql.rst:
##########
@@ -0,0 +1,48 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+.. _howto/connection:spark-sql:
+
+Apache Spark Sql Connection
Review Comment:
fixed it :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] change spark connection form and add spark connections docs [airflow]
Posted by "shohamy7 (via GitHub)" <gi...@apache.org>.
shohamy7 commented on PR #36419:
URL: https://github.com/apache/airflow/pull/36419#issuecomment-1869107822
@Taragolis
Please review, would like to here what you are think on this PR :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] change spark connection form and add spark connections docs [airflow]
Posted by "josh-fell (via GitHub)" <gi...@apache.org>.
josh-fell commented on code in PR #36419:
URL: https://github.com/apache/airflow/pull/36419#discussion_r1436608468
##########
airflow/providers/apache/spark/hooks/spark_submit.py:
##########
@@ -98,10 +98,42 @@ class SparkSubmitHook(BaseHook, LoggingMixin):
def get_ui_field_behaviour(cls) -> dict[str, Any]:
"""Return custom field behaviour."""
return {
- "hidden_fields": ["schema", "login", "password"],
+ "hidden_fields": ["schema", "login", "password", "extra"],
"relabeling": {},
}
+ @classmethod
+ def get_connection_form_widgets(cls) -> dict[str, Any]:
+ """Returns connection widgets to add to connection form."""
+ from flask_appbuilder.fieldwidgets import BS3TextFieldWidget
+ from flask_babel import lazy_gettext
+ from wtforms import StringField
+ from wtforms.validators import Optional, any_of
+
+ return {
+ "queue": StringField(
+ lazy_gettext("YARN queue"),
+ widget=BS3TextFieldWidget(),
+ description="Default YARN queue to use",
+ validators=[Optional()],
+ ),
+ "deploy-mode": StringField(
+ lazy_gettext("Deploy mode"),
+ widget=BS3TextFieldWidget(),
+ description="Must be client or cluster",
+ validators=[any_of(["client", "cluster"])],
+ ),
+ "spark-binary": StringField(
+ lazy_gettext("Spark binary"),
+ widget=BS3TextFieldWidget(),
+ description=f"Must be one of: {', '.join(ALLOWED_SPARK_BINARIES)}",
+ validators=[any_of(ALLOWED_SPARK_BINARIES)],
Review Comment:
WDYT about making these [`SelectField`](https://wtforms.readthedocs.io/en/2.3.x/fields/#wtforms.fields.SelectField)s instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] change spark connection form and add spark connections docs [airflow]
Posted by "bolkedebruin (via GitHub)" <gi...@apache.org>.
bolkedebruin commented on code in PR #36419:
URL: https://github.com/apache/airflow/pull/36419#discussion_r1436918236
##########
docs/apache-airflow-providers-apache-spark/connections/spark.rst:
##########
@@ -19,15 +19,15 @@
.. _howto/connection:spark:
Review Comment:
Lets rename the file as then to spark-submit.rst
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] change spark connection form and add spark connections docs [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal merged PR #36419:
URL: https://github.com/apache/airflow/pull/36419
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] change spark connection form and add spark connections docs [airflow]
Posted by "shohamy7 (via GitHub)" <gi...@apache.org>.
shohamy7 commented on code in PR #36419:
URL: https://github.com/apache/airflow/pull/36419#discussion_r1436621647
##########
airflow/providers/apache/spark/hooks/spark_submit.py:
##########
@@ -98,10 +98,42 @@ class SparkSubmitHook(BaseHook, LoggingMixin):
def get_ui_field_behaviour(cls) -> dict[str, Any]:
"""Return custom field behaviour."""
return {
- "hidden_fields": ["schema", "login", "password"],
+ "hidden_fields": ["schema", "login", "password", "extra"],
"relabeling": {},
}
+ @classmethod
+ def get_connection_form_widgets(cls) -> dict[str, Any]:
+ """Returns connection widgets to add to connection form."""
+ from flask_appbuilder.fieldwidgets import BS3TextFieldWidget
+ from flask_babel import lazy_gettext
+ from wtforms import StringField
+ from wtforms.validators import Optional, any_of
+
+ return {
+ "queue": StringField(
+ lazy_gettext("YARN queue"),
+ widget=BS3TextFieldWidget(),
+ description="Default YARN queue to use",
+ validators=[Optional()],
+ ),
+ "deploy-mode": StringField(
+ lazy_gettext("Deploy mode"),
+ widget=BS3TextFieldWidget(),
+ description="Must be client or cluster",
+ validators=[any_of(["client", "cluster"])],
+ ),
+ "spark-binary": StringField(
+ lazy_gettext("Spark binary"),
+ widget=BS3TextFieldWidget(),
+ description=f"Must be one of: {', '.join(ALLOWED_SPARK_BINARIES)}",
+ validators=[any_of(ALLOWED_SPARK_BINARIES)],
Review Comment:
I've tried but it won't get rendered.
It looks like there is a restriction on which field we can use.
https://github.com/apache/airflow/blob/73d87945e9a78195278d0a4b495483062ddc9b35/airflow/providers_manager.py#L948
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] change spark connection form and add spark connections docs [airflow]
Posted by "shohamy7 (via GitHub)" <gi...@apache.org>.
shohamy7 commented on code in PR #36419:
URL: https://github.com/apache/airflow/pull/36419#discussion_r1436848168
##########
airflow/providers/apache/spark/hooks/spark_submit.py:
##########
@@ -98,10 +98,42 @@ class SparkSubmitHook(BaseHook, LoggingMixin):
def get_ui_field_behaviour(cls) -> dict[str, Any]:
"""Return custom field behaviour."""
return {
- "hidden_fields": ["schema", "login", "password"],
+ "hidden_fields": ["schema", "login", "password", "extra"],
"relabeling": {},
}
+ @classmethod
+ def get_connection_form_widgets(cls) -> dict[str, Any]:
+ """Returns connection widgets to add to connection form."""
+ from flask_appbuilder.fieldwidgets import BS3TextFieldWidget
+ from flask_babel import lazy_gettext
+ from wtforms import StringField
+ from wtforms.validators import Optional, any_of
+
+ return {
+ "queue": StringField(
+ lazy_gettext("YARN queue"),
+ widget=BS3TextFieldWidget(),
+ description="Default YARN queue to use",
+ validators=[Optional()],
+ ),
+ "deploy-mode": StringField(
+ lazy_gettext("Deploy mode"),
+ widget=BS3TextFieldWidget(),
+ description="Must be client or cluster",
+ validators=[any_of(["client", "cluster"])],
+ ),
+ "spark-binary": StringField(
+ lazy_gettext("Spark binary"),
+ widget=BS3TextFieldWidget(),
+ description=f"Must be one of: {', '.join(ALLOWED_SPARK_BINARIES)}",
+ validators=[any_of(ALLOWED_SPARK_BINARIES)],
Review Comment:
I see.
So for now I will keep it as it is :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] change spark connection form and add spark connections docs [airflow]
Posted by "shohamy7 (via GitHub)" <gi...@apache.org>.
shohamy7 commented on PR #36419:
URL: https://github.com/apache/airflow/pull/36419#issuecomment-1870649740
@josh-fell
Good point!
I didn't know about this one. I've checked and saw that it is causing issues with other connections.
But I saw that when I add default values, the problem solved.
I saw that we already implement this kind of logic in the `VaultHook`.
https://github.com/apache/airflow/blob/71314b86cfb73478b929c5728c5b969f6b3caeae/airflow/providers/hashicorp/hooks/vault.py#L380-L386
I will push a fix with default values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] change spark connection form and add spark connections docs [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #36419:
URL: https://github.com/apache/airflow/pull/36419#issuecomment-2018756477
> How should we pass the extra params that have been removed in this PR now?
It's not removed.
This is just friendlier UI form.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] change spark connection form and add spark connections docs [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #36419:
URL: https://github.com/apache/airflow/pull/36419#discussion_r1436880841
##########
airflow/providers/apache/spark/hooks/spark_submit.py:
##########
@@ -98,10 +98,42 @@ class SparkSubmitHook(BaseHook, LoggingMixin):
def get_ui_field_behaviour(cls) -> dict[str, Any]:
"""Return custom field behaviour."""
return {
- "hidden_fields": ["schema", "login", "password"],
+ "hidden_fields": ["schema", "login", "password", "extra"],
"relabeling": {},
}
+ @classmethod
+ def get_connection_form_widgets(cls) -> dict[str, Any]:
+ """Returns connection widgets to add to connection form."""
+ from flask_appbuilder.fieldwidgets import BS3TextFieldWidget
+ from flask_babel import lazy_gettext
+ from wtforms import StringField
+ from wtforms.validators import Optional, any_of
+
+ return {
+ "queue": StringField(
+ lazy_gettext("YARN queue"),
+ widget=BS3TextFieldWidget(),
+ description="Default YARN queue to use",
+ validators=[Optional()],
+ ),
+ "deploy-mode": StringField(
+ lazy_gettext("Deploy mode"),
+ widget=BS3TextFieldWidget(),
+ description="Must be client or cluster",
+ validators=[any_of(["client", "cluster"])],
+ ),
+ "spark-binary": StringField(
+ lazy_gettext("Spark binary"),
+ widget=BS3TextFieldWidget(),
+ description=f"Must be one of: {', '.join(ALLOWED_SPARK_BINARIES)}",
+ validators=[any_of(ALLOWED_SPARK_BINARIES)],
Review Comment:
That is might be a good idea extend allowed list by a separate PR, so one day it is possible to extend community providers connection UI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] change spark connection form and add spark connections docs [airflow]
Posted by "josh-fell (via GitHub)" <gi...@apache.org>.
josh-fell commented on code in PR #36419:
URL: https://github.com/apache/airflow/pull/36419#discussion_r1437086157
##########
airflow/providers/apache/spark/hooks/spark_submit.py:
##########
@@ -98,10 +98,42 @@ class SparkSubmitHook(BaseHook, LoggingMixin):
def get_ui_field_behaviour(cls) -> dict[str, Any]:
"""Return custom field behaviour."""
return {
- "hidden_fields": ["schema", "login", "password"],
+ "hidden_fields": ["schema", "login", "password", "extra"],
"relabeling": {},
}
+ @classmethod
+ def get_connection_form_widgets(cls) -> dict[str, Any]:
+ """Returns connection widgets to add to connection form."""
+ from flask_appbuilder.fieldwidgets import BS3TextFieldWidget
+ from flask_babel import lazy_gettext
+ from wtforms import StringField
+ from wtforms.validators import Optional, any_of
+
+ return {
+ "queue": StringField(
+ lazy_gettext("YARN queue"),
+ widget=BS3TextFieldWidget(),
+ description="Default YARN queue to use",
+ validators=[Optional()],
+ ),
+ "deploy-mode": StringField(
+ lazy_gettext("Deploy mode"),
+ widget=BS3TextFieldWidget(),
+ description="Must be client or cluster",
+ validators=[any_of(["client", "cluster"])],
+ ),
+ "spark-binary": StringField(
+ lazy_gettext("Spark binary"),
+ widget=BS3TextFieldWidget(),
+ description=f"Must be one of: {', '.join(ALLOWED_SPARK_BINARIES)}",
+ validators=[any_of(ALLOWED_SPARK_BINARIES)],
Review Comment:
Agreed, separate PR, but there was a [particular reason](https://github.com/apache/airflow/issues/17193) why `SelectField` wasn't in the original list of allowed types it seems. The issue is 2 years old, but obviously we don't want to resurface an old bug. Just a heads up if/when making the change later.
It would be very useful to have this form field type. (It's used in the Connection form itself already.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] change spark connection form and add spark connections docs [airflow]
Posted by "josh-fell (via GitHub)" <gi...@apache.org>.
josh-fell commented on PR #36419:
URL: https://github.com/apache/airflow/pull/36419#issuecomment-1870393776
@shohamy7 Have you tried to create a connection that _isn't_ one of the connections being updated here? Behind the scenes with connections, _all_ of the connection form data is submitted across all connection types. My concern is that adding the `any_of` validation might block other connections from being created in the UI because other forms don't have these exact fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] change spark connection form and add spark connections docs [airflow]
Posted by "YvesMentens (via GitHub)" <gi...@apache.org>.
YvesMentens commented on PR #36419:
URL: https://github.com/apache/airflow/pull/36419#issuecomment-2018787802
But how do I pass the extra parameters then? If I cant add them in the UI?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] change spark connection form and add spark connections docs [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #36419:
URL: https://github.com/apache/airflow/pull/36419#discussion_r1436345176
##########
docs/apache-airflow-providers-apache-spark/connections/spark-sql.rst:
##########
@@ -0,0 +1,48 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+.. _howto/connection:spark-sql:
+
+Apache Spark Sql Connection
Review Comment:
```suggestion
Apache Spark SQL Connection
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] change spark connection form and add spark connections docs [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #36419:
URL: https://github.com/apache/airflow/pull/36419#discussion_r1436826808
##########
airflow/providers/apache/spark/hooks/spark_submit.py:
##########
@@ -98,10 +98,42 @@ class SparkSubmitHook(BaseHook, LoggingMixin):
def get_ui_field_behaviour(cls) -> dict[str, Any]:
"""Return custom field behaviour."""
return {
- "hidden_fields": ["schema", "login", "password"],
+ "hidden_fields": ["schema", "login", "password", "extra"],
"relabeling": {},
}
+ @classmethod
+ def get_connection_form_widgets(cls) -> dict[str, Any]:
+ """Returns connection widgets to add to connection form."""
+ from flask_appbuilder.fieldwidgets import BS3TextFieldWidget
+ from flask_babel import lazy_gettext
+ from wtforms import StringField
+ from wtforms.validators import Optional, any_of
+
+ return {
+ "queue": StringField(
+ lazy_gettext("YARN queue"),
+ widget=BS3TextFieldWidget(),
+ description="Default YARN queue to use",
+ validators=[Optional()],
+ ),
+ "deploy-mode": StringField(
+ lazy_gettext("Deploy mode"),
+ widget=BS3TextFieldWidget(),
+ description="Must be client or cluster",
+ validators=[any_of(["client", "cluster"])],
+ ),
+ "spark-binary": StringField(
+ lazy_gettext("Spark binary"),
+ widget=BS3TextFieldWidget(),
+ description=f"Must be one of: {', '.join(ALLOWED_SPARK_BINARIES)}",
+ validators=[any_of(ALLOWED_SPARK_BINARIES)],
Review Comment:
This change in Airflow core, so it would not work in providers until it meet minimum Airflow version where this change would be applied, approx 1 year after commit to the core
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] change spark connection form and add spark connections docs [airflow]
Posted by "YvesMentens (via GitHub)" <gi...@apache.org>.
YvesMentens commented on PR #36419:
URL: https://github.com/apache/airflow/pull/36419#issuecomment-2018731687
How should we pass the extra params that have been removed in this PR now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] change spark connection form and add spark connections docs [airflow]
Posted by "shohamy7 (via GitHub)" <gi...@apache.org>.
shohamy7 commented on code in PR #36419:
URL: https://github.com/apache/airflow/pull/36419#discussion_r1436621647
##########
airflow/providers/apache/spark/hooks/spark_submit.py:
##########
@@ -98,10 +98,42 @@ class SparkSubmitHook(BaseHook, LoggingMixin):
def get_ui_field_behaviour(cls) -> dict[str, Any]:
"""Return custom field behaviour."""
return {
- "hidden_fields": ["schema", "login", "password"],
+ "hidden_fields": ["schema", "login", "password", "extra"],
"relabeling": {},
}
+ @classmethod
+ def get_connection_form_widgets(cls) -> dict[str, Any]:
+ """Returns connection widgets to add to connection form."""
+ from flask_appbuilder.fieldwidgets import BS3TextFieldWidget
+ from flask_babel import lazy_gettext
+ from wtforms import StringField
+ from wtforms.validators import Optional, any_of
+
+ return {
+ "queue": StringField(
+ lazy_gettext("YARN queue"),
+ widget=BS3TextFieldWidget(),
+ description="Default YARN queue to use",
+ validators=[Optional()],
+ ),
+ "deploy-mode": StringField(
+ lazy_gettext("Deploy mode"),
+ widget=BS3TextFieldWidget(),
+ description="Must be client or cluster",
+ validators=[any_of(["client", "cluster"])],
+ ),
+ "spark-binary": StringField(
+ lazy_gettext("Spark binary"),
+ widget=BS3TextFieldWidget(),
+ description=f"Must be one of: {', '.join(ALLOWED_SPARK_BINARIES)}",
+ validators=[any_of(ALLOWED_SPARK_BINARIES)],
Review Comment:
I've tried but it won't get rendered.
It looks like there is a restriction on which field we can use.
https://github.com/apache/airflow/blob/73d87945e9a78195278d0a4b495483062ddc9b35/airflow/providers_manager.py#L934-L954
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] change spark connection form and add spark connections docs [airflow]
Posted by "shohamy7 (via GitHub)" <gi...@apache.org>.
shohamy7 commented on code in PR #36419:
URL: https://github.com/apache/airflow/pull/36419#discussion_r1436635037
##########
airflow/providers/apache/spark/hooks/spark_submit.py:
##########
@@ -98,10 +98,42 @@ class SparkSubmitHook(BaseHook, LoggingMixin):
def get_ui_field_behaviour(cls) -> dict[str, Any]:
"""Return custom field behaviour."""
return {
- "hidden_fields": ["schema", "login", "password"],
+ "hidden_fields": ["schema", "login", "password", "extra"],
"relabeling": {},
}
+ @classmethod
+ def get_connection_form_widgets(cls) -> dict[str, Any]:
+ """Returns connection widgets to add to connection form."""
+ from flask_appbuilder.fieldwidgets import BS3TextFieldWidget
+ from flask_babel import lazy_gettext
+ from wtforms import StringField
+ from wtforms.validators import Optional, any_of
+
+ return {
+ "queue": StringField(
+ lazy_gettext("YARN queue"),
+ widget=BS3TextFieldWidget(),
+ description="Default YARN queue to use",
+ validators=[Optional()],
+ ),
+ "deploy-mode": StringField(
+ lazy_gettext("Deploy mode"),
+ widget=BS3TextFieldWidget(),
+ description="Must be client or cluster",
+ validators=[any_of(["client", "cluster"])],
+ ),
+ "spark-binary": StringField(
+ lazy_gettext("Spark binary"),
+ widget=BS3TextFieldWidget(),
+ description=f"Must be one of: {', '.join(ALLOWED_SPARK_BINARIES)}",
+ validators=[any_of(ALLOWED_SPARK_BINARIES)],
Review Comment:
@josh-fell
I can add it into the `allowed_field_classes`, WDYT?
Or maybe I should do it in a different PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] change spark connection form and add spark connections docs [airflow]
Posted by "shohamy7 (via GitHub)" <gi...@apache.org>.
shohamy7 commented on code in PR #36419:
URL: https://github.com/apache/airflow/pull/36419#discussion_r1437270084
##########
docs/apache-airflow-providers-apache-spark/connections/spark.rst:
##########
@@ -19,15 +19,15 @@
.. _howto/connection:spark:
Review Comment:
renamed :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org