You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/05/04 23:31:20 UTC
[GitHub] [airflow] boittega opened a new issue #8713: Spark SQL Hook not using connections
boittega opened a new issue #8713:
URL: https://github.com/apache/airflow/issues/8713
**Apache Airflow version**: 1.10.10
**What happened**:
`SparkSqlHook` is not using any connection, the default conn_id is `spark_sql_default`, if this connection doesn't exist, the hook returns an error:
```
Traceback (most recent call last):
File "/Users/rbottega/Documents/airflow_latest/env/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 983, in _run_raw_task
result = task_copy.execute(context=context)
File "/Users/rbottega/Documents/airflow_latest/env/lib/python3.7/site-packages/airflow/contrib/operators/spark_sql_operator.py", line 109, in execute
yarn_queue=self._yarn_queue
File "/Users/rbottega/Documents/airflow_latest/env/lib/python3.7/site-packages/airflow/contrib/hooks/spark_sql_hook.py", line 75, in __init__
self._conn = self.get_connection(conn_id)
File "/Users/rbottega/Documents/airflow_latest/env/lib/python3.7/site-packages/airflow/hooks/base_hook.py", line 84, in get_connection
conn = random.choice(list(cls.get_connections(conn_id)))
File "/Users/rbottega/Documents/airflow_latest/env/lib/python3.7/site-packages/airflow/hooks/base_hook.py", line 80, in get_connections
return secrets.get_connections(conn_id)
File "/Users/rbottega/Documents/airflow_latest/env/lib/python3.7/site-packages/airflow/secrets/__init__.py", line 56, in get_connections
raise AirflowException("The conn_id `{0}` isn't defined".format(conn_id))
airflow.exceptions.AirflowException: The conn_id `spark_sql_default` isn't defined
```
If specified any valid connection, it does nothing, the `self._conn` variable is never used and there is an empty `get_conn` method.
```
def get_conn(self):
pass
```
**What you expected to happen**:
It should follow the same behaviour of `SparkSubmitHook` to receive the master host and extra parameters from the connection OR don't request a connection ID.
**How to reproduce it**:
Just create a DAG with a `SparkSqlOperator` and have not created the connection `spark_sql_default`.
```
sql_job = SparkSqlOperator(
sql="SELECT * FROM test",
master="local",
task_id="sql_job"
)
```
**Anything else we need to know**:
I am happy to implement any of these solutions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boittega commented on issue #8713: Spark SQL Hook not using connections
Posted by GitBox <gi...@apache.org>.
boittega commented on issue #8713:
URL: https://github.com/apache/airflow/issues/8713#issuecomment-730985988
Hi @danielenricocahall
Sure, go ahead.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] danielenricocahall commented on issue #8713: Spark SQL Hook not using connections
Posted by GitBox <gi...@apache.org>.
danielenricocahall commented on issue #8713:
URL: https://github.com/apache/airflow/issues/8713#issuecomment-730794729
Can I take this issue?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #8713: Spark SQL Hook not using connections
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #8713:
URL: https://github.com/apache/airflow/issues/8713#issuecomment-839627318
I’ve submitted #15794 for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #8713: Spark SQL Hook not using connections
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #8713:
URL: https://github.com/apache/airflow/issues/8713
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #8713: Spark SQL Hook not using connections
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #8713:
URL: https://github.com/apache/airflow/issues/8713#issuecomment-830790606
@danielenricocahall I'm unasinging you as you didn't complete the PR if you wish to finish it let us know.
This issue is open to anyone who wants to work on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #8713: Spark SQL Hook not using connections
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #8713:
URL: https://github.com/apache/airflow/issues/8713#issuecomment-731114223
I've assigned it to you @danielenricocahall
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #8713: Spark SQL Hook not using connections
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #8713:
URL: https://github.com/apache/airflow/issues/8713#issuecomment-623761771
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org