You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/15 18:44:35 UTC
[GitHub] [airflow] gauravguptabits opened a new issue #19602: Spark Operator not working in Airflow Standalone version 2.2.1
gauravguptabits opened a new issue #19602:
URL: https://github.com/apache/airflow/issues/19602
### Apache Airflow version
2.2.1 (latest released)
### Operating System
Ubuntu
### Versions of Apache Airflow Providers
Apache Web UI is unable to load the dag file.
Logs:
```
Traceback (most recent call last):
File "/home/gaurav.gupta/miniconda3/envs/venv_pyspark/lib/python3.7/site-packages/airflow/models/dagbag.py", line 331, in _load_modules_from_file
loader.exec_module(new_module)
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/gaurav.gupta/airflow/dags/prepare_complaint_raw_dataset_dag.py", line 4, in <module>
from airflow.contrib.operators.spark_submit_operator import SparkSubmitOperator
File "/home/gaurav.gupta/miniconda3/envs/venv_pyspark/lib/python3.7/site-packages/airflow/contrib/operators/spark_submit_operator.py", line 22, in <module>
from airflow.providers.apache.spark.operators.spark_submit import SparkSubmitOperator # noqa
ModuleNotFoundError: No module named 'airflow.providers.apache'
```
DAG file
```
from datetime import datetime, timedelta
import pendulum
from airflow import DAG
from airflow.contrib.operators.spark_submit_operator import SparkSubmitOperator
from airflow.models import Variable
local_tz = pendulum.timezone("Asia/Kolkata")
default_args = {
'owner': 'Impressico',
'depends_on_past': False,
'start_date': datetime(2021, 11, 15, tzinfo=local_tz),
'email': ['gaurav.gupta@impressico.com'],
'email_on_failure': True,
'email_on_retry': True,
'retries': 0,
'retry_delay': timedelta(minutes=5)
}
dag = DAG(dag_id='prepare_complaint_raw_dataset',
default_args=default_args,
catchup=False,
schedule_interval="0 * * * *")
pyspark_app_home = Variable.get("PYSPARK_APP_HOME")
prepare_complaint_raw_dataset = SparkSubmitOperator(task_id='prepare_complaint_raw_dataset',
conn_id='Spark_Miniconda_Venv_pyspark',
py_files='jobs.zip,libs.zip',
application=f'{pyspark_app_home}/dist/main.py',
total_executor_cores=4,
name='prepare_complaint_raw_dataset',
dag=dag)
```
### Deployment
Virtualenv installation
### Deployment details
I have installed via pip in Anaconda environment.
### What happened
After defining dag file, web ui of airflow is popping up an error. Following the stacktrace it resulted in some import error which has been attached in description
### What you expected to happen
I was expecting the file to get at least accepted by WEB UI as the definition looks fine. Being newbee to Airflow, I feel there can be configuration issue within the SparkSubmitOperator parameters.
### How to reproduce
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #19602: Spark Operator not working in Airflow Standalone version 2.2.1
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #19602:
URL: https://github.com/apache/airflow/issues/19602#issuecomment-969300202
To use apache spark operators, you need to install an apache spark provider. See: http://airflow.apache.org/docs/apache-airflow-providers-apache-spark/stable/index.html and http://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #19602: Spark Operator not working in Airflow Standalone version 2.2.1
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #19602:
URL: https://github.com/apache/airflow/issues/19602#issuecomment-969208119
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj closed issue #19602: Spark Operator not working in Airflow Standalone version 2.2.1
Posted by GitBox <gi...@apache.org>.
mik-laj closed issue #19602:
URL: https://github.com/apache/airflow/issues/19602
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org