You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Brian Candler <b....@pobox.com> on 2016/05/31 16:33:12 UTC
ImportError: cannot import name HiveOperator (missing dependency on
unicodecsv)
Trying to follow the quickstart at
http://pythonhosted.org/airflow/start.html (inside a fresh virtualenv,
under ubuntu 14.04)
I get the following error:
(venv)brian@cfprov:~/airflow$ airflow initdb
[2016-05-31 17:02:25,939] {__init__.py:36} INFO - Using executor
SequentialExecutor
[2016-05-31 17:02:26,049] {driver.py:120} INFO - Generating grammar
tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-05-31 17:02:26,114] {driver.py:120} INFO - Generating grammar
tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
DB: sqlite:////home/brian/airflow/airflow.db
[2016-05-31 17:02:26,361] {db.py:222} INFO - Creating tables
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Running upgrade -> e3a246e0dc1,
current schema
INFO [alembic.runtime.migration] Running upgrade e3a246e0dc1 ->
1507a7289a2f, create is_encrypted
/home/brian/airflow/venv/local/lib/python2.7/site-packages/alembic/util/messaging.py:69:
UserWarning: Skipping unsupported ALTER for creation of implicit constraint
warnings.warn(msg)
INFO [alembic.runtime.migration] Running upgrade 1507a7289a2f ->
13eb55f81627, maintain history for compatibility with earlier migrations
INFO [alembic.runtime.migration] Running upgrade 13eb55f81627 ->
338e90f54d61, More logging into task_isntance
INFO [alembic.runtime.migration] Running upgrade 338e90f54d61 ->
52d714495f0, job_id indices
INFO [alembic.runtime.migration] Running upgrade 52d714495f0 ->
502898887f84, Adding extra to Log
INFO [alembic.runtime.migration] Running upgrade 502898887f84 ->
1b38cef5b76e, add dagrun
INFO [alembic.runtime.migration] Running upgrade 1b38cef5b76e ->
2e541a1dcfed, task_duration
INFO [alembic.runtime.migration] Running upgrade 2e541a1dcfed ->
40e67319e3a9, dagrun_config
INFO [alembic.runtime.migration] Running upgrade 40e67319e3a9 ->
561833c1c74b, add password column to user
INFO [alembic.runtime.migration] Running upgrade 561833c1c74b ->
4446e08588, dagrun start end
INFO [alembic.runtime.migration] Running upgrade 4446e08588 ->
bbc73705a13e, Add notification_sent column to sla_miss
INFO [alembic.runtime.migration] Running upgrade bbc73705a13e ->
bba5a7cfc896, Add a column to track the encryption state of the 'Extra'
field in connection
INFO [alembic.runtime.migration] Running upgrade bba5a7cfc896 ->
1968acfc09e3, add is_encrypted column to variable table
INFO [alembic.runtime.migration] Running upgrade 1968acfc09e3 ->
2e82aab8ef20, rename user table
ERROR [airflow.models.DagBag] Failed to import:
/home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/example_dags/example_twitter_dag.py
Traceback (most recent call last):
File
"/home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/models.py",
line 247, in process_file
m = imp.load_source(mod_name, filepath)
File
"/home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/example_dags/example_twitter_dag.py",
line 26, in <module>
from airflow.operators import BashOperator, HiveOperator,
PythonOperator
ImportError: cannot import name HiveOperator
Done.
(venv)brian@cfprov:~/airflow$
Airflow *does* appear to work (the web UI comes up), but I also get the
same error logged for the execution of the example_bash_operator code:
[2016-05-31 17:04:16,451] {models.py:154} INFO - Filling up the DagBag from /home/brian/airflow/dags/example_dags/example_bash_operator.py
[2016-05-31 17:04:16,452] {models.py:250} ERROR - Failed to import: /home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/example_dags/example_twitter_dag.py
Traceback (most recent call last):
File "/home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/models.py", line 247, in process_file
m = imp.load_source(mod_name, filepath)
File "/home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/example_dags/example_twitter_dag.py", line 26, in <module>
from airflow.operators import BashOperator, HiveOperator, PythonOperator
ImportError: cannot import name HiveOperator
If I try it from the command line:
>>> from airflow.operators.hive_operator import HiveOperator
[2016-05-31 17:24:59,316] {__init__.py:36} INFO - Using executor
SequentialExecutor
[2016-05-31 17:24:59,427] {driver.py:120} INFO - Generating grammar
tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-05-31 17:24:59,484] {driver.py:120} INFO - Generating grammar
tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File
"/home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/operators/hive_operator.py",
line 4, in <module>
from airflow.hooks import HiveCliHook
ImportError: cannot import name HiveCliHook
And again:
>>> from airflow.hooks.hive_hooks import HiveCliHook
[2016-05-31 17:27:11,716] {__init__.py:36} INFO - Using executor
SequentialExecutor
[2016-05-31 17:27:11,824] {driver.py:120} INFO - Generating grammar
tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-05-31 17:27:11,872] {driver.py:120} INFO - Generating grammar
tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File
"/home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/hooks/hive_hooks.py",
line 19, in <module>
import unicodecsv as csv
ImportError: No module named unicodecsv
Aha. This error has been hidden by magic.After "pip install unicodecsv"
it seems happy:
>>> from airflow.operators import HiveOperator
[2016-05-31 17:28:15,828] {__init__.py:36} INFO - Using executor
SequentialExecutor
[2016-05-31 17:28:15,936] {driver.py:120} INFO - Generating grammar
tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-05-31 17:28:15,984] {driver.py:120} INFO - Generating grammar
tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
>>>
So: I think the "airflow" package is missing a dependency on "unicodecsv".
Regards,
Brian Candler.
P.S. Side problem: I wanted to create an issue on JIRA but couldn't see
how. I have an Apache JIRA account (username "candlerb"). Under the
"Create" menu it offers me two choices:
* New JIRA Project # really, I can do this?!
* Create Service Desk Request # whatever that is
But all I wanted to do was create an issue. Am I missing something
obvious? Or are only project developers allowed to raise issues?
Re: ImportError: cannot import name HiveOperator (missing dependency
on unicodecsv)
Posted by Chris Riccomini <cr...@apache.org>.
Hey Brian,
Try going here to create the JIRA:
https://issues.apache.org/jira/secure/CreateIssue!default.jspa
Cheers,
Chris
On Tue, May 31, 2016 at 9:33 AM, Brian Candler <b....@pobox.com> wrote:
> Trying to follow the quickstart at
> http://pythonhosted.org/airflow/start.html (inside a fresh virtualenv,
> under ubuntu 14.04)
>
> I get the following error:
>
> (venv)brian@cfprov:~/airflow$ airflow initdb
> [2016-05-31 17:02:25,939] {__init__.py:36} INFO - Using executor
> SequentialExecutor
> [2016-05-31 17:02:26,049] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2016-05-31 17:02:26,114] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> DB: sqlite:////home/brian/airflow/airflow.db
> [2016-05-31 17:02:26,361] {db.py:222} INFO - Creating tables
> INFO [alembic.runtime.migration] Context impl SQLiteImpl.
> INFO [alembic.runtime.migration] Will assume non-transactional DDL.
> INFO [alembic.runtime.migration] Running upgrade -> e3a246e0dc1, current
> schema
> INFO [alembic.runtime.migration] Running upgrade e3a246e0dc1 ->
> 1507a7289a2f, create is_encrypted
> /home/brian/airflow/venv/local/lib/python2.7/site-packages/alembic/util/messaging.py:69:
> UserWarning: Skipping unsupported ALTER for creation of implicit constraint
> warnings.warn(msg)
> INFO [alembic.runtime.migration] Running upgrade 1507a7289a2f ->
> 13eb55f81627, maintain history for compatibility with earlier migrations
> INFO [alembic.runtime.migration] Running upgrade 13eb55f81627 ->
> 338e90f54d61, More logging into task_isntance
> INFO [alembic.runtime.migration] Running upgrade 338e90f54d61 ->
> 52d714495f0, job_id indices
> INFO [alembic.runtime.migration] Running upgrade 52d714495f0 ->
> 502898887f84, Adding extra to Log
> INFO [alembic.runtime.migration] Running upgrade 502898887f84 ->
> 1b38cef5b76e, add dagrun
> INFO [alembic.runtime.migration] Running upgrade 1b38cef5b76e ->
> 2e541a1dcfed, task_duration
> INFO [alembic.runtime.migration] Running upgrade 2e541a1dcfed ->
> 40e67319e3a9, dagrun_config
> INFO [alembic.runtime.migration] Running upgrade 40e67319e3a9 ->
> 561833c1c74b, add password column to user
> INFO [alembic.runtime.migration] Running upgrade 561833c1c74b ->
> 4446e08588, dagrun start end
> INFO [alembic.runtime.migration] Running upgrade 4446e08588 ->
> bbc73705a13e, Add notification_sent column to sla_miss
> INFO [alembic.runtime.migration] Running upgrade bbc73705a13e ->
> bba5a7cfc896, Add a column to track the encryption state of the 'Extra'
> field in connection
> INFO [alembic.runtime.migration] Running upgrade bba5a7cfc896 ->
> 1968acfc09e3, add is_encrypted column to variable table
> INFO [alembic.runtime.migration] Running upgrade 1968acfc09e3 ->
> 2e82aab8ef20, rename user table
> ERROR [airflow.models.DagBag] Failed to import:
> /home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/example_dags/example_twitter_dag.py
> Traceback (most recent call last):
> File
> "/home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/models.py",
> line 247, in process_file
> m = imp.load_source(mod_name, filepath)
> File
> "/home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/example_dags/example_twitter_dag.py",
> line 26, in <module>
> from airflow.operators import BashOperator, HiveOperator,
> PythonOperator
> ImportError: cannot import name HiveOperator
> Done.
> (venv)brian@cfprov:~/airflow$
>
> Airflow *does* appear to work (the web UI comes up), but I also get the
> same error logged for the execution of the example_bash_operator code:
>
> [2016-05-31 17:04:16,451] {models.py:154} INFO - Filling up the DagBag
> from /home/brian/airflow/dags/example_dags/example_bash_operator.py
> [2016-05-31 17:04:16,452] {models.py:250} ERROR - Failed to import:
> /home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/example_dags/example_twitter_dag.py
> Traceback (most recent call last):
> File
> "/home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/models.py",
> line 247, in process_file
> m = imp.load_source(mod_name, filepath)
> File
> "/home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/example_dags/example_twitter_dag.py",
> line 26, in <module>
> from airflow.operators import BashOperator, HiveOperator,
> PythonOperator
> ImportError: cannot import name HiveOperator
>
>
> If I try it from the command line:
>
> >>> from airflow.operators.hive_operator import HiveOperator
> [2016-05-31 17:24:59,316] {__init__.py:36} INFO - Using executor
> SequentialExecutor
> [2016-05-31 17:24:59,427] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2016-05-31 17:24:59,484] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File
> "/home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/operators/hive_operator.py",
> line 4, in <module>
> from airflow.hooks import HiveCliHook
> ImportError: cannot import name HiveCliHook
>
> And again:
>
> >>> from airflow.hooks.hive_hooks import HiveCliHook
> [2016-05-31 17:27:11,716] {__init__.py:36} INFO - Using executor
> SequentialExecutor
> [2016-05-31 17:27:11,824] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2016-05-31 17:27:11,872] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File
> "/home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/hooks/hive_hooks.py",
> line 19, in <module>
> import unicodecsv as csv
> ImportError: No module named unicodecsv
>
> Aha. This error has been hidden by magic.After "pip install unicodecsv" it
> seems happy:
>
> >>> from airflow.operators import HiveOperator
> [2016-05-31 17:28:15,828] {__init__.py:36} INFO - Using executor
> SequentialExecutor
> [2016-05-31 17:28:15,936] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2016-05-31 17:28:15,984] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> >>>
>
> So: I think the "airflow" package is missing a dependency on "unicodecsv".
>
> Regards,
>
> Brian Candler.
>
> P.S. Side problem: I wanted to create an issue on JIRA but couldn't see
> how. I have an Apache JIRA account (username "candlerb"). Under the
> "Create" menu it offers me two choices:
>
> * New JIRA Project # really, I can do this?!
>
> * Create Service Desk Request # whatever that is
>
> But all I wanted to do was create an issue. Am I missing something
> obvious? Or are only project developers allowed to raise issues?
>
>