You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Vineet Goel <vi...@robinhood.com> on 2016/06/17 00:22:23 UTC
Backfills error using Celery Executor
Hi,
I am new to Airflow. I set up a local Airflow instance with CeleryExecutor.
When I execute the command:
airflow backfill remote_execution_test -s 2014-07-01 -e 2014-07-02
I get the following error in the worker logs:
[2016-06-16 17:03:48,753: ERROR/Worker-9] Command 'airflow run
remote_execution_test print_date 2014-07-01T00:00:00 --pickle 7 --local '
returned non-zero exit status 1
with the following stack trace:
Traceback (most recent call last):
File "/Users/vineet/.virtualenvs/airflow/bin/airflow", line 15, in
<module>
args.func(args)
File
"/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/airflow/bin/cli.py",
line 211, in run
DagPickle).filter(DagPickle.id == args.pickle).first()
File
"/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/query.py",
line 2659, in first
ret = list(self[0:1])
File
"/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/query.py",
line 2457, in __getitem__
return list(res)
File
"/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/loading.py",
line 86, in instances
util.raise_from_cause(err)
File
"/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/sqlalchemy/util/compat.py",
line 202, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File
"/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/loading.py",
line 71, in instances
rows = [proc(row) for row in fetch]
File
"/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/loading.py",
line 428, in _instance
loaded_instance, populate_existing, populators)
File
"/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/loading.py",
line 486, in _populate_full
dict_[key] = getter(row)
File
"/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/sqlalchemy/sql/sqltypes.py",
line 1253, in process
return loads(value)
File
"/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/dill/dill.py",
line 260, in loads
return load(file)
File
"/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/dill/dill.py",
line 250, in load
obj = pik.load()
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py",
line 858, in load
dispatch[key](self)
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py",
line 1090, in load_global
klass = self.find_class(module, name)
File
"/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/dill/dill.py",
line 406, in find_class
return StockUnpickler.find_class(self, module, name)
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py",
line 1124, in find_class
__import__(module)
ImportError: No module named unusual_prefix_remote_execution_test
I am trying to understand what I might be doing wrong here. I tried to run
the command that celery is trying to run without the --pickle flag and it
works.
Note: I am running a single worker on the same machine.
Thanks!
Re: Backfills error using Celery Executor
Posted by Vineet Goel <vi...@robinhood.com>.
Note: If I disable pickling dags, it all works: donot_pickle = True
Though I don't understand what might be causing the pickling issue
considering I am running everything on the same machine.
On Thu, Jun 16, 2016 at 5:22 PM, Vineet Goel <vi...@robinhood.com> wrote:
> Hi,
>
> I am new to Airflow. I set up a local Airflow instance with
> CeleryExecutor. When I execute the command:
>
> airflow backfill remote_execution_test -s 2014-07-01 -e 2014-07-02
>
>
> I get the following error in the worker logs:
>
> [2016-06-16 17:03:48,753: ERROR/Worker-9] Command 'airflow run
> remote_execution_test print_date 2014-07-01T00:00:00 --pickle 7 --local '
> returned non-zero exit status 1
>
> with the following stack trace:
>
> Traceback (most recent call last):
> File "/Users/vineet/.virtualenvs/airflow/bin/airflow", line 15, in
> <module>
> args.func(args)
> File
> "/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/airflow/bin/cli.py",
> line 211, in run
> DagPickle).filter(DagPickle.id == args.pickle).first()
> File
> "/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/query.py",
> line 2659, in first
> ret = list(self[0:1])
> File
> "/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/query.py",
> line 2457, in __getitem__
> return list(res)
> File
> "/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/loading.py",
> line 86, in instances
> util.raise_from_cause(err)
> File
> "/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/sqlalchemy/util/compat.py",
> line 202, in raise_from_cause
> reraise(type(exception), exception, tb=exc_tb, cause=cause)
> File
> "/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/loading.py",
> line 71, in instances
> rows = [proc(row) for row in fetch]
> File
> "/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/loading.py",
> line 428, in _instance
> loaded_instance, populate_existing, populators)
> File
> "/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/loading.py",
> line 486, in _populate_full
> dict_[key] = getter(row)
> File
> "/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/sqlalchemy/sql/sqltypes.py",
> line 1253, in process
> return loads(value)
> File
> "/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/dill/dill.py",
> line 260, in loads
> return load(file)
> File
> "/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/dill/dill.py",
> line 250, in load
> obj = pik.load()
> File
> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py",
> line 858, in load
> dispatch[key](self)
> File
> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py",
> line 1090, in load_global
> klass = self.find_class(module, name)
> File
> "/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/dill/dill.py",
> line 406, in find_class
> return StockUnpickler.find_class(self, module, name)
> File
> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py",
> line 1124, in find_class
> __import__(module)
> ImportError: No module named unusual_prefix_remote_execution_test
>
> I am trying to understand what I might be doing wrong here. I tried to run
> the command that celery is trying to run without the --pickle flag and it
> works.
>
> Note: I am running a single worker on the same machine.
>
> Thanks!
>