You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Ben Tallman <be...@apigee.com> on 2016/10/05 01:23:50 UTC
New Scheduler process seems incompatible with sqlite...
When testing the scheduler locally, on master, using sqlite and sequential
executor, dag.sync_to_db never returns when querying the db...
orm_dag = session.query(
> DagModel).filter(DagModel.dag_id == dag.dag_id).first()
Any insights? My guess is that this is due to the new multi-process nature
of dag processing conflicting with sqlite?
Thanks,
Ben
*--*
*ben tallman* | *apigee
<http://t.sidekickopen06.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs4WJfgqW4WJj7n3MP7VWW3LqXLC56dWRRf2H8CkP02?t=http%3A%2F%2Fwww.apigee.com%2F&si=5141814536306688&pi=291525b6-758e-488a-f6fb-b8a6ee46e8a8>*
| m: +1.503.680.5709 | o: +1.503.608.7552 | twitter @anonymousmanage
<http://t.sidekickopen06.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs4WJfgqW4WJj7n3MP7VWW3LqXLC56dWRRf2H8CkP02?t=http%3A%2F%2Ftwitter.com%2Fanonymousmanage&si=5141814536306688&pi=291525b6-758e-488a-f6fb-b8a6ee46e8a8>
@apigee
<http://t.sidekickopen06.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs4WJfgqW4WJj7n3MP7VWW3LqXLC56dWRRf2H8CkP02?t=https%3A%2F%2Ftwitter.com%2Fapigee&si=5141814536306688&pi=291525b6-758e-488a-f6fb-b8a6ee46e8a8>
<http://t.sidekickopen06.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs4WJfgqW4WJj7n3MP7VWW3LqXLC56dWRRf2H8CkP02?t=http%3A%2F%2Fadapt.apigee.com%2F&si=5141814536306688&pi=291525b6-758e-488a-f6fb-b8a6ee46e8a8>
Re: New Scheduler process seems incompatible with sqlite...
Posted by siddharth anand <sa...@apache.org>.
That sounds possible. Can you set min_file_process_interval = 0? My airflow
config looks as shown below.
[scheduler]
# Task instances listen for external kill signal (when you clear tasks
# from the CLI or the UI), this defines the frequency at which they should
# listen (in seconds).
job_heartbeat_sec = 5
# The scheduler constantly tries to trigger new tasks (look at the
# scheduler section in the docs for more information). This defines
# how often the scheduler should run (in seconds).
scheduler_heartbeat_sec = 5
min_file_process_interval = 0
max_threads = 4
I tried running the sequential executor with this set and it improved
matters as shown here :
[image: Inline image 2]
-s
On Tue, Oct 4, 2016 at 6:23 PM, Ben Tallman <be...@apigee.com> wrote:
> When testing the scheduler locally, on master, using sqlite and sequential
> executor, dag.sync_to_db never returns when querying the db...
>
> orm_dag = session.query(
> > DagModel).filter(DagModel.dag_id == dag.dag_id).first()
>
>
> Any insights? My guess is that this is due to the new multi-process nature
> of dag processing conflicting with sqlite?
>
> Thanks,
> Ben
>
> *--*
> *ben tallman* | *apigee
> <http://t.sidekickopen06.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nM
> JW7t5XZs4WJfgqW4WJj7n3MP7VWW3LqXLC56dWRRf2H8CkP02?t=http%3A%
> 2F%2Fwww.apigee.com%2F&si=5141814536306688&pi=291525b6-
> 758e-488a-f6fb-b8a6ee46e8a8>*
> | m: +1.503.680.5709 | o: +1.503.608.7552 | twitter @anonymousmanage
> <http://t.sidekickopen06.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nM
> JW7t5XZs4WJfgqW4WJj7n3MP7VWW3LqXLC56dWRRf2H8CkP02?t=http%3A%
> 2F%2Ftwitter.com%2Fanonymousmanage&si=5141814536306688&pi=291525b6-
> 758e-488a-f6fb-b8a6ee46e8a8>
> @apigee
> <http://t.sidekickopen06.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nM
> JW7t5XZs4WJfgqW4WJj7n3MP7VWW3LqXLC56dWRRf2H8CkP02?t=https%
> 3A%2F%2Ftwitter.com%2Fapigee&si=5141814536306688&pi=
> 291525b6-758e-488a-f6fb-b8a6ee46e8a8>
> <http://t.sidekickopen06.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nM
> JW7t5XZs4WJfgqW4WJj7n3MP7VWW3LqXLC56dWRRf2H8CkP02?t=http%3A%
> 2F%2Fadapt.apigee.com%2F&si=5141814536306688&pi=291525b6-
> 758e-488a-f6fb-b8a6ee46e8a8>
>