You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jason Chen <ch...@gmail.com> on 2016/07/05 18:22:09 UTC
Running a task from the Airflow UI
Hi Airflow team,
I am using the "LocalExecutor" and it works very well to run the workflow
I setup.
I noticed that, from the UI, it can trigger a task to run.
However, I got the error "Only works with the CeleryExecutor, sorry ".
I can ssh into airflow node and run the command line from there.
However, it would be nice to just run it from airflow UI.
Is it possible to do that (with "LocalExecutor") or it's a future feature
to consider ?
Thanks.
Jason
Re: Running a task from the Airflow UI
Posted by Jeremiah Lowin <jl...@apache.org>.
It's funny but I never thought about it either until your email. We can't
guarantee there's a scheduler running... but then again we can't guarantee
a Celery worker is running either, so I think it might be a good
enhancement (along with an informative status message).
On Wed, Jul 6, 2016 at 8:32 PM Maxime Beauchemin <ma...@gmail.com>
wrote:
> Oh right, somehow I didn't even think of that. It's a fair assumption that
> there should be a scheduler up.
>
> Max
>
> On Wed, Jul 6, 2016 at 3:12 PM, Jeremiah Lowin <jl...@apache.org> wrote:
>
> > Perhaps it's a good chance to revisit the functionality. Right now the UI
> > "run" button actually runs the task via CeleryExecutor. Perhaps instead
> (or
> > just when using a non-Celery executor) it should queue the task and let
> the
> > Scheduler pick it up. I guess in that case it would just be sugar for
> > marking a TI as QUEUED. Just a thought.
> >
> > On Wed, Jul 6, 2016 at 2:54 AM Maxime Beauchemin <
> > maximebeauchemin@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > The problem is that a web server isn't the right place to run an
> airflow
> > > task. From the context of the web request scope we have to somehow
> pass a
> > > message to an external executor to run the task. For LocalExecutor to
> > work
> > > the web server would have to start a LocalExecutor as a sub process and
> > > that doesn't sound like a great idea...
> > >
> > > Max
> > >
> > > On Tue, Jul 5, 2016 at 11:22 AM, Jason Chen <chingchien.chen@gmail.com
> >
> > > wrote:
> > >
> > > > Hi Airflow team,
> > > > I am using the "LocalExecutor" and it works very well to run the
> > > workflow
> > > > I setup.
> > > >
> > > > I noticed that, from the UI, it can trigger a task to run.
> > > > However, I got the error "Only works with the CeleryExecutor, sorry
> ".
> > > > I can ssh into airflow node and run the command line from there.
> > > > However, it would be nice to just run it from airflow UI.
> > > > Is it possible to do that (with "LocalExecutor") or it's a future
> > feature
> > > > to consider ?
> > > >
> > > > Thanks.
> > > > Jason
> > > >
> > >
> >
>
Re: Running a task from the Airflow UI
Posted by Maxime Beauchemin <ma...@gmail.com>.
Oh right, somehow I didn't even think of that. It's a fair assumption that
there should be a scheduler up.
Max
On Wed, Jul 6, 2016 at 3:12 PM, Jeremiah Lowin <jl...@apache.org> wrote:
> Perhaps it's a good chance to revisit the functionality. Right now the UI
> "run" button actually runs the task via CeleryExecutor. Perhaps instead (or
> just when using a non-Celery executor) it should queue the task and let the
> Scheduler pick it up. I guess in that case it would just be sugar for
> marking a TI as QUEUED. Just a thought.
>
> On Wed, Jul 6, 2016 at 2:54 AM Maxime Beauchemin <
> maximebeauchemin@gmail.com>
> wrote:
>
> > Hi,
> >
> > The problem is that a web server isn't the right place to run an airflow
> > task. From the context of the web request scope we have to somehow pass a
> > message to an external executor to run the task. For LocalExecutor to
> work
> > the web server would have to start a LocalExecutor as a sub process and
> > that doesn't sound like a great idea...
> >
> > Max
> >
> > On Tue, Jul 5, 2016 at 11:22 AM, Jason Chen <ch...@gmail.com>
> > wrote:
> >
> > > Hi Airflow team,
> > > I am using the "LocalExecutor" and it works very well to run the
> > workflow
> > > I setup.
> > >
> > > I noticed that, from the UI, it can trigger a task to run.
> > > However, I got the error "Only works with the CeleryExecutor, sorry ".
> > > I can ssh into airflow node and run the command line from there.
> > > However, it would be nice to just run it from airflow UI.
> > > Is it possible to do that (with "LocalExecutor") or it's a future
> feature
> > > to consider ?
> > >
> > > Thanks.
> > > Jason
> > >
> >
>
Re: Suggested way of passing "input parameters" to a DAG run?
Posted by Andrew Phillips <an...@apache.org>.
On 2016-08-03 18:42, Joseph Napolitano wrote:
> What I can say is that we use it a lot, but very lightly. We basically
> use
> it to communicate the S3 key for a flat file between operators.
Just to follow up on this: here's a Gist with some example usage based
on the "store in XCom" approach:
https://gist.github.com/anonymous/b41228011275fb362aa5b95b476de542
We're using a macro to make retrieval a little nicer, and the end result
is not too different from reading from {{ params }}.
Nevertheless, +1 for a more first-class approach, if possible. {{
dag_run.params.foo }}, anyone? ;-)
On a related note, out of curiosity: does someone have any information
on use case(s) where changes to dag.params are useful during a DAG run?
Or is that just a consequence of the fact that the DAG object is
instantiated multiple times during a single run?
Is there an implicit assumption that the values passed to the DAG object
are the same every time within a single run? If so, that might be a good
topic for an FAQ entry/blog post?
Regards
ap
Re: Suggested way of passing "input parameters" to a DAG run?
Posted by Joseph Napolitano <jo...@blueapron.com.INVALID>.
What I can say is that we use it a lot, but very lightly. We basically use
it to communicate the S3 key for a flat file between operators.
Definitely don't use it to send actual data :)
Cheers
On Wed, Aug 3, 2016 at 6:21 PM, Andrew Phillips <an...@apache.org> wrote:
> Let me know if that helps, or if I completely misunderstood :)
>>
>
> That helps, indeed - thanks, Joe! We were in fact going down exactly this
> path as an alternative; we were just a bit hesitant to use XComs based on
> the following comment in the docs [1]:
>
> "If it absolutely can’t be avoided, Airflow does have a feature for
> operator cross-communication called XCom that is described elsewhere in
> this document."
>
> The statements talks about sharing information *between* tasks, but we
> weren't sure if this should be read as "stay away from XComs unless there's
> no other option". Curious to hear the community's thoughts on that.
>
> Thanks for the quick response!
>
> ap
>
> [1] https://pythonhosted.org/airflow/concepts.html#operators
>
--
*Joe Napolitano *| Sr. Data Engineer
www.blueapron.com | 5 Crosby Street, New York, NY 10013
Re: Suggested way of passing "input parameters" to a DAG run?
Posted by Andrew Phillips <an...@apache.org>.
> Let me know if that helps, or if I completely misunderstood :)
That helps, indeed - thanks, Joe! We were in fact going down exactly
this path as an alternative; we were just a bit hesitant to use XComs
based on the following comment in the docs [1]:
"If it absolutely can\u2019t be avoided, Airflow does have a feature for
operator cross-communication called XCom that is described elsewhere in
this document."
The statements talks about sharing information *between* tasks, but we
weren't sure if this should be read as "stay away from XComs unless
there's no other option". Curious to hear the community's thoughts on
that.
Thanks for the quick response!
ap
[1] https://pythonhosted.org/airflow/concepts.html#operators
Re: Suggested way of passing "input parameters" to a DAG run?
Posted by Joseph Napolitano <jo...@blueapron.com.INVALID>.
There are a lot of ways to define the input source. Let's suppose you have
these inputs in a relational database, or a flat file on S3.
The first task in your DAG would be a matter of querying for those inputs,
or grabbing the file. The trick is getting the inputs to later tasks. The
XCOM feature is a way to share data between your tasks, so it's a matter of
pulling the XCOM from the task that originally queried the inputs.
Suppose you had an "input operator"
class InputOperator(BaseOperator)
.... with an execute method ...
def execute(self, context):
... whatever you return it retrievable in later tasks through XCOM
....
return {"input_key": "input_value"}
Then in your DAG
input_operator_task = ... your InputOperator ....
downstream_task = SomeExistingOperator(
task_id='downstream_task',
keyword_arg_using_your_inputs="{{ti.xcom_pull(task_ids='input_operator_task')}}",
dag=dag
)
The XCOM pull is evaluated through the Jinja template.
Let me know if that helps, or if I completely misunderstood :)
Joe Nap
On Wed, Aug 3, 2016 at 5:29 PM, Andrew Phillips <an...@apache.org> wrote:
> Hi all
>
> What is/are the suggested way(s) of passing "input parameters" to a DAG
> run (adding quotes since, as far as we can tell, that concept doesn't exist
> natively in Airflow, probably by design)?
>
> This would be information that is used by one or multiple operators in a
> DAG run and that should not change for all task instances in that DAG run,
> but may be different for another DAG run executing concurrently. An example
> would be a Git pull request number.
>
> What we tried first was to use a Variable for this, but it doesn't look
> like that will work because the value can change during the execution of
> the DAG run. At least, that seems to be the case in the way we're using it:
>
> input_params = Variable.get(<variable_for_dag>)
> dag = DAG(..., params=input_params)
>
> We had hoped that this would "fix" the values of the parameters when the
> DAG run was created, but that does not seem to be the case: if the variable
> is updated (in preparation for a new DAG run) while a DAG run is active,
> tasks that haven't executed yet see the new value. I.e. we end up seeing
> this:
>
> set Variable my_param to "foo"
> dag_run_1 starts, gets the variable and passes my_param to the Dag object
> dag_run_1.op_1 evaluates {{ params.my_param }} and gets "foo"
> set Variable my_param to "bar"
> dag_run_2 starts and passes var to the Dag object
> dag_run_1.op_2 evaluates {{ params.my_param }} and sees "bar" # want this
> to still be foo!
>
> Not sure at this point whether this is a bug or, if not, whether there's a
> different way to retrieve the value of a variable that allows us to "fix"
> it for the duration of the DAG run.
>
> Or, taking a step back, is there some other approach that we could use to
> store and retrieve input data to DAGs?
>
> Regards
>
> ap
>
>
>
--
*Joe Napolitano *| Sr. Data Engineer
www.blueapron.com | 5 Crosby Street, New York, NY 10013
Suggested way of passing "input parameters" to a DAG run?
Posted by Andrew Phillips <an...@apache.org>.
Hi all
What is/are the suggested way(s) of passing "input parameters" to a DAG
run (adding quotes since, as far as we can tell, that concept doesn't
exist natively in Airflow, probably by design)?
This would be information that is used by one or multiple operators in a
DAG run and that should not change for all task instances in that DAG
run, but may be different for another DAG run executing concurrently. An
example would be a Git pull request number.
What we tried first was to use a Variable for this, but it doesn't look
like that will work because the value can change during the execution of
the DAG run. At least, that seems to be the case in the way we're using
it:
input_params = Variable.get(<variable_for_dag>)
dag = DAG(..., params=input_params)
We had hoped that this would "fix" the values of the parameters when the
DAG run was created, but that does not seem to be the case: if the
variable is updated (in preparation for a new DAG run) while a DAG run
is active, tasks that haven't executed yet see the new value. I.e. we
end up seeing this:
set Variable my_param to "foo"
dag_run_1 starts, gets the variable and passes my_param to the Dag
object
dag_run_1.op_1 evaluates {{ params.my_param }} and gets "foo"
set Variable my_param to "bar"
dag_run_2 starts and passes var to the Dag object
dag_run_1.op_2 evaluates {{ params.my_param }} and sees "bar" # want
this to still be foo!
Not sure at this point whether this is a bug or, if not, whether there's
a different way to retrieve the value of a variable that allows us to
"fix" it for the duration of the DAG run.
Or, taking a step back, is there some other approach that we could use
to store and retrieve input data to DAGs?
Regards
ap
Re: Running a task from the Airflow UI
Posted by Andrew Phillips <an...@apache.org>.
> A REST api is long overdue. I suggest anyone in the community that has
> the
> cycles to start implementing.. your PRs would be welcome. Currently, we
> have a very powerful CLI that should ideally have similar
> functionality
> exposed via the API.
On the subject of "exposing the CLI via an API", this might be useful:
http://mail-archives.apache.org/mod_mbox/incubator-airflow-dev/201607.mbox/%3Cfd1792dd9a8fddc5d38a0c5da98541ee%40qrmedia.com%3E
Regards
ap
Re: Running a task from the Airflow UI
Posted by siddharth anand <sa...@apache.org>.
A REST api is long overdue. I suggest anyone in the community that has the
cycles to start implementing.. your PRs would be welcome. Currently, we
have a very powerful CLI that should ideally have similar functionality
exposed via the API. The CLI's trigger_dag command is one of the first ones
I'd like to see. I've seen requests to have AWS Lambda's trigger_dag in
response to messages coming over SQS or Kinesis.
Seems like a good first REST API candidate to me.
-s
On Mon, Jul 11, 2016 at 11:17 AM, Paul Minton <pm...@change.org> wrote:
> Thanks Jeremiah. I just noticed this feature this weekend.
>
> I'm wondering if that functionality could be exposed to the REST api. Like
> a post to "airflow.run" with a json object in the payload that gets
> included with the pickled context object.
>
> That would 1) allow for dynamic runs and 2) (more importantly for our use
> case) expose dynamic runs via HTTP making it super simple for external
> resources to trigger dag runs.
>
> I'm still getting familiar with the source so I'm not sure how difficult
> that would be to implement.
>
> On Mon, Jul 11, 2016 at 6:44 AM, Jeremiah Lowin <jl...@apache.org> wrote:
>
>> Paul,
>>
>> The trigger_dag cli command accepts a json "conf" parameter. To be honest
>> I'm not familiar with the feature (perhaps Sid can provide more detail)
>> but
>> I think it might accomplish your goals.
>>
>> J
>>
>> On Fri, Jul 8, 2016 at 1:11 PM Paul Minton <pm...@change.org> wrote:
>>
>> > +1 to this feature as well.
>> >
>> > I wonder if it would be possible to pass along some context when
>> triggering
>> > a job. This may be outside the scope of this thread, but it would allow
>> for
>> > more dynamic runs of a job. As a simple example, I may want to kick off
>> a
>> > job and pass along the key to a file on s3. Right now we would depend
>> on a
>> > initial s3 sensor, but that would require that the filename be static
>> > across runs.
>> >
>> > On Thu, Jul 7, 2016 at 9:55 AM, Chris Riccomini <cr...@apache.org>
>> > wrote:
>> >
>> > > +1
>> > >
>> > > On Thu, Jul 7, 2016 at 5:18 AM, Bolke de Bruin <bd...@gmail.com>
>> > wrote:
>> > >
>> > > > Ideally the CLI and WebUI should both access an API that handles
>> > > > authentication and authorization. This would resolve both issues.
>> > > However,
>> > > > the UI already allows for authentication and to a lesser extent
>> > > > authorization. Thus allowing this from the UI (which we already do
>> for
>> > > > Celery) is not a big change.
>> > > >
>> > > > - Bolke
>> > > >
>> > > >
>> > > > > Op 7 jul. 2016, om 11:01 heeft Alexander Alten-Lorenz <
>> > > > wget.null@gmail.com> het volgende geschreven:
>> > > > >
>> > > > > Sounds good, but on the other hand I'm with Maxime. Given that the
>> > task
>> > > > can be triggered per CLI, the functionality is available but needs a
>> > > local
>> > > > login. When the "run" button now would be available for everyone who
>> > has
>> > > > access to the UI, I can imagine that would cause some serious load
>> > issues
>> > > > in a production environment, especially with SLA based workflow
>> setups.
>> > > > > On the other hand, when the "run" button with a local executor
>> would
>> > > > queue the task in a control queue (like "external triggered") a
>> admin
>> > > could
>> > > > finally mark them as "approved".
>> > > > >
>> > > > > --alex
>> > > > >
>> > > > >> On Jul 7, 2016, at 12:12 AM, Jeremiah Lowin <jl...@apache.org>
>> > > wrote:
>> > > > >>
>> > > > >> Perhaps it's a good chance to revisit the functionality. Right
>> now
>> > the
>> > > > UI
>> > > > >> "run" button actually runs the task via CeleryExecutor. Perhaps
>> > > instead
>> > > > (or
>> > > > >> just when using a non-Celery executor) it should queue the task
>> and
>> > > let
>> > > > the
>> > > > >> Scheduler pick it up. I guess in that case it would just be sugar
>> > for
>> > > > >> marking a TI as QUEUED. Just a thought.
>> > > > >>
>> > > > >> On Wed, Jul 6, 2016 at 2:54 AM Maxime Beauchemin <
>> > > > maximebeauchemin@gmail.com>
>> > > > >> wrote:
>> > > > >>
>> > > > >>> Hi,
>> > > > >>>
>> > > > >>> The problem is that a web server isn't the right place to run an
>> > > > airflow
>> > > > >>> task. From the context of the web request scope we have to
>> somehow
>> > > > pass a
>> > > > >>> message to an external executor to run the task. For
>> LocalExecutor
>> > to
>> > > > work
>> > > > >>> the web server would have to start a LocalExecutor as a sub
>> process
>> > > and
>> > > > >>> that doesn't sound like a great idea...
>> > > > >>>
>> > > > >>> Max
>> > > > >>>
>> > > > >>> On Tue, Jul 5, 2016 at 11:22 AM, Jason Chen <
>> > > chingchien.chen@gmail.com
>> > > > >
>> > > > >>> wrote:
>> > > > >>>
>> > > > >>>> Hi Airflow team,
>> > > > >>>> I am using the "LocalExecutor" and it works very well to run
>> the
>> > > > >>> workflow
>> > > > >>>> I setup.
>> > > > >>>>
>> > > > >>>> I noticed that, from the UI, it can trigger a task to run.
>> > > > >>>> However, I got the error "Only works with the CeleryExecutor,
>> > sorry
>> > > ".
>> > > > >>>> I can ssh into airflow node and run the command line from
>> there.
>> > > > >>>> However, it would be nice to just run it from airflow UI.
>> > > > >>>> Is it possible to do that (with "LocalExecutor") or it's a
>> future
>> > > > feature
>> > > > >>>> to consider ?
>> > > > >>>>
>> > > > >>>> Thanks.
>> > > > >>>> Jason
>> > > > >>>>
>> > > > >>>
>> > > > >
>> > > >
>> > > >
>> > >
>> >
>>
>
>
Re: Running a task from the Airflow UI
Posted by Paul Minton <pm...@change.org>.
Thanks Jeremiah. I just noticed this feature this weekend.
I'm wondering if that functionality could be exposed to the REST api. Like
a post to "airflow.run" with a json object in the payload that gets
included with the pickled context object.
That would 1) allow for dynamic runs and 2) (more importantly for our use
case) expose dynamic runs via HTTP making it super simple for external
resources to trigger dag runs.
I'm still getting familiar with the source so I'm not sure how difficult
that would be to implement.
On Mon, Jul 11, 2016 at 6:44 AM, Jeremiah Lowin <jl...@apache.org> wrote:
> Paul,
>
> The trigger_dag cli command accepts a json "conf" parameter. To be honest
> I'm not familiar with the feature (perhaps Sid can provide more detail) but
> I think it might accomplish your goals.
>
> J
>
> On Fri, Jul 8, 2016 at 1:11 PM Paul Minton <pm...@change.org> wrote:
>
> > +1 to this feature as well.
> >
> > I wonder if it would be possible to pass along some context when
> triggering
> > a job. This may be outside the scope of this thread, but it would allow
> for
> > more dynamic runs of a job. As a simple example, I may want to kick off a
> > job and pass along the key to a file on s3. Right now we would depend on
> a
> > initial s3 sensor, but that would require that the filename be static
> > across runs.
> >
> > On Thu, Jul 7, 2016 at 9:55 AM, Chris Riccomini <cr...@apache.org>
> > wrote:
> >
> > > +1
> > >
> > > On Thu, Jul 7, 2016 at 5:18 AM, Bolke de Bruin <bd...@gmail.com>
> > wrote:
> > >
> > > > Ideally the CLI and WebUI should both access an API that handles
> > > > authentication and authorization. This would resolve both issues.
> > > However,
> > > > the UI already allows for authentication and to a lesser extent
> > > > authorization. Thus allowing this from the UI (which we already do
> for
> > > > Celery) is not a big change.
> > > >
> > > > - Bolke
> > > >
> > > >
> > > > > Op 7 jul. 2016, om 11:01 heeft Alexander Alten-Lorenz <
> > > > wget.null@gmail.com> het volgende geschreven:
> > > > >
> > > > > Sounds good, but on the other hand I'm with Maxime. Given that the
> > task
> > > > can be triggered per CLI, the functionality is available but needs a
> > > local
> > > > login. When the "run" button now would be available for everyone who
> > has
> > > > access to the UI, I can imagine that would cause some serious load
> > issues
> > > > in a production environment, especially with SLA based workflow
> setups.
> > > > > On the other hand, when the "run" button with a local executor
> would
> > > > queue the task in a control queue (like "external triggered") a admin
> > > could
> > > > finally mark them as "approved".
> > > > >
> > > > > --alex
> > > > >
> > > > >> On Jul 7, 2016, at 12:12 AM, Jeremiah Lowin <jl...@apache.org>
> > > wrote:
> > > > >>
> > > > >> Perhaps it's a good chance to revisit the functionality. Right now
> > the
> > > > UI
> > > > >> "run" button actually runs the task via CeleryExecutor. Perhaps
> > > instead
> > > > (or
> > > > >> just when using a non-Celery executor) it should queue the task
> and
> > > let
> > > > the
> > > > >> Scheduler pick it up. I guess in that case it would just be sugar
> > for
> > > > >> marking a TI as QUEUED. Just a thought.
> > > > >>
> > > > >> On Wed, Jul 6, 2016 at 2:54 AM Maxime Beauchemin <
> > > > maximebeauchemin@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >>> Hi,
> > > > >>>
> > > > >>> The problem is that a web server isn't the right place to run an
> > > > airflow
> > > > >>> task. From the context of the web request scope we have to
> somehow
> > > > pass a
> > > > >>> message to an external executor to run the task. For
> LocalExecutor
> > to
> > > > work
> > > > >>> the web server would have to start a LocalExecutor as a sub
> process
> > > and
> > > > >>> that doesn't sound like a great idea...
> > > > >>>
> > > > >>> Max
> > > > >>>
> > > > >>> On Tue, Jul 5, 2016 at 11:22 AM, Jason Chen <
> > > chingchien.chen@gmail.com
> > > > >
> > > > >>> wrote:
> > > > >>>
> > > > >>>> Hi Airflow team,
> > > > >>>> I am using the "LocalExecutor" and it works very well to run the
> > > > >>> workflow
> > > > >>>> I setup.
> > > > >>>>
> > > > >>>> I noticed that, from the UI, it can trigger a task to run.
> > > > >>>> However, I got the error "Only works with the CeleryExecutor,
> > sorry
> > > ".
> > > > >>>> I can ssh into airflow node and run the command line from there.
> > > > >>>> However, it would be nice to just run it from airflow UI.
> > > > >>>> Is it possible to do that (with "LocalExecutor") or it's a
> future
> > > > feature
> > > > >>>> to consider ?
> > > > >>>>
> > > > >>>> Thanks.
> > > > >>>> Jason
> > > > >>>>
> > > > >>>
> > > > >
> > > >
> > > >
> > >
> >
>
Re: Running a task from the Airflow UI
Posted by Jeremiah Lowin <jl...@apache.org>.
Paul,
The trigger_dag cli command accepts a json "conf" parameter. To be honest
I'm not familiar with the feature (perhaps Sid can provide more detail) but
I think it might accomplish your goals.
J
On Fri, Jul 8, 2016 at 1:11 PM Paul Minton <pm...@change.org> wrote:
> +1 to this feature as well.
>
> I wonder if it would be possible to pass along some context when triggering
> a job. This may be outside the scope of this thread, but it would allow for
> more dynamic runs of a job. As a simple example, I may want to kick off a
> job and pass along the key to a file on s3. Right now we would depend on a
> initial s3 sensor, but that would require that the filename be static
> across runs.
>
> On Thu, Jul 7, 2016 at 9:55 AM, Chris Riccomini <cr...@apache.org>
> wrote:
>
> > +1
> >
> > On Thu, Jul 7, 2016 at 5:18 AM, Bolke de Bruin <bd...@gmail.com>
> wrote:
> >
> > > Ideally the CLI and WebUI should both access an API that handles
> > > authentication and authorization. This would resolve both issues.
> > However,
> > > the UI already allows for authentication and to a lesser extent
> > > authorization. Thus allowing this from the UI (which we already do for
> > > Celery) is not a big change.
> > >
> > > - Bolke
> > >
> > >
> > > > Op 7 jul. 2016, om 11:01 heeft Alexander Alten-Lorenz <
> > > wget.null@gmail.com> het volgende geschreven:
> > > >
> > > > Sounds good, but on the other hand I'm with Maxime. Given that the
> task
> > > can be triggered per CLI, the functionality is available but needs a
> > local
> > > login. When the "run" button now would be available for everyone who
> has
> > > access to the UI, I can imagine that would cause some serious load
> issues
> > > in a production environment, especially with SLA based workflow setups.
> > > > On the other hand, when the "run" button with a local executor would
> > > queue the task in a control queue (like "external triggered") a admin
> > could
> > > finally mark them as "approved".
> > > >
> > > > --alex
> > > >
> > > >> On Jul 7, 2016, at 12:12 AM, Jeremiah Lowin <jl...@apache.org>
> > wrote:
> > > >>
> > > >> Perhaps it's a good chance to revisit the functionality. Right now
> the
> > > UI
> > > >> "run" button actually runs the task via CeleryExecutor. Perhaps
> > instead
> > > (or
> > > >> just when using a non-Celery executor) it should queue the task and
> > let
> > > the
> > > >> Scheduler pick it up. I guess in that case it would just be sugar
> for
> > > >> marking a TI as QUEUED. Just a thought.
> > > >>
> > > >> On Wed, Jul 6, 2016 at 2:54 AM Maxime Beauchemin <
> > > maximebeauchemin@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> Hi,
> > > >>>
> > > >>> The problem is that a web server isn't the right place to run an
> > > airflow
> > > >>> task. From the context of the web request scope we have to somehow
> > > pass a
> > > >>> message to an external executor to run the task. For LocalExecutor
> to
> > > work
> > > >>> the web server would have to start a LocalExecutor as a sub process
> > and
> > > >>> that doesn't sound like a great idea...
> > > >>>
> > > >>> Max
> > > >>>
> > > >>> On Tue, Jul 5, 2016 at 11:22 AM, Jason Chen <
> > chingchien.chen@gmail.com
> > > >
> > > >>> wrote:
> > > >>>
> > > >>>> Hi Airflow team,
> > > >>>> I am using the "LocalExecutor" and it works very well to run the
> > > >>> workflow
> > > >>>> I setup.
> > > >>>>
> > > >>>> I noticed that, from the UI, it can trigger a task to run.
> > > >>>> However, I got the error "Only works with the CeleryExecutor,
> sorry
> > ".
> > > >>>> I can ssh into airflow node and run the command line from there.
> > > >>>> However, it would be nice to just run it from airflow UI.
> > > >>>> Is it possible to do that (with "LocalExecutor") or it's a future
> > > feature
> > > >>>> to consider ?
> > > >>>>
> > > >>>> Thanks.
> > > >>>> Jason
> > > >>>>
> > > >>>
> > > >
> > >
> > >
> >
>
Re: Running a task from the Airflow UI
Posted by Paul Minton <pm...@change.org>.
+1 to this feature as well.
I wonder if it would be possible to pass along some context when triggering
a job. This may be outside the scope of this thread, but it would allow for
more dynamic runs of a job. As a simple example, I may want to kick off a
job and pass along the key to a file on s3. Right now we would depend on a
initial s3 sensor, but that would require that the filename be static
across runs.
On Thu, Jul 7, 2016 at 9:55 AM, Chris Riccomini <cr...@apache.org>
wrote:
> +1
>
> On Thu, Jul 7, 2016 at 5:18 AM, Bolke de Bruin <bd...@gmail.com> wrote:
>
> > Ideally the CLI and WebUI should both access an API that handles
> > authentication and authorization. This would resolve both issues.
> However,
> > the UI already allows for authentication and to a lesser extent
> > authorization. Thus allowing this from the UI (which we already do for
> > Celery) is not a big change.
> >
> > - Bolke
> >
> >
> > > Op 7 jul. 2016, om 11:01 heeft Alexander Alten-Lorenz <
> > wget.null@gmail.com> het volgende geschreven:
> > >
> > > Sounds good, but on the other hand I'm with Maxime. Given that the task
> > can be triggered per CLI, the functionality is available but needs a
> local
> > login. When the "run" button now would be available for everyone who has
> > access to the UI, I can imagine that would cause some serious load issues
> > in a production environment, especially with SLA based workflow setups.
> > > On the other hand, when the "run" button with a local executor would
> > queue the task in a control queue (like "external triggered") a admin
> could
> > finally mark them as "approved".
> > >
> > > --alex
> > >
> > >> On Jul 7, 2016, at 12:12 AM, Jeremiah Lowin <jl...@apache.org>
> wrote:
> > >>
> > >> Perhaps it's a good chance to revisit the functionality. Right now the
> > UI
> > >> "run" button actually runs the task via CeleryExecutor. Perhaps
> instead
> > (or
> > >> just when using a non-Celery executor) it should queue the task and
> let
> > the
> > >> Scheduler pick it up. I guess in that case it would just be sugar for
> > >> marking a TI as QUEUED. Just a thought.
> > >>
> > >> On Wed, Jul 6, 2016 at 2:54 AM Maxime Beauchemin <
> > maximebeauchemin@gmail.com>
> > >> wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> The problem is that a web server isn't the right place to run an
> > airflow
> > >>> task. From the context of the web request scope we have to somehow
> > pass a
> > >>> message to an external executor to run the task. For LocalExecutor to
> > work
> > >>> the web server would have to start a LocalExecutor as a sub process
> and
> > >>> that doesn't sound like a great idea...
> > >>>
> > >>> Max
> > >>>
> > >>> On Tue, Jul 5, 2016 at 11:22 AM, Jason Chen <
> chingchien.chen@gmail.com
> > >
> > >>> wrote:
> > >>>
> > >>>> Hi Airflow team,
> > >>>> I am using the "LocalExecutor" and it works very well to run the
> > >>> workflow
> > >>>> I setup.
> > >>>>
> > >>>> I noticed that, from the UI, it can trigger a task to run.
> > >>>> However, I got the error "Only works with the CeleryExecutor, sorry
> ".
> > >>>> I can ssh into airflow node and run the command line from there.
> > >>>> However, it would be nice to just run it from airflow UI.
> > >>>> Is it possible to do that (with "LocalExecutor") or it's a future
> > feature
> > >>>> to consider ?
> > >>>>
> > >>>> Thanks.
> > >>>> Jason
> > >>>>
> > >>>
> > >
> >
> >
>
Re: Running a task from the Airflow UI
Posted by Chris Riccomini <cr...@apache.org>.
+1
On Thu, Jul 7, 2016 at 5:18 AM, Bolke de Bruin <bd...@gmail.com> wrote:
> Ideally the CLI and WebUI should both access an API that handles
> authentication and authorization. This would resolve both issues. However,
> the UI already allows for authentication and to a lesser extent
> authorization. Thus allowing this from the UI (which we already do for
> Celery) is not a big change.
>
> - Bolke
>
>
> > Op 7 jul. 2016, om 11:01 heeft Alexander Alten-Lorenz <
> wget.null@gmail.com> het volgende geschreven:
> >
> > Sounds good, but on the other hand I'm with Maxime. Given that the task
> can be triggered per CLI, the functionality is available but needs a local
> login. When the "run" button now would be available for everyone who has
> access to the UI, I can imagine that would cause some serious load issues
> in a production environment, especially with SLA based workflow setups.
> > On the other hand, when the "run" button with a local executor would
> queue the task in a control queue (like "external triggered") a admin could
> finally mark them as "approved".
> >
> > --alex
> >
> >> On Jul 7, 2016, at 12:12 AM, Jeremiah Lowin <jl...@apache.org> wrote:
> >>
> >> Perhaps it's a good chance to revisit the functionality. Right now the
> UI
> >> "run" button actually runs the task via CeleryExecutor. Perhaps instead
> (or
> >> just when using a non-Celery executor) it should queue the task and let
> the
> >> Scheduler pick it up. I guess in that case it would just be sugar for
> >> marking a TI as QUEUED. Just a thought.
> >>
> >> On Wed, Jul 6, 2016 at 2:54 AM Maxime Beauchemin <
> maximebeauchemin@gmail.com>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> The problem is that a web server isn't the right place to run an
> airflow
> >>> task. From the context of the web request scope we have to somehow
> pass a
> >>> message to an external executor to run the task. For LocalExecutor to
> work
> >>> the web server would have to start a LocalExecutor as a sub process and
> >>> that doesn't sound like a great idea...
> >>>
> >>> Max
> >>>
> >>> On Tue, Jul 5, 2016 at 11:22 AM, Jason Chen <chingchien.chen@gmail.com
> >
> >>> wrote:
> >>>
> >>>> Hi Airflow team,
> >>>> I am using the "LocalExecutor" and it works very well to run the
> >>> workflow
> >>>> I setup.
> >>>>
> >>>> I noticed that, from the UI, it can trigger a task to run.
> >>>> However, I got the error "Only works with the CeleryExecutor, sorry ".
> >>>> I can ssh into airflow node and run the command line from there.
> >>>> However, it would be nice to just run it from airflow UI.
> >>>> Is it possible to do that (with "LocalExecutor") or it's a future
> feature
> >>>> to consider ?
> >>>>
> >>>> Thanks.
> >>>> Jason
> >>>>
> >>>
> >
>
>
Re: Running a task from the Airflow UI
Posted by Bolke de Bruin <bd...@gmail.com>.
Ideally the CLI and WebUI should both access an API that handles authentication and authorization. This would resolve both issues. However, the UI already allows for authentication and to a lesser extent authorization. Thus allowing this from the UI (which we already do for Celery) is not a big change.
- Bolke
> Op 7 jul. 2016, om 11:01 heeft Alexander Alten-Lorenz <wg...@gmail.com> het volgende geschreven:
>
> Sounds good, but on the other hand I'm with Maxime. Given that the task can be triggered per CLI, the functionality is available but needs a local login. When the "run" button now would be available for everyone who has access to the UI, I can imagine that would cause some serious load issues in a production environment, especially with SLA based workflow setups.
> On the other hand, when the "run" button with a local executor would queue the task in a control queue (like "external triggered") a admin could finally mark them as "approved".
>
> --alex
>
>> On Jul 7, 2016, at 12:12 AM, Jeremiah Lowin <jl...@apache.org> wrote:
>>
>> Perhaps it's a good chance to revisit the functionality. Right now the UI
>> "run" button actually runs the task via CeleryExecutor. Perhaps instead (or
>> just when using a non-Celery executor) it should queue the task and let the
>> Scheduler pick it up. I guess in that case it would just be sugar for
>> marking a TI as QUEUED. Just a thought.
>>
>> On Wed, Jul 6, 2016 at 2:54 AM Maxime Beauchemin <ma...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> The problem is that a web server isn't the right place to run an airflow
>>> task. From the context of the web request scope we have to somehow pass a
>>> message to an external executor to run the task. For LocalExecutor to work
>>> the web server would have to start a LocalExecutor as a sub process and
>>> that doesn't sound like a great idea...
>>>
>>> Max
>>>
>>> On Tue, Jul 5, 2016 at 11:22 AM, Jason Chen <ch...@gmail.com>
>>> wrote:
>>>
>>>> Hi Airflow team,
>>>> I am using the "LocalExecutor" and it works very well to run the
>>> workflow
>>>> I setup.
>>>>
>>>> I noticed that, from the UI, it can trigger a task to run.
>>>> However, I got the error "Only works with the CeleryExecutor, sorry ".
>>>> I can ssh into airflow node and run the command line from there.
>>>> However, it would be nice to just run it from airflow UI.
>>>> Is it possible to do that (with "LocalExecutor") or it's a future feature
>>>> to consider ?
>>>>
>>>> Thanks.
>>>> Jason
>>>>
>>>
>
Re: Running a task from the Airflow UI
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Sounds good, but on the other hand I'm with Maxime. Given that the task can be triggered per CLI, the functionality is available but needs a local login. When the "run" button now would be available for everyone who has access to the UI, I can imagine that would cause some serious load issues in a production environment, especially with SLA based workflow setups.
On the other hand, when the "run" button with a local executor would queue the task in a control queue (like "external triggered") a admin could finally mark them as "approved".
--alex
> On Jul 7, 2016, at 12:12 AM, Jeremiah Lowin <jl...@apache.org> wrote:
>
> Perhaps it's a good chance to revisit the functionality. Right now the UI
> "run" button actually runs the task via CeleryExecutor. Perhaps instead (or
> just when using a non-Celery executor) it should queue the task and let the
> Scheduler pick it up. I guess in that case it would just be sugar for
> marking a TI as QUEUED. Just a thought.
>
> On Wed, Jul 6, 2016 at 2:54 AM Maxime Beauchemin <ma...@gmail.com>
> wrote:
>
>> Hi,
>>
>> The problem is that a web server isn't the right place to run an airflow
>> task. From the context of the web request scope we have to somehow pass a
>> message to an external executor to run the task. For LocalExecutor to work
>> the web server would have to start a LocalExecutor as a sub process and
>> that doesn't sound like a great idea...
>>
>> Max
>>
>> On Tue, Jul 5, 2016 at 11:22 AM, Jason Chen <ch...@gmail.com>
>> wrote:
>>
>>> Hi Airflow team,
>>> I am using the "LocalExecutor" and it works very well to run the
>> workflow
>>> I setup.
>>>
>>> I noticed that, from the UI, it can trigger a task to run.
>>> However, I got the error "Only works with the CeleryExecutor, sorry ".
>>> I can ssh into airflow node and run the command line from there.
>>> However, it would be nice to just run it from airflow UI.
>>> Is it possible to do that (with "LocalExecutor") or it's a future feature
>>> to consider ?
>>>
>>> Thanks.
>>> Jason
>>>
>>
Re: Running a task from the Airflow UI
Posted by Jeremiah Lowin <jl...@apache.org>.
Perhaps it's a good chance to revisit the functionality. Right now the UI
"run" button actually runs the task via CeleryExecutor. Perhaps instead (or
just when using a non-Celery executor) it should queue the task and let the
Scheduler pick it up. I guess in that case it would just be sugar for
marking a TI as QUEUED. Just a thought.
On Wed, Jul 6, 2016 at 2:54 AM Maxime Beauchemin <ma...@gmail.com>
wrote:
> Hi,
>
> The problem is that a web server isn't the right place to run an airflow
> task. From the context of the web request scope we have to somehow pass a
> message to an external executor to run the task. For LocalExecutor to work
> the web server would have to start a LocalExecutor as a sub process and
> that doesn't sound like a great idea...
>
> Max
>
> On Tue, Jul 5, 2016 at 11:22 AM, Jason Chen <ch...@gmail.com>
> wrote:
>
> > Hi Airflow team,
> > I am using the "LocalExecutor" and it works very well to run the
> workflow
> > I setup.
> >
> > I noticed that, from the UI, it can trigger a task to run.
> > However, I got the error "Only works with the CeleryExecutor, sorry ".
> > I can ssh into airflow node and run the command line from there.
> > However, it would be nice to just run it from airflow UI.
> > Is it possible to do that (with "LocalExecutor") or it's a future feature
> > to consider ?
> >
> > Thanks.
> > Jason
> >
>
Re: Running a task from the Airflow UI
Posted by Maxime Beauchemin <ma...@gmail.com>.
Hi,
The problem is that a web server isn't the right place to run an airflow
task. From the context of the web request scope we have to somehow pass a
message to an external executor to run the task. For LocalExecutor to work
the web server would have to start a LocalExecutor as a sub process and
that doesn't sound like a great idea...
Max
On Tue, Jul 5, 2016 at 11:22 AM, Jason Chen <ch...@gmail.com>
wrote:
> Hi Airflow team,
> I am using the "LocalExecutor" and it works very well to run the workflow
> I setup.
>
> I noticed that, from the UI, it can trigger a task to run.
> However, I got the error "Only works with the CeleryExecutor, sorry ".
> I can ssh into airflow node and run the command line from there.
> However, it would be nice to just run it from airflow UI.
> Is it possible to do that (with "LocalExecutor") or it's a future feature
> to consider ?
>
> Thanks.
> Jason
>