You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Michael Gong <go...@hotmail.com> on 2017/03/03 16:49:52 UTC
Airflow running different with different user id ?
Hi,
Suppose I have 1 airflow instance running 2 different DAGs, is it possible to specify the 2 DAGs running under 2 different ids ?
Any advises are welcomed.
Thanks.
Michael
Re: Airflow running different with different user id ?
Posted by Bolke de Bruin <bd...@gmail.com>.
Nice management of expectations ;-).
Sent from my iPhone
> On 3 Mar 2017, at 21:44, Dan Davydov <da...@airbnb.com.INVALID> wrote:
>
> Within a couple of weeks.
>
>> On Fri, Mar 3, 2017 at 12:34 PM, Michael Gong <go...@hotmail.com> wrote:
>>
>> When approximately will it be released?
>>
>> Sent from my PP•KING™ smartphone
>>
>> On Mar 3, 2017 1:42 PM, Dan Davydov <da...@airbnb.com.INVALID>
>> wrote:
>> Yes it is starting on 1.8.0 which will be released soon, you can look in
>> the documentation/grep for "run_as".
>>
>>> On Mar 3, 2017 8:50 AM, "Michael Gong" <go...@hotmail.com> wrote:
>>>
>>> Hi,
>>>
>>>
>>> Suppose I have 1 airflow instance running 2 different DAGs, is it
>> possible
>>> to specify the 2 DAGs running under 2 different ids ?
>>>
>>>
>>> Any advises are welcomed.
>>>
>>>
>>> Thanks.
>>>
>>> Michael
>>>
>>>
>>>
>>>
>>>
>>
Re: "last_expired" column in the dag table
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Hi Michael,
Could you share the schedule with us? How is it configured. Please provide
the DAG object with the cron definition.
Cheers, Fokko
2018-02-21 5:09 GMT+01:00 Michael Gong <go...@hotmail.com>:
> Hi, airflow dev,
>
>
> We set up several dags running on the airflow server. Most dags run fine,
> but there is one dag which doesn't start at the scheduled time.
>
>
> Then we found in the "dag" table, there is a "last_expired" column. The
> not-started dag has a value, but it's empty for all other dags.
>
>
> Did anyone know what "last_expired" column means ? Could it prevent a dag
> to run ?
>
>
> Any comments are welcomed.
>
>
> Thanks.
>
> Michael
>
>
>
"last_expired" column in the dag table
Posted by Michael Gong <go...@hotmail.com>.
Hi, airflow dev,
We set up several dags running on the airflow server. Most dags run fine, but there is one dag which doesn't start at the scheduled time.
Then we found in the "dag" table, there is a "last_expired" column. The not-started dag has a value, but it's empty for all other dags.
Did anyone know what "last_expired" column means ? Could it prevent a dag to run ?
Any comments are welcomed.
Thanks.
Michael
Re: Airflow (1.6) dag's status "Failed" even when a task was running
Posted by Bolke de Bruin <bd...@gmail.com>.
Airflow 1.6 ??!!
I don’t think any of the maintainers is running that version anymore.
> On 16 Jan 2018, at 19:30, Michael Gong <go...@hotmail.com> wrote:
>
> Hi, Airflow developers,
>
>
> On Airflow 1.6, we found sometimes a dag's status was set to "Failed" even when some tasks were still running.
>
>
> The steps to illustrate this behavior is: (suppose a dag contains 2 tasks , A and B.)
>
>
> 1.
> let task A and B starts running. --> The dag status is "Running".
> 2.
> Clear task A. --> the task A process will be killed and the task A will set to "Failed".
> 3.
> Task B is still running.
> 4.
> The dag status becomes "Failed", even though the Task B is still running.
>
>
> I wonder whether this is an Airflow bug or an expected behavior.
>
>
>
> Any comments are welcome.
>
>
> Thanks.
>
> Michael
>
>
>
>
>
>
>
> ________________________________
> From: Michael Gong <go...@hotmail.com>
> Sent: Friday, March 3, 2017 8:56 PM
> To: dev@airflow.incubator.apache.org
> Subject: Re: Airflow running different with different user id ?
>
> Hi, Dan,
>
>
> Thanks for the encouraging news.
>
>
> Maybe you could direct me to the documentation about "run_as" so I could see whether it is for my purpose.
>
>
> I guess it will be passed as an argument for the DAG object , right ?
>
>
> Thanks again.
>
>
> Michael
>
>
>
>
> ________________________________
> From: Dan Davydov <da...@airbnb.com.INVALID>
> Sent: Friday, March 3, 2017 8:44 PM
> To: dev@airflow.incubator.apache.org
> Subject: Re: Airflow running different with different user id ?
>
> Within a couple of weeks.
>
> On Fri, Mar 3, 2017 at 12:34 PM, Michael Gong <go...@hotmail.com> wrote:
>
>> When approximately will it be released?
>>
>> Sent from my PP•KING™ smartphone
>>
>> On Mar 3, 2017 1:42 PM, Dan Davydov <da...@airbnb.com.INVALID>
>> wrote:
>> Yes it is starting on 1.8.0 which will be released soon, you can look in
>> the documentation/grep for "run_as".
>>
>> On Mar 3, 2017 8:50 AM, "Michael Gong" <go...@hotmail.com> wrote:
>>
>>> Hi,
>>>
>>>
>>> Suppose I have 1 airflow instance running 2 different DAGs, is it
>> possible
>>> to specify the 2 DAGs running under 2 different ids ?
>>>
>>>
>>> Any advises are welcomed.
>>>
>>>
>>> Thanks.
>>>
>>> Michael
>>>
>>>
>>>
>>>
>>>
>>
Airflow (1.6) dag's status "Failed" even when a task was running
Posted by Michael Gong <go...@hotmail.com>.
Hi, Airflow developers,
On Airflow 1.6, we found sometimes a dag's status was set to "Failed" even when some tasks were still running.
The steps to illustrate this behavior is: (suppose a dag contains 2 tasks , A and B.)
1.
let task A and B starts running. --> The dag status is "Running".
2.
Clear task A. --> the task A process will be killed and the task A will set to "Failed".
3.
Task B is still running.
4.
The dag status becomes "Failed", even though the Task B is still running.
I wonder whether this is an Airflow bug or an expected behavior.
Any comments are welcome.
Thanks.
Michael
________________________________
From: Michael Gong <go...@hotmail.com>
Sent: Friday, March 3, 2017 8:56 PM
To: dev@airflow.incubator.apache.org
Subject: Re: Airflow running different with different user id ?
Hi, Dan,
Thanks for the encouraging news.
Maybe you could direct me to the documentation about "run_as" so I could see whether it is for my purpose.
I guess it will be passed as an argument for the DAG object , right ?
Thanks again.
Michael
________________________________
From: Dan Davydov <da...@airbnb.com.INVALID>
Sent: Friday, March 3, 2017 8:44 PM
To: dev@airflow.incubator.apache.org
Subject: Re: Airflow running different with different user id ?
Within a couple of weeks.
On Fri, Mar 3, 2017 at 12:34 PM, Michael Gong <go...@hotmail.com> wrote:
> When approximately will it be released?
>
> Sent from my PP•KING™ smartphone
>
> On Mar 3, 2017 1:42 PM, Dan Davydov <da...@airbnb.com.INVALID>
> wrote:
> Yes it is starting on 1.8.0 which will be released soon, you can look in
> the documentation/grep for "run_as".
>
> On Mar 3, 2017 8:50 AM, "Michael Gong" <go...@hotmail.com> wrote:
>
> > Hi,
> >
> >
> > Suppose I have 1 airflow instance running 2 different DAGs, is it
> possible
> > to specify the 2 DAGs running under 2 different ids ?
> >
> >
> > Any advises are welcomed.
> >
> >
> > Thanks.
> >
> > Michael
> >
> >
> >
> >
> >
>
Re: Airflow running different with different user id ?
Posted by Michael Gong <go...@hotmail.com>.
Hi, Dan,
The only doc mentioning "run_as" was in the airflow.operators.HiveOperator.
Is this what you mean "run_as"?
If not, can you provide more information about it? We are very interested to know more .
Thanks.
Michael
Sent from my PP•KING™ smartphone
On Mar 3, 2017 3:57 PM, Michael Gong <go...@hotmail.com> wrote:
Hi, Dan,
Thanks for the encouraging news.
Maybe you could direct me to the documentation about "run_as" so I could see whether it is for my purpose.
I guess it will be passed as an argument for the DAG object , right ?
Thanks again.
Michael
________________________________
From: Dan Davydov <da...@airbnb.com.INVALID>
Sent: Friday, March 3, 2017 8:44 PM
To: dev@airflow.incubator.apache.org
Subject: Re: Airflow running different with different user id ?
Within a couple of weeks.
On Fri, Mar 3, 2017 at 12:34 PM, Michael Gong <go...@hotmail.com> wrote:
> When approximately will it be released?
>
> Sent from my PP•KING™ smartphone
>
> On Mar 3, 2017 1:42 PM, Dan Davydov <da...@airbnb.com.INVALID>
> wrote:
> Yes it is starting on 1.8.0 which will be released soon, you can look in
> the documentation/grep for "run_as".
>
> On Mar 3, 2017 8:50 AM, "Michael Gong" <go...@hotmail.com> wrote:
>
> > Hi,
> >
> >
> > Suppose I have 1 airflow instance running 2 different DAGs, is it
> possible
> > to specify the 2 DAGs running under 2 different ids ?
> >
> >
> > Any advises are welcomed.
> >
> >
> > Thanks.
> >
> > Michael
> >
> >
> >
> >
> >
>
Re: airflow 1.8.0 documentation ?
Posted by Maxime Beauchemin <ma...@gmail.com>.
Thanks to Daniel and Arthur, we now have a few versions of the docs up on
ReadTheDocs.
https://readthedocs.org/projects/airflow/
I updated the wiki with the links. Let me know if you think we need to
publish older versions of the docs.
Max
On Wed, Mar 8, 2017 at 2:16 PM, Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:
> Thanks for taking that on. It's mostly an effort in mocking all the libs
> that cannot be built in the readthedocs environment. This may require a
> specific `requirements.txt`->`readthedocs-reqs.txt` for that purpose with
> only the core dependencies needed to parse the modules.
>
> Max
>
> On Wed, Mar 8, 2017 at 1:40 PM, Daniel Huang <dx...@gmail.com> wrote:
>
>> Just filed https://issues.apache.org/jira/browse/AIRFLOW-956 to setup
>> readthedocs. I'll give it a shot.
>>
>> On Wed, Mar 8, 2017 at 12:29 PM, Jake Gysland <jg...@optoro.com>
>> wrote:
>>
>> > I'm glad there's new documentation available for 1.8.0, but replacing
>> the
>> > documentation for the latest stable release with the documentation for a
>> > future release candidate can lead to experiences that are ...
>> frustrating,
>> > as I've discovered this afternoon while trying to help a colleague QA a
>> DAG
>> > and finding documentation for CLI flags that don't exist.
>> >
>> > Can we at least get a version number (or... a commit hash?) into the
>> > documentation somewhere that's reasonably visible?
>> >
>> > On Mon, Mar 6, 2017 at 12:20 PM, Maxime Beauchemin <
>> > maximebeauchemin@gmail.com> wrote:
>> >
>> > > I just pushed a new version of the docs out.
>> > >
>> > > We may want to set up a service like https://readthedocs.org/ to
>> offer
>> > > versioned docs. It would require setting up some "mocking" of external
>> > libs
>> > > that can't be installed on the RTD build nodes and that actually
>> > shouldn't
>> > > be required to build the docs. From my understanding the mocking of
>> libs
>> > > would have to be done here in this file:
>> > > https://github.com/airbnb/superset/blob/master/docs/conf.py
>> > >
>> > > It would be great if someone wanted to take that on.
>> > >
>> > > Max
>> > >
>> > > On Mon, Mar 6, 2017 at 8:46 AM, Michael Gong <go...@hotmail.com>
>> wrote:
>> > >
>> > > > Hi,
>> > > >
>> > > >
>> > > > Does anyone know where I can find the documentation for the coming
>> > > airflow
>> > > > 1.8.0 release ?
>> > > >
>> > > >
>> > > > thanks.
>> > > >
>> > > > Michael
>> > > >
>> > > >
>> > > > ________________________________
>> > > > From: Michael Gong <go...@hotmail.com>
>> > > > Sent: Monday, March 6, 2017 2:37 PM
>> > > > To: dev; Michael Gong
>> > > > Subject: Re: Airflow running different with different user id ?
>> > > >
>> > > > Hi, Dan,
>> > > >
>> > > > The only doc mentioning "run_as" was in the airflow.operators.
>> > > > HiveOperator.
>> > > >
>> > > > Is this what you mean "run_as"?
>> > > >
>> > > > If not, can you provide more information about it? We are very
>> > interested
>> > > > to know more .
>> > > >
>> > > > Thanks.
>> > > > Michael
>> > > >
>> > > > Sent from my PP•KING™ smartphone
>> > > >
>> > > > On Mar 3, 2017 3:57 PM, Michael Gong <go...@hotmail.com> wrote:
>> > > > Hi, Dan,
>> > > >
>> > > >
>> > > > Thanks for the encouraging news.
>> > > >
>> > > >
>> > > > Maybe you could direct me to the documentation about "run_as" so I
>> > could
>> > > > see whether it is for my purpose.
>> > > >
>> > > >
>> > > > I guess it will be passed as an argument for the DAG object , right
>> ?
>> > > >
>> > > >
>> > > > Thanks again.
>> > > >
>> > > >
>> > > > Michael
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > ________________________________
>> > > > From: Dan Davydov <da...@airbnb.com.INVALID>
>> > > > Sent: Friday, March 3, 2017 8:44 PM
>> > > > To: dev@airflow.incubator.apache.org
>> > > > Subject: Re: Airflow running different with different user id ?
>> > > >
>> > > > Within a couple of weeks.
>> > > >
>> > > > On Fri, Mar 3, 2017 at 12:34 PM, Michael Gong <go...@hotmail.com>
>> > wrote:
>> > > >
>> > > > > When approximately will it be released?
>> > > > >
>> > > > > Sent from my PP•KING™ smartphone
>> > > > >
>> > > > > On Mar 3, 2017 1:42 PM, Dan Davydov <dan.davydov@airbnb.com
>> .INVALID>
>> > > > > wrote:
>> > > > > Yes it is starting on 1.8.0 which will be released soon, you can
>> look
>> > > in
>> > > > > the documentation/grep for "run_as".
>> > > > >
>> > > > > On Mar 3, 2017 8:50 AM, "Michael Gong" <go...@hotmail.com> wrote:
>> > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > >
>> > > > > > Suppose I have 1 airflow instance running 2 different DAGs, is
>> it
>> > > > > possible
>> > > > > > to specify the 2 DAGs running under 2 different ids ?
>> > > > > >
>> > > > > >
>> > > > > > Any advises are welcomed.
>> > > > > >
>> > > > > >
>> > > > > > Thanks.
>> > > > > >
>> > > > > > Michael
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> >
>> > Jake Gysland
>> >
>> > Data Pipeline Engineer
>> >
>> > www.optoro.com
>> >
>> > [image: Optoro, Inc.] <http://www.optoro.com/>
>> >
>>
>
>
Re: airflow 1.8.0 documentation ?
Posted by Maxime Beauchemin <ma...@gmail.com>.
Thanks for taking that on. It's mostly an effort in mocking all the libs
that cannot be built in the readthedocs environment. This may require a
specific `requirements.txt`->`readthedocs-reqs.txt` for that purpose with
only the core dependencies needed to parse the modules.
Max
On Wed, Mar 8, 2017 at 1:40 PM, Daniel Huang <dx...@gmail.com> wrote:
> Just filed https://issues.apache.org/jira/browse/AIRFLOW-956 to setup
> readthedocs. I'll give it a shot.
>
> On Wed, Mar 8, 2017 at 12:29 PM, Jake Gysland <jg...@optoro.com> wrote:
>
> > I'm glad there's new documentation available for 1.8.0, but replacing the
> > documentation for the latest stable release with the documentation for a
> > future release candidate can lead to experiences that are ...
> frustrating,
> > as I've discovered this afternoon while trying to help a colleague QA a
> DAG
> > and finding documentation for CLI flags that don't exist.
> >
> > Can we at least get a version number (or... a commit hash?) into the
> > documentation somewhere that's reasonably visible?
> >
> > On Mon, Mar 6, 2017 at 12:20 PM, Maxime Beauchemin <
> > maximebeauchemin@gmail.com> wrote:
> >
> > > I just pushed a new version of the docs out.
> > >
> > > We may want to set up a service like https://readthedocs.org/ to offer
> > > versioned docs. It would require setting up some "mocking" of external
> > libs
> > > that can't be installed on the RTD build nodes and that actually
> > shouldn't
> > > be required to build the docs. From my understanding the mocking of
> libs
> > > would have to be done here in this file:
> > > https://github.com/airbnb/superset/blob/master/docs/conf.py
> > >
> > > It would be great if someone wanted to take that on.
> > >
> > > Max
> > >
> > > On Mon, Mar 6, 2017 at 8:46 AM, Michael Gong <go...@hotmail.com>
> wrote:
> > >
> > > > Hi,
> > > >
> > > >
> > > > Does anyone know where I can find the documentation for the coming
> > > airflow
> > > > 1.8.0 release ?
> > > >
> > > >
> > > > thanks.
> > > >
> > > > Michael
> > > >
> > > >
> > > > ________________________________
> > > > From: Michael Gong <go...@hotmail.com>
> > > > Sent: Monday, March 6, 2017 2:37 PM
> > > > To: dev; Michael Gong
> > > > Subject: Re: Airflow running different with different user id ?
> > > >
> > > > Hi, Dan,
> > > >
> > > > The only doc mentioning "run_as" was in the airflow.operators.
> > > > HiveOperator.
> > > >
> > > > Is this what you mean "run_as"?
> > > >
> > > > If not, can you provide more information about it? We are very
> > interested
> > > > to know more .
> > > >
> > > > Thanks.
> > > > Michael
> > > >
> > > > Sent from my PP•KING™ smartphone
> > > >
> > > > On Mar 3, 2017 3:57 PM, Michael Gong <go...@hotmail.com> wrote:
> > > > Hi, Dan,
> > > >
> > > >
> > > > Thanks for the encouraging news.
> > > >
> > > >
> > > > Maybe you could direct me to the documentation about "run_as" so I
> > could
> > > > see whether it is for my purpose.
> > > >
> > > >
> > > > I guess it will be passed as an argument for the DAG object , right ?
> > > >
> > > >
> > > > Thanks again.
> > > >
> > > >
> > > > Michael
> > > >
> > > >
> > > >
> > > >
> > > > ________________________________
> > > > From: Dan Davydov <da...@airbnb.com.INVALID>
> > > > Sent: Friday, March 3, 2017 8:44 PM
> > > > To: dev@airflow.incubator.apache.org
> > > > Subject: Re: Airflow running different with different user id ?
> > > >
> > > > Within a couple of weeks.
> > > >
> > > > On Fri, Mar 3, 2017 at 12:34 PM, Michael Gong <go...@hotmail.com>
> > wrote:
> > > >
> > > > > When approximately will it be released?
> > > > >
> > > > > Sent from my PP•KING™ smartphone
> > > > >
> > > > > On Mar 3, 2017 1:42 PM, Dan Davydov <dan.davydov@airbnb.com.
> INVALID>
> > > > > wrote:
> > > > > Yes it is starting on 1.8.0 which will be released soon, you can
> look
> > > in
> > > > > the documentation/grep for "run_as".
> > > > >
> > > > > On Mar 3, 2017 8:50 AM, "Michael Gong" <go...@hotmail.com> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > >
> > > > > > Suppose I have 1 airflow instance running 2 different DAGs, is it
> > > > > possible
> > > > > > to specify the 2 DAGs running under 2 different ids ?
> > > > > >
> > > > > >
> > > > > > Any advises are welcomed.
> > > > > >
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > Michael
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> >
> > Jake Gysland
> >
> > Data Pipeline Engineer
> >
> > www.optoro.com
> >
> > [image: Optoro, Inc.] <http://www.optoro.com/>
> >
>
Re: airflow 1.8.0 documentation ?
Posted by Daniel Huang <dx...@gmail.com>.
Just filed https://issues.apache.org/jira/browse/AIRFLOW-956 to setup
readthedocs. I'll give it a shot.
On Wed, Mar 8, 2017 at 12:29 PM, Jake Gysland <jg...@optoro.com> wrote:
> I'm glad there's new documentation available for 1.8.0, but replacing the
> documentation for the latest stable release with the documentation for a
> future release candidate can lead to experiences that are ... frustrating,
> as I've discovered this afternoon while trying to help a colleague QA a DAG
> and finding documentation for CLI flags that don't exist.
>
> Can we at least get a version number (or... a commit hash?) into the
> documentation somewhere that's reasonably visible?
>
> On Mon, Mar 6, 2017 at 12:20 PM, Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
>
> > I just pushed a new version of the docs out.
> >
> > We may want to set up a service like https://readthedocs.org/ to offer
> > versioned docs. It would require setting up some "mocking" of external
> libs
> > that can't be installed on the RTD build nodes and that actually
> shouldn't
> > be required to build the docs. From my understanding the mocking of libs
> > would have to be done here in this file:
> > https://github.com/airbnb/superset/blob/master/docs/conf.py
> >
> > It would be great if someone wanted to take that on.
> >
> > Max
> >
> > On Mon, Mar 6, 2017 at 8:46 AM, Michael Gong <go...@hotmail.com> wrote:
> >
> > > Hi,
> > >
> > >
> > > Does anyone know where I can find the documentation for the coming
> > airflow
> > > 1.8.0 release ?
> > >
> > >
> > > thanks.
> > >
> > > Michael
> > >
> > >
> > > ________________________________
> > > From: Michael Gong <go...@hotmail.com>
> > > Sent: Monday, March 6, 2017 2:37 PM
> > > To: dev; Michael Gong
> > > Subject: Re: Airflow running different with different user id ?
> > >
> > > Hi, Dan,
> > >
> > > The only doc mentioning "run_as" was in the airflow.operators.
> > > HiveOperator.
> > >
> > > Is this what you mean "run_as"?
> > >
> > > If not, can you provide more information about it? We are very
> interested
> > > to know more .
> > >
> > > Thanks.
> > > Michael
> > >
> > > Sent from my PP•KING™ smartphone
> > >
> > > On Mar 3, 2017 3:57 PM, Michael Gong <go...@hotmail.com> wrote:
> > > Hi, Dan,
> > >
> > >
> > > Thanks for the encouraging news.
> > >
> > >
> > > Maybe you could direct me to the documentation about "run_as" so I
> could
> > > see whether it is for my purpose.
> > >
> > >
> > > I guess it will be passed as an argument for the DAG object , right ?
> > >
> > >
> > > Thanks again.
> > >
> > >
> > > Michael
> > >
> > >
> > >
> > >
> > > ________________________________
> > > From: Dan Davydov <da...@airbnb.com.INVALID>
> > > Sent: Friday, March 3, 2017 8:44 PM
> > > To: dev@airflow.incubator.apache.org
> > > Subject: Re: Airflow running different with different user id ?
> > >
> > > Within a couple of weeks.
> > >
> > > On Fri, Mar 3, 2017 at 12:34 PM, Michael Gong <go...@hotmail.com>
> wrote:
> > >
> > > > When approximately will it be released?
> > > >
> > > > Sent from my PP•KING™ smartphone
> > > >
> > > > On Mar 3, 2017 1:42 PM, Dan Davydov <da...@airbnb.com.INVALID>
> > > > wrote:
> > > > Yes it is starting on 1.8.0 which will be released soon, you can look
> > in
> > > > the documentation/grep for "run_as".
> > > >
> > > > On Mar 3, 2017 8:50 AM, "Michael Gong" <go...@hotmail.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > >
> > > > > Suppose I have 1 airflow instance running 2 different DAGs, is it
> > > > possible
> > > > > to specify the 2 DAGs running under 2 different ids ?
> > > > >
> > > > >
> > > > > Any advises are welcomed.
> > > > >
> > > > >
> > > > > Thanks.
> > > > >
> > > > > Michael
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
>
> Jake Gysland
>
> Data Pipeline Engineer
>
> www.optoro.com
>
> [image: Optoro, Inc.] <http://www.optoro.com/>
>
Re: airflow 1.8.0 documentation ?
Posted by Jake Gysland <jg...@optoro.com>.
I'm glad there's new documentation available for 1.8.0, but replacing the
documentation for the latest stable release with the documentation for a
future release candidate can lead to experiences that are ... frustrating,
as I've discovered this afternoon while trying to help a colleague QA a DAG
and finding documentation for CLI flags that don't exist.
Can we at least get a version number (or... a commit hash?) into the
documentation somewhere that's reasonably visible?
On Mon, Mar 6, 2017 at 12:20 PM, Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:
> I just pushed a new version of the docs out.
>
> We may want to set up a service like https://readthedocs.org/ to offer
> versioned docs. It would require setting up some "mocking" of external libs
> that can't be installed on the RTD build nodes and that actually shouldn't
> be required to build the docs. From my understanding the mocking of libs
> would have to be done here in this file:
> https://github.com/airbnb/superset/blob/master/docs/conf.py
>
> It would be great if someone wanted to take that on.
>
> Max
>
> On Mon, Mar 6, 2017 at 8:46 AM, Michael Gong <go...@hotmail.com> wrote:
>
> > Hi,
> >
> >
> > Does anyone know where I can find the documentation for the coming
> airflow
> > 1.8.0 release ?
> >
> >
> > thanks.
> >
> > Michael
> >
> >
> > ________________________________
> > From: Michael Gong <go...@hotmail.com>
> > Sent: Monday, March 6, 2017 2:37 PM
> > To: dev; Michael Gong
> > Subject: Re: Airflow running different with different user id ?
> >
> > Hi, Dan,
> >
> > The only doc mentioning "run_as" was in the airflow.operators.
> > HiveOperator.
> >
> > Is this what you mean "run_as"?
> >
> > If not, can you provide more information about it? We are very interested
> > to know more .
> >
> > Thanks.
> > Michael
> >
> > Sent from my PP•KING™ smartphone
> >
> > On Mar 3, 2017 3:57 PM, Michael Gong <go...@hotmail.com> wrote:
> > Hi, Dan,
> >
> >
> > Thanks for the encouraging news.
> >
> >
> > Maybe you could direct me to the documentation about "run_as" so I could
> > see whether it is for my purpose.
> >
> >
> > I guess it will be passed as an argument for the DAG object , right ?
> >
> >
> > Thanks again.
> >
> >
> > Michael
> >
> >
> >
> >
> > ________________________________
> > From: Dan Davydov <da...@airbnb.com.INVALID>
> > Sent: Friday, March 3, 2017 8:44 PM
> > To: dev@airflow.incubator.apache.org
> > Subject: Re: Airflow running different with different user id ?
> >
> > Within a couple of weeks.
> >
> > On Fri, Mar 3, 2017 at 12:34 PM, Michael Gong <go...@hotmail.com> wrote:
> >
> > > When approximately will it be released?
> > >
> > > Sent from my PP•KING™ smartphone
> > >
> > > On Mar 3, 2017 1:42 PM, Dan Davydov <da...@airbnb.com.INVALID>
> > > wrote:
> > > Yes it is starting on 1.8.0 which will be released soon, you can look
> in
> > > the documentation/grep for "run_as".
> > >
> > > On Mar 3, 2017 8:50 AM, "Michael Gong" <go...@hotmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > >
> > > > Suppose I have 1 airflow instance running 2 different DAGs, is it
> > > possible
> > > > to specify the 2 DAGs running under 2 different ids ?
> > > >
> > > >
> > > > Any advises are welcomed.
> > > >
> > > >
> > > > Thanks.
> > > >
> > > > Michael
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
>
--
Jake Gysland
Data Pipeline Engineer
www.optoro.com
[image: Optoro, Inc.] <http://www.optoro.com/>
Re: airflow 1.8.0 documentation ?
Posted by Maxime Beauchemin <ma...@gmail.com>.
I just pushed a new version of the docs out.
We may want to set up a service like https://readthedocs.org/ to offer
versioned docs. It would require setting up some "mocking" of external libs
that can't be installed on the RTD build nodes and that actually shouldn't
be required to build the docs. From my understanding the mocking of libs
would have to be done here in this file:
https://github.com/airbnb/superset/blob/master/docs/conf.py
It would be great if someone wanted to take that on.
Max
On Mon, Mar 6, 2017 at 8:46 AM, Michael Gong <go...@hotmail.com> wrote:
> Hi,
>
>
> Does anyone know where I can find the documentation for the coming airflow
> 1.8.0 release ?
>
>
> thanks.
>
> Michael
>
>
> ________________________________
> From: Michael Gong <go...@hotmail.com>
> Sent: Monday, March 6, 2017 2:37 PM
> To: dev; Michael Gong
> Subject: Re: Airflow running different with different user id ?
>
> Hi, Dan,
>
> The only doc mentioning "run_as" was in the airflow.operators.
> HiveOperator.
>
> Is this what you mean "run_as"?
>
> If not, can you provide more information about it? We are very interested
> to know more .
>
> Thanks.
> Michael
>
> Sent from my PP•KING™ smartphone
>
> On Mar 3, 2017 3:57 PM, Michael Gong <go...@hotmail.com> wrote:
> Hi, Dan,
>
>
> Thanks for the encouraging news.
>
>
> Maybe you could direct me to the documentation about "run_as" so I could
> see whether it is for my purpose.
>
>
> I guess it will be passed as an argument for the DAG object , right ?
>
>
> Thanks again.
>
>
> Michael
>
>
>
>
> ________________________________
> From: Dan Davydov <da...@airbnb.com.INVALID>
> Sent: Friday, March 3, 2017 8:44 PM
> To: dev@airflow.incubator.apache.org
> Subject: Re: Airflow running different with different user id ?
>
> Within a couple of weeks.
>
> On Fri, Mar 3, 2017 at 12:34 PM, Michael Gong <go...@hotmail.com> wrote:
>
> > When approximately will it be released?
> >
> > Sent from my PP•KING™ smartphone
> >
> > On Mar 3, 2017 1:42 PM, Dan Davydov <da...@airbnb.com.INVALID>
> > wrote:
> > Yes it is starting on 1.8.0 which will be released soon, you can look in
> > the documentation/grep for "run_as".
> >
> > On Mar 3, 2017 8:50 AM, "Michael Gong" <go...@hotmail.com> wrote:
> >
> > > Hi,
> > >
> > >
> > > Suppose I have 1 airflow instance running 2 different DAGs, is it
> > possible
> > > to specify the 2 DAGs running under 2 different ids ?
> > >
> > >
> > > Any advises are welcomed.
> > >
> > >
> > > Thanks.
> > >
> > > Michael
> > >
> > >
> > >
> > >
> > >
> >
>
airflow 1.8.0 documentation ?
Posted by Michael Gong <go...@hotmail.com>.
Hi,
Does anyone know where I can find the documentation for the coming airflow 1.8.0 release ?
thanks.
Michael
________________________________
From: Michael Gong <go...@hotmail.com>
Sent: Monday, March 6, 2017 2:37 PM
To: dev; Michael Gong
Subject: Re: Airflow running different with different user id ?
Hi, Dan,
The only doc mentioning "run_as" was in the airflow.operators.HiveOperator.
Is this what you mean "run_as"?
If not, can you provide more information about it? We are very interested to know more .
Thanks.
Michael
Sent from my PP•KING™ smartphone
On Mar 3, 2017 3:57 PM, Michael Gong <go...@hotmail.com> wrote:
Hi, Dan,
Thanks for the encouraging news.
Maybe you could direct me to the documentation about "run_as" so I could see whether it is for my purpose.
I guess it will be passed as an argument for the DAG object , right ?
Thanks again.
Michael
________________________________
From: Dan Davydov <da...@airbnb.com.INVALID>
Sent: Friday, March 3, 2017 8:44 PM
To: dev@airflow.incubator.apache.org
Subject: Re: Airflow running different with different user id ?
Within a couple of weeks.
On Fri, Mar 3, 2017 at 12:34 PM, Michael Gong <go...@hotmail.com> wrote:
> When approximately will it be released?
>
> Sent from my PP•KING™ smartphone
>
> On Mar 3, 2017 1:42 PM, Dan Davydov <da...@airbnb.com.INVALID>
> wrote:
> Yes it is starting on 1.8.0 which will be released soon, you can look in
> the documentation/grep for "run_as".
>
> On Mar 3, 2017 8:50 AM, "Michael Gong" <go...@hotmail.com> wrote:
>
> > Hi,
> >
> >
> > Suppose I have 1 airflow instance running 2 different DAGs, is it
> possible
> > to specify the 2 DAGs running under 2 different ids ?
> >
> >
> > Any advises are welcomed.
> >
> >
> > Thanks.
> >
> > Michael
> >
> >
> >
> >
> >
>
Airflow Impersonation feature using ssh ? ( Re: Airflow running
different with different user id ? )
Posted by Michael Gong <go...@hotmail.com>.
Hi,
In the coming airflow 1.8, a feature of Impersonation will be added to allow a task using another user name.
#####################################
Impersonation¶<https://airflow.apache.org/security.html?highlight=impersonation#impersonation>
Airflow has the ability to impersonate a unix user while running task instances based on the task’s run_as_user parameter, which takes a user’s name.
NOTE For impersonations to work, Airflow must be run with sudo as subtasks are run with sudo -u and permissions of files are changed. Furthermore, the unix user needs to exist on the worker. Here is what a simple sudoers file entry could look like to achieve this, assuming as airflow is running as the airflow user. Note that this means that the airflow user must be trusted and treated the same way as the root user.
...
####################################
From the above doc, it seems the new user must be on the sudoers file entry of the root user.
However, due to some internal policy , we are not allowed to add a user to the sudoers file.
Hence, I wonder whether airflow could add an option to use ssh , instead of sudo, to work with the Impersonation feature, ie. the "run_as_user" parameter ?
Any suggestions ?
Thanks.
Michael
________________________________
From: Maxime Beauchemin <ma...@gmail.com>
Sent: Monday, March 6, 2017 5:15 PM
To: dev@airflow.incubator.apache.org
Cc: Michael Gong
Subject: Re: Airflow running different with different user id ?
Hi,
I just rebuilt + pushed the docs manually, and updated the wiki page as to how to do it.
https://cwiki.apache.org/confluence/display/AIRFLOW/Building+and+deploying+the+docs
We may want to automate this in the future though I hit a few bumps when trying to do that in the past.
Here's the doc entry:
https://airflow.apache.org/security.html?highlight=impersonation#impersonation
Max
On Mon, Mar 6, 2017 at 6:37 AM, Michael Gong <go...@hotmail.com>> wrote:
Hi, Dan,
The only doc mentioning "run_as" was in the airflow.operators.HiveOperator.
Is this what you mean "run_as"?
If not, can you provide more information about it? We are very interested to know more .
Thanks.
Michael
Sent from my PP•KING™ smartphone
On Mar 3, 2017 3:57 PM, Michael Gong <go...@hotmail.com>> wrote:
Hi, Dan,
Thanks for the encouraging news.
Maybe you could direct me to the documentation about "run_as" so I could see whether it is for my purpose.
I guess it will be passed as an argument for the DAG object , right ?
Thanks again.
Michael
________________________________
From: Dan Davydov <da...@airbnb.com>.INVALID>
Sent: Friday, March 3, 2017 8:44 PM
To: dev@airflow.incubator.apache.org<ma...@airflow.incubator.apache.org>
Subject: Re: Airflow running different with different user id ?
Within a couple of weeks.
On Fri, Mar 3, 2017 at 12:34 PM, Michael Gong <go...@hotmail.com>> wrote:
> When approximately will it be released?
>
> Sent from my PP•KING™ smartphone
>
> On Mar 3, 2017 1:42 PM, Dan Davydov <da...@airbnb.com>.INVALID>
> wrote:
> Yes it is starting on 1.8.0 which will be released soon, you can look in
> the documentation/grep for "run_as".
>
> On Mar 3, 2017 8:50 AM, "Michael Gong" <go...@hotmail.com>> wrote:
>
> > Hi,
> >
> >
> > Suppose I have 1 airflow instance running 2 different DAGs, is it
> possible
> > to specify the 2 DAGs running under 2 different ids ?
> >
> >
> > Any advises are welcomed.
> >
> >
> > Thanks.
> >
> > Michael
> >
> >
> >
> >
> >
>
Re: Airflow running different with different user id ?
Posted by Maxime Beauchemin <ma...@gmail.com>.
Hi,
I just rebuilt + pushed the docs manually, and updated the wiki page as to
how to do it.
https://cwiki.apache.org/confluence/display/AIRFLOW/Building+and+deploying+the+docs
We may want to automate this in the future though I hit a few bumps when
trying to do that in the past.
Here's the doc entry:
https://airflow.apache.org/security.html?highlight=impersonation#impersonation
Max
On Mon, Mar 6, 2017 at 6:37 AM, Michael Gong <go...@hotmail.com> wrote:
> Hi, Dan,
>
> The only doc mentioning "run_as" was in the airflow.operators.
> HiveOperator.
>
> Is this what you mean "run_as"?
>
> If not, can you provide more information about it? We are very interested
> to know more .
>
> Thanks.
> Michael
>
> Sent from my PP•KING™ smartphone
>
> On Mar 3, 2017 3:57 PM, Michael Gong <go...@hotmail.com> wrote:
> Hi, Dan,
>
>
> Thanks for the encouraging news.
>
>
> Maybe you could direct me to the documentation about "run_as" so I could
> see whether it is for my purpose.
>
>
> I guess it will be passed as an argument for the DAG object , right ?
>
>
> Thanks again.
>
>
> Michael
>
>
>
>
> ________________________________
> From: Dan Davydov <da...@airbnb.com.INVALID>
> Sent: Friday, March 3, 2017 8:44 PM
> To: dev@airflow.incubator.apache.org
> Subject: Re: Airflow running different with different user id ?
>
> Within a couple of weeks.
>
> On Fri, Mar 3, 2017 at 12:34 PM, Michael Gong <go...@hotmail.com> wrote:
>
> > When approximately will it be released?
> >
> > Sent from my PP•KING™ smartphone
> >
> > On Mar 3, 2017 1:42 PM, Dan Davydov <da...@airbnb.com.INVALID>
> > wrote:
> > Yes it is starting on 1.8.0 which will be released soon, you can look in
> > the documentation/grep for "run_as".
> >
> > On Mar 3, 2017 8:50 AM, "Michael Gong" <go...@hotmail.com> wrote:
> >
> > > Hi,
> > >
> > >
> > > Suppose I have 1 airflow instance running 2 different DAGs, is it
> > possible
> > > to specify the 2 DAGs running under 2 different ids ?
> > >
> > >
> > > Any advises are welcomed.
> > >
> > >
> > > Thanks.
> > >
> > > Michael
> > >
> > >
> > >
> > >
> > >
> >
>
Re: Airflow running different with different user id ?
Posted by Michael Gong <go...@hotmail.com>.
Hi, Dan,
The only doc mentioning "run_as" was in the airflow.operators.HiveOperator.
Is this what you mean "run_as"?
If not, can you provide more information about it? We are very interested to know more .
Thanks.
Michael
Sent from my PP•KING™ smartphone
On Mar 3, 2017 3:57 PM, Michael Gong <go...@hotmail.com> wrote:
Hi, Dan,
Thanks for the encouraging news.
Maybe you could direct me to the documentation about "run_as" so I could see whether it is for my purpose.
I guess it will be passed as an argument for the DAG object , right ?
Thanks again.
Michael
________________________________
From: Dan Davydov <da...@airbnb.com.INVALID>
Sent: Friday, March 3, 2017 8:44 PM
To: dev@airflow.incubator.apache.org
Subject: Re: Airflow running different with different user id ?
Within a couple of weeks.
On Fri, Mar 3, 2017 at 12:34 PM, Michael Gong <go...@hotmail.com> wrote:
> When approximately will it be released?
>
> Sent from my PP•KING™ smartphone
>
> On Mar 3, 2017 1:42 PM, Dan Davydov <da...@airbnb.com.INVALID>
> wrote:
> Yes it is starting on 1.8.0 which will be released soon, you can look in
> the documentation/grep for "run_as".
>
> On Mar 3, 2017 8:50 AM, "Michael Gong" <go...@hotmail.com> wrote:
>
> > Hi,
> >
> >
> > Suppose I have 1 airflow instance running 2 different DAGs, is it
> possible
> > to specify the 2 DAGs running under 2 different ids ?
> >
> >
> > Any advises are welcomed.
> >
> >
> > Thanks.
> >
> > Michael
> >
> >
> >
> >
> >
>
Re: Airflow running different with different user id ?
Posted by Michael Gong <go...@hotmail.com>.
Hi, Dan,
Thanks for the encouraging news.
Maybe you could direct me to the documentation about "run_as" so I could see whether it is for my purpose.
I guess it will be passed as an argument for the DAG object , right ?
Thanks again.
Michael
________________________________
From: Dan Davydov <da...@airbnb.com.INVALID>
Sent: Friday, March 3, 2017 8:44 PM
To: dev@airflow.incubator.apache.org
Subject: Re: Airflow running different with different user id ?
Within a couple of weeks.
On Fri, Mar 3, 2017 at 12:34 PM, Michael Gong <go...@hotmail.com> wrote:
> When approximately will it be released?
>
> Sent from my PP•KING™ smartphone
>
> On Mar 3, 2017 1:42 PM, Dan Davydov <da...@airbnb.com.INVALID>
> wrote:
> Yes it is starting on 1.8.0 which will be released soon, you can look in
> the documentation/grep for "run_as".
>
> On Mar 3, 2017 8:50 AM, "Michael Gong" <go...@hotmail.com> wrote:
>
> > Hi,
> >
> >
> > Suppose I have 1 airflow instance running 2 different DAGs, is it
> possible
> > to specify the 2 DAGs running under 2 different ids ?
> >
> >
> > Any advises are welcomed.
> >
> >
> > Thanks.
> >
> > Michael
> >
> >
> >
> >
> >
>
Re: Airflow running different with different user id ?
Posted by Dan Davydov <da...@airbnb.com.INVALID>.
Within a couple of weeks.
On Fri, Mar 3, 2017 at 12:34 PM, Michael Gong <go...@hotmail.com> wrote:
> When approximately will it be released?
>
> Sent from my PP•KING™ smartphone
>
> On Mar 3, 2017 1:42 PM, Dan Davydov <da...@airbnb.com.INVALID>
> wrote:
> Yes it is starting on 1.8.0 which will be released soon, you can look in
> the documentation/grep for "run_as".
>
> On Mar 3, 2017 8:50 AM, "Michael Gong" <go...@hotmail.com> wrote:
>
> > Hi,
> >
> >
> > Suppose I have 1 airflow instance running 2 different DAGs, is it
> possible
> > to specify the 2 DAGs running under 2 different ids ?
> >
> >
> > Any advises are welcomed.
> >
> >
> > Thanks.
> >
> > Michael
> >
> >
> >
> >
> >
>
Re: Airflow running different with different user id ?
Posted by Michael Gong <go...@hotmail.com>.
When approximately will it be released?
Sent from my PP•KING™ smartphone
On Mar 3, 2017 1:42 PM, Dan Davydov <da...@airbnb.com.INVALID> wrote:
Yes it is starting on 1.8.0 which will be released soon, you can look in
the documentation/grep for "run_as".
On Mar 3, 2017 8:50 AM, "Michael Gong" <go...@hotmail.com> wrote:
> Hi,
>
>
> Suppose I have 1 airflow instance running 2 different DAGs, is it possible
> to specify the 2 DAGs running under 2 different ids ?
>
>
> Any advises are welcomed.
>
>
> Thanks.
>
> Michael
>
>
>
>
>
Re: Airflow running different with different user id ?
Posted by Dan Davydov <da...@airbnb.com.INVALID>.
Yes it is starting on 1.8.0 which will be released soon, you can look in
the documentation/grep for "run_as".
On Mar 3, 2017 8:50 AM, "Michael Gong" <go...@hotmail.com> wrote:
> Hi,
>
>
> Suppose I have 1 airflow instance running 2 different DAGs, is it possible
> to specify the 2 DAGs running under 2 different ids ?
>
>
> Any advises are welcomed.
>
>
> Thanks.
>
> Michael
>
>
>
>
>