You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Bolke de Bruin <bd...@gmail.com> on 2017/01/28 15:07:15 UTC
Airflow 1.8.0 BETA 5
Hi All,
I have made the FIFTH beta of Airflow 1.8.0 available at: https://dist.apache.org/repos/dist/dev/incubator/airflow/ <https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public keys are available at https://dist.apache.org/repos/dist/release/incubator/airflow/ <https://dist.apache.org/repos/dist/release/incubator/airflow/> . It is tagged with a local version “apache.incubating” so it allows upgrading from earlier releases.
Issues fixed:
* Parsing errors not showing up in UI fixing a regression**
* Scheduler would terminate immediately if no dag files present
** As this touches the scheduler logic I though it warranted another beta.
This should be the last beta in my opinion and we can prepare changelog, upgrade notes and release notes for the RC (Feb 2).
Cheers
Bolke
Re: Airflow 1.8.0 BETA 5
Posted by Chris Riccomini <cr...@apache.org>.
Data point: b5 is working fine for me so far.
On Tue, Jan 31, 2017 at 8:25 AM, Alex Van Boxel <al...@vanboxel.be> wrote:
> I'll try to identify the core problem
>
> On Tue, Jan 31, 2017, 16:43 Bolke de Bruin <bd...@gmail.com> wrote:
>
> > Hey Alex
> >
> > Can you provide some info on the scheduler paths thing. I don't have/see
> > that issue. Do you mean cli paths or by cfg? Jira would be nice in any
> case.
> >
> > I don't think the dag processor respects cli parameters.
> >
> > Bolke
> >
> > Sent from my iPhone
> >
> > > On 31 Jan 2017, at 15:10, Alex Van Boxel <al...@vanboxel.be> wrote:
> > >
> > > It's quite hard to share my complete dags. I don't have this locally,
> > but I
> > > have it in my production environment where I use Celery. I rolled back
> to
> > > beta 4 to make it work again.
> > >
> > > Also @bolke the scheduler logs don't respect the log path.
> > >
> > > On Tue, Jan 31, 2017 at 1:02 AM Dan Davydov <dan.davydov@airbnb.com
> > .invalid>
> > > wrote:
> > >
> > >> @Alex
> > >> I'm not able to reproduce locally (assuming the two python files are
> in
> > the
> > >> same folder or is on your PYTHONPATH). I don't see that import error
> > >> anyways.
> > >>
> > >> Just in case, what is your complete DAG definition? Is anyone else
> able
> > to
> > >> repro?
> > >>
> > >>> On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel <al...@vanboxel.be>
> > wrote:
> > >>>
> > >>> Well this means none of my DAG's work anymore:
> > >>>
> > >>> you just can do this anymore:
> > >>>
> > >>> file bqschema.py with
> > >>>
> > >>> def marketing_segment():
> > >>> return [
> > >>> {"name": "user_id", "type": "integer", "mode": "nullable"},
> > >>> {"name": "bucket_date", "type": "timestamp", "mode":
> > "nullable"},
> > >>> {"name": "segment_main", "type": "string", "mode":
> "nullable"},
> > >>> {"name": "segment_sub", "type": "integer", "mode":
> "nullable"},
> > >>>
> > >>>
> > >>> In marketing_segmentation.py:
> > >>>
> > >>>
> > >>> import bqschema
> > >>>
> > >>> Gives an error:
> > >>>
> > >>> Traceback (most recent call last):
> > >>> File
> > >>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+
> > >>> apache.incubating-py2.7.egg/airflow/models.py",
> > >>> line 264, in process_file
> > >>> m = imp.load_source(mod_name, filepath)
> > >>> File "/home/airflow/dags/marketing_segmentation.py", line 17, in
> > >>> <module>
> > >>> import bqschema
> > >>> ImportError: No module named bqschema
> > >>>
> > >>> *I don't think this is incorrect?!*
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov <dan.davydov@airbnb.com
> .
> > >>> invalid>
> > >>> wrote:
> > >>>
> > >>>> The latest commit fixed a regression since 1.7 that files with
> parsing
> > >>>> errors no longer showed up on the UI.
> > >>>>
> > >>>> On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel <al...@vanboxel.be>
> > >>> wrote:
> > >>>>
> > >>>>> Just installed beta 5 on our dev environment it lighted up as a
> > >>> christmas
> > >>>>> tree. I got a a screen full of import errors. I see that the latest
> > >>>> commit
> > >>>>> did something with import errors... is it coorect?!
> > >>>>>
> > >>>>> On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin <bd...@gmail.com>
> > >>>> wrote:
> > >>>>>
> > >>>>>> Hey Boris
> > >>>>>>
> > >>>>>> The scheduler is a bit more aggressive and can use multiple
> > >>> processors,
> > >>>>> so
> > >>>>>> higher CPU usage is actually a good thing.
> > >>>>>>
> > >>>>>> I case it is really out of hand look at the new scheduler options
> > >> and
> > >>>>>> heartbeat options (see PR for updating.md not in the beta yet).
> > >>>>>>
> > >>>>>> Bolke
> > >>>>>>
> > >>>>>> Sent from my iPhone
> > >>>>>>
> > >>>>>>> On 29 Jan 2017, at 15:35, Boris Tyukin <bo...@boristyukin.com>
> > >>>> wrote:
> > >>>>>>>
> > >>>>>>> I am not sure if it is my config or something, but looks like
> > >> after
> > >>>> the
> > >>>>>>> upgrade and start of scheduler, airflow would totally hose CPU.
> > >> The
> > >>>>>> reason
> > >>>>>>> is two new examples that start running right away - latest only
> > >> and
> > >>>>>> latest
> > >>>>>>> with trigger. Once I pause them, CPU goes back to idle. Is this
> > >>>> because
> > >>>>>> now
> > >>>>>>> dags are not paused by default like it was before?
> > >>>>>>>
> > >>>>>>> As I mentioned before, I also had to upgrade mysql to 5.7 - if
> > >>>> someone
> > >>>>>>> needs a step by step instruction, make sure to follow all steps
> > >>>>> precisely
> > >>>>>>> here for in-place upgrade or you will have heck of the time (like
> > >>>> me).
> > >>>>>>>
> > >>>>>> https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#
> > >>>>> upgrade-procedure-inplace
> > >>>>>>>
> > >>>>>>> BTW official Oracle repository for Oracle Linux only has MySql
> > >> 5.6
> > >>> -
> > >>>>> for
> > >>>>>>> 5.7 you have to use MySql community repo.
> > >>>>>>>
> > >>>>>>>> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin <
> > >>> bdbruin@gmail.com
> > >>>>>
> > >>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> Hi All,
> > >>>>>>>>
> > >>>>>>>> I have made the FIFTH beta of Airflow 1.8.0 available at:
> > >>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> > >>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> ,
> > >>> public
> > >>>>>> keys
> > >>>>>>>> are available at https://dist.apache.org/repos/
> > >>>>> dist/release/incubator/
> > >>>>>>>> airflow/ <https://dist.apache.org/repos/dist/release/incubator/
> > >>>>> airflow/
> > >>>>>>>
> > >>>>>>>> . It is tagged with a local version “apache.incubating” so it
> > >>> allows
> > >>>>>>>> upgrading from earlier releases.
> > >>>>>>>>
> > >>>>>>>> Issues fixed:
> > >>>>>>>> * Parsing errors not showing up in UI fixing a regression**
> > >>>>>>>> * Scheduler would terminate immediately if no dag files present
> > >>>>>>>>
> > >>>>>>>> ** As this touches the scheduler logic I though it warranted
> > >>> another
> > >>>>>> beta.
> > >>>>>>>>
> > >>>>>>>> This should be the last beta in my opinion and we can prepare
> > >>>>> changelog,
> > >>>>>>>> upgrade notes and release notes for the RC (Feb 2).
> > >>>>>>>>
> > >>>>>>>> Cheers
> > >>>>>>>> Bolke
> > >>>>>>
> > >>>>> --
> > >>>>> _/
> > >>>>> _/ Alex Van Boxel
> > >>>>>
> > >>>>
> > >>> --
> > >>> _/
> > >>> _/ Alex Van Boxel
> > >>>
> > >>
> > > --
> > > _/
> > > _/ Alex Van Boxel
> >
> --
> _/
> _/ Alex Van Boxel
>
Re: Airflow 1.8.0 BETA 5
Posted by Bolke de Bruin <bd...@gmail.com>.
Rc 1 isn't out yet, differences will be small.
Sent from my iPhone
> On 31 Jan 2017, at 22:00, Chris Riccomini <cr...@apache.org> wrote:
>
> I'm still on b5. Should I upgrade to rc1, or is it still in progress?
>
>> On Tue, Jan 31, 2017 at 12:26 PM, Alex Van Boxel <al...@vanboxel.be> wrote:
>>
>> I identified my root cause: it's was a problem at my site with a faulty
>> rebase. So all green.
>>
>>> On Tue, Jan 31, 2017 at 8:52 PM Alex Van Boxel <al...@vanboxel.be> wrote:
>>>
>>> So bumped to RC1 and this seems fine. I don't get it.
>>>
>>> So it's a go. Sorry for the noise, but better safe then sorry. I also
>>> found that the scheduler logs setting are not equal from the normal
>> logging
>>> so that's also a go.
>>>
>>> You get green light from me.
>>>
>>> On Tue, Jan 31, 2017 at 8:32 PM Bolke de Bruin <bd...@gmail.com>
>> wrote:
>>>
>>> And the scheduler doesn't log anything in beta 4?
>>>
>>> Sent from my iPhone
>>>
>>>> On 31 Jan 2017, at 19:26, Alex Van Boxel <al...@vanboxel.be> wrote:
>>>>
>>>> I see it in the scheduler and in the UI. Currently trying to do a new
>>>> upgrade.
>>>>
>>>>> On Tue, Jan 31, 2017 at 8:00 PM Bolke de Bruin <bd...@gmail.com>
>>> wrote:
>>>>>
>>>>> Please note I will be holding off on the RC, before we understand this
>>>>> issue better.
>>>>>
>>>>> Bolke
>>>>>
>>>>>> On 31 Jan 2017, at 18:06, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>>>
>>>>>> Hey Alex,
>>>>>>
>>>>>> Could it actually be that Airflow is doing the right thing? Earlier
>> it
>>>>> was swallowing the errors. Where do you see the errors? In the
>> scheduler
>>>>> logs, UI, processor logs?
>>>>>>
>>>>>> - Bolke
>>>>>>
>>>>>>> On 31 Jan 2017, at 16:25, Alex Van Boxel <al...@vanboxel.be> wrote:
>>>>>>>
>>>>>>> I'll try to identify the core problem
>>>>>>>
>>>>>>>> On Tue, Jan 31, 2017, 16:43 Bolke de Bruin <bd...@gmail.com>
>>> wrote:
>>>>>>>>
>>>>>>>> Hey Alex
>>>>>>>>
>>>>>>>> Can you provide some info on the scheduler paths thing. I don't
>>>>> have/see
>>>>>>>> that issue. Do you mean cli paths or by cfg? Jira would be nice in
>>> any
>>>>> case.
>>>>>>>>
>>>>>>>> I don't think the dag processor respects cli parameters.
>>>>>>>>
>>>>>>>> Bolke
>>>>>>>>
>>>>>>>> Sent from my iPhone
>>>>>>>>
>>>>>>>>> On 31 Jan 2017, at 15:10, Alex Van Boxel <al...@vanboxel.be>
>> wrote:
>>>>>>>>>
>>>>>>>>> It's quite hard to share my complete dags. I don't have this
>>> locally,
>>>>>>>> but I
>>>>>>>>> have it in my production environment where I use Celery. I rolled
>>>>> back to
>>>>>>>>> beta 4 to make it work again.
>>>>>>>>>
>>>>>>>>> Also @bolke the scheduler logs don't respect the log path.
>>>>>>>>>
>>>>>>>>> On Tue, Jan 31, 2017 at 1:02 AM Dan Davydov <
>> dan.davydov@airbnb.com
>>>>>>>> .invalid>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> @Alex
>>>>>>>>>> I'm not able to reproduce locally (assuming the two python files
>>> are
>>>>> in
>>>>>>>> the
>>>>>>>>>> same folder or is on your PYTHONPATH). I don't see that import
>>> error
>>>>>>>>>> anyways.
>>>>>>>>>>
>>>>>>>>>> Just in case, what is your complete DAG definition? Is anyone
>> else
>>>>> able
>>>>>>>> to
>>>>>>>>>> repro?
>>>>>>>>>>
>>>>>>>>>>> On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel <
>> alex@vanboxel.be
>>>>
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Well this means none of my DAG's work anymore:
>>>>>>>>>>>
>>>>>>>>>>> you just can do this anymore:
>>>>>>>>>>>
>>>>>>>>>>> file bqschema.py with
>>>>>>>>>>>
>>>>>>>>>>> def marketing_segment():
>>>>>>>>>>> return [
>>>>>>>>>>> {"name": "user_id", "type": "integer", "mode": "nullable"},
>>>>>>>>>>> {"name": "bucket_date", "type": "timestamp", "mode":
>>>>>>>> "nullable"},
>>>>>>>>>>> {"name": "segment_main", "type": "string", "mode":
>>> "nullable"},
>>>>>>>>>>> {"name": "segment_sub", "type": "integer", "mode":
>>> "nullable"},
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> In marketing_segmentation.py:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> import bqschema
>>>>>>>>>>>
>>>>>>>>>>> Gives an error:
>>>>>>>>>>>
>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>> File
>>>>>>>>>>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+
>>>>>>>>>>> apache.incubating-py2.7.egg/airflow/models.py",
>>>>>>>>>>> line 264, in process_file
>>>>>>>>>>> m = imp.load_source(mod_name, filepath)
>>>>>>>>>>> File "/home/airflow/dags/marketing_segmentation.py", line 17,
>> in
>>>>>>>>>>> <module>
>>>>>>>>>>> import bqschema
>>>>>>>>>>> ImportError: No module named bqschema
>>>>>>>>>>>
>>>>>>>>>>> *I don't think this is incorrect?!*
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov <
>>>>> dan.davydov@airbnb.com.
>>>>>>>>>>> invalid>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> The latest commit fixed a regression since 1.7 that files with
>>>>> parsing
>>>>>>>>>>>> errors no longer showed up on the UI.
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel <
>>> alex@vanboxel.be>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Just installed beta 5 on our dev environment it lighted up as
>> a
>>>>>>>>>>> christmas
>>>>>>>>>>>>> tree. I got a a screen full of import errors. I see that the
>>>>> latest
>>>>>>>>>>>> commit
>>>>>>>>>>>>> did something with import errors... is it coorect?!
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin <
>>> bdbruin@gmail.com
>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hey Boris
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The scheduler is a bit more aggressive and can use multiple
>>>>>>>>>>> processors,
>>>>>>>>>>>>> so
>>>>>>>>>>>>>> higher CPU usage is actually a good thing.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I case it is really out of hand look at the new scheduler
>>> options
>>>>>>>>>> and
>>>>>>>>>>>>>> heartbeat options (see PR for updating.md not in the beta
>>> yet).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sent from my iPhone
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 29 Jan 2017, at 15:35, Boris Tyukin <
>> boris@boristyukin.com
>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I am not sure if it is my config or something, but looks
>> like
>>>>>>>>>> after
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> upgrade and start of scheduler, airflow would totally hose
>>> CPU.
>>>>>>>>>> The
>>>>>>>>>>>>>> reason
>>>>>>>>>>>>>>> is two new examples that start running right away - latest
>>> only
>>>>>>>>>> and
>>>>>>>>>>>>>> latest
>>>>>>>>>>>>>>> with trigger. Once I pause them, CPU goes back to idle. Is
>>> this
>>>>>>>>>>>> because
>>>>>>>>>>>>>> now
>>>>>>>>>>>>>>> dags are not paused by default like it was before?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> As I mentioned before, I also had to upgrade mysql to 5.7 -
>> if
>>>>>>>>>>>> someone
>>>>>>>>>>>>>>> needs a step by step instruction, make sure to follow all
>>> steps
>>>>>>>>>>>>> precisely
>>>>>>>>>>>>>>> here for in-place upgrade or you will have heck of the time
>>>>> (like
>>>>>>>>>>>> me).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#
>>>>>>>>>>>>> upgrade-procedure-inplace
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> BTW official Oracle repository for Oracle Linux only has
>> MySql
>>>>>>>>>> 5.6
>>>>>>>>>>> -
>>>>>>>>>>>>> for
>>>>>>>>>>>>>>> 5.7 you have to use MySql community repo.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin <
>>>>>>>>>>> bdbruin@gmail.com
>>>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I have made the FIFTH beta of Airflow 1.8.0 available at:
>>>>>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/
>> <
>>>>>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/>
>> ,
>>>>>>>>>>> public
>>>>>>>>>>>>>> keys
>>>>>>>>>>>>>>>> are available at https://dist.apache.org/repos/
>>>>>>>>>>>>> dist/release/incubator/
>>>>>>>>>>>>>>>> airflow/ <
>>>>> https://dist.apache.org/repos/dist/release/incubator/
>>>>>>>>>>>>> airflow/
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> . It is tagged with a local version “apache.incubating” so
>> it
>>>>>>>>>>> allows
>>>>>>>>>>>>>>>> upgrading from earlier releases.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Issues fixed:
>>>>>>>>>>>>>>>> * Parsing errors not showing up in UI fixing a regression**
>>>>>>>>>>>>>>>> * Scheduler would terminate immediately if no dag files
>>> present
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ** As this touches the scheduler logic I though it
>> warranted
>>>>>>>>>>> another
>>>>>>>>>>>>>> beta.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This should be the last beta in my opinion and we can
>> prepare
>>>>>>>>>>>>> changelog,
>>>>>>>>>>>>>>>> upgrade notes and release notes for the RC (Feb 2).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> _/
>>>>>>>>>>>>> _/ Alex Van Boxel
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> _/
>>>>>>>>>>> _/ Alex Van Boxel
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> _/
>>>>>>>>> _/ Alex Van Boxel
>>>>>>>>
>>>>>>> --
>>>>>>> _/
>>>>>>> _/ Alex Van Boxel
>>>>>>
>>>>>
>>>>> --
>>>> _/
>>>> _/ Alex Van Boxel
>>>
>>> --
>>> _/
>>> _/ Alex Van Boxel
>>>
>> --
>> _/
>> _/ Alex Van Boxel
>>
Re: Airflow 1.8.0 BETA 5
Posted by Chris Riccomini <cr...@apache.org>.
I'm still on b5. Should I upgrade to rc1, or is it still in progress?
On Tue, Jan 31, 2017 at 12:26 PM, Alex Van Boxel <al...@vanboxel.be> wrote:
> I identified my root cause: it's was a problem at my site with a faulty
> rebase. So all green.
>
> On Tue, Jan 31, 2017 at 8:52 PM Alex Van Boxel <al...@vanboxel.be> wrote:
>
> > So bumped to RC1 and this seems fine. I don't get it.
> >
> > So it's a go. Sorry for the noise, but better safe then sorry. I also
> > found that the scheduler logs setting are not equal from the normal
> logging
> > so that's also a go.
> >
> > You get green light from me.
> >
> > On Tue, Jan 31, 2017 at 8:32 PM Bolke de Bruin <bd...@gmail.com>
> wrote:
> >
> > And the scheduler doesn't log anything in beta 4?
> >
> > Sent from my iPhone
> >
> > > On 31 Jan 2017, at 19:26, Alex Van Boxel <al...@vanboxel.be> wrote:
> > >
> > > I see it in the scheduler and in the UI. Currently trying to do a new
> > > upgrade.
> > >
> > >> On Tue, Jan 31, 2017 at 8:00 PM Bolke de Bruin <bd...@gmail.com>
> > wrote:
> > >>
> > >> Please note I will be holding off on the RC, before we understand this
> > >> issue better.
> > >>
> > >> Bolke
> > >>
> > >>> On 31 Jan 2017, at 18:06, Bolke de Bruin <bd...@gmail.com> wrote:
> > >>>
> > >>> Hey Alex,
> > >>>
> > >>> Could it actually be that Airflow is doing the right thing? Earlier
> it
> > >> was swallowing the errors. Where do you see the errors? In the
> scheduler
> > >> logs, UI, processor logs?
> > >>>
> > >>> - Bolke
> > >>>
> > >>>> On 31 Jan 2017, at 16:25, Alex Van Boxel <al...@vanboxel.be> wrote:
> > >>>>
> > >>>> I'll try to identify the core problem
> > >>>>
> > >>>>> On Tue, Jan 31, 2017, 16:43 Bolke de Bruin <bd...@gmail.com>
> > wrote:
> > >>>>>
> > >>>>> Hey Alex
> > >>>>>
> > >>>>> Can you provide some info on the scheduler paths thing. I don't
> > >> have/see
> > >>>>> that issue. Do you mean cli paths or by cfg? Jira would be nice in
> > any
> > >> case.
> > >>>>>
> > >>>>> I don't think the dag processor respects cli parameters.
> > >>>>>
> > >>>>> Bolke
> > >>>>>
> > >>>>> Sent from my iPhone
> > >>>>>
> > >>>>>> On 31 Jan 2017, at 15:10, Alex Van Boxel <al...@vanboxel.be>
> wrote:
> > >>>>>>
> > >>>>>> It's quite hard to share my complete dags. I don't have this
> > locally,
> > >>>>> but I
> > >>>>>> have it in my production environment where I use Celery. I rolled
> > >> back to
> > >>>>>> beta 4 to make it work again.
> > >>>>>>
> > >>>>>> Also @bolke the scheduler logs don't respect the log path.
> > >>>>>>
> > >>>>>> On Tue, Jan 31, 2017 at 1:02 AM Dan Davydov <
> dan.davydov@airbnb.com
> > >>>>> .invalid>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> @Alex
> > >>>>>>> I'm not able to reproduce locally (assuming the two python files
> > are
> > >> in
> > >>>>> the
> > >>>>>>> same folder or is on your PYTHONPATH). I don't see that import
> > error
> > >>>>>>> anyways.
> > >>>>>>>
> > >>>>>>> Just in case, what is your complete DAG definition? Is anyone
> else
> > >> able
> > >>>>> to
> > >>>>>>> repro?
> > >>>>>>>
> > >>>>>>>> On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel <
> alex@vanboxel.be
> > >
> > >>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> Well this means none of my DAG's work anymore:
> > >>>>>>>>
> > >>>>>>>> you just can do this anymore:
> > >>>>>>>>
> > >>>>>>>> file bqschema.py with
> > >>>>>>>>
> > >>>>>>>> def marketing_segment():
> > >>>>>>>> return [
> > >>>>>>>> {"name": "user_id", "type": "integer", "mode": "nullable"},
> > >>>>>>>> {"name": "bucket_date", "type": "timestamp", "mode":
> > >>>>> "nullable"},
> > >>>>>>>> {"name": "segment_main", "type": "string", "mode":
> > "nullable"},
> > >>>>>>>> {"name": "segment_sub", "type": "integer", "mode":
> > "nullable"},
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> In marketing_segmentation.py:
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> import bqschema
> > >>>>>>>>
> > >>>>>>>> Gives an error:
> > >>>>>>>>
> > >>>>>>>> Traceback (most recent call last):
> > >>>>>>>> File
> > >>>>>>>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+
> > >>>>>>>> apache.incubating-py2.7.egg/airflow/models.py",
> > >>>>>>>> line 264, in process_file
> > >>>>>>>> m = imp.load_source(mod_name, filepath)
> > >>>>>>>> File "/home/airflow/dags/marketing_segmentation.py", line 17,
> in
> > >>>>>>>> <module>
> > >>>>>>>> import bqschema
> > >>>>>>>> ImportError: No module named bqschema
> > >>>>>>>>
> > >>>>>>>> *I don't think this is incorrect?!*
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov <
> > >> dan.davydov@airbnb.com.
> > >>>>>>>> invalid>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> The latest commit fixed a regression since 1.7 that files with
> > >> parsing
> > >>>>>>>>> errors no longer showed up on the UI.
> > >>>>>>>>>
> > >>>>>>>>> On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel <
> > alex@vanboxel.be>
> > >>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Just installed beta 5 on our dev environment it lighted up as
> a
> > >>>>>>>> christmas
> > >>>>>>>>>> tree. I got a a screen full of import errors. I see that the
> > >> latest
> > >>>>>>>>> commit
> > >>>>>>>>>> did something with import errors... is it coorect?!
> > >>>>>>>>>>
> > >>>>>>>>>> On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin <
> > bdbruin@gmail.com
> > >>>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hey Boris
> > >>>>>>>>>>>
> > >>>>>>>>>>> The scheduler is a bit more aggressive and can use multiple
> > >>>>>>>> processors,
> > >>>>>>>>>> so
> > >>>>>>>>>>> higher CPU usage is actually a good thing.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I case it is really out of hand look at the new scheduler
> > options
> > >>>>>>> and
> > >>>>>>>>>>> heartbeat options (see PR for updating.md not in the beta
> > yet).
> > >>>>>>>>>>>
> > >>>>>>>>>>> Bolke
> > >>>>>>>>>>>
> > >>>>>>>>>>> Sent from my iPhone
> > >>>>>>>>>>>
> > >>>>>>>>>>>> On 29 Jan 2017, at 15:35, Boris Tyukin <
> boris@boristyukin.com
> > >
> > >>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I am not sure if it is my config or something, but looks
> like
> > >>>>>>> after
> > >>>>>>>>> the
> > >>>>>>>>>>>> upgrade and start of scheduler, airflow would totally hose
> > CPU.
> > >>>>>>> The
> > >>>>>>>>>>> reason
> > >>>>>>>>>>>> is two new examples that start running right away - latest
> > only
> > >>>>>>> and
> > >>>>>>>>>>> latest
> > >>>>>>>>>>>> with trigger. Once I pause them, CPU goes back to idle. Is
> > this
> > >>>>>>>>> because
> > >>>>>>>>>>> now
> > >>>>>>>>>>>> dags are not paused by default like it was before?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> As I mentioned before, I also had to upgrade mysql to 5.7 -
> if
> > >>>>>>>>> someone
> > >>>>>>>>>>>> needs a step by step instruction, make sure to follow all
> > steps
> > >>>>>>>>>> precisely
> > >>>>>>>>>>>> here for in-place upgrade or you will have heck of the time
> > >> (like
> > >>>>>>>>> me).
> > >>>>>>>>>>>>
> > >>>>>>>>>>> https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#
> > >>>>>>>>>> upgrade-procedure-inplace
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> BTW official Oracle repository for Oracle Linux only has
> MySql
> > >>>>>>> 5.6
> > >>>>>>>> -
> > >>>>>>>>>> for
> > >>>>>>>>>>>> 5.7 you have to use MySql community repo.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin <
> > >>>>>>>> bdbruin@gmail.com
> > >>>>>>>>>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi All,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I have made the FIFTH beta of Airflow 1.8.0 available at:
> > >>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/
> <
> > >>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/>
> ,
> > >>>>>>>> public
> > >>>>>>>>>>> keys
> > >>>>>>>>>>>>> are available at https://dist.apache.org/repos/
> > >>>>>>>>>> dist/release/incubator/
> > >>>>>>>>>>>>> airflow/ <
> > >> https://dist.apache.org/repos/dist/release/incubator/
> > >>>>>>>>>> airflow/
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> . It is tagged with a local version “apache.incubating” so
> it
> > >>>>>>>> allows
> > >>>>>>>>>>>>> upgrading from earlier releases.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Issues fixed:
> > >>>>>>>>>>>>> * Parsing errors not showing up in UI fixing a regression**
> > >>>>>>>>>>>>> * Scheduler would terminate immediately if no dag files
> > present
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> ** As this touches the scheduler logic I though it
> warranted
> > >>>>>>>> another
> > >>>>>>>>>>> beta.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> This should be the last beta in my opinion and we can
> prepare
> > >>>>>>>>>> changelog,
> > >>>>>>>>>>>>> upgrade notes and release notes for the RC (Feb 2).
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Cheers
> > >>>>>>>>>>>>> Bolke
> > >>>>>>>>>>>
> > >>>>>>>>>> --
> > >>>>>>>>>> _/
> > >>>>>>>>>> _/ Alex Van Boxel
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>> --
> > >>>>>>>> _/
> > >>>>>>>> _/ Alex Van Boxel
> > >>>>>>>>
> > >>>>>>>
> > >>>>>> --
> > >>>>>> _/
> > >>>>>> _/ Alex Van Boxel
> > >>>>>
> > >>>> --
> > >>>> _/
> > >>>> _/ Alex Van Boxel
> > >>>
> > >>
> > >> --
> > > _/
> > > _/ Alex Van Boxel
> >
> > --
> > _/
> > _/ Alex Van Boxel
> >
> --
> _/
> _/ Alex Van Boxel
>
Re: Airflow 1.8.0 BETA 5
Posted by Alex Van Boxel <al...@vanboxel.be>.
I identified my root cause: it's was a problem at my site with a faulty
rebase. So all green.
On Tue, Jan 31, 2017 at 8:52 PM Alex Van Boxel <al...@vanboxel.be> wrote:
> So bumped to RC1 and this seems fine. I don't get it.
>
> So it's a go. Sorry for the noise, but better safe then sorry. I also
> found that the scheduler logs setting are not equal from the normal logging
> so that's also a go.
>
> You get green light from me.
>
> On Tue, Jan 31, 2017 at 8:32 PM Bolke de Bruin <bd...@gmail.com> wrote:
>
> And the scheduler doesn't log anything in beta 4?
>
> Sent from my iPhone
>
> > On 31 Jan 2017, at 19:26, Alex Van Boxel <al...@vanboxel.be> wrote:
> >
> > I see it in the scheduler and in the UI. Currently trying to do a new
> > upgrade.
> >
> >> On Tue, Jan 31, 2017 at 8:00 PM Bolke de Bruin <bd...@gmail.com>
> wrote:
> >>
> >> Please note I will be holding off on the RC, before we understand this
> >> issue better.
> >>
> >> Bolke
> >>
> >>> On 31 Jan 2017, at 18:06, Bolke de Bruin <bd...@gmail.com> wrote:
> >>>
> >>> Hey Alex,
> >>>
> >>> Could it actually be that Airflow is doing the right thing? Earlier it
> >> was swallowing the errors. Where do you see the errors? In the scheduler
> >> logs, UI, processor logs?
> >>>
> >>> - Bolke
> >>>
> >>>> On 31 Jan 2017, at 16:25, Alex Van Boxel <al...@vanboxel.be> wrote:
> >>>>
> >>>> I'll try to identify the core problem
> >>>>
> >>>>> On Tue, Jan 31, 2017, 16:43 Bolke de Bruin <bd...@gmail.com>
> wrote:
> >>>>>
> >>>>> Hey Alex
> >>>>>
> >>>>> Can you provide some info on the scheduler paths thing. I don't
> >> have/see
> >>>>> that issue. Do you mean cli paths or by cfg? Jira would be nice in
> any
> >> case.
> >>>>>
> >>>>> I don't think the dag processor respects cli parameters.
> >>>>>
> >>>>> Bolke
> >>>>>
> >>>>> Sent from my iPhone
> >>>>>
> >>>>>> On 31 Jan 2017, at 15:10, Alex Van Boxel <al...@vanboxel.be> wrote:
> >>>>>>
> >>>>>> It's quite hard to share my complete dags. I don't have this
> locally,
> >>>>> but I
> >>>>>> have it in my production environment where I use Celery. I rolled
> >> back to
> >>>>>> beta 4 to make it work again.
> >>>>>>
> >>>>>> Also @bolke the scheduler logs don't respect the log path.
> >>>>>>
> >>>>>> On Tue, Jan 31, 2017 at 1:02 AM Dan Davydov <dan.davydov@airbnb.com
> >>>>> .invalid>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> @Alex
> >>>>>>> I'm not able to reproduce locally (assuming the two python files
> are
> >> in
> >>>>> the
> >>>>>>> same folder or is on your PYTHONPATH). I don't see that import
> error
> >>>>>>> anyways.
> >>>>>>>
> >>>>>>> Just in case, what is your complete DAG definition? Is anyone else
> >> able
> >>>>> to
> >>>>>>> repro?
> >>>>>>>
> >>>>>>>> On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel <alex@vanboxel.be
> >
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>> Well this means none of my DAG's work anymore:
> >>>>>>>>
> >>>>>>>> you just can do this anymore:
> >>>>>>>>
> >>>>>>>> file bqschema.py with
> >>>>>>>>
> >>>>>>>> def marketing_segment():
> >>>>>>>> return [
> >>>>>>>> {"name": "user_id", "type": "integer", "mode": "nullable"},
> >>>>>>>> {"name": "bucket_date", "type": "timestamp", "mode":
> >>>>> "nullable"},
> >>>>>>>> {"name": "segment_main", "type": "string", "mode":
> "nullable"},
> >>>>>>>> {"name": "segment_sub", "type": "integer", "mode":
> "nullable"},
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> In marketing_segmentation.py:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> import bqschema
> >>>>>>>>
> >>>>>>>> Gives an error:
> >>>>>>>>
> >>>>>>>> Traceback (most recent call last):
> >>>>>>>> File
> >>>>>>>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+
> >>>>>>>> apache.incubating-py2.7.egg/airflow/models.py",
> >>>>>>>> line 264, in process_file
> >>>>>>>> m = imp.load_source(mod_name, filepath)
> >>>>>>>> File "/home/airflow/dags/marketing_segmentation.py", line 17, in
> >>>>>>>> <module>
> >>>>>>>> import bqschema
> >>>>>>>> ImportError: No module named bqschema
> >>>>>>>>
> >>>>>>>> *I don't think this is incorrect?!*
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov <
> >> dan.davydov@airbnb.com.
> >>>>>>>> invalid>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> The latest commit fixed a regression since 1.7 that files with
> >> parsing
> >>>>>>>>> errors no longer showed up on the UI.
> >>>>>>>>>
> >>>>>>>>> On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel <
> alex@vanboxel.be>
> >>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Just installed beta 5 on our dev environment it lighted up as a
> >>>>>>>> christmas
> >>>>>>>>>> tree. I got a a screen full of import errors. I see that the
> >> latest
> >>>>>>>>> commit
> >>>>>>>>>> did something with import errors... is it coorect?!
> >>>>>>>>>>
> >>>>>>>>>> On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin <
> bdbruin@gmail.com
> >>>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hey Boris
> >>>>>>>>>>>
> >>>>>>>>>>> The scheduler is a bit more aggressive and can use multiple
> >>>>>>>> processors,
> >>>>>>>>>> so
> >>>>>>>>>>> higher CPU usage is actually a good thing.
> >>>>>>>>>>>
> >>>>>>>>>>> I case it is really out of hand look at the new scheduler
> options
> >>>>>>> and
> >>>>>>>>>>> heartbeat options (see PR for updating.md not in the beta
> yet).
> >>>>>>>>>>>
> >>>>>>>>>>> Bolke
> >>>>>>>>>>>
> >>>>>>>>>>> Sent from my iPhone
> >>>>>>>>>>>
> >>>>>>>>>>>> On 29 Jan 2017, at 15:35, Boris Tyukin <boris@boristyukin.com
> >
> >>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> I am not sure if it is my config or something, but looks like
> >>>>>>> after
> >>>>>>>>> the
> >>>>>>>>>>>> upgrade and start of scheduler, airflow would totally hose
> CPU.
> >>>>>>> The
> >>>>>>>>>>> reason
> >>>>>>>>>>>> is two new examples that start running right away - latest
> only
> >>>>>>> and
> >>>>>>>>>>> latest
> >>>>>>>>>>>> with trigger. Once I pause them, CPU goes back to idle. Is
> this
> >>>>>>>>> because
> >>>>>>>>>>> now
> >>>>>>>>>>>> dags are not paused by default like it was before?
> >>>>>>>>>>>>
> >>>>>>>>>>>> As I mentioned before, I also had to upgrade mysql to 5.7 - if
> >>>>>>>>> someone
> >>>>>>>>>>>> needs a step by step instruction, make sure to follow all
> steps
> >>>>>>>>>> precisely
> >>>>>>>>>>>> here for in-place upgrade or you will have heck of the time
> >> (like
> >>>>>>>>> me).
> >>>>>>>>>>>>
> >>>>>>>>>>> https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#
> >>>>>>>>>> upgrade-procedure-inplace
> >>>>>>>>>>>>
> >>>>>>>>>>>> BTW official Oracle repository for Oracle Linux only has MySql
> >>>>>>> 5.6
> >>>>>>>> -
> >>>>>>>>>> for
> >>>>>>>>>>>> 5.7 you have to use MySql community repo.
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin <
> >>>>>>>> bdbruin@gmail.com
> >>>>>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi All,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I have made the FIFTH beta of Airflow 1.8.0 available at:
> >>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> >>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> ,
> >>>>>>>> public
> >>>>>>>>>>> keys
> >>>>>>>>>>>>> are available at https://dist.apache.org/repos/
> >>>>>>>>>> dist/release/incubator/
> >>>>>>>>>>>>> airflow/ <
> >> https://dist.apache.org/repos/dist/release/incubator/
> >>>>>>>>>> airflow/
> >>>>>>>>>>>>
> >>>>>>>>>>>>> . It is tagged with a local version “apache.incubating” so it
> >>>>>>>> allows
> >>>>>>>>>>>>> upgrading from earlier releases.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Issues fixed:
> >>>>>>>>>>>>> * Parsing errors not showing up in UI fixing a regression**
> >>>>>>>>>>>>> * Scheduler would terminate immediately if no dag files
> present
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> ** As this touches the scheduler logic I though it warranted
> >>>>>>>> another
> >>>>>>>>>>> beta.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> This should be the last beta in my opinion and we can prepare
> >>>>>>>>>> changelog,
> >>>>>>>>>>>>> upgrade notes and release notes for the RC (Feb 2).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Cheers
> >>>>>>>>>>>>> Bolke
> >>>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> _/
> >>>>>>>>>> _/ Alex Van Boxel
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>> --
> >>>>>>>> _/
> >>>>>>>> _/ Alex Van Boxel
> >>>>>>>>
> >>>>>>>
> >>>>>> --
> >>>>>> _/
> >>>>>> _/ Alex Van Boxel
> >>>>>
> >>>> --
> >>>> _/
> >>>> _/ Alex Van Boxel
> >>>
> >>
> >> --
> > _/
> > _/ Alex Van Boxel
>
> --
> _/
> _/ Alex Van Boxel
>
--
_/
_/ Alex Van Boxel
Re: Airflow 1.8.0 BETA 5
Posted by Alex Van Boxel <al...@vanboxel.be>.
So bumped to RC1 and this seems fine. I don't get it.
So it's a go. Sorry for the noise, but better safe then sorry. I also found
that the scheduler logs setting are not equal from the normal logging so
that's also a go.
You get green light from me.
On Tue, Jan 31, 2017 at 8:32 PM Bolke de Bruin <bd...@gmail.com> wrote:
> And the scheduler doesn't log anything in beta 4?
>
> Sent from my iPhone
>
> > On 31 Jan 2017, at 19:26, Alex Van Boxel <al...@vanboxel.be> wrote:
> >
> > I see it in the scheduler and in the UI. Currently trying to do a new
> > upgrade.
> >
> >> On Tue, Jan 31, 2017 at 8:00 PM Bolke de Bruin <bd...@gmail.com>
> wrote:
> >>
> >> Please note I will be holding off on the RC, before we understand this
> >> issue better.
> >>
> >> Bolke
> >>
> >>> On 31 Jan 2017, at 18:06, Bolke de Bruin <bd...@gmail.com> wrote:
> >>>
> >>> Hey Alex,
> >>>
> >>> Could it actually be that Airflow is doing the right thing? Earlier it
> >> was swallowing the errors. Where do you see the errors? In the scheduler
> >> logs, UI, processor logs?
> >>>
> >>> - Bolke
> >>>
> >>>> On 31 Jan 2017, at 16:25, Alex Van Boxel <al...@vanboxel.be> wrote:
> >>>>
> >>>> I'll try to identify the core problem
> >>>>
> >>>>> On Tue, Jan 31, 2017, 16:43 Bolke de Bruin <bd...@gmail.com>
> wrote:
> >>>>>
> >>>>> Hey Alex
> >>>>>
> >>>>> Can you provide some info on the scheduler paths thing. I don't
> >> have/see
> >>>>> that issue. Do you mean cli paths or by cfg? Jira would be nice in
> any
> >> case.
> >>>>>
> >>>>> I don't think the dag processor respects cli parameters.
> >>>>>
> >>>>> Bolke
> >>>>>
> >>>>> Sent from my iPhone
> >>>>>
> >>>>>> On 31 Jan 2017, at 15:10, Alex Van Boxel <al...@vanboxel.be> wrote:
> >>>>>>
> >>>>>> It's quite hard to share my complete dags. I don't have this
> locally,
> >>>>> but I
> >>>>>> have it in my production environment where I use Celery. I rolled
> >> back to
> >>>>>> beta 4 to make it work again.
> >>>>>>
> >>>>>> Also @bolke the scheduler logs don't respect the log path.
> >>>>>>
> >>>>>> On Tue, Jan 31, 2017 at 1:02 AM Dan Davydov <dan.davydov@airbnb.com
> >>>>> .invalid>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> @Alex
> >>>>>>> I'm not able to reproduce locally (assuming the two python files
> are
> >> in
> >>>>> the
> >>>>>>> same folder or is on your PYTHONPATH). I don't see that import
> error
> >>>>>>> anyways.
> >>>>>>>
> >>>>>>> Just in case, what is your complete DAG definition? Is anyone else
> >> able
> >>>>> to
> >>>>>>> repro?
> >>>>>>>
> >>>>>>>> On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel <alex@vanboxel.be
> >
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>> Well this means none of my DAG's work anymore:
> >>>>>>>>
> >>>>>>>> you just can do this anymore:
> >>>>>>>>
> >>>>>>>> file bqschema.py with
> >>>>>>>>
> >>>>>>>> def marketing_segment():
> >>>>>>>> return [
> >>>>>>>> {"name": "user_id", "type": "integer", "mode": "nullable"},
> >>>>>>>> {"name": "bucket_date", "type": "timestamp", "mode":
> >>>>> "nullable"},
> >>>>>>>> {"name": "segment_main", "type": "string", "mode":
> "nullable"},
> >>>>>>>> {"name": "segment_sub", "type": "integer", "mode":
> "nullable"},
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> In marketing_segmentation.py:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> import bqschema
> >>>>>>>>
> >>>>>>>> Gives an error:
> >>>>>>>>
> >>>>>>>> Traceback (most recent call last):
> >>>>>>>> File
> >>>>>>>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+
> >>>>>>>> apache.incubating-py2.7.egg/airflow/models.py",
> >>>>>>>> line 264, in process_file
> >>>>>>>> m = imp.load_source(mod_name, filepath)
> >>>>>>>> File "/home/airflow/dags/marketing_segmentation.py", line 17, in
> >>>>>>>> <module>
> >>>>>>>> import bqschema
> >>>>>>>> ImportError: No module named bqschema
> >>>>>>>>
> >>>>>>>> *I don't think this is incorrect?!*
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov <
> >> dan.davydov@airbnb.com.
> >>>>>>>> invalid>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> The latest commit fixed a regression since 1.7 that files with
> >> parsing
> >>>>>>>>> errors no longer showed up on the UI.
> >>>>>>>>>
> >>>>>>>>> On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel <
> alex@vanboxel.be>
> >>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Just installed beta 5 on our dev environment it lighted up as a
> >>>>>>>> christmas
> >>>>>>>>>> tree. I got a a screen full of import errors. I see that the
> >> latest
> >>>>>>>>> commit
> >>>>>>>>>> did something with import errors... is it coorect?!
> >>>>>>>>>>
> >>>>>>>>>> On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin <
> bdbruin@gmail.com
> >>>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hey Boris
> >>>>>>>>>>>
> >>>>>>>>>>> The scheduler is a bit more aggressive and can use multiple
> >>>>>>>> processors,
> >>>>>>>>>> so
> >>>>>>>>>>> higher CPU usage is actually a good thing.
> >>>>>>>>>>>
> >>>>>>>>>>> I case it is really out of hand look at the new scheduler
> options
> >>>>>>> and
> >>>>>>>>>>> heartbeat options (see PR for updating.md not in the beta
> yet).
> >>>>>>>>>>>
> >>>>>>>>>>> Bolke
> >>>>>>>>>>>
> >>>>>>>>>>> Sent from my iPhone
> >>>>>>>>>>>
> >>>>>>>>>>>> On 29 Jan 2017, at 15:35, Boris Tyukin <boris@boristyukin.com
> >
> >>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> I am not sure if it is my config or something, but looks like
> >>>>>>> after
> >>>>>>>>> the
> >>>>>>>>>>>> upgrade and start of scheduler, airflow would totally hose
> CPU.
> >>>>>>> The
> >>>>>>>>>>> reason
> >>>>>>>>>>>> is two new examples that start running right away - latest
> only
> >>>>>>> and
> >>>>>>>>>>> latest
> >>>>>>>>>>>> with trigger. Once I pause them, CPU goes back to idle. Is
> this
> >>>>>>>>> because
> >>>>>>>>>>> now
> >>>>>>>>>>>> dags are not paused by default like it was before?
> >>>>>>>>>>>>
> >>>>>>>>>>>> As I mentioned before, I also had to upgrade mysql to 5.7 - if
> >>>>>>>>> someone
> >>>>>>>>>>>> needs a step by step instruction, make sure to follow all
> steps
> >>>>>>>>>> precisely
> >>>>>>>>>>>> here for in-place upgrade or you will have heck of the time
> >> (like
> >>>>>>>>> me).
> >>>>>>>>>>>>
> >>>>>>>>>>> https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#
> >>>>>>>>>> upgrade-procedure-inplace
> >>>>>>>>>>>>
> >>>>>>>>>>>> BTW official Oracle repository for Oracle Linux only has MySql
> >>>>>>> 5.6
> >>>>>>>> -
> >>>>>>>>>> for
> >>>>>>>>>>>> 5.7 you have to use MySql community repo.
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin <
> >>>>>>>> bdbruin@gmail.com
> >>>>>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi All,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I have made the FIFTH beta of Airflow 1.8.0 available at:
> >>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> >>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> ,
> >>>>>>>> public
> >>>>>>>>>>> keys
> >>>>>>>>>>>>> are available at https://dist.apache.org/repos/
> >>>>>>>>>> dist/release/incubator/
> >>>>>>>>>>>>> airflow/ <
> >> https://dist.apache.org/repos/dist/release/incubator/
> >>>>>>>>>> airflow/
> >>>>>>>>>>>>
> >>>>>>>>>>>>> . It is tagged with a local version “apache.incubating” so it
> >>>>>>>> allows
> >>>>>>>>>>>>> upgrading from earlier releases.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Issues fixed:
> >>>>>>>>>>>>> * Parsing errors not showing up in UI fixing a regression**
> >>>>>>>>>>>>> * Scheduler would terminate immediately if no dag files
> present
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> ** As this touches the scheduler logic I though it warranted
> >>>>>>>> another
> >>>>>>>>>>> beta.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> This should be the last beta in my opinion and we can prepare
> >>>>>>>>>> changelog,
> >>>>>>>>>>>>> upgrade notes and release notes for the RC (Feb 2).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Cheers
> >>>>>>>>>>>>> Bolke
> >>>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> _/
> >>>>>>>>>> _/ Alex Van Boxel
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>> --
> >>>>>>>> _/
> >>>>>>>> _/ Alex Van Boxel
> >>>>>>>>
> >>>>>>>
> >>>>>> --
> >>>>>> _/
> >>>>>> _/ Alex Van Boxel
> >>>>>
> >>>> --
> >>>> _/
> >>>> _/ Alex Van Boxel
> >>>
> >>
> >> --
> > _/
> > _/ Alex Van Boxel
>
--
_/
_/ Alex Van Boxel
Re: Airflow 1.8.0 BETA 5
Posted by Bolke de Bruin <bd...@gmail.com>.
And the scheduler doesn't log anything in beta 4?
Sent from my iPhone
> On 31 Jan 2017, at 19:26, Alex Van Boxel <al...@vanboxel.be> wrote:
>
> I see it in the scheduler and in the UI. Currently trying to do a new
> upgrade.
>
>> On Tue, Jan 31, 2017 at 8:00 PM Bolke de Bruin <bd...@gmail.com> wrote:
>>
>> Please note I will be holding off on the RC, before we understand this
>> issue better.
>>
>> Bolke
>>
>>> On 31 Jan 2017, at 18:06, Bolke de Bruin <bd...@gmail.com> wrote:
>>>
>>> Hey Alex,
>>>
>>> Could it actually be that Airflow is doing the right thing? Earlier it
>> was swallowing the errors. Where do you see the errors? In the scheduler
>> logs, UI, processor logs?
>>>
>>> - Bolke
>>>
>>>> On 31 Jan 2017, at 16:25, Alex Van Boxel <al...@vanboxel.be> wrote:
>>>>
>>>> I'll try to identify the core problem
>>>>
>>>>> On Tue, Jan 31, 2017, 16:43 Bolke de Bruin <bd...@gmail.com> wrote:
>>>>>
>>>>> Hey Alex
>>>>>
>>>>> Can you provide some info on the scheduler paths thing. I don't
>> have/see
>>>>> that issue. Do you mean cli paths or by cfg? Jira would be nice in any
>> case.
>>>>>
>>>>> I don't think the dag processor respects cli parameters.
>>>>>
>>>>> Bolke
>>>>>
>>>>> Sent from my iPhone
>>>>>
>>>>>> On 31 Jan 2017, at 15:10, Alex Van Boxel <al...@vanboxel.be> wrote:
>>>>>>
>>>>>> It's quite hard to share my complete dags. I don't have this locally,
>>>>> but I
>>>>>> have it in my production environment where I use Celery. I rolled
>> back to
>>>>>> beta 4 to make it work again.
>>>>>>
>>>>>> Also @bolke the scheduler logs don't respect the log path.
>>>>>>
>>>>>> On Tue, Jan 31, 2017 at 1:02 AM Dan Davydov <dan.davydov@airbnb.com
>>>>> .invalid>
>>>>>> wrote:
>>>>>>
>>>>>>> @Alex
>>>>>>> I'm not able to reproduce locally (assuming the two python files are
>> in
>>>>> the
>>>>>>> same folder or is on your PYTHONPATH). I don't see that import error
>>>>>>> anyways.
>>>>>>>
>>>>>>> Just in case, what is your complete DAG definition? Is anyone else
>> able
>>>>> to
>>>>>>> repro?
>>>>>>>
>>>>>>>> On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel <al...@vanboxel.be>
>>>>> wrote:
>>>>>>>>
>>>>>>>> Well this means none of my DAG's work anymore:
>>>>>>>>
>>>>>>>> you just can do this anymore:
>>>>>>>>
>>>>>>>> file bqschema.py with
>>>>>>>>
>>>>>>>> def marketing_segment():
>>>>>>>> return [
>>>>>>>> {"name": "user_id", "type": "integer", "mode": "nullable"},
>>>>>>>> {"name": "bucket_date", "type": "timestamp", "mode":
>>>>> "nullable"},
>>>>>>>> {"name": "segment_main", "type": "string", "mode": "nullable"},
>>>>>>>> {"name": "segment_sub", "type": "integer", "mode": "nullable"},
>>>>>>>>
>>>>>>>>
>>>>>>>> In marketing_segmentation.py:
>>>>>>>>
>>>>>>>>
>>>>>>>> import bqschema
>>>>>>>>
>>>>>>>> Gives an error:
>>>>>>>>
>>>>>>>> Traceback (most recent call last):
>>>>>>>> File
>>>>>>>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+
>>>>>>>> apache.incubating-py2.7.egg/airflow/models.py",
>>>>>>>> line 264, in process_file
>>>>>>>> m = imp.load_source(mod_name, filepath)
>>>>>>>> File "/home/airflow/dags/marketing_segmentation.py", line 17, in
>>>>>>>> <module>
>>>>>>>> import bqschema
>>>>>>>> ImportError: No module named bqschema
>>>>>>>>
>>>>>>>> *I don't think this is incorrect?!*
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov <
>> dan.davydov@airbnb.com.
>>>>>>>> invalid>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> The latest commit fixed a regression since 1.7 that files with
>> parsing
>>>>>>>>> errors no longer showed up on the UI.
>>>>>>>>>
>>>>>>>>> On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel <al...@vanboxel.be>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Just installed beta 5 on our dev environment it lighted up as a
>>>>>>>> christmas
>>>>>>>>>> tree. I got a a screen full of import errors. I see that the
>> latest
>>>>>>>>> commit
>>>>>>>>>> did something with import errors... is it coorect?!
>>>>>>>>>>
>>>>>>>>>> On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin <bdbruin@gmail.com
>>>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hey Boris
>>>>>>>>>>>
>>>>>>>>>>> The scheduler is a bit more aggressive and can use multiple
>>>>>>>> processors,
>>>>>>>>>> so
>>>>>>>>>>> higher CPU usage is actually a good thing.
>>>>>>>>>>>
>>>>>>>>>>> I case it is really out of hand look at the new scheduler options
>>>>>>> and
>>>>>>>>>>> heartbeat options (see PR for updating.md not in the beta yet).
>>>>>>>>>>>
>>>>>>>>>>> Bolke
>>>>>>>>>>>
>>>>>>>>>>> Sent from my iPhone
>>>>>>>>>>>
>>>>>>>>>>>> On 29 Jan 2017, at 15:35, Boris Tyukin <bo...@boristyukin.com>
>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I am not sure if it is my config or something, but looks like
>>>>>>> after
>>>>>>>>> the
>>>>>>>>>>>> upgrade and start of scheduler, airflow would totally hose CPU.
>>>>>>> The
>>>>>>>>>>> reason
>>>>>>>>>>>> is two new examples that start running right away - latest only
>>>>>>> and
>>>>>>>>>>> latest
>>>>>>>>>>>> with trigger. Once I pause them, CPU goes back to idle. Is this
>>>>>>>>> because
>>>>>>>>>>> now
>>>>>>>>>>>> dags are not paused by default like it was before?
>>>>>>>>>>>>
>>>>>>>>>>>> As I mentioned before, I also had to upgrade mysql to 5.7 - if
>>>>>>>>> someone
>>>>>>>>>>>> needs a step by step instruction, make sure to follow all steps
>>>>>>>>>> precisely
>>>>>>>>>>>> here for in-place upgrade or you will have heck of the time
>> (like
>>>>>>>>> me).
>>>>>>>>>>>>
>>>>>>>>>>> https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#
>>>>>>>>>> upgrade-procedure-inplace
>>>>>>>>>>>>
>>>>>>>>>>>> BTW official Oracle repository for Oracle Linux only has MySql
>>>>>>> 5.6
>>>>>>>> -
>>>>>>>>>> for
>>>>>>>>>>>> 5.7 you have to use MySql community repo.
>>>>>>>>>>>>
>>>>>>>>>>>>> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin <
>>>>>>>> bdbruin@gmail.com
>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have made the FIFTH beta of Airflow 1.8.0 available at:
>>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
>>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> ,
>>>>>>>> public
>>>>>>>>>>> keys
>>>>>>>>>>>>> are available at https://dist.apache.org/repos/
>>>>>>>>>> dist/release/incubator/
>>>>>>>>>>>>> airflow/ <
>> https://dist.apache.org/repos/dist/release/incubator/
>>>>>>>>>> airflow/
>>>>>>>>>>>>
>>>>>>>>>>>>> . It is tagged with a local version “apache.incubating” so it
>>>>>>>> allows
>>>>>>>>>>>>> upgrading from earlier releases.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Issues fixed:
>>>>>>>>>>>>> * Parsing errors not showing up in UI fixing a regression**
>>>>>>>>>>>>> * Scheduler would terminate immediately if no dag files present
>>>>>>>>>>>>>
>>>>>>>>>>>>> ** As this touches the scheduler logic I though it warranted
>>>>>>>> another
>>>>>>>>>>> beta.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This should be the last beta in my opinion and we can prepare
>>>>>>>>>> changelog,
>>>>>>>>>>>>> upgrade notes and release notes for the RC (Feb 2).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>> Bolke
>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> _/
>>>>>>>>>> _/ Alex Van Boxel
>>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>>> _/
>>>>>>>> _/ Alex Van Boxel
>>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> _/
>>>>>> _/ Alex Van Boxel
>>>>>
>>>> --
>>>> _/
>>>> _/ Alex Van Boxel
>>>
>>
>> --
> _/
> _/ Alex Van Boxel
Re: Airflow 1.8.0 BETA 5
Posted by Alex Van Boxel <al...@vanboxel.be>.
I see it in the scheduler and in the UI. Currently trying to do a new
upgrade.
On Tue, Jan 31, 2017 at 8:00 PM Bolke de Bruin <bd...@gmail.com> wrote:
> Please note I will be holding off on the RC, before we understand this
> issue better.
>
> Bolke
>
> > On 31 Jan 2017, at 18:06, Bolke de Bruin <bd...@gmail.com> wrote:
> >
> > Hey Alex,
> >
> > Could it actually be that Airflow is doing the right thing? Earlier it
> was swallowing the errors. Where do you see the errors? In the scheduler
> logs, UI, processor logs?
> >
> > - Bolke
> >
> >> On 31 Jan 2017, at 16:25, Alex Van Boxel <al...@vanboxel.be> wrote:
> >>
> >> I'll try to identify the core problem
> >>
> >> On Tue, Jan 31, 2017, 16:43 Bolke de Bruin <bd...@gmail.com> wrote:
> >>
> >>> Hey Alex
> >>>
> >>> Can you provide some info on the scheduler paths thing. I don't
> have/see
> >>> that issue. Do you mean cli paths or by cfg? Jira would be nice in any
> case.
> >>>
> >>> I don't think the dag processor respects cli parameters.
> >>>
> >>> Bolke
> >>>
> >>> Sent from my iPhone
> >>>
> >>>> On 31 Jan 2017, at 15:10, Alex Van Boxel <al...@vanboxel.be> wrote:
> >>>>
> >>>> It's quite hard to share my complete dags. I don't have this locally,
> >>> but I
> >>>> have it in my production environment where I use Celery. I rolled
> back to
> >>>> beta 4 to make it work again.
> >>>>
> >>>> Also @bolke the scheduler logs don't respect the log path.
> >>>>
> >>>> On Tue, Jan 31, 2017 at 1:02 AM Dan Davydov <dan.davydov@airbnb.com
> >>> .invalid>
> >>>> wrote:
> >>>>
> >>>>> @Alex
> >>>>> I'm not able to reproduce locally (assuming the two python files are
> in
> >>> the
> >>>>> same folder or is on your PYTHONPATH). I don't see that import error
> >>>>> anyways.
> >>>>>
> >>>>> Just in case, what is your complete DAG definition? Is anyone else
> able
> >>> to
> >>>>> repro?
> >>>>>
> >>>>>> On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel <al...@vanboxel.be>
> >>> wrote:
> >>>>>>
> >>>>>> Well this means none of my DAG's work anymore:
> >>>>>>
> >>>>>> you just can do this anymore:
> >>>>>>
> >>>>>> file bqschema.py with
> >>>>>>
> >>>>>> def marketing_segment():
> >>>>>> return [
> >>>>>> {"name": "user_id", "type": "integer", "mode": "nullable"},
> >>>>>> {"name": "bucket_date", "type": "timestamp", "mode":
> >>> "nullable"},
> >>>>>> {"name": "segment_main", "type": "string", "mode": "nullable"},
> >>>>>> {"name": "segment_sub", "type": "integer", "mode": "nullable"},
> >>>>>>
> >>>>>>
> >>>>>> In marketing_segmentation.py:
> >>>>>>
> >>>>>>
> >>>>>> import bqschema
> >>>>>>
> >>>>>> Gives an error:
> >>>>>>
> >>>>>> Traceback (most recent call last):
> >>>>>> File
> >>>>>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+
> >>>>>> apache.incubating-py2.7.egg/airflow/models.py",
> >>>>>> line 264, in process_file
> >>>>>> m = imp.load_source(mod_name, filepath)
> >>>>>> File "/home/airflow/dags/marketing_segmentation.py", line 17, in
> >>>>>> <module>
> >>>>>> import bqschema
> >>>>>> ImportError: No module named bqschema
> >>>>>>
> >>>>>> *I don't think this is incorrect?!*
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov <
> dan.davydov@airbnb.com.
> >>>>>> invalid>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> The latest commit fixed a regression since 1.7 that files with
> parsing
> >>>>>>> errors no longer showed up on the UI.
> >>>>>>>
> >>>>>>> On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel <al...@vanboxel.be>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>> Just installed beta 5 on our dev environment it lighted up as a
> >>>>>> christmas
> >>>>>>>> tree. I got a a screen full of import errors. I see that the
> latest
> >>>>>>> commit
> >>>>>>>> did something with import errors... is it coorect?!
> >>>>>>>>
> >>>>>>>> On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin <bdbruin@gmail.com
> >
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hey Boris
> >>>>>>>>>
> >>>>>>>>> The scheduler is a bit more aggressive and can use multiple
> >>>>>> processors,
> >>>>>>>> so
> >>>>>>>>> higher CPU usage is actually a good thing.
> >>>>>>>>>
> >>>>>>>>> I case it is really out of hand look at the new scheduler options
> >>>>> and
> >>>>>>>>> heartbeat options (see PR for updating.md not in the beta yet).
> >>>>>>>>>
> >>>>>>>>> Bolke
> >>>>>>>>>
> >>>>>>>>> Sent from my iPhone
> >>>>>>>>>
> >>>>>>>>>> On 29 Jan 2017, at 15:35, Boris Tyukin <bo...@boristyukin.com>
> >>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> I am not sure if it is my config or something, but looks like
> >>>>> after
> >>>>>>> the
> >>>>>>>>>> upgrade and start of scheduler, airflow would totally hose CPU.
> >>>>> The
> >>>>>>>>> reason
> >>>>>>>>>> is two new examples that start running right away - latest only
> >>>>> and
> >>>>>>>>> latest
> >>>>>>>>>> with trigger. Once I pause them, CPU goes back to idle. Is this
> >>>>>>> because
> >>>>>>>>> now
> >>>>>>>>>> dags are not paused by default like it was before?
> >>>>>>>>>>
> >>>>>>>>>> As I mentioned before, I also had to upgrade mysql to 5.7 - if
> >>>>>>> someone
> >>>>>>>>>> needs a step by step instruction, make sure to follow all steps
> >>>>>>>> precisely
> >>>>>>>>>> here for in-place upgrade or you will have heck of the time
> (like
> >>>>>>> me).
> >>>>>>>>>>
> >>>>>>>>> https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#
> >>>>>>>> upgrade-procedure-inplace
> >>>>>>>>>>
> >>>>>>>>>> BTW official Oracle repository for Oracle Linux only has MySql
> >>>>> 5.6
> >>>>>> -
> >>>>>>>> for
> >>>>>>>>>> 5.7 you have to use MySql community repo.
> >>>>>>>>>>
> >>>>>>>>>>> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin <
> >>>>>> bdbruin@gmail.com
> >>>>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Hi All,
> >>>>>>>>>>>
> >>>>>>>>>>> I have made the FIFTH beta of Airflow 1.8.0 available at:
> >>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> >>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> ,
> >>>>>> public
> >>>>>>>>> keys
> >>>>>>>>>>> are available at https://dist.apache.org/repos/
> >>>>>>>> dist/release/incubator/
> >>>>>>>>>>> airflow/ <
> https://dist.apache.org/repos/dist/release/incubator/
> >>>>>>>> airflow/
> >>>>>>>>>>
> >>>>>>>>>>> . It is tagged with a local version “apache.incubating” so it
> >>>>>> allows
> >>>>>>>>>>> upgrading from earlier releases.
> >>>>>>>>>>>
> >>>>>>>>>>> Issues fixed:
> >>>>>>>>>>> * Parsing errors not showing up in UI fixing a regression**
> >>>>>>>>>>> * Scheduler would terminate immediately if no dag files present
> >>>>>>>>>>>
> >>>>>>>>>>> ** As this touches the scheduler logic I though it warranted
> >>>>>> another
> >>>>>>>>> beta.
> >>>>>>>>>>>
> >>>>>>>>>>> This should be the last beta in my opinion and we can prepare
> >>>>>>>> changelog,
> >>>>>>>>>>> upgrade notes and release notes for the RC (Feb 2).
> >>>>>>>>>>>
> >>>>>>>>>>> Cheers
> >>>>>>>>>>> Bolke
> >>>>>>>>>
> >>>>>>>> --
> >>>>>>>> _/
> >>>>>>>> _/ Alex Van Boxel
> >>>>>>>>
> >>>>>>>
> >>>>>> --
> >>>>>> _/
> >>>>>> _/ Alex Van Boxel
> >>>>>>
> >>>>>
> >>>> --
> >>>> _/
> >>>> _/ Alex Van Boxel
> >>>
> >> --
> >> _/
> >> _/ Alex Van Boxel
> >
>
> --
_/
_/ Alex Van Boxel
Re: Airflow 1.8.0 BETA 5
Posted by Bolke de Bruin <bd...@gmail.com>.
Please note I will be holding off on the RC, before we understand this issue better.
Bolke
> On 31 Jan 2017, at 18:06, Bolke de Bruin <bd...@gmail.com> wrote:
>
> Hey Alex,
>
> Could it actually be that Airflow is doing the right thing? Earlier it was swallowing the errors. Where do you see the errors? In the scheduler logs, UI, processor logs?
>
> - Bolke
>
>> On 31 Jan 2017, at 16:25, Alex Van Boxel <al...@vanboxel.be> wrote:
>>
>> I'll try to identify the core problem
>>
>> On Tue, Jan 31, 2017, 16:43 Bolke de Bruin <bd...@gmail.com> wrote:
>>
>>> Hey Alex
>>>
>>> Can you provide some info on the scheduler paths thing. I don't have/see
>>> that issue. Do you mean cli paths or by cfg? Jira would be nice in any case.
>>>
>>> I don't think the dag processor respects cli parameters.
>>>
>>> Bolke
>>>
>>> Sent from my iPhone
>>>
>>>> On 31 Jan 2017, at 15:10, Alex Van Boxel <al...@vanboxel.be> wrote:
>>>>
>>>> It's quite hard to share my complete dags. I don't have this locally,
>>> but I
>>>> have it in my production environment where I use Celery. I rolled back to
>>>> beta 4 to make it work again.
>>>>
>>>> Also @bolke the scheduler logs don't respect the log path.
>>>>
>>>> On Tue, Jan 31, 2017 at 1:02 AM Dan Davydov <dan.davydov@airbnb.com
>>> .invalid>
>>>> wrote:
>>>>
>>>>> @Alex
>>>>> I'm not able to reproduce locally (assuming the two python files are in
>>> the
>>>>> same folder or is on your PYTHONPATH). I don't see that import error
>>>>> anyways.
>>>>>
>>>>> Just in case, what is your complete DAG definition? Is anyone else able
>>> to
>>>>> repro?
>>>>>
>>>>>> On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel <al...@vanboxel.be>
>>> wrote:
>>>>>>
>>>>>> Well this means none of my DAG's work anymore:
>>>>>>
>>>>>> you just can do this anymore:
>>>>>>
>>>>>> file bqschema.py with
>>>>>>
>>>>>> def marketing_segment():
>>>>>> return [
>>>>>> {"name": "user_id", "type": "integer", "mode": "nullable"},
>>>>>> {"name": "bucket_date", "type": "timestamp", "mode":
>>> "nullable"},
>>>>>> {"name": "segment_main", "type": "string", "mode": "nullable"},
>>>>>> {"name": "segment_sub", "type": "integer", "mode": "nullable"},
>>>>>>
>>>>>>
>>>>>> In marketing_segmentation.py:
>>>>>>
>>>>>>
>>>>>> import bqschema
>>>>>>
>>>>>> Gives an error:
>>>>>>
>>>>>> Traceback (most recent call last):
>>>>>> File
>>>>>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+
>>>>>> apache.incubating-py2.7.egg/airflow/models.py",
>>>>>> line 264, in process_file
>>>>>> m = imp.load_source(mod_name, filepath)
>>>>>> File "/home/airflow/dags/marketing_segmentation.py", line 17, in
>>>>>> <module>
>>>>>> import bqschema
>>>>>> ImportError: No module named bqschema
>>>>>>
>>>>>> *I don't think this is incorrect?!*
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov <dan.davydov@airbnb.com.
>>>>>> invalid>
>>>>>> wrote:
>>>>>>
>>>>>>> The latest commit fixed a regression since 1.7 that files with parsing
>>>>>>> errors no longer showed up on the UI.
>>>>>>>
>>>>>>> On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel <al...@vanboxel.be>
>>>>>> wrote:
>>>>>>>
>>>>>>>> Just installed beta 5 on our dev environment it lighted up as a
>>>>>> christmas
>>>>>>>> tree. I got a a screen full of import errors. I see that the latest
>>>>>>> commit
>>>>>>>> did something with import errors... is it coorect?!
>>>>>>>>
>>>>>>>> On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin <bd...@gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hey Boris
>>>>>>>>>
>>>>>>>>> The scheduler is a bit more aggressive and can use multiple
>>>>>> processors,
>>>>>>>> so
>>>>>>>>> higher CPU usage is actually a good thing.
>>>>>>>>>
>>>>>>>>> I case it is really out of hand look at the new scheduler options
>>>>> and
>>>>>>>>> heartbeat options (see PR for updating.md not in the beta yet).
>>>>>>>>>
>>>>>>>>> Bolke
>>>>>>>>>
>>>>>>>>> Sent from my iPhone
>>>>>>>>>
>>>>>>>>>> On 29 Jan 2017, at 15:35, Boris Tyukin <bo...@boristyukin.com>
>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> I am not sure if it is my config or something, but looks like
>>>>> after
>>>>>>> the
>>>>>>>>>> upgrade and start of scheduler, airflow would totally hose CPU.
>>>>> The
>>>>>>>>> reason
>>>>>>>>>> is two new examples that start running right away - latest only
>>>>> and
>>>>>>>>> latest
>>>>>>>>>> with trigger. Once I pause them, CPU goes back to idle. Is this
>>>>>>> because
>>>>>>>>> now
>>>>>>>>>> dags are not paused by default like it was before?
>>>>>>>>>>
>>>>>>>>>> As I mentioned before, I also had to upgrade mysql to 5.7 - if
>>>>>>> someone
>>>>>>>>>> needs a step by step instruction, make sure to follow all steps
>>>>>>>> precisely
>>>>>>>>>> here for in-place upgrade or you will have heck of the time (like
>>>>>>> me).
>>>>>>>>>>
>>>>>>>>> https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#
>>>>>>>> upgrade-procedure-inplace
>>>>>>>>>>
>>>>>>>>>> BTW official Oracle repository for Oracle Linux only has MySql
>>>>> 5.6
>>>>>> -
>>>>>>>> for
>>>>>>>>>> 5.7 you have to use MySql community repo.
>>>>>>>>>>
>>>>>>>>>>> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin <
>>>>>> bdbruin@gmail.com
>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi All,
>>>>>>>>>>>
>>>>>>>>>>> I have made the FIFTH beta of Airflow 1.8.0 available at:
>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> ,
>>>>>> public
>>>>>>>>> keys
>>>>>>>>>>> are available at https://dist.apache.org/repos/
>>>>>>>> dist/release/incubator/
>>>>>>>>>>> airflow/ <https://dist.apache.org/repos/dist/release/incubator/
>>>>>>>> airflow/
>>>>>>>>>>
>>>>>>>>>>> . It is tagged with a local version “apache.incubating” so it
>>>>>> allows
>>>>>>>>>>> upgrading from earlier releases.
>>>>>>>>>>>
>>>>>>>>>>> Issues fixed:
>>>>>>>>>>> * Parsing errors not showing up in UI fixing a regression**
>>>>>>>>>>> * Scheduler would terminate immediately if no dag files present
>>>>>>>>>>>
>>>>>>>>>>> ** As this touches the scheduler logic I though it warranted
>>>>>> another
>>>>>>>>> beta.
>>>>>>>>>>>
>>>>>>>>>>> This should be the last beta in my opinion and we can prepare
>>>>>>>> changelog,
>>>>>>>>>>> upgrade notes and release notes for the RC (Feb 2).
>>>>>>>>>>>
>>>>>>>>>>> Cheers
>>>>>>>>>>> Bolke
>>>>>>>>>
>>>>>>>> --
>>>>>>>> _/
>>>>>>>> _/ Alex Van Boxel
>>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> _/
>>>>>> _/ Alex Van Boxel
>>>>>>
>>>>>
>>>> --
>>>> _/
>>>> _/ Alex Van Boxel
>>>
>> --
>> _/
>> _/ Alex Van Boxel
>
Re: Airflow 1.8.0 BETA 5
Posted by Bolke de Bruin <bd...@gmail.com>.
Hey Alex,
Could it actually be that Airflow is doing the right thing? Earlier it was swallowing the errors. Where do you see the errors? In the scheduler logs, UI, processor logs?
- Bolke
> On 31 Jan 2017, at 16:25, Alex Van Boxel <al...@vanboxel.be> wrote:
>
> I'll try to identify the core problem
>
> On Tue, Jan 31, 2017, 16:43 Bolke de Bruin <bd...@gmail.com> wrote:
>
>> Hey Alex
>>
>> Can you provide some info on the scheduler paths thing. I don't have/see
>> that issue. Do you mean cli paths or by cfg? Jira would be nice in any case.
>>
>> I don't think the dag processor respects cli parameters.
>>
>> Bolke
>>
>> Sent from my iPhone
>>
>>> On 31 Jan 2017, at 15:10, Alex Van Boxel <al...@vanboxel.be> wrote:
>>>
>>> It's quite hard to share my complete dags. I don't have this locally,
>> but I
>>> have it in my production environment where I use Celery. I rolled back to
>>> beta 4 to make it work again.
>>>
>>> Also @bolke the scheduler logs don't respect the log path.
>>>
>>> On Tue, Jan 31, 2017 at 1:02 AM Dan Davydov <dan.davydov@airbnb.com
>> .invalid>
>>> wrote:
>>>
>>>> @Alex
>>>> I'm not able to reproduce locally (assuming the two python files are in
>> the
>>>> same folder or is on your PYTHONPATH). I don't see that import error
>>>> anyways.
>>>>
>>>> Just in case, what is your complete DAG definition? Is anyone else able
>> to
>>>> repro?
>>>>
>>>>> On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel <al...@vanboxel.be>
>> wrote:
>>>>>
>>>>> Well this means none of my DAG's work anymore:
>>>>>
>>>>> you just can do this anymore:
>>>>>
>>>>> file bqschema.py with
>>>>>
>>>>> def marketing_segment():
>>>>> return [
>>>>> {"name": "user_id", "type": "integer", "mode": "nullable"},
>>>>> {"name": "bucket_date", "type": "timestamp", "mode":
>> "nullable"},
>>>>> {"name": "segment_main", "type": "string", "mode": "nullable"},
>>>>> {"name": "segment_sub", "type": "integer", "mode": "nullable"},
>>>>>
>>>>>
>>>>> In marketing_segmentation.py:
>>>>>
>>>>>
>>>>> import bqschema
>>>>>
>>>>> Gives an error:
>>>>>
>>>>> Traceback (most recent call last):
>>>>> File
>>>>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+
>>>>> apache.incubating-py2.7.egg/airflow/models.py",
>>>>> line 264, in process_file
>>>>> m = imp.load_source(mod_name, filepath)
>>>>> File "/home/airflow/dags/marketing_segmentation.py", line 17, in
>>>>> <module>
>>>>> import bqschema
>>>>> ImportError: No module named bqschema
>>>>>
>>>>> *I don't think this is incorrect?!*
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov <dan.davydov@airbnb.com.
>>>>> invalid>
>>>>> wrote:
>>>>>
>>>>>> The latest commit fixed a regression since 1.7 that files with parsing
>>>>>> errors no longer showed up on the UI.
>>>>>>
>>>>>> On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel <al...@vanboxel.be>
>>>>> wrote:
>>>>>>
>>>>>>> Just installed beta 5 on our dev environment it lighted up as a
>>>>> christmas
>>>>>>> tree. I got a a screen full of import errors. I see that the latest
>>>>>> commit
>>>>>>> did something with import errors... is it coorect?!
>>>>>>>
>>>>>>> On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin <bd...@gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>>> Hey Boris
>>>>>>>>
>>>>>>>> The scheduler is a bit more aggressive and can use multiple
>>>>> processors,
>>>>>>> so
>>>>>>>> higher CPU usage is actually a good thing.
>>>>>>>>
>>>>>>>> I case it is really out of hand look at the new scheduler options
>>>> and
>>>>>>>> heartbeat options (see PR for updating.md not in the beta yet).
>>>>>>>>
>>>>>>>> Bolke
>>>>>>>>
>>>>>>>> Sent from my iPhone
>>>>>>>>
>>>>>>>>> On 29 Jan 2017, at 15:35, Boris Tyukin <bo...@boristyukin.com>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> I am not sure if it is my config or something, but looks like
>>>> after
>>>>>> the
>>>>>>>>> upgrade and start of scheduler, airflow would totally hose CPU.
>>>> The
>>>>>>>> reason
>>>>>>>>> is two new examples that start running right away - latest only
>>>> and
>>>>>>>> latest
>>>>>>>>> with trigger. Once I pause them, CPU goes back to idle. Is this
>>>>>> because
>>>>>>>> now
>>>>>>>>> dags are not paused by default like it was before?
>>>>>>>>>
>>>>>>>>> As I mentioned before, I also had to upgrade mysql to 5.7 - if
>>>>>> someone
>>>>>>>>> needs a step by step instruction, make sure to follow all steps
>>>>>>> precisely
>>>>>>>>> here for in-place upgrade or you will have heck of the time (like
>>>>>> me).
>>>>>>>>>
>>>>>>>> https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#
>>>>>>> upgrade-procedure-inplace
>>>>>>>>>
>>>>>>>>> BTW official Oracle repository for Oracle Linux only has MySql
>>>> 5.6
>>>>> -
>>>>>>> for
>>>>>>>>> 5.7 you have to use MySql community repo.
>>>>>>>>>
>>>>>>>>>> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin <
>>>>> bdbruin@gmail.com
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi All,
>>>>>>>>>>
>>>>>>>>>> I have made the FIFTH beta of Airflow 1.8.0 available at:
>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> ,
>>>>> public
>>>>>>>> keys
>>>>>>>>>> are available at https://dist.apache.org/repos/
>>>>>>> dist/release/incubator/
>>>>>>>>>> airflow/ <https://dist.apache.org/repos/dist/release/incubator/
>>>>>>> airflow/
>>>>>>>>>
>>>>>>>>>> . It is tagged with a local version “apache.incubating” so it
>>>>> allows
>>>>>>>>>> upgrading from earlier releases.
>>>>>>>>>>
>>>>>>>>>> Issues fixed:
>>>>>>>>>> * Parsing errors not showing up in UI fixing a regression**
>>>>>>>>>> * Scheduler would terminate immediately if no dag files present
>>>>>>>>>>
>>>>>>>>>> ** As this touches the scheduler logic I though it warranted
>>>>> another
>>>>>>>> beta.
>>>>>>>>>>
>>>>>>>>>> This should be the last beta in my opinion and we can prepare
>>>>>>> changelog,
>>>>>>>>>> upgrade notes and release notes for the RC (Feb 2).
>>>>>>>>>>
>>>>>>>>>> Cheers
>>>>>>>>>> Bolke
>>>>>>>>
>>>>>>> --
>>>>>>> _/
>>>>>>> _/ Alex Van Boxel
>>>>>>>
>>>>>>
>>>>> --
>>>>> _/
>>>>> _/ Alex Van Boxel
>>>>>
>>>>
>>> --
>>> _/
>>> _/ Alex Van Boxel
>>
> --
> _/
> _/ Alex Van Boxel
Re: Airflow 1.8.0 BETA 5
Posted by Alex Van Boxel <al...@vanboxel.be>.
I'll try to identify the core problem
On Tue, Jan 31, 2017, 16:43 Bolke de Bruin <bd...@gmail.com> wrote:
> Hey Alex
>
> Can you provide some info on the scheduler paths thing. I don't have/see
> that issue. Do you mean cli paths or by cfg? Jira would be nice in any case.
>
> I don't think the dag processor respects cli parameters.
>
> Bolke
>
> Sent from my iPhone
>
> > On 31 Jan 2017, at 15:10, Alex Van Boxel <al...@vanboxel.be> wrote:
> >
> > It's quite hard to share my complete dags. I don't have this locally,
> but I
> > have it in my production environment where I use Celery. I rolled back to
> > beta 4 to make it work again.
> >
> > Also @bolke the scheduler logs don't respect the log path.
> >
> > On Tue, Jan 31, 2017 at 1:02 AM Dan Davydov <dan.davydov@airbnb.com
> .invalid>
> > wrote:
> >
> >> @Alex
> >> I'm not able to reproduce locally (assuming the two python files are in
> the
> >> same folder or is on your PYTHONPATH). I don't see that import error
> >> anyways.
> >>
> >> Just in case, what is your complete DAG definition? Is anyone else able
> to
> >> repro?
> >>
> >>> On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel <al...@vanboxel.be>
> wrote:
> >>>
> >>> Well this means none of my DAG's work anymore:
> >>>
> >>> you just can do this anymore:
> >>>
> >>> file bqschema.py with
> >>>
> >>> def marketing_segment():
> >>> return [
> >>> {"name": "user_id", "type": "integer", "mode": "nullable"},
> >>> {"name": "bucket_date", "type": "timestamp", "mode":
> "nullable"},
> >>> {"name": "segment_main", "type": "string", "mode": "nullable"},
> >>> {"name": "segment_sub", "type": "integer", "mode": "nullable"},
> >>>
> >>>
> >>> In marketing_segmentation.py:
> >>>
> >>>
> >>> import bqschema
> >>>
> >>> Gives an error:
> >>>
> >>> Traceback (most recent call last):
> >>> File
> >>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+
> >>> apache.incubating-py2.7.egg/airflow/models.py",
> >>> line 264, in process_file
> >>> m = imp.load_source(mod_name, filepath)
> >>> File "/home/airflow/dags/marketing_segmentation.py", line 17, in
> >>> <module>
> >>> import bqschema
> >>> ImportError: No module named bqschema
> >>>
> >>> *I don't think this is incorrect?!*
> >>>
> >>>
> >>>
> >>> On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov <dan.davydov@airbnb.com.
> >>> invalid>
> >>> wrote:
> >>>
> >>>> The latest commit fixed a regression since 1.7 that files with parsing
> >>>> errors no longer showed up on the UI.
> >>>>
> >>>> On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel <al...@vanboxel.be>
> >>> wrote:
> >>>>
> >>>>> Just installed beta 5 on our dev environment it lighted up as a
> >>> christmas
> >>>>> tree. I got a a screen full of import errors. I see that the latest
> >>>> commit
> >>>>> did something with import errors... is it coorect?!
> >>>>>
> >>>>> On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin <bd...@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>>> Hey Boris
> >>>>>>
> >>>>>> The scheduler is a bit more aggressive and can use multiple
> >>> processors,
> >>>>> so
> >>>>>> higher CPU usage is actually a good thing.
> >>>>>>
> >>>>>> I case it is really out of hand look at the new scheduler options
> >> and
> >>>>>> heartbeat options (see PR for updating.md not in the beta yet).
> >>>>>>
> >>>>>> Bolke
> >>>>>>
> >>>>>> Sent from my iPhone
> >>>>>>
> >>>>>>> On 29 Jan 2017, at 15:35, Boris Tyukin <bo...@boristyukin.com>
> >>>> wrote:
> >>>>>>>
> >>>>>>> I am not sure if it is my config or something, but looks like
> >> after
> >>>> the
> >>>>>>> upgrade and start of scheduler, airflow would totally hose CPU.
> >> The
> >>>>>> reason
> >>>>>>> is two new examples that start running right away - latest only
> >> and
> >>>>>> latest
> >>>>>>> with trigger. Once I pause them, CPU goes back to idle. Is this
> >>>> because
> >>>>>> now
> >>>>>>> dags are not paused by default like it was before?
> >>>>>>>
> >>>>>>> As I mentioned before, I also had to upgrade mysql to 5.7 - if
> >>>> someone
> >>>>>>> needs a step by step instruction, make sure to follow all steps
> >>>>> precisely
> >>>>>>> here for in-place upgrade or you will have heck of the time (like
> >>>> me).
> >>>>>>>
> >>>>>> https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#
> >>>>> upgrade-procedure-inplace
> >>>>>>>
> >>>>>>> BTW official Oracle repository for Oracle Linux only has MySql
> >> 5.6
> >>> -
> >>>>> for
> >>>>>>> 5.7 you have to use MySql community repo.
> >>>>>>>
> >>>>>>>> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin <
> >>> bdbruin@gmail.com
> >>>>>
> >>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi All,
> >>>>>>>>
> >>>>>>>> I have made the FIFTH beta of Airflow 1.8.0 available at:
> >>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> >>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> ,
> >>> public
> >>>>>> keys
> >>>>>>>> are available at https://dist.apache.org/repos/
> >>>>> dist/release/incubator/
> >>>>>>>> airflow/ <https://dist.apache.org/repos/dist/release/incubator/
> >>>>> airflow/
> >>>>>>>
> >>>>>>>> . It is tagged with a local version “apache.incubating” so it
> >>> allows
> >>>>>>>> upgrading from earlier releases.
> >>>>>>>>
> >>>>>>>> Issues fixed:
> >>>>>>>> * Parsing errors not showing up in UI fixing a regression**
> >>>>>>>> * Scheduler would terminate immediately if no dag files present
> >>>>>>>>
> >>>>>>>> ** As this touches the scheduler logic I though it warranted
> >>> another
> >>>>>> beta.
> >>>>>>>>
> >>>>>>>> This should be the last beta in my opinion and we can prepare
> >>>>> changelog,
> >>>>>>>> upgrade notes and release notes for the RC (Feb 2).
> >>>>>>>>
> >>>>>>>> Cheers
> >>>>>>>> Bolke
> >>>>>>
> >>>>> --
> >>>>> _/
> >>>>> _/ Alex Van Boxel
> >>>>>
> >>>>
> >>> --
> >>> _/
> >>> _/ Alex Van Boxel
> >>>
> >>
> > --
> > _/
> > _/ Alex Van Boxel
>
--
_/
_/ Alex Van Boxel
Re: Airflow 1.8.0 BETA 5
Posted by Bolke de Bruin <bd...@gmail.com>.
Hey Alex
Can you provide some info on the scheduler paths thing. I don't have/see that issue. Do you mean cli paths or by cfg? Jira would be nice in any case.
I don't think the dag processor respects cli parameters.
Bolke
Sent from my iPhone
> On 31 Jan 2017, at 15:10, Alex Van Boxel <al...@vanboxel.be> wrote:
>
> It's quite hard to share my complete dags. I don't have this locally, but I
> have it in my production environment where I use Celery. I rolled back to
> beta 4 to make it work again.
>
> Also @bolke the scheduler logs don't respect the log path.
>
> On Tue, Jan 31, 2017 at 1:02 AM Dan Davydov <da...@airbnb.com.invalid>
> wrote:
>
>> @Alex
>> I'm not able to reproduce locally (assuming the two python files are in the
>> same folder or is on your PYTHONPATH). I don't see that import error
>> anyways.
>>
>> Just in case, what is your complete DAG definition? Is anyone else able to
>> repro?
>>
>>> On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel <al...@vanboxel.be> wrote:
>>>
>>> Well this means none of my DAG's work anymore:
>>>
>>> you just can do this anymore:
>>>
>>> file bqschema.py with
>>>
>>> def marketing_segment():
>>> return [
>>> {"name": "user_id", "type": "integer", "mode": "nullable"},
>>> {"name": "bucket_date", "type": "timestamp", "mode": "nullable"},
>>> {"name": "segment_main", "type": "string", "mode": "nullable"},
>>> {"name": "segment_sub", "type": "integer", "mode": "nullable"},
>>>
>>>
>>> In marketing_segmentation.py:
>>>
>>>
>>> import bqschema
>>>
>>> Gives an error:
>>>
>>> Traceback (most recent call last):
>>> File
>>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+
>>> apache.incubating-py2.7.egg/airflow/models.py",
>>> line 264, in process_file
>>> m = imp.load_source(mod_name, filepath)
>>> File "/home/airflow/dags/marketing_segmentation.py", line 17, in
>>> <module>
>>> import bqschema
>>> ImportError: No module named bqschema
>>>
>>> *I don't think this is incorrect?!*
>>>
>>>
>>>
>>> On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov <dan.davydov@airbnb.com.
>>> invalid>
>>> wrote:
>>>
>>>> The latest commit fixed a regression since 1.7 that files with parsing
>>>> errors no longer showed up on the UI.
>>>>
>>>> On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel <al...@vanboxel.be>
>>> wrote:
>>>>
>>>>> Just installed beta 5 on our dev environment it lighted up as a
>>> christmas
>>>>> tree. I got a a screen full of import errors. I see that the latest
>>>> commit
>>>>> did something with import errors... is it coorect?!
>>>>>
>>>>> On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin <bd...@gmail.com>
>>>> wrote:
>>>>>
>>>>>> Hey Boris
>>>>>>
>>>>>> The scheduler is a bit more aggressive and can use multiple
>>> processors,
>>>>> so
>>>>>> higher CPU usage is actually a good thing.
>>>>>>
>>>>>> I case it is really out of hand look at the new scheduler options
>> and
>>>>>> heartbeat options (see PR for updating.md not in the beta yet).
>>>>>>
>>>>>> Bolke
>>>>>>
>>>>>> Sent from my iPhone
>>>>>>
>>>>>>> On 29 Jan 2017, at 15:35, Boris Tyukin <bo...@boristyukin.com>
>>>> wrote:
>>>>>>>
>>>>>>> I am not sure if it is my config or something, but looks like
>> after
>>>> the
>>>>>>> upgrade and start of scheduler, airflow would totally hose CPU.
>> The
>>>>>> reason
>>>>>>> is two new examples that start running right away - latest only
>> and
>>>>>> latest
>>>>>>> with trigger. Once I pause them, CPU goes back to idle. Is this
>>>> because
>>>>>> now
>>>>>>> dags are not paused by default like it was before?
>>>>>>>
>>>>>>> As I mentioned before, I also had to upgrade mysql to 5.7 - if
>>>> someone
>>>>>>> needs a step by step instruction, make sure to follow all steps
>>>>> precisely
>>>>>>> here for in-place upgrade or you will have heck of the time (like
>>>> me).
>>>>>>>
>>>>>> https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#
>>>>> upgrade-procedure-inplace
>>>>>>>
>>>>>>> BTW official Oracle repository for Oracle Linux only has MySql
>> 5.6
>>> -
>>>>> for
>>>>>>> 5.7 you have to use MySql community repo.
>>>>>>>
>>>>>>>> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin <
>>> bdbruin@gmail.com
>>>>>
>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> I have made the FIFTH beta of Airflow 1.8.0 available at:
>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> ,
>>> public
>>>>>> keys
>>>>>>>> are available at https://dist.apache.org/repos/
>>>>> dist/release/incubator/
>>>>>>>> airflow/ <https://dist.apache.org/repos/dist/release/incubator/
>>>>> airflow/
>>>>>>>
>>>>>>>> . It is tagged with a local version “apache.incubating” so it
>>> allows
>>>>>>>> upgrading from earlier releases.
>>>>>>>>
>>>>>>>> Issues fixed:
>>>>>>>> * Parsing errors not showing up in UI fixing a regression**
>>>>>>>> * Scheduler would terminate immediately if no dag files present
>>>>>>>>
>>>>>>>> ** As this touches the scheduler logic I though it warranted
>>> another
>>>>>> beta.
>>>>>>>>
>>>>>>>> This should be the last beta in my opinion and we can prepare
>>>>> changelog,
>>>>>>>> upgrade notes and release notes for the RC (Feb 2).
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>> Bolke
>>>>>>
>>>>> --
>>>>> _/
>>>>> _/ Alex Van Boxel
>>>>>
>>>>
>>> --
>>> _/
>>> _/ Alex Van Boxel
>>>
>>
> --
> _/
> _/ Alex Van Boxel
Re: Airflow 1.8.0 BETA 5
Posted by Alex Van Boxel <al...@vanboxel.be>.
It's quite hard to share my complete dags. I don't have this locally, but I
have it in my production environment where I use Celery. I rolled back to
beta 4 to make it work again.
Also @bolke the scheduler logs don't respect the log path.
On Tue, Jan 31, 2017 at 1:02 AM Dan Davydov <da...@airbnb.com.invalid>
wrote:
> @Alex
> I'm not able to reproduce locally (assuming the two python files are in the
> same folder or is on your PYTHONPATH). I don't see that import error
> anyways.
>
> Just in case, what is your complete DAG definition? Is anyone else able to
> repro?
>
> On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel <al...@vanboxel.be> wrote:
>
> > Well this means none of my DAG's work anymore:
> >
> > you just can do this anymore:
> >
> > file bqschema.py with
> >
> > def marketing_segment():
> > return [
> > {"name": "user_id", "type": "integer", "mode": "nullable"},
> > {"name": "bucket_date", "type": "timestamp", "mode": "nullable"},
> > {"name": "segment_main", "type": "string", "mode": "nullable"},
> > {"name": "segment_sub", "type": "integer", "mode": "nullable"},
> >
> >
> > In marketing_segmentation.py:
> >
> >
> > import bqschema
> >
> > Gives an error:
> >
> > Traceback (most recent call last):
> > File
> > "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+
> > apache.incubating-py2.7.egg/airflow/models.py",
> > line 264, in process_file
> > m = imp.load_source(mod_name, filepath)
> > File "/home/airflow/dags/marketing_segmentation.py", line 17, in
> > <module>
> > import bqschema
> > ImportError: No module named bqschema
> >
> > *I don't think this is incorrect?!*
> >
> >
> >
> > On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov <dan.davydov@airbnb.com.
> > invalid>
> > wrote:
> >
> > > The latest commit fixed a regression since 1.7 that files with parsing
> > > errors no longer showed up on the UI.
> > >
> > > On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel <al...@vanboxel.be>
> > wrote:
> > >
> > > > Just installed beta 5 on our dev environment it lighted up as a
> > christmas
> > > > tree. I got a a screen full of import errors. I see that the latest
> > > commit
> > > > did something with import errors... is it coorect?!
> > > >
> > > > On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin <bd...@gmail.com>
> > > wrote:
> > > >
> > > > > Hey Boris
> > > > >
> > > > > The scheduler is a bit more aggressive and can use multiple
> > processors,
> > > > so
> > > > > higher CPU usage is actually a good thing.
> > > > >
> > > > > I case it is really out of hand look at the new scheduler options
> and
> > > > > heartbeat options (see PR for updating.md not in the beta yet).
> > > > >
> > > > > Bolke
> > > > >
> > > > > Sent from my iPhone
> > > > >
> > > > > > On 29 Jan 2017, at 15:35, Boris Tyukin <bo...@boristyukin.com>
> > > wrote:
> > > > > >
> > > > > > I am not sure if it is my config or something, but looks like
> after
> > > the
> > > > > > upgrade and start of scheduler, airflow would totally hose CPU.
> The
> > > > > reason
> > > > > > is two new examples that start running right away - latest only
> and
> > > > > latest
> > > > > > with trigger. Once I pause them, CPU goes back to idle. Is this
> > > because
> > > > > now
> > > > > > dags are not paused by default like it was before?
> > > > > >
> > > > > > As I mentioned before, I also had to upgrade mysql to 5.7 - if
> > > someone
> > > > > > needs a step by step instruction, make sure to follow all steps
> > > > precisely
> > > > > > here for in-place upgrade or you will have heck of the time (like
> > > me).
> > > > > >
> > > > > https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#
> > > > upgrade-procedure-inplace
> > > > > >
> > > > > > BTW official Oracle repository for Oracle Linux only has MySql
> 5.6
> > -
> > > > for
> > > > > > 5.7 you have to use MySql community repo.
> > > > > >
> > > > > >> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin <
> > bdbruin@gmail.com
> > > >
> > > > > wrote:
> > > > > >>
> > > > > >> Hi All,
> > > > > >>
> > > > > >> I have made the FIFTH beta of Airflow 1.8.0 available at:
> > > > > >> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> > > > > >> https://dist.apache.org/repos/dist/dev/incubator/airflow/> ,
> > public
> > > > > keys
> > > > > >> are available at https://dist.apache.org/repos/
> > > > dist/release/incubator/
> > > > > >> airflow/ <https://dist.apache.org/repos/dist/release/incubator/
> > > > airflow/
> > > > > >
> > > > > >> . It is tagged with a local version “apache.incubating” so it
> > allows
> > > > > >> upgrading from earlier releases.
> > > > > >>
> > > > > >> Issues fixed:
> > > > > >> * Parsing errors not showing up in UI fixing a regression**
> > > > > >> * Scheduler would terminate immediately if no dag files present
> > > > > >>
> > > > > >> ** As this touches the scheduler logic I though it warranted
> > another
> > > > > beta.
> > > > > >>
> > > > > >> This should be the last beta in my opinion and we can prepare
> > > > changelog,
> > > > > >> upgrade notes and release notes for the RC (Feb 2).
> > > > > >>
> > > > > >> Cheers
> > > > > >> Bolke
> > > > >
> > > > --
> > > > _/
> > > > _/ Alex Van Boxel
> > > >
> > >
> > --
> > _/
> > _/ Alex Van Boxel
> >
>
--
_/
_/ Alex Van Boxel
Re: Airflow 1.8.0 BETA 5
Posted by Dan Davydov <da...@airbnb.com.INVALID>.
@Alex
I'm not able to reproduce locally (assuming the two python files are in the
same folder or is on your PYTHONPATH). I don't see that import error
anyways.
Just in case, what is your complete DAG definition? Is anyone else able to
repro?
On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel <al...@vanboxel.be> wrote:
> Well this means none of my DAG's work anymore:
>
> you just can do this anymore:
>
> file bqschema.py with
>
> def marketing_segment():
> return [
> {"name": "user_id", "type": "integer", "mode": "nullable"},
> {"name": "bucket_date", "type": "timestamp", "mode": "nullable"},
> {"name": "segment_main", "type": "string", "mode": "nullable"},
> {"name": "segment_sub", "type": "integer", "mode": "nullable"},
>
>
> In marketing_segmentation.py:
>
>
> import bqschema
>
> Gives an error:
>
> Traceback (most recent call last):
> File
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+
> apache.incubating-py2.7.egg/airflow/models.py",
> line 264, in process_file
> m = imp.load_source(mod_name, filepath)
> File "/home/airflow/dags/marketing_segmentation.py", line 17, in
> <module>
> import bqschema
> ImportError: No module named bqschema
>
> *I don't think this is incorrect?!*
>
>
>
> On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov <dan.davydov@airbnb.com.
> invalid>
> wrote:
>
> > The latest commit fixed a regression since 1.7 that files with parsing
> > errors no longer showed up on the UI.
> >
> > On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel <al...@vanboxel.be>
> wrote:
> >
> > > Just installed beta 5 on our dev environment it lighted up as a
> christmas
> > > tree. I got a a screen full of import errors. I see that the latest
> > commit
> > > did something with import errors... is it coorect?!
> > >
> > > On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin <bd...@gmail.com>
> > wrote:
> > >
> > > > Hey Boris
> > > >
> > > > The scheduler is a bit more aggressive and can use multiple
> processors,
> > > so
> > > > higher CPU usage is actually a good thing.
> > > >
> > > > I case it is really out of hand look at the new scheduler options and
> > > > heartbeat options (see PR for updating.md not in the beta yet).
> > > >
> > > > Bolke
> > > >
> > > > Sent from my iPhone
> > > >
> > > > > On 29 Jan 2017, at 15:35, Boris Tyukin <bo...@boristyukin.com>
> > wrote:
> > > > >
> > > > > I am not sure if it is my config or something, but looks like after
> > the
> > > > > upgrade and start of scheduler, airflow would totally hose CPU. The
> > > > reason
> > > > > is two new examples that start running right away - latest only and
> > > > latest
> > > > > with trigger. Once I pause them, CPU goes back to idle. Is this
> > because
> > > > now
> > > > > dags are not paused by default like it was before?
> > > > >
> > > > > As I mentioned before, I also had to upgrade mysql to 5.7 - if
> > someone
> > > > > needs a step by step instruction, make sure to follow all steps
> > > precisely
> > > > > here for in-place upgrade or you will have heck of the time (like
> > me).
> > > > >
> > > > https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#
> > > upgrade-procedure-inplace
> > > > >
> > > > > BTW official Oracle repository for Oracle Linux only has MySql 5.6
> -
> > > for
> > > > > 5.7 you have to use MySql community repo.
> > > > >
> > > > >> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin <
> bdbruin@gmail.com
> > >
> > > > wrote:
> > > > >>
> > > > >> Hi All,
> > > > >>
> > > > >> I have made the FIFTH beta of Airflow 1.8.0 available at:
> > > > >> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> > > > >> https://dist.apache.org/repos/dist/dev/incubator/airflow/> ,
> public
> > > > keys
> > > > >> are available at https://dist.apache.org/repos/
> > > dist/release/incubator/
> > > > >> airflow/ <https://dist.apache.org/repos/dist/release/incubator/
> > > airflow/
> > > > >
> > > > >> . It is tagged with a local version “apache.incubating” so it
> allows
> > > > >> upgrading from earlier releases.
> > > > >>
> > > > >> Issues fixed:
> > > > >> * Parsing errors not showing up in UI fixing a regression**
> > > > >> * Scheduler would terminate immediately if no dag files present
> > > > >>
> > > > >> ** As this touches the scheduler logic I though it warranted
> another
> > > > beta.
> > > > >>
> > > > >> This should be the last beta in my opinion and we can prepare
> > > changelog,
> > > > >> upgrade notes and release notes for the RC (Feb 2).
> > > > >>
> > > > >> Cheers
> > > > >> Bolke
> > > >
> > > --
> > > _/
> > > _/ Alex Van Boxel
> > >
> >
> --
> _/
> _/ Alex Van Boxel
>
Re: Airflow 1.8.0 BETA 5
Posted by Alex Van Boxel <al...@vanboxel.be>.
Well this means none of my DAG's work anymore:
you just can do this anymore:
file bqschema.py with
def marketing_segment():
return [
{"name": "user_id", "type": "integer", "mode": "nullable"},
{"name": "bucket_date", "type": "timestamp", "mode": "nullable"},
{"name": "segment_main", "type": "string", "mode": "nullable"},
{"name": "segment_sub", "type": "integer", "mode": "nullable"},
In marketing_segmentation.py:
import bqschema
Gives an error:
Traceback (most recent call last):
File
"/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+apache.incubating-py2.7.egg/airflow/models.py",
line 264, in process_file
m = imp.load_source(mod_name, filepath)
File "/home/airflow/dags/marketing_segmentation.py", line 17, in <module>
import bqschema
ImportError: No module named bqschema
*I don't think this is incorrect?!*
On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov <da...@airbnb.com.invalid>
wrote:
> The latest commit fixed a regression since 1.7 that files with parsing
> errors no longer showed up on the UI.
>
> On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel <al...@vanboxel.be> wrote:
>
> > Just installed beta 5 on our dev environment it lighted up as a christmas
> > tree. I got a a screen full of import errors. I see that the latest
> commit
> > did something with import errors... is it coorect?!
> >
> > On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin <bd...@gmail.com>
> wrote:
> >
> > > Hey Boris
> > >
> > > The scheduler is a bit more aggressive and can use multiple processors,
> > so
> > > higher CPU usage is actually a good thing.
> > >
> > > I case it is really out of hand look at the new scheduler options and
> > > heartbeat options (see PR for updating.md not in the beta yet).
> > >
> > > Bolke
> > >
> > > Sent from my iPhone
> > >
> > > > On 29 Jan 2017, at 15:35, Boris Tyukin <bo...@boristyukin.com>
> wrote:
> > > >
> > > > I am not sure if it is my config or something, but looks like after
> the
> > > > upgrade and start of scheduler, airflow would totally hose CPU. The
> > > reason
> > > > is two new examples that start running right away - latest only and
> > > latest
> > > > with trigger. Once I pause them, CPU goes back to idle. Is this
> because
> > > now
> > > > dags are not paused by default like it was before?
> > > >
> > > > As I mentioned before, I also had to upgrade mysql to 5.7 - if
> someone
> > > > needs a step by step instruction, make sure to follow all steps
> > precisely
> > > > here for in-place upgrade or you will have heck of the time (like
> me).
> > > >
> > > https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#
> > upgrade-procedure-inplace
> > > >
> > > > BTW official Oracle repository for Oracle Linux only has MySql 5.6 -
> > for
> > > > 5.7 you have to use MySql community repo.
> > > >
> > > >> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin <bdbruin@gmail.com
> >
> > > wrote:
> > > >>
> > > >> Hi All,
> > > >>
> > > >> I have made the FIFTH beta of Airflow 1.8.0 available at:
> > > >> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> > > >> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public
> > > keys
> > > >> are available at https://dist.apache.org/repos/
> > dist/release/incubator/
> > > >> airflow/ <https://dist.apache.org/repos/dist/release/incubator/
> > airflow/
> > > >
> > > >> . It is tagged with a local version “apache.incubating” so it allows
> > > >> upgrading from earlier releases.
> > > >>
> > > >> Issues fixed:
> > > >> * Parsing errors not showing up in UI fixing a regression**
> > > >> * Scheduler would terminate immediately if no dag files present
> > > >>
> > > >> ** As this touches the scheduler logic I though it warranted another
> > > beta.
> > > >>
> > > >> This should be the last beta in my opinion and we can prepare
> > changelog,
> > > >> upgrade notes and release notes for the RC (Feb 2).
> > > >>
> > > >> Cheers
> > > >> Bolke
> > >
> > --
> > _/
> > _/ Alex Van Boxel
> >
>
--
_/
_/ Alex Van Boxel
Re: Airflow 1.8.0 BETA 5
Posted by Chris Riccomini <cr...@apache.org>.
We're upgraded to b5 in dev/prod as of an hour ago. Will keep you posted.
On Mon, Jan 30, 2017 at 2:45 PM, Dan Davydov <dan.davydov@airbnb.com.invalid
> wrote:
> The latest commit fixed a regression since 1.7 that files with parsing
> errors no longer showed up on the UI.
>
> On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel <al...@vanboxel.be> wrote:
>
> > Just installed beta 5 on our dev environment it lighted up as a christmas
> > tree. I got a a screen full of import errors. I see that the latest
> commit
> > did something with import errors... is it coorect?!
> >
> > On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin <bd...@gmail.com>
> wrote:
> >
> > > Hey Boris
> > >
> > > The scheduler is a bit more aggressive and can use multiple processors,
> > so
> > > higher CPU usage is actually a good thing.
> > >
> > > I case it is really out of hand look at the new scheduler options and
> > > heartbeat options (see PR for updating.md not in the beta yet).
> > >
> > > Bolke
> > >
> > > Sent from my iPhone
> > >
> > > > On 29 Jan 2017, at 15:35, Boris Tyukin <bo...@boristyukin.com>
> wrote:
> > > >
> > > > I am not sure if it is my config or something, but looks like after
> the
> > > > upgrade and start of scheduler, airflow would totally hose CPU. The
> > > reason
> > > > is two new examples that start running right away - latest only and
> > > latest
> > > > with trigger. Once I pause them, CPU goes back to idle. Is this
> because
> > > now
> > > > dags are not paused by default like it was before?
> > > >
> > > > As I mentioned before, I also had to upgrade mysql to 5.7 - if
> someone
> > > > needs a step by step instruction, make sure to follow all steps
> > precisely
> > > > here for in-place upgrade or you will have heck of the time (like
> me).
> > > >
> > > https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#
> > upgrade-procedure-inplace
> > > >
> > > > BTW official Oracle repository for Oracle Linux only has MySql 5.6 -
> > for
> > > > 5.7 you have to use MySql community repo.
> > > >
> > > >> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin <bdbruin@gmail.com
> >
> > > wrote:
> > > >>
> > > >> Hi All,
> > > >>
> > > >> I have made the FIFTH beta of Airflow 1.8.0 available at:
> > > >> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> > > >> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public
> > > keys
> > > >> are available at https://dist.apache.org/repos/
> > dist/release/incubator/
> > > >> airflow/ <https://dist.apache.org/repos/dist/release/incubator/
> > airflow/
> > > >
> > > >> . It is tagged with a local version “apache.incubating” so it allows
> > > >> upgrading from earlier releases.
> > > >>
> > > >> Issues fixed:
> > > >> * Parsing errors not showing up in UI fixing a regression**
> > > >> * Scheduler would terminate immediately if no dag files present
> > > >>
> > > >> ** As this touches the scheduler logic I though it warranted another
> > > beta.
> > > >>
> > > >> This should be the last beta in my opinion and we can prepare
> > changelog,
> > > >> upgrade notes and release notes for the RC (Feb 2).
> > > >>
> > > >> Cheers
> > > >> Bolke
> > >
> > --
> > _/
> > _/ Alex Van Boxel
> >
>
Re: Airflow 1.8.0 BETA 5
Posted by Dan Davydov <da...@airbnb.com.INVALID>.
The latest commit fixed a regression since 1.7 that files with parsing
errors no longer showed up on the UI.
On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel <al...@vanboxel.be> wrote:
> Just installed beta 5 on our dev environment it lighted up as a christmas
> tree. I got a a screen full of import errors. I see that the latest commit
> did something with import errors... is it coorect?!
>
> On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin <bd...@gmail.com> wrote:
>
> > Hey Boris
> >
> > The scheduler is a bit more aggressive and can use multiple processors,
> so
> > higher CPU usage is actually a good thing.
> >
> > I case it is really out of hand look at the new scheduler options and
> > heartbeat options (see PR for updating.md not in the beta yet).
> >
> > Bolke
> >
> > Sent from my iPhone
> >
> > > On 29 Jan 2017, at 15:35, Boris Tyukin <bo...@boristyukin.com> wrote:
> > >
> > > I am not sure if it is my config or something, but looks like after the
> > > upgrade and start of scheduler, airflow would totally hose CPU. The
> > reason
> > > is two new examples that start running right away - latest only and
> > latest
> > > with trigger. Once I pause them, CPU goes back to idle. Is this because
> > now
> > > dags are not paused by default like it was before?
> > >
> > > As I mentioned before, I also had to upgrade mysql to 5.7 - if someone
> > > needs a step by step instruction, make sure to follow all steps
> precisely
> > > here for in-place upgrade or you will have heck of the time (like me).
> > >
> > https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#
> upgrade-procedure-inplace
> > >
> > > BTW official Oracle repository for Oracle Linux only has MySql 5.6 -
> for
> > > 5.7 you have to use MySql community repo.
> > >
> > >> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin <bd...@gmail.com>
> > wrote:
> > >>
> > >> Hi All,
> > >>
> > >> I have made the FIFTH beta of Airflow 1.8.0 available at:
> > >> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> > >> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public
> > keys
> > >> are available at https://dist.apache.org/repos/
> dist/release/incubator/
> > >> airflow/ <https://dist.apache.org/repos/dist/release/incubator/
> airflow/
> > >
> > >> . It is tagged with a local version “apache.incubating” so it allows
> > >> upgrading from earlier releases.
> > >>
> > >> Issues fixed:
> > >> * Parsing errors not showing up in UI fixing a regression**
> > >> * Scheduler would terminate immediately if no dag files present
> > >>
> > >> ** As this touches the scheduler logic I though it warranted another
> > beta.
> > >>
> > >> This should be the last beta in my opinion and we can prepare
> changelog,
> > >> upgrade notes and release notes for the RC (Feb 2).
> > >>
> > >> Cheers
> > >> Bolke
> >
> --
> _/
> _/ Alex Van Boxel
>
Re: Airflow 1.8.0 BETA 5
Posted by Alex Van Boxel <al...@vanboxel.be>.
Just installed beta 5 on our dev environment it lighted up as a christmas
tree. I got a a screen full of import errors. I see that the latest commit
did something with import errors... is it coorect?!
On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin <bd...@gmail.com> wrote:
> Hey Boris
>
> The scheduler is a bit more aggressive and can use multiple processors, so
> higher CPU usage is actually a good thing.
>
> I case it is really out of hand look at the new scheduler options and
> heartbeat options (see PR for updating.md not in the beta yet).
>
> Bolke
>
> Sent from my iPhone
>
> > On 29 Jan 2017, at 15:35, Boris Tyukin <bo...@boristyukin.com> wrote:
> >
> > I am not sure if it is my config or something, but looks like after the
> > upgrade and start of scheduler, airflow would totally hose CPU. The
> reason
> > is two new examples that start running right away - latest only and
> latest
> > with trigger. Once I pause them, CPU goes back to idle. Is this because
> now
> > dags are not paused by default like it was before?
> >
> > As I mentioned before, I also had to upgrade mysql to 5.7 - if someone
> > needs a step by step instruction, make sure to follow all steps precisely
> > here for in-place upgrade or you will have heck of the time (like me).
> >
> https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#upgrade-procedure-inplace
> >
> > BTW official Oracle repository for Oracle Linux only has MySql 5.6 - for
> > 5.7 you have to use MySql community repo.
> >
> >> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin <bd...@gmail.com>
> wrote:
> >>
> >> Hi All,
> >>
> >> I have made the FIFTH beta of Airflow 1.8.0 available at:
> >> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> >> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public
> keys
> >> are available at https://dist.apache.org/repos/dist/release/incubator/
> >> airflow/ <https://dist.apache.org/repos/dist/release/incubator/airflow/
> >
> >> . It is tagged with a local version “apache.incubating” so it allows
> >> upgrading from earlier releases.
> >>
> >> Issues fixed:
> >> * Parsing errors not showing up in UI fixing a regression**
> >> * Scheduler would terminate immediately if no dag files present
> >>
> >> ** As this touches the scheduler logic I though it warranted another
> beta.
> >>
> >> This should be the last beta in my opinion and we can prepare changelog,
> >> upgrade notes and release notes for the RC (Feb 2).
> >>
> >> Cheers
> >> Bolke
>
--
_/
_/ Alex Van Boxel
Re: Airflow 1.8.0 BETA 5
Posted by Bolke de Bruin <bd...@gmail.com>.
Hey Boris
The scheduler is a bit more aggressive and can use multiple processors, so higher CPU usage is actually a good thing.
I case it is really out of hand look at the new scheduler options and heartbeat options (see PR for updating.md not in the beta yet).
Bolke
Sent from my iPhone
> On 29 Jan 2017, at 15:35, Boris Tyukin <bo...@boristyukin.com> wrote:
>
> I am not sure if it is my config or something, but looks like after the
> upgrade and start of scheduler, airflow would totally hose CPU. The reason
> is two new examples that start running right away - latest only and latest
> with trigger. Once I pause them, CPU goes back to idle. Is this because now
> dags are not paused by default like it was before?
>
> As I mentioned before, I also had to upgrade mysql to 5.7 - if someone
> needs a step by step instruction, make sure to follow all steps precisely
> here for in-place upgrade or you will have heck of the time (like me).
> https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#upgrade-procedure-inplace
>
> BTW official Oracle repository for Oracle Linux only has MySql 5.6 - for
> 5.7 you have to use MySql community repo.
>
>> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin <bd...@gmail.com> wrote:
>>
>> Hi All,
>>
>> I have made the FIFTH beta of Airflow 1.8.0 available at:
>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public keys
>> are available at https://dist.apache.org/repos/dist/release/incubator/
>> airflow/ <https://dist.apache.org/repos/dist/release/incubator/airflow/>
>> . It is tagged with a local version “apache.incubating” so it allows
>> upgrading from earlier releases.
>>
>> Issues fixed:
>> * Parsing errors not showing up in UI fixing a regression**
>> * Scheduler would terminate immediately if no dag files present
>>
>> ** As this touches the scheduler logic I though it warranted another beta.
>>
>> This should be the last beta in my opinion and we can prepare changelog,
>> upgrade notes and release notes for the RC (Feb 2).
>>
>> Cheers
>> Bolke
Re: Airflow 1.8.0 BETA 5
Posted by Boris Tyukin <bo...@boristyukin.com>.
I am not sure if it is my config or something, but looks like after the
upgrade and start of scheduler, airflow would totally hose CPU. The reason
is two new examples that start running right away - latest only and latest
with trigger. Once I pause them, CPU goes back to idle. Is this because now
dags are not paused by default like it was before?
As I mentioned before, I also had to upgrade mysql to 5.7 - if someone
needs a step by step instruction, make sure to follow all steps precisely
here for in-place upgrade or you will have heck of the time (like me).
https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#upgrade-procedure-inplace
BTW official Oracle repository for Oracle Linux only has MySql 5.6 - for
5.7 you have to use MySql community repo.
On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin <bd...@gmail.com> wrote:
> Hi All,
>
> I have made the FIFTH beta of Airflow 1.8.0 available at:
> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public keys
> are available at https://dist.apache.org/repos/dist/release/incubator/
> airflow/ <https://dist.apache.org/repos/dist/release/incubator/airflow/>
> . It is tagged with a local version “apache.incubating” so it allows
> upgrading from earlier releases.
>
> Issues fixed:
> * Parsing errors not showing up in UI fixing a regression**
> * Scheduler would terminate immediately if no dag files present
>
> ** As this touches the scheduler logic I though it warranted another beta.
>
> This should be the last beta in my opinion and we can prepare changelog,
> upgrade notes and release notes for the RC (Feb 2).
>
> Cheers
> Bolke