You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Bolke de Bruin <bd...@gmail.com> on 2017/01/26 18:49:38 UTC
Airflow 1.8.0 BETA 4
Hi All,
I have made the FOURTH beta of Airflow 1.8.0 available at: https://dist.apache.org/repos/dist/dev/incubator/airflow/ <https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public keys are available at https://dist.apache.org/repos/dist/release/incubator/airflow/ <https://dist.apache.org/repos/dist/release/incubator/airflow/> . It is tagged with a local version “apache.incubating” so it allows upgrading from earlier releases. This beta is available for testing in a more production like setting (acceptance environment?).
I would like to encourage everyone to try it out, to report back any issues so we get to a rock solid release of 1.8.0. When reporting issues a test case or even a fix is highly appreciated.
Issues fixed:
* Incorrect Alembic reference due to revert (initdb/upgradedb/resetdb should work again)
* Py3 incompatibility in base_taskrunner.
Under investigation:
* DAG marked success, with half of the Tasks never scheduled (Alex)
Kind regards,
Bolke
Re: Airflow 1.8.0 BETA 4
Posted by Chris Riccomini <cr...@apache.org>.
Any more details on this one?
* DAG marked success, with half of the Tasks never scheduled (Alex)
On Thu, Jan 26, 2017 at 10:49 AM, Bolke de Bruin <bd...@gmail.com> wrote:
> Hi All,
>
> I have made the FOURTH beta of Airflow 1.8.0 available at:
> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public keys
> are available at https://dist.apache.org/repos/dist/release/incubator/
> airflow/ <https://dist.apache.org/repos/dist/release/incubator/airflow/>
> . It is tagged with a local version “apache.incubating” so it allows
> upgrading from earlier releases. This beta is available for testing in a
> more production like setting (acceptance environment?).
>
> I would like to encourage everyone to try it out, to report back any
> issues so we get to a rock solid release of 1.8.0. When reporting issues a
> test case or even a fix is highly appreciated.
>
> Issues fixed:
> * Incorrect Alembic reference due to revert (initdb/upgradedb/resetdb
> should work again)
> * Py3 incompatibility in base_taskrunner.
>
> Under investigation:
> * DAG marked success, with half of the Tasks never scheduled (Alex)
>
> Kind regards,
> Bolke
Re: Airflow 1.8.0 BETA 4
Posted by Alex Van Boxel <al...@vanboxel.be>.
I'm also abandoning the follow problem:
* DAG marked success, with half of the Tasks never scheduled (Alex)
I *can't simulate it locally*... I'll keep an eye on the production runs
though.
On Fri, Jan 27, 2017 at 8:45 PM Bolke de Bruin <bd...@gmail.com> wrote:
> 5.6.4 is the minimum. Yes it was required. updating.md was required.
> 5.6.4 is already really really old. Most distro's replaced it with mariadb
> which is compatible.
>
> Bolke
>
> Sent from my iPhone
>
> > On 27 Jan 2017, at 20:19, Boris Tyukin <bo...@boristyukin.com> wrote:
> >
> > sorry mysql version i have is mysql Ver 14.14 Distrib 5.1.73
> >
> > I see that DATETIME(6) is only supported on MySQL 5.7 and later.
> >
> > Was it really necessary to do use DATETIME(6) not just DATETIME?
> >
> > Probably should be mention in change doc that this requires an upgrade of
> > mysql which in my opinion is not cool. epel repo only has 5.1.73 as
> latest
> > version and CDH distro includes 5.1.73 as well.
> >
> > Some companies might not allow an upgrade of mysql unless it is in one of
> > the official linux repos.
> >
> >
> >> On Fri, Jan 27, 2017 at 2:14 PM, Boris Tyukin <bo...@boristyukin.com>
> wrote:
> >>
> >> was requirement to MySQL database version changes with 1.8.0? I run
> mysql
> >> 14.14 and it worked fine with 1.7.
> >>
> >> I just installed 1.8.4 beta 4 and got this error below when i ran
> airflow
> >> upgradedb command
> >>
> >> File "/usr/local/lib/python2.7/site-packages/MySQLdb/cursors.py", line
> >> 205, in execute
> >> self.errorhandler(self, exc, value)
> >> File "/usr/local/lib/python2.7/site-packages/MySQLdb/connections.py",
> >> line 36, in defaulterrorhandler
> >> raise errorclass, errorvalue
> >> sqlalchemy.exc.ProgrammingError: (_mysql_exceptions.ProgrammingError)
> >> (1064, "You have an error in your SQL syntax; check the manual that
> >> corresponds to your MySQL server version for the right syntax to use
> near
> >> '(6) NULL' at line 1") [SQL: u'ALTER TABLE dag MODIFY last_scheduler_run
> >> DATETIME(6) NULL']
> >>
> >>
> >>
> >>> On Thu, Jan 26, 2017 at 1:49 PM, Bolke de Bruin <bd...@gmail.com>
> wrote:
> >>>
> >>> Hi All,
> >>>
> >>> I have made the FOURTH beta of Airflow 1.8.0 available at:
> >>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> >>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public
> keys
> >>> are available at https://dist.apache.org/repos/
> >>> dist/release/incubator/airflow/ <https://dist.apache.org/repos
> >>> /dist/release/incubator/airflow/> . It is tagged with a local version
> >>> “apache.incubating” so it allows upgrading from earlier releases. This
> beta
> >>> is available for testing in a more production like setting (acceptance
> >>> environment?).
> >>>
> >>> I would like to encourage everyone to try it out, to report back any
> >>> issues so we get to a rock solid release of 1.8.0. When reporting
> issues a
> >>> test case or even a fix is highly appreciated.
> >>>
> >>> Issues fixed:
> >>> * Incorrect Alembic reference due to revert (initdb/upgradedb/resetdb
> >>> should work again)
> >>> * Py3 incompatibility in base_taskrunner.
> >>>
> >>> Under investigation:
> >>> * DAG marked success, with half of the Tasks never scheduled (Alex)
> >>>
> >>> Kind regards,
> >>> Bolke
> >>
> >>
> >>
>
--
_/
_/ Alex Van Boxel
Re: Airflow 1.8.0 BETA 4
Posted by Bolke de Bruin <bd...@gmail.com>.
And, I need check my spelling and grammar more often :-).
> On 27 Jan 2017, at 22:09, Bolke de Bruin <bd...@gmail.com> wrote:
>
> Big enterprise here so I know your pain. It will be in the release docs. I don’t think we can put in into “pip install” as a check. If you have suggestions there that would be appreciated.
>
> Mariadb supports it since 5.3 and is available as 5.5.X in epel 7. Its a drop in replacement and should work for Airflow. Another option is to use Postgres (I understand that that ‘upgrading’ to Postgres I a bit more difficult).
>
> Bolke
>
>
>
>> On 27 Jan 2017, at 22:00, Boris Tyukin <bo...@boristyukin.com> wrote:
>>
>> yeah i hear you but you know how these big enterprise companies behind on
>> everything. official epel repo only has 5.1.73, same with Oracle linux
>> repo. We have to use normally only officially approved repos. Looks like
>> airflow 1.7.1 worked fine with 5.1.73 and now it does not.
>>
>> Not a big deal but I still do not see where it was mentioned in
>> documentation - pip installed 1.8.0 without any complains as well.
>>
>> On Fri, Jan 27, 2017 at 3:41 PM, Bolke de Bruin <bd...@gmail.com> wrote:
>>
>>> BTW:
>>>
>>> * MySQL 5.6.10 (5.6.4 was a beta I just figured out - released in 2011)
>>> was released in 2013
>>> * MySQL 5.1.73 was released on 2013-12-04
>>>
>>> We are really not ahead of the pack with our requirements.
>>>
>>> Bolke
>>>
>>>> On 27 Jan 2017, at 21:32, Boris Tyukin <bo...@boristyukin.com> wrote:
>>>>
>>>> thanks for confirmation Bolke
>>>>
>>>> On Fri, Jan 27, 2017 at 2:45 PM, Bolke de Bruin <bd...@gmail.com>
>>> wrote:
>>>>
>>>>> 5.6.4 is the minimum. Yes it was required. updating.md was required.
>>>>> 5.6.4 is already really really old. Most distro's replaced it with
>>> mariadb
>>>>> which is compatible.
>>>>>
>>>>> Bolke
>>>>>
>>>>> Sent from my iPhone
>>>>>
>>>>>> On 27 Jan 2017, at 20:19, Boris Tyukin <bo...@boristyukin.com> wrote:
>>>>>>
>>>>>> sorry mysql version i have is mysql Ver 14.14 Distrib 5.1.73
>>>>>>
>>>>>> I see that DATETIME(6) is only supported on MySQL 5.7 and later.
>>>>>>
>>>>>> Was it really necessary to do use DATETIME(6) not just DATETIME?
>>>>>>
>>>>>> Probably should be mention in change doc that this requires an upgrade
>>> of
>>>>>> mysql which in my opinion is not cool. epel repo only has 5.1.73 as
>>>>> latest
>>>>>> version and CDH distro includes 5.1.73 as well.
>>>>>>
>>>>>> Some companies might not allow an upgrade of mysql unless it is in one
>>> of
>>>>>> the official linux repos.
>>>>>>
>>>>>>
>>>>>>> On Fri, Jan 27, 2017 at 2:14 PM, Boris Tyukin <bo...@boristyukin.com>
>>>>> wrote:
>>>>>>>
>>>>>>> was requirement to MySQL database version changes with 1.8.0? I run
>>>>> mysql
>>>>>>> 14.14 and it worked fine with 1.7.
>>>>>>>
>>>>>>> I just installed 1.8.4 beta 4 and got this error below when i ran
>>>>> airflow
>>>>>>> upgradedb command
>>>>>>>
>>>>>>> File "/usr/local/lib/python2.7/site-packages/MySQLdb/cursors.py",
>>> line
>>>>>>> 205, in execute
>>>>>>> self.errorhandler(self, exc, value)
>>>>>>> File "/usr/local/lib/python2.7/site-packages/MySQLdb/connections.py",
>>>>>>> line 36, in defaulterrorhandler
>>>>>>> raise errorclass, errorvalue
>>>>>>> sqlalchemy.exc.ProgrammingError: (_mysql_exceptions.ProgrammingError)
>>>>>>> (1064, "You have an error in your SQL syntax; check the manual that
>>>>>>> corresponds to your MySQL server version for the right syntax to use
>>>>> near
>>>>>>> '(6) NULL' at line 1") [SQL: u'ALTER TABLE dag MODIFY
>>> last_scheduler_run
>>>>>>> DATETIME(6) NULL']
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On Thu, Jan 26, 2017 at 1:49 PM, Bolke de Bruin <bd...@gmail.com>
>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> I have made the FOURTH beta of Airflow 1.8.0 available at:
>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public
>>>>> keys
>>>>>>>> are available at https://dist.apache.org/repos/
>>>>>>>> dist/release/incubator/airflow/ <https://dist.apache.org/repos
>>>>>>>> /dist/release/incubator/airflow/> . It is tagged with a local
>>> version
>>>>>>>> “apache.incubating” so it allows upgrading from earlier releases.
>>> This
>>>>> beta
>>>>>>>> is available for testing in a more production like setting
>>> (acceptance
>>>>>>>> environment?).
>>>>>>>>
>>>>>>>> I would like to encourage everyone to try it out, to report back any
>>>>>>>> issues so we get to a rock solid release of 1.8.0. When reporting
>>>>> issues a
>>>>>>>> test case or even a fix is highly appreciated.
>>>>>>>>
>>>>>>>> Issues fixed:
>>>>>>>> * Incorrect Alembic reference due to revert (initdb/upgradedb/resetdb
>>>>>>>> should work again)
>>>>>>>> * Py3 incompatibility in base_taskrunner.
>>>>>>>>
>>>>>>>> Under investigation:
>>>>>>>> * DAG marked success, with half of the Tasks never scheduled (Alex)
>>>>>>>>
>>>>>>>> Kind regards,
>>>>>>>> Bolke
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>
>>>
>
Re: Airflow 1.8.0 BETA 4
Posted by Bolke de Bruin <bd...@gmail.com>.
Big enterprise here so I know your pain. It will be in the release docs. I don’t think we can put in into “pip install” as a check. If you have suggestions there that would be appreciated.
Mariadb supports it since 5.3 and is available as 5.5.X in epel 7. Its a drop in replacement and should work for Airflow. Another option is to use Postgres (I understand that that ‘upgrading’ to Postgres I a bit more difficult).
Bolke
> On 27 Jan 2017, at 22:00, Boris Tyukin <bo...@boristyukin.com> wrote:
>
> yeah i hear you but you know how these big enterprise companies behind on
> everything. official epel repo only has 5.1.73, same with Oracle linux
> repo. We have to use normally only officially approved repos. Looks like
> airflow 1.7.1 worked fine with 5.1.73 and now it does not.
>
> Not a big deal but I still do not see where it was mentioned in
> documentation - pip installed 1.8.0 without any complains as well.
>
> On Fri, Jan 27, 2017 at 3:41 PM, Bolke de Bruin <bd...@gmail.com> wrote:
>
>> BTW:
>>
>> * MySQL 5.6.10 (5.6.4 was a beta I just figured out - released in 2011)
>> was released in 2013
>> * MySQL 5.1.73 was released on 2013-12-04
>>
>> We are really not ahead of the pack with our requirements.
>>
>> Bolke
>>
>>> On 27 Jan 2017, at 21:32, Boris Tyukin <bo...@boristyukin.com> wrote:
>>>
>>> thanks for confirmation Bolke
>>>
>>> On Fri, Jan 27, 2017 at 2:45 PM, Bolke de Bruin <bd...@gmail.com>
>> wrote:
>>>
>>>> 5.6.4 is the minimum. Yes it was required. updating.md was required.
>>>> 5.6.4 is already really really old. Most distro's replaced it with
>> mariadb
>>>> which is compatible.
>>>>
>>>> Bolke
>>>>
>>>> Sent from my iPhone
>>>>
>>>>> On 27 Jan 2017, at 20:19, Boris Tyukin <bo...@boristyukin.com> wrote:
>>>>>
>>>>> sorry mysql version i have is mysql Ver 14.14 Distrib 5.1.73
>>>>>
>>>>> I see that DATETIME(6) is only supported on MySQL 5.7 and later.
>>>>>
>>>>> Was it really necessary to do use DATETIME(6) not just DATETIME?
>>>>>
>>>>> Probably should be mention in change doc that this requires an upgrade
>> of
>>>>> mysql which in my opinion is not cool. epel repo only has 5.1.73 as
>>>> latest
>>>>> version and CDH distro includes 5.1.73 as well.
>>>>>
>>>>> Some companies might not allow an upgrade of mysql unless it is in one
>> of
>>>>> the official linux repos.
>>>>>
>>>>>
>>>>>> On Fri, Jan 27, 2017 at 2:14 PM, Boris Tyukin <bo...@boristyukin.com>
>>>> wrote:
>>>>>>
>>>>>> was requirement to MySQL database version changes with 1.8.0? I run
>>>> mysql
>>>>>> 14.14 and it worked fine with 1.7.
>>>>>>
>>>>>> I just installed 1.8.4 beta 4 and got this error below when i ran
>>>> airflow
>>>>>> upgradedb command
>>>>>>
>>>>>> File "/usr/local/lib/python2.7/site-packages/MySQLdb/cursors.py",
>> line
>>>>>> 205, in execute
>>>>>> self.errorhandler(self, exc, value)
>>>>>> File "/usr/local/lib/python2.7/site-packages/MySQLdb/connections.py",
>>>>>> line 36, in defaulterrorhandler
>>>>>> raise errorclass, errorvalue
>>>>>> sqlalchemy.exc.ProgrammingError: (_mysql_exceptions.ProgrammingError)
>>>>>> (1064, "You have an error in your SQL syntax; check the manual that
>>>>>> corresponds to your MySQL server version for the right syntax to use
>>>> near
>>>>>> '(6) NULL' at line 1") [SQL: u'ALTER TABLE dag MODIFY
>> last_scheduler_run
>>>>>> DATETIME(6) NULL']
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Thu, Jan 26, 2017 at 1:49 PM, Bolke de Bruin <bd...@gmail.com>
>>>> wrote:
>>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> I have made the FOURTH beta of Airflow 1.8.0 available at:
>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public
>>>> keys
>>>>>>> are available at https://dist.apache.org/repos/
>>>>>>> dist/release/incubator/airflow/ <https://dist.apache.org/repos
>>>>>>> /dist/release/incubator/airflow/> . It is tagged with a local
>> version
>>>>>>> “apache.incubating” so it allows upgrading from earlier releases.
>> This
>>>> beta
>>>>>>> is available for testing in a more production like setting
>> (acceptance
>>>>>>> environment?).
>>>>>>>
>>>>>>> I would like to encourage everyone to try it out, to report back any
>>>>>>> issues so we get to a rock solid release of 1.8.0. When reporting
>>>> issues a
>>>>>>> test case or even a fix is highly appreciated.
>>>>>>>
>>>>>>> Issues fixed:
>>>>>>> * Incorrect Alembic reference due to revert (initdb/upgradedb/resetdb
>>>>>>> should work again)
>>>>>>> * Py3 incompatibility in base_taskrunner.
>>>>>>>
>>>>>>> Under investigation:
>>>>>>> * DAG marked success, with half of the Tasks never scheduled (Alex)
>>>>>>>
>>>>>>> Kind regards,
>>>>>>> Bolke
>>>>>>
>>>>>>
>>>>>>
>>>>
>>
>>
Re: Airflow 1.8.0 BETA 4
Posted by Boris Tyukin <bo...@boristyukin.com>.
yeah i hear you but you know how these big enterprise companies behind on
everything. official epel repo only has 5.1.73, same with Oracle linux
repo. We have to use normally only officially approved repos. Looks like
airflow 1.7.1 worked fine with 5.1.73 and now it does not.
Not a big deal but I still do not see where it was mentioned in
documentation - pip installed 1.8.0 without any complains as well.
On Fri, Jan 27, 2017 at 3:41 PM, Bolke de Bruin <bd...@gmail.com> wrote:
> BTW:
>
> * MySQL 5.6.10 (5.6.4 was a beta I just figured out - released in 2011)
> was released in 2013
> * MySQL 5.1.73 was released on 2013-12-04
>
> We are really not ahead of the pack with our requirements.
>
> Bolke
>
> > On 27 Jan 2017, at 21:32, Boris Tyukin <bo...@boristyukin.com> wrote:
> >
> > thanks for confirmation Bolke
> >
> > On Fri, Jan 27, 2017 at 2:45 PM, Bolke de Bruin <bd...@gmail.com>
> wrote:
> >
> >> 5.6.4 is the minimum. Yes it was required. updating.md was required.
> >> 5.6.4 is already really really old. Most distro's replaced it with
> mariadb
> >> which is compatible.
> >>
> >> Bolke
> >>
> >> Sent from my iPhone
> >>
> >>> On 27 Jan 2017, at 20:19, Boris Tyukin <bo...@boristyukin.com> wrote:
> >>>
> >>> sorry mysql version i have is mysql Ver 14.14 Distrib 5.1.73
> >>>
> >>> I see that DATETIME(6) is only supported on MySQL 5.7 and later.
> >>>
> >>> Was it really necessary to do use DATETIME(6) not just DATETIME?
> >>>
> >>> Probably should be mention in change doc that this requires an upgrade
> of
> >>> mysql which in my opinion is not cool. epel repo only has 5.1.73 as
> >> latest
> >>> version and CDH distro includes 5.1.73 as well.
> >>>
> >>> Some companies might not allow an upgrade of mysql unless it is in one
> of
> >>> the official linux repos.
> >>>
> >>>
> >>>> On Fri, Jan 27, 2017 at 2:14 PM, Boris Tyukin <bo...@boristyukin.com>
> >> wrote:
> >>>>
> >>>> was requirement to MySQL database version changes with 1.8.0? I run
> >> mysql
> >>>> 14.14 and it worked fine with 1.7.
> >>>>
> >>>> I just installed 1.8.4 beta 4 and got this error below when i ran
> >> airflow
> >>>> upgradedb command
> >>>>
> >>>> File "/usr/local/lib/python2.7/site-packages/MySQLdb/cursors.py",
> line
> >>>> 205, in execute
> >>>> self.errorhandler(self, exc, value)
> >>>> File "/usr/local/lib/python2.7/site-packages/MySQLdb/connections.py",
> >>>> line 36, in defaulterrorhandler
> >>>> raise errorclass, errorvalue
> >>>> sqlalchemy.exc.ProgrammingError: (_mysql_exceptions.ProgrammingError)
> >>>> (1064, "You have an error in your SQL syntax; check the manual that
> >>>> corresponds to your MySQL server version for the right syntax to use
> >> near
> >>>> '(6) NULL' at line 1") [SQL: u'ALTER TABLE dag MODIFY
> last_scheduler_run
> >>>> DATETIME(6) NULL']
> >>>>
> >>>>
> >>>>
> >>>>> On Thu, Jan 26, 2017 at 1:49 PM, Bolke de Bruin <bd...@gmail.com>
> >> wrote:
> >>>>>
> >>>>> Hi All,
> >>>>>
> >>>>> I have made the FOURTH beta of Airflow 1.8.0 available at:
> >>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> >>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public
> >> keys
> >>>>> are available at https://dist.apache.org/repos/
> >>>>> dist/release/incubator/airflow/ <https://dist.apache.org/repos
> >>>>> /dist/release/incubator/airflow/> . It is tagged with a local
> version
> >>>>> “apache.incubating” so it allows upgrading from earlier releases.
> This
> >> beta
> >>>>> is available for testing in a more production like setting
> (acceptance
> >>>>> environment?).
> >>>>>
> >>>>> I would like to encourage everyone to try it out, to report back any
> >>>>> issues so we get to a rock solid release of 1.8.0. When reporting
> >> issues a
> >>>>> test case or even a fix is highly appreciated.
> >>>>>
> >>>>> Issues fixed:
> >>>>> * Incorrect Alembic reference due to revert (initdb/upgradedb/resetdb
> >>>>> should work again)
> >>>>> * Py3 incompatibility in base_taskrunner.
> >>>>>
> >>>>> Under investigation:
> >>>>> * DAG marked success, with half of the Tasks never scheduled (Alex)
> >>>>>
> >>>>> Kind regards,
> >>>>> Bolke
> >>>>
> >>>>
> >>>>
> >>
>
>
Re: Airflow 1.8.0 BETA 4
Posted by Bolke de Bruin <bd...@gmail.com>.
BTW:
* MySQL 5.6.10 (5.6.4 was a beta I just figured out - released in 2011) was released in 2013
* MySQL 5.1.73 was released on 2013-12-04
We are really not ahead of the pack with our requirements.
Bolke
> On 27 Jan 2017, at 21:32, Boris Tyukin <bo...@boristyukin.com> wrote:
>
> thanks for confirmation Bolke
>
> On Fri, Jan 27, 2017 at 2:45 PM, Bolke de Bruin <bd...@gmail.com> wrote:
>
>> 5.6.4 is the minimum. Yes it was required. updating.md was required.
>> 5.6.4 is already really really old. Most distro's replaced it with mariadb
>> which is compatible.
>>
>> Bolke
>>
>> Sent from my iPhone
>>
>>> On 27 Jan 2017, at 20:19, Boris Tyukin <bo...@boristyukin.com> wrote:
>>>
>>> sorry mysql version i have is mysql Ver 14.14 Distrib 5.1.73
>>>
>>> I see that DATETIME(6) is only supported on MySQL 5.7 and later.
>>>
>>> Was it really necessary to do use DATETIME(6) not just DATETIME?
>>>
>>> Probably should be mention in change doc that this requires an upgrade of
>>> mysql which in my opinion is not cool. epel repo only has 5.1.73 as
>> latest
>>> version and CDH distro includes 5.1.73 as well.
>>>
>>> Some companies might not allow an upgrade of mysql unless it is in one of
>>> the official linux repos.
>>>
>>>
>>>> On Fri, Jan 27, 2017 at 2:14 PM, Boris Tyukin <bo...@boristyukin.com>
>> wrote:
>>>>
>>>> was requirement to MySQL database version changes with 1.8.0? I run
>> mysql
>>>> 14.14 and it worked fine with 1.7.
>>>>
>>>> I just installed 1.8.4 beta 4 and got this error below when i ran
>> airflow
>>>> upgradedb command
>>>>
>>>> File "/usr/local/lib/python2.7/site-packages/MySQLdb/cursors.py", line
>>>> 205, in execute
>>>> self.errorhandler(self, exc, value)
>>>> File "/usr/local/lib/python2.7/site-packages/MySQLdb/connections.py",
>>>> line 36, in defaulterrorhandler
>>>> raise errorclass, errorvalue
>>>> sqlalchemy.exc.ProgrammingError: (_mysql_exceptions.ProgrammingError)
>>>> (1064, "You have an error in your SQL syntax; check the manual that
>>>> corresponds to your MySQL server version for the right syntax to use
>> near
>>>> '(6) NULL' at line 1") [SQL: u'ALTER TABLE dag MODIFY last_scheduler_run
>>>> DATETIME(6) NULL']
>>>>
>>>>
>>>>
>>>>> On Thu, Jan 26, 2017 at 1:49 PM, Bolke de Bruin <bd...@gmail.com>
>> wrote:
>>>>>
>>>>> Hi All,
>>>>>
>>>>> I have made the FOURTH beta of Airflow 1.8.0 available at:
>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public
>> keys
>>>>> are available at https://dist.apache.org/repos/
>>>>> dist/release/incubator/airflow/ <https://dist.apache.org/repos
>>>>> /dist/release/incubator/airflow/> . It is tagged with a local version
>>>>> “apache.incubating” so it allows upgrading from earlier releases. This
>> beta
>>>>> is available for testing in a more production like setting (acceptance
>>>>> environment?).
>>>>>
>>>>> I would like to encourage everyone to try it out, to report back any
>>>>> issues so we get to a rock solid release of 1.8.0. When reporting
>> issues a
>>>>> test case or even a fix is highly appreciated.
>>>>>
>>>>> Issues fixed:
>>>>> * Incorrect Alembic reference due to revert (initdb/upgradedb/resetdb
>>>>> should work again)
>>>>> * Py3 incompatibility in base_taskrunner.
>>>>>
>>>>> Under investigation:
>>>>> * DAG marked success, with half of the Tasks never scheduled (Alex)
>>>>>
>>>>> Kind regards,
>>>>> Bolke
>>>>
>>>>
>>>>
>>
Re: Airflow 1.8.0 BETA 4
Posted by Boris Tyukin <bo...@boristyukin.com>.
thanks for confirmation Bolke
On Fri, Jan 27, 2017 at 2:45 PM, Bolke de Bruin <bd...@gmail.com> wrote:
> 5.6.4 is the minimum. Yes it was required. updating.md was required.
> 5.6.4 is already really really old. Most distro's replaced it with mariadb
> which is compatible.
>
> Bolke
>
> Sent from my iPhone
>
> > On 27 Jan 2017, at 20:19, Boris Tyukin <bo...@boristyukin.com> wrote:
> >
> > sorry mysql version i have is mysql Ver 14.14 Distrib 5.1.73
> >
> > I see that DATETIME(6) is only supported on MySQL 5.7 and later.
> >
> > Was it really necessary to do use DATETIME(6) not just DATETIME?
> >
> > Probably should be mention in change doc that this requires an upgrade of
> > mysql which in my opinion is not cool. epel repo only has 5.1.73 as
> latest
> > version and CDH distro includes 5.1.73 as well.
> >
> > Some companies might not allow an upgrade of mysql unless it is in one of
> > the official linux repos.
> >
> >
> >> On Fri, Jan 27, 2017 at 2:14 PM, Boris Tyukin <bo...@boristyukin.com>
> wrote:
> >>
> >> was requirement to MySQL database version changes with 1.8.0? I run
> mysql
> >> 14.14 and it worked fine with 1.7.
> >>
> >> I just installed 1.8.4 beta 4 and got this error below when i ran
> airflow
> >> upgradedb command
> >>
> >> File "/usr/local/lib/python2.7/site-packages/MySQLdb/cursors.py", line
> >> 205, in execute
> >> self.errorhandler(self, exc, value)
> >> File "/usr/local/lib/python2.7/site-packages/MySQLdb/connections.py",
> >> line 36, in defaulterrorhandler
> >> raise errorclass, errorvalue
> >> sqlalchemy.exc.ProgrammingError: (_mysql_exceptions.ProgrammingError)
> >> (1064, "You have an error in your SQL syntax; check the manual that
> >> corresponds to your MySQL server version for the right syntax to use
> near
> >> '(6) NULL' at line 1") [SQL: u'ALTER TABLE dag MODIFY last_scheduler_run
> >> DATETIME(6) NULL']
> >>
> >>
> >>
> >>> On Thu, Jan 26, 2017 at 1:49 PM, Bolke de Bruin <bd...@gmail.com>
> wrote:
> >>>
> >>> Hi All,
> >>>
> >>> I have made the FOURTH beta of Airflow 1.8.0 available at:
> >>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> >>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public
> keys
> >>> are available at https://dist.apache.org/repos/
> >>> dist/release/incubator/airflow/ <https://dist.apache.org/repos
> >>> /dist/release/incubator/airflow/> . It is tagged with a local version
> >>> “apache.incubating” so it allows upgrading from earlier releases. This
> beta
> >>> is available for testing in a more production like setting (acceptance
> >>> environment?).
> >>>
> >>> I would like to encourage everyone to try it out, to report back any
> >>> issues so we get to a rock solid release of 1.8.0. When reporting
> issues a
> >>> test case or even a fix is highly appreciated.
> >>>
> >>> Issues fixed:
> >>> * Incorrect Alembic reference due to revert (initdb/upgradedb/resetdb
> >>> should work again)
> >>> * Py3 incompatibility in base_taskrunner.
> >>>
> >>> Under investigation:
> >>> * DAG marked success, with half of the Tasks never scheduled (Alex)
> >>>
> >>> Kind regards,
> >>> Bolke
> >>
> >>
> >>
>
Re: Airflow 1.8.0 BETA 4
Posted by Bolke de Bruin <bd...@gmail.com>.
5.6.4 is the minimum. Yes it was required. updating.md was required. 5.6.4 is already really really old. Most distro's replaced it with mariadb which is compatible.
Bolke
Sent from my iPhone
> On 27 Jan 2017, at 20:19, Boris Tyukin <bo...@boristyukin.com> wrote:
>
> sorry mysql version i have is mysql Ver 14.14 Distrib 5.1.73
>
> I see that DATETIME(6) is only supported on MySQL 5.7 and later.
>
> Was it really necessary to do use DATETIME(6) not just DATETIME?
>
> Probably should be mention in change doc that this requires an upgrade of
> mysql which in my opinion is not cool. epel repo only has 5.1.73 as latest
> version and CDH distro includes 5.1.73 as well.
>
> Some companies might not allow an upgrade of mysql unless it is in one of
> the official linux repos.
>
>
>> On Fri, Jan 27, 2017 at 2:14 PM, Boris Tyukin <bo...@boristyukin.com> wrote:
>>
>> was requirement to MySQL database version changes with 1.8.0? I run mysql
>> 14.14 and it worked fine with 1.7.
>>
>> I just installed 1.8.4 beta 4 and got this error below when i ran airflow
>> upgradedb command
>>
>> File "/usr/local/lib/python2.7/site-packages/MySQLdb/cursors.py", line
>> 205, in execute
>> self.errorhandler(self, exc, value)
>> File "/usr/local/lib/python2.7/site-packages/MySQLdb/connections.py",
>> line 36, in defaulterrorhandler
>> raise errorclass, errorvalue
>> sqlalchemy.exc.ProgrammingError: (_mysql_exceptions.ProgrammingError)
>> (1064, "You have an error in your SQL syntax; check the manual that
>> corresponds to your MySQL server version for the right syntax to use near
>> '(6) NULL' at line 1") [SQL: u'ALTER TABLE dag MODIFY last_scheduler_run
>> DATETIME(6) NULL']
>>
>>
>>
>>> On Thu, Jan 26, 2017 at 1:49 PM, Bolke de Bruin <bd...@gmail.com> wrote:
>>>
>>> Hi All,
>>>
>>> I have made the FOURTH beta of Airflow 1.8.0 available at:
>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public keys
>>> are available at https://dist.apache.org/repos/
>>> dist/release/incubator/airflow/ <https://dist.apache.org/repos
>>> /dist/release/incubator/airflow/> . It is tagged with a local version
>>> “apache.incubating” so it allows upgrading from earlier releases. This beta
>>> is available for testing in a more production like setting (acceptance
>>> environment?).
>>>
>>> I would like to encourage everyone to try it out, to report back any
>>> issues so we get to a rock solid release of 1.8.0. When reporting issues a
>>> test case or even a fix is highly appreciated.
>>>
>>> Issues fixed:
>>> * Incorrect Alembic reference due to revert (initdb/upgradedb/resetdb
>>> should work again)
>>> * Py3 incompatibility in base_taskrunner.
>>>
>>> Under investigation:
>>> * DAG marked success, with half of the Tasks never scheduled (Alex)
>>>
>>> Kind regards,
>>> Bolke
>>
>>
>>
Re: Airflow 1.8.0 BETA 4
Posted by Boris Tyukin <bo...@boristyukin.com>.
sorry mysql version i have is mysql Ver 14.14 Distrib 5.1.73
I see that DATETIME(6) is only supported on MySQL 5.7 and later.
Was it really necessary to do use DATETIME(6) not just DATETIME?
Probably should be mention in change doc that this requires an upgrade of
mysql which in my opinion is not cool. epel repo only has 5.1.73 as latest
version and CDH distro includes 5.1.73 as well.
Some companies might not allow an upgrade of mysql unless it is in one of
the official linux repos.
On Fri, Jan 27, 2017 at 2:14 PM, Boris Tyukin <bo...@boristyukin.com> wrote:
> was requirement to MySQL database version changes with 1.8.0? I run mysql
> 14.14 and it worked fine with 1.7.
>
> I just installed 1.8.4 beta 4 and got this error below when i ran airflow
> upgradedb command
>
> File "/usr/local/lib/python2.7/site-packages/MySQLdb/cursors.py", line
> 205, in execute
> self.errorhandler(self, exc, value)
> File "/usr/local/lib/python2.7/site-packages/MySQLdb/connections.py",
> line 36, in defaulterrorhandler
> raise errorclass, errorvalue
> sqlalchemy.exc.ProgrammingError: (_mysql_exceptions.ProgrammingError)
> (1064, "You have an error in your SQL syntax; check the manual that
> corresponds to your MySQL server version for the right syntax to use near
> '(6) NULL' at line 1") [SQL: u'ALTER TABLE dag MODIFY last_scheduler_run
> DATETIME(6) NULL']
>
>
>
> On Thu, Jan 26, 2017 at 1:49 PM, Bolke de Bruin <bd...@gmail.com> wrote:
>
>> Hi All,
>>
>> I have made the FOURTH beta of Airflow 1.8.0 available at:
>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public keys
>> are available at https://dist.apache.org/repos/
>> dist/release/incubator/airflow/ <https://dist.apache.org/repos
>> /dist/release/incubator/airflow/> . It is tagged with a local version
>> “apache.incubating” so it allows upgrading from earlier releases. This beta
>> is available for testing in a more production like setting (acceptance
>> environment?).
>>
>> I would like to encourage everyone to try it out, to report back any
>> issues so we get to a rock solid release of 1.8.0. When reporting issues a
>> test case or even a fix is highly appreciated.
>>
>> Issues fixed:
>> * Incorrect Alembic reference due to revert (initdb/upgradedb/resetdb
>> should work again)
>> * Py3 incompatibility in base_taskrunner.
>>
>> Under investigation:
>> * DAG marked success, with half of the Tasks never scheduled (Alex)
>>
>> Kind regards,
>> Bolke
>
>
>
Re: Airflow 1.8.0 BETA 4
Posted by Boris Tyukin <bo...@boristyukin.com>.
was requirement to MySQL database version changes with 1.8.0? I run mysql
14.14 and it worked fine with 1.7.
I just installed 1.8.4 beta 4 and got this error below when i ran airflow
upgradedb command
File "/usr/local/lib/python2.7/site-packages/MySQLdb/cursors.py", line
205, in execute
self.errorhandler(self, exc, value)
File "/usr/local/lib/python2.7/site-packages/MySQLdb/connections.py",
line 36, in defaulterrorhandler
raise errorclass, errorvalue
sqlalchemy.exc.ProgrammingError: (_mysql_exceptions.ProgrammingError)
(1064, "You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use near
'(6) NULL' at line 1") [SQL: u'ALTER TABLE dag MODIFY last_scheduler_run
DATETIME(6) NULL']
On Thu, Jan 26, 2017 at 1:49 PM, Bolke de Bruin <bd...@gmail.com> wrote:
> Hi All,
>
> I have made the FOURTH beta of Airflow 1.8.0 available at:
> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public keys
> are available at https://dist.apache.org/repos/dist/release/incubator/
> airflow/ <https://dist.apache.org/repos/dist/release/incubator/airflow/>
> . It is tagged with a local version “apache.incubating” so it allows
> upgrading from earlier releases. This beta is available for testing in a
> more production like setting (acceptance environment?).
>
> I would like to encourage everyone to try it out, to report back any
> issues so we get to a rock solid release of 1.8.0. When reporting issues a
> test case or even a fix is highly appreciated.
>
> Issues fixed:
> * Incorrect Alembic reference due to revert (initdb/upgradedb/resetdb
> should work again)
> * Py3 incompatibility in base_taskrunner.
>
> Under investigation:
> * DAG marked success, with half of the Tasks never scheduled (Alex)
>
> Kind regards,
> Bolke