You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Bolke de Bruin <bd...@gmail.com> on 2017/01/25 22:15:06 UTC
Airflow 1.8.0 BETA 3
Hi All,
I have made the THIRD beta of Airflow 1.8.0 available at: https://dist.apache.org/repos/dist/dev/incubator/airflow/ <https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public keys are available at https://dist.apache.org/repos/dist/release/incubator/airflow/ <https://dist.apache.org/repos/dist/release/incubator/airflow/> . It is tagged with a local version “apache.incubating” so it allows upgrading from earlier releases. This beta is available for testing in a more production like setting (acceptance environment?).
I would like to encourage everyone to try it out, to report back any issues so we get to a rock solid release of 1.8.0. When reporting issues a test case or even a fix is highly appreciated.
Issues cleared:
* Manual trigger not working
* Performance issues with MySQL
* Postgres auto-commit support left over to the driver
* Poison pill taken while task has exited
* Keep cgroups optional
* Funcsigs pinned to 1.0.0
Issue(s) remaining (blocker for RC):
* Cgroups not py3 compatible
If all goes well we should have a Release Candidate on Feb 2. Thanks for reporting issues and keep on testing please :). Moving towards RC I tend to like small bug fixes only. When we mark RC (do we need to vote on this?) the procedure becomes even more strict. Please remember that the FINAL release is dependent on a vote on the IPMC mailinglist.
Cheers
Bolke
Re: Airflow 1.8.0 BETA 3
Posted by Bolke de Bruin <bd...@gmail.com>.
Simple workaround: downgrade to beta 1 first.
Sent from my iPhone
> On 26 Jan 2017, at 07:34, Bolke de Bruin <bd...@gmail.com> wrote:
>
> Mmm that is due to the reverting of one changes to the db. Need to look into that how to fix it.
>
> Sent from my iPhone
>
>> On 26 Jan 2017, at 00:51, Alex Van Boxel <al...@vanboxel.be> wrote:
>>
>> I do seem to have a problem upgrading the MySQL database with the last
>> commit:
>>
>>
>> 2017-01-25T23:41:55.662572654Z
>> /usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/messaging.py:69:
>> UserWarning: Revision 1a5a9e6bf2b5 referenced from 1a5a9e6bf2b5 ->
>> 127d2bf2dfa7 (head), Add dag_id/state index on dag_run table is not present
>> 2017-01-25T23:41:55.662613670Z warnings.warn(msg)
>> 2017-01-25T23:41:55.664560884Z Traceback (most recent call last):
>> 2017-01-25T23:41:55.664582238Z File "/usr/local/bin/airflow", line 4, in
>> <module>
>> 2017-01-25T23:41:55.664758457Z
>> __import__('pkg_resources').run_script('airflow==1.8.0b1+apache.incubating',
>> 'airflow')
>> 2017-01-25T23:41:55.664776553Z File
>> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line
>> 739, in run_script
>> 2017-01-25T23:41:55.664937644Z
>> self.require(requires)[0].run_script(script_name, ns)
>> 2017-01-25T23:41:55.664952001Z File
>> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line
>> 1494, in run_script
>> 2017-01-25T23:41:55.665072840Z exec(code, namespace, namespace)
>> 2017-01-25T23:41:55.665086884Z File
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/EGG-INFO/scripts/airflow",
>> line 28, in <module>
>> 2017-01-25T23:41:55.665211755Z args.func(args)
>> 2017-01-25T23:41:55.665225157Z File
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/bin/cli.py",
>> line 931, in upgradedb
>> 2017-01-25T23:41:55.665326691Z db_utils.upgradedb()
>> 2017-01-25T23:41:55.665346979Z File
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/utils/db.py",
>> line 292, in upgradedb
>> 2017-01-25T23:41:55.665449297Z command.upgrade(config, 'heads')
>> 2017-01-25T23:41:55.665462333Z File
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
>> line 174, in upgrade
>> 2017-01-25T23:41:55.665521466Z script.run_env()
>> 2017-01-25T23:41:55.665527232Z File
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
>> line 416, in run_env2017-01-25T23:41:55.665638879Z
>> util.load_python_file(self.dir, 'env.py')
>> 2017-01-25T23:41:55.665656437Z File
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/pyfiles.py",
>> line 93, in load_python_file
>> 2017-01-25T23:41:55.665682760Z module = load_module_py(module_id, path)
>> 2017-01-25T23:41:55.665699257Z File
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/compat.py",
>> line 79, in load_module_py
>> 2017-01-25T23:41:55.665725036Z mod = imp.load_source(module_id, path,
>> fp)
>> 2017-01-25T23:41:55.665751433Z File
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
>> line 88, in <module>
>> 2017-01-25T23:41:55.665778219Z run_migrations_online()
>> 2017-01-25T23:41:55.665787820Z File
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
>> line 83, in run_migrations_online
>> 2017-01-25T23:41:55.665811353Z context.run_migrations()
>> 2017-01-25T23:41:55.665818933Z File "<string>", line 8, in run_migrations
>> 2017-01-25T23:41:55.666707547Z File
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/environment.py",
>> line 797, in run_migrations
>> 2017-01-25T23:41:55.666839945Z self.get_context().run_migrations(**kw)
>> 2017-01-25T23:41:55.666859336Z File
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/migration.py",
>> line 305, in run_migrations
>> 2017-01-25T23:41:55.666970791Z for step in self._migrations_fn(heads,
>> self):
>> 2017-01-25T23:41:55.666992265Z File
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
>> line 163, in upgrade
>> 2017-01-25T23:41:55.667021804Z return script._upgrade_revs(revision,
>> rev)
>> 2017-01-25T23:41:55.667032496Z File
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
>> line 329, in _upgrade_revs
>> 2017-01-25T23:41:55.667144295Z revs = list(revs)
>> 2017-01-25T23:41:55.667157407Z File
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>> line 641, in _iterate_revisions
>> 2017-01-25T23:41:55.667302027Z requested_lowers =
>> self.get_revisions(lower)
>> 2017-01-25T23:41:55.667315913Z File
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>> line 298, in get_revisions
>> 2017-01-25T23:41:55.667390590Z return sum([self.get_revisions(id_elem)
>> for id_elem in id_], ())
>> 2017-01-25T23:41:55.667411867Z File
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>> line 300, in get_revisions
>> 2017-01-25T23:41:55.667419226Z resolved_id, branch_label =
>> self._resolve_revision_number(id_)
>> 2017-01-25T23:41:55.667434110Z File
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>> line 433, in _resolve_revision_number
>> 2017-01-25T23:41:55.667576273Z self._revision_map
>> 2017-01-25T23:41:55.667590641Z File
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/langhelpers.py",
>> line 240, in __get__
>> 2017-01-25T23:41:55.667648561Z obj.__dict__[self.__name__] = result =
>> self.fget(obj)
>> 2017-01-25T23:41:55.667658598Z File
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>> line 151, in _revision_map
>> 2017-01-25T23:41:55.667680311Z down_revision = map_[downrev]
>> 2017-01-25T23:41:55.667746364Z KeyError: '1a5a9e6bf2b5'
>>
>>
>>
>>
>>
>>
>>> On Wed, Jan 25, 2017 at 11:15 PM Bolke de Bruin <bd...@gmail.com> wrote:
>>>
>>> Hi All,
>>>
>>> I have made the THIRD beta of Airflow 1.8.0 available at:
>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public keys
>>> are available at
>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>>> https://dist.apache.org/repos/dist/release/incubator/airflow/> . It is
>>> tagged with a local version “apache.incubating” so it allows upgrading from
>>> earlier releases. This beta is available for testing in a more production
>>> like setting (acceptance environment?).
>>>
>>> I would like to encourage everyone to try it out, to report back any
>>> issues so we get to a rock solid release of 1.8.0. When reporting issues a
>>> test case or even a fix is highly appreciated.
>>>
>>> Issues cleared:
>>>
>>> * Manual trigger not working
>>> * Performance issues with MySQL
>>> * Postgres auto-commit support left over to the driver
>>> * Poison pill taken while task has exited
>>> * Keep cgroups optional
>>> * Funcsigs pinned to 1.0.0
>>>
>>> Issue(s) remaining (blocker for RC):
>>> * Cgroups not py3 compatible
>>>
>>> If all goes well we should have a Release Candidate on Feb 2. Thanks for
>>> reporting issues and keep on testing please :). Moving towards RC I tend to
>>> like small bug fixes only. When we mark RC (do we need to vote on this?)
>>> the procedure becomes even more strict. Please remember that the FINAL
>>> release is dependent on a vote on the IPMC mailinglist.
>>>
>>> Cheers
>>> Bolke
>>
>> --
>> _/
>> _/ Alex Van Boxel
Re: Airflow 1.8.0 BETA 3
Posted by Bolke de Bruin <bd...@gmail.com>.
Sorry, it was working due to my unclean build environment. Please try https://github.com/apache/incubator-airflow/pull/2023 , +1 if it works and I will make Beta 4 available.
Bolke
> On 26 Jan 2017, at 13:27, Alex Van Boxel <al...@vanboxel.be> wrote:
>
> so, yes. doesn't work:
>
> I did:
> - create new conda environment
> - python setup.py install
> - rm airflow.db (sqllite)
> - airflow initdb
>
> same error:
> self._revision_map
> File
> "/Users/alexvanboxel/miniconda2/lib/python2.7/site-packages/alembic-0.8.10-py2.7.egg/alembic/util/langhelpers.py",
> line 240, in __get__
> obj.__dict__[self.__name__] = result = self.fget(obj)
> File
> "/Users/alexvanboxel/miniconda2/lib/python2.7/site-packages/alembic-0.8.10-py2.7.egg/alembic/script/revision.py",
> line 151, in _revision_map
> down_revision = map_[downrev]
> KeyError: '1a5a9e6bf2b5'
>
> consistent as my cluster setup (that one is build in the docker container)
> and this one my mac (both give the same error)
>
>
>
>
>
> On Thu, Jan 26, 2017 at 11:40 AM Bolke de Bruin <bd...@gmail.com> wrote:
>
>> Hi Alex
>>
>> Are you sure you are in a fully clean environment? Sometimes things remain
>> in the airflow directories. This worked for me:
>>
>> rm -rf site-packages/airflow*
>> python setup.py install
>> airflow initdb
>>
>> Bolke
>>
>>
>>> On 26 Jan 2017, at 10:53, Alex Van Boxel <al...@vanboxel.be> wrote:
>>>
>>> About the database error: starting from scratch also gives the same
>> error:
>>>
>>> Fresh install. Delete airflow.db sqllite db. And the : airflow initdb
>>>
>>> same error as above.
>>>
>>> On Thu, Jan 26, 2017 at 10:12 AM Alex Van Boxel <al...@vanboxel.be>
>> wrote:
>>>
>>>> Not directly one I can share. I'll spend some time looking at it. I'll
>> try
>>>> to create a unittest of it.
>>>>
>>>> On Thu, Jan 26, 2017 at 9:47 AM Bolke de Bruin <bd...@gmail.com>
>> wrote:
>>>>
>>>> Do you have a example dag for this? That makes it easier to work on.
>>>>
>>>> Bolke
>>>>
>>>> Sent from my iPhone
>>>>
>>>>> On 26 Jan 2017, at 08:36, Alex Van Boxel <al...@vanboxel.be> wrote:
>>>>>
>>>>> Another thing that I noticed (but observed it in beta 2 as well). Is
>> the
>>>>> following:
>>>>>
>>>>> - The following trigger should not fire.
>>>>> --- Trigger rule is ONE_SUCCESS
>>>>> --- upstream: UP_FOR_RETRY + SKIPPED
>>>>> => task get's triggered
>>>>> => resulting SKIPPED
>>>>> => DAG marked success, with half of the Tasks never scheduled
>>>>>
>>>>> - UP_FOR_RETRY is propagated downstream (actually resulting in failure
>>>>> described above)
>>>>> --- *Does this make sense* ?!
>>>>>
>>>>> Both are a problem, and certainly the combination of both. I'll see if
>> I
>>>>> can spend some time investigating this.
>>>>>
>>>>>
>>>>>
>>>>>> On Thu, Jan 26, 2017 at 7:34 AM Bolke de Bruin <bd...@gmail.com>
>>>> wrote:
>>>>>>
>>>>>> Mmm that is due to the reverting of one changes to the db. Need to
>> look
>>>>>> into that how to fix it.
>>>>>>
>>>>>> Sent from my iPhone
>>>>>>
>>>>>>> On 26 Jan 2017, at 00:51, Alex Van Boxel <al...@vanboxel.be> wrote:
>>>>>>>
>>>>>>> I do seem to have a problem upgrading the MySQL database with the
>> last
>>>>>>> commit:
>>>>>>>
>>>>>>>
>>>>>>> 2017-01-25T23:41:55.662572654Z
>>>>>>>
>>>>>>
>>>>
>> /usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/messaging.py:69:
>>>>>>> UserWarning: Revision 1a5a9e6bf2b5 referenced from 1a5a9e6bf2b5 ->
>>>>>>> 127d2bf2dfa7 (head), Add dag_id/state index on dag_run table is not
>>>>>> present
>>>>>>> 2017-01-25T23:41:55.662613670Z warnings.warn(msg)
>>>>>>> 2017-01-25T23:41:55.664560884Z Traceback (most recent call last):
>>>>>>> 2017-01-25T23:41:55.664582238Z File "/usr/local/bin/airflow", line
>> 4,
>>>>>> in
>>>>>>> <module>
>>>>>>> 2017-01-25T23:41:55.664758457Z
>>>>>>>
>>>>>>
>>>>
>> __import__('pkg_resources').run_script('airflow==1.8.0b1+apache.incubating',
>>>>>>> 'airflow')
>>>>>>> 2017-01-25T23:41:55.664776553Z File
>>>>>>> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py",
>>>> line
>>>>>>> 739, in run_script
>>>>>>> 2017-01-25T23:41:55.664937644Z
>>>>>>> self.require(requires)[0].run_script(script_name, ns)
>>>>>>> 2017-01-25T23:41:55.664952001Z File
>>>>>>> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py",
>>>> line
>>>>>>> 1494, in run_script
>>>>>>> 2017-01-25T23:41:55.665072840Z exec(code, namespace, namespace)
>>>>>>> 2017-01-25T23:41:55.665086884Z File
>>>>>>>
>>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/EGG-INFO/scripts/airflow",
>>>>>>> line 28, in <module>
>>>>>>> 2017-01-25T23:41:55.665211755Z args.func(args)
>>>>>>> 2017-01-25T23:41:55.665225157Z File
>>>>>>>
>>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/bin/cli.py",
>>>>>>> line 931, in upgradedb
>>>>>>> 2017-01-25T23:41:55.665326691Z db_utils.upgradedb()
>>>>>>> 2017-01-25T23:41:55.665346979Z File
>>>>>>>
>>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/utils/db.py",
>>>>>>> line 292, in upgradedb
>>>>>>> 2017-01-25T23:41:55.665449297Z command.upgrade(config, 'heads')
>>>>>>> 2017-01-25T23:41:55.665462333Z File
>>>>>>>
>>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
>>>>>>> line 174, in upgrade
>>>>>>> 2017-01-25T23:41:55.665521466Z script.run_env()
>>>>>>> 2017-01-25T23:41:55.665527232Z File
>>>>>>>
>>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
>>>>>>> line 416, in run_env2017-01-25T23:41:55.665638879Z
>>>>>>> util.load_python_file(self.dir, 'env.py')
>>>>>>> 2017-01-25T23:41:55.665656437Z File
>>>>>>>
>>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/pyfiles.py",
>>>>>>> line 93, in load_python_file
>>>>>>> 2017-01-25T23:41:55.665682760Z module = load_module_py(module_id,
>>>>>> path)
>>>>>>> 2017-01-25T23:41:55.665699257Z File
>>>>>>>
>>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/compat.py",
>>>>>>> line 79, in load_module_py
>>>>>>> 2017-01-25T23:41:55.665725036Z mod = imp.load_source(module_id,
>>>> path,
>>>>>>> fp)
>>>>>>> 2017-01-25T23:41:55.665751433Z File
>>>>>>>
>>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
>>>>>>> line 88, in <module>
>>>>>>> 2017-01-25T23:41:55.665778219Z run_migrations_online()
>>>>>>> 2017-01-25T23:41:55.665787820Z File
>>>>>>>
>>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
>>>>>>> line 83, in run_migrations_online
>>>>>>> 2017-01-25T23:41:55.665811353Z context.run_migrations()
>>>>>>> 2017-01-25T23:41:55.665818933Z File "<string>", line 8, in
>>>>>> run_migrations
>>>>>>> 2017-01-25T23:41:55.666707547Z File
>>>>>>>
>>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/environment.py",
>>>>>>> line 797, in run_migrations
>>>>>>> 2017-01-25T23:41:55.666839945Z
>>>>>> self.get_context().run_migrations(**kw)
>>>>>>> 2017-01-25T23:41:55.666859336Z File
>>>>>>>
>>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/migration.py",
>>>>>>> line 305, in run_migrations
>>>>>>> 2017-01-25T23:41:55.666970791Z for step in
>>>> self._migrations_fn(heads,
>>>>>>> self):
>>>>>>> 2017-01-25T23:41:55.666992265Z File
>>>>>>>
>>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
>>>>>>> line 163, in upgrade
>>>>>>> 2017-01-25T23:41:55.667021804Z return
>>>> script._upgrade_revs(revision,
>>>>>>> rev)
>>>>>>> 2017-01-25T23:41:55.667032496Z File
>>>>>>>
>>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
>>>>>>> line 329, in _upgrade_revs
>>>>>>> 2017-01-25T23:41:55.667144295Z revs = list(revs)
>>>>>>> 2017-01-25T23:41:55.667157407Z File
>>>>>>>
>>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>>>>>> line 641, in _iterate_revisions
>>>>>>> 2017-01-25T23:41:55.667302027Z requested_lowers =
>>>>>>> self.get_revisions(lower)
>>>>>>> 2017-01-25T23:41:55.667315913Z File
>>>>>>>
>>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>>>>>> line 298, in get_revisions
>>>>>>> 2017-01-25T23:41:55.667390590Z return
>>>>>> sum([self.get_revisions(id_elem)
>>>>>>> for id_elem in id_], ())
>>>>>>> 2017-01-25T23:41:55.667411867Z File
>>>>>>>
>>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>>>>>> line 300, in get_revisions
>>>>>>> 2017-01-25T23:41:55.667419226Z resolved_id, branch_label =
>>>>>>> self._resolve_revision_number(id_)
>>>>>>> 2017-01-25T23:41:55.667434110Z File
>>>>>>>
>>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>>>>>> line 433, in _resolve_revision_number
>>>>>>> 2017-01-25T23:41:55.667576273Z self._revision_map
>>>>>>> 2017-01-25T23:41:55.667590641Z File
>>>>>>>
>>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/langhelpers.py",
>>>>>>> line 240, in __get__
>>>>>>> 2017-01-25T23:41:55.667648561Z obj.__dict__[self.__name__] =
>>>> result =
>>>>>>> self.fget(obj)
>>>>>>> 2017-01-25T23:41:55.667658598Z File
>>>>>>>
>>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>>>>>> line 151, in _revision_map
>>>>>>> 2017-01-25T23:41:55.667680311Z down_revision = map_[downrev]
>>>>>>> 2017-01-25T23:41:55.667746364Z KeyError: '1a5a9e6bf2b5'
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On Wed, Jan 25, 2017 at 11:15 PM Bolke de Bruin <bd...@gmail.com>
>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> I have made the THIRD beta of Airflow 1.8.0 available at:
>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public
>>>>>> keys
>>>>>>>> are available at
>>>>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>>>>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/> . It
>>>> is
>>>>>>>> tagged with a local version “apache.incubating” so it allows
>> upgrading
>>>>>> from
>>>>>>>> earlier releases. This beta is available for testing in a more
>>>>>> production
>>>>>>>> like setting (acceptance environment?).
>>>>>>>>
>>>>>>>> I would like to encourage everyone to try it out, to report back
>> any
>>>>>>>> issues so we get to a rock solid release of 1.8.0. When reporting
>>>>>> issues a
>>>>>>>> test case or even a fix is highly appreciated.
>>>>>>>>
>>>>>>>> Issues cleared:
>>>>>>>>
>>>>>>>> * Manual trigger not working
>>>>>>>> * Performance issues with MySQL
>>>>>>>> * Postgres auto-commit support left over to the driver
>>>>>>>> * Poison pill taken while task has exited
>>>>>>>> * Keep cgroups optional
>>>>>>>> * Funcsigs pinned to 1.0.0
>>>>>>>>
>>>>>>>> Issue(s) remaining (blocker for RC):
>>>>>>>> * Cgroups not py3 compatible
>>>>>>>>
>>>>>>>> If all goes well we should have a Release Candidate on Feb 2. Thanks
>>>> for
>>>>>>>> reporting issues and keep on testing please :). Moving towards RC I
>>>>>> tend to
>>>>>>>> like small bug fixes only. When we mark RC (do we need to vote on
>>>> this?)
>>>>>>>> the procedure becomes even more strict. Please remember that the
>> FINAL
>>>>>>>> release is dependent on a vote on the IPMC mailinglist.
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>> Bolke
>>>>>>>
>>>>>>> --
>>>>>>> _/
>>>>>>> _/ Alex Van Boxel
>>>>>>
>>>>> --
>>>>> _/
>>>>> _/ Alex Van Boxel
>>>>
>>>> --
>>>> _/
>>>> _/ Alex Van Boxel
>>>>
>>> --
>>> _/
>>> _/ Alex Van Boxel
>>
>> --
> _/
> _/ Alex Van Boxel
Re: Airflow 1.8.0 BETA 3
Posted by Alex Van Boxel <al...@vanboxel.be>.
so, yes. doesn't work:
I did:
- create new conda environment
- python setup.py install
- rm airflow.db (sqllite)
- airflow initdb
same error:
self._revision_map
File
"/Users/alexvanboxel/miniconda2/lib/python2.7/site-packages/alembic-0.8.10-py2.7.egg/alembic/util/langhelpers.py",
line 240, in __get__
obj.__dict__[self.__name__] = result = self.fget(obj)
File
"/Users/alexvanboxel/miniconda2/lib/python2.7/site-packages/alembic-0.8.10-py2.7.egg/alembic/script/revision.py",
line 151, in _revision_map
down_revision = map_[downrev]
KeyError: '1a5a9e6bf2b5'
consistent as my cluster setup (that one is build in the docker container)
and this one my mac (both give the same error)
On Thu, Jan 26, 2017 at 11:40 AM Bolke de Bruin <bd...@gmail.com> wrote:
> Hi Alex
>
> Are you sure you are in a fully clean environment? Sometimes things remain
> in the airflow directories. This worked for me:
>
> rm -rf site-packages/airflow*
> python setup.py install
> airflow initdb
>
> Bolke
>
>
> > On 26 Jan 2017, at 10:53, Alex Van Boxel <al...@vanboxel.be> wrote:
> >
> > About the database error: starting from scratch also gives the same
> error:
> >
> > Fresh install. Delete airflow.db sqllite db. And the : airflow initdb
> >
> > same error as above.
> >
> > On Thu, Jan 26, 2017 at 10:12 AM Alex Van Boxel <al...@vanboxel.be>
> wrote:
> >
> >> Not directly one I can share. I'll spend some time looking at it. I'll
> try
> >> to create a unittest of it.
> >>
> >> On Thu, Jan 26, 2017 at 9:47 AM Bolke de Bruin <bd...@gmail.com>
> wrote:
> >>
> >> Do you have a example dag for this? That makes it easier to work on.
> >>
> >> Bolke
> >>
> >> Sent from my iPhone
> >>
> >>> On 26 Jan 2017, at 08:36, Alex Van Boxel <al...@vanboxel.be> wrote:
> >>>
> >>> Another thing that I noticed (but observed it in beta 2 as well). Is
> the
> >>> following:
> >>>
> >>> - The following trigger should not fire.
> >>> --- Trigger rule is ONE_SUCCESS
> >>> --- upstream: UP_FOR_RETRY + SKIPPED
> >>> => task get's triggered
> >>> => resulting SKIPPED
> >>> => DAG marked success, with half of the Tasks never scheduled
> >>>
> >>> - UP_FOR_RETRY is propagated downstream (actually resulting in failure
> >>> described above)
> >>> --- *Does this make sense* ?!
> >>>
> >>> Both are a problem, and certainly the combination of both. I'll see if
> I
> >>> can spend some time investigating this.
> >>>
> >>>
> >>>
> >>>> On Thu, Jan 26, 2017 at 7:34 AM Bolke de Bruin <bd...@gmail.com>
> >> wrote:
> >>>>
> >>>> Mmm that is due to the reverting of one changes to the db. Need to
> look
> >>>> into that how to fix it.
> >>>>
> >>>> Sent from my iPhone
> >>>>
> >>>>> On 26 Jan 2017, at 00:51, Alex Van Boxel <al...@vanboxel.be> wrote:
> >>>>>
> >>>>> I do seem to have a problem upgrading the MySQL database with the
> last
> >>>>> commit:
> >>>>>
> >>>>>
> >>>>> 2017-01-25T23:41:55.662572654Z
> >>>>>
> >>>>
> >>
> /usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/messaging.py:69:
> >>>>> UserWarning: Revision 1a5a9e6bf2b5 referenced from 1a5a9e6bf2b5 ->
> >>>>> 127d2bf2dfa7 (head), Add dag_id/state index on dag_run table is not
> >>>> present
> >>>>> 2017-01-25T23:41:55.662613670Z warnings.warn(msg)
> >>>>> 2017-01-25T23:41:55.664560884Z Traceback (most recent call last):
> >>>>> 2017-01-25T23:41:55.664582238Z File "/usr/local/bin/airflow", line
> 4,
> >>>> in
> >>>>> <module>
> >>>>> 2017-01-25T23:41:55.664758457Z
> >>>>>
> >>>>
> >>
> __import__('pkg_resources').run_script('airflow==1.8.0b1+apache.incubating',
> >>>>> 'airflow')
> >>>>> 2017-01-25T23:41:55.664776553Z File
> >>>>> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py",
> >> line
> >>>>> 739, in run_script
> >>>>> 2017-01-25T23:41:55.664937644Z
> >>>>> self.require(requires)[0].run_script(script_name, ns)
> >>>>> 2017-01-25T23:41:55.664952001Z File
> >>>>> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py",
> >> line
> >>>>> 1494, in run_script
> >>>>> 2017-01-25T23:41:55.665072840Z exec(code, namespace, namespace)
> >>>>> 2017-01-25T23:41:55.665086884Z File
> >>>>>
> >>>>
> >>
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/EGG-INFO/scripts/airflow",
> >>>>> line 28, in <module>
> >>>>> 2017-01-25T23:41:55.665211755Z args.func(args)
> >>>>> 2017-01-25T23:41:55.665225157Z File
> >>>>>
> >>>>
> >>
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/bin/cli.py",
> >>>>> line 931, in upgradedb
> >>>>> 2017-01-25T23:41:55.665326691Z db_utils.upgradedb()
> >>>>> 2017-01-25T23:41:55.665346979Z File
> >>>>>
> >>>>
> >>
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/utils/db.py",
> >>>>> line 292, in upgradedb
> >>>>> 2017-01-25T23:41:55.665449297Z command.upgrade(config, 'heads')
> >>>>> 2017-01-25T23:41:55.665462333Z File
> >>>>>
> >>>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
> >>>>> line 174, in upgrade
> >>>>> 2017-01-25T23:41:55.665521466Z script.run_env()
> >>>>> 2017-01-25T23:41:55.665527232Z File
> >>>>>
> >>>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
> >>>>> line 416, in run_env2017-01-25T23:41:55.665638879Z
> >>>>> util.load_python_file(self.dir, 'env.py')
> >>>>> 2017-01-25T23:41:55.665656437Z File
> >>>>>
> >>>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/pyfiles.py",
> >>>>> line 93, in load_python_file
> >>>>> 2017-01-25T23:41:55.665682760Z module = load_module_py(module_id,
> >>>> path)
> >>>>> 2017-01-25T23:41:55.665699257Z File
> >>>>>
> >>>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/compat.py",
> >>>>> line 79, in load_module_py
> >>>>> 2017-01-25T23:41:55.665725036Z mod = imp.load_source(module_id,
> >> path,
> >>>>> fp)
> >>>>> 2017-01-25T23:41:55.665751433Z File
> >>>>>
> >>>>
> >>
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
> >>>>> line 88, in <module>
> >>>>> 2017-01-25T23:41:55.665778219Z run_migrations_online()
> >>>>> 2017-01-25T23:41:55.665787820Z File
> >>>>>
> >>>>
> >>
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
> >>>>> line 83, in run_migrations_online
> >>>>> 2017-01-25T23:41:55.665811353Z context.run_migrations()
> >>>>> 2017-01-25T23:41:55.665818933Z File "<string>", line 8, in
> >>>> run_migrations
> >>>>> 2017-01-25T23:41:55.666707547Z File
> >>>>>
> >>>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/environment.py",
> >>>>> line 797, in run_migrations
> >>>>> 2017-01-25T23:41:55.666839945Z
> >>>> self.get_context().run_migrations(**kw)
> >>>>> 2017-01-25T23:41:55.666859336Z File
> >>>>>
> >>>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/migration.py",
> >>>>> line 305, in run_migrations
> >>>>> 2017-01-25T23:41:55.666970791Z for step in
> >> self._migrations_fn(heads,
> >>>>> self):
> >>>>> 2017-01-25T23:41:55.666992265Z File
> >>>>>
> >>>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
> >>>>> line 163, in upgrade
> >>>>> 2017-01-25T23:41:55.667021804Z return
> >> script._upgrade_revs(revision,
> >>>>> rev)
> >>>>> 2017-01-25T23:41:55.667032496Z File
> >>>>>
> >>>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
> >>>>> line 329, in _upgrade_revs
> >>>>> 2017-01-25T23:41:55.667144295Z revs = list(revs)
> >>>>> 2017-01-25T23:41:55.667157407Z File
> >>>>>
> >>>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> >>>>> line 641, in _iterate_revisions
> >>>>> 2017-01-25T23:41:55.667302027Z requested_lowers =
> >>>>> self.get_revisions(lower)
> >>>>> 2017-01-25T23:41:55.667315913Z File
> >>>>>
> >>>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> >>>>> line 298, in get_revisions
> >>>>> 2017-01-25T23:41:55.667390590Z return
> >>>> sum([self.get_revisions(id_elem)
> >>>>> for id_elem in id_], ())
> >>>>> 2017-01-25T23:41:55.667411867Z File
> >>>>>
> >>>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> >>>>> line 300, in get_revisions
> >>>>> 2017-01-25T23:41:55.667419226Z resolved_id, branch_label =
> >>>>> self._resolve_revision_number(id_)
> >>>>> 2017-01-25T23:41:55.667434110Z File
> >>>>>
> >>>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> >>>>> line 433, in _resolve_revision_number
> >>>>> 2017-01-25T23:41:55.667576273Z self._revision_map
> >>>>> 2017-01-25T23:41:55.667590641Z File
> >>>>>
> >>>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/langhelpers.py",
> >>>>> line 240, in __get__
> >>>>> 2017-01-25T23:41:55.667648561Z obj.__dict__[self.__name__] =
> >> result =
> >>>>> self.fget(obj)
> >>>>> 2017-01-25T23:41:55.667658598Z File
> >>>>>
> >>>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> >>>>> line 151, in _revision_map
> >>>>> 2017-01-25T23:41:55.667680311Z down_revision = map_[downrev]
> >>>>> 2017-01-25T23:41:55.667746364Z KeyError: '1a5a9e6bf2b5'
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>> On Wed, Jan 25, 2017 at 11:15 PM Bolke de Bruin <bd...@gmail.com>
> >>>> wrote:
> >>>>>>
> >>>>>> Hi All,
> >>>>>>
> >>>>>> I have made the THIRD beta of Airflow 1.8.0 available at:
> >>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> >>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public
> >>>> keys
> >>>>>> are available at
> >>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
> >>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/> . It
> >> is
> >>>>>> tagged with a local version “apache.incubating” so it allows
> upgrading
> >>>> from
> >>>>>> earlier releases. This beta is available for testing in a more
> >>>> production
> >>>>>> like setting (acceptance environment?).
> >>>>>>
> >>>>>> I would like to encourage everyone to try it out, to report back
> any
> >>>>>> issues so we get to a rock solid release of 1.8.0. When reporting
> >>>> issues a
> >>>>>> test case or even a fix is highly appreciated.
> >>>>>>
> >>>>>> Issues cleared:
> >>>>>>
> >>>>>> * Manual trigger not working
> >>>>>> * Performance issues with MySQL
> >>>>>> * Postgres auto-commit support left over to the driver
> >>>>>> * Poison pill taken while task has exited
> >>>>>> * Keep cgroups optional
> >>>>>> * Funcsigs pinned to 1.0.0
> >>>>>>
> >>>>>> Issue(s) remaining (blocker for RC):
> >>>>>> * Cgroups not py3 compatible
> >>>>>>
> >>>>>> If all goes well we should have a Release Candidate on Feb 2. Thanks
> >> for
> >>>>>> reporting issues and keep on testing please :). Moving towards RC I
> >>>> tend to
> >>>>>> like small bug fixes only. When we mark RC (do we need to vote on
> >> this?)
> >>>>>> the procedure becomes even more strict. Please remember that the
> FINAL
> >>>>>> release is dependent on a vote on the IPMC mailinglist.
> >>>>>>
> >>>>>> Cheers
> >>>>>> Bolke
> >>>>>
> >>>>> --
> >>>>> _/
> >>>>> _/ Alex Van Boxel
> >>>>
> >>> --
> >>> _/
> >>> _/ Alex Van Boxel
> >>
> >> --
> >> _/
> >> _/ Alex Van Boxel
> >>
> > --
> > _/
> > _/ Alex Van Boxel
>
> --
_/
_/ Alex Van Boxel
Re: Airflow 1.8.0 BETA 3
Posted by Bolke de Bruin <bd...@gmail.com>.
Hi Alex
Are you sure you are in a fully clean environment? Sometimes things remain in the airflow directories. This worked for me:
rm -rf site-packages/airflow*
python setup.py install
airflow initdb
Bolke
> On 26 Jan 2017, at 10:53, Alex Van Boxel <al...@vanboxel.be> wrote:
>
> About the database error: starting from scratch also gives the same error:
>
> Fresh install. Delete airflow.db sqllite db. And the : airflow initdb
>
> same error as above.
>
> On Thu, Jan 26, 2017 at 10:12 AM Alex Van Boxel <al...@vanboxel.be> wrote:
>
>> Not directly one I can share. I'll spend some time looking at it. I'll try
>> to create a unittest of it.
>>
>> On Thu, Jan 26, 2017 at 9:47 AM Bolke de Bruin <bd...@gmail.com> wrote:
>>
>> Do you have a example dag for this? That makes it easier to work on.
>>
>> Bolke
>>
>> Sent from my iPhone
>>
>>> On 26 Jan 2017, at 08:36, Alex Van Boxel <al...@vanboxel.be> wrote:
>>>
>>> Another thing that I noticed (but observed it in beta 2 as well). Is the
>>> following:
>>>
>>> - The following trigger should not fire.
>>> --- Trigger rule is ONE_SUCCESS
>>> --- upstream: UP_FOR_RETRY + SKIPPED
>>> => task get's triggered
>>> => resulting SKIPPED
>>> => DAG marked success, with half of the Tasks never scheduled
>>>
>>> - UP_FOR_RETRY is propagated downstream (actually resulting in failure
>>> described above)
>>> --- *Does this make sense* ?!
>>>
>>> Both are a problem, and certainly the combination of both. I'll see if I
>>> can spend some time investigating this.
>>>
>>>
>>>
>>>> On Thu, Jan 26, 2017 at 7:34 AM Bolke de Bruin <bd...@gmail.com>
>> wrote:
>>>>
>>>> Mmm that is due to the reverting of one changes to the db. Need to look
>>>> into that how to fix it.
>>>>
>>>> Sent from my iPhone
>>>>
>>>>> On 26 Jan 2017, at 00:51, Alex Van Boxel <al...@vanboxel.be> wrote:
>>>>>
>>>>> I do seem to have a problem upgrading the MySQL database with the last
>>>>> commit:
>>>>>
>>>>>
>>>>> 2017-01-25T23:41:55.662572654Z
>>>>>
>>>>
>> /usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/messaging.py:69:
>>>>> UserWarning: Revision 1a5a9e6bf2b5 referenced from 1a5a9e6bf2b5 ->
>>>>> 127d2bf2dfa7 (head), Add dag_id/state index on dag_run table is not
>>>> present
>>>>> 2017-01-25T23:41:55.662613670Z warnings.warn(msg)
>>>>> 2017-01-25T23:41:55.664560884Z Traceback (most recent call last):
>>>>> 2017-01-25T23:41:55.664582238Z File "/usr/local/bin/airflow", line 4,
>>>> in
>>>>> <module>
>>>>> 2017-01-25T23:41:55.664758457Z
>>>>>
>>>>
>> __import__('pkg_resources').run_script('airflow==1.8.0b1+apache.incubating',
>>>>> 'airflow')
>>>>> 2017-01-25T23:41:55.664776553Z File
>>>>> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py",
>> line
>>>>> 739, in run_script
>>>>> 2017-01-25T23:41:55.664937644Z
>>>>> self.require(requires)[0].run_script(script_name, ns)
>>>>> 2017-01-25T23:41:55.664952001Z File
>>>>> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py",
>> line
>>>>> 1494, in run_script
>>>>> 2017-01-25T23:41:55.665072840Z exec(code, namespace, namespace)
>>>>> 2017-01-25T23:41:55.665086884Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/EGG-INFO/scripts/airflow",
>>>>> line 28, in <module>
>>>>> 2017-01-25T23:41:55.665211755Z args.func(args)
>>>>> 2017-01-25T23:41:55.665225157Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/bin/cli.py",
>>>>> line 931, in upgradedb
>>>>> 2017-01-25T23:41:55.665326691Z db_utils.upgradedb()
>>>>> 2017-01-25T23:41:55.665346979Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/utils/db.py",
>>>>> line 292, in upgradedb
>>>>> 2017-01-25T23:41:55.665449297Z command.upgrade(config, 'heads')
>>>>> 2017-01-25T23:41:55.665462333Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
>>>>> line 174, in upgrade
>>>>> 2017-01-25T23:41:55.665521466Z script.run_env()
>>>>> 2017-01-25T23:41:55.665527232Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
>>>>> line 416, in run_env2017-01-25T23:41:55.665638879Z
>>>>> util.load_python_file(self.dir, 'env.py')
>>>>> 2017-01-25T23:41:55.665656437Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/pyfiles.py",
>>>>> line 93, in load_python_file
>>>>> 2017-01-25T23:41:55.665682760Z module = load_module_py(module_id,
>>>> path)
>>>>> 2017-01-25T23:41:55.665699257Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/compat.py",
>>>>> line 79, in load_module_py
>>>>> 2017-01-25T23:41:55.665725036Z mod = imp.load_source(module_id,
>> path,
>>>>> fp)
>>>>> 2017-01-25T23:41:55.665751433Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
>>>>> line 88, in <module>
>>>>> 2017-01-25T23:41:55.665778219Z run_migrations_online()
>>>>> 2017-01-25T23:41:55.665787820Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
>>>>> line 83, in run_migrations_online
>>>>> 2017-01-25T23:41:55.665811353Z context.run_migrations()
>>>>> 2017-01-25T23:41:55.665818933Z File "<string>", line 8, in
>>>> run_migrations
>>>>> 2017-01-25T23:41:55.666707547Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/environment.py",
>>>>> line 797, in run_migrations
>>>>> 2017-01-25T23:41:55.666839945Z
>>>> self.get_context().run_migrations(**kw)
>>>>> 2017-01-25T23:41:55.666859336Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/migration.py",
>>>>> line 305, in run_migrations
>>>>> 2017-01-25T23:41:55.666970791Z for step in
>> self._migrations_fn(heads,
>>>>> self):
>>>>> 2017-01-25T23:41:55.666992265Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
>>>>> line 163, in upgrade
>>>>> 2017-01-25T23:41:55.667021804Z return
>> script._upgrade_revs(revision,
>>>>> rev)
>>>>> 2017-01-25T23:41:55.667032496Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
>>>>> line 329, in _upgrade_revs
>>>>> 2017-01-25T23:41:55.667144295Z revs = list(revs)
>>>>> 2017-01-25T23:41:55.667157407Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>>>> line 641, in _iterate_revisions
>>>>> 2017-01-25T23:41:55.667302027Z requested_lowers =
>>>>> self.get_revisions(lower)
>>>>> 2017-01-25T23:41:55.667315913Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>>>> line 298, in get_revisions
>>>>> 2017-01-25T23:41:55.667390590Z return
>>>> sum([self.get_revisions(id_elem)
>>>>> for id_elem in id_], ())
>>>>> 2017-01-25T23:41:55.667411867Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>>>> line 300, in get_revisions
>>>>> 2017-01-25T23:41:55.667419226Z resolved_id, branch_label =
>>>>> self._resolve_revision_number(id_)
>>>>> 2017-01-25T23:41:55.667434110Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>>>> line 433, in _resolve_revision_number
>>>>> 2017-01-25T23:41:55.667576273Z self._revision_map
>>>>> 2017-01-25T23:41:55.667590641Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/langhelpers.py",
>>>>> line 240, in __get__
>>>>> 2017-01-25T23:41:55.667648561Z obj.__dict__[self.__name__] =
>> result =
>>>>> self.fget(obj)
>>>>> 2017-01-25T23:41:55.667658598Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>>>> line 151, in _revision_map
>>>>> 2017-01-25T23:41:55.667680311Z down_revision = map_[downrev]
>>>>> 2017-01-25T23:41:55.667746364Z KeyError: '1a5a9e6bf2b5'
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> On Wed, Jan 25, 2017 at 11:15 PM Bolke de Bruin <bd...@gmail.com>
>>>> wrote:
>>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> I have made the THIRD beta of Airflow 1.8.0 available at:
>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public
>>>> keys
>>>>>> are available at
>>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/> . It
>> is
>>>>>> tagged with a local version “apache.incubating” so it allows upgrading
>>>> from
>>>>>> earlier releases. This beta is available for testing in a more
>>>> production
>>>>>> like setting (acceptance environment?).
>>>>>>
>>>>>> I would like to encourage everyone to try it out, to report back any
>>>>>> issues so we get to a rock solid release of 1.8.0. When reporting
>>>> issues a
>>>>>> test case or even a fix is highly appreciated.
>>>>>>
>>>>>> Issues cleared:
>>>>>>
>>>>>> * Manual trigger not working
>>>>>> * Performance issues with MySQL
>>>>>> * Postgres auto-commit support left over to the driver
>>>>>> * Poison pill taken while task has exited
>>>>>> * Keep cgroups optional
>>>>>> * Funcsigs pinned to 1.0.0
>>>>>>
>>>>>> Issue(s) remaining (blocker for RC):
>>>>>> * Cgroups not py3 compatible
>>>>>>
>>>>>> If all goes well we should have a Release Candidate on Feb 2. Thanks
>> for
>>>>>> reporting issues and keep on testing please :). Moving towards RC I
>>>> tend to
>>>>>> like small bug fixes only. When we mark RC (do we need to vote on
>> this?)
>>>>>> the procedure becomes even more strict. Please remember that the FINAL
>>>>>> release is dependent on a vote on the IPMC mailinglist.
>>>>>>
>>>>>> Cheers
>>>>>> Bolke
>>>>>
>>>>> --
>>>>> _/
>>>>> _/ Alex Van Boxel
>>>>
>>> --
>>> _/
>>> _/ Alex Van Boxel
>>
>> --
>> _/
>> _/ Alex Van Boxel
>>
> --
> _/
> _/ Alex Van Boxel
Re: Airflow 1.8.0 BETA 3
Posted by Bolke de Bruin <bd...@gmail.com>.
I’ll have a look at it, and if needed release Beta 4.
- Bolke
> On 26 Jan 2017, at 10:53, Alex Van Boxel <al...@vanboxel.be> wrote:
>
> About the database error: starting from scratch also gives the same error:
>
> Fresh install. Delete airflow.db sqllite db. And the : airflow initdb
>
> same error as above.
>
> On Thu, Jan 26, 2017 at 10:12 AM Alex Van Boxel <al...@vanboxel.be> wrote:
>
>> Not directly one I can share. I'll spend some time looking at it. I'll try
>> to create a unittest of it.
>>
>> On Thu, Jan 26, 2017 at 9:47 AM Bolke de Bruin <bd...@gmail.com> wrote:
>>
>> Do you have a example dag for this? That makes it easier to work on.
>>
>> Bolke
>>
>> Sent from my iPhone
>>
>>> On 26 Jan 2017, at 08:36, Alex Van Boxel <al...@vanboxel.be> wrote:
>>>
>>> Another thing that I noticed (but observed it in beta 2 as well). Is the
>>> following:
>>>
>>> - The following trigger should not fire.
>>> --- Trigger rule is ONE_SUCCESS
>>> --- upstream: UP_FOR_RETRY + SKIPPED
>>> => task get's triggered
>>> => resulting SKIPPED
>>> => DAG marked success, with half of the Tasks never scheduled
>>>
>>> - UP_FOR_RETRY is propagated downstream (actually resulting in failure
>>> described above)
>>> --- *Does this make sense* ?!
>>>
>>> Both are a problem, and certainly the combination of both. I'll see if I
>>> can spend some time investigating this.
>>>
>>>
>>>
>>>> On Thu, Jan 26, 2017 at 7:34 AM Bolke de Bruin <bd...@gmail.com>
>> wrote:
>>>>
>>>> Mmm that is due to the reverting of one changes to the db. Need to look
>>>> into that how to fix it.
>>>>
>>>> Sent from my iPhone
>>>>
>>>>> On 26 Jan 2017, at 00:51, Alex Van Boxel <al...@vanboxel.be> wrote:
>>>>>
>>>>> I do seem to have a problem upgrading the MySQL database with the last
>>>>> commit:
>>>>>
>>>>>
>>>>> 2017-01-25T23:41:55.662572654Z
>>>>>
>>>>
>> /usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/messaging.py:69:
>>>>> UserWarning: Revision 1a5a9e6bf2b5 referenced from 1a5a9e6bf2b5 ->
>>>>> 127d2bf2dfa7 (head), Add dag_id/state index on dag_run table is not
>>>> present
>>>>> 2017-01-25T23:41:55.662613670Z warnings.warn(msg)
>>>>> 2017-01-25T23:41:55.664560884Z Traceback (most recent call last):
>>>>> 2017-01-25T23:41:55.664582238Z File "/usr/local/bin/airflow", line 4,
>>>> in
>>>>> <module>
>>>>> 2017-01-25T23:41:55.664758457Z
>>>>>
>>>>
>> __import__('pkg_resources').run_script('airflow==1.8.0b1+apache.incubating',
>>>>> 'airflow')
>>>>> 2017-01-25T23:41:55.664776553Z File
>>>>> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py",
>> line
>>>>> 739, in run_script
>>>>> 2017-01-25T23:41:55.664937644Z
>>>>> self.require(requires)[0].run_script(script_name, ns)
>>>>> 2017-01-25T23:41:55.664952001Z File
>>>>> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py",
>> line
>>>>> 1494, in run_script
>>>>> 2017-01-25T23:41:55.665072840Z exec(code, namespace, namespace)
>>>>> 2017-01-25T23:41:55.665086884Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/EGG-INFO/scripts/airflow",
>>>>> line 28, in <module>
>>>>> 2017-01-25T23:41:55.665211755Z args.func(args)
>>>>> 2017-01-25T23:41:55.665225157Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/bin/cli.py",
>>>>> line 931, in upgradedb
>>>>> 2017-01-25T23:41:55.665326691Z db_utils.upgradedb()
>>>>> 2017-01-25T23:41:55.665346979Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/utils/db.py",
>>>>> line 292, in upgradedb
>>>>> 2017-01-25T23:41:55.665449297Z command.upgrade(config, 'heads')
>>>>> 2017-01-25T23:41:55.665462333Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
>>>>> line 174, in upgrade
>>>>> 2017-01-25T23:41:55.665521466Z script.run_env()
>>>>> 2017-01-25T23:41:55.665527232Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
>>>>> line 416, in run_env2017-01-25T23:41:55.665638879Z
>>>>> util.load_python_file(self.dir, 'env.py')
>>>>> 2017-01-25T23:41:55.665656437Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/pyfiles.py",
>>>>> line 93, in load_python_file
>>>>> 2017-01-25T23:41:55.665682760Z module = load_module_py(module_id,
>>>> path)
>>>>> 2017-01-25T23:41:55.665699257Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/compat.py",
>>>>> line 79, in load_module_py
>>>>> 2017-01-25T23:41:55.665725036Z mod = imp.load_source(module_id,
>> path,
>>>>> fp)
>>>>> 2017-01-25T23:41:55.665751433Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
>>>>> line 88, in <module>
>>>>> 2017-01-25T23:41:55.665778219Z run_migrations_online()
>>>>> 2017-01-25T23:41:55.665787820Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
>>>>> line 83, in run_migrations_online
>>>>> 2017-01-25T23:41:55.665811353Z context.run_migrations()
>>>>> 2017-01-25T23:41:55.665818933Z File "<string>", line 8, in
>>>> run_migrations
>>>>> 2017-01-25T23:41:55.666707547Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/environment.py",
>>>>> line 797, in run_migrations
>>>>> 2017-01-25T23:41:55.666839945Z
>>>> self.get_context().run_migrations(**kw)
>>>>> 2017-01-25T23:41:55.666859336Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/migration.py",
>>>>> line 305, in run_migrations
>>>>> 2017-01-25T23:41:55.666970791Z for step in
>> self._migrations_fn(heads,
>>>>> self):
>>>>> 2017-01-25T23:41:55.666992265Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
>>>>> line 163, in upgrade
>>>>> 2017-01-25T23:41:55.667021804Z return
>> script._upgrade_revs(revision,
>>>>> rev)
>>>>> 2017-01-25T23:41:55.667032496Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
>>>>> line 329, in _upgrade_revs
>>>>> 2017-01-25T23:41:55.667144295Z revs = list(revs)
>>>>> 2017-01-25T23:41:55.667157407Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>>>> line 641, in _iterate_revisions
>>>>> 2017-01-25T23:41:55.667302027Z requested_lowers =
>>>>> self.get_revisions(lower)
>>>>> 2017-01-25T23:41:55.667315913Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>>>> line 298, in get_revisions
>>>>> 2017-01-25T23:41:55.667390590Z return
>>>> sum([self.get_revisions(id_elem)
>>>>> for id_elem in id_], ())
>>>>> 2017-01-25T23:41:55.667411867Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>>>> line 300, in get_revisions
>>>>> 2017-01-25T23:41:55.667419226Z resolved_id, branch_label =
>>>>> self._resolve_revision_number(id_)
>>>>> 2017-01-25T23:41:55.667434110Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>>>> line 433, in _resolve_revision_number
>>>>> 2017-01-25T23:41:55.667576273Z self._revision_map
>>>>> 2017-01-25T23:41:55.667590641Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/langhelpers.py",
>>>>> line 240, in __get__
>>>>> 2017-01-25T23:41:55.667648561Z obj.__dict__[self.__name__] =
>> result =
>>>>> self.fget(obj)
>>>>> 2017-01-25T23:41:55.667658598Z File
>>>>>
>>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>>>> line 151, in _revision_map
>>>>> 2017-01-25T23:41:55.667680311Z down_revision = map_[downrev]
>>>>> 2017-01-25T23:41:55.667746364Z KeyError: '1a5a9e6bf2b5'
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> On Wed, Jan 25, 2017 at 11:15 PM Bolke de Bruin <bd...@gmail.com>
>>>> wrote:
>>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> I have made the THIRD beta of Airflow 1.8.0 available at:
>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public
>>>> keys
>>>>>> are available at
>>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/> . It
>> is
>>>>>> tagged with a local version “apache.incubating” so it allows upgrading
>>>> from
>>>>>> earlier releases. This beta is available for testing in a more
>>>> production
>>>>>> like setting (acceptance environment?).
>>>>>>
>>>>>> I would like to encourage everyone to try it out, to report back any
>>>>>> issues so we get to a rock solid release of 1.8.0. When reporting
>>>> issues a
>>>>>> test case or even a fix is highly appreciated.
>>>>>>
>>>>>> Issues cleared:
>>>>>>
>>>>>> * Manual trigger not working
>>>>>> * Performance issues with MySQL
>>>>>> * Postgres auto-commit support left over to the driver
>>>>>> * Poison pill taken while task has exited
>>>>>> * Keep cgroups optional
>>>>>> * Funcsigs pinned to 1.0.0
>>>>>>
>>>>>> Issue(s) remaining (blocker for RC):
>>>>>> * Cgroups not py3 compatible
>>>>>>
>>>>>> If all goes well we should have a Release Candidate on Feb 2. Thanks
>> for
>>>>>> reporting issues and keep on testing please :). Moving towards RC I
>>>> tend to
>>>>>> like small bug fixes only. When we mark RC (do we need to vote on
>> this?)
>>>>>> the procedure becomes even more strict. Please remember that the FINAL
>>>>>> release is dependent on a vote on the IPMC mailinglist.
>>>>>>
>>>>>> Cheers
>>>>>> Bolke
>>>>>
>>>>> --
>>>>> _/
>>>>> _/ Alex Van Boxel
>>>>
>>> --
>>> _/
>>> _/ Alex Van Boxel
>>
>> --
>> _/
>> _/ Alex Van Boxel
>>
> --
> _/
> _/ Alex Van Boxel
Re: Airflow 1.8.0 BETA 3
Posted by Alex Van Boxel <al...@vanboxel.be>.
About the database error: starting from scratch also gives the same error:
Fresh install. Delete airflow.db sqllite db. And the : airflow initdb
same error as above.
On Thu, Jan 26, 2017 at 10:12 AM Alex Van Boxel <al...@vanboxel.be> wrote:
> Not directly one I can share. I'll spend some time looking at it. I'll try
> to create a unittest of it.
>
> On Thu, Jan 26, 2017 at 9:47 AM Bolke de Bruin <bd...@gmail.com> wrote:
>
> Do you have a example dag for this? That makes it easier to work on.
>
> Bolke
>
> Sent from my iPhone
>
> > On 26 Jan 2017, at 08:36, Alex Van Boxel <al...@vanboxel.be> wrote:
> >
> > Another thing that I noticed (but observed it in beta 2 as well). Is the
> > following:
> >
> > - The following trigger should not fire.
> > --- Trigger rule is ONE_SUCCESS
> > --- upstream: UP_FOR_RETRY + SKIPPED
> > => task get's triggered
> > => resulting SKIPPED
> > => DAG marked success, with half of the Tasks never scheduled
> >
> > - UP_FOR_RETRY is propagated downstream (actually resulting in failure
> > described above)
> > --- *Does this make sense* ?!
> >
> > Both are a problem, and certainly the combination of both. I'll see if I
> > can spend some time investigating this.
> >
> >
> >
> >> On Thu, Jan 26, 2017 at 7:34 AM Bolke de Bruin <bd...@gmail.com>
> wrote:
> >>
> >> Mmm that is due to the reverting of one changes to the db. Need to look
> >> into that how to fix it.
> >>
> >> Sent from my iPhone
> >>
> >>> On 26 Jan 2017, at 00:51, Alex Van Boxel <al...@vanboxel.be> wrote:
> >>>
> >>> I do seem to have a problem upgrading the MySQL database with the last
> >>> commit:
> >>>
> >>>
> >>> 2017-01-25T23:41:55.662572654Z
> >>>
> >>
> /usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/messaging.py:69:
> >>> UserWarning: Revision 1a5a9e6bf2b5 referenced from 1a5a9e6bf2b5 ->
> >>> 127d2bf2dfa7 (head), Add dag_id/state index on dag_run table is not
> >> present
> >>> 2017-01-25T23:41:55.662613670Z warnings.warn(msg)
> >>> 2017-01-25T23:41:55.664560884Z Traceback (most recent call last):
> >>> 2017-01-25T23:41:55.664582238Z File "/usr/local/bin/airflow", line 4,
> >> in
> >>> <module>
> >>> 2017-01-25T23:41:55.664758457Z
> >>>
> >>
> __import__('pkg_resources').run_script('airflow==1.8.0b1+apache.incubating',
> >>> 'airflow')
> >>> 2017-01-25T23:41:55.664776553Z File
> >>> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py",
> line
> >>> 739, in run_script
> >>> 2017-01-25T23:41:55.664937644Z
> >>> self.require(requires)[0].run_script(script_name, ns)
> >>> 2017-01-25T23:41:55.664952001Z File
> >>> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py",
> line
> >>> 1494, in run_script
> >>> 2017-01-25T23:41:55.665072840Z exec(code, namespace, namespace)
> >>> 2017-01-25T23:41:55.665086884Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/EGG-INFO/scripts/airflow",
> >>> line 28, in <module>
> >>> 2017-01-25T23:41:55.665211755Z args.func(args)
> >>> 2017-01-25T23:41:55.665225157Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/bin/cli.py",
> >>> line 931, in upgradedb
> >>> 2017-01-25T23:41:55.665326691Z db_utils.upgradedb()
> >>> 2017-01-25T23:41:55.665346979Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/utils/db.py",
> >>> line 292, in upgradedb
> >>> 2017-01-25T23:41:55.665449297Z command.upgrade(config, 'heads')
> >>> 2017-01-25T23:41:55.665462333Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
> >>> line 174, in upgrade
> >>> 2017-01-25T23:41:55.665521466Z script.run_env()
> >>> 2017-01-25T23:41:55.665527232Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
> >>> line 416, in run_env2017-01-25T23:41:55.665638879Z
> >>> util.load_python_file(self.dir, 'env.py')
> >>> 2017-01-25T23:41:55.665656437Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/pyfiles.py",
> >>> line 93, in load_python_file
> >>> 2017-01-25T23:41:55.665682760Z module = load_module_py(module_id,
> >> path)
> >>> 2017-01-25T23:41:55.665699257Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/compat.py",
> >>> line 79, in load_module_py
> >>> 2017-01-25T23:41:55.665725036Z mod = imp.load_source(module_id,
> path,
> >>> fp)
> >>> 2017-01-25T23:41:55.665751433Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
> >>> line 88, in <module>
> >>> 2017-01-25T23:41:55.665778219Z run_migrations_online()
> >>> 2017-01-25T23:41:55.665787820Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
> >>> line 83, in run_migrations_online
> >>> 2017-01-25T23:41:55.665811353Z context.run_migrations()
> >>> 2017-01-25T23:41:55.665818933Z File "<string>", line 8, in
> >> run_migrations
> >>> 2017-01-25T23:41:55.666707547Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/environment.py",
> >>> line 797, in run_migrations
> >>> 2017-01-25T23:41:55.666839945Z
> >> self.get_context().run_migrations(**kw)
> >>> 2017-01-25T23:41:55.666859336Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/migration.py",
> >>> line 305, in run_migrations
> >>> 2017-01-25T23:41:55.666970791Z for step in
> self._migrations_fn(heads,
> >>> self):
> >>> 2017-01-25T23:41:55.666992265Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
> >>> line 163, in upgrade
> >>> 2017-01-25T23:41:55.667021804Z return
> script._upgrade_revs(revision,
> >>> rev)
> >>> 2017-01-25T23:41:55.667032496Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
> >>> line 329, in _upgrade_revs
> >>> 2017-01-25T23:41:55.667144295Z revs = list(revs)
> >>> 2017-01-25T23:41:55.667157407Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> >>> line 641, in _iterate_revisions
> >>> 2017-01-25T23:41:55.667302027Z requested_lowers =
> >>> self.get_revisions(lower)
> >>> 2017-01-25T23:41:55.667315913Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> >>> line 298, in get_revisions
> >>> 2017-01-25T23:41:55.667390590Z return
> >> sum([self.get_revisions(id_elem)
> >>> for id_elem in id_], ())
> >>> 2017-01-25T23:41:55.667411867Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> >>> line 300, in get_revisions
> >>> 2017-01-25T23:41:55.667419226Z resolved_id, branch_label =
> >>> self._resolve_revision_number(id_)
> >>> 2017-01-25T23:41:55.667434110Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> >>> line 433, in _resolve_revision_number
> >>> 2017-01-25T23:41:55.667576273Z self._revision_map
> >>> 2017-01-25T23:41:55.667590641Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/langhelpers.py",
> >>> line 240, in __get__
> >>> 2017-01-25T23:41:55.667648561Z obj.__dict__[self.__name__] =
> result =
> >>> self.fget(obj)
> >>> 2017-01-25T23:41:55.667658598Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> >>> line 151, in _revision_map
> >>> 2017-01-25T23:41:55.667680311Z down_revision = map_[downrev]
> >>> 2017-01-25T23:41:55.667746364Z KeyError: '1a5a9e6bf2b5'
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>> On Wed, Jan 25, 2017 at 11:15 PM Bolke de Bruin <bd...@gmail.com>
> >> wrote:
> >>>>
> >>>> Hi All,
> >>>>
> >>>> I have made the THIRD beta of Airflow 1.8.0 available at:
> >>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> >>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public
> >> keys
> >>>> are available at
> >>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
> >>>> https://dist.apache.org/repos/dist/release/incubator/airflow/> . It
> is
> >>>> tagged with a local version “apache.incubating” so it allows upgrading
> >> from
> >>>> earlier releases. This beta is available for testing in a more
> >> production
> >>>> like setting (acceptance environment?).
> >>>>
> >>>> I would like to encourage everyone to try it out, to report back any
> >>>> issues so we get to a rock solid release of 1.8.0. When reporting
> >> issues a
> >>>> test case or even a fix is highly appreciated.
> >>>>
> >>>> Issues cleared:
> >>>>
> >>>> * Manual trigger not working
> >>>> * Performance issues with MySQL
> >>>> * Postgres auto-commit support left over to the driver
> >>>> * Poison pill taken while task has exited
> >>>> * Keep cgroups optional
> >>>> * Funcsigs pinned to 1.0.0
> >>>>
> >>>> Issue(s) remaining (blocker for RC):
> >>>> * Cgroups not py3 compatible
> >>>>
> >>>> If all goes well we should have a Release Candidate on Feb 2. Thanks
> for
> >>>> reporting issues and keep on testing please :). Moving towards RC I
> >> tend to
> >>>> like small bug fixes only. When we mark RC (do we need to vote on
> this?)
> >>>> the procedure becomes even more strict. Please remember that the FINAL
> >>>> release is dependent on a vote on the IPMC mailinglist.
> >>>>
> >>>> Cheers
> >>>> Bolke
> >>>
> >>> --
> >>> _/
> >>> _/ Alex Van Boxel
> >>
> > --
> > _/
> > _/ Alex Van Boxel
>
> --
> _/
> _/ Alex Van Boxel
>
--
_/
_/ Alex Van Boxel
Re: Airflow 1.8.0 BETA 3
Posted by Alex Van Boxel <al...@vanboxel.be>.
Not directly one I can share. I'll spend some time looking at it. I'll try
to create a unittest of it.
On Thu, Jan 26, 2017 at 9:47 AM Bolke de Bruin <bd...@gmail.com> wrote:
> Do you have a example dag for this? That makes it easier to work on.
>
> Bolke
>
> Sent from my iPhone
>
> > On 26 Jan 2017, at 08:36, Alex Van Boxel <al...@vanboxel.be> wrote:
> >
> > Another thing that I noticed (but observed it in beta 2 as well). Is the
> > following:
> >
> > - The following trigger should not fire.
> > --- Trigger rule is ONE_SUCCESS
> > --- upstream: UP_FOR_RETRY + SKIPPED
> > => task get's triggered
> > => resulting SKIPPED
> > => DAG marked success, with half of the Tasks never scheduled
> >
> > - UP_FOR_RETRY is propagated downstream (actually resulting in failure
> > described above)
> > --- *Does this make sense* ?!
> >
> > Both are a problem, and certainly the combination of both. I'll see if I
> > can spend some time investigating this.
> >
> >
> >
> >> On Thu, Jan 26, 2017 at 7:34 AM Bolke de Bruin <bd...@gmail.com>
> wrote:
> >>
> >> Mmm that is due to the reverting of one changes to the db. Need to look
> >> into that how to fix it.
> >>
> >> Sent from my iPhone
> >>
> >>> On 26 Jan 2017, at 00:51, Alex Van Boxel <al...@vanboxel.be> wrote:
> >>>
> >>> I do seem to have a problem upgrading the MySQL database with the last
> >>> commit:
> >>>
> >>>
> >>> 2017-01-25T23:41:55.662572654Z
> >>>
> >>
> /usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/messaging.py:69:
> >>> UserWarning: Revision 1a5a9e6bf2b5 referenced from 1a5a9e6bf2b5 ->
> >>> 127d2bf2dfa7 (head), Add dag_id/state index on dag_run table is not
> >> present
> >>> 2017-01-25T23:41:55.662613670Z warnings.warn(msg)
> >>> 2017-01-25T23:41:55.664560884Z Traceback (most recent call last):
> >>> 2017-01-25T23:41:55.664582238Z File "/usr/local/bin/airflow", line 4,
> >> in
> >>> <module>
> >>> 2017-01-25T23:41:55.664758457Z
> >>>
> >>
> __import__('pkg_resources').run_script('airflow==1.8.0b1+apache.incubating',
> >>> 'airflow')
> >>> 2017-01-25T23:41:55.664776553Z File
> >>> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py",
> line
> >>> 739, in run_script
> >>> 2017-01-25T23:41:55.664937644Z
> >>> self.require(requires)[0].run_script(script_name, ns)
> >>> 2017-01-25T23:41:55.664952001Z File
> >>> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py",
> line
> >>> 1494, in run_script
> >>> 2017-01-25T23:41:55.665072840Z exec(code, namespace, namespace)
> >>> 2017-01-25T23:41:55.665086884Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/EGG-INFO/scripts/airflow",
> >>> line 28, in <module>
> >>> 2017-01-25T23:41:55.665211755Z args.func(args)
> >>> 2017-01-25T23:41:55.665225157Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/bin/cli.py",
> >>> line 931, in upgradedb
> >>> 2017-01-25T23:41:55.665326691Z db_utils.upgradedb()
> >>> 2017-01-25T23:41:55.665346979Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/utils/db.py",
> >>> line 292, in upgradedb
> >>> 2017-01-25T23:41:55.665449297Z command.upgrade(config, 'heads')
> >>> 2017-01-25T23:41:55.665462333Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
> >>> line 174, in upgrade
> >>> 2017-01-25T23:41:55.665521466Z script.run_env()
> >>> 2017-01-25T23:41:55.665527232Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
> >>> line 416, in run_env2017-01-25T23:41:55.665638879Z
> >>> util.load_python_file(self.dir, 'env.py')
> >>> 2017-01-25T23:41:55.665656437Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/pyfiles.py",
> >>> line 93, in load_python_file
> >>> 2017-01-25T23:41:55.665682760Z module = load_module_py(module_id,
> >> path)
> >>> 2017-01-25T23:41:55.665699257Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/compat.py",
> >>> line 79, in load_module_py
> >>> 2017-01-25T23:41:55.665725036Z mod = imp.load_source(module_id,
> path,
> >>> fp)
> >>> 2017-01-25T23:41:55.665751433Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
> >>> line 88, in <module>
> >>> 2017-01-25T23:41:55.665778219Z run_migrations_online()
> >>> 2017-01-25T23:41:55.665787820Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
> >>> line 83, in run_migrations_online
> >>> 2017-01-25T23:41:55.665811353Z context.run_migrations()
> >>> 2017-01-25T23:41:55.665818933Z File "<string>", line 8, in
> >> run_migrations
> >>> 2017-01-25T23:41:55.666707547Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/environment.py",
> >>> line 797, in run_migrations
> >>> 2017-01-25T23:41:55.666839945Z
> >> self.get_context().run_migrations(**kw)
> >>> 2017-01-25T23:41:55.666859336Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/migration.py",
> >>> line 305, in run_migrations
> >>> 2017-01-25T23:41:55.666970791Z for step in
> self._migrations_fn(heads,
> >>> self):
> >>> 2017-01-25T23:41:55.666992265Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
> >>> line 163, in upgrade
> >>> 2017-01-25T23:41:55.667021804Z return
> script._upgrade_revs(revision,
> >>> rev)
> >>> 2017-01-25T23:41:55.667032496Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
> >>> line 329, in _upgrade_revs
> >>> 2017-01-25T23:41:55.667144295Z revs = list(revs)
> >>> 2017-01-25T23:41:55.667157407Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> >>> line 641, in _iterate_revisions
> >>> 2017-01-25T23:41:55.667302027Z requested_lowers =
> >>> self.get_revisions(lower)
> >>> 2017-01-25T23:41:55.667315913Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> >>> line 298, in get_revisions
> >>> 2017-01-25T23:41:55.667390590Z return
> >> sum([self.get_revisions(id_elem)
> >>> for id_elem in id_], ())
> >>> 2017-01-25T23:41:55.667411867Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> >>> line 300, in get_revisions
> >>> 2017-01-25T23:41:55.667419226Z resolved_id, branch_label =
> >>> self._resolve_revision_number(id_)
> >>> 2017-01-25T23:41:55.667434110Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> >>> line 433, in _resolve_revision_number
> >>> 2017-01-25T23:41:55.667576273Z self._revision_map
> >>> 2017-01-25T23:41:55.667590641Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/langhelpers.py",
> >>> line 240, in __get__
> >>> 2017-01-25T23:41:55.667648561Z obj.__dict__[self.__name__] =
> result =
> >>> self.fget(obj)
> >>> 2017-01-25T23:41:55.667658598Z File
> >>>
> >>
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> >>> line 151, in _revision_map
> >>> 2017-01-25T23:41:55.667680311Z down_revision = map_[downrev]
> >>> 2017-01-25T23:41:55.667746364Z KeyError: '1a5a9e6bf2b5'
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>> On Wed, Jan 25, 2017 at 11:15 PM Bolke de Bruin <bd...@gmail.com>
> >> wrote:
> >>>>
> >>>> Hi All,
> >>>>
> >>>> I have made the THIRD beta of Airflow 1.8.0 available at:
> >>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> >>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public
> >> keys
> >>>> are available at
> >>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
> >>>> https://dist.apache.org/repos/dist/release/incubator/airflow/> . It
> is
> >>>> tagged with a local version “apache.incubating” so it allows upgrading
> >> from
> >>>> earlier releases. This beta is available for testing in a more
> >> production
> >>>> like setting (acceptance environment?).
> >>>>
> >>>> I would like to encourage everyone to try it out, to report back any
> >>>> issues so we get to a rock solid release of 1.8.0. When reporting
> >> issues a
> >>>> test case or even a fix is highly appreciated.
> >>>>
> >>>> Issues cleared:
> >>>>
> >>>> * Manual trigger not working
> >>>> * Performance issues with MySQL
> >>>> * Postgres auto-commit support left over to the driver
> >>>> * Poison pill taken while task has exited
> >>>> * Keep cgroups optional
> >>>> * Funcsigs pinned to 1.0.0
> >>>>
> >>>> Issue(s) remaining (blocker for RC):
> >>>> * Cgroups not py3 compatible
> >>>>
> >>>> If all goes well we should have a Release Candidate on Feb 2. Thanks
> for
> >>>> reporting issues and keep on testing please :). Moving towards RC I
> >> tend to
> >>>> like small bug fixes only. When we mark RC (do we need to vote on
> this?)
> >>>> the procedure becomes even more strict. Please remember that the FINAL
> >>>> release is dependent on a vote on the IPMC mailinglist.
> >>>>
> >>>> Cheers
> >>>> Bolke
> >>>
> >>> --
> >>> _/
> >>> _/ Alex Van Boxel
> >>
> > --
> > _/
> > _/ Alex Van Boxel
>
--
_/
_/ Alex Van Boxel
Re: Airflow 1.8.0 BETA 3
Posted by Bolke de Bruin <bd...@gmail.com>.
Do you have a example dag for this? That makes it easier to work on.
Bolke
Sent from my iPhone
> On 26 Jan 2017, at 08:36, Alex Van Boxel <al...@vanboxel.be> wrote:
>
> Another thing that I noticed (but observed it in beta 2 as well). Is the
> following:
>
> - The following trigger should not fire.
> --- Trigger rule is ONE_SUCCESS
> --- upstream: UP_FOR_RETRY + SKIPPED
> => task get's triggered
> => resulting SKIPPED
> => DAG marked success, with half of the Tasks never scheduled
>
> - UP_FOR_RETRY is propagated downstream (actually resulting in failure
> described above)
> --- *Does this make sense* ?!
>
> Both are a problem, and certainly the combination of both. I'll see if I
> can spend some time investigating this.
>
>
>
>> On Thu, Jan 26, 2017 at 7:34 AM Bolke de Bruin <bd...@gmail.com> wrote:
>>
>> Mmm that is due to the reverting of one changes to the db. Need to look
>> into that how to fix it.
>>
>> Sent from my iPhone
>>
>>> On 26 Jan 2017, at 00:51, Alex Van Boxel <al...@vanboxel.be> wrote:
>>>
>>> I do seem to have a problem upgrading the MySQL database with the last
>>> commit:
>>>
>>>
>>> 2017-01-25T23:41:55.662572654Z
>>>
>> /usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/messaging.py:69:
>>> UserWarning: Revision 1a5a9e6bf2b5 referenced from 1a5a9e6bf2b5 ->
>>> 127d2bf2dfa7 (head), Add dag_id/state index on dag_run table is not
>> present
>>> 2017-01-25T23:41:55.662613670Z warnings.warn(msg)
>>> 2017-01-25T23:41:55.664560884Z Traceback (most recent call last):
>>> 2017-01-25T23:41:55.664582238Z File "/usr/local/bin/airflow", line 4,
>> in
>>> <module>
>>> 2017-01-25T23:41:55.664758457Z
>>>
>> __import__('pkg_resources').run_script('airflow==1.8.0b1+apache.incubating',
>>> 'airflow')
>>> 2017-01-25T23:41:55.664776553Z File
>>> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line
>>> 739, in run_script
>>> 2017-01-25T23:41:55.664937644Z
>>> self.require(requires)[0].run_script(script_name, ns)
>>> 2017-01-25T23:41:55.664952001Z File
>>> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line
>>> 1494, in run_script
>>> 2017-01-25T23:41:55.665072840Z exec(code, namespace, namespace)
>>> 2017-01-25T23:41:55.665086884Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/EGG-INFO/scripts/airflow",
>>> line 28, in <module>
>>> 2017-01-25T23:41:55.665211755Z args.func(args)
>>> 2017-01-25T23:41:55.665225157Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/bin/cli.py",
>>> line 931, in upgradedb
>>> 2017-01-25T23:41:55.665326691Z db_utils.upgradedb()
>>> 2017-01-25T23:41:55.665346979Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/utils/db.py",
>>> line 292, in upgradedb
>>> 2017-01-25T23:41:55.665449297Z command.upgrade(config, 'heads')
>>> 2017-01-25T23:41:55.665462333Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
>>> line 174, in upgrade
>>> 2017-01-25T23:41:55.665521466Z script.run_env()
>>> 2017-01-25T23:41:55.665527232Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
>>> line 416, in run_env2017-01-25T23:41:55.665638879Z
>>> util.load_python_file(self.dir, 'env.py')
>>> 2017-01-25T23:41:55.665656437Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/pyfiles.py",
>>> line 93, in load_python_file
>>> 2017-01-25T23:41:55.665682760Z module = load_module_py(module_id,
>> path)
>>> 2017-01-25T23:41:55.665699257Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/compat.py",
>>> line 79, in load_module_py
>>> 2017-01-25T23:41:55.665725036Z mod = imp.load_source(module_id, path,
>>> fp)
>>> 2017-01-25T23:41:55.665751433Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
>>> line 88, in <module>
>>> 2017-01-25T23:41:55.665778219Z run_migrations_online()
>>> 2017-01-25T23:41:55.665787820Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
>>> line 83, in run_migrations_online
>>> 2017-01-25T23:41:55.665811353Z context.run_migrations()
>>> 2017-01-25T23:41:55.665818933Z File "<string>", line 8, in
>> run_migrations
>>> 2017-01-25T23:41:55.666707547Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/environment.py",
>>> line 797, in run_migrations
>>> 2017-01-25T23:41:55.666839945Z
>> self.get_context().run_migrations(**kw)
>>> 2017-01-25T23:41:55.666859336Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/migration.py",
>>> line 305, in run_migrations
>>> 2017-01-25T23:41:55.666970791Z for step in self._migrations_fn(heads,
>>> self):
>>> 2017-01-25T23:41:55.666992265Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
>>> line 163, in upgrade
>>> 2017-01-25T23:41:55.667021804Z return script._upgrade_revs(revision,
>>> rev)
>>> 2017-01-25T23:41:55.667032496Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
>>> line 329, in _upgrade_revs
>>> 2017-01-25T23:41:55.667144295Z revs = list(revs)
>>> 2017-01-25T23:41:55.667157407Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>> line 641, in _iterate_revisions
>>> 2017-01-25T23:41:55.667302027Z requested_lowers =
>>> self.get_revisions(lower)
>>> 2017-01-25T23:41:55.667315913Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>> line 298, in get_revisions
>>> 2017-01-25T23:41:55.667390590Z return
>> sum([self.get_revisions(id_elem)
>>> for id_elem in id_], ())
>>> 2017-01-25T23:41:55.667411867Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>> line 300, in get_revisions
>>> 2017-01-25T23:41:55.667419226Z resolved_id, branch_label =
>>> self._resolve_revision_number(id_)
>>> 2017-01-25T23:41:55.667434110Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>> line 433, in _resolve_revision_number
>>> 2017-01-25T23:41:55.667576273Z self._revision_map
>>> 2017-01-25T23:41:55.667590641Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/langhelpers.py",
>>> line 240, in __get__
>>> 2017-01-25T23:41:55.667648561Z obj.__dict__[self.__name__] = result =
>>> self.fget(obj)
>>> 2017-01-25T23:41:55.667658598Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>> line 151, in _revision_map
>>> 2017-01-25T23:41:55.667680311Z down_revision = map_[downrev]
>>> 2017-01-25T23:41:55.667746364Z KeyError: '1a5a9e6bf2b5'
>>>
>>>
>>>
>>>
>>>
>>>
>>>> On Wed, Jan 25, 2017 at 11:15 PM Bolke de Bruin <bd...@gmail.com>
>> wrote:
>>>>
>>>> Hi All,
>>>>
>>>> I have made the THIRD beta of Airflow 1.8.0 available at:
>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public
>> keys
>>>> are available at
>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/> . It is
>>>> tagged with a local version “apache.incubating” so it allows upgrading
>> from
>>>> earlier releases. This beta is available for testing in a more
>> production
>>>> like setting (acceptance environment?).
>>>>
>>>> I would like to encourage everyone to try it out, to report back any
>>>> issues so we get to a rock solid release of 1.8.0. When reporting
>> issues a
>>>> test case or even a fix is highly appreciated.
>>>>
>>>> Issues cleared:
>>>>
>>>> * Manual trigger not working
>>>> * Performance issues with MySQL
>>>> * Postgres auto-commit support left over to the driver
>>>> * Poison pill taken while task has exited
>>>> * Keep cgroups optional
>>>> * Funcsigs pinned to 1.0.0
>>>>
>>>> Issue(s) remaining (blocker for RC):
>>>> * Cgroups not py3 compatible
>>>>
>>>> If all goes well we should have a Release Candidate on Feb 2. Thanks for
>>>> reporting issues and keep on testing please :). Moving towards RC I
>> tend to
>>>> like small bug fixes only. When we mark RC (do we need to vote on this?)
>>>> the procedure becomes even more strict. Please remember that the FINAL
>>>> release is dependent on a vote on the IPMC mailinglist.
>>>>
>>>> Cheers
>>>> Bolke
>>>
>>> --
>>> _/
>>> _/ Alex Van Boxel
>>
> --
> _/
> _/ Alex Van Boxel
Re: Airflow 1.8.0 BETA 3
Posted by Bolke de Bruin <bd...@gmail.com>.
It is only an issue when you go from beta 2 to beta 3. Not if you upgrade from 1.7.1.
Sent from my iPhone
> On 26 Jan 2017, at 08:38, Alex Van Boxel <al...@vanboxel.be> wrote:
>
> The downgrade is ok for testing, but we can't release with this change
> (can't expect people to install a beta first).
>
>> On Thu, Jan 26, 2017 at 8:36 AM Alex Van Boxel <al...@vanboxel.be> wrote:
>>
>> Another thing that I noticed (but observed it in beta 2 as well). Is the
>> following:
>>
>> - The following trigger should not fire.
>> --- Trigger rule is ONE_SUCCESS
>> --- upstream: UP_FOR_RETRY + SKIPPED
>> => task get's triggered
>> => resulting SKIPPED
>> => DAG marked success, with half of the Tasks never scheduled
>>
>> - UP_FOR_RETRY is propagated downstream (actually resulting in failure
>> described above)
>> --- *Does this make sense* ?!
>>
>> Both are a problem, and certainly the combination of both. I'll see if I
>> can spend some time investigating this.
>>
>>
>>
>> On Thu, Jan 26, 2017 at 7:34 AM Bolke de Bruin <bd...@gmail.com> wrote:
>>
>> Mmm that is due to the reverting of one changes to the db. Need to look
>> into that how to fix it.
>>
>> Sent from my iPhone
>>
>>> On 26 Jan 2017, at 00:51, Alex Van Boxel <al...@vanboxel.be> wrote:
>>>
>>> I do seem to have a problem upgrading the MySQL database with the last
>>> commit:
>>>
>>>
>>> 2017-01-25T23:41:55.662572654Z
>>>
>> /usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/messaging.py:69:
>>> UserWarning: Revision 1a5a9e6bf2b5 referenced from 1a5a9e6bf2b5 ->
>>> 127d2bf2dfa7 (head), Add dag_id/state index on dag_run table is not
>> present
>>> 2017-01-25T23:41:55.662613670Z warnings.warn(msg)
>>> 2017-01-25T23:41:55.664560884Z Traceback (most recent call last):
>>> 2017-01-25T23:41:55.664582238Z File "/usr/local/bin/airflow", line 4,
>> in
>>> <module>
>>> 2017-01-25T23:41:55.664758457Z
>>>
>> __import__('pkg_resources').run_script('airflow==1.8.0b1+apache.incubating',
>>> 'airflow')
>>> 2017-01-25T23:41:55.664776553Z File
>>> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line
>>> 739, in run_script
>>> 2017-01-25T23:41:55.664937644Z
>>> self.require(requires)[0].run_script(script_name, ns)
>>> 2017-01-25T23:41:55.664952001Z File
>>> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line
>>> 1494, in run_script
>>> 2017-01-25T23:41:55.665072840Z exec(code, namespace, namespace)
>>> 2017-01-25T23:41:55.665086884Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/EGG-INFO/scripts/airflow",
>>> line 28, in <module>
>>> 2017-01-25T23:41:55.665211755Z args.func(args)
>>> 2017-01-25T23:41:55.665225157Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/bin/cli.py",
>>> line 931, in upgradedb
>>> 2017-01-25T23:41:55.665326691Z db_utils.upgradedb()
>>> 2017-01-25T23:41:55.665346979Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/utils/db.py",
>>> line 292, in upgradedb
>>> 2017-01-25T23:41:55.665449297Z command.upgrade(config, 'heads')
>>> 2017-01-25T23:41:55.665462333Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
>>> line 174, in upgrade
>>> 2017-01-25T23:41:55.665521466Z script.run_env()
>>> 2017-01-25T23:41:55.665527232Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
>>> line 416, in run_env2017-01-25T23:41:55.665638879Z
>>> util.load_python_file(self.dir, 'env.py')
>>> 2017-01-25T23:41:55.665656437Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/pyfiles.py",
>>> line 93, in load_python_file
>>> 2017-01-25T23:41:55.665682760Z module = load_module_py(module_id,
>> path)
>>> 2017-01-25T23:41:55.665699257Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/compat.py",
>>> line 79, in load_module_py
>>> 2017-01-25T23:41:55.665725036Z mod = imp.load_source(module_id, path,
>>> fp)
>>> 2017-01-25T23:41:55.665751433Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
>>> line 88, in <module>
>>> 2017-01-25T23:41:55.665778219Z run_migrations_online()
>>> 2017-01-25T23:41:55.665787820Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
>>> line 83, in run_migrations_online
>>> 2017-01-25T23:41:55.665811353Z context.run_migrations()
>>> 2017-01-25T23:41:55.665818933Z File "<string>", line 8, in
>> run_migrations
>>> 2017-01-25T23:41:55.666707547Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/environment.py",
>>> line 797, in run_migrations
>>> 2017-01-25T23:41:55.666839945Z
>> self.get_context().run_migrations(**kw)
>>> 2017-01-25T23:41:55.666859336Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/migration.py",
>>> line 305, in run_migrations
>>> 2017-01-25T23:41:55.666970791Z for step in self._migrations_fn(heads,
>>> self):
>>> 2017-01-25T23:41:55.666992265Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
>>> line 163, in upgrade
>>> 2017-01-25T23:41:55.667021804Z return script._upgrade_revs(revision,
>>> rev)
>>> 2017-01-25T23:41:55.667032496Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
>>> line 329, in _upgrade_revs
>>> 2017-01-25T23:41:55.667144295Z revs = list(revs)
>>> 2017-01-25T23:41:55.667157407Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>> line 641, in _iterate_revisions
>>> 2017-01-25T23:41:55.667302027Z requested_lowers =
>>> self.get_revisions(lower)
>>> 2017-01-25T23:41:55.667315913Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>> line 298, in get_revisions
>>> 2017-01-25T23:41:55.667390590Z return
>> sum([self.get_revisions(id_elem)
>>> for id_elem in id_], ())
>>> 2017-01-25T23:41:55.667411867Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>> line 300, in get_revisions
>>> 2017-01-25T23:41:55.667419226Z resolved_id, branch_label =
>>> self._resolve_revision_number(id_)
>>> 2017-01-25T23:41:55.667434110Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>> line 433, in _resolve_revision_number
>>> 2017-01-25T23:41:55.667576273Z self._revision_map
>>> 2017-01-25T23:41:55.667590641Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/langhelpers.py",
>>> line 240, in __get__
>>> 2017-01-25T23:41:55.667648561Z obj.__dict__[self.__name__] = result =
>>> self.fget(obj)
>>> 2017-01-25T23:41:55.667658598Z File
>>>
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>> line 151, in _revision_map
>>> 2017-01-25T23:41:55.667680311Z down_revision = map_[downrev]
>>> 2017-01-25T23:41:55.667746364Z KeyError: '1a5a9e6bf2b5'
>>>
>>>
>>>
>>>
>>>
>>>
>>>> On Wed, Jan 25, 2017 at 11:15 PM Bolke de Bruin <bd...@gmail.com>
>> wrote:
>>>>
>>>> Hi All,
>>>>
>>>> I have made the THIRD beta of Airflow 1.8.0 available at:
>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public
>> keys
>>>> are available at
>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/> . It is
>>>> tagged with a local version “apache.incubating” so it allows upgrading
>> from
>>>> earlier releases. This beta is available for testing in a more
>> production
>>>> like setting (acceptance environment?).
>>>>
>>>> I would like to encourage everyone to try it out, to report back any
>>>> issues so we get to a rock solid release of 1.8.0. When reporting
>> issues a
>>>> test case or even a fix is highly appreciated.
>>>>
>>>> Issues cleared:
>>>>
>>>> * Manual trigger not working
>>>> * Performance issues with MySQL
>>>> * Postgres auto-commit support left over to the driver
>>>> * Poison pill taken while task has exited
>>>> * Keep cgroups optional
>>>> * Funcsigs pinned to 1.0.0
>>>>
>>>> Issue(s) remaining (blocker for RC):
>>>> * Cgroups not py3 compatible
>>>>
>>>> If all goes well we should have a Release Candidate on Feb 2. Thanks for
>>>> reporting issues and keep on testing please :). Moving towards RC I
>> tend to
>>>> like small bug fixes only. When we mark RC (do we need to vote on this?)
>>>> the procedure becomes even more strict. Please remember that the FINAL
>>>> release is dependent on a vote on the IPMC mailinglist.
>>>>
>>>> Cheers
>>>> Bolke
>>>
>>> --
>>> _/
>>> _/ Alex Van Boxel
>>
>> --
>> _/
>> _/ Alex Van Boxel
>>
> --
> _/
> _/ Alex Van Boxel
Re: Airflow 1.8.0 BETA 3
Posted by Alex Van Boxel <al...@vanboxel.be>.
The downgrade is ok for testing, but we can't release with this change
(can't expect people to install a beta first).
On Thu, Jan 26, 2017 at 8:36 AM Alex Van Boxel <al...@vanboxel.be> wrote:
> Another thing that I noticed (but observed it in beta 2 as well). Is the
> following:
>
> - The following trigger should not fire.
> --- Trigger rule is ONE_SUCCESS
> --- upstream: UP_FOR_RETRY + SKIPPED
> => task get's triggered
> => resulting SKIPPED
> => DAG marked success, with half of the Tasks never scheduled
>
> - UP_FOR_RETRY is propagated downstream (actually resulting in failure
> described above)
> --- *Does this make sense* ?!
>
> Both are a problem, and certainly the combination of both. I'll see if I
> can spend some time investigating this.
>
>
>
> On Thu, Jan 26, 2017 at 7:34 AM Bolke de Bruin <bd...@gmail.com> wrote:
>
> Mmm that is due to the reverting of one changes to the db. Need to look
> into that how to fix it.
>
> Sent from my iPhone
>
> > On 26 Jan 2017, at 00:51, Alex Van Boxel <al...@vanboxel.be> wrote:
> >
> > I do seem to have a problem upgrading the MySQL database with the last
> > commit:
> >
> >
> > 2017-01-25T23:41:55.662572654Z
> >
> /usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/messaging.py:69:
> > UserWarning: Revision 1a5a9e6bf2b5 referenced from 1a5a9e6bf2b5 ->
> > 127d2bf2dfa7 (head), Add dag_id/state index on dag_run table is not
> present
> > 2017-01-25T23:41:55.662613670Z warnings.warn(msg)
> > 2017-01-25T23:41:55.664560884Z Traceback (most recent call last):
> > 2017-01-25T23:41:55.664582238Z File "/usr/local/bin/airflow", line 4,
> in
> > <module>
> > 2017-01-25T23:41:55.664758457Z
> >
> __import__('pkg_resources').run_script('airflow==1.8.0b1+apache.incubating',
> > 'airflow')
> > 2017-01-25T23:41:55.664776553Z File
> > "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> > 739, in run_script
> > 2017-01-25T23:41:55.664937644Z
> > self.require(requires)[0].run_script(script_name, ns)
> > 2017-01-25T23:41:55.664952001Z File
> > "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> > 1494, in run_script
> > 2017-01-25T23:41:55.665072840Z exec(code, namespace, namespace)
> > 2017-01-25T23:41:55.665086884Z File
> >
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/EGG-INFO/scripts/airflow",
> > line 28, in <module>
> > 2017-01-25T23:41:55.665211755Z args.func(args)
> > 2017-01-25T23:41:55.665225157Z File
> >
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/bin/cli.py",
> > line 931, in upgradedb
> > 2017-01-25T23:41:55.665326691Z db_utils.upgradedb()
> > 2017-01-25T23:41:55.665346979Z File
> >
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/utils/db.py",
> > line 292, in upgradedb
> > 2017-01-25T23:41:55.665449297Z command.upgrade(config, 'heads')
> > 2017-01-25T23:41:55.665462333Z File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
> > line 174, in upgrade
> > 2017-01-25T23:41:55.665521466Z script.run_env()
> > 2017-01-25T23:41:55.665527232Z File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
> > line 416, in run_env2017-01-25T23:41:55.665638879Z
> > util.load_python_file(self.dir, 'env.py')
> > 2017-01-25T23:41:55.665656437Z File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/pyfiles.py",
> > line 93, in load_python_file
> > 2017-01-25T23:41:55.665682760Z module = load_module_py(module_id,
> path)
> > 2017-01-25T23:41:55.665699257Z File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/compat.py",
> > line 79, in load_module_py
> > 2017-01-25T23:41:55.665725036Z mod = imp.load_source(module_id, path,
> > fp)
> > 2017-01-25T23:41:55.665751433Z File
> >
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
> > line 88, in <module>
> > 2017-01-25T23:41:55.665778219Z run_migrations_online()
> > 2017-01-25T23:41:55.665787820Z File
> >
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
> > line 83, in run_migrations_online
> > 2017-01-25T23:41:55.665811353Z context.run_migrations()
> > 2017-01-25T23:41:55.665818933Z File "<string>", line 8, in
> run_migrations
> > 2017-01-25T23:41:55.666707547Z File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/environment.py",
> > line 797, in run_migrations
> > 2017-01-25T23:41:55.666839945Z
> self.get_context().run_migrations(**kw)
> > 2017-01-25T23:41:55.666859336Z File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/migration.py",
> > line 305, in run_migrations
> > 2017-01-25T23:41:55.666970791Z for step in self._migrations_fn(heads,
> > self):
> > 2017-01-25T23:41:55.666992265Z File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
> > line 163, in upgrade
> > 2017-01-25T23:41:55.667021804Z return script._upgrade_revs(revision,
> > rev)
> > 2017-01-25T23:41:55.667032496Z File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
> > line 329, in _upgrade_revs
> > 2017-01-25T23:41:55.667144295Z revs = list(revs)
> > 2017-01-25T23:41:55.667157407Z File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> > line 641, in _iterate_revisions
> > 2017-01-25T23:41:55.667302027Z requested_lowers =
> > self.get_revisions(lower)
> > 2017-01-25T23:41:55.667315913Z File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> > line 298, in get_revisions
> > 2017-01-25T23:41:55.667390590Z return
> sum([self.get_revisions(id_elem)
> > for id_elem in id_], ())
> > 2017-01-25T23:41:55.667411867Z File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> > line 300, in get_revisions
> > 2017-01-25T23:41:55.667419226Z resolved_id, branch_label =
> > self._resolve_revision_number(id_)
> > 2017-01-25T23:41:55.667434110Z File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> > line 433, in _resolve_revision_number
> > 2017-01-25T23:41:55.667576273Z self._revision_map
> > 2017-01-25T23:41:55.667590641Z File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/langhelpers.py",
> > line 240, in __get__
> > 2017-01-25T23:41:55.667648561Z obj.__dict__[self.__name__] = result =
> > self.fget(obj)
> > 2017-01-25T23:41:55.667658598Z File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> > line 151, in _revision_map
> > 2017-01-25T23:41:55.667680311Z down_revision = map_[downrev]
> > 2017-01-25T23:41:55.667746364Z KeyError: '1a5a9e6bf2b5'
> >
> >
> >
> >
> >
> >
> >> On Wed, Jan 25, 2017 at 11:15 PM Bolke de Bruin <bd...@gmail.com>
> wrote:
> >>
> >> Hi All,
> >>
> >> I have made the THIRD beta of Airflow 1.8.0 available at:
> >> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> >> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public
> keys
> >> are available at
> >> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
> >> https://dist.apache.org/repos/dist/release/incubator/airflow/> . It is
> >> tagged with a local version “apache.incubating” so it allows upgrading
> from
> >> earlier releases. This beta is available for testing in a more
> production
> >> like setting (acceptance environment?).
> >>
> >> I would like to encourage everyone to try it out, to report back any
> >> issues so we get to a rock solid release of 1.8.0. When reporting
> issues a
> >> test case or even a fix is highly appreciated.
> >>
> >> Issues cleared:
> >>
> >> * Manual trigger not working
> >> * Performance issues with MySQL
> >> * Postgres auto-commit support left over to the driver
> >> * Poison pill taken while task has exited
> >> * Keep cgroups optional
> >> * Funcsigs pinned to 1.0.0
> >>
> >> Issue(s) remaining (blocker for RC):
> >> * Cgroups not py3 compatible
> >>
> >> If all goes well we should have a Release Candidate on Feb 2. Thanks for
> >> reporting issues and keep on testing please :). Moving towards RC I
> tend to
> >> like small bug fixes only. When we mark RC (do we need to vote on this?)
> >> the procedure becomes even more strict. Please remember that the FINAL
> >> release is dependent on a vote on the IPMC mailinglist.
> >>
> >> Cheers
> >> Bolke
> >
> > --
> > _/
> > _/ Alex Van Boxel
>
> --
> _/
> _/ Alex Van Boxel
>
--
_/
_/ Alex Van Boxel
Re: Airflow 1.8.0 BETA 3
Posted by Alex Van Boxel <al...@vanboxel.be>.
Another thing that I noticed (but observed it in beta 2 as well). Is the
following:
- The following trigger should not fire.
--- Trigger rule is ONE_SUCCESS
--- upstream: UP_FOR_RETRY + SKIPPED
=> task get's triggered
=> resulting SKIPPED
=> DAG marked success, with half of the Tasks never scheduled
- UP_FOR_RETRY is propagated downstream (actually resulting in failure
described above)
--- *Does this make sense* ?!
Both are a problem, and certainly the combination of both. I'll see if I
can spend some time investigating this.
On Thu, Jan 26, 2017 at 7:34 AM Bolke de Bruin <bd...@gmail.com> wrote:
> Mmm that is due to the reverting of one changes to the db. Need to look
> into that how to fix it.
>
> Sent from my iPhone
>
> > On 26 Jan 2017, at 00:51, Alex Van Boxel <al...@vanboxel.be> wrote:
> >
> > I do seem to have a problem upgrading the MySQL database with the last
> > commit:
> >
> >
> > 2017-01-25T23:41:55.662572654Z
> >
> /usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/messaging.py:69:
> > UserWarning: Revision 1a5a9e6bf2b5 referenced from 1a5a9e6bf2b5 ->
> > 127d2bf2dfa7 (head), Add dag_id/state index on dag_run table is not
> present
> > 2017-01-25T23:41:55.662613670Z warnings.warn(msg)
> > 2017-01-25T23:41:55.664560884Z Traceback (most recent call last):
> > 2017-01-25T23:41:55.664582238Z File "/usr/local/bin/airflow", line 4,
> in
> > <module>
> > 2017-01-25T23:41:55.664758457Z
> >
> __import__('pkg_resources').run_script('airflow==1.8.0b1+apache.incubating',
> > 'airflow')
> > 2017-01-25T23:41:55.664776553Z File
> > "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> > 739, in run_script
> > 2017-01-25T23:41:55.664937644Z
> > self.require(requires)[0].run_script(script_name, ns)
> > 2017-01-25T23:41:55.664952001Z File
> > "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> > 1494, in run_script
> > 2017-01-25T23:41:55.665072840Z exec(code, namespace, namespace)
> > 2017-01-25T23:41:55.665086884Z File
> >
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/EGG-INFO/scripts/airflow",
> > line 28, in <module>
> > 2017-01-25T23:41:55.665211755Z args.func(args)
> > 2017-01-25T23:41:55.665225157Z File
> >
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/bin/cli.py",
> > line 931, in upgradedb
> > 2017-01-25T23:41:55.665326691Z db_utils.upgradedb()
> > 2017-01-25T23:41:55.665346979Z File
> >
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/utils/db.py",
> > line 292, in upgradedb
> > 2017-01-25T23:41:55.665449297Z command.upgrade(config, 'heads')
> > 2017-01-25T23:41:55.665462333Z File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
> > line 174, in upgrade
> > 2017-01-25T23:41:55.665521466Z script.run_env()
> > 2017-01-25T23:41:55.665527232Z File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
> > line 416, in run_env2017-01-25T23:41:55.665638879Z
> > util.load_python_file(self.dir, 'env.py')
> > 2017-01-25T23:41:55.665656437Z File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/pyfiles.py",
> > line 93, in load_python_file
> > 2017-01-25T23:41:55.665682760Z module = load_module_py(module_id,
> path)
> > 2017-01-25T23:41:55.665699257Z File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/compat.py",
> > line 79, in load_module_py
> > 2017-01-25T23:41:55.665725036Z mod = imp.load_source(module_id, path,
> > fp)
> > 2017-01-25T23:41:55.665751433Z File
> >
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
> > line 88, in <module>
> > 2017-01-25T23:41:55.665778219Z run_migrations_online()
> > 2017-01-25T23:41:55.665787820Z File
> >
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
> > line 83, in run_migrations_online
> > 2017-01-25T23:41:55.665811353Z context.run_migrations()
> > 2017-01-25T23:41:55.665818933Z File "<string>", line 8, in
> run_migrations
> > 2017-01-25T23:41:55.666707547Z File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/environment.py",
> > line 797, in run_migrations
> > 2017-01-25T23:41:55.666839945Z
> self.get_context().run_migrations(**kw)
> > 2017-01-25T23:41:55.666859336Z File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/migration.py",
> > line 305, in run_migrations
> > 2017-01-25T23:41:55.666970791Z for step in self._migrations_fn(heads,
> > self):
> > 2017-01-25T23:41:55.666992265Z File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
> > line 163, in upgrade
> > 2017-01-25T23:41:55.667021804Z return script._upgrade_revs(revision,
> > rev)
> > 2017-01-25T23:41:55.667032496Z File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
> > line 329, in _upgrade_revs
> > 2017-01-25T23:41:55.667144295Z revs = list(revs)
> > 2017-01-25T23:41:55.667157407Z File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> > line 641, in _iterate_revisions
> > 2017-01-25T23:41:55.667302027Z requested_lowers =
> > self.get_revisions(lower)
> > 2017-01-25T23:41:55.667315913Z File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> > line 298, in get_revisions
> > 2017-01-25T23:41:55.667390590Z return
> sum([self.get_revisions(id_elem)
> > for id_elem in id_], ())
> > 2017-01-25T23:41:55.667411867Z File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> > line 300, in get_revisions
> > 2017-01-25T23:41:55.667419226Z resolved_id, branch_label =
> > self._resolve_revision_number(id_)
> > 2017-01-25T23:41:55.667434110Z File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> > line 433, in _resolve_revision_number
> > 2017-01-25T23:41:55.667576273Z self._revision_map
> > 2017-01-25T23:41:55.667590641Z File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/langhelpers.py",
> > line 240, in __get__
> > 2017-01-25T23:41:55.667648561Z obj.__dict__[self.__name__] = result =
> > self.fget(obj)
> > 2017-01-25T23:41:55.667658598Z File
> >
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> > line 151, in _revision_map
> > 2017-01-25T23:41:55.667680311Z down_revision = map_[downrev]
> > 2017-01-25T23:41:55.667746364Z KeyError: '1a5a9e6bf2b5'
> >
> >
> >
> >
> >
> >
> >> On Wed, Jan 25, 2017 at 11:15 PM Bolke de Bruin <bd...@gmail.com>
> wrote:
> >>
> >> Hi All,
> >>
> >> I have made the THIRD beta of Airflow 1.8.0 available at:
> >> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> >> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public
> keys
> >> are available at
> >> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
> >> https://dist.apache.org/repos/dist/release/incubator/airflow/> . It is
> >> tagged with a local version “apache.incubating” so it allows upgrading
> from
> >> earlier releases. This beta is available for testing in a more
> production
> >> like setting (acceptance environment?).
> >>
> >> I would like to encourage everyone to try it out, to report back any
> >> issues so we get to a rock solid release of 1.8.0. When reporting
> issues a
> >> test case or even a fix is highly appreciated.
> >>
> >> Issues cleared:
> >>
> >> * Manual trigger not working
> >> * Performance issues with MySQL
> >> * Postgres auto-commit support left over to the driver
> >> * Poison pill taken while task has exited
> >> * Keep cgroups optional
> >> * Funcsigs pinned to 1.0.0
> >>
> >> Issue(s) remaining (blocker for RC):
> >> * Cgroups not py3 compatible
> >>
> >> If all goes well we should have a Release Candidate on Feb 2. Thanks for
> >> reporting issues and keep on testing please :). Moving towards RC I
> tend to
> >> like small bug fixes only. When we mark RC (do we need to vote on this?)
> >> the procedure becomes even more strict. Please remember that the FINAL
> >> release is dependent on a vote on the IPMC mailinglist.
> >>
> >> Cheers
> >> Bolke
> >
> > --
> > _/
> > _/ Alex Van Boxel
>
--
_/
_/ Alex Van Boxel
Re: Airflow 1.8.0 BETA 3
Posted by Bolke de Bruin <bd...@gmail.com>.
Mmm that is due to the reverting of one changes to the db. Need to look into that how to fix it.
Sent from my iPhone
> On 26 Jan 2017, at 00:51, Alex Van Boxel <al...@vanboxel.be> wrote:
>
> I do seem to have a problem upgrading the MySQL database with the last
> commit:
>
>
> 2017-01-25T23:41:55.662572654Z
> /usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/messaging.py:69:
> UserWarning: Revision 1a5a9e6bf2b5 referenced from 1a5a9e6bf2b5 ->
> 127d2bf2dfa7 (head), Add dag_id/state index on dag_run table is not present
> 2017-01-25T23:41:55.662613670Z warnings.warn(msg)
> 2017-01-25T23:41:55.664560884Z Traceback (most recent call last):
> 2017-01-25T23:41:55.664582238Z File "/usr/local/bin/airflow", line 4, in
> <module>
> 2017-01-25T23:41:55.664758457Z
> __import__('pkg_resources').run_script('airflow==1.8.0b1+apache.incubating',
> 'airflow')
> 2017-01-25T23:41:55.664776553Z File
> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 739, in run_script
> 2017-01-25T23:41:55.664937644Z
> self.require(requires)[0].run_script(script_name, ns)
> 2017-01-25T23:41:55.664952001Z File
> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1494, in run_script
> 2017-01-25T23:41:55.665072840Z exec(code, namespace, namespace)
> 2017-01-25T23:41:55.665086884Z File
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/EGG-INFO/scripts/airflow",
> line 28, in <module>
> 2017-01-25T23:41:55.665211755Z args.func(args)
> 2017-01-25T23:41:55.665225157Z File
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/bin/cli.py",
> line 931, in upgradedb
> 2017-01-25T23:41:55.665326691Z db_utils.upgradedb()
> 2017-01-25T23:41:55.665346979Z File
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/utils/db.py",
> line 292, in upgradedb
> 2017-01-25T23:41:55.665449297Z command.upgrade(config, 'heads')
> 2017-01-25T23:41:55.665462333Z File
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
> line 174, in upgrade
> 2017-01-25T23:41:55.665521466Z script.run_env()
> 2017-01-25T23:41:55.665527232Z File
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
> line 416, in run_env2017-01-25T23:41:55.665638879Z
> util.load_python_file(self.dir, 'env.py')
> 2017-01-25T23:41:55.665656437Z File
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/pyfiles.py",
> line 93, in load_python_file
> 2017-01-25T23:41:55.665682760Z module = load_module_py(module_id, path)
> 2017-01-25T23:41:55.665699257Z File
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/compat.py",
> line 79, in load_module_py
> 2017-01-25T23:41:55.665725036Z mod = imp.load_source(module_id, path,
> fp)
> 2017-01-25T23:41:55.665751433Z File
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
> line 88, in <module>
> 2017-01-25T23:41:55.665778219Z run_migrations_online()
> 2017-01-25T23:41:55.665787820Z File
> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
> line 83, in run_migrations_online
> 2017-01-25T23:41:55.665811353Z context.run_migrations()
> 2017-01-25T23:41:55.665818933Z File "<string>", line 8, in run_migrations
> 2017-01-25T23:41:55.666707547Z File
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/environment.py",
> line 797, in run_migrations
> 2017-01-25T23:41:55.666839945Z self.get_context().run_migrations(**kw)
> 2017-01-25T23:41:55.666859336Z File
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/migration.py",
> line 305, in run_migrations
> 2017-01-25T23:41:55.666970791Z for step in self._migrations_fn(heads,
> self):
> 2017-01-25T23:41:55.666992265Z File
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
> line 163, in upgrade
> 2017-01-25T23:41:55.667021804Z return script._upgrade_revs(revision,
> rev)
> 2017-01-25T23:41:55.667032496Z File
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
> line 329, in _upgrade_revs
> 2017-01-25T23:41:55.667144295Z revs = list(revs)
> 2017-01-25T23:41:55.667157407Z File
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> line 641, in _iterate_revisions
> 2017-01-25T23:41:55.667302027Z requested_lowers =
> self.get_revisions(lower)
> 2017-01-25T23:41:55.667315913Z File
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> line 298, in get_revisions
> 2017-01-25T23:41:55.667390590Z return sum([self.get_revisions(id_elem)
> for id_elem in id_], ())
> 2017-01-25T23:41:55.667411867Z File
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> line 300, in get_revisions
> 2017-01-25T23:41:55.667419226Z resolved_id, branch_label =
> self._resolve_revision_number(id_)
> 2017-01-25T23:41:55.667434110Z File
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> line 433, in _resolve_revision_number
> 2017-01-25T23:41:55.667576273Z self._revision_map
> 2017-01-25T23:41:55.667590641Z File
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/langhelpers.py",
> line 240, in __get__
> 2017-01-25T23:41:55.667648561Z obj.__dict__[self.__name__] = result =
> self.fget(obj)
> 2017-01-25T23:41:55.667658598Z File
> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
> line 151, in _revision_map
> 2017-01-25T23:41:55.667680311Z down_revision = map_[downrev]
> 2017-01-25T23:41:55.667746364Z KeyError: '1a5a9e6bf2b5'
>
>
>
>
>
>
>> On Wed, Jan 25, 2017 at 11:15 PM Bolke de Bruin <bd...@gmail.com> wrote:
>>
>> Hi All,
>>
>> I have made the THIRD beta of Airflow 1.8.0 available at:
>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public keys
>> are available at
>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>> https://dist.apache.org/repos/dist/release/incubator/airflow/> . It is
>> tagged with a local version “apache.incubating” so it allows upgrading from
>> earlier releases. This beta is available for testing in a more production
>> like setting (acceptance environment?).
>>
>> I would like to encourage everyone to try it out, to report back any
>> issues so we get to a rock solid release of 1.8.0. When reporting issues a
>> test case or even a fix is highly appreciated.
>>
>> Issues cleared:
>>
>> * Manual trigger not working
>> * Performance issues with MySQL
>> * Postgres auto-commit support left over to the driver
>> * Poison pill taken while task has exited
>> * Keep cgroups optional
>> * Funcsigs pinned to 1.0.0
>>
>> Issue(s) remaining (blocker for RC):
>> * Cgroups not py3 compatible
>>
>> If all goes well we should have a Release Candidate on Feb 2. Thanks for
>> reporting issues and keep on testing please :). Moving towards RC I tend to
>> like small bug fixes only. When we mark RC (do we need to vote on this?)
>> the procedure becomes even more strict. Please remember that the FINAL
>> release is dependent on a vote on the IPMC mailinglist.
>>
>> Cheers
>> Bolke
>
> --
> _/
> _/ Alex Van Boxel
Re: Airflow 1.8.0 BETA 3
Posted by Alex Van Boxel <al...@vanboxel.be>.
I do seem to have a problem upgrading the MySQL database with the last
commit:
2017-01-25T23:41:55.662572654Z
/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/messaging.py:69:
UserWarning: Revision 1a5a9e6bf2b5 referenced from 1a5a9e6bf2b5 ->
127d2bf2dfa7 (head), Add dag_id/state index on dag_run table is not present
2017-01-25T23:41:55.662613670Z warnings.warn(msg)
2017-01-25T23:41:55.664560884Z Traceback (most recent call last):
2017-01-25T23:41:55.664582238Z File "/usr/local/bin/airflow", line 4, in
<module>
2017-01-25T23:41:55.664758457Z
__import__('pkg_resources').run_script('airflow==1.8.0b1+apache.incubating',
'airflow')
2017-01-25T23:41:55.664776553Z File
"/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line
739, in run_script
2017-01-25T23:41:55.664937644Z
self.require(requires)[0].run_script(script_name, ns)
2017-01-25T23:41:55.664952001Z File
"/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line
1494, in run_script
2017-01-25T23:41:55.665072840Z exec(code, namespace, namespace)
2017-01-25T23:41:55.665086884Z File
"/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/EGG-INFO/scripts/airflow",
line 28, in <module>
2017-01-25T23:41:55.665211755Z args.func(args)
2017-01-25T23:41:55.665225157Z File
"/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/bin/cli.py",
line 931, in upgradedb
2017-01-25T23:41:55.665326691Z db_utils.upgradedb()
2017-01-25T23:41:55.665346979Z File
"/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/utils/db.py",
line 292, in upgradedb
2017-01-25T23:41:55.665449297Z command.upgrade(config, 'heads')
2017-01-25T23:41:55.665462333Z File
"/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
line 174, in upgrade
2017-01-25T23:41:55.665521466Z script.run_env()
2017-01-25T23:41:55.665527232Z File
"/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
line 416, in run_env2017-01-25T23:41:55.665638879Z
util.load_python_file(self.dir, 'env.py')
2017-01-25T23:41:55.665656437Z File
"/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/pyfiles.py",
line 93, in load_python_file
2017-01-25T23:41:55.665682760Z module = load_module_py(module_id, path)
2017-01-25T23:41:55.665699257Z File
"/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/compat.py",
line 79, in load_module_py
2017-01-25T23:41:55.665725036Z mod = imp.load_source(module_id, path,
fp)
2017-01-25T23:41:55.665751433Z File
"/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
line 88, in <module>
2017-01-25T23:41:55.665778219Z run_migrations_online()
2017-01-25T23:41:55.665787820Z File
"/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
line 83, in run_migrations_online
2017-01-25T23:41:55.665811353Z context.run_migrations()
2017-01-25T23:41:55.665818933Z File "<string>", line 8, in run_migrations
2017-01-25T23:41:55.666707547Z File
"/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/environment.py",
line 797, in run_migrations
2017-01-25T23:41:55.666839945Z self.get_context().run_migrations(**kw)
2017-01-25T23:41:55.666859336Z File
"/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/migration.py",
line 305, in run_migrations
2017-01-25T23:41:55.666970791Z for step in self._migrations_fn(heads,
self):
2017-01-25T23:41:55.666992265Z File
"/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
line 163, in upgrade
2017-01-25T23:41:55.667021804Z return script._upgrade_revs(revision,
rev)
2017-01-25T23:41:55.667032496Z File
"/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
line 329, in _upgrade_revs
2017-01-25T23:41:55.667144295Z revs = list(revs)
2017-01-25T23:41:55.667157407Z File
"/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
line 641, in _iterate_revisions
2017-01-25T23:41:55.667302027Z requested_lowers =
self.get_revisions(lower)
2017-01-25T23:41:55.667315913Z File
"/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
line 298, in get_revisions
2017-01-25T23:41:55.667390590Z return sum([self.get_revisions(id_elem)
for id_elem in id_], ())
2017-01-25T23:41:55.667411867Z File
"/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
line 300, in get_revisions
2017-01-25T23:41:55.667419226Z resolved_id, branch_label =
self._resolve_revision_number(id_)
2017-01-25T23:41:55.667434110Z File
"/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
line 433, in _resolve_revision_number
2017-01-25T23:41:55.667576273Z self._revision_map
2017-01-25T23:41:55.667590641Z File
"/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/langhelpers.py",
line 240, in __get__
2017-01-25T23:41:55.667648561Z obj.__dict__[self.__name__] = result =
self.fget(obj)
2017-01-25T23:41:55.667658598Z File
"/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
line 151, in _revision_map
2017-01-25T23:41:55.667680311Z down_revision = map_[downrev]
2017-01-25T23:41:55.667746364Z KeyError: '1a5a9e6bf2b5'
On Wed, Jan 25, 2017 at 11:15 PM Bolke de Bruin <bd...@gmail.com> wrote:
> Hi All,
>
> I have made the THIRD beta of Airflow 1.8.0 available at:
> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public keys
> are available at
> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
> https://dist.apache.org/repos/dist/release/incubator/airflow/> . It is
> tagged with a local version “apache.incubating” so it allows upgrading from
> earlier releases. This beta is available for testing in a more production
> like setting (acceptance environment?).
>
> I would like to encourage everyone to try it out, to report back any
> issues so we get to a rock solid release of 1.8.0. When reporting issues a
> test case or even a fix is highly appreciated.
>
> Issues cleared:
>
> * Manual trigger not working
> * Performance issues with MySQL
> * Postgres auto-commit support left over to the driver
> * Poison pill taken while task has exited
> * Keep cgroups optional
> * Funcsigs pinned to 1.0.0
>
> Issue(s) remaining (blocker for RC):
> * Cgroups not py3 compatible
>
> If all goes well we should have a Release Candidate on Feb 2. Thanks for
> reporting issues and keep on testing please :). Moving towards RC I tend to
> like small bug fixes only. When we mark RC (do we need to vote on this?)
> the procedure becomes even more strict. Please remember that the FINAL
> release is dependent on a vote on the IPMC mailinglist.
>
> Cheers
> Bolke
--
_/
_/ Alex Van Boxel