You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Bolke de Bruin <bd...@gmail.com> on 2018/07/08 20:02:18 UTC
[VOTE] Airflow 1.10.0rc1
Hey all,
I have cut Airflow 1.10.0 RC1. This email is calling a vote on the release,
which will last for 72 hours. Consider this my (binding) +1.
Airflow 1.10.0 RC 1 is available at:
https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release that
comes with INSTALL instructions.
apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python "sdist"
release.
Public keys are available at:
https://dist.apache.org/repos/dist/release/incubator/airflow/ <https://dist.apache.org/repos/dist/release/incubator/airflow/>
The amount of JIRAs fixed is over 700. Please have a look at the changelog.
Please note that the version number excludes the `rcX` string as well
as the "+incubating" string, so it's now simply 1.10.0. This will allow us
to rename the artifact without modifying the artifact checksums when we
actually release.
Cheers,
Bolke
Re: [VOTE] Airflow 1.10.0rc1
Posted by Naik Kaxil <k....@reply.com>.
+1 (binding)
Thanks @bolke for correcting me.
On 10/07/2018, 06:48, "Bolke de Bruin" <bd...@gmail.com> wrote:
Kaxil,
Please vote with binding / non-binding. This is important for the IPMC vote that won’t know all committers.
B.
Verstuurd vanaf mijn iPad
> Op 10 jul. 2018 om 03:22 heeft Naik Kaxil <k....@reply.com> het volgende geschreven:
>
> +1
>
> Regards,
> Kaxil
>
> On 10/07/2018, 02:09, "Kengo Seki" <se...@apache.org> wrote:
>
> +1 (non-binding)
>
> - Checked signatures and checksums
> - Confirmed the source distribution corresponds to v1-10-test's HEAD
> (f1083cb).
> - Installed the binary distribution with cryptography package ran some
> example dags with no error
> - Checked version info via WebUI and confirmed RBAC worked
>
> Kengo Seki <sekikn@ <se...@gmail.com>apache.org>
>
> 2018-07-09 5:02 GMT+09:00 Bolke de Bruin <bd...@gmail.com>:
>
>> Hey all,
>>
>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the release,
>> which will last for 72 hours. Consider this my (binding) +1.
>>
>> Airflow 1.10.0 RC 1 is available at:
>>
>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <
>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
>>
>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release that
>> comes with INSTALL instructions.
>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python
>> "sdist"
>> release.
>>
>> Public keys are available at:
>>
>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>> https://dist.apache.org/repos/dist/release/incubator/airflow/>
>>
>> The amount of JIRAs fixed is over 700. Please have a look at the changelog.
>>
>> Please note that the version number excludes the `rcX` string as well
>> as the "+incubating" string, so it's now simply 1.10.0. This will allow us
>> to rename the artifact without modifying the artifact checksums when we
>> actually release.
>>
>>
>> Cheers,
>> Bolke
>
>
>
>
>
>
> Kaxil Naik
>
> Data Reply
> 2nd Floor, Nova South
> 160 Victoria Street, Westminster
> London SW1E 5LB - UK
> phone: +44 (0)20 7730 6000
> k.naik@reply.com
> www.reply.com
Kaxil Naik
Data Reply
2nd Floor, Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK
phone: +44 (0)20 7730 6000
k.naik@reply.com
www.reply.com
Re: [VOTE] Airflow 1.10.0rc1
Posted by Bolke de Bruin <bd...@gmail.com>.
Kaxil,
Please vote with binding / non-binding. This is important for the IPMC vote that won’t know all committers.
B.
Verstuurd vanaf mijn iPad
> Op 10 jul. 2018 om 03:22 heeft Naik Kaxil <k....@reply.com> het volgende geschreven:
>
> +1
>
> Regards,
> Kaxil
>
> On 10/07/2018, 02:09, "Kengo Seki" <se...@apache.org> wrote:
>
> +1 (non-binding)
>
> - Checked signatures and checksums
> - Confirmed the source distribution corresponds to v1-10-test's HEAD
> (f1083cb).
> - Installed the binary distribution with cryptography package ran some
> example dags with no error
> - Checked version info via WebUI and confirmed RBAC worked
>
> Kengo Seki <sekikn@ <se...@gmail.com>apache.org>
>
> 2018-07-09 5:02 GMT+09:00 Bolke de Bruin <bd...@gmail.com>:
>
>> Hey all,
>>
>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the release,
>> which will last for 72 hours. Consider this my (binding) +1.
>>
>> Airflow 1.10.0 RC 1 is available at:
>>
>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <
>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
>>
>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release that
>> comes with INSTALL instructions.
>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python
>> "sdist"
>> release.
>>
>> Public keys are available at:
>>
>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>> https://dist.apache.org/repos/dist/release/incubator/airflow/>
>>
>> The amount of JIRAs fixed is over 700. Please have a look at the changelog.
>>
>> Please note that the version number excludes the `rcX` string as well
>> as the "+incubating" string, so it's now simply 1.10.0. This will allow us
>> to rename the artifact without modifying the artifact checksums when we
>> actually release.
>>
>>
>> Cheers,
>> Bolke
>
>
>
>
>
>
> Kaxil Naik
>
> Data Reply
> 2nd Floor, Nova South
> 160 Victoria Street, Westminster
> London SW1E 5LB - UK
> phone: +44 (0)20 7730 6000
> k.naik@reply.com
> www.reply.com
Re: [VOTE] Airflow 1.10.0rc1
Posted by Naik Kaxil <k....@reply.com>.
+1
Regards,
Kaxil
On 10/07/2018, 02:09, "Kengo Seki" <se...@apache.org> wrote:
+1 (non-binding)
- Checked signatures and checksums
- Confirmed the source distribution corresponds to v1-10-test's HEAD
(f1083cb).
- Installed the binary distribution with cryptography package ran some
example dags with no error
- Checked version info via WebUI and confirmed RBAC worked
Kengo Seki <sekikn@ <se...@gmail.com>apache.org>
2018-07-09 5:02 GMT+09:00 Bolke de Bruin <bd...@gmail.com>:
> Hey all,
>
> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the release,
> which will last for 72 hours. Consider this my (binding) +1.
>
> Airflow 1.10.0 RC 1 is available at:
>
> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <
> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
>
> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release that
> comes with INSTALL instructions.
> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python
> "sdist"
> release.
>
> Public keys are available at:
>
> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
> https://dist.apache.org/repos/dist/release/incubator/airflow/>
>
> The amount of JIRAs fixed is over 700. Please have a look at the changelog.
>
> Please note that the version number excludes the `rcX` string as well
> as the "+incubating" string, so it's now simply 1.10.0. This will allow us
> to rename the artifact without modifying the artifact checksums when we
> actually release.
>
>
> Cheers,
> Bolke
Kaxil Naik
Data Reply
2nd Floor, Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK
phone: +44 (0)20 7730 6000
k.naik@reply.com
www.reply.com
Re: [VOTE] Airflow 1.10.0rc1
Posted by Kengo Seki <se...@apache.org>.
+1 (non-binding)
- Checked signatures and checksums
- Confirmed the source distribution corresponds to v1-10-test's HEAD
(f1083cb).
- Installed the binary distribution with cryptography package ran some
example dags with no error
- Checked version info via WebUI and confirmed RBAC worked
Kengo Seki <sekikn@ <se...@gmail.com>apache.org>
2018-07-09 5:02 GMT+09:00 Bolke de Bruin <bd...@gmail.com>:
> Hey all,
>
> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the release,
> which will last for 72 hours. Consider this my (binding) +1.
>
> Airflow 1.10.0 RC 1 is available at:
>
> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <
> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
>
> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release that
> comes with INSTALL instructions.
> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python
> "sdist"
> release.
>
> Public keys are available at:
>
> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
> https://dist.apache.org/repos/dist/release/incubator/airflow/>
>
> The amount of JIRAs fixed is over 700. Please have a look at the changelog.
>
> Please note that the version number excludes the `rcX` string as well
> as the "+incubating" string, so it's now simply 1.10.0. This will allow us
> to rename the artifact without modifying the artifact checksums when we
> actually release.
>
>
> Cheers,
> Bolke
Re: [VOTE] Airflow 1.10.0rc1
Posted by Ash Berlin-Taylor <as...@firemirror.com>.
I'm afraid I won't have any time to test it out before Friday at the earliest. There have been some people using it successfully in the Gitter chat room, so +0.5 (binding) from me.
-ash
> On 11 Jul 2018, at 22:24, Bolke de Bruin <bd...@gmail.com> wrote:
>
> Committers can you please cast your vote? We are 2 +1s now...
>
> Thanks
> Bolke
>
> Verstuurd vanaf mijn iPad
>
>> Op 10 jul. 2018 om 11:32 heeft Stefan Seelmann <ma...@stefan-seelmann.de> het volgende geschreven:
>>
>> +1 (non-binding)
>>
>> * Verified checksums and signatures of the packages
>> * Checked license and notice files
>> * Run the tests
>> * Installed from git tag and run some example DAGs
>>
>> Two minor findings:
>> * In airflow/api/auth/backend/kerberos_auth.py the original license
>> header was replaced with the ASF one, it should be restored, see PR [1]
>> * MD5 files should no longer be distributed, see [2]
>>
>> Kind Regards,
>> Stefan
>>
>> [1] https://github.com/apache/incubator-airflow/pull/3591
>> [2] https://www.apache.org/dev/release-distribution#sigs-and-sums
>>
>>
>>> On 07/08/2018 10:02 PM, Bolke de Bruin wrote:
>>> Hey all,
>>>
>>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the release,
>>> which will last for 72 hours. Consider this my (binding) +1.
>>>
>>> Airflow 1.10.0 RC 1 is available at:
>>>
>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
>>>
>>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release that
>>> comes with INSTALL instructions.
>>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python "sdist"
>>> release.
>>>
>>> Public keys are available at:
>>>
>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <https://dist.apache.org/repos/dist/release/incubator/airflow/>
>>>
>>> The amount of JIRAs fixed is over 700. Please have a look at the changelog.
>>>
>>> Please note that the version number excludes the `rcX` string as well
>>> as the "+incubating" string, so it's now simply 1.10.0. This will allow us
>>> to rename the artifact without modifying the artifact checksums when we
>>> actually release.
>>>
>>>
>>> Cheers,
>>> Bolke
>>>
>>
Re: [VOTE] Airflow 1.10.0rc1
Posted by Bolke de Bruin <bd...@gmail.com>.
Committers can you please cast your vote? We are 2 +1s now...
Thanks
Bolke
Verstuurd vanaf mijn iPad
> Op 10 jul. 2018 om 11:32 heeft Stefan Seelmann <ma...@stefan-seelmann.de> het volgende geschreven:
>
> +1 (non-binding)
>
> * Verified checksums and signatures of the packages
> * Checked license and notice files
> * Run the tests
> * Installed from git tag and run some example DAGs
>
> Two minor findings:
> * In airflow/api/auth/backend/kerberos_auth.py the original license
> header was replaced with the ASF one, it should be restored, see PR [1]
> * MD5 files should no longer be distributed, see [2]
>
> Kind Regards,
> Stefan
>
> [1] https://github.com/apache/incubator-airflow/pull/3591
> [2] https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>> On 07/08/2018 10:02 PM, Bolke de Bruin wrote:
>> Hey all,
>>
>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the release,
>> which will last for 72 hours. Consider this my (binding) +1.
>>
>> Airflow 1.10.0 RC 1 is available at:
>>
>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
>>
>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release that
>> comes with INSTALL instructions.
>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python "sdist"
>> release.
>>
>> Public keys are available at:
>>
>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <https://dist.apache.org/repos/dist/release/incubator/airflow/>
>>
>> The amount of JIRAs fixed is over 700. Please have a look at the changelog.
>>
>> Please note that the version number excludes the `rcX` string as well
>> as the "+incubating" string, so it's now simply 1.10.0. This will allow us
>> to rename the artifact without modifying the artifact checksums when we
>> actually release.
>>
>>
>> Cheers,
>> Bolke
>>
>
Re: [VOTE] Airflow 1.10.0rc1
Posted by Stefan Seelmann <ma...@stefan-seelmann.de>.
+1 (non-binding)
* Verified checksums and signatures of the packages
* Checked license and notice files
* Run the tests
* Installed from git tag and run some example DAGs
Two minor findings:
* In airflow/api/auth/backend/kerberos_auth.py the original license
header was replaced with the ASF one, it should be restored, see PR [1]
* MD5 files should no longer be distributed, see [2]
Kind Regards,
Stefan
[1] https://github.com/apache/incubator-airflow/pull/3591
[2] https://www.apache.org/dev/release-distribution#sigs-and-sums
On 07/08/2018 10:02 PM, Bolke de Bruin wrote:
> Hey all,
>
> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the release,
> which will last for 72 hours. Consider this my (binding) +1.
>
> Airflow 1.10.0 RC 1 is available at:
>
> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
>
> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release that
> comes with INSTALL instructions.
> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python "sdist"
> release.
>
> Public keys are available at:
>
> https://dist.apache.org/repos/dist/release/incubator/airflow/ <https://dist.apache.org/repos/dist/release/incubator/airflow/>
>
> The amount of JIRAs fixed is over 700. Please have a look at the changelog.
>
> Please note that the version number excludes the `rcX` string as well
> as the "+incubating" string, so it's now simply 1.10.0. This will allow us
> to rename the artifact without modifying the artifact checksums when we
> actually release.
>
>
> Cheers,
> Bolke
>
Re: [VOTE] Airflow 1.10.0rc1
Posted by Carl Johan Gustavsson <ca...@gmail.com>.
I made a PR for fixing the issue
https://github.com/apache/incubator-airflow/pull/3593
On 10 July 2018 at 15:10:50, Carl Johan Gustavsson (
carl.j.gustavsson@gmail.com) wrote:
Hi Bolke,
I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all but 3.7
breaks
Quick repro:
➜ ~ pyenv local 3.5.5
➜ ~ locale
LANG=
LC_COLLATE="C"
LC_CTYPE="C"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=
➜ ~ cat testweird.txt
’
➜ ~ python
Python 3.5.5 (default, Jul 7 2018, 17:00:56)
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> open('testweird.txt').read()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/ascii.py",
line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0:
ordinal not in range(128)
>>>
Maybe not a blocking change but it is a breaking change from 1.9 I guess.
/ Carl Johan
On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbruin@gmail.com) wrote:
Hi Carl,
Did you this on python 3.5 as well? 3.6 is not an officially supported
(yet). As a workaround is available I won’t consider this blocking btw.
Bolke
Verstuurd vanaf mijn iPad
Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson <
carl.j.gustavsson@gmail.com> het volgende geschreven:
Hi,
First of all, thank you for all the work with the release management.
I ran in to a weird issue testing the RC1, running under Python 3.6.0 /
Ubuntu 14.04.5, upgrading from a master build from February.
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback (most
recent call last):
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
"/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in <module>
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
import configuration
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
"/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/__init__.py",
line 35, in <module>
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
import configuration as conf
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
"/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/configuration.py",
line 106, in <module>
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
DEFAULT_CONFIG = f.read()
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
"/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py", line
26, in decode
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return
codecs.ascii_decode(input, self.errors)[0]
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 20770:
ordinal not in range(128)
Removing the ’ on
https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L613
solved
the issue for me, and digging a bit deeper it seems Airflow now requires
setting LC_ALL=en_US.UTF-8 in the environment or similar to force Python to
read the file as utf-8 and not ascii. (I think this was changed in to
default to utf-8 in Python 3.7).
I see 3 solutions for this
1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or similar.
2. Change the default config file to not contain non-ascii characters.
3. Always read the file as unicode regardless of the LC_ALL environment,
by the encoding='utf-8’ parameter to open().
I think 3 is the best solution, and I can prepare a PR for that if
necessary .
I guess this counts as -1 (non-binding)
All the best
Carl Johan
On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbruin@gmail.com) wrote:
Hey all,
I have cut Airflow 1.10.0 RC1. This email is calling a vote on the release,
which will last for 72 hours. Consider this my (binding) +1.
Airflow 1.10.0 RC 1 is available at:
https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <
https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release that
comes with INSTALL instructions.
apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python "sdist"
release.
Public keys are available at:
https://dist.apache.org/repos/dist/release/incubator/airflow/ <
https://dist.apache.org/repos/dist/release/incubator/airflow/>
The amount of JIRAs fixed is over 700. Please have a look at the changelog.
Please note that the version number excludes the `rcX` string as well
as the "+incubating" string, so it's now simply 1.10.0. This will allow us
to rename the artifact without modifying the artifact checksums when we
actually release.
Cheers,
Bolke
Re: [VOTE] Airflow 1.10.0rc1
Posted by Bolke de Bruin <bd...@gmail.com>.
Thanks Fokko, will include your PR.
B.
> On 15 Jul 2018, at 12:07, Driesprong, Fokko <fo...@driesprong.frl> wrote:
>
> Hi all,
>
> I've did some more tests and it looks good. I was under the assumption that
> the sequential executor runs within the webserver, but this was a wrong
> assumption on my end. The behaviour is still the same as in 1.9. I've did
> some tests on Python 2.7, 3.5 and 3.6 and it looks good. Python 3.7 does
> not work yet, but it isn't supported anyway.
>
> I would like to have https://github.com/apache/incubator-airflow/pull/3604 in
> RC2. It isn't a critical bug, but it looks messy.
>
> Cheers, Fokko
>
> 2018-07-14 0:06 GMT+02:00 Driesprong, Fokko <fo...@driesprong.frl>:
>
>> Thanks Bolke for all the effort.
>>
>> I think I've miscommunicated the issue. It doesn't schedule the runs, and
>> when I explicitly kick of a run, it also isn't being picked up.
>>
>> Currently I'm doing a git bisect to check when this bug was introduced
>> happend. To be continued.
>>
>> Cheers, Fokko
>>
>> 2018-07-13 23:28 GMT+02:00 Bolke de Bruin <bd...@gmail.com>:
>>
>>> Hi Fokko,
>>>
>>> Please confirm this, because I tried sequential with a clean install. The
>>> only thing I found was that example DAGs were not picked up.
>>>
>>> I’m rolling rc2 anyway though so it would be good to get it fixed.
>>>
>>> B.
>>>
>>> Verstuurd vanaf mijn iPad
>>>
>>>> Op 13 jul. 2018 om 22:23 heeft Driesprong, Fokko <fo...@driesprong.frl>
>>> het volgende geschreven:
>>>>
>>>> Ok, I've did some testing.
>>>>
>>>> 1.10 works fine with the LocalExecutor. With the SequentialExecutor it
>>> does
>>>> not pick up any task, even with a different database as sqlite. Found
>>> this
>>>> one along the way: https://github.com/apache/incu
>>> bator-airflow/pull/3604
>>>>
>>>> There are no recent changes to the SequentialExecutor, so I'm still
>>> looking
>>>> how this bug found its way into the source. For me this is a -1, right
>>> now
>>>> it is not possible to just give Airflow a try using a basic setup with a
>>>> SequentialExecutor.
>>>>
>>>> Along the way this also makes me reconsider the tests. Like with the
>>>> Kubernetes test we just run a task, and then assert if it ran properly.
>>>> This might also be an idea for the sequential executor.
>>>>
>>>> Cheers, Fokko
>>>>
>>>> 2018-07-13 20:15 GMT+02:00 Jakob Homan <jg...@gmail.com>:
>>>>
>>>>> @Bolke - I didn't raise the concern, so I can't speak to whether or
>>>>> not Sebb will be ok with that. He tends to be pretty fastidious on
>>>>> this stuff and 'but some other TLP does it' hasn't gone over well
>>>>> before (trust me... I've tried). Totally up to you if you'd rather
>>>>> discuss it as part of the IPMC vote or just fix it to avoid
>>>>> discussion.
>>>>>
>>>>> -jakob
>>>>>
>>>>> On 13 July 2018 at 09:48, Ash Berlin-Taylor
>>>>> <as...@firemirror.com> wrote:
>>>>>> Cloud that be related to my ignorefile change? `airflow list_dags`
>>> still
>>>>> shows the example dags - the output is the same for that command as on
>>>>> v1-9-stable.
>>>>>>
>>>>>> Though I just noticed I'd left `self.log.info <http://self.log.info/
>>>> ()`
>>>>> in there. That's going to be noisy. https://github.com/apache/
>>>>> incubator-airflow/pull/3603 <https://github.com/apache/
>>>>> incubator-airflow/pull/3603>
>>>>>>
>>>>>> -ash
>>>>>>
>>>>>>> On 13 Jul 2018, at 17:36, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>>>>
>>>>>>> Example dags are not picked up. If you put a dag in the normal dag
>>>>> folder it works fine.
>>>>>>>
>>>>>>> Please create a jira for this @fokko. A pr would be appreciated.
>>>>>>>
>>>>>>> B.
>>>>>>>
>>>>>>> Sent from my iPhone
>>>>>>>
>>>>>>>> On 13 Jul 2018, at 15:46, Driesprong, Fokko <fo...@driesprong.frl>
>>>>> wrote:
>>>>>>>>
>>>>>>>> With the SequentialExecutor the webserver also acts as the scheduler
>>>>>>>> (without parallelism)
>>>>>>>>
>>>>>>>> 2018-07-13 15:43 GMT+02:00 Carl Johan Gustavsson <
>>>>> carl.johan@tictail.com>:
>>>>>>>>
>>>>>>
>>>>>
>>>
>>
>>
Re: [VOTE] Airflow 1.10.0rc1
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Hi all,
I've did some more tests and it looks good. I was under the assumption that
the sequential executor runs within the webserver, but this was a wrong
assumption on my end. The behaviour is still the same as in 1.9. I've did
some tests on Python 2.7, 3.5 and 3.6 and it looks good. Python 3.7 does
not work yet, but it isn't supported anyway.
I would like to have https://github.com/apache/incubator-airflow/pull/3604 in
RC2. It isn't a critical bug, but it looks messy.
Cheers, Fokko
2018-07-14 0:06 GMT+02:00 Driesprong, Fokko <fo...@driesprong.frl>:
> Thanks Bolke for all the effort.
>
> I think I've miscommunicated the issue. It doesn't schedule the runs, and
> when I explicitly kick of a run, it also isn't being picked up.
>
> Currently I'm doing a git bisect to check when this bug was introduced
> happend. To be continued.
>
> Cheers, Fokko
>
> 2018-07-13 23:28 GMT+02:00 Bolke de Bruin <bd...@gmail.com>:
>
>> Hi Fokko,
>>
>> Please confirm this, because I tried sequential with a clean install. The
>> only thing I found was that example DAGs were not picked up.
>>
>> I’m rolling rc2 anyway though so it would be good to get it fixed.
>>
>> B.
>>
>> Verstuurd vanaf mijn iPad
>>
>> > Op 13 jul. 2018 om 22:23 heeft Driesprong, Fokko <fo...@driesprong.frl>
>> het volgende geschreven:
>> >
>> > Ok, I've did some testing.
>> >
>> > 1.10 works fine with the LocalExecutor. With the SequentialExecutor it
>> does
>> > not pick up any task, even with a different database as sqlite. Found
>> this
>> > one along the way: https://github.com/apache/incu
>> bator-airflow/pull/3604
>> >
>> > There are no recent changes to the SequentialExecutor, so I'm still
>> looking
>> > how this bug found its way into the source. For me this is a -1, right
>> now
>> > it is not possible to just give Airflow a try using a basic setup with a
>> > SequentialExecutor.
>> >
>> > Along the way this also makes me reconsider the tests. Like with the
>> > Kubernetes test we just run a task, and then assert if it ran properly.
>> > This might also be an idea for the sequential executor.
>> >
>> > Cheers, Fokko
>> >
>> > 2018-07-13 20:15 GMT+02:00 Jakob Homan <jg...@gmail.com>:
>> >
>> >> @Bolke - I didn't raise the concern, so I can't speak to whether or
>> >> not Sebb will be ok with that. He tends to be pretty fastidious on
>> >> this stuff and 'but some other TLP does it' hasn't gone over well
>> >> before (trust me... I've tried). Totally up to you if you'd rather
>> >> discuss it as part of the IPMC vote or just fix it to avoid
>> >> discussion.
>> >>
>> >> -jakob
>> >>
>> >> On 13 July 2018 at 09:48, Ash Berlin-Taylor
>> >> <as...@firemirror.com> wrote:
>> >>> Cloud that be related to my ignorefile change? `airflow list_dags`
>> still
>> >> shows the example dags - the output is the same for that command as on
>> >> v1-9-stable.
>> >>>
>> >>> Though I just noticed I'd left `self.log.info <http://self.log.info/
>> >()`
>> >> in there. That's going to be noisy. https://github.com/apache/
>> >> incubator-airflow/pull/3603 <https://github.com/apache/
>> >> incubator-airflow/pull/3603>
>> >>>
>> >>> -ash
>> >>>
>> >>>> On 13 Jul 2018, at 17:36, Bolke de Bruin <bd...@gmail.com> wrote:
>> >>>>
>> >>>> Example dags are not picked up. If you put a dag in the normal dag
>> >> folder it works fine.
>> >>>>
>> >>>> Please create a jira for this @fokko. A pr would be appreciated.
>> >>>>
>> >>>> B.
>> >>>>
>> >>>> Sent from my iPhone
>> >>>>
>> >>>>> On 13 Jul 2018, at 15:46, Driesprong, Fokko <fo...@driesprong.frl>
>> >> wrote:
>> >>>>>
>> >>>>> With the SequentialExecutor the webserver also acts as the scheduler
>> >>>>> (without parallelism)
>> >>>>>
>> >>>>> 2018-07-13 15:43 GMT+02:00 Carl Johan Gustavsson <
>> >> carl.johan@tictail.com>:
>> >>>>>
>> >>>
>> >>
>>
>
>
Re: [VOTE] Airflow 1.10.0rc1
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Thanks Bolke for all the effort.
I think I've miscommunicated the issue. It doesn't schedule the runs, and
when I explicitly kick of a run, it also isn't being picked up.
Currently I'm doing a git bisect to check when this bug was introduced
happend. To be continued.
Cheers, Fokko
2018-07-13 23:28 GMT+02:00 Bolke de Bruin <bd...@gmail.com>:
> Hi Fokko,
>
> Please confirm this, because I tried sequential with a clean install. The
> only thing I found was that example DAGs were not picked up.
>
> I’m rolling rc2 anyway though so it would be good to get it fixed.
>
> B.
>
> Verstuurd vanaf mijn iPad
>
> > Op 13 jul. 2018 om 22:23 heeft Driesprong, Fokko <fo...@driesprong.frl>
> het volgende geschreven:
> >
> > Ok, I've did some testing.
> >
> > 1.10 works fine with the LocalExecutor. With the SequentialExecutor it
> does
> > not pick up any task, even with a different database as sqlite. Found
> this
> > one along the way: https://github.com/apache/incubator-airflow/pull/3604
> >
> > There are no recent changes to the SequentialExecutor, so I'm still
> looking
> > how this bug found its way into the source. For me this is a -1, right
> now
> > it is not possible to just give Airflow a try using a basic setup with a
> > SequentialExecutor.
> >
> > Along the way this also makes me reconsider the tests. Like with the
> > Kubernetes test we just run a task, and then assert if it ran properly.
> > This might also be an idea for the sequential executor.
> >
> > Cheers, Fokko
> >
> > 2018-07-13 20:15 GMT+02:00 Jakob Homan <jg...@gmail.com>:
> >
> >> @Bolke - I didn't raise the concern, so I can't speak to whether or
> >> not Sebb will be ok with that. He tends to be pretty fastidious on
> >> this stuff and 'but some other TLP does it' hasn't gone over well
> >> before (trust me... I've tried). Totally up to you if you'd rather
> >> discuss it as part of the IPMC vote or just fix it to avoid
> >> discussion.
> >>
> >> -jakob
> >>
> >> On 13 July 2018 at 09:48, Ash Berlin-Taylor
> >> <as...@firemirror.com> wrote:
> >>> Cloud that be related to my ignorefile change? `airflow list_dags`
> still
> >> shows the example dags - the output is the same for that command as on
> >> v1-9-stable.
> >>>
> >>> Though I just noticed I'd left `self.log.info <http://self.log.info/
> >()`
> >> in there. That's going to be noisy. https://github.com/apache/
> >> incubator-airflow/pull/3603 <https://github.com/apache/
> >> incubator-airflow/pull/3603>
> >>>
> >>> -ash
> >>>
> >>>> On 13 Jul 2018, at 17:36, Bolke de Bruin <bd...@gmail.com> wrote:
> >>>>
> >>>> Example dags are not picked up. If you put a dag in the normal dag
> >> folder it works fine.
> >>>>
> >>>> Please create a jira for this @fokko. A pr would be appreciated.
> >>>>
> >>>> B.
> >>>>
> >>>> Sent from my iPhone
> >>>>
> >>>>> On 13 Jul 2018, at 15:46, Driesprong, Fokko <fo...@driesprong.frl>
> >> wrote:
> >>>>>
> >>>>> With the SequentialExecutor the webserver also acts as the scheduler
> >>>>> (without parallelism)
> >>>>>
> >>>>> 2018-07-13 15:43 GMT+02:00 Carl Johan Gustavsson <
> >> carl.johan@tictail.com>:
> >>>>>
> >>>
> >>
>
Re: [VOTE] Airflow 1.10.0rc1
Posted by Bolke de Bruin <bd...@gmail.com>.
Hi Fokko,
Please confirm this, because I tried sequential with a clean install. The only thing I found was that example DAGs were not picked up.
I’m rolling rc2 anyway though so it would be good to get it fixed.
B.
Verstuurd vanaf mijn iPad
> Op 13 jul. 2018 om 22:23 heeft Driesprong, Fokko <fo...@driesprong.frl> het volgende geschreven:
>
> Ok, I've did some testing.
>
> 1.10 works fine with the LocalExecutor. With the SequentialExecutor it does
> not pick up any task, even with a different database as sqlite. Found this
> one along the way: https://github.com/apache/incubator-airflow/pull/3604
>
> There are no recent changes to the SequentialExecutor, so I'm still looking
> how this bug found its way into the source. For me this is a -1, right now
> it is not possible to just give Airflow a try using a basic setup with a
> SequentialExecutor.
>
> Along the way this also makes me reconsider the tests. Like with the
> Kubernetes test we just run a task, and then assert if it ran properly.
> This might also be an idea for the sequential executor.
>
> Cheers, Fokko
>
> 2018-07-13 20:15 GMT+02:00 Jakob Homan <jg...@gmail.com>:
>
>> @Bolke - I didn't raise the concern, so I can't speak to whether or
>> not Sebb will be ok with that. He tends to be pretty fastidious on
>> this stuff and 'but some other TLP does it' hasn't gone over well
>> before (trust me... I've tried). Totally up to you if you'd rather
>> discuss it as part of the IPMC vote or just fix it to avoid
>> discussion.
>>
>> -jakob
>>
>> On 13 July 2018 at 09:48, Ash Berlin-Taylor
>> <as...@firemirror.com> wrote:
>>> Cloud that be related to my ignorefile change? `airflow list_dags` still
>> shows the example dags - the output is the same for that command as on
>> v1-9-stable.
>>>
>>> Though I just noticed I'd left `self.log.info <http://self.log.info/>()`
>> in there. That's going to be noisy. https://github.com/apache/
>> incubator-airflow/pull/3603 <https://github.com/apache/
>> incubator-airflow/pull/3603>
>>>
>>> -ash
>>>
>>>> On 13 Jul 2018, at 17:36, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>
>>>> Example dags are not picked up. If you put a dag in the normal dag
>> folder it works fine.
>>>>
>>>> Please create a jira for this @fokko. A pr would be appreciated.
>>>>
>>>> B.
>>>>
>>>> Sent from my iPhone
>>>>
>>>>> On 13 Jul 2018, at 15:46, Driesprong, Fokko <fo...@driesprong.frl>
>> wrote:
>>>>>
>>>>> With the SequentialExecutor the webserver also acts as the scheduler
>>>>> (without parallelism)
>>>>>
>>>>> 2018-07-13 15:43 GMT+02:00 Carl Johan Gustavsson <
>> carl.johan@tictail.com>:
>>>>>
>>>
>>
Re: [VOTE] Airflow 1.10.0rc1
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Ok, I've did some testing.
1.10 works fine with the LocalExecutor. With the SequentialExecutor it does
not pick up any task, even with a different database as sqlite. Found this
one along the way: https://github.com/apache/incubator-airflow/pull/3604
There are no recent changes to the SequentialExecutor, so I'm still looking
how this bug found its way into the source. For me this is a -1, right now
it is not possible to just give Airflow a try using a basic setup with a
SequentialExecutor.
Along the way this also makes me reconsider the tests. Like with the
Kubernetes test we just run a task, and then assert if it ran properly.
This might also be an idea for the sequential executor.
Cheers, Fokko
2018-07-13 20:15 GMT+02:00 Jakob Homan <jg...@gmail.com>:
> @Bolke - I didn't raise the concern, so I can't speak to whether or
> not Sebb will be ok with that. He tends to be pretty fastidious on
> this stuff and 'but some other TLP does it' hasn't gone over well
> before (trust me... I've tried). Totally up to you if you'd rather
> discuss it as part of the IPMC vote or just fix it to avoid
> discussion.
>
> -jakob
>
> On 13 July 2018 at 09:48, Ash Berlin-Taylor
> <as...@firemirror.com> wrote:
> > Cloud that be related to my ignorefile change? `airflow list_dags` still
> shows the example dags - the output is the same for that command as on
> v1-9-stable.
> >
> > Though I just noticed I'd left `self.log.info <http://self.log.info/>()`
> in there. That's going to be noisy. https://github.com/apache/
> incubator-airflow/pull/3603 <https://github.com/apache/
> incubator-airflow/pull/3603>
> >
> > -ash
> >
> >> On 13 Jul 2018, at 17:36, Bolke de Bruin <bd...@gmail.com> wrote:
> >>
> >> Example dags are not picked up. If you put a dag in the normal dag
> folder it works fine.
> >>
> >> Please create a jira for this @fokko. A pr would be appreciated.
> >>
> >> B.
> >>
> >> Sent from my iPhone
> >>
> >>> On 13 Jul 2018, at 15:46, Driesprong, Fokko <fo...@driesprong.frl>
> wrote:
> >>>
> >>> With the SequentialExecutor the webserver also acts as the scheduler
> >>> (without parallelism)
> >>>
> >>> 2018-07-13 15:43 GMT+02:00 Carl Johan Gustavsson <
> carl.johan@tictail.com>:
> >>>
> >
>
Re: [VOTE] Airflow 1.10.0rc1
Posted by Jakob Homan <jg...@gmail.com>.
@Bolke - I didn't raise the concern, so I can't speak to whether or
not Sebb will be ok with that. He tends to be pretty fastidious on
this stuff and 'but some other TLP does it' hasn't gone over well
before (trust me... I've tried). Totally up to you if you'd rather
discuss it as part of the IPMC vote or just fix it to avoid
discussion.
-jakob
On 13 July 2018 at 09:48, Ash Berlin-Taylor
<as...@firemirror.com> wrote:
> Cloud that be related to my ignorefile change? `airflow list_dags` still shows the example dags - the output is the same for that command as on v1-9-stable.
>
> Though I just noticed I'd left `self.log.info <http://self.log.info/>()` in there. That's going to be noisy. https://github.com/apache/incubator-airflow/pull/3603 <https://github.com/apache/incubator-airflow/pull/3603>
>
> -ash
>
>> On 13 Jul 2018, at 17:36, Bolke de Bruin <bd...@gmail.com> wrote:
>>
>> Example dags are not picked up. If you put a dag in the normal dag folder it works fine.
>>
>> Please create a jira for this @fokko. A pr would be appreciated.
>>
>> B.
>>
>> Sent from my iPhone
>>
>>> On 13 Jul 2018, at 15:46, Driesprong, Fokko <fo...@driesprong.frl> wrote:
>>>
>>> With the SequentialExecutor the webserver also acts as the scheduler
>>> (without parallelism)
>>>
>>> 2018-07-13 15:43 GMT+02:00 Carl Johan Gustavsson <ca...@tictail.com>:
>>>
>
Re: [VOTE] Airflow 1.10.0rc1
Posted by Ash Berlin-Taylor <as...@firemirror.com>.
Cloud that be related to my ignorefile change? `airflow list_dags` still shows the example dags - the output is the same for that command as on v1-9-stable.
Though I just noticed I'd left `self.log.info <http://self.log.info/>()` in there. That's going to be noisy. https://github.com/apache/incubator-airflow/pull/3603 <https://github.com/apache/incubator-airflow/pull/3603>
-ash
> On 13 Jul 2018, at 17:36, Bolke de Bruin <bd...@gmail.com> wrote:
>
> Example dags are not picked up. If you put a dag in the normal dag folder it works fine.
>
> Please create a jira for this @fokko. A pr would be appreciated.
>
> B.
>
> Sent from my iPhone
>
>> On 13 Jul 2018, at 15:46, Driesprong, Fokko <fo...@driesprong.frl> wrote:
>>
>> With the SequentialExecutor the webserver also acts as the scheduler
>> (without parallelism)
>>
>> 2018-07-13 15:43 GMT+02:00 Carl Johan Gustavsson <ca...@tictail.com>:
>>
Re: [VOTE] Airflow 1.10.0rc1
Posted by Bolke de Bruin <bd...@gmail.com>.
Example dags are not picked up. If you put a dag in the normal dag folder it works fine.
Please create a jira for this @fokko. A pr would be appreciated.
B.
Sent from my iPhone
> On 13 Jul 2018, at 15:46, Driesprong, Fokko <fo...@driesprong.frl> wrote:
>
> With the SequentialExecutor the webserver also acts as the scheduler
> (without parallelism)
>
> 2018-07-13 15:43 GMT+02:00 Carl Johan Gustavsson <ca...@tictail.com>:
>
>> Need to run the scheduler also right?
>>
>> --
>> Carl Johan Gustavsson
>>
>> On 13 July 2018 at 15:37:22, Driesprong, Fokko (fokko@driesprong.frl)
>> wrote:
>>
>> I've installed Airflow, but using a clean install and the sequential
>> executor, the tasks are not being picked up.
>>
>> This is easy to replicate:
>>
>> docker run -t -i --rm -p 8080:8080 python:3.5 bash
>> pip install https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
>> 10.0rc1/apache-airflow-1.10.0rc1+incubating-bin.tar.gz
>> airflow initdb
>> airflow webserver
>>
>> I've enabled the http example dag and kicked of a dag, but it isn't being
>> picked up by the SequentialScheduler.
>>
>> - Fokko
>>
>>
>>
>>
>> 2018-07-13 11:51 GMT+02:00 Bolke de Bruin <bd...@gmail.com>:
>>
>>> Hi Sid,
>>>
>>> Do you have a JIRA and a PR to address it? I can then consider it for RC2
>>>
>>> B.
>>>
>>>> On 12 Jul 2018, at 04:50, Sid Anand <sa...@apache.org> wrote:
>>>>
>>>> FYI!
>>>> I just installed the release candidate. The first thing I noticed is a
>>> missing tool tip for the Null State in the Recent Tasks column on the
>>> landing page. Since the null globe is new to this UI, users will likely
>>> hover over it to inquire what it means... and will be left wanting. Of
>>> course, they could click on the globe, which will take them to
>>> http://localhost:8080/admin/taskinstance/?flt1_dag_id_equals
>>> =example_bash_operator&flt2_state_equals=null <
>>> http://localhost:8080/admin/taskinstance/?flt1_dag_id_equal
>>> s=example_bash_operator&flt2_state_equals=null>, which will always show
>>> an empty list, leaving them a bit more confused.
>>>>
>>>> -s
>>>>
>>>> On Wed, Jul 11, 2018 at 3:13 PM Carl Johan Gustavsson <
>>> carl.johan@tictail.com.invalid> wrote:
>>>> Hi Bolke,
>>>>
>>>> (Switching email to avoid moderation on my emails.)
>>>>
>>>> The normal Airflow test suite does not fail as it uses a LC_ALL set to
>>>> utf-8.
>>>>
>>>> I think it is a proper test though, it is a minimal reproducible
>>> version of
>>>> the code that fails. And the only difference in behaviour is at 3.7
>>> which
>>>> we don’t support anyway so I’m fairly sure it is broken for all
>>> supported
>>>> Python 3 versions.
>>>>
>>>> I now tried running the tests in docker using 3.5 with the LC_ALL/LANG
>>>> unset and I see the same failure.
>>>>
>>>> I don’t think this is a big thing though and we could release it without
>>>> the fix I made. I think most people run it with a sane LC_ALL, but
>>>> apparently we didn’t.
>>>> Here’s the log for the test:
>>>>
>>>>> docker run -t -i -v `pwd`:/airflow/ python:3.5 bash
>>>> root@b99b297df111:/# locale
>>>> LANG=C.UTF-8
>>>> LANGUAGE=
>>>> LC_CTYPE="C.UTF-8"
>>>> LC_NUMERIC="C.UTF-8"
>>>> LC_TIME="C.UTF-8"
>>>> LC_COLLATE="C.UTF-8"
>>>> LC_MONETARY="C.UTF-8"
>>>> LC_MESSAGES="C.UTF-8"
>>>> LC_PAPER="C.UTF-8"
>>>> LC_NAME="C.UTF-8"
>>>> LC_ADDRESS="C.UTF-8"
>>>> LC_TELEPHONE="C.UTF-8"
>>>> LC_MEASUREMENT="C.UTF-8"
>>>> LC_IDENTIFICATION="C.UTF-8"
>>>> LC_ALL=
>>>>> unset LANG
>>>> root@b99b297df111:/# locale
>>>> LANG=
>>>> LANGUAGE=
>>>> LC_CTYPE="POSIX"
>>>> LC_NUMERIC="POSIX"
>>>> LC_TIME="POSIX"
>>>> LC_COLLATE="POSIX"
>>>> LC_MONETARY="POSIX"
>>>> LC_MESSAGES="POSIX"
>>>> LC_PAPER="POSIX"
>>>> LC_NAME="POSIX"
>>>> LC_ADDRESS="POSIX"
>>>> LC_TELEPHONE="POSIX"
>>>> LC_MEASUREMENT="POSIX"
>>>> LC_IDENTIFICATION="POSIX"
>>>> LC_ALL=
>>>> root@b99b297df111:/# pip install -e .[devel]
>>>> root@b99b297df111:/airflow# ./run_unit_tests.sh
>>>> + export AIRFLOW_HOME=/root/airflow
>>>> + AIRFLOW_HOME=/root/airflow
>>>> + export AIRFLOW__CORE__UNIT_TEST_MODE=True
>>>> + AIRFLOW__CORE__UNIT_TEST_MODE=True
>>>> + export AIRFLOW__TESTSECTION__TESTKEY=testvalue
>>>> + AIRFLOW__TESTSECTION__TESTKEY=testvalue
>>>> + export AIRFLOW_USE_NEW_IMPORTS=1
>>>> + AIRFLOW_USE_NEW_IMPORTS=1
>>>> +++ dirname ./run_unit_tests.sh
>>>> ++ cd .
>>>> ++ pwd
>>>> + DIR=/airflow
>>>> + export PYTHONPATH=:/airflow/tests/test_utils
>>>> + PYTHONPATH=:/airflow/tests/test_utils
>>>> + nose_args=
>>>> + which airflow
>>>> + echo 'Initializing the DB'
>>>> Initializing the DB
>>>> + airflow resetdb
>>>> + yes
>>>> Traceback (most recent call last):
>>>> File "/usr/local/bin/airflow", line 6, in <module>
>>>> exec(compile(open(__file__).read(), __file__, 'exec'))
>>>> File "/airflow/airflow/bin/airflow", line 21, in <module>
>>>> from airflow import configuration
>>>> File "/airflow/airflow/__init__.py", line 35, in <module>
>>>> from airflow import configuration as conf
>>>> File "/airflow/airflow/configuration.py", line 106, in <module>
>>>> DEFAULT_CONFIG = f.read()
>>>> File "/usr/local/lib/python3.5/encodings/ascii.py", line 26, in
>>> decode
>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>> 21082:
>>>> ordinal not in range(128)
>>>> + '[' '' ']'
>>>> + '[' -z '' ']'
>>>> + nose_args='--with-coverage --cover-erase --cover-html
>>>> --cover-package=airflow --cover-html-dir=airflow/www/
>>> static/coverage
>>>> --with-ignore-docstrings --rednose --with-timer -s -v
>>>> --logging-level=DEBUG '
>>>> + echo 'Starting the unit tests with the following nose arguments:
>>>> --with-coverage' --cover-erase --cover-html --cover-package=airflow
>>>> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
>>>> --rednose --with-timer -s -v --logging-level=DEBUG
>>>> Starting the unit tests with the following nose arguments:
>>> --with-coverage
>>>> --cover-erase --cover-html --cover-package=airflow
>>>> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
>>>> --rednose --with-timer -s -v --logging-level=DEBUG
>>>> + nosetests --with-coverage --cover-erase --cover-html
>>>> --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
>>>> --with-ignore-docstrings --rednose --with-timer -s -v
>>> --logging-level=DEBUG
>>>> nose.plugins.cover: ERROR: Coverage not available: unable to import
>>>> coverage module
>>>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>>>> position 21082: ordinal not in range(128)) ... ERROR
>>>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>>>> position 21082: ordinal not in range(128)) ... ERROR
>>>> ======================================================================
>>>> 1) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte
>>> 0xe2
>>>> in position 21082: ordinal not in range(128))
>>>> ----------------------------------------------------------------------
>>>> Traceback (most recent call last):
>>>> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>>>> runTest
>>>> raise self.exc_val.with_traceback(self.tb)
>>>> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>>>> loadTestsFromName
>>>> addr.filename, addr.module)
>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>>>> importFromPath
>>>> return self.importFromDir(dir_path, fqname)
>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>>>> importFromDir
>>>> mod = load_module(part_fqname, fh, filename, desc)
>>>> /usr/local/lib/python3.5/imp.py line 245 in load_module
>>>> return load_package(name, filename)
>>>> /usr/local/lib/python3.5/imp.py line 217 in load_package
>>>> return _load(spec)
>>>> <frozen importlib._bootstrap> line 693 in _load
>>>>
>>>> <frozen importlib._bootstrap> line 673 in _load_unlocked
>>>>
>>>> <frozen importlib._bootstrap_external> line 697 in exec_module
>>>>
>>>> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>>>>
>>>> airflow/__init__.py line 35 in <module>
>>>> from airflow import configuration as conf
>>>> airflow/configuration.py line 106 in <module>
>>>> DEFAULT_CONFIG = f.read()
>>>> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>>> 21082: ordinal not in range(128)
>>>> ======================================================================
>>>> 2) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte
>>> 0xe2
>>>> in position 21082: ordinal not in range(128))
>>>> ----------------------------------------------------------------------
>>>> Traceback (most recent call last):
>>>> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>>>> runTest
>>>> raise self.exc_val.with_traceback(self.tb)
>>>> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>>>> loadTestsFromName
>>>> addr.filename, addr.module)
>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>>>> importFromPath
>>>> return self.importFromDir(dir_path, fqname)
>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>>>> importFromDir
>>>> mod = load_module(part_fqname, fh, filename, desc)
>>>> /usr/local/lib/python3.5/imp.py line 245 in load_module
>>>> return load_package(name, filename)
>>>> /usr/local/lib/python3.5/imp.py line 217 in load_package
>>>> return _load(spec)
>>>> <frozen importlib._bootstrap> line 693 in _load
>>>>
>>>> <frozen importlib._bootstrap> line 673 in _load_unlocked
>>>>
>>>> <frozen importlib._bootstrap_external> line 697 in exec_module
>>>>
>>>> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>>>>
>>>> tests/__init__.py line 25 in <module>
>>>> from .configuration import *
>>>> tests/configuration.py line 28 in <module>
>>>> from airflow import configuration
>>>> airflow/__init__.py line 35 in <module>
>>>> from airflow import configuration as conf
>>>> airflow/configuration.py line 106 in <module>
>>>> DEFAULT_CONFIG = f.read()
>>>> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>>> 21082: ordinal not in range(128)
>>>>
>>>> [error] 100.00% nose.failure.Failure.runTest: 0.0004s
>>>> ------------------------------------------------------------
>>> -----------------
>>>> 2 tests run in 0.060 seconds.
>>>> 2 errors (0 tests passed)
>>>>
>>>>
>>>> --
>>>> Carl Johan Gustavsson
>>>>
>>>> On 11 July 2018 at 23:24:01, Bolke de Bruin (bdbruin@gmail.com <mailto:
>>> bdbruin@gmail.com>) wrote:
>>>>
>>>> Hi Carl,
>>>>
>>>> That is not a real check, ie. Does Airflow have the same issue clean
>>>> install on 3.5? Travis’ tests run on 3.5.
>>>>
>>>> B.
>>>>
>>>> Verstuurd vanaf mijn iPad
>>>>
>>>>> Op 10 jul. 2018 om 15:10 heeft Carl Johan Gustavsson <
>>>> carl.j.gustavsson@gmail.com <ma...@gmail.com>> het
>>> volgende geschreven:
>>>>>
>>>>> Hi Bolke,
>>>>>
>>>>>
>>>>> I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all but 3.7
>>>> breaks
>>>>>
>>>>> Quick repro:
>>>>>
>>>>> ➜ ~ pyenv local 3.5.5
>>>>> ➜ ~ locale
>>>>> LANG=
>>>>> LC_COLLATE="C"
>>>>> LC_CTYPE="C"
>>>>> LC_MESSAGES="C"
>>>>> LC_MONETARY="C"
>>>>> LC_NUMERIC="C"
>>>>> LC_TIME="C"
>>>>> LC_ALL=
>>>>> ➜ ~ cat testweird.txt
>>>>> ’
>>>>> ➜ ~ python
>>>>> Python 3.5.5 (default, Jul 7 2018, 17:00:56)
>>>>> [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
>>>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>>>> open('testweird.txt').read()
>>>>> Traceback (most recent call last):
>>>>> File "<stdin>", line 1, in <module>
>>>>> File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/as
>>> cii.py",
>>>> line 26, in decode
>>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>> 0:
>>>> ordinal not in range(128)
>>>>>>>>
>>>>>
>>>>>
>>>>> Maybe not a blocking change but it is a breaking change from 1.9 I
>>> guess.
>>>>>
>>>>>
>>>>> / Carl Johan
>>>>>
>>>>>
>>>>>> On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbruin@gmail.com
>>> <ma...@gmail.com>) wrote:
>>>>>>
>>>>>> Hi Carl,
>>>>>>
>>>>>> Did you this on python 3.5 as well? 3.6 is not an officially
>>> supported
>>>> (yet). As a workaround is available I won’t consider this blocking btw.
>>>>>>
>>>>>> Bolke
>>>>>>
>>>>>> Verstuurd vanaf mijn iPad
>>>>>>
>>>>>> Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson <
>>>> carl.j.gustavsson@gmail.com <ma...@gmail.com>> het
>>> volgende geschreven:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> First of all, thank you for all the work with the release
>>> management.
>>>>>>>
>>>>>>> I ran in to a weird issue testing the RC1, running under Python
>>> 3.6.0 /
>>>> Ubuntu 14.04.5, upgrading from a master build from February.
>>>>>>>
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback
>>>> (most recent call last):
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>> "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in <module>
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from
>>> airflow
>>>> import configuration
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-package
>>> s/airflow/__init__.py",
>>>> line 35, in <module>
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from
>>> airflow
>>>> import configuration as conf
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-package
>>> s/airflow/configuration.py",
>>>> line 106, in <module>
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>>>> DEFAULT_CONFIG = f.read()
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py",
>>> line
>>>> 26, in decode
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return
>>>> codecs.ascii_decode(input, self.errors)[0]
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>> 20770:
>>>> ordinal not in range(128)
>>>>>>>
>>>>>>> Removing the ’ on
>>>> https://github.com/apache/incubator-airflow/blob/master/airf
>>> low/config_templates/default_airflow.cfg#L613 <
>>> https://github.com/apache/incubator-airflow/blob/master/air
>>> flow/config_templates/default_airflow.cfg#L613>
>>>> solved the issue for me, and digging a bit deeper it seems Airflow now
>>>> requires setting LC_ALL=en_US.UTF-8 in the environment or similar to
>>> force
>>>> Python to read the file as utf-8 and not ascii. (I think this was
>>> changed
>>>> in to default to utf-8 in Python 3.7).
>>>>>>>
>>>>>>> I see 3 solutions for this
>>>>>>> 1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or
>>>> similar.
>>>>>>> 2. Change the default config file to not contain non-ascii
>>> characters.
>>>>>>> 3. Always read the file as unicode regardless of the LC_ALL
>>>> environment, by the encoding='utf-8’ parameter to open().
>>>>>>>
>>>>>>> I think 3 is the best solution, and I can prepare a PR for that if
>>>> necessary .
>>>>>>>
>>>>>>> I guess this counts as -1 (non-binding)
>>>>>>>
>>>>>>> All the best
>>>>>>>
>>>>>>> Carl Johan
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbruin@gmail.com
>>> <ma...@gmail.com>) wrote:
>>>>>>>>
>>>>>>>> Hey all,
>>>>>>>>
>>>>>>>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the
>>>> release,
>>>>>>>> which will last for 72 hours. Consider this my (binding) +1.
>>>>>>>>
>>>>>>>> Airflow 1.10.0 RC 1 is available at:
>>>>>>>>
>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.1
>>> 0.0rc1/ <https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
>>> 10.0rc1/> <
>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <
>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>>
>>>>>>>>
>>>>>>>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source
>>> release
>>>> that
>>>>>>>> comes with INSTALL instructions.
>>>>>>>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary
>>> Python
>>>> "sdist"
>>>>>>>> release.
>>>>>>>>
>>>>>>>> Public keys are available at:
>>>>>>>>
>>>>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>>> https://dist.apache.org/repos/dist/release/incubator/airflow/> <
>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>>> https://dist.apache.org/repos/dist/release/incubator/airflow/>>
>>>>>>>>
>>>>>>>> The amount of JIRAs fixed is over 700. Please have a look at the
>>>> changelog.
>>>>>>>>
>>>>>>>> Please note that the version number excludes the `rcX` string as
>>> well
>>>>>>>> as the "+incubating" string, so it's now simply 1.10.0. This will
>>>> allow us
>>>>>>>> to rename the artifact without modifying the artifact checksums
>>> when
>>>> we
>>>>>>>> actually release.
>>>>>>>>
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Bolke
>>>
>>>
>>
Re: [VOTE] Airflow 1.10.0rc1
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
With the SequentialExecutor the webserver also acts as the scheduler
(without parallelism)
2018-07-13 15:43 GMT+02:00 Carl Johan Gustavsson <ca...@tictail.com>:
> Need to run the scheduler also right?
>
> --
> Carl Johan Gustavsson
>
> On 13 July 2018 at 15:37:22, Driesprong, Fokko (fokko@driesprong.frl)
> wrote:
>
> I've installed Airflow, but using a clean install and the sequential
> executor, the tasks are not being picked up.
>
> This is easy to replicate:
>
> docker run -t -i --rm -p 8080:8080 python:3.5 bash
> pip install https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
> 10.0rc1/apache-airflow-1.10.0rc1+incubating-bin.tar.gz
> airflow initdb
> airflow webserver
>
> I've enabled the http example dag and kicked of a dag, but it isn't being
> picked up by the SequentialScheduler.
>
> - Fokko
>
>
>
>
> 2018-07-13 11:51 GMT+02:00 Bolke de Bruin <bd...@gmail.com>:
>
>> Hi Sid,
>>
>> Do you have a JIRA and a PR to address it? I can then consider it for RC2
>>
>> B.
>>
>> > On 12 Jul 2018, at 04:50, Sid Anand <sa...@apache.org> wrote:
>> >
>> > FYI!
>> > I just installed the release candidate. The first thing I noticed is a
>> missing tool tip for the Null State in the Recent Tasks column on the
>> landing page. Since the null globe is new to this UI, users will likely
>> hover over it to inquire what it means... and will be left wanting. Of
>> course, they could click on the globe, which will take them to
>> http://localhost:8080/admin/taskinstance/?flt1_dag_id_equals
>> =example_bash_operator&flt2_state_equals=null <
>> http://localhost:8080/admin/taskinstance/?flt1_dag_id_equal
>> s=example_bash_operator&flt2_state_equals=null>, which will always show
>> an empty list, leaving them a bit more confused.
>> >
>> > -s
>> >
>> > On Wed, Jul 11, 2018 at 3:13 PM Carl Johan Gustavsson <
>> carl.johan@tictail.com.invalid> wrote:
>> > Hi Bolke,
>> >
>> > (Switching email to avoid moderation on my emails.)
>> >
>> > The normal Airflow test suite does not fail as it uses a LC_ALL set to
>> > utf-8.
>> >
>> > I think it is a proper test though, it is a minimal reproducible
>> version of
>> > the code that fails. And the only difference in behaviour is at 3.7
>> which
>> > we don’t support anyway so I’m fairly sure it is broken for all
>> supported
>> > Python 3 versions.
>> >
>> > I now tried running the tests in docker using 3.5 with the LC_ALL/LANG
>> > unset and I see the same failure.
>> >
>> > I don’t think this is a big thing though and we could release it without
>> > the fix I made. I think most people run it with a sane LC_ALL, but
>> > apparently we didn’t.
>> > Here’s the log for the test:
>> >
>> > > docker run -t -i -v `pwd`:/airflow/ python:3.5 bash
>> > root@b99b297df111:/# locale
>> > LANG=C.UTF-8
>> > LANGUAGE=
>> > LC_CTYPE="C.UTF-8"
>> > LC_NUMERIC="C.UTF-8"
>> > LC_TIME="C.UTF-8"
>> > LC_COLLATE="C.UTF-8"
>> > LC_MONETARY="C.UTF-8"
>> > LC_MESSAGES="C.UTF-8"
>> > LC_PAPER="C.UTF-8"
>> > LC_NAME="C.UTF-8"
>> > LC_ADDRESS="C.UTF-8"
>> > LC_TELEPHONE="C.UTF-8"
>> > LC_MEASUREMENT="C.UTF-8"
>> > LC_IDENTIFICATION="C.UTF-8"
>> > LC_ALL=
>> > > unset LANG
>> > root@b99b297df111:/# locale
>> > LANG=
>> > LANGUAGE=
>> > LC_CTYPE="POSIX"
>> > LC_NUMERIC="POSIX"
>> > LC_TIME="POSIX"
>> > LC_COLLATE="POSIX"
>> > LC_MONETARY="POSIX"
>> > LC_MESSAGES="POSIX"
>> > LC_PAPER="POSIX"
>> > LC_NAME="POSIX"
>> > LC_ADDRESS="POSIX"
>> > LC_TELEPHONE="POSIX"
>> > LC_MEASUREMENT="POSIX"
>> > LC_IDENTIFICATION="POSIX"
>> > LC_ALL=
>> > root@b99b297df111:/# pip install -e .[devel]
>> > root@b99b297df111:/airflow# ./run_unit_tests.sh
>> > + export AIRFLOW_HOME=/root/airflow
>> > + AIRFLOW_HOME=/root/airflow
>> > + export AIRFLOW__CORE__UNIT_TEST_MODE=True
>> > + AIRFLOW__CORE__UNIT_TEST_MODE=True
>> > + export AIRFLOW__TESTSECTION__TESTKEY=testvalue
>> > + AIRFLOW__TESTSECTION__TESTKEY=testvalue
>> > + export AIRFLOW_USE_NEW_IMPORTS=1
>> > + AIRFLOW_USE_NEW_IMPORTS=1
>> > +++ dirname ./run_unit_tests.sh
>> > ++ cd .
>> > ++ pwd
>> > + DIR=/airflow
>> > + export PYTHONPATH=:/airflow/tests/test_utils
>> > + PYTHONPATH=:/airflow/tests/test_utils
>> > + nose_args=
>> > + which airflow
>> > + echo 'Initializing the DB'
>> > Initializing the DB
>> > + airflow resetdb
>> > + yes
>> > Traceback (most recent call last):
>> > File "/usr/local/bin/airflow", line 6, in <module>
>> > exec(compile(open(__file__).read(), __file__, 'exec'))
>> > File "/airflow/airflow/bin/airflow", line 21, in <module>
>> > from airflow import configuration
>> > File "/airflow/airflow/__init__.py", line 35, in <module>
>> > from airflow import configuration as conf
>> > File "/airflow/airflow/configuration.py", line 106, in <module>
>> > DEFAULT_CONFIG = f.read()
>> > File "/usr/local/lib/python3.5/encodings/ascii.py", line 26, in
>> decode
>> > return codecs.ascii_decode(input, self.errors)[0]
>> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>> 21082:
>> > ordinal not in range(128)
>> > + '[' '' ']'
>> > + '[' -z '' ']'
>> > + nose_args='--with-coverage --cover-erase --cover-html
>> > --cover-package=airflow --cover-html-dir=airflow/www/
>> static/coverage
>> > --with-ignore-docstrings --rednose --with-timer -s -v
>> > --logging-level=DEBUG '
>> > + echo 'Starting the unit tests with the following nose arguments:
>> > --with-coverage' --cover-erase --cover-html --cover-package=airflow
>> > --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
>> > --rednose --with-timer -s -v --logging-level=DEBUG
>> > Starting the unit tests with the following nose arguments:
>> --with-coverage
>> > --cover-erase --cover-html --cover-package=airflow
>> > --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
>> > --rednose --with-timer -s -v --logging-level=DEBUG
>> > + nosetests --with-coverage --cover-erase --cover-html
>> > --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
>> > --with-ignore-docstrings --rednose --with-timer -s -v
>> --logging-level=DEBUG
>> > nose.plugins.cover: ERROR: Coverage not available: unable to import
>> > coverage module
>> > Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>> > position 21082: ordinal not in range(128)) ... ERROR
>> > Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>> > position 21082: ordinal not in range(128)) ... ERROR
>> > ======================================================================
>> > 1) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte
>> 0xe2
>> > in position 21082: ordinal not in range(128))
>> > ----------------------------------------------------------------------
>> > Traceback (most recent call last):
>> > /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>> > runTest
>> > raise self.exc_val.with_traceback(self.tb)
>> > /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>> > loadTestsFromName
>> > addr.filename, addr.module)
>> > /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>> > importFromPath
>> > return self.importFromDir(dir_path, fqname)
>> > /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>> > importFromDir
>> > mod = load_module(part_fqname, fh, filename, desc)
>> > /usr/local/lib/python3.5/imp.py line 245 in load_module
>> > return load_package(name, filename)
>> > /usr/local/lib/python3.5/imp.py line 217 in load_package
>> > return _load(spec)
>> > <frozen importlib._bootstrap> line 693 in _load
>> >
>> > <frozen importlib._bootstrap> line 673 in _load_unlocked
>> >
>> > <frozen importlib._bootstrap_external> line 697 in exec_module
>> >
>> > <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>> >
>> > airflow/__init__.py line 35 in <module>
>> > from airflow import configuration as conf
>> > airflow/configuration.py line 106 in <module>
>> > DEFAULT_CONFIG = f.read()
>> > /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>> > return codecs.ascii_decode(input, self.errors)[0]
>> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>> > 21082: ordinal not in range(128)
>> > ======================================================================
>> > 2) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte
>> 0xe2
>> > in position 21082: ordinal not in range(128))
>> > ----------------------------------------------------------------------
>> > Traceback (most recent call last):
>> > /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>> > runTest
>> > raise self.exc_val.with_traceback(self.tb)
>> > /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>> > loadTestsFromName
>> > addr.filename, addr.module)
>> > /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>> > importFromPath
>> > return self.importFromDir(dir_path, fqname)
>> > /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>> > importFromDir
>> > mod = load_module(part_fqname, fh, filename, desc)
>> > /usr/local/lib/python3.5/imp.py line 245 in load_module
>> > return load_package(name, filename)
>> > /usr/local/lib/python3.5/imp.py line 217 in load_package
>> > return _load(spec)
>> > <frozen importlib._bootstrap> line 693 in _load
>> >
>> > <frozen importlib._bootstrap> line 673 in _load_unlocked
>> >
>> > <frozen importlib._bootstrap_external> line 697 in exec_module
>> >
>> > <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>> >
>> > tests/__init__.py line 25 in <module>
>> > from .configuration import *
>> > tests/configuration.py line 28 in <module>
>> > from airflow import configuration
>> > airflow/__init__.py line 35 in <module>
>> > from airflow import configuration as conf
>> > airflow/configuration.py line 106 in <module>
>> > DEFAULT_CONFIG = f.read()
>> > /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>> > return codecs.ascii_decode(input, self.errors)[0]
>> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>> > 21082: ordinal not in range(128)
>> >
>> > [error] 100.00% nose.failure.Failure.runTest: 0.0004s
>> > ------------------------------------------------------------
>> -----------------
>> > 2 tests run in 0.060 seconds.
>> > 2 errors (0 tests passed)
>> >
>> >
>> > --
>> > Carl Johan Gustavsson
>> >
>> > On 11 July 2018 at 23:24:01, Bolke de Bruin (bdbruin@gmail.com <mailto:
>> bdbruin@gmail.com>) wrote:
>> >
>> > Hi Carl,
>> >
>> > That is not a real check, ie. Does Airflow have the same issue clean
>> > install on 3.5? Travis’ tests run on 3.5.
>> >
>> > B.
>> >
>> > Verstuurd vanaf mijn iPad
>> >
>> > > Op 10 jul. 2018 om 15:10 heeft Carl Johan Gustavsson <
>> > carl.j.gustavsson@gmail.com <ma...@gmail.com>> het
>> volgende geschreven:
>> > >
>> > > Hi Bolke,
>> > >
>> > >
>> > > I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all but 3.7
>> > breaks
>> > >
>> > > Quick repro:
>> > >
>> > > ➜ ~ pyenv local 3.5.5
>> > > ➜ ~ locale
>> > > LANG=
>> > > LC_COLLATE="C"
>> > > LC_CTYPE="C"
>> > > LC_MESSAGES="C"
>> > > LC_MONETARY="C"
>> > > LC_NUMERIC="C"
>> > > LC_TIME="C"
>> > > LC_ALL=
>> > > ➜ ~ cat testweird.txt
>> > > ’
>> > > ➜ ~ python
>> > > Python 3.5.5 (default, Jul 7 2018, 17:00:56)
>> > > [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
>> > > Type "help", "copyright", "credits" or "license" for more information.
>> > > >>> open('testweird.txt').read()
>> > > Traceback (most recent call last):
>> > > File "<stdin>", line 1, in <module>
>> > > File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/as
>> cii.py",
>> > line 26, in decode
>> > > return codecs.ascii_decode(input, self.errors)[0]
>> > > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>> 0:
>> > ordinal not in range(128)
>> > > >>>
>> > >
>> > >
>> > > Maybe not a blocking change but it is a breaking change from 1.9 I
>> guess.
>> > >
>> > >
>> > > / Carl Johan
>> > >
>> > >
>> > >> On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbruin@gmail.com
>> <ma...@gmail.com>) wrote:
>> > >>
>> > >> Hi Carl,
>> > >>
>> > >> Did you this on python 3.5 as well? 3.6 is not an officially
>> supported
>> > (yet). As a workaround is available I won’t consider this blocking btw.
>> > >>
>> > >> Bolke
>> > >>
>> > >> Verstuurd vanaf mijn iPad
>> > >>
>> > >> Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson <
>> > carl.j.gustavsson@gmail.com <ma...@gmail.com>> het
>> volgende geschreven:
>> > >>
>> > >>> Hi,
>> > >>>
>> > >>> First of all, thank you for all the work with the release
>> management.
>> > >>>
>> > >>> I ran in to a weird issue testing the RC1, running under Python
>> 3.6.0 /
>> > Ubuntu 14.04.5, upgrading from a master build from February.
>> > >>>
>> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback
>> > (most recent call last):
>> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>> > "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in <module>
>> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from
>> airflow
>> > import configuration
>> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>> > "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-package
>> s/airflow/__init__.py",
>> > line 35, in <module>
>> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from
>> airflow
>> > import configuration as conf
>> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>> > "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-package
>> s/airflow/configuration.py",
>> > line 106, in <module>
>> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>> > DEFAULT_CONFIG = f.read()
>> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>> > "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py",
>> line
>> > 26, in decode
>> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return
>> > codecs.ascii_decode(input, self.errors)[0]
>> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>> 20770:
>> > ordinal not in range(128)
>> > >>>
>> > >>> Removing the ’ on
>> > https://github.com/apache/incubator-airflow/blob/master/airf
>> low/config_templates/default_airflow.cfg#L613 <
>> https://github.com/apache/incubator-airflow/blob/master/air
>> flow/config_templates/default_airflow.cfg#L613>
>> > solved the issue for me, and digging a bit deeper it seems Airflow now
>> > requires setting LC_ALL=en_US.UTF-8 in the environment or similar to
>> force
>> > Python to read the file as utf-8 and not ascii. (I think this was
>> changed
>> > in to default to utf-8 in Python 3.7).
>> > >>>
>> > >>> I see 3 solutions for this
>> > >>> 1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or
>> > similar.
>> > >>> 2. Change the default config file to not contain non-ascii
>> characters.
>> > >>> 3. Always read the file as unicode regardless of the LC_ALL
>> > environment, by the encoding='utf-8’ parameter to open().
>> > >>>
>> > >>> I think 3 is the best solution, and I can prepare a PR for that if
>> > necessary .
>> > >>>
>> > >>> I guess this counts as -1 (non-binding)
>> > >>>
>> > >>> All the best
>> > >>>
>> > >>> Carl Johan
>> > >>>
>> > >>>
>> > >>>
>> > >>>> On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbruin@gmail.com
>> <ma...@gmail.com>) wrote:
>> > >>>>
>> > >>>> Hey all,
>> > >>>>
>> > >>>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the
>> > release,
>> > >>>> which will last for 72 hours. Consider this my (binding) +1.
>> > >>>>
>> > >>>> Airflow 1.10.0 RC 1 is available at:
>> > >>>>
>> > >>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.1
>> 0.0rc1/ <https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
>> 10.0rc1/> <
>> > https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <
>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>>
>> > >>>>
>> > >>>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source
>> release
>> > that
>> > >>>> comes with INSTALL instructions.
>> > >>>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary
>> Python
>> > "sdist"
>> > >>>> release.
>> > >>>>
>> > >>>> Public keys are available at:
>> > >>>>
>> > >>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>> https://dist.apache.org/repos/dist/release/incubator/airflow/> <
>> > https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>> https://dist.apache.org/repos/dist/release/incubator/airflow/>>
>> > >>>>
>> > >>>> The amount of JIRAs fixed is over 700. Please have a look at the
>> > changelog.
>> > >>>>
>> > >>>> Please note that the version number excludes the `rcX` string as
>> well
>> > >>>> as the "+incubating" string, so it's now simply 1.10.0. This will
>> > allow us
>> > >>>> to rename the artifact without modifying the artifact checksums
>> when
>> > we
>> > >>>> actually release.
>> > >>>>
>> > >>>>
>> > >>>> Cheers,
>> > >>>> Bolke
>>
>>
>
Re: [VOTE] Airflow 1.10.0rc1
Posted by Carl Johan Gustavsson <ca...@tictail.com.INVALID>.
Need to run the scheduler also right?
--
Carl Johan Gustavsson
On 13 July 2018 at 15:37:22, Driesprong, Fokko (fokko@driesprong.frl) wrote:
I've installed Airflow, but using a clean install and the sequential
executor, the tasks are not being picked up.
This is easy to replicate:
docker run -t -i --rm -p 8080:8080 python:3.5 bash
pip install
https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/apache-airflow-1.10.0rc1+incubating-bin.tar.gz
airflow initdb
airflow webserver
I've enabled the http example dag and kicked of a dag, but it isn't being
picked up by the SequentialScheduler.
- Fokko
2018-07-13 11:51 GMT+02:00 Bolke de Bruin <bd...@gmail.com>:
> Hi Sid,
>
> Do you have a JIRA and a PR to address it? I can then consider it for RC2
>
> B.
>
> > On 12 Jul 2018, at 04:50, Sid Anand <sa...@apache.org> wrote:
> >
> > FYI!
> > I just installed the release candidate. The first thing I noticed is a
> missing tool tip for the Null State in the Recent Tasks column on the
> landing page. Since the null globe is new to this UI, users will likely
> hover over it to inquire what it means... and will be left wanting. Of
> course, they could click on the globe, which will take them to
> http://localhost:8080/admin/taskinstance/?flt1_dag_id_
> equals=example_bash_operator&flt2_state_equals=null <
> http://localhost:8080/admin/taskinstance/?flt1_dag_id_
> equals=example_bash_operator&flt2_state_equals=null>, which will always
> show an empty list, leaving them a bit more confused.
> >
> > -s
> >
> > On Wed, Jul 11, 2018 at 3:13 PM Carl Johan Gustavsson <
> carl.johan@tictail.com.invalid> wrote:
> > Hi Bolke,
> >
> > (Switching email to avoid moderation on my emails.)
> >
> > The normal Airflow test suite does not fail as it uses a LC_ALL set to
> > utf-8.
> >
> > I think it is a proper test though, it is a minimal reproducible version
> of
> > the code that fails. And the only difference in behaviour is at 3.7 which
> > we don’t support anyway so I’m fairly sure it is broken for all supported
> > Python 3 versions.
> >
> > I now tried running the tests in docker using 3.5 with the LC_ALL/LANG
> > unset and I see the same failure.
> >
> > I don’t think this is a big thing though and we could release it without
> > the fix I made. I think most people run it with a sane LC_ALL, but
> > apparently we didn’t.
> > Here’s the log for the test:
> >
> > > docker run -t -i -v `pwd`:/airflow/ python:3.5 bash
> > root@b99b297df111:/# locale
> > LANG=C.UTF-8
> > LANGUAGE=
> > LC_CTYPE="C.UTF-8"
> > LC_NUMERIC="C.UTF-8"
> > LC_TIME="C.UTF-8"
> > LC_COLLATE="C.UTF-8"
> > LC_MONETARY="C.UTF-8"
> > LC_MESSAGES="C.UTF-8"
> > LC_PAPER="C.UTF-8"
> > LC_NAME="C.UTF-8"
> > LC_ADDRESS="C.UTF-8"
> > LC_TELEPHONE="C.UTF-8"
> > LC_MEASUREMENT="C.UTF-8"
> > LC_IDENTIFICATION="C.UTF-8"
> > LC_ALL=
> > > unset LANG
> > root@b99b297df111:/# locale
> > LANG=
> > LANGUAGE=
> > LC_CTYPE="POSIX"
> > LC_NUMERIC="POSIX"
> > LC_TIME="POSIX"
> > LC_COLLATE="POSIX"
> > LC_MONETARY="POSIX"
> > LC_MESSAGES="POSIX"
> > LC_PAPER="POSIX"
> > LC_NAME="POSIX"
> > LC_ADDRESS="POSIX"
> > LC_TELEPHONE="POSIX"
> > LC_MEASUREMENT="POSIX"
> > LC_IDENTIFICATION="POSIX"
> > LC_ALL=
> > root@b99b297df111:/# pip install -e .[devel]
> > root@b99b297df111:/airflow# ./run_unit_tests.sh
> > + export AIRFLOW_HOME=/root/airflow
> > + AIRFLOW_HOME=/root/airflow
> > + export AIRFLOW__CORE__UNIT_TEST_MODE=True
> > + AIRFLOW__CORE__UNIT_TEST_MODE=True
> > + export AIRFLOW__TESTSECTION__TESTKEY=testvalue
> > + AIRFLOW__TESTSECTION__TESTKEY=testvalue
> > + export AIRFLOW_USE_NEW_IMPORTS=1
> > + AIRFLOW_USE_NEW_IMPORTS=1
> > +++ dirname ./run_unit_tests.sh
> > ++ cd .
> > ++ pwd
> > + DIR=/airflow
> > + export PYTHONPATH=:/airflow/tests/test_utils
> > + PYTHONPATH=:/airflow/tests/test_utils
> > + nose_args=
> > + which airflow
> > + echo 'Initializing the DB'
> > Initializing the DB
> > + airflow resetdb
> > + yes
> > Traceback (most recent call last):
> > File "/usr/local/bin/airflow", line 6, in <module>
> > exec(compile(open(__file__).read(), __file__, 'exec'))
> > File "/airflow/airflow/bin/airflow", line 21, in <module>
> > from airflow import configuration
> > File "/airflow/airflow/__init__.py", line 35, in <module>
> > from airflow import configuration as conf
> > File "/airflow/airflow/configuration.py", line 106, in <module>
> > DEFAULT_CONFIG = f.read()
> > File "/usr/local/lib/python3.5/encodings/ascii.py", line 26, in decode
> > return codecs.ascii_decode(input, self.errors)[0]
> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
> 21082:
> > ordinal not in range(128)
> > + '[' '' ']'
> > + '[' -z '' ']'
> > + nose_args='--with-coverage --cover-erase --cover-html
> > --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
> > --with-ignore-docstrings --rednose --with-timer -s -v
> > --logging-level=DEBUG '
> > + echo 'Starting the unit tests with the following nose arguments:
> > --with-coverage' --cover-erase --cover-html --cover-package=airflow
> > --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
> > --rednose --with-timer -s -v --logging-level=DEBUG
> > Starting the unit tests with the following nose arguments:
> --with-coverage
> > --cover-erase --cover-html --cover-package=airflow
> > --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
> > --rednose --with-timer -s -v --logging-level=DEBUG
> > + nosetests --with-coverage --cover-erase --cover-html
> > --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
> > --with-ignore-docstrings --rednose --with-timer -s -v
> --logging-level=DEBUG
> > nose.plugins.cover: ERROR: Coverage not available: unable to import
> > coverage module
> > Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
> > position 21082: ordinal not in range(128)) ... ERROR
> > Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
> > position 21082: ordinal not in range(128)) ... ERROR
> > ======================================================================
> > 1) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte
> 0xe2
> > in position 21082: ordinal not in range(128))
> > ----------------------------------------------------------------------
> > Traceback (most recent call last):
> > /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
> > runTest
> > raise self.exc_val.with_traceback(self.tb)
> > /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
> > loadTestsFromName
> > addr.filename, addr.module)
> > /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
> > importFromPath
> > return self.importFromDir(dir_path, fqname)
> > /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
> > importFromDir
> > mod = load_module(part_fqname, fh, filename, desc)
> > /usr/local/lib/python3.5/imp.py line 245 in load_module
> > return load_package(name, filename)
> > /usr/local/lib/python3.5/imp.py line 217 in load_package
> > return _load(spec)
> > <frozen importlib._bootstrap> line 693 in _load
> >
> > <frozen importlib._bootstrap> line 673 in _load_unlocked
> >
> > <frozen importlib._bootstrap_external> line 697 in exec_module
> >
> > <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
> >
> > airflow/__init__.py line 35 in <module>
> > from airflow import configuration as conf
> > airflow/configuration.py line 106 in <module>
> > DEFAULT_CONFIG = f.read()
> > /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
> > return codecs.ascii_decode(input, self.errors)[0]
> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
> > 21082: ordinal not in range(128)
> > ======================================================================
> > 2) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte
> 0xe2
> > in position 21082: ordinal not in range(128))
> > ----------------------------------------------------------------------
> > Traceback (most recent call last):
> > /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
> > runTest
> > raise self.exc_val.with_traceback(self.tb)
> > /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
> > loadTestsFromName
> > addr.filename, addr.module)
> > /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
> > importFromPath
> > return self.importFromDir(dir_path, fqname)
> > /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
> > importFromDir
> > mod = load_module(part_fqname, fh, filename, desc)
> > /usr/local/lib/python3.5/imp.py line 245 in load_module
> > return load_package(name, filename)
> > /usr/local/lib/python3.5/imp.py line 217 in load_package
> > return _load(spec)
> > <frozen importlib._bootstrap> line 693 in _load
> >
> > <frozen importlib._bootstrap> line 673 in _load_unlocked
> >
> > <frozen importlib._bootstrap_external> line 697 in exec_module
> >
> > <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
> >
> > tests/__init__.py line 25 in <module>
> > from .configuration import *
> > tests/configuration.py line 28 in <module>
> > from airflow import configuration
> > airflow/__init__.py line 35 in <module>
> > from airflow import configuration as conf
> > airflow/configuration.py line 106 in <module>
> > DEFAULT_CONFIG = f.read()
> > /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
> > return codecs.ascii_decode(input, self.errors)[0]
> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
> > 21082: ordinal not in range(128)
> >
> > [error] 100.00% nose.failure.Failure.runTest: 0.0004s
> > ------------------------------------------------------------
> -----------------
> > 2 tests run in 0.060 seconds.
> > 2 errors (0 tests passed)
> >
> >
> > --
> > Carl Johan Gustavsson
> >
> > On 11 July 2018 at 23:24:01, Bolke de Bruin (bdbruin@gmail.com <mailto:
> bdbruin@gmail.com>) wrote:
> >
> > Hi Carl,
> >
> > That is not a real check, ie. Does Airflow have the same issue clean
> > install on 3.5? Travis’ tests run on 3.5.
> >
> > B.
> >
> > Verstuurd vanaf mijn iPad
> >
> > > Op 10 jul. 2018 om 15:10 heeft Carl Johan Gustavsson <
> > carl.j.gustavsson@gmail.com <ma...@gmail.com>> het
> volgende geschreven:
> > >
> > > Hi Bolke,
> > >
> > >
> > > I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all but 3.7
> > breaks
> > >
> > > Quick repro:
> > >
> > > ➜ ~ pyenv local 3.5.5
> > > ➜ ~ locale
> > > LANG=
> > > LC_COLLATE="C"
> > > LC_CTYPE="C"
> > > LC_MESSAGES="C"
> > > LC_MONETARY="C"
> > > LC_NUMERIC="C"
> > > LC_TIME="C"
> > > LC_ALL=
> > > ➜ ~ cat testweird.txt
> > > ’
> > > ➜ ~ python
> > > Python 3.5.5 (default, Jul 7 2018, 17:00:56)
> > > [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
> > > Type "help", "copyright", "credits" or "license" for more information.
> > > >>> open('testweird.txt').read()
> > > Traceback (most recent call last):
> > > File "<stdin>", line 1, in <module>
> > > File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/
> ascii.py",
> > line 26, in decode
> > > return codecs.ascii_decode(input, self.errors)[0]
> > > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0:
> > ordinal not in range(128)
> > > >>>
> > >
> > >
> > > Maybe not a blocking change but it is a breaking change from 1.9 I
> guess.
> > >
> > >
> > > / Carl Johan
> > >
> > >
> > >> On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbruin@gmail.com
> <ma...@gmail.com>) wrote:
> > >>
> > >> Hi Carl,
> > >>
> > >> Did you this on python 3.5 as well? 3.6 is not an officially supported
> > (yet). As a workaround is available I won’t consider this blocking btw.
> > >>
> > >> Bolke
> > >>
> > >> Verstuurd vanaf mijn iPad
> > >>
> > >> Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson <
> > carl.j.gustavsson@gmail.com <ma...@gmail.com>> het
> volgende geschreven:
> > >>
> > >>> Hi,
> > >>>
> > >>> First of all, thank you for all the work with the release management.
> > >>>
> > >>> I ran in to a weird issue testing the RC1, running under Python
> 3.6.0 /
> > Ubuntu 14.04.5, upgrading from a master build from February.
> > >>>
> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback
> > (most recent call last):
> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> > "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in <module>
> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from
> airflow
> > import configuration
> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> > "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-
> packages/airflow/__init__.py",
> > line 35, in <module>
> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from
> airflow
> > import configuration as conf
> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> > "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/
> configuration.py",
> > line 106, in <module>
> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
> > DEFAULT_CONFIG = f.read()
> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> > "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py",
> line
> > 26, in decode
> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return
> > codecs.ascii_decode(input, self.errors)[0]
> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
> 20770:
> > ordinal not in range(128)
> > >>>
> > >>> Removing the ’ on
> > https://github.com/apache/incubator-airflow/blob/master/
> airflow/config_templates/default_airflow.cfg#L613 <
> https://github.com/apache/incubator-airflow/blob/master/
> airflow/config_templates/default_airflow.cfg#L613>
> > solved the issue for me, and digging a bit deeper it seems Airflow now
> > requires setting LC_ALL=en_US.UTF-8 in the environment or similar to
> force
> > Python to read the file as utf-8 and not ascii. (I think this was changed
> > in to default to utf-8 in Python 3.7).
> > >>>
> > >>> I see 3 solutions for this
> > >>> 1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or
> > similar.
> > >>> 2. Change the default config file to not contain non-ascii
> characters.
> > >>> 3. Always read the file as unicode regardless of the LC_ALL
> > environment, by the encoding='utf-8’ parameter to open().
> > >>>
> > >>> I think 3 is the best solution, and I can prepare a PR for that if
> > necessary .
> > >>>
> > >>> I guess this counts as -1 (non-binding)
> > >>>
> > >>> All the best
> > >>>
> > >>> Carl Johan
> > >>>
> > >>>
> > >>>
> > >>>> On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbruin@gmail.com
> <ma...@gmail.com>) wrote:
> > >>>>
> > >>>> Hey all,
> > >>>>
> > >>>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the
> > release,
> > >>>> which will last for 72 hours. Consider this my (binding) +1.
> > >>>>
> > >>>> Airflow 1.10.0 RC 1 is available at:
> > >>>>
> > >>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/
> <https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/> <
> > https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <
> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>>
> > >>>>
> > >>>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source
> release
> > that
> > >>>> comes with INSTALL instructions.
> > >>>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python
> > "sdist"
> > >>>> release.
> > >>>>
> > >>>> Public keys are available at:
> > >>>>
> > >>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
> https://dist.apache.org/repos/dist/release/incubator/airflow/> <
> > https://dist.apache.org/repos/dist/release/incubator/airflow/ <
> https://dist.apache.org/repos/dist/release/incubator/airflow/>>
> > >>>>
> > >>>> The amount of JIRAs fixed is over 700. Please have a look at the
> > changelog.
> > >>>>
> > >>>> Please note that the version number excludes the `rcX` string as
> well
> > >>>> as the "+incubating" string, so it's now simply 1.10.0. This will
> > allow us
> > >>>> to rename the artifact without modifying the artifact checksums when
> > we
> > >>>> actually release.
> > >>>>
> > >>>>
> > >>>> Cheers,
> > >>>> Bolke
>
>
Re: [VOTE] Airflow 1.10.0rc1
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
I've installed Airflow, but using a clean install and the sequential
executor, the tasks are not being picked up.
This is easy to replicate:
docker run -t -i --rm -p 8080:8080 python:3.5 bash
pip install
https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/apache-airflow-1.10.0rc1+incubating-bin.tar.gz
airflow initdb
airflow webserver
I've enabled the http example dag and kicked of a dag, but it isn't being
picked up by the SequentialScheduler.
- Fokko
2018-07-13 11:51 GMT+02:00 Bolke de Bruin <bd...@gmail.com>:
> Hi Sid,
>
> Do you have a JIRA and a PR to address it? I can then consider it for RC2
>
> B.
>
> > On 12 Jul 2018, at 04:50, Sid Anand <sa...@apache.org> wrote:
> >
> > FYI!
> > I just installed the release candidate. The first thing I noticed is a
> missing tool tip for the Null State in the Recent Tasks column on the
> landing page. Since the null globe is new to this UI, users will likely
> hover over it to inquire what it means... and will be left wanting. Of
> course, they could click on the globe, which will take them to
> http://localhost:8080/admin/taskinstance/?flt1_dag_id_
> equals=example_bash_operator&flt2_state_equals=null <
> http://localhost:8080/admin/taskinstance/?flt1_dag_id_
> equals=example_bash_operator&flt2_state_equals=null>, which will always
> show an empty list, leaving them a bit more confused.
> >
> > -s
> >
> > On Wed, Jul 11, 2018 at 3:13 PM Carl Johan Gustavsson <
> carl.johan@tictail.com.invalid> wrote:
> > Hi Bolke,
> >
> > (Switching email to avoid moderation on my emails.)
> >
> > The normal Airflow test suite does not fail as it uses a LC_ALL set to
> > utf-8.
> >
> > I think it is a proper test though, it is a minimal reproducible version
> of
> > the code that fails. And the only difference in behaviour is at 3.7 which
> > we don’t support anyway so I’m fairly sure it is broken for all supported
> > Python 3 versions.
> >
> > I now tried running the tests in docker using 3.5 with the LC_ALL/LANG
> > unset and I see the same failure.
> >
> > I don’t think this is a big thing though and we could release it without
> > the fix I made. I think most people run it with a sane LC_ALL, but
> > apparently we didn’t.
> > Here’s the log for the test:
> >
> > > docker run -t -i -v `pwd`:/airflow/ python:3.5 bash
> > root@b99b297df111:/# locale
> > LANG=C.UTF-8
> > LANGUAGE=
> > LC_CTYPE="C.UTF-8"
> > LC_NUMERIC="C.UTF-8"
> > LC_TIME="C.UTF-8"
> > LC_COLLATE="C.UTF-8"
> > LC_MONETARY="C.UTF-8"
> > LC_MESSAGES="C.UTF-8"
> > LC_PAPER="C.UTF-8"
> > LC_NAME="C.UTF-8"
> > LC_ADDRESS="C.UTF-8"
> > LC_TELEPHONE="C.UTF-8"
> > LC_MEASUREMENT="C.UTF-8"
> > LC_IDENTIFICATION="C.UTF-8"
> > LC_ALL=
> > > unset LANG
> > root@b99b297df111:/# locale
> > LANG=
> > LANGUAGE=
> > LC_CTYPE="POSIX"
> > LC_NUMERIC="POSIX"
> > LC_TIME="POSIX"
> > LC_COLLATE="POSIX"
> > LC_MONETARY="POSIX"
> > LC_MESSAGES="POSIX"
> > LC_PAPER="POSIX"
> > LC_NAME="POSIX"
> > LC_ADDRESS="POSIX"
> > LC_TELEPHONE="POSIX"
> > LC_MEASUREMENT="POSIX"
> > LC_IDENTIFICATION="POSIX"
> > LC_ALL=
> > root@b99b297df111:/# pip install -e .[devel]
> > root@b99b297df111:/airflow# ./run_unit_tests.sh
> > + export AIRFLOW_HOME=/root/airflow
> > + AIRFLOW_HOME=/root/airflow
> > + export AIRFLOW__CORE__UNIT_TEST_MODE=True
> > + AIRFLOW__CORE__UNIT_TEST_MODE=True
> > + export AIRFLOW__TESTSECTION__TESTKEY=testvalue
> > + AIRFLOW__TESTSECTION__TESTKEY=testvalue
> > + export AIRFLOW_USE_NEW_IMPORTS=1
> > + AIRFLOW_USE_NEW_IMPORTS=1
> > +++ dirname ./run_unit_tests.sh
> > ++ cd .
> > ++ pwd
> > + DIR=/airflow
> > + export PYTHONPATH=:/airflow/tests/test_utils
> > + PYTHONPATH=:/airflow/tests/test_utils
> > + nose_args=
> > + which airflow
> > + echo 'Initializing the DB'
> > Initializing the DB
> > + airflow resetdb
> > + yes
> > Traceback (most recent call last):
> > File "/usr/local/bin/airflow", line 6, in <module>
> > exec(compile(open(__file__).read(), __file__, 'exec'))
> > File "/airflow/airflow/bin/airflow", line 21, in <module>
> > from airflow import configuration
> > File "/airflow/airflow/__init__.py", line 35, in <module>
> > from airflow import configuration as conf
> > File "/airflow/airflow/configuration.py", line 106, in <module>
> > DEFAULT_CONFIG = f.read()
> > File "/usr/local/lib/python3.5/encodings/ascii.py", line 26, in decode
> > return codecs.ascii_decode(input, self.errors)[0]
> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
> 21082:
> > ordinal not in range(128)
> > + '[' '' ']'
> > + '[' -z '' ']'
> > + nose_args='--with-coverage --cover-erase --cover-html
> > --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
> > --with-ignore-docstrings --rednose --with-timer -s -v
> > --logging-level=DEBUG '
> > + echo 'Starting the unit tests with the following nose arguments:
> > --with-coverage' --cover-erase --cover-html --cover-package=airflow
> > --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
> > --rednose --with-timer -s -v --logging-level=DEBUG
> > Starting the unit tests with the following nose arguments:
> --with-coverage
> > --cover-erase --cover-html --cover-package=airflow
> > --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
> > --rednose --with-timer -s -v --logging-level=DEBUG
> > + nosetests --with-coverage --cover-erase --cover-html
> > --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
> > --with-ignore-docstrings --rednose --with-timer -s -v
> --logging-level=DEBUG
> > nose.plugins.cover: ERROR: Coverage not available: unable to import
> > coverage module
> > Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
> > position 21082: ordinal not in range(128)) ... ERROR
> > Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
> > position 21082: ordinal not in range(128)) ... ERROR
> > ======================================================================
> > 1) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte
> 0xe2
> > in position 21082: ordinal not in range(128))
> > ----------------------------------------------------------------------
> > Traceback (most recent call last):
> > /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
> > runTest
> > raise self.exc_val.with_traceback(self.tb)
> > /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
> > loadTestsFromName
> > addr.filename, addr.module)
> > /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
> > importFromPath
> > return self.importFromDir(dir_path, fqname)
> > /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
> > importFromDir
> > mod = load_module(part_fqname, fh, filename, desc)
> > /usr/local/lib/python3.5/imp.py line 245 in load_module
> > return load_package(name, filename)
> > /usr/local/lib/python3.5/imp.py line 217 in load_package
> > return _load(spec)
> > <frozen importlib._bootstrap> line 693 in _load
> >
> > <frozen importlib._bootstrap> line 673 in _load_unlocked
> >
> > <frozen importlib._bootstrap_external> line 697 in exec_module
> >
> > <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
> >
> > airflow/__init__.py line 35 in <module>
> > from airflow import configuration as conf
> > airflow/configuration.py line 106 in <module>
> > DEFAULT_CONFIG = f.read()
> > /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
> > return codecs.ascii_decode(input, self.errors)[0]
> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
> > 21082: ordinal not in range(128)
> > ======================================================================
> > 2) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte
> 0xe2
> > in position 21082: ordinal not in range(128))
> > ----------------------------------------------------------------------
> > Traceback (most recent call last):
> > /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
> > runTest
> > raise self.exc_val.with_traceback(self.tb)
> > /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
> > loadTestsFromName
> > addr.filename, addr.module)
> > /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
> > importFromPath
> > return self.importFromDir(dir_path, fqname)
> > /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
> > importFromDir
> > mod = load_module(part_fqname, fh, filename, desc)
> > /usr/local/lib/python3.5/imp.py line 245 in load_module
> > return load_package(name, filename)
> > /usr/local/lib/python3.5/imp.py line 217 in load_package
> > return _load(spec)
> > <frozen importlib._bootstrap> line 693 in _load
> >
> > <frozen importlib._bootstrap> line 673 in _load_unlocked
> >
> > <frozen importlib._bootstrap_external> line 697 in exec_module
> >
> > <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
> >
> > tests/__init__.py line 25 in <module>
> > from .configuration import *
> > tests/configuration.py line 28 in <module>
> > from airflow import configuration
> > airflow/__init__.py line 35 in <module>
> > from airflow import configuration as conf
> > airflow/configuration.py line 106 in <module>
> > DEFAULT_CONFIG = f.read()
> > /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
> > return codecs.ascii_decode(input, self.errors)[0]
> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
> > 21082: ordinal not in range(128)
> >
> > [error] 100.00% nose.failure.Failure.runTest: 0.0004s
> > ------------------------------------------------------------
> -----------------
> > 2 tests run in 0.060 seconds.
> > 2 errors (0 tests passed)
> >
> >
> > --
> > Carl Johan Gustavsson
> >
> > On 11 July 2018 at 23:24:01, Bolke de Bruin (bdbruin@gmail.com <mailto:
> bdbruin@gmail.com>) wrote:
> >
> > Hi Carl,
> >
> > That is not a real check, ie. Does Airflow have the same issue clean
> > install on 3.5? Travis’ tests run on 3.5.
> >
> > B.
> >
> > Verstuurd vanaf mijn iPad
> >
> > > Op 10 jul. 2018 om 15:10 heeft Carl Johan Gustavsson <
> > carl.j.gustavsson@gmail.com <ma...@gmail.com>> het
> volgende geschreven:
> > >
> > > Hi Bolke,
> > >
> > >
> > > I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all but 3.7
> > breaks
> > >
> > > Quick repro:
> > >
> > > ➜ ~ pyenv local 3.5.5
> > > ➜ ~ locale
> > > LANG=
> > > LC_COLLATE="C"
> > > LC_CTYPE="C"
> > > LC_MESSAGES="C"
> > > LC_MONETARY="C"
> > > LC_NUMERIC="C"
> > > LC_TIME="C"
> > > LC_ALL=
> > > ➜ ~ cat testweird.txt
> > > ’
> > > ➜ ~ python
> > > Python 3.5.5 (default, Jul 7 2018, 17:00:56)
> > > [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
> > > Type "help", "copyright", "credits" or "license" for more information.
> > > >>> open('testweird.txt').read()
> > > Traceback (most recent call last):
> > > File "<stdin>", line 1, in <module>
> > > File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/
> ascii.py",
> > line 26, in decode
> > > return codecs.ascii_decode(input, self.errors)[0]
> > > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0:
> > ordinal not in range(128)
> > > >>>
> > >
> > >
> > > Maybe not a blocking change but it is a breaking change from 1.9 I
> guess.
> > >
> > >
> > > / Carl Johan
> > >
> > >
> > >> On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbruin@gmail.com
> <ma...@gmail.com>) wrote:
> > >>
> > >> Hi Carl,
> > >>
> > >> Did you this on python 3.5 as well? 3.6 is not an officially supported
> > (yet). As a workaround is available I won’t consider this blocking btw.
> > >>
> > >> Bolke
> > >>
> > >> Verstuurd vanaf mijn iPad
> > >>
> > >> Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson <
> > carl.j.gustavsson@gmail.com <ma...@gmail.com>> het
> volgende geschreven:
> > >>
> > >>> Hi,
> > >>>
> > >>> First of all, thank you for all the work with the release management.
> > >>>
> > >>> I ran in to a weird issue testing the RC1, running under Python
> 3.6.0 /
> > Ubuntu 14.04.5, upgrading from a master build from February.
> > >>>
> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback
> > (most recent call last):
> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> > "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in <module>
> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from
> airflow
> > import configuration
> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> > "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-
> packages/airflow/__init__.py",
> > line 35, in <module>
> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from
> airflow
> > import configuration as conf
> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> > "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/
> configuration.py",
> > line 106, in <module>
> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
> > DEFAULT_CONFIG = f.read()
> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> > "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py",
> line
> > 26, in decode
> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return
> > codecs.ascii_decode(input, self.errors)[0]
> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
> 20770:
> > ordinal not in range(128)
> > >>>
> > >>> Removing the ’ on
> > https://github.com/apache/incubator-airflow/blob/master/
> airflow/config_templates/default_airflow.cfg#L613 <
> https://github.com/apache/incubator-airflow/blob/master/
> airflow/config_templates/default_airflow.cfg#L613>
> > solved the issue for me, and digging a bit deeper it seems Airflow now
> > requires setting LC_ALL=en_US.UTF-8 in the environment or similar to
> force
> > Python to read the file as utf-8 and not ascii. (I think this was changed
> > in to default to utf-8 in Python 3.7).
> > >>>
> > >>> I see 3 solutions for this
> > >>> 1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or
> > similar.
> > >>> 2. Change the default config file to not contain non-ascii
> characters.
> > >>> 3. Always read the file as unicode regardless of the LC_ALL
> > environment, by the encoding='utf-8’ parameter to open().
> > >>>
> > >>> I think 3 is the best solution, and I can prepare a PR for that if
> > necessary .
> > >>>
> > >>> I guess this counts as -1 (non-binding)
> > >>>
> > >>> All the best
> > >>>
> > >>> Carl Johan
> > >>>
> > >>>
> > >>>
> > >>>> On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbruin@gmail.com
> <ma...@gmail.com>) wrote:
> > >>>>
> > >>>> Hey all,
> > >>>>
> > >>>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the
> > release,
> > >>>> which will last for 72 hours. Consider this my (binding) +1.
> > >>>>
> > >>>> Airflow 1.10.0 RC 1 is available at:
> > >>>>
> > >>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/
> <https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/> <
> > https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <
> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>>
> > >>>>
> > >>>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source
> release
> > that
> > >>>> comes with INSTALL instructions.
> > >>>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python
> > "sdist"
> > >>>> release.
> > >>>>
> > >>>> Public keys are available at:
> > >>>>
> > >>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
> https://dist.apache.org/repos/dist/release/incubator/airflow/> <
> > https://dist.apache.org/repos/dist/release/incubator/airflow/ <
> https://dist.apache.org/repos/dist/release/incubator/airflow/>>
> > >>>>
> > >>>> The amount of JIRAs fixed is over 700. Please have a look at the
> > changelog.
> > >>>>
> > >>>> Please note that the version number excludes the `rcX` string as
> well
> > >>>> as the "+incubating" string, so it's now simply 1.10.0. This will
> > allow us
> > >>>> to rename the artifact without modifying the artifact checksums when
> > we
> > >>>> actually release.
> > >>>>
> > >>>>
> > >>>> Cheers,
> > >>>> Bolke
>
>
Re: [VOTE] Airflow 1.10.0rc1
Posted by Bolke de Bruin <bd...@gmail.com>.
Hi Sid,
Do you have a JIRA and a PR to address it? I can then consider it for RC2
B.
> On 12 Jul 2018, at 04:50, Sid Anand <sa...@apache.org> wrote:
>
> FYI!
> I just installed the release candidate. The first thing I noticed is a missing tool tip for the Null State in the Recent Tasks column on the landing page. Since the null globe is new to this UI, users will likely hover over it to inquire what it means... and will be left wanting. Of course, they could click on the globe, which will take them to http://localhost:8080/admin/taskinstance/?flt1_dag_id_equals=example_bash_operator&flt2_state_equals=null <http://localhost:8080/admin/taskinstance/?flt1_dag_id_equals=example_bash_operator&flt2_state_equals=null>, which will always show an empty list, leaving them a bit more confused.
>
> -s
>
> On Wed, Jul 11, 2018 at 3:13 PM Carl Johan Gustavsson <ca...@tictail.com.invalid> wrote:
> Hi Bolke,
>
> (Switching email to avoid moderation on my emails.)
>
> The normal Airflow test suite does not fail as it uses a LC_ALL set to
> utf-8.
>
> I think it is a proper test though, it is a minimal reproducible version of
> the code that fails. And the only difference in behaviour is at 3.7 which
> we don’t support anyway so I’m fairly sure it is broken for all supported
> Python 3 versions.
>
> I now tried running the tests in docker using 3.5 with the LC_ALL/LANG
> unset and I see the same failure.
>
> I don’t think this is a big thing though and we could release it without
> the fix I made. I think most people run it with a sane LC_ALL, but
> apparently we didn’t.
> Here’s the log for the test:
>
> > docker run -t -i -v `pwd`:/airflow/ python:3.5 bash
> root@b99b297df111:/# locale
> LANG=C.UTF-8
> LANGUAGE=
> LC_CTYPE="C.UTF-8"
> LC_NUMERIC="C.UTF-8"
> LC_TIME="C.UTF-8"
> LC_COLLATE="C.UTF-8"
> LC_MONETARY="C.UTF-8"
> LC_MESSAGES="C.UTF-8"
> LC_PAPER="C.UTF-8"
> LC_NAME="C.UTF-8"
> LC_ADDRESS="C.UTF-8"
> LC_TELEPHONE="C.UTF-8"
> LC_MEASUREMENT="C.UTF-8"
> LC_IDENTIFICATION="C.UTF-8"
> LC_ALL=
> > unset LANG
> root@b99b297df111:/# locale
> LANG=
> LANGUAGE=
> LC_CTYPE="POSIX"
> LC_NUMERIC="POSIX"
> LC_TIME="POSIX"
> LC_COLLATE="POSIX"
> LC_MONETARY="POSIX"
> LC_MESSAGES="POSIX"
> LC_PAPER="POSIX"
> LC_NAME="POSIX"
> LC_ADDRESS="POSIX"
> LC_TELEPHONE="POSIX"
> LC_MEASUREMENT="POSIX"
> LC_IDENTIFICATION="POSIX"
> LC_ALL=
> root@b99b297df111:/# pip install -e .[devel]
> root@b99b297df111:/airflow# ./run_unit_tests.sh
> + export AIRFLOW_HOME=/root/airflow
> + AIRFLOW_HOME=/root/airflow
> + export AIRFLOW__CORE__UNIT_TEST_MODE=True
> + AIRFLOW__CORE__UNIT_TEST_MODE=True
> + export AIRFLOW__TESTSECTION__TESTKEY=testvalue
> + AIRFLOW__TESTSECTION__TESTKEY=testvalue
> + export AIRFLOW_USE_NEW_IMPORTS=1
> + AIRFLOW_USE_NEW_IMPORTS=1
> +++ dirname ./run_unit_tests.sh
> ++ cd .
> ++ pwd
> + DIR=/airflow
> + export PYTHONPATH=:/airflow/tests/test_utils
> + PYTHONPATH=:/airflow/tests/test_utils
> + nose_args=
> + which airflow
> + echo 'Initializing the DB'
> Initializing the DB
> + airflow resetdb
> + yes
> Traceback (most recent call last):
> File "/usr/local/bin/airflow", line 6, in <module>
> exec(compile(open(__file__).read(), __file__, 'exec'))
> File "/airflow/airflow/bin/airflow", line 21, in <module>
> from airflow import configuration
> File "/airflow/airflow/__init__.py", line 35, in <module>
> from airflow import configuration as conf
> File "/airflow/airflow/configuration.py", line 106, in <module>
> DEFAULT_CONFIG = f.read()
> File "/usr/local/lib/python3.5/encodings/ascii.py", line 26, in decode
> return codecs.ascii_decode(input, self.errors)[0]
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 21082:
> ordinal not in range(128)
> + '[' '' ']'
> + '[' -z '' ']'
> + nose_args='--with-coverage --cover-erase --cover-html
> --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
> --with-ignore-docstrings --rednose --with-timer -s -v
> --logging-level=DEBUG '
> + echo 'Starting the unit tests with the following nose arguments:
> --with-coverage' --cover-erase --cover-html --cover-package=airflow
> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
> --rednose --with-timer -s -v --logging-level=DEBUG
> Starting the unit tests with the following nose arguments: --with-coverage
> --cover-erase --cover-html --cover-package=airflow
> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
> --rednose --with-timer -s -v --logging-level=DEBUG
> + nosetests --with-coverage --cover-erase --cover-html
> --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
> --with-ignore-docstrings --rednose --with-timer -s -v --logging-level=DEBUG
> nose.plugins.cover: ERROR: Coverage not available: unable to import
> coverage module
> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
> position 21082: ordinal not in range(128)) ... ERROR
> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
> position 21082: ordinal not in range(128)) ... ERROR
> ======================================================================
> 1) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
> in position 21082: ordinal not in range(128))
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
> runTest
> raise self.exc_val.with_traceback(self.tb)
> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
> loadTestsFromName
> addr.filename, addr.module)
> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
> importFromPath
> return self.importFromDir(dir_path, fqname)
> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
> importFromDir
> mod = load_module(part_fqname, fh, filename, desc)
> /usr/local/lib/python3.5/imp.py line 245 in load_module
> return load_package(name, filename)
> /usr/local/lib/python3.5/imp.py line 217 in load_package
> return _load(spec)
> <frozen importlib._bootstrap> line 693 in _load
>
> <frozen importlib._bootstrap> line 673 in _load_unlocked
>
> <frozen importlib._bootstrap_external> line 697 in exec_module
>
> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>
> airflow/__init__.py line 35 in <module>
> from airflow import configuration as conf
> airflow/configuration.py line 106 in <module>
> DEFAULT_CONFIG = f.read()
> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
> return codecs.ascii_decode(input, self.errors)[0]
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
> 21082: ordinal not in range(128)
> ======================================================================
> 2) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
> in position 21082: ordinal not in range(128))
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
> runTest
> raise self.exc_val.with_traceback(self.tb)
> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
> loadTestsFromName
> addr.filename, addr.module)
> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
> importFromPath
> return self.importFromDir(dir_path, fqname)
> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
> importFromDir
> mod = load_module(part_fqname, fh, filename, desc)
> /usr/local/lib/python3.5/imp.py line 245 in load_module
> return load_package(name, filename)
> /usr/local/lib/python3.5/imp.py line 217 in load_package
> return _load(spec)
> <frozen importlib._bootstrap> line 693 in _load
>
> <frozen importlib._bootstrap> line 673 in _load_unlocked
>
> <frozen importlib._bootstrap_external> line 697 in exec_module
>
> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>
> tests/__init__.py line 25 in <module>
> from .configuration import *
> tests/configuration.py line 28 in <module>
> from airflow import configuration
> airflow/__init__.py line 35 in <module>
> from airflow import configuration as conf
> airflow/configuration.py line 106 in <module>
> DEFAULT_CONFIG = f.read()
> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
> return codecs.ascii_decode(input, self.errors)[0]
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
> 21082: ordinal not in range(128)
>
> [error] 100.00% nose.failure.Failure.runTest: 0.0004s
> -----------------------------------------------------------------------------
> 2 tests run in 0.060 seconds.
> 2 errors (0 tests passed)
>
>
> --
> Carl Johan Gustavsson
>
> On 11 July 2018 at 23:24:01, Bolke de Bruin (bdbruin@gmail.com <ma...@gmail.com>) wrote:
>
> Hi Carl,
>
> That is not a real check, ie. Does Airflow have the same issue clean
> install on 3.5? Travis’ tests run on 3.5.
>
> B.
>
> Verstuurd vanaf mijn iPad
>
> > Op 10 jul. 2018 om 15:10 heeft Carl Johan Gustavsson <
> carl.j.gustavsson@gmail.com <ma...@gmail.com>> het volgende geschreven:
> >
> > Hi Bolke,
> >
> >
> > I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all but 3.7
> breaks
> >
> > Quick repro:
> >
> > ➜ ~ pyenv local 3.5.5
> > ➜ ~ locale
> > LANG=
> > LC_COLLATE="C"
> > LC_CTYPE="C"
> > LC_MESSAGES="C"
> > LC_MONETARY="C"
> > LC_NUMERIC="C"
> > LC_TIME="C"
> > LC_ALL=
> > ➜ ~ cat testweird.txt
> > ’
> > ➜ ~ python
> > Python 3.5.5 (default, Jul 7 2018, 17:00:56)
> > [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
> > Type "help", "copyright", "credits" or "license" for more information.
> > >>> open('testweird.txt').read()
> > Traceback (most recent call last):
> > File "<stdin>", line 1, in <module>
> > File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/ascii.py",
> line 26, in decode
> > return codecs.ascii_decode(input, self.errors)[0]
> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0:
> ordinal not in range(128)
> > >>>
> >
> >
> > Maybe not a blocking change but it is a breaking change from 1.9 I guess.
> >
> >
> > / Carl Johan
> >
> >
> >> On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbruin@gmail.com <ma...@gmail.com>) wrote:
> >>
> >> Hi Carl,
> >>
> >> Did you this on python 3.5 as well? 3.6 is not an officially supported
> (yet). As a workaround is available I won’t consider this blocking btw.
> >>
> >> Bolke
> >>
> >> Verstuurd vanaf mijn iPad
> >>
> >> Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson <
> carl.j.gustavsson@gmail.com <ma...@gmail.com>> het volgende geschreven:
> >>
> >>> Hi,
> >>>
> >>> First of all, thank you for all the work with the release management.
> >>>
> >>> I ran in to a weird issue testing the RC1, running under Python 3.6.0 /
> Ubuntu 14.04.5, upgrading from a master build from February.
> >>>
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback
> (most recent call last):
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in <module>
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
> import configuration
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/__init__.py",
> line 35, in <module>
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
> import configuration as conf
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/configuration.py",
> line 106, in <module>
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
> DEFAULT_CONFIG = f.read()
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py", line
> 26, in decode
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return
> codecs.ascii_decode(input, self.errors)[0]
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 20770:
> ordinal not in range(128)
> >>>
> >>> Removing the ’ on
> https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L613 <https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L613>
> solved the issue for me, and digging a bit deeper it seems Airflow now
> requires setting LC_ALL=en_US.UTF-8 in the environment or similar to force
> Python to read the file as utf-8 and not ascii. (I think this was changed
> in to default to utf-8 in Python 3.7).
> >>>
> >>> I see 3 solutions for this
> >>> 1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or
> similar.
> >>> 2. Change the default config file to not contain non-ascii characters.
> >>> 3. Always read the file as unicode regardless of the LC_ALL
> environment, by the encoding='utf-8’ parameter to open().
> >>>
> >>> I think 3 is the best solution, and I can prepare a PR for that if
> necessary .
> >>>
> >>> I guess this counts as -1 (non-binding)
> >>>
> >>> All the best
> >>>
> >>> Carl Johan
> >>>
> >>>
> >>>
> >>>> On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbruin@gmail.com <ma...@gmail.com>) wrote:
> >>>>
> >>>> Hey all,
> >>>>
> >>>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the
> release,
> >>>> which will last for 72 hours. Consider this my (binding) +1.
> >>>>
> >>>> Airflow 1.10.0 RC 1 is available at:
> >>>>
> >>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/> <
> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>>
> >>>>
> >>>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release
> that
> >>>> comes with INSTALL instructions.
> >>>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python
> "sdist"
> >>>> release.
> >>>>
> >>>> Public keys are available at:
> >>>>
> >>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <https://dist.apache.org/repos/dist/release/incubator/airflow/> <
> https://dist.apache.org/repos/dist/release/incubator/airflow/ <https://dist.apache.org/repos/dist/release/incubator/airflow/>>
> >>>>
> >>>> The amount of JIRAs fixed is over 700. Please have a look at the
> changelog.
> >>>>
> >>>> Please note that the version number excludes the `rcX` string as well
> >>>> as the "+incubating" string, so it's now simply 1.10.0. This will
> allow us
> >>>> to rename the artifact without modifying the artifact checksums when
> we
> >>>> actually release.
> >>>>
> >>>>
> >>>> Cheers,
> >>>> Bolke
Re: [VOTE] Airflow 1.10.0rc1
Posted by Bolke de Bruin <bd...@gmail.com>.
Hi Jakob,
As I’m preparing the RC2 I am also addressing the below issues. However
> * Nit: Last release Sebb called out "Copyright 2016 and onwards" in
> NOTICE as being imprecise. This language remains.
is equal to what is used in e.g. Apache Spark (https://github.com/apache/spark/blob/master/NOTICE <https://github.com/apache/spark/blob/master/NOTICE>)
I would rather like to keep it, because it reduces maintenance.
What is your opinion?
Cheers
Bolke
> On 12 Jul 2018, at 07:54, Jakob Homan <jg...@gmail.com> wrote:
>
> +1 (binding)
>
> * Sigs look good
> * Artifact has incubating in name
> * LICENSE/NOTICE/DISCLAIMER look good
> - nit: DISCLAIMER is not word wrapped to 80 chars as I've seen in
> other projects.
> * Nit: Last release Sebb called out "Copyright 2016 and onwards" in
> NOTICE as being imprecise. This language remains.
> * Spot check on license headers looks ok.
> * Nit: Last release there was a request for instructions on how to
> build the code. This isn't included.
>
> -Jakob
>
> On 11 July 2018 at 19:50, Sid Anand <sa...@apache.org> wrote:
>> FYI!
>> I just installed the release candidate. The first thing I noticed is a
>> missing tool tip for the Null State in the Recent Tasks column on the
>> landing page. Since the null globe is new to this UI, users will likely
>> hover over it to inquire what it means... and will be left wanting. Of
>> course, they could click on the globe, which will take them to
>> http://localhost:8080/admin/taskinstance/?flt1_dag_id_equals=example_bash_operator&flt2_state_equals=null,
>> which will always show an empty list, leaving them a bit more confused.
>>
>> -s
>>
>> On Wed, Jul 11, 2018 at 3:13 PM Carl Johan Gustavsson
>> <ca...@tictail.com.invalid> wrote:
>>
>>> Hi Bolke,
>>>
>>> (Switching email to avoid moderation on my emails.)
>>>
>>> The normal Airflow test suite does not fail as it uses a LC_ALL set to
>>> utf-8.
>>>
>>> I think it is a proper test though, it is a minimal reproducible version of
>>> the code that fails. And the only difference in behaviour is at 3.7 which
>>> we don’t support anyway so I’m fairly sure it is broken for all supported
>>> Python 3 versions.
>>>
>>> I now tried running the tests in docker using 3.5 with the LC_ALL/LANG
>>> unset and I see the same failure.
>>>
>>> I don’t think this is a big thing though and we could release it without
>>> the fix I made. I think most people run it with a sane LC_ALL, but
>>> apparently we didn’t.
>>> Here’s the log for the test:
>>>
>>>> docker run -t -i -v `pwd`:/airflow/ python:3.5 bash
>>> root@b99b297df111:/# locale
>>> LANG=C.UTF-8
>>> LANGUAGE=
>>> LC_CTYPE="C.UTF-8"
>>> LC_NUMERIC="C.UTF-8"
>>> LC_TIME="C.UTF-8"
>>> LC_COLLATE="C.UTF-8"
>>> LC_MONETARY="C.UTF-8"
>>> LC_MESSAGES="C.UTF-8"
>>> LC_PAPER="C.UTF-8"
>>> LC_NAME="C.UTF-8"
>>> LC_ADDRESS="C.UTF-8"
>>> LC_TELEPHONE="C.UTF-8"
>>> LC_MEASUREMENT="C.UTF-8"
>>> LC_IDENTIFICATION="C.UTF-8"
>>> LC_ALL=
>>>> unset LANG
>>> root@b99b297df111:/# locale
>>> LANG=
>>> LANGUAGE=
>>> LC_CTYPE="POSIX"
>>> LC_NUMERIC="POSIX"
>>> LC_TIME="POSIX"
>>> LC_COLLATE="POSIX"
>>> LC_MONETARY="POSIX"
>>> LC_MESSAGES="POSIX"
>>> LC_PAPER="POSIX"
>>> LC_NAME="POSIX"
>>> LC_ADDRESS="POSIX"
>>> LC_TELEPHONE="POSIX"
>>> LC_MEASUREMENT="POSIX"
>>> LC_IDENTIFICATION="POSIX"
>>> LC_ALL=
>>> root@b99b297df111:/# pip install -e .[devel]
>>> root@b99b297df111:/airflow# ./run_unit_tests.sh
>>> + export AIRFLOW_HOME=/root/airflow
>>> + AIRFLOW_HOME=/root/airflow
>>> + export AIRFLOW__CORE__UNIT_TEST_MODE=True
>>> + AIRFLOW__CORE__UNIT_TEST_MODE=True
>>> + export AIRFLOW__TESTSECTION__TESTKEY=testvalue
>>> + AIRFLOW__TESTSECTION__TESTKEY=testvalue
>>> + export AIRFLOW_USE_NEW_IMPORTS=1
>>> + AIRFLOW_USE_NEW_IMPORTS=1
>>> +++ dirname ./run_unit_tests.sh
>>> ++ cd .
>>> ++ pwd
>>> + DIR=/airflow
>>> + export PYTHONPATH=:/airflow/tests/test_utils
>>> + PYTHONPATH=:/airflow/tests/test_utils
>>> + nose_args=
>>> + which airflow
>>> + echo 'Initializing the DB'
>>> Initializing the DB
>>> + airflow resetdb
>>> + yes
>>> Traceback (most recent call last):
>>> File "/usr/local/bin/airflow", line 6, in <module>
>>> exec(compile(open(__file__).read(), __file__, 'exec'))
>>> File "/airflow/airflow/bin/airflow", line 21, in <module>
>>> from airflow import configuration
>>> File "/airflow/airflow/__init__.py", line 35, in <module>
>>> from airflow import configuration as conf
>>> File "/airflow/airflow/configuration.py", line 106, in <module>
>>> DEFAULT_CONFIG = f.read()
>>> File "/usr/local/lib/python3.5/encodings/ascii.py", line 26, in decode
>>> return codecs.ascii_decode(input, self.errors)[0]
>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 21082:
>>> ordinal not in range(128)
>>> + '[' '' ']'
>>> + '[' -z '' ']'
>>> + nose_args='--with-coverage --cover-erase --cover-html
>>> --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
>>> --with-ignore-docstrings --rednose --with-timer -s -v
>>> --logging-level=DEBUG '
>>> + echo 'Starting the unit tests with the following nose arguments:
>>> --with-coverage' --cover-erase --cover-html --cover-package=airflow
>>> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
>>> --rednose --with-timer -s -v --logging-level=DEBUG
>>> Starting the unit tests with the following nose arguments: --with-coverage
>>> --cover-erase --cover-html --cover-package=airflow
>>> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
>>> --rednose --with-timer -s -v --logging-level=DEBUG
>>> + nosetests --with-coverage --cover-erase --cover-html
>>> --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
>>> --with-ignore-docstrings --rednose --with-timer -s -v --logging-level=DEBUG
>>> nose.plugins.cover: ERROR: Coverage not available: unable to import
>>> coverage module
>>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>>> position 21082: ordinal not in range(128)) ... ERROR
>>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>>> position 21082: ordinal not in range(128)) ... ERROR
>>> ======================================================================
>>> 1) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
>>> in position 21082: ordinal not in range(128))
>>> ----------------------------------------------------------------------
>>> Traceback (most recent call last):
>>> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>>> runTest
>>> raise self.exc_val.with_traceback(self.tb)
>>> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>>> loadTestsFromName
>>> addr.filename, addr.module)
>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>>> importFromPath
>>> return self.importFromDir(dir_path, fqname)
>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>>> importFromDir
>>> mod = load_module(part_fqname, fh, filename, desc)
>>> /usr/local/lib/python3.5/imp.py line 245 in load_module
>>> return load_package(name, filename)
>>> /usr/local/lib/python3.5/imp.py line 217 in load_package
>>> return _load(spec)
>>> <frozen importlib._bootstrap> line 693 in _load
>>>
>>> <frozen importlib._bootstrap> line 673 in _load_unlocked
>>>
>>> <frozen importlib._bootstrap_external> line 697 in exec_module
>>>
>>> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>>>
>>> airflow/__init__.py line 35 in <module>
>>> from airflow import configuration as conf
>>> airflow/configuration.py line 106 in <module>
>>> DEFAULT_CONFIG = f.read()
>>> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>>> return codecs.ascii_decode(input, self.errors)[0]
>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>> 21082: ordinal not in range(128)
>>> ======================================================================
>>> 2) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
>>> in position 21082: ordinal not in range(128))
>>> ----------------------------------------------------------------------
>>> Traceback (most recent call last):
>>> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>>> runTest
>>> raise self.exc_val.with_traceback(self.tb)
>>> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>>> loadTestsFromName
>>> addr.filename, addr.module)
>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>>> importFromPath
>>> return self.importFromDir(dir_path, fqname)
>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>>> importFromDir
>>> mod = load_module(part_fqname, fh, filename, desc)
>>> /usr/local/lib/python3.5/imp.py line 245 in load_module
>>> return load_package(name, filename)
>>> /usr/local/lib/python3.5/imp.py line 217 in load_package
>>> return _load(spec)
>>> <frozen importlib._bootstrap> line 693 in _load
>>>
>>> <frozen importlib._bootstrap> line 673 in _load_unlocked
>>>
>>> <frozen importlib._bootstrap_external> line 697 in exec_module
>>>
>>> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>>>
>>> tests/__init__.py line 25 in <module>
>>> from .configuration import *
>>> tests/configuration.py line 28 in <module>
>>> from airflow import configuration
>>> airflow/__init__.py line 35 in <module>
>>> from airflow import configuration as conf
>>> airflow/configuration.py line 106 in <module>
>>> DEFAULT_CONFIG = f.read()
>>> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>>> return codecs.ascii_decode(input, self.errors)[0]
>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>> 21082: ordinal not in range(128)
>>>
>>> [error] 100.00% nose.failure.Failure.runTest: 0.0004s
>>>
>>> -----------------------------------------------------------------------------
>>> 2 tests run in 0.060 seconds.
>>> 2 errors (0 tests passed)
>>>
>>>
>>> --
>>> Carl Johan Gustavsson
>>>
>>> On 11 July 2018 at 23:24:01, Bolke de Bruin (bdbruin@gmail.com) wrote:
>>>
>>> Hi Carl,
>>>
>>> That is not a real check, ie. Does Airflow have the same issue clean
>>> install on 3.5? Travis’ tests run on 3.5.
>>>
>>> B.
>>>
>>> Verstuurd vanaf mijn iPad
>>>
>>>> Op 10 jul. 2018 om 15:10 heeft Carl Johan Gustavsson <
>>> carl.j.gustavsson@gmail.com> het volgende geschreven:
>>>>
>>>> Hi Bolke,
>>>>
>>>>
>>>> I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all but 3.7
>>> breaks
>>>>
>>>> Quick repro:
>>>>
>>>> ➜ ~ pyenv local 3.5.5
>>>> ➜ ~ locale
>>>> LANG=
>>>> LC_COLLATE="C"
>>>> LC_CTYPE="C"
>>>> LC_MESSAGES="C"
>>>> LC_MONETARY="C"
>>>> LC_NUMERIC="C"
>>>> LC_TIME="C"
>>>> LC_ALL=
>>>> ➜ ~ cat testweird.txt
>>>> ’
>>>> ➜ ~ python
>>>> Python 3.5.5 (default, Jul 7 2018, 17:00:56)
>>>> [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
>>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>>> open('testweird.txt').read()
>>>> Traceback (most recent call last):
>>>> File "<stdin>", line 1, in <module>
>>>> File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/ascii.py",
>>> line 26, in decode
>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0:
>>> ordinal not in range(128)
>>>>>>>
>>>>
>>>>
>>>> Maybe not a blocking change but it is a breaking change from 1.9 I guess.
>>>>
>>>>
>>>> / Carl Johan
>>>>
>>>>
>>>>> On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbruin@gmail.com) wrote:
>>>>>
>>>>> Hi Carl,
>>>>>
>>>>> Did you this on python 3.5 as well? 3.6 is not an officially supported
>>> (yet). As a workaround is available I won’t consider this blocking btw.
>>>>>
>>>>> Bolke
>>>>>
>>>>> Verstuurd vanaf mijn iPad
>>>>>
>>>>> Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson <
>>> carl.j.gustavsson@gmail.com> het volgende geschreven:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> First of all, thank you for all the work with the release management.
>>>>>>
>>>>>> I ran in to a weird issue testing the RC1, running under Python 3.6.0 /
>>> Ubuntu 14.04.5, upgrading from a master build from February.
>>>>>>
>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback
>>> (most recent call last):
>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>> "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in <module>
>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
>>> import configuration
>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>
>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/__init__.py",
>>> line 35, in <module>
>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
>>> import configuration as conf
>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>
>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/configuration.py",
>>> line 106, in <module>
>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>>> DEFAULT_CONFIG = f.read()
>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py", line
>>> 26, in decode
>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return
>>> codecs.ascii_decode(input, self.errors)[0]
>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 20770:
>>> ordinal not in range(128)
>>>>>>
>>>>>> Removing the ’ on
>>>
>>> https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L613
>>> solved the issue for me, and digging a bit deeper it seems Airflow now
>>> requires setting LC_ALL=en_US.UTF-8 in the environment or similar to force
>>> Python to read the file as utf-8 and not ascii. (I think this was changed
>>> in to default to utf-8 in Python 3.7).
>>>>>>
>>>>>> I see 3 solutions for this
>>>>>> 1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or
>>> similar.
>>>>>> 2. Change the default config file to not contain non-ascii characters.
>>>>>> 3. Always read the file as unicode regardless of the LC_ALL
>>> environment, by the encoding='utf-8’ parameter to open().
>>>>>>
>>>>>> I think 3 is the best solution, and I can prepare a PR for that if
>>> necessary .
>>>>>>
>>>>>> I guess this counts as -1 (non-binding)
>>>>>>
>>>>>> All the best
>>>>>>
>>>>>> Carl Johan
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbruin@gmail.com) wrote:
>>>>>>>
>>>>>>> Hey all,
>>>>>>>
>>>>>>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the
>>> release,
>>>>>>> which will last for 72 hours. Consider this my (binding) +1.
>>>>>>>
>>>>>>> Airflow 1.10.0 RC 1 is available at:
>>>>>>>
>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <
>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
>>>>>>>
>>>>>>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release
>>> that
>>>>>>> comes with INSTALL instructions.
>>>>>>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python
>>> "sdist"
>>>>>>> release.
>>>>>>>
>>>>>>> Public keys are available at:
>>>>>>>
>>>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>>> https://dist.apache.org/repos/dist/release/incubator/airflow/>
>>>>>>>
>>>>>>> The amount of JIRAs fixed is over 700. Please have a look at the
>>> changelog.
>>>>>>>
>>>>>>> Please note that the version number excludes the `rcX` string as well
>>>>>>> as the "+incubating" string, so it's now simply 1.10.0. This will
>>> allow us
>>>>>>> to rename the artifact without modifying the artifact checksums when
>>> we
>>>>>>> actually release.
>>>>>>>
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Bolke
>>>
Re: [VOTE] Airflow 1.10.0rc1
Posted by Ash Berlin-Taylor <as...@firemirror.com>.
https://github.com/apache/incubator-airflow/pull/3602 <https://github.com/apache/incubator-airflow/pull/3602>
This fixes the ignore file being ignored in dirs, and fixes AIRFLOW-1729 correctly to not walk into dirs that have been ignored. (The previous PR targeted at that Jira didn't change that behaviour)
It doesn't contain unittests as we don't have anything in place to test that. I'm not happy with that situation, but it doesn't make anything worse. It removes duplication between models.py and utils/dag_processing.py too.
-ash
> On 12 Jul 2018, at 21:54, Bolke de Bruin <bd...@gmail.com> wrote:
>
> If a PR is quick, I’m willing to restart the vote for a RC2 (we are 3 +1, 0.5 -1).
>
> B.
>
> Verstuurd vanaf mijn iPad
>
>> Op 12 jul. 2018 om 15:39 heeft James Meickle <jm...@quantopian.com.INVALID> het volgende geschreven:
>>
>> Looking at that diff, it seems like the function as a whole needs some
>> love, even if that commit were reverted. The use of os.walk means it's
>> going to crawl the entire tree every time, and use accumulated patterns to
>> check against each file in each folder. The behavior should be to use
>> patterns to also exclude entire folders from consideration, which would
>> likely dramatically speed up the function for most use cases.
>>
>> On Thu, Jul 12, 2018 at 9:14 AM, Ash Berlin-Taylor <
>> ash_airflowlist@firemirror.com> wrote:
>>
>>> I wasn't paying enough attention. I can reproduce it.
>>>
>>> I'm not sure it was intentional but the bug was introduced in
>>> https://github.com/apache/incubator-airflow/pull/3171 <
>>> https://github.com/apache/incubator-airflow/pull/3171>
>>>
>>> I'd rather not release with a regression since 1.9, so updating my vote to
>>> -0.5 (binding), but not blocking/vetoing release if others are okay with it.
>>>
>>> -ash
>>>
>>>> On 12 Jul 2018, at 13:46, Ash Berlin-Taylor <ash_airflowlist@firemirror.
>>> com> wrote:
>>>>
>>>> That said I can't reproduce it myself (I will take discussion to the
>>> Jira ticket)
>>>>
>>>> -ash
>>>>
>>>>> On 12 Jul 2018, at 13:36, Ash Berlin-Taylor <
>>> ash_airflowlist@firemirror.com> wrote:
>>>>>
>>>>> Possible blocking regression here - .airflowignore doesn't seem to be
>>> respected anymore
>>>>>
>>>>> https://issues.apache.org/jira/browse/AIRFLOW-2729 <
>>> https://issues.apache.org/jira/browse/AIRFLOW-2729>
>>>>>
>>>>> -ash
>>>>>> On 12 Jul 2018, at 08:45, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>>>
>>>>>> Hi Jakob,
>>>>>>
>>>>>> Thanks. We do include an INSTALL document that explains how to well
>>> install airflow and is quite a standard location for install instructions.
>>> Or did the install instruction needed to be included in the Vote?
>>>>>>
>>>>>> B.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Verstuurd vanaf mijn iPad
>>>>>>
>>>>>>> Op 12 jul. 2018 om 07:54 heeft Jakob Homan <jg...@gmail.com> het
>>> volgende geschreven:
>>>>>>>
>>>>>>> +1 (binding)
>>>>>>>
>>>>>>> * Sigs look good
>>>>>>> * Artifact has incubating in name
>>>>>>> * LICENSE/NOTICE/DISCLAIMER look good
>>>>>>> - nit: DISCLAIMER is not word wrapped to 80 chars as I've seen in
>>>>>>> other projects.
>>>>>>> * Nit: Last release Sebb called out "Copyright 2016 and onwards" in
>>>>>>> NOTICE as being imprecise. This language remains.
>>>>>>> * Spot check on license headers looks ok.
>>>>>>> * Nit: Last release there was a request for instructions on how to
>>>>>>> build the code. This isn't included.
>>>>>>>
>>>>>>> -Jakob
>>>>>>>
>>>>>>>> On 11 July 2018 at 19:50, Sid Anand <sa...@apache.org> wrote:
>>>>>>>> FYI!
>>>>>>>> I just installed the release candidate. The first thing I noticed is
>>> a
>>>>>>>> missing tool tip for the Null State in the Recent Tasks column on the
>>>>>>>> landing page. Since the null globe is new to this UI, users will
>>> likely
>>>>>>>> hover over it to inquire what it means... and will be left wanting.
>>> Of
>>>>>>>> course, they could click on the globe, which will take them to
>>>>>>>> http://localhost:8080/admin/taskinstance/?flt1_dag_id_
>>> equals=example_bash_operator&flt2_state_equals=null,
>>>>>>>> which will always show an empty list, leaving them a bit more
>>> confused.
>>>>>>>>
>>>>>>>> -s
>>>>>>>>
>>>>>>>> On Wed, Jul 11, 2018 at 3:13 PM Carl Johan Gustavsson
>>>>>>>> <ca...@tictail.com.invalid> wrote:
>>>>>>>>
>>>>>>>>> Hi Bolke,
>>>>>>>>>
>>>>>>>>> (Switching email to avoid moderation on my emails.)
>>>>>>>>>
>>>>>>>>> The normal Airflow test suite does not fail as it uses a LC_ALL set
>>> to
>>>>>>>>> utf-8.
>>>>>>>>>
>>>>>>>>> I think it is a proper test though, it is a minimal reproducible
>>> version of
>>>>>>>>> the code that fails. And the only difference in behaviour is at 3.7
>>> which
>>>>>>>>> we don’t support anyway so I’m fairly sure it is broken for all
>>> supported
>>>>>>>>> Python 3 versions.
>>>>>>>>>
>>>>>>>>> I now tried running the tests in docker using 3.5 with the
>>> LC_ALL/LANG
>>>>>>>>> unset and I see the same failure.
>>>>>>>>>
>>>>>>>>> I don’t think this is a big thing though and we could release it
>>> without
>>>>>>>>> the fix I made. I think most people run it with a sane LC_ALL, but
>>>>>>>>> apparently we didn’t.
>>>>>>>>> Here’s the log for the test:
>>>>>>>>>
>>>>>>>>>> docker run -t -i -v `pwd`:/airflow/ python:3.5 bash
>>>>>>>>> root@b99b297df111:/# locale
>>>>>>>>> LANG=C.UTF-8
>>>>>>>>> LANGUAGE=
>>>>>>>>> LC_CTYPE="C.UTF-8"
>>>>>>>>> LC_NUMERIC="C.UTF-8"
>>>>>>>>> LC_TIME="C.UTF-8"
>>>>>>>>> LC_COLLATE="C.UTF-8"
>>>>>>>>> LC_MONETARY="C.UTF-8"
>>>>>>>>> LC_MESSAGES="C.UTF-8"
>>>>>>>>> LC_PAPER="C.UTF-8"
>>>>>>>>> LC_NAME="C.UTF-8"
>>>>>>>>> LC_ADDRESS="C.UTF-8"
>>>>>>>>> LC_TELEPHONE="C.UTF-8"
>>>>>>>>> LC_MEASUREMENT="C.UTF-8"
>>>>>>>>> LC_IDENTIFICATION="C.UTF-8"
>>>>>>>>> LC_ALL=
>>>>>>>>>> unset LANG
>>>>>>>>> root@b99b297df111:/# locale
>>>>>>>>> LANG=
>>>>>>>>> LANGUAGE=
>>>>>>>>> LC_CTYPE="POSIX"
>>>>>>>>> LC_NUMERIC="POSIX"
>>>>>>>>> LC_TIME="POSIX"
>>>>>>>>> LC_COLLATE="POSIX"
>>>>>>>>> LC_MONETARY="POSIX"
>>>>>>>>> LC_MESSAGES="POSIX"
>>>>>>>>> LC_PAPER="POSIX"
>>>>>>>>> LC_NAME="POSIX"
>>>>>>>>> LC_ADDRESS="POSIX"
>>>>>>>>> LC_TELEPHONE="POSIX"
>>>>>>>>> LC_MEASUREMENT="POSIX"
>>>>>>>>> LC_IDENTIFICATION="POSIX"
>>>>>>>>> LC_ALL=
>>>>>>>>> root@b99b297df111:/# pip install -e .[devel]
>>>>>>>>> root@b99b297df111:/airflow# ./run_unit_tests.sh
>>>>>>>>> + export AIRFLOW_HOME=/root/airflow
>>>>>>>>> + AIRFLOW_HOME=/root/airflow
>>>>>>>>> + export AIRFLOW__CORE__UNIT_TEST_MODE=True
>>>>>>>>> + AIRFLOW__CORE__UNIT_TEST_MODE=True
>>>>>>>>> + export AIRFLOW__TESTSECTION__TESTKEY=testvalue
>>>>>>>>> + AIRFLOW__TESTSECTION__TESTKEY=testvalue
>>>>>>>>> + export AIRFLOW_USE_NEW_IMPORTS=1
>>>>>>>>> + AIRFLOW_USE_NEW_IMPORTS=1
>>>>>>>>> +++ dirname ./run_unit_tests.sh
>>>>>>>>> ++ cd .
>>>>>>>>> ++ pwd
>>>>>>>>> + DIR=/airflow
>>>>>>>>> + export PYTHONPATH=:/airflow/tests/test_utils
>>>>>>>>> + PYTHONPATH=:/airflow/tests/test_utils
>>>>>>>>> + nose_args=
>>>>>>>>> + which airflow
>>>>>>>>> + echo 'Initializing the DB'
>>>>>>>>> Initializing the DB
>>>>>>>>> + airflow resetdb
>>>>>>>>> + yes
>>>>>>>>> Traceback (most recent call last):
>>>>>>>>> File "/usr/local/bin/airflow", line 6, in <module>
>>>>>>>>> exec(compile(open(__file__).read(), __file__, 'exec'))
>>>>>>>>> File "/airflow/airflow/bin/airflow", line 21, in <module>
>>>>>>>>> from airflow import configuration
>>>>>>>>> File "/airflow/airflow/__init__.py", line 35, in <module>
>>>>>>>>> from airflow import configuration as conf
>>>>>>>>> File "/airflow/airflow/configuration.py", line 106, in <module>
>>>>>>>>> DEFAULT_CONFIG = f.read()
>>>>>>>>> File "/usr/local/lib/python3.5/encodings/ascii.py", line 26, in
>>> decode
>>>>>>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in
>>> position 21082:
>>>>>>>>> ordinal not in range(128)
>>>>>>>>> + '[' '' ']'
>>>>>>>>> + '[' -z '' ']'
>>>>>>>>> + nose_args='--with-coverage --cover-erase --cover-html
>>>>>>>>> --cover-package=airflow --cover-html-dir=airflow/www/
>>> static/coverage
>>>>>>>>> --with-ignore-docstrings --rednose --with-timer -s
>>> -v
>>>>>>>>> --logging-level=DEBUG '
>>>>>>>>> + echo 'Starting the unit tests with the following nose arguments:
>>>>>>>>> --with-coverage' --cover-erase --cover-html --cover-package=airflow
>>>>>>>>> --cover-html-dir=airflow/www/static/coverage
>>> --with-ignore-docstrings
>>>>>>>>> --rednose --with-timer -s -v --logging-level=DEBUG
>>>>>>>>> Starting the unit tests with the following nose arguments:
>>> --with-coverage
>>>>>>>>> --cover-erase --cover-html --cover-package=airflow
>>>>>>>>> --cover-html-dir=airflow/www/static/coverage
>>> --with-ignore-docstrings
>>>>>>>>> --rednose --with-timer -s -v --logging-level=DEBUG
>>>>>>>>> + nosetests --with-coverage --cover-erase --cover-html
>>>>>>>>> --cover-package=airflow --cover-html-dir=airflow/www/
>>> static/coverage
>>>>>>>>> --with-ignore-docstrings --rednose --with-timer -s -v
>>> --logging-level=DEBUG
>>>>>>>>> nose.plugins.cover: ERROR: Coverage not available: unable to import
>>>>>>>>> coverage module
>>>>>>>>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>>>>>>>>> position 21082: ordinal not in range(128)) ... ERROR
>>>>>>>>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>>>>>>>>> position 21082: ordinal not in range(128)) ... ERROR
>>>>>>>>> ============================================================
>>> ==========
>>>>>>>>> 1) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode
>>> byte 0xe2
>>>>>>>>> in position 21082: ordinal not in range(128))
>>>>>>>>> ------------------------------------------------------------
>>> ----------
>>>>>>>>> Traceback (most recent call last):
>>>>>>>>> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>>>>>>>>> runTest
>>>>>>>>> raise self.exc_val.with_traceback(self.tb)
>>>>>>>>> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>>>>>>>>> loadTestsFromName
>>>>>>>>> addr.filename, addr.module)
>>>>>>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>>>>>>>>> importFromPath
>>>>>>>>> return self.importFromDir(dir_path, fqname)
>>>>>>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>>>>>>>>> importFromDir
>>>>>>>>> mod = load_module(part_fqname, fh, filename, desc)
>>>>>>>>> /usr/local/lib/python3.5/imp.py line 245 in load_module
>>>>>>>>> return load_package(name, filename)
>>>>>>>>> /usr/local/lib/python3.5/imp.py line 217 in load_package
>>>>>>>>> return _load(spec)
>>>>>>>>> <frozen importlib._bootstrap> line 693 in _load
>>>>>>>>>
>>>>>>>>> <frozen importlib._bootstrap> line 673 in _load_unlocked
>>>>>>>>>
>>>>>>>>> <frozen importlib._bootstrap_external> line 697 in exec_module
>>>>>>>>>
>>>>>>>>> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>>>>>>>>>
>>>>>>>>> airflow/__init__.py line 35 in <module>
>>>>>>>>> from airflow import configuration as conf
>>>>>>>>> airflow/configuration.py line 106 in <module>
>>>>>>>>> DEFAULT_CONFIG = f.read()
>>>>>>>>> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>>>>>>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>>>>>>>> 21082: ordinal not in range(128)
>>>>>>>>> ============================================================
>>> ==========
>>>>>>>>> 2) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode
>>> byte 0xe2
>>>>>>>>> in position 21082: ordinal not in range(128))
>>>>>>>>> ------------------------------------------------------------
>>> ----------
>>>>>>>>> Traceback (most recent call last):
>>>>>>>>> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>>>>>>>>> runTest
>>>>>>>>> raise self.exc_val.with_traceback(self.tb)
>>>>>>>>> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>>>>>>>>> loadTestsFromName
>>>>>>>>> addr.filename, addr.module)
>>>>>>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>>>>>>>>> importFromPath
>>>>>>>>> return self.importFromDir(dir_path, fqname)
>>>>>>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>>>>>>>>> importFromDir
>>>>>>>>> mod = load_module(part_fqname, fh, filename, desc)
>>>>>>>>> /usr/local/lib/python3.5/imp.py line 245 in load_module
>>>>>>>>> return load_package(name, filename)
>>>>>>>>> /usr/local/lib/python3.5/imp.py line 217 in load_package
>>>>>>>>> return _load(spec)
>>>>>>>>> <frozen importlib._bootstrap> line 693 in _load
>>>>>>>>>
>>>>>>>>> <frozen importlib._bootstrap> line 673 in _load_unlocked
>>>>>>>>>
>>>>>>>>> <frozen importlib._bootstrap_external> line 697 in exec_module
>>>>>>>>>
>>>>>>>>> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>>>>>>>>>
>>>>>>>>> tests/__init__.py line 25 in <module>
>>>>>>>>> from .configuration import *
>>>>>>>>> tests/configuration.py line 28 in <module>
>>>>>>>>> from airflow import configuration
>>>>>>>>> airflow/__init__.py line 35 in <module>
>>>>>>>>> from airflow import configuration as conf
>>>>>>>>> airflow/configuration.py line 106 in <module>
>>>>>>>>> DEFAULT_CONFIG = f.read()
>>>>>>>>> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>>>>>>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>>>>>>>> 21082: ordinal not in range(128)
>>>>>>>>>
>>>>>>>>> [error] 100.00% nose.failure.Failure.runTest: 0.0004s
>>>>>>>>>
>>>>>>>>> ------------------------------------------------------------
>>> -----------------
>>>>>>>>> 2 tests run in 0.060 seconds.
>>>>>>>>> 2 errors (0 tests passed)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Carl Johan Gustavsson
>>>>>>>>>
>>>>>>>>> On 11 July 2018 at 23:24:01, Bolke de Bruin (bdbruin@gmail.com)
>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi Carl,
>>>>>>>>>
>>>>>>>>> That is not a real check, ie. Does Airflow have the same issue clean
>>>>>>>>> install on 3.5? Travis’ tests run on 3.5.
>>>>>>>>>
>>>>>>>>> B.
>>>>>>>>>
>>>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>>>>
>>>>>>>>>> Op 10 jul. 2018 om 15:10 heeft Carl Johan Gustavsson <
>>>>>>>>> carl.j.gustavsson@gmail.com> het volgende geschreven:
>>>>>>>>>>
>>>>>>>>>> Hi Bolke,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all but
>>> 3.7
>>>>>>>>> breaks
>>>>>>>>>>
>>>>>>>>>> Quick repro:
>>>>>>>>>>
>>>>>>>>>> ➜ ~ pyenv local 3.5.5
>>>>>>>>>> ➜ ~ locale
>>>>>>>>>> LANG=
>>>>>>>>>> LC_COLLATE="C"
>>>>>>>>>> LC_CTYPE="C"
>>>>>>>>>> LC_MESSAGES="C"
>>>>>>>>>> LC_MONETARY="C"
>>>>>>>>>> LC_NUMERIC="C"
>>>>>>>>>> LC_TIME="C"
>>>>>>>>>> LC_ALL=
>>>>>>>>>> ➜ ~ cat testweird.txt
>>>>>>>>>> ’
>>>>>>>>>> ➜ ~ python
>>>>>>>>>> Python 3.5.5 (default, Jul 7 2018, 17:00:56)
>>>>>>>>>> [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on
>>> darwin
>>>>>>>>>> Type "help", "copyright", "credits" or "license" for more
>>> information.
>>>>>>>>>>>>> open('testweird.txt').read()
>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>> File "<stdin>", line 1, in <module>
>>>>>>>>>> File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/
>>> ascii.py",
>>>>>>>>> line 26, in decode
>>>>>>>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>>>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in
>>> position 0:
>>>>>>>>> ordinal not in range(128)
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Maybe not a blocking change but it is a breaking change from 1.9 I
>>> guess.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> / Carl Johan
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbruin@gmail.com)
>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Carl,
>>>>>>>>>>>
>>>>>>>>>>> Did you this on python 3.5 as well? 3.6 is not an officially
>>> supported
>>>>>>>>> (yet). As a workaround is available I won’t consider this blocking
>>> btw.
>>>>>>>>>>>
>>>>>>>>>>> Bolke
>>>>>>>>>>>
>>>>>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>>>>>>
>>>>>>>>>>> Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson <
>>>>>>>>> carl.j.gustavsson@gmail.com> het volgende geschreven:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> First of all, thank you for all the work with the release
>>> management.
>>>>>>>>>>>>
>>>>>>>>>>>> I ran in to a weird issue testing the RC1, running under Python
>>> 3.6.0 /
>>>>>>>>> Ubuntu 14.04.5, upgrading from a master build from February.
>>>>>>>>>>>>
>>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>>> Traceback
>>>>>>>>> (most recent call last):
>>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>>>>>>> "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in
>>> <module>
>>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from
>>> airflow
>>>>>>>>> import configuration
>>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>>>>>>>
>>>>>>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-
>>> packages/airflow/__init__.py",
>>>>>>>>> line 35, in <module>
>>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from
>>> airflow
>>>>>>>>> import configuration as conf
>>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>>>>>>>
>>>>>>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-
>>> packages/airflow/configuration.py",
>>>>>>>>> line 106, in <module>
>>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>>>>>>>>> DEFAULT_CONFIG = f.read()
>>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>>>>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py",
>>> line
>>>>>>>>> 26, in decode
>>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return
>>>>>>>>> codecs.ascii_decode(input, self.errors)[0]
>>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>>>>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in
>>> position 20770:
>>>>>>>>> ordinal not in range(128)
>>>>>>>>>>>>
>>>>>>>>>>>> Removing the ’ on
>>>>>>>>>
>>>>>>>>> https://github.com/apache/incubator-airflow/blob/master/
>>> airflow/config_templates/default_airflow.cfg#L613
>>>>>>>>> solved the issue for me, and digging a bit deeper it seems Airflow
>>> now
>>>>>>>>> requires setting LC_ALL=en_US.UTF-8 in the environment or similar
>>> to force
>>>>>>>>> Python to read the file as utf-8 and not ascii. (I think this was
>>> changed
>>>>>>>>> in to default to utf-8 in Python 3.7).
>>>>>>>>>>>>
>>>>>>>>>>>> I see 3 solutions for this
>>>>>>>>>>>> 1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or
>>>>>>>>> similar.
>>>>>>>>>>>> 2. Change the default config file to not contain non-ascii
>>> characters.
>>>>>>>>>>>> 3. Always read the file as unicode regardless of the LC_ALL
>>>>>>>>> environment, by the encoding='utf-8’ parameter to open().
>>>>>>>>>>>>
>>>>>>>>>>>> I think 3 is the best solution, and I can prepare a PR for that
>>> if
>>>>>>>>> necessary .
>>>>>>>>>>>>
>>>>>>>>>>>> I guess this counts as -1 (non-binding)
>>>>>>>>>>>>
>>>>>>>>>>>> All the best
>>>>>>>>>>>>
>>>>>>>>>>>> Carl Johan
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbruin@gmail.com)
>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on
>>> the
>>>>>>>>> release,
>>>>>>>>>>>>> which will last for 72 hours. Consider this my (binding) +1.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Airflow 1.10.0 RC 1 is available at:
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
>>> 10.0rc1/ <
>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/
>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source
>>> release
>>>>>>>>> that
>>>>>>>>>>>>> comes with INSTALL instructions.
>>>>>>>>>>>>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary
>>> Python
>>>>>>>>> "sdist"
>>>>>>>>>>>>> release.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Public keys are available at:
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>>>>>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/>
>>>>>>>>>>>>>
>>>>>>>>>>>>> The amount of JIRAs fixed is over 700. Please have a look at the
>>>>>>>>> changelog.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please note that the version number excludes the `rcX` string
>>> as well
>>>>>>>>>>>>> as the "+incubating" string, so it's now simply 1.10.0. This
>>> will
>>>>>>>>> allow us
>>>>>>>>>>>>> to rename the artifact without modifying the artifact checksums
>>> when
>>>>>>>>> we
>>>>>>>>>>>>> actually release.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Bolke
>>>>>>>>>
>>>>>
>>>>
>>>
>>>
Re: [VOTE] Airflow 1.10.0rc1
Posted by Ash Berlin-Taylor <as...@firemirror.com>.
https://github.com/apache/incubator-airflow/pull/3602 <https://github.com/apache/incubator-airflow/pull/3602>
This fixes the ignore file being ignored in dirs, and fixes AIRFLOW-1729 correctly to not walk into dirs that have been ignored. (The previous PR targeted at that Jira didn't change that behaviour)
It doesn't contain unittests as we don't have anything in place to test that. I'm not happy with that situation, but it doesn't make anything worse. It removes duplication between models.py and utils/dag_processing.py too.
(sent again from my correct email so it doesn't need moderation. Sorry)
-ash
> On 12 Jul 2018, at 21:54, Bolke de Bruin <bd...@gmail.com> wrote:
>
> If a PR is quick, I’m willing to restart the vote for a RC2 (we are 3 +1, 0.5 -1).
>
> B.
>
> Verstuurd vanaf mijn iPad
>
>> Op 12 jul. 2018 om 15:39 heeft James Meickle <jm...@quantopian.com.INVALID> het volgende geschreven:
>>
>> Looking at that diff, it seems like the function as a whole needs some
>> love, even if that commit were reverted. The use of os.walk means it's
>> going to crawl the entire tree every time, and use accumulated patterns to
>> check against each file in each folder. The behavior should be to use
>> patterns to also exclude entire folders from consideration, which would
>> likely dramatically speed up the function for most use cases.
>>
>> On Thu, Jul 12, 2018 at 9:14 AM, Ash Berlin-Taylor <
>> ash_airflowlist@firemirror.com> wrote:
>>
>>> I wasn't paying enough attention. I can reproduce it.
>>>
>>> I'm not sure it was intentional but the bug was introduced in
>>> https://github.com/apache/incubator-airflow/pull/3171 <
>>> https://github.com/apache/incubator-airflow/pull/3171>
>>>
>>> I'd rather not release with a regression since 1.9, so updating my vote to
>>> -0.5 (binding), but not blocking/vetoing release if others are okay with it.
>>>
>>> -ash
>>>
>>>> On 12 Jul 2018, at 13:46, Ash Berlin-Taylor <ash_airflowlist@firemirror.
>>> com> wrote:
>>>>
>>>> That said I can't reproduce it myself (I will take discussion to the
>>> Jira ticket)
>>>>
>>>> -ash
>>>>
>>>>> On 12 Jul 2018, at 13:36, Ash Berlin-Taylor <
>>> ash_airflowlist@firemirror.com> wrote:
>>>>>
>>>>> Possible blocking regression here - .airflowignore doesn't seem to be
>>> respected anymore
>>>>>
>>>>> https://issues.apache.org/jira/browse/AIRFLOW-2729 <
>>> https://issues.apache.org/jira/browse/AIRFLOW-2729>
>>>>>
>>>>> -ash
>>>>>> On 12 Jul 2018, at 08:45, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>>>
>>>>>> Hi Jakob,
>>>>>>
>>>>>> Thanks. We do include an INSTALL document that explains how to well
>>> install airflow and is quite a standard location for install instructions.
>>> Or did the install instruction needed to be included in the Vote?
>>>>>>
>>>>>> B.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Verstuurd vanaf mijn iPad
>>>>>>
>>>>>>> Op 12 jul. 2018 om 07:54 heeft Jakob Homan <jg...@gmail.com> het
>>> volgende geschreven:
>>>>>>>
>>>>>>> +1 (binding)
>>>>>>>
>>>>>>> * Sigs look good
>>>>>>> * Artifact has incubating in name
>>>>>>> * LICENSE/NOTICE/DISCLAIMER look good
>>>>>>> - nit: DISCLAIMER is not word wrapped to 80 chars as I've seen in
>>>>>>> other projects.
>>>>>>> * Nit: Last release Sebb called out "Copyright 2016 and onwards" in
>>>>>>> NOTICE as being imprecise. This language remains.
>>>>>>> * Spot check on license headers looks ok.
>>>>>>> * Nit: Last release there was a request for instructions on how to
>>>>>>> build the code. This isn't included.
>>>>>>>
>>>>>>> -Jakob
>>>>>>>
>>>>>>>> On 11 July 2018 at 19:50, Sid Anand <sa...@apache.org> wrote:
>>>>>>>> FYI!
>>>>>>>> I just installed the release candidate. The first thing I noticed is
>>> a
>>>>>>>> missing tool tip for the Null State in the Recent Tasks column on the
>>>>>>>> landing page. Since the null globe is new to this UI, users will
>>> likely
>>>>>>>> hover over it to inquire what it means... and will be left wanting.
>>> Of
>>>>>>>> course, they could click on the globe, which will take them to
>>>>>>>> http://localhost:8080/admin/taskinstance/?flt1_dag_id_
>>> equals=example_bash_operator&flt2_state_equals=null,
>>>>>>>> which will always show an empty list, leaving them a bit more
>>> confused.
>>>>>>>>
>>>>>>>> -s
>>>>>>>>
>>>>>>>> On Wed, Jul 11, 2018 at 3:13 PM Carl Johan Gustavsson
>>>>>>>> <ca...@tictail.com.invalid> wrote:
>>>>>>>>
>>>>>>>>> Hi Bolke,
>>>>>>>>>
>>>>>>>>> (Switching email to avoid moderation on my emails.)
>>>>>>>>>
>>>>>>>>> The normal Airflow test suite does not fail as it uses a LC_ALL set
>>> to
>>>>>>>>> utf-8.
>>>>>>>>>
>>>>>>>>> I think it is a proper test though, it is a minimal reproducible
>>> version of
>>>>>>>>> the code that fails. And the only difference in behaviour is at 3.7
>>> which
>>>>>>>>> we don’t support anyway so I’m fairly sure it is broken for all
>>> supported
>>>>>>>>> Python 3 versions.
>>>>>>>>>
>>>>>>>>> I now tried running the tests in docker using 3.5 with the
>>> LC_ALL/LANG
>>>>>>>>> unset and I see the same failure.
>>>>>>>>>
>>>>>>>>> I don’t think this is a big thing though and we could release it
>>> without
>>>>>>>>> the fix I made. I think most people run it with a sane LC_ALL, but
>>>>>>>>> apparently we didn’t.
>>>>>>>>> Here’s the log for the test:
>>>>>>>>>
>>>>>>>>>> docker run -t -i -v `pwd`:/airflow/ python:3.5 bash
>>>>>>>>> root@b99b297df111:/# locale
>>>>>>>>> LANG=C.UTF-8
>>>>>>>>> LANGUAGE=
>>>>>>>>> LC_CTYPE="C.UTF-8"
>>>>>>>>> LC_NUMERIC="C.UTF-8"
>>>>>>>>> LC_TIME="C.UTF-8"
>>>>>>>>> LC_COLLATE="C.UTF-8"
>>>>>>>>> LC_MONETARY="C.UTF-8"
>>>>>>>>> LC_MESSAGES="C.UTF-8"
>>>>>>>>> LC_PAPER="C.UTF-8"
>>>>>>>>> LC_NAME="C.UTF-8"
>>>>>>>>> LC_ADDRESS="C.UTF-8"
>>>>>>>>> LC_TELEPHONE="C.UTF-8"
>>>>>>>>> LC_MEASUREMENT="C.UTF-8"
>>>>>>>>> LC_IDENTIFICATION="C.UTF-8"
>>>>>>>>> LC_ALL=
>>>>>>>>>> unset LANG
>>>>>>>>> root@b99b297df111:/# locale
>>>>>>>>> LANG=
>>>>>>>>> LANGUAGE=
>>>>>>>>> LC_CTYPE="POSIX"
>>>>>>>>> LC_NUMERIC="POSIX"
>>>>>>>>> LC_TIME="POSIX"
>>>>>>>>> LC_COLLATE="POSIX"
>>>>>>>>> LC_MONETARY="POSIX"
>>>>>>>>> LC_MESSAGES="POSIX"
>>>>>>>>> LC_PAPER="POSIX"
>>>>>>>>> LC_NAME="POSIX"
>>>>>>>>> LC_ADDRESS="POSIX"
>>>>>>>>> LC_TELEPHONE="POSIX"
>>>>>>>>> LC_MEASUREMENT="POSIX"
>>>>>>>>> LC_IDENTIFICATION="POSIX"
>>>>>>>>> LC_ALL=
>>>>>>>>> root@b99b297df111:/# pip install -e .[devel]
>>>>>>>>> root@b99b297df111:/airflow# ./run_unit_tests.sh
>>>>>>>>> + export AIRFLOW_HOME=/root/airflow
>>>>>>>>> + AIRFLOW_HOME=/root/airflow
>>>>>>>>> + export AIRFLOW__CORE__UNIT_TEST_MODE=True
>>>>>>>>> + AIRFLOW__CORE__UNIT_TEST_MODE=True
>>>>>>>>> + export AIRFLOW__TESTSECTION__TESTKEY=testvalue
>>>>>>>>> + AIRFLOW__TESTSECTION__TESTKEY=testvalue
>>>>>>>>> + export AIRFLOW_USE_NEW_IMPORTS=1
>>>>>>>>> + AIRFLOW_USE_NEW_IMPORTS=1
>>>>>>>>> +++ dirname ./run_unit_tests.sh
>>>>>>>>> ++ cd .
>>>>>>>>> ++ pwd
>>>>>>>>> + DIR=/airflow
>>>>>>>>> + export PYTHONPATH=:/airflow/tests/test_utils
>>>>>>>>> + PYTHONPATH=:/airflow/tests/test_utils
>>>>>>>>> + nose_args=
>>>>>>>>> + which airflow
>>>>>>>>> + echo 'Initializing the DB'
>>>>>>>>> Initializing the DB
>>>>>>>>> + airflow resetdb
>>>>>>>>> + yes
>>>>>>>>> Traceback (most recent call last):
>>>>>>>>> File "/usr/local/bin/airflow", line 6, in <module>
>>>>>>>>> exec(compile(open(__file__).read(), __file__, 'exec'))
>>>>>>>>> File "/airflow/airflow/bin/airflow", line 21, in <module>
>>>>>>>>> from airflow import configuration
>>>>>>>>> File "/airflow/airflow/__init__.py", line 35, in <module>
>>>>>>>>> from airflow import configuration as conf
>>>>>>>>> File "/airflow/airflow/configuration.py", line 106, in <module>
>>>>>>>>> DEFAULT_CONFIG = f.read()
>>>>>>>>> File "/usr/local/lib/python3.5/encodings/ascii.py", line 26, in
>>> decode
>>>>>>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in
>>> position 21082:
>>>>>>>>> ordinal not in range(128)
>>>>>>>>> + '[' '' ']'
>>>>>>>>> + '[' -z '' ']'
>>>>>>>>> + nose_args='--with-coverage --cover-erase --cover-html
>>>>>>>>> --cover-package=airflow --cover-html-dir=airflow/www/
>>> static/coverage
>>>>>>>>> --with-ignore-docstrings --rednose --with-timer -s
>>> -v
>>>>>>>>> --logging-level=DEBUG '
>>>>>>>>> + echo 'Starting the unit tests with the following nose arguments:
>>>>>>>>> --with-coverage' --cover-erase --cover-html --cover-package=airflow
>>>>>>>>> --cover-html-dir=airflow/www/static/coverage
>>> --with-ignore-docstrings
>>>>>>>>> --rednose --with-timer -s -v --logging-level=DEBUG
>>>>>>>>> Starting the unit tests with the following nose arguments:
>>> --with-coverage
>>>>>>>>> --cover-erase --cover-html --cover-package=airflow
>>>>>>>>> --cover-html-dir=airflow/www/static/coverage
>>> --with-ignore-docstrings
>>>>>>>>> --rednose --with-timer -s -v --logging-level=DEBUG
>>>>>>>>> + nosetests --with-coverage --cover-erase --cover-html
>>>>>>>>> --cover-package=airflow --cover-html-dir=airflow/www/
>>> static/coverage
>>>>>>>>> --with-ignore-docstrings --rednose --with-timer -s -v
>>> --logging-level=DEBUG
>>>>>>>>> nose.plugins.cover: ERROR: Coverage not available: unable to import
>>>>>>>>> coverage module
>>>>>>>>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>>>>>>>>> position 21082: ordinal not in range(128)) ... ERROR
>>>>>>>>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>>>>>>>>> position 21082: ordinal not in range(128)) ... ERROR
>>>>>>>>> ============================================================
>>> ==========
>>>>>>>>> 1) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode
>>> byte 0xe2
>>>>>>>>> in position 21082: ordinal not in range(128))
>>>>>>>>> ------------------------------------------------------------
>>> ----------
>>>>>>>>> Traceback (most recent call last):
>>>>>>>>> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>>>>>>>>> runTest
>>>>>>>>> raise self.exc_val.with_traceback(self.tb)
>>>>>>>>> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>>>>>>>>> loadTestsFromName
>>>>>>>>> addr.filename, addr.module)
>>>>>>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>>>>>>>>> importFromPath
>>>>>>>>> return self.importFromDir(dir_path, fqname)
>>>>>>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>>>>>>>>> importFromDir
>>>>>>>>> mod = load_module(part_fqname, fh, filename, desc)
>>>>>>>>> /usr/local/lib/python3.5/imp.py line 245 in load_module
>>>>>>>>> return load_package(name, filename)
>>>>>>>>> /usr/local/lib/python3.5/imp.py line 217 in load_package
>>>>>>>>> return _load(spec)
>>>>>>>>> <frozen importlib._bootstrap> line 693 in _load
>>>>>>>>>
>>>>>>>>> <frozen importlib._bootstrap> line 673 in _load_unlocked
>>>>>>>>>
>>>>>>>>> <frozen importlib._bootstrap_external> line 697 in exec_module
>>>>>>>>>
>>>>>>>>> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>>>>>>>>>
>>>>>>>>> airflow/__init__.py line 35 in <module>
>>>>>>>>> from airflow import configuration as conf
>>>>>>>>> airflow/configuration.py line 106 in <module>
>>>>>>>>> DEFAULT_CONFIG = f.read()
>>>>>>>>> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>>>>>>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>>>>>>>> 21082: ordinal not in range(128)
>>>>>>>>> ============================================================
>>> ==========
>>>>>>>>> 2) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode
>>> byte 0xe2
>>>>>>>>> in position 21082: ordinal not in range(128))
>>>>>>>>> ------------------------------------------------------------
>>> ----------
>>>>>>>>> Traceback (most recent call last):
>>>>>>>>> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>>>>>>>>> runTest
>>>>>>>>> raise self.exc_val.with_traceback(self.tb)
>>>>>>>>> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>>>>>>>>> loadTestsFromName
>>>>>>>>> addr.filename, addr.module)
>>>>>>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>>>>>>>>> importFromPath
>>>>>>>>> return self.importFromDir(dir_path, fqname)
>>>>>>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>>>>>>>>> importFromDir
>>>>>>>>> mod = load_module(part_fqname, fh, filename, desc)
>>>>>>>>> /usr/local/lib/python3.5/imp.py line 245 in load_module
>>>>>>>>> return load_package(name, filename)
>>>>>>>>> /usr/local/lib/python3.5/imp.py line 217 in load_package
>>>>>>>>> return _load(spec)
>>>>>>>>> <frozen importlib._bootstrap> line 693 in _load
>>>>>>>>>
>>>>>>>>> <frozen importlib._bootstrap> line 673 in _load_unlocked
>>>>>>>>>
>>>>>>>>> <frozen importlib._bootstrap_external> line 697 in exec_module
>>>>>>>>>
>>>>>>>>> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>>>>>>>>>
>>>>>>>>> tests/__init__.py line 25 in <module>
>>>>>>>>> from .configuration import *
>>>>>>>>> tests/configuration.py line 28 in <module>
>>>>>>>>> from airflow import configuration
>>>>>>>>> airflow/__init__.py line 35 in <module>
>>>>>>>>> from airflow import configuration as conf
>>>>>>>>> airflow/configuration.py line 106 in <module>
>>>>>>>>> DEFAULT_CONFIG = f.read()
>>>>>>>>> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>>>>>>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>>>>>>>> 21082: ordinal not in range(128)
>>>>>>>>>
>>>>>>>>> [error] 100.00% nose.failure.Failure.runTest: 0.0004s
>>>>>>>>>
>>>>>>>>> ------------------------------------------------------------
>>> -----------------
>>>>>>>>> 2 tests run in 0.060 seconds.
>>>>>>>>> 2 errors (0 tests passed)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Carl Johan Gustavsson
>>>>>>>>>
>>>>>>>>> On 11 July 2018 at 23:24:01, Bolke de Bruin (bdbruin@gmail.com)
>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi Carl,
>>>>>>>>>
>>>>>>>>> That is not a real check, ie. Does Airflow have the same issue clean
>>>>>>>>> install on 3.5? Travis’ tests run on 3.5.
>>>>>>>>>
>>>>>>>>> B.
>>>>>>>>>
>>>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>>>>
>>>>>>>>>> Op 10 jul. 2018 om 15:10 heeft Carl Johan Gustavsson <
>>>>>>>>> carl.j.gustavsson@gmail.com> het volgende geschreven:
>>>>>>>>>>
>>>>>>>>>> Hi Bolke,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all but
>>> 3.7
>>>>>>>>> breaks
>>>>>>>>>>
>>>>>>>>>> Quick repro:
>>>>>>>>>>
>>>>>>>>>> ➜ ~ pyenv local 3.5.5
>>>>>>>>>> ➜ ~ locale
>>>>>>>>>> LANG=
>>>>>>>>>> LC_COLLATE="C"
>>>>>>>>>> LC_CTYPE="C"
>>>>>>>>>> LC_MESSAGES="C"
>>>>>>>>>> LC_MONETARY="C"
>>>>>>>>>> LC_NUMERIC="C"
>>>>>>>>>> LC_TIME="C"
>>>>>>>>>> LC_ALL=
>>>>>>>>>> ➜ ~ cat testweird.txt
>>>>>>>>>> ’
>>>>>>>>>> ➜ ~ python
>>>>>>>>>> Python 3.5.5 (default, Jul 7 2018, 17:00:56)
>>>>>>>>>> [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on
>>> darwin
>>>>>>>>>> Type "help", "copyright", "credits" or "license" for more
>>> information.
>>>>>>>>>>>>> open('testweird.txt').read()
>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>> File "<stdin>", line 1, in <module>
>>>>>>>>>> File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/
>>> ascii.py",
>>>>>>>>> line 26, in decode
>>>>>>>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>>>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in
>>> position 0:
>>>>>>>>> ordinal not in range(128)
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Maybe not a blocking change but it is a breaking change from 1.9 I
>>> guess.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> / Carl Johan
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbruin@gmail.com)
>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Carl,
>>>>>>>>>>>
>>>>>>>>>>> Did you this on python 3.5 as well? 3.6 is not an officially
>>> supported
>>>>>>>>> (yet). As a workaround is available I won’t consider this blocking
>>> btw.
>>>>>>>>>>>
>>>>>>>>>>> Bolke
>>>>>>>>>>>
>>>>>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>>>>>>
>>>>>>>>>>> Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson <
>>>>>>>>> carl.j.gustavsson@gmail.com> het volgende geschreven:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> First of all, thank you for all the work with the release
>>> management.
>>>>>>>>>>>>
>>>>>>>>>>>> I ran in to a weird issue testing the RC1, running under Python
>>> 3.6.0 /
>>>>>>>>> Ubuntu 14.04.5, upgrading from a master build from February.
>>>>>>>>>>>>
>>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>>> Traceback
>>>>>>>>> (most recent call last):
>>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>>>>>>> "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in
>>> <module>
>>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from
>>> airflow
>>>>>>>>> import configuration
>>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>>>>>>>
>>>>>>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-
>>> packages/airflow/__init__.py",
>>>>>>>>> line 35, in <module>
>>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from
>>> airflow
>>>>>>>>> import configuration as conf
>>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>>>>>>>
>>>>>>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-
>>> packages/airflow/configuration.py",
>>>>>>>>> line 106, in <module>
>>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>>>>>>>>> DEFAULT_CONFIG = f.read()
>>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>>>>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py",
>>> line
>>>>>>>>> 26, in decode
>>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return
>>>>>>>>> codecs.ascii_decode(input, self.errors)[0]
>>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>>>>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in
>>> position 20770:
>>>>>>>>> ordinal not in range(128)
>>>>>>>>>>>>
>>>>>>>>>>>> Removing the ’ on
>>>>>>>>>
>>>>>>>>> https://github.com/apache/incubator-airflow/blob/master/
>>> airflow/config_templates/default_airflow.cfg#L613
>>>>>>>>> solved the issue for me, and digging a bit deeper it seems Airflow
>>> now
>>>>>>>>> requires setting LC_ALL=en_US.UTF-8 in the environment or similar
>>> to force
>>>>>>>>> Python to read the file as utf-8 and not ascii. (I think this was
>>> changed
>>>>>>>>> in to default to utf-8 in Python 3.7).
>>>>>>>>>>>>
>>>>>>>>>>>> I see 3 solutions for this
>>>>>>>>>>>> 1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or
>>>>>>>>> similar.
>>>>>>>>>>>> 2. Change the default config file to not contain non-ascii
>>> characters.
>>>>>>>>>>>> 3. Always read the file as unicode regardless of the LC_ALL
>>>>>>>>> environment, by the encoding='utf-8’ parameter to open().
>>>>>>>>>>>>
>>>>>>>>>>>> I think 3 is the best solution, and I can prepare a PR for that
>>> if
>>>>>>>>> necessary .
>>>>>>>>>>>>
>>>>>>>>>>>> I guess this counts as -1 (non-binding)
>>>>>>>>>>>>
>>>>>>>>>>>> All the best
>>>>>>>>>>>>
>>>>>>>>>>>> Carl Johan
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbruin@gmail.com)
>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on
>>> the
>>>>>>>>> release,
>>>>>>>>>>>>> which will last for 72 hours. Consider this my (binding) +1.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Airflow 1.10.0 RC 1 is available at:
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
>>> 10.0rc1/ <
>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/
>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source
>>> release
>>>>>>>>> that
>>>>>>>>>>>>> comes with INSTALL instructions.
>>>>>>>>>>>>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary
>>> Python
>>>>>>>>> "sdist"
>>>>>>>>>>>>> release.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Public keys are available at:
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>>>>>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/>
>>>>>>>>>>>>>
>>>>>>>>>>>>> The amount of JIRAs fixed is over 700. Please have a look at the
>>>>>>>>> changelog.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please note that the version number excludes the `rcX` string
>>> as well
>>>>>>>>>>>>> as the "+incubating" string, so it's now simply 1.10.0. This
>>> will
>>>>>>>>> allow us
>>>>>>>>>>>>> to rename the artifact without modifying the artifact checksums
>>> when
>>>>>>>>> we
>>>>>>>>>>>>> actually release.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Bolke
>>>>>>>>>
>>>>>
>>>>
>>>
>>>
Re: [VOTE] Airflow 1.10.0rc1
Posted by Bolke de Bruin <bd...@gmail.com>.
If a PR is quick, I’m willing to restart the vote for a RC2 (we are 3 +1, 0.5 -1).
B.
Verstuurd vanaf mijn iPad
> Op 12 jul. 2018 om 15:39 heeft James Meickle <jm...@quantopian.com.INVALID> het volgende geschreven:
>
> Looking at that diff, it seems like the function as a whole needs some
> love, even if that commit were reverted. The use of os.walk means it's
> going to crawl the entire tree every time, and use accumulated patterns to
> check against each file in each folder. The behavior should be to use
> patterns to also exclude entire folders from consideration, which would
> likely dramatically speed up the function for most use cases.
>
> On Thu, Jul 12, 2018 at 9:14 AM, Ash Berlin-Taylor <
> ash_airflowlist@firemirror.com> wrote:
>
>> I wasn't paying enough attention. I can reproduce it.
>>
>> I'm not sure it was intentional but the bug was introduced in
>> https://github.com/apache/incubator-airflow/pull/3171 <
>> https://github.com/apache/incubator-airflow/pull/3171>
>>
>> I'd rather not release with a regression since 1.9, so updating my vote to
>> -0.5 (binding), but not blocking/vetoing release if others are okay with it.
>>
>> -ash
>>
>>> On 12 Jul 2018, at 13:46, Ash Berlin-Taylor <ash_airflowlist@firemirror.
>> com> wrote:
>>>
>>> That said I can't reproduce it myself (I will take discussion to the
>> Jira ticket)
>>>
>>> -ash
>>>
>>>> On 12 Jul 2018, at 13:36, Ash Berlin-Taylor <
>> ash_airflowlist@firemirror.com> wrote:
>>>>
>>>> Possible blocking regression here - .airflowignore doesn't seem to be
>> respected anymore
>>>>
>>>> https://issues.apache.org/jira/browse/AIRFLOW-2729 <
>> https://issues.apache.org/jira/browse/AIRFLOW-2729>
>>>>
>>>> -ash
>>>>> On 12 Jul 2018, at 08:45, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>>
>>>>> Hi Jakob,
>>>>>
>>>>> Thanks. We do include an INSTALL document that explains how to well
>> install airflow and is quite a standard location for install instructions.
>> Or did the install instruction needed to be included in the Vote?
>>>>>
>>>>> B.
>>>>>
>>>>>
>>>>>
>>>>> Verstuurd vanaf mijn iPad
>>>>>
>>>>>> Op 12 jul. 2018 om 07:54 heeft Jakob Homan <jg...@gmail.com> het
>> volgende geschreven:
>>>>>>
>>>>>> +1 (binding)
>>>>>>
>>>>>> * Sigs look good
>>>>>> * Artifact has incubating in name
>>>>>> * LICENSE/NOTICE/DISCLAIMER look good
>>>>>> - nit: DISCLAIMER is not word wrapped to 80 chars as I've seen in
>>>>>> other projects.
>>>>>> * Nit: Last release Sebb called out "Copyright 2016 and onwards" in
>>>>>> NOTICE as being imprecise. This language remains.
>>>>>> * Spot check on license headers looks ok.
>>>>>> * Nit: Last release there was a request for instructions on how to
>>>>>> build the code. This isn't included.
>>>>>>
>>>>>> -Jakob
>>>>>>
>>>>>>> On 11 July 2018 at 19:50, Sid Anand <sa...@apache.org> wrote:
>>>>>>> FYI!
>>>>>>> I just installed the release candidate. The first thing I noticed is
>> a
>>>>>>> missing tool tip for the Null State in the Recent Tasks column on the
>>>>>>> landing page. Since the null globe is new to this UI, users will
>> likely
>>>>>>> hover over it to inquire what it means... and will be left wanting.
>> Of
>>>>>>> course, they could click on the globe, which will take them to
>>>>>>> http://localhost:8080/admin/taskinstance/?flt1_dag_id_
>> equals=example_bash_operator&flt2_state_equals=null,
>>>>>>> which will always show an empty list, leaving them a bit more
>> confused.
>>>>>>>
>>>>>>> -s
>>>>>>>
>>>>>>> On Wed, Jul 11, 2018 at 3:13 PM Carl Johan Gustavsson
>>>>>>> <ca...@tictail.com.invalid> wrote:
>>>>>>>
>>>>>>>> Hi Bolke,
>>>>>>>>
>>>>>>>> (Switching email to avoid moderation on my emails.)
>>>>>>>>
>>>>>>>> The normal Airflow test suite does not fail as it uses a LC_ALL set
>> to
>>>>>>>> utf-8.
>>>>>>>>
>>>>>>>> I think it is a proper test though, it is a minimal reproducible
>> version of
>>>>>>>> the code that fails. And the only difference in behaviour is at 3.7
>> which
>>>>>>>> we don’t support anyway so I’m fairly sure it is broken for all
>> supported
>>>>>>>> Python 3 versions.
>>>>>>>>
>>>>>>>> I now tried running the tests in docker using 3.5 with the
>> LC_ALL/LANG
>>>>>>>> unset and I see the same failure.
>>>>>>>>
>>>>>>>> I don’t think this is a big thing though and we could release it
>> without
>>>>>>>> the fix I made. I think most people run it with a sane LC_ALL, but
>>>>>>>> apparently we didn’t.
>>>>>>>> Here’s the log for the test:
>>>>>>>>
>>>>>>>>> docker run -t -i -v `pwd`:/airflow/ python:3.5 bash
>>>>>>>> root@b99b297df111:/# locale
>>>>>>>> LANG=C.UTF-8
>>>>>>>> LANGUAGE=
>>>>>>>> LC_CTYPE="C.UTF-8"
>>>>>>>> LC_NUMERIC="C.UTF-8"
>>>>>>>> LC_TIME="C.UTF-8"
>>>>>>>> LC_COLLATE="C.UTF-8"
>>>>>>>> LC_MONETARY="C.UTF-8"
>>>>>>>> LC_MESSAGES="C.UTF-8"
>>>>>>>> LC_PAPER="C.UTF-8"
>>>>>>>> LC_NAME="C.UTF-8"
>>>>>>>> LC_ADDRESS="C.UTF-8"
>>>>>>>> LC_TELEPHONE="C.UTF-8"
>>>>>>>> LC_MEASUREMENT="C.UTF-8"
>>>>>>>> LC_IDENTIFICATION="C.UTF-8"
>>>>>>>> LC_ALL=
>>>>>>>>> unset LANG
>>>>>>>> root@b99b297df111:/# locale
>>>>>>>> LANG=
>>>>>>>> LANGUAGE=
>>>>>>>> LC_CTYPE="POSIX"
>>>>>>>> LC_NUMERIC="POSIX"
>>>>>>>> LC_TIME="POSIX"
>>>>>>>> LC_COLLATE="POSIX"
>>>>>>>> LC_MONETARY="POSIX"
>>>>>>>> LC_MESSAGES="POSIX"
>>>>>>>> LC_PAPER="POSIX"
>>>>>>>> LC_NAME="POSIX"
>>>>>>>> LC_ADDRESS="POSIX"
>>>>>>>> LC_TELEPHONE="POSIX"
>>>>>>>> LC_MEASUREMENT="POSIX"
>>>>>>>> LC_IDENTIFICATION="POSIX"
>>>>>>>> LC_ALL=
>>>>>>>> root@b99b297df111:/# pip install -e .[devel]
>>>>>>>> root@b99b297df111:/airflow# ./run_unit_tests.sh
>>>>>>>> + export AIRFLOW_HOME=/root/airflow
>>>>>>>> + AIRFLOW_HOME=/root/airflow
>>>>>>>> + export AIRFLOW__CORE__UNIT_TEST_MODE=True
>>>>>>>> + AIRFLOW__CORE__UNIT_TEST_MODE=True
>>>>>>>> + export AIRFLOW__TESTSECTION__TESTKEY=testvalue
>>>>>>>> + AIRFLOW__TESTSECTION__TESTKEY=testvalue
>>>>>>>> + export AIRFLOW_USE_NEW_IMPORTS=1
>>>>>>>> + AIRFLOW_USE_NEW_IMPORTS=1
>>>>>>>> +++ dirname ./run_unit_tests.sh
>>>>>>>> ++ cd .
>>>>>>>> ++ pwd
>>>>>>>> + DIR=/airflow
>>>>>>>> + export PYTHONPATH=:/airflow/tests/test_utils
>>>>>>>> + PYTHONPATH=:/airflow/tests/test_utils
>>>>>>>> + nose_args=
>>>>>>>> + which airflow
>>>>>>>> + echo 'Initializing the DB'
>>>>>>>> Initializing the DB
>>>>>>>> + airflow resetdb
>>>>>>>> + yes
>>>>>>>> Traceback (most recent call last):
>>>>>>>> File "/usr/local/bin/airflow", line 6, in <module>
>>>>>>>> exec(compile(open(__file__).read(), __file__, 'exec'))
>>>>>>>> File "/airflow/airflow/bin/airflow", line 21, in <module>
>>>>>>>> from airflow import configuration
>>>>>>>> File "/airflow/airflow/__init__.py", line 35, in <module>
>>>>>>>> from airflow import configuration as conf
>>>>>>>> File "/airflow/airflow/configuration.py", line 106, in <module>
>>>>>>>> DEFAULT_CONFIG = f.read()
>>>>>>>> File "/usr/local/lib/python3.5/encodings/ascii.py", line 26, in
>> decode
>>>>>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in
>> position 21082:
>>>>>>>> ordinal not in range(128)
>>>>>>>> + '[' '' ']'
>>>>>>>> + '[' -z '' ']'
>>>>>>>> + nose_args='--with-coverage --cover-erase --cover-html
>>>>>>>> --cover-package=airflow --cover-html-dir=airflow/www/
>> static/coverage
>>>>>>>> --with-ignore-docstrings --rednose --with-timer -s
>> -v
>>>>>>>> --logging-level=DEBUG '
>>>>>>>> + echo 'Starting the unit tests with the following nose arguments:
>>>>>>>> --with-coverage' --cover-erase --cover-html --cover-package=airflow
>>>>>>>> --cover-html-dir=airflow/www/static/coverage
>> --with-ignore-docstrings
>>>>>>>> --rednose --with-timer -s -v --logging-level=DEBUG
>>>>>>>> Starting the unit tests with the following nose arguments:
>> --with-coverage
>>>>>>>> --cover-erase --cover-html --cover-package=airflow
>>>>>>>> --cover-html-dir=airflow/www/static/coverage
>> --with-ignore-docstrings
>>>>>>>> --rednose --with-timer -s -v --logging-level=DEBUG
>>>>>>>> + nosetests --with-coverage --cover-erase --cover-html
>>>>>>>> --cover-package=airflow --cover-html-dir=airflow/www/
>> static/coverage
>>>>>>>> --with-ignore-docstrings --rednose --with-timer -s -v
>> --logging-level=DEBUG
>>>>>>>> nose.plugins.cover: ERROR: Coverage not available: unable to import
>>>>>>>> coverage module
>>>>>>>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>>>>>>>> position 21082: ordinal not in range(128)) ... ERROR
>>>>>>>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>>>>>>>> position 21082: ordinal not in range(128)) ... ERROR
>>>>>>>> ============================================================
>> ==========
>>>>>>>> 1) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode
>> byte 0xe2
>>>>>>>> in position 21082: ordinal not in range(128))
>>>>>>>> ------------------------------------------------------------
>> ----------
>>>>>>>> Traceback (most recent call last):
>>>>>>>> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>>>>>>>> runTest
>>>>>>>> raise self.exc_val.with_traceback(self.tb)
>>>>>>>> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>>>>>>>> loadTestsFromName
>>>>>>>> addr.filename, addr.module)
>>>>>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>>>>>>>> importFromPath
>>>>>>>> return self.importFromDir(dir_path, fqname)
>>>>>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>>>>>>>> importFromDir
>>>>>>>> mod = load_module(part_fqname, fh, filename, desc)
>>>>>>>> /usr/local/lib/python3.5/imp.py line 245 in load_module
>>>>>>>> return load_package(name, filename)
>>>>>>>> /usr/local/lib/python3.5/imp.py line 217 in load_package
>>>>>>>> return _load(spec)
>>>>>>>> <frozen importlib._bootstrap> line 693 in _load
>>>>>>>>
>>>>>>>> <frozen importlib._bootstrap> line 673 in _load_unlocked
>>>>>>>>
>>>>>>>> <frozen importlib._bootstrap_external> line 697 in exec_module
>>>>>>>>
>>>>>>>> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>>>>>>>>
>>>>>>>> airflow/__init__.py line 35 in <module>
>>>>>>>> from airflow import configuration as conf
>>>>>>>> airflow/configuration.py line 106 in <module>
>>>>>>>> DEFAULT_CONFIG = f.read()
>>>>>>>> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>>>>>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>>>>>>> 21082: ordinal not in range(128)
>>>>>>>> ============================================================
>> ==========
>>>>>>>> 2) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode
>> byte 0xe2
>>>>>>>> in position 21082: ordinal not in range(128))
>>>>>>>> ------------------------------------------------------------
>> ----------
>>>>>>>> Traceback (most recent call last):
>>>>>>>> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>>>>>>>> runTest
>>>>>>>> raise self.exc_val.with_traceback(self.tb)
>>>>>>>> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>>>>>>>> loadTestsFromName
>>>>>>>> addr.filename, addr.module)
>>>>>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>>>>>>>> importFromPath
>>>>>>>> return self.importFromDir(dir_path, fqname)
>>>>>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>>>>>>>> importFromDir
>>>>>>>> mod = load_module(part_fqname, fh, filename, desc)
>>>>>>>> /usr/local/lib/python3.5/imp.py line 245 in load_module
>>>>>>>> return load_package(name, filename)
>>>>>>>> /usr/local/lib/python3.5/imp.py line 217 in load_package
>>>>>>>> return _load(spec)
>>>>>>>> <frozen importlib._bootstrap> line 693 in _load
>>>>>>>>
>>>>>>>> <frozen importlib._bootstrap> line 673 in _load_unlocked
>>>>>>>>
>>>>>>>> <frozen importlib._bootstrap_external> line 697 in exec_module
>>>>>>>>
>>>>>>>> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>>>>>>>>
>>>>>>>> tests/__init__.py line 25 in <module>
>>>>>>>> from .configuration import *
>>>>>>>> tests/configuration.py line 28 in <module>
>>>>>>>> from airflow import configuration
>>>>>>>> airflow/__init__.py line 35 in <module>
>>>>>>>> from airflow import configuration as conf
>>>>>>>> airflow/configuration.py line 106 in <module>
>>>>>>>> DEFAULT_CONFIG = f.read()
>>>>>>>> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>>>>>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>>>>>>> 21082: ordinal not in range(128)
>>>>>>>>
>>>>>>>> [error] 100.00% nose.failure.Failure.runTest: 0.0004s
>>>>>>>>
>>>>>>>> ------------------------------------------------------------
>> -----------------
>>>>>>>> 2 tests run in 0.060 seconds.
>>>>>>>> 2 errors (0 tests passed)
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Carl Johan Gustavsson
>>>>>>>>
>>>>>>>> On 11 July 2018 at 23:24:01, Bolke de Bruin (bdbruin@gmail.com)
>> wrote:
>>>>>>>>
>>>>>>>> Hi Carl,
>>>>>>>>
>>>>>>>> That is not a real check, ie. Does Airflow have the same issue clean
>>>>>>>> install on 3.5? Travis’ tests run on 3.5.
>>>>>>>>
>>>>>>>> B.
>>>>>>>>
>>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>>>
>>>>>>>>> Op 10 jul. 2018 om 15:10 heeft Carl Johan Gustavsson <
>>>>>>>> carl.j.gustavsson@gmail.com> het volgende geschreven:
>>>>>>>>>
>>>>>>>>> Hi Bolke,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all but
>> 3.7
>>>>>>>> breaks
>>>>>>>>>
>>>>>>>>> Quick repro:
>>>>>>>>>
>>>>>>>>> ➜ ~ pyenv local 3.5.5
>>>>>>>>> ➜ ~ locale
>>>>>>>>> LANG=
>>>>>>>>> LC_COLLATE="C"
>>>>>>>>> LC_CTYPE="C"
>>>>>>>>> LC_MESSAGES="C"
>>>>>>>>> LC_MONETARY="C"
>>>>>>>>> LC_NUMERIC="C"
>>>>>>>>> LC_TIME="C"
>>>>>>>>> LC_ALL=
>>>>>>>>> ➜ ~ cat testweird.txt
>>>>>>>>> ’
>>>>>>>>> ➜ ~ python
>>>>>>>>> Python 3.5.5 (default, Jul 7 2018, 17:00:56)
>>>>>>>>> [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on
>> darwin
>>>>>>>>> Type "help", "copyright", "credits" or "license" for more
>> information.
>>>>>>>>>>>> open('testweird.txt').read()
>>>>>>>>> Traceback (most recent call last):
>>>>>>>>> File "<stdin>", line 1, in <module>
>>>>>>>>> File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/
>> ascii.py",
>>>>>>>> line 26, in decode
>>>>>>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in
>> position 0:
>>>>>>>> ordinal not in range(128)
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Maybe not a blocking change but it is a breaking change from 1.9 I
>> guess.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> / Carl Johan
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbruin@gmail.com)
>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Carl,
>>>>>>>>>>
>>>>>>>>>> Did you this on python 3.5 as well? 3.6 is not an officially
>> supported
>>>>>>>> (yet). As a workaround is available I won’t consider this blocking
>> btw.
>>>>>>>>>>
>>>>>>>>>> Bolke
>>>>>>>>>>
>>>>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>>>>>
>>>>>>>>>> Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson <
>>>>>>>> carl.j.gustavsson@gmail.com> het volgende geschreven:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> First of all, thank you for all the work with the release
>> management.
>>>>>>>>>>>
>>>>>>>>>>> I ran in to a weird issue testing the RC1, running under Python
>> 3.6.0 /
>>>>>>>> Ubuntu 14.04.5, upgrading from a master build from February.
>>>>>>>>>>>
>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>> Traceback
>>>>>>>> (most recent call last):
>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>>>>>> "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in
>> <module>
>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from
>> airflow
>>>>>>>> import configuration
>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>>>>>>
>>>>>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-
>> packages/airflow/__init__.py",
>>>>>>>> line 35, in <module>
>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from
>> airflow
>>>>>>>> import configuration as conf
>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>>>>>>
>>>>>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-
>> packages/airflow/configuration.py",
>>>>>>>> line 106, in <module>
>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>>>>>>>> DEFAULT_CONFIG = f.read()
>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>>>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py",
>> line
>>>>>>>> 26, in decode
>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return
>>>>>>>> codecs.ascii_decode(input, self.errors)[0]
>>>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>>>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in
>> position 20770:
>>>>>>>> ordinal not in range(128)
>>>>>>>>>>>
>>>>>>>>>>> Removing the ’ on
>>>>>>>>
>>>>>>>> https://github.com/apache/incubator-airflow/blob/master/
>> airflow/config_templates/default_airflow.cfg#L613
>>>>>>>> solved the issue for me, and digging a bit deeper it seems Airflow
>> now
>>>>>>>> requires setting LC_ALL=en_US.UTF-8 in the environment or similar
>> to force
>>>>>>>> Python to read the file as utf-8 and not ascii. (I think this was
>> changed
>>>>>>>> in to default to utf-8 in Python 3.7).
>>>>>>>>>>>
>>>>>>>>>>> I see 3 solutions for this
>>>>>>>>>>> 1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or
>>>>>>>> similar.
>>>>>>>>>>> 2. Change the default config file to not contain non-ascii
>> characters.
>>>>>>>>>>> 3. Always read the file as unicode regardless of the LC_ALL
>>>>>>>> environment, by the encoding='utf-8’ parameter to open().
>>>>>>>>>>>
>>>>>>>>>>> I think 3 is the best solution, and I can prepare a PR for that
>> if
>>>>>>>> necessary .
>>>>>>>>>>>
>>>>>>>>>>> I guess this counts as -1 (non-binding)
>>>>>>>>>>>
>>>>>>>>>>> All the best
>>>>>>>>>>>
>>>>>>>>>>> Carl Johan
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbruin@gmail.com)
>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>
>>>>>>>>>>>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on
>> the
>>>>>>>> release,
>>>>>>>>>>>> which will last for 72 hours. Consider this my (binding) +1.
>>>>>>>>>>>>
>>>>>>>>>>>> Airflow 1.10.0 RC 1 is available at:
>>>>>>>>>>>>
>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
>> 10.0rc1/ <
>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/
>>>
>>>>>>>>>>>>
>>>>>>>>>>>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source
>> release
>>>>>>>> that
>>>>>>>>>>>> comes with INSTALL instructions.
>>>>>>>>>>>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary
>> Python
>>>>>>>> "sdist"
>>>>>>>>>>>> release.
>>>>>>>>>>>>
>>>>>>>>>>>> Public keys are available at:
>>>>>>>>>>>>
>>>>>>>>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>>>>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/>
>>>>>>>>>>>>
>>>>>>>>>>>> The amount of JIRAs fixed is over 700. Please have a look at the
>>>>>>>> changelog.
>>>>>>>>>>>>
>>>>>>>>>>>> Please note that the version number excludes the `rcX` string
>> as well
>>>>>>>>>>>> as the "+incubating" string, so it's now simply 1.10.0. This
>> will
>>>>>>>> allow us
>>>>>>>>>>>> to rename the artifact without modifying the artifact checksums
>> when
>>>>>>>> we
>>>>>>>>>>>> actually release.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> Bolke
>>>>>>>>
>>>>
>>>
>>
>>
Re: [VOTE] Airflow 1.10.0rc1
Posted by James Meickle <jm...@quantopian.com.INVALID>.
Looking at that diff, it seems like the function as a whole needs some
love, even if that commit were reverted. The use of os.walk means it's
going to crawl the entire tree every time, and use accumulated patterns to
check against each file in each folder. The behavior should be to use
patterns to also exclude entire folders from consideration, which would
likely dramatically speed up the function for most use cases.
On Thu, Jul 12, 2018 at 9:14 AM, Ash Berlin-Taylor <
ash_airflowlist@firemirror.com> wrote:
> I wasn't paying enough attention. I can reproduce it.
>
> I'm not sure it was intentional but the bug was introduced in
> https://github.com/apache/incubator-airflow/pull/3171 <
> https://github.com/apache/incubator-airflow/pull/3171>
>
> I'd rather not release with a regression since 1.9, so updating my vote to
> -0.5 (binding), but not blocking/vetoing release if others are okay with it.
>
> -ash
>
> > On 12 Jul 2018, at 13:46, Ash Berlin-Taylor <ash_airflowlist@firemirror.
> com> wrote:
> >
> > That said I can't reproduce it myself (I will take discussion to the
> Jira ticket)
> >
> > -ash
> >
> >> On 12 Jul 2018, at 13:36, Ash Berlin-Taylor <
> ash_airflowlist@firemirror.com> wrote:
> >>
> >> Possible blocking regression here - .airflowignore doesn't seem to be
> respected anymore
> >>
> >> https://issues.apache.org/jira/browse/AIRFLOW-2729 <
> https://issues.apache.org/jira/browse/AIRFLOW-2729>
> >>
> >> -ash
> >>> On 12 Jul 2018, at 08:45, Bolke de Bruin <bd...@gmail.com> wrote:
> >>>
> >>> Hi Jakob,
> >>>
> >>> Thanks. We do include an INSTALL document that explains how to well
> install airflow and is quite a standard location for install instructions.
> Or did the install instruction needed to be included in the Vote?
> >>>
> >>> B.
> >>>
> >>>
> >>>
> >>> Verstuurd vanaf mijn iPad
> >>>
> >>>> Op 12 jul. 2018 om 07:54 heeft Jakob Homan <jg...@gmail.com> het
> volgende geschreven:
> >>>>
> >>>> +1 (binding)
> >>>>
> >>>> * Sigs look good
> >>>> * Artifact has incubating in name
> >>>> * LICENSE/NOTICE/DISCLAIMER look good
> >>>> - nit: DISCLAIMER is not word wrapped to 80 chars as I've seen in
> >>>> other projects.
> >>>> * Nit: Last release Sebb called out "Copyright 2016 and onwards" in
> >>>> NOTICE as being imprecise. This language remains.
> >>>> * Spot check on license headers looks ok.
> >>>> * Nit: Last release there was a request for instructions on how to
> >>>> build the code. This isn't included.
> >>>>
> >>>> -Jakob
> >>>>
> >>>>> On 11 July 2018 at 19:50, Sid Anand <sa...@apache.org> wrote:
> >>>>> FYI!
> >>>>> I just installed the release candidate. The first thing I noticed is
> a
> >>>>> missing tool tip for the Null State in the Recent Tasks column on the
> >>>>> landing page. Since the null globe is new to this UI, users will
> likely
> >>>>> hover over it to inquire what it means... and will be left wanting.
> Of
> >>>>> course, they could click on the globe, which will take them to
> >>>>> http://localhost:8080/admin/taskinstance/?flt1_dag_id_
> equals=example_bash_operator&flt2_state_equals=null,
> >>>>> which will always show an empty list, leaving them a bit more
> confused.
> >>>>>
> >>>>> -s
> >>>>>
> >>>>> On Wed, Jul 11, 2018 at 3:13 PM Carl Johan Gustavsson
> >>>>> <ca...@tictail.com.invalid> wrote:
> >>>>>
> >>>>>> Hi Bolke,
> >>>>>>
> >>>>>> (Switching email to avoid moderation on my emails.)
> >>>>>>
> >>>>>> The normal Airflow test suite does not fail as it uses a LC_ALL set
> to
> >>>>>> utf-8.
> >>>>>>
> >>>>>> I think it is a proper test though, it is a minimal reproducible
> version of
> >>>>>> the code that fails. And the only difference in behaviour is at 3.7
> which
> >>>>>> we don’t support anyway so I’m fairly sure it is broken for all
> supported
> >>>>>> Python 3 versions.
> >>>>>>
> >>>>>> I now tried running the tests in docker using 3.5 with the
> LC_ALL/LANG
> >>>>>> unset and I see the same failure.
> >>>>>>
> >>>>>> I don’t think this is a big thing though and we could release it
> without
> >>>>>> the fix I made. I think most people run it with a sane LC_ALL, but
> >>>>>> apparently we didn’t.
> >>>>>> Here’s the log for the test:
> >>>>>>
> >>>>>>> docker run -t -i -v `pwd`:/airflow/ python:3.5 bash
> >>>>>> root@b99b297df111:/# locale
> >>>>>> LANG=C.UTF-8
> >>>>>> LANGUAGE=
> >>>>>> LC_CTYPE="C.UTF-8"
> >>>>>> LC_NUMERIC="C.UTF-8"
> >>>>>> LC_TIME="C.UTF-8"
> >>>>>> LC_COLLATE="C.UTF-8"
> >>>>>> LC_MONETARY="C.UTF-8"
> >>>>>> LC_MESSAGES="C.UTF-8"
> >>>>>> LC_PAPER="C.UTF-8"
> >>>>>> LC_NAME="C.UTF-8"
> >>>>>> LC_ADDRESS="C.UTF-8"
> >>>>>> LC_TELEPHONE="C.UTF-8"
> >>>>>> LC_MEASUREMENT="C.UTF-8"
> >>>>>> LC_IDENTIFICATION="C.UTF-8"
> >>>>>> LC_ALL=
> >>>>>>> unset LANG
> >>>>>> root@b99b297df111:/# locale
> >>>>>> LANG=
> >>>>>> LANGUAGE=
> >>>>>> LC_CTYPE="POSIX"
> >>>>>> LC_NUMERIC="POSIX"
> >>>>>> LC_TIME="POSIX"
> >>>>>> LC_COLLATE="POSIX"
> >>>>>> LC_MONETARY="POSIX"
> >>>>>> LC_MESSAGES="POSIX"
> >>>>>> LC_PAPER="POSIX"
> >>>>>> LC_NAME="POSIX"
> >>>>>> LC_ADDRESS="POSIX"
> >>>>>> LC_TELEPHONE="POSIX"
> >>>>>> LC_MEASUREMENT="POSIX"
> >>>>>> LC_IDENTIFICATION="POSIX"
> >>>>>> LC_ALL=
> >>>>>> root@b99b297df111:/# pip install -e .[devel]
> >>>>>> root@b99b297df111:/airflow# ./run_unit_tests.sh
> >>>>>> + export AIRFLOW_HOME=/root/airflow
> >>>>>> + AIRFLOW_HOME=/root/airflow
> >>>>>> + export AIRFLOW__CORE__UNIT_TEST_MODE=True
> >>>>>> + AIRFLOW__CORE__UNIT_TEST_MODE=True
> >>>>>> + export AIRFLOW__TESTSECTION__TESTKEY=testvalue
> >>>>>> + AIRFLOW__TESTSECTION__TESTKEY=testvalue
> >>>>>> + export AIRFLOW_USE_NEW_IMPORTS=1
> >>>>>> + AIRFLOW_USE_NEW_IMPORTS=1
> >>>>>> +++ dirname ./run_unit_tests.sh
> >>>>>> ++ cd .
> >>>>>> ++ pwd
> >>>>>> + DIR=/airflow
> >>>>>> + export PYTHONPATH=:/airflow/tests/test_utils
> >>>>>> + PYTHONPATH=:/airflow/tests/test_utils
> >>>>>> + nose_args=
> >>>>>> + which airflow
> >>>>>> + echo 'Initializing the DB'
> >>>>>> Initializing the DB
> >>>>>> + airflow resetdb
> >>>>>> + yes
> >>>>>> Traceback (most recent call last):
> >>>>>> File "/usr/local/bin/airflow", line 6, in <module>
> >>>>>> exec(compile(open(__file__).read(), __file__, 'exec'))
> >>>>>> File "/airflow/airflow/bin/airflow", line 21, in <module>
> >>>>>> from airflow import configuration
> >>>>>> File "/airflow/airflow/__init__.py", line 35, in <module>
> >>>>>> from airflow import configuration as conf
> >>>>>> File "/airflow/airflow/configuration.py", line 106, in <module>
> >>>>>> DEFAULT_CONFIG = f.read()
> >>>>>> File "/usr/local/lib/python3.5/encodings/ascii.py", line 26, in
> decode
> >>>>>> return codecs.ascii_decode(input, self.errors)[0]
> >>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in
> position 21082:
> >>>>>> ordinal not in range(128)
> >>>>>> + '[' '' ']'
> >>>>>> + '[' -z '' ']'
> >>>>>> + nose_args='--with-coverage --cover-erase --cover-html
> >>>>>> --cover-package=airflow --cover-html-dir=airflow/www/
> static/coverage
> >>>>>> --with-ignore-docstrings --rednose --with-timer -s
> -v
> >>>>>> --logging-level=DEBUG '
> >>>>>> + echo 'Starting the unit tests with the following nose arguments:
> >>>>>> --with-coverage' --cover-erase --cover-html --cover-package=airflow
> >>>>>> --cover-html-dir=airflow/www/static/coverage
> --with-ignore-docstrings
> >>>>>> --rednose --with-timer -s -v --logging-level=DEBUG
> >>>>>> Starting the unit tests with the following nose arguments:
> --with-coverage
> >>>>>> --cover-erase --cover-html --cover-package=airflow
> >>>>>> --cover-html-dir=airflow/www/static/coverage
> --with-ignore-docstrings
> >>>>>> --rednose --with-timer -s -v --logging-level=DEBUG
> >>>>>> + nosetests --with-coverage --cover-erase --cover-html
> >>>>>> --cover-package=airflow --cover-html-dir=airflow/www/
> static/coverage
> >>>>>> --with-ignore-docstrings --rednose --with-timer -s -v
> --logging-level=DEBUG
> >>>>>> nose.plugins.cover: ERROR: Coverage not available: unable to import
> >>>>>> coverage module
> >>>>>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
> >>>>>> position 21082: ordinal not in range(128)) ... ERROR
> >>>>>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
> >>>>>> position 21082: ordinal not in range(128)) ... ERROR
> >>>>>> ============================================================
> ==========
> >>>>>> 1) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode
> byte 0xe2
> >>>>>> in position 21082: ordinal not in range(128))
> >>>>>> ------------------------------------------------------------
> ----------
> >>>>>> Traceback (most recent call last):
> >>>>>> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
> >>>>>> runTest
> >>>>>> raise self.exc_val.with_traceback(self.tb)
> >>>>>> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
> >>>>>> loadTestsFromName
> >>>>>> addr.filename, addr.module)
> >>>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
> >>>>>> importFromPath
> >>>>>> return self.importFromDir(dir_path, fqname)
> >>>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
> >>>>>> importFromDir
> >>>>>> mod = load_module(part_fqname, fh, filename, desc)
> >>>>>> /usr/local/lib/python3.5/imp.py line 245 in load_module
> >>>>>> return load_package(name, filename)
> >>>>>> /usr/local/lib/python3.5/imp.py line 217 in load_package
> >>>>>> return _load(spec)
> >>>>>> <frozen importlib._bootstrap> line 693 in _load
> >>>>>>
> >>>>>> <frozen importlib._bootstrap> line 673 in _load_unlocked
> >>>>>>
> >>>>>> <frozen importlib._bootstrap_external> line 697 in exec_module
> >>>>>>
> >>>>>> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
> >>>>>>
> >>>>>> airflow/__init__.py line 35 in <module>
> >>>>>> from airflow import configuration as conf
> >>>>>> airflow/configuration.py line 106 in <module>
> >>>>>> DEFAULT_CONFIG = f.read()
> >>>>>> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
> >>>>>> return codecs.ascii_decode(input, self.errors)[0]
> >>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
> >>>>>> 21082: ordinal not in range(128)
> >>>>>> ============================================================
> ==========
> >>>>>> 2) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode
> byte 0xe2
> >>>>>> in position 21082: ordinal not in range(128))
> >>>>>> ------------------------------------------------------------
> ----------
> >>>>>> Traceback (most recent call last):
> >>>>>> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
> >>>>>> runTest
> >>>>>> raise self.exc_val.with_traceback(self.tb)
> >>>>>> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
> >>>>>> loadTestsFromName
> >>>>>> addr.filename, addr.module)
> >>>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
> >>>>>> importFromPath
> >>>>>> return self.importFromDir(dir_path, fqname)
> >>>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
> >>>>>> importFromDir
> >>>>>> mod = load_module(part_fqname, fh, filename, desc)
> >>>>>> /usr/local/lib/python3.5/imp.py line 245 in load_module
> >>>>>> return load_package(name, filename)
> >>>>>> /usr/local/lib/python3.5/imp.py line 217 in load_package
> >>>>>> return _load(spec)
> >>>>>> <frozen importlib._bootstrap> line 693 in _load
> >>>>>>
> >>>>>> <frozen importlib._bootstrap> line 673 in _load_unlocked
> >>>>>>
> >>>>>> <frozen importlib._bootstrap_external> line 697 in exec_module
> >>>>>>
> >>>>>> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
> >>>>>>
> >>>>>> tests/__init__.py line 25 in <module>
> >>>>>> from .configuration import *
> >>>>>> tests/configuration.py line 28 in <module>
> >>>>>> from airflow import configuration
> >>>>>> airflow/__init__.py line 35 in <module>
> >>>>>> from airflow import configuration as conf
> >>>>>> airflow/configuration.py line 106 in <module>
> >>>>>> DEFAULT_CONFIG = f.read()
> >>>>>> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
> >>>>>> return codecs.ascii_decode(input, self.errors)[0]
> >>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
> >>>>>> 21082: ordinal not in range(128)
> >>>>>>
> >>>>>> [error] 100.00% nose.failure.Failure.runTest: 0.0004s
> >>>>>>
> >>>>>> ------------------------------------------------------------
> -----------------
> >>>>>> 2 tests run in 0.060 seconds.
> >>>>>> 2 errors (0 tests passed)
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Carl Johan Gustavsson
> >>>>>>
> >>>>>> On 11 July 2018 at 23:24:01, Bolke de Bruin (bdbruin@gmail.com)
> wrote:
> >>>>>>
> >>>>>> Hi Carl,
> >>>>>>
> >>>>>> That is not a real check, ie. Does Airflow have the same issue clean
> >>>>>> install on 3.5? Travis’ tests run on 3.5.
> >>>>>>
> >>>>>> B.
> >>>>>>
> >>>>>> Verstuurd vanaf mijn iPad
> >>>>>>
> >>>>>>> Op 10 jul. 2018 om 15:10 heeft Carl Johan Gustavsson <
> >>>>>> carl.j.gustavsson@gmail.com> het volgende geschreven:
> >>>>>>>
> >>>>>>> Hi Bolke,
> >>>>>>>
> >>>>>>>
> >>>>>>> I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all but
> 3.7
> >>>>>> breaks
> >>>>>>>
> >>>>>>> Quick repro:
> >>>>>>>
> >>>>>>> ➜ ~ pyenv local 3.5.5
> >>>>>>> ➜ ~ locale
> >>>>>>> LANG=
> >>>>>>> LC_COLLATE="C"
> >>>>>>> LC_CTYPE="C"
> >>>>>>> LC_MESSAGES="C"
> >>>>>>> LC_MONETARY="C"
> >>>>>>> LC_NUMERIC="C"
> >>>>>>> LC_TIME="C"
> >>>>>>> LC_ALL=
> >>>>>>> ➜ ~ cat testweird.txt
> >>>>>>> ’
> >>>>>>> ➜ ~ python
> >>>>>>> Python 3.5.5 (default, Jul 7 2018, 17:00:56)
> >>>>>>> [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on
> darwin
> >>>>>>> Type "help", "copyright", "credits" or "license" for more
> information.
> >>>>>>>>>> open('testweird.txt').read()
> >>>>>>> Traceback (most recent call last):
> >>>>>>> File "<stdin>", line 1, in <module>
> >>>>>>> File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/
> ascii.py",
> >>>>>> line 26, in decode
> >>>>>>> return codecs.ascii_decode(input, self.errors)[0]
> >>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in
> position 0:
> >>>>>> ordinal not in range(128)
> >>>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Maybe not a blocking change but it is a breaking change from 1.9 I
> guess.
> >>>>>>>
> >>>>>>>
> >>>>>>> / Carl Johan
> >>>>>>>
> >>>>>>>
> >>>>>>>> On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbruin@gmail.com)
> wrote:
> >>>>>>>>
> >>>>>>>> Hi Carl,
> >>>>>>>>
> >>>>>>>> Did you this on python 3.5 as well? 3.6 is not an officially
> supported
> >>>>>> (yet). As a workaround is available I won’t consider this blocking
> btw.
> >>>>>>>>
> >>>>>>>> Bolke
> >>>>>>>>
> >>>>>>>> Verstuurd vanaf mijn iPad
> >>>>>>>>
> >>>>>>>> Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson <
> >>>>>> carl.j.gustavsson@gmail.com> het volgende geschreven:
> >>>>>>>>
> >>>>>>>>> Hi,
> >>>>>>>>>
> >>>>>>>>> First of all, thank you for all the work with the release
> management.
> >>>>>>>>>
> >>>>>>>>> I ran in to a weird issue testing the RC1, running under Python
> 3.6.0 /
> >>>>>> Ubuntu 14.04.5, upgrading from a master build from February.
> >>>>>>>>>
> >>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
> Traceback
> >>>>>> (most recent call last):
> >>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> >>>>>> "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in
> <module>
> >>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from
> airflow
> >>>>>> import configuration
> >>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> >>>>>>
> >>>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-
> packages/airflow/__init__.py",
> >>>>>> line 35, in <module>
> >>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from
> airflow
> >>>>>> import configuration as conf
> >>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> >>>>>>
> >>>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-
> packages/airflow/configuration.py",
> >>>>>> line 106, in <module>
> >>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
> >>>>>> DEFAULT_CONFIG = f.read()
> >>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> >>>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py",
> line
> >>>>>> 26, in decode
> >>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return
> >>>>>> codecs.ascii_decode(input, self.errors)[0]
> >>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
> >>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in
> position 20770:
> >>>>>> ordinal not in range(128)
> >>>>>>>>>
> >>>>>>>>> Removing the ’ on
> >>>>>>
> >>>>>> https://github.com/apache/incubator-airflow/blob/master/
> airflow/config_templates/default_airflow.cfg#L613
> >>>>>> solved the issue for me, and digging a bit deeper it seems Airflow
> now
> >>>>>> requires setting LC_ALL=en_US.UTF-8 in the environment or similar
> to force
> >>>>>> Python to read the file as utf-8 and not ascii. (I think this was
> changed
> >>>>>> in to default to utf-8 in Python 3.7).
> >>>>>>>>>
> >>>>>>>>> I see 3 solutions for this
> >>>>>>>>> 1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or
> >>>>>> similar.
> >>>>>>>>> 2. Change the default config file to not contain non-ascii
> characters.
> >>>>>>>>> 3. Always read the file as unicode regardless of the LC_ALL
> >>>>>> environment, by the encoding='utf-8’ parameter to open().
> >>>>>>>>>
> >>>>>>>>> I think 3 is the best solution, and I can prepare a PR for that
> if
> >>>>>> necessary .
> >>>>>>>>>
> >>>>>>>>> I guess this counts as -1 (non-binding)
> >>>>>>>>>
> >>>>>>>>> All the best
> >>>>>>>>>
> >>>>>>>>> Carl Johan
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbruin@gmail.com)
> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hey all,
> >>>>>>>>>>
> >>>>>>>>>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on
> the
> >>>>>> release,
> >>>>>>>>>> which will last for 72 hours. Consider this my (binding) +1.
> >>>>>>>>>>
> >>>>>>>>>> Airflow 1.10.0 RC 1 is available at:
> >>>>>>>>>>
> >>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
> 10.0rc1/ <
> >>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/
> >
> >>>>>>>>>>
> >>>>>>>>>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source
> release
> >>>>>> that
> >>>>>>>>>> comes with INSTALL instructions.
> >>>>>>>>>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary
> Python
> >>>>>> "sdist"
> >>>>>>>>>> release.
> >>>>>>>>>>
> >>>>>>>>>> Public keys are available at:
> >>>>>>>>>>
> >>>>>>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
> >>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/>
> >>>>>>>>>>
> >>>>>>>>>> The amount of JIRAs fixed is over 700. Please have a look at the
> >>>>>> changelog.
> >>>>>>>>>>
> >>>>>>>>>> Please note that the version number excludes the `rcX` string
> as well
> >>>>>>>>>> as the "+incubating" string, so it's now simply 1.10.0. This
> will
> >>>>>> allow us
> >>>>>>>>>> to rename the artifact without modifying the artifact checksums
> when
> >>>>>> we
> >>>>>>>>>> actually release.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Cheers,
> >>>>>>>>>> Bolke
> >>>>>>
> >>
> >
>
>
Re: [VOTE] Airflow 1.10.0rc1
Posted by Ash Berlin-Taylor <as...@firemirror.com>.
I wasn't paying enough attention. I can reproduce it.
I'm not sure it was intentional but the bug was introduced in https://github.com/apache/incubator-airflow/pull/3171 <https://github.com/apache/incubator-airflow/pull/3171>
I'd rather not release with a regression since 1.9, so updating my vote to -0.5 (binding), but not blocking/vetoing release if others are okay with it.
-ash
> On 12 Jul 2018, at 13:46, Ash Berlin-Taylor <as...@firemirror.com> wrote:
>
> That said I can't reproduce it myself (I will take discussion to the Jira ticket)
>
> -ash
>
>> On 12 Jul 2018, at 13:36, Ash Berlin-Taylor <as...@firemirror.com> wrote:
>>
>> Possible blocking regression here - .airflowignore doesn't seem to be respected anymore
>>
>> https://issues.apache.org/jira/browse/AIRFLOW-2729 <https://issues.apache.org/jira/browse/AIRFLOW-2729>
>>
>> -ash
>>> On 12 Jul 2018, at 08:45, Bolke de Bruin <bd...@gmail.com> wrote:
>>>
>>> Hi Jakob,
>>>
>>> Thanks. We do include an INSTALL document that explains how to well install airflow and is quite a standard location for install instructions. Or did the install instruction needed to be included in the Vote?
>>>
>>> B.
>>>
>>>
>>>
>>> Verstuurd vanaf mijn iPad
>>>
>>>> Op 12 jul. 2018 om 07:54 heeft Jakob Homan <jg...@gmail.com> het volgende geschreven:
>>>>
>>>> +1 (binding)
>>>>
>>>> * Sigs look good
>>>> * Artifact has incubating in name
>>>> * LICENSE/NOTICE/DISCLAIMER look good
>>>> - nit: DISCLAIMER is not word wrapped to 80 chars as I've seen in
>>>> other projects.
>>>> * Nit: Last release Sebb called out "Copyright 2016 and onwards" in
>>>> NOTICE as being imprecise. This language remains.
>>>> * Spot check on license headers looks ok.
>>>> * Nit: Last release there was a request for instructions on how to
>>>> build the code. This isn't included.
>>>>
>>>> -Jakob
>>>>
>>>>> On 11 July 2018 at 19:50, Sid Anand <sa...@apache.org> wrote:
>>>>> FYI!
>>>>> I just installed the release candidate. The first thing I noticed is a
>>>>> missing tool tip for the Null State in the Recent Tasks column on the
>>>>> landing page. Since the null globe is new to this UI, users will likely
>>>>> hover over it to inquire what it means... and will be left wanting. Of
>>>>> course, they could click on the globe, which will take them to
>>>>> http://localhost:8080/admin/taskinstance/?flt1_dag_id_equals=example_bash_operator&flt2_state_equals=null,
>>>>> which will always show an empty list, leaving them a bit more confused.
>>>>>
>>>>> -s
>>>>>
>>>>> On Wed, Jul 11, 2018 at 3:13 PM Carl Johan Gustavsson
>>>>> <ca...@tictail.com.invalid> wrote:
>>>>>
>>>>>> Hi Bolke,
>>>>>>
>>>>>> (Switching email to avoid moderation on my emails.)
>>>>>>
>>>>>> The normal Airflow test suite does not fail as it uses a LC_ALL set to
>>>>>> utf-8.
>>>>>>
>>>>>> I think it is a proper test though, it is a minimal reproducible version of
>>>>>> the code that fails. And the only difference in behaviour is at 3.7 which
>>>>>> we don’t support anyway so I’m fairly sure it is broken for all supported
>>>>>> Python 3 versions.
>>>>>>
>>>>>> I now tried running the tests in docker using 3.5 with the LC_ALL/LANG
>>>>>> unset and I see the same failure.
>>>>>>
>>>>>> I don’t think this is a big thing though and we could release it without
>>>>>> the fix I made. I think most people run it with a sane LC_ALL, but
>>>>>> apparently we didn’t.
>>>>>> Here’s the log for the test:
>>>>>>
>>>>>>> docker run -t -i -v `pwd`:/airflow/ python:3.5 bash
>>>>>> root@b99b297df111:/# locale
>>>>>> LANG=C.UTF-8
>>>>>> LANGUAGE=
>>>>>> LC_CTYPE="C.UTF-8"
>>>>>> LC_NUMERIC="C.UTF-8"
>>>>>> LC_TIME="C.UTF-8"
>>>>>> LC_COLLATE="C.UTF-8"
>>>>>> LC_MONETARY="C.UTF-8"
>>>>>> LC_MESSAGES="C.UTF-8"
>>>>>> LC_PAPER="C.UTF-8"
>>>>>> LC_NAME="C.UTF-8"
>>>>>> LC_ADDRESS="C.UTF-8"
>>>>>> LC_TELEPHONE="C.UTF-8"
>>>>>> LC_MEASUREMENT="C.UTF-8"
>>>>>> LC_IDENTIFICATION="C.UTF-8"
>>>>>> LC_ALL=
>>>>>>> unset LANG
>>>>>> root@b99b297df111:/# locale
>>>>>> LANG=
>>>>>> LANGUAGE=
>>>>>> LC_CTYPE="POSIX"
>>>>>> LC_NUMERIC="POSIX"
>>>>>> LC_TIME="POSIX"
>>>>>> LC_COLLATE="POSIX"
>>>>>> LC_MONETARY="POSIX"
>>>>>> LC_MESSAGES="POSIX"
>>>>>> LC_PAPER="POSIX"
>>>>>> LC_NAME="POSIX"
>>>>>> LC_ADDRESS="POSIX"
>>>>>> LC_TELEPHONE="POSIX"
>>>>>> LC_MEASUREMENT="POSIX"
>>>>>> LC_IDENTIFICATION="POSIX"
>>>>>> LC_ALL=
>>>>>> root@b99b297df111:/# pip install -e .[devel]
>>>>>> root@b99b297df111:/airflow# ./run_unit_tests.sh
>>>>>> + export AIRFLOW_HOME=/root/airflow
>>>>>> + AIRFLOW_HOME=/root/airflow
>>>>>> + export AIRFLOW__CORE__UNIT_TEST_MODE=True
>>>>>> + AIRFLOW__CORE__UNIT_TEST_MODE=True
>>>>>> + export AIRFLOW__TESTSECTION__TESTKEY=testvalue
>>>>>> + AIRFLOW__TESTSECTION__TESTKEY=testvalue
>>>>>> + export AIRFLOW_USE_NEW_IMPORTS=1
>>>>>> + AIRFLOW_USE_NEW_IMPORTS=1
>>>>>> +++ dirname ./run_unit_tests.sh
>>>>>> ++ cd .
>>>>>> ++ pwd
>>>>>> + DIR=/airflow
>>>>>> + export PYTHONPATH=:/airflow/tests/test_utils
>>>>>> + PYTHONPATH=:/airflow/tests/test_utils
>>>>>> + nose_args=
>>>>>> + which airflow
>>>>>> + echo 'Initializing the DB'
>>>>>> Initializing the DB
>>>>>> + airflow resetdb
>>>>>> + yes
>>>>>> Traceback (most recent call last):
>>>>>> File "/usr/local/bin/airflow", line 6, in <module>
>>>>>> exec(compile(open(__file__).read(), __file__, 'exec'))
>>>>>> File "/airflow/airflow/bin/airflow", line 21, in <module>
>>>>>> from airflow import configuration
>>>>>> File "/airflow/airflow/__init__.py", line 35, in <module>
>>>>>> from airflow import configuration as conf
>>>>>> File "/airflow/airflow/configuration.py", line 106, in <module>
>>>>>> DEFAULT_CONFIG = f.read()
>>>>>> File "/usr/local/lib/python3.5/encodings/ascii.py", line 26, in decode
>>>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 21082:
>>>>>> ordinal not in range(128)
>>>>>> + '[' '' ']'
>>>>>> + '[' -z '' ']'
>>>>>> + nose_args='--with-coverage --cover-erase --cover-html
>>>>>> --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
>>>>>> --with-ignore-docstrings --rednose --with-timer -s -v
>>>>>> --logging-level=DEBUG '
>>>>>> + echo 'Starting the unit tests with the following nose arguments:
>>>>>> --with-coverage' --cover-erase --cover-html --cover-package=airflow
>>>>>> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
>>>>>> --rednose --with-timer -s -v --logging-level=DEBUG
>>>>>> Starting the unit tests with the following nose arguments: --with-coverage
>>>>>> --cover-erase --cover-html --cover-package=airflow
>>>>>> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
>>>>>> --rednose --with-timer -s -v --logging-level=DEBUG
>>>>>> + nosetests --with-coverage --cover-erase --cover-html
>>>>>> --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
>>>>>> --with-ignore-docstrings --rednose --with-timer -s -v --logging-level=DEBUG
>>>>>> nose.plugins.cover: ERROR: Coverage not available: unable to import
>>>>>> coverage module
>>>>>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>>>>>> position 21082: ordinal not in range(128)) ... ERROR
>>>>>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>>>>>> position 21082: ordinal not in range(128)) ... ERROR
>>>>>> ======================================================================
>>>>>> 1) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
>>>>>> in position 21082: ordinal not in range(128))
>>>>>> ----------------------------------------------------------------------
>>>>>> Traceback (most recent call last):
>>>>>> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>>>>>> runTest
>>>>>> raise self.exc_val.with_traceback(self.tb)
>>>>>> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>>>>>> loadTestsFromName
>>>>>> addr.filename, addr.module)
>>>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>>>>>> importFromPath
>>>>>> return self.importFromDir(dir_path, fqname)
>>>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>>>>>> importFromDir
>>>>>> mod = load_module(part_fqname, fh, filename, desc)
>>>>>> /usr/local/lib/python3.5/imp.py line 245 in load_module
>>>>>> return load_package(name, filename)
>>>>>> /usr/local/lib/python3.5/imp.py line 217 in load_package
>>>>>> return _load(spec)
>>>>>> <frozen importlib._bootstrap> line 693 in _load
>>>>>>
>>>>>> <frozen importlib._bootstrap> line 673 in _load_unlocked
>>>>>>
>>>>>> <frozen importlib._bootstrap_external> line 697 in exec_module
>>>>>>
>>>>>> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>>>>>>
>>>>>> airflow/__init__.py line 35 in <module>
>>>>>> from airflow import configuration as conf
>>>>>> airflow/configuration.py line 106 in <module>
>>>>>> DEFAULT_CONFIG = f.read()
>>>>>> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>>>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>>>>> 21082: ordinal not in range(128)
>>>>>> ======================================================================
>>>>>> 2) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
>>>>>> in position 21082: ordinal not in range(128))
>>>>>> ----------------------------------------------------------------------
>>>>>> Traceback (most recent call last):
>>>>>> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>>>>>> runTest
>>>>>> raise self.exc_val.with_traceback(self.tb)
>>>>>> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>>>>>> loadTestsFromName
>>>>>> addr.filename, addr.module)
>>>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>>>>>> importFromPath
>>>>>> return self.importFromDir(dir_path, fqname)
>>>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>>>>>> importFromDir
>>>>>> mod = load_module(part_fqname, fh, filename, desc)
>>>>>> /usr/local/lib/python3.5/imp.py line 245 in load_module
>>>>>> return load_package(name, filename)
>>>>>> /usr/local/lib/python3.5/imp.py line 217 in load_package
>>>>>> return _load(spec)
>>>>>> <frozen importlib._bootstrap> line 693 in _load
>>>>>>
>>>>>> <frozen importlib._bootstrap> line 673 in _load_unlocked
>>>>>>
>>>>>> <frozen importlib._bootstrap_external> line 697 in exec_module
>>>>>>
>>>>>> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>>>>>>
>>>>>> tests/__init__.py line 25 in <module>
>>>>>> from .configuration import *
>>>>>> tests/configuration.py line 28 in <module>
>>>>>> from airflow import configuration
>>>>>> airflow/__init__.py line 35 in <module>
>>>>>> from airflow import configuration as conf
>>>>>> airflow/configuration.py line 106 in <module>
>>>>>> DEFAULT_CONFIG = f.read()
>>>>>> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>>>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>>>>> 21082: ordinal not in range(128)
>>>>>>
>>>>>> [error] 100.00% nose.failure.Failure.runTest: 0.0004s
>>>>>>
>>>>>> -----------------------------------------------------------------------------
>>>>>> 2 tests run in 0.060 seconds.
>>>>>> 2 errors (0 tests passed)
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Carl Johan Gustavsson
>>>>>>
>>>>>> On 11 July 2018 at 23:24:01, Bolke de Bruin (bdbruin@gmail.com) wrote:
>>>>>>
>>>>>> Hi Carl,
>>>>>>
>>>>>> That is not a real check, ie. Does Airflow have the same issue clean
>>>>>> install on 3.5? Travis’ tests run on 3.5.
>>>>>>
>>>>>> B.
>>>>>>
>>>>>> Verstuurd vanaf mijn iPad
>>>>>>
>>>>>>> Op 10 jul. 2018 om 15:10 heeft Carl Johan Gustavsson <
>>>>>> carl.j.gustavsson@gmail.com> het volgende geschreven:
>>>>>>>
>>>>>>> Hi Bolke,
>>>>>>>
>>>>>>>
>>>>>>> I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all but 3.7
>>>>>> breaks
>>>>>>>
>>>>>>> Quick repro:
>>>>>>>
>>>>>>> ➜ ~ pyenv local 3.5.5
>>>>>>> ➜ ~ locale
>>>>>>> LANG=
>>>>>>> LC_COLLATE="C"
>>>>>>> LC_CTYPE="C"
>>>>>>> LC_MESSAGES="C"
>>>>>>> LC_MONETARY="C"
>>>>>>> LC_NUMERIC="C"
>>>>>>> LC_TIME="C"
>>>>>>> LC_ALL=
>>>>>>> ➜ ~ cat testweird.txt
>>>>>>> ’
>>>>>>> ➜ ~ python
>>>>>>> Python 3.5.5 (default, Jul 7 2018, 17:00:56)
>>>>>>> [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
>>>>>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>>>>>> open('testweird.txt').read()
>>>>>>> Traceback (most recent call last):
>>>>>>> File "<stdin>", line 1, in <module>
>>>>>>> File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/ascii.py",
>>>>>> line 26, in decode
>>>>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0:
>>>>>> ordinal not in range(128)
>>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Maybe not a blocking change but it is a breaking change from 1.9 I guess.
>>>>>>>
>>>>>>>
>>>>>>> / Carl Johan
>>>>>>>
>>>>>>>
>>>>>>>> On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbruin@gmail.com) wrote:
>>>>>>>>
>>>>>>>> Hi Carl,
>>>>>>>>
>>>>>>>> Did you this on python 3.5 as well? 3.6 is not an officially supported
>>>>>> (yet). As a workaround is available I won’t consider this blocking btw.
>>>>>>>>
>>>>>>>> Bolke
>>>>>>>>
>>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>>>
>>>>>>>> Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson <
>>>>>> carl.j.gustavsson@gmail.com> het volgende geschreven:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> First of all, thank you for all the work with the release management.
>>>>>>>>>
>>>>>>>>> I ran in to a weird issue testing the RC1, running under Python 3.6.0 /
>>>>>> Ubuntu 14.04.5, upgrading from a master build from February.
>>>>>>>>>
>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback
>>>>>> (most recent call last):
>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>>>> "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in <module>
>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
>>>>>> import configuration
>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>>>>
>>>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/__init__.py",
>>>>>> line 35, in <module>
>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
>>>>>> import configuration as conf
>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>>>>
>>>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/configuration.py",
>>>>>> line 106, in <module>
>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>>>>>> DEFAULT_CONFIG = f.read()
>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py", line
>>>>>> 26, in decode
>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return
>>>>>> codecs.ascii_decode(input, self.errors)[0]
>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 20770:
>>>>>> ordinal not in range(128)
>>>>>>>>>
>>>>>>>>> Removing the ’ on
>>>>>>
>>>>>> https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L613
>>>>>> solved the issue for me, and digging a bit deeper it seems Airflow now
>>>>>> requires setting LC_ALL=en_US.UTF-8 in the environment or similar to force
>>>>>> Python to read the file as utf-8 and not ascii. (I think this was changed
>>>>>> in to default to utf-8 in Python 3.7).
>>>>>>>>>
>>>>>>>>> I see 3 solutions for this
>>>>>>>>> 1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or
>>>>>> similar.
>>>>>>>>> 2. Change the default config file to not contain non-ascii characters.
>>>>>>>>> 3. Always read the file as unicode regardless of the LC_ALL
>>>>>> environment, by the encoding='utf-8’ parameter to open().
>>>>>>>>>
>>>>>>>>> I think 3 is the best solution, and I can prepare a PR for that if
>>>>>> necessary .
>>>>>>>>>
>>>>>>>>> I guess this counts as -1 (non-binding)
>>>>>>>>>
>>>>>>>>> All the best
>>>>>>>>>
>>>>>>>>> Carl Johan
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbruin@gmail.com) wrote:
>>>>>>>>>>
>>>>>>>>>> Hey all,
>>>>>>>>>>
>>>>>>>>>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the
>>>>>> release,
>>>>>>>>>> which will last for 72 hours. Consider this my (binding) +1.
>>>>>>>>>>
>>>>>>>>>> Airflow 1.10.0 RC 1 is available at:
>>>>>>>>>>
>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <
>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
>>>>>>>>>>
>>>>>>>>>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release
>>>>>> that
>>>>>>>>>> comes with INSTALL instructions.
>>>>>>>>>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python
>>>>>> "sdist"
>>>>>>>>>> release.
>>>>>>>>>>
>>>>>>>>>> Public keys are available at:
>>>>>>>>>>
>>>>>>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/>
>>>>>>>>>>
>>>>>>>>>> The amount of JIRAs fixed is over 700. Please have a look at the
>>>>>> changelog.
>>>>>>>>>>
>>>>>>>>>> Please note that the version number excludes the `rcX` string as well
>>>>>>>>>> as the "+incubating" string, so it's now simply 1.10.0. This will
>>>>>> allow us
>>>>>>>>>> to rename the artifact without modifying the artifact checksums when
>>>>>> we
>>>>>>>>>> actually release.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Bolke
>>>>>>
>>
>
Re: [VOTE] Airflow 1.10.0rc1
Posted by Ash Berlin-Taylor <as...@firemirror.com>.
That said I can't reproduce it myself (I will take discussion to the Jira ticket)
-ash
> On 12 Jul 2018, at 13:36, Ash Berlin-Taylor <as...@firemirror.com> wrote:
>
> Possible blocking regression here - .airflowignore doesn't seem to be respected anymore
>
> https://issues.apache.org/jira/browse/AIRFLOW-2729 <https://issues.apache.org/jira/browse/AIRFLOW-2729>
>
> -ash
>> On 12 Jul 2018, at 08:45, Bolke de Bruin <bd...@gmail.com> wrote:
>>
>> Hi Jakob,
>>
>> Thanks. We do include an INSTALL document that explains how to well install airflow and is quite a standard location for install instructions. Or did the install instruction needed to be included in the Vote?
>>
>> B.
>>
>>
>>
>> Verstuurd vanaf mijn iPad
>>
>>> Op 12 jul. 2018 om 07:54 heeft Jakob Homan <jg...@gmail.com> het volgende geschreven:
>>>
>>> +1 (binding)
>>>
>>> * Sigs look good
>>> * Artifact has incubating in name
>>> * LICENSE/NOTICE/DISCLAIMER look good
>>> - nit: DISCLAIMER is not word wrapped to 80 chars as I've seen in
>>> other projects.
>>> * Nit: Last release Sebb called out "Copyright 2016 and onwards" in
>>> NOTICE as being imprecise. This language remains.
>>> * Spot check on license headers looks ok.
>>> * Nit: Last release there was a request for instructions on how to
>>> build the code. This isn't included.
>>>
>>> -Jakob
>>>
>>>> On 11 July 2018 at 19:50, Sid Anand <sa...@apache.org> wrote:
>>>> FYI!
>>>> I just installed the release candidate. The first thing I noticed is a
>>>> missing tool tip for the Null State in the Recent Tasks column on the
>>>> landing page. Since the null globe is new to this UI, users will likely
>>>> hover over it to inquire what it means... and will be left wanting. Of
>>>> course, they could click on the globe, which will take them to
>>>> http://localhost:8080/admin/taskinstance/?flt1_dag_id_equals=example_bash_operator&flt2_state_equals=null,
>>>> which will always show an empty list, leaving them a bit more confused.
>>>>
>>>> -s
>>>>
>>>> On Wed, Jul 11, 2018 at 3:13 PM Carl Johan Gustavsson
>>>> <ca...@tictail.com.invalid> wrote:
>>>>
>>>>> Hi Bolke,
>>>>>
>>>>> (Switching email to avoid moderation on my emails.)
>>>>>
>>>>> The normal Airflow test suite does not fail as it uses a LC_ALL set to
>>>>> utf-8.
>>>>>
>>>>> I think it is a proper test though, it is a minimal reproducible version of
>>>>> the code that fails. And the only difference in behaviour is at 3.7 which
>>>>> we don’t support anyway so I’m fairly sure it is broken for all supported
>>>>> Python 3 versions.
>>>>>
>>>>> I now tried running the tests in docker using 3.5 with the LC_ALL/LANG
>>>>> unset and I see the same failure.
>>>>>
>>>>> I don’t think this is a big thing though and we could release it without
>>>>> the fix I made. I think most people run it with a sane LC_ALL, but
>>>>> apparently we didn’t.
>>>>> Here’s the log for the test:
>>>>>
>>>>>> docker run -t -i -v `pwd`:/airflow/ python:3.5 bash
>>>>> root@b99b297df111:/# locale
>>>>> LANG=C.UTF-8
>>>>> LANGUAGE=
>>>>> LC_CTYPE="C.UTF-8"
>>>>> LC_NUMERIC="C.UTF-8"
>>>>> LC_TIME="C.UTF-8"
>>>>> LC_COLLATE="C.UTF-8"
>>>>> LC_MONETARY="C.UTF-8"
>>>>> LC_MESSAGES="C.UTF-8"
>>>>> LC_PAPER="C.UTF-8"
>>>>> LC_NAME="C.UTF-8"
>>>>> LC_ADDRESS="C.UTF-8"
>>>>> LC_TELEPHONE="C.UTF-8"
>>>>> LC_MEASUREMENT="C.UTF-8"
>>>>> LC_IDENTIFICATION="C.UTF-8"
>>>>> LC_ALL=
>>>>>> unset LANG
>>>>> root@b99b297df111:/# locale
>>>>> LANG=
>>>>> LANGUAGE=
>>>>> LC_CTYPE="POSIX"
>>>>> LC_NUMERIC="POSIX"
>>>>> LC_TIME="POSIX"
>>>>> LC_COLLATE="POSIX"
>>>>> LC_MONETARY="POSIX"
>>>>> LC_MESSAGES="POSIX"
>>>>> LC_PAPER="POSIX"
>>>>> LC_NAME="POSIX"
>>>>> LC_ADDRESS="POSIX"
>>>>> LC_TELEPHONE="POSIX"
>>>>> LC_MEASUREMENT="POSIX"
>>>>> LC_IDENTIFICATION="POSIX"
>>>>> LC_ALL=
>>>>> root@b99b297df111:/# pip install -e .[devel]
>>>>> root@b99b297df111:/airflow# ./run_unit_tests.sh
>>>>> + export AIRFLOW_HOME=/root/airflow
>>>>> + AIRFLOW_HOME=/root/airflow
>>>>> + export AIRFLOW__CORE__UNIT_TEST_MODE=True
>>>>> + AIRFLOW__CORE__UNIT_TEST_MODE=True
>>>>> + export AIRFLOW__TESTSECTION__TESTKEY=testvalue
>>>>> + AIRFLOW__TESTSECTION__TESTKEY=testvalue
>>>>> + export AIRFLOW_USE_NEW_IMPORTS=1
>>>>> + AIRFLOW_USE_NEW_IMPORTS=1
>>>>> +++ dirname ./run_unit_tests.sh
>>>>> ++ cd .
>>>>> ++ pwd
>>>>> + DIR=/airflow
>>>>> + export PYTHONPATH=:/airflow/tests/test_utils
>>>>> + PYTHONPATH=:/airflow/tests/test_utils
>>>>> + nose_args=
>>>>> + which airflow
>>>>> + echo 'Initializing the DB'
>>>>> Initializing the DB
>>>>> + airflow resetdb
>>>>> + yes
>>>>> Traceback (most recent call last):
>>>>> File "/usr/local/bin/airflow", line 6, in <module>
>>>>> exec(compile(open(__file__).read(), __file__, 'exec'))
>>>>> File "/airflow/airflow/bin/airflow", line 21, in <module>
>>>>> from airflow import configuration
>>>>> File "/airflow/airflow/__init__.py", line 35, in <module>
>>>>> from airflow import configuration as conf
>>>>> File "/airflow/airflow/configuration.py", line 106, in <module>
>>>>> DEFAULT_CONFIG = f.read()
>>>>> File "/usr/local/lib/python3.5/encodings/ascii.py", line 26, in decode
>>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 21082:
>>>>> ordinal not in range(128)
>>>>> + '[' '' ']'
>>>>> + '[' -z '' ']'
>>>>> + nose_args='--with-coverage --cover-erase --cover-html
>>>>> --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
>>>>> --with-ignore-docstrings --rednose --with-timer -s -v
>>>>> --logging-level=DEBUG '
>>>>> + echo 'Starting the unit tests with the following nose arguments:
>>>>> --with-coverage' --cover-erase --cover-html --cover-package=airflow
>>>>> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
>>>>> --rednose --with-timer -s -v --logging-level=DEBUG
>>>>> Starting the unit tests with the following nose arguments: --with-coverage
>>>>> --cover-erase --cover-html --cover-package=airflow
>>>>> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
>>>>> --rednose --with-timer -s -v --logging-level=DEBUG
>>>>> + nosetests --with-coverage --cover-erase --cover-html
>>>>> --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
>>>>> --with-ignore-docstrings --rednose --with-timer -s -v --logging-level=DEBUG
>>>>> nose.plugins.cover: ERROR: Coverage not available: unable to import
>>>>> coverage module
>>>>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>>>>> position 21082: ordinal not in range(128)) ... ERROR
>>>>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>>>>> position 21082: ordinal not in range(128)) ... ERROR
>>>>> ======================================================================
>>>>> 1) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
>>>>> in position 21082: ordinal not in range(128))
>>>>> ----------------------------------------------------------------------
>>>>> Traceback (most recent call last):
>>>>> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>>>>> runTest
>>>>> raise self.exc_val.with_traceback(self.tb)
>>>>> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>>>>> loadTestsFromName
>>>>> addr.filename, addr.module)
>>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>>>>> importFromPath
>>>>> return self.importFromDir(dir_path, fqname)
>>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>>>>> importFromDir
>>>>> mod = load_module(part_fqname, fh, filename, desc)
>>>>> /usr/local/lib/python3.5/imp.py line 245 in load_module
>>>>> return load_package(name, filename)
>>>>> /usr/local/lib/python3.5/imp.py line 217 in load_package
>>>>> return _load(spec)
>>>>> <frozen importlib._bootstrap> line 693 in _load
>>>>>
>>>>> <frozen importlib._bootstrap> line 673 in _load_unlocked
>>>>>
>>>>> <frozen importlib._bootstrap_external> line 697 in exec_module
>>>>>
>>>>> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>>>>>
>>>>> airflow/__init__.py line 35 in <module>
>>>>> from airflow import configuration as conf
>>>>> airflow/configuration.py line 106 in <module>
>>>>> DEFAULT_CONFIG = f.read()
>>>>> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>>>> 21082: ordinal not in range(128)
>>>>> ======================================================================
>>>>> 2) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
>>>>> in position 21082: ordinal not in range(128))
>>>>> ----------------------------------------------------------------------
>>>>> Traceback (most recent call last):
>>>>> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>>>>> runTest
>>>>> raise self.exc_val.with_traceback(self.tb)
>>>>> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>>>>> loadTestsFromName
>>>>> addr.filename, addr.module)
>>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>>>>> importFromPath
>>>>> return self.importFromDir(dir_path, fqname)
>>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>>>>> importFromDir
>>>>> mod = load_module(part_fqname, fh, filename, desc)
>>>>> /usr/local/lib/python3.5/imp.py line 245 in load_module
>>>>> return load_package(name, filename)
>>>>> /usr/local/lib/python3.5/imp.py line 217 in load_package
>>>>> return _load(spec)
>>>>> <frozen importlib._bootstrap> line 693 in _load
>>>>>
>>>>> <frozen importlib._bootstrap> line 673 in _load_unlocked
>>>>>
>>>>> <frozen importlib._bootstrap_external> line 697 in exec_module
>>>>>
>>>>> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>>>>>
>>>>> tests/__init__.py line 25 in <module>
>>>>> from .configuration import *
>>>>> tests/configuration.py line 28 in <module>
>>>>> from airflow import configuration
>>>>> airflow/__init__.py line 35 in <module>
>>>>> from airflow import configuration as conf
>>>>> airflow/configuration.py line 106 in <module>
>>>>> DEFAULT_CONFIG = f.read()
>>>>> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>>>> 21082: ordinal not in range(128)
>>>>>
>>>>> [error] 100.00% nose.failure.Failure.runTest: 0.0004s
>>>>>
>>>>> -----------------------------------------------------------------------------
>>>>> 2 tests run in 0.060 seconds.
>>>>> 2 errors (0 tests passed)
>>>>>
>>>>>
>>>>> --
>>>>> Carl Johan Gustavsson
>>>>>
>>>>> On 11 July 2018 at 23:24:01, Bolke de Bruin (bdbruin@gmail.com) wrote:
>>>>>
>>>>> Hi Carl,
>>>>>
>>>>> That is not a real check, ie. Does Airflow have the same issue clean
>>>>> install on 3.5? Travis’ tests run on 3.5.
>>>>>
>>>>> B.
>>>>>
>>>>> Verstuurd vanaf mijn iPad
>>>>>
>>>>>> Op 10 jul. 2018 om 15:10 heeft Carl Johan Gustavsson <
>>>>> carl.j.gustavsson@gmail.com> het volgende geschreven:
>>>>>>
>>>>>> Hi Bolke,
>>>>>>
>>>>>>
>>>>>> I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all but 3.7
>>>>> breaks
>>>>>>
>>>>>> Quick repro:
>>>>>>
>>>>>> ➜ ~ pyenv local 3.5.5
>>>>>> ➜ ~ locale
>>>>>> LANG=
>>>>>> LC_COLLATE="C"
>>>>>> LC_CTYPE="C"
>>>>>> LC_MESSAGES="C"
>>>>>> LC_MONETARY="C"
>>>>>> LC_NUMERIC="C"
>>>>>> LC_TIME="C"
>>>>>> LC_ALL=
>>>>>> ➜ ~ cat testweird.txt
>>>>>> ’
>>>>>> ➜ ~ python
>>>>>> Python 3.5.5 (default, Jul 7 2018, 17:00:56)
>>>>>> [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
>>>>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>>>>> open('testweird.txt').read()
>>>>>> Traceback (most recent call last):
>>>>>> File "<stdin>", line 1, in <module>
>>>>>> File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/ascii.py",
>>>>> line 26, in decode
>>>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0:
>>>>> ordinal not in range(128)
>>>>>>>>>
>>>>>>
>>>>>>
>>>>>> Maybe not a blocking change but it is a breaking change from 1.9 I guess.
>>>>>>
>>>>>>
>>>>>> / Carl Johan
>>>>>>
>>>>>>
>>>>>>> On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbruin@gmail.com) wrote:
>>>>>>>
>>>>>>> Hi Carl,
>>>>>>>
>>>>>>> Did you this on python 3.5 as well? 3.6 is not an officially supported
>>>>> (yet). As a workaround is available I won’t consider this blocking btw.
>>>>>>>
>>>>>>> Bolke
>>>>>>>
>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>>
>>>>>>> Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson <
>>>>> carl.j.gustavsson@gmail.com> het volgende geschreven:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> First of all, thank you for all the work with the release management.
>>>>>>>>
>>>>>>>> I ran in to a weird issue testing the RC1, running under Python 3.6.0 /
>>>>> Ubuntu 14.04.5, upgrading from a master build from February.
>>>>>>>>
>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback
>>>>> (most recent call last):
>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>>> "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in <module>
>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
>>>>> import configuration
>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>>>
>>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/__init__.py",
>>>>> line 35, in <module>
>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
>>>>> import configuration as conf
>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>>>
>>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/configuration.py",
>>>>> line 106, in <module>
>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>>>>> DEFAULT_CONFIG = f.read()
>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py", line
>>>>> 26, in decode
>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return
>>>>> codecs.ascii_decode(input, self.errors)[0]
>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 20770:
>>>>> ordinal not in range(128)
>>>>>>>>
>>>>>>>> Removing the ’ on
>>>>>
>>>>> https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L613
>>>>> solved the issue for me, and digging a bit deeper it seems Airflow now
>>>>> requires setting LC_ALL=en_US.UTF-8 in the environment or similar to force
>>>>> Python to read the file as utf-8 and not ascii. (I think this was changed
>>>>> in to default to utf-8 in Python 3.7).
>>>>>>>>
>>>>>>>> I see 3 solutions for this
>>>>>>>> 1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or
>>>>> similar.
>>>>>>>> 2. Change the default config file to not contain non-ascii characters.
>>>>>>>> 3. Always read the file as unicode regardless of the LC_ALL
>>>>> environment, by the encoding='utf-8’ parameter to open().
>>>>>>>>
>>>>>>>> I think 3 is the best solution, and I can prepare a PR for that if
>>>>> necessary .
>>>>>>>>
>>>>>>>> I guess this counts as -1 (non-binding)
>>>>>>>>
>>>>>>>> All the best
>>>>>>>>
>>>>>>>> Carl Johan
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbruin@gmail.com) wrote:
>>>>>>>>>
>>>>>>>>> Hey all,
>>>>>>>>>
>>>>>>>>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the
>>>>> release,
>>>>>>>>> which will last for 72 hours. Consider this my (binding) +1.
>>>>>>>>>
>>>>>>>>> Airflow 1.10.0 RC 1 is available at:
>>>>>>>>>
>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <
>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
>>>>>>>>>
>>>>>>>>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release
>>>>> that
>>>>>>>>> comes with INSTALL instructions.
>>>>>>>>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python
>>>>> "sdist"
>>>>>>>>> release.
>>>>>>>>>
>>>>>>>>> Public keys are available at:
>>>>>>>>>
>>>>>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/>
>>>>>>>>>
>>>>>>>>> The amount of JIRAs fixed is over 700. Please have a look at the
>>>>> changelog.
>>>>>>>>>
>>>>>>>>> Please note that the version number excludes the `rcX` string as well
>>>>>>>>> as the "+incubating" string, so it's now simply 1.10.0. This will
>>>>> allow us
>>>>>>>>> to rename the artifact without modifying the artifact checksums when
>>>>> we
>>>>>>>>> actually release.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Bolke
>>>>>
>
Re: [VOTE] Airflow 1.10.0rc1
Posted by Ash Berlin-Taylor <as...@firemirror.com>.
Possible blocking regression here - .airflowignore doesn't seem to be respected anymore
https://issues.apache.org/jira/browse/AIRFLOW-2729 <https://issues.apache.org/jira/browse/AIRFLOW-2729>
-ash
> On 12 Jul 2018, at 08:45, Bolke de Bruin <bd...@gmail.com> wrote:
>
> Hi Jakob,
>
> Thanks. We do include an INSTALL document that explains how to well install airflow and is quite a standard location for install instructions. Or did the install instruction needed to be included in the Vote?
>
> B.
>
>
>
> Verstuurd vanaf mijn iPad
>
>> Op 12 jul. 2018 om 07:54 heeft Jakob Homan <jg...@gmail.com> het volgende geschreven:
>>
>> +1 (binding)
>>
>> * Sigs look good
>> * Artifact has incubating in name
>> * LICENSE/NOTICE/DISCLAIMER look good
>> - nit: DISCLAIMER is not word wrapped to 80 chars as I've seen in
>> other projects.
>> * Nit: Last release Sebb called out "Copyright 2016 and onwards" in
>> NOTICE as being imprecise. This language remains.
>> * Spot check on license headers looks ok.
>> * Nit: Last release there was a request for instructions on how to
>> build the code. This isn't included.
>>
>> -Jakob
>>
>>> On 11 July 2018 at 19:50, Sid Anand <sa...@apache.org> wrote:
>>> FYI!
>>> I just installed the release candidate. The first thing I noticed is a
>>> missing tool tip for the Null State in the Recent Tasks column on the
>>> landing page. Since the null globe is new to this UI, users will likely
>>> hover over it to inquire what it means... and will be left wanting. Of
>>> course, they could click on the globe, which will take them to
>>> http://localhost:8080/admin/taskinstance/?flt1_dag_id_equals=example_bash_operator&flt2_state_equals=null,
>>> which will always show an empty list, leaving them a bit more confused.
>>>
>>> -s
>>>
>>> On Wed, Jul 11, 2018 at 3:13 PM Carl Johan Gustavsson
>>> <ca...@tictail.com.invalid> wrote:
>>>
>>>> Hi Bolke,
>>>>
>>>> (Switching email to avoid moderation on my emails.)
>>>>
>>>> The normal Airflow test suite does not fail as it uses a LC_ALL set to
>>>> utf-8.
>>>>
>>>> I think it is a proper test though, it is a minimal reproducible version of
>>>> the code that fails. And the only difference in behaviour is at 3.7 which
>>>> we don’t support anyway so I’m fairly sure it is broken for all supported
>>>> Python 3 versions.
>>>>
>>>> I now tried running the tests in docker using 3.5 with the LC_ALL/LANG
>>>> unset and I see the same failure.
>>>>
>>>> I don’t think this is a big thing though and we could release it without
>>>> the fix I made. I think most people run it with a sane LC_ALL, but
>>>> apparently we didn’t.
>>>> Here’s the log for the test:
>>>>
>>>>> docker run -t -i -v `pwd`:/airflow/ python:3.5 bash
>>>> root@b99b297df111:/# locale
>>>> LANG=C.UTF-8
>>>> LANGUAGE=
>>>> LC_CTYPE="C.UTF-8"
>>>> LC_NUMERIC="C.UTF-8"
>>>> LC_TIME="C.UTF-8"
>>>> LC_COLLATE="C.UTF-8"
>>>> LC_MONETARY="C.UTF-8"
>>>> LC_MESSAGES="C.UTF-8"
>>>> LC_PAPER="C.UTF-8"
>>>> LC_NAME="C.UTF-8"
>>>> LC_ADDRESS="C.UTF-8"
>>>> LC_TELEPHONE="C.UTF-8"
>>>> LC_MEASUREMENT="C.UTF-8"
>>>> LC_IDENTIFICATION="C.UTF-8"
>>>> LC_ALL=
>>>>> unset LANG
>>>> root@b99b297df111:/# locale
>>>> LANG=
>>>> LANGUAGE=
>>>> LC_CTYPE="POSIX"
>>>> LC_NUMERIC="POSIX"
>>>> LC_TIME="POSIX"
>>>> LC_COLLATE="POSIX"
>>>> LC_MONETARY="POSIX"
>>>> LC_MESSAGES="POSIX"
>>>> LC_PAPER="POSIX"
>>>> LC_NAME="POSIX"
>>>> LC_ADDRESS="POSIX"
>>>> LC_TELEPHONE="POSIX"
>>>> LC_MEASUREMENT="POSIX"
>>>> LC_IDENTIFICATION="POSIX"
>>>> LC_ALL=
>>>> root@b99b297df111:/# pip install -e .[devel]
>>>> root@b99b297df111:/airflow# ./run_unit_tests.sh
>>>> + export AIRFLOW_HOME=/root/airflow
>>>> + AIRFLOW_HOME=/root/airflow
>>>> + export AIRFLOW__CORE__UNIT_TEST_MODE=True
>>>> + AIRFLOW__CORE__UNIT_TEST_MODE=True
>>>> + export AIRFLOW__TESTSECTION__TESTKEY=testvalue
>>>> + AIRFLOW__TESTSECTION__TESTKEY=testvalue
>>>> + export AIRFLOW_USE_NEW_IMPORTS=1
>>>> + AIRFLOW_USE_NEW_IMPORTS=1
>>>> +++ dirname ./run_unit_tests.sh
>>>> ++ cd .
>>>> ++ pwd
>>>> + DIR=/airflow
>>>> + export PYTHONPATH=:/airflow/tests/test_utils
>>>> + PYTHONPATH=:/airflow/tests/test_utils
>>>> + nose_args=
>>>> + which airflow
>>>> + echo 'Initializing the DB'
>>>> Initializing the DB
>>>> + airflow resetdb
>>>> + yes
>>>> Traceback (most recent call last):
>>>> File "/usr/local/bin/airflow", line 6, in <module>
>>>> exec(compile(open(__file__).read(), __file__, 'exec'))
>>>> File "/airflow/airflow/bin/airflow", line 21, in <module>
>>>> from airflow import configuration
>>>> File "/airflow/airflow/__init__.py", line 35, in <module>
>>>> from airflow import configuration as conf
>>>> File "/airflow/airflow/configuration.py", line 106, in <module>
>>>> DEFAULT_CONFIG = f.read()
>>>> File "/usr/local/lib/python3.5/encodings/ascii.py", line 26, in decode
>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 21082:
>>>> ordinal not in range(128)
>>>> + '[' '' ']'
>>>> + '[' -z '' ']'
>>>> + nose_args='--with-coverage --cover-erase --cover-html
>>>> --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
>>>> --with-ignore-docstrings --rednose --with-timer -s -v
>>>> --logging-level=DEBUG '
>>>> + echo 'Starting the unit tests with the following nose arguments:
>>>> --with-coverage' --cover-erase --cover-html --cover-package=airflow
>>>> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
>>>> --rednose --with-timer -s -v --logging-level=DEBUG
>>>> Starting the unit tests with the following nose arguments: --with-coverage
>>>> --cover-erase --cover-html --cover-package=airflow
>>>> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
>>>> --rednose --with-timer -s -v --logging-level=DEBUG
>>>> + nosetests --with-coverage --cover-erase --cover-html
>>>> --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
>>>> --with-ignore-docstrings --rednose --with-timer -s -v --logging-level=DEBUG
>>>> nose.plugins.cover: ERROR: Coverage not available: unable to import
>>>> coverage module
>>>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>>>> position 21082: ordinal not in range(128)) ... ERROR
>>>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>>>> position 21082: ordinal not in range(128)) ... ERROR
>>>> ======================================================================
>>>> 1) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
>>>> in position 21082: ordinal not in range(128))
>>>> ----------------------------------------------------------------------
>>>> Traceback (most recent call last):
>>>> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>>>> runTest
>>>> raise self.exc_val.with_traceback(self.tb)
>>>> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>>>> loadTestsFromName
>>>> addr.filename, addr.module)
>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>>>> importFromPath
>>>> return self.importFromDir(dir_path, fqname)
>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>>>> importFromDir
>>>> mod = load_module(part_fqname, fh, filename, desc)
>>>> /usr/local/lib/python3.5/imp.py line 245 in load_module
>>>> return load_package(name, filename)
>>>> /usr/local/lib/python3.5/imp.py line 217 in load_package
>>>> return _load(spec)
>>>> <frozen importlib._bootstrap> line 693 in _load
>>>>
>>>> <frozen importlib._bootstrap> line 673 in _load_unlocked
>>>>
>>>> <frozen importlib._bootstrap_external> line 697 in exec_module
>>>>
>>>> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>>>>
>>>> airflow/__init__.py line 35 in <module>
>>>> from airflow import configuration as conf
>>>> airflow/configuration.py line 106 in <module>
>>>> DEFAULT_CONFIG = f.read()
>>>> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>>> 21082: ordinal not in range(128)
>>>> ======================================================================
>>>> 2) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
>>>> in position 21082: ordinal not in range(128))
>>>> ----------------------------------------------------------------------
>>>> Traceback (most recent call last):
>>>> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>>>> runTest
>>>> raise self.exc_val.with_traceback(self.tb)
>>>> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>>>> loadTestsFromName
>>>> addr.filename, addr.module)
>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>>>> importFromPath
>>>> return self.importFromDir(dir_path, fqname)
>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>>>> importFromDir
>>>> mod = load_module(part_fqname, fh, filename, desc)
>>>> /usr/local/lib/python3.5/imp.py line 245 in load_module
>>>> return load_package(name, filename)
>>>> /usr/local/lib/python3.5/imp.py line 217 in load_package
>>>> return _load(spec)
>>>> <frozen importlib._bootstrap> line 693 in _load
>>>>
>>>> <frozen importlib._bootstrap> line 673 in _load_unlocked
>>>>
>>>> <frozen importlib._bootstrap_external> line 697 in exec_module
>>>>
>>>> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>>>>
>>>> tests/__init__.py line 25 in <module>
>>>> from .configuration import *
>>>> tests/configuration.py line 28 in <module>
>>>> from airflow import configuration
>>>> airflow/__init__.py line 35 in <module>
>>>> from airflow import configuration as conf
>>>> airflow/configuration.py line 106 in <module>
>>>> DEFAULT_CONFIG = f.read()
>>>> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>>> 21082: ordinal not in range(128)
>>>>
>>>> [error] 100.00% nose.failure.Failure.runTest: 0.0004s
>>>>
>>>> -----------------------------------------------------------------------------
>>>> 2 tests run in 0.060 seconds.
>>>> 2 errors (0 tests passed)
>>>>
>>>>
>>>> --
>>>> Carl Johan Gustavsson
>>>>
>>>> On 11 July 2018 at 23:24:01, Bolke de Bruin (bdbruin@gmail.com) wrote:
>>>>
>>>> Hi Carl,
>>>>
>>>> That is not a real check, ie. Does Airflow have the same issue clean
>>>> install on 3.5? Travis’ tests run on 3.5.
>>>>
>>>> B.
>>>>
>>>> Verstuurd vanaf mijn iPad
>>>>
>>>>> Op 10 jul. 2018 om 15:10 heeft Carl Johan Gustavsson <
>>>> carl.j.gustavsson@gmail.com> het volgende geschreven:
>>>>>
>>>>> Hi Bolke,
>>>>>
>>>>>
>>>>> I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all but 3.7
>>>> breaks
>>>>>
>>>>> Quick repro:
>>>>>
>>>>> ➜ ~ pyenv local 3.5.5
>>>>> ➜ ~ locale
>>>>> LANG=
>>>>> LC_COLLATE="C"
>>>>> LC_CTYPE="C"
>>>>> LC_MESSAGES="C"
>>>>> LC_MONETARY="C"
>>>>> LC_NUMERIC="C"
>>>>> LC_TIME="C"
>>>>> LC_ALL=
>>>>> ➜ ~ cat testweird.txt
>>>>> ’
>>>>> ➜ ~ python
>>>>> Python 3.5.5 (default, Jul 7 2018, 17:00:56)
>>>>> [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
>>>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>>>> open('testweird.txt').read()
>>>>> Traceback (most recent call last):
>>>>> File "<stdin>", line 1, in <module>
>>>>> File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/ascii.py",
>>>> line 26, in decode
>>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0:
>>>> ordinal not in range(128)
>>>>>>>>
>>>>>
>>>>>
>>>>> Maybe not a blocking change but it is a breaking change from 1.9 I guess.
>>>>>
>>>>>
>>>>> / Carl Johan
>>>>>
>>>>>
>>>>>> On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbruin@gmail.com) wrote:
>>>>>>
>>>>>> Hi Carl,
>>>>>>
>>>>>> Did you this on python 3.5 as well? 3.6 is not an officially supported
>>>> (yet). As a workaround is available I won’t consider this blocking btw.
>>>>>>
>>>>>> Bolke
>>>>>>
>>>>>> Verstuurd vanaf mijn iPad
>>>>>>
>>>>>> Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson <
>>>> carl.j.gustavsson@gmail.com> het volgende geschreven:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> First of all, thank you for all the work with the release management.
>>>>>>>
>>>>>>> I ran in to a weird issue testing the RC1, running under Python 3.6.0 /
>>>> Ubuntu 14.04.5, upgrading from a master build from February.
>>>>>>>
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback
>>>> (most recent call last):
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>> "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in <module>
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
>>>> import configuration
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>>
>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/__init__.py",
>>>> line 35, in <module>
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
>>>> import configuration as conf
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>>
>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/configuration.py",
>>>> line 106, in <module>
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>>>> DEFAULT_CONFIG = f.read()
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py", line
>>>> 26, in decode
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return
>>>> codecs.ascii_decode(input, self.errors)[0]
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 20770:
>>>> ordinal not in range(128)
>>>>>>>
>>>>>>> Removing the ’ on
>>>>
>>>> https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L613
>>>> solved the issue for me, and digging a bit deeper it seems Airflow now
>>>> requires setting LC_ALL=en_US.UTF-8 in the environment or similar to force
>>>> Python to read the file as utf-8 and not ascii. (I think this was changed
>>>> in to default to utf-8 in Python 3.7).
>>>>>>>
>>>>>>> I see 3 solutions for this
>>>>>>> 1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or
>>>> similar.
>>>>>>> 2. Change the default config file to not contain non-ascii characters.
>>>>>>> 3. Always read the file as unicode regardless of the LC_ALL
>>>> environment, by the encoding='utf-8’ parameter to open().
>>>>>>>
>>>>>>> I think 3 is the best solution, and I can prepare a PR for that if
>>>> necessary .
>>>>>>>
>>>>>>> I guess this counts as -1 (non-binding)
>>>>>>>
>>>>>>> All the best
>>>>>>>
>>>>>>> Carl Johan
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbruin@gmail.com) wrote:
>>>>>>>>
>>>>>>>> Hey all,
>>>>>>>>
>>>>>>>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the
>>>> release,
>>>>>>>> which will last for 72 hours. Consider this my (binding) +1.
>>>>>>>>
>>>>>>>> Airflow 1.10.0 RC 1 is available at:
>>>>>>>>
>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <
>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
>>>>>>>>
>>>>>>>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release
>>>> that
>>>>>>>> comes with INSTALL instructions.
>>>>>>>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python
>>>> "sdist"
>>>>>>>> release.
>>>>>>>>
>>>>>>>> Public keys are available at:
>>>>>>>>
>>>>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/>
>>>>>>>>
>>>>>>>> The amount of JIRAs fixed is over 700. Please have a look at the
>>>> changelog.
>>>>>>>>
>>>>>>>> Please note that the version number excludes the `rcX` string as well
>>>>>>>> as the "+incubating" string, so it's now simply 1.10.0. This will
>>>> allow us
>>>>>>>> to rename the artifact without modifying the artifact checksums when
>>>> we
>>>>>>>> actually release.
>>>>>>>>
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Bolke
>>>>
Re: [VOTE] Airflow 1.10.0rc1
Posted by Bolke de Bruin <bd...@gmail.com>.
Hi Jakob,
Thanks. We do include an INSTALL document that explains how to well install airflow and is quite a standard location for install instructions. Or did the install instruction needed to be included in the Vote?
B.
Verstuurd vanaf mijn iPad
> Op 12 jul. 2018 om 07:54 heeft Jakob Homan <jg...@gmail.com> het volgende geschreven:
>
> +1 (binding)
>
> * Sigs look good
> * Artifact has incubating in name
> * LICENSE/NOTICE/DISCLAIMER look good
> - nit: DISCLAIMER is not word wrapped to 80 chars as I've seen in
> other projects.
> * Nit: Last release Sebb called out "Copyright 2016 and onwards" in
> NOTICE as being imprecise. This language remains.
> * Spot check on license headers looks ok.
> * Nit: Last release there was a request for instructions on how to
> build the code. This isn't included.
>
> -Jakob
>
>> On 11 July 2018 at 19:50, Sid Anand <sa...@apache.org> wrote:
>> FYI!
>> I just installed the release candidate. The first thing I noticed is a
>> missing tool tip for the Null State in the Recent Tasks column on the
>> landing page. Since the null globe is new to this UI, users will likely
>> hover over it to inquire what it means... and will be left wanting. Of
>> course, they could click on the globe, which will take them to
>> http://localhost:8080/admin/taskinstance/?flt1_dag_id_equals=example_bash_operator&flt2_state_equals=null,
>> which will always show an empty list, leaving them a bit more confused.
>>
>> -s
>>
>> On Wed, Jul 11, 2018 at 3:13 PM Carl Johan Gustavsson
>> <ca...@tictail.com.invalid> wrote:
>>
>>> Hi Bolke,
>>>
>>> (Switching email to avoid moderation on my emails.)
>>>
>>> The normal Airflow test suite does not fail as it uses a LC_ALL set to
>>> utf-8.
>>>
>>> I think it is a proper test though, it is a minimal reproducible version of
>>> the code that fails. And the only difference in behaviour is at 3.7 which
>>> we don’t support anyway so I’m fairly sure it is broken for all supported
>>> Python 3 versions.
>>>
>>> I now tried running the tests in docker using 3.5 with the LC_ALL/LANG
>>> unset and I see the same failure.
>>>
>>> I don’t think this is a big thing though and we could release it without
>>> the fix I made. I think most people run it with a sane LC_ALL, but
>>> apparently we didn’t.
>>> Here’s the log for the test:
>>>
>>>> docker run -t -i -v `pwd`:/airflow/ python:3.5 bash
>>> root@b99b297df111:/# locale
>>> LANG=C.UTF-8
>>> LANGUAGE=
>>> LC_CTYPE="C.UTF-8"
>>> LC_NUMERIC="C.UTF-8"
>>> LC_TIME="C.UTF-8"
>>> LC_COLLATE="C.UTF-8"
>>> LC_MONETARY="C.UTF-8"
>>> LC_MESSAGES="C.UTF-8"
>>> LC_PAPER="C.UTF-8"
>>> LC_NAME="C.UTF-8"
>>> LC_ADDRESS="C.UTF-8"
>>> LC_TELEPHONE="C.UTF-8"
>>> LC_MEASUREMENT="C.UTF-8"
>>> LC_IDENTIFICATION="C.UTF-8"
>>> LC_ALL=
>>>> unset LANG
>>> root@b99b297df111:/# locale
>>> LANG=
>>> LANGUAGE=
>>> LC_CTYPE="POSIX"
>>> LC_NUMERIC="POSIX"
>>> LC_TIME="POSIX"
>>> LC_COLLATE="POSIX"
>>> LC_MONETARY="POSIX"
>>> LC_MESSAGES="POSIX"
>>> LC_PAPER="POSIX"
>>> LC_NAME="POSIX"
>>> LC_ADDRESS="POSIX"
>>> LC_TELEPHONE="POSIX"
>>> LC_MEASUREMENT="POSIX"
>>> LC_IDENTIFICATION="POSIX"
>>> LC_ALL=
>>> root@b99b297df111:/# pip install -e .[devel]
>>> root@b99b297df111:/airflow# ./run_unit_tests.sh
>>> + export AIRFLOW_HOME=/root/airflow
>>> + AIRFLOW_HOME=/root/airflow
>>> + export AIRFLOW__CORE__UNIT_TEST_MODE=True
>>> + AIRFLOW__CORE__UNIT_TEST_MODE=True
>>> + export AIRFLOW__TESTSECTION__TESTKEY=testvalue
>>> + AIRFLOW__TESTSECTION__TESTKEY=testvalue
>>> + export AIRFLOW_USE_NEW_IMPORTS=1
>>> + AIRFLOW_USE_NEW_IMPORTS=1
>>> +++ dirname ./run_unit_tests.sh
>>> ++ cd .
>>> ++ pwd
>>> + DIR=/airflow
>>> + export PYTHONPATH=:/airflow/tests/test_utils
>>> + PYTHONPATH=:/airflow/tests/test_utils
>>> + nose_args=
>>> + which airflow
>>> + echo 'Initializing the DB'
>>> Initializing the DB
>>> + airflow resetdb
>>> + yes
>>> Traceback (most recent call last):
>>> File "/usr/local/bin/airflow", line 6, in <module>
>>> exec(compile(open(__file__).read(), __file__, 'exec'))
>>> File "/airflow/airflow/bin/airflow", line 21, in <module>
>>> from airflow import configuration
>>> File "/airflow/airflow/__init__.py", line 35, in <module>
>>> from airflow import configuration as conf
>>> File "/airflow/airflow/configuration.py", line 106, in <module>
>>> DEFAULT_CONFIG = f.read()
>>> File "/usr/local/lib/python3.5/encodings/ascii.py", line 26, in decode
>>> return codecs.ascii_decode(input, self.errors)[0]
>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 21082:
>>> ordinal not in range(128)
>>> + '[' '' ']'
>>> + '[' -z '' ']'
>>> + nose_args='--with-coverage --cover-erase --cover-html
>>> --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
>>> --with-ignore-docstrings --rednose --with-timer -s -v
>>> --logging-level=DEBUG '
>>> + echo 'Starting the unit tests with the following nose arguments:
>>> --with-coverage' --cover-erase --cover-html --cover-package=airflow
>>> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
>>> --rednose --with-timer -s -v --logging-level=DEBUG
>>> Starting the unit tests with the following nose arguments: --with-coverage
>>> --cover-erase --cover-html --cover-package=airflow
>>> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
>>> --rednose --with-timer -s -v --logging-level=DEBUG
>>> + nosetests --with-coverage --cover-erase --cover-html
>>> --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
>>> --with-ignore-docstrings --rednose --with-timer -s -v --logging-level=DEBUG
>>> nose.plugins.cover: ERROR: Coverage not available: unable to import
>>> coverage module
>>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>>> position 21082: ordinal not in range(128)) ... ERROR
>>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>>> position 21082: ordinal not in range(128)) ... ERROR
>>> ======================================================================
>>> 1) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
>>> in position 21082: ordinal not in range(128))
>>> ----------------------------------------------------------------------
>>> Traceback (most recent call last):
>>> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>>> runTest
>>> raise self.exc_val.with_traceback(self.tb)
>>> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>>> loadTestsFromName
>>> addr.filename, addr.module)
>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>>> importFromPath
>>> return self.importFromDir(dir_path, fqname)
>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>>> importFromDir
>>> mod = load_module(part_fqname, fh, filename, desc)
>>> /usr/local/lib/python3.5/imp.py line 245 in load_module
>>> return load_package(name, filename)
>>> /usr/local/lib/python3.5/imp.py line 217 in load_package
>>> return _load(spec)
>>> <frozen importlib._bootstrap> line 693 in _load
>>>
>>> <frozen importlib._bootstrap> line 673 in _load_unlocked
>>>
>>> <frozen importlib._bootstrap_external> line 697 in exec_module
>>>
>>> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>>>
>>> airflow/__init__.py line 35 in <module>
>>> from airflow import configuration as conf
>>> airflow/configuration.py line 106 in <module>
>>> DEFAULT_CONFIG = f.read()
>>> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>>> return codecs.ascii_decode(input, self.errors)[0]
>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>> 21082: ordinal not in range(128)
>>> ======================================================================
>>> 2) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
>>> in position 21082: ordinal not in range(128))
>>> ----------------------------------------------------------------------
>>> Traceback (most recent call last):
>>> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>>> runTest
>>> raise self.exc_val.with_traceback(self.tb)
>>> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>>> loadTestsFromName
>>> addr.filename, addr.module)
>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>>> importFromPath
>>> return self.importFromDir(dir_path, fqname)
>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>>> importFromDir
>>> mod = load_module(part_fqname, fh, filename, desc)
>>> /usr/local/lib/python3.5/imp.py line 245 in load_module
>>> return load_package(name, filename)
>>> /usr/local/lib/python3.5/imp.py line 217 in load_package
>>> return _load(spec)
>>> <frozen importlib._bootstrap> line 693 in _load
>>>
>>> <frozen importlib._bootstrap> line 673 in _load_unlocked
>>>
>>> <frozen importlib._bootstrap_external> line 697 in exec_module
>>>
>>> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>>>
>>> tests/__init__.py line 25 in <module>
>>> from .configuration import *
>>> tests/configuration.py line 28 in <module>
>>> from airflow import configuration
>>> airflow/__init__.py line 35 in <module>
>>> from airflow import configuration as conf
>>> airflow/configuration.py line 106 in <module>
>>> DEFAULT_CONFIG = f.read()
>>> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>>> return codecs.ascii_decode(input, self.errors)[0]
>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>> 21082: ordinal not in range(128)
>>>
>>> [error] 100.00% nose.failure.Failure.runTest: 0.0004s
>>>
>>> -----------------------------------------------------------------------------
>>> 2 tests run in 0.060 seconds.
>>> 2 errors (0 tests passed)
>>>
>>>
>>> --
>>> Carl Johan Gustavsson
>>>
>>> On 11 July 2018 at 23:24:01, Bolke de Bruin (bdbruin@gmail.com) wrote:
>>>
>>> Hi Carl,
>>>
>>> That is not a real check, ie. Does Airflow have the same issue clean
>>> install on 3.5? Travis’ tests run on 3.5.
>>>
>>> B.
>>>
>>> Verstuurd vanaf mijn iPad
>>>
>>>> Op 10 jul. 2018 om 15:10 heeft Carl Johan Gustavsson <
>>> carl.j.gustavsson@gmail.com> het volgende geschreven:
>>>>
>>>> Hi Bolke,
>>>>
>>>>
>>>> I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all but 3.7
>>> breaks
>>>>
>>>> Quick repro:
>>>>
>>>> ➜ ~ pyenv local 3.5.5
>>>> ➜ ~ locale
>>>> LANG=
>>>> LC_COLLATE="C"
>>>> LC_CTYPE="C"
>>>> LC_MESSAGES="C"
>>>> LC_MONETARY="C"
>>>> LC_NUMERIC="C"
>>>> LC_TIME="C"
>>>> LC_ALL=
>>>> ➜ ~ cat testweird.txt
>>>> ’
>>>> ➜ ~ python
>>>> Python 3.5.5 (default, Jul 7 2018, 17:00:56)
>>>> [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
>>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>>> open('testweird.txt').read()
>>>> Traceback (most recent call last):
>>>> File "<stdin>", line 1, in <module>
>>>> File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/ascii.py",
>>> line 26, in decode
>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0:
>>> ordinal not in range(128)
>>>>>>>
>>>>
>>>>
>>>> Maybe not a blocking change but it is a breaking change from 1.9 I guess.
>>>>
>>>>
>>>> / Carl Johan
>>>>
>>>>
>>>>> On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbruin@gmail.com) wrote:
>>>>>
>>>>> Hi Carl,
>>>>>
>>>>> Did you this on python 3.5 as well? 3.6 is not an officially supported
>>> (yet). As a workaround is available I won’t consider this blocking btw.
>>>>>
>>>>> Bolke
>>>>>
>>>>> Verstuurd vanaf mijn iPad
>>>>>
>>>>> Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson <
>>> carl.j.gustavsson@gmail.com> het volgende geschreven:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> First of all, thank you for all the work with the release management.
>>>>>>
>>>>>> I ran in to a weird issue testing the RC1, running under Python 3.6.0 /
>>> Ubuntu 14.04.5, upgrading from a master build from February.
>>>>>>
>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback
>>> (most recent call last):
>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>> "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in <module>
>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
>>> import configuration
>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>
>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/__init__.py",
>>> line 35, in <module>
>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
>>> import configuration as conf
>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>
>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/configuration.py",
>>> line 106, in <module>
>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>>> DEFAULT_CONFIG = f.read()
>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py", line
>>> 26, in decode
>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return
>>> codecs.ascii_decode(input, self.errors)[0]
>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 20770:
>>> ordinal not in range(128)
>>>>>>
>>>>>> Removing the ’ on
>>>
>>> https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L613
>>> solved the issue for me, and digging a bit deeper it seems Airflow now
>>> requires setting LC_ALL=en_US.UTF-8 in the environment or similar to force
>>> Python to read the file as utf-8 and not ascii. (I think this was changed
>>> in to default to utf-8 in Python 3.7).
>>>>>>
>>>>>> I see 3 solutions for this
>>>>>> 1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or
>>> similar.
>>>>>> 2. Change the default config file to not contain non-ascii characters.
>>>>>> 3. Always read the file as unicode regardless of the LC_ALL
>>> environment, by the encoding='utf-8’ parameter to open().
>>>>>>
>>>>>> I think 3 is the best solution, and I can prepare a PR for that if
>>> necessary .
>>>>>>
>>>>>> I guess this counts as -1 (non-binding)
>>>>>>
>>>>>> All the best
>>>>>>
>>>>>> Carl Johan
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbruin@gmail.com) wrote:
>>>>>>>
>>>>>>> Hey all,
>>>>>>>
>>>>>>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the
>>> release,
>>>>>>> which will last for 72 hours. Consider this my (binding) +1.
>>>>>>>
>>>>>>> Airflow 1.10.0 RC 1 is available at:
>>>>>>>
>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <
>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
>>>>>>>
>>>>>>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release
>>> that
>>>>>>> comes with INSTALL instructions.
>>>>>>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python
>>> "sdist"
>>>>>>> release.
>>>>>>>
>>>>>>> Public keys are available at:
>>>>>>>
>>>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>>> https://dist.apache.org/repos/dist/release/incubator/airflow/>
>>>>>>>
>>>>>>> The amount of JIRAs fixed is over 700. Please have a look at the
>>> changelog.
>>>>>>>
>>>>>>> Please note that the version number excludes the `rcX` string as well
>>>>>>> as the "+incubating" string, so it's now simply 1.10.0. This will
>>> allow us
>>>>>>> to rename the artifact without modifying the artifact checksums when
>>> we
>>>>>>> actually release.
>>>>>>>
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Bolke
>>>
Re: [VOTE] Airflow 1.10.0rc1
Posted by Jakob Homan <jg...@gmail.com>.
+1 (binding)
* Sigs look good
* Artifact has incubating in name
* LICENSE/NOTICE/DISCLAIMER look good
- nit: DISCLAIMER is not word wrapped to 80 chars as I've seen in
other projects.
* Nit: Last release Sebb called out "Copyright 2016 and onwards" in
NOTICE as being imprecise. This language remains.
* Spot check on license headers looks ok.
* Nit: Last release there was a request for instructions on how to
build the code. This isn't included.
-Jakob
On 11 July 2018 at 19:50, Sid Anand <sa...@apache.org> wrote:
> FYI!
> I just installed the release candidate. The first thing I noticed is a
> missing tool tip for the Null State in the Recent Tasks column on the
> landing page. Since the null globe is new to this UI, users will likely
> hover over it to inquire what it means... and will be left wanting. Of
> course, they could click on the globe, which will take them to
> http://localhost:8080/admin/taskinstance/?flt1_dag_id_equals=example_bash_operator&flt2_state_equals=null,
> which will always show an empty list, leaving them a bit more confused.
>
> -s
>
> On Wed, Jul 11, 2018 at 3:13 PM Carl Johan Gustavsson
> <ca...@tictail.com.invalid> wrote:
>
>> Hi Bolke,
>>
>> (Switching email to avoid moderation on my emails.)
>>
>> The normal Airflow test suite does not fail as it uses a LC_ALL set to
>> utf-8.
>>
>> I think it is a proper test though, it is a minimal reproducible version of
>> the code that fails. And the only difference in behaviour is at 3.7 which
>> we don’t support anyway so I’m fairly sure it is broken for all supported
>> Python 3 versions.
>>
>> I now tried running the tests in docker using 3.5 with the LC_ALL/LANG
>> unset and I see the same failure.
>>
>> I don’t think this is a big thing though and we could release it without
>> the fix I made. I think most people run it with a sane LC_ALL, but
>> apparently we didn’t.
>> Here’s the log for the test:
>>
>> > docker run -t -i -v `pwd`:/airflow/ python:3.5 bash
>> root@b99b297df111:/# locale
>> LANG=C.UTF-8
>> LANGUAGE=
>> LC_CTYPE="C.UTF-8"
>> LC_NUMERIC="C.UTF-8"
>> LC_TIME="C.UTF-8"
>> LC_COLLATE="C.UTF-8"
>> LC_MONETARY="C.UTF-8"
>> LC_MESSAGES="C.UTF-8"
>> LC_PAPER="C.UTF-8"
>> LC_NAME="C.UTF-8"
>> LC_ADDRESS="C.UTF-8"
>> LC_TELEPHONE="C.UTF-8"
>> LC_MEASUREMENT="C.UTF-8"
>> LC_IDENTIFICATION="C.UTF-8"
>> LC_ALL=
>> > unset LANG
>> root@b99b297df111:/# locale
>> LANG=
>> LANGUAGE=
>> LC_CTYPE="POSIX"
>> LC_NUMERIC="POSIX"
>> LC_TIME="POSIX"
>> LC_COLLATE="POSIX"
>> LC_MONETARY="POSIX"
>> LC_MESSAGES="POSIX"
>> LC_PAPER="POSIX"
>> LC_NAME="POSIX"
>> LC_ADDRESS="POSIX"
>> LC_TELEPHONE="POSIX"
>> LC_MEASUREMENT="POSIX"
>> LC_IDENTIFICATION="POSIX"
>> LC_ALL=
>> root@b99b297df111:/# pip install -e .[devel]
>> root@b99b297df111:/airflow# ./run_unit_tests.sh
>> + export AIRFLOW_HOME=/root/airflow
>> + AIRFLOW_HOME=/root/airflow
>> + export AIRFLOW__CORE__UNIT_TEST_MODE=True
>> + AIRFLOW__CORE__UNIT_TEST_MODE=True
>> + export AIRFLOW__TESTSECTION__TESTKEY=testvalue
>> + AIRFLOW__TESTSECTION__TESTKEY=testvalue
>> + export AIRFLOW_USE_NEW_IMPORTS=1
>> + AIRFLOW_USE_NEW_IMPORTS=1
>> +++ dirname ./run_unit_tests.sh
>> ++ cd .
>> ++ pwd
>> + DIR=/airflow
>> + export PYTHONPATH=:/airflow/tests/test_utils
>> + PYTHONPATH=:/airflow/tests/test_utils
>> + nose_args=
>> + which airflow
>> + echo 'Initializing the DB'
>> Initializing the DB
>> + airflow resetdb
>> + yes
>> Traceback (most recent call last):
>> File "/usr/local/bin/airflow", line 6, in <module>
>> exec(compile(open(__file__).read(), __file__, 'exec'))
>> File "/airflow/airflow/bin/airflow", line 21, in <module>
>> from airflow import configuration
>> File "/airflow/airflow/__init__.py", line 35, in <module>
>> from airflow import configuration as conf
>> File "/airflow/airflow/configuration.py", line 106, in <module>
>> DEFAULT_CONFIG = f.read()
>> File "/usr/local/lib/python3.5/encodings/ascii.py", line 26, in decode
>> return codecs.ascii_decode(input, self.errors)[0]
>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 21082:
>> ordinal not in range(128)
>> + '[' '' ']'
>> + '[' -z '' ']'
>> + nose_args='--with-coverage --cover-erase --cover-html
>> --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
>> --with-ignore-docstrings --rednose --with-timer -s -v
>> --logging-level=DEBUG '
>> + echo 'Starting the unit tests with the following nose arguments:
>> --with-coverage' --cover-erase --cover-html --cover-package=airflow
>> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
>> --rednose --with-timer -s -v --logging-level=DEBUG
>> Starting the unit tests with the following nose arguments: --with-coverage
>> --cover-erase --cover-html --cover-package=airflow
>> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
>> --rednose --with-timer -s -v --logging-level=DEBUG
>> + nosetests --with-coverage --cover-erase --cover-html
>> --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
>> --with-ignore-docstrings --rednose --with-timer -s -v --logging-level=DEBUG
>> nose.plugins.cover: ERROR: Coverage not available: unable to import
>> coverage module
>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>> position 21082: ordinal not in range(128)) ... ERROR
>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>> position 21082: ordinal not in range(128)) ... ERROR
>> ======================================================================
>> 1) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
>> in position 21082: ordinal not in range(128))
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>> runTest
>> raise self.exc_val.with_traceback(self.tb)
>> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>> loadTestsFromName
>> addr.filename, addr.module)
>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>> importFromPath
>> return self.importFromDir(dir_path, fqname)
>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>> importFromDir
>> mod = load_module(part_fqname, fh, filename, desc)
>> /usr/local/lib/python3.5/imp.py line 245 in load_module
>> return load_package(name, filename)
>> /usr/local/lib/python3.5/imp.py line 217 in load_package
>> return _load(spec)
>> <frozen importlib._bootstrap> line 693 in _load
>>
>> <frozen importlib._bootstrap> line 673 in _load_unlocked
>>
>> <frozen importlib._bootstrap_external> line 697 in exec_module
>>
>> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>>
>> airflow/__init__.py line 35 in <module>
>> from airflow import configuration as conf
>> airflow/configuration.py line 106 in <module>
>> DEFAULT_CONFIG = f.read()
>> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>> return codecs.ascii_decode(input, self.errors)[0]
>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>> 21082: ordinal not in range(128)
>> ======================================================================
>> 2) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
>> in position 21082: ordinal not in range(128))
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>> runTest
>> raise self.exc_val.with_traceback(self.tb)
>> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>> loadTestsFromName
>> addr.filename, addr.module)
>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>> importFromPath
>> return self.importFromDir(dir_path, fqname)
>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>> importFromDir
>> mod = load_module(part_fqname, fh, filename, desc)
>> /usr/local/lib/python3.5/imp.py line 245 in load_module
>> return load_package(name, filename)
>> /usr/local/lib/python3.5/imp.py line 217 in load_package
>> return _load(spec)
>> <frozen importlib._bootstrap> line 693 in _load
>>
>> <frozen importlib._bootstrap> line 673 in _load_unlocked
>>
>> <frozen importlib._bootstrap_external> line 697 in exec_module
>>
>> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>>
>> tests/__init__.py line 25 in <module>
>> from .configuration import *
>> tests/configuration.py line 28 in <module>
>> from airflow import configuration
>> airflow/__init__.py line 35 in <module>
>> from airflow import configuration as conf
>> airflow/configuration.py line 106 in <module>
>> DEFAULT_CONFIG = f.read()
>> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>> return codecs.ascii_decode(input, self.errors)[0]
>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>> 21082: ordinal not in range(128)
>>
>> [error] 100.00% nose.failure.Failure.runTest: 0.0004s
>>
>> -----------------------------------------------------------------------------
>> 2 tests run in 0.060 seconds.
>> 2 errors (0 tests passed)
>>
>>
>> --
>> Carl Johan Gustavsson
>>
>> On 11 July 2018 at 23:24:01, Bolke de Bruin (bdbruin@gmail.com) wrote:
>>
>> Hi Carl,
>>
>> That is not a real check, ie. Does Airflow have the same issue clean
>> install on 3.5? Travis’ tests run on 3.5.
>>
>> B.
>>
>> Verstuurd vanaf mijn iPad
>>
>> > Op 10 jul. 2018 om 15:10 heeft Carl Johan Gustavsson <
>> carl.j.gustavsson@gmail.com> het volgende geschreven:
>> >
>> > Hi Bolke,
>> >
>> >
>> > I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all but 3.7
>> breaks
>> >
>> > Quick repro:
>> >
>> > ➜ ~ pyenv local 3.5.5
>> > ➜ ~ locale
>> > LANG=
>> > LC_COLLATE="C"
>> > LC_CTYPE="C"
>> > LC_MESSAGES="C"
>> > LC_MONETARY="C"
>> > LC_NUMERIC="C"
>> > LC_TIME="C"
>> > LC_ALL=
>> > ➜ ~ cat testweird.txt
>> > ’
>> > ➜ ~ python
>> > Python 3.5.5 (default, Jul 7 2018, 17:00:56)
>> > [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
>> > Type "help", "copyright", "credits" or "license" for more information.
>> > >>> open('testweird.txt').read()
>> > Traceback (most recent call last):
>> > File "<stdin>", line 1, in <module>
>> > File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/ascii.py",
>> line 26, in decode
>> > return codecs.ascii_decode(input, self.errors)[0]
>> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0:
>> ordinal not in range(128)
>> > >>>
>> >
>> >
>> > Maybe not a blocking change but it is a breaking change from 1.9 I guess.
>> >
>> >
>> > / Carl Johan
>> >
>> >
>> >> On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbruin@gmail.com) wrote:
>> >>
>> >> Hi Carl,
>> >>
>> >> Did you this on python 3.5 as well? 3.6 is not an officially supported
>> (yet). As a workaround is available I won’t consider this blocking btw.
>> >>
>> >> Bolke
>> >>
>> >> Verstuurd vanaf mijn iPad
>> >>
>> >> Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson <
>> carl.j.gustavsson@gmail.com> het volgende geschreven:
>> >>
>> >>> Hi,
>> >>>
>> >>> First of all, thank you for all the work with the release management.
>> >>>
>> >>> I ran in to a weird issue testing the RC1, running under Python 3.6.0 /
>> Ubuntu 14.04.5, upgrading from a master build from February.
>> >>>
>> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback
>> (most recent call last):
>> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>> "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in <module>
>> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
>> import configuration
>> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>
>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/__init__.py",
>> line 35, in <module>
>> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
>> import configuration as conf
>> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>
>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/configuration.py",
>> line 106, in <module>
>> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>> DEFAULT_CONFIG = f.read()
>> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py", line
>> 26, in decode
>> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return
>> codecs.ascii_decode(input, self.errors)[0]
>> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 20770:
>> ordinal not in range(128)
>> >>>
>> >>> Removing the ’ on
>>
>> https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L613
>> solved the issue for me, and digging a bit deeper it seems Airflow now
>> requires setting LC_ALL=en_US.UTF-8 in the environment or similar to force
>> Python to read the file as utf-8 and not ascii. (I think this was changed
>> in to default to utf-8 in Python 3.7).
>> >>>
>> >>> I see 3 solutions for this
>> >>> 1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or
>> similar.
>> >>> 2. Change the default config file to not contain non-ascii characters.
>> >>> 3. Always read the file as unicode regardless of the LC_ALL
>> environment, by the encoding='utf-8’ parameter to open().
>> >>>
>> >>> I think 3 is the best solution, and I can prepare a PR for that if
>> necessary .
>> >>>
>> >>> I guess this counts as -1 (non-binding)
>> >>>
>> >>> All the best
>> >>>
>> >>> Carl Johan
>> >>>
>> >>>
>> >>>
>> >>>> On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbruin@gmail.com) wrote:
>> >>>>
>> >>>> Hey all,
>> >>>>
>> >>>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the
>> release,
>> >>>> which will last for 72 hours. Consider this my (binding) +1.
>> >>>>
>> >>>> Airflow 1.10.0 RC 1 is available at:
>> >>>>
>> >>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <
>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
>> >>>>
>> >>>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release
>> that
>> >>>> comes with INSTALL instructions.
>> >>>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python
>> "sdist"
>> >>>> release.
>> >>>>
>> >>>> Public keys are available at:
>> >>>>
>> >>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>> https://dist.apache.org/repos/dist/release/incubator/airflow/>
>> >>>>
>> >>>> The amount of JIRAs fixed is over 700. Please have a look at the
>> changelog.
>> >>>>
>> >>>> Please note that the version number excludes the `rcX` string as well
>> >>>> as the "+incubating" string, so it's now simply 1.10.0. This will
>> allow us
>> >>>> to rename the artifact without modifying the artifact checksums when
>> we
>> >>>> actually release.
>> >>>>
>> >>>>
>> >>>> Cheers,
>> >>>> Bolke
>>
Re: [VOTE] Airflow 1.10.0rc1
Posted by Sid Anand <sa...@apache.org>.
FYI!
I just installed the release candidate. The first thing I noticed is a
missing tool tip for the Null State in the Recent Tasks column on the
landing page. Since the null globe is new to this UI, users will likely
hover over it to inquire what it means... and will be left wanting. Of
course, they could click on the globe, which will take them to
http://localhost:8080/admin/taskinstance/?flt1_dag_id_equals=example_bash_operator&flt2_state_equals=null,
which will always show an empty list, leaving them a bit more confused.
-s
On Wed, Jul 11, 2018 at 3:13 PM Carl Johan Gustavsson
<ca...@tictail.com.invalid> wrote:
> Hi Bolke,
>
> (Switching email to avoid moderation on my emails.)
>
> The normal Airflow test suite does not fail as it uses a LC_ALL set to
> utf-8.
>
> I think it is a proper test though, it is a minimal reproducible version of
> the code that fails. And the only difference in behaviour is at 3.7 which
> we don’t support anyway so I’m fairly sure it is broken for all supported
> Python 3 versions.
>
> I now tried running the tests in docker using 3.5 with the LC_ALL/LANG
> unset and I see the same failure.
>
> I don’t think this is a big thing though and we could release it without
> the fix I made. I think most people run it with a sane LC_ALL, but
> apparently we didn’t.
> Here’s the log for the test:
>
> > docker run -t -i -v `pwd`:/airflow/ python:3.5 bash
> root@b99b297df111:/# locale
> LANG=C.UTF-8
> LANGUAGE=
> LC_CTYPE="C.UTF-8"
> LC_NUMERIC="C.UTF-8"
> LC_TIME="C.UTF-8"
> LC_COLLATE="C.UTF-8"
> LC_MONETARY="C.UTF-8"
> LC_MESSAGES="C.UTF-8"
> LC_PAPER="C.UTF-8"
> LC_NAME="C.UTF-8"
> LC_ADDRESS="C.UTF-8"
> LC_TELEPHONE="C.UTF-8"
> LC_MEASUREMENT="C.UTF-8"
> LC_IDENTIFICATION="C.UTF-8"
> LC_ALL=
> > unset LANG
> root@b99b297df111:/# locale
> LANG=
> LANGUAGE=
> LC_CTYPE="POSIX"
> LC_NUMERIC="POSIX"
> LC_TIME="POSIX"
> LC_COLLATE="POSIX"
> LC_MONETARY="POSIX"
> LC_MESSAGES="POSIX"
> LC_PAPER="POSIX"
> LC_NAME="POSIX"
> LC_ADDRESS="POSIX"
> LC_TELEPHONE="POSIX"
> LC_MEASUREMENT="POSIX"
> LC_IDENTIFICATION="POSIX"
> LC_ALL=
> root@b99b297df111:/# pip install -e .[devel]
> root@b99b297df111:/airflow# ./run_unit_tests.sh
> + export AIRFLOW_HOME=/root/airflow
> + AIRFLOW_HOME=/root/airflow
> + export AIRFLOW__CORE__UNIT_TEST_MODE=True
> + AIRFLOW__CORE__UNIT_TEST_MODE=True
> + export AIRFLOW__TESTSECTION__TESTKEY=testvalue
> + AIRFLOW__TESTSECTION__TESTKEY=testvalue
> + export AIRFLOW_USE_NEW_IMPORTS=1
> + AIRFLOW_USE_NEW_IMPORTS=1
> +++ dirname ./run_unit_tests.sh
> ++ cd .
> ++ pwd
> + DIR=/airflow
> + export PYTHONPATH=:/airflow/tests/test_utils
> + PYTHONPATH=:/airflow/tests/test_utils
> + nose_args=
> + which airflow
> + echo 'Initializing the DB'
> Initializing the DB
> + airflow resetdb
> + yes
> Traceback (most recent call last):
> File "/usr/local/bin/airflow", line 6, in <module>
> exec(compile(open(__file__).read(), __file__, 'exec'))
> File "/airflow/airflow/bin/airflow", line 21, in <module>
> from airflow import configuration
> File "/airflow/airflow/__init__.py", line 35, in <module>
> from airflow import configuration as conf
> File "/airflow/airflow/configuration.py", line 106, in <module>
> DEFAULT_CONFIG = f.read()
> File "/usr/local/lib/python3.5/encodings/ascii.py", line 26, in decode
> return codecs.ascii_decode(input, self.errors)[0]
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 21082:
> ordinal not in range(128)
> + '[' '' ']'
> + '[' -z '' ']'
> + nose_args='--with-coverage --cover-erase --cover-html
> --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
> --with-ignore-docstrings --rednose --with-timer -s -v
> --logging-level=DEBUG '
> + echo 'Starting the unit tests with the following nose arguments:
> --with-coverage' --cover-erase --cover-html --cover-package=airflow
> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
> --rednose --with-timer -s -v --logging-level=DEBUG
> Starting the unit tests with the following nose arguments: --with-coverage
> --cover-erase --cover-html --cover-package=airflow
> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
> --rednose --with-timer -s -v --logging-level=DEBUG
> + nosetests --with-coverage --cover-erase --cover-html
> --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
> --with-ignore-docstrings --rednose --with-timer -s -v --logging-level=DEBUG
> nose.plugins.cover: ERROR: Coverage not available: unable to import
> coverage module
> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
> position 21082: ordinal not in range(128)) ... ERROR
> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
> position 21082: ordinal not in range(128)) ... ERROR
> ======================================================================
> 1) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
> in position 21082: ordinal not in range(128))
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
> runTest
> raise self.exc_val.with_traceback(self.tb)
> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
> loadTestsFromName
> addr.filename, addr.module)
> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
> importFromPath
> return self.importFromDir(dir_path, fqname)
> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
> importFromDir
> mod = load_module(part_fqname, fh, filename, desc)
> /usr/local/lib/python3.5/imp.py line 245 in load_module
> return load_package(name, filename)
> /usr/local/lib/python3.5/imp.py line 217 in load_package
> return _load(spec)
> <frozen importlib._bootstrap> line 693 in _load
>
> <frozen importlib._bootstrap> line 673 in _load_unlocked
>
> <frozen importlib._bootstrap_external> line 697 in exec_module
>
> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>
> airflow/__init__.py line 35 in <module>
> from airflow import configuration as conf
> airflow/configuration.py line 106 in <module>
> DEFAULT_CONFIG = f.read()
> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
> return codecs.ascii_decode(input, self.errors)[0]
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
> 21082: ordinal not in range(128)
> ======================================================================
> 2) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
> in position 21082: ordinal not in range(128))
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
> runTest
> raise self.exc_val.with_traceback(self.tb)
> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
> loadTestsFromName
> addr.filename, addr.module)
> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
> importFromPath
> return self.importFromDir(dir_path, fqname)
> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
> importFromDir
> mod = load_module(part_fqname, fh, filename, desc)
> /usr/local/lib/python3.5/imp.py line 245 in load_module
> return load_package(name, filename)
> /usr/local/lib/python3.5/imp.py line 217 in load_package
> return _load(spec)
> <frozen importlib._bootstrap> line 693 in _load
>
> <frozen importlib._bootstrap> line 673 in _load_unlocked
>
> <frozen importlib._bootstrap_external> line 697 in exec_module
>
> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>
> tests/__init__.py line 25 in <module>
> from .configuration import *
> tests/configuration.py line 28 in <module>
> from airflow import configuration
> airflow/__init__.py line 35 in <module>
> from airflow import configuration as conf
> airflow/configuration.py line 106 in <module>
> DEFAULT_CONFIG = f.read()
> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
> return codecs.ascii_decode(input, self.errors)[0]
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
> 21082: ordinal not in range(128)
>
> [error] 100.00% nose.failure.Failure.runTest: 0.0004s
>
> -----------------------------------------------------------------------------
> 2 tests run in 0.060 seconds.
> 2 errors (0 tests passed)
>
>
> --
> Carl Johan Gustavsson
>
> On 11 July 2018 at 23:24:01, Bolke de Bruin (bdbruin@gmail.com) wrote:
>
> Hi Carl,
>
> That is not a real check, ie. Does Airflow have the same issue clean
> install on 3.5? Travis’ tests run on 3.5.
>
> B.
>
> Verstuurd vanaf mijn iPad
>
> > Op 10 jul. 2018 om 15:10 heeft Carl Johan Gustavsson <
> carl.j.gustavsson@gmail.com> het volgende geschreven:
> >
> > Hi Bolke,
> >
> >
> > I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all but 3.7
> breaks
> >
> > Quick repro:
> >
> > ➜ ~ pyenv local 3.5.5
> > ➜ ~ locale
> > LANG=
> > LC_COLLATE="C"
> > LC_CTYPE="C"
> > LC_MESSAGES="C"
> > LC_MONETARY="C"
> > LC_NUMERIC="C"
> > LC_TIME="C"
> > LC_ALL=
> > ➜ ~ cat testweird.txt
> > ’
> > ➜ ~ python
> > Python 3.5.5 (default, Jul 7 2018, 17:00:56)
> > [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
> > Type "help", "copyright", "credits" or "license" for more information.
> > >>> open('testweird.txt').read()
> > Traceback (most recent call last):
> > File "<stdin>", line 1, in <module>
> > File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/ascii.py",
> line 26, in decode
> > return codecs.ascii_decode(input, self.errors)[0]
> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0:
> ordinal not in range(128)
> > >>>
> >
> >
> > Maybe not a blocking change but it is a breaking change from 1.9 I guess.
> >
> >
> > / Carl Johan
> >
> >
> >> On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbruin@gmail.com) wrote:
> >>
> >> Hi Carl,
> >>
> >> Did you this on python 3.5 as well? 3.6 is not an officially supported
> (yet). As a workaround is available I won’t consider this blocking btw.
> >>
> >> Bolke
> >>
> >> Verstuurd vanaf mijn iPad
> >>
> >> Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson <
> carl.j.gustavsson@gmail.com> het volgende geschreven:
> >>
> >>> Hi,
> >>>
> >>> First of all, thank you for all the work with the release management.
> >>>
> >>> I ran in to a weird issue testing the RC1, running under Python 3.6.0 /
> Ubuntu 14.04.5, upgrading from a master build from February.
> >>>
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback
> (most recent call last):
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in <module>
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
> import configuration
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>
> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/__init__.py",
> line 35, in <module>
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
> import configuration as conf
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>
> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/configuration.py",
> line 106, in <module>
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
> DEFAULT_CONFIG = f.read()
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py", line
> 26, in decode
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return
> codecs.ascii_decode(input, self.errors)[0]
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 20770:
> ordinal not in range(128)
> >>>
> >>> Removing the ’ on
>
> https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L613
> solved the issue for me, and digging a bit deeper it seems Airflow now
> requires setting LC_ALL=en_US.UTF-8 in the environment or similar to force
> Python to read the file as utf-8 and not ascii. (I think this was changed
> in to default to utf-8 in Python 3.7).
> >>>
> >>> I see 3 solutions for this
> >>> 1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or
> similar.
> >>> 2. Change the default config file to not contain non-ascii characters.
> >>> 3. Always read the file as unicode regardless of the LC_ALL
> environment, by the encoding='utf-8’ parameter to open().
> >>>
> >>> I think 3 is the best solution, and I can prepare a PR for that if
> necessary .
> >>>
> >>> I guess this counts as -1 (non-binding)
> >>>
> >>> All the best
> >>>
> >>> Carl Johan
> >>>
> >>>
> >>>
> >>>> On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbruin@gmail.com) wrote:
> >>>>
> >>>> Hey all,
> >>>>
> >>>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the
> release,
> >>>> which will last for 72 hours. Consider this my (binding) +1.
> >>>>
> >>>> Airflow 1.10.0 RC 1 is available at:
> >>>>
> >>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <
> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
> >>>>
> >>>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release
> that
> >>>> comes with INSTALL instructions.
> >>>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python
> "sdist"
> >>>> release.
> >>>>
> >>>> Public keys are available at:
> >>>>
> >>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
> https://dist.apache.org/repos/dist/release/incubator/airflow/>
> >>>>
> >>>> The amount of JIRAs fixed is over 700. Please have a look at the
> changelog.
> >>>>
> >>>> Please note that the version number excludes the `rcX` string as well
> >>>> as the "+incubating" string, so it's now simply 1.10.0. This will
> allow us
> >>>> to rename the artifact without modifying the artifact checksums when
> we
> >>>> actually release.
> >>>>
> >>>>
> >>>> Cheers,
> >>>> Bolke
>
Re: [VOTE] Airflow 1.10.0rc1
Posted by Carl Johan Gustavsson <ca...@tictail.com.INVALID>.
Hi Bolke,
(Switching email to avoid moderation on my emails.)
The normal Airflow test suite does not fail as it uses a LC_ALL set to
utf-8.
I think it is a proper test though, it is a minimal reproducible version of
the code that fails. And the only difference in behaviour is at 3.7 which
we don’t support anyway so I’m fairly sure it is broken for all supported
Python 3 versions.
I now tried running the tests in docker using 3.5 with the LC_ALL/LANG
unset and I see the same failure.
I don’t think this is a big thing though and we could release it without
the fix I made. I think most people run it with a sane LC_ALL, but
apparently we didn’t.
Here’s the log for the test:
> docker run -t -i -v `pwd`:/airflow/ python:3.5 bash
root@b99b297df111:/# locale
LANG=C.UTF-8
LANGUAGE=
LC_CTYPE="C.UTF-8"
LC_NUMERIC="C.UTF-8"
LC_TIME="C.UTF-8"
LC_COLLATE="C.UTF-8"
LC_MONETARY="C.UTF-8"
LC_MESSAGES="C.UTF-8"
LC_PAPER="C.UTF-8"
LC_NAME="C.UTF-8"
LC_ADDRESS="C.UTF-8"
LC_TELEPHONE="C.UTF-8"
LC_MEASUREMENT="C.UTF-8"
LC_IDENTIFICATION="C.UTF-8"
LC_ALL=
> unset LANG
root@b99b297df111:/# locale
LANG=
LANGUAGE=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=
root@b99b297df111:/# pip install -e .[devel]
root@b99b297df111:/airflow# ./run_unit_tests.sh
+ export AIRFLOW_HOME=/root/airflow
+ AIRFLOW_HOME=/root/airflow
+ export AIRFLOW__CORE__UNIT_TEST_MODE=True
+ AIRFLOW__CORE__UNIT_TEST_MODE=True
+ export AIRFLOW__TESTSECTION__TESTKEY=testvalue
+ AIRFLOW__TESTSECTION__TESTKEY=testvalue
+ export AIRFLOW_USE_NEW_IMPORTS=1
+ AIRFLOW_USE_NEW_IMPORTS=1
+++ dirname ./run_unit_tests.sh
++ cd .
++ pwd
+ DIR=/airflow
+ export PYTHONPATH=:/airflow/tests/test_utils
+ PYTHONPATH=:/airflow/tests/test_utils
+ nose_args=
+ which airflow
+ echo 'Initializing the DB'
Initializing the DB
+ airflow resetdb
+ yes
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 6, in <module>
exec(compile(open(__file__).read(), __file__, 'exec'))
File "/airflow/airflow/bin/airflow", line 21, in <module>
from airflow import configuration
File "/airflow/airflow/__init__.py", line 35, in <module>
from airflow import configuration as conf
File "/airflow/airflow/configuration.py", line 106, in <module>
DEFAULT_CONFIG = f.read()
File "/usr/local/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 21082:
ordinal not in range(128)
+ '[' '' ']'
+ '[' -z '' ']'
+ nose_args='--with-coverage --cover-erase --cover-html
--cover-package=airflow --cover-html-dir=airflow/www/static/coverage
--with-ignore-docstrings --rednose --with-timer -s -v
--logging-level=DEBUG '
+ echo 'Starting the unit tests with the following nose arguments:
--with-coverage' --cover-erase --cover-html --cover-package=airflow
--cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
--rednose --with-timer -s -v --logging-level=DEBUG
Starting the unit tests with the following nose arguments: --with-coverage
--cover-erase --cover-html --cover-package=airflow
--cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
--rednose --with-timer -s -v --logging-level=DEBUG
+ nosetests --with-coverage --cover-erase --cover-html
--cover-package=airflow --cover-html-dir=airflow/www/static/coverage
--with-ignore-docstrings --rednose --with-timer -s -v --logging-level=DEBUG
nose.plugins.cover: ERROR: Coverage not available: unable to import
coverage module
Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
position 21082: ordinal not in range(128)) ... ERROR
Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
position 21082: ordinal not in range(128)) ... ERROR
======================================================================
1) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
in position 21082: ordinal not in range(128))
----------------------------------------------------------------------
Traceback (most recent call last):
/usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
runTest
raise self.exc_val.with_traceback(self.tb)
/usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
loadTestsFromName
addr.filename, addr.module)
/usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
importFromPath
return self.importFromDir(dir_path, fqname)
/usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
importFromDir
mod = load_module(part_fqname, fh, filename, desc)
/usr/local/lib/python3.5/imp.py line 245 in load_module
return load_package(name, filename)
/usr/local/lib/python3.5/imp.py line 217 in load_package
return _load(spec)
<frozen importlib._bootstrap> line 693 in _load
<frozen importlib._bootstrap> line 673 in _load_unlocked
<frozen importlib._bootstrap_external> line 697 in exec_module
<frozen importlib._bootstrap> line 222 in _call_with_frames_removed
airflow/__init__.py line 35 in <module>
from airflow import configuration as conf
airflow/configuration.py line 106 in <module>
DEFAULT_CONFIG = f.read()
/usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
21082: ordinal not in range(128)
======================================================================
2) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
in position 21082: ordinal not in range(128))
----------------------------------------------------------------------
Traceback (most recent call last):
/usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
runTest
raise self.exc_val.with_traceback(self.tb)
/usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
loadTestsFromName
addr.filename, addr.module)
/usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
importFromPath
return self.importFromDir(dir_path, fqname)
/usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
importFromDir
mod = load_module(part_fqname, fh, filename, desc)
/usr/local/lib/python3.5/imp.py line 245 in load_module
return load_package(name, filename)
/usr/local/lib/python3.5/imp.py line 217 in load_package
return _load(spec)
<frozen importlib._bootstrap> line 693 in _load
<frozen importlib._bootstrap> line 673 in _load_unlocked
<frozen importlib._bootstrap_external> line 697 in exec_module
<frozen importlib._bootstrap> line 222 in _call_with_frames_removed
tests/__init__.py line 25 in <module>
from .configuration import *
tests/configuration.py line 28 in <module>
from airflow import configuration
airflow/__init__.py line 35 in <module>
from airflow import configuration as conf
airflow/configuration.py line 106 in <module>
DEFAULT_CONFIG = f.read()
/usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
21082: ordinal not in range(128)
[error] 100.00% nose.failure.Failure.runTest: 0.0004s
-----------------------------------------------------------------------------
2 tests run in 0.060 seconds.
2 errors (0 tests passed)
--
Carl Johan Gustavsson
On 11 July 2018 at 23:24:01, Bolke de Bruin (bdbruin@gmail.com) wrote:
Hi Carl,
That is not a real check, ie. Does Airflow have the same issue clean
install on 3.5? Travis’ tests run on 3.5.
B.
Verstuurd vanaf mijn iPad
> Op 10 jul. 2018 om 15:10 heeft Carl Johan Gustavsson <
carl.j.gustavsson@gmail.com> het volgende geschreven:
>
> Hi Bolke,
>
>
> I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all but 3.7
breaks
>
> Quick repro:
>
> ➜ ~ pyenv local 3.5.5
> ➜ ~ locale
> LANG=
> LC_COLLATE="C"
> LC_CTYPE="C"
> LC_MESSAGES="C"
> LC_MONETARY="C"
> LC_NUMERIC="C"
> LC_TIME="C"
> LC_ALL=
> ➜ ~ cat testweird.txt
> ’
> ➜ ~ python
> Python 3.5.5 (default, Jul 7 2018, 17:00:56)
> [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>> open('testweird.txt').read()
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/ascii.py",
line 26, in decode
> return codecs.ascii_decode(input, self.errors)[0]
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0:
ordinal not in range(128)
> >>>
>
>
> Maybe not a blocking change but it is a breaking change from 1.9 I guess.
>
>
> / Carl Johan
>
>
>> On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbruin@gmail.com) wrote:
>>
>> Hi Carl,
>>
>> Did you this on python 3.5 as well? 3.6 is not an officially supported
(yet). As a workaround is available I won’t consider this blocking btw.
>>
>> Bolke
>>
>> Verstuurd vanaf mijn iPad
>>
>> Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson <
carl.j.gustavsson@gmail.com> het volgende geschreven:
>>
>>> Hi,
>>>
>>> First of all, thank you for all the work with the release management.
>>>
>>> I ran in to a weird issue testing the RC1, running under Python 3.6.0 /
Ubuntu 14.04.5, upgrading from a master build from February.
>>>
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback
(most recent call last):
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
"/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in <module>
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
import configuration
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
"/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/__init__.py",
line 35, in <module>
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
import configuration as conf
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
"/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/configuration.py",
line 106, in <module>
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
DEFAULT_CONFIG = f.read()
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
"/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py", line
26, in decode
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return
codecs.ascii_decode(input, self.errors)[0]
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 20770:
ordinal not in range(128)
>>>
>>> Removing the ’ on
https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L613
solved the issue for me, and digging a bit deeper it seems Airflow now
requires setting LC_ALL=en_US.UTF-8 in the environment or similar to force
Python to read the file as utf-8 and not ascii. (I think this was changed
in to default to utf-8 in Python 3.7).
>>>
>>> I see 3 solutions for this
>>> 1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or
similar.
>>> 2. Change the default config file to not contain non-ascii characters.
>>> 3. Always read the file as unicode regardless of the LC_ALL
environment, by the encoding='utf-8’ parameter to open().
>>>
>>> I think 3 is the best solution, and I can prepare a PR for that if
necessary .
>>>
>>> I guess this counts as -1 (non-binding)
>>>
>>> All the best
>>>
>>> Carl Johan
>>>
>>>
>>>
>>>> On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbruin@gmail.com) wrote:
>>>>
>>>> Hey all,
>>>>
>>>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the
release,
>>>> which will last for 72 hours. Consider this my (binding) +1.
>>>>
>>>> Airflow 1.10.0 RC 1 is available at:
>>>>
>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <
https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
>>>>
>>>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release
that
>>>> comes with INSTALL instructions.
>>>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python
"sdist"
>>>> release.
>>>>
>>>> Public keys are available at:
>>>>
>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
https://dist.apache.org/repos/dist/release/incubator/airflow/>
>>>>
>>>> The amount of JIRAs fixed is over 700. Please have a look at the
changelog.
>>>>
>>>> Please note that the version number excludes the `rcX` string as well
>>>> as the "+incubating" string, so it's now simply 1.10.0. This will
allow us
>>>> to rename the artifact without modifying the artifact checksums when
we
>>>> actually release.
>>>>
>>>>
>>>> Cheers,
>>>> Bolke
Re: [VOTE] Airflow 1.10.0rc1
Posted by Bolke de Bruin <bd...@gmail.com>.
Hi Carl,
That is not a real check, ie. Does Airflow have the same issue clean install on 3.5? Travis’ tests run on 3.5.
B.
Verstuurd vanaf mijn iPad
> Op 10 jul. 2018 om 15:10 heeft Carl Johan Gustavsson <ca...@gmail.com> het volgende geschreven:
>
> Hi Bolke,
>
>
> I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all but 3.7 breaks
>
> Quick repro:
>
> ➜ ~ pyenv local 3.5.5
> ➜ ~ locale
> LANG=
> LC_COLLATE="C"
> LC_CTYPE="C"
> LC_MESSAGES="C"
> LC_MONETARY="C"
> LC_NUMERIC="C"
> LC_TIME="C"
> LC_ALL=
> ➜ ~ cat testweird.txt
> ’
> ➜ ~ python
> Python 3.5.5 (default, Jul 7 2018, 17:00:56)
> [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>> open('testweird.txt').read()
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/ascii.py", line 26, in decode
> return codecs.ascii_decode(input, self.errors)[0]
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)
> >>>
>
>
> Maybe not a blocking change but it is a breaking change from 1.9 I guess.
>
>
> / Carl Johan
>
>
>> On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbruin@gmail.com) wrote:
>>
>> Hi Carl,
>>
>> Did you this on python 3.5 as well? 3.6 is not an officially supported (yet). As a workaround is available I won’t consider this blocking btw.
>>
>> Bolke
>>
>> Verstuurd vanaf mijn iPad
>>
>> Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson <ca...@gmail.com> het volgende geschreven:
>>
>>> Hi,
>>>
>>> First of all, thank you for all the work with the release management.
>>>
>>> I ran in to a weird issue testing the RC1, running under Python 3.6.0 / Ubuntu 14.04.5, upgrading from a master build from February.
>>>
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback (most recent call last):
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in <module>
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow import configuration
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/__init__.py", line 35, in <module>
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow import configuration as conf
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/configuration.py", line 106, in <module>
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 DEFAULT_CONFIG = f.read()
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py", line 26, in decode
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return codecs.ascii_decode(input, self.errors)[0]
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 20770: ordinal not in range(128)
>>>
>>> Removing the ’ on https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L613 solved the issue for me, and digging a bit deeper it seems Airflow now requires setting LC_ALL=en_US.UTF-8 in the environment or similar to force Python to read the file as utf-8 and not ascii. (I think this was changed in to default to utf-8 in Python 3.7).
>>>
>>> I see 3 solutions for this
>>> 1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or similar.
>>> 2. Change the default config file to not contain non-ascii characters.
>>> 3. Always read the file as unicode regardless of the LC_ALL environment, by the encoding='utf-8’ parameter to open().
>>>
>>> I think 3 is the best solution, and I can prepare a PR for that if necessary .
>>>
>>> I guess this counts as -1 (non-binding)
>>>
>>> All the best
>>>
>>> Carl Johan
>>>
>>>
>>>
>>>> On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbruin@gmail.com) wrote:
>>>>
>>>> Hey all,
>>>>
>>>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the release,
>>>> which will last for 72 hours. Consider this my (binding) +1.
>>>>
>>>> Airflow 1.10.0 RC 1 is available at:
>>>>
>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
>>>>
>>>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release that
>>>> comes with INSTALL instructions.
>>>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python "sdist"
>>>> release.
>>>>
>>>> Public keys are available at:
>>>>
>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <https://dist.apache.org/repos/dist/release/incubator/airflow/>
>>>>
>>>> The amount of JIRAs fixed is over 700. Please have a look at the changelog.
>>>>
>>>> Please note that the version number excludes the `rcX` string as well
>>>> as the "+incubating" string, so it's now simply 1.10.0. This will allow us
>>>> to rename the artifact without modifying the artifact checksums when we
>>>> actually release.
>>>>
>>>>
>>>> Cheers,
>>>> Bolke
Re: [VOTE] Airflow 1.10.0rc1
Posted by Carl Johan Gustavsson <ca...@gmail.com>.
Hi Bolke,
I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all but 3.7 breaks
Quick repro:
➜ ~ pyenv local 3.5.5
➜ ~ locale
LANG=
LC_COLLATE="C"
LC_CTYPE="C"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=
➜ ~ cat testweird.txt
’
➜ ~ python
Python 3.5.5 (default, Jul 7 2018, 17:00:56)
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> open('testweird.txt').read()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/ascii.py",
line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0:
ordinal not in range(128)
>>>
Maybe not a blocking change but it is a breaking change from 1.9 I guess.
/ Carl Johan
On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbruin@gmail.com) wrote:
Hi Carl,
Did you this on python 3.5 as well? 3.6 is not an officially supported
(yet). As a workaround is available I won’t consider this blocking btw.
Bolke
Verstuurd vanaf mijn iPad
Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson <
carl.j.gustavsson@gmail.com> het volgende geschreven:
Hi,
First of all, thank you for all the work with the release management.
I ran in to a weird issue testing the RC1, running under Python 3.6.0 /
Ubuntu 14.04.5, upgrading from a master build from February.
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback (most
recent call last):
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
"/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in <module>
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
import configuration
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
"/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/__init__.py",
line 35, in <module>
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
import configuration as conf
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
"/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/configuration.py",
line 106, in <module>
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
DEFAULT_CONFIG = f.read()
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
"/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py", line
26, in decode
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return
codecs.ascii_decode(input, self.errors)[0]
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 20770:
ordinal not in range(128)
Removing the ’ on
https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L613
solved
the issue for me, and digging a bit deeper it seems Airflow now requires
setting LC_ALL=en_US.UTF-8 in the environment or similar to force Python to
read the file as utf-8 and not ascii. (I think this was changed in to
default to utf-8 in Python 3.7).
I see 3 solutions for this
1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or similar.
2. Change the default config file to not contain non-ascii characters.
3. Always read the file as unicode regardless of the LC_ALL environment,
by the encoding='utf-8’ parameter to open().
I think 3 is the best solution, and I can prepare a PR for that if
necessary .
I guess this counts as -1 (non-binding)
All the best
Carl Johan
On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbruin@gmail.com) wrote:
Hey all,
I have cut Airflow 1.10.0 RC1. This email is calling a vote on the release,
which will last for 72 hours. Consider this my (binding) +1.
Airflow 1.10.0 RC 1 is available at:
https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <
https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release that
comes with INSTALL instructions.
apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python "sdist"
release.
Public keys are available at:
https://dist.apache.org/repos/dist/release/incubator/airflow/ <
https://dist.apache.org/repos/dist/release/incubator/airflow/>
The amount of JIRAs fixed is over 700. Please have a look at the changelog.
Please note that the version number excludes the `rcX` string as well
as the "+incubating" string, so it's now simply 1.10.0. This will allow us
to rename the artifact without modifying the artifact checksums when we
actually release.
Cheers,
Bolke
Re: [VOTE] Airflow 1.10.0rc1
Posted by Bolke de Bruin <bd...@gmail.com>.
Hi Carl,
Did you this on python 3.5 as well? 3.6 is not an officially supported (yet). As a workaround is available I won’t consider this blocking btw.
Bolke
Verstuurd vanaf mijn iPad
> Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson <ca...@gmail.com> het volgende geschreven:
>
> Hi,
>
> First of all, thank you for all the work with the release management.
>
> I ran in to a weird issue testing the RC1, running under Python 3.6.0 / Ubuntu 14.04.5, upgrading from a master build from February.
>
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback (most recent call last):
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in <module>
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow import configuration
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/__init__.py", line 35, in <module>
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow import configuration as conf
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/configuration.py", line 106, in <module>
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 DEFAULT_CONFIG = f.read()
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py", line 26, in decode
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return codecs.ascii_decode(input, self.errors)[0]
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 20770: ordinal not in range(128)
>
> Removing the ’ on https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L613 solved the issue for me, and digging a bit deeper it seems Airflow now requires setting LC_ALL=en_US.UTF-8 in the environment or similar to force Python to read the file as utf-8 and not ascii. (I think this was changed in to default to utf-8 in Python 3.7).
>
> I see 3 solutions for this
> 1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or similar.
> 2. Change the default config file to not contain non-ascii characters.
> 3. Always read the file as unicode regardless of the LC_ALL environment, by the encoding='utf-8’ parameter to open().
>
> I think 3 is the best solution, and I can prepare a PR for that if necessary .
>
> I guess this counts as -1 (non-binding)
>
> All the best
>
> Carl Johan
>
>
>
>> On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbruin@gmail.com) wrote:
>>
>> Hey all,
>>
>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the release,
>> which will last for 72 hours. Consider this my (binding) +1.
>>
>> Airflow 1.10.0 RC 1 is available at:
>>
>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
>>
>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release that
>> comes with INSTALL instructions.
>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python "sdist"
>> release.
>>
>> Public keys are available at:
>>
>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <https://dist.apache.org/repos/dist/release/incubator/airflow/>
>>
>> The amount of JIRAs fixed is over 700. Please have a look at the changelog.
>>
>> Please note that the version number excludes the `rcX` string as well
>> as the "+incubating" string, so it's now simply 1.10.0. This will allow us
>> to rename the artifact without modifying the artifact checksums when we
>> actually release.
>>
>>
>> Cheers,
>> Bolke
Re: [VOTE] Airflow 1.10.0rc1
Posted by Carl Johan Gustavsson <ca...@gmail.com>.
Hi,
First of all, thank you for all the work with the release management.
I ran in to a weird issue testing the RC1, running under Python 3.6.0 / Ubuntu 14.04.5, upgrading from a master build from February.
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback (most recent call last):
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in <module>
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow import configuration
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/__init__.py", line 35, in <module>
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow import configuration as conf
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/configuration.py", line 106, in <module>
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 DEFAULT_CONFIG = f.read()
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py", line 26, in decode
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return codecs.ascii_decode(input, self.errors)[0]
Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 20770: ordinal not in range(128)
Removing the ’ on https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L613 solved the issue for me, and digging a bit deeper it seems Airflow now requires setting LC_ALL=en_US.UTF-8 in the environment or similar to force Python to read the file as utf-8 and not ascii. (I think this was changed in to default to utf-8 in Python 3.7).
I see 3 solutions for this
1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or similar.
2. Change the default config file to not contain non-ascii characters.
3. Always read the file as unicode regardless of the LC_ALL environment, by the encoding='utf-8’ parameter to open().
I think 3 is the best solution, and I can prepare a PR for that if necessary .
I guess this counts as -1 (non-binding)
All the best
Carl Johan
On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbruin@gmail.com) wrote:
Hey all,
I have cut Airflow 1.10.0 RC1. This email is calling a vote on the release,
which will last for 72 hours. Consider this my (binding) +1.
Airflow 1.10.0 RC 1 is available at:
https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release that
comes with INSTALL instructions.
apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python "sdist"
release.
Public keys are available at:
https://dist.apache.org/repos/dist/release/incubator/airflow/ <https://dist.apache.org/repos/dist/release/incubator/airflow/>
The amount of JIRAs fixed is over 700. Please have a look at the changelog.
Please note that the version number excludes the `rcX` string as well
as the "+incubating" string, so it's now simply 1.10.0. This will allow us
to rename the artifact without modifying the artifact checksums when we
actually release.
Cheers,
Bolke