You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/07/12 17:25:00 UTC
[jira] [Work logged] (BEAM-4749) fastavro breaks macos tests
[ https://issues.apache.org/jira/browse/BEAM-4749?focusedWorklogId=122440&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-122440 ]
ASF GitHub Bot logged work on BEAM-4749:
----------------------------------------
Author: ASF GitHub Bot
Created on: 12/Jul/18 17:24
Start Date: 12/Jul/18 17:24
Worklog Time Spent: 10m
Work Description: aaltay opened a new pull request #5939: [BEAM-4749] Install fastavro only in linux
URL: https://github.com/apache/beam/pull/5939
Installing fastavro breaks in MacOS at compilation time. This in turn prevents SDK from being installed. Until we have tests and know that fastavro installs cleanly in other OSes I will limit it to linux only.
R: @udim
cc: @ryan-williams
------------------------
Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it.
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) </br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | --- | --- | --- | ---
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 122440)
Time Spent: 10m
Remaining Estimate: 0h
> fastavro breaks macos tests
> ---------------------------
>
> Key: BEAM-4749
> URL: https://issues.apache.org/jira/browse/BEAM-4749
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
> Reporter: Ahmet Altay
> Assignee: Chamikara Jayalath
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Recent addition of the fastavro dependency is breaking python linter in macos. At least for some cases, because it requires a compiler.
> Could we optionally depend on fastavro, and fallback to regular avro package?
>
> Log:
> *> Task :beam-sdks-python:lintPy27*
> ERROR: invocation failed (exit code 1), logfile: /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/log/py27-lint-2.log
> ERROR: actionid: py27-lint
> msg: installpkg
> cmdargs: ['/Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python', '/Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/pip', 'install', '/Users/relax/beam-gradle/beam/sdks/python/target/.tox/dist/apache-beam-2.6.0.dev0.zip[test]']
>
> Processing ./target/.tox/dist/apache-beam-2.6.0.dev0.zip
> Collecting avro<2.0.0,>=1.8.1 (from apache-beam==2.6.0.dev0)
> Collecting crcmod<2.0,>=1.7 (from apache-beam==2.6.0.dev0)
> Collecting dill==0.2.6 (from apache-beam==2.6.0.dev0)
> Collecting fastavro==0.19.7 (from apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/a7/0a/b08ba5cef63c675e8442c2bf1cbcef90c8b9f824be2202d492f0cedb0913/fastavro-0.19.7.tar.gz]
> Collecting grpcio<2,>=1.8 (from apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/66/89/4a90caabd51c17686cbb48a9bbe8c592c4be929c0d2542d2ffde76b0d671/grpcio-1.13.0-cp27-cp27m-macosx_10_12_x86_64.whl]
> Collecting hdfs<3.0.0,>=2.1.0 (from apache-beam==2.6.0.dev0)
> Collecting httplib2<=0.11.3,>=0.8 (from apache-beam==2.6.0.dev0)
> Collecting mock<3.0.0,>=1.0.1 (from apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/e6/35/f187bdf23be87092bd0f1200d43d23076cee4d0dec109f195173fd3ebc79/mock-2.0.0-py2.py3-none-any.whl]
> Collecting oauth2client<5,>=2.0.1 (from apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/82/d8/3eab58811282ac7271a081ba5c0d4b875ce786ca68ce43e2a62ade32e9a8/oauth2client-4.1.2-py2.py3-none-any.whl]
> Collecting protobuf<4,>=3.5.0.post1 (from apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/4f/56/a21f2d077ceae7fd521c0ed31fb8bb1c7f13ffbb09bf7dd27de6cf6bad08/protobuf-3.6.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl]
> Collecting pytz<=2018.4,>=2018.3 (from apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/dc/83/15f7833b70d3e067ca91467ca245bae0f6fe56ddc7451aa0dc5606b120f2/pytz-2018.4-py2.py3-none-any.whl]
> Collecting pyyaml<4.0.0,>=3.12 (from apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/4a/85/db5a2df477072b2902b0eb892feb37d88ac635d36245a72a6a69b23b383a/PyYAML-3.12.tar.gz]
> Collecting pyvcf<0.7.0,>=0.6.8 (from apache-beam==2.6.0.dev0)
> Requirement already satisfied: six<1.12,>=1.9 in ./target/.tox/py27-lint/lib/python2.7/site-packages (from apache-beam==2.6.0.dev0) (1.11.0)
> Collecting typing<3.7.0,>=3.6.0 (from apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/0d/4d/4e5985d075d241d686a1663fa1f88b61d544658d08c1375c7c6aac32afc3/typing-3.6.4-py2-none-any.whl]
> Collecting futures<4.0.0,>=3.1.1 (from apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl]
> Requirement already satisfied: future<1.0.0,>=0.16.0 in ./target/.tox/py27-lint/lib/python2.7/site-packages (from apache-beam==2.6.0.dev0) (0.16.0)
> Collecting nose>=1.3.7 (from apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/99/4f/13fb671119e65c4dce97c60e67d3fd9e6f7f809f2b307e2611f4701205cb/nose-1.3.7-py2-none-any.whl]
> Collecting pyhamcrest<2.0,>=1.9 (from apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/9a/d5/d37fd731b7d0e91afcc84577edeccf4638b4f9b82f5ffe2f8b62e2ddc609/PyHamcrest-1.9.0-py2.py3-none-any.whl]
> Requirement already satisfied: setuptools>=18.0 in ./target/.tox/py27-lint/lib/python2.7/site-packages (from fastavro==0.19.7->apache-beam==2.6.0.dev0) (39.2.0)
> Requirement already satisfied: enum34>=1.0.4 in ./target/.tox/py27-lint/lib/python2.7/site-packages (from grpcio<2,>=1.8->apache-beam==2.6.0.dev0) (1.1.6)
> Collecting docopt (from hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
> Collecting requests>=2.7.0 (from hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/65/47/7e02164a2a3db50ed6d8a6ab1d6d60b69c4c3fdf57a284257925dfc12bda/requests-2.19.1-py2.py3-none-any.whl]
> Collecting pbr>=0.11 (from mock<3.0.0,>=1.0.1->apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/b3/5d/c196041ffdf3e34ba206db6d61d1f893a75e1f3435699ade9bd65e089a3d/pbr-4.0.4-py2.py3-none-any.whl]
> Collecting funcsigs>=1; python_version < "3.3" (from mock<3.0.0,>=1.0.1->apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/69/cb/f5be453359271714c01b9bd06126eaf2e368f1fddfff30818754b5ac2328/funcsigs-1.0.2-py2.py3-none-any.whl]
> Collecting rsa>=3.1.4 (from oauth2client<5,>=2.0.1->apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/e1/ae/baedc9cb175552e95f3395c43055a6a5e125ae4d48a1d7a924baca83e92e/rsa-3.4.2-py2.py3-none-any.whl]
> Collecting pyasn1>=0.1.7 (from oauth2client<5,>=2.0.1->apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/a0/70/2c27740f08e477499ce19eefe05dbcae6f19fdc49e9e82ce4768be0643b9/pyasn1-0.4.3-py2.py3-none-any.whl]
> Collecting pyasn1-modules>=0.0.5 (from oauth2client<5,>=2.0.1->apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/19/02/fa63f7ba30a0d7b925ca29d034510fc1ffde53264b71b4155022ddf3ab5d/pyasn1_modules-0.2.2-py2.py3-none-any.whl]
> Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl]
> Collecting idna<2.8,>=2.5 (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl]
> Collecting urllib3<1.24,>=1.21.1 (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl]
> Collecting certifi>=2017.4.17 (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl]
> Building wheels for collected packages: apache-beam, fastavro, pyyaml
> Running setup.py bdist_wheel for apache-beam: started
> Running setup.py bdist_wheel for apache-beam: finished with status 'done'
> Stored in directory: /Users/relax/Library/Caches/pip/wheels/a3/18/08/60ce57e27e8b04460419c9463b9b6da19a8aab9fb64afc612f
> Running setup.py bdist_wheel for fastavro: started
> Running setup.py bdist_wheel for fastavro: finished with status 'error'
> Complete output from command /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-install-WX5Ntf/fastavro/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-wheel-p6UGao --python-tag cp27:
> running bdist_wheel
> running build
> running build_py
> creating build
> creating build/lib.macosx-10.4-x86_64-2.7
> creating build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_schema_common.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_schema_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/write.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_write_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/__init__.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_read_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_read_common.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_validate_common.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_validation_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/six.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_timezone.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/__main__.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/const.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/schema.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/read.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/validation.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> running build_ext
> building 'fastavro._read' extension
> creating build/temp.macosx-10.4-x86_64-2.7
> creating build/temp.macosx-10.4-x86_64-2.7/fastavro
> gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -I/Library/GoogleCorpSupport/Python/2.7/include/python2.7 -c fastavro/_read.c -o build/temp.macosx-10.4-x86_64-2.7/fastavro/_read.o
> xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
> error: command 'gcc' failed with exit status 1
>
> ----------------------------------------
> Failed building wheel for fastavro
> Running setup.py clean for fastavro
> Running setup.py bdist_wheel for pyyaml: started
> Running setup.py bdist_wheel for pyyaml: finished with status 'done'
> Stored in directory: /Users/relax/Library/Caches/pip/wheels/03/05/65/bdc14f2c6e09e82ae3e0f13d021e1b6b2481437ea2f207df3f
> Successfully built apache-beam pyyaml
> Failed to build fastavro
> Installing collected packages: avro, crcmod, dill, fastavro, futures, grpcio, docopt, chardet, idna, urllib3, certifi, requests, hdfs, httplib2, pbr, funcsigs, mock, pyasn1, rsa, pyasn1-modules, oauth2client, protobuf, pytz, pyyaml, pyvcf, typing, nose, pyhamcrest, apache-beam
> Running setup.py install for fastavro: started
> Running setup.py install for fastavro: finished with status 'error'
> Complete output from command /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-install-WX5Ntf/fastavro/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-record-sAEfXi/install-record.txt --single-version-externally-managed --compile --install-headers /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/include/site/python2.7/fastavro:
> running install
> running build
> running build_py
> creating build
> creating build/lib.macosx-10.4-x86_64-2.7
> creating build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_schema_common.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_schema_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/write.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_write_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/__init__.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_read_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_read_common.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_validate_common.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_validation_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/six.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_timezone.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/__main__.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/const.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/schema.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/read.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/validation.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> running build_ext
> building 'fastavro._read' extension
> creating build/temp.macosx-10.4-x86_64-2.7
> creating build/temp.macosx-10.4-x86_64-2.7/fastavro
> gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -I/Library/GoogleCorpSupport/Python/2.7/include/python2.7 -c fastavro/_read.c -o build/temp.macosx-10.4-x86_64-2.7/fastavro/_read.o
> xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
> error: command 'gcc' failed with exit status 1
>
> ----------------------------------------
> Command "/Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-install-WX5Ntf/fastavro/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-record-sAEfXi/install-record.txt --single-version-externally-managed --compile --install-headers /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/include/site/python2.7/fastavro" failed with error code 1 in /private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-install-WX5Ntf/fastavro/
>
> py27-lint installed: astroid==1.6.5,avro==1.8.2,backports.functools-lru-cache==1.5,configparser==3.5.0,crcmod==1.7,dill==0.2.6,enum34==1.1.6,flake8==3.5.0,future==0.16.0,isort==4.2.15,lazy-object-proxy==1.3.1,mccabe==0.6.1,pycodestyle==2.3.1,pyflakes==1.6.0,pylint==1.7.2,singledispatch==3.4.0.3,six==1.11.0,wrapt==1.10.11
> ___________________________________ summary ____________________________________
> ERROR: py27-lint: InvocationError for command /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/pip install /Users/relax/beam-gradle/beam/sdks/python/target/.tox/dist/apache-beam-2.6.0.dev0.zip[test] (see /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/log/py27-lint-2.log) (exited with code 1)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)