You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/08/02 15:45:00 UTC
[jira] [Work logged] (BEAM-4749) fastavro breaks macos tests
[ https://issues.apache.org/jira/browse/BEAM-4749?focusedWorklogId=130289&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-130289 ]
ASF GitHub Bot logged work on BEAM-4749:
----------------------------------------
Author: ASF GitHub Bot
Created on: 02/Aug/18 15:44
Start Date: 02/Aug/18 15:44
Worklog Time Spent: 10m
Work Description: aaltay closed pull request #6100: [BEAM-4749] Revert "Install fastavro only in linux"
URL: https://github.com/apache/beam/pull/6100
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/sdks/python/apache_beam/io/avroio.py b/sdks/python/apache_beam/io/avroio.py
index 7b654d02d80..f90dc3c6833 100644
--- a/sdks/python/apache_beam/io/avroio.py
+++ b/sdks/python/apache_beam/io/avroio.py
@@ -53,6 +53,8 @@
from avro import io as avroio
from avro import datafile
from avro import schema
+from fastavro.read import block_reader
+from fastavro.write import Writer
import apache_beam as beam
from apache_beam.io import filebasedsink
@@ -441,9 +443,6 @@ def split_points_unclaimed(stop_position):
start_offset = 0
with self.open_file(file_name) as f:
- # TODO(BEAM-4749): fastavro fails to install in MacOS.
- from fastavro.read import block_reader # pylint: disable=wrong-import-position
-
blocks = block_reader(f)
sync_marker = blocks._header['sync']
@@ -597,9 +596,6 @@ def display_data(self):
class _FastAvroSink(_AvroSink):
"""A sink for avro files that uses the `fastavro` library"""
def open(self, temp_path):
- # TODO(BEAM-4749): fastavro fails to install in MacOS.
- from fastavro.write import Writer # pylint: disable=wrong-import-position
-
file_handle = super(_AvroSink, self).open(temp_path)
return Writer(file_handle, self._schema.to_json(), self._codec)
diff --git a/sdks/python/apache_beam/io/avroio_test.py b/sdks/python/apache_beam/io/avroio_test.py
index 214753965fa..93f2ba9ebfd 100644
--- a/sdks/python/apache_beam/io/avroio_test.py
+++ b/sdks/python/apache_beam/io/avroio_test.py
@@ -453,13 +453,6 @@ def __init__(self, methodName='runTest'):
super(TestFastAvro, self).__init__(methodName)
self.use_fastavro = True
- def setUp(self):
- super(TestFastAvro, self).setUp()
- try:
- from fastavro.read import block_reader # pylint: disable=unused-variable
- except ImportError:
- self.skipTest("fastavro is not installed.")
-
if __name__ == '__main__':
logging.getLogger().setLevel(logging.INFO)
diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index 3bf4da0d21b..2cdd374522b 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -98,6 +98,7 @@ def get_version():
'avro>=1.8.1,<2.0.0',
'crcmod>=1.7,<2.0',
'dill>=0.2.6,<=0.2.8.2',
+ 'fastavro==0.19.7',
'grpcio>=1.8,<2',
'hdfs>=2.1.0,<3.0.0',
'httplib2>=0.8,<=0.11.3',
@@ -115,14 +116,6 @@ def get_version():
'future>=0.16.0,<1.0.0',
]
-REQUIRED_PACKAGES_LINUX_ONLY = [
- 'fastavro==0.19.7',
-]
-
-# TODO(BEAM-4749): fastavro fails to install in MacOS.
-if 'Linux' in platform.system():
- REQUIRED_PACKAGES.extend(REQUIRED_PACKAGES_LINUX_ONLY)
-
REQUIRED_TEST_PACKAGES = [
'nose>=1.3.7',
'numpy>=1.14.3,<2',
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 130289)
Time Spent: 50m (was: 40m)
> fastavro breaks macos tests
> ---------------------------
>
> Key: BEAM-4749
> URL: https://issues.apache.org/jira/browse/BEAM-4749
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
> Reporter: Ahmet Altay
> Assignee: Ryan Williams
> Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Recent addition of the fastavro dependency is breaking python linter in macos. At least for some cases, because it requires a compiler.
> Could we optionally depend on fastavro, and fallback to regular avro package?
>
> Log:
> *> Task :beam-sdks-python:lintPy27*
> ERROR: invocation failed (exit code 1), logfile: /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/log/py27-lint-2.log
> ERROR: actionid: py27-lint
> msg: installpkg
> cmdargs: ['/Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python', '/Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/pip', 'install', '/Users/relax/beam-gradle/beam/sdks/python/target/.tox/dist/apache-beam-2.6.0.dev0.zip[test]']
>
> Processing ./target/.tox/dist/apache-beam-2.6.0.dev0.zip
> Collecting avro<2.0.0,>=1.8.1 (from apache-beam==2.6.0.dev0)
> Collecting crcmod<2.0,>=1.7 (from apache-beam==2.6.0.dev0)
> Collecting dill==0.2.6 (from apache-beam==2.6.0.dev0)
> Collecting fastavro==0.19.7 (from apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/a7/0a/b08ba5cef63c675e8442c2bf1cbcef90c8b9f824be2202d492f0cedb0913/fastavro-0.19.7.tar.gz]
> Collecting grpcio<2,>=1.8 (from apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/66/89/4a90caabd51c17686cbb48a9bbe8c592c4be929c0d2542d2ffde76b0d671/grpcio-1.13.0-cp27-cp27m-macosx_10_12_x86_64.whl]
> Collecting hdfs<3.0.0,>=2.1.0 (from apache-beam==2.6.0.dev0)
> Collecting httplib2<=0.11.3,>=0.8 (from apache-beam==2.6.0.dev0)
> Collecting mock<3.0.0,>=1.0.1 (from apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/e6/35/f187bdf23be87092bd0f1200d43d23076cee4d0dec109f195173fd3ebc79/mock-2.0.0-py2.py3-none-any.whl]
> Collecting oauth2client<5,>=2.0.1 (from apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/82/d8/3eab58811282ac7271a081ba5c0d4b875ce786ca68ce43e2a62ade32e9a8/oauth2client-4.1.2-py2.py3-none-any.whl]
> Collecting protobuf<4,>=3.5.0.post1 (from apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/4f/56/a21f2d077ceae7fd521c0ed31fb8bb1c7f13ffbb09bf7dd27de6cf6bad08/protobuf-3.6.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl]
> Collecting pytz<=2018.4,>=2018.3 (from apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/dc/83/15f7833b70d3e067ca91467ca245bae0f6fe56ddc7451aa0dc5606b120f2/pytz-2018.4-py2.py3-none-any.whl]
> Collecting pyyaml<4.0.0,>=3.12 (from apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/4a/85/db5a2df477072b2902b0eb892feb37d88ac635d36245a72a6a69b23b383a/PyYAML-3.12.tar.gz]
> Collecting pyvcf<0.7.0,>=0.6.8 (from apache-beam==2.6.0.dev0)
> Requirement already satisfied: six<1.12,>=1.9 in ./target/.tox/py27-lint/lib/python2.7/site-packages (from apache-beam==2.6.0.dev0) (1.11.0)
> Collecting typing<3.7.0,>=3.6.0 (from apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/0d/4d/4e5985d075d241d686a1663fa1f88b61d544658d08c1375c7c6aac32afc3/typing-3.6.4-py2-none-any.whl]
> Collecting futures<4.0.0,>=3.1.1 (from apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl]
> Requirement already satisfied: future<1.0.0,>=0.16.0 in ./target/.tox/py27-lint/lib/python2.7/site-packages (from apache-beam==2.6.0.dev0) (0.16.0)
> Collecting nose>=1.3.7 (from apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/99/4f/13fb671119e65c4dce97c60e67d3fd9e6f7f809f2b307e2611f4701205cb/nose-1.3.7-py2-none-any.whl]
> Collecting pyhamcrest<2.0,>=1.9 (from apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/9a/d5/d37fd731b7d0e91afcc84577edeccf4638b4f9b82f5ffe2f8b62e2ddc609/PyHamcrest-1.9.0-py2.py3-none-any.whl]
> Requirement already satisfied: setuptools>=18.0 in ./target/.tox/py27-lint/lib/python2.7/site-packages (from fastavro==0.19.7->apache-beam==2.6.0.dev0) (39.2.0)
> Requirement already satisfied: enum34>=1.0.4 in ./target/.tox/py27-lint/lib/python2.7/site-packages (from grpcio<2,>=1.8->apache-beam==2.6.0.dev0) (1.1.6)
> Collecting docopt (from hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
> Collecting requests>=2.7.0 (from hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/65/47/7e02164a2a3db50ed6d8a6ab1d6d60b69c4c3fdf57a284257925dfc12bda/requests-2.19.1-py2.py3-none-any.whl]
> Collecting pbr>=0.11 (from mock<3.0.0,>=1.0.1->apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/b3/5d/c196041ffdf3e34ba206db6d61d1f893a75e1f3435699ade9bd65e089a3d/pbr-4.0.4-py2.py3-none-any.whl]
> Collecting funcsigs>=1; python_version < "3.3" (from mock<3.0.0,>=1.0.1->apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/69/cb/f5be453359271714c01b9bd06126eaf2e368f1fddfff30818754b5ac2328/funcsigs-1.0.2-py2.py3-none-any.whl]
> Collecting rsa>=3.1.4 (from oauth2client<5,>=2.0.1->apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/e1/ae/baedc9cb175552e95f3395c43055a6a5e125ae4d48a1d7a924baca83e92e/rsa-3.4.2-py2.py3-none-any.whl]
> Collecting pyasn1>=0.1.7 (from oauth2client<5,>=2.0.1->apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/a0/70/2c27740f08e477499ce19eefe05dbcae6f19fdc49e9e82ce4768be0643b9/pyasn1-0.4.3-py2.py3-none-any.whl]
> Collecting pyasn1-modules>=0.0.5 (from oauth2client<5,>=2.0.1->apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/19/02/fa63f7ba30a0d7b925ca29d034510fc1ffde53264b71b4155022ddf3ab5d/pyasn1_modules-0.2.2-py2.py3-none-any.whl]
> Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl]
> Collecting idna<2.8,>=2.5 (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl]
> Collecting urllib3<1.24,>=1.21.1 (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl]
> Collecting certifi>=2017.4.17 (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
> Using cached [https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl]
> Building wheels for collected packages: apache-beam, fastavro, pyyaml
> Running setup.py bdist_wheel for apache-beam: started
> Running setup.py bdist_wheel for apache-beam: finished with status 'done'
> Stored in directory: /Users/relax/Library/Caches/pip/wheels/a3/18/08/60ce57e27e8b04460419c9463b9b6da19a8aab9fb64afc612f
> Running setup.py bdist_wheel for fastavro: started
> Running setup.py bdist_wheel for fastavro: finished with status 'error'
> Complete output from command /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-install-WX5Ntf/fastavro/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-wheel-p6UGao --python-tag cp27:
> running bdist_wheel
> running build
> running build_py
> creating build
> creating build/lib.macosx-10.4-x86_64-2.7
> creating build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_schema_common.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_schema_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/write.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_write_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/__init__.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_read_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_read_common.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_validate_common.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_validation_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/six.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_timezone.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/__main__.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/const.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/schema.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/read.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/validation.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> running build_ext
> building 'fastavro._read' extension
> creating build/temp.macosx-10.4-x86_64-2.7
> creating build/temp.macosx-10.4-x86_64-2.7/fastavro
> gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -I/Library/GoogleCorpSupport/Python/2.7/include/python2.7 -c fastavro/_read.c -o build/temp.macosx-10.4-x86_64-2.7/fastavro/_read.o
> xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
> error: command 'gcc' failed with exit status 1
>
> ----------------------------------------
> Failed building wheel for fastavro
> Running setup.py clean for fastavro
> Running setup.py bdist_wheel for pyyaml: started
> Running setup.py bdist_wheel for pyyaml: finished with status 'done'
> Stored in directory: /Users/relax/Library/Caches/pip/wheels/03/05/65/bdc14f2c6e09e82ae3e0f13d021e1b6b2481437ea2f207df3f
> Successfully built apache-beam pyyaml
> Failed to build fastavro
> Installing collected packages: avro, crcmod, dill, fastavro, futures, grpcio, docopt, chardet, idna, urllib3, certifi, requests, hdfs, httplib2, pbr, funcsigs, mock, pyasn1, rsa, pyasn1-modules, oauth2client, protobuf, pytz, pyyaml, pyvcf, typing, nose, pyhamcrest, apache-beam
> Running setup.py install for fastavro: started
> Running setup.py install for fastavro: finished with status 'error'
> Complete output from command /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-install-WX5Ntf/fastavro/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-record-sAEfXi/install-record.txt --single-version-externally-managed --compile --install-headers /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/include/site/python2.7/fastavro:
> running install
> running build
> running build_py
> creating build
> creating build/lib.macosx-10.4-x86_64-2.7
> creating build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_schema_common.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_schema_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/write.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_write_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/__init__.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_read_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_read_common.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_validate_common.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_validation_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/six.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_timezone.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/__main__.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/const.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/schema.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/read.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/validation.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> running build_ext
> building 'fastavro._read' extension
> creating build/temp.macosx-10.4-x86_64-2.7
> creating build/temp.macosx-10.4-x86_64-2.7/fastavro
> gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -I/Library/GoogleCorpSupport/Python/2.7/include/python2.7 -c fastavro/_read.c -o build/temp.macosx-10.4-x86_64-2.7/fastavro/_read.o
> xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
> error: command 'gcc' failed with exit status 1
>
> ----------------------------------------
> Command "/Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-install-WX5Ntf/fastavro/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-record-sAEfXi/install-record.txt --single-version-externally-managed --compile --install-headers /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/include/site/python2.7/fastavro" failed with error code 1 in /private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-install-WX5Ntf/fastavro/
>
> py27-lint installed: astroid==1.6.5,avro==1.8.2,backports.functools-lru-cache==1.5,configparser==3.5.0,crcmod==1.7,dill==0.2.6,enum34==1.1.6,flake8==3.5.0,future==0.16.0,isort==4.2.15,lazy-object-proxy==1.3.1,mccabe==0.6.1,pycodestyle==2.3.1,pyflakes==1.6.0,pylint==1.7.2,singledispatch==3.4.0.3,six==1.11.0,wrapt==1.10.11
> ___________________________________ summary ____________________________________
> ERROR: py27-lint: InvocationError for command /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/pip install /Users/relax/beam-gradle/beam/sdks/python/target/.tox/dist/apache-beam-2.6.0.dev0.zip[test] (see /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/log/py27-lint-2.log) (exited with code 1)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)