You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Kyle Weaver <kc...@google.com> on 2020/04/10 17:01:03 UTC
Beam 2.21 release update
Hi everyone,
Just a heads up that the Beam 2.21 release branch [1] is cut.
- If you find any important issues that you think should be addressed in
the release, please tag the jira with fix version 2.21.0 and cc me
(username `ibzib`).
- Make sure to update the change log [2] with any significant changes if
you haven't already. Send a PR with the change and tag me. (I imagine I'm
not the only one who forgot to do this :).)
Thanks,
Kyle
[1] https://github.com/apache/beam/blob/release-2.21.0
[2] https://github.com/apache/beam/blob/master/CHANGES.md
Re: Beam 2.21 release update
Posted by Kyle Weaver <kc...@google.com>.
Thanks for the heads up Thomas. Please let us know as soon as possible what
you find.
On Fri, May 8, 2020 at 2:43 PM Thomas Weise <th...@apache.org> wrote:
> I could not find a way for the hidden pip install to succeed within the
> virtual environment.
>
> Also, heads up that we found another issue related to timer encoding that
> looks like a release blocker. Details to follow.
>
>
> On Fri, May 8, 2020 at 9:53 AM Udi Meiri <eh...@google.com> wrote:
>
>> +Chad Dombrova <ch...@gmail.com> , who added _find_protoc_gen_mypy.
>>
>> I'm guessing that the code
>> in _install_grpcio_tools_and_generate_proto_files creates a kind of
>> virtualenv, but it only works well for staging Python modules and not
>> binaries like protoc-gen-mypy.
>> (I assume there's a reason why it doesn't invoke virtualenv, probably
>> since the list of things setup.py can expect to be installed is very
>> minimal (setuptools).)
>>
>> One solution would be to make these setup.py dependencies explicit in
>> pyproject.toml, such that pip installs them before running setup.py:
>> https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support
>> It would help when using tools like pip ("pip wheel"), but I'm not sure
>> what the alternative for "python setup.py sdist" is.
>>
>>
>> On Thu, May 7, 2020 at 10:40 PM Thomas Weise <th...@apache.org> wrote:
>>
>>> No additional stacktraces. Full error output below.
>>>
>>> It's not clear what is going wrong.
>>>
>>> There isn't any exception from the subprocess execution since the
>>> "WARNING:root:Installing grpcio-tools took 305.39 seconds." is printed.
>>>
>>> Also, the time it takes to perform the install is equivalent to
>>> successfully running the pip command.
>>>
>>> I will report back if I find anything else. Currently doing the
>>> explicit install via pip install -r sdks/python/build-requirements.txt
>>>
>>> Thanks,
>>> Thomas
>>>
>>> WARNING:root:Installing grpcio-tools took 269.27 seconds.
>>> INFO:gen_protos:Regenerating Python proto definitions (no output files).
>>> Process Process-1:
>>> Traceback (most recent call last):
>>> File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in
>>> _bootstrap
>>> self.run()
>>> File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
>>> self._target(*self._args, **self._kwargs)
>>> File
>>> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
>>> 378, in _install_grpcio_tools_and_generate_proto_files
>>> generate_proto_files(force=force)
>>> File
>>> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
>>> 315, in generate_proto_files
>>> protoc_gen_mypy = _find_protoc_gen_mypy()
>>> File
>>> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
>>> 233, in _find_protoc_gen_mypy
>>> (fname, ', '.join(search_paths)))
>>> RuntimeError: Could not find protoc-gen-mypy in /code/venvs/venv2/bin,
>>> /code/venvs/venv2/bin, /code/venvs/venv3/bin, /usr/local/sbin,
>>> /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin
>>> Traceback (most recent call last):
>>> File "setup.py", line 311, in <module>
>>> 'mypy': generate_protos_first(mypy),
>>> File
>>> "/code/venvs/venv2/local/lib/python2.7/site-packages/setuptools/__init__.py",
>>> line 129, in setup
>>> return distutils.core.setup(**attrs)
>>> File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
>>> dist.run_commands()
>>> File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
>>> self.run_command(cmd)
>>> File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
>>> cmd_obj.run()
>>> File
>>> "/code/venvs/venv2/local/lib/python2.7/site-packages/wheel/bdist_wheel.py",
>>> line 204, in run
>>> self.run_command('build')
>>> File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
>>> self.distribution.run_command(command)
>>> File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
>>> cmd_obj.run()
>>> File "/usr/lib/python2.7/distutils/command/build.py", line 128, in run
>>> self.run_command(cmd_name)
>>> File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
>>> self.distribution.run_command(command)
>>> File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
>>> cmd_obj.run()
>>> File "setup.py", line 235, in run
>>> gen_protos.generate_proto_files()
>>> File
>>> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
>>> 310, in generate_proto_files
>>> raise ValueError("Proto generation failed (see log for details).")
>>> ValueError: Proto generation failed (see log for details).
>>>
>>>
>>> On Thu, May 7, 2020 at 2:25 PM Udi Meiri <eh...@google.com> wrote:
>>>
>>>> It's hard to say without more details what's going on. Ahmet you're
>>>> right that it installs build-requirements.txt and retries calling
>>>> generate_proto_files().
>>>>
>>>> Thomas, were there additional stacktraces? (after a "During handling of
>>>> the above exception, another exception occurred:" message?)
>>>>
>>>>
>>>> On Thu, May 7, 2020 at 11:59 AM Ahmet Altay <al...@google.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Thu, May 7, 2020 at 11:56 AM Thomas Weise <th...@apache.org> wrote:
>>>>>
>>>>>> Thanks Udi! This is the issue. I'm trying to upgrade from 2.18 where
>>>>>> build-requirements.txt didn't exist.
>>>>>>
>>>>>> Is there a reason why this cannot happen automatically when
>>>>>> running python3.6 setup.py sdist bdist_wheel ?
>>>>>>
>>>>>
>>>>> I _believe_ this should happen automatically here:
>>>>> https://github.com/apache/beam/blob/master/sdks/python/gen_protos.py#L365.
>>>>> Maybe there is a problem there?
>>>>>
>>>>>
>>>>>>
>>>>>> Thomas
>>>>>>
>>>>>>
>>>>>> On Thu, May 7, 2020 at 11:07 AM Udi Meiri <eh...@google.com> wrote:
>>>>>>
>>>>>>> Probably not the issue, but double checking: are you running "pip
>>>>>>> install -r sdks/python/build-requirements.txt" first?
>>>>>>>
>>>>>>> On Wed, May 6, 2020 at 7:22 PM Thomas Weise <th...@apache.org> wrote:
>>>>>>>
>>>>>>>> I'm working on rebasing our fork to 2.21.0 and run into a problem
>>>>>>>> installing grpcio-tools that leads to *ModuleNotFoundError: No
>>>>>>>> module named 'grpc_tools' *(see details below)
>>>>>>>>
>>>>>>>> I cannot reproduce this locally.
>>>>>>>>
>>>>>>>> Any =suggestions on what to look for?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Thomas
>>>>>>>>
>>>>>>>> [?25hBuilding wheels for collected packages: future, gr
>>>>>>>> Running setup.py bdist_wheel for future ... [?25l- \
>>>>>>>> [?25h Stored in directory:
>>>>>>>> /root/.cache/pip/wheels/bf/c9/a3/c538d90ef17cf7823fa51fc701a7a7a910a80f6a405bf1
>>>>>>>> Running setup.py bdist_wheel for grpcio ... [?25l- \ | / - \ | /
>>>>>>>> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
>>>>>>>> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
>>>>>>>> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
>>>>>>>> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
>>>>>>>> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
>>>>>>>> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
>>>>>>>> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
>>>>>>>> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
>>>>>>>> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
>>>>>>>> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
>>>>>>>> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
>>>>>>>> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
>>>>>>>> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
>>>>>>>> | / - \ | / - \ | / - \ |
>>>>>>>> [?25h Stored in directory:
>>>>>>>> /root/.cache/pip/wheels/00/4d/5f/07d0d4283911d2b917b867a11b1622d9d2cc8c286eefd1
>>>>>>>> Successfully built future grpcio
>>>>>>>> Installing collected packages: six, grpcio, setuptools, protobuf,
>>>>>>>> grpcio-tools, future, mypy-protobuf
>>>>>>>> Successfully installed future-0.16.0 grpcio-1.28.1
>>>>>>>> grpcio-tools-1.14.2 mypy-protobuf-1.18 protobuf-3.11.3 setuptools-46.1.3
>>>>>>>> six-1.14.0
>>>>>>>> WARNING:root:Installing grpcio-tools took 305.39 seconds.
>>>>>>>> INFO:gen_protos:Regenerating Python proto definitions (no output
>>>>>>>> files).
>>>>>>>> Process Process-1:
>>>>>>>> Traceback (most recent call last):
>>>>>>>> File
>>>>>>>> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
>>>>>>>> 292, in generate_proto_files
>>>>>>>> from grpc_tools import protoc
>>>>>>>> ModuleNotFoundError: No module named 'grpc_tools'
>>>>>>>>
>>>>>>>> On Fri, Apr 10, 2020 at 10:01 AM Kyle Weaver <kc...@google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi everyone,
>>>>>>>>>
>>>>>>>>> Just a heads up that the Beam 2.21 release branch [1] is cut.
>>>>>>>>> - If you find any important issues that you think should be
>>>>>>>>> addressed in the release, please tag the jira with fix version 2.21.0 and
>>>>>>>>> cc me (username `ibzib`).
>>>>>>>>> - Make sure to update the change log [2] with any significant
>>>>>>>>> changes if you haven't already. Send a PR with the change and tag me. (I
>>>>>>>>> imagine I'm not the only one who forgot to do this :).)
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Kyle
>>>>>>>>>
>>>>>>>>> [1] https://github.com/apache/beam/blob/release-2.21.0
>>>>>>>>> [2] https://github.com/apache/beam/blob/master/CHANGES.md
>>>>>>>>>
>>>>>>>>
Re: Beam 2.21 release update
Posted by Thomas Weise <th...@apache.org>.
I could not find a way for the hidden pip install to succeed within the
virtual environment.
Also, heads up that we found another issue related to timer encoding that
looks like a release blocker. Details to follow.
On Fri, May 8, 2020 at 9:53 AM Udi Meiri <eh...@google.com> wrote:
> +Chad Dombrova <ch...@gmail.com> , who added _find_protoc_gen_mypy.
>
> I'm guessing that the code
> in _install_grpcio_tools_and_generate_proto_files creates a kind of
> virtualenv, but it only works well for staging Python modules and not
> binaries like protoc-gen-mypy.
> (I assume there's a reason why it doesn't invoke virtualenv, probably
> since the list of things setup.py can expect to be installed is very
> minimal (setuptools).)
>
> One solution would be to make these setup.py dependencies explicit in
> pyproject.toml, such that pip installs them before running setup.py:
> https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support
> It would help when using tools like pip ("pip wheel"), but I'm not sure
> what the alternative for "python setup.py sdist" is.
>
>
> On Thu, May 7, 2020 at 10:40 PM Thomas Weise <th...@apache.org> wrote:
>
>> No additional stacktraces. Full error output below.
>>
>> It's not clear what is going wrong.
>>
>> There isn't any exception from the subprocess execution since the
>> "WARNING:root:Installing grpcio-tools took 305.39 seconds." is printed.
>>
>> Also, the time it takes to perform the install is equivalent to
>> successfully running the pip command.
>>
>> I will report back if I find anything else. Currently doing the
>> explicit install via pip install -r sdks/python/build-requirements.txt
>>
>> Thanks,
>> Thomas
>>
>> WARNING:root:Installing grpcio-tools took 269.27 seconds.
>> INFO:gen_protos:Regenerating Python proto definitions (no output files).
>> Process Process-1:
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in
>> _bootstrap
>> self.run()
>> File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
>> self._target(*self._args, **self._kwargs)
>> File
>> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
>> 378, in _install_grpcio_tools_and_generate_proto_files
>> generate_proto_files(force=force)
>> File
>> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
>> 315, in generate_proto_files
>> protoc_gen_mypy = _find_protoc_gen_mypy()
>> File
>> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
>> 233, in _find_protoc_gen_mypy
>> (fname, ', '.join(search_paths)))
>> RuntimeError: Could not find protoc-gen-mypy in /code/venvs/venv2/bin,
>> /code/venvs/venv2/bin, /code/venvs/venv3/bin, /usr/local/sbin,
>> /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin
>> Traceback (most recent call last):
>> File "setup.py", line 311, in <module>
>> 'mypy': generate_protos_first(mypy),
>> File
>> "/code/venvs/venv2/local/lib/python2.7/site-packages/setuptools/__init__.py",
>> line 129, in setup
>> return distutils.core.setup(**attrs)
>> File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
>> dist.run_commands()
>> File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
>> self.run_command(cmd)
>> File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
>> cmd_obj.run()
>> File
>> "/code/venvs/venv2/local/lib/python2.7/site-packages/wheel/bdist_wheel.py",
>> line 204, in run
>> self.run_command('build')
>> File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
>> self.distribution.run_command(command)
>> File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
>> cmd_obj.run()
>> File "/usr/lib/python2.7/distutils/command/build.py", line 128, in run
>> self.run_command(cmd_name)
>> File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
>> self.distribution.run_command(command)
>> File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
>> cmd_obj.run()
>> File "setup.py", line 235, in run
>> gen_protos.generate_proto_files()
>> File
>> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
>> 310, in generate_proto_files
>> raise ValueError("Proto generation failed (see log for details).")
>> ValueError: Proto generation failed (see log for details).
>>
>>
>> On Thu, May 7, 2020 at 2:25 PM Udi Meiri <eh...@google.com> wrote:
>>
>>> It's hard to say without more details what's going on. Ahmet you're
>>> right that it installs build-requirements.txt and retries calling
>>> generate_proto_files().
>>>
>>> Thomas, were there additional stacktraces? (after a "During handling of
>>> the above exception, another exception occurred:" message?)
>>>
>>>
>>> On Thu, May 7, 2020 at 11:59 AM Ahmet Altay <al...@google.com> wrote:
>>>
>>>>
>>>>
>>>> On Thu, May 7, 2020 at 11:56 AM Thomas Weise <th...@apache.org> wrote:
>>>>
>>>>> Thanks Udi! This is the issue. I'm trying to upgrade from 2.18 where
>>>>> build-requirements.txt didn't exist.
>>>>>
>>>>> Is there a reason why this cannot happen automatically when
>>>>> running python3.6 setup.py sdist bdist_wheel ?
>>>>>
>>>>
>>>> I _believe_ this should happen automatically here:
>>>> https://github.com/apache/beam/blob/master/sdks/python/gen_protos.py#L365.
>>>> Maybe there is a problem there?
>>>>
>>>>
>>>>>
>>>>> Thomas
>>>>>
>>>>>
>>>>> On Thu, May 7, 2020 at 11:07 AM Udi Meiri <eh...@google.com> wrote:
>>>>>
>>>>>> Probably not the issue, but double checking: are you running "pip
>>>>>> install -r sdks/python/build-requirements.txt" first?
>>>>>>
>>>>>> On Wed, May 6, 2020 at 7:22 PM Thomas Weise <th...@apache.org> wrote:
>>>>>>
>>>>>>> I'm working on rebasing our fork to 2.21.0 and run into a problem
>>>>>>> installing grpcio-tools that leads to *ModuleNotFoundError: No
>>>>>>> module named 'grpc_tools' *(see details below)
>>>>>>>
>>>>>>> I cannot reproduce this locally.
>>>>>>>
>>>>>>> Any =suggestions on what to look for?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Thomas
>>>>>>>
>>>>>>> [?25hBuilding wheels for collected packages: future, gr
>>>>>>> Running setup.py bdist_wheel for future ... [?25l- \
>>>>>>> [?25h Stored in directory:
>>>>>>> /root/.cache/pip/wheels/bf/c9/a3/c538d90ef17cf7823fa51fc701a7a7a910a80f6a405bf1
>>>>>>> Running setup.py bdist_wheel for grpcio ... [?25l- \ | / - \ | / -
>>>>>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
>>>>>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
>>>>>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
>>>>>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
>>>>>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
>>>>>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
>>>>>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
>>>>>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
>>>>>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
>>>>>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
>>>>>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
>>>>>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
>>>>>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
>>>>>>> / - \ | / - \ | / - \ |
>>>>>>> [?25h Stored in directory:
>>>>>>> /root/.cache/pip/wheels/00/4d/5f/07d0d4283911d2b917b867a11b1622d9d2cc8c286eefd1
>>>>>>> Successfully built future grpcio
>>>>>>> Installing collected packages: six, grpcio, setuptools, protobuf,
>>>>>>> grpcio-tools, future, mypy-protobuf
>>>>>>> Successfully installed future-0.16.0 grpcio-1.28.1
>>>>>>> grpcio-tools-1.14.2 mypy-protobuf-1.18 protobuf-3.11.3 setuptools-46.1.3
>>>>>>> six-1.14.0
>>>>>>> WARNING:root:Installing grpcio-tools took 305.39 seconds.
>>>>>>> INFO:gen_protos:Regenerating Python proto definitions (no output
>>>>>>> files).
>>>>>>> Process Process-1:
>>>>>>> Traceback (most recent call last):
>>>>>>> File
>>>>>>> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
>>>>>>> 292, in generate_proto_files
>>>>>>> from grpc_tools import protoc
>>>>>>> ModuleNotFoundError: No module named 'grpc_tools'
>>>>>>>
>>>>>>> On Fri, Apr 10, 2020 at 10:01 AM Kyle Weaver <kc...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi everyone,
>>>>>>>>
>>>>>>>> Just a heads up that the Beam 2.21 release branch [1] is cut.
>>>>>>>> - If you find any important issues that you think should be
>>>>>>>> addressed in the release, please tag the jira with fix version 2.21.0 and
>>>>>>>> cc me (username `ibzib`).
>>>>>>>> - Make sure to update the change log [2] with any significant
>>>>>>>> changes if you haven't already. Send a PR with the change and tag me. (I
>>>>>>>> imagine I'm not the only one who forgot to do this :).)
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Kyle
>>>>>>>>
>>>>>>>> [1] https://github.com/apache/beam/blob/release-2.21.0
>>>>>>>> [2] https://github.com/apache/beam/blob/master/CHANGES.md
>>>>>>>>
>>>>>>>
Re: Beam 2.21 release update
Posted by Luke Cwik <lc...@google.com>.
Thanks for the fix Max.
On Mon, May 11, 2020 at 5:46 AM Maximilian Michels <mx...@apache.org> wrote:
> FYI I've created this issue and marked it as a blocker:
> https://jira.apache.org/jira/browse/BEAM-9947
>
> Essentially, the timer encoding is broken for all non-standard key
> coders. The fix can be found here:
> https://github.com/apache/beam/pull/11658
>
> -Max
>
> On 08.05.20 18:53, Udi Meiri wrote:
> > +Chad Dombrova <ma...@gmail.com> , who added
> _find_protoc_gen_mypy.
> >
> > I'm guessing that the code
> > in _install_grpcio_tools_and_generate_proto_files creates a kind of
> > virtualenv, but it only works well for staging Python modules and not
> > binaries like protoc-gen-mypy.
> > (I assume there's a reason why it doesn't invoke virtualenv, probably
> > since the list of things setup.py can expect to be installed is very
> > minimal (setuptools).)
> >
> > One solution would be to make these setup.py dependencies explicit in
> > pyproject.toml, such that pip installs them before running
> > setup.py:
> https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support
> > It would help when using tools like pip ("pip wheel"), but I'm not sure
> > what the alternative for "python setup.py sdist" is.
> >
> >
> > On Thu, May 7, 2020 at 10:40 PM Thomas Weise <thw@apache.org
> > <ma...@apache.org>> wrote:
> >
> > No additional stacktraces. Full error output below.
> >
> > It's not clear what is going wrong.
> >
> > There isn't any exception from the subprocess execution since the
> > "WARNING:root:Installing grpcio-tools took 305.39 seconds." is
> printed.
> >
> > Also, the time it takes to perform the install is equivalent to
> > successfully running the pip command.
> >
> > I will report back if I find anything else. Currently doing the
> > explicit install via pip install -r
> sdks/python/build-requirements.txt
> >
> > Thanks,
> > Thomas
> >
> > WARNING:root:Installing grpcio-tools took 269.27 seconds.
> > INFO:gen_protos:Regenerating Python proto definitions (no output
> files).
> > Process Process-1:
> > Traceback (most recent call last):
> > File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in
> > _bootstrap
> > self.run()
> > File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in
> run
> > self._target(*self._args, **self._kwargs)
> > File
> >
> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
> > 378, in _install_grpcio_tools_and_generate_proto_files
> > generate_proto_files(force=force)
> > File
> >
> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
> > 315, in generate_proto_files
> > protoc_gen_mypy = _find_protoc_gen_mypy()
> > File
> >
> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
> > 233, in _find_protoc_gen_mypy
> > (fname, ', '.join(search_paths)))
> > RuntimeError: Could not find protoc-gen-mypy in
> > /code/venvs/venv2/bin, /code/venvs/venv2/bin, /code/venvs/venv3/bin,
> > /usr/local/sbin, /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin
> > Traceback (most recent call last):
> > File "setup.py", line 311, in <module>
> > 'mypy': generate_protos_first(mypy),
> > File
> >
> "/code/venvs/venv2/local/lib/python2.7/site-packages/setuptools/__init__.py",
> > line 129, in setup
> > return distutils.core.setup(**attrs)
> > File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
> > dist.run_commands()
> > File "/usr/lib/python2.7/distutils/dist.py", line 953, in
> run_commands
> > self.run_command(cmd)
> > File "/usr/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> > cmd_obj.run()
> > File
> >
> "/code/venvs/venv2/local/lib/python2.7/site-packages/wheel/bdist_wheel.py",
> > line 204, in run
> > self.run_command('build')
> > File "/usr/lib/python2.7/distutils/cmd.py", line 326, in
> run_command
> > self.distribution.run_command(command)
> > File "/usr/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> > cmd_obj.run()
> > File "/usr/lib/python2.7/distutils/command/build.py", line 128, in
> run
> > self.run_command(cmd_name)
> > File "/usr/lib/python2.7/distutils/cmd.py", line 326, in
> run_command
> > self.distribution.run_command(command)
> > File "/usr/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> > cmd_obj.run()
> > File "setup.py", line 235, in run
> > gen_protos.generate_proto_files()
> > File
> >
> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
> > 310, in generate_proto_files
> > raise ValueError("Proto generation failed (see log for
> details).")
> > ValueError: Proto generation failed (see log for details).
> >
> >
> > On Thu, May 7, 2020 at 2:25 PM Udi Meiri <ehudm@google.com
> > <ma...@google.com>> wrote:
> >
> > It's hard to say without more details what's going on. Ahmet
> > you're right that it installs build-requirements.txt and retries
> > calling generate_proto_files().
> >
> > Thomas, were there additional stacktraces? (after a "During
> > handling of the above exception, another exception occurred:"
> > message?)
> >
> >
> > On Thu, May 7, 2020 at 11:59 AM Ahmet Altay <altay@google.com
> > <ma...@google.com>> wrote:
> >
> >
> >
> > On Thu, May 7, 2020 at 11:56 AM Thomas Weise <thw@apache.org
> > <ma...@apache.org>> wrote:
> >
> > Thanks Udi! This is the issue. I'm trying to upgrade
> > from 2.18 where build-requirements.txt didn't exist.
> >
> > Is there a reason why this cannot happen automatically
> > when running python3.6 setup.py sdist bdist_wheel ?
> >
> >
> > I _believe_ this should happen automatically
> > here:
> https://github.com/apache/beam/blob/master/sdks/python/gen_protos.py#L365.
> > Maybe there is a problem there?
> >
> >
> >
> > Thomas
> >
> >
> > On Thu, May 7, 2020 at 11:07 AM Udi Meiri
> > <ehudm@google.com <ma...@google.com>> wrote:
> >
> > Probably not the issue, but double checking: are you
> > running "pip install -r
> > sdks/python/build-requirements.txt" first?
> >
> > On Wed, May 6, 2020 at 7:22 PM Thomas Weise
> > <thw@apache.org <ma...@apache.org>> wrote:
> >
> > I'm working on rebasing our fork to 2.21.0 and
> > run into a problem installing grpcio-tools that
> > leads to *ModuleNotFoundError: No module named
> > 'grpc_tools' *(see details below)
> >
> > I cannot reproduce this locally.
> >
> > Any =suggestions on what to look for?
> >
> > Thanks,
> > Thomas
> >
> > [?25hBuilding wheels for collected packages:
> > future, gr
> > Running setup.py bdist_wheel for future ...
> > [?25l- \
> > [?25h Stored in directory:
> >
> /root/.cache/pip/wheels/bf/c9/a3/c538d90ef17cf7823fa51fc701a7a7a910a80f6a405bf1
> > Running setup.py bdist_wheel for grpcio ...
> > [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \
> > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> > | / - \ | / - \ | / - \ |
> > [?25h Stored in directory:
> >
> /root/.cache/pip/wheels/00/4d/5f/07d0d4283911d2b917b867a11b1622d9d2cc8c286eefd1
> > Successfully built future grpcio
> > Installing collected packages: six, grpcio,
> > setuptools, protobuf, grpcio-tools, future,
> > mypy-protobuf
> > Successfully installed future-0.16.0
> > grpcio-1.28.1 grpcio-tools-1.14.2
> > mypy-protobuf-1.18 protobuf-3.11.3
> > setuptools-46.1.3 six-1.14.0
> > WARNING:root:Installing grpcio-tools took 305.39
> > seconds.
> > INFO:gen_protos:Regenerating Python proto
> > definitions (no output files).
> > Process Process-1:
> > Traceback (most recent call last):
> > File
> >
> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py",
> > line 292, in generate_proto_files
> > from grpc_tools import protoc
> > ModuleNotFoundError: No module named 'grpc_tools'
> >
> > On Fri, Apr 10, 2020 at 10:01 AM Kyle Weaver
> > <kcweaver@google.com
> > <ma...@google.com>> wrote:
> >
> > Hi everyone,
> >
> > Just a heads up that the Beam 2.21 release
> > branch [1] is cut.
> > - If you find any important issues that you
> > think should be addressed in the release,
> > please tag the jira with fix version 2.21.0
> > and cc me (username `ibzib`).
> > - Make sure to update the change log [2]
> > with any significant changes if you haven't
> > already. Send a PR with the change and tag
> > me. (I imagine I'm not the only one who
> > forgot to do this :).)
> >
> > Thanks,
> > Kyle
> >
> > [1]
> https://github.com/apache/beam/blob/release-2.21.0
> > [2]
> https://github.com/apache/beam/blob/master/CHANGES.md
> >
>
Re: Beam 2.21 release update
Posted by Maximilian Michels <mx...@apache.org>.
FYI I've created this issue and marked it as a blocker:
https://jira.apache.org/jira/browse/BEAM-9947
Essentially, the timer encoding is broken for all non-standard key
coders. The fix can be found here: https://github.com/apache/beam/pull/11658
-Max
On 08.05.20 18:53, Udi Meiri wrote:
> +Chad Dombrova <ma...@gmail.com> , who added _find_protoc_gen_mypy.
>
> I'm guessing that the code
> in _install_grpcio_tools_and_generate_proto_files creates a kind of
> virtualenv, but it only works well for staging Python modules and not
> binaries like protoc-gen-mypy.
> (I assume there's a reason why it doesn't invoke virtualenv, probably
> since the list of things setup.py can expect to be installed is very
> minimal (setuptools).)
>
> One solution would be to make these setup.py dependencies explicit in
> pyproject.toml, such that pip installs them before running
> setup.py: https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support
> It would help when using tools like pip ("pip wheel"), but I'm not sure
> what the alternative for "python setup.py sdist" is.
>
>
> On Thu, May 7, 2020 at 10:40 PM Thomas Weise <thw@apache.org
> <ma...@apache.org>> wrote:
>
> No additional stacktraces. Full error output below.
>
> It's not clear what is going wrong.
>
> There isn't any exception from the subprocess execution since the
> "WARNING:root:Installing grpcio-tools took 305.39 seconds." is printed.
>
> Also, the time it takes to perform the install is equivalent to
> successfully running the pip command.
>
> I will report back if I find anything else. Currently doing the
> explicit install via pip install -r sdks/python/build-requirements.txt
>
> Thanks,
> Thomas
>
> WARNING:root:Installing grpcio-tools took 269.27 seconds.
> INFO:gen_protos:Regenerating Python proto definitions (no output files).
> Process Process-1:
> Traceback (most recent call last):
> File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in
> _bootstrap
> self.run()
> File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
> self._target(*self._args, **self._kwargs)
> File
> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
> 378, in _install_grpcio_tools_and_generate_proto_files
> generate_proto_files(force=force)
> File
> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
> 315, in generate_proto_files
> protoc_gen_mypy = _find_protoc_gen_mypy()
> File
> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
> 233, in _find_protoc_gen_mypy
> (fname, ', '.join(search_paths)))
> RuntimeError: Could not find protoc-gen-mypy in
> /code/venvs/venv2/bin, /code/venvs/venv2/bin, /code/venvs/venv3/bin,
> /usr/local/sbin, /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin
> Traceback (most recent call last):
> File "setup.py", line 311, in <module>
> 'mypy': generate_protos_first(mypy),
> File
> "/code/venvs/venv2/local/lib/python2.7/site-packages/setuptools/__init__.py",
> line 129, in setup
> return distutils.core.setup(**attrs)
> File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
> dist.run_commands()
> File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
> self.run_command(cmd)
> File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
> cmd_obj.run()
> File
> "/code/venvs/venv2/local/lib/python2.7/site-packages/wheel/bdist_wheel.py",
> line 204, in run
> self.run_command('build')
> File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
> self.distribution.run_command(command)
> File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
> cmd_obj.run()
> File "/usr/lib/python2.7/distutils/command/build.py", line 128, in run
> self.run_command(cmd_name)
> File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
> self.distribution.run_command(command)
> File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
> cmd_obj.run()
> File "setup.py", line 235, in run
> gen_protos.generate_proto_files()
> File
> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
> 310, in generate_proto_files
> raise ValueError("Proto generation failed (see log for details).")
> ValueError: Proto generation failed (see log for details).
>
>
> On Thu, May 7, 2020 at 2:25 PM Udi Meiri <ehudm@google.com
> <ma...@google.com>> wrote:
>
> It's hard to say without more details what's going on. Ahmet
> you're right that it installs build-requirements.txt and retries
> calling generate_proto_files().
>
> Thomas, were there additional stacktraces? (after a "During
> handling of the above exception, another exception occurred:"
> message?)
>
>
> On Thu, May 7, 2020 at 11:59 AM Ahmet Altay <altay@google.com
> <ma...@google.com>> wrote:
>
>
>
> On Thu, May 7, 2020 at 11:56 AM Thomas Weise <thw@apache.org
> <ma...@apache.org>> wrote:
>
> Thanks Udi! This is the issue. I'm trying to upgrade
> from 2.18 where build-requirements.txt didn't exist.
>
> Is there a reason why this cannot happen automatically
> when running python3.6 setup.py sdist bdist_wheel ?
>
>
> I _believe_ this should happen automatically
> here: https://github.com/apache/beam/blob/master/sdks/python/gen_protos.py#L365.
> Maybe there is a problem there?
>
>
>
> Thomas
>
>
> On Thu, May 7, 2020 at 11:07 AM Udi Meiri
> <ehudm@google.com <ma...@google.com>> wrote:
>
> Probably not the issue, but double checking: are you
> running "pip install -r
> sdks/python/build-requirements.txt" first?
>
> On Wed, May 6, 2020 at 7:22 PM Thomas Weise
> <thw@apache.org <ma...@apache.org>> wrote:
>
> I'm working on rebasing our fork to 2.21.0 and
> run into a problem installing grpcio-tools that
> leads to *ModuleNotFoundError: No module named
> 'grpc_tools' *(see details below)
>
> I cannot reproduce this locally.
>
> Any =suggestions on what to look for?
>
> Thanks,
> Thomas
>
> [?25hBuilding wheels for collected packages:
> future, gr
> Running setup.py bdist_wheel for future ...
> [?25l- \
> [?25h Stored in directory:
> /root/.cache/pip/wheels/bf/c9/a3/c538d90ef17cf7823fa51fc701a7a7a910a80f6a405bf1
> Running setup.py bdist_wheel for grpcio ...
> [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \
> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> | / - \ | / - \ | / - \ |
> [?25h Stored in directory:
> /root/.cache/pip/wheels/00/4d/5f/07d0d4283911d2b917b867a11b1622d9d2cc8c286eefd1
> Successfully built future grpcio
> Installing collected packages: six, grpcio,
> setuptools, protobuf, grpcio-tools, future,
> mypy-protobuf
> Successfully installed future-0.16.0
> grpcio-1.28.1 grpcio-tools-1.14.2
> mypy-protobuf-1.18 protobuf-3.11.3
> setuptools-46.1.3 six-1.14.0
> WARNING:root:Installing grpcio-tools took 305.39
> seconds.
> INFO:gen_protos:Regenerating Python proto
> definitions (no output files).
> Process Process-1:
> Traceback (most recent call last):
> File
> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py",
> line 292, in generate_proto_files
> from grpc_tools import protoc
> ModuleNotFoundError: No module named 'grpc_tools'
>
> On Fri, Apr 10, 2020 at 10:01 AM Kyle Weaver
> <kcweaver@google.com
> <ma...@google.com>> wrote:
>
> Hi everyone,
>
> Just a heads up that the Beam 2.21 release
> branch [1] is cut.
> - If you find any important issues that you
> think should be addressed in the release,
> please tag the jira with fix version 2.21.0
> and cc me (username `ibzib`).
> - Make sure to update the change log [2]
> with any significant changes if you haven't
> already. Send a PR with the change and tag
> me. (I imagine I'm not the only one who
> forgot to do this :).)
>
> Thanks,
> Kyle
>
> [1] https://github.com/apache/beam/blob/release-2.21.0
> [2] https://github.com/apache/beam/blob/master/CHANGES.md
>
Re: Beam 2.21 release update
Posted by Udi Meiri <eh...@google.com>.
+Chad Dombrova <ch...@gmail.com> , who added _find_protoc_gen_mypy.
I'm guessing that the code
in _install_grpcio_tools_and_generate_proto_files creates a kind of
virtualenv, but it only works well for staging Python modules and not
binaries like protoc-gen-mypy.
(I assume there's a reason why it doesn't invoke virtualenv, probably since
the list of things setup.py can expect to be installed is very minimal
(setuptools).)
One solution would be to make these setup.py dependencies explicit in
pyproject.toml, such that pip installs them before running setup.py:
https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support
It would help when using tools like pip ("pip wheel"), but I'm not sure
what the alternative for "python setup.py sdist" is.
On Thu, May 7, 2020 at 10:40 PM Thomas Weise <th...@apache.org> wrote:
> No additional stacktraces. Full error output below.
>
> It's not clear what is going wrong.
>
> There isn't any exception from the subprocess execution since the
> "WARNING:root:Installing grpcio-tools took 305.39 seconds." is printed.
>
> Also, the time it takes to perform the install is equivalent to
> successfully running the pip command.
>
> I will report back if I find anything else. Currently doing the
> explicit install via pip install -r sdks/python/build-requirements.txt
>
> Thanks,
> Thomas
>
> WARNING:root:Installing grpcio-tools took 269.27 seconds.
> INFO:gen_protos:Regenerating Python proto definitions (no output files).
> Process Process-1:
> Traceback (most recent call last):
> File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in
> _bootstrap
> self.run()
> File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
> self._target(*self._args, **self._kwargs)
> File
> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
> 378, in _install_grpcio_tools_and_generate_proto_files
> generate_proto_files(force=force)
> File
> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
> 315, in generate_proto_files
> protoc_gen_mypy = _find_protoc_gen_mypy()
> File
> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
> 233, in _find_protoc_gen_mypy
> (fname, ', '.join(search_paths)))
> RuntimeError: Could not find protoc-gen-mypy in /code/venvs/venv2/bin,
> /code/venvs/venv2/bin, /code/venvs/venv3/bin, /usr/local/sbin,
> /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin
> Traceback (most recent call last):
> File "setup.py", line 311, in <module>
> 'mypy': generate_protos_first(mypy),
> File
> "/code/venvs/venv2/local/lib/python2.7/site-packages/setuptools/__init__.py",
> line 129, in setup
> return distutils.core.setup(**attrs)
> File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
> dist.run_commands()
> File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
> self.run_command(cmd)
> File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
> cmd_obj.run()
> File
> "/code/venvs/venv2/local/lib/python2.7/site-packages/wheel/bdist_wheel.py",
> line 204, in run
> self.run_command('build')
> File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
> self.distribution.run_command(command)
> File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
> cmd_obj.run()
> File "/usr/lib/python2.7/distutils/command/build.py", line 128, in run
> self.run_command(cmd_name)
> File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
> self.distribution.run_command(command)
> File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
> cmd_obj.run()
> File "setup.py", line 235, in run
> gen_protos.generate_proto_files()
> File
> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
> 310, in generate_proto_files
> raise ValueError("Proto generation failed (see log for details).")
> ValueError: Proto generation failed (see log for details).
>
>
> On Thu, May 7, 2020 at 2:25 PM Udi Meiri <eh...@google.com> wrote:
>
>> It's hard to say without more details what's going on. Ahmet you're right
>> that it installs build-requirements.txt and retries calling
>> generate_proto_files().
>>
>> Thomas, were there additional stacktraces? (after a "During handling of
>> the above exception, another exception occurred:" message?)
>>
>>
>> On Thu, May 7, 2020 at 11:59 AM Ahmet Altay <al...@google.com> wrote:
>>
>>>
>>>
>>> On Thu, May 7, 2020 at 11:56 AM Thomas Weise <th...@apache.org> wrote:
>>>
>>>> Thanks Udi! This is the issue. I'm trying to upgrade from 2.18 where
>>>> build-requirements.txt didn't exist.
>>>>
>>>> Is there a reason why this cannot happen automatically when
>>>> running python3.6 setup.py sdist bdist_wheel ?
>>>>
>>>
>>> I _believe_ this should happen automatically here:
>>> https://github.com/apache/beam/blob/master/sdks/python/gen_protos.py#L365.
>>> Maybe there is a problem there?
>>>
>>>
>>>>
>>>> Thomas
>>>>
>>>>
>>>> On Thu, May 7, 2020 at 11:07 AM Udi Meiri <eh...@google.com> wrote:
>>>>
>>>>> Probably not the issue, but double checking: are you running "pip
>>>>> install -r sdks/python/build-requirements.txt" first?
>>>>>
>>>>> On Wed, May 6, 2020 at 7:22 PM Thomas Weise <th...@apache.org> wrote:
>>>>>
>>>>>> I'm working on rebasing our fork to 2.21.0 and run into a problem
>>>>>> installing grpcio-tools that leads to *ModuleNotFoundError: No
>>>>>> module named 'grpc_tools' *(see details below)
>>>>>>
>>>>>> I cannot reproduce this locally.
>>>>>>
>>>>>> Any =suggestions on what to look for?
>>>>>>
>>>>>> Thanks,
>>>>>> Thomas
>>>>>>
>>>>>> [?25hBuilding wheels for collected packages: future, gr
>>>>>> Running setup.py bdist_wheel for future ... [?25l- \
>>>>>> [?25h Stored in directory:
>>>>>> /root/.cache/pip/wheels/bf/c9/a3/c538d90ef17cf7823fa51fc701a7a7a910a80f6a405bf1
>>>>>> Running setup.py bdist_wheel for grpcio ... [?25l- \ | / - \ | / -
>>>>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
>>>>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
>>>>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
>>>>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
>>>>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
>>>>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
>>>>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
>>>>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
>>>>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
>>>>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
>>>>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
>>>>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
>>>>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
>>>>>> / - \ | / - \ | / - \ |
>>>>>> [?25h Stored in directory:
>>>>>> /root/.cache/pip/wheels/00/4d/5f/07d0d4283911d2b917b867a11b1622d9d2cc8c286eefd1
>>>>>> Successfully built future grpcio
>>>>>> Installing collected packages: six, grpcio, setuptools, protobuf,
>>>>>> grpcio-tools, future, mypy-protobuf
>>>>>> Successfully installed future-0.16.0 grpcio-1.28.1
>>>>>> grpcio-tools-1.14.2 mypy-protobuf-1.18 protobuf-3.11.3 setuptools-46.1.3
>>>>>> six-1.14.0
>>>>>> WARNING:root:Installing grpcio-tools took 305.39 seconds.
>>>>>> INFO:gen_protos:Regenerating Python proto definitions (no output
>>>>>> files).
>>>>>> Process Process-1:
>>>>>> Traceback (most recent call last):
>>>>>> File
>>>>>> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
>>>>>> 292, in generate_proto_files
>>>>>> from grpc_tools import protoc
>>>>>> ModuleNotFoundError: No module named 'grpc_tools'
>>>>>>
>>>>>> On Fri, Apr 10, 2020 at 10:01 AM Kyle Weaver <kc...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> Just a heads up that the Beam 2.21 release branch [1] is cut.
>>>>>>> - If you find any important issues that you think should be
>>>>>>> addressed in the release, please tag the jira with fix version 2.21.0 and
>>>>>>> cc me (username `ibzib`).
>>>>>>> - Make sure to update the change log [2] with any significant
>>>>>>> changes if you haven't already. Send a PR with the change and tag me. (I
>>>>>>> imagine I'm not the only one who forgot to do this :).)
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Kyle
>>>>>>>
>>>>>>> [1] https://github.com/apache/beam/blob/release-2.21.0
>>>>>>> [2] https://github.com/apache/beam/blob/master/CHANGES.md
>>>>>>>
>>>>>>
Re: Beam 2.21 release update
Posted by Thomas Weise <th...@apache.org>.
No additional stacktraces. Full error output below.
It's not clear what is going wrong.
There isn't any exception from the subprocess execution since the
"WARNING:root:Installing grpcio-tools took 305.39 seconds." is printed.
Also, the time it takes to perform the install is equivalent to
successfully running the pip command.
I will report back if I find anything else. Currently doing the
explicit install via pip install -r sdks/python/build-requirements.txt
Thanks,
Thomas
WARNING:root:Installing grpcio-tools took 269.27 seconds.
INFO:gen_protos:Regenerating Python proto definitions (no output files).
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in
_bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File
"/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
378, in _install_grpcio_tools_and_generate_proto_files
generate_proto_files(force=force)
File
"/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
315, in generate_proto_files
protoc_gen_mypy = _find_protoc_gen_mypy()
File
"/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
233, in _find_protoc_gen_mypy
(fname, ', '.join(search_paths)))
RuntimeError: Could not find protoc-gen-mypy in /code/venvs/venv2/bin,
/code/venvs/venv2/bin, /code/venvs/venv3/bin, /usr/local/sbin,
/usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin
Traceback (most recent call last):
File "setup.py", line 311, in <module>
'mypy': generate_protos_first(mypy),
File
"/code/venvs/venv2/local/lib/python2.7/site-packages/setuptools/__init__.py",
line 129, in setup
return distutils.core.setup(**attrs)
File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File
"/code/venvs/venv2/local/lib/python2.7/site-packages/wheel/bdist_wheel.py",
line 204, in run
self.run_command('build')
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/lib/python2.7/distutils/command/build.py", line 128, in run
self.run_command(cmd_name)
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "setup.py", line 235, in run
gen_protos.generate_proto_files()
File
"/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
310, in generate_proto_files
raise ValueError("Proto generation failed (see log for details).")
ValueError: Proto generation failed (see log for details).
On Thu, May 7, 2020 at 2:25 PM Udi Meiri <eh...@google.com> wrote:
> It's hard to say without more details what's going on. Ahmet you're right
> that it installs build-requirements.txt and retries calling
> generate_proto_files().
>
> Thomas, were there additional stacktraces? (after a "During handling of
> the above exception, another exception occurred:" message?)
>
>
> On Thu, May 7, 2020 at 11:59 AM Ahmet Altay <al...@google.com> wrote:
>
>>
>>
>> On Thu, May 7, 2020 at 11:56 AM Thomas Weise <th...@apache.org> wrote:
>>
>>> Thanks Udi! This is the issue. I'm trying to upgrade from 2.18 where
>>> build-requirements.txt didn't exist.
>>>
>>> Is there a reason why this cannot happen automatically when
>>> running python3.6 setup.py sdist bdist_wheel ?
>>>
>>
>> I _believe_ this should happen automatically here:
>> https://github.com/apache/beam/blob/master/sdks/python/gen_protos.py#L365.
>> Maybe there is a problem there?
>>
>>
>>>
>>> Thomas
>>>
>>>
>>> On Thu, May 7, 2020 at 11:07 AM Udi Meiri <eh...@google.com> wrote:
>>>
>>>> Probably not the issue, but double checking: are you running "pip
>>>> install -r sdks/python/build-requirements.txt" first?
>>>>
>>>> On Wed, May 6, 2020 at 7:22 PM Thomas Weise <th...@apache.org> wrote:
>>>>
>>>>> I'm working on rebasing our fork to 2.21.0 and run into a problem
>>>>> installing grpcio-tools that leads to *ModuleNotFoundError: No module
>>>>> named 'grpc_tools' *(see details below)
>>>>>
>>>>> I cannot reproduce this locally.
>>>>>
>>>>> Any =suggestions on what to look for?
>>>>>
>>>>> Thanks,
>>>>> Thomas
>>>>>
>>>>> [?25hBuilding wheels for collected packages: future, gr
>>>>> Running setup.py bdist_wheel for future ... [?25l- \
>>>>> [?25h Stored in directory:
>>>>> /root/.cache/pip/wheels/bf/c9/a3/c538d90ef17cf7823fa51fc701a7a7a910a80f6a405bf1
>>>>> Running setup.py bdist_wheel for grpcio ... [?25l- \ | / - \ | / - \
>>>>> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
>>>>> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
>>>>> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
>>>>> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
>>>>> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
>>>>> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
>>>>> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
>>>>> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
>>>>> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
>>>>> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
>>>>> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
>>>>> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
>>>>> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
>>>>> - \ | / - \ | / - \ |
>>>>> [?25h Stored in directory:
>>>>> /root/.cache/pip/wheels/00/4d/5f/07d0d4283911d2b917b867a11b1622d9d2cc8c286eefd1
>>>>> Successfully built future grpcio
>>>>> Installing collected packages: six, grpcio, setuptools, protobuf,
>>>>> grpcio-tools, future, mypy-protobuf
>>>>> Successfully installed future-0.16.0 grpcio-1.28.1 grpcio-tools-1.14.2
>>>>> mypy-protobuf-1.18 protobuf-3.11.3 setuptools-46.1.3 six-1.14.0
>>>>> WARNING:root:Installing grpcio-tools took 305.39 seconds.
>>>>> INFO:gen_protos:Regenerating Python proto definitions (no output
>>>>> files).
>>>>> Process Process-1:
>>>>> Traceback (most recent call last):
>>>>> File
>>>>> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
>>>>> 292, in generate_proto_files
>>>>> from grpc_tools import protoc
>>>>> ModuleNotFoundError: No module named 'grpc_tools'
>>>>>
>>>>> On Fri, Apr 10, 2020 at 10:01 AM Kyle Weaver <kc...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> Just a heads up that the Beam 2.21 release branch [1] is cut.
>>>>>> - If you find any important issues that you think should be addressed
>>>>>> in the release, please tag the jira with fix version 2.21.0 and cc me
>>>>>> (username `ibzib`).
>>>>>> - Make sure to update the change log [2] with any significant changes
>>>>>> if you haven't already. Send a PR with the change and tag me. (I imagine
>>>>>> I'm not the only one who forgot to do this :).)
>>>>>>
>>>>>> Thanks,
>>>>>> Kyle
>>>>>>
>>>>>> [1] https://github.com/apache/beam/blob/release-2.21.0
>>>>>> [2] https://github.com/apache/beam/blob/master/CHANGES.md
>>>>>>
>>>>>
Re: Beam 2.21 release update
Posted by Udi Meiri <eh...@google.com>.
It's hard to say without more details what's going on. Ahmet you're right
that it installs build-requirements.txt and retries calling
generate_proto_files().
Thomas, were there additional stacktraces? (after a "During handling of the
above exception, another exception occurred:" message?)
On Thu, May 7, 2020 at 11:59 AM Ahmet Altay <al...@google.com> wrote:
>
>
> On Thu, May 7, 2020 at 11:56 AM Thomas Weise <th...@apache.org> wrote:
>
>> Thanks Udi! This is the issue. I'm trying to upgrade from 2.18 where
>> build-requirements.txt didn't exist.
>>
>> Is there a reason why this cannot happen automatically when
>> running python3.6 setup.py sdist bdist_wheel ?
>>
>
> I _believe_ this should happen automatically here:
> https://github.com/apache/beam/blob/master/sdks/python/gen_protos.py#L365.
> Maybe there is a problem there?
>
>
>>
>> Thomas
>>
>>
>> On Thu, May 7, 2020 at 11:07 AM Udi Meiri <eh...@google.com> wrote:
>>
>>> Probably not the issue, but double checking: are you running "pip
>>> install -r sdks/python/build-requirements.txt" first?
>>>
>>> On Wed, May 6, 2020 at 7:22 PM Thomas Weise <th...@apache.org> wrote:
>>>
>>>> I'm working on rebasing our fork to 2.21.0 and run into a problem
>>>> installing grpcio-tools that leads to *ModuleNotFoundError: No module
>>>> named 'grpc_tools' *(see details below)
>>>>
>>>> I cannot reproduce this locally.
>>>>
>>>> Any =suggestions on what to look for?
>>>>
>>>> Thanks,
>>>> Thomas
>>>>
>>>> [?25hBuilding wheels for collected packages: future, gr
>>>> Running setup.py bdist_wheel for future ... [?25l- \
>>>> [?25h Stored in directory:
>>>> /root/.cache/pip/wheels/bf/c9/a3/c538d90ef17cf7823fa51fc701a7a7a910a80f6a405bf1
>>>> Running setup.py bdist_wheel for grpcio ... [?25l- \ | / - \ | / - \
>>>> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
>>>> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
>>>> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
>>>> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
>>>> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
>>>> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
>>>> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
>>>> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
>>>> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
>>>> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
>>>> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
>>>> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
>>>> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
>>>> - \ | / - \ | / - \ |
>>>> [?25h Stored in directory:
>>>> /root/.cache/pip/wheels/00/4d/5f/07d0d4283911d2b917b867a11b1622d9d2cc8c286eefd1
>>>> Successfully built future grpcio
>>>> Installing collected packages: six, grpcio, setuptools, protobuf,
>>>> grpcio-tools, future, mypy-protobuf
>>>> Successfully installed future-0.16.0 grpcio-1.28.1 grpcio-tools-1.14.2
>>>> mypy-protobuf-1.18 protobuf-3.11.3 setuptools-46.1.3 six-1.14.0
>>>> WARNING:root:Installing grpcio-tools took 305.39 seconds.
>>>> INFO:gen_protos:Regenerating Python proto definitions (no output files).
>>>> Process Process-1:
>>>> Traceback (most recent call last):
>>>> File
>>>> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
>>>> 292, in generate_proto_files
>>>> from grpc_tools import protoc
>>>> ModuleNotFoundError: No module named 'grpc_tools'
>>>>
>>>> On Fri, Apr 10, 2020 at 10:01 AM Kyle Weaver <kc...@google.com>
>>>> wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> Just a heads up that the Beam 2.21 release branch [1] is cut.
>>>>> - If you find any important issues that you think should be addressed
>>>>> in the release, please tag the jira with fix version 2.21.0 and cc me
>>>>> (username `ibzib`).
>>>>> - Make sure to update the change log [2] with any significant changes
>>>>> if you haven't already. Send a PR with the change and tag me. (I imagine
>>>>> I'm not the only one who forgot to do this :).)
>>>>>
>>>>> Thanks,
>>>>> Kyle
>>>>>
>>>>> [1] https://github.com/apache/beam/blob/release-2.21.0
>>>>> [2] https://github.com/apache/beam/blob/master/CHANGES.md
>>>>>
>>>>
Re: Beam 2.21 release update
Posted by Ahmet Altay <al...@google.com>.
On Thu, May 7, 2020 at 11:56 AM Thomas Weise <th...@apache.org> wrote:
> Thanks Udi! This is the issue. I'm trying to upgrade from 2.18 where
> build-requirements.txt didn't exist.
>
> Is there a reason why this cannot happen automatically when
> running python3.6 setup.py sdist bdist_wheel ?
>
I _believe_ this should happen automatically here:
https://github.com/apache/beam/blob/master/sdks/python/gen_protos.py#L365.
Maybe there is a problem there?
>
> Thomas
>
>
> On Thu, May 7, 2020 at 11:07 AM Udi Meiri <eh...@google.com> wrote:
>
>> Probably not the issue, but double checking: are you running "pip install
>> -r sdks/python/build-requirements.txt" first?
>>
>> On Wed, May 6, 2020 at 7:22 PM Thomas Weise <th...@apache.org> wrote:
>>
>>> I'm working on rebasing our fork to 2.21.0 and run into a problem
>>> installing grpcio-tools that leads to *ModuleNotFoundError: No module
>>> named 'grpc_tools' *(see details below)
>>>
>>> I cannot reproduce this locally.
>>>
>>> Any =suggestions on what to look for?
>>>
>>> Thanks,
>>> Thomas
>>>
>>> [?25hBuilding wheels for collected packages: future, gr
>>> Running setup.py bdist_wheel for future ... [?25l- \
>>> [?25h Stored in directory:
>>> /root/.cache/pip/wheels/bf/c9/a3/c538d90ef17cf7823fa51fc701a7a7a910a80f6a405bf1
>>> Running setup.py bdist_wheel for grpcio ... [?25l- \ | / - \ | / - \ |
>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
>>> \ | / - \ | / - \ |
>>> [?25h Stored in directory:
>>> /root/.cache/pip/wheels/00/4d/5f/07d0d4283911d2b917b867a11b1622d9d2cc8c286eefd1
>>> Successfully built future grpcio
>>> Installing collected packages: six, grpcio, setuptools, protobuf,
>>> grpcio-tools, future, mypy-protobuf
>>> Successfully installed future-0.16.0 grpcio-1.28.1 grpcio-tools-1.14.2
>>> mypy-protobuf-1.18 protobuf-3.11.3 setuptools-46.1.3 six-1.14.0
>>> WARNING:root:Installing grpcio-tools took 305.39 seconds.
>>> INFO:gen_protos:Regenerating Python proto definitions (no output files).
>>> Process Process-1:
>>> Traceback (most recent call last):
>>> File
>>> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
>>> 292, in generate_proto_files
>>> from grpc_tools import protoc
>>> ModuleNotFoundError: No module named 'grpc_tools'
>>>
>>> On Fri, Apr 10, 2020 at 10:01 AM Kyle Weaver <kc...@google.com>
>>> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> Just a heads up that the Beam 2.21 release branch [1] is cut.
>>>> - If you find any important issues that you think should be addressed
>>>> in the release, please tag the jira with fix version 2.21.0 and cc me
>>>> (username `ibzib`).
>>>> - Make sure to update the change log [2] with any significant changes
>>>> if you haven't already. Send a PR with the change and tag me. (I imagine
>>>> I'm not the only one who forgot to do this :).)
>>>>
>>>> Thanks,
>>>> Kyle
>>>>
>>>> [1] https://github.com/apache/beam/blob/release-2.21.0
>>>> [2] https://github.com/apache/beam/blob/master/CHANGES.md
>>>>
>>>
Re: Beam 2.21 release update
Posted by Thomas Weise <th...@apache.org>.
Thanks Udi! This is the issue. I'm trying to upgrade from 2.18 where
build-requirements.txt didn't exist.
Is there a reason why this cannot happen automatically when
running python3.6 setup.py sdist bdist_wheel ?
Thomas
On Thu, May 7, 2020 at 11:07 AM Udi Meiri <eh...@google.com> wrote:
> Probably not the issue, but double checking: are you running "pip install
> -r sdks/python/build-requirements.txt" first?
>
> On Wed, May 6, 2020 at 7:22 PM Thomas Weise <th...@apache.org> wrote:
>
>> I'm working on rebasing our fork to 2.21.0 and run into a problem
>> installing grpcio-tools that leads to *ModuleNotFoundError: No module
>> named 'grpc_tools' *(see details below)
>>
>> I cannot reproduce this locally.
>>
>> Any =suggestions on what to look for?
>>
>> Thanks,
>> Thomas
>>
>> [?25hBuilding wheels for collected packages: future, gr
>> Running setup.py bdist_wheel for future ... [?25l- \
>> [?25h Stored in directory:
>> /root/.cache/pip/wheels/bf/c9/a3/c538d90ef17cf7823fa51fc701a7a7a910a80f6a405bf1
>> Running setup.py bdist_wheel for grpcio ... [?25l- \ | / - \ | / - \ |
>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
>> \ | / - \ | / - \ |
>> [?25h Stored in directory:
>> /root/.cache/pip/wheels/00/4d/5f/07d0d4283911d2b917b867a11b1622d9d2cc8c286eefd1
>> Successfully built future grpcio
>> Installing collected packages: six, grpcio, setuptools, protobuf,
>> grpcio-tools, future, mypy-protobuf
>> Successfully installed future-0.16.0 grpcio-1.28.1 grpcio-tools-1.14.2
>> mypy-protobuf-1.18 protobuf-3.11.3 setuptools-46.1.3 six-1.14.0
>> WARNING:root:Installing grpcio-tools took 305.39 seconds.
>> INFO:gen_protos:Regenerating Python proto definitions (no output files).
>> Process Process-1:
>> Traceback (most recent call last):
>> File
>> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
>> 292, in generate_proto_files
>> from grpc_tools import protoc
>> ModuleNotFoundError: No module named 'grpc_tools'
>>
>> On Fri, Apr 10, 2020 at 10:01 AM Kyle Weaver <kc...@google.com> wrote:
>>
>>> Hi everyone,
>>>
>>> Just a heads up that the Beam 2.21 release branch [1] is cut.
>>> - If you find any important issues that you think should be addressed in
>>> the release, please tag the jira with fix version 2.21.0 and cc me
>>> (username `ibzib`).
>>> - Make sure to update the change log [2] with any significant changes if
>>> you haven't already. Send a PR with the change and tag me. (I imagine I'm
>>> not the only one who forgot to do this :).)
>>>
>>> Thanks,
>>> Kyle
>>>
>>> [1] https://github.com/apache/beam/blob/release-2.21.0
>>> [2] https://github.com/apache/beam/blob/master/CHANGES.md
>>>
>>
Re: Beam 2.21 release update
Posted by Udi Meiri <eh...@google.com>.
Probably not the issue, but double checking: are you running "pip install
-r sdks/python/build-requirements.txt" first?
On Wed, May 6, 2020 at 7:22 PM Thomas Weise <th...@apache.org> wrote:
> I'm working on rebasing our fork to 2.21.0 and run into a problem
> installing grpcio-tools that leads to *ModuleNotFoundError: No module
> named 'grpc_tools' *(see details below)
>
> I cannot reproduce this locally.
>
> Any =suggestions on what to look for?
>
> Thanks,
> Thomas
>
> [?25hBuilding wheels for collected packages: future, gr
> Running setup.py bdist_wheel for future ... [?25l- \
> [?25h Stored in directory:
> /root/.cache/pip/wheels/bf/c9/a3/c538d90ef17cf7823fa51fc701a7a7a910a80f6a405bf1
> Running setup.py bdist_wheel for grpcio ... [?25l- \ | / - \ | / - \ | /
> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
> | / - \ | / - \ |
> [?25h Stored in directory:
> /root/.cache/pip/wheels/00/4d/5f/07d0d4283911d2b917b867a11b1622d9d2cc8c286eefd1
> Successfully built future grpcio
> Installing collected packages: six, grpcio, setuptools, protobuf,
> grpcio-tools, future, mypy-protobuf
> Successfully installed future-0.16.0 grpcio-1.28.1 grpcio-tools-1.14.2
> mypy-protobuf-1.18 protobuf-3.11.3 setuptools-46.1.3 six-1.14.0
> WARNING:root:Installing grpcio-tools took 305.39 seconds.
> INFO:gen_protos:Regenerating Python proto definitions (no output files).
> Process Process-1:
> Traceback (most recent call last):
> File
> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
> 292, in generate_proto_files
> from grpc_tools import protoc
> ModuleNotFoundError: No module named 'grpc_tools'
>
> On Fri, Apr 10, 2020 at 10:01 AM Kyle Weaver <kc...@google.com> wrote:
>
>> Hi everyone,
>>
>> Just a heads up that the Beam 2.21 release branch [1] is cut.
>> - If you find any important issues that you think should be addressed in
>> the release, please tag the jira with fix version 2.21.0 and cc me
>> (username `ibzib`).
>> - Make sure to update the change log [2] with any significant changes if
>> you haven't already. Send a PR with the change and tag me. (I imagine I'm
>> not the only one who forgot to do this :).)
>>
>> Thanks,
>> Kyle
>>
>> [1] https://github.com/apache/beam/blob/release-2.21.0
>> [2] https://github.com/apache/beam/blob/master/CHANGES.md
>>
>
Re: Beam 2.21 release update
Posted by Thomas Weise <th...@apache.org>.
I'm working on rebasing our fork to 2.21.0 and run into a problem
installing grpcio-tools that leads to *ModuleNotFoundError: No module named
'grpc_tools' *(see details below)
I cannot reproduce this locally.
Any =suggestions on what to look for?
Thanks,
Thomas
[?25hBuilding wheels for collected packages: future, gr
Running setup.py bdist_wheel for future ... [?25l- \
[?25h Stored in directory:
/root/.cache/pip/wheels/bf/c9/a3/c538d90ef17cf7823fa51fc701a7a7a910a80f6a405bf1
Running setup.py bdist_wheel for grpcio ... [?25l- \ | / - \ | / - \ | /
- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
| / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
| / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
| / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
| / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
| / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
| / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
| / - \ | / - \ |
[?25h Stored in directory:
/root/.cache/pip/wheels/00/4d/5f/07d0d4283911d2b917b867a11b1622d9d2cc8c286eefd1
Successfully built future grpcio
Installing collected packages: six, grpcio, setuptools, protobuf,
grpcio-tools, future, mypy-protobuf
Successfully installed future-0.16.0 grpcio-1.28.1 grpcio-tools-1.14.2
mypy-protobuf-1.18 protobuf-3.11.3 setuptools-46.1.3 six-1.14.0
WARNING:root:Installing grpcio-tools took 305.39 seconds.
INFO:gen_protos:Regenerating Python proto definitions (no output files).
Process Process-1:
Traceback (most recent call last):
File
"/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line
292, in generate_proto_files
from grpc_tools import protoc
ModuleNotFoundError: No module named 'grpc_tools'
On Fri, Apr 10, 2020 at 10:01 AM Kyle Weaver <kc...@google.com> wrote:
> Hi everyone,
>
> Just a heads up that the Beam 2.21 release branch [1] is cut.
> - If you find any important issues that you think should be addressed in
> the release, please tag the jira with fix version 2.21.0 and cc me
> (username `ibzib`).
> - Make sure to update the change log [2] with any significant changes if
> you haven't already. Send a PR with the change and tag me. (I imagine I'm
> not the only one who forgot to do this :).)
>
> Thanks,
> Kyle
>
> [1] https://github.com/apache/beam/blob/release-2.21.0
> [2] https://github.com/apache/beam/blob/master/CHANGES.md
>