You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Alex Amato <aj...@google.com> on 2020/06/26 21:51:03 UTC
Commands to detect style issues quickly before sending PR
I sent out some PRs a few days ago, and quickly discovered a bunch of
errors and have been spending most of my time playing wack-a-mole without
knowing how to repro them all locally.
I asked this a few years ago, and wanted to make sure I have something up
to date to work with. Ideally, I'd like a single command line for
simplicity. Here is what I've been using. I'm not sure if we have a script
or gradle target which already covers this or not
*Java*
time ./gradlew spotlessApply && ./gradlew checkstyleMain checkstyleTest
javadoc spotbugsMain compileJava compileTestJava
*Python *
./gradlew :sdks:python:test-suites:tox:py2:lintPy27_3 && ./gradlew
:sdks:python:test-suites:tox:py37:lintPy37
&& ./gradlew :sdks:python:test-suites:tox:py38:formatter
(I think this might be correct, maybe there is a faster way to run it
directly with tox as well)
========
Though the python command is failing for me, perhaps I need to install
another python version. I think we have setup steps for those in the wiki...
creating
build/temp.linux-x86_64-3.8/third_party/protobuf/src/google/protobuf/util/internal
creating
build/temp.linux-x86_64-3.8/third_party/protobuf/src/google/protobuf/stubs
creating
build/temp.linux-x86_64-3.8/third_party/protobuf/src/google/protobuf/io
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare
-DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat
-Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat
-Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC
-DHAVE_PTHREAD=1 -I. -Igrpc_root -Igrpc_root/include
-Ithird_party/protobuf/src -I/usr/include/python3.8
-I/usr/local/google/home/ajamato/beam/build/gradleenv/-1227304282/include/python3.8
-c grpc_tools/_protoc_compiler.cpp -o
build/temp.linux-x86_64-3.8/grpc_tools/_protoc_compiler.o -std=c++11
-fno-wrapv -frtti
grpc_tools/_protoc_compiler.cpp:216:10: fatal error: Python.h: No such
file or directory
216 | #include "Python.h"
| ^~~~~~~~~~
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1:
/usr/local/google/home/ajamato/beam/build/gradleenv/-1227304282/bin/python3.8
-u -c 'import sys, setuptools, tokenize; sys.argv[0] =
'"'"'/tmp/pip-install-xmf_k_sy/grpcio-tools/setup.py'"'"';
__file__='"'"'/tmp/pip-install-xmf_k_sy/grpcio-tools/setup.py'"'"';f=getattr(tokenize,
'"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"',
'"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))'
install --record /tmp/pip-record-9bhhuq55/install-record.txt
--single-version-externally-managed --compile --install-headers
/usr/local/google/home/ajamato/beam/build/gradleenv/-1227304282/include/site/python3.8/grpcio-tools
Check the logs for full command output.
*> Task :sdks:python:test-suites:tox:py38:setupVirtualenv* FAILED
Re: Commands to detect style issues quickly before sending PR
Posted by Udi Meiri <eh...@google.com>.
Another tip for Python:
Also "pip install pre-commit" will run yapf and pylint on changed files
during "git commit".
On Fri, Jun 26, 2020 at 4:03 PM Valentyn Tymofieiev <va...@google.com>
wrote:
> See also:
> https://cwiki.apache.org/confluence/display/BEAM/Python+Tips#PythonTips-Formatting
>
> > Task :sdks:python:test-suites:tox:py38:setupVirtualenv FAILED
> https://github.com/apache/beam/pull/12109 should fix that.
>
> On Fri, Jun 26, 2020 at 2:51 PM Alex Amato <aj...@google.com> wrote:
>
>> I sent out some PRs a few days ago, and quickly discovered a bunch of
>> errors and have been spending most of my time playing wack-a-mole without
>> knowing how to repro them all locally.
>>
>> I asked this a few years ago, and wanted to make sure I have something up
>> to date to work with. Ideally, I'd like a single command line for
>> simplicity. Here is what I've been using. I'm not sure if we have a script
>> or gradle target which already covers this or not
>>
>> *Java*
>> time ./gradlew spotlessApply && ./gradlew checkstyleMain checkstyleTest
>> javadoc spotbugsMain compileJava compileTestJava
>>
>> *Python *
>> ./gradlew :sdks:python:test-suites:tox:py2:lintPy27_3 && ./gradlew :sdks:python:test-suites:tox:py37:lintPy37
>> && ./gradlew :sdks:python:test-suites:tox:py38:formatter
>>
>> (I think this might be correct, maybe there is a faster way to run it
>> directly with tox as well)
>>
>> ========
>> Though the python command is failing for me, perhaps I need to install
>> another python version. I think we have setup steps for those in the wiki...
>>
>>
>> creating
>> build/temp.linux-x86_64-3.8/third_party/protobuf/src/google/protobuf/util/internal
>>
>> creating
>> build/temp.linux-x86_64-3.8/third_party/protobuf/src/google/protobuf/stubs
>>
>> creating
>> build/temp.linux-x86_64-3.8/third_party/protobuf/src/google/protobuf/io
>>
>> x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare
>> -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat
>> -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat
>> -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC
>> -DHAVE_PTHREAD=1 -I. -Igrpc_root -Igrpc_root/include
>> -Ithird_party/protobuf/src -I/usr/include/python3.8
>> -I/usr/local/google/home/ajamato/beam/build/gradleenv/-1227304282/include/python3.8
>> -c grpc_tools/_protoc_compiler.cpp -o
>> build/temp.linux-x86_64-3.8/grpc_tools/_protoc_compiler.o -std=c++11
>> -fno-wrapv -frtti
>>
>> grpc_tools/_protoc_compiler.cpp:216:10: fatal error: Python.h: No
>> such file or directory
>>
>> 216 | #include "Python.h"
>>
>> | ^~~~~~~~~~
>>
>> compilation terminated.
>>
>> error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
>>
>> ----------------------------------------
>>
>> ERROR: Command errored out with exit status 1:
>> /usr/local/google/home/ajamato/beam/build/gradleenv/-1227304282/bin/python3.8
>> -u -c 'import sys, setuptools, tokenize; sys.argv[0] =
>> '"'"'/tmp/pip-install-xmf_k_sy/grpcio-tools/setup.py'"'"';
>> __file__='"'"'/tmp/pip-install-xmf_k_sy/grpcio-tools/setup.py'"'"';f=getattr(tokenize,
>> '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"',
>> '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))'
>> install --record /tmp/pip-record-9bhhuq55/install-record.txt
>> --single-version-externally-managed --compile --install-headers
>> /usr/local/google/home/ajamato/beam/build/gradleenv/-1227304282/include/site/python3.8/grpcio-tools
>> Check the logs for full command output.
>>
>>
>> *> Task :sdks:python:test-suites:tox:py38:setupVirtualenv* FAILED
>>
>
Re: Commands to detect style issues quickly before sending PR
Posted by Valentyn Tymofieiev <va...@google.com>.
See also:
https://cwiki.apache.org/confluence/display/BEAM/Python+Tips#PythonTips-Formatting
> Task :sdks:python:test-suites:tox:py38:setupVirtualenv FAILED
https://github.com/apache/beam/pull/12109 should fix that.
On Fri, Jun 26, 2020 at 2:51 PM Alex Amato <aj...@google.com> wrote:
> I sent out some PRs a few days ago, and quickly discovered a bunch of
> errors and have been spending most of my time playing wack-a-mole without
> knowing how to repro them all locally.
>
> I asked this a few years ago, and wanted to make sure I have something up
> to date to work with. Ideally, I'd like a single command line for
> simplicity. Here is what I've been using. I'm not sure if we have a script
> or gradle target which already covers this or not
>
> *Java*
> time ./gradlew spotlessApply && ./gradlew checkstyleMain checkstyleTest
> javadoc spotbugsMain compileJava compileTestJava
>
> *Python *
> ./gradlew :sdks:python:test-suites:tox:py2:lintPy27_3 && ./gradlew :sdks:python:test-suites:tox:py37:lintPy37
> && ./gradlew :sdks:python:test-suites:tox:py38:formatter
>
> (I think this might be correct, maybe there is a faster way to run it
> directly with tox as well)
>
> ========
> Though the python command is failing for me, perhaps I need to install
> another python version. I think we have setup steps for those in the wiki...
>
>
> creating
> build/temp.linux-x86_64-3.8/third_party/protobuf/src/google/protobuf/util/internal
>
> creating
> build/temp.linux-x86_64-3.8/third_party/protobuf/src/google/protobuf/stubs
>
> creating
> build/temp.linux-x86_64-3.8/third_party/protobuf/src/google/protobuf/io
>
> x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare
> -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat
> -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat
> -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC
> -DHAVE_PTHREAD=1 -I. -Igrpc_root -Igrpc_root/include
> -Ithird_party/protobuf/src -I/usr/include/python3.8
> -I/usr/local/google/home/ajamato/beam/build/gradleenv/-1227304282/include/python3.8
> -c grpc_tools/_protoc_compiler.cpp -o
> build/temp.linux-x86_64-3.8/grpc_tools/_protoc_compiler.o -std=c++11
> -fno-wrapv -frtti
>
> grpc_tools/_protoc_compiler.cpp:216:10: fatal error: Python.h: No
> such file or directory
>
> 216 | #include "Python.h"
>
> | ^~~~~~~~~~
>
> compilation terminated.
>
> error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
>
> ----------------------------------------
>
> ERROR: Command errored out with exit status 1:
> /usr/local/google/home/ajamato/beam/build/gradleenv/-1227304282/bin/python3.8
> -u -c 'import sys, setuptools, tokenize; sys.argv[0] =
> '"'"'/tmp/pip-install-xmf_k_sy/grpcio-tools/setup.py'"'"';
> __file__='"'"'/tmp/pip-install-xmf_k_sy/grpcio-tools/setup.py'"'"';f=getattr(tokenize,
> '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"',
> '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))'
> install --record /tmp/pip-record-9bhhuq55/install-record.txt
> --single-version-externally-managed --compile --install-headers
> /usr/local/google/home/ajamato/beam/build/gradleenv/-1227304282/include/site/python3.8/grpcio-tools
> Check the logs for full command output.
>
>
> *> Task :sdks:python:test-suites:tox:py38:setupVirtualenv* FAILED
>