You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Kenneth Knowles <kl...@google.com.INVALID> on 2017/04/07 00:58:35 UTC
Python build artifacts seem to be misconfigured
Hi all,
It appears that the Python build process creates quite a few files that are
not accounted for in our .gitignore and that also trip the RAT check next
time around. These should be set up so that RAT and git both ignore the
files.
It is possible that others have defaults that differ from mine, but
droppings from a recent `mvn verify` include:
sdks/python/apache_beam/coders/coder_impl.c
sdks/python/apache_beam/coders/coder_impl.so
sdks/python/apache_beam/coders/stream.c
sdks/python/apache_beam/coders/stream.so
sdks/python/apache_beam/metrics/execution.c
sdks/python/apache_beam/metrics/execution.so
sdks/python/apache_beam/runners/common.c
sdks/python/apache_beam/runners/common.so
sdks/python/apache_beam/transforms/cy_combiners.c
sdks/python/apache_beam/transforms/cy_combiners.so
sdks/python/apache_beam/utils/counters.c
sdks/python/apache_beam/utils/counters.so
sdks/python/apache_beam/utils/windowed_value.c
sdks/python/apache_beam/utils/windowed_value.so
sdks/python/nose-1.3.7-py2.7.egg/
Can someone who knows the Python SDK build process rectify?
Kenn
Re: Python build artifacts seem to be misconfigured
Posted by Sergio Fernández <wi...@apache.org>.
Maybe we have to push-up the configurations the Maven build does
https://github.com/apache/beam/blob/master/sdks/python/pom.xml#L35
to the main Python build, so it stays the same whatever is invoked.
On Sat, Apr 15, 2017 at 3:28 AM, Ahmet Altay <al...@google.com.invalid>
wrote:
> Robert, would passing `build_dir='target/build'` to `cythonize` accomplish
> what Davor is saying?
>
> Using build_dir seems to be working but I could not find documentation of
> it.
>
> Ahmet
>
> On Wed, Apr 12, 2017 at 7:11 PM, Davor Bonaci <da...@apache.org> wrote:
>
> > I think it would be great if it can be configured that all (or, as many
> as
> > possible) build-generated files use one specific directory -- "target/".
> > Likely, all problems would just go away.
> >
> > On Wed, Apr 12, 2017 at 3:24 PM, Ahmet Altay <al...@google.com.invalid>
> > wrote:
> >
> > > This is also root cause for the flakiness in test_using_slow_impl very
> > > flaky locally tests (https://issues.apache.org/jira/browse/BEAM-1910).
> > >
> > > Kenn, have you found anything that might explain why tox is not
> deleting
> > > them?
> > >
> > > Ahmet
> > >
> > > On Tue, Apr 11, 2017 at 11:50 AM, Robert Bradshaw <
> > > robertwb@google.com.invalid> wrote:
> > >
> > > > We should also ignore them: https://github.com/apache/beam/pull/2494
> > > >
> > > > On Thu, Apr 6, 2017 at 6:45 PM, Kenneth Knowles
> <klk@google.com.invalid
> > >
> > > > wrote:
> > > > > Thanks for the pointer. I'll dig in to tox docs to see why this
> isn't
> > > > > happening. Probably something to do with unclean shutdowns.
> > > > >
> > > > > On Thu, Apr 6, 2017 at 6:10 PM, Vikas RK <vi...@gmail.com>
> wrote:
> > > > >
> > > > >> Those are cython generated files that should be deleted according
> to
> > > > >> https://github.com/apache/beam/blob/master/sdks/python/
> tox.ini#L54
> > > > >>
> > > > >>
> > > > >>
> > > > >> On 6 April 2017 at 17:58, Kenneth Knowles <klk@google.com.invalid
> >
> > > > wrote:
> > > > >>
> > > > >> > Hi all,
> > > > >> >
> > > > >> > It appears that the Python build process creates quite a few
> files
> > > > that
> > > > >> are
> > > > >> > not accounted for in our .gitignore and that also trip the RAT
> > check
> > > > next
> > > > >> > time around. These should be set up so that RAT and git both
> > ignore
> > > > the
> > > > >> > files.
> > > > >> >
> > > > >> > It is possible that others have defaults that differ from mine,
> > but
> > > > >> > droppings from a recent `mvn verify` include:
> > > > >> >
> > > > >> > sdks/python/apache_beam/coders/coder_impl.c
> > > > >> > sdks/python/apache_beam/coders/coder_impl.so
> > > > >> > sdks/python/apache_beam/coders/stream.c
> > > > >> > sdks/python/apache_beam/coders/stream.so
> > > > >> > sdks/python/apache_beam/metrics/execution.c
> > > > >> > sdks/python/apache_beam/metrics/execution.so
> > > > >> > sdks/python/apache_beam/runners/common.c
> > > > >> > sdks/python/apache_beam/runners/common.so
> > > > >> > sdks/python/apache_beam/transforms/cy_combiners.c
> > > > >> > sdks/python/apache_beam/transforms/cy_combiners.so
> > > > >> > sdks/python/apache_beam/utils/counters.c
> > > > >> > sdks/python/apache_beam/utils/counters.so
> > > > >> > sdks/python/apache_beam/utils/windowed_value.c
> > > > >> > sdks/python/apache_beam/utils/windowed_value.so
> > > > >> > sdks/python/nose-1.3.7-py2.7.egg/
> > > > >> >
> > > > >> > Can someone who knows the Python SDK build process rectify?
> > > > >> >
> > > > >> > Kenn
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
Re: Python build artifacts seem to be misconfigured
Posted by Ahmet Altay <al...@google.com.INVALID>.
Robert, would passing `build_dir='target/build'` to `cythonize` accomplish
what Davor is saying?
Using build_dir seems to be working but I could not find documentation of
it.
Ahmet
On Wed, Apr 12, 2017 at 7:11 PM, Davor Bonaci <da...@apache.org> wrote:
> I think it would be great if it can be configured that all (or, as many as
> possible) build-generated files use one specific directory -- "target/".
> Likely, all problems would just go away.
>
> On Wed, Apr 12, 2017 at 3:24 PM, Ahmet Altay <al...@google.com.invalid>
> wrote:
>
> > This is also root cause for the flakiness in test_using_slow_impl very
> > flaky locally tests (https://issues.apache.org/jira/browse/BEAM-1910).
> >
> > Kenn, have you found anything that might explain why tox is not deleting
> > them?
> >
> > Ahmet
> >
> > On Tue, Apr 11, 2017 at 11:50 AM, Robert Bradshaw <
> > robertwb@google.com.invalid> wrote:
> >
> > > We should also ignore them: https://github.com/apache/beam/pull/2494
> > >
> > > On Thu, Apr 6, 2017 at 6:45 PM, Kenneth Knowles <klk@google.com.invalid
> >
> > > wrote:
> > > > Thanks for the pointer. I'll dig in to tox docs to see why this isn't
> > > > happening. Probably something to do with unclean shutdowns.
> > > >
> > > > On Thu, Apr 6, 2017 at 6:10 PM, Vikas RK <vi...@gmail.com> wrote:
> > > >
> > > >> Those are cython generated files that should be deleted according to
> > > >> https://github.com/apache/beam/blob/master/sdks/python/tox.ini#L54
> > > >>
> > > >>
> > > >>
> > > >> On 6 April 2017 at 17:58, Kenneth Knowles <kl...@google.com.invalid>
> > > wrote:
> > > >>
> > > >> > Hi all,
> > > >> >
> > > >> > It appears that the Python build process creates quite a few files
> > > that
> > > >> are
> > > >> > not accounted for in our .gitignore and that also trip the RAT
> check
> > > next
> > > >> > time around. These should be set up so that RAT and git both
> ignore
> > > the
> > > >> > files.
> > > >> >
> > > >> > It is possible that others have defaults that differ from mine,
> but
> > > >> > droppings from a recent `mvn verify` include:
> > > >> >
> > > >> > sdks/python/apache_beam/coders/coder_impl.c
> > > >> > sdks/python/apache_beam/coders/coder_impl.so
> > > >> > sdks/python/apache_beam/coders/stream.c
> > > >> > sdks/python/apache_beam/coders/stream.so
> > > >> > sdks/python/apache_beam/metrics/execution.c
> > > >> > sdks/python/apache_beam/metrics/execution.so
> > > >> > sdks/python/apache_beam/runners/common.c
> > > >> > sdks/python/apache_beam/runners/common.so
> > > >> > sdks/python/apache_beam/transforms/cy_combiners.c
> > > >> > sdks/python/apache_beam/transforms/cy_combiners.so
> > > >> > sdks/python/apache_beam/utils/counters.c
> > > >> > sdks/python/apache_beam/utils/counters.so
> > > >> > sdks/python/apache_beam/utils/windowed_value.c
> > > >> > sdks/python/apache_beam/utils/windowed_value.so
> > > >> > sdks/python/nose-1.3.7-py2.7.egg/
> > > >> >
> > > >> > Can someone who knows the Python SDK build process rectify?
> > > >> >
> > > >> > Kenn
> > > >> >
> > > >>
> > >
> >
>
Re: Python build artifacts seem to be misconfigured
Posted by Davor Bonaci <da...@apache.org>.
I think it would be great if it can be configured that all (or, as many as
possible) build-generated files use one specific directory -- "target/".
Likely, all problems would just go away.
On Wed, Apr 12, 2017 at 3:24 PM, Ahmet Altay <al...@google.com.invalid>
wrote:
> This is also root cause for the flakiness in test_using_slow_impl very
> flaky locally tests (https://issues.apache.org/jira/browse/BEAM-1910).
>
> Kenn, have you found anything that might explain why tox is not deleting
> them?
>
> Ahmet
>
> On Tue, Apr 11, 2017 at 11:50 AM, Robert Bradshaw <
> robertwb@google.com.invalid> wrote:
>
> > We should also ignore them: https://github.com/apache/beam/pull/2494
> >
> > On Thu, Apr 6, 2017 at 6:45 PM, Kenneth Knowles <kl...@google.com.invalid>
> > wrote:
> > > Thanks for the pointer. I'll dig in to tox docs to see why this isn't
> > > happening. Probably something to do with unclean shutdowns.
> > >
> > > On Thu, Apr 6, 2017 at 6:10 PM, Vikas RK <vi...@gmail.com> wrote:
> > >
> > >> Those are cython generated files that should be deleted according to
> > >> https://github.com/apache/beam/blob/master/sdks/python/tox.ini#L54
> > >>
> > >>
> > >>
> > >> On 6 April 2017 at 17:58, Kenneth Knowles <kl...@google.com.invalid>
> > wrote:
> > >>
> > >> > Hi all,
> > >> >
> > >> > It appears that the Python build process creates quite a few files
> > that
> > >> are
> > >> > not accounted for in our .gitignore and that also trip the RAT check
> > next
> > >> > time around. These should be set up so that RAT and git both ignore
> > the
> > >> > files.
> > >> >
> > >> > It is possible that others have defaults that differ from mine, but
> > >> > droppings from a recent `mvn verify` include:
> > >> >
> > >> > sdks/python/apache_beam/coders/coder_impl.c
> > >> > sdks/python/apache_beam/coders/coder_impl.so
> > >> > sdks/python/apache_beam/coders/stream.c
> > >> > sdks/python/apache_beam/coders/stream.so
> > >> > sdks/python/apache_beam/metrics/execution.c
> > >> > sdks/python/apache_beam/metrics/execution.so
> > >> > sdks/python/apache_beam/runners/common.c
> > >> > sdks/python/apache_beam/runners/common.so
> > >> > sdks/python/apache_beam/transforms/cy_combiners.c
> > >> > sdks/python/apache_beam/transforms/cy_combiners.so
> > >> > sdks/python/apache_beam/utils/counters.c
> > >> > sdks/python/apache_beam/utils/counters.so
> > >> > sdks/python/apache_beam/utils/windowed_value.c
> > >> > sdks/python/apache_beam/utils/windowed_value.so
> > >> > sdks/python/nose-1.3.7-py2.7.egg/
> > >> >
> > >> > Can someone who knows the Python SDK build process rectify?
> > >> >
> > >> > Kenn
> > >> >
> > >>
> >
>
Re: Python build artifacts seem to be misconfigured
Posted by Ahmet Altay <al...@google.com.INVALID>.
This is also root cause for the flakiness in test_using_slow_impl very
flaky locally tests (https://issues.apache.org/jira/browse/BEAM-1910).
Kenn, have you found anything that might explain why tox is not deleting
them?
Ahmet
On Tue, Apr 11, 2017 at 11:50 AM, Robert Bradshaw <
robertwb@google.com.invalid> wrote:
> We should also ignore them: https://github.com/apache/beam/pull/2494
>
> On Thu, Apr 6, 2017 at 6:45 PM, Kenneth Knowles <kl...@google.com.invalid>
> wrote:
> > Thanks for the pointer. I'll dig in to tox docs to see why this isn't
> > happening. Probably something to do with unclean shutdowns.
> >
> > On Thu, Apr 6, 2017 at 6:10 PM, Vikas RK <vi...@gmail.com> wrote:
> >
> >> Those are cython generated files that should be deleted according to
> >> https://github.com/apache/beam/blob/master/sdks/python/tox.ini#L54
> >>
> >>
> >>
> >> On 6 April 2017 at 17:58, Kenneth Knowles <kl...@google.com.invalid>
> wrote:
> >>
> >> > Hi all,
> >> >
> >> > It appears that the Python build process creates quite a few files
> that
> >> are
> >> > not accounted for in our .gitignore and that also trip the RAT check
> next
> >> > time around. These should be set up so that RAT and git both ignore
> the
> >> > files.
> >> >
> >> > It is possible that others have defaults that differ from mine, but
> >> > droppings from a recent `mvn verify` include:
> >> >
> >> > sdks/python/apache_beam/coders/coder_impl.c
> >> > sdks/python/apache_beam/coders/coder_impl.so
> >> > sdks/python/apache_beam/coders/stream.c
> >> > sdks/python/apache_beam/coders/stream.so
> >> > sdks/python/apache_beam/metrics/execution.c
> >> > sdks/python/apache_beam/metrics/execution.so
> >> > sdks/python/apache_beam/runners/common.c
> >> > sdks/python/apache_beam/runners/common.so
> >> > sdks/python/apache_beam/transforms/cy_combiners.c
> >> > sdks/python/apache_beam/transforms/cy_combiners.so
> >> > sdks/python/apache_beam/utils/counters.c
> >> > sdks/python/apache_beam/utils/counters.so
> >> > sdks/python/apache_beam/utils/windowed_value.c
> >> > sdks/python/apache_beam/utils/windowed_value.so
> >> > sdks/python/nose-1.3.7-py2.7.egg/
> >> >
> >> > Can someone who knows the Python SDK build process rectify?
> >> >
> >> > Kenn
> >> >
> >>
>
Re: Python build artifacts seem to be misconfigured
Posted by Robert Bradshaw <ro...@google.com.INVALID>.
We should also ignore them: https://github.com/apache/beam/pull/2494
On Thu, Apr 6, 2017 at 6:45 PM, Kenneth Knowles <kl...@google.com.invalid> wrote:
> Thanks for the pointer. I'll dig in to tox docs to see why this isn't
> happening. Probably something to do with unclean shutdowns.
>
> On Thu, Apr 6, 2017 at 6:10 PM, Vikas RK <vi...@gmail.com> wrote:
>
>> Those are cython generated files that should be deleted according to
>> https://github.com/apache/beam/blob/master/sdks/python/tox.ini#L54
>>
>>
>>
>> On 6 April 2017 at 17:58, Kenneth Knowles <kl...@google.com.invalid> wrote:
>>
>> > Hi all,
>> >
>> > It appears that the Python build process creates quite a few files that
>> are
>> > not accounted for in our .gitignore and that also trip the RAT check next
>> > time around. These should be set up so that RAT and git both ignore the
>> > files.
>> >
>> > It is possible that others have defaults that differ from mine, but
>> > droppings from a recent `mvn verify` include:
>> >
>> > sdks/python/apache_beam/coders/coder_impl.c
>> > sdks/python/apache_beam/coders/coder_impl.so
>> > sdks/python/apache_beam/coders/stream.c
>> > sdks/python/apache_beam/coders/stream.so
>> > sdks/python/apache_beam/metrics/execution.c
>> > sdks/python/apache_beam/metrics/execution.so
>> > sdks/python/apache_beam/runners/common.c
>> > sdks/python/apache_beam/runners/common.so
>> > sdks/python/apache_beam/transforms/cy_combiners.c
>> > sdks/python/apache_beam/transforms/cy_combiners.so
>> > sdks/python/apache_beam/utils/counters.c
>> > sdks/python/apache_beam/utils/counters.so
>> > sdks/python/apache_beam/utils/windowed_value.c
>> > sdks/python/apache_beam/utils/windowed_value.so
>> > sdks/python/nose-1.3.7-py2.7.egg/
>> >
>> > Can someone who knows the Python SDK build process rectify?
>> >
>> > Kenn
>> >
>>
Re: Python build artifacts seem to be misconfigured
Posted by Kenneth Knowles <kl...@google.com.INVALID>.
Thanks for the pointer. I'll dig in to tox docs to see why this isn't
happening. Probably something to do with unclean shutdowns.
On Thu, Apr 6, 2017 at 6:10 PM, Vikas RK <vi...@gmail.com> wrote:
> Those are cython generated files that should be deleted according to
> https://github.com/apache/beam/blob/master/sdks/python/tox.ini#L54
>
>
>
> On 6 April 2017 at 17:58, Kenneth Knowles <kl...@google.com.invalid> wrote:
>
> > Hi all,
> >
> > It appears that the Python build process creates quite a few files that
> are
> > not accounted for in our .gitignore and that also trip the RAT check next
> > time around. These should be set up so that RAT and git both ignore the
> > files.
> >
> > It is possible that others have defaults that differ from mine, but
> > droppings from a recent `mvn verify` include:
> >
> > sdks/python/apache_beam/coders/coder_impl.c
> > sdks/python/apache_beam/coders/coder_impl.so
> > sdks/python/apache_beam/coders/stream.c
> > sdks/python/apache_beam/coders/stream.so
> > sdks/python/apache_beam/metrics/execution.c
> > sdks/python/apache_beam/metrics/execution.so
> > sdks/python/apache_beam/runners/common.c
> > sdks/python/apache_beam/runners/common.so
> > sdks/python/apache_beam/transforms/cy_combiners.c
> > sdks/python/apache_beam/transforms/cy_combiners.so
> > sdks/python/apache_beam/utils/counters.c
> > sdks/python/apache_beam/utils/counters.so
> > sdks/python/apache_beam/utils/windowed_value.c
> > sdks/python/apache_beam/utils/windowed_value.so
> > sdks/python/nose-1.3.7-py2.7.egg/
> >
> > Can someone who knows the Python SDK build process rectify?
> >
> > Kenn
> >
>
Re: Python build artifacts seem to be misconfigured
Posted by Vikas RK <vi...@gmail.com>.
Those are cython generated files that should be deleted according to
https://github.com/apache/beam/blob/master/sdks/python/tox.ini#L54
On 6 April 2017 at 17:58, Kenneth Knowles <kl...@google.com.invalid> wrote:
> Hi all,
>
> It appears that the Python build process creates quite a few files that are
> not accounted for in our .gitignore and that also trip the RAT check next
> time around. These should be set up so that RAT and git both ignore the
> files.
>
> It is possible that others have defaults that differ from mine, but
> droppings from a recent `mvn verify` include:
>
> sdks/python/apache_beam/coders/coder_impl.c
> sdks/python/apache_beam/coders/coder_impl.so
> sdks/python/apache_beam/coders/stream.c
> sdks/python/apache_beam/coders/stream.so
> sdks/python/apache_beam/metrics/execution.c
> sdks/python/apache_beam/metrics/execution.so
> sdks/python/apache_beam/runners/common.c
> sdks/python/apache_beam/runners/common.so
> sdks/python/apache_beam/transforms/cy_combiners.c
> sdks/python/apache_beam/transforms/cy_combiners.so
> sdks/python/apache_beam/utils/counters.c
> sdks/python/apache_beam/utils/counters.so
> sdks/python/apache_beam/utils/windowed_value.c
> sdks/python/apache_beam/utils/windowed_value.so
> sdks/python/nose-1.3.7-py2.7.egg/
>
> Can someone who knows the Python SDK build process rectify?
>
> Kenn
>