You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Charles Chen <cc...@google.com> on 2018/09/07 21:56:51 UTC
[VOTE] Release 2.7.0, release candidate #1
Hi everyone,
Please review and vote on the release candidate #1 for the version 2.7.0,
as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)
The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release to be deployed to dist.apache.org [2],
which is signed with the key with fingerprint 45C60AAAD115F560 [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "v2.7.0-RC1" [5],
* website pull request listing the release and publishing the API reference
manual [6].
* Java artifacts were built with Gradle 4.8 and OpenJDK
1.8.0_181-8u181-b13-1~deb9u1-b13.
* Python artifacts are deployed along with the source release to the
dist.apache.org [2].
The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.
Thanks,
Charles
[1]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
[2] https://dist.apache.org/repos/dist/dev/beam/2.7.0
[3] https://dist.apache.org/repos/dist/dev/beam/KEYS
[4] https://repository.apache.org/content/repositories/orgapachebeam-1046/
[5] https://github.com/apache/beam/tree/v2.7.0-RC1
[6] https://github.com/apache/beam-site/pull/549
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Maximilian Michels <mx...@apache.org>.
Could we still include some fixes for the RC2? I just discovered two
JIRA issues which were not properly marked with "Fix Version".
https://issues.apache.org/jira/browse/BEAM-5239
https://issues.apache.org/jira/browse/BEAM-5246
They are not show-stoppers, so also fine with me if we don't backport them.
-Max
On 11.09.18 11:44, Romain Manni-Bucau wrote:
> -1, seems spark integration is broken (tested with spark 2.3.1 and 2.2.1):
>
> 18/09/11 11:33:29 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, RMANNIBUCAU, executor 0): java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to fieldorg.apache.spark.rdd.RDD.org <http://org.apache.spark.rdd.RDD.org>$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD
> at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>
>
> Also the issue Lukasz identified is important even if workarounds can be
> put in place so +1 to fix it as well if possible.
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github
> <https://github.com/rmannibucau> | LinkedIn
> <https://www.linkedin.com/in/rmannibucau> | Book
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>
>
> Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik <lcwik@google.com
> <ma...@google.com>> a écrit :
>
> I found an issue where we are no longer packaging the pom.xml within
> the artifact jars at META-INF/maven/groupId/artifactId. More details
> in https://issues.apache.org/jira/browse/BEAM-5351. I wouldn't
> consider this a blocker but it was an easy fix
> (https://github.com/apache/beam/pull/6358) and users may rely on the
> pom.xml.
>
> Should we recut the release candidate to include this?
>
> On Mon, Sep 10, 2018 at 4:58 AM Jean-Baptiste Onofré
> <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
>
> +1 (binding)
>
> Tested successfully on Beam Samples.
>
> Thanks !
>
> Regards
> JB
>
> On 07/09/2018 23:56, Charles Chen wrote:
> > Hi everyone,
> >
> > Please review and vote on the release candidate #1 for the
> version
> > 2.7.0, as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific
> comments)
> >
> > The complete staging area is available for your review, which
> includes:
> > * JIRA release notes [1],
> > * the official Apache source release to be deployed to
> dist.apache.org <http://dist.apache.org>
> > <http://dist.apache.org> [2], which is signed with the key with
> > fingerprint 45C60AAAD115F560 [3],
> > * all artifacts to be deployed to the Maven Central
> Repository [4],
> > * source code tag "v2.7.0-RC1" [5],
> > * website pull request listing the release and publishing the API
> > reference manual [6].
> > * Java artifacts were built with Gradle 4.8 and OpenJDK
> > 1.8.0_181-8u181-b13-1~deb9u1-b13.
> > * Python artifacts are deployed along with the source release
> to the
> > dist.apache.org <http://dist.apache.org>
> <http://dist.apache.org> [2].
> >
> > The vote will be open for at least 72 hours. It is adopted by
> majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Charles
> >
> > [1]
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
> > [2] https://dist.apache.org/repos/dist/dev/beam/2.7.0
> > [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
> > [4]
> https://repository.apache.org/content/repositories/orgapachebeam-1046/
> > [5] https://github.com/apache/beam/tree/v2.7.0-RC1
> > [6] https://github.com/apache/beam-site/pull/549
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org <ma...@apache.org>
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Le mer. 12 sept. 2018 00:37, Lukasz Cwik <lc...@google.com> a écrit :
> I was unaware that users would use multiple versions of Apache Beam on the
> classpath at the same time. In that case I don't believe shading is
> something that will be there number one problem since we don't have a
> stable API surface between internal Apache Beam components.
>
Agree was exactly what I tried to say.
> For users who aren't using multiple Apache Beam packages, I would not
> expect non Apache Beam packages to ever export anything underneath the
> org.apache.beam package namespace.
>
Agree too.
> Also, I did add tooling to our build process to make sure that we only
> release classes underneath the org.apache.beam package namespace with the
> validateShadedJarDoesntLeakNonOrgApacheBeamClasses[1] task.
> 1:
> https://github.com/apache/beam/blob/a3f6f7e3b147f5a65e5b419d9baf24b35750974b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L751
>
> Romain, I think this is something we could continue outside of the release
> thread. Feel free to start a new thread or follow up on Slack.
>
The point was that Beam is hiding non beam issues with such a delivery
which is a blocker to upgrade. So beam alone is ok but if you add anything
- and since you will likely for any pipeline - then your app is no more in
a workable state while shades are a recommended solution.
> On Tue, Sep 11, 2018 at 2:48 PM Romain Manni-Bucau <rm...@gmail.com>
> wrote:
>
>> I understand Lukasz but it makes using shades properly pretty impossible
>> since this warning is not just something you can ignore but something you
>> have to fix since it can hide bugs. I get the "it is ok while you have a
>> single beam version" point but why would you get only beam in your
>> classpath, from the moment you use an IO it is not true anymore so this
>> warning is key to ensure your deployment is under control. In general you
>> accept something which fits the screen (like 20 overlapping classes or so)
>> but having 6600 classes to check is way more than something which would be
>> done just by a quick visual check. It requires you to add tooling on top of
>> it which is not really good overall. Wonder if it wouldn't be better to
>> revert that if it can't be completed short term and reapplied when possible
>> (probably using a working branch).
>>
>> Romain Manni-Bucau
>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>> <https://rmannibucau.metawerx.net/> | Old Blog
>> <http://rmannibucau.wordpress.com> | Github
>> <https://github.com/rmannibucau> | LinkedIn
>> <https://www.linkedin.com/in/rmannibucau> | Book
>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>
>>
>> Le mar. 11 sept. 2018 à 23:41, Lukasz Cwik <lc...@google.com> a écrit :
>>
>>> Romain, the beam-model-fn-execution-2.7.0.jar,
>>> beam-model-job-management-2.7.0.jar, beam-model-pipeline-2.7.0.jar have
>>> duplicates of the same classes to satisfy their dependencies (gRPC and
>>> protobuf and their transitive dependencies). Producing a separate artifact
>>> is still not done to prevent the message that your describing and other
>>> then size of jars, that message is benign in this case.
>>>
>>> Note that much of our vendoring goal that the community had discussed
>>> and agreed upon is still not unfinished, for example Guava:
>>> https://issues.apache.org/jira/browse/BEAM-3608
>>>
>>>
>>>
>>> On Tue, Sep 11, 2018 at 2:29 PM Romain Manni-Bucau <
>>> rmannibucau@gmail.com> wrote:
>>>
>>>> BTW, did you notice that doing a shade now logs something like:
>>>>
>>>> [WARNING] beam-model-fn-execution-2.7.0.jar,
>>>> beam-model-job-management-2.7.0.jar, beam-model-pipeline-2.7.0.jar define
>>>> 6660 overlapping classes:
>>>> [WARNING] -
>>>> org.apache.beam.vendor.netty.v4.io.netty.handler.codec.http.HttpClientCodec$1
>>>> [WARNING] -
>>>> org.apache.beam.vendor.guava.v20.com.google.common.util.concurrent.AggregateFutureState$SafeAtomicHelper
>>>> [WARNING] -
>>>> org.apache.beam.vendor.netty.v4.io.netty.util.concurrent.DefaultFutureListeners
>>>> [WARNING] -
>>>> org.apache.beam.vendor.netty.v4.io.netty.handler.ssl.OpenSslSessionContext$1
>>>> [WARNING] -
>>>> org.apache.beam.vendor.netty.v4.io.netty.handler.ssl.Java9SslUtils$4
>>>> [WARNING] -
>>>> org.apache.beam.vendor.guava.v20.com.google.common.collect.ImmutableMultimap$Builder
>>>> [WARNING] -
>>>> org.apache.beam.vendor.netty.v4.io.netty.handler.codec.spdy.SpdyHeaders
>>>> [WARNING] -
>>>> org.apache.beam.vendor.protobuf.v3.com.google.protobuf.DescriptorProtos$FieldDescriptorProtoOrBuilder
>>>> [WARNING] -
>>>> org.apache.beam.vendor.guava.v20.com.google.common.collect.AbstractMultimap
>>>> [WARNING] -
>>>> org.apache.beam.vendor.guava.v20.com.google.common.io.BaseEncoding$3
>>>> [WARNING] - 6650 more...
>>>>
>>>> Looks like the new shading policy impl was merged a bit too fast ;)
>>>>
>>>> Romain Manni-Bucau
>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>> <http://rmannibucau.wordpress.com> | Github
>>>> <https://github.com/rmannibucau> | LinkedIn
>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>
>>>>
>>>> Le mar. 11 sept. 2018 à 21:42, Jean-Baptiste Onofré <jb...@nanthrax.net>
>>>> a écrit :
>>>>
>>>>> I'm taking the Spark runner one.
>>>>>
>>>>> Regards
>>>>> JB
>>>>>
>>>>> On 11/09/2018 21:15, Ahmet Altay wrote:
>>>>> > Could anyone else help with looking at these issues earlier?
>>>>> >
>>>>> > On Tue, Sep 11, 2018 at 12:03 PM, Romain Manni-Bucau
>>>>> > <rmannibucau@gmail.com <ma...@gmail.com>> wrote:
>>>>> >
>>>>> > Im running this main [1] through this IT [2]. Was working fine
>>>>> since
>>>>> > ~1 year but 2.7.0 broke it. Didnt investigate more but can have a
>>>>> > look later this month if it helps.
>>>>> >
>>>>> > [1]
>>>>> >
>>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
>>>>> > <
>>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
>>>>> >
>>>>> > [2]
>>>>> >
>>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
>>>>> > <
>>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
>>>>> >
>>>>> >
>>>>> > Le mar. 11 sept. 2018 20:54, Charles Chen <ccy@google.com
>>>>> > <ma...@google.com>> a écrit :
>>>>> >
>>>>> > Romain: can you give more details on the failure you're
>>>>> > encountering, i.e. how you are performing this validation?
>>>>> >
>>>>> > On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste Onofré
>>>>> > <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
>>>>> >
>>>>> > Hi,
>>>>> >
>>>>> > weird, I didn't have it on Beam samples. Let me try to
>>>>> > reproduce and I
>>>>> > will create the Jira.
>>>>> >
>>>>> > Regards
>>>>> > JB
>>>>> >
>>>>> > On 11/09/2018 11:44, Romain Manni-Bucau wrote:
>>>>> > > -1, seems spark integration is broken (tested with
>>>>> spark
>>>>> > 2.3.1 and 2.2.1):
>>>>> > >
>>>>> > > 18/09/11 11:33:29 WARN TaskSetManager: Lost task 0.0
>>>>> in
>>>>> > stage 0.0 (TID 0, RMANNIBUCAU, executor 0):
>>>>> > java.lang.ClassCastException: cannot assign instance of
>>>>> > scala.collection.immutable.List$SerializationProxy to
>>>>> > fieldorg.apache.spark.rdd.RDD.org
>>>>> > <http://fieldorg.apache.spark.rdd.RDD.org>
>>>>> > <http://org.apache.spark.rdd.RDD.org
>>>>> > <http://org.apache.spark.rdd.RDD.org
>>>>> >>$apache$spark$rdd$RDD$$dependencies_
>>>>> > of type scala.collection.Seq in instance of
>>>>> > org.apache.spark.rdd.MapPartitionsRDD
>>>>> > > at
>>>>> >
>>>>> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>>>>> > >
>>>>> > >
>>>>> > > Also the issue Lukasz identified is important even if
>>>>> > workarounds can be
>>>>> > > put in place so +1 to fix it as well if possible.
>>>>> > >
>>>>> > > Romain Manni-Bucau
>>>>> > > @rmannibucau <https://twitter.com/rmannibucau
>>>>> > <https://twitter.com/rmannibucau>> | Blog
>>>>> > > <https://rmannibucau.metawerx.net/
>>>>> > <https://rmannibucau.metawerx.net/>> | Old Blog
>>>>> > > <http://rmannibucau.wordpress.com
>>>>> > <http://rmannibucau.wordpress.com>> | Github
>>>>> > > <https://github.com/rmannibucau
>>>>> > <https://github.com/rmannibucau>> | LinkedIn
>>>>> > > <https://www.linkedin.com/in/rmannibucau
>>>>> > <https://www.linkedin.com/in/rmannibucau>> | Book
>>>>> > >
>>>>> > <
>>>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>>> > <
>>>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>>> >>
>>>>> > >
>>>>> > >
>>>>> > > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik
>>>>> > <lcwik@google.com <ma...@google.com>
>>>>> > > <mailto:lcwik@google.com <ma...@google.com>>>
>>>>> a
>>>>> > écrit :
>>>>> > >
>>>>> > > I found an issue where we are no longer packaging
>>>>> the
>>>>> > pom.xml within
>>>>> > > the artifact jars at
>>>>> > META-INF/maven/groupId/artifactId. More details
>>>>> > > in
>>>>> https://issues.apache.org/jira/browse/BEAM-5351
>>>>> > <https://issues.apache.org/jira/browse/BEAM-5351>. I
>>>>> wouldn't
>>>>> > > consider this a blocker but it was an easy fix
>>>>> > > (https://github.com/apache/beam/pull/6358
>>>>> > <https://github.com/apache/beam/pull/6358>) and users
>>>>> may
>>>>> > rely on the
>>>>> > > pom.xml.
>>>>> > >
>>>>> > > Should we recut the release candidate to include
>>>>> this?
>>>>> > >
>>>>> > > On Mon, Sep 10, 2018 at 4:58 AM Jean-Baptiste
>>>>> Onofré
>>>>> > > <jb@nanthrax.net <ma...@nanthrax.net>
>>>>> > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>>
>>>>> wrote:
>>>>> > >
>>>>> > > +1 (binding)
>>>>> > >
>>>>> > > Tested successfully on Beam Samples.
>>>>> > >
>>>>> > > Thanks !
>>>>> > >
>>>>> > > Regards
>>>>> > > JB
>>>>> > >
>>>>> > > On 07/09/2018 23:56, Charles Chen wrote:
>>>>> > > > Hi everyone,
>>>>> > > >
>>>>> > > > Please review and vote on the release
>>>>> > candidate #1 for the
>>>>> > > version
>>>>> > > > 2.7.0, as follows:
>>>>> > > > [ ] +1, Approve the release
>>>>> > > > [ ] -1, Do not approve the release (please
>>>>> > provide specific
>>>>> > > comments)
>>>>> > > >
>>>>> > > > The complete staging area is available for
>>>>> > your review, which
>>>>> > > includes:
>>>>> > > > * JIRA release notes [1],
>>>>> > > > * the official Apache source release to be
>>>>> > deployed to
>>>>> > > dist.apache.org <http://dist.apache.org>
>>>>> > <http://dist.apache.org>
>>>>> > > > <http://dist.apache.org> [2], which is
>>>>> signed
>>>>> > with the key with
>>>>> > > > fingerprint 45C60AAAD115F560 [3],
>>>>> > > > * all artifacts to be deployed to the Maven
>>>>> > Central
>>>>> > > Repository [4],
>>>>> > > > * source code tag "v2.7.0-RC1" [5],
>>>>> > > > * website pull request listing the release
>>>>> and
>>>>> > publishing the API
>>>>> > > > reference manual [6].
>>>>> > > > * Java artifacts were built with Gradle 4.8
>>>>> > and OpenJDK
>>>>> > > > 1.8.0_181-8u181-b13-1~deb9u1-b13.
>>>>> > > > * Python artifacts are deployed along with
>>>>> the
>>>>> > source release
>>>>> > > to the
>>>>> > > > dist.apache.org <http://dist.apache.org>
>>>>> > <http://dist.apache.org>
>>>>> > > <http://dist.apache.org> [2].
>>>>> > > >
>>>>> > > > The vote will be open for at least 72
>>>>> hours.
>>>>> > It is adopted by
>>>>> > > majority
>>>>> > > > approval, with at least 3 PMC affirmative
>>>>> votes.
>>>>> > > >
>>>>> > > > Thanks,
>>>>> > > > Charles
>>>>> > > >
>>>>> > > > [1]
>>>>> > > >
>>>>> > >
>>>>> >
>>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>>>>> > <
>>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>>>>> >
>>>>> > > > [2]
>>>>> > https://dist.apache.org/repos/dist/dev/beam/2.7.0
>>>>> > <https://dist.apache.org/repos/dist/dev/beam/2.7.0>
>>>>> > > > [3]
>>>>> > https://dist.apache.org/repos/dist/dev/beam/KEYS
>>>>> > <https://dist.apache.org/repos/dist/dev/beam/KEYS>
>>>>> > > > [4]
>>>>> > >
>>>>> >
>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1046/
>>>>> > <
>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1046/
>>>>> >
>>>>> > > > [5]
>>>>> > https://github.com/apache/beam/tree/v2.7.0-RC1
>>>>> > <https://github.com/apache/beam/tree/v2.7.0-RC1>
>>>>> > > > [6]
>>>>> > https://github.com/apache/beam-site/pull/549
>>>>> > <https://github.com/apache/beam-site/pull/549>
>>>>> > >
>>>>> > > --
>>>>> > > Jean-Baptiste Onofré
>>>>> > > jbonofre@apache.org <ma...@apache.org>
>>>>> > <mailto:jbonofre@apache.org <mailto:jbonofre@apache.org
>>>>> >>
>>>>> > > http://blog.nanthrax.net
>>>>> > > Talend - http://www.talend.com
>>>>> > >
>>>>> >
>>>>> >
>>>>>
>>>>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Lukasz Cwik <lc...@google.com>.
I was unaware that users would use multiple versions of Apache Beam on the
classpath at the same time. In that case I don't believe shading is
something that will be there number one problem since we don't have a
stable API surface between internal Apache Beam components.
For users who aren't using multiple Apache Beam packages, I would not
expect non Apache Beam packages to ever export anything underneath the
org.apache.beam package namespace.
Also, I did add tooling to our build process to make sure that we only
release classes underneath the org.apache.beam package namespace with the
validateShadedJarDoesntLeakNonOrgApacheBeamClasses[1] task.
1:
https://github.com/apache/beam/blob/a3f6f7e3b147f5a65e5b419d9baf24b35750974b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L751
Romain, I think this is something we could continue outside of the release
thread. Feel free to start a new thread or follow up on Slack.
On Tue, Sep 11, 2018 at 2:48 PM Romain Manni-Bucau <rm...@gmail.com>
wrote:
> I understand Lukasz but it makes using shades properly pretty impossible
> since this warning is not just something you can ignore but something you
> have to fix since it can hide bugs. I get the "it is ok while you have a
> single beam version" point but why would you get only beam in your
> classpath, from the moment you use an IO it is not true anymore so this
> warning is key to ensure your deployment is under control. In general you
> accept something which fits the screen (like 20 overlapping classes or so)
> but having 6600 classes to check is way more than something which would be
> done just by a quick visual check. It requires you to add tooling on top of
> it which is not really good overall. Wonder if it wouldn't be better to
> revert that if it can't be completed short term and reapplied when possible
> (probably using a working branch).
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github
> <https://github.com/rmannibucau> | LinkedIn
> <https://www.linkedin.com/in/rmannibucau> | Book
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>
>
> Le mar. 11 sept. 2018 à 23:41, Lukasz Cwik <lc...@google.com> a écrit :
>
>> Romain, the beam-model-fn-execution-2.7.0.jar,
>> beam-model-job-management-2.7.0.jar, beam-model-pipeline-2.7.0.jar have
>> duplicates of the same classes to satisfy their dependencies (gRPC and
>> protobuf and their transitive dependencies). Producing a separate artifact
>> is still not done to prevent the message that your describing and other
>> then size of jars, that message is benign in this case.
>>
>> Note that much of our vendoring goal that the community had discussed and
>> agreed upon is still not unfinished, for example Guava:
>> https://issues.apache.org/jira/browse/BEAM-3608
>>
>>
>>
>> On Tue, Sep 11, 2018 at 2:29 PM Romain Manni-Bucau <rm...@gmail.com>
>> wrote:
>>
>>> BTW, did you notice that doing a shade now logs something like:
>>>
>>> [WARNING] beam-model-fn-execution-2.7.0.jar,
>>> beam-model-job-management-2.7.0.jar, beam-model-pipeline-2.7.0.jar define
>>> 6660 overlapping classes:
>>> [WARNING] -
>>> org.apache.beam.vendor.netty.v4.io.netty.handler.codec.http.HttpClientCodec$1
>>> [WARNING] -
>>> org.apache.beam.vendor.guava.v20.com.google.common.util.concurrent.AggregateFutureState$SafeAtomicHelper
>>> [WARNING] -
>>> org.apache.beam.vendor.netty.v4.io.netty.util.concurrent.DefaultFutureListeners
>>> [WARNING] -
>>> org.apache.beam.vendor.netty.v4.io.netty.handler.ssl.OpenSslSessionContext$1
>>> [WARNING] -
>>> org.apache.beam.vendor.netty.v4.io.netty.handler.ssl.Java9SslUtils$4
>>> [WARNING] -
>>> org.apache.beam.vendor.guava.v20.com.google.common.collect.ImmutableMultimap$Builder
>>> [WARNING] -
>>> org.apache.beam.vendor.netty.v4.io.netty.handler.codec.spdy.SpdyHeaders
>>> [WARNING] -
>>> org.apache.beam.vendor.protobuf.v3.com.google.protobuf.DescriptorProtos$FieldDescriptorProtoOrBuilder
>>> [WARNING] -
>>> org.apache.beam.vendor.guava.v20.com.google.common.collect.AbstractMultimap
>>> [WARNING] -
>>> org.apache.beam.vendor.guava.v20.com.google.common.io.BaseEncoding$3
>>> [WARNING] - 6650 more...
>>>
>>> Looks like the new shading policy impl was merged a bit too fast ;)
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>> <http://rmannibucau.wordpress.com> | Github
>>> <https://github.com/rmannibucau> | LinkedIn
>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>
>>>
>>> Le mar. 11 sept. 2018 à 21:42, Jean-Baptiste Onofré <jb...@nanthrax.net> a
>>> écrit :
>>>
>>>> I'm taking the Spark runner one.
>>>>
>>>> Regards
>>>> JB
>>>>
>>>> On 11/09/2018 21:15, Ahmet Altay wrote:
>>>> > Could anyone else help with looking at these issues earlier?
>>>> >
>>>> > On Tue, Sep 11, 2018 at 12:03 PM, Romain Manni-Bucau
>>>> > <rmannibucau@gmail.com <ma...@gmail.com>> wrote:
>>>> >
>>>> > Im running this main [1] through this IT [2]. Was working fine
>>>> since
>>>> > ~1 year but 2.7.0 broke it. Didnt investigate more but can have a
>>>> > look later this month if it helps.
>>>> >
>>>> > [1]
>>>> >
>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
>>>> > <
>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
>>>> >
>>>> > [2]
>>>> >
>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
>>>> > <
>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
>>>> >
>>>> >
>>>> > Le mar. 11 sept. 2018 20:54, Charles Chen <ccy@google.com
>>>> > <ma...@google.com>> a écrit :
>>>> >
>>>> > Romain: can you give more details on the failure you're
>>>> > encountering, i.e. how you are performing this validation?
>>>> >
>>>> > On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste Onofré
>>>> > <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
>>>> >
>>>> > Hi,
>>>> >
>>>> > weird, I didn't have it on Beam samples. Let me try to
>>>> > reproduce and I
>>>> > will create the Jira.
>>>> >
>>>> > Regards
>>>> > JB
>>>> >
>>>> > On 11/09/2018 11:44, Romain Manni-Bucau wrote:
>>>> > > -1, seems spark integration is broken (tested with
>>>> spark
>>>> > 2.3.1 and 2.2.1):
>>>> > >
>>>> > > 18/09/11 11:33:29 WARN TaskSetManager: Lost task 0.0 in
>>>> > stage 0.0 (TID 0, RMANNIBUCAU, executor 0):
>>>> > java.lang.ClassCastException: cannot assign instance of
>>>> > scala.collection.immutable.List$SerializationProxy to
>>>> > fieldorg.apache.spark.rdd.RDD.org
>>>> > <http://fieldorg.apache.spark.rdd.RDD.org>
>>>> > <http://org.apache.spark.rdd.RDD.org
>>>> > <http://org.apache.spark.rdd.RDD.org
>>>> >>$apache$spark$rdd$RDD$$dependencies_
>>>> > of type scala.collection.Seq in instance of
>>>> > org.apache.spark.rdd.MapPartitionsRDD
>>>> > > at
>>>> >
>>>> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>>>> > >
>>>> > >
>>>> > > Also the issue Lukasz identified is important even if
>>>> > workarounds can be
>>>> > > put in place so +1 to fix it as well if possible.
>>>> > >
>>>> > > Romain Manni-Bucau
>>>> > > @rmannibucau <https://twitter.com/rmannibucau
>>>> > <https://twitter.com/rmannibucau>> | Blog
>>>> > > <https://rmannibucau.metawerx.net/
>>>> > <https://rmannibucau.metawerx.net/>> | Old Blog
>>>> > > <http://rmannibucau.wordpress.com
>>>> > <http://rmannibucau.wordpress.com>> | Github
>>>> > > <https://github.com/rmannibucau
>>>> > <https://github.com/rmannibucau>> | LinkedIn
>>>> > > <https://www.linkedin.com/in/rmannibucau
>>>> > <https://www.linkedin.com/in/rmannibucau>> | Book
>>>> > >
>>>> > <
>>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>> > <
>>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>> >>
>>>> > >
>>>> > >
>>>> > > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik
>>>> > <lcwik@google.com <ma...@google.com>
>>>> > > <mailto:lcwik@google.com <ma...@google.com>>> a
>>>> > écrit :
>>>> > >
>>>> > > I found an issue where we are no longer packaging
>>>> the
>>>> > pom.xml within
>>>> > > the artifact jars at
>>>> > META-INF/maven/groupId/artifactId. More details
>>>> > > in https://issues.apache.org/jira/browse/BEAM-5351
>>>> > <https://issues.apache.org/jira/browse/BEAM-5351>. I
>>>> wouldn't
>>>> > > consider this a blocker but it was an easy fix
>>>> > > (https://github.com/apache/beam/pull/6358
>>>> > <https://github.com/apache/beam/pull/6358>) and users may
>>>> > rely on the
>>>> > > pom.xml.
>>>> > >
>>>> > > Should we recut the release candidate to include
>>>> this?
>>>> > >
>>>> > > On Mon, Sep 10, 2018 at 4:58 AM Jean-Baptiste
>>>> Onofré
>>>> > > <jb@nanthrax.net <ma...@nanthrax.net>
>>>> > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> wrote:
>>>> > >
>>>> > > +1 (binding)
>>>> > >
>>>> > > Tested successfully on Beam Samples.
>>>> > >
>>>> > > Thanks !
>>>> > >
>>>> > > Regards
>>>> > > JB
>>>> > >
>>>> > > On 07/09/2018 23:56, Charles Chen wrote:
>>>> > > > Hi everyone,
>>>> > > >
>>>> > > > Please review and vote on the release
>>>> > candidate #1 for the
>>>> > > version
>>>> > > > 2.7.0, as follows:
>>>> > > > [ ] +1, Approve the release
>>>> > > > [ ] -1, Do not approve the release (please
>>>> > provide specific
>>>> > > comments)
>>>> > > >
>>>> > > > The complete staging area is available for
>>>> > your review, which
>>>> > > includes:
>>>> > > > * JIRA release notes [1],
>>>> > > > * the official Apache source release to be
>>>> > deployed to
>>>> > > dist.apache.org <http://dist.apache.org>
>>>> > <http://dist.apache.org>
>>>> > > > <http://dist.apache.org> [2], which is
>>>> signed
>>>> > with the key with
>>>> > > > fingerprint 45C60AAAD115F560 [3],
>>>> > > > * all artifacts to be deployed to the Maven
>>>> > Central
>>>> > > Repository [4],
>>>> > > > * source code tag "v2.7.0-RC1" [5],
>>>> > > > * website pull request listing the release
>>>> and
>>>> > publishing the API
>>>> > > > reference manual [6].
>>>> > > > * Java artifacts were built with Gradle 4.8
>>>> > and OpenJDK
>>>> > > > 1.8.0_181-8u181-b13-1~deb9u1-b13.
>>>> > > > * Python artifacts are deployed along with
>>>> the
>>>> > source release
>>>> > > to the
>>>> > > > dist.apache.org <http://dist.apache.org>
>>>> > <http://dist.apache.org>
>>>> > > <http://dist.apache.org> [2].
>>>> > > >
>>>> > > > The vote will be open for at least 72 hours.
>>>> > It is adopted by
>>>> > > majority
>>>> > > > approval, with at least 3 PMC affirmative
>>>> votes.
>>>> > > >
>>>> > > > Thanks,
>>>> > > > Charles
>>>> > > >
>>>> > > > [1]
>>>> > > >
>>>> > >
>>>> >
>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>>>> > <
>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>>>> >
>>>> > > > [2]
>>>> > https://dist.apache.org/repos/dist/dev/beam/2.7.0
>>>> > <https://dist.apache.org/repos/dist/dev/beam/2.7.0>
>>>> > > > [3]
>>>> > https://dist.apache.org/repos/dist/dev/beam/KEYS
>>>> > <https://dist.apache.org/repos/dist/dev/beam/KEYS>
>>>> > > > [4]
>>>> > >
>>>> >
>>>> https://repository.apache.org/content/repositories/orgapachebeam-1046/
>>>> > <
>>>> https://repository.apache.org/content/repositories/orgapachebeam-1046/>
>>>> > > > [5]
>>>> > https://github.com/apache/beam/tree/v2.7.0-RC1
>>>> > <https://github.com/apache/beam/tree/v2.7.0-RC1>
>>>> > > > [6]
>>>> > https://github.com/apache/beam-site/pull/549
>>>> > <https://github.com/apache/beam-site/pull/549>
>>>> > >
>>>> > > --
>>>> > > Jean-Baptiste Onofré
>>>> > > jbonofre@apache.org <ma...@apache.org>
>>>> > <mailto:jbonofre@apache.org <ma...@apache.org>>
>>>> > > http://blog.nanthrax.net
>>>> > > Talend - http://www.talend.com
>>>> > >
>>>> >
>>>> >
>>>>
>>>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Romain Manni-Bucau <rm...@gmail.com>.
I understand Lukasz but it makes using shades properly pretty impossible
since this warning is not just something you can ignore but something you
have to fix since it can hide bugs. I get the "it is ok while you have a
single beam version" point but why would you get only beam in your
classpath, from the moment you use an IO it is not true anymore so this
warning is key to ensure your deployment is under control. In general you
accept something which fits the screen (like 20 overlapping classes or so)
but having 6600 classes to check is way more than something which would be
done just by a quick visual check. It requires you to add tooling on top of
it which is not really good overall. Wonder if it wouldn't be better to
revert that if it can't be completed short term and reapplied when possible
(probably using a working branch).
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>
Le mar. 11 sept. 2018 à 23:41, Lukasz Cwik <lc...@google.com> a écrit :
> Romain, the beam-model-fn-execution-2.7.0.jar,
> beam-model-job-management-2.7.0.jar, beam-model-pipeline-2.7.0.jar have
> duplicates of the same classes to satisfy their dependencies (gRPC and
> protobuf and their transitive dependencies). Producing a separate artifact
> is still not done to prevent the message that your describing and other
> then size of jars, that message is benign in this case.
>
> Note that much of our vendoring goal that the community had discussed and
> agreed upon is still not unfinished, for example Guava:
> https://issues.apache.org/jira/browse/BEAM-3608
>
>
>
> On Tue, Sep 11, 2018 at 2:29 PM Romain Manni-Bucau <rm...@gmail.com>
> wrote:
>
>> BTW, did you notice that doing a shade now logs something like:
>>
>> [WARNING] beam-model-fn-execution-2.7.0.jar,
>> beam-model-job-management-2.7.0.jar, beam-model-pipeline-2.7.0.jar define
>> 6660 overlapping classes:
>> [WARNING] -
>> org.apache.beam.vendor.netty.v4.io.netty.handler.codec.http.HttpClientCodec$1
>> [WARNING] -
>> org.apache.beam.vendor.guava.v20.com.google.common.util.concurrent.AggregateFutureState$SafeAtomicHelper
>> [WARNING] -
>> org.apache.beam.vendor.netty.v4.io.netty.util.concurrent.DefaultFutureListeners
>> [WARNING] -
>> org.apache.beam.vendor.netty.v4.io.netty.handler.ssl.OpenSslSessionContext$1
>> [WARNING] -
>> org.apache.beam.vendor.netty.v4.io.netty.handler.ssl.Java9SslUtils$4
>> [WARNING] -
>> org.apache.beam.vendor.guava.v20.com.google.common.collect.ImmutableMultimap$Builder
>> [WARNING] -
>> org.apache.beam.vendor.netty.v4.io.netty.handler.codec.spdy.SpdyHeaders
>> [WARNING] -
>> org.apache.beam.vendor.protobuf.v3.com.google.protobuf.DescriptorProtos$FieldDescriptorProtoOrBuilder
>> [WARNING] -
>> org.apache.beam.vendor.guava.v20.com.google.common.collect.AbstractMultimap
>> [WARNING] -
>> org.apache.beam.vendor.guava.v20.com.google.common.io.BaseEncoding$3
>> [WARNING] - 6650 more...
>>
>> Looks like the new shading policy impl was merged a bit too fast ;)
>>
>> Romain Manni-Bucau
>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>> <https://rmannibucau.metawerx.net/> | Old Blog
>> <http://rmannibucau.wordpress.com> | Github
>> <https://github.com/rmannibucau> | LinkedIn
>> <https://www.linkedin.com/in/rmannibucau> | Book
>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>
>>
>> Le mar. 11 sept. 2018 à 21:42, Jean-Baptiste Onofré <jb...@nanthrax.net> a
>> écrit :
>>
>>> I'm taking the Spark runner one.
>>>
>>> Regards
>>> JB
>>>
>>> On 11/09/2018 21:15, Ahmet Altay wrote:
>>> > Could anyone else help with looking at these issues earlier?
>>> >
>>> > On Tue, Sep 11, 2018 at 12:03 PM, Romain Manni-Bucau
>>> > <rmannibucau@gmail.com <ma...@gmail.com>> wrote:
>>> >
>>> > Im running this main [1] through this IT [2]. Was working fine
>>> since
>>> > ~1 year but 2.7.0 broke it. Didnt investigate more but can have a
>>> > look later this month if it helps.
>>> >
>>> > [1]
>>> >
>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
>>> > <
>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
>>> >
>>> > [2]
>>> >
>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
>>> > <
>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
>>> >
>>> >
>>> > Le mar. 11 sept. 2018 20:54, Charles Chen <ccy@google.com
>>> > <ma...@google.com>> a écrit :
>>> >
>>> > Romain: can you give more details on the failure you're
>>> > encountering, i.e. how you are performing this validation?
>>> >
>>> > On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste Onofré
>>> > <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
>>> >
>>> > Hi,
>>> >
>>> > weird, I didn't have it on Beam samples. Let me try to
>>> > reproduce and I
>>> > will create the Jira.
>>> >
>>> > Regards
>>> > JB
>>> >
>>> > On 11/09/2018 11:44, Romain Manni-Bucau wrote:
>>> > > -1, seems spark integration is broken (tested with spark
>>> > 2.3.1 and 2.2.1):
>>> > >
>>> > > 18/09/11 11:33:29 WARN TaskSetManager: Lost task 0.0 in
>>> > stage 0.0 (TID 0, RMANNIBUCAU, executor 0):
>>> > java.lang.ClassCastException: cannot assign instance of
>>> > scala.collection.immutable.List$SerializationProxy to
>>> > fieldorg.apache.spark.rdd.RDD.org
>>> > <http://fieldorg.apache.spark.rdd.RDD.org>
>>> > <http://org.apache.spark.rdd.RDD.org
>>> > <http://org.apache.spark.rdd.RDD.org
>>> >>$apache$spark$rdd$RDD$$dependencies_
>>> > of type scala.collection.Seq in instance of
>>> > org.apache.spark.rdd.MapPartitionsRDD
>>> > > at
>>> >
>>> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>>> > >
>>> > >
>>> > > Also the issue Lukasz identified is important even if
>>> > workarounds can be
>>> > > put in place so +1 to fix it as well if possible.
>>> > >
>>> > > Romain Manni-Bucau
>>> > > @rmannibucau <https://twitter.com/rmannibucau
>>> > <https://twitter.com/rmannibucau>> | Blog
>>> > > <https://rmannibucau.metawerx.net/
>>> > <https://rmannibucau.metawerx.net/>> | Old Blog
>>> > > <http://rmannibucau.wordpress.com
>>> > <http://rmannibucau.wordpress.com>> | Github
>>> > > <https://github.com/rmannibucau
>>> > <https://github.com/rmannibucau>> | LinkedIn
>>> > > <https://www.linkedin.com/in/rmannibucau
>>> > <https://www.linkedin.com/in/rmannibucau>> | Book
>>> > >
>>> > <
>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>> > <
>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>> >>
>>> > >
>>> > >
>>> > > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik
>>> > <lcwik@google.com <ma...@google.com>
>>> > > <mailto:lcwik@google.com <ma...@google.com>>> a
>>> > écrit :
>>> > >
>>> > > I found an issue where we are no longer packaging
>>> the
>>> > pom.xml within
>>> > > the artifact jars at
>>> > META-INF/maven/groupId/artifactId. More details
>>> > > in https://issues.apache.org/jira/browse/BEAM-5351
>>> > <https://issues.apache.org/jira/browse/BEAM-5351>. I
>>> wouldn't
>>> > > consider this a blocker but it was an easy fix
>>> > > (https://github.com/apache/beam/pull/6358
>>> > <https://github.com/apache/beam/pull/6358>) and users may
>>> > rely on the
>>> > > pom.xml.
>>> > >
>>> > > Should we recut the release candidate to include
>>> this?
>>> > >
>>> > > On Mon, Sep 10, 2018 at 4:58 AM Jean-Baptiste Onofré
>>> > > <jb@nanthrax.net <ma...@nanthrax.net>
>>> > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> wrote:
>>> > >
>>> > > +1 (binding)
>>> > >
>>> > > Tested successfully on Beam Samples.
>>> > >
>>> > > Thanks !
>>> > >
>>> > > Regards
>>> > > JB
>>> > >
>>> > > On 07/09/2018 23:56, Charles Chen wrote:
>>> > > > Hi everyone,
>>> > > >
>>> > > > Please review and vote on the release
>>> > candidate #1 for the
>>> > > version
>>> > > > 2.7.0, as follows:
>>> > > > [ ] +1, Approve the release
>>> > > > [ ] -1, Do not approve the release (please
>>> > provide specific
>>> > > comments)
>>> > > >
>>> > > > The complete staging area is available for
>>> > your review, which
>>> > > includes:
>>> > > > * JIRA release notes [1],
>>> > > > * the official Apache source release to be
>>> > deployed to
>>> > > dist.apache.org <http://dist.apache.org>
>>> > <http://dist.apache.org>
>>> > > > <http://dist.apache.org> [2], which is
>>> signed
>>> > with the key with
>>> > > > fingerprint 45C60AAAD115F560 [3],
>>> > > > * all artifacts to be deployed to the Maven
>>> > Central
>>> > > Repository [4],
>>> > > > * source code tag "v2.7.0-RC1" [5],
>>> > > > * website pull request listing the release
>>> and
>>> > publishing the API
>>> > > > reference manual [6].
>>> > > > * Java artifacts were built with Gradle 4.8
>>> > and OpenJDK
>>> > > > 1.8.0_181-8u181-b13-1~deb9u1-b13.
>>> > > > * Python artifacts are deployed along with
>>> the
>>> > source release
>>> > > to the
>>> > > > dist.apache.org <http://dist.apache.org>
>>> > <http://dist.apache.org>
>>> > > <http://dist.apache.org> [2].
>>> > > >
>>> > > > The vote will be open for at least 72 hours.
>>> > It is adopted by
>>> > > majority
>>> > > > approval, with at least 3 PMC affirmative
>>> votes.
>>> > > >
>>> > > > Thanks,
>>> > > > Charles
>>> > > >
>>> > > > [1]
>>> > > >
>>> > >
>>> >
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>>> > <
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>>> >
>>> > > > [2]
>>> > https://dist.apache.org/repos/dist/dev/beam/2.7.0
>>> > <https://dist.apache.org/repos/dist/dev/beam/2.7.0>
>>> > > > [3]
>>> > https://dist.apache.org/repos/dist/dev/beam/KEYS
>>> > <https://dist.apache.org/repos/dist/dev/beam/KEYS>
>>> > > > [4]
>>> > >
>>> >
>>> https://repository.apache.org/content/repositories/orgapachebeam-1046/
>>> > <
>>> https://repository.apache.org/content/repositories/orgapachebeam-1046/>
>>> > > > [5]
>>> > https://github.com/apache/beam/tree/v2.7.0-RC1
>>> > <https://github.com/apache/beam/tree/v2.7.0-RC1>
>>> > > > [6]
>>> > https://github.com/apache/beam-site/pull/549
>>> > <https://github.com/apache/beam-site/pull/549>
>>> > >
>>> > > --
>>> > > Jean-Baptiste Onofré
>>> > > jbonofre@apache.org <ma...@apache.org>
>>> > <mailto:jbonofre@apache.org <ma...@apache.org>>
>>> > > http://blog.nanthrax.net
>>> > > Talend - http://www.talend.com
>>> > >
>>> >
>>> >
>>>
>>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Lukasz Cwik <lc...@google.com>.
Romain, the beam-model-fn-execution-2.7.0.jar,
beam-model-job-management-2.7.0.jar, beam-model-pipeline-2.7.0.jar have
duplicates of the same classes to satisfy their dependencies (gRPC and
protobuf and their transitive dependencies). Producing a separate artifact
is still not done to prevent the message that your describing and other
then size of jars, that message is benign in this case.
Note that much of our vendoring goal that the community had discussed and
agreed upon is still not unfinished, for example Guava:
https://issues.apache.org/jira/browse/BEAM-3608
On Tue, Sep 11, 2018 at 2:29 PM Romain Manni-Bucau <rm...@gmail.com>
wrote:
> BTW, did you notice that doing a shade now logs something like:
>
> [WARNING] beam-model-fn-execution-2.7.0.jar,
> beam-model-job-management-2.7.0.jar, beam-model-pipeline-2.7.0.jar define
> 6660 overlapping classes:
> [WARNING] -
> org.apache.beam.vendor.netty.v4.io.netty.handler.codec.http.HttpClientCodec$1
> [WARNING] -
> org.apache.beam.vendor.guava.v20.com.google.common.util.concurrent.AggregateFutureState$SafeAtomicHelper
> [WARNING] -
> org.apache.beam.vendor.netty.v4.io.netty.util.concurrent.DefaultFutureListeners
> [WARNING] -
> org.apache.beam.vendor.netty.v4.io.netty.handler.ssl.OpenSslSessionContext$1
> [WARNING] -
> org.apache.beam.vendor.netty.v4.io.netty.handler.ssl.Java9SslUtils$4
> [WARNING] -
> org.apache.beam.vendor.guava.v20.com.google.common.collect.ImmutableMultimap$Builder
> [WARNING] -
> org.apache.beam.vendor.netty.v4.io.netty.handler.codec.spdy.SpdyHeaders
> [WARNING] -
> org.apache.beam.vendor.protobuf.v3.com.google.protobuf.DescriptorProtos$FieldDescriptorProtoOrBuilder
> [WARNING] -
> org.apache.beam.vendor.guava.v20.com.google.common.collect.AbstractMultimap
> [WARNING] -
> org.apache.beam.vendor.guava.v20.com.google.common.io.BaseEncoding$3
> [WARNING] - 6650 more...
>
> Looks like the new shading policy impl was merged a bit too fast ;)
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github
> <https://github.com/rmannibucau> | LinkedIn
> <https://www.linkedin.com/in/rmannibucau> | Book
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>
>
> Le mar. 11 sept. 2018 à 21:42, Jean-Baptiste Onofré <jb...@nanthrax.net> a
> écrit :
>
>> I'm taking the Spark runner one.
>>
>> Regards
>> JB
>>
>> On 11/09/2018 21:15, Ahmet Altay wrote:
>> > Could anyone else help with looking at these issues earlier?
>> >
>> > On Tue, Sep 11, 2018 at 12:03 PM, Romain Manni-Bucau
>> > <rmannibucau@gmail.com <ma...@gmail.com>> wrote:
>> >
>> > Im running this main [1] through this IT [2]. Was working fine since
>> > ~1 year but 2.7.0 broke it. Didnt investigate more but can have a
>> > look later this month if it helps.
>> >
>> > [1]
>> >
>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
>> > <
>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
>> >
>> > [2]
>> >
>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
>> > <
>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
>> >
>> >
>> > Le mar. 11 sept. 2018 20:54, Charles Chen <ccy@google.com
>> > <ma...@google.com>> a écrit :
>> >
>> > Romain: can you give more details on the failure you're
>> > encountering, i.e. how you are performing this validation?
>> >
>> > On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste Onofré
>> > <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
>> >
>> > Hi,
>> >
>> > weird, I didn't have it on Beam samples. Let me try to
>> > reproduce and I
>> > will create the Jira.
>> >
>> > Regards
>> > JB
>> >
>> > On 11/09/2018 11:44, Romain Manni-Bucau wrote:
>> > > -1, seems spark integration is broken (tested with spark
>> > 2.3.1 and 2.2.1):
>> > >
>> > > 18/09/11 11:33:29 WARN TaskSetManager: Lost task 0.0 in
>> > stage 0.0 (TID 0, RMANNIBUCAU, executor 0):
>> > java.lang.ClassCastException: cannot assign instance of
>> > scala.collection.immutable.List$SerializationProxy to
>> > fieldorg.apache.spark.rdd.RDD.org
>> > <http://fieldorg.apache.spark.rdd.RDD.org>
>> > <http://org.apache.spark.rdd.RDD.org
>> > <http://org.apache.spark.rdd.RDD.org
>> >>$apache$spark$rdd$RDD$$dependencies_
>> > of type scala.collection.Seq in instance of
>> > org.apache.spark.rdd.MapPartitionsRDD
>> > > at
>> >
>> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>> > >
>> > >
>> > > Also the issue Lukasz identified is important even if
>> > workarounds can be
>> > > put in place so +1 to fix it as well if possible.
>> > >
>> > > Romain Manni-Bucau
>> > > @rmannibucau <https://twitter.com/rmannibucau
>> > <https://twitter.com/rmannibucau>> | Blog
>> > > <https://rmannibucau.metawerx.net/
>> > <https://rmannibucau.metawerx.net/>> | Old Blog
>> > > <http://rmannibucau.wordpress.com
>> > <http://rmannibucau.wordpress.com>> | Github
>> > > <https://github.com/rmannibucau
>> > <https://github.com/rmannibucau>> | LinkedIn
>> > > <https://www.linkedin.com/in/rmannibucau
>> > <https://www.linkedin.com/in/rmannibucau>> | Book
>> > >
>> > <
>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>> > <
>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>> >>
>> > >
>> > >
>> > > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik
>> > <lcwik@google.com <ma...@google.com>
>> > > <mailto:lcwik@google.com <ma...@google.com>>> a
>> > écrit :
>> > >
>> > > I found an issue where we are no longer packaging the
>> > pom.xml within
>> > > the artifact jars at
>> > META-INF/maven/groupId/artifactId. More details
>> > > in https://issues.apache.org/jira/browse/BEAM-5351
>> > <https://issues.apache.org/jira/browse/BEAM-5351>. I
>> wouldn't
>> > > consider this a blocker but it was an easy fix
>> > > (https://github.com/apache/beam/pull/6358
>> > <https://github.com/apache/beam/pull/6358>) and users may
>> > rely on the
>> > > pom.xml.
>> > >
>> > > Should we recut the release candidate to include
>> this?
>> > >
>> > > On Mon, Sep 10, 2018 at 4:58 AM Jean-Baptiste Onofré
>> > > <jb@nanthrax.net <ma...@nanthrax.net>
>> > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> wrote:
>> > >
>> > > +1 (binding)
>> > >
>> > > Tested successfully on Beam Samples.
>> > >
>> > > Thanks !
>> > >
>> > > Regards
>> > > JB
>> > >
>> > > On 07/09/2018 23:56, Charles Chen wrote:
>> > > > Hi everyone,
>> > > >
>> > > > Please review and vote on the release
>> > candidate #1 for the
>> > > version
>> > > > 2.7.0, as follows:
>> > > > [ ] +1, Approve the release
>> > > > [ ] -1, Do not approve the release (please
>> > provide specific
>> > > comments)
>> > > >
>> > > > The complete staging area is available for
>> > your review, which
>> > > includes:
>> > > > * JIRA release notes [1],
>> > > > * the official Apache source release to be
>> > deployed to
>> > > dist.apache.org <http://dist.apache.org>
>> > <http://dist.apache.org>
>> > > > <http://dist.apache.org> [2], which is signed
>> > with the key with
>> > > > fingerprint 45C60AAAD115F560 [3],
>> > > > * all artifacts to be deployed to the Maven
>> > Central
>> > > Repository [4],
>> > > > * source code tag "v2.7.0-RC1" [5],
>> > > > * website pull request listing the release and
>> > publishing the API
>> > > > reference manual [6].
>> > > > * Java artifacts were built with Gradle 4.8
>> > and OpenJDK
>> > > > 1.8.0_181-8u181-b13-1~deb9u1-b13.
>> > > > * Python artifacts are deployed along with the
>> > source release
>> > > to the
>> > > > dist.apache.org <http://dist.apache.org>
>> > <http://dist.apache.org>
>> > > <http://dist.apache.org> [2].
>> > > >
>> > > > The vote will be open for at least 72 hours.
>> > It is adopted by
>> > > majority
>> > > > approval, with at least 3 PMC affirmative
>> votes.
>> > > >
>> > > > Thanks,
>> > > > Charles
>> > > >
>> > > > [1]
>> > > >
>> > >
>> >
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>> > <
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>> >
>> > > > [2]
>> > https://dist.apache.org/repos/dist/dev/beam/2.7.0
>> > <https://dist.apache.org/repos/dist/dev/beam/2.7.0>
>> > > > [3]
>> > https://dist.apache.org/repos/dist/dev/beam/KEYS
>> > <https://dist.apache.org/repos/dist/dev/beam/KEYS>
>> > > > [4]
>> > >
>> >
>> https://repository.apache.org/content/repositories/orgapachebeam-1046/
>> > <
>> https://repository.apache.org/content/repositories/orgapachebeam-1046/>
>> > > > [5]
>> > https://github.com/apache/beam/tree/v2.7.0-RC1
>> > <https://github.com/apache/beam/tree/v2.7.0-RC1>
>> > > > [6]
>> > https://github.com/apache/beam-site/pull/549
>> > <https://github.com/apache/beam-site/pull/549>
>> > >
>> > > --
>> > > Jean-Baptiste Onofré
>> > > jbonofre@apache.org <ma...@apache.org>
>> > <mailto:jbonofre@apache.org <ma...@apache.org>>
>> > > http://blog.nanthrax.net
>> > > Talend - http://www.talend.com
>> > >
>> >
>> >
>>
>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Romain Manni-Bucau <rm...@gmail.com>.
BTW, did you notice that doing a shade now logs something like:
[WARNING] beam-model-fn-execution-2.7.0.jar,
beam-model-job-management-2.7.0.jar, beam-model-pipeline-2.7.0.jar define
6660 overlapping classes:
[WARNING] -
org.apache.beam.vendor.netty.v4.io.netty.handler.codec.http.HttpClientCodec$1
[WARNING] -
org.apache.beam.vendor.guava.v20.com.google.common.util.concurrent.AggregateFutureState$SafeAtomicHelper
[WARNING] -
org.apache.beam.vendor.netty.v4.io.netty.util.concurrent.DefaultFutureListeners
[WARNING] -
org.apache.beam.vendor.netty.v4.io.netty.handler.ssl.OpenSslSessionContext$1
[WARNING] -
org.apache.beam.vendor.netty.v4.io.netty.handler.ssl.Java9SslUtils$4
[WARNING] -
org.apache.beam.vendor.guava.v20.com.google.common.collect.ImmutableMultimap$Builder
[WARNING] -
org.apache.beam.vendor.netty.v4.io.netty.handler.codec.spdy.SpdyHeaders
[WARNING] -
org.apache.beam.vendor.protobuf.v3.com.google.protobuf.DescriptorProtos$FieldDescriptorProtoOrBuilder
[WARNING] -
org.apache.beam.vendor.guava.v20.com.google.common.collect.AbstractMultimap
[WARNING] -
org.apache.beam.vendor.guava.v20.com.google.common.io.BaseEncoding$3
[WARNING] - 6650 more...
Looks like the new shading policy impl was merged a bit too fast ;)
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>
Le mar. 11 sept. 2018 à 21:42, Jean-Baptiste Onofré <jb...@nanthrax.net> a
écrit :
> I'm taking the Spark runner one.
>
> Regards
> JB
>
> On 11/09/2018 21:15, Ahmet Altay wrote:
> > Could anyone else help with looking at these issues earlier?
> >
> > On Tue, Sep 11, 2018 at 12:03 PM, Romain Manni-Bucau
> > <rmannibucau@gmail.com <ma...@gmail.com>> wrote:
> >
> > Im running this main [1] through this IT [2]. Was working fine since
> > ~1 year but 2.7.0 broke it. Didnt investigate more but can have a
> > look later this month if it helps.
> >
> > [1]
> >
> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
> > <
> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
> >
> > [2]
> >
> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
> > <
> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
> >
> >
> > Le mar. 11 sept. 2018 20:54, Charles Chen <ccy@google.com
> > <ma...@google.com>> a écrit :
> >
> > Romain: can you give more details on the failure you're
> > encountering, i.e. how you are performing this validation?
> >
> > On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste Onofré
> > <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
> >
> > Hi,
> >
> > weird, I didn't have it on Beam samples. Let me try to
> > reproduce and I
> > will create the Jira.
> >
> > Regards
> > JB
> >
> > On 11/09/2018 11:44, Romain Manni-Bucau wrote:
> > > -1, seems spark integration is broken (tested with spark
> > 2.3.1 and 2.2.1):
> > >
> > > 18/09/11 11:33:29 WARN TaskSetManager: Lost task 0.0 in
> > stage 0.0 (TID 0, RMANNIBUCAU, executor 0):
> > java.lang.ClassCastException: cannot assign instance of
> > scala.collection.immutable.List$SerializationProxy to
> > fieldorg.apache.spark.rdd.RDD.org
> > <http://fieldorg.apache.spark.rdd.RDD.org>
> > <http://org.apache.spark.rdd.RDD.org
> > <http://org.apache.spark.rdd.RDD.org
> >>$apache$spark$rdd$RDD$$dependencies_
> > of type scala.collection.Seq in instance of
> > org.apache.spark.rdd.MapPartitionsRDD
> > > at
> >
> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
> > >
> > >
> > > Also the issue Lukasz identified is important even if
> > workarounds can be
> > > put in place so +1 to fix it as well if possible.
> > >
> > > Romain Manni-Bucau
> > > @rmannibucau <https://twitter.com/rmannibucau
> > <https://twitter.com/rmannibucau>> | Blog
> > > <https://rmannibucau.metawerx.net/
> > <https://rmannibucau.metawerx.net/>> | Old Blog
> > > <http://rmannibucau.wordpress.com
> > <http://rmannibucau.wordpress.com>> | Github
> > > <https://github.com/rmannibucau
> > <https://github.com/rmannibucau>> | LinkedIn
> > > <https://www.linkedin.com/in/rmannibucau
> > <https://www.linkedin.com/in/rmannibucau>> | Book
> > >
> > <
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> > <
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> >>
> > >
> > >
> > > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik
> > <lcwik@google.com <ma...@google.com>
> > > <mailto:lcwik@google.com <ma...@google.com>>> a
> > écrit :
> > >
> > > I found an issue where we are no longer packaging the
> > pom.xml within
> > > the artifact jars at
> > META-INF/maven/groupId/artifactId. More details
> > > in https://issues.apache.org/jira/browse/BEAM-5351
> > <https://issues.apache.org/jira/browse/BEAM-5351>. I
> wouldn't
> > > consider this a blocker but it was an easy fix
> > > (https://github.com/apache/beam/pull/6358
> > <https://github.com/apache/beam/pull/6358>) and users may
> > rely on the
> > > pom.xml.
> > >
> > > Should we recut the release candidate to include this?
> > >
> > > On Mon, Sep 10, 2018 at 4:58 AM Jean-Baptiste Onofré
> > > <jb@nanthrax.net <ma...@nanthrax.net>
> > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> wrote:
> > >
> > > +1 (binding)
> > >
> > > Tested successfully on Beam Samples.
> > >
> > > Thanks !
> > >
> > > Regards
> > > JB
> > >
> > > On 07/09/2018 23:56, Charles Chen wrote:
> > > > Hi everyone,
> > > >
> > > > Please review and vote on the release
> > candidate #1 for the
> > > version
> > > > 2.7.0, as follows:
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please
> > provide specific
> > > comments)
> > > >
> > > > The complete staging area is available for
> > your review, which
> > > includes:
> > > > * JIRA release notes [1],
> > > > * the official Apache source release to be
> > deployed to
> > > dist.apache.org <http://dist.apache.org>
> > <http://dist.apache.org>
> > > > <http://dist.apache.org> [2], which is signed
> > with the key with
> > > > fingerprint 45C60AAAD115F560 [3],
> > > > * all artifacts to be deployed to the Maven
> > Central
> > > Repository [4],
> > > > * source code tag "v2.7.0-RC1" [5],
> > > > * website pull request listing the release and
> > publishing the API
> > > > reference manual [6].
> > > > * Java artifacts were built with Gradle 4.8
> > and OpenJDK
> > > > 1.8.0_181-8u181-b13-1~deb9u1-b13.
> > > > * Python artifacts are deployed along with the
> > source release
> > > to the
> > > > dist.apache.org <http://dist.apache.org>
> > <http://dist.apache.org>
> > > <http://dist.apache.org> [2].
> > > >
> > > > The vote will be open for at least 72 hours.
> > It is adopted by
> > > majority
> > > > approval, with at least 3 PMC affirmative
> votes.
> > > >
> > > > Thanks,
> > > > Charles
> > > >
> > > > [1]
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
> > <
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
> >
> > > > [2]
> > https://dist.apache.org/repos/dist/dev/beam/2.7.0
> > <https://dist.apache.org/repos/dist/dev/beam/2.7.0>
> > > > [3]
> > https://dist.apache.org/repos/dist/dev/beam/KEYS
> > <https://dist.apache.org/repos/dist/dev/beam/KEYS>
> > > > [4]
> > >
> >
> https://repository.apache.org/content/repositories/orgapachebeam-1046/
> > <
> https://repository.apache.org/content/repositories/orgapachebeam-1046/>
> > > > [5]
> > https://github.com/apache/beam/tree/v2.7.0-RC1
> > <https://github.com/apache/beam/tree/v2.7.0-RC1>
> > > > [6]
> > https://github.com/apache/beam-site/pull/549
> > <https://github.com/apache/beam-site/pull/549>
> > >
> > > --
> > > Jean-Baptiste Onofré
> > > jbonofre@apache.org <ma...@apache.org>
> > <mailto:jbonofre@apache.org <ma...@apache.org>>
> > > http://blog.nanthrax.net
> > > Talend - http://www.talend.com
> > >
> >
> >
>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
I'm taking the Spark runner one.
Regards
JB
On 11/09/2018 21:15, Ahmet Altay wrote:
> Could anyone else help with looking at these issues earlier?
>
> On Tue, Sep 11, 2018 at 12:03 PM, Romain Manni-Bucau
> <rmannibucau@gmail.com <ma...@gmail.com>> wrote:
>
> Im running this main [1] through this IT [2]. Was working fine since
> ~1 year but 2.7.0 broke it. Didnt investigate more but can have a
> look later this month if it helps.
>
> [1]
> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
> <https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java>
> [2]
> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
> <https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java>
>
> Le mar. 11 sept. 2018 20:54, Charles Chen <ccy@google.com
> <ma...@google.com>> a écrit :
>
> Romain: can you give more details on the failure you're
> encountering, i.e. how you are performing this validation?
>
> On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste Onofré
> <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
>
> Hi,
>
> weird, I didn't have it on Beam samples. Let me try to
> reproduce and I
> will create the Jira.
>
> Regards
> JB
>
> On 11/09/2018 11:44, Romain Manni-Bucau wrote:
> > -1, seems spark integration is broken (tested with spark
> 2.3.1 and 2.2.1):
> >
> > 18/09/11 11:33:29 WARN TaskSetManager: Lost task 0.0 in
> stage 0.0 (TID 0, RMANNIBUCAU, executor 0):
> java.lang.ClassCastException: cannot assign instance of
> scala.collection.immutable.List$SerializationProxy to
> fieldorg.apache.spark.rdd.RDD.org
> <http://fieldorg.apache.spark.rdd.RDD.org>
> <http://org.apache.spark.rdd.RDD.org
> <http://org.apache.spark.rdd.RDD.org>>$apache$spark$rdd$RDD$$dependencies_
> of type scala.collection.Seq in instance of
> org.apache.spark.rdd.MapPartitionsRDD
> > at
> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
> >
> >
> > Also the issue Lukasz identified is important even if
> workarounds can be
> > put in place so +1 to fix it as well if possible.
> >
> > Romain Manni-Bucau
> > @rmannibucau <https://twitter.com/rmannibucau
> <https://twitter.com/rmannibucau>> | Blog
> > <https://rmannibucau.metawerx.net/
> <https://rmannibucau.metawerx.net/>> | Old Blog
> > <http://rmannibucau.wordpress.com
> <http://rmannibucau.wordpress.com>> | Github
> > <https://github.com/rmannibucau
> <https://github.com/rmannibucau>> | LinkedIn
> > <https://www.linkedin.com/in/rmannibucau
> <https://www.linkedin.com/in/rmannibucau>> | Book
> >
> <https://www.packtpub.com/application-development/java-ee-8-high-performance
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>>
> >
> >
> > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik
> <lcwik@google.com <ma...@google.com>
> > <mailto:lcwik@google.com <ma...@google.com>>> a
> écrit :
> >
> > I found an issue where we are no longer packaging the
> pom.xml within
> > the artifact jars at
> META-INF/maven/groupId/artifactId. More details
> > in https://issues.apache.org/jira/browse/BEAM-5351
> <https://issues.apache.org/jira/browse/BEAM-5351>. I wouldn't
> > consider this a blocker but it was an easy fix
> > (https://github.com/apache/beam/pull/6358
> <https://github.com/apache/beam/pull/6358>) and users may
> rely on the
> > pom.xml.
> >
> > Should we recut the release candidate to include this?
> >
> > On Mon, Sep 10, 2018 at 4:58 AM Jean-Baptiste Onofré
> > <jb@nanthrax.net <ma...@nanthrax.net>
> <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> wrote:
> >
> > +1 (binding)
> >
> > Tested successfully on Beam Samples.
> >
> > Thanks !
> >
> > Regards
> > JB
> >
> > On 07/09/2018 23:56, Charles Chen wrote:
> > > Hi everyone,
> > >
> > > Please review and vote on the release
> candidate #1 for the
> > version
> > > 2.7.0, as follows:
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please
> provide specific
> > comments)
> > >
> > > The complete staging area is available for
> your review, which
> > includes:
> > > * JIRA release notes [1],
> > > * the official Apache source release to be
> deployed to
> > dist.apache.org <http://dist.apache.org>
> <http://dist.apache.org>
> > > <http://dist.apache.org> [2], which is signed
> with the key with
> > > fingerprint 45C60AAAD115F560 [3],
> > > * all artifacts to be deployed to the Maven
> Central
> > Repository [4],
> > > * source code tag "v2.7.0-RC1" [5],
> > > * website pull request listing the release and
> publishing the API
> > > reference manual [6].
> > > * Java artifacts were built with Gradle 4.8
> and OpenJDK
> > > 1.8.0_181-8u181-b13-1~deb9u1-b13.
> > > * Python artifacts are deployed along with the
> source release
> > to the
> > > dist.apache.org <http://dist.apache.org>
> <http://dist.apache.org>
> > <http://dist.apache.org> [2].
> > >
> > > The vote will be open for at least 72 hours.
> It is adopted by
> > majority
> > > approval, with at least 3 PMC affirmative votes.
> > >
> > > Thanks,
> > > Charles
> > >
> > > [1]
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654>
> > > [2]
> https://dist.apache.org/repos/dist/dev/beam/2.7.0
> <https://dist.apache.org/repos/dist/dev/beam/2.7.0>
> > > [3]
> https://dist.apache.org/repos/dist/dev/beam/KEYS
> <https://dist.apache.org/repos/dist/dev/beam/KEYS>
> > > [4]
> >
> https://repository.apache.org/content/repositories/orgapachebeam-1046/
> <https://repository.apache.org/content/repositories/orgapachebeam-1046/>
> > > [5]
> https://github.com/apache/beam/tree/v2.7.0-RC1
> <https://github.com/apache/beam/tree/v2.7.0-RC1>
> > > [6]
> https://github.com/apache/beam-site/pull/549
> <https://github.com/apache/beam-site/pull/549>
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org <ma...@apache.org>
> <mailto:jbonofre@apache.org <ma...@apache.org>>
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Romain Manni-Bucau <rm...@gmail.com>.
@Charles: just "mvn verify" (it is an integration-test)
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>
Le lun. 17 sept. 2018 à 19:54, Charles Chen <cc...@google.com> a écrit :
> Can you send the specific command / config you are using to reproduce?
>
> On Mon, Sep 17, 2018 at 10:28 AM Romain Manni-Bucau <rm...@gmail.com>
> wrote:
>
>> @Charles: guess you can download
>> https://github.com/Talend/component-runtime/tree/master/component-runtime-beam/src/it/serialization-over-cluster
>> subproject, replace project.version by 1.0.4 (other placeholders are on the
>> net/more known) and you should be able to reproduce forcing beam to be in
>> v2.7.0. I dont have much time this week to check out this particular issue
>> but hopefully next one should be more doable if the issue is still pending.
>>
>> Romain Manni-Bucau
>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>> <https://rmannibucau.metawerx.net/> | Old Blog
>> <http://rmannibucau.wordpress.com> | Github
>> <https://github.com/rmannibucau> | LinkedIn
>> <https://www.linkedin.com/in/rmannibucau> | Book
>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>
>>
>> Le lun. 17 sept. 2018 à 19:18, Charles Chen <cc...@google.com> a écrit :
>>
>>> Luke, Maximillian, Raghu, can you please propose cherry-pick PRs to the
>>> release-2.7.0 for your issues and add me as a reviewer (@charlesccychen)?
>>>
>>> Romain, JB: is there any way I can help with debugging the issue you're
>>> facing so we can unblock the release?
>>>
>>> On Fri, Sep 14, 2018 at 1:49 PM Raghu Angadi <ra...@google.com> wrote:
>>>
>>>> I would like propose one more cherrypick for RC2 :
>>>> https://github.com/apache/beam/pull/6391
>>>> This is a KafkaIO bug fix. Once a user hits this bug, there is no easy
>>>> work around for them, especially on Dataflow. Only work around in Dataflow
>>>> is to restart or reload the job.
>>>>
>>>> The fix itself fairly safe and is tested.
>>>> Raghu.
>>>>
>>>> On Fri, Sep 14, 2018 at 12:52 AM Alexey Romanenko <
>>>> aromanenko.dev@gmail.com> wrote:
>>>>
>>>>> Perhaps it could help, but I run simple WordCount (built with Beam
>>>>> 2.7) on YARN/Spark (HDP Sandbox) cluster and it worked fine for me.
>>>>>
>>>>> On 14 Sep 2018, at 06:56, Romain Manni-Bucau <rm...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi Charles,
>>>>>
>>>>> I didn't get enough time to check deeply but it is clearly a
>>>>> dependency issue and it is not in beam spark runner itself but in another
>>>>> transitive module of beam. It does not happen in existing spark test cause
>>>>> none of them are in a cluster (even just with 1 worker) but this seems to
>>>>> be a regression since 2.6 works OOTB.
>>>>>
>>>>> Romain Manni-Bucau
>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>> <http://rmannibucau.wordpress.com/> | Github
>>>>> <https://github.com/rmannibucau> | LinkedIn
>>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>>
>>>>>
>>>>> Le jeu. 13 sept. 2018 à 22:15, Charles Chen <cc...@google.com> a écrit :
>>>>>
>>>>>> Romain and JB, can you please add the results of your investigations
>>>>>> into the errors you've seen above? Given that the existing SparkRunner
>>>>>> tests pass for this RC, and that the integration test you ran is in another
>>>>>> repo that is not continuously tested with Beam, it is not clear how we
>>>>>> should move forward and whether this is a blocking issue, unless we can
>>>>>> find a root cause in Beam.
>>>>>>
>>>>>> On Wed, Sep 12, 2018 at 2:08 AM Etienne Chauchot <
>>>>>> echauchot@apache.org> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> on a performance and functional regression stand point I see no
>>>>>>> regression:
>>>>>>>
>>>>>>> I looked at nexmark graphs "output pcollection size" and "execution
>>>>>>> time" around release cut date on dataflow, spark, flink and direct runner
>>>>>>> in batch and streaming modes. There seems to be no regression.
>>>>>>>
>>>>>>> Etienne
>>>>>>>
>>>>>>> Le mardi 11 septembre 2018 à 12:25 -0700, Charles Chen a écrit :
>>>>>>>
>>>>>>> The SparkRunner validation test (here:
>>>>>>> https://beam.apache.org/contribute/release-guide/#run-validation-tests)
>>>>>>> passes on my machine. It looks like we are likely missing test coverage
>>>>>>> where Romain is hitting issues.
>>>>>>>
>>>>>>> On Tue, Sep 11, 2018 at 12:15 PM Ahmet Altay <al...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Could anyone else help with looking at these issues earlier?
>>>>>>>
>>>>>>> On Tue, Sep 11, 2018 at 12:03 PM, Romain Manni-Bucau <
>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>
>>>>>>> Im running this main [1] through this IT [2]. Was working fine since
>>>>>>> ~1 year but 2.7.0 broke it. Didnt investigate more but can have a look
>>>>>>> later this month if it helps.
>>>>>>>
>>>>>>> [1]
>>>>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
>>>>>>> [2]
>>>>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
>>>>>>>
>>>>>>> Le mar. 11 sept. 2018 20:54, Charles Chen <cc...@google.com> a écrit :
>>>>>>>
>>>>>>> Romain: can you give more details on the failure you're
>>>>>>> encountering, i.e. how you are performing this validation?
>>>>>>>
>>>>>>> On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste Onofré <
>>>>>>> jb@nanthrax.net> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> weird, I didn't have it on Beam samples. Let me try to reproduce and
>>>>>>> I
>>>>>>> will create the Jira.
>>>>>>>
>>>>>>> Regards
>>>>>>> JB
>>>>>>>
>>>>>>> On 11/09/2018 11:44, Romain Manni-Bucau wrote:
>>>>>>> > -1, seems spark integration is broken (tested with spark 2.3.1 and
>>>>>>> 2.2.1):
>>>>>>> >
>>>>>>> > 18/09/11 11:33:29 WARN TaskSetManager: Lost task 0.0 in stage 0.0
>>>>>>> (TID 0, RMANNIBUCAU, executor 0): java.lang.ClassCastException: cannot
>>>>>>> assign instance of scala.collection.immutable.List$SerializationProxy to
>>>>>>> fieldorg.apache.spark.rdd.RDD.org
>>>>>>> <http://fieldorg.apache.spark.rdd.rdd.org/> <
>>>>>>> http://org.apache.spark.rdd.RDD.org
>>>>>>> <http://org.apache.spark.rdd.rdd.org/>>$apache$spark$rdd$RDD$$dependencies_
>>>>>>> of type scala.collection.Seq in instance of
>>>>>>> org.apache.spark.rdd.MapPartitionsRDD
>>>>>>> > at
>>>>>>> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>>>>>>> >
>>>>>>> >
>>>>>>> > Also the issue Lukasz identified is important even if workarounds
>>>>>>> can be
>>>>>>> > put in place so +1 to fix it as well if possible.
>>>>>>> >
>>>>>>> > Romain Manni-Bucau
>>>>>>> > @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>>>> > <https://rmannibucau.metawerx.net/> | Old Blog
>>>>>>> > <http://rmannibucau.wordpress.com> | Github
>>>>>>> > <https://github.com/rmannibucau> | LinkedIn
>>>>>>> > <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>>> > <
>>>>>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik <lcwik@google.com
>>>>>>> > <ma...@google.com>> a écrit :
>>>>>>> >
>>>>>>> > I found an issue where we are no longer packaging the pom.xml
>>>>>>> within
>>>>>>> > the artifact jars at META-INF/maven/groupId/artifactId. More
>>>>>>> details
>>>>>>> > in https://issues.apache.org/jira/browse/BEAM-5351. I wouldn't
>>>>>>> > consider this a blocker but it was an easy fix
>>>>>>> > (https://github.com/apache/beam/pull/6358) and users may rely
>>>>>>> on the
>>>>>>> > pom.xml.
>>>>>>> >
>>>>>>> > Should we recut the release candidate to include this?
>>>>>>> >
>>>>>>> > On Mon, Sep 10, 2018 at 4:58 AM Jean-Baptiste Onofré
>>>>>>> > <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
>>>>>>> >
>>>>>>> > +1 (binding)
>>>>>>> >
>>>>>>> > Tested successfully on Beam Samples.
>>>>>>> >
>>>>>>> > Thanks !
>>>>>>> >
>>>>>>> > Regards
>>>>>>> > JB
>>>>>>> >
>>>>>>> > On 07/09/2018 23:56, Charles Chen wrote:
>>>>>>> > > Hi everyone,
>>>>>>> > >
>>>>>>> > > Please review and vote on the release candidate #1 for
>>>>>>> the
>>>>>>> > version
>>>>>>> > > 2.7.0, as follows:
>>>>>>> > > [ ] +1, Approve the release
>>>>>>> > > [ ] -1, Do not approve the release (please provide
>>>>>>> specific
>>>>>>> > comments)
>>>>>>> > >
>>>>>>> > > The complete staging area is available for your review,
>>>>>>> which
>>>>>>> > includes:
>>>>>>> > > * JIRA release notes [1],
>>>>>>> > > * the official Apache source release to be deployed to
>>>>>>> > dist.apache.org <http://dist.apache.org>
>>>>>>> > > <http://dist.apache.org> [2], which is signed with the
>>>>>>> key with
>>>>>>> > > fingerprint 45C60AAAD115F560 [3],
>>>>>>> > > * all artifacts to be deployed to the Maven Central
>>>>>>> > Repository [4],
>>>>>>> > > * source code tag "v2.7.0-RC1" [5],
>>>>>>> > > * website pull request listing the release and
>>>>>>> publishing the API
>>>>>>> > > reference manual [6].
>>>>>>> > > * Java artifacts were built with Gradle 4.8 and OpenJDK
>>>>>>> > > 1.8.0_181-8u181-b13-1~deb9u1-b13.
>>>>>>> > > * Python artifacts are deployed along with the source
>>>>>>> release
>>>>>>> > to the
>>>>>>> > > dist.apache.org <http://dist.apache.org>
>>>>>>> > <http://dist.apache.org> [2].
>>>>>>> > >
>>>>>>> > > The vote will be open for at least 72 hours. It is
>>>>>>> adopted by
>>>>>>> > majority
>>>>>>> > > approval, with at least 3 PMC affirmative votes.
>>>>>>> > >
>>>>>>> > > Thanks,
>>>>>>> > > Charles
>>>>>>> > >
>>>>>>> > > [1]
>>>>>>> > >
>>>>>>> >
>>>>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>>>>>>> > > [2] https://dist.apache.org/repos/dist/dev/beam/2.7.0
>>>>>>> > > [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
>>>>>>> > > [4]
>>>>>>> >
>>>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1046/
>>>>>>> > > [5] https://github.com/apache/beam/tree/v2.7.0-RC1
>>>>>>> > > [6] https://github.com/apache/beam-site/pull/549
>>>>>>> >
>>>>>>> > --
>>>>>>> > Jean-Baptiste Onofré
>>>>>>> > jbonofre@apache.org <ma...@apache.org>
>>>>>>> > http://blog.nanthrax.net
>>>>>>> > Talend - http://www.talend.com
>>>>>>> >
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Lukasz Cwik <lc...@google.com>.
I created this cherry pick for the missing pom.xml issue:
https://github.com/apache/beam/pull/6412
On Mon, Sep 17, 2018 at 10:54 AM Charles Chen <cc...@google.com> wrote:
> Can you send the specific command / config you are using to reproduce?
>
> On Mon, Sep 17, 2018 at 10:28 AM Romain Manni-Bucau <rm...@gmail.com>
> wrote:
>
>> @Charles: guess you can download
>> https://github.com/Talend/component-runtime/tree/master/component-runtime-beam/src/it/serialization-over-cluster
>> subproject, replace project.version by 1.0.4 (other placeholders are on the
>> net/more known) and you should be able to reproduce forcing beam to be in
>> v2.7.0. I dont have much time this week to check out this particular issue
>> but hopefully next one should be more doable if the issue is still pending.
>>
>> Romain Manni-Bucau
>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>> <https://rmannibucau.metawerx.net/> | Old Blog
>> <http://rmannibucau.wordpress.com> | Github
>> <https://github.com/rmannibucau> | LinkedIn
>> <https://www.linkedin.com/in/rmannibucau> | Book
>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>
>>
>> Le lun. 17 sept. 2018 à 19:18, Charles Chen <cc...@google.com> a écrit :
>>
>>> Luke, Maximillian, Raghu, can you please propose cherry-pick PRs to the
>>> release-2.7.0 for your issues and add me as a reviewer (@charlesccychen)?
>>>
>>> Romain, JB: is there any way I can help with debugging the issue you're
>>> facing so we can unblock the release?
>>>
>>> On Fri, Sep 14, 2018 at 1:49 PM Raghu Angadi <ra...@google.com> wrote:
>>>
>>>> I would like propose one more cherrypick for RC2 :
>>>> https://github.com/apache/beam/pull/6391
>>>> This is a KafkaIO bug fix. Once a user hits this bug, there is no easy
>>>> work around for them, especially on Dataflow. Only work around in Dataflow
>>>> is to restart or reload the job.
>>>>
>>>> The fix itself fairly safe and is tested.
>>>> Raghu.
>>>>
>>>> On Fri, Sep 14, 2018 at 12:52 AM Alexey Romanenko <
>>>> aromanenko.dev@gmail.com> wrote:
>>>>
>>>>> Perhaps it could help, but I run simple WordCount (built with Beam
>>>>> 2.7) on YARN/Spark (HDP Sandbox) cluster and it worked fine for me.
>>>>>
>>>>> On 14 Sep 2018, at 06:56, Romain Manni-Bucau <rm...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi Charles,
>>>>>
>>>>> I didn't get enough time to check deeply but it is clearly a
>>>>> dependency issue and it is not in beam spark runner itself but in another
>>>>> transitive module of beam. It does not happen in existing spark test cause
>>>>> none of them are in a cluster (even just with 1 worker) but this seems to
>>>>> be a regression since 2.6 works OOTB.
>>>>>
>>>>> Romain Manni-Bucau
>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>> <http://rmannibucau.wordpress.com/> | Github
>>>>> <https://github.com/rmannibucau> | LinkedIn
>>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>>
>>>>>
>>>>> Le jeu. 13 sept. 2018 à 22:15, Charles Chen <cc...@google.com> a écrit :
>>>>>
>>>>>> Romain and JB, can you please add the results of your investigations
>>>>>> into the errors you've seen above? Given that the existing SparkRunner
>>>>>> tests pass for this RC, and that the integration test you ran is in another
>>>>>> repo that is not continuously tested with Beam, it is not clear how we
>>>>>> should move forward and whether this is a blocking issue, unless we can
>>>>>> find a root cause in Beam.
>>>>>>
>>>>>> On Wed, Sep 12, 2018 at 2:08 AM Etienne Chauchot <
>>>>>> echauchot@apache.org> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> on a performance and functional regression stand point I see no
>>>>>>> regression:
>>>>>>>
>>>>>>> I looked at nexmark graphs "output pcollection size" and "execution
>>>>>>> time" around release cut date on dataflow, spark, flink and direct runner
>>>>>>> in batch and streaming modes. There seems to be no regression.
>>>>>>>
>>>>>>> Etienne
>>>>>>>
>>>>>>> Le mardi 11 septembre 2018 à 12:25 -0700, Charles Chen a écrit :
>>>>>>>
>>>>>>> The SparkRunner validation test (here:
>>>>>>> https://beam.apache.org/contribute/release-guide/#run-validation-tests)
>>>>>>> passes on my machine. It looks like we are likely missing test coverage
>>>>>>> where Romain is hitting issues.
>>>>>>>
>>>>>>> On Tue, Sep 11, 2018 at 12:15 PM Ahmet Altay <al...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Could anyone else help with looking at these issues earlier?
>>>>>>>
>>>>>>> On Tue, Sep 11, 2018 at 12:03 PM, Romain Manni-Bucau <
>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>
>>>>>>> Im running this main [1] through this IT [2]. Was working fine since
>>>>>>> ~1 year but 2.7.0 broke it. Didnt investigate more but can have a look
>>>>>>> later this month if it helps.
>>>>>>>
>>>>>>> [1]
>>>>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
>>>>>>> [2]
>>>>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
>>>>>>>
>>>>>>> Le mar. 11 sept. 2018 20:54, Charles Chen <cc...@google.com> a écrit :
>>>>>>>
>>>>>>> Romain: can you give more details on the failure you're
>>>>>>> encountering, i.e. how you are performing this validation?
>>>>>>>
>>>>>>> On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste Onofré <
>>>>>>> jb@nanthrax.net> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> weird, I didn't have it on Beam samples. Let me try to reproduce and
>>>>>>> I
>>>>>>> will create the Jira.
>>>>>>>
>>>>>>> Regards
>>>>>>> JB
>>>>>>>
>>>>>>> On 11/09/2018 11:44, Romain Manni-Bucau wrote:
>>>>>>> > -1, seems spark integration is broken (tested with spark 2.3.1 and
>>>>>>> 2.2.1):
>>>>>>> >
>>>>>>> > 18/09/11 11:33:29 WARN TaskSetManager: Lost task 0.0 in stage 0.0
>>>>>>> (TID 0, RMANNIBUCAU, executor 0): java.lang.ClassCastException: cannot
>>>>>>> assign instance of scala.collection.immutable.List$SerializationProxy to
>>>>>>> fieldorg.apache.spark.rdd.RDD.org
>>>>>>> <http://fieldorg.apache.spark.rdd.rdd.org/> <
>>>>>>> http://org.apache.spark.rdd.RDD.org
>>>>>>> <http://org.apache.spark.rdd.rdd.org/>>$apache$spark$rdd$RDD$$dependencies_
>>>>>>> of type scala.collection.Seq in instance of
>>>>>>> org.apache.spark.rdd.MapPartitionsRDD
>>>>>>> > at
>>>>>>> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>>>>>>> >
>>>>>>> >
>>>>>>> > Also the issue Lukasz identified is important even if workarounds
>>>>>>> can be
>>>>>>> > put in place so +1 to fix it as well if possible.
>>>>>>> >
>>>>>>> > Romain Manni-Bucau
>>>>>>> > @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>>>> > <https://rmannibucau.metawerx.net/> | Old Blog
>>>>>>> > <http://rmannibucau.wordpress.com> | Github
>>>>>>> > <https://github.com/rmannibucau> | LinkedIn
>>>>>>> > <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>>> > <
>>>>>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik <lcwik@google.com
>>>>>>> > <ma...@google.com>> a écrit :
>>>>>>> >
>>>>>>> > I found an issue where we are no longer packaging the pom.xml
>>>>>>> within
>>>>>>> > the artifact jars at META-INF/maven/groupId/artifactId. More
>>>>>>> details
>>>>>>> > in https://issues.apache.org/jira/browse/BEAM-5351. I wouldn't
>>>>>>> > consider this a blocker but it was an easy fix
>>>>>>> > (https://github.com/apache/beam/pull/6358) and users may rely
>>>>>>> on the
>>>>>>> > pom.xml.
>>>>>>> >
>>>>>>> > Should we recut the release candidate to include this?
>>>>>>> >
>>>>>>> > On Mon, Sep 10, 2018 at 4:58 AM Jean-Baptiste Onofré
>>>>>>> > <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
>>>>>>> >
>>>>>>> > +1 (binding)
>>>>>>> >
>>>>>>> > Tested successfully on Beam Samples.
>>>>>>> >
>>>>>>> > Thanks !
>>>>>>> >
>>>>>>> > Regards
>>>>>>> > JB
>>>>>>> >
>>>>>>> > On 07/09/2018 23:56, Charles Chen wrote:
>>>>>>> > > Hi everyone,
>>>>>>> > >
>>>>>>> > > Please review and vote on the release candidate #1 for
>>>>>>> the
>>>>>>> > version
>>>>>>> > > 2.7.0, as follows:
>>>>>>> > > [ ] +1, Approve the release
>>>>>>> > > [ ] -1, Do not approve the release (please provide
>>>>>>> specific
>>>>>>> > comments)
>>>>>>> > >
>>>>>>> > > The complete staging area is available for your review,
>>>>>>> which
>>>>>>> > includes:
>>>>>>> > > * JIRA release notes [1],
>>>>>>> > > * the official Apache source release to be deployed to
>>>>>>> > dist.apache.org <http://dist.apache.org>
>>>>>>> > > <http://dist.apache.org> [2], which is signed with the
>>>>>>> key with
>>>>>>> > > fingerprint 45C60AAAD115F560 [3],
>>>>>>> > > * all artifacts to be deployed to the Maven Central
>>>>>>> > Repository [4],
>>>>>>> > > * source code tag "v2.7.0-RC1" [5],
>>>>>>> > > * website pull request listing the release and
>>>>>>> publishing the API
>>>>>>> > > reference manual [6].
>>>>>>> > > * Java artifacts were built with Gradle 4.8 and OpenJDK
>>>>>>> > > 1.8.0_181-8u181-b13-1~deb9u1-b13.
>>>>>>> > > * Python artifacts are deployed along with the source
>>>>>>> release
>>>>>>> > to the
>>>>>>> > > dist.apache.org <http://dist.apache.org>
>>>>>>> > <http://dist.apache.org> [2].
>>>>>>> > >
>>>>>>> > > The vote will be open for at least 72 hours. It is
>>>>>>> adopted by
>>>>>>> > majority
>>>>>>> > > approval, with at least 3 PMC affirmative votes.
>>>>>>> > >
>>>>>>> > > Thanks,
>>>>>>> > > Charles
>>>>>>> > >
>>>>>>> > > [1]
>>>>>>> > >
>>>>>>> >
>>>>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>>>>>>> > > [2] https://dist.apache.org/repos/dist/dev/beam/2.7.0
>>>>>>> > > [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
>>>>>>> > > [4]
>>>>>>> >
>>>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1046/
>>>>>>> > > [5] https://github.com/apache/beam/tree/v2.7.0-RC1
>>>>>>> > > [6] https://github.com/apache/beam-site/pull/549
>>>>>>> >
>>>>>>> > --
>>>>>>> > Jean-Baptiste Onofré
>>>>>>> > jbonofre@apache.org <ma...@apache.org>
>>>>>>> > http://blog.nanthrax.net
>>>>>>> > Talend - http://www.talend.com
>>>>>>> >
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Charles Chen <cc...@google.com>.
Can you send the specific command / config you are using to reproduce?
On Mon, Sep 17, 2018 at 10:28 AM Romain Manni-Bucau <rm...@gmail.com>
wrote:
> @Charles: guess you can download
> https://github.com/Talend/component-runtime/tree/master/component-runtime-beam/src/it/serialization-over-cluster
> subproject, replace project.version by 1.0.4 (other placeholders are on the
> net/more known) and you should be able to reproduce forcing beam to be in
> v2.7.0. I dont have much time this week to check out this particular issue
> but hopefully next one should be more doable if the issue is still pending.
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github
> <https://github.com/rmannibucau> | LinkedIn
> <https://www.linkedin.com/in/rmannibucau> | Book
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>
>
> Le lun. 17 sept. 2018 à 19:18, Charles Chen <cc...@google.com> a écrit :
>
>> Luke, Maximillian, Raghu, can you please propose cherry-pick PRs to the
>> release-2.7.0 for your issues and add me as a reviewer (@charlesccychen)?
>>
>> Romain, JB: is there any way I can help with debugging the issue you're
>> facing so we can unblock the release?
>>
>> On Fri, Sep 14, 2018 at 1:49 PM Raghu Angadi <ra...@google.com> wrote:
>>
>>> I would like propose one more cherrypick for RC2 :
>>> https://github.com/apache/beam/pull/6391
>>> This is a KafkaIO bug fix. Once a user hits this bug, there is no easy
>>> work around for them, especially on Dataflow. Only work around in Dataflow
>>> is to restart or reload the job.
>>>
>>> The fix itself fairly safe and is tested.
>>> Raghu.
>>>
>>> On Fri, Sep 14, 2018 at 12:52 AM Alexey Romanenko <
>>> aromanenko.dev@gmail.com> wrote:
>>>
>>>> Perhaps it could help, but I run simple WordCount (built with Beam 2.7)
>>>> on YARN/Spark (HDP Sandbox) cluster and it worked fine for me.
>>>>
>>>> On 14 Sep 2018, at 06:56, Romain Manni-Bucau <rm...@gmail.com>
>>>> wrote:
>>>>
>>>> Hi Charles,
>>>>
>>>> I didn't get enough time to check deeply but it is clearly a dependency
>>>> issue and it is not in beam spark runner itself but in another transitive
>>>> module of beam. It does not happen in existing spark test cause none of
>>>> them are in a cluster (even just with 1 worker) but this seems to be a
>>>> regression since 2.6 works OOTB.
>>>>
>>>> Romain Manni-Bucau
>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>> <http://rmannibucau.wordpress.com/> | Github
>>>> <https://github.com/rmannibucau> | LinkedIn
>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>
>>>>
>>>> Le jeu. 13 sept. 2018 à 22:15, Charles Chen <cc...@google.com> a écrit :
>>>>
>>>>> Romain and JB, can you please add the results of your investigations
>>>>> into the errors you've seen above? Given that the existing SparkRunner
>>>>> tests pass for this RC, and that the integration test you ran is in another
>>>>> repo that is not continuously tested with Beam, it is not clear how we
>>>>> should move forward and whether this is a blocking issue, unless we can
>>>>> find a root cause in Beam.
>>>>>
>>>>> On Wed, Sep 12, 2018 at 2:08 AM Etienne Chauchot <ec...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> on a performance and functional regression stand point I see no
>>>>>> regression:
>>>>>>
>>>>>> I looked at nexmark graphs "output pcollection size" and "execution
>>>>>> time" around release cut date on dataflow, spark, flink and direct runner
>>>>>> in batch and streaming modes. There seems to be no regression.
>>>>>>
>>>>>> Etienne
>>>>>>
>>>>>> Le mardi 11 septembre 2018 à 12:25 -0700, Charles Chen a écrit :
>>>>>>
>>>>>> The SparkRunner validation test (here:
>>>>>> https://beam.apache.org/contribute/release-guide/#run-validation-tests)
>>>>>> passes on my machine. It looks like we are likely missing test coverage
>>>>>> where Romain is hitting issues.
>>>>>>
>>>>>> On Tue, Sep 11, 2018 at 12:15 PM Ahmet Altay <al...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>> Could anyone else help with looking at these issues earlier?
>>>>>>
>>>>>> On Tue, Sep 11, 2018 at 12:03 PM, Romain Manni-Bucau <
>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>
>>>>>> Im running this main [1] through this IT [2]. Was working fine since
>>>>>> ~1 year but 2.7.0 broke it. Didnt investigate more but can have a look
>>>>>> later this month if it helps.
>>>>>>
>>>>>> [1]
>>>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
>>>>>> [2]
>>>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
>>>>>>
>>>>>> Le mar. 11 sept. 2018 20:54, Charles Chen <cc...@google.com> a écrit :
>>>>>>
>>>>>> Romain: can you give more details on the failure you're encountering,
>>>>>> i.e. how you are performing this validation?
>>>>>>
>>>>>> On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>>>>>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> weird, I didn't have it on Beam samples. Let me try to reproduce and
>>>>>> I
>>>>>> will create the Jira.
>>>>>>
>>>>>> Regards
>>>>>> JB
>>>>>>
>>>>>> On 11/09/2018 11:44, Romain Manni-Bucau wrote:
>>>>>> > -1, seems spark integration is broken (tested with spark 2.3.1 and
>>>>>> 2.2.1):
>>>>>> >
>>>>>> > 18/09/11 11:33:29 WARN TaskSetManager: Lost task 0.0 in stage 0.0
>>>>>> (TID 0, RMANNIBUCAU, executor 0): java.lang.ClassCastException: cannot
>>>>>> assign instance of scala.collection.immutable.List$SerializationProxy to
>>>>>> fieldorg.apache.spark.rdd.RDD.org
>>>>>> <http://fieldorg.apache.spark.rdd.rdd.org/> <
>>>>>> http://org.apache.spark.rdd.RDD.org
>>>>>> <http://org.apache.spark.rdd.rdd.org/>>$apache$spark$rdd$RDD$$dependencies_
>>>>>> of type scala.collection.Seq in instance of
>>>>>> org.apache.spark.rdd.MapPartitionsRDD
>>>>>> > at
>>>>>> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>>>>>> >
>>>>>> >
>>>>>> > Also the issue Lukasz identified is important even if workarounds
>>>>>> can be
>>>>>> > put in place so +1 to fix it as well if possible.
>>>>>> >
>>>>>> > Romain Manni-Bucau
>>>>>> > @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>>> > <https://rmannibucau.metawerx.net/> | Old Blog
>>>>>> > <http://rmannibucau.wordpress.com> | Github
>>>>>> > <https://github.com/rmannibucau> | LinkedIn
>>>>>> > <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>> > <
>>>>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik <lcwik@google.com
>>>>>> > <ma...@google.com>> a écrit :
>>>>>> >
>>>>>> > I found an issue where we are no longer packaging the pom.xml
>>>>>> within
>>>>>> > the artifact jars at META-INF/maven/groupId/artifactId. More
>>>>>> details
>>>>>> > in https://issues.apache.org/jira/browse/BEAM-5351. I wouldn't
>>>>>> > consider this a blocker but it was an easy fix
>>>>>> > (https://github.com/apache/beam/pull/6358) and users may rely
>>>>>> on the
>>>>>> > pom.xml.
>>>>>> >
>>>>>> > Should we recut the release candidate to include this?
>>>>>> >
>>>>>> > On Mon, Sep 10, 2018 at 4:58 AM Jean-Baptiste Onofré
>>>>>> > <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
>>>>>> >
>>>>>> > +1 (binding)
>>>>>> >
>>>>>> > Tested successfully on Beam Samples.
>>>>>> >
>>>>>> > Thanks !
>>>>>> >
>>>>>> > Regards
>>>>>> > JB
>>>>>> >
>>>>>> > On 07/09/2018 23:56, Charles Chen wrote:
>>>>>> > > Hi everyone,
>>>>>> > >
>>>>>> > > Please review and vote on the release candidate #1 for
>>>>>> the
>>>>>> > version
>>>>>> > > 2.7.0, as follows:
>>>>>> > > [ ] +1, Approve the release
>>>>>> > > [ ] -1, Do not approve the release (please provide
>>>>>> specific
>>>>>> > comments)
>>>>>> > >
>>>>>> > > The complete staging area is available for your review,
>>>>>> which
>>>>>> > includes:
>>>>>> > > * JIRA release notes [1],
>>>>>> > > * the official Apache source release to be deployed to
>>>>>> > dist.apache.org <http://dist.apache.org>
>>>>>> > > <http://dist.apache.org> [2], which is signed with the
>>>>>> key with
>>>>>> > > fingerprint 45C60AAAD115F560 [3],
>>>>>> > > * all artifacts to be deployed to the Maven Central
>>>>>> > Repository [4],
>>>>>> > > * source code tag "v2.7.0-RC1" [5],
>>>>>> > > * website pull request listing the release and
>>>>>> publishing the API
>>>>>> > > reference manual [6].
>>>>>> > > * Java artifacts were built with Gradle 4.8 and OpenJDK
>>>>>> > > 1.8.0_181-8u181-b13-1~deb9u1-b13.
>>>>>> > > * Python artifacts are deployed along with the source
>>>>>> release
>>>>>> > to the
>>>>>> > > dist.apache.org <http://dist.apache.org>
>>>>>> > <http://dist.apache.org> [2].
>>>>>> > >
>>>>>> > > The vote will be open for at least 72 hours. It is
>>>>>> adopted by
>>>>>> > majority
>>>>>> > > approval, with at least 3 PMC affirmative votes.
>>>>>> > >
>>>>>> > > Thanks,
>>>>>> > > Charles
>>>>>> > >
>>>>>> > > [1]
>>>>>> > >
>>>>>> >
>>>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>>>>>> > > [2] https://dist.apache.org/repos/dist/dev/beam/2.7.0
>>>>>> > > [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
>>>>>> > > [4]
>>>>>> >
>>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1046/
>>>>>> > > [5] https://github.com/apache/beam/tree/v2.7.0-RC1
>>>>>> > > [6] https://github.com/apache/beam-site/pull/549
>>>>>> >
>>>>>> > --
>>>>>> > Jean-Baptiste Onofré
>>>>>> > jbonofre@apache.org <ma...@apache.org>
>>>>>> > http://blog.nanthrax.net
>>>>>> > Talend - http://www.talend.com
>>>>>> >
>>>>>>
>>>>>>
>>>>>>
>>>>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Romain Manni-Bucau <rm...@gmail.com>.
@Charles: guess you can download
https://github.com/Talend/component-runtime/tree/master/component-runtime-beam/src/it/serialization-over-cluster
subproject, replace project.version by 1.0.4 (other placeholders are on the
net/more known) and you should be able to reproduce forcing beam to be in
v2.7.0. I dont have much time this week to check out this particular issue
but hopefully next one should be more doable if the issue is still pending.
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>
Le lun. 17 sept. 2018 à 19:18, Charles Chen <cc...@google.com> a écrit :
> Luke, Maximillian, Raghu, can you please propose cherry-pick PRs to the
> release-2.7.0 for your issues and add me as a reviewer (@charlesccychen)?
>
> Romain, JB: is there any way I can help with debugging the issue you're
> facing so we can unblock the release?
>
> On Fri, Sep 14, 2018 at 1:49 PM Raghu Angadi <ra...@google.com> wrote:
>
>> I would like propose one more cherrypick for RC2 :
>> https://github.com/apache/beam/pull/6391
>> This is a KafkaIO bug fix. Once a user hits this bug, there is no easy
>> work around for them, especially on Dataflow. Only work around in Dataflow
>> is to restart or reload the job.
>>
>> The fix itself fairly safe and is tested.
>> Raghu.
>>
>> On Fri, Sep 14, 2018 at 12:52 AM Alexey Romanenko <
>> aromanenko.dev@gmail.com> wrote:
>>
>>> Perhaps it could help, but I run simple WordCount (built with Beam 2.7)
>>> on YARN/Spark (HDP Sandbox) cluster and it worked fine for me.
>>>
>>> On 14 Sep 2018, at 06:56, Romain Manni-Bucau <rm...@gmail.com>
>>> wrote:
>>>
>>> Hi Charles,
>>>
>>> I didn't get enough time to check deeply but it is clearly a dependency
>>> issue and it is not in beam spark runner itself but in another transitive
>>> module of beam. It does not happen in existing spark test cause none of
>>> them are in a cluster (even just with 1 worker) but this seems to be a
>>> regression since 2.6 works OOTB.
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>> <http://rmannibucau.wordpress.com/> | Github
>>> <https://github.com/rmannibucau> | LinkedIn
>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>
>>>
>>> Le jeu. 13 sept. 2018 à 22:15, Charles Chen <cc...@google.com> a écrit :
>>>
>>>> Romain and JB, can you please add the results of your investigations
>>>> into the errors you've seen above? Given that the existing SparkRunner
>>>> tests pass for this RC, and that the integration test you ran is in another
>>>> repo that is not continuously tested with Beam, it is not clear how we
>>>> should move forward and whether this is a blocking issue, unless we can
>>>> find a root cause in Beam.
>>>>
>>>> On Wed, Sep 12, 2018 at 2:08 AM Etienne Chauchot <ec...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> on a performance and functional regression stand point I see no
>>>>> regression:
>>>>>
>>>>> I looked at nexmark graphs "output pcollection size" and "execution
>>>>> time" around release cut date on dataflow, spark, flink and direct runner
>>>>> in batch and streaming modes. There seems to be no regression.
>>>>>
>>>>> Etienne
>>>>>
>>>>> Le mardi 11 septembre 2018 à 12:25 -0700, Charles Chen a écrit :
>>>>>
>>>>> The SparkRunner validation test (here:
>>>>> https://beam.apache.org/contribute/release-guide/#run-validation-tests)
>>>>> passes on my machine. It looks like we are likely missing test coverage
>>>>> where Romain is hitting issues.
>>>>>
>>>>> On Tue, Sep 11, 2018 at 12:15 PM Ahmet Altay <al...@google.com> wrote:
>>>>>
>>>>> Could anyone else help with looking at these issues earlier?
>>>>>
>>>>> On Tue, Sep 11, 2018 at 12:03 PM, Romain Manni-Bucau <
>>>>> rmannibucau@gmail.com> wrote:
>>>>>
>>>>> Im running this main [1] through this IT [2]. Was working fine since
>>>>> ~1 year but 2.7.0 broke it. Didnt investigate more but can have a look
>>>>> later this month if it helps.
>>>>>
>>>>> [1]
>>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
>>>>> [2]
>>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
>>>>>
>>>>> Le mar. 11 sept. 2018 20:54, Charles Chen <cc...@google.com> a écrit :
>>>>>
>>>>> Romain: can you give more details on the failure you're encountering,
>>>>> i.e. how you are performing this validation?
>>>>>
>>>>> On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> weird, I didn't have it on Beam samples. Let me try to reproduce and I
>>>>> will create the Jira.
>>>>>
>>>>> Regards
>>>>> JB
>>>>>
>>>>> On 11/09/2018 11:44, Romain Manni-Bucau wrote:
>>>>> > -1, seems spark integration is broken (tested with spark 2.3.1 and
>>>>> 2.2.1):
>>>>> >
>>>>> > 18/09/11 11:33:29 WARN TaskSetManager: Lost task 0.0 in stage 0.0
>>>>> (TID 0, RMANNIBUCAU, executor 0): java.lang.ClassCastException: cannot
>>>>> assign instance of scala.collection.immutable.List$SerializationProxy to
>>>>> fieldorg.apache.spark.rdd.RDD.org
>>>>> <http://fieldorg.apache.spark.rdd.rdd.org/> <
>>>>> http://org.apache.spark.rdd.RDD.org
>>>>> <http://org.apache.spark.rdd.rdd.org/>>$apache$spark$rdd$RDD$$dependencies_
>>>>> of type scala.collection.Seq in instance of
>>>>> org.apache.spark.rdd.MapPartitionsRDD
>>>>> > at
>>>>> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>>>>> >
>>>>> >
>>>>> > Also the issue Lukasz identified is important even if workarounds
>>>>> can be
>>>>> > put in place so +1 to fix it as well if possible.
>>>>> >
>>>>> > Romain Manni-Bucau
>>>>> > @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>> > <https://rmannibucau.metawerx.net/> | Old Blog
>>>>> > <http://rmannibucau.wordpress.com> | Github
>>>>> > <https://github.com/rmannibucau> | LinkedIn
>>>>> > <https://www.linkedin.com/in/rmannibucau> | Book
>>>>> > <
>>>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>>> >
>>>>> >
>>>>> >
>>>>> > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik <lcwik@google.com
>>>>> > <ma...@google.com>> a écrit :
>>>>> >
>>>>> > I found an issue where we are no longer packaging the pom.xml
>>>>> within
>>>>> > the artifact jars at META-INF/maven/groupId/artifactId. More
>>>>> details
>>>>> > in https://issues.apache.org/jira/browse/BEAM-5351. I wouldn't
>>>>> > consider this a blocker but it was an easy fix
>>>>> > (https://github.com/apache/beam/pull/6358) and users may rely
>>>>> on the
>>>>> > pom.xml.
>>>>> >
>>>>> > Should we recut the release candidate to include this?
>>>>> >
>>>>> > On Mon, Sep 10, 2018 at 4:58 AM Jean-Baptiste Onofré
>>>>> > <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
>>>>> >
>>>>> > +1 (binding)
>>>>> >
>>>>> > Tested successfully on Beam Samples.
>>>>> >
>>>>> > Thanks !
>>>>> >
>>>>> > Regards
>>>>> > JB
>>>>> >
>>>>> > On 07/09/2018 23:56, Charles Chen wrote:
>>>>> > > Hi everyone,
>>>>> > >
>>>>> > > Please review and vote on the release candidate #1 for the
>>>>> > version
>>>>> > > 2.7.0, as follows:
>>>>> > > [ ] +1, Approve the release
>>>>> > > [ ] -1, Do not approve the release (please provide
>>>>> specific
>>>>> > comments)
>>>>> > >
>>>>> > > The complete staging area is available for your review,
>>>>> which
>>>>> > includes:
>>>>> > > * JIRA release notes [1],
>>>>> > > * the official Apache source release to be deployed to
>>>>> > dist.apache.org <http://dist.apache.org>
>>>>> > > <http://dist.apache.org> [2], which is signed with the
>>>>> key with
>>>>> > > fingerprint 45C60AAAD115F560 [3],
>>>>> > > * all artifacts to be deployed to the Maven Central
>>>>> > Repository [4],
>>>>> > > * source code tag "v2.7.0-RC1" [5],
>>>>> > > * website pull request listing the release and publishing
>>>>> the API
>>>>> > > reference manual [6].
>>>>> > > * Java artifacts were built with Gradle 4.8 and OpenJDK
>>>>> > > 1.8.0_181-8u181-b13-1~deb9u1-b13.
>>>>> > > * Python artifacts are deployed along with the source
>>>>> release
>>>>> > to the
>>>>> > > dist.apache.org <http://dist.apache.org>
>>>>> > <http://dist.apache.org> [2].
>>>>> > >
>>>>> > > The vote will be open for at least 72 hours. It is
>>>>> adopted by
>>>>> > majority
>>>>> > > approval, with at least 3 PMC affirmative votes.
>>>>> > >
>>>>> > > Thanks,
>>>>> > > Charles
>>>>> > >
>>>>> > > [1]
>>>>> > >
>>>>> >
>>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>>>>> > > [2] https://dist.apache.org/repos/dist/dev/beam/2.7.0
>>>>> > > [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
>>>>> > > [4]
>>>>> >
>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1046/
>>>>> > > [5] https://github.com/apache/beam/tree/v2.7.0-RC1
>>>>> > > [6] https://github.com/apache/beam-site/pull/549
>>>>> >
>>>>> > --
>>>>> > Jean-Baptiste Onofré
>>>>> > jbonofre@apache.org <ma...@apache.org>
>>>>> > http://blog.nanthrax.net
>>>>> > Talend - http://www.talend.com
>>>>> >
>>>>>
>>>>>
>>>>>
>>>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Raghu Angadi <ra...@google.com>.
Thanks Charles. Sent cherry-pick for KafkaIO fix:
https://github.com/apache/beam/pull/6421
On Mon, Sep 17, 2018 at 10:18 AM Charles Chen <cc...@google.com> wrote:
> Luke, Maximillian, Raghu, can you please propose cherry-pick PRs to the
> release-2.7.0 for your issues and add me as a reviewer (@charlesccychen)?
>
> Romain, JB: is there any way I can help with debugging the issue you're
> facing so we can unblock the release?
>
> On Fri, Sep 14, 2018 at 1:49 PM Raghu Angadi <ra...@google.com> wrote:
>
>> I would like propose one more cherrypick for RC2 :
>> https://github.com/apache/beam/pull/6391
>> This is a KafkaIO bug fix. Once a user hits this bug, there is no easy
>> work around for them, especially on Dataflow. Only work around in Dataflow
>> is to restart or reload the job.
>>
>> The fix itself fairly safe and is tested.
>> Raghu.
>>
>> On Fri, Sep 14, 2018 at 12:52 AM Alexey Romanenko <
>> aromanenko.dev@gmail.com> wrote:
>>
>>> Perhaps it could help, but I run simple WordCount (built with Beam 2.7)
>>> on YARN/Spark (HDP Sandbox) cluster and it worked fine for me.
>>>
>>> On 14 Sep 2018, at 06:56, Romain Manni-Bucau <rm...@gmail.com>
>>> wrote:
>>>
>>> Hi Charles,
>>>
>>> I didn't get enough time to check deeply but it is clearly a dependency
>>> issue and it is not in beam spark runner itself but in another transitive
>>> module of beam. It does not happen in existing spark test cause none of
>>> them are in a cluster (even just with 1 worker) but this seems to be a
>>> regression since 2.6 works OOTB.
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>> <http://rmannibucau.wordpress.com/> | Github
>>> <https://github.com/rmannibucau> | LinkedIn
>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>
>>>
>>> Le jeu. 13 sept. 2018 à 22:15, Charles Chen <cc...@google.com> a écrit :
>>>
>>>> Romain and JB, can you please add the results of your investigations
>>>> into the errors you've seen above? Given that the existing SparkRunner
>>>> tests pass for this RC, and that the integration test you ran is in another
>>>> repo that is not continuously tested with Beam, it is not clear how we
>>>> should move forward and whether this is a blocking issue, unless we can
>>>> find a root cause in Beam.
>>>>
>>>> On Wed, Sep 12, 2018 at 2:08 AM Etienne Chauchot <ec...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> on a performance and functional regression stand point I see no
>>>>> regression:
>>>>>
>>>>> I looked at nexmark graphs "output pcollection size" and "execution
>>>>> time" around release cut date on dataflow, spark, flink and direct runner
>>>>> in batch and streaming modes. There seems to be no regression.
>>>>>
>>>>> Etienne
>>>>>
>>>>> Le mardi 11 septembre 2018 à 12:25 -0700, Charles Chen a écrit :
>>>>>
>>>>> The SparkRunner validation test (here:
>>>>> https://beam.apache.org/contribute/release-guide/#run-validation-tests)
>>>>> passes on my machine. It looks like we are likely missing test coverage
>>>>> where Romain is hitting issues.
>>>>>
>>>>> On Tue, Sep 11, 2018 at 12:15 PM Ahmet Altay <al...@google.com> wrote:
>>>>>
>>>>> Could anyone else help with looking at these issues earlier?
>>>>>
>>>>> On Tue, Sep 11, 2018 at 12:03 PM, Romain Manni-Bucau <
>>>>> rmannibucau@gmail.com> wrote:
>>>>>
>>>>> Im running this main [1] through this IT [2]. Was working fine since
>>>>> ~1 year but 2.7.0 broke it. Didnt investigate more but can have a look
>>>>> later this month if it helps.
>>>>>
>>>>> [1]
>>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
>>>>> [2]
>>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
>>>>>
>>>>> Le mar. 11 sept. 2018 20:54, Charles Chen <cc...@google.com> a écrit :
>>>>>
>>>>> Romain: can you give more details on the failure you're encountering,
>>>>> i.e. how you are performing this validation?
>>>>>
>>>>> On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> weird, I didn't have it on Beam samples. Let me try to reproduce and I
>>>>> will create the Jira.
>>>>>
>>>>> Regards
>>>>> JB
>>>>>
>>>>> On 11/09/2018 11:44, Romain Manni-Bucau wrote:
>>>>> > -1, seems spark integration is broken (tested with spark 2.3.1 and
>>>>> 2.2.1):
>>>>> >
>>>>> > 18/09/11 11:33:29 WARN TaskSetManager: Lost task 0.0 in stage 0.0
>>>>> (TID 0, RMANNIBUCAU, executor 0): java.lang.ClassCastException: cannot
>>>>> assign instance of scala.collection.immutable.List$SerializationProxy to
>>>>> fieldorg.apache.spark.rdd.RDD.org
>>>>> <http://fieldorg.apache.spark.rdd.rdd.org/> <
>>>>> http://org.apache.spark.rdd.RDD.org
>>>>> <http://org.apache.spark.rdd.rdd.org/>>$apache$spark$rdd$RDD$$dependencies_
>>>>> of type scala.collection.Seq in instance of
>>>>> org.apache.spark.rdd.MapPartitionsRDD
>>>>> > at
>>>>> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>>>>> >
>>>>> >
>>>>> > Also the issue Lukasz identified is important even if workarounds
>>>>> can be
>>>>> > put in place so +1 to fix it as well if possible.
>>>>> >
>>>>> > Romain Manni-Bucau
>>>>> > @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>> > <https://rmannibucau.metawerx.net/> | Old Blog
>>>>> > <http://rmannibucau.wordpress.com> | Github
>>>>> > <https://github.com/rmannibucau> | LinkedIn
>>>>> > <https://www.linkedin.com/in/rmannibucau> | Book
>>>>> > <
>>>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>>> >
>>>>> >
>>>>> >
>>>>> > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik <lcwik@google.com
>>>>> > <ma...@google.com>> a écrit :
>>>>> >
>>>>> > I found an issue where we are no longer packaging the pom.xml
>>>>> within
>>>>> > the artifact jars at META-INF/maven/groupId/artifactId. More
>>>>> details
>>>>> > in https://issues.apache.org/jira/browse/BEAM-5351. I wouldn't
>>>>> > consider this a blocker but it was an easy fix
>>>>> > (https://github.com/apache/beam/pull/6358) and users may rely
>>>>> on the
>>>>> > pom.xml.
>>>>> >
>>>>> > Should we recut the release candidate to include this?
>>>>> >
>>>>> > On Mon, Sep 10, 2018 at 4:58 AM Jean-Baptiste Onofré
>>>>> > <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
>>>>> >
>>>>> > +1 (binding)
>>>>> >
>>>>> > Tested successfully on Beam Samples.
>>>>> >
>>>>> > Thanks !
>>>>> >
>>>>> > Regards
>>>>> > JB
>>>>> >
>>>>> > On 07/09/2018 23:56, Charles Chen wrote:
>>>>> > > Hi everyone,
>>>>> > >
>>>>> > > Please review and vote on the release candidate #1 for the
>>>>> > version
>>>>> > > 2.7.0, as follows:
>>>>> > > [ ] +1, Approve the release
>>>>> > > [ ] -1, Do not approve the release (please provide
>>>>> specific
>>>>> > comments)
>>>>> > >
>>>>> > > The complete staging area is available for your review,
>>>>> which
>>>>> > includes:
>>>>> > > * JIRA release notes [1],
>>>>> > > * the official Apache source release to be deployed to
>>>>> > dist.apache.org <http://dist.apache.org>
>>>>> > > <http://dist.apache.org> [2], which is signed with the
>>>>> key with
>>>>> > > fingerprint 45C60AAAD115F560 [3],
>>>>> > > * all artifacts to be deployed to the Maven Central
>>>>> > Repository [4],
>>>>> > > * source code tag "v2.7.0-RC1" [5],
>>>>> > > * website pull request listing the release and publishing
>>>>> the API
>>>>> > > reference manual [6].
>>>>> > > * Java artifacts were built with Gradle 4.8 and OpenJDK
>>>>> > > 1.8.0_181-8u181-b13-1~deb9u1-b13.
>>>>> > > * Python artifacts are deployed along with the source
>>>>> release
>>>>> > to the
>>>>> > > dist.apache.org <http://dist.apache.org>
>>>>> > <http://dist.apache.org> [2].
>>>>> > >
>>>>> > > The vote will be open for at least 72 hours. It is
>>>>> adopted by
>>>>> > majority
>>>>> > > approval, with at least 3 PMC affirmative votes.
>>>>> > >
>>>>> > > Thanks,
>>>>> > > Charles
>>>>> > >
>>>>> > > [1]
>>>>> > >
>>>>> >
>>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>>>>> > > [2] https://dist.apache.org/repos/dist/dev/beam/2.7.0
>>>>> > > [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
>>>>> > > [4]
>>>>> >
>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1046/
>>>>> > > [5] https://github.com/apache/beam/tree/v2.7.0-RC1
>>>>> > > [6] https://github.com/apache/beam-site/pull/549
>>>>> >
>>>>> > --
>>>>> > Jean-Baptiste Onofré
>>>>> > jbonofre@apache.org <ma...@apache.org>
>>>>> > http://blog.nanthrax.net
>>>>> > Talend - http://www.talend.com
>>>>> >
>>>>>
>>>>>
>>>>>
>>>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Nithin Sujir <ns...@google.com>.
Hi,
Do we have an ETA on the 2.7.0 release?
Thanks,
Nithin.
On Fri, Sep 21, 2018 at 12:10 AM Romain Manni-Bucau <rm...@gmail.com>
wrote:
> Hi Charles,
>
> I didnt get a chance to work more on it but the sample shows that just
> changing beam version breaks existing code.
>
> Since Beam does not manage its dependency compatibility with runners -
> understand this as "it never managed this issue" - I guess you can proceed
> with 2.7 ignoring this breaking change. For > 2.7.0 versions, testing with
> the officially supported engine versions (kind of matrix compatibility) can
> be required with some advanced apps (with sides, unions, etc... maybe
> nextmark can be a start?).
>
> My blind guess is that 2.6 was compiled with spark 2.2.1 and 2.7 with
> spark 2.3.1 and therefore the method can have changed even if signature
> didn't (thanks scala and java method lookup which uses returned types vs
> signatures ignoring them).
> An interesting test would be to compile Beam 2.7.0 with spark 2.2.1 and
> run it with my project, I guess it would work.
>
> Side note: during my tests i realized that if you use avro 1.8 new API it
> fails in spark since only spark master was upgraded to avro 1.7 and not
> earlier versions so beam providing avro 1.8 is another issue.
>
> Anyway, fine to proceed on my side even if there is a "user regression",
> nobody being available to identify it would mean delaying the release of
> too much and beam is far to be only spark runner so no reason to block
> others ;).
>
> Le ven. 21 sept. 2018 03:32, Ahmet Altay <al...@google.com> a écrit :
>
>> Good point. However, we agreed that our release policy would be to patch
>> only long term support (LTS) releases. Given that we have not made any LTS
>> releases yet, perhaps we should use 2.8.0 as the opportunity to make our
>> first LTS release.
>>
>> On Thu, Sep 20, 2018 at 6:26 PM, Thomas Weise <th...@apache.org> wrote:
>>
>>> That's not the same for a user though. 2.7.1 would be a patch compatible
>>> release that only fixes bugs. 2.8.0 adds new features and potentially also
>>> new issues..
>>>
>>> On Thu, Sep 20, 2018 at 3:16 PM Ahmet Altay <al...@google.com> wrote:
>>>
>>>> +1 to Thomas's suggestion. Instead of 2.7.1 we can follow up with 2.8.0
>>>> though. 2.8.0 has a release branch cut date of 10/10 according to our
>>>> release calendar.
>>>>
>>>> On Thu, Sep 20, 2018 at 2:47 PM, Connell O'Callaghan <
>>>> connello@google.com> wrote:
>>>>
>>>>> +1 to Thomas's suggestion - if Charles or others cannot reproduce.
>>>>>
>>>>> On Thu, Sep 20, 2018 at 2:40 PM Thomas Weise <th...@apache.org> wrote:
>>>>>
>>>>>> We can also consider releasing 2.7.0 and then follow up with 2.7.1 if
>>>>>> the problem can be reproduced and requires a fix. Just food for thought :)
>>>>>>
>>>>>>
>>>>>> On Thu, Sep 20, 2018 at 2:13 PM Charles Chen <cc...@google.com> wrote:
>>>>>>
>>>>>>> My mistake, it looks like the correct beam staging repository (
>>>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1046/)
>>>>>>> is specified in your pom file.
>>>>>>>
>>>>>>> On Thu, Sep 20, 2018 at 2:10 PM Charles Chen <cc...@google.com> wrote:
>>>>>>>
>>>>>>>> Hey Romain and JB, do you have any progress on this? One thing I
>>>>>>>> would like to point out is that 2.7.0 isn't yet pushed to Maven Central, so
>>>>>>>> referring to it by version is not expected to work (and it looks like this
>>>>>>>> is what is done in your repo:
>>>>>>>> https://github.com/rmannibucau/beam-2.7.0-fails). Luke indicated
>>>>>>>> above that he doesn't see any dependency changes. Can you isolate and
>>>>>>>> reproduce this problem so that we can develop a fix, if necessary? I would
>>>>>>>> like to proceed with an RC2 as soon as possible.
>>>>>>>>
>>>>>>>> On Wed, Sep 19, 2018 at 6:37 AM Romain Manni-Bucau <
>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Quick update on the spark issue: I didn't get enough time to
>>>>>>>>> identify it clearly but managed to have a passing run of my test changing a
>>>>>>>>> bunch of versions.
>>>>>>>>> I suspect my code triggers some class conflicting between spark
>>>>>>>>> and my shade leading to a serialization issue. I didn't test
>>>>>>>>> userClassPathFirst option of spark but it can be an interesting thing to
>>>>>>>>> enable in beam runner.
>>>>>>>>> However it is still very confusing to have it not running just
>>>>>>>>> upgrading beam version and the spark error is very hard to understand.
>>>>>>>>>
>>>>>>>>> Romain Manni-Bucau
>>>>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>>>>>> <http://rmannibucau.wordpress.com> | Github
>>>>>>>>> <https://github.com/rmannibucau> | LinkedIn
>>>>>>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Le mar. 18 sept. 2018 à 20:17, Lukasz Cwik <lc...@google.com> a
>>>>>>>>> écrit :
>>>>>>>>>
>>>>>>>>>> Romain hinted that this was a dependency issue but when comparing
>>>>>>>>>> the two dependency trees I don't get much of a difference:
>>>>>>>>>>
>>>>>>>>>> lcwik@lcwik0: ~$ diff /tmp/260 /tmp/270
>>>>>>>>>> < [INFO] +- org.apache.beam:beam-runners-spark:jar:2.6.0:compile
>>>>>>>>>> < [INFO] | +-
>>>>>>>>>> org.apache.beam:beam-model-pipeline:jar:2.6.0:compile
>>>>>>>>>> ---
>>>>>>>>>> > [INFO] +- org.apache.beam:beam-runners-spark:jar:2.7.0:compile
>>>>>>>>>> > [INFO] | +-
>>>>>>>>>> org.apache.beam:beam-model-pipeline:jar:2.7.0:compile
>>>>>>>>>> 5c6
>>>>>>>>>> < [INFO] | +-
>>>>>>>>>> org.apache.beam:beam-sdks-java-core:jar:2.6.0:compile
>>>>>>>>>> ---
>>>>>>>>>> > [INFO] | +-
>>>>>>>>>> org.apache.beam:beam-sdks-java-core:jar:2.7.0:compile
>>>>>>>>>> 14,18c15,19
>>>>>>>>>> < [INFO] | | \- org.tukaani:xz:jar:1.5:compile
>>>>>>>>>> < [INFO] | +-
>>>>>>>>>> org.apache.beam:beam-runners-core-construction-java:jar:2.6.0:compile
>>>>>>>>>> < [INFO] | | \-
>>>>>>>>>> org.apache.beam:beam-model-job-management:jar:2.6.0:compile
>>>>>>>>>> < [INFO] | +-
>>>>>>>>>> org.apache.beam:beam-runners-core-java:jar:2.6.0:compile
>>>>>>>>>> < [INFO] | | \-
>>>>>>>>>> org.apache.beam:beam-model-fn-execution:jar:2.6.0:compile
>>>>>>>>>> ---
>>>>>>>>>> > [INFO] | | \- org.tukaani:xz:jar:1.8:compile
>>>>>>>>>> > [INFO] | +-
>>>>>>>>>> org.apache.beam:beam-runners-core-construction-java:jar:2.7.0:compile
>>>>>>>>>> > [INFO] | | \-
>>>>>>>>>> org.apache.beam:beam-model-job-management:jar:2.7.0:compile
>>>>>>>>>> > [INFO] | +-
>>>>>>>>>> org.apache.beam:beam-runners-core-java:jar:2.7.0:compile
>>>>>>>>>> > [INFO] | | \-
>>>>>>>>>> org.apache.beam:beam-model-fn-execution:jar:2.7.0:compile
>>>>>>>>>>
>>>>>>>>>> Other then Beam package changes, the only other change is xz
>>>>>>>>>> which I don't believe could be causing the issue.
>>>>>>>>>>
>>>>>>>>>> On Tue, Sep 18, 2018 at 8:38 AM Jean-Baptiste Onofré <
>>>>>>>>>> jb@nanthrax.net> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks, let me take a look.
>>>>>>>>>>>
>>>>>>>>>>> Regards
>>>>>>>>>>> JB
>>>>>>>>>>>
>>>>>>>>>>> On 18/09/2018 17:36, Romain Manni-Bucau wrote:
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > Le mar. 18 sept. 2018 à 16:44, Jean-Baptiste Onofré <
>>>>>>>>>>> jb@nanthrax.net
>>>>>>>>>>> > <ma...@nanthrax.net>> a écrit :
>>>>>>>>>>> >
>>>>>>>>>>> > Hi,
>>>>>>>>>>> >
>>>>>>>>>>> > I don't have the issue ;)
>>>>>>>>>>> >
>>>>>>>>>>> > As said in my vote, I tested 2.7.0 RC1 on beam-samples
>>>>>>>>>>> with Spark
>>>>>>>>>>> > without problem.
>>>>>>>>>>> >
>>>>>>>>>>> > I don't reproduce Romain issue as well.
>>>>>>>>>>> >
>>>>>>>>>>> > @Romain can you provide some details to reproduce the
>>>>>>>>>>> issue ?
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > Sure, you can use this
>>>>>>>>>>> > reproducer: https://github.com/rmannibucau/beam-2.7.0-fails
>>>>>>>>>>> > It shows that it suceeds on 2.6 and fails on 2.7.
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > Regards
>>>>>>>>>>> > JB
>>>>>>>>>>> >
>>>>>>>>>>> > On 17/09/2018 19:17, Charles Chen wrote:
>>>>>>>>>>> > > Luke, Maximillian, Raghu, can you please propose
>>>>>>>>>>> cherry-pick PRs
>>>>>>>>>>> > to the
>>>>>>>>>>> > > release-2.7.0 for your issues and add me as a reviewer
>>>>>>>>>>> > (@charlesccychen)?
>>>>>>>>>>> > >
>>>>>>>>>>> > > Romain, JB: is there any way I can help with debugging
>>>>>>>>>>> the issue
>>>>>>>>>>> > you're
>>>>>>>>>>> > > facing so we can unblock the release?
>>>>>>>>>>> > >
>>>>>>>>>>> > > On Fri, Sep 14, 2018 at 1:49 PM Raghu Angadi <
>>>>>>>>>>> rangadi@google.com
>>>>>>>>>>> > <ma...@google.com>
>>>>>>>>>>> > > <mailto:rangadi@google.com <ma...@google.com>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>> > >
>>>>>>>>>>> > > I would like propose one more cherrypick for RC2
>>>>>>>>>>> > > : https://github.com/apache/beam/pull/6391
>>>>>>>>>>> > > This is a KafkaIO bug fix. Once a user hits this
>>>>>>>>>>> bug, there is no
>>>>>>>>>>> > > easy work around for them, especially on Dataflow.
>>>>>>>>>>> Only work
>>>>>>>>>>> > around
>>>>>>>>>>> > > in Dataflow is to restart or reload the job.
>>>>>>>>>>> > >
>>>>>>>>>>> > > The fix itself fairly safe and is tested.
>>>>>>>>>>> > > Raghu.
>>>>>>>>>>> > >
>>>>>>>>>>> > > On Fri, Sep 14, 2018 at 12:52 AM Alexey Romanenko
>>>>>>>>>>> > > <aromanenko.dev@gmail.com <mailto:
>>>>>>>>>>> aromanenko.dev@gmail.com>
>>>>>>>>>>> > <mailto:aromanenko.dev@gmail.com <mailto:
>>>>>>>>>>> aromanenko.dev@gmail.com>>>
>>>>>>>>>>> > wrote:
>>>>>>>>>>> > >
>>>>>>>>>>> > > Perhaps it could help, but I run simple
>>>>>>>>>>> WordCount (built with
>>>>>>>>>>> > > Beam 2.7) on YARN/Spark (HDP Sandbox) cluster
>>>>>>>>>>> and it
>>>>>>>>>>> > worked fine
>>>>>>>>>>> > > for me.
>>>>>>>>>>> > >
>>>>>>>>>>> > >> On 14 Sep 2018, at 06:56, Romain Manni-Bucau
>>>>>>>>>>> > >> <rmannibucau@gmail.com <mailto:
>>>>>>>>>>> rmannibucau@gmail.com>
>>>>>>>>>>> > <mailto:rmannibucau@gmail.com <mailto:
>>>>>>>>>>> rmannibucau@gmail.com>>> wrote:
>>>>>>>>>>> > >>
>>>>>>>>>>> > >> Hi Charles,
>>>>>>>>>>> > >>
>>>>>>>>>>> > >> I didn't get enough time to check deeply but it
>>>>>>>>>>> is clearly a
>>>>>>>>>>> > >> dependency issue and it is not in beam spark
>>>>>>>>>>> runner
>>>>>>>>>>> > itself but
>>>>>>>>>>> > >> in another transitive module of beam. It does
>>>>>>>>>>> not happen in
>>>>>>>>>>> > >> existing spark test cause none of them are in a
>>>>>>>>>>> cluster (even
>>>>>>>>>>> > >> just with 1 worker) but this seems to be a
>>>>>>>>>>> regression since
>>>>>>>>>>> > >> 2.6 works OOTB.
>>>>>>>>>>> > >>
>>>>>>>>>>> > >> Romain Manni-Bucau
>>>>>>>>>>> > >> @rmannibucau <https://twitter.com/rmannibucau>
>>>>>>>>>>> | Blog
>>>>>>>>>>> > >> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>>>>>>>> > >> <http://rmannibucau.wordpress.com/> | Github
>>>>>>>>>>> > >> <https://github.com/rmannibucau> | LinkedIn
>>>>>>>>>>> > >> <https://www.linkedin.com/in/rmannibucau> |
>>>>>>>>>>> Book
>>>>>>>>>>> > >>
>>>>>>>>>>> > <
>>>>>>>>>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>>>>>>>>> >
>>>>>>>>>>> > >>
>>>>>>>>>>> > >>
>>>>>>>>>>> > >> Le jeu. 13 sept. 2018 à 22:15, Charles Chen
>>>>>>>>>>> > <ccy@google.com <ma...@google.com>
>>>>>>>>>>> > >> <mailto:ccy@google.com <ma...@google.com>>>
>>>>>>>>>>> a écrit :
>>>>>>>>>>> > >>
>>>>>>>>>>> > >> Romain and JB, can you please add the
>>>>>>>>>>> results of your
>>>>>>>>>>> > >> investigations into the errors you've seen
>>>>>>>>>>> above? Given
>>>>>>>>>>> > >> that the existing SparkRunner tests pass
>>>>>>>>>>> for this RC, and
>>>>>>>>>>> > >> that the integration test you ran is in
>>>>>>>>>>> another repo that
>>>>>>>>>>> > >> is not continuously tested with Beam, it is
>>>>>>>>>>> not clear how
>>>>>>>>>>> > >> we should move forward and whether this is
>>>>>>>>>>> a blocking
>>>>>>>>>>> > >> issue, unless we can find a root cause in
>>>>>>>>>>> Beam.
>>>>>>>>>>> > >>
>>>>>>>>>>> > >> On Wed, Sep 12, 2018 at 2:08 AM Etienne
>>>>>>>>>>> Chauchot
>>>>>>>>>>> > >> <echauchot@apache.org <mailto:
>>>>>>>>>>> echauchot@apache.org>
>>>>>>>>>>> > <mailto:echauchot@apache.org <ma...@apache.org>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>> > >>
>>>>>>>>>>> > >> Hi all,
>>>>>>>>>>> > >>
>>>>>>>>>>> > >> on a performance and functional
>>>>>>>>>>> regression stand
>>>>>>>>>>> > point
>>>>>>>>>>> > >> I see no regression:
>>>>>>>>>>> > >>
>>>>>>>>>>> > >> I looked at nexmark graphs "output
>>>>>>>>>>> pcollection size"
>>>>>>>>>>> > >> and "execution time" around release cut
>>>>>>>>>>> date on
>>>>>>>>>>> > >> dataflow, spark, flink and direct
>>>>>>>>>>> runner in batch and
>>>>>>>>>>> > >> streaming modes. There seems to be no
>>>>>>>>>>> regression.
>>>>>>>>>>> > >>
>>>>>>>>>>> > >> Etienne
>>>>>>>>>>> > >>
>>>>>>>>>>> > >> Le mardi 11 septembre 2018 à 12:25
>>>>>>>>>>> -0700, Charles
>>>>>>>>>>> > Chen
>>>>>>>>>>> > >> a écrit :
>>>>>>>>>>> > >>> The SparkRunner validation test
>>>>>>>>>>> > >>>
>>>>>>>>>>> > (here:
>>>>>>>>>>> https://beam.apache.org/contribute/release-guide/#run-validation-tests
>>>>>>>>>>> )
>>>>>>>>>>> > >>> passes on my machine. It looks like
>>>>>>>>>>> we are likely
>>>>>>>>>>> > >>> missing test coverage where Romain is
>>>>>>>>>>> hitting
>>>>>>>>>>> > issues.
>>>>>>>>>>> > >>>
>>>>>>>>>>> > >>> On Tue, Sep 11, 2018 at 12:15 PM Ahmet
>>>>>>>>>>> Altay
>>>>>>>>>>> > >>> <altay@google.com <mailto:
>>>>>>>>>>> altay@google.com>
>>>>>>>>>>> > <mailto:altay@google.com <ma...@google.com>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>> > >>>> Could anyone else help with looking
>>>>>>>>>>> at these issues
>>>>>>>>>>> > >>>> earlier?
>>>>>>>>>>> > >>>>
>>>>>>>>>>> > >>>> On Tue, Sep 11, 2018 at 12:03 PM,
>>>>>>>>>>> Romain
>>>>>>>>>>> > Manni-Bucau
>>>>>>>>>>> > >>>> <rmannibucau@gmail.com
>>>>>>>>>>> > <ma...@gmail.com>
>>>>>>>>>>> > >>>> <mailto:rmannibucau@gmail.com
>>>>>>>>>>> > <ma...@gmail.com>>> wrote:
>>>>>>>>>>> > >>>>> Im running this main [1] through
>>>>>>>>>>> this IT [2]. Was
>>>>>>>>>>> > >>>>> working fine since ~1 year but 2.7.0
>>>>>>>>>>> broke it.
>>>>>>>>>>> > >>>>> Didnt investigate more but can have
>>>>>>>>>>> a look later
>>>>>>>>>>> > >>>>> this month if it helps.
>>>>>>>>>>> > >>>>>
>>>>>>>>>>> > >>>>>
>>>>>>>>>>> > [1]
>>>>>>>>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
>>>>>>>>>>> > >>>>>
>>>>>>>>>>> > [2]
>>>>>>>>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
>>>>>>>>>>> > >>>>>
>>>>>>>>>>> > >>>>> Le mar. 11 sept. 2018 20:54, Charles
>>>>>>>>>>> Chen
>>>>>>>>>>> > >>>>> <ccy@google.com <mailto:
>>>>>>>>>>> ccy@google.com>
>>>>>>>>>>> > <mailto:ccy@google.com <ma...@google.com>>> a écrit :
>>>>>>>>>>> > >>>>>> Romain: can you give more details
>>>>>>>>>>> on the failure
>>>>>>>>>>> > >>>>>> you're encountering, i.e. how you
>>>>>>>>>>> are performing
>>>>>>>>>>> > >>>>>> this validation?
>>>>>>>>>>> > >>>>>>
>>>>>>>>>>> > >>>>>> On Tue, Sep 11, 2018 at 9:36 AM
>>>>>>>>>>> Jean-Baptiste
>>>>>>>>>>> > >>>>>> Onofré <jb@nanthrax.net
>>>>>>>>>>> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
>>>>>>>>>>> > <ma...@nanthrax.net>>>
>>>>>>>>>>> > >>>>>> wrote:
>>>>>>>>>>> > >>>>>>> Hi,
>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>> > >>>>>>> weird, I didn't have it on Beam
>>>>>>>>>>> samples. Let me
>>>>>>>>>>> > >>>>>>> try to reproduce and I
>>>>>>>>>>> > >>>>>>> will create the Jira.
>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>> > >>>>>>> Regards
>>>>>>>>>>> > >>>>>>> JB
>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>> > >>>>>>> On 11/09/2018 11:44, Romain
>>>>>>>>>>> Manni-Bucau wrote:
>>>>>>>>>>> > >>>>>>> > -1, seems spark integration is
>>>>>>>>>>> broken (tested
>>>>>>>>>>> > >>>>>>> with spark 2.3.1 and 2.2.1):
>>>>>>>>>>> > >>>>>>> >
>>>>>>>>>>> > >>>>>>> > 18/09/11 11:33:29 WARN
>>>>>>>>>>> TaskSetManager: Lost
>>>>>>>>>>> > >>>>>>> task 0.0 in stage 0.0 (TID 0,
>>>>>>>>>>> RMANNIBUCAU,
>>>>>>>>>>> > >>>>>>> executor 0):
>>>>>>>>>>> java.lang.ClassCastException:
>>>>>>>>>>> > cannot
>>>>>>>>>>> > >>>>>>> assign instance of
>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>> > scala.collection.immutable.List$SerializationProxy to
>>>>>>>>>>> > >>>>>>> fieldorg.apache.spark.rdd.RDD.org
>>>>>>>>>>> > <http://fieldorg.apache.spark.rdd.RDD.org>
>>>>>>>>>>> > >>>>>>> <
>>>>>>>>>>> http://fieldorg.apache.spark.rdd.rdd.org/>
>>>>>>>>>>> > >>>>>>> <
>>>>>>>>>>> http://org.apache.spark.rdd.RDD.org
>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>> > <http://org.apache.spark.rdd.rdd.org/
>>>>>>>>>>> >>$apache$spark$rdd$RDD$$dependencies_
>>>>>>>>>>> > >>>>>>> of type scala.collection.Seq in
>>>>>>>>>>> instance of
>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>> org.apache.spark.rdd.MapPartitionsRDD
>>>>>>>>>>> > >>>>>>> > at
>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>> >
>>>>>>>>>>> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>>>>>>>>>>> > >>>>>>> >
>>>>>>>>>>> > >>>>>>> >
>>>>>>>>>>> > >>>>>>> > Also the issue Lukasz identified
>>>>>>>>>>> is important
>>>>>>>>>>> > >>>>>>> even if workarounds can be
>>>>>>>>>>> > >>>>>>> > put in place so +1 to fix it as
>>>>>>>>>>> well if
>>>>>>>>>>> > possible.
>>>>>>>>>>> > >>>>>>> >
>>>>>>>>>>> > >>>>>>> > Romain Manni-Bucau
>>>>>>>>>>> > >>>>>>> > @rmannibucau <
>>>>>>>>>>> https://twitter.com/rmannibucau>
>>>>>>>>>>> > >>>>>>> | Blog
>>>>>>>>>>> > >>>>>>> > <
>>>>>>>>>>> https://rmannibucau.metawerx.net/> | Old Blog
>>>>>>>>>>> > >>>>>>> > <
>>>>>>>>>>> http://rmannibucau.wordpress.com
>>>>>>>>>>> > >>>>>>> <http://rmannibucau.wordpress.com/>>
>>>>>>>>>>> | Github
>>>>>>>>>>> > >>>>>>> > <https://github.com/rmannibucau> |
>>>>>>>>>>> LinkedIn
>>>>>>>>>>> > >>>>>>> >
>>>>>>>>>>> > <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>>>>>>> > >>>>>>> >
>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>> > <
>>>>>>>>>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>>>>>>>>> >
>>>>>>>>>>> > >>>>>>> >
>>>>>>>>>>> > >>>>>>> >
>>>>>>>>>>> > >>>>>>> > Le lun. 10 sept. 2018 à 20:48,
>>>>>>>>>>> Lukasz Cwik
>>>>>>>>>>> > >>>>>>> <lcwik@google.com <mailto:
>>>>>>>>>>> lcwik@google.com>
>>>>>>>>>>> > <mailto:lcwik@google.com <ma...@google.com>>
>>>>>>>>>>> > >>>>>>> > <mailto:lcwik@google.com
>>>>>>>>>>> > <ma...@google.com>
>>>>>>>>>>> > >>>>>>> <mailto:lcwik@google.com
>>>>>>>>>>> > <ma...@google.com>>>> a écrit :
>>>>>>>>>>> > >>>>>>> >
>>>>>>>>>>> > >>>>>>> > I found an issue where we
>>>>>>>>>>> are no longer
>>>>>>>>>>> > >>>>>>> packaging the pom.xml within
>>>>>>>>>>> > >>>>>>> > the artifact jars at
>>>>>>>>>>> > >>>>>>> META-INF/maven/groupId/artifactId.
>>>>>>>>>>> More details
>>>>>>>>>>> > >>>>>>> > in
>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-5351.
>>>>>>>>>>> > >>>>>>> I wouldn't
>>>>>>>>>>> > >>>>>>> > consider this a blocker but
>>>>>>>>>>> it was an
>>>>>>>>>>> > easy fix
>>>>>>>>>>> > >>>>>>> > (
>>>>>>>>>>> https://github.com/apache/beam/pull/6358)
>>>>>>>>>>> > >>>>>>> and users may rely on the
>>>>>>>>>>> > >>>>>>> > pom.xml.
>>>>>>>>>>> > >>>>>>> >
>>>>>>>>>>> > >>>>>>> > Should we recut the release
>>>>>>>>>>> candidate to
>>>>>>>>>>> > >>>>>>> include this?
>>>>>>>>>>> > >>>>>>> >
>>>>>>>>>>> > >>>>>>> > On Mon, Sep 10, 2018 at 4:58
>>>>>>>>>>> AM
>>>>>>>>>>> > >>>>>>> Jean-Baptiste Onofré
>>>>>>>>>>> > >>>>>>> > <jb@nanthrax.net
>>>>>>>>>>> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
>>>>>>>>>>> > <ma...@nanthrax.net>>
>>>>>>>>>>> > >>>>>>> <mailto:jb@nanthrax.net <mailto:
>>>>>>>>>>> jb@nanthrax.net>
>>>>>>>>>>> > >>>>>>> <mailto:jb@nanthrax.net
>>>>>>>>>>> > <ma...@nanthrax.net>>>> wrote:
>>>>>>>>>>> > >>>>>>> >
>>>>>>>>>>> > >>>>>>> > +1 (binding)
>>>>>>>>>>> > >>>>>>> >
>>>>>>>>>>> > >>>>>>> > Tested successfully on
>>>>>>>>>>> Beam Samples.
>>>>>>>>>>> > >>>>>>> >
>>>>>>>>>>> > >>>>>>> > Thanks !
>>>>>>>>>>> > >>>>>>> >
>>>>>>>>>>> > >>>>>>> > Regards
>>>>>>>>>>> > >>>>>>> > JB
>>>>>>>>>>> > >>>>>>> >
>>>>>>>>>>> > >>>>>>> > On 07/09/2018 23:56,
>>>>>>>>>>> Charles Chen
>>>>>>>>>>> > wrote:
>>>>>>>>>>> > >>>>>>> > > Hi everyone,
>>>>>>>>>>> > >>>>>>> > >
>>>>>>>>>>> > >>>>>>> > > Please review and
>>>>>>>>>>> vote on the
>>>>>>>>>>> > >>>>>>> release candidate #1 for the
>>>>>>>>>>> > >>>>>>> > version
>>>>>>>>>>> > >>>>>>> > > 2.7.0, as follows:
>>>>>>>>>>> > >>>>>>> > > [ ] +1, Approve the
>>>>>>>>>>> release
>>>>>>>>>>> > >>>>>>> > > [ ] -1, Do not
>>>>>>>>>>> approve the release
>>>>>>>>>>> > >>>>>>> (please provide specific
>>>>>>>>>>> > >>>>>>> > comments)
>>>>>>>>>>> > >>>>>>> > >
>>>>>>>>>>> > >>>>>>> > > The complete staging
>>>>>>>>>>> area is
>>>>>>>>>>> > >>>>>>> available for your review, which
>>>>>>>>>>> > >>>>>>> > includes:
>>>>>>>>>>> > >>>>>>> > > * JIRA release notes
>>>>>>>>>>> [1],
>>>>>>>>>>> > >>>>>>> > > * the official Apache
>>>>>>>>>>> source
>>>>>>>>>>> > release
>>>>>>>>>>> > >>>>>>> to be deployed to
>>>>>>>>>>> > >>>>>>> > dist.apache.org
>>>>>>>>>>> > <http://dist.apache.org>
>>>>>>>>>>> > >>>>>>> <http://dist.apache.org/>
>>>>>>>>>>> > <http://dist.apache.org
>>>>>>>>>>> > >>>>>>> <http://dist.apache.org/>>
>>>>>>>>>>> > >>>>>>> > > <
>>>>>>>>>>> http://dist.apache.org
>>>>>>>>>>> > >>>>>>> <http://dist.apache.org/>> [2],
>>>>>>>>>>> which is signed
>>>>>>>>>>> > >>>>>>> with the key with
>>>>>>>>>>> > >>>>>>> > > fingerprint
>>>>>>>>>>> 45C60AAAD115F560 [3],
>>>>>>>>>>> > >>>>>>> > > * all artifacts to be
>>>>>>>>>>> deployed to
>>>>>>>>>>> > >>>>>>> the Maven Central
>>>>>>>>>>> > >>>>>>> > Repository [4],
>>>>>>>>>>> > >>>>>>> > > * source code tag
>>>>>>>>>>> "v2.7.0-RC1" [5],
>>>>>>>>>>> > >>>>>>> > > * website pull
>>>>>>>>>>> request listing the
>>>>>>>>>>> > >>>>>>> release and publishing the API
>>>>>>>>>>> > >>>>>>> > > reference manual [6].
>>>>>>>>>>> > >>>>>>> > > * Java artifacts were
>>>>>>>>>>> built with
>>>>>>>>>>> > >>>>>>> Gradle 4.8 and OpenJDK
>>>>>>>>>>> > >>>>>>> > >
>>>>>>>>>>> 1.8.0_181-8u181-b13-1~deb9u1-b13.
>>>>>>>>>>> > >>>>>>> > > * Python artifacts
>>>>>>>>>>> are deployed
>>>>>>>>>>> > >>>>>>> along with the source release
>>>>>>>>>>> > >>>>>>> > to the
>>>>>>>>>>> > >>>>>>> > > dist.apache.org
>>>>>>>>>>> > <http://dist.apache.org>
>>>>>>>>>>> > >>>>>>> <http://dist.apache.org/>
>>>>>>>>>>> > <http://dist.apache.org
>>>>>>>>>>> > >>>>>>> <http://dist.apache.org/>>
>>>>>>>>>>> > >>>>>>> > <http://dist.apache.org
>>>>>>>>>>> > >>>>>>> <http://dist.apache.org/>> [2].
>>>>>>>>>>> > >>>>>>> > >
>>>>>>>>>>> > >>>>>>> > > The vote will be open
>>>>>>>>>>> for at least
>>>>>>>>>>> > >>>>>>> 72 hours. It is adopted by
>>>>>>>>>>> > >>>>>>> > majority
>>>>>>>>>>> > >>>>>>> > > approval, with at
>>>>>>>>>>> least 3 PMC
>>>>>>>>>>> > >>>>>>> affirmative votes.
>>>>>>>>>>> > >>>>>>> > >
>>>>>>>>>>> > >>>>>>> > > Thanks,
>>>>>>>>>>> > >>>>>>> > > Charles
>>>>>>>>>>> > >>>>>>> > >
>>>>>>>>>>> > >>>>>>> > > [1]
>>>>>>>>>>> > >>>>>>> > >
>>>>>>>>>>> > >>>>>>> >
>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>> >
>>>>>>>>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>>>>>>>>>>> > >>>>>>> > > [2]
>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>> > https://dist.apache.org/repos/dist/dev/beam/2.7.0
>>>>>>>>>>> > >>>>>>> > > [3]
>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/beam/KEYS
>>>>>>>>>>> > >>>>>>> > > [4]
>>>>>>>>>>> > >>>>>>> >
>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>> >
>>>>>>>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1046/
>>>>>>>>>>> > >>>>>>> > > [5]
>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>> https://github.com/apache/beam/tree/v2.7.0-RC1
>>>>>>>>>>> > >>>>>>> > > [6]
>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>> https://github.com/apache/beam-site/pull/549
>>>>>>>>>>> > >>>>>>> >
>>>>>>>>>>> > >>>>>>> > --
>>>>>>>>>>> > >>>>>>> > Jean-Baptiste Onofré
>>>>>>>>>>> > >>>>>>> > jbonofre@apache.org
>>>>>>>>>>> > <ma...@apache.org>
>>>>>>>>>>> > >>>>>>> <mailto:jbonofre@apache.org
>>>>>>>>>>> > <ma...@apache.org>>
>>>>>>>>>>> > >>>>>>> <mailto:jbonofre@apache.org
>>>>>>>>>>> > <ma...@apache.org>
>>>>>>>>>>> > >>>>>>> <mailto:jbonofre@apache.org
>>>>>>>>>>> > <ma...@apache.org>>>
>>>>>>>>>>> > >>>>>>> > http://blog.nanthrax.net
>>>>>>>>>>> > >>>>>>> <http://blog.nanthrax.net/>
>>>>>>>>>>> > >>>>>>> > Talend -
>>>>>>>>>>> http://www.talend.com
>>>>>>>>>>> > >>>>>>> <http://www.talend.com/>
>>>>>>>>>>> > >>>>>>> >
>>>>>>>>>>> > >>>>
>>>>>>>>>>> > >
>>>>>>>>>>> >
>>>>>>>>>>> > --
>>>>>>>>>>> > Jean-Baptiste Onofré
>>>>>>>>>>> > jbonofre@apache.org <ma...@apache.org>
>>>>>>>>>>> > http://blog.nanthrax.net
>>>>>>>>>>> > Talend - http://www.talend.com
>>>>>>>>>>> >
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Jean-Baptiste Onofré
>>>>>>>>>>> jbonofre@apache.org
>>>>>>>>>>> http://blog.nanthrax.net
>>>>>>>>>>> Talend - http://www.talend.com
>>>>>>>>>>>
>>>>>>>>>>
>>>>
>>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Hi Charles,
I didnt get a chance to work more on it but the sample shows that just
changing beam version breaks existing code.
Since Beam does not manage its dependency compatibility with runners -
understand this as "it never managed this issue" - I guess you can proceed
with 2.7 ignoring this breaking change. For > 2.7.0 versions, testing with
the officially supported engine versions (kind of matrix compatibility) can
be required with some advanced apps (with sides, unions, etc... maybe
nextmark can be a start?).
My blind guess is that 2.6 was compiled with spark 2.2.1 and 2.7 with spark
2.3.1 and therefore the method can have changed even if signature didn't
(thanks scala and java method lookup which uses returned types vs
signatures ignoring them).
An interesting test would be to compile Beam 2.7.0 with spark 2.2.1 and run
it with my project, I guess it would work.
Side note: during my tests i realized that if you use avro 1.8 new API it
fails in spark since only spark master was upgraded to avro 1.7 and not
earlier versions so beam providing avro 1.8 is another issue.
Anyway, fine to proceed on my side even if there is a "user regression",
nobody being available to identify it would mean delaying the release of
too much and beam is far to be only spark runner so no reason to block
others ;).
Le ven. 21 sept. 2018 03:32, Ahmet Altay <al...@google.com> a écrit :
> Good point. However, we agreed that our release policy would be to patch
> only long term support (LTS) releases. Given that we have not made any LTS
> releases yet, perhaps we should use 2.8.0 as the opportunity to make our
> first LTS release.
>
> On Thu, Sep 20, 2018 at 6:26 PM, Thomas Weise <th...@apache.org> wrote:
>
>> That's not the same for a user though. 2.7.1 would be a patch compatible
>> release that only fixes bugs. 2.8.0 adds new features and potentially also
>> new issues..
>>
>> On Thu, Sep 20, 2018 at 3:16 PM Ahmet Altay <al...@google.com> wrote:
>>
>>> +1 to Thomas's suggestion. Instead of 2.7.1 we can follow up with 2.8.0
>>> though. 2.8.0 has a release branch cut date of 10/10 according to our
>>> release calendar.
>>>
>>> On Thu, Sep 20, 2018 at 2:47 PM, Connell O'Callaghan <
>>> connello@google.com> wrote:
>>>
>>>> +1 to Thomas's suggestion - if Charles or others cannot reproduce.
>>>>
>>>> On Thu, Sep 20, 2018 at 2:40 PM Thomas Weise <th...@apache.org> wrote:
>>>>
>>>>> We can also consider releasing 2.7.0 and then follow up with 2.7.1 if
>>>>> the problem can be reproduced and requires a fix. Just food for thought :)
>>>>>
>>>>>
>>>>> On Thu, Sep 20, 2018 at 2:13 PM Charles Chen <cc...@google.com> wrote:
>>>>>
>>>>>> My mistake, it looks like the correct beam staging repository (
>>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1046/)
>>>>>> is specified in your pom file.
>>>>>>
>>>>>> On Thu, Sep 20, 2018 at 2:10 PM Charles Chen <cc...@google.com> wrote:
>>>>>>
>>>>>>> Hey Romain and JB, do you have any progress on this? One thing I
>>>>>>> would like to point out is that 2.7.0 isn't yet pushed to Maven Central, so
>>>>>>> referring to it by version is not expected to work (and it looks like this
>>>>>>> is what is done in your repo:
>>>>>>> https://github.com/rmannibucau/beam-2.7.0-fails). Luke indicated
>>>>>>> above that he doesn't see any dependency changes. Can you isolate and
>>>>>>> reproduce this problem so that we can develop a fix, if necessary? I would
>>>>>>> like to proceed with an RC2 as soon as possible.
>>>>>>>
>>>>>>> On Wed, Sep 19, 2018 at 6:37 AM Romain Manni-Bucau <
>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>
>>>>>>>> Quick update on the spark issue: I didn't get enough time to
>>>>>>>> identify it clearly but managed to have a passing run of my test changing a
>>>>>>>> bunch of versions.
>>>>>>>> I suspect my code triggers some class conflicting between spark and
>>>>>>>> my shade leading to a serialization issue. I didn't test userClassPathFirst
>>>>>>>> option of spark but it can be an interesting thing to enable in beam runner.
>>>>>>>> However it is still very confusing to have it not running just
>>>>>>>> upgrading beam version and the spark error is very hard to understand.
>>>>>>>>
>>>>>>>> Romain Manni-Bucau
>>>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>>>>> <http://rmannibucau.wordpress.com> | Github
>>>>>>>> <https://github.com/rmannibucau> | LinkedIn
>>>>>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>>>>>
>>>>>>>>
>>>>>>>> Le mar. 18 sept. 2018 à 20:17, Lukasz Cwik <lc...@google.com> a
>>>>>>>> écrit :
>>>>>>>>
>>>>>>>>> Romain hinted that this was a dependency issue but when comparing
>>>>>>>>> the two dependency trees I don't get much of a difference:
>>>>>>>>>
>>>>>>>>> lcwik@lcwik0: ~$ diff /tmp/260 /tmp/270
>>>>>>>>> < [INFO] +- org.apache.beam:beam-runners-spark:jar:2.6.0:compile
>>>>>>>>> < [INFO] | +-
>>>>>>>>> org.apache.beam:beam-model-pipeline:jar:2.6.0:compile
>>>>>>>>> ---
>>>>>>>>> > [INFO] +- org.apache.beam:beam-runners-spark:jar:2.7.0:compile
>>>>>>>>> > [INFO] | +-
>>>>>>>>> org.apache.beam:beam-model-pipeline:jar:2.7.0:compile
>>>>>>>>> 5c6
>>>>>>>>> < [INFO] | +-
>>>>>>>>> org.apache.beam:beam-sdks-java-core:jar:2.6.0:compile
>>>>>>>>> ---
>>>>>>>>> > [INFO] | +-
>>>>>>>>> org.apache.beam:beam-sdks-java-core:jar:2.7.0:compile
>>>>>>>>> 14,18c15,19
>>>>>>>>> < [INFO] | | \- org.tukaani:xz:jar:1.5:compile
>>>>>>>>> < [INFO] | +-
>>>>>>>>> org.apache.beam:beam-runners-core-construction-java:jar:2.6.0:compile
>>>>>>>>> < [INFO] | | \-
>>>>>>>>> org.apache.beam:beam-model-job-management:jar:2.6.0:compile
>>>>>>>>> < [INFO] | +-
>>>>>>>>> org.apache.beam:beam-runners-core-java:jar:2.6.0:compile
>>>>>>>>> < [INFO] | | \-
>>>>>>>>> org.apache.beam:beam-model-fn-execution:jar:2.6.0:compile
>>>>>>>>> ---
>>>>>>>>> > [INFO] | | \- org.tukaani:xz:jar:1.8:compile
>>>>>>>>> > [INFO] | +-
>>>>>>>>> org.apache.beam:beam-runners-core-construction-java:jar:2.7.0:compile
>>>>>>>>> > [INFO] | | \-
>>>>>>>>> org.apache.beam:beam-model-job-management:jar:2.7.0:compile
>>>>>>>>> > [INFO] | +-
>>>>>>>>> org.apache.beam:beam-runners-core-java:jar:2.7.0:compile
>>>>>>>>> > [INFO] | | \-
>>>>>>>>> org.apache.beam:beam-model-fn-execution:jar:2.7.0:compile
>>>>>>>>>
>>>>>>>>> Other then Beam package changes, the only other change is xz which
>>>>>>>>> I don't believe could be causing the issue.
>>>>>>>>>
>>>>>>>>> On Tue, Sep 18, 2018 at 8:38 AM Jean-Baptiste Onofré <
>>>>>>>>> jb@nanthrax.net> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks, let me take a look.
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>> JB
>>>>>>>>>>
>>>>>>>>>> On 18/09/2018 17:36, Romain Manni-Bucau wrote:
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > Le mar. 18 sept. 2018 à 16:44, Jean-Baptiste Onofré <
>>>>>>>>>> jb@nanthrax.net
>>>>>>>>>> > <ma...@nanthrax.net>> a écrit :
>>>>>>>>>> >
>>>>>>>>>> > Hi,
>>>>>>>>>> >
>>>>>>>>>> > I don't have the issue ;)
>>>>>>>>>> >
>>>>>>>>>> > As said in my vote, I tested 2.7.0 RC1 on beam-samples with
>>>>>>>>>> Spark
>>>>>>>>>> > without problem.
>>>>>>>>>> >
>>>>>>>>>> > I don't reproduce Romain issue as well.
>>>>>>>>>> >
>>>>>>>>>> > @Romain can you provide some details to reproduce the issue
>>>>>>>>>> ?
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > Sure, you can use this
>>>>>>>>>> > reproducer: https://github.com/rmannibucau/beam-2.7.0-fails
>>>>>>>>>> > It shows that it suceeds on 2.6 and fails on 2.7.
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > Regards
>>>>>>>>>> > JB
>>>>>>>>>> >
>>>>>>>>>> > On 17/09/2018 19:17, Charles Chen wrote:
>>>>>>>>>> > > Luke, Maximillian, Raghu, can you please propose
>>>>>>>>>> cherry-pick PRs
>>>>>>>>>> > to the
>>>>>>>>>> > > release-2.7.0 for your issues and add me as a reviewer
>>>>>>>>>> > (@charlesccychen)?
>>>>>>>>>> > >
>>>>>>>>>> > > Romain, JB: is there any way I can help with debugging
>>>>>>>>>> the issue
>>>>>>>>>> > you're
>>>>>>>>>> > > facing so we can unblock the release?
>>>>>>>>>> > >
>>>>>>>>>> > > On Fri, Sep 14, 2018 at 1:49 PM Raghu Angadi <
>>>>>>>>>> rangadi@google.com
>>>>>>>>>> > <ma...@google.com>
>>>>>>>>>> > > <mailto:rangadi@google.com <ma...@google.com>>>
>>>>>>>>>> wrote:
>>>>>>>>>> > >
>>>>>>>>>> > > I would like propose one more cherrypick for RC2
>>>>>>>>>> > > : https://github.com/apache/beam/pull/6391
>>>>>>>>>> > > This is a KafkaIO bug fix. Once a user hits this bug,
>>>>>>>>>> there is no
>>>>>>>>>> > > easy work around for them, especially on Dataflow.
>>>>>>>>>> Only work
>>>>>>>>>> > around
>>>>>>>>>> > > in Dataflow is to restart or reload the job.
>>>>>>>>>> > >
>>>>>>>>>> > > The fix itself fairly safe and is tested.
>>>>>>>>>> > > Raghu.
>>>>>>>>>> > >
>>>>>>>>>> > > On Fri, Sep 14, 2018 at 12:52 AM Alexey Romanenko
>>>>>>>>>> > > <aromanenko.dev@gmail.com <mailto:
>>>>>>>>>> aromanenko.dev@gmail.com>
>>>>>>>>>> > <mailto:aromanenko.dev@gmail.com <mailto:
>>>>>>>>>> aromanenko.dev@gmail.com>>>
>>>>>>>>>> > wrote:
>>>>>>>>>> > >
>>>>>>>>>> > > Perhaps it could help, but I run simple WordCount
>>>>>>>>>> (built with
>>>>>>>>>> > > Beam 2.7) on YARN/Spark (HDP Sandbox) cluster and
>>>>>>>>>> it
>>>>>>>>>> > worked fine
>>>>>>>>>> > > for me.
>>>>>>>>>> > >
>>>>>>>>>> > >> On 14 Sep 2018, at 06:56, Romain Manni-Bucau
>>>>>>>>>> > >> <rmannibucau@gmail.com <mailto:
>>>>>>>>>> rmannibucau@gmail.com>
>>>>>>>>>> > <mailto:rmannibucau@gmail.com <ma...@gmail.com>>>
>>>>>>>>>> wrote:
>>>>>>>>>> > >>
>>>>>>>>>> > >> Hi Charles,
>>>>>>>>>> > >>
>>>>>>>>>> > >> I didn't get enough time to check deeply but it
>>>>>>>>>> is clearly a
>>>>>>>>>> > >> dependency issue and it is not in beam spark
>>>>>>>>>> runner
>>>>>>>>>> > itself but
>>>>>>>>>> > >> in another transitive module of beam. It does
>>>>>>>>>> not happen in
>>>>>>>>>> > >> existing spark test cause none of them are in a
>>>>>>>>>> cluster (even
>>>>>>>>>> > >> just with 1 worker) but this seems to be a
>>>>>>>>>> regression since
>>>>>>>>>> > >> 2.6 works OOTB.
>>>>>>>>>> > >>
>>>>>>>>>> > >> Romain Manni-Bucau
>>>>>>>>>> > >> @rmannibucau <https://twitter.com/rmannibucau>
>>>>>>>>>> | Blog
>>>>>>>>>> > >> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>>>>>>> > >> <http://rmannibucau.wordpress.com/> | Github
>>>>>>>>>> > >> <https://github.com/rmannibucau> | LinkedIn
>>>>>>>>>> > >> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>>>>>> > >>
>>>>>>>>>> > <
>>>>>>>>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>>>>>>>> >
>>>>>>>>>> > >>
>>>>>>>>>> > >>
>>>>>>>>>> > >> Le jeu. 13 sept. 2018 à 22:15, Charles Chen
>>>>>>>>>> > <ccy@google.com <ma...@google.com>
>>>>>>>>>> > >> <mailto:ccy@google.com <ma...@google.com>>>
>>>>>>>>>> a écrit :
>>>>>>>>>> > >>
>>>>>>>>>> > >> Romain and JB, can you please add the
>>>>>>>>>> results of your
>>>>>>>>>> > >> investigations into the errors you've seen
>>>>>>>>>> above? Given
>>>>>>>>>> > >> that the existing SparkRunner tests pass for
>>>>>>>>>> this RC, and
>>>>>>>>>> > >> that the integration test you ran is in
>>>>>>>>>> another repo that
>>>>>>>>>> > >> is not continuously tested with Beam, it is
>>>>>>>>>> not clear how
>>>>>>>>>> > >> we should move forward and whether this is a
>>>>>>>>>> blocking
>>>>>>>>>> > >> issue, unless we can find a root cause in
>>>>>>>>>> Beam.
>>>>>>>>>> > >>
>>>>>>>>>> > >> On Wed, Sep 12, 2018 at 2:08 AM Etienne
>>>>>>>>>> Chauchot
>>>>>>>>>> > >> <echauchot@apache.org <mailto:
>>>>>>>>>> echauchot@apache.org>
>>>>>>>>>> > <mailto:echauchot@apache.org <ma...@apache.org>>>
>>>>>>>>>> wrote:
>>>>>>>>>> > >>
>>>>>>>>>> > >> Hi all,
>>>>>>>>>> > >>
>>>>>>>>>> > >> on a performance and functional
>>>>>>>>>> regression stand
>>>>>>>>>> > point
>>>>>>>>>> > >> I see no regression:
>>>>>>>>>> > >>
>>>>>>>>>> > >> I looked at nexmark graphs "output
>>>>>>>>>> pcollection size"
>>>>>>>>>> > >> and "execution time" around release cut
>>>>>>>>>> date on
>>>>>>>>>> > >> dataflow, spark, flink and direct runner
>>>>>>>>>> in batch and
>>>>>>>>>> > >> streaming modes. There seems to be no
>>>>>>>>>> regression.
>>>>>>>>>> > >>
>>>>>>>>>> > >> Etienne
>>>>>>>>>> > >>
>>>>>>>>>> > >> Le mardi 11 septembre 2018 à 12:25
>>>>>>>>>> -0700, Charles
>>>>>>>>>> > Chen
>>>>>>>>>> > >> a écrit :
>>>>>>>>>> > >>> The SparkRunner validation test
>>>>>>>>>> > >>>
>>>>>>>>>> > (here:
>>>>>>>>>> https://beam.apache.org/contribute/release-guide/#run-validation-tests
>>>>>>>>>> )
>>>>>>>>>> > >>> passes on my machine. It looks like we
>>>>>>>>>> are likely
>>>>>>>>>> > >>> missing test coverage where Romain is
>>>>>>>>>> hitting
>>>>>>>>>> > issues.
>>>>>>>>>> > >>>
>>>>>>>>>> > >>> On Tue, Sep 11, 2018 at 12:15 PM Ahmet
>>>>>>>>>> Altay
>>>>>>>>>> > >>> <altay@google.com <mailto:
>>>>>>>>>> altay@google.com>
>>>>>>>>>> > <mailto:altay@google.com <ma...@google.com>>> wrote:
>>>>>>>>>> > >>>> Could anyone else help with looking at
>>>>>>>>>> these issues
>>>>>>>>>> > >>>> earlier?
>>>>>>>>>> > >>>>
>>>>>>>>>> > >>>> On Tue, Sep 11, 2018 at 12:03 PM,
>>>>>>>>>> Romain
>>>>>>>>>> > Manni-Bucau
>>>>>>>>>> > >>>> <rmannibucau@gmail.com
>>>>>>>>>> > <ma...@gmail.com>
>>>>>>>>>> > >>>> <mailto:rmannibucau@gmail.com
>>>>>>>>>> > <ma...@gmail.com>>> wrote:
>>>>>>>>>> > >>>>> Im running this main [1] through this
>>>>>>>>>> IT [2]. Was
>>>>>>>>>> > >>>>> working fine since ~1 year but 2.7.0
>>>>>>>>>> broke it.
>>>>>>>>>> > >>>>> Didnt investigate more but can have a
>>>>>>>>>> look later
>>>>>>>>>> > >>>>> this month if it helps.
>>>>>>>>>> > >>>>>
>>>>>>>>>> > >>>>>
>>>>>>>>>> > [1]
>>>>>>>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
>>>>>>>>>> > >>>>>
>>>>>>>>>> > [2]
>>>>>>>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
>>>>>>>>>> > >>>>>
>>>>>>>>>> > >>>>> Le mar. 11 sept. 2018 20:54, Charles
>>>>>>>>>> Chen
>>>>>>>>>> > >>>>> <ccy@google.com <mailto:
>>>>>>>>>> ccy@google.com>
>>>>>>>>>> > <mailto:ccy@google.com <ma...@google.com>>> a écrit :
>>>>>>>>>> > >>>>>> Romain: can you give more details on
>>>>>>>>>> the failure
>>>>>>>>>> > >>>>>> you're encountering, i.e. how you
>>>>>>>>>> are performing
>>>>>>>>>> > >>>>>> this validation?
>>>>>>>>>> > >>>>>>
>>>>>>>>>> > >>>>>> On Tue, Sep 11, 2018 at 9:36 AM
>>>>>>>>>> Jean-Baptiste
>>>>>>>>>> > >>>>>> Onofré <jb@nanthrax.net
>>>>>>>>>> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
>>>>>>>>>> > <ma...@nanthrax.net>>>
>>>>>>>>>> > >>>>>> wrote:
>>>>>>>>>> > >>>>>>> Hi,
>>>>>>>>>> > >>>>>>>
>>>>>>>>>> > >>>>>>> weird, I didn't have it on Beam
>>>>>>>>>> samples. Let me
>>>>>>>>>> > >>>>>>> try to reproduce and I
>>>>>>>>>> > >>>>>>> will create the Jira.
>>>>>>>>>> > >>>>>>>
>>>>>>>>>> > >>>>>>> Regards
>>>>>>>>>> > >>>>>>> JB
>>>>>>>>>> > >>>>>>>
>>>>>>>>>> > >>>>>>> On 11/09/2018 11:44, Romain
>>>>>>>>>> Manni-Bucau wrote:
>>>>>>>>>> > >>>>>>> > -1, seems spark integration is
>>>>>>>>>> broken (tested
>>>>>>>>>> > >>>>>>> with spark 2.3.1 and 2.2.1):
>>>>>>>>>> > >>>>>>> >
>>>>>>>>>> > >>>>>>> > 18/09/11 11:33:29 WARN
>>>>>>>>>> TaskSetManager: Lost
>>>>>>>>>> > >>>>>>> task 0.0 in stage 0.0 (TID 0,
>>>>>>>>>> RMANNIBUCAU,
>>>>>>>>>> > >>>>>>> executor 0):
>>>>>>>>>> java.lang.ClassCastException:
>>>>>>>>>> > cannot
>>>>>>>>>> > >>>>>>> assign instance of
>>>>>>>>>> > >>>>>>>
>>>>>>>>>> > scala.collection.immutable.List$SerializationProxy to
>>>>>>>>>> > >>>>>>> fieldorg.apache.spark.rdd.RDD.org
>>>>>>>>>> > <http://fieldorg.apache.spark.rdd.RDD.org>
>>>>>>>>>> > >>>>>>> <
>>>>>>>>>> http://fieldorg.apache.spark.rdd.rdd.org/>
>>>>>>>>>> > >>>>>>> <
>>>>>>>>>> http://org.apache.spark.rdd.RDD.org
>>>>>>>>>> > >>>>>>>
>>>>>>>>>> > <http://org.apache.spark.rdd.rdd.org/
>>>>>>>>>> >>$apache$spark$rdd$RDD$$dependencies_
>>>>>>>>>> > >>>>>>> of type scala.collection.Seq in
>>>>>>>>>> instance of
>>>>>>>>>> > >>>>>>>
>>>>>>>>>> org.apache.spark.rdd.MapPartitionsRDD
>>>>>>>>>> > >>>>>>> > at
>>>>>>>>>> > >>>>>>>
>>>>>>>>>> >
>>>>>>>>>> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>>>>>>>>>> > >>>>>>> >
>>>>>>>>>> > >>>>>>> >
>>>>>>>>>> > >>>>>>> > Also the issue Lukasz identified
>>>>>>>>>> is important
>>>>>>>>>> > >>>>>>> even if workarounds can be
>>>>>>>>>> > >>>>>>> > put in place so +1 to fix it as
>>>>>>>>>> well if
>>>>>>>>>> > possible.
>>>>>>>>>> > >>>>>>> >
>>>>>>>>>> > >>>>>>> > Romain Manni-Bucau
>>>>>>>>>> > >>>>>>> > @rmannibucau <
>>>>>>>>>> https://twitter.com/rmannibucau>
>>>>>>>>>> > >>>>>>> | Blog
>>>>>>>>>> > >>>>>>> > <
>>>>>>>>>> https://rmannibucau.metawerx.net/> | Old Blog
>>>>>>>>>> > >>>>>>> > <http://rmannibucau.wordpress.com
>>>>>>>>>> > >>>>>>> <http://rmannibucau.wordpress.com/>>
>>>>>>>>>> | Github
>>>>>>>>>> > >>>>>>> > <https://github.com/rmannibucau> |
>>>>>>>>>> LinkedIn
>>>>>>>>>> > >>>>>>> >
>>>>>>>>>> > <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>>>>>> > >>>>>>> >
>>>>>>>>>> > >>>>>>>
>>>>>>>>>> > <
>>>>>>>>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>>>>>>>> >
>>>>>>>>>> > >>>>>>> >
>>>>>>>>>> > >>>>>>> >
>>>>>>>>>> > >>>>>>> > Le lun. 10 sept. 2018 à 20:48,
>>>>>>>>>> Lukasz Cwik
>>>>>>>>>> > >>>>>>> <lcwik@google.com <mailto:
>>>>>>>>>> lcwik@google.com>
>>>>>>>>>> > <mailto:lcwik@google.com <ma...@google.com>>
>>>>>>>>>> > >>>>>>> > <mailto:lcwik@google.com
>>>>>>>>>> > <ma...@google.com>
>>>>>>>>>> > >>>>>>> <mailto:lcwik@google.com
>>>>>>>>>> > <ma...@google.com>>>> a écrit :
>>>>>>>>>> > >>>>>>> >
>>>>>>>>>> > >>>>>>> > I found an issue where we are
>>>>>>>>>> no longer
>>>>>>>>>> > >>>>>>> packaging the pom.xml within
>>>>>>>>>> > >>>>>>> > the artifact jars at
>>>>>>>>>> > >>>>>>> META-INF/maven/groupId/artifactId.
>>>>>>>>>> More details
>>>>>>>>>> > >>>>>>> > in
>>>>>>>>>> > >>>>>>>
>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-5351.
>>>>>>>>>> > >>>>>>> I wouldn't
>>>>>>>>>> > >>>>>>> > consider this a blocker but
>>>>>>>>>> it was an
>>>>>>>>>> > easy fix
>>>>>>>>>> > >>>>>>> > (
>>>>>>>>>> https://github.com/apache/beam/pull/6358)
>>>>>>>>>> > >>>>>>> and users may rely on the
>>>>>>>>>> > >>>>>>> > pom.xml.
>>>>>>>>>> > >>>>>>> >
>>>>>>>>>> > >>>>>>> > Should we recut the release
>>>>>>>>>> candidate to
>>>>>>>>>> > >>>>>>> include this?
>>>>>>>>>> > >>>>>>> >
>>>>>>>>>> > >>>>>>> > On Mon, Sep 10, 2018 at 4:58
>>>>>>>>>> AM
>>>>>>>>>> > >>>>>>> Jean-Baptiste Onofré
>>>>>>>>>> > >>>>>>> > <jb@nanthrax.net
>>>>>>>>>> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
>>>>>>>>>> > <ma...@nanthrax.net>>
>>>>>>>>>> > >>>>>>> <mailto:jb@nanthrax.net <mailto:
>>>>>>>>>> jb@nanthrax.net>
>>>>>>>>>> > >>>>>>> <mailto:jb@nanthrax.net
>>>>>>>>>> > <ma...@nanthrax.net>>>> wrote:
>>>>>>>>>> > >>>>>>> >
>>>>>>>>>> > >>>>>>> > +1 (binding)
>>>>>>>>>> > >>>>>>> >
>>>>>>>>>> > >>>>>>> > Tested successfully on
>>>>>>>>>> Beam Samples.
>>>>>>>>>> > >>>>>>> >
>>>>>>>>>> > >>>>>>> > Thanks !
>>>>>>>>>> > >>>>>>> >
>>>>>>>>>> > >>>>>>> > Regards
>>>>>>>>>> > >>>>>>> > JB
>>>>>>>>>> > >>>>>>> >
>>>>>>>>>> > >>>>>>> > On 07/09/2018 23:56,
>>>>>>>>>> Charles Chen
>>>>>>>>>> > wrote:
>>>>>>>>>> > >>>>>>> > > Hi everyone,
>>>>>>>>>> > >>>>>>> > >
>>>>>>>>>> > >>>>>>> > > Please review and vote
>>>>>>>>>> on the
>>>>>>>>>> > >>>>>>> release candidate #1 for the
>>>>>>>>>> > >>>>>>> > version
>>>>>>>>>> > >>>>>>> > > 2.7.0, as follows:
>>>>>>>>>> > >>>>>>> > > [ ] +1, Approve the
>>>>>>>>>> release
>>>>>>>>>> > >>>>>>> > > [ ] -1, Do not approve
>>>>>>>>>> the release
>>>>>>>>>> > >>>>>>> (please provide specific
>>>>>>>>>> > >>>>>>> > comments)
>>>>>>>>>> > >>>>>>> > >
>>>>>>>>>> > >>>>>>> > > The complete staging
>>>>>>>>>> area is
>>>>>>>>>> > >>>>>>> available for your review, which
>>>>>>>>>> > >>>>>>> > includes:
>>>>>>>>>> > >>>>>>> > > * JIRA release notes
>>>>>>>>>> [1],
>>>>>>>>>> > >>>>>>> > > * the official Apache
>>>>>>>>>> source
>>>>>>>>>> > release
>>>>>>>>>> > >>>>>>> to be deployed to
>>>>>>>>>> > >>>>>>> > dist.apache.org
>>>>>>>>>> > <http://dist.apache.org>
>>>>>>>>>> > >>>>>>> <http://dist.apache.org/>
>>>>>>>>>> > <http://dist.apache.org
>>>>>>>>>> > >>>>>>> <http://dist.apache.org/>>
>>>>>>>>>> > >>>>>>> > > <
>>>>>>>>>> http://dist.apache.org
>>>>>>>>>> > >>>>>>> <http://dist.apache.org/>> [2],
>>>>>>>>>> which is signed
>>>>>>>>>> > >>>>>>> with the key with
>>>>>>>>>> > >>>>>>> > > fingerprint
>>>>>>>>>> 45C60AAAD115F560 [3],
>>>>>>>>>> > >>>>>>> > > * all artifacts to be
>>>>>>>>>> deployed to
>>>>>>>>>> > >>>>>>> the Maven Central
>>>>>>>>>> > >>>>>>> > Repository [4],
>>>>>>>>>> > >>>>>>> > > * source code tag
>>>>>>>>>> "v2.7.0-RC1" [5],
>>>>>>>>>> > >>>>>>> > > * website pull request
>>>>>>>>>> listing the
>>>>>>>>>> > >>>>>>> release and publishing the API
>>>>>>>>>> > >>>>>>> > > reference manual [6].
>>>>>>>>>> > >>>>>>> > > * Java artifacts were
>>>>>>>>>> built with
>>>>>>>>>> > >>>>>>> Gradle 4.8 and OpenJDK
>>>>>>>>>> > >>>>>>> > >
>>>>>>>>>> 1.8.0_181-8u181-b13-1~deb9u1-b13.
>>>>>>>>>> > >>>>>>> > > * Python artifacts are
>>>>>>>>>> deployed
>>>>>>>>>> > >>>>>>> along with the source release
>>>>>>>>>> > >>>>>>> > to the
>>>>>>>>>> > >>>>>>> > > dist.apache.org
>>>>>>>>>> > <http://dist.apache.org>
>>>>>>>>>> > >>>>>>> <http://dist.apache.org/>
>>>>>>>>>> > <http://dist.apache.org
>>>>>>>>>> > >>>>>>> <http://dist.apache.org/>>
>>>>>>>>>> > >>>>>>> > <http://dist.apache.org
>>>>>>>>>> > >>>>>>> <http://dist.apache.org/>> [2].
>>>>>>>>>> > >>>>>>> > >
>>>>>>>>>> > >>>>>>> > > The vote will be open
>>>>>>>>>> for at least
>>>>>>>>>> > >>>>>>> 72 hours. It is adopted by
>>>>>>>>>> > >>>>>>> > majority
>>>>>>>>>> > >>>>>>> > > approval, with at
>>>>>>>>>> least 3 PMC
>>>>>>>>>> > >>>>>>> affirmative votes.
>>>>>>>>>> > >>>>>>> > >
>>>>>>>>>> > >>>>>>> > > Thanks,
>>>>>>>>>> > >>>>>>> > > Charles
>>>>>>>>>> > >>>>>>> > >
>>>>>>>>>> > >>>>>>> > > [1]
>>>>>>>>>> > >>>>>>> > >
>>>>>>>>>> > >>>>>>> >
>>>>>>>>>> > >>>>>>>
>>>>>>>>>> >
>>>>>>>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>>>>>>>>>> > >>>>>>> > > [2]
>>>>>>>>>> > >>>>>>>
>>>>>>>>>> > https://dist.apache.org/repos/dist/dev/beam/2.7.0
>>>>>>>>>> > >>>>>>> > > [3]
>>>>>>>>>> > >>>>>>>
>>>>>>>>>> https://dist.apache.org/repos/dist/dev/beam/KEYS
>>>>>>>>>> > >>>>>>> > > [4]
>>>>>>>>>> > >>>>>>> >
>>>>>>>>>> > >>>>>>>
>>>>>>>>>> >
>>>>>>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1046/
>>>>>>>>>> > >>>>>>> > > [5]
>>>>>>>>>> > >>>>>>>
>>>>>>>>>> https://github.com/apache/beam/tree/v2.7.0-RC1
>>>>>>>>>> > >>>>>>> > > [6]
>>>>>>>>>> > >>>>>>>
>>>>>>>>>> https://github.com/apache/beam-site/pull/549
>>>>>>>>>> > >>>>>>> >
>>>>>>>>>> > >>>>>>> > --
>>>>>>>>>> > >>>>>>> > Jean-Baptiste Onofré
>>>>>>>>>> > >>>>>>> > jbonofre@apache.org
>>>>>>>>>> > <ma...@apache.org>
>>>>>>>>>> > >>>>>>> <mailto:jbonofre@apache.org
>>>>>>>>>> > <ma...@apache.org>>
>>>>>>>>>> > >>>>>>> <mailto:jbonofre@apache.org
>>>>>>>>>> > <ma...@apache.org>
>>>>>>>>>> > >>>>>>> <mailto:jbonofre@apache.org
>>>>>>>>>> > <ma...@apache.org>>>
>>>>>>>>>> > >>>>>>> > http://blog.nanthrax.net
>>>>>>>>>> > >>>>>>> <http://blog.nanthrax.net/>
>>>>>>>>>> > >>>>>>> > Talend -
>>>>>>>>>> http://www.talend.com
>>>>>>>>>> > >>>>>>> <http://www.talend.com/>
>>>>>>>>>> > >>>>>>> >
>>>>>>>>>> > >>>>
>>>>>>>>>> > >
>>>>>>>>>> >
>>>>>>>>>> > --
>>>>>>>>>> > Jean-Baptiste Onofré
>>>>>>>>>> > jbonofre@apache.org <ma...@apache.org>
>>>>>>>>>> > http://blog.nanthrax.net
>>>>>>>>>> > Talend - http://www.talend.com
>>>>>>>>>> >
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Jean-Baptiste Onofré
>>>>>>>>>> jbonofre@apache.org
>>>>>>>>>> http://blog.nanthrax.net
>>>>>>>>>> Talend - http://www.talend.com
>>>>>>>>>>
>>>>>>>>>
>>>
>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Ahmet Altay <al...@google.com>.
Good point. However, we agreed that our release policy would be to patch
only long term support (LTS) releases. Given that we have not made any LTS
releases yet, perhaps we should use 2.8.0 as the opportunity to make our
first LTS release.
On Thu, Sep 20, 2018 at 6:26 PM, Thomas Weise <th...@apache.org> wrote:
> That's not the same for a user though. 2.7.1 would be a patch compatible
> release that only fixes bugs. 2.8.0 adds new features and potentially also
> new issues..
>
> On Thu, Sep 20, 2018 at 3:16 PM Ahmet Altay <al...@google.com> wrote:
>
>> +1 to Thomas's suggestion. Instead of 2.7.1 we can follow up with 2.8.0
>> though. 2.8.0 has a release branch cut date of 10/10 according to our
>> release calendar.
>>
>> On Thu, Sep 20, 2018 at 2:47 PM, Connell O'Callaghan <connello@google.com
>> > wrote:
>>
>>> +1 to Thomas's suggestion - if Charles or others cannot reproduce.
>>>
>>> On Thu, Sep 20, 2018 at 2:40 PM Thomas Weise <th...@apache.org> wrote:
>>>
>>>> We can also consider releasing 2.7.0 and then follow up with 2.7.1 if
>>>> the problem can be reproduced and requires a fix. Just food for thought :)
>>>>
>>>>
>>>> On Thu, Sep 20, 2018 at 2:13 PM Charles Chen <cc...@google.com> wrote:
>>>>
>>>>> My mistake, it looks like the correct beam staging repository (
>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1046/)
>>>>> is specified in your pom file.
>>>>>
>>>>> On Thu, Sep 20, 2018 at 2:10 PM Charles Chen <cc...@google.com> wrote:
>>>>>
>>>>>> Hey Romain and JB, do you have any progress on this? One thing I
>>>>>> would like to point out is that 2.7.0 isn't yet pushed to Maven Central, so
>>>>>> referring to it by version is not expected to work (and it looks like this
>>>>>> is what is done in your repo: https://github.com/
>>>>>> rmannibucau/beam-2.7.0-fails). Luke indicated above that he doesn't
>>>>>> see any dependency changes. Can you isolate and reproduce this problem so
>>>>>> that we can develop a fix, if necessary? I would like to proceed with an
>>>>>> RC2 as soon as possible.
>>>>>>
>>>>>> On Wed, Sep 19, 2018 at 6:37 AM Romain Manni-Bucau <
>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>
>>>>>>> Quick update on the spark issue: I didn't get enough time to
>>>>>>> identify it clearly but managed to have a passing run of my test changing a
>>>>>>> bunch of versions.
>>>>>>> I suspect my code triggers some class conflicting between spark and
>>>>>>> my shade leading to a serialization issue. I didn't test userClassPathFirst
>>>>>>> option of spark but it can be an interesting thing to enable in beam runner.
>>>>>>> However it is still very confusing to have it not running just
>>>>>>> upgrading beam version and the spark error is very hard to understand.
>>>>>>>
>>>>>>> Romain Manni-Bucau
>>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>>>> <http://rmannibucau.wordpress.com> | Github
>>>>>>> <https://github.com/rmannibucau> | LinkedIn
>>>>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>>>>
>>>>>>>
>>>>>>> Le mar. 18 sept. 2018 à 20:17, Lukasz Cwik <lc...@google.com> a
>>>>>>> écrit :
>>>>>>>
>>>>>>>> Romain hinted that this was a dependency issue but when comparing
>>>>>>>> the two dependency trees I don't get much of a difference:
>>>>>>>>
>>>>>>>> lcwik@lcwik0: ~$ diff /tmp/260 /tmp/270
>>>>>>>> < [INFO] +- org.apache.beam:beam-runners-spark:jar:2.6.0:compile
>>>>>>>> < [INFO] | +- org.apache.beam:beam-model-
>>>>>>>> pipeline:jar:2.6.0:compile
>>>>>>>> ---
>>>>>>>> > [INFO] +- org.apache.beam:beam-runners-spark:jar:2.7.0:compile
>>>>>>>> > [INFO] | +- org.apache.beam:beam-model-
>>>>>>>> pipeline:jar:2.7.0:compile
>>>>>>>> 5c6
>>>>>>>> < [INFO] | +- org.apache.beam:beam-sdks-
>>>>>>>> java-core:jar:2.6.0:compile
>>>>>>>> ---
>>>>>>>> > [INFO] | +- org.apache.beam:beam-sdks-
>>>>>>>> java-core:jar:2.7.0:compile
>>>>>>>> 14,18c15,19
>>>>>>>> < [INFO] | | \- org.tukaani:xz:jar:1.5:compile
>>>>>>>> < [INFO] | +- org.apache.beam:beam-runners-
>>>>>>>> core-construction-java:jar:2.6.0:compile
>>>>>>>> < [INFO] | | \- org.apache.beam:beam-model-
>>>>>>>> job-management:jar:2.6.0:compile
>>>>>>>> < [INFO] | +- org.apache.beam:beam-runners-
>>>>>>>> core-java:jar:2.6.0:compile
>>>>>>>> < [INFO] | | \- org.apache.beam:beam-model-fn-
>>>>>>>> execution:jar:2.6.0:compile
>>>>>>>> ---
>>>>>>>> > [INFO] | | \- org.tukaani:xz:jar:1.8:compile
>>>>>>>> > [INFO] | +- org.apache.beam:beam-runners-
>>>>>>>> core-construction-java:jar:2.7.0:compile
>>>>>>>> > [INFO] | | \- org.apache.beam:beam-model-
>>>>>>>> job-management:jar:2.7.0:compile
>>>>>>>> > [INFO] | +- org.apache.beam:beam-runners-
>>>>>>>> core-java:jar:2.7.0:compile
>>>>>>>> > [INFO] | | \- org.apache.beam:beam-model-fn-
>>>>>>>> execution:jar:2.7.0:compile
>>>>>>>>
>>>>>>>> Other then Beam package changes, the only other change is xz which
>>>>>>>> I don't believe could be causing the issue.
>>>>>>>>
>>>>>>>> On Tue, Sep 18, 2018 at 8:38 AM Jean-Baptiste Onofré <
>>>>>>>> jb@nanthrax.net> wrote:
>>>>>>>>
>>>>>>>>> Thanks, let me take a look.
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> JB
>>>>>>>>>
>>>>>>>>> On 18/09/2018 17:36, Romain Manni-Bucau wrote:
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > Le mar. 18 sept. 2018 à 16:44, Jean-Baptiste Onofré <
>>>>>>>>> jb@nanthrax.net
>>>>>>>>> > <ma...@nanthrax.net>> a écrit :
>>>>>>>>> >
>>>>>>>>> > Hi,
>>>>>>>>> >
>>>>>>>>> > I don't have the issue ;)
>>>>>>>>> >
>>>>>>>>> > As said in my vote, I tested 2.7.0 RC1 on beam-samples with
>>>>>>>>> Spark
>>>>>>>>> > without problem.
>>>>>>>>> >
>>>>>>>>> > I don't reproduce Romain issue as well.
>>>>>>>>> >
>>>>>>>>> > @Romain can you provide some details to reproduce the issue ?
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > Sure, you can use this
>>>>>>>>> > reproducer: https://github.com/rmannibucau/beam-2.7.0-fails
>>>>>>>>> > It shows that it suceeds on 2.6 and fails on 2.7.
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > Regards
>>>>>>>>> > JB
>>>>>>>>> >
>>>>>>>>> > On 17/09/2018 19:17, Charles Chen wrote:
>>>>>>>>> > > Luke, Maximillian, Raghu, can you please propose
>>>>>>>>> cherry-pick PRs
>>>>>>>>> > to the
>>>>>>>>> > > release-2.7.0 for your issues and add me as a reviewer
>>>>>>>>> > (@charlesccychen)?
>>>>>>>>> > >
>>>>>>>>> > > Romain, JB: is there any way I can help with debugging the
>>>>>>>>> issue
>>>>>>>>> > you're
>>>>>>>>> > > facing so we can unblock the release?
>>>>>>>>> > >
>>>>>>>>> > > On Fri, Sep 14, 2018 at 1:49 PM Raghu Angadi <
>>>>>>>>> rangadi@google.com
>>>>>>>>> > <ma...@google.com>
>>>>>>>>> > > <mailto:rangadi@google.com <ma...@google.com>>>
>>>>>>>>> wrote:
>>>>>>>>> > >
>>>>>>>>> > > I would like propose one more cherrypick for RC2
>>>>>>>>> > > : https://github.com/apache/beam/pull/6391
>>>>>>>>> > > This is a KafkaIO bug fix. Once a user hits this bug,
>>>>>>>>> there is no
>>>>>>>>> > > easy work around for them, especially on Dataflow.
>>>>>>>>> Only work
>>>>>>>>> > around
>>>>>>>>> > > in Dataflow is to restart or reload the job.
>>>>>>>>> > >
>>>>>>>>> > > The fix itself fairly safe and is tested.
>>>>>>>>> > > Raghu.
>>>>>>>>> > >
>>>>>>>>> > > On Fri, Sep 14, 2018 at 12:52 AM Alexey Romanenko
>>>>>>>>> > > <aromanenko.dev@gmail.com <mailto:
>>>>>>>>> aromanenko.dev@gmail.com>
>>>>>>>>> > <mailto:aromanenko.dev@gmail.com <mailto:
>>>>>>>>> aromanenko.dev@gmail.com>>>
>>>>>>>>> > wrote:
>>>>>>>>> > >
>>>>>>>>> > > Perhaps it could help, but I run simple WordCount
>>>>>>>>> (built with
>>>>>>>>> > > Beam 2.7) on YARN/Spark (HDP Sandbox) cluster and
>>>>>>>>> it
>>>>>>>>> > worked fine
>>>>>>>>> > > for me.
>>>>>>>>> > >
>>>>>>>>> > >> On 14 Sep 2018, at 06:56, Romain Manni-Bucau
>>>>>>>>> > >> <rmannibucau@gmail.com <mailto:
>>>>>>>>> rmannibucau@gmail.com>
>>>>>>>>> > <mailto:rmannibucau@gmail.com <ma...@gmail.com>>>
>>>>>>>>> wrote:
>>>>>>>>> > >>
>>>>>>>>> > >> Hi Charles,
>>>>>>>>> > >>
>>>>>>>>> > >> I didn't get enough time to check deeply but it
>>>>>>>>> is clearly a
>>>>>>>>> > >> dependency issue and it is not in beam spark
>>>>>>>>> runner
>>>>>>>>> > itself but
>>>>>>>>> > >> in another transitive module of beam. It does not
>>>>>>>>> happen in
>>>>>>>>> > >> existing spark test cause none of them are in a
>>>>>>>>> cluster (even
>>>>>>>>> > >> just with 1 worker) but this seems to be a
>>>>>>>>> regression since
>>>>>>>>> > >> 2.6 works OOTB.
>>>>>>>>> > >>
>>>>>>>>> > >> Romain Manni-Bucau
>>>>>>>>> > >> @rmannibucau <https://twitter.com/rmannibucau> |
>>>>>>>>> Blog
>>>>>>>>> > >> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>>>>>> > >> <http://rmannibucau.wordpress.com/> | Github
>>>>>>>>> > >> <https://github.com/rmannibucau> | LinkedIn
>>>>>>>>> > >> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>>>>> > >>
>>>>>>>>> > <https://www.packtpub.com/application-development/java-
>>>>>>>>> ee-8-high-performance>
>>>>>>>>> > >>
>>>>>>>>> > >>
>>>>>>>>> > >> Le jeu. 13 sept. 2018 à 22:15, Charles Chen
>>>>>>>>> > <ccy@google.com <ma...@google.com>
>>>>>>>>> > >> <mailto:ccy@google.com <ma...@google.com>>>
>>>>>>>>> a écrit :
>>>>>>>>> > >>
>>>>>>>>> > >> Romain and JB, can you please add the results
>>>>>>>>> of your
>>>>>>>>> > >> investigations into the errors you've seen
>>>>>>>>> above? Given
>>>>>>>>> > >> that the existing SparkRunner tests pass for
>>>>>>>>> this RC, and
>>>>>>>>> > >> that the integration test you ran is in
>>>>>>>>> another repo that
>>>>>>>>> > >> is not continuously tested with Beam, it is
>>>>>>>>> not clear how
>>>>>>>>> > >> we should move forward and whether this is a
>>>>>>>>> blocking
>>>>>>>>> > >> issue, unless we can find a root cause in
>>>>>>>>> Beam.
>>>>>>>>> > >>
>>>>>>>>> > >> On Wed, Sep 12, 2018 at 2:08 AM Etienne
>>>>>>>>> Chauchot
>>>>>>>>> > >> <echauchot@apache.org <mailto:
>>>>>>>>> echauchot@apache.org>
>>>>>>>>> > <mailto:echauchot@apache.org <ma...@apache.org>>>
>>>>>>>>> wrote:
>>>>>>>>> > >>
>>>>>>>>> > >> Hi all,
>>>>>>>>> > >>
>>>>>>>>> > >> on a performance and functional
>>>>>>>>> regression stand
>>>>>>>>> > point
>>>>>>>>> > >> I see no regression:
>>>>>>>>> > >>
>>>>>>>>> > >> I looked at nexmark graphs "output
>>>>>>>>> pcollection size"
>>>>>>>>> > >> and "execution time" around release cut
>>>>>>>>> date on
>>>>>>>>> > >> dataflow, spark, flink and direct runner
>>>>>>>>> in batch and
>>>>>>>>> > >> streaming modes. There seems to be no
>>>>>>>>> regression.
>>>>>>>>> > >>
>>>>>>>>> > >> Etienne
>>>>>>>>> > >>
>>>>>>>>> > >> Le mardi 11 septembre 2018 à 12:25 -0700,
>>>>>>>>> Charles
>>>>>>>>> > Chen
>>>>>>>>> > >> a écrit :
>>>>>>>>> > >>> The SparkRunner validation test
>>>>>>>>> > >>>
>>>>>>>>> > (here: https://beam.apache.org/contribute/release-guide/#
>>>>>>>>> run-validation-tests)
>>>>>>>>> > >>> passes on my machine. It looks like we
>>>>>>>>> are likely
>>>>>>>>> > >>> missing test coverage where Romain is
>>>>>>>>> hitting
>>>>>>>>> > issues.
>>>>>>>>> > >>>
>>>>>>>>> > >>> On Tue, Sep 11, 2018 at 12:15 PM Ahmet
>>>>>>>>> Altay
>>>>>>>>> > >>> <altay@google.com <mailto:
>>>>>>>>> altay@google.com>
>>>>>>>>> > <mailto:altay@google.com <ma...@google.com>>> wrote:
>>>>>>>>> > >>>> Could anyone else help with looking at
>>>>>>>>> these issues
>>>>>>>>> > >>>> earlier?
>>>>>>>>> > >>>>
>>>>>>>>> > >>>> On Tue, Sep 11, 2018 at 12:03 PM, Romain
>>>>>>>>> > Manni-Bucau
>>>>>>>>> > >>>> <rmannibucau@gmail.com
>>>>>>>>> > <ma...@gmail.com>
>>>>>>>>> > >>>> <mailto:rmannibucau@gmail.com
>>>>>>>>> > <ma...@gmail.com>>> wrote:
>>>>>>>>> > >>>>> Im running this main [1] through this
>>>>>>>>> IT [2]. Was
>>>>>>>>> > >>>>> working fine since ~1 year but 2.7.0
>>>>>>>>> broke it.
>>>>>>>>> > >>>>> Didnt investigate more but can have a
>>>>>>>>> look later
>>>>>>>>> > >>>>> this month if it helps.
>>>>>>>>> > >>>>>
>>>>>>>>> > >>>>>
>>>>>>>>> > [1] https://github.com/Talend/component-runtime/blob/
>>>>>>>>> master/component-runtime-beam/src/it/serialization-over-
>>>>>>>>> cluster/src/main/java/org/talend/sdk/component/beam/it/
>>>>>>>>> clusterserialization/Main.java
>>>>>>>>> > >>>>>
>>>>>>>>> > [2] https://github.com/Talend/component-runtime/blob/
>>>>>>>>> master/component-runtime-beam/src/it/serialization-over-
>>>>>>>>> cluster/src/test/java/org/talend/sdk/component/beam/it/
>>>>>>>>> SerializationOverClusterIT.java
>>>>>>>>> > >>>>>
>>>>>>>>> > >>>>> Le mar. 11 sept. 2018 20:54, Charles
>>>>>>>>> Chen
>>>>>>>>> > >>>>> <ccy@google.com <mailto:ccy@google.com
>>>>>>>>> >
>>>>>>>>> > <mailto:ccy@google.com <ma...@google.com>>> a écrit :
>>>>>>>>> > >>>>>> Romain: can you give more details on
>>>>>>>>> the failure
>>>>>>>>> > >>>>>> you're encountering, i.e. how you are
>>>>>>>>> performing
>>>>>>>>> > >>>>>> this validation?
>>>>>>>>> > >>>>>>
>>>>>>>>> > >>>>>> On Tue, Sep 11, 2018 at 9:36 AM
>>>>>>>>> Jean-Baptiste
>>>>>>>>> > >>>>>> Onofré <jb@nanthrax.net
>>>>>>>>> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
>>>>>>>>> > <ma...@nanthrax.net>>>
>>>>>>>>> > >>>>>> wrote:
>>>>>>>>> > >>>>>>> Hi,
>>>>>>>>> > >>>>>>>
>>>>>>>>> > >>>>>>> weird, I didn't have it on Beam
>>>>>>>>> samples. Let me
>>>>>>>>> > >>>>>>> try to reproduce and I
>>>>>>>>> > >>>>>>> will create the Jira.
>>>>>>>>> > >>>>>>>
>>>>>>>>> > >>>>>>> Regards
>>>>>>>>> > >>>>>>> JB
>>>>>>>>> > >>>>>>>
>>>>>>>>> > >>>>>>> On 11/09/2018 11:44, Romain
>>>>>>>>> Manni-Bucau wrote:
>>>>>>>>> > >>>>>>> > -1, seems spark integration is
>>>>>>>>> broken (tested
>>>>>>>>> > >>>>>>> with spark 2.3.1 and 2.2.1):
>>>>>>>>> > >>>>>>> >
>>>>>>>>> > >>>>>>> > 18/09/11 11:33:29 WARN
>>>>>>>>> TaskSetManager: Lost
>>>>>>>>> > >>>>>>> task 0.0 in stage 0.0 (TID 0,
>>>>>>>>> RMANNIBUCAU,
>>>>>>>>> > >>>>>>> executor 0):
>>>>>>>>> java.lang.ClassCastException:
>>>>>>>>> > cannot
>>>>>>>>> > >>>>>>> assign instance of
>>>>>>>>> > >>>>>>>
>>>>>>>>> > scala.collection.immutable.List$SerializationProxy to
>>>>>>>>> > >>>>>>> fieldorg.apache.spark.rdd.RDD.org
>>>>>>>>> > <http://fieldorg.apache.spark.rdd.RDD.org>
>>>>>>>>> > >>>>>>> <http://fieldorg.apache.
>>>>>>>>> spark.rdd.rdd.org/>
>>>>>>>>> > >>>>>>> <http://org.apache.spark.rdd.RDD.org
>>>>>>>>> > >>>>>>>
>>>>>>>>> > <http://org.apache.spark.rdd.rdd.org/>>$apache$spark$
>>>>>>>>> rdd$RDD$$dependencies_
>>>>>>>>> > >>>>>>> of type scala.collection.Seq in
>>>>>>>>> instance of
>>>>>>>>> > >>>>>>> org.apache.spark.rdd.
>>>>>>>>> MapPartitionsRDD
>>>>>>>>> > >>>>>>> > at
>>>>>>>>> > >>>>>>>
>>>>>>>>> > java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(
>>>>>>>>> ObjectStreamClass.java:2233)
>>>>>>>>> > >>>>>>> >
>>>>>>>>> > >>>>>>> >
>>>>>>>>> > >>>>>>> > Also the issue Lukasz identified
>>>>>>>>> is important
>>>>>>>>> > >>>>>>> even if workarounds can be
>>>>>>>>> > >>>>>>> > put in place so +1 to fix it as
>>>>>>>>> well if
>>>>>>>>> > possible.
>>>>>>>>> > >>>>>>> >
>>>>>>>>> > >>>>>>> > Romain Manni-Bucau
>>>>>>>>> > >>>>>>> > @rmannibucau <https://twitter.com/
>>>>>>>>> rmannibucau>
>>>>>>>>> > >>>>>>> | Blog
>>>>>>>>> > >>>>>>> > <https://rmannibucau.metawerx.net/> |
>>>>>>>>> Old Blog
>>>>>>>>> > >>>>>>> > <http://rmannibucau.wordpress.com
>>>>>>>>> > >>>>>>> <http://rmannibucau.wordpress.com/>>
>>>>>>>>> | Github
>>>>>>>>> > >>>>>>> > <https://github.com/rmannibucau> |
>>>>>>>>> LinkedIn
>>>>>>>>> > >>>>>>> >
>>>>>>>>> > <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>>>>> > >>>>>>> >
>>>>>>>>> > >>>>>>>
>>>>>>>>> > <https://www.packtpub.com/application-development/java-
>>>>>>>>> ee-8-high-performance>
>>>>>>>>> > >>>>>>> >
>>>>>>>>> > >>>>>>> >
>>>>>>>>> > >>>>>>> > Le lun. 10 sept. 2018 à 20:48,
>>>>>>>>> Lukasz Cwik
>>>>>>>>> > >>>>>>> <lcwik@google.com <mailto:
>>>>>>>>> lcwik@google.com>
>>>>>>>>> > <mailto:lcwik@google.com <ma...@google.com>>
>>>>>>>>> > >>>>>>> > <mailto:lcwik@google.com
>>>>>>>>> > <ma...@google.com>
>>>>>>>>> > >>>>>>> <mailto:lcwik@google.com
>>>>>>>>> > <ma...@google.com>>>> a écrit :
>>>>>>>>> > >>>>>>> >
>>>>>>>>> > >>>>>>> > I found an issue where we are
>>>>>>>>> no longer
>>>>>>>>> > >>>>>>> packaging the pom.xml within
>>>>>>>>> > >>>>>>> > the artifact jars at
>>>>>>>>> > >>>>>>> META-INF/maven/groupId/artifactId.
>>>>>>>>> More details
>>>>>>>>> > >>>>>>> > in
>>>>>>>>> > >>>>>>> https://issues.apache.org/
>>>>>>>>> jira/browse/BEAM-5351.
>>>>>>>>> > >>>>>>> I wouldn't
>>>>>>>>> > >>>>>>> > consider this a blocker but it
>>>>>>>>> was an
>>>>>>>>> > easy fix
>>>>>>>>> > >>>>>>> > (https://github.com/apache/
>>>>>>>>> beam/pull/6358)
>>>>>>>>> > >>>>>>> and users may rely on the
>>>>>>>>> > >>>>>>> > pom.xml.
>>>>>>>>> > >>>>>>> >
>>>>>>>>> > >>>>>>> > Should we recut the release
>>>>>>>>> candidate to
>>>>>>>>> > >>>>>>> include this?
>>>>>>>>> > >>>>>>> >
>>>>>>>>> > >>>>>>> > On Mon, Sep 10, 2018 at 4:58 AM
>>>>>>>>> > >>>>>>> Jean-Baptiste Onofré
>>>>>>>>> > >>>>>>> > <jb@nanthrax.net
>>>>>>>>> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
>>>>>>>>> > <ma...@nanthrax.net>>
>>>>>>>>> > >>>>>>> <mailto:jb@nanthrax.net <mailto:
>>>>>>>>> jb@nanthrax.net>
>>>>>>>>> > >>>>>>> <mailto:jb@nanthrax.net
>>>>>>>>> > <ma...@nanthrax.net>>>> wrote:
>>>>>>>>> > >>>>>>> >
>>>>>>>>> > >>>>>>> > +1 (binding)
>>>>>>>>> > >>>>>>> >
>>>>>>>>> > >>>>>>> > Tested successfully on
>>>>>>>>> Beam Samples.
>>>>>>>>> > >>>>>>> >
>>>>>>>>> > >>>>>>> > Thanks !
>>>>>>>>> > >>>>>>> >
>>>>>>>>> > >>>>>>> > Regards
>>>>>>>>> > >>>>>>> > JB
>>>>>>>>> > >>>>>>> >
>>>>>>>>> > >>>>>>> > On 07/09/2018 23:56,
>>>>>>>>> Charles Chen
>>>>>>>>> > wrote:
>>>>>>>>> > >>>>>>> > > Hi everyone,
>>>>>>>>> > >>>>>>> > >
>>>>>>>>> > >>>>>>> > > Please review and vote
>>>>>>>>> on the
>>>>>>>>> > >>>>>>> release candidate #1 for the
>>>>>>>>> > >>>>>>> > version
>>>>>>>>> > >>>>>>> > > 2.7.0, as follows:
>>>>>>>>> > >>>>>>> > > [ ] +1, Approve the
>>>>>>>>> release
>>>>>>>>> > >>>>>>> > > [ ] -1, Do not approve
>>>>>>>>> the release
>>>>>>>>> > >>>>>>> (please provide specific
>>>>>>>>> > >>>>>>> > comments)
>>>>>>>>> > >>>>>>> > >
>>>>>>>>> > >>>>>>> > > The complete staging
>>>>>>>>> area is
>>>>>>>>> > >>>>>>> available for your review, which
>>>>>>>>> > >>>>>>> > includes:
>>>>>>>>> > >>>>>>> > > * JIRA release notes
>>>>>>>>> [1],
>>>>>>>>> > >>>>>>> > > * the official Apache
>>>>>>>>> source
>>>>>>>>> > release
>>>>>>>>> > >>>>>>> to be deployed to
>>>>>>>>> > >>>>>>> > dist.apache.org
>>>>>>>>> > <http://dist.apache.org>
>>>>>>>>> > >>>>>>> <http://dist.apache.org/>
>>>>>>>>> > <http://dist.apache.org
>>>>>>>>> > >>>>>>> <http://dist.apache.org/>>
>>>>>>>>> > >>>>>>> > > <http://dist.apache.org
>>>>>>>>> > >>>>>>> <http://dist.apache.org/>> [2],
>>>>>>>>> which is signed
>>>>>>>>> > >>>>>>> with the key with
>>>>>>>>> > >>>>>>> > > fingerprint
>>>>>>>>> 45C60AAAD115F560 [3],
>>>>>>>>> > >>>>>>> > > * all artifacts to be
>>>>>>>>> deployed to
>>>>>>>>> > >>>>>>> the Maven Central
>>>>>>>>> > >>>>>>> > Repository [4],
>>>>>>>>> > >>>>>>> > > * source code tag
>>>>>>>>> "v2.7.0-RC1" [5],
>>>>>>>>> > >>>>>>> > > * website pull request
>>>>>>>>> listing the
>>>>>>>>> > >>>>>>> release and publishing the API
>>>>>>>>> > >>>>>>> > > reference manual [6].
>>>>>>>>> > >>>>>>> > > * Java artifacts were
>>>>>>>>> built with
>>>>>>>>> > >>>>>>> Gradle 4.8 and OpenJDK
>>>>>>>>> > >>>>>>> > >
>>>>>>>>> 1.8.0_181-8u181-b13-1~deb9u1-b13.
>>>>>>>>> > >>>>>>> > > * Python artifacts are
>>>>>>>>> deployed
>>>>>>>>> > >>>>>>> along with the source release
>>>>>>>>> > >>>>>>> > to the
>>>>>>>>> > >>>>>>> > > dist.apache.org
>>>>>>>>> > <http://dist.apache.org>
>>>>>>>>> > >>>>>>> <http://dist.apache.org/>
>>>>>>>>> > <http://dist.apache.org
>>>>>>>>> > >>>>>>> <http://dist.apache.org/>>
>>>>>>>>> > >>>>>>> > <http://dist.apache.org
>>>>>>>>> > >>>>>>> <http://dist.apache.org/>> [2].
>>>>>>>>> > >>>>>>> > >
>>>>>>>>> > >>>>>>> > > The vote will be open
>>>>>>>>> for at least
>>>>>>>>> > >>>>>>> 72 hours. It is adopted by
>>>>>>>>> > >>>>>>> > majority
>>>>>>>>> > >>>>>>> > > approval, with at least
>>>>>>>>> 3 PMC
>>>>>>>>> > >>>>>>> affirmative votes.
>>>>>>>>> > >>>>>>> > >
>>>>>>>>> > >>>>>>> > > Thanks,
>>>>>>>>> > >>>>>>> > > Charles
>>>>>>>>> > >>>>>>> > >
>>>>>>>>> > >>>>>>> > > [1]
>>>>>>>>> > >>>>>>> > >
>>>>>>>>> > >>>>>>> >
>>>>>>>>> > >>>>>>>
>>>>>>>>> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>>>>>>>>> projectId=12319527&version=12343654
>>>>>>>>> > >>>>>>> > > [2]
>>>>>>>>> > >>>>>>>
>>>>>>>>> > https://dist.apache.org/repos/dist/dev/beam/2.7.0
>>>>>>>>> > >>>>>>> > > [3]
>>>>>>>>> > >>>>>>> https://dist.apache.org/
>>>>>>>>> repos/dist/dev/beam/KEYS
>>>>>>>>> > >>>>>>> > > [4]
>>>>>>>>> > >>>>>>> >
>>>>>>>>> > >>>>>>>
>>>>>>>>> > https://repository.apache.org/content/repositories/
>>>>>>>>> orgapachebeam-1046/
>>>>>>>>> > >>>>>>> > > [5]
>>>>>>>>> > >>>>>>> https://github.com/apache/
>>>>>>>>> beam/tree/v2.7.0-RC1
>>>>>>>>> > >>>>>>> > > [6]
>>>>>>>>> > >>>>>>> https://github.com/apache/
>>>>>>>>> beam-site/pull/549
>>>>>>>>> > >>>>>>> >
>>>>>>>>> > >>>>>>> > --
>>>>>>>>> > >>>>>>> > Jean-Baptiste Onofré
>>>>>>>>> > >>>>>>> > jbonofre@apache.org
>>>>>>>>> > <ma...@apache.org>
>>>>>>>>> > >>>>>>> <mailto:jbonofre@apache.org
>>>>>>>>> > <ma...@apache.org>>
>>>>>>>>> > >>>>>>> <mailto:jbonofre@apache.org
>>>>>>>>> > <ma...@apache.org>
>>>>>>>>> > >>>>>>> <mailto:jbonofre@apache.org
>>>>>>>>> > <ma...@apache.org>>>
>>>>>>>>> > >>>>>>> > http://blog.nanthrax.net
>>>>>>>>> > >>>>>>> <http://blog.nanthrax.net/>
>>>>>>>>> > >>>>>>> > Talend -
>>>>>>>>> http://www.talend.com
>>>>>>>>> > >>>>>>> <http://www.talend.com/>
>>>>>>>>> > >>>>>>> >
>>>>>>>>> > >>>>
>>>>>>>>> > >
>>>>>>>>> >
>>>>>>>>> > --
>>>>>>>>> > Jean-Baptiste Onofré
>>>>>>>>> > jbonofre@apache.org <ma...@apache.org>
>>>>>>>>> > http://blog.nanthrax.net
>>>>>>>>> > Talend - http://www.talend.com
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Jean-Baptiste Onofré
>>>>>>>>> jbonofre@apache.org
>>>>>>>>> http://blog.nanthrax.net
>>>>>>>>> Talend - http://www.talend.com
>>>>>>>>>
>>>>>>>>
>>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Thomas Weise <th...@apache.org>.
That's not the same for a user though. 2.7.1 would be a patch compatible
release that only fixes bugs. 2.8.0 adds new features and potentially also
new issues..
On Thu, Sep 20, 2018 at 3:16 PM Ahmet Altay <al...@google.com> wrote:
> +1 to Thomas's suggestion. Instead of 2.7.1 we can follow up with 2.8.0
> though. 2.8.0 has a release branch cut date of 10/10 according to our
> release calendar.
>
> On Thu, Sep 20, 2018 at 2:47 PM, Connell O'Callaghan <co...@google.com>
> wrote:
>
>> +1 to Thomas's suggestion - if Charles or others cannot reproduce.
>>
>> On Thu, Sep 20, 2018 at 2:40 PM Thomas Weise <th...@apache.org> wrote:
>>
>>> We can also consider releasing 2.7.0 and then follow up with 2.7.1 if
>>> the problem can be reproduced and requires a fix. Just food for thought :)
>>>
>>>
>>> On Thu, Sep 20, 2018 at 2:13 PM Charles Chen <cc...@google.com> wrote:
>>>
>>>> My mistake, it looks like the correct beam staging repository (
>>>> https://repository.apache.org/content/repositories/orgapachebeam-1046/)
>>>> is specified in your pom file.
>>>>
>>>> On Thu, Sep 20, 2018 at 2:10 PM Charles Chen <cc...@google.com> wrote:
>>>>
>>>>> Hey Romain and JB, do you have any progress on this? One thing I
>>>>> would like to point out is that 2.7.0 isn't yet pushed to Maven Central, so
>>>>> referring to it by version is not expected to work (and it looks like this
>>>>> is what is done in your repo:
>>>>> https://github.com/rmannibucau/beam-2.7.0-fails). Luke indicated
>>>>> above that he doesn't see any dependency changes. Can you isolate and
>>>>> reproduce this problem so that we can develop a fix, if necessary? I would
>>>>> like to proceed with an RC2 as soon as possible.
>>>>>
>>>>> On Wed, Sep 19, 2018 at 6:37 AM Romain Manni-Bucau <
>>>>> rmannibucau@gmail.com> wrote:
>>>>>
>>>>>> Quick update on the spark issue: I didn't get enough time to identify
>>>>>> it clearly but managed to have a passing run of my test changing a bunch of
>>>>>> versions.
>>>>>> I suspect my code triggers some class conflicting between spark and
>>>>>> my shade leading to a serialization issue. I didn't test userClassPathFirst
>>>>>> option of spark but it can be an interesting thing to enable in beam runner.
>>>>>> However it is still very confusing to have it not running just
>>>>>> upgrading beam version and the spark error is very hard to understand.
>>>>>>
>>>>>> Romain Manni-Bucau
>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>>> <http://rmannibucau.wordpress.com> | Github
>>>>>> <https://github.com/rmannibucau> | LinkedIn
>>>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>>>
>>>>>>
>>>>>> Le mar. 18 sept. 2018 à 20:17, Lukasz Cwik <lc...@google.com> a
>>>>>> écrit :
>>>>>>
>>>>>>> Romain hinted that this was a dependency issue but when comparing
>>>>>>> the two dependency trees I don't get much of a difference:
>>>>>>>
>>>>>>> lcwik@lcwik0: ~$ diff /tmp/260 /tmp/270
>>>>>>> < [INFO] +- org.apache.beam:beam-runners-spark:jar:2.6.0:compile
>>>>>>> < [INFO] | +- org.apache.beam:beam-model-pipeline:jar:2.6.0:compile
>>>>>>> ---
>>>>>>> > [INFO] +- org.apache.beam:beam-runners-spark:jar:2.7.0:compile
>>>>>>> > [INFO] | +- org.apache.beam:beam-model-pipeline:jar:2.7.0:compile
>>>>>>> 5c6
>>>>>>> < [INFO] | +- org.apache.beam:beam-sdks-java-core:jar:2.6.0:compile
>>>>>>> ---
>>>>>>> > [INFO] | +- org.apache.beam:beam-sdks-java-core:jar:2.7.0:compile
>>>>>>> 14,18c15,19
>>>>>>> < [INFO] | | \- org.tukaani:xz:jar:1.5:compile
>>>>>>> < [INFO] | +-
>>>>>>> org.apache.beam:beam-runners-core-construction-java:jar:2.6.0:compile
>>>>>>> < [INFO] | | \-
>>>>>>> org.apache.beam:beam-model-job-management:jar:2.6.0:compile
>>>>>>> < [INFO] | +-
>>>>>>> org.apache.beam:beam-runners-core-java:jar:2.6.0:compile
>>>>>>> < [INFO] | | \-
>>>>>>> org.apache.beam:beam-model-fn-execution:jar:2.6.0:compile
>>>>>>> ---
>>>>>>> > [INFO] | | \- org.tukaani:xz:jar:1.8:compile
>>>>>>> > [INFO] | +-
>>>>>>> org.apache.beam:beam-runners-core-construction-java:jar:2.7.0:compile
>>>>>>> > [INFO] | | \-
>>>>>>> org.apache.beam:beam-model-job-management:jar:2.7.0:compile
>>>>>>> > [INFO] | +-
>>>>>>> org.apache.beam:beam-runners-core-java:jar:2.7.0:compile
>>>>>>> > [INFO] | | \-
>>>>>>> org.apache.beam:beam-model-fn-execution:jar:2.7.0:compile
>>>>>>>
>>>>>>> Other then Beam package changes, the only other change is xz which I
>>>>>>> don't believe could be causing the issue.
>>>>>>>
>>>>>>> On Tue, Sep 18, 2018 at 8:38 AM Jean-Baptiste Onofré <
>>>>>>> jb@nanthrax.net> wrote:
>>>>>>>
>>>>>>>> Thanks, let me take a look.
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> JB
>>>>>>>>
>>>>>>>> On 18/09/2018 17:36, Romain Manni-Bucau wrote:
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > Le mar. 18 sept. 2018 à 16:44, Jean-Baptiste Onofré <
>>>>>>>> jb@nanthrax.net
>>>>>>>> > <ma...@nanthrax.net>> a écrit :
>>>>>>>> >
>>>>>>>> > Hi,
>>>>>>>> >
>>>>>>>> > I don't have the issue ;)
>>>>>>>> >
>>>>>>>> > As said in my vote, I tested 2.7.0 RC1 on beam-samples with
>>>>>>>> Spark
>>>>>>>> > without problem.
>>>>>>>> >
>>>>>>>> > I don't reproduce Romain issue as well.
>>>>>>>> >
>>>>>>>> > @Romain can you provide some details to reproduce the issue ?
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > Sure, you can use this
>>>>>>>> > reproducer: https://github.com/rmannibucau/beam-2.7.0-fails
>>>>>>>> > It shows that it suceeds on 2.6 and fails on 2.7.
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > Regards
>>>>>>>> > JB
>>>>>>>> >
>>>>>>>> > On 17/09/2018 19:17, Charles Chen wrote:
>>>>>>>> > > Luke, Maximillian, Raghu, can you please propose
>>>>>>>> cherry-pick PRs
>>>>>>>> > to the
>>>>>>>> > > release-2.7.0 for your issues and add me as a reviewer
>>>>>>>> > (@charlesccychen)?
>>>>>>>> > >
>>>>>>>> > > Romain, JB: is there any way I can help with debugging the
>>>>>>>> issue
>>>>>>>> > you're
>>>>>>>> > > facing so we can unblock the release?
>>>>>>>> > >
>>>>>>>> > > On Fri, Sep 14, 2018 at 1:49 PM Raghu Angadi <
>>>>>>>> rangadi@google.com
>>>>>>>> > <ma...@google.com>
>>>>>>>> > > <mailto:rangadi@google.com <ma...@google.com>>>
>>>>>>>> wrote:
>>>>>>>> > >
>>>>>>>> > > I would like propose one more cherrypick for RC2
>>>>>>>> > > : https://github.com/apache/beam/pull/6391
>>>>>>>> > > This is a KafkaIO bug fix. Once a user hits this bug,
>>>>>>>> there is no
>>>>>>>> > > easy work around for them, especially on Dataflow. Only
>>>>>>>> work
>>>>>>>> > around
>>>>>>>> > > in Dataflow is to restart or reload the job.
>>>>>>>> > >
>>>>>>>> > > The fix itself fairly safe and is tested.
>>>>>>>> > > Raghu.
>>>>>>>> > >
>>>>>>>> > > On Fri, Sep 14, 2018 at 12:52 AM Alexey Romanenko
>>>>>>>> > > <aromanenko.dev@gmail.com <mailto:
>>>>>>>> aromanenko.dev@gmail.com>
>>>>>>>> > <mailto:aromanenko.dev@gmail.com <mailto:
>>>>>>>> aromanenko.dev@gmail.com>>>
>>>>>>>> > wrote:
>>>>>>>> > >
>>>>>>>> > > Perhaps it could help, but I run simple WordCount
>>>>>>>> (built with
>>>>>>>> > > Beam 2.7) on YARN/Spark (HDP Sandbox) cluster and it
>>>>>>>> > worked fine
>>>>>>>> > > for me.
>>>>>>>> > >
>>>>>>>> > >> On 14 Sep 2018, at 06:56, Romain Manni-Bucau
>>>>>>>> > >> <rmannibucau@gmail.com <mailto:
>>>>>>>> rmannibucau@gmail.com>
>>>>>>>> > <mailto:rmannibucau@gmail.com <ma...@gmail.com>>>
>>>>>>>> wrote:
>>>>>>>> > >>
>>>>>>>> > >> Hi Charles,
>>>>>>>> > >>
>>>>>>>> > >> I didn't get enough time to check deeply but it is
>>>>>>>> clearly a
>>>>>>>> > >> dependency issue and it is not in beam spark runner
>>>>>>>> > itself but
>>>>>>>> > >> in another transitive module of beam. It does not
>>>>>>>> happen in
>>>>>>>> > >> existing spark test cause none of them are in a
>>>>>>>> cluster (even
>>>>>>>> > >> just with 1 worker) but this seems to be a
>>>>>>>> regression since
>>>>>>>> > >> 2.6 works OOTB.
>>>>>>>> > >>
>>>>>>>> > >> Romain Manni-Bucau
>>>>>>>> > >> @rmannibucau <https://twitter.com/rmannibucau> |
>>>>>>>> Blog
>>>>>>>> > >> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>>>>> > >> <http://rmannibucau.wordpress.com/> | Github
>>>>>>>> > >> <https://github.com/rmannibucau> | LinkedIn
>>>>>>>> > >> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>>>> > >>
>>>>>>>> > <
>>>>>>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>>>>>> >
>>>>>>>> > >>
>>>>>>>> > >>
>>>>>>>> > >> Le jeu. 13 sept. 2018 à 22:15, Charles Chen
>>>>>>>> > <ccy@google.com <ma...@google.com>
>>>>>>>> > >> <mailto:ccy@google.com <ma...@google.com>>>
>>>>>>>> a écrit :
>>>>>>>> > >>
>>>>>>>> > >> Romain and JB, can you please add the results
>>>>>>>> of your
>>>>>>>> > >> investigations into the errors you've seen
>>>>>>>> above? Given
>>>>>>>> > >> that the existing SparkRunner tests pass for
>>>>>>>> this RC, and
>>>>>>>> > >> that the integration test you ran is in
>>>>>>>> another repo that
>>>>>>>> > >> is not continuously tested with Beam, it is
>>>>>>>> not clear how
>>>>>>>> > >> we should move forward and whether this is a
>>>>>>>> blocking
>>>>>>>> > >> issue, unless we can find a root cause in Beam.
>>>>>>>> > >>
>>>>>>>> > >> On Wed, Sep 12, 2018 at 2:08 AM Etienne
>>>>>>>> Chauchot
>>>>>>>> > >> <echauchot@apache.org <mailto:
>>>>>>>> echauchot@apache.org>
>>>>>>>> > <mailto:echauchot@apache.org <ma...@apache.org>>>
>>>>>>>> wrote:
>>>>>>>> > >>
>>>>>>>> > >> Hi all,
>>>>>>>> > >>
>>>>>>>> > >> on a performance and functional regression
>>>>>>>> stand
>>>>>>>> > point
>>>>>>>> > >> I see no regression:
>>>>>>>> > >>
>>>>>>>> > >> I looked at nexmark graphs "output
>>>>>>>> pcollection size"
>>>>>>>> > >> and "execution time" around release cut
>>>>>>>> date on
>>>>>>>> > >> dataflow, spark, flink and direct runner
>>>>>>>> in batch and
>>>>>>>> > >> streaming modes. There seems to be no
>>>>>>>> regression.
>>>>>>>> > >>
>>>>>>>> > >> Etienne
>>>>>>>> > >>
>>>>>>>> > >> Le mardi 11 septembre 2018 à 12:25 -0700,
>>>>>>>> Charles
>>>>>>>> > Chen
>>>>>>>> > >> a écrit :
>>>>>>>> > >>> The SparkRunner validation test
>>>>>>>> > >>>
>>>>>>>> > (here:
>>>>>>>> https://beam.apache.org/contribute/release-guide/#run-validation-tests
>>>>>>>> )
>>>>>>>> > >>> passes on my machine. It looks like we
>>>>>>>> are likely
>>>>>>>> > >>> missing test coverage where Romain is
>>>>>>>> hitting
>>>>>>>> > issues.
>>>>>>>> > >>>
>>>>>>>> > >>> On Tue, Sep 11, 2018 at 12:15 PM Ahmet
>>>>>>>> Altay
>>>>>>>> > >>> <altay@google.com <mailto:
>>>>>>>> altay@google.com>
>>>>>>>> > <mailto:altay@google.com <ma...@google.com>>> wrote:
>>>>>>>> > >>>> Could anyone else help with looking at
>>>>>>>> these issues
>>>>>>>> > >>>> earlier?
>>>>>>>> > >>>>
>>>>>>>> > >>>> On Tue, Sep 11, 2018 at 12:03 PM, Romain
>>>>>>>> > Manni-Bucau
>>>>>>>> > >>>> <rmannibucau@gmail.com
>>>>>>>> > <ma...@gmail.com>
>>>>>>>> > >>>> <mailto:rmannibucau@gmail.com
>>>>>>>> > <ma...@gmail.com>>> wrote:
>>>>>>>> > >>>>> Im running this main [1] through this
>>>>>>>> IT [2]. Was
>>>>>>>> > >>>>> working fine since ~1 year but 2.7.0
>>>>>>>> broke it.
>>>>>>>> > >>>>> Didnt investigate more but can have a
>>>>>>>> look later
>>>>>>>> > >>>>> this month if it helps.
>>>>>>>> > >>>>>
>>>>>>>> > >>>>>
>>>>>>>> > [1]
>>>>>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
>>>>>>>> > >>>>>
>>>>>>>> > [2]
>>>>>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
>>>>>>>> > >>>>>
>>>>>>>> > >>>>> Le mar. 11 sept. 2018 20:54, Charles
>>>>>>>> Chen
>>>>>>>> > >>>>> <ccy@google.com <ma...@google.com>
>>>>>>>> > <mailto:ccy@google.com <ma...@google.com>>> a écrit :
>>>>>>>> > >>>>>> Romain: can you give more details on
>>>>>>>> the failure
>>>>>>>> > >>>>>> you're encountering, i.e. how you are
>>>>>>>> performing
>>>>>>>> > >>>>>> this validation?
>>>>>>>> > >>>>>>
>>>>>>>> > >>>>>> On Tue, Sep 11, 2018 at 9:36 AM
>>>>>>>> Jean-Baptiste
>>>>>>>> > >>>>>> Onofré <jb@nanthrax.net
>>>>>>>> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
>>>>>>>> > <ma...@nanthrax.net>>>
>>>>>>>> > >>>>>> wrote:
>>>>>>>> > >>>>>>> Hi,
>>>>>>>> > >>>>>>>
>>>>>>>> > >>>>>>> weird, I didn't have it on Beam
>>>>>>>> samples. Let me
>>>>>>>> > >>>>>>> try to reproduce and I
>>>>>>>> > >>>>>>> will create the Jira.
>>>>>>>> > >>>>>>>
>>>>>>>> > >>>>>>> Regards
>>>>>>>> > >>>>>>> JB
>>>>>>>> > >>>>>>>
>>>>>>>> > >>>>>>> On 11/09/2018 11:44, Romain
>>>>>>>> Manni-Bucau wrote:
>>>>>>>> > >>>>>>> > -1, seems spark integration is
>>>>>>>> broken (tested
>>>>>>>> > >>>>>>> with spark 2.3.1 and 2.2.1):
>>>>>>>> > >>>>>>> >
>>>>>>>> > >>>>>>> > 18/09/11 11:33:29 WARN
>>>>>>>> TaskSetManager: Lost
>>>>>>>> > >>>>>>> task 0.0 in stage 0.0 (TID 0,
>>>>>>>> RMANNIBUCAU,
>>>>>>>> > >>>>>>> executor 0):
>>>>>>>> java.lang.ClassCastException:
>>>>>>>> > cannot
>>>>>>>> > >>>>>>> assign instance of
>>>>>>>> > >>>>>>>
>>>>>>>> > scala.collection.immutable.List$SerializationProxy to
>>>>>>>> > >>>>>>> fieldorg.apache.spark.rdd.RDD.org
>>>>>>>> > <http://fieldorg.apache.spark.rdd.RDD.org>
>>>>>>>> > >>>>>>> <
>>>>>>>> http://fieldorg.apache.spark.rdd.rdd.org/>
>>>>>>>> > >>>>>>> <http://org.apache.spark.rdd.RDD.org
>>>>>>>> > >>>>>>>
>>>>>>>> > <http://org.apache.spark.rdd.rdd.org/
>>>>>>>> >>$apache$spark$rdd$RDD$$dependencies_
>>>>>>>> > >>>>>>> of type scala.collection.Seq in
>>>>>>>> instance of
>>>>>>>> > >>>>>>> org.apache.spark.rdd.MapPartitionsRDD
>>>>>>>> > >>>>>>> > at
>>>>>>>> > >>>>>>>
>>>>>>>> >
>>>>>>>> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>>>>>>>> > >>>>>>> >
>>>>>>>> > >>>>>>> >
>>>>>>>> > >>>>>>> > Also the issue Lukasz identified is
>>>>>>>> important
>>>>>>>> > >>>>>>> even if workarounds can be
>>>>>>>> > >>>>>>> > put in place so +1 to fix it as
>>>>>>>> well if
>>>>>>>> > possible.
>>>>>>>> > >>>>>>> >
>>>>>>>> > >>>>>>> > Romain Manni-Bucau
>>>>>>>> > >>>>>>> > @rmannibucau <
>>>>>>>> https://twitter.com/rmannibucau>
>>>>>>>> > >>>>>>> | Blog
>>>>>>>> > >>>>>>> > <https://rmannibucau.metawerx.net/> |
>>>>>>>> Old Blog
>>>>>>>> > >>>>>>> > <http://rmannibucau.wordpress.com
>>>>>>>> > >>>>>>> <http://rmannibucau.wordpress.com/>>
>>>>>>>> | Github
>>>>>>>> > >>>>>>> > <https://github.com/rmannibucau> |
>>>>>>>> LinkedIn
>>>>>>>> > >>>>>>> >
>>>>>>>> > <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>>>> > >>>>>>> >
>>>>>>>> > >>>>>>>
>>>>>>>> > <
>>>>>>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>>>>>> >
>>>>>>>> > >>>>>>> >
>>>>>>>> > >>>>>>> >
>>>>>>>> > >>>>>>> > Le lun. 10 sept. 2018 à 20:48,
>>>>>>>> Lukasz Cwik
>>>>>>>> > >>>>>>> <lcwik@google.com <mailto:
>>>>>>>> lcwik@google.com>
>>>>>>>> > <mailto:lcwik@google.com <ma...@google.com>>
>>>>>>>> > >>>>>>> > <mailto:lcwik@google.com
>>>>>>>> > <ma...@google.com>
>>>>>>>> > >>>>>>> <mailto:lcwik@google.com
>>>>>>>> > <ma...@google.com>>>> a écrit :
>>>>>>>> > >>>>>>> >
>>>>>>>> > >>>>>>> > I found an issue where we are
>>>>>>>> no longer
>>>>>>>> > >>>>>>> packaging the pom.xml within
>>>>>>>> > >>>>>>> > the artifact jars at
>>>>>>>> > >>>>>>> META-INF/maven/groupId/artifactId.
>>>>>>>> More details
>>>>>>>> > >>>>>>> > in
>>>>>>>> > >>>>>>>
>>>>>>>> https://issues.apache.org/jira/browse/BEAM-5351.
>>>>>>>> > >>>>>>> I wouldn't
>>>>>>>> > >>>>>>> > consider this a blocker but it
>>>>>>>> was an
>>>>>>>> > easy fix
>>>>>>>> > >>>>>>> > (
>>>>>>>> https://github.com/apache/beam/pull/6358)
>>>>>>>> > >>>>>>> and users may rely on the
>>>>>>>> > >>>>>>> > pom.xml.
>>>>>>>> > >>>>>>> >
>>>>>>>> > >>>>>>> > Should we recut the release
>>>>>>>> candidate to
>>>>>>>> > >>>>>>> include this?
>>>>>>>> > >>>>>>> >
>>>>>>>> > >>>>>>> > On Mon, Sep 10, 2018 at 4:58 AM
>>>>>>>> > >>>>>>> Jean-Baptiste Onofré
>>>>>>>> > >>>>>>> > <jb@nanthrax.net
>>>>>>>> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
>>>>>>>> > <ma...@nanthrax.net>>
>>>>>>>> > >>>>>>> <mailto:jb@nanthrax.net <mailto:
>>>>>>>> jb@nanthrax.net>
>>>>>>>> > >>>>>>> <mailto:jb@nanthrax.net
>>>>>>>> > <ma...@nanthrax.net>>>> wrote:
>>>>>>>> > >>>>>>> >
>>>>>>>> > >>>>>>> > +1 (binding)
>>>>>>>> > >>>>>>> >
>>>>>>>> > >>>>>>> > Tested successfully on Beam
>>>>>>>> Samples.
>>>>>>>> > >>>>>>> >
>>>>>>>> > >>>>>>> > Thanks !
>>>>>>>> > >>>>>>> >
>>>>>>>> > >>>>>>> > Regards
>>>>>>>> > >>>>>>> > JB
>>>>>>>> > >>>>>>> >
>>>>>>>> > >>>>>>> > On 07/09/2018 23:56,
>>>>>>>> Charles Chen
>>>>>>>> > wrote:
>>>>>>>> > >>>>>>> > > Hi everyone,
>>>>>>>> > >>>>>>> > >
>>>>>>>> > >>>>>>> > > Please review and vote
>>>>>>>> on the
>>>>>>>> > >>>>>>> release candidate #1 for the
>>>>>>>> > >>>>>>> > version
>>>>>>>> > >>>>>>> > > 2.7.0, as follows:
>>>>>>>> > >>>>>>> > > [ ] +1, Approve the
>>>>>>>> release
>>>>>>>> > >>>>>>> > > [ ] -1, Do not approve
>>>>>>>> the release
>>>>>>>> > >>>>>>> (please provide specific
>>>>>>>> > >>>>>>> > comments)
>>>>>>>> > >>>>>>> > >
>>>>>>>> > >>>>>>> > > The complete staging
>>>>>>>> area is
>>>>>>>> > >>>>>>> available for your review, which
>>>>>>>> > >>>>>>> > includes:
>>>>>>>> > >>>>>>> > > * JIRA release notes [1],
>>>>>>>> > >>>>>>> > > * the official Apache
>>>>>>>> source
>>>>>>>> > release
>>>>>>>> > >>>>>>> to be deployed to
>>>>>>>> > >>>>>>> > dist.apache.org
>>>>>>>> > <http://dist.apache.org>
>>>>>>>> > >>>>>>> <http://dist.apache.org/>
>>>>>>>> > <http://dist.apache.org
>>>>>>>> > >>>>>>> <http://dist.apache.org/>>
>>>>>>>> > >>>>>>> > > <http://dist.apache.org
>>>>>>>> > >>>>>>> <http://dist.apache.org/>> [2],
>>>>>>>> which is signed
>>>>>>>> > >>>>>>> with the key with
>>>>>>>> > >>>>>>> > > fingerprint
>>>>>>>> 45C60AAAD115F560 [3],
>>>>>>>> > >>>>>>> > > * all artifacts to be
>>>>>>>> deployed to
>>>>>>>> > >>>>>>> the Maven Central
>>>>>>>> > >>>>>>> > Repository [4],
>>>>>>>> > >>>>>>> > > * source code tag
>>>>>>>> "v2.7.0-RC1" [5],
>>>>>>>> > >>>>>>> > > * website pull request
>>>>>>>> listing the
>>>>>>>> > >>>>>>> release and publishing the API
>>>>>>>> > >>>>>>> > > reference manual [6].
>>>>>>>> > >>>>>>> > > * Java artifacts were
>>>>>>>> built with
>>>>>>>> > >>>>>>> Gradle 4.8 and OpenJDK
>>>>>>>> > >>>>>>> > >
>>>>>>>> 1.8.0_181-8u181-b13-1~deb9u1-b13.
>>>>>>>> > >>>>>>> > > * Python artifacts are
>>>>>>>> deployed
>>>>>>>> > >>>>>>> along with the source release
>>>>>>>> > >>>>>>> > to the
>>>>>>>> > >>>>>>> > > dist.apache.org
>>>>>>>> > <http://dist.apache.org>
>>>>>>>> > >>>>>>> <http://dist.apache.org/>
>>>>>>>> > <http://dist.apache.org
>>>>>>>> > >>>>>>> <http://dist.apache.org/>>
>>>>>>>> > >>>>>>> > <http://dist.apache.org
>>>>>>>> > >>>>>>> <http://dist.apache.org/>> [2].
>>>>>>>> > >>>>>>> > >
>>>>>>>> > >>>>>>> > > The vote will be open
>>>>>>>> for at least
>>>>>>>> > >>>>>>> 72 hours. It is adopted by
>>>>>>>> > >>>>>>> > majority
>>>>>>>> > >>>>>>> > > approval, with at least
>>>>>>>> 3 PMC
>>>>>>>> > >>>>>>> affirmative votes.
>>>>>>>> > >>>>>>> > >
>>>>>>>> > >>>>>>> > > Thanks,
>>>>>>>> > >>>>>>> > > Charles
>>>>>>>> > >>>>>>> > >
>>>>>>>> > >>>>>>> > > [1]
>>>>>>>> > >>>>>>> > >
>>>>>>>> > >>>>>>> >
>>>>>>>> > >>>>>>>
>>>>>>>> >
>>>>>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>>>>>>>> > >>>>>>> > > [2]
>>>>>>>> > >>>>>>>
>>>>>>>> > https://dist.apache.org/repos/dist/dev/beam/2.7.0
>>>>>>>> > >>>>>>> > > [3]
>>>>>>>> > >>>>>>>
>>>>>>>> https://dist.apache.org/repos/dist/dev/beam/KEYS
>>>>>>>> > >>>>>>> > > [4]
>>>>>>>> > >>>>>>> >
>>>>>>>> > >>>>>>>
>>>>>>>> >
>>>>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1046/
>>>>>>>> > >>>>>>> > > [5]
>>>>>>>> > >>>>>>>
>>>>>>>> https://github.com/apache/beam/tree/v2.7.0-RC1
>>>>>>>> > >>>>>>> > > [6]
>>>>>>>> > >>>>>>>
>>>>>>>> https://github.com/apache/beam-site/pull/549
>>>>>>>> > >>>>>>> >
>>>>>>>> > >>>>>>> > --
>>>>>>>> > >>>>>>> > Jean-Baptiste Onofré
>>>>>>>> > >>>>>>> > jbonofre@apache.org
>>>>>>>> > <ma...@apache.org>
>>>>>>>> > >>>>>>> <mailto:jbonofre@apache.org
>>>>>>>> > <ma...@apache.org>>
>>>>>>>> > >>>>>>> <mailto:jbonofre@apache.org
>>>>>>>> > <ma...@apache.org>
>>>>>>>> > >>>>>>> <mailto:jbonofre@apache.org
>>>>>>>> > <ma...@apache.org>>>
>>>>>>>> > >>>>>>> > http://blog.nanthrax.net
>>>>>>>> > >>>>>>> <http://blog.nanthrax.net/>
>>>>>>>> > >>>>>>> > Talend -
>>>>>>>> http://www.talend.com
>>>>>>>> > >>>>>>> <http://www.talend.com/>
>>>>>>>> > >>>>>>> >
>>>>>>>> > >>>>
>>>>>>>> > >
>>>>>>>> >
>>>>>>>> > --
>>>>>>>> > Jean-Baptiste Onofré
>>>>>>>> > jbonofre@apache.org <ma...@apache.org>
>>>>>>>> > http://blog.nanthrax.net
>>>>>>>> > Talend - http://www.talend.com
>>>>>>>> >
>>>>>>>>
>>>>>>>> --
>>>>>>>> Jean-Baptiste Onofré
>>>>>>>> jbonofre@apache.org
>>>>>>>> http://blog.nanthrax.net
>>>>>>>> Talend - http://www.talend.com
>>>>>>>>
>>>>>>>
>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Ahmet Altay <al...@google.com>.
+1 to Thomas's suggestion. Instead of 2.7.1 we can follow up with 2.8.0
though. 2.8.0 has a release branch cut date of 10/10 according to our
release calendar.
On Thu, Sep 20, 2018 at 2:47 PM, Connell O'Callaghan <co...@google.com>
wrote:
> +1 to Thomas's suggestion - if Charles or others cannot reproduce.
>
> On Thu, Sep 20, 2018 at 2:40 PM Thomas Weise <th...@apache.org> wrote:
>
>> We can also consider releasing 2.7.0 and then follow up with 2.7.1 if the
>> problem can be reproduced and requires a fix. Just food for thought :)
>>
>>
>> On Thu, Sep 20, 2018 at 2:13 PM Charles Chen <cc...@google.com> wrote:
>>
>>> My mistake, it looks like the correct beam staging repository (
>>> https://repository.apache.org/content/repositories/orgapachebeam-1046/)
>>> is specified in your pom file.
>>>
>>> On Thu, Sep 20, 2018 at 2:10 PM Charles Chen <cc...@google.com> wrote:
>>>
>>>> Hey Romain and JB, do you have any progress on this? One thing I would
>>>> like to point out is that 2.7.0 isn't yet pushed to Maven Central, so
>>>> referring to it by version is not expected to work (and it looks like this
>>>> is what is done in your repo: https://github.com/
>>>> rmannibucau/beam-2.7.0-fails). Luke indicated above that he doesn't
>>>> see any dependency changes. Can you isolate and reproduce this problem so
>>>> that we can develop a fix, if necessary? I would like to proceed with an
>>>> RC2 as soon as possible.
>>>>
>>>> On Wed, Sep 19, 2018 at 6:37 AM Romain Manni-Bucau <
>>>> rmannibucau@gmail.com> wrote:
>>>>
>>>>> Quick update on the spark issue: I didn't get enough time to identify
>>>>> it clearly but managed to have a passing run of my test changing a bunch of
>>>>> versions.
>>>>> I suspect my code triggers some class conflicting between spark and my
>>>>> shade leading to a serialization issue. I didn't test userClassPathFirst
>>>>> option of spark but it can be an interesting thing to enable in beam runner.
>>>>> However it is still very confusing to have it not running just
>>>>> upgrading beam version and the spark error is very hard to understand.
>>>>>
>>>>> Romain Manni-Bucau
>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>> <http://rmannibucau.wordpress.com> | Github
>>>>> <https://github.com/rmannibucau> | LinkedIn
>>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>>
>>>>>
>>>>> Le mar. 18 sept. 2018 à 20:17, Lukasz Cwik <lc...@google.com> a
>>>>> écrit :
>>>>>
>>>>>> Romain hinted that this was a dependency issue but when comparing the
>>>>>> two dependency trees I don't get much of a difference:
>>>>>>
>>>>>> lcwik@lcwik0: ~$ diff /tmp/260 /tmp/270
>>>>>> < [INFO] +- org.apache.beam:beam-runners-spark:jar:2.6.0:compile
>>>>>> < [INFO] | +- org.apache.beam:beam-model-pipeline:jar:2.6.0:compile
>>>>>> ---
>>>>>> > [INFO] +- org.apache.beam:beam-runners-spark:jar:2.7.0:compile
>>>>>> > [INFO] | +- org.apache.beam:beam-model-pipeline:jar:2.7.0:compile
>>>>>> 5c6
>>>>>> < [INFO] | +- org.apache.beam:beam-sdks-java-core:jar:2.6.0:compile
>>>>>> ---
>>>>>> > [INFO] | +- org.apache.beam:beam-sdks-java-core:jar:2.7.0:compile
>>>>>> 14,18c15,19
>>>>>> < [INFO] | | \- org.tukaani:xz:jar:1.5:compile
>>>>>> < [INFO] | +- org.apache.beam:beam-runners-
>>>>>> core-construction-java:jar:2.6.0:compile
>>>>>> < [INFO] | | \- org.apache.beam:beam-model-
>>>>>> job-management:jar:2.6.0:compile
>>>>>> < [INFO] | +- org.apache.beam:beam-runners-
>>>>>> core-java:jar:2.6.0:compile
>>>>>> < [INFO] | | \- org.apache.beam:beam-model-fn-
>>>>>> execution:jar:2.6.0:compile
>>>>>> ---
>>>>>> > [INFO] | | \- org.tukaani:xz:jar:1.8:compile
>>>>>> > [INFO] | +- org.apache.beam:beam-runners-
>>>>>> core-construction-java:jar:2.7.0:compile
>>>>>> > [INFO] | | \- org.apache.beam:beam-model-
>>>>>> job-management:jar:2.7.0:compile
>>>>>> > [INFO] | +- org.apache.beam:beam-runners-
>>>>>> core-java:jar:2.7.0:compile
>>>>>> > [INFO] | | \- org.apache.beam:beam-model-fn-
>>>>>> execution:jar:2.7.0:compile
>>>>>>
>>>>>> Other then Beam package changes, the only other change is xz which I
>>>>>> don't believe could be causing the issue.
>>>>>>
>>>>>> On Tue, Sep 18, 2018 at 8:38 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks, let me take a look.
>>>>>>>
>>>>>>> Regards
>>>>>>> JB
>>>>>>>
>>>>>>> On 18/09/2018 17:36, Romain Manni-Bucau wrote:
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > Le mar. 18 sept. 2018 à 16:44, Jean-Baptiste Onofré <
>>>>>>> jb@nanthrax.net
>>>>>>> > <ma...@nanthrax.net>> a écrit :
>>>>>>> >
>>>>>>> > Hi,
>>>>>>> >
>>>>>>> > I don't have the issue ;)
>>>>>>> >
>>>>>>> > As said in my vote, I tested 2.7.0 RC1 on beam-samples with
>>>>>>> Spark
>>>>>>> > without problem.
>>>>>>> >
>>>>>>> > I don't reproduce Romain issue as well.
>>>>>>> >
>>>>>>> > @Romain can you provide some details to reproduce the issue ?
>>>>>>> >
>>>>>>> >
>>>>>>> > Sure, you can use this
>>>>>>> > reproducer: https://github.com/rmannibucau/beam-2.7.0-fails
>>>>>>> > It shows that it suceeds on 2.6 and fails on 2.7.
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > Regards
>>>>>>> > JB
>>>>>>> >
>>>>>>> > On 17/09/2018 19:17, Charles Chen wrote:
>>>>>>> > > Luke, Maximillian, Raghu, can you please propose cherry-pick
>>>>>>> PRs
>>>>>>> > to the
>>>>>>> > > release-2.7.0 for your issues and add me as a reviewer
>>>>>>> > (@charlesccychen)?
>>>>>>> > >
>>>>>>> > > Romain, JB: is there any way I can help with debugging the
>>>>>>> issue
>>>>>>> > you're
>>>>>>> > > facing so we can unblock the release?
>>>>>>> > >
>>>>>>> > > On Fri, Sep 14, 2018 at 1:49 PM Raghu Angadi <
>>>>>>> rangadi@google.com
>>>>>>> > <ma...@google.com>
>>>>>>> > > <mailto:rangadi@google.com <ma...@google.com>>>
>>>>>>> wrote:
>>>>>>> > >
>>>>>>> > > I would like propose one more cherrypick for RC2
>>>>>>> > > : https://github.com/apache/beam/pull/6391
>>>>>>> > > This is a KafkaIO bug fix. Once a user hits this bug,
>>>>>>> there is no
>>>>>>> > > easy work around for them, especially on Dataflow. Only
>>>>>>> work
>>>>>>> > around
>>>>>>> > > in Dataflow is to restart or reload the job.
>>>>>>> > >
>>>>>>> > > The fix itself fairly safe and is tested.
>>>>>>> > > Raghu.
>>>>>>> > >
>>>>>>> > > On Fri, Sep 14, 2018 at 12:52 AM Alexey Romanenko
>>>>>>> > > <aromanenko.dev@gmail.com <mailto:aromanenko.dev@gmail.
>>>>>>> com>
>>>>>>> > <mailto:aromanenko.dev@gmail.com <mailto:aromanenko.dev@gmail.
>>>>>>> com>>>
>>>>>>> > wrote:
>>>>>>> > >
>>>>>>> > > Perhaps it could help, but I run simple WordCount
>>>>>>> (built with
>>>>>>> > > Beam 2.7) on YARN/Spark (HDP Sandbox) cluster and it
>>>>>>> > worked fine
>>>>>>> > > for me.
>>>>>>> > >
>>>>>>> > >> On 14 Sep 2018, at 06:56, Romain Manni-Bucau
>>>>>>> > >> <rmannibucau@gmail.com <mailto:
>>>>>>> rmannibucau@gmail.com>
>>>>>>> > <mailto:rmannibucau@gmail.com <ma...@gmail.com>>>
>>>>>>> wrote:
>>>>>>> > >>
>>>>>>> > >> Hi Charles,
>>>>>>> > >>
>>>>>>> > >> I didn't get enough time to check deeply but it is
>>>>>>> clearly a
>>>>>>> > >> dependency issue and it is not in beam spark runner
>>>>>>> > itself but
>>>>>>> > >> in another transitive module of beam. It does not
>>>>>>> happen in
>>>>>>> > >> existing spark test cause none of them are in a
>>>>>>> cluster (even
>>>>>>> > >> just with 1 worker) but this seems to be a
>>>>>>> regression since
>>>>>>> > >> 2.6 works OOTB.
>>>>>>> > >>
>>>>>>> > >> Romain Manni-Bucau
>>>>>>> > >> @rmannibucau <https://twitter.com/rmannibucau> |
>>>>>>> Blog
>>>>>>> > >> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>>>> > >> <http://rmannibucau.wordpress.com/> | Github
>>>>>>> > >> <https://github.com/rmannibucau> | LinkedIn
>>>>>>> > >> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>>> > >>
>>>>>>> > <https://www.packtpub.com/application-development/java-
>>>>>>> ee-8-high-performance>
>>>>>>> > >>
>>>>>>> > >>
>>>>>>> > >> Le jeu. 13 sept. 2018 à 22:15, Charles Chen
>>>>>>> > <ccy@google.com <ma...@google.com>
>>>>>>> > >> <mailto:ccy@google.com <ma...@google.com>>> a
>>>>>>> écrit :
>>>>>>> > >>
>>>>>>> > >> Romain and JB, can you please add the results
>>>>>>> of your
>>>>>>> > >> investigations into the errors you've seen
>>>>>>> above? Given
>>>>>>> > >> that the existing SparkRunner tests pass for
>>>>>>> this RC, and
>>>>>>> > >> that the integration test you ran is in another
>>>>>>> repo that
>>>>>>> > >> is not continuously tested with Beam, it is not
>>>>>>> clear how
>>>>>>> > >> we should move forward and whether this is a
>>>>>>> blocking
>>>>>>> > >> issue, unless we can find a root cause in Beam.
>>>>>>> > >>
>>>>>>> > >> On Wed, Sep 12, 2018 at 2:08 AM Etienne Chauchot
>>>>>>> > >> <echauchot@apache.org <mailto:
>>>>>>> echauchot@apache.org>
>>>>>>> > <mailto:echauchot@apache.org <ma...@apache.org>>>
>>>>>>> wrote:
>>>>>>> > >>
>>>>>>> > >> Hi all,
>>>>>>> > >>
>>>>>>> > >> on a performance and functional regression
>>>>>>> stand
>>>>>>> > point
>>>>>>> > >> I see no regression:
>>>>>>> > >>
>>>>>>> > >> I looked at nexmark graphs "output
>>>>>>> pcollection size"
>>>>>>> > >> and "execution time" around release cut
>>>>>>> date on
>>>>>>> > >> dataflow, spark, flink and direct runner in
>>>>>>> batch and
>>>>>>> > >> streaming modes. There seems to be no
>>>>>>> regression.
>>>>>>> > >>
>>>>>>> > >> Etienne
>>>>>>> > >>
>>>>>>> > >> Le mardi 11 septembre 2018 à 12:25 -0700,
>>>>>>> Charles
>>>>>>> > Chen
>>>>>>> > >> a écrit :
>>>>>>> > >>> The SparkRunner validation test
>>>>>>> > >>>
>>>>>>> > (here: https://beam.apache.org/contribute/release-guide/#
>>>>>>> run-validation-tests)
>>>>>>> > >>> passes on my machine. It looks like we
>>>>>>> are likely
>>>>>>> > >>> missing test coverage where Romain is
>>>>>>> hitting
>>>>>>> > issues.
>>>>>>> > >>>
>>>>>>> > >>> On Tue, Sep 11, 2018 at 12:15 PM Ahmet
>>>>>>> Altay
>>>>>>> > >>> <altay@google.com <mailto:altay@google.com
>>>>>>> >
>>>>>>> > <mailto:altay@google.com <ma...@google.com>>> wrote:
>>>>>>> > >>>> Could anyone else help with looking at
>>>>>>> these issues
>>>>>>> > >>>> earlier?
>>>>>>> > >>>>
>>>>>>> > >>>> On Tue, Sep 11, 2018 at 12:03 PM, Romain
>>>>>>> > Manni-Bucau
>>>>>>> > >>>> <rmannibucau@gmail.com
>>>>>>> > <ma...@gmail.com>
>>>>>>> > >>>> <mailto:rmannibucau@gmail.com
>>>>>>> > <ma...@gmail.com>>> wrote:
>>>>>>> > >>>>> Im running this main [1] through this IT
>>>>>>> [2]. Was
>>>>>>> > >>>>> working fine since ~1 year but 2.7.0
>>>>>>> broke it.
>>>>>>> > >>>>> Didnt investigate more but can have a
>>>>>>> look later
>>>>>>> > >>>>> this month if it helps.
>>>>>>> > >>>>>
>>>>>>> > >>>>>
>>>>>>> > [1] https://github.com/Talend/component-runtime/blob/
>>>>>>> master/component-runtime-beam/src/it/serialization-over-
>>>>>>> cluster/src/main/java/org/talend/sdk/component/beam/it/
>>>>>>> clusterserialization/Main.java
>>>>>>> > >>>>>
>>>>>>> > [2] https://github.com/Talend/component-runtime/blob/
>>>>>>> master/component-runtime-beam/src/it/serialization-over-
>>>>>>> cluster/src/test/java/org/talend/sdk/component/beam/it/
>>>>>>> SerializationOverClusterIT.java
>>>>>>> > >>>>>
>>>>>>> > >>>>> Le mar. 11 sept. 2018 20:54, Charles Chen
>>>>>>> > >>>>> <ccy@google.com <ma...@google.com>
>>>>>>> > <mailto:ccy@google.com <ma...@google.com>>> a écrit :
>>>>>>> > >>>>>> Romain: can you give more details on
>>>>>>> the failure
>>>>>>> > >>>>>> you're encountering, i.e. how you are
>>>>>>> performing
>>>>>>> > >>>>>> this validation?
>>>>>>> > >>>>>>
>>>>>>> > >>>>>> On Tue, Sep 11, 2018 at 9:36 AM
>>>>>>> Jean-Baptiste
>>>>>>> > >>>>>> Onofré <jb@nanthrax.net
>>>>>>> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
>>>>>>> > <ma...@nanthrax.net>>>
>>>>>>> > >>>>>> wrote:
>>>>>>> > >>>>>>> Hi,
>>>>>>> > >>>>>>>
>>>>>>> > >>>>>>> weird, I didn't have it on Beam
>>>>>>> samples. Let me
>>>>>>> > >>>>>>> try to reproduce and I
>>>>>>> > >>>>>>> will create the Jira.
>>>>>>> > >>>>>>>
>>>>>>> > >>>>>>> Regards
>>>>>>> > >>>>>>> JB
>>>>>>> > >>>>>>>
>>>>>>> > >>>>>>> On 11/09/2018 11:44, Romain
>>>>>>> Manni-Bucau wrote:
>>>>>>> > >>>>>>> > -1, seems spark integration is
>>>>>>> broken (tested
>>>>>>> > >>>>>>> with spark 2.3.1 and 2.2.1):
>>>>>>> > >>>>>>> >
>>>>>>> > >>>>>>> > 18/09/11 11:33:29 WARN
>>>>>>> TaskSetManager: Lost
>>>>>>> > >>>>>>> task 0.0 in stage 0.0 (TID 0,
>>>>>>> RMANNIBUCAU,
>>>>>>> > >>>>>>> executor 0):
>>>>>>> java.lang.ClassCastException:
>>>>>>> > cannot
>>>>>>> > >>>>>>> assign instance of
>>>>>>> > >>>>>>>
>>>>>>> > scala.collection.immutable.List$SerializationProxy to
>>>>>>> > >>>>>>> fieldorg.apache.spark.rdd.RDD.org
>>>>>>> > <http://fieldorg.apache.spark.rdd.RDD.org>
>>>>>>> > >>>>>>> <http://fieldorg.apache.
>>>>>>> spark.rdd.rdd.org/>
>>>>>>> > >>>>>>> <http://org.apache.spark.rdd.RDD.org
>>>>>>> > >>>>>>>
>>>>>>> > <http://org.apache.spark.rdd.rdd.org/>>$apache$spark$
>>>>>>> rdd$RDD$$dependencies_
>>>>>>> > >>>>>>> of type scala.collection.Seq in
>>>>>>> instance of
>>>>>>> > >>>>>>> org.apache.spark.rdd.MapPartitionsRDD
>>>>>>> > >>>>>>> > at
>>>>>>> > >>>>>>>
>>>>>>> > java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(
>>>>>>> ObjectStreamClass.java:2233)
>>>>>>> > >>>>>>> >
>>>>>>> > >>>>>>> >
>>>>>>> > >>>>>>> > Also the issue Lukasz identified is
>>>>>>> important
>>>>>>> > >>>>>>> even if workarounds can be
>>>>>>> > >>>>>>> > put in place so +1 to fix it as well
>>>>>>> if
>>>>>>> > possible.
>>>>>>> > >>>>>>> >
>>>>>>> > >>>>>>> > Romain Manni-Bucau
>>>>>>> > >>>>>>> > @rmannibucau <https://twitter.com/
>>>>>>> rmannibucau>
>>>>>>> > >>>>>>> | Blog
>>>>>>> > >>>>>>> > <https://rmannibucau.metawerx.net/> |
>>>>>>> Old Blog
>>>>>>> > >>>>>>> > <http://rmannibucau.wordpress.com
>>>>>>> > >>>>>>> <http://rmannibucau.wordpress.com/>>
>>>>>>> | Github
>>>>>>> > >>>>>>> > <https://github.com/rmannibucau> |
>>>>>>> LinkedIn
>>>>>>> > >>>>>>> >
>>>>>>> > <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>>> > >>>>>>> >
>>>>>>> > >>>>>>>
>>>>>>> > <https://www.packtpub.com/application-development/java-
>>>>>>> ee-8-high-performance>
>>>>>>> > >>>>>>> >
>>>>>>> > >>>>>>> >
>>>>>>> > >>>>>>> > Le lun. 10 sept. 2018 à 20:48,
>>>>>>> Lukasz Cwik
>>>>>>> > >>>>>>> <lcwik@google.com <mailto:
>>>>>>> lcwik@google.com>
>>>>>>> > <mailto:lcwik@google.com <ma...@google.com>>
>>>>>>> > >>>>>>> > <mailto:lcwik@google.com
>>>>>>> > <ma...@google.com>
>>>>>>> > >>>>>>> <mailto:lcwik@google.com
>>>>>>> > <ma...@google.com>>>> a écrit :
>>>>>>> > >>>>>>> >
>>>>>>> > >>>>>>> > I found an issue where we are no
>>>>>>> longer
>>>>>>> > >>>>>>> packaging the pom.xml within
>>>>>>> > >>>>>>> > the artifact jars at
>>>>>>> > >>>>>>> META-INF/maven/groupId/artifactId.
>>>>>>> More details
>>>>>>> > >>>>>>> > in
>>>>>>> > >>>>>>> https://issues.apache.org/
>>>>>>> jira/browse/BEAM-5351.
>>>>>>> > >>>>>>> I wouldn't
>>>>>>> > >>>>>>> > consider this a blocker but it
>>>>>>> was an
>>>>>>> > easy fix
>>>>>>> > >>>>>>> > (https://github.com/apache/
>>>>>>> beam/pull/6358)
>>>>>>> > >>>>>>> and users may rely on the
>>>>>>> > >>>>>>> > pom.xml.
>>>>>>> > >>>>>>> >
>>>>>>> > >>>>>>> > Should we recut the release
>>>>>>> candidate to
>>>>>>> > >>>>>>> include this?
>>>>>>> > >>>>>>> >
>>>>>>> > >>>>>>> > On Mon, Sep 10, 2018 at 4:58 AM
>>>>>>> > >>>>>>> Jean-Baptiste Onofré
>>>>>>> > >>>>>>> > <jb@nanthrax.net
>>>>>>> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
>>>>>>> > <ma...@nanthrax.net>>
>>>>>>> > >>>>>>> <mailto:jb@nanthrax.net <mailto:
>>>>>>> jb@nanthrax.net>
>>>>>>> > >>>>>>> <mailto:jb@nanthrax.net
>>>>>>> > <ma...@nanthrax.net>>>> wrote:
>>>>>>> > >>>>>>> >
>>>>>>> > >>>>>>> > +1 (binding)
>>>>>>> > >>>>>>> >
>>>>>>> > >>>>>>> > Tested successfully on Beam
>>>>>>> Samples.
>>>>>>> > >>>>>>> >
>>>>>>> > >>>>>>> > Thanks !
>>>>>>> > >>>>>>> >
>>>>>>> > >>>>>>> > Regards
>>>>>>> > >>>>>>> > JB
>>>>>>> > >>>>>>> >
>>>>>>> > >>>>>>> > On 07/09/2018 23:56, Charles
>>>>>>> Chen
>>>>>>> > wrote:
>>>>>>> > >>>>>>> > > Hi everyone,
>>>>>>> > >>>>>>> > >
>>>>>>> > >>>>>>> > > Please review and vote on
>>>>>>> the
>>>>>>> > >>>>>>> release candidate #1 for the
>>>>>>> > >>>>>>> > version
>>>>>>> > >>>>>>> > > 2.7.0, as follows:
>>>>>>> > >>>>>>> > > [ ] +1, Approve the
>>>>>>> release
>>>>>>> > >>>>>>> > > [ ] -1, Do not approve
>>>>>>> the release
>>>>>>> > >>>>>>> (please provide specific
>>>>>>> > >>>>>>> > comments)
>>>>>>> > >>>>>>> > >
>>>>>>> > >>>>>>> > > The complete staging area
>>>>>>> is
>>>>>>> > >>>>>>> available for your review, which
>>>>>>> > >>>>>>> > includes:
>>>>>>> > >>>>>>> > > * JIRA release notes [1],
>>>>>>> > >>>>>>> > > * the official Apache
>>>>>>> source
>>>>>>> > release
>>>>>>> > >>>>>>> to be deployed to
>>>>>>> > >>>>>>> > dist.apache.org
>>>>>>> > <http://dist.apache.org>
>>>>>>> > >>>>>>> <http://dist.apache.org/>
>>>>>>> > <http://dist.apache.org
>>>>>>> > >>>>>>> <http://dist.apache.org/>>
>>>>>>> > >>>>>>> > > <http://dist.apache.org
>>>>>>> > >>>>>>> <http://dist.apache.org/>> [2], which
>>>>>>> is signed
>>>>>>> > >>>>>>> with the key with
>>>>>>> > >>>>>>> > > fingerprint
>>>>>>> 45C60AAAD115F560 [3],
>>>>>>> > >>>>>>> > > * all artifacts to be
>>>>>>> deployed to
>>>>>>> > >>>>>>> the Maven Central
>>>>>>> > >>>>>>> > Repository [4],
>>>>>>> > >>>>>>> > > * source code tag
>>>>>>> "v2.7.0-RC1" [5],
>>>>>>> > >>>>>>> > > * website pull request
>>>>>>> listing the
>>>>>>> > >>>>>>> release and publishing the API
>>>>>>> > >>>>>>> > > reference manual [6].
>>>>>>> > >>>>>>> > > * Java artifacts were
>>>>>>> built with
>>>>>>> > >>>>>>> Gradle 4.8 and OpenJDK
>>>>>>> > >>>>>>> > >
>>>>>>> 1.8.0_181-8u181-b13-1~deb9u1-b13.
>>>>>>> > >>>>>>> > > * Python artifacts are
>>>>>>> deployed
>>>>>>> > >>>>>>> along with the source release
>>>>>>> > >>>>>>> > to the
>>>>>>> > >>>>>>> > > dist.apache.org
>>>>>>> > <http://dist.apache.org>
>>>>>>> > >>>>>>> <http://dist.apache.org/>
>>>>>>> > <http://dist.apache.org
>>>>>>> > >>>>>>> <http://dist.apache.org/>>
>>>>>>> > >>>>>>> > <http://dist.apache.org
>>>>>>> > >>>>>>> <http://dist.apache.org/>> [2].
>>>>>>> > >>>>>>> > >
>>>>>>> > >>>>>>> > > The vote will be open for
>>>>>>> at least
>>>>>>> > >>>>>>> 72 hours. It is adopted by
>>>>>>> > >>>>>>> > majority
>>>>>>> > >>>>>>> > > approval, with at least 3
>>>>>>> PMC
>>>>>>> > >>>>>>> affirmative votes.
>>>>>>> > >>>>>>> > >
>>>>>>> > >>>>>>> > > Thanks,
>>>>>>> > >>>>>>> > > Charles
>>>>>>> > >>>>>>> > >
>>>>>>> > >>>>>>> > > [1]
>>>>>>> > >>>>>>> > >
>>>>>>> > >>>>>>> >
>>>>>>> > >>>>>>>
>>>>>>> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>>>>>>> projectId=12319527&version=12343654
>>>>>>> > >>>>>>> > > [2]
>>>>>>> > >>>>>>>
>>>>>>> > https://dist.apache.org/repos/dist/dev/beam/2.7.0
>>>>>>> > >>>>>>> > > [3]
>>>>>>> > >>>>>>> https://dist.apache.org/
>>>>>>> repos/dist/dev/beam/KEYS
>>>>>>> > >>>>>>> > > [4]
>>>>>>> > >>>>>>> >
>>>>>>> > >>>>>>>
>>>>>>> > https://repository.apache.org/content/repositories/
>>>>>>> orgapachebeam-1046/
>>>>>>> > >>>>>>> > > [5]
>>>>>>> > >>>>>>> https://github.com/apache/
>>>>>>> beam/tree/v2.7.0-RC1
>>>>>>> > >>>>>>> > > [6]
>>>>>>> > >>>>>>> https://github.com/apache/
>>>>>>> beam-site/pull/549
>>>>>>> > >>>>>>> >
>>>>>>> > >>>>>>> > --
>>>>>>> > >>>>>>> > Jean-Baptiste Onofré
>>>>>>> > >>>>>>> > jbonofre@apache.org
>>>>>>> > <ma...@apache.org>
>>>>>>> > >>>>>>> <mailto:jbonofre@apache.org
>>>>>>> > <ma...@apache.org>>
>>>>>>> > >>>>>>> <mailto:jbonofre@apache.org
>>>>>>> > <ma...@apache.org>
>>>>>>> > >>>>>>> <mailto:jbonofre@apache.org
>>>>>>> > <ma...@apache.org>>>
>>>>>>> > >>>>>>> > http://blog.nanthrax.net
>>>>>>> > >>>>>>> <http://blog.nanthrax.net/>
>>>>>>> > >>>>>>> > Talend -
>>>>>>> http://www.talend.com
>>>>>>> > >>>>>>> <http://www.talend.com/>
>>>>>>> > >>>>>>> >
>>>>>>> > >>>>
>>>>>>> > >
>>>>>>> >
>>>>>>> > --
>>>>>>> > Jean-Baptiste Onofré
>>>>>>> > jbonofre@apache.org <ma...@apache.org>
>>>>>>> > http://blog.nanthrax.net
>>>>>>> > Talend - http://www.talend.com
>>>>>>> >
>>>>>>>
>>>>>>> --
>>>>>>> Jean-Baptiste Onofré
>>>>>>> jbonofre@apache.org
>>>>>>> http://blog.nanthrax.net
>>>>>>> Talend - http://www.talend.com
>>>>>>>
>>>>>>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Connell O'Callaghan <co...@google.com>.
+1 to Thomas's suggestion - if Charles or others cannot reproduce.
On Thu, Sep 20, 2018 at 2:40 PM Thomas Weise <th...@apache.org> wrote:
> We can also consider releasing 2.7.0 and then follow up with 2.7.1 if the
> problem can be reproduced and requires a fix. Just food for thought :)
>
>
> On Thu, Sep 20, 2018 at 2:13 PM Charles Chen <cc...@google.com> wrote:
>
>> My mistake, it looks like the correct beam staging repository (
>> https://repository.apache.org/content/repositories/orgapachebeam-1046/)
>> is specified in your pom file.
>>
>> On Thu, Sep 20, 2018 at 2:10 PM Charles Chen <cc...@google.com> wrote:
>>
>>> Hey Romain and JB, do you have any progress on this? One thing I would
>>> like to point out is that 2.7.0 isn't yet pushed to Maven Central, so
>>> referring to it by version is not expected to work (and it looks like this
>>> is what is done in your repo:
>>> https://github.com/rmannibucau/beam-2.7.0-fails). Luke indicated above
>>> that he doesn't see any dependency changes. Can you isolate and reproduce
>>> this problem so that we can develop a fix, if necessary? I would like to
>>> proceed with an RC2 as soon as possible.
>>>
>>> On Wed, Sep 19, 2018 at 6:37 AM Romain Manni-Bucau <
>>> rmannibucau@gmail.com> wrote:
>>>
>>>> Quick update on the spark issue: I didn't get enough time to identify
>>>> it clearly but managed to have a passing run of my test changing a bunch of
>>>> versions.
>>>> I suspect my code triggers some class conflicting between spark and my
>>>> shade leading to a serialization issue. I didn't test userClassPathFirst
>>>> option of spark but it can be an interesting thing to enable in beam runner.
>>>> However it is still very confusing to have it not running just
>>>> upgrading beam version and the spark error is very hard to understand.
>>>>
>>>> Romain Manni-Bucau
>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>> <http://rmannibucau.wordpress.com> | Github
>>>> <https://github.com/rmannibucau> | LinkedIn
>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>
>>>>
>>>> Le mar. 18 sept. 2018 à 20:17, Lukasz Cwik <lc...@google.com> a écrit :
>>>>
>>>>> Romain hinted that this was a dependency issue but when comparing the
>>>>> two dependency trees I don't get much of a difference:
>>>>>
>>>>> lcwik@lcwik0: ~$ diff /tmp/260 /tmp/270
>>>>> < [INFO] +- org.apache.beam:beam-runners-spark:jar:2.6.0:compile
>>>>> < [INFO] | +- org.apache.beam:beam-model-pipeline:jar:2.6.0:compile
>>>>> ---
>>>>> > [INFO] +- org.apache.beam:beam-runners-spark:jar:2.7.0:compile
>>>>> > [INFO] | +- org.apache.beam:beam-model-pipeline:jar:2.7.0:compile
>>>>> 5c6
>>>>> < [INFO] | +- org.apache.beam:beam-sdks-java-core:jar:2.6.0:compile
>>>>> ---
>>>>> > [INFO] | +- org.apache.beam:beam-sdks-java-core:jar:2.7.0:compile
>>>>> 14,18c15,19
>>>>> < [INFO] | | \- org.tukaani:xz:jar:1.5:compile
>>>>> < [INFO] | +-
>>>>> org.apache.beam:beam-runners-core-construction-java:jar:2.6.0:compile
>>>>> < [INFO] | | \-
>>>>> org.apache.beam:beam-model-job-management:jar:2.6.0:compile
>>>>> < [INFO] | +- org.apache.beam:beam-runners-core-java:jar:2.6.0:compile
>>>>> < [INFO] | | \-
>>>>> org.apache.beam:beam-model-fn-execution:jar:2.6.0:compile
>>>>> ---
>>>>> > [INFO] | | \- org.tukaani:xz:jar:1.8:compile
>>>>> > [INFO] | +-
>>>>> org.apache.beam:beam-runners-core-construction-java:jar:2.7.0:compile
>>>>> > [INFO] | | \-
>>>>> org.apache.beam:beam-model-job-management:jar:2.7.0:compile
>>>>> > [INFO] | +- org.apache.beam:beam-runners-core-java:jar:2.7.0:compile
>>>>> > [INFO] | | \-
>>>>> org.apache.beam:beam-model-fn-execution:jar:2.7.0:compile
>>>>>
>>>>> Other then Beam package changes, the only other change is xz which I
>>>>> don't believe could be causing the issue.
>>>>>
>>>>> On Tue, Sep 18, 2018 at 8:38 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>>>>> wrote:
>>>>>
>>>>>> Thanks, let me take a look.
>>>>>>
>>>>>> Regards
>>>>>> JB
>>>>>>
>>>>>> On 18/09/2018 17:36, Romain Manni-Bucau wrote:
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > Le mar. 18 sept. 2018 à 16:44, Jean-Baptiste Onofré <
>>>>>> jb@nanthrax.net
>>>>>> > <ma...@nanthrax.net>> a écrit :
>>>>>> >
>>>>>> > Hi,
>>>>>> >
>>>>>> > I don't have the issue ;)
>>>>>> >
>>>>>> > As said in my vote, I tested 2.7.0 RC1 on beam-samples with
>>>>>> Spark
>>>>>> > without problem.
>>>>>> >
>>>>>> > I don't reproduce Romain issue as well.
>>>>>> >
>>>>>> > @Romain can you provide some details to reproduce the issue ?
>>>>>> >
>>>>>> >
>>>>>> > Sure, you can use this
>>>>>> > reproducer: https://github.com/rmannibucau/beam-2.7.0-fails
>>>>>> > It shows that it suceeds on 2.6 and fails on 2.7.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > Regards
>>>>>> > JB
>>>>>> >
>>>>>> > On 17/09/2018 19:17, Charles Chen wrote:
>>>>>> > > Luke, Maximillian, Raghu, can you please propose cherry-pick
>>>>>> PRs
>>>>>> > to the
>>>>>> > > release-2.7.0 for your issues and add me as a reviewer
>>>>>> > (@charlesccychen)?
>>>>>> > >
>>>>>> > > Romain, JB: is there any way I can help with debugging the
>>>>>> issue
>>>>>> > you're
>>>>>> > > facing so we can unblock the release?
>>>>>> > >
>>>>>> > > On Fri, Sep 14, 2018 at 1:49 PM Raghu Angadi <
>>>>>> rangadi@google.com
>>>>>> > <ma...@google.com>
>>>>>> > > <mailto:rangadi@google.com <ma...@google.com>>>
>>>>>> wrote:
>>>>>> > >
>>>>>> > > I would like propose one more cherrypick for RC2
>>>>>> > > : https://github.com/apache/beam/pull/6391
>>>>>> > > This is a KafkaIO bug fix. Once a user hits this bug,
>>>>>> there is no
>>>>>> > > easy work around for them, especially on Dataflow. Only
>>>>>> work
>>>>>> > around
>>>>>> > > in Dataflow is to restart or reload the job.
>>>>>> > >
>>>>>> > > The fix itself fairly safe and is tested.
>>>>>> > > Raghu.
>>>>>> > >
>>>>>> > > On Fri, Sep 14, 2018 at 12:52 AM Alexey Romanenko
>>>>>> > > <aromanenko.dev@gmail.com <mailto:
>>>>>> aromanenko.dev@gmail.com>
>>>>>> > <mailto:aromanenko.dev@gmail.com <mailto:
>>>>>> aromanenko.dev@gmail.com>>>
>>>>>> > wrote:
>>>>>> > >
>>>>>> > > Perhaps it could help, but I run simple WordCount
>>>>>> (built with
>>>>>> > > Beam 2.7) on YARN/Spark (HDP Sandbox) cluster and it
>>>>>> > worked fine
>>>>>> > > for me.
>>>>>> > >
>>>>>> > >> On 14 Sep 2018, at 06:56, Romain Manni-Bucau
>>>>>> > >> <rmannibucau@gmail.com <mailto:rmannibucau@gmail.com
>>>>>> >
>>>>>> > <mailto:rmannibucau@gmail.com <ma...@gmail.com>>>
>>>>>> wrote:
>>>>>> > >>
>>>>>> > >> Hi Charles,
>>>>>> > >>
>>>>>> > >> I didn't get enough time to check deeply but it is
>>>>>> clearly a
>>>>>> > >> dependency issue and it is not in beam spark runner
>>>>>> > itself but
>>>>>> > >> in another transitive module of beam. It does not
>>>>>> happen in
>>>>>> > >> existing spark test cause none of them are in a
>>>>>> cluster (even
>>>>>> > >> just with 1 worker) but this seems to be a
>>>>>> regression since
>>>>>> > >> 2.6 works OOTB.
>>>>>> > >>
>>>>>> > >> Romain Manni-Bucau
>>>>>> > >> @rmannibucau <https://twitter.com/rmannibucau> |
>>>>>> Blog
>>>>>> > >> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>>> > >> <http://rmannibucau.wordpress.com/> | Github
>>>>>> > >> <https://github.com/rmannibucau> | LinkedIn
>>>>>> > >> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>> > >>
>>>>>> > <
>>>>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>>>> >
>>>>>> > >>
>>>>>> > >>
>>>>>> > >> Le jeu. 13 sept. 2018 à 22:15, Charles Chen
>>>>>> > <ccy@google.com <ma...@google.com>
>>>>>> > >> <mailto:ccy@google.com <ma...@google.com>>> a
>>>>>> écrit :
>>>>>> > >>
>>>>>> > >> Romain and JB, can you please add the results of
>>>>>> your
>>>>>> > >> investigations into the errors you've seen
>>>>>> above? Given
>>>>>> > >> that the existing SparkRunner tests pass for
>>>>>> this RC, and
>>>>>> > >> that the integration test you ran is in another
>>>>>> repo that
>>>>>> > >> is not continuously tested with Beam, it is not
>>>>>> clear how
>>>>>> > >> we should move forward and whether this is a
>>>>>> blocking
>>>>>> > >> issue, unless we can find a root cause in Beam.
>>>>>> > >>
>>>>>> > >> On Wed, Sep 12, 2018 at 2:08 AM Etienne Chauchot
>>>>>> > >> <echauchot@apache.org <mailto:
>>>>>> echauchot@apache.org>
>>>>>> > <mailto:echauchot@apache.org <ma...@apache.org>>>
>>>>>> wrote:
>>>>>> > >>
>>>>>> > >> Hi all,
>>>>>> > >>
>>>>>> > >> on a performance and functional regression
>>>>>> stand
>>>>>> > point
>>>>>> > >> I see no regression:
>>>>>> > >>
>>>>>> > >> I looked at nexmark graphs "output
>>>>>> pcollection size"
>>>>>> > >> and "execution time" around release cut date
>>>>>> on
>>>>>> > >> dataflow, spark, flink and direct runner in
>>>>>> batch and
>>>>>> > >> streaming modes. There seems to be no
>>>>>> regression.
>>>>>> > >>
>>>>>> > >> Etienne
>>>>>> > >>
>>>>>> > >> Le mardi 11 septembre 2018 à 12:25 -0700,
>>>>>> Charles
>>>>>> > Chen
>>>>>> > >> a écrit :
>>>>>> > >>> The SparkRunner validation test
>>>>>> > >>>
>>>>>> > (here:
>>>>>> https://beam.apache.org/contribute/release-guide/#run-validation-tests
>>>>>> )
>>>>>> > >>> passes on my machine. It looks like we are
>>>>>> likely
>>>>>> > >>> missing test coverage where Romain is
>>>>>> hitting
>>>>>> > issues.
>>>>>> > >>>
>>>>>> > >>> On Tue, Sep 11, 2018 at 12:15 PM Ahmet Altay
>>>>>> > >>> <altay@google.com <ma...@google.com>
>>>>>> > <mailto:altay@google.com <ma...@google.com>>> wrote:
>>>>>> > >>>> Could anyone else help with looking at
>>>>>> these issues
>>>>>> > >>>> earlier?
>>>>>> > >>>>
>>>>>> > >>>> On Tue, Sep 11, 2018 at 12:03 PM, Romain
>>>>>> > Manni-Bucau
>>>>>> > >>>> <rmannibucau@gmail.com
>>>>>> > <ma...@gmail.com>
>>>>>> > >>>> <mailto:rmannibucau@gmail.com
>>>>>> > <ma...@gmail.com>>> wrote:
>>>>>> > >>>>> Im running this main [1] through this IT
>>>>>> [2]. Was
>>>>>> > >>>>> working fine since ~1 year but 2.7.0
>>>>>> broke it.
>>>>>> > >>>>> Didnt investigate more but can have a
>>>>>> look later
>>>>>> > >>>>> this month if it helps.
>>>>>> > >>>>>
>>>>>> > >>>>>
>>>>>> > [1]
>>>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
>>>>>> > >>>>>
>>>>>> > [2]
>>>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
>>>>>> > >>>>>
>>>>>> > >>>>> Le mar. 11 sept. 2018 20:54, Charles Chen
>>>>>> > >>>>> <ccy@google.com <ma...@google.com>
>>>>>> > <mailto:ccy@google.com <ma...@google.com>>> a écrit :
>>>>>> > >>>>>> Romain: can you give more details on the
>>>>>> failure
>>>>>> > >>>>>> you're encountering, i.e. how you are
>>>>>> performing
>>>>>> > >>>>>> this validation?
>>>>>> > >>>>>>
>>>>>> > >>>>>> On Tue, Sep 11, 2018 at 9:36 AM
>>>>>> Jean-Baptiste
>>>>>> > >>>>>> Onofré <jb@nanthrax.net
>>>>>> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
>>>>>> > <ma...@nanthrax.net>>>
>>>>>> > >>>>>> wrote:
>>>>>> > >>>>>>> Hi,
>>>>>> > >>>>>>>
>>>>>> > >>>>>>> weird, I didn't have it on Beam
>>>>>> samples. Let me
>>>>>> > >>>>>>> try to reproduce and I
>>>>>> > >>>>>>> will create the Jira.
>>>>>> > >>>>>>>
>>>>>> > >>>>>>> Regards
>>>>>> > >>>>>>> JB
>>>>>> > >>>>>>>
>>>>>> > >>>>>>> On 11/09/2018 11:44, Romain Manni-Bucau
>>>>>> wrote:
>>>>>> > >>>>>>> > -1, seems spark integration is broken
>>>>>> (tested
>>>>>> > >>>>>>> with spark 2.3.1 and 2.2.1):
>>>>>> > >>>>>>> >
>>>>>> > >>>>>>> > 18/09/11 11:33:29 WARN
>>>>>> TaskSetManager: Lost
>>>>>> > >>>>>>> task 0.0 in stage 0.0 (TID 0,
>>>>>> RMANNIBUCAU,
>>>>>> > >>>>>>> executor 0):
>>>>>> java.lang.ClassCastException:
>>>>>> > cannot
>>>>>> > >>>>>>> assign instance of
>>>>>> > >>>>>>>
>>>>>> > scala.collection.immutable.List$SerializationProxy to
>>>>>> > >>>>>>> fieldorg.apache.spark.rdd.RDD.org
>>>>>> > <http://fieldorg.apache.spark.rdd.RDD.org>
>>>>>> > >>>>>>> <
>>>>>> http://fieldorg.apache.spark.rdd.rdd.org/>
>>>>>> > >>>>>>> <http://org.apache.spark.rdd.RDD.org
>>>>>> > >>>>>>>
>>>>>> > <http://org.apache.spark.rdd.rdd.org/
>>>>>> >>$apache$spark$rdd$RDD$$dependencies_
>>>>>> > >>>>>>> of type scala.collection.Seq in
>>>>>> instance of
>>>>>> > >>>>>>> org.apache.spark.rdd.MapPartitionsRDD
>>>>>> > >>>>>>> > at
>>>>>> > >>>>>>>
>>>>>> >
>>>>>> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>>>>>> > >>>>>>> >
>>>>>> > >>>>>>> >
>>>>>> > >>>>>>> > Also the issue Lukasz identified is
>>>>>> important
>>>>>> > >>>>>>> even if workarounds can be
>>>>>> > >>>>>>> > put in place so +1 to fix it as well
>>>>>> if
>>>>>> > possible.
>>>>>> > >>>>>>> >
>>>>>> > >>>>>>> > Romain Manni-Bucau
>>>>>> > >>>>>>> > @rmannibucau <
>>>>>> https://twitter.com/rmannibucau>
>>>>>> > >>>>>>> | Blog
>>>>>> > >>>>>>> > <https://rmannibucau.metawerx.net/> |
>>>>>> Old Blog
>>>>>> > >>>>>>> > <http://rmannibucau.wordpress.com
>>>>>> > >>>>>>> <http://rmannibucau.wordpress.com/>> |
>>>>>> Github
>>>>>> > >>>>>>> > <https://github.com/rmannibucau> |
>>>>>> LinkedIn
>>>>>> > >>>>>>> >
>>>>>> > <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>> > >>>>>>> >
>>>>>> > >>>>>>>
>>>>>> > <
>>>>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>>>> >
>>>>>> > >>>>>>> >
>>>>>> > >>>>>>> >
>>>>>> > >>>>>>> > Le lun. 10 sept. 2018 à 20:48, Lukasz
>>>>>> Cwik
>>>>>> > >>>>>>> <lcwik@google.com <mailto:
>>>>>> lcwik@google.com>
>>>>>> > <mailto:lcwik@google.com <ma...@google.com>>
>>>>>> > >>>>>>> > <mailto:lcwik@google.com
>>>>>> > <ma...@google.com>
>>>>>> > >>>>>>> <mailto:lcwik@google.com
>>>>>> > <ma...@google.com>>>> a écrit :
>>>>>> > >>>>>>> >
>>>>>> > >>>>>>> > I found an issue where we are no
>>>>>> longer
>>>>>> > >>>>>>> packaging the pom.xml within
>>>>>> > >>>>>>> > the artifact jars at
>>>>>> > >>>>>>> META-INF/maven/groupId/artifactId. More
>>>>>> details
>>>>>> > >>>>>>> > in
>>>>>> > >>>>>>>
>>>>>> https://issues.apache.org/jira/browse/BEAM-5351.
>>>>>> > >>>>>>> I wouldn't
>>>>>> > >>>>>>> > consider this a blocker but it
>>>>>> was an
>>>>>> > easy fix
>>>>>> > >>>>>>> > (
>>>>>> https://github.com/apache/beam/pull/6358)
>>>>>> > >>>>>>> and users may rely on the
>>>>>> > >>>>>>> > pom.xml.
>>>>>> > >>>>>>> >
>>>>>> > >>>>>>> > Should we recut the release
>>>>>> candidate to
>>>>>> > >>>>>>> include this?
>>>>>> > >>>>>>> >
>>>>>> > >>>>>>> > On Mon, Sep 10, 2018 at 4:58 AM
>>>>>> > >>>>>>> Jean-Baptiste Onofré
>>>>>> > >>>>>>> > <jb@nanthrax.net
>>>>>> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
>>>>>> > <ma...@nanthrax.net>>
>>>>>> > >>>>>>> <mailto:jb@nanthrax.net <mailto:
>>>>>> jb@nanthrax.net>
>>>>>> > >>>>>>> <mailto:jb@nanthrax.net
>>>>>> > <ma...@nanthrax.net>>>> wrote:
>>>>>> > >>>>>>> >
>>>>>> > >>>>>>> > +1 (binding)
>>>>>> > >>>>>>> >
>>>>>> > >>>>>>> > Tested successfully on Beam
>>>>>> Samples.
>>>>>> > >>>>>>> >
>>>>>> > >>>>>>> > Thanks !
>>>>>> > >>>>>>> >
>>>>>> > >>>>>>> > Regards
>>>>>> > >>>>>>> > JB
>>>>>> > >>>>>>> >
>>>>>> > >>>>>>> > On 07/09/2018 23:56, Charles
>>>>>> Chen
>>>>>> > wrote:
>>>>>> > >>>>>>> > > Hi everyone,
>>>>>> > >>>>>>> > >
>>>>>> > >>>>>>> > > Please review and vote on
>>>>>> the
>>>>>> > >>>>>>> release candidate #1 for the
>>>>>> > >>>>>>> > version
>>>>>> > >>>>>>> > > 2.7.0, as follows:
>>>>>> > >>>>>>> > > [ ] +1, Approve the release
>>>>>> > >>>>>>> > > [ ] -1, Do not approve the
>>>>>> release
>>>>>> > >>>>>>> (please provide specific
>>>>>> > >>>>>>> > comments)
>>>>>> > >>>>>>> > >
>>>>>> > >>>>>>> > > The complete staging area
>>>>>> is
>>>>>> > >>>>>>> available for your review, which
>>>>>> > >>>>>>> > includes:
>>>>>> > >>>>>>> > > * JIRA release notes [1],
>>>>>> > >>>>>>> > > * the official Apache
>>>>>> source
>>>>>> > release
>>>>>> > >>>>>>> to be deployed to
>>>>>> > >>>>>>> > dist.apache.org
>>>>>> > <http://dist.apache.org>
>>>>>> > >>>>>>> <http://dist.apache.org/>
>>>>>> > <http://dist.apache.org
>>>>>> > >>>>>>> <http://dist.apache.org/>>
>>>>>> > >>>>>>> > > <http://dist.apache.org
>>>>>> > >>>>>>> <http://dist.apache.org/>> [2], which
>>>>>> is signed
>>>>>> > >>>>>>> with the key with
>>>>>> > >>>>>>> > > fingerprint
>>>>>> 45C60AAAD115F560 [3],
>>>>>> > >>>>>>> > > * all artifacts to be
>>>>>> deployed to
>>>>>> > >>>>>>> the Maven Central
>>>>>> > >>>>>>> > Repository [4],
>>>>>> > >>>>>>> > > * source code tag
>>>>>> "v2.7.0-RC1" [5],
>>>>>> > >>>>>>> > > * website pull request
>>>>>> listing the
>>>>>> > >>>>>>> release and publishing the API
>>>>>> > >>>>>>> > > reference manual [6].
>>>>>> > >>>>>>> > > * Java artifacts were
>>>>>> built with
>>>>>> > >>>>>>> Gradle 4.8 and OpenJDK
>>>>>> > >>>>>>> > >
>>>>>> 1.8.0_181-8u181-b13-1~deb9u1-b13.
>>>>>> > >>>>>>> > > * Python artifacts are
>>>>>> deployed
>>>>>> > >>>>>>> along with the source release
>>>>>> > >>>>>>> > to the
>>>>>> > >>>>>>> > > dist.apache.org
>>>>>> > <http://dist.apache.org>
>>>>>> > >>>>>>> <http://dist.apache.org/>
>>>>>> > <http://dist.apache.org
>>>>>> > >>>>>>> <http://dist.apache.org/>>
>>>>>> > >>>>>>> > <http://dist.apache.org
>>>>>> > >>>>>>> <http://dist.apache.org/>> [2].
>>>>>> > >>>>>>> > >
>>>>>> > >>>>>>> > > The vote will be open for
>>>>>> at least
>>>>>> > >>>>>>> 72 hours. It is adopted by
>>>>>> > >>>>>>> > majority
>>>>>> > >>>>>>> > > approval, with at least 3
>>>>>> PMC
>>>>>> > >>>>>>> affirmative votes.
>>>>>> > >>>>>>> > >
>>>>>> > >>>>>>> > > Thanks,
>>>>>> > >>>>>>> > > Charles
>>>>>> > >>>>>>> > >
>>>>>> > >>>>>>> > > [1]
>>>>>> > >>>>>>> > >
>>>>>> > >>>>>>> >
>>>>>> > >>>>>>>
>>>>>> >
>>>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>>>>>> > >>>>>>> > > [2]
>>>>>> > >>>>>>>
>>>>>> > https://dist.apache.org/repos/dist/dev/beam/2.7.0
>>>>>> > >>>>>>> > > [3]
>>>>>> > >>>>>>>
>>>>>> https://dist.apache.org/repos/dist/dev/beam/KEYS
>>>>>> > >>>>>>> > > [4]
>>>>>> > >>>>>>> >
>>>>>> > >>>>>>>
>>>>>> >
>>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1046/
>>>>>> > >>>>>>> > > [5]
>>>>>> > >>>>>>>
>>>>>> https://github.com/apache/beam/tree/v2.7.0-RC1
>>>>>> > >>>>>>> > > [6]
>>>>>> > >>>>>>>
>>>>>> https://github.com/apache/beam-site/pull/549
>>>>>> > >>>>>>> >
>>>>>> > >>>>>>> > --
>>>>>> > >>>>>>> > Jean-Baptiste Onofré
>>>>>> > >>>>>>> > jbonofre@apache.org
>>>>>> > <ma...@apache.org>
>>>>>> > >>>>>>> <mailto:jbonofre@apache.org
>>>>>> > <ma...@apache.org>>
>>>>>> > >>>>>>> <mailto:jbonofre@apache.org
>>>>>> > <ma...@apache.org>
>>>>>> > >>>>>>> <mailto:jbonofre@apache.org
>>>>>> > <ma...@apache.org>>>
>>>>>> > >>>>>>> > http://blog.nanthrax.net
>>>>>> > >>>>>>> <http://blog.nanthrax.net/>
>>>>>> > >>>>>>> > Talend -
>>>>>> http://www.talend.com
>>>>>> > >>>>>>> <http://www.talend.com/>
>>>>>> > >>>>>>> >
>>>>>> > >>>>
>>>>>> > >
>>>>>> >
>>>>>> > --
>>>>>> > Jean-Baptiste Onofré
>>>>>> > jbonofre@apache.org <ma...@apache.org>
>>>>>> > http://blog.nanthrax.net
>>>>>> > Talend - http://www.talend.com
>>>>>> >
>>>>>>
>>>>>> --
>>>>>> Jean-Baptiste Onofré
>>>>>> jbonofre@apache.org
>>>>>> http://blog.nanthrax.net
>>>>>> Talend - http://www.talend.com
>>>>>>
>>>>>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Thomas Weise <th...@apache.org>.
We can also consider releasing 2.7.0 and then follow up with 2.7.1 if the
problem can be reproduced and requires a fix. Just food for thought :)
On Thu, Sep 20, 2018 at 2:13 PM Charles Chen <cc...@google.com> wrote:
> My mistake, it looks like the correct beam staging repository (
> https://repository.apache.org/content/repositories/orgapachebeam-1046/)
> is specified in your pom file.
>
> On Thu, Sep 20, 2018 at 2:10 PM Charles Chen <cc...@google.com> wrote:
>
>> Hey Romain and JB, do you have any progress on this? One thing I would
>> like to point out is that 2.7.0 isn't yet pushed to Maven Central, so
>> referring to it by version is not expected to work (and it looks like this
>> is what is done in your repo:
>> https://github.com/rmannibucau/beam-2.7.0-fails). Luke indicated above
>> that he doesn't see any dependency changes. Can you isolate and reproduce
>> this problem so that we can develop a fix, if necessary? I would like to
>> proceed with an RC2 as soon as possible.
>>
>> On Wed, Sep 19, 2018 at 6:37 AM Romain Manni-Bucau <rm...@gmail.com>
>> wrote:
>>
>>> Quick update on the spark issue: I didn't get enough time to identify it
>>> clearly but managed to have a passing run of my test changing a bunch of
>>> versions.
>>> I suspect my code triggers some class conflicting between spark and my
>>> shade leading to a serialization issue. I didn't test userClassPathFirst
>>> option of spark but it can be an interesting thing to enable in beam runner.
>>> However it is still very confusing to have it not running just upgrading
>>> beam version and the spark error is very hard to understand.
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>> <http://rmannibucau.wordpress.com> | Github
>>> <https://github.com/rmannibucau> | LinkedIn
>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>
>>>
>>> Le mar. 18 sept. 2018 à 20:17, Lukasz Cwik <lc...@google.com> a écrit :
>>>
>>>> Romain hinted that this was a dependency issue but when comparing the
>>>> two dependency trees I don't get much of a difference:
>>>>
>>>> lcwik@lcwik0: ~$ diff /tmp/260 /tmp/270
>>>> < [INFO] +- org.apache.beam:beam-runners-spark:jar:2.6.0:compile
>>>> < [INFO] | +- org.apache.beam:beam-model-pipeline:jar:2.6.0:compile
>>>> ---
>>>> > [INFO] +- org.apache.beam:beam-runners-spark:jar:2.7.0:compile
>>>> > [INFO] | +- org.apache.beam:beam-model-pipeline:jar:2.7.0:compile
>>>> 5c6
>>>> < [INFO] | +- org.apache.beam:beam-sdks-java-core:jar:2.6.0:compile
>>>> ---
>>>> > [INFO] | +- org.apache.beam:beam-sdks-java-core:jar:2.7.0:compile
>>>> 14,18c15,19
>>>> < [INFO] | | \- org.tukaani:xz:jar:1.5:compile
>>>> < [INFO] | +-
>>>> org.apache.beam:beam-runners-core-construction-java:jar:2.6.0:compile
>>>> < [INFO] | | \-
>>>> org.apache.beam:beam-model-job-management:jar:2.6.0:compile
>>>> < [INFO] | +- org.apache.beam:beam-runners-core-java:jar:2.6.0:compile
>>>> < [INFO] | | \-
>>>> org.apache.beam:beam-model-fn-execution:jar:2.6.0:compile
>>>> ---
>>>> > [INFO] | | \- org.tukaani:xz:jar:1.8:compile
>>>> > [INFO] | +-
>>>> org.apache.beam:beam-runners-core-construction-java:jar:2.7.0:compile
>>>> > [INFO] | | \-
>>>> org.apache.beam:beam-model-job-management:jar:2.7.0:compile
>>>> > [INFO] | +- org.apache.beam:beam-runners-core-java:jar:2.7.0:compile
>>>> > [INFO] | | \-
>>>> org.apache.beam:beam-model-fn-execution:jar:2.7.0:compile
>>>>
>>>> Other then Beam package changes, the only other change is xz which I
>>>> don't believe could be causing the issue.
>>>>
>>>> On Tue, Sep 18, 2018 at 8:38 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>>>> wrote:
>>>>
>>>>> Thanks, let me take a look.
>>>>>
>>>>> Regards
>>>>> JB
>>>>>
>>>>> On 18/09/2018 17:36, Romain Manni-Bucau wrote:
>>>>> >
>>>>> >
>>>>> >
>>>>> > Le mar. 18 sept. 2018 à 16:44, Jean-Baptiste Onofré <jb@nanthrax.net
>>>>> > <ma...@nanthrax.net>> a écrit :
>>>>> >
>>>>> > Hi,
>>>>> >
>>>>> > I don't have the issue ;)
>>>>> >
>>>>> > As said in my vote, I tested 2.7.0 RC1 on beam-samples with Spark
>>>>> > without problem.
>>>>> >
>>>>> > I don't reproduce Romain issue as well.
>>>>> >
>>>>> > @Romain can you provide some details to reproduce the issue ?
>>>>> >
>>>>> >
>>>>> > Sure, you can use this
>>>>> > reproducer: https://github.com/rmannibucau/beam-2.7.0-fails
>>>>> > It shows that it suceeds on 2.6 and fails on 2.7.
>>>>> >
>>>>> >
>>>>> >
>>>>> > Regards
>>>>> > JB
>>>>> >
>>>>> > On 17/09/2018 19:17, Charles Chen wrote:
>>>>> > > Luke, Maximillian, Raghu, can you please propose cherry-pick
>>>>> PRs
>>>>> > to the
>>>>> > > release-2.7.0 for your issues and add me as a reviewer
>>>>> > (@charlesccychen)?
>>>>> > >
>>>>> > > Romain, JB: is there any way I can help with debugging the
>>>>> issue
>>>>> > you're
>>>>> > > facing so we can unblock the release?
>>>>> > >
>>>>> > > On Fri, Sep 14, 2018 at 1:49 PM Raghu Angadi <
>>>>> rangadi@google.com
>>>>> > <ma...@google.com>
>>>>> > > <mailto:rangadi@google.com <ma...@google.com>>>
>>>>> wrote:
>>>>> > >
>>>>> > > I would like propose one more cherrypick for RC2
>>>>> > > : https://github.com/apache/beam/pull/6391
>>>>> > > This is a KafkaIO bug fix. Once a user hits this bug,
>>>>> there is no
>>>>> > > easy work around for them, especially on Dataflow. Only
>>>>> work
>>>>> > around
>>>>> > > in Dataflow is to restart or reload the job.
>>>>> > >
>>>>> > > The fix itself fairly safe and is tested.
>>>>> > > Raghu.
>>>>> > >
>>>>> > > On Fri, Sep 14, 2018 at 12:52 AM Alexey Romanenko
>>>>> > > <aromanenko.dev@gmail.com <mailto:aromanenko.dev@gmail.com
>>>>> >
>>>>> > <mailto:aromanenko.dev@gmail.com <mailto:
>>>>> aromanenko.dev@gmail.com>>>
>>>>> > wrote:
>>>>> > >
>>>>> > > Perhaps it could help, but I run simple WordCount
>>>>> (built with
>>>>> > > Beam 2.7) on YARN/Spark (HDP Sandbox) cluster and it
>>>>> > worked fine
>>>>> > > for me.
>>>>> > >
>>>>> > >> On 14 Sep 2018, at 06:56, Romain Manni-Bucau
>>>>> > >> <rmannibucau@gmail.com <ma...@gmail.com>
>>>>> > <mailto:rmannibucau@gmail.com <ma...@gmail.com>>>
>>>>> wrote:
>>>>> > >>
>>>>> > >> Hi Charles,
>>>>> > >>
>>>>> > >> I didn't get enough time to check deeply but it is
>>>>> clearly a
>>>>> > >> dependency issue and it is not in beam spark runner
>>>>> > itself but
>>>>> > >> in another transitive module of beam. It does not
>>>>> happen in
>>>>> > >> existing spark test cause none of them are in a
>>>>> cluster (even
>>>>> > >> just with 1 worker) but this seems to be a regression
>>>>> since
>>>>> > >> 2.6 works OOTB.
>>>>> > >>
>>>>> > >> Romain Manni-Bucau
>>>>> > >> @rmannibucau <https://twitter.com/rmannibucau> |
>>>>> Blog
>>>>> > >> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>> > >> <http://rmannibucau.wordpress.com/> | Github
>>>>> > >> <https://github.com/rmannibucau> | LinkedIn
>>>>> > >> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>> > >>
>>>>> > <
>>>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>>> >
>>>>> > >>
>>>>> > >>
>>>>> > >> Le jeu. 13 sept. 2018 à 22:15, Charles Chen
>>>>> > <ccy@google.com <ma...@google.com>
>>>>> > >> <mailto:ccy@google.com <ma...@google.com>>> a
>>>>> écrit :
>>>>> > >>
>>>>> > >> Romain and JB, can you please add the results of
>>>>> your
>>>>> > >> investigations into the errors you've seen
>>>>> above? Given
>>>>> > >> that the existing SparkRunner tests pass for this
>>>>> RC, and
>>>>> > >> that the integration test you ran is in another
>>>>> repo that
>>>>> > >> is not continuously tested with Beam, it is not
>>>>> clear how
>>>>> > >> we should move forward and whether this is a
>>>>> blocking
>>>>> > >> issue, unless we can find a root cause in Beam.
>>>>> > >>
>>>>> > >> On Wed, Sep 12, 2018 at 2:08 AM Etienne Chauchot
>>>>> > >> <echauchot@apache.org <mailto:
>>>>> echauchot@apache.org>
>>>>> > <mailto:echauchot@apache.org <ma...@apache.org>>>
>>>>> wrote:
>>>>> > >>
>>>>> > >> Hi all,
>>>>> > >>
>>>>> > >> on a performance and functional regression
>>>>> stand
>>>>> > point
>>>>> > >> I see no regression:
>>>>> > >>
>>>>> > >> I looked at nexmark graphs "output
>>>>> pcollection size"
>>>>> > >> and "execution time" around release cut date
>>>>> on
>>>>> > >> dataflow, spark, flink and direct runner in
>>>>> batch and
>>>>> > >> streaming modes. There seems to be no
>>>>> regression.
>>>>> > >>
>>>>> > >> Etienne
>>>>> > >>
>>>>> > >> Le mardi 11 septembre 2018 à 12:25 -0700,
>>>>> Charles
>>>>> > Chen
>>>>> > >> a écrit :
>>>>> > >>> The SparkRunner validation test
>>>>> > >>>
>>>>> > (here:
>>>>> https://beam.apache.org/contribute/release-guide/#run-validation-tests
>>>>> )
>>>>> > >>> passes on my machine. It looks like we are
>>>>> likely
>>>>> > >>> missing test coverage where Romain is hitting
>>>>> > issues.
>>>>> > >>>
>>>>> > >>> On Tue, Sep 11, 2018 at 12:15 PM Ahmet Altay
>>>>> > >>> <altay@google.com <ma...@google.com>
>>>>> > <mailto:altay@google.com <ma...@google.com>>> wrote:
>>>>> > >>>> Could anyone else help with looking at
>>>>> these issues
>>>>> > >>>> earlier?
>>>>> > >>>>
>>>>> > >>>> On Tue, Sep 11, 2018 at 12:03 PM, Romain
>>>>> > Manni-Bucau
>>>>> > >>>> <rmannibucau@gmail.com
>>>>> > <ma...@gmail.com>
>>>>> > >>>> <mailto:rmannibucau@gmail.com
>>>>> > <ma...@gmail.com>>> wrote:
>>>>> > >>>>> Im running this main [1] through this IT
>>>>> [2]. Was
>>>>> > >>>>> working fine since ~1 year but 2.7.0 broke
>>>>> it.
>>>>> > >>>>> Didnt investigate more but can have a look
>>>>> later
>>>>> > >>>>> this month if it helps.
>>>>> > >>>>>
>>>>> > >>>>>
>>>>> > [1]
>>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
>>>>> > >>>>>
>>>>> > [2]
>>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
>>>>> > >>>>>
>>>>> > >>>>> Le mar. 11 sept. 2018 20:54, Charles Chen
>>>>> > >>>>> <ccy@google.com <ma...@google.com>
>>>>> > <mailto:ccy@google.com <ma...@google.com>>> a écrit :
>>>>> > >>>>>> Romain: can you give more details on the
>>>>> failure
>>>>> > >>>>>> you're encountering, i.e. how you are
>>>>> performing
>>>>> > >>>>>> this validation?
>>>>> > >>>>>>
>>>>> > >>>>>> On Tue, Sep 11, 2018 at 9:36 AM
>>>>> Jean-Baptiste
>>>>> > >>>>>> Onofré <jb@nanthrax.net
>>>>> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
>>>>> > <ma...@nanthrax.net>>>
>>>>> > >>>>>> wrote:
>>>>> > >>>>>>> Hi,
>>>>> > >>>>>>>
>>>>> > >>>>>>> weird, I didn't have it on Beam samples.
>>>>> Let me
>>>>> > >>>>>>> try to reproduce and I
>>>>> > >>>>>>> will create the Jira.
>>>>> > >>>>>>>
>>>>> > >>>>>>> Regards
>>>>> > >>>>>>> JB
>>>>> > >>>>>>>
>>>>> > >>>>>>> On 11/09/2018 11:44, Romain Manni-Bucau
>>>>> wrote:
>>>>> > >>>>>>> > -1, seems spark integration is broken
>>>>> (tested
>>>>> > >>>>>>> with spark 2.3.1 and 2.2.1):
>>>>> > >>>>>>> >
>>>>> > >>>>>>> > 18/09/11 11:33:29 WARN TaskSetManager:
>>>>> Lost
>>>>> > >>>>>>> task 0.0 in stage 0.0 (TID 0,
>>>>> RMANNIBUCAU,
>>>>> > >>>>>>> executor 0):
>>>>> java.lang.ClassCastException:
>>>>> > cannot
>>>>> > >>>>>>> assign instance of
>>>>> > >>>>>>>
>>>>> > scala.collection.immutable.List$SerializationProxy to
>>>>> > >>>>>>> fieldorg.apache.spark.rdd.RDD.org
>>>>> > <http://fieldorg.apache.spark.rdd.RDD.org>
>>>>> > >>>>>>> <
>>>>> http://fieldorg.apache.spark.rdd.rdd.org/>
>>>>> > >>>>>>> <http://org.apache.spark.rdd.RDD.org
>>>>> > >>>>>>>
>>>>> > <http://org.apache.spark.rdd.rdd.org/
>>>>> >>$apache$spark$rdd$RDD$$dependencies_
>>>>> > >>>>>>> of type scala.collection.Seq in instance
>>>>> of
>>>>> > >>>>>>> org.apache.spark.rdd.MapPartitionsRDD
>>>>> > >>>>>>> > at
>>>>> > >>>>>>>
>>>>> >
>>>>> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>>>>> > >>>>>>> >
>>>>> > >>>>>>> >
>>>>> > >>>>>>> > Also the issue Lukasz identified is
>>>>> important
>>>>> > >>>>>>> even if workarounds can be
>>>>> > >>>>>>> > put in place so +1 to fix it as well if
>>>>> > possible.
>>>>> > >>>>>>> >
>>>>> > >>>>>>> > Romain Manni-Bucau
>>>>> > >>>>>>> > @rmannibucau <
>>>>> https://twitter.com/rmannibucau>
>>>>> > >>>>>>> | Blog
>>>>> > >>>>>>> > <https://rmannibucau.metawerx.net/> |
>>>>> Old Blog
>>>>> > >>>>>>> > <http://rmannibucau.wordpress.com
>>>>> > >>>>>>> <http://rmannibucau.wordpress.com/>> |
>>>>> Github
>>>>> > >>>>>>> > <https://github.com/rmannibucau> |
>>>>> LinkedIn
>>>>> > >>>>>>> >
>>>>> > <https://www.linkedin.com/in/rmannibucau> | Book
>>>>> > >>>>>>> >
>>>>> > >>>>>>>
>>>>> > <
>>>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>>> >
>>>>> > >>>>>>> >
>>>>> > >>>>>>> >
>>>>> > >>>>>>> > Le lun. 10 sept. 2018 à 20:48, Lukasz
>>>>> Cwik
>>>>> > >>>>>>> <lcwik@google.com <mailto:
>>>>> lcwik@google.com>
>>>>> > <mailto:lcwik@google.com <ma...@google.com>>
>>>>> > >>>>>>> > <mailto:lcwik@google.com
>>>>> > <ma...@google.com>
>>>>> > >>>>>>> <mailto:lcwik@google.com
>>>>> > <ma...@google.com>>>> a écrit :
>>>>> > >>>>>>> >
>>>>> > >>>>>>> > I found an issue where we are no
>>>>> longer
>>>>> > >>>>>>> packaging the pom.xml within
>>>>> > >>>>>>> > the artifact jars at
>>>>> > >>>>>>> META-INF/maven/groupId/artifactId. More
>>>>> details
>>>>> > >>>>>>> > in
>>>>> > >>>>>>>
>>>>> https://issues.apache.org/jira/browse/BEAM-5351.
>>>>> > >>>>>>> I wouldn't
>>>>> > >>>>>>> > consider this a blocker but it was
>>>>> an
>>>>> > easy fix
>>>>> > >>>>>>> > (
>>>>> https://github.com/apache/beam/pull/6358)
>>>>> > >>>>>>> and users may rely on the
>>>>> > >>>>>>> > pom.xml.
>>>>> > >>>>>>> >
>>>>> > >>>>>>> > Should we recut the release
>>>>> candidate to
>>>>> > >>>>>>> include this?
>>>>> > >>>>>>> >
>>>>> > >>>>>>> > On Mon, Sep 10, 2018 at 4:58 AM
>>>>> > >>>>>>> Jean-Baptiste Onofré
>>>>> > >>>>>>> > <jb@nanthrax.net
>>>>> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
>>>>> > <ma...@nanthrax.net>>
>>>>> > >>>>>>> <mailto:jb@nanthrax.net <mailto:
>>>>> jb@nanthrax.net>
>>>>> > >>>>>>> <mailto:jb@nanthrax.net
>>>>> > <ma...@nanthrax.net>>>> wrote:
>>>>> > >>>>>>> >
>>>>> > >>>>>>> > +1 (binding)
>>>>> > >>>>>>> >
>>>>> > >>>>>>> > Tested successfully on Beam
>>>>> Samples.
>>>>> > >>>>>>> >
>>>>> > >>>>>>> > Thanks !
>>>>> > >>>>>>> >
>>>>> > >>>>>>> > Regards
>>>>> > >>>>>>> > JB
>>>>> > >>>>>>> >
>>>>> > >>>>>>> > On 07/09/2018 23:56, Charles
>>>>> Chen
>>>>> > wrote:
>>>>> > >>>>>>> > > Hi everyone,
>>>>> > >>>>>>> > >
>>>>> > >>>>>>> > > Please review and vote on
>>>>> the
>>>>> > >>>>>>> release candidate #1 for the
>>>>> > >>>>>>> > version
>>>>> > >>>>>>> > > 2.7.0, as follows:
>>>>> > >>>>>>> > > [ ] +1, Approve the release
>>>>> > >>>>>>> > > [ ] -1, Do not approve the
>>>>> release
>>>>> > >>>>>>> (please provide specific
>>>>> > >>>>>>> > comments)
>>>>> > >>>>>>> > >
>>>>> > >>>>>>> > > The complete staging area is
>>>>> > >>>>>>> available for your review, which
>>>>> > >>>>>>> > includes:
>>>>> > >>>>>>> > > * JIRA release notes [1],
>>>>> > >>>>>>> > > * the official Apache source
>>>>> > release
>>>>> > >>>>>>> to be deployed to
>>>>> > >>>>>>> > dist.apache.org
>>>>> > <http://dist.apache.org>
>>>>> > >>>>>>> <http://dist.apache.org/>
>>>>> > <http://dist.apache.org
>>>>> > >>>>>>> <http://dist.apache.org/>>
>>>>> > >>>>>>> > > <http://dist.apache.org
>>>>> > >>>>>>> <http://dist.apache.org/>> [2], which
>>>>> is signed
>>>>> > >>>>>>> with the key with
>>>>> > >>>>>>> > > fingerprint
>>>>> 45C60AAAD115F560 [3],
>>>>> > >>>>>>> > > * all artifacts to be
>>>>> deployed to
>>>>> > >>>>>>> the Maven Central
>>>>> > >>>>>>> > Repository [4],
>>>>> > >>>>>>> > > * source code tag
>>>>> "v2.7.0-RC1" [5],
>>>>> > >>>>>>> > > * website pull request
>>>>> listing the
>>>>> > >>>>>>> release and publishing the API
>>>>> > >>>>>>> > > reference manual [6].
>>>>> > >>>>>>> > > * Java artifacts were built
>>>>> with
>>>>> > >>>>>>> Gradle 4.8 and OpenJDK
>>>>> > >>>>>>> > >
>>>>> 1.8.0_181-8u181-b13-1~deb9u1-b13.
>>>>> > >>>>>>> > > * Python artifacts are
>>>>> deployed
>>>>> > >>>>>>> along with the source release
>>>>> > >>>>>>> > to the
>>>>> > >>>>>>> > > dist.apache.org
>>>>> > <http://dist.apache.org>
>>>>> > >>>>>>> <http://dist.apache.org/>
>>>>> > <http://dist.apache.org
>>>>> > >>>>>>> <http://dist.apache.org/>>
>>>>> > >>>>>>> > <http://dist.apache.org
>>>>> > >>>>>>> <http://dist.apache.org/>> [2].
>>>>> > >>>>>>> > >
>>>>> > >>>>>>> > > The vote will be open for
>>>>> at least
>>>>> > >>>>>>> 72 hours. It is adopted by
>>>>> > >>>>>>> > majority
>>>>> > >>>>>>> > > approval, with at least 3
>>>>> PMC
>>>>> > >>>>>>> affirmative votes.
>>>>> > >>>>>>> > >
>>>>> > >>>>>>> > > Thanks,
>>>>> > >>>>>>> > > Charles
>>>>> > >>>>>>> > >
>>>>> > >>>>>>> > > [1]
>>>>> > >>>>>>> > >
>>>>> > >>>>>>> >
>>>>> > >>>>>>>
>>>>> >
>>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>>>>> > >>>>>>> > > [2]
>>>>> > >>>>>>>
>>>>> > https://dist.apache.org/repos/dist/dev/beam/2.7.0
>>>>> > >>>>>>> > > [3]
>>>>> > >>>>>>>
>>>>> https://dist.apache.org/repos/dist/dev/beam/KEYS
>>>>> > >>>>>>> > > [4]
>>>>> > >>>>>>> >
>>>>> > >>>>>>>
>>>>> >
>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1046/
>>>>> > >>>>>>> > > [5]
>>>>> > >>>>>>>
>>>>> https://github.com/apache/beam/tree/v2.7.0-RC1
>>>>> > >>>>>>> > > [6]
>>>>> > >>>>>>>
>>>>> https://github.com/apache/beam-site/pull/549
>>>>> > >>>>>>> >
>>>>> > >>>>>>> > --
>>>>> > >>>>>>> > Jean-Baptiste Onofré
>>>>> > >>>>>>> > jbonofre@apache.org
>>>>> > <ma...@apache.org>
>>>>> > >>>>>>> <mailto:jbonofre@apache.org
>>>>> > <ma...@apache.org>>
>>>>> > >>>>>>> <mailto:jbonofre@apache.org
>>>>> > <ma...@apache.org>
>>>>> > >>>>>>> <mailto:jbonofre@apache.org
>>>>> > <ma...@apache.org>>>
>>>>> > >>>>>>> > http://blog.nanthrax.net
>>>>> > >>>>>>> <http://blog.nanthrax.net/>
>>>>> > >>>>>>> > Talend - http://www.talend.com
>>>>> > >>>>>>> <http://www.talend.com/>
>>>>> > >>>>>>> >
>>>>> > >>>>
>>>>> > >
>>>>> >
>>>>> > --
>>>>> > Jean-Baptiste Onofré
>>>>> > jbonofre@apache.org <ma...@apache.org>
>>>>> > http://blog.nanthrax.net
>>>>> > Talend - http://www.talend.com
>>>>> >
>>>>>
>>>>> --
>>>>> Jean-Baptiste Onofré
>>>>> jbonofre@apache.org
>>>>> http://blog.nanthrax.net
>>>>> Talend - http://www.talend.com
>>>>>
>>>>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Charles Chen <cc...@google.com>.
My mistake, it looks like the correct beam staging repository (
https://repository.apache.org/content/repositories/orgapachebeam-1046/) is
specified in your pom file.
On Thu, Sep 20, 2018 at 2:10 PM Charles Chen <cc...@google.com> wrote:
> Hey Romain and JB, do you have any progress on this? One thing I would
> like to point out is that 2.7.0 isn't yet pushed to Maven Central, so
> referring to it by version is not expected to work (and it looks like this
> is what is done in your repo:
> https://github.com/rmannibucau/beam-2.7.0-fails). Luke indicated above
> that he doesn't see any dependency changes. Can you isolate and reproduce
> this problem so that we can develop a fix, if necessary? I would like to
> proceed with an RC2 as soon as possible.
>
> On Wed, Sep 19, 2018 at 6:37 AM Romain Manni-Bucau <rm...@gmail.com>
> wrote:
>
>> Quick update on the spark issue: I didn't get enough time to identify it
>> clearly but managed to have a passing run of my test changing a bunch of
>> versions.
>> I suspect my code triggers some class conflicting between spark and my
>> shade leading to a serialization issue. I didn't test userClassPathFirst
>> option of spark but it can be an interesting thing to enable in beam runner.
>> However it is still very confusing to have it not running just upgrading
>> beam version and the spark error is very hard to understand.
>>
>> Romain Manni-Bucau
>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>> <https://rmannibucau.metawerx.net/> | Old Blog
>> <http://rmannibucau.wordpress.com> | Github
>> <https://github.com/rmannibucau> | LinkedIn
>> <https://www.linkedin.com/in/rmannibucau> | Book
>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>
>>
>> Le mar. 18 sept. 2018 à 20:17, Lukasz Cwik <lc...@google.com> a écrit :
>>
>>> Romain hinted that this was a dependency issue but when comparing the
>>> two dependency trees I don't get much of a difference:
>>>
>>> lcwik@lcwik0: ~$ diff /tmp/260 /tmp/270
>>> < [INFO] +- org.apache.beam:beam-runners-spark:jar:2.6.0:compile
>>> < [INFO] | +- org.apache.beam:beam-model-pipeline:jar:2.6.0:compile
>>> ---
>>> > [INFO] +- org.apache.beam:beam-runners-spark:jar:2.7.0:compile
>>> > [INFO] | +- org.apache.beam:beam-model-pipeline:jar:2.7.0:compile
>>> 5c6
>>> < [INFO] | +- org.apache.beam:beam-sdks-java-core:jar:2.6.0:compile
>>> ---
>>> > [INFO] | +- org.apache.beam:beam-sdks-java-core:jar:2.7.0:compile
>>> 14,18c15,19
>>> < [INFO] | | \- org.tukaani:xz:jar:1.5:compile
>>> < [INFO] | +-
>>> org.apache.beam:beam-runners-core-construction-java:jar:2.6.0:compile
>>> < [INFO] | | \-
>>> org.apache.beam:beam-model-job-management:jar:2.6.0:compile
>>> < [INFO] | +- org.apache.beam:beam-runners-core-java:jar:2.6.0:compile
>>> < [INFO] | | \-
>>> org.apache.beam:beam-model-fn-execution:jar:2.6.0:compile
>>> ---
>>> > [INFO] | | \- org.tukaani:xz:jar:1.8:compile
>>> > [INFO] | +-
>>> org.apache.beam:beam-runners-core-construction-java:jar:2.7.0:compile
>>> > [INFO] | | \-
>>> org.apache.beam:beam-model-job-management:jar:2.7.0:compile
>>> > [INFO] | +- org.apache.beam:beam-runners-core-java:jar:2.7.0:compile
>>> > [INFO] | | \-
>>> org.apache.beam:beam-model-fn-execution:jar:2.7.0:compile
>>>
>>> Other then Beam package changes, the only other change is xz which I
>>> don't believe could be causing the issue.
>>>
>>> On Tue, Sep 18, 2018 at 8:38 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>>> wrote:
>>>
>>>> Thanks, let me take a look.
>>>>
>>>> Regards
>>>> JB
>>>>
>>>> On 18/09/2018 17:36, Romain Manni-Bucau wrote:
>>>> >
>>>> >
>>>> >
>>>> > Le mar. 18 sept. 2018 à 16:44, Jean-Baptiste Onofré <jb@nanthrax.net
>>>> > <ma...@nanthrax.net>> a écrit :
>>>> >
>>>> > Hi,
>>>> >
>>>> > I don't have the issue ;)
>>>> >
>>>> > As said in my vote, I tested 2.7.0 RC1 on beam-samples with Spark
>>>> > without problem.
>>>> >
>>>> > I don't reproduce Romain issue as well.
>>>> >
>>>> > @Romain can you provide some details to reproduce the issue ?
>>>> >
>>>> >
>>>> > Sure, you can use this
>>>> > reproducer: https://github.com/rmannibucau/beam-2.7.0-fails
>>>> > It shows that it suceeds on 2.6 and fails on 2.7.
>>>> >
>>>> >
>>>> >
>>>> > Regards
>>>> > JB
>>>> >
>>>> > On 17/09/2018 19:17, Charles Chen wrote:
>>>> > > Luke, Maximillian, Raghu, can you please propose cherry-pick PRs
>>>> > to the
>>>> > > release-2.7.0 for your issues and add me as a reviewer
>>>> > (@charlesccychen)?
>>>> > >
>>>> > > Romain, JB: is there any way I can help with debugging the issue
>>>> > you're
>>>> > > facing so we can unblock the release?
>>>> > >
>>>> > > On Fri, Sep 14, 2018 at 1:49 PM Raghu Angadi <
>>>> rangadi@google.com
>>>> > <ma...@google.com>
>>>> > > <mailto:rangadi@google.com <ma...@google.com>>> wrote:
>>>> > >
>>>> > > I would like propose one more cherrypick for RC2
>>>> > > : https://github.com/apache/beam/pull/6391
>>>> > > This is a KafkaIO bug fix. Once a user hits this bug, there
>>>> is no
>>>> > > easy work around for them, especially on Dataflow. Only work
>>>> > around
>>>> > > in Dataflow is to restart or reload the job.
>>>> > >
>>>> > > The fix itself fairly safe and is tested.
>>>> > > Raghu.
>>>> > >
>>>> > > On Fri, Sep 14, 2018 at 12:52 AM Alexey Romanenko
>>>> > > <aromanenko.dev@gmail.com <ma...@gmail.com>
>>>> > <mailto:aromanenko.dev@gmail.com <mailto:aromanenko.dev@gmail.com
>>>> >>>
>>>> > wrote:
>>>> > >
>>>> > > Perhaps it could help, but I run simple WordCount
>>>> (built with
>>>> > > Beam 2.7) on YARN/Spark (HDP Sandbox) cluster and it
>>>> > worked fine
>>>> > > for me.
>>>> > >
>>>> > >> On 14 Sep 2018, at 06:56, Romain Manni-Bucau
>>>> > >> <rmannibucau@gmail.com <ma...@gmail.com>
>>>> > <mailto:rmannibucau@gmail.com <ma...@gmail.com>>>
>>>> wrote:
>>>> > >>
>>>> > >> Hi Charles,
>>>> > >>
>>>> > >> I didn't get enough time to check deeply but it is
>>>> clearly a
>>>> > >> dependency issue and it is not in beam spark runner
>>>> > itself but
>>>> > >> in another transitive module of beam. It does not
>>>> happen in
>>>> > >> existing spark test cause none of them are in a
>>>> cluster (even
>>>> > >> just with 1 worker) but this seems to be a regression
>>>> since
>>>> > >> 2.6 works OOTB.
>>>> > >>
>>>> > >> Romain Manni-Bucau
>>>> > >> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>> > >> <https://rmannibucau.metawerx.net/> | Old Blog
>>>> > >> <http://rmannibucau.wordpress.com/> | Github
>>>> > >> <https://github.com/rmannibucau> | LinkedIn
>>>> > >> <https://www.linkedin.com/in/rmannibucau> | Book
>>>> > >>
>>>> > <
>>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>> >
>>>> > >>
>>>> > >>
>>>> > >> Le jeu. 13 sept. 2018 à 22:15, Charles Chen
>>>> > <ccy@google.com <ma...@google.com>
>>>> > >> <mailto:ccy@google.com <ma...@google.com>>> a
>>>> écrit :
>>>> > >>
>>>> > >> Romain and JB, can you please add the results of
>>>> your
>>>> > >> investigations into the errors you've seen above?
>>>> Given
>>>> > >> that the existing SparkRunner tests pass for this
>>>> RC, and
>>>> > >> that the integration test you ran is in another
>>>> repo that
>>>> > >> is not continuously tested with Beam, it is not
>>>> clear how
>>>> > >> we should move forward and whether this is a
>>>> blocking
>>>> > >> issue, unless we can find a root cause in Beam.
>>>> > >>
>>>> > >> On Wed, Sep 12, 2018 at 2:08 AM Etienne Chauchot
>>>> > >> <echauchot@apache.org <mailto:echauchot@apache.org
>>>> >
>>>> > <mailto:echauchot@apache.org <ma...@apache.org>>>
>>>> wrote:
>>>> > >>
>>>> > >> Hi all,
>>>> > >>
>>>> > >> on a performance and functional regression
>>>> stand
>>>> > point
>>>> > >> I see no regression:
>>>> > >>
>>>> > >> I looked at nexmark graphs "output pcollection
>>>> size"
>>>> > >> and "execution time" around release cut date on
>>>> > >> dataflow, spark, flink and direct runner in
>>>> batch and
>>>> > >> streaming modes. There seems to be no
>>>> regression.
>>>> > >>
>>>> > >> Etienne
>>>> > >>
>>>> > >> Le mardi 11 septembre 2018 à 12:25 -0700,
>>>> Charles
>>>> > Chen
>>>> > >> a écrit :
>>>> > >>> The SparkRunner validation test
>>>> > >>>
>>>> > (here:
>>>> https://beam.apache.org/contribute/release-guide/#run-validation-tests)
>>>> > >>> passes on my machine. It looks like we are
>>>> likely
>>>> > >>> missing test coverage where Romain is hitting
>>>> > issues.
>>>> > >>>
>>>> > >>> On Tue, Sep 11, 2018 at 12:15 PM Ahmet Altay
>>>> > >>> <altay@google.com <ma...@google.com>
>>>> > <mailto:altay@google.com <ma...@google.com>>> wrote:
>>>> > >>>> Could anyone else help with looking at these
>>>> issues
>>>> > >>>> earlier?
>>>> > >>>>
>>>> > >>>> On Tue, Sep 11, 2018 at 12:03 PM, Romain
>>>> > Manni-Bucau
>>>> > >>>> <rmannibucau@gmail.com
>>>> > <ma...@gmail.com>
>>>> > >>>> <mailto:rmannibucau@gmail.com
>>>> > <ma...@gmail.com>>> wrote:
>>>> > >>>>> Im running this main [1] through this IT
>>>> [2]. Was
>>>> > >>>>> working fine since ~1 year but 2.7.0 broke
>>>> it.
>>>> > >>>>> Didnt investigate more but can have a look
>>>> later
>>>> > >>>>> this month if it helps.
>>>> > >>>>>
>>>> > >>>>>
>>>> > [1]
>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
>>>> > >>>>>
>>>> > [2]
>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
>>>> > >>>>>
>>>> > >>>>> Le mar. 11 sept. 2018 20:54, Charles Chen
>>>> > >>>>> <ccy@google.com <ma...@google.com>
>>>> > <mailto:ccy@google.com <ma...@google.com>>> a écrit :
>>>> > >>>>>> Romain: can you give more details on the
>>>> failure
>>>> > >>>>>> you're encountering, i.e. how you are
>>>> performing
>>>> > >>>>>> this validation?
>>>> > >>>>>>
>>>> > >>>>>> On Tue, Sep 11, 2018 at 9:36 AM
>>>> Jean-Baptiste
>>>> > >>>>>> Onofré <jb@nanthrax.net
>>>> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
>>>> > <ma...@nanthrax.net>>>
>>>> > >>>>>> wrote:
>>>> > >>>>>>> Hi,
>>>> > >>>>>>>
>>>> > >>>>>>> weird, I didn't have it on Beam samples.
>>>> Let me
>>>> > >>>>>>> try to reproduce and I
>>>> > >>>>>>> will create the Jira.
>>>> > >>>>>>>
>>>> > >>>>>>> Regards
>>>> > >>>>>>> JB
>>>> > >>>>>>>
>>>> > >>>>>>> On 11/09/2018 11:44, Romain Manni-Bucau
>>>> wrote:
>>>> > >>>>>>> > -1, seems spark integration is broken
>>>> (tested
>>>> > >>>>>>> with spark 2.3.1 and 2.2.1):
>>>> > >>>>>>> >
>>>> > >>>>>>> > 18/09/11 11:33:29 WARN TaskSetManager:
>>>> Lost
>>>> > >>>>>>> task 0.0 in stage 0.0 (TID 0, RMANNIBUCAU,
>>>> > >>>>>>> executor 0): java.lang.ClassCastException:
>>>> > cannot
>>>> > >>>>>>> assign instance of
>>>> > >>>>>>>
>>>> > scala.collection.immutable.List$SerializationProxy to
>>>> > >>>>>>> fieldorg.apache.spark.rdd.RDD.org
>>>> > <http://fieldorg.apache.spark.rdd.RDD.org>
>>>> > >>>>>>> <
>>>> http://fieldorg.apache.spark.rdd.rdd.org/>
>>>> > >>>>>>> <http://org.apache.spark.rdd.RDD.org
>>>> > >>>>>>>
>>>> > <http://org.apache.spark.rdd.rdd.org/
>>>> >>$apache$spark$rdd$RDD$$dependencies_
>>>> > >>>>>>> of type scala.collection.Seq in instance
>>>> of
>>>> > >>>>>>> org.apache.spark.rdd.MapPartitionsRDD
>>>> > >>>>>>> > at
>>>> > >>>>>>>
>>>> >
>>>> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>>>> > >>>>>>> >
>>>> > >>>>>>> >
>>>> > >>>>>>> > Also the issue Lukasz identified is
>>>> important
>>>> > >>>>>>> even if workarounds can be
>>>> > >>>>>>> > put in place so +1 to fix it as well if
>>>> > possible.
>>>> > >>>>>>> >
>>>> > >>>>>>> > Romain Manni-Bucau
>>>> > >>>>>>> > @rmannibucau <
>>>> https://twitter.com/rmannibucau>
>>>> > >>>>>>> | Blog
>>>> > >>>>>>> > <https://rmannibucau.metawerx.net/> |
>>>> Old Blog
>>>> > >>>>>>> > <http://rmannibucau.wordpress.com
>>>> > >>>>>>> <http://rmannibucau.wordpress.com/>> |
>>>> Github
>>>> > >>>>>>> > <https://github.com/rmannibucau> |
>>>> LinkedIn
>>>> > >>>>>>> >
>>>> > <https://www.linkedin.com/in/rmannibucau> | Book
>>>> > >>>>>>> >
>>>> > >>>>>>>
>>>> > <
>>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>> >
>>>> > >>>>>>> >
>>>> > >>>>>>> >
>>>> > >>>>>>> > Le lun. 10 sept. 2018 à 20:48, Lukasz
>>>> Cwik
>>>> > >>>>>>> <lcwik@google.com <mailto:
>>>> lcwik@google.com>
>>>> > <mailto:lcwik@google.com <ma...@google.com>>
>>>> > >>>>>>> > <mailto:lcwik@google.com
>>>> > <ma...@google.com>
>>>> > >>>>>>> <mailto:lcwik@google.com
>>>> > <ma...@google.com>>>> a écrit :
>>>> > >>>>>>> >
>>>> > >>>>>>> > I found an issue where we are no
>>>> longer
>>>> > >>>>>>> packaging the pom.xml within
>>>> > >>>>>>> > the artifact jars at
>>>> > >>>>>>> META-INF/maven/groupId/artifactId. More
>>>> details
>>>> > >>>>>>> > in
>>>> > >>>>>>>
>>>> https://issues.apache.org/jira/browse/BEAM-5351.
>>>> > >>>>>>> I wouldn't
>>>> > >>>>>>> > consider this a blocker but it was
>>>> an
>>>> > easy fix
>>>> > >>>>>>> > (
>>>> https://github.com/apache/beam/pull/6358)
>>>> > >>>>>>> and users may rely on the
>>>> > >>>>>>> > pom.xml.
>>>> > >>>>>>> >
>>>> > >>>>>>> > Should we recut the release
>>>> candidate to
>>>> > >>>>>>> include this?
>>>> > >>>>>>> >
>>>> > >>>>>>> > On Mon, Sep 10, 2018 at 4:58 AM
>>>> > >>>>>>> Jean-Baptiste Onofré
>>>> > >>>>>>> > <jb@nanthrax.net
>>>> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
>>>> > <ma...@nanthrax.net>>
>>>> > >>>>>>> <mailto:jb@nanthrax.net <mailto:
>>>> jb@nanthrax.net>
>>>> > >>>>>>> <mailto:jb@nanthrax.net
>>>> > <ma...@nanthrax.net>>>> wrote:
>>>> > >>>>>>> >
>>>> > >>>>>>> > +1 (binding)
>>>> > >>>>>>> >
>>>> > >>>>>>> > Tested successfully on Beam
>>>> Samples.
>>>> > >>>>>>> >
>>>> > >>>>>>> > Thanks !
>>>> > >>>>>>> >
>>>> > >>>>>>> > Regards
>>>> > >>>>>>> > JB
>>>> > >>>>>>> >
>>>> > >>>>>>> > On 07/09/2018 23:56, Charles
>>>> Chen
>>>> > wrote:
>>>> > >>>>>>> > > Hi everyone,
>>>> > >>>>>>> > >
>>>> > >>>>>>> > > Please review and vote on the
>>>> > >>>>>>> release candidate #1 for the
>>>> > >>>>>>> > version
>>>> > >>>>>>> > > 2.7.0, as follows:
>>>> > >>>>>>> > > [ ] +1, Approve the release
>>>> > >>>>>>> > > [ ] -1, Do not approve the
>>>> release
>>>> > >>>>>>> (please provide specific
>>>> > >>>>>>> > comments)
>>>> > >>>>>>> > >
>>>> > >>>>>>> > > The complete staging area is
>>>> > >>>>>>> available for your review, which
>>>> > >>>>>>> > includes:
>>>> > >>>>>>> > > * JIRA release notes [1],
>>>> > >>>>>>> > > * the official Apache source
>>>> > release
>>>> > >>>>>>> to be deployed to
>>>> > >>>>>>> > dist.apache.org
>>>> > <http://dist.apache.org>
>>>> > >>>>>>> <http://dist.apache.org/>
>>>> > <http://dist.apache.org
>>>> > >>>>>>> <http://dist.apache.org/>>
>>>> > >>>>>>> > > <http://dist.apache.org
>>>> > >>>>>>> <http://dist.apache.org/>> [2], which is
>>>> signed
>>>> > >>>>>>> with the key with
>>>> > >>>>>>> > > fingerprint 45C60AAAD115F560
>>>> [3],
>>>> > >>>>>>> > > * all artifacts to be
>>>> deployed to
>>>> > >>>>>>> the Maven Central
>>>> > >>>>>>> > Repository [4],
>>>> > >>>>>>> > > * source code tag
>>>> "v2.7.0-RC1" [5],
>>>> > >>>>>>> > > * website pull request
>>>> listing the
>>>> > >>>>>>> release and publishing the API
>>>> > >>>>>>> > > reference manual [6].
>>>> > >>>>>>> > > * Java artifacts were built
>>>> with
>>>> > >>>>>>> Gradle 4.8 and OpenJDK
>>>> > >>>>>>> > >
>>>> 1.8.0_181-8u181-b13-1~deb9u1-b13.
>>>> > >>>>>>> > > * Python artifacts are
>>>> deployed
>>>> > >>>>>>> along with the source release
>>>> > >>>>>>> > to the
>>>> > >>>>>>> > > dist.apache.org
>>>> > <http://dist.apache.org>
>>>> > >>>>>>> <http://dist.apache.org/>
>>>> > <http://dist.apache.org
>>>> > >>>>>>> <http://dist.apache.org/>>
>>>> > >>>>>>> > <http://dist.apache.org
>>>> > >>>>>>> <http://dist.apache.org/>> [2].
>>>> > >>>>>>> > >
>>>> > >>>>>>> > > The vote will be open for at
>>>> least
>>>> > >>>>>>> 72 hours. It is adopted by
>>>> > >>>>>>> > majority
>>>> > >>>>>>> > > approval, with at least 3 PMC
>>>> > >>>>>>> affirmative votes.
>>>> > >>>>>>> > >
>>>> > >>>>>>> > > Thanks,
>>>> > >>>>>>> > > Charles
>>>> > >>>>>>> > >
>>>> > >>>>>>> > > [1]
>>>> > >>>>>>> > >
>>>> > >>>>>>> >
>>>> > >>>>>>>
>>>> >
>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>>>> > >>>>>>> > > [2]
>>>> > >>>>>>>
>>>> > https://dist.apache.org/repos/dist/dev/beam/2.7.0
>>>> > >>>>>>> > > [3]
>>>> > >>>>>>>
>>>> https://dist.apache.org/repos/dist/dev/beam/KEYS
>>>> > >>>>>>> > > [4]
>>>> > >>>>>>> >
>>>> > >>>>>>>
>>>> >
>>>> https://repository.apache.org/content/repositories/orgapachebeam-1046/
>>>> > >>>>>>> > > [5]
>>>> > >>>>>>>
>>>> https://github.com/apache/beam/tree/v2.7.0-RC1
>>>> > >>>>>>> > > [6]
>>>> > >>>>>>>
>>>> https://github.com/apache/beam-site/pull/549
>>>> > >>>>>>> >
>>>> > >>>>>>> > --
>>>> > >>>>>>> > Jean-Baptiste Onofré
>>>> > >>>>>>> > jbonofre@apache.org
>>>> > <ma...@apache.org>
>>>> > >>>>>>> <mailto:jbonofre@apache.org
>>>> > <ma...@apache.org>>
>>>> > >>>>>>> <mailto:jbonofre@apache.org
>>>> > <ma...@apache.org>
>>>> > >>>>>>> <mailto:jbonofre@apache.org
>>>> > <ma...@apache.org>>>
>>>> > >>>>>>> > http://blog.nanthrax.net
>>>> > >>>>>>> <http://blog.nanthrax.net/>
>>>> > >>>>>>> > Talend - http://www.talend.com
>>>> > >>>>>>> <http://www.talend.com/>
>>>> > >>>>>>> >
>>>> > >>>>
>>>> > >
>>>> >
>>>> > --
>>>> > Jean-Baptiste Onofré
>>>> > jbonofre@apache.org <ma...@apache.org>
>>>> > http://blog.nanthrax.net
>>>> > Talend - http://www.talend.com
>>>> >
>>>>
>>>> --
>>>> Jean-Baptiste Onofré
>>>> jbonofre@apache.org
>>>> http://blog.nanthrax.net
>>>> Talend - http://www.talend.com
>>>>
>>>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Charles Chen <cc...@google.com>.
Hey Romain and JB, do you have any progress on this? One thing I would
like to point out is that 2.7.0 isn't yet pushed to Maven Central, so
referring to it by version is not expected to work (and it looks like this
is what is done in your repo:
https://github.com/rmannibucau/beam-2.7.0-fails). Luke indicated above
that he doesn't see any dependency changes. Can you isolate and reproduce
this problem so that we can develop a fix, if necessary? I would like to
proceed with an RC2 as soon as possible.
On Wed, Sep 19, 2018 at 6:37 AM Romain Manni-Bucau <rm...@gmail.com>
wrote:
> Quick update on the spark issue: I didn't get enough time to identify it
> clearly but managed to have a passing run of my test changing a bunch of
> versions.
> I suspect my code triggers some class conflicting between spark and my
> shade leading to a serialization issue. I didn't test userClassPathFirst
> option of spark but it can be an interesting thing to enable in beam runner.
> However it is still very confusing to have it not running just upgrading
> beam version and the spark error is very hard to understand.
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github
> <https://github.com/rmannibucau> | LinkedIn
> <https://www.linkedin.com/in/rmannibucau> | Book
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>
>
> Le mar. 18 sept. 2018 à 20:17, Lukasz Cwik <lc...@google.com> a écrit :
>
>> Romain hinted that this was a dependency issue but when comparing the two
>> dependency trees I don't get much of a difference:
>>
>> lcwik@lcwik0: ~$ diff /tmp/260 /tmp/270
>> < [INFO] +- org.apache.beam:beam-runners-spark:jar:2.6.0:compile
>> < [INFO] | +- org.apache.beam:beam-model-pipeline:jar:2.6.0:compile
>> ---
>> > [INFO] +- org.apache.beam:beam-runners-spark:jar:2.7.0:compile
>> > [INFO] | +- org.apache.beam:beam-model-pipeline:jar:2.7.0:compile
>> 5c6
>> < [INFO] | +- org.apache.beam:beam-sdks-java-core:jar:2.6.0:compile
>> ---
>> > [INFO] | +- org.apache.beam:beam-sdks-java-core:jar:2.7.0:compile
>> 14,18c15,19
>> < [INFO] | | \- org.tukaani:xz:jar:1.5:compile
>> < [INFO] | +-
>> org.apache.beam:beam-runners-core-construction-java:jar:2.6.0:compile
>> < [INFO] | | \-
>> org.apache.beam:beam-model-job-management:jar:2.6.0:compile
>> < [INFO] | +- org.apache.beam:beam-runners-core-java:jar:2.6.0:compile
>> < [INFO] | | \-
>> org.apache.beam:beam-model-fn-execution:jar:2.6.0:compile
>> ---
>> > [INFO] | | \- org.tukaani:xz:jar:1.8:compile
>> > [INFO] | +-
>> org.apache.beam:beam-runners-core-construction-java:jar:2.7.0:compile
>> > [INFO] | | \-
>> org.apache.beam:beam-model-job-management:jar:2.7.0:compile
>> > [INFO] | +- org.apache.beam:beam-runners-core-java:jar:2.7.0:compile
>> > [INFO] | | \-
>> org.apache.beam:beam-model-fn-execution:jar:2.7.0:compile
>>
>> Other then Beam package changes, the only other change is xz which I
>> don't believe could be causing the issue.
>>
>> On Tue, Sep 18, 2018 at 8:38 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>> wrote:
>>
>>> Thanks, let me take a look.
>>>
>>> Regards
>>> JB
>>>
>>> On 18/09/2018 17:36, Romain Manni-Bucau wrote:
>>> >
>>> >
>>> >
>>> > Le mar. 18 sept. 2018 à 16:44, Jean-Baptiste Onofré <jb@nanthrax.net
>>> > <ma...@nanthrax.net>> a écrit :
>>> >
>>> > Hi,
>>> >
>>> > I don't have the issue ;)
>>> >
>>> > As said in my vote, I tested 2.7.0 RC1 on beam-samples with Spark
>>> > without problem.
>>> >
>>> > I don't reproduce Romain issue as well.
>>> >
>>> > @Romain can you provide some details to reproduce the issue ?
>>> >
>>> >
>>> > Sure, you can use this
>>> > reproducer: https://github.com/rmannibucau/beam-2.7.0-fails
>>> > It shows that it suceeds on 2.6 and fails on 2.7.
>>> >
>>> >
>>> >
>>> > Regards
>>> > JB
>>> >
>>> > On 17/09/2018 19:17, Charles Chen wrote:
>>> > > Luke, Maximillian, Raghu, can you please propose cherry-pick PRs
>>> > to the
>>> > > release-2.7.0 for your issues and add me as a reviewer
>>> > (@charlesccychen)?
>>> > >
>>> > > Romain, JB: is there any way I can help with debugging the issue
>>> > you're
>>> > > facing so we can unblock the release?
>>> > >
>>> > > On Fri, Sep 14, 2018 at 1:49 PM Raghu Angadi <rangadi@google.com
>>> > <ma...@google.com>
>>> > > <mailto:rangadi@google.com <ma...@google.com>>> wrote:
>>> > >
>>> > > I would like propose one more cherrypick for RC2
>>> > > : https://github.com/apache/beam/pull/6391
>>> > > This is a KafkaIO bug fix. Once a user hits this bug, there
>>> is no
>>> > > easy work around for them, especially on Dataflow. Only work
>>> > around
>>> > > in Dataflow is to restart or reload the job.
>>> > >
>>> > > The fix itself fairly safe and is tested.
>>> > > Raghu.
>>> > >
>>> > > On Fri, Sep 14, 2018 at 12:52 AM Alexey Romanenko
>>> > > <aromanenko.dev@gmail.com <ma...@gmail.com>
>>> > <mailto:aromanenko.dev@gmail.com <mailto:aromanenko.dev@gmail.com
>>> >>>
>>> > wrote:
>>> > >
>>> > > Perhaps it could help, but I run simple WordCount (built
>>> with
>>> > > Beam 2.7) on YARN/Spark (HDP Sandbox) cluster and it
>>> > worked fine
>>> > > for me.
>>> > >
>>> > >> On 14 Sep 2018, at 06:56, Romain Manni-Bucau
>>> > >> <rmannibucau@gmail.com <ma...@gmail.com>
>>> > <mailto:rmannibucau@gmail.com <ma...@gmail.com>>>
>>> wrote:
>>> > >>
>>> > >> Hi Charles,
>>> > >>
>>> > >> I didn't get enough time to check deeply but it is
>>> clearly a
>>> > >> dependency issue and it is not in beam spark runner
>>> > itself but
>>> > >> in another transitive module of beam. It does not
>>> happen in
>>> > >> existing spark test cause none of them are in a cluster
>>> (even
>>> > >> just with 1 worker) but this seems to be a regression
>>> since
>>> > >> 2.6 works OOTB.
>>> > >>
>>> > >> Romain Manni-Bucau
>>> > >> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>> > >> <https://rmannibucau.metawerx.net/> | Old Blog
>>> > >> <http://rmannibucau.wordpress.com/> | Github
>>> > >> <https://github.com/rmannibucau> | LinkedIn
>>> > >> <https://www.linkedin.com/in/rmannibucau> | Book
>>> > >>
>>> > <
>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>> >
>>> > >>
>>> > >>
>>> > >> Le jeu. 13 sept. 2018 à 22:15, Charles Chen
>>> > <ccy@google.com <ma...@google.com>
>>> > >> <mailto:ccy@google.com <ma...@google.com>>> a
>>> écrit :
>>> > >>
>>> > >> Romain and JB, can you please add the results of
>>> your
>>> > >> investigations into the errors you've seen above?
>>> Given
>>> > >> that the existing SparkRunner tests pass for this
>>> RC, and
>>> > >> that the integration test you ran is in another
>>> repo that
>>> > >> is not continuously tested with Beam, it is not
>>> clear how
>>> > >> we should move forward and whether this is a
>>> blocking
>>> > >> issue, unless we can find a root cause in Beam.
>>> > >>
>>> > >> On Wed, Sep 12, 2018 at 2:08 AM Etienne Chauchot
>>> > >> <echauchot@apache.org <ma...@apache.org>
>>> > <mailto:echauchot@apache.org <ma...@apache.org>>>
>>> wrote:
>>> > >>
>>> > >> Hi all,
>>> > >>
>>> > >> on a performance and functional regression stand
>>> > point
>>> > >> I see no regression:
>>> > >>
>>> > >> I looked at nexmark graphs "output pcollection
>>> size"
>>> > >> and "execution time" around release cut date on
>>> > >> dataflow, spark, flink and direct runner in
>>> batch and
>>> > >> streaming modes. There seems to be no
>>> regression.
>>> > >>
>>> > >> Etienne
>>> > >>
>>> > >> Le mardi 11 septembre 2018 à 12:25 -0700,
>>> Charles
>>> > Chen
>>> > >> a écrit :
>>> > >>> The SparkRunner validation test
>>> > >>>
>>> > (here:
>>> https://beam.apache.org/contribute/release-guide/#run-validation-tests)
>>> > >>> passes on my machine. It looks like we are
>>> likely
>>> > >>> missing test coverage where Romain is hitting
>>> > issues.
>>> > >>>
>>> > >>> On Tue, Sep 11, 2018 at 12:15 PM Ahmet Altay
>>> > >>> <altay@google.com <ma...@google.com>
>>> > <mailto:altay@google.com <ma...@google.com>>> wrote:
>>> > >>>> Could anyone else help with looking at these
>>> issues
>>> > >>>> earlier?
>>> > >>>>
>>> > >>>> On Tue, Sep 11, 2018 at 12:03 PM, Romain
>>> > Manni-Bucau
>>> > >>>> <rmannibucau@gmail.com
>>> > <ma...@gmail.com>
>>> > >>>> <mailto:rmannibucau@gmail.com
>>> > <ma...@gmail.com>>> wrote:
>>> > >>>>> Im running this main [1] through this IT
>>> [2]. Was
>>> > >>>>> working fine since ~1 year but 2.7.0 broke
>>> it.
>>> > >>>>> Didnt investigate more but can have a look
>>> later
>>> > >>>>> this month if it helps.
>>> > >>>>>
>>> > >>>>>
>>> > [1]
>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
>>> > >>>>>
>>> > [2]
>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
>>> > >>>>>
>>> > >>>>> Le mar. 11 sept. 2018 20:54, Charles Chen
>>> > >>>>> <ccy@google.com <ma...@google.com>
>>> > <mailto:ccy@google.com <ma...@google.com>>> a écrit :
>>> > >>>>>> Romain: can you give more details on the
>>> failure
>>> > >>>>>> you're encountering, i.e. how you are
>>> performing
>>> > >>>>>> this validation?
>>> > >>>>>>
>>> > >>>>>> On Tue, Sep 11, 2018 at 9:36 AM
>>> Jean-Baptiste
>>> > >>>>>> Onofré <jb@nanthrax.net
>>> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
>>> > <ma...@nanthrax.net>>>
>>> > >>>>>> wrote:
>>> > >>>>>>> Hi,
>>> > >>>>>>>
>>> > >>>>>>> weird, I didn't have it on Beam samples.
>>> Let me
>>> > >>>>>>> try to reproduce and I
>>> > >>>>>>> will create the Jira.
>>> > >>>>>>>
>>> > >>>>>>> Regards
>>> > >>>>>>> JB
>>> > >>>>>>>
>>> > >>>>>>> On 11/09/2018 11:44, Romain Manni-Bucau
>>> wrote:
>>> > >>>>>>> > -1, seems spark integration is broken
>>> (tested
>>> > >>>>>>> with spark 2.3.1 and 2.2.1):
>>> > >>>>>>> >
>>> > >>>>>>> > 18/09/11 11:33:29 WARN TaskSetManager:
>>> Lost
>>> > >>>>>>> task 0.0 in stage 0.0 (TID 0, RMANNIBUCAU,
>>> > >>>>>>> executor 0): java.lang.ClassCastException:
>>> > cannot
>>> > >>>>>>> assign instance of
>>> > >>>>>>>
>>> > scala.collection.immutable.List$SerializationProxy to
>>> > >>>>>>> fieldorg.apache.spark.rdd.RDD.org
>>> > <http://fieldorg.apache.spark.rdd.RDD.org>
>>> > >>>>>>> <http://fieldorg.apache.spark.rdd.rdd.org/
>>> >
>>> > >>>>>>> <http://org.apache.spark.rdd.RDD.org
>>> > >>>>>>>
>>> > <http://org.apache.spark.rdd.rdd.org/
>>> >>$apache$spark$rdd$RDD$$dependencies_
>>> > >>>>>>> of type scala.collection.Seq in instance of
>>> > >>>>>>> org.apache.spark.rdd.MapPartitionsRDD
>>> > >>>>>>> > at
>>> > >>>>>>>
>>> >
>>> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>>> > >>>>>>> >
>>> > >>>>>>> >
>>> > >>>>>>> > Also the issue Lukasz identified is
>>> important
>>> > >>>>>>> even if workarounds can be
>>> > >>>>>>> > put in place so +1 to fix it as well if
>>> > possible.
>>> > >>>>>>> >
>>> > >>>>>>> > Romain Manni-Bucau
>>> > >>>>>>> > @rmannibucau <
>>> https://twitter.com/rmannibucau>
>>> > >>>>>>> | Blog
>>> > >>>>>>> > <https://rmannibucau.metawerx.net/> |
>>> Old Blog
>>> > >>>>>>> > <http://rmannibucau.wordpress.com
>>> > >>>>>>> <http://rmannibucau.wordpress.com/>> |
>>> Github
>>> > >>>>>>> > <https://github.com/rmannibucau> |
>>> LinkedIn
>>> > >>>>>>> >
>>> > <https://www.linkedin.com/in/rmannibucau> | Book
>>> > >>>>>>> >
>>> > >>>>>>>
>>> > <
>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>> >
>>> > >>>>>>> >
>>> > >>>>>>> >
>>> > >>>>>>> > Le lun. 10 sept. 2018 à 20:48, Lukasz
>>> Cwik
>>> > >>>>>>> <lcwik@google.com <mailto:lcwik@google.com
>>> >
>>> > <mailto:lcwik@google.com <ma...@google.com>>
>>> > >>>>>>> > <mailto:lcwik@google.com
>>> > <ma...@google.com>
>>> > >>>>>>> <mailto:lcwik@google.com
>>> > <ma...@google.com>>>> a écrit :
>>> > >>>>>>> >
>>> > >>>>>>> > I found an issue where we are no
>>> longer
>>> > >>>>>>> packaging the pom.xml within
>>> > >>>>>>> > the artifact jars at
>>> > >>>>>>> META-INF/maven/groupId/artifactId. More
>>> details
>>> > >>>>>>> > in
>>> > >>>>>>>
>>> https://issues.apache.org/jira/browse/BEAM-5351.
>>> > >>>>>>> I wouldn't
>>> > >>>>>>> > consider this a blocker but it was an
>>> > easy fix
>>> > >>>>>>> > (
>>> https://github.com/apache/beam/pull/6358)
>>> > >>>>>>> and users may rely on the
>>> > >>>>>>> > pom.xml.
>>> > >>>>>>> >
>>> > >>>>>>> > Should we recut the release
>>> candidate to
>>> > >>>>>>> include this?
>>> > >>>>>>> >
>>> > >>>>>>> > On Mon, Sep 10, 2018 at 4:58 AM
>>> > >>>>>>> Jean-Baptiste Onofré
>>> > >>>>>>> > <jb@nanthrax.net
>>> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
>>> > <ma...@nanthrax.net>>
>>> > >>>>>>> <mailto:jb@nanthrax.net <mailto:
>>> jb@nanthrax.net>
>>> > >>>>>>> <mailto:jb@nanthrax.net
>>> > <ma...@nanthrax.net>>>> wrote:
>>> > >>>>>>> >
>>> > >>>>>>> > +1 (binding)
>>> > >>>>>>> >
>>> > >>>>>>> > Tested successfully on Beam
>>> Samples.
>>> > >>>>>>> >
>>> > >>>>>>> > Thanks !
>>> > >>>>>>> >
>>> > >>>>>>> > Regards
>>> > >>>>>>> > JB
>>> > >>>>>>> >
>>> > >>>>>>> > On 07/09/2018 23:56, Charles Chen
>>> > wrote:
>>> > >>>>>>> > > Hi everyone,
>>> > >>>>>>> > >
>>> > >>>>>>> > > Please review and vote on the
>>> > >>>>>>> release candidate #1 for the
>>> > >>>>>>> > version
>>> > >>>>>>> > > 2.7.0, as follows:
>>> > >>>>>>> > > [ ] +1, Approve the release
>>> > >>>>>>> > > [ ] -1, Do not approve the
>>> release
>>> > >>>>>>> (please provide specific
>>> > >>>>>>> > comments)
>>> > >>>>>>> > >
>>> > >>>>>>> > > The complete staging area is
>>> > >>>>>>> available for your review, which
>>> > >>>>>>> > includes:
>>> > >>>>>>> > > * JIRA release notes [1],
>>> > >>>>>>> > > * the official Apache source
>>> > release
>>> > >>>>>>> to be deployed to
>>> > >>>>>>> > dist.apache.org
>>> > <http://dist.apache.org>
>>> > >>>>>>> <http://dist.apache.org/>
>>> > <http://dist.apache.org
>>> > >>>>>>> <http://dist.apache.org/>>
>>> > >>>>>>> > > <http://dist.apache.org
>>> > >>>>>>> <http://dist.apache.org/>> [2], which is
>>> signed
>>> > >>>>>>> with the key with
>>> > >>>>>>> > > fingerprint 45C60AAAD115F560
>>> [3],
>>> > >>>>>>> > > * all artifacts to be
>>> deployed to
>>> > >>>>>>> the Maven Central
>>> > >>>>>>> > Repository [4],
>>> > >>>>>>> > > * source code tag
>>> "v2.7.0-RC1" [5],
>>> > >>>>>>> > > * website pull request
>>> listing the
>>> > >>>>>>> release and publishing the API
>>> > >>>>>>> > > reference manual [6].
>>> > >>>>>>> > > * Java artifacts were built
>>> with
>>> > >>>>>>> Gradle 4.8 and OpenJDK
>>> > >>>>>>> > >
>>> 1.8.0_181-8u181-b13-1~deb9u1-b13.
>>> > >>>>>>> > > * Python artifacts are
>>> deployed
>>> > >>>>>>> along with the source release
>>> > >>>>>>> > to the
>>> > >>>>>>> > > dist.apache.org
>>> > <http://dist.apache.org>
>>> > >>>>>>> <http://dist.apache.org/>
>>> > <http://dist.apache.org
>>> > >>>>>>> <http://dist.apache.org/>>
>>> > >>>>>>> > <http://dist.apache.org
>>> > >>>>>>> <http://dist.apache.org/>> [2].
>>> > >>>>>>> > >
>>> > >>>>>>> > > The vote will be open for at
>>> least
>>> > >>>>>>> 72 hours. It is adopted by
>>> > >>>>>>> > majority
>>> > >>>>>>> > > approval, with at least 3 PMC
>>> > >>>>>>> affirmative votes.
>>> > >>>>>>> > >
>>> > >>>>>>> > > Thanks,
>>> > >>>>>>> > > Charles
>>> > >>>>>>> > >
>>> > >>>>>>> > > [1]
>>> > >>>>>>> > >
>>> > >>>>>>> >
>>> > >>>>>>>
>>> >
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>>> > >>>>>>> > > [2]
>>> > >>>>>>>
>>> > https://dist.apache.org/repos/dist/dev/beam/2.7.0
>>> > >>>>>>> > > [3]
>>> > >>>>>>>
>>> https://dist.apache.org/repos/dist/dev/beam/KEYS
>>> > >>>>>>> > > [4]
>>> > >>>>>>> >
>>> > >>>>>>>
>>> >
>>> https://repository.apache.org/content/repositories/orgapachebeam-1046/
>>> > >>>>>>> > > [5]
>>> > >>>>>>>
>>> https://github.com/apache/beam/tree/v2.7.0-RC1
>>> > >>>>>>> > > [6]
>>> > >>>>>>>
>>> https://github.com/apache/beam-site/pull/549
>>> > >>>>>>> >
>>> > >>>>>>> > --
>>> > >>>>>>> > Jean-Baptiste Onofré
>>> > >>>>>>> > jbonofre@apache.org
>>> > <ma...@apache.org>
>>> > >>>>>>> <mailto:jbonofre@apache.org
>>> > <ma...@apache.org>>
>>> > >>>>>>> <mailto:jbonofre@apache.org
>>> > <ma...@apache.org>
>>> > >>>>>>> <mailto:jbonofre@apache.org
>>> > <ma...@apache.org>>>
>>> > >>>>>>> > http://blog.nanthrax.net
>>> > >>>>>>> <http://blog.nanthrax.net/>
>>> > >>>>>>> > Talend - http://www.talend.com
>>> > >>>>>>> <http://www.talend.com/>
>>> > >>>>>>> >
>>> > >>>>
>>> > >
>>> >
>>> > --
>>> > Jean-Baptiste Onofré
>>> > jbonofre@apache.org <ma...@apache.org>
>>> > http://blog.nanthrax.net
>>> > Talend - http://www.talend.com
>>> >
>>>
>>> --
>>> Jean-Baptiste Onofré
>>> jbonofre@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>>
>>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Quick update on the spark issue: I didn't get enough time to identify it
clearly but managed to have a passing run of my test changing a bunch of
versions.
I suspect my code triggers some class conflicting between spark and my
shade leading to a serialization issue. I didn't test userClassPathFirst
option of spark but it can be an interesting thing to enable in beam runner.
However it is still very confusing to have it not running just upgrading
beam version and the spark error is very hard to understand.
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>
Le mar. 18 sept. 2018 à 20:17, Lukasz Cwik <lc...@google.com> a écrit :
> Romain hinted that this was a dependency issue but when comparing the two
> dependency trees I don't get much of a difference:
>
> lcwik@lcwik0: ~$ diff /tmp/260 /tmp/270
> < [INFO] +- org.apache.beam:beam-runners-spark:jar:2.6.0:compile
> < [INFO] | +- org.apache.beam:beam-model-pipeline:jar:2.6.0:compile
> ---
> > [INFO] +- org.apache.beam:beam-runners-spark:jar:2.7.0:compile
> > [INFO] | +- org.apache.beam:beam-model-pipeline:jar:2.7.0:compile
> 5c6
> < [INFO] | +- org.apache.beam:beam-sdks-java-core:jar:2.6.0:compile
> ---
> > [INFO] | +- org.apache.beam:beam-sdks-java-core:jar:2.7.0:compile
> 14,18c15,19
> < [INFO] | | \- org.tukaani:xz:jar:1.5:compile
> < [INFO] | +-
> org.apache.beam:beam-runners-core-construction-java:jar:2.6.0:compile
> < [INFO] | | \-
> org.apache.beam:beam-model-job-management:jar:2.6.0:compile
> < [INFO] | +- org.apache.beam:beam-runners-core-java:jar:2.6.0:compile
> < [INFO] | | \- org.apache.beam:beam-model-fn-execution:jar:2.6.0:compile
> ---
> > [INFO] | | \- org.tukaani:xz:jar:1.8:compile
> > [INFO] | +-
> org.apache.beam:beam-runners-core-construction-java:jar:2.7.0:compile
> > [INFO] | | \-
> org.apache.beam:beam-model-job-management:jar:2.7.0:compile
> > [INFO] | +- org.apache.beam:beam-runners-core-java:jar:2.7.0:compile
> > [INFO] | | \- org.apache.beam:beam-model-fn-execution:jar:2.7.0:compile
>
> Other then Beam package changes, the only other change is xz which I don't
> believe could be causing the issue.
>
> On Tue, Sep 18, 2018 at 8:38 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> Thanks, let me take a look.
>>
>> Regards
>> JB
>>
>> On 18/09/2018 17:36, Romain Manni-Bucau wrote:
>> >
>> >
>> >
>> > Le mar. 18 sept. 2018 à 16:44, Jean-Baptiste Onofré <jb@nanthrax.net
>> > <ma...@nanthrax.net>> a écrit :
>> >
>> > Hi,
>> >
>> > I don't have the issue ;)
>> >
>> > As said in my vote, I tested 2.7.0 RC1 on beam-samples with Spark
>> > without problem.
>> >
>> > I don't reproduce Romain issue as well.
>> >
>> > @Romain can you provide some details to reproduce the issue ?
>> >
>> >
>> > Sure, you can use this
>> > reproducer: https://github.com/rmannibucau/beam-2.7.0-fails
>> > It shows that it suceeds on 2.6 and fails on 2.7.
>> >
>> >
>> >
>> > Regards
>> > JB
>> >
>> > On 17/09/2018 19:17, Charles Chen wrote:
>> > > Luke, Maximillian, Raghu, can you please propose cherry-pick PRs
>> > to the
>> > > release-2.7.0 for your issues and add me as a reviewer
>> > (@charlesccychen)?
>> > >
>> > > Romain, JB: is there any way I can help with debugging the issue
>> > you're
>> > > facing so we can unblock the release?
>> > >
>> > > On Fri, Sep 14, 2018 at 1:49 PM Raghu Angadi <rangadi@google.com
>> > <ma...@google.com>
>> > > <mailto:rangadi@google.com <ma...@google.com>>> wrote:
>> > >
>> > > I would like propose one more cherrypick for RC2
>> > > : https://github.com/apache/beam/pull/6391
>> > > This is a KafkaIO bug fix. Once a user hits this bug, there
>> is no
>> > > easy work around for them, especially on Dataflow. Only work
>> > around
>> > > in Dataflow is to restart or reload the job.
>> > >
>> > > The fix itself fairly safe and is tested.
>> > > Raghu.
>> > >
>> > > On Fri, Sep 14, 2018 at 12:52 AM Alexey Romanenko
>> > > <aromanenko.dev@gmail.com <ma...@gmail.com>
>> > <mailto:aromanenko.dev@gmail.com <mailto:aromanenko.dev@gmail.com
>> >>>
>> > wrote:
>> > >
>> > > Perhaps it could help, but I run simple WordCount (built
>> with
>> > > Beam 2.7) on YARN/Spark (HDP Sandbox) cluster and it
>> > worked fine
>> > > for me.
>> > >
>> > >> On 14 Sep 2018, at 06:56, Romain Manni-Bucau
>> > >> <rmannibucau@gmail.com <ma...@gmail.com>
>> > <mailto:rmannibucau@gmail.com <ma...@gmail.com>>>
>> wrote:
>> > >>
>> > >> Hi Charles,
>> > >>
>> > >> I didn't get enough time to check deeply but it is
>> clearly a
>> > >> dependency issue and it is not in beam spark runner
>> > itself but
>> > >> in another transitive module of beam. It does not happen
>> in
>> > >> existing spark test cause none of them are in a cluster
>> (even
>> > >> just with 1 worker) but this seems to be a regression
>> since
>> > >> 2.6 works OOTB.
>> > >>
>> > >> Romain Manni-Bucau
>> > >> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>> > >> <https://rmannibucau.metawerx.net/> | Old Blog
>> > >> <http://rmannibucau.wordpress.com/> | Github
>> > >> <https://github.com/rmannibucau> | LinkedIn
>> > >> <https://www.linkedin.com/in/rmannibucau> | Book
>> > >>
>> > <
>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>> >
>> > >>
>> > >>
>> > >> Le jeu. 13 sept. 2018 à 22:15, Charles Chen
>> > <ccy@google.com <ma...@google.com>
>> > >> <mailto:ccy@google.com <ma...@google.com>>> a
>> écrit :
>> > >>
>> > >> Romain and JB, can you please add the results of your
>> > >> investigations into the errors you've seen above?
>> Given
>> > >> that the existing SparkRunner tests pass for this
>> RC, and
>> > >> that the integration test you ran is in another repo
>> that
>> > >> is not continuously tested with Beam, it is not
>> clear how
>> > >> we should move forward and whether this is a blocking
>> > >> issue, unless we can find a root cause in Beam.
>> > >>
>> > >> On Wed, Sep 12, 2018 at 2:08 AM Etienne Chauchot
>> > >> <echauchot@apache.org <ma...@apache.org>
>> > <mailto:echauchot@apache.org <ma...@apache.org>>> wrote:
>> > >>
>> > >> Hi all,
>> > >>
>> > >> on a performance and functional regression stand
>> > point
>> > >> I see no regression:
>> > >>
>> > >> I looked at nexmark graphs "output pcollection
>> size"
>> > >> and "execution time" around release cut date on
>> > >> dataflow, spark, flink and direct runner in
>> batch and
>> > >> streaming modes. There seems to be no regression.
>> > >>
>> > >> Etienne
>> > >>
>> > >> Le mardi 11 septembre 2018 à 12:25 -0700, Charles
>> > Chen
>> > >> a écrit :
>> > >>> The SparkRunner validation test
>> > >>>
>> > (here:
>> https://beam.apache.org/contribute/release-guide/#run-validation-tests)
>> > >>> passes on my machine. It looks like we are
>> likely
>> > >>> missing test coverage where Romain is hitting
>> > issues.
>> > >>>
>> > >>> On Tue, Sep 11, 2018 at 12:15 PM Ahmet Altay
>> > >>> <altay@google.com <ma...@google.com>
>> > <mailto:altay@google.com <ma...@google.com>>> wrote:
>> > >>>> Could anyone else help with looking at these
>> issues
>> > >>>> earlier?
>> > >>>>
>> > >>>> On Tue, Sep 11, 2018 at 12:03 PM, Romain
>> > Manni-Bucau
>> > >>>> <rmannibucau@gmail.com
>> > <ma...@gmail.com>
>> > >>>> <mailto:rmannibucau@gmail.com
>> > <ma...@gmail.com>>> wrote:
>> > >>>>> Im running this main [1] through this IT [2].
>> Was
>> > >>>>> working fine since ~1 year but 2.7.0 broke it.
>> > >>>>> Didnt investigate more but can have a look
>> later
>> > >>>>> this month if it helps.
>> > >>>>>
>> > >>>>>
>> > [1]
>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
>> > >>>>>
>> > [2]
>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
>> > >>>>>
>> > >>>>> Le mar. 11 sept. 2018 20:54, Charles Chen
>> > >>>>> <ccy@google.com <ma...@google.com>
>> > <mailto:ccy@google.com <ma...@google.com>>> a écrit :
>> > >>>>>> Romain: can you give more details on the
>> failure
>> > >>>>>> you're encountering, i.e. how you are
>> performing
>> > >>>>>> this validation?
>> > >>>>>>
>> > >>>>>> On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste
>> > >>>>>> Onofré <jb@nanthrax.net
>> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
>> > <ma...@nanthrax.net>>>
>> > >>>>>> wrote:
>> > >>>>>>> Hi,
>> > >>>>>>>
>> > >>>>>>> weird, I didn't have it on Beam samples.
>> Let me
>> > >>>>>>> try to reproduce and I
>> > >>>>>>> will create the Jira.
>> > >>>>>>>
>> > >>>>>>> Regards
>> > >>>>>>> JB
>> > >>>>>>>
>> > >>>>>>> On 11/09/2018 11:44, Romain Manni-Bucau
>> wrote:
>> > >>>>>>> > -1, seems spark integration is broken
>> (tested
>> > >>>>>>> with spark 2.3.1 and 2.2.1):
>> > >>>>>>> >
>> > >>>>>>> > 18/09/11 11:33:29 WARN TaskSetManager:
>> Lost
>> > >>>>>>> task 0.0 in stage 0.0 (TID 0, RMANNIBUCAU,
>> > >>>>>>> executor 0): java.lang.ClassCastException:
>> > cannot
>> > >>>>>>> assign instance of
>> > >>>>>>>
>> > scala.collection.immutable.List$SerializationProxy to
>> > >>>>>>> fieldorg.apache.spark.rdd.RDD.org
>> > <http://fieldorg.apache.spark.rdd.RDD.org>
>> > >>>>>>> <http://fieldorg.apache.spark.rdd.rdd.org/>
>> > >>>>>>> <http://org.apache.spark.rdd.RDD.org
>> > >>>>>>>
>> > <http://org.apache.spark.rdd.rdd.org/
>> >>$apache$spark$rdd$RDD$$dependencies_
>> > >>>>>>> of type scala.collection.Seq in instance of
>> > >>>>>>> org.apache.spark.rdd.MapPartitionsRDD
>> > >>>>>>> > at
>> > >>>>>>>
>> >
>> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>> > >>>>>>> >
>> > >>>>>>> >
>> > >>>>>>> > Also the issue Lukasz identified is
>> important
>> > >>>>>>> even if workarounds can be
>> > >>>>>>> > put in place so +1 to fix it as well if
>> > possible.
>> > >>>>>>> >
>> > >>>>>>> > Romain Manni-Bucau
>> > >>>>>>> > @rmannibucau <
>> https://twitter.com/rmannibucau>
>> > >>>>>>> | Blog
>> > >>>>>>> > <https://rmannibucau.metawerx.net/> |
>> Old Blog
>> > >>>>>>> > <http://rmannibucau.wordpress.com
>> > >>>>>>> <http://rmannibucau.wordpress.com/>> |
>> Github
>> > >>>>>>> > <https://github.com/rmannibucau> |
>> LinkedIn
>> > >>>>>>> >
>> > <https://www.linkedin.com/in/rmannibucau> | Book
>> > >>>>>>> >
>> > >>>>>>>
>> > <
>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>> >
>> > >>>>>>> >
>> > >>>>>>> >
>> > >>>>>>> > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik
>> > >>>>>>> <lcwik@google.com <ma...@google.com>
>> > <mailto:lcwik@google.com <ma...@google.com>>
>> > >>>>>>> > <mailto:lcwik@google.com
>> > <ma...@google.com>
>> > >>>>>>> <mailto:lcwik@google.com
>> > <ma...@google.com>>>> a écrit :
>> > >>>>>>> >
>> > >>>>>>> > I found an issue where we are no
>> longer
>> > >>>>>>> packaging the pom.xml within
>> > >>>>>>> > the artifact jars at
>> > >>>>>>> META-INF/maven/groupId/artifactId. More
>> details
>> > >>>>>>> > in
>> > >>>>>>>
>> https://issues.apache.org/jira/browse/BEAM-5351.
>> > >>>>>>> I wouldn't
>> > >>>>>>> > consider this a blocker but it was an
>> > easy fix
>> > >>>>>>> > (
>> https://github.com/apache/beam/pull/6358)
>> > >>>>>>> and users may rely on the
>> > >>>>>>> > pom.xml.
>> > >>>>>>> >
>> > >>>>>>> > Should we recut the release candidate
>> to
>> > >>>>>>> include this?
>> > >>>>>>> >
>> > >>>>>>> > On Mon, Sep 10, 2018 at 4:58 AM
>> > >>>>>>> Jean-Baptiste Onofré
>> > >>>>>>> > <jb@nanthrax.net
>> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
>> > <ma...@nanthrax.net>>
>> > >>>>>>> <mailto:jb@nanthrax.net <mailto:
>> jb@nanthrax.net>
>> > >>>>>>> <mailto:jb@nanthrax.net
>> > <ma...@nanthrax.net>>>> wrote:
>> > >>>>>>> >
>> > >>>>>>> > +1 (binding)
>> > >>>>>>> >
>> > >>>>>>> > Tested successfully on Beam
>> Samples.
>> > >>>>>>> >
>> > >>>>>>> > Thanks !
>> > >>>>>>> >
>> > >>>>>>> > Regards
>> > >>>>>>> > JB
>> > >>>>>>> >
>> > >>>>>>> > On 07/09/2018 23:56, Charles Chen
>> > wrote:
>> > >>>>>>> > > Hi everyone,
>> > >>>>>>> > >
>> > >>>>>>> > > Please review and vote on the
>> > >>>>>>> release candidate #1 for the
>> > >>>>>>> > version
>> > >>>>>>> > > 2.7.0, as follows:
>> > >>>>>>> > > [ ] +1, Approve the release
>> > >>>>>>> > > [ ] -1, Do not approve the
>> release
>> > >>>>>>> (please provide specific
>> > >>>>>>> > comments)
>> > >>>>>>> > >
>> > >>>>>>> > > The complete staging area is
>> > >>>>>>> available for your review, which
>> > >>>>>>> > includes:
>> > >>>>>>> > > * JIRA release notes [1],
>> > >>>>>>> > > * the official Apache source
>> > release
>> > >>>>>>> to be deployed to
>> > >>>>>>> > dist.apache.org
>> > <http://dist.apache.org>
>> > >>>>>>> <http://dist.apache.org/>
>> > <http://dist.apache.org
>> > >>>>>>> <http://dist.apache.org/>>
>> > >>>>>>> > > <http://dist.apache.org
>> > >>>>>>> <http://dist.apache.org/>> [2], which is
>> signed
>> > >>>>>>> with the key with
>> > >>>>>>> > > fingerprint 45C60AAAD115F560
>> [3],
>> > >>>>>>> > > * all artifacts to be deployed
>> to
>> > >>>>>>> the Maven Central
>> > >>>>>>> > Repository [4],
>> > >>>>>>> > > * source code tag "v2.7.0-RC1"
>> [5],
>> > >>>>>>> > > * website pull request listing
>> the
>> > >>>>>>> release and publishing the API
>> > >>>>>>> > > reference manual [6].
>> > >>>>>>> > > * Java artifacts were built
>> with
>> > >>>>>>> Gradle 4.8 and OpenJDK
>> > >>>>>>> > >
>> 1.8.0_181-8u181-b13-1~deb9u1-b13.
>> > >>>>>>> > > * Python artifacts are deployed
>> > >>>>>>> along with the source release
>> > >>>>>>> > to the
>> > >>>>>>> > > dist.apache.org
>> > <http://dist.apache.org>
>> > >>>>>>> <http://dist.apache.org/>
>> > <http://dist.apache.org
>> > >>>>>>> <http://dist.apache.org/>>
>> > >>>>>>> > <http://dist.apache.org
>> > >>>>>>> <http://dist.apache.org/>> [2].
>> > >>>>>>> > >
>> > >>>>>>> > > The vote will be open for at
>> least
>> > >>>>>>> 72 hours. It is adopted by
>> > >>>>>>> > majority
>> > >>>>>>> > > approval, with at least 3 PMC
>> > >>>>>>> affirmative votes.
>> > >>>>>>> > >
>> > >>>>>>> > > Thanks,
>> > >>>>>>> > > Charles
>> > >>>>>>> > >
>> > >>>>>>> > > [1]
>> > >>>>>>> > >
>> > >>>>>>> >
>> > >>>>>>>
>> >
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>> > >>>>>>> > > [2]
>> > >>>>>>>
>> > https://dist.apache.org/repos/dist/dev/beam/2.7.0
>> > >>>>>>> > > [3]
>> > >>>>>>>
>> https://dist.apache.org/repos/dist/dev/beam/KEYS
>> > >>>>>>> > > [4]
>> > >>>>>>> >
>> > >>>>>>>
>> >
>> https://repository.apache.org/content/repositories/orgapachebeam-1046/
>> > >>>>>>> > > [5]
>> > >>>>>>>
>> https://github.com/apache/beam/tree/v2.7.0-RC1
>> > >>>>>>> > > [6]
>> > >>>>>>>
>> https://github.com/apache/beam-site/pull/549
>> > >>>>>>> >
>> > >>>>>>> > --
>> > >>>>>>> > Jean-Baptiste Onofré
>> > >>>>>>> > jbonofre@apache.org
>> > <ma...@apache.org>
>> > >>>>>>> <mailto:jbonofre@apache.org
>> > <ma...@apache.org>>
>> > >>>>>>> <mailto:jbonofre@apache.org
>> > <ma...@apache.org>
>> > >>>>>>> <mailto:jbonofre@apache.org
>> > <ma...@apache.org>>>
>> > >>>>>>> > http://blog.nanthrax.net
>> > >>>>>>> <http://blog.nanthrax.net/>
>> > >>>>>>> > Talend - http://www.talend.com
>> > >>>>>>> <http://www.talend.com/>
>> > >>>>>>> >
>> > >>>>
>> > >
>> >
>> > --
>> > Jean-Baptiste Onofré
>> > jbonofre@apache.org <ma...@apache.org>
>> > http://blog.nanthrax.net
>> > Talend - http://www.talend.com
>> >
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Lukasz Cwik <lc...@google.com>.
Romain hinted that this was a dependency issue but when comparing the two
dependency trees I don't get much of a difference:
lcwik@lcwik0: ~$ diff /tmp/260 /tmp/270
< [INFO] +- org.apache.beam:beam-runners-spark:jar:2.6.0:compile
< [INFO] | +- org.apache.beam:beam-model-pipeline:jar:2.6.0:compile
---
> [INFO] +- org.apache.beam:beam-runners-spark:jar:2.7.0:compile
> [INFO] | +- org.apache.beam:beam-model-pipeline:jar:2.7.0:compile
5c6
< [INFO] | +- org.apache.beam:beam-sdks-java-core:jar:2.6.0:compile
---
> [INFO] | +- org.apache.beam:beam-sdks-java-core:jar:2.7.0:compile
14,18c15,19
< [INFO] | | \- org.tukaani:xz:jar:1.5:compile
< [INFO] | +-
org.apache.beam:beam-runners-core-construction-java:jar:2.6.0:compile
< [INFO] | | \-
org.apache.beam:beam-model-job-management:jar:2.6.0:compile
< [INFO] | +- org.apache.beam:beam-runners-core-java:jar:2.6.0:compile
< [INFO] | | \- org.apache.beam:beam-model-fn-execution:jar:2.6.0:compile
---
> [INFO] | | \- org.tukaani:xz:jar:1.8:compile
> [INFO] | +-
org.apache.beam:beam-runners-core-construction-java:jar:2.7.0:compile
> [INFO] | | \-
org.apache.beam:beam-model-job-management:jar:2.7.0:compile
> [INFO] | +- org.apache.beam:beam-runners-core-java:jar:2.7.0:compile
> [INFO] | | \- org.apache.beam:beam-model-fn-execution:jar:2.7.0:compile
Other then Beam package changes, the only other change is xz which I don't
believe could be causing the issue.
On Tue, Sep 18, 2018 at 8:38 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Thanks, let me take a look.
>
> Regards
> JB
>
> On 18/09/2018 17:36, Romain Manni-Bucau wrote:
> >
> >
> >
> > Le mar. 18 sept. 2018 à 16:44, Jean-Baptiste Onofré <jb@nanthrax.net
> > <ma...@nanthrax.net>> a écrit :
> >
> > Hi,
> >
> > I don't have the issue ;)
> >
> > As said in my vote, I tested 2.7.0 RC1 on beam-samples with Spark
> > without problem.
> >
> > I don't reproduce Romain issue as well.
> >
> > @Romain can you provide some details to reproduce the issue ?
> >
> >
> > Sure, you can use this
> > reproducer: https://github.com/rmannibucau/beam-2.7.0-fails
> > It shows that it suceeds on 2.6 and fails on 2.7.
> >
> >
> >
> > Regards
> > JB
> >
> > On 17/09/2018 19:17, Charles Chen wrote:
> > > Luke, Maximillian, Raghu, can you please propose cherry-pick PRs
> > to the
> > > release-2.7.0 for your issues and add me as a reviewer
> > (@charlesccychen)?
> > >
> > > Romain, JB: is there any way I can help with debugging the issue
> > you're
> > > facing so we can unblock the release?
> > >
> > > On Fri, Sep 14, 2018 at 1:49 PM Raghu Angadi <rangadi@google.com
> > <ma...@google.com>
> > > <mailto:rangadi@google.com <ma...@google.com>>> wrote:
> > >
> > > I would like propose one more cherrypick for RC2
> > > : https://github.com/apache/beam/pull/6391
> > > This is a KafkaIO bug fix. Once a user hits this bug, there is
> no
> > > easy work around for them, especially on Dataflow. Only work
> > around
> > > in Dataflow is to restart or reload the job.
> > >
> > > The fix itself fairly safe and is tested.
> > > Raghu.
> > >
> > > On Fri, Sep 14, 2018 at 12:52 AM Alexey Romanenko
> > > <aromanenko.dev@gmail.com <ma...@gmail.com>
> > <mailto:aromanenko.dev@gmail.com <ma...@gmail.com>>>
> > wrote:
> > >
> > > Perhaps it could help, but I run simple WordCount (built
> with
> > > Beam 2.7) on YARN/Spark (HDP Sandbox) cluster and it
> > worked fine
> > > for me.
> > >
> > >> On 14 Sep 2018, at 06:56, Romain Manni-Bucau
> > >> <rmannibucau@gmail.com <ma...@gmail.com>
> > <mailto:rmannibucau@gmail.com <ma...@gmail.com>>>
> wrote:
> > >>
> > >> Hi Charles,
> > >>
> > >> I didn't get enough time to check deeply but it is
> clearly a
> > >> dependency issue and it is not in beam spark runner
> > itself but
> > >> in another transitive module of beam. It does not happen
> in
> > >> existing spark test cause none of them are in a cluster
> (even
> > >> just with 1 worker) but this seems to be a regression
> since
> > >> 2.6 works OOTB.
> > >>
> > >> Romain Manni-Bucau
> > >> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> > >> <https://rmannibucau.metawerx.net/> | Old Blog
> > >> <http://rmannibucau.wordpress.com/> | Github
> > >> <https://github.com/rmannibucau> | LinkedIn
> > >> <https://www.linkedin.com/in/rmannibucau> | Book
> > >>
> > <
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> >
> > >>
> > >>
> > >> Le jeu. 13 sept. 2018 à 22:15, Charles Chen
> > <ccy@google.com <ma...@google.com>
> > >> <mailto:ccy@google.com <ma...@google.com>>> a
> écrit :
> > >>
> > >> Romain and JB, can you please add the results of your
> > >> investigations into the errors you've seen above?
> Given
> > >> that the existing SparkRunner tests pass for this RC,
> and
> > >> that the integration test you ran is in another repo
> that
> > >> is not continuously tested with Beam, it is not clear
> how
> > >> we should move forward and whether this is a blocking
> > >> issue, unless we can find a root cause in Beam.
> > >>
> > >> On Wed, Sep 12, 2018 at 2:08 AM Etienne Chauchot
> > >> <echauchot@apache.org <ma...@apache.org>
> > <mailto:echauchot@apache.org <ma...@apache.org>>> wrote:
> > >>
> > >> Hi all,
> > >>
> > >> on a performance and functional regression stand
> > point
> > >> I see no regression:
> > >>
> > >> I looked at nexmark graphs "output pcollection
> size"
> > >> and "execution time" around release cut date on
> > >> dataflow, spark, flink and direct runner in batch
> and
> > >> streaming modes. There seems to be no regression.
> > >>
> > >> Etienne
> > >>
> > >> Le mardi 11 septembre 2018 à 12:25 -0700, Charles
> > Chen
> > >> a écrit :
> > >>> The SparkRunner validation test
> > >>>
> > (here:
> https://beam.apache.org/contribute/release-guide/#run-validation-tests)
> > >>> passes on my machine. It looks like we are
> likely
> > >>> missing test coverage where Romain is hitting
> > issues.
> > >>>
> > >>> On Tue, Sep 11, 2018 at 12:15 PM Ahmet Altay
> > >>> <altay@google.com <ma...@google.com>
> > <mailto:altay@google.com <ma...@google.com>>> wrote:
> > >>>> Could anyone else help with looking at these
> issues
> > >>>> earlier?
> > >>>>
> > >>>> On Tue, Sep 11, 2018 at 12:03 PM, Romain
> > Manni-Bucau
> > >>>> <rmannibucau@gmail.com
> > <ma...@gmail.com>
> > >>>> <mailto:rmannibucau@gmail.com
> > <ma...@gmail.com>>> wrote:
> > >>>>> Im running this main [1] through this IT [2].
> Was
> > >>>>> working fine since ~1 year but 2.7.0 broke it.
> > >>>>> Didnt investigate more but can have a look
> later
> > >>>>> this month if it helps.
> > >>>>>
> > >>>>>
> > [1]
> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
> > >>>>>
> > [2]
> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
> > >>>>>
> > >>>>> Le mar. 11 sept. 2018 20:54, Charles Chen
> > >>>>> <ccy@google.com <ma...@google.com>
> > <mailto:ccy@google.com <ma...@google.com>>> a écrit :
> > >>>>>> Romain: can you give more details on the
> failure
> > >>>>>> you're encountering, i.e. how you are
> performing
> > >>>>>> this validation?
> > >>>>>>
> > >>>>>> On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste
> > >>>>>> Onofré <jb@nanthrax.net
> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
> > <ma...@nanthrax.net>>>
> > >>>>>> wrote:
> > >>>>>>> Hi,
> > >>>>>>>
> > >>>>>>> weird, I didn't have it on Beam samples. Let
> me
> > >>>>>>> try to reproduce and I
> > >>>>>>> will create the Jira.
> > >>>>>>>
> > >>>>>>> Regards
> > >>>>>>> JB
> > >>>>>>>
> > >>>>>>> On 11/09/2018 11:44, Romain Manni-Bucau
> wrote:
> > >>>>>>> > -1, seems spark integration is broken
> (tested
> > >>>>>>> with spark 2.3.1 and 2.2.1):
> > >>>>>>> >
> > >>>>>>> > 18/09/11 11:33:29 WARN TaskSetManager: Lost
> > >>>>>>> task 0.0 in stage 0.0 (TID 0, RMANNIBUCAU,
> > >>>>>>> executor 0): java.lang.ClassCastException:
> > cannot
> > >>>>>>> assign instance of
> > >>>>>>>
> > scala.collection.immutable.List$SerializationProxy to
> > >>>>>>> fieldorg.apache.spark.rdd.RDD.org
> > <http://fieldorg.apache.spark.rdd.RDD.org>
> > >>>>>>> <http://fieldorg.apache.spark.rdd.rdd.org/>
> > >>>>>>> <http://org.apache.spark.rdd.RDD.org
> > >>>>>>>
> > <http://org.apache.spark.rdd.rdd.org/
> >>$apache$spark$rdd$RDD$$dependencies_
> > >>>>>>> of type scala.collection.Seq in instance of
> > >>>>>>> org.apache.spark.rdd.MapPartitionsRDD
> > >>>>>>> > at
> > >>>>>>>
> >
> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
> > >>>>>>> >
> > >>>>>>> >
> > >>>>>>> > Also the issue Lukasz identified is
> important
> > >>>>>>> even if workarounds can be
> > >>>>>>> > put in place so +1 to fix it as well if
> > possible.
> > >>>>>>> >
> > >>>>>>> > Romain Manni-Bucau
> > >>>>>>> > @rmannibucau <
> https://twitter.com/rmannibucau>
> > >>>>>>> | Blog
> > >>>>>>> > <https://rmannibucau.metawerx.net/> | Old
> Blog
> > >>>>>>> > <http://rmannibucau.wordpress.com
> > >>>>>>> <http://rmannibucau.wordpress.com/>> |
> Github
> > >>>>>>> > <https://github.com/rmannibucau> |
> LinkedIn
> > >>>>>>> >
> > <https://www.linkedin.com/in/rmannibucau> | Book
> > >>>>>>> >
> > >>>>>>>
> > <
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> >
> > >>>>>>> >
> > >>>>>>> >
> > >>>>>>> > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik
> > >>>>>>> <lcwik@google.com <ma...@google.com>
> > <mailto:lcwik@google.com <ma...@google.com>>
> > >>>>>>> > <mailto:lcwik@google.com
> > <ma...@google.com>
> > >>>>>>> <mailto:lcwik@google.com
> > <ma...@google.com>>>> a écrit :
> > >>>>>>> >
> > >>>>>>> > I found an issue where we are no longer
> > >>>>>>> packaging the pom.xml within
> > >>>>>>> > the artifact jars at
> > >>>>>>> META-INF/maven/groupId/artifactId. More
> details
> > >>>>>>> > in
> > >>>>>>>
> https://issues.apache.org/jira/browse/BEAM-5351.
> > >>>>>>> I wouldn't
> > >>>>>>> > consider this a blocker but it was an
> > easy fix
> > >>>>>>> > (
> https://github.com/apache/beam/pull/6358)
> > >>>>>>> and users may rely on the
> > >>>>>>> > pom.xml.
> > >>>>>>> >
> > >>>>>>> > Should we recut the release candidate
> to
> > >>>>>>> include this?
> > >>>>>>> >
> > >>>>>>> > On Mon, Sep 10, 2018 at 4:58 AM
> > >>>>>>> Jean-Baptiste Onofré
> > >>>>>>> > <jb@nanthrax.net
> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
> > <ma...@nanthrax.net>>
> > >>>>>>> <mailto:jb@nanthrax.net <mailto:
> jb@nanthrax.net>
> > >>>>>>> <mailto:jb@nanthrax.net
> > <ma...@nanthrax.net>>>> wrote:
> > >>>>>>> >
> > >>>>>>> > +1 (binding)
> > >>>>>>> >
> > >>>>>>> > Tested successfully on Beam
> Samples.
> > >>>>>>> >
> > >>>>>>> > Thanks !
> > >>>>>>> >
> > >>>>>>> > Regards
> > >>>>>>> > JB
> > >>>>>>> >
> > >>>>>>> > On 07/09/2018 23:56, Charles Chen
> > wrote:
> > >>>>>>> > > Hi everyone,
> > >>>>>>> > >
> > >>>>>>> > > Please review and vote on the
> > >>>>>>> release candidate #1 for the
> > >>>>>>> > version
> > >>>>>>> > > 2.7.0, as follows:
> > >>>>>>> > > [ ] +1, Approve the release
> > >>>>>>> > > [ ] -1, Do not approve the
> release
> > >>>>>>> (please provide specific
> > >>>>>>> > comments)
> > >>>>>>> > >
> > >>>>>>> > > The complete staging area is
> > >>>>>>> available for your review, which
> > >>>>>>> > includes:
> > >>>>>>> > > * JIRA release notes [1],
> > >>>>>>> > > * the official Apache source
> > release
> > >>>>>>> to be deployed to
> > >>>>>>> > dist.apache.org
> > <http://dist.apache.org>
> > >>>>>>> <http://dist.apache.org/>
> > <http://dist.apache.org
> > >>>>>>> <http://dist.apache.org/>>
> > >>>>>>> > > <http://dist.apache.org
> > >>>>>>> <http://dist.apache.org/>> [2], which is
> signed
> > >>>>>>> with the key with
> > >>>>>>> > > fingerprint 45C60AAAD115F560
> [3],
> > >>>>>>> > > * all artifacts to be deployed
> to
> > >>>>>>> the Maven Central
> > >>>>>>> > Repository [4],
> > >>>>>>> > > * source code tag "v2.7.0-RC1"
> [5],
> > >>>>>>> > > * website pull request listing
> the
> > >>>>>>> release and publishing the API
> > >>>>>>> > > reference manual [6].
> > >>>>>>> > > * Java artifacts were built with
> > >>>>>>> Gradle 4.8 and OpenJDK
> > >>>>>>> > >
> 1.8.0_181-8u181-b13-1~deb9u1-b13.
> > >>>>>>> > > * Python artifacts are deployed
> > >>>>>>> along with the source release
> > >>>>>>> > to the
> > >>>>>>> > > dist.apache.org
> > <http://dist.apache.org>
> > >>>>>>> <http://dist.apache.org/>
> > <http://dist.apache.org
> > >>>>>>> <http://dist.apache.org/>>
> > >>>>>>> > <http://dist.apache.org
> > >>>>>>> <http://dist.apache.org/>> [2].
> > >>>>>>> > >
> > >>>>>>> > > The vote will be open for at
> least
> > >>>>>>> 72 hours. It is adopted by
> > >>>>>>> > majority
> > >>>>>>> > > approval, with at least 3 PMC
> > >>>>>>> affirmative votes.
> > >>>>>>> > >
> > >>>>>>> > > Thanks,
> > >>>>>>> > > Charles
> > >>>>>>> > >
> > >>>>>>> > > [1]
> > >>>>>>> > >
> > >>>>>>> >
> > >>>>>>>
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
> > >>>>>>> > > [2]
> > >>>>>>>
> > https://dist.apache.org/repos/dist/dev/beam/2.7.0
> > >>>>>>> > > [3]
> > >>>>>>>
> https://dist.apache.org/repos/dist/dev/beam/KEYS
> > >>>>>>> > > [4]
> > >>>>>>> >
> > >>>>>>>
> >
> https://repository.apache.org/content/repositories/orgapachebeam-1046/
> > >>>>>>> > > [5]
> > >>>>>>>
> https://github.com/apache/beam/tree/v2.7.0-RC1
> > >>>>>>> > > [6]
> > >>>>>>> https://github.com/apache/beam-site/pull/549
> > >>>>>>> >
> > >>>>>>> > --
> > >>>>>>> > Jean-Baptiste Onofré
> > >>>>>>> > jbonofre@apache.org
> > <ma...@apache.org>
> > >>>>>>> <mailto:jbonofre@apache.org
> > <ma...@apache.org>>
> > >>>>>>> <mailto:jbonofre@apache.org
> > <ma...@apache.org>
> > >>>>>>> <mailto:jbonofre@apache.org
> > <ma...@apache.org>>>
> > >>>>>>> > http://blog.nanthrax.net
> > >>>>>>> <http://blog.nanthrax.net/>
> > >>>>>>> > Talend - http://www.talend.com
> > >>>>>>> <http://www.talend.com/>
> > >>>>>>> >
> > >>>>
> > >
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org <ma...@apache.org>
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Thanks, let me take a look.
Regards
JB
On 18/09/2018 17:36, Romain Manni-Bucau wrote:
>
>
>
> Le mar. 18 sept. 2018 à 16:44, Jean-Baptiste Onofré <jb@nanthrax.net
> <ma...@nanthrax.net>> a écrit :
>
> Hi,
>
> I don't have the issue ;)
>
> As said in my vote, I tested 2.7.0 RC1 on beam-samples with Spark
> without problem.
>
> I don't reproduce Romain issue as well.
>
> @Romain can you provide some details to reproduce the issue ?
>
>
> Sure, you can use this
> reproducer: https://github.com/rmannibucau/beam-2.7.0-fails
> It shows that it suceeds on 2.6 and fails on 2.7.
>
>
>
> Regards
> JB
>
> On 17/09/2018 19:17, Charles Chen wrote:
> > Luke, Maximillian, Raghu, can you please propose cherry-pick PRs
> to the
> > release-2.7.0 for your issues and add me as a reviewer
> (@charlesccychen)?
> >
> > Romain, JB: is there any way I can help with debugging the issue
> you're
> > facing so we can unblock the release?
> >
> > On Fri, Sep 14, 2018 at 1:49 PM Raghu Angadi <rangadi@google.com
> <ma...@google.com>
> > <mailto:rangadi@google.com <ma...@google.com>>> wrote:
> >
> > I would like propose one more cherrypick for RC2
> > : https://github.com/apache/beam/pull/6391
> > This is a KafkaIO bug fix. Once a user hits this bug, there is no
> > easy work around for them, especially on Dataflow. Only work
> around
> > in Dataflow is to restart or reload the job.
> >
> > The fix itself fairly safe and is tested.
> > Raghu.
> >
> > On Fri, Sep 14, 2018 at 12:52 AM Alexey Romanenko
> > <aromanenko.dev@gmail.com <ma...@gmail.com>
> <mailto:aromanenko.dev@gmail.com <ma...@gmail.com>>>
> wrote:
> >
> > Perhaps it could help, but I run simple WordCount (built with
> > Beam 2.7) on YARN/Spark (HDP Sandbox) cluster and it
> worked fine
> > for me.
> >
> >> On 14 Sep 2018, at 06:56, Romain Manni-Bucau
> >> <rmannibucau@gmail.com <ma...@gmail.com>
> <mailto:rmannibucau@gmail.com <ma...@gmail.com>>> wrote:
> >>
> >> Hi Charles,
> >>
> >> I didn't get enough time to check deeply but it is clearly a
> >> dependency issue and it is not in beam spark runner
> itself but
> >> in another transitive module of beam. It does not happen in
> >> existing spark test cause none of them are in a cluster (even
> >> just with 1 worker) but this seems to be a regression since
> >> 2.6 works OOTB.
> >>
> >> Romain Manni-Bucau
> >> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> >> <https://rmannibucau.metawerx.net/> | Old Blog
> >> <http://rmannibucau.wordpress.com/> | Github
> >> <https://github.com/rmannibucau> | LinkedIn
> >> <https://www.linkedin.com/in/rmannibucau> | Book
> >>
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
> >>
> >>
> >> Le jeu. 13 sept. 2018 à 22:15, Charles Chen
> <ccy@google.com <ma...@google.com>
> >> <mailto:ccy@google.com <ma...@google.com>>> a écrit :
> >>
> >> Romain and JB, can you please add the results of your
> >> investigations into the errors you've seen above? Given
> >> that the existing SparkRunner tests pass for this RC, and
> >> that the integration test you ran is in another repo that
> >> is not continuously tested with Beam, it is not clear how
> >> we should move forward and whether this is a blocking
> >> issue, unless we can find a root cause in Beam.
> >>
> >> On Wed, Sep 12, 2018 at 2:08 AM Etienne Chauchot
> >> <echauchot@apache.org <ma...@apache.org>
> <mailto:echauchot@apache.org <ma...@apache.org>>> wrote:
> >>
> >> Hi all,
> >>
> >> on a performance and functional regression stand
> point
> >> I see no regression:
> >>
> >> I looked at nexmark graphs "output pcollection size"
> >> and "execution time" around release cut date on
> >> dataflow, spark, flink and direct runner in batch and
> >> streaming modes. There seems to be no regression.
> >>
> >> Etienne
> >>
> >> Le mardi 11 septembre 2018 à 12:25 -0700, Charles
> Chen
> >> a écrit :
> >>> The SparkRunner validation test
> >>>
> (here: https://beam.apache.org/contribute/release-guide/#run-validation-tests)
> >>> passes on my machine. It looks like we are likely
> >>> missing test coverage where Romain is hitting
> issues.
> >>>
> >>> On Tue, Sep 11, 2018 at 12:15 PM Ahmet Altay
> >>> <altay@google.com <ma...@google.com>
> <mailto:altay@google.com <ma...@google.com>>> wrote:
> >>>> Could anyone else help with looking at these issues
> >>>> earlier?
> >>>>
> >>>> On Tue, Sep 11, 2018 at 12:03 PM, Romain
> Manni-Bucau
> >>>> <rmannibucau@gmail.com
> <ma...@gmail.com>
> >>>> <mailto:rmannibucau@gmail.com
> <ma...@gmail.com>>> wrote:
> >>>>> Im running this main [1] through this IT [2]. Was
> >>>>> working fine since ~1 year but 2.7.0 broke it.
> >>>>> Didnt investigate more but can have a look later
> >>>>> this month if it helps.
> >>>>>
> >>>>>
> [1] https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
> >>>>>
> [2] https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
> >>>>>
> >>>>> Le mar. 11 sept. 2018 20:54, Charles Chen
> >>>>> <ccy@google.com <ma...@google.com>
> <mailto:ccy@google.com <ma...@google.com>>> a écrit :
> >>>>>> Romain: can you give more details on the failure
> >>>>>> you're encountering, i.e. how you are performing
> >>>>>> this validation?
> >>>>>>
> >>>>>> On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste
> >>>>>> Onofré <jb@nanthrax.net
> <ma...@nanthrax.net> <mailto:jb@nanthrax.net
> <ma...@nanthrax.net>>>
> >>>>>> wrote:
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> weird, I didn't have it on Beam samples. Let me
> >>>>>>> try to reproduce and I
> >>>>>>> will create the Jira.
> >>>>>>>
> >>>>>>> Regards
> >>>>>>> JB
> >>>>>>>
> >>>>>>> On 11/09/2018 11:44, Romain Manni-Bucau wrote:
> >>>>>>> > -1, seems spark integration is broken (tested
> >>>>>>> with spark 2.3.1 and 2.2.1):
> >>>>>>> >
> >>>>>>> > 18/09/11 11:33:29 WARN TaskSetManager: Lost
> >>>>>>> task 0.0 in stage 0.0 (TID 0, RMANNIBUCAU,
> >>>>>>> executor 0): java.lang.ClassCastException:
> cannot
> >>>>>>> assign instance of
> >>>>>>>
> scala.collection.immutable.List$SerializationProxy to
> >>>>>>> fieldorg.apache.spark.rdd.RDD.org
> <http://fieldorg.apache.spark.rdd.RDD.org>
> >>>>>>> <http://fieldorg.apache.spark.rdd.rdd.org/>
> >>>>>>> <http://org.apache.spark.rdd.RDD.org
> >>>>>>>
> <http://org.apache.spark.rdd.rdd.org/>>$apache$spark$rdd$RDD$$dependencies_
> >>>>>>> of type scala.collection.Seq in instance of
> >>>>>>> org.apache.spark.rdd.MapPartitionsRDD
> >>>>>>> > at
> >>>>>>>
> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
> >>>>>>> >
> >>>>>>> >
> >>>>>>> > Also the issue Lukasz identified is important
> >>>>>>> even if workarounds can be
> >>>>>>> > put in place so +1 to fix it as well if
> possible.
> >>>>>>> >
> >>>>>>> > Romain Manni-Bucau
> >>>>>>> > @rmannibucau <https://twitter.com/rmannibucau>
> >>>>>>> | Blog
> >>>>>>> > <https://rmannibucau.metawerx.net/> | Old Blog
> >>>>>>> > <http://rmannibucau.wordpress.com
> >>>>>>> <http://rmannibucau.wordpress.com/>> | Github
> >>>>>>> > <https://github.com/rmannibucau> | LinkedIn
> >>>>>>> >
> <https://www.linkedin.com/in/rmannibucau> | Book
> >>>>>>> >
> >>>>>>>
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
> >>>>>>> >
> >>>>>>> >
> >>>>>>> > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik
> >>>>>>> <lcwik@google.com <ma...@google.com>
> <mailto:lcwik@google.com <ma...@google.com>>
> >>>>>>> > <mailto:lcwik@google.com
> <ma...@google.com>
> >>>>>>> <mailto:lcwik@google.com
> <ma...@google.com>>>> a écrit :
> >>>>>>> >
> >>>>>>> > I found an issue where we are no longer
> >>>>>>> packaging the pom.xml within
> >>>>>>> > the artifact jars at
> >>>>>>> META-INF/maven/groupId/artifactId. More details
> >>>>>>> > in
> >>>>>>> https://issues.apache.org/jira/browse/BEAM-5351.
> >>>>>>> I wouldn't
> >>>>>>> > consider this a blocker but it was an
> easy fix
> >>>>>>> > (https://github.com/apache/beam/pull/6358)
> >>>>>>> and users may rely on the
> >>>>>>> > pom.xml.
> >>>>>>> >
> >>>>>>> > Should we recut the release candidate to
> >>>>>>> include this?
> >>>>>>> >
> >>>>>>> > On Mon, Sep 10, 2018 at 4:58 AM
> >>>>>>> Jean-Baptiste Onofré
> >>>>>>> > <jb@nanthrax.net
> <ma...@nanthrax.net> <mailto:jb@nanthrax.net
> <ma...@nanthrax.net>>
> >>>>>>> <mailto:jb@nanthrax.net <ma...@nanthrax.net>
> >>>>>>> <mailto:jb@nanthrax.net
> <ma...@nanthrax.net>>>> wrote:
> >>>>>>> >
> >>>>>>> > +1 (binding)
> >>>>>>> >
> >>>>>>> > Tested successfully on Beam Samples.
> >>>>>>> >
> >>>>>>> > Thanks !
> >>>>>>> >
> >>>>>>> > Regards
> >>>>>>> > JB
> >>>>>>> >
> >>>>>>> > On 07/09/2018 23:56, Charles Chen
> wrote:
> >>>>>>> > > Hi everyone,
> >>>>>>> > >
> >>>>>>> > > Please review and vote on the
> >>>>>>> release candidate #1 for the
> >>>>>>> > version
> >>>>>>> > > 2.7.0, as follows:
> >>>>>>> > > [ ] +1, Approve the release
> >>>>>>> > > [ ] -1, Do not approve the release
> >>>>>>> (please provide specific
> >>>>>>> > comments)
> >>>>>>> > >
> >>>>>>> > > The complete staging area is
> >>>>>>> available for your review, which
> >>>>>>> > includes:
> >>>>>>> > > * JIRA release notes [1],
> >>>>>>> > > * the official Apache source
> release
> >>>>>>> to be deployed to
> >>>>>>> > dist.apache.org
> <http://dist.apache.org>
> >>>>>>> <http://dist.apache.org/>
> <http://dist.apache.org
> >>>>>>> <http://dist.apache.org/>>
> >>>>>>> > > <http://dist.apache.org
> >>>>>>> <http://dist.apache.org/>> [2], which is signed
> >>>>>>> with the key with
> >>>>>>> > > fingerprint 45C60AAAD115F560 [3],
> >>>>>>> > > * all artifacts to be deployed to
> >>>>>>> the Maven Central
> >>>>>>> > Repository [4],
> >>>>>>> > > * source code tag "v2.7.0-RC1" [5],
> >>>>>>> > > * website pull request listing the
> >>>>>>> release and publishing the API
> >>>>>>> > > reference manual [6].
> >>>>>>> > > * Java artifacts were built with
> >>>>>>> Gradle 4.8 and OpenJDK
> >>>>>>> > > 1.8.0_181-8u181-b13-1~deb9u1-b13.
> >>>>>>> > > * Python artifacts are deployed
> >>>>>>> along with the source release
> >>>>>>> > to the
> >>>>>>> > > dist.apache.org
> <http://dist.apache.org>
> >>>>>>> <http://dist.apache.org/>
> <http://dist.apache.org
> >>>>>>> <http://dist.apache.org/>>
> >>>>>>> > <http://dist.apache.org
> >>>>>>> <http://dist.apache.org/>> [2].
> >>>>>>> > >
> >>>>>>> > > The vote will be open for at least
> >>>>>>> 72 hours. It is adopted by
> >>>>>>> > majority
> >>>>>>> > > approval, with at least 3 PMC
> >>>>>>> affirmative votes.
> >>>>>>> > >
> >>>>>>> > > Thanks,
> >>>>>>> > > Charles
> >>>>>>> > >
> >>>>>>> > > [1]
> >>>>>>> > >
> >>>>>>> >
> >>>>>>>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
> >>>>>>> > > [2]
> >>>>>>>
> https://dist.apache.org/repos/dist/dev/beam/2.7.0
> >>>>>>> > > [3]
> >>>>>>> https://dist.apache.org/repos/dist/dev/beam/KEYS
> >>>>>>> > > [4]
> >>>>>>> >
> >>>>>>>
> https://repository.apache.org/content/repositories/orgapachebeam-1046/
> >>>>>>> > > [5]
> >>>>>>> https://github.com/apache/beam/tree/v2.7.0-RC1
> >>>>>>> > > [6]
> >>>>>>> https://github.com/apache/beam-site/pull/549
> >>>>>>> >
> >>>>>>> > --
> >>>>>>> > Jean-Baptiste Onofré
> >>>>>>> > jbonofre@apache.org
> <ma...@apache.org>
> >>>>>>> <mailto:jbonofre@apache.org
> <ma...@apache.org>>
> >>>>>>> <mailto:jbonofre@apache.org
> <ma...@apache.org>
> >>>>>>> <mailto:jbonofre@apache.org
> <ma...@apache.org>>>
> >>>>>>> > http://blog.nanthrax.net
> >>>>>>> <http://blog.nanthrax.net/>
> >>>>>>> > Talend - http://www.talend.com
> >>>>>>> <http://www.talend.com/>
> >>>>>>> >
> >>>>
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org <ma...@apache.org>
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Le mar. 18 sept. 2018 à 16:44, Jean-Baptiste Onofré <jb...@nanthrax.net> a
écrit :
> Hi,
>
> I don't have the issue ;)
>
> As said in my vote, I tested 2.7.0 RC1 on beam-samples with Spark
> without problem.
>
> I don't reproduce Romain issue as well.
>
> @Romain can you provide some details to reproduce the issue ?
>
Sure, you can use this reproducer:
https://github.com/rmannibucau/beam-2.7.0-fails
It shows that it suceeds on 2.6 and fails on 2.7.
>
> Regards
> JB
>
> On 17/09/2018 19:17, Charles Chen wrote:
> > Luke, Maximillian, Raghu, can you please propose cherry-pick PRs to the
> > release-2.7.0 for your issues and add me as a reviewer (@charlesccychen)?
> >
> > Romain, JB: is there any way I can help with debugging the issue you're
> > facing so we can unblock the release?
> >
> > On Fri, Sep 14, 2018 at 1:49 PM Raghu Angadi <rangadi@google.com
> > <ma...@google.com>> wrote:
> >
> > I would like propose one more cherrypick for RC2
> > : https://github.com/apache/beam/pull/6391
> > This is a KafkaIO bug fix. Once a user hits this bug, there is no
> > easy work around for them, especially on Dataflow. Only work around
> > in Dataflow is to restart or reload the job.
> >
> > The fix itself fairly safe and is tested.
> > Raghu.
> >
> > On Fri, Sep 14, 2018 at 12:52 AM Alexey Romanenko
> > <aromanenko.dev@gmail.com <ma...@gmail.com>> wrote:
> >
> > Perhaps it could help, but I run simple WordCount (built with
> > Beam 2.7) on YARN/Spark (HDP Sandbox) cluster and it worked fine
> > for me.
> >
> >> On 14 Sep 2018, at 06:56, Romain Manni-Bucau
> >> <rmannibucau@gmail.com <ma...@gmail.com>> wrote:
> >>
> >> Hi Charles,
> >>
> >> I didn't get enough time to check deeply but it is clearly a
> >> dependency issue and it is not in beam spark runner itself but
> >> in another transitive module of beam. It does not happen in
> >> existing spark test cause none of them are in a cluster (even
> >> just with 1 worker) but this seems to be a regression since
> >> 2.6 works OOTB.
> >>
> >> Romain Manni-Bucau
> >> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> >> <https://rmannibucau.metawerx.net/> | Old Blog
> >> <http://rmannibucau.wordpress.com/> | Github
> >> <https://github.com/rmannibucau> | LinkedIn
> >> <https://www.linkedin.com/in/rmannibucau> | Book
> >> <
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> >
> >>
> >>
> >> Le jeu. 13 sept. 2018 à 22:15, Charles Chen <ccy@google.com
> >> <ma...@google.com>> a écrit :
> >>
> >> Romain and JB, can you please add the results of your
> >> investigations into the errors you've seen above? Given
> >> that the existing SparkRunner tests pass for this RC, and
> >> that the integration test you ran is in another repo that
> >> is not continuously tested with Beam, it is not clear how
> >> we should move forward and whether this is a blocking
> >> issue, unless we can find a root cause in Beam.
> >>
> >> On Wed, Sep 12, 2018 at 2:08 AM Etienne Chauchot
> >> <echauchot@apache.org <ma...@apache.org>> wrote:
> >>
> >> Hi all,
> >>
> >> on a performance and functional regression stand point
> >> I see no regression:
> >>
> >> I looked at nexmark graphs "output pcollection size"
> >> and "execution time" around release cut date on
> >> dataflow, spark, flink and direct runner in batch and
> >> streaming modes. There seems to be no regression.
> >>
> >> Etienne
> >>
> >> Le mardi 11 septembre 2018 à 12:25 -0700, Charles Chen
> >> a écrit :
> >>> The SparkRunner validation test
> >>> (here:
> https://beam.apache.org/contribute/release-guide/#run-validation-tests)
> >>> passes on my machine. It looks like we are likely
> >>> missing test coverage where Romain is hitting issues.
> >>>
> >>> On Tue, Sep 11, 2018 at 12:15 PM Ahmet Altay
> >>> <altay@google.com <ma...@google.com>> wrote:
> >>>> Could anyone else help with looking at these issues
> >>>> earlier?
> >>>>
> >>>> On Tue, Sep 11, 2018 at 12:03 PM, Romain Manni-Bucau
> >>>> <rmannibucau@gmail.com
> >>>> <ma...@gmail.com>> wrote:
> >>>>> Im running this main [1] through this IT [2]. Was
> >>>>> working fine since ~1 year but 2.7.0 broke it.
> >>>>> Didnt investigate more but can have a look later
> >>>>> this month if it helps.
> >>>>>
> >>>>> [1]
> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
> >>>>> [2]
> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
> >>>>>
> >>>>> Le mar. 11 sept. 2018 20:54, Charles Chen
> >>>>> <ccy@google.com <ma...@google.com>> a écrit :
> >>>>>> Romain: can you give more details on the failure
> >>>>>> you're encountering, i.e. how you are performing
> >>>>>> this validation?
> >>>>>>
> >>>>>> On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste
> >>>>>> Onofré <jb@nanthrax.net <ma...@nanthrax.net>>
> >>>>>> wrote:
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> weird, I didn't have it on Beam samples. Let me
> >>>>>>> try to reproduce and I
> >>>>>>> will create the Jira.
> >>>>>>>
> >>>>>>> Regards
> >>>>>>> JB
> >>>>>>>
> >>>>>>> On 11/09/2018 11:44, Romain Manni-Bucau wrote:
> >>>>>>> > -1, seems spark integration is broken (tested
> >>>>>>> with spark 2.3.1 and 2.2.1):
> >>>>>>> >
> >>>>>>> > 18/09/11 11:33:29 WARN TaskSetManager: Lost
> >>>>>>> task 0.0 in stage 0.0 (TID 0, RMANNIBUCAU,
> >>>>>>> executor 0): java.lang.ClassCastException: cannot
> >>>>>>> assign instance of
> >>>>>>> scala.collection.immutable.List$SerializationProxy
> to
> >>>>>>> fieldorg.apache.spark.rdd.RDD.org
> >>>>>>> <http://fieldorg.apache.spark.rdd.rdd.org/>
> >>>>>>> <http://org.apache.spark.rdd.RDD.org
> >>>>>>> <http://org.apache.spark.rdd.rdd.org/
> >>$apache$spark$rdd$RDD$$dependencies_
> >>>>>>> of type scala.collection.Seq in instance of
> >>>>>>> org.apache.spark.rdd.MapPartitionsRDD
> >>>>>>> > at
> >>>>>>>
> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
> >>>>>>> >
> >>>>>>> >
> >>>>>>> > Also the issue Lukasz identified is important
> >>>>>>> even if workarounds can be
> >>>>>>> > put in place so +1 to fix it as well if possible.
> >>>>>>> >
> >>>>>>> > Romain Manni-Bucau
> >>>>>>> > @rmannibucau <https://twitter.com/rmannibucau>
> >>>>>>> | Blog
> >>>>>>> > <https://rmannibucau.metawerx.net/> | Old Blog
> >>>>>>> > <http://rmannibucau.wordpress.com
> >>>>>>> <http://rmannibucau.wordpress.com/>> | Github
> >>>>>>> > <https://github.com/rmannibucau> | LinkedIn
> >>>>>>> > <https://www.linkedin.com/in/rmannibucau> | Book
> >>>>>>> >
> >>>>>>> <
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> >
> >>>>>>> >
> >>>>>>> >
> >>>>>>> > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik
> >>>>>>> <lcwik@google.com <ma...@google.com>
> >>>>>>> > <mailto:lcwik@google.com
> >>>>>>> <ma...@google.com>>> a écrit :
> >>>>>>> >
> >>>>>>> > I found an issue where we are no longer
> >>>>>>> packaging the pom.xml within
> >>>>>>> > the artifact jars at
> >>>>>>> META-INF/maven/groupId/artifactId. More details
> >>>>>>> > in
> >>>>>>> https://issues.apache.org/jira/browse/BEAM-5351.
> >>>>>>> I wouldn't
> >>>>>>> > consider this a blocker but it was an easy
> fix
> >>>>>>> > (https://github.com/apache/beam/pull/6358)
> >>>>>>> and users may rely on the
> >>>>>>> > pom.xml.
> >>>>>>> >
> >>>>>>> > Should we recut the release candidate to
> >>>>>>> include this?
> >>>>>>> >
> >>>>>>> > On Mon, Sep 10, 2018 at 4:58 AM
> >>>>>>> Jean-Baptiste Onofré
> >>>>>>> > <jb@nanthrax.net <ma...@nanthrax.net>
> >>>>>>> <mailto:jb@nanthrax.net
> >>>>>>> <ma...@nanthrax.net>>> wrote:
> >>>>>>> >
> >>>>>>> > +1 (binding)
> >>>>>>> >
> >>>>>>> > Tested successfully on Beam Samples.
> >>>>>>> >
> >>>>>>> > Thanks !
> >>>>>>> >
> >>>>>>> > Regards
> >>>>>>> > JB
> >>>>>>> >
> >>>>>>> > On 07/09/2018 23:56, Charles Chen wrote:
> >>>>>>> > > Hi everyone,
> >>>>>>> > >
> >>>>>>> > > Please review and vote on the
> >>>>>>> release candidate #1 for the
> >>>>>>> > version
> >>>>>>> > > 2.7.0, as follows:
> >>>>>>> > > [ ] +1, Approve the release
> >>>>>>> > > [ ] -1, Do not approve the release
> >>>>>>> (please provide specific
> >>>>>>> > comments)
> >>>>>>> > >
> >>>>>>> > > The complete staging area is
> >>>>>>> available for your review, which
> >>>>>>> > includes:
> >>>>>>> > > * JIRA release notes [1],
> >>>>>>> > > * the official Apache source release
> >>>>>>> to be deployed to
> >>>>>>> > dist.apache.org
> >>>>>>> <http://dist.apache.org/> <http://dist.apache.org
> >>>>>>> <http://dist.apache.org/>>
> >>>>>>> > > <http://dist.apache.org
> >>>>>>> <http://dist.apache.org/>> [2], which is signed
> >>>>>>> with the key with
> >>>>>>> > > fingerprint 45C60AAAD115F560 [3],
> >>>>>>> > > * all artifacts to be deployed to
> >>>>>>> the Maven Central
> >>>>>>> > Repository [4],
> >>>>>>> > > * source code tag "v2.7.0-RC1" [5],
> >>>>>>> > > * website pull request listing the
> >>>>>>> release and publishing the API
> >>>>>>> > > reference manual [6].
> >>>>>>> > > * Java artifacts were built with
> >>>>>>> Gradle 4.8 and OpenJDK
> >>>>>>> > > 1.8.0_181-8u181-b13-1~deb9u1-b13.
> >>>>>>> > > * Python artifacts are deployed
> >>>>>>> along with the source release
> >>>>>>> > to the
> >>>>>>> > > dist.apache.org
> >>>>>>> <http://dist.apache.org/> <http://dist.apache.org
> >>>>>>> <http://dist.apache.org/>>
> >>>>>>> > <http://dist.apache.org
> >>>>>>> <http://dist.apache.org/>> [2].
> >>>>>>> > >
> >>>>>>> > > The vote will be open for at least
> >>>>>>> 72 hours. It is adopted by
> >>>>>>> > majority
> >>>>>>> > > approval, with at least 3 PMC
> >>>>>>> affirmative votes.
> >>>>>>> > >
> >>>>>>> > > Thanks,
> >>>>>>> > > Charles
> >>>>>>> > >
> >>>>>>> > > [1]
> >>>>>>> > >
> >>>>>>> >
> >>>>>>>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
> >>>>>>> > > [2]
> >>>>>>> https://dist.apache.org/repos/dist/dev/beam/2.7.0
> >>>>>>> > > [3]
> >>>>>>> https://dist.apache.org/repos/dist/dev/beam/KEYS
> >>>>>>> > > [4]
> >>>>>>> >
> >>>>>>>
> https://repository.apache.org/content/repositories/orgapachebeam-1046/
> >>>>>>> > > [5]
> >>>>>>> https://github.com/apache/beam/tree/v2.7.0-RC1
> >>>>>>> > > [6]
> >>>>>>> https://github.com/apache/beam-site/pull/549
> >>>>>>> >
> >>>>>>> > --
> >>>>>>> > Jean-Baptiste Onofré
> >>>>>>> > jbonofre@apache.org
> >>>>>>> <ma...@apache.org>
> >>>>>>> <mailto:jbonofre@apache.org
> >>>>>>> <ma...@apache.org>>
> >>>>>>> > http://blog.nanthrax.net
> >>>>>>> <http://blog.nanthrax.net/>
> >>>>>>> > Talend - http://www.talend.com
> >>>>>>> <http://www.talend.com/>
> >>>>>>> >
> >>>>
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi,
I don't have the issue ;)
As said in my vote, I tested 2.7.0 RC1 on beam-samples with Spark
without problem.
I don't reproduce Romain issue as well.
@Romain can you provide some details to reproduce the issue ?
Regards
JB
On 17/09/2018 19:17, Charles Chen wrote:
> Luke, Maximillian, Raghu, can you please propose cherry-pick PRs to the
> release-2.7.0 for your issues and add me as a reviewer (@charlesccychen)?
>
> Romain, JB: is there any way I can help with debugging the issue you're
> facing so we can unblock the release?
>
> On Fri, Sep 14, 2018 at 1:49 PM Raghu Angadi <rangadi@google.com
> <ma...@google.com>> wrote:
>
> I would like propose one more cherrypick for RC2
> : https://github.com/apache/beam/pull/6391
> This is a KafkaIO bug fix. Once a user hits this bug, there is no
> easy work around for them, especially on Dataflow. Only work around
> in Dataflow is to restart or reload the job.
>
> The fix itself fairly safe and is tested.
> Raghu.
>
> On Fri, Sep 14, 2018 at 12:52 AM Alexey Romanenko
> <aromanenko.dev@gmail.com <ma...@gmail.com>> wrote:
>
> Perhaps it could help, but I run simple WordCount (built with
> Beam 2.7) on YARN/Spark (HDP Sandbox) cluster and it worked fine
> for me.
>
>> On 14 Sep 2018, at 06:56, Romain Manni-Bucau
>> <rmannibucau@gmail.com <ma...@gmail.com>> wrote:
>>
>> Hi Charles,
>>
>> I didn't get enough time to check deeply but it is clearly a
>> dependency issue and it is not in beam spark runner itself but
>> in another transitive module of beam. It does not happen in
>> existing spark test cause none of them are in a cluster (even
>> just with 1 worker) but this seems to be a regression since
>> 2.6 works OOTB.
>>
>> Romain Manni-Bucau
>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>> <https://rmannibucau.metawerx.net/> | Old Blog
>> <http://rmannibucau.wordpress.com/> | Github
>> <https://github.com/rmannibucau> | LinkedIn
>> <https://www.linkedin.com/in/rmannibucau> | Book
>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>
>>
>> Le jeu. 13 sept. 2018 à 22:15, Charles Chen <ccy@google.com
>> <ma...@google.com>> a écrit :
>>
>> Romain and JB, can you please add the results of your
>> investigations into the errors you've seen above? Given
>> that the existing SparkRunner tests pass for this RC, and
>> that the integration test you ran is in another repo that
>> is not continuously tested with Beam, it is not clear how
>> we should move forward and whether this is a blocking
>> issue, unless we can find a root cause in Beam.
>>
>> On Wed, Sep 12, 2018 at 2:08 AM Etienne Chauchot
>> <echauchot@apache.org <ma...@apache.org>> wrote:
>>
>> Hi all,
>>
>> on a performance and functional regression stand point
>> I see no regression:
>>
>> I looked at nexmark graphs "output pcollection size"
>> and "execution time" around release cut date on
>> dataflow, spark, flink and direct runner in batch and
>> streaming modes. There seems to be no regression.
>>
>> Etienne
>>
>> Le mardi 11 septembre 2018 à 12:25 -0700, Charles Chen
>> a écrit :
>>> The SparkRunner validation test
>>> (here: https://beam.apache.org/contribute/release-guide/#run-validation-tests)
>>> passes on my machine. It looks like we are likely
>>> missing test coverage where Romain is hitting issues.
>>>
>>> On Tue, Sep 11, 2018 at 12:15 PM Ahmet Altay
>>> <altay@google.com <ma...@google.com>> wrote:
>>>> Could anyone else help with looking at these issues
>>>> earlier?
>>>>
>>>> On Tue, Sep 11, 2018 at 12:03 PM, Romain Manni-Bucau
>>>> <rmannibucau@gmail.com
>>>> <ma...@gmail.com>> wrote:
>>>>> Im running this main [1] through this IT [2]. Was
>>>>> working fine since ~1 year but 2.7.0 broke it.
>>>>> Didnt investigate more but can have a look later
>>>>> this month if it helps.
>>>>>
>>>>> [1] https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
>>>>> [2] https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
>>>>>
>>>>> Le mar. 11 sept. 2018 20:54, Charles Chen
>>>>> <ccy@google.com <ma...@google.com>> a écrit :
>>>>>> Romain: can you give more details on the failure
>>>>>> you're encountering, i.e. how you are performing
>>>>>> this validation?
>>>>>>
>>>>>> On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste
>>>>>> Onofré <jb@nanthrax.net <ma...@nanthrax.net>>
>>>>>> wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> weird, I didn't have it on Beam samples. Let me
>>>>>>> try to reproduce and I
>>>>>>> will create the Jira.
>>>>>>>
>>>>>>> Regards
>>>>>>> JB
>>>>>>>
>>>>>>> On 11/09/2018 11:44, Romain Manni-Bucau wrote:
>>>>>>> > -1, seems spark integration is broken (tested
>>>>>>> with spark 2.3.1 and 2.2.1):
>>>>>>> >
>>>>>>> > 18/09/11 11:33:29 WARN TaskSetManager: Lost
>>>>>>> task 0.0 in stage 0.0 (TID 0, RMANNIBUCAU,
>>>>>>> executor 0): java.lang.ClassCastException: cannot
>>>>>>> assign instance of
>>>>>>> scala.collection.immutable.List$SerializationProxy to
>>>>>>> fieldorg.apache.spark.rdd.RDD.org
>>>>>>> <http://fieldorg.apache.spark.rdd.rdd.org/>
>>>>>>> <http://org.apache.spark.rdd.RDD.org
>>>>>>> <http://org.apache.spark.rdd.rdd.org/>>$apache$spark$rdd$RDD$$dependencies_
>>>>>>> of type scala.collection.Seq in instance of
>>>>>>> org.apache.spark.rdd.MapPartitionsRDD
>>>>>>> > at
>>>>>>> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>>>>>>> >
>>>>>>> >
>>>>>>> > Also the issue Lukasz identified is important
>>>>>>> even if workarounds can be
>>>>>>> > put in place so +1 to fix it as well if possible.
>>>>>>> >
>>>>>>> > Romain Manni-Bucau
>>>>>>> > @rmannibucau <https://twitter.com/rmannibucau>
>>>>>>> | Blog
>>>>>>> > <https://rmannibucau.metawerx.net/> | Old Blog
>>>>>>> > <http://rmannibucau.wordpress.com
>>>>>>> <http://rmannibucau.wordpress.com/>> | Github
>>>>>>> > <https://github.com/rmannibucau> | LinkedIn
>>>>>>> > <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>>> >
>>>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>>>> >
>>>>>>> >
>>>>>>> > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik
>>>>>>> <lcwik@google.com <ma...@google.com>
>>>>>>> > <mailto:lcwik@google.com
>>>>>>> <ma...@google.com>>> a écrit :
>>>>>>> >
>>>>>>> > I found an issue where we are no longer
>>>>>>> packaging the pom.xml within
>>>>>>> > the artifact jars at
>>>>>>> META-INF/maven/groupId/artifactId. More details
>>>>>>> > in
>>>>>>> https://issues.apache.org/jira/browse/BEAM-5351.
>>>>>>> I wouldn't
>>>>>>> > consider this a blocker but it was an easy fix
>>>>>>> > (https://github.com/apache/beam/pull/6358)
>>>>>>> and users may rely on the
>>>>>>> > pom.xml.
>>>>>>> >
>>>>>>> > Should we recut the release candidate to
>>>>>>> include this?
>>>>>>> >
>>>>>>> > On Mon, Sep 10, 2018 at 4:58 AM
>>>>>>> Jean-Baptiste Onofré
>>>>>>> > <jb@nanthrax.net <ma...@nanthrax.net>
>>>>>>> <mailto:jb@nanthrax.net
>>>>>>> <ma...@nanthrax.net>>> wrote:
>>>>>>> >
>>>>>>> > +1 (binding)
>>>>>>> >
>>>>>>> > Tested successfully on Beam Samples.
>>>>>>> >
>>>>>>> > Thanks !
>>>>>>> >
>>>>>>> > Regards
>>>>>>> > JB
>>>>>>> >
>>>>>>> > On 07/09/2018 23:56, Charles Chen wrote:
>>>>>>> > > Hi everyone,
>>>>>>> > >
>>>>>>> > > Please review and vote on the
>>>>>>> release candidate #1 for the
>>>>>>> > version
>>>>>>> > > 2.7.0, as follows:
>>>>>>> > > [ ] +1, Approve the release
>>>>>>> > > [ ] -1, Do not approve the release
>>>>>>> (please provide specific
>>>>>>> > comments)
>>>>>>> > >
>>>>>>> > > The complete staging area is
>>>>>>> available for your review, which
>>>>>>> > includes:
>>>>>>> > > * JIRA release notes [1],
>>>>>>> > > * the official Apache source release
>>>>>>> to be deployed to
>>>>>>> > dist.apache.org
>>>>>>> <http://dist.apache.org/> <http://dist.apache.org
>>>>>>> <http://dist.apache.org/>>
>>>>>>> > > <http://dist.apache.org
>>>>>>> <http://dist.apache.org/>> [2], which is signed
>>>>>>> with the key with
>>>>>>> > > fingerprint 45C60AAAD115F560 [3],
>>>>>>> > > * all artifacts to be deployed to
>>>>>>> the Maven Central
>>>>>>> > Repository [4],
>>>>>>> > > * source code tag "v2.7.0-RC1" [5],
>>>>>>> > > * website pull request listing the
>>>>>>> release and publishing the API
>>>>>>> > > reference manual [6].
>>>>>>> > > * Java artifacts were built with
>>>>>>> Gradle 4.8 and OpenJDK
>>>>>>> > > 1.8.0_181-8u181-b13-1~deb9u1-b13.
>>>>>>> > > * Python artifacts are deployed
>>>>>>> along with the source release
>>>>>>> > to the
>>>>>>> > > dist.apache.org
>>>>>>> <http://dist.apache.org/> <http://dist.apache.org
>>>>>>> <http://dist.apache.org/>>
>>>>>>> > <http://dist.apache.org
>>>>>>> <http://dist.apache.org/>> [2].
>>>>>>> > >
>>>>>>> > > The vote will be open for at least
>>>>>>> 72 hours. It is adopted by
>>>>>>> > majority
>>>>>>> > > approval, with at least 3 PMC
>>>>>>> affirmative votes.
>>>>>>> > >
>>>>>>> > > Thanks,
>>>>>>> > > Charles
>>>>>>> > >
>>>>>>> > > [1]
>>>>>>> > >
>>>>>>> >
>>>>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>>>>>>> > > [2]
>>>>>>> https://dist.apache.org/repos/dist/dev/beam/2.7.0
>>>>>>> > > [3]
>>>>>>> https://dist.apache.org/repos/dist/dev/beam/KEYS
>>>>>>> > > [4]
>>>>>>> >
>>>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1046/
>>>>>>> > > [5]
>>>>>>> https://github.com/apache/beam/tree/v2.7.0-RC1
>>>>>>> > > [6]
>>>>>>> https://github.com/apache/beam-site/pull/549
>>>>>>> >
>>>>>>> > --
>>>>>>> > Jean-Baptiste Onofré
>>>>>>> > jbonofre@apache.org
>>>>>>> <ma...@apache.org>
>>>>>>> <mailto:jbonofre@apache.org
>>>>>>> <ma...@apache.org>>
>>>>>>> > http://blog.nanthrax.net
>>>>>>> <http://blog.nanthrax.net/>
>>>>>>> > Talend - http://www.talend.com
>>>>>>> <http://www.talend.com/>
>>>>>>> >
>>>>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Maximilian Michels <mx...@apache.org>.
Hi Charles,
Thanks for driving the release. Here are the cherrypicks I mentioned:
https://github.com/apache/beam/pull/6427
Thanks,
Max
On 17.09.18 19:17, Charles Chen wrote:
> Luke, Maximillian, Raghu, can you please propose cherry-pick PRs to the
> release-2.7.0 for your issues and add me as a reviewer (@charlesccychen)?
>
> Romain, JB: is there any way I can help with debugging the issue you're
> facing so we can unblock the release?
>
> On Fri, Sep 14, 2018 at 1:49 PM Raghu Angadi <rangadi@google.com
> <ma...@google.com>> wrote:
>
> I would like propose one more cherrypick for RC2 :
> https://github.com/apache/beam/pull/6391
> This is a KafkaIO bug fix. Once a user hits this bug, there is no
> easy work around for them, especially on Dataflow. Only work around
> in Dataflow is to restart or reload the job.
>
> The fix itself fairly safe and is tested.
> Raghu.
>
> On Fri, Sep 14, 2018 at 12:52 AM Alexey Romanenko
> <aromanenko.dev@gmail.com <ma...@gmail.com>> wrote:
>
> Perhaps it could help, but I run simple WordCount (built with
> Beam 2.7) on YARN/Spark (HDP Sandbox) cluster and it worked fine
> for me.
>
>> On 14 Sep 2018, at 06:56, Romain Manni-Bucau
>> <rmannibucau@gmail.com <ma...@gmail.com>> wrote:
>>
>> Hi Charles,
>>
>> I didn't get enough time to check deeply but it is clearly a
>> dependency issue and it is not in beam spark runner itself but
>> in another transitive module of beam. It does not happen in
>> existing spark test cause none of them are in a cluster (even
>> just with 1 worker) but this seems to be a regression since
>> 2.6 works OOTB.
>>
>> Romain Manni-Bucau
>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>> <https://rmannibucau.metawerx.net/> | Old Blog
>> <http://rmannibucau.wordpress.com/> | Github
>> <https://github.com/rmannibucau> | LinkedIn
>> <https://www.linkedin.com/in/rmannibucau> | Book
>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>
>>
>> Le jeu. 13 sept. 2018 à 22:15, Charles Chen <ccy@google.com
>> <ma...@google.com>> a écrit :
>>
>> Romain and JB, can you please add the results of your
>> investigations into the errors you've seen above? Given
>> that the existing SparkRunner tests pass for this RC, and
>> that the integration test you ran is in another repo that
>> is not continuously tested with Beam, it is not clear how
>> we should move forward and whether this is a blocking
>> issue, unless we can find a root cause in Beam.
>>
>> On Wed, Sep 12, 2018 at 2:08 AM Etienne Chauchot
>> <echauchot@apache.org <ma...@apache.org>> wrote:
>>
>> Hi all,
>>
>> on a performance and functional regression stand point
>> I see no regression:
>>
>> I looked at nexmark graphs "output pcollection size"
>> and "execution time" around release cut date on
>> dataflow, spark, flink and direct runner in batch and
>> streaming modes. There seems to be no regression.
>>
>> Etienne
>>
>> Le mardi 11 septembre 2018 à 12:25 -0700, Charles Chen
>> a écrit :
>>> The SparkRunner validation test (here:
>>> https://beam.apache.org/contribute/release-guide/#run-validation-tests)
>>> passes on my machine. It looks like we are likely
>>> missing test coverage where Romain is hitting issues.
>>>
>>> On Tue, Sep 11, 2018 at 12:15 PM Ahmet Altay
>>> <altay@google.com <ma...@google.com>> wrote:
>>>> Could anyone else help with looking at these issues
>>>> earlier?
>>>>
>>>> On Tue, Sep 11, 2018 at 12:03 PM, Romain Manni-Bucau
>>>> <rmannibucau@gmail.com
>>>> <ma...@gmail.com>> wrote:
>>>>> Im running this main [1] through this IT [2]. Was
>>>>> working fine since ~1 year but 2.7.0 broke it.
>>>>> Didnt investigate more but can have a look later
>>>>> this month if it helps.
>>>>>
>>>>> [1]
>>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
>>>>> [2]
>>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
>>>>>
>>>>> Le mar. 11 sept. 2018 20:54, Charles Chen
>>>>> <ccy@google.com <ma...@google.com>> a écrit :
>>>>>> Romain: can you give more details on the failure
>>>>>> you're encountering, i.e. how you are performing
>>>>>> this validation?
>>>>>>
>>>>>> On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste
>>>>>> Onofré <jb@nanthrax.net <ma...@nanthrax.net>>
>>>>>> wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> weird, I didn't have it on Beam samples. Let me
>>>>>>> try to reproduce and I
>>>>>>> will create the Jira.
>>>>>>>
>>>>>>> Regards
>>>>>>> JB
>>>>>>>
>>>>>>> On 11/09/2018 11:44, Romain Manni-Bucau wrote:
>>>>>>> > -1, seems spark integration is broken (tested
>>>>>>> with spark 2.3.1 and 2.2.1):
>>>>>>> >
>>>>>>> > 18/09/11 11:33:29 WARN TaskSetManager: Lost
>>>>>>> task 0.0 in stage 0.0 (TID 0, RMANNIBUCAU,
>>>>>>> executor 0): java.lang.ClassCastException: cannot
>>>>>>> assign instance of
>>>>>>> scala.collection.immutable.List$SerializationProxy to
>>>>>>> fieldorg.apache.spark.rdd.RDD.org
>>>>>>> <http://fieldorg.apache.spark.rdd.rdd.org/>
>>>>>>> <http://org.apache.spark.rdd.RDD.org
>>>>>>> <http://org.apache.spark.rdd.rdd.org/>>$apache$spark$rdd$RDD$$dependencies_
>>>>>>> of type scala.collection.Seq in instance of
>>>>>>> org.apache.spark.rdd.MapPartitionsRDD
>>>>>>> > at
>>>>>>> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>>>>>>> >
>>>>>>> >
>>>>>>> > Also the issue Lukasz identified is important
>>>>>>> even if workarounds can be
>>>>>>> > put in place so +1 to fix it as well if possible.
>>>>>>> >
>>>>>>> > Romain Manni-Bucau
>>>>>>> > @rmannibucau <https://twitter.com/rmannibucau>
>>>>>>> | Blog
>>>>>>> > <https://rmannibucau.metawerx.net/> | Old Blog
>>>>>>> > <http://rmannibucau.wordpress.com
>>>>>>> <http://rmannibucau.wordpress.com/>> | Github
>>>>>>> > <https://github.com/rmannibucau> | LinkedIn
>>>>>>> > <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>>> >
>>>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>>>> >
>>>>>>> >
>>>>>>> > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik
>>>>>>> <lcwik@google.com <ma...@google.com>
>>>>>>> > <mailto:lcwik@google.com
>>>>>>> <ma...@google.com>>> a écrit :
>>>>>>> >
>>>>>>> > I found an issue where we are no longer
>>>>>>> packaging the pom.xml within
>>>>>>> > the artifact jars at
>>>>>>> META-INF/maven/groupId/artifactId. More details
>>>>>>> > in
>>>>>>> https://issues.apache.org/jira/browse/BEAM-5351.
>>>>>>> I wouldn't
>>>>>>> > consider this a blocker but it was an easy fix
>>>>>>> > (https://github.com/apache/beam/pull/6358)
>>>>>>> and users may rely on the
>>>>>>> > pom.xml.
>>>>>>> >
>>>>>>> > Should we recut the release candidate to
>>>>>>> include this?
>>>>>>> >
>>>>>>> > On Mon, Sep 10, 2018 at 4:58 AM
>>>>>>> Jean-Baptiste Onofré
>>>>>>> > <jb@nanthrax.net <ma...@nanthrax.net>
>>>>>>> <mailto:jb@nanthrax.net
>>>>>>> <ma...@nanthrax.net>>> wrote:
>>>>>>> >
>>>>>>> > +1 (binding)
>>>>>>> >
>>>>>>> > Tested successfully on Beam Samples.
>>>>>>> >
>>>>>>> > Thanks !
>>>>>>> >
>>>>>>> > Regards
>>>>>>> > JB
>>>>>>> >
>>>>>>> > On 07/09/2018 23:56, Charles Chen wrote:
>>>>>>> > > Hi everyone,
>>>>>>> > >
>>>>>>> > > Please review and vote on the
>>>>>>> release candidate #1 for the
>>>>>>> > version
>>>>>>> > > 2.7.0, as follows:
>>>>>>> > > [ ] +1, Approve the release
>>>>>>> > > [ ] -1, Do not approve the release
>>>>>>> (please provide specific
>>>>>>> > comments)
>>>>>>> > >
>>>>>>> > > The complete staging area is
>>>>>>> available for your review, which
>>>>>>> > includes:
>>>>>>> > > * JIRA release notes [1],
>>>>>>> > > * the official Apache source release
>>>>>>> to be deployed to
>>>>>>> > dist.apache.org <http://dist.apache.org/>
>>>>>>> <http://dist.apache.org <http://dist.apache.org/>>
>>>>>>> > > <http://dist.apache.org
>>>>>>> <http://dist.apache.org/>> [2], which is signed
>>>>>>> with the key with
>>>>>>> > > fingerprint 45C60AAAD115F560 [3],
>>>>>>> > > * all artifacts to be deployed to
>>>>>>> the Maven Central
>>>>>>> > Repository [4],
>>>>>>> > > * source code tag "v2.7.0-RC1" [5],
>>>>>>> > > * website pull request listing the
>>>>>>> release and publishing the API
>>>>>>> > > reference manual [6].
>>>>>>> > > * Java artifacts were built with
>>>>>>> Gradle 4.8 and OpenJDK
>>>>>>> > > 1.8.0_181-8u181-b13-1~deb9u1-b13.
>>>>>>> > > * Python artifacts are deployed
>>>>>>> along with the source release
>>>>>>> > to the
>>>>>>> > > dist.apache.org
>>>>>>> <http://dist.apache.org/> <http://dist.apache.org
>>>>>>> <http://dist.apache.org/>>
>>>>>>> > <http://dist.apache.org
>>>>>>> <http://dist.apache.org/>> [2].
>>>>>>> > >
>>>>>>> > > The vote will be open for at least
>>>>>>> 72 hours. It is adopted by
>>>>>>> > majority
>>>>>>> > > approval, with at least 3 PMC
>>>>>>> affirmative votes.
>>>>>>> > >
>>>>>>> > > Thanks,
>>>>>>> > > Charles
>>>>>>> > >
>>>>>>> > > [1]
>>>>>>> > >
>>>>>>> >
>>>>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>>>>>>> > > [2]
>>>>>>> https://dist.apache.org/repos/dist/dev/beam/2.7.0
>>>>>>> > > [3]
>>>>>>> https://dist.apache.org/repos/dist/dev/beam/KEYS
>>>>>>> > > [4]
>>>>>>> >
>>>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1046/
>>>>>>> > > [5]
>>>>>>> https://github.com/apache/beam/tree/v2.7.0-RC1
>>>>>>> > > [6]
>>>>>>> https://github.com/apache/beam-site/pull/549
>>>>>>> >
>>>>>>> > --
>>>>>>> > Jean-Baptiste Onofré
>>>>>>> > jbonofre@apache.org
>>>>>>> <ma...@apache.org>
>>>>>>> <mailto:jbonofre@apache.org
>>>>>>> <ma...@apache.org>>
>>>>>>> > http://blog.nanthrax.net
>>>>>>> <http://blog.nanthrax.net/>
>>>>>>> > Talend - http://www.talend.com
>>>>>>> <http://www.talend.com/>
>>>>>>> >
>>>>
>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Charles Chen <cc...@google.com>.
Luke, Maximillian, Raghu, can you please propose cherry-pick PRs to the
release-2.7.0 for your issues and add me as a reviewer (@charlesccychen)?
Romain, JB: is there any way I can help with debugging the issue you're
facing so we can unblock the release?
On Fri, Sep 14, 2018 at 1:49 PM Raghu Angadi <ra...@google.com> wrote:
> I would like propose one more cherrypick for RC2 :
> https://github.com/apache/beam/pull/6391
> This is a KafkaIO bug fix. Once a user hits this bug, there is no easy
> work around for them, especially on Dataflow. Only work around in Dataflow
> is to restart or reload the job.
>
> The fix itself fairly safe and is tested.
> Raghu.
>
> On Fri, Sep 14, 2018 at 12:52 AM Alexey Romanenko <
> aromanenko.dev@gmail.com> wrote:
>
>> Perhaps it could help, but I run simple WordCount (built with Beam 2.7)
>> on YARN/Spark (HDP Sandbox) cluster and it worked fine for me.
>>
>> On 14 Sep 2018, at 06:56, Romain Manni-Bucau <rm...@gmail.com>
>> wrote:
>>
>> Hi Charles,
>>
>> I didn't get enough time to check deeply but it is clearly a dependency
>> issue and it is not in beam spark runner itself but in another transitive
>> module of beam. It does not happen in existing spark test cause none of
>> them are in a cluster (even just with 1 worker) but this seems to be a
>> regression since 2.6 works OOTB.
>>
>> Romain Manni-Bucau
>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>> <https://rmannibucau.metawerx.net/> | Old Blog
>> <http://rmannibucau.wordpress.com/> | Github
>> <https://github.com/rmannibucau> | LinkedIn
>> <https://www.linkedin.com/in/rmannibucau> | Book
>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>
>>
>> Le jeu. 13 sept. 2018 à 22:15, Charles Chen <cc...@google.com> a écrit :
>>
>>> Romain and JB, can you please add the results of your investigations
>>> into the errors you've seen above? Given that the existing SparkRunner
>>> tests pass for this RC, and that the integration test you ran is in another
>>> repo that is not continuously tested with Beam, it is not clear how we
>>> should move forward and whether this is a blocking issue, unless we can
>>> find a root cause in Beam.
>>>
>>> On Wed, Sep 12, 2018 at 2:08 AM Etienne Chauchot <ec...@apache.org>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> on a performance and functional regression stand point I see no
>>>> regression:
>>>>
>>>> I looked at nexmark graphs "output pcollection size" and "execution
>>>> time" around release cut date on dataflow, spark, flink and direct runner
>>>> in batch and streaming modes. There seems to be no regression.
>>>>
>>>> Etienne
>>>>
>>>> Le mardi 11 septembre 2018 à 12:25 -0700, Charles Chen a écrit :
>>>>
>>>> The SparkRunner validation test (here:
>>>> https://beam.apache.org/contribute/release-guide/#run-validation-tests)
>>>> passes on my machine. It looks like we are likely missing test coverage
>>>> where Romain is hitting issues.
>>>>
>>>> On Tue, Sep 11, 2018 at 12:15 PM Ahmet Altay <al...@google.com> wrote:
>>>>
>>>> Could anyone else help with looking at these issues earlier?
>>>>
>>>> On Tue, Sep 11, 2018 at 12:03 PM, Romain Manni-Bucau <
>>>> rmannibucau@gmail.com> wrote:
>>>>
>>>> Im running this main [1] through this IT [2]. Was working fine since ~1
>>>> year but 2.7.0 broke it. Didnt investigate more but can have a look later
>>>> this month if it helps.
>>>>
>>>> [1]
>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
>>>> [2]
>>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
>>>>
>>>> Le mar. 11 sept. 2018 20:54, Charles Chen <cc...@google.com> a écrit :
>>>>
>>>> Romain: can you give more details on the failure you're encountering,
>>>> i.e. how you are performing this validation?
>>>>
>>>> On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> weird, I didn't have it on Beam samples. Let me try to reproduce and I
>>>> will create the Jira.
>>>>
>>>> Regards
>>>> JB
>>>>
>>>> On 11/09/2018 11:44, Romain Manni-Bucau wrote:
>>>> > -1, seems spark integration is broken (tested with spark 2.3.1 and
>>>> 2.2.1):
>>>> >
>>>> > 18/09/11 11:33:29 WARN TaskSetManager: Lost task 0.0 in stage 0.0
>>>> (TID 0, RMANNIBUCAU, executor 0): java.lang.ClassCastException: cannot
>>>> assign instance of scala.collection.immutable.List$SerializationProxy to
>>>> fieldorg.apache.spark.rdd.RDD.org
>>>> <http://fieldorg.apache.spark.rdd.rdd.org/> <
>>>> http://org.apache.spark.rdd.RDD.org
>>>> <http://org.apache.spark.rdd.rdd.org/>>$apache$spark$rdd$RDD$$dependencies_
>>>> of type scala.collection.Seq in instance of
>>>> org.apache.spark.rdd.MapPartitionsRDD
>>>> > at
>>>> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>>>> >
>>>> >
>>>> > Also the issue Lukasz identified is important even if workarounds can
>>>> be
>>>> > put in place so +1 to fix it as well if possible.
>>>> >
>>>> > Romain Manni-Bucau
>>>> > @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>> > <https://rmannibucau.metawerx.net/> | Old Blog
>>>> > <http://rmannibucau.wordpress.com> | Github
>>>> > <https://github.com/rmannibucau> | LinkedIn
>>>> > <https://www.linkedin.com/in/rmannibucau> | Book
>>>> > <
>>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>> >
>>>> >
>>>> >
>>>> > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik <lcwik@google.com
>>>> > <ma...@google.com>> a écrit :
>>>> >
>>>> > I found an issue where we are no longer packaging the pom.xml
>>>> within
>>>> > the artifact jars at META-INF/maven/groupId/artifactId. More
>>>> details
>>>> > in https://issues.apache.org/jira/browse/BEAM-5351. I wouldn't
>>>> > consider this a blocker but it was an easy fix
>>>> > (https://github.com/apache/beam/pull/6358) and users may rely on
>>>> the
>>>> > pom.xml.
>>>> >
>>>> > Should we recut the release candidate to include this?
>>>> >
>>>> > On Mon, Sep 10, 2018 at 4:58 AM Jean-Baptiste Onofré
>>>> > <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
>>>> >
>>>> > +1 (binding)
>>>> >
>>>> > Tested successfully on Beam Samples.
>>>> >
>>>> > Thanks !
>>>> >
>>>> > Regards
>>>> > JB
>>>> >
>>>> > On 07/09/2018 23:56, Charles Chen wrote:
>>>> > > Hi everyone,
>>>> > >
>>>> > > Please review and vote on the release candidate #1 for the
>>>> > version
>>>> > > 2.7.0, as follows:
>>>> > > [ ] +1, Approve the release
>>>> > > [ ] -1, Do not approve the release (please provide specific
>>>> > comments)
>>>> > >
>>>> > > The complete staging area is available for your review,
>>>> which
>>>> > includes:
>>>> > > * JIRA release notes [1],
>>>> > > * the official Apache source release to be deployed to
>>>> > dist.apache.org <http://dist.apache.org>
>>>> > > <http://dist.apache.org> [2], which is signed with the
>>>> key with
>>>> > > fingerprint 45C60AAAD115F560 [3],
>>>> > > * all artifacts to be deployed to the Maven Central
>>>> > Repository [4],
>>>> > > * source code tag "v2.7.0-RC1" [5],
>>>> > > * website pull request listing the release and publishing
>>>> the API
>>>> > > reference manual [6].
>>>> > > * Java artifacts were built with Gradle 4.8 and OpenJDK
>>>> > > 1.8.0_181-8u181-b13-1~deb9u1-b13.
>>>> > > * Python artifacts are deployed along with the source
>>>> release
>>>> > to the
>>>> > > dist.apache.org <http://dist.apache.org>
>>>> > <http://dist.apache.org> [2].
>>>> > >
>>>> > > The vote will be open for at least 72 hours. It is adopted
>>>> by
>>>> > majority
>>>> > > approval, with at least 3 PMC affirmative votes.
>>>> > >
>>>> > > Thanks,
>>>> > > Charles
>>>> > >
>>>> > > [1]
>>>> > >
>>>> >
>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>>>> > > [2] https://dist.apache.org/repos/dist/dev/beam/2.7.0
>>>> > > [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
>>>> > > [4]
>>>> >
>>>> https://repository.apache.org/content/repositories/orgapachebeam-1046/
>>>> > > [5] https://github.com/apache/beam/tree/v2.7.0-RC1
>>>> > > [6] https://github.com/apache/beam-site/pull/549
>>>> >
>>>> > --
>>>> > Jean-Baptiste Onofré
>>>> > jbonofre@apache.org <ma...@apache.org>
>>>> > http://blog.nanthrax.net
>>>> > Talend - http://www.talend.com
>>>> >
>>>>
>>>>
>>>>
>>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Raghu Angadi <ra...@google.com>.
I would like propose one more cherrypick for RC2 :
https://github.com/apache/beam/pull/6391
This is a KafkaIO bug fix. Once a user hits this bug, there is no easy work
around for them, especially on Dataflow. Only work around in Dataflow is to
restart or reload the job.
The fix itself fairly safe and is tested.
Raghu.
On Fri, Sep 14, 2018 at 12:52 AM Alexey Romanenko <ar...@gmail.com>
wrote:
> Perhaps it could help, but I run simple WordCount (built with Beam 2.7) on
> YARN/Spark (HDP Sandbox) cluster and it worked fine for me.
>
> On 14 Sep 2018, at 06:56, Romain Manni-Bucau <rm...@gmail.com>
> wrote:
>
> Hi Charles,
>
> I didn't get enough time to check deeply but it is clearly a dependency
> issue and it is not in beam spark runner itself but in another transitive
> module of beam. It does not happen in existing spark test cause none of
> them are in a cluster (even just with 1 worker) but this seems to be a
> regression since 2.6 works OOTB.
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com/> | Github
> <https://github.com/rmannibucau> | LinkedIn
> <https://www.linkedin.com/in/rmannibucau> | Book
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>
>
> Le jeu. 13 sept. 2018 à 22:15, Charles Chen <cc...@google.com> a écrit :
>
>> Romain and JB, can you please add the results of your investigations into
>> the errors you've seen above? Given that the existing SparkRunner tests
>> pass for this RC, and that the integration test you ran is in another repo
>> that is not continuously tested with Beam, it is not clear how we should
>> move forward and whether this is a blocking issue, unless we can find a
>> root cause in Beam.
>>
>> On Wed, Sep 12, 2018 at 2:08 AM Etienne Chauchot <ec...@apache.org>
>> wrote:
>>
>>> Hi all,
>>>
>>> on a performance and functional regression stand point I see no
>>> regression:
>>>
>>> I looked at nexmark graphs "output pcollection size" and "execution
>>> time" around release cut date on dataflow, spark, flink and direct runner
>>> in batch and streaming modes. There seems to be no regression.
>>>
>>> Etienne
>>>
>>> Le mardi 11 septembre 2018 à 12:25 -0700, Charles Chen a écrit :
>>>
>>> The SparkRunner validation test (here:
>>> https://beam.apache.org/contribute/release-guide/#run-validation-tests)
>>> passes on my machine. It looks like we are likely missing test coverage
>>> where Romain is hitting issues.
>>>
>>> On Tue, Sep 11, 2018 at 12:15 PM Ahmet Altay <al...@google.com> wrote:
>>>
>>> Could anyone else help with looking at these issues earlier?
>>>
>>> On Tue, Sep 11, 2018 at 12:03 PM, Romain Manni-Bucau <
>>> rmannibucau@gmail.com> wrote:
>>>
>>> Im running this main [1] through this IT [2]. Was working fine since ~1
>>> year but 2.7.0 broke it. Didnt investigate more but can have a look later
>>> this month if it helps.
>>>
>>> [1]
>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
>>> [2]
>>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
>>>
>>> Le mar. 11 sept. 2018 20:54, Charles Chen <cc...@google.com> a écrit :
>>>
>>> Romain: can you give more details on the failure you're encountering,
>>> i.e. how you are performing this validation?
>>>
>>> On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>>> wrote:
>>>
>>> Hi,
>>>
>>> weird, I didn't have it on Beam samples. Let me try to reproduce and I
>>> will create the Jira.
>>>
>>> Regards
>>> JB
>>>
>>> On 11/09/2018 11:44, Romain Manni-Bucau wrote:
>>> > -1, seems spark integration is broken (tested with spark 2.3.1 and
>>> 2.2.1):
>>> >
>>> > 18/09/11 11:33:29 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID
>>> 0, RMANNIBUCAU, executor 0): java.lang.ClassCastException: cannot assign
>>> instance of scala.collection.immutable.List$SerializationProxy to
>>> fieldorg.apache.spark.rdd.RDD.org
>>> <http://fieldorg.apache.spark.rdd.rdd.org/> <
>>> http://org.apache.spark.rdd.RDD.org
>>> <http://org.apache.spark.rdd.rdd.org/>>$apache$spark$rdd$RDD$$dependencies_
>>> of type scala.collection.Seq in instance of
>>> org.apache.spark.rdd.MapPartitionsRDD
>>> > at
>>> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>>> >
>>> >
>>> > Also the issue Lukasz identified is important even if workarounds can
>>> be
>>> > put in place so +1 to fix it as well if possible.
>>> >
>>> > Romain Manni-Bucau
>>> > @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>> > <https://rmannibucau.metawerx.net/> | Old Blog
>>> > <http://rmannibucau.wordpress.com> | Github
>>> > <https://github.com/rmannibucau> | LinkedIn
>>> > <https://www.linkedin.com/in/rmannibucau> | Book
>>> > <
>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>> >
>>> >
>>> >
>>> > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik <lcwik@google.com
>>> > <ma...@google.com>> a écrit :
>>> >
>>> > I found an issue where we are no longer packaging the pom.xml
>>> within
>>> > the artifact jars at META-INF/maven/groupId/artifactId. More
>>> details
>>> > in https://issues.apache.org/jira/browse/BEAM-5351. I wouldn't
>>> > consider this a blocker but it was an easy fix
>>> > (https://github.com/apache/beam/pull/6358) and users may rely on
>>> the
>>> > pom.xml.
>>> >
>>> > Should we recut the release candidate to include this?
>>> >
>>> > On Mon, Sep 10, 2018 at 4:58 AM Jean-Baptiste Onofré
>>> > <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
>>> >
>>> > +1 (binding)
>>> >
>>> > Tested successfully on Beam Samples.
>>> >
>>> > Thanks !
>>> >
>>> > Regards
>>> > JB
>>> >
>>> > On 07/09/2018 23:56, Charles Chen wrote:
>>> > > Hi everyone,
>>> > >
>>> > > Please review and vote on the release candidate #1 for the
>>> > version
>>> > > 2.7.0, as follows:
>>> > > [ ] +1, Approve the release
>>> > > [ ] -1, Do not approve the release (please provide specific
>>> > comments)
>>> > >
>>> > > The complete staging area is available for your review,
>>> which
>>> > includes:
>>> > > * JIRA release notes [1],
>>> > > * the official Apache source release to be deployed to
>>> > dist.apache.org <http://dist.apache.org>
>>> > > <http://dist.apache.org> [2], which is signed with the key
>>> with
>>> > > fingerprint 45C60AAAD115F560 [3],
>>> > > * all artifacts to be deployed to the Maven Central
>>> > Repository [4],
>>> > > * source code tag "v2.7.0-RC1" [5],
>>> > > * website pull request listing the release and publishing
>>> the API
>>> > > reference manual [6].
>>> > > * Java artifacts were built with Gradle 4.8 and OpenJDK
>>> > > 1.8.0_181-8u181-b13-1~deb9u1-b13.
>>> > > * Python artifacts are deployed along with the source
>>> release
>>> > to the
>>> > > dist.apache.org <http://dist.apache.org>
>>> > <http://dist.apache.org> [2].
>>> > >
>>> > > The vote will be open for at least 72 hours. It is adopted
>>> by
>>> > majority
>>> > > approval, with at least 3 PMC affirmative votes.
>>> > >
>>> > > Thanks,
>>> > > Charles
>>> > >
>>> > > [1]
>>> > >
>>> >
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>>> > > [2] https://dist.apache.org/repos/dist/dev/beam/2.7.0
>>> > > [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
>>> > > [4]
>>> >
>>> https://repository.apache.org/content/repositories/orgapachebeam-1046/
>>> > > [5] https://github.com/apache/beam/tree/v2.7.0-RC1
>>> > > [6] https://github.com/apache/beam-site/pull/549
>>> >
>>> > --
>>> > Jean-Baptiste Onofré
>>> > jbonofre@apache.org <ma...@apache.org>
>>> > http://blog.nanthrax.net
>>> > Talend - http://www.talend.com
>>> >
>>>
>>>
>>>
>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Alexey Romanenko <ar...@gmail.com>.
Perhaps it could help, but I run simple WordCount (built with Beam 2.7) on YARN/Spark (HDP Sandbox) cluster and it worked fine for me.
> On 14 Sep 2018, at 06:56, Romain Manni-Bucau <rm...@gmail.com> wrote:
>
> Hi Charles,
>
> I didn't get enough time to check deeply but it is clearly a dependency issue and it is not in beam spark runner itself but in another transitive module of beam. It does not happen in existing spark test cause none of them are in a cluster (even just with 1 worker) but this seems to be a regression since 2.6 works OOTB.
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com/> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>
> Le jeu. 13 sept. 2018 à 22:15, Charles Chen <ccy@google.com <ma...@google.com>> a écrit :
> Romain and JB, can you please add the results of your investigations into the errors you've seen above? Given that the existing SparkRunner tests pass for this RC, and that the integration test you ran is in another repo that is not continuously tested with Beam, it is not clear how we should move forward and whether this is a blocking issue, unless we can find a root cause in Beam.
>
> On Wed, Sep 12, 2018 at 2:08 AM Etienne Chauchot <echauchot@apache.org <ma...@apache.org>> wrote:
> Hi all,
>
> on a performance and functional regression stand point I see no regression:
>
> I looked at nexmark graphs "output pcollection size" and "execution time" around release cut date on dataflow, spark, flink and direct runner in batch and streaming modes. There seems to be no regression.
>
> Etienne
>
> Le mardi 11 septembre 2018 à 12:25 -0700, Charles Chen a écrit :
>> The SparkRunner validation test (here: https://beam.apache.org/contribute/release-guide/#run-validation-tests <https://beam.apache.org/contribute/release-guide/#run-validation-tests>) passes on my machine. It looks like we are likely missing test coverage where Romain is hitting issues.
>>
>> On Tue, Sep 11, 2018 at 12:15 PM Ahmet Altay <altay@google.com <ma...@google.com>> wrote:
>>> Could anyone else help with looking at these issues earlier?
>>>
>>> On Tue, Sep 11, 2018 at 12:03 PM, Romain Manni-Bucau <rmannibucau@gmail.com <ma...@gmail.com>> wrote:
>>>> Im running this main [1] through this IT [2]. Was working fine since ~1 year but 2.7.0 broke it. Didnt investigate more but can have a look later this month if it helps.
>>>>
>>>> [1] https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java <https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java>
>>>> [2] https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java <https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java>
>>>>
>>>> Le mar. 11 sept. 2018 20:54, Charles Chen <ccy@google.com <ma...@google.com>> a écrit :
>>>>> Romain: can you give more details on the failure you're encountering, i.e. how you are performing this validation?
>>>>>
>>>>> On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste Onofré <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> weird, I didn't have it on Beam samples. Let me try to reproduce and I
>>>>>> will create the Jira.
>>>>>>
>>>>>> Regards
>>>>>> JB
>>>>>>
>>>>>> On 11/09/2018 11:44, Romain Manni-Bucau wrote:
>>>>>> > -1, seems spark integration is broken (tested with spark 2.3.1 and 2.2.1):
>>>>>> >
>>>>>> > 18/09/11 11:33:29 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, RMANNIBUCAU, executor 0): java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to fieldorg.apache.spark.rdd.RDD.org <http://fieldorg.apache.spark.rdd.rdd.org/> <http://org.apache.spark.rdd.RDD.org <http://org.apache.spark.rdd.rdd.org/>>$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD
>>>>>> > at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>>>>>> >
>>>>>> >
>>>>>> > Also the issue Lukasz identified is important even if workarounds can be
>>>>>> > put in place so +1 to fix it as well if possible.
>>>>>> >
>>>>>> > Romain Manni-Bucau
>>>>>> > @rmannibucau <https://twitter.com/rmannibucau <https://twitter.com/rmannibucau>> | Blog
>>>>>> > <https://rmannibucau.metawerx.net/ <https://rmannibucau.metawerx.net/>> | Old Blog
>>>>>> > <http://rmannibucau.wordpress.com <http://rmannibucau.wordpress.com/>> | Github
>>>>>> > <https://github.com/rmannibucau <https://github.com/rmannibucau>> | LinkedIn
>>>>>> > <https://www.linkedin.com/in/rmannibucau <https://www.linkedin.com/in/rmannibucau>> | Book
>>>>>> > <https://www.packtpub.com/application-development/java-ee-8-high-performance <https://www.packtpub.com/application-development/java-ee-8-high-performance>>
>>>>>> >
>>>>>> >
>>>>>> > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik <lcwik@google.com <ma...@google.com>
>>>>>> > <mailto:lcwik@google.com <ma...@google.com>>> a écrit :
>>>>>> >
>>>>>> > I found an issue where we are no longer packaging the pom.xml within
>>>>>> > the artifact jars at META-INF/maven/groupId/artifactId. More details
>>>>>> > in https://issues.apache.org/jira/browse/BEAM-5351 <https://issues.apache.org/jira/browse/BEAM-5351>. I wouldn't
>>>>>> > consider this a blocker but it was an easy fix
>>>>>> > (https://github.com/apache/beam/pull/6358 <https://github.com/apache/beam/pull/6358>) and users may rely on the
>>>>>> > pom.xml.
>>>>>> >
>>>>>> > Should we recut the release candidate to include this?
>>>>>> >
>>>>>> > On Mon, Sep 10, 2018 at 4:58 AM Jean-Baptiste Onofré
>>>>>> > <jb@nanthrax.net <ma...@nanthrax.net> <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> wrote:
>>>>>> >
>>>>>> > +1 (binding)
>>>>>> >
>>>>>> > Tested successfully on Beam Samples.
>>>>>> >
>>>>>> > Thanks !
>>>>>> >
>>>>>> > Regards
>>>>>> > JB
>>>>>> >
>>>>>> > On 07/09/2018 23:56, Charles Chen wrote:
>>>>>> > > Hi everyone,
>>>>>> > >
>>>>>> > > Please review and vote on the release candidate #1 for the
>>>>>> > version
>>>>>> > > 2.7.0, as follows:
>>>>>> > > [ ] +1, Approve the release
>>>>>> > > [ ] -1, Do not approve the release (please provide specific
>>>>>> > comments)
>>>>>> > >
>>>>>> > > The complete staging area is available for your review, which
>>>>>> > includes:
>>>>>> > > * JIRA release notes [1],
>>>>>> > > * the official Apache source release to be deployed to
>>>>>> > dist.apache.org <http://dist.apache.org/> <http://dist.apache.org <http://dist.apache.org/>>
>>>>>> > > <http://dist.apache.org <http://dist.apache.org/>> [2], which is signed with the key with
>>>>>> > > fingerprint 45C60AAAD115F560 [3],
>>>>>> > > * all artifacts to be deployed to the Maven Central
>>>>>> > Repository [4],
>>>>>> > > * source code tag "v2.7.0-RC1" [5],
>>>>>> > > * website pull request listing the release and publishing the API
>>>>>> > > reference manual [6].
>>>>>> > > * Java artifacts were built with Gradle 4.8 and OpenJDK
>>>>>> > > 1.8.0_181-8u181-b13-1~deb9u1-b13.
>>>>>> > > * Python artifacts are deployed along with the source release
>>>>>> > to the
>>>>>> > > dist.apache.org <http://dist.apache.org/> <http://dist.apache.org <http://dist.apache.org/>>
>>>>>> > <http://dist.apache.org <http://dist.apache.org/>> [2].
>>>>>> > >
>>>>>> > > The vote will be open for at least 72 hours. It is adopted by
>>>>>> > majority
>>>>>> > > approval, with at least 3 PMC affirmative votes.
>>>>>> > >
>>>>>> > > Thanks,
>>>>>> > > Charles
>>>>>> > >
>>>>>> > > [1]
>>>>>> > >
>>>>>> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654 <https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654>
>>>>>> > > [2] https://dist.apache.org/repos/dist/dev/beam/2.7.0 <https://dist.apache.org/repos/dist/dev/beam/2.7.0>
>>>>>> > > [3] https://dist.apache.org/repos/dist/dev/beam/KEYS <https://dist.apache.org/repos/dist/dev/beam/KEYS>
>>>>>> > > [4]
>>>>>> > https://repository.apache.org/content/repositories/orgapachebeam-1046/ <https://repository.apache.org/content/repositories/orgapachebeam-1046/>
>>>>>> > > [5] https://github.com/apache/beam/tree/v2.7.0-RC1 <https://github.com/apache/beam/tree/v2.7.0-RC1>
>>>>>> > > [6] https://github.com/apache/beam-site/pull/549 <https://github.com/apache/beam-site/pull/549>
>>>>>> >
>>>>>> > --
>>>>>> > Jean-Baptiste Onofré
>>>>>> > jbonofre@apache.org <ma...@apache.org> <mailto:jbonofre@apache.org <ma...@apache.org>>
>>>>>> > http://blog.nanthrax.net <http://blog.nanthrax.net/>
>>>>>> > Talend - http://www.talend.com <http://www.talend.com/>
>>>>>> >
>>>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Hi Charles,
I didn't get enough time to check deeply but it is clearly a dependency
issue and it is not in beam spark runner itself but in another transitive
module of beam. It does not happen in existing spark test cause none of
them are in a cluster (even just with 1 worker) but this seems to be a
regression since 2.6 works OOTB.
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>
Le jeu. 13 sept. 2018 à 22:15, Charles Chen <cc...@google.com> a écrit :
> Romain and JB, can you please add the results of your investigations into
> the errors you've seen above? Given that the existing SparkRunner tests
> pass for this RC, and that the integration test you ran is in another repo
> that is not continuously tested with Beam, it is not clear how we should
> move forward and whether this is a blocking issue, unless we can find a
> root cause in Beam.
>
> On Wed, Sep 12, 2018 at 2:08 AM Etienne Chauchot <ec...@apache.org>
> wrote:
>
>> Hi all,
>>
>> on a performance and functional regression stand point I see no
>> regression:
>>
>> I looked at nexmark graphs "output pcollection size" and "execution time"
>> around release cut date on dataflow, spark, flink and direct runner in
>> batch and streaming modes. There seems to be no regression.
>>
>> Etienne
>>
>> Le mardi 11 septembre 2018 à 12:25 -0700, Charles Chen a écrit :
>>
>> The SparkRunner validation test (here:
>> https://beam.apache.org/contribute/release-guide/#run-validation-tests)
>> passes on my machine. It looks like we are likely missing test coverage
>> where Romain is hitting issues.
>>
>> On Tue, Sep 11, 2018 at 12:15 PM Ahmet Altay <al...@google.com> wrote:
>>
>> Could anyone else help with looking at these issues earlier?
>>
>> On Tue, Sep 11, 2018 at 12:03 PM, Romain Manni-Bucau <
>> rmannibucau@gmail.com> wrote:
>>
>> Im running this main [1] through this IT [2]. Was working fine since ~1
>> year but 2.7.0 broke it. Didnt investigate more but can have a look later
>> this month if it helps.
>>
>> [1]
>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
>> [2]
>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
>>
>> Le mar. 11 sept. 2018 20:54, Charles Chen <cc...@google.com> a écrit :
>>
>> Romain: can you give more details on the failure you're encountering,
>> i.e. how you are performing this validation?
>>
>> On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>> wrote:
>>
>> Hi,
>>
>> weird, I didn't have it on Beam samples. Let me try to reproduce and I
>> will create the Jira.
>>
>> Regards
>> JB
>>
>> On 11/09/2018 11:44, Romain Manni-Bucau wrote:
>> > -1, seems spark integration is broken (tested with spark 2.3.1 and
>> 2.2.1):
>> >
>> > 18/09/11 11:33:29 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID
>> 0, RMANNIBUCAU, executor 0): java.lang.ClassCastException: cannot assign
>> instance of scala.collection.immutable.List$SerializationProxy to
>> fieldorg.apache.spark.rdd.RDD.org <http://org.apache.spark.rdd.RDD.org>$apache$spark$rdd$RDD$$dependencies_
>> of type scala.collection.Seq in instance of
>> org.apache.spark.rdd.MapPartitionsRDD
>> > at
>> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>> >
>> >
>> > Also the issue Lukasz identified is important even if workarounds can
>> be
>> > put in place so +1 to fix it as well if possible.
>> >
>> > Romain Manni-Bucau
>> > @rmannibucau <https://twitter.com/rmannibucau> | Blog
>> > <https://rmannibucau.metawerx.net/> | Old Blog
>> > <http://rmannibucau.wordpress.com> | Github
>> > <https://github.com/rmannibucau> | LinkedIn
>> > <https://www.linkedin.com/in/rmannibucau> | Book
>> > <
>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>> >
>> >
>> >
>> > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik <lcwik@google.com
>> > <ma...@google.com>> a écrit :
>> >
>> > I found an issue where we are no longer packaging the pom.xml within
>> > the artifact jars at META-INF/maven/groupId/artifactId. More details
>> > in https://issues.apache.org/jira/browse/BEAM-5351. I wouldn't
>> > consider this a blocker but it was an easy fix
>> > (https://github.com/apache/beam/pull/6358) and users may rely on
>> the
>> > pom.xml.
>> >
>> > Should we recut the release candidate to include this?
>> >
>> > On Mon, Sep 10, 2018 at 4:58 AM Jean-Baptiste Onofré
>> > <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
>> >
>> > +1 (binding)
>> >
>> > Tested successfully on Beam Samples.
>> >
>> > Thanks !
>> >
>> > Regards
>> > JB
>> >
>> > On 07/09/2018 23:56, Charles Chen wrote:
>> > > Hi everyone,
>> > >
>> > > Please review and vote on the release candidate #1 for the
>> > version
>> > > 2.7.0, as follows:
>> > > [ ] +1, Approve the release
>> > > [ ] -1, Do not approve the release (please provide specific
>> > comments)
>> > >
>> > > The complete staging area is available for your review, which
>> > includes:
>> > > * JIRA release notes [1],
>> > > * the official Apache source release to be deployed to
>> > dist.apache.org <http://dist.apache.org>
>> > > <http://dist.apache.org> [2], which is signed with the key
>> with
>> > > fingerprint 45C60AAAD115F560 [3],
>> > > * all artifacts to be deployed to the Maven Central
>> > Repository [4],
>> > > * source code tag "v2.7.0-RC1" [5],
>> > > * website pull request listing the release and publishing
>> the API
>> > > reference manual [6].
>> > > * Java artifacts were built with Gradle 4.8 and OpenJDK
>> > > 1.8.0_181-8u181-b13-1~deb9u1-b13.
>> > > * Python artifacts are deployed along with the source release
>> > to the
>> > > dist.apache.org <http://dist.apache.org>
>> > <http://dist.apache.org> [2].
>> > >
>> > > The vote will be open for at least 72 hours. It is adopted by
>> > majority
>> > > approval, with at least 3 PMC affirmative votes.
>> > >
>> > > Thanks,
>> > > Charles
>> > >
>> > > [1]
>> > >
>> >
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>> > > [2] https://dist.apache.org/repos/dist/dev/beam/2.7.0
>> > > [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
>> > > [4]
>> >
>> https://repository.apache.org/content/repositories/orgapachebeam-1046/
>> > > [5] https://github.com/apache/beam/tree/v2.7.0-RC1
>> > > [6] https://github.com/apache/beam-site/pull/549
>> >
>> > --
>> > Jean-Baptiste Onofré
>> > jbonofre@apache.org <ma...@apache.org>
>> > http://blog.nanthrax.net
>> > Talend - http://www.talend.com
>> >
>>
>>
>>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Charles Chen <cc...@google.com>.
Romain and JB, can you please add the results of your investigations into
the errors you've seen above? Given that the existing SparkRunner tests
pass for this RC, and that the integration test you ran is in another repo
that is not continuously tested with Beam, it is not clear how we should
move forward and whether this is a blocking issue, unless we can find a
root cause in Beam.
On Wed, Sep 12, 2018 at 2:08 AM Etienne Chauchot <ec...@apache.org>
wrote:
> Hi all,
>
> on a performance and functional regression stand point I see no regression:
>
> I looked at nexmark graphs "output pcollection size" and "execution time"
> around release cut date on dataflow, spark, flink and direct runner in
> batch and streaming modes. There seems to be no regression.
>
> Etienne
>
> Le mardi 11 septembre 2018 à 12:25 -0700, Charles Chen a écrit :
>
> The SparkRunner validation test (here:
> https://beam.apache.org/contribute/release-guide/#run-validation-tests)
> passes on my machine. It looks like we are likely missing test coverage
> where Romain is hitting issues.
>
> On Tue, Sep 11, 2018 at 12:15 PM Ahmet Altay <al...@google.com> wrote:
>
> Could anyone else help with looking at these issues earlier?
>
> On Tue, Sep 11, 2018 at 12:03 PM, Romain Manni-Bucau <
> rmannibucau@gmail.com> wrote:
>
> Im running this main [1] through this IT [2]. Was working fine since ~1
> year but 2.7.0 broke it. Didnt investigate more but can have a look later
> this month if it helps.
>
> [1]
> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
> [2]
> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
>
> Le mar. 11 sept. 2018 20:54, Charles Chen <cc...@google.com> a écrit :
>
> Romain: can you give more details on the failure you're encountering, i.e.
> how you are performing this validation?
>
> On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> Hi,
>
> weird, I didn't have it on Beam samples. Let me try to reproduce and I
> will create the Jira.
>
> Regards
> JB
>
> On 11/09/2018 11:44, Romain Manni-Bucau wrote:
> > -1, seems spark integration is broken (tested with spark 2.3.1 and
> 2.2.1):
> >
> > 18/09/11 11:33:29 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID
> 0, RMANNIBUCAU, executor 0): java.lang.ClassCastException: cannot assign
> instance of scala.collection.immutable.List$SerializationProxy to
> fieldorg.apache.spark.rdd.RDD.org <http://org.apache.spark.rdd.RDD.org>$apache$spark$rdd$RDD$$dependencies_
> of type scala.collection.Seq in instance of
> org.apache.spark.rdd.MapPartitionsRDD
> > at
> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
> >
> >
> > Also the issue Lukasz identified is important even if workarounds can be
> > put in place so +1 to fix it as well if possible.
> >
> > Romain Manni-Bucau
> > @rmannibucau <https://twitter.com/rmannibucau> | Blog
> > <https://rmannibucau.metawerx.net/> | Old Blog
> > <http://rmannibucau.wordpress.com> | Github
> > <https://github.com/rmannibucau> | LinkedIn
> > <https://www.linkedin.com/in/rmannibucau> | Book
> > <
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> >
> >
> >
> > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik <lcwik@google.com
> > <ma...@google.com>> a écrit :
> >
> > I found an issue where we are no longer packaging the pom.xml within
> > the artifact jars at META-INF/maven/groupId/artifactId. More details
> > in https://issues.apache.org/jira/browse/BEAM-5351. I wouldn't
> > consider this a blocker but it was an easy fix
> > (https://github.com/apache/beam/pull/6358) and users may rely on the
> > pom.xml.
> >
> > Should we recut the release candidate to include this?
> >
> > On Mon, Sep 10, 2018 at 4:58 AM Jean-Baptiste Onofré
> > <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
> >
> > +1 (binding)
> >
> > Tested successfully on Beam Samples.
> >
> > Thanks !
> >
> > Regards
> > JB
> >
> > On 07/09/2018 23:56, Charles Chen wrote:
> > > Hi everyone,
> > >
> > > Please review and vote on the release candidate #1 for the
> > version
> > > 2.7.0, as follows:
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific
> > comments)
> > >
> > > The complete staging area is available for your review, which
> > includes:
> > > * JIRA release notes [1],
> > > * the official Apache source release to be deployed to
> > dist.apache.org <http://dist.apache.org>
> > > <http://dist.apache.org> [2], which is signed with the key
> with
> > > fingerprint 45C60AAAD115F560 [3],
> > > * all artifacts to be deployed to the Maven Central
> > Repository [4],
> > > * source code tag "v2.7.0-RC1" [5],
> > > * website pull request listing the release and publishing the
> API
> > > reference manual [6].
> > > * Java artifacts were built with Gradle 4.8 and OpenJDK
> > > 1.8.0_181-8u181-b13-1~deb9u1-b13.
> > > * Python artifacts are deployed along with the source release
> > to the
> > > dist.apache.org <http://dist.apache.org>
> > <http://dist.apache.org> [2].
> > >
> > > The vote will be open for at least 72 hours. It is adopted by
> > majority
> > > approval, with at least 3 PMC affirmative votes.
> > >
> > > Thanks,
> > > Charles
> > >
> > > [1]
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
> > > [2] https://dist.apache.org/repos/dist/dev/beam/2.7.0
> > > [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
> > > [4]
> >
> https://repository.apache.org/content/repositories/orgapachebeam-1046/
> > > [5] https://github.com/apache/beam/tree/v2.7.0-RC1
> > > [6] https://github.com/apache/beam-site/pull/549
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org <ma...@apache.org>
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
>
>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Etienne Chauchot <ec...@apache.org>.
Hi all,
on a performance and functional regression stand point I see no regression:
I looked at nexmark graphs "output pcollection size" and "execution time" around release cut date on dataflow, spark,
flink and direct runner in batch and streaming modes. There seems to be no regression.
Etienne
Le mardi 11 septembre 2018 à 12:25 -0700, Charles Chen a écrit :
> The SparkRunner validation test (here: https://beam.apache.org/contribute/release-guide/#run-validation-tests) passes
> on my machine. It looks like we are likely missing test coverage where Romain is hitting issues.
> On Tue, Sep 11, 2018 at 12:15 PM Ahmet Altay <al...@google.com> wrote:
> > Could anyone else help with looking at these issues earlier?
> > On Tue, Sep 11, 2018 at 12:03 PM, Romain Manni-Bucau <rm...@gmail.com> wrote:
> > > Im running this main [1] through this IT [2]. Was working fine since ~1 year but 2.7.0 broke it. Didnt investigate
> > > more but can have a look later this month if it helps.
> > > [1] https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-clust
> > > er/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
> > > [2] https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-clust
> > > er/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
> > >
> > > Le mar. 11 sept. 2018 20:54, Charles Chen <cc...@google.com> a écrit :
> > > > Romain: can you give more details on the failure you're encountering, i.e. how you are performing this
> > > > validation?
> > > >
> > > > On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> > > > > Hi,
> > > > >
> > > > >
> > > > >
> > > > > weird, I didn't have it on Beam samples. Let me try to reproduce and I
> > > > >
> > > > > will create the Jira.
> > > > >
> > > > >
> > > > >
> > > > > Regards
> > > > >
> > > > > JB
> > > > >
> > > > >
> > > > >
> > > > > On 11/09/2018 11:44, Romain Manni-Bucau wrote:
> > > > >
> > > > > > -1, seems spark integration is broken (tested with spark 2.3.1 and 2.2.1):
> > > > >
> > > > > >
> > > > >
> > > > > > 18/09/11 11:33:29 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, RMANNIBUCAU, executor 0):
> > > > > java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to
> > > > > fieldorg.apache.spark.rdd.RDD.org <http://org.apache.spark.rdd.RDD.org>$apache$spark$rdd$RDD$$dependencies_ of
> > > > > type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD
> > > > >
> > > > > > at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
> > > > >
> > > > > >
> > > > >
> > > > > >
> > > > >
> > > > > > Also the issue Lukasz identified is important even if workarounds can be
> > > > >
> > > > > > put in place so +1 to fix it as well if possible.
> > > > >
> > > > > >
> > > > >
> > > > > > Romain Manni-Bucau
> > > > >
> > > > > > @rmannibucau <https://twitter.com/rmannibucau> | Blog
> > > > >
> > > > > > <https://rmannibucau.metawerx.net/>t; | Old Blog
> > > > >
> > > > > > <http://rmannibucau.wordpress.com> | Github
> > > > >
> > > > > > <https://github.com/rmannibucau>t; | LinkedIn
> > > > >
> > > > > > <https://www.linkedin.com/in/rmannibucau>t; | Book
> > > > >
> > > > > > <https://www.packtpub.com/application-development/java-ee-8-high-performance>
> > > > >
> > > > > >
> > > > >
> > > > > >
> > > > >
> > > > > > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik <lcwik@google.com
> > > > >
> > > > > > <ma...@google.com>> a écrit :
> > > > >
> > > > > >
> > > > >
> > > > > > I found an issue where we are no longer packaging the pom.xml within
> > > > >
> > > > > > the artifact jars at META-INF/maven/groupId/artifactId. More details
> > > > >
> > > > > > in https://issues.apache.org/jira/browse/BEAM-5351. I wouldn't
> > > > >
> > > > > > consider this a blocker but it was an easy fix
> > > > >
> > > > > > (https://github.com/apache/beam/pull/6358) and users may rely on the
> > > > >
> > > > > > pom.xml.
> > > > >
> > > > > >
> > > > >
> > > > > > Should we recut the release candidate to include this?
> > > > >
> > > > > >
> > > > >
> > > > > > On Mon, Sep 10, 2018 at 4:58 AM Jean-Baptiste Onofré
> > > > >
> > > > > > <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
> > > > >
> > > > > >
> > > > >
> > > > > > +1 (binding)
> > > > >
> > > > > >
> > > > >
> > > > > > Tested successfully on Beam Samples.
> > > > >
> > > > > >
> > > > >
> > > > > > Thanks !
> > > > >
> > > > > >
> > > > >
> > > > > > Regards
> > > > >
> > > > > > JB
> > > > >
> > > > > >
> > > > >
> > > > > > On 07/09/2018 23:56, Charles Chen wrote:
> > > > >
> > > > > > > Hi everyone,
> > > > >
> > > > > > >
> > > > >
> > > > > > > Please review and vote on the release candidate #1 for the
> > > > >
> > > > > > version
> > > > >
> > > > > > > 2.7.0, as follows:
> > > > >
> > > > > > > [ ] +1, Approve the release
> > > > >
> > > > > > > [ ] -1, Do not approve the release (please provide specific
> > > > >
> > > > > > comments)
> > > > >
> > > > > > >
> > > > >
> > > > > > > The complete staging area is available for your review, which
> > > > >
> > > > > > includes:
> > > > >
> > > > > > > * JIRA release notes [1],
> > > > >
> > > > > > > * the official Apache source release to be deployed to
> > > > >
> > > > > > dist.apache.org <http://dist.apache.org>
> > > > >
> > > > > > > <http://dist.apache.org> [2], which is signed with the key with
> > > > >
> > > > > > > fingerprint 45C60AAAD115F560 [3],
> > > > >
> > > > > > > * all artifacts to be deployed to the Maven Central
> > > > >
> > > > > > Repository [4],
> > > > >
> > > > > > > * source code tag "v2.7.0-RC1" [5],
> > > > >
> > > > > > > * website pull request listing the release and publishing the API
> > > > >
> > > > > > > reference manual [6].
> > > > >
> > > > > > > * Java artifacts were built with Gradle 4.8 and OpenJDK
> > > > >
> > > > > > > 1.8.0_181-8u181-b13-1~deb9u1-b13.
> > > > >
> > > > > > > * Python artifacts are deployed along with the source release
> > > > >
> > > > > > to the
> > > > >
> > > > > > > dist.apache.org <http://dist.apache.org>
> > > > >
> > > > > > <http://dist.apache.org> [2].
> > > > >
> > > > > > >
> > > > >
> > > > > > > The vote will be open for at least 72 hours. It is adopted by
> > > > >
> > > > > > majority
> > > > >
> > > > > > > approval, with at least 3 PMC affirmative votes.
> > > > >
> > > > > > >
> > > > >
> > > > > > > Thanks,
> > > > >
> > > > > > > Charles
> > > > >
> > > > > > >
> > > > >
> > > > > > > [1]
> > > > >
> > > > > > >
> > > > >
> > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
> > > > >
> > > > > > > [2] https://dist.apache.org/repos/dist/dev/beam/2.7.0
> > > > >
> > > > > > > [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
> > > > >
> > > > > > > [4]
> > > > >
> > > > > > https://repository.apache.org/content/repositories/orgapachebeam-1046/
> > > > >
> > > > > > > [5] https://github.com/apache/beam/tree/v2.7.0-RC1
> > > > >
> > > > > > > [6] https://github.com/apache/beam-site/pull/549
> > > > >
> > > > > >
> > > > >
> > > > > > --
> > > > >
> > > > > > Jean-Baptiste Onofré
> > > > >
> > > > > > jbonofre@apache.org <ma...@apache.org>
> > > > >
> > > > > > http://blog.nanthrax.net
> > > > >
> > > > > > Talend - http://www.talend.com
> > > > >
> > > > > >
> > > > >
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Charles Chen <cc...@google.com>.
The SparkRunner validation test (here:
https://beam.apache.org/contribute/release-guide/#run-validation-tests)
passes on my machine. It looks like we are likely missing test coverage
where Romain is hitting issues.
On Tue, Sep 11, 2018 at 12:15 PM Ahmet Altay <al...@google.com> wrote:
> Could anyone else help with looking at these issues earlier?
>
> On Tue, Sep 11, 2018 at 12:03 PM, Romain Manni-Bucau <
> rmannibucau@gmail.com> wrote:
>
>> Im running this main [1] through this IT [2]. Was working fine since ~1
>> year but 2.7.0 broke it. Didnt investigate more but can have a look later
>> this month if it helps.
>>
>> [1]
>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
>> [2]
>> https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
>>
>> Le mar. 11 sept. 2018 20:54, Charles Chen <cc...@google.com> a écrit :
>>
>>> Romain: can you give more details on the failure you're encountering,
>>> i.e. how you are performing this validation?
>>>
>>> On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> weird, I didn't have it on Beam samples. Let me try to reproduce and I
>>>> will create the Jira.
>>>>
>>>> Regards
>>>> JB
>>>>
>>>> On 11/09/2018 11:44, Romain Manni-Bucau wrote:
>>>> > -1, seems spark integration is broken (tested with spark 2.3.1 and
>>>> 2.2.1):
>>>> >
>>>> > 18/09/11 11:33:29 WARN TaskSetManager: Lost task 0.0 in stage 0.0
>>>> (TID 0, RMANNIBUCAU, executor 0): java.lang.ClassCastException: cannot
>>>> assign instance of scala.collection.immutable.List$SerializationProxy to
>>>> fieldorg.apache.spark.rdd.RDD.org <http://org.apache.spark.rdd.RDD.org>$apache$spark$rdd$RDD$$dependencies_
>>>> of type scala.collection.Seq in instance of
>>>> org.apache.spark.rdd.MapPartitionsRDD
>>>> > at
>>>> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>>>> >
>>>> >
>>>> > Also the issue Lukasz identified is important even if workarounds can
>>>> be
>>>> > put in place so +1 to fix it as well if possible.
>>>> >
>>>> > Romain Manni-Bucau
>>>> > @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>> > <https://rmannibucau.metawerx.net/> | Old Blog
>>>> > <http://rmannibucau.wordpress.com> | Github
>>>> > <https://github.com/rmannibucau> | LinkedIn
>>>> > <https://www.linkedin.com/in/rmannibucau> | Book
>>>> > <
>>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>> >
>>>> >
>>>> >
>>>> > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik <lcwik@google.com
>>>> > <ma...@google.com>> a écrit :
>>>> >
>>>> > I found an issue where we are no longer packaging the pom.xml
>>>> within
>>>> > the artifact jars at META-INF/maven/groupId/artifactId. More
>>>> details
>>>> > in https://issues.apache.org/jira/browse/BEAM-5351. I wouldn't
>>>> > consider this a blocker but it was an easy fix
>>>> > (https://github.com/apache/beam/pull/6358) and users may rely on
>>>> the
>>>> > pom.xml.
>>>> >
>>>> > Should we recut the release candidate to include this?
>>>> >
>>>> > On Mon, Sep 10, 2018 at 4:58 AM Jean-Baptiste Onofré
>>>> > <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
>>>> >
>>>> > +1 (binding)
>>>> >
>>>> > Tested successfully on Beam Samples.
>>>> >
>>>> > Thanks !
>>>> >
>>>> > Regards
>>>> > JB
>>>> >
>>>> > On 07/09/2018 23:56, Charles Chen wrote:
>>>> > > Hi everyone,
>>>> > >
>>>> > > Please review and vote on the release candidate #1 for the
>>>> > version
>>>> > > 2.7.0, as follows:
>>>> > > [ ] +1, Approve the release
>>>> > > [ ] -1, Do not approve the release (please provide specific
>>>> > comments)
>>>> > >
>>>> > > The complete staging area is available for your review,
>>>> which
>>>> > includes:
>>>> > > * JIRA release notes [1],
>>>> > > * the official Apache source release to be deployed to
>>>> > dist.apache.org <http://dist.apache.org>
>>>> > > <http://dist.apache.org> [2], which is signed with the
>>>> key with
>>>> > > fingerprint 45C60AAAD115F560 [3],
>>>> > > * all artifacts to be deployed to the Maven Central
>>>> > Repository [4],
>>>> > > * source code tag "v2.7.0-RC1" [5],
>>>> > > * website pull request listing the release and publishing
>>>> the API
>>>> > > reference manual [6].
>>>> > > * Java artifacts were built with Gradle 4.8 and OpenJDK
>>>> > > 1.8.0_181-8u181-b13-1~deb9u1-b13.
>>>> > > * Python artifacts are deployed along with the source
>>>> release
>>>> > to the
>>>> > > dist.apache.org <http://dist.apache.org>
>>>> > <http://dist.apache.org> [2].
>>>> > >
>>>> > > The vote will be open for at least 72 hours. It is adopted
>>>> by
>>>> > majority
>>>> > > approval, with at least 3 PMC affirmative votes.
>>>> > >
>>>> > > Thanks,
>>>> > > Charles
>>>> > >
>>>> > > [1]
>>>> > >
>>>> >
>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>>>> > > [2] https://dist.apache.org/repos/dist/dev/beam/2.7.0
>>>> > > [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
>>>> > > [4]
>>>> >
>>>> https://repository.apache.org/content/repositories/orgapachebeam-1046/
>>>> > > [5] https://github.com/apache/beam/tree/v2.7.0-RC1
>>>> > > [6] https://github.com/apache/beam-site/pull/549
>>>> >
>>>> > --
>>>> > Jean-Baptiste Onofré
>>>> > jbonofre@apache.org <ma...@apache.org>
>>>> > http://blog.nanthrax.net
>>>> > Talend - http://www.talend.com
>>>> >
>>>>
>>>
>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Ahmet Altay <al...@google.com>.
Could anyone else help with looking at these issues earlier?
On Tue, Sep 11, 2018 at 12:03 PM, Romain Manni-Bucau <rm...@gmail.com>
wrote:
> Im running this main [1] through this IT [2]. Was working fine since ~1
> year but 2.7.0 broke it. Didnt investigate more but can have a look later
> this month if it helps.
>
> [1] https://github.com/Talend/component-runtime/blob/master/
> component-runtime-beam/src/it/serialization-over-cluster/
> src/main/java/org/talend/sdk/component/beam/it/
> clusterserialization/Main.java
> [2] https://github.com/Talend/component-runtime/blob/master/
> component-runtime-beam/src/it/serialization-over-cluster/
> src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.
> java
>
> Le mar. 11 sept. 2018 20:54, Charles Chen <cc...@google.com> a écrit :
>
>> Romain: can you give more details on the failure you're encountering,
>> i.e. how you are performing this validation?
>>
>> On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>> wrote:
>>
>>> Hi,
>>>
>>> weird, I didn't have it on Beam samples. Let me try to reproduce and I
>>> will create the Jira.
>>>
>>> Regards
>>> JB
>>>
>>> On 11/09/2018 11:44, Romain Manni-Bucau wrote:
>>> > -1, seems spark integration is broken (tested with spark 2.3.1 and
>>> 2.2.1):
>>> >
>>> > 18/09/11 11:33:29 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID
>>> 0, RMANNIBUCAU, executor 0): java.lang.ClassCastException: cannot assign
>>> instance of scala.collection.immutable.List$SerializationProxy to
>>> fieldorg.apache.spark.rdd.RDD.org <http://org.apache.spark.rdd.RDD.org
>>> >$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in
>>> instance of org.apache.spark.rdd.MapPartitionsRDD
>>> > at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(
>>> ObjectStreamClass.java:2233)
>>> >
>>> >
>>> > Also the issue Lukasz identified is important even if workarounds can
>>> be
>>> > put in place so +1 to fix it as well if possible.
>>> >
>>> > Romain Manni-Bucau
>>> > @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>> > <https://rmannibucau.metawerx.net/> | Old Blog
>>> > <http://rmannibucau.wordpress.com> | Github
>>> > <https://github.com/rmannibucau> | LinkedIn
>>> > <https://www.linkedin.com/in/rmannibucau> | Book
>>> > <https://www.packtpub.com/application-development/java-
>>> ee-8-high-performance>
>>> >
>>> >
>>> > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik <lcwik@google.com
>>> > <ma...@google.com>> a écrit :
>>> >
>>> > I found an issue where we are no longer packaging the pom.xml
>>> within
>>> > the artifact jars at META-INF/maven/groupId/artifactId. More
>>> details
>>> > in https://issues.apache.org/jira/browse/BEAM-5351. I wouldn't
>>> > consider this a blocker but it was an easy fix
>>> > (https://github.com/apache/beam/pull/6358) and users may rely on
>>> the
>>> > pom.xml.
>>> >
>>> > Should we recut the release candidate to include this?
>>> >
>>> > On Mon, Sep 10, 2018 at 4:58 AM Jean-Baptiste Onofré
>>> > <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
>>> >
>>> > +1 (binding)
>>> >
>>> > Tested successfully on Beam Samples.
>>> >
>>> > Thanks !
>>> >
>>> > Regards
>>> > JB
>>> >
>>> > On 07/09/2018 23:56, Charles Chen wrote:
>>> > > Hi everyone,
>>> > >
>>> > > Please review and vote on the release candidate #1 for the
>>> > version
>>> > > 2.7.0, as follows:
>>> > > [ ] +1, Approve the release
>>> > > [ ] -1, Do not approve the release (please provide specific
>>> > comments)
>>> > >
>>> > > The complete staging area is available for your review,
>>> which
>>> > includes:
>>> > > * JIRA release notes [1],
>>> > > * the official Apache source release to be deployed to
>>> > dist.apache.org <http://dist.apache.org>
>>> > > <http://dist.apache.org> [2], which is signed with the key
>>> with
>>> > > fingerprint 45C60AAAD115F560 [3],
>>> > > * all artifacts to be deployed to the Maven Central
>>> > Repository [4],
>>> > > * source code tag "v2.7.0-RC1" [5],
>>> > > * website pull request listing the release and publishing
>>> the API
>>> > > reference manual [6].
>>> > > * Java artifacts were built with Gradle 4.8 and OpenJDK
>>> > > 1.8.0_181-8u181-b13-1~deb9u1-b13.
>>> > > * Python artifacts are deployed along with the source
>>> release
>>> > to the
>>> > > dist.apache.org <http://dist.apache.org>
>>> > <http://dist.apache.org> [2].
>>> > >
>>> > > The vote will be open for at least 72 hours. It is adopted
>>> by
>>> > majority
>>> > > approval, with at least 3 PMC affirmative votes.
>>> > >
>>> > > Thanks,
>>> > > Charles
>>> > >
>>> > > [1]
>>> > >
>>> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>>> projectId=12319527&version=12343654
>>> > > [2] https://dist.apache.org/repos/dist/dev/beam/2.7.0
>>> > > [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
>>> > > [4]
>>> > https://repository.apache.org/content/repositories/
>>> orgapachebeam-1046/
>>> > > [5] https://github.com/apache/beam/tree/v2.7.0-RC1
>>> > > [6] https://github.com/apache/beam-site/pull/549
>>> >
>>> > --
>>> > Jean-Baptiste Onofré
>>> > jbonofre@apache.org <ma...@apache.org>
>>> > http://blog.nanthrax.net
>>> > Talend - http://www.talend.com
>>> >
>>>
>>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Im running this main [1] through this IT [2]. Was working fine since ~1
year but 2.7.0 broke it. Didnt investigate more but can have a look later
this month if it helps.
[1]
https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
[2]
https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
Le mar. 11 sept. 2018 20:54, Charles Chen <cc...@google.com> a écrit :
> Romain: can you give more details on the failure you're encountering, i.e.
> how you are performing this validation?
>
> On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> Hi,
>>
>> weird, I didn't have it on Beam samples. Let me try to reproduce and I
>> will create the Jira.
>>
>> Regards
>> JB
>>
>> On 11/09/2018 11:44, Romain Manni-Bucau wrote:
>> > -1, seems spark integration is broken (tested with spark 2.3.1 and
>> 2.2.1):
>> >
>> > 18/09/11 11:33:29 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID
>> 0, RMANNIBUCAU, executor 0): java.lang.ClassCastException: cannot assign
>> instance of scala.collection.immutable.List$SerializationProxy to
>> fieldorg.apache.spark.rdd.RDD.org <http://org.apache.spark.rdd.RDD.org>$apache$spark$rdd$RDD$$dependencies_
>> of type scala.collection.Seq in instance of
>> org.apache.spark.rdd.MapPartitionsRDD
>> > at
>> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>> >
>> >
>> > Also the issue Lukasz identified is important even if workarounds can
>> be
>> > put in place so +1 to fix it as well if possible.
>> >
>> > Romain Manni-Bucau
>> > @rmannibucau <https://twitter.com/rmannibucau> | Blog
>> > <https://rmannibucau.metawerx.net/> | Old Blog
>> > <http://rmannibucau.wordpress.com> | Github
>> > <https://github.com/rmannibucau> | LinkedIn
>> > <https://www.linkedin.com/in/rmannibucau> | Book
>> > <
>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>> >
>> >
>> >
>> > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik <lcwik@google.com
>> > <ma...@google.com>> a écrit :
>> >
>> > I found an issue where we are no longer packaging the pom.xml within
>> > the artifact jars at META-INF/maven/groupId/artifactId. More details
>> > in https://issues.apache.org/jira/browse/BEAM-5351. I wouldn't
>> > consider this a blocker but it was an easy fix
>> > (https://github.com/apache/beam/pull/6358) and users may rely on
>> the
>> > pom.xml.
>> >
>> > Should we recut the release candidate to include this?
>> >
>> > On Mon, Sep 10, 2018 at 4:58 AM Jean-Baptiste Onofré
>> > <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
>> >
>> > +1 (binding)
>> >
>> > Tested successfully on Beam Samples.
>> >
>> > Thanks !
>> >
>> > Regards
>> > JB
>> >
>> > On 07/09/2018 23:56, Charles Chen wrote:
>> > > Hi everyone,
>> > >
>> > > Please review and vote on the release candidate #1 for the
>> > version
>> > > 2.7.0, as follows:
>> > > [ ] +1, Approve the release
>> > > [ ] -1, Do not approve the release (please provide specific
>> > comments)
>> > >
>> > > The complete staging area is available for your review, which
>> > includes:
>> > > * JIRA release notes [1],
>> > > * the official Apache source release to be deployed to
>> > dist.apache.org <http://dist.apache.org>
>> > > <http://dist.apache.org> [2], which is signed with the key
>> with
>> > > fingerprint 45C60AAAD115F560 [3],
>> > > * all artifacts to be deployed to the Maven Central
>> > Repository [4],
>> > > * source code tag "v2.7.0-RC1" [5],
>> > > * website pull request listing the release and publishing
>> the API
>> > > reference manual [6].
>> > > * Java artifacts were built with Gradle 4.8 and OpenJDK
>> > > 1.8.0_181-8u181-b13-1~deb9u1-b13.
>> > > * Python artifacts are deployed along with the source release
>> > to the
>> > > dist.apache.org <http://dist.apache.org>
>> > <http://dist.apache.org> [2].
>> > >
>> > > The vote will be open for at least 72 hours. It is adopted by
>> > majority
>> > > approval, with at least 3 PMC affirmative votes.
>> > >
>> > > Thanks,
>> > > Charles
>> > >
>> > > [1]
>> > >
>> >
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>> > > [2] https://dist.apache.org/repos/dist/dev/beam/2.7.0
>> > > [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
>> > > [4]
>> >
>> https://repository.apache.org/content/repositories/orgapachebeam-1046/
>> > > [5] https://github.com/apache/beam/tree/v2.7.0-RC1
>> > > [6] https://github.com/apache/beam-site/pull/549
>> >
>> > --
>> > Jean-Baptiste Onofré
>> > jbonofre@apache.org <ma...@apache.org>
>> > http://blog.nanthrax.net
>> > Talend - http://www.talend.com
>> >
>>
>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Charles Chen <cc...@google.com>.
Romain: can you give more details on the failure you're encountering, i.e.
how you are performing this validation?
On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi,
>
> weird, I didn't have it on Beam samples. Let me try to reproduce and I
> will create the Jira.
>
> Regards
> JB
>
> On 11/09/2018 11:44, Romain Manni-Bucau wrote:
> > -1, seems spark integration is broken (tested with spark 2.3.1 and
> 2.2.1):
> >
> > 18/09/11 11:33:29 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID
> 0, RMANNIBUCAU, executor 0): java.lang.ClassCastException: cannot assign
> instance of scala.collection.immutable.List$SerializationProxy to
> fieldorg.apache.spark.rdd.RDD.org <http://org.apache.spark.rdd.RDD.org>$apache$spark$rdd$RDD$$dependencies_
> of type scala.collection.Seq in instance of
> org.apache.spark.rdd.MapPartitionsRDD
> > at
> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
> >
> >
> > Also the issue Lukasz identified is important even if workarounds can be
> > put in place so +1 to fix it as well if possible.
> >
> > Romain Manni-Bucau
> > @rmannibucau <https://twitter.com/rmannibucau> | Blog
> > <https://rmannibucau.metawerx.net/> | Old Blog
> > <http://rmannibucau.wordpress.com> | Github
> > <https://github.com/rmannibucau> | LinkedIn
> > <https://www.linkedin.com/in/rmannibucau> | Book
> > <
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> >
> >
> >
> > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik <lcwik@google.com
> > <ma...@google.com>> a écrit :
> >
> > I found an issue where we are no longer packaging the pom.xml within
> > the artifact jars at META-INF/maven/groupId/artifactId. More details
> > in https://issues.apache.org/jira/browse/BEAM-5351. I wouldn't
> > consider this a blocker but it was an easy fix
> > (https://github.com/apache/beam/pull/6358) and users may rely on the
> > pom.xml.
> >
> > Should we recut the release candidate to include this?
> >
> > On Mon, Sep 10, 2018 at 4:58 AM Jean-Baptiste Onofré
> > <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
> >
> > +1 (binding)
> >
> > Tested successfully on Beam Samples.
> >
> > Thanks !
> >
> > Regards
> > JB
> >
> > On 07/09/2018 23:56, Charles Chen wrote:
> > > Hi everyone,
> > >
> > > Please review and vote on the release candidate #1 for the
> > version
> > > 2.7.0, as follows:
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific
> > comments)
> > >
> > > The complete staging area is available for your review, which
> > includes:
> > > * JIRA release notes [1],
> > > * the official Apache source release to be deployed to
> > dist.apache.org <http://dist.apache.org>
> > > <http://dist.apache.org> [2], which is signed with the key
> with
> > > fingerprint 45C60AAAD115F560 [3],
> > > * all artifacts to be deployed to the Maven Central
> > Repository [4],
> > > * source code tag "v2.7.0-RC1" [5],
> > > * website pull request listing the release and publishing the
> API
> > > reference manual [6].
> > > * Java artifacts were built with Gradle 4.8 and OpenJDK
> > > 1.8.0_181-8u181-b13-1~deb9u1-b13.
> > > * Python artifacts are deployed along with the source release
> > to the
> > > dist.apache.org <http://dist.apache.org>
> > <http://dist.apache.org> [2].
> > >
> > > The vote will be open for at least 72 hours. It is adopted by
> > majority
> > > approval, with at least 3 PMC affirmative votes.
> > >
> > > Thanks,
> > > Charles
> > >
> > > [1]
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
> > > [2] https://dist.apache.org/repos/dist/dev/beam/2.7.0
> > > [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
> > > [4]
> >
> https://repository.apache.org/content/repositories/orgapachebeam-1046/
> > > [5] https://github.com/apache/beam/tree/v2.7.0-RC1
> > > [6] https://github.com/apache/beam-site/pull/549
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org <ma...@apache.org>
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi,
weird, I didn't have it on Beam samples. Let me try to reproduce and I
will create the Jira.
Regards
JB
On 11/09/2018 11:44, Romain Manni-Bucau wrote:
> -1, seems spark integration is broken (tested with spark 2.3.1 and 2.2.1):
>
> 18/09/11 11:33:29 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, RMANNIBUCAU, executor 0): java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to fieldorg.apache.spark.rdd.RDD.org <http://org.apache.spark.rdd.RDD.org>$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD
> at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>
>
> Also the issue Lukasz identified is important even if workarounds can be
> put in place so +1 to fix it as well if possible.
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github
> <https://github.com/rmannibucau> | LinkedIn
> <https://www.linkedin.com/in/rmannibucau> | Book
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>
>
> Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik <lcwik@google.com
> <ma...@google.com>> a écrit :
>
> I found an issue where we are no longer packaging the pom.xml within
> the artifact jars at META-INF/maven/groupId/artifactId. More details
> in https://issues.apache.org/jira/browse/BEAM-5351. I wouldn't
> consider this a blocker but it was an easy fix
> (https://github.com/apache/beam/pull/6358) and users may rely on the
> pom.xml.
>
> Should we recut the release candidate to include this?
>
> On Mon, Sep 10, 2018 at 4:58 AM Jean-Baptiste Onofré
> <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
>
> +1 (binding)
>
> Tested successfully on Beam Samples.
>
> Thanks !
>
> Regards
> JB
>
> On 07/09/2018 23:56, Charles Chen wrote:
> > Hi everyone,
> >
> > Please review and vote on the release candidate #1 for the
> version
> > 2.7.0, as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific
> comments)
> >
> > The complete staging area is available for your review, which
> includes:
> > * JIRA release notes [1],
> > * the official Apache source release to be deployed to
> dist.apache.org <http://dist.apache.org>
> > <http://dist.apache.org> [2], which is signed with the key with
> > fingerprint 45C60AAAD115F560 [3],
> > * all artifacts to be deployed to the Maven Central
> Repository [4],
> > * source code tag "v2.7.0-RC1" [5],
> > * website pull request listing the release and publishing the API
> > reference manual [6].
> > * Java artifacts were built with Gradle 4.8 and OpenJDK
> > 1.8.0_181-8u181-b13-1~deb9u1-b13.
> > * Python artifacts are deployed along with the source release
> to the
> > dist.apache.org <http://dist.apache.org>
> <http://dist.apache.org> [2].
> >
> > The vote will be open for at least 72 hours. It is adopted by
> majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Charles
> >
> > [1]
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
> > [2] https://dist.apache.org/repos/dist/dev/beam/2.7.0
> > [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
> > [4]
> https://repository.apache.org/content/repositories/orgapachebeam-1046/
> > [5] https://github.com/apache/beam/tree/v2.7.0-RC1
> > [6] https://github.com/apache/beam-site/pull/549
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org <ma...@apache.org>
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Romain Manni-Bucau <rm...@gmail.com>.
-1, seems spark integration is broken (tested with spark 2.3.1 and 2.2.1):
18/09/11 11:33:29 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID
0, RMANNIBUCAU, executor 0): java.lang.ClassCastException: cannot
assign instance of scala.collection.immutable.List$SerializationProxy
to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_
of type scala.collection.Seq in instance of
org.apache.spark.rdd.MapPartitionsRDD
at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
Also the issue Lukasz identified is important even if workarounds can be
put in place so +1 to fix it as well if possible.
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>
Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik <lc...@google.com> a écrit :
> I found an issue where we are no longer packaging the pom.xml within the
> artifact jars at META-INF/maven/groupId/artifactId. More details in
> https://issues.apache.org/jira/browse/BEAM-5351. I wouldn't consider this
> a blocker but it was an easy fix (https://github.com/apache/beam/pull/6358)
> and users may rely on the pom.xml.
>
> Should we recut the release candidate to include this?
>
> On Mon, Sep 10, 2018 at 4:58 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> +1 (binding)
>>
>> Tested successfully on Beam Samples.
>>
>> Thanks !
>>
>> Regards
>> JB
>>
>> On 07/09/2018 23:56, Charles Chen wrote:
>> > Hi everyone,
>> >
>> > Please review and vote on the release candidate #1 for the version
>> > 2.7.0, as follows:
>> > [ ] +1, Approve the release
>> > [ ] -1, Do not approve the release (please provide specific comments)
>> >
>> > The complete staging area is available for your review, which includes:
>> > * JIRA release notes [1],
>> > * the official Apache source release to be deployed to dist.apache.org
>> > <http://dist.apache.org> [2], which is signed with the key with
>> > fingerprint 45C60AAAD115F560 [3],
>> > * all artifacts to be deployed to the Maven Central Repository [4],
>> > * source code tag "v2.7.0-RC1" [5],
>> > * website pull request listing the release and publishing the API
>> > reference manual [6].
>> > * Java artifacts were built with Gradle 4.8 and OpenJDK
>> > 1.8.0_181-8u181-b13-1~deb9u1-b13.
>> > * Python artifacts are deployed along with the source release to the
>> > dist.apache.org <http://dist.apache.org> [2].
>> >
>> > The vote will be open for at least 72 hours. It is adopted by majority
>> > approval, with at least 3 PMC affirmative votes.
>> >
>> > Thanks,
>> > Charles
>> >
>> > [1]
>> >
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
>> > [2] https://dist.apache.org/repos/dist/dev/beam/2.7.0
>> > [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
>> > [4]
>> https://repository.apache.org/content/repositories/orgapachebeam-1046/
>> > [5] https://github.com/apache/beam/tree/v2.7.0-RC1
>> > [6] https://github.com/apache/beam-site/pull/549
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Lukasz Cwik <lc...@google.com>.
I found an issue where we are no longer packaging the pom.xml within the
artifact jars at META-INF/maven/groupId/artifactId. More details in
https://issues.apache.org/jira/browse/BEAM-5351. I wouldn't consider this a
blocker but it was an easy fix (https://github.com/apache/beam/pull/6358)
and users may rely on the pom.xml.
Should we recut the release candidate to include this?
On Mon, Sep 10, 2018 at 4:58 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> +1 (binding)
>
> Tested successfully on Beam Samples.
>
> Thanks !
>
> Regards
> JB
>
> On 07/09/2018 23:56, Charles Chen wrote:
> > Hi everyone,
> >
> > Please review and vote on the release candidate #1 for the version
> > 2.7.0, as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > The complete staging area is available for your review, which includes:
> > * JIRA release notes [1],
> > * the official Apache source release to be deployed to dist.apache.org
> > <http://dist.apache.org> [2], which is signed with the key with
> > fingerprint 45C60AAAD115F560 [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tag "v2.7.0-RC1" [5],
> > * website pull request listing the release and publishing the API
> > reference manual [6].
> > * Java artifacts were built with Gradle 4.8 and OpenJDK
> > 1.8.0_181-8u181-b13-1~deb9u1-b13.
> > * Python artifacts are deployed along with the source release to the
> > dist.apache.org <http://dist.apache.org> [2].
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Charles
> >
> > [1]
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
> > [2] https://dist.apache.org/repos/dist/dev/beam/2.7.0
> > [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
> > [4]
> https://repository.apache.org/content/repositories/orgapachebeam-1046/
> > [5] https://github.com/apache/beam/tree/v2.7.0-RC1
> > [6] https://github.com/apache/beam-site/pull/549
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: [VOTE] Release 2.7.0, release candidate #1
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
+1 (binding)
Tested successfully on Beam Samples.
Thanks !
Regards
JB
On 07/09/2018 23:56, Charles Chen wrote:
> Hi everyone,
>
> Please review and vote on the release candidate #1 for the version
> 2.7.0, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org
> <http://dist.apache.org> [2], which is signed with the key with
> fingerprint 45C60AAAD115F560 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.7.0-RC1" [5],
> * website pull request listing the release and publishing the API
> reference manual [6].
> * Java artifacts were built with Gradle 4.8 and OpenJDK
> 1.8.0_181-8u181-b13-1~deb9u1-b13.
> * Python artifacts are deployed along with the source release to the
> dist.apache.org <http://dist.apache.org> [2].
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Charles
>
> [1]
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
> [2] https://dist.apache.org/repos/dist/dev/beam/2.7.0
> [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1046/
> [5] https://github.com/apache/beam/tree/v2.7.0-RC1
> [6] https://github.com/apache/beam-site/pull/549
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com