You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Stanislav Los <st...@magnetic.com> on 2016/01/06 16:52:13 UTC
hello-samza doesn't run on CDH5.4
Hi,
Any ideas why Samza can't start a container while running hello-samza on
CDH5.4?
It runs fine on local grid. But when running on cluster I can see it talks
fine to YARN's ResourceManager and Kafka, get's resources, but fails to
start container. At first, I thought it's because CDH5.4 uses YARN 2.6.0,
while Samza uses 2.6.1, so I recompiled Samza with YARN 2.6.0 and
re-assambled hello-samza. No luck so far. Also, I thought it's because of
different protobuf version, but I checked, it's the same in CDH and Samza.
Followed
http://samza.apache.org/learn/tutorials/0.10/deploy-samza-to-CDH.html to
deploy.
See full logs:
Samza Deploy Script output http://pastebin.com/0K8HVMzt
YARN logs http://pastebin.com/UvNp6240
An error is:
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
com.google.protobuf.InvalidProtocolBufferException: Protocol message
contained an invalid tag (zero).
at
com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:89)
at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:108)
at
org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.<init>(YarnSecurityTokenProtos.java:1078)
at
org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.<init>(YarnSecurityTokenProtos.java:1042)
at
org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto$1.parsePartialFrom(YarnSecurityTokenProtos.java:1138)
at
org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto$1.parsePartialFrom(YarnSecurityTokenProtos.java:1133)
at
com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
at
org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.parseFrom(YarnSecurityTokenProtos.java:1307)
at
org.apache.hadoop.yarn.security.AMRMTokenIdentifier.readFields(AMRMTokenIdentifier.java:81)
at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:142)
at
org.apache.samza.job.yarn.ContainerUtil.startContainer(ContainerUtil.java:181)
at
org.apache.samza.job.yarn.ContainerUtil.runContainer(ContainerUtil.java:119)
at
org.apache.samza.job.yarn.ContainerAllocator.run(ContainerAllocator.java:65)
at java.lang.Thread.run(Thread.java:745)
Exception in thread "Container Allocator Thread"
org.apache.samza.SamzaException: IO Exception when writing credentials to
output buffer
at
org.apache.samza.job.yarn.ContainerUtil.startContainer(ContainerUtil.java:190)
at
org.apache.samza.job.yarn.ContainerUtil.runContainer(ContainerUtil.java:119)
at
org.apache.samza.job.yarn.ContainerAllocator.run(ContainerAllocator.java:65)
at java.lang.Thread.run(Thread.java:745)
Regards,
Stan
Re: hello-samza doesn't run on CDH5.4
Posted by Stanislav Los <st...@magnetic.com>.
https://issues.apache.org/jira/browse/SAMZA-851 and pull request that adds
optional Maven profile
On Wed, Jan 6, 2016 at 5:28 PM, Yi Pan <ni...@gmail.com> wrote:
> Glad to hear that it has worked. Yeah, I guess right now, if there is any
> dependency conflicts, your solution would be the way to resolve it.
>
> Could you open a JIRA to enhance the hello-samza tutorial docs to address
> this issue?
>
> Thanks!
>
> -Yi
>
> On Wed, Jan 6, 2016 at 12:47 PM, Stanislav Los <st...@magnetic.com>
> wrote:
>
> > So, for now, I made it work by replacing any generic hadoop-* jar in
> > hello-samza distribution with related jar from CDH:
> >
> > [stanislav@hcm001.us-east-1.mgnt.cc samza]$ ls -l deploy/samza-cdh/lib/
> |
> > grep hadoop
> >
> > -rwxr-xr-x 1 stanislav stanislav 21533 Jan 6 15:19
> > hadoop-annotations-2.6.0-cdh5.4.0.jar
> >
> > -rwxr-xr-x 1 stanislav stanislav 73113 Jan 6 15:22
> > hadoop-auth-2.6.0-cdh5.4.0.jar
> >
> > -rwxr-xr-x 1 stanislav stanislav 3339429 Jan 6 15:22
> > hadoop-common-2.6.0-cdh5.4.0.jar
> >
> > -rw-r--r-- 1 stanislav stanislav 7970062 Jan 6 15:22
> > hadoop-hdfs-2.6.0-cdh5.4.0.jar
> >
> > -rw-r--r-- 1 stanislav stanislav 1875075 Jan 6 15:23
> > hadoop-yarn-api-2.6.0-cdh5.4.0.jar
> >
> > -rw-r--r-- 1 stanislav stanislav 119623 Jan 6 15:23
> > hadoop-yarn-client-2.6.0-cdh5.4.0.jar
> >
> > -rw-r--r-- 1 stanislav stanislav 1540461 Jan 6 15:23
> > hadoop-yarn-common-2.6.0-cdh5.4.0.jar
> >
> > I guess, I can update hello-samza pom.xml by explicitly excluding generic
> > hadoop dependencies and replacing them with related stuff from CDH repo
> to
> > achieve the same on a build level.
> >
> > On Wed, Jan 6, 2016 at 3:18 PM, Yi Pan <ni...@gmail.com> wrote:
> >
> > > Token expiration is still a problem w/o Kerberos in YARN 2.6.0 client.
> > But
> > > the exception stack trace from that bug is also different from yours. I
> > am
> > > suspecting there is some incompatible dependency libraries between
> CDH5.4
> > > distro and Samza that caused your serialization problems here.
> > >
> > > On Wed, Jan 6, 2016 at 11:50 AM, Stanislav Los <stanislav@magnetic.com
> >
> > > wrote:
> > >
> > > > Hi Yi,
> > > >
> > > > Our cluster doesn't have Kerberos security in place, so I don't know
> > how
> > > it
> > > > could be related.
> > > > CDH5.4 runs on YARN 2.6.0.
> > > > I compiled Samza with YARN 2.6.0 and YARN 2.6.1.
> > > > In both cases I get the same error.
> > > > Should token expiration be a problem for the cluster without
> Kerberos?
> > > >
> > > > Regards.
> > > >
> > > > On Wed, Jan 6, 2016 at 2:39 PM, Yi Pan <ni...@gmail.com> wrote:
> > > >
> > > > > Hi, Stanislav,
> > > > >
> > > > > Could it be related to SAMZA-727? Your exception stack trace is
> > > different
> > > > > though. Besides that, Samza 0.10 requires minimum YARN 2.6.1 now,
> due
> > > to
> > > > a
> > > > > token expiration issue in YARN 2.6.0 client lib.
> > > > >
> > > > > -Yi
> > > > >
> > > > > On Wed, Jan 6, 2016 at 7:52 AM, Stanislav Los <
> > stanislav@magnetic.com>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Any ideas why Samza can't start a container while running
> > hello-samza
> > > > on
> > > > > > CDH5.4?
> > > > > >
> > > > > > It runs fine on local grid. But when running on cluster I can see
> > it
> > > > > talks
> > > > > > fine to YARN's ResourceManager and Kafka, get's resources, but
> > fails
> > > > to
> > > > > > start container. At first, I thought it's because CDH5.4 uses
> YARN
> > > > 2.6.0,
> > > > > > while Samza uses 2.6.1, so I recompiled Samza with YARN 2.6.0 and
> > > > > > re-assambled hello-samza. No luck so far. Also, I thought it's
> > > because
> > > > of
> > > > > > different protobuf version, but I checked, it's the same in CDH
> and
> > > > > Samza.
> > > > > >
> > > > > > Followed
> > > > > >
> > > http://samza.apache.org/learn/tutorials/0.10/deploy-samza-to-CDH.html
> > > > to
> > > > > > deploy.
> > > > > >
> > > > > > See full logs:
> > > > > > Samza Deploy Script output http://pastebin.com/0K8HVMzt
> > > > > > YARN logs http://pastebin.com/UvNp6240
> > > > > >
> > > > > > An error is:
> > > > > >
> > > > > > java version "1.7.0_80"
> > > > > >
> > > > > > Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
> > > > > >
> > > > > > Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
> > > > > >
> > > > > > com.google.protobuf.InvalidProtocolBufferException: Protocol
> > message
> > > > > > contained an invalid tag (zero).
> > > > > >
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:89)
> > > > > >
> > > > > > at
> > > > >
> > com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:108)
> > > > > >
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.<init>(YarnSecurityTokenProtos.java:1078)
> > > > > >
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.<init>(YarnSecurityTokenProtos.java:1042)
> > > > > >
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto$1.parsePartialFrom(YarnSecurityTokenProtos.java:1138)
> > > > > >
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto$1.parsePartialFrom(YarnSecurityTokenProtos.java:1133)
> > > > > >
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
> > > > > >
> > > > > > at
> > > > com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
> > > > > >
> > > > > > at
> > > > com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
> > > > > >
> > > > > > at
> > > com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
> > > > > >
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.parseFrom(YarnSecurityTokenProtos.java:1307)
> > > > > >
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.yarn.security.AMRMTokenIdentifier.readFields(AMRMTokenIdentifier.java:81)
> > > > > >
> > > > > > at
> > > > >
> > org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:142)
> > > > > >
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.samza.job.yarn.ContainerUtil.startContainer(ContainerUtil.java:181)
> > > > > >
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.samza.job.yarn.ContainerUtil.runContainer(ContainerUtil.java:119)
> > > > > >
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.samza.job.yarn.ContainerAllocator.run(ContainerAllocator.java:65)
> > > > > >
> > > > > > at java.lang.Thread.run(Thread.java:745)
> > > > > >
> > > > > > Exception in thread "Container Allocator Thread"
> > > > > > org.apache.samza.SamzaException: IO Exception when writing
> > > credentials
> > > > to
> > > > > > output buffer
> > > > > >
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.samza.job.yarn.ContainerUtil.startContainer(ContainerUtil.java:190)
> > > > > >
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.samza.job.yarn.ContainerUtil.runContainer(ContainerUtil.java:119)
> > > > > >
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.samza.job.yarn.ContainerAllocator.run(ContainerAllocator.java:65)
> > > > > >
> > > > > > at java.lang.Thread.run(Thread.java:745)
> > > > > >
> > > > > >
> > > > > > Regards,
> > > > > > Stan
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: hello-samza doesn't run on CDH5.4
Posted by Yi Pan <ni...@gmail.com>.
Glad to hear that it has worked. Yeah, I guess right now, if there is any
dependency conflicts, your solution would be the way to resolve it.
Could you open a JIRA to enhance the hello-samza tutorial docs to address
this issue?
Thanks!
-Yi
On Wed, Jan 6, 2016 at 12:47 PM, Stanislav Los <st...@magnetic.com>
wrote:
> So, for now, I made it work by replacing any generic hadoop-* jar in
> hello-samza distribution with related jar from CDH:
>
> [stanislav@hcm001.us-east-1.mgnt.cc samza]$ ls -l deploy/samza-cdh/lib/ |
> grep hadoop
>
> -rwxr-xr-x 1 stanislav stanislav 21533 Jan 6 15:19
> hadoop-annotations-2.6.0-cdh5.4.0.jar
>
> -rwxr-xr-x 1 stanislav stanislav 73113 Jan 6 15:22
> hadoop-auth-2.6.0-cdh5.4.0.jar
>
> -rwxr-xr-x 1 stanislav stanislav 3339429 Jan 6 15:22
> hadoop-common-2.6.0-cdh5.4.0.jar
>
> -rw-r--r-- 1 stanislav stanislav 7970062 Jan 6 15:22
> hadoop-hdfs-2.6.0-cdh5.4.0.jar
>
> -rw-r--r-- 1 stanislav stanislav 1875075 Jan 6 15:23
> hadoop-yarn-api-2.6.0-cdh5.4.0.jar
>
> -rw-r--r-- 1 stanislav stanislav 119623 Jan 6 15:23
> hadoop-yarn-client-2.6.0-cdh5.4.0.jar
>
> -rw-r--r-- 1 stanislav stanislav 1540461 Jan 6 15:23
> hadoop-yarn-common-2.6.0-cdh5.4.0.jar
>
> I guess, I can update hello-samza pom.xml by explicitly excluding generic
> hadoop dependencies and replacing them with related stuff from CDH repo to
> achieve the same on a build level.
>
> On Wed, Jan 6, 2016 at 3:18 PM, Yi Pan <ni...@gmail.com> wrote:
>
> > Token expiration is still a problem w/o Kerberos in YARN 2.6.0 client.
> But
> > the exception stack trace from that bug is also different from yours. I
> am
> > suspecting there is some incompatible dependency libraries between CDH5.4
> > distro and Samza that caused your serialization problems here.
> >
> > On Wed, Jan 6, 2016 at 11:50 AM, Stanislav Los <st...@magnetic.com>
> > wrote:
> >
> > > Hi Yi,
> > >
> > > Our cluster doesn't have Kerberos security in place, so I don't know
> how
> > it
> > > could be related.
> > > CDH5.4 runs on YARN 2.6.0.
> > > I compiled Samza with YARN 2.6.0 and YARN 2.6.1.
> > > In both cases I get the same error.
> > > Should token expiration be a problem for the cluster without Kerberos?
> > >
> > > Regards.
> > >
> > > On Wed, Jan 6, 2016 at 2:39 PM, Yi Pan <ni...@gmail.com> wrote:
> > >
> > > > Hi, Stanislav,
> > > >
> > > > Could it be related to SAMZA-727? Your exception stack trace is
> > different
> > > > though. Besides that, Samza 0.10 requires minimum YARN 2.6.1 now, due
> > to
> > > a
> > > > token expiration issue in YARN 2.6.0 client lib.
> > > >
> > > > -Yi
> > > >
> > > > On Wed, Jan 6, 2016 at 7:52 AM, Stanislav Los <
> stanislav@magnetic.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Any ideas why Samza can't start a container while running
> hello-samza
> > > on
> > > > > CDH5.4?
> > > > >
> > > > > It runs fine on local grid. But when running on cluster I can see
> it
> > > > talks
> > > > > fine to YARN's ResourceManager and Kafka, get's resources, but
> fails
> > > to
> > > > > start container. At first, I thought it's because CDH5.4 uses YARN
> > > 2.6.0,
> > > > > while Samza uses 2.6.1, so I recompiled Samza with YARN 2.6.0 and
> > > > > re-assambled hello-samza. No luck so far. Also, I thought it's
> > because
> > > of
> > > > > different protobuf version, but I checked, it's the same in CDH and
> > > > Samza.
> > > > >
> > > > > Followed
> > > > >
> > http://samza.apache.org/learn/tutorials/0.10/deploy-samza-to-CDH.html
> > > to
> > > > > deploy.
> > > > >
> > > > > See full logs:
> > > > > Samza Deploy Script output http://pastebin.com/0K8HVMzt
> > > > > YARN logs http://pastebin.com/UvNp6240
> > > > >
> > > > > An error is:
> > > > >
> > > > > java version "1.7.0_80"
> > > > >
> > > > > Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
> > > > >
> > > > > Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
> > > > >
> > > > > com.google.protobuf.InvalidProtocolBufferException: Protocol
> message
> > > > > contained an invalid tag (zero).
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:89)
> > > > >
> > > > > at
> > > >
> com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:108)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.<init>(YarnSecurityTokenProtos.java:1078)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.<init>(YarnSecurityTokenProtos.java:1042)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto$1.parsePartialFrom(YarnSecurityTokenProtos.java:1138)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto$1.parsePartialFrom(YarnSecurityTokenProtos.java:1133)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
> > > > >
> > > > > at
> > > com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
> > > > >
> > > > > at
> > > com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
> > > > >
> > > > > at
> > com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.parseFrom(YarnSecurityTokenProtos.java:1307)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.yarn.security.AMRMTokenIdentifier.readFields(AMRMTokenIdentifier.java:81)
> > > > >
> > > > > at
> > > >
> org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:142)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.samza.job.yarn.ContainerUtil.startContainer(ContainerUtil.java:181)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.samza.job.yarn.ContainerUtil.runContainer(ContainerUtil.java:119)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.samza.job.yarn.ContainerAllocator.run(ContainerAllocator.java:65)
> > > > >
> > > > > at java.lang.Thread.run(Thread.java:745)
> > > > >
> > > > > Exception in thread "Container Allocator Thread"
> > > > > org.apache.samza.SamzaException: IO Exception when writing
> > credentials
> > > to
> > > > > output buffer
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.samza.job.yarn.ContainerUtil.startContainer(ContainerUtil.java:190)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.samza.job.yarn.ContainerUtil.runContainer(ContainerUtil.java:119)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.samza.job.yarn.ContainerAllocator.run(ContainerAllocator.java:65)
> > > > >
> > > > > at java.lang.Thread.run(Thread.java:745)
> > > > >
> > > > >
> > > > > Regards,
> > > > > Stan
> > > > >
> > > >
> > >
> >
>
Re: hello-samza doesn't run on CDH5.4
Posted by Stanislav Los <st...@magnetic.com>.
So, for now, I made it work by replacing any generic hadoop-* jar in
hello-samza distribution with related jar from CDH:
[stanislav@hcm001.us-east-1.mgnt.cc samza]$ ls -l deploy/samza-cdh/lib/ |
grep hadoop
-rwxr-xr-x 1 stanislav stanislav 21533 Jan 6 15:19
hadoop-annotations-2.6.0-cdh5.4.0.jar
-rwxr-xr-x 1 stanislav stanislav 73113 Jan 6 15:22
hadoop-auth-2.6.0-cdh5.4.0.jar
-rwxr-xr-x 1 stanislav stanislav 3339429 Jan 6 15:22
hadoop-common-2.6.0-cdh5.4.0.jar
-rw-r--r-- 1 stanislav stanislav 7970062 Jan 6 15:22
hadoop-hdfs-2.6.0-cdh5.4.0.jar
-rw-r--r-- 1 stanislav stanislav 1875075 Jan 6 15:23
hadoop-yarn-api-2.6.0-cdh5.4.0.jar
-rw-r--r-- 1 stanislav stanislav 119623 Jan 6 15:23
hadoop-yarn-client-2.6.0-cdh5.4.0.jar
-rw-r--r-- 1 stanislav stanislav 1540461 Jan 6 15:23
hadoop-yarn-common-2.6.0-cdh5.4.0.jar
I guess, I can update hello-samza pom.xml by explicitly excluding generic
hadoop dependencies and replacing them with related stuff from CDH repo to
achieve the same on a build level.
On Wed, Jan 6, 2016 at 3:18 PM, Yi Pan <ni...@gmail.com> wrote:
> Token expiration is still a problem w/o Kerberos in YARN 2.6.0 client. But
> the exception stack trace from that bug is also different from yours. I am
> suspecting there is some incompatible dependency libraries between CDH5.4
> distro and Samza that caused your serialization problems here.
>
> On Wed, Jan 6, 2016 at 11:50 AM, Stanislav Los <st...@magnetic.com>
> wrote:
>
> > Hi Yi,
> >
> > Our cluster doesn't have Kerberos security in place, so I don't know how
> it
> > could be related.
> > CDH5.4 runs on YARN 2.6.0.
> > I compiled Samza with YARN 2.6.0 and YARN 2.6.1.
> > In both cases I get the same error.
> > Should token expiration be a problem for the cluster without Kerberos?
> >
> > Regards.
> >
> > On Wed, Jan 6, 2016 at 2:39 PM, Yi Pan <ni...@gmail.com> wrote:
> >
> > > Hi, Stanislav,
> > >
> > > Could it be related to SAMZA-727? Your exception stack trace is
> different
> > > though. Besides that, Samza 0.10 requires minimum YARN 2.6.1 now, due
> to
> > a
> > > token expiration issue in YARN 2.6.0 client lib.
> > >
> > > -Yi
> > >
> > > On Wed, Jan 6, 2016 at 7:52 AM, Stanislav Los <st...@magnetic.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Any ideas why Samza can't start a container while running hello-samza
> > on
> > > > CDH5.4?
> > > >
> > > > It runs fine on local grid. But when running on cluster I can see it
> > > talks
> > > > fine to YARN's ResourceManager and Kafka, get's resources, but fails
> > to
> > > > start container. At first, I thought it's because CDH5.4 uses YARN
> > 2.6.0,
> > > > while Samza uses 2.6.1, so I recompiled Samza with YARN 2.6.0 and
> > > > re-assambled hello-samza. No luck so far. Also, I thought it's
> because
> > of
> > > > different protobuf version, but I checked, it's the same in CDH and
> > > Samza.
> > > >
> > > > Followed
> > > >
> http://samza.apache.org/learn/tutorials/0.10/deploy-samza-to-CDH.html
> > to
> > > > deploy.
> > > >
> > > > See full logs:
> > > > Samza Deploy Script output http://pastebin.com/0K8HVMzt
> > > > YARN logs http://pastebin.com/UvNp6240
> > > >
> > > > An error is:
> > > >
> > > > java version "1.7.0_80"
> > > >
> > > > Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
> > > >
> > > > Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
> > > >
> > > > com.google.protobuf.InvalidProtocolBufferException: Protocol message
> > > > contained an invalid tag (zero).
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:89)
> > > >
> > > > at
> > > com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:108)
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.<init>(YarnSecurityTokenProtos.java:1078)
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.<init>(YarnSecurityTokenProtos.java:1042)
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto$1.parsePartialFrom(YarnSecurityTokenProtos.java:1138)
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto$1.parsePartialFrom(YarnSecurityTokenProtos.java:1133)
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
> > > >
> > > > at
> > com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
> > > >
> > > > at
> > com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
> > > >
> > > > at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.parseFrom(YarnSecurityTokenProtos.java:1307)
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.yarn.security.AMRMTokenIdentifier.readFields(AMRMTokenIdentifier.java:81)
> > > >
> > > > at
> > > org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:142)
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.samza.job.yarn.ContainerUtil.startContainer(ContainerUtil.java:181)
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.samza.job.yarn.ContainerUtil.runContainer(ContainerUtil.java:119)
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.samza.job.yarn.ContainerAllocator.run(ContainerAllocator.java:65)
> > > >
> > > > at java.lang.Thread.run(Thread.java:745)
> > > >
> > > > Exception in thread "Container Allocator Thread"
> > > > org.apache.samza.SamzaException: IO Exception when writing
> credentials
> > to
> > > > output buffer
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.samza.job.yarn.ContainerUtil.startContainer(ContainerUtil.java:190)
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.samza.job.yarn.ContainerUtil.runContainer(ContainerUtil.java:119)
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.samza.job.yarn.ContainerAllocator.run(ContainerAllocator.java:65)
> > > >
> > > > at java.lang.Thread.run(Thread.java:745)
> > > >
> > > >
> > > > Regards,
> > > > Stan
> > > >
> > >
> >
>
Re: hello-samza doesn't run on CDH5.4
Posted by Yi Pan <ni...@gmail.com>.
Token expiration is still a problem w/o Kerberos in YARN 2.6.0 client. But
the exception stack trace from that bug is also different from yours. I am
suspecting there is some incompatible dependency libraries between CDH5.4
distro and Samza that caused your serialization problems here.
On Wed, Jan 6, 2016 at 11:50 AM, Stanislav Los <st...@magnetic.com>
wrote:
> Hi Yi,
>
> Our cluster doesn't have Kerberos security in place, so I don't know how it
> could be related.
> CDH5.4 runs on YARN 2.6.0.
> I compiled Samza with YARN 2.6.0 and YARN 2.6.1.
> In both cases I get the same error.
> Should token expiration be a problem for the cluster without Kerberos?
>
> Regards.
>
> On Wed, Jan 6, 2016 at 2:39 PM, Yi Pan <ni...@gmail.com> wrote:
>
> > Hi, Stanislav,
> >
> > Could it be related to SAMZA-727? Your exception stack trace is different
> > though. Besides that, Samza 0.10 requires minimum YARN 2.6.1 now, due to
> a
> > token expiration issue in YARN 2.6.0 client lib.
> >
> > -Yi
> >
> > On Wed, Jan 6, 2016 at 7:52 AM, Stanislav Los <st...@magnetic.com>
> > wrote:
> >
> > > Hi,
> > >
> > > Any ideas why Samza can't start a container while running hello-samza
> on
> > > CDH5.4?
> > >
> > > It runs fine on local grid. But when running on cluster I can see it
> > talks
> > > fine to YARN's ResourceManager and Kafka, get's resources, but fails
> to
> > > start container. At first, I thought it's because CDH5.4 uses YARN
> 2.6.0,
> > > while Samza uses 2.6.1, so I recompiled Samza with YARN 2.6.0 and
> > > re-assambled hello-samza. No luck so far. Also, I thought it's because
> of
> > > different protobuf version, but I checked, it's the same in CDH and
> > Samza.
> > >
> > > Followed
> > > http://samza.apache.org/learn/tutorials/0.10/deploy-samza-to-CDH.html
> to
> > > deploy.
> > >
> > > See full logs:
> > > Samza Deploy Script output http://pastebin.com/0K8HVMzt
> > > YARN logs http://pastebin.com/UvNp6240
> > >
> > > An error is:
> > >
> > > java version "1.7.0_80"
> > >
> > > Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
> > >
> > > Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
> > >
> > > com.google.protobuf.InvalidProtocolBufferException: Protocol message
> > > contained an invalid tag (zero).
> > >
> > > at
> > >
> > >
> >
> com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:89)
> > >
> > > at
> > com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:108)
> > >
> > > at
> > >
> > >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.<init>(YarnSecurityTokenProtos.java:1078)
> > >
> > > at
> > >
> > >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.<init>(YarnSecurityTokenProtos.java:1042)
> > >
> > > at
> > >
> > >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto$1.parsePartialFrom(YarnSecurityTokenProtos.java:1138)
> > >
> > > at
> > >
> > >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto$1.parsePartialFrom(YarnSecurityTokenProtos.java:1133)
> > >
> > > at
> > >
> > >
> >
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
> > >
> > > at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
> > >
> > > at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
> > >
> > > at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
> > >
> > > at
> > >
> > >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.parseFrom(YarnSecurityTokenProtos.java:1307)
> > >
> > > at
> > >
> > >
> >
> org.apache.hadoop.yarn.security.AMRMTokenIdentifier.readFields(AMRMTokenIdentifier.java:81)
> > >
> > > at
> > org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:142)
> > >
> > > at
> > >
> > >
> >
> org.apache.samza.job.yarn.ContainerUtil.startContainer(ContainerUtil.java:181)
> > >
> > > at
> > >
> > >
> >
> org.apache.samza.job.yarn.ContainerUtil.runContainer(ContainerUtil.java:119)
> > >
> > > at
> > >
> > >
> >
> org.apache.samza.job.yarn.ContainerAllocator.run(ContainerAllocator.java:65)
> > >
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> > > Exception in thread "Container Allocator Thread"
> > > org.apache.samza.SamzaException: IO Exception when writing credentials
> to
> > > output buffer
> > >
> > > at
> > >
> > >
> >
> org.apache.samza.job.yarn.ContainerUtil.startContainer(ContainerUtil.java:190)
> > >
> > > at
> > >
> > >
> >
> org.apache.samza.job.yarn.ContainerUtil.runContainer(ContainerUtil.java:119)
> > >
> > > at
> > >
> > >
> >
> org.apache.samza.job.yarn.ContainerAllocator.run(ContainerAllocator.java:65)
> > >
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> > >
> > > Regards,
> > > Stan
> > >
> >
>
Re: hello-samza doesn't run on CDH5.4
Posted by Stanislav Los <st...@magnetic.com>.
Hi Yi,
Our cluster doesn't have Kerberos security in place, so I don't know how it
could be related.
CDH5.4 runs on YARN 2.6.0.
I compiled Samza with YARN 2.6.0 and YARN 2.6.1.
In both cases I get the same error.
Should token expiration be a problem for the cluster without Kerberos?
Regards.
On Wed, Jan 6, 2016 at 2:39 PM, Yi Pan <ni...@gmail.com> wrote:
> Hi, Stanislav,
>
> Could it be related to SAMZA-727? Your exception stack trace is different
> though. Besides that, Samza 0.10 requires minimum YARN 2.6.1 now, due to a
> token expiration issue in YARN 2.6.0 client lib.
>
> -Yi
>
> On Wed, Jan 6, 2016 at 7:52 AM, Stanislav Los <st...@magnetic.com>
> wrote:
>
> > Hi,
> >
> > Any ideas why Samza can't start a container while running hello-samza on
> > CDH5.4?
> >
> > It runs fine on local grid. But when running on cluster I can see it
> talks
> > fine to YARN's ResourceManager and Kafka, get's resources, but fails to
> > start container. At first, I thought it's because CDH5.4 uses YARN 2.6.0,
> > while Samza uses 2.6.1, so I recompiled Samza with YARN 2.6.0 and
> > re-assambled hello-samza. No luck so far. Also, I thought it's because of
> > different protobuf version, but I checked, it's the same in CDH and
> Samza.
> >
> > Followed
> > http://samza.apache.org/learn/tutorials/0.10/deploy-samza-to-CDH.html to
> > deploy.
> >
> > See full logs:
> > Samza Deploy Script output http://pastebin.com/0K8HVMzt
> > YARN logs http://pastebin.com/UvNp6240
> >
> > An error is:
> >
> > java version "1.7.0_80"
> >
> > Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
> >
> > Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
> >
> > com.google.protobuf.InvalidProtocolBufferException: Protocol message
> > contained an invalid tag (zero).
> >
> > at
> >
> >
> com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:89)
> >
> > at
> com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:108)
> >
> > at
> >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.<init>(YarnSecurityTokenProtos.java:1078)
> >
> > at
> >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.<init>(YarnSecurityTokenProtos.java:1042)
> >
> > at
> >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto$1.parsePartialFrom(YarnSecurityTokenProtos.java:1138)
> >
> > at
> >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto$1.parsePartialFrom(YarnSecurityTokenProtos.java:1133)
> >
> > at
> >
> >
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
> >
> > at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
> >
> > at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
> >
> > at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
> >
> > at
> >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.parseFrom(YarnSecurityTokenProtos.java:1307)
> >
> > at
> >
> >
> org.apache.hadoop.yarn.security.AMRMTokenIdentifier.readFields(AMRMTokenIdentifier.java:81)
> >
> > at
> org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:142)
> >
> > at
> >
> >
> org.apache.samza.job.yarn.ContainerUtil.startContainer(ContainerUtil.java:181)
> >
> > at
> >
> >
> org.apache.samza.job.yarn.ContainerUtil.runContainer(ContainerUtil.java:119)
> >
> > at
> >
> >
> org.apache.samza.job.yarn.ContainerAllocator.run(ContainerAllocator.java:65)
> >
> > at java.lang.Thread.run(Thread.java:745)
> >
> > Exception in thread "Container Allocator Thread"
> > org.apache.samza.SamzaException: IO Exception when writing credentials to
> > output buffer
> >
> > at
> >
> >
> org.apache.samza.job.yarn.ContainerUtil.startContainer(ContainerUtil.java:190)
> >
> > at
> >
> >
> org.apache.samza.job.yarn.ContainerUtil.runContainer(ContainerUtil.java:119)
> >
> > at
> >
> >
> org.apache.samza.job.yarn.ContainerAllocator.run(ContainerAllocator.java:65)
> >
> > at java.lang.Thread.run(Thread.java:745)
> >
> >
> > Regards,
> > Stan
> >
>
Re: hello-samza doesn't run on CDH5.4
Posted by Yi Pan <ni...@gmail.com>.
Hi, Stanislav,
Could it be related to SAMZA-727? Your exception stack trace is different
though. Besides that, Samza 0.10 requires minimum YARN 2.6.1 now, due to a
token expiration issue in YARN 2.6.0 client lib.
-Yi
On Wed, Jan 6, 2016 at 7:52 AM, Stanislav Los <st...@magnetic.com>
wrote:
> Hi,
>
> Any ideas why Samza can't start a container while running hello-samza on
> CDH5.4?
>
> It runs fine on local grid. But when running on cluster I can see it talks
> fine to YARN's ResourceManager and Kafka, get's resources, but fails to
> start container. At first, I thought it's because CDH5.4 uses YARN 2.6.0,
> while Samza uses 2.6.1, so I recompiled Samza with YARN 2.6.0 and
> re-assambled hello-samza. No luck so far. Also, I thought it's because of
> different protobuf version, but I checked, it's the same in CDH and Samza.
>
> Followed
> http://samza.apache.org/learn/tutorials/0.10/deploy-samza-to-CDH.html to
> deploy.
>
> See full logs:
> Samza Deploy Script output http://pastebin.com/0K8HVMzt
> YARN logs http://pastebin.com/UvNp6240
>
> An error is:
>
> java version "1.7.0_80"
>
> Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
>
> Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
>
> com.google.protobuf.InvalidProtocolBufferException: Protocol message
> contained an invalid tag (zero).
>
> at
>
> com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:89)
>
> at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:108)
>
> at
>
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.<init>(YarnSecurityTokenProtos.java:1078)
>
> at
>
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.<init>(YarnSecurityTokenProtos.java:1042)
>
> at
>
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto$1.parsePartialFrom(YarnSecurityTokenProtos.java:1138)
>
> at
>
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto$1.parsePartialFrom(YarnSecurityTokenProtos.java:1133)
>
> at
>
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
>
> at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
>
> at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
>
> at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
>
> at
>
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.parseFrom(YarnSecurityTokenProtos.java:1307)
>
> at
>
> org.apache.hadoop.yarn.security.AMRMTokenIdentifier.readFields(AMRMTokenIdentifier.java:81)
>
> at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:142)
>
> at
>
> org.apache.samza.job.yarn.ContainerUtil.startContainer(ContainerUtil.java:181)
>
> at
>
> org.apache.samza.job.yarn.ContainerUtil.runContainer(ContainerUtil.java:119)
>
> at
>
> org.apache.samza.job.yarn.ContainerAllocator.run(ContainerAllocator.java:65)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Exception in thread "Container Allocator Thread"
> org.apache.samza.SamzaException: IO Exception when writing credentials to
> output buffer
>
> at
>
> org.apache.samza.job.yarn.ContainerUtil.startContainer(ContainerUtil.java:190)
>
> at
>
> org.apache.samza.job.yarn.ContainerUtil.runContainer(ContainerUtil.java:119)
>
> at
>
> org.apache.samza.job.yarn.ContainerAllocator.run(ContainerAllocator.java:65)
>
> at java.lang.Thread.run(Thread.java:745)
>
>
> Regards,
> Stan
>