You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Kenneth Knowles (JIRA)" <ji...@apache.org> on 2017/04/28 19:38:04 UTC
[jira] [Resolved] (BEAM-1981) Serialization error with
TimerInternals in ApexGroupByKeyOperator
[ https://issues.apache.org/jira/browse/BEAM-1981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kenneth Knowles resolved BEAM-1981.
-----------------------------------
Resolution: Fixed
> Serialization error with TimerInternals in ApexGroupByKeyOperator
> -----------------------------------------------------------------
>
> Key: BEAM-1981
> URL: https://issues.apache.org/jira/browse/BEAM-1981
> Project: Beam
> Issue Type: Bug
> Components: runner-apex
> Reporter: Daniel Halperin
> Assignee: Thomas Weise
> Fix For: First stable release
>
>
> Logs below. We tried switching to Java serialization, but that didn't work. We made the field transient (which is broken but let us make progress) and that did.
> Stack trace
> {code}
> com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
> Serialization trace:
> activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
> at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
> at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
> at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
> at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
> at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
> at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
> at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
> at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
> context: PTContainer[id=6(container_1492195730173_0001_01_000012),state=ACTIVE,operators=[
> {code}
> Larger logs with more scope:
> {code}
> 2017-04-14 18:56:49,961 INFO com.datatorrent.stram.StreamingAppMaster: Master starting with classpath: ./portability-demo-bundled-apex.jar:/etc/hadoop/conf:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-annotations.jar:/usr/lib/hadoop/hadoop-auth-2.7.3.jar:/usr/lib/hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-common-2.7.3-tests.jar:/usr/lib/hadoop/hadoop-annotations-2.7.3.jar:/usr/lib/hadoop/hadoop-nfs-2.7.3.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/hadoop/hadoop-common-2.7.3.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/snappy-java-1.0.5.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/lib/hadoop-lzo-0.4.19.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop/lib/bigquery-connector-0.10.1-hadoop2.jar:/usr/lib/hadoop/lib/slf4j-api-1.7.10.jar:/usr/lib/hadoop/lib/avro-1.7.7.jar:/usr/lib/hadoop/lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/gcs-connector-1.6.0-hadoop2.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/zookeeper-3.4.6.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26.jar:/usr/lib/hadoop/lib/curator-client-2.7.1.jar:/usr/lib/hadoop/lib/htrace-core-3.1.0-incubating.jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop/lib/jetty-6.1.26.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/httpclient-4.2.5.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/jersey-server-1.9.jar:/usr/lib/hadoop/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/lib/hadoop/lib/gson-2.2.4.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/jersey-json-1.9.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop/lib/httpcore-4.2.5.jar:/usr/lib/hadoop/lib/junit-4.11.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop/lib/commons-math3-3.1.1.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.7.3.jar:/usr/lib/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26.jar:/usr/lib/hadoop-hdfs/lib/htrace-core-3.1.0-incubating.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-mapreduce/commons-configuration-1.6.jar:/usr/lib/hadoop-mapreduce/hadoop-auth.jar:/usr/lib/hadoop-mapreduce/snappy-java-1.0.5.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/usr/lib/hadoop-mapreduce/commons-lang-2.6.jar:/usr/lib/hadoop-mapreduce/hamcrest-core-1.3.jar:/usr/lib/hadoop-mapreduce/hadoop-auth-2.7.3.jar:/usr/lib/hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-1.7.4.jar:/usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.7.3.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common.jar:/usr/lib/hadoop-mapreduce/hadoop-extras-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-sls-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-aws-2.7.3.jar:/usr/lib/hadoop-mapreduce/avro-1.7.7.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.7.3.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/lib/hadoop-mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.7.3.jar:/usr/lib/hadoop-mapreduce/mockito-all-1.8.5.jar:/usr/lib/hadoop-mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/usr/lib/hadoop-mapreduce/api-util-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle-2.7.3.jar:/usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/commons-digester-1.8.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-archives.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-mapreduce/zookeeper-3.4.6.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/usr/lib/hadoop-mapreduce/jetty-util-6.1.26.jar:/usr/lib/hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/hadoop-mapreduce/htrace-core-3.1.0-incubating.jar:/usr/lib/hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/joda-time-2.9.9.jar:/usr/lib/hadoop-mapreduce/jetty-6.1.26.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/jsch-0.1.42.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix-2.7.3.jar:/usr/lib/hadoop-mapreduce/xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/usr/lib/hadoop-mapreduce/azure-storage-2.0.0.jar:/usr/lib/hadoop-mapreduce/jettison-1.1.jar:/usr/lib/hadoop-mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming-2.7.3.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/httpclient-4.2.5.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2.jar:/usr/lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar:/usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar:/usr/lib/hadoop-mapreduce/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/hadoop-mapreduce/jackson-annotations-2.2.3.jar:/usr/lib/hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/jackson-databind-2.2.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app-2.7.3.jar:/usr/lib/hadoop-mapreduce/jersey-json-1.9.jar:/usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop-mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/httpcore-4.2.5.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-ant-2.7.3.jar:/usr/lib/hadoop-mapreduce/junit-4.11.jar:/usr/lib/hadoop-mapreduce/hadoop-aws.jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/hadoop-mapreduce/commons-net-3.1.jar:/usr/lib/hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/jackson-core-2.2.3.jar:/usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13.jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/usr/lib/hadoop-mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/hadoop-mapreduce/lib/snappy-java-1.0.5.jar:/usr/lib/hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop-mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.7.jar:/usr/lib/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/lib/xz-1.0.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/usr/lib/hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common.jar:/usr/lib/hadoop-yarn/lib/*'
> 2017-04-14 18:56:49,966 INFO com.datatorrent.stram.StreamingAppMaster: version: 3.5.0 from rev: 6de8828 branch: 6de8828e4f3d5734d0a6f9c1be0aa7057cb60ac8 by Thomas Weise on 06.12.2016 @ 22:11:51 PST
> 2017-04-14 18:56:49,967 INFO com.datatorrent.stram.StreamingAppMaster: appmaster env:
> PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
> HADOOP_CONF_DIR=/etc/hadoop/conf
> MAX_APP_ATTEMPTS=2
> DATAPROC_MASTER_STANDALONE_COMPONENTS=hadoop-hdfs-secondarynamenode
> DATAPROC_COMMON_COMPONENTS=openjdk-8-jdk libjansi-java python-numpy libmysql-java hadoop-client hive pig spark-core spark-python spark-r autofs nfs-common libhdfs0 libsnappy1 libatlas3-base libopenblas-base libapr1 vim git bash-completion spark-yarn-shuffle spark-datanucleus spark-extras hadoop-lzo
> MAIL=/var/mail/yarn
> LD_LIBRARY_PATH=:/usr/lib/hadoop/lib/native:/usr/lib/hadoop/lib/native
> HADOOP_HOME_WARN_SUPPRESS=true
> DATAPROC_WORKER_COMPONENTS=hadoop-hdfs-datanode hadoop-yarn-nodemanager
> LOGNAME=dhalperi
> JVM_PID=2123
> JSVC_HOME=/usr/lib/bigtop-utils
> PWD=/hadoop/yarn/nm-local-dir/usercache/dhalperi/appcache/application_1492195730173_0001/container_1492195730173_0001_01_000001
> HADOOP_YARN_USER=yarn
> HADOOP_PREFIX=/usr/lib/hadoop
> LOCAL_DIRS=/hadoop/yarn/nm-local-dir/usercache/dhalperi/appcache/application_1492195730173_0001
> YARN_IDENT_STRING=yarn
> SHELL=/bin/bash
> ALPN_JAR=/usr/local/share/google/alpn/alpn-boot-8.1.7.v20160121.jar
> YARN_CONF_DIR=/etc/hadoop/conf
> _SYSTEMCTL_SKIP_REDIRECT=true
> LOG_DIRS=/var/log/hadoop-yarn/userlogs/application_1492195730173_0001/container_1492195730173_0001_01_000001
> NM_AUX_SERVICE_mapreduce_shuffle=AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
> BDUTIL_DIR=/usr/local/share/google/dataproc/bdutil-dataproc-20170410-124434-RC1
> HADOOP_YARN_HOME=/usr/lib/hadoop-yarn
> YARN_PID_DIR=/var/run/hadoop-yarn
> DATAPROC_MASTER_EXCLUSIVE_COMPONENTS=hadoop-mapreduce-historyserver hive-metastore hive-server2 nfs-kernel-server spark-history-server
> NM_AUX_SERVICE_spark_shuffle=
> HADOOP_HOME=/usr/lib/hadoop
> SHLVL=5
> YARN_ROOT_LOGGER=INFO,RFA
> JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
> LANG=en_US.UTF-8
> APP_SUBMIT_TIME_ENV=1492196207488
> NM_HOST=gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal
> YARN_LOGFILE=yarn-yarn-nodemanager-gaming-apex-dhalperi-w-15.log
> HADOOP_USER_NAME=dhalperi
> DATAPROC_AGENT_JAR=/usr/local/share/google/dataproc/agent-20170410-124434-RC1.jar
> HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec
> YARN_LOG_DIR=/var/log/hadoop-yarn
> HADOOP_HDFS_HOME=/usr/lib/hadoop-hdfs
> HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce
> HADOOP_COMMON_HOME=/usr/lib/hadoop
> _=/usr/lib/jvm/java-8-openjdk-amd64/bin/java
> APPLICATION_WEB_PROXY_BASE=/proxy/application_1492195730173_0001
> NM_HTTP_PORT=8042
> NM_PORT=53054
> USER=dhalperi
> CLASSPATH=./*:/etc/hadoop/conf:/usr/lib/hadoop/*:/usr/lib/hadoop/lib/*:/usr/lib/hadoop-hdfs/*:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-mapreduce/*:/usr/lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-yarn/*:/usr/lib/hadoop-yarn/lib/*'
> DATAPROC_MASTER_HA_COMPONENTS=hadoop-hdfs-journalnode hadoop-hdfs-zkfc zookeeper-server
> HADOOP_TOKEN_FILE_LOCATION=/hadoop/yarn/nm-local-dir/usercache/dhalperi/appcache/application_1492195730173_0001/container_1492195730173_0001_01_000001/container_tokens
> DATAPROC_MASTER_COMPONENTS=hadoop-hdfs-namenode hadoop-yarn-resourcemanager mysql-server
> YARN_NICENESS=0
> HOME=/home/
> CONTAINER_ID=container_1492195730173_0001_01_000001
> MALLOC_ARENA_MAX=4
> 2017-04-14 18:56:50,236 INFO com.datatorrent.stram.StreamingAppMaster: Initializing Application Master.
> 2017-04-14 18:56:50,311 INFO com.datatorrent.stram.StreamingAppMasterService: Application master, appId=1, clustertimestamp=1492195730173, attemptId=1
> 2017-04-14 18:56:52,099 INFO com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase: GHFS version: 1.6.0-hadoop2
> 2017-04-14 18:56:52,491 INFO com.datatorrent.common.util.AsyncFSStorageAgent: using /hadoop/yarn/nm-local-dir/usercache/dhalperi/appcache/application_1492195730173_0001/container_1492195730173_0001_01_000001/tmp/chkp5833413517908828871 as the basepath for checkpointing.
> 2017-04-14 18:56:56,041 INFO com.datatorrent.stram.FSRecoveryHandler: Creating hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001/recovery/log
> 2017-04-14 18:56:56,172 INFO com.datatorrent.stram.StreamingAppMasterService: Starting application with 10 operators in 10 containers
> 2017-04-14 18:56:56,182 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Upper bound of the thread pool size is 500
> 2017-04-14 18:56:56,183 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
> 2017-04-14 18:56:56,201 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at gaming-apex-dhalperi-m/10.240.169.55:8030
> 2017-04-14 18:56:56,224 INFO com.datatorrent.stram.StreamingContainerParent: Config: Configuration: core-default.xml, core-site.xml, yarn-default.xml, yarn-site.xml, mapred-default.xml, mapred-site.xml, hdfs-default.xml, hdfs-site.xml
> 2017-04-14 18:56:56,224 INFO com.datatorrent.stram.StreamingContainerParent: Listener thread count 30
> 2017-04-14 18:56:56,230 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
> 2017-04-14 18:56:56,235 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 57363
> 2017-04-14 18:56:56,256 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> 2017-04-14 18:56:56,257 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 57363: starting
> 2017-04-14 18:56:56,289 INFO com.datatorrent.stram.StreamingContainerParent: Container callback server listening at gaming-apex-dhalperi-w-15/10.240.169.52:57363
> 2017-04-14 18:56:56,334 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
> 2017-04-14 18:56:56,417 INFO org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
> 2017-04-14 18:56:56,424 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.stram is not defined
> 2017-04-14 18:56:56,432 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
> 2017-04-14 18:56:56,434 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context stram
> 2017-04-14 18:56:56,434 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
> 2017-04-14 18:56:56,435 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
> 2017-04-14 18:56:56,437 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /stram/*
> 2017-04-14 18:56:56,437 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
> 2017-04-14 18:56:56,658 INFO org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
> 2017-04-14 18:56:56,659 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 32869
> 2017-04-14 18:57:08,363 WARN com.datatorrent.stram.webapp.TypeGraphFactory: The size of precomputed type graph is 8136 KB
> 2017-04-14 18:57:08,838 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app stram started at 32869
> 2017-04-14 18:57:08,838 INFO com.datatorrent.stram.StreamingAppMasterService: Started web service at port: 32869
> 2017-04-14 18:57:08,839 INFO com.datatorrent.stram.StreamingAppMasterService: Setting tracking URL to: gaming-apex-dhalperi-w-15:32869
> 2017-04-14 18:57:08,849 INFO com.datatorrent.stram.StreamingAppMasterService: Starting ApplicationMaster
> 2017-04-14 18:57:08,850 INFO com.datatorrent.stram.StreamingAppMasterService: number of tokens: 1
> 2017-04-14 18:57:08,908 INFO com.datatorrent.stram.StreamingAppMasterService: Max mem 12288m, Min mem 1024m, Max vcores 32000 and Min vcores 1 capabililty of resources in this cluster
> 2017-04-14 18:57:08,909 INFO com.datatorrent.stram.StreamingAppMasterService: Blacklist removal time in millis = 3600000, max consecutive node failure count = 2147483647
> 2017-04-14 18:57:08,913 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at gaming-apex-dhalperi-m/10.240.169.55:8032
> 2017-04-14 18:57:10,127 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=9,name=SumTeamScores/GroupByKey]
> 2017-04-14 18:57:10,127 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
> 2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=1,name=TextIO.Read/Read]
> 2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
> 2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=4,name=FixedWindows/Window.Assign]
> 2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
> 2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]
> 2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
> 2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=2,name=ParseGameEvent/ParMultiDo(ParseEvent)]
> 2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
> 2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)]
> 2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
> 2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]
> 2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
> 2017-04-14 18:57:10,132 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=5,name=SumTeamScores/ParDo(KeyScoreByTeam)/ParMultiDo(KeyScoreByTeam)]
> 2017-04-14 18:57:10,132 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
> 2017-04-14 18:57:10,132 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)]
> 2017-04-14 18:57:10,132 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
> 2017-04-14 18:57:10,132 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=3,name=SetTimestamps/ParMultiDo(SetTimestamps)]
> 2017-04-14 18:57:10,132 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
> 2017-04-14 18:57:10,132 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[0], Capability[<memory:1536, vCores:1>]Priority[1], Capability[<memory:1536, vCores:1>]Priority[2], Capability[<memory:1024, vCores:1>]Priority[3], Capability[<memory:1536, vCores:1>]Priority[4], Capability[<memory:1536, vCores:1>]Priority[5], Capability[<memory:1536, vCores:1>]Priority[6], Capability[<memory:1536, vCores:1>]Priority[7], Capability[<memory:1536, vCores:1>]Priority[8], Capability[<memory:1536, vCores:1>]Priority[9]]
> 2017-04-14 18:57:10,132 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[0] on host: [null]
> 2017-04-14 18:57:10,137 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[1] on host: [null]
> 2017-04-14 18:57:10,137 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[2] on host: [null]
> 2017-04-14 18:57:10,138 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1024, vCores:1>]Priority[3] on host: [null]
> 2017-04-14 18:57:10,138 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[4] on host: [null]
> 2017-04-14 18:57:10,138 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[5] on host: [null]
> 2017-04-14 18:57:10,138 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[6] on host: [null]
> 2017-04-14 18:57:10,138 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[7] on host: [null]
> 2017-04-14 18:57:10,138 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[8] on host: [null]
> 2017-04-14 18:57:10,138 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[9] on host: [null]
> 2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-9.c.clouddfe.google.com.internal:37663
> 2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-11.c.clouddfe.google.com.internal:50931
> 2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-5.c.clouddfe.google.com.internal:33276
> 2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-1.c.clouddfe.google.com.internal:37140
> 2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928
> 2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-12.c.clouddfe.google.com.internal:55762
> 2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743
> 2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-2.c.clouddfe.google.com.internal:46370
> 2017-04-14 18:57:11,217 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054
> 2017-04-14 18:57:11,217 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-18.c.clouddfe.google.com.internal:39804
> 2017-04-14 18:57:11,218 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000002, containerNode=gaming-apex-dhalperi-w-9.c.clouddfe.google.com.internal:37663, containerNodeURI=gaming-apex-dhalperi-w-9.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority0
> 2017-04-14 18:57:11,245 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000002
> 2017-04-14 18:57:11,251 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
> 2017-04-14 18:57:11,307 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
> 2017-04-14 18:57:11,307 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000002
> 2017-04-14 18:57:11,307 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-9.c.clouddfe.google.com.internal:37663 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000002 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
> 2017-04-14 18:57:11,317 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000003, containerNode=gaming-apex-dhalperi-w-11.c.clouddfe.google.com.internal:50931, containerNodeURI=gaming-apex-dhalperi-w-11.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority1
> 2017-04-14 18:57:11,321 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000002
> 2017-04-14 18:57:11,325 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000003
> 2017-04-14 18:57:11,326 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
> 2017-04-14 18:57:11,332 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
> 2017-04-14 18:57:11,332 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000003
> 2017-04-14 18:57:11,334 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-11.c.clouddfe.google.com.internal:50931 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000003 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
> 2017-04-14 18:57:11,334 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-9.c.clouddfe.google.com.internal:37663
> 2017-04-14 18:57:11,339 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000004, containerNode=gaming-apex-dhalperi-w-5.c.clouddfe.google.com.internal:33276, containerNodeURI=gaming-apex-dhalperi-w-5.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority2
> 2017-04-14 18:57:11,342 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000003
> 2017-04-14 18:57:11,345 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000004
> 2017-04-14 18:57:11,346 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
> 2017-04-14 18:57:11,350 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
> 2017-04-14 18:57:11,351 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000004
> 2017-04-14 18:57:11,351 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-5.c.clouddfe.google.com.internal:33276 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000004 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
> 2017-04-14 18:57:11,352 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000005, containerNode=gaming-apex-dhalperi-w-1.c.clouddfe.google.com.internal:37140, containerNodeURI=gaming-apex-dhalperi-w-1.c.clouddfe.google.com.internal:8042, containerResourceMemory1024, priority3
> 2017-04-14 18:57:11,359 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000005
> 2017-04-14 18:57:11,360 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
> 2017-04-14 18:57:11,361 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000004
> 2017-04-14 18:57:11,371 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
> 2017-04-14 18:57:11,372 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx805306368 for container container_1492195730173_0001_01_000005
> 2017-04-14 18:57:11,372 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-1.c.clouddfe.google.com.internal:37140 command: $JAVA_HOME/bin/java -Xmx805306368 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000005 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
> 2017-04-14 18:57:11,373 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000006, containerNode=gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928, containerNodeURI=gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority4
> 2017-04-14 18:57:11,373 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000005
> 2017-04-14 18:57:11,384 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000006
> 2017-04-14 18:57:11,384 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
> 2017-04-14 18:57:11,387 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
> 2017-04-14 18:57:11,388 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000006
> 2017-04-14 18:57:11,388 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000006 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
> 2017-04-14 18:57:11,389 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000007, containerNode=gaming-apex-dhalperi-w-12.c.clouddfe.google.com.internal:55762, containerNodeURI=gaming-apex-dhalperi-w-12.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority5
> 2017-04-14 18:57:11,394 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000007
> 2017-04-14 18:57:11,394 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
> 2017-04-14 18:57:11,401 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000006
> 2017-04-14 18:57:11,401 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
> 2017-04-14 18:57:11,406 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000007
> 2017-04-14 18:57:11,407 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-12.c.clouddfe.google.com.internal:55762 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000007 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
> 2017-04-14 18:57:11,407 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000008, containerNode=gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743, containerNodeURI=gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority6
> 2017-04-14 18:57:11,412 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-1.c.clouddfe.google.com.internal:37140
> 2017-04-14 18:57:11,413 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000007
> 2017-04-14 18:57:11,415 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000008
> 2017-04-14 18:57:11,415 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
> 2017-04-14 18:57:11,418 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-5.c.clouddfe.google.com.internal:33276
> 2017-04-14 18:57:11,421 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
> 2017-04-14 18:57:11,422 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000008
> 2017-04-14 18:57:11,422 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000008 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
> 2017-04-14 18:57:11,422 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000009, containerNode=gaming-apex-dhalperi-w-2.c.clouddfe.google.com.internal:46370, containerNodeURI=gaming-apex-dhalperi-w-2.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority7
> 2017-04-14 18:57:11,426 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000008
> 2017-04-14 18:57:11,430 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000009
> 2017-04-14 18:57:11,430 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
> 2017-04-14 18:57:11,433 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-11.c.clouddfe.google.com.internal:50931
> 2017-04-14 18:57:11,436 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
> 2017-04-14 18:57:11,436 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000009
> 2017-04-14 18:57:11,436 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-2.c.clouddfe.google.com.internal:46370 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000009 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
> 2017-04-14 18:57:11,437 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000010, containerNode=gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054, containerNodeURI=gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority8
> 2017-04-14 18:57:11,437 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000009
> 2017-04-14 18:57:11,449 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743
> 2017-04-14 18:57:11,449 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000010
> 2017-04-14 18:57:11,450 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
> 2017-04-14 18:57:11,454 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-12.c.clouddfe.google.com.internal:55762
> 2017-04-14 18:57:11,454 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
> 2017-04-14 18:57:11,455 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000010
> 2017-04-14 18:57:11,455 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000010 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
> 2017-04-14 18:57:11,455 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000011, containerNode=gaming-apex-dhalperi-w-18.c.clouddfe.google.com.internal:39804, containerNodeURI=gaming-apex-dhalperi-w-18.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority9
> 2017-04-14 18:57:11,459 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928
> 2017-04-14 18:57:11,463 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000011
> 2017-04-14 18:57:11,463 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
> 2017-04-14 18:57:11,460 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000010
> 2017-04-14 18:57:11,473 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-2.c.clouddfe.google.com.internal:46370
> 2017-04-14 18:57:11,474 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054
> 2017-04-14 18:57:11,482 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
> 2017-04-14 18:57:11,482 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000011
> 2017-04-14 18:57:11,482 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-18.c.clouddfe.google.com.internal:39804 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000011 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
> 2017-04-14 18:57:11,490 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000011
> 2017-04-14 18:57:11,500 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-18.c.clouddfe.google.com.internal:39804
> 2017-04-14 18:57:13,756 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000010] Entering heartbeat loop.. context: PTContainer[id=7(container_1492195730173_0001_01_000010),state=ALLOCATED,operators=[PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)]]]
> 2017-04-14 18:57:14,770 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000010 buffer server: gaming-apex-dhalperi-w-15:55080
> 2017-04-14 18:57:14,941 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000008] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000008),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
> 2017-04-14 18:57:15,054 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000002] Entering heartbeat loop.. context: PTContainer[id=9(container_1492195730173_0001_01_000002),state=ALLOCATED,operators=[PTOperator[id=9,name=SumTeamScores/GroupByKey]]]
> 2017-04-14 18:57:15,073 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000009] Entering heartbeat loop.. context: PTContainer[id=5(container_1492195730173_0001_01_000009),state=ALLOCATED,operators=[PTOperator[id=5,name=SumTeamScores/ParDo(KeyScoreByTeam)/ParMultiDo(KeyScoreByTeam)]]]
> 2017-04-14 18:57:15,106 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000003] Entering heartbeat loop.. context: PTContainer[id=1(container_1492195730173_0001_01_000003),state=ALLOCATED,operators=[PTOperator[id=1,name=TextIO.Read/Read]]]
> 2017-04-14 18:57:15,125 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000011] Entering heartbeat loop.. context: PTContainer[id=3(container_1492195730173_0001_01_000011),state=ALLOCATED,operators=[PTOperator[id=3,name=SetTimestamps/ParMultiDo(SetTimestamps)]]]
> 2017-04-14 18:57:15,232 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000006] Entering heartbeat loop.. context: PTContainer[id=2(container_1492195730173_0001_01_000006),state=ALLOCATED,operators=[PTOperator[id=2,name=ParseGameEvent/ParMultiDo(ParseEvent)]]]
> 2017-04-14 18:57:15,333 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000007] Entering heartbeat loop.. context: PTContainer[id=8(container_1492195730173_0001_01_000007),state=ALLOCATED,operators=[PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)]]]
> 2017-04-14 18:57:15,346 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000004] Entering heartbeat loop.. context: PTContainer[id=4(container_1492195730173_0001_01_000004),state=ALLOCATED,operators=[PTOperator[id=4,name=FixedWindows/Window.Assign]]]
> 2017-04-14 18:57:15,472 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000005] Entering heartbeat loop.. context: PTContainer[id=10(container_1492195730173_0001_01_000005),state=ALLOCATED,operators=[PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]]]
> 2017-04-14 18:57:15,954 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000008 buffer server: gaming-apex-dhalperi-w-3:60148
> 2017-04-14 18:57:16,066 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000002 buffer server: gaming-apex-dhalperi-w-9:52966
> 2017-04-14 18:57:16,085 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000009 buffer server: gaming-apex-dhalperi-w-2:41280
> 2017-04-14 18:57:16,116 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000003 buffer server: gaming-apex-dhalperi-w-11:41089
> 2017-04-14 18:57:16,136 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000011 buffer server: gaming-apex-dhalperi-w-18:50437
> 2017-04-14 18:57:16,244 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000006 buffer server: gaming-apex-dhalperi-w-16:55760
> 2017-04-14 18:57:16,345 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000007 buffer server: gaming-apex-dhalperi-w-12:42777
> 2017-04-14 18:57:16,359 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000004 buffer server: gaming-apex-dhalperi-w-5:58095
> 2017-04-14 18:57:16,485 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000005 buffer server: gaming-apex-dhalperi-w-1:55004
> 2017-04-14 19:00:14,948 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196414947, last window id change time 1492196354898, window processing timeout millis 60000
> 2017-04-14 19:00:14,948 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=9,name=SumTeamScores/GroupByKey] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196414947, last window id change time 1492196354896, window processing timeout millis 60000
> 2017-04-14 19:00:15,950 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196415950, last window id change time 1492196355249, window processing timeout millis 60000
> 2017-04-14 19:00:15,950 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196415950, last window id change time 1492196355318, window processing timeout millis 60000
> 2017-04-14 19:00:15,950 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196415950, last window id change time 1492196354898, window processing timeout millis 60000
> 2017-04-14 19:00:15,950 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=9,name=SumTeamScores/GroupByKey] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196415950, last window id change time 1492196354896, window processing timeout millis 60000
> 2017-04-14 19:00:15,950 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196415950, last window id change time 1492196354966, window processing timeout millis 60000
> 2017-04-14 19:00:15,951 INFO com.datatorrent.stram.StreamingContainerManager: Blocked operator PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey] container PTContainer[id=6(container_1492195730173_0001_01_000008),state=ACTIVE] time 60701ms
> 2017-04-14 19:00:16,951 INFO com.datatorrent.stram.StreamingAppMasterService: Requested stop container container_1492195730173_0001_01_000008
> 2017-04-14 19:00:16,951 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: STOP_CONTAINER for Container container_1492195730173_0001_01_000008
> 2017-04-14 19:00:16,953 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196416953, last window id change time 1492196355249, window processing timeout millis 60000
> 2017-04-14 19:00:16,953 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196416953, last window id change time 1492196355318, window processing timeout millis 60000
> 2017-04-14 19:00:16,954 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196416953, last window id change time 1492196354898, window processing timeout millis 60000
> 2017-04-14 19:00:16,954 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=9,name=SumTeamScores/GroupByKey] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196416953, last window id change time 1492196354896, window processing timeout millis 60000
> 2017-04-14 19:00:16,954 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196416953, last window id change time 1492196354966, window processing timeout millis 60000
> 2017-04-14 19:00:16,954 INFO com.datatorrent.stram.StreamingContainerManager: Blocked operator PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey] container PTContainer[id=6(container_1492195730173_0001_01_000008),state=ACTIVE] time 61704ms
> 2017-04-14 19:00:16,957 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743
> 2017-04-14 19:00:17,962 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000008, state=COMPLETE, exitStatus=-105, diagnostics=Container killed by the ApplicationMaster.
> Container killed on request. Exit code is 143
> Container exited with a non-zero exit code 143
> 2017-04-14 19:00:17,962 INFO com.datatorrent.stram.StreamingContainerManager: Initiating recovery for container_1492195730173_0001_01_000008@gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743
> 2017-04-14 19:00:17,965 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196417965, last window id change time 1492196355249, window processing timeout millis 60000
> 2017-04-14 19:00:17,965 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196417965, last window id change time 1492196355318, window processing timeout millis 60000
> 2017-04-14 19:00:17,965 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196417965, last window id change time 1492196354898, window processing timeout millis 60000
> 2017-04-14 19:00:17,965 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=9,name=SumTeamScores/GroupByKey] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196417965, last window id change time 1492196354896, window processing timeout millis 60000
> 2017-04-14 19:00:17,965 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196417965, last window id change time 1492196354966, window processing timeout millis 60000
> 2017-04-14 19:00:17,965 INFO com.datatorrent.stram.StreamingContainerManager: Affected operators [PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey], PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)], PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)], PTOperator[id=9,name=SumTeamScores/GroupByKey], PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]]
> 2017-04-14 19:00:19,013 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]
> 2017-04-14 19:00:19,013 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
> 2017-04-14 19:00:19,014 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[10]]
> 2017-04-14 19:00:19,014 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[10] on host: [null]
> 2017-04-14 19:00:20,021 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000012, containerNode=gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743, containerNodeURI=gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority10
> 2017-04-14 19:00:20,022 INFO com.datatorrent.stram.StreamingContainerManager: Removing container agent container_1492195730173_0001_01_000008
> 2017-04-14 19:00:20,024 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000012
> 2017-04-14 19:00:20,024 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
> 2017-04-14 19:00:20,034 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
> 2017-04-14 19:00:20,035 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000012
> 2017-04-14 19:00:20,035 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000012 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
> 2017-04-14 19:00:20,035 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000012
> 2017-04-14 19:00:20,036 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743
> 2017-04-14 19:00:21,762 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000012] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000012),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
> 2017-04-14 19:00:22,774 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000012 buffer server: gaming-apex-dhalperi-w-3:54029
> 2017-04-14 19:00:23,006 INFO com.datatorrent.stram.StreamingContainerParent: child msg: deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-3]]]] com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
> Serialization trace:
> activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
> at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
> at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
> at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
> at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
> at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
> at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
> at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
> at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
> context: PTContainer[id=6(container_1492195730173_0001_01_000012),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
> 2017-04-14 19:00:23,011 INFO com.datatorrent.stram.StreamingContainerParent: child msg: java.lang.IllegalStateException: Deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-3]]]]
> at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:829)
> at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
> at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
> Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
> Serialization trace:
> activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
> at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
> at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
> at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
> at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
> at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
> at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
> ... 2 more
> context: PTContainer[id=6(container_1492195730173_0001_01_000012),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
> 2017-04-14 19:00:24,048 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000012, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch.
> Container id: container_1492195730173_0001_01_000012
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
> at org.apache.hadoop.util.Shell.run(Shell.java:479)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1
> 2017-04-14 19:00:24,049 INFO com.datatorrent.stram.StreamingContainerManager: Initiating recovery for container_1492195730173_0001_01_000012@gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743
> 2017-04-14 19:00:24,049 INFO com.datatorrent.stram.StreamingContainerManager: Affected operators [PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey], PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)], PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)], PTOperator[id=9,name=SumTeamScores/GroupByKey], PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]]
> 2017-04-14 19:00:25,072 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]
> 2017-04-14 19:00:25,072 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
> 2017-04-14 19:00:25,072 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[11]]
> 2017-04-14 19:00:25,072 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[11] on host: [null]
> 2017-04-14 19:00:26,078 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000013, containerNode=gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928, containerNodeURI=gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority11
> 2017-04-14 19:00:26,079 INFO com.datatorrent.stram.StreamingContainerManager: Removing container agent container_1492195730173_0001_01_000012
> 2017-04-14 19:00:26,081 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000013
> 2017-04-14 19:00:26,081 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
> 2017-04-14 19:00:26,084 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
> 2017-04-14 19:00:26,084 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000013
> 2017-04-14 19:00:26,085 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000013 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
> 2017-04-14 19:00:26,085 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000013
> 2017-04-14 19:00:26,096 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928
> 2017-04-14 19:00:30,185 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000013] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000013),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
> 2017-04-14 19:00:31,201 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000013 buffer server: gaming-apex-dhalperi-w-16:43881
> 2017-04-14 19:00:31,590 INFO com.datatorrent.stram.StreamingContainerParent: child msg: deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-16]]]] com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
> Serialization trace:
> activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
> at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
> at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
> at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
> at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
> at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
> at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
> at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
> at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
> context: PTContainer[id=6(container_1492195730173_0001_01_000013),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
> 2017-04-14 19:00:31,618 INFO com.datatorrent.stram.StreamingContainerParent: child msg: java.lang.IllegalStateException: Deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-16]]]]
> at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:829)
> at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
> at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
> Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
> Serialization trace:
> activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
> at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
> at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
> at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
> at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
> at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
> at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
> ... 2 more
> context: PTContainer[id=6(container_1492195730173_0001_01_000013),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
> 2017-04-14 19:00:34,104 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000013, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch.
> Container id: container_1492195730173_0001_01_000013
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
> at org.apache.hadoop.util.Shell.run(Shell.java:479)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1
> 2017-04-14 19:00:34,105 INFO com.datatorrent.stram.StreamingContainerManager: Initiating recovery for container_1492195730173_0001_01_000013@gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928
> 2017-04-14 19:00:34,105 INFO com.datatorrent.stram.StreamingContainerManager: Affected operators [PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey], PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)], PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)], PTOperator[id=9,name=SumTeamScores/GroupByKey], PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]]
> 2017-04-14 19:00:35,127 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]
> 2017-04-14 19:00:35,127 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
> 2017-04-14 19:00:35,127 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[12]]
> 2017-04-14 19:00:35,127 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[12] on host: [null]
> 2017-04-14 19:00:36,135 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000014, containerNode=gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054, containerNodeURI=gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority12
> 2017-04-14 19:00:36,135 INFO com.datatorrent.stram.StreamingContainerManager: Removing container agent container_1492195730173_0001_01_000013
> 2017-04-14 19:00:36,138 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000014
> 2017-04-14 19:00:36,138 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
> 2017-04-14 19:00:36,144 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
> 2017-04-14 19:00:36,144 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000014
> 2017-04-14 19:00:36,144 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000014 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
> 2017-04-14 19:00:36,144 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000014
> 2017-04-14 19:00:36,145 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054
> 2017-04-14 19:00:37,902 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000014] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000014),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
> 2017-04-14 19:00:38,913 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000014 buffer server: gaming-apex-dhalperi-w-15:59584
> 2017-04-14 19:00:39,107 INFO com.datatorrent.stram.StreamingContainerParent: child msg: deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-15]]]] com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
> Serialization trace:
> activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
> at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
> at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
> at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
> at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
> at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
> at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
> at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
> at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
> context: PTContainer[id=6(container_1492195730173_0001_01_000014),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
> 2017-04-14 19:00:39,112 INFO com.datatorrent.stram.StreamingContainerParent: child msg: java.lang.IllegalStateException: Deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-15]]]]
> at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:829)
> at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
> at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
> Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
> Serialization trace:
> activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
> at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
> at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
> at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
> at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
> at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
> at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
> ... 2 more
> context: PTContainer[id=6(container_1492195730173_0001_01_000014),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
> 2017-04-14 19:00:40,160 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000014, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch.
> Container id: container_1492195730173_0001_01_000014
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
> at org.apache.hadoop.util.Shell.run(Shell.java:479)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1
> 2017-04-14 19:00:40,161 INFO com.datatorrent.stram.StreamingContainerManager: Initiating recovery for container_1492195730173_0001_01_000014@gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054
> 2017-04-14 19:00:40,161 INFO com.datatorrent.stram.StreamingContainerManager: Affected operators [PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey], PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)], PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)], PTOperator[id=9,name=SumTeamScores/GroupByKey], PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]]
> 2017-04-14 19:00:41,199 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]
> 2017-04-14 19:00:41,199 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
> 2017-04-14 19:00:41,199 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[13]]
> 2017-04-14 19:00:41,199 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[13] on host: [null]
> 2017-04-14 19:00:42,206 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-8.c.clouddfe.google.com.internal:53733
> 2017-04-14 19:00:42,206 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000015, containerNode=gaming-apex-dhalperi-w-8.c.clouddfe.google.com.internal:53733, containerNodeURI=gaming-apex-dhalperi-w-8.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority13
> 2017-04-14 19:00:42,206 INFO com.datatorrent.stram.StreamingContainerManager: Removing container agent container_1492195730173_0001_01_000014
> 2017-04-14 19:00:42,208 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000015
> 2017-04-14 19:00:42,208 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
> 2017-04-14 19:00:42,212 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
> 2017-04-14 19:00:42,212 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000015
> 2017-04-14 19:00:42,212 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-8.c.clouddfe.google.com.internal:53733 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000015 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
> 2017-04-14 19:00:42,212 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Set NMClientAsync thread pool size to 21 as the number of nodes to talk to is 11
> 2017-04-14 19:00:42,214 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000015
> 2017-04-14 19:00:42,219 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-8.c.clouddfe.google.com.internal:53733
> 2017-04-14 19:00:45,476 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000015] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000015),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
> 2017-04-14 19:00:46,487 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000015 buffer server: gaming-apex-dhalperi-w-8:57550
> 2017-04-14 19:00:46,661 INFO com.datatorrent.stram.StreamingContainerParent: child msg: deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-8]]]] com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
> Serialization trace:
> activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
> at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
> at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
> at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
> at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
> at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
> at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
> at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
> at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
> context: PTContainer[id=6(container_1492195730173_0001_01_000015),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
> 2017-04-14 19:00:46,668 INFO com.datatorrent.stram.StreamingContainerParent: child msg: java.lang.IllegalStateException: Deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-8]]]]
> at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:829)
> at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
> at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
> Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
> Serialization trace:
> activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
> at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
> at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
> at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
> at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
> at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
> at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
> ... 2 more
> context: PTContainer[id=6(container_1492195730173_0001_01_000015),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
> 2017-04-14 19:00:47,225 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000015, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch.
> Container id: container_1492195730173_0001_01_000015
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
> at org.apache.hadoop.util.Shell.run(Shell.java:479)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1
> 2017-04-14 19:00:47,226 INFO com.datatorrent.stram.StreamingContainerManager: Initiating recovery for container_1492195730173_0001_01_000015@gaming-apex-dhalperi-w-8.c.clouddfe.google.com.internal:53733
> 2017-04-14 19:00:47,226 INFO com.datatorrent.stram.StreamingContainerManager: Affected operators [PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey], PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)], PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)], PTOperator[id=9,name=SumTeamScores/GroupByKey], PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]]
> 2017-04-14 19:00:48,263 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]
> 2017-04-14 19:00:48,263 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
> 2017-04-14 19:00:48,264 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[14]]
> 2017-04-14 19:00:48,264 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[14] on host: [null]
> 2017-04-14 19:00:49,271 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000016, containerNode=gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928, containerNodeURI=gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority14
> 2017-04-14 19:00:49,272 INFO com.datatorrent.stram.StreamingContainerManager: Removing container agent container_1492195730173_0001_01_000015
> 2017-04-14 19:00:49,274 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000016
> 2017-04-14 19:00:49,274 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
> 2017-04-14 19:00:49,277 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
> 2017-04-14 19:00:49,277 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000016
> 2017-04-14 19:00:49,277 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000016 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
> 2017-04-14 19:00:49,278 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000016
> 2017-04-14 19:00:49,279 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928
> 2017-04-14 19:00:52,751 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000016] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000016),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
> 2017-04-14 19:00:53,775 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000016 buffer server: gaming-apex-dhalperi-w-16:52403
> 2017-04-14 19:00:54,135 INFO com.datatorrent.stram.StreamingContainerParent: child msg: deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-16]]]] com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
> Serialization trace:
> activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
> at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
> at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
> at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
> at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
> at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
> at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
> at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
> at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
> context: PTContainer[id=6(container_1492195730173_0001_01_000016),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
> 2017-04-14 19:00:54,145 INFO com.datatorrent.stram.StreamingContainerParent: child msg: java.lang.IllegalStateException: Deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-16]]]]
> at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:829)
> at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
> at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
> Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
> Serialization trace:
> activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
> at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
> at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
> at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
> at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
> at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
> at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
> ... 2 more
> context: PTContainer[id=6(container_1492195730173_0001_01_000016),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
> 2017-04-14 19:00:55,291 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000016, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch.
> Container id: container_1492195730173_0001_01_000016
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
> at org.apache.hadoop.util.Shell.run(Shell.java:479)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1
> 2017-04-14 19:00:55,292 INFO com.datatorrent.stram.StreamingContainerManager: Initiating recovery for container_1492195730173_0001_01_000016@gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928
> 2017-04-14 19:00:55,292 INFO com.datatorrent.stram.StreamingContainerManager: Affected operators [PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey], PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)], PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)], PTOperator[id=9,name=SumTeamScores/GroupByKey], PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]]
> 2017-04-14 19:00:56,328 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]
> 2017-04-14 19:00:56,328 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
> 2017-04-14 19:00:56,329 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[15]]
> 2017-04-14 19:00:56,329 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[15] on host: [null]
> 2017-04-14 19:00:57,337 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-14.c.clouddfe.google.com.internal:56866
> 2017-04-14 19:00:57,337 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000017, containerNode=gaming-apex-dhalperi-w-14.c.clouddfe.google.com.internal:56866, containerNodeURI=gaming-apex-dhalperi-w-14.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority15
> 2017-04-14 19:00:57,337 INFO com.datatorrent.stram.StreamingContainerManager: Removing container agent container_1492195730173_0001_01_000016
> 2017-04-14 19:00:57,339 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000017
> 2017-04-14 19:00:57,340 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
> 2017-04-14 19:00:57,348 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
> 2017-04-14 19:00:57,348 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000017
> 2017-04-14 19:00:57,348 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-14.c.clouddfe.google.com.internal:56866 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000017 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
> 2017-04-14 19:00:57,349 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000017
> 2017-04-14 19:00:57,354 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-14.c.clouddfe.google.com.internal:56866
> 2017-04-14 19:01:01,038 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000017] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000017),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
> 2017-04-14 19:01:02,049 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000017 buffer server: gaming-apex-dhalperi-w-14:50842
> 2017-04-14 19:01:02,198 INFO com.datatorrent.stram.StreamingContainerParent: child msg: deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-14]]]] com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
> Serialization trace:
> activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
> at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
> at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
> at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
> at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
> at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
> at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
> at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
> at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
> context: PTContainer[id=6(container_1492195730173_0001_01_000017),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
> 2017-04-14 19:01:02,204 INFO com.datatorrent.stram.StreamingContainerParent: child msg: java.lang.IllegalStateException: Deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-14]]]]
> at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:829)
> at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
> at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
> Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
> Serialization trace:
> activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
> at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
> at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
> at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
> at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
> at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
> at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
> ... 2 more
> context: PTContainer[id=6(container_1492195730173_0001_01_000017),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
> 2017-04-14 19:01:03,363 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000017, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch.
> Container id: container_1492195730173_0001_01_000017
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
> at org.apache.hadoop.util.Shell.run(Shell.java:479)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1
> 2017-04-14 19:01:03,363 INFO com.datatorrent.stram.StreamingContainerManager: Initiating recovery for container_1492195730173_0001_01_000017@gaming-apex-dhalperi-w-14.c.clouddfe.google.com.internal:56866
> 2017-04-14 19:01:03,364 INFO com.datatorrent.stram.StreamingContainerManager: Affected operators [PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey], PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)], PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)], PTOperator[id=9,name=SumTeamScores/GroupByKey], PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]]
> 2017-04-14 19:01:04,387 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]
> 2017-04-14 19:01:04,387 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
> 2017-04-14 19:01:04,388 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[16]]
> 2017-04-14 19:01:04,388 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[16] on host: [null]
> 2017-04-14 19:01:05,396 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-6.c.clouddfe.google.com.internal:43371
> 2017-04-14 19:01:05,396 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000018, containerNode=gaming-apex-dhalperi-w-6.c.clouddfe.google.com.internal:43371, containerNodeURI=gaming-apex-dhalperi-w-6.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority16
> 2017-04-14 19:01:05,397 INFO com.datatorrent.stram.StreamingContainerManager: Removing container agent container_1492195730173_0001_01_000017
> 2017-04-14 19:01:05,399 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000018
> 2017-04-14 19:01:05,399 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
> 2017-04-14 19:01:05,401 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
> 2017-04-14 19:01:05,401 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000018
> 2017-04-14 19:01:05,401 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-6.c.clouddfe.google.com.internal:43371 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000018 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
> 2017-04-14 19:01:05,402 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000018
> 2017-04-14 19:01:05,407 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-6.c.clouddfe.google.com.internal:43371
> 2017-04-14 19:01:09,197 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000018] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000018),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
> 2017-04-14 19:01:10,209 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000018 buffer server: gaming-apex-dhalperi-w-6:49764
> 2017-04-14 19:01:10,377 INFO com.datatorrent.stram.StreamingContainerParent: child msg: deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-6]]]] com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
> Serialization trace:
> activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
> at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
> at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
> at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
> at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
> at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
> at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
> at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
> at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
> context: PTContainer[id=6(container_1492195730173_0001_01_000018),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
> 2017-04-14 19:01:10,385 INFO com.datatorrent.stram.StreamingContainerParent: child msg: java.lang.IllegalStateException: Deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-6]]]]
> at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:829)
> at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
> at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
> Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
> Serialization trace:
> activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
> at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
> at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
> at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
> at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
> at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
> at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
> at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
> ... 2 more
> context: PTContainer[id=6(container_1492195730173_0001_01_000018),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
> 2017-04-14 19:01:11,416 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000018, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch.
> Container id: container_1492195730173_0001_01_000018
> Exit code: 1
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)