You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by "Bandaru, Srinivas" <sr...@optum.com> on 2016/10/13 17:46:42 UTC
DT Fault Tolerance UHG
Hi,
Need some help. While running DT application with “Kafka” running into issues with application. When monitoring the application, We are observing that the operator getting inactive and restarts continuously. Could you please refer the below log let us know if any configuration need to be changed?
2016-10-122016-10-12 16:40:16,835 INFO org.apache.kafka.common.utils.AppInfoParser: Kafka version : 0.9.0.1
2016-10-12 16:40:16,835 INFO org.apache.kafka.common.utils.AppInfoParser: Kafka commitId : 23c69d62a0cabf06
16:40:17,153 INFO org.apache.apex.malhar.kafka.KafkaSinglePortExactlyOnceOutputOperator: Rebuild the partial window after 6340695403456888950
2016-10-12 16:40:18,827 ERROR com.datatorrent.stram.engine.StreamingContainer: Operator set [OperatorDeployInfo[id=4,name=kafkaOut1,type=GENERIC,checkpoint={57feace00000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=inputPort,streamId=updateTopic1,sourceNodeId=3,sourcePortName=outTopic1,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[]]] stopped running due to an exception.
java.lang.RuntimeException: Violates Exactly once. Not all the tuples received after operator reset.
at org.apache.apex.malhar.kafka.KafkaSinglePortExactlyOnceOutputOperator.endWindow(KafkaSinglePortExactlyOnceOutputOperator.java:174)
at com.datatorrent.stram.engine.GenericNode.processEndWindow(GenericNode.java:146)
at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:357)
at com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1407)
2016-10-12 16:40:18,838 INFO org.apache.kafka.clients.producer.KafkaProducer: Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
2016-10-12 16:40:18,862 INFO com.datatorrent.stram.engine.StreamingContainer: Undeploy request: [4]
2016-10-12 16:40:18,863 INFO com.datatorrent.stram.engine.StreamingContainer: Undeploy complete.
__stderr__0__stdout__0,_*container_e31_1476212051326_0045_01_000087¨Oø__dt.log__204352016-10-12 16:51:42,333 INFO com.datatorrent.stram.engine.StreamingContainer: Child starting with classpath: ./commons-beanutils-1.8.3.jar:./apex-api-3.4.0.jar:./apex-bufferserver-3.4.0.jar:./commons-lang3-3.1.jar:./httpcore-4.3.2.jar:./snappy-java-1.1.1.7.jar:./zkclient-0.7.jar:./jctools-core-1.1.jar:./jopt-simple-3.2.jar:./apex-shaded-ning19-1.0.0.jar:./malhar-library-3.5.0.jar:./Kafka2Datatorrent-1.0-SNAPSHOT.jar:./bval-jsr303-0.5.jar:./httpclient-4.3.5.jar:./jackson-mapper-asl-1.9.2.jar:./kafka_2.10-0.9.0.1.jar:./bval-core-0.5.jar:./minlog-1.2.jar:./jersey-apache-client4-1.9.jar:./malhar-contrib-3.4.0.jar:./metrics-core-2.2.0.jar:./jackson-core-asl-1.9.2.jar:./validation-api-1.1.0.Final.jar:./gson-2.0.jar:./kryo-2.24.0.jar:./netlet-1.2.1.jar:./lz4-1.2.0.jar:./mbassador-1.1.9.jar:./slf4j-api-1.7.5.jar:./kafka-clients-0.9.0.1.jar:./scala-library-2.10.5.jar:./kafka-unit-0.4.jar:./apex-common-3.4.0.jar:./xbean-asm5-shaded-4.3.jar:./jersey-client-1.9.jar:./apex-engine-4.jar:./malhar-kafka-3.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-nfs-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-common-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-common-2.7.0-mapr-1602-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/stax-api-1.0-2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-diagnostic-tools-5.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-client-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-5.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jets3t-0.9.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-framework-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-hbase-5.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hamcrest-core-1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/httpclient-4.2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/json-20080701.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/slf4j-log4j12-1.7.12.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/zookeeper-3.4.5-mapr-1503.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-mapreduce-5.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/activation-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-databind-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-core-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-io-2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/servlet-api-2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-tools-5.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-tools-5.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-cli-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-auth-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-azure-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/ojai-1.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/avro-1.7.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/log4j-1.2.17.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-5.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/central-logging-5.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-aws-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsch-0.1.42.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-math3-3.1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/ojai-mapreduce-1.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-json-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/azure-storage-2.0.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jettison-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/paranamer-2.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-hbase-5.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/asm-3.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/xz-1.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-annotations-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/antlr4-runtime-4.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/aws-java-sdk-1.7.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/httpcore-4.2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-collections-3.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-shell-5.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-server-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jetty-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-annotations-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-5.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-httpclient-3.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-java-utils-5.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/junit-4.11.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-codec-1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-logging-1.1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-digester-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-mapreduce-5.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/gson-2.2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-core-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/guava-14.0.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/xmlenc-0.52.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsr305-3.0.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-net-3.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/slf4j-api-1.7.12.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mysql-connector-java-5.1.25.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-lang-2.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-2.7.0-mapr-1602-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-io-2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/asm-3.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/guava-14.0.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-common-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-registry-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-client-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-api-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-common-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/javax.inject-1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/zookeeper-3.4.5-mapr-1503.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/zookeeper-3.4.5-mapr-1503-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/activation-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/guice-3.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-io-2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/servlet-api-2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-cli-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-json-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jettison-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/asm-3.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/xz-1.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/eval-0.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jetty-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-codec-1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/guava-14.0.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-client-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-lang-2.6.jar:.
2016-10-12 16:51:43,750 INFO com.datatorrent.bufferserver.storage.DiskStorage: using /opt/mapr/tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1476212051326_0045/container_e31_1476212051326_0045_01_000087/tmp as the basepath for spooling.
2016-10-12 16:51:43,755 INFO com.datatorrent.bufferserver.server.Server: Server started listening at /0:0:0:0:0:0:0:0:55309
2016-10-12 16:51:44,796 INFO com.datatorrent.stram.engine.StreamingContainer: Deploy request: [OperatorDeployInfo[id=4,name=kafkaOut1,type=GENERIC,checkpoint={57feace00000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=inputPort,streamId=updateTopic1,sourceNodeId=3,sourcePortName=outTopic1,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[]]]
2016-10-12 16:51:44,800 INFO org.apache.hadoop.conf.Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2016-10-12 16:51:44,910 INFO org.apache.kafka.clients.producer.ProducerConfig: ProducerConfig values:
compression.type = none
metric.reporters = []
metadata.max.age.ms = 300000
metadata.fetch.timeout.ms = 60000
reconnect.backoff.ms = 50
sasl.kerberos.ticket.renew.window.factor = 0.8
bootstrap.servers = [apsrd7102.uhc.com:9092]
retry.backoff.ms = 100
sasl.kerberos.kinit.cmd = /usr/bin/kinit
buffer.memory = 33554432
timeout.ms = 30000
key.serializer = class org.apache.kafka.common.serialization.StringSerializer
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
ssl.keystore.type = JKS
ssl.trustmanager.algorithm = PKIX
block.on.buffer.full = false
ssl.key.password = null
max.block.ms = 60000
sasl.kerberos.min.time.before.relogin = 60000
connections.max.idle.ms = 540000
ssl.truststore.password = null
max.in.flight.requests.per.connection = 5
metrics.num.samples = 2
client.id =
ssl.endpoint.identification.algorithm = null
ssl.protocol = TLS
request.timeout.ms = 30000
ssl.provider = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
acks = all
batch.size = 16384
ssl.keystore.location = null
receive.buffer.bytes = 32768
ssl.cipher.suites = null
ssl.truststore.type = JKS
security.protocol = PLAINTEXT
retries = 2
max.request.size = 1048576
value.serializer = class com.example.datatorrent.Application$TenantSerializer
ssl.truststore.location = null
ssl.keystore.password = null
ssl.keymanager.algorithm = SunX509
metrics.sample.window.ms = 30000
partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
send.buffer.bytes = 131072
linger.ms = 1
2016-10-122016-10-12 16:40:16,835 INFO org.apache.kafka.common.utils.AppInfoParser: Kafka version : 0.9.0.1
2016-10-12 16:40:16,835 INFO org.apache.kafka.common.utils.AppInfoParser: Kafka commitId : 23c69d62a0cabf06
16:40:17,153 INFO org.apache.apex.malhar.kafka.KafkaSinglePortExactlyOnceOutputOperator: Rebuild the partial window after 6340695403456888950
2016-10-12 16:40:18,827 ERROR com.datatorrent.stram.engine.StreamingContainer: Operator set [OperatorDeployInfo[id=4,name=kafkaOut1,type=GENERIC,checkpoint={57feace00000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=inputPort,streamId=updateTopic1,sourceNodeId=3,sourcePortName=outTopic1,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[]]] stopped running due to an exception.
java.lang.RuntimeException: Violates Exactly once. Not all the tuples received after operator reset.
at org.apache.apex.malhar.kafka.KafkaSinglePortExactlyOnceOutputOperator.endWindow(KafkaSinglePortExactlyOnceOutputOperator.java:174)
at com.datatorrent.stram.engine.GenericNode.processEndWindow(GenericNode.java:146)
at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:357)
at com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1407)
2016-10-12 16:40:18,838 INFO org.apache.kafka.clients.producer.KafkaProducer: Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
2016-10-12 16:40:18,862 INFO com.datatorrent.stram.engine.StreamingContainer: Undeploy request: [4]
2016-10-12 16:40:18,863 INFO com.datatorrent.stram.engine.StreamingContainer: Undeploy complete.
__stderr__0__stdout__0,_*container_e31_1476212051326_0045_01_000087¨Oø__dt.log__204352016-10-12 16:51:42,333 INFO com.datatorrent.stram.engine.StreamingContainer: Child starting with classpath: ./commons-beanutils-1.8.3.jar:./apex-api-3.4.0.jar:./apex-bufferserver-3.4.0.jar:./commons-lang3-3.1.jar:./httpcore-4.3.2.jar:./snappy-java-1.1.1.7.jar:./zkclient-0.7.jar:./jctools-core-1.1.jar:./jopt-simple-3.2.jar:./apex-shaded-ning19-1.0.0.jar:./malhar-library-3.5.0.jar:./Kafka2Datatorrent-1.0-SNAPSHOT.jar:./bval-jsr303-0.5.jar:./httpclient-4.3.5.jar:./jackson-mapper-asl-1.9.2.jar:./kafka_2.10-0.9.0.1.jar:./bval-core-0.5.jar:./minlog-1.2.jar:./jersey-apache-client4-1.9.jar:./malhar-contrib-3.4.0.jar:./metrics-core-2.2.0.jar:./jackson-core-asl-1.9.2.jar:./validation-api-1.1.0.Final.jar:./gson-2.0.jar:./kryo-2.24.0.jar:./netlet-1.2.1.jar:./lz4-1.2.0.jar:./mbassador-1.1.9.jar:./slf4j-api-1.7.5.jar:./kafka-clients-0.9.0.1.jar:./scala-library-2.10.5.jar:./kafka-unit-0.4.jar:./apex-common-3.4.0.jar:./xbean-asm5-shaded-4.3.jar:./jersey-client-1.9.jar:./apex-engine-4.jar:./malhar-kafka-3.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-nfs-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-common-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-common-2.7.0-mapr-1602-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/stax-api-1.0-2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-diagnostic-tools-5.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-client-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-5.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jets3t-0.9.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-framework-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-hbase-5.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hamcrest-core-1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/httpclient-4.2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/json-20080701.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/slf4j-log4j12-1.7.12.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/zookeeper-3.4.5-mapr-1503.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-mapreduce-5.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/activation-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/opt/m
Thanks,
Srinivas
This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the intended
recipient or his or her authorized agent, the reader is hereby notified
that any dissemination, distribution or copying of this e-mail is
prohibited. If you have received this e-mail in error, please notify the
sender by replying to this message and delete this e-mail immediately.
Re: DT Fault Tolerance UHG
Posted by Thomas Weise <th...@apache.org>.
If the tuples processed by the output operator are not of type String, then
the recovery code may fail because it attempts to interpret the messages
that were already stored as String. That's a bug in the operator. The
workaround is to convert the object to String in the upstream operator and
then pass the String to the Kafka output operator.
Thanks,
Thomas
On Thu, Oct 13, 2016 at 10:46 AM, Bandaru, Srinivas <
srinivas.bandaru@optum.com> wrote:
> Hi,
>
> Need some help. While running DT application with “*Kafka”* running into
> issues with application. When monitoring the application, We are observing
> that the *operator getting inactive* and restarts continuously. Could
> you please refer the below log let us know if any configuration need to
> be changed?
>
>
>
>
>
> 2016-10-122016-10-12 16:40:16,835 INFO org.apache.kafka.common.utils.AppInfoParser:
> Kafka version : 0.9.0.1
>
> 2016-10-12 16:40:16,835 INFO org.apache.kafka.common.utils.AppInfoParser:
> Kafka commitId : 23c69d62a0cabf06
>
> 16:40:17,153 INFO org.apache.apex.malhar.kafka.
> KafkaSinglePortExactlyOnceOutputOperator: Rebuild the partial window
> after 6340695403456888950
>
> 2016-10-12 16:40:18,827 ERROR com.datatorrent.stram.engine.StreamingContainer:
> Operator set [OperatorDeployInfo[id=4,name=kafkaOut1,type=GENERIC,checkpoint={57feace00000003b,
> 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=
> inputPort,streamId=updateTopic1,sourceNodeId=3,sourcePortName=outTopic1,
> locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[]]]
> stopped running due to an exception.
>
> java.lang.RuntimeException: Violates Exactly once. Not all the tuples
> received after operator reset.
>
> at org.apache.apex.malhar.kafka.KafkaSinglePortExactlyOnceOutp
> utOperator.endWindow(KafkaSinglePortExactlyOnceOutputOperator.java:174)
>
> at com.datatorrent.stram.engine.GenericNode.processEndWindow(
> GenericNode.java:146)
>
> at com.datatorrent.stram.engine.GenericNode.run(GenericNode.
> java:357)
>
> at com.datatorrent.stram.engine.StreamingContainer$2.run(
> StreamingContainer.java:1407)
>
> 2016-10-12 16:40:18,838 INFO org.apache.kafka.clients.producer.KafkaProducer:
> Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
>
> 2016-10-12 16:40:18,862 INFO com.datatorrent.stram.engine.StreamingContainer:
> Undeploy request: [4]
>
> 2016-10-12 16:40:18,863 INFO com.datatorrent.stram.engine.StreamingContainer:
> Undeploy complete.
>
> __stderr__0__stdout__0,_*container_e31_1476212051326_
> 0045_01_000087¨Oø__dt.log__204352016-10-12 16:51:42,333 INFO
> com.datatorrent.stram.engine.StreamingContainer: Child starting with
> classpath: ./commons-beanutils-1.8.3.jar:./apex-api-3.4.0.jar:./apex-
> bufferserver-3.4.0.jar:./commons-lang3-3.1.jar:./
> httpcore-4.3.2.jar:./snappy-java-1.1.1.7.jar:./zkclient-0.
> 7.jar:./jctools-core-1.1.jar:./jopt-simple-3.2.jar:./apex-
> shaded-ning19-1.0.0.jar:./malhar-library-3.5.0.jar:./
> Kafka2Datatorrent-1.0-SNAPSHOT.jar:./bval-jsr303-0.
> 5.jar:./httpclient-4.3.5.jar:./jackson-mapper-asl-1.9.2.jar:
> ./kafka_2.10-0.9.0.1.jar:./bval-core-0.5.jar:./minlog-1.
> 2.jar:./jersey-apache-client4-1.9.jar:./malhar-contrib-3.4.
> 0.jar:./metrics-core-2.2.0.jar:./jackson-core-asl-1.9.2.
> jar:./validation-api-1.1.0.Final.jar:./gson-2.0.jar:./
> kryo-2.24.0.jar:./netlet-1.2.1.jar:./lz4-1.2.0.jar:./
> mbassador-1.1.9.jar:./slf4j-api-1.7.5.jar:./kafka-clients-
> 0.9.0.1.jar:./scala-library-2.10.5.jar:./kafka-unit-0.4.jar:
> ./apex-common-3.4.0.jar:./xbean-asm5-shaded-4.3.jar:./
> jersey-client-1.9.jar:./apex-engine-4.jar:./malhar-kafka-3.
> 5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop:/opt/
> mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-
> nfs-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/common/hadoop-common-2.7.0-mapr-1602.jar:/
> opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-
> common-2.7.0-mapr-1602-tests.jar:/opt/mapr/hadoop/hadoop-2.
> 7.0/share/hadoop/common/lib/stax-api-1.0-2.jar:/opt/mapr/
> hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-
> diagnostic-tools-5.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.
> 7.0/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/
> opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/
> commons-compress-1.4.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/
> opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/
> curator-client-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/common/lib/maprdb-5.1.0-mapr-tests.jar:/
> opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/
> jets3t-0.9.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/
> hadoop/common/lib/jsp-api-2.1.jar:/opt/mapr/hadoop/hadoop-2.
> 7.0/share/hadoop/common/lib/curator-framework-2.7.1.jar:/
> opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-
> hbase-5.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/common/lib/hamcrest-core-1.3.jar:/opt/
> mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/
> htrace-core-3.1.0-incubating.jar:/opt/mapr/hadoop/hadoop-2.
> 7.0/share/hadoop/common/lib/httpclient-4.2.5.jar:/opt/
> mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/json-
> 20080701.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/
> common/lib/slf4j-log4j12-1.7.12.jar:/opt/mapr/hadoop/
> hadoop-2.7.0/share/hadoop/common/lib/zookeeper-3.4.5-
> mapr-1503.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/
> hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/mapr/
> hadoop/hadoop-2.7.0/share/hadoop/common/lib/netty-3.6.2.
> Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-
> configuration-1.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/common/lib/maprdb-mapreduce-5.1.0-mapr.
> jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/
> activation-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/
> hadoop/common/lib/jetty-util-6.1.26.jar:/opt/mapr/hadoop/
> hadoop-2.7.0/share/hadoop/common/lib/jackson-databind-2.
> 7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/
> common/lib/jackson-core-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.
> 7.0/share/hadoop/common/lib/commons-io-2.4.jar:/opt/mapr/
> hadoop/hadoop-2.7.0/share/hadoop/common/lib/servlet-api-
> 2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/
> common/lib/mapr-tools-5.1.0-mapr-tests.jar:/opt/mapr/
> hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-
> recipes-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/
> hadoop/common/lib/mapr-tools-5.1.0-mapr.jar:/opt/mapr/
> hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-cli-
> 1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/
> common/lib/jackson-mapper-asl-1.9.13.jar:/opt/mapr/hadoop/
> hadoop-2.7.0/share/hadoop/common/lib/hadoop-auth-2.7.0-
> mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/
> hadoop/common/lib/hadoop-azure-2.7.0-mapr-1602.jar:/
> opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/ojai-
> 1.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/
> common/lib/avro-1.7.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/common/lib/log4j-1.2.17.jar:/opt/mapr/hadoop/
> hadoop-2.7.0/share/hadoop/common/lib/maprdb-5.1.0-mapr.
> jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/
> central-logging-5.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.
> 7.0/share/hadoop/common/lib/hadoop-aws-2.7.0-mapr-1602.
> jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/
> jsch-0.1.42.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/
> hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/opt/mapr/hadoop/
> hadoop-2.7.0/share/hadoop/common/lib/commons-math3-3.1.
> 1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/
> snappy-java-1.0.4.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/common/lib/ojai-mapreduce-1.0.jar:/opt/mapr/
> hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-json-
> 1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/
> common/lib/commons-beanutils-1.7.0.jar:/opt/mapr/hadoop/
> hadoop-2.7.0/share/hadoop/common/lib/azure-storage-2.0.
> 0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/
> jettison-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/
> hadoop/common/lib/paranamer-2.3.jar:/opt/mapr/hadoop/hadoop-
> 2.7.0/share/hadoop/common/lib/mapr-hbase-5.1.0-mapr-tests.
> jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/
> commons-beanutils-core-1.8.0.jar:/opt/mapr/hadoop/hadoop-2.
> 7.0/share/hadoop/common/lib/asm-3.2.jar:/opt/mapr/hadoop/
> hadoop-2.7.0/share/hadoop/common/lib/xz-1.0.jar:/opt/
> mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/
> hadoop-annotations-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/
> hadoop-2.7.0/share/hadoop/common/lib/antlr4-runtime-4.5.
> jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/
> aws-java-sdk-1.7.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/common/lib/httpcore-4.2.5.jar:/opt/mapr/
> hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-
> collections-3.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/common/lib/jaxb-api-2.2.2.jar:/opt/mapr/
> hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-
> shell-5.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/common/lib/jersey-server-1.9.jar:/opt/
> mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/
> jackson-xc-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/common/lib/jetty-6.1.26.jar:/opt/mapr/hadoop/
> hadoop-2.7.0/share/hadoop/common/lib/jackson-annotations-2.7.1.jar:/opt/
> mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/
> maprfs-5.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/common/lib/commons-httpclient-3.1.jar:/
> opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-
> java-utils-5.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.
> 7.0/share/hadoop/common/lib/junit-4.11.jar:/opt/mapr/
> hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-
> codec-1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/
> hadoop/common/lib/commons-logging-1.1.3.jar:/opt/mapr/
> hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-
> digester-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/
> hadoop/common/lib/api-util-1.0.0-M20.jar:/opt/mapr/hadoop/
> hadoop-2.7.0/share/hadoop/common/lib/jackson-jaxrs-1.9.
> 13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/
> common/lib/maprdb-mapreduce-5.1.0-mapr-tests.jar:/opt/mapr/
> hadoop/hadoop-2.7.0/share/hadoop/common/lib/java-
> xmlbuilder-0.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/
> hadoop/common/lib/gson-2.2.4.jar:/opt/mapr/hadoop/hadoop-2.
> 7.0/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-
> M15.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/
> common/lib/jersey-core-1.9.jar:/opt/mapr/hadoop/hadoop-2.
> 7.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/opt/
> mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/guava-
> 14.0.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/
> common/lib/xmlenc-0.52.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/opt/
> mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/
> jsr305-3.0.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/
> hadoop/common/lib/commons-net-3.1.jar:/opt/mapr/hadoop/
> hadoop-2.7.0/share/hadoop/common/lib/slf4j-api-1.7.12.
> jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/
> mysql-connector-java-5.1.25.jar:/opt/mapr/hadoop/hadoop-2.
> 7.0/share/hadoop/common/lib/commons-lang-2.6.jar:/opt/
> mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-
> nfs-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/hdfs/hadoop-hdfs-2.7.0-mapr-1602.jar:/opt/mapr/
> hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-2.7.0-
> mapr-1602-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/
> hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/opt/mapr/
> hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xercesImpl-2.
> 9.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/
> hdfs/lib/htrace-core-3.1.0-incubating.jar:/opt/mapr/
> hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/protobuf-java-
> 2.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/
> hdfs/lib/netty-3.6.2.Final.jar:/opt/mapr/hadoop/hadoop-2.
> 7.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/opt/
> mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-
> io-2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/
> hdfs/lib/servlet-api-2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/hdfs/lib/commons-cli-1.2.jar:/opt/mapr/hadoop/
> hadoop-2.7.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.
> 9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/
> hdfs/lib/leveldbjni-all-1.8.jar:/opt/mapr/hadoop/hadoop-2.
> 7.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/opt/mapr/
> hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xml-apis-1.3.
> 04.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/
> hdfs/lib/asm-3.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/hdfs/lib/jersey-server-1.9.jar:/opt/mapr/
> hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jetty-6.1.26.
> jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/
> netty-all-4.0.23.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/hdfs/lib/commons-codec-1.4.jar:/opt/mapr/
> hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-
> logging-1.1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/
> hadoop/hdfs/lib/jersey-core-1.9.jar:/opt/mapr/hadoop/hadoop-
> 2.7.0/share/hadoop/hdfs/lib/guava-14.0.1.jar:/opt/mapr/
> hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-
> daemon-1.0.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/
> hadoop/hdfs/lib/xmlenc-0.52.jar:/opt/mapr/hadoop/hadoop-2.
> 7.0/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/opt/
> mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-
> server-tests-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.
> 7.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.
> 0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/
> hadoop/yarn/hadoop-yarn-server-common-2.7.0-mapr-1602.
> jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-
> yarn-applications-distributedshell-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/
> hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-
> sharedcachemanager-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/
> hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-applications-
> unmanaged-am-launcher-2.7.0-mapr-1602.jar:/opt/mapr/
> hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-
> registry-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/yarn/hadoop-yarn-client-2.7.0-mapr-1602.jar:/
> opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-
> server-nodemanager-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/
> hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-web-
> proxy-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/yarn/hadoop-yarn-api-2.7.0-mapr-1602.jar:/opt/
> mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-
> common-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.0-
> mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/
> hadoop/yarn/lib/stax-api-1.0-2.jar:/opt/mapr/hadoop/hadoop-
> 2.7.0/share/hadoop/yarn/lib/javax.inject-1.jar:/opt/mapr/
> hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-
> compress-1.4.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/
> hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/opt/mapr/
> hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/zookeeper-3.4.
> 5-mapr-1503.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/
> hadoop/yarn/lib/protobuf-java-2.5.0.jar:/opt/mapr/hadoop/
> hadoop-2.7.0/share/hadoop/yarn/lib/netty-3.6.2.Final.
> jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/
> zookeeper-3.4.5-mapr-1503-tests.jar:/opt/mapr/hadoop/
> hadoop-2.7.0/share/hadoop/yarn/lib/activation-1.1.jar:/
> opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/guice-3.
> 0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/
> jetty-util-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/yarn/lib/commons-io-2.4.jar:/opt/mapr/hadoop/
> hadoop-2.7.0/share/hadoop/yarn/lib/servlet-api-2.5.jar:/
> opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-
> cli-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/
> yarn/lib/jackson-mapper-asl-1.9.13.jar:/opt/mapr/hadoop/
> hadoop-2.7.0/share/hadoop/yarn/lib/leveldbjni-all-1.8.
> jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/
> log4j-1.2.17.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/
> hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/opt/mapr/hadoop/
> hadoop-2.7.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/
> opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-
> json-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/
> yarn/lib/jettison-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/yarn/lib/asm-3.2.jar:/opt/mapr/hadoop/hadoop-2.
> 7.0/share/hadoop/yarn/lib/xz-1.0.jar:/opt/mapr/hadoop/
> hadoop-2.7.0/share/hadoop/yarn/lib/guice-servlet-3.0.
> jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/
> jersey-guice-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/yarn/lib/eval-0.5.jar:/opt/mapr/hadoop/hadoop-
> 2.7.0/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:
> /opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jaxb-
> api-2.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/
> hadoop/yarn/lib/jersey-server-1.9.jar:/opt/mapr/hadoop/
> hadoop-2.7.0/share/hadoop/yarn/lib/jackson-xc-1.9.13.
> jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/
> jetty-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/
> hadoop/yarn/lib/commons-codec-1.4.jar:/opt/mapr/hadoop/
> hadoop-2.7.0/share/hadoop/yarn/lib/commons-logging-1.1.
> 3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/
> jackson-jaxrs-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/yarn/lib/jersey-core-1.9.jar:/opt/mapr/hadoop/
> hadoop-2.7.0/share/hadoop/yarn/lib/guava-14.0.1.jar:/
> opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-
> client-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/
> hadoop/yarn/lib/commons-lang-2.6.jar:.
>
> 2016-10-12 16:51:43,750 INFO com.datatorrent.bufferserver.storage.DiskStorage:
> using /opt/mapr/tmp/hadoop-mapr/nm-local-dir/usercache/mapr/
> appcache/application_1476212051326_0045/container_
> e31_1476212051326_0045_01_000087/tmp as the basepath for spooling.
>
> 2016-10-12 16:51:43,755 INFO com.datatorrent.bufferserver.server.Server:
> Server started listening at /0:0:0:0:0:0:0:0:55309
>
> 2016-10-12 16:51:44,796 INFO com.datatorrent.stram.engine.StreamingContainer:
> Deploy request: [OperatorDeployInfo[id=4,name=kafkaOut1,type=GENERIC,checkpoint={57feace00000003b,
> 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=
> inputPort,streamId=updateTopic1,sourceNodeId=3,sourcePortName=outTopic1,
> locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[]]]
>
> 2016-10-12 16:51:44,800 INFO org.apache.hadoop.conf.Configuration.deprecation:
> io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
>
> 2016-10-12 16:51:44,910 INFO org.apache.kafka.clients.producer.ProducerConfig:
> ProducerConfig values:
>
> compression.type = none
>
> metric.reporters = []
>
> metadata.max.age.ms = 300000
>
> metadata.fetch.timeout.ms = 60000
>
> reconnect.backoff.ms = 50
>
> sasl.kerberos.ticket.renew.window.factor = 0.8
>
> bootstrap.servers = [apsrd7102.uhc.com:9092]
>
> retry.backoff.ms = 100
>
> sasl.kerberos.kinit.cmd = /usr/bin/kinit
>
> buffer.memory = 33554432
>
> timeout.ms = 30000
>
> key.serializer = class org.apache.kafka.common.
> serialization.StringSerializer
>
> sasl.kerberos.service.name = null
>
> sasl.kerberos.ticket.renew.jitter = 0.05
>
> ssl.keystore.type = JKS
>
> ssl.trustmanager.algorithm = PKIX
>
> block.on.buffer.full = false
>
> ssl.key.password = null
>
> max.block.ms = 60000
>
> sasl.kerberos.min.time.before.relogin = 60000
>
> connections.max.idle.ms = 540000
>
> ssl.truststore.password = null
>
> max.in.flight.requests.per.connection = 5
>
> metrics.num.samples = 2
>
> client.id =
>
> ssl.endpoint.identification.algorithm = null
>
> ssl.protocol = TLS
>
> request.timeout.ms = 30000
>
> ssl.provider = null
>
> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
>
> acks = all
>
> batch.size = 16384
>
> ssl.keystore.location = null
>
> receive.buffer.bytes = 32768
>
> ssl.cipher.suites = null
>
> ssl.truststore.type = JKS
>
> security.protocol = PLAINTEXT
>
> retries = 2
>
> max.request.size = 1048576
>
> value.serializer = class com.example.datatorrent.
> Application$TenantSerializer
>
> ssl.truststore.location = null
>
> ssl.keystore.password = null
>
> ssl.keymanager.algorithm = SunX509
>
> metrics.sample.window.ms = 30000
>
> partitioner.class = class org.apache.kafka.clients.
> producer.internals.DefaultPartitioner
>
> send.buffer.bytes = 131072
>
> linger.ms = 1
>
>
>
>
>
>
>
> 2016-10-122016-10-12 16:40:16,835 INFO org.apache.kafka.common.utils.AppInfoParser:
> Kafka version : 0.9.0.1
>
> 2016-10-12 16:40:16,835 INFO org.apache.kafka.common.utils.AppInfoParser:
> Kafka commitId : 23c69d62a0cabf06
>
> 16:40:17,153 INFO org.apache.apex.malhar.kafka.
> KafkaSinglePortExactlyOnceOutputOperator: Rebuild the partial window
> after 6340695403456888950
>
> 2016-10-12 16:40:18,827 ERROR com.datatorrent.stram.engine.StreamingContainer:
> Operator set [OperatorDeployInfo[id=4,name=kafkaOut1,type=GENERIC,checkpoint={57feace00000003b,
> 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=
> inputPort,streamId=updateTopic1,sourceNodeId=3,sourcePortName=outTopic1,
> locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[]]]
> stopped running due to an exception.
>
> java.lang.RuntimeException: Violates Exactly once. Not all the tuples
> received after operator reset.
>
> at org.apache.apex.malhar.kafka.KafkaSinglePortExactlyOnceOutp
> utOperator.endWindow(KafkaSinglePortExactlyOnceOutputOperator.java:174)
>
> at com.datatorrent.stram.engine.GenericNode.processEndWindow(
> GenericNode.java:146)
>
> at com.datatorrent.stram.engine.GenericNode.run(GenericNode.
> java:357)
>
> at com.datatorrent.stram.engine.StreamingContainer$2.run(
> StreamingContainer.java:1407)
>
> 2016-10-12 16:40:18,838 INFO org.apache.kafka.clients.producer.KafkaProducer:
> Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
>
> 2016-10-12 16:40:18,862 INFO com.datatorrent.stram.engine.StreamingContainer:
> Undeploy request: [4]
>
> 2016-10-12 16:40:18,863 INFO com.datatorrent.stram.engine.StreamingContainer:
> Undeploy complete.
>
> __stderr__0__stdout__0,_*container_e31_1476212051326_
> 0045_01_000087¨Oø__dt.log__204352016-10-12 16:51:42,333 INFO
> com.datatorrent.stram.engine.StreamingContainer: Child starting with
> classpath: ./commons-beanutils-1.8.3.jar:./apex-api-3.4.0.jar:./apex-
> bufferserver-3.4.0.jar:./commons-lang3-3.1.jar:./
> httpcore-4.3.2.jar:./snappy-java-1.1.1.7.jar:./zkclient-0.
> 7.jar:./jctools-core-1.1.jar:./jopt-simple-3.2.jar:./apex-
> shaded-ning19-1.0.0.jar:./malhar-library-3.5.0.jar:./
> Kafka2Datatorrent-1.0-SNAPSHOT.jar:./bval-jsr303-0.
> 5.jar:./httpclient-4.3.5.jar:./jackson-mapper-asl-1.9.2.jar:
> ./kafka_2.10-0.9.0.1.jar:./bval-core-0.5.jar:./minlog-1.
> 2.jar:./jersey-apache-client4-1.9.jar:./malhar-contrib-3.4.
> 0.jar:./metrics-core-2.2.0.jar:./jackson-core-asl-1.9.2.
> jar:./validation-api-1.1.0.Final.jar:./gson-2.0.jar:./
> kryo-2.24.0.jar:./netlet-1.2.1.jar:./lz4-1.2.0.jar:./
> mbassador-1.1.9.jar:./slf4j-api-1.7.5.jar:./kafka-clients-
> 0.9.0.1.jar:./scala-library-2.10.5.jar:./kafka-unit-0.4.jar:
> ./apex-common-3.4.0.jar:./xbean-asm5-shaded-4.3.jar:./
> jersey-client-1.9.jar:./apex-engine-4.jar:./malhar-kafka-3.
> 5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop:/opt/
> mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-
> nfs-2.7.0-mapr-1602.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/common/hadoop-common-2.7.0-mapr-1602.jar:/
> opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-
> common-2.7.0-mapr-1602-tests.jar:/opt/mapr/hadoop/hadoop-2.
> 7.0/share/hadoop/common/lib/stax-api-1.0-2.jar:/opt/mapr/
> hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-
> diagnostic-tools-5.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.
> 7.0/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/
> opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/
> commons-compress-1.4.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/
> opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/
> curator-client-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/common/lib/maprdb-5.1.0-mapr-tests.jar:/
> opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/
> jets3t-0.9.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/
> hadoop/common/lib/jsp-api-2.1.jar:/opt/mapr/hadoop/hadoop-2.
> 7.0/share/hadoop/common/lib/curator-framework-2.7.1.jar:/
> opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-
> hbase-5.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/common/lib/hamcrest-core-1.3.jar:/opt/
> mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/
> htrace-core-3.1.0-incubating.jar:/opt/mapr/hadoop/hadoop-2.
> 7.0/share/hadoop/common/lib/httpclient-4.2.5.jar:/opt/
> mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/json-
> 20080701.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/
> common/lib/slf4j-log4j12-1.7.12.jar:/opt/mapr/hadoop/
> hadoop-2.7.0/share/hadoop/common/lib/zookeeper-3.4.5-
> mapr-1503.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/
> hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/mapr/
> hadoop/hadoop-2.7.0/share/hadoop/common/lib/netty-3.6.2.
> Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-
> configuration-1.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/
> share/hadoop/common/lib/maprdb-mapreduce-5.1.0-mapr.
> jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/
> activation-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/
> hadoop/common/lib/jetty-util-6.1.26.jar:/opt/m
>
>
>
> Thanks,
>
> Srinivas
>
>
> This e-mail, including attachments, may include confidential and/or
> proprietary information, and may be used only by the person or entity
> to which it is addressed. If the reader of this e-mail is not the intended
> recipient or his or her authorized agent, the reader is hereby notified
> that any dissemination, distribution or copying of this e-mail is
> prohibited. If you have received this e-mail in error, please notify the
> sender by replying to this message and delete this e-mail immediately.
>