You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by Vivek Bhide <bh...@gmail.com> on 2017/06/08 23:25:25 UTC
BoundedDedupOperator failing with
java.lang.IllegalArgumentException: bucket conflict
Hi,
I am using the BoundedDedupOperator and with default value of numBuckets
(46340) the container is failing with below bucket conflict exception
2017-06-08 17:52:10,140 INFO stram.StreamingContainerParent
(StreamingContainerParent.java:log(170)) - child msg: Stopped running due to
an exception. java.lang.IllegalArgumentException: bucket conflict -27877
27877
at
org.apache.apex.malhar.lib.state.managed.AbstractManagedStateImpl.handleBucketConflict(AbstractManagedStateImpl.java:320)
at
org.apache.apex.malhar.lib.state.managed.AbstractManagedStateImpl.prepareBucket(AbstractManagedStateImpl.java:269)
at
org.apache.apex.malhar.lib.state.managed.AbstractManagedStateImpl.getValueFromBucketAsync(AbstractManagedStateImpl.java:305)
at
org.apache.apex.malhar.lib.state.managed.ManagedTimeStateImpl.getAsync(ManagedTimeStateImpl.java:76)
at
org.apache.apex.malhar.lib.dedup.BoundedDedupOperator.getAsyncManagedState(BoundedDedupOperator.java:151)
at
org.apache.apex.malhar.lib.dedup.AbstractDeduper.processTuple(AbstractDeduper.java:163)
at
org.apache.apex.malhar.lib.dedup.BoundedDedupOperator$1.process(BoundedDedupOperator.java:91)
at com.datatorrent.api.DefaultInputPort.put(DefaultInputPort.java:79)
at
com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:288)
at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
at
com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1428)
I tried changing the numBucket value to something else but it still fails
with same error. I was planning to use it with just 1 bucket but I think
that will slow the operator too much since the unique keys in our
applications are very high
Has anybody else encountered the same problem?
Regards
Vivek
--
View this message in context: http://apache-apex-users-list.78494.x6.nabble.com/BoundedDedupOperator-failing-with-java-lang-IllegalArgumentException-bucket-conflict-tp1698.html
Sent from the Apache Apex Users list mailing list archive at Nabble.com.
Re: BoundedDedupOperator failing with java.lang.IllegalArgumentException:
bucket conflict
Posted by Bhupesh Chawda <bh...@datatorrent.com>.
Hi Vivek,
The release of the malhar artifacts depends on the community to decide and
whether there are sufficient new features to warrant a release; there is no
planned date as such.
~ Bhupesh
_______________________________________________________
Bhupesh Chawda
E: bhupesh@datatorrent.com | Twitter: @bhupeshsc
www.datatorrent.com | apex.apache.org
On Sun, Jun 18, 2017 at 10:09 PM, Vivek Bhide <bh...@gmail.com> wrote:
> When can we expect 3.8.0 to be published to maven? Latest Malhar version in
> maven is still 3.7.0
>
> Regards
> Vivek
>
>
>
> --
> View this message in context: http://apache-apex-users-list.
> 78494.x6.nabble.com/BoundedDedupOperator-failing-with-java-lang-
> IllegalArgumentException-bucket-conflict-tp1698p1742.html
> Sent from the Apache Apex Users list mailing list archive at Nabble.com.
>
Re: BoundedDedupOperator failing with
java.lang.IllegalArgumentException: bucket conflict
Posted by Vivek Bhide <bh...@gmail.com>.
When can we expect 3.8.0 to be published to maven? Latest Malhar version in
maven is still 3.7.0
Regards
Vivek
--
View this message in context: http://apache-apex-users-list.78494.x6.nabble.com/BoundedDedupOperator-failing-with-java-lang-IllegalArgumentException-bucket-conflict-tp1698p1742.html
Sent from the Apache Apex Users list mailing list archive at Nabble.com.
Re: BoundedDedupOperator failing with
java.lang.IllegalArgumentException: bucket conflict
Posted by Vivek Bhide <bh...@gmail.com>.
Sure. Will check and let you know
Regards
Vivek
--
View this message in context: http://apache-apex-users-list.78494.x6.nabble.com/BoundedDedupOperator-failing-with-java-lang-IllegalArgumentException-bucket-conflict-tp1698p1720.html
Sent from the Apache Apex Users list mailing list archive at Nabble.com.
Re: BoundedDedupOperator failing with java.lang.IllegalArgumentException:
bucket conflict
Posted by Bhupesh Chawda <bh...@datatorrent.com>.
Hi Vivek,
I tried out the BoundedDedupOperator and processed ~ 50k records, seems to
be running okay. Here is the example I tried out: https://github.com/apache
/apex-malhar/tree/master/examples/dedup.
Make sure to replace TimeBasedDedupOperator with BoundedDedupOperator. Also
note that this uses the latest code 3.8.0-SNAPSHOT (master).
Can you check if this helps?
~ Bhupesh
_______________________________________________________
Bhupesh Chawda
E: bhupesh@datatorrent.com | Twitter: @bhupeshsc
www.datatorrent.com | apex.apache.org
On Fri, Jun 9, 2017 at 11:01 PM, Vivek Bhide <bh...@gmail.com> wrote:
> Hi Bhupesh,
>
> I even tried using the TimeBoundedDedupe instead of BoundedDedup and even
> that one fails with exception. In this case, the container starts properly
> but as soon as it tries to process the tuples it fails.
>
> Below are configurations
> ================
>
> <property>
>
> <name>dt.application.DataUsageIngest.operator.dedupeOperator.port.input.
> attr.TUPLE_CLASS</name>
>
> <value>com.tgt.dqs.datausageingest.object.DataSetAttributeWithChecksum</
> value>
> </property>
> <property>
>
> <name>dt.application.DataUsageIngest.operator.dedupeOperator.prop.
> keyExpression</name>
> <value>checksum</value>
> </property>
> <property>
>
> <name>dt.application.DataUsageIngest.operator.dedupeOperator.prop.
> timeExpression</name>
> <value>date.getTime()</value>
> </property>
> <property>
>
> <name>dt.application.DataUsageIngest.operator.dedupeOperator.prop.
> bucketSpan</name>
> <value>1800</value>
> </property>
> <property>
>
> <name>dt.application.DataUsageIngest.operator.dedupeOperator.prop.
> expireBefore</name>
> <value>180000</value>
> </property>
>
> Below are the container logs
> ===================
>
> 2017-06-08 18:59:53,569 INFO util.LoggerUtil
> (LoggerUtil.java:changeLoggersLevel(274)) - changing level of
> com.datatorrent.stram.util.LoggerUtil to INFO
> 2017-06-08 18:59:53,590 INFO engine.StreamingContainer
> (StreamingContainer.java:main(291)) - Child starting with classpath:
> ./kafka-clients-0.9.0.0.jar:./jetty-io-8.1.10.v20130312.jar:
> ./jetty-http-8.1.10.v20130312.jar:./jetty-server-8.1.10.
> v20130312.jar:./activemq-client-5.8.0.jar:./bval-core-
> 0.5.jar:./joda-time-2.9.1.jar:./jetty-servlet-8.1.10.
> v20130312.jar:./mbassador-1.1.9.jar:./jersey-core-1.9.jar:./
> httpcore-4.3.2.jar:./avro-1.7.4.jar:./validation-api-1.1.0.
> Final.jar:./malhar-contrib-3.7.0.jar:./named-regexp-0.2.3.
> jar:./jetty-security-8.1.10.v20130312.jar:./jackson-core-
> asl-1.9.13.jar:./commons-lang3-3.1.jar:./httpcore-4.3.
> 3.jar:./jcip-annotations-1.0.jar:./malhar-hive-3.7.0.jar:./
> rhino-1.7R4.jar:./httpclient-4.3.5.jar:./commons-compress-
> 1.4.1.jar:./jersey-client-1.9.jar:./xbean-asm5-shaded-4.3.
> jar:./jetty-websocket-8.1.10.v20130312.jar:./apex-common-3.
> 7.0-SNAPSHOT.jar:./jackson-annotations-2.7.0.jar:./
> mailapi-1.4.3.jar:./kafka_2.12-0.10.2.0.jar:./bval-jsr303-
> 0.5.jar:./jctools-core-1.1.jar:./netlet-1.3.0.jar:./
> commons-collections-3.2.1.jar:./libthrift-0.9.3.jar:./
> malhar-kafka-3.7.0.jar:./aws-java-sdk-s3-1.10.73.jar:./
> janino-3.0.7.jar:./malhar-library-3.7.0.jar:./minlog-1.
> 2.jar:./hawtbuf-1.9.jar:./datausageingest-1.0-SNAPSHOT.
> jar:./jetty-continuation-8.1.10.v20130312.jar:./jetty-util-
> 8.1.10.v20130312.jar:./jsr305-1.3.9.jar:./lz4-1.2.0.jar:./
> httpclient-4.3.6.jar:./apex-api-3.7.0-SNAPSHOT.jar:./
> log4j-1.2.17.jar:./activation-1.1.jar:./json-schema-core-1.
> 0.2.jar:./scala-library-2.12.1.jar:./jopt-simple-5.0.3.jar:
> ./kryo-2.24.0.jar:./snappy-java-1.0.4.1.jar:./fastutil-7.
> 0.6.jar:./apex-engine.jar:./guava-11.0.2.jar:./adaptor-
> commons-0.0.2-SNAPSHOT.jar:./geronimo-j2ee-management_1.1_
> spec-1.0.1.jar:./jersey-apache-client4-1.9.jar:./
> zookeeper-3.4.9.jar:./jooq-3.6.4.jar:./slf4j-api-1.7.5.jar:
> ./hive-jdbc-2.0.0.jar:./metrics-core-2.2.0.jar:./
> commons-beanutils-1.9.2.jar:./slf4j-log4j12-1.7.21.jar:./
> apex-shaded-ning19-1.0.0.jar:./libphonenumber-5.3.jar:./aws-
> java-sdk-kms-1.10.73.jar:./aws-java-sdk-core-1.10.73.jar:
> ./hive-service-2.0.0.jar:./commons-logging-1.1.1.jar:./
> zkclient-0.10.jar:./jackson-core-2.7.0.jar:./scala-parser-
> combinators_2.12-1.0.4.jar:./jackson-databind-2.5.4.jar:./
> jms-api-1.1-rev-1.jar:./paranamer-2.3.jar:./apex-
> bufferserver-3.7.0-SNAPSHOT.jar:./hive-exec-0.13.1.jar:./
> json-schema-validator-2.0.1.jar:./commons-compiler-3.0.7.
> jar:./javax.mail-1.5.0.jar:./geronimo-jms_1.1_spec-1.1.1.
> jar:./jackson-mapper-asl-1.9.13.jar:./jackson-dataformat-
> cbor-2.5.3.jar:./xz-1.0.jar:/usr/hdp/current/hadoop-client/
> conf:/usr/hdp/current/hadoop-client/hadoop-azure.jar:/usr/
> hdp/current/hadoop-client/hadoop-annotations.jar:/usr/
> hdp/current/hadoop-client/hadoop-nfs-2.7.3.2.5.3.0-37.
> jar:/usr/hdp/current/hadoop-client/hadoop-nfs.jar:/usr/
> hdp/current/hadoop-client/hadoop-annotations-2.7.3.2.5.
> 3.0-37.jar:/usr/hdp/current/hadoop-client/hadoop-azure-2.
> 7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-client/hadoop-
> auth.jar:/usr/hdp/current/hadoop-client/hadoop-auth-2.7.
> 3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-client/hadoop-
> common-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-client/
> hadoop-aws-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-
> client/hadoop-common-tests.jar:/usr/hdp/current/hadoop-
> client/hadoop-aws.jar:/usr/hdp/current/hadoop-client/
> hadoop-common-2.7.3.2.5.3.0-37-tests.jar:/usr/hdp/current/
> hadoop-client/hadoop-common.jar:/usr/hdp/current/hadoop-
> client/lib/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/current/
> hadoop-client/lib/ojdbc6.jar:/usr/hdp/current/hadoop-client/
> lib/asm-3.2.jar:/usr/hdp/current/hadoop-client/lib/
> ranger-yarn-plugin-shim-0.6.0.2.5.3.0-37.jar:/usr/hdp/
> current/hadoop-client/lib/snappy-java-1.0.4.1.jar:/usr/
> hdp/current/hadoop-client/lib/nimbus-jose-jwt-3.9.jar:/usr/
> hdp/current/hadoop-client/lib/curator-framework-2.7.1.jar:/
> usr/hdp/current/hadoop-client/lib/jaxb-impl-2.2.3-1.jar:/
> usr/hdp/current/hadoop-client/lib/commons-codec-1.4.jar:/
> usr/hdp/current/hadoop-client/lib/jsp-api-2.1.jar:/usr/hdp/
> current/hadoop-client/lib/slf4j-log4j12-1.7.10.jar:/usr/
> hdp/current/hadoop-client/lib/stax-api-1.0-2.jar:/usr/hdp/
> current/hadoop-client/lib/jackson-jaxrs-1.9.13.jar:/usr/
> hdp/current/hadoop-client/lib/junit-4.11.jar:/usr/hdp/
> current/hadoop-client/lib/paranamer-2.3.jar:/usr/hdp/
> current/hadoop-client/lib/aws-java-sdk-core-1.10.6.jar:/usr/
> hdp/current/hadoop-client/lib/java-xmlbuilder-0.4.jar:/usr/
> hdp/current/hadoop-client/lib/commons-net-3.1.jar:/usr/hdp/
> current/hadoop-client/lib/jsch-0.1.42.jar:/usr/hdp/
> current/hadoop-client/lib/jackson-databind-2.2.3.jar:/
> usr/hdp/current/hadoop-client/lib/commons-lang-2.6.jar:/usr/
> hdp/current/hadoop-client/lib/apacheds-i18n-2.0.0-M15.jar:/
> usr/hdp/current/hadoop-client/lib/ranger-hdfs-plugin-shim-0.
> 6.0.2.5.3.0-37.jar:/usr/hdp/current/hadoop-client/lib/
> commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-client/
> lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-client/
> lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-
> client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/
> hadoop-client/lib/commons-configuration-1.6.jar:/usr/
> hdp/current/hadoop-client/lib/jackson-annotations-2.2.3.jar:
> /usr/hdp/current/hadoop-client/lib/xz-1.0.jar:/usr/
> hdp/current/hadoop-client/lib/guava-11.0.2.jar:/usr/hdp/
> current/hadoop-client/lib/commons-beanutils-core-1.8.0.
> jar:/usr/hdp/current/hadoop-client/lib/gson-2.2.4.jar:/
> usr/hdp/current/hadoop-client/lib/htrace-core-3.1.0-
> incubating.jar:/usr/hdp/current/hadoop-client/lib/
> commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-client/
> lib/jettison-1.1.jar:/usr/hdp/current/hadoop-client/lib/
> ranger-plugin-classloader-0.6.0.2.5.3.0-37.jar:/usr/hdp/
> current/hadoop-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/
> current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar:/usr/
> hdp/current/hadoop-client/lib/httpcore-4.4.4.jar:/usr/hdp/
> current/hadoop-client/lib/netty-3.6.2.Final.jar:/usr/
> hdp/current/hadoop-client/lib/jersey-json-1.9.jar:/usr/hdp/
> current/hadoop-client/lib/jcip-annotations-1.0.jar:/usr/
> hdp/current/hadoop-client/lib/httpclient-4.5.2.jar:/usr/hdp/
> current/hadoop-client/lib/hamcrest-core-1.3.jar:/usr/
> hdp/current/hadoop-client/lib/curator-recipes-2.7.1.jar:/
> usr/hdp/current/hadoop-client/lib/commons-io-2.4.jar:/usr/
> hdp/current/hadoop-client/lib/commons-compress-1.4.1.jar:/
> usr/hdp/current/hadoop-client/lib/apacheds-kerberos-codec-2.
> 0.0-M15.jar:/usr/hdp/current/hadoop-client/lib/protobuf-
> java-2.5.0.jar:/usr/hdp/current/hadoop-client/lib/
> jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/
> jersey-core-1.9.jar:/usr/hdp/current/hadoop-client/lib/api-
> asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-client/lib/
> jsr305-3.0.0.jar:/usr/hdp/current/hadoop-client/lib/
> xmlenc-0.52.jar:/usr/hdp/current/hadoop-client/lib/
> curator-client-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/
> commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-client/lib/
> jets3t-0.9.0.jar:/usr/hdp/current/hadoop-client/lib/
> jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/
> activation-1.1.jar:/usr/hdp/current/hadoop-client/lib/api-
> util-1.0.0-M20.jar:/usr/hdp/current/hadoop-client/lib/
> azure-keyvault-core-0.8.0.jar:/usr/hdp/current/hadoop-
> client/lib/slf4j-api-1.7.10.jar:/usr/hdp/current/hadoop-
> client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-client/
> lib/commons-lang3-3.4.jar:/usr/hdp/current/hadoop-client/
> lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-client/lib/
> joda-time-2.8.1.jar:/usr/hdp/current/hadoop-client/lib/
> jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/
> lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-client/lib/
> log4j-1.2.17.jar:/usr/hdp/current/hadoop-client/lib/
> commons-digester-1.8.jar:/usr/hdp/current/hadoop-client/lib/
> jersey-server-1.9.jar:/usr/hdp/current/hadoop-client/lib/
> zookeeper-3.4.6.2.5.3.0-37.jar:/usr/hdp/current/hadoop-
> client/lib/azure-storage-4.2.0.jar:/usr/hdp/current/hadoop-
> client/lib/commons-collections-3.2.2.jar:/usr/
> hdp/current/hadoop-client/lib/mockito-all-1.8.5.jar:/usr/
> hdp/current/hadoop-client/lib/json-smart-1.1.1.jar:/usr/hdp/
> current/hadoop-hdfs-client/hadoop-hdfs.jar:/usr/hdp/
> current/hadoop-hdfs-client/hadoop-hdfs-2.7.3.2.5.3.0-37-
> tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-
> hdfs-tests.jar:/usr/hdp/current/hadoop-hdfs-client/
> hadoop-hdfs-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-
> hdfs-client/hadoop-hdfs-nfs-2.7.3.2.5.3.0-37.jar:/usr/hdp/
> current/hadoop-hdfs-client/hadoop-hdfs-nfs.jar:/usr/hdp/
> current/hadoop-hdfs-client/lib/asm-3.2.jar:/usr/hdp/
> current/hadoop-hdfs-client/lib/commons-codec-1.4.jar:/
> usr/hdp/current/hadoop-hdfs-client/lib/commons-lang-2.6.
> jar:/usr/hdp/current/hadoop-hdfs-client/lib/xercesImpl-2.
> 9.1.jar:/usr/hdp/current/hadoop-hdfs-client/lib/
> commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-hdfs-
> client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/
> hadoop-hdfs-client/lib/jackson-core-asl-1.9.13.jar:/
> usr/hdp/current/hadoop-hdfs-client/lib/okio-1.4.0.jar:/
> usr/hdp/current/hadoop-hdfs-client/lib/netty-all-4.0.23.
> Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/guava-
> 11.0.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/htrace-
> core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-hdfs-
> client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-
> hdfs-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-
> hdfs-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/
> hadoop-hdfs-client/lib/leveldbjni-all-1.8.jar:/usr/
> hdp/current/hadoop-hdfs-client/lib/jetty-6.1.26.hwx.
> jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-core-1.
> 9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jsr305-3.0.0.
> jar:/usr/hdp/current/hadoop-hdfs-client/lib/xmlenc-0.52.
> jar:/usr/hdp/current/hadoop-hdfs-client/lib/okhttp-2.4.0.
> jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-cli-1.
> 2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xml-apis-1.3.
> 04.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-
> util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/
> lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-hdfs-
> client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-hdfs-
> client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-
> hdfs-client/lib/commons-daemon-1.0.13.jar:/usr/hdp/
> current/hadoop-yarn-client/hadoop-yarn-server-
> nodemanager-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-
> yarn-client/hadoop-yarn-server-tests.jar:/usr/hdp/
> current/hadoop-yarn-client/hadoop-yarn-common-2.7.3.2.5.
> 3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-
> applicationhistoryservice.jar:/usr/hdp/current/hadoop-yarn-
> client/hadoop-yarn-applications-distributedshell.
> jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-
> server-sharedcachemanager-2.7.3.2.5.3.0-37.jar:/usr/hdp/
> current/hadoop-yarn-client/hadoop-yarn-client-2.7.3.2.5.
> 3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-
> yarn-server-resourcemanager-2.7.3.2.5.3.0-37.jar:/usr/hdp/
> current/hadoop-yarn-client/hadoop-yarn-registry-2.7.3.2.
> 5.3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-
> yarn-server-common-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/
> hadoop-yarn-client/hadoop-yarn-server-web-proxy.jar:/
> usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-
> timeline-pluginstorage-2.7.3.2.5.3.0-37.jar:/usr/hdp/
> current/hadoop-yarn-client/hadoop-yarn-api-2.7.3.2.5.3.0-
> 37.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-
> yarn-common.jar:/usr/hdp/current/hadoop-yarn-client/
> hadoop-yarn-server-timeline-pluginstorage.jar:/usr/hdp/
> current/hadoop-yarn-client/hadoop-yarn-server-common.jar:
> /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-
> nodemanager.jar:/usr/hdp/current/hadoop-yarn-client/
> hadoop-yarn-api.jar:/usr/hdp/current/hadoop-yarn-client/
> hadoop-yarn-server-sharedcachemanager.jar:/usr/hdp/current/hadoop-yarn-
> client/hadoop-yarn-registry.jar:/usr/hdp/current/hadoop-
> yarn-client/hadoop-yarn-server-tests-2.7.3.2.5.3.0-37.
> jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-
> applications-unmanaged-am-launcher.jar:/usr/hdp/current/
> hadoop-yarn-client/hadoop-yarn-client.jar:/usr/hdp/
> current/hadoop-yarn-client/hadoop-yarn-server-web-proxy-
> 2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/
> hadoop-yarn-applications-distributedshell-2.7.3.2.5.3.
> 0-37.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-
> yarn-server-resourcemanager.jar:/usr/hdp/current/hadoop-
> yarn-client/hadoop-yarn-server-applicationhistoryservice-2.7.
> 3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/
> hadoop-yarn-applications-unmanaged-am-launcher-2.7.3.2.
> 5.3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/lib/asm-3.
> 2.jar:/usr/hdp/current/hadoop-yarn-client/lib/snappy-java-1.
> 0.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/nimbus-
> jose-jwt-3.9.jar:/usr/hdp/current/hadoop-yarn-client/
> lib/guice-3.0.jar:/usr/hdp/current/hadoop-yarn-client/
> lib/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-
> yarn-client/lib/jersey-client-1.9.jar:/usr/hdp/current/
> hadoop-yarn-client/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/
> current/hadoop-yarn-client/lib/commons-codec-1.4.jar:/
> usr/hdp/current/hadoop-yarn-client/lib/jsp-api-2.1.jar:/
> usr/hdp/current/hadoop-yarn-client/lib/stax-api-1.0-2.jar:
> /usr/hdp/current/hadoop-yarn-client/lib/jackson-jaxrs-1.9.
> 13.jar:/usr/hdp/current/hadoop-yarn-client/lib/paranamer-2.3.jar:/usr/hdp/
> current/hadoop-yarn-client/lib/java-xmlbuilder-0.4.jar:/
> usr/hdp/current/hadoop-yarn-client/lib/commons-net-3.1.
> jar:/usr/hdp/current/hadoop-yarn-client/lib/jsch-0.1.42.
> jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-
> databind-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/
> lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-yarn-
> client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/
> hadoop-yarn-client/lib/commons-logging-1.1.3.jar:/
> usr/hdp/current/hadoop-yarn-client/lib/jackson-core-2.2.3.
> jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-
> mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-
> client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/
> hadoop-yarn-client/lib/metrics-core-3.0.1.jar:/usr/
> hdp/current/hadoop-yarn-client/lib/commons-configuration-1.6.jar:/usr/
> hdp/current/hadoop-yarn-client/lib/jackson-annotations-2.2.3.jar:/usr/
> hdp/current/hadoop-yarn-client/lib/xz-1.0.jar:/usr/
> hdp/current/hadoop-yarn-client/lib/guava-11.0.2.jar:/
> usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-
> core-1.8.0.jar:/usr/hdp/current/hadoop-yarn-client/
> lib/gson-2.2.4.jar:/usr/hdp/current/hadoop-yarn-client/
> lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-yarn-client/
> lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-
> yarn-client/lib/jettison-1.1.jar:/usr/hdp/current/hadoop-
> yarn-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-
> yarn-client/lib/httpcore-4.4.4.jar:/usr/hdp/current/hadoop-
> yarn-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/
> hadoop-yarn-client/lib/jersey-json-1.9.jar:/usr/hdp/current/
> hadoop-yarn-client/lib/jcip-annotations-1.0.jar:/usr/hdp/
> current/hadoop-yarn-client/lib/javax.inject-1.jar:/usr/
> hdp/current/hadoop-yarn-client/lib/httpclient-4.5.2.
> jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-
> recipes-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/
> lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-yarn-
> client/lib/objenesis-2.1.jar:/usr/hdp/current/hadoop-yarn-
> client/lib/commons-compress-1.4.1.jar:/usr/hdp/current/
> hadoop-yarn-client/lib/apacheds-kerberos-codec-2.0.0-
> M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/
> protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-yarn-
> client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-
> yarn-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/
> hadoop-yarn-client/lib/javassist-3.18.1-GA.jar:/usr/
> hdp/current/hadoop-yarn-client/lib/jersey-core-1.9.
> jar:/usr/hdp/current/hadoop-yarn-client/lib/api-asn1-api-
> 1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/
> lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-yarn-client/
> lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-yarn-client/
> lib/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-yarn-
> client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-
> yarn-client/lib/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-
> yarn-client/lib/jackson-xc-1.9.13.jar:/usr/hdp/current/
> hadoop-yarn-client/lib/guice-servlet-3.0.jar:/usr/hdp/
> current/hadoop-yarn-client/lib/activation-1.1.jar:/usr/
> hdp/current/hadoop-yarn-client/lib/api-util-1.0.0-M20.
> jar:/usr/hdp/current/hadoop-yarn-client/lib/azure-
> keyvault-core-0.8.0.jar:/usr/hdp/current/hadoop-yarn-
> client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-yarn-
> client/lib/jersey-guice-1.9.jar:/usr/hdp/current/hadoop-
> yarn-client/lib/commons-lang3-3.4.jar:/usr/hdp/current/
> hadoop-yarn-client/lib/commons-cli-1.2.jar:/usr/hdp/
> current/hadoop-yarn-client/lib/fst-2.24.jar:/usr/hdp/
> current/hadoop-yarn-client/lib/jetty-util-6.1.26.hwx.jar:
> /usr/hdp/current/hadoop-yarn-client/lib/servlet-api-2.5.
> jar:/usr/hdp/current/hadoop-yarn-client/lib/log4j-1.2.17.
> jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-
> digester-1.8.jar:/usr/hdp/current/hadoop-yarn-client/
> lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-yarn-
> client/lib/aopalliance-1.0.jar:/usr/hdp/current/hadoop-
> yarn-client/lib/zookeeper-3.4.6.2.5.3.0-37-tests.jar:/usr/
> hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.
> 5.3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/lib/azure-
> storage-4.2.0.jar:/usr/hdp/current/hadoop-yarn-client/
> lib/commons-collections-3.2.2.jar:/usr/hdp/current/hadoop-
> yarn-client/lib/json-smart-1.1.1.jar:.
> 2017-06-08 18:59:54,005 WARN util.NativeCodeLoader
> (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop
> library
> for your platform... using builtin-java classes where applicable
> 2017-06-08 18:59:55,009 WARN shortcircuit.DomainSocketFactory
> (DomainSocketFactory.java:<init>(117)) - The short-circuit local reads
> feature cannot be used because libhadoop cannot be loaded.
> 2017-06-08 18:59:55,435 INFO storage.DiskStorage
> (DiskStorage.java:<init>(53)) - using
> /grid/10/hadoop/yarn/local/usercache/SVDATHDP/appcache/
> application_1496931225841_2096/container_e3093_
> 1496931225841_2096_01_000002/tmp
> as the basepath for spooling.
> 2017-06-08 18:59:55,438 INFO server.Server (Server.java:registered(112)) -
> Server started listening at /0.0.0.0:41023
> 2017-06-08 18:59:56,471 INFO engine.StreamingContainer
> (StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending
> request.
> 2017-06-08 18:59:56,976 INFO engine.StreamingContainer
> (StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending
> request.
> 2017-06-08 18:59:57,482 INFO engine.StreamingContainer
> (StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending
> request.
> 2017-06-08 18:59:57,987 INFO engine.StreamingContainer
> (StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending
> request.
> 2017-06-08 18:59:58,491 INFO engine.StreamingContainer
> (StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending
> request.
> 2017-06-08 18:59:58,996 INFO engine.StreamingContainer
> (StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending
> request.
> 2017-06-08 18:59:59,500 INFO engine.StreamingContainer
> (StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending
> request.
> 2017-06-08 19:00:00,004 INFO engine.StreamingContainer
> (StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending
> request.
> 2017-06-08 19:00:00,414 INFO server.Server (Server.java:onMessage(599)) -
> Received subscriber request: SubscribeRequestTuple{version=1.0,
> identifier=tcp://d-2vwlw12.target.com:41023/5.unique.1,
> windowId=ffffffffffffffff, type=uniquMessages/6.inputPort,
> upstreamIdentifier=5.unique.1, mask=0, partitions=null, bufferSize=1024}
> 2017-06-08 19:00:00,528 INFO engine.StreamingContainer
> (StreamingContainer.java:processHeartbeatResponse(825)) - Deploy request:
> [OperatorDeployInfo[id=5,name=dedupeOperator,type=GENERIC,
> checkpoint={ffffffffffffffff,
> 0,
> 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=
> checkDuplicates,sourceNodeId=3,sourcePortName=dedupePort,
> locality=<null>,partitionMask=0,partitionKeys=<null>]],
> outputs=[OperatorDeployInfo.OutputDeployInfo[portName=
> unique,streamId=uniquMessages,bufferServer=d-2vwlw12.target.com]]]]
> 2017-06-08 19:00:00,664 INFO server.Server (Server.java:onMessage(555)) -
> Received publisher request: PublishRequestTuple{version=1.0,
> identifier=5.unique.1, windowId=ffffffffffffffff}
> 2017-06-08 19:00:03,105 INFO util.AsyncFSStorageAgent
> (AsyncFSStorageAgent.java:save(91)) - using
> /grid/10/hadoop/yarn/local/usercache/SVDATHDP/appcache/
> application_1496931225841_2096/container_e3093_
> 1496931225841_2096_01_000002/tmp/chkp4165334308239559126
> as the basepath for checkpointing.
> 2017-06-08 19:05:57,806 ERROR engine.StreamingContainer
> (StreamingContainer.java:run(1456)) - Operator set
> [OperatorDeployInfo[id=5,name=dedupeOperator,type=GENERIC,
> checkpoint={ffffffffffffffff,
> 0,
> 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=
> checkDuplicates,sourceNodeId=3,sourcePortName=dedupePort,
> locality=<null>,partitionMask=0,partitionKeys=<null>]],
> outputs=[OperatorDeployInfo.OutputDeployInfo[portName=
> unique,streamId=uniquMessages,bufferServer=d-2vwlw12.target.com]]]]
> stopped running due to an exception.
> java.lang.IllegalArgumentException: Invalid slice: offset=0, length=0
> array.length=0
> at com.datatorrent.netlet.util.Slice.<init>(Slice.java:43)
> at
> org.apache.apex.malhar.lib.utils.serde.BufferSlice.<init>
> (BufferSlice.java:48)
> at
> org.apache.apex.malhar.lib.utils.serde.BufferSlice.<init>
> (BufferSlice.java:58)
> at
> org.apache.apex.malhar.lib.utils.serde.SliceUtils.
> toBufferSlice(SliceUtils.java:111)
> at
> org.apache.apex.malhar.lib.state.managed.Bucket$
> DefaultBucket.put(Bucket.java:421)
> at
> org.apache.apex.malhar.lib.state.managed.AbstractManagedStateImpl.
> putInBucket(AbstractManagedStateImpl.java:286)
> at
> org.apache.apex.malhar.lib.state.managed.ManagedTimeUnifiedStateImpl.put(
> ManagedTimeUnifiedStateImpl.java:72)
> at
> org.apache.apex.malhar.lib.dedup.TimeBasedDedupOperator.putManagedState(
> TimeBasedDedupOperator.java:189)
> at
> org.apache.apex.malhar.lib.dedup.AbstractDeduper.processAuxiliary(
> AbstractDeduper.java:316)
> at
> org.apache.apex.malhar.lib.dedup.AbstractDeduper.
> endWindow(AbstractDeduper.java:337)
> at
> com.datatorrent.stram.engine.GenericNode.processEndWindow(
> GenericNode.java:153)
> at com.datatorrent.stram.engine.GenericNode.run(GenericNode.
> java:397)
> at
> com.datatorrent.stram.engine.StreamingContainer$2.run(
> StreamingContainer.java:1428)
> 2017-06-08 19:05:58,082 INFO engine.StreamingContainer
> (StreamingContainer.java:processHeartbeatResponse(808)) - Undeploy
> request:
> [5]
> 2017-06-08 19:05:58,084 INFO engine.StreamingContainer
> (StreamingContainer.java:undeploy(561)) - Undeploy complete.
> 2017-06-08 19:05:58,085 INFO server.Server (Server.java:run(414)) -
> Removing ln
> LogicalNode@411ca26bidentifier=tcp://d-2vwlw12.target.com:41023/5.unique.1
> ,
> upstream=5.unique.1, group=uniquMessages/6.inputPort, partitions=[],
> iterator=com.datatorrent.bufferserver.internal.DataList$DataListIterator@
> 64b843d2{da=com.datatorrent.bufferserver.internal.DataList$Block@2c9a38f1{
> identifier=5.unique.1,
> data=67108864, readingOffset=5661, writingOffset=6062,
> starting_window=5939e4e6000002cf, ending_window=5939e4e6000002ff,
> refCount=2, uniqueIdentifier=0, next=null, future=null}}} from dl
> com.datatorrent.bufferserver.internal.DataList@e9ca021 {5.unique.1}
>
> Regards
> Vivek
>
>
>
> --
> View this message in context: http://apache-apex-users-list.
> 78494.x6.nabble.com/BoundedDedupOperator-failing-with-java-lang-
> IllegalArgumentException-bucket-conflict-tp1698p1703.html
> Sent from the Apache Apex Users list mailing list archive at Nabble.com.
>
Re: BoundedDedupOperator failing with
java.lang.IllegalArgumentException: bucket conflict
Posted by Vivek Bhide <bh...@gmail.com>.
Hi Bhupesh,
I even tried using the TimeBoundedDedupe instead of BoundedDedup and even
that one fails with exception. In this case, the container starts properly
but as soon as it tries to process the tuples it fails.
Below are configurations
================
<property>
<name>dt.application.DataUsageIngest.operator.dedupeOperator.port.input.attr.TUPLE_CLASS</name>
<value>com.tgt.dqs.datausageingest.object.DataSetAttributeWithChecksum</value>
</property>
<property>
<name>dt.application.DataUsageIngest.operator.dedupeOperator.prop.keyExpression</name>
<value>checksum</value>
</property>
<property>
<name>dt.application.DataUsageIngest.operator.dedupeOperator.prop.timeExpression</name>
<value>date.getTime()</value>
</property>
<property>
<name>dt.application.DataUsageIngest.operator.dedupeOperator.prop.bucketSpan</name>
<value>1800</value>
</property>
<property>
<name>dt.application.DataUsageIngest.operator.dedupeOperator.prop.expireBefore</name>
<value>180000</value>
</property>
Below are the container logs
===================
2017-06-08 18:59:53,569 INFO util.LoggerUtil
(LoggerUtil.java:changeLoggersLevel(274)) - changing level of
com.datatorrent.stram.util.LoggerUtil to INFO
2017-06-08 18:59:53,590 INFO engine.StreamingContainer
(StreamingContainer.java:main(291)) - Child starting with classpath:
./kafka-clients-0.9.0.0.jar:./jetty-io-8.1.10.v20130312.jar:./jetty-http-8.1.10.v20130312.jar:./jetty-server-8.1.10.v20130312.jar:./activemq-client-5.8.0.jar:./bval-core-0.5.jar:./joda-time-2.9.1.jar:./jetty-servlet-8.1.10.v20130312.jar:./mbassador-1.1.9.jar:./jersey-core-1.9.jar:./httpcore-4.3.2.jar:./avro-1.7.4.jar:./validation-api-1.1.0.Final.jar:./malhar-contrib-3.7.0.jar:./named-regexp-0.2.3.jar:./jetty-security-8.1.10.v20130312.jar:./jackson-core-asl-1.9.13.jar:./commons-lang3-3.1.jar:./httpcore-4.3.3.jar:./jcip-annotations-1.0.jar:./malhar-hive-3.7.0.jar:./rhino-1.7R4.jar:./httpclient-4.3.5.jar:./commons-compress-1.4.1.jar:./jersey-client-1.9.jar:./xbean-asm5-shaded-4.3.jar:./jetty-websocket-8.1.10.v20130312.jar:./apex-common-3.7.0-SNAPSHOT.jar:./jackson-annotations-2.7.0.jar:./mailapi-1.4.3.jar:./kafka_2.12-0.10.2.0.jar:./bval-jsr303-0.5.jar:./jctools-core-1.1.jar:./netlet-1.3.0.jar:./commons-collections-3.2.1.jar:./libthrift-0.9.3.jar:./malhar-kafka-3.7.0.jar:./aws-java-sdk-s3-1.10.73.jar:./janino-3.0.7.jar:./malhar-library-3.7.0.jar:./minlog-1.2.jar:./hawtbuf-1.9.jar:./datausageingest-1.0-SNAPSHOT.jar:./jetty-continuation-8.1.10.v20130312.jar:./jetty-util-8.1.10.v20130312.jar:./jsr305-1.3.9.jar:./lz4-1.2.0.jar:./httpclient-4.3.6.jar:./apex-api-3.7.0-SNAPSHOT.jar:./log4j-1.2.17.jar:./activation-1.1.jar:./json-schema-core-1.0.2.jar:./scala-library-2.12.1.jar:./jopt-simple-5.0.3.jar:./kryo-2.24.0.jar:./snappy-java-1.0.4.1.jar:./fastutil-7.0.6.jar:./apex-engine.jar:./guava-11.0.2.jar:./adaptor-commons-0.0.2-SNAPSHOT.jar:./geronimo-j2ee-management_1.1_spec-1.0.1.jar:./jersey-apache-client4-1.9.jar:./zookeeper-3.4.9.jar:./jooq-3.6.4.jar:./slf4j-api-1.7.5.jar:./hive-jdbc-2.0.0.jar:./metrics-core-2.2.0.jar:./commons-beanutils-1.9.2.jar:./slf4j-log4j12-1.7.21.jar:./apex-shaded-ning19-1.0.0.jar:./libphonenumber-5.3.jar:./aws-java-sdk-kms-1.10.73.jar:./aws-java-sdk-core-1.10.73.jar:./hive-service-2.0.0.jar:./commons-logging-1.1.1.jar:./zkclient-0.10.jar:./jackson-core-2.7.0.jar:./scala-parser-combinators_2.12-1.0.4.jar:./jackson-databind-2.5.4.jar:./jms-api-1.1-rev-1.jar:./paranamer-2.3.jar:./apex-bufferserver-3.7.0-SNAPSHOT.jar:./hive-exec-0.13.1.jar:./json-schema-validator-2.0.1.jar:./commons-compiler-3.0.7.jar:./javax.mail-1.5.0.jar:./geronimo-jms_1.1_spec-1.1.1.jar:./jackson-mapper-asl-1.9.13.jar:./jackson-dataformat-cbor-2.5.3.jar:./xz-1.0.jar:/usr/hdp/current/hadoop-client/conf:/usr/hdp/current/hadoop-client/hadoop-azure.jar:/usr/hdp/current/hadoop-client/hadoop-annotations.jar:/usr/hdp/current/hadoop-client/hadoop-nfs-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-client/hadoop-nfs.jar:/usr/hdp/current/hadoop-client/hadoop-annotations-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-client/hadoop-azure-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-client/hadoop-auth.jar:/usr/hdp/current/hadoop-client/hadoop-auth-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-client/hadoop-common-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-client/hadoop-aws-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-client/hadoop-common-tests.jar:/usr/hdp/current/hadoop-client/hadoop-aws.jar:/usr/hdp/current/hadoop-client/hadoop-common-2.7.3.2.5.3.0-37-tests.jar:/usr/hdp/current/hadoop-client/hadoop-common.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/ojdbc6.jar:/usr/hdp/current/hadoop-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-client/lib/ranger-yarn-plugin-shim-0.6.0.2.5.3.0-37.jar:/usr/hdp/current/hadoop-client/lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hadoop-client/lib/nimbus-jose-jwt-3.9.jar:/usr/hdp/current/hadoop-client/lib/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hadoop-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-client/lib/jsp-api-2.1.jar:/usr/hdp/current/hadoop-client/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/current/hadoop-client/lib/stax-api-1.0-2.jar:/usr/hdp/current/hadoop-client/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/junit-4.11.jar:/usr/hdp/current/hadoop-client/lib/paranamer-2.3.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hadoop-client/lib/commons-net-3.1.jar:/usr/hdp/current/hadoop-client/lib/jsch-0.1.42.jar:/usr/hdp/current/hadoop-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hadoop-client/lib/ranger-hdfs-plugin-shim-0.6.0.2.5.3.0-37.jar:/usr/hdp/current/hadoop-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/commons-configuration-1.6.jar:/usr/hdp/current/hadoop-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/xz-1.0.jar:/usr/hdp/current/hadoop-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-client/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hadoop-client/lib/gson-2.2.4.jar:/usr/hdp/current/hadoop-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-client/lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-client/lib/jettison-1.1.jar:/usr/hdp/current/hadoop-client/lib/ranger-plugin-classloader-0.6.0.2.5.3.0-37.jar:/usr/hdp/current/hadoop-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/httpcore-4.4.4.jar:/usr/hdp/current/hadoop-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-client/lib/jersey-json-1.9.jar:/usr/hdp/current/hadoop-client/lib/jcip-annotations-1.0.jar:/usr/hdp/current/hadoop-client/lib/httpclient-4.5.2.jar:/usr/hdp/current/hadoop-client/lib/hamcrest-core-1.3.jar:/usr/hdp/current/hadoop-client/lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-client/lib/commons-compress-1.4.1.jar:/usr/hdp/current/hadoop-client/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hadoop-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-client/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-client/lib/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-client/lib/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-client/lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/activation-1.1.jar:/usr/hdp/current/hadoop-client/lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hadoop-client/lib/azure-keyvault-core-0.8.0.jar:/usr/hdp/current/hadoop-client/lib/slf4j-api-1.7.10.jar:/usr/hdp/current/hadoop-client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-client/lib/commons-lang3-3.4.jar:/usr/hdp/current/hadoop-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-client/lib/joda-time-2.8.1.jar:/usr/hdp/current/hadoop-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-client/lib/commons-digester-1.8.jar:/usr/hdp/current/hadoop-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-client/lib/zookeeper-3.4.6.2.5.3.0-37.jar:/usr/hdp/current/hadoop-client/lib/azure-storage-4.2.0.jar:/usr/hdp/current/hadoop-client/lib/commons-collections-3.2.2.jar:/usr/hdp/current/hadoop-client/lib/mockito-all-1.8.5.jar:/usr/hdp/current/hadoop-client/lib/json-smart-1.1.1.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-2.7.3.2.5.3.0-37-tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-nfs-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-nfs.jar:/usr/hdp/current/hadoop-hdfs-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xercesImpl-2.9.1.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/okio-1.4.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/netty-all-4.0.23.Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-hdfs-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-hdfs-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-hdfs-client/lib/okhttp-2.4.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xml-apis-1.3.04.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-hdfs-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-daemon-1.0.13.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-nodemanager-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-tests.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-applicationhistoryservice.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-sharedcachemanager-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-resourcemanager-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-registry-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-timeline-pluginstorage-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-timeline-pluginstorage.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-nodemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-sharedcachemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-registry.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-tests-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-resourcemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-applicationhistoryservice-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-unmanaged-am-launcher-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/nimbus-jose-jwt-3.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/guice-3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-client-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsp-api-2.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/stax-api-1.0-2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/paranamer-2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-net-3.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsch-0.1.42.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-yarn-client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/metrics-core-3.0.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-configuration-1.6.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/xz-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/gson-2.2.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jettison-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/httpcore-4.4.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-json-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/jcip-annotations-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/javax.inject-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/httpclient-4.5.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/objenesis-2.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-compress-1.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/javassist-3.18.1-GA.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/guice-servlet-3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/activation-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/lib/azure-keyvault-core-0.8.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-guice-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-lang3-3.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/fst-2.24.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-digester-1.8.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/aopalliance-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.5.3.0-37-tests.jar:/usr/hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.5.3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/lib/azure-storage-4.2.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-collections-3.2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/json-smart-1.1.1.jar:.
2017-06-08 18:59:54,005 WARN util.NativeCodeLoader
(NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
2017-06-08 18:59:55,009 WARN shortcircuit.DomainSocketFactory
(DomainSocketFactory.java:<init>(117)) - The short-circuit local reads
feature cannot be used because libhadoop cannot be loaded.
2017-06-08 18:59:55,435 INFO storage.DiskStorage
(DiskStorage.java:<init>(53)) - using
/grid/10/hadoop/yarn/local/usercache/SVDATHDP/appcache/application_1496931225841_2096/container_e3093_1496931225841_2096_01_000002/tmp
as the basepath for spooling.
2017-06-08 18:59:55,438 INFO server.Server (Server.java:registered(112)) -
Server started listening at /0.0.0.0:41023
2017-06-08 18:59:56,471 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 18:59:56,976 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 18:59:57,482 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 18:59:57,987 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 18:59:58,491 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 18:59:58,996 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 18:59:59,500 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 19:00:00,004 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 19:00:00,414 INFO server.Server (Server.java:onMessage(599)) -
Received subscriber request: SubscribeRequestTuple{version=1.0,
identifier=tcp://d-2vwlw12.target.com:41023/5.unique.1,
windowId=ffffffffffffffff, type=uniquMessages/6.inputPort,
upstreamIdentifier=5.unique.1, mask=0, partitions=null, bufferSize=1024}
2017-06-08 19:00:00,528 INFO engine.StreamingContainer
(StreamingContainer.java:processHeartbeatResponse(825)) - Deploy request:
[OperatorDeployInfo[id=5,name=dedupeOperator,type=GENERIC,checkpoint={ffffffffffffffff,
0,
0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=checkDuplicates,sourceNodeId=3,sourcePortName=dedupePort,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=unique,streamId=uniquMessages,bufferServer=d-2vwlw12.target.com]]]]
2017-06-08 19:00:00,664 INFO server.Server (Server.java:onMessage(555)) -
Received publisher request: PublishRequestTuple{version=1.0,
identifier=5.unique.1, windowId=ffffffffffffffff}
2017-06-08 19:00:03,105 INFO util.AsyncFSStorageAgent
(AsyncFSStorageAgent.java:save(91)) - using
/grid/10/hadoop/yarn/local/usercache/SVDATHDP/appcache/application_1496931225841_2096/container_e3093_1496931225841_2096_01_000002/tmp/chkp4165334308239559126
as the basepath for checkpointing.
2017-06-08 19:05:57,806 ERROR engine.StreamingContainer
(StreamingContainer.java:run(1456)) - Operator set
[OperatorDeployInfo[id=5,name=dedupeOperator,type=GENERIC,checkpoint={ffffffffffffffff,
0,
0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=checkDuplicates,sourceNodeId=3,sourcePortName=dedupePort,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=unique,streamId=uniquMessages,bufferServer=d-2vwlw12.target.com]]]]
stopped running due to an exception.
java.lang.IllegalArgumentException: Invalid slice: offset=0, length=0
array.length=0
at com.datatorrent.netlet.util.Slice.<init>(Slice.java:43)
at
org.apache.apex.malhar.lib.utils.serde.BufferSlice.<init>(BufferSlice.java:48)
at
org.apache.apex.malhar.lib.utils.serde.BufferSlice.<init>(BufferSlice.java:58)
at
org.apache.apex.malhar.lib.utils.serde.SliceUtils.toBufferSlice(SliceUtils.java:111)
at
org.apache.apex.malhar.lib.state.managed.Bucket$DefaultBucket.put(Bucket.java:421)
at
org.apache.apex.malhar.lib.state.managed.AbstractManagedStateImpl.putInBucket(AbstractManagedStateImpl.java:286)
at
org.apache.apex.malhar.lib.state.managed.ManagedTimeUnifiedStateImpl.put(ManagedTimeUnifiedStateImpl.java:72)
at
org.apache.apex.malhar.lib.dedup.TimeBasedDedupOperator.putManagedState(TimeBasedDedupOperator.java:189)
at
org.apache.apex.malhar.lib.dedup.AbstractDeduper.processAuxiliary(AbstractDeduper.java:316)
at
org.apache.apex.malhar.lib.dedup.AbstractDeduper.endWindow(AbstractDeduper.java:337)
at
com.datatorrent.stram.engine.GenericNode.processEndWindow(GenericNode.java:153)
at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:397)
at
com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1428)
2017-06-08 19:05:58,082 INFO engine.StreamingContainer
(StreamingContainer.java:processHeartbeatResponse(808)) - Undeploy request:
[5]
2017-06-08 19:05:58,084 INFO engine.StreamingContainer
(StreamingContainer.java:undeploy(561)) - Undeploy complete.
2017-06-08 19:05:58,085 INFO server.Server (Server.java:run(414)) -
Removing ln
LogicalNode@411ca26bidentifier=tcp://d-2vwlw12.target.com:41023/5.unique.1,
upstream=5.unique.1, group=uniquMessages/6.inputPort, partitions=[],
iterator=com.datatorrent.bufferserver.internal.DataList$DataListIterator@64b843d2{da=com.datatorrent.bufferserver.internal.DataList$Block@2c9a38f1{identifier=5.unique.1,
data=67108864, readingOffset=5661, writingOffset=6062,
starting_window=5939e4e6000002cf, ending_window=5939e4e6000002ff,
refCount=2, uniqueIdentifier=0, next=null, future=null}}} from dl
com.datatorrent.bufferserver.internal.DataList@e9ca021 {5.unique.1}
Regards
Vivek
--
View this message in context: http://apache-apex-users-list.78494.x6.nabble.com/BoundedDedupOperator-failing-with-java-lang-IllegalArgumentException-bucket-conflict-tp1698p1703.html
Sent from the Apache Apex Users list mailing list archive at Nabble.com.
Re: BoundedDedupOperator failing with
java.lang.IllegalArgumentException: bucket conflict
Posted by Vivek Bhide <bh...@gmail.com>.
Hi Bhupesh,
The exception occurred immediately. Rather operator didn't even initialized
completely and failed before that causing all further operators to stuck in
pending deploy state. Regarding properties file, as i said, there was no
configuration with numBucket in config file and only config for dedup
operator was below
<property>
<name>dt.application.DataUsageIngest.operator.dedupeOperator.port.input.attr.TUPLE_CLASS</name>
<value>com.tgt.dqs.datausageingest.object.DataSetAttributeWithChecksum</value>
</property>
<property>
<name>dt.application.DataUsageIngest.operator.dedupeOperator.prop.keyExpression</name>
<value>checksum</value>
</property>
Below are the full container logs for dedup operator
2017-06-08 17:52:03,491 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 17:52:03,996 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 17:52:04,500 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 17:52:05,004 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 17:52:05,512 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 17:52:06,016 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 17:52:06,519 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 17:52:07,023 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 17:52:07,527 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 17:52:08,030 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 17:52:08,535 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 17:52:09,039 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 17:52:09,340 INFO server.Server (Server.java:onMessage(599)) -
Received subscriber request: SubscribeRequestTuple{version=1.0,
identifier=tcp://brdn1251.target.com:38212/5.unique.1,
windowId=ffffffffffffffff, type=uniquMessages/6.inputPort,
upstreamIdentifier=5.unique.1, mask=0, partitions=null, bufferSize=1024}
2017-06-08 17:52:09,558 INFO engine.StreamingContainer
(StreamingContainer.java:processHeartbeatResponse(825)) - Deploy request:
[OperatorDeployInfo[id=5,name=dedupeOperator,type=GENERIC,checkpoint={ffffffffffffffff,
0,
0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=checkDuplicates,sourceNodeId=3,sourcePortName=dedupePort,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=unique,streamId=uniquMessages,bufferServer=brdn1251.target.com]]]]
2017-06-08 17:52:09,672 INFO server.Server (Server.java:onMessage(555)) -
Received publisher request: PublishRequestTuple{version=1.0,
identifier=5.unique.1, windowId=ffffffffffffffff}
2017-06-08 17:52:10,128 ERROR engine.StreamingContainer
(StreamingContainer.java:run(1456)) - Operator set
[OperatorDeployInfo[id=5,name=dedupeOperator,type=GENERIC,checkpoint={ffffffffffffffff,
0,
0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=checkDuplicates,sourceNodeId=3,sourcePortName=dedupePort,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=unique,streamId=uniquMessages,bufferServer=brdn1251.target.com]]]]
stopped running due to an exception.
java.lang.IllegalArgumentException: bucket conflict -27877 27877
at
org.apache.apex.malhar.lib.state.managed.AbstractManagedStateImpl.handleBucketConflict(AbstractManagedStateImpl.java:320)
at
org.apache.apex.malhar.lib.state.managed.AbstractManagedStateImpl.prepareBucket(AbstractManagedStateImpl.java:269)
at
org.apache.apex.malhar.lib.state.managed.AbstractManagedStateImpl.getValueFromBucketAsync(AbstractManagedStateImpl.java:305)
at
org.apache.apex.malhar.lib.state.managed.ManagedTimeStateImpl.getAsync(ManagedTimeStateImpl.java:76)
at
org.apache.apex.malhar.lib.dedup.BoundedDedupOperator.getAsyncManagedState(BoundedDedupOperator.java:151)
at
org.apache.apex.malhar.lib.dedup.AbstractDeduper.processTuple(AbstractDeduper.java:163)
at
org.apache.apex.malhar.lib.dedup.BoundedDedupOperator$1.process(BoundedDedupOperator.java:91)
at com.datatorrent.api.DefaultInputPort.put(DefaultInputPort.java:79)
at
com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:288)
at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
at
com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1428)
2017-06-08 17:52:11,691 INFO engine.StreamingContainer
(StreamingContainer.java:processHeartbeatResponse(808)) - Undeploy request:
[5]
2017-06-08 17:52:11,692 INFO engine.StreamingContainer
(StreamingContainer.java:undeploy(561)) - Undeploy complete.
2017-06-08 17:52:11,694 INFO server.Server (Server.java:run(414)) -
Removing ln
LogicalNode@3f062c75identifier=tcp://brdn1251.target.com:38212/5.unique.1,
upstream=5.unique.1, group=uniquMessages/6.inputPort, partitions=[],
iterator=com.datatorrent.bufferserver.internal.DataList$DataListIterator@7e693f46{da=com.datatorrent.bufferserver.internal.DataList$Block@137d497f{identifier=5.unique.1,
data=67108864, readingOffset=0, writingOffset=306,
starting_window=5939d50100000001, ending_window=5939d50100000032,
refCount=2, uniqueIdentifier=0, next=null, future=null}}} from dl
com.datatorrent.bufferserver.internal.DataList@799564b7 {5.unique.1}
Regards
Vivek
--
View this message in context: http://apache-apex-users-list.78494.x6.nabble.com/BoundedDedupOperator-failing-with-java-lang-IllegalArgumentException-bucket-conflict-tp1698p1702.html
Sent from the Apache Apex Users list mailing list archive at Nabble.com.
Re: BoundedDedupOperator failing with java.lang.IllegalArgumentException:
bucket conflict
Posted by Bhupesh Chawda <bh...@datatorrent.com>.
Hi Vivek,
Did the exception occur immediately, or after some records were processed?
Can you send the full container log for this operator.
Please also share the properties file that you used for configuration.
~ Bhupesh
_______________________________________________________
Bhupesh Chawda
E: bhupesh@datatorrent.com | Twitter: @bhupeshsc
www.datatorrent.com | apex.apache.org
On Fri, Jun 9, 2017 at 4:55 AM, Vivek Bhide <bh...@gmail.com> wrote:
> Hi,
>
> I am using the BoundedDedupOperator and with default value of numBuckets
> (46340) the container is failing with below bucket conflict exception
>
> 2017-06-08 17:52:10,140 INFO stram.StreamingContainerParent
> (StreamingContainerParent.java:log(170)) - child msg: Stopped running due
> to
> an exception. java.lang.IllegalArgumentException: bucket conflict -27877
> 27877
> at
> org.apache.apex.malhar.lib.state.managed.AbstractManagedStateImpl.
> handleBucketConflict(AbstractManagedStateImpl.java:320)
> at
> org.apache.apex.malhar.lib.state.managed.AbstractManagedStateImpl.
> prepareBucket(AbstractManagedStateImpl.java:269)
> at
> org.apache.apex.malhar.lib.state.managed.AbstractManagedStateImpl.
> getValueFromBucketAsync(AbstractManagedStateImpl.java:305)
> at
> org.apache.apex.malhar.lib.state.managed.ManagedTimeStateImpl.getAsync(
> ManagedTimeStateImpl.java:76)
> at
> org.apache.apex.malhar.lib.dedup.BoundedDedupOperator.
> getAsyncManagedState(BoundedDedupOperator.java:151)
> at
> org.apache.apex.malhar.lib.dedup.AbstractDeduper.
> processTuple(AbstractDeduper.java:163)
> at
> org.apache.apex.malhar.lib.dedup.BoundedDedupOperator$1.
> process(BoundedDedupOperator.java:91)
> at com.datatorrent.api.DefaultInputPort.put(
> DefaultInputPort.java:79)
> at
> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(
> BufferServerSubscriber.java:288)
> at com.datatorrent.stram.engine.GenericNode.run(GenericNode.
> java:269)
> at
> com.datatorrent.stram.engine.StreamingContainer$2.run(
> StreamingContainer.java:1428)
>
>
> I tried changing the numBucket value to something else but it still fails
> with same error. I was planning to use it with just 1 bucket but I think
> that will slow the operator too much since the unique keys in our
> applications are very high
>
> Has anybody else encountered the same problem?
>
> Regards
> Vivek
>
>
>
> --
> View this message in context: http://apache-apex-users-list.
> 78494.x6.nabble.com/BoundedDedupOperator-failing-with-java-lang-
> IllegalArgumentException-bucket-conflict-tp1698.html
> Sent from the Apache Apex Users list mailing list archive at Nabble.com.
>