You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by Vivek Bhide <vi...@target.com> on 2018/03/14 17:02:44 UTC
KafkaSinglePortExactlyOnceOutputOperator throwing exception about
violation about Exactly once
Hi,
I was testing the KafkaSinglePortExactlyOnceOutputOperator and I see below
error in case of operator fails. This is not reproducible every time but
occurs more than 50% of the time. I tried testing it with single operator
instance, with multiple partitions and parallel partitions but i couldn't
get the pattern since failure doesn't happen always.
Is there anything I missing out during operator setup or cleanup
Regards
Vivek
--
Sent from: http://apache-apex-users-list.78494.x6.nabble.com/
Re: KafkaSinglePortExactlyOnceOutputOperator throwing exception
about violation about Exactly once
Posted by Vivek Bhide <vi...@target.com>.
Thanks Sandesh.. It helped
Regards
Vivek
--
Sent from: http://apache-apex-users-list.78494.x6.nabble.com/
Re: KafkaSinglePortExactlyOnceOutputOperator throwing exception about
violation about Exactly once
Posted by Sandesh Hegde <sa...@datatorrent.com>.
1. Upstream needs to be idempotent, if not then during recovery you will
see that exception.
2. If you are using a custom tuple, please make sure that it HashCode &
Equals methods are properly implemented.
On Wed, Mar 14, 2018 at 10:09 AM Vivek Bhide <vi...@target.com> wrote:
> Exception stack trace is below
>
> 2018-03-13 17:00:53,219 INFO
> com.datatorrent.stram.StreamingContainerManager: Container
> container_1520983362676_0002_01_000007 buffer server: 80e65028f6a8:60970
> 2018-03-13 17:00:54,811 INFO
> com.datatorrent.stram.StreamingContainerParent:
> child msg: Stopped running due to an exception. java.lang.RuntimeException:
> Violates Exactly once. Not all the tuples received after operator reset.
> at
>
> org.apache.apex.malhar.kafka.KafkaSinglePortExactlyOnceOutputOperator.endWindow(KafkaSinglePortExactlyOnceOutputOperator.java:190)
> at
>
> com.datatorrent.stram.engine.GenericNode.processEndWindow(GenericNode.java:153)
> at
> com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:397)
> at
>
> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1428)
> context:
>
> PTContainer[id=3(container_1520983362676_0002_01_000007),state=ACTIVE,operators=[PTOperator[id=3,name=kafkaOutputOperator,state=ACTIVE]]]
>
>
>
>
>
> --
> Sent from: http://apache-apex-users-list.78494.x6.nabble.com/
>
Re: KafkaSinglePortExactlyOnceOutputOperator throwing exception
about violation about Exactly once
Posted by Vivek Bhide <vi...@target.com>.
Exception stack trace is below
2018-03-13 17:00:53,219 INFO
com.datatorrent.stram.StreamingContainerManager: Container
container_1520983362676_0002_01_000007 buffer server: 80e65028f6a8:60970
2018-03-13 17:00:54,811 INFO com.datatorrent.stram.StreamingContainerParent:
child msg: Stopped running due to an exception. java.lang.RuntimeException:
Violates Exactly once. Not all the tuples received after operator reset.
at
org.apache.apex.malhar.kafka.KafkaSinglePortExactlyOnceOutputOperator.endWindow(KafkaSinglePortExactlyOnceOutputOperator.java:190)
at
com.datatorrent.stram.engine.GenericNode.processEndWindow(GenericNode.java:153)
at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:397)
at
com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1428)
context:
PTContainer[id=3(container_1520983362676_0002_01_000007),state=ACTIVE,operators=[PTOperator[id=3,name=kafkaOutputOperator,state=ACTIVE]]]
--
Sent from: http://apache-apex-users-list.78494.x6.nabble.com/