[jira] [Created] (KAFKA-14742) Flaky ExactlyOnceSourceIntegrationTest.testConnectorBoundary OOMs

["Greg Harris (Jira)" <ji...@apache.org>]

Greg Harris created KAFKA-14742:
-----------------------------------

             Summary: Flaky ExactlyOnceSourceIntegrationTest.testConnectorBoundary OOMs
                 Key: KAFKA-14742
                 URL: https://issues.apache.org/jira/browse/KAFKA-14742
             Project: Kafka
          Issue Type: Improvement
            Reporter: Greg Harris
            Assignee: Greg Harris


The ExactlyOnceSourceIntegrationTest appears to occasionally throw the following exception in my local test runs:
{noformat}
java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.base/java.util.HashMap.newNode(HashMap.java:1901)
	at java.base/java.util.HashMap.putVal(HashMap.java:629)
	at java.base/java.util.HashMap.put(HashMap.java:610)
	at java.base/java.util.HashSet.add(HashSet.java:221)
	at java.base/java.util.stream.Collectors$$Lambda$6/0x800000011.accept(Unknown Source)
	at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
	at java.base/java.util.stream.LongPipeline$1$1.accept(LongPipeline.java:177)
	at java.base/java.util.stream.Streams$RangeLongSpliterator.forEachRemaining(Streams.java:228)
	at java.base/java.util.Spliterator$OfLong.forEachRemaining(Spliterator.java:775)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
	at org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest.lambda$assertSeqnos$9(ExactlyOnceSourceIntegrationTest.java:964)
	at org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest$$Lambda$2500/0x00000008015a1908.accept(Unknown Source)
	at java.base/java.util.HashMap.forEach(HashMap.java:1421)
	at org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest.assertSeqnos(ExactlyOnceSourceIntegrationTest.java:961)
	at org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest.assertExactlyOnceSeqnos(ExactlyOnceSourceIntegrationTest.java:939)
	at org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest.testIntervalBoundary(ExactlyOnceSourceIntegrationTest.java:358)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100){noformat}
It appears that the data produced by the connectors under test is too large to be asserted on with the current assertions' memory overhead. We should try to optimize the assertions' overhead and or reduce the number of records being asserted on.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)