[jira] [Commented] (FLINK-11397) Speed up initialization of AbstractStreamOperatorTestHarness

["Piotr Nowojski (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-11397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16750256#comment-16750256 ] 

Piotr Nowojski commented on FLINK-11397:
----------------------------------------

[~Tison] thanks for reporting the issue. Can you tell how have you investigated it? Don't get me wrong, I'm a big opponent of using mockito for things like that and it would be very valuable on it's own to replace this {{mockTask = mock(StreamTask.class)}} with a proper mock or some dummy implementation, but I couldn't fully reproduce your findings.

I wrote a simple micro benchmark for both {{mock(StreamTask.class)}} and {{new OneInputStreamOperatorTestHarness()}} constructor and it seemed like first one takes under 0.01ms to completed. Second one is significantly slower, somewhere between 0.1 and 1ms, but I couldn't get the more precise number nor I couldn't investigate/profile the code what's taking so much time because apparently {{new OneInputStreamOperatorTestHarness()}} is leaking some resources and I was getting:

{code:java}
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:717)
	at org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync.<init>(IOManagerAsync.java:88)
	at org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync.<init>(IOManagerAsync.java:69)
	at org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync.<init>(IOManagerAsync.java:60)
	at org.apache.flink.runtime.operators.testutils.MockEnvironment.<init>(MockEnvironment.java:144)
	at org.apache.flink.runtime.operators.testutils.MockEnvironmentBuilder.build(MockEnvironmentBuilder.java:126)
	at org.apache.flink.streaming.util.AbstractStreamOperatorTestHarness.<init>(AbstractStreamOperatorTestHarness.java:164)
	at org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness.<init>(OneInputStreamOperatorTestHarness.java:92)
	at org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness.<init>(OneInputStreamOperatorTestHarness.java:83)
	at org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness.<init>(OneInputStreamOperatorTestHarness.java:75)
{code}

Regardless of that, 1ms is still pretty far away from "a few seconds".

But as I wrote above - even ignoring the speed up I would more then happy to review/merge the code that gets rid of this mockito mock :)

> Speed up initialization of AbstractStreamOperatorTestHarness
> ------------------------------------------------------------
>
>                 Key: FLINK-11397
>                 URL: https://issues.apache.org/jira/browse/FLINK-11397
>             Project: Flink
>          Issue Type: Improvement
>          Components: Tests
>    Affects Versions: 1.8.0
>            Reporter: TisonKun
>            Priority: Major
>
> Currently Kafka connector tests are unbearably slow, which is tracked by FLINK-10603. With investigation, the construction and initialization of {{AbstractStreamOperatorTestHarness}} is quite slow.
> When walk down the code, it amazed me that {{mockTask = mock(StreamTask.class);}} cost a few of second to finish. If we can introduce a test class instead of mock framework, the situation might be loosen.
> cc [~Zentol] [~pnowojski]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)