You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Katarzyna Kucharczyk <ka...@gmail.com> on 2019/01/10 13:55:13 UTC
Load testing on DirectRunner
Hi Everyone,
My name is Kasia and I contribute to Beam's tests. Currently, I am working
with Łukasz Gajowy on load tests and we created Jenkins configuration to
run Synthetic Sources test on DirectRunner. It was decided to generate 1
000 000 000 records (bytes) for a small suite (details you can find in this
proposal [1] ). Running this test on the Beam’s Jenkins is causing runtime
exception with the message:
"java.lang.OutOfMemoryError: GC overhead limit exceeded".
Of course, this is not a surprise since it's a lot of data. That's why I am
asking for your advice/opinion:
Do you think if this test should be smaller? On the other hand, if it's
going to be smaller would it be still worth testing as a load test?
Maybe it would be better to wait for the UniversalLocalRunner instead and
use it while it's there? What is the status of ULR? Do you know if the ULR
will replace DirectRunner?
I created an issue [2] with details of this problem where you can find the
link to the example of a failing job.
Thanks,
Kasia
[1] https://s.apache.org/load-test-basic-operations
<https://s.apache.org/load-test-basic-operations>
[2] https://issues.apache.org/jira/browse/BEAM-6351
Re: Load testing on DirectRunner
Posted by Rui Wang <ru...@google.com>.
Agree with Reuven.
It depends on the purpose of running load test on direct runner. As a
runner for testing, direct runner might not need load testing. However, if
running load test on direct runner is used for verifying if load testing
work, then reducing the size of test data definitely works.
-Rui
On Thu, Jan 10, 2019 at 10:10 AM Reuven Lax <re...@google.com> wrote:
> The Direct Runner as currently implemented is purposely inefficient. It
> was designed for testing, and therefore does many things that are meant to
> expose bugs in user pipelines (e.g. randomly sorting PCollections,
> serializing/deserializing every element, etc.). So it's not surprising that
> it doesn't behave well under load tests.
>
> Reuven
>
> On Thu, Jan 10, 2019 at 5:55 AM Katarzyna Kucharczyk <
> ka.kucharczyk@gmail.com> wrote:
>
>> Hi Everyone,
>>
>> My name is Kasia and I contribute to Beam's tests. Currently, I am
>> working with Łukasz Gajowy on load tests and we created Jenkins
>> configuration to run Synthetic Sources test on DirectRunner. It was decided
>> to generate 1 000 000 000 records (bytes) for a small suite (details you
>> can find in this proposal [1] ). Running this test on the Beam’s Jenkins is
>> causing runtime exception with the message:
>> "java.lang.OutOfMemoryError: GC overhead limit exceeded".
>>
>> Of course, this is not a surprise since it's a lot of data. That's why I
>> am asking for your advice/opinion:
>> Do you think if this test should be smaller? On the other hand, if it's
>> going to be smaller would it be still worth testing as a load test?
>> Maybe it would be better to wait for the UniversalLocalRunner instead and
>> use it while it's there? What is the status of ULR? Do you know if the ULR
>> will replace DirectRunner?
>>
>> I created an issue [2] with details of this problem where you can find
>> the link to the example of a failing job.
>>
>> Thanks,
>> Kasia
>>
>> [1] https://s.apache.org/load-test-basic-operations
>> <https://s.apache.org/load-test-basic-operations>
>> [2] https://issues.apache.org/jira/browse/BEAM-6351
>>
>
Re: Load testing on DirectRunner
Posted by Łukasz Gajowy <lg...@apache.org>.
Thanks all. We won't run full-size load tests on Direct runner. I defined a
separate smoke job with "tiny" versions of load tests for each runner
(well, Direct and Dataflow for now, but we can extend this later with other
runners). Feel free to comment: https://github.com/apache/beam/pull/7497
Maybe it's a good idea to run a "smoke" job with such tests post-commit
(same as NEXMark does). WDYT?
Łukasz
PS: There's also a separate matter of how frequently we should run the
full-fledged load tests on Jenkins but let me address this later in a
separate thread when we have more data on the time/worker/space overhead of
tests. Currently, load tests can be phrase triggered only.
czw., 10 sty 2019 o 19:14 Andrew Pilloud <ap...@google.com> napisał(a):
> My default advice here is to use the Direct Runner for small smoke tests,
> and use the Flink LocalRunner for larger datasets that can still be run
> locally. As Reuven points out, the Direct Runner is more of a validation
> test itself, it does many things designed to test pipelines for the worst
> combinations of conditions they may encounter on other runners.
>
> Andrew
>
> On Thu, Jan 10, 2019 at 10:10 AM Reuven Lax <re...@google.com> wrote:
>
>> The Direct Runner as currently implemented is purposely inefficient. It
>> was designed for testing, and therefore does many things that are meant to
>> expose bugs in user pipelines (e.g. randomly sorting PCollections,
>> serializing/deserializing every element, etc.). So it's not surprising that
>> it doesn't behave well under load tests.
>>
>> Reuven
>>
>> On Thu, Jan 10, 2019 at 5:55 AM Katarzyna Kucharczyk <
>> ka.kucharczyk@gmail.com> wrote:
>>
>>> Hi Everyone,
>>>
>>> My name is Kasia and I contribute to Beam's tests. Currently, I am
>>> working with Łukasz Gajowy on load tests and we created Jenkins
>>> configuration to run Synthetic Sources test on DirectRunner. It was decided
>>> to generate 1 000 000 000 records (bytes) for a small suite (details you
>>> can find in this proposal [1] ). Running this test on the Beam’s Jenkins is
>>> causing runtime exception with the message:
>>> "java.lang.OutOfMemoryError: GC overhead limit exceeded".
>>>
>>> Of course, this is not a surprise since it's a lot of data. That's why I
>>> am asking for your advice/opinion:
>>> Do you think if this test should be smaller? On the other hand, if it's
>>> going to be smaller would it be still worth testing as a load test?
>>> Maybe it would be better to wait for the UniversalLocalRunner instead
>>> and use it while it's there? What is the status of ULR? Do you know if the
>>> ULR will replace DirectRunner?
>>>
>>> I created an issue [2] with details of this problem where you can find
>>> the link to the example of a failing job.
>>>
>>> Thanks,
>>> Kasia
>>>
>>> [1] https://s.apache.org/load-test-basic-operations
>>> <https://s.apache.org/load-test-basic-operations>
>>> [2] https://issues.apache.org/jira/browse/BEAM-6351
>>>
>>
Re: Load testing on DirectRunner
Posted by Andrew Pilloud <ap...@google.com>.
My default advice here is to use the Direct Runner for small smoke tests,
and use the Flink LocalRunner for larger datasets that can still be run
locally. As Reuven points out, the Direct Runner is more of a validation
test itself, it does many things designed to test pipelines for the worst
combinations of conditions they may encounter on other runners.
Andrew
On Thu, Jan 10, 2019 at 10:10 AM Reuven Lax <re...@google.com> wrote:
> The Direct Runner as currently implemented is purposely inefficient. It
> was designed for testing, and therefore does many things that are meant to
> expose bugs in user pipelines (e.g. randomly sorting PCollections,
> serializing/deserializing every element, etc.). So it's not surprising that
> it doesn't behave well under load tests.
>
> Reuven
>
> On Thu, Jan 10, 2019 at 5:55 AM Katarzyna Kucharczyk <
> ka.kucharczyk@gmail.com> wrote:
>
>> Hi Everyone,
>>
>> My name is Kasia and I contribute to Beam's tests. Currently, I am
>> working with Łukasz Gajowy on load tests and we created Jenkins
>> configuration to run Synthetic Sources test on DirectRunner. It was decided
>> to generate 1 000 000 000 records (bytes) for a small suite (details you
>> can find in this proposal [1] ). Running this test on the Beam’s Jenkins is
>> causing runtime exception with the message:
>> "java.lang.OutOfMemoryError: GC overhead limit exceeded".
>>
>> Of course, this is not a surprise since it's a lot of data. That's why I
>> am asking for your advice/opinion:
>> Do you think if this test should be smaller? On the other hand, if it's
>> going to be smaller would it be still worth testing as a load test?
>> Maybe it would be better to wait for the UniversalLocalRunner instead and
>> use it while it's there? What is the status of ULR? Do you know if the ULR
>> will replace DirectRunner?
>>
>> I created an issue [2] with details of this problem where you can find
>> the link to the example of a failing job.
>>
>> Thanks,
>> Kasia
>>
>> [1] https://s.apache.org/load-test-basic-operations
>> <https://s.apache.org/load-test-basic-operations>
>> [2] https://issues.apache.org/jira/browse/BEAM-6351
>>
>
Re: Load testing on DirectRunner
Posted by Reuven Lax <re...@google.com>.
The Direct Runner as currently implemented is purposely inefficient. It was
designed for testing, and therefore does many things that are meant to
expose bugs in user pipelines (e.g. randomly sorting PCollections,
serializing/deserializing every element, etc.). So it's not surprising that
it doesn't behave well under load tests.
Reuven
On Thu, Jan 10, 2019 at 5:55 AM Katarzyna Kucharczyk <
ka.kucharczyk@gmail.com> wrote:
> Hi Everyone,
>
> My name is Kasia and I contribute to Beam's tests. Currently, I am working
> with Łukasz Gajowy on load tests and we created Jenkins configuration to
> run Synthetic Sources test on DirectRunner. It was decided to generate 1
> 000 000 000 records (bytes) for a small suite (details you can find in this
> proposal [1] ). Running this test on the Beam’s Jenkins is causing runtime
> exception with the message:
> "java.lang.OutOfMemoryError: GC overhead limit exceeded".
>
> Of course, this is not a surprise since it's a lot of data. That's why I
> am asking for your advice/opinion:
> Do you think if this test should be smaller? On the other hand, if it's
> going to be smaller would it be still worth testing as a load test?
> Maybe it would be better to wait for the UniversalLocalRunner instead and
> use it while it's there? What is the status of ULR? Do you know if the ULR
> will replace DirectRunner?
>
> I created an issue [2] with details of this problem where you can find the
> link to the example of a failing job.
>
> Thanks,
> Kasia
>
> [1] https://s.apache.org/load-test-basic-operations
> <https://s.apache.org/load-test-basic-operations>
> [2] https://issues.apache.org/jira/browse/BEAM-6351
>