You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Minwoo Kang <it...@gmail.com> on 2023/02/06 04:13:58 UTC

Why can not change THRIFT_CHUNK_SIZE_BYTES in StormSubmitter?

Hello, Users.

When I submit storm topology, Occasionally it takes a long time.
So I investigated that, and I found THRIFT_CHUNK_SIZE_BYTES[1] is too small.

However, That value can not be modified because it is the final value.
It can be changed (STORM-241[2]) for the first time. But 70174d[3] commit
changed chunk size to private constant.

I wonder why the user cannot change THRIFT_CHUNK_SIZE_BYTES.

Thanks.

[1]:
https://github.com/apache/storm/blob/bf29d1cc9914d4fe596b5e65532322e3dfd3e4ff/storm-client/src/jvm/org/apache/storm/StormSubmitter.java#L62
[2]: https://issues.apache.org/jira/browse/STORM-241
[3]:
https://github.com/apache/storm/commit/70174deaa758fc1e3f3a881445e8750ae7218b3d

Re: Why can not change THRIFT_CHUNK_SIZE_BYTES in StormSubmitter?

Posted by Minwoo Kang <it...@gmail.com>.
Thank you for your reply.

I also upload a jar fast in the 1.X cluster.
In the 1.X cluster is not a problem.

However 2.X cluster is too slow to upload a jar.
I do not understand why the 2.X cluster is slower than the 1.X cluster.

2023년 2월 9일 (목) 오전 4:14, Rui Abreu <ru...@gmail.com>님이 작성:

> I can upload topologies bigger than 90 MB in around 30 seconds in a Storm
> 1.X cluster.
> I'm using a large value for *nimbus.thrift.max_buffer_size* and I have
> also increased *nimbus.thrift.threads. *Don't know if this helps the
> upload process or how slow your upload is.
>
> On Tue, 7 Feb 2023 at 00:35, Minwoo Kang <it...@gmail.com>
> wrote:
>
>> Thank you for your reply.
>>
>> It is about 60 MB.
>> The size is not huge. But the upload speed is too slow.
>>
>> No, I did not change "nimbus.thrift.max_buffer_size".
>> The "getMaxBufferSize" method uses a "nimbus.thrift.max_buffer_size".
>> So I thought the "getMaxBufferSize" method only impacts a size check in
>>  "stringLengthLimit_" and "maxLength" in the thrift frame.
>>
>> There are two upload processes when submitting a topology.
>> first: client -> nimbus
>> second: nimbus -> supervisor
>>
>> The second(nimbus -> supervisor) is also slow. However, the upload speed
>> is faster after I increase "storm.blobstore.inputstream.buffer.size.bytes"
>> to 4MB.
>>
>> I changed the code to increase the "THRIFT_CHUNK_SIZE_BYTES" value.
>> client upload speed is faster after I increase "THRIFT_CHUNK_SIZE_BYTES"
>> to 4MB.
>>
>> So I thought maybe the client should change the "THRIFT_CHUNK_SIZE_BYTES"
>> like nimbus ("storm.blobstore.inputstream.buffer.size.bytes" value).
>>
>> Thanks.
>>
>> 2023년 2월 6일 (월) 오후 9:51, Rui Abreu <ru...@gmail.com>님이 작성:
>>
>>> How big the uber jar that you are submitting?
>>> Have you tried tweaking the Nimbus side property?
>>> https://github.com/apache/storm/blob/master/conf/defaults.yaml#L67
>>>
>>> On Mon, 6 Feb 2023 at 04:14, Minwoo Kang <it...@gmail.com>
>>> wrote:
>>>
>>>> Hello, Users.
>>>>
>>>> When I submit storm topology, Occasionally it takes a long time.
>>>> So I investigated that, and I found THRIFT_CHUNK_SIZE_BYTES[1] is too
>>>> small.
>>>>
>>>> However, That value can not be modified because it is the final value.
>>>> It can be changed (STORM-241[2]) for the first time. But 70174d[3]
>>>> commit changed chunk size to private constant.
>>>>
>>>> I wonder why the user cannot change THRIFT_CHUNK_SIZE_BYTES.
>>>>
>>>> Thanks.
>>>>
>>>> [1]:
>>>> https://github.com/apache/storm/blob/bf29d1cc9914d4fe596b5e65532322e3dfd3e4ff/storm-client/src/jvm/org/apache/storm/StormSubmitter.java#L62
>>>> [2]: https://issues.apache.org/jira/browse/STORM-241
>>>> [3]:
>>>> https://github.com/apache/storm/commit/70174deaa758fc1e3f3a881445e8750ae7218b3d
>>>>
>>>

Re: Why can not change THRIFT_CHUNK_SIZE_BYTES in StormSubmitter?

Posted by Rui Abreu <ru...@gmail.com>.
I can upload topologies bigger than 90 MB in around 30 seconds in a Storm
1.X cluster.
I'm using a large value for *nimbus.thrift.max_buffer_size* and I have also
increased *nimbus.thrift.threads. *Don't know if this helps the upload
process or how slow your upload is.

On Tue, 7 Feb 2023 at 00:35, Minwoo Kang <it...@gmail.com> wrote:

> Thank you for your reply.
>
> It is about 60 MB.
> The size is not huge. But the upload speed is too slow.
>
> No, I did not change "nimbus.thrift.max_buffer_size".
> The "getMaxBufferSize" method uses a "nimbus.thrift.max_buffer_size".
> So I thought the "getMaxBufferSize" method only impacts a size check in
>  "stringLengthLimit_" and "maxLength" in the thrift frame.
>
> There are two upload processes when submitting a topology.
> first: client -> nimbus
> second: nimbus -> supervisor
>
> The second(nimbus -> supervisor) is also slow. However, the upload speed
> is faster after I increase "storm.blobstore.inputstream.buffer.size.bytes"
> to 4MB.
>
> I changed the code to increase the "THRIFT_CHUNK_SIZE_BYTES" value.
> client upload speed is faster after I increase "THRIFT_CHUNK_SIZE_BYTES"
> to 4MB.
>
> So I thought maybe the client should change the "THRIFT_CHUNK_SIZE_BYTES"
> like nimbus ("storm.blobstore.inputstream.buffer.size.bytes" value).
>
> Thanks.
>
> 2023년 2월 6일 (월) 오후 9:51, Rui Abreu <ru...@gmail.com>님이 작성:
>
>> How big the uber jar that you are submitting?
>> Have you tried tweaking the Nimbus side property?
>> https://github.com/apache/storm/blob/master/conf/defaults.yaml#L67
>>
>> On Mon, 6 Feb 2023 at 04:14, Minwoo Kang <it...@gmail.com>
>> wrote:
>>
>>> Hello, Users.
>>>
>>> When I submit storm topology, Occasionally it takes a long time.
>>> So I investigated that, and I found THRIFT_CHUNK_SIZE_BYTES[1] is too
>>> small.
>>>
>>> However, That value can not be modified because it is the final value.
>>> It can be changed (STORM-241[2]) for the first time. But 70174d[3]
>>> commit changed chunk size to private constant.
>>>
>>> I wonder why the user cannot change THRIFT_CHUNK_SIZE_BYTES.
>>>
>>> Thanks.
>>>
>>> [1]:
>>> https://github.com/apache/storm/blob/bf29d1cc9914d4fe596b5e65532322e3dfd3e4ff/storm-client/src/jvm/org/apache/storm/StormSubmitter.java#L62
>>> [2]: https://issues.apache.org/jira/browse/STORM-241
>>> [3]:
>>> https://github.com/apache/storm/commit/70174deaa758fc1e3f3a881445e8750ae7218b3d
>>>
>>

Re: Why can not change THRIFT_CHUNK_SIZE_BYTES in StormSubmitter?

Posted by Minwoo Kang <it...@gmail.com>.
Thank you for your reply.

It is about 60 MB.
The size is not huge. But the upload speed is too slow.

No, I did not change "nimbus.thrift.max_buffer_size".
The "getMaxBufferSize" method uses a "nimbus.thrift.max_buffer_size".
So I thought the "getMaxBufferSize" method only impacts a size check in
 "stringLengthLimit_" and "maxLength" in the thrift frame.

There are two upload processes when submitting a topology.
first: client -> nimbus
second: nimbus -> supervisor

The second(nimbus -> supervisor) is also slow. However, the upload speed is
faster after I increase "storm.blobstore.inputstream.buffer.size.bytes" to
4MB.

I changed the code to increase the "THRIFT_CHUNK_SIZE_BYTES" value.
client upload speed is faster after I increase "THRIFT_CHUNK_SIZE_BYTES" to
4MB.

So I thought maybe the client should change the "THRIFT_CHUNK_SIZE_BYTES"
like nimbus ("storm.blobstore.inputstream.buffer.size.bytes" value).

Thanks.

2023년 2월 6일 (월) 오후 9:51, Rui Abreu <ru...@gmail.com>님이 작성:

> How big the uber jar that you are submitting?
> Have you tried tweaking the Nimbus side property?
> https://github.com/apache/storm/blob/master/conf/defaults.yaml#L67
>
> On Mon, 6 Feb 2023 at 04:14, Minwoo Kang <it...@gmail.com>
> wrote:
>
>> Hello, Users.
>>
>> When I submit storm topology, Occasionally it takes a long time.
>> So I investigated that, and I found THRIFT_CHUNK_SIZE_BYTES[1] is too
>> small.
>>
>> However, That value can not be modified because it is the final value.
>> It can be changed (STORM-241[2]) for the first time. But 70174d[3] commit
>> changed chunk size to private constant.
>>
>> I wonder why the user cannot change THRIFT_CHUNK_SIZE_BYTES.
>>
>> Thanks.
>>
>> [1]:
>> https://github.com/apache/storm/blob/bf29d1cc9914d4fe596b5e65532322e3dfd3e4ff/storm-client/src/jvm/org/apache/storm/StormSubmitter.java#L62
>> [2]: https://issues.apache.org/jira/browse/STORM-241
>> [3]:
>> https://github.com/apache/storm/commit/70174deaa758fc1e3f3a881445e8750ae7218b3d
>>
>

Re: Why can not change THRIFT_CHUNK_SIZE_BYTES in StormSubmitter?

Posted by Rui Abreu <ru...@gmail.com>.
How big the uber jar that you are submitting?
Have you tried tweaking the Nimbus side property?
https://github.com/apache/storm/blob/master/conf/defaults.yaml#L67

On Mon, 6 Feb 2023 at 04:14, Minwoo Kang <it...@gmail.com> wrote:

> Hello, Users.
>
> When I submit storm topology, Occasionally it takes a long time.
> So I investigated that, and I found THRIFT_CHUNK_SIZE_BYTES[1] is too
> small.
>
> However, That value can not be modified because it is the final value.
> It can be changed (STORM-241[2]) for the first time. But 70174d[3] commit
> changed chunk size to private constant.
>
> I wonder why the user cannot change THRIFT_CHUNK_SIZE_BYTES.
>
> Thanks.
>
> [1]:
> https://github.com/apache/storm/blob/bf29d1cc9914d4fe596b5e65532322e3dfd3e4ff/storm-client/src/jvm/org/apache/storm/StormSubmitter.java#L62
> [2]: https://issues.apache.org/jira/browse/STORM-241
> [3]:
> https://github.com/apache/storm/commit/70174deaa758fc1e3f3a881445e8750ae7218b3d
>