Suggestion on optimise kryo Output

[Bright Chen <br...@datatorrent.com>]

Hi,

The kryo Output has some limitation

   - The size of the data is limited. kryo write data to a fixed buffer, it
   will throw the overflow exception if the data exceed the size
   - The Output.toBytes() will copy the data to temporary buffer and
   output, it will decrease the performance and introduce garbage collection.

When I was tuning Spillable Data structure and Manage State. I create a
mechanism to share and reuse the memory to avoid above problem. And it can
be reused in core serialization with small change. Please see jira:
https://issues.apache.org/jira/browse/APEXMALHAR-2190


Any suggestion or comments, please put in jira:
https://issues.apache.org/jira/browse/APEXCORE-606


thanks

Bright

Re: Suggestion on optimise kryo Output

[Sandesh Hegde <sa...@datatorrent.com>]

Kryo is used in a default implementation of the StreamCodec interface.
Ideally, if the StreamCodec interface itself allows the buffer to be passed
then we can also send the buffer from the BufferServer in future.

On Mon, Jan 9, 2017 at 4:10 PM Bright Chen <br...@datatorrent.com> wrote:

> Hi,
>
> The kryo Output has some limitation
>
>    - The size of the data is limited. kryo write data to a fixed buffer, it
>    will throw the overflow exception if the data exceed the size
>    - The Output.toBytes() will copy the data to temporary buffer and
>    output, it will decrease the performance and introduce garbage
> collection.
>
> When I was tuning Spillable Data structure and Manage State. I create a
> mechanism to share and reuse the memory to avoid above problem. And it can
> be reused in core serialization with small change. Please see jira:
> https://issues.apache.org/jira/browse/APEXMALHAR-2190
>
>
> Any suggestion or comments, please put in jira:
> https://issues.apache.org/jira/browse/APEXCORE-606
>
>
> thanks
>
> Bright
>