You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directmemory.apache.org by Christoph Engelbert <no...@apache.org> on 2012/10/29 20:56:20 UTC
KryoSerializer
Hey guys,
while having a look how to implement the Lightning Serializer
adapter I recognized that maybe the current implementation of the
KryoSerializer and maybe some more could be absolutely thread unsafe.
private final byte[] buffer = new byte[ BUFFER_SIZE ];
private final Output output = new Output( buffer, -1 );
private final Input input = new Input( buffer );
Only one byte[] buffer is used for serialization and
deserialization. Is somewhere deep in DM an instantiater hidden that
generates new instances of the serializer or will there be a problem
if different threads read / write at the same time?
Cheers Chris
Re: KryoSerializer
Posted by David Yu <da...@gmail.com>.
On Tue, Oct 30, 2012 at 3:56 AM, Christoph Engelbert
<no...@apache.org>wrote:
> Hey guys,
>
> while having a look how to implement the Lightning Serializer
> adapter I recognized that maybe the current implementation of the
> KryoSerializer and maybe some more could be absolutely thread unsafe.
>
> private final byte[] buffer = new byte[ BUFFER_SIZE ];
> private final Output output = new Output( buffer, -1 );
> private final Input input = new Input( buffer );
>
> Only one byte[] buffer is used for serialization and
> deserialization. Is somewhere deep in DM an instantiater hidden that
> generates new instances of the serializer or will there be a problem
> if different threads read / write at the same time?
>
Good catch. Its definitely not thread safe. Other serializers(including
kryo) on jvm-serializers benchmark (referenced by tatu in this thread), are
not thread-safe either.
> Cheers Chris
>
--
When the cat is away, the mouse is alone.
- David Yu
Re: KryoSerializer
Posted by Tatu Saloranta <ts...@gmail.com>.
On Thu, Nov 1, 2012 at 11:26 AM, Christoph Engelbert
<no...@apache.org> wrote:
> Am 01.11.2012 18:35, schrieb Tatu Saloranta:
>> On Thu, Nov 1, 2012 at 10:06 AM, Christoph Engelbert
>> <no...@apache.org> wrote:
>>> Am 01.11.2012 17:48, schrieb Roman Levenstein:
>>>> Hi,
>>>>
>>>> On Thu, Nov 1, 2012 at 4:45 PM, Tatu Saloranta <ts...@gmail.com> wrote:
>>>>> On Thu, Nov 1, 2012 at 8:37 AM, Christoph Engelbert
>>>>> <no...@apache.org> wrote:
>>>>>> Am 01.11.2012 16:35, schrieb Tatu Saloranta:
>>>>>>> On Thu, Nov 1, 2012 at 2:53 AM, Christoph Engelbert
>>>>>>> <no...@apache.org> wrote:
>>>>>>>> Am 30.10.2012 18:01, schrieb Tatu Saloranta:
>>>>>>>>> You may want to see how serializer/deserializer in
>>>>>>>>>
>>>>>>>>> https://github.com/eishay/jvm-serializers
>>>>>>>>>
>>>>>>>>> is done -- Martin wrote it (or at least modified & has maintained it),
>>>>>>>>> so it should be along best practices.
>>>>>>>>>
>>>>>>>>> -+ Tatu +-
>>>>>>>> This is only best practice for non concurrent access. The unittests
>>>>>>>> are executed one by one so the implementation using one byte array
>>>>>>>> is not a problem but this cannot be threadsafe.
>>>>>>>>
>>>>>>>> This applies to this one as well. Never use this implementation in
>>>>>>>> concurrent environments :-)
>>>>>>>> https://github.com/eishay/jvm-serializers/blob/kannan/tpc/src/serializers/Kryo.java
>>>>>>> Correct. What I meant was it was right way to do it for the use case
>>>>>>> in question. Which granted requires one to understand the use case...
>>>>>>> I forgot that this particular test does not do multi-threaded
>>>>>>> processing like others do.
>>>>>>>
>>>>>>> It actually is one of the things for jvm-serializers to consider;
>>>>>>> efficient reuse techniques vary a lot between codecs.
>>>>>>>
>>>>>>> -+ Tatu +-
>>>>>> I guess there's still the problem about the kryo instance itself. On
>>>>>> googlecode it is marked that the kryo instance itself is not
>>>>>> threadsafe. Anyone has some more informations about that? Tatu? :-)
>>>>> No, I have only tried to use it without success once or twice, outside
>>>>> of jvm-serializers.
>>>>> My weapon of choice if Smile+Jakcson. :-D
>>>>>
>>>>> But Kryo author (Nate something?) should know -- I can try to dig
>>>>> contact info, point him to this mailing list.
>>>>>
>>>>> -+ Tatu +-
>>>> I'm one of Kryo's contributors.
>>>> Kryo instances do keep some context information during serialization
>>>> and deserialization. Therefore, the same instance cannot be safely
>>>> used at the same time with multiple input or output streams for
>>>> performing simultaneous (de) serializations. And using the same
>>>> instance with multiple threads can result in even more problems.
>>>>
>>>> Therefore, when I need to use Kryo in multi-threaded environment, I
>>>> usually make sure that each thread gets its own instance of Kryo at
>>>> least for the duration of the operation. The reduce the cost of Kryo
>>>> instances creation I usually use a pool of such instances. Every time
>>>> a thread needs to perform a (de) serialization operation, it would get
>>>> an instance from the pool, perform the operation and return the
>>>> instance back to the pool. Alternatively, one could use ThreadLocal
>>>> Kryo instances if applicable.
>>>>
>>>> -Roman
>>> Ok that is a nice information. So I would suggest to use
>>> concurrencylevel to instantiate a bunch of kryo instances and queue
>>> them. If non of the preinstantiated kryo instances is available than
>>> the (de-)serialization must wait until one's free again (but that
>>> would mean the concurrencylevel is choosen to low ;-)).
>> Instead of queue which needs synchronization, it may be better to use
>> ThreadLocal with soft reference. This requires no synchronization for
>> access, and also cleans up instances if needed for GC. This can make a
>> big difference for multi-threaded access due to reduced lock
>> contestion.
>> And the details can be hidden behind a shared factory if that simplifies things.
>>
>> -+ Tatu +-
>
> But I guess I would need to register classes in all of the kryo
> instances at the same time, or how creates kryo ids for the
> different classes?
They'd need to get registered when instances are created (or via
factory used to create them). Same way I think as what queue approach
would do, except can't do eagerly (which may or may not be done with
queues).
This is why a separate factory would make sense, and
ThreadLocal+SoftReference is just the mechanism for retaining them for
possible reuse.
-+ Tatu +-
Re: KryoSerializer
Posted by Christoph Engelbert <no...@apache.org>.
Am 01.11.2012 18:35, schrieb Tatu Saloranta:
> On Thu, Nov 1, 2012 at 10:06 AM, Christoph Engelbert
> <no...@apache.org> wrote:
>> Am 01.11.2012 17:48, schrieb Roman Levenstein:
>>> Hi,
>>>
>>> On Thu, Nov 1, 2012 at 4:45 PM, Tatu Saloranta <ts...@gmail.com> wrote:
>>>> On Thu, Nov 1, 2012 at 8:37 AM, Christoph Engelbert
>>>> <no...@apache.org> wrote:
>>>>> Am 01.11.2012 16:35, schrieb Tatu Saloranta:
>>>>>> On Thu, Nov 1, 2012 at 2:53 AM, Christoph Engelbert
>>>>>> <no...@apache.org> wrote:
>>>>>>> Am 30.10.2012 18:01, schrieb Tatu Saloranta:
>>>>>>>> You may want to see how serializer/deserializer in
>>>>>>>>
>>>>>>>> https://github.com/eishay/jvm-serializers
>>>>>>>>
>>>>>>>> is done -- Martin wrote it (or at least modified & has maintained it),
>>>>>>>> so it should be along best practices.
>>>>>>>>
>>>>>>>> -+ Tatu +-
>>>>>>> This is only best practice for non concurrent access. The unittests
>>>>>>> are executed one by one so the implementation using one byte array
>>>>>>> is not a problem but this cannot be threadsafe.
>>>>>>>
>>>>>>> This applies to this one as well. Never use this implementation in
>>>>>>> concurrent environments :-)
>>>>>>> https://github.com/eishay/jvm-serializers/blob/kannan/tpc/src/serializers/Kryo.java
>>>>>> Correct. What I meant was it was right way to do it for the use case
>>>>>> in question. Which granted requires one to understand the use case...
>>>>>> I forgot that this particular test does not do multi-threaded
>>>>>> processing like others do.
>>>>>>
>>>>>> It actually is one of the things for jvm-serializers to consider;
>>>>>> efficient reuse techniques vary a lot between codecs.
>>>>>>
>>>>>> -+ Tatu +-
>>>>> I guess there's still the problem about the kryo instance itself. On
>>>>> googlecode it is marked that the kryo instance itself is not
>>>>> threadsafe. Anyone has some more informations about that? Tatu? :-)
>>>> No, I have only tried to use it without success once or twice, outside
>>>> of jvm-serializers.
>>>> My weapon of choice if Smile+Jakcson. :-D
>>>>
>>>> But Kryo author (Nate something?) should know -- I can try to dig
>>>> contact info, point him to this mailing list.
>>>>
>>>> -+ Tatu +-
>>> I'm one of Kryo's contributors.
>>> Kryo instances do keep some context information during serialization
>>> and deserialization. Therefore, the same instance cannot be safely
>>> used at the same time with multiple input or output streams for
>>> performing simultaneous (de) serializations. And using the same
>>> instance with multiple threads can result in even more problems.
>>>
>>> Therefore, when I need to use Kryo in multi-threaded environment, I
>>> usually make sure that each thread gets its own instance of Kryo at
>>> least for the duration of the operation. The reduce the cost of Kryo
>>> instances creation I usually use a pool of such instances. Every time
>>> a thread needs to perform a (de) serialization operation, it would get
>>> an instance from the pool, perform the operation and return the
>>> instance back to the pool. Alternatively, one could use ThreadLocal
>>> Kryo instances if applicable.
>>>
>>> -Roman
>> Ok that is a nice information. So I would suggest to use
>> concurrencylevel to instantiate a bunch of kryo instances and queue
>> them. If non of the preinstantiated kryo instances is available than
>> the (de-)serialization must wait until one's free again (but that
>> would mean the concurrencylevel is choosen to low ;-)).
> Instead of queue which needs synchronization, it may be better to use
> ThreadLocal with soft reference. This requires no synchronization for
> access, and also cleans up instances if needed for GC. This can make a
> big difference for multi-threaded access due to reduced lock
> contestion.
> And the details can be hidden behind a shared factory if that simplifies things.
>
> -+ Tatu +-
But I guess I would need to register classes in all of the kryo
instances at the same time, or how creates kryo ids for the
different classes?
Re: KryoSerializer
Posted by Tatu Saloranta <ts...@gmail.com>.
On Thu, Nov 1, 2012 at 10:06 AM, Christoph Engelbert
<no...@apache.org> wrote:
> Am 01.11.2012 17:48, schrieb Roman Levenstein:
>> Hi,
>>
>> On Thu, Nov 1, 2012 at 4:45 PM, Tatu Saloranta <ts...@gmail.com> wrote:
>>> On Thu, Nov 1, 2012 at 8:37 AM, Christoph Engelbert
>>> <no...@apache.org> wrote:
>>>> Am 01.11.2012 16:35, schrieb Tatu Saloranta:
>>>>> On Thu, Nov 1, 2012 at 2:53 AM, Christoph Engelbert
>>>>> <no...@apache.org> wrote:
>>>>>> Am 30.10.2012 18:01, schrieb Tatu Saloranta:
>>>>>>> You may want to see how serializer/deserializer in
>>>>>>>
>>>>>>> https://github.com/eishay/jvm-serializers
>>>>>>>
>>>>>>> is done -- Martin wrote it (or at least modified & has maintained it),
>>>>>>> so it should be along best practices.
>>>>>>>
>>>>>>> -+ Tatu +-
>>>>>> This is only best practice for non concurrent access. The unittests
>>>>>> are executed one by one so the implementation using one byte array
>>>>>> is not a problem but this cannot be threadsafe.
>>>>>>
>>>>>> This applies to this one as well. Never use this implementation in
>>>>>> concurrent environments :-)
>>>>>> https://github.com/eishay/jvm-serializers/blob/kannan/tpc/src/serializers/Kryo.java
>>>>> Correct. What I meant was it was right way to do it for the use case
>>>>> in question. Which granted requires one to understand the use case...
>>>>> I forgot that this particular test does not do multi-threaded
>>>>> processing like others do.
>>>>>
>>>>> It actually is one of the things for jvm-serializers to consider;
>>>>> efficient reuse techniques vary a lot between codecs.
>>>>>
>>>>> -+ Tatu +-
>>>> I guess there's still the problem about the kryo instance itself. On
>>>> googlecode it is marked that the kryo instance itself is not
>>>> threadsafe. Anyone has some more informations about that? Tatu? :-)
>>> No, I have only tried to use it without success once or twice, outside
>>> of jvm-serializers.
>>> My weapon of choice if Smile+Jakcson. :-D
>>>
>>> But Kryo author (Nate something?) should know -- I can try to dig
>>> contact info, point him to this mailing list.
>>>
>>> -+ Tatu +-
>> I'm one of Kryo's contributors.
>> Kryo instances do keep some context information during serialization
>> and deserialization. Therefore, the same instance cannot be safely
>> used at the same time with multiple input or output streams for
>> performing simultaneous (de) serializations. And using the same
>> instance with multiple threads can result in even more problems.
>>
>> Therefore, when I need to use Kryo in multi-threaded environment, I
>> usually make sure that each thread gets its own instance of Kryo at
>> least for the duration of the operation. The reduce the cost of Kryo
>> instances creation I usually use a pool of such instances. Every time
>> a thread needs to perform a (de) serialization operation, it would get
>> an instance from the pool, perform the operation and return the
>> instance back to the pool. Alternatively, one could use ThreadLocal
>> Kryo instances if applicable.
>>
>> -Roman
>
> Ok that is a nice information. So I would suggest to use
> concurrencylevel to instantiate a bunch of kryo instances and queue
> them. If non of the preinstantiated kryo instances is available than
> the (de-)serialization must wait until one's free again (but that
> would mean the concurrencylevel is choosen to low ;-)).
Instead of queue which needs synchronization, it may be better to use
ThreadLocal with soft reference. This requires no synchronization for
access, and also cleans up instances if needed for GC. This can make a
big difference for multi-threaded access due to reduced lock
contestion.
And the details can be hidden behind a shared factory if that simplifies things.
-+ Tatu +-
Re: KryoSerializer
Posted by Roman Levenstein <ro...@gmail.com>.
On Thu, Nov 1, 2012 at 6:06 PM, Christoph Engelbert <no...@apache.org>wrote:
> Am 01.11.2012 17:48, schrieb Roman Levenstein:
> > Hi,
> >
> > On Thu, Nov 1, 2012 at 4:45 PM, Tatu Saloranta <ts...@gmail.com>
> wrote:
> >> On Thu, Nov 1, 2012 at 8:37 AM, Christoph Engelbert
> >> <no...@apache.org> wrote:
> >>> Am 01.11.2012 16:35, schrieb Tatu Saloranta:
> >>>> On Thu, Nov 1, 2012 at 2:53 AM, Christoph Engelbert
> >>>> <no...@apache.org> wrote:
> >>>>> Am 30.10.2012 18:01, schrieb Tatu Saloranta:
> >>>>>> You may want to see how serializer/deserializer in
> >>>>>>
> >>>>>> https://github.com/eishay/jvm-serializers
> >>>>>>
> >>>>>> is done -- Martin wrote it (or at least modified & has maintained
> it),
> >>>>>> so it should be along best practices.
> >>>>>>
> >>>>>> -+ Tatu +-
> >>>>> This is only best practice for non concurrent access. The unittests
> >>>>> are executed one by one so the implementation using one byte array
> >>>>> is not a problem but this cannot be threadsafe.
> >>>>>
> >>>>> This applies to this one as well. Never use this implementation in
> >>>>> concurrent environments :-)
> >>>>>
> https://github.com/eishay/jvm-serializers/blob/kannan/tpc/src/serializers/Kryo.java
> >>>> Correct. What I meant was it was right way to do it for the use case
> >>>> in question. Which granted requires one to understand the use case...
> >>>> I forgot that this particular test does not do multi-threaded
> >>>> processing like others do.
> >>>>
> >>>> It actually is one of the things for jvm-serializers to consider;
> >>>> efficient reuse techniques vary a lot between codecs.
> >>>>
> >>>> -+ Tatu +-
> >>> I guess there's still the problem about the kryo instance itself. On
> >>> googlecode it is marked that the kryo instance itself is not
> >>> threadsafe. Anyone has some more informations about that? Tatu? :-)
> >> No, I have only tried to use it without success once or twice, outside
> >> of jvm-serializers.
> >> My weapon of choice if Smile+Jakcson. :-D
> >>
> >> But Kryo author (Nate something?) should know -- I can try to dig
> >> contact info, point him to this mailing list.
> >>
> >> -+ Tatu +-
> > I'm one of Kryo's contributors.
> > Kryo instances do keep some context information during serialization
> > and deserialization. Therefore, the same instance cannot be safely
> > used at the same time with multiple input or output streams for
> > performing simultaneous (de) serializations. And using the same
> > instance with multiple threads can result in even more problems.
> >
> > Therefore, when I need to use Kryo in multi-threaded environment, I
> > usually make sure that each thread gets its own instance of Kryo at
> > least for the duration of the operation. The reduce the cost of Kryo
> > instances creation I usually use a pool of such instances. Every time
> > a thread needs to perform a (de) serialization operation, it would get
> > an instance from the pool, perform the operation and return the
> > instance back to the pool. Alternatively, one could use ThreadLocal
> > Kryo instances if applicable.
> >
> > -Roman
>
> Ok that is a nice information. So I would suggest to use
> concurrencylevel to instantiate a bunch of kryo instances and queue
> them. If non of the preinstantiated kryo instances is available than
> the (de-)serialization must wait until one's free again (but that
> would mean the concurrencylevel is choosen to low ;-)).
>
> Or you grow the pool on demand. You may also shrink it afterwards based on
the concurrency level or any other kind of metric, e.g. you could remove a
Kryo instance from the pool if it is not used for a certain time period .
Cheers,
Roman
Re: KryoSerializer
Posted by Christoph Engelbert <no...@apache.org>.
Am 01.11.2012 17:48, schrieb Roman Levenstein:
> Hi,
>
> On Thu, Nov 1, 2012 at 4:45 PM, Tatu Saloranta <ts...@gmail.com> wrote:
>> On Thu, Nov 1, 2012 at 8:37 AM, Christoph Engelbert
>> <no...@apache.org> wrote:
>>> Am 01.11.2012 16:35, schrieb Tatu Saloranta:
>>>> On Thu, Nov 1, 2012 at 2:53 AM, Christoph Engelbert
>>>> <no...@apache.org> wrote:
>>>>> Am 30.10.2012 18:01, schrieb Tatu Saloranta:
>>>>>> You may want to see how serializer/deserializer in
>>>>>>
>>>>>> https://github.com/eishay/jvm-serializers
>>>>>>
>>>>>> is done -- Martin wrote it (or at least modified & has maintained it),
>>>>>> so it should be along best practices.
>>>>>>
>>>>>> -+ Tatu +-
>>>>> This is only best practice for non concurrent access. The unittests
>>>>> are executed one by one so the implementation using one byte array
>>>>> is not a problem but this cannot be threadsafe.
>>>>>
>>>>> This applies to this one as well. Never use this implementation in
>>>>> concurrent environments :-)
>>>>> https://github.com/eishay/jvm-serializers/blob/kannan/tpc/src/serializers/Kryo.java
>>>> Correct. What I meant was it was right way to do it for the use case
>>>> in question. Which granted requires one to understand the use case...
>>>> I forgot that this particular test does not do multi-threaded
>>>> processing like others do.
>>>>
>>>> It actually is one of the things for jvm-serializers to consider;
>>>> efficient reuse techniques vary a lot between codecs.
>>>>
>>>> -+ Tatu +-
>>> I guess there's still the problem about the kryo instance itself. On
>>> googlecode it is marked that the kryo instance itself is not
>>> threadsafe. Anyone has some more informations about that? Tatu? :-)
>> No, I have only tried to use it without success once or twice, outside
>> of jvm-serializers.
>> My weapon of choice if Smile+Jakcson. :-D
>>
>> But Kryo author (Nate something?) should know -- I can try to dig
>> contact info, point him to this mailing list.
>>
>> -+ Tatu +-
> I'm one of Kryo's contributors.
> Kryo instances do keep some context information during serialization
> and deserialization. Therefore, the same instance cannot be safely
> used at the same time with multiple input or output streams for
> performing simultaneous (de) serializations. And using the same
> instance with multiple threads can result in even more problems.
>
> Therefore, when I need to use Kryo in multi-threaded environment, I
> usually make sure that each thread gets its own instance of Kryo at
> least for the duration of the operation. The reduce the cost of Kryo
> instances creation I usually use a pool of such instances. Every time
> a thread needs to perform a (de) serialization operation, it would get
> an instance from the pool, perform the operation and return the
> instance back to the pool. Alternatively, one could use ThreadLocal
> Kryo instances if applicable.
>
> -Roman
Ok that is a nice information. So I would suggest to use
concurrencylevel to instantiate a bunch of kryo instances and queue
them. If non of the preinstantiated kryo instances is available than
the (de-)serialization must wait until one's free again (but that
would mean the concurrencylevel is choosen to low ;-)).
Cheers Chris
Re: KryoSerializer
Posted by Roman Levenstein <ro...@gmail.com>.
Hi,
On Thu, Nov 1, 2012 at 4:45 PM, Tatu Saloranta <ts...@gmail.com> wrote:
> On Thu, Nov 1, 2012 at 8:37 AM, Christoph Engelbert
> <no...@apache.org> wrote:
>> Am 01.11.2012 16:35, schrieb Tatu Saloranta:
>>> On Thu, Nov 1, 2012 at 2:53 AM, Christoph Engelbert
>>> <no...@apache.org> wrote:
>>>> Am 30.10.2012 18:01, schrieb Tatu Saloranta:
>>>>> You may want to see how serializer/deserializer in
>>>>>
>>>>> https://github.com/eishay/jvm-serializers
>>>>>
>>>>> is done -- Martin wrote it (or at least modified & has maintained it),
>>>>> so it should be along best practices.
>>>>>
>>>>> -+ Tatu +-
>>>> This is only best practice for non concurrent access. The unittests
>>>> are executed one by one so the implementation using one byte array
>>>> is not a problem but this cannot be threadsafe.
>>>>
>>>> This applies to this one as well. Never use this implementation in
>>>> concurrent environments :-)
>>>> https://github.com/eishay/jvm-serializers/blob/kannan/tpc/src/serializers/Kryo.java
>>> Correct. What I meant was it was right way to do it for the use case
>>> in question. Which granted requires one to understand the use case...
>>> I forgot that this particular test does not do multi-threaded
>>> processing like others do.
>>>
>>> It actually is one of the things for jvm-serializers to consider;
>>> efficient reuse techniques vary a lot between codecs.
>>>
>>> -+ Tatu +-
>>
>> I guess there's still the problem about the kryo instance itself. On
>> googlecode it is marked that the kryo instance itself is not
>> threadsafe. Anyone has some more informations about that? Tatu? :-)
>
> No, I have only tried to use it without success once or twice, outside
> of jvm-serializers.
> My weapon of choice if Smile+Jakcson. :-D
>
> But Kryo author (Nate something?) should know -- I can try to dig
> contact info, point him to this mailing list.
>
> -+ Tatu +-
I'm one of Kryo's contributors.
Kryo instances do keep some context information during serialization
and deserialization. Therefore, the same instance cannot be safely
used at the same time with multiple input or output streams for
performing simultaneous (de) serializations. And using the same
instance with multiple threads can result in even more problems.
Therefore, when I need to use Kryo in multi-threaded environment, I
usually make sure that each thread gets its own instance of Kryo at
least for the duration of the operation. The reduce the cost of Kryo
instances creation I usually use a pool of such instances. Every time
a thread needs to perform a (de) serialization operation, it would get
an instance from the pool, perform the operation and return the
instance back to the pool. Alternatively, one could use ThreadLocal
Kryo instances if applicable.
-Roman
Re: KryoSerializer
Posted by Tatu Saloranta <ts...@gmail.com>.
On Thu, Nov 1, 2012 at 8:37 AM, Christoph Engelbert
<no...@apache.org> wrote:
> Am 01.11.2012 16:35, schrieb Tatu Saloranta:
>> On Thu, Nov 1, 2012 at 2:53 AM, Christoph Engelbert
>> <no...@apache.org> wrote:
>>> Am 30.10.2012 18:01, schrieb Tatu Saloranta:
>>>> You may want to see how serializer/deserializer in
>>>>
>>>> https://github.com/eishay/jvm-serializers
>>>>
>>>> is done -- Martin wrote it (or at least modified & has maintained it),
>>>> so it should be along best practices.
>>>>
>>>> -+ Tatu +-
>>> This is only best practice for non concurrent access. The unittests
>>> are executed one by one so the implementation using one byte array
>>> is not a problem but this cannot be threadsafe.
>>>
>>> This applies to this one as well. Never use this implementation in
>>> concurrent environments :-)
>>> https://github.com/eishay/jvm-serializers/blob/kannan/tpc/src/serializers/Kryo.java
>> Correct. What I meant was it was right way to do it for the use case
>> in question. Which granted requires one to understand the use case...
>> I forgot that this particular test does not do multi-threaded
>> processing like others do.
>>
>> It actually is one of the things for jvm-serializers to consider;
>> efficient reuse techniques vary a lot between codecs.
>>
>> -+ Tatu +-
>
> I guess there's still the problem about the kryo instance itself. On
> googlecode it is marked that the kryo instance itself is not
> threadsafe. Anyone has some more informations about that? Tatu? :-)
No, I have only tried to use it without success once or twice, outside
of jvm-serializers.
My weapon of choice if Smile+Jakcson. :-D
But Kryo author (Nate something?) should know -- I can try to dig
contact info, point him to this mailing list.
-+ Tatu +-
Re: KryoSerializer
Posted by Christoph Engelbert <no...@apache.org>.
Am 01.11.2012 16:35, schrieb Tatu Saloranta:
> On Thu, Nov 1, 2012 at 2:53 AM, Christoph Engelbert
> <no...@apache.org> wrote:
>> Am 30.10.2012 18:01, schrieb Tatu Saloranta:
>>> You may want to see how serializer/deserializer in
>>>
>>> https://github.com/eishay/jvm-serializers
>>>
>>> is done -- Martin wrote it (or at least modified & has maintained it),
>>> so it should be along best practices.
>>>
>>> -+ Tatu +-
>> This is only best practice for non concurrent access. The unittests
>> are executed one by one so the implementation using one byte array
>> is not a problem but this cannot be threadsafe.
>>
>> This applies to this one as well. Never use this implementation in
>> concurrent environments :-)
>> https://github.com/eishay/jvm-serializers/blob/kannan/tpc/src/serializers/Kryo.java
> Correct. What I meant was it was right way to do it for the use case
> in question. Which granted requires one to understand the use case...
> I forgot that this particular test does not do multi-threaded
> processing like others do.
>
> It actually is one of the things for jvm-serializers to consider;
> efficient reuse techniques vary a lot between codecs.
>
> -+ Tatu +-
I guess there's still the problem about the kryo instance itself. On
googlecode it is marked that the kryo instance itself is not
threadsafe. Anyone has some more informations about that? Tatu? :-)
Cheers Chris
Re: KryoSerializer
Posted by Tatu Saloranta <ts...@gmail.com>.
On Thu, Nov 1, 2012 at 2:53 AM, Christoph Engelbert
<no...@apache.org> wrote:
> Am 30.10.2012 18:01, schrieb Tatu Saloranta:
>> You may want to see how serializer/deserializer in
>>
>> https://github.com/eishay/jvm-serializers
>>
>> is done -- Martin wrote it (or at least modified & has maintained it),
>> so it should be along best practices.
>>
>> -+ Tatu +-
>
> This is only best practice for non concurrent access. The unittests
> are executed one by one so the implementation using one byte array
> is not a problem but this cannot be threadsafe.
>
> This applies to this one as well. Never use this implementation in
> concurrent environments :-)
> https://github.com/eishay/jvm-serializers/blob/kannan/tpc/src/serializers/Kryo.java
Correct. What I meant was it was right way to do it for the use case
in question. Which granted requires one to understand the use case...
I forgot that this particular test does not do multi-threaded
processing like others do.
It actually is one of the things for jvm-serializers to consider;
efficient reuse techniques vary a lot between codecs.
-+ Tatu +-
Re: KryoSerializer
Posted by Christoph Engelbert <no...@apache.org>.
Am 30.10.2012 18:01, schrieb Tatu Saloranta:
> You may want to see how serializer/deserializer in
>
> https://github.com/eishay/jvm-serializers
>
> is done -- Martin wrote it (or at least modified & has maintained it),
> so it should be along best practices.
>
> -+ Tatu +-
This is only best practice for non concurrent access. The unittests
are executed one by one so the implementation using one byte array
is not a problem but this cannot be threadsafe.
This applies to this one as well. Never use this implementation in
concurrent environments :-)
https://github.com/eishay/jvm-serializers/blob/kannan/tpc/src/serializers/Kryo.java
> On Tue, Oct 30, 2012 at 9:33 AM, Simone Tripodi
> <si...@apache.org> wrote:
>>> Take care of the field private final Kryo kryo; too.
>>> Is this class declared as thread safe ?
>> IIRC, yes, a Kryo instance should be thread safe, but much better is
>> we double check - I just sent Martin a tweet message :P
>>
>> best,
>> -Simo
>>
>> http://people.apache.org/~simonetripodi/
>> http://simonetripodi.livejournal.com/
>> http://twitter.com/simonetripodi
>> http://www.99soft.org/
>>
>>
>> On Tue, Oct 30, 2012 at 4:43 PM, Olivier Lamy <ol...@apache.org> wrote:
>>> 2012/10/29 Christoph Engelbert <no...@apache.org>:
>>>> Hey guys,
>>>>
>>>> while having a look how to implement the Lightning Serializer
>>>> adapter I recognized that maybe the current implementation of the
>>>> KryoSerializer and maybe some more could be absolutely thread unsafe.
>>>>
>>>> private final byte[] buffer = new byte[ BUFFER_SIZE ];
>>>> private final Output output = new Output( buffer, -1 );
>>>> private final Input input = new Input( buffer );
>>>>
>>>> Only one byte[] buffer is used for serialization and
>>>> deserialization. Is somewhere deep in DM an instantiater hidden that
>>>> generates new instances of the serializer or will there be a problem
>>>> if different threads read / write at the same time?
>>> Yup you must write a KryoSerializer which is thread safe (others
>>> de/serializer are thread safe).
>>>
>>> Take care of the field private final Kryo kryo; too.
>>> Is this class declared as thread safe ?
>>>
>>>
>>>> Cheers Chris
>>>
>>>
>>> --
>>> Olivier Lamy
>>> Talend: http://coders.talend.com
>>> http://twitter.com/olamy | http://linkedin.com/in/olamy
Re: KryoSerializer
Posted by Tatu Saloranta <ts...@gmail.com>.
You may want to see how serializer/deserializer in
https://github.com/eishay/jvm-serializers
is done -- Martin wrote it (or at least modified & has maintained it),
so it should be along best practices.
-+ Tatu +-
On Tue, Oct 30, 2012 at 9:33 AM, Simone Tripodi
<si...@apache.org> wrote:
>> Take care of the field private final Kryo kryo; too.
>> Is this class declared as thread safe ?
>
> IIRC, yes, a Kryo instance should be thread safe, but much better is
> we double check - I just sent Martin a tweet message :P
>
> best,
> -Simo
>
> http://people.apache.org/~simonetripodi/
> http://simonetripodi.livejournal.com/
> http://twitter.com/simonetripodi
> http://www.99soft.org/
>
>
> On Tue, Oct 30, 2012 at 4:43 PM, Olivier Lamy <ol...@apache.org> wrote:
>> 2012/10/29 Christoph Engelbert <no...@apache.org>:
>>> Hey guys,
>>>
>>> while having a look how to implement the Lightning Serializer
>>> adapter I recognized that maybe the current implementation of the
>>> KryoSerializer and maybe some more could be absolutely thread unsafe.
>>>
>>> private final byte[] buffer = new byte[ BUFFER_SIZE ];
>>> private final Output output = new Output( buffer, -1 );
>>> private final Input input = new Input( buffer );
>>>
>>> Only one byte[] buffer is used for serialization and
>>> deserialization. Is somewhere deep in DM an instantiater hidden that
>>> generates new instances of the serializer or will there be a problem
>>> if different threads read / write at the same time?
>> Yup you must write a KryoSerializer which is thread safe (others
>> de/serializer are thread safe).
>>
>> Take care of the field private final Kryo kryo; too.
>> Is this class declared as thread safe ?
>>
>>
>>>
>>> Cheers Chris
>>
>>
>>
>> --
>> Olivier Lamy
>> Talend: http://coders.talend.com
>> http://twitter.com/olamy | http://linkedin.com/in/olamy
Re: KryoSerializer
Posted by Simone Tripodi <si...@apache.org>.
> Take care of the field private final Kryo kryo; too.
> Is this class declared as thread safe ?
IIRC, yes, a Kryo instance should be thread safe, but much better is
we double check - I just sent Martin a tweet message :P
best,
-Simo
http://people.apache.org/~simonetripodi/
http://simonetripodi.livejournal.com/
http://twitter.com/simonetripodi
http://www.99soft.org/
On Tue, Oct 30, 2012 at 4:43 PM, Olivier Lamy <ol...@apache.org> wrote:
> 2012/10/29 Christoph Engelbert <no...@apache.org>:
>> Hey guys,
>>
>> while having a look how to implement the Lightning Serializer
>> adapter I recognized that maybe the current implementation of the
>> KryoSerializer and maybe some more could be absolutely thread unsafe.
>>
>> private final byte[] buffer = new byte[ BUFFER_SIZE ];
>> private final Output output = new Output( buffer, -1 );
>> private final Input input = new Input( buffer );
>>
>> Only one byte[] buffer is used for serialization and
>> deserialization. Is somewhere deep in DM an instantiater hidden that
>> generates new instances of the serializer or will there be a problem
>> if different threads read / write at the same time?
> Yup you must write a KryoSerializer which is thread safe (others
> de/serializer are thread safe).
>
> Take care of the field private final Kryo kryo; too.
> Is this class declared as thread safe ?
>
>
>>
>> Cheers Chris
>
>
>
> --
> Olivier Lamy
> Talend: http://coders.talend.com
> http://twitter.com/olamy | http://linkedin.com/in/olamy
Re: KryoSerializer
Posted by Olivier Lamy <ol...@apache.org>.
2012/10/29 Christoph Engelbert <no...@apache.org>:
> Hey guys,
>
> while having a look how to implement the Lightning Serializer
> adapter I recognized that maybe the current implementation of the
> KryoSerializer and maybe some more could be absolutely thread unsafe.
>
> private final byte[] buffer = new byte[ BUFFER_SIZE ];
> private final Output output = new Output( buffer, -1 );
> private final Input input = new Input( buffer );
>
> Only one byte[] buffer is used for serialization and
> deserialization. Is somewhere deep in DM an instantiater hidden that
> generates new instances of the serializer or will there be a problem
> if different threads read / write at the same time?
Yup you must write a KryoSerializer which is thread safe (others
de/serializer are thread safe).
Take care of the field private final Kryo kryo; too.
Is this class declared as thread safe ?
>
> Cheers Chris
--
Olivier Lamy
Talend: http://coders.talend.com
http://twitter.com/olamy | http://linkedin.com/in/olamy
Re: KryoSerializer
Posted by Simone Tripodi <si...@apache.org>.
Hi Noctarius,
I'm by your side - that (In|Out)put objects have to be instantiated
each (de)serialize invocation.
Feel free to fill an issue and fix it - unfortunately I am with almost
0 spare time during these months :(
Have a nice day, all the best!
-Simo
http://people.apache.org/~simonetripodi/
http://simonetripodi.livejournal.com/
http://twitter.com/simonetripodi
http://www.99soft.org/
On Mon, Oct 29, 2012 at 8:56 PM, Christoph Engelbert
<no...@apache.org> wrote:
> Hey guys,
>
> while having a look how to implement the Lightning Serializer
> adapter I recognized that maybe the current implementation of the
> KryoSerializer and maybe some more could be absolutely thread unsafe.
>
> private final byte[] buffer = new byte[ BUFFER_SIZE ];
> private final Output output = new Output( buffer, -1 );
> private final Input input = new Input( buffer );
>
> Only one byte[] buffer is used for serialization and
> deserialization. Is somewhere deep in DM an instantiater hidden that
> generates new instances of the serializer or will there be a problem
> if different threads read / write at the same time?
>
> Cheers Chris