You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Myron Chelyada <my...@gmail.com> on 2016/02/15 18:25:50 UTC
Externalizable in cache
Hello team,
I am faced to issue with putting "Externalizable" value into cache.
I.e. when I am trying to put instance of my custom "externalizable" class
into cache program hangs on put operation.
After some investigation I've figured out that it happens when "put" event
is included.
Test class that allows to reproduce this issue can be found in attachment.
Is it perhaps already known issue?
Re: Externalizable in cache
Posted by Myron Chelyada <my...@gmail.com>.
Thanks,
And sorry for a little bit confusing test I provided.
2016-02-16 13:46 GMT+02:00 Vladimir Ozerov <vo...@gridgain.com>:
> Myron,
> Thank you for pointing this. it looks like we violate stream contract
> because regular Java ObjectOutputStream will return -1 in this case.
> Forwarding to dev list.
>
> Igniters,
> As Myron mentioned, out implementation of ObjectInput returns 0 when
> performing ObjectInput.read(byte[]) and there is no more array data
> available. To the contrast, Java's ObjectInputStream returns -1 in this
> case.
> Looks like we should fix that.
>
> Thoughts?
>
> Vladimir.
>
> On Mon, Feb 15, 2016 at 10:32 PM, Myron Chelyada <myron.chelyada@gmail.com
> > wrote:
>
>> Hello Alexey,
>>
>> This is actually not real code but just extracted part for test. That's
>> why it looks incorrect.
>> And I've already captured infinite loop because of 0 being returned. And
>> imagine use case when I need to read until end of stream. So question is
>> why 0 is returned but not -1 (according to API).
>>
>>
>> 2016-02-15 21:10 GMT+02:00 Dmitriy Setrakyan <ds...@apache.org>:
>>
>>> Alexey, is this something that Ignite could prevent automatically?
>>>
>>> On Mon, Feb 15, 2016 at 10:17 AM, Alexey Goncharuk <
>>> alexey.goncharuk@gmail.com> wrote:
>>>
>>>> A little correction: in this particular case inputStream does return 0
>>>> which leads to an infinite loop, however, generally this may be not the
>>>> case, so the implementation should not read beyond object boundary anyways.
>>>>
>>>
>>>
>>
>
Re: Externalizable in cache
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Alex,
In ObjectOutputStream if you read byte array and still have some other data
in the stream, it will return you -1. So as I understand, from user
perspective it is normal to wait for -1. What do you think?
Vladimir.
On Tue, Feb 16, 2016 at 2:54 PM, Alexey Goncharuk <
alexey.goncharuk@gmail.com> wrote:
> Agree that this should be fixed. However, from the end-user perspective
> this should never be an issue because at some point in future Ignite might
> need to append some extra bytes to the Externalizable object layout, thus
> reading beyond that limit will break the unmarshalling process.
>
> If, by the time we invoke readExternal, we know the size of the written
> externalizable object then we can create a thin wrapper around the existing
> input stream which will provide the desired behavior. However, this will
> come at a cost of generating more garbage, so I would just fix the return
> value for the input stream.
>
Re: Externalizable in cache
Posted by Alexey Goncharuk <al...@gmail.com>.
Agree that this should be fixed. However, from the end-user perspective
this should never be an issue because at some point in future Ignite might
need to append some extra bytes to the Externalizable object layout, thus
reading beyond that limit will break the unmarshalling process.
If, by the time we invoke readExternal, we know the size of the written
externalizable object then we can create a thin wrapper around the existing
input stream which will provide the desired behavior. However, this will
come at a cost of generating more garbage, so I would just fix the return
value for the input stream.
Re: Externalizable in cache
Posted by Myron Chelyada <my...@gmail.com>.
Thanks,
And sorry for a little bit confusing test I provided.
2016-02-16 13:46 GMT+02:00 Vladimir Ozerov <vo...@gridgain.com>:
> Myron,
> Thank you for pointing this. it looks like we violate stream contract
> because regular Java ObjectOutputStream will return -1 in this case.
> Forwarding to dev list.
>
> Igniters,
> As Myron mentioned, out implementation of ObjectInput returns 0 when
> performing ObjectInput.read(byte[]) and there is no more array data
> available. To the contrast, Java's ObjectInputStream returns -1 in this
> case.
> Looks like we should fix that.
>
> Thoughts?
>
> Vladimir.
>
> On Mon, Feb 15, 2016 at 10:32 PM, Myron Chelyada <myron.chelyada@gmail.com
> > wrote:
>
>> Hello Alexey,
>>
>> This is actually not real code but just extracted part for test. That's
>> why it looks incorrect.
>> And I've already captured infinite loop because of 0 being returned. And
>> imagine use case when I need to read until end of stream. So question is
>> why 0 is returned but not -1 (according to API).
>>
>>
>> 2016-02-15 21:10 GMT+02:00 Dmitriy Setrakyan <ds...@apache.org>:
>>
>>> Alexey, is this something that Ignite could prevent automatically?
>>>
>>> On Mon, Feb 15, 2016 at 10:17 AM, Alexey Goncharuk <
>>> alexey.goncharuk@gmail.com> wrote:
>>>
>>>> A little correction: in this particular case inputStream does return 0
>>>> which leads to an infinite loop, however, generally this may be not the
>>>> case, so the implementation should not read beyond object boundary anyways.
>>>>
>>>
>>>
>>
>
Re: Externalizable in cache
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Myron,
Thank you for pointing this. it looks like we violate stream contract
because regular Java ObjectOutputStream will return -1 in this case.
Forwarding to dev list.
Igniters,
As Myron mentioned, out implementation of ObjectInput returns 0 when
performing ObjectInput.read(byte[]) and there is no more array data
available. To the contrast, Java's ObjectInputStream returns -1 in this
case.
Looks like we should fix that.
Thoughts?
Vladimir.
On Mon, Feb 15, 2016 at 10:32 PM, Myron Chelyada <my...@gmail.com>
wrote:
> Hello Alexey,
>
> This is actually not real code but just extracted part for test. That's
> why it looks incorrect.
> And I've already captured infinite loop because of 0 being returned. And
> imagine use case when I need to read until end of stream. So question is
> why 0 is returned but not -1 (according to API).
>
>
> 2016-02-15 21:10 GMT+02:00 Dmitriy Setrakyan <ds...@apache.org>:
>
>> Alexey, is this something that Ignite could prevent automatically?
>>
>> On Mon, Feb 15, 2016 at 10:17 AM, Alexey Goncharuk <
>> alexey.goncharuk@gmail.com> wrote:
>>
>>> A little correction: in this particular case inputStream does return 0
>>> which leads to an infinite loop, however, generally this may be not the
>>> case, so the implementation should not read beyond object boundary anyways.
>>>
>>
>>
>
Re: Externalizable in cache
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Myron,
Thank you for pointing this. it looks like we violate stream contract
because regular Java ObjectOutputStream will return -1 in this case.
Forwarding to dev list.
Igniters,
As Myron mentioned, out implementation of ObjectInput returns 0 when
performing ObjectInput.read(byte[]) and there is no more array data
available. To the contrast, Java's ObjectInputStream returns -1 in this
case.
Looks like we should fix that.
Thoughts?
Vladimir.
On Mon, Feb 15, 2016 at 10:32 PM, Myron Chelyada <my...@gmail.com>
wrote:
> Hello Alexey,
>
> This is actually not real code but just extracted part for test. That's
> why it looks incorrect.
> And I've already captured infinite loop because of 0 being returned. And
> imagine use case when I need to read until end of stream. So question is
> why 0 is returned but not -1 (according to API).
>
>
> 2016-02-15 21:10 GMT+02:00 Dmitriy Setrakyan <ds...@apache.org>:
>
>> Alexey, is this something that Ignite could prevent automatically?
>>
>> On Mon, Feb 15, 2016 at 10:17 AM, Alexey Goncharuk <
>> alexey.goncharuk@gmail.com> wrote:
>>
>>> A little correction: in this particular case inputStream does return 0
>>> which leads to an infinite loop, however, generally this may be not the
>>> case, so the implementation should not read beyond object boundary anyways.
>>>
>>
>>
>
Re: Externalizable in cache
Posted by Myron Chelyada <my...@gmail.com>.
Hello Alexey,
This is actually not real code but just extracted part for test. That's why
it looks incorrect.
And I've already captured infinite loop because of 0 being returned. And
imagine use case when I need to read until end of stream. So question is
why 0 is returned but not -1 (according to API).
2016-02-15 21:10 GMT+02:00 Dmitriy Setrakyan <ds...@apache.org>:
> Alexey, is this something that Ignite could prevent automatically?
>
> On Mon, Feb 15, 2016 at 10:17 AM, Alexey Goncharuk <
> alexey.goncharuk@gmail.com> wrote:
>
>> A little correction: in this particular case inputStream does return 0
>> which leads to an infinite loop, however, generally this may be not the
>> case, so the implementation should not read beyond object boundary anyways.
>>
>
>
Re: Externalizable in cache
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Alexey, is this something that Ignite could prevent automatically?
On Mon, Feb 15, 2016 at 10:17 AM, Alexey Goncharuk <
alexey.goncharuk@gmail.com> wrote:
> A little correction: in this particular case inputStream does return 0
> which leads to an infinite loop, however, generally this may be not the
> case, so the implementation should not read beyond object boundary anyways.
>
Re: Externalizable in cache
Posted by Alexey Goncharuk <al...@gmail.com>.
A little correction: in this particular case inputStream does return 0
which leads to an infinite loop, however, generally this may be not the
case, so the implementation should not read beyond object boundary anyways.
Re: Externalizable in cache
Posted by Alexey Goncharuk <al...@gmail.com>.
Hello Myron,
Your implementation of Externalizable interface is incorrect. The number of
bytes that can be read from the object input stream passed to the
readExternal() method is not limited, so you need to make sure that you do
not read more bytes than the number of bytes written.
The correct implementation of Externalizable for your case is below:
@Override public void writeExternal(ObjectOutput out) throws IOException {
out.writeInt(data.length);
out.write(data);
}
@Override public void readExternal(ObjectInput in) throws IOException,
ClassNotFoundException {
int len = in.readInt();
byte[] buffer = new byte[len];
int total = 0;
while (total < len) {
int remaining = len - total;
int read0 = in.read(buffer, total, remaining);
total += read0;
}
data = buffer;
}