You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Alexei Scherbakov <al...@gmail.com> on 2017/09/29 10:29:14 UTC
Support for packed int and long primitives in raw binary API
Guys,
I notices we do not have support for packed ints and longs in raw binary
API [1] [2]
Such methods are essential for implementing efficient custom compression
schemes.
Their addition can simplify implementing custom serializers for the cases
then default binary marshaller is not enough, without additional library
dependencies.
Proposed API extension for rawReader/rawWriter:
org.apache.ignite.binary.BinaryRawWriter#writePackedInt
org.apache.ignite.binary.BinaryRawWriter#writePackedLong
org.apache.ignite.binary.BinaryRawReader#readPackedInt
org.apache.ignite.binary.BinaryRawReader#readPackedLong
JIRA ticket: [3]
Thoughs ?
[1] org.apache.ignite.binary.BinaryRawReader
[2] org.apache.ignite.binary.BinaryRawWriter
[3] https://issues.apache.org/jira/browse/IGNITE-6426
--
Best regards,
Alexei Scherbakov
Re: Support for packed int and long primitives in raw binary API
Posted by Valentin Kulichenko <va...@gmail.com>.
Alexey,
Any reason why you propose this only for raw data? I think this should be
an option for binary marshaller (probably per type). If we also add special
methods to Binarylizable, then they should be applied to both raw and
non-raw data.
BTW, there are couple other tickets regarding this, and looks like there is
already some progress:
- https://issues.apache.org/jira/browse/IGNITE-5097
- https://issues.apache.org/jira/browse/IGNITE-6418
-Val
On Fri, Sep 29, 2017 at 5:37 AM, Alexei Scherbakov <
alexey.scherbakoff@gmail.com> wrote:
> Dmitriy,
>
> Not arrays, just primitives.
>
> Using special binary representation, ints and longs can be represented by
> 1-9 bytes, depending on their cardinality.
>
> I want to add such implementation in raw binary API to help implementing
> custom serializers.
>
>
>
> 2017-09-29 15:12 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
>
> > Alexey, are you talking about arrays of ints and longs?
> >
> > On Fri, Sep 29, 2017 at 3:29 AM, Alexei Scherbakov <
> > alexey.scherbakoff@gmail.com> wrote:
> >
> > > Guys,
> > >
> > > I notices we do not have support for packed ints and longs in raw
> binary
> > > API [1] [2]
> > >
> > > Such methods are essential for implementing efficient custom
> compression
> > > schemes.
> > >
> > > Their addition can simplify implementing custom serializers for the
> cases
> > > then default binary marshaller is not enough, without additional
> library
> > > dependencies.
> > >
> > > Proposed API extension for rawReader/rawWriter:
> > >
> > > org.apache.ignite.binary.BinaryRawWriter#writePackedInt
> > >
> > > org.apache.ignite.binary.BinaryRawWriter#writePackedLong
> > >
> > > org.apache.ignite.binary.BinaryRawReader#readPackedInt
> > >
> > > org.apache.ignite.binary.BinaryRawReader#readPackedLong
> > >
> > > JIRA ticket: [3]
> > >
> > > Thoughs ?
> > >
> > > [1] org.apache.ignite.binary.BinaryRawReader
> > >
> > > [2] org.apache.ignite.binary.BinaryRawWriter
> > >
> > > [3] https://issues.apache.org/jira/browse/IGNITE-6426
> > >
> > > --
> > >
> > > Best regards,
> > > Alexei Scherbakov
> > >
> >
>
>
>
> --
>
> Best regards,
> Alexei Scherbakov
>
Re: Support for packed int and long primitives in raw binary API
Posted by Alexei Scherbakov <al...@gmail.com>.
Dmitriy,
Not arrays, just primitives.
Using special binary representation, ints and longs can be represented by
1-9 bytes, depending on their cardinality.
I want to add such implementation in raw binary API to help implementing
custom serializers.
2017-09-29 15:12 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
> Alexey, are you talking about arrays of ints and longs?
>
> On Fri, Sep 29, 2017 at 3:29 AM, Alexei Scherbakov <
> alexey.scherbakoff@gmail.com> wrote:
>
> > Guys,
> >
> > I notices we do not have support for packed ints and longs in raw binary
> > API [1] [2]
> >
> > Such methods are essential for implementing efficient custom compression
> > schemes.
> >
> > Their addition can simplify implementing custom serializers for the cases
> > then default binary marshaller is not enough, without additional library
> > dependencies.
> >
> > Proposed API extension for rawReader/rawWriter:
> >
> > org.apache.ignite.binary.BinaryRawWriter#writePackedInt
> >
> > org.apache.ignite.binary.BinaryRawWriter#writePackedLong
> >
> > org.apache.ignite.binary.BinaryRawReader#readPackedInt
> >
> > org.apache.ignite.binary.BinaryRawReader#readPackedLong
> >
> > JIRA ticket: [3]
> >
> > Thoughs ?
> >
> > [1] org.apache.ignite.binary.BinaryRawReader
> >
> > [2] org.apache.ignite.binary.BinaryRawWriter
> >
> > [3] https://issues.apache.org/jira/browse/IGNITE-6426
> >
> > --
> >
> > Best regards,
> > Alexei Scherbakov
> >
>
--
Best regards,
Alexei Scherbakov
Re: Support for packed int and long primitives in raw binary API
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Alexey, are you talking about arrays of ints and longs?
On Fri, Sep 29, 2017 at 3:29 AM, Alexei Scherbakov <
alexey.scherbakoff@gmail.com> wrote:
> Guys,
>
> I notices we do not have support for packed ints and longs in raw binary
> API [1] [2]
>
> Such methods are essential for implementing efficient custom compression
> schemes.
>
> Their addition can simplify implementing custom serializers for the cases
> then default binary marshaller is not enough, without additional library
> dependencies.
>
> Proposed API extension for rawReader/rawWriter:
>
> org.apache.ignite.binary.BinaryRawWriter#writePackedInt
>
> org.apache.ignite.binary.BinaryRawWriter#writePackedLong
>
> org.apache.ignite.binary.BinaryRawReader#readPackedInt
>
> org.apache.ignite.binary.BinaryRawReader#readPackedLong
>
> JIRA ticket: [3]
>
> Thoughs ?
>
> [1] org.apache.ignite.binary.BinaryRawReader
>
> [2] org.apache.ignite.binary.BinaryRawWriter
>
> [3] https://issues.apache.org/jira/browse/IGNITE-6426
>
> --
>
> Best regards,
> Alexei Scherbakov
>