You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@ignite.apache.org by Pavel Tupitsyn <pt...@apache.org> on 2017/11/22 13:03:51 UTC

Thin Client Protocol documentation

Igniters,

I've put together a detailed description of our Thin Client protocol
in form of IEP on wiki:
https://cwiki.apache.org/confluence/display/IGNITE/IEP-9+Thin+Client+Protocol


To clarify:
- Protocol implementation is in master (see ClientMessageParser class)
- Protocol has not been released yet, so we are free to change anything
- Protocol is only used by .NET Thin Client for now, but is supposed to be
used from other languages by third party contributors
- More operations will be added in future, this is a first set of them,
cache-related


Please review the document and let me know your thoughts.
Is there anything missing or wrong?

We should make sure that the foundation is solid and extensible.


Thanks,
Pavel

Re: Thin Client Protocol documentation

Posted by Andrey Kornev <an...@hotmail.com>.

>>> I don't think it makes sense to do that. Another problem is that it will take weeks :)

I'm sorry, Pavel. None of what you've said so far makes it obvious why it doesn't make sense to produce the complete spec. Would you care to elaborate, please?

Also, what's the problem with the "another problem" of the effort taking weeks? If based on your best estimate the effort requires a few weeks of your time, why is this a problem then?

>>> But, of course, you are welcome to take the initiative.

Saying something like that is not very helpful, Pavel. But let me just say, I believe that as the current spec lead and an employee of the company behind Apache Ignite project you're in a much better position to continue to lead the effort (you've been doing great so far!). From my side, I'd be more than happy to review the spec and provide my feedback. Also, I'm very interested in becoming a contributor/early adopter of the Java-based implementation of the protocol.

Regards
Andrey

________________________________
From: Pavel Tupitsyn <pt...@gridgain.com>
Sent: Tuesday, December 5, 2017 11:15 PM
To: dev@ignite.apache.org
Subject: Re: Thin Client Protocol documentation

Andrey,

As I understand, you suggest to document every prospective feature right
now.
That would be (at least) compute, clustering, transactions, services,
messages, events, failover, data structures, metrics.

I don't think it makes sense to do that. Another problem is that it will
take weeks :)
But, of course, you are welcome to take the initiative.

Thanks,
Pavel

On Tue, Dec 5, 2017 at 10:20 PM, Andrey Kornev <an...@hotmail.com>
wrote:

> Pavel,
>
> I have absolutely no doubts that support for all those features will be
> added eventually. And if so, wouldn't it be the right thing to do to
> document the operations and their semantics right now without
> necessarily implementing them? It should really help to ensure that the
> protocol can accommodate all those use cases before it gets released to the
> public.
>
> Thanks
> Andrey
> ------------------------------
> *From:* Pavel Tupitsyn <pt...@gridgain.com>
> *Sent:* Tuesday, December 5, 2017 12:07 AM
> *Cc:* dev@ignite.apache.org
> *Subject:* Re: Thin Client Protocol documentation
>
> Andrey,
>
> All of this is to come :)
>
>
> Prachi,
>
> 1) There are no flags, see the doc
> 2) String is simply 4 byte length (n) + n bytes
> 3) Op codes have changed
>
> writeByteLittleEndian(1051, out); // Type code
> writeIntLittleEndian(20, out); // String length
> out.writeUTF("myNewCache"); // Cache name
>
> On Tue, Dec 5, 2017 at 4:39 AM, Prachi Garg <pg...@gridgain.com> wrote:
>
> > Hi Pavel,
> >
> > I am trying to use the OP_CACHE_CREATE_WITH_NAME operation, but can't get
> > it to work.  Digging deeper into the source code, it seems like I have to
> > provide a flag, string length, and position, in addition to the type code
> > and the actual string. Is that correct?
> >
> > Here is the request I am sending to the server-
> >
> > DataOutputStream out = new DataOutputStream(socket.getOutputStream());
> >
> > // Message length
> > writeIntLittleEndian(24, out);
> >
> > // Op code = OP_CACHE_CREATE_WITH_NAME
> > writeShortLittleEndian(1051, out);
> >
> > // Request id
> > long reqId = 1;
> > writeLongLittleEndian(reqId, out);
> >
> > // String (cache name)
> > writeByteLittleEndian(9, out); // Type code
> > writeByteLittleEndian(0, out); // Flag
> > writeIntLittleEndian(20, out); // String length
> > writeIntLittleEndian(0, out); // Position
> > out.writeUTF("myNewCache"); // Cache name
> >
> > // Send request
> > out.flush();
> >
> > But I get the following error on the server side.
> >
> > [2017-12-04 17:27:39,421][ERROR][client-connector-#53][
> ClientListenerNioListener]
> > Failed to parse client request.
> > java.lang.StringIndexOutOfBoundsException: String index out of range:
> 2575
> > at java.lang.String.checkBounds(String.java:385)
> > at java.lang.String.<init>(String.java:462)
> > at org.apache.ignite.internal.binary.BinaryUtils.
> > doReadString(BinaryUtils.java:1314)
> > at org.apache.ignite.internal.binary.BinaryReaderExImpl.
> > readString(BinaryReaderExImpl.java:1055)
> > at org.apache.ignite.internal.processors.platform.client.cache.
> > ClientCacheCreateWithNameRequest.<init>(ClientCacheCreateWithNameReque
> > st.java:43)
> > at org.apache.ignite.internal.processors.platform.client.
> > ClientMessageParser.decode(ClientMessageParser.java:318)
> > at org.apache.ignite.internal.processors.platform.client.
> > ClientMessageParser.decode(ClientMessageParser.java:220)
> > at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.
> > onMessage(ClientListenerNioListener.java:119)
> > at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.
> > onMessage(ClientListenerNioListener.java:40)
> > at org.apache.ignite.internal.util.nio.GridNioFilterChain$
> > TailFilter.onMessageReceived(GridNioFilterChain.java:279)
> > at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.
> > proceedMessageReceived(GridNioFilterAdapter.java:109)
> > at org.apache.ignite.internal.util.nio.GridNioAsyncNotifyFilter$3.
> > body(GridNioAsyncNotifyFilter.java:97)
> > at org.apache.ignite.internal.util.worker.GridWorker.run(
> > GridWorker.java:110)
> > at org.apache.ignite.internal.util.worker.GridWorkerPool$1.
> > run(GridWorkerPool.java:70)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1142)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > What am I missing here? Can you provide an example of how to send a
> > 'String' type request?
> >
> > -Prachi
> >
> > On Mon, Dec 4, 2017 at 1:06 PM, Andrey Kornev <an...@hotmail.com>
> > wrote:
> >
> >> Pavel,
> >>
> >> Thanks! While we're at it, are there any plans to add cluster-related
> >> operations? For example, I think it'd be nice to allow the thin clients
> to
> >> obtain a current topology snapshot. This would make it possible the
> clients
> >> to send requests directly to the affinity host for colocated
> computation.
> >> To make it even more useful, all server responses could optionally
> include
> >> the topology version the operation has been executed against. This would
> >> effectively give us a kind out-of-band topology change notification
> >> mechanism. This way the clients can detect a topology change and refresh
> >> the topology snapshot next time they need to compute affinity.
> >>
> >> Regards
> >> Andrey
> >> ________________________________
> >> From: Pavel Tupitsyn <pt...@apache.org>
> >> Sent: Sunday, December 3, 2017 9:23 AM
> >> To: dev@ignite.apache.org
> >> Subject: Re: Thin Client Protocol documentation
> >>
> >> Hi Andrey,
> >>
> >> Compute and other APIs are certainly planned, cache is just a start.
> >> We intentionally limit the scope to actually release something in 2.4
> and
> >> not delay it further.
> >>
> >> Adding operations to existing protocol is relatively easy.
> >> Current focus is to make sure that the protocol itself is solid and
> >> future-proof.
> >>
> >> Thanks,
> >> Pavel
> >>
> >
> >
>

Re: Thin Client Protocol documentation

Posted by Denis Magda <dm...@apache.org>.

Hi Andrey,

I’m afraid there are no plans for this features yet (at least I didn’t spot any relevant discussion on @dev).

Feel free to start a separate discussion proposing the protocol expansion. It’s better not to intermix this with the documentation thread.

—
Denis

> On Dec 15, 2017, at 11:12 AM, Andrey Kornev <an...@hotmail.com> wrote:
> 
> Pavel, could you please at least share your time frame wrt adding compute, clustering and services features to the spec?
> 
> Thanks
> Andrey
> ________________________________
> From: Pavel Tupitsyn <pt...@gridgain.com>
> Sent: Tuesday, December 5, 2017 11:15 PM
> To: dev@ignite.apache.org
> Subject: Re: Thin Client Protocol documentation
> 
> Andrey,
> 
> As I understand, you suggest to document every prospective feature right
> now.
> That would be (at least) compute, clustering, transactions, services,
> messages, events, failover, data structures, metrics.
> 
> I don't think it makes sense to do that. Another problem is that it will
> take weeks :)
> But, of course, you are welcome to take the initiative.
> 
> Thanks,
> Pavel
> 
> On Tue, Dec 5, 2017 at 10:20 PM, Andrey Kornev <an...@hotmail.com>
> wrote:
> 
>> Pavel,
>> 
>> I have absolutely no doubts that support for all those features will be
>> added eventually. And if so, wouldn't it be the right thing to do to
>> document the operations and their semantics right now without
>> necessarily implementing them? It should really help to ensure that the
>> protocol can accommodate all those use cases before it gets released to the
>> public.
>> 
>> Thanks
>> Andrey
>> ------------------------------
>> *From:* Pavel Tupitsyn <pt...@gridgain.com>
>> *Sent:* Tuesday, December 5, 2017 12:07 AM
>> *Cc:* dev@ignite.apache.org
>> *Subject:* Re: Thin Client Protocol documentation
>> 
>> Andrey,
>> 
>> All of this is to come :)
>> 
>> 
>> Prachi,
>> 
>> 1) There are no flags, see the doc
>> 2) String is simply 4 byte length (n) + n bytes
>> 3) Op codes have changed
>> 
>> writeByteLittleEndian(1051, out); // Type code
>> writeIntLittleEndian(20, out); // String length
>> out.writeUTF("myNewCache"); // Cache name
>> 
>> On Tue, Dec 5, 2017 at 4:39 AM, Prachi Garg <pg...@gridgain.com> wrote:
>> 
>>> Hi Pavel,
>>> 
>>> I am trying to use the OP_CACHE_CREATE_WITH_NAME operation, but can't get
>>> it to work.  Digging deeper into the source code, it seems like I have to
>>> provide a flag, string length, and position, in addition to the type code
>>> and the actual string. Is that correct?
>>> 
>>> Here is the request I am sending to the server-
>>> 
>>> DataOutputStream out = new DataOutputStream(socket.getOutputStream());
>>> 
>>> // Message length
>>> writeIntLittleEndian(24, out);
>>> 
>>> // Op code = OP_CACHE_CREATE_WITH_NAME
>>> writeShortLittleEndian(1051, out);
>>> 
>>> // Request id
>>> long reqId = 1;
>>> writeLongLittleEndian(reqId, out);
>>> 
>>> // String (cache name)
>>> writeByteLittleEndian(9, out); // Type code
>>> writeByteLittleEndian(0, out); // Flag
>>> writeIntLittleEndian(20, out); // String length
>>> writeIntLittleEndian(0, out); // Position
>>> out.writeUTF("myNewCache"); // Cache name
>>> 
>>> // Send request
>>> out.flush();
>>> 
>>> But I get the following error on the server side.
>>> 
>>> [2017-12-04 17:27:39,421][ERROR][client-connector-#53][
>> ClientListenerNioListener]
>>> Failed to parse client request.
>>> java.lang.StringIndexOutOfBoundsException: String index out of range:
>> 2575
>>> at java.lang.String.checkBounds(String.java:385)
>>> at java.lang.String.<init>(String.java:462)
>>> at org.apache.ignite.internal.binary.BinaryUtils.
>>> doReadString(BinaryUtils.java:1314)
>>> at org.apache.ignite.internal.binary.BinaryReaderExImpl.
>>> readString(BinaryReaderExImpl.java:1055)
>>> at org.apache.ignite.internal.processors.platform.client.cache.
>>> ClientCacheCreateWithNameRequest.<init>(ClientCacheCreateWithNameReque
>>> st.java:43)
>>> at org.apache.ignite.internal.processors.platform.client.
>>> ClientMessageParser.decode(ClientMessageParser.java:318)
>>> at org.apache.ignite.internal.processors.platform.client.
>>> ClientMessageParser.decode(ClientMessageParser.java:220)
>>> at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.
>>> onMessage(ClientListenerNioListener.java:119)
>>> at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.
>>> onMessage(ClientListenerNioListener.java:40)
>>> at org.apache.ignite.internal.util.nio.GridNioFilterChain$
>>> TailFilter.onMessageReceived(GridNioFilterChain.java:279)
>>> at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.
>>> proceedMessageReceived(GridNioFilterAdapter.java:109)
>>> at org.apache.ignite.internal.util.nio.GridNioAsyncNotifyFilter$3.
>>> body(GridNioAsyncNotifyFilter.java:97)
>>> at org.apache.ignite.internal.util.worker.GridWorker.run(
>>> GridWorker.java:110)
>>> at org.apache.ignite.internal.util.worker.GridWorkerPool$1.
>>> run(GridWorkerPool.java:70)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>> ThreadPoolExecutor.java:1142)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>> ThreadPoolExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:745)
>>> 
>>> What am I missing here? Can you provide an example of how to send a
>>> 'String' type request?
>>> 
>>> -Prachi
>>> 
>>> On Mon, Dec 4, 2017 at 1:06 PM, Andrey Kornev <an...@hotmail.com>
>>> wrote:
>>> 
>>>> Pavel,
>>>> 
>>>> Thanks! While we're at it, are there any plans to add cluster-related
>>>> operations? For example, I think it'd be nice to allow the thin clients
>> to
>>>> obtain a current topology snapshot. This would make it possible the
>> clients
>>>> to send requests directly to the affinity host for colocated
>> computation.
>>>> To make it even more useful, all server responses could optionally
>> include
>>>> the topology version the operation has been executed against. This would
>>>> effectively give us a kind out-of-band topology change notification
>>>> mechanism. This way the clients can detect a topology change and refresh
>>>> the topology snapshot next time they need to compute affinity.
>>>> 
>>>> Regards
>>>> Andrey
>>>> ________________________________
>>>> From: Pavel Tupitsyn <pt...@apache.org>
>>>> Sent: Sunday, December 3, 2017 9:23 AM
>>>> To: dev@ignite.apache.org
>>>> Subject: Re: Thin Client Protocol documentation
>>>> 
>>>> Hi Andrey,
>>>> 
>>>> Compute and other APIs are certainly planned, cache is just a start.
>>>> We intentionally limit the scope to actually release something in 2.4
>> and
>>>> not delay it further.
>>>> 
>>>> Adding operations to existing protocol is relatively easy.
>>>> Current focus is to make sure that the protocol itself is solid and
>>>> future-proof.
>>>> 
>>>> Thanks,
>>>> Pavel
>>>> 
>>> 
>>> 
>>

Re: Thin Client Protocol documentation

Posted by Andrey Kornev <an...@hotmail.com>.

Pavel, could you please at least share your time frame wrt adding compute, clustering and services features to the spec?

Thanks
Andrey
________________________________
From: Pavel Tupitsyn <pt...@gridgain.com>
Sent: Tuesday, December 5, 2017 11:15 PM
To: dev@ignite.apache.org
Subject: Re: Thin Client Protocol documentation

Andrey,

As I understand, you suggest to document every prospective feature right
now.
That would be (at least) compute, clustering, transactions, services,
messages, events, failover, data structures, metrics.

I don't think it makes sense to do that. Another problem is that it will
take weeks :)
But, of course, you are welcome to take the initiative.

Thanks,
Pavel

On Tue, Dec 5, 2017 at 10:20 PM, Andrey Kornev <an...@hotmail.com>
wrote:

> Pavel,
>
> I have absolutely no doubts that support for all those features will be
> added eventually. And if so, wouldn't it be the right thing to do to
> document the operations and their semantics right now without
> necessarily implementing them? It should really help to ensure that the
> protocol can accommodate all those use cases before it gets released to the
> public.
>
> Thanks
> Andrey
> ------------------------------
> *From:* Pavel Tupitsyn <pt...@gridgain.com>
> *Sent:* Tuesday, December 5, 2017 12:07 AM
> *Cc:* dev@ignite.apache.org
> *Subject:* Re: Thin Client Protocol documentation
>
> Andrey,
>
> All of this is to come :)
>
>
> Prachi,
>
> 1) There are no flags, see the doc
> 2) String is simply 4 byte length (n) + n bytes
> 3) Op codes have changed
>
> writeByteLittleEndian(1051, out); // Type code
> writeIntLittleEndian(20, out); // String length
> out.writeUTF("myNewCache"); // Cache name
>
> On Tue, Dec 5, 2017 at 4:39 AM, Prachi Garg <pg...@gridgain.com> wrote:
>
> > Hi Pavel,
> >
> > I am trying to use the OP_CACHE_CREATE_WITH_NAME operation, but can't get
> > it to work.  Digging deeper into the source code, it seems like I have to
> > provide a flag, string length, and position, in addition to the type code
> > and the actual string. Is that correct?
> >
> > Here is the request I am sending to the server-
> >
> > DataOutputStream out = new DataOutputStream(socket.getOutputStream());
> >
> > // Message length
> > writeIntLittleEndian(24, out);
> >
> > // Op code = OP_CACHE_CREATE_WITH_NAME
> > writeShortLittleEndian(1051, out);
> >
> > // Request id
> > long reqId = 1;
> > writeLongLittleEndian(reqId, out);
> >
> > // String (cache name)
> > writeByteLittleEndian(9, out); // Type code
> > writeByteLittleEndian(0, out); // Flag
> > writeIntLittleEndian(20, out); // String length
> > writeIntLittleEndian(0, out); // Position
> > out.writeUTF("myNewCache"); // Cache name
> >
> > // Send request
> > out.flush();
> >
> > But I get the following error on the server side.
> >
> > [2017-12-04 17:27:39,421][ERROR][client-connector-#53][
> ClientListenerNioListener]
> > Failed to parse client request.
> > java.lang.StringIndexOutOfBoundsException: String index out of range:
> 2575
> > at java.lang.String.checkBounds(String.java:385)
> > at java.lang.String.<init>(String.java:462)
> > at org.apache.ignite.internal.binary.BinaryUtils.
> > doReadString(BinaryUtils.java:1314)
> > at org.apache.ignite.internal.binary.BinaryReaderExImpl.
> > readString(BinaryReaderExImpl.java:1055)
> > at org.apache.ignite.internal.processors.platform.client.cache.
> > ClientCacheCreateWithNameRequest.<init>(ClientCacheCreateWithNameReque
> > st.java:43)
> > at org.apache.ignite.internal.processors.platform.client.
> > ClientMessageParser.decode(ClientMessageParser.java:318)
> > at org.apache.ignite.internal.processors.platform.client.
> > ClientMessageParser.decode(ClientMessageParser.java:220)
> > at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.
> > onMessage(ClientListenerNioListener.java:119)
> > at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.
> > onMessage(ClientListenerNioListener.java:40)
> > at org.apache.ignite.internal.util.nio.GridNioFilterChain$
> > TailFilter.onMessageReceived(GridNioFilterChain.java:279)
> > at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.
> > proceedMessageReceived(GridNioFilterAdapter.java:109)
> > at org.apache.ignite.internal.util.nio.GridNioAsyncNotifyFilter$3.
> > body(GridNioAsyncNotifyFilter.java:97)
> > at org.apache.ignite.internal.util.worker.GridWorker.run(
> > GridWorker.java:110)
> > at org.apache.ignite.internal.util.worker.GridWorkerPool$1.
> > run(GridWorkerPool.java:70)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1142)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > What am I missing here? Can you provide an example of how to send a
> > 'String' type request?
> >
> > -Prachi
> >
> > On Mon, Dec 4, 2017 at 1:06 PM, Andrey Kornev <an...@hotmail.com>
> > wrote:
> >
> >> Pavel,
> >>
> >> Thanks! While we're at it, are there any plans to add cluster-related
> >> operations? For example, I think it'd be nice to allow the thin clients
> to
> >> obtain a current topology snapshot. This would make it possible the
> clients
> >> to send requests directly to the affinity host for colocated
> computation.
> >> To make it even more useful, all server responses could optionally
> include
> >> the topology version the operation has been executed against. This would
> >> effectively give us a kind out-of-band topology change notification
> >> mechanism. This way the clients can detect a topology change and refresh
> >> the topology snapshot next time they need to compute affinity.
> >>
> >> Regards
> >> Andrey
> >> ________________________________
> >> From: Pavel Tupitsyn <pt...@apache.org>
> >> Sent: Sunday, December 3, 2017 9:23 AM
> >> To: dev@ignite.apache.org
> >> Subject: Re: Thin Client Protocol documentation
> >>
> >> Hi Andrey,
> >>
> >> Compute and other APIs are certainly planned, cache is just a start.
> >> We intentionally limit the scope to actually release something in 2.4
> and
> >> not delay it further.
> >>
> >> Adding operations to existing protocol is relatively easy.
> >> Current focus is to make sure that the protocol itself is solid and
> >> future-proof.
> >>
> >> Thanks,
> >> Pavel
> >>
> >
> >
>

Re: Thin Client Protocol documentation

Posted by Pavel Tupitsyn <pt...@gridgain.com>.

Andrey,

As I understand, you suggest to document every prospective feature right
now.
That would be (at least) compute, clustering, transactions, services,
messages, events, failover, data structures, metrics.

I don't think it makes sense to do that. Another problem is that it will
take weeks :)
But, of course, you are welcome to take the initiative.

Thanks,
Pavel

On Tue, Dec 5, 2017 at 10:20 PM, Andrey Kornev <an...@hotmail.com>
wrote:

> Pavel,
>
> I have absolutely no doubts that support for all those features will be
> added eventually. And if so, wouldn't it be the right thing to do to
> document the operations and their semantics right now without
> necessarily implementing them? It should really help to ensure that the
> protocol can accommodate all those use cases before it gets released to the
> public.
>
> Thanks
> Andrey
> ------------------------------
> *From:* Pavel Tupitsyn <pt...@gridgain.com>
> *Sent:* Tuesday, December 5, 2017 12:07 AM
> *Cc:* dev@ignite.apache.org
> *Subject:* Re: Thin Client Protocol documentation
>
> Andrey,
>
> All of this is to come :)
>
>
> Prachi,
>
> 1) There are no flags, see the doc
> 2) String is simply 4 byte length (n) + n bytes
> 3) Op codes have changed
>
> writeByteLittleEndian(1051, out); // Type code
> writeIntLittleEndian(20, out); // String length
> out.writeUTF("myNewCache"); // Cache name
>
> On Tue, Dec 5, 2017 at 4:39 AM, Prachi Garg <pg...@gridgain.com> wrote:
>
> > Hi Pavel,
> >
> > I am trying to use the OP_CACHE_CREATE_WITH_NAME operation, but can't get
> > it to work.  Digging deeper into the source code, it seems like I have to
> > provide a flag, string length, and position, in addition to the type code
> > and the actual string. Is that correct?
> >
> > Here is the request I am sending to the server-
> >
> > DataOutputStream out = new DataOutputStream(socket.getOutputStream());
> >
> > // Message length
> > writeIntLittleEndian(24, out);
> >
> > // Op code = OP_CACHE_CREATE_WITH_NAME
> > writeShortLittleEndian(1051, out);
> >
> > // Request id
> > long reqId = 1;
> > writeLongLittleEndian(reqId, out);
> >
> > // String (cache name)
> > writeByteLittleEndian(9, out); // Type code
> > writeByteLittleEndian(0, out); // Flag
> > writeIntLittleEndian(20, out); // String length
> > writeIntLittleEndian(0, out); // Position
> > out.writeUTF("myNewCache"); // Cache name
> >
> > // Send request
> > out.flush();
> >
> > But I get the following error on the server side.
> >
> > [2017-12-04 17:27:39,421][ERROR][client-connector-#53][
> ClientListenerNioListener]
> > Failed to parse client request.
> > java.lang.StringIndexOutOfBoundsException: String index out of range:
> 2575
> > at java.lang.String.checkBounds(String.java:385)
> > at java.lang.String.<init>(String.java:462)
> > at org.apache.ignite.internal.binary.BinaryUtils.
> > doReadString(BinaryUtils.java:1314)
> > at org.apache.ignite.internal.binary.BinaryReaderExImpl.
> > readString(BinaryReaderExImpl.java:1055)
> > at org.apache.ignite.internal.processors.platform.client.cache.
> > ClientCacheCreateWithNameRequest.<init>(ClientCacheCreateWithNameReque
> > st.java:43)
> > at org.apache.ignite.internal.processors.platform.client.
> > ClientMessageParser.decode(ClientMessageParser.java:318)
> > at org.apache.ignite.internal.processors.platform.client.
> > ClientMessageParser.decode(ClientMessageParser.java:220)
> > at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.
> > onMessage(ClientListenerNioListener.java:119)
> > at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.
> > onMessage(ClientListenerNioListener.java:40)
> > at org.apache.ignite.internal.util.nio.GridNioFilterChain$
> > TailFilter.onMessageReceived(GridNioFilterChain.java:279)
> > at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.
> > proceedMessageReceived(GridNioFilterAdapter.java:109)
> > at org.apache.ignite.internal.util.nio.GridNioAsyncNotifyFilter$3.
> > body(GridNioAsyncNotifyFilter.java:97)
> > at org.apache.ignite.internal.util.worker.GridWorker.run(
> > GridWorker.java:110)
> > at org.apache.ignite.internal.util.worker.GridWorkerPool$1.
> > run(GridWorkerPool.java:70)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1142)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > What am I missing here? Can you provide an example of how to send a
> > 'String' type request?
> >
> > -Prachi
> >
> > On Mon, Dec 4, 2017 at 1:06 PM, Andrey Kornev <an...@hotmail.com>
> > wrote:
> >
> >> Pavel,
> >>
> >> Thanks! While we're at it, are there any plans to add cluster-related
> >> operations? For example, I think it'd be nice to allow the thin clients
> to
> >> obtain a current topology snapshot. This would make it possible the
> clients
> >> to send requests directly to the affinity host for colocated
> computation.
> >> To make it even more useful, all server responses could optionally
> include
> >> the topology version the operation has been executed against. This would
> >> effectively give us a kind out-of-band topology change notification
> >> mechanism. This way the clients can detect a topology change and refresh
> >> the topology snapshot next time they need to compute affinity.
> >>
> >> Regards
> >> Andrey
> >> ________________________________
> >> From: Pavel Tupitsyn <pt...@apache.org>
> >> Sent: Sunday, December 3, 2017 9:23 AM
> >> To: dev@ignite.apache.org
> >> Subject: Re: Thin Client Protocol documentation
> >>
> >> Hi Andrey,
> >>
> >> Compute and other APIs are certainly planned, cache is just a start.
> >> We intentionally limit the scope to actually release something in 2.4
> and
> >> not delay it further.
> >>
> >> Adding operations to existing protocol is relatively easy.
> >> Current focus is to make sure that the protocol itself is solid and
> >> future-proof.
> >>
> >> Thanks,
> >> Pavel
> >>
> >
> >
>

Re: Thin Client Protocol documentation

Posted by Andrey Kornev <an...@hotmail.com>.

Pavel,

I have absolutely no doubts that support for all those features will be added eventually. And if so, wouldn't it be the right thing to do to document the operations and their semantics right now without necessarily implementing them? It should really help to ensure that the protocol can accommodate all those use cases before it gets released to the public.

Thanks
Andrey
________________________________
From: Pavel Tupitsyn <pt...@gridgain.com>
Sent: Tuesday, December 5, 2017 12:07 AM
Cc: dev@ignite.apache.org
Subject: Re: Thin Client Protocol documentation

Andrey,

All of this is to come :)


Prachi,

1) There are no flags, see the doc
2) String is simply 4 byte length (n) + n bytes
3) Op codes have changed

writeByteLittleEndian(1051, out); // Type code
writeIntLittleEndian(20, out); // String length
out.writeUTF("myNewCache"); // Cache name

On Tue, Dec 5, 2017 at 4:39 AM, Prachi Garg <pg...@gridgain.com> wrote:

> Hi Pavel,
>
> I am trying to use the OP_CACHE_CREATE_WITH_NAME operation, but can't get
> it to work.  Digging deeper into the source code, it seems like I have to
> provide a flag, string length, and position, in addition to the type code
> and the actual string. Is that correct?
>
> Here is the request I am sending to the server-
>
> DataOutputStream out = new DataOutputStream(socket.getOutputStream());
>
> // Message length
> writeIntLittleEndian(24, out);
>
> // Op code = OP_CACHE_CREATE_WITH_NAME
> writeShortLittleEndian(1051, out);
>
> // Request id
> long reqId = 1;
> writeLongLittleEndian(reqId, out);
>
> // String (cache name)
> writeByteLittleEndian(9, out); // Type code
> writeByteLittleEndian(0, out); // Flag
> writeIntLittleEndian(20, out); // String length
> writeIntLittleEndian(0, out); // Position
> out.writeUTF("myNewCache"); // Cache name
>
> // Send request
> out.flush();
>
> But I get the following error on the server side.
>
> [2017-12-04 17:27:39,421][ERROR][client-connector-#53][ClientListenerNioListener]
> Failed to parse client request.
> java.lang.StringIndexOutOfBoundsException: String index out of range: 2575
> at java.lang.String.checkBounds(String.java:385)
> at java.lang.String.<init>(String.java:462)
> at org.apache.ignite.internal.binary.BinaryUtils.
> doReadString(BinaryUtils.java:1314)
> at org.apache.ignite.internal.binary.BinaryReaderExImpl.
> readString(BinaryReaderExImpl.java:1055)
> at org.apache.ignite.internal.processors.platform.client.cache.
> ClientCacheCreateWithNameRequest.<init>(ClientCacheCreateWithNameReque
> st.java:43)
> at org.apache.ignite.internal.processors.platform.client.
> ClientMessageParser.decode(ClientMessageParser.java:318)
> at org.apache.ignite.internal.processors.platform.client.
> ClientMessageParser.decode(ClientMessageParser.java:220)
> at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.
> onMessage(ClientListenerNioListener.java:119)
> at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.
> onMessage(ClientListenerNioListener.java:40)
> at org.apache.ignite.internal.util.nio.GridNioFilterChain$
> TailFilter.onMessageReceived(GridNioFilterChain.java:279)
> at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.
> proceedMessageReceived(GridNioFilterAdapter.java:109)
> at org.apache.ignite.internal.util.nio.GridNioAsyncNotifyFilter$3.
> body(GridNioAsyncNotifyFilter.java:97)
> at org.apache.ignite.internal.util.worker.GridWorker.run(
> GridWorker.java:110)
> at org.apache.ignite.internal.util.worker.GridWorkerPool$1.
> run(GridWorkerPool.java:70)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> What am I missing here? Can you provide an example of how to send a
> 'String' type request?
>
> -Prachi
>
> On Mon, Dec 4, 2017 at 1:06 PM, Andrey Kornev <an...@hotmail.com>
> wrote:
>
>> Pavel,
>>
>> Thanks! While we're at it, are there any plans to add cluster-related
>> operations? For example, I think it'd be nice to allow the thin clients to
>> obtain a current topology snapshot. This would make it possible the clients
>> to send requests directly to the affinity host for colocated computation.
>> To make it even more useful, all server responses could optionally include
>> the topology version the operation has been executed against. This would
>> effectively give us a kind out-of-band topology change notification
>> mechanism. This way the clients can detect a topology change and refresh
>> the topology snapshot next time they need to compute affinity.
>>
>> Regards
>> Andrey
>> ________________________________
>> From: Pavel Tupitsyn <pt...@apache.org>
>> Sent: Sunday, December 3, 2017 9:23 AM
>> To: dev@ignite.apache.org
>> Subject: Re: Thin Client Protocol documentation
>>
>> Hi Andrey,
>>
>> Compute and other APIs are certainly planned, cache is just a start.
>> We intentionally limit the scope to actually release something in 2.4 and
>> not delay it further.
>>
>> Adding operations to existing protocol is relatively easy.
>> Current focus is to make sure that the protocol itself is solid and
>> future-proof.
>>
>> Thanks,
>> Pavel
>>
>
>

Re: Thin Client Protocol documentation

Posted by Pavel Tupitsyn <pt...@gridgain.com>.

Andrey,

All of this is to come :)


Prachi,

1) There are no flags, see the doc
2) String is simply 4 byte length (n) + n bytes
3) Op codes have changed

writeByteLittleEndian(1051, out); // Type code
writeIntLittleEndian(20, out); // String length
out.writeUTF("myNewCache"); // Cache name

On Tue, Dec 5, 2017 at 4:39 AM, Prachi Garg <pg...@gridgain.com> wrote:

> Hi Pavel,
>
> I am trying to use the OP_CACHE_CREATE_WITH_NAME operation, but can't get
> it to work.  Digging deeper into the source code, it seems like I have to
> provide a flag, string length, and position, in addition to the type code
> and the actual string. Is that correct?
>
> Here is the request I am sending to the server-
>
> DataOutputStream out = new DataOutputStream(socket.getOutputStream());
>
> // Message length
> writeIntLittleEndian(24, out);
>
> // Op code = OP_CACHE_CREATE_WITH_NAME
> writeShortLittleEndian(1051, out);
>
> // Request id
> long reqId = 1;
> writeLongLittleEndian(reqId, out);
>
> // String (cache name)
> writeByteLittleEndian(9, out); // Type code
> writeByteLittleEndian(0, out); // Flag
> writeIntLittleEndian(20, out); // String length
> writeIntLittleEndian(0, out); // Position
> out.writeUTF("myNewCache"); // Cache name
>
> // Send request
> out.flush();
>
> But I get the following error on the server side.
>
> [2017-12-04 17:27:39,421][ERROR][client-connector-#53][ClientListenerNioListener]
> Failed to parse client request.
> java.lang.StringIndexOutOfBoundsException: String index out of range: 2575
> at java.lang.String.checkBounds(String.java:385)
> at java.lang.String.<init>(String.java:462)
> at org.apache.ignite.internal.binary.BinaryUtils.
> doReadString(BinaryUtils.java:1314)
> at org.apache.ignite.internal.binary.BinaryReaderExImpl.
> readString(BinaryReaderExImpl.java:1055)
> at org.apache.ignite.internal.processors.platform.client.cache.
> ClientCacheCreateWithNameRequest.<init>(ClientCacheCreateWithNameReque
> st.java:43)
> at org.apache.ignite.internal.processors.platform.client.
> ClientMessageParser.decode(ClientMessageParser.java:318)
> at org.apache.ignite.internal.processors.platform.client.
> ClientMessageParser.decode(ClientMessageParser.java:220)
> at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.
> onMessage(ClientListenerNioListener.java:119)
> at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.
> onMessage(ClientListenerNioListener.java:40)
> at org.apache.ignite.internal.util.nio.GridNioFilterChain$
> TailFilter.onMessageReceived(GridNioFilterChain.java:279)
> at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.
> proceedMessageReceived(GridNioFilterAdapter.java:109)
> at org.apache.ignite.internal.util.nio.GridNioAsyncNotifyFilter$3.
> body(GridNioAsyncNotifyFilter.java:97)
> at org.apache.ignite.internal.util.worker.GridWorker.run(
> GridWorker.java:110)
> at org.apache.ignite.internal.util.worker.GridWorkerPool$1.
> run(GridWorkerPool.java:70)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> What am I missing here? Can you provide an example of how to send a
> 'String' type request?
>
> -Prachi
>
> On Mon, Dec 4, 2017 at 1:06 PM, Andrey Kornev <an...@hotmail.com>
> wrote:
>
>> Pavel,
>>
>> Thanks! While we're at it, are there any plans to add cluster-related
>> operations? For example, I think it'd be nice to allow the thin clients to
>> obtain a current topology snapshot. This would make it possible the clients
>> to send requests directly to the affinity host for colocated computation.
>> To make it even more useful, all server responses could optionally include
>> the topology version the operation has been executed against. This would
>> effectively give us a kind out-of-band topology change notification
>> mechanism. This way the clients can detect a topology change and refresh
>> the topology snapshot next time they need to compute affinity.
>>
>> Regards
>> Andrey
>> ________________________________
>> From: Pavel Tupitsyn <pt...@apache.org>
>> Sent: Sunday, December 3, 2017 9:23 AM
>> To: dev@ignite.apache.org
>> Subject: Re: Thin Client Protocol documentation
>>
>> Hi Andrey,
>>
>> Compute and other APIs are certainly planned, cache is just a start.
>> We intentionally limit the scope to actually release something in 2.4 and
>> not delay it further.
>>
>> Adding operations to existing protocol is relatively easy.
>> Current focus is to make sure that the protocol itself is solid and
>> future-proof.
>>
>> Thanks,
>> Pavel
>>
>
>

Re: Thin Client Protocol documentation

Posted by Prachi Garg <pg...@gridgain.com>.

Hi Pavel,

I am trying to use the OP_CACHE_CREATE_WITH_NAME operation, but can't get
it to work.  Digging deeper into the source code, it seems like I have to
provide a flag, string length, and position, in addition to the type code
and the actual string. Is that correct?

Here is the request I am sending to the server-

DataOutputStream out = new DataOutputStream(socket.getOutputStream());

// Message length
writeIntLittleEndian(24, out);

// Op code = OP_CACHE_CREATE_WITH_NAME
writeShortLittleEndian(1051, out);

// Request id
long reqId = 1;
writeLongLittleEndian(reqId, out);

// String (cache name)
writeByteLittleEndian(9, out); // Type code
writeByteLittleEndian(0, out); // Flag
writeIntLittleEndian(20, out); // String length
writeIntLittleEndian(0, out); // Position
out.writeUTF("myNewCache"); // Cache name

// Send request
out.flush();

But I get the following error on the server side.

[2017-12-04
17:27:39,421][ERROR][client-connector-#53][ClientListenerNioListener]
Failed to parse client request.
java.lang.StringIndexOutOfBoundsException: String index out of range: 2575
at java.lang.String.checkBounds(String.java:385)
at java.lang.String.<init>(String.java:462)
at
org.apache.ignite.internal.binary.BinaryUtils.doReadString(BinaryUtils.java:1314)
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.readString(BinaryReaderExImpl.java:1055)
at
org.apache.ignite.internal.processors.platform.client.cache.ClientCacheCreateWithNameRequest.<init>(ClientCacheCreateWithNameRequest.java:43)
at
org.apache.ignite.internal.processors.platform.client.ClientMessageParser.decode(ClientMessageParser.java:318)
at
org.apache.ignite.internal.processors.platform.client.ClientMessageParser.decode(ClientMessageParser.java:220)
at
org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.onMessage(ClientListenerNioListener.java:119)
at
org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.onMessage(ClientListenerNioListener.java:40)
at
org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onMessageReceived(GridNioFilterChain.java:279)
at
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
at
org.apache.ignite.internal.util.nio.GridNioAsyncNotifyFilter$3.body(GridNioAsyncNotifyFilter.java:97)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
org.apache.ignite.internal.util.worker.GridWorkerPool$1.run(GridWorkerPool.java:70)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

What am I missing here? Can you provide an example of how to send a
'String' type request?

-Prachi

On Mon, Dec 4, 2017 at 1:06 PM, Andrey Kornev <an...@hotmail.com>
wrote:

> Pavel,
>
> Thanks! While we're at it, are there any plans to add cluster-related
> operations? For example, I think it'd be nice to allow the thin clients to
> obtain a current topology snapshot. This would make it possible the clients
> to send requests directly to the affinity host for colocated computation.
> To make it even more useful, all server responses could optionally include
> the topology version the operation has been executed against. This would
> effectively give us a kind out-of-band topology change notification
> mechanism. This way the clients can detect a topology change and refresh
> the topology snapshot next time they need to compute affinity.
>
> Regards
> Andrey
> ________________________________
> From: Pavel Tupitsyn <pt...@apache.org>
> Sent: Sunday, December 3, 2017 9:23 AM
> To: dev@ignite.apache.org
> Subject: Re: Thin Client Protocol documentation
>
> Hi Andrey,
>
> Compute and other APIs are certainly planned, cache is just a start.
> We intentionally limit the scope to actually release something in 2.4 and
> not delay it further.
>
> Adding operations to existing protocol is relatively easy.
> Current focus is to make sure that the protocol itself is solid and
> future-proof.
>
> Thanks,
> Pavel
>

Re: Thin Client Protocol documentation

Posted by Andrey Kornev <an...@hotmail.com>.

Pavel,

Thanks! While we're at it, are there any plans to add cluster-related operations? For example, I think it'd be nice to allow the thin clients to obtain a current topology snapshot. This would make it possible the clients to send requests directly to the affinity host for colocated computation. To make it even more useful, all server responses could optionally include the topology version the operation has been executed against. This would effectively give us a kind out-of-band topology change notification mechanism. This way the clients can detect a topology change and refresh the topology snapshot next time they need to compute affinity.

Regards
Andrey
________________________________
From: Pavel Tupitsyn <pt...@apache.org>
Sent: Sunday, December 3, 2017 9:23 AM
To: dev@ignite.apache.org
Subject: Re: Thin Client Protocol documentation

Hi Andrey,

Compute and other APIs are certainly planned, cache is just a start.
We intentionally limit the scope to actually release something in 2.4 and
not delay it further.

Adding operations to existing protocol is relatively easy.
Current focus is to make sure that the protocol itself is solid and
future-proof.

Thanks,
Pavel

Re: Thin Client Protocol documentation

Posted by Pavel Tupitsyn <pt...@apache.org>.

Prachi,

Added "Cache flags" note to the doc.

Thanks,
Pavel

On Mon, Dec 4, 2017 at 7:46 PM, Prachi Garg <pg...@gridgain.com> wrote:

> Pavel,
>
> How do you specify the flags (withSkipStore, withExpiryPolicy, etc..) in
> bytes? Do you have a code for each of them?
>
> On Sun, Dec 3, 2017 at 9:23 AM, Pavel Tupitsyn <pt...@apache.org>
> wrote:
>
> > Hi Andrey,
> >
> > Compute and other APIs are certainly planned, cache is just a start.
> > We intentionally limit the scope to actually release something in 2.4 and
> > not delay it further.
> >
> > Adding operations to existing protocol is relatively easy.
> > Current focus is to make sure that the protocol itself is solid and
> > future-proof.
> >
> > Thanks,
> > Pavel
> >
>

Re: Thin Client Protocol documentation

Posted by Prachi Garg <pg...@gridgain.com>.

Pavel,

How do you specify the flags (withSkipStore, withExpiryPolicy, etc..) in
bytes? Do you have a code for each of them?

On Sun, Dec 3, 2017 at 9:23 AM, Pavel Tupitsyn <pt...@apache.org> wrote:

> Hi Andrey,
>
> Compute and other APIs are certainly planned, cache is just a start.
> We intentionally limit the scope to actually release something in 2.4 and
> not delay it further.
>
> Adding operations to existing protocol is relatively easy.
> Current focus is to make sure that the protocol itself is solid and
> future-proof.
>
> Thanks,
> Pavel
>

Re: Thin Client Protocol documentation

Posted by Pavel Tupitsyn <pt...@apache.org>.

Hi Andrey,

Compute and other APIs are certainly planned, cache is just a start.
We intentionally limit the scope to actually release something in 2.4 and
not delay it further.

Adding operations to existing protocol is relatively easy.
Current focus is to make sure that the protocol itself is solid and
future-proof.

Thanks,
Pavel

Re: Thin Client Protocol documentation

Posted by Andrey Kornev <an...@hotmail.com>.

Pavel,

Great job on the Thin Client protocol design!

I'm a bit late to the party, but I'm wondering why none of the compute features are supported? Such omission is unfortunate. Is it intentional? If so, what are the reasons?

I think it would be useful for the clients to be able to invoke already deployed tasks/services, or send a closure (if the client happens to be written in Java) for execution.

Thanks
Andrey

________________________________
From: Pavel Tupitsyn <pt...@apache.org>
Sent: Friday, December 1, 2017 5:48 AM
To: dev@ignite.apache.org
Subject: Re: Thin Client Protocol documentation

There are no legacy formats. JDBC and ODBC clients are not "legacy", quite
the opposite.

In future we may even want to combine JDBC Thin and general-purpose Thin
clients since they have a lot in common.
So let's keep the handshake format consistent across clients.

> exceptions for message formats
Handshake is an exception anyway, it does not have (or need) requestId, etc.

On Fri, Dec 1, 2017 at 1:38 PM, Alexey Popov <ta...@gmail.com> wrote:

> Pavel,
>
> I believe ClientListenerNioListener.onHandshake() could support more than
> one Handshake request format (be backward compatible), for instance, if we
> will have a new handshake code = 0xABCD that differs from 0x01 byte.
>
> It is a design vs architecture question.
>
> I can’t understand why the legacy Handshake format should be used for a
> new protocol. If this protocol is supposed to be public it should have no
> exceptions for message formats.
>
> Thank you,
> Alexey
>
> From: Pavel Tupitsyn
> Sent: Thursday, November 30, 2017 12:11 PM
> To: dev@ignite.apache.org
> Subject: Re: Thin Client Protocol documentation
>
> Hi Alexey,
>
> 1,2,3 are related only to handshake. All other operations are consistent.
>
> Handshake request format is dictated by existing client connector that is
> shared with ODBC and JDBC clients (see
> ClientListenerNioListener.onHandshake).
> so we can't add magic numbers or change operation code.
>
> But yes, we can add server version to the handshake response, and I think
> this makes sense.
>
> > 4. The same comments for success flag (1 byte) and status code (4 bytes)
> in responses. Let's leave only status code.
> We don't have a success flag in responses, there is just a 4-byte status
> code, 0 indicates success, everything else is an error.
>
> Thanks,
> Pavel
>
> On Thu, Nov 30, 2017 at 12:01 PM, Alexey Popov <ta...@gmail.com>
> wrote:
>
> > Hi Pavel,
> >
> > Let me add my 5 cents.
> >
> > 1. It would be great if both Handshake request & response have some
> > "magic" number (2 or 4 bytes) inside their msg body. That will simplify
> > handling situations when non-Ignite client connects to Ignite server and
> > vice versa.
> >
> > 2. It makes sense to add server version to successful Handshake response
> > as well. It will help to understand & debug possible backward
> compatibility
> > issues in the field by *.pcap logs analysis & etc.
> >
> > 3. Can we have a more strict header for all message types?
> > As far as I understand,
> > Handshake request has:
> > 1) length - 4 byte
> > 2) Handshake code - 1 byte
> > 3) body - (length - 1) bytes
> >
> > while OP_CACHE_GET request has:
> > 1) length - 4 byte
> > 2) OP_CACHE_GET code - 2 bytes
> > 3) request id - 4 bytes
> > 4) body - (length - 2 - 4) bytes
> >
> > Why some messages have Operation code with 1 byte while others - 2 bytes?
> > Why some requests/responses have request-id while others don't? Let's
> > simplify parser work )
> >
> > 4. The same comments for success flag (1 byte) and status code (4 bytes)
> > in responses. Let's leave only status code.
> >
> > Thank you,
> > Alexey
> >
> > From: Pavel Tupitsyn
> > Sent: Wednesday, November 22, 2017 4:04 PM
> > To: dev@ignite.apache.org
> > Subject: Thin Client Protocol documentation
> >
> > Igniters,
> >
> > I've put together a detailed description of our Thin Client protocol
> > in form of IEP on wiki:
> > https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> > 9+Thin+Client+Protocol
> >
> >
> > To clarify:
> > - Protocol implementation is in master (see ClientMessageParser class)
> > - Protocol has not been released yet, so we are free to change anything
> > - Protocol is only used by .NET Thin Client for now, but is supposed to
> be
> > used from other languages by third party contributors
> > - More operations will be added in future, this is a first set of them,
> > cache-related
> >
> >
> > Please review the document and let me know your thoughts.
> > Is there anything missing or wrong?
> >
> > We should make sure that the foundation is solid and extensible.
> >
> >
> > Thanks,
> > Pavel
> >
> >
>
>

Re: Thin Client Protocol documentation

Posted by Pavel Tupitsyn <pt...@apache.org>.

There are no legacy formats. JDBC and ODBC clients are not "legacy", quite
the opposite.

In future we may even want to combine JDBC Thin and general-purpose Thin
clients since they have a lot in common.
So let's keep the handshake format consistent across clients.

> exceptions for message formats
Handshake is an exception anyway, it does not have (or need) requestId, etc.

On Fri, Dec 1, 2017 at 1:38 PM, Alexey Popov <ta...@gmail.com> wrote:

> Pavel,
>
> I believe ClientListenerNioListener.onHandshake() could support more than
> one Handshake request format (be backward compatible), for instance, if we
> will have a new handshake code = 0xABCD that differs from 0x01 byte.
>
> It is a design vs architecture question.
>
> I can’t understand why the legacy Handshake format should be used for a
> new protocol. If this protocol is supposed to be public it should have no
> exceptions for message formats.
>
> Thank you,
> Alexey
>
> From: Pavel Tupitsyn
> Sent: Thursday, November 30, 2017 12:11 PM
> To: dev@ignite.apache.org
> Subject: Re: Thin Client Protocol documentation
>
> Hi Alexey,
>
> 1,2,3 are related only to handshake. All other operations are consistent.
>
> Handshake request format is dictated by existing client connector that is
> shared with ODBC and JDBC clients (see
> ClientListenerNioListener.onHandshake).
> so we can't add magic numbers or change operation code.
>
> But yes, we can add server version to the handshake response, and I think
> this makes sense.
>
> > 4. The same comments for success flag (1 byte) and status code (4 bytes)
> in responses. Let's leave only status code.
> We don't have a success flag in responses, there is just a 4-byte status
> code, 0 indicates success, everything else is an error.
>
> Thanks,
> Pavel
>
> On Thu, Nov 30, 2017 at 12:01 PM, Alexey Popov <ta...@gmail.com>
> wrote:
>
> > Hi Pavel,
> >
> > Let me add my 5 cents.
> >
> > 1. It would be great if both Handshake request & response have some
> > "magic" number (2 or 4 bytes) inside their msg body. That will simplify
> > handling situations when non-Ignite client connects to Ignite server and
> > vice versa.
> >
> > 2. It makes sense to add server version to successful Handshake response
> > as well. It will help to understand & debug possible backward
> compatibility
> > issues in the field by *.pcap logs analysis & etc.
> >
> > 3. Can we have a more strict header for all message types?
> > As far as I understand,
> > Handshake request has:
> > 1) length - 4 byte
> > 2) Handshake code - 1 byte
> > 3) body - (length - 1) bytes
> >
> > while OP_CACHE_GET request has:
> > 1) length - 4 byte
> > 2) OP_CACHE_GET code - 2 bytes
> > 3) request id - 4 bytes
> > 4) body - (length - 2 - 4) bytes
> >
> > Why some messages have Operation code with 1 byte while others - 2 bytes?
> > Why some requests/responses have request-id while others don't? Let's
> > simplify parser work )
> >
> > 4. The same comments for success flag (1 byte) and status code (4 bytes)
> > in responses. Let's leave only status code.
> >
> > Thank you,
> > Alexey
> >
> > From: Pavel Tupitsyn
> > Sent: Wednesday, November 22, 2017 4:04 PM
> > To: dev@ignite.apache.org
> > Subject: Thin Client Protocol documentation
> >
> > Igniters,
> >
> > I've put together a detailed description of our Thin Client protocol
> > in form of IEP on wiki:
> > https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> > 9+Thin+Client+Protocol
> >
> >
> > To clarify:
> > - Protocol implementation is in master (see ClientMessageParser class)
> > - Protocol has not been released yet, so we are free to change anything
> > - Protocol is only used by .NET Thin Client for now, but is supposed to
> be
> > used from other languages by third party contributors
> > - More operations will be added in future, this is a first set of them,
> > cache-related
> >
> >
> > Please review the document and let me know your thoughts.
> > Is there anything missing or wrong?
> >
> > We should make sure that the foundation is solid and extensible.
> >
> >
> > Thanks,
> > Pavel
> >
> >
>
>

RE: Thin Client Protocol documentation

Posted by Alexey Popov <ta...@gmail.com>.

Pavel,

I believe ClientListenerNioListener.onHandshake() could support more than one Handshake request format (be backward compatible), for instance, if we will have a new handshake code = 0xABCD that differs from 0x01 byte. 

It is a design vs architecture question.

I can’t understand why the legacy Handshake format should be used for a new protocol. If this protocol is supposed to be public it should have no exceptions for message formats.

Thank you,
Alexey

From: Pavel Tupitsyn
Sent: Thursday, November 30, 2017 12:11 PM
To: dev@ignite.apache.org
Subject: Re: Thin Client Protocol documentation

Hi Alexey,

1,2,3 are related only to handshake. All other operations are consistent.

Handshake request format is dictated by existing client connector that is
shared with ODBC and JDBC clients (see
ClientListenerNioListener.onHandshake).
so we can't add magic numbers or change operation code.

But yes, we can add server version to the handshake response, and I think
this makes sense.

> 4. The same comments for success flag (1 byte) and status code (4 bytes)
in responses. Let's leave only status code.
We don't have a success flag in responses, there is just a 4-byte status
code, 0 indicates success, everything else is an error.

Thanks,
Pavel

On Thu, Nov 30, 2017 at 12:01 PM, Alexey Popov <ta...@gmail.com> wrote:

> Hi Pavel,
>
> Let me add my 5 cents.
>
> 1. It would be great if both Handshake request & response have some
> "magic" number (2 or 4 bytes) inside their msg body. That will simplify
> handling situations when non-Ignite client connects to Ignite server and
> vice versa.
>
> 2. It makes sense to add server version to successful Handshake response
> as well. It will help to understand & debug possible backward compatibility
> issues in the field by *.pcap logs analysis & etc.
>
> 3. Can we have a more strict header for all message types?
> As far as I understand,
> Handshake request has:
> 1) length - 4 byte
> 2) Handshake code - 1 byte
> 3) body - (length - 1) bytes
>
> while OP_CACHE_GET request has:
> 1) length - 4 byte
> 2) OP_CACHE_GET code - 2 bytes
> 3) request id - 4 bytes
> 4) body - (length - 2 - 4) bytes
>
> Why some messages have Operation code with 1 byte while others - 2 bytes?
> Why some requests/responses have request-id while others don't? Let's
> simplify parser work )
>
> 4. The same comments for success flag (1 byte) and status code (4 bytes)
> in responses. Let's leave only status code.
>
> Thank you,
> Alexey
>
> From: Pavel Tupitsyn
> Sent: Wednesday, November 22, 2017 4:04 PM
> To: dev@ignite.apache.org
> Subject: Thin Client Protocol documentation
>
> Igniters,
>
> I've put together a detailed description of our Thin Client protocol
> in form of IEP on wiki:
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> 9+Thin+Client+Protocol
>
>
> To clarify:
> - Protocol implementation is in master (see ClientMessageParser class)
> - Protocol has not been released yet, so we are free to change anything
> - Protocol is only used by .NET Thin Client for now, but is supposed to be
> used from other languages by third party contributors
> - More operations will be added in future, this is a first set of them,
> cache-related
>
>
> Please review the document and let me know your thoughts.
> Is there anything missing or wrong?
>
> We should make sure that the foundation is solid and extensible.
>
>
> Thanks,
> Pavel
>
>

Re: Thin Client Protocol documentation

Posted by Pavel Tupitsyn <pt...@apache.org>.

Hi Alexey,

1,2,3 are related only to handshake. All other operations are consistent.

Handshake request format is dictated by existing client connector that is
shared with ODBC and JDBC clients (see
ClientListenerNioListener.onHandshake).
so we can't add magic numbers or change operation code.

But yes, we can add server version to the handshake response, and I think
this makes sense.

> 4. The same comments for success flag (1 byte) and status code (4 bytes)
in responses. Let's leave only status code.
We don't have a success flag in responses, there is just a 4-byte status
code, 0 indicates success, everything else is an error.

Thanks,
Pavel

On Thu, Nov 30, 2017 at 12:01 PM, Alexey Popov <ta...@gmail.com> wrote:

> Hi Pavel,
>
> Let me add my 5 cents.
>
> 1. It would be great if both Handshake request & response have some
> "magic" number (2 or 4 bytes) inside their msg body. That will simplify
> handling situations when non-Ignite client connects to Ignite server and
> vice versa.
>
> 2. It makes sense to add server version to successful Handshake response
> as well. It will help to understand & debug possible backward compatibility
> issues in the field by *.pcap logs analysis & etc.
>
> 3. Can we have a more strict header for all message types?
> As far as I understand,
> Handshake request has:
> 1) length - 4 byte
> 2) Handshake code - 1 byte
> 3) body - (length - 1) bytes
>
> while OP_CACHE_GET request has:
> 1) length - 4 byte
> 2) OP_CACHE_GET code - 2 bytes
> 3) request id - 4 bytes
> 4) body - (length - 2 - 4) bytes
>
> Why some messages have Operation code with 1 byte while others - 2 bytes?
> Why some requests/responses have request-id while others don't? Let's
> simplify parser work )
>
> 4. The same comments for success flag (1 byte) and status code (4 bytes)
> in responses. Let's leave only status code.
>
> Thank you,
> Alexey
>
> From: Pavel Tupitsyn
> Sent: Wednesday, November 22, 2017 4:04 PM
> To: dev@ignite.apache.org
> Subject: Thin Client Protocol documentation
>
> Igniters,
>
> I've put together a detailed description of our Thin Client protocol
> in form of IEP on wiki:
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> 9+Thin+Client+Protocol
>
>
> To clarify:
> - Protocol implementation is in master (see ClientMessageParser class)
> - Protocol has not been released yet, so we are free to change anything
> - Protocol is only used by .NET Thin Client for now, but is supposed to be
> used from other languages by third party contributors
> - More operations will be added in future, this is a first set of them,
> cache-related
>
>
> Please review the document and let me know your thoughts.
> Is there anything missing or wrong?
>
> We should make sure that the foundation is solid and extensible.
>
>
> Thanks,
> Pavel
>
>

RE: Thin Client Protocol documentation

Posted by Alexey Popov <ta...@gmail.com>.

Hi Pavel,

Let me add my 5 cents. 

1. It would be great if both Handshake request & response have some "magic" number (2 or 4 bytes) inside their msg body. That will simplify handling situations when non-Ignite client connects to Ignite server and vice versa. 

2. It makes sense to add server version to successful Handshake response as well. It will help to understand & debug possible backward compatibility issues in the field by *.pcap logs analysis & etc. 

3. Can we have a more strict header for all message types? 
As far as I understand, 
Handshake request has: 
1) length - 4 byte 
2) Handshake code - 1 byte 
3) body - (length - 1) bytes 

while OP_CACHE_GET request has: 
1) length - 4 byte 
2) OP_CACHE_GET code - 2 bytes 
3) request id - 4 bytes 
4) body - (length - 2 - 4) bytes 

Why some messages have Operation code with 1 byte while others - 2 bytes? Why some requests/responses have request-id while others don't? Let's simplify parser work ) 

4. The same comments for success flag (1 byte) and status code (4 bytes) in responses. Let's leave only status code. 

Thank you,
Alexey

From: Pavel Tupitsyn
Sent: Wednesday, November 22, 2017 4:04 PM
To: dev@ignite.apache.org
Subject: Thin Client Protocol documentation

Igniters,

I've put together a detailed description of our Thin Client protocol
in form of IEP on wiki:
https://cwiki.apache.org/confluence/display/IGNITE/IEP-9+Thin+Client+Protocol


To clarify:
- Protocol implementation is in master (see ClientMessageParser class)
- Protocol has not been released yet, so we are free to change anything
- Protocol is only used by .NET Thin Client for now, but is supposed to be
used from other languages by third party contributors
- More operations will be added in future, this is a first set of them,
cache-related


Please review the document and let me know your thoughts.
Is there anything missing or wrong?

We should make sure that the foundation is solid and extensible.


Thanks,
Pavel

Re: Thin Client Protocol documentation

Posted by Pavel Tupitsyn <pt...@apache.org>.

Prachi, see "Response (failure)" table.
Failed response includes server protocol version and an error message.

On Thu, Nov 30, 2017 at 4:55 AM, Prachi Garg <pg...@gridgain.com> wrote:

> Pavel,
>
> If the connection handshake fails, what should be the message length in
> the response. When I try to fail the handshake, I get 32 as the message
> length. Is this expected?
>
> -Prachi
>
> On Wed, Nov 29, 2017 at 8:20 AM, Pavel Tupitsyn <pt...@gridgain.com>
> wrote:
>
>> Hi Prachi,
>>
>> Flags parameter in all cache operations is reserved for things like
>> withSkipStore, withExpiryPolicy, withKeepBinary, withNoRetries,
>> withPartitionRecover.
>> See methods in IgniteCache interface.
>>
>> Thanks,
>> Pavel
>>
>> On Wed, Nov 29, 2017 at 7:10 PM, Prachi Garg <pg...@gridgain.com> wrote:
>>
>>> Hi Pavel,
>>>
>>> In the operation request, what does the 'flags' parameter mean?
>>>
>>>
>>> Thanks,
>>> -Prachi
>>>
>>> On Wed, Nov 29, 2017 at 5:27 AM, Pavel Tupitsyn <pt...@apache.org>
>>> wrote:
>>>
>>>> Sergey, good point, done.
>>>>
>>>> On Wed, Nov 29, 2017 at 2:30 PM, Sergey Kozlov <sk...@gridgain.com>
>>>> wrote:
>>>>
>>>> > Pavel
>>>> >
>>>> > Could you update the page by following:
>>>> >
>>>> >  - String, date, UUID arrays allow to put NULL. Due to that every
>>>> item in
>>>> > the array written as type code byte (default item type or null type
>>>> code) +
>>>> > type data. It should be detailed explained (looks like that the table
>>>> > should have an addtional column called for instance "nullable")
>>>> >
>>>> > -  UUID type takes 16 bytes length
>>>> >
>>>> > thanks
>>>> >
>>>> >
>>>> > On Mon, Nov 27, 2017 at 4:37 PM, Sergey Kozlov <sk...@gridgain.com>
>>>> > wrote:
>>>> >
>>>> > > Pavel
>>>> > >
>>>> > > Thanks for explanations!
>>>> > >
>>>> > > On Mon, Nov 27, 2017 at 2:46 PM, Pavel Tupitsyn <
>>>> ptupitsyn@apache.org>
>>>> > > wrote:
>>>> > >
>>>> > >> Sergey,
>>>> > >>
>>>> > >> 1. Code table size does not affect anything, as I understand, so
>>>> there
>>>> > is
>>>> > >> no reason to introduce an extra byte.
>>>> > >> 2. We have object arrays (code 23), I forgot to mention them,
>>>> fixed.
>>>> > >> 3. Also forgot, see code 25 in the updated document.
>>>> > >>
>>>> > >> Also note that operation codes have been updated (grouped by
>>>> purpose) as
>>>> > >> part of https://issues.apache.org/jira/browse/IGNITE-6989.
>>>> > >>
>>>> > >> Thanks,
>>>> > >> Pavel
>>>> > >>
>>>> > >> On Sun, Nov 26, 2017 at 9:54 PM, Sergey Kozlov <
>>>> skozlov@gridgain.com>
>>>> > >> wrote:
>>>> > >>
>>>> > >> > Pavel
>>>> > >> >
>>>> > >> > Thanks for the document and your efforts for new protocol. It was
>>>> > really
>>>> > >> > helpful for playing around the python thin client design.
>>>> > >> >
>>>> > >> > Could you explain some things that were still not clear for
>>>> binary
>>>> > >> object
>>>> > >> > format:
>>>> > >> >
>>>> > >> > 1. What a reason to introduce separated type codes for arrays?
>>>> Why
>>>> > just
>>>> > >> we
>>>> > >> > can't use the following?
>>>> > >> > *<1 byte universal array code>*
>>>> > >> > *<1 byte primitive code>*
>>>> > >> > *<4 bytes length>*
>>>> > >> > *<N bytes data>*
>>>> > >> >
>>>> > >> > We get 1 byte overhead but save 10 bytes in the code table. For
>>>> arrays
>>>> > >> the
>>>> > >> > overhead is really insignificant:10 longs array takes now
>>>> 1+4+4*10=45
>>>> > >> bytes
>>>> > >> > vs 1+1+4+4*10=46 bytes for the approach
>>>> > >> > Moreover for that appoach a new primitive code will be available
>>>> for
>>>> > >> using
>>>> > >> > for array immediately.
>>>> > >> >
>>>> > >> > 2. Why the arrays force to use a selected type? For python
>>>> there's no
>>>> > >> > limitations to use different types across one array (list).
>>>> Would be
>>>> > >> good
>>>> > >> > to introduce a new type that will allow that. It could be look
>>>> like
>>>> > >> > following
>>>> > >> > *<1 byte universal array code>*
>>>> > >> > *<1 byte no common type code*> <-- this says that every item must
>>>> > >> provide
>>>> > >> > its date type code like it does regular primitive data
>>>> > >> > *<4 bytes length>*
>>>> > >> > *<1 byte item 0 type code>*     <-- item provides its code
>>>> > >> > *<N byte item 0 data>*      <-- item provides its data
>>>> > >> > *<1 byte item 1 type code>*
>>>> > >> > *<N byte item 1 data>*
>>>> > >> > etc
>>>> > >> >
>>>> > >> > Also that allow to put nested arrays without changes in type code
>>>> > table!
>>>> > >> > For instance if we want to store 9 longs and 1 boolean it will
>>>> take
>>>> > >> > now 1+1+4+(1+9)*4+(1+1)=48
>>>> > >> > bytes (vs 45 bytes to store as 10 longs as usual).
>>>> > >> >
>>>> > >> > 3. Ther's only one way to store a dictionary (key-value)
>>>> structure as
>>>> > >> value
>>>> > >> > in the cache via Complex Object. But it looks like
>>>> overcomplicated. I
>>>> > >> > suppose to introduce a code for that
>>>> > >> > *<1 byte key-value dictionary code>*
>>>> > >> >
>>>> > >> > *<4 bytes length>*
>>>> > >> > *<1 byte key 1 **name **type code>*
>>>> > >> > *<N byte key 1 name data>*
>>>> > >> > *<1 byte value 1 type code>*
>>>> > >> > *<N byte value 1 value>*
>>>> > >> > *<1 byte key 2 **name **type code>*
>>>> > >> > *<N byte key 2 name data>*
>>>> > >> > *<1 byte value 2 type code>*
>>>> > >> > *<N byte value 2 value>*
>>>> > >> > etc
>>>> > >> >
>>>> > >> > Also that allow to put nested dictionaries without changes in
>>>> type
>>>> > code
>>>> > >> > table!
>>>> > >> > Of  course for the appoach above we get significat overhead for
>>>> key
>>>> > >> > storing. But I think it is acceptable for some cases and
>>>> definitely ok
>>>> > >> for
>>>> > >> > Python
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> > On Wed, Nov 22, 2017 at 9:14 PM, Prachi Garg <pgarg@gridgain.com
>>>> >
>>>> > >> wrote:
>>>> > >> >
>>>> > >> > > Thanks Pavel! The document has good information. I'll create
>>>> one on
>>>> > >> > > readme.io; will also add some examples there.
>>>> > >> > >
>>>> > >> > > On Wed, Nov 22, 2017 at 5:03 AM, Pavel Tupitsyn <
>>>> > ptupitsyn@apache.org
>>>> > >> >
>>>> > >> > > wrote:
>>>> > >> > >
>>>> > >> > > > Igniters,
>>>> > >> > > >
>>>> > >> > > > I've put together a detailed description of our Thin Client
>>>> > protocol
>>>> > >> > > > in form of IEP on wiki:
>>>> > >> > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-
>>>> > >> > > > 9+Thin+Client+Protocol
>>>> > >> > > >
>>>> > >> > > >
>>>> > >> > > > To clarify:
>>>> > >> > > > - Protocol implementation is in master (see
>>>> ClientMessageParser
>>>> > >> class)
>>>> > >> > > > - Protocol has not been released yet, so we are free to
>>>> change
>>>> > >> anything
>>>> > >> > > > - Protocol is only used by .NET Thin Client for now, but is
>>>> > >> supposed to
>>>> > >> > > be
>>>> > >> > > > used from other languages by third party contributors
>>>> > >> > > > - More operations will be added in future, this is a first
>>>> set of
>>>> > >> them,
>>>> > >> > > > cache-related
>>>> > >> > > >
>>>> > >> > > >
>>>> > >> > > > Please review the document and let me know your thoughts.
>>>> > >> > > > Is there anything missing or wrong?
>>>> > >> > > >
>>>> > >> > > > We should make sure that the foundation is solid and
>>>> extensible.
>>>> > >> > > >
>>>> > >> > > >
>>>> > >> > > > Thanks,
>>>> > >> > > > Pavel
>>>> > >> > > >
>>>> > >> > >
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> > --
>>>> > >> > Sergey Kozlov
>>>> > >> > GridGain Systems
>>>> > >> > www.gridgain.com
>>>> > >> >
>>>> > >>
>>>> > >
>>>> > >
>>>> > >
>>>> > > --
>>>> > > Sergey Kozlov
>>>> > > GridGain Systems
>>>> > > www.gridgain.com
>>>> > >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Sergey Kozlov
>>>> > GridGain Systems
>>>> > www.gridgain.com
>>>> >
>>>>
>>>
>>>
>>
>

Re: Thin Client Protocol documentation

Posted by Prachi Garg <pg...@gridgain.com>.

Pavel,

If the connection handshake fails, what should be the message length in the
response. When I try to fail the handshake, I get 32 as the message length.
Is this expected?

-Prachi

On Wed, Nov 29, 2017 at 8:20 AM, Pavel Tupitsyn <pt...@gridgain.com>
wrote:

> Hi Prachi,
>
> Flags parameter in all cache operations is reserved for things like
> withSkipStore, withExpiryPolicy, withKeepBinary, withNoRetries,
> withPartitionRecover.
> See methods in IgniteCache interface.
>
> Thanks,
> Pavel
>
> On Wed, Nov 29, 2017 at 7:10 PM, Prachi Garg <pg...@gridgain.com> wrote:
>
>> Hi Pavel,
>>
>> In the operation request, what does the 'flags' parameter mean?
>>
>>
>> Thanks,
>> -Prachi
>>
>> On Wed, Nov 29, 2017 at 5:27 AM, Pavel Tupitsyn <pt...@apache.org>
>> wrote:
>>
>>> Sergey, good point, done.
>>>
>>> On Wed, Nov 29, 2017 at 2:30 PM, Sergey Kozlov <sk...@gridgain.com>
>>> wrote:
>>>
>>> > Pavel
>>> >
>>> > Could you update the page by following:
>>> >
>>> >  - String, date, UUID arrays allow to put NULL. Due to that every item
>>> in
>>> > the array written as type code byte (default item type or null type
>>> code) +
>>> > type data. It should be detailed explained (looks like that the table
>>> > should have an addtional column called for instance "nullable")
>>> >
>>> > -  UUID type takes 16 bytes length
>>> >
>>> > thanks
>>> >
>>> >
>>> > On Mon, Nov 27, 2017 at 4:37 PM, Sergey Kozlov <sk...@gridgain.com>
>>> > wrote:
>>> >
>>> > > Pavel
>>> > >
>>> > > Thanks for explanations!
>>> > >
>>> > > On Mon, Nov 27, 2017 at 2:46 PM, Pavel Tupitsyn <
>>> ptupitsyn@apache.org>
>>> > > wrote:
>>> > >
>>> > >> Sergey,
>>> > >>
>>> > >> 1. Code table size does not affect anything, as I understand, so
>>> there
>>> > is
>>> > >> no reason to introduce an extra byte.
>>> > >> 2. We have object arrays (code 23), I forgot to mention them, fixed.
>>> > >> 3. Also forgot, see code 25 in the updated document.
>>> > >>
>>> > >> Also note that operation codes have been updated (grouped by
>>> purpose) as
>>> > >> part of https://issues.apache.org/jira/browse/IGNITE-6989.
>>> > >>
>>> > >> Thanks,
>>> > >> Pavel
>>> > >>
>>> > >> On Sun, Nov 26, 2017 at 9:54 PM, Sergey Kozlov <
>>> skozlov@gridgain.com>
>>> > >> wrote:
>>> > >>
>>> > >> > Pavel
>>> > >> >
>>> > >> > Thanks for the document and your efforts for new protocol. It was
>>> > really
>>> > >> > helpful for playing around the python thin client design.
>>> > >> >
>>> > >> > Could you explain some things that were still not clear for binary
>>> > >> object
>>> > >> > format:
>>> > >> >
>>> > >> > 1. What a reason to introduce separated type codes for arrays? Why
>>> > just
>>> > >> we
>>> > >> > can't use the following?
>>> > >> > *<1 byte universal array code>*
>>> > >> > *<1 byte primitive code>*
>>> > >> > *<4 bytes length>*
>>> > >> > *<N bytes data>*
>>> > >> >
>>> > >> > We get 1 byte overhead but save 10 bytes in the code table. For
>>> arrays
>>> > >> the
>>> > >> > overhead is really insignificant:10 longs array takes now
>>> 1+4+4*10=45
>>> > >> bytes
>>> > >> > vs 1+1+4+4*10=46 bytes for the approach
>>> > >> > Moreover for that appoach a new primitive code will be available
>>> for
>>> > >> using
>>> > >> > for array immediately.
>>> > >> >
>>> > >> > 2. Why the arrays force to use a selected type? For python
>>> there's no
>>> > >> > limitations to use different types across one array (list). Would
>>> be
>>> > >> good
>>> > >> > to introduce a new type that will allow that. It could be look
>>> like
>>> > >> > following
>>> > >> > *<1 byte universal array code>*
>>> > >> > *<1 byte no common type code*> <-- this says that every item must
>>> > >> provide
>>> > >> > its date type code like it does regular primitive data
>>> > >> > *<4 bytes length>*
>>> > >> > *<1 byte item 0 type code>*     <-- item provides its code
>>> > >> > *<N byte item 0 data>*      <-- item provides its data
>>> > >> > *<1 byte item 1 type code>*
>>> > >> > *<N byte item 1 data>*
>>> > >> > etc
>>> > >> >
>>> > >> > Also that allow to put nested arrays without changes in type code
>>> > table!
>>> > >> > For instance if we want to store 9 longs and 1 boolean it will
>>> take
>>> > >> > now 1+1+4+(1+9)*4+(1+1)=48
>>> > >> > bytes (vs 45 bytes to store as 10 longs as usual).
>>> > >> >
>>> > >> > 3. Ther's only one way to store a dictionary (key-value)
>>> structure as
>>> > >> value
>>> > >> > in the cache via Complex Object. But it looks like
>>> overcomplicated. I
>>> > >> > suppose to introduce a code for that
>>> > >> > *<1 byte key-value dictionary code>*
>>> > >> >
>>> > >> > *<4 bytes length>*
>>> > >> > *<1 byte key 1 **name **type code>*
>>> > >> > *<N byte key 1 name data>*
>>> > >> > *<1 byte value 1 type code>*
>>> > >> > *<N byte value 1 value>*
>>> > >> > *<1 byte key 2 **name **type code>*
>>> > >> > *<N byte key 2 name data>*
>>> > >> > *<1 byte value 2 type code>*
>>> > >> > *<N byte value 2 value>*
>>> > >> > etc
>>> > >> >
>>> > >> > Also that allow to put nested dictionaries without changes in type
>>> > code
>>> > >> > table!
>>> > >> > Of  course for the appoach above we get significat overhead for
>>> key
>>> > >> > storing. But I think it is acceptable for some cases and
>>> definitely ok
>>> > >> for
>>> > >> > Python
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> > On Wed, Nov 22, 2017 at 9:14 PM, Prachi Garg <pg...@gridgain.com>
>>> > >> wrote:
>>> > >> >
>>> > >> > > Thanks Pavel! The document has good information. I'll create
>>> one on
>>> > >> > > readme.io; will also add some examples there.
>>> > >> > >
>>> > >> > > On Wed, Nov 22, 2017 at 5:03 AM, Pavel Tupitsyn <
>>> > ptupitsyn@apache.org
>>> > >> >
>>> > >> > > wrote:
>>> > >> > >
>>> > >> > > > Igniters,
>>> > >> > > >
>>> > >> > > > I've put together a detailed description of our Thin Client
>>> > protocol
>>> > >> > > > in form of IEP on wiki:
>>> > >> > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-
>>> > >> > > > 9+Thin+Client+Protocol
>>> > >> > > >
>>> > >> > > >
>>> > >> > > > To clarify:
>>> > >> > > > - Protocol implementation is in master (see
>>> ClientMessageParser
>>> > >> class)
>>> > >> > > > - Protocol has not been released yet, so we are free to change
>>> > >> anything
>>> > >> > > > - Protocol is only used by .NET Thin Client for now, but is
>>> > >> supposed to
>>> > >> > > be
>>> > >> > > > used from other languages by third party contributors
>>> > >> > > > - More operations will be added in future, this is a first
>>> set of
>>> > >> them,
>>> > >> > > > cache-related
>>> > >> > > >
>>> > >> > > >
>>> > >> > > > Please review the document and let me know your thoughts.
>>> > >> > > > Is there anything missing or wrong?
>>> > >> > > >
>>> > >> > > > We should make sure that the foundation is solid and
>>> extensible.
>>> > >> > > >
>>> > >> > > >
>>> > >> > > > Thanks,
>>> > >> > > > Pavel
>>> > >> > > >
>>> > >> > >
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> > --
>>> > >> > Sergey Kozlov
>>> > >> > GridGain Systems
>>> > >> > www.gridgain.com
>>> > >> >
>>> > >>
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > Sergey Kozlov
>>> > > GridGain Systems
>>> > > www.gridgain.com
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > Sergey Kozlov
>>> > GridGain Systems
>>> > www.gridgain.com
>>> >
>>>
>>
>>
>

Re: Thin Client Protocol documentation

Posted by Pavel Tupitsyn <pt...@gridgain.com>.

Hi Prachi,

Flags parameter in all cache operations is reserved for things like
withSkipStore, withExpiryPolicy, withKeepBinary, withNoRetries,
withPartitionRecover.
See methods in IgniteCache interface.

Thanks,
Pavel

On Wed, Nov 29, 2017 at 7:10 PM, Prachi Garg <pg...@gridgain.com> wrote:

> Hi Pavel,
>
> In the operation request, what does the 'flags' parameter mean?
>
>
> Thanks,
> -Prachi
>
> On Wed, Nov 29, 2017 at 5:27 AM, Pavel Tupitsyn <pt...@apache.org>
> wrote:
>
>> Sergey, good point, done.
>>
>> On Wed, Nov 29, 2017 at 2:30 PM, Sergey Kozlov <sk...@gridgain.com>
>> wrote:
>>
>> > Pavel
>> >
>> > Could you update the page by following:
>> >
>> >  - String, date, UUID arrays allow to put NULL. Due to that every item
>> in
>> > the array written as type code byte (default item type or null type
>> code) +
>> > type data. It should be detailed explained (looks like that the table
>> > should have an addtional column called for instance "nullable")
>> >
>> > -  UUID type takes 16 bytes length
>> >
>> > thanks
>> >
>> >
>> > On Mon, Nov 27, 2017 at 4:37 PM, Sergey Kozlov <sk...@gridgain.com>
>> > wrote:
>> >
>> > > Pavel
>> > >
>> > > Thanks for explanations!
>> > >
>> > > On Mon, Nov 27, 2017 at 2:46 PM, Pavel Tupitsyn <ptupitsyn@apache.org
>> >
>> > > wrote:
>> > >
>> > >> Sergey,
>> > >>
>> > >> 1. Code table size does not affect anything, as I understand, so
>> there
>> > is
>> > >> no reason to introduce an extra byte.
>> > >> 2. We have object arrays (code 23), I forgot to mention them, fixed.
>> > >> 3. Also forgot, see code 25 in the updated document.
>> > >>
>> > >> Also note that operation codes have been updated (grouped by
>> purpose) as
>> > >> part of https://issues.apache.org/jira/browse/IGNITE-6989.
>> > >>
>> > >> Thanks,
>> > >> Pavel
>> > >>
>> > >> On Sun, Nov 26, 2017 at 9:54 PM, Sergey Kozlov <skozlov@gridgain.com
>> >
>> > >> wrote:
>> > >>
>> > >> > Pavel
>> > >> >
>> > >> > Thanks for the document and your efforts for new protocol. It was
>> > really
>> > >> > helpful for playing around the python thin client design.
>> > >> >
>> > >> > Could you explain some things that were still not clear for binary
>> > >> object
>> > >> > format:
>> > >> >
>> > >> > 1. What a reason to introduce separated type codes for arrays? Why
>> > just
>> > >> we
>> > >> > can't use the following?
>> > >> > *<1 byte universal array code>*
>> > >> > *<1 byte primitive code>*
>> > >> > *<4 bytes length>*
>> > >> > *<N bytes data>*
>> > >> >
>> > >> > We get 1 byte overhead but save 10 bytes in the code table. For
>> arrays
>> > >> the
>> > >> > overhead is really insignificant:10 longs array takes now
>> 1+4+4*10=45
>> > >> bytes
>> > >> > vs 1+1+4+4*10=46 bytes for the approach
>> > >> > Moreover for that appoach a new primitive code will be available
>> for
>> > >> using
>> > >> > for array immediately.
>> > >> >
>> > >> > 2. Why the arrays force to use a selected type? For python there's
>> no
>> > >> > limitations to use different types across one array (list). Would
>> be
>> > >> good
>> > >> > to introduce a new type that will allow that. It could be look like
>> > >> > following
>> > >> > *<1 byte universal array code>*
>> > >> > *<1 byte no common type code*> <-- this says that every item must
>> > >> provide
>> > >> > its date type code like it does regular primitive data
>> > >> > *<4 bytes length>*
>> > >> > *<1 byte item 0 type code>*     <-- item provides its code
>> > >> > *<N byte item 0 data>*      <-- item provides its data
>> > >> > *<1 byte item 1 type code>*
>> > >> > *<N byte item 1 data>*
>> > >> > etc
>> > >> >
>> > >> > Also that allow to put nested arrays without changes in type code
>> > table!
>> > >> > For instance if we want to store 9 longs and 1 boolean it will take
>> > >> > now 1+1+4+(1+9)*4+(1+1)=48
>> > >> > bytes (vs 45 bytes to store as 10 longs as usual).
>> > >> >
>> > >> > 3. Ther's only one way to store a dictionary (key-value) structure
>> as
>> > >> value
>> > >> > in the cache via Complex Object. But it looks like
>> overcomplicated. I
>> > >> > suppose to introduce a code for that
>> > >> > *<1 byte key-value dictionary code>*
>> > >> >
>> > >> > *<4 bytes length>*
>> > >> > *<1 byte key 1 **name **type code>*
>> > >> > *<N byte key 1 name data>*
>> > >> > *<1 byte value 1 type code>*
>> > >> > *<N byte value 1 value>*
>> > >> > *<1 byte key 2 **name **type code>*
>> > >> > *<N byte key 2 name data>*
>> > >> > *<1 byte value 2 type code>*
>> > >> > *<N byte value 2 value>*
>> > >> > etc
>> > >> >
>> > >> > Also that allow to put nested dictionaries without changes in type
>> > code
>> > >> > table!
>> > >> > Of  course for the appoach above we get significat overhead for key
>> > >> > storing. But I think it is acceptable for some cases and
>> definitely ok
>> > >> for
>> > >> > Python
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Wed, Nov 22, 2017 at 9:14 PM, Prachi Garg <pg...@gridgain.com>
>> > >> wrote:
>> > >> >
>> > >> > > Thanks Pavel! The document has good information. I'll create one
>> on
>> > >> > > readme.io; will also add some examples there.
>> > >> > >
>> > >> > > On Wed, Nov 22, 2017 at 5:03 AM, Pavel Tupitsyn <
>> > ptupitsyn@apache.org
>> > >> >
>> > >> > > wrote:
>> > >> > >
>> > >> > > > Igniters,
>> > >> > > >
>> > >> > > > I've put together a detailed description of our Thin Client
>> > protocol
>> > >> > > > in form of IEP on wiki:
>> > >> > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-
>> > >> > > > 9+Thin+Client+Protocol
>> > >> > > >
>> > >> > > >
>> > >> > > > To clarify:
>> > >> > > > - Protocol implementation is in master (see ClientMessageParser
>> > >> class)
>> > >> > > > - Protocol has not been released yet, so we are free to change
>> > >> anything
>> > >> > > > - Protocol is only used by .NET Thin Client for now, but is
>> > >> supposed to
>> > >> > > be
>> > >> > > > used from other languages by third party contributors
>> > >> > > > - More operations will be added in future, this is a first set
>> of
>> > >> them,
>> > >> > > > cache-related
>> > >> > > >
>> > >> > > >
>> > >> > > > Please review the document and let me know your thoughts.
>> > >> > > > Is there anything missing or wrong?
>> > >> > > >
>> > >> > > > We should make sure that the foundation is solid and
>> extensible.
>> > >> > > >
>> > >> > > >
>> > >> > > > Thanks,
>> > >> > > > Pavel
>> > >> > > >
>> > >> > >
>> > >> >
>> > >> >
>> > >> >
>> > >> > --
>> > >> > Sergey Kozlov
>> > >> > GridGain Systems
>> > >> > www.gridgain.com
>> > >> >
>> > >>
>> > >
>> > >
>> > >
>> > > --
>> > > Sergey Kozlov
>> > > GridGain Systems
>> > > www.gridgain.com
>> > >
>> >
>> >
>> >
>> > --
>> > Sergey Kozlov
>> > GridGain Systems
>> > www.gridgain.com
>> >
>>
>
>

Re: Thin Client Protocol documentation

Posted by Prachi Garg <pg...@gridgain.com>.

Hi Pavel,

In the operation request, what does the 'flags' parameter mean?


Thanks,
-Prachi

On Wed, Nov 29, 2017 at 5:27 AM, Pavel Tupitsyn <pt...@apache.org>
wrote:

> Sergey, good point, done.
>
> On Wed, Nov 29, 2017 at 2:30 PM, Sergey Kozlov <sk...@gridgain.com>
> wrote:
>
> > Pavel
> >
> > Could you update the page by following:
> >
> >  - String, date, UUID arrays allow to put NULL. Due to that every item in
> > the array written as type code byte (default item type or null type
> code) +
> > type data. It should be detailed explained (looks like that the table
> > should have an addtional column called for instance "nullable")
> >
> > -  UUID type takes 16 bytes length
> >
> > thanks
> >
> >
> > On Mon, Nov 27, 2017 at 4:37 PM, Sergey Kozlov <sk...@gridgain.com>
> > wrote:
> >
> > > Pavel
> > >
> > > Thanks for explanations!
> > >
> > > On Mon, Nov 27, 2017 at 2:46 PM, Pavel Tupitsyn <pt...@apache.org>
> > > wrote:
> > >
> > >> Sergey,
> > >>
> > >> 1. Code table size does not affect anything, as I understand, so there
> > is
> > >> no reason to introduce an extra byte.
> > >> 2. We have object arrays (code 23), I forgot to mention them, fixed.
> > >> 3. Also forgot, see code 25 in the updated document.
> > >>
> > >> Also note that operation codes have been updated (grouped by purpose)
> as
> > >> part of https://issues.apache.org/jira/browse/IGNITE-6989.
> > >>
> > >> Thanks,
> > >> Pavel
> > >>
> > >> On Sun, Nov 26, 2017 at 9:54 PM, Sergey Kozlov <sk...@gridgain.com>
> > >> wrote:
> > >>
> > >> > Pavel
> > >> >
> > >> > Thanks for the document and your efforts for new protocol. It was
> > really
> > >> > helpful for playing around the python thin client design.
> > >> >
> > >> > Could you explain some things that were still not clear for binary
> > >> object
> > >> > format:
> > >> >
> > >> > 1. What a reason to introduce separated type codes for arrays? Why
> > just
> > >> we
> > >> > can't use the following?
> > >> > *<1 byte universal array code>*
> > >> > *<1 byte primitive code>*
> > >> > *<4 bytes length>*
> > >> > *<N bytes data>*
> > >> >
> > >> > We get 1 byte overhead but save 10 bytes in the code table. For
> arrays
> > >> the
> > >> > overhead is really insignificant:10 longs array takes now
> 1+4+4*10=45
> > >> bytes
> > >> > vs 1+1+4+4*10=46 bytes for the approach
> > >> > Moreover for that appoach a new primitive code will be available for
> > >> using
> > >> > for array immediately.
> > >> >
> > >> > 2. Why the arrays force to use a selected type? For python there's
> no
> > >> > limitations to use different types across one array (list). Would be
> > >> good
> > >> > to introduce a new type that will allow that. It could be look like
> > >> > following
> > >> > *<1 byte universal array code>*
> > >> > *<1 byte no common type code*> <-- this says that every item must
> > >> provide
> > >> > its date type code like it does regular primitive data
> > >> > *<4 bytes length>*
> > >> > *<1 byte item 0 type code>*     <-- item provides its code
> > >> > *<N byte item 0 data>*      <-- item provides its data
> > >> > *<1 byte item 1 type code>*
> > >> > *<N byte item 1 data>*
> > >> > etc
> > >> >
> > >> > Also that allow to put nested arrays without changes in type code
> > table!
> > >> > For instance if we want to store 9 longs and 1 boolean it will take
> > >> > now 1+1+4+(1+9)*4+(1+1)=48
> > >> > bytes (vs 45 bytes to store as 10 longs as usual).
> > >> >
> > >> > 3. Ther's only one way to store a dictionary (key-value) structure
> as
> > >> value
> > >> > in the cache via Complex Object. But it looks like overcomplicated.
> I
> > >> > suppose to introduce a code for that
> > >> > *<1 byte key-value dictionary code>*
> > >> >
> > >> > *<4 bytes length>*
> > >> > *<1 byte key 1 **name **type code>*
> > >> > *<N byte key 1 name data>*
> > >> > *<1 byte value 1 type code>*
> > >> > *<N byte value 1 value>*
> > >> > *<1 byte key 2 **name **type code>*
> > >> > *<N byte key 2 name data>*
> > >> > *<1 byte value 2 type code>*
> > >> > *<N byte value 2 value>*
> > >> > etc
> > >> >
> > >> > Also that allow to put nested dictionaries without changes in type
> > code
> > >> > table!
> > >> > Of  course for the appoach above we get significat overhead for key
> > >> > storing. But I think it is acceptable for some cases and definitely
> ok
> > >> for
> > >> > Python
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On Wed, Nov 22, 2017 at 9:14 PM, Prachi Garg <pg...@gridgain.com>
> > >> wrote:
> > >> >
> > >> > > Thanks Pavel! The document has good information. I'll create one
> on
> > >> > > readme.io; will also add some examples there.
> > >> > >
> > >> > > On Wed, Nov 22, 2017 at 5:03 AM, Pavel Tupitsyn <
> > ptupitsyn@apache.org
> > >> >
> > >> > > wrote:
> > >> > >
> > >> > > > Igniters,
> > >> > > >
> > >> > > > I've put together a detailed description of our Thin Client
> > protocol
> > >> > > > in form of IEP on wiki:
> > >> > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> > >> > > > 9+Thin+Client+Protocol
> > >> > > >
> > >> > > >
> > >> > > > To clarify:
> > >> > > > - Protocol implementation is in master (see ClientMessageParser
> > >> class)
> > >> > > > - Protocol has not been released yet, so we are free to change
> > >> anything
> > >> > > > - Protocol is only used by .NET Thin Client for now, but is
> > >> supposed to
> > >> > > be
> > >> > > > used from other languages by third party contributors
> > >> > > > - More operations will be added in future, this is a first set
> of
> > >> them,
> > >> > > > cache-related
> > >> > > >
> > >> > > >
> > >> > > > Please review the document and let me know your thoughts.
> > >> > > > Is there anything missing or wrong?
> > >> > > >
> > >> > > > We should make sure that the foundation is solid and extensible.
> > >> > > >
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Pavel
> > >> > > >
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Sergey Kozlov
> > >> > GridGain Systems
> > >> > www.gridgain.com
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Sergey Kozlov
> > > GridGain Systems
> > > www.gridgain.com
> > >
> >
> >
> >
> > --
> > Sergey Kozlov
> > GridGain Systems
> > www.gridgain.com
> >
>

Re: Thin Client Protocol documentation

Posted by Pavel Tupitsyn <pt...@apache.org>.

Sergey, good point, done.

On Wed, Nov 29, 2017 at 2:30 PM, Sergey Kozlov <sk...@gridgain.com> wrote:

> Pavel
>
> Could you update the page by following:
>
>  - String, date, UUID arrays allow to put NULL. Due to that every item in
> the array written as type code byte (default item type or null type code) +
> type data. It should be detailed explained (looks like that the table
> should have an addtional column called for instance "nullable")
>
> -  UUID type takes 16 bytes length
>
> thanks
>
>
> On Mon, Nov 27, 2017 at 4:37 PM, Sergey Kozlov <sk...@gridgain.com>
> wrote:
>
> > Pavel
> >
> > Thanks for explanations!
> >
> > On Mon, Nov 27, 2017 at 2:46 PM, Pavel Tupitsyn <pt...@apache.org>
> > wrote:
> >
> >> Sergey,
> >>
> >> 1. Code table size does not affect anything, as I understand, so there
> is
> >> no reason to introduce an extra byte.
> >> 2. We have object arrays (code 23), I forgot to mention them, fixed.
> >> 3. Also forgot, see code 25 in the updated document.
> >>
> >> Also note that operation codes have been updated (grouped by purpose) as
> >> part of https://issues.apache.org/jira/browse/IGNITE-6989.
> >>
> >> Thanks,
> >> Pavel
> >>
> >> On Sun, Nov 26, 2017 at 9:54 PM, Sergey Kozlov <sk...@gridgain.com>
> >> wrote:
> >>
> >> > Pavel
> >> >
> >> > Thanks for the document and your efforts for new protocol. It was
> really
> >> > helpful for playing around the python thin client design.
> >> >
> >> > Could you explain some things that were still not clear for binary
> >> object
> >> > format:
> >> >
> >> > 1. What a reason to introduce separated type codes for arrays? Why
> just
> >> we
> >> > can't use the following?
> >> > *<1 byte universal array code>*
> >> > *<1 byte primitive code>*
> >> > *<4 bytes length>*
> >> > *<N bytes data>*
> >> >
> >> > We get 1 byte overhead but save 10 bytes in the code table. For arrays
> >> the
> >> > overhead is really insignificant:10 longs array takes now 1+4+4*10=45
> >> bytes
> >> > vs 1+1+4+4*10=46 bytes for the approach
> >> > Moreover for that appoach a new primitive code will be available for
> >> using
> >> > for array immediately.
> >> >
> >> > 2. Why the arrays force to use a selected type? For python there's no
> >> > limitations to use different types across one array (list). Would be
> >> good
> >> > to introduce a new type that will allow that. It could be look like
> >> > following
> >> > *<1 byte universal array code>*
> >> > *<1 byte no common type code*> <-- this says that every item must
> >> provide
> >> > its date type code like it does regular primitive data
> >> > *<4 bytes length>*
> >> > *<1 byte item 0 type code>*     <-- item provides its code
> >> > *<N byte item 0 data>*      <-- item provides its data
> >> > *<1 byte item 1 type code>*
> >> > *<N byte item 1 data>*
> >> > etc
> >> >
> >> > Also that allow to put nested arrays without changes in type code
> table!
> >> > For instance if we want to store 9 longs and 1 boolean it will take
> >> > now 1+1+4+(1+9)*4+(1+1)=48
> >> > bytes (vs 45 bytes to store as 10 longs as usual).
> >> >
> >> > 3. Ther's only one way to store a dictionary (key-value) structure as
> >> value
> >> > in the cache via Complex Object. But it looks like overcomplicated. I
> >> > suppose to introduce a code for that
> >> > *<1 byte key-value dictionary code>*
> >> >
> >> > *<4 bytes length>*
> >> > *<1 byte key 1 **name **type code>*
> >> > *<N byte key 1 name data>*
> >> > *<1 byte value 1 type code>*
> >> > *<N byte value 1 value>*
> >> > *<1 byte key 2 **name **type code>*
> >> > *<N byte key 2 name data>*
> >> > *<1 byte value 2 type code>*
> >> > *<N byte value 2 value>*
> >> > etc
> >> >
> >> > Also that allow to put nested dictionaries without changes in type
> code
> >> > table!
> >> > Of  course for the appoach above we get significat overhead for key
> >> > storing. But I think it is acceptable for some cases and definitely ok
> >> for
> >> > Python
> >> >
> >> >
> >> >
> >> >
> >> > On Wed, Nov 22, 2017 at 9:14 PM, Prachi Garg <pg...@gridgain.com>
> >> wrote:
> >> >
> >> > > Thanks Pavel! The document has good information. I'll create one on
> >> > > readme.io; will also add some examples there.
> >> > >
> >> > > On Wed, Nov 22, 2017 at 5:03 AM, Pavel Tupitsyn <
> ptupitsyn@apache.org
> >> >
> >> > > wrote:
> >> > >
> >> > > > Igniters,
> >> > > >
> >> > > > I've put together a detailed description of our Thin Client
> protocol
> >> > > > in form of IEP on wiki:
> >> > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> >> > > > 9+Thin+Client+Protocol
> >> > > >
> >> > > >
> >> > > > To clarify:
> >> > > > - Protocol implementation is in master (see ClientMessageParser
> >> class)
> >> > > > - Protocol has not been released yet, so we are free to change
> >> anything
> >> > > > - Protocol is only used by .NET Thin Client for now, but is
> >> supposed to
> >> > > be
> >> > > > used from other languages by third party contributors
> >> > > > - More operations will be added in future, this is a first set of
> >> them,
> >> > > > cache-related
> >> > > >
> >> > > >
> >> > > > Please review the document and let me know your thoughts.
> >> > > > Is there anything missing or wrong?
> >> > > >
> >> > > > We should make sure that the foundation is solid and extensible.
> >> > > >
> >> > > >
> >> > > > Thanks,
> >> > > > Pavel
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Sergey Kozlov
> >> > GridGain Systems
> >> > www.gridgain.com
> >> >
> >>
> >
> >
> >
> > --
> > Sergey Kozlov
> > GridGain Systems
> > www.gridgain.com
> >
>
>
>
> --
> Sergey Kozlov
> GridGain Systems
> www.gridgain.com
>

Re: Thin Client Protocol documentation

Posted by Sergey Kozlov <sk...@gridgain.com>.

Pavel

Could you update the page by following:

 - String, date, UUID arrays allow to put NULL. Due to that every item in
the array written as type code byte (default item type or null type code) +
type data. It should be detailed explained (looks like that the table
should have an addtional column called for instance "nullable")

-  UUID type takes 16 bytes length

thanks


On Mon, Nov 27, 2017 at 4:37 PM, Sergey Kozlov <sk...@gridgain.com> wrote:

> Pavel
>
> Thanks for explanations!
>
> On Mon, Nov 27, 2017 at 2:46 PM, Pavel Tupitsyn <pt...@apache.org>
> wrote:
>
>> Sergey,
>>
>> 1. Code table size does not affect anything, as I understand, so there is
>> no reason to introduce an extra byte.
>> 2. We have object arrays (code 23), I forgot to mention them, fixed.
>> 3. Also forgot, see code 25 in the updated document.
>>
>> Also note that operation codes have been updated (grouped by purpose) as
>> part of https://issues.apache.org/jira/browse/IGNITE-6989.
>>
>> Thanks,
>> Pavel
>>
>> On Sun, Nov 26, 2017 at 9:54 PM, Sergey Kozlov <sk...@gridgain.com>
>> wrote:
>>
>> > Pavel
>> >
>> > Thanks for the document and your efforts for new protocol. It was really
>> > helpful for playing around the python thin client design.
>> >
>> > Could you explain some things that were still not clear for binary
>> object
>> > format:
>> >
>> > 1. What a reason to introduce separated type codes for arrays? Why just
>> we
>> > can't use the following?
>> > *<1 byte universal array code>*
>> > *<1 byte primitive code>*
>> > *<4 bytes length>*
>> > *<N bytes data>*
>> >
>> > We get 1 byte overhead but save 10 bytes in the code table. For arrays
>> the
>> > overhead is really insignificant:10 longs array takes now 1+4+4*10=45
>> bytes
>> > vs 1+1+4+4*10=46 bytes for the approach
>> > Moreover for that appoach a new primitive code will be available for
>> using
>> > for array immediately.
>> >
>> > 2. Why the arrays force to use a selected type? For python there's no
>> > limitations to use different types across one array (list). Would be
>> good
>> > to introduce a new type that will allow that. It could be look like
>> > following
>> > *<1 byte universal array code>*
>> > *<1 byte no common type code*> <-- this says that every item must
>> provide
>> > its date type code like it does regular primitive data
>> > *<4 bytes length>*
>> > *<1 byte item 0 type code>*     <-- item provides its code
>> > *<N byte item 0 data>*      <-- item provides its data
>> > *<1 byte item 1 type code>*
>> > *<N byte item 1 data>*
>> > etc
>> >
>> > Also that allow to put nested arrays without changes in type code table!
>> > For instance if we want to store 9 longs and 1 boolean it will take
>> > now 1+1+4+(1+9)*4+(1+1)=48
>> > bytes (vs 45 bytes to store as 10 longs as usual).
>> >
>> > 3. Ther's only one way to store a dictionary (key-value) structure as
>> value
>> > in the cache via Complex Object. But it looks like overcomplicated. I
>> > suppose to introduce a code for that
>> > *<1 byte key-value dictionary code>*
>> >
>> > *<4 bytes length>*
>> > *<1 byte key 1 **name **type code>*
>> > *<N byte key 1 name data>*
>> > *<1 byte value 1 type code>*
>> > *<N byte value 1 value>*
>> > *<1 byte key 2 **name **type code>*
>> > *<N byte key 2 name data>*
>> > *<1 byte value 2 type code>*
>> > *<N byte value 2 value>*
>> > etc
>> >
>> > Also that allow to put nested dictionaries without changes in type code
>> > table!
>> > Of  course for the appoach above we get significat overhead for key
>> > storing. But I think it is acceptable for some cases and definitely ok
>> for
>> > Python
>> >
>> >
>> >
>> >
>> > On Wed, Nov 22, 2017 at 9:14 PM, Prachi Garg <pg...@gridgain.com>
>> wrote:
>> >
>> > > Thanks Pavel! The document has good information. I'll create one on
>> > > readme.io; will also add some examples there.
>> > >
>> > > On Wed, Nov 22, 2017 at 5:03 AM, Pavel Tupitsyn <ptupitsyn@apache.org
>> >
>> > > wrote:
>> > >
>> > > > Igniters,
>> > > >
>> > > > I've put together a detailed description of our Thin Client protocol
>> > > > in form of IEP on wiki:
>> > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-
>> > > > 9+Thin+Client+Protocol
>> > > >
>> > > >
>> > > > To clarify:
>> > > > - Protocol implementation is in master (see ClientMessageParser
>> class)
>> > > > - Protocol has not been released yet, so we are free to change
>> anything
>> > > > - Protocol is only used by .NET Thin Client for now, but is
>> supposed to
>> > > be
>> > > > used from other languages by third party contributors
>> > > > - More operations will be added in future, this is a first set of
>> them,
>> > > > cache-related
>> > > >
>> > > >
>> > > > Please review the document and let me know your thoughts.
>> > > > Is there anything missing or wrong?
>> > > >
>> > > > We should make sure that the foundation is solid and extensible.
>> > > >
>> > > >
>> > > > Thanks,
>> > > > Pavel
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Sergey Kozlov
>> > GridGain Systems
>> > www.gridgain.com
>> >
>>
>
>
>
> --
> Sergey Kozlov
> GridGain Systems
> www.gridgain.com
>



-- 
Sergey Kozlov
GridGain Systems
www.gridgain.com

Re: Thin Client Protocol documentation

Posted by Sergey Kozlov <sk...@gridgain.com>.

Pavel

Thanks for explanations!

On Mon, Nov 27, 2017 at 2:46 PM, Pavel Tupitsyn <pt...@apache.org>
wrote:

> Sergey,
>
> 1. Code table size does not affect anything, as I understand, so there is
> no reason to introduce an extra byte.
> 2. We have object arrays (code 23), I forgot to mention them, fixed.
> 3. Also forgot, see code 25 in the updated document.
>
> Also note that operation codes have been updated (grouped by purpose) as
> part of https://issues.apache.org/jira/browse/IGNITE-6989.
>
> Thanks,
> Pavel
>
> On Sun, Nov 26, 2017 at 9:54 PM, Sergey Kozlov <sk...@gridgain.com>
> wrote:
>
> > Pavel
> >
> > Thanks for the document and your efforts for new protocol. It was really
> > helpful for playing around the python thin client design.
> >
> > Could you explain some things that were still not clear for binary object
> > format:
> >
> > 1. What a reason to introduce separated type codes for arrays? Why just
> we
> > can't use the following?
> > *<1 byte universal array code>*
> > *<1 byte primitive code>*
> > *<4 bytes length>*
> > *<N bytes data>*
> >
> > We get 1 byte overhead but save 10 bytes in the code table. For arrays
> the
> > overhead is really insignificant:10 longs array takes now 1+4+4*10=45
> bytes
> > vs 1+1+4+4*10=46 bytes for the approach
> > Moreover for that appoach a new primitive code will be available for
> using
> > for array immediately.
> >
> > 2. Why the arrays force to use a selected type? For python there's no
> > limitations to use different types across one array (list). Would be good
> > to introduce a new type that will allow that. It could be look like
> > following
> > *<1 byte universal array code>*
> > *<1 byte no common type code*> <-- this says that every item must provide
> > its date type code like it does regular primitive data
> > *<4 bytes length>*
> > *<1 byte item 0 type code>*     <-- item provides its code
> > *<N byte item 0 data>*      <-- item provides its data
> > *<1 byte item 1 type code>*
> > *<N byte item 1 data>*
> > etc
> >
> > Also that allow to put nested arrays without changes in type code table!
> > For instance if we want to store 9 longs and 1 boolean it will take
> > now 1+1+4+(1+9)*4+(1+1)=48
> > bytes (vs 45 bytes to store as 10 longs as usual).
> >
> > 3. Ther's only one way to store a dictionary (key-value) structure as
> value
> > in the cache via Complex Object. But it looks like overcomplicated. I
> > suppose to introduce a code for that
> > *<1 byte key-value dictionary code>*
> >
> > *<4 bytes length>*
> > *<1 byte key 1 **name **type code>*
> > *<N byte key 1 name data>*
> > *<1 byte value 1 type code>*
> > *<N byte value 1 value>*
> > *<1 byte key 2 **name **type code>*
> > *<N byte key 2 name data>*
> > *<1 byte value 2 type code>*
> > *<N byte value 2 value>*
> > etc
> >
> > Also that allow to put nested dictionaries without changes in type code
> > table!
> > Of  course for the appoach above we get significat overhead for key
> > storing. But I think it is acceptable for some cases and definitely ok
> for
> > Python
> >
> >
> >
> >
> > On Wed, Nov 22, 2017 at 9:14 PM, Prachi Garg <pg...@gridgain.com> wrote:
> >
> > > Thanks Pavel! The document has good information. I'll create one on
> > > readme.io; will also add some examples there.
> > >
> > > On Wed, Nov 22, 2017 at 5:03 AM, Pavel Tupitsyn <pt...@apache.org>
> > > wrote:
> > >
> > > > Igniters,
> > > >
> > > > I've put together a detailed description of our Thin Client protocol
> > > > in form of IEP on wiki:
> > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> > > > 9+Thin+Client+Protocol
> > > >
> > > >
> > > > To clarify:
> > > > - Protocol implementation is in master (see ClientMessageParser
> class)
> > > > - Protocol has not been released yet, so we are free to change
> anything
> > > > - Protocol is only used by .NET Thin Client for now, but is supposed
> to
> > > be
> > > > used from other languages by third party contributors
> > > > - More operations will be added in future, this is a first set of
> them,
> > > > cache-related
> > > >
> > > >
> > > > Please review the document and let me know your thoughts.
> > > > Is there anything missing or wrong?
> > > >
> > > > We should make sure that the foundation is solid and extensible.
> > > >
> > > >
> > > > Thanks,
> > > > Pavel
> > > >
> > >
> >
> >
> >
> > --
> > Sergey Kozlov
> > GridGain Systems
> > www.gridgain.com
> >
>



-- 
Sergey Kozlov
GridGain Systems
www.gridgain.com

Re: Thin Client Protocol documentation

Posted by Pavel Tupitsyn <pt...@apache.org>.

Sergey,

1. Code table size does not affect anything, as I understand, so there is
no reason to introduce an extra byte.
2. We have object arrays (code 23), I forgot to mention them, fixed.
3. Also forgot, see code 25 in the updated document.

Also note that operation codes have been updated (grouped by purpose) as
part of https://issues.apache.org/jira/browse/IGNITE-6989.

Thanks,
Pavel

On Sun, Nov 26, 2017 at 9:54 PM, Sergey Kozlov <sk...@gridgain.com> wrote:

> Pavel
>
> Thanks for the document and your efforts for new protocol. It was really
> helpful for playing around the python thin client design.
>
> Could you explain some things that were still not clear for binary object
> format:
>
> 1. What a reason to introduce separated type codes for arrays? Why just we
> can't use the following?
> *<1 byte universal array code>*
> *<1 byte primitive code>*
> *<4 bytes length>*
> *<N bytes data>*
>
> We get 1 byte overhead but save 10 bytes in the code table. For arrays the
> overhead is really insignificant:10 longs array takes now 1+4+4*10=45 bytes
> vs 1+1+4+4*10=46 bytes for the approach
> Moreover for that appoach a new primitive code will be available for using
> for array immediately.
>
> 2. Why the arrays force to use a selected type? For python there's no
> limitations to use different types across one array (list). Would be good
> to introduce a new type that will allow that. It could be look like
> following
> *<1 byte universal array code>*
> *<1 byte no common type code*> <-- this says that every item must provide
> its date type code like it does regular primitive data
> *<4 bytes length>*
> *<1 byte item 0 type code>*     <-- item provides its code
> *<N byte item 0 data>*      <-- item provides its data
> *<1 byte item 1 type code>*
> *<N byte item 1 data>*
> etc
>
> Also that allow to put nested arrays without changes in type code table!
> For instance if we want to store 9 longs and 1 boolean it will take
> now 1+1+4+(1+9)*4+(1+1)=48
> bytes (vs 45 bytes to store as 10 longs as usual).
>
> 3. Ther's only one way to store a dictionary (key-value) structure as value
> in the cache via Complex Object. But it looks like overcomplicated. I
> suppose to introduce a code for that
> *<1 byte key-value dictionary code>*
>
> *<4 bytes length>*
> *<1 byte key 1 **name **type code>*
> *<N byte key 1 name data>*
> *<1 byte value 1 type code>*
> *<N byte value 1 value>*
> *<1 byte key 2 **name **type code>*
> *<N byte key 2 name data>*
> *<1 byte value 2 type code>*
> *<N byte value 2 value>*
> etc
>
> Also that allow to put nested dictionaries without changes in type code
> table!
> Of  course for the appoach above we get significat overhead for key
> storing. But I think it is acceptable for some cases and definitely ok for
> Python
>
>
>
>
> On Wed, Nov 22, 2017 at 9:14 PM, Prachi Garg <pg...@gridgain.com> wrote:
>
> > Thanks Pavel! The document has good information. I'll create one on
> > readme.io; will also add some examples there.
> >
> > On Wed, Nov 22, 2017 at 5:03 AM, Pavel Tupitsyn <pt...@apache.org>
> > wrote:
> >
> > > Igniters,
> > >
> > > I've put together a detailed description of our Thin Client protocol
> > > in form of IEP on wiki:
> > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> > > 9+Thin+Client+Protocol
> > >
> > >
> > > To clarify:
> > > - Protocol implementation is in master (see ClientMessageParser class)
> > > - Protocol has not been released yet, so we are free to change anything
> > > - Protocol is only used by .NET Thin Client for now, but is supposed to
> > be
> > > used from other languages by third party contributors
> > > - More operations will be added in future, this is a first set of them,
> > > cache-related
> > >
> > >
> > > Please review the document and let me know your thoughts.
> > > Is there anything missing or wrong?
> > >
> > > We should make sure that the foundation is solid and extensible.
> > >
> > >
> > > Thanks,
> > > Pavel
> > >
> >
>
>
>
> --
> Sergey Kozlov
> GridGain Systems
> www.gridgain.com
>

Re: Thin Client Protocol documentation

Posted by Sergey Kozlov <sk...@gridgain.com>.

Pavel

Thanks for the document and your efforts for new protocol. It was really
helpful for playing around the python thin client design.

Could you explain some things that were still not clear for binary object
format:

1. What a reason to introduce separated type codes for arrays? Why just we
can't use the following?
*<1 byte universal array code>*
*<1 byte primitive code>*
*<4 bytes length>*
*<N bytes data>*

We get 1 byte overhead but save 10 bytes in the code table. For arrays the
overhead is really insignificant:10 longs array takes now 1+4+4*10=45 bytes
vs 1+1+4+4*10=46 bytes for the approach
Moreover for that appoach a new primitive code will be available for using
for array immediately.

2. Why the arrays force to use a selected type? For python there's no
limitations to use different types across one array (list). Would be good
to introduce a new type that will allow that. It could be look like
following
*<1 byte universal array code>*
*<1 byte no common type code*> <-- this says that every item must provide
its date type code like it does regular primitive data
*<4 bytes length>*
*<1 byte item 0 type code>*     <-- item provides its code
*<N byte item 0 data>*      <-- item provides its data
*<1 byte item 1 type code>*
*<N byte item 1 data>*
etc

Also that allow to put nested arrays without changes in type code table!
For instance if we want to store 9 longs and 1 boolean it will take
now 1+1+4+(1+9)*4+(1+1)=48
bytes (vs 45 bytes to store as 10 longs as usual).

3. Ther's only one way to store a dictionary (key-value) structure as value
in the cache via Complex Object. But it looks like overcomplicated. I
suppose to introduce a code for that
*<1 byte key-value dictionary code>*

*<4 bytes length>*
*<1 byte key 1 **name **type code>*
*<N byte key 1 name data>*
*<1 byte value 1 type code>*
*<N byte value 1 value>*
*<1 byte key 2 **name **type code>*
*<N byte key 2 name data>*
*<1 byte value 2 type code>*
*<N byte value 2 value>*
etc

Also that allow to put nested dictionaries without changes in type code
table!
Of  course for the appoach above we get significat overhead for key
storing. But I think it is acceptable for some cases and definitely ok for
Python

On Wed, Nov 22, 2017 at 9:14 PM, Prachi Garg <pg...@gridgain.com> wrote:

> Thanks Pavel! The document has good information. I'll create one on
> readme.io; will also add some examples there.
>
> On Wed, Nov 22, 2017 at 5:03 AM, Pavel Tupitsyn <pt...@apache.org>
> wrote:
>
> > Igniters,
> >
> > I've put together a detailed description of our Thin Client protocol
> > in form of IEP on wiki:
> > https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> > 9+Thin+Client+Protocol
> >
> >
> > To clarify:
> > - Protocol implementation is in master (see ClientMessageParser class)
> > - Protocol has not been released yet, so we are free to change anything
> > - Protocol is only used by .NET Thin Client for now, but is supposed to
> be
> > used from other languages by third party contributors
> > - More operations will be added in future, this is a first set of them,
> > cache-related
> >
> >
> > Please review the document and let me know your thoughts.
> > Is there anything missing or wrong?
> >
> > We should make sure that the foundation is solid and extensible.
> >
> >
> > Thanks,
> > Pavel
> >
>

-- 
Sergey Kozlov
GridGain Systems
www.gridgain.com

Re: Thin Client Protocol documentation

Posted by Prachi Garg <pg...@gridgain.com>.

Thanks Pavel! The document has good information. I'll create one on
readme.io; will also add some examples there.

On Wed, Nov 22, 2017 at 5:03 AM, Pavel Tupitsyn <pt...@apache.org>
wrote:

> Igniters,
>
> I've put together a detailed description of our Thin Client protocol
> in form of IEP on wiki:
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> 9+Thin+Client+Protocol
>
>
> To clarify:
> - Protocol implementation is in master (see ClientMessageParser class)
> - Protocol has not been released yet, so we are free to change anything
> - Protocol is only used by .NET Thin Client for now, but is supposed to be
> used from other languages by third party contributors
> - More operations will be added in future, this is a first set of them,
> cache-related
>
>
> Please review the document and let me know your thoughts.
> Is there anything missing or wrong?
>
> We should make sure that the foundation is solid and extensible.
>
>
> Thanks,
> Pavel
>