Doxygen patch and questions on Spec, Maps, Blocks and RPC

[Matt Massie <ma...@cloudera.com>]

I've just submitted a patch to build C documentation using doxygen

https://issues.apache.org/jira/browse/AVRO-37


I'm starting work on building file/socket handles and processing the  
Avro compound data types (expect a patch with proper code/unit tests  
soon) and I have a few (hopefully quick) questions:

(1) Spec related: Should we have a max-length attribute for the  
variable length objects?

Since we're going to be using Avro for RPC, do we need to consider the  
possibility of malicious data in the Avro streams?  Malicious clients  
could swamp out the memory on the server by sending intentionally long  
values (e.g. a string that is 2GB in length).  Instead of having a  
server-wide max, it might make sense to allow a maximum length to be  
specified per object.


(2) Maps: Do we need to maintain the key/value pair order for maps?

I will be converting Avro maps to apr_table_t structures in order to  
make key search constant time.  Do I need to guarantee that the order  
I decode the map is the order I encode it later?  Just want to know if  
I need to store a private key array to maintain the order.  It's ok if  
I do, but I'd like to use less memory if I can avoid it.


(3) Blocks: Sanity check

If the elements of an array are fixed length (e.g. 8 bytes), then the  
block of 100 of them would look like...

[ long = 100 ][ 100 * 8 = 800 bytes of data in the block][ long = 0 ]

... terminated with a zero.. or

[ long = 90 ][ 90 * 8 = 720 bytes of data in the block ][ long = 10 ] 
[ 10 * 8 = 80 bytes in the block ][ long = 0]

.. correct?

However, if the objects are variable length, there is no way to  
calculate the size of the block based on the element sizes so we use  
the negative "count" value.  For example...

[ long = -1 ][ long = 23948 ][ 23948 bytes of data in the block ] 
[ long = 0 ]

.. which is terminated with a zero.


(4) RPC related: Should we explicitly specify the entire RPC  
communication as an Avro schema?

For examples, the entire RPC communication schema can be expressed in  
a single XDR .x file.  The zeroc guys who wrote ICE express the RPC of  
all their components using their IDL.

Having the Avro RPC shema would make implementing RPC automatic and  
flexible.


Hope you all have a great Memorial Day weekend!

-Matt

Re: Doxygen patch and questions on Spec, Maps, Blocks and RPC

[Doug Cutting <cu...@apache.org>]

Matt Massie wrote:
>>> (4) RPC related: Should we explicitly specify the entire RPC 
>>> communication as an Avro schema?
>>
>> You mean the handshake stuff?  I've thought about that, but felt that 
>> the bootstrapping got complicated.
> 
> Yes.  Actually, I believe that we should express all messages exchanged 
> between Avro components in Avro schema so that we don't need to 
> hand-craft the RPC layer.

I've added this to AVRO-2.

It's tempting to use an Avro protocol to describe the exchange, but 
since this data is appended to the front of initial request and response 
payloads I don't see how it would be able to share code with other Avro 
protocols.

https://issues.apache.org/jira/browse/AVRO-2

Please comment there.

Doug

Re: Doxygen patch and questions on Spec, Maps, Blocks and RPC

[Matt Massie <ma...@cloudera.com>]

On May 22, 2009, at 4:32 PM, Doug Cutting wrote:

> Matt Massie wrote:
>> (1) Spec related: Should we have a max-length attribute for the  
>> variable length objects?
>
> My instinct would be to put this in the server implementation--it  
> should guard against requests that use too much memory.

That will work.  If this approach doesn't work as well as we'd like,  
we can update the spec at a later time.

>
>> (2) Maps: Do we need to maintain the key/value pair order for maps?
>
> I think re-ordering is fine.  Does anyone disagree?

My vote would be to not impose any ordering on maps.

>
>> (3) Blocks: Sanity check
>> If the elements of an array are fixed length (e.g. 8 bytes), then  
>> the block of 100 of them would look like...
>> [ long = 100 ][ 100 * 8 = 800 bytes of data in the block][ long = 0 ]
>> ... terminated with a zero.. or
>> [ long = 90 ][ 90 * 8 = 720 bytes of data in the block ][ long =  
>> 10 ][ 10 * 8 = 80 bytes in the block ][ long = 0]
>> .. correct?
>
> Yes, that's the idea.
>
>> However, if the objects are variable length, there is no way to  
>> calculate the size of the block based on the element sizes so we  
>> use the negative "count" value.  For example...
>> [ long = -1 ][ long = 23948 ][ 23948 bytes of data in the block ] 
>> [ long = 0 ]
>> .. which is terminated with a zero.
>
> Almost.  AVRO-25 proposes to permit, for your first example:
>
> [long = -100] [long = 800] [100 * 8 = 800 bytes of data] [long = 0]
>
> The item count is always required, but when its negative, its  
> followed by the byte count.  This is the same for variable and fixed- 
> sized data.

I guess one advantage of requiring the "count" is to allow an extra  
check once the block is processed.


>> (4) RPC related: Should we explicitly specify the entire RPC  
>> communication as an Avro schema?
>
> You mean the handshake stuff?  I've thought about that, but felt  
> that the bootstrapping got complicated.

Yes.  Actually, I believe that we should express all messages  
exchanged between Avro components in Avro schema so that we don't need  
to hand-craft the RPC layer.  Having the schema will also make the  
protocols more transparent and Avro more adaptive to changes in RPC.

Are there any reasons you can see for hand-coding the RPC layer?

-Matt

Re: Doxygen patch and questions on Spec, Maps, Blocks and RPC

[Doug Cutting <cu...@apache.org>]

Matt Massie wrote:
> (1) Spec related: Should we have a max-length attribute for the variable 
> length objects?

My instinct would be to put this in the server implementation--it should 
guard against requests that use too much memory.

> (2) Maps: Do we need to maintain the key/value pair order for maps?

I think re-ordering is fine.  Does anyone disagree?

> (3) Blocks: Sanity check
> 
> If the elements of an array are fixed length (e.g. 8 bytes), then the 
> block of 100 of them would look like...
> 
> [ long = 100 ][ 100 * 8 = 800 bytes of data in the block][ long = 0 ]
> 
> ... terminated with a zero.. or
> 
> [ long = 90 ][ 90 * 8 = 720 bytes of data in the block ][ long = 10 ][ 
> 10 * 8 = 80 bytes in the block ][ long = 0]
> 
> .. correct?

Yes, that's the idea.

> However, if the objects are variable length, there is no way to 
> calculate the size of the block based on the element sizes so we use the 
> negative "count" value.  For example...
> 
> [ long = -1 ][ long = 23948 ][ 23948 bytes of data in the block ][ long 
> = 0 ]
> 
> .. which is terminated with a zero.

Almost.  AVRO-25 proposes to permit, for your first example:

  [long = -100] [long = 800] [100 * 8 = 800 bytes of data] [long = 0]

The item count is always required, but when its negative, its followed 
by the byte count.  This is the same for variable and fixed-sized data.

> (4) RPC related: Should we explicitly specify the entire RPC 
> communication as an Avro schema?

You mean the handshake stuff?  I've thought about that, but felt that 
the bootstrapping got complicated.

Cheers,

Doug