Re: Streaming Support / Project Status

[scott comer <we...@mac.com>]

hi peter.

this sort of question generally should go to etch-user and not etch-dev. 
i'm copying etch-users, as it might be useful to other users of etch who 
aren't developers of the etch system.

several things come to mind:

0) welcome. always good to see another user!

1) defining serializers is the way to go because the higher level data 
they deal with is way more compact than java object serializations, by a 
LOT. (in one simple test, 49 bytes vs. several hundred). not only that, 
it is way FASTER.

2) object serialization is absolutely not portable between versions and 
different languages, whereas the etch object serialization is. by that i 
mean between different versions of your application, where you might 
have added or deleted fields or types. and different languages such as 
c, python, ruby, c#, etc. you might not care about that now, but i've 
been on many projects where that became important later.

3) etch imposes a limit on the size of packets, around 16k. you can 
change that, but not arbitrarily high. the max is around 4 mb. the 
larger the packet size, the less reactive your system will be and the 
fewer concurrent connections you will be able to support.

4) you can define methods in your protocol to deliver byte arrays from 
one side to the other. but you have to chunk them if they're too large. 
that's ok, too. just define a subclass of OutputStream that makes calls 
on your etch connection to write bytes to the stream. wrap it in a 
BufferedOutputStream, voila, you're in business. you'll need some 
identifier to define which stream you're writing to, to keep data from 
multiple streams separate.

5) on the receiving end, when you receive one of these byte arrays, 
you'll have to reassemble them or buffer them or something until you get 
the whole thing. say, using a ByteArrayOutputStream. when you get the 
last, you can then turn around and make a ByteArrayInputStream out of 
the resulting array and pass it off to an ObjectInputStream.

6) some of us are still here, and i'm using the 1.1 candidate in my 
current project. that's a startup, and my wife is in school, and the 
kids are fun but a lot of work, and so i'm pretty busy and don't have as 
much time as i'd like for etch.

scott out

On 3/30/2010 3:00 PM, Peter Rodgers wrote:
> I've been playing with Etch in Java and have successfully got a client 
> talking with a server with a full bi-directional conversation. 
> Currently the messages are just simple string exchanges.
>
> I can see that I can easily extend types by using @Extern and defining 
> a serializer by extending ImportExportHelper.  However for various 
> reasons, mostly to do with the memory overhead of using byte[], I 
> really want to be able to send byte streams.
>
> In Java the objects I need to send over the wire can give me an 
> InputStream or can write(OutputStream).  Is there any way for me to 
> issue streams?  Presumably somehow I could go below the Message layer 
> and write to the Packetizer / Stream Transport layer - but that seems 
> a bit hacky.
>
> I can see that part of Etch's intent is to abstract away from streams 
> - but I need a mixed solution where I can send struct-like object 
> messages for protocol-state transfer and streaming for data payload.
>
> Reading over the mailing list I see that activity has been pretty 
> quiet and the 1.1 release doesn't seem to have made it yet.  Is this 
> project still active?
>
> Thanks in advance,
>
> Peter