You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@etch.apache.org by Peter Rodgers <pj...@1060.org> on 2010/03/30 22:00:40 UTC
Streaming Support / Project Status
I've been playing with Etch in Java and have successfully got a client
talking with a server with a full bi-directional conversation. Currently
the messages are just simple string exchanges.
I can see that I can easily extend types by using @Extern and defining a
serializer by extending ImportExportHelper. However for various
reasons, mostly to do with the memory overhead of using byte[], I really
want to be able to send byte streams.
In Java the objects I need to send over the wire can give me an
InputStream or can write(OutputStream). Is there any way for me to
issue streams? Presumably somehow I could go below the Message layer
and write to the Packetizer / Stream Transport layer - but that seems a
bit hacky.
I can see that part of Etch's intent is to abstract away from streams -
but I need a mixed solution where I can send struct-like object messages
for protocol-state transfer and streaming for data payload.
Reading over the mailing list I see that activity has been pretty quiet
and the 1.1 release doesn't seem to have made it yet. Is this project
still active?
Thanks in advance,
Peter
Re: Streaming Support / Project Status
Posted by scott comer <we...@mac.com>.
hi peter.
this sort of question generally should go to etch-user and not etch-dev.
i'm copying etch-users, as it might be useful to other users of etch who
aren't developers of the etch system.
several things come to mind:
0) welcome. always good to see another user!
1) defining serializers is the way to go because the higher level data
they deal with is way more compact than java object serializations, by a
LOT. (in one simple test, 49 bytes vs. several hundred). not only that,
it is way FASTER.
2) object serialization is absolutely not portable between versions and
different languages, whereas the etch object serialization is. by that i
mean between different versions of your application, where you might
have added or deleted fields or types. and different languages such as
c, python, ruby, c#, etc. you might not care about that now, but i've
been on many projects where that became important later.
3) etch imposes a limit on the size of packets, around 16k. you can
change that, but not arbitrarily high. the max is around 4 mb. the
larger the packet size, the less reactive your system will be and the
fewer concurrent connections you will be able to support.
4) you can define methods in your protocol to deliver byte arrays from
one side to the other. but you have to chunk them if they're too large.
that's ok, too. just define a subclass of OutputStream that makes calls
on your etch connection to write bytes to the stream. wrap it in a
BufferedOutputStream, voila, you're in business. you'll need some
identifier to define which stream you're writing to, to keep data from
multiple streams separate.
5) on the receiving end, when you receive one of these byte arrays,
you'll have to reassemble them or buffer them or something until you get
the whole thing. say, using a ByteArrayOutputStream. when you get the
last, you can then turn around and make a ByteArrayInputStream out of
the resulting array and pass it off to an ObjectInputStream.
6) some of us are still here, and i'm using the 1.1 candidate in my
current project. that's a startup, and my wife is in school, and the
kids are fun but a lot of work, and so i'm pretty busy and don't have as
much time as i'd like for etch.
scott out
On 3/30/2010 3:00 PM, Peter Rodgers wrote:
> I've been playing with Etch in Java and have successfully got a client
> talking with a server with a full bi-directional conversation.
> Currently the messages are just simple string exchanges.
>
> I can see that I can easily extend types by using @Extern and defining
> a serializer by extending ImportExportHelper. However for various
> reasons, mostly to do with the memory overhead of using byte[], I
> really want to be able to send byte streams.
>
> In Java the objects I need to send over the wire can give me an
> InputStream or can write(OutputStream). Is there any way for me to
> issue streams? Presumably somehow I could go below the Message layer
> and write to the Packetizer / Stream Transport layer - but that seems
> a bit hacky.
>
> I can see that part of Etch's intent is to abstract away from streams
> - but I need a mixed solution where I can send struct-like object
> messages for protocol-state transfer and streaming for data payload.
>
> Reading over the mailing list I see that activity has been pretty
> quiet and the 1.1 release doesn't seem to have made it yet. Is this
> project still active?
>
> Thanks in advance,
>
> Peter
Re: Streaming Support / Project Status
Posted by scott comer <we...@mac.com>.
hi peter.
this sort of question generally should go to etch-user and not etch-dev.
i'm copying etch-users, as it might be useful to other users of etch who
aren't developers of the etch system.
several things come to mind:
0) welcome. always good to see another user!
1) defining serializers is the way to go because the higher level data
they deal with is way more compact than java object serializations, by a
LOT. (in one simple test, 49 bytes vs. several hundred). not only that,
it is way FASTER.
2) object serialization is absolutely not portable between versions and
different languages, whereas the etch object serialization is. by that i
mean between different versions of your application, where you might
have added or deleted fields or types. and different languages such as
c, python, ruby, c#, etc. you might not care about that now, but i've
been on many projects where that became important later.
3) etch imposes a limit on the size of packets, around 16k. you can
change that, but not arbitrarily high. the max is around 4 mb. the
larger the packet size, the less reactive your system will be and the
fewer concurrent connections you will be able to support.
4) you can define methods in your protocol to deliver byte arrays from
one side to the other. but you have to chunk them if they're too large.
that's ok, too. just define a subclass of OutputStream that makes calls
on your etch connection to write bytes to the stream. wrap it in a
BufferedOutputStream, voila, you're in business. you'll need some
identifier to define which stream you're writing to, to keep data from
multiple streams separate.
5) on the receiving end, when you receive one of these byte arrays,
you'll have to reassemble them or buffer them or something until you get
the whole thing. say, using a ByteArrayOutputStream. when you get the
last, you can then turn around and make a ByteArrayInputStream out of
the resulting array and pass it off to an ObjectInputStream.
6) some of us are still here, and i'm using the 1.1 candidate in my
current project. that's a startup, and my wife is in school, and the
kids are fun but a lot of work, and so i'm pretty busy and don't have as
much time as i'd like for etch.
scott out
On 3/30/2010 3:00 PM, Peter Rodgers wrote:
> I've been playing with Etch in Java and have successfully got a client
> talking with a server with a full bi-directional conversation.
> Currently the messages are just simple string exchanges.
>
> I can see that I can easily extend types by using @Extern and defining
> a serializer by extending ImportExportHelper. However for various
> reasons, mostly to do with the memory overhead of using byte[], I
> really want to be able to send byte streams.
>
> In Java the objects I need to send over the wire can give me an
> InputStream or can write(OutputStream). Is there any way for me to
> issue streams? Presumably somehow I could go below the Message layer
> and write to the Packetizer / Stream Transport layer - but that seems
> a bit hacky.
>
> I can see that part of Etch's intent is to abstract away from streams
> - but I need a mixed solution where I can send struct-like object
> messages for protocol-state transfer and streaming for data payload.
>
> Reading over the mailing list I see that activity has been pretty
> quiet and the 1.1 release doesn't seem to have made it yet. Is this
> project still active?
>
> Thanks in advance,
>
> Peter