You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@etch.apache.org by HERKEL Jakub <J....@cpce.net> on 2011/01/08 12:02:32 UTC
Etch messages serialization subsystem
Hello,
Is there a possibility how we can use only etch messages serialization
subsystem? In our case we want to use IPC transport between processes so
we don't need TPC/IP transport layer and I would like to use etch
messages that would be serialized to binary stream/memory and
transported via IPC.
Thanks for your ideas
jakub
Re: Etch messages serialization subsystem
Posted by scott comer <we...@mac.com>.
as you probably know from reading the big message problem, the technique
is encode, split, reassemble, decode. the architecture problem is how to
split and reassemble. transport needs to move the pieces reliably and
reassemble in the proper order.
the current packet format doesn't allow for specifying fragment k out of
n fragments.
the current receiver has no idea of buffering.
the current binary input and output codec has no idea of incremental coding.
one workaround, easily implemented, is to bump the packetizer's
maxbuffersize (gives you up to 4 meg).
harder workaround, easily implemented, is to raise the hard 4m limit in
the buffer pool. note that java doesn't perform very well with buffers
larger than 4m. it will trash your heap.
the best workaround is to start with tcp2 impl, and using bufferpool
blocks to code and decode large packets as a series of 1-4k blocks.
doesn't trash the heap as badly. you may still need to tune but mostly
should work.
scott out
On 1/11/2011 2:52 AM, Jakub Herkel wrote:
> Hello,
>
> Thanks for overview, it doesn't seem like a hard task. But after
> having read "Big message problem" article I see one possible pitfall
> there. We want to use Etch as a middle layer between two our parts.
> But these parts have communication where we cannot know message size
> exactly. For better understanding, one of our part is something like a
> device driver. Every driver contains variables which we want to read
> or write. But we don't know any information how large variable can be.
> We only know information about the type of a variable, the name of a
> variable but not its maximum size. It is no problem for integer
> variables but for example for binary data or strings it can be one. We
> have implemented IPC communication based on Boost libs that can split
> one memory block to some smaller memory blocks and send it over IPC
> queue.
> My question is, is there a possibility how to split a message in Etch?
> After reading the abovementioned article and some emails from mailing
> list I think that it is not possible, but I haven't studied source
> code. If it isn't possible, is there a possibility how to use only
> message serialization subsystem? If I understand correctly, this
> limitation is coming from network/service layer, isn't it? So maybe we
> can serialize etch message to memory stream and after that send it via
> our IPC subsystem which is able to split memory stream to some chunks.
>
> best regards
>
> jakub
>
>
> On Sat, 2011-01-08 at 14:47 +0100, Holger Grandy wrote:
>> Hello Jakub,
>>
>> that should be possible. What you need to do is implement your own
>> transport (No Tcp, but shared memory or whatever you like). First check
>> http://incubator.apache.org/etch/architecture.html for an overview of
>> the Etch architecture (the diagram is not perfectly in sync with current
>> trunk, but you will get the idea). You will see that Etch has a layered
>> architecture which supports exchanging the single layers, which is
>> what you want (for the lowest layer)
>>
>> You can take
>> examples/helloworld/target/generated-sources/main/etch/java/org/apache/etch/examples/helloworld/HelloWorldHelper.java
>> as an example on how the communication stack is built up. Check e.g.
>> the newServer Method there. What you need to do is implement your own
>> TransportMessage implementation for your desired communication method.
>> The Interface implementation is built by the TransportFactory (e.g.
>> for Java in
>> binding-java/runtime/src/main/java/org/apache/etch/bindings/java/support/)
>> Add your own TransportFactory Implementation and add your transport
>> mechanism as an implementation of the TransportMessage interface.
>>
>> Info on how to build trunk is available at
>> http://incubator.apache.org/etch/building-trunk.html
>> Hope this helps you as a first quick introduction.
>>
>> If you want to do implement this, I would be very interested in your
>> experiences.
>> Keep us up to date!
>>
>> Regards,
>> Holger
>>
>> On Sat, Jan 8, 2011 at 12:02 PM, HERKEL Jakub <J....@cpce.net> wrote:
>> > Hello,
>> >
>> >
>> >
>> > Is there a possibility how we can use only etch messages serialization
>> > subsystem? In our case we want to use IPC transport between
>> processes so we
>> > don’t need TPC/IP transport layer and I would like to use etch
>> messages that
>> > would be serialized to binary stream/memory and transported via IPC.
>> >
>> >
>> >
>> > Thanks for your ideas
>> >
>> >
>> >
>> > jakub
>> >
>> >
>>
>>
>
Re: Etch messages serialization subsystem
Posted by Jakub Herkel <j....@cpce.net>.
Hello,
Thanks for overview, it doesn't seem like a hard task. But after having
read "Big message problem" article I see one possible pitfall there. We
want to use Etch as a middle layer between two our parts. But these
parts have communication where we cannot know message size exactly. For
better understanding, one of our part is something like a device driver.
Every driver contains variables which we want to read or write. But we
don't know any information how large variable can be. We only know
information about the type of a variable, the name of a variable but not
its maximum size. It is no problem for integer variables but for example
for binary data or strings it can be one. We have implemented IPC
communication based on Boost libs that can split one memory block to
some smaller memory blocks and send it over IPC queue.
My question is, is there a possibility how to split a message in Etch?
After reading the abovementioned article and some emails from mailing
list I think that it is not possible, but I haven't studied source code.
If it isn't possible, is there a possibility how to use only message
serialization subsystem? If I understand correctly, this limitation is
coming from network/service layer, isn't it? So maybe we can serialize
etch message to memory stream and after that send it via our IPC
subsystem which is able to split memory stream to some chunks.
best regards
jakub
On Sat, 2011-01-08 at 14:47 +0100, Holger Grandy wrote:
> Hello Jakub,
>
> that should be possible. What you need to do is implement your own
> transport (No Tcp, but shared memory or whatever you like). First
> check
> http://incubator.apache.org/etch/architecture.html for an overview of
> the Etch architecture (the diagram is not perfectly in sync with
> current
> trunk, but you will get the idea). You will see that Etch has a
> layered
> architecture which supports exchanging the single layers, which is
> what you want (for the lowest layer)
>
> You can take
> examples/helloworld/target/generated-sources/main/etch/java/org/apache/etch/examples/helloworld/HelloWorldHelper.java
> as an example on how the communication stack is built up. Check e.g.
> the newServer Method there. What you need to do is implement your own
> TransportMessage implementation for your desired communication method.
> The Interface implementation is built by the TransportFactory (e.g.
> for Java in
> binding-java/runtime/src/main/java/org/apache/etch/bindings/java/support/)
> Add your own TransportFactory Implementation and add your transport
> mechanism as an implementation of the TransportMessage interface.
>
> Info on how to build trunk is available at
> http://incubator.apache.org/etch/building-trunk.html
> Hope this helps you as a first quick introduction.
>
> If you want to do implement this, I would be very interested in your
> experiences.
> Keep us up to date!
>
> Regards,
> Holger
>
> On Sat, Jan 8, 2011 at 12:02 PM, HERKEL Jakub <J....@cpce.net>
> wrote:
> > Hello,
> >
> >
> >
> > Is there a possibility how we can use only etch messages
> serialization
> > subsystem? In our case we want to use IPC transport between
> processes so we
> > don’t need TPC/IP transport layer and I would like to use etch
> messages that
> > would be serialized to binary stream/memory and transported via IPC.
> >
> >
> >
> > Thanks for your ideas
> >
> >
> >
> > jakub
> >
> >
>
>
Re: Etch messages serialization subsystem
Posted by Holger Grandy <gr...@apache.org>.
Hello Jakub,
that should be possible. What you need to do is implement your own
transport (No Tcp, but shared memory or whatever you like). First check
http://incubator.apache.org/etch/architecture.html for an overview of
the Etch architecture (the diagram is not perfectly in sync with current
trunk, but you will get the idea). You will see that Etch has a layered
architecture which supports exchanging the single layers, which is
what you want (for the lowest layer)
You can take examples/helloworld/target/generated-sources/main/etch/java/org/apache/etch/examples/helloworld/HelloWorldHelper.java
as an example on how the communication stack is built up. Check e.g.
the newServer Method there. What you need to do is implement your own
TransportMessage implementation for your desired communication method.
The Interface implementation is built by the TransportFactory (e.g.
for Java in binding-java/runtime/src/main/java/org/apache/etch/bindings/java/support/)
Add your own TransportFactory Implementation and add your transport
mechanism as an implementation of the TransportMessage interface.
Info on how to build trunk is available at
http://incubator.apache.org/etch/building-trunk.html
Hope this helps you as a first quick introduction.
If you want to do implement this, I would be very interested in your
experiences.
Keep us up to date!
Regards,
Holger
On Sat, Jan 8, 2011 at 12:02 PM, HERKEL Jakub <J....@cpce.net> wrote:
> Hello,
>
>
>
> Is there a possibility how we can use only etch messages serialization
> subsystem? In our case we want to use IPC transport between processes so we
> don’t need TPC/IP transport layer and I would like to use etch messages that
> would be serialized to binary stream/memory and transported via IPC.
>
>
>
> Thanks for your ideas
>
>
>
> jakub
>
>
Re: Etch messages serialization subsystem
Posted by scott comer <we...@mac.com>.
as holger said, but an example of what you want to implement is
TcpTransportFactory.
just make your IpcTransportFactory, define it using
TransportFactory.add("ipc", "fully.qualified.IpcTransportFactory"), and
then it will be invoked to
construct stacks for uris like "ipc:..."
On 1/8/2011 5:02 AM, HERKEL Jakub wrote:
>
> Hello,
>
> Is there a possibility how we can use only etch messages serialization
> subsystem? In our case we want to use IPC transport between processes
> so we don't need TPC/IP transport layer and I would like to use etch
> messages that would be serialized to binary stream/memory and
> transported via IPC.
>
> Thanks for your ideas
>
> jakub
>