Enhancement to XMI deserializer as a foundation for remote parallel processing

[Adam Lally <al...@alum.rpi.edu>]

I am working on some modifications to the XMI deserializer that will
allow the following scenario:

1) A CAS is serialized to XMI.
2) Copies of the XMI documents are sent to multiple remote services
3) Each remote service appends to the CAS (does not delete or modify
existing stuff) and responds with a new serialized XMI CAS)
4) The multiple XMI responses are all merged back into a single CAS instance

This would permit multiple remote services that don't depend on each
other to run in parallel, assuming they only append to the CAS (which
is common).  Of course there's other work on the runtime needed to
actually do the parallel invocations, but XMI serializer support is a
prerequisite.

The basic XMI deserializer changes aren't too complicated.  First,
when the CAS is originally serialized in step 1, we make available to
the caller the maximum xmi:id value in that CAS (this is mostly
already done, we just need to add a public accessor for this value).
Next, we allow passing this value to the deserializer as an optional
argument that essentially says "deserialize only XMI elements whose
xmi:id is greater than the specified value".

Any comments on this approach?

-Adam

Re: Enhancement to XMI deserializer as a foundation for remote parallel processing

[Michael Baessler <mb...@michael-baessler.de>]

Adam Lally wrote:
> On 2/22/07, Michael Baessler <mb...@michael-baessler.de> wrote:
>> Currently don't understand how the xmi:id is generated. For example we
>> have a CAS in step 1 that is serialized with a max xmi:id of 100.
>> Now two other remote services get this XMI document and do their
>> processing on it and serialize the CAS again. First service adds for
>> example FS with ids from 101 to 150, right?
>> And what do the second process? I think it also starts with numbering at
>> 101.
>
> Yes, that's correct.  The IDs used by the two services for new FS they
> create will not be unique.
>
>> So how are the XMI documents merged in the XMI deserializer. Is it
>> necessary
>> so merge the xmi:id attributes before creating the CAS or is it
>> sufficient to just read the additional xmi:ids (greater than 100) and
>> add them to the CAS?
>>
>
> Good questions...
>
> It actually should work without having to merge (if you mean to make
> unique) the IDs produced by the services.  Take your example, when the
> original max ID is 100 (I call 100 the "merge point"), and the two
> responses "xmi1" and "xmi2" both have appended FS with xmi:id=101:
>
> The deserialization of xmi1 works as normal - all FS including the one
> with ID 101 are added to the CAS.  When the deserialization of xmi2 is
> done, the deserializer code is told that the "merge point" is 100.
> This effect of this is that all of the FS with xmi:id <= 100 are
> ignored.  The FS with xmi:id = 101 will be added to the CAS as a new
> FeatureStruture.  It doesn't matter that xmi1 also had an FS with the
> same id - the deserializer will know that because 101 is greater than
> the mergePoint, a new FS should be created.
>
> As for references between FS.. during the deserialization of xmi2, the
> deserializer knows that any reference to an xmi:id with value <= 100
> is a reference to an FS that pre-existed in the CAS, while any
> reference to an xmi:id with value > 100 is a reference to an FS that
> is part of the xmi2 document that's currently being deserialized.  So
> all the information is there to do this correctly.  If xmi2 contained
> a reference to id 101, the deserializer would know that this was
> supposed to refer to id 101 in xmi2, NOT in xmi1.
>
>
>> I think when xmi:ids must be merged before the CAS is created the XMI
>> deserializer have to take care about the references with special offset
>> values. That is easy when using
>>
>> <myproj:Baz xmi:id="2">
>>    <myFoo href="#1"/>
>> <myproj.Baz>
>>
>> but more difficult when using
>>
>> <myproj:Baz xmi:id="2" myFoo="1"/>
>>
>
> The deserializer knows the TypeSystem of the CAS, so will know that
> myFoo is defined as a reference and not an integer.  This is in fact
> needed for "normal" deserialization to work, even without merging.
Thanks!

So the xmi:ids that are not unique must not be merged they can just be 
added to the CAS one XMI document after the other.

-- Michael

Re: Enhancement to XMI deserializer as a foundation for remote parallel processing

[Adam Lally <al...@alum.rpi.edu>]

On 2/22/07, Michael Baessler <mb...@michael-baessler.de> wrote:
> Currently don't understand how the xmi:id is generated. For example we
> have a CAS in step 1 that is serialized with a max xmi:id of 100.
> Now two other remote services get this XMI document and do their
> processing on it and serialize the CAS again. First service adds for
> example FS with ids from 101 to 150, right?
> And what do the second process? I think it also starts with numbering at
> 101.

Yes, that's correct.  The IDs used by the two services for new FS they
create will not be unique.

> So how are the XMI documents merged in the XMI deserializer. Is it
> necessary
> so merge the xmi:id attributes before creating the CAS or is it
> sufficient to just read the additional xmi:ids (greater than 100) and
> add them to the CAS?
>

Good questions...

It actually should work without having to merge (if you mean to make
unique) the IDs produced by the services.  Take your example, when the
original max ID is 100 (I call 100 the "merge point"), and the two
responses "xmi1" and "xmi2" both have appended FS with xmi:id=101:

The deserialization of xmi1 works as normal - all FS including the one
with ID 101 are added to the CAS.  When the deserialization of xmi2 is
done, the deserializer code is told that the "merge point" is 100.
This effect of this is that all of the FS with xmi:id <= 100 are
ignored.  The FS with xmi:id = 101 will be added to the CAS as a new
FeatureStruture.  It doesn't matter that xmi1 also had an FS with the
same id - the deserializer will know that because 101 is greater than
the mergePoint, a new FS should be created.

As for references between FS.. during the deserialization of xmi2, the
deserializer knows that any reference to an xmi:id with value <= 100
is a reference to an FS that pre-existed in the CAS, while any
reference to an xmi:id with value > 100 is a reference to an FS that
is part of the xmi2 document that's currently being deserialized.  So
all the information is there to do this correctly.  If xmi2 contained
a reference to id 101, the deserializer would know that this was
supposed to refer to id 101 in xmi2, NOT in xmi1.


> I think when xmi:ids must be merged before the CAS is created the XMI
> deserializer have to take care about the references with special offset
> values. That is easy when using
>
> <myproj:Baz xmi:id="2">
>    <myFoo href="#1"/>
> <myproj.Baz>
>
> but more difficult when using
>
> <myproj:Baz xmi:id="2" myFoo="1"/>
>

The deserializer knows the TypeSystem of the CAS, so will know that
myFoo is defined as a reference and not an integer.  This is in fact
needed for "normal" deserialization to work, even without merging.

-Adam

Re: Enhancement to XMI deserializer as a foundation for remote parallel processing

[Michael Baessler <mb...@michael-baessler.de>]

Adam Lally wrote:
> I am working on some modifications to the XMI deserializer that will
> allow the following scenario:
>
> 1) A CAS is serialized to XMI.
> 2) Copies of the XMI documents are sent to multiple remote services
> 3) Each remote service appends to the CAS (does not delete or modify
> existing stuff) and responds with a new serialized XMI CAS)
> 4) The multiple XMI responses are all merged back into a single CAS 
> instance
>
> This would permit multiple remote services that don't depend on each
> other to run in parallel, assuming they only append to the CAS (which
> is common).  Of course there's other work on the runtime needed to
> actually do the parallel invocations, but XMI serializer support is a
> prerequisite.
>
> The basic XMI deserializer changes aren't too complicated.  First,
> when the CAS is originally serialized in step 1, we make available to
> the caller the maximum xmi:id value in that CAS (this is mostly
> already done, we just need to add a public accessor for this value).
> Next, we allow passing this value to the deserializer as an optional
> argument that essentially says "deserialize only XMI elements whose
> xmi:id is greater than the specified value".
>
> Any comments on this approach?
Currently don't understand how the xmi:id is generated. For example we 
have a CAS in step 1 that is serialized with a max xmi:id of 100.
Now two other remote services get this XMI document and do their 
processing on it and serialize the CAS again. First service adds for 
example FS with ids from 101 to 150, right?
And what do the second process? I think it also starts with numbering at 
101. So how are the XMI documents merged in the XMI deserializer. Is it 
necessary
so merge the xmi:id attributes before creating the CAS or is it 
sufficient to just read the additional xmi:ids (greater than 100) and 
add them to the CAS?

I think when xmi:ids must be merged before the CAS is created the XMI 
deserializer have to take care about the references with special offset 
values. That is easy when using

<myproj:Baz xmi:id="2">
   <myFoo href="#1"/>
<myproj.Baz>

but more difficult when using

<myproj:Baz xmi:id="2" myFoo="1"/>


How will be merging be done?

-- Michael