You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@servicemix.apache.org by Daniel Kulp <dk...@apache.org> on 2010/05/07 22:01:35 UTC
CXF NMR transport ideas and enchancments...
I've started digging through the CXF NMR transport thing trying to understand
what it's doing and why and wanted to start a quick discussion to help me
understand things as well as propose some enhancements.
There are definitely some obvious deficiencies like not mapping any sort of
protocol headers in either direction and such, but I'm not even at that level
yet. Thing like attachments (for MTOM) are also obviously an issue that will
need to be looked at.
On the incoming side, it seems to grab a Source object from the NM, then
always uses a transformer to make a byte[] for it, then create an InputStream
which is fed into CXF. This seems INCREDIBLY inefficient and is one of the
first things I'd like to change. With some changes I've done for CXF 2.2.8,
I can not take the Source and create an XMLStreamReader from it
(StaxUtils.createXMLStreamReader(source)) and pass that directly into CXF.
This avoid all the funky transformations, byte[] things, etc... All the
"stream" level interceptors that CXF puts on the chain will not see an
inputstream and will just skip themselves.
Pros: much faster, much lower memory, seems more "natural", not subject to
XSL transformer configuration issues.
Cons: stream level interceptors won't work. The main one here would be the
Logging interceptors. Most of the interceptors at this level (like gzip and
fastinfoset and similar) really cannot apply for the NMR stuff anyway as that
functionality would need to be at the http (or other transport) component
level anyway. Thus, it's mostly just the Logging. Does anyone feel that is
really a big issue? We could have a special "XMLLoggingInInterceptor" that
would operate at the XMLStreamReader level if we really need something to
work.
Extending this a bit further, we could also check if it's a StreamSource with
an InputStream and feed the InputStream in that case and the XMLStreamReader
in the other cases. That would allow the stream interceptors to work in
"some" cases. That might cause some confusion though.
On the outgoing side, it's also a bit strange. We write to an OutputStream,
but then we seem to parse it to a DOM so we can set a DOMSource into the NM.
Two questions:
1) Why not just write to a DOM directly from the CXF side (like set an
W3CDOMStreamWriter instead of an OutputStream) and let it write to that? This
would have the same logging/stream issue as on the in side.
2) Or, keep the OutputStream stuff, but set a StreamSource on the NM instead
of a DOMSource. Is there an issue with doing that?
Anyway, I'm starting to really look into this a bit and will probably start
filing issues (and patches) for the above.
Thoughts?
--
Daniel Kulp
dkulp@apache.org
http://dankulp.com/blog
Re: CXF NMR transport ideas and enchancments...
Posted by Guillaume Nodet <gn...@gmail.com>.
Right, I suppose most of the interceptors (at least) could be shared.
On Mon, May 17, 2010 at 15:52, Daniel Kulp <dk...@apache.org> wrote:
> On Monday 17 May 2010 8:44:04 am Guillaume Nodet wrote:
> > All those enhancements look good to me.
> > Something interesting would be to see if we can port those enhancements
> to
> > the JBI transport too.
> > The JBI one is still much more used.
>
> I guess my question is why have two with nearly identical goals and code?
> Is
> it possible to have just one with maybe a flag to turn on any JBI specific
> behavior? Either that or have the JBI one extend/subclass the non-jbi
> one?
> Maybe a shared lib they both use? It would be a lot easier to maintain
> if
> we just had to deal with a single place to deal with these types of
> enhancements.
>
>
> Dan
>
>
> > On Fri, May 7, 2010 at 10:01 PM, Daniel Kulp <dk...@apache.org> wrote:
> > > I've started digging through the CXF NMR transport thing trying to
> > > understand
> > > what it's doing and why and wanted to start a quick discussion to help
> me
> > > understand things as well as propose some enhancements.
> > >
> > > There are definitely some obvious deficiencies like not mapping any
> sort
> > > of protocol headers in either direction and such, but I'm not even at
> > > that level
> > > yet. Thing like attachments (for MTOM) are also obviously an issue that
> > > will
> > > need to be looked at.
> > >
> > > On the incoming side, it seems to grab a Source object from the NM,
> then
> > > always uses a transformer to make a byte[] for it, then create an
> > > InputStream
> > > which is fed into CXF. This seems INCREDIBLY inefficient and is one
> of
> > > the
> > > first things I'd like to change. With some changes I've done for CXF
> > > 2.2.8,
> > > I can not take the Source and create an XMLStreamReader from it
> > > (StaxUtils.createXMLStreamReader(source)) and pass that directly into
> > > CXF. This avoid all the funky transformations, byte[] things, etc...
> > > All the "stream" level interceptors that CXF puts on the chain will not
> > > see an inputstream and will just skip themselves.
> > >
> > > Pros: much faster, much lower memory, seems more "natural", not
> subject
> > > to XSL transformer configuration issues.
> > >
> > > Cons: stream level interceptors won't work. The main one here would
> be
> > > the
> > > Logging interceptors. Most of the interceptors at this level (like
> gzip
> > > and
> > > fastinfoset and similar) really cannot apply for the NMR stuff anyway
> as
> > > that
> > > functionality would need to be at the http (or other transport)
> component
> > > level anyway. Thus, it's mostly just the Logging. Does anyone feel
> > > that is
> > > really a big issue? We could have a special "XMLLoggingInInterceptor"
> > > that
> > > would operate at the XMLStreamReader level if we really need something
> to
> > > work.
> > >
> > > Extending this a bit further, we could also check if it's a
> StreamSource
> > > with
> > > an InputStream and feed the InputStream in that case and the
> > > XMLStreamReader
> > > in the other cases. That would allow the stream interceptors to work
> in
> > > "some" cases. That might cause some confusion though.
> > >
> > >
> > > On the outgoing side, it's also a bit strange. We write to an
> > > OutputStream,
> > > but then we seem to parse it to a DOM so we can set a DOMSource into
> the
> > > NM.
> > > Two questions:
> > > 1) Why not just write to a DOM directly from the CXF side (like set an
> > > W3CDOMStreamWriter instead of an OutputStream) and let it write to
> that?
> > >
> > > This
> > >
> > > would have the same logging/stream issue as on the in side.
> > >
> > > 2) Or, keep the OutputStream stuff, but set a StreamSource on the NM
> > > instead
> > > of a DOMSource. Is there an issue with doing that?
> > >
> > > Anyway, I'm starting to really look into this a bit and will probably
> > > start filing issues (and patches) for the above.
> > >
> > > Thoughts?
> > >
> > > --
> > > Daniel Kulp
> > > dkulp@apache.org
> > > http://dankulp.com/blog
>
> --
> Daniel Kulp
> dkulp@apache.org
> http://dankulp.com/blog
>
--
Cheers,
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
------------------------
Open Source SOA
http://fusesource.com
Re: CXF NMR transport ideas and enchancments...
Posted by Daniel Kulp <dk...@apache.org>.
On Monday 17 May 2010 8:44:04 am Guillaume Nodet wrote:
> All those enhancements look good to me.
> Something interesting would be to see if we can port those enhancements to
> the JBI transport too.
> The JBI one is still much more used.
I guess my question is why have two with nearly identical goals and code? Is
it possible to have just one with maybe a flag to turn on any JBI specific
behavior? Either that or have the JBI one extend/subclass the non-jbi one?
Maybe a shared lib they both use? It would be a lot easier to maintain if
we just had to deal with a single place to deal with these types of
enhancements.
Dan
> On Fri, May 7, 2010 at 10:01 PM, Daniel Kulp <dk...@apache.org> wrote:
> > I've started digging through the CXF NMR transport thing trying to
> > understand
> > what it's doing and why and wanted to start a quick discussion to help me
> > understand things as well as propose some enhancements.
> >
> > There are definitely some obvious deficiencies like not mapping any sort
> > of protocol headers in either direction and such, but I'm not even at
> > that level
> > yet. Thing like attachments (for MTOM) are also obviously an issue that
> > will
> > need to be looked at.
> >
> > On the incoming side, it seems to grab a Source object from the NM, then
> > always uses a transformer to make a byte[] for it, then create an
> > InputStream
> > which is fed into CXF. This seems INCREDIBLY inefficient and is one of
> > the
> > first things I'd like to change. With some changes I've done for CXF
> > 2.2.8,
> > I can not take the Source and create an XMLStreamReader from it
> > (StaxUtils.createXMLStreamReader(source)) and pass that directly into
> > CXF. This avoid all the funky transformations, byte[] things, etc...
> > All the "stream" level interceptors that CXF puts on the chain will not
> > see an inputstream and will just skip themselves.
> >
> > Pros: much faster, much lower memory, seems more "natural", not subject
> > to XSL transformer configuration issues.
> >
> > Cons: stream level interceptors won't work. The main one here would be
> > the
> > Logging interceptors. Most of the interceptors at this level (like gzip
> > and
> > fastinfoset and similar) really cannot apply for the NMR stuff anyway as
> > that
> > functionality would need to be at the http (or other transport) component
> > level anyway. Thus, it's mostly just the Logging. Does anyone feel
> > that is
> > really a big issue? We could have a special "XMLLoggingInInterceptor"
> > that
> > would operate at the XMLStreamReader level if we really need something to
> > work.
> >
> > Extending this a bit further, we could also check if it's a StreamSource
> > with
> > an InputStream and feed the InputStream in that case and the
> > XMLStreamReader
> > in the other cases. That would allow the stream interceptors to work in
> > "some" cases. That might cause some confusion though.
> >
> >
> > On the outgoing side, it's also a bit strange. We write to an
> > OutputStream,
> > but then we seem to parse it to a DOM so we can set a DOMSource into the
> > NM.
> > Two questions:
> > 1) Why not just write to a DOM directly from the CXF side (like set an
> > W3CDOMStreamWriter instead of an OutputStream) and let it write to that?
> >
> > This
> >
> > would have the same logging/stream issue as on the in side.
> >
> > 2) Or, keep the OutputStream stuff, but set a StreamSource on the NM
> > instead
> > of a DOMSource. Is there an issue with doing that?
> >
> > Anyway, I'm starting to really look into this a bit and will probably
> > start filing issues (and patches) for the above.
> >
> > Thoughts?
> >
> > --
> > Daniel Kulp
> > dkulp@apache.org
> > http://dankulp.com/blog
--
Daniel Kulp
dkulp@apache.org
http://dankulp.com/blog
Re: CXF NMR transport ideas and enchancments...
Posted by Guillaume Nodet <gn...@gmail.com>.
All those enhancements look good to me.
Something interesting would be to see if we can port those enhancements to
the JBI transport too.
The JBI one is still much more used.
On Fri, May 7, 2010 at 10:01 PM, Daniel Kulp <dk...@apache.org> wrote:
>
> I've started digging through the CXF NMR transport thing trying to
> understand
> what it's doing and why and wanted to start a quick discussion to help me
> understand things as well as propose some enhancements.
>
> There are definitely some obvious deficiencies like not mapping any sort of
> protocol headers in either direction and such, but I'm not even at that
> level
> yet. Thing like attachments (for MTOM) are also obviously an issue that
> will
> need to be looked at.
>
> On the incoming side, it seems to grab a Source object from the NM, then
> always uses a transformer to make a byte[] for it, then create an
> InputStream
> which is fed into CXF. This seems INCREDIBLY inefficient and is one of
> the
> first things I'd like to change. With some changes I've done for CXF
> 2.2.8,
> I can not take the Source and create an XMLStreamReader from it
> (StaxUtils.createXMLStreamReader(source)) and pass that directly into CXF.
> This avoid all the funky transformations, byte[] things, etc... All the
> "stream" level interceptors that CXF puts on the chain will not see an
> inputstream and will just skip themselves.
>
> Pros: much faster, much lower memory, seems more "natural", not subject to
> XSL transformer configuration issues.
>
> Cons: stream level interceptors won't work. The main one here would be
> the
> Logging interceptors. Most of the interceptors at this level (like gzip
> and
> fastinfoset and similar) really cannot apply for the NMR stuff anyway as
> that
> functionality would need to be at the http (or other transport) component
> level anyway. Thus, it's mostly just the Logging. Does anyone feel that
> is
> really a big issue? We could have a special "XMLLoggingInInterceptor"
> that
> would operate at the XMLStreamReader level if we really need something to
> work.
>
> Extending this a bit further, we could also check if it's a StreamSource
> with
> an InputStream and feed the InputStream in that case and the
> XMLStreamReader
> in the other cases. That would allow the stream interceptors to work in
> "some" cases. That might cause some confusion though.
>
>
> On the outgoing side, it's also a bit strange. We write to an
> OutputStream,
> but then we seem to parse it to a DOM so we can set a DOMSource into the
> NM.
> Two questions:
> 1) Why not just write to a DOM directly from the CXF side (like set an
> W3CDOMStreamWriter instead of an OutputStream) and let it write to that?
> This
> would have the same logging/stream issue as on the in side.
>
> 2) Or, keep the OutputStream stuff, but set a StreamSource on the NM
> instead
> of a DOMSource. Is there an issue with doing that?
>
> Anyway, I'm starting to really look into this a bit and will probably start
> filing issues (and patches) for the above.
>
> Thoughts?
>
> --
> Daniel Kulp
> dkulp@apache.org
> http://dankulp.com/blog
>
--
Cheers,
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
------------------------
Open Source SOA
http://fusesource.com
Re: CXF NMR transport ideas and enchancments...
Posted by Freeman Fang <fr...@gmail.com>.
On 2010-5-11, at 上午1:24, Daniel Kulp wrote:
> On Monday 10 May 2010 4:39:44 am Freeman Fang wrote:
>> I'm a little bit concern about the Logging Interceptors, as this is
>> popularly used interceptors, for backward compatibility，I think we
>> MAY need keep inputstream still working as before.
>> Is there a way to transform from Source to InputStream directly,
>> without using byte[] as bridge?
>
> If the Source is a StreamSource, we can grab the stream directly.
> If it's not
> a StreamSource, some level of transform would need to be done to get
> an
> InputStream. I would most likely use our CachedOutputStream to
> create the
> InputStream to avoid byte[] for larger messages, but it would still
> involve a
> lot more work than would be ideal.
>
> Another option MIGHT be to update the LoggingInInterceptor and if
> there isn't
> an InputStream, have it automatically add a new interceptor that
> lives after
> the StaxIn or similar that would wrapper the XMLStreamReader with
> one that
> would provide some level of logging of the XML stream. Thus, we
> could avoid
> the stream, yet still provide some logging.
+1 for this option, hide the tricky in LoggingInInterceptor, so that
LoggingInterceptor/Logging feature still work as before.
Freeman
>
>
>
>>> On the outgoing side, it's also a bit strange. We write to an
>>> OutputStream,
>>> but then we seem to parse it to a DOM so we can set a DOMSource into
>>> the NM.
>>> Two questions:
>>> 1) Why not just write to a DOM directly from the CXF side (like
>>> set an
>>> W3CDOMStreamWriter instead of an OutputStream) and let it write to
>>> that? This
>>> would have the same logging/stream issue as on the in side.
>>>
>>> 2) Or, keep the OutputStream stuff, but set a StreamSource on the NM
>>> instead
>>> of a DOMSource. Is there an issue with doing that?
>>
>> +1 for keeping the OutputStream stuff and using StreamSource, I can't
>> see any issue with StreamSource.
>
> OK. I'll play around with this a bit.
>
>
> --
> Daniel Kulp
> dkulp@apache.org
> http://dankulp.com/blog
--
Freeman Fang
------------------------
Open Source SOA: http://fusesource.com
Re: CXF NMR transport ideas and enchancments...
Posted by Daniel Kulp <dk...@apache.org>.
On Monday 10 May 2010 4:39:44 am Freeman Fang wrote:
> I'm a little bit concern about the Logging Interceptors, as this is
> popularly used interceptors, for backward compatibility，I think we
> MAY need keep inputstream still working as before.
> Is there a way to transform from Source to InputStream directly,
> without using byte[] as bridge?
If the Source is a StreamSource, we can grab the stream directly. If it's not
a StreamSource, some level of transform would need to be done to get an
InputStream. I would most likely use our CachedOutputStream to create the
InputStream to avoid byte[] for larger messages, but it would still involve a
lot more work than would be ideal.
Another option MIGHT be to update the LoggingInInterceptor and if there isn't
an InputStream, have it automatically add a new interceptor that lives after
the StaxIn or similar that would wrapper the XMLStreamReader with one that
would provide some level of logging of the XML stream. Thus, we could avoid
the stream, yet still provide some logging.
> > On the outgoing side, it's also a bit strange. We write to an
> > OutputStream,
> > but then we seem to parse it to a DOM so we can set a DOMSource into
> > the NM.
> > Two questions:
> > 1) Why not just write to a DOM directly from the CXF side (like set an
> > W3CDOMStreamWriter instead of an OutputStream) and let it write to
> > that? This
> > would have the same logging/stream issue as on the in side.
> >
> > 2) Or, keep the OutputStream stuff, but set a StreamSource on the NM
> > instead
> > of a DOMSource. Is there an issue with doing that?
>
> +1 for keeping the OutputStream stuff and using StreamSource, I can't
> see any issue with StreamSource.
OK. I'll play around with this a bit.
--
Daniel Kulp
dkulp@apache.org
http://dankulp.com/blog
Re: CXF NMR transport ideas and enchancments...
Posted by Freeman Fang <fr...@gmail.com>.
Hi Dan,
Thanks for the feedback and my comment inline.
On 2010-5-8, at 上午4:01, Daniel Kulp wrote:
>
> I've started digging through the CXF NMR transport thing trying to
> understand
> what it's doing and why and wanted to start a quick discussion to
> help me
> understand things as well as propose some enhancements.
>
> There are definitely some obvious deficiencies like not mapping any
> sort of
> protocol headers in either direction and such, but I'm not even at
> that level
> yet. Thing like attachments (for MTOM) are also obviously an issue
> that will
> need to be looked at.
>
Yeah, we've fixed similar issue(to copy over property headers and
attachment between cxf message and JBI message) in JBI transport, and
we also need do same thing for NMR transport. Create SMX4-527[1] to
track this issue.
[1]https://issues.apache.org/activemq/browse/SMX4-527
> On the incoming side, it seems to grab a Source object from the NM,
> then
> always uses a transformer to make a byte[] for it, then create an
> InputStream
> which is fed into CXF. This seems INCREDIBLY inefficient and is
> one of the
> first things I'd like to change. With some changes I've done for
> CXF 2.2.8,
> I can not take the Source and create an XMLStreamReader from it
> (StaxUtils.createXMLStreamReader(source)) and pass that directly
> into CXF.
> This avoid all the funky transformations, byte[] things, etc...
> All the
> "stream" level interceptors that CXF puts on the chain will not see an
> inputstream and will just skip themselves.
>
Is this new functionality for CXF 2.2.8, as CXF 2.2.8 is in VOTE and
not officially out yet, we can refactor NMR transport afterwards.
Agree we should avoid byte[] transform in any case.
> Pros: much faster, much lower memory, seems more "natural", not
> subject to
> XSL transformer configuration issues.
>
> Cons: stream level interceptors won't work. The main one here
> would be the
> Logging interceptors. Most of the interceptors at this level (like
> gzip and
> fastinfoset and similar) really cannot apply for the NMR stuff
> anyway as that
> functionality would need to be at the http (or other transport)
> component
> level anyway. Thus, it's mostly just the Logging. Does anyone
> feel that is
> really a big issue? We could have a special
> "XMLLoggingInInterceptor" that
> would operate at the XMLStreamReader level if we really need
> something to
> work.
>
> Extending this a bit further, we could also check if it's a
> StreamSource with
> an InputStream and feed the InputStream in that case and the
> XMLStreamReader
> in the other cases. That would allow the stream interceptors to
> work in
> "some" cases. That might cause some confusion though.
>
I'm a little bit concern about the Logging Interceptors, as this is
popularly used interceptors, for backward compatibility，I think we
MAY need keep inputstream still working as before.
Is there a way to transform from Source to InputStream directly,
without using byte[] as bridge? If we can transform from Source to
InputStream directly and then pass this InputStream to CXF, IMHO we
can say we resolve the performance problem.
>
> On the outgoing side, it's also a bit strange. We write to an
> OutputStream,
> but then we seem to parse it to a DOM so we can set a DOMSource into
> the NM.
> Two questions:
> 1) Why not just write to a DOM directly from the CXF side (like set an
> W3CDOMStreamWriter instead of an OutputStream) and let it write to
> that? This
> would have the same logging/stream issue as on the in side.
>
> 2) Or, keep the OutputStream stuff, but set a StreamSource on the NM
> instead
> of a DOMSource. Is there an issue with doing that?
>
+1 for keeping the OutputStream stuff and using StreamSource, I can't
see any issue with StreamSource.
> Anyway, I'm starting to really look into this a bit and will
> probably start
> filing issues (and patches) for the above.
>
> Thoughts?
Welcome and thanks for any contribution :-)
Freeman
>
> --
> Daniel Kulp
> dkulp@apache.org
> http://dankulp.com/blog
--
Freeman Fang
------------------------
Open Source SOA: http://fusesource.com
Re: CXF NMR transport ideas and enchancments...
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Dan,
Thanks for this first feedback.
Concerning the default behavior using InputStream, it looks like a big issue around performances and memory consumption.
I think that Freeman can give us more feedback around it.
Defintely we should have an improved behavior which is able to choose between the usage of a InputStream or a XMLStreamReader. So +1 regarding your advice.
I think that the best way is make a kind of audit on CXF (features and components) to identify the leaks or improvements and raise the corresponding Jira tasks.
I will have time in the following weeks if you need.
Regards
JB
-----Original Message-----
From: Daniel Kulp <dk...@apache.org>
Date: Fri, 7 May 2010 16:01:35
To: <de...@servicemix.apache.org>
Subject: CXF NMR transport ideas and enchancments...
I've started digging through the CXF NMR transport thing trying to understand
what it's doing and why and wanted to start a quick discussion to help me
understand things as well as propose some enhancements.
There are definitely some obvious deficiencies like not mapping any sort of
protocol headers in either direction and such, but I'm not even at that level
yet. Thing like attachments (for MTOM) are also obviously an issue that will
need to be looked at.
On the incoming side, it seems to grab a Source object from the NM, then
always uses a transformer to make a byte[] for it, then create an InputStream
which is fed into CXF. This seems INCREDIBLY inefficient and is one of the
first things I'd like to change. With some changes I've done for CXF 2.2.8,
I can not take the Source and create an XMLStreamReader from it
(StaxUtils.createXMLStreamReader(source)) and pass that directly into CXF.
This avoid all the funky transformations, byte[] things, etc... All the
"stream" level interceptors that CXF puts on the chain will not see an
inputstream and will just skip themselves.
Pros: much faster, much lower memory, seems more "natural", not subject to
XSL transformer configuration issues.
Cons: stream level interceptors won't work. The main one here would be the
Logging interceptors. Most of the interceptors at this level (like gzip and
fastinfoset and similar) really cannot apply for the NMR stuff anyway as that
functionality would need to be at the http (or other transport) component
level anyway. Thus, it's mostly just the Logging. Does anyone feel that is
really a big issue? We could have a special "XMLLoggingInInterceptor" that
would operate at the XMLStreamReader level if we really need something to
work.
Extending this a bit further, we could also check if it's a StreamSource with
an InputStream and feed the InputStream in that case and the XMLStreamReader
in the other cases. That would allow the stream interceptors to work in
"some" cases. That might cause some confusion though.
On the outgoing side, it's also a bit strange. We write to an OutputStream,
but then we seem to parse it to a DOM so we can set a DOMSource into the NM.
Two questions:
1) Why not just write to a DOM directly from the CXF side (like set an
W3CDOMStreamWriter instead of an OutputStream) and let it write to that? This
would have the same logging/stream issue as on the in side.
2) Or, keep the OutputStream stuff, but set a StreamSource on the NM instead
of a DOMSource. Is there an issue with doing that?
Anyway, I'm starting to really look into this a bit and will probably start
filing issues (and patches) for the above.
Thoughts?
--
Daniel Kulp
dkulp@apache.org
http://dankulp.com/blog