You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tuscany.apache.org by Yang ZHONG <le...@gmail.com> on 2006/12/09 20:58:16 UTC
[C++] Duplicated/infinite loading? How to avoid
While working on http://issues.apache.org/jira/browse/TUSCANY-907
I've observed duplicated XSD loading. It could be infinite loading. I might
have a way to avoid that.
Could you verify current implementation may load redundantly even
infinitely?
Could you brainstorm how to avoid that?
http://pecl.php.net/bugs/bug.php?id=9243
has the Test Case:
1. http://ping.chip.org/phr/xml/insurance.xsd imports
http://ping.chip.org/phr/xml/types.xsd
2. http://ping.chip.org/phr/xml/insurance.xsd imports
http://ping.chip.org/phr/xml/contact.xsd which also imports
http://ping.chip.org/phr/xml/types.xsd
I've observed http://ping.chip.org/phr/xml/types.xsd is loaded/parsed
multiple times during loading http://ping.chip.org/phr/xml/insurance.xsd
That impacts performance, especially types.xsd is remote.
More deadly, that may cause infinite loading or stack overflow.
Imagine A imports B which imports A, or X imports Y which imports Z which
imports X.
If above is true, let's brainstorm how to avoid that.
We can have a LoadingContext/ResourceSet to cache loaded/parsed resources,
and reuse parsing results if a location/URI has been loaded/parsed already.
3-1. The LoadingContext/ResourceSet can be a Thread Local
3-2. The LoadingContext/ResourceSet can be explicitly passed into each
parser
3-3. Currently ParserErrorSetter is explicitly passed into each parser,
which can host the LoadingContext/ResourceSet
Do you have alternatives? Which would you prefer?
--
Yang ZHONG
Re: [C++] Duplicated/infinite loading? How to avoid
Posted by Pete Robbins <ro...@googlemail.com>.
On 13/12/06, Pete Robbins <ro...@googlemail.com> wrote:
>
> The patch works for me (or at least doesn't break anything ;-) ).
>
> good work
>
I'm getting an access violation in this code when running hte SCA samples. I
need to investigate some more.
> On 13/12/06, Yang ZHONG <le...@gmail.com> wrote:
> >
> > A patch has been attached into
> > http://issues.apache.org/jira/browse/TUSCANY-990
> >
> > BTW, Caroline has also requested this improvement (
> > http://www.mail-archive.com/tuscany-dev%40ws.apache.org/msg11839.html)
> >
> > On 12/10/06, Pete Robbins <ro...@googlemail.com> wrote:
> > >
> > > On 09/12/06, Yang ZHONG <leiwang.yangzhong@gmail.com > wrote:
> > > >
> > > > Thank you Pete for the alternative to host
> > LoadingContext/ResourceSet
> > > > within
> > > > DataFactory.
> > > > How to browse from SDOSchemaSAX2Parser to DataFactory please?
> > > > My prototype checks LoadingContext/ResourceSet
> > > > within SDOSchemaSAX2Parser::startInclude
> > > >
> > > > My LoadingContext/ResourceSet is internal to parsers, not a user
> > > context.
> > >
> > >
> > > I wasn't thinking clearly ;-) It's SDOSchemaSAX2Parser that needs to
> > > remember what it has loaded. Your suggsetion sounds good. Could you
> > post a
> > > patch so we can see your idea?
> > >
> > >
> > >
> > > On 12/9/06, Pete Robbins <ro...@googlemail.com> wrote:
> > > > >
> > > > > Yang,
> > > > > Yes we've noticed this when loading the Atom schema.
> > > > >
> > > > >
> > > > > On 09/12/06, Yang ZHONG <le...@gmail.com> wrote:
> > > > > >
> > > > > > While working on
> > http://issues.apache.org/jira/browse/TUSCANY-907
> > > > > > I've observed duplicated XSD loading. It could be infinite
> > loading.
> > > I
> > > > > > might
> > > > > > have a way to avoid that.
> > > > > >
> > > > > > Could you verify current implementation may load redundantly
> > even
> > > > > > infinitely?
> > > > > > Could you brainstorm how to avoid that?
> > > > > >
> > > > > > http://pecl.php.net/bugs/bug.php?id=9243
> > > > > > has the Test Case:
> > > > > > 1. http://ping.chip.org/phr/xml/insurance.xsd imports
> > > > > > http://ping.chip.org/phr/xml/types.xsd
> > > > > > 2. http://ping.chip.org/phr/xml/insurance.xsd imports
> > > > > > http://ping.chip.org/phr/xml/contact.xsd which also imports
> > > > > > http://ping.chip.org/phr/xml/types.xsd
> > > > > >
> > > > > > I've observed http://ping.chip.org/phr/xml/types.xsd is
> > > loaded/parsed
> > > > > > multiple times during loading
> > > > http://ping.chip.org/phr/xml/insurance.xsd
> > > > > >
> > > > > > That impacts performance, especially types.xsd is remote.
> > > > > >
> > > > > > More deadly, that may cause infinite loading or stack overflow.
> > > > > > Imagine A imports B which imports A, or X imports Y which
> > imports Z
> > > > > which
> > > > > > imports X.
> > > > > >
> > > > > > If above is true, let's brainstorm how to avoid that.
> > > > > >
> > > > > > We can have a LoadingContext/ResourceSet to cache loaded/parsed
> > > > > resources,
> > > > > > and reuse parsing results if a location/URI has been
> > loaded/parsed
> > > > > > already.
> > > > > >
> > > > > > 3-1. The LoadingContext/ResourceSet can be a Thread Local
> > > > > > 3-2. The LoadingContext/ResourceSet can be explicitly passed
> > into
> > > each
> > > > > > parser
> > > > > > 3-3. Currently ParserErrorSetter is explicitly passed into each
> > > > parser,
> > > > > > which can host the LoadingContext/ResourceSet
> > > > > >
> > > > > > Do you have alternatives? Which would you prefer?
> > > > >
> > > > >
> > > > > The most basic problem is when a user uses XSDHelper to load
> > X.xsdwhich
> > > > > imports other schema... which in turn include some of the same
> > schema.
> > > > At
> > > > > the moment his causes redundant parsing. A simple solution is to
> > > > remember
> > > > > the schema namespace/location so the import/include of a schema we
> > > > already
> > > > > have parsed can be ignored. This information could be held in the
> > > > > DataFactory so that if the user calls XSDHelper::define() for the
> > same
> > > > > schema we could ignore it (as we know the types are already
> > defined).
> > > > >
> > > > > I'm not sure if you are suggesting a wider ranging improvement of
> > > some
> > > > > context the user can pass into XSDHelper?
> > > > >
> > > > > Cheers,
> > > > > --
> > > > > Pete
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Yang ZHONG
> > > >
> > > >
> > >
> > >
> > > --
> > > Pete
> > >
> > >
> >
> >
> > --
> >
> > Yang ZHONG
> >
> >
>
>
> --
> Pete
--
Pete
Re: [C++] Duplicated/infinite loading? How to avoid
Posted by Pete Robbins <ro...@googlemail.com>.
The patch works for me (or at least doesn't break anything ;-) ).
good work
On 13/12/06, Yang ZHONG <le...@gmail.com> wrote:
>
> A patch has been attached into
> http://issues.apache.org/jira/browse/TUSCANY-990
>
> BTW, Caroline has also requested this improvement (
> http://www.mail-archive.com/tuscany-dev%40ws.apache.org/msg11839.html)
>
> On 12/10/06, Pete Robbins <ro...@googlemail.com> wrote:
> >
> > On 09/12/06, Yang ZHONG <le...@gmail.com> wrote:
> > >
> > > Thank you Pete for the alternative to host LoadingContext/ResourceSet
> > > within
> > > DataFactory.
> > > How to browse from SDOSchemaSAX2Parser to DataFactory please?
> > > My prototype checks LoadingContext/ResourceSet
> > > within SDOSchemaSAX2Parser::startInclude
> > >
> > > My LoadingContext/ResourceSet is internal to parsers, not a user
> > context.
> >
> >
> > I wasn't thinking clearly ;-) It's SDOSchemaSAX2Parser that needs to
> > remember what it has loaded. Your suggsetion sounds good. Could you post
> a
> > patch so we can see your idea?
> >
> >
> >
> > On 12/9/06, Pete Robbins <ro...@googlemail.com> wrote:
> > > >
> > > > Yang,
> > > > Yes we've noticed this when loading the Atom schema.
> > > >
> > > >
> > > > On 09/12/06, Yang ZHONG <le...@gmail.com> wrote:
> > > > >
> > > > > While working on http://issues.apache.org/jira/browse/TUSCANY-907
> > > > > I've observed duplicated XSD loading. It could be infinite
> loading.
> > I
> > > > > might
> > > > > have a way to avoid that.
> > > > >
> > > > > Could you verify current implementation may load redundantly even
> > > > > infinitely?
> > > > > Could you brainstorm how to avoid that?
> > > > >
> > > > > http://pecl.php.net/bugs/bug.php?id=9243
> > > > > has the Test Case:
> > > > > 1. http://ping.chip.org/phr/xml/insurance.xsd imports
> > > > > http://ping.chip.org/phr/xml/types.xsd
> > > > > 2. http://ping.chip.org/phr/xml/insurance.xsd imports
> > > > > http://ping.chip.org/phr/xml/contact.xsd which also imports
> > > > > http://ping.chip.org/phr/xml/types.xsd
> > > > >
> > > > > I've observed http://ping.chip.org/phr/xml/types.xsd is
> > loaded/parsed
> > > > > multiple times during loading
> > > http://ping.chip.org/phr/xml/insurance.xsd
> > > > >
> > > > > That impacts performance, especially types.xsd is remote.
> > > > >
> > > > > More deadly, that may cause infinite loading or stack overflow.
> > > > > Imagine A imports B which imports A, or X imports Y which imports
> Z
> > > > which
> > > > > imports X.
> > > > >
> > > > > If above is true, let's brainstorm how to avoid that.
> > > > >
> > > > > We can have a LoadingContext/ResourceSet to cache loaded/parsed
> > > > resources,
> > > > > and reuse parsing results if a location/URI has been loaded/parsed
> > > > > already.
> > > > >
> > > > > 3-1. The LoadingContext/ResourceSet can be a Thread Local
> > > > > 3-2. The LoadingContext/ResourceSet can be explicitly passed into
> > each
> > > > > parser
> > > > > 3-3. Currently ParserErrorSetter is explicitly passed into each
> > > parser,
> > > > > which can host the LoadingContext/ResourceSet
> > > > >
> > > > > Do you have alternatives? Which would you prefer?
> > > >
> > > >
> > > > The most basic problem is when a user uses XSDHelper to load
> X.xsdwhich
> > > > imports other schema... which in turn include some of the same
> schema.
> > > At
> > > > the moment his causes redundant parsing. A simple solution is to
> > > remember
> > > > the schema namespace/location so the import/include of a schema we
> > > already
> > > > have parsed can be ignored. This information could be held in the
> > > > DataFactory so that if the user calls XSDHelper::define() for the
> same
> > > > schema we could ignore it (as we know the types are already
> defined).
> > > >
> > > > I'm not sure if you are suggesting a wider ranging improvement of
> > some
> > > > context the user can pass into XSDHelper?
> > > >
> > > > Cheers,
> > > > --
> > > > Pete
> > > >
> > > >
> > >
> > >
> > > --
> > >
> > > Yang ZHONG
> > >
> > >
> >
> >
> > --
> > Pete
> >
> >
>
>
> --
>
> Yang ZHONG
>
>
--
Pete
Re: [C++] Duplicated/infinite loading? How to avoid
Posted by Yang ZHONG <le...@gmail.com>.
A patch has been attached into
http://issues.apache.org/jira/browse/TUSCANY-990
BTW, Caroline has also requested this improvement (
http://www.mail-archive.com/tuscany-dev%40ws.apache.org/msg11839.html)
On 12/10/06, Pete Robbins <ro...@googlemail.com> wrote:
>
> On 09/12/06, Yang ZHONG <le...@gmail.com> wrote:
> >
> > Thank you Pete for the alternative to host LoadingContext/ResourceSet
> > within
> > DataFactory.
> > How to browse from SDOSchemaSAX2Parser to DataFactory please?
> > My prototype checks LoadingContext/ResourceSet
> > within SDOSchemaSAX2Parser::startInclude
> >
> > My LoadingContext/ResourceSet is internal to parsers, not a user
> context.
>
>
> I wasn't thinking clearly ;-) It's SDOSchemaSAX2Parser that needs to
> remember what it has loaded. Your suggsetion sounds good. Could you post a
> patch so we can see your idea?
>
>
>
> On 12/9/06, Pete Robbins <ro...@googlemail.com> wrote:
> > >
> > > Yang,
> > > Yes we've noticed this when loading the Atom schema.
> > >
> > >
> > > On 09/12/06, Yang ZHONG <le...@gmail.com> wrote:
> > > >
> > > > While working on http://issues.apache.org/jira/browse/TUSCANY-907
> > > > I've observed duplicated XSD loading. It could be infinite loading.
> I
> > > > might
> > > > have a way to avoid that.
> > > >
> > > > Could you verify current implementation may load redundantly even
> > > > infinitely?
> > > > Could you brainstorm how to avoid that?
> > > >
> > > > http://pecl.php.net/bugs/bug.php?id=9243
> > > > has the Test Case:
> > > > 1. http://ping.chip.org/phr/xml/insurance.xsd imports
> > > > http://ping.chip.org/phr/xml/types.xsd
> > > > 2. http://ping.chip.org/phr/xml/insurance.xsd imports
> > > > http://ping.chip.org/phr/xml/contact.xsd which also imports
> > > > http://ping.chip.org/phr/xml/types.xsd
> > > >
> > > > I've observed http://ping.chip.org/phr/xml/types.xsd is
> loaded/parsed
> > > > multiple times during loading
> > http://ping.chip.org/phr/xml/insurance.xsd
> > > >
> > > > That impacts performance, especially types.xsd is remote.
> > > >
> > > > More deadly, that may cause infinite loading or stack overflow.
> > > > Imagine A imports B which imports A, or X imports Y which imports Z
> > > which
> > > > imports X.
> > > >
> > > > If above is true, let's brainstorm how to avoid that.
> > > >
> > > > We can have a LoadingContext/ResourceSet to cache loaded/parsed
> > > resources,
> > > > and reuse parsing results if a location/URI has been loaded/parsed
> > > > already.
> > > >
> > > > 3-1. The LoadingContext/ResourceSet can be a Thread Local
> > > > 3-2. The LoadingContext/ResourceSet can be explicitly passed into
> each
> > > > parser
> > > > 3-3. Currently ParserErrorSetter is explicitly passed into each
> > parser,
> > > > which can host the LoadingContext/ResourceSet
> > > >
> > > > Do you have alternatives? Which would you prefer?
> > >
> > >
> > > The most basic problem is when a user uses XSDHelper to load X.xsdwhich
> > > imports other schema... which in turn include some of the same schema.
> > At
> > > the moment his causes redundant parsing. A simple solution is to
> > remember
> > > the schema namespace/location so the import/include of a schema we
> > already
> > > have parsed can be ignored. This information could be held in the
> > > DataFactory so that if the user calls XSDHelper::define() for the same
> > > schema we could ignore it (as we know the types are already defined).
> > >
> > > I'm not sure if you are suggesting a wider ranging improvement of
> some
> > > context the user can pass into XSDHelper?
> > >
> > > Cheers,
> > > --
> > > Pete
> > >
> > >
> >
> >
> > --
> >
> > Yang ZHONG
> >
> >
>
>
> --
> Pete
>
>
--
Yang ZHONG
Re: [C++] Duplicated/infinite loading? How to avoid
Posted by Pete Robbins <ro...@googlemail.com>.
On 09/12/06, Yang ZHONG <le...@gmail.com> wrote:
>
> Thank you Pete for the alternative to host LoadingContext/ResourceSet
> within
> DataFactory.
> How to browse from SDOSchemaSAX2Parser to DataFactory please?
> My prototype checks LoadingContext/ResourceSet
> within SDOSchemaSAX2Parser::startInclude
>
> My LoadingContext/ResourceSet is internal to parsers, not a user context.
I wasn't thinking clearly ;-) It's SDOSchemaSAX2Parser that needs to
remember what it has loaded. Your suggsetion sounds good. Could you post a
patch so we can see your idea?
On 12/9/06, Pete Robbins <ro...@googlemail.com> wrote:
> >
> > Yang,
> > Yes we've noticed this when loading the Atom schema.
> >
> >
> > On 09/12/06, Yang ZHONG <le...@gmail.com> wrote:
> > >
> > > While working on http://issues.apache.org/jira/browse/TUSCANY-907
> > > I've observed duplicated XSD loading. It could be infinite loading. I
> > > might
> > > have a way to avoid that.
> > >
> > > Could you verify current implementation may load redundantly even
> > > infinitely?
> > > Could you brainstorm how to avoid that?
> > >
> > > http://pecl.php.net/bugs/bug.php?id=9243
> > > has the Test Case:
> > > 1. http://ping.chip.org/phr/xml/insurance.xsd imports
> > > http://ping.chip.org/phr/xml/types.xsd
> > > 2. http://ping.chip.org/phr/xml/insurance.xsd imports
> > > http://ping.chip.org/phr/xml/contact.xsd which also imports
> > > http://ping.chip.org/phr/xml/types.xsd
> > >
> > > I've observed http://ping.chip.org/phr/xml/types.xsd is loaded/parsed
> > > multiple times during loading
> http://ping.chip.org/phr/xml/insurance.xsd
> > >
> > > That impacts performance, especially types.xsd is remote.
> > >
> > > More deadly, that may cause infinite loading or stack overflow.
> > > Imagine A imports B which imports A, or X imports Y which imports Z
> > which
> > > imports X.
> > >
> > > If above is true, let's brainstorm how to avoid that.
> > >
> > > We can have a LoadingContext/ResourceSet to cache loaded/parsed
> > resources,
> > > and reuse parsing results if a location/URI has been loaded/parsed
> > > already.
> > >
> > > 3-1. The LoadingContext/ResourceSet can be a Thread Local
> > > 3-2. The LoadingContext/ResourceSet can be explicitly passed into each
> > > parser
> > > 3-3. Currently ParserErrorSetter is explicitly passed into each
> parser,
> > > which can host the LoadingContext/ResourceSet
> > >
> > > Do you have alternatives? Which would you prefer?
> >
> >
> > The most basic problem is when a user uses XSDHelper to load X.xsd which
> > imports other schema... which in turn include some of the same schema.
> At
> > the moment his causes redundant parsing. A simple solution is to
> remember
> > the schema namespace/location so the import/include of a schema we
> already
> > have parsed can be ignored. This information could be held in the
> > DataFactory so that if the user calls XSDHelper::define() for the same
> > schema we could ignore it (as we know the types are already defined).
> >
> > I'm not sure if you are suggesting a wider ranging improvement of some
> > context the user can pass into XSDHelper?
> >
> > Cheers,
> > --
> > Pete
> >
> >
>
>
> --
>
> Yang ZHONG
>
>
--
Pete
Re: [C++] Duplicated/infinite loading? How to avoid
Posted by Yang ZHONG <le...@gmail.com>.
Thank you Pete for the alternative to host LoadingContext/ResourceSet within
DataFactory.
How to browse from SDOSchemaSAX2Parser to DataFactory please?
My prototype checks LoadingContext/ResourceSet
within SDOSchemaSAX2Parser::startInclude
My LoadingContext/ResourceSet is internal to parsers, not a user context.
On 12/9/06, Pete Robbins <ro...@googlemail.com> wrote:
>
> Yang,
> Yes we've noticed this when loading the Atom schema.
>
>
> On 09/12/06, Yang ZHONG <le...@gmail.com> wrote:
> >
> > While working on http://issues.apache.org/jira/browse/TUSCANY-907
> > I've observed duplicated XSD loading. It could be infinite loading. I
> > might
> > have a way to avoid that.
> >
> > Could you verify current implementation may load redundantly even
> > infinitely?
> > Could you brainstorm how to avoid that?
> >
> > http://pecl.php.net/bugs/bug.php?id=9243
> > has the Test Case:
> > 1. http://ping.chip.org/phr/xml/insurance.xsd imports
> > http://ping.chip.org/phr/xml/types.xsd
> > 2. http://ping.chip.org/phr/xml/insurance.xsd imports
> > http://ping.chip.org/phr/xml/contact.xsd which also imports
> > http://ping.chip.org/phr/xml/types.xsd
> >
> > I've observed http://ping.chip.org/phr/xml/types.xsd is loaded/parsed
> > multiple times during loading http://ping.chip.org/phr/xml/insurance.xsd
> >
> > That impacts performance, especially types.xsd is remote.
> >
> > More deadly, that may cause infinite loading or stack overflow.
> > Imagine A imports B which imports A, or X imports Y which imports Z
> which
> > imports X.
> >
> > If above is true, let's brainstorm how to avoid that.
> >
> > We can have a LoadingContext/ResourceSet to cache loaded/parsed
> resources,
> > and reuse parsing results if a location/URI has been loaded/parsed
> > already.
> >
> > 3-1. The LoadingContext/ResourceSet can be a Thread Local
> > 3-2. The LoadingContext/ResourceSet can be explicitly passed into each
> > parser
> > 3-3. Currently ParserErrorSetter is explicitly passed into each parser,
> > which can host the LoadingContext/ResourceSet
> >
> > Do you have alternatives? Which would you prefer?
>
>
> The most basic problem is when a user uses XSDHelper to load X.xsd which
> imports other schema... which in turn include some of the same schema. At
> the moment his causes redundant parsing. A simple solution is to remember
> the schema namespace/location so the import/include of a schema we already
> have parsed can be ignored. This information could be held in the
> DataFactory so that if the user calls XSDHelper::define() for the same
> schema we could ignore it (as we know the types are already defined).
>
> I'm not sure if you are suggesting a wider ranging improvement of some
> context the user can pass into XSDHelper?
>
> Cheers,
> --
> Pete
>
>
--
Yang ZHONG
Re: [C++] Duplicated/infinite loading? How to avoid
Posted by Pete Robbins <ro...@googlemail.com>.
Yang,
Yes we've noticed this when loading the Atom schema.
On 09/12/06, Yang ZHONG <le...@gmail.com> wrote:
>
> While working on http://issues.apache.org/jira/browse/TUSCANY-907
> I've observed duplicated XSD loading. It could be infinite loading. I
> might
> have a way to avoid that.
>
> Could you verify current implementation may load redundantly even
> infinitely?
> Could you brainstorm how to avoid that?
>
> http://pecl.php.net/bugs/bug.php?id=9243
> has the Test Case:
> 1. http://ping.chip.org/phr/xml/insurance.xsd imports
> http://ping.chip.org/phr/xml/types.xsd
> 2. http://ping.chip.org/phr/xml/insurance.xsd imports
> http://ping.chip.org/phr/xml/contact.xsd which also imports
> http://ping.chip.org/phr/xml/types.xsd
>
> I've observed http://ping.chip.org/phr/xml/types.xsd is loaded/parsed
> multiple times during loading http://ping.chip.org/phr/xml/insurance.xsd
>
> That impacts performance, especially types.xsd is remote.
>
> More deadly, that may cause infinite loading or stack overflow.
> Imagine A imports B which imports A, or X imports Y which imports Z which
> imports X.
>
> If above is true, let's brainstorm how to avoid that.
>
> We can have a LoadingContext/ResourceSet to cache loaded/parsed resources,
> and reuse parsing results if a location/URI has been loaded/parsed
> already.
>
> 3-1. The LoadingContext/ResourceSet can be a Thread Local
> 3-2. The LoadingContext/ResourceSet can be explicitly passed into each
> parser
> 3-3. Currently ParserErrorSetter is explicitly passed into each parser,
> which can host the LoadingContext/ResourceSet
>
> Do you have alternatives? Which would you prefer?
The most basic problem is when a user uses XSDHelper to load X.xsd which
imports other schema... which in turn include some of the same schema. At
the moment his causes redundant parsing. A simple solution is to remember
the schema namespace/location so the import/include of a schema we already
have parsed can be ignored. This information could be held in the
DataFactory so that if the user calls XSDHelper::define() for the same
schema we could ignore it (as we know the types are already defined).
I'm not sure if you are suggesting a wider ranging improvement of some
context the user can pass into XSDHelper?
Cheers,
--
Pete