You are viewing a plain text version of this content. The canonical link for it is here.
Posted to c-dev@xerces.apache.org by Daniel McLean <da...@mds.rmit.edu.au> on 2004/12/02 07:53:16 UTC
DTD caching problems
I am trying to set up grammar caching and have things working nicely for
XML schemas but not for DTDs. I am effectively doing the following:
xercesc::XMLGrammarPool* grammarPool =
xercesc::XMLGrammarPoolImpl(xercesc::XMLPlatformUtils::fgMemoryManager);
parser = xercesc::XMLReaderFactory::createXMLReader(xercesc::XMLPlatformUtils::fgMemoryManager,
grammarPool));
parser->setFeature(xercesc::XMLUni::fgXercesCacheGrammarFromParse, true);
parser->setProperty(xercesc::XMLUni::fgXercesScannerName,
static_cast<void*>(const_cast<XMLCh*>(xercesc::XMLUni::fgDGXMLScanner)));
parser->setFeature(xercesc::XMLUni::fgSAX2CoreValidation, true);
When an XML document with no internal subset is parsed referring to an
external DTD:
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE library SYSTEM "example.dtd">
<library> ...
I get the error:
(system id 'example-dtd.xml', line 2, char 40): An exception occurred!
Type:NoSuchElementException, Message:The key '{0}' could not be found
in the hash table
I put some printf()s into my resolveEntity() call back and was interested
to see resolveEntity() being called twice. The first time with a null
publicId and 'example.dtd' as the systemId, the second time with an empty
string for the publicId and 'example.dtd' as the systemId.
This error does not occur if I replace the lines
parser = xercesc::XMLReaderFactory::createXMLReader(xercesc::XMLPlatformUtils::fgMemoryManager,
grammarPool));
parser->setFeature(xercesc::XMLUni::fgXercesCacheGrammarFromParse, true);
with
parser = xercesc::XMLReaderFactory::createXMLReader();
and therefore do not use grammar caching. When no caching is used, I just
get one call to resolveEntity().
As mentioned above, caching is working fine for XML schemas when I use the
SGXMLScanner.
At one point, I suspected that maybe I needed to call setSystemId() and/or
setPublicId() on the InputSource I return from resolveEntity(), and did:
buf->setSystemId(p_systemId);
buf->setPublicId(NIL);
But this didn't help.
Is this a known problem? Any suggestions beyond submitting a testcase to
jira?
Daniel
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
Re: DTD caching problems
Posted by David Cargill <ca...@ca.ibm.com>.
Hi David B & Alberto,
That is the change I am working on...
Regards,
David A. Cargill
XML Parser Development
IBM Toronto Lab
(905) 413-2371, tie 969
cargilld@ca.ibm.com
Alberto Massari
<amassari@progres
s.com> To
xerces-c-dev@xml.apache.org
12/03/2004 12:58 cc
PM
Subject
Re: DTD caching problems
Please respond to
xerces-c-dev
Hi David,
At 12.50 03/12/2004 -0500, david_n_bertoni@us.ibm.com wrote:
>Hi Alberto,
>
>One problem I noticed with your fix is the call to resolveEntity() in the
>various overloads of resolveSystemId() always result in a null pointer for
>the public ID, meaning that any mechanism that relies on public IDs will
>not work. For example, in IGXMLScanner.::resolveSystemId():
>
> XMLResourceIdentifier
>resourceIdentifier(XMLResourceIdentifier::ExternalEntity,
> expSysId.getRawBuffer(), 0,
>XMLUni::fgZeroLenString, lastInfo.systemId);
> srcToFill = fEntityHandler->resolveEntity(&resourceIdentifier);
>
>You can see the XMLResourceIdentifier instance here is constructed with a
>null pointer for the public ID parameter.
>
>Perhaps changing the signature for resolveSystemId() to provide the public
>ID, if any, will solve this problem?
Yes, I too think that resolveSystemId should propagate the publicId (in my
commit I also fixed DGXMLScanner::resolveSystemId so that it propagates the
baseUri, as IGXMLScanner was doing)
Please go ahead and change the signature.
Thanks,
Alberto
>Thanks!
>
>Dave
>
>
>
>
>
>Alberto Massari <am...@progress.com>
>12/03/2004 09:14 AM
>Please respond to xerces-c-dev
>
> To: xerces-c-dev@xml.apache.org
> cc: (bcc: David N Bertoni/Cambridge/IBM)
> Subject: Re: DTD caching problems
>
>Hi David,
>
>At 12.05 03/12/2004 -0500, David Cargill wrote:
>
> >Hi Alberto,
> >Looks like we are both working on the same problem. I have a different
> >solution for #1 and that is to call switchGrammar in scanDocType. That
>is
> >what happens in IGXMLScanner but not DGXMLScanner and that is why it
>works
> >for IG and not DG.
>
>I have thought about that, but I think that changing orphanGrammar would
>be
>a safer also for other cases when we forget to keep the two maps in synch.
>
>But feel free to revert my change.
>
>
> >For #2, still looking at how to avoid the 2nd call. Another customer
> >reported a similar problem as this as well...
>
>Have a look at what I committed, and let me know if this solves your
>problem too.
>
>Alberto
>
>
> >Regards,
> >David A. Cargill
> >XML Parser Development
> >IBM Toronto Lab
> >(905) 413-2371, tie 969
> >cargilld@ca.ibm.com
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
Re: DTD caching problems
Posted by Alberto Massari <am...@progress.com>.
Hi David,
At 12.50 03/12/2004 -0500, david_n_bertoni@us.ibm.com wrote:
>Hi Alberto,
>
>One problem I noticed with your fix is the call to resolveEntity() in the
>various overloads of resolveSystemId() always result in a null pointer for
>the public ID, meaning that any mechanism that relies on public IDs will
>not work. For example, in IGXMLScanner.::resolveSystemId():
>
> XMLResourceIdentifier
>resourceIdentifier(XMLResourceIdentifier::ExternalEntity,
> expSysId.getRawBuffer(), 0,
>XMLUni::fgZeroLenString, lastInfo.systemId);
> srcToFill = fEntityHandler->resolveEntity(&resourceIdentifier);
>
>You can see the XMLResourceIdentifier instance here is constructed with a
>null pointer for the public ID parameter.
>
>Perhaps changing the signature for resolveSystemId() to provide the public
>ID, if any, will solve this problem?
Yes, I too think that resolveSystemId should propagate the publicId (in my
commit I also fixed DGXMLScanner::resolveSystemId so that it propagates the
baseUri, as IGXMLScanner was doing)
Please go ahead and change the signature.
Thanks,
Alberto
>Thanks!
>
>Dave
>
>
>
>
>
>Alberto Massari <am...@progress.com>
>12/03/2004 09:14 AM
>Please respond to xerces-c-dev
>
> To: xerces-c-dev@xml.apache.org
> cc: (bcc: David N Bertoni/Cambridge/IBM)
> Subject: Re: DTD caching problems
>
>Hi David,
>
>At 12.05 03/12/2004 -0500, David Cargill wrote:
>
> >Hi Alberto,
> >Looks like we are both working on the same problem. I have a different
> >solution for #1 and that is to call switchGrammar in scanDocType. That
>is
> >what happens in IGXMLScanner but not DGXMLScanner and that is why it
>works
> >for IG and not DG.
>
>I have thought about that, but I think that changing orphanGrammar would
>be
>a safer also for other cases when we forget to keep the two maps in synch.
>
>But feel free to revert my change.
>
>
> >For #2, still looking at how to avoid the 2nd call. Another customer
> >reported a similar problem as this as well...
>
>Have a look at what I committed, and let me know if this solves your
>problem too.
>
>Alberto
>
>
> >Regards,
> >David A. Cargill
> >XML Parser Development
> >IBM Toronto Lab
> >(905) 413-2371, tie 969
> >cargilld@ca.ibm.com
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
Re: DTD caching problems
Posted by da...@us.ibm.com.
Hi Alberto,
One problem I noticed with your fix is the call to resolveEntity() in the
various overloads of resolveSystemId() always result in a null pointer for
the public ID, meaning that any mechanism that relies on public IDs will
not work. For example, in IGXMLScanner.::resolveSystemId():
XMLResourceIdentifier
resourceIdentifier(XMLResourceIdentifier::ExternalEntity,
expSysId.getRawBuffer(), 0,
XMLUni::fgZeroLenString, lastInfo.systemId);
srcToFill = fEntityHandler->resolveEntity(&resourceIdentifier);
You can see the XMLResourceIdentifier instance here is constructed with a
null pointer for the public ID parameter.
Perhaps changing the signature for resolveSystemId() to provide the public
ID, if any, will solve this problem?
Thanks!
Dave
Alberto Massari <am...@progress.com>
12/03/2004 09:14 AM
Please respond to xerces-c-dev
To: xerces-c-dev@xml.apache.org
cc: (bcc: David N Bertoni/Cambridge/IBM)
Subject: Re: DTD caching problems
Hi David,
At 12.05 03/12/2004 -0500, David Cargill wrote:
>Hi Alberto,
>Looks like we are both working on the same problem. I have a different
>solution for #1 and that is to call switchGrammar in scanDocType. That
is
>what happens in IGXMLScanner but not DGXMLScanner and that is why it
works
>for IG and not DG.
I have thought about that, but I think that changing orphanGrammar would
be
a safer also for other cases when we forget to keep the two maps in synch.
But feel free to revert my change.
>For #2, still looking at how to avoid the 2nd call. Another customer
>reported a similar problem as this as well...
Have a look at what I committed, and let me know if this solves your
problem too.
Alberto
>Regards,
>David A. Cargill
>XML Parser Development
>IBM Toronto Lab
>(905) 413-2371, tie 969
>cargilld@ca.ibm.com
>
>
>
> Alberto Massari
> <amassari@progres
> s.com> To
> xerces-c-dev@xml.apache.org
> 12/03/2004 11:51 cc
> AM
> Subject
> Re: DTD caching problems
> Please respond to
> xerces-c-dev
>
>
>
>
>
>
>
>
>Hi Daniel,
>I have looked at the testcase you submitted, and I found two things:
>1) the exception you get when using cached grammars and the DGXMLScanner
is
>
>caused by GrammarResolver::orphanGrammar; to fix it, apply this diff
>
>diff -u -r1.29 GrammarResolver.cpp
>--- GrammarResolver.cpp 29 Sep 2004 19:27:07 -0000 1.29
>+++ GrammarResolver.cpp 3 Dec 2004 16:16:23 -0000
>@@ -439,7 +439,8 @@
> Grammar* grammar = fGrammarPool->orphanGrammar(nameSpaceKey);
> if (grammar)
> {
>- fGrammarFromPool->removeKey(nameSpaceKey);
>+ if (fGrammarFromPool->containsKey(nameSpaceKey))
>+ fGrammarFromPool->removeKey(nameSpaceKey);
> return grammar;
> }
> // It failed to remove it from the grammar pool either because
it
>
>2) as for the extra resolveEntity() you see, it's by design;
>GrammarResolver now stores the DTDs using their systemID as key (instead
of
>
>the root element name), so, when caching is on, an extra call to
>resolveEntity is done in order to be able to invoke
>InputSource::getSystemId(). At that point, if the systemId is already in
>the cache, the input source is not parsed; if it is not in the cache, a
new
>
>reader is created (and this triggers a new resolveEntity call)
>I will change this to at least avoid the second call (as we have already
>resolved the DTD).
>
>Thanks for the testcase,
>Alberto
>
>At 16.42 03/12/2004 +1100, Daniel McLean wrote:
> >I've put together a demo based on one of the sample programs shipped
with
> >Xerces. The demo creates a Xerces GrammarPoolImpl then conducts a
number
> >of validating XML parses using that grammar pool when caching is
enabled
> >for the parser. The test documents include an XML file referring to a
DTD
> >and an XML file referring to an XML schema. Each type of parse is done
> >twice in succession to observe the caching behaviour of Xerces, with a
new
> >parser instance created for each parse. The parses done include:
> >
> > DGXMLScanner without caching involving DTD validation
> > DGXMLScanner with caching involving DTD validation
> >
> > IGXMLScanner without caching involving DTD validation
> > IGXMLScanner with caching involving DTD validation
> >
> > SGXMLScanner without caching involving XML schema validation
> > SGXMLScanner with caching involving XML schema validation
> >
> >I ran with on an Fedora Core 1 linux x86 platform using Xerces 2.6.0.
> >
> >I see a number of surprising things (refer to output below):
> > - parsing fails when caching is enabled whilst using the DGXMLScanner
> > - when using DGXMLScanner or IGXMLScanner for DTD validation, the DTD
> > appears to be re-resolved for each parse. I haven't checked
whether
> > the DTD is actually being re-parsed after being resolved each time,
>but
> > I wonder why it is being resolved each time if it is supposedly
being
> > cached
> > - the DTD is being resolved twice on the first parse when parsing
with a
> > DTD and caching enabled. This seems questionable.
> >
> >I've attached a tarball containing the files for the demo, which should
> >work after being unpacked into the samples directory.
> >
> >Regards,
> >
> >Daniel
> >
> >doParse dtdDGXML
> >parse #1
> >resolveEntity("", "example.dtd")
> >parse #2
> >resolveEntity("", "example.dtd")
> >----------------------------
> >doParse dtdDGXMLCaching
> >parse #1
> >resolveEntity(null, "example.dtd")
> >resolveEntity("", "example.dtd")
> >
> >Fatal Error at (file
>
>/home/daniel/xerces/xerces-c-src_2_6_0/samples/GrammarCache/example-dtd.xml,
>
> >line 2, char 40): An exception occurred! Type:NoSuchElementException,
> >Message:The key '{0}' could not be found in the hash table
> >parse #2
> >resolveEntity(null, "example.dtd")
> >resolveEntity("", "example.dtd")
> >
> >Fatal Error at (file
>
>/home/daniel/xerces/xerces-c-src_2_6_0/samples/GrammarCache/example-dtd.xml,
>
> >line 2, char 40): An exception occurred! Type:NoSuchElementException,
> >Message:The key '{0}' could not be found in the hash table
> >----------------------------
> >doParse dtdIGXML
> >parse #1
> >resolveEntity("", "example.dtd")
> >parse #2
> >resolveEntity("", "example.dtd")
> >----------------------------
> >doParse dtdIGXMLCaching
> >parse #1
> >resolveEntity("", "example.dtd")
> >resolveEntity("", "example.dtd")
> >parse #2
> >resolveEntity("", "example.dtd")
> >----------------------------
> >doParse xsd
> >parse #1
> >resolveEntity("", "example.xsd")
> >parse #2
> >resolveEntity("", "example.xsd")
> >----------------------------
> >doParse xsdCaching
> >parse #1
> >resolveEntity("", "example.xsd")
> >parse #2
> >----------------------------
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
> >For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
>For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
>
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
>For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
Re: DTD caching problems
Posted by David Cargill <ca...@ca.ibm.com>.
Hi Alberto,
Thanks. I will try and look and see which way is more performant (calling
switchGrammar or calling containsKey). I don't think there are any other
cases where the maps aren't in synch. Of course, orphanGrammar is only
currently called for XMLUni::fgDTDEntityString.
For #2, your fix is a good start. There are a few things I need to modify
(move where Janitor is set for one). The bigger change is that with the
original code the first call to resolveEntity (from getSystemId) only
passes in the systemId. The second call was also passing in the publicId.
So for the other problem I am working on there will still be two calls to
resolveEntity as they aren't resolve using the SystemId so the first one
will fail.
Regards,
David A. Cargill
XML Parser Development
IBM Toronto Lab
(905) 413-2371, tie 969
cargilld@ca.ibm.com
Alberto Massari
<amassari@progres
s.com> To
xerces-c-dev@xml.apache.org
12/03/2004 12:14 cc
PM
Subject
Re: DTD caching problems
Please respond to
xerces-c-dev
Hi David,
At 12.05 03/12/2004 -0500, David Cargill wrote:
>Hi Alberto,
>Looks like we are both working on the same problem. I have a different
>solution for #1 and that is to call switchGrammar in scanDocType. That is
>what happens in IGXMLScanner but not DGXMLScanner and that is why it works
>for IG and not DG.
I have thought about that, but I think that changing orphanGrammar would be
a safer also for other cases when we forget to keep the two maps in synch.
But feel free to revert my change.
>For #2, still looking at how to avoid the 2nd call. Another customer
>reported a similar problem as this as well...
Have a look at what I committed, and let me know if this solves your
problem too.
Alberto
>Regards,
>David A. Cargill
>XML Parser Development
>IBM Toronto Lab
>(905) 413-2371, tie 969
>cargilld@ca.ibm.com
>
>
>
> Alberto Massari
> <amassari@progres
> s.com>
To
> xerces-c-dev@xml.apache.org
> 12/03/2004 11:51
cc
> AM
>
Subject
> Re: DTD caching problems
> Please respond to
> xerces-c-dev
>
>
>
>
>
>
>
>
>Hi Daniel,
>I have looked at the testcase you submitted, and I found two things:
>1) the exception you get when using cached grammars and the DGXMLScanner
is
>
>caused by GrammarResolver::orphanGrammar; to fix it, apply this diff
>
>diff -u -r1.29 GrammarResolver.cpp
>--- GrammarResolver.cpp 29 Sep 2004 19:27:07 -0000 1.29
>+++ GrammarResolver.cpp 3 Dec 2004 16:16:23 -0000
>@@ -439,7 +439,8 @@
> Grammar* grammar = fGrammarPool->orphanGrammar(nameSpaceKey);
> if (grammar)
> {
>- fGrammarFromPool->removeKey(nameSpaceKey);
>+ if (fGrammarFromPool->containsKey(nameSpaceKey))
>+ fGrammarFromPool->removeKey(nameSpaceKey);
> return grammar;
> }
> // It failed to remove it from the grammar pool either because
it
>
>2) as for the extra resolveEntity() you see, it's by design;
>GrammarResolver now stores the DTDs using their systemID as key (instead
of
>
>the root element name), so, when caching is on, an extra call to
>resolveEntity is done in order to be able to invoke
>InputSource::getSystemId(). At that point, if the systemId is already in
>the cache, the input source is not parsed; if it is not in the cache, a
new
>
>reader is created (and this triggers a new resolveEntity call)
>I will change this to at least avoid the second call (as we have already
>resolved the DTD).
>
>Thanks for the testcase,
>Alberto
>
>At 16.42 03/12/2004 +1100, Daniel McLean wrote:
> >I've put together a demo based on one of the sample programs shipped
with
> >Xerces. The demo creates a Xerces GrammarPoolImpl then conducts a
number
> >of validating XML parses using that grammar pool when caching is enabled
> >for the parser. The test documents include an XML file referring to a
DTD
> >and an XML file referring to an XML schema. Each type of parse is done
> >twice in succession to observe the caching behaviour of Xerces, with a
new
> >parser instance created for each parse. The parses done include:
> >
> > DGXMLScanner without caching involving DTD validation
> > DGXMLScanner with caching involving DTD validation
> >
> > IGXMLScanner without caching involving DTD validation
> > IGXMLScanner with caching involving DTD validation
> >
> > SGXMLScanner without caching involving XML schema validation
> > SGXMLScanner with caching involving XML schema validation
> >
> >I ran with on an Fedora Core 1 linux x86 platform using Xerces 2.6.0.
> >
> >I see a number of surprising things (refer to output below):
> > - parsing fails when caching is enabled whilst using the DGXMLScanner
> > - when using DGXMLScanner or IGXMLScanner for DTD validation, the DTD
> > appears to be re-resolved for each parse. I haven't checked whether
> > the DTD is actually being re-parsed after being resolved each time,
>but
> > I wonder why it is being resolved each time if it is supposedly
being
> > cached
> > - the DTD is being resolved twice on the first parse when parsing with
a
> > DTD and caching enabled. This seems questionable.
> >
> >I've attached a tarball containing the files for the demo, which should
> >work after being unpacked into the samples directory.
> >
> >Regards,
> >
> >Daniel
> >
> >doParse dtdDGXML
> >parse #1
> >resolveEntity("", "example.dtd")
> >parse #2
> >resolveEntity("", "example.dtd")
> >----------------------------
> >doParse dtdDGXMLCaching
> >parse #1
> >resolveEntity(null, "example.dtd")
> >resolveEntity("", "example.dtd")
> >
> >Fatal Error at (file
>
>/home/daniel/xerces/xerces-c-src_2_6_0/samples/GrammarCache/example-dtd.xml,
>
> >line 2, char 40): An exception occurred! Type:NoSuchElementException,
> >Message:The key '{0}' could not be found in the hash table
> >parse #2
> >resolveEntity(null, "example.dtd")
> >resolveEntity("", "example.dtd")
> >
> >Fatal Error at (file
>
>/home/daniel/xerces/xerces-c-src_2_6_0/samples/GrammarCache/example-dtd.xml,
>
> >line 2, char 40): An exception occurred! Type:NoSuchElementException,
> >Message:The key '{0}' could not be found in the hash table
> >----------------------------
> >doParse dtdIGXML
> >parse #1
> >resolveEntity("", "example.dtd")
> >parse #2
> >resolveEntity("", "example.dtd")
> >----------------------------
> >doParse dtdIGXMLCaching
> >parse #1
> >resolveEntity("", "example.dtd")
> >resolveEntity("", "example.dtd")
> >parse #2
> >resolveEntity("", "example.dtd")
> >----------------------------
> >doParse xsd
> >parse #1
> >resolveEntity("", "example.xsd")
> >parse #2
> >resolveEntity("", "example.xsd")
> >----------------------------
> >doParse xsdCaching
> >parse #1
> >resolveEntity("", "example.xsd")
> >parse #2
> >----------------------------
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
> >For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
>For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
>
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
>For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
Re: DTD caching problems
Posted by Alberto Massari <am...@progress.com>.
Hi David,
At 12.05 03/12/2004 -0500, David Cargill wrote:
>Hi Alberto,
>Looks like we are both working on the same problem. I have a different
>solution for #1 and that is to call switchGrammar in scanDocType. That is
>what happens in IGXMLScanner but not DGXMLScanner and that is why it works
>for IG and not DG.
I have thought about that, but I think that changing orphanGrammar would be
a safer also for other cases when we forget to keep the two maps in synch.
But feel free to revert my change.
>For #2, still looking at how to avoid the 2nd call. Another customer
>reported a similar problem as this as well...
Have a look at what I committed, and let me know if this solves your
problem too.
Alberto
>Regards,
>David A. Cargill
>XML Parser Development
>IBM Toronto Lab
>(905) 413-2371, tie 969
>cargilld@ca.ibm.com
>
>
>
> Alberto Massari
> <amassari@progres
> s.com> To
> xerces-c-dev@xml.apache.org
> 12/03/2004 11:51 cc
> AM
> Subject
> Re: DTD caching problems
> Please respond to
> xerces-c-dev
>
>
>
>
>
>
>
>
>Hi Daniel,
>I have looked at the testcase you submitted, and I found two things:
>1) the exception you get when using cached grammars and the DGXMLScanner is
>
>caused by GrammarResolver::orphanGrammar; to fix it, apply this diff
>
>diff -u -r1.29 GrammarResolver.cpp
>--- GrammarResolver.cpp 29 Sep 2004 19:27:07 -0000 1.29
>+++ GrammarResolver.cpp 3 Dec 2004 16:16:23 -0000
>@@ -439,7 +439,8 @@
> Grammar* grammar = fGrammarPool->orphanGrammar(nameSpaceKey);
> if (grammar)
> {
>- fGrammarFromPool->removeKey(nameSpaceKey);
>+ if (fGrammarFromPool->containsKey(nameSpaceKey))
>+ fGrammarFromPool->removeKey(nameSpaceKey);
> return grammar;
> }
> // It failed to remove it from the grammar pool either because it
>
>2) as for the extra resolveEntity() you see, it's by design;
>GrammarResolver now stores the DTDs using their systemID as key (instead of
>
>the root element name), so, when caching is on, an extra call to
>resolveEntity is done in order to be able to invoke
>InputSource::getSystemId(). At that point, if the systemId is already in
>the cache, the input source is not parsed; if it is not in the cache, a new
>
>reader is created (and this triggers a new resolveEntity call)
>I will change this to at least avoid the second call (as we have already
>resolved the DTD).
>
>Thanks for the testcase,
>Alberto
>
>At 16.42 03/12/2004 +1100, Daniel McLean wrote:
> >I've put together a demo based on one of the sample programs shipped with
> >Xerces. The demo creates a Xerces GrammarPoolImpl then conducts a number
> >of validating XML parses using that grammar pool when caching is enabled
> >for the parser. The test documents include an XML file referring to a DTD
> >and an XML file referring to an XML schema. Each type of parse is done
> >twice in succession to observe the caching behaviour of Xerces, with a new
> >parser instance created for each parse. The parses done include:
> >
> > DGXMLScanner without caching involving DTD validation
> > DGXMLScanner with caching involving DTD validation
> >
> > IGXMLScanner without caching involving DTD validation
> > IGXMLScanner with caching involving DTD validation
> >
> > SGXMLScanner without caching involving XML schema validation
> > SGXMLScanner with caching involving XML schema validation
> >
> >I ran with on an Fedora Core 1 linux x86 platform using Xerces 2.6.0.
> >
> >I see a number of surprising things (refer to output below):
> > - parsing fails when caching is enabled whilst using the DGXMLScanner
> > - when using DGXMLScanner or IGXMLScanner for DTD validation, the DTD
> > appears to be re-resolved for each parse. I haven't checked whether
> > the DTD is actually being re-parsed after being resolved each time,
>but
> > I wonder why it is being resolved each time if it is supposedly being
> > cached
> > - the DTD is being resolved twice on the first parse when parsing with a
> > DTD and caching enabled. This seems questionable.
> >
> >I've attached a tarball containing the files for the demo, which should
> >work after being unpacked into the samples directory.
> >
> >Regards,
> >
> >Daniel
> >
> >doParse dtdDGXML
> >parse #1
> >resolveEntity("", "example.dtd")
> >parse #2
> >resolveEntity("", "example.dtd")
> >----------------------------
> >doParse dtdDGXMLCaching
> >parse #1
> >resolveEntity(null, "example.dtd")
> >resolveEntity("", "example.dtd")
> >
> >Fatal Error at (file
> >/home/daniel/xerces/xerces-c-src_2_6_0/samples/GrammarCache/example-dtd.xml,
>
> >line 2, char 40): An exception occurred! Type:NoSuchElementException,
> >Message:The key '{0}' could not be found in the hash table
> >parse #2
> >resolveEntity(null, "example.dtd")
> >resolveEntity("", "example.dtd")
> >
> >Fatal Error at (file
> >/home/daniel/xerces/xerces-c-src_2_6_0/samples/GrammarCache/example-dtd.xml,
>
> >line 2, char 40): An exception occurred! Type:NoSuchElementException,
> >Message:The key '{0}' could not be found in the hash table
> >----------------------------
> >doParse dtdIGXML
> >parse #1
> >resolveEntity("", "example.dtd")
> >parse #2
> >resolveEntity("", "example.dtd")
> >----------------------------
> >doParse dtdIGXMLCaching
> >parse #1
> >resolveEntity("", "example.dtd")
> >resolveEntity("", "example.dtd")
> >parse #2
> >resolveEntity("", "example.dtd")
> >----------------------------
> >doParse xsd
> >parse #1
> >resolveEntity("", "example.xsd")
> >parse #2
> >resolveEntity("", "example.xsd")
> >----------------------------
> >doParse xsdCaching
> >parse #1
> >resolveEntity("", "example.xsd")
> >parse #2
> >----------------------------
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
> >For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
>For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
>
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
>For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
Re: DTD caching problems
Posted by David Cargill <ca...@ca.ibm.com>.
Hi Alberto,
Looks like we are both working on the same problem. I have a different
solution for #1 and that is to call switchGrammar in scanDocType. That is
what happens in IGXMLScanner but not DGXMLScanner and that is why it works
for IG and not DG.
For #2, still looking at how to avoid the 2nd call. Another customer
reported a similar problem as this as well...
Regards,
David A. Cargill
XML Parser Development
IBM Toronto Lab
(905) 413-2371, tie 969
cargilld@ca.ibm.com
Alberto Massari
<amassari@progres
s.com> To
xerces-c-dev@xml.apache.org
12/03/2004 11:51 cc
AM
Subject
Re: DTD caching problems
Please respond to
xerces-c-dev
Hi Daniel,
I have looked at the testcase you submitted, and I found two things:
1) the exception you get when using cached grammars and the DGXMLScanner is
caused by GrammarResolver::orphanGrammar; to fix it, apply this diff
diff -u -r1.29 GrammarResolver.cpp
--- GrammarResolver.cpp 29 Sep 2004 19:27:07 -0000 1.29
+++ GrammarResolver.cpp 3 Dec 2004 16:16:23 -0000
@@ -439,7 +439,8 @@
Grammar* grammar = fGrammarPool->orphanGrammar(nameSpaceKey);
if (grammar)
{
- fGrammarFromPool->removeKey(nameSpaceKey);
+ if (fGrammarFromPool->containsKey(nameSpaceKey))
+ fGrammarFromPool->removeKey(nameSpaceKey);
return grammar;
}
// It failed to remove it from the grammar pool either because it
2) as for the extra resolveEntity() you see, it's by design;
GrammarResolver now stores the DTDs using their systemID as key (instead of
the root element name), so, when caching is on, an extra call to
resolveEntity is done in order to be able to invoke
InputSource::getSystemId(). At that point, if the systemId is already in
the cache, the input source is not parsed; if it is not in the cache, a new
reader is created (and this triggers a new resolveEntity call)
I will change this to at least avoid the second call (as we have already
resolved the DTD).
Thanks for the testcase,
Alberto
At 16.42 03/12/2004 +1100, Daniel McLean wrote:
>I've put together a demo based on one of the sample programs shipped with
>Xerces. The demo creates a Xerces GrammarPoolImpl then conducts a number
>of validating XML parses using that grammar pool when caching is enabled
>for the parser. The test documents include an XML file referring to a DTD
>and an XML file referring to an XML schema. Each type of parse is done
>twice in succession to observe the caching behaviour of Xerces, with a new
>parser instance created for each parse. The parses done include:
>
> DGXMLScanner without caching involving DTD validation
> DGXMLScanner with caching involving DTD validation
>
> IGXMLScanner without caching involving DTD validation
> IGXMLScanner with caching involving DTD validation
>
> SGXMLScanner without caching involving XML schema validation
> SGXMLScanner with caching involving XML schema validation
>
>I ran with on an Fedora Core 1 linux x86 platform using Xerces 2.6.0.
>
>I see a number of surprising things (refer to output below):
> - parsing fails when caching is enabled whilst using the DGXMLScanner
> - when using DGXMLScanner or IGXMLScanner for DTD validation, the DTD
> appears to be re-resolved for each parse. I haven't checked whether
> the DTD is actually being re-parsed after being resolved each time,
but
> I wonder why it is being resolved each time if it is supposedly being
> cached
> - the DTD is being resolved twice on the first parse when parsing with a
> DTD and caching enabled. This seems questionable.
>
>I've attached a tarball containing the files for the demo, which should
>work after being unpacked into the samples directory.
>
>Regards,
>
>Daniel
>
>doParse dtdDGXML
>parse #1
>resolveEntity("", "example.dtd")
>parse #2
>resolveEntity("", "example.dtd")
>----------------------------
>doParse dtdDGXMLCaching
>parse #1
>resolveEntity(null, "example.dtd")
>resolveEntity("", "example.dtd")
>
>Fatal Error at (file
>/home/daniel/xerces/xerces-c-src_2_6_0/samples/GrammarCache/example-dtd.xml,
>line 2, char 40): An exception occurred! Type:NoSuchElementException,
>Message:The key '{0}' could not be found in the hash table
>parse #2
>resolveEntity(null, "example.dtd")
>resolveEntity("", "example.dtd")
>
>Fatal Error at (file
>/home/daniel/xerces/xerces-c-src_2_6_0/samples/GrammarCache/example-dtd.xml,
>line 2, char 40): An exception occurred! Type:NoSuchElementException,
>Message:The key '{0}' could not be found in the hash table
>----------------------------
>doParse dtdIGXML
>parse #1
>resolveEntity("", "example.dtd")
>parse #2
>resolveEntity("", "example.dtd")
>----------------------------
>doParse dtdIGXMLCaching
>parse #1
>resolveEntity("", "example.dtd")
>resolveEntity("", "example.dtd")
>parse #2
>resolveEntity("", "example.dtd")
>----------------------------
>doParse xsd
>parse #1
>resolveEntity("", "example.xsd")
>parse #2
>resolveEntity("", "example.xsd")
>----------------------------
>doParse xsdCaching
>parse #1
>resolveEntity("", "example.xsd")
>parse #2
>----------------------------
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
>For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
Re: DTD caching problems
Posted by Alberto Massari <am...@progress.com>.
Hi Daniel,
I have looked at the testcase you submitted, and I found two things:
1) the exception you get when using cached grammars and the DGXMLScanner is
caused by GrammarResolver::orphanGrammar; to fix it, apply this diff
diff -u -r1.29 GrammarResolver.cpp
--- GrammarResolver.cpp 29 Sep 2004 19:27:07 -0000 1.29
+++ GrammarResolver.cpp 3 Dec 2004 16:16:23 -0000
@@ -439,7 +439,8 @@
Grammar* grammar = fGrammarPool->orphanGrammar(nameSpaceKey);
if (grammar)
{
- fGrammarFromPool->removeKey(nameSpaceKey);
+ if (fGrammarFromPool->containsKey(nameSpaceKey))
+ fGrammarFromPool->removeKey(nameSpaceKey);
return grammar;
}
// It failed to remove it from the grammar pool either because it
2) as for the extra resolveEntity() you see, it's by design;
GrammarResolver now stores the DTDs using their systemID as key (instead of
the root element name), so, when caching is on, an extra call to
resolveEntity is done in order to be able to invoke
InputSource::getSystemId(). At that point, if the systemId is already in
the cache, the input source is not parsed; if it is not in the cache, a new
reader is created (and this triggers a new resolveEntity call)
I will change this to at least avoid the second call (as we have already
resolved the DTD).
Thanks for the testcase,
Alberto
At 16.42 03/12/2004 +1100, Daniel McLean wrote:
>I've put together a demo based on one of the sample programs shipped with
>Xerces. The demo creates a Xerces GrammarPoolImpl then conducts a number
>of validating XML parses using that grammar pool when caching is enabled
>for the parser. The test documents include an XML file referring to a DTD
>and an XML file referring to an XML schema. Each type of parse is done
>twice in succession to observe the caching behaviour of Xerces, with a new
>parser instance created for each parse. The parses done include:
>
> DGXMLScanner without caching involving DTD validation
> DGXMLScanner with caching involving DTD validation
>
> IGXMLScanner without caching involving DTD validation
> IGXMLScanner with caching involving DTD validation
>
> SGXMLScanner without caching involving XML schema validation
> SGXMLScanner with caching involving XML schema validation
>
>I ran with on an Fedora Core 1 linux x86 platform using Xerces 2.6.0.
>
>I see a number of surprising things (refer to output below):
> - parsing fails when caching is enabled whilst using the DGXMLScanner
> - when using DGXMLScanner or IGXMLScanner for DTD validation, the DTD
> appears to be re-resolved for each parse. I haven't checked whether
> the DTD is actually being re-parsed after being resolved each time, but
> I wonder why it is being resolved each time if it is supposedly being
> cached
> - the DTD is being resolved twice on the first parse when parsing with a
> DTD and caching enabled. This seems questionable.
>
>I've attached a tarball containing the files for the demo, which should
>work after being unpacked into the samples directory.
>
>Regards,
>
>Daniel
>
>doParse dtdDGXML
>parse #1
>resolveEntity("", "example.dtd")
>parse #2
>resolveEntity("", "example.dtd")
>----------------------------
>doParse dtdDGXMLCaching
>parse #1
>resolveEntity(null, "example.dtd")
>resolveEntity("", "example.dtd")
>
>Fatal Error at (file
>/home/daniel/xerces/xerces-c-src_2_6_0/samples/GrammarCache/example-dtd.xml,
>line 2, char 40): An exception occurred! Type:NoSuchElementException,
>Message:The key '{0}' could not be found in the hash table
>parse #2
>resolveEntity(null, "example.dtd")
>resolveEntity("", "example.dtd")
>
>Fatal Error at (file
>/home/daniel/xerces/xerces-c-src_2_6_0/samples/GrammarCache/example-dtd.xml,
>line 2, char 40): An exception occurred! Type:NoSuchElementException,
>Message:The key '{0}' could not be found in the hash table
>----------------------------
>doParse dtdIGXML
>parse #1
>resolveEntity("", "example.dtd")
>parse #2
>resolveEntity("", "example.dtd")
>----------------------------
>doParse dtdIGXMLCaching
>parse #1
>resolveEntity("", "example.dtd")
>resolveEntity("", "example.dtd")
>parse #2
>resolveEntity("", "example.dtd")
>----------------------------
>doParse xsd
>parse #1
>resolveEntity("", "example.xsd")
>parse #2
>resolveEntity("", "example.xsd")
>----------------------------
>doParse xsdCaching
>parse #1
>resolveEntity("", "example.xsd")
>parse #2
>----------------------------
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
>For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
Re: DTD caching problems
Posted by Daniel McLean <da...@mds.rmit.edu.au>.
I've put together a demo based on one of the sample programs shipped with
Xerces. The demo creates a Xerces GrammarPoolImpl then conducts a number
of validating XML parses using that grammar pool when caching is enabled
for the parser. The test documents include an XML file referring to a DTD
and an XML file referring to an XML schema. Each type of parse is done
twice in succession to observe the caching behaviour of Xerces, with a new
parser instance created for each parse. The parses done include:
DGXMLScanner without caching involving DTD validation
DGXMLScanner with caching involving DTD validation
IGXMLScanner without caching involving DTD validation
IGXMLScanner with caching involving DTD validation
SGXMLScanner without caching involving XML schema validation
SGXMLScanner with caching involving XML schema validation
I ran with on an Fedora Core 1 linux x86 platform using Xerces 2.6.0.
I see a number of surprising things (refer to output below):
- parsing fails when caching is enabled whilst using the DGXMLScanner
- when using DGXMLScanner or IGXMLScanner for DTD validation, the DTD
appears to be re-resolved for each parse. I haven't checked whether
the DTD is actually being re-parsed after being resolved each time, but
I wonder why it is being resolved each time if it is supposedly being
cached
- the DTD is being resolved twice on the first parse when parsing with a
DTD and caching enabled. This seems questionable.
I've attached a tarball containing the files for the demo, which should
work after being unpacked into the samples directory.
Regards,
Daniel
doParse dtdDGXML
parse #1
resolveEntity("", "example.dtd")
parse #2
resolveEntity("", "example.dtd")
----------------------------
doParse dtdDGXMLCaching
parse #1
resolveEntity(null, "example.dtd")
resolveEntity("", "example.dtd")
Fatal Error at (file
/home/daniel/xerces/xerces-c-src_2_6_0/samples/GrammarCache/example-dtd.xml,
line 2, char 40): An exception occurred! Type:NoSuchElementException,
Message:The key '{0}' could not be found in the hash table
parse #2
resolveEntity(null, "example.dtd")
resolveEntity("", "example.dtd")
Fatal Error at (file
/home/daniel/xerces/xerces-c-src_2_6_0/samples/GrammarCache/example-dtd.xml,
line 2, char 40): An exception occurred! Type:NoSuchElementException,
Message:The key '{0}' could not be found in the hash table
----------------------------
doParse dtdIGXML
parse #1
resolveEntity("", "example.dtd")
parse #2
resolveEntity("", "example.dtd")
----------------------------
doParse dtdIGXMLCaching
parse #1
resolveEntity("", "example.dtd")
resolveEntity("", "example.dtd")
parse #2
resolveEntity("", "example.dtd")
----------------------------
doParse xsd
parse #1
resolveEntity("", "example.xsd")
parse #2
resolveEntity("", "example.xsd")
----------------------------
doParse xsdCaching
parse #1
resolveEntity("", "example.xsd")
parse #2
----------------------------