You are viewing a plain text version of this content. The canonical link for it is here.
Posted to c-dev@xerces.apache.org by PeiYong Zhang <pe...@ca.ibm.com> on 2003/05/13 21:31:13 UTC
grammar pool
Hi everyone,
We have a request for grammar pool and I thought that I would post the
requirement and proposed design to the mailing list to gather feedback.
This new feature would NOT be in the upcoming Xercesc2.3 .
Requirement
-----------------
Currently, grammars carry around state about the instance document they
are validating. This makes it impossible to share grammar instances amongst
parsers, which is desirable to reduce memory footprint.
Design Objectives
--------------------------
Basically the idea is to have grammar exist beyond the lifetime of the parser.
Currently the function loadGrammar returns the Grammar pointer to the user.
But such grammar is owned by the parser, and will be gone when the parser is
deleted.
This feature is to make the grammar independent of the parser, and thus able
to be shared among parsers. External application can plug a grammar pool to
a parser so that it can cache any parsed grammars to the grammar pool and
retrieve grammars from the grammar pool.
Design Issues
------------------------
1. Thread Safety
In a multi-threading environment, it is possible that more than one parser
instances parsing the same instance document with "toCacheGrammarFromParse",
and they may step on each other in putting grammar in to a shared grammar pool,
and therefore cause memory leakage.
A grammar pool implementation needs to ensure thread safety if such an
implementation is to be used by an external application which runs in a
multi-threading environment.
2. Grammar Identity
During validation, grammar is identified through (only) Namespace URI. But in
reality, more than one grammar may have the same Namespace URI, especially when
they are used by different instance documents.
A grammar pool implementation may use the information in Grammar Description,
in addition to the Namespace URI, to uniquely identify a grammar.
A grammar pool implementation also needs to ensure the proper deletion of a grammar
rejected (due to the duplicated grammar identification) to avoid memory leakage,
and it is up to the implemention whether an exception to throw or normal return
is appropriate.
3. Ownership of Grammar Pool
External application owns the grammar pool it creates, and needs to ensure its
persistency through the call to XMLPlatformUtils::Terminate().
4. Ownership of Grammar in Grammar Pool
The objectives of grammar pool is to extend grammar life scope (which is created
and owned by individual parser in the current Xerces), beyond its owning parser,
and therefore be shared among parsers.
With the introduction of plugable memory manager, memory owned by a grammar is
allocated by the parser's specific memory manager, which could possibly no longer
exist by the time when grammar needs to be released from pool.
Thus the grammar pool (and its memory manager) would be the one to create/own grammar.
5. Specification of Grammar Pool on Parser
Parser intended to be attached to a grammar pool, must be explicitly provided with
a grammar pool during its creation. Otherwise a new instance of Xerces default
grammar pool implementation (XMLGrammarPoolImpl) is created when needed, which indeed
is a stateful, parser specific grammar pool (the current behaviour).
No re-specification of grammar pool on parser after creation is allowed.
Design Overview:
---------------------------
1. Provide a grammar pool interface that external applications can implement.
2. Provide a default implementation of grammar pool that is used if external
application does not specify one during parser creation.
3. Provide grammar description interfaces that external application can implement.
4. Provide default implementations of grammar descriptions.
5. Grammar components, such as ElementDecl, NotationDecl, is to be created by their
owning grammar.
6. Allow external applications to plug in their own grammar pool to parser.
7. Grammar/GrammarDescription is to be created by grammar pool.
Design Details:
---------------------------
1. Grammar Pool Interface
public XMLGrammarPool : public XMemory
{
public:
virtual ~XMLGrammarPool(){};
/***
* Virtual interface for grammar pool
*/
virtual void cacheGrammar(const Grammar* const) = 0;
virtual Grammar* retrieveGrammar(const XMLGrammarDescription* const) = 0;
// grammar removed from the pool and owned by the caller
virtual Grammar* orphanGrammar(const XMLGrammarDescription* const) = 0;
virtual void clear() = 0;
virtual void lockPool() = 0;
virtual void unlockPool() = 0;
/***
* Factory methods to create Grammar and GrammarDescription
*/
virtual DTDGrammar* createDTDGrammar() = 0;
virtual SchemaGrammar* createSchemaGrammar() = 0;
virtual XMLDTDDescription* createDTDDescription() = 0;
virtual XMLSchemaDescription* createSchemaDescription() = 0;
/***
* Getter
*/
const MemoryManager* getMemoryManager()
{ return fMemMgr; }
protected:
XMLGrammarPool(MemoryManager* memMgr = XMLPlatformUtils::fgMemoryManager);
/***
* plugged-in memory manager, not owned
*/
MemoryManager* const fMemMgr;
private:
/***
* Unimplemented copy constructor and operator=
*/
XMLGrammarPool(const XMLGrammarPool& );
XMLGrammarPool& operator=(const XMLGrammarPool& );
};
XMLGrammarPool::XMLGrammarPool(MemoryManager* memMgr)
:fMemMgr(memMgr)
{}
2. Xerces Default XMLGrammarPool Implementation
class XMLGrammarPoolImpl : public XMLGrammarPool
{
public:
XMLGrammarPoolImpl(MemoryManager* memMgr);
~XMLGrammarPoolImpl();
virtual void cacheGrammar(const Grammar* const);
virtual Grammar* retrieveGrammar(const XMLGrammarDescription* const);
virtual Grammar* orphanGrammar(const XMLGrammarDescription* const);
virtual void clear();
virtual void lockPool(); // thread-saft mechanism not supported
virtual void unlockPool(); // thread-saft mechanism not supported
/***
* Factory methods to create Grammar and GrammarDescription
*/
virtual DTDGrammar* createDTDGrammar()
{ return new (fMemMgr) DTDGrammar(fMemMgr); }
virutal SchemaGrammar* createSchemaGrammar()
{ return new (fMemMgr) SchemaGrammar(fMemMgr); }
virtual XMLDTDDescription* createDTDDescription()
{ return new (fMemMgr) XMLDTDDescriptionImpl(fMemMgr); }
virtual XMLSchemaDescription* createSchemaDescription()
{ return new (fMemMgr) XMLSchemaDescriptionImpl(fMemMgr); }
private:
/***
* Unimplemented copy constructor and operator=
*/
XMLGrammarPoolImpl(const XMLGrammarPoolImpl& );
XMLGrammarPoolImpl& operator=(const XMLGrammarPoolImpl& );
RefHashTableOf<Grammar>* fGrammarRegistry;
};
XMLGrammarPoolImpl::XMLGrammarPoolImpl(MemoryManager* memMgr)
,fGrammarRegistry(0)
{
fGrammarRegistry = new (fMemMgr) RefHashTableOf<Grammar>(29, true);
}
XMLGrammarPoolImpl::~XMLGrammarPoolImpl()
{
delete fGrammarRegistry;
}
void XMLGrammarPoolImpl:: cacheGrammar(Grammar* const grammar)
{
if (!grammar)
return;
XMLCh* grammarKey = grammar->getDescription()->getGrammarKey();
if (fGrammarRegistry->containsKey(grammarKey))
{
ThrowXML(RuntimeException, XMLExcepts::GC_ExistingGrammar);
}
fGrammarRegistry->put((void*) grammarKey, grammar);
}
Grammar* XMLGrammarPoolImpl:: retrieveGrammar(const XMLGrammarDescript* const desc)
{
if (!desc)
return 0;
return fGrammarRegistry->get(desc->getGrammarKey());
}
Grammar* XMLGrammarPoolImpl:: orphanGrammar(const XMLGrammarDescript* const desc)
{
if (!desc)
return 0;
return fGrammarRegistry->orphanKey(desc->getGrammarKey());
}
GrmmarPoolImpl::clear()
{
fGrammarRegistry->removeAll();
}
3. Grammar Description Interfaces
XMLGrammarDescription encapsulate extra grammar type specific information,
to allow better grammar retrieve/cache operation in grammar pool.
public XMLGrammarDescription
{
public:
virtual ~XMLGrammarDescription(){};
virtual Grammar::GrammarType getGrammarType() = 0;
virtual const XMLCh* const getGrammarKey() = 0;
protected:
XMLGrammarDescription(){};
private:
/***
* Unimplemented copy constructor and operator=
*/
XMLGrammarDescription(const XMLGrammarDescription &);
XMLGrammarDescription& operator=(const XMLGrammarDescription &);
}
public XMLDTDDescription : public XMLGrammarDescription
{
public:
virtual ~XMLDTDDescription(){};
virtual const XMLCh* const getRootName() = 0;
protected:
XMLDTDDescription(){};
private:
/***
* Unimplemented copy constructor and operator=
*/
XMLDTDDescription(const XMLDTDDescription &);
XMLDTDDescription& operator=(const XMLDTDDescription &);
}
typdef const XMLCh* const LocationHint;
public XMLSchemaDescription : public XMLGrammarDescription
{
public:
enum ContextType
{
CONTEXT_INCLUDE;
CONTEXT_REDEFINE;
CONTEXT_IMPORT;
CONTEXT_PREPARSE;
CONTEXT_INSTANCE;
CONTEXT_ELEMENT;
CONTEXT_ATTRIBUTE;
CONTEXT_XSITYPE;
};
virtual short getContextType() = 0;
virtual const XMLCh* const getTargetNamespace() = 0;
virtual LocationHint* getLocationHints() = 0;
virtual const QName* const getTriggeringComponent() = 0;
virtual const QName* const getEnclosingElementName() = 0;
virtual const XMLAttDef* const getAttributes() = 0;
protected:
XMLSchemaDescription(){};
private
/***
* Unimplemented copy constructor and operator=
*/
XMLSchemaDescription(const XMLSchemaDescription &);
XMLSchemaDescription& operator=(const XMLSchemaDescription &);
}
4. Xerces Default Implementation of XMLDTDDescription and XMLSchemaDescription
public XMLDTDDescriptionImpl : public XMLDTDDescription
{
public:
XMLDTDDescriptionImpl();
~XMLDTDDescriptionImpl();
/***
* Implementation of XMLGrammarDescription interface
*/
virtual Grammar::GrammarType getGrammarType()
{ return Grammar::DTDGrammarType; }
virtual const XMLCh* const getGrammarKey()
{ return getRootName(); )
/***
* Implementation of XMLDTDDescription interface
*/
virtual const XMLCh* const getRootName()
{ return fRootName; }
/***
* Setter
*/
void setRootName(const XMLCh* const rootName)
{
if (fRootName)
delete fRootName;
fRootName = rootName;
}
private:
/***
* Unimplemented copy constructor and operator=
*/
XMLDTDDescriptionImpl(const XMLDTDDescriptionImpl &);
XMLDTDDescriptionImpl& operator=(const XMLDTDDescriptionImpl &);
const XMLCh* fRootName;
}
XMLDTDDescriptionImpl::XMLDTDDescriptionImpl()
:fRootName(0)
{}
XMLDTDDescriptionImpl::~XMLDTDDescriptionImpl()
{
if (fRootName)
delete fRootName;
}
public XMLSchemaDescriptionImpl : public XMLSchemaDescription
{
public:
XMLSchemaDescriptionImpl();
~XMLSchemaDescriptionImpl();
/***
* Implementation of XMLGrammarDescription interface
*/
virtual Grammar::GrammarType getGrammarType()
{ return Grammar::SchemaGrammarType; }
virtual const XMLCh* const getGrammarKey()
{ return getTargetNamespace(); )
/***
* Implementation of XMLSchemaDescription interface
*/
virtual short getContextType()
{ return fContextType; }
virtual const XMLCh* const getTargetNamespace()
{ return fNamespace; }
virtual LocationHint* getLocationHints()
{ return fLocationHints; }
virtual const QName* const getTriggeringComponent()
{ return fTriggeringComponent; }
virtual const QName* const getEnclosingElementName()
{ return fEnclosingElementName; }
virtual const XMLAttDef* const getAttributes()
{ return fAttributes; }
/***
* Setter
*/
void setContextType(ContextType type)
{ fContextType = type; }
void setTargetNamespace(const XMLCh* const namespace)
{
if (fNamespace)
delete fNamespace;
fNamespace = namespace;
}
void setLocationHints(RefVectorOf<LocationHint> *hints)
{
if (fLocationHints)
delete fLocationHints;
fLocationHints = hints;
}
void setTriggeringComponent(const QName* const trigComponent)
{
if ( fTriggeringComponent)
delete fTriggeringComponent;
fTriggeringComponent = trigComponent;
}
void setEnclosingElementName(const QName* const encElement)
{
if (fEnclosingElementName)
delete fEnclosingElementName;
fEnclosingElementName = encElement;
}
void setAttributes(XMLAttDefs* const AttDefs)
{
if (fAttributes)
delete fAttributes;
fAttributes = attDefs;
}
private
/***
* Unimplemented copy constructor and operator=
*/
XMLSchemaDescriptionImpl(const XMLSchemaDescriptionImpl &);
XMLSchemaDescriptionImpl& operator=(const XMLSchemaDescriptionImpl &);
short fContextType;
const XMLCh* fNamespace;
RefVectorOf<LocationHint>* fLocationHints;
const QName* fTriggeringComponent;
const QName* fEnclosingElementName;
const XMLAttDefs* fAttributes;
}
XMLSchemaDescriptionImpl::XMLSchemaDescriptionImpl()
:fContextType(0)
,fNamespace(0)
,fLocationHints(0)
,fTriggeringComponent(0)
,fEnclosingElementName(0)
,fAttributes(0)
{
}
XMLSchemaDescriptionImpl::~XMLSchemaDescriptionImpl()
{
delete fNamespace;
delete fLocationHints;
delete fTriggeringComponent;
delete fEnclosingElementName;
delete fAttributes;
}
5. Grammar/DTDGrammar/SchemaGrammar
class Grammar
{
public:
//Factory methods to create various grammar components.
//parameters and overloaded methods to be put in later.
virtual XMLElementDecl* createElementDecl() = 0;
virtual XMLEntityDecl* createNotationDecl() = 0;
virtual XMLNotationDecl* createNotationDecl() = 0;
virtual XMLAttDefs* createAttDefs() = 0;
virtual ComplexTypeInfo* createComplexTypeInfo() = 0;
virtual XercesGroupInfo* createGroupInfo() = 0;
virtual XercesAttGroupInfo* createAttGroupInfo() = 0;
virtual XMLRefInfo* createRefInfo() = 0;
XMLGrammarDescription* const getDescription()
{ return fGramDesc; };
void setDescription(const XMLGrammarDescription* const desc)
{
if (fGramDesc)
delete fGramDesc;
fGramDesc = desc;
}
...
private:
XMLGrammarDescription* fGramDesc; // owned
}
Grammar::Grammar()
:fGramDesc(0)
{}
Grammar::~Grammar()
{
if (fGramDesc)
delete fGramDesc;
}
class DTDGrammar : public Grammar
{
public:
DTDGrammar(MemoryManager*);
virtual XMLElementDecl* createElementDecl()
{ return new (fMemMgr) DTDElementDecl(fMemMgr); };
virtual XMLNotationDecl* createNotationDecl()
{ return new (fMemMgr) DTDNotationDecl(fMemMgr); };
virtual XMLEntityDecl* createEntityDecl()
{ return new (fMemMgr) XMLEntityDecl(fMemMgr); };
virtual XMLAttDefs* createAttDefs()
{ return new (fMemMgr) DTDAttDefs(fMemMgr); };
virtual ComplexTypeInfo* createComplexTypeInfo()
{ ThrowXML(Grammar_Component_Not_Supported); };
virtual XercesGroupInfo* createGroupInfo()
{ ThrowXML(Grammar_Component_Not_Supported); };
virtual XercesAttGroupInfo* createAttGroupInfo()
{ ThrowXML(Grammar_Component_Not_Supported); };
virtual XMLRefInfo* createRefInfo()
{ ThrowXML(Grammar_Component_Not_Supported); };
...
private:
MemoryManager* fMemMgr;
};
class SchemaGrammar : public Grammar
{
public:
SchemaGrammar(MemoryManager*);
virtual XMLElementDecl* createElementDecl()
{ return new (fMemMgr) SchemaElementDecl(fMemMgr); };
virtual XMLNotationDecl* createNotationDecl()
{ return new (fMemMgr) SchemaNotationDecl(fMemMgr); };
virtual XMLEntityDecl* createEntityDecl()
{ return new (fMemMgr) XMLEntityDecl(fMemMgr); };
virtual XMLAttDefs* createAttDefs()
{ return new (fMemMgr) SchemaAttDefs(fMemMgr); };
virtual ComplexTypeInfo* createComplexTypeInfo()
{ return new (fMemMgr) ComplexTypeInfo(fMemMgr); };
virtual XercesGroupInfo* createGroupInfo()
{ return new (fMemMgr) XercesGroupInfo(fMemMgr); };
virtual XercesAttGroupInfo* createAttGroupInfo()
{ return new (fMemMgr) XercesAttGroupInfo(fMemMgr); };
virtual XMLRefInfo* createRefInfo()
{ return new (fMemMgr) RefInfo(fMemMgr); };
...
private:
MemoryManager* fMemMgr;
}
6. Hold instance of grammar pool
One more parameter added to various parsers' constructor(s) (or corresponding factory method),
to allow specifying XMLGrammarPool (default to 0) from external application.
XercesDOMParser
{
public:
XercesDOMParser (XMLValidator* const valToAdopt = 0
, MemoryManager* const memMgr = XMLPlatformUtils::fgMemoryManager
, XMLGrammarPool* const gramPool = 0
);
/***
* Getter
*
* GrammarResolver has the actual grammar pool in use, which is
* either the same one passed to the parser
* or the the one created by the GrammarResolver.
*/
XMLGrammarPool* const getXMLGrammarPool()
{ return fGrammarResolver->getXMLGrammarPool(); }
/***
* Setter
* Not provided, non-immutable grammar pool/parser
*/
...
private:
MemoryManager* const fMemMgr;
GrammarResolver* fGrammarResolver;
XMLGrammarPool* const fGramPool; // not owned
}
XercesDOMParser::XercesDOMParser(XMLValidator* const valToAdopt
, MemoryManager* const memMgr
, XMLGrammarPool* const gramPool
)
: ...
,fMemMgr(memMgr)
,fGramPool(gramPool)
{
}
7. GrammarResolver
class GrammarResolver : public XMemory
{
public:
GrammarResolver(MemoryManager* const memMgr
, XMLGrammarPool* const gramPool = 0);
void GrammarResolver::cacheGrammarFromParse(const bool );
Grammar* getGrammar(const XMLGrammarDescription* const);
Grammar* orphanGrammar(const XMLGrammarDescription* const);
void putGrammar(const XMLGrammarDescription* const
Grammar* const grammarToAdopt );
XMLGrammarPool* getXMLGrammarPool()
{ return fGramPool; }
private:
/***
* helper
*/
void createDefaultXMLGrammarPool()
{
/***
* using the parser specific memory manager to create the grammar pool
*/
if (!fGramPool)
fGramPool = new (fMemMgr) XMLGrammarPoolImpl(fMemMgr);
}
MemoryManger* fMemMgr; // parser specific memory manager
XMLGrammarPool* fGramPool;
bool fGramPoolFromExternalApplication;
RefHashTableOf<Grammar>* fGrammarRegistry; //local grammar bucket
}
GrammarResolver::GrammarResolver(MemoryManager* const memMgr
, XMLGrammarPool* const gramPool)
:fMemMgr(memMgr)
:fGramPool(gramPool)
,fGramPoolFromExternalApplication(true)
{
if (!gramPool)
{
//Lazy instantiation
//we create the actual grammar pool later on until it is needed
fGramPoolFromExternalApplication = false;
}
}
GrammarResolver::~GrammarResolver()
{
if (!fGramPoolFromExternalApplication && fGramPool)
delete fGramPool;
}
void GrammarResolver::cacheGrammarFromParse(const bool aValue)
{
if (aValue && !fGramPool)
{
createDefaultXMLGrammarPool();
}
fCacheGrammar = aValue;
...
}
void GrammarResolver::cacheGrammars()
{
if (!fGramPool)
{
createDefaultXMLGrammarPool();
}
RefHashTableOfEnumerator<Grammar> grammarEnum(fGrammarRegistry);
while (grammarEnum.hasMoreElements()) {
XMLCh* grammarKey = (XMLCh*) grammarEnum.nextElementKey();
Grammar* grammar = fGrammarRegistry->orphanKey(grammarKey);
fGramPool->cacheGrammar(grammar);
}
}
Grammar* GrammarResolver::getGrammar(const XMLGrammarDescription* const desc )
{
if (!desc)
return 0;
XMLCh* grammarKey = desc->getGrammarKey();
Grammar* aGrammar = fGrammarRegistry->get(grammarKey);
if (!aGrammar && fUseCachedGrammar && fGramPool)
{
aGrammar = fGramPool->retrieveGrammar(desc);
}
return aGrammar;
}
void GrammarResolver::putGrammar( const XMLGrammarDescription* const desc
, Grammar* const grammarToAdopt )
{
if (!desc)
return;
if (fCacheGrammar)
fGramPool->cacheGrammar(grammarToAdopt);
fGrammarRegistry->put( (void*) desc->getGrammarKey(), grammarToAdopt );
}
Grammar* GrammarResolver::orphanGrammar(const XMLGrammarDescription* const desc)
{
if (!desc)
return 0;
if (fCacheGrammar)
return fGramPool->orphanGrammar(desc->getGrammarKey());
return fGrammarRegistry->orphanKey(desc->getGrammarKey());
}
8. Scanners and TraverseSchema
a. Container Creation
There are two groups of containers in SchemaGrammar, one is created by the SchemaGrammar
itself, the other by TraverseSchema. For the latter, TraverseSchema needs to create
the container with the grammar pool's Memory Manager.
eg.
current code:
if (fGroupRegistry == 0 )
{
fGroupRegistry = new RefHashTableOf<XercesGroupInfo>(13);
fSchemaGrammar->setGroupInfoRegistry(fGroupRegistry);
}
new code:
if (fGroupRegistry == 0 )
{
MemoryManager* pMemMgr = fGrammarResolver->getXMLGrammarPool()->getMemoryManager();
fGroupRegistry = new (pMemMgr) RefHashTableOf<XercesGroupInfo>(13);
fSchemaGrammar->setGroupInfoRegistry(fGroupRegistry);
}
b. Grammar Component Creation
All grammar component shall be created by its containing grammar.
Sample SchemaElementDecl
current code:
SchemaElementDecl* elemDecl = new SchemaElementDecl(prefix, name, uriIndex,
(SchemaElementDecl::ModelTypes) elemType,
enclosingScope);
new code:
SchemaElementDecl* elemDecl = fSchemaGrammar->createElementDecl(prefix, name, uriIndex,
(SchemaElementDecl::ModelTypes) elemType,
enclosingScope);
Sample DTDElementDecl
current code:
elemDecl = new DTDElementDecl(bbName.getRawBuffer(), fEmptyNamespaceId);
elemDecl->setCreateReason(XMLElementDecl::AttList);
elemDecl->setExternalElemDeclaration(isReadingExternalEntity());
fDTDGrammar->putElemDecl((XMLElementDecl*) elemDecl);
new code:
elemDecl = fDTDGrammar->createElemDecl(bbName.getRawBuffer(), fEmptyNamespaceId);
elemDecl->setCreateReason(XMLElementDecl::AttList);
elemDecl->setExternalElemDeclaration(isReadingExternalEntity());
fDTDGrammar->putElemDecl((XMLElementDecl*) elemDecl);
c. XMLGrammarDescription
To construct XMLGrammarDescription with regard to all grammar operation on
GrammarResolver
Other issues:
-------------------------
Though it is fine for an external application, instead of having its own
grammar pool implementation, plug an instantce of the Xerces default grammar pool
implementation, XMLGrammarPoolImpl, to parser as a shared grammar pool among parsers,
external application shall be aware that this implementation does not provide
thread safety mechanism.
Your comments and suggestions are appreciated
Rgds,
PeiYong