You are viewing a plain text version of this content. The canonical link for it is here.
Posted to c-dev@xerces.apache.org by "Christian Will (JIRA)" <xe...@xml.apache.org> on 2005/03/16 23:16:24 UTC
[jira] Created: (XERCESC-1381) Memory Leak in GrammarResolver class
Memory Leak in GrammarResolver class
------------------------------------
Key: XERCESC-1381
URL: http://issues.apache.org/jira/browse/XERCESC-1381
Project: Xerces-C++
Type: Bug
Components: Validating Parser (Schema) (Xerces 1.5 or up only)
Versions: 2.6.0
Environment: all
Reporter: Christian Will
Attachments: GrammarResolver.cpp.patch
Hi,
the grammar resolver class produces memory leaks under some circumstances. If we set the features fgXercesUseCachedGrammarInParse, fgXercesCacheGrammarFromParse and lock the grammar pool all new created grammars will never be deleted.
Here is my first example. I create a parser, cache the grammar, lock the grammar pool and parse one instance document. I delete the parser object, unlock and delete the grammar pool, and terminate the platform utils. After all objects are deleted we can see that still 3 kbytes are allocated from the grammar pool memory manager.
I could locate all allocations to [\internal\IGXMLScanner2.cpp(1146)].
=> "fDTDGrammar = new (fGrammarPoolMemoryManager) DTDGrammar(fGrammarPoolMemoryManager);"
Sample Log 1
------------
init...
parser1 set feature SchemaFullChecking,UseCachedGrammarInParse,CacheGrammarFromParse
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
Parser One : 26456 [ 26456]bytes, 101 [ 101, 0]blocks (0)
GrammarPool : 1214 [ 1214]bytes, 20 [ 20, 0]blocks (0)
cache grammar... (T:/testfiles/_FIXML/schema/FIXML-main-4-4.xsd)
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
Parser One : 26456 [ 308510]bytes, 101 [ 2885, 4064]blocks (5)
GrammarPool : 4239962 [ 12829334]bytes, 62711 [117134, 70374]blocks (8)
lock grammarpool...
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
Parser One : 26456 [ 308510]bytes, 101 [ 2885, 4064]blocks (0)
GrammarPool : 5501456 [ 12829334]bytes, 97049 [151874, 70776]blocks (0)
---------------------------------------------------------------------------
parse with parser1... (T:/testfiles/_FIXML/order100.xml)
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
Parser One : 50846 [ 308510]bytes, 517 [ 3395, 4370]blocks (40)
GrammarPool : 5504580 [ 12829334]bytes, 97064 [151889, 70776]blocks (4)
---------------------------------------------------------------------------
delete parser1...
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
GrammarPool : 5504580 [ 12829334]bytes, 97064 [151889, 70776]blocks (0)
unlock grammarPool...
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
GrammarPool : 4243086 [ 12829334]bytes, 62726 [151889,105114]blocks (0)
delete grammarPool...
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
GrammarPool : 3020 [ 12829334]bytes, 12 [151889,176419]blocks (0)
terminate PlatformUtils...
GrammarPool : 3020 [ 12829334]bytes, 12 [151889,176419]blocks (0)
done.
==-still-allocated-memory-=================================================
pointer: 3df83c0 size: 44bytes sequence: 151875
T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
T:\xerces_cvs\src\xercesc\internal\XMLScanner.cpp(434)
pointer: 3df86d0 size: 32bytes sequence: 151876
T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(169)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
pointer: 3df87d8 size: 436bytes sequence: 151877
..\..\..\..\..\src\xercesc/util/NameIdPool.c(122)
T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(169)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
pointer: 3df8a70 size: 512bytes sequence: 151878
..\..\..\..\..\src\xercesc/util/NameIdPool.c(136)
T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(169)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
pointer: 3df8d58 size: 32bytes sequence: 151879
T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(173)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
pointer: 3df8e60 size: 436bytes sequence: 151880
..\..\..\..\..\src\xercesc/util/NameIdPool.c(122)
T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(173)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
pointer: 3df90f8 size: 512bytes sequence: 151881
..\..\..\..\..\src\xercesc/util/NameIdPool.c(136)
T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(173)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
pointer: 3df93e0 size: 32bytes sequence: 151882
T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(174)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
pointer: 3df94e8 size: 436bytes sequence: 151883
..\..\..\..\..\src\xercesc/util/NameIdPool.c(122)
T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(174)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
pointer: 3df9780 size: 512bytes sequence: 151884
..\..\..\..\..\src\xercesc/util/NameIdPool.c(136)
T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(174)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
pointer: 3df9a68 size: 24bytes sequence: 151885
T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(177)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
pointer: 3df9b60 size: 12bytes sequence: 151886
..\..\..\..\..\src\xercesc/util/XMLString.hpp(1667)
T:\xerces_cvs\src\xercesc\validators\DTD\XMLDTDDescriptionImpl.cpp(59)
T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(177)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
==-still-allocated-memory-=================================================
My second example shows the impact a little bit better. I lock the grammar pool without loading the grammar. I parse the instance document 3 times. And after I delete all objects and terminate the platform utils I can see that the associated grammar is still 3 times (11.5MB) in memory.
Sample Log 2
------------
init...
parser1 set feature SchemaFullChecking,UseCachedGrammarInParse,CacheGrammarFromParse
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
Parser One : 26456 [ 26456]bytes, 101 [ 101, 0]blocks (0)
GrammarPool : 1214 [ 1214]bytes, 20 [ 20, 0]blocks (0)
lock grammarpool...
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
Parser One : 26456 [ 26456]bytes, 101 [ 101, 0]blocks (0)
GrammarPool : 23934 [ 24098]bytes, 597 [ 629, 32]blocks (0)
---------------------------------------------------------------------------
parse with parser1... (T:/testfiles/_FIXML/order100.xml)
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
Parser One : 50862 [ 486594]bytes, 517 [ 3392, 4374]blocks (40)
GrammarPool : 4265794 [ 12855178]bytes, 63302 [117757, 70406]blocks (5)
parse with parser1... (T:/testfiles/_FIXML/order100.xml)
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
Parser One : 52466 [ 497174]bytes, 586 [ 6317, 8535]blocks (65)
GrammarPool : 8120878 [ 16710262]bytes,115564 [224426,140764]blocks (0)
parse with parser1... (T:/testfiles/_FIXML/order100.xml)
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
Parser One : 54586 [ 498778]bytes, 655 [ 9241, 12695]blocks (144)
GrammarPool : 11975962 [ 20565346]bytes,167826 [331095,211122]blocks (0)
---------------------------------------------------------------------------
delete parser1...
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
GrammarPool : 11975962 [ 20565346]bytes,167826 [331095,211122]blocks (0)
unlock grammarPool...
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
GrammarPool : 11953242 [ 20565346]bytes,167249 [331095,211699]blocks (0)
delete grammarPool...
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
GrammarPool : 11565252 [ 20565346]bytes,156786 [331095,222162]blocks (0)
terminate PlatformUtils...
GrammarPool : 11565252 [ 20565346]bytes,156786 [331095,222162]blocks (0)
Let me try to explain why this is happening. In GrammarResolver::putGrammar() can we see that we put our grammar in the table 'fGrammarBucket' if we don't want to cache our grammar or the grammar pool doesn't want cache our grammar (because it is locked).
void GrammarResolver::putGrammar(Grammar* const grammarToAdopt)
{
...
if (!fCacheGrammar || !fGrammarPool->cacheGrammar(grammarToAdopt))
{
// either we aren't caching or the grammar pool doesn't want it
// so we need to look after it
fGrammarBucket->put( (void*) grammarToAdopt->getGrammarDescription()->getGrammarKey(), grammarToAdopt );
if (grammarToAdopt->getGrammarType() == Grammar::SchemaGrammarType)
{
fGrammarsToAddToXSModel->addElement((SchemaGrammar*) grammarToAdopt);
}
}
}
So far so good. We have all grammars that we don't want to cache or/and can't be cached in this table. So normaly if it deletes the table fGrammarBucket all grammars will be deleted. But if the feature fgXercesCacheGrammarFromParse is set it changes the behavior of the table to not adopte the elements. What we can see in GrammarResolver::cacheGrammarFromParse().
void GrammarResolver::cacheGrammarFromParse(const bool aValue)
{
reset();
fCacheGrammar = aValue;
fGrammarBucket->setAdoptElements(!fCacheGrammar);
}
I think that is wrong to change the property of our table to not adopt the elements. Whenever we remove an grammar from our list, for example to cache to grammar, we call fGrammarBucket->orphanKey(grammarKey), and that means already that we remove the grammar wihtout deleting it.
I removed this line "fGrammarBucket->setAdoptElements(!fCacheGrammar);" made some tests and everything looks good.
But I'm not sure if there are no side effects. And that is the reason why I would like to get a second opinion from somebody. I can also send parts of my code If you have problems to reproduce the bug.
Regards,
Christian Will
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
[jira] Commented: (XERCESC-1381) Memory Leak in GrammarResolver class
Posted by "Christian Will (JIRA)" <xe...@xml.apache.org>.
[ http://issues.apache.org/jira/browse/XERCESC-1381?page=comments#action_61292 ]
Christian Will commented on XERCESC-1381:
-----------------------------------------
Thanks Khaled,
I saw your check-in and everything looks fine. So I guess you can close this bug.
Regards,
Christian
> Memory Leak in GrammarResolver class
> ------------------------------------
>
> Key: XERCESC-1381
> URL: http://issues.apache.org/jira/browse/XERCESC-1381
> Project: Xerces-C++
> Type: Bug
> Components: Validating Parser (Schema) (Xerces 1.5 or up only)
> Versions: 2.6.0
> Environment: all
> Reporter: Christian Will
> Attachments: GrammarResolver.cpp.patch
>
> Hi,
> the grammar resolver class produces memory leaks under some circumstances. If we set the features fgXercesUseCachedGrammarInParse, fgXercesCacheGrammarFromParse and lock the grammar pool all new created grammars will never be deleted.
> Here is my first example. I create a parser, cache the grammar, lock the grammar pool and parse one instance document. I delete the parser object, unlock and delete the grammar pool, and terminate the platform utils. After all objects are deleted we can see that still 3 kbytes are allocated from the grammar pool memory manager.
> I could locate all allocations to [\internal\IGXMLScanner2.cpp(1146)].
> => "fDTDGrammar = new (fGrammarPoolMemoryManager) DTDGrammar(fGrammarPoolMemoryManager);"
> Sample Log 1
> ------------
> init...
> parser1 set feature SchemaFullChecking,UseCachedGrammarInParse,CacheGrammarFromParse
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 26456 [ 26456]bytes, 101 [ 101, 0]blocks (0)
> GrammarPool : 1214 [ 1214]bytes, 20 [ 20, 0]blocks (0)
> cache grammar... (T:/testfiles/_FIXML/schema/FIXML-main-4-4.xsd)
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 26456 [ 308510]bytes, 101 [ 2885, 4064]blocks (5)
> GrammarPool : 4239962 [ 12829334]bytes, 62711 [117134, 70374]blocks (8)
> lock grammarpool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 26456 [ 308510]bytes, 101 [ 2885, 4064]blocks (0)
> GrammarPool : 5501456 [ 12829334]bytes, 97049 [151874, 70776]blocks (0)
> ---------------------------------------------------------------------------
> parse with parser1... (T:/testfiles/_FIXML/order100.xml)
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 50846 [ 308510]bytes, 517 [ 3395, 4370]blocks (40)
> GrammarPool : 5504580 [ 12829334]bytes, 97064 [151889, 70776]blocks (4)
> ---------------------------------------------------------------------------
> delete parser1...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 5504580 [ 12829334]bytes, 97064 [151889, 70776]blocks (0)
> unlock grammarPool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 4243086 [ 12829334]bytes, 62726 [151889,105114]blocks (0)
> delete grammarPool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 3020 [ 12829334]bytes, 12 [151889,176419]blocks (0)
> terminate PlatformUtils...
> GrammarPool : 3020 [ 12829334]bytes, 12 [151889,176419]blocks (0)
> done.
> ==-still-allocated-memory-=================================================
> pointer: 3df83c0 size: 44bytes sequence: 151875
> T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> T:\xerces_cvs\src\xercesc\internal\XMLScanner.cpp(434)
> pointer: 3df86d0 size: 32bytes sequence: 151876
> T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(169)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df87d8 size: 436bytes sequence: 151877
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(122)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(169)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df8a70 size: 512bytes sequence: 151878
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(136)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(169)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df8d58 size: 32bytes sequence: 151879
> T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(173)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df8e60 size: 436bytes sequence: 151880
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(122)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(173)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df90f8 size: 512bytes sequence: 151881
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(136)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(173)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df93e0 size: 32bytes sequence: 151882
> T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(174)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df94e8 size: 436bytes sequence: 151883
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(122)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(174)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df9780 size: 512bytes sequence: 151884
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(136)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(174)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df9a68 size: 24bytes sequence: 151885
> T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(177)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df9b60 size: 12bytes sequence: 151886
> ..\..\..\..\..\src\xercesc/util/XMLString.hpp(1667)
> T:\xerces_cvs\src\xercesc\validators\DTD\XMLDTDDescriptionImpl.cpp(59)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(177)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> ==-still-allocated-memory-=================================================
> My second example shows the impact a little bit better. I lock the grammar pool without loading the grammar. I parse the instance document 3 times. And after I delete all objects and terminate the platform utils I can see that the associated grammar is still 3 times (11.5MB) in memory.
> Sample Log 2
> ------------
> init...
> parser1 set feature SchemaFullChecking,UseCachedGrammarInParse,CacheGrammarFromParse
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 26456 [ 26456]bytes, 101 [ 101, 0]blocks (0)
> GrammarPool : 1214 [ 1214]bytes, 20 [ 20, 0]blocks (0)
> lock grammarpool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 26456 [ 26456]bytes, 101 [ 101, 0]blocks (0)
> GrammarPool : 23934 [ 24098]bytes, 597 [ 629, 32]blocks (0)
> ---------------------------------------------------------------------------
> parse with parser1... (T:/testfiles/_FIXML/order100.xml)
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 50862 [ 486594]bytes, 517 [ 3392, 4374]blocks (40)
> GrammarPool : 4265794 [ 12855178]bytes, 63302 [117757, 70406]blocks (5)
> parse with parser1... (T:/testfiles/_FIXML/order100.xml)
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 52466 [ 497174]bytes, 586 [ 6317, 8535]blocks (65)
> GrammarPool : 8120878 [ 16710262]bytes,115564 [224426,140764]blocks (0)
> parse with parser1... (T:/testfiles/_FIXML/order100.xml)
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 54586 [ 498778]bytes, 655 [ 9241, 12695]blocks (144)
> GrammarPool : 11975962 [ 20565346]bytes,167826 [331095,211122]blocks (0)
> ---------------------------------------------------------------------------
> delete parser1...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 11975962 [ 20565346]bytes,167826 [331095,211122]blocks (0)
> unlock grammarPool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 11953242 [ 20565346]bytes,167249 [331095,211699]blocks (0)
> delete grammarPool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 11565252 [ 20565346]bytes,156786 [331095,222162]blocks (0)
> terminate PlatformUtils...
> GrammarPool : 11565252 [ 20565346]bytes,156786 [331095,222162]blocks (0)
> Let me try to explain why this is happening. In GrammarResolver::putGrammar() can we see that we put our grammar in the table 'fGrammarBucket' if we don't want to cache our grammar or the grammar pool doesn't want cache our grammar (because it is locked).
> void GrammarResolver::putGrammar(Grammar* const grammarToAdopt)
> {
> ...
> if (!fCacheGrammar || !fGrammarPool->cacheGrammar(grammarToAdopt))
> {
> // either we aren't caching or the grammar pool doesn't want it
> // so we need to look after it
> fGrammarBucket->put( (void*) grammarToAdopt->getGrammarDescription()->getGrammarKey(), grammarToAdopt );
> if (grammarToAdopt->getGrammarType() == Grammar::SchemaGrammarType)
> {
> fGrammarsToAddToXSModel->addElement((SchemaGrammar*) grammarToAdopt);
> }
> }
> }
> So far so good. We have all grammars that we don't want to cache or/and can't be cached in this table. So normaly if it deletes the table fGrammarBucket all grammars will be deleted. But if the feature fgXercesCacheGrammarFromParse is set it changes the behavior of the table to not adopte the elements. What we can see in GrammarResolver::cacheGrammarFromParse().
> void GrammarResolver::cacheGrammarFromParse(const bool aValue)
> {
> reset();
> fCacheGrammar = aValue;
> fGrammarBucket->setAdoptElements(!fCacheGrammar);
> }
> I think that is wrong to change the property of our table to not adopt the elements. Whenever we remove an grammar from our list, for example to cache to grammar, we call fGrammarBucket->orphanKey(grammarKey), and that means already that we remove the grammar wihtout deleting it.
> I removed this line "fGrammarBucket->setAdoptElements(!fCacheGrammar);" made some tests and everything looks good.
> But I'm not sure if there are no side effects. And that is the reason why I would like to get a second opinion from somebody. I can also send parts of my code If you have problems to reproduce the bug.
> Regards,
> Christian Will
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
[jira] Resolved: (XERCESC-1381) Memory Leak in GrammarResolver class
Posted by "Khaled Noaman (JIRA)" <xe...@xml.apache.org>.
[ http://issues.apache.org/jira/browse/XERCESC-1381?page=history ]
Khaled Noaman resolved XERCESC-1381:
------------------------------------
Resolution: Fixed
Hi Christian,
I've committed your fix. Would you please verify? I've looked in the code for potential side effects, and I think that your fix is valid. When we cache grammars, we remove it from the grammar resolver table, so I agree with you that we should not change the table not to adopt elements.
Regards,
Khaled
> Memory Leak in GrammarResolver class
> ------------------------------------
>
> Key: XERCESC-1381
> URL: http://issues.apache.org/jira/browse/XERCESC-1381
> Project: Xerces-C++
> Type: Bug
> Components: Validating Parser (Schema) (Xerces 1.5 or up only)
> Versions: 2.6.0
> Environment: all
> Reporter: Christian Will
> Attachments: GrammarResolver.cpp.patch
>
> Hi,
> the grammar resolver class produces memory leaks under some circumstances. If we set the features fgXercesUseCachedGrammarInParse, fgXercesCacheGrammarFromParse and lock the grammar pool all new created grammars will never be deleted.
> Here is my first example. I create a parser, cache the grammar, lock the grammar pool and parse one instance document. I delete the parser object, unlock and delete the grammar pool, and terminate the platform utils. After all objects are deleted we can see that still 3 kbytes are allocated from the grammar pool memory manager.
> I could locate all allocations to [\internal\IGXMLScanner2.cpp(1146)].
> => "fDTDGrammar = new (fGrammarPoolMemoryManager) DTDGrammar(fGrammarPoolMemoryManager);"
> Sample Log 1
> ------------
> init...
> parser1 set feature SchemaFullChecking,UseCachedGrammarInParse,CacheGrammarFromParse
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 26456 [ 26456]bytes, 101 [ 101, 0]blocks (0)
> GrammarPool : 1214 [ 1214]bytes, 20 [ 20, 0]blocks (0)
> cache grammar... (T:/testfiles/_FIXML/schema/FIXML-main-4-4.xsd)
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 26456 [ 308510]bytes, 101 [ 2885, 4064]blocks (5)
> GrammarPool : 4239962 [ 12829334]bytes, 62711 [117134, 70374]blocks (8)
> lock grammarpool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 26456 [ 308510]bytes, 101 [ 2885, 4064]blocks (0)
> GrammarPool : 5501456 [ 12829334]bytes, 97049 [151874, 70776]blocks (0)
> ---------------------------------------------------------------------------
> parse with parser1... (T:/testfiles/_FIXML/order100.xml)
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 50846 [ 308510]bytes, 517 [ 3395, 4370]blocks (40)
> GrammarPool : 5504580 [ 12829334]bytes, 97064 [151889, 70776]blocks (4)
> ---------------------------------------------------------------------------
> delete parser1...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 5504580 [ 12829334]bytes, 97064 [151889, 70776]blocks (0)
> unlock grammarPool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 4243086 [ 12829334]bytes, 62726 [151889,105114]blocks (0)
> delete grammarPool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 3020 [ 12829334]bytes, 12 [151889,176419]blocks (0)
> terminate PlatformUtils...
> GrammarPool : 3020 [ 12829334]bytes, 12 [151889,176419]blocks (0)
> done.
> ==-still-allocated-memory-=================================================
> pointer: 3df83c0 size: 44bytes sequence: 151875
> T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> T:\xerces_cvs\src\xercesc\internal\XMLScanner.cpp(434)
> pointer: 3df86d0 size: 32bytes sequence: 151876
> T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(169)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df87d8 size: 436bytes sequence: 151877
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(122)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(169)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df8a70 size: 512bytes sequence: 151878
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(136)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(169)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df8d58 size: 32bytes sequence: 151879
> T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(173)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df8e60 size: 436bytes sequence: 151880
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(122)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(173)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df90f8 size: 512bytes sequence: 151881
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(136)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(173)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df93e0 size: 32bytes sequence: 151882
> T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(174)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df94e8 size: 436bytes sequence: 151883
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(122)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(174)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df9780 size: 512bytes sequence: 151884
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(136)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(174)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df9a68 size: 24bytes sequence: 151885
> T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(177)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df9b60 size: 12bytes sequence: 151886
> ..\..\..\..\..\src\xercesc/util/XMLString.hpp(1667)
> T:\xerces_cvs\src\xercesc\validators\DTD\XMLDTDDescriptionImpl.cpp(59)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(177)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> ==-still-allocated-memory-=================================================
> My second example shows the impact a little bit better. I lock the grammar pool without loading the grammar. I parse the instance document 3 times. And after I delete all objects and terminate the platform utils I can see that the associated grammar is still 3 times (11.5MB) in memory.
> Sample Log 2
> ------------
> init...
> parser1 set feature SchemaFullChecking,UseCachedGrammarInParse,CacheGrammarFromParse
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 26456 [ 26456]bytes, 101 [ 101, 0]blocks (0)
> GrammarPool : 1214 [ 1214]bytes, 20 [ 20, 0]blocks (0)
> lock grammarpool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 26456 [ 26456]bytes, 101 [ 101, 0]blocks (0)
> GrammarPool : 23934 [ 24098]bytes, 597 [ 629, 32]blocks (0)
> ---------------------------------------------------------------------------
> parse with parser1... (T:/testfiles/_FIXML/order100.xml)
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 50862 [ 486594]bytes, 517 [ 3392, 4374]blocks (40)
> GrammarPool : 4265794 [ 12855178]bytes, 63302 [117757, 70406]blocks (5)
> parse with parser1... (T:/testfiles/_FIXML/order100.xml)
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 52466 [ 497174]bytes, 586 [ 6317, 8535]blocks (65)
> GrammarPool : 8120878 [ 16710262]bytes,115564 [224426,140764]blocks (0)
> parse with parser1... (T:/testfiles/_FIXML/order100.xml)
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 54586 [ 498778]bytes, 655 [ 9241, 12695]blocks (144)
> GrammarPool : 11975962 [ 20565346]bytes,167826 [331095,211122]blocks (0)
> ---------------------------------------------------------------------------
> delete parser1...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 11975962 [ 20565346]bytes,167826 [331095,211122]blocks (0)
> unlock grammarPool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 11953242 [ 20565346]bytes,167249 [331095,211699]blocks (0)
> delete grammarPool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 11565252 [ 20565346]bytes,156786 [331095,222162]blocks (0)
> terminate PlatformUtils...
> GrammarPool : 11565252 [ 20565346]bytes,156786 [331095,222162]blocks (0)
> Let me try to explain why this is happening. In GrammarResolver::putGrammar() can we see that we put our grammar in the table 'fGrammarBucket' if we don't want to cache our grammar or the grammar pool doesn't want cache our grammar (because it is locked).
> void GrammarResolver::putGrammar(Grammar* const grammarToAdopt)
> {
> ...
> if (!fCacheGrammar || !fGrammarPool->cacheGrammar(grammarToAdopt))
> {
> // either we aren't caching or the grammar pool doesn't want it
> // so we need to look after it
> fGrammarBucket->put( (void*) grammarToAdopt->getGrammarDescription()->getGrammarKey(), grammarToAdopt );
> if (grammarToAdopt->getGrammarType() == Grammar::SchemaGrammarType)
> {
> fGrammarsToAddToXSModel->addElement((SchemaGrammar*) grammarToAdopt);
> }
> }
> }
> So far so good. We have all grammars that we don't want to cache or/and can't be cached in this table. So normaly if it deletes the table fGrammarBucket all grammars will be deleted. But if the feature fgXercesCacheGrammarFromParse is set it changes the behavior of the table to not adopte the elements. What we can see in GrammarResolver::cacheGrammarFromParse().
> void GrammarResolver::cacheGrammarFromParse(const bool aValue)
> {
> reset();
> fCacheGrammar = aValue;
> fGrammarBucket->setAdoptElements(!fCacheGrammar);
> }
> I think that is wrong to change the property of our table to not adopt the elements. Whenever we remove an grammar from our list, for example to cache to grammar, we call fGrammarBucket->orphanKey(grammarKey), and that means already that we remove the grammar wihtout deleting it.
> I removed this line "fGrammarBucket->setAdoptElements(!fCacheGrammar);" made some tests and everything looks good.
> But I'm not sure if there are no side effects. And that is the reason why I would like to get a second opinion from somebody. I can also send parts of my code If you have problems to reproduce the bug.
> Regards,
> Christian Will
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
[jira] Updated: (XERCESC-1381) Memory Leak in GrammarResolver class
Posted by "Christian Will (JIRA)" <xe...@xml.apache.org>.
[ http://issues.apache.org/jira/browse/XERCESC-1381?page=history ]
Christian Will updated XERCESC-1381:
------------------------------------
Attachment: GrammarResolver.cpp.patch
my changes...
> Memory Leak in GrammarResolver class
> ------------------------------------
>
> Key: XERCESC-1381
> URL: http://issues.apache.org/jira/browse/XERCESC-1381
> Project: Xerces-C++
> Type: Bug
> Components: Validating Parser (Schema) (Xerces 1.5 or up only)
> Versions: 2.6.0
> Environment: all
> Reporter: Christian Will
> Attachments: GrammarResolver.cpp.patch
>
> Hi,
> the grammar resolver class produces memory leaks under some circumstances. If we set the features fgXercesUseCachedGrammarInParse, fgXercesCacheGrammarFromParse and lock the grammar pool all new created grammars will never be deleted.
> Here is my first example. I create a parser, cache the grammar, lock the grammar pool and parse one instance document. I delete the parser object, unlock and delete the grammar pool, and terminate the platform utils. After all objects are deleted we can see that still 3 kbytes are allocated from the grammar pool memory manager.
> I could locate all allocations to [\internal\IGXMLScanner2.cpp(1146)].
> => "fDTDGrammar = new (fGrammarPoolMemoryManager) DTDGrammar(fGrammarPoolMemoryManager);"
> Sample Log 1
> ------------
> init...
> parser1 set feature SchemaFullChecking,UseCachedGrammarInParse,CacheGrammarFromParse
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 26456 [ 26456]bytes, 101 [ 101, 0]blocks (0)
> GrammarPool : 1214 [ 1214]bytes, 20 [ 20, 0]blocks (0)
> cache grammar... (T:/testfiles/_FIXML/schema/FIXML-main-4-4.xsd)
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 26456 [ 308510]bytes, 101 [ 2885, 4064]blocks (5)
> GrammarPool : 4239962 [ 12829334]bytes, 62711 [117134, 70374]blocks (8)
> lock grammarpool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 26456 [ 308510]bytes, 101 [ 2885, 4064]blocks (0)
> GrammarPool : 5501456 [ 12829334]bytes, 97049 [151874, 70776]blocks (0)
> ---------------------------------------------------------------------------
> parse with parser1... (T:/testfiles/_FIXML/order100.xml)
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 50846 [ 308510]bytes, 517 [ 3395, 4370]blocks (40)
> GrammarPool : 5504580 [ 12829334]bytes, 97064 [151889, 70776]blocks (4)
> ---------------------------------------------------------------------------
> delete parser1...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 5504580 [ 12829334]bytes, 97064 [151889, 70776]blocks (0)
> unlock grammarPool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 4243086 [ 12829334]bytes, 62726 [151889,105114]blocks (0)
> delete grammarPool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 3020 [ 12829334]bytes, 12 [151889,176419]blocks (0)
> terminate PlatformUtils...
> GrammarPool : 3020 [ 12829334]bytes, 12 [151889,176419]blocks (0)
> done.
> ==-still-allocated-memory-=================================================
> pointer: 3df83c0 size: 44bytes sequence: 151875
> T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> T:\xerces_cvs\src\xercesc\internal\XMLScanner.cpp(434)
> pointer: 3df86d0 size: 32bytes sequence: 151876
> T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(169)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df87d8 size: 436bytes sequence: 151877
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(122)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(169)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df8a70 size: 512bytes sequence: 151878
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(136)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(169)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df8d58 size: 32bytes sequence: 151879
> T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(173)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df8e60 size: 436bytes sequence: 151880
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(122)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(173)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df90f8 size: 512bytes sequence: 151881
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(136)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(173)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df93e0 size: 32bytes sequence: 151882
> T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(174)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df94e8 size: 436bytes sequence: 151883
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(122)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(174)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df9780 size: 512bytes sequence: 151884
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(136)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(174)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df9a68 size: 24bytes sequence: 151885
> T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(177)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df9b60 size: 12bytes sequence: 151886
> ..\..\..\..\..\src\xercesc/util/XMLString.hpp(1667)
> T:\xerces_cvs\src\xercesc\validators\DTD\XMLDTDDescriptionImpl.cpp(59)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(177)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> ==-still-allocated-memory-=================================================
> My second example shows the impact a little bit better. I lock the grammar pool without loading the grammar. I parse the instance document 3 times. And after I delete all objects and terminate the platform utils I can see that the associated grammar is still 3 times (11.5MB) in memory.
> Sample Log 2
> ------------
> init...
> parser1 set feature SchemaFullChecking,UseCachedGrammarInParse,CacheGrammarFromParse
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 26456 [ 26456]bytes, 101 [ 101, 0]blocks (0)
> GrammarPool : 1214 [ 1214]bytes, 20 [ 20, 0]blocks (0)
> lock grammarpool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 26456 [ 26456]bytes, 101 [ 101, 0]blocks (0)
> GrammarPool : 23934 [ 24098]bytes, 597 [ 629, 32]blocks (0)
> ---------------------------------------------------------------------------
> parse with parser1... (T:/testfiles/_FIXML/order100.xml)
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 50862 [ 486594]bytes, 517 [ 3392, 4374]blocks (40)
> GrammarPool : 4265794 [ 12855178]bytes, 63302 [117757, 70406]blocks (5)
> parse with parser1... (T:/testfiles/_FIXML/order100.xml)
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 52466 [ 497174]bytes, 586 [ 6317, 8535]blocks (65)
> GrammarPool : 8120878 [ 16710262]bytes,115564 [224426,140764]blocks (0)
> parse with parser1... (T:/testfiles/_FIXML/order100.xml)
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 54586 [ 498778]bytes, 655 [ 9241, 12695]blocks (144)
> GrammarPool : 11975962 [ 20565346]bytes,167826 [331095,211122]blocks (0)
> ---------------------------------------------------------------------------
> delete parser1...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 11975962 [ 20565346]bytes,167826 [331095,211122]blocks (0)
> unlock grammarPool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 11953242 [ 20565346]bytes,167249 [331095,211699]blocks (0)
> delete grammarPool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 11565252 [ 20565346]bytes,156786 [331095,222162]blocks (0)
> terminate PlatformUtils...
> GrammarPool : 11565252 [ 20565346]bytes,156786 [331095,222162]blocks (0)
> Let me try to explain why this is happening. In GrammarResolver::putGrammar() can we see that we put our grammar in the table 'fGrammarBucket' if we don't want to cache our grammar or the grammar pool doesn't want cache our grammar (because it is locked).
> void GrammarResolver::putGrammar(Grammar* const grammarToAdopt)
> {
> ...
> if (!fCacheGrammar || !fGrammarPool->cacheGrammar(grammarToAdopt))
> {
> // either we aren't caching or the grammar pool doesn't want it
> // so we need to look after it
> fGrammarBucket->put( (void*) grammarToAdopt->getGrammarDescription()->getGrammarKey(), grammarToAdopt );
> if (grammarToAdopt->getGrammarType() == Grammar::SchemaGrammarType)
> {
> fGrammarsToAddToXSModel->addElement((SchemaGrammar*) grammarToAdopt);
> }
> }
> }
> So far so good. We have all grammars that we don't want to cache or/and can't be cached in this table. So normaly if it deletes the table fGrammarBucket all grammars will be deleted. But if the feature fgXercesCacheGrammarFromParse is set it changes the behavior of the table to not adopte the elements. What we can see in GrammarResolver::cacheGrammarFromParse().
> void GrammarResolver::cacheGrammarFromParse(const bool aValue)
> {
> reset();
> fCacheGrammar = aValue;
> fGrammarBucket->setAdoptElements(!fCacheGrammar);
> }
> I think that is wrong to change the property of our table to not adopt the elements. Whenever we remove an grammar from our list, for example to cache to grammar, we call fGrammarBucket->orphanKey(grammarKey), and that means already that we remove the grammar wihtout deleting it.
> I removed this line "fGrammarBucket->setAdoptElements(!fCacheGrammar);" made some tests and everything looks good.
> But I'm not sure if there are no side effects. And that is the reason why I would like to get a second opinion from somebody. I can also send parts of my code If you have problems to reproduce the bug.
> Regards,
> Christian Will
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
[jira] Closed: (XERCESC-1381) Memory Leak in GrammarResolver class
Posted by "Khaled Noaman (JIRA)" <xe...@xml.apache.org>.
[ http://issues.apache.org/jira/browse/XERCESC-1381?page=history ]
Khaled Noaman closed XERCESC-1381:
----------------------------------
> Memory Leak in GrammarResolver class
> ------------------------------------
>
> Key: XERCESC-1381
> URL: http://issues.apache.org/jira/browse/XERCESC-1381
> Project: Xerces-C++
> Type: Bug
> Components: Validating Parser (Schema) (Xerces 1.5 or up only)
> Versions: 2.6.0
> Environment: all
> Reporter: Christian Will
> Attachments: GrammarResolver.cpp.patch
>
> Hi,
> the grammar resolver class produces memory leaks under some circumstances. If we set the features fgXercesUseCachedGrammarInParse, fgXercesCacheGrammarFromParse and lock the grammar pool all new created grammars will never be deleted.
> Here is my first example. I create a parser, cache the grammar, lock the grammar pool and parse one instance document. I delete the parser object, unlock and delete the grammar pool, and terminate the platform utils. After all objects are deleted we can see that still 3 kbytes are allocated from the grammar pool memory manager.
> I could locate all allocations to [\internal\IGXMLScanner2.cpp(1146)].
> => "fDTDGrammar = new (fGrammarPoolMemoryManager) DTDGrammar(fGrammarPoolMemoryManager);"
> Sample Log 1
> ------------
> init...
> parser1 set feature SchemaFullChecking,UseCachedGrammarInParse,CacheGrammarFromParse
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 26456 [ 26456]bytes, 101 [ 101, 0]blocks (0)
> GrammarPool : 1214 [ 1214]bytes, 20 [ 20, 0]blocks (0)
> cache grammar... (T:/testfiles/_FIXML/schema/FIXML-main-4-4.xsd)
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 26456 [ 308510]bytes, 101 [ 2885, 4064]blocks (5)
> GrammarPool : 4239962 [ 12829334]bytes, 62711 [117134, 70374]blocks (8)
> lock grammarpool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 26456 [ 308510]bytes, 101 [ 2885, 4064]blocks (0)
> GrammarPool : 5501456 [ 12829334]bytes, 97049 [151874, 70776]blocks (0)
> ---------------------------------------------------------------------------
> parse with parser1... (T:/testfiles/_FIXML/order100.xml)
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 50846 [ 308510]bytes, 517 [ 3395, 4370]blocks (40)
> GrammarPool : 5504580 [ 12829334]bytes, 97064 [151889, 70776]blocks (4)
> ---------------------------------------------------------------------------
> delete parser1...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 5504580 [ 12829334]bytes, 97064 [151889, 70776]blocks (0)
> unlock grammarPool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 4243086 [ 12829334]bytes, 62726 [151889,105114]blocks (0)
> delete grammarPool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 3020 [ 12829334]bytes, 12 [151889,176419]blocks (0)
> terminate PlatformUtils...
> GrammarPool : 3020 [ 12829334]bytes, 12 [151889,176419]blocks (0)
> done.
> ==-still-allocated-memory-=================================================
> pointer: 3df83c0 size: 44bytes sequence: 151875
> T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> T:\xerces_cvs\src\xercesc\internal\XMLScanner.cpp(434)
> pointer: 3df86d0 size: 32bytes sequence: 151876
> T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(169)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df87d8 size: 436bytes sequence: 151877
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(122)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(169)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df8a70 size: 512bytes sequence: 151878
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(136)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(169)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df8d58 size: 32bytes sequence: 151879
> T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(173)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df8e60 size: 436bytes sequence: 151880
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(122)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(173)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df90f8 size: 512bytes sequence: 151881
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(136)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(173)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df93e0 size: 32bytes sequence: 151882
> T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(174)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df94e8 size: 436bytes sequence: 151883
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(122)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(174)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df9780 size: 512bytes sequence: 151884
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(136)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(174)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df9a68 size: 24bytes sequence: 151885
> T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(177)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df9b60 size: 12bytes sequence: 151886
> ..\..\..\..\..\src\xercesc/util/XMLString.hpp(1667)
> T:\xerces_cvs\src\xercesc\validators\DTD\XMLDTDDescriptionImpl.cpp(59)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(177)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> ==-still-allocated-memory-=================================================
> My second example shows the impact a little bit better. I lock the grammar pool without loading the grammar. I parse the instance document 3 times. And after I delete all objects and terminate the platform utils I can see that the associated grammar is still 3 times (11.5MB) in memory.
> Sample Log 2
> ------------
> init...
> parser1 set feature SchemaFullChecking,UseCachedGrammarInParse,CacheGrammarFromParse
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 26456 [ 26456]bytes, 101 [ 101, 0]blocks (0)
> GrammarPool : 1214 [ 1214]bytes, 20 [ 20, 0]blocks (0)
> lock grammarpool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 26456 [ 26456]bytes, 101 [ 101, 0]blocks (0)
> GrammarPool : 23934 [ 24098]bytes, 597 [ 629, 32]blocks (0)
> ---------------------------------------------------------------------------
> parse with parser1... (T:/testfiles/_FIXML/order100.xml)
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 50862 [ 486594]bytes, 517 [ 3392, 4374]blocks (40)
> GrammarPool : 4265794 [ 12855178]bytes, 63302 [117757, 70406]blocks (5)
> parse with parser1... (T:/testfiles/_FIXML/order100.xml)
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 52466 [ 497174]bytes, 586 [ 6317, 8535]blocks (65)
> GrammarPool : 8120878 [ 16710262]bytes,115564 [224426,140764]blocks (0)
> parse with parser1... (T:/testfiles/_FIXML/order100.xml)
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 54586 [ 498778]bytes, 655 [ 9241, 12695]blocks (144)
> GrammarPool : 11975962 [ 20565346]bytes,167826 [331095,211122]blocks (0)
> ---------------------------------------------------------------------------
> delete parser1...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 11975962 [ 20565346]bytes,167826 [331095,211122]blocks (0)
> unlock grammarPool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 11953242 [ 20565346]bytes,167249 [331095,211699]blocks (0)
> delete grammarPool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 11565252 [ 20565346]bytes,156786 [331095,222162]blocks (0)
> terminate PlatformUtils...
> GrammarPool : 11565252 [ 20565346]bytes,156786 [331095,222162]blocks (0)
> Let me try to explain why this is happening. In GrammarResolver::putGrammar() can we see that we put our grammar in the table 'fGrammarBucket' if we don't want to cache our grammar or the grammar pool doesn't want cache our grammar (because it is locked).
> void GrammarResolver::putGrammar(Grammar* const grammarToAdopt)
> {
> ...
> if (!fCacheGrammar || !fGrammarPool->cacheGrammar(grammarToAdopt))
> {
> // either we aren't caching or the grammar pool doesn't want it
> // so we need to look after it
> fGrammarBucket->put( (void*) grammarToAdopt->getGrammarDescription()->getGrammarKey(), grammarToAdopt );
> if (grammarToAdopt->getGrammarType() == Grammar::SchemaGrammarType)
> {
> fGrammarsToAddToXSModel->addElement((SchemaGrammar*) grammarToAdopt);
> }
> }
> }
> So far so good. We have all grammars that we don't want to cache or/and can't be cached in this table. So normaly if it deletes the table fGrammarBucket all grammars will be deleted. But if the feature fgXercesCacheGrammarFromParse is set it changes the behavior of the table to not adopte the elements. What we can see in GrammarResolver::cacheGrammarFromParse().
> void GrammarResolver::cacheGrammarFromParse(const bool aValue)
> {
> reset();
> fCacheGrammar = aValue;
> fGrammarBucket->setAdoptElements(!fCacheGrammar);
> }
> I think that is wrong to change the property of our table to not adopt the elements. Whenever we remove an grammar from our list, for example to cache to grammar, we call fGrammarBucket->orphanKey(grammarKey), and that means already that we remove the grammar wihtout deleting it.
> I removed this line "fGrammarBucket->setAdoptElements(!fCacheGrammar);" made some tests and everything looks good.
> But I'm not sure if there are no side effects. And that is the reason why I would like to get a second opinion from somebody. I can also send parts of my code If you have problems to reproduce the bug.
> Regards,
> Christian Will
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-c-dev-help@xml.apache.org