Extensive memory usage

[Ludwig Magnusson <lu...@itcatapult.com>]

Hi!

I am using torque in one of my projects to manage the database. Torque uses
velocity to generate sql-files for the data to be inserted in the database.
For those not familiar with torque it takes (in my case) an xml-file, like
this one:

 

<dataset name="all">

  <Role Id="0" Name="Role_0"/>

  <Role Id="1" Name="Role_1"/>

  .

  <Role Id="n" Name="Role_n"/>

</dataset>

 

And creates an sql-file like this one:

 

INSERT INTO role (id,name)

    VALUES (1,'Role_1');

 

INSERT INTO role (id,name)

    VALUES (2,'Role_2');

.

 

INSERT INTO role (id,name)

    VALUES (n,'Role_n');

 

A problem I encountered was that when working with large files (but normal
size for describing databases) was that the program threw an
java.lang.OutOfMemoryError. It turns out that for an input xml-file of 60Kb,
that generates an output file of 100 Kb, the program used 50Mb of memory.
The amount of memory used was proportional to the size of the input file.
For my actual database file, which currently has a size of ca 700kb, the
program needs 415Mb of memory. It seems to me that a great efficiency
improvement could be done here. And even though we have access to a lot of
memory, extending the allowed amount of memory for the program doesn't seem
like a long term solution if the database grows.

We did some tracking using Java Visual VM, which confirms memory usage
constantly increasing while running the program. The velocity engine creates
a lot of instances of org.apache.velocity.runtime.parser.Token that are
never garbage collected. It also creates a lot of strings and chars, but
they, or some of them, are destroyed when running the GC. I'm not very
familiar with the velocity code and I don't know if all the instances of the
token class are needed or if they can be destroyed over time. That said, one
can of course not say that the token class definitely is the reason for the
crash.

I have tried this with velocity 1.5 and 1.6.2. It said in the release notes
that velocity 1.6 contained a major performance improvement but I could not
notice any difference in this case.

 

Does anyone have any ideas about this?

/Ludwig