Problem when importing huge files

[Sridhar Raman <sr...@gmail.com>]

I have 25000 thousand records in a file.  I tried importing 5000 in one
shot, it happened quickly.  The same case for 10000 and 15000.  So I thought
there wouldn't be a problem, and I attempted to import the 25000 in one go.
The importXML command is executed properly (I got this from the log), but
the session.save() seems to go on forever.  After a few hours, the file to
which I am redirected the import output, becomes 4gb in size!!!  I checked
the workspace folder, and saw that the size of this was also around the same
(3.9gb).

Any idea what could be the cause of this problem?

Thanks,
Sridhar

Re: Problem when importing huge files

[Stefan Guggisberg <st...@gmail.com>]

On 3/11/07, Sridhar Raman <sr...@gmail.com> wrote:
> I have 25000 thousand records in a file.  I tried importing 5000 in one
> shot, it happened quickly.  The same case for 10000 and 15000.  So I thought
> there wouldn't be a problem, and I attempted to import the 25000 in one go.
> The importXML command is executed properly (I got this from the log), but
> the session.save() seems to go on forever.  After a few hours, the file to

try increasing your jvm heap, seems like your jvm's gc got extremely busy
(just a wild guess of course).

i further suggest to use Workspace.importXML rather than Session.importXML
since it circumvents the transient space, i.e. lower memory consumption
and you don't have to call Session.save at the end.

cheers
stefan

> which I am redirected the import output, becomes 4gb in size!!!  I checked
> the workspace folder, and saw that the size of this was also around the same
> (3.9gb).
>
> Any idea what could be the cause of this problem?
>
> Thanks,
> Sridhar
>