You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by Cservenak Tamas <cs...@is-micro.hu> on 2005/11/22 13:38:29 UTC
[transaction] OutOfMemory error on continous work with lot of new files
Hi!
I am using Commons Tx to handle a lot of small files in transactional
way with long term runs (weeks or days at least). New files arrives at
cca 100000 piece/hour. These files leaves the app at the end of
processing. We have noticed, that during work, the Commons Tx eates up
all heap memory (not confirmed yet, but I'm 90% sure about this
statement)...
Some investigation (profiling) pointed out, that in GenericLockManager,
the globalLocks Map just grows, and nobody removes lock entries. At
least lock instances multiples like viruses :)
I am aware that the supplied patch is probalby WRONG, but i am hoping to
reduce memory leak (is it leak?) even at some lock handling overhead
costs (the lock manager is now always reinserting locks).
Keep in mind, that a lot of new files enters and LEAVES my app, and will
never return to it. So, (at least in my case) the reuse of locks is not
as important as memory footprint reduction.
Thanx
~t~
Index:
/home/cstamas/worx/projects/ext/eclipse/commons-transaction/src/java/org/apache/commons/transaction/locking/GenericLockManager.java
===================================================================
---
/home/cstamas/worx/projects/ext/eclipse/commons-transaction/src/java/org/apache/commons/transaction/locking/GenericLockManager.java
(revision 348140)
+++
/home/cstamas/worx/projects/ext/eclipse/commons-transaction/src/java/org/apache/commons/transaction/locking/GenericLockManager.java
(working copy)
@@ -307,6 +307,8 @@
if (lock != null) {
released = lock.release(ownerId);
removeOwner(ownerId, lock);
+ // XXX to shrink the globalLocks map
+ removeLock(lock);
}
return released;
}
@@ -336,6 +338,8 @@
GenericLock lock = (GenericLock) it.next();
lock.release(ownerId);
locks.remove(lock);
+ // XXX to shrink the globalLocks map
+ removeLock(lock);
}
}
}
Re: [transaction] OutOfMemory error on continous work with lot of new files
Posted by Oliver Zeigermann <ol...@gmail.com>.
Hi Tamas.
You are right. All the locks remain inside the GenericLockManager
remain there until you call removeLock. That's intentionally.
If you call removeLock everytime an owner releases a lock, no other
owner will be able to get access to this lock using the
GenericLockManager. So you patch isn't actually a good idea.
A better approach to your problem is to remove all locks from the
manager that no longer have an owner. If a lock has an owner can be
checked using getLockLevel.
As generally lock managers are rather small (aren't they?) I would
rather not include this procedure into the release methods of
GenericLockManager, but would advise you to add this to your custom
code. I am open to convincing arguments (combined with a patch) to add
it to the GenericLockManager, though ;)
Oliver
2005/11/22, Cservenak Tamas <cs...@is-micro.hu>:
> Hi!
>
> I am using Commons Tx to handle a lot of small files in transactional
> way with long term runs (weeks or days at least). New files arrives at
> cca 100000 piece/hour. These files leaves the app at the end of
> processing. We have noticed, that during work, the Commons Tx eates up
> all heap memory (not confirmed yet, but I'm 90% sure about this
> statement)...
>
> Some investigation (profiling) pointed out, that in GenericLockManager,
> the globalLocks Map just grows, and nobody removes lock entries. At
> least lock instances multiples like viruses :)
>
> I am aware that the supplied patch is probalby WRONG, but i am hoping to
> reduce memory leak (is it leak?) even at some lock handling overhead
> costs (the lock manager is now always reinserting locks).
>
> Keep in mind, that a lot of new files enters and LEAVES my app, and will
> never return to it. So, (at least in my case) the reuse of locks is not
> as important as memory footprint reduction.
>
>
> Thanx
> ~t~
>
>
> Index:
> /home/cstamas/worx/projects/ext/eclipse/commons-transaction/src/java/org/apache/commons/transaction/locking/GenericLockManager.java
> ===================================================================
> ---
> /home/cstamas/worx/projects/ext/eclipse/commons-transaction/src/java/org/apache/commons/transaction/locking/GenericLockManager.java
> (revision 348140)
> +++
> /home/cstamas/worx/projects/ext/eclipse/commons-transaction/src/java/org/apache/commons/transaction/locking/GenericLockManager.java
> (working copy)
> @@ -307,6 +307,8 @@
> if (lock != null) {
> released = lock.release(ownerId);
> removeOwner(ownerId, lock);
> + // XXX to shrink the globalLocks map
> + removeLock(lock);
> }
> return released;
> }
> @@ -336,6 +338,8 @@
> GenericLock lock = (GenericLock) it.next();
> lock.release(ownerId);
> locks.remove(lock);
> + // XXX to shrink the globalLocks map
> + removeLock(lock);
> }
> }
> }
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-dev-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org