You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@camel.apache.org by "Glenn Moss (JIRA)" <ji...@apache.org> on 2010/10/19 01:20:40 UTC
[jira] Created: (CAMEL-3249) HawtDB file grows indefinitely over
time
HawtDB file grows indefinitely over time
----------------------------------------
Key: CAMEL-3249
URL: https://issues.apache.org/activemq/browse/CAMEL-3249
Project: Apache Camel
Issue Type: Bug
Components: camel-hawtdb
Affects Versions: 2.5.0
Environment: Using 2.5-SNAPSHOT
Reporter: Glenn Moss
When using an aggregator with the HawtDB persistent store, the file grows indefinitely until it fills the volume on which it's stored. This prevents the persistence from being used in any kind of long-running production deployment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CAMEL-3249) HawtDB file grows indefinitely over
time
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen updated CAMEL-3249:
-------------------------------
Fix Version/s: (was: 2.6.0)
2.5.0
Hiram is working on a fix so we should try to get it fixed in Camel 2.5. eg a new release of hawtDB that camel-hawtdb uses.
> HawtDB file grows indefinitely over time
> ----------------------------------------
>
> Key: CAMEL-3249
> URL: https://issues.apache.org/activemq/browse/CAMEL-3249
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-hawtdb
> Affects Versions: 2.3.0, 2.4.0, 2.5.0
> Environment: Using 2.5-SNAPSHOT
> Reporter: Glenn Moss
> Assignee: Hiram Chirino
> Fix For: 2.5.0
>
>
> When using an aggregator with the HawtDB persistent store, the file grows indefinitely until it fills the volume on which it's stored. This prevents the persistence from being used in any kind of long-running production deployment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CAMEL-3249) HawtDB file grows indefinitely over
time
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62673#action_62673 ]
Claus Ibsen commented on CAMEL-3249:
------------------------------------
Can you post how you have setup this?
> HawtDB file grows indefinitely over time
> ----------------------------------------
>
> Key: CAMEL-3249
> URL: https://issues.apache.org/activemq/browse/CAMEL-3249
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-hawtdb
> Affects Versions: 2.5.0
> Environment: Using 2.5-SNAPSHOT
> Reporter: Glenn Moss
>
> When using an aggregator with the HawtDB persistent store, the file grows indefinitely until it fills the volume on which it's stored. This prevents the persistence from being used in any kind of long-running production deployment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CAMEL-3249) HawtDB file grows indefinitely over
time
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62707#action_62707 ]
Claus Ibsen commented on CAMEL-3249:
------------------------------------
Please leave all details its important.
What version of HawtDB are you using?
What JDK are you using?
> HawtDB file grows indefinitely over time
> ----------------------------------------
>
> Key: CAMEL-3249
> URL: https://issues.apache.org/activemq/browse/CAMEL-3249
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-hawtdb
> Affects Versions: 2.5.0
> Environment: Using 2.5-SNAPSHOT
> Reporter: Glenn Moss
> Assignee: Claus Ibsen
> Fix For: 2.5.0
>
>
> When using an aggregator with the HawtDB persistent store, the file grows indefinitely until it fills the volume on which it's stored. This prevents the persistence from being used in any kind of long-running production deployment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CAMEL-3249) HawtDB file grows indefinitely over
time
Posted by "Glenn Moss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62710#action_62710 ]
Glenn Moss commented on CAMEL-3249:
-----------------------------------
We're using HawtDB 1.4 and JDK 1.6. (I don't have the exact version number as I don't have access to the machine I was using for testing right now.)
> HawtDB file grows indefinitely over time
> ----------------------------------------
>
> Key: CAMEL-3249
> URL: https://issues.apache.org/activemq/browse/CAMEL-3249
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-hawtdb
> Affects Versions: 2.5.0
> Environment: Using 2.5-SNAPSHOT
> Reporter: Glenn Moss
> Assignee: Claus Ibsen
> Fix For: 2.5.0
>
>
> When using an aggregator with the HawtDB persistent store, the file grows indefinitely until it fills the volume on which it's stored. This prevents the persistence from being used in any kind of long-running production deployment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CAMEL-3249) HawtDB file grows indefinitely over
time
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62679#action_62679 ]
Claus Ibsen commented on CAMEL-3249:
------------------------------------
Ah your problem is that you do not have any completion conditions. So data is never gone be sent out of the aggregator, and hence why the hawtdb file keeps growing.
You need to add 1 .. n completion conditions. See more at
http://camel.apache.org/aggregator2
For example a completionTimeout or completionSize etc.
Also Camel in Action chapter 8 covers the aggregator pattern and how to use it with HawtDB
> HawtDB file grows indefinitely over time
> ----------------------------------------
>
> Key: CAMEL-3249
> URL: https://issues.apache.org/activemq/browse/CAMEL-3249
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-hawtdb
> Affects Versions: 2.5.0
> Environment: Using 2.5-SNAPSHOT
> Reporter: Glenn Moss
>
> When using an aggregator with the HawtDB persistent store, the file grows indefinitely until it fills the volume on which it's stored. This prevents the persistence from being used in any kind of long-running production deployment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CAMEL-3249) HawtDB file grows indefinitely over
time
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen resolved CAMEL-3249.
--------------------------------
Resolution: Fixed
trunk: 1026081.
Gleen HawtDB 1.5 has been released which has the fix. So you should be able to upgrade and use that version.
Thanks for reporting.
Its already in central maven repo
> HawtDB file grows indefinitely over time
> ----------------------------------------
>
> Key: CAMEL-3249
> URL: https://issues.apache.org/activemq/browse/CAMEL-3249
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-hawtdb
> Affects Versions: 2.3.0, 2.4.0, 2.5.0
> Environment: Using 2.5-SNAPSHOT
> Reporter: Glenn Moss
> Assignee: Hiram Chirino
> Fix For: 2.5.0
>
>
> When using an aggregator with the HawtDB persistent store, the file grows indefinitely until it fills the volume on which it's stored. This prevents the persistence from being used in any kind of long-running production deployment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CAMEL-3249) HawtDB file grows indefinitely over
time
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62737#action_62737 ]
Claus Ibsen commented on CAMEL-3249:
------------------------------------
Also remember to update the release notes and camel-hawtdb that v1.5 is required due the bug in older releases
> HawtDB file grows indefinitely over time
> ----------------------------------------
>
> Key: CAMEL-3249
> URL: https://issues.apache.org/activemq/browse/CAMEL-3249
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-hawtdb
> Affects Versions: 2.3.0, 2.4.0, 2.5.0
> Environment: Using 2.5-SNAPSHOT
> Reporter: Glenn Moss
> Assignee: Hiram Chirino
> Fix For: 2.5.0
>
>
> When using an aggregator with the HawtDB persistent store, the file grows indefinitely until it fills the volume on which it's stored. This prevents the persistence from being used in any kind of long-running production deployment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CAMEL-3249) HawtDB file grows indefinitely over
time
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen updated CAMEL-3249:
-------------------------------
Affects Version/s: 2.3.0
2.4.0
Fix Version/s: (was: 2.5.0)
2.6.0
> HawtDB file grows indefinitely over time
> ----------------------------------------
>
> Key: CAMEL-3249
> URL: https://issues.apache.org/activemq/browse/CAMEL-3249
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-hawtdb
> Affects Versions: 2.3.0, 2.4.0, 2.5.0
> Environment: Using 2.5-SNAPSHOT
> Reporter: Glenn Moss
> Assignee: Claus Ibsen
> Fix For: 2.6.0
>
>
> When using an aggregator with the HawtDB persistent store, the file grows indefinitely until it fills the volume on which it's stored. This prevents the persistence from being used in any kind of long-running production deployment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CAMEL-3249) HawtDB file grows indefinitely over
time
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen reassigned CAMEL-3249:
----------------------------------
Assignee: Hiram Chirino (was: Claus Ibsen)
Hiram can you take a look.
It looks like HawtDB doesn't register the old data as free.
I have created an unit test in camel-hawtdb
org.apache.camel.component.hawtdb.HawtDBGrowIssueTest
> HawtDB file grows indefinitely over time
> ----------------------------------------
>
> Key: CAMEL-3249
> URL: https://issues.apache.org/activemq/browse/CAMEL-3249
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-hawtdb
> Affects Versions: 2.3.0, 2.4.0, 2.5.0
> Environment: Using 2.5-SNAPSHOT
> Reporter: Glenn Moss
> Assignee: Hiram Chirino
> Fix For: 2.6.0
>
>
> When using an aggregator with the HawtDB persistent store, the file grows indefinitely until it fills the volume on which it's stored. This prevents the persistence from being used in any kind of long-running production deployment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CAMEL-3249) HawtDB file grows indefinitely over
time
Posted by "Glenn Moss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62783#action_62783 ]
Glenn Moss commented on CAMEL-3249:
-----------------------------------
Thanks for your work on this. I'm currently out if the office for a while, but I'm excited to try it out once I'm back.
> HawtDB file grows indefinitely over time
> ----------------------------------------
>
> Key: CAMEL-3249
> URL: https://issues.apache.org/activemq/browse/CAMEL-3249
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-hawtdb
> Affects Versions: 2.3.0, 2.4.0, 2.5.0
> Environment: Using 2.5-SNAPSHOT
> Reporter: Glenn Moss
> Assignee: Hiram Chirino
> Fix For: 2.5.0
>
>
> When using an aggregator with the HawtDB persistent store, the file grows indefinitely until it fills the volume on which it's stored. This prevents the persistence from being used in any kind of long-running production deployment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CAMEL-3249) HawtDB file grows indefinitely over
time
Posted by "Glenn Moss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62677#action_62677 ]
Glenn Moss commented on CAMEL-3249:
-----------------------------------
Here's how I've used the aggregator:
{code}
from("activemq:source")
.aggregate(header("some-id"), new MyAggregationStrategy())
.aggregationRepository(new HawtDBAggregationRepository("repo", "hawtdbfile"))
.to("activemq:destination");
{code}
I've replicated the behaviour with only HawtDB. By opening a new HawtDB file, and only putting in data of a fixed size into the identical key, the file size grows indefinitely, depending on the size of the data. In the following code, the breakpoint appears to be at 489 characters of data. The file size remains constant with 488 character string, but grows indefinitely with a 489-character string. I think this probably has something to do with how the size of the data relates to the page size (set on TxPageFileFactory).
{code}
public class App {
public static void main (String[] args) {
String filename = args[0];
int dataSize = Integer.parseInt(args[1]);
BTreeIndexFactory<String, String> indexFact =
new BTreeIndexFactory<String, String>();
indexFact.setKeyCodec(StringCodec.INSTANCE);
indexFact.setValueCodec(StringCodec.INSTANCE);
indexFact.setDeferredEncoding(true);
TxPageFileFactory pageFact = new TxPageFileFactory();
pageFact.setFile(new File(filename));
pageFact.setPageSize((short)512);
pageFact.setMappingSegementSize(8 * 1024 * 1024);
pageFact.setSync(true);
pageFact.open();
TxPageFile pageFile = pageFact.getTxPageFile();
Transaction tx = pageFile.tx();
if (!tx.allocator().isAllocated(0)) {
SortedIndex<String, String> created = indexFact.create(tx);
}
tx.commit();
pageFile.flush();
String data = "";
for (int i = 0; i < dataSize; i++) {
data += "0";
}
try {
while (true) {
tx = pageFile.tx();
SortedIndex<String, String> index = indexFact.open(tx);
index.put("foo", data);
tx.commit();
pageFile.flush();
}
} finally {
pageFact.close();
}
}
}
{code}
> HawtDB file grows indefinitely over time
> ----------------------------------------
>
> Key: CAMEL-3249
> URL: https://issues.apache.org/activemq/browse/CAMEL-3249
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-hawtdb
> Affects Versions: 2.5.0
> Environment: Using 2.5-SNAPSHOT
> Reporter: Glenn Moss
>
> When using an aggregator with the HawtDB persistent store, the file grows indefinitely until it fills the volume on which it's stored. This prevents the persistence from being used in any kind of long-running production deployment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CAMEL-3249) HawtDB file grows indefinitely over
time
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen resolved CAMEL-3249.
--------------------------------
Resolution: Working as Designed
Fix Version/s: 2.5.0
Assignee: Claus Ibsen
> HawtDB file grows indefinitely over time
> ----------------------------------------
>
> Key: CAMEL-3249
> URL: https://issues.apache.org/activemq/browse/CAMEL-3249
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-hawtdb
> Affects Versions: 2.5.0
> Environment: Using 2.5-SNAPSHOT
> Reporter: Glenn Moss
> Assignee: Claus Ibsen
> Fix For: 2.5.0
>
>
> When using an aggregator with the HawtDB persistent store, the file grows indefinitely until it fills the volume on which it's stored. This prevents the persistence from being used in any kind of long-running production deployment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CAMEL-3249) HawtDB file grows indefinitely over
time
Posted by "Hiram Chirino (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62734#action_62734 ]
Hiram Chirino commented on CAMEL-3249:
--------------------------------------
Thanks for the test case. I found an allocation leak in HawtDB. I've got it fixed in the latest snapshot now and am working on spinning a new release so you guys can pick the fix up.
> HawtDB file grows indefinitely over time
> ----------------------------------------
>
> Key: CAMEL-3249
> URL: https://issues.apache.org/activemq/browse/CAMEL-3249
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-hawtdb
> Affects Versions: 2.3.0, 2.4.0, 2.5.0
> Environment: Using 2.5-SNAPSHOT
> Reporter: Glenn Moss
> Assignee: Hiram Chirino
> Fix For: 2.5.0
>
>
> When using an aggregator with the HawtDB persistent store, the file grows indefinitely until it fills the volume on which it's stored. This prevents the persistence from being used in any kind of long-running production deployment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (CAMEL-3249) HawtDB file grows indefinitely over
time
Posted by "Glenn Moss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Glenn Moss reopened CAMEL-3249:
-------------------------------
Sorry, in sanitizing the code, I did leave the completion condition out, but be assured that I do have one. As I detailed in the remainder of my comment, the problem is reproducible with only HawtDB code, and appears to not be dependent on the aggregator component.
> HawtDB file grows indefinitely over time
> ----------------------------------------
>
> Key: CAMEL-3249
> URL: https://issues.apache.org/activemq/browse/CAMEL-3249
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-hawtdb
> Affects Versions: 2.5.0
> Environment: Using 2.5-SNAPSHOT
> Reporter: Glenn Moss
> Assignee: Claus Ibsen
> Fix For: 2.5.0
>
>
> When using an aggregator with the HawtDB persistent store, the file grows indefinitely until it fills the volume on which it's stored. This prevents the persistence from being used in any kind of long-running production deployment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.