You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jiansheng Huang (JIRA)" <ji...@apache.org> on 2009/05/29 18:50:45 UTC
[jira] Created: (CASSANDRA-208) jvm crashes intermittently during
compaction
jvm crashes intermittently during compaction
--------------------------------------------
Key: CASSANDRA-208
URL: https://issues.apache.org/jira/browse/CASSANDRA-208
Project: Cassandra
Issue Type: Bug
Affects Versions: trunk
Environment: arch: x86_64
os: Linux version 2.6.18-92.1.22.el5
java: nio2-ea-bin-b99-linux-x64-05_feb_2009
Reporter: Jiansheng Huang
jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717309#action_12717309 ]
Jonathan Ellis commented on CASSANDRA-208:
------------------------------------------
do those tests patch after applying just 01 and 02?
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt, 0004-fix-ColumnReader-add-tests.patch, 0005-fix-test-order-dependent-failures.patch
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-208:
-------------------------------------
Attachment: 0003-split-sstable-into-data-index-and-bloom-filter-files.txt
0002-r-m-touch-code-that-populates-a-cache-that-is-never.txt
0001-CASSANDRA-208-cleanup.txt
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.3
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch-code-that-populates-a-cache-that-is-never.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CASSANDRA-208) OOM intermittently
during compaction
Posted by "Jiansheng Huang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716359#action_12716359 ]
Jiansheng Huang edited comment on CASSANDRA-208 at 6/4/09 11:28 AM:
--------------------------------------------------------------------
To answer Jonathan's questions: (1) we have tested around several million keys, (2) avg row size ~ 2k, max row size ~ 10k. (3) tried several Xmx settings, max tried is 5G. Before compaction starts, pretty much most of the heap is free. I think the problem is easy to run into if the system is run with continuous traffic for over a week or so given that our test dataset has been fairly small.
was (Author: jihuang):
To answer Jonathan's questions: (1) we have tested around several million keys, (2) avg row size ~ 2k, max row size ~ 10k. (3) tried several Xmx settings, max tried is 5G. Before compaction starts, pretty much most of the heap is free. I think the problem is easy to run into if the system is run with continuous traffic for over a week or so.
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch-code-that-populates-a-cache-that-is-never.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CASSANDRA-208) OOM intermittently
during compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717309#action_12717309 ]
Jonathan Ellis edited comment on CASSANDRA-208 at 6/8/09 9:19 AM:
------------------------------------------------------------------
do those tests pass after applying just 01 and 02?
was (Author: jbellis):
do those tests patch after applying just 01 and 02?
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt, 0004-fix-ColumnReader-add-tests.patch, 0005-fix-test-order-dependent-failures.patch
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716453#action_12716453 ]
Jun Rao commented on CASSANDRA-208:
-----------------------------------
Some comments:
1. In SSTable.loadIndexFile, indexEntries could be empty if there are fewer than indexInterval keys. I don't see code dealing with empty indexEntries. A simple solution is to add the last index key to indexEntries when indexEntries is empty. Similarly, you have to fix the same case when an SSTable is first created. You have to remember the last index key being appended.
2. In SSTable.next, need to deal with the case that getPosition() returns -1.
3. In SSTable.getColumnGroupReader, change getNearestPostion() to getPostion(). Similar to 2, need to deal with returned position being -1.
Also, we need to remember opening another issue to get rid of partitioner from KeyPosition.
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Michael Greene (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Greene updated CASSANDRA-208:
-------------------------------------
Component/s: Core
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt, 0004-fix-ColumnReader-add-tests.patch, 0005-fix-test-order-dependent-failures.patch, 208-2.patch, 208-3.patch, 208.patch
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CASSANDRA-208) OOM intermittently
during compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717453#action_12717453 ]
Jonathan Ellis edited comment on CASSANDRA-208 at 6/8/09 3:12 PM:
------------------------------------------------------------------
208-2 fixes syncing of the bloom filter files. this fixes the compactions failure and all the others except timesort, which is not actually a new bug.
CASSANDRA-223 shows that the timesort failure is caused by a bug in TimeFilter that dates back to the FB import.
was (Author: jbellis):
208-2 fixes syncing of the bloom filter files. this fixes the compactions failure.
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt, 0004-fix-ColumnReader-add-tests.patch, 0005-fix-test-order-dependent-failures.patch, 208-2.patch, 208.patch
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717466#action_12717466 ]
Jun Rao commented on CASSANDRA-208:
-----------------------------------
Ok, now all tests pass except timesort.
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt, 0004-fix-ColumnReader-add-tests.patch, 0005-fix-test-order-dependent-failures.patch, 208-2.patch, 208.patch
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717377#action_12717377 ]
Jonathan Ellis commented on CASSANDRA-208:
------------------------------------------
committed 01 to avoid rebase hell. (02 was merged via CASSANDRA-222 from 0.3.)
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt, 0004-fix-ColumnReader-add-tests.patch, 0005-fix-test-order-dependent-failures.patch
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-208:
-------------------------------------
Attachment: 208.patch
208.patch integrates the old 03, 04, and 05 for convenience
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt, 0004-fix-ColumnReader-add-tests.patch, 0005-fix-test-order-dependent-failures.patch, 208-2.patch, 208.patch
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-208:
-------------------------------------
Attachment: 0005-fix-test-order-dependent-failures.patch
I did have NameSortTest and TimeSortTest failing because they were extending CFST and hence running those tests as well, which caused problems since those sets of tests are supposed to be run independently. Making NST and TST not inherit from CFST (which was only done for convenience) fixes that. Here is the patch.
I haven't seen compactions et al ever fail, no.
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt, 0004-fix-ColumnReader-add-tests.patch, 0005-fix-test-order-dependent-failures.patch
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716476#action_12716476 ]
Jun Rao commented on CASSANDRA-208:
-----------------------------------
3. That's true, but at the column level. At the row level, get_slice_from only cares about exact row match.
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) jvm crashes intermittently during
compaction
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714521#action_12714521 ]
Jun Rao commented on CASSANDRA-208:
-----------------------------------
In your data set, do you have a large row? Today, during compaction, a full row may have to be buffered in memory. See https://issues.apache.org/jira/browse/CASSANDRA-16 for more on this.
> jvm crashes intermittently during compaction
> --------------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716299#action_12716299 ]
Jonathan Ellis commented on CASSANDRA-208:
------------------------------------------
Jiansheng: can you give us more details about your data set? We are trying to figure out how likely it is for others to run into this, leaning towards moving this patch to trunk/0.4.
The relevant data are
- how many keys in the CF that fails to compact
- average / max row size (column size x count) in that CF
- jvm memory settings, and how much of the heap is free (according to e.g. jconsole) before compaction starts
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.3
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch-code-that-populates-a-cache-that-is-never.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) jvm crashes intermittently during
compaction
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715554#action_12715554 ]
Jun Rao commented on CASSANDRA-208:
-----------------------------------
When HBase was using the old format, rows are stored as <key,value> pairs in MapFile, where a key is rowkey:columnName:timestamp. The keys for every 128 rows are promoted to the row index. The benefit is that it's simple. There is only a single-level index (compared with row index and column index within a row in cassandra) and it can be used to efficiently look up a full row, a column within a row, or a version of a column in a row. On the other hand, if you make the index dense, the row keys are duplicated for columns within the same row.
> jvm crashes intermittently during compaction
> --------------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.3
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-208:
-------------------------------------
Attachment: (was: 0003-split-sstable-into-data-index-and-bloom-filter-files.txt)
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716369#action_12716369 ]
Jun Rao commented on CASSANDRA-208:
-----------------------------------
Jiansheng, what's the average key size of a row?
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch-code-that-populates-a-cache-that-is-never.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-208:
-------------------------------------
Attachment: 208-2.patch
208-2 fixes syncing of the bloom filter files. this fixes the compactions failure.
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt, 0004-fix-ColumnReader-add-tests.patch, 0005-fix-test-order-dependent-failures.patch, 208-2.patch, 208.patch
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-208:
-------------------------------------
Attachment: 0003-split-sstable-into-data-index-and-bloom-filter-files.txt
0002-r-m-touch.txt
0001-CASSANDRA-208-cleanup.txt
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716073#action_12716073 ]
Jonathan Ellis commented on CASSANDRA-208:
------------------------------------------
03
split sstable into data, index, and bloom filter files
02
r/m 'touch' code that populates a cache that is never used (and never updated on compaction either, so it's buggy too)
01
cleanup. mostly just moving code around so its position makes more sense to me
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.3
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch-code-that-populates-a-cache-that-is-never.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CASSANDRA-208) jvm crashes intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-208:
----------------------------------------
Assignee: Jonathan Ellis
> jvm crashes intermittently during compaction
> --------------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.3
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716477#action_12716477 ]
Jonathan Ellis commented on CASSANDRA-208:
------------------------------------------
Ah, thanks. I got mixed up. :)
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) jvm crashes intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715334#action_12715334 ]
Jonathan Ellis commented on CASSANDRA-208:
------------------------------------------
Another related link: http://wiki.apache.org/hadoop/Hbase/NewFileFormat
My take is that the designs are different enough that their reasons for moving to a single file don't really apply to cassandra.
- the old MapFile has a bunch of properties that make it general enough for Hadoop core but inefficient for hbase (e.g. storing the CF name once per key, keys appearing multiple times in the index)
- they only index one key per block, so their index is much much smaller than ours, and they can get away with storing the index at the end of the file as cassandra currently does
> Even if we take out the row index and BF, data is still mixed with column index.
Not at the SSTable key/value level. To sstable the value is just byte[] so the fact that CF serializes with indexes is an implementation detail. (To the degree that SSTable or SF does care, that is an encapsulation violation -- one of the reasons this code is one of the less pleasant parts of cassandra to work in.)
I will get a patch together that will implement the file splitting I proposed and we will see how that looks. I think that's going to get us to a stable 0.3 fastest; if we want to radically re-think how indexing works (so we can go back to index-at-the-end-of-one-file) then I think that is a change to make in 0.4. (The non-sparse index Cassandra uses may be necessary if you want to support large CF rows, or you will waste too much time scanning through those rows looking for keys when you only get to within 128 keys from the index.)
One thing that piqued my curiosity: what is the hbase "row index?" Looks like their "key index" is like our sstable indexes (with the difference mentioned above, that it only indexes one key per block).
> jvm crashes intermittently during compaction
> --------------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.3
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jiansheng Huang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716375#action_12716375 ]
Jiansheng Huang commented on CASSANDRA-208:
-------------------------------------------
Jun, our key size is about 20 characters.
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch-code-that-populates-a-cache-that-is-never.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-208:
-------------------------------------
Summary: OOM intermittently during compaction (was: jvm crashes intermittently during compaction)
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.3
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716485#action_12716485 ]
Jonathan Ellis commented on CASSANDRA-208:
------------------------------------------
2. is handled by FileReader.next being a no-op if position == -1
3. attached 04 patch to fix, and add more tests. Also, makes sliceFrom able to handle "" as start column.
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716346#action_12716346 ]
Jun Rao commented on CASSANDRA-208:
-----------------------------------
Just browsed through the patch for now and found the following bug.
In SSTable.afterAppend, the position added to indexEntries should be the position in the index file, not the position in the data file.
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.3
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch-code-that-populates-a-cache-that-is-never.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716399#action_12716399 ]
Jonathan Ellis commented on CASSANDRA-208:
------------------------------------------
You're forgetting the BlockMetadata longs (extra 16 bytes per key) and Map overhead (32 bytes per key or more), which would be 880MB for 10M keys.
Still smaller than 5G, though. I guess we can get Jiansheng to test after applying the patch. :)
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch-code-that-populates-a-cache-that-is-never.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717664#action_12717664 ]
Hudson commented on CASSANDRA-208:
----------------------------------
Integrated in Cassandra #103 (See [http://hudson.zones.apache.org/hudson/job/Cassandra/103/])
cleanup SSTable-related code. patch by jbellis; reviewed by Jun Rao for
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt, 0004-fix-ColumnReader-add-tests.patch, 0005-fix-test-order-dependent-failures.patch, 208-2.patch, 208.patch
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-208) jvm crashes intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-208:
-------------------------------------
Fix Version/s: 0.3
Priority: Critical (was: Major)
> jvm crashes intermittently during compaction
> --------------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Priority: Critical
> Fix For: 0.3
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-208:
-------------------------------------
Fix Version/s: (was: 0.3)
0.4
Affects Version/s: (was: trunk)
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch-code-that-populates-a-cache-that-is-never.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716430#action_12716430 ]
Jonathan Ellis commented on CASSANDRA-208:
------------------------------------------
removed patches against 0.3 (to avoid confusion, i still have copies should they become necessary).
added patches against trunk.
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) jvm crashes intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714635#action_12714635 ]
Jonathan Ellis commented on CASSANDRA-208:
------------------------------------------
blockindexes_ keeps the indexes in memory as the flush or compaction is performed, so they can be dumped at the end of the SSTable file.
The old code on code.google dumped each index at the end of its block which avoided this problem (only one block's index would ever be in memory at once). I'm not sure why FB changed this behavior. The only reason I can think of is that reading the index later won't require as many seeks, but this seems like a bad optimization to make since index reading is done once per sstable.
I couldn't tell at first why this should cause OOM -- in both the google and apache versions we have indexMetadataMap_ which contains the same block index info, right? Wrong, the difference is that iMM only contains the first entry from each block index. So much less memory is used there.
I guess we should switch back to the old, interleaved block indexing method.
> jvm crashes intermittently during compaction
> --------------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) jvm crashes intermittently during
compaction
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715322#action_12715322 ]
Jun Rao commented on CASSANDRA-208:
-----------------------------------
Even if we take out the row index and BF, data is still mixed with column index.
Interestingly, HBase's sstable format started with MapFile (1 file for index, 1 file for BF and 1 file for data) and has recently moved to the TFile (a spec can be found at https://issues.apache.org/jira/browse/HADOOP-3315) format (1 file for everything). Although HBase probably did that mainly for reducing # of files (problem for HDFS master), it's probably worth while to take a look at their new design (https://issues.apache.org/jira/browse/HBASE-61).
> jvm crashes intermittently during compaction
> --------------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.3
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) jvm crashes intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715329#action_12715329 ]
Jonathan Ellis commented on CASSANDRA-208:
------------------------------------------
Thanks for the links, that is interesting context.
> jvm crashes intermittently during compaction
> --------------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.3
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717339#action_12717339 ]
Jun Rao commented on CASSANDRA-208:
-----------------------------------
The tests pass with just 01 and 02. They start failing with 03.
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt, 0004-fix-ColumnReader-add-tests.patch, 0005-fix-test-order-dependent-failures.patch
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) jvm crashes intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715301#action_12715301 ]
Jonathan Ellis commented on CASSANDRA-208:
------------------------------------------
IMO we need to address the hard-coded-"keys" problem sooner or later, and having separate files is really the only sane solution.
> jvm crashes intermittently during compaction
> --------------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.3
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716650#action_12716650 ]
Jun Rao commented on CASSANDRA-208:
-----------------------------------
Were you able to run unit tests successfully? I saw a bunch of failures in tests like CompactionsTest and NameSortTest.
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt, 0004-fix-ColumnReader-add-tests.patch
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jiansheng Huang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716359#action_12716359 ]
Jiansheng Huang commented on CASSANDRA-208:
-------------------------------------------
To answer Jonathan's questions: (1) we have tested around several million keys, (2) avg row size ~ 2k, max row size ~ 10k. (3) tried several Xmx settings, max tried is 5G. Before compaction starts, pretty much most of the heap is free. I think the problem is easy to run into if the system is run with continuous traffic for over a week or so.
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch-code-that-populates-a-cache-that-is-never.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-208:
-------------------------------------
Attachment: (was: 0002-r-m-touch-code-that-populates-a-cache-that-is-never.txt)
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716472#action_12716472 ]
Jonathan Ellis commented on CASSANDRA-208:
------------------------------------------
1. empty memtables are not flushed, so there will be at least one key, and the first key will always be in indexEntries since 0 % interval == 0 for any interval. lots of the tests have only one row. I don't think there is a problem here.
2. ok
3. Hmm, if you have columns B C and D in a row, and you do a slice_from starting with A, don't you want B C D to be returned? getNearestPosition will start the scan at B, but vanilla getPosition will return -1 if there is not an exact match.
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716101#action_12716101 ]
Jun Rao commented on CASSANDRA-208:
-----------------------------------
just realize this is off 0.3 branch, never mind.
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.3
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch-code-that-populates-a-cache-that-is-never.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-208:
-------------------------------------
Attachment: (was: 0001-CASSANDRA-208-cleanup.txt)
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-208:
-------------------------------------
Attachment: 208-3.patch
208-3 patches TST to only test what it was intended to test, not CASSANDRA-223.
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt, 0004-fix-ColumnReader-add-tests.patch, 0005-fix-test-order-dependent-failures.patch, 208-2.patch, 208-3.patch, 208.patch
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jiansheng Huang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716452#action_12716452 ]
Jiansheng Huang commented on CASSANDRA-208:
-------------------------------------------
Ok, then scale up the number of keys by 5 times, you will run into the problem. :)
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718068#action_12718068 ]
Hudson commented on CASSANDRA-208:
----------------------------------
Integrated in Cassandra #104 (See [http://hudson.zones.apache.org/hudson/job/Cassandra/104/])
apply rows atomically, rather than one-column-at-a-time. this avoids exposing the bug in time-sorted
columns discussed in #223.
patch by jbellis; reviewed by Jun Rao for
split sstable into data, index, and bloom filter files. this allows us to avoid saving up index chunks
in memory until the sstable data is completely written, while still allowing fast scanning of the index
on startup. it also simplifies the sstable/sequencefile code considerably.
patch by jbellis; reviewed by Jun Rao for
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt, 0004-fix-ColumnReader-add-tests.patch, 0005-fix-test-order-dependent-failures.patch, 208-2.patch, 208-3.patch, 208.patch
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717758#action_12717758 ]
Jun Rao commented on CASSANDRA-208:
-----------------------------------
Patch 208-3 looks fine. All tests pass.
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt, 0004-fix-ColumnReader-add-tests.patch, 0005-fix-test-order-dependent-failures.patch, 208-2.patch, 208-3.patch, 208.patch
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716387#action_12716387 ]
Jun Rao commented on CASSANDRA-208:
-----------------------------------
Hmm, with a key size of 40 bytes, even with 10 million keys, the space needed for all row index entries is about 400MB. That's way smaller than 5G. So, not sure whether the new SSTable format really solves this particular OOME problem.
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch-code-that-populates-a-cache-that-is-never.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716099#action_12716099 ]
Jun Rao commented on CASSANDRA-208:
-----------------------------------
can't apply patch to trunk. could you rebase?
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.3
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch-code-that-populates-a-cache-that-is-never.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716696#action_12716696 ]
Jun Rao commented on CASSANDRA-208:
-----------------------------------
My ant test succeeds on linux, but fails under cygwin on windows. Here are some of the failures that I get. Those tests succeed on windows without your patch.
[junit] Testcase: testCompactions(org.apache.cassandra.db.CompactionsTest): FAILED
[junit] attempted to delete non-existing file Table1-Standard1-14-Filter.db
[junit] junit.framework.AssertionFailedError: attempted to delete non-existing file Table1-Standard1-14-Filter.db
[junit] at org.apache.cassandra.io.SSTable.deleteWithConfirm(SSTable.java:103)
[junit] at org.apache.cassandra.io.SSTable.delete(SSTable.java:98)
[junit] at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:1373)
[junit] at org.apache.cassandra.db.ColumnFamilyStore.doCompaction(ColumnFamilyStore.java:849)
[junit] at org.apache.cassandra.db.CompactionsTest.testCompactions(CompactionsTest.java:63)
[junit] Testcase: testNameSort10(org.apache.cassandra.db.NameSortTest): FAILED
[junit] null
[junit] junit.framework.AssertionFailedError
[junit] at org.apache.cassandra.db.NameSortTest.validateNameSort(NameSortTest.java:111)
[junit] at org.apache.cassandra.db.NameSortTest.testNameSort(NameSortTest.java:89)
[junit] at org.apache.cassandra.db.NameSortTest.testNameSort10(NameSortTest.java:43)
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt, 0004-fix-ColumnReader-add-tests.patch, 0005-fix-test-order-dependent-failures.patch
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716352#action_12716352 ]
Jonathan Ellis commented on CASSANDRA-208:
------------------------------------------
Good catch.
I will add a test for this in the version for trunk.
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch-code-that-populates-a-cache-that-is-never.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) jvm crashes intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715300#action_12715300 ]
Jonathan Ellis commented on CASSANDRA-208:
------------------------------------------
Or, how about this: we just stop storing multiple kinds of data in the SSTable file, and instead store the index entries and the bloom filter in separate on-disk files.
Gains (vs interleaving):
- simplicity (don't have to skip non-data keys, EOF is really EOF)
- no more hard-coded "keys" in the sstable that will result in very weird-ass bugs if a client every uses one
- behaves more like the FS cache expects since each section (index, data, bloom) is homogeneous and the "if I read block A, I'm more likely to need the next block" assumption holds more often
- retains goal of loading index on server start w/o seeking
Losses:
- some seeking during flush/compaction to switch between writing data blocks and index blocks
Although the "no seeks on writes, at all" claim is a cool one, in practice the amount of seeks we'll be doing is still negligible when buffering is done, i.e., still a huge win over traditional B-tree design where every update requires a seek.
Thoughts?
> jvm crashes intermittently during compaction
> --------------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.3
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-208) jvm crashes intermittently during
compaction
Posted by "Jiansheng Huang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714564#action_12714564 ]
Jiansheng Huang commented on CASSANDRA-208:
-------------------------------------------
we don't have a large row, definitely not to a size which cannot fit in memory.
> jvm crashes intermittently during compaction
> --------------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-208) OOM intermittently during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-208:
-------------------------------------
Attachment: 0004-fix-ColumnReader-add-tests.patch
> OOM intermittently during compaction
> ------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 0.4
>
> Attachments: 0001-CASSANDRA-208-cleanup.txt, 0002-r-m-touch.txt, 0003-split-sstable-into-data-index-and-bloom-filter-files.txt, 0004-fix-ColumnReader-add-tests.patch
>
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.