You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2010/01/05 16:58:55 UTC
[jira] Created: (CASSANDRA-669) Handle mmapping index files greater
than 2GB
Handle mmapping index files greater than 2GB
--------------------------------------------
Key: CASSANDRA-669
URL: https://issues.apache.org/jira/browse/CASSANDRA-669
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.9
Reporter: Jonathan Ellis
Assignee: Jonathan Ellis
Fix For: 0.9
"Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-669) Handle mmapping index files greater
than 2GB
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-669:
-------------------------------------
Attachment: (was: 0001-store-data-information-for-any-index-entries-spanning-.txt)
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt, 0003-use-long-for-mmap-segment-position-instead-of-int-whic.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-669) Handle mmapping index files
greater than 2GB
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797155#action_12797155 ]
Brandon Williams commented on CASSANDRA-669:
--------------------------------------------
Oops, nevermind my +1, receiving the following traceback when trying to get_slice now:
Caused by: java.lang.AssertionError
at org.apache.cassandra.io.util.MappedFileDataInput.length(MappedFileDataInput.java:36)
at org.apache.cassandra.io.util.MappedFileDataInput.read(MappedFileDataInput.java:52)
at java.io.InputStream.read(InputStream.java:171)
at org.apache.cassandra.io.util.MappedFileDataInput.readUnsignedShort(MappedFileDataInput.java:358)
at java.io.DataInputStream.readUTF(DataInputStream.java:589)
at org.apache.cassandra.io.util.MappedFileDataInput.readUTF(MappedFileDataInput.java:381)
at org.apache.cassandra.io.SSTableReader.getPosition(SSTableReader.java:419)
at org.apache.cassandra.io.SSTableReader.getFileDataInput(SSTableReader.java:537)
at org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:54)
at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamilyInternal(ColumnFamilyStore.java:859)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:817)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:786)
at org.apache.cassandra.db.Table.getRow(Table.java:405)
at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
at org.apache.cassandra.service.StorageProxy$weakReadLocalCallable.call(StorageProxy.java:692)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
... 3 more
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt, 0003-use-long-for-mmap-segment-position-instead-of-int-whic.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-669) Handle mmapping index files greater
than 2GB
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-669:
-------------------------------------
Attachment: 0003-instead-of-providing-a-RandomAccessFile-like-interface.txt
0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt
0001-store-data-information-for-any-index-entries-spanning-.txt
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt, 0003-instead-of-providing-a-RandomAccessFile-like-interface.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-669) Handle mmapping index files
greater than 2GB
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797614#action_12797614 ]
Hudson commented on CASSANDRA-669:
----------------------------------
Integrated in Cassandra #316 (See [http://hudson.zones.apache.org/hudson/job/Cassandra/316/])
instead of providing a RandomAccessFile-like interface in FileDataInput implementing seek and trying to keep people from shooting themselves in the foot by forgetting that it may only represent a 2GB segment of a larger file, provide an InputStream-like interface emphasizing mark/reset
patch by jbellis; tested by Brandon Williams for
add support for multiple mmapped index segments, and add mmap_index_only option
patch by jbellis; tested by Brandon Williams for
store data information for any index entries spanning a mmap segment boundary when reading the index (with a BufferedRAF) so we don't have to deal with that at read time.
patch by jbellis; tested by Brandon Williams for
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt, 0003-instead-of-providing-a-RandomAccessFile-like-interface.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-669) Handle mmapping index files
greater than 2GB
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797997#action_12797997 ]
Hudson commented on CASSANDRA-669:
----------------------------------
Integrated in Cassandra #317 (See [http://hudson.zones.apache.org/hudson/job/Cassandra/317/])
fix bad rebase causing regression in the patchset (use index path to open index file). patch by jbellis
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt, 0003-instead-of-providing-a-RandomAccessFile-like-interface.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-669) Handle mmapping index files greater
than 2GB
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-669:
-------------------------------------
Attachment: 0004-instead-of-providing-a-RandomAccessFile-like-interface.txt
0003-use-long-for-mmap-segment-position-instead-of-int-whic.txt
0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt
0001-store-data-information-for-any-index-entries-spanning-.txt
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt, 0003-use-long-for-mmap-segment-position-instead-of-int-whic.txt, 0004-instead-of-providing-a-RandomAccessFile-like-interface.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-669) Handle mmapping index files
greater than 2GB
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797462#action_12797462 ]
Brandon Williams commented on CASSANDRA-669:
--------------------------------------------
+1, for real this time. 40M narrow keys used to create a 2.2GB index when fully compacted, all is well.
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt, 0003-instead-of-providing-a-RandomAccessFile-like-interface.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-669) Handle mmapping index files greater
than 2GB
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-669:
-------------------------------------
Attachment: (was: 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt)
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt, 0003-use-long-for-mmap-segment-position-instead-of-int-whic.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-669) Handle mmapping index files
greater than 2GB
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797329#action_12797329 ]
Jonathan Ellis commented on CASSANDRA-669:
------------------------------------------
... latest patches fix the fd leak and index reading bugs, and rebased bugfixes into 01 and 02.
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt, 0003-instead-of-providing-a-RandomAccessFile-like-interface.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-669) Handle mmapping index files greater
than 2GB
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-669:
-------------------------------------
Attachment: (was: 0003-use-long-for-mmap-segment-position-instead-of-int-whic.txt)
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt, 0003-use-long-for-mmap-segment-position-instead-of-int-whic.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-669) Handle mmapping index files
greater than 2GB
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796989#action_12796989 ]
Jonathan Ellis commented on CASSANDRA-669:
------------------------------------------
fix the -- in the xml in 02, and change BUFFER_SIZE to long in 03 as well to force the promotion of the position parameter.
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt, 0003-use-long-for-mmap-segment-position-instead-of-int-whic.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-669) Handle mmapping index files greater
than 2GB
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-669:
-------------------------------------
Attachment: (was: 0003-use-long-for-mmap-segment-position-instead-of-int-whic.txt)
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt, 0003-instead-of-providing-a-RandomAccessFile-like-interface.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-669) Handle mmapping index files
greater than 2GB
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796979#action_12796979 ]
Jonathan Ellis commented on CASSANDRA-669:
------------------------------------------
heh, that's actually a 2nd bug. patch 03 attached to fix.
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt, 0003-use-long-for-mmap-segment-position-instead-of-int-whic.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CASSANDRA-669) Handle mmapping index files
greater than 2GB
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-669.
--------------------------------------
Resolution: Fixed
committed
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt, 0003-instead-of-providing-a-RandomAccessFile-like-interface.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-669) Handle mmapping index files greater
than 2GB
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-669:
-------------------------------------
Attachment: 0003-use-long-for-mmap-segment-position-instead-of-int-whic.txt
0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt
0001-store-data-information-for-any-index-entries-spanning-.txt
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt, 0003-use-long-for-mmap-segment-position-instead-of-int-whic.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-669) Handle mmapping index files greater
than 2GB
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-669:
-------------------------------------
Attachment: (was: 0001-store-data-information-for-any-index-entries-spanning-.txt)
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt, 0003-use-long-for-mmap-segment-position-instead-of-int-whic.txt, 0004-instead-of-providing-a-RandomAccessFile-like-interface.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-669) Handle mmapping index files greater
than 2GB
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-669:
-------------------------------------
Attachment: (was: 0004-instead-of-providing-a-RandomAccessFile-like-interface.txt)
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt, 0003-instead-of-providing-a-RandomAccessFile-like-interface.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-669) Handle mmapping index files
greater than 2GB
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796851#action_12796851 ]
Jonathan Ellis commented on CASSANDRA-669:
------------------------------------------
02
add support for multiple mmapped index segments, and add mmap_index_only option
01
store data information for any index entries spanning a mmap segment boundary when reading the index (with a BufferedRAF)
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-669) Handle mmapping index files
greater than 2GB
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796913#action_12796913 ]
Brandon Williams commented on CASSANDRA-669:
--------------------------------------------
Received the following traceback after testing (during compaction):
ERROR - Error in executor futuretask
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Negative position
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:53)
at org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:591)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1118)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.IllegalArgumentException: Negative position
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:739)
at org.apache.cassandra.io.SSTableReader.mmap(SSTableReader.java:273)
at org.apache.cassandra.io.SSTableReader.<init>(SSTableReader.java:234)
at org.apache.cassandra.io.SSTableWriter.closeAndOpenReader(SSTableWriter.java:157)
at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:307)
at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:102)
at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:83)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-669) Handle mmapping index files greater
than 2GB
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-669:
-------------------------------------
Attachment: (was: 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt)
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt, 0003-instead-of-providing-a-RandomAccessFile-like-interface.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-669) Handle mmapping index files greater
than 2GB
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-669:
-------------------------------------
Attachment: (was: 0001-store-data-information-for-any-index-entries-spanning-.txt)
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt, 0003-instead-of-providing-a-RandomAccessFile-like-interface.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-669) Handle mmapping index files
greater than 2GB
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797186#action_12797186 ]
Jonathan Ellis commented on CASSANDRA-669:
------------------------------------------
patch 04 to fix.
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt, 0003-use-long-for-mmap-segment-position-instead-of-int-whic.txt, 0004-instead-of-providing-a-RandomAccessFile-like-interface.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-669) Handle mmapping index files greater
than 2GB
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-669:
-------------------------------------
Attachment: (was: 0003-use-long-for-mmap-segment-position-instead-of-int-whic.txt)
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt, 0003-use-long-for-mmap-segment-position-instead-of-int-whic.txt, 0004-instead-of-providing-a-RandomAccessFile-like-interface.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-669) Handle mmapping index files greater
than 2GB
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-669:
-------------------------------------
Attachment: (was: 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt)
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt, 0003-use-long-for-mmap-segment-position-instead-of-int-whic.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-669) Handle mmapping index files greater
than 2GB
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-669:
-------------------------------------
Attachment: 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt
0001-store-data-information-for-any-index-entries-spanning-.txt
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-669) Handle mmapping index files
greater than 2GB
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797132#action_12797132 ]
Brandon Williams commented on CASSANDRA-669:
--------------------------------------------
+1
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt, 0003-use-long-for-mmap-segment-position-instead-of-int-whic.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-669) Handle mmapping index files greater
than 2GB
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-669:
-------------------------------------
Attachment: (was: 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt)
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt, 0003-use-long-for-mmap-segment-position-instead-of-int-whic.txt, 0004-instead-of-providing-a-RandomAccessFile-like-interface.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-669) Handle mmapping index files greater
than 2GB
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-669:
-------------------------------------
Attachment: (was: 0001-store-data-information-for-any-index-entries-spanning-.txt)
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt, 0003-use-long-for-mmap-segment-position-instead-of-int-whic.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-669) Handle mmapping index files greater
than 2GB
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-669:
-------------------------------------
Attachment: 0003-use-long-for-mmap-segment-position-instead-of-int-whic.txt
0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt
0001-store-data-information-for-any-index-entries-spanning-.txt
> Handle mmapping index files greater than 2GB
> --------------------------------------------
>
> Key: CASSANDRA-669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.9
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.9
>
> Attachments: 0001-store-data-information-for-any-index-entries-spanning-.txt, 0002-add-support-for-multiple-mmapped-index-segments-and-ad.txt, 0003-use-long-for-mmap-segment-position-instead-of-int-whic.txt
>
>
> "Who would ever have an index file larger than 2GB?" I thought. Turns out it's not that hard with narrow rows... :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.