You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Radim Kolar (Created) (JIRA)" <ji...@apache.org> on 2012/01/13 20:02:40 UTC
[jira] [Created] (CASSANDRA-3743) Lower memory consumption used by
index sampling
Lower memory consumption used by index sampling
-----------------------------------------------
Key: CASSANDRA-3743
URL: https://issues.apache.org/jira/browse/CASSANDRA-3743
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 1.0.6
Reporter: Radim Kolar
currently j.o.a.c.io.sstable.indexsummary is implemented as ArrayList of KeyPosition (RowPosition key, long offset)i propose to change it to:
RowPosition keys[]
long offsets[]
and use standard binary search on it. This will lower number of java objects used per entry from 2 (KeyPosition + RowPosition) to 1 (RowPosition).
For building these arrays convenient ArrayList class can be used and then call to .toArray() on it.
This is very important because index sampling uses a lot of memory on nodes with billions rows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3743) Lower memory consumption used by
index sampling
Posted by "Radim Kolar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Radim Kolar updated CASSANDRA-3743:
-----------------------------------
Attachment: cassandra-3743.txt
> Lower memory consumption used by index sampling
> -----------------------------------------------
>
> Key: CASSANDRA-3743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3743
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.6
> Reporter: Radim Kolar
> Labels: optimization
> Fix For: 1.0.8
>
> Attachments: cassandra-3743.txt
>
>
> currently j.o.a.c.io.sstable.indexsummary is implemented as ArrayList of KeyPosition (RowPosition key, long offset)i propose to change it to:
> RowPosition keys[]
> long offsets[]
> and use standard binary search on it. This will lower number of java objects used per entry from 2 (KeyPosition + RowPosition) to 1 (RowPosition).
> For building these arrays convenient ArrayList class can be used and then call to .toArray() on it.
> This is very important because index sampling uses a lot of memory on nodes with billions rows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3743) Lower memory consumption used
by index sampling
Posted by "Yuki Morishita (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189286#comment-13189286 ]
Yuki Morishita commented on CASSANDRA-3743:
-------------------------------------------
+1 except coding style:
- you should use 4 spaces instead of tab
- always place { and } on new line
> Lower memory consumption used by index sampling
> -----------------------------------------------
>
> Key: CASSANDRA-3743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3743
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Radim Kolar
> Assignee: Radim Kolar
> Labels: optimization
> Fix For: 1.1
>
> Attachments: cassandra-3743.txt
>
>
> currently j.o.a.c.io.sstable.indexsummary is implemented as ArrayList of KeyPosition (RowPosition key, long offset)i propose to change it to:
> RowPosition keys[]
> long offsets[]
> and use standard binary search on it. This will lower number of java objects used per entry from 2 (KeyPosition + RowPosition) to 1 (RowPosition).
> For building these arrays convenient ArrayList class can be used and then call to .toArray() on it.
> This is very important because index sampling uses a lot of memory on nodes with billions rows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3743) Lower memory consumption used
by index sampling
Posted by "Radim Kolar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186912#comment-13186912 ]
Radim Kolar commented on CASSANDRA-3743:
----------------------------------------
i will load cassandra into Eclipse and check how many times is IndexSummary referenced
> Lower memory consumption used by index sampling
> -----------------------------------------------
>
> Key: CASSANDRA-3743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3743
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.6
> Reporter: Radim Kolar
>
> currently j.o.a.c.io.sstable.indexsummary is implemented as ArrayList of KeyPosition (RowPosition key, long offset)i propose to change it to:
> RowPosition keys[]
> long offsets[]
> and use standard binary search on it. This will lower number of java objects used per entry from 2 (KeyPosition + RowPosition) to 1 (RowPosition).
> For building these arrays convenient ArrayList class can be used and then call to .toArray() on it.
> This is very important because index sampling uses a lot of memory on nodes with billions rows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3743) Lower memory consumption used
by index sampling
Posted by "Yuki Morishita (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189978#comment-13189978 ]
Yuki Morishita commented on CASSANDRA-3743:
-------------------------------------------
I thought patch is for 1.0 branch.
Is it going to trunk?
> Lower memory consumption used by index sampling
> -----------------------------------------------
>
> Key: CASSANDRA-3743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3743
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Radim Kolar
> Assignee: Radim Kolar
> Labels: optimization
> Fix For: 1.1
>
> Attachments: cassandra-3743-codestyle.txt, cassandra-3743.txt
>
>
> currently j.o.a.c.io.sstable.indexsummary is implemented as ArrayList of KeyPosition (RowPosition key, long offset)i propose to change it to:
> RowPosition keys[]
> long offsets[]
> and use standard binary search on it. This will lower number of java objects used per entry from 2 (KeyPosition + RowPosition) to 1 (RowPosition).
> For building these arrays convenient ArrayList class can be used and then call to .toArray() on it.
> This is very important because index sampling uses a lot of memory on nodes with billions rows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3743) Lower memory consumption used
by index sampling
Posted by "Radim Kolar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190549#comment-13190549 ]
Radim Kolar commented on CASSANDRA-3743:
----------------------------------------
i have no plans to use 1.1, lets close ticket.
> Lower memory consumption used by index sampling
> -----------------------------------------------
>
> Key: CASSANDRA-3743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3743
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Radim Kolar
> Assignee: Radim Kolar
> Labels: optimization
> Fix For: 1.1
>
> Attachments: cassandra-3743-codestyle.txt
>
>
> currently j.o.a.c.io.sstable.indexsummary is implemented as ArrayList of KeyPosition (RowPosition key, long offset)i propose to change it to:
> RowPosition keys[]
> long offsets[]
> and use standard binary search on it. This will lower number of java objects used per entry from 2 (KeyPosition + RowPosition) to 1 (RowPosition).
> For building these arrays convenient ArrayList class can be used and then call to .toArray() on it.
> This is very important because index sampling uses a lot of memory on nodes with billions rows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3743) Lower memory consumption used
by index sampling
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189569#comment-13189569 ]
Jonathan Ellis commented on CASSANDRA-3743:
-------------------------------------------
Please submit an updated patch against 1.1, Radim.
> Lower memory consumption used by index sampling
> -----------------------------------------------
>
> Key: CASSANDRA-3743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3743
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Radim Kolar
> Assignee: Radim Kolar
> Labels: optimization
> Fix For: 1.1
>
> Attachments: cassandra-3743.txt
>
>
> currently j.o.a.c.io.sstable.indexsummary is implemented as ArrayList of KeyPosition (RowPosition key, long offset)i propose to change it to:
> RowPosition keys[]
> long offsets[]
> and use standard binary search on it. This will lower number of java objects used per entry from 2 (KeyPosition + RowPosition) to 1 (RowPosition).
> For building these arrays convenient ArrayList class can be used and then call to .toArray() on it.
> This is very important because index sampling uses a lot of memory on nodes with billions rows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3743) Lower memory consumption used
by index sampling
Posted by "Radim Kolar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185758#comment-13185758 ]
Radim Kolar commented on CASSANDRA-3743:
----------------------------------------
Its all about nuking KeyPosition class
> Lower memory consumption used by index sampling
> -----------------------------------------------
>
> Key: CASSANDRA-3743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3743
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.6
> Reporter: Radim Kolar
>
> currently j.o.a.c.io.sstable.indexsummary is implemented as ArrayList of KeyPosition (RowPosition key, long offset)i propose to change it to:
> RowPosition keys[]
> long offsets[]
> and use standard binary search on it. This will lower number of java objects used per entry from 2 (KeyPosition + RowPosition) to 1 (RowPosition).
> For building these arrays convenient ArrayList class can be used and then call to .toArray() on it.
> This is very important because index sampling uses a lot of memory on nodes with billions rows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3743) Lower memory consumption used
by index sampling
Posted by "Radim Kolar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189561#comment-13189561 ]
Radim Kolar commented on CASSANDRA-3743:
----------------------------------------
Are you going to fix these codestyle errors? its just few lines
> Lower memory consumption used by index sampling
> -----------------------------------------------
>
> Key: CASSANDRA-3743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3743
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Radim Kolar
> Assignee: Radim Kolar
> Labels: optimization
> Fix For: 1.1
>
> Attachments: cassandra-3743.txt
>
>
> currently j.o.a.c.io.sstable.indexsummary is implemented as ArrayList of KeyPosition (RowPosition key, long offset)i propose to change it to:
> RowPosition keys[]
> long offsets[]
> and use standard binary search on it. This will lower number of java objects used per entry from 2 (KeyPosition + RowPosition) to 1 (RowPosition).
> For building these arrays convenient ArrayList class can be used and then call to .toArray() on it.
> This is very important because index sampling uses a lot of memory on nodes with billions rows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3743) Lower memory consumption used
by index sampling
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185763#comment-13185763 ]
Jonathan Ellis commented on CASSANDRA-3743:
-------------------------------------------
This is a good idea. Are you going to take a stab at it, Radim?
> Lower memory consumption used by index sampling
> -----------------------------------------------
>
> Key: CASSANDRA-3743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3743
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.6
> Reporter: Radim Kolar
>
> currently j.o.a.c.io.sstable.indexsummary is implemented as ArrayList of KeyPosition (RowPosition key, long offset)i propose to change it to:
> RowPosition keys[]
> long offsets[]
> and use standard binary search on it. This will lower number of java objects used per entry from 2 (KeyPosition + RowPosition) to 1 (RowPosition).
> For building these arrays convenient ArrayList class can be used and then call to .toArray() on it.
> This is very important because index sampling uses a lot of memory on nodes with billions rows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3743) Lower memory consumption used
by index sampling
Posted by "Yuki Morishita (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189912#comment-13189912 ]
Yuki Morishita commented on CASSANDRA-3743:
-------------------------------------------
new patch attached lgtm.
> Lower memory consumption used by index sampling
> -----------------------------------------------
>
> Key: CASSANDRA-3743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3743
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Radim Kolar
> Assignee: Radim Kolar
> Labels: optimization
> Fix For: 1.1
>
> Attachments: cassandra-3743-codestyle.txt, cassandra-3743.txt
>
>
> currently j.o.a.c.io.sstable.indexsummary is implemented as ArrayList of KeyPosition (RowPosition key, long offset)i propose to change it to:
> RowPosition keys[]
> long offsets[]
> and use standard binary search on it. This will lower number of java objects used per entry from 2 (KeyPosition + RowPosition) to 1 (RowPosition).
> For building these arrays convenient ArrayList class can be used and then call to .toArray() on it.
> This is very important because index sampling uses a lot of memory on nodes with billions rows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3743) Lower memory consumption used
by index sampling
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189820#comment-13189820 ]
Jonathan Ellis commented on CASSANDRA-3743:
-------------------------------------------
There's Arrays.binarySearch. But do we always know how many positions we have before performing the sampling? If not you could just use AL.trimToSize at the end to avoid wasted space. With that the difference b/t AL and array are negligible here.
> Lower memory consumption used by index sampling
> -----------------------------------------------
>
> Key: CASSANDRA-3743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3743
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Radim Kolar
> Assignee: Radim Kolar
> Labels: optimization
> Fix For: 1.1
>
> Attachments: cassandra-3743.txt
>
>
> currently j.o.a.c.io.sstable.indexsummary is implemented as ArrayList of KeyPosition (RowPosition key, long offset)i propose to change it to:
> RowPosition keys[]
> long offsets[]
> and use standard binary search on it. This will lower number of java objects used per entry from 2 (KeyPosition + RowPosition) to 1 (RowPosition).
> For building these arrays convenient ArrayList class can be used and then call to .toArray() on it.
> This is very important because index sampling uses a lot of memory on nodes with billions rows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3743) Lower memory consumption used
by index sampling
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189975#comment-13189975 ]
Jonathan Ellis commented on CASSANDRA-3743:
-------------------------------------------
Really? I'm getting 7 of 11 hunks failing to apply against current trunk.
> Lower memory consumption used by index sampling
> -----------------------------------------------
>
> Key: CASSANDRA-3743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3743
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Radim Kolar
> Assignee: Radim Kolar
> Labels: optimization
> Fix For: 1.1
>
> Attachments: cassandra-3743-codestyle.txt, cassandra-3743.txt
>
>
> currently j.o.a.c.io.sstable.indexsummary is implemented as ArrayList of KeyPosition (RowPosition key, long offset)i propose to change it to:
> RowPosition keys[]
> long offsets[]
> and use standard binary search on it. This will lower number of java objects used per entry from 2 (KeyPosition + RowPosition) to 1 (RowPosition).
> For building these arrays convenient ArrayList class can be used and then call to .toArray() on it.
> This is very important because index sampling uses a lot of memory on nodes with billions rows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3743) Lower memory consumption used
by index sampling
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189990#comment-13189990 ]
Jonathan Ellis commented on CASSANDRA-3743:
-------------------------------------------
Yes, we should really be keeping 1.0 for bugfixes at this point. 1.1 freeze is in less than a week.
> Lower memory consumption used by index sampling
> -----------------------------------------------
>
> Key: CASSANDRA-3743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3743
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Radim Kolar
> Assignee: Radim Kolar
> Labels: optimization
> Fix For: 1.1
>
> Attachments: cassandra-3743-codestyle.txt, cassandra-3743.txt
>
>
> currently j.o.a.c.io.sstable.indexsummary is implemented as ArrayList of KeyPosition (RowPosition key, long offset)i propose to change it to:
> RowPosition keys[]
> long offsets[]
> and use standard binary search on it. This will lower number of java objects used per entry from 2 (KeyPosition + RowPosition) to 1 (RowPosition).
> For building these arrays convenient ArrayList class can be used and then call to .toArray() on it.
> This is very important because index sampling uses a lot of memory on nodes with billions rows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3743) Lower memory consumption used
by index sampling
Posted by "Radim Kolar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188349#comment-13188349 ]
Radim Kolar commented on CASSANDRA-3743:
----------------------------------------
I am working on it now
> Lower memory consumption used by index sampling
> -----------------------------------------------
>
> Key: CASSANDRA-3743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3743
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.6
> Reporter: Radim Kolar
>
> currently j.o.a.c.io.sstable.indexsummary is implemented as ArrayList of KeyPosition (RowPosition key, long offset)i propose to change it to:
> RowPosition keys[]
> long offsets[]
> and use standard binary search on it. This will lower number of java objects used per entry from 2 (KeyPosition + RowPosition) to 1 (RowPosition).
> For building these arrays convenient ArrayList class can be used and then call to .toArray() on it.
> This is very important because index sampling uses a lot of memory on nodes with billions rows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3743) Lower memory consumption used by
index sampling
Posted by "Radim Kolar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Radim Kolar updated CASSANDRA-3743:
-----------------------------------
Attachment: cassandra-3743-codestyle.txt
> Lower memory consumption used by index sampling
> -----------------------------------------------
>
> Key: CASSANDRA-3743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3743
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Radim Kolar
> Assignee: Radim Kolar
> Labels: optimization
> Fix For: 1.1
>
> Attachments: cassandra-3743-codestyle.txt, cassandra-3743.txt
>
>
> currently j.o.a.c.io.sstable.indexsummary is implemented as ArrayList of KeyPosition (RowPosition key, long offset)i propose to change it to:
> RowPosition keys[]
> long offsets[]
> and use standard binary search on it. This will lower number of java objects used per entry from 2 (KeyPosition + RowPosition) to 1 (RowPosition).
> For building these arrays convenient ArrayList class can be used and then call to .toArray() on it.
> This is very important because index sampling uses a lot of memory on nodes with billions rows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3743) Lower memory consumption used
by index sampling
Posted by "Vijay (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189629#comment-13189629 ]
Vijay commented on CASSANDRA-3743:
----------------------------------
Wondering if we can use DecoratedKey[] instead of ArrayList in the attached patch... Just my 2 cents (I noticed this while doing another patch).
> Lower memory consumption used by index sampling
> -----------------------------------------------
>
> Key: CASSANDRA-3743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3743
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Radim Kolar
> Assignee: Radim Kolar
> Labels: optimization
> Fix For: 1.1
>
> Attachments: cassandra-3743.txt
>
>
> currently j.o.a.c.io.sstable.indexsummary is implemented as ArrayList of KeyPosition (RowPosition key, long offset)i propose to change it to:
> RowPosition keys[]
> long offsets[]
> and use standard binary search on it. This will lower number of java objects used per entry from 2 (KeyPosition + RowPosition) to 1 (RowPosition).
> For building these arrays convenient ArrayList class can be used and then call to .toArray() on it.
> This is very important because index sampling uses a lot of memory on nodes with billions rows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3743) Lower memory consumption used by
index sampling
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3743:
--------------------------------------
Attachment: 3743-trunk-trim.txt
Slight update attached to use positions.trimToSize instead of leaving an "orphan" arraylist around. As mentioned above, the size difference is negligible.
Question: is the getKeySamples transform still necessary, or can we just cast the keys collection?
Or could we just make it a List of DK in the first place instead of RP?
> Lower memory consumption used by index sampling
> -----------------------------------------------
>
> Key: CASSANDRA-3743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3743
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Radim Kolar
> Assignee: Radim Kolar
> Labels: optimization
> Fix For: 1.1
>
> Attachments: 3743-trunk-trim.txt, 3743-trunk.txt, cassandra-3743-codestyle.txt
>
>
> currently j.o.a.c.io.sstable.indexsummary is implemented as ArrayList of KeyPosition (RowPosition key, long offset)i propose to change it to:
> RowPosition keys[]
> long offsets[]
> and use standard binary search on it. This will lower number of java objects used per entry from 2 (KeyPosition + RowPosition) to 1 (RowPosition).
> For building these arrays convenient ArrayList class can be used and then call to .toArray() on it.
> This is very important because index sampling uses a lot of memory on nodes with billions rows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3743) Lower memory consumption used by
index sampling
Posted by "Radim Kolar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Radim Kolar updated CASSANDRA-3743:
-----------------------------------
Attachment: (was: cassandra-3743.txt)
> Lower memory consumption used by index sampling
> -----------------------------------------------
>
> Key: CASSANDRA-3743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3743
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Radim Kolar
> Assignee: Radim Kolar
> Labels: optimization
> Fix For: 1.1
>
> Attachments: cassandra-3743-codestyle.txt
>
>
> currently j.o.a.c.io.sstable.indexsummary is implemented as ArrayList of KeyPosition (RowPosition key, long offset)i propose to change it to:
> RowPosition keys[]
> long offsets[]
> and use standard binary search on it. This will lower number of java objects used per entry from 2 (KeyPosition + RowPosition) to 1 (RowPosition).
> For building these arrays convenient ArrayList class can be used and then call to .toArray() on it.
> This is very important because index sampling uses a lot of memory on nodes with billions rows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3743) Lower memory consumption used
by index sampling
Posted by "Vijay (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190138#comment-13190138 ]
Vijay commented on CASSANDRA-3743:
----------------------------------
>>> But do we always know how many positions we have before performing the sampling?
I think we do while doing the flush... not a big deal i just noticed long position[] hence was wondering why not DecoratedKeys[]
>>> With that the difference b/t AL and array are negligible here.
Agreed!
> Lower memory consumption used by index sampling
> -----------------------------------------------
>
> Key: CASSANDRA-3743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3743
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Radim Kolar
> Assignee: Radim Kolar
> Labels: optimization
> Fix For: 1.1
>
> Attachments: cassandra-3743-codestyle.txt, cassandra-3743.txt
>
>
> currently j.o.a.c.io.sstable.indexsummary is implemented as ArrayList of KeyPosition (RowPosition key, long offset)i propose to change it to:
> RowPosition keys[]
> long offsets[]
> and use standard binary search on it. This will lower number of java objects used per entry from 2 (KeyPosition + RowPosition) to 1 (RowPosition).
> For building these arrays convenient ArrayList class can be used and then call to .toArray() on it.
> This is very important because index sampling uses a lot of memory on nodes with billions rows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3743) Lower memory consumption used
by index sampling
Posted by "Radim Kolar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188950#comment-13188950 ]
Radim Kolar commented on CASSANDRA-3743:
----------------------------------------
patch is against-1.0. This was expected to go into 1.0-branch. Its small change KeyPosition was referenced by just one other class
> Lower memory consumption used by index sampling
> -----------------------------------------------
>
> Key: CASSANDRA-3743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3743
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Radim Kolar
> Assignee: Radim Kolar
> Labels: optimization
> Fix For: 1.1
>
> Attachments: cassandra-3743.txt
>
>
> currently j.o.a.c.io.sstable.indexsummary is implemented as ArrayList of KeyPosition (RowPosition key, long offset)i propose to change it to:
> RowPosition keys[]
> long offsets[]
> and use standard binary search on it. This will lower number of java objects used per entry from 2 (KeyPosition + RowPosition) to 1 (RowPosition).
> For building these arrays convenient ArrayList class can be used and then call to .toArray() on it.
> This is very important because index sampling uses a lot of memory on nodes with billions rows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-3743) Lower memory consumption used by
index sampling
Posted by "Jonathan Ellis (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-3743.
---------------------------------------
Resolution: Fixed
committed, thanks!
> Lower memory consumption used by index sampling
> -----------------------------------------------
>
> Key: CASSANDRA-3743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3743
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Radim Kolar
> Assignee: Radim Kolar
> Labels: optimization
> Fix For: 1.1
>
> Attachments: 3743-trunk-trim.txt, 3743-trunk-v2.txt, 3743-trunk.txt, cassandra-3743-codestyle.txt
>
>
> currently j.o.a.c.io.sstable.indexsummary is implemented as ArrayList of KeyPosition (RowPosition key, long offset)i propose to change it to:
> RowPosition keys[]
> long offsets[]
> and use standard binary search on it. This will lower number of java objects used per entry from 2 (KeyPosition + RowPosition) to 1 (RowPosition).
> For building these arrays convenient ArrayList class can be used and then call to .toArray() on it.
> This is very important because index sampling uses a lot of memory on nodes with billions rows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3743) Lower memory consumption used by
index sampling
Posted by "Yuki Morishita (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuki Morishita updated CASSANDRA-3743:
--------------------------------------
Attachment: 3743-trunk-v2.txt
bq. is the getKeySamples transform still necessary, or can we just cast the keys collection? Or could we just make it a List of DK in the first place instead of RP?
I think it is better just make a list of DK since IndexSummary.addEntry accepts only DK. Attached v2.
> Lower memory consumption used by index sampling
> -----------------------------------------------
>
> Key: CASSANDRA-3743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3743
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Radim Kolar
> Assignee: Radim Kolar
> Labels: optimization
> Fix For: 1.1
>
> Attachments: 3743-trunk-trim.txt, 3743-trunk-v2.txt, 3743-trunk.txt, cassandra-3743-codestyle.txt
>
>
> currently j.o.a.c.io.sstable.indexsummary is implemented as ArrayList of KeyPosition (RowPosition key, long offset)i propose to change it to:
> RowPosition keys[]
> long offsets[]
> and use standard binary search on it. This will lower number of java objects used per entry from 2 (KeyPosition + RowPosition) to 1 (RowPosition).
> For building these arrays convenient ArrayList class can be used and then call to .toArray() on it.
> This is very important because index sampling uses a lot of memory on nodes with billions rows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3743) Lower memory consumption used
by index sampling
Posted by "Radim Kolar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189814#comment-13189814 ]
Radim Kolar commented on CASSANDRA-3743:
----------------------------------------
has java builtin binary search operating on array[] ?
> Lower memory consumption used by index sampling
> -----------------------------------------------
>
> Key: CASSANDRA-3743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3743
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Radim Kolar
> Assignee: Radim Kolar
> Labels: optimization
> Fix For: 1.1
>
> Attachments: cassandra-3743.txt
>
>
> currently j.o.a.c.io.sstable.indexsummary is implemented as ArrayList of KeyPosition (RowPosition key, long offset)i propose to change it to:
> RowPosition keys[]
> long offsets[]
> and use standard binary search on it. This will lower number of java objects used per entry from 2 (KeyPosition + RowPosition) to 1 (RowPosition).
> For building these arrays convenient ArrayList class can be used and then call to .toArray() on it.
> This is very important because index sampling uses a lot of memory on nodes with billions rows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3743) Lower memory consumption used by
index sampling
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3743:
--------------------------------------
Reviewer: yukim
Affects Version/s: (was: 1.0.6)
1.0.0
Fix Version/s: (was: 1.0.8)
1.1
Assignee: Radim Kolar
> Lower memory consumption used by index sampling
> -----------------------------------------------
>
> Key: CASSANDRA-3743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3743
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Radim Kolar
> Assignee: Radim Kolar
> Labels: optimization
> Fix For: 1.1
>
> Attachments: cassandra-3743.txt
>
>
> currently j.o.a.c.io.sstable.indexsummary is implemented as ArrayList of KeyPosition (RowPosition key, long offset)i propose to change it to:
> RowPosition keys[]
> long offsets[]
> and use standard binary search on it. This will lower number of java objects used per entry from 2 (KeyPosition + RowPosition) to 1 (RowPosition).
> For building these arrays convenient ArrayList class can be used and then call to .toArray() on it.
> This is very important because index sampling uses a lot of memory on nodes with billions rows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3743) Lower memory consumption used by
index sampling
Posted by "Yuki Morishita (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuki Morishita updated CASSANDRA-3743:
--------------------------------------
Attachment: 3743-trunk.txt
Rebased Radim's patch for trunk.
> Lower memory consumption used by index sampling
> -----------------------------------------------
>
> Key: CASSANDRA-3743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3743
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Radim Kolar
> Assignee: Radim Kolar
> Labels: optimization
> Fix For: 1.1
>
> Attachments: 3743-trunk.txt, cassandra-3743-codestyle.txt
>
>
> currently j.o.a.c.io.sstable.indexsummary is implemented as ArrayList of KeyPosition (RowPosition key, long offset)i propose to change it to:
> RowPosition keys[]
> long offsets[]
> and use standard binary search on it. This will lower number of java objects used per entry from 2 (KeyPosition + RowPosition) to 1 (RowPosition).
> For building these arrays convenient ArrayList class can be used and then call to .toArray() on it.
> This is very important because index sampling uses a lot of memory on nodes with billions rows
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira