You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2011/09/12 19:44:09 UTC
[jira] [Created] (CASSANDRA-3183) Make SerializingCacheProvider the
default if JNA is available
Make SerializingCacheProvider the default if JNA is available
-------------------------------------------------------------
Key: CASSANDRA-3183
URL: https://issues.apache.org/jira/browse/CASSANDRA-3183
Project: Cassandra
Issue Type: Task
Components: Core
Reporter: Jonathan Ellis
Assignee: Jonathan Ellis
Priority: Minor
Fix For: 1.0.0
The serializing cache is a better choice for most users:
- Lower total memory usage (serialized data is usually 8x-12x smaller than "live" data in the JVM with all the overhead that involves) means you can cache more rows for a given memory footprint
- Moving the serialized rows off-heap means you can use smaller heaps, reducing the impact of GC pauses
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3183) Make SerializingCacheProvider the
default if JNA is available
Posted by "Yang Yang (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Yang updated CASSANDRA-3183:
---------------------------------
Comment: was deleted
(was: I actually ran into some issues with the serializing cache, it seems to make the perm gen unable to GC (even with a forced full GC), and increases memory consumption much faster.
I had to invoke invalidate cache in JMX followed by a forced GC to release that memory.
just a FYI for now, I'm still going through the tests and comparison with the hashmap-based cache, will update more definitively when I get more results)
> Make SerializingCacheProvider the default if JNA is available
> -------------------------------------------------------------
>
> Key: CASSANDRA-3183
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3183
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 1.0.0
>
> Attachments: 3183-additions.patch, 3183-alt.patch, 3183.txt, CASSANDRA-3183-alt-v2.patch, CASSANDRA-3183-alt-v3.patch
>
>
> The serializing cache is a better choice for most users:
> - Lower total memory usage (serialized data is usually 8x-12x smaller than "live" data in the JVM with all the overhead that involves) means you can cache more rows for a given memory footprint
> - Moving the serialized rows off-heap means you can use smaller heaps, reducing the impact of GC pauses
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3183) Make SerializingCacheProvider the
default if JNA is available
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-3183:
---------------------------------------
Attachment: CASSANDRA-3183-alt-v2.patch
made SCP to use CLibrary.isJNAEnabled()
fixed thrift and CLI documentation, this also fixes bug in CFMetaData.toThrift() method which was not including row_cache_provider to the resulting CfDef
> Make SerializingCacheProvider the default if JNA is available
> -------------------------------------------------------------
>
> Key: CASSANDRA-3183
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3183
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 1.0.0
>
> Attachments: 3183-additions.patch, 3183-alt.patch, 3183.txt, CASSANDRA-3183-alt-v2.patch
>
>
> The serializing cache is a better choice for most users:
> - Lower total memory usage (serialized data is usually 8x-12x smaller than "live" data in the JVM with all the overhead that involves) means you can cache more rows for a given memory footprint
> - Moving the serialized rows off-heap means you can use smaller heaps, reducing the impact of GC pauses
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3183) Make SerializingCacheProvider
the default if JNA is available
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103140#comment-13103140 ]
Jonathan Ellis commented on CASSANDRA-3183:
-------------------------------------------
I don't think you can actually end up w/ a null r_c_p so this is unnecessary?
{code}
- writeAttr(sb, false, "row_cache_provider", normaliseType(cfDef.row_cache_provider, "org.apache.cassandra.cache"));
+ writeAttr(sb, false, "row_cache_provider", cfDef.row_cache_provider == null ? "" : cfDef.row_cache_provider);
{code}
> Make SerializingCacheProvider the default if JNA is available
> -------------------------------------------------------------
>
> Key: CASSANDRA-3183
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3183
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 1.0.0
>
> Attachments: 3183-additions.patch, 3183-alt.patch, 3183.txt, CASSANDRA-3183-alt-v2.patch, CASSANDRA-3183-alt-v3.patch
>
>
> The serializing cache is a better choice for most users:
> - Lower total memory usage (serialized data is usually 8x-12x smaller than "live" data in the JVM with all the overhead that involves) means you can cache more rows for a given memory footprint
> - Moving the serialized rows off-heap means you can use smaller heaps, reducing the impact of GC pauses
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3183) Make SerializingCacheProvider
the default if JNA is available
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103147#comment-13103147 ]
Jonathan Ellis commented on CASSANDRA-3183:
-------------------------------------------
+1
> Make SerializingCacheProvider the default if JNA is available
> -------------------------------------------------------------
>
> Key: CASSANDRA-3183
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3183
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 1.0.0
>
> Attachments: 3183-additions.patch, 3183-alt.patch, 3183.txt, CASSANDRA-3183-alt-v2.patch, CASSANDRA-3183-alt-v3.patch
>
>
> The serializing cache is a better choice for most users:
> - Lower total memory usage (serialized data is usually 8x-12x smaller than "live" data in the JVM with all the overhead that involves) means you can cache more rows for a given memory footprint
> - Moving the serialized rows off-heap means you can use smaller heaps, reducing the impact of GC pauses
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3183) Make SerializingCacheProvider
the default if JNA is available
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103014#comment-13103014 ]
Jonathan Ellis commented on CASSANDRA-3183:
-------------------------------------------
- I don't like having two different checks (isJNAEnabled in CLibrary, and does Memory class exist in SCP)
- -alt patch doesn't address other problems (thrift always including a CLHCP value, which means default in CFM gets ignored, or being able to startup on a machine w/o JNA, w/ a schema including SCP)
> Make SerializingCacheProvider the default if JNA is available
> -------------------------------------------------------------
>
> Key: CASSANDRA-3183
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3183
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 1.0.0
>
> Attachments: 3183-alt.patch, 3183.txt
>
>
> The serializing cache is a better choice for most users:
> - Lower total memory usage (serialized data is usually 8x-12x smaller than "live" data in the JVM with all the overhead that involves) means you can cache more rows for a given memory footprint
> - Moving the serialized rows off-heap means you can use smaller heaps, reducing the impact of GC pauses
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3183) Make SerializingCacheProvider
the default if JNA is available
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103126#comment-13103126 ]
Pavel Yaskevich commented on CASSANDRA-3183:
--------------------------------------------
bq. I don't think this is actually technically correct, though – JNA Memory does not require libc to work.
we can still use Memory.class.getName() there if you like
> Make SerializingCacheProvider the default if JNA is available
> -------------------------------------------------------------
>
> Key: CASSANDRA-3183
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3183
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 1.0.0
>
> Attachments: 3183-additions.patch, 3183-alt.patch, 3183.txt, CASSANDRA-3183-alt-v2.patch
>
>
> The serializing cache is a better choice for most users:
> - Lower total memory usage (serialized data is usually 8x-12x smaller than "live" data in the JVM with all the overhead that involves) means you can cache more rows for a given memory footprint
> - Moving the serialized rows off-heap means you can use smaller heaps, reducing the impact of GC pauses
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3183) Make SerializingCacheProvider the
default if JNA is available
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-3183:
---------------------------------------
Attachment: (was: CASSANDRA-3183-alt-v3.patch)
> Make SerializingCacheProvider the default if JNA is available
> -------------------------------------------------------------
>
> Key: CASSANDRA-3183
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3183
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 1.0.0
>
> Attachments: 3183-additions.patch, 3183-alt.patch, 3183.txt, CASSANDRA-3183-alt-v2.patch, CASSANDRA-3183-alt-v3.patch
>
>
> The serializing cache is a better choice for most users:
> - Lower total memory usage (serialized data is usually 8x-12x smaller than "live" data in the JVM with all the overhead that involves) means you can cache more rows for a given memory footprint
> - Moving the serialized rows off-heap means you can use smaller heaps, reducing the impact of GC pauses
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3183) Make SerializingCacheProvider
the default if JNA is available
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103120#comment-13103120 ]
Jonathan Ellis commented on CASSANDRA-3183:
-------------------------------------------
bq. made SCP to use CLibrary.isJNAEnabled()
I don't think this is actually technically correct, though -- JNA Memory does not require libc to work.
> Make SerializingCacheProvider the default if JNA is available
> -------------------------------------------------------------
>
> Key: CASSANDRA-3183
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3183
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 1.0.0
>
> Attachments: 3183-additions.patch, 3183-alt.patch, 3183.txt, CASSANDRA-3183-alt-v2.patch
>
>
> The serializing cache is a better choice for most users:
> - Lower total memory usage (serialized data is usually 8x-12x smaller than "live" data in the JVM with all the overhead that involves) means you can cache more rows for a given memory footprint
> - Moving the serialized rows off-heap means you can use smaller heaps, reducing the impact of GC pauses
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3183) Make SerializingCacheProvider the
default if JNA is available
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-3183:
---------------------------------------
Attachment: CASSANDRA-3183-alt-v3.patch
v3 without CLibrary.isJNAEnabled changes.
> Make SerializingCacheProvider the default if JNA is available
> -------------------------------------------------------------
>
> Key: CASSANDRA-3183
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3183
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 1.0.0
>
> Attachments: 3183-additions.patch, 3183-alt.patch, 3183.txt, CASSANDRA-3183-alt-v2.patch, CASSANDRA-3183-alt-v3.patch
>
>
> The serializing cache is a better choice for most users:
> - Lower total memory usage (serialized data is usually 8x-12x smaller than "live" data in the JVM with all the overhead that involves) means you can cache more rows for a given memory footprint
> - Moving the serialized rows off-heap means you can use smaller heaps, reducing the impact of GC pauses
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3183) Make SerializingCacheProvider
the default if JNA is available
Posted by "Yang Yang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118185#comment-13118185 ]
Yang Yang commented on CASSANDRA-3183:
--------------------------------------
I actually ran into some issues with the serializing cache, it seems to make the perm gen unable to GC (even with a forced full GC), and increases memory consumption much faster.
I had to invoke invalidate cache in JMX followed by a forced GC to release that memory.
just a FYI for now, I'm still going through the tests and comparison with the hashmap-based cache, will update more definitively when I get more results
> Make SerializingCacheProvider the default if JNA is available
> -------------------------------------------------------------
>
> Key: CASSANDRA-3183
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3183
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 1.0.0
>
> Attachments: 3183-additions.patch, 3183-alt.patch, 3183.txt, CASSANDRA-3183-alt-v2.patch, CASSANDRA-3183-alt-v3.patch
>
>
> The serializing cache is a better choice for most users:
> - Lower total memory usage (serialized data is usually 8x-12x smaller than "live" data in the JVM with all the overhead that involves) means you can cache more rows for a given memory footprint
> - Moving the serialized rows off-heap means you can use smaller heaps, reducing the impact of GC pauses
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3183) Make SerializingCacheProvider the
default if JNA is available
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-3183:
---------------------------------------
Attachment: 3183-alt.patch
alternative version using boolean flag from CLibrary.
> Make SerializingCacheProvider the default if JNA is available
> -------------------------------------------------------------
>
> Key: CASSANDRA-3183
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3183
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 1.0.0
>
> Attachments: 3183-alt.patch, 3183.txt
>
>
> The serializing cache is a better choice for most users:
> - Lower total memory usage (serialized data is usually 8x-12x smaller than "live" data in the JVM with all the overhead that involves) means you can cache more rows for a given memory footprint
> - Moving the serialized rows off-heap means you can use smaller heaps, reducing the impact of GC pauses
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3183) Make SerializingCacheProvider the
default if JNA is available
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-3183:
---------------------------------------
Attachment: CASSANDRA-3183-alt-v3.patch
You are absolutely right, I forgot to revert that but fixed now, v3 is re-attached.
> Make SerializingCacheProvider the default if JNA is available
> -------------------------------------------------------------
>
> Key: CASSANDRA-3183
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3183
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 1.0.0
>
> Attachments: 3183-additions.patch, 3183-alt.patch, 3183.txt, CASSANDRA-3183-alt-v2.patch, CASSANDRA-3183-alt-v3.patch
>
>
> The serializing cache is a better choice for most users:
> - Lower total memory usage (serialized data is usually 8x-12x smaller than "live" data in the JVM with all the overhead that involves) means you can cache more rows for a given memory footprint
> - Moving the serialized rows off-heap means you can use smaller heaps, reducing the impact of GC pauses
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3183) Make SerializingCacheProvider the
default if JNA is available
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3183:
--------------------------------------
Attachment: 3183.txt
> Make SerializingCacheProvider the default if JNA is available
> -------------------------------------------------------------
>
> Key: CASSANDRA-3183
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3183
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 1.0.0
>
> Attachments: 3183.txt
>
>
> The serializing cache is a better choice for most users:
> - Lower total memory usage (serialized data is usually 8x-12x smaller than "live" data in the JVM with all the overhead that involves) means you can cache more rows for a given memory footprint
> - Moving the serialized rows off-heap means you can use smaller heaps, reducing the impact of GC pauses
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3183) Make SerializingCacheProvider the
default if JNA is available
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-3183:
---------------------------------------
Attachment: 3183-additions.patch
Ok, additions to fix broken CreateColumnFamilyStatement in CQL
and here is a problem I have discovered in your patch running `ant test -Dtest.name=CliTest`
{noformat}
[junit] Testcase: testCli(org.apache.cassandra.cli.CliTest): Caused an ERROR
[junit] null
[junit] java.lang.NullPointerException
[junit] at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:209)
[junit] at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:322)
[junit] at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:292)
[junit] at org.apache.cassandra.db.Table.initCf(Table.java:372)
[junit] at org.apache.cassandra.db.Table.<init>(Table.java:320)
[junit] at org.apache.cassandra.db.Table.open(Table.java:121)
[junit] at org.apache.cassandra.db.Table.open(Table.java:104)
[junit] at org.apache.cassandra.db.SystemTable.checkHealth(SystemTable.java:217)
[junit] at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:148)
[junit] at org.apache.cassandra.service.AbstractCassandraDaemon.init(AbstractCassandraDaemon.java:236)
[junit] at org.apache.cassandra.service.EmbeddedCassandraService.start(EmbeddedCassandraService.java:56)
[junit] at org.apache.cassandra.cli.CliTest.testCli(CliTest.java:211)
{noformat}
> Make SerializingCacheProvider the default if JNA is available
> -------------------------------------------------------------
>
> Key: CASSANDRA-3183
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3183
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 1.0.0
>
> Attachments: 3183-additions.patch, 3183-alt.patch, 3183.txt
>
>
> The serializing cache is a better choice for most users:
> - Lower total memory usage (serialized data is usually 8x-12x smaller than "live" data in the JVM with all the overhead that involves) means you can cache more rows for a given memory footprint
> - Moving the serialized rows off-heap means you can use smaller heaps, reducing the impact of GC pauses
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira