You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Vitalii Tymchyshyn (Created) (JIRA)" <ji...@apache.org> on 2012/01/13 13:34:39 UTC
[jira] [Created] (CASSANDRA-3741) OOMs because delete operations
are not accounted
OOMs because delete operations are not accounted
------------------------------------------------
Key: CASSANDRA-3741
URL: https://issues.apache.org/jira/browse/CASSANDRA-3741
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1
Environment: FreeBSD
Reporter: Vitalii Tymchyshyn
Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key.
I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager.
This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3741) OOMs because delete operations
are not accounted
Posted by "Vitalii Tymchyshyn (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186448#comment-13186448 ]
Vitalii Tymchyshyn commented on CASSANDRA-3741:
-----------------------------------------------
Throughput size for deletion IS 0 for me. I am not deleting for some columns, but all columns in a key and operation has 0 columns in this case, this means 0 throughput so I did introduce an overhead for memory operation storage that looks good for me (20 bytes is ~ConcurrentMap size).
To note: I was getting OOM on startup/log replay.
The problem is that I have started delete-only workload from some Column Families and this won't change as all inserts go into another column families: we are moving. And for full-key delete-only throughput is 0
> OOMs because delete operations are not accounted
> ------------------------------------------------
>
> Key: CASSANDRA-3741
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3741
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1
> Environment: FreeBSD
> Reporter: Vitalii Tymchyshyn
>
> Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key.
> I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager.
> This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3741) OOMs because delete operations
are not accounted
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458860#comment-13458860 ]
Jonathan Ellis commented on CASSANDRA-3741:
-------------------------------------------
No, this is a sensitive area of the code and we don't want to risk destabilizing it.
> OOMs because delete operations are not accounted
> ------------------------------------------------
>
> Key: CASSANDRA-3741
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3741
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: FreeBSD
> Reporter: Vitalii Tymchyshyn
> Assignee: Andriy Kolyadenko
> Fix For: 1.1.1
>
>
> Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key.
> I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager.
> This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3741) OOMs because delete operations
are not accounted
Posted by "Radim Kolar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185836#comment-13185836 ]
Radim Kolar commented on CASSANDRA-3741:
----------------------------------------
How can you OOM if you replace large inserts with small deletes?
> OOMs because delete operations are not accounted
> ------------------------------------------------
>
> Key: CASSANDRA-3741
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3741
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1
> Environment: FreeBSD
> Reporter: Vitalii Tymchyshyn
>
> Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key.
> I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager.
> This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3741) OOMs because delete operations
are not accounted
Posted by "Andriy Kolyadenko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267238#comment-13267238 ]
Andriy Kolyadenko commented on CASSANDRA-3741:
----------------------------------------------
Another attempt to fix this: https://github.com/apache/cassandra/pull/10
Please consider my patch. It's very annoying to have Cassandra dying in such situations.
> OOMs because delete operations are not accounted
> ------------------------------------------------
>
> Key: CASSANDRA-3741
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3741
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: FreeBSD
> Reporter: Vitalii Tymchyshyn
>
> Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key.
> I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager.
> This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3741) OOMs because delete operations
are not accounted
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267783#comment-13267783 ]
Jonathan Ellis commented on CASSANDRA-3741:
-------------------------------------------
Intent to contribute in public is enough. github pull requests (and patches sent to the mailing list for that matter) are fine.
For the purposes of record-keeping though, we like to associate these with Jira tickets.
> OOMs because delete operations are not accounted
> ------------------------------------------------
>
> Key: CASSANDRA-3741
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3741
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: FreeBSD
> Reporter: Vitalii Tymchyshyn
> Assignee: Andriy Kolyadenko
> Fix For: 1.1.1
>
>
> Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key.
> I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager.
> This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3741) OOMs because delete operations
are not accounted
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185742#comment-13185742 ]
Jonathan Ellis commented on CASSANDRA-3741:
-------------------------------------------
Thanks, Vitalii!
Unfortunately we can't use that patch as is because adding ops * 20 in there is going to throw off the size calculation for other workloads.
Note that the "throughput" size for a deletion is NOT zero (see Column.size implementation). It sounds like you abruptly changed your workload from doing a bunch of larger inserts, then hit it with a ton of deletes all at once and OOMed before it was able to update its liveRatio estimate.
So the real problem is that if you change workloads dramatically enough, Cassandra's estimates can be off.
> OOMs because delete operations are not accounted
> ------------------------------------------------
>
> Key: CASSANDRA-3741
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3741
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1
> Environment: FreeBSD
> Reporter: Vitalii Tymchyshyn
>
> Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key.
> I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager.
> This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3741) OOMs because delete operations
are not accounted
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3741:
--------------------------------------
Fix Version/s: (was: 1.2)
1.1.1
> OOMs because delete operations are not accounted
> ------------------------------------------------
>
> Key: CASSANDRA-3741
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3741
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: FreeBSD
> Reporter: Vitalii Tymchyshyn
> Assignee: Andriy Kolyadenko
> Fix For: 1.1.1
>
>
> Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key.
> I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager.
> This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3741) OOMs because delete operations
are not accounted
Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458849#comment-13458849 ]
Radim Kolar commented on CASSANDRA-3741:
----------------------------------------
was this backported to cassandra 1.0?
> OOMs because delete operations are not accounted
> ------------------------------------------------
>
> Key: CASSANDRA-3741
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3741
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: FreeBSD
> Reporter: Vitalii Tymchyshyn
> Assignee: Andriy Kolyadenko
> Fix For: 1.1.1
>
>
> Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key.
> I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager.
> This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3741) OOMs because delete operations
are not accounted
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3741:
--------------------------------------
Affects Version/s: (was: 1.1.0)
1.0.0
> OOMs because delete operations are not accounted
> ------------------------------------------------
>
> Key: CASSANDRA-3741
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3741
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: FreeBSD
> Reporter: Vitalii Tymchyshyn
> Assignee: Andriy Kolyadenko
> Fix For: 1.2
>
>
> Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key.
> I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager.
> This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3741) OOMs because delete operations
are not accounted
Posted by "Jeremiah Jordan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267781#comment-13267781 ]
Jeremiah Jordan commented on CASSANDRA-3741:
--------------------------------------------
Procedural question, for git stuff should we still be attaching patches to JIRA? Or is a link to the github diff enough?
> OOMs because delete operations are not accounted
> ------------------------------------------------
>
> Key: CASSANDRA-3741
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3741
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: FreeBSD
> Reporter: Vitalii Tymchyshyn
> Assignee: Andriy Kolyadenko
> Fix For: 1.1.1
>
>
> Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key.
> I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager.
> This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3741) OOMs because delete operations
are not accounted
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267506#comment-13267506 ]
Jonathan Ellis commented on CASSANDRA-3741:
-------------------------------------------
Thanks, Andriy. You (and Vitalii) are right; whole-row deletions did indeed have zero throughput because of that behavior.
Committed with a change to 12 bytes (long + int from deletion info) to match what we do with Column sizes. (We measure the serialized size in Memtable.currentThroughput, then multiply by liveRatio to get a better estimate of the in-memory size. Mixing internal overhead as in CSLM would actually double count that. I've also introduced CASSANDRA-4215 to clean this up more for 1.2.
> OOMs because delete operations are not accounted
> ------------------------------------------------
>
> Key: CASSANDRA-3741
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3741
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: FreeBSD
> Reporter: Vitalii Tymchyshyn
>
> Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key.
> I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager.
> This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3741) OOMs because delete operations
are not accounted
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267510#comment-13267510 ]
Jonathan Ellis commented on CASSANDRA-3741:
-------------------------------------------
(That said, truncate is still the right solution for mass deletes.)
> OOMs because delete operations are not accounted
> ------------------------------------------------
>
> Key: CASSANDRA-3741
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3741
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: FreeBSD
> Reporter: Vitalii Tymchyshyn
>
> Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key.
> I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager.
> This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-3741) OOMs because delete operations
are not accounted
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-3741.
---------------------------------------
Resolution: Fixed
Fix Version/s: 1.2
Reviewer: jbellis
Assignee: Andriy Kolyadenko
> OOMs because delete operations are not accounted
> ------------------------------------------------
>
> Key: CASSANDRA-3741
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3741
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: FreeBSD
> Reporter: Vitalii Tymchyshyn
> Assignee: Andriy Kolyadenko
> Fix For: 1.2
>
>
> Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key.
> I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager.
> This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3741) OOMs because delete operations
are not accounted
Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458932#comment-13458932 ]
Radim Kolar commented on CASSANDRA-3741:
----------------------------------------
1.0 is already destabilized by this bug. If you want to have minimal effect, then add just 1 byte to live bytes count for every delete. In non delete only workload, it will have minimal effect.
Its important to have non zero count otherwise it will not be flushed to disk on memory pressure.
> OOMs because delete operations are not accounted
> ------------------------------------------------
>
> Key: CASSANDRA-3741
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3741
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: FreeBSD
> Reporter: Vitalii Tymchyshyn
> Assignee: Andriy Kolyadenko
> Fix For: 1.1.1
>
>
> Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key.
> I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager.
> This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3741) OOMs because delete operations
are not accounted
Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461662#comment-13461662 ]
Radim Kolar commented on CASSANDRA-3741:
----------------------------------------
If you do not want to get fixed, then it should be documented as known bug in NEWS.TXT
> OOMs because delete operations are not accounted
> ------------------------------------------------
>
> Key: CASSANDRA-3741
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3741
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: FreeBSD
> Reporter: Vitalii Tymchyshyn
> Assignee: Andriy Kolyadenko
> Fix For: 1.1.1
>
>
> Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key.
> I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager.
> This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira