[jira] [Commented] (CASSANDRA-7360) CQLSSTableWriter consumes all memory for table with compound primary key

["Benjamin Lerer (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-7360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14121104#comment-14121104 ] 

Benjamin Lerer commented on CASSANDRA-7360:
-------------------------------------------

I noticed 2 things:
1) I think that in BufferedWriter.close() the code should probably try to call super.close before trying to rethrow the exception. Just to make sure that we terminate as properly as possible.

2) I have the following error message in my log when I run the Unit tests:
Exception in thread "StorageServiceShutdownHook" java.lang.IllegalStateException: No configured daemon
	at org.apache.cassandra.service.StorageService.stopRPCServer(StorageService.java:309)
	at org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:381)
	at org.apache.cassandra.service.StorageService.access$3(StorageService.java:379)
	at org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:568)
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
	at java.lang.Thread.run(Thread.java:745)
I am not sure that it is a real problem but idealy it would be nice if we could get rid of it.

Personnaly, I think that I instead of storing the error I would have wrapped it into a UncheckedException (e.g. SyncException) that I would have caught in rawAddRow and rethrow as IOException as it is more direct. Nevertheless it is just a matter of taste in term of "non clean stuff" and I am fine with your solution.  

> CQLSSTableWriter consumes all memory for table with compound primary key
> ------------------------------------------------------------------------
>
>                 Key: CASSANDRA-7360
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7360
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Xu Zhongxing
>            Assignee: Sylvain Lebresne
>             Fix For: 2.0.11
>
>         Attachments: 7360.txt
>
>
> When using CQLSSTableWriter to write a table with compound primary key, if the partition key is identical for a huge amount of records, the sync() method is never called, and the memory usage keeps growing until the memory is exhausted. 
> Could the code be improved to do sync() even when there is no new row  created? The relevant code is in SSTableSimpleUnsortedWriter.java and AbstractSSTableSimpleWriter.java. I am new to the code and cannot produce a reasonable patch for now.
> The problem can be reproduced by the following test case:
> {code}
> import org.apache.cassandra.io.sstable.CQLSSTableWriter;
> import org.apache.cassandra.exceptions.InvalidRequestException;
> import java.io.IOException;
> import java.util.UUID;
> class SS {
>     public static void main(String[] args) {
>         String schema = "create table test.t (x uuid, y uuid, primary key (x, y))";
>         String insert = "insert into test.t (x, y) values (?, ?)";
>         CQLSSTableWriter writer = CQLSSTableWriter.builder()
>             .inDirectory("/tmp/test/t")
>             .forTable(schema).withBufferSizeInMB(32)
>             .using(insert).build();
>         UUID id = UUID.randomUUID();
>         try {
>             for (int i = 0; i < 50000000; i++) {
>                 UUID id2 = UUID.randomUUID();
>                 writer.addRow(id, id2);
>             }
>             writer.close();
>         } catch (Exception e) {
>             System.err.println("hell");
>         }
>     }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)