questions about SSTableExport/Import

[aaron morton <aa...@thelastpickle.com>]

I was trying to help this guy http://www.mail-archive.com/user@cassandra.apache.org/msg07297.html who seemed to have troubles loading a json file. And I started taking a look at SSTableExport and SSTableImport.

SSTableExport does not encode any information about the Column sub type (ExpiringColumn or DeletedColumn). It records isMarkedForDelete(), the timestamp and the localDeletionTime as the col value if its a DeletedColumn. SSTableImport then calls either cf.addColumn() or cf.addTombstone() based on the deleted flag. 

First question is is the code in SSTableImport.addToStandardCF() correct to call cf.addColumn() if when the column was serialised it was isMarkedForDelete() ?

Next is it OK to lose the fact that a column is an ExpiringColumn (and its ttl) when it's exported to json? 

On my local machine I modified the unit test for SSTableExport as below and the assertion that the col was not returned failed.

Thanks
Aaron

diff --git a/test/unit/org/apache/cassandra/tools/SSTableExportTest.java b/test/unit/org/apache/cassandra/tools/SSTableExportTest.java
index 6f79f62..53d2a9c 100644
--- a/test/unit/org/apache/cassandra/tools/SSTableExportTest.java
+++ b/test/unit/org/apache/cassandra/tools/SSTableExportTest.java
@@ -179,6 +179,7 @@ public class SSTableExportTest extends SchemaLoader
         
         // Add rowA
         cfamily.addColumn(new QueryPath("Standard1", null, ByteBufferUtil.bytes("name")), ByteBufferUtil.bytes("val"), 1);
+        cfamily.addColumn(new QueryPath("Standard1", null, ByteBufferUtil.bytes("ttl")), ByteBufferUtil.bytes("val"), 1, 1);
         writer.append(Util.dk("rowA"), cfamily);
         cfamily.clear();
 
@@ -187,6 +188,15 @@ public class SSTableExportTest extends SchemaLoader
         writer.append(Util.dk("rowExclude"), cfamily);
         cfamily.clear();
 
+        //make sure the ttl col has expired
+        try
+        {
+            Thread.sleep(1500);
+        }
+        catch (InterruptedException e)
+        {
+            throw new AssertionError(e);
+        }
         SSTableReader reader = writer.closeAndOpenReader();
         
         // Export to JSON and verify
@@ -203,6 +213,11 @@ public class SSTableExportTest extends SchemaLoader
         assertTrue(cf != null);
         assertTrue(cf.getColumn(ByteBufferUtil.bytes("name")).value().equals(ByteBuffer.wrap(hexToBytes("76616c"))));
 
+        qf = QueryFilter.getNamesFilter(Util.dk("rowA"), new QueryPath("Standard1", null, null), ByteBufferUtil.bytes("ttl"));
+        cf = qf.getSSTableColumnIterator(reader).getColumnFamily();
+        assertTrue(cf != null);
+        assertTrue(cf.getColumn(ByteBufferUtil.bytes("ttl")) == null);
+

Re: questions about SSTableExport/Import

[Jonathan Ellis <jb...@gmail.com>]

On Wed, Nov 17, 2010 at 5:19 AM, aaron morton <aa...@thelastpickle.com> wrote:
> I was trying to help this guy http://www.mail-archive.com/user@cassandra.apache.org/msg07297.html who seemed to have troubles loading a json file. And I started taking a look at SSTableExport and SSTableImport.
>
> SSTableExport does not encode any information about the Column sub type (ExpiringColumn or DeletedColumn). It records isMarkedForDelete(), the timestamp and the localDeletionTime as the col value if its a DeletedColumn. SSTableImport then calls either cf.addColumn() or cf.addTombstone() based on the deleted flag.
>
> First question is is the code in SSTableImport.addToStandardCF() correct to call cf.addColumn() if when the column was serialised it was isMarkedForDelete() ?

That does look backwards.

> Next is it OK to lose the fact that a column is an ExpiringColumn (and its ttl) when it's exported to json?

It's okay in the sense that we make no effort to support TTL in json
export/import.  You're welcome to add that, but I don't consider it
being missing a bug.

-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com