[jira] [Created] (CASSANDRA-15354) Cassandra CQLSSTableWriter and sstableloader support HDFS

["YAN Bo (Jira)" <ji...@apache.org>]

YAN Bo created CASSANDRA-15354:
----------------------------------

             Summary: Cassandra CQLSSTableWriter and sstableloader support HDFS
                 Key: CASSANDRA-15354
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-15354
             Project: Cassandra
          Issue Type: New Feature
          Components: Legacy/Local Write-Read Paths, Local/SSTable, Tool/sstable
            Reporter: YAN Bo


{code:java}
//代码占位符
rdd.foreachPartition( msgIterator => {
  val writer = CQLSSTableWriter.builder()
    .inDirectory(outputDir)
    // set target schema
    .forTable(SCHEMA)
    // set CQL statement to put data
    .using(INSERT_STMT)
    // set partitioner if needed
    // default is Murmur3Partitioner so set if you use different one.
    .withPartitioner( new Murmur3Partitioner()).build()
  msgIterator.foreach(msg => {
    val items = msg.toString().split(",")
    val  javaList = new util.ArrayList[Object]();
    items.foreach(t=> javaList.add(t))
    writer.addRow(javaList)
  })
  writer.close()
})
{code}
Cassandra has provided bulkdata's export/import via SSTable, which is very fancy for users. In some case we have TB-level data from HDFS to Cassandra, and we can use spark to generate SSTable files by distributed computation with codes like above. Unfortunately  CQLSSTableWriter can only write data to local path, and sstableloader can only load from local path. So if we use CQLSSTableWriter in Spark or Hadoop MR program, we need to write other codes put local sstables distributed in distributed nodes to HDFS, then download all sstables from HDFS to the machine with sstableloader,  bigdata stored and transferred between pysical machines will bring many reliability problems.  

 

So we'd better let CQLSSTableWriter can write data to HDFS directly or have other writer which supports HDFS, and let sstableloader can load from HDFS path.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org