You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "James Berragan (Jira)" <ji...@apache.org> on 2020/10/22 16:49:00 UTC
[jira] [Updated] (CASSANDRA-16222) Spark-Cassandra Bulk Reader
[ https://issues.apache.org/jira/browse/CASSANDRA-16222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
James Berragan updated CASSANDRA-16222:
---------------------------------------
Description:
*Description:*
This ticket introduces the Spark-Cassandra Bulk Reader: a Spark library able to read and compact Cassandra raw sstables into SparkSQL along the principles of streaming compaction. The full code is attached as a patch file and will be submitted to a GitHub repo.
**
*Motivation*
For analytics or "select *" use cases at scale, the performance is prohibitively expensive to read via the normal CQL read path - either using the Java driver or the Open Source Spark connector. By reading the raw sstables, the bulk reader is able to read with near-zero impact to a production cluster at speeds many orders of magnitudes faster than alternatives. We have seen very good performance results, exporting a 32TB table (~46bn CQL rows) to HDFS in Parquet format in around 1h10m; a 20x reduction compared to previous solutions. By reading from multiple replicas and ‘compacting’ duplicate data together, it can achieve consistency at a user defined level (i.e. ONE, TWO, LOCAL_QUORUM etc).
*Overview*
This library provides the core code for reading a set of SSTables into SparkSQL through a DataLayer abstraction. The role of the DataLayer is to:
* return a SchemaStruct, mapping the Cassandra CQL table schema to the SparkSQL schema.
* a list of sstables available for reading.
* a method to open an InputStream for any file component of an sstable (e.g. data, compression, summary etc).
The PartitionedDataLayer abstraction builds on the DataLayer interface for partitioning Spark workers across a Cassandra token ring - allowing the Spark job to scale linearly - and reading from sufficient Cassandra replicas to achieve a user-specified consistency level.
A simple example LocalDataLayer implementation is included for reading from a local file system. Users of the library can build their own implementations to read from wherever they wish e.g. reading from a backup in a cloud storage system, or reading from the snapshot directory of a live Cassandra cluster.
At the core, the Bulk Reader uses the Apache Cassandra CompactionIterator to perform the streaming compaction. As it iterates through the CompactionIterator it deserializes the ByteBuffers, converts into the appropriate SparkSQL data type and pivots each cell into a SparkSQL row.
Supporting this library is a robust property-based test framework for writing Cassandra sstables with arbitrary schemas using the CQLSSTableWriter, and reading back through SparkSQL to verify the library achieves both consistency and correctness.
was:
*Description:*
This ticket introduces the Spark-Cassandra Bulk Reader: a Spark library able to read and compact Cassandra raw sstables into SparkSQL along the principles of streaming compaction. The full code is attached as a patch file and will be submitted to a GitHub repo.
**
*Motivation*
For analytics or "select *" use cases at scale, the performance is prohibitively expensive to read via the normal CQL read path - either using the Java driver or the Open Source Spark connector. By reading the raw sstables, the bulk reader is able to read with near-zero impact to a production cluster at speeds many orders of magnitudes faster than alternatives. We have seen very good performance results, exporting a 32TB table (~46bn CQL rows) to HDFS in Parquet format in around 1h10m; a 20x reduction compared to previous solutions. By reading from multiple replicas and ‘compacting’ duplicate data together, it can achieve consistency at a user defined level (i.e. ONE, TWO, LOCAL_QUORUM etc).
*Overview*
This library provides the core code for reading a set of SSTables into SparkSQL through an a DataLayer abstraction. The role of the DataLayer is to:
* return a SchemaStruct, mapping the Cassandra CQL table schema to the SparkSQL schema.
* a list of sstables available for reading.
* a method to open an InputStream for any file component of an sstable (e.g. data, compression, summary etc).
The PartitionedDataLayer abstraction builds on the DataLayer interface for partitioning Spark workers across a Cassandra token ring - allowing the Spark job to scale linearly - and reading from sufficient Cassandra replicas to achieve a user-specified consistency level.
A simple example LocalDataLayer implementation is included for reading from a local file system. Users of the library can build their own implementations to read from wherever they wish e.g. reading from a backup in a cloud storage system, or reading from the snapshot directory of a live Cassandra cluster.
At the core, the Bulk Reader uses the Apache Cassandra CompactionIterator to perform the streaming compaction. As it iterates through the CompactionIterator it deserializes the ByteBuffers, converts into the appropriate SparkSQL data type and pivots each cell into a SparkSQL row.
Supporting this library is a robust property-based test framework for writing Cassandra sstables with arbitrary schemas using the CQLSSTableWriter, and reading back through SparkSQL to verify the library achieves both consistency and correctness.
> Spark-Cassandra Bulk Reader
> ---------------------------
>
> Key: CASSANDRA-16222
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16222
> Project: Cassandra
> Issue Type: New Feature
> Reporter: James Berragan
> Priority: Normal
> Attachments: sparkbulkreader.patch
>
>
> *Description:*
> This ticket introduces the Spark-Cassandra Bulk Reader: a Spark library able to read and compact Cassandra raw sstables into SparkSQL along the principles of streaming compaction. The full code is attached as a patch file and will be submitted to a GitHub repo.
> **
> *Motivation*
> For analytics or "select *" use cases at scale, the performance is prohibitively expensive to read via the normal CQL read path - either using the Java driver or the Open Source Spark connector. By reading the raw sstables, the bulk reader is able to read with near-zero impact to a production cluster at speeds many orders of magnitudes faster than alternatives. We have seen very good performance results, exporting a 32TB table (~46bn CQL rows) to HDFS in Parquet format in around 1h10m; a 20x reduction compared to previous solutions. By reading from multiple replicas and ‘compacting’ duplicate data together, it can achieve consistency at a user defined level (i.e. ONE, TWO, LOCAL_QUORUM etc).
> *Overview*
> This library provides the core code for reading a set of SSTables into SparkSQL through a DataLayer abstraction. The role of the DataLayer is to:
> * return a SchemaStruct, mapping the Cassandra CQL table schema to the SparkSQL schema.
> * a list of sstables available for reading.
> * a method to open an InputStream for any file component of an sstable (e.g. data, compression, summary etc).
> The PartitionedDataLayer abstraction builds on the DataLayer interface for partitioning Spark workers across a Cassandra token ring - allowing the Spark job to scale linearly - and reading from sufficient Cassandra replicas to achieve a user-specified consistency level.
> A simple example LocalDataLayer implementation is included for reading from a local file system. Users of the library can build their own implementations to read from wherever they wish e.g. reading from a backup in a cloud storage system, or reading from the snapshot directory of a live Cassandra cluster.
>
> At the core, the Bulk Reader uses the Apache Cassandra CompactionIterator to perform the streaming compaction. As it iterates through the CompactionIterator it deserializes the ByteBuffers, converts into the appropriate SparkSQL data type and pivots each cell into a SparkSQL row.
>
> Supporting this library is a robust property-based test framework for writing Cassandra sstables with arbitrary schemas using the CQLSSTableWriter, and reading back through SparkSQL to verify the library achieves both consistency and correctness.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org