You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by "Issac Buenrostro (JIRA)" <ji...@apache.org> on 2015/03/23 19:39:53 UTC

[jira] [Created] (PARQUET-224) Implement writing Parquet files into Cassandra natively

Issac Buenrostro created PARQUET-224:
----------------------------------------

             Summary: Implement writing Parquet files into Cassandra natively
                 Key: PARQUET-224
                 URL: https://issues.apache.org/jira/browse/PARQUET-224
             Project: Parquet
          Issue Type: New Feature
            Reporter: Issac Buenrostro
            Priority: Minor


Writing Parquet files into Cassandra could allow parallel writes of multiple pages into different cells, and low latency reads with a persistent connection to C*.

Each page could be written to separate C* cells, with metadata written into a separate column family.

A possible way of implementing is:

- abstract ParquetFileWriter -> ParquetDataWriter. writeDictionaryPage, writeDataPage are abstract methods.
- ParquetFileWriter implements ParquetDataWriter, writing the data to Hadoop compatible files.
- ParquetCassandraWriter implements ParquetDataWriter, writing data to Cassandra
-- for each page, metadata is written to Metadata CF, with key <parquet-file-name>:<row-chunk>:<column>:<page>
-- for each page, data is written to Data CF, with key <parquet-file-name>:<row-chunk>:<column>:<page>
-- footer is written to Metadata CF, with key <parquet-file-name>

- abstract ParquetFileReader -> ParquetDataReader. readNextRowGroup, readFooter are abstract methods. Chunk will also need to be abstract.
- ParquetFileReader implements ParquetDataReader, reading from Hadoop compatible files.
- ParquetCassandraReader implements ParquetDataReader, reading from Cassandra

- ParquetDataWriter and ParquetDataReader are instantiated through reflection.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)