You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by Selina Tech <sw...@gmail.com> on 2015/11/06 02:56:33 UTC
Reading Parquet data from input stream and write to output stream
Dear all:
I am wondering if I could read input stream such as Kafka and convert
it Parquet data and write back to output stream? All example I found
convert data file to Parquet data.
I know this feature is not available last year. How about right now?
I am trying to aggregate Kafka message by Samza and convert it to
Parquet data and then save it to S3. What is the best one to implement it?
Sincerely,
Selina
reference:
https://github.com/Parquet/parquet-mr/issues/231
Re: Reading Parquet data from input stream and write to output stream
Posted by Ryan Blue <bl...@cloudera.com>.
Selina,
I would use parquet-avro to create a writer. Kafka messages are commonly
encoded as Avro, so you may already be working with Avro objects. If
not, then convert to Avro and then write to the AvroParquetWriter.
You can create a the writer that creates S3 files by setting up your S3
file system settings in a Configuration and then using paths that look
like this: s3n://s3bucket-name/path/within/bucket. You would just pass
that Path to the AvroParquetWriter.builder method, configure the
builder, and call build() to get a configured writer.
rb
On 11/09/2015 04:55 PM, Selina Tech wrote:
> Hi, Ryan:
>
> Thanks a lot for your suggestion. I do not have to get the
> output stream if I could write my continually Kafka message (in json,
> cvs or avro format) to AWS S3 in parquet format. Would you like to
> introduce a little bit more detail about it and then I find some
> solution in detail?
>
> There is one solution. Create a Parquet table by presto in
> Hive, and use Presto Hive connector to sql data and save the data to
> Hive and then send the data to S3. I am wondering if there is a better
> solution?
>
> Sincerely,
> Selina
>
> On Mon, Nov 9, 2015 at 9:35 AM, Ryan Blue <blue@cloudera.com
> <ma...@cloudera.com>> wrote:
>
> Selina,
>
> You should be able to write to S3 without needing to flush to an
> output stream. You would just use the S3 FileSystem to write data
> instead of HDFS. This doesn't need to require Parquet to write to an
> OutputStream instead of a file. Is there a reason why you want to
> supply an output stream instead?
>
> rb
>
>
> On 11/05/2015 05:56 PM, Selina Tech wrote:
>
> Dear all:
>
> I am wondering if I could read input stream such as
> Kafka and convert
> it Parquet data and write back to output stream? All example I
> found
> convert data file to Parquet data.
>
> I know this feature is not available last year. How
> about right now?
>
> I am trying to aggregate Kafka message by Samza and
> convert it to
> Parquet data and then save it to S3. What is the best one to
> implement it?
>
>
> Sincerely,
> Selina
>
> reference:
> https://github.com/Parquet/parquet-mr/issues/231
>
>
>
> --
> Ryan Blue
> Software Engineer
> Cloudera, Inc.
>
>
--
Ryan Blue
Software Engineer
Cloudera, Inc.
Re: Reading Parquet data from input stream and write to output stream
Posted by Selina Tech <sw...@gmail.com>.
Hi, Ryan:
Thanks a lot for your suggestion. I do not have to get the output
stream if I could write my continually Kafka message (in json, cvs or avro
format) to AWS S3 in parquet format. Would you like to introduce a little
bit more detail about it and then I find some solution in detail?
There is one solution. Create a Parquet table by presto in Hive,
and use Presto Hive connector to sql data and save the data to Hive and
then send the data to S3. I am wondering if there is a better solution?
Sincerely,
Selina
On Mon, Nov 9, 2015 at 9:35 AM, Ryan Blue <bl...@cloudera.com> wrote:
> Selina,
>
> You should be able to write to S3 without needing to flush to an output
> stream. You would just use the S3 FileSystem to write data instead of HDFS.
> This doesn't need to require Parquet to write to an OutputStream instead of
> a file. Is there a reason why you want to supply an output stream instead?
>
> rb
>
>
> On 11/05/2015 05:56 PM, Selina Tech wrote:
>
>> Dear all:
>>
>> I am wondering if I could read input stream such as Kafka and
>> convert
>> it Parquet data and write back to output stream? All example I found
>> convert data file to Parquet data.
>>
>> I know this feature is not available last year. How about right
>> now?
>>
>> I am trying to aggregate Kafka message by Samza and convert it to
>> Parquet data and then save it to S3. What is the best one to implement it?
>>
>>
>> Sincerely,
>> Selina
>>
>> reference:
>> https://github.com/Parquet/parquet-mr/issues/231
>>
>>
>
> --
> Ryan Blue
> Software Engineer
> Cloudera, Inc.
>
Re: Reading Parquet data from input stream and write to output stream
Posted by Ryan Blue <bl...@cloudera.com>.
Selina,
You should be able to write to S3 without needing to flush to an output
stream. You would just use the S3 FileSystem to write data instead of
HDFS. This doesn't need to require Parquet to write to an OutputStream
instead of a file. Is there a reason why you want to supply an output
stream instead?
rb
On 11/05/2015 05:56 PM, Selina Tech wrote:
> Dear all:
>
> I am wondering if I could read input stream such as Kafka and convert
> it Parquet data and write back to output stream? All example I found
> convert data file to Parquet data.
>
> I know this feature is not available last year. How about right now?
>
> I am trying to aggregate Kafka message by Samza and convert it to
> Parquet data and then save it to S3. What is the best one to implement it?
>
>
> Sincerely,
> Selina
>
> reference:
> https://github.com/Parquet/parquet-mr/issues/231
>
--
Ryan Blue
Software Engineer
Cloudera, Inc.