You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Kevin Tran <ke...@gmail.com> on 2016/08/28 14:43:51 UTC
Best practises to storing data in Parquet files
Hi,
Does anyone know what is the best practises to store data to parquet file?
Does parquet file has limit in size ( 1TB ) ?
Should we use SaveMode.APPEND for long running streaming app ?
How should we store in HDFS (directory structure, ... )?
Thanks,
Kevin.
Re: Best practises to storing data in Parquet files
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi Kevin.
When you say Kafka interacting with Oracle database (if I understand you
correctly) are you using GoldenGate with Kafka interface to push data from
Oracle to Kafka?
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 29 August 2016 at 03:23, Chanh Le <gi...@gmail.com> wrote:
> > Does parquet file has limit in size ( 1TB ) ?
> I did’t see any problem but 1TB is too big to operation need to divide
> into small pieces.
> > Should we use SaveMode.APPEND for long running streaming app ?
> Yes, but you need to partition it by time so it easy to maintain like
> update or delete a specific time without rebuild them all.
> > How should we store in HDFS (directory structure, ... )?
> Should partition the file into small pieces.
>
>
> > On Aug 28, 2016, at 9:43 PM, Kevin Tran <ke...@gmail.com> wrote:
> >
> > Hi,
> > Does anyone know what is the best practises to store data to parquet
> file?
> > Does parquet file has limit in size ( 1TB ) ?
> > Should we use SaveMode.APPEND for long running streaming app ?
> > How should we store in HDFS (directory structure, ... )?
> >
> > Thanks,
> > Kevin.
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
Re: Best practises to storing data in Parquet files
Posted by Chanh Le <gi...@gmail.com>.
> Does parquet file has limit in size ( 1TB ) ?
I did’t see any problem but 1TB is too big to operation need to divide into small pieces.
> Should we use SaveMode.APPEND for long running streaming app ?
Yes, but you need to partition it by time so it easy to maintain like update or delete a specific time without rebuild them all.
> How should we store in HDFS (directory structure, ... )?
Should partition the file into small pieces.
> On Aug 28, 2016, at 9:43 PM, Kevin Tran <ke...@gmail.com> wrote:
>
> Hi,
> Does anyone know what is the best practises to store data to parquet file?
> Does parquet file has limit in size ( 1TB ) ?
> Should we use SaveMode.APPEND for long running streaming app ?
> How should we store in HDFS (directory structure, ... )?
>
> Thanks,
> Kevin.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Best practises to storing data in Parquet files
Posted by Kevin Tran <ke...@gmail.com>.
Hi Mich,
My stack is as following:
Data sources:
* IBM MQ
* Oracle database
Kafka to store all messages from data sources
Spark Streaming fetching messages from Kafka and do a bit transform and
write parquet files to HDFS
Hive / SparkSQL / Impala will query on parquet files.
Do you have any reference architecture which HBase is apart of ?
Please share with me best practises you might know or your favourite
designs.
Thanks,
Kevin.
On Mon, Aug 29, 2016 at 5:18 AM, Mich Talebzadeh <mi...@gmail.com>
wrote:
> Hi,
>
> Can you explain about you particular stack.
>
> Example what is the source of streaming data and the role that Spark plays.
>
> Are you dealing with Real Time and Batch and why Parquet and not something
> like Hbase to ingest data real time.
>
> HTH
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 28 August 2016 at 15:43, Kevin Tran <ke...@gmail.com> wrote:
>
>> Hi,
>> Does anyone know what is the best practises to store data to parquet file?
>> Does parquet file has limit in size ( 1TB ) ?
>> Should we use SaveMode.APPEND for long running streaming app ?
>> How should we store in HDFS (directory structure, ... )?
>>
>> Thanks,
>> Kevin.
>>
>
>
Re: Best practises to storing data in Parquet files
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi,
Can you explain about you particular stack.
Example what is the source of streaming data and the role that Spark plays.
Are you dealing with Real Time and Batch and why Parquet and not something
like Hbase to ingest data real time.
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 28 August 2016 at 15:43, Kevin Tran <ke...@gmail.com> wrote:
> Hi,
> Does anyone know what is the best practises to store data to parquet file?
> Does parquet file has limit in size ( 1TB ) ?
> Should we use SaveMode.APPEND for long running streaming app ?
> How should we store in HDFS (directory structure, ... )?
>
> Thanks,
> Kevin.
>