Re: CSV bulk loading using map reduce

[Gabriel Reid <ga...@gmail.com>]

Hi Noam,

I think that the things that most typically can affect MR loading
performance are:
* number of regions (as this affects the number of reducers used to
create the HFiles)
* amount of memory used for sort buffers
* use of compression on map output

With your 32-region salted table, it sounds like things should be
well-balanced, so that shouldn't be an issue.

I would expect the biggest impact you can possibly get is by tuning
the sort buffers. You can spot if there are potential issues here by
looking at the "Spilled records" counter in the list of job counters
in the job tracker. If you're getting multiple times the number of
spilled records vs the number of output records, then it'll probably
be advantageous to up the sort buffer settings. There's an old but
good overview of these concepts starting from slide 12 in this
presentation: http://www.slideshare.net/cloudera/mr-perf

About the compression on map output, the mapreduce.map.output.compress
should almost certainly be set to true, with (probably) snappy used
for compression. Having this set to true compresses the output of the
map task before it's sent across the network to the reducers. Since
HBase KeyValues are very bulky, having this setting turned on can have
quite an impact on throughput.

As far as the performance comparison with Impala, I assume you're
referring to impala that is backed by text files or Parquet, correct?

- Gabriel


On Tue, Dec 23, 2014 at 9:43 AM, Bulvik, Noam <No...@teoco.com> wrote:
> We are testing the CSV bulk loader tool (using MR) and trying to understand
> what are the parameter that can be used to enhanced its performance.
>
> we have 8 nodes with 4 disks each (1 core per disk) we created 32  regions
> and pre-split our table using salt, the table has 55 columns
>
>
>
> So far we managed to load ~40,000 rows per second (from csv file of ~ 8G),
> when we compare this loading to impala  loading  on the same cluster, same
> table structure we see rates of about 300,000 rows per seconds.
>
>
>
> Any idea which parameters of Hbase /Phoenix can we use to enhance the
> performance
>
>
>
> Regards,
>
> Noam
>
>
>
> Information in this e-mail and its attachments is confidential and
> privileged under the TEOCO confidentiality terms that can be reviewed here.

Re: CSV bulk loading using map reduce

[Gabriel Reid <ga...@gmail.com>]

Ok, thanks for confirming that the Impala table is backed by Parquet.

It's not surprising to hear that you can get higher throughput while
loading a Parquet table compared to Phoenix. Even when using Parquet,
the file structure used by Impala is quite lightweight compared to
HFiles. For example, each column in non-key row in Phoenix contains
the full row key and the name of the column qualifier, and all of
these need to be sorted before writing them as HFiles.

The trade-off is of course that you can query single rows extremely
quickly in HBase, whereas that won't be as performant in Impala.

By using the suggestions I gave, you will likely be able to improve
performance of the loading, but I think that creating parquet files
for Impala will always give substantially higher throughput.

- Gabriel


On Tue, Dec 23, 2014 at 1:40 PM, Bulvik, Noam <No...@teoco.com> wrote:
> Thanks for the answer we will look in to it and update
> The impala is impala parquet table
>
> -----Original Message-----
> From: Gabriel Reid [mailto:gabriel.reid@gmail.com]
> Sent: Tuesday, December 23, 2014 2:27 PM
> To: user@phoenix.apache.org
> Subject: Re: CSV bulk loading using map reduce
>
> Hi Noam,
>
> I think that the things that most typically can affect MR loading performance are:
> * number of regions (as this affects the number of reducers used to create the HFiles)
> * amount of memory used for sort buffers
> * use of compression on map output
>
> With your 32-region salted table, it sounds like things should be well-balanced, so that shouldn't be an issue.
>
> I would expect the biggest impact you can possibly get is by tuning the sort buffers. You can spot if there are potential issues here by looking at the "Spilled records" counter in the list of job counters in the job tracker. If you're getting multiple times the number of spilled records vs the number of output records, then it'll probably be advantageous to up the sort buffer settings. There's an old but good overview of these concepts starting from slide 12 in this
> presentation: http://www.slideshare.net/cloudera/mr-perf
>
> About the compression on map output, the mapreduce.map.output.compress should almost certainly be set to true, with (probably) snappy used for compression. Having this set to true compresses the output of the map task before it's sent across the network to the reducers. Since HBase KeyValues are very bulky, having this setting turned on can have quite an impact on throughput.
>
> As far as the performance comparison with Impala, I assume you're referring to impala that is backed by text files or Parquet, correct?
>
> - Gabriel
>
>
> On Tue, Dec 23, 2014 at 9:43 AM, Bulvik, Noam <No...@teoco.com> wrote:
>> We are testing the CSV bulk loader tool (using MR) and trying to
>> understand what are the parameter that can be used to enhanced its performance.
>>
>> we have 8 nodes with 4 disks each (1 core per disk) we created 32
>> regions and pre-split our table using salt, the table has 55 columns
>>
>>
>>
>> So far we managed to load ~40,000 rows per second (from csv file of ~
>> 8G), when we compare this loading to impala  loading  on the same
>> cluster, same table structure we see rates of about 300,000 rows per seconds.
>>
>>
>>
>> Any idea which parameters of Hbase /Phoenix can we use to enhance the
>> performance
>>
>>
>>
>> Regards,
>>
>> Noam
>>
>>
>>
>> Information in this e-mail and its attachments is confidential and
>> privileged under the TEOCO confidentiality terms that can be reviewed here.
> Information in this e-mail and its attachments is confidential and privileged under the TEOCO confidentiality terms that can be reviewed here<http://www.teoco.com/email-disclaimer>.

RE: CSV bulk loading using map reduce

["Bulvik, Noam" <No...@teoco.com>]

Thanks for the answer we will look in to it and update
The impala is impala parquet table

-----Original Message-----
From: Gabriel Reid [mailto:gabriel.reid@gmail.com]
Sent: Tuesday, December 23, 2014 2:27 PM
To: user@phoenix.apache.org
Subject: Re: CSV bulk loading using map reduce

Hi Noam,

I think that the things that most typically can affect MR loading performance are:
* number of regions (as this affects the number of reducers used to create the HFiles)
* amount of memory used for sort buffers
* use of compression on map output

With your 32-region salted table, it sounds like things should be well-balanced, so that shouldn't be an issue.

I would expect the biggest impact you can possibly get is by tuning the sort buffers. You can spot if there are potential issues here by looking at the "Spilled records" counter in the list of job counters in the job tracker. If you're getting multiple times the number of spilled records vs the number of output records, then it'll probably be advantageous to up the sort buffer settings. There's an old but good overview of these concepts starting from slide 12 in this
presentation: http://www.slideshare.net/cloudera/mr-perf

About the compression on map output, the mapreduce.map.output.compress should almost certainly be set to true, with (probably) snappy used for compression. Having this set to true compresses the output of the map task before it's sent across the network to the reducers. Since HBase KeyValues are very bulky, having this setting turned on can have quite an impact on throughput.

As far as the performance comparison with Impala, I assume you're referring to impala that is backed by text files or Parquet, correct?

- Gabriel


On Tue, Dec 23, 2014 at 9:43 AM, Bulvik, Noam <No...@teoco.com> wrote:
> We are testing the CSV bulk loader tool (using MR) and trying to
> understand what are the parameter that can be used to enhanced its performance.
>
> we have 8 nodes with 4 disks each (1 core per disk) we created 32
> regions and pre-split our table using salt, the table has 55 columns
>
>
>
> So far we managed to load ~40,000 rows per second (from csv file of ~
> 8G), when we compare this loading to impala  loading  on the same
> cluster, same table structure we see rates of about 300,000 rows per seconds.
>
>
>
> Any idea which parameters of Hbase /Phoenix can we use to enhance the
> performance
>
>
>
> Regards,
>
> Noam
>
>
>
> Information in this e-mail and its attachments is confidential and
> privileged under the TEOCO confidentiality terms that can be reviewed here.
Information in this e-mail and its attachments is confidential and privileged under the TEOCO confidentiality terms that can be reviewed here<http://www.teoco.com/email-disclaimer>.