You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by "Riesland, Zack" <Za...@sensus.com> on 2015/08/31 15:19:42 UTC
Help Tuning CsvBulkImport MapReduce
I'm looking for some pointers on speeding up CsvBulkImport.
Here's an example:
I took about 2 billion rows from hive and exported them to CSV.
HDFS decided to translate this to 257 files, each about 1 GB.
Running the CsvBulkImport tool against this folder results in 1,835 mappers and then 1 reducer per region on the HBase table.
The whole process takes something like 2 hours, the bulk of which is spent on mappers.
Any suggestions on how to possibly make this faster?
When I create the CSV files, I'm doing a pretty simple select statement from hive. The results tend to be mostly sorted.
I honestly don't know this space well enough to know whether that's good, bad, or neutral.
Thanks for any feedback!
Re: Help Tuning CsvBulkImport MapReduce
Posted by 丁桂涛 <di...@baixing.com>.
BTW, since hive represents NULL values as \\N in the textfile, how do you
handle those NULL values when using the CsvBulkImport tool?
On Tue, Sep 1, 2015 at 9:04 AM, Behdad Forghani <be...@exapackets.com>
wrote:
> Hi,
>
>
>
> In my experience the fastest way to load data is directly write to HFile.
> I have measured a performance gain of 10x. Also, if you have binary data or
> need to escape characters HBase bulk loader does not escape characters.
> For my use case, I create HFiles and load the HFIle. Then, I create a view
> on HBase table.
>
>
>
> Behdad
>
>
>
> *From:* Riesland, Zack [mailto:Zack.Riesland@sensus.com]
> *Sent:* Monday, August 31, 2015 6:20 AM
> *To:* user@phoenix.apache.org
> *Subject:* Help Tuning CsvBulkImport MapReduce
>
>
>
> I’m looking for some pointers on speeding up CsvBulkImport.
>
>
>
> Here’s an example:
>
>
>
> I took about 2 billion rows from hive and exported them to CSV.
>
>
>
> HDFS decided to translate this to 257 files, each about 1 GB.
>
>
>
> Running the CsvBulkImport tool against this folder results in 1,835
> mappers and then 1 reducer per region on the HBase table.
>
>
>
> The whole process takes something like 2 hours, the bulk of which is spent
> on mappers.
>
>
>
> Any suggestions on how to possibly make this faster?
>
>
>
> When I create the CSV files, I’m doing a pretty simple select statement
> from hive. The results tend to be mostly sorted.
>
>
>
> I honestly don’t know this space well enough to know whether that’s good,
> bad, or neutral.
>
>
>
> Thanks for any feedback!
>
>
>
Re: Help Tuning CsvBulkImport MapReduce
Posted by Behdad Forghani <be...@exapackets.com>.
Hi,
I thought I should have explained my use case after I sent the email. This
is not for the case where your data is already in CSV format rather than if
your application has a choice of writing to HBase or dumping the records to
CSV and bulk loading the resulting CSV files. In my case, my application
writes protocol traces and network switch logs to HBase. My choices were to
write the records to HFile directly, or, create CSV files and then bulk
loading these CSV files. As you can imagine, there is at least an extra
write and read to and from CSV file in the second case. Then, there is the
issue of handling binary data as TSV/CSV files are designed to be text.
I found writing to HFile took less time than writing to CSV files. In my
measurement, it took about 504 seconds just to create the TSV files and 378
seconds to create and load the HFiles. In both cases, I was using 16
parallel threads to write about 45 million records. I attributed the speed
of writing to HFiles being faster to SNAPPY and the fact that I could
directly write the binary data.
Regards,
Behdad
On Tue, Sep 1, 2015 at 2:14 AM, Gabriel Reid <ga...@gmail.com> wrote:
> On Tue, Sep 1, 2015 at 3:04 AM, Behdad Forghani <be...@exapackets.com>
> wrote:
>
> > In my experience the fastest way to load data is directly write to
> HFile. I
> > have measured a performance gain of 10x. Also, if you have binary data or
> > need to escape characters HBase bulk loader does not escape characters.
> For
> > my use case, I create HFiles and load the HFIle. Then, I create a view on
> > HBase table.
>
> The CSV bulk import tool[1] does write to HFiles in a MapReduce job.
> Are you saying that you've gotten 10x better performance than this
> tool? If so, it would certainly be interesting to hear about how you
> able to get such good performance. Or were you comparing to bulk
> loading via PSQL?
>
> 1. http://phoenix.apache.org/bulk_dataload.html
>
Re: Help Tuning CsvBulkImport MapReduce
Posted by Gabriel Reid <ga...@gmail.com>.
On Tue, Sep 1, 2015 at 3:04 AM, Behdad Forghani <be...@exapackets.com> wrote:
> In my experience the fastest way to load data is directly write to HFile. I
> have measured a performance gain of 10x. Also, if you have binary data or
> need to escape characters HBase bulk loader does not escape characters. For
> my use case, I create HFiles and load the HFIle. Then, I create a view on
> HBase table.
The CSV bulk import tool[1] does write to HFiles in a MapReduce job.
Are you saying that you've gotten 10x better performance than this
tool? If so, it would certainly be interesting to hear about how you
able to get such good performance. Or were you comparing to bulk
loading via PSQL?
1. http://phoenix.apache.org/bulk_dataload.html
RE: Help Tuning CsvBulkImport MapReduce
Posted by Behdad Forghani <be...@exapackets.com>.
Hi,
In my experience the fastest way to load data is directly write to HFile. I
have measured a performance gain of 10x. Also, if you have binary data or
need to escape characters HBase bulk loader does not escape characters. For
my use case, I create HFiles and load the HFIle. Then, I create a view on
HBase table.
Behdad
From: Riesland, Zack [mailto:Zack.Riesland@sensus.com]
Sent: Monday, August 31, 2015 6:20 AM
To: user@phoenix.apache.org
Subject: Help Tuning CsvBulkImport MapReduce
I'm looking for some pointers on speeding up CsvBulkImport.
Here's an example:
I took about 2 billion rows from hive and exported them to CSV.
HDFS decided to translate this to 257 files, each about 1 GB.
Running the CsvBulkImport tool against this folder results in 1,835 mappers
and then 1 reducer per region on the HBase table.
The whole process takes something like 2 hours, the bulk of which is spent
on mappers.
Any suggestions on how to possibly make this faster?
When I create the CSV files, I'm doing a pretty simple select statement from
hive. The results tend to be mostly sorted.
I honestly don't know this space well enough to know whether that's good,
bad, or neutral.
Thanks for any feedback!
RE: Help Tuning CsvBulkImport MapReduce
Posted by "Riesland, Zack" <Za...@sensus.com>.
Found it.
Never knew all those helpful metrics were there in the GUI!
Looks like we are right at the 2x threshold on our spilled records vs map output records ratio.
I will play with this this week.
Thanks again!
-----Original Message-----
From: Gabriel Reid [mailto:gabriel.reid@gmail.com]
Sent: Tuesday, September 01, 2015 6:43 AM
To: user@phoenix.apache.org
Subject: Re: Help Tuning CsvBulkImport MapReduce
On Tue, Sep 1, 2015 at 11:29 AM, Riesland, Zack <Za...@sensus.com> wrote:
> You say I can find information about spills in the job counters. Are
> you talking about “failed” map tasks, or is there something else that
> will help me identify spill scenarios?
"Spilled records" is a counter that is available at the job level and at individual task level -- you can see it in the Counters view of a job or task in the web interface of the YARN Resource Manager or History Server. This is a counter that will only be included if a task was successful (this is the case for all counters in MapReduce jobs).
"Map Output Records" is the other counter that you'll want to compare with this.
- Gabriel
Re: Help Tuning CsvBulkImport MapReduce
Posted by Gabriel Reid <ga...@gmail.com>.
On Tue, Sep 1, 2015 at 11:29 AM, Riesland, Zack
<Za...@sensus.com> wrote:
> You say I can find information about spills in the job counters. Are you
> talking about “failed” map tasks, or is there something else that will help
> me identify spill scenarios?
"Spilled records" is a counter that is available at the job level and
at individual task level -- you can see it in the Counters view of a
job or task in the web interface of the YARN Resource Manager or
History Server. This is a counter that will only be included if a task
was successful (this is the case for all counters in MapReduce jobs).
"Map Output Records" is the other counter that you'll want to compare
with this.
- Gabriel
RE: Help Tuning CsvBulkImport MapReduce
Posted by "Riesland, Zack" <Za...@sensus.com>.
Thanks Gabriel,
That is extremely helpful.
One clarification:
You say I can find information about spills in the job counters. Are you talking about “failed” map tasks, or is there something else that will help me identify spill scenarios?
From: Gabriel Reid [mailto:gabriel.reid@gmail.com]
Sent: Monday, August 31, 2015 4:39 PM
To: user@phoenix.apache.org
Subject: Re: Help Tuning CsvBulkImport MapReduce
If the bulk of the time is being spent in the map phase, then there probably isn't all that much that can be done in terms of tuning that will make a huge difference. However, there may be a few things to look at.
You mentioned that HDFS decided to translate the hive export to 257 files -- do you mean blocks, or are there actually 257 files on HDFS? If so, it's Hive (and/or MapReduce) that's responsible for the 257 files, but that's probably just a detail and not all that important.
How long are each of the map tasks taking? If they're only taking something like 30 seconds or so, then it would be worth trying to have each task process more data. This is most easily accomplished by using a bigger block size on HDFS, as each HDFS block typically results in a single map task. However, you'll want to first check how long each map task is taking -- if they're each taking 3-5 minutes (or more), then you won't gain much by increasing the block size.
A second thing to look at is the number of spills compared to the number of map output records -- you can find this information in the job counters. If the number of spills in the map phase are two (or more) times the number of map output records, you'll likely get an increase in performance by upping the mapreduce.task.io.sort.mb setting (or some other sort settings). However, before getting into this you'll want to see if the spills are an issue.
As I said above though, most likely if the map phase is taking up the most time, it's probably CPU-bound on the conversion of CSV data to HBase KeyValues. This is likely if you're dealing with really wide rows. How many columns are you importing into your table?
- Gabriel
On Mon, Aug 31, 2015 at 3:20 PM Riesland, Zack <Za...@sensus.com>> wrote:
I’m looking for some pointers on speeding up CsvBulkImport.
Here’s an example:
I took about 2 billion rows from hive and exported them to CSV.
HDFS decided to translate this to 257 files, each about 1 GB.
Running the CsvBulkImport tool against this folder results in 1,835 mappers and then 1 reducer per region on the HBase table.
The whole process takes something like 2 hours, the bulk of which is spent on mappers.
Any suggestions on how to possibly make this faster?
When I create the CSV files, I’m doing a pretty simple select statement from hive. The results tend to be mostly sorted.
I honestly don’t know this space well enough to know whether that’s good, bad, or neutral.
Thanks for any feedback!
Re: Help Tuning CsvBulkImport MapReduce
Posted by Gabriel Reid <ga...@gmail.com>.
If the bulk of the time is being spent in the map phase, then there
probably isn't all that much that can be done in terms of tuning that will
make a huge difference. However, there may be a few things to look at.
You mentioned that HDFS decided to translate the hive export to 257 files
-- do you mean blocks, or are there actually 257 files on HDFS? If so, it's
Hive (and/or MapReduce) that's responsible for the 257 files, but that's
probably just a detail and not all that important.
How long are each of the map tasks taking? If they're only taking something
like 30 seconds or so, then it would be worth trying to have each task
process more data. This is most easily accomplished by using a bigger block
size on HDFS, as each HDFS block typically results in a single map task.
However, you'll want to first check how long each map task is taking -- if
they're each taking 3-5 minutes (or more), then you won't gain much by
increasing the block size.
A second thing to look at is the number of spills compared to the number of
map output records -- you can find this information in the job counters. If
the number of spills in the map phase are two (or more) times the number of
map output records, you'll likely get an increase in performance by upping
the mapreduce.task.io.sort.mb setting (or some other sort settings).
However, before getting into this you'll want to see if the spills are an
issue.
As I said above though, most likely if the map phase is taking up the most
time, it's probably CPU-bound on the conversion of CSV data to HBase
KeyValues. This is likely if you're dealing with really wide rows. How many
columns are you importing into your table?
- Gabriel
On Mon, Aug 31, 2015 at 3:20 PM Riesland, Zack <Za...@sensus.com>
wrote:
> I’m looking for some pointers on speeding up CsvBulkImport.
>
>
>
> Here’s an example:
>
>
>
> I took about 2 billion rows from hive and exported them to CSV.
>
>
>
> HDFS decided to translate this to 257 files, each about 1 GB.
>
>
>
> Running the CsvBulkImport tool against this folder results in 1,835
> mappers and then 1 reducer per region on the HBase table.
>
>
>
> The whole process takes something like 2 hours, the bulk of which is spent
> on mappers.
>
>
>
> Any suggestions on how to possibly make this faster?
>
>
>
> When I create the CSV files, I’m doing a pretty simple select statement
> from hive. The results tend to be mostly sorted.
>
>
>
> I honestly don’t know this space well enough to know whether that’s good,
> bad, or neutral.
>
>
>
> Thanks for any feedback!
>
>
>