You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by milindshah <mi...@gmail.com> on 2015/09/03 22:40:30 UTC

Multiwal performance with HBase 1.x

Hi

I have a generic question regarding the performance improvement that
Multiwal should provide over single WAL. Has anyone tried to measure the
performance gain while loading a large data set , about 1-2TB size on a
cluster? I would appreciate if someone could share numbers and observed
improvement. I would like to try this on a 10 node cluster with 12 drives
each and load 2 TB of data using YCSB.

Thanks,
Milind



--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Multiwal-performance-with-HBase-1-x-tp4074403.html
Sent from the HBase User mailing list archive at Nabble.com.

Re: Multiwal performance with HBase 1.x

Posted by Jingcheng Du <ji...@intel.com>.
Ted, I ran the tests several months ago, I can re-ran the cases again and
share them.



--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Multiwal-performance-with-HBase-1-x-tp4074403p4074715.html
Sent from the HBase User mailing list archive at Nabble.com.

Re: Multiwal performance with HBase 1.x

Posted by Jingcheng Du <ji...@intel.com>.
Thanks for comments, and so sorry to use settings that are not practical for
productions without telling the context in advance:)
Actually I wanted to see what multiple wal could do with less impact of
other factors in HBase:)
I can re-test the cases with more reasonable settings if needed. If you guys
may suggest the correct settings in the non-default configuration list I
gave, I'll be very very thankful.



--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Multiwal-performance-with-HBase-1-x-tp4074403p4074754.html
Sent from the HBase User mailing list archive at Nabble.com.

Re: Multiwal performance with HBase 1.x

Posted by Yu Li <ca...@gmail.com>.
@Ted, Sure, I've opened HBASE-14457
<https://issues.apache.org/jira/browse/HBASE-14457> as an umbrella for all
works done on multiwal and please allow me to give a more detailed sharing
there, in format of documents rather than emails. :-)

@Jingcheng and @Vlad, any comments/sharing from you will be warmly welcomed
in the JIRA. :-)

Best Regards,
Yu

On 21 September 2015 at 19:35, Ted Yu <yu...@gmail.com> wrote:

> Thanks for sharing, Yu.
>
> Images didn't go through. Can you use third party site for sharing ?
>
> Cheers
>
> > On Sep 20, 2015, at 11:09 PM, Yu Li <ca...@gmail.com> wrote:
> >
> > Hi Vlad,
> >
> > >> the existing write performance is more than adequate (avg load per RS
> usually less than 1MB/sec)
> > We have some different user scenarios and I'd like to share with you. We
> are using hbase to store data for building search index, and features like
> pv/uv of each online item will be recorded, so the write load would reach
> as high as 10MB/s (below is a screenshot of the ganglia metrics data) per
> RS. OTOH, as a database I think the online write performance of HBase is as
> important as read, bulkload is for offline and it cannot resolve all
> problem.
> >
> >
> > Another advantage of using multiple wal is that we could do
> user/business level isolation on wal. For example you could use one
> namespace per business and one wal group per namespace, and you could
> replicate only the data for the business in need.
> >
> > Regarding compaction IO, as I mentioned before, we could use tiered
> storage to prevent compaction to affect wal sync. This way we've observed
> an obvious improvement on the avg mutate RT, from 0.5ms to 0.3ms on our
> online cluster, FYI.
> >
> > Best Regards,
> > Yu
> >
> >> On 19 September 2015 at 00:55, Vladimir Rodionov <
> vladrodionov@gmail.com> wrote:
> >> Hi, Jingcheng
> >>
> >> You postpone compaction until your test completes by setting number of
> >> blocking stores to 120. That is kind of cheating :)
> >> As I said previously, in a long run, compaction rules the world - not
> >> number of wal files. In a real production setting, the existing write
> >> performance
> >> is more than adequate (avg load per RS usually less than 1MB/sec).
> Multiwal
> >> has probably its value if someone need to load quick large volume of
> data,
> >> but ... why do not use bulk load instead?
> >>
> >> Thank for letting us know that beefy servers with 8 SSDs can sustain
> such a
> >> huge load.
> >>
> >> -Vlad
> >>
> >>
> >>
> >> On Thu, Sep 17, 2015 at 10:30 PM, Jingcheng Du <ji...@intel.com>
> >> wrote:
> >>
> >> > More information for the test.
> >> > I use ycsb 0.3.0 for the test.
> >> > The command line is "./ycsb load hbase-10 -P ../workloads/workload
> -threads
> >> > 200 -p columnfamily=family -p clientbuffering=true -s > workload.dat"
> >> > The workload is, the data size is slightly less than 1TB:
> >> > fieldcount=5
> >> > fieldlength=200
> >> > recordcount=1000000000
> >> > maxexecutiontime=86400
> >> >
> >> >
> >> >
> >> > --
> >> > View this message in context:
> >> >
> http://apache-hbase.679495.n3.nabble.com/Multiwal-performance-with-HBase-1-x-tp4074403p4074731.html
> >> > Sent from the HBase User mailing list archive at Nabble.com.
> >> >
> >
>

Re: Multiwal performance with HBase 1.x

Posted by Ted Yu <yu...@gmail.com>.
Thanks for sharing, Yu. 

Images didn't go through. Can you use third party site for sharing ?

Cheers

> On Sep 20, 2015, at 11:09 PM, Yu Li <ca...@gmail.com> wrote:
> 
> Hi Vlad,
> 
> >> the existing write performance is more than adequate (avg load per RS usually less than 1MB/sec)
> We have some different user scenarios and I'd like to share with you. We are using hbase to store data for building search index, and features like pv/uv of each online item will be recorded, so the write load would reach as high as 10MB/s (below is a screenshot of the ganglia metrics data) per RS. OTOH, as a database I think the online write performance of HBase is as important as read, bulkload is for offline and it cannot resolve all problem.
> 
> 
> Another advantage of using multiple wal is that we could do user/business level isolation on wal. For example you could use one namespace per business and one wal group per namespace, and you could replicate only the data for the business in need.
> 
> Regarding compaction IO, as I mentioned before, we could use tiered storage to prevent compaction to affect wal sync. This way we've observed an obvious improvement on the avg mutate RT, from 0.5ms to 0.3ms on our online cluster, FYI.
> 
> Best Regards,
> Yu
> 
>> On 19 September 2015 at 00:55, Vladimir Rodionov <vl...@gmail.com> wrote:
>> Hi, Jingcheng
>> 
>> You postpone compaction until your test completes by setting number of
>> blocking stores to 120. That is kind of cheating :)
>> As I said previously, in a long run, compaction rules the world - not
>> number of wal files. In a real production setting, the existing write
>> performance
>> is more than adequate (avg load per RS usually less than 1MB/sec). Multiwal
>> has probably its value if someone need to load quick large volume of data,
>> but ... why do not use bulk load instead?
>> 
>> Thank for letting us know that beefy servers with 8 SSDs can sustain such a
>> huge load.
>> 
>> -Vlad
>> 
>> 
>> 
>> On Thu, Sep 17, 2015 at 10:30 PM, Jingcheng Du <ji...@intel.com>
>> wrote:
>> 
>> > More information for the test.
>> > I use ycsb 0.3.0 for the test.
>> > The command line is "./ycsb load hbase-10 -P ../workloads/workload -threads
>> > 200 -p columnfamily=family -p clientbuffering=true -s > workload.dat"
>> > The workload is, the data size is slightly less than 1TB:
>> > fieldcount=5
>> > fieldlength=200
>> > recordcount=1000000000
>> > maxexecutiontime=86400
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> > http://apache-hbase.679495.n3.nabble.com/Multiwal-performance-with-HBase-1-x-tp4074403p4074731.html
>> > Sent from the HBase User mailing list archive at Nabble.com.
>> >
> 

Re: Multiwal performance with HBase 1.x

Posted by Yu Li <ca...@gmail.com>.
Hi Vlad,

>> the existing write performance is more than adequate (avg load per RS
usually less than 1MB/sec)
We have some different user scenarios and I'd like to share with you. We
are using hbase to store data for building search index, and features like
pv/uv of each online item will be recorded, so the write load would reach
as high as 10MB/s (below is a screenshot of the ganglia metrics data) per
RS. OTOH, as a database I think the online write performance of HBase is as
important as read, bulkload is for offline and it cannot resolve all
problem.
[image: Inline images 2][image: Inline images 3]

Another advantage of using multiple wal is that we could do user/business
level isolation on wal. For example you could use one namespace per
business and one wal group per namespace, and you could replicate only the
data for the business in need.

Regarding compaction IO, as I mentioned before, we could use tiered storage
to prevent compaction to affect wal sync. This way we've observed an
obvious improvement on the avg mutate RT, from 0.5ms to 0.3ms on our online
cluster, FYI.

Best Regards,
Yu

On 19 September 2015 at 00:55, Vladimir Rodionov <vl...@gmail.com>
wrote:

> Hi, Jingcheng
>
> You postpone compaction until your test completes by setting number of
> blocking stores to 120. That is kind of cheating :)
> As I said previously, in a long run, compaction rules the world - not
> number of wal files. In a real production setting, the existing write
> performance
> is more than adequate (avg load per RS usually less than 1MB/sec). Multiwal
> has probably its value if someone need to load quick large volume of data,
> but ... why do not use bulk load instead?
>
> Thank for letting us know that beefy servers with 8 SSDs can sustain such a
> huge load.
>
> -Vlad
>
>
>
> On Thu, Sep 17, 2015 at 10:30 PM, Jingcheng Du <ji...@intel.com>
> wrote:
>
> > More information for the test.
> > I use ycsb 0.3.0 for the test.
> > The command line is "./ycsb load hbase-10 -P ../workloads/workload
> -threads
> > 200 -p columnfamily=family -p clientbuffering=true -s > workload.dat"
> > The workload is, the data size is slightly less than 1TB:
> > fieldcount=5
> > fieldlength=200
> > recordcount=1000000000
> > maxexecutiontime=86400
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-hbase.679495.n3.nabble.com/Multiwal-performance-with-HBase-1-x-tp4074403p4074731.html
> > Sent from the HBase User mailing list archive at Nabble.com.
> >
>

Re: Multiwal performance with HBase 1.x

Posted by Jingcheng Du <ji...@intel.com>.
Thanks Andy for suggestions.
I will use the suggested settings to re-do the cases.



--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Multiwal-performance-with-HBase-1-x-tp4074403p4074755.html
Sent from the HBase User mailing list archive at Nabble.com.

Re: Multiwal performance with HBase 1.x

Posted by Andrew Purtell <ap...@apache.org>.
> You postpone compaction until your test completes by setting number of blocking
stores to 120.

We don't recommend a setting like that for production. I'm not sure what
the consensus would be but in our production that setting is 20. The
default is half that so is very likely typical.



On Fri, Sep 18, 2015 at 9:55 AM, Vladimir Rodionov <vl...@gmail.com>
wrote:

> Hi, Jingcheng
>
> You postpone compaction until your test completes by setting number of
> blocking stores to 120. That is kind of cheating :)
> As I said previously, in a long run, compaction rules the world - not
> number of wal files. In a real production setting, the existing write
> performance
> is more than adequate (avg load per RS usually less than 1MB/sec). Multiwal
> has probably its value if someone need to load quick large volume of data,
> but ... why do not use bulk load instead?
>
> Thank for letting us know that beefy servers with 8 SSDs can sustain such a
> huge load.
>
> -Vlad
>
>
>
> On Thu, Sep 17, 2015 at 10:30 PM, Jingcheng Du <ji...@intel.com>
> wrote:
>
> > More information for the test.
> > I use ycsb 0.3.0 for the test.
> > The command line is "./ycsb load hbase-10 -P ../workloads/workload
> -threads
> > 200 -p columnfamily=family -p clientbuffering=true -s > workload.dat"
> > The workload is, the data size is slightly less than 1TB:
> > fieldcount=5
> > fieldlength=200
> > recordcount=1000000000
> > maxexecutiontime=86400
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-hbase.679495.n3.nabble.com/Multiwal-performance-with-HBase-1-x-tp4074403p4074731.html
> > Sent from the HBase User mailing list archive at Nabble.com.
> >
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: Multiwal performance with HBase 1.x

Posted by Vladimir Rodionov <vl...@gmail.com>.
Hi, Jingcheng

You postpone compaction until your test completes by setting number of
blocking stores to 120. That is kind of cheating :)
As I said previously, in a long run, compaction rules the world - not
number of wal files. In a real production setting, the existing write
performance
is more than adequate (avg load per RS usually less than 1MB/sec). Multiwal
has probably its value if someone need to load quick large volume of data,
but ... why do not use bulk load instead?

Thank for letting us know that beefy servers with 8 SSDs can sustain such a
huge load.

-Vlad



On Thu, Sep 17, 2015 at 10:30 PM, Jingcheng Du <ji...@intel.com>
wrote:

> More information for the test.
> I use ycsb 0.3.0 for the test.
> The command line is "./ycsb load hbase-10 -P ../workloads/workload -threads
> 200 -p columnfamily=family -p clientbuffering=true -s > workload.dat"
> The workload is, the data size is slightly less than 1TB:
> fieldcount=5
> fieldlength=200
> recordcount=1000000000
> maxexecutiontime=86400
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/Multiwal-performance-with-HBase-1-x-tp4074403p4074731.html
> Sent from the HBase User mailing list archive at Nabble.com.
>

Re: Multiwal performance with HBase 1.x

Posted by Jingcheng Du <ji...@intel.com>.
Thanks Yu. I have some similar POC implementation to do the tiered storage to
separate the WAL, flushed hfiles and compactions.



--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Multiwal-performance-with-HBase-1-x-tp4074403p4074733.html
Sent from the HBase User mailing list archive at Nabble.com.

Re: Multiwal performance with HBase 1.x

Posted by Yu Li <ca...@gmail.com>.
JingCheng's test result is similar to what we observed with multiwal
enabled on our online cluster, say ~20% improvements with 4 wal groups.
It's a 800 nodes cluster and each node with 3 ssd and 9 sata disks, and we
set storage policy of wals to ALL_SSD.

Compaction impact could be further reduced by separating compaction IO and
wal sync IO, like setting storage policy of WALs dir to ALL_SSD and data
dir to HOT(default), FYI.

Best Regards,
Yu

On 18 September 2015 at 13:30, Jingcheng Du <ji...@intel.com> wrote:

> More information for the test.
> I use ycsb 0.3.0 for the test.
> The command line is "./ycsb load hbase-10 -P ../workloads/workload -threads
> 200 -p columnfamily=family -p clientbuffering=true -s > workload.dat"
> The workload is, the data size is slightly less than 1TB:
> fieldcount=5
> fieldlength=200
> recordcount=1000000000
> maxexecutiontime=86400
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/Multiwal-performance-with-HBase-1-x-tp4074403p4074731.html
> Sent from the HBase User mailing list archive at Nabble.com.
>

Re: Multiwal performance with HBase 1.x

Posted by Jingcheng Du <ji...@intel.com>.
More information for the test.
I use ycsb 0.3.0 for the test.
The command line is "./ycsb load hbase-10 -P ../workloads/workload -threads
200 -p columnfamily=family -p clientbuffering=true -s > workload.dat"
The workload is, the data size is slightly less than 1TB:
fieldcount=5
fieldlength=200
recordcount=1000000000
maxexecutiontime=86400



--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Multiwal-performance-with-HBase-1-x-tp4074403p4074731.html
Sent from the HBase User mailing list archive at Nabble.com.

Re: Multiwal performance with HBase 1.x

Posted by Jingcheng Du <ji...@intel.com>.
Some more supplements.
>>for 3 nodes cluster?
I use 3 nodes for RS, and another node for HM in this tests. Totally 4 nodes
in this cluster.



--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Multiwal-performance-with-HBase-1-x-tp4074403p4074730.html
Sent from the HBase User mailing list archive at Nabble.com.

Re: Multiwal performance with HBase 1.x

Posted by Jingcheng Du <ji...@intel.com>.
>>Compaction is no longer a serious problem?
I noticed there a few hundreds of compaction requests in queue. I might take
many disk I/O and lead to less throughput at that time.

>>Can you publish test/hardware configuration and HBase version/config 
>>options: Max Heap 64GB, how much was allocated to memstores? What was JVM 
>>version/options? What GC did you use?
Sure.
8 SSD (product name is INTEL SSDSC2BB800G4, each disk has around 600GB, some
of the disks had data besides the data for tests) in each node, we have 290G
memory in each node (the OS cache might help in the tests?, we assign 64GB
heap for RS, and use default memstore limit (0.4).
I use G1GC for the test, and use jdk1.8.0_60.
I use hbase 1.0.0 release to do the test.
<property>
    <name>hbase.hstore.compaction.throughput.higher.bound</name>
    <value>20971520</value>
  </property>
  <property>
    <name>hbase.hstore.compaction.throughput.lower.bound</name>
    <value>10485760</value>
  </property>
  <property>
    <name>hbase.regionserver.thread.compaction.small</name>
    <value>100</value>
  </property>
  <property>
    <name>hbase.regionserver.thread.compaction.large</name>
    <value>5</value>
  </property>
<property>
    <name>hbase.wal.regiongrouping.numgroups</name>
    <value>4</value>
  </property>
<property>
    <name>hbase.wal.provider</name>
    <value>multiwal</value>
  </property>
<property>
    <name>hbase.hstore.flusher.count</name>
    <value>15</value>
  </property>
<property>
    <name>hbase.ipc.server.read.threadpool.size</name>
    <value>50</value>
  </property>
<property>
    <name>hbase.hstore.blockingStoreFiles</name>
    <value>120</value>
  </property>
<property>
    <name>hbase.regionserver.maxlogs</name>
    <value>50</value>
  </property>
<property>
    <name>hbase.regionserver.metahandler.count</name>
    <value>30</value>
  </property>
<property>
    <name>hbase.regionserver.handler.count</name>
    <value>100</value>
  </property>

Meanwhile, I use hdfs 2.6.0 and 200 for dfs.datanode.handler.count in hdfs.
<property>
 <name>dfs.datanode.handler.count</name>
 <value>200</value>
</property>



--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Multiwal-performance-with-HBase-1-x-tp4074403p4074729.html
Sent from the HBase User mailing list archive at Nabble.com.

Re: Multiwal performance with HBase 1.x

Posted by Vladimir Rodionov <vl...@gmail.com>.
 Compaction is no longer a serious problem? Close to 850GB/h ingest rate
for 3 nodes cluster? Effect of 8 SSDs? Looks great, but ...

This is 78MB/sec load rate per server, taking into account compaction write
amplification looks unbelievable even for 8 SSD.

Can you publish test/hardware configuration and HBase version/config
options: Max Heap 64GB, how much was allocated to memstores? What was JVM
version/options? What GC did you use?

Thanks for results
-Vlad

On Thu, Sep 17, 2015 at 8:37 PM, Jingcheng Du <ji...@intel.com>
wrote:

> Now I've finished 3 tests for defaultWAL/singleWAL, 4 groups and 8 groups
> with 1TB data.
> Test env: 3 nodes, 8 SSD in each node, 64 GB heap size, pre-split 210
> regions in the cluster.
> Results:
> 1. DefaultWAL
> RunTime(ms), 5340650.0
> Throughput(ops/sec), 187243.12583674272
> AverageLatency(us), 1060.6468652781823
>
> 2. 4 groups
> RunTime(ms), 4326704.0
> Throughput(ops/sec), 231122.8131159423
> AverageLatency(us), 859.4096927494501
>
> 3. 8 groups
> RunTime(ms), 4222285.0
> Throughput(ops/sec), 236838.58384737174
> AverageLatency(us), 849.488522633065
>
>
>
>
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/Multiwal-performance-with-HBase-1-x-tp4074403p4074727.html
> Sent from the HBase User mailing list archive at Nabble.com.
>

Re: Multiwal performance with HBase 1.x

Posted by Jingcheng Du <ji...@intel.com>.
Now I've finished 3 tests for defaultWAL/singleWAL, 4 groups and 8 groups
with 1TB data.
Test env: 3 nodes, 8 SSD in each node, 64 GB heap size, pre-split 210
regions in the cluster.
Results:
1. DefaultWAL
RunTime(ms), 5340650.0
Throughput(ops/sec), 187243.12583674272
AverageLatency(us), 1060.6468652781823

2. 4 groups
RunTime(ms), 4326704.0
Throughput(ops/sec), 231122.8131159423
AverageLatency(us), 859.4096927494501

3. 8 groups
RunTime(ms), 4222285.0
Throughput(ops/sec), 236838.58384737174
AverageLatency(us), 849.488522633065







--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Multiwal-performance-with-HBase-1-x-tp4074403p4074727.html
Sent from the HBase User mailing list archive at Nabble.com.

Re: Multiwal performance with HBase 1.x

Posted by Ted Yu <yu...@gmail.com>.
Since you have 8 SSDs, can you run the test with 8 groups of WAL ?

Thanks

On Tue, Sep 15, 2015 at 7:44 PM, Jingcheng Du <ji...@intel.com>
wrote:

> I did some tests on multiple wal, but with a much smaller data size, only
> 100G.
> I have 1 HMaster and 3 RegionServers. Each node has 8 SSD.
> When I ran the test with single wal, the test cost 500s.
> When I ran the test with 4 groups of wal, it cost 340s.
> I tried wal per region (since I had too many regions in RSs, about 120
> totally, the performance seemed bad), it cost 740s.
> FYI.
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/Multiwal-performance-with-HBase-1-x-tp4074403p4074696.html
> Sent from the HBase User mailing list archive at Nabble.com.
>

Re: Multiwal performance with HBase 1.x

Posted by Vladimir Rodionov <vl...@gmail.com>.
Yes, sure you can pre-split tables to avoid splitting during test. You will
need at least 200 regions with default max size 10GB and
ConstantSizeRegionSplitPolicy. Its 70 regions per server. I would create
300 regions with default size, or 150 with double size to guarantee
splitless run - it is up to you.

-Vlad

On Wed, Sep 16, 2015 at 5:54 PM, Jingcheng Du <ji...@intel.com>
wrote:

> Currently the capacity of disks in each node is only around 4TB, I have 3
> nodes. so I am afraid 10TB is too much for them:) How about 1TB? Is region
> split okay during the test? Do you have suggestion how many regions we
> should pre-split or max region size with that data size to avoid the region
> split? Please advise. Thanks a lot
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/Multiwal-performance-with-HBase-1-x-tp4074403p4074716.html
> Sent from the HBase User mailing list archive at Nabble.com.
>

Re: Multiwal performance with HBase 1.x

Posted by Jingcheng Du <ji...@intel.com>.
Currently the capacity of disks in each node is only around 4TB, I have 3
nodes. so I am afraid 10TB is too much for them:) How about 1TB? Is region
split okay during the test? Do you have suggestion how many regions we
should pre-split or max region size with that data size to avoid the region
split? Please advise. Thanks a lot



--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Multiwal-performance-with-HBase-1-x-tp4074403p4074716.html
Sent from the HBase User mailing list archive at Nabble.com.

Re: Multiwal performance with HBase 1.x

Posted by Vladimir Rodionov <vl...@gmail.com>.
Please load 10TB and publish performance numbers. Should be pretty close.

-Vlad

On Tue, Sep 15, 2015 at 7:44 PM, Jingcheng Du <ji...@intel.com>
wrote:

> I did some tests on multiple wal, but with a much smaller data size, only
> 100G.
> I have 1 HMaster and 3 RegionServers. Each node has 8 SSD.
> When I ran the test with single wal, the test cost 500s.
> When I ran the test with 4 groups of wal, it cost 340s.
> I tried wal per region (since I had too many regions in RSs, about 120
> totally, the performance seemed bad), it cost 740s.
> FYI.
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/Multiwal-performance-with-HBase-1-x-tp4074403p4074696.html
> Sent from the HBase User mailing list archive at Nabble.com.
>

Re: Multiwal performance with HBase 1.x

Posted by Jingcheng Du <ji...@intel.com>.
I did some tests on multiple wal, but with a much smaller data size, only
100G.
I have 1 HMaster and 3 RegionServers. Each node has 8 SSD.
When I ran the test with single wal, the test cost 500s.
When I ran the test with 4 groups of wal, it cost 340s.
I tried wal per region (since I had too many regions in RSs, about 120
totally, the performance seemed bad), it cost 740s.
FYI.



--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Multiwal-performance-with-HBase-1-x-tp4074403p4074696.html
Sent from the HBase User mailing list archive at Nabble.com.

Re: Multiwal performance with HBase 1.x

Posted by milindshah <mi...@gmail.com>.
Unfortunately, I don't have SSDs at this point. The reason I put that test
configuration is because I have 12 drives. With replication = 3, I am
planning to go set numgroups=4 (1/3 of total spindles).

Milind



--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Multiwal-performance-with-HBase-1-x-tp4074403p4074410.html
Sent from the HBase User mailing list archive at Nabble.com.

Re: Multiwal performance with HBase 1.x

Posted by Ted Yu <yu...@gmail.com>.
Do you have SSD's in your cluster ?

If so, remember placing WAL files on SSD's

Cheers

On Thu, Sep 3, 2015 at 3:25 PM, milindshah <mi...@gmail.com> wrote:

> I agree that compaction is a separate discussion. What I am trying to do
> here
> is:
>
> 1) Load 2 TB of data with single WAL on an empty table.
> 2) Load 2 TB of data with regiongrouping.numgroups = 2 on an empty table.
> 3) Load 2 TB of data with regiongrouping.numgroups = 4 on an empty table.
>
> to measure throughput improvements. Does this make a relevant test?
>
> Thanks,
> Milind
>
>
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/Multiwal-performance-with-HBase-1-x-tp4074403p4074407.html
> Sent from the HBase User mailing list archive at Nabble.com.
>

Re: Multiwal performance with HBase 1.x

Posted by milindshah <mi...@gmail.com>.
I agree that compaction is a separate discussion. What I am trying to do here
is:

1) Load 2 TB of data with single WAL on an empty table.
2) Load 2 TB of data with regiongrouping.numgroups = 2 on an empty table.
3) Load 2 TB of data with regiongrouping.numgroups = 4 on an empty table.

to measure throughput improvements. Does this make a relevant test?

Thanks,
Milind





--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Multiwal-performance-with-HBase-1-x-tp4074403p4074407.html
Sent from the HBase User mailing list archive at Nabble.com.

Re: Multiwal performance with HBase 1.x

Posted by Vladimir Rodionov <vl...@gmail.com>.
>> Compactions have not been a limiting factor for throughput.

This is the subject of a separate discussion.

>> I need to go do the scientific tests on write throughput for more wals.

Yes, please, it would be nice to see some numbers.

-Vlad

On Thu, Sep 3, 2015 at 2:20 PM, Elliott Clark <ec...@apache.org> wrote:

> On Thu, Sep 3, 2015 at 1:53 PM, Vladimir Rodionov <vl...@gmail.com>
> wrote:
>
> > In a long run, it does not matter how many WAL files per RS do you have
>
>
> That's not been my experience in production. We got more write throughput
> when adding in more regionservers on a single box ( thus adding in more
> wals ). Compactions have not been a limiting factor for throughput.
> I need to go do the scientific tests on write throughput for more wals.
>

Re: Multiwal performance with HBase 1.x

Posted by Elliott Clark <ec...@apache.org>.
On Thu, Sep 3, 2015 at 1:53 PM, Vladimir Rodionov <vl...@gmail.com>
wrote:

> In a long run, it does not matter how many WAL files per RS do you have


That's not been my experience in production. We got more write throughput
when adding in more regionservers on a single box ( thus adding in more
wals ). Compactions have not been a limiting factor for throughput.
I need to go do the scientific tests on write throughput for more wals.

Re: Multiwal performance with HBase 1.x

Posted by Vladimir Rodionov <vl...@gmail.com>.
In a long run, it does not matter how many WAL files per RS do you have,
unless you disable compaction completely. You load rate will decrease over
time as 1/LOG(S), where S is RS data size, base of LOG is something between
min/max files for compaction ( 3 - 10). This is because of compaction.
Compactions - this is what you should worry about.

-Vlad

On Thu, Sep 3, 2015 at 1:40 PM, milindshah <mi...@gmail.com> wrote:

> Hi
>
> I have a generic question regarding the performance improvement that
> Multiwal should provide over single WAL. Has anyone tried to measure the
> performance gain while loading a large data set , about 1-2TB size on a
> cluster? I would appreciate if someone could share numbers and observed
> improvement. I would like to try this on a 10 node cluster with 12 drives
> each and load 2 TB of data using YCSB.
>
> Thanks,
> Milind
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/Multiwal-performance-with-HBase-1-x-tp4074403.html
> Sent from the HBase User mailing list archive at Nabble.com.
>