You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Da Zheng <zh...@gmail.com> on 2011/01/25 21:33:32 UTC
the performance of HDFS
Hello,
I try to measure the performance of HDFS, but the writing rate is quite
low. When the replication factor is 1, the rate of writing to HDFS is
about 60MB/s. When the replication factor is 3, the rate drops
significantly to about 15MB/s. Even though the actual rate of writing
data to the disk is about 45MB/s, it's still much lower than when
replication factor is 1. The link between two nodes in the cluster is
1Gbps. CPU is Dual-Core AMD Opteron(tm) Processor 2212, so CPU isn't
bottleneck either. I thought I should be able to saturate the disk very
easily. I wonder where the bottleneck is. What is the throughput for
writing on a Hadoop cluster when the replication factor is 3?
Thanks,
Da
Re: the performance of HDFS
Posted by Ted Dunning <td...@maprtech.com>.
This is a bit lower than it should be, but it is not so far out of line with
what is reasonable.
Did you make sure that have multiple separate disks for HDFS to use? With
many disks, you should be able to get local disk write speeds up to a few
hundred MB/s.
Once you involve replication then you need to have data go out the network
interface, back in to another machine, back out and back in to a third
machine. There are lots of copies going on and if you are writing lots of
files, you will typically be limited to 1/2 of your network bandwidth at
most, but doing a bit less than that is to be expected. What you are seeing
is lower than it should be but a moderate factor.
On Tue, Jan 25, 2011 at 12:33 PM, Da Zheng <zh...@gmail.com> wrote:
> Hello,
>
> I try to measure the performance of HDFS, but the writing rate is quite
> low. When the replication factor is 1, the rate of writing to HDFS is about
> 60MB/s. When the replication factor is 3, the rate drops significantly to
> about 15MB/s. Even though the actual rate of writing data to the disk is
> about 45MB/s, it's still much lower than when replication factor is 1. The
> link between two nodes in the cluster is 1Gbps. CPU is Dual-Core AMD
> Opteron(tm) Processor 2212, so CPU isn't bottleneck either. I thought I
> should be able to saturate the disk very easily. I wonder where the
> bottleneck is. What is the throughput for writing on a Hadoop cluster when
> the replication factor is 3?
>
> Thanks,
> Da
>
Re: the performance of HDFS
Posted by Da Zheng <zh...@gmail.com>.
Right, thank you. it tells me the model of the hard drive I use is Hitachi
HUA72101. The spec of the drive is here:
http://www.hitachigst.com/tech/techlib.nsf/techdocs/DF2EF568E18716F5862572C20067A757/$file/Ultrastar_A7K1000_final_DS.pdf
It says the Sustained transfer rate (MB/sec) is 85 - 42 (zone 0-29). I guess the
result I get is reasonable, right?
Thanks,
Da
On 1/26/11 3:12 AM, Harsh J wrote:
> You may also see it in /proc/scsi/scsi, perhaps.
>
> On Wed, Jan 26, 2011 at 9:27 AM, Da Zheng <zh...@gmail.com> wrote:
>> unfortunately, the command isn't available in the system and I don't have
>> privilege to install software:-(
>>
>>
>> On 1/25/11 9:37 PM, Ted Dunning wrote:
>>> Perhaps lshw would help you.
>>>
>>> ubuntu:~$ sudo lshw
>>> ...
>>> *-storage
>>> description: RAID bus controller
>>> product: SB700/SB800 SATA Controller [Non-RAID5 mode]
>>> vendor: ATI Technologies Inc
>>> physical id: 11
>>> bus info: pci@0000:00:11.0
>>> logical name: scsi0
>>> logical name: scsi1
>>> version: 00
>>> width: 32 bits
>>> clock: 66MHz
>>> capabilities: storage pm bus_master cap_list emulated
>>> configuration: driver=ahci latency=64
>>> resources: irq:22 ioport:b000(size=8) ioport:a000(size=4)
>>> ioport:9000(size=8) ioport:8000(size=4) ioport:7000(size=16)
>>> memory:fe7ffc00-fe7fffff
>>> *-disk
>>> description: ATA Disk
>>> product: ST3750528AS
>>> vendor: Seagate
>>> physical id: 0
>>> bus info: scsi@0:0.0.0
>>> ...
>>>
>>> On Tue, Jan 25, 2011 at 6:29 PM, Da Zheng <zh...@gmail.com> wrote:
>>>
>>>> No, each node in the cluster is powerful server. I was told the nodes are
>>>> Dell
>>>> Poweredge SC1435, but I cannot figure out the configuration of hard drives.
>>>> Dell
>>>> provides several possible hard drives for this model.
>>>>
>>>> On 1/25/11 7:59 PM, Ted Dunning wrote:
>>>>> This is a really slow drive or controller.
>>>>>
>>>>> Consumer grade 3.5 inch 2TB drives typically can handle 100MB/s. I would
>>>>> suspect in the absence of real information that your controller is more
>>>>> likely to be deficient than your drive. If this is on a laptop or
>>>>> something, then I withdraw my thought.
>>>>>
>>>>> On Tue, Jan 25, 2011 at 4:50 PM, Da Zheng <zh...@gmail.com> wrote:
>>>>>
>>>>>> The aggregate write-rate can get much higher if you use more drives, but
>>>> a
>>>>>>> single stream throughput is limited to the speed of one disk spindle.
>>>>>>>
>>>>>>> You are right. I measure the performance of the hard drive. It seems
>>>> the
>>>>>> bottleneck is the hard drive, but the hard drive is a little too slow.
>>>> The
>>>>>> average writing rate is 50MB/s.
>>>>>
>>>>
>>>>
>>>
>>
>>
>
>
>
Re: the performance of HDFS
Posted by Harsh J <qw...@gmail.com>.
You may also see it in /proc/scsi/scsi, perhaps.
On Wed, Jan 26, 2011 at 9:27 AM, Da Zheng <zh...@gmail.com> wrote:
> unfortunately, the command isn't available in the system and I don't have
> privilege to install software:-(
>
>
> On 1/25/11 9:37 PM, Ted Dunning wrote:
>> Perhaps lshw would help you.
>>
>> ubuntu:~$ sudo lshw
>> ...
>> *-storage
>> description: RAID bus controller
>> product: SB700/SB800 SATA Controller [Non-RAID5 mode]
>> vendor: ATI Technologies Inc
>> physical id: 11
>> bus info: pci@0000:00:11.0
>> logical name: scsi0
>> logical name: scsi1
>> version: 00
>> width: 32 bits
>> clock: 66MHz
>> capabilities: storage pm bus_master cap_list emulated
>> configuration: driver=ahci latency=64
>> resources: irq:22 ioport:b000(size=8) ioport:a000(size=4)
>> ioport:9000(size=8) ioport:8000(size=4) ioport:7000(size=16)
>> memory:fe7ffc00-fe7fffff
>> *-disk
>> description: ATA Disk
>> product: ST3750528AS
>> vendor: Seagate
>> physical id: 0
>> bus info: scsi@0:0.0.0
>> ...
>>
>> On Tue, Jan 25, 2011 at 6:29 PM, Da Zheng <zh...@gmail.com> wrote:
>>
>>> No, each node in the cluster is powerful server. I was told the nodes are
>>> Dell
>>> Poweredge SC1435, but I cannot figure out the configuration of hard drives.
>>> Dell
>>> provides several possible hard drives for this model.
>>>
>>> On 1/25/11 7:59 PM, Ted Dunning wrote:
>>>> This is a really slow drive or controller.
>>>>
>>>> Consumer grade 3.5 inch 2TB drives typically can handle 100MB/s. I would
>>>> suspect in the absence of real information that your controller is more
>>>> likely to be deficient than your drive. If this is on a laptop or
>>>> something, then I withdraw my thought.
>>>>
>>>> On Tue, Jan 25, 2011 at 4:50 PM, Da Zheng <zh...@gmail.com> wrote:
>>>>
>>>>> The aggregate write-rate can get much higher if you use more drives, but
>>> a
>>>>>> single stream throughput is limited to the speed of one disk spindle.
>>>>>>
>>>>>> You are right. I measure the performance of the hard drive. It seems
>>> the
>>>>> bottleneck is the hard drive, but the hard drive is a little too slow.
>>> The
>>>>> average writing rate is 50MB/s.
>>>>
>>>
>>>
>>
>
>
--
Harsh J
www.harshj.com
Re: the performance of HDFS
Posted by Da Zheng <zh...@gmail.com>.
unfortunately, the command isn't available in the system and I don't have
privilege to install software:-(
On 1/25/11 9:37 PM, Ted Dunning wrote:
> Perhaps lshw would help you.
>
> ubuntu:~$ sudo lshw
> ...
> *-storage
> description: RAID bus controller
> product: SB700/SB800 SATA Controller [Non-RAID5 mode]
> vendor: ATI Technologies Inc
> physical id: 11
> bus info: pci@0000:00:11.0
> logical name: scsi0
> logical name: scsi1
> version: 00
> width: 32 bits
> clock: 66MHz
> capabilities: storage pm bus_master cap_list emulated
> configuration: driver=ahci latency=64
> resources: irq:22 ioport:b000(size=8) ioport:a000(size=4)
> ioport:9000(size=8) ioport:8000(size=4) ioport:7000(size=16)
> memory:fe7ffc00-fe7fffff
> *-disk
> description: ATA Disk
> product: ST3750528AS
> vendor: Seagate
> physical id: 0
> bus info: scsi@0:0.0.0
> ...
>
> On Tue, Jan 25, 2011 at 6:29 PM, Da Zheng <zh...@gmail.com> wrote:
>
>> No, each node in the cluster is powerful server. I was told the nodes are
>> Dell
>> Poweredge SC1435, but I cannot figure out the configuration of hard drives.
>> Dell
>> provides several possible hard drives for this model.
>>
>> On 1/25/11 7:59 PM, Ted Dunning wrote:
>>> This is a really slow drive or controller.
>>>
>>> Consumer grade 3.5 inch 2TB drives typically can handle 100MB/s. I would
>>> suspect in the absence of real information that your controller is more
>>> likely to be deficient than your drive. If this is on a laptop or
>>> something, then I withdraw my thought.
>>>
>>> On Tue, Jan 25, 2011 at 4:50 PM, Da Zheng <zh...@gmail.com> wrote:
>>>
>>>> The aggregate write-rate can get much higher if you use more drives, but
>> a
>>>>> single stream throughput is limited to the speed of one disk spindle.
>>>>>
>>>>> You are right. I measure the performance of the hard drive. It seems
>> the
>>>> bottleneck is the hard drive, but the hard drive is a little too slow.
>> The
>>>> average writing rate is 50MB/s.
>>>
>>
>>
>
Re: the performance of HDFS
Posted by Ted Dunning <td...@maprtech.com>.
Perhaps lshw would help you.
ubuntu:~$ sudo lshw
...
*-storage
description: RAID bus controller
product: SB700/SB800 SATA Controller [Non-RAID5 mode]
vendor: ATI Technologies Inc
physical id: 11
bus info: pci@0000:00:11.0
logical name: scsi0
logical name: scsi1
version: 00
width: 32 bits
clock: 66MHz
capabilities: storage pm bus_master cap_list emulated
configuration: driver=ahci latency=64
resources: irq:22 ioport:b000(size=8) ioport:a000(size=4)
ioport:9000(size=8) ioport:8000(size=4) ioport:7000(size=16)
memory:fe7ffc00-fe7fffff
*-disk
description: ATA Disk
product: ST3750528AS
vendor: Seagate
physical id: 0
bus info: scsi@0:0.0.0
...
On Tue, Jan 25, 2011 at 6:29 PM, Da Zheng <zh...@gmail.com> wrote:
> No, each node in the cluster is powerful server. I was told the nodes are
> Dell
> Poweredge SC1435, but I cannot figure out the configuration of hard drives.
> Dell
> provides several possible hard drives for this model.
>
> On 1/25/11 7:59 PM, Ted Dunning wrote:
> > This is a really slow drive or controller.
> >
> > Consumer grade 3.5 inch 2TB drives typically can handle 100MB/s. I would
> > suspect in the absence of real information that your controller is more
> > likely to be deficient than your drive. If this is on a laptop or
> > something, then I withdraw my thought.
> >
> > On Tue, Jan 25, 2011 at 4:50 PM, Da Zheng <zh...@gmail.com> wrote:
> >
> >> The aggregate write-rate can get much higher if you use more drives, but
> a
> >>> single stream throughput is limited to the speed of one disk spindle.
> >>>
> >>> You are right. I measure the performance of the hard drive. It seems
> the
> >> bottleneck is the hard drive, but the hard drive is a little too slow.
> The
> >> average writing rate is 50MB/s.
> >
>
>
Re: the performance of HDFS
Posted by Da Zheng <zh...@gmail.com>.
Esteban,
Tools like iostat and iotop isn't installed.
>From the specification, there is a RAID controller, but I'm not sure if it's used.
zhengda@occalit28:~$ cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: ATA Model: WDC WD800AAJS-18 Rev: 01.0
Type: Direct-Access ANSI SCSI revision: 05
Host: scsi1 Channel: 00 Id: 00 Lun: 00
Vendor: ATA Model: Hitachi HUA72101 Rev: GKAO
Type: Direct-Access ANSI SCSI revision: 05
Thanks,
Da
On 1/26/11 12:14 PM, Esteban Gutierrez Moguel wrote:
> Da,
>
> Do you have permissions to run iostat? That could help you to find
> persistent IO activity from other processes.
>
> Another possibility is that your system has a RAID controller without a
> cache memory card, from my experience that makes writes awfully slow.
>
>
> esteban.
>
>
> On Tue, Jan 25, 2011 at 20:29, Da Zheng <zh...@gmail.com> wrote:
>
>> No, each node in the cluster is powerful server. I was told the nodes are
>> Dell
>> Poweredge SC1435, but I cannot figure out the configuration of hard drives.
>> Dell
>> provides several possible hard drives for this model.
>>
>> On 1/25/11 7:59 PM, Ted Dunning wrote:
>>> This is a really slow drive or controller.
>>>
>>> Consumer grade 3.5 inch 2TB drives typically can handle 100MB/s. I would
>>> suspect in the absence of real information that your controller is more
>>> likely to be deficient than your drive. If this is on a laptop or
>>> something, then I withdraw my thought.
>>>
>>> On Tue, Jan 25, 2011 at 4:50 PM, Da Zheng <zh...@gmail.com> wrote:
>>>
>>>> The aggregate write-rate can get much higher if you use more drives, but
>> a
>>>>> single stream throughput is limited to the speed of one disk spindle.
>>>>>
>>>>> You are right. I measure the performance of the hard drive. It seems
>> the
>>>> bottleneck is the hard drive, but the hard drive is a little too slow.
>> The
>>>> average writing rate is 50MB/s.
>>>
>>
>>
>
Re: the performance of HDFS
Posted by Esteban Gutierrez Moguel <es...@gmail.com>.
Da,
Do you have permissions to run iostat? That could help you to find
persistent IO activity from other processes.
Another possibility is that your system has a RAID controller without a
cache memory card, from my experience that makes writes awfully slow.
esteban.
On Tue, Jan 25, 2011 at 20:29, Da Zheng <zh...@gmail.com> wrote:
> No, each node in the cluster is powerful server. I was told the nodes are
> Dell
> Poweredge SC1435, but I cannot figure out the configuration of hard drives.
> Dell
> provides several possible hard drives for this model.
>
> On 1/25/11 7:59 PM, Ted Dunning wrote:
> > This is a really slow drive or controller.
> >
> > Consumer grade 3.5 inch 2TB drives typically can handle 100MB/s. I would
> > suspect in the absence of real information that your controller is more
> > likely to be deficient than your drive. If this is on a laptop or
> > something, then I withdraw my thought.
> >
> > On Tue, Jan 25, 2011 at 4:50 PM, Da Zheng <zh...@gmail.com> wrote:
> >
> >> The aggregate write-rate can get much higher if you use more drives, but
> a
> >>> single stream throughput is limited to the speed of one disk spindle.
> >>>
> >>> You are right. I measure the performance of the hard drive. It seems
> the
> >> bottleneck is the hard drive, but the hard drive is a little too slow.
> The
> >> average writing rate is 50MB/s.
> >
>
>
Re: the performance of HDFS
Posted by Da Zheng <zh...@gmail.com>.
No, each node in the cluster is powerful server. I was told the nodes are Dell
Poweredge SC1435, but I cannot figure out the configuration of hard drives. Dell
provides several possible hard drives for this model.
On 1/25/11 7:59 PM, Ted Dunning wrote:
> This is a really slow drive or controller.
>
> Consumer grade 3.5 inch 2TB drives typically can handle 100MB/s. I would
> suspect in the absence of real information that your controller is more
> likely to be deficient than your drive. If this is on a laptop or
> something, then I withdraw my thought.
>
> On Tue, Jan 25, 2011 at 4:50 PM, Da Zheng <zh...@gmail.com> wrote:
>
>> The aggregate write-rate can get much higher if you use more drives, but a
>>> single stream throughput is limited to the speed of one disk spindle.
>>>
>>> You are right. I measure the performance of the hard drive. It seems the
>> bottleneck is the hard drive, but the hard drive is a little too slow. The
>> average writing rate is 50MB/s.
>
Re: the performance of HDFS
Posted by Ted Dunning <td...@maprtech.com>.
This is a really slow drive or controller.
Consumer grade 3.5 inch 2TB drives typically can handle 100MB/s. I would
suspect in the absence of real information that your controller is more
likely to be deficient than your drive. If this is on a laptop or
something, then I withdraw my thought.
On Tue, Jan 25, 2011 at 4:50 PM, Da Zheng <zh...@gmail.com> wrote:
> The aggregate write-rate can get much higher if you use more drives, but a
>> single stream throughput is limited to the speed of one disk spindle.
>>
>> You are right. I measure the performance of the hard drive. It seems the
> bottleneck is the hard drive, but the hard drive is a little too slow. The
> average writing rate is 50MB/s.
Re: the performance of HDFS
Posted by Da Zheng <zh...@gmail.com>.
On 01/25/2011 05:49 PM, M. C. Srivas wrote:
> On Tue, Jan 25, 2011 at 12:33 PM, Da Zheng<zh...@gmail.com> wrote:
>
>> Hello,
>>
>> I try to measure the performance of HDFS, but the writing rate is quite
>> low. When the replication factor is 1, the rate of writing to HDFS is about
>> 60MB/s. When the replication factor is 3, the rate drops significantly to
>> about 15MB/s. Even though the actual rate of writing data to the disk is
>> about 45MB/s, it's still much lower than when replication factor is 1. The
>> link between two nodes in the cluster is 1Gbps. CPU is Dual-Core AMD
>> Opteron(tm) Processor 2212, so CPU isn't bottleneck either. I thought I
>> should be able to saturate the disk very easily. I wonder where the
>> bottleneck is. What is the throughput for writing on a Hadoop cluster when
>> the replication factor is 3?
>>
> The numbers above seem correct as per my observations. If your data is
> 3-way replicated, the data-node writes about 3x the actual data written.
> Conversely, your write-rate will be limited to 1/3 of how fast the disk can
> write, minus some overhead for replication.
>
> The aggregate write-rate can get much higher if you use more drives, but a
> single stream throughput is limited to the speed of one disk spindle.
>
You are right. I measure the performance of the hard drive. It seems the
bottleneck is the hard drive, but the hard drive is a little too slow.
The average writing rate is 50MB/s.
Thanks,
Da
Re: the performance of HDFS
Posted by "M. C. Srivas" <mc...@gmail.com>.
On Tue, Jan 25, 2011 at 12:33 PM, Da Zheng <zh...@gmail.com> wrote:
> Hello,
>
> I try to measure the performance of HDFS, but the writing rate is quite
> low. When the replication factor is 1, the rate of writing to HDFS is about
> 60MB/s. When the replication factor is 3, the rate drops significantly to
> about 15MB/s. Even though the actual rate of writing data to the disk is
> about 45MB/s, it's still much lower than when replication factor is 1. The
> link between two nodes in the cluster is 1Gbps. CPU is Dual-Core AMD
> Opteron(tm) Processor 2212, so CPU isn't bottleneck either. I thought I
> should be able to saturate the disk very easily. I wonder where the
> bottleneck is. What is the throughput for writing on a Hadoop cluster when
> the replication factor is 3?
>
The numbers above seem correct as per my observations. If your data is
3-way replicated, the data-node writes about 3x the actual data written.
Conversely, your write-rate will be limited to 1/3 of how fast the disk can
write, minus some overhead for replication.
The aggregate write-rate can get much higher if you use more drives, but a
single stream throughput is limited to the speed of one disk spindle.
> Thanks,
> Da
>