You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Hao Yang <ha...@gmail.com> on 2011/11/29 06:43:39 UTC
how to write hadoop client to test HDFS with fault injection
Hi, all:
I am a graduate student, now working on HDFS fault injection course
project. I used FI framework to create aspects, and hdfs-fi.jar was
generated. How can I write Hadoop Client code to test the injection? Thank
you.
Thank you very much for your time.
Best regards
Hao Yang
Re: how to write hadoop client to test HDFS with fault injection
Posted by Hao Yang <ha...@gmail.com>.
Thanks very much for the above. In 0.21 I found TestDFSIO in the
mapred-test.jar, but I didn't find Teragen and Terasort in the three test
jars (I found Teragen and Terasort in 0.20.04 test jars, but they cannot be
executed in 0.21). When I run TestDFSIO benchmark, I sometimes encounter
the situation below. I met the same problem when I test it in my laptop
with one datanode or two datanodes launched.
java.io.IOException: File
/benchmarks/TestDFSIO/io_control/in_file_test_io_0 could only be replicated
to 0 nodes, instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1448)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:691)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:342)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1350)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1346)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1344)
Thanks.
Hao Yang
2011/11/29 Konstantin Boudnik <co...@apache.org>
> Once you have an instrumented cluster you can run some end-to-end tests
> (TeraSort
> has been suggested and it would do fine). The tricky part is to assemble
> all
> moving parts of the Hadoop cluster together from three different projects
> (if
> you are on .21+ branch) and will be a way easier for 0.20.+ based versions.
>
> However, one fine point here is that your faults (implemented as aspects)
> should be triggered some how. You can do it logically (say every 3rd
> attempt
> to get new block) or via environment, or from the test application.
>
> The latter might be a most tricky part because you'll have to essentially
> communicate between different JVM processes perhaps located on different
> physical hosts if you're running in a real cluster environment.
>
> To address these issues we have project Herriot which is based on byte code
> injection (read aspectj) and allows for distributed fault injections in a
> real
> cluster. System test framework adds a few APIs to the Hadoop daemons and
> allow
> to call them via standard Hadoop RPC.
>
> You can find framework sources and tests code in src/aop/system.
> There's a separate set of build targets which will help you instrument
> everything you need (jar-test-system should do). More info is here
> http://is.gd/SBkjrC
>
> One last point: Herriot isn't fully integrated in trunk's maven build as
> of today.
>
> Hope it helps,
> Cos
>
> >> From: Hao Yang [mailto:hao.yang0614@gmail.com]
> >> Sent: Monday, November 28, 2011 9:44 PM
> >> To: common-dev@hadoop.apache.org
> >> Subject: how to write hadoop client to test HDFS with fault injection
> >>
> >> Hi, all:
> >>
> >> I am a graduate student, now working on HDFS fault injection course
> >> project. I used FI framework to create aspects, and hdfs-fi.jar was
> >> generated. How can I write Hadoop Client code to test the injection?
> Thank
> >> you.
> >>
> >>
> >> Thank you very much for your time.
> >>
> >>
> >> Best regards
> >> Hao Yang
> >>
> >
>
Re: how to write hadoop client to test HDFS with fault injection
Posted by Konstantin Boudnik <co...@apache.org>.
Once you have an instrumented cluster you can run some end-to-end tests (TeraSort
has been suggested and it would do fine). The tricky part is to assemble all
moving parts of the Hadoop cluster together from three different projects (if
you are on .21+ branch) and will be a way easier for 0.20.+ based versions.
However, one fine point here is that your faults (implemented as aspects)
should be triggered some how. You can do it logically (say every 3rd attempt
to get new block) or via environment, or from the test application.
The latter might be a most tricky part because you'll have to essentially
communicate between different JVM processes perhaps located on different
physical hosts if you're running in a real cluster environment.
To address these issues we have project Herriot which is based on byte code
injection (read aspectj) and allows for distributed fault injections in a real
cluster. System test framework adds a few APIs to the Hadoop daemons and allow
to call them via standard Hadoop RPC.
You can find framework sources and tests code in src/aop/system.
There's a separate set of build targets which will help you instrument
everything you need (jar-test-system should do). More info is here http://is.gd/SBkjrC
One last point: Herriot isn't fully integrated in trunk's maven build as of today.
Hope it helps,
Cos
>> From: Hao Yang [mailto:hao.yang0614@gmail.com]
>> Sent: Monday, November 28, 2011 9:44 PM
>> To: common-dev@hadoop.apache.org
>> Subject: how to write hadoop client to test HDFS with fault injection
>>
>> Hi, all:
>>
>> I am a graduate student, now working on HDFS fault injection course
>> project. I used FI framework to create aspects, and hdfs-fi.jar was
>> generated. How can I write Hadoop Client code to test the injection? Thank
>> you.
>>
>>
>> Thank you very much for your time.
>>
>>
>> Best regards
>> Hao Yang
>>
>
Re: how to write hadoop client to test HDFS with fault injection
Posted by Steve Loughran <st...@apache.org>.
I'd add that one thing I'd like is better low level hardware fault
simulation, something that doesn't inject faults into the software, so
much as stamp on the underlying data or kill bits of the stack entirely
That is
-deliberately corrupting blocks or checksums
-deleting blocks
-dropping entire racks in the real/simulated cluster
http://www.slideshare.net/steve_l/did-you-reallywantthatdata
Then follow Brad's suggestion and see how terasort copes (you can scale
it down to GB for local tests first), before having a look at how the
layers above -especially hbase- cope with problems.
On 29/11/11 16:36, Brad Sarsfield wrote:
> Hi Hao,
>
> You may want to setup a full Hadoop 'one box' install and start running the TestDFSIO, Teragen and Terasort (with a small amount of data) scenarios using the injected jar to run hdfs. That may be the fastest and best way to get up and running.
>
> When it comes to fault injection the valuable issues that you find may be found using the end to end tests versus targeted api calls. It's much more realistic. It's a much stronger case to get something fixed / fix something when you can point to the end to end scenario, showing how it can fail. That's where you find the good ones.
>
> You may also want to consider what a data validation oracle would look like that would allow you to do data validation to catch data corruption issues. If you do go further and create a scenario that, say generates a non-random / repeatable sequence that reads/writes to HDFS, you could use the fault injection and the oracle would validate data correctness.
>
> You'd get extra bonus points if you packaged things up and contributed your work to Bigtop so others could benefit long term.
>
> ~Brad
>
> -----Original Message-----
> From: Hao Yang [mailto:hao.yang0614@gmail.com]
> Sent: Monday, November 28, 2011 9:44 PM
> To: common-dev@hadoop.apache.org
> Subject: how to write hadoop client to test HDFS with fault injection
>
> Hi, all:
>
> I am a graduate student, now working on HDFS fault injection course project. I used FI framework to create aspects, and hdfs-fi.jar was generated. How can I write Hadoop Client code to test the injection? Thank you.
>
>
> Thank you very much for your time.
>
>
> Best regards
> Hao Yang
>
RE: how to write hadoop client to test HDFS with fault injection
Posted by Brad Sarsfield <br...@bing.com>.
Hi Hao,
You may want to setup a full Hadoop 'one box' install and start running the TestDFSIO, Teragen and Terasort (with a small amount of data) scenarios using the injected jar to run hdfs. That may be the fastest and best way to get up and running.
When it comes to fault injection the valuable issues that you find may be found using the end to end tests versus targeted api calls. It's much more realistic. It's a much stronger case to get something fixed / fix something when you can point to the end to end scenario, showing how it can fail. That's where you find the good ones.
You may also want to consider what a data validation oracle would look like that would allow you to do data validation to catch data corruption issues. If you do go further and create a scenario that, say generates a non-random / repeatable sequence that reads/writes to HDFS, you could use the fault injection and the oracle would validate data correctness.
You'd get extra bonus points if you packaged things up and contributed your work to Bigtop so others could benefit long term.
~Brad
-----Original Message-----
From: Hao Yang [mailto:hao.yang0614@gmail.com]
Sent: Monday, November 28, 2011 9:44 PM
To: common-dev@hadoop.apache.org
Subject: how to write hadoop client to test HDFS with fault injection
Hi, all:
I am a graduate student, now working on HDFS fault injection course project. I used FI framework to create aspects, and hdfs-fi.jar was generated. How can I write Hadoop Client code to test the injection? Thank you.
Thank you very much for your time.
Best regards
Hao Yang