Hadoop Testing?

[Anandkumar R <an...@gmail.com>]

Dear Friends,

I am Anandkumar, working as a test engineer in eBay and we use Hadoop
extensively to store our log. I am in situation to validate or test our Data
perfectly reaching the Hadoop infrastructure or not. could anyone of you
recommend me the best testing methodologies and if there is any existing
framework for testing Hadoop, please recommend to me.

My scenario is simple of Client will dump millions of data to Hadoop, I need
to validate that the data has reached Hadoop perfectly and also there is not
Data loss and also other testing like scalability and reliability.

Anticipating your support

Thanks,
Anandkumar

Re: Hadoop Testing?

[James Seigel <ja...@tynt.com>]

Anandkumar

Count the records in.  See if the same number show up in hdfs.

number the records in.  See if the sequential set of numbers show up in hdfs.

Other than that I am not sure what kinds of things you’d like to test.

Scaling:  What kind of scaling?  Size?  requests/sec?  data in / sec..

reliability?  What kinds of measures are you thinking of?  With replication set to 3, in the most unlucky scenario you can have two machines go down and you still won’t lose data.

Performance of mapreduce, a lot of people run terasort/teragen to get a baseline of how their changes affected a cluster (more nodes, configuration tweaks, etc...)

I hope this helps a little.

Cheers
James
On 2011-03-17, at 11:48 AM, Anandkumar R wrote:

> Dear Friends,
> 
> I am Anandkumar, working as a test engineer in eBay and we use Hadoop
> extensively to store our log. I am in situation to validate or test our Data
> perfectly reaching the Hadoop infrastructure or not. could anyone of you
> recommend me the best testing methodologies and if there is any existing
> framework for testing Hadoop, please recommend to me.
> 
> My scenario is simple of Client will dump millions of data to Hadoop, I need
> to validate that the data has reached Hadoop perfectly and also there is not
> Data loss and also other testing like scalability and reliability.
> 
> Anticipating your support
> 
> Thanks,
> Anandkumar

Re: Hadoop Testing?

[Konstantin Boudnik <co...@apache.org>]

We just have pushed an update of stack validation framework (the one
Roman and I have  presented at eBay a few weeks ago) which allows you
to formalize and simplify Hadoop testing. This is still a pre-Beta
stage (e.g. no user docs are ready yet), but it is working and has a
lot of merit in it as of now. This framework allows you to seamlessly
reuse a existing Hadoop oriented test-artifacts available from variety
of components such as Pig, Sqoop, Hadoop proper, Hive, etc.

Pl. check github.com/cloudera/iTest - more is coming daily. And feel
free to contribute ;)

--
  Take care,
Konstantin (Cos) Boudnik

On Thu, Mar 17, 2011 at 10:48, Anandkumar R <an...@gmail.com> wrote:
> Dear Friends,
>
> I am Anandkumar, working as a test engineer in eBay and we use Hadoop
> extensively to store our log. I am in situation to validate or test our Data
> perfectly reaching the Hadoop infrastructure or not. could anyone of you
> recommend me the best testing methodologies and if there is any existing
> framework for testing Hadoop, please recommend to me.
>
> My scenario is simple of Client will dump millions of data to Hadoop, I need
> to validate that the data has reached Hadoop perfectly and also there is not
> Data loss and also other testing like scalability and reliability.
>
> Anticipating your support
>
> Thanks,
> Anandkumar
>

Re: Hadoop Testing?

[Konstantin Boudnik <co...@apache.org>]

We just have pushed an update of stack validation framework (the one
Roman and I have  presented at eBay a few weeks ago) which allows you
to formalize and simplify Hadoop testing. This is still a pre-Beta
stage (e.g. no user docs are ready yet), but it is working and has a
lot of merit in it as of now. This framework allows you to seamlessly
reuse a existing Hadoop oriented test-artifacts available from variety
of components such as Pig, Sqoop, Hadoop proper, Hive, etc.

Pl. check github.com/cloudera/iTest - more is coming daily. And feel
free to contribute ;)

--
  Take care,
Konstantin (Cos) Boudnik

On Thu, Mar 17, 2011 at 10:48, Anandkumar R <an...@gmail.com> wrote:
> Dear Friends,
>
> I am Anandkumar, working as a test engineer in eBay and we use Hadoop
> extensively to store our log. I am in situation to validate or test our Data
> perfectly reaching the Hadoop infrastructure or not. could anyone of you
> recommend me the best testing methodologies and if there is any existing
> framework for testing Hadoop, please recommend to me.
>
> My scenario is simple of Client will dump millions of data to Hadoop, I need
> to validate that the data has reached Hadoop perfectly and also there is not
> Data loss and also other testing like scalability and reliability.
>
> Anticipating your support
>
> Thanks,
> Anandkumar
>