You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mrunit.apache.org by "Dandona, Rohit" <Ro...@SYNTELINC.COM> on 2012/10/15 08:59:28 UTC
MRunit test invokes setup() and cleanup() repeatedly
Hi,
I am trying to implement a MRunit test a MR job which picks up the best record from a set. The execution of the test invokes the setup() and cleanup() methods of the mapper/reducer class repeatedly (for every test input case). This results in the resetting of variables that are supposed to be set once during the job. Thus, incorrect results are obtained. Please suggest a way out.
Thanks
Confidential: This electronic message and all contents contain information from Syntel, Inc. which may be privileged, confidential or otherwise protected from disclosure. The information is intended to be for the addressee only. If you are not the addressee, any disclosure, copy, distribution or use of the contents of this message is prohibited. If you have received this electronic message in error, please notify the sender immediately and destroy the original message and all copies.
RE: MRunit test invokes setup() and cleanup() repeatedly
Posted by "Dandona, Rohit" <Ro...@SYNTELINC.COM>.
Thanks anyway !
-----Original Message-----
From: Brock Noland [mailto:brock@cloudera.com]
Sent: Monday, October 15, 2012 8:02 PM
To: user@mrunit.apache.org
Subject: Re: MRunit test invokes setup() and cleanup() repeatedly
Hi,
Not that I am aware of. The good news is that we are planning on
releasing soon with changes that allow you to specify multiple input
key/value pairs.
Until then, you should be able to use a snapshot of trunk.
Brock
On Mon, Oct 15, 2012 at 6:08 AM, Dandona, Rohit
<Ro...@syntelinc.com> wrote:
> Hi Bertrand,
>
>
>
> Thanks for your reply.
>
> The problem which I am facing is in accordance with the issue reported in
> MRUNIT-64.
>
> Is there any other way to perform this test with multiple key value pair
> inputs ? Is it possible to set number of times the setup() and cleanup()
> methods get executed in the mrunit test to one ?
>
>
>
> Thanks again for your help.
>
> Rohit
>
>
>
>
>
> From: Bertrand Dechoux [mailto:dechouxb@gmail.com]
> Sent: Monday, October 15, 2012 1:05 PM
>
>
> To: user@mrunit.apache.org
> Subject: Re: MRunit test invokes setup() and cleanup() repeatedly
>
>
>
> Hi,
>
>
>
> I don't have a simple way to check MRUnit source code right now but :
> 1) could you provide a simple example? (if you add println to
> setup/map/reduce/cleanup methods, it should provide with enough information,
> we don't even need the values, hashes are fine).
> 2) your problem might be read like you are using a global shared variable
> which won't be possible within a cluster, could you provide more information
> about your problem? Are you specifying a specific input format where you now
> all related information will be in the same mapper? If MRunit actually call
> setup/cleanup for every map input, I don't see the harm in this except when
> using non standard input format. As for the reducer, if setup/cleanup are
> indeed called for every input, it might be a issue (with MRunit
> implementation) if you know and rely on the fact that you have only a single
> reducer.
>
> Regards
>
> Bertrand
>
> On Mon, Oct 15, 2012 at 8:59 AM, Dandona, Rohit
> <Ro...@syntelinc.com> wrote:
>
> Hi,
>
>
>
> I am trying to implement a MRunit test a MR job which picks up the best
> record from a set. The execution of the test invokes the setup() and
> cleanup() methods of the mapper/reducer class repeatedly (for every test
> input case). This results in the resetting of variables that are supposed to
> be set once during the job. Thus, incorrect results are obtained. Please
> suggest a way out.
>
>
>
> Thanks
>
>
>
>
>
> Confidential: This electronic message and all contents contain information
> from Syntel, Inc. which may be privileged, confidential or otherwise
> protected from disclosure. The information is intended to be for the
> addressee only. If you are not the addressee, any disclosure, copy,
> distribution or use of the contents of this message is prohibited. If you
> have received this electronic message in error, please notify the sender
> immediately and destroy the original message and all copies.
>
>
>
>
> --
> Bertrand Dechoux
>
> Confidential: This electronic message and all contents contain information
> from Syntel, Inc. which may be privileged, confidential or otherwise
> protected from disclosure. The information is intended to be for the
> addressee only. If you are not the addressee, any disclosure, copy,
> distribution or use of the contents of this message is prohibited. If you
> have received this electronic message in error, please notify the sender
> immediately and destroy the original message and all copies.
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Confidential: This electronic message and all contents contain information from Syntel, Inc. which may be privileged, confidential or otherwise protected from disclosure. The information is intended to be for the addressee only. If you are not the addressee, any disclosure, copy, distribution or use of the contents of this message is prohibited. If you have received this electronic message in error, please notify the sender immediately and destroy the original message and all copies.
Re: MRunit test invokes setup() and cleanup() repeatedly
Posted by Brock Noland <br...@cloudera.com>.
Hi,
Not that I am aware of. The good news is that we are planning on
releasing soon with changes that allow you to specify multiple input
key/value pairs.
Until then, you should be able to use a snapshot of trunk.
Brock
On Mon, Oct 15, 2012 at 6:08 AM, Dandona, Rohit
<Ro...@syntelinc.com> wrote:
> Hi Bertrand,
>
>
>
> Thanks for your reply.
>
> The problem which I am facing is in accordance with the issue reported in
> MRUNIT-64.
>
> Is there any other way to perform this test with multiple key value pair
> inputs ? Is it possible to set number of times the setup() and cleanup()
> methods get executed in the mrunit test to one ?
>
>
>
> Thanks again for your help.
>
> Rohit
>
>
>
>
>
> From: Bertrand Dechoux [mailto:dechouxb@gmail.com]
> Sent: Monday, October 15, 2012 1:05 PM
>
>
> To: user@mrunit.apache.org
> Subject: Re: MRunit test invokes setup() and cleanup() repeatedly
>
>
>
> Hi,
>
>
>
> I don't have a simple way to check MRUnit source code right now but :
> 1) could you provide a simple example? (if you add println to
> setup/map/reduce/cleanup methods, it should provide with enough information,
> we don't even need the values, hashes are fine).
> 2) your problem might be read like you are using a global shared variable
> which won't be possible within a cluster, could you provide more information
> about your problem? Are you specifying a specific input format where you now
> all related information will be in the same mapper? If MRunit actually call
> setup/cleanup for every map input, I don't see the harm in this except when
> using non standard input format. As for the reducer, if setup/cleanup are
> indeed called for every input, it might be a issue (with MRunit
> implementation) if you know and rely on the fact that you have only a single
> reducer.
>
> Regards
>
> Bertrand
>
> On Mon, Oct 15, 2012 at 8:59 AM, Dandona, Rohit
> <Ro...@syntelinc.com> wrote:
>
> Hi,
>
>
>
> I am trying to implement a MRunit test a MR job which picks up the best
> record from a set. The execution of the test invokes the setup() and
> cleanup() methods of the mapper/reducer class repeatedly (for every test
> input case). This results in the resetting of variables that are supposed to
> be set once during the job. Thus, incorrect results are obtained. Please
> suggest a way out.
>
>
>
> Thanks
>
>
>
>
>
> Confidential: This electronic message and all contents contain information
> from Syntel, Inc. which may be privileged, confidential or otherwise
> protected from disclosure. The information is intended to be for the
> addressee only. If you are not the addressee, any disclosure, copy,
> distribution or use of the contents of this message is prohibited. If you
> have received this electronic message in error, please notify the sender
> immediately and destroy the original message and all copies.
>
>
>
>
> --
> Bertrand Dechoux
>
> Confidential: This electronic message and all contents contain information
> from Syntel, Inc. which may be privileged, confidential or otherwise
> protected from disclosure. The information is intended to be for the
> addressee only. If you are not the addressee, any disclosure, copy,
> distribution or use of the contents of this message is prohibited. If you
> have received this electronic message in error, please notify the sender
> immediately and destroy the original message and all copies.
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
RE: MRunit test invokes setup() and cleanup() repeatedly
Posted by "Dandona, Rohit" <Ro...@SYNTELINC.COM>.
Hi Bertrand,
Thanks for your reply.
The problem which I am facing is in accordance with the issue reported in MRUNIT-64.
Is there any other way to perform this test with multiple key value pair inputs ? Is it possible to set number of times the setup() and cleanup() methods get executed in the mrunit test to one ?
Thanks again for your help.
Rohit
From: Bertrand Dechoux [mailto:dechouxb@gmail.com]
Sent: Monday, October 15, 2012 1:05 PM
To: user@mrunit.apache.org
Subject: Re: MRunit test invokes setup() and cleanup() repeatedly
Hi,
I don't have a simple way to check MRUnit source code right now but :
1) could you provide a simple example? (if you add println to setup/map/reduce/cleanup methods, it should provide with enough information, we don't even need the values, hashes are fine).
2) your problem might be read like you are using a global shared variable which won't be possible within a cluster, could you provide more information about your problem? Are you specifying a specific input format where you now all related information will be in the same mapper? If MRunit actually call setup/cleanup for every map input, I don't see the harm in this except when using non standard input format. As for the reducer, if setup/cleanup are indeed called for every input, it might be a issue (with MRunit implementation) if you know and rely on the fact that you have only a single reducer.
Regards
Bertrand
On Mon, Oct 15, 2012 at 8:59 AM, Dandona, Rohit <Ro...@syntelinc.com>> wrote:
Hi,
I am trying to implement a MRunit test a MR job which picks up the best record from a set. The execution of the test invokes the setup() and cleanup() methods of the mapper/reducer class repeatedly (for every test input case). This results in the resetting of variables that are supposed to be set once during the job. Thus, incorrect results are obtained. Please suggest a way out.
Thanks
Confidential: This electronic message and all contents contain information from Syntel, Inc. which may be privileged, confidential or otherwise protected from disclosure. The information is intended to be for the addressee only. If you are not the addressee, any disclosure, copy, distribution or use of the contents of this message is prohibited. If you have received this electronic message in error, please notify the sender immediately and destroy the original message and all copies.
--
Bertrand Dechoux
Confidential: This electronic message and all contents contain information from Syntel, Inc. which may be privileged, confidential or otherwise protected from disclosure. The information is intended to be for the addressee only. If you are not the addressee, any disclosure, copy, distribution or use of the contents of this message is prohibited. If you have received this electronic message in error, please notify the sender immediately and destroy the original message and all copies.
Re: MRunit test invokes setup() and cleanup() repeatedly
Posted by Bertrand Dechoux <de...@gmail.com>.
Hi,
I don't have a simple way to check MRUnit source code right now but :
1) could you provide a simple example? (if you add println to
setup/map/reduce/cleanup methods, it should provide with enough
information, we don't even need the values, hashes are fine).
2) your problem might be read like you are using a global shared variable
which won't be possible within a cluster, could you provide more
information about your problem? Are you specifying a specific input format
where you now all related information will be in the same mapper? If MRunit
actually call setup/cleanup for every map input, I don't see the harm in
this except when using non standard input format. As for the reducer, if
setup/cleanup are indeed called for every input, it might be a issue (with
MRunit implementation) if you know and rely on the fact that you have only
a single reducer.
Regards
Bertrand
On Mon, Oct 15, 2012 at 8:59 AM, Dandona, Rohit <Rohit_Dandona@syntelinc.com
> wrote:
> Hi,****
>
> ** **
>
> I am trying to implement a MRunit test a MR job which picks up the best
> record from a set. The execution of the test invokes the setup() and
> cleanup() methods of the mapper/reducer class repeatedly (for every test
> input case). This results in the resetting of variables that are supposed
> to be set once during the job. Thus, incorrect results are obtained.
> Please suggest a way out. ****
>
> ** **
>
> Thanks****
>
> ** **
>
> ** **
> Confidential: This electronic message and all contents contain
> information from Syntel, Inc. which may be privileged, confidential or
> otherwise protected from disclosure. The information is intended to be for
> the addressee only. If you are not the addressee, any disclosure, copy,
> distribution or use of the contents of this message is prohibited. If you
> have received this electronic message in error, please notify the sender
> immediately and destroy the original message and all copies.
>
--
Bertrand Dechoux
Re: MRunit test invokes setup() and cleanup() repeatedly
Posted by Dave Beech <da...@paraliatech.com>.
No problem about the code - fully appreciate the confidentiality issue.
The new (as yet unreleased) version of MRUnit does support multiple
key/value pairs, added in MRUNIT-64 as you've seen.
Maybe you could try building the new version of MRUnit from source
(located in git here:
http://git-wip-us.apache.org/repos/asf/mrunit.git) to see if this
solves your problem?
Thanks,
Dave
On 15 October 2012 11:50, Dandona, Rohit <Ro...@syntelinc.com> wrote:
> Hi Dave,
>
>
>
> Thanks for your reply.
>
>
>
> The problem which I am facing is in accordance with the issue reported in
> MRUNIT-64.
>
> As I mentioned earlier, my MR job picks the best record from a set of
> records. So the MRtest contains multiple key value pair inputs where a
> single pair would be the expected output and the rest will be ignored. The
> problem surfaces when the setup() and cleanup() methods of the
> mapper/reducer are invoked for every input pair, rendering a false result.
> Is there a new version of mrunit that supports multiple key value pairs ? Is
> there any other way to perform this test ?
>
> I am sorry, I cannot share the code with you for the reasons of
> confidentiality (I regret that L ).
>
>
>
> Thank you for your support.
>
> Rohit
>
>
>
>
>
> From: Dave Beech [mailto:dave@paraliatech.com]
> Sent: Monday, October 15, 2012 12:58 PM
> To: user@mrunit.apache.org
> Subject: Re: MRunit test invokes setup() and cleanup() repeatedly
>
>
>
> Hi Rohit
>
>
>
> Any chance you could share your mapper and test code to help us work out the
> problem? You can't attach files to these emails unfortunately so a tool like
> pastebin may help.
>
>
>
> I think your objects are being reused where they shouldn't be. Ideally you
> should be creating a new MapDriver and Mapper instance per test case; a
> setup method with the @Before method annotation from JUnit will help here.
>
>
>
> Thanks
>
> Dave
>
> On 15 Oct 2012, at 07:59, "Dandona, Rohit" <Ro...@SYNTELINC.COM>
> wrote:
>
> Hi,
>
>
>
> I am trying to implement a MRunit test a MR job which picks up the best
> record from a set. The execution of the test invokes the setup() and
> cleanup() methods of the mapper/reducer class repeatedly (for every test
> input case). This results in the resetting of variables that are supposed to
> be set once during the job. Thus, incorrect results are obtained. Please
> suggest a way out.
>
>
>
> Thanks
>
>
>
>
>
> Confidential: This electronic message and all contents contain information
> from Syntel, Inc. which may be privileged, confidential or otherwise
> protected from disclosure. The information is intended to be for the
> addressee only. If you are not the addressee, any disclosure, copy,
> distribution or use of the contents of this message is prohibited. If you
> have received this electronic message in error, please notify the sender
> immediately and destroy the original message and all copies.
>
> Confidential: This electronic message and all contents contain information
> from Syntel, Inc. which may be privileged, confidential or otherwise
> protected from disclosure. The information is intended to be for the
> addressee only. If you are not the addressee, any disclosure, copy,
> distribution or use of the contents of this message is prohibited. If you
> have received this electronic message in error, please notify the sender
> immediately and destroy the original message and all copies.
RE: MRunit test invokes setup() and cleanup() repeatedly
Posted by "Dandona, Rohit" <Ro...@SYNTELINC.COM>.
Hi Dave,
Thanks for your reply.
The problem which I am facing is in accordance with the issue reported in MRUNIT-64.
As I mentioned earlier, my MR job picks the best record from a set of records. So the MRtest contains multiple key value pair inputs where a single pair would be the expected output and the rest will be ignored. The problem surfaces when the setup() and cleanup() methods of the mapper/reducer are invoked for every input pair, rendering a false result. Is there a new version of mrunit that supports multiple key value pairs ? Is there any other way to perform this test ?
I am sorry, I cannot share the code with you for the reasons of confidentiality (I regret that ☹ ).
Thank you for your support.
Rohit
From: Dave Beech [mailto:dave@paraliatech.com]
Sent: Monday, October 15, 2012 12:58 PM
To: user@mrunit.apache.org
Subject: Re: MRunit test invokes setup() and cleanup() repeatedly
Hi Rohit
Any chance you could share your mapper and test code to help us work out the problem? You can't attach files to these emails unfortunately so a tool like pastebin may help.
I think your objects are being reused where they shouldn't be. Ideally you should be creating a new MapDriver and Mapper instance per test case; a setup method with the @Before method annotation from JUnit will help here.
Thanks
Dave
On 15 Oct 2012, at 07:59, "Dandona, Rohit" <Ro...@SYNTELINC.COM>> wrote:
Hi,
I am trying to implement a MRunit test a MR job which picks up the best record from a set. The execution of the test invokes the setup() and cleanup() methods of the mapper/reducer class repeatedly (for every test input case). This results in the resetting of variables that are supposed to be set once during the job. Thus, incorrect results are obtained. Please suggest a way out.
Thanks
Confidential: This electronic message and all contents contain information from Syntel, Inc. which may be privileged, confidential or otherwise protected from disclosure. The information is intended to be for the addressee only. If you are not the addressee, any disclosure, copy, distribution or use of the contents of this message is prohibited. If you have received this electronic message in error, please notify the sender immediately and destroy the original message and all copies.
Confidential: This electronic message and all contents contain information from Syntel, Inc. which may be privileged, confidential or otherwise protected from disclosure. The information is intended to be for the addressee only. If you are not the addressee, any disclosure, copy, distribution or use of the contents of this message is prohibited. If you have received this electronic message in error, please notify the sender immediately and destroy the original message and all copies.
Re: MRunit test invokes setup() and cleanup() repeatedly
Posted by Dave Beech <da...@paraliatech.com>.
Hi Rohit
Any chance you could share your mapper and test code to help us work out the problem? You can't attach files to these emails unfortunately so a tool like pastebin may help.
I think your objects are being reused where they shouldn't be. Ideally you should be creating a new MapDriver and Mapper instance per test case; a setup method with the @Before method annotation from JUnit will help here.
Thanks
Dave
On 15 Oct 2012, at 07:59, "Dandona, Rohit" <Ro...@SYNTELINC.COM> wrote:
> Hi,
>
> I am trying to implement a MRunit test a MR job which picks up the best record from a set. The execution of the test invokes the setup() and cleanup() methods of the mapper/reducer class repeatedly (for every test input case). This results in the resetting of variables that are supposed to be set once during the job. Thus, incorrect results are obtained. Please suggest a way out.
>
> Thanks
>
>
> Confidential: This electronic message and all contents contain information from Syntel, Inc. which may be privileged, confidential or otherwise protected from disclosure. The information is intended to be for the addressee only. If you are not the addressee, any disclosure, copy, distribution or use of the contents of this message is prohibited. If you have received this electronic message in error, please notify the sender immediately and destroy the original message and all copies.