You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@crunch.apache.org by Matthias Friedrich <ma...@mafr.de> on 2013/04/02 19:33:11 UTC
Exceptions in integration test cases
Hi,
I'm a bit worried about the IO exceptions we get in our integration
test (see [1], grep for EOFException). Since we don't validate the
results, there may be data loss, at least in local mode.
I have stumbled upon a test case [2] where Crunch seems to reliably
lose some data when using Writables, while the same code works fine
with Avros. This time, I was able to reproduce this on two machines
(with slightly different error messages, ie. [3]).
Has anyone seen this before? Is anyone able to reproduce the error
or is it just me? :)
Thanks,
Matthias
[1] https://builds.apache.org/job/Crunch-master-integration/244/consoleFull
[2] http://users.mafr.de/~matthias/crunch/CogroupIT.patch
[3] http://users.mafr.de/~matthias/crunch/CogroupIT-error.txt
Re: Exceptions in integration test cases
Posted by Matthias Friedrich <ma...@mafr.de>.
Fantastic, Gabriel. Thanks a lot!
Regards,
Matthias
On Tuesday, 2013-04-02, Gabriel Reid wrote:
>
> On 02 Apr 2013, at 21:39, Josh Wills <jo...@gmail.com> wrote:
>
> > In the context of that test (PCollectionGetSizeIT), the EOFException is
> > expected-- there isn't any data to read b/c the file is empty. We could
> > change up the impl to throw a CrunchRuntimeException instead.
> >
> > For the cogroup test, that sounds suspiciously like a deep copy issue.
> > Gabriel is the most knowledgable person to ask, he always fixes my deep
> > copy errors. :)
>
> Well that's a bizarre coincidence -- I just encountered this object reuse bug in CoGroup earlier today at work, and was going over mails from today while running the integration tests after having written a patch for it :-)
>
> In any case, yes, there is a deep copy issue with CoGroup -- I'm actually very surprised that this is still in there, as I was sure that I had gone over CoGroup in the past to make sure that object reuse wasn't an issue.
>
> In any case, I'll commit the fix shortly.
>
> - Gabriel
>
>
> > On Tue, Apr 2, 2013 at 10:33 AM, Matthias Friedrich <ma...@mafr.de> wrote:
> >
> >> Hi,
> >>
> >> I'm a bit worried about the IO exceptions we get in our integration
> >> test (see [1], grep for EOFException). Since we don't validate the
> >> results, there may be data loss, at least in local mode.
> >>
> >> I have stumbled upon a test case [2] where Crunch seems to reliably
> >> lose some data when using Writables, while the same code works fine
> >> with Avros. This time, I was able to reproduce this on two machines
> >> (with slightly different error messages, ie. [3]).
> >>
> >> Has anyone seen this before? Is anyone able to reproduce the error
> >> or is it just me? :)
> >>
> >> Thanks,
> >> Matthias
> >>
> >> [1]
> >> https://builds.apache.org/job/Crunch-master-integration/244/consoleFull
> >> [2] http://users.mafr.de/~matthias/crunch/CogroupIT.patch
> >> [3] http://users.mafr.de/~matthias/crunch/CogroupIT-error.txt
> >>
>
Re: Exceptions in integration test cases
Posted by Gabriel Reid <ga...@gmail.com>.
On 02 Apr 2013, at 21:39, Josh Wills <jo...@gmail.com> wrote:
> In the context of that test (PCollectionGetSizeIT), the EOFException is
> expected-- there isn't any data to read b/c the file is empty. We could
> change up the impl to throw a CrunchRuntimeException instead.
>
> For the cogroup test, that sounds suspiciously like a deep copy issue.
> Gabriel is the most knowledgable person to ask, he always fixes my deep
> copy errors. :)
Well that's a bizarre coincidence -- I just encountered this object reuse bug in CoGroup earlier today at work, and was going over mails from today while running the integration tests after having written a patch for it :-)
In any case, yes, there is a deep copy issue with CoGroup -- I'm actually very surprised that this is still in there, as I was sure that I had gone over CoGroup in the past to make sure that object reuse wasn't an issue.
In any case, I'll commit the fix shortly.
- Gabriel
> On Tue, Apr 2, 2013 at 10:33 AM, Matthias Friedrich <ma...@mafr.de> wrote:
>
>> Hi,
>>
>> I'm a bit worried about the IO exceptions we get in our integration
>> test (see [1], grep for EOFException). Since we don't validate the
>> results, there may be data loss, at least in local mode.
>>
>> I have stumbled upon a test case [2] where Crunch seems to reliably
>> lose some data when using Writables, while the same code works fine
>> with Avros. This time, I was able to reproduce this on two machines
>> (with slightly different error messages, ie. [3]).
>>
>> Has anyone seen this before? Is anyone able to reproduce the error
>> or is it just me? :)
>>
>> Thanks,
>> Matthias
>>
>> [1]
>> https://builds.apache.org/job/Crunch-master-integration/244/consoleFull
>> [2] http://users.mafr.de/~matthias/crunch/CogroupIT.patch
>> [3] http://users.mafr.de/~matthias/crunch/CogroupIT-error.txt
>>
Re: Exceptions in integration test cases
Posted by Josh Wills <jo...@gmail.com>.
In the context of that test (PCollectionGetSizeIT), the EOFException is
expected-- there isn't any data to read b/c the file is empty. We could
change up the impl to throw a CrunchRuntimeException instead.
For the cogroup test, that sounds suspiciously like a deep copy issue.
Gabriel is the most knowledgable person to ask, he always fixes my deep
copy errors. :)
Josh
On Tue, Apr 2, 2013 at 10:33 AM, Matthias Friedrich <ma...@mafr.de> wrote:
> Hi,
>
> I'm a bit worried about the IO exceptions we get in our integration
> test (see [1], grep for EOFException). Since we don't validate the
> results, there may be data loss, at least in local mode.
>
> I have stumbled upon a test case [2] where Crunch seems to reliably
> lose some data when using Writables, while the same code works fine
> with Avros. This time, I was able to reproduce this on two machines
> (with slightly different error messages, ie. [3]).
>
> Has anyone seen this before? Is anyone able to reproduce the error
> or is it just me? :)
>
> Thanks,
> Matthias
>
> [1]
> https://builds.apache.org/job/Crunch-master-integration/244/consoleFull
> [2] http://users.mafr.de/~matthias/crunch/CogroupIT.patch
> [3] http://users.mafr.de/~matthias/crunch/CogroupIT-error.txt
>