You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Abhijeet Apsunde <Ab...@persistent.co.in> on 2013/12/25 14:38:19 UTC
Intermittent test errors in Hadoop-hdfs project
Hi all,
I'm running Junit tests suite for Hadoop-hdfs project, however none of 3 run attempts could complete successfully.
There were random test failing due to some error and one consistent failure.
I believe one consistently broken test is due to following change, since corresponding "editsStored" binary file is not checked in with modified "editsStored.xml" file in this particular revision.
###############################
Revision: 1552841
Author: cmccabe
Date: Saturday, December 21, 2013 4:57:20 AM
Message:
HDFS-5636. Enforce a max TTL per cache pool (awang via cmccabe)
----
.
.
Modified : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
.
###############################
'Tests in error' passed successfully when ran separately. Test results snippet for all 3 runs is attached at the end of this mail.
I would like to know if others also have noted similar observation or there's something wrong with my setup.
If it's a common behaviour how does apache CI handles such scenarios while testing a patch?
Thanks,
Abhijeet.
______________________Test results snippets ________________________
Run1
Failed tests:
TestOfflineEditsViewer.testStored:193 Reference XML edits and parsed to XML should be same
Tests in error:
TestHDFSFileSystemContract.setUp:39 » IO Failed to save in any storage directo...
TestDatanodeReport.testDatanodeReport:48 » InconsistentFSState Directory /home...
#######################
Run2
Failed tests:
TestOfflineEditsViewer.testStored:193 Reference XML edits and parsed to XML should be same
Tests in error:
TestLeaseRecovery.testBlockSynchronization:80 » IO NameNode is not formatted.
TestDataNodeVolumeFailureToleration.setUp:71 » IO Timed out waiting for Mini H...
TestDataNodeVolumeFailureToleration.tearDown:83 NullPointer
#######################
Run 3
Failed tests:
TestOfflineEditsViewer.testStored:193 Reference XML edits and parsed to XML should be same
Tests in error:
TestAppendDifferentChecksum.setupCluster:52 » IO Cannot lock storage /home/abh...
TestFileAppend3.testTC12 » Remote No lease on /TC12/foo: File is not open for ...
TestHDFSFileSystemContract.setUp:39 » InconsistentFSState Directory /home/abhi...
######################
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
RE: Intermittent test errors in Hadoop-hdfs project
Posted by Abhijeet Apsunde <Ab...@persistent.co.in>.
Arpit,
No I haven't made any changes, it's the latest trunk.
In fact I made some changes and ran the suite when I stumbled upon these errors. To make sure that my changes aren't breaking anything I ran the suite on unpatched trunk and observed that those issues exist on trunk also.
But the strange thing is these 'tests in error' are not the same for every run. That's why I felt it could be related to machine. I'm attaching latest run's failed test reports.
In case the attachments don't get through, logs for tests also available at this location
http://abhizit.cloudapp.net/machine1/
http://abhizit.cloudapp.net/machine2/
These logs contains :
1. Console logs of test run : nohup.out.txt
2. Compressed 'target' folder for Hadoop-hdfs subproject. [no included in attachment because of its size] : tgz file
3. Test logs for 'tests in error' .
Thanks !
Abhijeet.
-----Original Message-----
From: Arpit Agarwal [mailto:aagarwal@hortonworks.com]
Sent: Saturday, December 28, 2013 12:04 AM
To: hdfs-dev@hadoop.apache.org
Subject: Re: Intermittent test errors in Hadoop-hdfs project
Hi Abhijeet, these tests pass consistently on my OS X dev machine.
Have you made any changes in your source tree? If you post logs for the failed tests perhaps someone can take a look.
On Thu, Dec 26, 2013 at 9:41 PM, Abhijeet Apsunde < Abhijeet_Apsunde@persistent.co.in> wrote:
> Thanks Arpit,
> As Ted also pointed out failed test is related to HDFS-5636, my
> question was mainly for 'tests in error' which I think is not faced by everyone.
>
> I'm running HDFS test suite and hitting some intermittent test errors.
> Tests in error vary for each run. And these error test cases execute
> successfully when ran separately.
> I tried these on 2 machines following is their configuration.
>
> 1. CentOS 6.2 , 8 core Xeon(R) CPU E5320 @ 1.86GHz, 32GB RAM.
> 2. Ubuntu 12.04 running on Windows Azure VM with 8 cores and 14 GB
> RAM
>
> Is there a recommended configuration for build machine ? if such
> errors are common how does apache CI handles these ?
>
> Following are results from machine 2.
>
> Tests in error:
> TestPersistBlocks.TestRestartDfsWithFlush:81->testRestartDfs:123 »
> IO Timed ou...
>
> TestDFSStartupVersions.testVersions:264->Object.wait:502->Object.wait:
> -2
> » te...
>
> Tests run: 2260, Failures: 2, Errors: 2, Skipped: 49
>
> Thanks,
> Abhijeet.
>
> -----Original Message-----
> From: Arpit Agarwal [mailto:aagarwal@hortonworks.com]
> Sent: Friday, December 27, 2013 1:09 AM
> To: hdfs-dev@hadoop.apache.org; Andrew Wang
> Subject: Re: Intermittent test errors in Hadoop-hdfs project
>
> It looks like HDFS-5636 missed updating editsStored.
>
>
> On Wed, Dec 25, 2013 at 8:31 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > TestOfflineEditsViewer is the one appearing in Jenkins builds.
> >
> > I ran the other tests locally on Mac and they passed.
> >
> > Cheers
> >
> >
> > On Wed, Dec 25, 2013 at 5:38 AM, Abhijeet Apsunde <
> > Abhijeet_Apsunde@persistent.co.in> wrote:
> >
> > > Hi all,
> > >
> > > I'm running Junit tests suite for Hadoop-hdfs project, however
> > > none of 3 run attempts could complete successfully.
> > > There were random test failing due to some error and one
> > > consistent failure.
> > >
> > > I believe one consistently broken test is due to following change,
> > > since corresponding "editsStored" binary file is not checked in
> > > with modified "editsStored.xml" file in this particular revision.
> > > ###############################
> > > Revision: 1552841
> > > Author: cmccabe
> > > Date: Saturday, December 21, 2013 4:57:20 AM
> > > Message:
> > > HDFS-5636. Enforce a max TTL per cache pool (awang via cmccabe)
> > > ----
> > > .
> > > .
> > > Modified :
> > >
> > /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resour
> > ce
> > s/editsStored.xml
> > > .
> > > ###############################
> > >
> > > 'Tests in error' passed successfully when ran separately. Test
> > > results snippet for all 3 runs is attached at the end of this mail.
> > > I would like to know if others also have noted similar observation
> > > or there's something wrong with my setup.
> > > If it's a common behaviour how does apache CI handles such
> > > scenarios
> > while
> > > testing a patch?
> > >
> > > Thanks,
> > > Abhijeet.
> > >
> > >
> > >
> > >
> > >
> > > ______________________Test results snippets
> > > ________________________
> > >
> > > Run1
> > > Failed tests:
> > > TestOfflineEditsViewer.testStored:193 Reference XML edits and
> > > parsed to XML should be same
> > >
> > > Tests in error:
> > > TestHDFSFileSystemContract.setUp:39 » IO Failed to save in any
> > > storage directo...
> > > TestDatanodeReport.testDatanodeReport:48 » InconsistentFSState
> > > Directory /home...
> > > #######################
> > > Run2
> > > Failed tests:
> > > TestOfflineEditsViewer.testStored:193 Reference XML edits and
> > > parsed to XML should be same
> > >
> > > Tests in error:
> > > TestLeaseRecovery.testBlockSynchronization:80 » IO NameNode is
> > > not formatted.
> > > TestDataNodeVolumeFailureToleration.setUp:71 » IO Timed out
> > > waiting
> > for
> > > Mini H...
> > > TestDataNodeVolumeFailureToleration.tearDown:83 NullPointer
> > > ####################### Run 3 Failed tests:
> > > TestOfflineEditsViewer.testStored:193 Reference XML edits and
> > > parsed to XML should be same
> > >
> > > Tests in error:
> > > TestAppendDifferentChecksum.setupCluster:52 » IO Cannot lock
> > > storage /home/abh...
> > > TestFileAppend3.testTC12 » Remote No lease on /TC12/foo: File
> > > is not open for ...
> > > TestHDFSFileSystemContract.setUp:39 » InconsistentFSState
> > > Directory /home/abhi...
> > > ######################
> > >
> > >
> > > DISCLAIMER
> > > ==========
> > > This e-mail may contain privileged and confidential information
> > > which is the property of Persistent Systems Ltd. It is intended
> > > only for the use
> > of
> > > the individual or entity to which it is addressed. If you are not
> > > the intended recipient, you are not authorized to read, retain,
> > > copy, print, distribute or use this message. If you have received
> > > this communication
> > in
> > > error, please notify the sender and delete all copies of this message.
> > > Persistent Systems Ltd. does not accept any liability for virus
> > > infected mails.
> > >
> > >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or
> entity to which it is addressed and may contain information that is
> confidential, privileged and exempt from disclosure under applicable
> law. If the reader of this message is not the intended recipient, you
> are hereby notified that any printing, copying, dissemination,
> distribution, disclosure or forwarding of this communication is
> strictly prohibited. If you have received this communication in error,
> please contact the sender immediately and delete it from your system. Thank You.
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which
> is the property of Persistent Systems Ltd. It is intended only for the
> use of the individual or entity to which it is addressed. If you are
> not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus
> infected mails.
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
Re: Intermittent test errors in Hadoop-hdfs project
Posted by Arpit Agarwal <aa...@hortonworks.com>.
Hi Abhijeet, these tests pass consistently on my OS X dev machine.
Have you made any changes in your source tree? If you post logs for the
failed tests perhaps someone can take a look.
On Thu, Dec 26, 2013 at 9:41 PM, Abhijeet Apsunde <
Abhijeet_Apsunde@persistent.co.in> wrote:
> Thanks Arpit,
> As Ted also pointed out failed test is related to HDFS-5636, my question
> was mainly for 'tests in error' which I think is not faced by everyone.
>
> I'm running HDFS test suite and hitting some intermittent test errors.
> Tests in error vary for each run. And these error test cases execute
> successfully when ran separately.
> I tried these on 2 machines following is their configuration.
>
> 1. CentOS 6.2 , 8 core Xeon(R) CPU E5320 @ 1.86GHz, 32GB RAM.
> 2. Ubuntu 12.04 running on Windows Azure VM with 8 cores and 14 GB RAM
>
> Is there a recommended configuration for build machine ? if such errors
> are common how does apache CI handles these ?
>
> Following are results from machine 2.
>
> Tests in error:
> TestPersistBlocks.TestRestartDfsWithFlush:81->testRestartDfs:123 » IO
> Timed ou...
> TestDFSStartupVersions.testVersions:264->Object.wait:502->Object.wait:-2
> » te...
>
> Tests run: 2260, Failures: 2, Errors: 2, Skipped: 49
>
> Thanks,
> Abhijeet.
>
> -----Original Message-----
> From: Arpit Agarwal [mailto:aagarwal@hortonworks.com]
> Sent: Friday, December 27, 2013 1:09 AM
> To: hdfs-dev@hadoop.apache.org; Andrew Wang
> Subject: Re: Intermittent test errors in Hadoop-hdfs project
>
> It looks like HDFS-5636 missed updating editsStored.
>
>
> On Wed, Dec 25, 2013 at 8:31 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > TestOfflineEditsViewer is the one appearing in Jenkins builds.
> >
> > I ran the other tests locally on Mac and they passed.
> >
> > Cheers
> >
> >
> > On Wed, Dec 25, 2013 at 5:38 AM, Abhijeet Apsunde <
> > Abhijeet_Apsunde@persistent.co.in> wrote:
> >
> > > Hi all,
> > >
> > > I'm running Junit tests suite for Hadoop-hdfs project, however none
> > > of 3 run attempts could complete successfully.
> > > There were random test failing due to some error and one consistent
> > > failure.
> > >
> > > I believe one consistently broken test is due to following change,
> > > since corresponding "editsStored" binary file is not checked in with
> > > modified "editsStored.xml" file in this particular revision.
> > > ###############################
> > > Revision: 1552841
> > > Author: cmccabe
> > > Date: Saturday, December 21, 2013 4:57:20 AM
> > > Message:
> > > HDFS-5636. Enforce a max TTL per cache pool (awang via cmccabe)
> > > ----
> > > .
> > > .
> > > Modified :
> > >
> > /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resource
> > s/editsStored.xml
> > > .
> > > ###############################
> > >
> > > 'Tests in error' passed successfully when ran separately. Test
> > > results snippet for all 3 runs is attached at the end of this mail.
> > > I would like to know if others also have noted similar observation
> > > or there's something wrong with my setup.
> > > If it's a common behaviour how does apache CI handles such scenarios
> > while
> > > testing a patch?
> > >
> > > Thanks,
> > > Abhijeet.
> > >
> > >
> > >
> > >
> > >
> > > ______________________Test results snippets ________________________
> > >
> > > Run1
> > > Failed tests:
> > > TestOfflineEditsViewer.testStored:193 Reference XML edits and
> > > parsed to XML should be same
> > >
> > > Tests in error:
> > > TestHDFSFileSystemContract.setUp:39 » IO Failed to save in any
> > > storage directo...
> > > TestDatanodeReport.testDatanodeReport:48 » InconsistentFSState
> > > Directory /home...
> > > #######################
> > > Run2
> > > Failed tests:
> > > TestOfflineEditsViewer.testStored:193 Reference XML edits and
> > > parsed to XML should be same
> > >
> > > Tests in error:
> > > TestLeaseRecovery.testBlockSynchronization:80 » IO NameNode is
> > > not formatted.
> > > TestDataNodeVolumeFailureToleration.setUp:71 » IO Timed out
> > > waiting
> > for
> > > Mini H...
> > > TestDataNodeVolumeFailureToleration.tearDown:83 NullPointer
> > > ####################### Run 3 Failed tests:
> > > TestOfflineEditsViewer.testStored:193 Reference XML edits and
> > > parsed to XML should be same
> > >
> > > Tests in error:
> > > TestAppendDifferentChecksum.setupCluster:52 » IO Cannot lock
> > > storage /home/abh...
> > > TestFileAppend3.testTC12 » Remote No lease on /TC12/foo: File is
> > > not open for ...
> > > TestHDFSFileSystemContract.setUp:39 » InconsistentFSState
> > > Directory /home/abhi...
> > > ######################
> > >
> > >
> > > DISCLAIMER
> > > ==========
> > > This e-mail may contain privileged and confidential information
> > > which is the property of Persistent Systems Ltd. It is intended only
> > > for the use
> > of
> > > the individual or entity to which it is addressed. If you are not
> > > the intended recipient, you are not authorized to read, retain,
> > > copy, print, distribute or use this message. If you have received
> > > this communication
> > in
> > > error, please notify the sender and delete all copies of this message.
> > > Persistent Systems Ltd. does not accept any liability for virus
> > > infected mails.
> > >
> > >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Persistent Systems Ltd. It is intended only for the use of
> the individual or entity to which it is addressed. If you are not the
> intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication in
> error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus infected
> mails.
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
RE: Intermittent test errors in Hadoop-hdfs project
Posted by Abhijeet Apsunde <Ab...@persistent.co.in>.
Thanks Arpit,
As Ted also pointed out failed test is related to HDFS-5636, my question was mainly for 'tests in error' which I think is not faced by everyone.
I'm running HDFS test suite and hitting some intermittent test errors. Tests in error vary for each run. And these error test cases execute successfully when ran separately.
I tried these on 2 machines following is their configuration.
1. CentOS 6.2 , 8 core Xeon(R) CPU E5320 @ 1.86GHz, 32GB RAM.
2. Ubuntu 12.04 running on Windows Azure VM with 8 cores and 14 GB RAM
Is there a recommended configuration for build machine ? if such errors are common how does apache CI handles these ?
Following are results from machine 2.
Tests in error:
TestPersistBlocks.TestRestartDfsWithFlush:81->testRestartDfs:123 » IO Timed ou...
TestDFSStartupVersions.testVersions:264->Object.wait:502->Object.wait:-2 » te...
Tests run: 2260, Failures: 2, Errors: 2, Skipped: 49
Thanks,
Abhijeet.
-----Original Message-----
From: Arpit Agarwal [mailto:aagarwal@hortonworks.com]
Sent: Friday, December 27, 2013 1:09 AM
To: hdfs-dev@hadoop.apache.org; Andrew Wang
Subject: Re: Intermittent test errors in Hadoop-hdfs project
It looks like HDFS-5636 missed updating editsStored.
On Wed, Dec 25, 2013 at 8:31 AM, Ted Yu <yu...@gmail.com> wrote:
> TestOfflineEditsViewer is the one appearing in Jenkins builds.
>
> I ran the other tests locally on Mac and they passed.
>
> Cheers
>
>
> On Wed, Dec 25, 2013 at 5:38 AM, Abhijeet Apsunde <
> Abhijeet_Apsunde@persistent.co.in> wrote:
>
> > Hi all,
> >
> > I'm running Junit tests suite for Hadoop-hdfs project, however none
> > of 3 run attempts could complete successfully.
> > There were random test failing due to some error and one consistent
> > failure.
> >
> > I believe one consistently broken test is due to following change,
> > since corresponding "editsStored" binary file is not checked in with
> > modified "editsStored.xml" file in this particular revision.
> > ###############################
> > Revision: 1552841
> > Author: cmccabe
> > Date: Saturday, December 21, 2013 4:57:20 AM
> > Message:
> > HDFS-5636. Enforce a max TTL per cache pool (awang via cmccabe)
> > ----
> > .
> > .
> > Modified :
> >
> /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resource
> s/editsStored.xml
> > .
> > ###############################
> >
> > 'Tests in error' passed successfully when ran separately. Test
> > results snippet for all 3 runs is attached at the end of this mail.
> > I would like to know if others also have noted similar observation
> > or there's something wrong with my setup.
> > If it's a common behaviour how does apache CI handles such scenarios
> while
> > testing a patch?
> >
> > Thanks,
> > Abhijeet.
> >
> >
> >
> >
> >
> > ______________________Test results snippets ________________________
> >
> > Run1
> > Failed tests:
> > TestOfflineEditsViewer.testStored:193 Reference XML edits and
> > parsed to XML should be same
> >
> > Tests in error:
> > TestHDFSFileSystemContract.setUp:39 » IO Failed to save in any
> > storage directo...
> > TestDatanodeReport.testDatanodeReport:48 » InconsistentFSState
> > Directory /home...
> > #######################
> > Run2
> > Failed tests:
> > TestOfflineEditsViewer.testStored:193 Reference XML edits and
> > parsed to XML should be same
> >
> > Tests in error:
> > TestLeaseRecovery.testBlockSynchronization:80 » IO NameNode is
> > not formatted.
> > TestDataNodeVolumeFailureToleration.setUp:71 » IO Timed out
> > waiting
> for
> > Mini H...
> > TestDataNodeVolumeFailureToleration.tearDown:83 NullPointer
> > ####################### Run 3 Failed tests:
> > TestOfflineEditsViewer.testStored:193 Reference XML edits and
> > parsed to XML should be same
> >
> > Tests in error:
> > TestAppendDifferentChecksum.setupCluster:52 » IO Cannot lock
> > storage /home/abh...
> > TestFileAppend3.testTC12 » Remote No lease on /TC12/foo: File is
> > not open for ...
> > TestHDFSFileSystemContract.setUp:39 » InconsistentFSState
> > Directory /home/abhi...
> > ######################
> >
> >
> > DISCLAIMER
> > ==========
> > This e-mail may contain privileged and confidential information
> > which is the property of Persistent Systems Ltd. It is intended only
> > for the use
> of
> > the individual or entity to which it is addressed. If you are not
> > the intended recipient, you are not authorized to read, retain,
> > copy, print, distribute or use this message. If you have received
> > this communication
> in
> > error, please notify the sender and delete all copies of this message.
> > Persistent Systems Ltd. does not accept any liability for virus
> > infected mails.
> >
> >
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
Re: Intermittent test errors in Hadoop-hdfs project
Posted by Arpit Agarwal <aa...@hortonworks.com>.
It looks like HDFS-5636 missed updating editsStored.
On Wed, Dec 25, 2013 at 8:31 AM, Ted Yu <yu...@gmail.com> wrote:
> TestOfflineEditsViewer is the one appearing in Jenkins builds.
>
> I ran the other tests locally on Mac and they passed.
>
> Cheers
>
>
> On Wed, Dec 25, 2013 at 5:38 AM, Abhijeet Apsunde <
> Abhijeet_Apsunde@persistent.co.in> wrote:
>
> > Hi all,
> >
> > I'm running Junit tests suite for Hadoop-hdfs project, however none of 3
> > run attempts could complete successfully.
> > There were random test failing due to some error and one consistent
> > failure.
> >
> > I believe one consistently broken test is due to following change, since
> > corresponding "editsStored" binary file is not checked in with modified
> > "editsStored.xml" file in this particular revision.
> > ###############################
> > Revision: 1552841
> > Author: cmccabe
> > Date: Saturday, December 21, 2013 4:57:20 AM
> > Message:
> > HDFS-5636. Enforce a max TTL per cache pool (awang via cmccabe)
> > ----
> > .
> > .
> > Modified :
> >
> /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
> > .
> > ###############################
> >
> > 'Tests in error' passed successfully when ran separately. Test results
> > snippet for all 3 runs is attached at the end of this mail.
> > I would like to know if others also have noted similar observation or
> > there's something wrong with my setup.
> > If it's a common behaviour how does apache CI handles such scenarios
> while
> > testing a patch?
> >
> > Thanks,
> > Abhijeet.
> >
> >
> >
> >
> >
> > ______________________Test results snippets ________________________
> >
> > Run1
> > Failed tests:
> > TestOfflineEditsViewer.testStored:193 Reference XML edits and parsed to
> > XML should be same
> >
> > Tests in error:
> > TestHDFSFileSystemContract.setUp:39 » IO Failed to save in any storage
> > directo...
> > TestDatanodeReport.testDatanodeReport:48 » InconsistentFSState
> > Directory /home...
> > #######################
> > Run2
> > Failed tests:
> > TestOfflineEditsViewer.testStored:193 Reference XML edits and parsed to
> > XML should be same
> >
> > Tests in error:
> > TestLeaseRecovery.testBlockSynchronization:80 » IO NameNode is not
> > formatted.
> > TestDataNodeVolumeFailureToleration.setUp:71 » IO Timed out waiting
> for
> > Mini H...
> > TestDataNodeVolumeFailureToleration.tearDown:83 NullPointer
> > #######################
> > Run 3
> > Failed tests:
> > TestOfflineEditsViewer.testStored:193 Reference XML edits and parsed to
> > XML should be same
> >
> > Tests in error:
> > TestAppendDifferentChecksum.setupCluster:52 » IO Cannot lock storage
> > /home/abh...
> > TestFileAppend3.testTC12 » Remote No lease on /TC12/foo: File is not
> > open for ...
> > TestHDFSFileSystemContract.setUp:39 » InconsistentFSState Directory
> > /home/abhi...
> > ######################
> >
> >
> > DISCLAIMER
> > ==========
> > This e-mail may contain privileged and confidential information which is
> > the property of Persistent Systems Ltd. It is intended only for the use
> of
> > the individual or entity to which it is addressed. If you are not the
> > intended recipient, you are not authorized to read, retain, copy, print,
> > distribute or use this message. If you have received this communication
> in
> > error, please notify the sender and delete all copies of this message.
> > Persistent Systems Ltd. does not accept any liability for virus infected
> > mails.
> >
> >
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Intermittent test errors in Hadoop-hdfs project
Posted by Ted Yu <yu...@gmail.com>.
TestOfflineEditsViewer is the one appearing in Jenkins builds.
I ran the other tests locally on Mac and they passed.
Cheers
On Wed, Dec 25, 2013 at 5:38 AM, Abhijeet Apsunde <
Abhijeet_Apsunde@persistent.co.in> wrote:
> Hi all,
>
> I'm running Junit tests suite for Hadoop-hdfs project, however none of 3
> run attempts could complete successfully.
> There were random test failing due to some error and one consistent
> failure.
>
> I believe one consistently broken test is due to following change, since
> corresponding "editsStored" binary file is not checked in with modified
> "editsStored.xml" file in this particular revision.
> ###############################
> Revision: 1552841
> Author: cmccabe
> Date: Saturday, December 21, 2013 4:57:20 AM
> Message:
> HDFS-5636. Enforce a max TTL per cache pool (awang via cmccabe)
> ----
> .
> .
> Modified :
> /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
> .
> ###############################
>
> 'Tests in error' passed successfully when ran separately. Test results
> snippet for all 3 runs is attached at the end of this mail.
> I would like to know if others also have noted similar observation or
> there's something wrong with my setup.
> If it's a common behaviour how does apache CI handles such scenarios while
> testing a patch?
>
> Thanks,
> Abhijeet.
>
>
>
>
>
> ______________________Test results snippets ________________________
>
> Run1
> Failed tests:
> TestOfflineEditsViewer.testStored:193 Reference XML edits and parsed to
> XML should be same
>
> Tests in error:
> TestHDFSFileSystemContract.setUp:39 » IO Failed to save in any storage
> directo...
> TestDatanodeReport.testDatanodeReport:48 » InconsistentFSState
> Directory /home...
> #######################
> Run2
> Failed tests:
> TestOfflineEditsViewer.testStored:193 Reference XML edits and parsed to
> XML should be same
>
> Tests in error:
> TestLeaseRecovery.testBlockSynchronization:80 » IO NameNode is not
> formatted.
> TestDataNodeVolumeFailureToleration.setUp:71 » IO Timed out waiting for
> Mini H...
> TestDataNodeVolumeFailureToleration.tearDown:83 NullPointer
> #######################
> Run 3
> Failed tests:
> TestOfflineEditsViewer.testStored:193 Reference XML edits and parsed to
> XML should be same
>
> Tests in error:
> TestAppendDifferentChecksum.setupCluster:52 » IO Cannot lock storage
> /home/abh...
> TestFileAppend3.testTC12 » Remote No lease on /TC12/foo: File is not
> open for ...
> TestHDFSFileSystemContract.setUp:39 » InconsistentFSState Directory
> /home/abhi...
> ######################
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Persistent Systems Ltd. It is intended only for the use of
> the individual or entity to which it is addressed. If you are not the
> intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication in
> error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus infected
> mails.
>
>