You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Hadoop QA (JIRA)" <ji...@apache.org> on 2015/08/06 04:31:04 UTC
[jira] [Commented] (MAPREDUCE-6444) Add a checkpointable version of
shuffle and reduce context supported by a checkpoint manager which uses the
HDFS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659365#comment-14659365 ]
Hadoop QA commented on MAPREDUCE-6444:
--------------------------------------
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 21m 59s | Pre-patch trunk compilation is healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 6 new or modified test files. |
| {color:red}-1{color} | javac | 8m 38s | The applied patch generated 1 additional warning messages. |
| {color:green}+1{color} | javadoc | 10m 17s | There were no new javadoc warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 39s | The applied patch generated 7 new checkstyle issues (total was 294, now 296). |
| {color:green}+1{color} | whitespace | 0m 6s | The patch has no lines that end in whitespace. |
| {color:green}+1{color} | install | 1m 26s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 3m 25s | The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | mapreduce tests | 9m 15s | Tests failed in hadoop-mapreduce-client-app. |
| {color:red}-1{color} | mapreduce tests | 1m 44s | Tests failed in hadoop-mapreduce-client-core. |
| {color:red}-1{color} | mapreduce tests | 149m 19s | Tests failed in hadoop-mapreduce-client-jobclient. |
| | | 209m 0s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-mapreduce-client-core |
| Failed unit tests | hadoop.mapreduce.TestReducerPreemption |
| | hadoop.mapreduce.task.reduce.TestMerger |
| | hadoop.mapreduce.task.reduce.TestShufflePreemption |
| | hadoop.mapreduce.security.ssl.TestEncryptedShuffle |
| | hadoop.mapreduce.lib.map.TestMultithreadedMapper |
| | hadoop.mapred.TestLineRecordReaderJobs |
| | hadoop.mapreduce.lib.join.TestJoinDatamerge |
| | hadoop.mapred.TestMapOutputType |
| | hadoop.conf.TestNoDefaultsJobConf |
| | hadoop.mapreduce.lib.input.TestMultipleInputs |
| | hadoop.mapred.jobcontrol.TestLocalJobControl |
| | hadoop.mapred.lib.TestChainMapReduce |
| | hadoop.mapreduce.lib.chain.TestSingleElementChain |
| | hadoop.fs.TestDFSIO |
| | hadoop.mapreduce.TestMROutputFormat |
| | hadoop.mapreduce.TestLocalRunner |
| | hadoop.mapred.TestFieldSelection |
| | hadoop.mapred.TestMiniMRClientCluster |
| | hadoop.mapred.TestClusterMRNotification |
| | hadoop.mapreduce.lib.partition.TestMRKeyFieldBasedComparator |
| | hadoop.mapred.TestMRTimelineEventHandling |
| | hadoop.mapreduce.TestMapCollection |
| | hadoop.mapreduce.lib.chain.TestMapReduceChain |
| | hadoop.mapred.TestLocalMRNotification |
| | hadoop.mapreduce.v2.TestMRJobsWithHistoryService |
| | hadoop.mapreduce.lib.aggregate.TestMapReduceAggregates |
| | hadoop.mapred.TestUserDefinedCounters |
| | hadoop.mapreduce.security.TestBinaryTokenFile |
| | hadoop.mapred.TestReduceFetch |
| | hadoop.mapreduce.v2.TestMiniMRProxyUser |
| | hadoop.mapred.TestOldCombinerGrouping |
| | hadoop.mapreduce.lib.db.TestDataDrivenDBInputFormat |
| | hadoop.mapreduce.lib.chain.TestChainErrors |
| | hadoop.mapreduce.TestValueIterReset |
| | hadoop.mapreduce.v2.TestMROldApiJobs |
| | hadoop.fs.slive.TestSlive |
| | hadoop.mapred.TestClusterMapReduceTestCase |
| | hadoop.mapred.TestMRIntermediateDataEncryption |
| | hadoop.mapred.TestReduceFetchFromPartialMem |
| | hadoop.mapreduce.TestNewCombinerGrouping |
| | hadoop.fs.TestFileSystem |
| | hadoop.mapreduce.lib.fieldsel.TestMRFieldSelection |
| | hadoop.mapred.TestJobName |
| | hadoop.mapreduce.lib.input.TestLineRecordReaderJobs |
| | hadoop.mapred.TestCollect |
| | hadoop.mapreduce.v2.TestMRJobsWithProfiler |
| | hadoop.mapred.TestLocalJobSubmission |
| | hadoop.mapred.TestJobCounters |
| | hadoop.mapreduce.TestMapReduceLazyOutput |
| | hadoop.mapred.TestReduceTask |
| | hadoop.mapred.TestJavaSerialization |
| | hadoop.mapreduce.lib.output.TestMRMultipleOutputs |
| | hadoop.mapred.TestSpecialCharactersInOutputPath |
| | hadoop.mapreduce.v2.TestUberAM |
| | hadoop.mapreduce.security.TestMRCredentials |
| | hadoop.mapred.TestLazyOutput |
| | hadoop.mapred.TestFileOutputFormat |
| | hadoop.mapreduce.v2.TestMRAMWithNonNormalizedCapabilities |
| | hadoop.mapred.pipes.TestPipeApplication |
| | hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl |
| | hadoop.mapreduce.TestChild |
| | hadoop.mapred.lib.TestMultithreadedMapRunner |
| | hadoop.mapred.TestReporter |
| | hadoop.mapred.lib.aggregate.TestAggregates |
| | hadoop.mapred.TestMapRed |
| | hadoop.mapred.lib.TestMultipleOutputs |
| | hadoop.mapreduce.TestMapReduce |
| | hadoop.mapred.TestComparators |
| | hadoop.mapreduce.TestMRJobClient |
| | hadoop.mapred.lib.TestKeyFieldBasedComparator |
| | hadoop.mapred.join.TestDatamerge |
| Timed out tests | org.apache.hadoop.mapred.TestJobSysDirWithDFS |
| | org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers |
| | org.apache.hadoop.mapred.TestMiniMRClasspath |
| | org.apache.hadoop.mapreduce.v2.TestMRJobs |
| | org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | http://issues.apache.org/jira/secure/attachment/12748950/MAPREDUCE-6444.001.patch |
| Optional Tests | javac unit findbugs checkstyle javadoc |
| git revision | trunk / 3da0bed |
| javac | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5930/artifact/patchprocess/diffJavacWarnings.txt |
| checkstyle | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5930/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt |
| Findbugs warnings | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5930/artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html |
| hadoop-mapreduce-client-app test log | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5930/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt |
| hadoop-mapreduce-client-core test log | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5930/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt |
| hadoop-mapreduce-client-jobclient test log | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5930/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt |
| Test Results | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5930/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5930/console |
This message was automatically generated.
> Add a checkpointable version of shuffle and reduce context supported by a checkpoint manager which uses the HDFS
> ----------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-6444
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6444
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Augusto Souza
> Assignee: Augusto Souza
> Attachments: MAPREDUCE-6444.001.patch
>
>
> Patch to add classes to provide a checkpointable version of shuffle and reduce context. Also, add a classe which makes possible to save a checkpoint of a task in HDFS. Those classes are respectively: CheckpointableShuffle, CheckpointableReduceContextImpl and TaskCheckpointManager.
> This improvement also helps in the splitting of patch in MAPREDUCE-5269 that is too large for being reviewed or accepted by Jenkins CI scripts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)