You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Junping Du <jd...@hortonworks.com> on 2015/12/01 13:26:44 UTC
Re: continuing releases on Apache Hadoop 2.6.x
Thanks Vinod!
I went through all 31 fixes (Common: 8, HDFS: 6, MAPREDUCE: 8, YARN: 9) landed on branch-2.6? now, and all outstanding patches (>=5KB) are listed below and they are all bug fix.
Haven't found any high risky patches so far but I noticed we have 3 significant fixes (MAPREDUCE-5948, MAPREDUCE-6481 and MAPREDUCE-6549) for LineRecordReader. Among them, MAPREDUCE-5948 and MAPREDUCE-6481 are completed fixes and we will ship them in 2.7.2, so I think there is no risk with them. MAPREDUCE-6549, together with an open JIRA: MAPREDUCE-6558, sounds like fix the same issue for uncompressed/compressed input file. I already ping the JIRA owners to get suggestions on if we could release MAPREDUCE-6549 independently or together with MAPREDUCE-6558 in next release.
I already pinged all owners for open JIRAs that targeted for 2.6.3 to move it to 2.6.4. Based on the feedback from JIRAs (and voices here too), I will create branch-2.6.3 in next 1 day or 2. Thanks!?
List of Outstanding Patches:
HADOOP-10134 [JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments. (apurtell via stevel) 38K
HDFS-9289. Make DataStreamer#block thread safe and verify genStamp in commitBlock. (Chang Li via zhz) 12K
HDFS-9470. Encryption zone on root not loaded from fsimage after NN restart. (Xiao Chen via wang) 5K
MAPREDUCE-6273. HistoryFileManager should check whether summaryFile exists to avoid FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state. (zhihai xu via devaraj) 5K
MAPREDUCE-5948. org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record delimiters well (Vinayakumar B, Rushabh Shah, and Akira AJISAKA via jlowe) 20K
MAPREDUCE-6481. LineRecordReader may give incomplete record and wrong position/key information for uncompressed input sometimes. (Zhihai Xu via jlowe) 19K
MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause duplicate records (wilfreds via rkanter) 31K
YARN-2859. ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster.(Vinod Kumar Vavilapalli via xgong) 6K (with addendum)
YARN-4344. NMs reconnecting with changed capabilities can lead to wrong cluster resource calculations (Varun Vasudev via jlowe) 12K
YARN-3925. ContainerLogsUtils#getContainerLogFile fails to read container log files from full disks. (zhihai xu via jlowe) 7K
Thanks,
Junping
________________________________
From: Vinod Kumar Vavilapalli <vi...@apache.org>
Sent: Monday, November 30, 2015 9:23 PM
To: Junping Du
Cc: Sangjin Lee; mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; Hadoop Common; hdfs-dev@hadoop.apache.org; Haohui Mai; Chris Trezzo
Subject: Re: continuing releases on Apache Hadoop 2.6.x
+1, this sounds good to me.
Thanks
+Vinod
On Nov 25, 2015, at 11:44 AM, Junping Du <jd...@hortonworks.com>> wrote:
May be we should quickly go through landed patches in 2.6.3 list and postpone any risky patches to 2.6.4?
Release plan for Apache Hadoop 2.6.3
Posted by Junping Du <jd...@hortonworks.com>.
Hi all,
Based on the feedback from previous email threads and JIRAs, we are in good time for releasing 2.6.3!
Things Done/TODO:
+ Branch
-- Based on branch-2.6, which has been open to 2.6.3 commits for a while, I already branch out branch-2.6.3 today.
-- New commits coming as fix to 2.6.3 need to explicitly cherry-pick to branch-2.6.3 as well as branch-2.6.
+ Patches
-- 2.6.3 already has 31 fixes as list of [1] shows.
-- The list of open tickets [2] on 2.6.3 is relative small (3 for now) as we already move non-blocker / non-critical ones out to 2.6.4. I'll keep triaging and see what we can get in within a week or so.
+ Release
-- Given the open ticket list is relative short and all of them are patch available, I plan to start RC process for voting early next week with leaving aside some buffer time.
Thoughts?
Appreciate any help in moving forward open tickets [2].
Thanks,
Junping
[1] 2.6.3 Fixed Tickets:
http://s.apache.org/Ylq
[2] 2.6.3 Tickets:
http://s.apache.org/9A7
________________________________
From: Junping Du
Sent: Tuesday, December 01, 2015 12:26 PM
To: Vinod Kumar Vavilapalli
Cc: Sangjin Lee; mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; Hadoop Common; hdfs-dev@hadoop.apache.org; Haohui Mai; Chris Trezzo; junping_du@apache.org
Subject: Re: continuing releases on Apache Hadoop 2.6.x
Thanks Vinod!
I went through all 31 fixes (Common: 8, HDFS: 6, MAPREDUCE: 8, YARN: 9) landed on branch-2.6 now, and all outstanding patches (>=5KB) are listed below and they are all bug fix.
Haven't found any high risky patches so far but I noticed we have 3 significant fixes (MAPREDUCE-5948, MAPREDUCE-6481 and MAPREDUCE-6549) for LineRecordReader. Among them, MAPREDUCE-5948 and MAPREDUCE-6481 are completed fixes and we will ship them in 2.7.2, so I think there is no risk with them. MAPREDUCE-6549, together with an open JIRA: MAPREDUCE-6558, sounds like fix the same issue for uncompressed/compressed input file. I already ping the JIRA owners to get suggestions on if we could release MAPREDUCE-6549 independently or together with MAPREDUCE-6558 in next release.
I already pinged all owners for open JIRAs that targeted for 2.6.3 to move it to 2.6.4. Based on the feedback from JIRAs (and voices here too), I will create branch-2.6.3 in next 1 day or 2. Thanks!
List of Outstanding Patches:
HADOOP-10134 [JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments. (apurtell via stevel) 38K
HDFS-9289. Make DataStreamer#block thread safe and verify genStamp in commitBlock. (Chang Li via zhz) 12K
HDFS-9470. Encryption zone on root not loaded from fsimage after NN restart. (Xiao Chen via wang) 5K
MAPREDUCE-6273. HistoryFileManager should check whether summaryFile exists to avoid FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state. (zhihai xu via devaraj) 5K
MAPREDUCE-5948. org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record delimiters well (Vinayakumar B, Rushabh Shah, and Akira AJISAKA via jlowe) 20K
MAPREDUCE-6481. LineRecordReader may give incomplete record and wrong position/key information for uncompressed input sometimes. (Zhihai Xu via jlowe) 19K
MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause duplicate records (wilfreds via rkanter) 31K
YARN-2859. ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster.(Vinod Kumar Vavilapalli via xgong) 6K (with addendum)
YARN-4344. NMs reconnecting with changed capabilities can lead to wrong cluster resource calculations (Varun Vasudev via jlowe) 12K
YARN-3925. ContainerLogsUtils#getContainerLogFile fails to read container log files from full disks. (zhihai xu via jlowe) 7K
Thanks,
Junping
________________________________
From: Vinod Kumar Vavilapalli <vi...@apache.org>
Sent: Monday, November 30, 2015 9:23 PM
To: Junping Du
Cc: Sangjin Lee; mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; Hadoop Common; hdfs-dev@hadoop.apache.org; Haohui Mai; Chris Trezzo
Subject: Re: continuing releases on Apache Hadoop 2.6.x
+1, this sounds good to me.
Thanks
+Vinod
On Nov 25, 2015, at 11:44 AM, Junping Du <jd...@hortonworks.com>> wrote:
May be we should quickly go through landed patches in 2.6.3 list and postpone any risky patches to 2.6.4?
Release plan for Apache Hadoop 2.6.3
Posted by Junping Du <jd...@hortonworks.com>.
Hi all,
Based on the feedback from previous email threads and JIRAs, we are in good time for releasing 2.6.3!
Things Done/TODO:
+ Branch
-- Based on branch-2.6, which has been open to 2.6.3 commits for a while, I already branch out branch-2.6.3 today.
-- New commits coming as fix to 2.6.3 need to explicitly cherry-pick to branch-2.6.3 as well as branch-2.6.
+ Patches
-- 2.6.3 already has 31 fixes as list of [1] shows.
-- The list of open tickets [2] on 2.6.3 is relative small (3 for now) as we already move non-blocker / non-critical ones out to 2.6.4. I'll keep triaging and see what we can get in within a week or so.
+ Release
-- Given the open ticket list is relative short and all of them are patch available, I plan to start RC process for voting early next week with leaving aside some buffer time.
Thoughts?
Appreciate any help in moving forward open tickets [2].
Thanks,
Junping
[1] 2.6.3 Fixed Tickets:
http://s.apache.org/Ylq
[2] 2.6.3 Tickets:
http://s.apache.org/9A7
________________________________
From: Junping Du
Sent: Tuesday, December 01, 2015 12:26 PM
To: Vinod Kumar Vavilapalli
Cc: Sangjin Lee; mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; Hadoop Common; hdfs-dev@hadoop.apache.org; Haohui Mai; Chris Trezzo; junping_du@apache.org
Subject: Re: continuing releases on Apache Hadoop 2.6.x
Thanks Vinod!
I went through all 31 fixes (Common: 8, HDFS: 6, MAPREDUCE: 8, YARN: 9) landed on branch-2.6 now, and all outstanding patches (>=5KB) are listed below and they are all bug fix.
Haven't found any high risky patches so far but I noticed we have 3 significant fixes (MAPREDUCE-5948, MAPREDUCE-6481 and MAPREDUCE-6549) for LineRecordReader. Among them, MAPREDUCE-5948 and MAPREDUCE-6481 are completed fixes and we will ship them in 2.7.2, so I think there is no risk with them. MAPREDUCE-6549, together with an open JIRA: MAPREDUCE-6558, sounds like fix the same issue for uncompressed/compressed input file. I already ping the JIRA owners to get suggestions on if we could release MAPREDUCE-6549 independently or together with MAPREDUCE-6558 in next release.
I already pinged all owners for open JIRAs that targeted for 2.6.3 to move it to 2.6.4. Based on the feedback from JIRAs (and voices here too), I will create branch-2.6.3 in next 1 day or 2. Thanks!
List of Outstanding Patches:
HADOOP-10134 [JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments. (apurtell via stevel) 38K
HDFS-9289. Make DataStreamer#block thread safe and verify genStamp in commitBlock. (Chang Li via zhz) 12K
HDFS-9470. Encryption zone on root not loaded from fsimage after NN restart. (Xiao Chen via wang) 5K
MAPREDUCE-6273. HistoryFileManager should check whether summaryFile exists to avoid FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state. (zhihai xu via devaraj) 5K
MAPREDUCE-5948. org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record delimiters well (Vinayakumar B, Rushabh Shah, and Akira AJISAKA via jlowe) 20K
MAPREDUCE-6481. LineRecordReader may give incomplete record and wrong position/key information for uncompressed input sometimes. (Zhihai Xu via jlowe) 19K
MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause duplicate records (wilfreds via rkanter) 31K
YARN-2859. ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster.(Vinod Kumar Vavilapalli via xgong) 6K (with addendum)
YARN-4344. NMs reconnecting with changed capabilities can lead to wrong cluster resource calculations (Varun Vasudev via jlowe) 12K
YARN-3925. ContainerLogsUtils#getContainerLogFile fails to read container log files from full disks. (zhihai xu via jlowe) 7K
Thanks,
Junping
________________________________
From: Vinod Kumar Vavilapalli <vi...@apache.org>
Sent: Monday, November 30, 2015 9:23 PM
To: Junping Du
Cc: Sangjin Lee; mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; Hadoop Common; hdfs-dev@hadoop.apache.org; Haohui Mai; Chris Trezzo
Subject: Re: continuing releases on Apache Hadoop 2.6.x
+1, this sounds good to me.
Thanks
+Vinod
On Nov 25, 2015, at 11:44 AM, Junping Du <jd...@hortonworks.com>> wrote:
May be we should quickly go through landed patches in 2.6.3 list and postpone any risky patches to 2.6.4?
Release plan for Apache Hadoop 2.6.3
Posted by Junping Du <jd...@hortonworks.com>.
Hi all,
Based on the feedback from previous email threads and JIRAs, we are in good time for releasing 2.6.3!
Things Done/TODO:
+ Branch
-- Based on branch-2.6, which has been open to 2.6.3 commits for a while, I already branch out branch-2.6.3 today.
-- New commits coming as fix to 2.6.3 need to explicitly cherry-pick to branch-2.6.3 as well as branch-2.6.
+ Patches
-- 2.6.3 already has 31 fixes as list of [1] shows.
-- The list of open tickets [2] on 2.6.3 is relative small (3 for now) as we already move non-blocker / non-critical ones out to 2.6.4. I'll keep triaging and see what we can get in within a week or so.
+ Release
-- Given the open ticket list is relative short and all of them are patch available, I plan to start RC process for voting early next week with leaving aside some buffer time.
Thoughts?
Appreciate any help in moving forward open tickets [2].
Thanks,
Junping
[1] 2.6.3 Fixed Tickets:
http://s.apache.org/Ylq
[2] 2.6.3 Tickets:
http://s.apache.org/9A7
________________________________
From: Junping Du
Sent: Tuesday, December 01, 2015 12:26 PM
To: Vinod Kumar Vavilapalli
Cc: Sangjin Lee; mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; Hadoop Common; hdfs-dev@hadoop.apache.org; Haohui Mai; Chris Trezzo; junping_du@apache.org
Subject: Re: continuing releases on Apache Hadoop 2.6.x
Thanks Vinod!
I went through all 31 fixes (Common: 8, HDFS: 6, MAPREDUCE: 8, YARN: 9) landed on branch-2.6 now, and all outstanding patches (>=5KB) are listed below and they are all bug fix.
Haven't found any high risky patches so far but I noticed we have 3 significant fixes (MAPREDUCE-5948, MAPREDUCE-6481 and MAPREDUCE-6549) for LineRecordReader. Among them, MAPREDUCE-5948 and MAPREDUCE-6481 are completed fixes and we will ship them in 2.7.2, so I think there is no risk with them. MAPREDUCE-6549, together with an open JIRA: MAPREDUCE-6558, sounds like fix the same issue for uncompressed/compressed input file. I already ping the JIRA owners to get suggestions on if we could release MAPREDUCE-6549 independently or together with MAPREDUCE-6558 in next release.
I already pinged all owners for open JIRAs that targeted for 2.6.3 to move it to 2.6.4. Based on the feedback from JIRAs (and voices here too), I will create branch-2.6.3 in next 1 day or 2. Thanks!
List of Outstanding Patches:
HADOOP-10134 [JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments. (apurtell via stevel) 38K
HDFS-9289. Make DataStreamer#block thread safe and verify genStamp in commitBlock. (Chang Li via zhz) 12K
HDFS-9470. Encryption zone on root not loaded from fsimage after NN restart. (Xiao Chen via wang) 5K
MAPREDUCE-6273. HistoryFileManager should check whether summaryFile exists to avoid FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state. (zhihai xu via devaraj) 5K
MAPREDUCE-5948. org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record delimiters well (Vinayakumar B, Rushabh Shah, and Akira AJISAKA via jlowe) 20K
MAPREDUCE-6481. LineRecordReader may give incomplete record and wrong position/key information for uncompressed input sometimes. (Zhihai Xu via jlowe) 19K
MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause duplicate records (wilfreds via rkanter) 31K
YARN-2859. ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster.(Vinod Kumar Vavilapalli via xgong) 6K (with addendum)
YARN-4344. NMs reconnecting with changed capabilities can lead to wrong cluster resource calculations (Varun Vasudev via jlowe) 12K
YARN-3925. ContainerLogsUtils#getContainerLogFile fails to read container log files from full disks. (zhihai xu via jlowe) 7K
Thanks,
Junping
________________________________
From: Vinod Kumar Vavilapalli <vi...@apache.org>
Sent: Monday, November 30, 2015 9:23 PM
To: Junping Du
Cc: Sangjin Lee; mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; Hadoop Common; hdfs-dev@hadoop.apache.org; Haohui Mai; Chris Trezzo
Subject: Re: continuing releases on Apache Hadoop 2.6.x
+1, this sounds good to me.
Thanks
+Vinod
On Nov 25, 2015, at 11:44 AM, Junping Du <jd...@hortonworks.com>> wrote:
May be we should quickly go through landed patches in 2.6.3 list and postpone any risky patches to 2.6.4?
Release plan for Apache Hadoop 2.6.3
Posted by Junping Du <jd...@hortonworks.com>.
Hi all,
Based on the feedback from previous email threads and JIRAs, we are in good time for releasing 2.6.3!
Things Done/TODO:
+ Branch
-- Based on branch-2.6, which has been open to 2.6.3 commits for a while, I already branch out branch-2.6.3 today.
-- New commits coming as fix to 2.6.3 need to explicitly cherry-pick to branch-2.6.3 as well as branch-2.6.
+ Patches
-- 2.6.3 already has 31 fixes as list of [1] shows.
-- The list of open tickets [2] on 2.6.3 is relative small (3 for now) as we already move non-blocker / non-critical ones out to 2.6.4. I'll keep triaging and see what we can get in within a week or so.
+ Release
-- Given the open ticket list is relative short and all of them are patch available, I plan to start RC process for voting early next week with leaving aside some buffer time.
Thoughts?
Appreciate any help in moving forward open tickets [2].
Thanks,
Junping
[1] 2.6.3 Fixed Tickets:
http://s.apache.org/Ylq
[2] 2.6.3 Tickets:
http://s.apache.org/9A7
________________________________
From: Junping Du
Sent: Tuesday, December 01, 2015 12:26 PM
To: Vinod Kumar Vavilapalli
Cc: Sangjin Lee; mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; Hadoop Common; hdfs-dev@hadoop.apache.org; Haohui Mai; Chris Trezzo; junping_du@apache.org
Subject: Re: continuing releases on Apache Hadoop 2.6.x
Thanks Vinod!
I went through all 31 fixes (Common: 8, HDFS: 6, MAPREDUCE: 8, YARN: 9) landed on branch-2.6 now, and all outstanding patches (>=5KB) are listed below and they are all bug fix.
Haven't found any high risky patches so far but I noticed we have 3 significant fixes (MAPREDUCE-5948, MAPREDUCE-6481 and MAPREDUCE-6549) for LineRecordReader. Among them, MAPREDUCE-5948 and MAPREDUCE-6481 are completed fixes and we will ship them in 2.7.2, so I think there is no risk with them. MAPREDUCE-6549, together with an open JIRA: MAPREDUCE-6558, sounds like fix the same issue for uncompressed/compressed input file. I already ping the JIRA owners to get suggestions on if we could release MAPREDUCE-6549 independently or together with MAPREDUCE-6558 in next release.
I already pinged all owners for open JIRAs that targeted for 2.6.3 to move it to 2.6.4. Based on the feedback from JIRAs (and voices here too), I will create branch-2.6.3 in next 1 day or 2. Thanks!
List of Outstanding Patches:
HADOOP-10134 [JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments. (apurtell via stevel) 38K
HDFS-9289. Make DataStreamer#block thread safe and verify genStamp in commitBlock. (Chang Li via zhz) 12K
HDFS-9470. Encryption zone on root not loaded from fsimage after NN restart. (Xiao Chen via wang) 5K
MAPREDUCE-6273. HistoryFileManager should check whether summaryFile exists to avoid FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state. (zhihai xu via devaraj) 5K
MAPREDUCE-5948. org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record delimiters well (Vinayakumar B, Rushabh Shah, and Akira AJISAKA via jlowe) 20K
MAPREDUCE-6481. LineRecordReader may give incomplete record and wrong position/key information for uncompressed input sometimes. (Zhihai Xu via jlowe) 19K
MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause duplicate records (wilfreds via rkanter) 31K
YARN-2859. ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster.(Vinod Kumar Vavilapalli via xgong) 6K (with addendum)
YARN-4344. NMs reconnecting with changed capabilities can lead to wrong cluster resource calculations (Varun Vasudev via jlowe) 12K
YARN-3925. ContainerLogsUtils#getContainerLogFile fails to read container log files from full disks. (zhihai xu via jlowe) 7K
Thanks,
Junping
________________________________
From: Vinod Kumar Vavilapalli <vi...@apache.org>
Sent: Monday, November 30, 2015 9:23 PM
To: Junping Du
Cc: Sangjin Lee; mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; Hadoop Common; hdfs-dev@hadoop.apache.org; Haohui Mai; Chris Trezzo
Subject: Re: continuing releases on Apache Hadoop 2.6.x
+1, this sounds good to me.
Thanks
+Vinod
On Nov 25, 2015, at 11:44 AM, Junping Du <jd...@hortonworks.com>> wrote:
May be we should quickly go through landed patches in 2.6.3 list and postpone any risky patches to 2.6.4?