You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Konstantin Shvachko <sh...@gmail.com> on 2017/06/15 23:42:56 UTC
Re: About 2.7.4 Release
Hey guys.
An update on 2.7.4 progress.
We are down to 4 blockers. There is some work remaining on those.
https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
Would be good if people could follow up on review comments.
I looked through nightly Jenkins build results for 2.7.4 both on Apache
Jenkins and internal.
Some test fail intermittently, but there no consistent failures. I filed
HDFS-11985 to track some of them.
https://issues.apache.org/jira/browse/HDFS-11985
I do not currently consider these failures as blockers. LMK if some of them
are.
We started internal testing of branch-2.7 on one of our smallish (100+
nodes) test clusters.
Will update on the results.
There is a plan to enable BigTop for 2.7.4 testing.
Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
Thank you everybody for contributing to this effort.
Regards,
--Konstantin
On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka <aa...@apache.org> wrote:
> Sure.
> If you want to edit the wiki, please tell me your ASF confluence account.
>
> -Akira
>
> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>
>> Couple of more JIRAs need to be back ported for 2.7.4 release. These will
>> solve RM HA unstability issues.
>> https://issues.apache.org/jira/browse/YARN-5333
>> https://issues.apache.org/jira/browse/YARN-5988
>> https://issues.apache.org/jira/browse/YARN-6304
>>
>> I will raise a JIRAs to back port it.
>>
>> @Akira , could you help to add these JIRAs into wiki?
>>
>> Thanks & Regards
>> Rohith Sharma K S
>>
>> On 29 May 2017 at 12:19, Akira Ajisaka <aa...@apache.org> wrote:
>>
>> Created a page for 2.7.4 release.
>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
>>>
>>> If you want to edit this wiki, please ping me.
>>>
>>> Regards,
>>> Akira
>>>
>>>
>>> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
>>>
>>> Hi Konstantin Shvachko
>>>>
>>>>
>>>> how about creating a wiki page for 2.7.4 release status like 2.8 and
>>>> trunk in following link.??
>>>>
>>>>
>>>> https://cwiki.apache.org/confluence/display/HADOOP
>>>>
>>>>
>>>> ________________________________
>>>> From: Konstantin Shvachko <sh...@gmail.com>
>>>> Sent: Saturday, May 13, 2017 3:58 AM
>>>> To: Akira Ajisaka
>>>> Cc: Hadoop Common; Hdfs-dev; mapreduce-dev@hadoop.apache.org;
>>>> yarn-dev@hadoop.apache.org
>>>> Subject: Re: About 2.7.4 Release
>>>>
>>>> Latest update on the links and filters. Here is the correct link for the
>>>> filter:
>>>> https://issues.apache.org/jira/secure/IssueNavigator.jspa?
>>>> requestId=12340814
>>>>
>>>> Also updated: https://s.apache.org/Dzg4
>>>>
>>>> Had to do some Jira debugging. Sorry for confusion.
>>>>
>>>> Thanks,
>>>> --Konstantin
>>>>
>>>> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <
>>>> shv.hadoop@gmail.com>
>>>> wrote:
>>>>
>>>> Hey Akira,
>>>>
>>>>>
>>>>> I didn't have private filters. Most probably Jira caches something.
>>>>> Your filter is in the right direction, but for some reason it lists
>>>>> only
>>>>> 22 issues, while mine has 29.
>>>>> It misses e.g. YARN-5543 <https://issues.apache.org/jir
>>>>> a/browse/YARN-5543>
>>>>> .
>>>>>
>>>>> Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
>>>>> shared it with "everybody", and updated my link to point to that
>>>>> filter.
>>>>> So
>>>>> you can use any of the three methods below to get the correct list:
>>>>> 1. Go to https://s.apache.org/Dzg4
>>>>> 2. Go to the filter via
>>>>> https://issues.apache.org/jira/issues?filter=12340814
>>>>> or by finding "Hadoop 2.7.4 release blockers" filter in the jira
>>>>> 3. On Advanced issues search page paste this:
>>>>> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels = release-blocker
>>>>> AND "Target Version/s" = 2.7.4
>>>>>
>>>>> Hope this solves the confusion for which issues are included.
>>>>> Please LMK if it doesn't, as it is important.
>>>>>
>>>>> Thanks,
>>>>> --Konstantin
>>>>>
>>>>> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka <aa...@apache.org>
>>>>> wrote:
>>>>>
>>>>> Hi Konstantin,
>>>>>
>>>>>>
>>>>>> Thank you for volunteering as release manager!
>>>>>>
>>>>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>>>>>
>>>>>>>
>>>>>>> I couldn't see the link. Maybe is it private filter?
>>>>>>
>>>>>> Here is a link I generated: https://s.apache.org/ehKy
>>>>>> This filter includes resolved issue and excludes fixversion == 2.7.4
>>>>>>
>>>>>> Thanks and Regards,
>>>>>> Akira
>>>>>>
>>>>>> On 2017/05/08 19:20, Konstantin Shvachko wrote:
>>>>>>
>>>>>> Hi Brahma Reddy Battula,
>>>>>>
>>>>>>>
>>>>>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>>>>>> Your link excludes closed and resolved issues, which needs
>>>>>>> backporting,
>>>>>>> and
>>>>>>> which we cannot reopen, as discussed in this thread earlier.
>>>>>>>
>>>>>>> Looked through the issues you proposed:
>>>>>>>
>>>>>>> HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
>>>>>>> Seems like a new feature. It helps failover to standby node when
>>>>>>> primary
>>>>>>> is
>>>>>>> under heavy load, but it introduces new APIs, addresses, config
>>>>>>> parameters.
>>>>>>> And needs at least one follow up jira.
>>>>>>> Looks like a backward compatible change, though.
>>>>>>> Did you have a chance to run it in production?
>>>>>>>
>>>>>>> +1 on
>>>>>>> HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987>
>>>>>>>
>>>>>>> [HDFS-10987] Make Decommission less expensive when lot of ...<
>>>>>>
>>>>> https://issues.apache.org/jira/browse/HDFS-10987>
>>>> issues.apache.org
>>>> When user want to decommission a node which having 50M blocks ,it could
>>>> hold the namesystem lock for long time.We've seen it is taking 36 sec.
>>>> As
>>>> we knew during this ...
>>>>
>>>>
>>>>
>>>> HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902>
>>>>
>>>>>
>>>>>>> [HDFS-9902] Support different values of dfs.datanode.du ...<
>>>>>>
>>>>> https://issues.apache.org/jira/browse/HDFS-9902>
>>>> issues.apache.org
>>>> Now Hadoop support different storage type for DISK, SSD, ARCHIVE and
>>>> RAM_DISK, but they share one configuration dfs.datanode.du.reserved. The
>>>> DISK size may be several ...
>>>>
>>>>
>>>>
>>>> HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312>
>>>>
>>>>>
>>>>>>> Trash does not descent into child directories to check for ...<
>>>>>>
>>>>> https://issues.apache.org/jira/browse/HDFS-8312>
>>>> issues.apache.org
>>>> HDFS trash does not descent into child directory to check if user has
>>>> permission to delete files. For example: Run the following command to
>>>> initialize directory ...
>>>>
>>>>
>>>>
>>>> HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100>
>>>>
>>>>>
>>>>>>> Upgrade Jsch jar to latest version to fix vulnerability in ...<
>>>>>>
>>>>> https://issues.apache.org/jira/browse/HADOOP-14100>
>>>> issues.apache.org
>>>> Recently there was on vulnerability reported on jsch library. Its fixed
>>>> in latest 0.1.54 version before CVE was made public.
>>>> https://cve.mitre.org/cgi-bin/cvename.cgi ...
>>>>
>>>>
>>>>
>>>>
>>>> Added them to 2.7.4 release. You should see them via the above link
>>>>>>> now.
>>>>>>> Would be good if you could attach backport patches for some of them?
>>>>>>>
>>>>>>> Appreciate your help,
>>>>>>> --Konstantin
>>>>>>>
>>>>>>> On Mon, May 8, 2017 at 8:39 AM, Brahma Reddy Battula <
>>>>>>> brahmareddy.battula@huawei.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>> Looks following link is not correct..
>>>>>>>
>>>>>>>>
>>>>>>>> https://s.apache.org/Dzg4
>>>>>>>>
>>>>>>>> It should be like following..?
>>>>>>>>
>>>>>>>> https://s.apache.org/wi3U
>>>>>>>>
>>>>>>>>
>>>>>>>> Apart from Konstantin mentioned,Following also good to go..? let me
>>>>>>>> know
>>>>>>>> your thoughts on this.
>>>>>>>>
>>>>>>>> For Large Cluster:
>>>>>>>> =============
>>>>>>>>
>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9311===Life Line
>>>>>>>> Protocol
>>>>>>>> https://issues.apache.org/jira/browse/HDFS-10987===Deecommission
>>>>>>>> Expensive when lot's of blocks are present
>>>>>>>>
>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9902===
>>>>>>>> "dfs.datanode.du.reserved" per Storage Type
>>>>>>>>
>>>>>>>> For Security:
>>>>>>>> =========
>>>>>>>> https://issues.apache.org/jira/browse/HDFS-8312===Trash does not
>>>>>>>> descent
>>>>>>>> into child directories to check for permission
>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-14100===Upgrade Jsch
>>>>>>>> jar
>>>>>>>> to
>>>>>>>> latest version to fix vulnerability in old versions
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Brahma Reddy Battula
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Erik Krogen [mailto:ekrogen@linkedin.com.INVALID]
>>>>>>>> Sent: 06 May 2017 02:40
>>>>>>>> To: Konstantin Shvachko
>>>>>>>> Cc: Zhe Zhang; Hadoop Common; Hdfs-dev;
>>>>>>>> mapreduce-dev@hadoop.apache.org
>>>>>>>> ;
>>>>>>>> yarn-dev@hadoop.apache.org
>>>>>>>> Subject: Re: About 2.7.4 Release
>>>>>>>>
>>>>>>>> List LGTM Konstantin!
>>>>>>>>
>>>>>>>> Let's say that we will only create a new tracking JIRA for patches
>>>>>>>> which
>>>>>>>> do not backport cleanly, to avoid having too many lying around.
>>>>>>>> Otherwise
>>>>>>>> we can directly attach to old ticket. If a clean backport does
>>>>>>>> happen
>>>>>>>> to
>>>>>>>> break a test the nightly build will help us catch it.
>>>>>>>>
>>>>>>>> Erik
>>>>>>>>
>>>>>>>> On Thu, May 4, 2017 at 7:21 PM, Konstantin Shvachko <
>>>>>>>> shv.hadoop@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Great Zhe. Let's monitor the build.
>>>>>>>>
>>>>>>>>
>>>>>>>>> I marked all jiras I knew of for inclusion into 2.7.4 as I
>>>>>>>>> described
>>>>>>>>> before.
>>>>>>>>> Target Version/s: 2.7.4
>>>>>>>>> Label: release-blocker
>>>>>>>>>
>>>>>>>>> Here is the link to the list: https://s.apache.org/Dzg4 Please let
>>>>>>>>> me
>>>>>>>>> know if I missed anything.
>>>>>>>>> And feel free to pick up any. Most of backports are pretty
>>>>>>>>> straightforward, but not all.
>>>>>>>>>
>>>>>>>>> We can create tracking jiras for backporting if you need to run
>>>>>>>>> Jenkins on the patch (and since Allen does not allow reopening
>>>>>>>>> them).
>>>>>>>>> But I think the final patch should be attached to the original
>>>>>>>>> jira.
>>>>>>>>> Otherwise history will be hard to follow.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> --Konstantin
>>>>>>>>>
>>>>>>>>> On Wed, May 3, 2017 at 4:53 PM, Zhe Zhang <zh...@apache.org> wrote:
>>>>>>>>>
>>>>>>>>> Thanks for volunteering as RM Konstantin! The plan LGTM.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> I've created a nightly Jenkins job for branch-2.7 (unit tests):
>>>>>>>>>> https://builds.apache.org/job/Hadoop-branch2.7-nightly/
>>>>>>>>>>
>>>>>>>>>> On Wed, May 3, 2017 at 12:42 AM Konstantin Shvachko <
>>>>>>>>>>
>>>>>>>>>> shv.hadoop@gmail.com>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hey guys,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> I and a few of my colleagues would like to help here and move
>>>>>>>>>>> 2.7.4
>>>>>>>>>>> release forward. A few points in this regard.
>>>>>>>>>>>
>>>>>>>>>>> 1. Reading through this thread since March 1 I see that Vinod
>>>>>>>>>>> hinted on managing the release. Vinod, if you still want the job
>>>>>>>>>>> /
>>>>>>>>>>> have bandwidth will be happy to work with you.
>>>>>>>>>>> Otherwise I am glad to volunteer as the release manager.
>>>>>>>>>>>
>>>>>>>>>>> 2. In addition to current blockers and criticals, I would like to
>>>>>>>>>>>
>>>>>>>>>>> propose
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> a
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> few issues to be included in the release, see the list below.
>>>>>>>>>>> Those
>>>>>>>>>>> are mostly bug fixes and optimizations, which we already have in
>>>>>>>>>>> our
>>>>>>>>>>>
>>>>>>>>>>> internal
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> branch and run in production. Plus one minor feature "node
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> labeling", which we found very handy, when you have heterogeneous
>>>>>>>>>>> environments and mixed workloads, like MR and Spark.
>>>>>>>>>>>
>>>>>>>>>>> 3. For marking issues for the release I propose to
>>>>>>>>>>> - set the target version to 2.7.4, and
>>>>>>>>>>> - add a new label "release-blocker"
>>>>>>>>>>> That way we will know issues targeted for the release without
>>>>>>>>>>> reopening them for backports.
>>>>>>>>>>>
>>>>>>>>>>> 4. I see quite a few people are interested in the release. With
>>>>>>>>>>> all
>>>>>>>>>>> the help I think we can target to release by the end of May.
>>>>>>>>>>>
>>>>>>>>>>> Other things include fixing CHANGES.txt and fixing Jenkins build
>>>>>>>>>>> for
>>>>>>>>>>>
>>>>>>>>>>> 2.7.4
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> branch.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> --Konstantin
>>>>>>>>>>>
>>>>>>>>>>> ========== List of issue for 2.7.4 ===========
>>>>>>>>>>> ------ Backports
>>>>>>>>>>> HADOOP-12975 <https://issues.apache.org/jira/browse/HADOOP-12975
>>>>>>>>>>> >.
>>>>>>>>>>> Add
>>>>>>>>>>>
>>>>>>>>>>> du
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> jitters
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> HDFS-9710 <https://issues.apache.org/jira/browse/HDFS-9710>. IBR
>>>>>>>>>>>
>>>>>>>>>>> batching
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> HDFS-10715 <https://issues.apache.org/jira/browse/HDFS-10715>. NPE
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> when applying AvailableSpaceBlockPlacementPolicy
>>>>>>>>>>> HDFS-2538 <https://issues.apache.org/jira/browse/HDFS-2538>.
>>>>>>>>>>> fsck
>>>>>>>>>>>
>>>>>>>>>>> removal
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> of dot printing
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> HDFS-8131 <https://issues.apache.org/jira/browse/HDFS-8131>.
>>>>>>>>>>> space-balanced
>>>>>>>>>>> policy for balancer
>>>>>>>>>>> HDFS-8549 <https://issues.apache.org/jira/browse/HDFS-8549>.
>>>>>>>>>>> abort
>>>>>>>>>>> balancer if upgrade in progress
>>>>>>>>>>> HDFS-9412 <https://issues.apache.org/jira/browse/HDFS-9412>.
>>>>>>>>>>> skip
>>>>>>>>>>> small blocks in getBlocks
>>>>>>>>>>>
>>>>>>>>>>> YARN-1471 <https://issues.apache.org/jira/browse/YARN-1471>. SLS
>>>>>>>>>>> simulator
>>>>>>>>>>> YARN-4302 <https://issues.apache.org/jira/browse/YARN-4302>. SLS
>>>>>>>>>>> YARN-4367 <https://issues.apache.org/jira/browse/YARN-4367>. SLS
>>>>>>>>>>> YARN-4612 <https://issues.apache.org/jira/browse/YARN-4612>. SLS
>>>>>>>>>>>
>>>>>>>>>>> ----- Node labeling
>>>>>>>>>>> MAPREDUCE-6304
>>>>>>>>>>> <https://issues.apache.org/jira/browse/MAPREDUCE-6304>
>>>>>>>>>>> YARN-2943 <https://issues.apache.org/jira/browse/YARN-2943>
>>>>>>>>>>> YARN-4109 <https://issues.apache.org/jira/browse/YARN-4109>
>>>>>>>>>>> YARN-4140 <https://issues.apache.org/jira/browse/YARN-4140>
>>>>>>>>>>> YARN-4250 <https://issues.apache.org/jira/browse/YARN-4250>
>>>>>>>>>>> YARN-4925 <https://issues.apache.org/jira/browse/YARN-4925>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>> Zhe Zhang
>>>>>>>>>> Apache Hadoop Committer
>>>>>>>>>> http://zhe-thoughts.github.io/about/ | @oldcap
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>
>>>
>>>
>>
Re: About 2.7.4 Release
Posted by Konstantin Shvachko <sh...@gmail.com>.
Hey guys,
Looks like we are done with blockers for Apache Hadoop 2.7.4 release.
https://issues.apache.org/jira/issues/?filter=12340814
I just committed HDFS-11896
<https://issues.apache.org/jira/browse/HDFS-11896>, and decided not to wait
for HDFS-11576 <https://issues.apache.org/jira/browse/HDFS-11576>, see Jira
comment.
Thanks Vinod for pointing out HDFS-11742
<https://issues.apache.org/jira/browse/HDFS-11742>. It was thoroughly
tested on a small cluster and Kihwal committed it last week, thanks.
Will start building initial RC. Please refrain from committing to
branch-2.7 for some time.
Thank you everybody for contributing.
--Konst
On Thu, Jul 20, 2017 at 12:07 PM, Vinod Kumar Vavilapalli <
vinodkv@apache.org> wrote:
> Thanks for taking 2.7.4 over Konstantin!
>
> Regarding rolling RC next week, I still see that there are 4 blocker /
> critical tickets targeted for 2.7.4: https://issues.apache.
> org/jira/issues/?jql=project%20in%20(HDFS%2C%20MAPREDUCE%
> 2C%20HADOOP%2C%20YARN)%20AND%20priority%20in%20(Blocker%2C%
> 20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20%
> 22Target%20Version%2Fs%22%20%3D%202.7.4
> <https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS,%20MAPREDUCE,%20HADOOP,%20YARN)%20AND%20priority%20in%20(Blocker,%20Critical)%20AND%20resolution%20=%20Unresolved%20AND%20%22Target%20Version/s%22%20=%202.7.4>
> .
>
> We should get closure on them. https://issues.apache.
> org/jira/browse/HDFS-11742 definitely was something that was deemed a
> blocker for 2.8.2, not sure about 2.7.4.
>
> I’m ‘back’ - let me know if you need any help.
>
> Thanks
> +Vinod
>
> On Jul 13, 2017, at 5:45 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
> Hi everybody.
>
> We have been doing some internal testing of Hadoop 2.7.4. The testing is
> going well.
> Did not find any major issues on our workloads.
> Used an internal tool called Dynamometer to check NameNode performance on
> real cluster traces. Good.
> Overall test cluster performance looks good.
> Some more testing is still going on.
>
> I plan to build an RC next week. If there are no objection.
>
> Thanks,
> --Konst
>
> On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <shv.hadoop@gmail.com
> >
> wrote:
>
> Hey guys.
>
> An update on 2.7.4 progress.
> We are down to 4 blockers. There is some work remaining on those.
> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
> Would be good if people could follow up on review comments.
>
> I looked through nightly Jenkins build results for 2.7.4 both on Apache
> Jenkins and internal.
> Some test fail intermittently, but there no consistent failures. I filed
> HDFS-11985 to track some of them.
> https://issues.apache.org/jira/browse/HDFS-11985
> I do not currently consider these failures as blockers. LMK if some of
> them are.
>
> We started internal testing of branch-2.7 on one of our smallish (100+
> nodes) test clusters.
> Will update on the results.
>
> There is a plan to enable BigTop for 2.7.4 testing.
>
> Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
> Thank you everybody for contributing to this effort.
>
> Regards,
> --Konstantin
>
>
> On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka <aa...@apache.org>
> wrote:
>
> Sure.
> If you want to edit the wiki, please tell me your ASF confluence account.
>
> -Akira
>
> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>
> Couple of more JIRAs need to be back ported for 2.7.4 release. These will
> solve RM HA unstability issues.
> https://issues.apache.org/jira/browse/YARN-5333
> https://issues.apache.org/jira/browse/YARN-5988
> https://issues.apache.org/jira/browse/YARN-6304
>
> I will raise a JIRAs to back port it.
>
> @Akira , could you help to add these JIRAs into wiki?
>
> Thanks & Regards
> Rohith Sharma K S
>
> On 29 May 2017 at 12:19, Akira Ajisaka <aa...@apache.org> wrote:
>
> Created a page for 2.7.4 release.
>
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
>
> If you want to edit this wiki, please ping me.
>
> Regards,
> Akira
>
>
> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
>
> Hi Konstantin Shvachko
>
>
>
> how about creating a wiki page for 2.7.4 release status like 2.8 and
> trunk in following link.??
>
>
> https://cwiki.apache.org/confluence/display/HADOOP
>
>
> ________________________________
> From: Konstantin Shvachko <sh...@gmail.com>
> Sent: Saturday, May 13, 2017 3:58 AM
> To: Akira Ajisaka
> Cc: Hadoop Common; Hdfs-dev; mapreduce-dev@hadoop.apache.org;
> yarn-dev@hadoop.apache.org
> Subject: Re: About 2.7.4 Release
>
> Latest update on the links and filters. Here is the correct link for
> the
> filter:
> https://issues.apache.org/jira/secure/IssueNavigator.jspa?
> requestId=12340814
>
> Also updated: https://s.apache.org/Dzg4
>
> Had to do some Jira debugging. Sorry for confusion.
>
> Thanks,
> --Konstantin
>
> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>
> wrote:
>
> Hey Akira,
>
>
> I didn't have private filters. Most probably Jira caches something.
> Your filter is in the right direction, but for some reason it lists
> only
> 22 issues, while mine has 29.
> It misses e.g. YARN-5543 <https://issues.apache.org/jir
> a/browse/YARN-5543>
> .
>
> Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
> shared it with "everybody", and updated my link to point to that
> filter.
> So
> you can use any of the three methods below to get the correct list:
> 1. Go to https://s.apache.org/Dzg4
> 2. Go to the filter via
> https://issues.apache.org/jira/issues?filter=12340814
> or by finding "Hadoop 2.7.4 release blockers" filter in the jira
> 3. On Advanced issues search page paste this:
> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels =
> release-blocker
> AND "Target Version/s" = 2.7.4
>
> Hope this solves the confusion for which issues are included.
> Please LMK if it doesn't, as it is important.
>
> Thanks,
> --Konstantin
>
> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka <aa...@apache.org>
> wrote:
>
> Hi Konstantin,
>
>
> Thank you for volunteering as release manager!
>
> Actually the original link works fine: https://s.apache.org/Dzg4
>
>
> I couldn't see the link. Maybe is it private filter?
>
>
> Here is a link I generated: https://s.apache.org/ehKy
> This filter includes resolved issue and excludes fixversion == 2.7.4
>
> Thanks and Regards,
> Akira
>
> On 2017/05/08 19:20, Konstantin Shvachko wrote:
>
> Hi Brahma Reddy Battula,
>
>
> Actually the original link works fine: https://s.apache.org/Dzg4
> Your link excludes closed and resolved issues, which needs
> backporting,
> and
> which we cannot reopen, as discussed in this thread earlier.
>
> Looked through the issues you proposed:
>
> HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
> Seems like a new feature. It helps failover to standby node when
> primary
> is
> under heavy load, but it introduces new APIs, addresses, config
> parameters.
> And needs at least one follow up jira.
> Looks like a backward compatible change, though.
> Did you have a chance to run it in production?
>
> +1 on
> HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987>
>
> [HDFS-10987] Make Decommission less expensive when lot of ...<
>
>
> https://issues.apache.org/jira/browse/HDFS-10987>
>
> issues.apache.org
> When user want to decommission a node which having 50M blocks ,it could
> hold the namesystem lock for long time.We've seen it is taking 36 sec.
> As
> we knew during this ...
>
>
>
> HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902>
>
>
> [HDFS-9902] Support different values of dfs.datanode.du ...<
>
>
> https://issues.apache.org/jira/browse/HDFS-9902>
>
> issues.apache.org
> Now Hadoop support different storage type for DISK, SSD, ARCHIVE and
> RAM_DISK, but they share one configuration dfs.datanode.du.reserved.
> The
> DISK size may be several ...
>
>
>
> HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312>
>
>
> Trash does not descent into child directories to check for ...<
>
>
> https://issues.apache.org/jira/browse/HDFS-8312>
>
> issues.apache.org
> HDFS trash does not descent into child directory to check if user has
> permission to delete files. For example: Run the following command to
> initialize directory ...
>
>
>
> HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100>
>
>
> Upgrade Jsch jar to latest version to fix vulnerability in ...<
>
>
> https://issues.apache.org/jira/browse/HADOOP-14100>
>
> issues.apache.org
> Recently there was on vulnerability reported on jsch library. Its fixed
> in latest 0.1.54 version before CVE was made public.
> https://cve.mitre.org/cgi-bin/cvename.cgi ...
>
>
>
>
> Added them to 2.7.4 release. You should see them via the above link
>
> now.
> Would be good if you could attach backport patches for some of them?
>
> Appreciate your help,
> --Konstantin
>
> On Mon, May 8, 2017 at 8:39 AM, Brahma Reddy Battula <
> brahmareddy.battula@huawei.com> wrote:
>
>
> Looks following link is not correct..
>
>
> https://s.apache.org/Dzg4
>
> It should be like following..?
>
> https://s.apache.org/wi3U
>
>
> Apart from Konstantin mentioned,Following also good to go..? let me
> know
> your thoughts on this.
>
> For Large Cluster:
> =============
>
> https://issues.apache.org/jira/browse/HDFS-9311===Life Line
> Protocol
> https://issues.apache.org/jira/browse/HDFS-10987===Deecommission
> Expensive when lot's of blocks are present
>
> https://issues.apache.org/jira/browse/HDFS-9902===
> "dfs.datanode.du.reserved" per Storage Type
>
> For Security:
> =========
> https://issues.apache.org/jira/browse/HDFS-8312===Trash does not
> descent
> into child directories to check for permission
> https://issues.apache.org/jira/browse/HADOOP-14100===Upgrade Jsch
> jar
> to
> latest version to fix vulnerability in old versions
>
>
>
> Regards
> Brahma Reddy Battula
>
> -----Original Message-----
> From: Erik Krogen [mailto:ekrogen@linkedin.com.INVALID]
> Sent: 06 May 2017 02:40
> To: Konstantin Shvachko
> Cc: Zhe Zhang; Hadoop Common; Hdfs-dev;
> mapreduce-dev@hadoop.apache.org
> ;
> yarn-dev@hadoop.apache.org
> Subject: Re: About 2.7.4 Release
>
> List LGTM Konstantin!
>
> Let's say that we will only create a new tracking JIRA for patches
> which
> do not backport cleanly, to avoid having too many lying around.
> Otherwise
> we can directly attach to old ticket. If a clean backport does
> happen
> to
> break a test the nightly build will help us catch it.
>
> Erik
>
> On Thu, May 4, 2017 at 7:21 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>
> wrote:
>
> Great Zhe. Let's monitor the build.
>
>
> I marked all jiras I knew of for inclusion into 2.7.4 as I
> described
> before.
> Target Version/s: 2.7.4
> Label: release-blocker
>
> Here is the link to the list: https://s.apache.org/Dzg4 Please
> let
> me
> know if I missed anything.
> And feel free to pick up any. Most of backports are pretty
> straightforward, but not all.
>
> We can create tracking jiras for backporting if you need to run
> Jenkins on the patch (and since Allen does not allow reopening
> them).
> But I think the final patch should be attached to the original
> jira.
> Otherwise history will be hard to follow.
>
> Thanks,
> --Konstantin
>
> On Wed, May 3, 2017 at 4:53 PM, Zhe Zhang <zh...@apache.org> wrote:
>
> Thanks for volunteering as RM Konstantin! The plan LGTM.
>
>
> I've created a nightly Jenkins job for branch-2.7 (unit tests):
> https://builds.apache.org/job/Hadoop-branch2.7-nightly/
>
> On Wed, May 3, 2017 at 12:42 AM Konstantin Shvachko <
>
> shv.hadoop@gmail.com>
>
>
> wrote:
>
>
> Hey guys,
>
>
> I and a few of my colleagues would like to help here and move
> 2.7.4
> release forward. A few points in this regard.
>
> 1. Reading through this thread since March 1 I see that Vinod
> hinted on managing the release. Vinod, if you still want the
> job /
> have bandwidth will be happy to work with you.
> Otherwise I am glad to volunteer as the release manager.
>
> 2. In addition to current blockers and criticals, I would like
> to
>
> propose
>
>
>
> a
>
>
> few issues to be included in the release, see the list below.
>
> Those
> are mostly bug fixes and optimizations, which we already have in
> our
>
> internal
>
>
>
> branch and run in production. Plus one minor feature "node
>
>
> labeling", which we found very handy, when you have heterogeneous
>
> environments and mixed workloads, like MR and Spark.
>
> 3. For marking issues for the release I propose to
> - set the target version to 2.7.4, and
> - add a new label "release-blocker"
> That way we will know issues targeted for the release without
> reopening them for backports.
>
> 4. I see quite a few people are interested in the release. With
> all
> the help I think we can target to release by the end of May.
>
> Other things include fixing CHANGES.txt and fixing Jenkins build
> for
>
> 2.7.4
>
>
>
> branch.
>
>
>
> Thanks,
> --Konstantin
>
> ========== List of issue for 2.7.4 ===========
> ------ Backports
> HADOOP-12975 <https://issues.apache.org/jir
> a/browse/HADOOP-12975>.
> Add
>
> du
>
>
>
> jitters
>
>
> HDFS-9710 <https://issues.apache.org/jira/browse/HDFS-9710>. IBR
>
>
> batching
>
>
>
> HDFS-10715 <https://issues.apache.org/jira/browse/HDFS-10715>.
> NPE
>
>
> when applying AvailableSpaceBlockPlacementPolicy
>
> HDFS-2538 <https://issues.apache.org/jira/browse/HDFS-2538>.
> fsck
>
> removal
>
>
>
> of dot printing
>
>
> HDFS-8131 <https://issues.apache.org/jira/browse/HDFS-8131>.
>
> space-balanced
> policy for balancer
> HDFS-8549 <https://issues.apache.org/jira/browse/HDFS-8549>.
> abort
> balancer if upgrade in progress
> HDFS-9412 <https://issues.apache.org/jira/browse/HDFS-9412>.
> skip
> small blocks in getBlocks
>
> YARN-1471 <https://issues.apache.org/jira/browse/YARN-1471>.
> SLS
> simulator
> YARN-4302 <https://issues.apache.org/jira/browse/YARN-4302>.
> SLS
> YARN-4367 <https://issues.apache.org/jira/browse/YARN-4367>.
> SLS
> YARN-4612 <https://issues.apache.org/jira/browse/YARN-4612>.
> SLS
>
> ----- Node labeling
> MAPREDUCE-6304
> <https://issues.apache.org/jira/browse/MAPREDUCE-6304>
> YARN-2943 <https://issues.apache.org/jira/browse/YARN-2943>
> YARN-4109 <https://issues.apache.org/jira/browse/YARN-4109>
> YARN-4140 <https://issues.apache.org/jira/browse/YARN-4140>
> YARN-4250 <https://issues.apache.org/jira/browse/YARN-4250>
> YARN-4925 <https://issues.apache.org/jira/browse/YARN-4925>
>
> --
>
> Zhe Zhang
>
> Apache Hadoop Committer
> http://zhe-thoughts.github.io/about/ | @oldcap
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
>
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>
>
>
>
>
>
Re: About 2.7.4 Release
Posted by Konstantin Shvachko <sh...@gmail.com>.
Hey guys,
Looks like we are done with blockers for Apache Hadoop 2.7.4 release.
https://issues.apache.org/jira/issues/?filter=12340814
I just committed HDFS-11896
<https://issues.apache.org/jira/browse/HDFS-11896>, and decided not to wait
for HDFS-11576 <https://issues.apache.org/jira/browse/HDFS-11576>, see Jira
comment.
Thanks Vinod for pointing out HDFS-11742
<https://issues.apache.org/jira/browse/HDFS-11742>. It was thoroughly
tested on a small cluster and Kihwal committed it last week, thanks.
Will start building initial RC. Please refrain from committing to
branch-2.7 for some time.
Thank you everybody for contributing.
--Konst
On Thu, Jul 20, 2017 at 12:07 PM, Vinod Kumar Vavilapalli <
vinodkv@apache.org> wrote:
> Thanks for taking 2.7.4 over Konstantin!
>
> Regarding rolling RC next week, I still see that there are 4 blocker /
> critical tickets targeted for 2.7.4: https://issues.apache.
> org/jira/issues/?jql=project%20in%20(HDFS%2C%20MAPREDUCE%
> 2C%20HADOOP%2C%20YARN)%20AND%20priority%20in%20(Blocker%2C%
> 20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20%
> 22Target%20Version%2Fs%22%20%3D%202.7.4
> <https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS,%20MAPREDUCE,%20HADOOP,%20YARN)%20AND%20priority%20in%20(Blocker,%20Critical)%20AND%20resolution%20=%20Unresolved%20AND%20%22Target%20Version/s%22%20=%202.7.4>
> .
>
> We should get closure on them. https://issues.apache.
> org/jira/browse/HDFS-11742 definitely was something that was deemed a
> blocker for 2.8.2, not sure about 2.7.4.
>
> I’m ‘back’ - let me know if you need any help.
>
> Thanks
> +Vinod
>
> On Jul 13, 2017, at 5:45 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
> Hi everybody.
>
> We have been doing some internal testing of Hadoop 2.7.4. The testing is
> going well.
> Did not find any major issues on our workloads.
> Used an internal tool called Dynamometer to check NameNode performance on
> real cluster traces. Good.
> Overall test cluster performance looks good.
> Some more testing is still going on.
>
> I plan to build an RC next week. If there are no objection.
>
> Thanks,
> --Konst
>
> On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <shv.hadoop@gmail.com
> >
> wrote:
>
> Hey guys.
>
> An update on 2.7.4 progress.
> We are down to 4 blockers. There is some work remaining on those.
> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
> Would be good if people could follow up on review comments.
>
> I looked through nightly Jenkins build results for 2.7.4 both on Apache
> Jenkins and internal.
> Some test fail intermittently, but there no consistent failures. I filed
> HDFS-11985 to track some of them.
> https://issues.apache.org/jira/browse/HDFS-11985
> I do not currently consider these failures as blockers. LMK if some of
> them are.
>
> We started internal testing of branch-2.7 on one of our smallish (100+
> nodes) test clusters.
> Will update on the results.
>
> There is a plan to enable BigTop for 2.7.4 testing.
>
> Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
> Thank you everybody for contributing to this effort.
>
> Regards,
> --Konstantin
>
>
> On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka <aa...@apache.org>
> wrote:
>
> Sure.
> If you want to edit the wiki, please tell me your ASF confluence account.
>
> -Akira
>
> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>
> Couple of more JIRAs need to be back ported for 2.7.4 release. These will
> solve RM HA unstability issues.
> https://issues.apache.org/jira/browse/YARN-5333
> https://issues.apache.org/jira/browse/YARN-5988
> https://issues.apache.org/jira/browse/YARN-6304
>
> I will raise a JIRAs to back port it.
>
> @Akira , could you help to add these JIRAs into wiki?
>
> Thanks & Regards
> Rohith Sharma K S
>
> On 29 May 2017 at 12:19, Akira Ajisaka <aa...@apache.org> wrote:
>
> Created a page for 2.7.4 release.
>
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
>
> If you want to edit this wiki, please ping me.
>
> Regards,
> Akira
>
>
> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
>
> Hi Konstantin Shvachko
>
>
>
> how about creating a wiki page for 2.7.4 release status like 2.8 and
> trunk in following link.??
>
>
> https://cwiki.apache.org/confluence/display/HADOOP
>
>
> ________________________________
> From: Konstantin Shvachko <sh...@gmail.com>
> Sent: Saturday, May 13, 2017 3:58 AM
> To: Akira Ajisaka
> Cc: Hadoop Common; Hdfs-dev; mapreduce-dev@hadoop.apache.org;
> yarn-dev@hadoop.apache.org
> Subject: Re: About 2.7.4 Release
>
> Latest update on the links and filters. Here is the correct link for
> the
> filter:
> https://issues.apache.org/jira/secure/IssueNavigator.jspa?
> requestId=12340814
>
> Also updated: https://s.apache.org/Dzg4
>
> Had to do some Jira debugging. Sorry for confusion.
>
> Thanks,
> --Konstantin
>
> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>
> wrote:
>
> Hey Akira,
>
>
> I didn't have private filters. Most probably Jira caches something.
> Your filter is in the right direction, but for some reason it lists
> only
> 22 issues, while mine has 29.
> It misses e.g. YARN-5543 <https://issues.apache.org/jir
> a/browse/YARN-5543>
> .
>
> Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
> shared it with "everybody", and updated my link to point to that
> filter.
> So
> you can use any of the three methods below to get the correct list:
> 1. Go to https://s.apache.org/Dzg4
> 2. Go to the filter via
> https://issues.apache.org/jira/issues?filter=12340814
> or by finding "Hadoop 2.7.4 release blockers" filter in the jira
> 3. On Advanced issues search page paste this:
> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels =
> release-blocker
> AND "Target Version/s" = 2.7.4
>
> Hope this solves the confusion for which issues are included.
> Please LMK if it doesn't, as it is important.
>
> Thanks,
> --Konstantin
>
> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka <aa...@apache.org>
> wrote:
>
> Hi Konstantin,
>
>
> Thank you for volunteering as release manager!
>
> Actually the original link works fine: https://s.apache.org/Dzg4
>
>
> I couldn't see the link. Maybe is it private filter?
>
>
> Here is a link I generated: https://s.apache.org/ehKy
> This filter includes resolved issue and excludes fixversion == 2.7.4
>
> Thanks and Regards,
> Akira
>
> On 2017/05/08 19:20, Konstantin Shvachko wrote:
>
> Hi Brahma Reddy Battula,
>
>
> Actually the original link works fine: https://s.apache.org/Dzg4
> Your link excludes closed and resolved issues, which needs
> backporting,
> and
> which we cannot reopen, as discussed in this thread earlier.
>
> Looked through the issues you proposed:
>
> HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
> Seems like a new feature. It helps failover to standby node when
> primary
> is
> under heavy load, but it introduces new APIs, addresses, config
> parameters.
> And needs at least one follow up jira.
> Looks like a backward compatible change, though.
> Did you have a chance to run it in production?
>
> +1 on
> HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987>
>
> [HDFS-10987] Make Decommission less expensive when lot of ...<
>
>
> https://issues.apache.org/jira/browse/HDFS-10987>
>
> issues.apache.org
> When user want to decommission a node which having 50M blocks ,it could
> hold the namesystem lock for long time.We've seen it is taking 36 sec.
> As
> we knew during this ...
>
>
>
> HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902>
>
>
> [HDFS-9902] Support different values of dfs.datanode.du ...<
>
>
> https://issues.apache.org/jira/browse/HDFS-9902>
>
> issues.apache.org
> Now Hadoop support different storage type for DISK, SSD, ARCHIVE and
> RAM_DISK, but they share one configuration dfs.datanode.du.reserved.
> The
> DISK size may be several ...
>
>
>
> HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312>
>
>
> Trash does not descent into child directories to check for ...<
>
>
> https://issues.apache.org/jira/browse/HDFS-8312>
>
> issues.apache.org
> HDFS trash does not descent into child directory to check if user has
> permission to delete files. For example: Run the following command to
> initialize directory ...
>
>
>
> HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100>
>
>
> Upgrade Jsch jar to latest version to fix vulnerability in ...<
>
>
> https://issues.apache.org/jira/browse/HADOOP-14100>
>
> issues.apache.org
> Recently there was on vulnerability reported on jsch library. Its fixed
> in latest 0.1.54 version before CVE was made public.
> https://cve.mitre.org/cgi-bin/cvename.cgi ...
>
>
>
>
> Added them to 2.7.4 release. You should see them via the above link
>
> now.
> Would be good if you could attach backport patches for some of them?
>
> Appreciate your help,
> --Konstantin
>
> On Mon, May 8, 2017 at 8:39 AM, Brahma Reddy Battula <
> brahmareddy.battula@huawei.com> wrote:
>
>
> Looks following link is not correct..
>
>
> https://s.apache.org/Dzg4
>
> It should be like following..?
>
> https://s.apache.org/wi3U
>
>
> Apart from Konstantin mentioned,Following also good to go..? let me
> know
> your thoughts on this.
>
> For Large Cluster:
> =============
>
> https://issues.apache.org/jira/browse/HDFS-9311===Life Line
> Protocol
> https://issues.apache.org/jira/browse/HDFS-10987===Deecommission
> Expensive when lot's of blocks are present
>
> https://issues.apache.org/jira/browse/HDFS-9902===
> "dfs.datanode.du.reserved" per Storage Type
>
> For Security:
> =========
> https://issues.apache.org/jira/browse/HDFS-8312===Trash does not
> descent
> into child directories to check for permission
> https://issues.apache.org/jira/browse/HADOOP-14100===Upgrade Jsch
> jar
> to
> latest version to fix vulnerability in old versions
>
>
>
> Regards
> Brahma Reddy Battula
>
> -----Original Message-----
> From: Erik Krogen [mailto:ekrogen@linkedin.com.INVALID]
> Sent: 06 May 2017 02:40
> To: Konstantin Shvachko
> Cc: Zhe Zhang; Hadoop Common; Hdfs-dev;
> mapreduce-dev@hadoop.apache.org
> ;
> yarn-dev@hadoop.apache.org
> Subject: Re: About 2.7.4 Release
>
> List LGTM Konstantin!
>
> Let's say that we will only create a new tracking JIRA for patches
> which
> do not backport cleanly, to avoid having too many lying around.
> Otherwise
> we can directly attach to old ticket. If a clean backport does
> happen
> to
> break a test the nightly build will help us catch it.
>
> Erik
>
> On Thu, May 4, 2017 at 7:21 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>
> wrote:
>
> Great Zhe. Let's monitor the build.
>
>
> I marked all jiras I knew of for inclusion into 2.7.4 as I
> described
> before.
> Target Version/s: 2.7.4
> Label: release-blocker
>
> Here is the link to the list: https://s.apache.org/Dzg4 Please
> let
> me
> know if I missed anything.
> And feel free to pick up any. Most of backports are pretty
> straightforward, but not all.
>
> We can create tracking jiras for backporting if you need to run
> Jenkins on the patch (and since Allen does not allow reopening
> them).
> But I think the final patch should be attached to the original
> jira.
> Otherwise history will be hard to follow.
>
> Thanks,
> --Konstantin
>
> On Wed, May 3, 2017 at 4:53 PM, Zhe Zhang <zh...@apache.org> wrote:
>
> Thanks for volunteering as RM Konstantin! The plan LGTM.
>
>
> I've created a nightly Jenkins job for branch-2.7 (unit tests):
> https://builds.apache.org/job/Hadoop-branch2.7-nightly/
>
> On Wed, May 3, 2017 at 12:42 AM Konstantin Shvachko <
>
> shv.hadoop@gmail.com>
>
>
> wrote:
>
>
> Hey guys,
>
>
> I and a few of my colleagues would like to help here and move
> 2.7.4
> release forward. A few points in this regard.
>
> 1. Reading through this thread since March 1 I see that Vinod
> hinted on managing the release. Vinod, if you still want the
> job /
> have bandwidth will be happy to work with you.
> Otherwise I am glad to volunteer as the release manager.
>
> 2. In addition to current blockers and criticals, I would like
> to
>
> propose
>
>
>
> a
>
>
> few issues to be included in the release, see the list below.
>
> Those
> are mostly bug fixes and optimizations, which we already have in
> our
>
> internal
>
>
>
> branch and run in production. Plus one minor feature "node
>
>
> labeling", which we found very handy, when you have heterogeneous
>
> environments and mixed workloads, like MR and Spark.
>
> 3. For marking issues for the release I propose to
> - set the target version to 2.7.4, and
> - add a new label "release-blocker"
> That way we will know issues targeted for the release without
> reopening them for backports.
>
> 4. I see quite a few people are interested in the release. With
> all
> the help I think we can target to release by the end of May.
>
> Other things include fixing CHANGES.txt and fixing Jenkins build
> for
>
> 2.7.4
>
>
>
> branch.
>
>
>
> Thanks,
> --Konstantin
>
> ========== List of issue for 2.7.4 ===========
> ------ Backports
> HADOOP-12975 <https://issues.apache.org/jir
> a/browse/HADOOP-12975>.
> Add
>
> du
>
>
>
> jitters
>
>
> HDFS-9710 <https://issues.apache.org/jira/browse/HDFS-9710>. IBR
>
>
> batching
>
>
>
> HDFS-10715 <https://issues.apache.org/jira/browse/HDFS-10715>.
> NPE
>
>
> when applying AvailableSpaceBlockPlacementPolicy
>
> HDFS-2538 <https://issues.apache.org/jira/browse/HDFS-2538>.
> fsck
>
> removal
>
>
>
> of dot printing
>
>
> HDFS-8131 <https://issues.apache.org/jira/browse/HDFS-8131>.
>
> space-balanced
> policy for balancer
> HDFS-8549 <https://issues.apache.org/jira/browse/HDFS-8549>.
> abort
> balancer if upgrade in progress
> HDFS-9412 <https://issues.apache.org/jira/browse/HDFS-9412>.
> skip
> small blocks in getBlocks
>
> YARN-1471 <https://issues.apache.org/jira/browse/YARN-1471>.
> SLS
> simulator
> YARN-4302 <https://issues.apache.org/jira/browse/YARN-4302>.
> SLS
> YARN-4367 <https://issues.apache.org/jira/browse/YARN-4367>.
> SLS
> YARN-4612 <https://issues.apache.org/jira/browse/YARN-4612>.
> SLS
>
> ----- Node labeling
> MAPREDUCE-6304
> <https://issues.apache.org/jira/browse/MAPREDUCE-6304>
> YARN-2943 <https://issues.apache.org/jira/browse/YARN-2943>
> YARN-4109 <https://issues.apache.org/jira/browse/YARN-4109>
> YARN-4140 <https://issues.apache.org/jira/browse/YARN-4140>
> YARN-4250 <https://issues.apache.org/jira/browse/YARN-4250>
> YARN-4925 <https://issues.apache.org/jira/browse/YARN-4925>
>
> --
>
> Zhe Zhang
>
> Apache Hadoop Committer
> http://zhe-thoughts.github.io/about/ | @oldcap
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
>
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>
>
>
>
>
>
Re: About 2.7.4 Release
Posted by Konstantin Shvachko <sh...@gmail.com>.
Hey guys,
Looks like we are done with blockers for Apache Hadoop 2.7.4 release.
https://issues.apache.org/jira/issues/?filter=12340814
I just committed HDFS-11896
<https://issues.apache.org/jira/browse/HDFS-11896>, and decided not to wait
for HDFS-11576 <https://issues.apache.org/jira/browse/HDFS-11576>, see Jira
comment.
Thanks Vinod for pointing out HDFS-11742
<https://issues.apache.org/jira/browse/HDFS-11742>. It was thoroughly
tested on a small cluster and Kihwal committed it last week, thanks.
Will start building initial RC. Please refrain from committing to
branch-2.7 for some time.
Thank you everybody for contributing.
--Konst
On Thu, Jul 20, 2017 at 12:07 PM, Vinod Kumar Vavilapalli <
vinodkv@apache.org> wrote:
> Thanks for taking 2.7.4 over Konstantin!
>
> Regarding rolling RC next week, I still see that there are 4 blocker /
> critical tickets targeted for 2.7.4: https://issues.apache.
> org/jira/issues/?jql=project%20in%20(HDFS%2C%20MAPREDUCE%
> 2C%20HADOOP%2C%20YARN)%20AND%20priority%20in%20(Blocker%2C%
> 20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20%
> 22Target%20Version%2Fs%22%20%3D%202.7.4
> <https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS,%20MAPREDUCE,%20HADOOP,%20YARN)%20AND%20priority%20in%20(Blocker,%20Critical)%20AND%20resolution%20=%20Unresolved%20AND%20%22Target%20Version/s%22%20=%202.7.4>
> .
>
> We should get closure on them. https://issues.apache.
> org/jira/browse/HDFS-11742 definitely was something that was deemed a
> blocker for 2.8.2, not sure about 2.7.4.
>
> I’m ‘back’ - let me know if you need any help.
>
> Thanks
> +Vinod
>
> On Jul 13, 2017, at 5:45 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
> Hi everybody.
>
> We have been doing some internal testing of Hadoop 2.7.4. The testing is
> going well.
> Did not find any major issues on our workloads.
> Used an internal tool called Dynamometer to check NameNode performance on
> real cluster traces. Good.
> Overall test cluster performance looks good.
> Some more testing is still going on.
>
> I plan to build an RC next week. If there are no objection.
>
> Thanks,
> --Konst
>
> On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <shv.hadoop@gmail.com
> >
> wrote:
>
> Hey guys.
>
> An update on 2.7.4 progress.
> We are down to 4 blockers. There is some work remaining on those.
> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
> Would be good if people could follow up on review comments.
>
> I looked through nightly Jenkins build results for 2.7.4 both on Apache
> Jenkins and internal.
> Some test fail intermittently, but there no consistent failures. I filed
> HDFS-11985 to track some of them.
> https://issues.apache.org/jira/browse/HDFS-11985
> I do not currently consider these failures as blockers. LMK if some of
> them are.
>
> We started internal testing of branch-2.7 on one of our smallish (100+
> nodes) test clusters.
> Will update on the results.
>
> There is a plan to enable BigTop for 2.7.4 testing.
>
> Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
> Thank you everybody for contributing to this effort.
>
> Regards,
> --Konstantin
>
>
> On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka <aa...@apache.org>
> wrote:
>
> Sure.
> If you want to edit the wiki, please tell me your ASF confluence account.
>
> -Akira
>
> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>
> Couple of more JIRAs need to be back ported for 2.7.4 release. These will
> solve RM HA unstability issues.
> https://issues.apache.org/jira/browse/YARN-5333
> https://issues.apache.org/jira/browse/YARN-5988
> https://issues.apache.org/jira/browse/YARN-6304
>
> I will raise a JIRAs to back port it.
>
> @Akira , could you help to add these JIRAs into wiki?
>
> Thanks & Regards
> Rohith Sharma K S
>
> On 29 May 2017 at 12:19, Akira Ajisaka <aa...@apache.org> wrote:
>
> Created a page for 2.7.4 release.
>
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
>
> If you want to edit this wiki, please ping me.
>
> Regards,
> Akira
>
>
> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
>
> Hi Konstantin Shvachko
>
>
>
> how about creating a wiki page for 2.7.4 release status like 2.8 and
> trunk in following link.??
>
>
> https://cwiki.apache.org/confluence/display/HADOOP
>
>
> ________________________________
> From: Konstantin Shvachko <sh...@gmail.com>
> Sent: Saturday, May 13, 2017 3:58 AM
> To: Akira Ajisaka
> Cc: Hadoop Common; Hdfs-dev; mapreduce-dev@hadoop.apache.org;
> yarn-dev@hadoop.apache.org
> Subject: Re: About 2.7.4 Release
>
> Latest update on the links and filters. Here is the correct link for
> the
> filter:
> https://issues.apache.org/jira/secure/IssueNavigator.jspa?
> requestId=12340814
>
> Also updated: https://s.apache.org/Dzg4
>
> Had to do some Jira debugging. Sorry for confusion.
>
> Thanks,
> --Konstantin
>
> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>
> wrote:
>
> Hey Akira,
>
>
> I didn't have private filters. Most probably Jira caches something.
> Your filter is in the right direction, but for some reason it lists
> only
> 22 issues, while mine has 29.
> It misses e.g. YARN-5543 <https://issues.apache.org/jir
> a/browse/YARN-5543>
> .
>
> Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
> shared it with "everybody", and updated my link to point to that
> filter.
> So
> you can use any of the three methods below to get the correct list:
> 1. Go to https://s.apache.org/Dzg4
> 2. Go to the filter via
> https://issues.apache.org/jira/issues?filter=12340814
> or by finding "Hadoop 2.7.4 release blockers" filter in the jira
> 3. On Advanced issues search page paste this:
> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels =
> release-blocker
> AND "Target Version/s" = 2.7.4
>
> Hope this solves the confusion for which issues are included.
> Please LMK if it doesn't, as it is important.
>
> Thanks,
> --Konstantin
>
> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka <aa...@apache.org>
> wrote:
>
> Hi Konstantin,
>
>
> Thank you for volunteering as release manager!
>
> Actually the original link works fine: https://s.apache.org/Dzg4
>
>
> I couldn't see the link. Maybe is it private filter?
>
>
> Here is a link I generated: https://s.apache.org/ehKy
> This filter includes resolved issue and excludes fixversion == 2.7.4
>
> Thanks and Regards,
> Akira
>
> On 2017/05/08 19:20, Konstantin Shvachko wrote:
>
> Hi Brahma Reddy Battula,
>
>
> Actually the original link works fine: https://s.apache.org/Dzg4
> Your link excludes closed and resolved issues, which needs
> backporting,
> and
> which we cannot reopen, as discussed in this thread earlier.
>
> Looked through the issues you proposed:
>
> HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
> Seems like a new feature. It helps failover to standby node when
> primary
> is
> under heavy load, but it introduces new APIs, addresses, config
> parameters.
> And needs at least one follow up jira.
> Looks like a backward compatible change, though.
> Did you have a chance to run it in production?
>
> +1 on
> HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987>
>
> [HDFS-10987] Make Decommission less expensive when lot of ...<
>
>
> https://issues.apache.org/jira/browse/HDFS-10987>
>
> issues.apache.org
> When user want to decommission a node which having 50M blocks ,it could
> hold the namesystem lock for long time.We've seen it is taking 36 sec.
> As
> we knew during this ...
>
>
>
> HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902>
>
>
> [HDFS-9902] Support different values of dfs.datanode.du ...<
>
>
> https://issues.apache.org/jira/browse/HDFS-9902>
>
> issues.apache.org
> Now Hadoop support different storage type for DISK, SSD, ARCHIVE and
> RAM_DISK, but they share one configuration dfs.datanode.du.reserved.
> The
> DISK size may be several ...
>
>
>
> HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312>
>
>
> Trash does not descent into child directories to check for ...<
>
>
> https://issues.apache.org/jira/browse/HDFS-8312>
>
> issues.apache.org
> HDFS trash does not descent into child directory to check if user has
> permission to delete files. For example: Run the following command to
> initialize directory ...
>
>
>
> HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100>
>
>
> Upgrade Jsch jar to latest version to fix vulnerability in ...<
>
>
> https://issues.apache.org/jira/browse/HADOOP-14100>
>
> issues.apache.org
> Recently there was on vulnerability reported on jsch library. Its fixed
> in latest 0.1.54 version before CVE was made public.
> https://cve.mitre.org/cgi-bin/cvename.cgi ...
>
>
>
>
> Added them to 2.7.4 release. You should see them via the above link
>
> now.
> Would be good if you could attach backport patches for some of them?
>
> Appreciate your help,
> --Konstantin
>
> On Mon, May 8, 2017 at 8:39 AM, Brahma Reddy Battula <
> brahmareddy.battula@huawei.com> wrote:
>
>
> Looks following link is not correct..
>
>
> https://s.apache.org/Dzg4
>
> It should be like following..?
>
> https://s.apache.org/wi3U
>
>
> Apart from Konstantin mentioned,Following also good to go..? let me
> know
> your thoughts on this.
>
> For Large Cluster:
> =============
>
> https://issues.apache.org/jira/browse/HDFS-9311===Life Line
> Protocol
> https://issues.apache.org/jira/browse/HDFS-10987===Deecommission
> Expensive when lot's of blocks are present
>
> https://issues.apache.org/jira/browse/HDFS-9902===
> "dfs.datanode.du.reserved" per Storage Type
>
> For Security:
> =========
> https://issues.apache.org/jira/browse/HDFS-8312===Trash does not
> descent
> into child directories to check for permission
> https://issues.apache.org/jira/browse/HADOOP-14100===Upgrade Jsch
> jar
> to
> latest version to fix vulnerability in old versions
>
>
>
> Regards
> Brahma Reddy Battula
>
> -----Original Message-----
> From: Erik Krogen [mailto:ekrogen@linkedin.com.INVALID]
> Sent: 06 May 2017 02:40
> To: Konstantin Shvachko
> Cc: Zhe Zhang; Hadoop Common; Hdfs-dev;
> mapreduce-dev@hadoop.apache.org
> ;
> yarn-dev@hadoop.apache.org
> Subject: Re: About 2.7.4 Release
>
> List LGTM Konstantin!
>
> Let's say that we will only create a new tracking JIRA for patches
> which
> do not backport cleanly, to avoid having too many lying around.
> Otherwise
> we can directly attach to old ticket. If a clean backport does
> happen
> to
> break a test the nightly build will help us catch it.
>
> Erik
>
> On Thu, May 4, 2017 at 7:21 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>
> wrote:
>
> Great Zhe. Let's monitor the build.
>
>
> I marked all jiras I knew of for inclusion into 2.7.4 as I
> described
> before.
> Target Version/s: 2.7.4
> Label: release-blocker
>
> Here is the link to the list: https://s.apache.org/Dzg4 Please
> let
> me
> know if I missed anything.
> And feel free to pick up any. Most of backports are pretty
> straightforward, but not all.
>
> We can create tracking jiras for backporting if you need to run
> Jenkins on the patch (and since Allen does not allow reopening
> them).
> But I think the final patch should be attached to the original
> jira.
> Otherwise history will be hard to follow.
>
> Thanks,
> --Konstantin
>
> On Wed, May 3, 2017 at 4:53 PM, Zhe Zhang <zh...@apache.org> wrote:
>
> Thanks for volunteering as RM Konstantin! The plan LGTM.
>
>
> I've created a nightly Jenkins job for branch-2.7 (unit tests):
> https://builds.apache.org/job/Hadoop-branch2.7-nightly/
>
> On Wed, May 3, 2017 at 12:42 AM Konstantin Shvachko <
>
> shv.hadoop@gmail.com>
>
>
> wrote:
>
>
> Hey guys,
>
>
> I and a few of my colleagues would like to help here and move
> 2.7.4
> release forward. A few points in this regard.
>
> 1. Reading through this thread since March 1 I see that Vinod
> hinted on managing the release. Vinod, if you still want the
> job /
> have bandwidth will be happy to work with you.
> Otherwise I am glad to volunteer as the release manager.
>
> 2. In addition to current blockers and criticals, I would like
> to
>
> propose
>
>
>
> a
>
>
> few issues to be included in the release, see the list below.
>
> Those
> are mostly bug fixes and optimizations, which we already have in
> our
>
> internal
>
>
>
> branch and run in production. Plus one minor feature "node
>
>
> labeling", which we found very handy, when you have heterogeneous
>
> environments and mixed workloads, like MR and Spark.
>
> 3. For marking issues for the release I propose to
> - set the target version to 2.7.4, and
> - add a new label "release-blocker"
> That way we will know issues targeted for the release without
> reopening them for backports.
>
> 4. I see quite a few people are interested in the release. With
> all
> the help I think we can target to release by the end of May.
>
> Other things include fixing CHANGES.txt and fixing Jenkins build
> for
>
> 2.7.4
>
>
>
> branch.
>
>
>
> Thanks,
> --Konstantin
>
> ========== List of issue for 2.7.4 ===========
> ------ Backports
> HADOOP-12975 <https://issues.apache.org/jir
> a/browse/HADOOP-12975>.
> Add
>
> du
>
>
>
> jitters
>
>
> HDFS-9710 <https://issues.apache.org/jira/browse/HDFS-9710>. IBR
>
>
> batching
>
>
>
> HDFS-10715 <https://issues.apache.org/jira/browse/HDFS-10715>.
> NPE
>
>
> when applying AvailableSpaceBlockPlacementPolicy
>
> HDFS-2538 <https://issues.apache.org/jira/browse/HDFS-2538>.
> fsck
>
> removal
>
>
>
> of dot printing
>
>
> HDFS-8131 <https://issues.apache.org/jira/browse/HDFS-8131>.
>
> space-balanced
> policy for balancer
> HDFS-8549 <https://issues.apache.org/jira/browse/HDFS-8549>.
> abort
> balancer if upgrade in progress
> HDFS-9412 <https://issues.apache.org/jira/browse/HDFS-9412>.
> skip
> small blocks in getBlocks
>
> YARN-1471 <https://issues.apache.org/jira/browse/YARN-1471>.
> SLS
> simulator
> YARN-4302 <https://issues.apache.org/jira/browse/YARN-4302>.
> SLS
> YARN-4367 <https://issues.apache.org/jira/browse/YARN-4367>.
> SLS
> YARN-4612 <https://issues.apache.org/jira/browse/YARN-4612>.
> SLS
>
> ----- Node labeling
> MAPREDUCE-6304
> <https://issues.apache.org/jira/browse/MAPREDUCE-6304>
> YARN-2943 <https://issues.apache.org/jira/browse/YARN-2943>
> YARN-4109 <https://issues.apache.org/jira/browse/YARN-4109>
> YARN-4140 <https://issues.apache.org/jira/browse/YARN-4140>
> YARN-4250 <https://issues.apache.org/jira/browse/YARN-4250>
> YARN-4925 <https://issues.apache.org/jira/browse/YARN-4925>
>
> --
>
> Zhe Zhang
>
> Apache Hadoop Committer
> http://zhe-thoughts.github.io/about/ | @oldcap
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
>
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>
>
>
>
>
>
Re: About 2.7.4 Release
Posted by Konstantin Shvachko <sh...@gmail.com>.
Hey guys,
Looks like we are done with blockers for Apache Hadoop 2.7.4 release.
https://issues.apache.org/jira/issues/?filter=12340814
I just committed HDFS-11896
<https://issues.apache.org/jira/browse/HDFS-11896>, and decided not to wait
for HDFS-11576 <https://issues.apache.org/jira/browse/HDFS-11576>, see Jira
comment.
Thanks Vinod for pointing out HDFS-11742
<https://issues.apache.org/jira/browse/HDFS-11742>. It was thoroughly
tested on a small cluster and Kihwal committed it last week, thanks.
Will start building initial RC. Please refrain from committing to
branch-2.7 for some time.
Thank you everybody for contributing.
--Konst
On Thu, Jul 20, 2017 at 12:07 PM, Vinod Kumar Vavilapalli <
vinodkv@apache.org> wrote:
> Thanks for taking 2.7.4 over Konstantin!
>
> Regarding rolling RC next week, I still see that there are 4 blocker /
> critical tickets targeted for 2.7.4: https://issues.apache.
> org/jira/issues/?jql=project%20in%20(HDFS%2C%20MAPREDUCE%
> 2C%20HADOOP%2C%20YARN)%20AND%20priority%20in%20(Blocker%2C%
> 20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20%
> 22Target%20Version%2Fs%22%20%3D%202.7.4
> <https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS,%20MAPREDUCE,%20HADOOP,%20YARN)%20AND%20priority%20in%20(Blocker,%20Critical)%20AND%20resolution%20=%20Unresolved%20AND%20%22Target%20Version/s%22%20=%202.7.4>
> .
>
> We should get closure on them. https://issues.apache.
> org/jira/browse/HDFS-11742 definitely was something that was deemed a
> blocker for 2.8.2, not sure about 2.7.4.
>
> I’m ‘back’ - let me know if you need any help.
>
> Thanks
> +Vinod
>
> On Jul 13, 2017, at 5:45 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
> Hi everybody.
>
> We have been doing some internal testing of Hadoop 2.7.4. The testing is
> going well.
> Did not find any major issues on our workloads.
> Used an internal tool called Dynamometer to check NameNode performance on
> real cluster traces. Good.
> Overall test cluster performance looks good.
> Some more testing is still going on.
>
> I plan to build an RC next week. If there are no objection.
>
> Thanks,
> --Konst
>
> On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <shv.hadoop@gmail.com
> >
> wrote:
>
> Hey guys.
>
> An update on 2.7.4 progress.
> We are down to 4 blockers. There is some work remaining on those.
> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
> Would be good if people could follow up on review comments.
>
> I looked through nightly Jenkins build results for 2.7.4 both on Apache
> Jenkins and internal.
> Some test fail intermittently, but there no consistent failures. I filed
> HDFS-11985 to track some of them.
> https://issues.apache.org/jira/browse/HDFS-11985
> I do not currently consider these failures as blockers. LMK if some of
> them are.
>
> We started internal testing of branch-2.7 on one of our smallish (100+
> nodes) test clusters.
> Will update on the results.
>
> There is a plan to enable BigTop for 2.7.4 testing.
>
> Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
> Thank you everybody for contributing to this effort.
>
> Regards,
> --Konstantin
>
>
> On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka <aa...@apache.org>
> wrote:
>
> Sure.
> If you want to edit the wiki, please tell me your ASF confluence account.
>
> -Akira
>
> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>
> Couple of more JIRAs need to be back ported for 2.7.4 release. These will
> solve RM HA unstability issues.
> https://issues.apache.org/jira/browse/YARN-5333
> https://issues.apache.org/jira/browse/YARN-5988
> https://issues.apache.org/jira/browse/YARN-6304
>
> I will raise a JIRAs to back port it.
>
> @Akira , could you help to add these JIRAs into wiki?
>
> Thanks & Regards
> Rohith Sharma K S
>
> On 29 May 2017 at 12:19, Akira Ajisaka <aa...@apache.org> wrote:
>
> Created a page for 2.7.4 release.
>
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
>
> If you want to edit this wiki, please ping me.
>
> Regards,
> Akira
>
>
> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
>
> Hi Konstantin Shvachko
>
>
>
> how about creating a wiki page for 2.7.4 release status like 2.8 and
> trunk in following link.??
>
>
> https://cwiki.apache.org/confluence/display/HADOOP
>
>
> ________________________________
> From: Konstantin Shvachko <sh...@gmail.com>
> Sent: Saturday, May 13, 2017 3:58 AM
> To: Akira Ajisaka
> Cc: Hadoop Common; Hdfs-dev; mapreduce-dev@hadoop.apache.org;
> yarn-dev@hadoop.apache.org
> Subject: Re: About 2.7.4 Release
>
> Latest update on the links and filters. Here is the correct link for
> the
> filter:
> https://issues.apache.org/jira/secure/IssueNavigator.jspa?
> requestId=12340814
>
> Also updated: https://s.apache.org/Dzg4
>
> Had to do some Jira debugging. Sorry for confusion.
>
> Thanks,
> --Konstantin
>
> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>
> wrote:
>
> Hey Akira,
>
>
> I didn't have private filters. Most probably Jira caches something.
> Your filter is in the right direction, but for some reason it lists
> only
> 22 issues, while mine has 29.
> It misses e.g. YARN-5543 <https://issues.apache.org/jir
> a/browse/YARN-5543>
> .
>
> Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
> shared it with "everybody", and updated my link to point to that
> filter.
> So
> you can use any of the three methods below to get the correct list:
> 1. Go to https://s.apache.org/Dzg4
> 2. Go to the filter via
> https://issues.apache.org/jira/issues?filter=12340814
> or by finding "Hadoop 2.7.4 release blockers" filter in the jira
> 3. On Advanced issues search page paste this:
> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels =
> release-blocker
> AND "Target Version/s" = 2.7.4
>
> Hope this solves the confusion for which issues are included.
> Please LMK if it doesn't, as it is important.
>
> Thanks,
> --Konstantin
>
> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka <aa...@apache.org>
> wrote:
>
> Hi Konstantin,
>
>
> Thank you for volunteering as release manager!
>
> Actually the original link works fine: https://s.apache.org/Dzg4
>
>
> I couldn't see the link. Maybe is it private filter?
>
>
> Here is a link I generated: https://s.apache.org/ehKy
> This filter includes resolved issue and excludes fixversion == 2.7.4
>
> Thanks and Regards,
> Akira
>
> On 2017/05/08 19:20, Konstantin Shvachko wrote:
>
> Hi Brahma Reddy Battula,
>
>
> Actually the original link works fine: https://s.apache.org/Dzg4
> Your link excludes closed and resolved issues, which needs
> backporting,
> and
> which we cannot reopen, as discussed in this thread earlier.
>
> Looked through the issues you proposed:
>
> HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
> Seems like a new feature. It helps failover to standby node when
> primary
> is
> under heavy load, but it introduces new APIs, addresses, config
> parameters.
> And needs at least one follow up jira.
> Looks like a backward compatible change, though.
> Did you have a chance to run it in production?
>
> +1 on
> HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987>
>
> [HDFS-10987] Make Decommission less expensive when lot of ...<
>
>
> https://issues.apache.org/jira/browse/HDFS-10987>
>
> issues.apache.org
> When user want to decommission a node which having 50M blocks ,it could
> hold the namesystem lock for long time.We've seen it is taking 36 sec.
> As
> we knew during this ...
>
>
>
> HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902>
>
>
> [HDFS-9902] Support different values of dfs.datanode.du ...<
>
>
> https://issues.apache.org/jira/browse/HDFS-9902>
>
> issues.apache.org
> Now Hadoop support different storage type for DISK, SSD, ARCHIVE and
> RAM_DISK, but they share one configuration dfs.datanode.du.reserved.
> The
> DISK size may be several ...
>
>
>
> HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312>
>
>
> Trash does not descent into child directories to check for ...<
>
>
> https://issues.apache.org/jira/browse/HDFS-8312>
>
> issues.apache.org
> HDFS trash does not descent into child directory to check if user has
> permission to delete files. For example: Run the following command to
> initialize directory ...
>
>
>
> HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100>
>
>
> Upgrade Jsch jar to latest version to fix vulnerability in ...<
>
>
> https://issues.apache.org/jira/browse/HADOOP-14100>
>
> issues.apache.org
> Recently there was on vulnerability reported on jsch library. Its fixed
> in latest 0.1.54 version before CVE was made public.
> https://cve.mitre.org/cgi-bin/cvename.cgi ...
>
>
>
>
> Added them to 2.7.4 release. You should see them via the above link
>
> now.
> Would be good if you could attach backport patches for some of them?
>
> Appreciate your help,
> --Konstantin
>
> On Mon, May 8, 2017 at 8:39 AM, Brahma Reddy Battula <
> brahmareddy.battula@huawei.com> wrote:
>
>
> Looks following link is not correct..
>
>
> https://s.apache.org/Dzg4
>
> It should be like following..?
>
> https://s.apache.org/wi3U
>
>
> Apart from Konstantin mentioned,Following also good to go..? let me
> know
> your thoughts on this.
>
> For Large Cluster:
> =============
>
> https://issues.apache.org/jira/browse/HDFS-9311===Life Line
> Protocol
> https://issues.apache.org/jira/browse/HDFS-10987===Deecommission
> Expensive when lot's of blocks are present
>
> https://issues.apache.org/jira/browse/HDFS-9902===
> "dfs.datanode.du.reserved" per Storage Type
>
> For Security:
> =========
> https://issues.apache.org/jira/browse/HDFS-8312===Trash does not
> descent
> into child directories to check for permission
> https://issues.apache.org/jira/browse/HADOOP-14100===Upgrade Jsch
> jar
> to
> latest version to fix vulnerability in old versions
>
>
>
> Regards
> Brahma Reddy Battula
>
> -----Original Message-----
> From: Erik Krogen [mailto:ekrogen@linkedin.com.INVALID]
> Sent: 06 May 2017 02:40
> To: Konstantin Shvachko
> Cc: Zhe Zhang; Hadoop Common; Hdfs-dev;
> mapreduce-dev@hadoop.apache.org
> ;
> yarn-dev@hadoop.apache.org
> Subject: Re: About 2.7.4 Release
>
> List LGTM Konstantin!
>
> Let's say that we will only create a new tracking JIRA for patches
> which
> do not backport cleanly, to avoid having too many lying around.
> Otherwise
> we can directly attach to old ticket. If a clean backport does
> happen
> to
> break a test the nightly build will help us catch it.
>
> Erik
>
> On Thu, May 4, 2017 at 7:21 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>
> wrote:
>
> Great Zhe. Let's monitor the build.
>
>
> I marked all jiras I knew of for inclusion into 2.7.4 as I
> described
> before.
> Target Version/s: 2.7.4
> Label: release-blocker
>
> Here is the link to the list: https://s.apache.org/Dzg4 Please
> let
> me
> know if I missed anything.
> And feel free to pick up any. Most of backports are pretty
> straightforward, but not all.
>
> We can create tracking jiras for backporting if you need to run
> Jenkins on the patch (and since Allen does not allow reopening
> them).
> But I think the final patch should be attached to the original
> jira.
> Otherwise history will be hard to follow.
>
> Thanks,
> --Konstantin
>
> On Wed, May 3, 2017 at 4:53 PM, Zhe Zhang <zh...@apache.org> wrote:
>
> Thanks for volunteering as RM Konstantin! The plan LGTM.
>
>
> I've created a nightly Jenkins job for branch-2.7 (unit tests):
> https://builds.apache.org/job/Hadoop-branch2.7-nightly/
>
> On Wed, May 3, 2017 at 12:42 AM Konstantin Shvachko <
>
> shv.hadoop@gmail.com>
>
>
> wrote:
>
>
> Hey guys,
>
>
> I and a few of my colleagues would like to help here and move
> 2.7.4
> release forward. A few points in this regard.
>
> 1. Reading through this thread since March 1 I see that Vinod
> hinted on managing the release. Vinod, if you still want the
> job /
> have bandwidth will be happy to work with you.
> Otherwise I am glad to volunteer as the release manager.
>
> 2. In addition to current blockers and criticals, I would like
> to
>
> propose
>
>
>
> a
>
>
> few issues to be included in the release, see the list below.
>
> Those
> are mostly bug fixes and optimizations, which we already have in
> our
>
> internal
>
>
>
> branch and run in production. Plus one minor feature "node
>
>
> labeling", which we found very handy, when you have heterogeneous
>
> environments and mixed workloads, like MR and Spark.
>
> 3. For marking issues for the release I propose to
> - set the target version to 2.7.4, and
> - add a new label "release-blocker"
> That way we will know issues targeted for the release without
> reopening them for backports.
>
> 4. I see quite a few people are interested in the release. With
> all
> the help I think we can target to release by the end of May.
>
> Other things include fixing CHANGES.txt and fixing Jenkins build
> for
>
> 2.7.4
>
>
>
> branch.
>
>
>
> Thanks,
> --Konstantin
>
> ========== List of issue for 2.7.4 ===========
> ------ Backports
> HADOOP-12975 <https://issues.apache.org/jir
> a/browse/HADOOP-12975>.
> Add
>
> du
>
>
>
> jitters
>
>
> HDFS-9710 <https://issues.apache.org/jira/browse/HDFS-9710>. IBR
>
>
> batching
>
>
>
> HDFS-10715 <https://issues.apache.org/jira/browse/HDFS-10715>.
> NPE
>
>
> when applying AvailableSpaceBlockPlacementPolicy
>
> HDFS-2538 <https://issues.apache.org/jira/browse/HDFS-2538>.
> fsck
>
> removal
>
>
>
> of dot printing
>
>
> HDFS-8131 <https://issues.apache.org/jira/browse/HDFS-8131>.
>
> space-balanced
> policy for balancer
> HDFS-8549 <https://issues.apache.org/jira/browse/HDFS-8549>.
> abort
> balancer if upgrade in progress
> HDFS-9412 <https://issues.apache.org/jira/browse/HDFS-9412>.
> skip
> small blocks in getBlocks
>
> YARN-1471 <https://issues.apache.org/jira/browse/YARN-1471>.
> SLS
> simulator
> YARN-4302 <https://issues.apache.org/jira/browse/YARN-4302>.
> SLS
> YARN-4367 <https://issues.apache.org/jira/browse/YARN-4367>.
> SLS
> YARN-4612 <https://issues.apache.org/jira/browse/YARN-4612>.
> SLS
>
> ----- Node labeling
> MAPREDUCE-6304
> <https://issues.apache.org/jira/browse/MAPREDUCE-6304>
> YARN-2943 <https://issues.apache.org/jira/browse/YARN-2943>
> YARN-4109 <https://issues.apache.org/jira/browse/YARN-4109>
> YARN-4140 <https://issues.apache.org/jira/browse/YARN-4140>
> YARN-4250 <https://issues.apache.org/jira/browse/YARN-4250>
> YARN-4925 <https://issues.apache.org/jira/browse/YARN-4925>
>
> --
>
> Zhe Zhang
>
> Apache Hadoop Committer
> http://zhe-thoughts.github.io/about/ | @oldcap
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
>
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>
>
>
>
>
>
Re: About 2.7.4 Release
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Thanks for taking 2.7.4 over Konstantin!
Regarding rolling RC next week, I still see that there are 4 blocker / critical tickets targeted for 2.7.4: https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20MAPREDUCE%2C%20HADOOP%2C%20YARN)%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%2Fs%22%20%3D%202.7.4 <https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS,%20MAPREDUCE,%20HADOOP,%20YARN)%20AND%20priority%20in%20(Blocker,%20Critical)%20AND%20resolution%20=%20Unresolved%20AND%20%22Target%20Version/s%22%20=%202.7.4>.
We should get closure on them. https://issues.apache.org/jira/browse/HDFS-11742 <https://issues.apache.org/jira/browse/HDFS-11742> definitely was something that was deemed a blocker for 2.8.2, not sure about 2.7.4.
I’m ‘back’ - let me know if you need any help.
Thanks
+Vinod
> On Jul 13, 2017, at 5:45 PM, Konstantin Shvachko <sh...@gmail.com> wrote:
>
> Hi everybody.
>
> We have been doing some internal testing of Hadoop 2.7.4. The testing is
> going well.
> Did not find any major issues on our workloads.
> Used an internal tool called Dynamometer to check NameNode performance on
> real cluster traces. Good.
> Overall test cluster performance looks good.
> Some more testing is still going on.
>
> I plan to build an RC next week. If there are no objection.
>
> Thanks,
> --Konst
>
> On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
>> Hey guys.
>>
>> An update on 2.7.4 progress.
>> We are down to 4 blockers. There is some work remaining on those.
>> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
>> Would be good if people could follow up on review comments.
>>
>> I looked through nightly Jenkins build results for 2.7.4 both on Apache
>> Jenkins and internal.
>> Some test fail intermittently, but there no consistent failures. I filed
>> HDFS-11985 to track some of them.
>> https://issues.apache.org/jira/browse/HDFS-11985
>> I do not currently consider these failures as blockers. LMK if some of
>> them are.
>>
>> We started internal testing of branch-2.7 on one of our smallish (100+
>> nodes) test clusters.
>> Will update on the results.
>>
>> There is a plan to enable BigTop for 2.7.4 testing.
>>
>> Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
>> Thank you everybody for contributing to this effort.
>>
>> Regards,
>> --Konstantin
>>
>>
>> On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka <aa...@apache.org>
>> wrote:
>>
>>> Sure.
>>> If you want to edit the wiki, please tell me your ASF confluence account.
>>>
>>> -Akira
>>>
>>> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>>>
>>>> Couple of more JIRAs need to be back ported for 2.7.4 release. These will
>>>> solve RM HA unstability issues.
>>>> https://issues.apache.org/jira/browse/YARN-5333
>>>> https://issues.apache.org/jira/browse/YARN-5988
>>>> https://issues.apache.org/jira/browse/YARN-6304
>>>>
>>>> I will raise a JIRAs to back port it.
>>>>
>>>> @Akira , could you help to add these JIRAs into wiki?
>>>>
>>>> Thanks & Regards
>>>> Rohith Sharma K S
>>>>
>>>> On 29 May 2017 at 12:19, Akira Ajisaka <aa...@apache.org> wrote:
>>>>
>>>> Created a page for 2.7.4 release.
>>>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
>>>>>
>>>>> If you want to edit this wiki, please ping me.
>>>>>
>>>>> Regards,
>>>>> Akira
>>>>>
>>>>>
>>>>> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
>>>>>
>>>>> Hi Konstantin Shvachko
>>>>>>
>>>>>>
>>>>>> how about creating a wiki page for 2.7.4 release status like 2.8 and
>>>>>> trunk in following link.??
>>>>>>
>>>>>>
>>>>>> https://cwiki.apache.org/confluence/display/HADOOP
>>>>>>
>>>>>>
>>>>>> ________________________________
>>>>>> From: Konstantin Shvachko <sh...@gmail.com>
>>>>>> Sent: Saturday, May 13, 2017 3:58 AM
>>>>>> To: Akira Ajisaka
>>>>>> Cc: Hadoop Common; Hdfs-dev; mapreduce-dev@hadoop.apache.org;
>>>>>> yarn-dev@hadoop.apache.org
>>>>>> Subject: Re: About 2.7.4 Release
>>>>>>
>>>>>> Latest update on the links and filters. Here is the correct link for
>>>>>> the
>>>>>> filter:
>>>>>> https://issues.apache.org/jira/secure/IssueNavigator.jspa?
>>>>>> requestId=12340814
>>>>>>
>>>>>> Also updated: https://s.apache.org/Dzg4
>>>>>>
>>>>>> Had to do some Jira debugging. Sorry for confusion.
>>>>>>
>>>>>> Thanks,
>>>>>> --Konstantin
>>>>>>
>>>>>> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <
>>>>>> shv.hadoop@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hey Akira,
>>>>>>
>>>>>>>
>>>>>>> I didn't have private filters. Most probably Jira caches something.
>>>>>>> Your filter is in the right direction, but for some reason it lists
>>>>>>> only
>>>>>>> 22 issues, while mine has 29.
>>>>>>> It misses e.g. YARN-5543 <https://issues.apache.org/jir
>>>>>>> a/browse/YARN-5543>
>>>>>>> .
>>>>>>>
>>>>>>> Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
>>>>>>> shared it with "everybody", and updated my link to point to that
>>>>>>> filter.
>>>>>>> So
>>>>>>> you can use any of the three methods below to get the correct list:
>>>>>>> 1. Go to https://s.apache.org/Dzg4
>>>>>>> 2. Go to the filter via
>>>>>>> https://issues.apache.org/jira/issues?filter=12340814
>>>>>>> or by finding "Hadoop 2.7.4 release blockers" filter in the jira
>>>>>>> 3. On Advanced issues search page paste this:
>>>>>>> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels =
>>>>>>> release-blocker
>>>>>>> AND "Target Version/s" = 2.7.4
>>>>>>>
>>>>>>> Hope this solves the confusion for which issues are included.
>>>>>>> Please LMK if it doesn't, as it is important.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> --Konstantin
>>>>>>>
>>>>>>> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka <aa...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi Konstantin,
>>>>>>>
>>>>>>>>
>>>>>>>> Thank you for volunteering as release manager!
>>>>>>>>
>>>>>>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I couldn't see the link. Maybe is it private filter?
>>>>>>>>
>>>>>>>> Here is a link I generated: https://s.apache.org/ehKy
>>>>>>>> This filter includes resolved issue and excludes fixversion == 2.7.4
>>>>>>>>
>>>>>>>> Thanks and Regards,
>>>>>>>> Akira
>>>>>>>>
>>>>>>>> On 2017/05/08 19:20, Konstantin Shvachko wrote:
>>>>>>>>
>>>>>>>> Hi Brahma Reddy Battula,
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>>>>>>>> Your link excludes closed and resolved issues, which needs
>>>>>>>>> backporting,
>>>>>>>>> and
>>>>>>>>> which we cannot reopen, as discussed in this thread earlier.
>>>>>>>>>
>>>>>>>>> Looked through the issues you proposed:
>>>>>>>>>
>>>>>>>>> HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
>>>>>>>>> Seems like a new feature. It helps failover to standby node when
>>>>>>>>> primary
>>>>>>>>> is
>>>>>>>>> under heavy load, but it introduces new APIs, addresses, config
>>>>>>>>> parameters.
>>>>>>>>> And needs at least one follow up jira.
>>>>>>>>> Looks like a backward compatible change, though.
>>>>>>>>> Did you have a chance to run it in production?
>>>>>>>>>
>>>>>>>>> +1 on
>>>>>>>>> HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987>
>>>>>>>>>
>>>>>>>>> [HDFS-10987] Make Decommission less expensive when lot of ...<
>>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/HDFS-10987>
>>>>>> issues.apache.org
>>>>>> When user want to decommission a node which having 50M blocks ,it could
>>>>>> hold the namesystem lock for long time.We've seen it is taking 36 sec.
>>>>>> As
>>>>>> we knew during this ...
>>>>>>
>>>>>>
>>>>>>
>>>>>> HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902>
>>>>>>
>>>>>>>
>>>>>>>>> [HDFS-9902] Support different values of dfs.datanode.du ...<
>>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/HDFS-9902>
>>>>>> issues.apache.org
>>>>>> Now Hadoop support different storage type for DISK, SSD, ARCHIVE and
>>>>>> RAM_DISK, but they share one configuration dfs.datanode.du.reserved.
>>>>>> The
>>>>>> DISK size may be several ...
>>>>>>
>>>>>>
>>>>>>
>>>>>> HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312>
>>>>>>
>>>>>>>
>>>>>>>>> Trash does not descent into child directories to check for ...<
>>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/HDFS-8312>
>>>>>> issues.apache.org
>>>>>> HDFS trash does not descent into child directory to check if user has
>>>>>> permission to delete files. For example: Run the following command to
>>>>>> initialize directory ...
>>>>>>
>>>>>>
>>>>>>
>>>>>> HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100>
>>>>>>
>>>>>>>
>>>>>>>>> Upgrade Jsch jar to latest version to fix vulnerability in ...<
>>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/HADOOP-14100>
>>>>>> issues.apache.org
>>>>>> Recently there was on vulnerability reported on jsch library. Its fixed
>>>>>> in latest 0.1.54 version before CVE was made public.
>>>>>> https://cve.mitre.org/cgi-bin/cvename.cgi ...
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Added them to 2.7.4 release. You should see them via the above link
>>>>>>>>> now.
>>>>>>>>> Would be good if you could attach backport patches for some of them?
>>>>>>>>>
>>>>>>>>> Appreciate your help,
>>>>>>>>> --Konstantin
>>>>>>>>>
>>>>>>>>> On Mon, May 8, 2017 at 8:39 AM, Brahma Reddy Battula <
>>>>>>>>> brahmareddy.battula@huawei.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Looks following link is not correct..
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> https://s.apache.org/Dzg4
>>>>>>>>>>
>>>>>>>>>> It should be like following..?
>>>>>>>>>>
>>>>>>>>>> https://s.apache.org/wi3U
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Apart from Konstantin mentioned,Following also good to go..? let me
>>>>>>>>>> know
>>>>>>>>>> your thoughts on this.
>>>>>>>>>>
>>>>>>>>>> For Large Cluster:
>>>>>>>>>> =============
>>>>>>>>>>
>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9311===Life Line
>>>>>>>>>> Protocol
>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-10987===Deecommission
>>>>>>>>>> Expensive when lot's of blocks are present
>>>>>>>>>>
>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9902===
>>>>>>>>>> "dfs.datanode.du.reserved" per Storage Type
>>>>>>>>>>
>>>>>>>>>> For Security:
>>>>>>>>>> =========
>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-8312===Trash does not
>>>>>>>>>> descent
>>>>>>>>>> into child directories to check for permission
>>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-14100===Upgrade Jsch
>>>>>>>>>> jar
>>>>>>>>>> to
>>>>>>>>>> latest version to fix vulnerability in old versions
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>> Brahma Reddy Battula
>>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Erik Krogen [mailto:ekrogen@linkedin.com.INVALID]
>>>>>>>>>> Sent: 06 May 2017 02:40
>>>>>>>>>> To: Konstantin Shvachko
>>>>>>>>>> Cc: Zhe Zhang; Hadoop Common; Hdfs-dev;
>>>>>>>>>> mapreduce-dev@hadoop.apache.org
>>>>>>>>>> ;
>>>>>>>>>> yarn-dev@hadoop.apache.org
>>>>>>>>>> Subject: Re: About 2.7.4 Release
>>>>>>>>>>
>>>>>>>>>> List LGTM Konstantin!
>>>>>>>>>>
>>>>>>>>>> Let's say that we will only create a new tracking JIRA for patches
>>>>>>>>>> which
>>>>>>>>>> do not backport cleanly, to avoid having too many lying around.
>>>>>>>>>> Otherwise
>>>>>>>>>> we can directly attach to old ticket. If a clean backport does
>>>>>>>>>> happen
>>>>>>>>>> to
>>>>>>>>>> break a test the nightly build will help us catch it.
>>>>>>>>>>
>>>>>>>>>> Erik
>>>>>>>>>>
>>>>>>>>>> On Thu, May 4, 2017 at 7:21 PM, Konstantin Shvachko <
>>>>>>>>>> shv.hadoop@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Great Zhe. Let's monitor the build.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> I marked all jiras I knew of for inclusion into 2.7.4 as I
>>>>>>>>>>> described
>>>>>>>>>>> before.
>>>>>>>>>>> Target Version/s: 2.7.4
>>>>>>>>>>> Label: release-blocker
>>>>>>>>>>>
>>>>>>>>>>> Here is the link to the list: https://s.apache.org/Dzg4 Please
>>>>>>>>>>> let
>>>>>>>>>>> me
>>>>>>>>>>> know if I missed anything.
>>>>>>>>>>> And feel free to pick up any. Most of backports are pretty
>>>>>>>>>>> straightforward, but not all.
>>>>>>>>>>>
>>>>>>>>>>> We can create tracking jiras for backporting if you need to run
>>>>>>>>>>> Jenkins on the patch (and since Allen does not allow reopening
>>>>>>>>>>> them).
>>>>>>>>>>> But I think the final patch should be attached to the original
>>>>>>>>>>> jira.
>>>>>>>>>>> Otherwise history will be hard to follow.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> --Konstantin
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 3, 2017 at 4:53 PM, Zhe Zhang <zh...@apache.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Thanks for volunteering as RM Konstantin! The plan LGTM.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> I've created a nightly Jenkins job for branch-2.7 (unit tests):
>>>>>>>>>>>> https://builds.apache.org/job/Hadoop-branch2.7-nightly/
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, May 3, 2017 at 12:42 AM Konstantin Shvachko <
>>>>>>>>>>>>
>>>>>>>>>>>> shv.hadoop@gmail.com>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> I and a few of my colleagues would like to help here and move
>>>>>>>>>>>>> 2.7.4
>>>>>>>>>>>>> release forward. A few points in this regard.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1. Reading through this thread since March 1 I see that Vinod
>>>>>>>>>>>>> hinted on managing the release. Vinod, if you still want the
>>>>>>>>>>>>> job /
>>>>>>>>>>>>> have bandwidth will be happy to work with you.
>>>>>>>>>>>>> Otherwise I am glad to volunteer as the release manager.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2. In addition to current blockers and criticals, I would like
>>>>>>>>>>>>> to
>>>>>>>>>>>>>
>>>>>>>>>>>>> propose
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> a
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> few issues to be included in the release, see the list below.
>>>>>>>>>>>>> Those
>>>>>>>>>>>>> are mostly bug fixes and optimizations, which we already have in
>>>>>>>>>>>>> our
>>>>>>>>>>>>>
>>>>>>>>>>>>> internal
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> branch and run in production. Plus one minor feature "node
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> labeling", which we found very handy, when you have heterogeneous
>>>>>>>>>>>>> environments and mixed workloads, like MR and Spark.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 3. For marking issues for the release I propose to
>>>>>>>>>>>>> - set the target version to 2.7.4, and
>>>>>>>>>>>>> - add a new label "release-blocker"
>>>>>>>>>>>>> That way we will know issues targeted for the release without
>>>>>>>>>>>>> reopening them for backports.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 4. I see quite a few people are interested in the release. With
>>>>>>>>>>>>> all
>>>>>>>>>>>>> the help I think we can target to release by the end of May.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Other things include fixing CHANGES.txt and fixing Jenkins build
>>>>>>>>>>>>> for
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2.7.4
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> branch.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> --Konstantin
>>>>>>>>>>>>>
>>>>>>>>>>>>> ========== List of issue for 2.7.4 ===========
>>>>>>>>>>>>> ------ Backports
>>>>>>>>>>>>> HADOOP-12975 <https://issues.apache.org/jir
>>>>>>>>>>>>> a/browse/HADOOP-12975>.
>>>>>>>>>>>>> Add
>>>>>>>>>>>>>
>>>>>>>>>>>>> du
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> jitters
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> HDFS-9710 <https://issues.apache.org/jira/browse/HDFS-9710>. IBR
>>>>>>>>>>>>>
>>>>>>>>>>>>> batching
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> HDFS-10715 <https://issues.apache.org/jira/browse/HDFS-10715>.
>>>>>>>>>>> NPE
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> when applying AvailableSpaceBlockPlacementPolicy
>>>>>>>>>>>>> HDFS-2538 <https://issues.apache.org/jira/browse/HDFS-2538>.
>>>>>>>>>>>>> fsck
>>>>>>>>>>>>>
>>>>>>>>>>>>> removal
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> of dot printing
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> HDFS-8131 <https://issues.apache.org/jira/browse/HDFS-8131>.
>>>>>>>>>>>>> space-balanced
>>>>>>>>>>>>> policy for balancer
>>>>>>>>>>>>> HDFS-8549 <https://issues.apache.org/jira/browse/HDFS-8549>.
>>>>>>>>>>>>> abort
>>>>>>>>>>>>> balancer if upgrade in progress
>>>>>>>>>>>>> HDFS-9412 <https://issues.apache.org/jira/browse/HDFS-9412>.
>>>>>>>>>>>>> skip
>>>>>>>>>>>>> small blocks in getBlocks
>>>>>>>>>>>>>
>>>>>>>>>>>>> YARN-1471 <https://issues.apache.org/jira/browse/YARN-1471>.
>>>>>>>>>>>>> SLS
>>>>>>>>>>>>> simulator
>>>>>>>>>>>>> YARN-4302 <https://issues.apache.org/jira/browse/YARN-4302>.
>>>>>>>>>>>>> SLS
>>>>>>>>>>>>> YARN-4367 <https://issues.apache.org/jira/browse/YARN-4367>.
>>>>>>>>>>>>> SLS
>>>>>>>>>>>>> YARN-4612 <https://issues.apache.org/jira/browse/YARN-4612>.
>>>>>>>>>>>>> SLS
>>>>>>>>>>>>>
>>>>>>>>>>>>> ----- Node labeling
>>>>>>>>>>>>> MAPREDUCE-6304
>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/MAPREDUCE-6304>
>>>>>>>>>>>>> YARN-2943 <https://issues.apache.org/jira/browse/YARN-2943>
>>>>>>>>>>>>> YARN-4109 <https://issues.apache.org/jira/browse/YARN-4109>
>>>>>>>>>>>>> YARN-4140 <https://issues.apache.org/jira/browse/YARN-4140>
>>>>>>>>>>>>> YARN-4250 <https://issues.apache.org/jira/browse/YARN-4250>
>>>>>>>>>>>>> YARN-4925 <https://issues.apache.org/jira/browse/YARN-4925>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>>
>>>>>>>>>>>>> Zhe Zhang
>>>>>>>>>>>> Apache Hadoop Committer
>>>>>>>>>>>> http://zhe-thoughts.github.io/about/ | @oldcap
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>>
>>>>>
>>>>>
>>>>
>>
Re: About 2.7.4 Release
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Thanks for taking 2.7.4 over Konstantin!
Regarding rolling RC next week, I still see that there are 4 blocker / critical tickets targeted for 2.7.4: https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20MAPREDUCE%2C%20HADOOP%2C%20YARN)%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%2Fs%22%20%3D%202.7.4 <https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS,%20MAPREDUCE,%20HADOOP,%20YARN)%20AND%20priority%20in%20(Blocker,%20Critical)%20AND%20resolution%20=%20Unresolved%20AND%20%22Target%20Version/s%22%20=%202.7.4>.
We should get closure on them. https://issues.apache.org/jira/browse/HDFS-11742 <https://issues.apache.org/jira/browse/HDFS-11742> definitely was something that was deemed a blocker for 2.8.2, not sure about 2.7.4.
I’m ‘back’ - let me know if you need any help.
Thanks
+Vinod
> On Jul 13, 2017, at 5:45 PM, Konstantin Shvachko <sh...@gmail.com> wrote:
>
> Hi everybody.
>
> We have been doing some internal testing of Hadoop 2.7.4. The testing is
> going well.
> Did not find any major issues on our workloads.
> Used an internal tool called Dynamometer to check NameNode performance on
> real cluster traces. Good.
> Overall test cluster performance looks good.
> Some more testing is still going on.
>
> I plan to build an RC next week. If there are no objection.
>
> Thanks,
> --Konst
>
> On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
>> Hey guys.
>>
>> An update on 2.7.4 progress.
>> We are down to 4 blockers. There is some work remaining on those.
>> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
>> Would be good if people could follow up on review comments.
>>
>> I looked through nightly Jenkins build results for 2.7.4 both on Apache
>> Jenkins and internal.
>> Some test fail intermittently, but there no consistent failures. I filed
>> HDFS-11985 to track some of them.
>> https://issues.apache.org/jira/browse/HDFS-11985
>> I do not currently consider these failures as blockers. LMK if some of
>> them are.
>>
>> We started internal testing of branch-2.7 on one of our smallish (100+
>> nodes) test clusters.
>> Will update on the results.
>>
>> There is a plan to enable BigTop for 2.7.4 testing.
>>
>> Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
>> Thank you everybody for contributing to this effort.
>>
>> Regards,
>> --Konstantin
>>
>>
>> On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka <aa...@apache.org>
>> wrote:
>>
>>> Sure.
>>> If you want to edit the wiki, please tell me your ASF confluence account.
>>>
>>> -Akira
>>>
>>> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>>>
>>>> Couple of more JIRAs need to be back ported for 2.7.4 release. These will
>>>> solve RM HA unstability issues.
>>>> https://issues.apache.org/jira/browse/YARN-5333
>>>> https://issues.apache.org/jira/browse/YARN-5988
>>>> https://issues.apache.org/jira/browse/YARN-6304
>>>>
>>>> I will raise a JIRAs to back port it.
>>>>
>>>> @Akira , could you help to add these JIRAs into wiki?
>>>>
>>>> Thanks & Regards
>>>> Rohith Sharma K S
>>>>
>>>> On 29 May 2017 at 12:19, Akira Ajisaka <aa...@apache.org> wrote:
>>>>
>>>> Created a page for 2.7.4 release.
>>>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
>>>>>
>>>>> If you want to edit this wiki, please ping me.
>>>>>
>>>>> Regards,
>>>>> Akira
>>>>>
>>>>>
>>>>> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
>>>>>
>>>>> Hi Konstantin Shvachko
>>>>>>
>>>>>>
>>>>>> how about creating a wiki page for 2.7.4 release status like 2.8 and
>>>>>> trunk in following link.??
>>>>>>
>>>>>>
>>>>>> https://cwiki.apache.org/confluence/display/HADOOP
>>>>>>
>>>>>>
>>>>>> ________________________________
>>>>>> From: Konstantin Shvachko <sh...@gmail.com>
>>>>>> Sent: Saturday, May 13, 2017 3:58 AM
>>>>>> To: Akira Ajisaka
>>>>>> Cc: Hadoop Common; Hdfs-dev; mapreduce-dev@hadoop.apache.org;
>>>>>> yarn-dev@hadoop.apache.org
>>>>>> Subject: Re: About 2.7.4 Release
>>>>>>
>>>>>> Latest update on the links and filters. Here is the correct link for
>>>>>> the
>>>>>> filter:
>>>>>> https://issues.apache.org/jira/secure/IssueNavigator.jspa?
>>>>>> requestId=12340814
>>>>>>
>>>>>> Also updated: https://s.apache.org/Dzg4
>>>>>>
>>>>>> Had to do some Jira debugging. Sorry for confusion.
>>>>>>
>>>>>> Thanks,
>>>>>> --Konstantin
>>>>>>
>>>>>> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <
>>>>>> shv.hadoop@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hey Akira,
>>>>>>
>>>>>>>
>>>>>>> I didn't have private filters. Most probably Jira caches something.
>>>>>>> Your filter is in the right direction, but for some reason it lists
>>>>>>> only
>>>>>>> 22 issues, while mine has 29.
>>>>>>> It misses e.g. YARN-5543 <https://issues.apache.org/jir
>>>>>>> a/browse/YARN-5543>
>>>>>>> .
>>>>>>>
>>>>>>> Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
>>>>>>> shared it with "everybody", and updated my link to point to that
>>>>>>> filter.
>>>>>>> So
>>>>>>> you can use any of the three methods below to get the correct list:
>>>>>>> 1. Go to https://s.apache.org/Dzg4
>>>>>>> 2. Go to the filter via
>>>>>>> https://issues.apache.org/jira/issues?filter=12340814
>>>>>>> or by finding "Hadoop 2.7.4 release blockers" filter in the jira
>>>>>>> 3. On Advanced issues search page paste this:
>>>>>>> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels =
>>>>>>> release-blocker
>>>>>>> AND "Target Version/s" = 2.7.4
>>>>>>>
>>>>>>> Hope this solves the confusion for which issues are included.
>>>>>>> Please LMK if it doesn't, as it is important.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> --Konstantin
>>>>>>>
>>>>>>> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka <aa...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi Konstantin,
>>>>>>>
>>>>>>>>
>>>>>>>> Thank you for volunteering as release manager!
>>>>>>>>
>>>>>>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I couldn't see the link. Maybe is it private filter?
>>>>>>>>
>>>>>>>> Here is a link I generated: https://s.apache.org/ehKy
>>>>>>>> This filter includes resolved issue and excludes fixversion == 2.7.4
>>>>>>>>
>>>>>>>> Thanks and Regards,
>>>>>>>> Akira
>>>>>>>>
>>>>>>>> On 2017/05/08 19:20, Konstantin Shvachko wrote:
>>>>>>>>
>>>>>>>> Hi Brahma Reddy Battula,
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>>>>>>>> Your link excludes closed and resolved issues, which needs
>>>>>>>>> backporting,
>>>>>>>>> and
>>>>>>>>> which we cannot reopen, as discussed in this thread earlier.
>>>>>>>>>
>>>>>>>>> Looked through the issues you proposed:
>>>>>>>>>
>>>>>>>>> HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
>>>>>>>>> Seems like a new feature. It helps failover to standby node when
>>>>>>>>> primary
>>>>>>>>> is
>>>>>>>>> under heavy load, but it introduces new APIs, addresses, config
>>>>>>>>> parameters.
>>>>>>>>> And needs at least one follow up jira.
>>>>>>>>> Looks like a backward compatible change, though.
>>>>>>>>> Did you have a chance to run it in production?
>>>>>>>>>
>>>>>>>>> +1 on
>>>>>>>>> HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987>
>>>>>>>>>
>>>>>>>>> [HDFS-10987] Make Decommission less expensive when lot of ...<
>>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/HDFS-10987>
>>>>>> issues.apache.org
>>>>>> When user want to decommission a node which having 50M blocks ,it could
>>>>>> hold the namesystem lock for long time.We've seen it is taking 36 sec.
>>>>>> As
>>>>>> we knew during this ...
>>>>>>
>>>>>>
>>>>>>
>>>>>> HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902>
>>>>>>
>>>>>>>
>>>>>>>>> [HDFS-9902] Support different values of dfs.datanode.du ...<
>>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/HDFS-9902>
>>>>>> issues.apache.org
>>>>>> Now Hadoop support different storage type for DISK, SSD, ARCHIVE and
>>>>>> RAM_DISK, but they share one configuration dfs.datanode.du.reserved.
>>>>>> The
>>>>>> DISK size may be several ...
>>>>>>
>>>>>>
>>>>>>
>>>>>> HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312>
>>>>>>
>>>>>>>
>>>>>>>>> Trash does not descent into child directories to check for ...<
>>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/HDFS-8312>
>>>>>> issues.apache.org
>>>>>> HDFS trash does not descent into child directory to check if user has
>>>>>> permission to delete files. For example: Run the following command to
>>>>>> initialize directory ...
>>>>>>
>>>>>>
>>>>>>
>>>>>> HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100>
>>>>>>
>>>>>>>
>>>>>>>>> Upgrade Jsch jar to latest version to fix vulnerability in ...<
>>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/HADOOP-14100>
>>>>>> issues.apache.org
>>>>>> Recently there was on vulnerability reported on jsch library. Its fixed
>>>>>> in latest 0.1.54 version before CVE was made public.
>>>>>> https://cve.mitre.org/cgi-bin/cvename.cgi ...
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Added them to 2.7.4 release. You should see them via the above link
>>>>>>>>> now.
>>>>>>>>> Would be good if you could attach backport patches for some of them?
>>>>>>>>>
>>>>>>>>> Appreciate your help,
>>>>>>>>> --Konstantin
>>>>>>>>>
>>>>>>>>> On Mon, May 8, 2017 at 8:39 AM, Brahma Reddy Battula <
>>>>>>>>> brahmareddy.battula@huawei.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Looks following link is not correct..
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> https://s.apache.org/Dzg4
>>>>>>>>>>
>>>>>>>>>> It should be like following..?
>>>>>>>>>>
>>>>>>>>>> https://s.apache.org/wi3U
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Apart from Konstantin mentioned,Following also good to go..? let me
>>>>>>>>>> know
>>>>>>>>>> your thoughts on this.
>>>>>>>>>>
>>>>>>>>>> For Large Cluster:
>>>>>>>>>> =============
>>>>>>>>>>
>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9311===Life Line
>>>>>>>>>> Protocol
>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-10987===Deecommission
>>>>>>>>>> Expensive when lot's of blocks are present
>>>>>>>>>>
>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9902===
>>>>>>>>>> "dfs.datanode.du.reserved" per Storage Type
>>>>>>>>>>
>>>>>>>>>> For Security:
>>>>>>>>>> =========
>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-8312===Trash does not
>>>>>>>>>> descent
>>>>>>>>>> into child directories to check for permission
>>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-14100===Upgrade Jsch
>>>>>>>>>> jar
>>>>>>>>>> to
>>>>>>>>>> latest version to fix vulnerability in old versions
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>> Brahma Reddy Battula
>>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Erik Krogen [mailto:ekrogen@linkedin.com.INVALID]
>>>>>>>>>> Sent: 06 May 2017 02:40
>>>>>>>>>> To: Konstantin Shvachko
>>>>>>>>>> Cc: Zhe Zhang; Hadoop Common; Hdfs-dev;
>>>>>>>>>> mapreduce-dev@hadoop.apache.org
>>>>>>>>>> ;
>>>>>>>>>> yarn-dev@hadoop.apache.org
>>>>>>>>>> Subject: Re: About 2.7.4 Release
>>>>>>>>>>
>>>>>>>>>> List LGTM Konstantin!
>>>>>>>>>>
>>>>>>>>>> Let's say that we will only create a new tracking JIRA for patches
>>>>>>>>>> which
>>>>>>>>>> do not backport cleanly, to avoid having too many lying around.
>>>>>>>>>> Otherwise
>>>>>>>>>> we can directly attach to old ticket. If a clean backport does
>>>>>>>>>> happen
>>>>>>>>>> to
>>>>>>>>>> break a test the nightly build will help us catch it.
>>>>>>>>>>
>>>>>>>>>> Erik
>>>>>>>>>>
>>>>>>>>>> On Thu, May 4, 2017 at 7:21 PM, Konstantin Shvachko <
>>>>>>>>>> shv.hadoop@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Great Zhe. Let's monitor the build.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> I marked all jiras I knew of for inclusion into 2.7.4 as I
>>>>>>>>>>> described
>>>>>>>>>>> before.
>>>>>>>>>>> Target Version/s: 2.7.4
>>>>>>>>>>> Label: release-blocker
>>>>>>>>>>>
>>>>>>>>>>> Here is the link to the list: https://s.apache.org/Dzg4 Please
>>>>>>>>>>> let
>>>>>>>>>>> me
>>>>>>>>>>> know if I missed anything.
>>>>>>>>>>> And feel free to pick up any. Most of backports are pretty
>>>>>>>>>>> straightforward, but not all.
>>>>>>>>>>>
>>>>>>>>>>> We can create tracking jiras for backporting if you need to run
>>>>>>>>>>> Jenkins on the patch (and since Allen does not allow reopening
>>>>>>>>>>> them).
>>>>>>>>>>> But I think the final patch should be attached to the original
>>>>>>>>>>> jira.
>>>>>>>>>>> Otherwise history will be hard to follow.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> --Konstantin
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 3, 2017 at 4:53 PM, Zhe Zhang <zh...@apache.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Thanks for volunteering as RM Konstantin! The plan LGTM.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> I've created a nightly Jenkins job for branch-2.7 (unit tests):
>>>>>>>>>>>> https://builds.apache.org/job/Hadoop-branch2.7-nightly/
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, May 3, 2017 at 12:42 AM Konstantin Shvachko <
>>>>>>>>>>>>
>>>>>>>>>>>> shv.hadoop@gmail.com>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> I and a few of my colleagues would like to help here and move
>>>>>>>>>>>>> 2.7.4
>>>>>>>>>>>>> release forward. A few points in this regard.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1. Reading through this thread since March 1 I see that Vinod
>>>>>>>>>>>>> hinted on managing the release. Vinod, if you still want the
>>>>>>>>>>>>> job /
>>>>>>>>>>>>> have bandwidth will be happy to work with you.
>>>>>>>>>>>>> Otherwise I am glad to volunteer as the release manager.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2. In addition to current blockers and criticals, I would like
>>>>>>>>>>>>> to
>>>>>>>>>>>>>
>>>>>>>>>>>>> propose
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> a
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> few issues to be included in the release, see the list below.
>>>>>>>>>>>>> Those
>>>>>>>>>>>>> are mostly bug fixes and optimizations, which we already have in
>>>>>>>>>>>>> our
>>>>>>>>>>>>>
>>>>>>>>>>>>> internal
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> branch and run in production. Plus one minor feature "node
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> labeling", which we found very handy, when you have heterogeneous
>>>>>>>>>>>>> environments and mixed workloads, like MR and Spark.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 3. For marking issues for the release I propose to
>>>>>>>>>>>>> - set the target version to 2.7.4, and
>>>>>>>>>>>>> - add a new label "release-blocker"
>>>>>>>>>>>>> That way we will know issues targeted for the release without
>>>>>>>>>>>>> reopening them for backports.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 4. I see quite a few people are interested in the release. With
>>>>>>>>>>>>> all
>>>>>>>>>>>>> the help I think we can target to release by the end of May.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Other things include fixing CHANGES.txt and fixing Jenkins build
>>>>>>>>>>>>> for
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2.7.4
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> branch.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> --Konstantin
>>>>>>>>>>>>>
>>>>>>>>>>>>> ========== List of issue for 2.7.4 ===========
>>>>>>>>>>>>> ------ Backports
>>>>>>>>>>>>> HADOOP-12975 <https://issues.apache.org/jir
>>>>>>>>>>>>> a/browse/HADOOP-12975>.
>>>>>>>>>>>>> Add
>>>>>>>>>>>>>
>>>>>>>>>>>>> du
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> jitters
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> HDFS-9710 <https://issues.apache.org/jira/browse/HDFS-9710>. IBR
>>>>>>>>>>>>>
>>>>>>>>>>>>> batching
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> HDFS-10715 <https://issues.apache.org/jira/browse/HDFS-10715>.
>>>>>>>>>>> NPE
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> when applying AvailableSpaceBlockPlacementPolicy
>>>>>>>>>>>>> HDFS-2538 <https://issues.apache.org/jira/browse/HDFS-2538>.
>>>>>>>>>>>>> fsck
>>>>>>>>>>>>>
>>>>>>>>>>>>> removal
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> of dot printing
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> HDFS-8131 <https://issues.apache.org/jira/browse/HDFS-8131>.
>>>>>>>>>>>>> space-balanced
>>>>>>>>>>>>> policy for balancer
>>>>>>>>>>>>> HDFS-8549 <https://issues.apache.org/jira/browse/HDFS-8549>.
>>>>>>>>>>>>> abort
>>>>>>>>>>>>> balancer if upgrade in progress
>>>>>>>>>>>>> HDFS-9412 <https://issues.apache.org/jira/browse/HDFS-9412>.
>>>>>>>>>>>>> skip
>>>>>>>>>>>>> small blocks in getBlocks
>>>>>>>>>>>>>
>>>>>>>>>>>>> YARN-1471 <https://issues.apache.org/jira/browse/YARN-1471>.
>>>>>>>>>>>>> SLS
>>>>>>>>>>>>> simulator
>>>>>>>>>>>>> YARN-4302 <https://issues.apache.org/jira/browse/YARN-4302>.
>>>>>>>>>>>>> SLS
>>>>>>>>>>>>> YARN-4367 <https://issues.apache.org/jira/browse/YARN-4367>.
>>>>>>>>>>>>> SLS
>>>>>>>>>>>>> YARN-4612 <https://issues.apache.org/jira/browse/YARN-4612>.
>>>>>>>>>>>>> SLS
>>>>>>>>>>>>>
>>>>>>>>>>>>> ----- Node labeling
>>>>>>>>>>>>> MAPREDUCE-6304
>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/MAPREDUCE-6304>
>>>>>>>>>>>>> YARN-2943 <https://issues.apache.org/jira/browse/YARN-2943>
>>>>>>>>>>>>> YARN-4109 <https://issues.apache.org/jira/browse/YARN-4109>
>>>>>>>>>>>>> YARN-4140 <https://issues.apache.org/jira/browse/YARN-4140>
>>>>>>>>>>>>> YARN-4250 <https://issues.apache.org/jira/browse/YARN-4250>
>>>>>>>>>>>>> YARN-4925 <https://issues.apache.org/jira/browse/YARN-4925>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>>
>>>>>>>>>>>>> Zhe Zhang
>>>>>>>>>>>> Apache Hadoop Committer
>>>>>>>>>>>> http://zhe-thoughts.github.io/about/ | @oldcap
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>>
>>>>>
>>>>>
>>>>
>>
Re: About 2.7.4 Release
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Thanks for taking 2.7.4 over Konstantin!
Regarding rolling RC next week, I still see that there are 4 blocker / critical tickets targeted for 2.7.4: https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20MAPREDUCE%2C%20HADOOP%2C%20YARN)%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%2Fs%22%20%3D%202.7.4 <https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS,%20MAPREDUCE,%20HADOOP,%20YARN)%20AND%20priority%20in%20(Blocker,%20Critical)%20AND%20resolution%20=%20Unresolved%20AND%20%22Target%20Version/s%22%20=%202.7.4>.
We should get closure on them. https://issues.apache.org/jira/browse/HDFS-11742 <https://issues.apache.org/jira/browse/HDFS-11742> definitely was something that was deemed a blocker for 2.8.2, not sure about 2.7.4.
I’m ‘back’ - let me know if you need any help.
Thanks
+Vinod
> On Jul 13, 2017, at 5:45 PM, Konstantin Shvachko <sh...@gmail.com> wrote:
>
> Hi everybody.
>
> We have been doing some internal testing of Hadoop 2.7.4. The testing is
> going well.
> Did not find any major issues on our workloads.
> Used an internal tool called Dynamometer to check NameNode performance on
> real cluster traces. Good.
> Overall test cluster performance looks good.
> Some more testing is still going on.
>
> I plan to build an RC next week. If there are no objection.
>
> Thanks,
> --Konst
>
> On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
>> Hey guys.
>>
>> An update on 2.7.4 progress.
>> We are down to 4 blockers. There is some work remaining on those.
>> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
>> Would be good if people could follow up on review comments.
>>
>> I looked through nightly Jenkins build results for 2.7.4 both on Apache
>> Jenkins and internal.
>> Some test fail intermittently, but there no consistent failures. I filed
>> HDFS-11985 to track some of them.
>> https://issues.apache.org/jira/browse/HDFS-11985
>> I do not currently consider these failures as blockers. LMK if some of
>> them are.
>>
>> We started internal testing of branch-2.7 on one of our smallish (100+
>> nodes) test clusters.
>> Will update on the results.
>>
>> There is a plan to enable BigTop for 2.7.4 testing.
>>
>> Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
>> Thank you everybody for contributing to this effort.
>>
>> Regards,
>> --Konstantin
>>
>>
>> On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka <aa...@apache.org>
>> wrote:
>>
>>> Sure.
>>> If you want to edit the wiki, please tell me your ASF confluence account.
>>>
>>> -Akira
>>>
>>> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>>>
>>>> Couple of more JIRAs need to be back ported for 2.7.4 release. These will
>>>> solve RM HA unstability issues.
>>>> https://issues.apache.org/jira/browse/YARN-5333
>>>> https://issues.apache.org/jira/browse/YARN-5988
>>>> https://issues.apache.org/jira/browse/YARN-6304
>>>>
>>>> I will raise a JIRAs to back port it.
>>>>
>>>> @Akira , could you help to add these JIRAs into wiki?
>>>>
>>>> Thanks & Regards
>>>> Rohith Sharma K S
>>>>
>>>> On 29 May 2017 at 12:19, Akira Ajisaka <aa...@apache.org> wrote:
>>>>
>>>> Created a page for 2.7.4 release.
>>>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
>>>>>
>>>>> If you want to edit this wiki, please ping me.
>>>>>
>>>>> Regards,
>>>>> Akira
>>>>>
>>>>>
>>>>> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
>>>>>
>>>>> Hi Konstantin Shvachko
>>>>>>
>>>>>>
>>>>>> how about creating a wiki page for 2.7.4 release status like 2.8 and
>>>>>> trunk in following link.??
>>>>>>
>>>>>>
>>>>>> https://cwiki.apache.org/confluence/display/HADOOP
>>>>>>
>>>>>>
>>>>>> ________________________________
>>>>>> From: Konstantin Shvachko <sh...@gmail.com>
>>>>>> Sent: Saturday, May 13, 2017 3:58 AM
>>>>>> To: Akira Ajisaka
>>>>>> Cc: Hadoop Common; Hdfs-dev; mapreduce-dev@hadoop.apache.org;
>>>>>> yarn-dev@hadoop.apache.org
>>>>>> Subject: Re: About 2.7.4 Release
>>>>>>
>>>>>> Latest update on the links and filters. Here is the correct link for
>>>>>> the
>>>>>> filter:
>>>>>> https://issues.apache.org/jira/secure/IssueNavigator.jspa?
>>>>>> requestId=12340814
>>>>>>
>>>>>> Also updated: https://s.apache.org/Dzg4
>>>>>>
>>>>>> Had to do some Jira debugging. Sorry for confusion.
>>>>>>
>>>>>> Thanks,
>>>>>> --Konstantin
>>>>>>
>>>>>> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <
>>>>>> shv.hadoop@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hey Akira,
>>>>>>
>>>>>>>
>>>>>>> I didn't have private filters. Most probably Jira caches something.
>>>>>>> Your filter is in the right direction, but for some reason it lists
>>>>>>> only
>>>>>>> 22 issues, while mine has 29.
>>>>>>> It misses e.g. YARN-5543 <https://issues.apache.org/jir
>>>>>>> a/browse/YARN-5543>
>>>>>>> .
>>>>>>>
>>>>>>> Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
>>>>>>> shared it with "everybody", and updated my link to point to that
>>>>>>> filter.
>>>>>>> So
>>>>>>> you can use any of the three methods below to get the correct list:
>>>>>>> 1. Go to https://s.apache.org/Dzg4
>>>>>>> 2. Go to the filter via
>>>>>>> https://issues.apache.org/jira/issues?filter=12340814
>>>>>>> or by finding "Hadoop 2.7.4 release blockers" filter in the jira
>>>>>>> 3. On Advanced issues search page paste this:
>>>>>>> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels =
>>>>>>> release-blocker
>>>>>>> AND "Target Version/s" = 2.7.4
>>>>>>>
>>>>>>> Hope this solves the confusion for which issues are included.
>>>>>>> Please LMK if it doesn't, as it is important.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> --Konstantin
>>>>>>>
>>>>>>> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka <aa...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi Konstantin,
>>>>>>>
>>>>>>>>
>>>>>>>> Thank you for volunteering as release manager!
>>>>>>>>
>>>>>>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I couldn't see the link. Maybe is it private filter?
>>>>>>>>
>>>>>>>> Here is a link I generated: https://s.apache.org/ehKy
>>>>>>>> This filter includes resolved issue and excludes fixversion == 2.7.4
>>>>>>>>
>>>>>>>> Thanks and Regards,
>>>>>>>> Akira
>>>>>>>>
>>>>>>>> On 2017/05/08 19:20, Konstantin Shvachko wrote:
>>>>>>>>
>>>>>>>> Hi Brahma Reddy Battula,
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>>>>>>>> Your link excludes closed and resolved issues, which needs
>>>>>>>>> backporting,
>>>>>>>>> and
>>>>>>>>> which we cannot reopen, as discussed in this thread earlier.
>>>>>>>>>
>>>>>>>>> Looked through the issues you proposed:
>>>>>>>>>
>>>>>>>>> HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
>>>>>>>>> Seems like a new feature. It helps failover to standby node when
>>>>>>>>> primary
>>>>>>>>> is
>>>>>>>>> under heavy load, but it introduces new APIs, addresses, config
>>>>>>>>> parameters.
>>>>>>>>> And needs at least one follow up jira.
>>>>>>>>> Looks like a backward compatible change, though.
>>>>>>>>> Did you have a chance to run it in production?
>>>>>>>>>
>>>>>>>>> +1 on
>>>>>>>>> HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987>
>>>>>>>>>
>>>>>>>>> [HDFS-10987] Make Decommission less expensive when lot of ...<
>>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/HDFS-10987>
>>>>>> issues.apache.org
>>>>>> When user want to decommission a node which having 50M blocks ,it could
>>>>>> hold the namesystem lock for long time.We've seen it is taking 36 sec.
>>>>>> As
>>>>>> we knew during this ...
>>>>>>
>>>>>>
>>>>>>
>>>>>> HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902>
>>>>>>
>>>>>>>
>>>>>>>>> [HDFS-9902] Support different values of dfs.datanode.du ...<
>>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/HDFS-9902>
>>>>>> issues.apache.org
>>>>>> Now Hadoop support different storage type for DISK, SSD, ARCHIVE and
>>>>>> RAM_DISK, but they share one configuration dfs.datanode.du.reserved.
>>>>>> The
>>>>>> DISK size may be several ...
>>>>>>
>>>>>>
>>>>>>
>>>>>> HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312>
>>>>>>
>>>>>>>
>>>>>>>>> Trash does not descent into child directories to check for ...<
>>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/HDFS-8312>
>>>>>> issues.apache.org
>>>>>> HDFS trash does not descent into child directory to check if user has
>>>>>> permission to delete files. For example: Run the following command to
>>>>>> initialize directory ...
>>>>>>
>>>>>>
>>>>>>
>>>>>> HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100>
>>>>>>
>>>>>>>
>>>>>>>>> Upgrade Jsch jar to latest version to fix vulnerability in ...<
>>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/HADOOP-14100>
>>>>>> issues.apache.org
>>>>>> Recently there was on vulnerability reported on jsch library. Its fixed
>>>>>> in latest 0.1.54 version before CVE was made public.
>>>>>> https://cve.mitre.org/cgi-bin/cvename.cgi ...
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Added them to 2.7.4 release. You should see them via the above link
>>>>>>>>> now.
>>>>>>>>> Would be good if you could attach backport patches for some of them?
>>>>>>>>>
>>>>>>>>> Appreciate your help,
>>>>>>>>> --Konstantin
>>>>>>>>>
>>>>>>>>> On Mon, May 8, 2017 at 8:39 AM, Brahma Reddy Battula <
>>>>>>>>> brahmareddy.battula@huawei.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Looks following link is not correct..
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> https://s.apache.org/Dzg4
>>>>>>>>>>
>>>>>>>>>> It should be like following..?
>>>>>>>>>>
>>>>>>>>>> https://s.apache.org/wi3U
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Apart from Konstantin mentioned,Following also good to go..? let me
>>>>>>>>>> know
>>>>>>>>>> your thoughts on this.
>>>>>>>>>>
>>>>>>>>>> For Large Cluster:
>>>>>>>>>> =============
>>>>>>>>>>
>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9311===Life Line
>>>>>>>>>> Protocol
>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-10987===Deecommission
>>>>>>>>>> Expensive when lot's of blocks are present
>>>>>>>>>>
>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9902===
>>>>>>>>>> "dfs.datanode.du.reserved" per Storage Type
>>>>>>>>>>
>>>>>>>>>> For Security:
>>>>>>>>>> =========
>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-8312===Trash does not
>>>>>>>>>> descent
>>>>>>>>>> into child directories to check for permission
>>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-14100===Upgrade Jsch
>>>>>>>>>> jar
>>>>>>>>>> to
>>>>>>>>>> latest version to fix vulnerability in old versions
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>> Brahma Reddy Battula
>>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Erik Krogen [mailto:ekrogen@linkedin.com.INVALID]
>>>>>>>>>> Sent: 06 May 2017 02:40
>>>>>>>>>> To: Konstantin Shvachko
>>>>>>>>>> Cc: Zhe Zhang; Hadoop Common; Hdfs-dev;
>>>>>>>>>> mapreduce-dev@hadoop.apache.org
>>>>>>>>>> ;
>>>>>>>>>> yarn-dev@hadoop.apache.org
>>>>>>>>>> Subject: Re: About 2.7.4 Release
>>>>>>>>>>
>>>>>>>>>> List LGTM Konstantin!
>>>>>>>>>>
>>>>>>>>>> Let's say that we will only create a new tracking JIRA for patches
>>>>>>>>>> which
>>>>>>>>>> do not backport cleanly, to avoid having too many lying around.
>>>>>>>>>> Otherwise
>>>>>>>>>> we can directly attach to old ticket. If a clean backport does
>>>>>>>>>> happen
>>>>>>>>>> to
>>>>>>>>>> break a test the nightly build will help us catch it.
>>>>>>>>>>
>>>>>>>>>> Erik
>>>>>>>>>>
>>>>>>>>>> On Thu, May 4, 2017 at 7:21 PM, Konstantin Shvachko <
>>>>>>>>>> shv.hadoop@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Great Zhe. Let's monitor the build.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> I marked all jiras I knew of for inclusion into 2.7.4 as I
>>>>>>>>>>> described
>>>>>>>>>>> before.
>>>>>>>>>>> Target Version/s: 2.7.4
>>>>>>>>>>> Label: release-blocker
>>>>>>>>>>>
>>>>>>>>>>> Here is the link to the list: https://s.apache.org/Dzg4 Please
>>>>>>>>>>> let
>>>>>>>>>>> me
>>>>>>>>>>> know if I missed anything.
>>>>>>>>>>> And feel free to pick up any. Most of backports are pretty
>>>>>>>>>>> straightforward, but not all.
>>>>>>>>>>>
>>>>>>>>>>> We can create tracking jiras for backporting if you need to run
>>>>>>>>>>> Jenkins on the patch (and since Allen does not allow reopening
>>>>>>>>>>> them).
>>>>>>>>>>> But I think the final patch should be attached to the original
>>>>>>>>>>> jira.
>>>>>>>>>>> Otherwise history will be hard to follow.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> --Konstantin
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 3, 2017 at 4:53 PM, Zhe Zhang <zh...@apache.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Thanks for volunteering as RM Konstantin! The plan LGTM.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> I've created a nightly Jenkins job for branch-2.7 (unit tests):
>>>>>>>>>>>> https://builds.apache.org/job/Hadoop-branch2.7-nightly/
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, May 3, 2017 at 12:42 AM Konstantin Shvachko <
>>>>>>>>>>>>
>>>>>>>>>>>> shv.hadoop@gmail.com>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> I and a few of my colleagues would like to help here and move
>>>>>>>>>>>>> 2.7.4
>>>>>>>>>>>>> release forward. A few points in this regard.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1. Reading through this thread since March 1 I see that Vinod
>>>>>>>>>>>>> hinted on managing the release. Vinod, if you still want the
>>>>>>>>>>>>> job /
>>>>>>>>>>>>> have bandwidth will be happy to work with you.
>>>>>>>>>>>>> Otherwise I am glad to volunteer as the release manager.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2. In addition to current blockers and criticals, I would like
>>>>>>>>>>>>> to
>>>>>>>>>>>>>
>>>>>>>>>>>>> propose
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> a
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> few issues to be included in the release, see the list below.
>>>>>>>>>>>>> Those
>>>>>>>>>>>>> are mostly bug fixes and optimizations, which we already have in
>>>>>>>>>>>>> our
>>>>>>>>>>>>>
>>>>>>>>>>>>> internal
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> branch and run in production. Plus one minor feature "node
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> labeling", which we found very handy, when you have heterogeneous
>>>>>>>>>>>>> environments and mixed workloads, like MR and Spark.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 3. For marking issues for the release I propose to
>>>>>>>>>>>>> - set the target version to 2.7.4, and
>>>>>>>>>>>>> - add a new label "release-blocker"
>>>>>>>>>>>>> That way we will know issues targeted for the release without
>>>>>>>>>>>>> reopening them for backports.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 4. I see quite a few people are interested in the release. With
>>>>>>>>>>>>> all
>>>>>>>>>>>>> the help I think we can target to release by the end of May.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Other things include fixing CHANGES.txt and fixing Jenkins build
>>>>>>>>>>>>> for
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2.7.4
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> branch.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> --Konstantin
>>>>>>>>>>>>>
>>>>>>>>>>>>> ========== List of issue for 2.7.4 ===========
>>>>>>>>>>>>> ------ Backports
>>>>>>>>>>>>> HADOOP-12975 <https://issues.apache.org/jir
>>>>>>>>>>>>> a/browse/HADOOP-12975>.
>>>>>>>>>>>>> Add
>>>>>>>>>>>>>
>>>>>>>>>>>>> du
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> jitters
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> HDFS-9710 <https://issues.apache.org/jira/browse/HDFS-9710>. IBR
>>>>>>>>>>>>>
>>>>>>>>>>>>> batching
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> HDFS-10715 <https://issues.apache.org/jira/browse/HDFS-10715>.
>>>>>>>>>>> NPE
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> when applying AvailableSpaceBlockPlacementPolicy
>>>>>>>>>>>>> HDFS-2538 <https://issues.apache.org/jira/browse/HDFS-2538>.
>>>>>>>>>>>>> fsck
>>>>>>>>>>>>>
>>>>>>>>>>>>> removal
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> of dot printing
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> HDFS-8131 <https://issues.apache.org/jira/browse/HDFS-8131>.
>>>>>>>>>>>>> space-balanced
>>>>>>>>>>>>> policy for balancer
>>>>>>>>>>>>> HDFS-8549 <https://issues.apache.org/jira/browse/HDFS-8549>.
>>>>>>>>>>>>> abort
>>>>>>>>>>>>> balancer if upgrade in progress
>>>>>>>>>>>>> HDFS-9412 <https://issues.apache.org/jira/browse/HDFS-9412>.
>>>>>>>>>>>>> skip
>>>>>>>>>>>>> small blocks in getBlocks
>>>>>>>>>>>>>
>>>>>>>>>>>>> YARN-1471 <https://issues.apache.org/jira/browse/YARN-1471>.
>>>>>>>>>>>>> SLS
>>>>>>>>>>>>> simulator
>>>>>>>>>>>>> YARN-4302 <https://issues.apache.org/jira/browse/YARN-4302>.
>>>>>>>>>>>>> SLS
>>>>>>>>>>>>> YARN-4367 <https://issues.apache.org/jira/browse/YARN-4367>.
>>>>>>>>>>>>> SLS
>>>>>>>>>>>>> YARN-4612 <https://issues.apache.org/jira/browse/YARN-4612>.
>>>>>>>>>>>>> SLS
>>>>>>>>>>>>>
>>>>>>>>>>>>> ----- Node labeling
>>>>>>>>>>>>> MAPREDUCE-6304
>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/MAPREDUCE-6304>
>>>>>>>>>>>>> YARN-2943 <https://issues.apache.org/jira/browse/YARN-2943>
>>>>>>>>>>>>> YARN-4109 <https://issues.apache.org/jira/browse/YARN-4109>
>>>>>>>>>>>>> YARN-4140 <https://issues.apache.org/jira/browse/YARN-4140>
>>>>>>>>>>>>> YARN-4250 <https://issues.apache.org/jira/browse/YARN-4250>
>>>>>>>>>>>>> YARN-4925 <https://issues.apache.org/jira/browse/YARN-4925>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>>
>>>>>>>>>>>>> Zhe Zhang
>>>>>>>>>>>> Apache Hadoop Committer
>>>>>>>>>>>> http://zhe-thoughts.github.io/about/ | @oldcap
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>>
>>>>>
>>>>>
>>>>
>>
Re: About 2.7.4 Release
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Thanks for taking 2.7.4 over Konstantin!
Regarding rolling RC next week, I still see that there are 4 blocker / critical tickets targeted for 2.7.4: https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20MAPREDUCE%2C%20HADOOP%2C%20YARN)%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%2Fs%22%20%3D%202.7.4 <https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS,%20MAPREDUCE,%20HADOOP,%20YARN)%20AND%20priority%20in%20(Blocker,%20Critical)%20AND%20resolution%20=%20Unresolved%20AND%20%22Target%20Version/s%22%20=%202.7.4>.
We should get closure on them. https://issues.apache.org/jira/browse/HDFS-11742 <https://issues.apache.org/jira/browse/HDFS-11742> definitely was something that was deemed a blocker for 2.8.2, not sure about 2.7.4.
I’m ‘back’ - let me know if you need any help.
Thanks
+Vinod
> On Jul 13, 2017, at 5:45 PM, Konstantin Shvachko <sh...@gmail.com> wrote:
>
> Hi everybody.
>
> We have been doing some internal testing of Hadoop 2.7.4. The testing is
> going well.
> Did not find any major issues on our workloads.
> Used an internal tool called Dynamometer to check NameNode performance on
> real cluster traces. Good.
> Overall test cluster performance looks good.
> Some more testing is still going on.
>
> I plan to build an RC next week. If there are no objection.
>
> Thanks,
> --Konst
>
> On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
>> Hey guys.
>>
>> An update on 2.7.4 progress.
>> We are down to 4 blockers. There is some work remaining on those.
>> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
>> Would be good if people could follow up on review comments.
>>
>> I looked through nightly Jenkins build results for 2.7.4 both on Apache
>> Jenkins and internal.
>> Some test fail intermittently, but there no consistent failures. I filed
>> HDFS-11985 to track some of them.
>> https://issues.apache.org/jira/browse/HDFS-11985
>> I do not currently consider these failures as blockers. LMK if some of
>> them are.
>>
>> We started internal testing of branch-2.7 on one of our smallish (100+
>> nodes) test clusters.
>> Will update on the results.
>>
>> There is a plan to enable BigTop for 2.7.4 testing.
>>
>> Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
>> Thank you everybody for contributing to this effort.
>>
>> Regards,
>> --Konstantin
>>
>>
>> On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka <aa...@apache.org>
>> wrote:
>>
>>> Sure.
>>> If you want to edit the wiki, please tell me your ASF confluence account.
>>>
>>> -Akira
>>>
>>> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>>>
>>>> Couple of more JIRAs need to be back ported for 2.7.4 release. These will
>>>> solve RM HA unstability issues.
>>>> https://issues.apache.org/jira/browse/YARN-5333
>>>> https://issues.apache.org/jira/browse/YARN-5988
>>>> https://issues.apache.org/jira/browse/YARN-6304
>>>>
>>>> I will raise a JIRAs to back port it.
>>>>
>>>> @Akira , could you help to add these JIRAs into wiki?
>>>>
>>>> Thanks & Regards
>>>> Rohith Sharma K S
>>>>
>>>> On 29 May 2017 at 12:19, Akira Ajisaka <aa...@apache.org> wrote:
>>>>
>>>> Created a page for 2.7.4 release.
>>>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
>>>>>
>>>>> If you want to edit this wiki, please ping me.
>>>>>
>>>>> Regards,
>>>>> Akira
>>>>>
>>>>>
>>>>> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
>>>>>
>>>>> Hi Konstantin Shvachko
>>>>>>
>>>>>>
>>>>>> how about creating a wiki page for 2.7.4 release status like 2.8 and
>>>>>> trunk in following link.??
>>>>>>
>>>>>>
>>>>>> https://cwiki.apache.org/confluence/display/HADOOP
>>>>>>
>>>>>>
>>>>>> ________________________________
>>>>>> From: Konstantin Shvachko <sh...@gmail.com>
>>>>>> Sent: Saturday, May 13, 2017 3:58 AM
>>>>>> To: Akira Ajisaka
>>>>>> Cc: Hadoop Common; Hdfs-dev; mapreduce-dev@hadoop.apache.org;
>>>>>> yarn-dev@hadoop.apache.org
>>>>>> Subject: Re: About 2.7.4 Release
>>>>>>
>>>>>> Latest update on the links and filters. Here is the correct link for
>>>>>> the
>>>>>> filter:
>>>>>> https://issues.apache.org/jira/secure/IssueNavigator.jspa?
>>>>>> requestId=12340814
>>>>>>
>>>>>> Also updated: https://s.apache.org/Dzg4
>>>>>>
>>>>>> Had to do some Jira debugging. Sorry for confusion.
>>>>>>
>>>>>> Thanks,
>>>>>> --Konstantin
>>>>>>
>>>>>> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <
>>>>>> shv.hadoop@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hey Akira,
>>>>>>
>>>>>>>
>>>>>>> I didn't have private filters. Most probably Jira caches something.
>>>>>>> Your filter is in the right direction, but for some reason it lists
>>>>>>> only
>>>>>>> 22 issues, while mine has 29.
>>>>>>> It misses e.g. YARN-5543 <https://issues.apache.org/jir
>>>>>>> a/browse/YARN-5543>
>>>>>>> .
>>>>>>>
>>>>>>> Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
>>>>>>> shared it with "everybody", and updated my link to point to that
>>>>>>> filter.
>>>>>>> So
>>>>>>> you can use any of the three methods below to get the correct list:
>>>>>>> 1. Go to https://s.apache.org/Dzg4
>>>>>>> 2. Go to the filter via
>>>>>>> https://issues.apache.org/jira/issues?filter=12340814
>>>>>>> or by finding "Hadoop 2.7.4 release blockers" filter in the jira
>>>>>>> 3. On Advanced issues search page paste this:
>>>>>>> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels =
>>>>>>> release-blocker
>>>>>>> AND "Target Version/s" = 2.7.4
>>>>>>>
>>>>>>> Hope this solves the confusion for which issues are included.
>>>>>>> Please LMK if it doesn't, as it is important.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> --Konstantin
>>>>>>>
>>>>>>> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka <aa...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi Konstantin,
>>>>>>>
>>>>>>>>
>>>>>>>> Thank you for volunteering as release manager!
>>>>>>>>
>>>>>>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I couldn't see the link. Maybe is it private filter?
>>>>>>>>
>>>>>>>> Here is a link I generated: https://s.apache.org/ehKy
>>>>>>>> This filter includes resolved issue and excludes fixversion == 2.7.4
>>>>>>>>
>>>>>>>> Thanks and Regards,
>>>>>>>> Akira
>>>>>>>>
>>>>>>>> On 2017/05/08 19:20, Konstantin Shvachko wrote:
>>>>>>>>
>>>>>>>> Hi Brahma Reddy Battula,
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>>>>>>>> Your link excludes closed and resolved issues, which needs
>>>>>>>>> backporting,
>>>>>>>>> and
>>>>>>>>> which we cannot reopen, as discussed in this thread earlier.
>>>>>>>>>
>>>>>>>>> Looked through the issues you proposed:
>>>>>>>>>
>>>>>>>>> HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
>>>>>>>>> Seems like a new feature. It helps failover to standby node when
>>>>>>>>> primary
>>>>>>>>> is
>>>>>>>>> under heavy load, but it introduces new APIs, addresses, config
>>>>>>>>> parameters.
>>>>>>>>> And needs at least one follow up jira.
>>>>>>>>> Looks like a backward compatible change, though.
>>>>>>>>> Did you have a chance to run it in production?
>>>>>>>>>
>>>>>>>>> +1 on
>>>>>>>>> HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987>
>>>>>>>>>
>>>>>>>>> [HDFS-10987] Make Decommission less expensive when lot of ...<
>>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/HDFS-10987>
>>>>>> issues.apache.org
>>>>>> When user want to decommission a node which having 50M blocks ,it could
>>>>>> hold the namesystem lock for long time.We've seen it is taking 36 sec.
>>>>>> As
>>>>>> we knew during this ...
>>>>>>
>>>>>>
>>>>>>
>>>>>> HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902>
>>>>>>
>>>>>>>
>>>>>>>>> [HDFS-9902] Support different values of dfs.datanode.du ...<
>>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/HDFS-9902>
>>>>>> issues.apache.org
>>>>>> Now Hadoop support different storage type for DISK, SSD, ARCHIVE and
>>>>>> RAM_DISK, but they share one configuration dfs.datanode.du.reserved.
>>>>>> The
>>>>>> DISK size may be several ...
>>>>>>
>>>>>>
>>>>>>
>>>>>> HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312>
>>>>>>
>>>>>>>
>>>>>>>>> Trash does not descent into child directories to check for ...<
>>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/HDFS-8312>
>>>>>> issues.apache.org
>>>>>> HDFS trash does not descent into child directory to check if user has
>>>>>> permission to delete files. For example: Run the following command to
>>>>>> initialize directory ...
>>>>>>
>>>>>>
>>>>>>
>>>>>> HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100>
>>>>>>
>>>>>>>
>>>>>>>>> Upgrade Jsch jar to latest version to fix vulnerability in ...<
>>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/HADOOP-14100>
>>>>>> issues.apache.org
>>>>>> Recently there was on vulnerability reported on jsch library. Its fixed
>>>>>> in latest 0.1.54 version before CVE was made public.
>>>>>> https://cve.mitre.org/cgi-bin/cvename.cgi ...
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Added them to 2.7.4 release. You should see them via the above link
>>>>>>>>> now.
>>>>>>>>> Would be good if you could attach backport patches for some of them?
>>>>>>>>>
>>>>>>>>> Appreciate your help,
>>>>>>>>> --Konstantin
>>>>>>>>>
>>>>>>>>> On Mon, May 8, 2017 at 8:39 AM, Brahma Reddy Battula <
>>>>>>>>> brahmareddy.battula@huawei.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Looks following link is not correct..
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> https://s.apache.org/Dzg4
>>>>>>>>>>
>>>>>>>>>> It should be like following..?
>>>>>>>>>>
>>>>>>>>>> https://s.apache.org/wi3U
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Apart from Konstantin mentioned,Following also good to go..? let me
>>>>>>>>>> know
>>>>>>>>>> your thoughts on this.
>>>>>>>>>>
>>>>>>>>>> For Large Cluster:
>>>>>>>>>> =============
>>>>>>>>>>
>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9311===Life Line
>>>>>>>>>> Protocol
>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-10987===Deecommission
>>>>>>>>>> Expensive when lot's of blocks are present
>>>>>>>>>>
>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9902===
>>>>>>>>>> "dfs.datanode.du.reserved" per Storage Type
>>>>>>>>>>
>>>>>>>>>> For Security:
>>>>>>>>>> =========
>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-8312===Trash does not
>>>>>>>>>> descent
>>>>>>>>>> into child directories to check for permission
>>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-14100===Upgrade Jsch
>>>>>>>>>> jar
>>>>>>>>>> to
>>>>>>>>>> latest version to fix vulnerability in old versions
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>> Brahma Reddy Battula
>>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Erik Krogen [mailto:ekrogen@linkedin.com.INVALID]
>>>>>>>>>> Sent: 06 May 2017 02:40
>>>>>>>>>> To: Konstantin Shvachko
>>>>>>>>>> Cc: Zhe Zhang; Hadoop Common; Hdfs-dev;
>>>>>>>>>> mapreduce-dev@hadoop.apache.org
>>>>>>>>>> ;
>>>>>>>>>> yarn-dev@hadoop.apache.org
>>>>>>>>>> Subject: Re: About 2.7.4 Release
>>>>>>>>>>
>>>>>>>>>> List LGTM Konstantin!
>>>>>>>>>>
>>>>>>>>>> Let's say that we will only create a new tracking JIRA for patches
>>>>>>>>>> which
>>>>>>>>>> do not backport cleanly, to avoid having too many lying around.
>>>>>>>>>> Otherwise
>>>>>>>>>> we can directly attach to old ticket. If a clean backport does
>>>>>>>>>> happen
>>>>>>>>>> to
>>>>>>>>>> break a test the nightly build will help us catch it.
>>>>>>>>>>
>>>>>>>>>> Erik
>>>>>>>>>>
>>>>>>>>>> On Thu, May 4, 2017 at 7:21 PM, Konstantin Shvachko <
>>>>>>>>>> shv.hadoop@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Great Zhe. Let's monitor the build.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> I marked all jiras I knew of for inclusion into 2.7.4 as I
>>>>>>>>>>> described
>>>>>>>>>>> before.
>>>>>>>>>>> Target Version/s: 2.7.4
>>>>>>>>>>> Label: release-blocker
>>>>>>>>>>>
>>>>>>>>>>> Here is the link to the list: https://s.apache.org/Dzg4 Please
>>>>>>>>>>> let
>>>>>>>>>>> me
>>>>>>>>>>> know if I missed anything.
>>>>>>>>>>> And feel free to pick up any. Most of backports are pretty
>>>>>>>>>>> straightforward, but not all.
>>>>>>>>>>>
>>>>>>>>>>> We can create tracking jiras for backporting if you need to run
>>>>>>>>>>> Jenkins on the patch (and since Allen does not allow reopening
>>>>>>>>>>> them).
>>>>>>>>>>> But I think the final patch should be attached to the original
>>>>>>>>>>> jira.
>>>>>>>>>>> Otherwise history will be hard to follow.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> --Konstantin
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 3, 2017 at 4:53 PM, Zhe Zhang <zh...@apache.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Thanks for volunteering as RM Konstantin! The plan LGTM.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> I've created a nightly Jenkins job for branch-2.7 (unit tests):
>>>>>>>>>>>> https://builds.apache.org/job/Hadoop-branch2.7-nightly/
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, May 3, 2017 at 12:42 AM Konstantin Shvachko <
>>>>>>>>>>>>
>>>>>>>>>>>> shv.hadoop@gmail.com>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> I and a few of my colleagues would like to help here and move
>>>>>>>>>>>>> 2.7.4
>>>>>>>>>>>>> release forward. A few points in this regard.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1. Reading through this thread since March 1 I see that Vinod
>>>>>>>>>>>>> hinted on managing the release. Vinod, if you still want the
>>>>>>>>>>>>> job /
>>>>>>>>>>>>> have bandwidth will be happy to work with you.
>>>>>>>>>>>>> Otherwise I am glad to volunteer as the release manager.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2. In addition to current blockers and criticals, I would like
>>>>>>>>>>>>> to
>>>>>>>>>>>>>
>>>>>>>>>>>>> propose
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> a
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> few issues to be included in the release, see the list below.
>>>>>>>>>>>>> Those
>>>>>>>>>>>>> are mostly bug fixes and optimizations, which we already have in
>>>>>>>>>>>>> our
>>>>>>>>>>>>>
>>>>>>>>>>>>> internal
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> branch and run in production. Plus one minor feature "node
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> labeling", which we found very handy, when you have heterogeneous
>>>>>>>>>>>>> environments and mixed workloads, like MR and Spark.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 3. For marking issues for the release I propose to
>>>>>>>>>>>>> - set the target version to 2.7.4, and
>>>>>>>>>>>>> - add a new label "release-blocker"
>>>>>>>>>>>>> That way we will know issues targeted for the release without
>>>>>>>>>>>>> reopening them for backports.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 4. I see quite a few people are interested in the release. With
>>>>>>>>>>>>> all
>>>>>>>>>>>>> the help I think we can target to release by the end of May.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Other things include fixing CHANGES.txt and fixing Jenkins build
>>>>>>>>>>>>> for
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2.7.4
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> branch.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> --Konstantin
>>>>>>>>>>>>>
>>>>>>>>>>>>> ========== List of issue for 2.7.4 ===========
>>>>>>>>>>>>> ------ Backports
>>>>>>>>>>>>> HADOOP-12975 <https://issues.apache.org/jir
>>>>>>>>>>>>> a/browse/HADOOP-12975>.
>>>>>>>>>>>>> Add
>>>>>>>>>>>>>
>>>>>>>>>>>>> du
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> jitters
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> HDFS-9710 <https://issues.apache.org/jira/browse/HDFS-9710>. IBR
>>>>>>>>>>>>>
>>>>>>>>>>>>> batching
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> HDFS-10715 <https://issues.apache.org/jira/browse/HDFS-10715>.
>>>>>>>>>>> NPE
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> when applying AvailableSpaceBlockPlacementPolicy
>>>>>>>>>>>>> HDFS-2538 <https://issues.apache.org/jira/browse/HDFS-2538>.
>>>>>>>>>>>>> fsck
>>>>>>>>>>>>>
>>>>>>>>>>>>> removal
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> of dot printing
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> HDFS-8131 <https://issues.apache.org/jira/browse/HDFS-8131>.
>>>>>>>>>>>>> space-balanced
>>>>>>>>>>>>> policy for balancer
>>>>>>>>>>>>> HDFS-8549 <https://issues.apache.org/jira/browse/HDFS-8549>.
>>>>>>>>>>>>> abort
>>>>>>>>>>>>> balancer if upgrade in progress
>>>>>>>>>>>>> HDFS-9412 <https://issues.apache.org/jira/browse/HDFS-9412>.
>>>>>>>>>>>>> skip
>>>>>>>>>>>>> small blocks in getBlocks
>>>>>>>>>>>>>
>>>>>>>>>>>>> YARN-1471 <https://issues.apache.org/jira/browse/YARN-1471>.
>>>>>>>>>>>>> SLS
>>>>>>>>>>>>> simulator
>>>>>>>>>>>>> YARN-4302 <https://issues.apache.org/jira/browse/YARN-4302>.
>>>>>>>>>>>>> SLS
>>>>>>>>>>>>> YARN-4367 <https://issues.apache.org/jira/browse/YARN-4367>.
>>>>>>>>>>>>> SLS
>>>>>>>>>>>>> YARN-4612 <https://issues.apache.org/jira/browse/YARN-4612>.
>>>>>>>>>>>>> SLS
>>>>>>>>>>>>>
>>>>>>>>>>>>> ----- Node labeling
>>>>>>>>>>>>> MAPREDUCE-6304
>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/MAPREDUCE-6304>
>>>>>>>>>>>>> YARN-2943 <https://issues.apache.org/jira/browse/YARN-2943>
>>>>>>>>>>>>> YARN-4109 <https://issues.apache.org/jira/browse/YARN-4109>
>>>>>>>>>>>>> YARN-4140 <https://issues.apache.org/jira/browse/YARN-4140>
>>>>>>>>>>>>> YARN-4250 <https://issues.apache.org/jira/browse/YARN-4250>
>>>>>>>>>>>>> YARN-4925 <https://issues.apache.org/jira/browse/YARN-4925>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>>
>>>>>>>>>>>>> Zhe Zhang
>>>>>>>>>>>> Apache Hadoop Committer
>>>>>>>>>>>> http://zhe-thoughts.github.io/about/ | @oldcap
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>>
>>>>>
>>>>>
>>>>
>>
Re: About 2.7.4 Release
Posted by Konstantin Shvachko <sh...@gmail.com>.
Hi everybody.
We have been doing some internal testing of Hadoop 2.7.4. The testing is
going well.
Did not find any major issues on our workloads.
Used an internal tool called Dynamometer to check NameNode performance on
real cluster traces. Good.
Overall test cluster performance looks good.
Some more testing is still going on.
I plan to build an RC next week. If there are no objection.
Thanks,
--Konst
On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <sh...@gmail.com>
wrote:
> Hey guys.
>
> An update on 2.7.4 progress.
> We are down to 4 blockers. There is some work remaining on those.
> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
> Would be good if people could follow up on review comments.
>
> I looked through nightly Jenkins build results for 2.7.4 both on Apache
> Jenkins and internal.
> Some test fail intermittently, but there no consistent failures. I filed
> HDFS-11985 to track some of them.
> https://issues.apache.org/jira/browse/HDFS-11985
> I do not currently consider these failures as blockers. LMK if some of
> them are.
>
> We started internal testing of branch-2.7 on one of our smallish (100+
> nodes) test clusters.
> Will update on the results.
>
> There is a plan to enable BigTop for 2.7.4 testing.
>
> Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
> Thank you everybody for contributing to this effort.
>
> Regards,
> --Konstantin
>
>
> On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka <aa...@apache.org>
> wrote:
>
>> Sure.
>> If you want to edit the wiki, please tell me your ASF confluence account.
>>
>> -Akira
>>
>> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>>
>>> Couple of more JIRAs need to be back ported for 2.7.4 release. These will
>>> solve RM HA unstability issues.
>>> https://issues.apache.org/jira/browse/YARN-5333
>>> https://issues.apache.org/jira/browse/YARN-5988
>>> https://issues.apache.org/jira/browse/YARN-6304
>>>
>>> I will raise a JIRAs to back port it.
>>>
>>> @Akira , could you help to add these JIRAs into wiki?
>>>
>>> Thanks & Regards
>>> Rohith Sharma K S
>>>
>>> On 29 May 2017 at 12:19, Akira Ajisaka <aa...@apache.org> wrote:
>>>
>>> Created a page for 2.7.4 release.
>>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
>>>>
>>>> If you want to edit this wiki, please ping me.
>>>>
>>>> Regards,
>>>> Akira
>>>>
>>>>
>>>> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
>>>>
>>>> Hi Konstantin Shvachko
>>>>>
>>>>>
>>>>> how about creating a wiki page for 2.7.4 release status like 2.8 and
>>>>> trunk in following link.??
>>>>>
>>>>>
>>>>> https://cwiki.apache.org/confluence/display/HADOOP
>>>>>
>>>>>
>>>>> ________________________________
>>>>> From: Konstantin Shvachko <sh...@gmail.com>
>>>>> Sent: Saturday, May 13, 2017 3:58 AM
>>>>> To: Akira Ajisaka
>>>>> Cc: Hadoop Common; Hdfs-dev; mapreduce-dev@hadoop.apache.org;
>>>>> yarn-dev@hadoop.apache.org
>>>>> Subject: Re: About 2.7.4 Release
>>>>>
>>>>> Latest update on the links and filters. Here is the correct link for
>>>>> the
>>>>> filter:
>>>>> https://issues.apache.org/jira/secure/IssueNavigator.jspa?
>>>>> requestId=12340814
>>>>>
>>>>> Also updated: https://s.apache.org/Dzg4
>>>>>
>>>>> Had to do some Jira debugging. Sorry for confusion.
>>>>>
>>>>> Thanks,
>>>>> --Konstantin
>>>>>
>>>>> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <
>>>>> shv.hadoop@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hey Akira,
>>>>>
>>>>>>
>>>>>> I didn't have private filters. Most probably Jira caches something.
>>>>>> Your filter is in the right direction, but for some reason it lists
>>>>>> only
>>>>>> 22 issues, while mine has 29.
>>>>>> It misses e.g. YARN-5543 <https://issues.apache.org/jir
>>>>>> a/browse/YARN-5543>
>>>>>> .
>>>>>>
>>>>>> Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
>>>>>> shared it with "everybody", and updated my link to point to that
>>>>>> filter.
>>>>>> So
>>>>>> you can use any of the three methods below to get the correct list:
>>>>>> 1. Go to https://s.apache.org/Dzg4
>>>>>> 2. Go to the filter via
>>>>>> https://issues.apache.org/jira/issues?filter=12340814
>>>>>> or by finding "Hadoop 2.7.4 release blockers" filter in the jira
>>>>>> 3. On Advanced issues search page paste this:
>>>>>> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels =
>>>>>> release-blocker
>>>>>> AND "Target Version/s" = 2.7.4
>>>>>>
>>>>>> Hope this solves the confusion for which issues are included.
>>>>>> Please LMK if it doesn't, as it is important.
>>>>>>
>>>>>> Thanks,
>>>>>> --Konstantin
>>>>>>
>>>>>> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka <aa...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>> Hi Konstantin,
>>>>>>
>>>>>>>
>>>>>>> Thank you for volunteering as release manager!
>>>>>>>
>>>>>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>>>>>>
>>>>>>>>
>>>>>>>> I couldn't see the link. Maybe is it private filter?
>>>>>>>
>>>>>>> Here is a link I generated: https://s.apache.org/ehKy
>>>>>>> This filter includes resolved issue and excludes fixversion == 2.7.4
>>>>>>>
>>>>>>> Thanks and Regards,
>>>>>>> Akira
>>>>>>>
>>>>>>> On 2017/05/08 19:20, Konstantin Shvachko wrote:
>>>>>>>
>>>>>>> Hi Brahma Reddy Battula,
>>>>>>>
>>>>>>>>
>>>>>>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>>>>>>> Your link excludes closed and resolved issues, which needs
>>>>>>>> backporting,
>>>>>>>> and
>>>>>>>> which we cannot reopen, as discussed in this thread earlier.
>>>>>>>>
>>>>>>>> Looked through the issues you proposed:
>>>>>>>>
>>>>>>>> HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
>>>>>>>> Seems like a new feature. It helps failover to standby node when
>>>>>>>> primary
>>>>>>>> is
>>>>>>>> under heavy load, but it introduces new APIs, addresses, config
>>>>>>>> parameters.
>>>>>>>> And needs at least one follow up jira.
>>>>>>>> Looks like a backward compatible change, though.
>>>>>>>> Did you have a chance to run it in production?
>>>>>>>>
>>>>>>>> +1 on
>>>>>>>> HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987>
>>>>>>>>
>>>>>>>> [HDFS-10987] Make Decommission less expensive when lot of ...<
>>>>>>>
>>>>>> https://issues.apache.org/jira/browse/HDFS-10987>
>>>>> issues.apache.org
>>>>> When user want to decommission a node which having 50M blocks ,it could
>>>>> hold the namesystem lock for long time.We've seen it is taking 36 sec.
>>>>> As
>>>>> we knew during this ...
>>>>>
>>>>>
>>>>>
>>>>> HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902>
>>>>>
>>>>>>
>>>>>>>> [HDFS-9902] Support different values of dfs.datanode.du ...<
>>>>>>>
>>>>>> https://issues.apache.org/jira/browse/HDFS-9902>
>>>>> issues.apache.org
>>>>> Now Hadoop support different storage type for DISK, SSD, ARCHIVE and
>>>>> RAM_DISK, but they share one configuration dfs.datanode.du.reserved.
>>>>> The
>>>>> DISK size may be several ...
>>>>>
>>>>>
>>>>>
>>>>> HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312>
>>>>>
>>>>>>
>>>>>>>> Trash does not descent into child directories to check for ...<
>>>>>>>
>>>>>> https://issues.apache.org/jira/browse/HDFS-8312>
>>>>> issues.apache.org
>>>>> HDFS trash does not descent into child directory to check if user has
>>>>> permission to delete files. For example: Run the following command to
>>>>> initialize directory ...
>>>>>
>>>>>
>>>>>
>>>>> HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100>
>>>>>
>>>>>>
>>>>>>>> Upgrade Jsch jar to latest version to fix vulnerability in ...<
>>>>>>>
>>>>>> https://issues.apache.org/jira/browse/HADOOP-14100>
>>>>> issues.apache.org
>>>>> Recently there was on vulnerability reported on jsch library. Its fixed
>>>>> in latest 0.1.54 version before CVE was made public.
>>>>> https://cve.mitre.org/cgi-bin/cvename.cgi ...
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Added them to 2.7.4 release. You should see them via the above link
>>>>>>>> now.
>>>>>>>> Would be good if you could attach backport patches for some of them?
>>>>>>>>
>>>>>>>> Appreciate your help,
>>>>>>>> --Konstantin
>>>>>>>>
>>>>>>>> On Mon, May 8, 2017 at 8:39 AM, Brahma Reddy Battula <
>>>>>>>> brahmareddy.battula@huawei.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Looks following link is not correct..
>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://s.apache.org/Dzg4
>>>>>>>>>
>>>>>>>>> It should be like following..?
>>>>>>>>>
>>>>>>>>> https://s.apache.org/wi3U
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Apart from Konstantin mentioned,Following also good to go..? let me
>>>>>>>>> know
>>>>>>>>> your thoughts on this.
>>>>>>>>>
>>>>>>>>> For Large Cluster:
>>>>>>>>> =============
>>>>>>>>>
>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9311===Life Line
>>>>>>>>> Protocol
>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-10987===Deecommission
>>>>>>>>> Expensive when lot's of blocks are present
>>>>>>>>>
>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9902===
>>>>>>>>> "dfs.datanode.du.reserved" per Storage Type
>>>>>>>>>
>>>>>>>>> For Security:
>>>>>>>>> =========
>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-8312===Trash does not
>>>>>>>>> descent
>>>>>>>>> into child directories to check for permission
>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-14100===Upgrade Jsch
>>>>>>>>> jar
>>>>>>>>> to
>>>>>>>>> latest version to fix vulnerability in old versions
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> Brahma Reddy Battula
>>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Erik Krogen [mailto:ekrogen@linkedin.com.INVALID]
>>>>>>>>> Sent: 06 May 2017 02:40
>>>>>>>>> To: Konstantin Shvachko
>>>>>>>>> Cc: Zhe Zhang; Hadoop Common; Hdfs-dev;
>>>>>>>>> mapreduce-dev@hadoop.apache.org
>>>>>>>>> ;
>>>>>>>>> yarn-dev@hadoop.apache.org
>>>>>>>>> Subject: Re: About 2.7.4 Release
>>>>>>>>>
>>>>>>>>> List LGTM Konstantin!
>>>>>>>>>
>>>>>>>>> Let's say that we will only create a new tracking JIRA for patches
>>>>>>>>> which
>>>>>>>>> do not backport cleanly, to avoid having too many lying around.
>>>>>>>>> Otherwise
>>>>>>>>> we can directly attach to old ticket. If a clean backport does
>>>>>>>>> happen
>>>>>>>>> to
>>>>>>>>> break a test the nightly build will help us catch it.
>>>>>>>>>
>>>>>>>>> Erik
>>>>>>>>>
>>>>>>>>> On Thu, May 4, 2017 at 7:21 PM, Konstantin Shvachko <
>>>>>>>>> shv.hadoop@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Great Zhe. Let's monitor the build.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> I marked all jiras I knew of for inclusion into 2.7.4 as I
>>>>>>>>>> described
>>>>>>>>>> before.
>>>>>>>>>> Target Version/s: 2.7.4
>>>>>>>>>> Label: release-blocker
>>>>>>>>>>
>>>>>>>>>> Here is the link to the list: https://s.apache.org/Dzg4 Please
>>>>>>>>>> let
>>>>>>>>>> me
>>>>>>>>>> know if I missed anything.
>>>>>>>>>> And feel free to pick up any. Most of backports are pretty
>>>>>>>>>> straightforward, but not all.
>>>>>>>>>>
>>>>>>>>>> We can create tracking jiras for backporting if you need to run
>>>>>>>>>> Jenkins on the patch (and since Allen does not allow reopening
>>>>>>>>>> them).
>>>>>>>>>> But I think the final patch should be attached to the original
>>>>>>>>>> jira.
>>>>>>>>>> Otherwise history will be hard to follow.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> --Konstantin
>>>>>>>>>>
>>>>>>>>>> On Wed, May 3, 2017 at 4:53 PM, Zhe Zhang <zh...@apache.org> wrote:
>>>>>>>>>>
>>>>>>>>>> Thanks for volunteering as RM Konstantin! The plan LGTM.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> I've created a nightly Jenkins job for branch-2.7 (unit tests):
>>>>>>>>>>> https://builds.apache.org/job/Hadoop-branch2.7-nightly/
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 3, 2017 at 12:42 AM Konstantin Shvachko <
>>>>>>>>>>>
>>>>>>>>>>> shv.hadoop@gmail.com>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hey guys,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> I and a few of my colleagues would like to help here and move
>>>>>>>>>>>> 2.7.4
>>>>>>>>>>>> release forward. A few points in this regard.
>>>>>>>>>>>>
>>>>>>>>>>>> 1. Reading through this thread since March 1 I see that Vinod
>>>>>>>>>>>> hinted on managing the release. Vinod, if you still want the
>>>>>>>>>>>> job /
>>>>>>>>>>>> have bandwidth will be happy to work with you.
>>>>>>>>>>>> Otherwise I am glad to volunteer as the release manager.
>>>>>>>>>>>>
>>>>>>>>>>>> 2. In addition to current blockers and criticals, I would like
>>>>>>>>>>>> to
>>>>>>>>>>>>
>>>>>>>>>>>> propose
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> a
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> few issues to be included in the release, see the list below.
>>>>>>>>>>>> Those
>>>>>>>>>>>> are mostly bug fixes and optimizations, which we already have in
>>>>>>>>>>>> our
>>>>>>>>>>>>
>>>>>>>>>>>> internal
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> branch and run in production. Plus one minor feature "node
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> labeling", which we found very handy, when you have heterogeneous
>>>>>>>>>>>> environments and mixed workloads, like MR and Spark.
>>>>>>>>>>>>
>>>>>>>>>>>> 3. For marking issues for the release I propose to
>>>>>>>>>>>> - set the target version to 2.7.4, and
>>>>>>>>>>>> - add a new label "release-blocker"
>>>>>>>>>>>> That way we will know issues targeted for the release without
>>>>>>>>>>>> reopening them for backports.
>>>>>>>>>>>>
>>>>>>>>>>>> 4. I see quite a few people are interested in the release. With
>>>>>>>>>>>> all
>>>>>>>>>>>> the help I think we can target to release by the end of May.
>>>>>>>>>>>>
>>>>>>>>>>>> Other things include fixing CHANGES.txt and fixing Jenkins build
>>>>>>>>>>>> for
>>>>>>>>>>>>
>>>>>>>>>>>> 2.7.4
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> branch.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> --Konstantin
>>>>>>>>>>>>
>>>>>>>>>>>> ========== List of issue for 2.7.4 ===========
>>>>>>>>>>>> ------ Backports
>>>>>>>>>>>> HADOOP-12975 <https://issues.apache.org/jir
>>>>>>>>>>>> a/browse/HADOOP-12975>.
>>>>>>>>>>>> Add
>>>>>>>>>>>>
>>>>>>>>>>>> du
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> jitters
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> HDFS-9710 <https://issues.apache.org/jira/browse/HDFS-9710>. IBR
>>>>>>>>>>>>
>>>>>>>>>>>> batching
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> HDFS-10715 <https://issues.apache.org/jira/browse/HDFS-10715>.
>>>>>>>>>> NPE
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> when applying AvailableSpaceBlockPlacementPolicy
>>>>>>>>>>>> HDFS-2538 <https://issues.apache.org/jira/browse/HDFS-2538>.
>>>>>>>>>>>> fsck
>>>>>>>>>>>>
>>>>>>>>>>>> removal
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> of dot printing
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> HDFS-8131 <https://issues.apache.org/jira/browse/HDFS-8131>.
>>>>>>>>>>>> space-balanced
>>>>>>>>>>>> policy for balancer
>>>>>>>>>>>> HDFS-8549 <https://issues.apache.org/jira/browse/HDFS-8549>.
>>>>>>>>>>>> abort
>>>>>>>>>>>> balancer if upgrade in progress
>>>>>>>>>>>> HDFS-9412 <https://issues.apache.org/jira/browse/HDFS-9412>.
>>>>>>>>>>>> skip
>>>>>>>>>>>> small blocks in getBlocks
>>>>>>>>>>>>
>>>>>>>>>>>> YARN-1471 <https://issues.apache.org/jira/browse/YARN-1471>.
>>>>>>>>>>>> SLS
>>>>>>>>>>>> simulator
>>>>>>>>>>>> YARN-4302 <https://issues.apache.org/jira/browse/YARN-4302>.
>>>>>>>>>>>> SLS
>>>>>>>>>>>> YARN-4367 <https://issues.apache.org/jira/browse/YARN-4367>.
>>>>>>>>>>>> SLS
>>>>>>>>>>>> YARN-4612 <https://issues.apache.org/jira/browse/YARN-4612>.
>>>>>>>>>>>> SLS
>>>>>>>>>>>>
>>>>>>>>>>>> ----- Node labeling
>>>>>>>>>>>> MAPREDUCE-6304
>>>>>>>>>>>> <https://issues.apache.org/jira/browse/MAPREDUCE-6304>
>>>>>>>>>>>> YARN-2943 <https://issues.apache.org/jira/browse/YARN-2943>
>>>>>>>>>>>> YARN-4109 <https://issues.apache.org/jira/browse/YARN-4109>
>>>>>>>>>>>> YARN-4140 <https://issues.apache.org/jira/browse/YARN-4140>
>>>>>>>>>>>> YARN-4250 <https://issues.apache.org/jira/browse/YARN-4250>
>>>>>>>>>>>> YARN-4925 <https://issues.apache.org/jira/browse/YARN-4925>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>>
>>>>>>>>>>>> Zhe Zhang
>>>>>>>>>>> Apache Hadoop Committer
>>>>>>>>>>> http://zhe-thoughts.github.io/about/ | @oldcap
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>
>>>>
>>>>
>>>
>
Re: About 2.7.4 Release
Posted by Konstantin Shvachko <sh...@gmail.com>.
Hi everybody.
We have been doing some internal testing of Hadoop 2.7.4. The testing is
going well.
Did not find any major issues on our workloads.
Used an internal tool called Dynamometer to check NameNode performance on
real cluster traces. Good.
Overall test cluster performance looks good.
Some more testing is still going on.
I plan to build an RC next week. If there are no objection.
Thanks,
--Konst
On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <sh...@gmail.com>
wrote:
> Hey guys.
>
> An update on 2.7.4 progress.
> We are down to 4 blockers. There is some work remaining on those.
> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
> Would be good if people could follow up on review comments.
>
> I looked through nightly Jenkins build results for 2.7.4 both on Apache
> Jenkins and internal.
> Some test fail intermittently, but there no consistent failures. I filed
> HDFS-11985 to track some of them.
> https://issues.apache.org/jira/browse/HDFS-11985
> I do not currently consider these failures as blockers. LMK if some of
> them are.
>
> We started internal testing of branch-2.7 on one of our smallish (100+
> nodes) test clusters.
> Will update on the results.
>
> There is a plan to enable BigTop for 2.7.4 testing.
>
> Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
> Thank you everybody for contributing to this effort.
>
> Regards,
> --Konstantin
>
>
> On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka <aa...@apache.org>
> wrote:
>
>> Sure.
>> If you want to edit the wiki, please tell me your ASF confluence account.
>>
>> -Akira
>>
>> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>>
>>> Couple of more JIRAs need to be back ported for 2.7.4 release. These will
>>> solve RM HA unstability issues.
>>> https://issues.apache.org/jira/browse/YARN-5333
>>> https://issues.apache.org/jira/browse/YARN-5988
>>> https://issues.apache.org/jira/browse/YARN-6304
>>>
>>> I will raise a JIRAs to back port it.
>>>
>>> @Akira , could you help to add these JIRAs into wiki?
>>>
>>> Thanks & Regards
>>> Rohith Sharma K S
>>>
>>> On 29 May 2017 at 12:19, Akira Ajisaka <aa...@apache.org> wrote:
>>>
>>> Created a page for 2.7.4 release.
>>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
>>>>
>>>> If you want to edit this wiki, please ping me.
>>>>
>>>> Regards,
>>>> Akira
>>>>
>>>>
>>>> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
>>>>
>>>> Hi Konstantin Shvachko
>>>>>
>>>>>
>>>>> how about creating a wiki page for 2.7.4 release status like 2.8 and
>>>>> trunk in following link.??
>>>>>
>>>>>
>>>>> https://cwiki.apache.org/confluence/display/HADOOP
>>>>>
>>>>>
>>>>> ________________________________
>>>>> From: Konstantin Shvachko <sh...@gmail.com>
>>>>> Sent: Saturday, May 13, 2017 3:58 AM
>>>>> To: Akira Ajisaka
>>>>> Cc: Hadoop Common; Hdfs-dev; mapreduce-dev@hadoop.apache.org;
>>>>> yarn-dev@hadoop.apache.org
>>>>> Subject: Re: About 2.7.4 Release
>>>>>
>>>>> Latest update on the links and filters. Here is the correct link for
>>>>> the
>>>>> filter:
>>>>> https://issues.apache.org/jira/secure/IssueNavigator.jspa?
>>>>> requestId=12340814
>>>>>
>>>>> Also updated: https://s.apache.org/Dzg4
>>>>>
>>>>> Had to do some Jira debugging. Sorry for confusion.
>>>>>
>>>>> Thanks,
>>>>> --Konstantin
>>>>>
>>>>> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <
>>>>> shv.hadoop@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hey Akira,
>>>>>
>>>>>>
>>>>>> I didn't have private filters. Most probably Jira caches something.
>>>>>> Your filter is in the right direction, but for some reason it lists
>>>>>> only
>>>>>> 22 issues, while mine has 29.
>>>>>> It misses e.g. YARN-5543 <https://issues.apache.org/jir
>>>>>> a/browse/YARN-5543>
>>>>>> .
>>>>>>
>>>>>> Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
>>>>>> shared it with "everybody", and updated my link to point to that
>>>>>> filter.
>>>>>> So
>>>>>> you can use any of the three methods below to get the correct list:
>>>>>> 1. Go to https://s.apache.org/Dzg4
>>>>>> 2. Go to the filter via
>>>>>> https://issues.apache.org/jira/issues?filter=12340814
>>>>>> or by finding "Hadoop 2.7.4 release blockers" filter in the jira
>>>>>> 3. On Advanced issues search page paste this:
>>>>>> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels =
>>>>>> release-blocker
>>>>>> AND "Target Version/s" = 2.7.4
>>>>>>
>>>>>> Hope this solves the confusion for which issues are included.
>>>>>> Please LMK if it doesn't, as it is important.
>>>>>>
>>>>>> Thanks,
>>>>>> --Konstantin
>>>>>>
>>>>>> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka <aa...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>> Hi Konstantin,
>>>>>>
>>>>>>>
>>>>>>> Thank you for volunteering as release manager!
>>>>>>>
>>>>>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>>>>>>
>>>>>>>>
>>>>>>>> I couldn't see the link. Maybe is it private filter?
>>>>>>>
>>>>>>> Here is a link I generated: https://s.apache.org/ehKy
>>>>>>> This filter includes resolved issue and excludes fixversion == 2.7.4
>>>>>>>
>>>>>>> Thanks and Regards,
>>>>>>> Akira
>>>>>>>
>>>>>>> On 2017/05/08 19:20, Konstantin Shvachko wrote:
>>>>>>>
>>>>>>> Hi Brahma Reddy Battula,
>>>>>>>
>>>>>>>>
>>>>>>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>>>>>>> Your link excludes closed and resolved issues, which needs
>>>>>>>> backporting,
>>>>>>>> and
>>>>>>>> which we cannot reopen, as discussed in this thread earlier.
>>>>>>>>
>>>>>>>> Looked through the issues you proposed:
>>>>>>>>
>>>>>>>> HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
>>>>>>>> Seems like a new feature. It helps failover to standby node when
>>>>>>>> primary
>>>>>>>> is
>>>>>>>> under heavy load, but it introduces new APIs, addresses, config
>>>>>>>> parameters.
>>>>>>>> And needs at least one follow up jira.
>>>>>>>> Looks like a backward compatible change, though.
>>>>>>>> Did you have a chance to run it in production?
>>>>>>>>
>>>>>>>> +1 on
>>>>>>>> HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987>
>>>>>>>>
>>>>>>>> [HDFS-10987] Make Decommission less expensive when lot of ...<
>>>>>>>
>>>>>> https://issues.apache.org/jira/browse/HDFS-10987>
>>>>> issues.apache.org
>>>>> When user want to decommission a node which having 50M blocks ,it could
>>>>> hold the namesystem lock for long time.We've seen it is taking 36 sec.
>>>>> As
>>>>> we knew during this ...
>>>>>
>>>>>
>>>>>
>>>>> HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902>
>>>>>
>>>>>>
>>>>>>>> [HDFS-9902] Support different values of dfs.datanode.du ...<
>>>>>>>
>>>>>> https://issues.apache.org/jira/browse/HDFS-9902>
>>>>> issues.apache.org
>>>>> Now Hadoop support different storage type for DISK, SSD, ARCHIVE and
>>>>> RAM_DISK, but they share one configuration dfs.datanode.du.reserved.
>>>>> The
>>>>> DISK size may be several ...
>>>>>
>>>>>
>>>>>
>>>>> HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312>
>>>>>
>>>>>>
>>>>>>>> Trash does not descent into child directories to check for ...<
>>>>>>>
>>>>>> https://issues.apache.org/jira/browse/HDFS-8312>
>>>>> issues.apache.org
>>>>> HDFS trash does not descent into child directory to check if user has
>>>>> permission to delete files. For example: Run the following command to
>>>>> initialize directory ...
>>>>>
>>>>>
>>>>>
>>>>> HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100>
>>>>>
>>>>>>
>>>>>>>> Upgrade Jsch jar to latest version to fix vulnerability in ...<
>>>>>>>
>>>>>> https://issues.apache.org/jira/browse/HADOOP-14100>
>>>>> issues.apache.org
>>>>> Recently there was on vulnerability reported on jsch library. Its fixed
>>>>> in latest 0.1.54 version before CVE was made public.
>>>>> https://cve.mitre.org/cgi-bin/cvename.cgi ...
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Added them to 2.7.4 release. You should see them via the above link
>>>>>>>> now.
>>>>>>>> Would be good if you could attach backport patches for some of them?
>>>>>>>>
>>>>>>>> Appreciate your help,
>>>>>>>> --Konstantin
>>>>>>>>
>>>>>>>> On Mon, May 8, 2017 at 8:39 AM, Brahma Reddy Battula <
>>>>>>>> brahmareddy.battula@huawei.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Looks following link is not correct..
>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://s.apache.org/Dzg4
>>>>>>>>>
>>>>>>>>> It should be like following..?
>>>>>>>>>
>>>>>>>>> https://s.apache.org/wi3U
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Apart from Konstantin mentioned,Following also good to go..? let me
>>>>>>>>> know
>>>>>>>>> your thoughts on this.
>>>>>>>>>
>>>>>>>>> For Large Cluster:
>>>>>>>>> =============
>>>>>>>>>
>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9311===Life Line
>>>>>>>>> Protocol
>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-10987===Deecommission
>>>>>>>>> Expensive when lot's of blocks are present
>>>>>>>>>
>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9902===
>>>>>>>>> "dfs.datanode.du.reserved" per Storage Type
>>>>>>>>>
>>>>>>>>> For Security:
>>>>>>>>> =========
>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-8312===Trash does not
>>>>>>>>> descent
>>>>>>>>> into child directories to check for permission
>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-14100===Upgrade Jsch
>>>>>>>>> jar
>>>>>>>>> to
>>>>>>>>> latest version to fix vulnerability in old versions
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> Brahma Reddy Battula
>>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Erik Krogen [mailto:ekrogen@linkedin.com.INVALID]
>>>>>>>>> Sent: 06 May 2017 02:40
>>>>>>>>> To: Konstantin Shvachko
>>>>>>>>> Cc: Zhe Zhang; Hadoop Common; Hdfs-dev;
>>>>>>>>> mapreduce-dev@hadoop.apache.org
>>>>>>>>> ;
>>>>>>>>> yarn-dev@hadoop.apache.org
>>>>>>>>> Subject: Re: About 2.7.4 Release
>>>>>>>>>
>>>>>>>>> List LGTM Konstantin!
>>>>>>>>>
>>>>>>>>> Let's say that we will only create a new tracking JIRA for patches
>>>>>>>>> which
>>>>>>>>> do not backport cleanly, to avoid having too many lying around.
>>>>>>>>> Otherwise
>>>>>>>>> we can directly attach to old ticket. If a clean backport does
>>>>>>>>> happen
>>>>>>>>> to
>>>>>>>>> break a test the nightly build will help us catch it.
>>>>>>>>>
>>>>>>>>> Erik
>>>>>>>>>
>>>>>>>>> On Thu, May 4, 2017 at 7:21 PM, Konstantin Shvachko <
>>>>>>>>> shv.hadoop@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Great Zhe. Let's monitor the build.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> I marked all jiras I knew of for inclusion into 2.7.4 as I
>>>>>>>>>> described
>>>>>>>>>> before.
>>>>>>>>>> Target Version/s: 2.7.4
>>>>>>>>>> Label: release-blocker
>>>>>>>>>>
>>>>>>>>>> Here is the link to the list: https://s.apache.org/Dzg4 Please
>>>>>>>>>> let
>>>>>>>>>> me
>>>>>>>>>> know if I missed anything.
>>>>>>>>>> And feel free to pick up any. Most of backports are pretty
>>>>>>>>>> straightforward, but not all.
>>>>>>>>>>
>>>>>>>>>> We can create tracking jiras for backporting if you need to run
>>>>>>>>>> Jenkins on the patch (and since Allen does not allow reopening
>>>>>>>>>> them).
>>>>>>>>>> But I think the final patch should be attached to the original
>>>>>>>>>> jira.
>>>>>>>>>> Otherwise history will be hard to follow.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> --Konstantin
>>>>>>>>>>
>>>>>>>>>> On Wed, May 3, 2017 at 4:53 PM, Zhe Zhang <zh...@apache.org> wrote:
>>>>>>>>>>
>>>>>>>>>> Thanks for volunteering as RM Konstantin! The plan LGTM.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> I've created a nightly Jenkins job for branch-2.7 (unit tests):
>>>>>>>>>>> https://builds.apache.org/job/Hadoop-branch2.7-nightly/
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 3, 2017 at 12:42 AM Konstantin Shvachko <
>>>>>>>>>>>
>>>>>>>>>>> shv.hadoop@gmail.com>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hey guys,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> I and a few of my colleagues would like to help here and move
>>>>>>>>>>>> 2.7.4
>>>>>>>>>>>> release forward. A few points in this regard.
>>>>>>>>>>>>
>>>>>>>>>>>> 1. Reading through this thread since March 1 I see that Vinod
>>>>>>>>>>>> hinted on managing the release. Vinod, if you still want the
>>>>>>>>>>>> job /
>>>>>>>>>>>> have bandwidth will be happy to work with you.
>>>>>>>>>>>> Otherwise I am glad to volunteer as the release manager.
>>>>>>>>>>>>
>>>>>>>>>>>> 2. In addition to current blockers and criticals, I would like
>>>>>>>>>>>> to
>>>>>>>>>>>>
>>>>>>>>>>>> propose
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> a
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> few issues to be included in the release, see the list below.
>>>>>>>>>>>> Those
>>>>>>>>>>>> are mostly bug fixes and optimizations, which we already have in
>>>>>>>>>>>> our
>>>>>>>>>>>>
>>>>>>>>>>>> internal
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> branch and run in production. Plus one minor feature "node
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> labeling", which we found very handy, when you have heterogeneous
>>>>>>>>>>>> environments and mixed workloads, like MR and Spark.
>>>>>>>>>>>>
>>>>>>>>>>>> 3. For marking issues for the release I propose to
>>>>>>>>>>>> - set the target version to 2.7.4, and
>>>>>>>>>>>> - add a new label "release-blocker"
>>>>>>>>>>>> That way we will know issues targeted for the release without
>>>>>>>>>>>> reopening them for backports.
>>>>>>>>>>>>
>>>>>>>>>>>> 4. I see quite a few people are interested in the release. With
>>>>>>>>>>>> all
>>>>>>>>>>>> the help I think we can target to release by the end of May.
>>>>>>>>>>>>
>>>>>>>>>>>> Other things include fixing CHANGES.txt and fixing Jenkins build
>>>>>>>>>>>> for
>>>>>>>>>>>>
>>>>>>>>>>>> 2.7.4
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> branch.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> --Konstantin
>>>>>>>>>>>>
>>>>>>>>>>>> ========== List of issue for 2.7.4 ===========
>>>>>>>>>>>> ------ Backports
>>>>>>>>>>>> HADOOP-12975 <https://issues.apache.org/jir
>>>>>>>>>>>> a/browse/HADOOP-12975>.
>>>>>>>>>>>> Add
>>>>>>>>>>>>
>>>>>>>>>>>> du
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> jitters
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> HDFS-9710 <https://issues.apache.org/jira/browse/HDFS-9710>. IBR
>>>>>>>>>>>>
>>>>>>>>>>>> batching
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> HDFS-10715 <https://issues.apache.org/jira/browse/HDFS-10715>.
>>>>>>>>>> NPE
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> when applying AvailableSpaceBlockPlacementPolicy
>>>>>>>>>>>> HDFS-2538 <https://issues.apache.org/jira/browse/HDFS-2538>.
>>>>>>>>>>>> fsck
>>>>>>>>>>>>
>>>>>>>>>>>> removal
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> of dot printing
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> HDFS-8131 <https://issues.apache.org/jira/browse/HDFS-8131>.
>>>>>>>>>>>> space-balanced
>>>>>>>>>>>> policy for balancer
>>>>>>>>>>>> HDFS-8549 <https://issues.apache.org/jira/browse/HDFS-8549>.
>>>>>>>>>>>> abort
>>>>>>>>>>>> balancer if upgrade in progress
>>>>>>>>>>>> HDFS-9412 <https://issues.apache.org/jira/browse/HDFS-9412>.
>>>>>>>>>>>> skip
>>>>>>>>>>>> small blocks in getBlocks
>>>>>>>>>>>>
>>>>>>>>>>>> YARN-1471 <https://issues.apache.org/jira/browse/YARN-1471>.
>>>>>>>>>>>> SLS
>>>>>>>>>>>> simulator
>>>>>>>>>>>> YARN-4302 <https://issues.apache.org/jira/browse/YARN-4302>.
>>>>>>>>>>>> SLS
>>>>>>>>>>>> YARN-4367 <https://issues.apache.org/jira/browse/YARN-4367>.
>>>>>>>>>>>> SLS
>>>>>>>>>>>> YARN-4612 <https://issues.apache.org/jira/browse/YARN-4612>.
>>>>>>>>>>>> SLS
>>>>>>>>>>>>
>>>>>>>>>>>> ----- Node labeling
>>>>>>>>>>>> MAPREDUCE-6304
>>>>>>>>>>>> <https://issues.apache.org/jira/browse/MAPREDUCE-6304>
>>>>>>>>>>>> YARN-2943 <https://issues.apache.org/jira/browse/YARN-2943>
>>>>>>>>>>>> YARN-4109 <https://issues.apache.org/jira/browse/YARN-4109>
>>>>>>>>>>>> YARN-4140 <https://issues.apache.org/jira/browse/YARN-4140>
>>>>>>>>>>>> YARN-4250 <https://issues.apache.org/jira/browse/YARN-4250>
>>>>>>>>>>>> YARN-4925 <https://issues.apache.org/jira/browse/YARN-4925>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>>
>>>>>>>>>>>> Zhe Zhang
>>>>>>>>>>> Apache Hadoop Committer
>>>>>>>>>>> http://zhe-thoughts.github.io/about/ | @oldcap
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>
>>>>
>>>>
>>>
>
Re: About 2.7.4 Release
Posted by Konstantin Shvachko <sh...@gmail.com>.
Hi everybody.
We have been doing some internal testing of Hadoop 2.7.4. The testing is
going well.
Did not find any major issues on our workloads.
Used an internal tool called Dynamometer to check NameNode performance on
real cluster traces. Good.
Overall test cluster performance looks good.
Some more testing is still going on.
I plan to build an RC next week. If there are no objection.
Thanks,
--Konst
On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <sh...@gmail.com>
wrote:
> Hey guys.
>
> An update on 2.7.4 progress.
> We are down to 4 blockers. There is some work remaining on those.
> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
> Would be good if people could follow up on review comments.
>
> I looked through nightly Jenkins build results for 2.7.4 both on Apache
> Jenkins and internal.
> Some test fail intermittently, but there no consistent failures. I filed
> HDFS-11985 to track some of them.
> https://issues.apache.org/jira/browse/HDFS-11985
> I do not currently consider these failures as blockers. LMK if some of
> them are.
>
> We started internal testing of branch-2.7 on one of our smallish (100+
> nodes) test clusters.
> Will update on the results.
>
> There is a plan to enable BigTop for 2.7.4 testing.
>
> Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
> Thank you everybody for contributing to this effort.
>
> Regards,
> --Konstantin
>
>
> On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka <aa...@apache.org>
> wrote:
>
>> Sure.
>> If you want to edit the wiki, please tell me your ASF confluence account.
>>
>> -Akira
>>
>> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>>
>>> Couple of more JIRAs need to be back ported for 2.7.4 release. These will
>>> solve RM HA unstability issues.
>>> https://issues.apache.org/jira/browse/YARN-5333
>>> https://issues.apache.org/jira/browse/YARN-5988
>>> https://issues.apache.org/jira/browse/YARN-6304
>>>
>>> I will raise a JIRAs to back port it.
>>>
>>> @Akira , could you help to add these JIRAs into wiki?
>>>
>>> Thanks & Regards
>>> Rohith Sharma K S
>>>
>>> On 29 May 2017 at 12:19, Akira Ajisaka <aa...@apache.org> wrote:
>>>
>>> Created a page for 2.7.4 release.
>>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
>>>>
>>>> If you want to edit this wiki, please ping me.
>>>>
>>>> Regards,
>>>> Akira
>>>>
>>>>
>>>> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
>>>>
>>>> Hi Konstantin Shvachko
>>>>>
>>>>>
>>>>> how about creating a wiki page for 2.7.4 release status like 2.8 and
>>>>> trunk in following link.??
>>>>>
>>>>>
>>>>> https://cwiki.apache.org/confluence/display/HADOOP
>>>>>
>>>>>
>>>>> ________________________________
>>>>> From: Konstantin Shvachko <sh...@gmail.com>
>>>>> Sent: Saturday, May 13, 2017 3:58 AM
>>>>> To: Akira Ajisaka
>>>>> Cc: Hadoop Common; Hdfs-dev; mapreduce-dev@hadoop.apache.org;
>>>>> yarn-dev@hadoop.apache.org
>>>>> Subject: Re: About 2.7.4 Release
>>>>>
>>>>> Latest update on the links and filters. Here is the correct link for
>>>>> the
>>>>> filter:
>>>>> https://issues.apache.org/jira/secure/IssueNavigator.jspa?
>>>>> requestId=12340814
>>>>>
>>>>> Also updated: https://s.apache.org/Dzg4
>>>>>
>>>>> Had to do some Jira debugging. Sorry for confusion.
>>>>>
>>>>> Thanks,
>>>>> --Konstantin
>>>>>
>>>>> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <
>>>>> shv.hadoop@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hey Akira,
>>>>>
>>>>>>
>>>>>> I didn't have private filters. Most probably Jira caches something.
>>>>>> Your filter is in the right direction, but for some reason it lists
>>>>>> only
>>>>>> 22 issues, while mine has 29.
>>>>>> It misses e.g. YARN-5543 <https://issues.apache.org/jir
>>>>>> a/browse/YARN-5543>
>>>>>> .
>>>>>>
>>>>>> Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
>>>>>> shared it with "everybody", and updated my link to point to that
>>>>>> filter.
>>>>>> So
>>>>>> you can use any of the three methods below to get the correct list:
>>>>>> 1. Go to https://s.apache.org/Dzg4
>>>>>> 2. Go to the filter via
>>>>>> https://issues.apache.org/jira/issues?filter=12340814
>>>>>> or by finding "Hadoop 2.7.4 release blockers" filter in the jira
>>>>>> 3. On Advanced issues search page paste this:
>>>>>> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels =
>>>>>> release-blocker
>>>>>> AND "Target Version/s" = 2.7.4
>>>>>>
>>>>>> Hope this solves the confusion for which issues are included.
>>>>>> Please LMK if it doesn't, as it is important.
>>>>>>
>>>>>> Thanks,
>>>>>> --Konstantin
>>>>>>
>>>>>> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka <aa...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>> Hi Konstantin,
>>>>>>
>>>>>>>
>>>>>>> Thank you for volunteering as release manager!
>>>>>>>
>>>>>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>>>>>>
>>>>>>>>
>>>>>>>> I couldn't see the link. Maybe is it private filter?
>>>>>>>
>>>>>>> Here is a link I generated: https://s.apache.org/ehKy
>>>>>>> This filter includes resolved issue and excludes fixversion == 2.7.4
>>>>>>>
>>>>>>> Thanks and Regards,
>>>>>>> Akira
>>>>>>>
>>>>>>> On 2017/05/08 19:20, Konstantin Shvachko wrote:
>>>>>>>
>>>>>>> Hi Brahma Reddy Battula,
>>>>>>>
>>>>>>>>
>>>>>>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>>>>>>> Your link excludes closed and resolved issues, which needs
>>>>>>>> backporting,
>>>>>>>> and
>>>>>>>> which we cannot reopen, as discussed in this thread earlier.
>>>>>>>>
>>>>>>>> Looked through the issues you proposed:
>>>>>>>>
>>>>>>>> HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
>>>>>>>> Seems like a new feature. It helps failover to standby node when
>>>>>>>> primary
>>>>>>>> is
>>>>>>>> under heavy load, but it introduces new APIs, addresses, config
>>>>>>>> parameters.
>>>>>>>> And needs at least one follow up jira.
>>>>>>>> Looks like a backward compatible change, though.
>>>>>>>> Did you have a chance to run it in production?
>>>>>>>>
>>>>>>>> +1 on
>>>>>>>> HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987>
>>>>>>>>
>>>>>>>> [HDFS-10987] Make Decommission less expensive when lot of ...<
>>>>>>>
>>>>>> https://issues.apache.org/jira/browse/HDFS-10987>
>>>>> issues.apache.org
>>>>> When user want to decommission a node which having 50M blocks ,it could
>>>>> hold the namesystem lock for long time.We've seen it is taking 36 sec.
>>>>> As
>>>>> we knew during this ...
>>>>>
>>>>>
>>>>>
>>>>> HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902>
>>>>>
>>>>>>
>>>>>>>> [HDFS-9902] Support different values of dfs.datanode.du ...<
>>>>>>>
>>>>>> https://issues.apache.org/jira/browse/HDFS-9902>
>>>>> issues.apache.org
>>>>> Now Hadoop support different storage type for DISK, SSD, ARCHIVE and
>>>>> RAM_DISK, but they share one configuration dfs.datanode.du.reserved.
>>>>> The
>>>>> DISK size may be several ...
>>>>>
>>>>>
>>>>>
>>>>> HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312>
>>>>>
>>>>>>
>>>>>>>> Trash does not descent into child directories to check for ...<
>>>>>>>
>>>>>> https://issues.apache.org/jira/browse/HDFS-8312>
>>>>> issues.apache.org
>>>>> HDFS trash does not descent into child directory to check if user has
>>>>> permission to delete files. For example: Run the following command to
>>>>> initialize directory ...
>>>>>
>>>>>
>>>>>
>>>>> HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100>
>>>>>
>>>>>>
>>>>>>>> Upgrade Jsch jar to latest version to fix vulnerability in ...<
>>>>>>>
>>>>>> https://issues.apache.org/jira/browse/HADOOP-14100>
>>>>> issues.apache.org
>>>>> Recently there was on vulnerability reported on jsch library. Its fixed
>>>>> in latest 0.1.54 version before CVE was made public.
>>>>> https://cve.mitre.org/cgi-bin/cvename.cgi ...
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Added them to 2.7.4 release. You should see them via the above link
>>>>>>>> now.
>>>>>>>> Would be good if you could attach backport patches for some of them?
>>>>>>>>
>>>>>>>> Appreciate your help,
>>>>>>>> --Konstantin
>>>>>>>>
>>>>>>>> On Mon, May 8, 2017 at 8:39 AM, Brahma Reddy Battula <
>>>>>>>> brahmareddy.battula@huawei.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Looks following link is not correct..
>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://s.apache.org/Dzg4
>>>>>>>>>
>>>>>>>>> It should be like following..?
>>>>>>>>>
>>>>>>>>> https://s.apache.org/wi3U
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Apart from Konstantin mentioned,Following also good to go..? let me
>>>>>>>>> know
>>>>>>>>> your thoughts on this.
>>>>>>>>>
>>>>>>>>> For Large Cluster:
>>>>>>>>> =============
>>>>>>>>>
>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9311===Life Line
>>>>>>>>> Protocol
>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-10987===Deecommission
>>>>>>>>> Expensive when lot's of blocks are present
>>>>>>>>>
>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9902===
>>>>>>>>> "dfs.datanode.du.reserved" per Storage Type
>>>>>>>>>
>>>>>>>>> For Security:
>>>>>>>>> =========
>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-8312===Trash does not
>>>>>>>>> descent
>>>>>>>>> into child directories to check for permission
>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-14100===Upgrade Jsch
>>>>>>>>> jar
>>>>>>>>> to
>>>>>>>>> latest version to fix vulnerability in old versions
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> Brahma Reddy Battula
>>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Erik Krogen [mailto:ekrogen@linkedin.com.INVALID]
>>>>>>>>> Sent: 06 May 2017 02:40
>>>>>>>>> To: Konstantin Shvachko
>>>>>>>>> Cc: Zhe Zhang; Hadoop Common; Hdfs-dev;
>>>>>>>>> mapreduce-dev@hadoop.apache.org
>>>>>>>>> ;
>>>>>>>>> yarn-dev@hadoop.apache.org
>>>>>>>>> Subject: Re: About 2.7.4 Release
>>>>>>>>>
>>>>>>>>> List LGTM Konstantin!
>>>>>>>>>
>>>>>>>>> Let's say that we will only create a new tracking JIRA for patches
>>>>>>>>> which
>>>>>>>>> do not backport cleanly, to avoid having too many lying around.
>>>>>>>>> Otherwise
>>>>>>>>> we can directly attach to old ticket. If a clean backport does
>>>>>>>>> happen
>>>>>>>>> to
>>>>>>>>> break a test the nightly build will help us catch it.
>>>>>>>>>
>>>>>>>>> Erik
>>>>>>>>>
>>>>>>>>> On Thu, May 4, 2017 at 7:21 PM, Konstantin Shvachko <
>>>>>>>>> shv.hadoop@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Great Zhe. Let's monitor the build.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> I marked all jiras I knew of for inclusion into 2.7.4 as I
>>>>>>>>>> described
>>>>>>>>>> before.
>>>>>>>>>> Target Version/s: 2.7.4
>>>>>>>>>> Label: release-blocker
>>>>>>>>>>
>>>>>>>>>> Here is the link to the list: https://s.apache.org/Dzg4 Please
>>>>>>>>>> let
>>>>>>>>>> me
>>>>>>>>>> know if I missed anything.
>>>>>>>>>> And feel free to pick up any. Most of backports are pretty
>>>>>>>>>> straightforward, but not all.
>>>>>>>>>>
>>>>>>>>>> We can create tracking jiras for backporting if you need to run
>>>>>>>>>> Jenkins on the patch (and since Allen does not allow reopening
>>>>>>>>>> them).
>>>>>>>>>> But I think the final patch should be attached to the original
>>>>>>>>>> jira.
>>>>>>>>>> Otherwise history will be hard to follow.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> --Konstantin
>>>>>>>>>>
>>>>>>>>>> On Wed, May 3, 2017 at 4:53 PM, Zhe Zhang <zh...@apache.org> wrote:
>>>>>>>>>>
>>>>>>>>>> Thanks for volunteering as RM Konstantin! The plan LGTM.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> I've created a nightly Jenkins job for branch-2.7 (unit tests):
>>>>>>>>>>> https://builds.apache.org/job/Hadoop-branch2.7-nightly/
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 3, 2017 at 12:42 AM Konstantin Shvachko <
>>>>>>>>>>>
>>>>>>>>>>> shv.hadoop@gmail.com>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hey guys,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> I and a few of my colleagues would like to help here and move
>>>>>>>>>>>> 2.7.4
>>>>>>>>>>>> release forward. A few points in this regard.
>>>>>>>>>>>>
>>>>>>>>>>>> 1. Reading through this thread since March 1 I see that Vinod
>>>>>>>>>>>> hinted on managing the release. Vinod, if you still want the
>>>>>>>>>>>> job /
>>>>>>>>>>>> have bandwidth will be happy to work with you.
>>>>>>>>>>>> Otherwise I am glad to volunteer as the release manager.
>>>>>>>>>>>>
>>>>>>>>>>>> 2. In addition to current blockers and criticals, I would like
>>>>>>>>>>>> to
>>>>>>>>>>>>
>>>>>>>>>>>> propose
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> a
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> few issues to be included in the release, see the list below.
>>>>>>>>>>>> Those
>>>>>>>>>>>> are mostly bug fixes and optimizations, which we already have in
>>>>>>>>>>>> our
>>>>>>>>>>>>
>>>>>>>>>>>> internal
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> branch and run in production. Plus one minor feature "node
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> labeling", which we found very handy, when you have heterogeneous
>>>>>>>>>>>> environments and mixed workloads, like MR and Spark.
>>>>>>>>>>>>
>>>>>>>>>>>> 3. For marking issues for the release I propose to
>>>>>>>>>>>> - set the target version to 2.7.4, and
>>>>>>>>>>>> - add a new label "release-blocker"
>>>>>>>>>>>> That way we will know issues targeted for the release without
>>>>>>>>>>>> reopening them for backports.
>>>>>>>>>>>>
>>>>>>>>>>>> 4. I see quite a few people are interested in the release. With
>>>>>>>>>>>> all
>>>>>>>>>>>> the help I think we can target to release by the end of May.
>>>>>>>>>>>>
>>>>>>>>>>>> Other things include fixing CHANGES.txt and fixing Jenkins build
>>>>>>>>>>>> for
>>>>>>>>>>>>
>>>>>>>>>>>> 2.7.4
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> branch.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> --Konstantin
>>>>>>>>>>>>
>>>>>>>>>>>> ========== List of issue for 2.7.4 ===========
>>>>>>>>>>>> ------ Backports
>>>>>>>>>>>> HADOOP-12975 <https://issues.apache.org/jir
>>>>>>>>>>>> a/browse/HADOOP-12975>.
>>>>>>>>>>>> Add
>>>>>>>>>>>>
>>>>>>>>>>>> du
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> jitters
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> HDFS-9710 <https://issues.apache.org/jira/browse/HDFS-9710>. IBR
>>>>>>>>>>>>
>>>>>>>>>>>> batching
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> HDFS-10715 <https://issues.apache.org/jira/browse/HDFS-10715>.
>>>>>>>>>> NPE
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> when applying AvailableSpaceBlockPlacementPolicy
>>>>>>>>>>>> HDFS-2538 <https://issues.apache.org/jira/browse/HDFS-2538>.
>>>>>>>>>>>> fsck
>>>>>>>>>>>>
>>>>>>>>>>>> removal
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> of dot printing
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> HDFS-8131 <https://issues.apache.org/jira/browse/HDFS-8131>.
>>>>>>>>>>>> space-balanced
>>>>>>>>>>>> policy for balancer
>>>>>>>>>>>> HDFS-8549 <https://issues.apache.org/jira/browse/HDFS-8549>.
>>>>>>>>>>>> abort
>>>>>>>>>>>> balancer if upgrade in progress
>>>>>>>>>>>> HDFS-9412 <https://issues.apache.org/jira/browse/HDFS-9412>.
>>>>>>>>>>>> skip
>>>>>>>>>>>> small blocks in getBlocks
>>>>>>>>>>>>
>>>>>>>>>>>> YARN-1471 <https://issues.apache.org/jira/browse/YARN-1471>.
>>>>>>>>>>>> SLS
>>>>>>>>>>>> simulator
>>>>>>>>>>>> YARN-4302 <https://issues.apache.org/jira/browse/YARN-4302>.
>>>>>>>>>>>> SLS
>>>>>>>>>>>> YARN-4367 <https://issues.apache.org/jira/browse/YARN-4367>.
>>>>>>>>>>>> SLS
>>>>>>>>>>>> YARN-4612 <https://issues.apache.org/jira/browse/YARN-4612>.
>>>>>>>>>>>> SLS
>>>>>>>>>>>>
>>>>>>>>>>>> ----- Node labeling
>>>>>>>>>>>> MAPREDUCE-6304
>>>>>>>>>>>> <https://issues.apache.org/jira/browse/MAPREDUCE-6304>
>>>>>>>>>>>> YARN-2943 <https://issues.apache.org/jira/browse/YARN-2943>
>>>>>>>>>>>> YARN-4109 <https://issues.apache.org/jira/browse/YARN-4109>
>>>>>>>>>>>> YARN-4140 <https://issues.apache.org/jira/browse/YARN-4140>
>>>>>>>>>>>> YARN-4250 <https://issues.apache.org/jira/browse/YARN-4250>
>>>>>>>>>>>> YARN-4925 <https://issues.apache.org/jira/browse/YARN-4925>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>>
>>>>>>>>>>>> Zhe Zhang
>>>>>>>>>>> Apache Hadoop Committer
>>>>>>>>>>> http://zhe-thoughts.github.io/about/ | @oldcap
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>
>>>>
>>>>
>>>
>
Re: About 2.7.4 Release
Posted by Konstantin Shvachko <sh...@gmail.com>.
Hi everybody.
We have been doing some internal testing of Hadoop 2.7.4. The testing is
going well.
Did not find any major issues on our workloads.
Used an internal tool called Dynamometer to check NameNode performance on
real cluster traces. Good.
Overall test cluster performance looks good.
Some more testing is still going on.
I plan to build an RC next week. If there are no objection.
Thanks,
--Konst
On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <sh...@gmail.com>
wrote:
> Hey guys.
>
> An update on 2.7.4 progress.
> We are down to 4 blockers. There is some work remaining on those.
> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
> Would be good if people could follow up on review comments.
>
> I looked through nightly Jenkins build results for 2.7.4 both on Apache
> Jenkins and internal.
> Some test fail intermittently, but there no consistent failures. I filed
> HDFS-11985 to track some of them.
> https://issues.apache.org/jira/browse/HDFS-11985
> I do not currently consider these failures as blockers. LMK if some of
> them are.
>
> We started internal testing of branch-2.7 on one of our smallish (100+
> nodes) test clusters.
> Will update on the results.
>
> There is a plan to enable BigTop for 2.7.4 testing.
>
> Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
> Thank you everybody for contributing to this effort.
>
> Regards,
> --Konstantin
>
>
> On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka <aa...@apache.org>
> wrote:
>
>> Sure.
>> If you want to edit the wiki, please tell me your ASF confluence account.
>>
>> -Akira
>>
>> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>>
>>> Couple of more JIRAs need to be back ported for 2.7.4 release. These will
>>> solve RM HA unstability issues.
>>> https://issues.apache.org/jira/browse/YARN-5333
>>> https://issues.apache.org/jira/browse/YARN-5988
>>> https://issues.apache.org/jira/browse/YARN-6304
>>>
>>> I will raise a JIRAs to back port it.
>>>
>>> @Akira , could you help to add these JIRAs into wiki?
>>>
>>> Thanks & Regards
>>> Rohith Sharma K S
>>>
>>> On 29 May 2017 at 12:19, Akira Ajisaka <aa...@apache.org> wrote:
>>>
>>> Created a page for 2.7.4 release.
>>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
>>>>
>>>> If you want to edit this wiki, please ping me.
>>>>
>>>> Regards,
>>>> Akira
>>>>
>>>>
>>>> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
>>>>
>>>> Hi Konstantin Shvachko
>>>>>
>>>>>
>>>>> how about creating a wiki page for 2.7.4 release status like 2.8 and
>>>>> trunk in following link.??
>>>>>
>>>>>
>>>>> https://cwiki.apache.org/confluence/display/HADOOP
>>>>>
>>>>>
>>>>> ________________________________
>>>>> From: Konstantin Shvachko <sh...@gmail.com>
>>>>> Sent: Saturday, May 13, 2017 3:58 AM
>>>>> To: Akira Ajisaka
>>>>> Cc: Hadoop Common; Hdfs-dev; mapreduce-dev@hadoop.apache.org;
>>>>> yarn-dev@hadoop.apache.org
>>>>> Subject: Re: About 2.7.4 Release
>>>>>
>>>>> Latest update on the links and filters. Here is the correct link for
>>>>> the
>>>>> filter:
>>>>> https://issues.apache.org/jira/secure/IssueNavigator.jspa?
>>>>> requestId=12340814
>>>>>
>>>>> Also updated: https://s.apache.org/Dzg4
>>>>>
>>>>> Had to do some Jira debugging. Sorry for confusion.
>>>>>
>>>>> Thanks,
>>>>> --Konstantin
>>>>>
>>>>> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <
>>>>> shv.hadoop@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hey Akira,
>>>>>
>>>>>>
>>>>>> I didn't have private filters. Most probably Jira caches something.
>>>>>> Your filter is in the right direction, but for some reason it lists
>>>>>> only
>>>>>> 22 issues, while mine has 29.
>>>>>> It misses e.g. YARN-5543 <https://issues.apache.org/jir
>>>>>> a/browse/YARN-5543>
>>>>>> .
>>>>>>
>>>>>> Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
>>>>>> shared it with "everybody", and updated my link to point to that
>>>>>> filter.
>>>>>> So
>>>>>> you can use any of the three methods below to get the correct list:
>>>>>> 1. Go to https://s.apache.org/Dzg4
>>>>>> 2. Go to the filter via
>>>>>> https://issues.apache.org/jira/issues?filter=12340814
>>>>>> or by finding "Hadoop 2.7.4 release blockers" filter in the jira
>>>>>> 3. On Advanced issues search page paste this:
>>>>>> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels =
>>>>>> release-blocker
>>>>>> AND "Target Version/s" = 2.7.4
>>>>>>
>>>>>> Hope this solves the confusion for which issues are included.
>>>>>> Please LMK if it doesn't, as it is important.
>>>>>>
>>>>>> Thanks,
>>>>>> --Konstantin
>>>>>>
>>>>>> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka <aa...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>> Hi Konstantin,
>>>>>>
>>>>>>>
>>>>>>> Thank you for volunteering as release manager!
>>>>>>>
>>>>>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>>>>>>
>>>>>>>>
>>>>>>>> I couldn't see the link. Maybe is it private filter?
>>>>>>>
>>>>>>> Here is a link I generated: https://s.apache.org/ehKy
>>>>>>> This filter includes resolved issue and excludes fixversion == 2.7.4
>>>>>>>
>>>>>>> Thanks and Regards,
>>>>>>> Akira
>>>>>>>
>>>>>>> On 2017/05/08 19:20, Konstantin Shvachko wrote:
>>>>>>>
>>>>>>> Hi Brahma Reddy Battula,
>>>>>>>
>>>>>>>>
>>>>>>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>>>>>>> Your link excludes closed and resolved issues, which needs
>>>>>>>> backporting,
>>>>>>>> and
>>>>>>>> which we cannot reopen, as discussed in this thread earlier.
>>>>>>>>
>>>>>>>> Looked through the issues you proposed:
>>>>>>>>
>>>>>>>> HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
>>>>>>>> Seems like a new feature. It helps failover to standby node when
>>>>>>>> primary
>>>>>>>> is
>>>>>>>> under heavy load, but it introduces new APIs, addresses, config
>>>>>>>> parameters.
>>>>>>>> And needs at least one follow up jira.
>>>>>>>> Looks like a backward compatible change, though.
>>>>>>>> Did you have a chance to run it in production?
>>>>>>>>
>>>>>>>> +1 on
>>>>>>>> HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987>
>>>>>>>>
>>>>>>>> [HDFS-10987] Make Decommission less expensive when lot of ...<
>>>>>>>
>>>>>> https://issues.apache.org/jira/browse/HDFS-10987>
>>>>> issues.apache.org
>>>>> When user want to decommission a node which having 50M blocks ,it could
>>>>> hold the namesystem lock for long time.We've seen it is taking 36 sec.
>>>>> As
>>>>> we knew during this ...
>>>>>
>>>>>
>>>>>
>>>>> HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902>
>>>>>
>>>>>>
>>>>>>>> [HDFS-9902] Support different values of dfs.datanode.du ...<
>>>>>>>
>>>>>> https://issues.apache.org/jira/browse/HDFS-9902>
>>>>> issues.apache.org
>>>>> Now Hadoop support different storage type for DISK, SSD, ARCHIVE and
>>>>> RAM_DISK, but they share one configuration dfs.datanode.du.reserved.
>>>>> The
>>>>> DISK size may be several ...
>>>>>
>>>>>
>>>>>
>>>>> HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312>
>>>>>
>>>>>>
>>>>>>>> Trash does not descent into child directories to check for ...<
>>>>>>>
>>>>>> https://issues.apache.org/jira/browse/HDFS-8312>
>>>>> issues.apache.org
>>>>> HDFS trash does not descent into child directory to check if user has
>>>>> permission to delete files. For example: Run the following command to
>>>>> initialize directory ...
>>>>>
>>>>>
>>>>>
>>>>> HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100>
>>>>>
>>>>>>
>>>>>>>> Upgrade Jsch jar to latest version to fix vulnerability in ...<
>>>>>>>
>>>>>> https://issues.apache.org/jira/browse/HADOOP-14100>
>>>>> issues.apache.org
>>>>> Recently there was on vulnerability reported on jsch library. Its fixed
>>>>> in latest 0.1.54 version before CVE was made public.
>>>>> https://cve.mitre.org/cgi-bin/cvename.cgi ...
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Added them to 2.7.4 release. You should see them via the above link
>>>>>>>> now.
>>>>>>>> Would be good if you could attach backport patches for some of them?
>>>>>>>>
>>>>>>>> Appreciate your help,
>>>>>>>> --Konstantin
>>>>>>>>
>>>>>>>> On Mon, May 8, 2017 at 8:39 AM, Brahma Reddy Battula <
>>>>>>>> brahmareddy.battula@huawei.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Looks following link is not correct..
>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://s.apache.org/Dzg4
>>>>>>>>>
>>>>>>>>> It should be like following..?
>>>>>>>>>
>>>>>>>>> https://s.apache.org/wi3U
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Apart from Konstantin mentioned,Following also good to go..? let me
>>>>>>>>> know
>>>>>>>>> your thoughts on this.
>>>>>>>>>
>>>>>>>>> For Large Cluster:
>>>>>>>>> =============
>>>>>>>>>
>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9311===Life Line
>>>>>>>>> Protocol
>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-10987===Deecommission
>>>>>>>>> Expensive when lot's of blocks are present
>>>>>>>>>
>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9902===
>>>>>>>>> "dfs.datanode.du.reserved" per Storage Type
>>>>>>>>>
>>>>>>>>> For Security:
>>>>>>>>> =========
>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-8312===Trash does not
>>>>>>>>> descent
>>>>>>>>> into child directories to check for permission
>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-14100===Upgrade Jsch
>>>>>>>>> jar
>>>>>>>>> to
>>>>>>>>> latest version to fix vulnerability in old versions
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> Brahma Reddy Battula
>>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Erik Krogen [mailto:ekrogen@linkedin.com.INVALID]
>>>>>>>>> Sent: 06 May 2017 02:40
>>>>>>>>> To: Konstantin Shvachko
>>>>>>>>> Cc: Zhe Zhang; Hadoop Common; Hdfs-dev;
>>>>>>>>> mapreduce-dev@hadoop.apache.org
>>>>>>>>> ;
>>>>>>>>> yarn-dev@hadoop.apache.org
>>>>>>>>> Subject: Re: About 2.7.4 Release
>>>>>>>>>
>>>>>>>>> List LGTM Konstantin!
>>>>>>>>>
>>>>>>>>> Let's say that we will only create a new tracking JIRA for patches
>>>>>>>>> which
>>>>>>>>> do not backport cleanly, to avoid having too many lying around.
>>>>>>>>> Otherwise
>>>>>>>>> we can directly attach to old ticket. If a clean backport does
>>>>>>>>> happen
>>>>>>>>> to
>>>>>>>>> break a test the nightly build will help us catch it.
>>>>>>>>>
>>>>>>>>> Erik
>>>>>>>>>
>>>>>>>>> On Thu, May 4, 2017 at 7:21 PM, Konstantin Shvachko <
>>>>>>>>> shv.hadoop@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Great Zhe. Let's monitor the build.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> I marked all jiras I knew of for inclusion into 2.7.4 as I
>>>>>>>>>> described
>>>>>>>>>> before.
>>>>>>>>>> Target Version/s: 2.7.4
>>>>>>>>>> Label: release-blocker
>>>>>>>>>>
>>>>>>>>>> Here is the link to the list: https://s.apache.org/Dzg4 Please
>>>>>>>>>> let
>>>>>>>>>> me
>>>>>>>>>> know if I missed anything.
>>>>>>>>>> And feel free to pick up any. Most of backports are pretty
>>>>>>>>>> straightforward, but not all.
>>>>>>>>>>
>>>>>>>>>> We can create tracking jiras for backporting if you need to run
>>>>>>>>>> Jenkins on the patch (and since Allen does not allow reopening
>>>>>>>>>> them).
>>>>>>>>>> But I think the final patch should be attached to the original
>>>>>>>>>> jira.
>>>>>>>>>> Otherwise history will be hard to follow.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> --Konstantin
>>>>>>>>>>
>>>>>>>>>> On Wed, May 3, 2017 at 4:53 PM, Zhe Zhang <zh...@apache.org> wrote:
>>>>>>>>>>
>>>>>>>>>> Thanks for volunteering as RM Konstantin! The plan LGTM.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> I've created a nightly Jenkins job for branch-2.7 (unit tests):
>>>>>>>>>>> https://builds.apache.org/job/Hadoop-branch2.7-nightly/
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 3, 2017 at 12:42 AM Konstantin Shvachko <
>>>>>>>>>>>
>>>>>>>>>>> shv.hadoop@gmail.com>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hey guys,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> I and a few of my colleagues would like to help here and move
>>>>>>>>>>>> 2.7.4
>>>>>>>>>>>> release forward. A few points in this regard.
>>>>>>>>>>>>
>>>>>>>>>>>> 1. Reading through this thread since March 1 I see that Vinod
>>>>>>>>>>>> hinted on managing the release. Vinod, if you still want the
>>>>>>>>>>>> job /
>>>>>>>>>>>> have bandwidth will be happy to work with you.
>>>>>>>>>>>> Otherwise I am glad to volunteer as the release manager.
>>>>>>>>>>>>
>>>>>>>>>>>> 2. In addition to current blockers and criticals, I would like
>>>>>>>>>>>> to
>>>>>>>>>>>>
>>>>>>>>>>>> propose
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> a
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> few issues to be included in the release, see the list below.
>>>>>>>>>>>> Those
>>>>>>>>>>>> are mostly bug fixes and optimizations, which we already have in
>>>>>>>>>>>> our
>>>>>>>>>>>>
>>>>>>>>>>>> internal
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> branch and run in production. Plus one minor feature "node
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> labeling", which we found very handy, when you have heterogeneous
>>>>>>>>>>>> environments and mixed workloads, like MR and Spark.
>>>>>>>>>>>>
>>>>>>>>>>>> 3. For marking issues for the release I propose to
>>>>>>>>>>>> - set the target version to 2.7.4, and
>>>>>>>>>>>> - add a new label "release-blocker"
>>>>>>>>>>>> That way we will know issues targeted for the release without
>>>>>>>>>>>> reopening them for backports.
>>>>>>>>>>>>
>>>>>>>>>>>> 4. I see quite a few people are interested in the release. With
>>>>>>>>>>>> all
>>>>>>>>>>>> the help I think we can target to release by the end of May.
>>>>>>>>>>>>
>>>>>>>>>>>> Other things include fixing CHANGES.txt and fixing Jenkins build
>>>>>>>>>>>> for
>>>>>>>>>>>>
>>>>>>>>>>>> 2.7.4
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> branch.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> --Konstantin
>>>>>>>>>>>>
>>>>>>>>>>>> ========== List of issue for 2.7.4 ===========
>>>>>>>>>>>> ------ Backports
>>>>>>>>>>>> HADOOP-12975 <https://issues.apache.org/jir
>>>>>>>>>>>> a/browse/HADOOP-12975>.
>>>>>>>>>>>> Add
>>>>>>>>>>>>
>>>>>>>>>>>> du
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> jitters
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> HDFS-9710 <https://issues.apache.org/jira/browse/HDFS-9710>. IBR
>>>>>>>>>>>>
>>>>>>>>>>>> batching
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> HDFS-10715 <https://issues.apache.org/jira/browse/HDFS-10715>.
>>>>>>>>>> NPE
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> when applying AvailableSpaceBlockPlacementPolicy
>>>>>>>>>>>> HDFS-2538 <https://issues.apache.org/jira/browse/HDFS-2538>.
>>>>>>>>>>>> fsck
>>>>>>>>>>>>
>>>>>>>>>>>> removal
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> of dot printing
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> HDFS-8131 <https://issues.apache.org/jira/browse/HDFS-8131>.
>>>>>>>>>>>> space-balanced
>>>>>>>>>>>> policy for balancer
>>>>>>>>>>>> HDFS-8549 <https://issues.apache.org/jira/browse/HDFS-8549>.
>>>>>>>>>>>> abort
>>>>>>>>>>>> balancer if upgrade in progress
>>>>>>>>>>>> HDFS-9412 <https://issues.apache.org/jira/browse/HDFS-9412>.
>>>>>>>>>>>> skip
>>>>>>>>>>>> small blocks in getBlocks
>>>>>>>>>>>>
>>>>>>>>>>>> YARN-1471 <https://issues.apache.org/jira/browse/YARN-1471>.
>>>>>>>>>>>> SLS
>>>>>>>>>>>> simulator
>>>>>>>>>>>> YARN-4302 <https://issues.apache.org/jira/browse/YARN-4302>.
>>>>>>>>>>>> SLS
>>>>>>>>>>>> YARN-4367 <https://issues.apache.org/jira/browse/YARN-4367>.
>>>>>>>>>>>> SLS
>>>>>>>>>>>> YARN-4612 <https://issues.apache.org/jira/browse/YARN-4612>.
>>>>>>>>>>>> SLS
>>>>>>>>>>>>
>>>>>>>>>>>> ----- Node labeling
>>>>>>>>>>>> MAPREDUCE-6304
>>>>>>>>>>>> <https://issues.apache.org/jira/browse/MAPREDUCE-6304>
>>>>>>>>>>>> YARN-2943 <https://issues.apache.org/jira/browse/YARN-2943>
>>>>>>>>>>>> YARN-4109 <https://issues.apache.org/jira/browse/YARN-4109>
>>>>>>>>>>>> YARN-4140 <https://issues.apache.org/jira/browse/YARN-4140>
>>>>>>>>>>>> YARN-4250 <https://issues.apache.org/jira/browse/YARN-4250>
>>>>>>>>>>>> YARN-4925 <https://issues.apache.org/jira/browse/YARN-4925>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>>
>>>>>>>>>>>> Zhe Zhang
>>>>>>>>>>> Apache Hadoop Committer
>>>>>>>>>>> http://zhe-thoughts.github.io/about/ | @oldcap
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>
>>>>
>>>>
>>>
>