Please cherry pick commits to lower branches

[Wei-Chiu Chuang <we...@apache.org>]

I spent the whole last week cherry picking commits from trunk/branch-3.2 to
branch-3.1 (should've done this prior to 3.1.4 code freeze). There were
about 50-60 of them, many of them are conflict-free, and several of them
are critical bug fixes.

If your commit stays in trunk, it'll be useless for the community until the
next minor release, and many months after people start using the new
release.

Here are a few tips:
(1) update dependency to address a know security vulnerability, should be
cherry picked into all lower branches, especially when it updates the
maintenance release number. Example: update commons-compress from 1.18 to
1.19.

(2) blocker/critical bug fixes should be backported to all applicable
branches.

(3) because of the removal of commons-logging and a few code refactors,
commits may apply cleanly but doesn't compile in branch-3.2, branch-3.1 and
lower branches. Please spend the time to verify a commit is good.

Best
Weichiu

Re: Please cherry pick commits to lower branches

[Steve Loughran <st...@cloudera.com.INVALID>]

created https://issues.apache.org/jira/browse/HADOOP-16646 for the s3a
stuff.

branch-3.2 doesn't build right now, someone needs to fix that first; I'll
start running the full ITest suites off a commit which does build...

On Wed, Oct 9, 2019 at 1:51 PM Steve Loughran <st...@cloudera.com> wrote:

>
> abfs stuff has gone in to 3.2 -credit Thomas Marquardt and Da Zhou there.
>
> I'll do the same for S3A, including SDK updates
> And I'd like to pull back two changes to the FS APIs
>
>
> HADOOP-15691 PathCapabilities. Lets apps probes for FS instances having
> specific features/semantics
> HADOOP-15229 Add FS/FC builder open() API.
>
> making the apis broadly available, even if any tuning in the object stores
> is left out, makes it possible to adopt.
>
> One thing to highlight which has cause problems on some backporting is the
> mockito update https://issues.apache.org/jira/browse/HADOOP-16275
>
> without that mock tests can still compile -but they go on to fail.I think
> we will need to put that into 3.2 at the very least. Same for SLF4J
>
> backporting big patches to 3.1.x is harder because of the moves to slf4j
> spanning things: you may not have merge conflicts but things don't compile.
>
>
> On Tue, Oct 8, 2019 at 10:15 AM Wei-Chiu Chuang <we...@apache.org>
> wrote:
>
>> I spent the whole last week cherry picking commits from trunk/branch-3.2
>> to
>> branch-3.1 (should've done this prior to 3.1.4 code freeze). There were
>> about 50-60 of them, many of them are conflict-free, and several of them
>> are critical bug fixes.
>>
>> If your commit stays in trunk, it'll be useless for the community until
>> the
>> next minor release, and many months after people start using the new
>> release.
>>
>> Here are a few tips:
>> (1) update dependency to address a know security vulnerability, should be
>> cherry picked into all lower branches, especially when it updates the
>> maintenance release number. Example: update commons-compress from 1.18 to
>> 1.19.
>>
>> (2) blocker/critical bug fixes should be backported to all applicable
>> branches.
>>
>> (3) because of the removal of commons-logging and a few code refactors,
>> commits may apply cleanly but doesn't compile in branch-3.2, branch-3.1
>> and
>> lower branches. Please spend the time to verify a commit is good.
>>
>> Best
>> Weichiu
>>
>

Re: Please cherry pick commits to lower branches

[Steve Loughran <st...@cloudera.com.INVALID>]

created https://issues.apache.org/jira/browse/HADOOP-16646 for the s3a
stuff.

branch-3.2 doesn't build right now, someone needs to fix that first; I'll
start running the full ITest suites off a commit which does build...

On Wed, Oct 9, 2019 at 1:51 PM Steve Loughran <st...@cloudera.com> wrote:

>
> abfs stuff has gone in to 3.2 -credit Thomas Marquardt and Da Zhou there.
>
> I'll do the same for S3A, including SDK updates
> And I'd like to pull back two changes to the FS APIs
>
>
> HADOOP-15691 PathCapabilities. Lets apps probes for FS instances having
> specific features/semantics
> HADOOP-15229 Add FS/FC builder open() API.
>
> making the apis broadly available, even if any tuning in the object stores
> is left out, makes it possible to adopt.
>
> One thing to highlight which has cause problems on some backporting is the
> mockito update https://issues.apache.org/jira/browse/HADOOP-16275
>
> without that mock tests can still compile -but they go on to fail.I think
> we will need to put that into 3.2 at the very least. Same for SLF4J
>
> backporting big patches to 3.1.x is harder because of the moves to slf4j
> spanning things: you may not have merge conflicts but things don't compile.
>
>
> On Tue, Oct 8, 2019 at 10:15 AM Wei-Chiu Chuang <we...@apache.org>
> wrote:
>
>> I spent the whole last week cherry picking commits from trunk/branch-3.2
>> to
>> branch-3.1 (should've done this prior to 3.1.4 code freeze). There were
>> about 50-60 of them, many of them are conflict-free, and several of them
>> are critical bug fixes.
>>
>> If your commit stays in trunk, it'll be useless for the community until
>> the
>> next minor release, and many months after people start using the new
>> release.
>>
>> Here are a few tips:
>> (1) update dependency to address a know security vulnerability, should be
>> cherry picked into all lower branches, especially when it updates the
>> maintenance release number. Example: update commons-compress from 1.18 to
>> 1.19.
>>
>> (2) blocker/critical bug fixes should be backported to all applicable
>> branches.
>>
>> (3) because of the removal of commons-logging and a few code refactors,
>> commits may apply cleanly but doesn't compile in branch-3.2, branch-3.1
>> and
>> lower branches. Please spend the time to verify a commit is good.
>>
>> Best
>> Weichiu
>>
>

Re: Please cherry pick commits to lower branches

[Steve Loughran <st...@cloudera.com.INVALID>]

created https://issues.apache.org/jira/browse/HADOOP-16646 for the s3a
stuff.

branch-3.2 doesn't build right now, someone needs to fix that first; I'll
start running the full ITest suites off a commit which does build...

On Wed, Oct 9, 2019 at 1:51 PM Steve Loughran <st...@cloudera.com> wrote:

>
> abfs stuff has gone in to 3.2 -credit Thomas Marquardt and Da Zhou there.
>
> I'll do the same for S3A, including SDK updates
> And I'd like to pull back two changes to the FS APIs
>
>
> HADOOP-15691 PathCapabilities. Lets apps probes for FS instances having
> specific features/semantics
> HADOOP-15229 Add FS/FC builder open() API.
>
> making the apis broadly available, even if any tuning in the object stores
> is left out, makes it possible to adopt.
>
> One thing to highlight which has cause problems on some backporting is the
> mockito update https://issues.apache.org/jira/browse/HADOOP-16275
>
> without that mock tests can still compile -but they go on to fail.I think
> we will need to put that into 3.2 at the very least. Same for SLF4J
>
> backporting big patches to 3.1.x is harder because of the moves to slf4j
> spanning things: you may not have merge conflicts but things don't compile.
>
>
> On Tue, Oct 8, 2019 at 10:15 AM Wei-Chiu Chuang <we...@apache.org>
> wrote:
>
>> I spent the whole last week cherry picking commits from trunk/branch-3.2
>> to
>> branch-3.1 (should've done this prior to 3.1.4 code freeze). There were
>> about 50-60 of them, many of them are conflict-free, and several of them
>> are critical bug fixes.
>>
>> If your commit stays in trunk, it'll be useless for the community until
>> the
>> next minor release, and many months after people start using the new
>> release.
>>
>> Here are a few tips:
>> (1) update dependency to address a know security vulnerability, should be
>> cherry picked into all lower branches, especially when it updates the
>> maintenance release number. Example: update commons-compress from 1.18 to
>> 1.19.
>>
>> (2) blocker/critical bug fixes should be backported to all applicable
>> branches.
>>
>> (3) because of the removal of commons-logging and a few code refactors,
>> commits may apply cleanly but doesn't compile in branch-3.2, branch-3.1
>> and
>> lower branches. Please spend the time to verify a commit is good.
>>
>> Best
>> Weichiu
>>
>

Re: Please cherry pick commits to lower branches

[Steve Loughran <st...@cloudera.com.INVALID>]

abfs stuff has gone in to 3.2 -credit Thomas Marquardt and Da Zhou there.

I'll do the same for S3A, including SDK updates
And I'd like to pull back two changes to the FS APIs


HADOOP-15691 PathCapabilities. Lets apps probes for FS instances having
specific features/semantics
HADOOP-15229 Add FS/FC builder open() API.

making the apis broadly available, even if any tuning in the object stores
is left out, makes it possible to adopt.

One thing to highlight which has cause problems on some backporting is the
mockito update https://issues.apache.org/jira/browse/HADOOP-16275

without that mock tests can still compile -but they go on to fail.I think
we will need to put that into 3.2 at the very least. Same for SLF4J

backporting big patches to 3.1.x is harder because of the moves to slf4j
spanning things: you may not have merge conflicts but things don't compile.


On Tue, Oct 8, 2019 at 10:15 AM Wei-Chiu Chuang <we...@apache.org> wrote:

> I spent the whole last week cherry picking commits from trunk/branch-3.2 to
> branch-3.1 (should've done this prior to 3.1.4 code freeze). There were
> about 50-60 of them, many of them are conflict-free, and several of them
> are critical bug fixes.
>
> If your commit stays in trunk, it'll be useless for the community until the
> next minor release, and many months after people start using the new
> release.
>
> Here are a few tips:
> (1) update dependency to address a know security vulnerability, should be
> cherry picked into all lower branches, especially when it updates the
> maintenance release number. Example: update commons-compress from 1.18 to
> 1.19.
>
> (2) blocker/critical bug fixes should be backported to all applicable
> branches.
>
> (3) because of the removal of commons-logging and a few code refactors,
> commits may apply cleanly but doesn't compile in branch-3.2, branch-3.1 and
> lower branches. Please spend the time to verify a commit is good.
>
> Best
> Weichiu
>

Re: Please cherry pick commits to lower branches

[Steve Loughran <st...@cloudera.com.INVALID>]

abfs stuff has gone in to 3.2 -credit Thomas Marquardt and Da Zhou there.

I'll do the same for S3A, including SDK updates
And I'd like to pull back two changes to the FS APIs


HADOOP-15691 PathCapabilities. Lets apps probes for FS instances having
specific features/semantics
HADOOP-15229 Add FS/FC builder open() API.

making the apis broadly available, even if any tuning in the object stores
is left out, makes it possible to adopt.

One thing to highlight which has cause problems on some backporting is the
mockito update https://issues.apache.org/jira/browse/HADOOP-16275

without that mock tests can still compile -but they go on to fail.I think
we will need to put that into 3.2 at the very least. Same for SLF4J

backporting big patches to 3.1.x is harder because of the moves to slf4j
spanning things: you may not have merge conflicts but things don't compile.


On Tue, Oct 8, 2019 at 10:15 AM Wei-Chiu Chuang <we...@apache.org> wrote:

> I spent the whole last week cherry picking commits from trunk/branch-3.2 to
> branch-3.1 (should've done this prior to 3.1.4 code freeze). There were
> about 50-60 of them, many of them are conflict-free, and several of them
> are critical bug fixes.
>
> If your commit stays in trunk, it'll be useless for the community until the
> next minor release, and many months after people start using the new
> release.
>
> Here are a few tips:
> (1) update dependency to address a know security vulnerability, should be
> cherry picked into all lower branches, especially when it updates the
> maintenance release number. Example: update commons-compress from 1.18 to
> 1.19.
>
> (2) blocker/critical bug fixes should be backported to all applicable
> branches.
>
> (3) because of the removal of commons-logging and a few code refactors,
> commits may apply cleanly but doesn't compile in branch-3.2, branch-3.1 and
> lower branches. Please spend the time to verify a commit is good.
>
> Best
> Weichiu
>

Re: Please cherry pick commits to lower branches

[Steve Loughran <st...@cloudera.com.INVALID>]

abfs stuff has gone in to 3.2 -credit Thomas Marquardt and Da Zhou there.

I'll do the same for S3A, including SDK updates
And I'd like to pull back two changes to the FS APIs


HADOOP-15691 PathCapabilities. Lets apps probes for FS instances having
specific features/semantics
HADOOP-15229 Add FS/FC builder open() API.

making the apis broadly available, even if any tuning in the object stores
is left out, makes it possible to adopt.

One thing to highlight which has cause problems on some backporting is the
mockito update https://issues.apache.org/jira/browse/HADOOP-16275

without that mock tests can still compile -but they go on to fail.I think
we will need to put that into 3.2 at the very least. Same for SLF4J

backporting big patches to 3.1.x is harder because of the moves to slf4j
spanning things: you may not have merge conflicts but things don't compile.


On Tue, Oct 8, 2019 at 10:15 AM Wei-Chiu Chuang <we...@apache.org> wrote:

> I spent the whole last week cherry picking commits from trunk/branch-3.2 to
> branch-3.1 (should've done this prior to 3.1.4 code freeze). There were
> about 50-60 of them, many of them are conflict-free, and several of them
> are critical bug fixes.
>
> If your commit stays in trunk, it'll be useless for the community until the
> next minor release, and many months after people start using the new
> release.
>
> Here are a few tips:
> (1) update dependency to address a know security vulnerability, should be
> cherry picked into all lower branches, especially when it updates the
> maintenance release number. Example: update commons-compress from 1.18 to
> 1.19.
>
> (2) blocker/critical bug fixes should be backported to all applicable
> branches.
>
> (3) because of the removal of commons-logging and a few code refactors,
> commits may apply cleanly but doesn't compile in branch-3.2, branch-3.1 and
> lower branches. Please spend the time to verify a commit is good.
>
> Best
> Weichiu
>