You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-dev@hadoop.apache.org by Wei-Chiu Chuang <we...@apache.org> on 2023/05/04 17:19:00 UTC

[DISCUSS] Hadoop 3.3.6 release planning

Hi community,

I'd like to kick off the discussion around Hadoop 3.3.6 release plan.

I'm being selfish but my intent for 3.3.6 is to have the new APIs in
HADOOP-18671 <https://issues.apache.org/jira/browse/HADOOP-18671> added so
we can have HBase to adopt this new API. Other than that, perhaps
thirdparty dependency updates.

If you have open items to be added in the coming weeks, please add 3.3.6 to
the target release version. Right now I am only seeing three open jiras
targeting 3.3.6.

I imagine this is going to be a small release as 3.3.5 (hat tip to Steve)
was only made two months back, and so far only 8 jiras were resolved in the
branch-3.3 line.

Best,
Weichiu

Re: [DISCUSS] Hadoop 3.3.6 release planning

Posted by Wei-Chiu Chuang <we...@apache.org>.

Thanks a lot for the help to move this release forward.

Some status update:

branch-3.3.6 just forked out of branch-3.3.
Starting a RC0 now.

I would like to add the text to highlight big features and improvements.
Let me know if you have something that's included in this release and would
also like to highlight. @Steve Loughran <st...@apache.org> I am aware the
S3A prefetch code is in this release, but I am not sure if it is in a state
where we can make it public. I'll let you decide.

SBOM artifacts
----------------------------------------
Starting from this release, Hadoop publishes Software Bill of Materials
(SBOM) using
CycloneDX Maven plugin. For more information about SBOM, please go to
[SBOM](https://cwiki.apache.org/confluence/display/COMDEV/SBOM).

HDFS RBF: RDBMS based token storage support
----------------------------------------
HDFS Router-Router Based Federation now supports storing delegation tokens
on MySQL,
[HADOOP-18535](https://issues.apache.org/jira/browse/HADOOP-18535)
which improves token operation through over the original Zookeeper-based
implementation.


New File System APIs
----------------------------------------
[HADOOP-18671](https://issues.apache.org/jira/browse/HADOOP-18671) moved a
number of
HDFS-specific APIs to Hadoop Common to make it possible for certain
applications that
depend on HDFS semantics to run on other Hadoop compatible file systems.

In particular, recoverLease() and isFileClosed() are exposed through
LeaseRecoverable
interface. While setSafeMode() is exposed through SafeMode interface.



On Thu, Jun 8, 2023 at 11:11 AM Wei-Chiu Chuang <we...@apache.org> wrote:

> Thanks for comments
>
> Looking at jiras fixed in 3.3.6 and 3.3.9 (my bad, forgot that most
> commits landing in branch-3.3 was 3.3.9), most are okay. We have about 119
> commits so it's manageable.
>
> I am planning to cut 3.3.6 out of branch-3.3 later today. Anything open
> that is still targeting 3.3.6 will be cherry picked one by one.
> I will also bulk-update jiras fixed in 3.3.9 to 3.3.6.
>
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12337047
> As of now, I am tracking 4 jiras that's targeting 3.3.6 -- update Kerby,
> fix hadoop shaded client to support Spark history server, a small
> regression in HDFS (probably will move this out), protobuf 2.5 dependency
> change.
>
> I had a dry-run of the RC. Env set up for me and everything works. So I
> expect to have a RC ready to vote on soon. If I can move out some of the
> jiras or help them resolved, I can probably have a RC0 on Monday for vote.
>
> On Mon, May 8, 2023 at 2:01 PM Ayush Saxena <ay...@gmail.com> wrote:
>
>> That openssl change ain't a blocker now from my side, that ABFS-Jdk-17
>> stuff got sorted out, Steve knew a way out....
>>
>> On Sat, 6 May 2023 at 00:51, Ayush Saxena <ay...@gmail.com> wrote:
>> >
>> > Thanx Wei-Chiu for the initiative, Good to have quick releases :)
>> >
>> > With my Hive developer hat on, I would like to bring some stuff up for
>> > consideration(feel free to say no, if it is beyond scope or feels even
>> > a bit unsafe, don't want to mess up the release)
>> >
>> > * HADOOP-18662: ListFiles with recursive fails with FNF : This broke
>> > compaction in Hive, bothers only with HDFS though. There is a
>> > workaround to that, if it doesn't feel safe. no issues, or if some
>> > improvements suggested. I can quickly do that :)
>> >
>> > * HADOOP-17649: Update wildfly openssl to 2.1.3.Final. Maybe not 2.1.3
>> > but if it works and is safe then to 2.2.5. I got flagged today that
>> > this openssl creates a bit of mess with JDK-17 for Hive with ABFS I
>> > think(need to dig in more),
>> >
>> > Now for the dependency upgrades:
>> >
>> > A big NO to Jackson, that ain't safe and the wounds are still fresh,
>> > it screwed the 3.3.3 release for many projects. So, let's not get into
>> > that. Infact anything that touches those shaded jars is risky, some
>> > package-json exclusion also created a mess recently. So, Lets not
>> > touch only and that too when we have less time.
>> >
>> > Avoid anything around Jetty upgrade, I have selfish reasons for that.
>> > Jetty messes something up with Hbase and Hive has a dependency on
>> > Hbase, and it is crazy, in case interested [1]. So, any upgrade to
>> > Jetty will block hive from upgrading Hadoop as of today. But that is a
>> > selfish reason and just up for consideration. Go ahead if necessary. I
>> > just wanted to let folks know
>> >
>> >
>> > Apart from the Jackson stuff, everything is suggestive in nature, your
>> > call feel free to ignore.
>> >
>> > @Xiaoqiao He , maybe pulling in all those 100+ would be risky,
>> > considering the timelines, but if we find a few fancy safe tickets,
>> > maybe if you have identified any already, can put them up on this
>> > thread and if folks are convinced. We can get them in? Juzz my
>> > thoughts, it is up to you and Wei-Chiu, (No skin in the game opinion)
>> >
>> >
>> > Goooood Luck!!!!
>> >
>> > -Ayush
>> >
>> > [1] https://github.com/apache/hive/pull/4290#issuecomment-1536553803
>> >
>> > On Fri, 5 May 2023 at 16:13, Steve Loughran <st...@cloudera.com.invalid>
>> wrote:
>> > >
>> > > Wei-Chiu has suggested a minimal "things in 3.3.5 which were very
>> broken,
>> > > api change for ozone and any critical jar updates"
>> > >
>> > > so much lower risk/easier to qualify and ship.
>> > >
>> > > I need to get https://issues.apache.org/jira/browse/HADOOP-18724 in
>> here;
>> > > maybe look at a refresh of the "classic" jars (slf4j, reload,
>> jackson*,
>> > > remove json-smart...)
>> > >
>> > > I'd also like to downgrade protobuf 2.5 from required to optional;
>> even
>> > > though hadoop uses the shaded one, to support hbase etc the IPC code
>> still
>> > > has direct use of the 2.5 classes. that coud be optional
>> > >
>> > > if anyone wants to take up this PR, I would be very happy
>> > > https://github.com/apache/hadoop/pull/4996
>> > >
>> > > On Fri, 5 May 2023 at 04:27, Xiaoqiao He <he...@apache.org>
>> wrote:
>> > >
>> > > > Thanks Wei-Chiu for driving this release.
>> > > > Cherry-pick YARN-11482 to branch-3.3 and mark 3.3.6 as the fixed
>> version.
>> > > >
>> > > > so far only 8 jiras were resolved in the branch-3.3 line.
>> > > >
>> > > >
>> > > > If we should consider both 3.3.6 and 3.3.9 (which is from
>> release-3.3.5
>> > > > discuss)[1] for this release line?
>> > > > I try to query with `project in (HDFS, YARN, HADOOP, MAPREDUCE) AND
>> > > > fixVersion in (3.3.6, 3.3.9)`[2],
>> > > > there are more than hundred jiras now.
>> > > >
>> > > > Best Regards,
>> > > > - He Xiaoqiao
>> > > >
>> > > > [1]
>> https://lists.apache.org/thread/kln96frt2tcg93x6ht99yck9m7r9qwxp
>> > > > [2]
>> > > >
>> > > >
>> https://issues.apache.org/jira/browse/YARN-11482?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20fixVersion%20in%20(3.3.6%2C%203.3.9)
>> > > >
>> > > >
>> > > > On Fri, May 5, 2023 at 1:19 AM Wei-Chiu Chuang <we...@apache.org>
>> wrote:
>> > > >
>> > > > > Hi community,
>> > > > >
>> > > > > I'd like to kick off the discussion around Hadoop 3.3.6 release
>> plan.
>> > > > >
>> > > > > I'm being selfish but my intent for 3.3.6 is to have the new APIs
>> in
>> > > > > HADOOP-18671 <https://issues.apache.org/jira/browse/HADOOP-18671>
>> added
>> > > > so
>> > > > > we can have HBase to adopt this new API. Other than that, perhaps
>> > > > > thirdparty dependency updates.
>> > > > >
>> > > > > If you have open items to be added in the coming weeks, please
>> add 3.3.6
>> > > > to
>> > > > > the target release version. Right now I am only seeing three open
>> jiras
>> > > > > targeting 3.3.6.
>> > > > >
>> > > > > I imagine this is going to be a small release as 3.3.5 (hat tip
>> to Steve)
>> > > > > was only made two months back, and so far only 8 jiras were
>> resolved in
>> > > > the
>> > > > > branch-3.3 line.
>> > > > >
>> > > > > Best,
>> > > > > Weichiu
>> > > > >
>> > > >
>>
>

Re: [DISCUSS] Hadoop 3.3.6 release planning

Posted by Wei-Chiu Chuang <we...@apache.org>.

Thanks a lot for the help to move this release forward.

Some status update:

branch-3.3.6 just forked out of branch-3.3.
Starting a RC0 now.

I would like to add the text to highlight big features and improvements.
Let me know if you have something that's included in this release and would
also like to highlight. @Steve Loughran <st...@apache.org> I am aware the
S3A prefetch code is in this release, but I am not sure if it is in a state
where we can make it public. I'll let you decide.

SBOM artifacts
----------------------------------------
Starting from this release, Hadoop publishes Software Bill of Materials
(SBOM) using
CycloneDX Maven plugin. For more information about SBOM, please go to
[SBOM](https://cwiki.apache.org/confluence/display/COMDEV/SBOM).

HDFS RBF: RDBMS based token storage support
----------------------------------------
HDFS Router-Router Based Federation now supports storing delegation tokens
on MySQL,
[HADOOP-18535](https://issues.apache.org/jira/browse/HADOOP-18535)
which improves token operation through over the original Zookeeper-based
implementation.


New File System APIs
----------------------------------------
[HADOOP-18671](https://issues.apache.org/jira/browse/HADOOP-18671) moved a
number of
HDFS-specific APIs to Hadoop Common to make it possible for certain
applications that
depend on HDFS semantics to run on other Hadoop compatible file systems.

In particular, recoverLease() and isFileClosed() are exposed through
LeaseRecoverable
interface. While setSafeMode() is exposed through SafeMode interface.



On Thu, Jun 8, 2023 at 11:11 AM Wei-Chiu Chuang <we...@apache.org> wrote:

> Thanks for comments
>
> Looking at jiras fixed in 3.3.6 and 3.3.9 (my bad, forgot that most
> commits landing in branch-3.3 was 3.3.9), most are okay. We have about 119
> commits so it's manageable.
>
> I am planning to cut 3.3.6 out of branch-3.3 later today. Anything open
> that is still targeting 3.3.6 will be cherry picked one by one.
> I will also bulk-update jiras fixed in 3.3.9 to 3.3.6.
>
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12337047
> As of now, I am tracking 4 jiras that's targeting 3.3.6 -- update Kerby,
> fix hadoop shaded client to support Spark history server, a small
> regression in HDFS (probably will move this out), protobuf 2.5 dependency
> change.
>
> I had a dry-run of the RC. Env set up for me and everything works. So I
> expect to have a RC ready to vote on soon. If I can move out some of the
> jiras or help them resolved, I can probably have a RC0 on Monday for vote.
>
> On Mon, May 8, 2023 at 2:01 PM Ayush Saxena <ay...@gmail.com> wrote:
>
>> That openssl change ain't a blocker now from my side, that ABFS-Jdk-17
>> stuff got sorted out, Steve knew a way out....
>>
>> On Sat, 6 May 2023 at 00:51, Ayush Saxena <ay...@gmail.com> wrote:
>> >
>> > Thanx Wei-Chiu for the initiative, Good to have quick releases :)
>> >
>> > With my Hive developer hat on, I would like to bring some stuff up for
>> > consideration(feel free to say no, if it is beyond scope or feels even
>> > a bit unsafe, don't want to mess up the release)
>> >
>> > * HADOOP-18662: ListFiles with recursive fails with FNF : This broke
>> > compaction in Hive, bothers only with HDFS though. There is a
>> > workaround to that, if it doesn't feel safe. no issues, or if some
>> > improvements suggested. I can quickly do that :)
>> >
>> > * HADOOP-17649: Update wildfly openssl to 2.1.3.Final. Maybe not 2.1.3
>> > but if it works and is safe then to 2.2.5. I got flagged today that
>> > this openssl creates a bit of mess with JDK-17 for Hive with ABFS I
>> > think(need to dig in more),
>> >
>> > Now for the dependency upgrades:
>> >
>> > A big NO to Jackson, that ain't safe and the wounds are still fresh,
>> > it screwed the 3.3.3 release for many projects. So, let's not get into
>> > that. Infact anything that touches those shaded jars is risky, some
>> > package-json exclusion also created a mess recently. So, Lets not
>> > touch only and that too when we have less time.
>> >
>> > Avoid anything around Jetty upgrade, I have selfish reasons for that.
>> > Jetty messes something up with Hbase and Hive has a dependency on
>> > Hbase, and it is crazy, in case interested [1]. So, any upgrade to
>> > Jetty will block hive from upgrading Hadoop as of today. But that is a
>> > selfish reason and just up for consideration. Go ahead if necessary. I
>> > just wanted to let folks know
>> >
>> >
>> > Apart from the Jackson stuff, everything is suggestive in nature, your
>> > call feel free to ignore.
>> >
>> > @Xiaoqiao He , maybe pulling in all those 100+ would be risky,
>> > considering the timelines, but if we find a few fancy safe tickets,
>> > maybe if you have identified any already, can put them up on this
>> > thread and if folks are convinced. We can get them in? Juzz my
>> > thoughts, it is up to you and Wei-Chiu, (No skin in the game opinion)
>> >
>> >
>> > Goooood Luck!!!!
>> >
>> > -Ayush
>> >
>> > [1] https://github.com/apache/hive/pull/4290#issuecomment-1536553803
>> >
>> > On Fri, 5 May 2023 at 16:13, Steve Loughran <st...@cloudera.com.invalid>
>> wrote:
>> > >
>> > > Wei-Chiu has suggested a minimal "things in 3.3.5 which were very
>> broken,
>> > > api change for ozone and any critical jar updates"
>> > >
>> > > so much lower risk/easier to qualify and ship.
>> > >
>> > > I need to get https://issues.apache.org/jira/browse/HADOOP-18724 in
>> here;
>> > > maybe look at a refresh of the "classic" jars (slf4j, reload,
>> jackson*,
>> > > remove json-smart...)
>> > >
>> > > I'd also like to downgrade protobuf 2.5 from required to optional;
>> even
>> > > though hadoop uses the shaded one, to support hbase etc the IPC code
>> still
>> > > has direct use of the 2.5 classes. that coud be optional
>> > >
>> > > if anyone wants to take up this PR, I would be very happy
>> > > https://github.com/apache/hadoop/pull/4996
>> > >
>> > > On Fri, 5 May 2023 at 04:27, Xiaoqiao He <he...@apache.org>
>> wrote:
>> > >
>> > > > Thanks Wei-Chiu for driving this release.
>> > > > Cherry-pick YARN-11482 to branch-3.3 and mark 3.3.6 as the fixed
>> version.
>> > > >
>> > > > so far only 8 jiras were resolved in the branch-3.3 line.
>> > > >
>> > > >
>> > > > If we should consider both 3.3.6 and 3.3.9 (which is from
>> release-3.3.5
>> > > > discuss)[1] for this release line?
>> > > > I try to query with `project in (HDFS, YARN, HADOOP, MAPREDUCE) AND
>> > > > fixVersion in (3.3.6, 3.3.9)`[2],
>> > > > there are more than hundred jiras now.
>> > > >
>> > > > Best Regards,
>> > > > - He Xiaoqiao
>> > > >
>> > > > [1]
>> https://lists.apache.org/thread/kln96frt2tcg93x6ht99yck9m7r9qwxp
>> > > > [2]
>> > > >
>> > > >
>> https://issues.apache.org/jira/browse/YARN-11482?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20fixVersion%20in%20(3.3.6%2C%203.3.9)
>> > > >
>> > > >
>> > > > On Fri, May 5, 2023 at 1:19 AM Wei-Chiu Chuang <we...@apache.org>
>> wrote:
>> > > >
>> > > > > Hi community,
>> > > > >
>> > > > > I'd like to kick off the discussion around Hadoop 3.3.6 release
>> plan.
>> > > > >
>> > > > > I'm being selfish but my intent for 3.3.6 is to have the new APIs
>> in
>> > > > > HADOOP-18671 <https://issues.apache.org/jira/browse/HADOOP-18671>
>> added
>> > > > so
>> > > > > we can have HBase to adopt this new API. Other than that, perhaps
>> > > > > thirdparty dependency updates.
>> > > > >
>> > > > > If you have open items to be added in the coming weeks, please
>> add 3.3.6
>> > > > to
>> > > > > the target release version. Right now I am only seeing three open
>> jiras
>> > > > > targeting 3.3.6.
>> > > > >
>> > > > > I imagine this is going to be a small release as 3.3.5 (hat tip
>> to Steve)
>> > > > > was only made two months back, and so far only 8 jiras were
>> resolved in
>> > > > the
>> > > > > branch-3.3 line.
>> > > > >
>> > > > > Best,
>> > > > > Weichiu
>> > > > >
>> > > >
>>
>

Re: [DISCUSS] Hadoop 3.3.6 release planning

Posted by Wei-Chiu Chuang <we...@apache.org>.

Thanks a lot for the help to move this release forward.

Some status update:

branch-3.3.6 just forked out of branch-3.3.
Starting a RC0 now.

I would like to add the text to highlight big features and improvements.
Let me know if you have something that's included in this release and would
also like to highlight. @Steve Loughran <st...@apache.org> I am aware the
S3A prefetch code is in this release, but I am not sure if it is in a state
where we can make it public. I'll let you decide.

SBOM artifacts
----------------------------------------
Starting from this release, Hadoop publishes Software Bill of Materials
(SBOM) using
CycloneDX Maven plugin. For more information about SBOM, please go to
[SBOM](https://cwiki.apache.org/confluence/display/COMDEV/SBOM).

HDFS RBF: RDBMS based token storage support
----------------------------------------
HDFS Router-Router Based Federation now supports storing delegation tokens
on MySQL,
[HADOOP-18535](https://issues.apache.org/jira/browse/HADOOP-18535)
which improves token operation through over the original Zookeeper-based
implementation.


New File System APIs
----------------------------------------
[HADOOP-18671](https://issues.apache.org/jira/browse/HADOOP-18671) moved a
number of
HDFS-specific APIs to Hadoop Common to make it possible for certain
applications that
depend on HDFS semantics to run on other Hadoop compatible file systems.

In particular, recoverLease() and isFileClosed() are exposed through
LeaseRecoverable
interface. While setSafeMode() is exposed through SafeMode interface.



On Thu, Jun 8, 2023 at 11:11 AM Wei-Chiu Chuang <we...@apache.org> wrote:

> Thanks for comments
>
> Looking at jiras fixed in 3.3.6 and 3.3.9 (my bad, forgot that most
> commits landing in branch-3.3 was 3.3.9), most are okay. We have about 119
> commits so it's manageable.
>
> I am planning to cut 3.3.6 out of branch-3.3 later today. Anything open
> that is still targeting 3.3.6 will be cherry picked one by one.
> I will also bulk-update jiras fixed in 3.3.9 to 3.3.6.
>
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12337047
> As of now, I am tracking 4 jiras that's targeting 3.3.6 -- update Kerby,
> fix hadoop shaded client to support Spark history server, a small
> regression in HDFS (probably will move this out), protobuf 2.5 dependency
> change.
>
> I had a dry-run of the RC. Env set up for me and everything works. So I
> expect to have a RC ready to vote on soon. If I can move out some of the
> jiras or help them resolved, I can probably have a RC0 on Monday for vote.
>
> On Mon, May 8, 2023 at 2:01 PM Ayush Saxena <ay...@gmail.com> wrote:
>
>> That openssl change ain't a blocker now from my side, that ABFS-Jdk-17
>> stuff got sorted out, Steve knew a way out....
>>
>> On Sat, 6 May 2023 at 00:51, Ayush Saxena <ay...@gmail.com> wrote:
>> >
>> > Thanx Wei-Chiu for the initiative, Good to have quick releases :)
>> >
>> > With my Hive developer hat on, I would like to bring some stuff up for
>> > consideration(feel free to say no, if it is beyond scope or feels even
>> > a bit unsafe, don't want to mess up the release)
>> >
>> > * HADOOP-18662: ListFiles with recursive fails with FNF : This broke
>> > compaction in Hive, bothers only with HDFS though. There is a
>> > workaround to that, if it doesn't feel safe. no issues, or if some
>> > improvements suggested. I can quickly do that :)
>> >
>> > * HADOOP-17649: Update wildfly openssl to 2.1.3.Final. Maybe not 2.1.3
>> > but if it works and is safe then to 2.2.5. I got flagged today that
>> > this openssl creates a bit of mess with JDK-17 for Hive with ABFS I
>> > think(need to dig in more),
>> >
>> > Now for the dependency upgrades:
>> >
>> > A big NO to Jackson, that ain't safe and the wounds are still fresh,
>> > it screwed the 3.3.3 release for many projects. So, let's not get into
>> > that. Infact anything that touches those shaded jars is risky, some
>> > package-json exclusion also created a mess recently. So, Lets not
>> > touch only and that too when we have less time.
>> >
>> > Avoid anything around Jetty upgrade, I have selfish reasons for that.
>> > Jetty messes something up with Hbase and Hive has a dependency on
>> > Hbase, and it is crazy, in case interested [1]. So, any upgrade to
>> > Jetty will block hive from upgrading Hadoop as of today. But that is a
>> > selfish reason and just up for consideration. Go ahead if necessary. I
>> > just wanted to let folks know
>> >
>> >
>> > Apart from the Jackson stuff, everything is suggestive in nature, your
>> > call feel free to ignore.
>> >
>> > @Xiaoqiao He , maybe pulling in all those 100+ would be risky,
>> > considering the timelines, but if we find a few fancy safe tickets,
>> > maybe if you have identified any already, can put them up on this
>> > thread and if folks are convinced. We can get them in? Juzz my
>> > thoughts, it is up to you and Wei-Chiu, (No skin in the game opinion)
>> >
>> >
>> > Goooood Luck!!!!
>> >
>> > -Ayush
>> >
>> > [1] https://github.com/apache/hive/pull/4290#issuecomment-1536553803
>> >
>> > On Fri, 5 May 2023 at 16:13, Steve Loughran <st...@cloudera.com.invalid>
>> wrote:
>> > >
>> > > Wei-Chiu has suggested a minimal "things in 3.3.5 which were very
>> broken,
>> > > api change for ozone and any critical jar updates"
>> > >
>> > > so much lower risk/easier to qualify and ship.
>> > >
>> > > I need to get https://issues.apache.org/jira/browse/HADOOP-18724 in
>> here;
>> > > maybe look at a refresh of the "classic" jars (slf4j, reload,
>> jackson*,
>> > > remove json-smart...)
>> > >
>> > > I'd also like to downgrade protobuf 2.5 from required to optional;
>> even
>> > > though hadoop uses the shaded one, to support hbase etc the IPC code
>> still
>> > > has direct use of the 2.5 classes. that coud be optional
>> > >
>> > > if anyone wants to take up this PR, I would be very happy
>> > > https://github.com/apache/hadoop/pull/4996
>> > >
>> > > On Fri, 5 May 2023 at 04:27, Xiaoqiao He <he...@apache.org>
>> wrote:
>> > >
>> > > > Thanks Wei-Chiu for driving this release.
>> > > > Cherry-pick YARN-11482 to branch-3.3 and mark 3.3.6 as the fixed
>> version.
>> > > >
>> > > > so far only 8 jiras were resolved in the branch-3.3 line.
>> > > >
>> > > >
>> > > > If we should consider both 3.3.6 and 3.3.9 (which is from
>> release-3.3.5
>> > > > discuss)[1] for this release line?
>> > > > I try to query with `project in (HDFS, YARN, HADOOP, MAPREDUCE) AND
>> > > > fixVersion in (3.3.6, 3.3.9)`[2],
>> > > > there are more than hundred jiras now.
>> > > >
>> > > > Best Regards,
>> > > > - He Xiaoqiao
>> > > >
>> > > > [1]
>> https://lists.apache.org/thread/kln96frt2tcg93x6ht99yck9m7r9qwxp
>> > > > [2]
>> > > >
>> > > >
>> https://issues.apache.org/jira/browse/YARN-11482?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20fixVersion%20in%20(3.3.6%2C%203.3.9)
>> > > >
>> > > >
>> > > > On Fri, May 5, 2023 at 1:19 AM Wei-Chiu Chuang <we...@apache.org>
>> wrote:
>> > > >
>> > > > > Hi community,
>> > > > >
>> > > > > I'd like to kick off the discussion around Hadoop 3.3.6 release
>> plan.
>> > > > >
>> > > > > I'm being selfish but my intent for 3.3.6 is to have the new APIs
>> in
>> > > > > HADOOP-18671 <https://issues.apache.org/jira/browse/HADOOP-18671>
>> added
>> > > > so
>> > > > > we can have HBase to adopt this new API. Other than that, perhaps
>> > > > > thirdparty dependency updates.
>> > > > >
>> > > > > If you have open items to be added in the coming weeks, please
>> add 3.3.6
>> > > > to
>> > > > > the target release version. Right now I am only seeing three open
>> jiras
>> > > > > targeting 3.3.6.
>> > > > >
>> > > > > I imagine this is going to be a small release as 3.3.5 (hat tip
>> to Steve)
>> > > > > was only made two months back, and so far only 8 jiras were
>> resolved in
>> > > > the
>> > > > > branch-3.3 line.
>> > > > >
>> > > > > Best,
>> > > > > Weichiu
>> > > > >
>> > > >
>>
>

Re: [DISCUSS] Hadoop 3.3.6 release planning

Posted by Wei-Chiu Chuang <we...@apache.org>.

Thanks a lot for the help to move this release forward.

Some status update:

branch-3.3.6 just forked out of branch-3.3.
Starting a RC0 now.

I would like to add the text to highlight big features and improvements.
Let me know if you have something that's included in this release and would
also like to highlight. @Steve Loughran <st...@apache.org> I am aware the
S3A prefetch code is in this release, but I am not sure if it is in a state
where we can make it public. I'll let you decide.

SBOM artifacts
----------------------------------------
Starting from this release, Hadoop publishes Software Bill of Materials
(SBOM) using
CycloneDX Maven plugin. For more information about SBOM, please go to
[SBOM](https://cwiki.apache.org/confluence/display/COMDEV/SBOM).

HDFS RBF: RDBMS based token storage support
----------------------------------------
HDFS Router-Router Based Federation now supports storing delegation tokens
on MySQL,
[HADOOP-18535](https://issues.apache.org/jira/browse/HADOOP-18535)
which improves token operation through over the original Zookeeper-based
implementation.


New File System APIs
----------------------------------------
[HADOOP-18671](https://issues.apache.org/jira/browse/HADOOP-18671) moved a
number of
HDFS-specific APIs to Hadoop Common to make it possible for certain
applications that
depend on HDFS semantics to run on other Hadoop compatible file systems.

In particular, recoverLease() and isFileClosed() are exposed through
LeaseRecoverable
interface. While setSafeMode() is exposed through SafeMode interface.



On Thu, Jun 8, 2023 at 11:11 AM Wei-Chiu Chuang <we...@apache.org> wrote:

> Thanks for comments
>
> Looking at jiras fixed in 3.3.6 and 3.3.9 (my bad, forgot that most
> commits landing in branch-3.3 was 3.3.9), most are okay. We have about 119
> commits so it's manageable.
>
> I am planning to cut 3.3.6 out of branch-3.3 later today. Anything open
> that is still targeting 3.3.6 will be cherry picked one by one.
> I will also bulk-update jiras fixed in 3.3.9 to 3.3.6.
>
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12337047
> As of now, I am tracking 4 jiras that's targeting 3.3.6 -- update Kerby,
> fix hadoop shaded client to support Spark history server, a small
> regression in HDFS (probably will move this out), protobuf 2.5 dependency
> change.
>
> I had a dry-run of the RC. Env set up for me and everything works. So I
> expect to have a RC ready to vote on soon. If I can move out some of the
> jiras or help them resolved, I can probably have a RC0 on Monday for vote.
>
> On Mon, May 8, 2023 at 2:01 PM Ayush Saxena <ay...@gmail.com> wrote:
>
>> That openssl change ain't a blocker now from my side, that ABFS-Jdk-17
>> stuff got sorted out, Steve knew a way out....
>>
>> On Sat, 6 May 2023 at 00:51, Ayush Saxena <ay...@gmail.com> wrote:
>> >
>> > Thanx Wei-Chiu for the initiative, Good to have quick releases :)
>> >
>> > With my Hive developer hat on, I would like to bring some stuff up for
>> > consideration(feel free to say no, if it is beyond scope or feels even
>> > a bit unsafe, don't want to mess up the release)
>> >
>> > * HADOOP-18662: ListFiles with recursive fails with FNF : This broke
>> > compaction in Hive, bothers only with HDFS though. There is a
>> > workaround to that, if it doesn't feel safe. no issues, or if some
>> > improvements suggested. I can quickly do that :)
>> >
>> > * HADOOP-17649: Update wildfly openssl to 2.1.3.Final. Maybe not 2.1.3
>> > but if it works and is safe then to 2.2.5. I got flagged today that
>> > this openssl creates a bit of mess with JDK-17 for Hive with ABFS I
>> > think(need to dig in more),
>> >
>> > Now for the dependency upgrades:
>> >
>> > A big NO to Jackson, that ain't safe and the wounds are still fresh,
>> > it screwed the 3.3.3 release for many projects. So, let's not get into
>> > that. Infact anything that touches those shaded jars is risky, some
>> > package-json exclusion also created a mess recently. So, Lets not
>> > touch only and that too when we have less time.
>> >
>> > Avoid anything around Jetty upgrade, I have selfish reasons for that.
>> > Jetty messes something up with Hbase and Hive has a dependency on
>> > Hbase, and it is crazy, in case interested [1]. So, any upgrade to
>> > Jetty will block hive from upgrading Hadoop as of today. But that is a
>> > selfish reason and just up for consideration. Go ahead if necessary. I
>> > just wanted to let folks know
>> >
>> >
>> > Apart from the Jackson stuff, everything is suggestive in nature, your
>> > call feel free to ignore.
>> >
>> > @Xiaoqiao He , maybe pulling in all those 100+ would be risky,
>> > considering the timelines, but if we find a few fancy safe tickets,
>> > maybe if you have identified any already, can put them up on this
>> > thread and if folks are convinced. We can get them in? Juzz my
>> > thoughts, it is up to you and Wei-Chiu, (No skin in the game opinion)
>> >
>> >
>> > Goooood Luck!!!!
>> >
>> > -Ayush
>> >
>> > [1] https://github.com/apache/hive/pull/4290#issuecomment-1536553803
>> >
>> > On Fri, 5 May 2023 at 16:13, Steve Loughran <st...@cloudera.com.invalid>
>> wrote:
>> > >
>> > > Wei-Chiu has suggested a minimal "things in 3.3.5 which were very
>> broken,
>> > > api change for ozone and any critical jar updates"
>> > >
>> > > so much lower risk/easier to qualify and ship.
>> > >
>> > > I need to get https://issues.apache.org/jira/browse/HADOOP-18724 in
>> here;
>> > > maybe look at a refresh of the "classic" jars (slf4j, reload,
>> jackson*,
>> > > remove json-smart...)
>> > >
>> > > I'd also like to downgrade protobuf 2.5 from required to optional;
>> even
>> > > though hadoop uses the shaded one, to support hbase etc the IPC code
>> still
>> > > has direct use of the 2.5 classes. that coud be optional
>> > >
>> > > if anyone wants to take up this PR, I would be very happy
>> > > https://github.com/apache/hadoop/pull/4996
>> > >
>> > > On Fri, 5 May 2023 at 04:27, Xiaoqiao He <he...@apache.org>
>> wrote:
>> > >
>> > > > Thanks Wei-Chiu for driving this release.
>> > > > Cherry-pick YARN-11482 to branch-3.3 and mark 3.3.6 as the fixed
>> version.
>> > > >
>> > > > so far only 8 jiras were resolved in the branch-3.3 line.
>> > > >
>> > > >
>> > > > If we should consider both 3.3.6 and 3.3.9 (which is from
>> release-3.3.5
>> > > > discuss)[1] for this release line?
>> > > > I try to query with `project in (HDFS, YARN, HADOOP, MAPREDUCE) AND
>> > > > fixVersion in (3.3.6, 3.3.9)`[2],
>> > > > there are more than hundred jiras now.
>> > > >
>> > > > Best Regards,
>> > > > - He Xiaoqiao
>> > > >
>> > > > [1]
>> https://lists.apache.org/thread/kln96frt2tcg93x6ht99yck9m7r9qwxp
>> > > > [2]
>> > > >
>> > > >
>> https://issues.apache.org/jira/browse/YARN-11482?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20fixVersion%20in%20(3.3.6%2C%203.3.9)
>> > > >
>> > > >
>> > > > On Fri, May 5, 2023 at 1:19 AM Wei-Chiu Chuang <we...@apache.org>
>> wrote:
>> > > >
>> > > > > Hi community,
>> > > > >
>> > > > > I'd like to kick off the discussion around Hadoop 3.3.6 release
>> plan.
>> > > > >
>> > > > > I'm being selfish but my intent for 3.3.6 is to have the new APIs
>> in
>> > > > > HADOOP-18671 <https://issues.apache.org/jira/browse/HADOOP-18671>
>> added
>> > > > so
>> > > > > we can have HBase to adopt this new API. Other than that, perhaps
>> > > > > thirdparty dependency updates.
>> > > > >
>> > > > > If you have open items to be added in the coming weeks, please
>> add 3.3.6
>> > > > to
>> > > > > the target release version. Right now I am only seeing three open
>> jiras
>> > > > > targeting 3.3.6.
>> > > > >
>> > > > > I imagine this is going to be a small release as 3.3.5 (hat tip
>> to Steve)
>> > > > > was only made two months back, and so far only 8 jiras were
>> resolved in
>> > > > the
>> > > > > branch-3.3 line.
>> > > > >
>> > > > > Best,
>> > > > > Weichiu
>> > > > >
>> > > >
>>
>

Re: [DISCUSS] Hadoop 3.3.6 release planning

Posted by Wei-Chiu Chuang <we...@apache.org>.

Thanks for comments

Looking at jiras fixed in 3.3.6 and 3.3.9 (my bad, forgot that most commits
landing in branch-3.3 was 3.3.9), most are okay. We have about 119 commits
so it's manageable.

I am planning to cut 3.3.6 out of branch-3.3 later today. Anything open
that is still targeting 3.3.6 will be cherry picked one by one.
I will also bulk-update jiras fixed in 3.3.9 to 3.3.6.

https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12337047
As of now, I am tracking 4 jiras that's targeting 3.3.6 -- update Kerby,
fix hadoop shaded client to support Spark history server, a small
regression in HDFS (probably will move this out), protobuf 2.5 dependency
change.

I had a dry-run of the RC. Env set up for me and everything works. So I
expect to have a RC ready to vote on soon. If I can move out some of the
jiras or help them resolved, I can probably have a RC0 on Monday for vote.

On Mon, May 8, 2023 at 2:01 PM Ayush Saxena <ay...@gmail.com> wrote:

> That openssl change ain't a blocker now from my side, that ABFS-Jdk-17
> stuff got sorted out, Steve knew a way out....
>
> On Sat, 6 May 2023 at 00:51, Ayush Saxena <ay...@gmail.com> wrote:
> >
> > Thanx Wei-Chiu for the initiative, Good to have quick releases :)
> >
> > With my Hive developer hat on, I would like to bring some stuff up for
> > consideration(feel free to say no, if it is beyond scope or feels even
> > a bit unsafe, don't want to mess up the release)
> >
> > * HADOOP-18662: ListFiles with recursive fails with FNF : This broke
> > compaction in Hive, bothers only with HDFS though. There is a
> > workaround to that, if it doesn't feel safe. no issues, or if some
> > improvements suggested. I can quickly do that :)
> >
> > * HADOOP-17649: Update wildfly openssl to 2.1.3.Final. Maybe not 2.1.3
> > but if it works and is safe then to 2.2.5. I got flagged today that
> > this openssl creates a bit of mess with JDK-17 for Hive with ABFS I
> > think(need to dig in more),
> >
> > Now for the dependency upgrades:
> >
> > A big NO to Jackson, that ain't safe and the wounds are still fresh,
> > it screwed the 3.3.3 release for many projects. So, let's not get into
> > that. Infact anything that touches those shaded jars is risky, some
> > package-json exclusion also created a mess recently. So, Lets not
> > touch only and that too when we have less time.
> >
> > Avoid anything around Jetty upgrade, I have selfish reasons for that.
> > Jetty messes something up with Hbase and Hive has a dependency on
> > Hbase, and it is crazy, in case interested [1]. So, any upgrade to
> > Jetty will block hive from upgrading Hadoop as of today. But that is a
> > selfish reason and just up for consideration. Go ahead if necessary. I
> > just wanted to let folks know
> >
> >
> > Apart from the Jackson stuff, everything is suggestive in nature, your
> > call feel free to ignore.
> >
> > @Xiaoqiao He , maybe pulling in all those 100+ would be risky,
> > considering the timelines, but if we find a few fancy safe tickets,
> > maybe if you have identified any already, can put them up on this
> > thread and if folks are convinced. We can get them in? Juzz my
> > thoughts, it is up to you and Wei-Chiu, (No skin in the game opinion)
> >
> >
> > Goooood Luck!!!!
> >
> > -Ayush
> >
> > [1] https://github.com/apache/hive/pull/4290#issuecomment-1536553803
> >
> > On Fri, 5 May 2023 at 16:13, Steve Loughran <st...@cloudera.com.invalid>
> wrote:
> > >
> > > Wei-Chiu has suggested a minimal "things in 3.3.5 which were very
> broken,
> > > api change for ozone and any critical jar updates"
> > >
> > > so much lower risk/easier to qualify and ship.
> > >
> > > I need to get https://issues.apache.org/jira/browse/HADOOP-18724 in
> here;
> > > maybe look at a refresh of the "classic" jars (slf4j, reload, jackson*,
> > > remove json-smart...)
> > >
> > > I'd also like to downgrade protobuf 2.5 from required to optional; even
> > > though hadoop uses the shaded one, to support hbase etc the IPC code
> still
> > > has direct use of the 2.5 classes. that coud be optional
> > >
> > > if anyone wants to take up this PR, I would be very happy
> > > https://github.com/apache/hadoop/pull/4996
> > >
> > > On Fri, 5 May 2023 at 04:27, Xiaoqiao He <he...@apache.org>
> wrote:
> > >
> > > > Thanks Wei-Chiu for driving this release.
> > > > Cherry-pick YARN-11482 to branch-3.3 and mark 3.3.6 as the fixed
> version.
> > > >
> > > > so far only 8 jiras were resolved in the branch-3.3 line.
> > > >
> > > >
> > > > If we should consider both 3.3.6 and 3.3.9 (which is from
> release-3.3.5
> > > > discuss)[1] for this release line?
> > > > I try to query with `project in (HDFS, YARN, HADOOP, MAPREDUCE) AND
> > > > fixVersion in (3.3.6, 3.3.9)`[2],
> > > > there are more than hundred jiras now.
> > > >
> > > > Best Regards,
> > > > - He Xiaoqiao
> > > >
> > > > [1] https://lists.apache.org/thread/kln96frt2tcg93x6ht99yck9m7r9qwxp
> > > > [2]
> > > >
> > > >
> https://issues.apache.org/jira/browse/YARN-11482?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20fixVersion%20in%20(3.3.6%2C%203.3.9)
> > > >
> > > >
> > > > On Fri, May 5, 2023 at 1:19 AM Wei-Chiu Chuang <we...@apache.org>
> wrote:
> > > >
> > > > > Hi community,
> > > > >
> > > > > I'd like to kick off the discussion around Hadoop 3.3.6 release
> plan.
> > > > >
> > > > > I'm being selfish but my intent for 3.3.6 is to have the new APIs
> in
> > > > > HADOOP-18671 <https://issues.apache.org/jira/browse/HADOOP-18671>
> added
> > > > so
> > > > > we can have HBase to adopt this new API. Other than that, perhaps
> > > > > thirdparty dependency updates.
> > > > >
> > > > > If you have open items to be added in the coming weeks, please add
> 3.3.6
> > > > to
> > > > > the target release version. Right now I am only seeing three open
> jiras
> > > > > targeting 3.3.6.
> > > > >
> > > > > I imagine this is going to be a small release as 3.3.5 (hat tip to
> Steve)
> > > > > was only made two months back, and so far only 8 jiras were
> resolved in
> > > > the
> > > > > branch-3.3 line.
> > > > >
> > > > > Best,
> > > > > Weichiu
> > > > >
> > > >
>

Re: [DISCUSS] Hadoop 3.3.6 release planning

Posted by Wei-Chiu Chuang <we...@apache.org>.

Thanks for comments

Looking at jiras fixed in 3.3.6 and 3.3.9 (my bad, forgot that most commits
landing in branch-3.3 was 3.3.9), most are okay. We have about 119 commits
so it's manageable.

I am planning to cut 3.3.6 out of branch-3.3 later today. Anything open
that is still targeting 3.3.6 will be cherry picked one by one.
I will also bulk-update jiras fixed in 3.3.9 to 3.3.6.

https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12337047
As of now, I am tracking 4 jiras that's targeting 3.3.6 -- update Kerby,
fix hadoop shaded client to support Spark history server, a small
regression in HDFS (probably will move this out), protobuf 2.5 dependency
change.

I had a dry-run of the RC. Env set up for me and everything works. So I
expect to have a RC ready to vote on soon. If I can move out some of the
jiras or help them resolved, I can probably have a RC0 on Monday for vote.

On Mon, May 8, 2023 at 2:01 PM Ayush Saxena <ay...@gmail.com> wrote:

> That openssl change ain't a blocker now from my side, that ABFS-Jdk-17
> stuff got sorted out, Steve knew a way out....
>
> On Sat, 6 May 2023 at 00:51, Ayush Saxena <ay...@gmail.com> wrote:
> >
> > Thanx Wei-Chiu for the initiative, Good to have quick releases :)
> >
> > With my Hive developer hat on, I would like to bring some stuff up for
> > consideration(feel free to say no, if it is beyond scope or feels even
> > a bit unsafe, don't want to mess up the release)
> >
> > * HADOOP-18662: ListFiles with recursive fails with FNF : This broke
> > compaction in Hive, bothers only with HDFS though. There is a
> > workaround to that, if it doesn't feel safe. no issues, or if some
> > improvements suggested. I can quickly do that :)
> >
> > * HADOOP-17649: Update wildfly openssl to 2.1.3.Final. Maybe not 2.1.3
> > but if it works and is safe then to 2.2.5. I got flagged today that
> > this openssl creates a bit of mess with JDK-17 for Hive with ABFS I
> > think(need to dig in more),
> >
> > Now for the dependency upgrades:
> >
> > A big NO to Jackson, that ain't safe and the wounds are still fresh,
> > it screwed the 3.3.3 release for many projects. So, let's not get into
> > that. Infact anything that touches those shaded jars is risky, some
> > package-json exclusion also created a mess recently. So, Lets not
> > touch only and that too when we have less time.
> >
> > Avoid anything around Jetty upgrade, I have selfish reasons for that.
> > Jetty messes something up with Hbase and Hive has a dependency on
> > Hbase, and it is crazy, in case interested [1]. So, any upgrade to
> > Jetty will block hive from upgrading Hadoop as of today. But that is a
> > selfish reason and just up for consideration. Go ahead if necessary. I
> > just wanted to let folks know
> >
> >
> > Apart from the Jackson stuff, everything is suggestive in nature, your
> > call feel free to ignore.
> >
> > @Xiaoqiao He , maybe pulling in all those 100+ would be risky,
> > considering the timelines, but if we find a few fancy safe tickets,
> > maybe if you have identified any already, can put them up on this
> > thread and if folks are convinced. We can get them in? Juzz my
> > thoughts, it is up to you and Wei-Chiu, (No skin in the game opinion)
> >
> >
> > Goooood Luck!!!!
> >
> > -Ayush
> >
> > [1] https://github.com/apache/hive/pull/4290#issuecomment-1536553803
> >
> > On Fri, 5 May 2023 at 16:13, Steve Loughran <st...@cloudera.com.invalid>
> wrote:
> > >
> > > Wei-Chiu has suggested a minimal "things in 3.3.5 which were very
> broken,
> > > api change for ozone and any critical jar updates"
> > >
> > > so much lower risk/easier to qualify and ship.
> > >
> > > I need to get https://issues.apache.org/jira/browse/HADOOP-18724 in
> here;
> > > maybe look at a refresh of the "classic" jars (slf4j, reload, jackson*,
> > > remove json-smart...)
> > >
> > > I'd also like to downgrade protobuf 2.5 from required to optional; even
> > > though hadoop uses the shaded one, to support hbase etc the IPC code
> still
> > > has direct use of the 2.5 classes. that coud be optional
> > >
> > > if anyone wants to take up this PR, I would be very happy
> > > https://github.com/apache/hadoop/pull/4996
> > >
> > > On Fri, 5 May 2023 at 04:27, Xiaoqiao He <he...@apache.org>
> wrote:
> > >
> > > > Thanks Wei-Chiu for driving this release.
> > > > Cherry-pick YARN-11482 to branch-3.3 and mark 3.3.6 as the fixed
> version.
> > > >
> > > > so far only 8 jiras were resolved in the branch-3.3 line.
> > > >
> > > >
> > > > If we should consider both 3.3.6 and 3.3.9 (which is from
> release-3.3.5
> > > > discuss)[1] for this release line?
> > > > I try to query with `project in (HDFS, YARN, HADOOP, MAPREDUCE) AND
> > > > fixVersion in (3.3.6, 3.3.9)`[2],
> > > > there are more than hundred jiras now.
> > > >
> > > > Best Regards,
> > > > - He Xiaoqiao
> > > >
> > > > [1] https://lists.apache.org/thread/kln96frt2tcg93x6ht99yck9m7r9qwxp
> > > > [2]
> > > >
> > > >
> https://issues.apache.org/jira/browse/YARN-11482?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20fixVersion%20in%20(3.3.6%2C%203.3.9)
> > > >
> > > >
> > > > On Fri, May 5, 2023 at 1:19 AM Wei-Chiu Chuang <we...@apache.org>
> wrote:
> > > >
> > > > > Hi community,
> > > > >
> > > > > I'd like to kick off the discussion around Hadoop 3.3.6 release
> plan.
> > > > >
> > > > > I'm being selfish but my intent for 3.3.6 is to have the new APIs
> in
> > > > > HADOOP-18671 <https://issues.apache.org/jira/browse/HADOOP-18671>
> added
> > > > so
> > > > > we can have HBase to adopt this new API. Other than that, perhaps
> > > > > thirdparty dependency updates.
> > > > >
> > > > > If you have open items to be added in the coming weeks, please add
> 3.3.6
> > > > to
> > > > > the target release version. Right now I am only seeing three open
> jiras
> > > > > targeting 3.3.6.
> > > > >
> > > > > I imagine this is going to be a small release as 3.3.5 (hat tip to
> Steve)
> > > > > was only made two months back, and so far only 8 jiras were
> resolved in
> > > > the
> > > > > branch-3.3 line.
> > > > >
> > > > > Best,
> > > > > Weichiu
> > > > >
> > > >
>

Re: [DISCUSS] Hadoop 3.3.6 release planning

Posted by Wei-Chiu Chuang <we...@apache.org>.

Thanks for comments

Looking at jiras fixed in 3.3.6 and 3.3.9 (my bad, forgot that most commits
landing in branch-3.3 was 3.3.9), most are okay. We have about 119 commits
so it's manageable.

I am planning to cut 3.3.6 out of branch-3.3 later today. Anything open
that is still targeting 3.3.6 will be cherry picked one by one.
I will also bulk-update jiras fixed in 3.3.9 to 3.3.6.

https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12337047
As of now, I am tracking 4 jiras that's targeting 3.3.6 -- update Kerby,
fix hadoop shaded client to support Spark history server, a small
regression in HDFS (probably will move this out), protobuf 2.5 dependency
change.

I had a dry-run of the RC. Env set up for me and everything works. So I
expect to have a RC ready to vote on soon. If I can move out some of the
jiras or help them resolved, I can probably have a RC0 on Monday for vote.

On Mon, May 8, 2023 at 2:01 PM Ayush Saxena <ay...@gmail.com> wrote:

> That openssl change ain't a blocker now from my side, that ABFS-Jdk-17
> stuff got sorted out, Steve knew a way out....
>
> On Sat, 6 May 2023 at 00:51, Ayush Saxena <ay...@gmail.com> wrote:
> >
> > Thanx Wei-Chiu for the initiative, Good to have quick releases :)
> >
> > With my Hive developer hat on, I would like to bring some stuff up for
> > consideration(feel free to say no, if it is beyond scope or feels even
> > a bit unsafe, don't want to mess up the release)
> >
> > * HADOOP-18662: ListFiles with recursive fails with FNF : This broke
> > compaction in Hive, bothers only with HDFS though. There is a
> > workaround to that, if it doesn't feel safe. no issues, or if some
> > improvements suggested. I can quickly do that :)
> >
> > * HADOOP-17649: Update wildfly openssl to 2.1.3.Final. Maybe not 2.1.3
> > but if it works and is safe then to 2.2.5. I got flagged today that
> > this openssl creates a bit of mess with JDK-17 for Hive with ABFS I
> > think(need to dig in more),
> >
> > Now for the dependency upgrades:
> >
> > A big NO to Jackson, that ain't safe and the wounds are still fresh,
> > it screwed the 3.3.3 release for many projects. So, let's not get into
> > that. Infact anything that touches those shaded jars is risky, some
> > package-json exclusion also created a mess recently. So, Lets not
> > touch only and that too when we have less time.
> >
> > Avoid anything around Jetty upgrade, I have selfish reasons for that.
> > Jetty messes something up with Hbase and Hive has a dependency on
> > Hbase, and it is crazy, in case interested [1]. So, any upgrade to
> > Jetty will block hive from upgrading Hadoop as of today. But that is a
> > selfish reason and just up for consideration. Go ahead if necessary. I
> > just wanted to let folks know
> >
> >
> > Apart from the Jackson stuff, everything is suggestive in nature, your
> > call feel free to ignore.
> >
> > @Xiaoqiao He , maybe pulling in all those 100+ would be risky,
> > considering the timelines, but if we find a few fancy safe tickets,
> > maybe if you have identified any already, can put them up on this
> > thread and if folks are convinced. We can get them in? Juzz my
> > thoughts, it is up to you and Wei-Chiu, (No skin in the game opinion)
> >
> >
> > Goooood Luck!!!!
> >
> > -Ayush
> >
> > [1] https://github.com/apache/hive/pull/4290#issuecomment-1536553803
> >
> > On Fri, 5 May 2023 at 16:13, Steve Loughran <st...@cloudera.com.invalid>
> wrote:
> > >
> > > Wei-Chiu has suggested a minimal "things in 3.3.5 which were very
> broken,
> > > api change for ozone and any critical jar updates"
> > >
> > > so much lower risk/easier to qualify and ship.
> > >
> > > I need to get https://issues.apache.org/jira/browse/HADOOP-18724 in
> here;
> > > maybe look at a refresh of the "classic" jars (slf4j, reload, jackson*,
> > > remove json-smart...)
> > >
> > > I'd also like to downgrade protobuf 2.5 from required to optional; even
> > > though hadoop uses the shaded one, to support hbase etc the IPC code
> still
> > > has direct use of the 2.5 classes. that coud be optional
> > >
> > > if anyone wants to take up this PR, I would be very happy
> > > https://github.com/apache/hadoop/pull/4996
> > >
> > > On Fri, 5 May 2023 at 04:27, Xiaoqiao He <he...@apache.org>
> wrote:
> > >
> > > > Thanks Wei-Chiu for driving this release.
> > > > Cherry-pick YARN-11482 to branch-3.3 and mark 3.3.6 as the fixed
> version.
> > > >
> > > > so far only 8 jiras were resolved in the branch-3.3 line.
> > > >
> > > >
> > > > If we should consider both 3.3.6 and 3.3.9 (which is from
> release-3.3.5
> > > > discuss)[1] for this release line?
> > > > I try to query with `project in (HDFS, YARN, HADOOP, MAPREDUCE) AND
> > > > fixVersion in (3.3.6, 3.3.9)`[2],
> > > > there are more than hundred jiras now.
> > > >
> > > > Best Regards,
> > > > - He Xiaoqiao
> > > >
> > > > [1] https://lists.apache.org/thread/kln96frt2tcg93x6ht99yck9m7r9qwxp
> > > > [2]
> > > >
> > > >
> https://issues.apache.org/jira/browse/YARN-11482?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20fixVersion%20in%20(3.3.6%2C%203.3.9)
> > > >
> > > >
> > > > On Fri, May 5, 2023 at 1:19 AM Wei-Chiu Chuang <we...@apache.org>
> wrote:
> > > >
> > > > > Hi community,
> > > > >
> > > > > I'd like to kick off the discussion around Hadoop 3.3.6 release
> plan.
> > > > >
> > > > > I'm being selfish but my intent for 3.3.6 is to have the new APIs
> in
> > > > > HADOOP-18671 <https://issues.apache.org/jira/browse/HADOOP-18671>
> added
> > > > so
> > > > > we can have HBase to adopt this new API. Other than that, perhaps
> > > > > thirdparty dependency updates.
> > > > >
> > > > > If you have open items to be added in the coming weeks, please add
> 3.3.6
> > > > to
> > > > > the target release version. Right now I am only seeing three open
> jiras
> > > > > targeting 3.3.6.
> > > > >
> > > > > I imagine this is going to be a small release as 3.3.5 (hat tip to
> Steve)
> > > > > was only made two months back, and so far only 8 jiras were
> resolved in
> > > > the
> > > > > branch-3.3 line.
> > > > >
> > > > > Best,
> > > > > Weichiu
> > > > >
> > > >
>