[jira] [Updated] (HADOOP-19971) Remove servlet and Jetty types from classes that do not need them

["Jose Luis López (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-19971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jose Luis López updated HADOOP-19971:
-------------------------------------
    Description: 
h3. Why

Hadoop will eventually move from the javax.servlet namespace to jakarta.servlet. When it does, every class that mentions a servlet type has to be revisited, and every class that mentions a Jetty type has to be revisited again when the Jetty API changes underneath it.

Several of those classes have no real connection to either. A component that generates signing secrets was handed the web server's context even though it only wanted somewhere to store an object. A certificate parser reported a bad certificate as a web server error. A Netty-based shuffle handler borrowed two header names from Jetty, and the YARN services client borrowed Jetty's URL encoder.

This change removes those references, so those classes drop out of the later migration entirely.

h3. What changes for anyone using Hadoop

Nothing is removed, no signature changes shape, and no dependency, version or setting changes. Code built against today's release keeps compiling and running without being rebuilt.

* Two entry points get a servlet-free alternative. The old ones stay, marked deprecated, until the namespace change:
** {{SignerSecretProvider.init(Properties, ServletContext, long)}} -> {{initialize(Properties, SecretProviderContext, long)}}
** {{CertificateUtil.parseRSAPublicKey(String)}} -> {{toRSAPublicKey(String)}}
* Providers that override {{init}} are still called directly. A provider that extends Hadoop's rolling secret provider and calls {{super.init}} still gets its rollover scheduler started. {{TestSignerSecretProviderCompatibility}} pins this contract.
* Exceptions are unchanged: same type, message and cause chain from {{parseRSAPublicKey}} and from {{JWTRedirectAuthenticationHandler}} when the PEM is corrupt.
* The shuffle response is unchanged, byte for byte: the same {{Connection}} and {{Keep-Alive}} header names, capitalised as before.
* The YARN services client now encodes {{user.name}} with {{java.net.URLEncoder}}. That differs from Jetty's encoder for two characters only: {{*}} is now sent as is where it used to be escaped, and {{~}} is now escaped where it used to be sent as is. Every code point decodes to the same value either way (checked across all of Unicode against Jetty 9.4.58), so the server reads the same user name.

Things a downstream project could notice:
* Deprecation warnings where code overrides {{init}} or calls {{parseRSAPublicKey}}.
* {{getDeclaredMethod("init", ...)}} on the shipped providers now throws {{NoSuchMethodException}}, because they declare {{initialize}} instead. {{getMethod}} still finds {{init}}.
* {{ZKSignerSecretProvider}} given a null ServletContext used to throw a NullPointerException. It now keeps its ZooKeeper client in a store private to itself and logs a WARN. Every caller in Hadoop passes a real context, and there the client is still stored as the ServletContext attribute of the same name.

h3. What changes inside

|| || before || after ||
| modules using Jetty in main sources | 12 | 11 |
| main-source files using Jetty | 26 | 24 |
| hadoop-auth main classes naming a servlet | 14 | 11 |

Counted on trunk at a7c2bea723d.

The MapReduce shuffle module no longer uses Jetty at all. In hadoop-auth, the secret providers and the authentication token no longer mention the servlet API. They are initialised against a small two-method store, {{SecretProviderContext}}, which holds whatever needs sharing; that is all the one provider that used the servlet context ever did with it. One new package-private class mentions the servlet API, and it exists only to keep the old path working.

h3. What this does not do

No namespace change, no Jetty upgrade, no Jersey change, no EE environments. The tree stays on Jetty 9.4, Jersey 2 and javax.servlet.

The authentication handler classes still mention the servlet API, deliberately. They are the extension point that HBase, Hive, Spark, Ozone and Knox build against, and changing it is exactly the kind of break this issue is meant to avoid. That belongs with the namespace change, where downstream projects will expect it.

h3. Ordering

This is preparation for the jakarta move. It depends on nothing, is based on trunk, and nothing in HADOOP-19972 depends on it. The plan is to land it with or just before the namespace change in HADOOP-19395, in the next major release.

When rebasing onto a later trunk:
* If HADOOP-19970 has landed, it declares jetty-http in the shuffle module's pom only because of the import this change removes. Drop that declaration here.
* If HADOOP-19972 has landed, re-check the shuffle headers and the {{user.name}} encoding against whatever Jetty classes it left in those two places.

16 files changed. Works on Jetty 9.4 and on the Jetty 12 ee8 environment of HADOOP-19912, and commits the namespace change in HADOOP-19395 to neither ee10 nor ee11.

Contains content generated by Claude.

  was:
h3. Why

Hadoop will eventually move from the javax.servlet namespace to jakarta.servlet. When it does, every class that mentions a servlet type has to be revisited, and every class that mentions a Jetty type has to be revisited again when the Jetty API changes underneath it.

Several of those classes have no real connection to either. A component that generates signing secrets was handed the web server's context even though it only wanted somewhere to store an object. A certificate parser reported a bad certificate as a web server error. A Netty-based shuffle handler borrowed two header names from Jetty, and the YARN services client borrowed Jetty's URL encoder.

This change removes those references, so those classes drop out of the later migration entirely.

h3. What changes for anyone using Hadoop

Nothing is removed, no signature changes shape, and no dependency, version or setting changes. Code built against today's release keeps compiling and running without being rebuilt.

* Two entry points get a servlet-free alternative. The old ones stay, marked deprecated, until the namespace change:
** {{SignerSecretProvider.init(Properties, ServletContext, long)}} -> {{initialize(Properties, SecretProviderContext, long)}}
** {{CertificateUtil.parseRSAPublicKey(String)}} -> {{toRSAPublicKey(String)}}
* Providers that override {{init}} are still called directly. A provider that extends Hadoop's rolling secret provider and calls {{super.init}} still gets its rollover scheduler started. {{TestSignerSecretProviderCompatibility}} pins this contract.
* Exceptions are unchanged: same type, message and cause chain from {{parseRSAPublicKey}} and from {{JWTRedirectAuthenticationHandler}} when the PEM is corrupt.
* The shuffle response is unchanged, byte for byte: the same {{Connection}} and {{Keep-Alive}} header names, capitalised as before.
* The YARN services client now encodes {{user.name}} with {{java.net.URLEncoder}}. That differs from Jetty's encoder for two characters only: {{*}} is now sent as is where it used to be escaped, and {{~}} is now escaped where it used to be sent as is. Every code point decodes to the same value either way (checked across all of Unicode against Jetty 9.4.58), so the server reads the same user name.

Things a downstream project could notice:
* Deprecation warnings where code overrides {{init}} or calls {{parseRSAPublicKey}}.
* {{getDeclaredMethod("init", ...)}} on the shipped providers now throws {{NoSuchMethodException}}, because they declare {{initialize}} instead. {{getMethod}} still finds {{init}}.
* {{ZKSignerSecretProvider}} given a null ServletContext used to throw a NullPointerException. It now keeps its ZooKeeper client in a store private to itself and logs a WARN. Every caller in Hadoop passes a real context, and there the client is still stored as the ServletContext attribute of the same name.

h3. What changes inside

|| || before || after ||
| modules using Jetty in main sources | 12 | 11 |
| main-source files using Jetty | 26 | 24 |
| hadoop-auth main classes naming a servlet | 14 | 11 |

Counted on trunk at a7c2bea723d.

The MapReduce shuffle module no longer uses Jetty at all. In hadoop-auth, the secret providers and the authentication token no longer mention the servlet API. They are initialised against a small two-method store, {{SecretProviderContext}}, which holds whatever needs sharing; that is all the one provider that used the servlet context ever did with it. One new package-private class mentions the servlet API, and it exists only to keep the old path working.

h3. What this does not do

No namespace change, no Jetty upgrade, no Jersey change, no EE environments. The tree stays on Jetty 9.4, Jersey 2 and javax.servlet.

The authentication handler classes still mention the servlet API, deliberately. They are the extension point that HBase, Hive, Spark, Ozone and Knox build against, and changing it is exactly the kind of break this issue is meant to avoid. That belongs with the namespace change, where downstream projects will expect it.

h3. Ordering

This is preparation for the jakarta move. It depends on nothing, is based on trunk, and nothing in HADOOP-19972 depends on it. The plan is to land it with or just before HADOOP-19912, in the next major release.

When rebasing onto a later trunk:
* If HADOOP-19970 has landed, it declares jetty-http in the shuffle module's pom only because of the import this change removes. Drop that declaration here.
* If HADOOP-19972 has landed, re-check the shuffle headers and the {{user.name}} encoding against whatever Jetty classes it left in those two places.

16 files changed. Works whether HADOOP-19912 ends up going through Jetty 12's ee8 environment or straight to ee10, and commits the project to neither.

Contains content generated by Claude.


> Remove servlet and Jetty types from classes that do not need them
> -----------------------------------------------------------------
>
>                 Key: HADOOP-19971
>                 URL: https://issues.apache.org/jira/browse/HADOOP-19971
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: auth, common
>            Reporter: Jose Luis López
>            Assignee: Jose Luis López
>            Priority: Minor
>              Labels: pull-request-available
>
> h3. Why
> Hadoop will eventually move from the javax.servlet namespace to jakarta.servlet. When it does, every class that mentions a servlet type has to be revisited, and every class that mentions a Jetty type has to be revisited again when the Jetty API changes underneath it.
> Several of those classes have no real connection to either. A component that generates signing secrets was handed the web server's context even though it only wanted somewhere to store an object. A certificate parser reported a bad certificate as a web server error. A Netty-based shuffle handler borrowed two header names from Jetty, and the YARN services client borrowed Jetty's URL encoder.
> This change removes those references, so those classes drop out of the later migration entirely.
> h3. What changes for anyone using Hadoop
> Nothing is removed, no signature changes shape, and no dependency, version or setting changes. Code built against today's release keeps compiling and running without being rebuilt.
> * Two entry points get a servlet-free alternative. The old ones stay, marked deprecated, until the namespace change:
> ** {{SignerSecretProvider.init(Properties, ServletContext, long)}} -> {{initialize(Properties, SecretProviderContext, long)}}
> ** {{CertificateUtil.parseRSAPublicKey(String)}} -> {{toRSAPublicKey(String)}}
> * Providers that override {{init}} are still called directly. A provider that extends Hadoop's rolling secret provider and calls {{super.init}} still gets its rollover scheduler started. {{TestSignerSecretProviderCompatibility}} pins this contract.
> * Exceptions are unchanged: same type, message and cause chain from {{parseRSAPublicKey}} and from {{JWTRedirectAuthenticationHandler}} when the PEM is corrupt.
> * The shuffle response is unchanged, byte for byte: the same {{Connection}} and {{Keep-Alive}} header names, capitalised as before.
> * The YARN services client now encodes {{user.name}} with {{java.net.URLEncoder}}. That differs from Jetty's encoder for two characters only: {{*}} is now sent as is where it used to be escaped, and {{~}} is now escaped where it used to be sent as is. Every code point decodes to the same value either way (checked across all of Unicode against Jetty 9.4.58), so the server reads the same user name.
> Things a downstream project could notice:
> * Deprecation warnings where code overrides {{init}} or calls {{parseRSAPublicKey}}.
> * {{getDeclaredMethod("init", ...)}} on the shipped providers now throws {{NoSuchMethodException}}, because they declare {{initialize}} instead. {{getMethod}} still finds {{init}}.
> * {{ZKSignerSecretProvider}} given a null ServletContext used to throw a NullPointerException. It now keeps its ZooKeeper client in a store private to itself and logs a WARN. Every caller in Hadoop passes a real context, and there the client is still stored as the ServletContext attribute of the same name.
> h3. What changes inside
> || || before || after ||
> | modules using Jetty in main sources | 12 | 11 |
> | main-source files using Jetty | 26 | 24 |
> | hadoop-auth main classes naming a servlet | 14 | 11 |
> Counted on trunk at a7c2bea723d.
> The MapReduce shuffle module no longer uses Jetty at all. In hadoop-auth, the secret providers and the authentication token no longer mention the servlet API. They are initialised against a small two-method store, {{SecretProviderContext}}, which holds whatever needs sharing; that is all the one provider that used the servlet context ever did with it. One new package-private class mentions the servlet API, and it exists only to keep the old path working.
> h3. What this does not do
> No namespace change, no Jetty upgrade, no Jersey change, no EE environments. The tree stays on Jetty 9.4, Jersey 2 and javax.servlet.
> The authentication handler classes still mention the servlet API, deliberately. They are the extension point that HBase, Hive, Spark, Ozone and Knox build against, and changing it is exactly the kind of break this issue is meant to avoid. That belongs with the namespace change, where downstream projects will expect it.
> h3. Ordering
> This is preparation for the jakarta move. It depends on nothing, is based on trunk, and nothing in HADOOP-19972 depends on it. The plan is to land it with or just before the namespace change in HADOOP-19395, in the next major release.
> When rebasing onto a later trunk:
> * If HADOOP-19970 has landed, it declares jetty-http in the shuffle module's pom only because of the import this change removes. Drop that declaration here.
> * If HADOOP-19972 has landed, re-check the shuffle headers and the {{user.name}} encoding against whatever Jetty classes it left in those two places.
> 16 files changed. Works on Jetty 9.4 and on the Jetty 12 ee8 environment of HADOOP-19912, and commits the namespace change in HADOOP-19395 to neither ee10 nor ee11.
> Contains content generated by Claude.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org