You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "aajisaka (via GitHub)" <gi...@apache.org> on 2026/01/15 06:16:29 UTC

[PR] [WIP] HADOOP-19785. mvn site fails in JDK17 [hadoop]

aajisaka opened a new pull request, #8182:
URL: https://github.com/apache/hadoop/pull/8182

   <!--
     Thanks for sending a pull request!
       1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
       2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
   -->
   
   ### Description of PR
   
   Fix the following error while running `mvn site` with JDK17
   
   ```
   Error:  /home/runner/work/hadoop/hadoop/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSWebApp.java:26: error: cannot find symbol
   Error:  import com.codahale.metrics.JmxReporter;
   Error:                             ^
   Error:    symbol:   class JmxReporter
   Error:    location: package com.codahale.metrics
   Error:  /home/runner/work/hadoop/hadoop/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSWebApp.java:69: error: cannot find symbol
   Error:    private JmxReporter jmxReporter;
   Error:            ^
   Error:    symbol:   class JmxReporter
   Error:    location: class KMSWebApp
   Error:  /home/runner/work/hadoop/hadoop/hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/SimulatedDataNodes.java:36: error: cannot find symbol
   Error:  import org.apache.hadoop.hdfs.MiniDFSCluster;
   Error:                               ^
   Error:    symbol:   class MiniDFSCluster
   Error:    location: package org.apache.hadoop.hdfs
   Error:  /home/runner/work/hadoop/hadoop/hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/SimulatedDataNodes.java:41: error: cannot find symbol
   Error:  import org.apache.hadoop.hdfs.server.datanode.DataNodeTestUtils;
   Error:                                               ^
   Error:    symbol:   class DataNodeTestUtils
   Error:    location: package org.apache.hadoop.hdfs.server.datanode
   Error:  /home/runner/work/hadoop/hadoop/hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/SimulatedDataNodes.java:42: error: cannot find symbol
   Error:  import org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset;
   Error:                                               ^
   Error:    symbol:   class SimulatedFSDataset
   Error:    location: package org.apache.hadoop.hdfs.server.datanode
   Error: [ERROR] 5 errors
   ```
   
   ### How was this patch tested?
   
   Ran the following command and verified the above error has been fixed:
   ```
   mvn clean install -DskipTests -DskipShade
   mvn site
   ```
   
   However, the Hadoop custom Doclet is failing due to the change in https://bugs.openjdk.org/browse/JDK-8253736
   
   ```
   [ERROR] java.lang.ClassCastException: class jdk.proxy1.$Proxy0 cannot be cast to
    class jdk.javadoc.internal.tool.DocEnvImpl (jdk.proxy1.$Proxy0 is in module jdk.proxy1 of loader 'app'; jdk.javadoc.internal.tool.DocEnvImpl is in module jdk.javadoc of 
   loader 'app')
   [ERROR]     at jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.WorkArounds.<init>(WorkArounds.java:104)
   [ERROR]     at jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.BaseConfiguration.initConfiguration(BaseConfiguration.java:251)
   [ERROR]     at jdk.javadoc/jdk.javadoc.internal.doclets.formats.html.HtmlConfiguration.initConfiguration(HtmlConfiguration.java:220)
   [ERROR]     at jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.AbstractDoclet.run(AbstractDoclet.java:104)
   [ERROR]     at jdk.javadoc/jdk.javadoc.doclet.StandardDoclet.run(StandardDoclet.java:103)
   [ERROR]     at org.apache.hadoop.classification.tools.IncludePublicAnnotationsStandardDoclet.run(IncludePublicAnnotationsStandardDoclet.java:152)
   [ERROR]     at jdk.javadoc/jdk.javadoc.internal.tool.Start.parseAndExecute(Start.java:556)
   [ERROR]     at jdk.javadoc/jdk.javadoc.internal.tool.Start.begin(Start.java:393)
   [ERROR]     at jdk.javadoc/jdk.javadoc.internal.tool.Start.begin(Start.java:342)
   [ERROR]     at jdk.javadoc/jdk.javadoc.internal.tool.Main.execute(Main.java:63)
   [ERROR]     at jdk.javadoc/jdk.javadoc.internal.tool.Main.main(Main.java:52)
   [ERROR] 2 errors
   ``` 
   
   ### For code changes:
   
   - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
   - [x] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
   - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [x] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
   
   ### AI Tooling
   
   If an AI tool was used:
   
   - [x] The PR includes the phrase "Contains content generated by <tool>"
         where <tool> is the name of the AI tool used.
   - [x] My use of AI contributions follows the ASF legal policy
         https://www.apache.org/legal/generative-tooling.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#issuecomment-3809165998

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   1m 23s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  1s |  |  detect-secrets was not available.  |
   | +0 :ok: |  xmllint  |   0m  1s |  |  xmllint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | -1 :x: |  test4tests  |   0m  0s |  |  The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.  |
   |||| _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  11m 37s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  48m 56s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  20m  1s |  |  trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  compile  |  20m 30s |  |  trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  checkstyle  |   4m 10s |  |  trunk passed  |
   | -1 :x: |  mvnsite  |  12m  6s | [/branch-mvnsite-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/5/artifact/out/branch-mvnsite-root.txt) |  root in trunk failed.  |
   | -1 :x: |  javadoc  |   9m 50s | [/branch-javadoc-root-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/5/artifact/out/branch-javadoc-root-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) |  root in trunk failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.  |
   | -1 :x: |  javadoc  |   9m 49s | [/branch-javadoc-root-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/5/artifact/out/branch-javadoc-root-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt) |  root in trunk failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.  |
   | +0 :ok: |  spotbugs  |   0m 28s |  |  branch/hadoop-project no spotbugs output file (spotbugsXml.xml)  |
   | -1 :x: |  spotbugs  |  41m  8s | [/branch-spotbugs-root-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/5/artifact/out/branch-spotbugs-root-warnings.html) |  root in trunk has 94 extant spotbugs warnings.  |
   | +1 :green_heart: |  shadedclient  |  74m 29s |  |  branch has no errors when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 44s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |  48m 57s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  20m 20s |  |  the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javac  |  20m 20s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  20m 11s |  |  the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javac  |  20m 11s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | -0 :warning: |  checkstyle  |   0m 41s | [/buildtool-patch-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/5/artifact/out/buildtool-patch-checkstyle-root.txt) |  The patch fails to run checkstyle in root  |
   | +1 :green_heart: |  mvnsite  |  21m 25s |  |  the patch passed  |
   | -1 :x: |  javadoc  |   0m 33s | [/patch-javadoc-root-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/5/artifact/out/patch-javadoc-root-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) |  root in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.  |
   | -1 :x: |  javadoc  |   0m 32s | [/patch-javadoc-root-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/5/artifact/out/patch-javadoc-root-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt) |  root in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.  |
   | +0 :ok: |  spotbugs  |   0m 27s |  |  hadoop-project has no data from spotbugs  |
   | +1 :green_heart: |  shadedclient  |  75m  5s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | -1 :x: |  unit  | 878m 13s | [/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/5/artifact/out/patch-unit-root.txt) |  root in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   1m 51s |  |  The patch does not generate ASF License warnings.  |
   |  |   | 1304m 33s |  |  |
   
   
   | Reason | Tests |
   |-------:|:------|
   | Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMHA |
   |   | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation |
   |   | hadoop.yarn.server.router.webapp.TestFederationWebApp |
   |   | hadoop.yarn.server.router.subcluster.fair.TestYarnFederationWithFairScheduler |
   |   | hadoop.yarn.sls.appmaster.TestAMSimulator |
   |   | hadoop.hdfs.tools.TestDFSAdmin |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.53 ServerAPI=1.53 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/5/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/8182 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
   | uname | Linux 1fffd08855eb 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 6f1cc3d40604413af576ca51770852fdaf840f73 |
   | Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
   | Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/5/testReport/ |
   | Max. process+thread count | 3792 (vs. ulimit of 5500) |
   | modules | C: hadoop-project hadoop-common-project/hadoop-annotations hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp hadoop-tools/hadoop-azure . U: . |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/5/console |
   | versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 |
   | Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#discussion_r2745818581


##########
hadoop-project/pom.xml:
##########
@@ -2823,7 +2830,8 @@
       </activation>
       <properties>
         <javac.version>17</javac.version>
-        <maven.compiler.release>${javac.version}</maven.compiler.release>

Review Comment:
   I think we want to keep this, so that we can use a higher version of JDK to build Hadoop, but produce Java 17 compatible bytecode. If this fails on some modules, can we just disable it on these specific modules?
   
   BTW, the activation condition of `jdk17+` profile is already true, we can remove it and move the content to the top level.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] [WIP] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#issuecomment-3757766601

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   1m 14s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  |
   | +0 :ok: |  xmllint  |   0m  0s |  |  xmllint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | -1 :x: |  test4tests  |   0m  0s |  |  The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.  |
   |||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  43m  2s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  19m  2s |  |  trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  compile  |  19m 26s |  |  trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04  |
   | -1 :x: |  mvnsite  |  10m 10s | [/branch-mvnsite-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/1/artifact/out/branch-mvnsite-root.txt) |  root in trunk failed.  |
   | -1 :x: |  javadoc  |   9m  7s | [/branch-javadoc-root-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/1/artifact/out/branch-javadoc-root-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) |  root in trunk failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.  |
   | -1 :x: |  javadoc  |   8m 32s | [/branch-javadoc-root-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/1/artifact/out/branch-javadoc-root-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt) |  root in trunk failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.  |
   | +1 :green_heart: |  shadedclient  | 141m 26s |  |  branch has no errors when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  34m 27s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  18m 20s |  |  the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javac  |  18m 20s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  19m 49s |  |  the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javac  |  19m 49s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | -1 :x: |  mvnsite  |   8m 16s | [/patch-mvnsite-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/1/artifact/out/patch-mvnsite-root.txt) |  root in the patch failed.  |
   | -1 :x: |  javadoc  |   7m 15s | [/patch-javadoc-root-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/1/artifact/out/patch-javadoc-root-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) |  root in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.  |
   | -1 :x: |  javadoc  |   7m 23s | [/patch-javadoc-root-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/1/artifact/out/patch-javadoc-root-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt) |  root in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.  |
   | +1 :green_heart: |  shadedclient  |  57m 23s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | -1 :x: |  unit  | 913m 20s | [/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/1/artifact/out/patch-unit-root.txt) |  root in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   1m 51s |  |  The patch does not generate ASF License warnings.  |
   |  |   | 1190m 50s |  |  |
   
   
   | Reason | Tests |
   |-------:|:------|
   | Failed junit tests | hadoop.yarn.service.TestYarnNativeServices |
   |   | hadoop.yarn.server.resourcemanager.TestRMHA |
   |   | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation |
   |   | hadoop.yarn.server.router.webapp.TestRouterWebServicesREST |
   |   | hadoop.yarn.server.router.subcluster.fair.TestYarnFederationWithFairScheduler |
   |   | hadoop.yarn.sls.appmaster.TestAMSimulator |
   |   | hadoop.hdfs.tools.TestDFSAdmin |
   |   | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList |
   |   | hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints |
   |   | hadoop.mapreduce.v2.app.TestRuntimeEstimators |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/1/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/8182 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint |
   | uname | Linux c4c5b5d51744 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 2996b57b356ff8690e5fddab42abcda8c064f944 |
   | Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
   | Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/1/testReport/ |
   | Max. process+thread count | 3935 (vs. ulimit of 5500) |
   | modules | C: . U: . |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/1/console |
   | versions | git=2.25.1 maven=3.9.11 |
   | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] [WIP] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#issuecomment-3766811692

   > Hi @zhtttylz, you've implemented new custom Doclet to support JDK17 in #8038, however, unfortunately `mvn site` fails in Doclet part. The detail is below:
   > 
   > 1. RootDocProcessor.process(env) creates a Proxy object that implements DocletEnvironment interface
   > 2. This proxy is passed to StandardDoclet.run(filtered)
   > 3. Inside the StandardDoclet, the javadoc internals (specifically WorkArounds.()) try to cast this proxy to the concrete implementation class jdk.javadoc.internal.tool.DocEnvImpl: https://github.com/jonathan-gibbons/jdk/blob/bc1e60c1bf91678ef18652a00aa2ce55b0446caa/src/jdk.javadoc/share/classes/jdk/javadoc/internal/doclets/toolkit/WorkArounds.java#L112
   > 4. The cast fails because a proxy object cannot be cast to a concrete implementation class
   > 
   > It seems our custom doclet implementation is prohibited after https://bugs.openjdk.org/browse/JDK-8253736:
   > 
   > > One particularly annoying wart is the cast on DocletEnvironment to DocEnvImpl, which effectively prevents using subtypes to carry additional info. It is not clear (even now) what the best way is to replace that logic.
   > 
   > Now I feel it's becoming really hard to maintain Hadoop's custom Doclets, and therefore I would like to drop the custom implementation. The primary change is we are going to build Hadoop JavaDoc with `@LimitedPrivate`, `@Private` or `@Unstable` classes, which are now excluded by our custom Doclets.
   > 
   > @slfan1989 @cnauroth @zhtttylz What do you think?
   
   @aajisaka Thanks for the detailed analysis — after reading through it, I fully agree that the cost of maintaining this custom Doclet has become unreasonably high. With OpenJDK continuing to clean up internal APIs (the trend starting from JDK-8253736 is only getting stronger), future compatibility will only get worse, and the next LTS might break it completely.
   I'm in favor of dropping the custom Doclet and switching to the standard doclet to generate complete JavaDocs (including all classes annotated with @Private / @Unstable / @LimitedPrivate). The main reasons are:
   
   - The maintenance burden is too heavy and takes away energy from more valuable work;
   - The visibility of these annotated classes has very limited impact on most downstream users — they shouldn't be depending on @Private APIs anyway;
   - On the positive side, having the full picture can actually help developers/contributors who want to dig into the implementation details.
   
   cc: @zhtttylz @cnauroth 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] [WIP] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#issuecomment-3753598322

   Hi @zhtttylz, you've implemented new custom Doclet to support JDK17 in https://github.com/apache/hadoop/pull/8038, however, unfortunately `mvn site` fails in Doclet part. The detail is below:
   
   1. RootDocProcessor.process(env) creates a Proxy object that implements DocletEnvironment interface
   2. This proxy is passed to StandardDoclet.run(filtered)
   3. Inside the StandardDoclet, the javadoc internals (specifically WorkArounds.<init>()) try to cast this proxy to the concrete implementation class jdk.javadoc.internal.tool.DocEnvImpl: https://github.com/jonathan-gibbons/jdk/blob/bc1e60c1bf91678ef18652a00aa2ce55b0446caa/src/jdk.javadoc/share/classes/jdk/javadoc/internal/doclets/toolkit/WorkArounds.java#L112
   4. The cast fails because a proxy object cannot be cast to a concrete implementation class
   
   It seems our custom doclet implementation is prohibited after https://bugs.openjdk.org/browse/JDK-8253736:
   
   > One particularly annoying wart is the cast on DocletEnvironment to DocEnvImpl, which effectively prevents using subtypes to carry additional info. It is not clear (even now) what the best way is to replace that logic.
   
   Now I feel it's becoming really hard to maintain Hadoop's custom Doclets, and therefore I would like to drop the custom implementation. The primary change is we are going to build Hadoop JavaDoc with `@LimitedPrivate`, `@Private` or `@Unstable` classes, which are now excluded by our custom Doclets.
   
   @slfan1989 @cnauroth @zhtttylz What do you think? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on code in PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#discussion_r2749393314


##########
hadoop-project/pom.xml:
##########
@@ -2823,7 +2830,8 @@
       </activation>
       <properties>
         <javac.version>17</javac.version>
-        <maven.compiler.release>${javac.version}</maven.compiler.release>

Review Comment:
   > I think we want to keep this, so that we can use a higher version of JDK to build Hadoop, but produce Java 17 compatible bytecode.
   
   Agreed. Removed `jdk17+` profile and moved the release flag into maven-compiler-plugin config, so that it can be overridden by hadoop-annotation module.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#discussion_r2730799138


##########
hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/HadoopDocEnvImpl.java:
##########
@@ -0,0 +1,156 @@
+package org.apache.hadoop.classification.tools;
+
+import com.sun.source.util.DocTrees;
+import jdk.javadoc.doclet.DocletEnvironment;
+import jdk.javadoc.internal.tool.DocEnvImpl;
+import jdk.javadoc.internal.tool.ToolEnvironment;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+
+import javax.lang.model.SourceVersion;
+import javax.lang.model.element.AnnotationMirror;
+import javax.lang.model.element.Element;
+import javax.lang.model.element.ElementKind;
+import javax.lang.model.element.TypeElement;
+import javax.lang.model.util.Elements;
+import javax.lang.model.util.Types;
+import javax.tools.JavaFileManager;
+import javax.tools.JavaFileObject;
+import java.util.LinkedHashSet;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * This class extends Java internal DocEnvImpl to avoid cast error while
+ * migrating to JDK17. It filters elements based on Hadoop's InterfaceAudience
+ * and InterfaceStability annotations.
+ * This class depends on JDK internal implementation, so we might need to
+ * update the source code when upgrading to upper JDK versions.

Review Comment:
   nice!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "cnauroth (via GitHub)" <gi...@apache.org>.
cnauroth commented on code in PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#discussion_r2733226547


##########
pom.xml:
##########
@@ -701,16 +701,28 @@ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/x
                   <version>${project.version}</version>
                 </docletArtifact>
               </docletArtifacts>
-              <useStandardDocletOptions>true</useStandardDocletOptions>
-
+              <additionalJOptions>
+                <additionalJOption>-J--add-exports=jdk.javadoc/jdk.javadoc.internal.tool=ALL-UNNAMED</additionalJOption>
+              </additionalJOptions>
+              <additionalOptions>

Review Comment:
   Is this necessary in `additionalOptions`, which is options passed to `javadoc`? Since `--add-exports` is a JVM argument, I would have expected we only need it in `additionalJOptions`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#issuecomment-3833532696

   Now the patch is ready for review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#issuecomment-3830543654

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   1m 10s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  1s |  |  detect-secrets was not available.  |
   | +0 :ok: |  xmllint  |   0m  1s |  |  xmllint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | -1 :x: |  test4tests  |   0m  0s |  |  The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.  |
   |||| _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |   2m  7s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  52m 15s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  20m 12s |  |  trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  compile  |  19m 34s |  |  trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  checkstyle  |   3m 19s |  |  trunk passed  |
   | -1 :x: |  mvnsite  |   9m 43s | [/branch-mvnsite-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/8/artifact/out/branch-mvnsite-root.txt) |  root in trunk failed.  |
   | -1 :x: |  javadoc  |   9m  3s | [/branch-javadoc-root-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/8/artifact/out/branch-javadoc-root-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) |  root in trunk failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.  |
   | -1 :x: |  javadoc  |   8m 33s | [/branch-javadoc-root-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/8/artifact/out/branch-javadoc-root-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt) |  root in trunk failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.  |
   | +0 :ok: |  spotbugs  |   0m 24s |  |  branch/hadoop-project no spotbugs output file (spotbugsXml.xml)  |
   | -1 :x: |  spotbugs  |  37m 57s | [/branch-spotbugs-root-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/8/artifact/out/branch-spotbugs-root-warnings.html) |  root in trunk has 94 extant spotbugs warnings.  |
   | +1 :green_heart: |  shadedclient  |  33m 19s |  |  branch has no errors when building and testing our client artifacts.  |
   | -0 :warning: |  patch  |  33m 45s |  |  Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.  |
   |||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 43s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |  46m 16s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  18m 47s |  |  the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javac  |  18m 47s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  19m 36s |  |  the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javac  |  19m 36s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | +1 :green_heart: |  checkstyle  |   5m 53s |  |  the patch passed  |
   | +1 :green_heart: |  mvnsite  |  19m 53s |  |  the patch passed  |
   | -1 :x: |  javadoc  |   9m 43s | [/results-javadoc-javadoc-root-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/8/artifact/out/results-javadoc-javadoc-root-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) |  root-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 generated 958 new + 8936 unchanged - 35326 fixed = 9894 total (was 44262)  |
   | -1 :x: |  javadoc  |   9m 40s | [/results-javadoc-javadoc-root-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/8/artifact/out/results-javadoc-javadoc-root-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt) |  root-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 generated 776 new + 9320 unchanged - 31841 fixed = 10096 total (was 41161)  |
   | +0 :ok: |  spotbugs  |   0m 21s |  |  hadoop-project has no data from spotbugs  |
   | +1 :green_heart: |  shadedclient  |  69m 47s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | -1 :x: |  unit  | 827m 33s | [/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/8/artifact/out/patch-unit-root.txt) |  root in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   1m 51s |  |  The patch does not generate ASF License warnings.  |
   |  |   | 1246m 55s |  |  |
   
   
   | Reason | Tests |
   |-------:|:------|
   | Failed junit tests | hadoop.yarn.service.TestYarnNativeServices |
   |   | hadoop.yarn.server.resourcemanager.TestRMHA |
   |   | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation |
   |   | hadoop.yarn.server.router.subcluster.fair.TestYarnFederationWithFairScheduler |
   |   | hadoop.yarn.sls.appmaster.TestAMSimulator |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.53 ServerAPI=1.53 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/8/artifact/out/Dockerfile |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
   | uname | Linux ba47af204f4b 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 90150aff3f84183cebaf21e977834b8a77e8ede7 |
   | Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
   | Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/8/testReport/ |
   | Max. process+thread count | 3811 (vs. ulimit of 5500) |
   | modules | C: hadoop-project hadoop-common-project/hadoop-annotations hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp hadoop-tools/hadoop-azure . U: . |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/8/console |
   | versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 |
   | Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#discussion_r2730661470


##########
hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/HadoopDocEnvImpl.java:
##########
@@ -0,0 +1,156 @@
+package org.apache.hadoop.classification.tools;

Review Comment:
   missing ALv2 header



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#issuecomment-3798000554

   > The current Hadoop 3.4.2 API docs show ~13,000 classes. Without the visibility annotation filtering, it goes up to ~126,000 classes.
   
   late chime in - the number sounds like a huge impact, was anyone considered migrating to [japicmp](https://siom79.github.io/japicmp/)? it looks like a modern and popular solution for binary compatibility check, I see that at least the Parquet project is using it. Specifically, the docs say it supports "exclusion based on annotations"
   
   > Per default all classes are tracked. If necessary, certain packages, classes, methods or fields can be excluded or explicitly included. Inclusion and exclusion is also possible based on annotations.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#discussion_r2730663778


##########
hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/HadoopDocEnvImpl.java:
##########
@@ -0,0 +1,156 @@
+package org.apache.hadoop.classification.tools;
+
+import com.sun.source.util.DocTrees;
+import jdk.javadoc.doclet.DocletEnvironment;
+import jdk.javadoc.internal.tool.DocEnvImpl;
+import jdk.javadoc.internal.tool.ToolEnvironment;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+
+import javax.lang.model.SourceVersion;
+import javax.lang.model.element.AnnotationMirror;
+import javax.lang.model.element.Element;
+import javax.lang.model.element.ElementKind;
+import javax.lang.model.element.TypeElement;
+import javax.lang.model.util.Elements;
+import javax.lang.model.util.Types;
+import javax.tools.JavaFileManager;
+import javax.tools.JavaFileObject;
+import java.util.LinkedHashSet;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * This class extends Java internal DocEnvImpl to avoid cast error while
+ * migrating to JDK17. It filters elements based on Hadoop's InterfaceAudience
+ * and InterfaceStability annotations.
+ * This class depends on JDK internal implementation, so we might need to
+ * update the source code when upgrading to upper JDK versions.

Review Comment:
   As we are here, could you please also test if it works with JDK 21 and 25?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#issuecomment-3809062300

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |  21m  8s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  |
   | +0 :ok: |  xmllint  |   0m  0s |  |  xmllint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | -1 :x: |  test4tests  |   0m  0s |  |  The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.  |
   |||| _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  12m 19s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  49m 40s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  20m 33s |  |  trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  compile  |  21m 35s |  |  trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  checkstyle  |   4m 12s |  |  trunk passed  |
   | -1 :x: |  mvnsite  |  11m 55s | [/branch-mvnsite-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/4/artifact/out/branch-mvnsite-root.txt) |  root in trunk failed.  |
   | -1 :x: |  javadoc  |   9m 43s | [/branch-javadoc-root-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/4/artifact/out/branch-javadoc-root-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) |  root in trunk failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.  |
   | -1 :x: |  javadoc  |   9m  9s | [/branch-javadoc-root-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/4/artifact/out/branch-javadoc-root-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt) |  root in trunk failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.  |
   | +0 :ok: |  spotbugs  |   0m 25s |  |  branch/hadoop-project no spotbugs output file (spotbugsXml.xml)  |
   | -1 :x: |  spotbugs  |  43m 35s | [/branch-spotbugs-root-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/4/artifact/out/branch-spotbugs-root-warnings.html) |  root in trunk has 94 extant spotbugs warnings.  |
   | +1 :green_heart: |  shadedclient  |  77m 48s |  |  branch has no errors when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 44s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |  48m 34s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  19m 16s |  |  the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javac  |  19m 16s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  21m  4s |  |  the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javac  |  21m  4s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | -0 :warning: |  checkstyle  |   0m 40s | [/buildtool-patch-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/4/artifact/out/buildtool-patch-checkstyle-root.txt) |  The patch fails to run checkstyle in root  |
   | +1 :green_heart: |  mvnsite  |  21m 59s |  |  the patch passed  |
   | -1 :x: |  javadoc  |   0m 31s | [/patch-javadoc-root-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/4/artifact/out/patch-javadoc-root-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) |  root in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.  |
   | -1 :x: |  javadoc  |   0m 27s | [/patch-javadoc-root-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/4/artifact/out/patch-javadoc-root-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt) |  root in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.  |
   | +0 :ok: |  spotbugs  |   0m 23s |  |  hadoop-project has no data from spotbugs  |
   | +1 :green_heart: |  shadedclient  |  74m 28s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | -1 :x: |  unit  | 873m 30s | [/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/4/artifact/out/patch-unit-root.txt) |  root in the patch passed.  |
   | -1 :x: |  asflicense  |   1m 55s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/4/artifact/out/results-asflicense.txt) |  The patch generated 1 ASF License warnings.  |
   |  |   | 1323m 18s |  |  |
   
   
   | Reason | Tests |
   |-------:|:------|
   | Failed junit tests | hadoop.yarn.service.TestYarnNativeServices |
   |   | hadoop.yarn.server.resourcemanager.TestRMHA |
   |   | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation |
   |   | hadoop.yarn.server.router.subcluster.fair.TestYarnFederationWithFairScheduler |
   |   | hadoop.yarn.sls.appmaster.TestAMSimulator |
   |   | hadoop.hdfs.tools.TestDFSAdmin |
   |   | hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.53 ServerAPI=1.53 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/4/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/8182 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
   | uname | Linux 5a606da41f9f 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 5e2fe4cae44057fcf9c9a866a944d6ff7cc5f82e |
   | Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
   | Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/4/testReport/ |
   | Max. process+thread count | 3856 (vs. ulimit of 5500) |
   | modules | C: hadoop-project hadoop-common-project/hadoop-annotations hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp hadoop-tools/hadoop-azure . U: . |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/4/console |
   | versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 |
   | Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] [WIP] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on code in PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#discussion_r2693491269


##########
pom.xml:
##########
@@ -710,7 +710,14 @@ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/x
                 <!-- include ONLY dependencies I control -->
                 <dependencySourceInclude>org.apache.hadoop:hadoop-annotations</dependencySourceInclude>
               </dependencySourceIncludes>
-
+              <additionalDependencies>
+                <additionalDependency>
+                  <groupId>org.apache.hadoop</groupId>
+                  <artifactId>hadoop-hdfs</artifactId>
+                  <version>${project.version}</version>

Review Comment:
   Hadoop Dynamometer production classes depend on HDFS test jar



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#issuecomment-3803545335

   > Specific to this issue, it looks like public class DocEnvImpl ... is not final, so it's feasible to create a derived class class HadoopDocEnvImpl extends DocEnvImpl that delegates all invocations to the Proxy one, to pass the explicitly cast check?
   
   You are right. I could successfully create HadoopDocEnvImpl extends DocEnvImpl, and also cleaned up invocations.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on code in PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#discussion_r2730771005


##########
hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/HadoopDocEnvImpl.java:
##########
@@ -0,0 +1,156 @@
+package org.apache.hadoop.classification.tools;

Review Comment:
   Thank you! Fixed in https://github.com/apache/hadoop/pull/8182/commits/6f1cc3d40604413af576ca51770852fdaf840f73



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "cnauroth (via GitHub)" <gi...@apache.org>.
cnauroth commented on code in PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#discussion_r2747255040


##########
pom.xml:
##########
@@ -100,14 +100,14 @@ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/x
 
     <!-- maven plugin versions -->
     <maven-deploy-plugin.version>2.8.1</maven-deploy-plugin.version>
-    <maven-site-plugin.version>3.9.1</maven-site-plugin.version>
+    <maven-site-plugin.version>3.21.0</maven-site-plugin.version>

Review Comment:
   I've been testing dry run releases of 3.5.0 from trunk with this patch applied. It generates the JavaDocs correctly, but then the JavaDocs don't show up in the final release of the site tarball. This looks similar to symptoms described in #5319, where we worked around it by downgrading maven-site-plugin.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka merged PR #8182:
URL: https://github.com/apache/hadoop/pull/8182


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] [WIP] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on code in PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#discussion_r2693491269


##########
pom.xml:
##########
@@ -710,7 +710,14 @@ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/x
                 <!-- include ONLY dependencies I control -->
                 <dependencySourceInclude>org.apache.hadoop:hadoop-annotations</dependencySourceInclude>
               </dependencySourceIncludes>
-
+              <additionalDependencies>
+                <additionalDependency>
+                  <groupId>org.apache.hadoop</groupId>
+                  <artifactId>hadoop-hdfs</artifactId>
+                  <version>${project.version}</version>

Review Comment:
   Added because Hadoop Dynamometer production classes depend on HDFS test jar



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#issuecomment-3798186425

   Thank you @pan3793
   
   There are 2 discussion points - jdiff and javadoc. I think jdiff can be covered by japicmp, but javadoc is not. Chris already started the thread in common-dev@ for doc side


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#issuecomment-3808544388

   Thank you @cnauroth for your review. Updated the default stability level to include `@Public` and `@Evolving` classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#discussion_r2745818581


##########
hadoop-project/pom.xml:
##########
@@ -2823,7 +2830,8 @@
       </activation>
       <properties>
         <javac.version>17</javac.version>
-        <maven.compiler.release>${javac.version}</maven.compiler.release>

Review Comment:
   I think we want to keep this, so that we can use a higher version of JDK to build Hadoop, but produce Java 17 compatible bytecode. If this fails on some modules, can we just disable it on these specific modules?
   
   BTW, the activation condition of `jdk17+` profile is always true, we can remove it and move the content to the top level.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#issuecomment-3838349598

   Merged. Thank you @slfan1989, @pan3793, and @cnauroth !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#issuecomment-3798289102

   @aajisaka, thanks for the clarification. I re-read the conversation carefully and understand the situation now.
   
   > 4. The cast fails because a proxy object cannot be cast to a concrete implementation class
   
   Specific to this issue, it looks like `public class DocEnvImpl ...` is not `final`, so it's feasible to create a derived class that delegates all invocations to the Proxy one, to pass the explicitly cast check?
   
   > Now I feel it's becoming really hard to maintain Hadoop's custom Doclets, and therefore I would like to drop the custom implementation. 
   
   Agree, but given the impact number is huge, and I suppose that Hadoop will use Java 17 as a baseline for several years, it's still valuable if the problem can be solved by a few hundred lines of hacky code.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on code in PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#discussion_r2730777603


##########
hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/HadoopDocEnvImpl.java:
##########
@@ -0,0 +1,156 @@
+package org.apache.hadoop.classification.tools;
+
+import com.sun.source.util.DocTrees;
+import jdk.javadoc.doclet.DocletEnvironment;
+import jdk.javadoc.internal.tool.DocEnvImpl;
+import jdk.javadoc.internal.tool.ToolEnvironment;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+
+import javax.lang.model.SourceVersion;
+import javax.lang.model.element.AnnotationMirror;
+import javax.lang.model.element.Element;
+import javax.lang.model.element.ElementKind;
+import javax.lang.model.element.TypeElement;
+import javax.lang.model.util.Elements;
+import javax.lang.model.util.Types;
+import javax.tools.JavaFileManager;
+import javax.tools.JavaFileObject;
+import java.util.LinkedHashSet;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * This class extends Java internal DocEnvImpl to avoid cast error while
+ * migrating to JDK17. It filters elements based on Hadoop's InterfaceAudience
+ * and InterfaceStability annotations.
+ * This class depends on JDK internal implementation, so we might need to
+ * update the source code when upgrading to upper JDK versions.

Review Comment:
   ```
   mvn clean install -DskipTests -DskipShade
   mvn site
   mvn site:stage -DstagingDirectory=/path/to/doc
   ```
   The above test commands worked with JDK 21 and 25.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on code in PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#discussion_r2734587908


##########
pom.xml:
##########
@@ -701,16 +701,28 @@ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/x
                   <version>${project.version}</version>
                 </docletArtifact>
               </docletArtifacts>
-              <useStandardDocletOptions>true</useStandardDocletOptions>
-
+              <additionalJOptions>
+                <additionalJOption>-J--add-exports=jdk.javadoc/jdk.javadoc.internal.tool=ALL-UNNAMED</additionalJOption>
+              </additionalJOptions>
+              <additionalOptions>

Review Comment:
   The both are passed to `javadoc` and they are necessary.
   
   * Compile error occurs without additionalOptions (`--add-exports=jdk.javadoc/jdk.javadoc.internal.tools=ALL-UNNAMED`):
   
   ```
   [ERROR] /Users/akajisak/git/github/hadoop/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/HadoopDocEnvImpl.java:22: error: package jdk.javadoc.internal.tool is not visible
   [ERROR] import jdk.javadoc.internal.tool.DocEnvImpl;
   [ERROR]                            ^
   [ERROR]   (package jdk.javadoc.internal.tool is declared in module jdk.javadoc, which does not export it)
   [ERROR] /Users/akajisak/git/github/hadoop/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/HadoopDocEnvImpl.java:23: error: package jdk.javadoc.internal.tool is not visible
   [ERROR] import jdk.javadoc.internal.tool.ToolEnvironment;
   [ERROR]                            ^
   [ERROR]   (package jdk.javadoc.internal.tool is declared in module jdk.javadoc, which does not export it)
   [ERROR] 2 errors
   ```
   
   * Runtime error occurs without additionalJOptions (`-J--add-exports=jdk.javadoc/jdk.javadoc.internal.tools=ALL-UNNAMED`):
   
   ```
   [ERROR] error: fatal error encountered: java.lang.IllegalAccessError: superclass access check failed: class org.apache.hadoop.classification.tools.HadoopDocEnvImpl (in unnamed module @0x7a55f88e) cannot access class jdk.javadoc.internal.tool.DocEnvImpl (in module jdk.javadoc) because module jdk.javadoc does not export jdk.javadoc.internal.tool to unnamed module @0x7a55f88e
   [ERROR] error: Please file a bug against the javadoc tool via the Java bug reporting page
   [ERROR]   (https://bugreport.java.com) after checking the Bug Database (https://bugs.java.com)
   [ERROR]   for duplicates. Include error messages and the following diagnostic in your report. Thank you.
   [ERROR] java.lang.IllegalAccessError: superclass access check failed: class org.apache.hadoop.classification.tools.HadoopDocEnvImpl (in unnamed module @0x7a55f88e) cannot access class jdk.javadoc.internal.tool.DocEnvImpl (in module jdk.javadoc) because module jdk.javadoc does not export jdk.javadoc.internal.tool to unnamed module @0x7a55f88e
   [ERROR] 	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
   [ERROR] 	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
   [ERROR] 	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
   [ERROR] 	at java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:524)
   [ERROR] 	at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:427)
   [ERROR] 	at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:421)
   [ERROR] 	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
   [ERROR] 	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:420)
   [ERROR] 	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:592)
   [ERROR] 	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
   [ERROR] 	at org.apache.hadoop.classification.tools.RootDocProcessor.process(RootDocProcessor.java:54)
   [ERROR] 	at org.apache.hadoop.classification.tools.IncludePublicAnnotationsStandardDoclet.run(IncludePublicAnnotationsStandardDoclet.java:151)
   [ERROR] 	at jdk.javadoc/jdk.javadoc.internal.tool.Start.parseAndExecute(Start.java:556)
   [ERROR] 	at jdk.javadoc/jdk.javadoc.internal.tool.Start.begin(Start.java:393)
   [ERROR] 	at jdk.javadoc/jdk.javadoc.internal.tool.Start.begin(Start.java:342)
   [ERROR] 	at jdk.javadoc/jdk.javadoc.internal.tool.Main.execute(Main.java:63)
   [ERROR] 	at jdk.javadoc/jdk.javadoc.internal.tool.Main.main(Main.java:52)
   [ERROR] 2 errors
   ```
   
   > I would have expected we only need it in additionalJOptions:
   
   The following config also worked:
   ```
                 <additionalJOptions>
                   <additionalJOption>--add-exports=jdk.javadoc/jdk.javadoc.internal.tool=ALL-UNNAMED</additionalJOption>
                   <additionalJOption>-J--add-exports=jdk.javadoc/jdk.javadoc.internal.tool=ALL-UNNAMED</additionalJOption>
                 </additionalJOptions>
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on code in PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#discussion_r2749390960


##########
pom.xml:
##########
@@ -100,14 +100,14 @@ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/x
 
     <!-- maven plugin versions -->
     <maven-deploy-plugin.version>2.8.1</maven-deploy-plugin.version>
-    <maven-site-plugin.version>3.9.1</maven-site-plugin.version>
+    <maven-site-plugin.version>3.21.0</maven-site-plugin.version>

Review Comment:
   Thank you @cnauroth for the details.
   
   Downgraded maven-site-plugin and maven-javadoc-plugin. maven-javadoc-plugin >= 3.10.0 cannot configure the output directory according to https://github.com/apache/maven-javadoc-plugin/issues/1305



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "cnauroth (via GitHub)" <gi...@apache.org>.
cnauroth commented on code in PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#discussion_r2747387846


##########
pom.xml:
##########
@@ -100,14 +100,14 @@ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/x
 
     <!-- maven plugin versions -->
     <maven-deploy-plugin.version>2.8.1</maven-deploy-plugin.version>
-    <maven-site-plugin.version>3.9.1</maven-site-plugin.version>
+    <maven-site-plugin.version>3.21.0</maven-site-plugin.version>

Review Comment:
   Here are some more details on this. When I build on branch-3.4 (without the plugin upgrade), the API docs land here:
   
   ```
   target/staging/hadoop-project/api
   ```
   
   On trunk, with this PR applied, they land here instead:
   
   ```
   target/staging/apidocs
   ```
   
   Then, the release script has an assumption that it can find everything in the `hadoop-project` sub-directory:
   
   https://github.com/apache/hadoop/blob/trunk/dev-support/bin/create-release#L624-L628
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] [WIP] HADOOP-19785. mvn site fails in JDK17 [hadoop]

Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on PR #8182:
URL: https://github.com/apache/hadoop/pull/8182#issuecomment-3770636111

   Thank you @slfan1989. I'll update the patch with using the standard doclet.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org