You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "GauthamBanasandra (via GitHub)" <gi...@apache.org> on 2026/10/19 16:04:15 UTC
[PR] HADOOP-20031. Fix the Windows native build: Docker image, libhdfspp CMake and YARN UI [hadoop]
GauthamBanasandra opened a new pull request, #8844:
URL: https://github.com/apache/hadoop/pull/8844
### Description of PR
`mvn clean install` with `-Pnative-win` cannot complete on Windows. The Docker image described in
BUILDING.txt no longer builds at all, and even with a working image the native build fails in
libhdfspp and again in the YARN capacity-scheduler UI. There is no single cause — eight separate
defects have accumulated, each of which independently stops the build.
**The build image no longer builds**
1. `Dockerfile_windows_10` installs Azul JDK 8. Hadoop requires JDK 17, so the enforcer rejects the
build immediately. This line has not been touched since 2022.
2. `vcpkg install` fails because the pinned `protobuf 3.25.5` cannot be resolved from vcpkg's
version database. The pin was added by HADOOP-19605 on 2025-11-10.
3. The pinned vcpkg tag `2025.03.19` references MSYS2 packages (`msys2-runtime-3.5.4-2`) that have
since been purged from the upstream mirror. MSYS2 keeps only current versions.
4. The `libzstd` and `rsync` MSYS2 pins in the Dockerfile have been purged for the same reason.
**The native build fails**
5. Boost is not pinned explicitly — its version comes from `builtin-baseline`, so it floats as the
baseline moves. Boost 1.87 removed `boost/asio/io_service.hpp`, which five libhdfspp files still
include (`ioservice.h`, `ioservice_impl.h`, `mock_connection.{h,cc}`,
`remote_block_reader_test.cc`).
6. The protobuf/protoc compatibility check in `libhdfspp/CMakeLists.txt` leaves
`CMAKE_REQUIRED_LIBRARIES` pointing at the Abseil targets. Abseil carries C++ feature
requirements that a later C-language `check_symbol_exists()` cannot satisfy under the Visual
Studio generator, so that check fails spuriously.
7. The legacy `PROTOBUF_LIBRARY`, `PROTOBUF_PROTOC_LIBRARY` and `PROTOBUF_LIBRARIES` variables
resolve to import DLLs under vcpkg, which MSVC rejects with `LNK1107`.
8. `hadoop-yarn-capacity-scheduler-ui` sets `packageManagerScript` to
`node_modules/.bin/react-router`. On Windows that path is an `sh` shim, which `node` cannot
execute.
**Why Linux is unaffected**
The Linux build *pins* its native dependencies inside the repository —
`dev-support/docker/install-boost.sh` hard-codes Boost 1.86.0 and `install-protobuf.sh` hard-codes
Protobuf 3.25.5, both built from source. The Windows build instead delegates to vcpkg and lets
versions float with the baseline. That asymmetry is the root cause of defects 2, 5 and arguably 3:
the Windows build has no equivalent of the version lock the Linux build has had all along.
**Fix**
- **Dockerfile**: install the Microsoft Build of OpenJDK 17; move to vcpkg tag `2026.07.29`; refresh
the `libzstd` and `rsync` pins; refresh CMake to 3.30.2. Switch the base image to
`windows/servercore`, which carries everything the build needs at a fraction of the size, and
relax the password policy that Server Core enables by default so the existing passwordless
`HadoopBuilder` account can still be created.
- **vcpkg.json**: narrow the Boost dependency from the `boost` metapackage to the four ports
libhdfspp actually includes (asio, date-time, program-options, system), and pin all 55
transitively required Boost ports to 1.86.0 — the same version `install-boost.sh` pins for Linux.
Pin protobuf to a version vcpkg publishes.
- **libhdfspp**: reset `CMAKE_REQUIRED_INCLUDES` and `CMAKE_REQUIRED_LIBRARIES` after the protoc
compatibility check, and migrate from the legacy `PROTOBUF_*` variables to the
`protobuf::libprotobuf` and `protobuf::libprotoc` imported targets.
- **hadoop-yarn-capacity-scheduler-ui**: point `packageManagerScript` at the real entry point,
`node_modules/@react-router/dev/bin.js`.
- **BUILDING.txt**: correct the Windows Docker section. It omitted that Docker Desktop must be
switched to Windows containers; it documented the Maven repository mount under
`ContainerAdministrator` although the Dockerfile ends with `USER HadoopBuilder`, so the documented
mount is silently ignored; it recommended `docker cp`, which fails against a running Hyper-V
isolated container; it omitted the container memory the build needs; and the manual vcpkg route
still pinned the purged `2025.03.19` tag.
### How was this patch tested?
Tested in the Windows container built from this PR's own `Dockerfile_windows_10`
(`windows/servercore:ltsc2019`, Hyper-V isolation, 8 GB), against a fresh clone of trunk with these
changes applied.
**Full build** — a single clean `mvn clean install` with `-Pnative-win,dist`, no resume:
- **120 of 120 modules SUCCESS**, no failures and no skipped modules
- 48:59 min against a warm Maven repository
- produces `winutils.exe`, `hadoop.dll` and `hdfs.dll`, present both in
`hadoop-dist/target/hadoop-3.6.0-SNAPSHOT/bin` and inside the distribution tarball
**winutils smoke test** — `systeminfo`, `ls`, `chmod` and `hardlink` all run correctly against the
produced binary.
**Unit tests** — `hadoop-common`, 5564 tests in 39:02 min: 5063 passed, 38 failures, 78 errors, 385
skipped. None of the failures are attributable to this PR: no file under `hadoop-common` or
`winutils` is modified here, so that module builds and tests as pristine trunk. The failures are the
pre-existing Windows set (`TestSSLHttpServer`, `TestHttpServer`, `TestShellBasedUnixGroupsMapping`,
`TestGroupFallback` and similar), plus some forked-JVM `OutOfMemoryError`s caused by the 8 GB
container limit rather than by any code defect.
Two honest caveats, both pre-existing and outside this PR's scope:
- `hadoop checknative -a` reports `winutils: true` and `hadoop: true`, but `zlib`, `bzip2`,
`openssl`, `ISA-L` and `PMDK` all `false`. This PR restores the ability to build; it does not add
codec support.
- `-Drequire.openssl` does not appear to be enforced on the Windows native path: vcpkg installs
openssl 3.6.3 and `libssl.lib`/`libcrypto.lib` are present, yet the build reports "build does not
support openssl" and still succeeds. Worth a separate issue.
Separately, `-Dcmake.prefix.path` as documented in BUILDING.txt is read by nothing — the only reader
in the tree is `${windows.cmake.prefix.path}` in `hadoop-hdfs-native-client/pom.xml`, and neither
property is defined in any pom, so the documented flag left `CMAKE_PREFIX_PATH` holding an
unexpanded literal. BUILDING.txt is corrected here. The same misspelling also appears in
`dev-support/bin/hadoop.sh` and may deserve its own issue. Note that the verified build above ran
with the *misspelled* flag, i.e. exactly as BUILDING.txt documented it before this change.
### For code changes:
- [x] Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: Have the integration tests been executed and the endpoint
declared according to the connector-specific documentation? *Note: Automated CI
testing doesn't cover all cases so manual testing with cloud storage is still
required.* — not applicable, no object storage code is touched.
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? — no new dependency is introduced. Boost, Protocol Buffers, OpenSSL and zlib were already used by the Windows build; this PR only pins the versions vcpkg resolves at image-build time.
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? — not applicable, nothing new is bundled into Hadoop's artifacts.
### AI Tooling
If an AI tool was used:
- [x] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [x] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
Contains content generated by Claude Code.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-20031. Fix the Windows native build: Docker image, libhdfspp CMake and YARN UI [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8844:
URL: https://github.com/apache/hadoop/pull/8844#issuecomment-6379776157
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 14s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | jsonlint | 0m 0s | | jsonlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 38s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 53m 44s | | trunk passed |
| +1 :green_heart: | compile | 20m 1s | | trunk passed with JDK Red Hat, Inc.-21.0.12.1+1-LTS |
| +1 :green_heart: | compile | 20m 53s | | trunk passed with JDK Red Hat, Inc.-17.0.20.1+1-LTS |
| +1 :green_heart: | mvnsite | 21m 50s | | trunk passed |
| +1 :green_heart: | javadoc | 10m 38s | | trunk passed with JDK Red Hat, Inc.-21.0.12.1+1-LTS |
| +1 :green_heart: | javadoc | 12m 32s | | trunk passed with JDK Red Hat, Inc.-17.0.20.1+1-LTS |
| +1 :green_heart: | shadedclient | 58m 59s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 50s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 52m 27s | | the patch passed |
| +1 :green_heart: | compile | 20m 5s | | the patch passed with JDK Red Hat, Inc.-21.0.12.1+1-LTS |
| +1 :green_heart: | cc | 20m 5s | | the patch passed |
| +1 :green_heart: | javac | 20m 5s | | the patch passed |
| +1 :green_heart: | compile | 20m 49s | | the patch passed with JDK Red Hat, Inc.-17.0.20.1+1-LTS |
| +1 :green_heart: | cc | 20m 49s | | the patch passed |
| +1 :green_heart: | javac | 20m 49s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | mvnsite | 21m 46s | | the patch passed |
| +1 :green_heart: | xmllint | 0m 1s | | No new issues. |
| +1 :green_heart: | javadoc | 10m 38s | | the patch passed with JDK Red Hat, Inc.-21.0.12.1+1-LTS |
| +1 :green_heart: | javadoc | 10m 21s | | the patch passed with JDK Red Hat, Inc.-17.0.20.1+1-LTS |
| +1 :green_heart: | shadedclient | 59m 36s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 844m 55s | [/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8844/2/artifact/out/patch-unit-root.txt) | root in the patch passed. |
| +1 :green_heart: | asflicense | 3m 48s | | The patch does not generate ASF License warnings. |
| | | 1208m 45s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.s3a.prefetch.TestS3ACachingBlockManager |
| | hadoop.yarn.service.TestYarnNativeServices |
| | hadoop.mapreduce.v2.app.webapp.TestAMWebApp |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8844/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8844 |
| Optional Tests | dupname asflicense codespell detsecrets jsonlint compile cc mvnsite javac unit javadoc mvninstall shadedclient xmllint |
| uname | Linux ab52cd7ef2fe 5.15.0-198-generic #208-Ubuntu SMP Fri Sep 4 10:22:38 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 46038bf1feb960049822c6379e6f1fabeb57f3ea |
| Default Java | Red Hat, Inc.-17.0.20.1+1-LTS |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-21.0.12.1.1-1.1.el8_10.x86_64:Red Hat, Inc.-21.0.12.1+1-LTS /usr/lib/jvm/java-17-openjdk-17.0.20.1.1-1.1.el8_10.x86_64:Red Hat, Inc.-17.0.20.1+1-LTS |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8844/2/testReport/ |
| Max. process+thread count | 4111 (vs. ulimit of 10000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui . U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8844/2/console |
| versions | git=2.43.7 maven=3.9.15 xmllint=20907 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-20031. Fix the Windows native build: Docker image, libhdfspp CMake and YARN UI [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8844:
URL: https://github.com/apache/hadoop/pull/8844#issuecomment-6290822604
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 21m 57s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | jsonlint | 0m 0s | | jsonlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 59s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 54m 1s | | trunk passed |
| +1 :green_heart: | compile | 20m 8s | | trunk passed with JDK Red Hat, Inc.-21.0.12.1+1-LTS |
| +1 :green_heart: | compile | 20m 38s | | trunk passed with JDK Red Hat, Inc.-17.0.20.1+1-LTS |
| +1 :green_heart: | mvnsite | 21m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 10m 36s | | trunk passed with JDK Red Hat, Inc.-21.0.12.1+1-LTS |
| +1 :green_heart: | javadoc | 10m 15s | | trunk passed with JDK Red Hat, Inc.-17.0.20.1+1-LTS |
| +1 :green_heart: | shadedclient | 56m 1s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 50s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 53m 7s | | the patch passed |
| +1 :green_heart: | compile | 20m 12s | | the patch passed with JDK Red Hat, Inc.-21.0.12.1+1-LTS |
| +1 :green_heart: | cc | 20m 12s | | the patch passed |
| +1 :green_heart: | javac | 20m 12s | | the patch passed |
| +1 :green_heart: | compile | 20m 15s | | the patch passed with JDK Red Hat, Inc.-17.0.20.1+1-LTS |
| +1 :green_heart: | cc | 20m 15s | | the patch passed |
| +1 :green_heart: | javac | 20m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | mvnsite | 21m 38s | | the patch passed |
| +1 :green_heart: | xmllint | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 10m 38s | | the patch passed with JDK Red Hat, Inc.-21.0.12.1+1-LTS |
| +1 :green_heart: | javadoc | 10m 14s | | the patch passed with JDK Red Hat, Inc.-17.0.20.1+1-LTS |
| +1 :green_heart: | shadedclient | 59m 24s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 839m 54s | [/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8844/1/artifact/out/patch-unit-root.txt) | root in the patch passed. |
| +1 :green_heart: | asflicense | 4m 8s | | The patch does not generate ASF License warnings. |
| | | 1221m 56s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.mapreduce.v2.app.webapp.TestAMWebApp |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8844/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8844 |
| Optional Tests | dupname asflicense codespell detsecrets jsonlint compile cc mvnsite javac unit javadoc mvninstall shadedclient xmllint |
| uname | Linux 54d0e085ad63 5.15.0-185-generic #195-Ubuntu SMP Fri Jun 19 17:11:50 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a8e0a4a52086da57fa2e84b85e3c32f902c955a9 |
| Default Java | Red Hat, Inc.-17.0.20.1+1-LTS |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-21.0.12.1.1-1.1.el8_10.x86_64:Red Hat, Inc.-21.0.12.1+1-LTS /usr/lib/jvm/java-17-openjdk-17.0.20.1.1-1.1.el8_10.x86_64:Red Hat, Inc.-17.0.20.1+1-LTS |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8844/1/testReport/ |
| Max. process+thread count | 4234 (vs. ulimit of 10000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui . U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8844/1/console |
| versions | git=2.43.7 maven=3.9.15 xmllint=20907 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-20031. Fix the Windows native build: Docker image, libhdfspp CMake and YARN UI [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #8844:
URL: https://github.com/apache/hadoop/pull/8844#issuecomment-6350806203
That's good. I have a windows 11 laptop for my college work, see. My PR #8811 updates some of docs related to windows, specifically states how to configure git to check out the repo. I don't look at the rest of the build (which I haven't tried).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-20031. Fix the Windows native build: Docker image, libhdfspp CMake and YARN UI [hadoop]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #8844:
URL: https://github.com/apache/hadoop/pull/8844#discussion_r4316607782
##########
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/CMakeLists.txt:
##########
@@ -151,8 +151,8 @@ endif (NOT THREAD_LOCAL_SUPPORTED)
unset (PROTOC_IS_COMPATIBLE CACHE)
set (CMAKE_REQUIRED_INCLUDES ${PROTOBUF_INCLUDE_DIRS})
set (CMAKE_REQUIRED_LIBRARIES
- ${PROTOBUF_LIBRARY}
- ${PROTOBUF_PROTOC_LIBRARY}
+ protobuf::libprotobuf
+ protobuf::libprotoc
${protobuf_ABSL_USED_TARGETS})
Review Comment:
This check now links against `protobuf::libprotobuf` / `protobuf::libprotoc` imported targets, but still relies on `${PROTOBUF_INCLUDE_DIRS}` for headers. With Config-mode Protobuf, `${PROTOBUF_INCLUDE_DIRS}` may be unset (the target carries include dirs instead), which can break `check_cxx_source_compiles()` by failing to find `<google/protobuf/...>` headers. Prefer deriving required include dirs from the imported target (e.g., via `get_target_property(... INTERFACE_INCLUDE_DIRECTORIES ...)`) or using the correct variable for how Protobuf is found in this build.
##########
dev-support/docker/Dockerfile_windows_10:
##########
@@ -50,54 +50,56 @@ RUN powershell Copy-Item -Recurse -Path 'C:\Program Files\Git' -Destination C:\G
RUN powershell git clone https://github.com/microsoft/vcpkg.git \
&& cd vcpkg \
&& git fetch --all \
- && git checkout 2025.03.19 \
+ && git checkout 2026.07.29 \
&& .\bootstrap-vcpkg.bat
ADD vcpkg/vcpkg.json .
RUN powershell .\vcpkg\vcpkg.exe install --x-install-root .\vcpkg\installed
-# Install Azul Java 8 JDK.
-RUN powershell Invoke-WebRequest -URI https://cdn.azul.com/zulu/bin/zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip -OutFile $Env:TEMP\zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip
-RUN powershell Expand-Archive -Path $Env:TEMP\zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip -DestinationPath "C:\Java"
+# Install Microsoft Build of OpenJDK 17.
+RUN powershell Invoke-WebRequest -URI https://aka.ms/download-jdk/microsoft-jdk-17.0.20.1-windows-x64.msi -OutFile C:\jdk17.msi
+RUN powershell Start-Process msiexec.exe -ArgumentList '/i','C:\jdk17.msi','INSTALLDIR=C:\Java\jdk17\','/quiet','/norestart' -Wait
+RUN powershell if (-not (Test-Path 'C:\Java\jdk17\bin\javac.exe')) { throw 'JDK 17 installation failed' }
+RUN powershell Remove-Item C:\jdk17.msi -Force
# Install Apache Maven.
RUN powershell Invoke-WebRequest -URI https://archive.apache.org/dist/maven/maven-3/3.9.15/binaries/apache-maven-3.9.15-bin.zip -OutFile $Env:TEMP\apache-maven-3.9.15-bin.zip
RUN powershell Expand-Archive -Path $Env:TEMP\apache-maven-3.9.15-bin.zip -DestinationPath "C:\Maven"
-# Install CMake 3.19.0.
-RUN powershell Invoke-WebRequest -URI https://cmake.org/files/v3.19/cmake-3.19.0-win64-x64.zip -OutFile $Env:TEMP\cmake-3.19.0-win64-x64.zip
-RUN powershell Expand-Archive -Path $Env:TEMP\cmake-3.19.0-win64-x64.zip -DestinationPath "C:\CMake"
+# Install CMake 3.30.2.
+RUN powershell Invoke-WebRequest -URI https://cmake.org/files/v3.30/cmake-3.30.2-windows-x86_64.zip -OutFile $Env:TEMP\cmake-3.30.2-windows-x86_64.zip
Review Comment:
These downloads/install steps don’t verify integrity (e.g., SHA-256) before executing installers/unzipping. This increases supply-chain risk for the build image. Consider adding checksum verification (or signature validation where applicable) for the JDK MSI and CMake ZIP before installation/extraction.
##########
dev-support/docker/vcpkg/vcpkg.json:
##########
@@ -1,16 +1,239 @@
{
"$schema": "https://raw.githubusercontent.com/microsoft/vcpkg-tool/main/docs/vcpkg.schema.json",
"dependencies": [
- "boost",
+ "boost-asio",
+ "boost-date-time",
+ "boost-system",
+ "boost-program-options",
"protobuf",
"openssl",
"zlib"
],
- "builtin-baseline": "289a69379604112a433874fe8b9812dad3103341",
+ "builtin-baseline": "9e593bb18ea69cc5095e012465dcd675a822ed0d",
"overrides": [
{
"name": "protobuf",
- "version": "3.25.5"
+ "version": "4.25.1"
Review Comment:
This pins Windows Protobuf to `4.25.1`, while the PR description notes Linux pins Protobuf `3.25.5` in-repo. Having different Protobuf major versions across platforms can lead to divergent build behavior and harder-to-debug inconsistencies in native components. Consider aligning the Protobuf version across platforms (or explicitly documenting why Windows must use a newer major and validating that the resulting native artifacts remain compatible with project expectations).
##########
dev-support/docker/vcpkg/vcpkg.json:
##########
@@ -1,16 +1,239 @@
{
"$schema": "https://raw.githubusercontent.com/microsoft/vcpkg-tool/main/docs/vcpkg.schema.json",
"dependencies": [
- "boost",
+ "boost-asio",
+ "boost-date-time",
+ "boost-system",
+ "boost-program-options",
Review Comment:
The manifest now pins Boost by enumerating dozens of `boost-*` overrides, which is hard to maintain as vcpkg port relationships evolve. If feasible, consider selecting a `builtin-baseline` commit where Boost is already 1.86.0, so you can drop most/all Boost overrides and only keep overrides for ports that truly must diverge from that baseline (reducing long-term churn and review surface).
##########
dev-support/docker/vcpkg/vcpkg.json:
##########
@@ -1,16 +1,239 @@
{
"$schema": "https://raw.githubusercontent.com/microsoft/vcpkg-tool/main/docs/vcpkg.schema.json",
"dependencies": [
- "boost",
+ "boost-asio",
+ "boost-date-time",
+ "boost-system",
+ "boost-program-options",
"protobuf",
"openssl",
"zlib"
],
- "builtin-baseline": "289a69379604112a433874fe8b9812dad3103341",
+ "builtin-baseline": "9e593bb18ea69cc5095e012465dcd675a822ed0d",
"overrides": [
{
"name": "protobuf",
- "version": "3.25.5"
+ "version": "4.25.1"
+ },
+ {
+ "name": "boost-algorithm",
+ "version": "1.86.0"
+ },
Review Comment:
The manifest now pins Boost by enumerating dozens of `boost-*` overrides, which is hard to maintain as vcpkg port relationships evolve. If feasible, consider selecting a `builtin-baseline` commit where Boost is already 1.86.0, so you can drop most/all Boost overrides and only keep overrides for ports that truly must diverge from that baseline (reducing long-term churn and review surface).
##########
dev-support/docker/Dockerfile_windows_10:
##########
@@ -50,54 +50,56 @@ RUN powershell Copy-Item -Recurse -Path 'C:\Program Files\Git' -Destination C:\G
RUN powershell git clone https://github.com/microsoft/vcpkg.git \
&& cd vcpkg \
&& git fetch --all \
- && git checkout 2025.03.19 \
+ && git checkout 2026.07.29 \
&& .\bootstrap-vcpkg.bat
ADD vcpkg/vcpkg.json .
RUN powershell .\vcpkg\vcpkg.exe install --x-install-root .\vcpkg\installed
-# Install Azul Java 8 JDK.
-RUN powershell Invoke-WebRequest -URI https://cdn.azul.com/zulu/bin/zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip -OutFile $Env:TEMP\zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip
-RUN powershell Expand-Archive -Path $Env:TEMP\zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip -DestinationPath "C:\Java"
+# Install Microsoft Build of OpenJDK 17.
+RUN powershell Invoke-WebRequest -URI https://aka.ms/download-jdk/microsoft-jdk-17.0.20.1-windows-x64.msi -OutFile C:\jdk17.msi
+RUN powershell Start-Process msiexec.exe -ArgumentList '/i','C:\jdk17.msi','INSTALLDIR=C:\Java\jdk17\','/quiet','/norestart' -Wait
Review Comment:
These downloads/install steps don’t verify integrity (e.g., SHA-256) before executing installers/unzipping. This increases supply-chain risk for the build image. Consider adding checksum verification (or signature validation where applicable) for the JDK MSI and CMake ZIP before installation/extraction.
##########
dev-support/docker/Dockerfile_windows_10:
##########
@@ -125,6 +127,13 @@ RUN powershell -Command Start-Process -FilePath .\vcredist_x64.exe -ArgumentList
RUN powershell -Command Remove-Item vcredist_x64.exe
# Create a user HadoopBuilder with basic privileges and use it for building Hadoop on Windows.
+# Windows Server Core enables password complexity by default (unlike the
+# Windows client image), which rejects the passwordless build account below.
+RUN powershell secedit /export /cfg "C:\pwpol.cfg"
+RUN powershell "(Get-Content C:\pwpol.cfg).Replace('PasswordComplexity = 1', 'PasswordComplexity = 0') | Out-File C:\pwpol.cfg"
+RUN powershell secedit /configure /db "C:\windows\security\local.sdb" /cfg "C:\pwpol.cfg"
+RUN powershell Remove-Item -Force "C:\pwpol.cfg" -Confirm:$false
+
RUN powershell New-LocalUser -Name 'HadoopBuilder' -Description 'User account for building Apache Hadoop' -Password ([securestring]::new()) -AccountNeverExpires -PasswordNeverExpires
Review Comment:
Disabling system-wide password complexity in the image weakens the container’s local security posture beyond just the build user. A safer alternative is to keep the policy enabled and create `HadoopBuilder` with a compliant strong random password (the `USER` directive does not require the password at runtime), avoiding the need to relax the policy.
##########
BUILDING.txt:
##########
@@ -603,12 +603,29 @@ still be immensely useful as a raw guide for all the steps involved in creating
needed to build Hadoop on Windows 10.
Building using the Docker:
+Docker Desktop must first be switched to Windows containers. In the default Linux container mode
+every command below fails immediately with "no matching manifest for linux/amd64".
+> "%ProgramFiles%\Docker\Docker\DockerCli.exe" -SwitchWindowsEngine
+
+Wait for the engine to restart, then confirm - this must print "windows" before proceeding.
+> docker info --format "{{.OSType}}"
+
We first need to build the Docker image for building Hadoop on Windows 10. Run this command from
the root of the Hadoop repository.
-> docker build -t hadoop-windows-10-builder -f .\dev-support\docker\Dockerfile_windows_10 .\dev-support\docker\
+> docker build -t hadoop-windows-10-builder -m 8g -f .\dev-support\docker\Dockerfile_windows_10 .\dev-support\docker\
+
+The -m 8g is needed because the vcpkg step compiles Boost, Protocol Buffers, OpenSSL and Zlib from
+source and the default container memory is too small. Expect the image build to take a few hours
+and the resulting image to be around 23 GB. Note that -m and --isolation are options of the classic
+builder, so leave DOCKER_BUILDKIT unset; BuildKit does not accept them.
Review Comment:
The instruction to 'leave DOCKER_BUILDKIT unset' is easy to misapply if users already have it set globally. Consider adding an explicit command example for both cmd.exe and PowerShell to disable it (e.g., setting it to 0 / clearing it) before running `docker build`, to reduce copy/paste failures.
##########
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/CMakeLists.txt:
##########
@@ -171,6 +171,13 @@ if (NOT PROTOC_IS_COMPATIBLE)
compiler are compatible with each other.")
endif (NOT PROTOC_IS_COMPATIBLE)
+# The checks above leave CMAKE_REQUIRED_INCLUDES and CMAKE_REQUIRED_LIBRARIES
+# pointing at the Protobuf and Abseil targets. Abseil carries C++ feature
+# requirements, which a C-language check_symbol_exists() below cannot satisfy
+# under the Visual Studio generator. Reset them so later checks run clean.
+unset (CMAKE_REQUIRED_INCLUDES)
+unset (CMAKE_REQUIRED_LIBRARIES)
Review Comment:
Unsetting `CMAKE_REQUIRED_*` after the compatibility check changes global state for the rest of this CMakeLists. To avoid unintended interactions with other checks that might have set these variables earlier, consider saving the previous values before the Protobuf check and restoring them afterward (rather than unsetting unconditionally).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-20031. Fix the Windows native build: Docker image, libhdfspp CMake and YARN UI [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8844:
URL: https://github.com/apache/hadoop/pull/8844#issuecomment-6391611071
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 18m 54s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | jsonlint | 0m 0s | | jsonlint was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 0m 49s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 46m 19s | | trunk passed |
| +1 :green_heart: | compile | 18m 15s | | trunk passed |
| +1 :green_heart: | mvnsite | 18m 48s | | trunk passed |
| +1 :green_heart: | javadoc | 10m 9s | | trunk passed |
| +1 :green_heart: | shadedclient | 129m 32s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 49m 51s | | the patch passed |
| +1 :green_heart: | compile | 17m 36s | | the patch passed |
| +1 :green_heart: | cc | 17m 36s | | the patch passed |
| +1 :green_heart: | javac | 17m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | mvnsite | 18m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 10m 2s | | the patch passed |
| +1 :green_heart: | shadedclient | 64m 39s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 944m 58s | [/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8844/2/artifact/out/patch-unit-root.txt) | root in the patch passed. |
| +1 :green_heart: | asflicense | 1m 57s | | The patch does not generate ASF License warnings. |
| | | 1230m 29s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | TEST-cetest |
| | hadoop.yarn.server.nodemanager.containermanager.resourceplugin.fpga.TestFpgaDiscoverer |
| | hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageDomain |
| | hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage |
| | hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageSchema |
| | hadoop.yarn.server.timelineservice.storage.TestTimelineWriterHBaseDown |
| | hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities |
| | hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps |
| | hadoop.yarn.server.timelineservice.storage.TestTimelineReaderHBaseDown |
| | hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun |
| | hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction |
| | hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8844/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8844 |
| Optional Tests | dupname asflicense codespell detsecrets jsonlint compile cc mvnsite javac unit javadoc mvninstall shadedclient xmllint |
| uname | Linux 611237952c76 5.15.0-198-generic #208-Ubuntu SMP Fri Sep 4 10:22:38 UTC 2026 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 46038bf1feb960049822c6379e6f1fabeb57f3ea |
| Default Java | Debian-25.0.4.1+1-1-deb13u1-Debian |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8844/2/testReport/ |
| Max. process+thread count | 4200 (vs. ulimit of 10000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui . U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8844/2/console |
| versions | git=2.47.3 maven=3.9.15 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-20031. Fix the Windows native build: Docker image, libhdfspp CMake and YARN UI [hadoop]
Posted by "GauthamBanasandra (via GitHub)" <gi...@apache.org>.
GauthamBanasandra commented on code in PR #8844:
URL: https://github.com/apache/hadoop/pull/8844#discussion_r4426603619
##########
dev-support/docker/Dockerfile_windows_10:
##########
@@ -50,54 +50,56 @@ RUN powershell Copy-Item -Recurse -Path 'C:\Program Files\Git' -Destination C:\G
RUN powershell git clone https://github.com/microsoft/vcpkg.git \
&& cd vcpkg \
&& git fetch --all \
- && git checkout 2025.03.19 \
+ && git checkout 2026.07.29 \
&& .\bootstrap-vcpkg.bat
ADD vcpkg/vcpkg.json .
RUN powershell .\vcpkg\vcpkg.exe install --x-install-root .\vcpkg\installed
-# Install Azul Java 8 JDK.
-RUN powershell Invoke-WebRequest -URI https://cdn.azul.com/zulu/bin/zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip -OutFile $Env:TEMP\zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip
-RUN powershell Expand-Archive -Path $Env:TEMP\zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip -DestinationPath "C:\Java"
+# Install Microsoft Build of OpenJDK 17.
+RUN powershell Invoke-WebRequest -URI https://aka.ms/download-jdk/microsoft-jdk-17.0.20.1-windows-x64.msi -OutFile C:\jdk17.msi
+RUN powershell Start-Process msiexec.exe -ArgumentList '/i','C:\jdk17.msi','INSTALLDIR=C:\Java\jdk17\','/quiet','/norestart' -Wait
Review Comment:
Added in 46038bf1. All nine downloaded artifacts now verify SHA-512 via `Get-FileHash`
before use, matching the pattern `pkg-resolver/install-pip.ps1` already uses for `get-pip.py`.
Where the publisher ships its own digest I checked against it before deriving the SHA-512,
so these aren't just self-attested: Microsoft publishes SHA-256 for the JDK, cmake.org for
CMake, and the ASF publishes SHA-512 for Maven. All three matched. Every gate was exercised
in a full image rebuild.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-20031. Fix the Windows native build: Docker image, libhdfspp CMake and YARN UI [hadoop]
Posted by "GauthamBanasandra (via GitHub)" <gi...@apache.org>.
GauthamBanasandra merged PR #8844:
URL: https://github.com/apache/hadoop/pull/8844
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-20031. Fix the Windows native build: Docker image, libhdfspp CMake and YARN UI [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on code in PR #8844:
URL: https://github.com/apache/hadoop/pull/8844#discussion_r4349217398
##########
dev-support/docker/Dockerfile_windows_10:
##########
@@ -50,54 +50,56 @@ RUN powershell Copy-Item -Recurse -Path 'C:\Program Files\Git' -Destination C:\G
RUN powershell git clone https://github.com/microsoft/vcpkg.git \
&& cd vcpkg \
&& git fetch --all \
- && git checkout 2025.03.19 \
+ && git checkout 2026.07.29 \
&& .\bootstrap-vcpkg.bat
ADD vcpkg/vcpkg.json .
RUN powershell .\vcpkg\vcpkg.exe install --x-install-root .\vcpkg\installed
-# Install Azul Java 8 JDK.
-RUN powershell Invoke-WebRequest -URI https://cdn.azul.com/zulu/bin/zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip -OutFile $Env:TEMP\zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip
-RUN powershell Expand-Archive -Path $Env:TEMP\zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip -DestinationPath "C:\Java"
+# Install Microsoft Build of OpenJDK 17.
+RUN powershell Invoke-WebRequest -URI https://aka.ms/download-jdk/microsoft-jdk-17.0.20.1-windows-x64.msi -OutFile C:\jdk17.msi
+RUN powershell Start-Process msiexec.exe -ArgumentList '/i','C:\jdk17.msi','INSTALLDIR=C:\Java\jdk17\','/quiet','/norestart' -Wait
Review Comment:
Yes, Please can we add a sha checksum stage here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-20031. Fix the Windows native build: Docker image, libhdfspp CMake and YARN UI [hadoop]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #8844:
URL: https://github.com/apache/hadoop/pull/8844#discussion_r4355509784
##########
dev-support/docker/Dockerfile_windows_10:
##########
@@ -50,54 +50,56 @@ RUN powershell Copy-Item -Recurse -Path 'C:\Program Files\Git' -Destination C:\G
RUN powershell git clone https://github.com/microsoft/vcpkg.git \
&& cd vcpkg \
&& git fetch --all \
- && git checkout 2025.03.19 \
+ && git checkout 2026.07.29 \
&& .\bootstrap-vcpkg.bat
ADD vcpkg/vcpkg.json .
RUN powershell .\vcpkg\vcpkg.exe install --x-install-root .\vcpkg\installed
-# Install Azul Java 8 JDK.
-RUN powershell Invoke-WebRequest -URI https://cdn.azul.com/zulu/bin/zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip -OutFile $Env:TEMP\zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip
-RUN powershell Expand-Archive -Path $Env:TEMP\zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip -DestinationPath "C:\Java"
+# Install Microsoft Build of OpenJDK 17.
+RUN powershell Invoke-WebRequest -URI https://aka.ms/download-jdk/microsoft-jdk-17.0.20.1-windows-x64.msi -OutFile C:\jdk17.msi
+RUN powershell Start-Process msiexec.exe -ArgumentList '/i','C:\jdk17.msi','INSTALLDIR=C:\Java\jdk17\','/quiet','/norestart' -Wait
+RUN powershell if (-not (Test-Path 'C:\Java\jdk17\bin\javac.exe')) { throw 'JDK 17 installation failed' }
+RUN powershell Remove-Item C:\jdk17.msi -Force
# Install Apache Maven.
RUN powershell Invoke-WebRequest -URI https://archive.apache.org/dist/maven/maven-3/3.9.15/binaries/apache-maven-3.9.15-bin.zip -OutFile $Env:TEMP\apache-maven-3.9.15-bin.zip
RUN powershell Expand-Archive -Path $Env:TEMP\apache-maven-3.9.15-bin.zip -DestinationPath "C:\Maven"
-# Install CMake 3.19.0.
-RUN powershell Invoke-WebRequest -URI https://cmake.org/files/v3.19/cmake-3.19.0-win64-x64.zip -OutFile $Env:TEMP\cmake-3.19.0-win64-x64.zip
-RUN powershell Expand-Archive -Path $Env:TEMP\cmake-3.19.0-win64-x64.zip -DestinationPath "C:\CMake"
+# Install CMake 3.30.2.
+RUN powershell Invoke-WebRequest -URI https://cmake.org/files/v3.30/cmake-3.30.2-windows-x86_64.zip -OutFile $Env:TEMP\cmake-3.30.2-windows-x86_64.zip
+RUN powershell Expand-Archive -Path $Env:TEMP\cmake-3.30.2-windows-x86_64.zip -DestinationPath "C:\CMake"
# Install zstd 1.5.4.
RUN powershell Invoke-WebRequest -Uri https://github.com/facebook/zstd/releases/download/v1.5.4/zstd-v1.5.4-win64.zip -OutFile $Env:TEMP\zstd-v1.5.4-win64.zip
RUN powershell Expand-Archive -Path $Env:TEMP\zstd-v1.5.4-win64.zip -DestinationPath "C:\ZStd"
RUN setx PATH "%PATH%;C:\ZStd"
-# Install libopenssl 3.5.2-1 needed for rsync 3.2.7.
+# Install libopenssl 3.5.2-1 needed for rsync 3.5.1.
RUN powershell Invoke-WebRequest -Uri https://repo.msys2.org/msys/x86_64/libopenssl-3.5.2-1-x86_64.pkg.tar.zst -OutFile $Env:TEMP\libopenssl-3.5.2-1-x86_64.pkg.tar.zst
RUN powershell zstd -d $Env:TEMP\libopenssl-3.5.2-1-x86_64.pkg.tar.zst -o $Env:TEMP\libopenssl-3.5.2-1-x86_64.pkg.tar
RUN powershell mkdir "C:\LibOpenSSL"
RUN powershell tar -xvf $Env:TEMP\libopenssl-3.5.2-1-x86_64.pkg.tar -C "C:\LibOpenSSL"
-# Install libxxhash 0.8.3 needed for rsync 3.2.7.
+# Install libxxhash 0.8.3 needed for rsync 3.5.1.
RUN powershell Invoke-WebRequest -Uri https://repo.msys2.org/msys/x86_64/libxxhash-0.8.3-1-x86_64.pkg.tar.zst -OutFile $Env:TEMP\libxxhash-0.8.3-1-x86_64.pkg.tar.zst
RUN powershell zstd -d $Env:TEMP\libxxhash-0.8.3-1-x86_64.pkg.tar.zst -o $Env:TEMP\libxxhash-0.8.3-1-x86_64.pkg.tar
RUN powershell mkdir "C:\LibXXHash"
RUN powershell tar -xvf $Env:TEMP\libxxhash-0.8.3-1-x86_64.pkg.tar -C "C:\LibXXHash"
-# Install libzstd 1.5.4 needed for rsync 3.2.7.
-RUN powershell Invoke-WebRequest -Uri https://repo.msys2.org/msys/x86_64/libzstd-1.5.5-1-x86_64.pkg.tar.zst -OutFile $Env:TEMP\libzstd-1.5.5-1-x86_64.pkg.tar.zst
-RUN powershell zstd -d $Env:TEMP\libzstd-1.5.5-1-x86_64.pkg.tar.zst -o $Env:TEMP\libzstd-1.5.5-1-x86_64.pkg.tar
+# Install libzstd 1.5.7 needed for rsync 3.5.1.
+RUN powershell Invoke-WebRequest -Uri https://repo.msys2.org/msys/x86_64/libzstd-1.5.7-1-x86_64.pkg.tar.zst -OutFile $Env:TEMP\libzstd-1.5.7-1-x86_64.pkg.tar.zst
Review Comment:
we don't need zstd native lib since HADOOP-19855 (https://github.com/apache/hadoop/pull/8399)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-20031. Fix the Windows native build: Docker image, libhdfspp CMake and YARN UI [hadoop]
Posted by "GauthamBanasandra (via GitHub)" <gi...@apache.org>.
GauthamBanasandra commented on code in PR #8844:
URL: https://github.com/apache/hadoop/pull/8844#discussion_r4426617820
##########
dev-support/docker/Dockerfile_windows_10:
##########
@@ -50,54 +50,56 @@ RUN powershell Copy-Item -Recurse -Path 'C:\Program Files\Git' -Destination C:\G
RUN powershell git clone https://github.com/microsoft/vcpkg.git \
&& cd vcpkg \
&& git fetch --all \
- && git checkout 2025.03.19 \
+ && git checkout 2026.07.29 \
&& .\bootstrap-vcpkg.bat
ADD vcpkg/vcpkg.json .
RUN powershell .\vcpkg\vcpkg.exe install --x-install-root .\vcpkg\installed
-# Install Azul Java 8 JDK.
-RUN powershell Invoke-WebRequest -URI https://cdn.azul.com/zulu/bin/zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip -OutFile $Env:TEMP\zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip
-RUN powershell Expand-Archive -Path $Env:TEMP\zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip -DestinationPath "C:\Java"
+# Install Microsoft Build of OpenJDK 17.
+RUN powershell Invoke-WebRequest -URI https://aka.ms/download-jdk/microsoft-jdk-17.0.20.1-windows-x64.msi -OutFile C:\jdk17.msi
+RUN powershell Start-Process msiexec.exe -ArgumentList '/i','C:\jdk17.msi','INSTALLDIR=C:\Java\jdk17\','/quiet','/norestart' -Wait
+RUN powershell if (-not (Test-Path 'C:\Java\jdk17\bin\javac.exe')) { throw 'JDK 17 installation failed' }
+RUN powershell Remove-Item C:\jdk17.msi -Force
# Install Apache Maven.
RUN powershell Invoke-WebRequest -URI https://archive.apache.org/dist/maven/maven-3/3.9.15/binaries/apache-maven-3.9.15-bin.zip -OutFile $Env:TEMP\apache-maven-3.9.15-bin.zip
RUN powershell Expand-Archive -Path $Env:TEMP\apache-maven-3.9.15-bin.zip -DestinationPath "C:\Maven"
-# Install CMake 3.19.0.
-RUN powershell Invoke-WebRequest -URI https://cmake.org/files/v3.19/cmake-3.19.0-win64-x64.zip -OutFile $Env:TEMP\cmake-3.19.0-win64-x64.zip
-RUN powershell Expand-Archive -Path $Env:TEMP\cmake-3.19.0-win64-x64.zip -DestinationPath "C:\CMake"
+# Install CMake 3.30.2.
+RUN powershell Invoke-WebRequest -URI https://cmake.org/files/v3.30/cmake-3.30.2-windows-x86_64.zip -OutFile $Env:TEMP\cmake-3.30.2-windows-x86_64.zip
+RUN powershell Expand-Archive -Path $Env:TEMP\cmake-3.30.2-windows-x86_64.zip -DestinationPath "C:\CMake"
# Install zstd 1.5.4.
RUN powershell Invoke-WebRequest -Uri https://github.com/facebook/zstd/releases/download/v1.5.4/zstd-v1.5.4-win64.zip -OutFile $Env:TEMP\zstd-v1.5.4-win64.zip
RUN powershell Expand-Archive -Path $Env:TEMP\zstd-v1.5.4-win64.zip -DestinationPath "C:\ZStd"
RUN setx PATH "%PATH%;C:\ZStd"
-# Install libopenssl 3.5.2-1 needed for rsync 3.2.7.
+# Install libopenssl 3.5.2-1 needed for rsync 3.5.1.
RUN powershell Invoke-WebRequest -Uri https://repo.msys2.org/msys/x86_64/libopenssl-3.5.2-1-x86_64.pkg.tar.zst -OutFile $Env:TEMP\libopenssl-3.5.2-1-x86_64.pkg.tar.zst
RUN powershell zstd -d $Env:TEMP\libopenssl-3.5.2-1-x86_64.pkg.tar.zst -o $Env:TEMP\libopenssl-3.5.2-1-x86_64.pkg.tar
RUN powershell mkdir "C:\LibOpenSSL"
RUN powershell tar -xvf $Env:TEMP\libopenssl-3.5.2-1-x86_64.pkg.tar -C "C:\LibOpenSSL"
-# Install libxxhash 0.8.3 needed for rsync 3.2.7.
+# Install libxxhash 0.8.3 needed for rsync 3.5.1.
RUN powershell Invoke-WebRequest -Uri https://repo.msys2.org/msys/x86_64/libxxhash-0.8.3-1-x86_64.pkg.tar.zst -OutFile $Env:TEMP\libxxhash-0.8.3-1-x86_64.pkg.tar.zst
RUN powershell zstd -d $Env:TEMP\libxxhash-0.8.3-1-x86_64.pkg.tar.zst -o $Env:TEMP\libxxhash-0.8.3-1-x86_64.pkg.tar
RUN powershell mkdir "C:\LibXXHash"
RUN powershell tar -xvf $Env:TEMP\libxxhash-0.8.3-1-x86_64.pkg.tar -C "C:\LibXXHash"
-# Install libzstd 1.5.4 needed for rsync 3.2.7.
-RUN powershell Invoke-WebRequest -Uri https://repo.msys2.org/msys/x86_64/libzstd-1.5.5-1-x86_64.pkg.tar.zst -OutFile $Env:TEMP\libzstd-1.5.5-1-x86_64.pkg.tar.zst
-RUN powershell zstd -d $Env:TEMP\libzstd-1.5.5-1-x86_64.pkg.tar.zst -o $Env:TEMP\libzstd-1.5.5-1-x86_64.pkg.tar
+# Install libzstd 1.5.7 needed for rsync 3.5.1.
+RUN powershell Invoke-WebRequest -Uri https://repo.msys2.org/msys/x86_64/libzstd-1.5.7-1-x86_64.pkg.tar.zst -OutFile $Env:TEMP\libzstd-1.5.7-1-x86_64.pkg.tar.zst
Review Comment:
Thanks for the review @pan3793.
You're right about HADOOP-19855, and it's already on trunk beneath this branch — there's no
zstd left in any CMakeLists.txt, config.h.cmake or pom other than `zstd-jni`, and nothing
consumes `-Drequire.zstd` any more. This PR doesn't re-add it: no zstd in `vcpkg.json`, and
the mvn command in BUILDING.txt only passes `-Drequire.openssl -Drequire.test.libhadoop`.
This line is a different thing though — it's the MSYS2 `libzstd` package, which ships just
`/usr/bin/msys-zstd-1.dll` and is a declared runtime dependency of MSYS2's `rsync.exe`
(packages.msys2.org lists rsync 3.5.1-2's deps as libiconv, libidn2, liblz4, libopenssl,
libxxhash, libzstd, popt). Line 105 copies it next to `rsync.exe` for that reason, and
`create-release` depends on rsync working.
The separate `C:\ZStd` CLI has to stay too: `zstd -d` unpacks all four `.pkg.tar.zst`
archives, including libopenssl and libxxhash, which aren't zstd-related at all.
These lines are only in the diff because repo.msys2.org serves only the current version of
each package, so trunk's pinned URLs now 404 and the image can't build at all. I've added a
comment at the zstd step noting it's an unpacking tool rather than the codec, to make this
less confusing next time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-20031. Fix the Windows native build: Docker image, libhdfspp CMake and YARN UI [hadoop]
Posted by "GauthamBanasandra (via GitHub)" <gi...@apache.org>.
GauthamBanasandra commented on code in PR #8844:
URL: https://github.com/apache/hadoop/pull/8844#discussion_r4426560161
##########
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/CMakeLists.txt:
##########
@@ -151,8 +151,8 @@ endif (NOT THREAD_LOCAL_SUPPORTED)
unset (PROTOC_IS_COMPATIBLE CACHE)
set (CMAKE_REQUIRED_INCLUDES ${PROTOBUF_INCLUDE_DIRS})
set (CMAKE_REQUIRED_LIBRARIES
- ${PROTOBUF_LIBRARY}
- ${PROTOBUF_PROTOC_LIBRARY}
+ protobuf::libprotobuf
+ protobuf::libprotoc
${protobuf_ABSL_USED_TARGETS})
Review Comment:
Checked this one and it doesn't bite here, though you're right about the premise.
We do resolve Protobuf in Config mode: the Windows build runs under vcpkg's toolchain, and `share/protobuf/vcpkg-cmake-wrapper.cmake` forces `_find_package(${ARGS} CONFIG)`, so CMake's own `FindProtobuf.cmake` never runs.
`PROTOBUF_INCLUDE_DIRS` is still populated, though. vcpkg's protobuf portfile rewrites `if(protobuf_MODULE_COMPATIBLE)` to always-true in the installed `protobuf-config.cmake`, which unconditionally loads `protobuf-module.cmake` -- and that file does exactly what you suggest, `get_target_property(Protobuf_INCLUDE_DIRS protobuf::libprotobuf INTERFACE_INCLUDE_DIRECTORIES)`, then mirrors it to the uppercase name.
Independently of that, an imported target placed in `CMAKE_REQUIRED_LIBRARIES` propagates its `INTERFACE_INCLUDE_DIRECTORIES` into `check_cxx_source_compiles()`, so the check would still find the headers even if the variable were empty. Confirmed by a full native build on CMake 3.30.2 with VS 2019. No change made.
##########
dev-support/docker/Dockerfile_windows_10:
##########
@@ -50,54 +50,56 @@ RUN powershell Copy-Item -Recurse -Path 'C:\Program Files\Git' -Destination C:\G
RUN powershell git clone https://github.com/microsoft/vcpkg.git \
&& cd vcpkg \
&& git fetch --all \
- && git checkout 2025.03.19 \
+ && git checkout 2026.07.29 \
&& .\bootstrap-vcpkg.bat
ADD vcpkg/vcpkg.json .
RUN powershell .\vcpkg\vcpkg.exe install --x-install-root .\vcpkg\installed
-# Install Azul Java 8 JDK.
-RUN powershell Invoke-WebRequest -URI https://cdn.azul.com/zulu/bin/zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip -OutFile $Env:TEMP\zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip
-RUN powershell Expand-Archive -Path $Env:TEMP\zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip -DestinationPath "C:\Java"
+# Install Microsoft Build of OpenJDK 17.
+RUN powershell Invoke-WebRequest -URI https://aka.ms/download-jdk/microsoft-jdk-17.0.20.1-windows-x64.msi -OutFile C:\jdk17.msi
+RUN powershell Start-Process msiexec.exe -ArgumentList '/i','C:\jdk17.msi','INSTALLDIR=C:\Java\jdk17\','/quiet','/norestart' -Wait
+RUN powershell if (-not (Test-Path 'C:\Java\jdk17\bin\javac.exe')) { throw 'JDK 17 installation failed' }
+RUN powershell Remove-Item C:\jdk17.msi -Force
# Install Apache Maven.
RUN powershell Invoke-WebRequest -URI https://archive.apache.org/dist/maven/maven-3/3.9.15/binaries/apache-maven-3.9.15-bin.zip -OutFile $Env:TEMP\apache-maven-3.9.15-bin.zip
RUN powershell Expand-Archive -Path $Env:TEMP\apache-maven-3.9.15-bin.zip -DestinationPath "C:\Maven"
-# Install CMake 3.19.0.
-RUN powershell Invoke-WebRequest -URI https://cmake.org/files/v3.19/cmake-3.19.0-win64-x64.zip -OutFile $Env:TEMP\cmake-3.19.0-win64-x64.zip
-RUN powershell Expand-Archive -Path $Env:TEMP\cmake-3.19.0-win64-x64.zip -DestinationPath "C:\CMake"
+# Install CMake 3.30.2.
+RUN powershell Invoke-WebRequest -URI https://cmake.org/files/v3.30/cmake-3.30.2-windows-x86_64.zip -OutFile $Env:TEMP\cmake-3.30.2-windows-x86_64.zip
Review Comment:
Done in 46038bf1. All nine downloaded artifacts now verify SHA-512 with `Get-FileHash` before they are used, following the pattern `pkg-resolver/install-pip.ps1` already uses for `get-pip.py`.
Where the publisher distributes its own digest I cross-checked against it before deriving the SHA-512, so the pinned values are not merely self-attested: Microsoft publishes SHA-256 for the JDK, cmake.org for CMake, and the ASF publishes SHA-512 for Maven. All three matched.
All nine gates were exercised in a full image rebuild.
One gap that cannot be closed the same way: `vs_buildtools.exe` comes from `aka.ms/vs/16/release/`, a rolling bootstrapper with no stable content and no published digest, so there is nothing to pin it to.
##########
dev-support/docker/Dockerfile_windows_10:
##########
@@ -125,6 +127,13 @@ RUN powershell -Command Start-Process -FilePath .\vcredist_x64.exe -ArgumentList
RUN powershell -Command Remove-Item vcredist_x64.exe
# Create a user HadoopBuilder with basic privileges and use it for building Hadoop on Windows.
+# Windows Server Core enables password complexity by default (unlike the
+# Windows client image), which rejects the passwordless build account below.
+RUN powershell secedit /export /cfg "C:\pwpol.cfg"
+RUN powershell "(Get-Content C:\pwpol.cfg).Replace('PasswordComplexity = 1', 'PasswordComplexity = 0') | Out-File C:\pwpol.cfg"
+RUN powershell secedit /configure /db "C:\windows\security\local.sdb" /cfg "C:\pwpol.cfg"
+RUN powershell Remove-Item -Force "C:\pwpol.cfg" -Confirm:$false
+
RUN powershell New-LocalUser -Name 'HadoopBuilder' -Description 'User account for building Apache Hadoop' -Password ([securestring]::new()) -AccountNeverExpires -PasswordNeverExpires
Review Comment:
Fixed in 46038bf1, though not by the suggested route.
The policy is now restored to `PasswordComplexity = 1` once `HadoopBuilder` has been created, so the shipped image is no longer left with a relaxed policy. The restore folds into the `secedit` block that already runs immediately afterwards, so it adds no extra layer.
I did not give the account a strong password instead. The load-bearing claim in the suggestion -- that `USER` never needs the password at runtime -- does not hold reliably for Windows containers: a non-builtin local account that *has* a password can fail at process creation under `USER`, and the passwordless account is what makes `USER HadoopBuilder` work here.
Verified by reading the policy back out of the finished image: `PasswordComplexity = 1`, and `SeCreateSymbolicLinkPrivilege = HadoopBuilder,*S-1-5-32-544` still granted.
##########
dev-support/docker/vcpkg/vcpkg.json:
##########
@@ -1,16 +1,239 @@
{
"$schema": "https://raw.githubusercontent.com/microsoft/vcpkg-tool/main/docs/vcpkg.schema.json",
"dependencies": [
- "boost",
+ "boost-asio",
+ "boost-date-time",
+ "boost-system",
+ "boost-program-options",
Review Comment:
I looked into moving `builtin-baseline` instead, and it does not work here.
The MSYS2 breakage this PR fixes comes from `scripts/cmake/vcpkg_acquire_msys.cmake`, which ships with the checked-out vcpkg **tag**, not with `builtin-baseline`. The older tags pin `msys2-runtime-3.5.4-2`, which is now a 404 on the mirror. So the tag has to move forward regardless, and keeping the new tag (2026.07.29) while pinning Boost explicitly was the combination I could verify end to end.
That said, this comment found a real bug, so thank you. `boost-coroutine` was missing from the override list, so it alone resolved to the baseline and installed at **1.91.0** while the other 55 ports sat at 1.86.0 -- a mixed Boost that happened to build anyway. It is now pinned, and I verified against vcpkg's own package inventory in the rebuilt image that no Boost port is off 1.86.0.
##########
dev-support/docker/vcpkg/vcpkg.json:
##########
@@ -1,16 +1,239 @@
{
"$schema": "https://raw.githubusercontent.com/microsoft/vcpkg-tool/main/docs/vcpkg.schema.json",
"dependencies": [
- "boost",
+ "boost-asio",
+ "boost-date-time",
+ "boost-system",
+ "boost-program-options",
"protobuf",
"openssl",
"zlib"
],
- "builtin-baseline": "289a69379604112a433874fe8b9812dad3103341",
+ "builtin-baseline": "9e593bb18ea69cc5095e012465dcd675a822ed0d",
"overrides": [
{
"name": "protobuf",
- "version": "3.25.5"
+ "version": "4.25.1"
Review Comment:
These are not different major versions -- the two numbers are per-language aliases for the same upstream release.
Protobuf publishes each release under several language-specific versions and tags each of them. Upstream tag `v3.25.5` declares `cpp 4.25.5` and `java 3.25.5`, so the Linux build, which compiles that tag via `install-protobuf.sh`, links **C++ runtime 4.25.5**; the `3.25.5` written there is that release's Java alias. vcpkg names its `protobuf` port after the C++ runtime version and resolves upstream as `REF v${VERSION}`.
So Windows 4.25.1 and Linux 4.25.5 are the same Protobuf 25.x generation, one patch apart, not a major split.
Aligning them exactly is not available either: vcpkg publishes neither `3.25.5` nor `4.25.5` for that port, so the nearest "alignment" would be a genuinely different major.
##########
dev-support/docker/vcpkg/vcpkg.json:
##########
@@ -1,16 +1,239 @@
{
"$schema": "https://raw.githubusercontent.com/microsoft/vcpkg-tool/main/docs/vcpkg.schema.json",
"dependencies": [
- "boost",
+ "boost-asio",
+ "boost-date-time",
+ "boost-system",
+ "boost-program-options",
"protobuf",
"openssl",
"zlib"
],
- "builtin-baseline": "289a69379604112a433874fe8b9812dad3103341",
+ "builtin-baseline": "9e593bb18ea69cc5095e012465dcd675a822ed0d",
"overrides": [
{
"name": "protobuf",
- "version": "3.25.5"
+ "version": "4.25.1"
+ },
+ {
+ "name": "boost-algorithm",
+ "version": "1.86.0"
+ },
Review Comment:
I looked into moving `builtin-baseline` instead, and it does not work here.
The MSYS2 breakage this PR fixes comes from `scripts/cmake/vcpkg_acquire_msys.cmake`, which ships with the checked-out vcpkg **tag**, not with `builtin-baseline`. The older tags pin `msys2-runtime-3.5.4-2`, which is now a 404 on the mirror. So the tag has to move forward regardless, and keeping the new tag (2026.07.29) while pinning Boost explicitly was the combination I could verify end to end.
That said, this comment found a real bug, so thank you. `boost-coroutine` was missing from the override list, so it alone resolved to the baseline and installed at **1.91.0** while the other 55 ports sat at 1.86.0 -- a mixed Boost that happened to build anyway. It is now pinned, and I verified against vcpkg's own package inventory in the rebuilt image that no Boost port is off 1.86.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-20031. Fix the Windows native build: Docker image, libhdfspp CMake and YARN UI [hadoop]
Posted by "GauthamBanasandra (via GitHub)" <gi...@apache.org>.
GauthamBanasandra commented on code in PR #8844:
URL: https://github.com/apache/hadoop/pull/8844#discussion_r4426562799
##########
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/CMakeLists.txt:
##########
@@ -171,6 +171,13 @@ if (NOT PROTOC_IS_COMPATIBLE)
compiler are compatible with each other.")
endif (NOT PROTOC_IS_COMPATIBLE)
+# The checks above leave CMAKE_REQUIRED_INCLUDES and CMAKE_REQUIRED_LIBRARIES
+# pointing at the Protobuf and Abseil targets. Abseil carries C++ feature
+# requirements, which a C-language check_symbol_exists() below cannot satisfy
+# under the Visual Studio generator. Reset them so later checks run clean.
+unset (CMAKE_REQUIRED_INCLUDES)
+unset (CMAKE_REQUIRED_LIBRARIES)
Review Comment:
Agreed, and fixed in 46038bf1 -- the values are now saved before the Protobuf check and restored afterwards, rather than unset.
You were right that unsetting discards state rather than restoring it: an earlier check in the same file sets `CMAKE_REQUIRED_LIBRARIES` to `${CMAKE_THREAD_LIBS_INIT}` for the `thread_local` probe, so the blanket `unset` left later checks with less than they started with.
##########
BUILDING.txt:
##########
@@ -603,12 +603,29 @@ still be immensely useful as a raw guide for all the steps involved in creating
needed to build Hadoop on Windows 10.
Building using the Docker:
+Docker Desktop must first be switched to Windows containers. In the default Linux container mode
+every command below fails immediately with "no matching manifest for linux/amd64".
+> "%ProgramFiles%\Docker\Docker\DockerCli.exe" -SwitchWindowsEngine
+
+Wait for the engine to restart, then confirm - this must print "windows" before proceeding.
+> docker info --format "{{.OSType}}"
+
We first need to build the Docker image for building Hadoop on Windows 10. Run this command from
the root of the Hadoop repository.
-> docker build -t hadoop-windows-10-builder -f .\dev-support\docker\Dockerfile_windows_10 .\dev-support\docker\
+> docker build -t hadoop-windows-10-builder -m 8g -f .\dev-support\docker\Dockerfile_windows_10 .\dev-support\docker\
+
+The -m 8g is needed because the vcpkg step compiles Boost, Protocol Buffers, OpenSSL and Zlib from
+source and the default container memory is too small. Expect the image build to take a few hours
+and the resulting image to be around 23 GB. Note that -m and --isolation are options of the classic
+builder, so leave DOCKER_BUILDKIT unset; BuildKit does not accept them.
Review Comment:
Done in 46038bf1. BUILDING.txt now gives the explicit command for both shells rather than only saying to leave it unset:
> set DOCKER_BUILDKIT= (cmd.exe)
> $Env:DOCKER_BUILDKIT = '' (PowerShell)
and notes that setting it to 0 works as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-20031. Fix the Windows native build: Docker image, libhdfspp CMake and YARN UI [hadoop]
Posted by "GauthamBanasandra (via GitHub)" <gi...@apache.org>.
GauthamBanasandra commented on PR #8844:
URL: https://github.com/apache/hadoop/pull/8844#issuecomment-6346806939
> Looks good, though I would like that sha 512 checksum validation in, assuming have it present in the target VMs.
>
> One point: text is all about Windows 10. Does this work in windows 11?
Thanks for the review @steveloughran. Yes, it works on Windows 11.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org