You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/10/22 16:57:51 UTC

[GitHub] [hadoop] GauthamBanasandra opened a new pull request, #5066: HADOOP-18506. Change build instructions for Windows using VS2019

GauthamBanasandra opened a new pull request, #5066:
URL: https://github.com/apache/hadoop/pull/5066

   * Changed the instructions to build Hadoop on Windows 10.
   
   <!--
     Thanks for sending a pull request!
       1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
       2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
   -->
   
   ### Description of PR
   With HADOOP-18133, we're finally able to build Hadoop on Windows using Visual Studio 2019. This PR updates the documentation with the latest instructions.
   
   ### How was this patch tested?
   N/A
   
   ### For code changes:
   
   - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
   - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
   - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

[GitHub] [hadoop] goiri merged pull request #5066: HADOOP-18506. Update build instructions for Windows using VS2019

Posted by GitBox <gi...@apache.org>.
goiri merged PR #5066:
URL: https://github.com/apache/hadoop/pull/5066


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

[GitHub] [hadoop] hadoop-yetus commented on pull request #5066: HADOOP-18506. Update build instructions for Windows using VS2019

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5066:
URL: https://github.com/apache/hadoop/pull/5066#issuecomment-1288054951

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   1m  4s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   |||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  shadedclient  |  34m 13s |  |  branch has no errors when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | +1 :green_heart: |  shadedclient  |  22m 28s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 42s |  |  The patch does not generate ASF License warnings.  |
   |  |   |  60m  5s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5066/3/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/5066 |
   | Optional Tests | dupname asflicense codespell detsecrets |
   | uname | Linux 57ab863dff40 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / a02c880898e1894152a70663fe80883daceee43d |
   | Max. process+thread count | 530 (vs. ulimit of 5500) |
   | modules | C: . U: . |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5066/3/console |
   | versions | git=2.25.1 maven=3.6.3 |
   | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

[GitHub] [hadoop] GauthamBanasandra commented on a diff in pull request #5066: HADOOP-18506. Update build instructions for Windows using VS2019

Posted by GitBox <gi...@apache.org>.
GauthamBanasandra commented on code in PR #5066:
URL: https://github.com/apache/hadoop/pull/5066#discussion_r1002660030


##########
BUILDING.txt:
##########
@@ -540,6 +564,11 @@ configure the bit-ness of the build, and set several optional components.
 Several tests require that the user must have the Create Symbolic Links
 privilege.
 
+To simplify the installation of Boost, Protocol buffers, OpenSSL and Zlib dependencies we can use
+vcpkg (https://github.com/Microsoft/vcpkg.git). Upon cloning the vcpkg repo, checkout the commit
+7ffa425e1db8b0c3edf9c50f2f3a0f25a324541d to get the required versions of the dependencies

Review Comment:
   > Why not a specific branch or latest?
   
   I couldn't find a branch/release in vcpkg where all these dependencies are at the required versions. Thus, we're using this specific commit.
   
   > It would also be good to add the command.
   
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

[GitHub] [hadoop] GauthamBanasandra commented on a diff in pull request #5066: HADOOP-18506. Update build instructions for Windows using VS2019

Posted by GitBox <gi...@apache.org>.
GauthamBanasandra commented on code in PR #5066:
URL: https://github.com/apache/hadoop/pull/5066#discussion_r1002660030


##########
BUILDING.txt:
##########
@@ -540,6 +564,11 @@ configure the bit-ness of the build, and set several optional components.
 Several tests require that the user must have the Create Symbolic Links
 privilege.
 
+To simplify the installation of Boost, Protocol buffers, OpenSSL and Zlib dependencies we can use
+vcpkg (https://github.com/Microsoft/vcpkg.git). Upon cloning the vcpkg repo, checkout the commit
+7ffa425e1db8b0c3edf9c50f2f3a0f25a324541d to get the required versions of the dependencies

Review Comment:
   > Why not a specific branch or latest?
   I couldn't find a branch/release in vcpkg where all these dependencies are at the required versions.
   
   > It would also be good to add the command.
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

[GitHub] [hadoop] hadoop-yetus commented on pull request #5066: HADOOP-18506. Update build instructions for Windows using VS2019

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5066:
URL: https://github.com/apache/hadoop/pull/5066#issuecomment-1287878069

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   1m  7s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   |||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  shadedclient  |  33m 18s |  |  branch has no errors when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | +1 :green_heart: |  shadedclient  |  22m 53s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 43s |  |  The patch does not generate ASF License warnings.  |
   |  |   |  59m 39s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5066/2/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/5066 |
   | Optional Tests | dupname asflicense codespell detsecrets |
   | uname | Linux 8ce34e055cbe 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / ac0ab9058df3c750597c2bcf43f7fa329341d35c |
   | Max. process+thread count | 601 (vs. ulimit of 5500) |
   | modules | C: . U: . |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5066/2/console |
   | versions | git=2.25.1 maven=3.6.3 |
   | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

[GitHub] [hadoop] hadoop-yetus commented on pull request #5066: HADOOP-18506. Update build instructions for Windows using VS2019

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5066:
URL: https://github.com/apache/hadoop/pull/5066#issuecomment-1287870377

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   1m  4s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   |||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  shadedclient  |  34m  7s |  |  branch has no errors when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | +1 :green_heart: |  shadedclient  |  22m 42s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 43s |  |  The patch does not generate ASF License warnings.  |
   |  |   |  60m 23s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5066/1/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/5066 |
   | Optional Tests | dupname asflicense codespell detsecrets |
   | uname | Linux 8f2ce564c477 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 40d79e36e2ffd36ecc563e624483978e599634e4 |
   | Max. process+thread count | 532 (vs. ulimit of 5500) |
   | modules | C: . U: . |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5066/1/console |
   | versions | git=2.25.1 maven=3.6.3 |
   | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

[GitHub] [hadoop] GauthamBanasandra commented on a diff in pull request #5066: HADOOP-18506. Update build instructions for Windows using VS2019

Posted by GitBox <gi...@apache.org>.
GauthamBanasandra commented on code in PR #5066:
URL: https://github.com/apache/hadoop/pull/5066#discussion_r1005994237


##########
BUILDING.txt:
##########
@@ -557,6 +586,22 @@ the zlib 1.2.7 source tree.
 
 http://www.zlib.net/
 
+
+Build command:
+The following command builds all the modules in the Hadoop project and generates the tar.gz file in
+hadoop-dist/target upon successful build. Run these commands from an
+"x64 Native Tools Command Prompt for VS 2019" which can be found under "Visual Studio 2019" in the
+Windows start menu.
+
+> set classpath=

Review Comment:
   I've created this JIRA - https://issues.apache.org/jira/browse/HADOOP-18509.



##########
BUILDING.txt:
##########
@@ -557,6 +586,22 @@ the zlib 1.2.7 source tree.
 
 http://www.zlib.net/
 
+
+Build command:
+The following command builds all the modules in the Hadoop project and generates the tar.gz file in
+hadoop-dist/target upon successful build. Run these commands from an
+"x64 Native Tools Command Prompt for VS 2019" which can be found under "Visual Studio 2019" in the
+Windows start menu.
+
+> set classpath=

Review Comment:
   I've created this JIRA - https://issues.apache.org/jira/browse/HADOOP-18509 to track this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

[GitHub] [hadoop] GauthamBanasandra commented on a diff in pull request #5066: HADOOP-18506. Update build instructions for Windows using VS2019

Posted by GitBox <gi...@apache.org>.
GauthamBanasandra commented on code in PR #5066:
URL: https://github.com/apache/hadoop/pull/5066#discussion_r1002660354


##########
BUILDING.txt:
##########
@@ -557,6 +586,22 @@ the zlib 1.2.7 source tree.
 
 http://www.zlib.net/
 
+
+Build command:
+The following command builds all the modules in the Hadoop project and generates the tar.gz file in
+hadoop-dist/target upon successful build. Run these commands from an
+"x64 Native Tools Command Prompt for VS 2019" which can be found under "Visual Studio 2019" in the
+Windows start menu.
+
+> set classpath=

Review Comment:
   Right, but this file (`BUILDING.txt`) isn't rendered as markdown. It should've been `BUILDING.md`.
   
   I'm using the `>` at the beginning to signify the start of the command in a Windows terminal. Like -
   
   ![image](https://user-images.githubusercontent.com/10280768/197380892-35ee5d16-ef1d-4336-8146-fcb75f977f45.png)
   



##########
BUILDING.txt:
##########
@@ -492,39 +492,63 @@ Building on CentOS 8
 
 ----------------------------------------------------------------------------------
 
-Building on Windows
+Building on Windows 10
 
 ----------------------------------------------------------------------------------
 Requirements:
 
-* Windows System
+* Windows 10
 * JDK 1.8
 * Maven 3.0 or later
 * Boost 1.72
 * Protocol Buffers 3.7.1
 * CMake 3.19 or newer
-* Visual Studio 2010 Professional or Higher
-* Windows SDK 8.1 (if building CPU rate control for the container executor)
-* zlib headers (if building native code bindings for zlib)
+* Visual Studio 2019
+* Windows SDK 8.1 (optional, if building CPU rate control for the container executor)
+* Zlib (if building native code bindings for zlib)

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

[GitHub] [hadoop] goiri commented on a diff in pull request #5066: HADOOP-18506. Update build instructions for Windows using VS2019

Posted by GitBox <gi...@apache.org>.
goiri commented on code in PR #5066:
URL: https://github.com/apache/hadoop/pull/5066#discussion_r1002544982


##########
BUILDING.txt:
##########
@@ -540,6 +564,11 @@ configure the bit-ness of the build, and set several optional components.
 Several tests require that the user must have the Create Symbolic Links
 privilege.
 
+To simplify the installation of Boost, Protocol buffers, OpenSSL and Zlib dependencies we can use
+vcpkg (https://github.com/Microsoft/vcpkg.git). Upon cloning the vcpkg repo, checkout the commit
+7ffa425e1db8b0c3edf9c50f2f3a0f25a324541d to get the required versions of the dependencies

Review Comment:
   Why not a specific branch or latest?
   It would also be good to add the command.



##########
BUILDING.txt:
##########
@@ -492,39 +492,63 @@ Building on CentOS 8
 
 ----------------------------------------------------------------------------------
 
-Building on Windows
+Building on Windows 10
 
 ----------------------------------------------------------------------------------
 Requirements:
 
-* Windows System
+* Windows 10
 * JDK 1.8
 * Maven 3.0 or later
 * Boost 1.72
 * Protocol Buffers 3.7.1
 * CMake 3.19 or newer
-* Visual Studio 2010 Professional or Higher
-* Windows SDK 8.1 (if building CPU rate control for the container executor)
-* zlib headers (if building native code bindings for zlib)
+* Visual Studio 2019
+* Windows SDK 8.1 (optional, if building CPU rate control for the container executor)
+* Zlib (if building native code bindings for zlib)

Review Comment:
   Can we add links to these components?



##########
BUILDING.txt:
##########
@@ -557,6 +586,22 @@ the zlib 1.2.7 source tree.
 
 http://www.zlib.net/
 
+
+Build command:
+The following command builds all the modules in the Hadoop project and generates the tar.gz file in
+hadoop-dist/target upon successful build. Run these commands from an
+"x64 Native Tools Command Prompt for VS 2019" which can be found under "Visual Studio 2019" in the
+Windows start menu.
+
+> set classpath=

Review Comment:
   In markdown I think you can use the triple quote to make it look like a command.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

[GitHub] [hadoop] goiri commented on a diff in pull request #5066: HADOOP-18506. Update build instructions for Windows using VS2019

Posted by GitBox <gi...@apache.org>.
goiri commented on code in PR #5066:
URL: https://github.com/apache/hadoop/pull/5066#discussion_r1002759332


##########
BUILDING.txt:
##########
@@ -557,6 +586,22 @@ the zlib 1.2.7 source tree.
 
 http://www.zlib.net/
 
+
+Build command:
+The following command builds all the modules in the Hadoop project and generates the tar.gz file in
+hadoop-dist/target upon successful build. Run these commands from an
+"x64 Native Tools Command Prompt for VS 2019" which can be found under "Visual Studio 2019" in the
+Windows start menu.
+
+> set classpath=

Review Comment:
   We have this inconsistent plain text, txt, and md files all over the project.
   Let's merge this one but we probably want to open a JIRA just to switch everything to markdown.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org