You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by cd...@apache.org on 2018/03/19 23:09:43 UTC
hadoop git commit: HADOOP-14667. Flexible Visual Studio support.
Contributed by Allen Wittenauer
Repository: hadoop
Updated Branches:
refs/heads/trunk f480367af -> 3fc3fa971
HADOOP-14667. Flexible Visual Studio support. Contributed by Allen Wittenauer
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/3fc3fa97
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/3fc3fa97
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/3fc3fa97
Branch: refs/heads/trunk
Commit: 3fc3fa9711d96677f6149e173df0f57cd06ee6b9
Parents: f480367
Author: Chris Douglas <cd...@apache.org>
Authored: Mon Mar 19 16:05:55 2018 -0700
Committer: Chris Douglas <cd...@apache.org>
Committed: Mon Mar 19 16:05:55 2018 -0700
----------------------------------------------------------------------
BUILDING.txt | 30 ++++--------
dev-support/bin/win-vs-upgrade.cmd | 39 ++++++++++++++++
dev-support/win-paths-eg.cmd | 49 ++++++++++++++++++++
hadoop-common-project/hadoop-common/pom.xml | 28 +++++++++++
.../src/main/native/native.vcxproj | 2 +
.../hadoop-hdfs-native-client/pom.xml | 5 +-
6 files changed, 128 insertions(+), 25 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/3fc3fa97/BUILDING.txt
----------------------------------------------------------------------
diff --git a/BUILDING.txt b/BUILDING.txt
index 6c266e5..6d752d4 100644
--- a/BUILDING.txt
+++ b/BUILDING.txt
@@ -348,7 +348,7 @@ Requirements:
* Maven 3.0 or later
* ProtocolBuffer 2.5.0
* CMake 3.1 or newer
-* Windows SDK 7.1 or Visual Studio 2010 Professional
+* Visual Studio 2010 Professional or Higher
* Windows SDK 8.1 (if building CPU rate control for the container executor)
* zlib headers (if building native code bindings for zlib)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
@@ -359,18 +359,15 @@ Requirements:
Unix command-line tools are also included with the Windows Git package which
can be downloaded from http://git-scm.com/downloads
-If using Visual Studio, it must be Visual Studio 2010 Professional (not 2012).
+If using Visual Studio, it must be Professional level or higher.
Do not use Visual Studio Express. It does not support compiling for 64-bit,
-which is problematic if running a 64-bit system. The Windows SDK 7.1 is free to
-download here:
-
-http://www.microsoft.com/en-us/download/details.aspx?id=8279
+which is problematic if running a 64-bit system.
The Windows SDK 8.1 is available to download at:
http://msdn.microsoft.com/en-us/windows/bg162891.aspx
-Cygwin is neither required nor supported.
+Cygwin is not required.
----------------------------------------------------------------------------------
Building:
@@ -378,21 +375,12 @@ Building:
Keep the source code tree in a short path to avoid running into problems related
to Windows maximum path length limitation (for example, C:\hdc).
-Run builds from a Windows SDK Command Prompt. (Start, All Programs,
-Microsoft Windows SDK v7.1, Windows SDK 7.1 Command Prompt).
-
-JAVA_HOME must be set, and the path must not contain spaces. If the full path
-would contain spaces, then use the Windows short path instead.
-
-You must set the Platform environment variable to either x64 or Win32 depending
-on whether you're running a 64-bit or 32-bit system. Note that this is
-case-sensitive. It must be "Platform", not "PLATFORM" or "platform".
-Environment variables on Windows are usually case-insensitive, but Maven treats
-them as case-sensitive. Failure to set this environment variable correctly will
-cause msbuild to fail while building the native code in hadoop-common.
+There is one support command file located in dev-support called win-paths-eg.cmd.
+It should be copied somewhere convenient and modified to fit your needs.
-set Platform=x64 (when building on a 64-bit system)
-set Platform=Win32 (when building on a 32-bit system)
+win-paths-eg.cmd sets up the environment for use. You will need to modify this
+file. It will put all of the required components in the command path,
+configure the bit-ness of the build, and set several optional components.
Several tests require that the user must have the Create Symbolic Links
privilege.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/3fc3fa97/dev-support/bin/win-vs-upgrade.cmd
----------------------------------------------------------------------
diff --git a/dev-support/bin/win-vs-upgrade.cmd b/dev-support/bin/win-vs-upgrade.cmd
new file mode 100644
index 0000000..d8c9d73
--- /dev/null
+++ b/dev-support/bin/win-vs-upgrade.cmd
@@ -0,0 +1,39 @@
+@ECHO OFF
+@REM Licensed to the Apache Software Foundation (ASF) under one or more
+@REM contributor license agreements. See the NOTICE file distributed with
+@REM this work for additional information regarding copyright ownership.
+@REM The ASF licenses this file to You under the Apache License, Version 2.0
+@REM (the "License"); you may not use this file except in compliance with
+@REM the License. You may obtain a copy of the License at
+@REM
+@REM http://www.apache.org/licenses/LICENSE-2.0
+@REM
+@REM Unless required by applicable law or agreed to in writing, software
+@REM distributed under the License is distributed on an "AS IS" BASIS,
+@REM WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+@REM See the License for the specific language governing permissions and
+@REM limitations under the License.
+
+@WHERE devenv
+IF %ERRORLEVEL% NEQ 0 (
+ @ECHO "devenv command was not found. Verify your compiler installation level."
+ EXIT /b 1
+)
+
+@REM Need to save output to a file because for loop will just
+@REM loop forever... :(
+
+SET srcdir=%1
+SET workdir=%2
+
+IF EXIST %srcdir%\Backup (
+ @ECHO "Solution files already upgraded."
+ EXIT /b 0
+)
+
+CD %srcdir%
+DIR /B *.sln > %workdir%\HADOOP-SLN-UPGRADE.TXT
+
+FOR /F %%f IN (%workdir%\HADOOP-SLN-UPGRADE.TXT) DO (
+ devenv %%f /upgrade
+)
http://git-wip-us.apache.org/repos/asf/hadoop/blob/3fc3fa97/dev-support/win-paths-eg.cmd
----------------------------------------------------------------------
diff --git a/dev-support/win-paths-eg.cmd b/dev-support/win-paths-eg.cmd
new file mode 100644
index 0000000..f9acae0
--- /dev/null
+++ b/dev-support/win-paths-eg.cmd
@@ -0,0 +1,49 @@
+@ECHO OFF
+@REM Licensed to the Apache Software Foundation (ASF) under one or more
+@REM contributor license agreements. See the NOTICE file distributed with
+@REM this work for additional information regarding copyright ownership.
+@REM The ASF licenses this file to You under the Apache License, Version 2.0
+@REM (the "License"); you may not use this file except in compliance with
+@REM the License. You may obtain a copy of the License at
+@REM
+@REM http://www.apache.org/licenses/LICENSE-2.0
+@REM
+@REM Unless required by applicable law or agreed to in writing, software
+@REM distributed under the License is distributed on an "AS IS" BASIS,
+@REM WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+@REM See the License for the specific language governing permissions and
+@REM limitations under the License.
+
+@REM *************************************************
+@REM JDK and these settings MUST MATCH
+@REM
+@REM 64-bit : Platform = x64, VCVARSPLAT = amd64
+@REM
+@REM 32-bit : Platform = Win32, VCVARSPLAT = x86
+@REM
+
+SET Platform=x64
+SET VCVARSPLAT=amd64
+
+@REM ******************
+@REM Forcibly move the Maven local repo
+
+SET MAVEN_OPTS=-Dmaven.repo.local=C:\Tools\m2
+
+@REM *******************************************
+@REM
+@REM Locations of your bits and pieces
+@REM
+@REM NOTE: cmake is assumed to already be on the
+@REM command path
+@REM
+
+SET MAVEN_HOME=C:\Tools\apache-maven-3.5.0
+SET JAVA_HOME=C:\Tools\jdk
+SET MSVS=C:\Program Files (x86)\Microsoft Visual Studio 12.0
+SET PROTO_BIN=C:\Tools\protobuf-2.5.0
+SET GIT_HOME=C:\Program Files\Git
+
+SET PATH=%JAVA_HOME%\bin;%MAVEN_HOME%\bin;%PROTO_BIN%;%GIT_HOME%\bin;%PATH%
+
+CALL "%MSVS%\VC\vcvarsall.bat" %VCVARSPLAT%
http://git-wip-us.apache.org/repos/asf/hadoop/blob/3fc3fa97/hadoop-common-project/hadoop-common/pom.xml
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/pom.xml b/hadoop-common-project/hadoop-common/pom.xml
index 49d3575..1a16dc4 100644
--- a/hadoop-common-project/hadoop-common/pom.xml
+++ b/hadoop-common-project/hadoop-common/pom.xml
@@ -839,6 +839,20 @@
<artifactId>exec-maven-plugin</artifactId>
<executions>
<execution>
+ <id>convert-ms-winutils</id>
+ <phase>generate-sources</phase>
+ <goals>
+ <goal>exec</goal>
+ </goals>
+ <configuration>
+ <executable>${basedir}\..\..\dev-support\bin\win-vs-upgrade.cmd</executable>
+ <arguments>
+ <argument>${basedir}\src\main\winutils</argument>
+ <argument>${project.build.directory}</argument>
+ </arguments>
+ </configuration>
+ </execution>
+ <execution>
<id>compile-ms-winutils</id>
<phase>compile</phase>
<goals>
@@ -858,6 +872,20 @@
</configuration>
</execution>
<execution>
+ <id>convert-ms-native-dll</id>
+ <phase>generate-sources</phase>
+ <goals>
+ <goal>exec</goal>
+ </goals>
+ <configuration>
+ <executable>${basedir}\..\..\dev-support\bin\win-vs-upgrade.cmd</executable>
+ <arguments>
+ <argument>${basedir}\src\main\native</argument>
+ <argument>${project.build.directory}</argument>
+ </arguments>
+ </configuration>
+ </execution>
+ <execution>
<id>compile-ms-native-dll</id>
<phase>compile</phase>
<goals>
http://git-wip-us.apache.org/repos/asf/hadoop/blob/3fc3fa97/hadoop-common-project/hadoop-common/src/main/native/native.vcxproj
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/native/native.vcxproj b/hadoop-common-project/hadoop-common/src/main/native/native.vcxproj
index 1119069..ac3767b 100644
--- a/hadoop-common-project/hadoop-common/src/main/native/native.vcxproj
+++ b/hadoop-common-project/hadoop-common/src/main/native/native.vcxproj
@@ -71,6 +71,7 @@
<PropertyGroup>
<SnappyLib Condition="Exists('$(CustomSnappyPrefix)\snappy.dll')">$(CustomSnappyPrefix)</SnappyLib>
<SnappyLib Condition="Exists('$(CustomSnappyPrefix)\lib\snappy.dll') And '$(SnappyLib)' == ''">$(CustomSnappyPrefix)\lib</SnappyLib>
+ <SnappyLib Condition="Exists('$(CustomSnappyPrefix)\bin\snappy.dll') And '$(SnappyLib)' == ''">$(CustomSnappyPrefix)\bin</SnappyLib>
<SnappyLib Condition="Exists('$(CustomSnappyLib)') And '$(SnappyLib)' == ''">$(CustomSnappyLib)</SnappyLib>
<SnappyInclude Condition="Exists('$(CustomSnappyPrefix)\snappy.h')">$(CustomSnappyPrefix)</SnappyInclude>
<SnappyInclude Condition="Exists('$(CustomSnappyPrefix)\include\snappy.h') And '$(SnappyInclude)' == ''">$(CustomSnappyPrefix)\include</SnappyInclude>
@@ -82,6 +83,7 @@
<PropertyGroup>
<IsalLib Condition="Exists('$(CustomIsalPrefix)\isa-l.dll')">$(CustomIsalPrefix)</IsalLib>
<IsalLib Condition="Exists('$(CustomIsalPrefix)\lib\isa-l.dll') And '$(IsalLib)' == ''">$(CustomIsalPrefix)\lib</IsalLib>
+ <IsalLib Condition="Exists('$(CustomIsalPrefix)\bin\isa-l.dll') And '$(IsalLib)' == ''">$(CustomIsalPrefix)\bin</IsalLib>
<IsalLib Condition="Exists('$(CustomIsalLib)') And '$(IsalLib)' == ''">$(CustomIsalLib)</IsalLib>
<IsalEnabled Condition="'$(IsalLib)' != ''">true</IsalEnabled>
</PropertyGroup>
http://git-wip-us.apache.org/repos/asf/hadoop/blob/3fc3fa97/hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml b/hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml
index fa0e174..d9d98e4 100644
--- a/hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml
+++ b/hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml
@@ -138,13 +138,10 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd">
</goals>
<configuration>
<target>
- <condition property="generator" value="Visual Studio 10" else="Visual Studio 10 Win64">
- <equals arg1="Win32" arg2="${env.PLATFORM}" />
- </condition>
<mkdir dir="${project.build.directory}/native"/>
<exec executable="cmake" dir="${project.build.directory}/native"
failonerror="true">
- <arg line="${basedir}/src/ -DGENERATED_JAVAH=${project.build.directory}/native/javah -DJVM_ARCH_DATA_MODEL=${sun.arch.data.model} -DREQUIRE_FUSE=${require.fuse} -G '${generator}'"/>
+ <arg line="${basedir}/src/ -DGENERATED_JAVAH=${project.build.directory}/native/javah -DJVM_ARCH_DATA_MODEL=${sun.arch.data.model} -DREQUIRE_FUSE=${require.fuse} -A '${env.PLATFORM}'"/>
</exec>
<exec executable="msbuild" dir="${project.build.directory}/native"
failonerror="true">
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org