You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hbase.apache.org by md...@apache.org on 2018/04/12 19:54:25 UTC
[1/2] hbase git commit: HBASE-20356 Make skipping protoc possible
Repository: hbase
Updated Branches:
refs/heads/branch-2 0c751dadf -> a50d9f435
refs/heads/master f3ec23093 -> 70d23214f
HBASE-20356 Make skipping protoc possible
Project: http://git-wip-us.apache.org/repos/asf/hbase/repo
Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/70d23214
Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/70d23214
Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/70d23214
Branch: refs/heads/master
Commit: 70d23214fbaacad90eaf0c0764405bb455e2ae85
Parents: f3ec230
Author: Mike Drob <md...@apache.org>
Authored: Wed Apr 11 21:25:04 2018 -0500
Committer: Mike Drob <md...@apache.org>
Committed: Thu Apr 12 13:31:54 2018 -0500
----------------------------------------------------------------------
dev-support/hbase-personality.sh | 2 +-
hbase-protocol-shaded/pom.xml | 5 ++++-
hbase-protocol/pom.xml | 5 ++++-
src/main/asciidoc/_chapters/developer.adoc | 28 ++++++++++++++++++++-----
4 files changed, 32 insertions(+), 8 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hbase/blob/70d23214/dev-support/hbase-personality.sh
----------------------------------------------------------------------
diff --git a/dev-support/hbase-personality.sh b/dev-support/hbase-personality.sh
index 90786f2..3507a1d 100755
--- a/dev-support/hbase-personality.sh
+++ b/dev-support/hbase-personality.sh
@@ -554,7 +554,7 @@ function hbaseprotoc_rebuild
# Need to run 'install' instead of 'compile' because shading plugin
# is hooked-up to 'install'; else hbase-protocol-shaded is left with
# half of its process done.
- modules_workers patch hbaseprotoc install -DskipTests -Pcompile-protobuf -X -DHBasePatchProcess
+ modules_workers patch hbaseprotoc install -DskipTests -X -DHBasePatchProcess
# shellcheck disable=SC2153
until [[ $i -eq "${#MODULE[@]}" ]]; do
http://git-wip-us.apache.org/repos/asf/hbase/blob/70d23214/hbase-protocol-shaded/pom.xml
----------------------------------------------------------------------
diff --git a/hbase-protocol-shaded/pom.xml b/hbase-protocol-shaded/pom.xml
index 021e424..25443e1 100644
--- a/hbase-protocol-shaded/pom.xml
+++ b/hbase-protocol-shaded/pom.xml
@@ -106,7 +106,7 @@
<version>1.5.3</version>
<executions>
<execution>
- <phase>generate-sources</phase>
+ <phase>process-sources</phase>
<goals>
<goal>replace</goal>
</goals>
@@ -117,6 +117,9 @@
<includes>
<include>**/*.java</include>
</includes>
+ <!-- Ignore errors when missing files, because it means this build
+ was run with -Dprotoc.skip and there is no -Dreplacer.skip -->
+ <ignoreErrors>true</ignoreErrors>
<replacements>
<replacement>
<token>([^\.])com.google.protobuf</token>
http://git-wip-us.apache.org/repos/asf/hbase/blob/70d23214/hbase-protocol/pom.xml
----------------------------------------------------------------------
diff --git a/hbase-protocol/pom.xml b/hbase-protocol/pom.xml
index e6d546e..bfe2588 100644
--- a/hbase-protocol/pom.xml
+++ b/hbase-protocol/pom.xml
@@ -81,7 +81,7 @@
<version>1.5.3</version>
<executions>
<execution>
- <phase>generate-sources</phase>
+ <phase>process-sources</phase>
<goals>
<goal>replace</goal>
</goals>
@@ -92,6 +92,9 @@
<includes>
<include>**/*.java</include>
</includes>
+ <!-- Ignore errors when missing files, because it means this build
+ was run with -Dprotoc.skip and there is no -Dreplacer.skip -->
+ <ignoreErrors>true</ignoreErrors>
<replacements>
<replacement>
<token>(public)(\W+static)?(\W+final)?(\W+class)</token>
http://git-wip-us.apache.org/repos/asf/hbase/blob/70d23214/src/main/asciidoc/_chapters/developer.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/developer.adoc b/src/main/asciidoc/_chapters/developer.adoc
index 48dc79e..9d9f564 100644
--- a/src/main/asciidoc/_chapters/developer.adoc
+++ b/src/main/asciidoc/_chapters/developer.adoc
@@ -415,22 +415,40 @@ use so we can freely change versions without upsetting any downstream project us
The protobuf files are located in _hbase-protocol/src/main/protobuf_.
For the change to be effective, you will need to regenerate the classes.
-You can use maven profile `compile-protobuf` to do this.
[source,bourne]
----
-mvn compile -Pcompile-protobuf
+mvn package -pl hbase-protocol -am
----
-You may also want to define `protoc.path` for the protoc binary, using the following command:
+Similarly, protobuf definitions for internal use are located in the _hbase-protocol-shaded_ module.
[source,bourne]
----
+mvn package -pl hbase-protocol-shaded -am
+----
+
+Typically, protobuf code generation is done using the native `protoc` binary. In our build we use a maven plugin for
+convenience; however, the plugin may not be able to retrieve appropriate binaries for all platforms. If you find yourself
+on a platform where protoc fails, you will have to compile protoc from source, and run it independent of our maven build.
+You can disable the inline code generation by specifying `-Dprotoc.skip` in your maven arguments, allowing your build to proceed further.
-mvn compile -Pcompile-protobuf -Dprotoc.path=/opt/local/bin/protoc
+A similar failure relates to the stock CentOS 6 docker image providing a too old version of glibc for the version of protoc that we use.
+In this case, you would have to install glibc 2.14 and protoc 3.5.1 manually, then execute something like:
+
+[source,bourne]
----
+cd hbase-protocol-shaded
+LD_LIBRARY_PATH=/opt/glibc-2.14/lib protoc \
+ --proto_path=src/main/protobuf \
+ --java_out=target/generated-sources/protobuf/java \
+ src/main/protobuf/*.proto
+----
+
+[NOTE]
+If you need to manually generate your protobuf files, you should not use `clean` in subsequent maven calls, as that will delete the newly generated files.
-Read the _hbase-protocol/README.txt_ for more details.
+Read the _hbase-protocol/README.txt_ for more details
[[build.thrift]]
==== Build Thrift
[2/2] hbase git commit: HBASE-20356 Make skipping protoc possible
Posted by md...@apache.org.
HBASE-20356 Make skipping protoc possible
Project: http://git-wip-us.apache.org/repos/asf/hbase/repo
Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/a50d9f43
Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/a50d9f43
Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/a50d9f43
Branch: refs/heads/branch-2
Commit: a50d9f43512a5321a00e141d929072300cb41490
Parents: 0c751da
Author: Mike Drob <md...@apache.org>
Authored: Wed Apr 11 21:25:04 2018 -0500
Committer: Mike Drob <md...@apache.org>
Committed: Thu Apr 12 14:54:03 2018 -0500
----------------------------------------------------------------------
dev-support/hbase-personality.sh | 2 +-
hbase-protocol-shaded/pom.xml | 5 ++++-
src/main/asciidoc/_chapters/developer.adoc | 28 ++++++++++++++++++++-----
3 files changed, 28 insertions(+), 7 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hbase/blob/a50d9f43/dev-support/hbase-personality.sh
----------------------------------------------------------------------
diff --git a/dev-support/hbase-personality.sh b/dev-support/hbase-personality.sh
index 90786f2..3507a1d 100755
--- a/dev-support/hbase-personality.sh
+++ b/dev-support/hbase-personality.sh
@@ -554,7 +554,7 @@ function hbaseprotoc_rebuild
# Need to run 'install' instead of 'compile' because shading plugin
# is hooked-up to 'install'; else hbase-protocol-shaded is left with
# half of its process done.
- modules_workers patch hbaseprotoc install -DskipTests -Pcompile-protobuf -X -DHBasePatchProcess
+ modules_workers patch hbaseprotoc install -DskipTests -X -DHBasePatchProcess
# shellcheck disable=SC2153
until [[ $i -eq "${#MODULE[@]}" ]]; do
http://git-wip-us.apache.org/repos/asf/hbase/blob/a50d9f43/hbase-protocol-shaded/pom.xml
----------------------------------------------------------------------
diff --git a/hbase-protocol-shaded/pom.xml b/hbase-protocol-shaded/pom.xml
index c409012..ee7d54b 100644
--- a/hbase-protocol-shaded/pom.xml
+++ b/hbase-protocol-shaded/pom.xml
@@ -106,7 +106,7 @@
<version>1.5.3</version>
<executions>
<execution>
- <phase>generate-sources</phase>
+ <phase>process-sources</phase>
<goals>
<goal>replace</goal>
</goals>
@@ -117,6 +117,9 @@
<includes>
<include>**/*.java</include>
</includes>
+ <!-- Ignore errors when missing files, because it means this build
+ was run with -Dprotoc.skip and there is no -Dreplacer.skip -->
+ <ignoreErrors>true</ignoreErrors>
<replacements>
<replacement>
<token>([^\.])com.google.protobuf</token>
http://git-wip-us.apache.org/repos/asf/hbase/blob/a50d9f43/src/main/asciidoc/_chapters/developer.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/developer.adoc b/src/main/asciidoc/_chapters/developer.adoc
index 6d959c2..40701e9 100644
--- a/src/main/asciidoc/_chapters/developer.adoc
+++ b/src/main/asciidoc/_chapters/developer.adoc
@@ -415,22 +415,40 @@ use so we can freely change versions without upsetting any downstream project us
The protobuf files are located in _hbase-protocol/src/main/protobuf_.
For the change to be effective, you will need to regenerate the classes.
-You can use maven profile `compile-protobuf` to do this.
[source,bourne]
----
-mvn compile -Pcompile-protobuf
+mvn package -pl hbase-protocol -am
----
-You may also want to define `protoc.path` for the protoc binary, using the following command:
+Similarly, protobuf definitions for internal use are located in the _hbase-protocol-shaded_ module.
[source,bourne]
----
+mvn package -pl hbase-protocol-shaded -am
+----
+
+Typically, protobuf code generation is done using the native `protoc` binary. In our build we use a maven plugin for
+convenience; however, the plugin may not be able to retrieve appropriate binaries for all platforms. If you find yourself
+on a platform where protoc fails, you will have to compile protoc from source, and run it independent of our maven build.
+You can disable the inline code generation by specifying `-Dprotoc.skip` in your maven arguments, allowing your build to proceed further.
-mvn compile -Pcompile-protobuf -Dprotoc.path=/opt/local/bin/protoc
+A similar failure relates to the stock CentOS 6 docker image providing a too old version of glibc for the version of protoc that we use.
+In this case, you would have to install glibc 2.14 and protoc 3.5.1 manually, then execute something like:
+
+[source,bourne]
----
+cd hbase-protocol-shaded
+LD_LIBRARY_PATH=/opt/glibc-2.14/lib protoc \
+ --proto_path=src/main/protobuf \
+ --java_out=target/generated-sources/protobuf/java \
+ src/main/protobuf/*.proto
+----
+
+[NOTE]
+If you need to manually generate your protobuf files, you should not use `clean` in subsequent maven calls, as that will delete the newly generated files.
-Read the _hbase-protocol/README.txt_ for more details.
+Read the _hbase-protocol/README.txt_ for more details
[[build.thrift]]
==== Build Thrift