[GitHub] [iceberg] rdsr commented on a change in pull request #1478: Enable Hive3 builds for iceberg-mr and iceberg-hive-metastore

[GitBox <gi...@apache.org>]

rdsr commented on a change in pull request #1478:
URL: https://github.com/apache/iceberg/pull/1478#discussion_r493661582



##########
File path: settings.gradle
##########
@@ -17,44 +17,64 @@
  * under the License.
  */
 
+gradle.ext.hive3Enabled = false
+if (System.getProperty("hive3") != null) {
+  if (JavaVersion.current() == JavaVersion.VERSION_1_8) {
+    println "*** 'hive3' flag detected - building with Hive3 / Hadoop3 ***"
+    gradle.ext.hive3Enabled = true
+  } else {
+    println "*** 'hive3' flag detected, but with JDK version other than 1.8 - skipping Hive3 / Hadoop3 build. ***"
+  }
+}
+
 rootProject.name = 'iceberg'
 include 'api'
 include 'common'
 include 'core'
 include 'data'
-include 'flink'
-include 'flink-runtime'
 include 'mr'
 include 'hive-runtime'
 include 'orc'
 include 'arrow'
 include 'parquet'
 include 'bundled-guava'
-include 'spark'
-include 'spark3'
-include 'spark3-runtime'
 include 'pig'
 include 'hive-metastore'
+if (gradle.ext.hive3Enabled) {
+  include 'mr-hive3'
+} else {
+  include 'mr-hive2'

Review comment:
       Why have we excluded these modules from the build? Seems like if the hive3 flag is used, none of these modules/artifacts will be published. Also, in the future, would we want to publish these modules compiled with hadoop3 artifacts if yes then it makes sense to build all modules, I think.
   What are your thoughts on the flag name, should it be hadoop3 or hive3. I feel like this flag is more general and affects Parquet, ORC, MR as you mentioned before.

##########
File path: build.gradle
##########
@@ -458,14 +471,29 @@ project(':iceberg-mr') {
     testCompile("org.apache.calcite:calcite-core")
     testCompile("com.esotericsoftware:kryo-shaded:4.0.2")
     testCompile("com.fasterxml.jackson.core:jackson-annotations:2.6.5")
-    testCompile("com.klarna:hiverunner:5.2.1") {
+    testCompile("com.klarna:hiverunner") {
       exclude group: 'javax.jms', module: 'jms'
       exclude group: 'org.apache.hive', module: 'hive-exec'
       exclude group: 'org.codehaus.jettison', module: 'jettison'
       exclude group: 'org.apache.calcite.avatica'
     }
   }
 }
+if (!gradle.ext.hive3Enabled) {
+  project(':iceberg-mr-hive2') {

Review comment:
       nit: we can possibly make this a var which can be set  when we check `hive3Enabled` to either `:iceberg-mr-hive2` or `:iceberg-mr-hive3` . This would the  remove if/else branches 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org