You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/11/30 13:27:55 UTC

[GitHub] [iceberg] pvary opened a new pull request #3627: Spark: Remove iceberg-hive-metastore compile time dependency

pvary opened a new pull request #3627:
URL: https://github.com/apache/iceberg/pull/3627


   icebeg-hive-metastore dependency is not needed at compile time (and should not be needed). Spark and spark extension modules should not depend on HiveCatalog being on the classpath.
   
   That said the dependency needed for running the tests, and also needed to be put into the fat runtime jar.
   
   The PR modifies these dependencies accordingly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] pvary merged pull request #3627: Spark: Remove iceberg-hive-metastore compile time dependency

Posted by GitBox <gi...@apache.org>.
pvary merged pull request #3627:
URL: https://github.com/apache/iceberg/pull/3627


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] kbendick commented on a change in pull request #3627: Spark: Remove iceberg-hive-metastore compile time dependency

Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3627:
URL: https://github.com/apache/iceberg/pull/3627#discussion_r759714220



##########
File path: spark/v3.1/build.gradle
##########
@@ -130,12 +130,12 @@ project(":iceberg-spark:iceberg-spark-3.1-extensions") {
     compileOnly project(':iceberg-orc')
     compileOnly project(':iceberg-common')
     compileOnly project(':iceberg-spark:iceberg-spark-3.1')
-    compileOnly project(':iceberg-hive-metastore')
     compileOnly("org.apache.spark:spark-hive_2.12:${sparkVersion}") {
       exclude group: 'org.apache.avro', module: 'avro'
       exclude group: 'org.apache.arrow'
     }
 
+    testImplementation project(path: ':iceberg-hive-metastore')
     testImplementation project(path: ':iceberg-hive-metastore', configuration: 'testArtifacts')

Review comment:
       Question: It looks like this line is duplicated, but the second one has `configuration: 'testArtifacts'`.
   
   Can we use just one or the other? This occurs in a few places.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] pvary commented on pull request #3627: Spark: Remove iceberg-hive-metastore compile time dependency

Posted by GitBox <gi...@apache.org>.
pvary commented on pull request #3627:
URL: https://github.com/apache/iceberg/pull/3627#issuecomment-988006120


   Thanks @rdblue for the review!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] pvary commented on a change in pull request #3627: Spark: Remove iceberg-hive-metastore compile time dependency

Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #3627:
URL: https://github.com/apache/iceberg/pull/3627#discussion_r759974835



##########
File path: spark/v3.1/build.gradle
##########
@@ -130,12 +130,12 @@ project(":iceberg-spark:iceberg-spark-3.1-extensions") {
     compileOnly project(':iceberg-orc')
     compileOnly project(':iceberg-common')
     compileOnly project(':iceberg-spark:iceberg-spark-3.1')
-    compileOnly project(':iceberg-hive-metastore')
     compileOnly("org.apache.spark:spark-hive_2.12:${sparkVersion}") {
       exclude group: 'org.apache.avro', module: 'avro'
       exclude group: 'org.apache.arrow'
     }
 
+    testImplementation project(path: ':iceberg-hive-metastore')
     testImplementation project(path: ':iceberg-hive-metastore', configuration: 'testArtifacts')

Review comment:
       We need to depend on the src and on the test code as well. The src contains the `HiveCatalog` while the test code contains the `TestHiveMetastore` classes.
   I am not a gradle wizard so I tried to google for a nicer way, but can not find any. I am open to other suggestions :)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] rdblue commented on pull request #3627: Spark: Remove iceberg-hive-metastore compile time dependency

Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #3627:
URL: https://github.com/apache/iceberg/pull/3627#issuecomment-987308704


   Thanks, @pvary! Looks good to me, but it has conflicts.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org