You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/07/13 20:15:48 UTC
[GitHub] [iceberg] RussellSpitzer opened a new pull request #2820: Spark: Moves Distribution and Ordering Utils to Spark3 Module
RussellSpitzer opened a new pull request #2820:
URL: https://github.com/apache/iceberg/pull/2820
To enable core Spark3 code to use distribution and ordering information when
performing writes or other operations. One example is to allow Sort Strategy
to transform SortOrder information so it can be used when rewriting data files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #2820: Spark: Moves Distribution and Ordering Utils to Spark3 Module
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #2820:
URL: https://github.com/apache/iceberg/pull/2820#issuecomment-915638970
Looks good to me. The other build settings weren't needed after all?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #2820: Spark: Moves Distribution and Ordering Utils to Spark3 Module
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #2820:
URL: https://github.com/apache/iceberg/pull/2820#issuecomment-904217481
I tried to convert some of these classes to Java today and didn't get anywhere. Some of the traits we need to implement just don't exist for some reason and I think it's probably better just to do it this way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on pull request #2820: Spark: Moves Distribution and Ordering Utils to Spark3 Module
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on pull request #2820:
URL: https://github.com/apache/iceberg/pull/2820#issuecomment-915566593
@rdblue Are we good to go with this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on pull request #2820: Spark: Moves Distribution and Ordering Utils to Spark3 Module
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on pull request #2820:
URL: https://github.com/apache/iceberg/pull/2820#issuecomment-916123699
> Looks good to me. The other build settings weren't needed after all?
Yep I'm not sure what I was thinking :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #2820: Spark: Moves Distribution and Ordering Utils to Spark3 Module
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #2820:
URL: https://github.com/apache/iceberg/pull/2820#discussion_r705361752
##########
File path: build.gradle
##########
@@ -986,7 +986,14 @@ if (jdkVersion == '8') {
}
project(':iceberg-spark3') {
+ apply plugin: 'scala'
+
sourceSets {
+ main {
+ // Use Scala Compiler for everything so we can do cross compilation within the module
+ scala.srcDirs = ['src/main/scala', 'src/main/java']
+ java.srcDirs = []
+ }
// Compile test source against Spark 3.1 and main classes compiled against Spark 3.0
Review comment:
Nope, we still use the "main" source set
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on pull request #2820: Spark: Moves Distribution and Ordering Utils to Spark3 Module
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on pull request #2820:
URL: https://github.com/apache/iceberg/pull/2820#issuecomment-916126980
Thanks for the review @wypoon + @rdblue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wypoon commented on a change in pull request #2820: Spark: Moves Distribution and Ordering Utils to Spark3 Module
Posted by GitBox <gi...@apache.org>.
wypoon commented on a change in pull request #2820:
URL: https://github.com/apache/iceberg/pull/2820#discussion_r694351251
##########
File path: build.gradle
##########
@@ -986,7 +986,14 @@ if (jdkVersion == '8') {
}
project(':iceberg-spark3') {
+ apply plugin: 'scala'
+
sourceSets {
+ main {
+ // Use Scala Compiler for everything so we can do cross compilation within the module
+ scala.srcDirs = ['src/main/scala', 'src/main/java']
+ java.srcDirs = []
+ }
// Compile test source against Spark 3.1 and main classes compiled against Spark 3.0
Review comment:
Do we need to do anything with the spark31 sourceSet?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer merged pull request #2820: Spark: Moves Distribution and Ordering Utils to Spark3 Module
Posted by GitBox <gi...@apache.org>.
RussellSpitzer merged pull request #2820:
URL: https://github.com/apache/iceberg/pull/2820
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on pull request #2820: Spark: Moves Distribution and Ordering Utils to Spark3 Module
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on pull request #2820:
URL: https://github.com/apache/iceberg/pull/2820#issuecomment-879376889
@aokolnychyi + @rdblue I would like to have access to these Utilities for doing some work with the Sort Rewrite Action. If anyone has another Idea on how to get access to this functionality I would be glad to hear it. Ideally I would like to keep Scala and this code out of the Spark3 module, but I can't have an Action use code from within the Extensions module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2820: Spark: Moves Distribution and Ordering Utils to Spark3 Module
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2820:
URL: https://github.com/apache/iceberg/pull/2820#discussion_r694388118
##########
File path: build.gradle
##########
@@ -986,7 +986,14 @@ if (jdkVersion == '8') {
}
project(':iceberg-spark3') {
+ apply plugin: 'scala'
+
sourceSets {
+ main {
+ // Use Scala Compiler for everything so we can do cross compilation within the module
Review comment:
I don't think I understand why we can't use the default scala plugin settings. This is disabling Java compilation by removing srcDirs. Should we just remove the Java plugin?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org