You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/01/30 15:38:39 UTC
[GitHub] [incubator-hudi] lamber-ken opened a new pull request #1293:
[HUDI-585] Optimize the steps of building with scala-2.12
lamber-ken opened a new pull request #1293: [HUDI-585] Optimize the steps of building with scala-2.12
URL: https://github.com/apache/incubator-hudi/pull/1293
## What is the purpose of the pull request
At the master branch, to build scala 2.12 version, will execute dev/change-scala-version.sh firstly.
This pr aims to optimize the steps.
## Brief change log
- *Remove dev/change-scala-version.sh script*
## Verify this pull request
**Test with `scala-2.11`**
```
mvn clean package -DskipTests -DskipITs
mvn clean package -DskipTests -DskipITs -Pscala-2.11
```
```
rm -rf /tmp/hudi_mor_table
export SPARK_HOME=/work/BigData/install/spark/spark-2.4.4-bin-hadoop2.7-2.11
${SPARK_HOME}/bin/spark-shell \
--packages org.apache.spark:spark-avro_2.11:2.4.4 \
--jars `ls packaging/hudi-spark-bundle/target/hudi-spark-bundle_*.*-*.*.*-SNAPSHOT.jar` \
--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer'
val basePath = "file:///tmp/hudi_mor_table"
var datas = List("""{ "name": "kenken", "ts": 1574297893836, "age": 12, "location": "latitude"}""")
val df = spark.read.json(spark.sparkContext.parallelize(datas, 2))
df.write.format("org.apache.hudi").
option("hoodie.insert.shuffle.parallelism", "10").
option("hoodie.upsert.shuffle.parallelism", "10").
option("hoodie.delete.shuffle.parallelism", "10").
option("hoodie.bulkinsert.shuffle.parallelism", "10").
option("hoodie.datasource.write.recordkey.field", "name").
option("hoodie.datasource.write.partitionpath.field", "location").
option("hoodie.datasource.write.precombine.field", "ts").
option("hoodie.table.name", "tableName").
mode("Overwrite").
save(basePath)
spark.read.format("org.apache.hudi").load(basePath + "/*/").show()
```
<br>
**Test with `scala-2.12`**
```
mvn clean package -DskipTests -DskipITs -Pscala-2.12
```
```
rm -rf /tmp/hudi_mor_table
export SPARK_HOME=/work/BigData/install/spark/spark-3.0.0-preview2-bin-hadoop2.7
${SPARK_HOME}/bin/spark-shell \
--packages org.apache.spark:spark-avro_2.12:3.0.0-preview2 \
--jars `ls packaging/hudi-spark-bundle/target/hudi-spark-bundle_*.*-*.*.*-SNAPSHOT.jar` \
--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer'
val basePath = "file:///tmp/hudi_mor_table"
var datas = List("""{ "name": "kenken", "ts": 1574297893836, "age": 12, "location": "latitude"}""")
val df = spark.read.json(spark.sparkContext.parallelize(datas, 2))
df.write.format("org.apache.hudi").
option("hoodie.insert.shuffle.parallelism", "10").
option("hoodie.upsert.shuffle.parallelism", "10").
option("hoodie.delete.shuffle.parallelism", "10").
option("hoodie.bulkinsert.shuffle.parallelism", "10").
option("hoodie.datasource.write.recordkey.field", "name").
option("hoodie.datasource.write.partitionpath.field", "location").
option("hoodie.datasource.write.precombine.field", "ts").
option("hoodie.table.name", "tableName").
mode("Overwrite").
save(basePath)
spark.read.format("org.apache.hudi").load(basePath + "/*/").show()
```
## Committer checklist
- [X] Has a corresponding JIRA in PR title & commit
- [X] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] lamber-ken commented on a change in pull request
#1293: [HUDI-585] Optimize the steps of building with scala-2.12
Posted by GitBox <gi...@apache.org>.
lamber-ken commented on a change in pull request #1293: [HUDI-585] Optimize the steps of building with scala-2.12
URL: https://github.com/apache/incubator-hudi/pull/1293#discussion_r375271334
##########
File path: pom.xml
##########
@@ -1052,6 +1052,10 @@
<!-- Exists for backwards compatibility; profile doesn't do anything -->
Review comment:
Already revert it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf commented on issue #1293: [HUDI-585]
Optimize the steps of building with scala-2.12
Posted by GitBox <gi...@apache.org>.
leesf commented on issue #1293: [HUDI-585] Optimize the steps of building with scala-2.12
URL: https://github.com/apache/incubator-hudi/pull/1293#issuecomment-581020774
@zhedoubushishi Did you also get the error https://github.com/apache/incubator-hudi/pull/1226#issuecomment-575513618 in windows?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf commented on issue #1293: [HUDI-585]
Optimize the steps of building with scala-2.12
Posted by GitBox <gi...@apache.org>.
leesf commented on issue #1293: [HUDI-585] Optimize the steps of building with scala-2.12
URL: https://github.com/apache/incubator-hudi/pull/1293#issuecomment-581020515
> Hi @leesf, from [README](https://github.com/apache/incubator-hudi#building-apache-hudi-from-source) doc, should use `Unix-like system (like Linux, Mac OS X)`.
Thanks for pointing it out @lamber-ken , now I would compile success in Linux.
Also I would compile success in Windows at master branch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on issue #1293: [HUDI-585]
Optimize the steps of building with scala-2.12
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #1293: [HUDI-585] Optimize the steps of building with scala-2.12
URL: https://github.com/apache/incubator-hudi/pull/1293#issuecomment-581788516
Looks like you folks already have a handle on this :) .. @leesf I will let you take this across the finish line
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf merged pull request #1293: [HUDI-585]
Optimize the steps of building with scala-2.12
Posted by GitBox <gi...@apache.org>.
leesf merged pull request #1293: [HUDI-585] Optimize the steps of building with scala-2.12
URL: https://github.com/apache/incubator-hudi/pull/1293
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf commented on issue #1293: [HUDI-585]
Optimize the steps of building with scala-2.12
Posted by GitBox <gi...@apache.org>.
leesf commented on issue #1293: [HUDI-585] Optimize the steps of building with scala-2.12
URL: https://github.com/apache/incubator-hudi/pull/1293#issuecomment-582405982
@lamber-ken Sorry for late update, tested and worked fine in Linux, let's go forward.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf commented on a change in pull request #1293:
[HUDI-585] Optimize the steps of building with scala-2.12
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #1293: [HUDI-585] Optimize the steps of building with scala-2.12
URL: https://github.com/apache/incubator-hudi/pull/1293#discussion_r375265150
##########
File path: pom.xml
##########
@@ -1052,6 +1052,10 @@
<!-- Exists for backwards compatibility; profile doesn't do anything -->
Review comment:
So this line would be removed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] lamber-ken commented on a change in pull request
#1293: [HUDI-585] Optimize the steps of building with scala-2.12
Posted by GitBox <gi...@apache.org>.
lamber-ken commented on a change in pull request #1293: [HUDI-585] Optimize the steps of building with scala-2.12
URL: https://github.com/apache/incubator-hudi/pull/1293#discussion_r375269481
##########
File path: pom.xml
##########
@@ -1052,6 +1052,10 @@
<!-- Exists for backwards compatibility; profile doesn't do anything -->
Review comment:
Not necessary, this profile doesn't do anything.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] lamber-ken commented on issue #1293: [HUDI-585]
Optimize the steps of building with scala-2.12
Posted by GitBox <gi...@apache.org>.
lamber-ken commented on issue #1293: [HUDI-585] Optimize the steps of building with scala-2.12
URL: https://github.com/apache/incubator-hudi/pull/1293#issuecomment-581016392
Hi @leesf, from [README](https://github.com/apache/incubator-hudi#building-apache-hudi-from-source) doc, should use `Unix-like system (like Linux, Mac OS X)`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] zhedoubushishi commented on issue #1293:
[HUDI-585] Optimize the steps of building with scala-2.12
Posted by GitBox <gi...@apache.org>.
zhedoubushishi commented on issue #1293: [HUDI-585] Optimize the steps of building with scala-2.12
URL: https://github.com/apache/incubator-hudi/pull/1293#issuecomment-581104630
> @zhedoubushishi Did you also get the error [#1226 (comment)](https://github.com/apache/incubator-hudi/pull/1226#issuecomment-575513618) in windows?
No I use mac. I tried to use the code in this branch and I still got this error:
```
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (process-resource-bundles) on project hudi-cli: Failed to resolve dependencies for one or more projects in the reactor. Reason: Unable to get dependency information for org.apache.hudi:hudi-spark_2.11:jar:0.5.2-SNAPSHOT: Failed to retrieve POM for org.apache.hudi:hudi-spark_2.11:jar:0.5.2-SNAPSHOT: Could not transfer artifact org.apache.hudi:hudi-spark_2.11:pom:0.5.2-SNAPSHOT
[ERROR] org.apache.hudi:hudi-spark_2.11:jar:0.5.2-SNAPSHOT
```
Can you delete this folder ```~/.m2/repository/org/apache/hudi``` first and build again to see if it works?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] lamber-ken commented on issue #1293: [HUDI-585]
Optimize the steps of building with scala-2.12
Posted by GitBox <gi...@apache.org>.
lamber-ken commented on issue #1293: [HUDI-585] Optimize the steps of building with scala-2.12
URL: https://github.com/apache/incubator-hudi/pull/1293#issuecomment-582415582
> @lamber-ken Sorry for late update, tested and worked fine in Linux, let's go forward.
Welcome :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] lamber-ken edited a comment on issue #1293:
[HUDI-585] Optimize the steps of building with scala-2.12
Posted by GitBox <gi...@apache.org>.
lamber-ken edited a comment on issue #1293: [HUDI-585] Optimize the steps of building with scala-2.12
URL: https://github.com/apache/incubator-hudi/pull/1293#issuecomment-582415582
> @lamber-ken Sorry for late update, tested and worked fine in Linux, let's go forward.
No problem :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf commented on issue #1293: [HUDI-585]
Optimize the steps of building with scala-2.12
Posted by GitBox <gi...@apache.org>.
leesf commented on issue #1293: [HUDI-585] Optimize the steps of building with scala-2.12
URL: https://github.com/apache/incubator-hudi/pull/1293#issuecomment-581015695
> Hi @leesf, you're welcome, I need some information about your env, thanks :)
>
> * Maven version
> * Java version
> * OS version
> * Build command
@lamber-ken FYI:
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)
Maven home: D:\01_Software\apache-maven-3.3.9-bin\apache-maven-3.3.9
Java version: 1.8.0_144, vendor: Oracle Corporation
Java home: C:\Program Files\Java\jdk1.8.0_144\jre
Default locale: zh_CN, platform encoding: GBK
OS name: "windows 7", version: "6.1", arch: "amd64", family: "dos"
Build Command: `mvn clean package -DskipTests -DskipITs -Pscala-2.12`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] lamber-ken commented on issue #1293: [HUDI-585]
Optimize the steps of building with scala-2.12
Posted by GitBox <gi...@apache.org>.
lamber-ken commented on issue #1293: [HUDI-585] Optimize the steps of building with scala-2.12
URL: https://github.com/apache/incubator-hudi/pull/1293#issuecomment-582456190
> +1 LGTM, also tested in window, except for hudi-integ-test(not install docker), other modules works fine. Thanks for your work @lamber-ken . Merging.
Thanks :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf commented on issue #1293: [HUDI-585]
Optimize the steps of building with scala-2.12
Posted by GitBox <gi...@apache.org>.
leesf commented on issue #1293: [HUDI-585] Optimize the steps of building with scala-2.12
URL: https://github.com/apache/incubator-hudi/pull/1293#issuecomment-581142414
Thanks for the update, will check it again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf commented on issue #1293: [HUDI-585]
Optimize the steps of building with scala-2.12
Posted by GitBox <gi...@apache.org>.
leesf commented on issue #1293: [HUDI-585] Optimize the steps of building with scala-2.12
URL: https://github.com/apache/incubator-hudi/pull/1293#issuecomment-581116499
Yes @zhedoubushishi , I also got the error after delete the folder. CC @lamber-ken
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] lamber-ken commented on issue #1293: [HUDI-585]
Optimize the steps of building with scala-2.12
Posted by GitBox <gi...@apache.org>.
lamber-ken commented on issue #1293: [HUDI-585] Optimize the steps of building with scala-2.12
URL: https://github.com/apache/incubator-hudi/pull/1293#issuecomment-581009348
Hi @leesf, you're welcome, I need some information about your env, thanks :)
- Maven version
- Java version
- OS version
- Build command
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] zhedoubushishi commented on issue #1293:
[HUDI-585] Optimize the steps of building with scala-2.12
Posted by GitBox <gi...@apache.org>.
zhedoubushishi commented on issue #1293: [HUDI-585] Optimize the steps of building with scala-2.12
URL: https://github.com/apache/incubator-hudi/pull/1293#issuecomment-582124317
@lamber-ken Tested. It works for me!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] lamber-ken commented on issue #1293: [HUDI-585]
Optimize the steps of building with scala-2.12
Posted by GitBox <gi...@apache.org>.
lamber-ken commented on issue #1293: [HUDI-585] Optimize the steps of building with scala-2.12
URL: https://github.com/apache/incubator-hudi/pull/1293#issuecomment-581135530
Thanks very much for testing this pr @leesf @zhedoubushishi. I updated the pr, please use bellow command
```
mvn clean package -DskipTests -DskipITs -Dscala-2.12
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf commented on issue #1293: [HUDI-585]
Optimize the steps of building with scala-2.12
Posted by GitBox <gi...@apache.org>.
leesf commented on issue #1293: [HUDI-585] Optimize the steps of building with scala-2.12
URL: https://github.com/apache/incubator-hudi/pull/1293#issuecomment-581007728
Thanks for opening the PR @lamber-ken ! I compile in my local with scala-2.12, and got the following exception:
```java
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (process-resource-bundles) on project hudi-cli: Failed to resolve dependencies for one
or more projects in the reactor. Reason: Unable to get dependency information for org.apache.hudi:hudi-spark_2.11:jar:0.5.2-SNAPSHOT: Failed to retrieve POM for org.apache.hudi:hudi-spa
rk_2.11:jar:0.5.2-SNAPSHOT: Could not transfer artifact org.apache.hudi:hudi-spark_2.11:pom:0.5.2-SNAPSHOT from/to central1 (http://central.maven.org/maven2/): Failed to transfer file:
http://central.maven.org/maven2/org/apache/hudi/hudi-spark_2.11/0.5.2-SNAPSHOT/hudi-spark_2.11-0.5.2-SNAPSHOT.pom. Return code is: 501 , ReasonPhrase:Not Implemented.
[ERROR] org.apache.hudi:hudi-spark_2.11:jar:0.5.2-SNAPSHOT
[ERROR]
[ERROR] from the specified remote repositories:
[ERROR] central1 (http://central.maven.org/maven2/, releases=true, snapshots=true),
[ERROR] Maven Central (https://repo.maven.apache.org/maven2, releases=true, snapshots=false),
[ERROR] cloudera-repo-releases (https://repository.cloudera.com/artifactory/public/, releases=true, snapshots=false),
[ERROR] confluent (https://packages.confluent.io/maven/, releases=true, snapshots=true),
[ERROR] libs-milestone (https://repo.spring.io/libs-milestone/, releases=true, snapshots=true),
[ERROR] libs-release (https://repo.spring.io/libs-release/, releases=true, snapshots=true),
[ERROR] apache.snapshots (https://repository.apache.org/snapshots, releases=false, snapshots=true),
[ERROR] alimaven (http://maven.aliyun.com/nexus/content/groups/public/, releases=true, snapshots=false)
[ERROR] Path to dependency:
[ERROR] 1) org.apache.hudi:hudi-cli:jar:0.5.2-SNAPSHOT
[ERROR] 2) org.apache.hudi:hudi-utilities_2.12:jar:0.5.2-SNAPSHOT
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :hudi-cli
```
The reason might be https://github.com/apache/incubator-hudi/pull/1226#issuecomment-575513618.
Also pull @zhedoubushishi in since he did support scala-2.12 in (https://github.com/apache/incubator-hudi/pull/1226)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services