You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/02/10 13:15:08 UTC
[GitHub] [hudi] nsivabalan opened a new pull request #2562: [HUDI-1540] Fixing commons codec depedency in bundle jars
nsivabalan opened a new pull request #2562:
URL: https://github.com/apache/hudi/pull/2562
## What is the purpose of the pull request
*commons-codec wasn't pulled in from any of the dep. So had to explicitly add it in our bundles*
## Brief change log
*(for example:)*
- *Added commons-codec dependency to spark and utilities bundle*
## Verify this pull request
manually verified.
```
tar -tvf packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.11-0.8.0-SNAPSHOT.jar | grep commons-codec
drwxrwxr-x 0 0 0 0 Feb 10 07:59 META-INF/maven/commons-codec/
drwxrwxr-x 0 0 0 0 Feb 10 07:59 META-INF/maven/commons-codec/commons-codec/
-rw-rw-r-- 0 0 0 10494 Aug 6 2009 META-INF/maven/commons-codec/commons-codec/pom.xml
-rw-rw-r-- 0 0 0 114 Aug 6 2009 META-INF/maven/commons-codec/commons-codec/pom.properties
sivabala-C02XG219JGH6:hudi sivabala$ tar -tvf packaging/hudi-spark-bundle/target/hudi-spark-bundle_2.11-0.8.0-SNAPSHOT.jar | grep commons-codec
drwxrwxr-x 0 0 0 0 Feb 10 08:00 META-INF/maven/commons-codec/
drwxrwxr-x 0 0 0 0 Feb 10 08:00 META-INF/maven/commons-codec/commons-codec/
-rw-rw-r-- 0 0 0 10494 Aug 6 2009 META-INF/maven/commons-codec/commons-codec/pom.xml
-rw-rw-r-- 0 0 0 114 Aug 6 2009 META-INF/maven/commons-codec/commons-codec/pom.properties
```
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #2562: [HUDI-1540] Fixing commons codec depedency in bundle jars
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #2562:
URL: https://github.com/apache/hudi/pull/2562#issuecomment-777083810
so the reason, this was left out was that we could use it from spark/hive etc, who should already have this? whats the actual problem we are addressing here , by including these jars?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2562: [HUDI-1540] Fixing commons codec depedency in bundle jars
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2562:
URL: https://github.com/apache/hudi/pull/2562#issuecomment-776721370
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2562?src=pr&el=h1) Report
> Merging [#2562](https://codecov.io/gh/apache/hudi/pull/2562?src=pr&el=desc) (5258866) into [master](https://codecov.io/gh/apache/hudi/commit/a2f85d90de73a58e924b4de757d09d6133b046a4?el=desc) (a2f85d9) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/2562?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2562 +/- ##
============================================
- Coverage 50.90% 50.90% -0.01%
- Complexity 3167 3168 +1
============================================
Files 433 433
Lines 19806 19806
Branches 2032 2032
============================================
- Hits 10083 10082 -1
- Misses 8904 8905 +1
Partials 819 819
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `36.90% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudicommon | `51.38% <ø> (-0.03%)` | `0.00 <ø> (ø)` | |
| hudiflink | `43.21% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudihadoopmr | `33.16% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisparkdatasource | `69.46% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisync | `48.61% <ø> (ø)` | `0.00 <ø> (ø)` | |
| huditimelineservice | `66.49% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudiutilities | `69.51% <ø> (+0.05%)` | `0.00 <ø> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2562?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...e/hudi/common/table/log/HoodieLogFormatWriter.java](https://codecov.io/gh/apache/hudi/pull/2562/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9Ib29kaWVMb2dGb3JtYXRXcml0ZXIuamF2YQ==) | `78.12% <0.00%> (-1.57%)` | `26.00% <0.00%> (ø%)` | |
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2562/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.86% <0.00%> (+0.35%)` | `51.00% <0.00%> (+1.00%)` | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar merged pull request #2562: [HUDI-1540] Fixing commons codec depedency in bundle jars
Posted by GitBox <gi...@apache.org>.
vinothchandar merged pull request #2562:
URL: https://github.com/apache/hudi/pull/2562
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #2562: [HUDI-1540] Fixing commons codec depedency in bundle jars
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #2562:
URL: https://github.com/apache/hudi/pull/2562#issuecomment-777084003
+1 on relocation. we need to be really sure about relocating or not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #2562: [HUDI-1540] Fixing commons codec depedency in bundle jars
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2562:
URL: https://github.com/apache/hudi/pull/2562#issuecomment-778267262
mvn dependency:tree -Dincludes=commons-codec:commons-codec
```
.
.
[INFO] ---------------< org.apache.hudi:hudi-spark-bundle_2.11 >---------------
[INFO] Building hudi-spark-bundle_2.11 0.8.0-SNAPSHOT [24/42]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ hudi-spark-bundle_2.11 ---
[INFO] org.apache.hudi:hudi-spark-bundle_2.11:jar:0.8.0-SNAPSHOT
[INFO] \- org.apache.hive:hive-service:jar:2.3.1:provided
[INFO] \- commons-codec:commons-codec:jar:1.4:compile
.
.
[INFO] ------------------< org.apache.hudi:hudi-spark2_2.11 >------------------
[INFO] Building hudi-spark2_2.11 0.8.0-SNAPSHOT [12/42]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ hudi-spark2_2.11 ---
[INFO] org.apache.hudi:hudi-spark2_2.11:jar:0.8.0-SNAPSHOT
[INFO] \- org.apache.hudi:hudi-client-common:jar:0.8.0-SNAPSHOT:compile
[INFO] \- org.apache.hadoop:hadoop-hdfs:jar:tests:2.7.3:compile
[INFO] \- commons-codec:commons-codec:jar:1.4:compile
[INFO]
[INFO] ------------------< org.apache.hudi:hudi-spark3_2.12 >------------------
[INFO] Building hudi-spark3_2.12 0.8.0-SNAPSHOT [13/42]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ hudi-spark3_2.12 ---
[INFO] org.apache.hudi:hudi-spark3_2.12:jar:0.8.0-SNAPSHOT
[INFO] \- org.apache.spark:spark-sql_2.12:jar:3.0.0:compile (optional)
[INFO] \- org.apache.spark:spark-catalyst_2.12:jar:3.0.0:compile (optional)
[INFO] \- commons-codec:commons-codec:jar:1.10:compile
[INFO]
[INFO] ------------------< org.apache.hudi:hudi-spark_2.11 >-------------------
[INFO] Building hudi-spark_2.11 0.8.0-SNAPSHOT [14/42]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ hudi-spark_2.11 ---
[INFO] org.apache.hudi:hudi-spark_2.11:jar:0.8.0-SNAPSHOT
[INFO] \- org.apache.hadoop:hadoop-common:jar:2.7.3:provided
[INFO] \- commons-codec:commons-codec:jar:1.4:compile
.
.
[INFO] ----------------< org.apache.hudi:hudi-utilities_2.11 >-----------------
[INFO] Building hudi-utilities_2.11 0.8.0-SNAPSHOT [15/42]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ hudi-utilities_2.11 ---
[INFO] org.apache.hudi:hudi-utilities_2.11:jar:0.8.0-SNAPSHOT
[INFO] \- org.apache.hadoop:hadoop-hdfs:jar:tests:2.7.3:test
[INFO] \- commons-codec:commons-codec:jar:1.4:compile
.
[INFO] -------------< org.apache.hudi:hudi-utilities-bundle_2.11 >-------------
[INFO] Building hudi-utilities-bundle_2.11 0.8.0-SNAPSHOT [16/42]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ hudi-utilities-bundle_2.11 ---
[INFO] org.apache.hudi:hudi-utilities-bundle_2.11:jar:0.8.0-SNAPSHOT
[INFO] \- org.apache.hive:hive-service:jar:2.3.1:provided
[INFO] \- commons-codec:commons-codec:jar:1.4:compile
.
.
```
Full output: https://gist.github.com/nsivabalan/aa5b78c8346045e966b2b916821cbed6
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io commented on pull request #2562: [HUDI-1540] Fixing commons codec depedency in bundle jars
Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #2562:
URL: https://github.com/apache/hudi/pull/2562#issuecomment-776721370
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2562?src=pr&el=h1) Report
> Merging [#2562](https://codecov.io/gh/apache/hudi/pull/2562?src=pr&el=desc) (5258866) into [master](https://codecov.io/gh/apache/hudi/commit/a2f85d90de73a58e924b4de757d09d6133b046a4?el=desc) (a2f85d9) will **decrease** coverage by `0.20%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/2562?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2562 +/- ##
============================================
- Coverage 50.90% 50.70% -0.21%
+ Complexity 3167 2985 -182
============================================
Files 433 412 -21
Lines 19806 18372 -1434
Branches 2032 1898 -134
============================================
- Hits 10083 9316 -767
+ Misses 8904 8303 -601
+ Partials 819 753 -66
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `36.90% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudicommon | `51.38% <ø> (-0.03%)` | `0.00 <ø> (ø)` | |
| hudiflink | `43.21% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudihadoopmr | `33.16% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisparkdatasource | `69.46% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisync | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `69.51% <ø> (+0.05%)` | `0.00 <ø> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2562?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...e/hudi/common/table/log/HoodieLogFormatWriter.java](https://codecov.io/gh/apache/hudi/pull/2562/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9Ib29kaWVMb2dGb3JtYXRXcml0ZXIuamF2YQ==) | `78.12% <0.00%> (-1.57%)` | `26.00% <0.00%> (ø%)` | |
| [.../apache/hudi/hive/MultiPartKeysValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/2562/diff?src=pr&el=tree#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTXVsdGlQYXJ0S2V5c1ZhbHVlRXh0cmFjdG9yLmphdmE=) | | | |
| [.../org/apache/hudi/hive/HoodieHiveSyncException.java](https://codecov.io/gh/apache/hudi/pull/2562/diff?src=pr&el=tree#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNFeGNlcHRpb24uamF2YQ==) | | | |
| [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/2562/diff?src=pr&el=tree#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | | | |
| [.../apache/hudi/timeline/service/TimelineService.java](https://codecov.io/gh/apache/hudi/pull/2562/diff?src=pr&el=tree#diff-aHVkaS10aW1lbGluZS1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RpbWVsaW5lL3NlcnZpY2UvVGltZWxpbmVTZXJ2aWNlLmphdmE=) | | | |
| [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/2562/diff?src=pr&el=tree#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | | | |
| [...i/hive/SlashEncodedDayPartitionValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/2562/diff?src=pr&el=tree#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvU2xhc2hFbmNvZGVkRGF5UGFydGl0aW9uVmFsdWVFeHRyYWN0b3IuamF2YQ==) | | | |
| [...g/apache/hudi/timeline/service/RequestHandler.java](https://codecov.io/gh/apache/hudi/pull/2562/diff?src=pr&el=tree#diff-aHVkaS10aW1lbGluZS1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RpbWVsaW5lL3NlcnZpY2UvUmVxdWVzdEhhbmRsZXIuamF2YQ==) | | | |
| [...in/java/org/apache/hudi/hive/SchemaDifference.java](https://codecov.io/gh/apache/hudi/pull/2562/diff?src=pr&el=tree#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvU2NoZW1hRGlmZmVyZW5jZS5qYXZh) | | | |
| [...udi/timeline/service/handlers/BaseFileHandler.java](https://codecov.io/gh/apache/hudi/pull/2562/diff?src=pr&el=tree#diff-aHVkaS10aW1lbGluZS1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RpbWVsaW5lL3NlcnZpY2UvaGFuZGxlcnMvQmFzZUZpbGVIYW5kbGVyLmphdmE=) | | | |
| ... and [13 more](https://codecov.io/gh/apache/hudi/pull/2562/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on pull request #2562: [HUDI-1540] Fixing commons codec depedency in bundle jars
Posted by GitBox <gi...@apache.org>.
n3nash commented on pull request #2562:
URL: https://github.com/apache/hudi/pull/2562#issuecomment-776886916
@nsivabalan can you confirm that the package is relocated and not a hard dependency on the actual package ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #2562: [HUDI-1540] Fixing commons codec depedency in bundle jars
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #2562:
URL: https://github.com/apache/hudi/pull/2562#issuecomment-788132839
cc @yanghua @garyli1019 @leesf to consider this for flink bundle as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #2562: [HUDI-1540] Fixing commons codec depedency in bundle jars
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2562:
URL: https://github.com/apache/hudi/pull/2562#issuecomment-778675900
and yes, it is relocated as well.
tar -tvf packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.11-0.8.0-SNAPSHOT.jar | grep commons.codec | head -3
drwxrwxr-x 0 0 0 0 Feb 13 15:39 org/apache/hudi/org/apache/commons/codec/
drwxrwxr-x 0 0 0 0 Feb 13 15:39 org/apache/hudi/org/apache/commons/codec/binary/
-rw-rw-r-- 0 0 0 11317 Feb 13 15:39 org/apache/hudi/org/apache/commons/codec/binary/Base64.class
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #2562: [HUDI-1540] Fixing commons codec depedency in bundle jars
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2562:
URL: https://github.com/apache/hudi/pull/2562#issuecomment-784267568
@vinothchandar @n3nash : gentle ping.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org