You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "FMX (via GitHub)" <gi...@apache.org> on 2023/08/10 02:54:08 UTC
[GitHub] [incubator-celeborn] FMX opened a new pull request, #1803: [CELEBORN-885][SPARK] Shade roaring bitmap to avoid dependency conflicts
FMX opened a new pull request, #1803:
URL: https://github.com/apache/incubator-celeborn/pull/1803
### What changes were proposed in this pull request?
Shade roaring bitmap to void dependency conflicts.
### Why are the changes needed?
Some user reports that celeborn client will introduce roaring bitmap conflicts.
### Does this PR introduce _any_ user-facing change?
NO.
### How was this patch tested?
GA and cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] codecov[bot] commented on pull request #1803: [CELEBORN-885][SPARK] Shade roaring bitmap to avoid dependency conflicts
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #1803:
URL: https://github.com/apache/incubator-celeborn/pull/1803#issuecomment-1672482151
## [Codecov](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1803?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#1803](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1803?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (07c8015) into [main](https://app.codecov.io/gh/apache/incubator-celeborn/commit/6ea1ee2ec424c1011eef80a5d74c54cc1464f471?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (6ea1ee2) will **increase** coverage by `0.01%`.
> Report is 4 commits behind head on main.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## main #1803 +/- ##
==========================================
+ Coverage 46.60% 46.61% +0.01%
==========================================
Files 162 162
Lines 10079 10077 -2
Branches 928 927 -1
==========================================
Hits 4696 4696
+ Misses 5073 5071 -2
Partials 310 310
```
[see 1 file with indirect coverage changes](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1803/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] FMX commented on pull request #1803: [CELEBORN-885][SPARK] Shade RoaringBitmap to avoid dependency conflicts
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on PR #1803:
URL: https://github.com/apache/incubator-celeborn/pull/1803#issuecomment-1674148553
> Not the NOTICE-binary, technically, each jar we published to maven central should contain its own NOTICE file, which reflects the real content itself. For shaded jars, we need to maintain the NOTICE file manually.
Any reference for bundle notice into jar file？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] FMX commented on pull request #1803: [CELEBORN-885][SPARK] Shade RoaringBitmap to avoid dependency conflicts
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on PR #1803:
URL: https://github.com/apache/incubator-celeborn/pull/1803#issuecomment-1674138068
> BTW, NOTICE for the shaded module should be updated too
RoarbingBitmap has no notice and its using apache license. We already contains its license info in LICENSE-binary. Is there anything else need to put in NOTICE?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on pull request #1803: [CELEBORN-885][SPARK] Shade roaring bitmap to avoid dependency conflicts
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1803:
URL: https://github.com/apache/incubator-celeborn/pull/1803#issuecomment-1672476171
sbt should be updated too, cc @cfmcgrady
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on pull request #1803: [CELEBORN-885][SPARK] Shade roaring bitmap to avoid dependency conflicts
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on PR #1803:
URL: https://github.com/apache/incubator-celeborn/pull/1803#issuecomment-1672495136
Would you mind updating the file `project/CelebornBuild.scala`, following the same rule as we do in the Flink assembly project?
```
ShadeRule.rename("org.roaringbitmap.**" -> "org.apache.celeborn.shaded.org.roaringbitmap.@1").inAll
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on pull request #1803: [CELEBORN-885][SPARK] Shade roaring bitmap to avoid dependency conflicts
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1803:
URL: https://github.com/apache/incubator-celeborn/pull/1803#issuecomment-1673692962
BTW, NOTICE for the shaded module should be updated too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on pull request #1803: [CELEBORN-885][SPARK] Shade RoaringBitmap to avoid dependency conflicts
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1803:
URL: https://github.com/apache/incubator-celeborn/pull/1803#issuecomment-1674145557
Not the NOTICE-binary, technically, each jar we published to maven central should contain its own NOTICE file, which reflects the real content itself. For shaded jars, we need to maintain the NOTICE file manually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] FMX commented on pull request #1803: [CELEBORN-885][SPARK] Shade RoaringBitmap to avoid dependency conflicts
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on PR #1803:
URL: https://github.com/apache/incubator-celeborn/pull/1803#issuecomment-1674137622
> Would you mind updating the file `project/CelebornBuild.scala`, following the same rule as we do in the Flink assembly project?
>
> ```diff
> val cp = (assembly / fullClasspath).value
> cp filter { v =>
> val name = v.data.getName
> - !(name.startsWith("celeborn-") || name.startsWith("protobuf-java-") ||
> - name.startsWith("guava-") || name.startsWith("netty-") || name.startsWith("commons-lang3-"))
> + !(name.startsWith("celeborn-") ||
> + name.startsWith("protobuf-java-") ||
> + name.startsWith("guava-") ||
> + name.startsWith("netty-") ||
> + name.startsWith("commons-lang3-") ||
> + name.startsWith("RoaringBitmap-"))
> }
> },
>
> ShadeRule.rename("com.google.protobuf.**" -> "org.apache.celeborn.shaded.com.google.protobuf.@1").inAll,
> ShadeRule.rename("com.google.common.**" -> "org.apache.celeborn.shaded.com.google.common.@1").inAll,
> ShadeRule.rename("io.netty.**" -> "org.apache.celeborn.shaded.io.netty.@1").inAll,
> - ShadeRule.rename("org.apache.commons.**" -> "org.apache.celeborn.shaded.org.apache.commons.@1").inAll
> + ShadeRule.rename("org.apache.commons.**" -> "org.apache.celeborn.shaded.org.apache.commons.@1").inAll,
> + ShadeRule.rename("org.roaringbitmap.**" -> "org.apache.celeborn.shaded.org.roaringbitmap.@1").inAll
> ),
>
> (assembly / assemblyMergeStrategy) := {
> ```
OK，I‘ll update it.
> BTW, NOTICE for the shaded module should be updated too
RoaringBitmap has no notice and its using Apache Licencese V2. We already contains its license info in LICENSE-binary file.
> Would you mind updating the file `project/CelebornBuild.scala`, following the same rule as we do in the Flink assembly project?
>
> ```diff
> val cp = (assembly / fullClasspath).value
> cp filter { v =>
> val name = v.data.getName
> - !(name.startsWith("celeborn-") || name.startsWith("protobuf-java-") ||
> - name.startsWith("guava-") || name.startsWith("netty-") || name.startsWith("commons-lang3-"))
> + !(name.startsWith("celeborn-") ||
> + name.startsWith("protobuf-java-") ||
> + name.startsWith("guava-") ||
> + name.startsWith("netty-") ||
> + name.startsWith("commons-lang3-") ||
> + name.startsWith("RoaringBitmap-"))
> }
> },
>
> ShadeRule.rename("com.google.protobuf.**" -> "org.apache.celeborn.shaded.com.google.protobuf.@1").inAll,
> ShadeRule.rename("com.google.common.**" -> "org.apache.celeborn.shaded.com.google.common.@1").inAll,
> ShadeRule.rename("io.netty.**" -> "org.apache.celeborn.shaded.io.netty.@1").inAll,
> - ShadeRule.rename("org.apache.commons.**" -> "org.apache.celeborn.shaded.org.apache.commons.@1").inAll
> + ShadeRule.rename("org.apache.commons.**" -> "org.apache.celeborn.shaded.org.apache.commons.@1").inAll,
> + ShadeRule.rename("org.roaringbitmap.**" -> "org.apache.celeborn.shaded.org.roaringbitmap.@1").inAll
> ),
>
> (assembly / assemblyMergeStrategy) := {
> ```
I'll update it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on pull request #1803: [CELEBORN-885][SPARK] Shade RoaringBitmap to avoid dependency conflicts
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1803:
URL: https://github.com/apache/incubator-celeborn/pull/1803#issuecomment-1674149112
https://github.com/apache/kyuubi/tree/master/externals/kyuubi-spark-sql-engine/src/main/resources/META-INF
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] FMX closed pull request #1803: [CELEBORN-885][SPARK] Shade RoaringBitmap to avoid dependency conflicts
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX closed pull request #1803: [CELEBORN-885][SPARK] Shade RoaringBitmap to avoid dependency conflicts
URL: https://github.com/apache/incubator-celeborn/pull/1803
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org