You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Jackie-Jiang (via GitHub)" <gi...@apache.org> on 2024/08/17 06:48:46 UTC

[PR] [Dependency] Cleanup unnecessary dependencies and exclusions [pinot]

Jackie-Jiang opened a new pull request, #13832:
URL: https://github.com/apache/pinot/pull/13832

   Also add the following libraries to the enforcer banned list:
   - `log4j:log4j`: Use `org.apache.logging.log4j:log4j-core`
   - `org.slf4j:slf4j-log4j12`: Use `org.apache.logging.log4j:log4j-slf4j2-impl`
   - `com.sun.jersey`: Use `org.glassfish.jersey`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Dependency] Cleanup unnecessary dependencies and exclusions [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #13832:
URL: https://github.com/apache/pinot/pull/13832#issuecomment-2294748191

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13832?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   All modified and coverable lines are covered by tests :white_check_mark:
   > Project coverage is 27.89%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`114687c`)](https://app.codecov.io/gh/apache/pinot/commit/114687cd8ce787d9ded1461bc2de2d6412c54c94?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 901 commits behind head on master.
   
   > :exclamation:  There is a different number of reports uploaded between BASE (59551e4) and HEAD (114687c). Click for more details.
   > 
   > <details><summary>HEAD has 23 uploads less than BASE</summary>
   >
   >| Flag | BASE (59551e4) | HEAD (114687c) |
   >|------|------|------|
   >|integration|7|5|
   >|temurin|12|7|
   >|java-21|7|6|
   >|skip-bytebuffers-false|7|4|
   >|unittests|5|2|
   >|unittests1|2|0|
   >|java-11|5|1|
   >|unittests2|3|2|
   >|custom-integration1|2|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@              Coverage Diff              @@
   ##             master   #13832       +/-   ##
   =============================================
   - Coverage     61.75%   27.89%   -33.86%     
   + Complexity      207      198        -9     
   =============================================
     Files          2436     2568      +132     
     Lines        133233   141942     +8709     
     Branches      20636    22083     +1447     
   =============================================
   - Hits          82274    39595    -42679     
   - Misses        44911    99297    +54386     
   + Partials       6048     3050     -2998     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/13832/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/13832/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/13832/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <ø> (-0.01%)` | :arrow_down: |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/13832/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <ø> (-0.01%)` | :arrow_down: |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/13832/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/13832/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-61.71%)` | :arrow_down: |
   | [java-21](https://app.codecov.io/gh/apache/pinot/pull/13832/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.89% <ø> (-33.73%)` | :arrow_down: |
   | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/13832/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.88% <ø> (-33.86%)` | :arrow_down: |
   | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/13832/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.89% <ø> (+0.16%)` | :arrow_up: |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/13832/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.89% <ø> (-33.86%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/13832/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.89% <ø> (-33.86%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/13832/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/13832/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.89% <ø> (+0.15%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/13832?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Dependency] Cleanup unnecessary dependencies and exclusions [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on PR #13832:
URL: https://github.com/apache/pinot/pull/13832#issuecomment-2300006904

   [Help wanted] I've been running into the following issue in quick-start:
   ```
   java.lang.NoClassDefFoundError: org/apache/pulsar/shade/org/yaml/snakeyaml/internal/Logger (wrong name: org/yaml/snakeyaml/internal/Logger)
           at java.base/java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
           at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
           at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) ~[?:?]
           at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800) ~[?:?]
           at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698) ~[?:?]
           at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621) ~[?:?]
           at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579) ~[?:?]
           at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) ~[?:?]
           at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[?:?]
           at org.yaml.snakeyaml.TypeDescription.<clinit>(TypeDescription.java:43) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-4e5a6a24b963fc0bdbbbb76e42cb7c1af4575132]
           at org.yaml.snakeyaml.constructor.Constructor.<init>(Constructor.java:60) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-4e5a6a24b963fc0bdbbbb76e42cb7c1af4575132]
           at org.yaml.snakeyaml.constructor.Constructor.<init>(Constructor.java:50) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-4e5a6a24b963fc0bdbbbb76e42cb7c1af4575132]
           at org.yaml.snakeyaml.Yaml.<init>(Yaml.java:64) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-4e5a6a24b963fc0bdbbbb76e42cb7c1af4575132]
           at org.apache.pinot.tools.BootstrapTableTool.setupOfflineData(BootstrapTableTool.java:217) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-4e5a6a24b963fc0bdbbbb76e42cb7c1af4575132]
           at org.apache.pinot.tools.BootstrapTableTool.bootstrapOfflineTable(BootstrapTableTool.java:194) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-4e5a6a24b963fc0bdbbbb76e42cb7c1af4575132]
           at org.apache.pinot.tools.BootstrapTableTool.execute(BootstrapTableTool.java:104) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-4e5a6a24b963fc0bdbbbb76e42cb7c1af4575132]
           at org.apache.pinot.tools.admin.command.QuickstartRunner.bootstrapTable(QuickstartRunner.java:232) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-4e5a6a24b963fc0bdbbbb76e42cb7c1af4575132]
           at org.apache.pinot.tools.Quickstart.execute(Quickstart.java:86) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-4e5a6a24b963fc0bdbbbb76e42cb7c1af4575132]
           at org.apache.pinot.tools.admin.command.QuickStartCommand.execute(QuickStartCommand.java:178) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-4e5a6a24b963fc0bdbbbb76e42cb7c1af4575132]
           at org.apache.pinot.tools.Command.call(Command.java:33) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-4e5a6a24b963fc0bdbbbb76e42cb7c1af4575132]
           at org.apache.pinot.tools.Command.call(Command.java:29) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-4e5a6a24b963fc0bdbbbb76e42cb7c1af4575132]
           at picocli.CommandLine.executeUserObject(CommandLine.java:2045) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-4e5a6a24b963fc0bdbbbb76e42cb7c1af4575132]
           at picocli.CommandLine.access$1500(CommandLine.java:148) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-4e5a6a24b963fc0bdbbbb76e42cb7c1af4575132]
           at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2465) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-4e5a6a24b963fc0bdbbbb76e42cb7c1af4575132]
           at picocli.CommandLine$RunLast.handle(CommandLine.java:2457) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-4e5a6a24b963fc0bdbbbb76e42cb7c1af4575132]
           at picocli.CommandLine$RunLast.handle(CommandLine.java:2419) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-4e5a6a24b963fc0bdbbbb76e42cb7c1af4575132]
           at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2277) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-4e5a6a24b963fc0bdbbbb76e42cb7c1af4575132]
           at picocli.CommandLine$RunLast.execute(CommandLine.java:2421) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-4e5a6a24b963fc0bdbbbb76e42cb7c1af4575132]
           at picocli.CommandLine.execute(CommandLine.java:2174) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-4e5a6a24b963fc0bdbbbb76e42cb7c1af4575132]
           at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:173) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-4e5a6a24b963fc0bdbbbb76e42cb7c1af4575132]
           at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:204) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-4e5a6a24b963fc0bdbbbb76e42cb7c1af4575132]
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Dependency] Cleanup unnecessary dependencies and exclusions [pinot]

Posted by "dang-stripe (via GitHub)" <gi...@apache.org>.
dang-stripe commented on PR #13832:
URL: https://github.com/apache/pinot/pull/13832#issuecomment-2477352667

   @Jackie-Jiang curator was pulled in by hadoop-common which some pinot modules depended on so it showed up in our pinot uberjar whereas it didn't exist before. the uberjar is at the beginning of our classpath before our plugins so the version was getting loaded first.
   
   ```
   [INFO] -------------------< org.apache.pinot:pinot-parquet >-------------------
   [INFO] org.apache.pinot:pinot-parquet:jar:1.2.0-SNAPSHOT
   [INFO] \- org.apache.hadoop:hadoop-common:jar:3.3.6:compile
   [INFO]    \- org.apache.hadoop:hadoop-auth:jar:3.3.6:compile
   [INFO]       \- org.apache.curator:curator-framework:jar:5.2.0:compile
   
   [INFO] --------------------< org.apache.pinot:pinot-tools >--------------------
   [INFO] org.apache.pinot:pinot-tools:jar:1.2.0-SNAPSHOT
   [INFO] \- org.apache.hadoop:hadoop-common:jar:3.3.6:compile
   [INFO]    \- org.apache.hadoop:hadoop-auth:jar:3.3.6:compile
   [INFO]       \- org.apache.curator:curator-framework:jar:5.2.0:compile
   ```
   
   we added back the exclusion for `hadoop-common` in `pinot-distribution` to work around it instead so it'd get excluded in the uberjar.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Dependency] Cleanup unnecessary dependencies and exclusions [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on PR #13832:
URL: https://github.com/apache/pinot/pull/13832#issuecomment-2303036730

   Find a similar [issue](https://github.com/prometheus/jmx_exporter/issues/802), and borrowed the solution from there


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Dependency] Cleanup unnecessary dependencies and exclusions [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on PR #13832:
URL: https://github.com/apache/pinot/pull/13832#issuecomment-2477311081

   @jadami10 @dang-stripe You should be able to override `curator.version` property to avoid the version conflict.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Dependency] Cleanup unnecessary dependencies and exclusions [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on PR #13832:
URL: https://github.com/apache/pinot/pull/13832#issuecomment-2475432507

   @jadami10 It is mostly for easier dependency management, and avoid unnecessary repeated declarations.
   Which exclusion do you find necessary? Most of the removed exclusions should be redundant, where the excluded dependency is actually defined. Or are you overriding the version of some dependency? If so, can you try modifying them in the dependency management section? Then the dependency should be resolved to the desired version.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Dependency] Cleanup unnecessary dependencies and exclusions [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #13832:
URL: https://github.com/apache/pinot/pull/13832


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Dependency] Cleanup unnecessary dependencies and exclusions [pinot]

Posted by "jadami10 (via GitHub)" <gi...@apache.org>.
jadami10 commented on PR #13832:
URL: https://github.com/apache/pinot/pull/13832#issuecomment-2463465110

   @Jackie-Jiang, what was the intention of this PR? Like why did we do it?
   
   On our end, removing exclusions has suddenly brought in transitive dependencies into the uber jar that are conflicting with our own plugins. We can probably work around it, but I want to understand why we're making such large changes to the dependency structure.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Dependency] Cleanup unnecessary dependencies and exclusions [pinot]

Posted by "jadami10 (via GitHub)" <gi...@apache.org>.
jadami10 commented on PR #13832:
URL: https://github.com/apache/pinot/pull/13832#issuecomment-2475472699

   I believe in our case the issue is that hadoop came with `curator`. And we use `curator` in a plugin as well. So the version in our plugin then clashed with the version hadoop brought in
   
   cc @dang-stripe who has the specific details/versions


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org