Re: How to override sticky dependencies in gradle?

[Ahmed Abualsaud via dev <de...@beam.apache.org>]

Update: "org.apache.hive:hive-exec" is actually a shaded jar that doesn't
do any repackaging for popular libraries (e.g. like "protobuf-java"). My
gradle logic was working as intended, but I was still running into issues
because multiple classes with the same name were available to the Java
class loader (one from my "protobuf-java" dependency, another from the
shaded "hive-exec").

We were able to work around this by shading "hive-exec" ourselves and
relocating the problematic namespace, similar to how it's done here:
https://github.com/GradleUp/shadow/issues/384

Shoutout to @Yi Hu <ya...@google.com> for the help!

On Mon, Jul 29, 2024 at 4:48 PM Tomo Suzuki <su...@google.com> wrote:

> "resolutionStrategy.force" should work. Use "./gradlew dependencies" to
> debug the dependencies.
> https://docs.gradle.org/current/userguide/viewing_debugging_dependencies.html
>
>
> On Mon, Jul 29, 2024 at 4:23 PM Ahmed Abualsaud via dev <
> dev@beam.apache.org> wrote:
>
>> Hey all,
>>
>> Does anyone have experience with sticky dependencies in gradle? I'm
>> experimenting with the new Managed Iceberg connector and trying to write to
>> GCS with a Hive catalog.
>>
>> I naturally need to pull some hive dependencies, but one particular
>> module is very stubborn with its dependencies:
>> *org.apache.hive:hive-exec:3.1.3.*
>>
>> I need to override its dependency on *com.google.protobuf:protobuf-java*
>> (version 2.5.0) to use a more up-to-date version, but nothing I do seems
>> to work
>>
>> I tried the following in my gradle file, but running the code always
>> defaults to classes in *protobuf-java:2.5.0*:
>>
>>
>> configurations.all {
>>     resolutionStrategy.force 'com.google.protobuf:protobuf-java:3.25.3'
>> }
>>
>> dependencies {
>>     testImplementation 'com.google.protobuf:protobuf-java:3.25.3'
>>
>>     testImplementation("org.apache.hive:hive-exec:3.1.3") {
>>         exclude group: "com.google.protobuf", module: "protobuf-java"
>>     }
>>     testImplementation "org.apache.iceberg:iceberg-hive-metastore:1.4.2"
>>     testImplementation("org.apache.hive.hcatalog:hive-hcatalog-core:3.1.3") {
>>         exclude group: "org.apache.hive", module: "hive-exec"
>>         exclude group: "com.google.protobuf", module: "protobuf-java"
>>     }
>>
>>     constraints {
>>         testImplementation('com.google.protobuf:protobuf-java') {
>>             version {
>>                 strictly '3.25.3'
>>             }
>>         }
>>     }
>> }
>>
>>
>> I also tried adding the following, which stripped *protobuf-java* from
>> everything except *hive-exec:*
>>
>>
>> configurations.all {
>>     exclude group: 'com.google.protobuf', module: 'protobuf-java'
>> }
>>
>>
>> I also had similar pain points when trying to override
>> org.apache.parquet:parquet-column. The hive module seems very intent on
>> keeping its dependencies.
>>
>> Can anyone share their experiences with stubborn dependencies such as
>> these? Would appreciate any tips!
>>
>> Best,
>> Ahmed
>>
>
>
> --
> Regards,
> Tomo
>