You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Ahmed Abualsaud via dev <de...@beam.apache.org> on 2024/07/29 20:23:23 UTC
How to override sticky dependencies in gradle?
Hey all,
Does anyone have experience with sticky dependencies in gradle? I'm
experimenting with the new Managed Iceberg connector and trying to write to
GCS with a Hive catalog.
I naturally need to pull some hive dependencies, but one particular module
is very stubborn with its dependencies: *org.apache.hive:hive-exec:3.1.3.*
I need to override its dependency on *com.google.protobuf:protobuf-java*
(version 2.5.0) to use a more up-to-date version, but nothing I do seems to
work
I tried the following in my gradle file, but running the code always
defaults to classes in *protobuf-java:2.5.0*:
configurations.all {
resolutionStrategy.force 'com.google.protobuf:protobuf-java:3.25.3'
}
dependencies {
testImplementation 'com.google.protobuf:protobuf-java:3.25.3'
testImplementation("org.apache.hive:hive-exec:3.1.3") {
exclude group: "com.google.protobuf", module: "protobuf-java"
}
testImplementation "org.apache.iceberg:iceberg-hive-metastore:1.4.2"
testImplementation("org.apache.hive.hcatalog:hive-hcatalog-core:3.1.3") {
exclude group: "org.apache.hive", module: "hive-exec"
exclude group: "com.google.protobuf", module: "protobuf-java"
}
constraints {
testImplementation('com.google.protobuf:protobuf-java') {
version {
strictly '3.25.3'
}
}
}
}
I also tried adding the following, which stripped *protobuf-java* from
everything except *hive-exec:*
configurations.all {
exclude group: 'com.google.protobuf', module: 'protobuf-java'
}
I also had similar pain points when trying to override
org.apache.parquet:parquet-column. The hive module seems very intent on
keeping its dependencies.
Can anyone share their experiences with stubborn dependencies such as
these? Would appreciate any tips!
Best,
Ahmed
Re: How to override sticky dependencies in gradle?
Posted by Ahmed Abualsaud via dev <de...@beam.apache.org>.
Update: "org.apache.hive:hive-exec" is actually a shaded jar that doesn't
do any repackaging for popular libraries (e.g. like "protobuf-java"). My
gradle logic was working as intended, but I was still running into issues
because multiple classes with the same name were available to the Java
class loader (one from my "protobuf-java" dependency, another from the
shaded "hive-exec").
We were able to work around this by shading "hive-exec" ourselves and
relocating the problematic namespace, similar to how it's done here:
https://github.com/GradleUp/shadow/issues/384
Shoutout to @Yi Hu <ya...@google.com> for the help!
On Mon, Jul 29, 2024 at 4:48 PM Tomo Suzuki <su...@google.com> wrote:
> "resolutionStrategy.force" should work. Use "./gradlew dependencies" to
> debug the dependencies.
> https://docs.gradle.org/current/userguide/viewing_debugging_dependencies.html
>
>
> On Mon, Jul 29, 2024 at 4:23 PM Ahmed Abualsaud via dev <
> dev@beam.apache.org> wrote:
>
>> Hey all,
>>
>> Does anyone have experience with sticky dependencies in gradle? I'm
>> experimenting with the new Managed Iceberg connector and trying to write to
>> GCS with a Hive catalog.
>>
>> I naturally need to pull some hive dependencies, but one particular
>> module is very stubborn with its dependencies:
>> *org.apache.hive:hive-exec:3.1.3.*
>>
>> I need to override its dependency on *com.google.protobuf:protobuf-java*
>> (version 2.5.0) to use a more up-to-date version, but nothing I do seems
>> to work
>>
>> I tried the following in my gradle file, but running the code always
>> defaults to classes in *protobuf-java:2.5.0*:
>>
>>
>> configurations.all {
>> resolutionStrategy.force 'com.google.protobuf:protobuf-java:3.25.3'
>> }
>>
>> dependencies {
>> testImplementation 'com.google.protobuf:protobuf-java:3.25.3'
>>
>> testImplementation("org.apache.hive:hive-exec:3.1.3") {
>> exclude group: "com.google.protobuf", module: "protobuf-java"
>> }
>> testImplementation "org.apache.iceberg:iceberg-hive-metastore:1.4.2"
>> testImplementation("org.apache.hive.hcatalog:hive-hcatalog-core:3.1.3") {
>> exclude group: "org.apache.hive", module: "hive-exec"
>> exclude group: "com.google.protobuf", module: "protobuf-java"
>> }
>>
>> constraints {
>> testImplementation('com.google.protobuf:protobuf-java') {
>> version {
>> strictly '3.25.3'
>> }
>> }
>> }
>> }
>>
>>
>> I also tried adding the following, which stripped *protobuf-java* from
>> everything except *hive-exec:*
>>
>>
>> configurations.all {
>> exclude group: 'com.google.protobuf', module: 'protobuf-java'
>> }
>>
>>
>> I also had similar pain points when trying to override
>> org.apache.parquet:parquet-column. The hive module seems very intent on
>> keeping its dependencies.
>>
>> Can anyone share their experiences with stubborn dependencies such as
>> these? Would appreciate any tips!
>>
>> Best,
>> Ahmed
>>
>
>
> --
> Regards,
> Tomo
>
Re: How to override sticky dependencies in gradle?
Posted by Tomo Suzuki via dev <de...@beam.apache.org>.
"resolutionStrategy.force" should work. Use "./gradlew dependencies" to
debug the dependencies.
https://docs.gradle.org/current/userguide/viewing_debugging_dependencies.html
On Mon, Jul 29, 2024 at 4:23 PM Ahmed Abualsaud via dev <de...@beam.apache.org>
wrote:
> Hey all,
>
> Does anyone have experience with sticky dependencies in gradle? I'm
> experimenting with the new Managed Iceberg connector and trying to write to
> GCS with a Hive catalog.
>
> I naturally need to pull some hive dependencies, but one particular module
> is very stubborn with its dependencies: *org.apache.hive:hive-exec:3.1.3.*
>
> I need to override its dependency on *com.google.protobuf:protobuf-java*
> (version 2.5.0) to use a more up-to-date version, but nothing I do seems
> to work
>
> I tried the following in my gradle file, but running the code always
> defaults to classes in *protobuf-java:2.5.0*:
>
>
> configurations.all {
> resolutionStrategy.force 'com.google.protobuf:protobuf-java:3.25.3'
> }
>
> dependencies {
> testImplementation 'com.google.protobuf:protobuf-java:3.25.3'
>
> testImplementation("org.apache.hive:hive-exec:3.1.3") {
> exclude group: "com.google.protobuf", module: "protobuf-java"
> }
> testImplementation "org.apache.iceberg:iceberg-hive-metastore:1.4.2"
> testImplementation("org.apache.hive.hcatalog:hive-hcatalog-core:3.1.3") {
> exclude group: "org.apache.hive", module: "hive-exec"
> exclude group: "com.google.protobuf", module: "protobuf-java"
> }
>
> constraints {
> testImplementation('com.google.protobuf:protobuf-java') {
> version {
> strictly '3.25.3'
> }
> }
> }
> }
>
>
> I also tried adding the following, which stripped *protobuf-java* from
> everything except *hive-exec:*
>
>
> configurations.all {
> exclude group: 'com.google.protobuf', module: 'protobuf-java'
> }
>
>
> I also had similar pain points when trying to override
> org.apache.parquet:parquet-column. The hive module seems very intent on
> keeping its dependencies.
>
> Can anyone share their experiences with stubborn dependencies such as
> these? Would appreciate any tips!
>
> Best,
> Ahmed
>
--
Regards,
Tomo