You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 09:09:15 UTC
[I] Use spark3 profile to build hudi-aws-bundle jars for release artifacts [hudi]
hudi-bot opened a new issue, #16265:
URL: https://github.com/apache/hudi/issues/16265
When hudi-aws-bundle.jar and hudi-spark3.3-bundle_2.12.jar are used at the same time, and hudi-aws-bundle.jar is loaded first in the Spark runtime, it can fails by NoSuchMethodError:
{noformat}
py4j.protocol.Py4JJavaError: An error occurred while calling ***.
: java.lang.NoSuchMethodError: org.apache.hudi.avro.model.HoodieCleanMetadata.getTotalFilesDeleted()I
at org.apache.hudi.client.BaseHoodieTableServiceClient.clean(BaseHoodieTableServiceClient.java:557
{noformat}
The problem is, currently hudi-aws-bundle jar in Maven central repo is built against spark2 profile and Avro 1.8.2 is used to generate source code from Avro schema file. Then, the generated source code is like
{noformat}
public Integer getTotalFilesDeleted() {
return this.totalFilesDeleted;
}
{noformat}
on the other hand, hudi-spark3.3-bundle_2.12.jar is built with Avro 1.11.1, and the generated source code is like
{noformat}
public int getTotalFilesDeleted() {
return this.totalFilesDeleted;
}
{noformat}
Since Avro 1.9.0, it uses primitive type for generated getters/setters (AVRO-2069). Therefore, if hudi-aws-bundle is loaded first in the runtime, it can fail with the above NoSuchMethodError.
Although it can be fixed by changing the classpath loading order or building hudi-aws-bundle by your own, is it possible to provide hudi-aws-spark3.3-bundle.jar in Maven central? or, is it possible to build hudi-aws-bundle jar using spark3 profile by default given most of AWS customer now use Spark 3.x for their runtime?
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6951
- Type: Improvement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Use spark3 profile to build hudi-aws-bundle jars for release artifacts [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #16265:
URL: https://github.com/apache/hudi/issues/16265#issuecomment-4795566296
Closing as outdated. The relevant code paths have evolved since this was reported (tracked in [HUDI-6951](https://issues.apache.org/jira/browse/HUDI-6951)). Please open a new issue if you can reproduce on a recent release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Use spark3 profile to build hudi-aws-bundle jars for release artifacts [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #16265: Use spark3 profile to build hudi-aws-bundle jars for release artifacts
URL: https://github.com/apache/hudi/issues/16265
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org