You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "rangareddy (via GitHub)" <gi...@apache.org> on 2026/07/15 08:36:13 UTC
[I] [BUG] PySpark write fails with ServiceConfigurationError: Provider org.apache.hudi.Spark32PlusDefaultSource not found in hudi-spark3.4-bundle [hudi]
rangareddy opened a new issue, #19297:
URL: https://github.com/apache/hudi/issues/19297
### Bug Description
**What happened:**
When using a locally built hudi-spark3.4-bundle (version 1.3.0-SNAPSHOT) built with the -Dspark3.4 profile, executing a DataFrame write with .format("hudi") fails immediately. The driver throws a java.util.ServiceConfigurationError stating that the provider class org.apache.hudi.Spark32PlusDefaultSource cannot be found on the classpath, despite the SPI configuration tracking it.
**What you expected:**
The write operation should succeed, resolving org.apache.hudi.Spark32PlusDefaultSource transparently via the internal DataSourceRegister mappings packaged inside the bundle.
**Steps to reproduce:**
1. Build the project: Clone the source repository and compile using Spark 3.4 and Flink 1.20 profiles (`mvn -T 2C install -DskipTests -Dspark3.4 -Dflink1.20`)
2. Launch PySpark shell: Point to the locally generated bundle jar:
`pyspark \
--jars packaging/hudi-spark-bundle/target/hudi-spark3.4-bundle_2.12-1.3.0-SNAPSHOT.jar \
--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' \
--conf 'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog' \
--conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' \
--conf 'spark.kryo.registrator=org.apache.spark.HoodieSparkKryoRegistrar'`
3. Execute a DataFrame write operation:
```python
from pyspark.sql.types import StructType, StructField, StringType, IntegerType
schema = StructType([
StructField("id", StringType(), nullable=False, metadata={"comment": "Unique identifier"}),
StructField("name", StringType(), nullable=True, metadata={"comment": "Name of the person"}),
StructField("age", IntegerType(), nullable=True, metadata={"comment": "Age of the person"}),
StructField("date", StringType(), nullable=True, metadata={"comment": "Partition field, date of entry"})
])
data = [("1", "John", 30, "2023-01-01"), ("2", "Jane", 25, "2023-01-01"), ("3", "Bob", 35, "2023-01-02")]
df = spark.createDataFrame(data, schema=schema)
hudi_options = {
'hoodie.table.name': 'Hudi_Table_With_Comments_130',
'hoodie.datasource.write.operation': 'insert',
'hoodie.datasource.write.partitionpath.field': 'date',
'hoodie.datasource.write.recordkey.field': 'id',
'hoodie.datasource.write.precombine.field': 'id',
'hoodie.schema.on.read.enable': 'true'
}
df.write.format("hudi").options(**hudi_options).mode("overwrite").save("/tmp/test/Hudi_Table_With_Comments_130")
```
### Environment
**Hudi version:** 1.3.0 (master)
**Query engine:** (Spark/Flink/Trino etc) Spark 3.4
**Relevant configs:** None
### Logs and Stack Trace
```python
>>> df.write.format("hudi").options(**hudi_options).mode("overwrite").save(tablePath)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/rangareddy/ranga_work/apache/spark/spark-3.4.1/python/pyspark/sql/readwriter.py", line 1398, in save
self._jwrite.save(path)
File "/Users/rangareddy/ranga_work/apache/spark/spark-3.4.1/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__
File "/Users/rangareddy/ranga_work/apache/spark/spark-3.4.1/python/pyspark/errors/exceptions/captured.py", line 169, in deco
return f(*a, **kw)
File "/Users/rangareddy/ranga_work/apache/spark/spark-3.4.1/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o79.save.
: java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.hudi.Spark32PlusDefaultSource not found
at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:593)
at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(ServiceLoader.java:1219)
at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1228)
at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1273)
at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1309)
at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1393)
at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at scala.collection.TraversableLike.filterImpl(TraversableLike.scala:303)
at scala.collection.TraversableLike.filterImpl$(TraversableLike.scala:297)
at scala.collection.AbstractTraversable.filterImpl(Traversable.scala:108)
at scala.collection.TraversableLike.filter(TraversableLike.scala:395)
at scala.collection.TraversableLike.filter$(TraversableLike.scala:395)
at scala.collection.AbstractTraversable.filter(Traversable.scala:108)
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:629)
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSourceV2(DataSource.scala:697)
at org.apache.spark.sql.DataFrameWriter.lookupV2Provider(DataFrameWriter.scala:860)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:256)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [BUG] PySpark write fails with ServiceConfigurationError: Provider org.apache.hudi.Spark32PlusDefaultSource not found in hudi-spark3.4-bundle [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy closed issue #19297: [BUG] PySpark write fails with ServiceConfigurationError: Provider org.apache.hudi.Spark32PlusDefaultSource not found in hudi-spark3.4-bundle
URL: https://github.com/apache/hudi/issues/19297
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [BUG] PySpark write fails with ServiceConfigurationError: Provider org.apache.hudi.Spark32PlusDefaultSource not found in hudi-spark3.4-bundle [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #19297:
URL: https://github.com/apache/hudi/issues/19297#issuecomment-5089183655
Thanks @deepakpanda93
It worked now and closing this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [BUG] PySpark write fails with ServiceConfigurationError: Provider org.apache.hudi.Spark32PlusDefaultSource not found in hudi-spark3.4-bundle [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 commented on issue #19297:
URL: https://github.com/apache/hudi/issues/19297#issuecomment-5002601815
Hi @rangareddy,
Thanks for the detailed report. I attempted to reproduce this on the master branch (Hudi 1.3.0-SNAPSHOT) using the same build profile and Spark 3.4 setup, and the DataFrame write completed successfully — the ```ServiceConfigurationError``` did not surface in my run.
```
>>> spark.read.format("hudi").load("/tmp/test/Hudi_Table_With_Comments_130").show(truncate=False)
+-------------------+---------------------+------------------+----------------------+------------------------------------------------------------------------+---+----+---+----------+
|_hoodie_commit_time|_hoodie_commit_seqno |_hoodie_record_key|_hoodie_partition_path|_hoodie_file_name |id |name|age|date |
+-------------------+---------------------+------------------+----------------------+------------------------------------------------------------------------+---+----+---+----------+
|20260717162936791 |20260717162936791_0_0|1 |2023-01-01 |e53ea9ba-c8a1-4c37-9326-99d7fd6aae40-0_0-21-58_20260717162936791.parquet|1 |John|30 |2023-01-01|
|20260717162936791 |20260717162936791_0_1|2 |2023-01-01 |e53ea9ba-c8a1-4c37-9326-99d7fd6aae40-0_0-21-58_20260717162936791.parquet|2 |Jane|25 |2023-01-01|
|20260717162936791 |20260717162936791_1_0|3 |2023-01-02 |33de3d04-68a0-4aa1-a53f-9d59cd82909a-0_1-21-59_20260717162936791.parquet|3 |Bob |35 |2023-01-02|
+-------------------+---------------------+------------------+----------------------+------------------------------------------------------------------------+---+----+---+----------+
```
The error signature ```Provider org.apache.hudi.Spark32PlusDefaultSource not found``` raised from ```ServiceLoader``` during ```DataSource.lookupDataSource``` is characteristic of a stale build artifact rather than a code-level defect.
Could you retry with a clean rebuild from scratch:
```
mvn clean
mvn -T 2C install -DskipTests -Dspark3.4 -Dflink1.20
```
I ran the exact PySpark script from the issue against a clean build on my machine and it completed the write successfully, so I suspect this will resolve after a full clean rebuild.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org