PySpark tests are failed with the java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.sources.FakeSourceOne not found

[Ranga Reddy <ra...@gmail.com>]

Hi Team,

I am running the pyspark tests in Spark version and it failed with P*rovider
org.apache.spark.sql.sources.FakeSourceOne not found.*

Spark Version: 3.4.0/3.5.0
Python Version: 3.8.10
OS: Ubuntu 20.04


*Steps: *

# /opt/data/spark/build/sbt -Phive clean package
# /opt/data/spark/build/sbt test:compile
# pip3 install -r /opt/data/spark/dev/requirements.txt
# /opt/data/spark/python/run-tests --python-executables=python3

*Exception:*

======================================================================
ERROR [15.081s]: test_read_images
(pyspark.ml.tests.test_image.ImageFileFormatTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/opt/data/spark/python/pyspark/ml/tests/test_image.py", line 29, in
test_read_images
self.spark.read.format("image")
File "/opt/data/spark/python/pyspark/sql/readwriter.py", line 300, in load
return self._df(self._jreader.load(path))
File "/opt/data/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py"
, line 1322, in __call__
return_value = get_return_value(
File "/opt/data/spark/python/pyspark/errors/exceptions/captured.py", line
176, in deco
return f(*a, **kw)
File "/opt/data/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py",
line 326, in get_return_value
raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o33.load.
: java.util.ServiceConfigurationError:
org.apache.spark.sql.sources.DataSourceRegister: Provider
org.apache.spark.sql.sources.FakeSourceOne not found
at java.util.ServiceLoader.fail(ServiceLoader.java:239)
at java.util.ServiceLoader.access$300(ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:372)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at scala.collection.convert.Wrappers$JIteratorWrapper
.next(Wrappers.scala:46)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at scala.collection.TraversableLike.filterImpl(TraversableLike.scala:303)
at scala.collection.TraversableLike.filterImpl$(TraversableLike.scala:297)
at scala.collection.AbstractTraversable.filterImpl(Traversable.scala:108)
at scala.collection.TraversableLike.filter(TraversableLike.scala:395)
at scala.collection.TraversableLike.filter$(TraversableLike.scala:395)
at scala.collection.AbstractTraversable.filter(Traversable.scala:108)
at org.apache.spark.sql.execution.datasources.DataSource$
.lookupDataSource(DataSource.scala:629)
at org.apache.spark.sql.execution.datasources.DataSource$
.lookupDataSourceV2(DataSource.scala:697)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:208)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:186)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at
py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.lang.Thread.run(Thread.java:750)


Could someone help me how to proceed further?


-- 
Thanks and Regards


*Ranga Reddy*
*------------------------------*

*Bangalore, Karnataka, India*
*Mobile : +91-9986183183 |  Email: rangareddy.avula@gmail.com
<ra...@gmail.com>*