You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Enrico Minack <en...@gmx.de.INVALID> on 2024/10/30 11:44:59 UTC
ShuffleDataIO: where is the reading part of the API?
Hi devs,
the docs of org.apache.spark.shuffle.api.ShuffleDataIO read:
An interface for plugging in modules for storing and reading
temporary shuffle data.
but the API does only provide interface for writing shuffle data:
- ShuffleExecutorComponents.createMapOutputWriter
- ShuffleExecutorComponents.createSingleFileMapOutputWriter
Where is the reading part of ShuffleDataIO?
How can I plug in implementations for reading shuffle data via
ShuffleDataIO?
I am aware of org.apache.spark.shuffle.ShuffleManager.
Cheers,
Enrico
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: ShuffleDataIO: where is the reading part of the API?
Posted by Mich Talebzadeh <mi...@gmail.com>.
If I recall correctly, we discussed some of these in the Apache Spark
community slack a couple of years ago.
The *ShuffleDataIO* interface is concerned with writing shuffle data. It
provides mechanisms for writing map output data to disk, which is then used
in the shuffle phase of a Spark job.
Reading shuffle data is primarily handled by the *ShuffleManager*
interface. Different shuffle implementations (like SortShuffleManager and
HashShuffleManager) are responsible for reading the written shuffle data
from disk.I
Implementing a custom ShuffleManager is a complex task, as it requires
familiarity with internal shuffle mechanics, data block management, and
potentially modifying how data is stored and retrieved on disk.You can
customize the reading process. However it is a complex task and requires an
expert understanding of Spark internals
HTH,
Mich Talebzadeh,
Architect | Data Engineer | Data Science | Financial Crime
PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College
London <https://en.wikipedia.org/wiki/Imperial_College_London>
London, United Kingdom
*Disclaimer:* The information provided is correct to the best of my
knowledge but of course cannot be guaranteed . It is essential to note
that, as with any advice, quote "one test result is worth one-thousand
expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von
Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)".
On Wed, 30 Oct 2024 at 11:46, Enrico Minack <en...@gmx.de.invalid>
wrote:
> Hi devs,
>
> the docs of org.apache.spark.shuffle.api.ShuffleDataIO read:
>
> An interface for plugging in modules for storing and reading
> temporary shuffle data.
>
> but the API does only provide interface for writing shuffle data:
> - ShuffleExecutorComponents.createMapOutputWriter
> - ShuffleExecutorComponents.createSingleFileMapOutputWriter
>
> Where is the reading part of ShuffleDataIO?
> How can I plug in implementations for reading shuffle data via
> ShuffleDataIO?
>
> I am aware of org.apache.spark.shuffle.ShuffleManager.
>
> Cheers,
> Enrico
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>