You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Peiyu Zhuang (JIRA)" <ji...@apache.org> on 2018/12/17 08:27:00 UTC
[jira] [Updated] (SPARK-26380) ExternalShuffleBlockResolver should
not block shuffle managers by name.
[ https://issues.apache.org/jira/browse/SPARK-26380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peiyu Zhuang updated SPARK-26380:
---------------------------------
Description:
We configure a shuffle manager that is compatible with the original Spark shuffle manager (but with other network/storage implementation) in our environment. It doesn't require any modification to Spark source code and It works fine until we enable dynamic allocation. The problem is that in {{ExternalShuffleBlockResolver}} class it checks the name of the shuffle manager class and raise an {{UnsupportedOperationException}} if shuffle manager is not {{SortShuffleManager}} or {{UnsafeShuffleManager}}.
Since shuffle manager is totally configurable, I think user should be able to decide which shuffle manager to use when dynamic allocation is enabled.
Maybe we could change {{UnsupportedOperationException}} to a warning log telling user that the shuffle manager they are using may not be compatible with dynamic allocation and give them detail reasons why. On the other hand, user may still choose to use their own shuffle manager while knowing the risk behind it.
was:
We configure a shuffle manager that is compatible with the original Spark shuffle manager (but with other network/storage implementation) in our environment. It doesn't require any modification to Spark source code and It works fine until we enable dynamic allocation. The problem is that in {{ExternalShuffleBlockResolver}} class it checks the name of the shuffle manager class and raise an {{UnsupportedOperationException}} if shuffle manager is not {{SortShuffleManager}} or {{UnsafeShuffleManager}}.
Since shuffle manager is totally configurable, I think user should be able to decide which shuffle manager to use when dynamic allocation is enabled.
Maybe we could change {{UnsupportedOperationException}} to a warning log telling user that the shuffle manager they are using may not be compatible with dynamic allocation and gives them detail reasons why. On the other hand, user may still choose to use their own shuffle manager while knowing the risk behind it.
> ExternalShuffleBlockResolver should not block shuffle managers by name.
> -----------------------------------------------------------------------
>
> Key: SPARK-26380
> URL: https://issues.apache.org/jira/browse/SPARK-26380
> Project: Spark
> Issue Type: Bug
> Components: Shuffle
> Affects Versions: 2.3.2
> Reporter: Peiyu Zhuang
> Priority: Major
>
> We configure a shuffle manager that is compatible with the original Spark shuffle manager (but with other network/storage implementation) in our environment. It doesn't require any modification to Spark source code and It works fine until we enable dynamic allocation. The problem is that in {{ExternalShuffleBlockResolver}} class it checks the name of the shuffle manager class and raise an {{UnsupportedOperationException}} if shuffle manager is not {{SortShuffleManager}} or {{UnsafeShuffleManager}}.
> Since shuffle manager is totally configurable, I think user should be able to decide which shuffle manager to use when dynamic allocation is enabled.
> Maybe we could change {{UnsupportedOperationException}} to a warning log telling user that the shuffle manager they are using may not be compatible with dynamic allocation and give them detail reasons why. On the other hand, user may still choose to use their own shuffle manager while knowing the risk behind it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org