You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "gaokui (Jira)" <ji...@apache.org> on 2022/04/07 01:44:00 UTC
[jira] [Updated] (SPARK-38812) rdd spil two rdd
[ https://issues.apache.org/jira/browse/SPARK-38812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
gaokui updated SPARK-38812:
---------------------------
Description:
when id do clean data, i hope rdd according one value(>or <) filter data, and then to generate two different set,one is error data file, another is right data file.
Now I use filter, but this filter must have two spark dag job, that cost too much.
exactly some code like iterator.span(preidicate) and then return one tuple(iter1,iter2)
Summary: rdd spil two rdd (was: rdd spilt,when id do clean data, i hope rdd according one value(>or <) filter data to two different set,one write error file, another is)
> rdd spil two rdd
> ----------------
>
> Key: SPARK-38812
> URL: https://issues.apache.org/jira/browse/SPARK-38812
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Affects Versions: 3.2.1
> Reporter: gaokui
> Priority: Major
>
> when id do clean data, i hope rdd according one value(>or <) filter data, and then to generate two different set,one is error data file, another is right data file.
> Now I use filter, but this filter must have two spark dag job, that cost too much.
> exactly some code like iterator.span(preidicate) and then return one tuple(iter1,iter2)
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org