You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@datafu.apache.org by "Eyal Allweil (Jira)" <ji...@apache.org> on 2020/10/08 18:59:00 UTC
[jira] [Updated] (DATAFU-154) Spark Explode Array method
[ https://issues.apache.org/jira/browse/DATAFU-154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eyal Allweil updated DATAFU-154:
--------------------------------
Description:
Spark has an [explode|https://spark.apache.org/docs/latest/api/sql/index.html#explode] method that divides an array into multiple rows. It is sometimes convenient to divide an array into multiple columns.
h5. For example, for input:
|label|sentence_arr |
|0.0 |[Hi, I heard, about, Spark] |
|0.0 |[I wish, Java, could use, case classes] |
|1.0 |[Logistic, regression, models, are neat]|
h5. the output could be
|label|sentence_arr |token0 |token1 |token2 |token3 |
|0.0 |[Hi, I heard, about, Spark] |Hi |I heard |about |Spark |
|0.0 |[I wish, Java, could use, case classes] |I wish |Java |could use|case classes|
|1.0 |[Logistic, regression, models, are neat]|Logistic|regression|models |are neat |
was:
Spark has an [explode|https://spark.apache.org/docs/latest/api/sql/index.html#explode] method that divides an array into multiple rows. It is sometimes convenient to divide an array into multiple columns.
For example, for input
+-----+----------------------------------------+
|label|sentence_arr |
+-----+----------------------------------------+
|0.0 |[Hi, I heard, about, Spark] |
|0.0 |[I wish, Java, could use, case classes] |
|1.0 |[Logistic, regression, models, are neat]|
+-----+----------------------------------------+
the output could be
+-----+----------------------------------------+--------+----------+---------+------------+
|label|sentence_arr |token0 |token1 |token2 |token3 |
+-----+----------------------------------------+--------+----------+---------+------------+
|0.0 |[Hi, I heard, about, Spark] |Hi |I heard |about |Spark |
|0.0 |[I wish, Java, could use, case classes] |I wish |Java |could use|case classes|
|1.0 |[Logistic, regression, models, are neat]|Logistic|regression|models |are neat |
+-----+----------------------------------------+--------+----------+---------+------------+
> Spark Explode Array method
> --------------------------
>
> Key: DATAFU-154
> URL: https://issues.apache.org/jira/browse/DATAFU-154
> Project: DataFu
> Issue Type: New Feature
> Environment: {noformat}
> *no* further _formatting_ is done here{noformat}
> {code:java}
>
> {code}
> {code:java}
>
> {code}
> Reporter: Eyal Allweil
> Priority: Major
>
> Spark has an [explode|https://spark.apache.org/docs/latest/api/sql/index.html#explode] method that divides an array into multiple rows. It is sometimes convenient to divide an array into multiple columns.
> h5. For example, for input:
> |label|sentence_arr |
> |0.0 |[Hi, I heard, about, Spark] |
> |0.0 |[I wish, Java, could use, case classes] |
> |1.0 |[Logistic, regression, models, are neat]|
> h5. the output could be
> |label|sentence_arr |token0 |token1 |token2 |token3 |
> |0.0 |[Hi, I heard, about, Spark] |Hi |I heard |about |Spark |
> |0.0 |[I wish, Java, could use, case classes] |I wish |Java |could use|case classes|
> |1.0 |[Logistic, regression, models, are neat]|Logistic|regression|models |are neat |
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)