You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Reynold Xin (JIRA)" <ji...@apache.org> on 2015/06/01 08:46:17 UTC
[jira] [Updated] (SPARK-7998) A better frequent item API
[ https://issues.apache.org/jira/browse/SPARK-7998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Reynold Xin updated SPARK-7998:
-------------------------------
Description:
The current freqItems API is really awkward to use. It returns a DataFrame with a single row, in which each value is an array of frequent items.
This design doesn't work well for exploratory data analysis (running show -- when there are more than 2 or 3 frequent values, the values get cut off):
{code}
In [74]: df.stat.freqItems(["a", "b", "c"], 0.4).show()
+------------------+------------------+-----------------+
| a_freqItems| b_freqItems| c_freqItems|
+------------------+------------------+-----------------+
|ArrayBuffer(11, 1)|ArrayBuffer(2, 22)|ArrayBuffer(1, 3)|
+------------------+------------------+-----------------+
{code}
It also doesn't work well for serious engineering, since it is hard to get the value out.
We should create a new function (so we maintain source/binary compatibility) that returns a list of list of values.
was:
The current freqItems API is really awkward to use. It returns a DataFrame with a single row, in which each value is an array of frequent items.
This design doesn't work well for exploratory data analysis (running show -- when there are more than 2 or 3 frequent values, the values get cut off):
{code}
In [74]: df.stat.freqItems(["a", "b", "c"], 0.4).show()
+------------------+------------------+-----------------+
| a_freqItems| b_freqItems| c_freqItems|
+------------------+------------------+-----------------+
|ArrayBuffer(11, 1)|ArrayBuffer(2, 22)|ArrayBuffer(1, 3)|
+------------------+------------------+-----------------+
{code}
It also doesn't work well for serious engineering, since it is hard to get the value out.
We should just create a new function (so we maintain source/binary compatibility) that returns a list of list of values.
> A better frequent item API
> --------------------------
>
> Key: SPARK-7998
> URL: https://issues.apache.org/jira/browse/SPARK-7998
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Reporter: Reynold Xin
>
> The current freqItems API is really awkward to use. It returns a DataFrame with a single row, in which each value is an array of frequent items.
> This design doesn't work well for exploratory data analysis (running show -- when there are more than 2 or 3 frequent values, the values get cut off):
> {code}
> In [74]: df.stat.freqItems(["a", "b", "c"], 0.4).show()
> +------------------+------------------+-----------------+
> | a_freqItems| b_freqItems| c_freqItems|
> +------------------+------------------+-----------------+
> |ArrayBuffer(11, 1)|ArrayBuffer(2, 22)|ArrayBuffer(1, 3)|
> +------------------+------------------+-----------------+
> {code}
> It also doesn't work well for serious engineering, since it is hard to get the value out.
> We should create a new function (so we maintain source/binary compatibility) that returns a list of list of values.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org