You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Xusen Yin (JIRA)" <ji...@apache.org> on 2015/10/23 09:39:27 UTC

[jira] [Commented] (SPARK-11234) What's cooking classification

    [ https://issues.apache.org/jira/browse/SPARK-11234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970596#comment-14970596 ] 

Xusen Yin commented on SPARK-11234:
-----------------------------------

[~mengxr] I add the cooking classification code here: https://gist.github.com/yinxusen/ad4372b8c0af5ae54a4a

Here are what I find:

1. Currently, multi-line per record JSON file is hard to handle, I have to load the data with JsonInputFormat in the json-pxf-ext package.

2. String indexer is easy to use. But it is hard to do beyond existing transformers. Like in the code, when I want to add all vectors that belong to the same id together, I have to write an aggregate function.

3. ParamGridBuilder accepts discrete parameter candidates, but I need to add some parameters with guess like Array(1.0, 0.1, 0.01). I don't know which parameter is suitable and how to fill in the array will get a better result. How about giving a range of real numbers so that the ParamGridBuilder can generate candidates for me like [0.0001, 1]?

4. The evaluator forces me to select a metric method. But sometimes I want to see all the evaluation results, say F1, precision-recall, AUC, etc. 

5. ML transformers will get stuck when facing with Int type. It's strange that we have to transform all Int values to double values before hand. I think a wise auto casting is helpful.

> What's cooking classification
> -----------------------------
>
>                 Key: SPARK-11234
>                 URL: https://issues.apache.org/jira/browse/SPARK-11234
>             Project: Spark
>          Issue Type: Sub-task
>          Components: ML
>            Reporter: Xusen Yin
>
> I add the subtask to post the work on this dataset:  https://www.kaggle.com/c/whats-cooking



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org