[jira] [Resolved] (SPARK-30347) LibSVMDataSource attach AttributeGroup

["zhengruifeng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-30347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

zhengruifeng resolved SPARK-30347.
----------------------------------
    Fix Version/s: 3.0.0
       Resolution: Fixed

Issue resolved by pull request 27003
[https://github.com/apache/spark/pull/27003]

> LibSVMDataSource attach AttributeGroup
> --------------------------------------
>
>                 Key: SPARK-30347
>                 URL: https://issues.apache.org/jira/browse/SPARK-30347
>             Project: Spark
>          Issue Type: Improvement
>          Components: ML
>    Affects Versions: 3.0.0
>            Reporter: zhengruifeng
>            Assignee: zhengruifeng
>            Priority: Minor
>             Fix For: 3.0.0
>
>
> LibSVMDataSource will attach a special metadata to indicate numFeatures.
> {code:java}
>  scala> val data = spark.read.format("libsvm").load("/data0/Dev/Opensource/spark/data/mllib/sample_multiclass_classification_data.txt")
> 19/12/24 18:40:09 WARN LibSVMFileFormat: 'numFeatures' option not specified, determining the number of features by going though the input. If you know the number in advance, please specify it via 'numFeatures' option to avoid the extra scan.
> data: org.apache.spark.sql.DataFrame = [label: double, features: vector]scala> data.schema("features").metadata
> res0: org.apache.spark.sql.types.Metadata = {"numFeatures":4}
> {code}
> However, ML impls all try to obtain the vector size via \{{AttributeGroup}}, which can not use this metadata:
> {code:java}
> scala> import org.apache.spark.ml.attribute._
> import org.apache.spark.ml.attribute._scala> AttributeGroup.fromStructField(data.schema("features")).size
> res1: Int = -1
>  {code}
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org