[jira] [Created] (SPARK-30347) LibSVMDataSource attach AttributeGroup

["zhengruifeng (Jira)" <ji...@apache.org>]

zhengruifeng created SPARK-30347:
------------------------------------

             Summary: LibSVMDataSource attach AttributeGroup
                 Key: SPARK-30347
                 URL: https://issues.apache.org/jira/browse/SPARK-30347
             Project: Spark
          Issue Type: Improvement
          Components: ML
    Affects Versions: 3.0.0
            Reporter: zhengruifeng


LibSVMDataSource will attach a special metadata to indicate numFeatures.
{code:java}
 scala> val data = spark.read.format("libsvm").load("/data0/Dev/Opensource/spark/data/mllib/sample_multiclass_classification_data.txt")
19/12/24 18:40:09 WARN LibSVMFileFormat: 'numFeatures' option not specified, determining the number of features by going though the input. If you know the number in advance, please specify it via 'numFeatures' option to avoid the extra scan.
data: org.apache.spark.sql.DataFrame = [label: double, features: vector]scala> data.schema("features").metadata
res0: org.apache.spark.sql.types.Metadata = {"numFeatures":4}
{code}
However, ML impls all try to obtain the vector size via \{{AttributeGroup}}, which can not use this metadata:
{code:java}
scala> import org.apache.spark.ml.attribute._
import org.apache.spark.ml.attribute._scala> AttributeGroup.fromStructField(data.schema("features")).size
res1: Int = -1
 {code}
 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org