[jira] [Created] (SPARK-49612) Feature transformers are case sensitive

["Chhavi Bansal (Jira)" <ji...@apache.org>]

Chhavi Bansal created SPARK-49612:
-------------------------------------

             Summary: Feature transformers are case sensitive
                 Key: SPARK-49612
                 URL: https://issues.apache.org/jira/browse/SPARK-49612
             Project: Spark
          Issue Type: Bug
          Components: ML, MLlib, Spark Core
    Affects Versions: 3.5.2
            Reporter: Chhavi Bansal


Hi team,

[https://spark.apache.org/docs/latest/ml-features]

The feature transformers are case sensitive even though the configuration 
{code:java}
spark.conf.get("spark.sql.caseSensitive") {code}
is set to false. The user of all these transformers are forced to abide by case of the column in the dataframe
{code:java}
  val data = List(Row("the movie was great", "positive", 10, "greatest of all time"),
    Row("the movie was average", "negative", 11, "just average things, average storyline"),
    Row("movie was fun", "positive", 2, "superb screen play"))
  val schema = new StructType()
    .add("comments", StringType, true)
    .add("reviews", StringType, true)
    .add("counts", IntegerType, true)
    .add("Additional_COMMENTS", StringType, true)
val df = spark.createDataFrame(data.asJava, schema)
  val si = new StringIndexer().setInputCol("additional_comments").setOutputCol("si_additional_comments")
  si.fit(df).transform(df).show() {code}
The above code fails with 
{code:java}
Exception in thread "main" org.apache.spark.SparkException: Input column additional_comments does not exist.
    at org.apache.spark.ml.feature.StringIndexerBase.$anonfun$validateAndTransformSchema$2(StringIndexer.scala:128)
    at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
    at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
    at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
    at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
    at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)
    at scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:198)
    at org.apache.spark.ml.feature.StringIndexerBase.validateAndTransformSchema(StringIndexer.scala:123)
    at org.apache.spark.ml.feature.StringIndexerBase.validateAndTransformSchema$(StringIndexer.scala:115) {code}
Which means that the column "additional_comments" needs to be provided in the same case as in the dataframe. 



I think when the caseSensitive  setting is set to *false* we should be able to use the naming in any case.

 

Can someone please help to solve this bug for all transformers.?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org