You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Amandeep Sharma <ha...@gmail.com> on 2021/02/09 09:37:06 UTC

dataFrame.na.fill() fails for column with dot

Hi guys,
Apologies for the long mail.

I am running below code snippet

import org.apache.spark.sql.SparkSession
object ColumnNameWithDot {
 def main(args: Array[String]): Unit = {

 val spark = SparkSession.builder.appName("Simple Application")
 .config("spark.master", "local").getOrCreate()

 spark.sparkContext.setLogLevel("OFF")

 import spark.implicits._
 val df = Seq(("abc", 23), ("def", 44), (null, 9)).toDF("ColWith.Dot",
"Col")
 df.na.fill(Map("`ColWith.Dot`" -> "n/a")).show()

 }
}

and it is failing with error
Exception in thread "main" org.apache.spark.sql.AnalysisException: Cannot
resolve column name "ColWith.Dot" among (ColWith.Dot, Col);

I checked that code-fix were made for the similar issue, found
https://issues.apache.org/jira/browse/SPARK-19473; but none of them fixed
all cases.

I debugged the code below are the observations

   1. In org.apache.spark.sql.DataFrameNaFunctions.fillMap(values:
   Seq[(String, Any)]) the df.resolve(colName) call succeeds, since column
   name is quoted with back tick it resolves the column
   2. val projections = df.schema.fields.map {
       ...
       ...
   }.getOrElse(df.col(f.name)) fails since resolved column name is not
   quoted with backtick

Problem lies in the
org.apache.spark.sql.catalyst.expressions
resolve(nameParts: Seq[String], resolver: Resolver): Option[NamedExpression]

where the comment says we try to resolve it as a column.

// If none of attributes match database.table.column pattern or
// `table.column` pattern, we try to resolve it as a column.
val (candidates, nestedFields) = matches match {
    case (Seq(), _) =>
        val name = nameParts.head
        val attributes = collectMatches(name,
direct.get(name.toLowerCase(Locale.ROOT)))
        (attributes, nameParts.tail)
    case _ => matches
}

should be changed to

// If none of attributes match database.table.column pattern or
// `table.column` pattern, we try to resolve it as a column.
val (candidates, nestedFields) = matches match {
    case (Seq(), _) =>
        val name = nameParts.mkString(".")
        val attributes = collectMatches(name,
direct.get(name.toLowerCase(Locale.ROOT)))
        (attributes, Seq.empty)
    case _ => matches
}
git diff is as below

-          val name = nameParts.head
+          val name = nameParts.mkString(".")
           val attributes = collectMatches(name,
direct.get(name.toLowerCase(Locale.ROOT)))
-          (attributes, nameParts.tail)
+          (attributes, Seq.empty)

I tested this change, there is no longer need to use backtick with columns
having dot in the name.
Can this change be merged?

Regards,
Amandeep

Re: dataFrame.na.fill() fails for column with dot

Posted by Terry Kim <yu...@gmail.com>.
You probably need to update f. name here
<https://github.com/apache/spark/blob/18b30107adb37d3c7a767a20cc02813f0fdb86da/sql/core/src/main/scala/org/apache/spark/sql/DataFrameNaFunctions.scala#L413>
as
well, but we can discuss further when you create a JIRA/PR.

Thanks,
Terry

On Tue, Feb 9, 2021 at 9:53 AM Terry Kim <yu...@gmail.com> wrote:

> Thanks Amandeep. This seems like a valid bug to me as quoted columns are
> not handled property for na.fill(). I think the better place to fix is in
> DataFrameNaFunctions.scala
> <https://github.com/apache/spark/blob/18b30107adb37d3c7a767a20cc02813f0fdb86da/sql/core/src/main/scala/org/apache/spark/sql/DataFrameNaFunctions.scala#L422> where
> "f.name" should be quoted.
>
> Could you create a JIRA
> <https://issues.apache.org/jira/projects/SPARK/issues>?
>
> Thanks,
> Terry
>

Re: dataFrame.na.fill() fails for column with dot

Posted by Terry Kim <yu...@gmail.com>.
Thanks Amandeep. This seems like a valid bug to me as quoted columns are
not handled property for na.fill(). I think the better place to fix is in
DataFrameNaFunctions.scala
<https://github.com/apache/spark/blob/18b30107adb37d3c7a767a20cc02813f0fdb86da/sql/core/src/main/scala/org/apache/spark/sql/DataFrameNaFunctions.scala#L422>
where
"f.name" should be quoted.

Could you create a JIRA
<https://issues.apache.org/jira/projects/SPARK/issues>?

Thanks,
Terry

Re: dataFrame.na.fill() fails for column with dot

Posted by Amandeep Sharma <ha...@gmail.com>.
Adding note to the previous email.
Test suite org.apache.spark.sql.catalyst.analysis.AnalysisSuite passed
after the aforementioned changes.

Regards,
Amandeep Sharma


On Tue, Feb 9, 2021 at 3:07 PM Amandeep Sharma <ha...@gmail.com>
wrote:

> Hi guys,
> Apologies for the long mail.
>
> I am running below code snippet
>
> import org.apache.spark.sql.SparkSession
> object ColumnNameWithDot {
>  def main(args: Array[String]): Unit = {
>
>  val spark = SparkSession.builder.appName("Simple Application")
>  .config("spark.master", "local").getOrCreate()
>
>  spark.sparkContext.setLogLevel("OFF")
>
>  import spark.implicits._
>  val df = Seq(("abc", 23), ("def", 44), (null, 9)).toDF("ColWith.Dot",
> "Col")
>  df.na.fill(Map("`ColWith.Dot`" -> "n/a")).show()
>
>  }
> }
>
> and it is failing with error
> Exception in thread "main" org.apache.spark.sql.AnalysisException: Cannot
> resolve column name "ColWith.Dot" among (ColWith.Dot, Col);
>
> I checked that code-fix were made for the similar issue, found
> https://issues.apache.org/jira/browse/SPARK-19473; but none of them fixed
> all cases.
>
> I debugged the code below are the observations
>
>    1. In org.apache.spark.sql.DataFrameNaFunctions.fillMap(values:
>    Seq[(String, Any)]) the df.resolve(colName) call succeeds, since
>    column name is quoted with back tick it resolves the column
>    2. val projections = df.schema.fields.map {
>        ...
>        ...
>    }.getOrElse(df.col(f.name)) fails since resolved column name is not
>    quoted with backtick
>
> Problem lies in the
> org.apache.spark.sql.catalyst.expressions
> resolve(nameParts: Seq[String], resolver: Resolver):
> Option[NamedExpression]
>
> where the comment says we try to resolve it as a column.
>
> // If none of attributes match database.table.column pattern or
> // `table.column` pattern, we try to resolve it as a column.
> val (candidates, nestedFields) = matches match {
>     case (Seq(), _) =>
>         val name = nameParts.head
>         val attributes = collectMatches(name,
> direct.get(name.toLowerCase(Locale.ROOT)))
>         (attributes, nameParts.tail)
>     case _ => matches
> }
>
> should be changed to
>
> // If none of attributes match database.table.column pattern or
> // `table.column` pattern, we try to resolve it as a column.
> val (candidates, nestedFields) = matches match {
>     case (Seq(), _) =>
>         val name = nameParts.mkString(".")
>         val attributes = collectMatches(name,
> direct.get(name.toLowerCase(Locale.ROOT)))
>         (attributes, Seq.empty)
>     case _ => matches
> }
> git diff is as below
>
> -          val name = nameParts.head
> +          val name = nameParts.mkString(".")
>            val attributes = collectMatches(name,
> direct.get(name.toLowerCase(Locale.ROOT)))
> -          (attributes, nameParts.tail)
> +          (attributes, Seq.empty)
>
> I tested this change, there is no longer need to use backtick with columns
> having dot in the name.
> Can this change be merged?
>
> Regards,
> Amandeep
>