[jira] [Created] (SPARK-44207) Where Clause throwing Resolved attribute(s) _metadata#398 missing from ... error

["huizhong xu (Jira)" <ji...@apache.org>]

huizhong xu created SPARK-44207:
-----------------------------------

             Summary: Where Clause throwing Resolved attribute(s) _metadata#398 missing from ... error
                 Key: SPARK-44207
                 URL: https://issues.apache.org/jira/browse/SPARK-44207
             Project: Spark
          Issue Type: Question
          Components: SQL
    Affects Versions: 3.3.1
            Reporter: huizhong xu


i have 2 data frames called lt and rt, both with same schema and only 1 row, generated separately by our own curation logic, all the columns are either String, boolean or Timestamp, i am trying to compare them, and i am running a join on two like this 

var joinedDF = lt.join(rt, "Id")

after that, i am trying to compare them by schema fist and then by  each column, how many % of rows are same,

code is kindof like this

for (column <- lt.schema) {
     if (rt.columns.contains(column.name) &&
     column.dataType == rt.schema(column.name).dataType) {

      var matchCount = joinedCount
      if (column.dataType.typeName == "string") {
             matchCount = joinedDF.where((lt(column.name) <=> rt(column.name))).count}

else

.....

 

on the last line where i am running a where clause, it is throwing an error called AnalysisException Resolved attribute(s) _metadata#398 missing from ...., i don't even have this _metadata column anywhere in my dataframe at all

and i searched online people are saying it is a problem of join, i tried to change the colunm names in rt and joinedDF, both doesn't work, same error is still thrown, can anybody help here



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org