[jira] [Created] (SPARK-21853) Getting an exception while calling the except method on the dataframe

["Shailesh Kini (JIRA)" <ji...@apache.org>]

Shailesh Kini created SPARK-21853:
-------------------------------------

             Summary: Getting an exception while calling the except method on the dataframe
                 Key: SPARK-21853
                 URL: https://issues.apache.org/jira/browse/SPARK-21853
             Project: Spark
          Issue Type: Question
          Components: Spark Shell
    Affects Versions: 2.1.1
            Reporter: Shailesh Kini


I am getting an exception while calling except on the Dataset.

org.apache.spark.sql.AnalysisException: resolved attribute(s) SVC_BILLING_PERIOD#37723 missing from

I have 2 csv. I create two dataset DS1 and DS2, which I join to create DS3. I need to filter out duplicates for further processing. I aggregate on DS3 dataset by on some columns and filter when the count > 1. This is DS4. I now join DS3 with DS4 on those columns and get DS5. DS5 has the same structure as DS3 as I drop the columns from the join. DS5 now has all the rows which are duplicate. I then call the except on DS3 to get me a dataset DS5 which all the rows not in DS5. I am planning to filter and remove of of the duplicates (all the columns are not duplicates so I need to use filter) and union it with DS6 to get the dataset free of duplicates.

Attaching the exception to this ticket.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org