You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Shailesh Kini (JIRA)" <ji...@apache.org> on 2017/08/29 03:42:00 UTC
[jira] [Commented] (SPARK-21853) Getting an exception while calling
the except method on the dataframe
[ https://issues.apache.org/jira/browse/SPARK-21853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16144697#comment-16144697 ]
Shailesh Kini commented on SPARK-21853:
---------------------------------------
As a work around, I saved the dataset DS3 into parquet format and read it again after which I was able to successfully call the except.
> Getting an exception while calling the except method on the dataframe
> ---------------------------------------------------------------------
>
> Key: SPARK-21853
> URL: https://issues.apache.org/jira/browse/SPARK-21853
> Project: Spark
> Issue Type: Question
> Components: Spark Shell
> Affects Versions: 2.1.1
> Reporter: Shailesh Kini
> Attachments: SparkException.txt
>
>
> I am getting an exception while calling except on the Dataset.
> org.apache.spark.sql.AnalysisException: resolved attribute(s) SVC_BILLING_PERIOD#37723 missing from
> I read 2 csv files into datasets DS1 and DS2, which I join (full outer) to create DS3. DS3 has some rows which are similar with the exception of one column. I need to isolate those rows and remove the similar rows. I use groupBy with the count > 1 on a few columns in DS3 to get those similar rows - dataset DS4. DS4 has only a few columns and not all so I join it back with DS3 on the aggregate columns to get a new dataset DS5 which has the same columns as DS3. To get a clean dataset without any of those similar rows, I am calling DS3.except(DS5) which throws the exception. The attribute is one of the filtering criteria I use which creating DS1.
> Attaching the exception to this ticket.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org