You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Xi Wang (JIRA)" <ji...@apache.org> on 2017/07/11 14:09:00 UTC
[jira] [Closed] (SPARK-20681) DataFram.Drop doesn't take effective,
neither does error
[ https://issues.apache.org/jira/browse/SPARK-20681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xi Wang closed SPARK-20681.
---------------------------
Resolution: Fixed
> DataFram.Drop doesn't take effective, neither does error
> --------------------------------------------------------
>
> Key: SPARK-20681
> URL: https://issues.apache.org/jira/browse/SPARK-20681
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.1.0
> Reporter: Xi Wang
> Priority: Critical
>
> I am running the following code trying to drop nested columns, but it doesn't work, it doesn't return error either.
> *I read the DF from this json:*
> {'parent':{'child':{'grandchild':{'val':'1',' val_to_be_deleted':'0'}}}}
> scala> spark.read.format("json").load("c:/tmp/spark_issue.json")
> res0: org.apache.spark.sql.DataFrame = [father: struct<child: struct<grandchild: struct<val: bigint, val_to_be_deleted: bigint>>>]
> *read the df:*
> scala> res0.printSchema
> root
> |-- parent: struct (nullable = true)
> | |-- child: struct (nullable = true)
> | | |-- grandchild: struct (nullable = true)
> | | | |-- val: long (nullable = true)
> | | | |-- val_to_be_deleted: long (nullable = true)
> *drop the column (I tried different ways, "quote", `back-tick`, col(object) ...) column remains anyway:*
> scala> res0.drop(col("father.child.grandchild.val_to_be_deleted")).printSchema
> root
> |-- father: struct (nullable = true)
> | |-- child: struct (nullable = true)
> | | |-- grandchild: struct (nullable = true)
> | | | |-- val: long (nullable = true)
> | | | |-- val_to_be_deleted: long (nullable = true)
> scala> res0.drop("father.child.grandchild.val_to_be_deleted").printSchema
> root
> |-- father: struct (nullable = true)
> | |-- child: struct (nullable = true)
> | | |-- grandchild: struct (nullable = true)
> | | | |-- val: long (nullable = true)
> | | | |-- val_to_be_deleted: long (nullable = true)
> Any help is appreciated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org