You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jean Bon (Jira)" <ji...@apache.org> on 2022/01/18 16:00:00 UTC
[jira] [Created] (SPARK-37954) old columns should not be available after select or drop
Jean Bon created SPARK-37954:
--------------------------------
Summary: old columns should not be available after select or drop
Key: SPARK-37954
URL: https://issues.apache.org/jira/browse/SPARK-37954
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 3.0.1
Reporter: Jean Bon
{code:java}
from pyspark.sql import SparkSession
from pyspark.sql.functions import col as col
spark = SparkSession.builder.appName('available_columns').getOrCreate()
df = spark.range(5).select((col("id")+10).alias("id2"))
assert df.columns==["id2"] #OK
try:
df.select("id")
error_raise = False
except:
error_raise = True
assert error_raise #OK
df = df.drop("id") #should raise an error
df.filter(col("id")!=2).count() #returns 4, should raise an error
{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org