[jira] [Created] (SPARK-32562) Pyspark drop duplicate columns

["abhijeet dada mote (Jira)" <ji...@apache.org>]

abhijeet dada mote created SPARK-32562:
------------------------------------------

             Summary: Pyspark drop duplicate columns
                 Key: SPARK-32562
                 URL: https://issues.apache.org/jira/browse/SPARK-32562
             Project: Spark
          Issue Type: Improvement
          Components: PySpark
    Affects Versions: 3.0.0
            Reporter: abhijeet dada mote
             Fix For: 3.0.0


Hi All,

This is one suggestion can we have a feature in pyspark to remove duplicate columns? 

I have come up with small code for that 

<code>

def drop_duplicate_columns(_rdd_df):
    column_names = _rdd_df.columns
    duplicate_columns = set([x for x in column_names if column_names.count(x) > 1])
    _rdd_df = _rdd_df.drop(*duplicate_columns)
    return _rdd_df

<code>

Your suggestions are appreciatd and can work on this PR, this would be my first contribution(PR) to Pyspark if you guys agree with it



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org