[jira] [Created] (SPARK-10073) Python withColumn for existing column name not consistent with scala

["Michael Armbrust (JIRA)" <ji...@apache.org>]

Michael Armbrust created SPARK-10073:
----------------------------------------

             Summary: Python withColumn for existing column name not consistent with scala
                 Key: SPARK-10073
                 URL: https://issues.apache.org/jira/browse/SPARK-10073
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 1.5.0
            Reporter: Michael Armbrust
            Assignee: Davies Liu
            Priority: Blocker


The same code as below works in Scala (replacing the old column with the new one).

{code}
from pyspark.sql import Row
df = sc.parallelize([Row(a=1)]).toDF()
df.withColumn("a", df.a).select("a")
---------------------------------------------------------------------------
AnalysisException                         Traceback (most recent call last)
<ipython-input-4-d5a4f4132506> in <module>()
      1 from pyspark.sql import Row
      2 df = sc.parallelize([Row(a=1)]).toDF()
----> 3 df.withColumn("a", df.a).select("a")

/home/ubuntu/databricks/spark/python/pyspark/sql/dataframe.py in select(self, *cols)
    764         [Row(name=u'Alice', age=12), Row(name=u'Bob', age=15)]
    765         """
--> 766         jdf = self._jdf.select(self._jcols(*cols))
    767         return DataFrame(jdf, self.sql_ctx)
    768 

/home/ubuntu/databricks/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py in __call__(self, *args)
    536         answer = self.gateway_client.send_command(command)
    537         return_value = get_return_value(answer, self.gateway_client,
--> 538                 self.target_id, self.name)
    539 
    540         for temp_arg in temp_args:

/home/ubuntu/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
     38             s = e.java_exception.toString()
     39             if s.startswith('org.apache.spark.sql.AnalysisException: '):
---> 40                 raise AnalysisException(s.split(': ', 1)[1])
     41             if s.startswith('java.lang.IllegalArgumentException: '):
     42                 raise IllegalArgumentException(s.split(': ', 1)[1])

AnalysisException: Reference 'a' is ambiguous, could be: a#894L, a#895L.;
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org