You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by danosipov <gi...@git.apache.org> on 2015/02/03 23:53:56 UTC

[GitHub] spark pull request: [SPARK-5576] saveAsTable into Hive fails due t...

GitHub user danosipov opened a pull request:

    https://github.com/apache/spark/pull/4346

    [SPARK-5576] saveAsTable into Hive fails due to duplicate columns

    Fix issue with Hive incompatibility with case sensitive columns

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/danosipov/spark case_sensitivity_hive

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/4346.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #4346
    
----
commit 64681658dc796e29639313b48e20ffb2f5b55b07
Author: Dan Osipov <da...@shazam.com>
Date:   2015-02-03T19:39:04Z

    Fix issue with Hive incompatibility with case sensitive columns

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-5576][SQL] saveAsTable into Hive fails ...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/4346#issuecomment-73108696
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26846/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-5576][SQL] saveAsTable into Hive fails ...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/4346#issuecomment-72962434
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26783/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-5576][SQL] saveAsTable into Hive fails ...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4346#issuecomment-73154921
  
      [Test build #26866 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26866/consoleFull) for   PR 4346 at commit [`3ce57cd`](https://github.com/apache/spark/commit/3ce57cd386d9273b78f0c5816c35ca209e92abcd).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-5576][SQL] saveAsTable into Hive fails ...

Posted by danosipov <gi...@git.apache.org>.

Github user danosipov commented on the pull request:

    https://github.com/apache/spark/pull/4346#issuecomment-72759153
  
    @sryza no problem, done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-5576][SQL] saveAsTable into Hive fails ...

Posted by danosipov <gi...@git.apache.org>.

Github user danosipov commented on the pull request:

    https://github.com/apache/spark/pull/4346#issuecomment-73162050
  
    Works for me - can you delete the PR if no longer need it? I'll keep the fork for local testing.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-5576][SQL] saveAsTable into Hive fails ...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4346#issuecomment-73098199
  
      [Test build #26846 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26846/consoleFull) for   PR 4346 at commit [`c649053`](https://github.com/apache/spark/commit/c64905333953a3f06ba0d0050a95690909b43324).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-5576][SQL] saveAsTable into Hive fails ...

Posted by JoshRosen <gi...@git.apache.org>.

Github user JoshRosen commented on the pull request:

    https://github.com/apache/spark/pull/4346#issuecomment-73134499
  
    Even though the Scala method has a default, from Java's POV it compiles down to a method without default arguments.  One way to fix this is to provide an overloaded Scala method that accepts the same set of old arguments and which calls the new method with an explicit default (e.g. the way that you'd add a new parameter in Java while retaining backwards-compatibility).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-5576][SQL] saveAsTable into Hive fails ...

Posted by marmbrus <gi...@git.apache.org>.

Github user marmbrus commented on the pull request:

    https://github.com/apache/spark/pull/4346#issuecomment-73154430
  
    Thanks for pointing out this issue and working on a solution!
    
    I'm a little hesitant to bake the solution into JSON as this seems like something that could happen with any data source.  Also it seems like you might be able to work around it without making changes to the library itself.  Would something like this work for you (assuming I add `coalesce` to the dataframe DSL)?
    
    ```scala
    val projection = df.schema
      .map(_.name)
      .groupBy(_.toLowerCase)
      .map { 
        case (lowerCase, otherCases) => 
          coalsece(otherCases.map(df(_)).as(lowerCase)) 
      }
    
    df.select(projection: _*)
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-5576][SQL] saveAsTable into Hive fails ...

Posted by JoshRosen <gi...@git.apache.org>.

Github user JoshRosen commented on the pull request:

    https://github.com/apache/spark/pull/4346#issuecomment-73126716
  
    I think that the failures here are due to changes to the method signatures in SQLContext: PySpark SQL code depended on the old signatures, which is causing this to fail:
    
    ```
    **********************************************************************
    File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/sql.py", line 1595, in pyspark.sql.SQLContext.jsonRDD
    Failed example:
        sqlCtx.jsonRDD(sc.parallelize(['{}',
                '{"key0": {"key1": "value1"}}'])).collect()
    Exception raised:
        Traceback (most recent call last):
          File "/usr/lib64/python2.6/doctest.py", line 1253, in __run
            compileflags, 1) in test.globs
          File "<doctest pyspark.sql.SQLContext.jsonRDD[13]>", line 2, in <module>
            '{"key0": {"key1": "value1"}}'])).collect()
          File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/sql.py", line 1614, in jsonRDD
            df = self._ssql_ctx.jsonRDD(jrdd.rdd(), samplingRatio)
          File "/home/jenkins/workspace/SparkPullRequestBuilder/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in __call__
            self.target_id, self.name)
          File "/home/jenkins/workspace/SparkPullRequestBuilder/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 304, in get_return_value
            format(target_id, '.', name, value))
        Py4JError: An error occurred while calling o76.jsonRDD. Trace:
        py4j.Py4JException: Method jsonRDD([class org.apache.spark.rdd.MapPartitionsRDD, class java.lang.Double]) does not exist
        	at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:333)
        	at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:342)
        	at py4j.Gateway.invoke(Gateway.java:252)
        	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
        	at py4j.commands.CallCommand.execute(CallCommand.java:79)
        	at py4j.GatewayConnection.run(GatewayConnection.java:207)
        	at java.lang.Thread.run(Thread.java:745)
    
    
    **********************************************************************
    File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/sql.py", line 1598, in pyspark.sql.SQLContext.jsonRDD
    Failed example:
        sqlCtx.jsonRDD(sc.parallelize(['{"key0": null}',
                '{"key0": {"key1": "value1"}}'])).collect()
    Exception raised:
        Traceback (most recent call last):
          File "/usr/lib64/python2.6/doctest.py", line 1253, in __run
            compileflags, 1) in test.globs
          File "<doctest pyspark.sql.SQLContext.jsonRDD[14]>", line 2, in <module>
            '{"key0": {"key1": "value1"}}'])).collect()
          File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/sql.py", line 1614, in jsonRDD
            df = self._ssql_ctx.jsonRDD(jrdd.rdd(), samplingRatio)
          File "/home/jenkins/workspace/SparkPullRequestBuilder/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in __call__
            self.target_id, self.name)
          File "/home/jenkins/workspace/SparkPullRequestBuilder/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 304, in get_return_value
            format(target_id, '.', name, value))
        Py4JError: An error occurred while calling o76.jsonRDD. Trace:
        py4j.Py4JException: Method jsonRDD([class org.apache.spark.rdd.MapPartitionsRDD, class java.lang.Double]) does not exist
        	at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:333)
        	at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:342)
        	at py4j.Gateway.invoke(Gateway.java:252)
        	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
        	at py4j.commands.CallCommand.execute(CallCommand.java:79)
        	at py4j.GatewayConnection.run(GatewayConnection.java:207)
        	at java.lang.Thread.run(Thread.java:745)
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-5576][SQL] saveAsTable into Hive fails ...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4346#issuecomment-72970324
  
      [Test build #26790 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26790/consoleFull) for   PR 4346 at commit [`de23c2e`](https://github.com/apache/spark/commit/de23c2e96d4ef07d0deec08c8f8fbbfd9f8756c1).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-5576][SQL] saveAsTable into Hive fails ...

Posted by JoshRosen <gi...@git.apache.org>.

Github user JoshRosen commented on the pull request:

    https://github.com/apache/spark/pull/4346#issuecomment-73133352
  
    Try grepping through the PySpark source to find places where we call `jsonRDD`.  It looks like your PR added another parameter, so any callers are going to have to be updated to provide that parameter.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org