You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by falaki <gi...@git.apache.org> on 2016/09/08 00:15:51 UTC

[GitHub] spark pull request #15007: [SPARKR][SPARK-17442] Additional arguments in wri...

GitHub user falaki opened a pull request:

    https://github.com/apache/spark/pull/15007

    [SPARKR][SPARK-17442] Additional arguments in write.df are not passed to data source

    ## What changes were proposed in this pull request?
    additional options were not passed down in `write.df`. This patch fixes that
    
    
    ## How was this patch tested?
    Adds unit test for passing `header = "true"` for CSV data source in R unit tests
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/falaki/spark SPARK-17442

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/15007.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #15007
    
----
commit 7da87cdb71434aea68f25e0da56b190dc55f4199
Author: Hossein <ho...@databricks.com>
Date:   2016-09-08T00:14:12Z

    Passing options in write.df method dispatch

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #15007: [SPARKR][SPARK-17442] Additional arguments in wri...

Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:

    https://github.com/apache/spark/pull/15007#discussion_r77942583
  
    --- Diff: R/pkg/inst/tests/testthat/test_sparkSQL.R ---
    @@ -243,7 +243,14 @@ test_that("read csv as DataFrame", {
       expect_equal(count(withoutna2), 3)
       expect_equal(count(where(withoutna2, withoutna2$make == "Dummy")), 0)
     
    +  # writing csv file
    +  csvPath2 <- tempfile(pattern = "csvtest2", fileext = ".csv")
    +  write.df(df2, path = csvPath2, "csv", header = "true")
    +  df3 <- read.df(csvPath2, "csv", header = "true")
    +  exect_equal(nrow(df3), nrow(df2))
    --- End diff --
    
    typo expect_equal


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15007: [SPARKR][SPARK-17442] Additional arguments in write.df a...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/15007
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65067/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15007: [SPARKR][SPARK-17442] Additional arguments in write.df a...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/15007
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15007: [SPARKR][SPARK-17442] Additional arguments in write.df a...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/15007
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65078/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15007: [SPARKR][SPARK-17442] Additional arguments in write.df a...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/15007
  
    **[Test build #65067 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65067/consoleFull)** for PR 15007 at commit [`78d1dce`](https://github.com/apache/spark/commit/78d1dcee73ac09dd30d19e57bda457706ae16101).
     * This patch **fails SparkR unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15007: [SPARKR][SPARK-17442] Additional arguments in write.df a...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/15007
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15007: [SPARKR][SPARK-17442] Additional arguments in write.df a...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/15007
  
    **[Test build #65078 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65078/consoleFull)** for PR 15007 at commit [`e2f88dc`](https://github.com/apache/spark/commit/e2f88dc5f020bef4e989a339e4beae2a8677f2cb).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #15007: [SPARKR][SPARK-17442] Additional arguments in wri...

Posted by falaki <gi...@git.apache.org>.
Github user falaki closed the pull request at:

    https://github.com/apache/spark/pull/15007


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15007: [SPARKR][SPARK-17442] Additional arguments in write.df a...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/15007
  
    **[Test build #65065 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65065/consoleFull)** for PR 15007 at commit [`7da87cd`](https://github.com/apache/spark/commit/7da87cdb71434aea68f25e0da56b190dc55f4199).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #15007: [SPARKR][SPARK-17442] Additional arguments in wri...

Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on a diff in the pull request:

    https://github.com/apache/spark/pull/15007#discussion_r77934359
  
    --- Diff: R/pkg/R/DataFrame.R ---
    @@ -2635,7 +2635,7 @@ setMethod("write.df",
                 write <- callJMethod(df@sdf, "write")
                 write <- callJMethod(write, "format", source)
                 write <- callJMethod(write, "mode", jmode)
    -            write <- callJMethod(write, "save", path)
    +            write <- callJMethod(write, "save", options)
    --- End diff --
    
    I am not sure I see a `save` method which takes in `options`. What if we just call `write <- callJMethod(write, "options", options)` in the line before `write <- callJMethod(write, "save", path)` ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15007: [SPARKR][SPARK-17442] Additional arguments in write.df a...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/15007
  
    **[Test build #65067 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65067/consoleFull)** for PR 15007 at commit [`78d1dce`](https://github.com/apache/spark/commit/78d1dcee73ac09dd30d19e57bda457706ae16101).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15007: [SPARKR][SPARK-17442] Additional arguments in write.df a...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/15007
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15007: [SPARKR][SPARK-17442] Additional arguments in write.df a...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/15007
  
    **[Test build #65078 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65078/consoleFull)** for PR 15007 at commit [`e2f88dc`](https://github.com/apache/spark/commit/e2f88dc5f020bef4e989a339e4beae2a8677f2cb).
     * This patch **fails SparkR unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15007: [SPARKR][SPARK-17442] Additional arguments in write.df a...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/15007
  
    **[Test build #65065 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65065/consoleFull)** for PR 15007 at commit [`7da87cd`](https://github.com/apache/spark/commit/7da87cdb71434aea68f25e0da56b190dc55f4199).
     * This patch **fails SparkR unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15007: [SPARKR][SPARK-17442] Additional arguments in write.df a...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/15007
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65065/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15007: [SPARKR][SPARK-17442] Additional arguments in write.df a...

Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on the issue:

    https://github.com/apache/spark/pull/15007
  
    @falaki We can use the PR at https://github.com/apache/spark/pull/15010 and close this one ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15007: [SPARKR][SPARK-17442] Additional arguments in write.df a...

Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:

    https://github.com/apache/spark/pull/15007
  
    sorry for the trouble, I opened #15010 with the fix


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #15007: [SPARKR][SPARK-17442] Additional arguments in wri...

Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:

    https://github.com/apache/spark/pull/15007#discussion_r77942163
  
    --- Diff: R/pkg/R/DataFrame.R ---
    @@ -2635,7 +2635,7 @@ setMethod("write.df",
                 write <- callJMethod(df@sdf, "write")
                 write <- callJMethod(write, "format", source)
                 write <- callJMethod(write, "mode", jmode)
    -            write <- callJMethod(write, "save", path)
    +            write <- callJMethod(write, "save", options)
    --- End diff --
    
    oops, sorry I broke it.
    right, I think the right approach is 
    ```
    write <- callJMethod(write, "options", options)
    write <- callJMethod(write, "save", path)
    ```
    
    `save` should be the last call.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org