You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by saurfang <gi...@git.apache.org> on 2015/10/23 00:07:33 UTC

[GitHub] spark pull request: [SPARK-8277][SPARKR] Faster createDataFrame us...

GitHub user saurfang opened a pull request:

    https://github.com/apache/spark/pull/9234

    [SPARK-8277][SPARKR] Faster createDataFrame using mapply

    With a single loop using `mapply`, I'm able to create DataFrame much faster from R data.frame.
    
    Please see benchmark results and code:
    ```r
    Unit: milliseconds
     expr        min         lq       mean     median        uq       max neval
      old 278.842059 316.711990 338.708011 327.762371 345.78249 549.76514   100
      new   6.545662   7.993431   9.596966   8.636927  10.16429  22.94926   100
    ```
    
    ```r
    library(nycflights13)
    library(microbenchmark)
    data <- head(flights, n = 1000)
    
    # get rid of factor type
    dropFactor <- function(x) {
      if (is.factor(x)) {
        as.character(x)
      } else {
        x
      }
    }
    
    createDataFrameNew <- function(data) {
      data <- data.frame(lapply(data, dropFactor), stringsAsFactors = FALSE)
      do.call(mapply, c(list, as.list(data), SIMPLIFY = FALSE))
    }
    
    createDataFrameOld <- function(data) {
      n <- nrow(data)
      m <- ncol(data)
      lapply(1:n, function(i) {
        lapply(1:m, function(j) { dropFactor(data[i,j]) })
      })
    }
    
    microbenchmark(old = createDataFrameOld(data), new = createDataFrameNew(data))
    ```

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/saurfang/spark createdataframe

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/9234.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #9234
    
----
commit 0bf8b7c932b609252a76a69d63aff8fb705730c3
Author: Forest Fang <fo...@outlook.com>
Date:   2015-10-22T21:59:42Z

    use mapply for faster createDataFrame

commit 397438d02d3f8c540e9e45b7a355f813f1970870
Author: Forest Fang <fo...@outlook.com>
Date:   2015-10-22T22:03:43Z

    drop no longer used m,n

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-8277][SPARKR] Faster createDataFrame us...

Posted by saurfang <gi...@git.apache.org>.

Github user saurfang commented on the pull request:

    https://github.com/apache/spark/pull/9234#issuecomment-150399405
  
    Ah. Thanks for the pointer and I didn't realize this issue has already been worked on. Looks like that PR already had all my brilliant idea ;) I'm closing this then. My apology on the duplicate work.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org