You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by nongli <gi...@git.apache.org> on 2016/01/05 07:03:26 UTC

[GitHub] spark pull request: [SPARK-12640][SQL] Add simple benchmarking uti...

GitHub user nongli opened a pull request:

    https://github.com/apache/spark/pull/10589

    [SPARK-12640][SQL] Add simple benchmarking utility class and add Parquet scan benchmarks.

    [SPARK-12640][SQL] Add simple benchmarking utility class and add Parquet scan benchmarks.
    
    We've run benchmarks ad hoc to measure the scanner performance. We will continue to invest in this
    and it makes sense to get these benchmarks into code. This adds a simple benchmarking utility to do
    this.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/nongli/spark spark-12640

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/10589.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #10589
    
----
commit 7eeff58298ceac076779a5cae05ca674ed0ac51a
Author: Nong <no...@gmail.com>
Date:   2015-12-31T22:45:30Z

    [SPARK-12636][SQL] Update UnsafeRowParquetRecordReader to support reading paths directly.
    
    As noted in the code, this change is to make this componenet easier to
    test in isolation.

commit 22afd1f0115b86cdb5ba661dd2c0714ff6a4243b
Author: Nong <no...@gmail.com>
Date:   2016-01-01T00:26:34Z

    [SPARK-12640][SQL] Add simple benchmarking utility class and add Parquet scan benchmarks.
    
    We've run benchmarks ad hoc to measure the scanner performance. We will continue to invest in this
    and it makes sense to get these benchmarks into code. This adds a simple benchmarking utility to do
    this.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12640][SQL] Add simple benchmarking uti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/10589#issuecomment-169512261
  
    **[Test build #48876 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48876/consoleFull)** for PR 10589 at commit [`f231e9a`](https://github.com/apache/spark/commit/f231e9ac3c4ae440e020b9f8494dcc34bbecf14c).
     * This patch passes all tests.
     * This patch **does not merge cleanly**.
     * This patch adds the following public classes _(experimental)_:
      * `  case class Case(name: String, fn: Int => Unit)`
      * `  case class Result(avgMs: Double, avgRate: Double)`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12640][SQL] Add simple benchmarking uti...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10589#discussion_r48921242
  
    --- Diff: core/src/test/scala/org/apache/spark/Benchmark.scala ---
    @@ -0,0 +1,102 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark
    +
    +import scala.collection.mutable
    +
    +import org.apache.commons.lang3.SystemUtils
    +import org.apache.spark.util.Utils
    +
    +/**
    + * Utility class to benchmark components. An example of how to use this is:
    + *  val benchmark = new Benchmark("My Benchmark", valuesPerIteration)
    + *   benchmark.addCase("V1", <function>")
    + *   benchmark.addCase("V2", <function>")
    + *   benchmark.run
    + * This will output the average time to run each function and the rate of each function.
    + *
    + * The benchmark function takes one argument that is the iteration that's being run
    + */
    +class Benchmark(name: String, valuesPerIteration: Long, iters: Int = 5) {
    +  val benchmarks = mutable.ArrayBuffer.empty[Benchmark.Case]
    +
    +  def addCase(name: String, f: Int => Unit): Unit = {
    +    benchmarks += Benchmark.Case(name, f)
    +  }
    +
    +  /**
    +   * Runs the benchmark and outputs the results to stdout. This should be copied and added as
    +   * a comment with the benchmark. Although the results vary from machine to machine, it should
    +   * provide some baseline.
    +   */
    +  def run(): Unit = {
    +    require(benchmarks.nonEmpty)
    +    val results = benchmarks.map { c =>
    +      Benchmark.measure(valuesPerIteration, c.fn, iters)
    +    }
    +    val firstRate = results.head.avgRate
    +    // scalastyle:off
    +    // The results are going to be processor specific so it is useful to include that.
    +    println(Benchmark.getProcessorName())
    +    printf("%-24s %16s %16s %14s\n", name + ":", "Avg Time(ms)", "Avg Rate(M/s)", "Relative Rate")
    +    println("-------------------------------------------------------------------------")
    +    results.zip(benchmarks).foreach { r =>
    +      printf("%-24s %16s %16s %14s\n", r._2.name, r._1.avgMs.toString, "%10.2f" format r._1.avgRate,
    +        "%6.2f X" format (r._1.avgRate / firstRate))
    +    }
    +    println
    +    // scalastyle:on
    +  }
    +}
    +
    +object Benchmark {
    +  case class Case(name: String, fn: Int => Unit)
    +  case class Result(avgMs: Double, avgRate: Double)
    +
    +  /**
    +   * This should return a user helpful processor information. Getting at this depends on the OS.
    +   * This should return something like "Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz"
    +   */
    +  def getProcessorName(): String = {
    +    if (SystemUtils.IS_OS_MAC_OSX) {
    +      Utils.executeAndGetOutput(Seq("/usr/sbin/sysctl", "-n", "machdep.cpu.brand_string"))
    +    } else if (SystemUtils.IS_OS_LINUX) {
    +      Utils.executeAndGetOutput(Seq("/usr/bin/grep", "-m", "1", "\"model name\"", "/proc/cpuinfo"))
    +    } else {
    +      System.getenv("PROCESSOR_IDENTIFIER")
    +    }
    +  }
    +
    +  /**
    +   * Runs a single function `f` for iters, returning the average time the function took and
    +   * the rate of the function.
    +   */
    +  def measure(num: Long, f: Int => Unit, iters: Int): Result = {
    +    var totalTime = 0L
    +    for (i <- 0 until iters + 1) {
    +      val start = System.currentTimeMillis()
    --- End diff --
    
    How about the calling System.nanoTime() for short-running benchmarks instead of calling System.currentTimeMillis()?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12640][SQL] Add simple benchmarking uti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/10589#issuecomment-169512584
  
    **[Test build #2341 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2341/consoleFull)** for PR 10589 at commit [`4f682c5`](https://github.com/apache/spark/commit/4f682c587bc92b226ea05844e0de85b353ddbbd8).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12640][SQL] Add simple benchmarking uti...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10589#discussion_r48929712
  
    --- Diff: core/src/test/scala/org/apache/spark/Benchmark.scala ---
    @@ -0,0 +1,102 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark
    +
    +import scala.collection.mutable
    +
    +import org.apache.commons.lang3.SystemUtils
    +import org.apache.spark.util.Utils
    +
    +/**
    + * Utility class to benchmark components. An example of how to use this is:
    + *  val benchmark = new Benchmark("My Benchmark", valuesPerIteration)
    + *   benchmark.addCase("V1", <function>")
    + *   benchmark.addCase("V2", <function>")
    + *   benchmark.run
    + * This will output the average time to run each function and the rate of each function.
    + *
    + * The benchmark function takes one argument that is the iteration that's being run
    + */
    +class Benchmark(name: String, valuesPerIteration: Long, iters: Int = 5) {
    +  val benchmarks = mutable.ArrayBuffer.empty[Benchmark.Case]
    +
    +  def addCase(name: String, f: Int => Unit): Unit = {
    +    benchmarks += Benchmark.Case(name, f)
    +  }
    +
    +  /**
    +   * Runs the benchmark and outputs the results to stdout. This should be copied and added as
    +   * a comment with the benchmark. Although the results vary from machine to machine, it should
    +   * provide some baseline.
    +   */
    +  def run(): Unit = {
    +    require(benchmarks.nonEmpty)
    +    val results = benchmarks.map { c =>
    +      Benchmark.measure(valuesPerIteration, c.fn, iters)
    +    }
    +    val firstRate = results.head.avgRate
    +    // scalastyle:off
    +    // The results are going to be processor specific so it is useful to include that.
    +    println(Benchmark.getProcessorName())
    +    printf("%-24s %16s %16s %14s\n", name + ":", "Avg Time(ms)", "Avg Rate(M/s)", "Relative Rate")
    +    println("-------------------------------------------------------------------------")
    +    results.zip(benchmarks).foreach { r =>
    +      printf("%-24s %16s %16s %14s\n", r._2.name, r._1.avgMs.toString, "%10.2f" format r._1.avgRate,
    +        "%6.2f X" format (r._1.avgRate / firstRate))
    +    }
    +    println
    +    // scalastyle:on
    +  }
    +}
    +
    +object Benchmark {
    +  case class Case(name: String, fn: Int => Unit)
    +  case class Result(avgMs: Double, avgRate: Double)
    +
    +  /**
    +   * This should return a user helpful processor information. Getting at this depends on the OS.
    +   * This should return something like "Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz"
    +   */
    +  def getProcessorName(): String = {
    +    if (SystemUtils.IS_OS_MAC_OSX) {
    +      Utils.executeAndGetOutput(Seq("/usr/sbin/sysctl", "-n", "machdep.cpu.brand_string"))
    +    } else if (SystemUtils.IS_OS_LINUX) {
    +      Utils.executeAndGetOutput(Seq("/usr/bin/grep", "-m", "1", "\"model name\"", "/proc/cpuinfo"))
    +    } else {
    +      System.getenv("PROCESSOR_IDENTIFIER")
    +    }
    +  }
    +
    +  /**
    +   * Runs a single function `f` for iters, returning the average time the function took and
    +   * the rate of the function.
    +   */
    +  def measure(num: Long, f: Int => Unit, iters: Int): Result = {
    +    var totalTime = 0L
    --- End diff --
    
    I'd consider printing the detailed timing about each run too. Anyway for the purpose of what @nongli is trying to accomplish, I don't think JIT is that big of a deal here.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12640][SQL] Add simple benchmarking uti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/10589#issuecomment-169483415
  
    **[Test build #48876 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48876/consoleFull)** for PR 10589 at commit [`f231e9a`](https://github.com/apache/spark/commit/f231e9ac3c4ae440e020b9f8494dcc34bbecf14c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12640][SQL] Add simple benchmarking uti...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10589#discussion_r48929696
  
    --- Diff: core/src/test/scala/org/apache/spark/Benchmark.scala ---
    @@ -0,0 +1,102 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark
    +
    +import scala.collection.mutable
    +
    +import org.apache.commons.lang3.SystemUtils
    +import org.apache.spark.util.Utils
    +
    +/**
    + * Utility class to benchmark components. An example of how to use this is:
    + *  val benchmark = new Benchmark("My Benchmark", valuesPerIteration)
    + *   benchmark.addCase("V1", <function>")
    + *   benchmark.addCase("V2", <function>")
    + *   benchmark.run
    + * This will output the average time to run each function and the rate of each function.
    + *
    + * The benchmark function takes one argument that is the iteration that's being run
    + */
    +class Benchmark(name: String, valuesPerIteration: Long, iters: Int = 5) {
    +  val benchmarks = mutable.ArrayBuffer.empty[Benchmark.Case]
    +
    +  def addCase(name: String, f: Int => Unit): Unit = {
    +    benchmarks += Benchmark.Case(name, f)
    +  }
    +
    +  /**
    +   * Runs the benchmark and outputs the results to stdout. This should be copied and added as
    +   * a comment with the benchmark. Although the results vary from machine to machine, it should
    +   * provide some baseline.
    +   */
    +  def run(): Unit = {
    +    require(benchmarks.nonEmpty)
    +    val results = benchmarks.map { c =>
    +      Benchmark.measure(valuesPerIteration, c.fn, iters)
    +    }
    +    val firstRate = results.head.avgRate
    +    // scalastyle:off
    +    // The results are going to be processor specific so it is useful to include that.
    +    println(Benchmark.getProcessorName())
    +    printf("%-24s %16s %16s %14s\n", name + ":", "Avg Time(ms)", "Avg Rate(M/s)", "Relative Rate")
    +    println("-------------------------------------------------------------------------")
    +    results.zip(benchmarks).foreach { r =>
    +      printf("%-24s %16s %16s %14s\n", r._2.name, r._1.avgMs.toString, "%10.2f" format r._1.avgRate,
    +        "%6.2f X" format (r._1.avgRate / firstRate))
    +    }
    +    println
    +    // scalastyle:on
    +  }
    +}
    +
    +object Benchmark {
    +  case class Case(name: String, fn: Int => Unit)
    +  case class Result(avgMs: Double, avgRate: Double)
    +
    +  /**
    +   * This should return a user helpful processor information. Getting at this depends on the OS.
    +   * This should return something like "Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz"
    +   */
    +  def getProcessorName(): String = {
    +    if (SystemUtils.IS_OS_MAC_OSX) {
    +      Utils.executeAndGetOutput(Seq("/usr/sbin/sysctl", "-n", "machdep.cpu.brand_string"))
    +    } else if (SystemUtils.IS_OS_LINUX) {
    +      Utils.executeAndGetOutput(Seq("/usr/bin/grep", "-m", "1", "\"model name\"", "/proc/cpuinfo"))
    +    } else {
    +      System.getenv("PROCESSOR_IDENTIFIER")
    +    }
    +  }
    +
    +  /**
    +   * Runs a single function `f` for iters, returning the average time the function took and
    +   * the rate of the function.
    +   */
    +  def measure(num: Long, f: Int => Unit, iters: Int): Result = {
    +    var totalTime = 0L
    +    for (i <- 0 until iters + 1) {
    +      val start = System.currentTimeMillis()
    --- End diff --
    
    +1 on nanoTime. It is more accurate than milli. See https://github.com/databricks/scala-style-guide#misc_currentTimeMillis_vs_nanoTime


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12640][SQL] Add simple benchmarking uti...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/10589


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12640][SQL] Add simple benchmarking uti...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10589#discussion_r48922143
  
    --- Diff: core/src/test/scala/org/apache/spark/Benchmark.scala ---
    @@ -0,0 +1,102 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark
    +
    +import scala.collection.mutable
    +
    +import org.apache.commons.lang3.SystemUtils
    +import org.apache.spark.util.Utils
    +
    +/**
    + * Utility class to benchmark components. An example of how to use this is:
    + *  val benchmark = new Benchmark("My Benchmark", valuesPerIteration)
    + *   benchmark.addCase("V1", <function>")
    + *   benchmark.addCase("V2", <function>")
    + *   benchmark.run
    + * This will output the average time to run each function and the rate of each function.
    + *
    + * The benchmark function takes one argument that is the iteration that's being run
    + */
    +class Benchmark(name: String, valuesPerIteration: Long, iters: Int = 5) {
    +  val benchmarks = mutable.ArrayBuffer.empty[Benchmark.Case]
    +
    +  def addCase(name: String, f: Int => Unit): Unit = {
    +    benchmarks += Benchmark.Case(name, f)
    +  }
    +
    +  /**
    +   * Runs the benchmark and outputs the results to stdout. This should be copied and added as
    +   * a comment with the benchmark. Although the results vary from machine to machine, it should
    +   * provide some baseline.
    +   */
    +  def run(): Unit = {
    +    require(benchmarks.nonEmpty)
    +    val results = benchmarks.map { c =>
    +      Benchmark.measure(valuesPerIteration, c.fn, iters)
    +    }
    +    val firstRate = results.head.avgRate
    +    // scalastyle:off
    +    // The results are going to be processor specific so it is useful to include that.
    +    println(Benchmark.getProcessorName())
    +    printf("%-24s %16s %16s %14s\n", name + ":", "Avg Time(ms)", "Avg Rate(M/s)", "Relative Rate")
    +    println("-------------------------------------------------------------------------")
    +    results.zip(benchmarks).foreach { r =>
    +      printf("%-24s %16s %16s %14s\n", r._2.name, r._1.avgMs.toString, "%10.2f" format r._1.avgRate,
    +        "%6.2f X" format (r._1.avgRate / firstRate))
    +    }
    +    println
    +    // scalastyle:on
    +  }
    +}
    +
    +object Benchmark {
    +  case class Case(name: String, fn: Int => Unit)
    +  case class Result(avgMs: Double, avgRate: Double)
    +
    +  /**
    +   * This should return a user helpful processor information. Getting at this depends on the OS.
    +   * This should return something like "Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz"
    +   */
    +  def getProcessorName(): String = {
    +    if (SystemUtils.IS_OS_MAC_OSX) {
    +      Utils.executeAndGetOutput(Seq("/usr/sbin/sysctl", "-n", "machdep.cpu.brand_string"))
    +    } else if (SystemUtils.IS_OS_LINUX) {
    +      Utils.executeAndGetOutput(Seq("/usr/bin/grep", "-m", "1", "\"model name\"", "/proc/cpuinfo"))
    +    } else {
    +      System.getenv("PROCESSOR_IDENTIFIER")
    +    }
    +  }
    +
    +  /**
    +   * Runs a single function `f` for iters, returning the average time the function took and
    +   * the rate of the function.
    +   */
    +  def measure(num: Long, f: Int => Unit, iters: Int): Result = {
    +    var totalTime = 0L
    --- End diff --
    
    If you would like to measure the peak performance, it is necessary for warm-up run before the measurement. The current driver may include execution time on a bytecode interpreter.
    This is because a just-in-time compiler in Java runtime usually starts translating java bytecode to CPU native code after the threshold of method invocations/branches exceeded the preset value (e.g. 10000 for OracleJDK). Before executing the native code, java bytecode is executed on the interpreter.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12640][SQL] Add simple benchmarking uti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/10589#issuecomment-168932322
  
    **[Test build #48738 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48738/consoleFull)** for PR 10589 at commit [`22afd1f`](https://github.com/apache/spark/commit/22afd1f0115b86cdb5ba661dd2c0714ff6a4243b).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `class Benchmark(name: String, valuesPerIteration: Long, iters: Int = 5) `
      * `  case class Case(name: String, fn: Int => Unit)`
      * `  case class Result(avgMs: Double, avgRate: Double)`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12640][SQL] Add simple benchmarking uti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/10589#issuecomment-169537008
  
    **[Test build #48885 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48885/consoleFull)** for PR 10589 at commit [`4f682c5`](https://github.com/apache/spark/commit/4f682c587bc92b226ea05844e0de85b353ddbbd8).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `  case class Case(name: String, fn: Int => Unit)`
      * `  case class Result(avgMs: Double, avgRate: Double)`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12640][SQL] Add simple benchmarking uti...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10589#discussion_r48815062
  
    --- Diff: core/src/test/scala/org/apache/spark/Benchmark.scala ---
    @@ -0,0 +1,102 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark
    +
    +import scala.collection.mutable
    +
    +import org.apache.commons.lang3.SystemUtils
    +import org.apache.spark.util.Utils
    +
    +/**
    + * Utility class to benchmark components. An example of how to use this is:
    + *  val benchmark = new Benchmark("My Benchmark", valuesPerIteration)
    + *   benchmark.addCase("V1", <function>")
    + *   benchmark.addCase("V2", <function>")
    + *   benchmark.run
    + * This will output the average time to run each function and the rate of each function.
    + *
    + * The benchmark function takes one argument that is the iteration that's being run
    + */
    +class Benchmark(name: String, valuesPerIteration: Long, iters: Int = 5) {
    --- End diff --
    
    move this into util package?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12640][SQL] Add simple benchmarking uti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/10589#issuecomment-168911272
  
    **[Test build #48738 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48738/consoleFull)** for PR 10589 at commit [`22afd1f`](https://github.com/apache/spark/commit/22afd1f0115b86cdb5ba661dd2c0714ff6a4243b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12640][SQL] Add simple benchmarking uti...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/10589#issuecomment-169537193
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48885/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12640][SQL] Add simple benchmarking uti...

Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10589#discussion_r48900041
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadBenchmark.scala ---
    @@ -0,0 +1,157 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.spark.sql.execution.datasources.parquet
    +
    +import java.io.File
    +
    +import scala.collection.JavaConverters._
    +import scala.util.Try
    +
    +import org.apache.spark.sql.{SQLConf, SQLContext}
    +import org.apache.spark.util.Utils
    +import org.apache.spark.{Benchmark, SparkConf, SparkContext}
    +
    +/**
    + * Benchmark to measure parquet read performance.
    + */
    +object ParquetReadBenchmark {
    +  val conf = new SparkConf()
    +  conf.set("spark.sql.parquet.compression.codec", "snappy")
    +  val sc = new SparkContext("local[1]", "test-sql-context", conf)
    +  val sqlContext = new SQLContext(sc)
    +
    +  def withTempPath(f: File => Unit): Unit = {
    +    val path = Utils.createTempDir()
    +    path.delete()
    +    try f(path) finally Utils.deleteRecursively(path)
    +  }
    +
    +  def withTempTable(tableNames: String*)(f: => Unit): Unit = {
    +    try f finally tableNames.foreach(sqlContext.dropTempTable)
    +  }
    +
    +  def withSQLConf(pairs: (String, String)*)(f: => Unit): Unit = {
    +    val (keys, values) = pairs.unzip
    +    val currentValues = keys.map(key => Try(sqlContext.conf.getConfString(key)).toOption)
    +    (keys, values).zipped.foreach(sqlContext.conf.setConfString)
    +    try f finally {
    +      keys.zip(currentValues).foreach {
    +        case (key, Some(value)) => sqlContext.conf.setConfString(key, value)
    +        case (key, None) => sqlContext.conf.unsetConf(key)
    +      }
    +    }
    +  }
    +
    +  def intScanBenchmark(values: Int): Unit = {
    +    withTempPath { dir =>
    +      sqlContext.range(values).write.parquet(dir.getCanonicalPath)
    +      withTempTable("tempTable") {
    +        sqlContext.read.parquet(dir.getCanonicalPath).registerTempTable("tempTable")
    +        val benchmark = new Benchmark("Single Int Column Scan", values)
    +
    +        benchmark.addCase("SQL", iter => {
    +          sqlContext.sql("select sum(id) from tempTable").collect()
    +        })
    +
    +        benchmark.addCase("SQL Parquet MR", iter => {
    +          withSQLConf(SQLConf.PARQUET_UNSAFE_ROW_RECORD_READER_ENABLED.key -> "false") {
    +            sqlContext.sql("select sum(id) from tempTable").collect()
    +          }
    +        })
    +
    +        val files = SpecificParquetRecordReaderBase.listDirectory(dir).toArray
    +        benchmark.addCase("ParquetReader", num => {
    +          var sum = 0L
    +          files.map(_.asInstanceOf[String]).foreach { p =>
    +            val reader = new UnsafeRowParquetRecordReader
    +            reader.initialize(p, ("id" :: Nil).asJava)
    +
    +            while (reader.nextKeyValue()) {
    +              val record = reader.getCurrentValue
    +              if (!record.isNullAt(0)) sum += record.getInt(0)
    +            }
    +            reader.close()
    +        }})
    +
    +        /*
    +          Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
    +          Single Int Column Scan:      Avg Time(ms)    Avg Rate(M/s)  Relative Rate
    +          -------------------------------------------------------------------------
    +          SQL                                1910.0            13.72         1.00 X
    --- End diff --
    
    This should be `SQL with ParquetReader`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12640][SQL] Add simple benchmarking uti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/10589#issuecomment-169534862
  
    **[Test build #2341 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2341/consoleFull)** for PR 10589 at commit [`4f682c5`](https://github.com/apache/spark/commit/4f682c587bc92b226ea05844e0de85b353ddbbd8).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `  case class Case(name: String, fn: Int => Unit)`
      * `  case class Result(avgMs: Double, avgRate: Double)`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12640][SQL] Add simple benchmarking uti...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/10589#issuecomment-169512424
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48876/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12640][SQL] Add simple benchmarking uti...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10589#discussion_r48960725
  
    --- Diff: core/src/test/scala/org/apache/spark/Benchmark.scala ---
    @@ -0,0 +1,102 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark
    +
    +import scala.collection.mutable
    +
    +import org.apache.commons.lang3.SystemUtils
    +import org.apache.spark.util.Utils
    +
    +/**
    + * Utility class to benchmark components. An example of how to use this is:
    + *  val benchmark = new Benchmark("My Benchmark", valuesPerIteration)
    + *   benchmark.addCase("V1", <function>")
    + *   benchmark.addCase("V2", <function>")
    + *   benchmark.run
    + * This will output the average time to run each function and the rate of each function.
    + *
    + * The benchmark function takes one argument that is the iteration that's being run
    + */
    +class Benchmark(name: String, valuesPerIteration: Long, iters: Int = 5) {
    +  val benchmarks = mutable.ArrayBuffer.empty[Benchmark.Case]
    +
    +  def addCase(name: String, f: Int => Unit): Unit = {
    +    benchmarks += Benchmark.Case(name, f)
    +  }
    +
    +  /**
    +   * Runs the benchmark and outputs the results to stdout. This should be copied and added as
    +   * a comment with the benchmark. Although the results vary from machine to machine, it should
    +   * provide some baseline.
    +   */
    +  def run(): Unit = {
    +    require(benchmarks.nonEmpty)
    +    val results = benchmarks.map { c =>
    +      Benchmark.measure(valuesPerIteration, c.fn, iters)
    +    }
    +    val firstRate = results.head.avgRate
    +    // scalastyle:off
    +    // The results are going to be processor specific so it is useful to include that.
    +    println(Benchmark.getProcessorName())
    +    printf("%-24s %16s %16s %14s\n", name + ":", "Avg Time(ms)", "Avg Rate(M/s)", "Relative Rate")
    +    println("-------------------------------------------------------------------------")
    +    results.zip(benchmarks).foreach { r =>
    +      printf("%-24s %16s %16s %14s\n", r._2.name, r._1.avgMs.toString, "%10.2f" format r._1.avgRate,
    +        "%6.2f X" format (r._1.avgRate / firstRate))
    +    }
    +    println
    +    // scalastyle:on
    +  }
    +}
    +
    +object Benchmark {
    +  case class Case(name: String, fn: Int => Unit)
    +  case class Result(avgMs: Double, avgRate: Double)
    +
    +  /**
    +   * This should return a user helpful processor information. Getting at this depends on the OS.
    +   * This should return something like "Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz"
    +   */
    +  def getProcessorName(): String = {
    +    if (SystemUtils.IS_OS_MAC_OSX) {
    +      Utils.executeAndGetOutput(Seq("/usr/sbin/sysctl", "-n", "machdep.cpu.brand_string"))
    +    } else if (SystemUtils.IS_OS_LINUX) {
    +      Utils.executeAndGetOutput(Seq("/usr/bin/grep", "-m", "1", "\"model name\"", "/proc/cpuinfo"))
    +    } else {
    +      System.getenv("PROCESSOR_IDENTIFIER")
    +    }
    +  }
    +
    +  /**
    +   * Runs a single function `f` for iters, returning the average time the function took and
    +   * the rate of the function.
    +   */
    +  def measure(num: Long, f: Int => Unit, iters: Int): Result = {
    +    var totalTime = 0L
    --- End diff --
    
    Sounds good for printing detailed timing about each run. It also allows us to check variations of elapsed time at each run.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12640][SQL] Add simple benchmarking uti...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/10589#issuecomment-169512866
  
    **[Test build #48885 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48885/consoleFull)** for PR 10589 at commit [`4f682c5`](https://github.com/apache/spark/commit/4f682c587bc92b226ea05844e0de85b353ddbbd8).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12640][SQL] Add simple benchmarking uti...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/10589#issuecomment-169537192
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12640][SQL] Add simple benchmarking uti...

Posted by nongli <gi...@git.apache.org>.
Github user nongli commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10589#discussion_r49010319
  
    --- Diff: core/src/test/scala/org/apache/spark/Benchmark.scala ---
    @@ -0,0 +1,102 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark
    +
    +import scala.collection.mutable
    +
    +import org.apache.commons.lang3.SystemUtils
    +import org.apache.spark.util.Utils
    +
    +/**
    + * Utility class to benchmark components. An example of how to use this is:
    + *  val benchmark = new Benchmark("My Benchmark", valuesPerIteration)
    + *   benchmark.addCase("V1", <function>")
    + *   benchmark.addCase("V2", <function>")
    + *   benchmark.run
    + * This will output the average time to run each function and the rate of each function.
    + *
    + * The benchmark function takes one argument that is the iteration that's being run
    + */
    +class Benchmark(name: String, valuesPerIteration: Long, iters: Int = 5) {
    +  val benchmarks = mutable.ArrayBuffer.empty[Benchmark.Case]
    +
    +  def addCase(name: String, f: Int => Unit): Unit = {
    +    benchmarks += Benchmark.Case(name, f)
    +  }
    +
    +  /**
    +   * Runs the benchmark and outputs the results to stdout. This should be copied and added as
    +   * a comment with the benchmark. Although the results vary from machine to machine, it should
    +   * provide some baseline.
    +   */
    +  def run(): Unit = {
    +    require(benchmarks.nonEmpty)
    +    val results = benchmarks.map { c =>
    +      Benchmark.measure(valuesPerIteration, c.fn, iters)
    +    }
    +    val firstRate = results.head.avgRate
    +    // scalastyle:off
    +    // The results are going to be processor specific so it is useful to include that.
    +    println(Benchmark.getProcessorName())
    +    printf("%-24s %16s %16s %14s\n", name + ":", "Avg Time(ms)", "Avg Rate(M/s)", "Relative Rate")
    +    println("-------------------------------------------------------------------------")
    +    results.zip(benchmarks).foreach { r =>
    +      printf("%-24s %16s %16s %14s\n", r._2.name, r._1.avgMs.toString, "%10.2f" format r._1.avgRate,
    +        "%6.2f X" format (r._1.avgRate / firstRate))
    +    }
    +    println
    +    // scalastyle:on
    +  }
    +}
    +
    +object Benchmark {
    +  case class Case(name: String, fn: Int => Unit)
    +  case class Result(avgMs: Double, avgRate: Double)
    +
    +  /**
    +   * This should return a user helpful processor information. Getting at this depends on the OS.
    +   * This should return something like "Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz"
    +   */
    +  def getProcessorName(): String = {
    +    if (SystemUtils.IS_OS_MAC_OSX) {
    +      Utils.executeAndGetOutput(Seq("/usr/sbin/sysctl", "-n", "machdep.cpu.brand_string"))
    +    } else if (SystemUtils.IS_OS_LINUX) {
    +      Utils.executeAndGetOutput(Seq("/usr/bin/grep", "-m", "1", "\"model name\"", "/proc/cpuinfo"))
    +    } else {
    +      System.getenv("PROCESSOR_IDENTIFIER")
    +    }
    +  }
    +
    +  /**
    +   * Runs a single function `f` for iters, returning the average time the function took and
    +   * the rate of the function.
    +   */
    +  def measure(num: Long, f: Int => Unit, iters: Int): Result = {
    +    var totalTime = 0L
    --- End diff --
    
    I skip the first iteration from the timing to deal with start up. So specifying iterations = 2 runs it 3 times.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12640][SQL] Add simple benchmarking uti...

Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:

    https://github.com/apache/spark/pull/10589#issuecomment-169140659
  
    @nongli You don't need to repeat the title in the description of the PR (or it get duplicated in commit message).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12640][SQL] Add simple benchmarking uti...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:

    https://github.com/apache/spark/pull/10589#issuecomment-169538037
  
    I've merged this. Thanks!



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12640][SQL] Add simple benchmarking uti...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/10589#issuecomment-168932596
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48738/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12640][SQL] Add simple benchmarking uti...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10589#discussion_r48815166
  
    --- Diff: core/src/test/scala/org/apache/spark/Benchmark.scala ---
    @@ -0,0 +1,102 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark
    +
    +import scala.collection.mutable
    +
    +import org.apache.commons.lang3.SystemUtils
    +import org.apache.spark.util.Utils
    +
    +/**
    + * Utility class to benchmark components. An example of how to use this is:
    + *  val benchmark = new Benchmark("My Benchmark", valuesPerIteration)
    + *   benchmark.addCase("V1", <function>")
    + *   benchmark.addCase("V2", <function>")
    + *   benchmark.run
    + * This will output the average time to run each function and the rate of each function.
    + *
    + * The benchmark function takes one argument that is the iteration that's being run
    + */
    +class Benchmark(name: String, valuesPerIteration: Long, iters: Int = 5) {
    +  val benchmarks = mutable.ArrayBuffer.empty[Benchmark.Case]
    +
    +  def addCase(name: String, f: Int => Unit): Unit = {
    --- End diff --
    
    same thing here, using multi parameter list makes the code look nicer


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12640][SQL] Add simple benchmarking uti...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10589#discussion_r48815123
  
    --- Diff: core/src/test/scala/org/apache/spark/Benchmark.scala ---
    @@ -0,0 +1,102 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark
    +
    +import scala.collection.mutable
    +
    +import org.apache.commons.lang3.SystemUtils
    +import org.apache.spark.util.Utils
    +
    +/**
    + * Utility class to benchmark components. An example of how to use this is:
    + *  val benchmark = new Benchmark("My Benchmark", valuesPerIteration)
    + *   benchmark.addCase("V1", <function>")
    + *   benchmark.addCase("V2", <function>")
    + *   benchmark.run
    + * This will output the average time to run each function and the rate of each function.
    + *
    + * The benchmark function takes one argument that is the iteration that's being run
    + */
    +class Benchmark(name: String, valuesPerIteration: Long, iters: Int = 5) {
    +  val benchmarks = mutable.ArrayBuffer.empty[Benchmark.Case]
    +
    +  def addCase(name: String, f: Int => Unit): Unit = {
    +    benchmarks += Benchmark.Case(name, f)
    +  }
    +
    +  /**
    +   * Runs the benchmark and outputs the results to stdout. This should be copied and added as
    +   * a comment with the benchmark. Although the results vary from machine to machine, it should
    +   * provide some baseline.
    +   */
    +  def run(): Unit = {
    +    require(benchmarks.nonEmpty)
    +    val results = benchmarks.map { c =>
    +      Benchmark.measure(valuesPerIteration, c.fn, iters)
    +    }
    +    val firstRate = results.head.avgRate
    +    // scalastyle:off
    +    // The results are going to be processor specific so it is useful to include that.
    +    println(Benchmark.getProcessorName())
    +    printf("%-24s %16s %16s %14s\n", name + ":", "Avg Time(ms)", "Avg Rate(M/s)", "Relative Rate")
    +    println("-------------------------------------------------------------------------")
    +    results.zip(benchmarks).foreach { r =>
    +      printf("%-24s %16s %16s %14s\n", r._2.name, r._1.avgMs.toString, "%10.2f" format r._1.avgRate,
    +        "%6.2f X" format (r._1.avgRate / firstRate))
    +    }
    +    println
    +    // scalastyle:on
    +  }
    +}
    +
    +object Benchmark {
    +  case class Case(name: String, fn: Int => Unit)
    +  case class Result(avgMs: Double, avgRate: Double)
    +
    +  /**
    +   * This should return a user helpful processor information. Getting at this depends on the OS.
    +   * This should return something like "Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz"
    +   */
    +  def getProcessorName(): String = {
    +    if (SystemUtils.IS_OS_MAC_OSX) {
    +      Utils.executeAndGetOutput(Seq("/usr/sbin/sysctl", "-n", "machdep.cpu.brand_string"))
    +    } else if (SystemUtils.IS_OS_LINUX) {
    +      Utils.executeAndGetOutput(Seq("/usr/bin/grep", "-m", "1", "\"model name\"", "/proc/cpuinfo"))
    +    } else {
    +      System.getenv("PROCESSOR_IDENTIFIER")
    +    }
    +  }
    +
    +  /**
    +   * Runs a single function `f` for iters, returning the average time the function took and
    +   * the rate of the function.
    +   */
    +  def measure(num: Long, f: Int => Unit, iters: Int): Result = {
    --- End diff --
    
    you can make this slightly nicer to call with multi parameter list, e.g.
    
    ```
    def measure(num: Long, iters: Int)(f: Int => Unit)
    ```
    
    Then you can call it with
    ```
    measure(10, 20) { iter =>
     ...
    }
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12640][SQL] Add simple benchmarking uti...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/10589#issuecomment-168932593
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12640][SQL] Add simple benchmarking uti...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/10589#issuecomment-169512422
  
    Build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org