You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by kayano-shoko <gi...@git.apache.org> on 2015/12/19 03:02:13 UTC
[GitHub] spark pull request: Branch 1.2
GitHub user kayano-shoko opened a pull request:
https://github.com/apache/spark/pull/10389
Branch 1.2
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/spark branch-1.2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10389.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10389
----
commit e7f9dd5cd10d18d0b712916750ac1643df169b4f
Author: Ernest <ea...@gmail.com>
Date: 2014-12-18T23:42:26Z
[SPARK-4880] remove spark.locality.wait in Analytics
spark.locality.wait set to 100000 in examples/graphx/Analytics.scala.
Should be left to the user.
Author: Ernest <ea...@gmail.com>
Closes #3730 from Earne/SPARK-4880 and squashes the following commits:
d79ed04 [Ernest] remove spark.locality.wait in Analytics
(cherry picked from commit a7ed6f3cc537f57de87d28e8466ca88fbfff53b5)
Signed-off-by: Reynold Xin <rx...@databricks.com>
commit 61c9b89d84c868e9ecf5cffb9718c46753c9996e
Author: Madhu Siddalingaiah <ma...@madhu.com>
Date: 2014-12-19T00:00:53Z
[SPARK-4884]: Improve Partition docs
Rewording was based on this discussion: http://apache-spark-developers-list.1001551.n3.nabble.com/RDD-data-flow-td9804.html
This is the associated JIRA ticket: https://issues.apache.org/jira/browse/SPARK-4884
Author: Madhu Siddalingaiah <ma...@madhu.com>
Closes #3722 from msiddalingaiah/master and squashes the following commits:
79e679f [Madhu Siddalingaiah] [DOC]: improve documentation
51d14b9 [Madhu Siddalingaiah] Merge remote-tracking branch 'upstream/master'
38faca4 [Madhu Siddalingaiah] Merge remote-tracking branch 'upstream/master'
cbccbfe [Madhu Siddalingaiah] Documentation: replace <b> with <code> (again)
332f7a2 [Madhu Siddalingaiah] Documentation: replace <b> with <code>
cd2b05a [Madhu Siddalingaiah] Merge remote-tracking branch 'upstream/master'
0fc12d7 [Madhu Siddalingaiah] Documentation: add description for repartitionAndSortWithinPartitions
(cherry picked from commit d5a596d4188bfa85ff49ee85039f54255c19a4de)
Signed-off-by: Josh Rosen <jo...@databricks.com>
commit 075b399c59b508251f4fb259e7b0c13b79ff5883
Author: Aaron Davidson <aa...@databricks.com>
Date: 2014-12-19T00:43:16Z
[SPARK-4837] NettyBlockTransferService should use spark.blockManager.port config
This is used in NioBlockTransferService here:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/network/nio/NioBlockTransferService.scala#L66
Author: Aaron Davidson <aa...@databricks.com>
Closes #3688 from aarondav/SPARK-4837 and squashes the following commits:
ebd2007 [Aaron Davidson] [SPARK-4837] NettyBlockTransferService should use spark.blockManager.port config
(cherry picked from commit 105293a7d06b26e7b179a0447eb802074ee9c218)
Signed-off-by: Josh Rosen <jo...@databricks.com>
commit ca37639aa1b537d0f9b56bf1362bf293635e235c
Author: Andrew Or <an...@databricks.com>
Date: 2014-12-19T01:37:42Z
[SPARK-4754] Refactor SparkContext into ExecutorAllocationClient
This is such that the `ExecutorAllocationManager` does not take in the `SparkContext` with all of its dependencies as an argument. This prevents future developers of this class to tie down this class further with the `SparkContext`, which has really become quite a monstrous object.
cc'ing pwendell who originally suggested this, and JoshRosen who may have thoughts about the trait mix-in style of `SparkContext`.
Author: Andrew Or <an...@databricks.com>
Closes #3614 from andrewor14/dynamic-allocation-sc and squashes the following commits:
187070d [Andrew Or] Merge branch 'master' of github.com:apache/spark into dynamic-allocation-sc
59baf6c [Andrew Or] Merge branch 'master' of github.com:apache/spark into dynamic-allocation-sc
347a348 [Andrew Or] Refactor SparkContext into ExecutorAllocationClient
(cherry picked from commit 9804a759b68f56eceb8a2f4ea90f76a92b5f9f67)
Signed-off-by: Andrew Or <an...@databricks.com>
Conflicts:
core/src/main/scala/org/apache/spark/SparkContext.scala
commit fd7bb9d9728fa2b4fc6f26ae6a31cfa60d560ad4
Author: Sandy Ryza <sa...@cloudera.com>
Date: 2014-12-19T06:40:44Z
SPARK-3428. TaskMetrics for running tasks is missing GC time metrics
Author: Sandy Ryza <sa...@cloudera.com>
Closes #3684 from sryza/sandy-spark-3428 and squashes the following commits:
cb827fe [Sandy Ryza] SPARK-3428. TaskMetrics for running tasks is missing GC time metrics
(cherry picked from commit 283263ffaa941e7e9ba147cf0ad377d9202d3761)
Signed-off-by: Josh Rosen <jo...@databricks.com>
commit 6aa88cc04b475320bb070019f48277e1de270de3
Author: Ryan Williams <ry...@gmail.com>
Date: 2014-12-19T21:56:04Z
[SPARK-4889] update history server example cmds
Author: Ryan Williams <ry...@gmail.com>
Closes #3736 from ryan-williams/hist and squashes the following commits:
421d8ff [Ryan Williams] add another random typo fix
76d6a4c [Ryan Williams] remove hdfs example
a2d0f82 [Ryan Williams] code review feedback
9ca7629 [Ryan Williams] [SPARK-4889] update history server example cmds
(cherry picked from commit cdb2c645ab769a8678dd81cff44a809fcfa4420b)
Signed-off-by: Andrew Or <an...@databricks.com>
commit f930fe893959959bd1e69f1e08df49dd5d6dab17
Author: Ryan Williams <ry...@gmail.com>
Date: 2014-12-19T23:24:41Z
[SPARK-4896] don’t redundantly overwrite executor JAR deps
Author: Ryan Williams <ry...@gmail.com>
Closes #2848 from ryan-williams/fetch-file and squashes the following commits:
c14daff [Ryan Williams] Fix copy that was changed to a move inadvertently
8e39c16 [Ryan Williams] code review feedback
788ed41 [Ryan Williams] don’t redundantly overwrite executor JAR deps
(cherry picked from commit 7981f969762e77f1752ef8f86c546d4fc32a1a4f)
Signed-off-by: Josh Rosen <jo...@databricks.com>
commit 4da1039840182e8e8bc836b89cda7b77fe7356d9
Author: Eran Medan <eh...@gmail.com>
Date: 2014-12-20T02:29:36Z
change signature of example to match released code
the signature of registerKryoClasses is actually of Array[Class[_]] not Seq
Author: Eran Medan <eh...@gmail.com>
Closes #3747 from eranation/patch-1 and squashes the following commits:
ee9885d [Eran Medan] change signature of example to match released code
commit a1a1361a9398a837129b87848e2cc3ae5acf6cf7
Author: Kanwaljit Singh <ka...@guavus.com>
Date: 2014-12-20T03:25:39Z
SPARK-2641: Passing num executors to spark arguments from properties file
Since we can set spark executor memory and executor cores using property file, we must also be allowed to set the executor instances.
Author: Kanwaljit Singh <ka...@guavus.com>
Closes #1657 from kjsingh/branch-1.0 and squashes the following commits:
d8a5a12 [Kanwaljit Singh] SPARK-2641: Fixing how spark arguments are loaded from properties file for num executors
Conflicts:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
commit 96d5b00ac27a203a5fe973f4e2e3031e602149ba
Author: Andrew Or <an...@databricks.com>
Date: 2014-12-20T03:36:20Z
[SPARK-4140] Document dynamic allocation
Once the external shuffle service is also documented, the dynamic allocation section will link to it. Let me know if the whole dynamic allocation should be moved to its separate page; I personally think the organization might be cleaner that way.
This patch builds on top of oza's work in #3689.
aarondav pwendell
Author: Andrew Or <an...@databricks.com>
Author: Tsuyoshi Ozawa <oz...@gmail.com>
Closes #3731 from andrewor14/document-dynamic-allocation and squashes the following commits:
1281447 [Andrew Or] Address a few comments
b9843f2 [Andrew Or] Document the configs as well
246fb44 [Andrew Or] Merge branch 'SPARK-4839' of github.com:oza/spark into document-dynamic-allocation
8c64004 [Andrew Or] Add documentation for dynamic allocation (without configs)
6827b56 [Tsuyoshi Ozawa] Fixing a documentation of spark.dynamicAllocation.enabled.
53cff58 [Tsuyoshi Ozawa] Adding a documentation about dynamic resource allocation.
(cherry picked from commit 15c03e1e0efac29855f32984da7c6b0321f0e37a)
Signed-off-by: Andrew Or <an...@databricks.com>
commit 4346a2ba1a1c62732c5e313b464019fe6259a4a8
Author: huangzhaowei <ca...@gmail.com>
Date: 2014-12-20T07:32:56Z
[Minor] Build Failed: value defaultProperties not found
Mvn Build Failed: value defaultProperties not found .Maybe related to this pr:
https://github.com/apache/spark/commit/1d648123a77bbcd9b7a34cc0d66c14fa85edfecd
andrewor14 can you look at this problem?
Author: huangzhaowei <ca...@gmail.com>
Closes #3749 from SaintBacchus/Mvn-Build-Fail and squashes the following commits:
8e2917c [huangzhaowei] Build Failed: value defaultProperties not found
(cherry picked from commit a764960b3b6d842eef7fa4777c8fa99d3f60fa1e)
Signed-off-by: Josh Rosen <jo...@databricks.com>
commit 665653d240c0563f23ad6922b217ff6f0d548874
Author: zsxwing <zs...@gmail.com>
Date: 2014-12-22T06:10:19Z
[SPARK-2075][Core] Make the compiler generate same bytes code for Hadoop 1.+ and Hadoop 2.+
`NullWritable` is a `Comparable` rather than `Comparable[NullWritable]` in Hadoop 1.+, so the compiler cannot find an implicit Ordering for it. It will generate different anonymous classes for `saveAsTextFile` in Hadoop 1.+ and Hadoop 2.+. Therefore, here we provide an Ordering for NullWritable so that the compiler will generate same codes.
I used the following commands to confirm the generated byte codes are some.
```
mvn -Dhadoop.version=1.2.1 -DskipTests clean package -pl core -am
javap -private -c -classpath core/target/scala-2.10/classes org.apache.spark.rdd.RDD > ~/hadoop1.txt
mvn -Pyarn -Phadoop-2.2 -Dhadoop.version=2.2.0 -DskipTests clean package -pl core -am
javap -private -c -classpath core/target/scala-2.10/classes org.apache.spark.rdd.RDD > ~/hadoop2.txt
diff ~/hadoop1.txt ~/hadoop2.txt
```
However, the compiler will generate different codes for the classes which call methods of `JobContext/TaskAttemptContext`. `JobContext/TaskAttemptContext` is a class in Hadoop 1.+, and calling its method will use `invokevirtual`, while it's an interface in Hadoop 2.+, and will use `invokeinterface`.
To fix it, we can use reflection to call `JobContext/TaskAttemptContext.getConfiguration`.
Author: zsxwing <zs...@gmail.com>
Closes #3740 from zsxwing/SPARK-2075 and squashes the following commits:
39d9df2 [zsxwing] Fix the code style
e4ad8b5 [zsxwing] Use null for the implicit Ordering
734bac9 [zsxwing] Explicitly set the implicit parameters
ca03559 [zsxwing] Use reflection to access JobContext/TaskAttemptContext.getConfiguration
fa40db0 [zsxwing] Add an Ordering for NullWritable to make the compiler generate same byte codes for RDD
(cherry picked from commit 6ee6aa70b7d52408cc66bd1434cbeae3212e3f01)
Signed-off-by: Reynold Xin <rx...@databricks.com>
commit b896963728154ec51689fd3e975fc5033eb24d8c
Author: zsxwing <zs...@gmail.com>
Date: 2014-12-22T19:13:40Z
[SPARK-2075][Core] backport for branch-1.2
backport #3740 for branch-1.2
Author: zsxwing <zs...@gmail.com>
Closes #3758 from zsxwing/SPARK-2075-branch-1.2 and squashes the following commits:
b57d440 [zsxwing] SPARK-2075 backport for branch-1.2
commit 31d42c4d8ceef461db43cdb709a365f65598aa3c
Author: Tsuyoshi Ozawa <oz...@lab.ntt.co.jp>
Date: 2014-12-22T19:28:05Z
[SPARK-4915][YARN] Fix classname to be specified for external shuffle service.
Author: Tsuyoshi Ozawa <oz...@lab.ntt.co.jp>
Closes #3757 from oza/SPARK-4915 and squashes the following commits:
3b0d6d6 [Tsuyoshi Ozawa] Fix classname to be specified for external shuffle service.
(cherry picked from commit 96606f69b7cd88edea5d4b6427f598a91ccb4c33)
Signed-off-by: Andrew Or <an...@databricks.com>
commit 70e69ef7b5005e5b4aa5160c41829fabe95bd092
Author: zsxwing <zs...@gmail.com>
Date: 2014-12-22T20:11:36Z
[SPARK-4883][Shuffle] Add a name to the directoryCleaner thread
Author: zsxwing <zs...@gmail.com>
Closes #3734 from zsxwing/SPARK-4883 and squashes the following commits:
e6f2b61 [zsxwing] Fix the name
cc74727 [zsxwing] Add a name to the directoryCleaner thread
(cherry picked from commit 8773705fd4ab167bb06bee7f274b337560e3c293)
Signed-off-by: Andrew Or <an...@databricks.com>
commit c7396b5887afe1bbe344ffcf06ef266847c378ac
Author: carlmartin <ca...@gmail.com>
Date: 2014-12-22T20:13:53Z
[Minor] Improve some code in BroadcastTest for short
Using
val arr1 = (0 until num).toArray
instead of
val arr1 = new Array[Int](num)
for (i <- 0 until arr1.length) {
arr1(i) = i
}
for short.
Author: carlmartin <ca...@gmail.com>
Closes #3750 from SaintBacchus/BroadcastTest and squashes the following commits:
43adb70 [carlmartin] Improve some code in BroadcastTest for short
commit 4b2bdedface53263d004b5c0306f2f2483a9c4bb
Author: Aaron Davidson <aa...@databricks.com>
Date: 2014-12-22T21:09:22Z
[SPARK-4864] Add documentation to Netty-based configs
Author: Aaron Davidson <aa...@databricks.com>
Closes #3713 from aarondav/netty-configs and squashes the following commits:
8a8b373 [Aaron Davidson] Address Patrick's comments
3b1f84e [Aaron Davidson] [SPARK-4864] Add documentation to Netty-based configs
(cherry picked from commit fbca6b6ce293b1997b40abeb9ab77b8a969a5fc9)
Signed-off-by: Patrick Wendell <pw...@gmail.com>
commit a8a8e0e8752194d82b6c6e20cedbb3871b221916
Author: genmao.ygm <ge...@alibaba-inc.com>
Date: 2014-12-22T22:14:39Z
[SPARK-4920][UI]:current spark version in UI is not striking.
It is not convenient to see the Spark version. We can keep the same style with Spark website.
Author: genmao.ygm <ge...@alibaba-inc.com>
Closes #3763 from uncleGen/master-clean-141222 and squashes the following commits:
0dcb9a9 [genmao.ygm] [SPARK-4920][UI]:current spark version in UI is not striking.
(cherry picked from commit de9d7d2b5b6d80963505571700e83779fd98f850)
Signed-off-by: Andrew Or <an...@databricks.com>
commit 58e37028a43883877a8f15bc2e0d3011a9ebd704
Author: zsxwing <zs...@gmail.com>
Date: 2014-12-22T22:26:28Z
[SPARK-4818][Core] Add 'iterator' to reduce memory consumed by join
In Scala, `map` and `flatMap` of `Iterable` will copy the contents of `Iterable` to a new `Seq`. Such as,
```Scala
val iterable = Seq(1, 2, 3).map(v => {
println(v)
v
})
println("Iterable map done")
val iterator = Seq(1, 2, 3).iterator.map(v => {
println(v)
v
})
println("Iterator map done")
```
outputed
```
1
2
3
Iterable map done
Iterator map done
```
So we should use 'iterator' to reduce memory consumed by join.
Found by Johannes Simon in http://mail-archives.apache.org/mod_mbox/spark-user/201412.mbox/%3C5BE70814-9D03-4F61-AE2C-0D63F2DE4446%40mail.de%3E
Author: zsxwing <zs...@gmail.com>
Closes #3671 from zsxwing/SPARK-4824 and squashes the following commits:
48ee7b9 [zsxwing] Remove the explicit types
95d59d6 [zsxwing] Add 'iterator' to reduce memory consumed by join
(cherry picked from commit c233ab3d8d75a33495298964fe73dbf7dd8fe305)
Signed-off-by: Josh Rosen <jo...@databricks.com>
commit f86fe089719db84ae3a53bc79ef663a087e3db29
Author: Nicholas Chammas <ni...@gmail.com>
Date: 2014-12-23T06:54:32Z
[Docs] Minor typo fixes
Author: Nicholas Chammas <ni...@gmail.com>
Closes #3772 from nchammas/patch-1 and squashes the following commits:
b7d9083 [Nicholas Chammas] [Docs] Minor typo fixes
(cherry picked from commit 0e532ccb2b282ea5f7b818e67d521dc44d94c951)
Signed-off-by: Patrick Wendell <pw...@gmail.com>
commit 9fb86b80a20807a0c797050aebb098f94a12e5ea
Author: zsxwing <zs...@gmail.com>
Date: 2014-12-23T19:18:06Z
[SPARK-4931][Yarn][Docs] Fix the format of running-on-yarn.md
Currently, the format about log4j in running-on-yarn.md is a bit messy.
Author: zsxwing <zs...@gmail.com>
Closes #3774 from zsxwing/SPARK-4931 and squashes the following commits:
4a5f853 [zsxwing] Fix the format of running-on-yarn.md
(cherry picked from commit 2d215aebaad4b4596354f1543d997d06ef340bd5)
Signed-off-by: Josh Rosen <jo...@databricks.com>
commit ec11ffddbb4074f42745ebd76cd8f32abeb4b3ce
Author: Marcelo Vanzin <va...@cloudera.com>
Date: 2014-12-23T20:02:08Z
[SPARK-4834] [standalone] Clean up application files after app finishes.
Commit 7aacb7bfa added support for sharing downloaded files among multiple
executors of the same app. That works great in Yarn, since the app's directory
is cleaned up after the app is done.
But Spark standalone mode didn't do that, so the lock/cache files created
by that change were left around and could eventually fill up the disk hosting
/tmp.
To solve that, create app-specific directories under the local dirs when
launching executors. Multiple executors launched by the same Worker will
use the same app directories, so they should be able to share the downloaded
files. When the application finishes, a new message is sent to all workers
telling them the application has finished; once that message has been received,
and all executors registered for the application shut down, then those
directories will be cleaned up by the Worker.
Note: Unit testing this is hard (if even possible), since local-cluster mode
doesn't seem to leave the Master/Worker daemons running long enough after
`sc.stop()` is called for the clean up protocol to take effect.
Author: Marcelo Vanzin <va...@cloudera.com>
Closes #3705 from vanzin/SPARK-4834 and squashes the following commits:
b430534 [Marcelo Vanzin] Remove seemingly unnecessary synchronization.
50eb4b9 [Marcelo Vanzin] Review feedback.
c0e5ea5 [Marcelo Vanzin] [SPARK-4834] [standalone] Clean up application files after app finishes.
(cherry picked from commit dd155369a04d7dfbf6a5745cbb243e22218367dc)
Signed-off-by: Josh Rosen <jo...@databricks.com>
commit e74ce14e8a25d2fc6c23feeedceb02fba5c2d66c
Author: Takeshi Yamamuro <li...@gmail.com>
Date: 2014-12-23T20:39:41Z
[SPARK-4932] Add help comments in Analytics
Trivial modifications for usability.
Author: Takeshi Yamamuro <li...@gmail.com>
Closes #3775 from maropu/AddHelpCommentInAnalytics and squashes the following commits:
fbea8f5 [Takeshi Yamamuro] Add help comments in Analytics
(cherry picked from commit 9c251c555f5ee527143d0cdb9e6c3cb7530fc8f8)
Signed-off-by: Josh Rosen <jo...@databricks.com>
commit 7b5ba85df342159f4b62f507bf1daa4dafe59059
Author: Cheng Lian <li...@databricks.com>
Date: 2014-12-23T20:54:20Z
[SPARK-4914][Build] Cleans lib_managed before compiling with Hive 0.13.1
This PR tries to fix the Hive tests failure encountered in PR #3157 by cleaning `lib_managed` before building assembly jar against Hive 0.13.1 in `dev/run-tests`. Otherwise two sets of datanucleus jars would be left in `lib_managed` and may mess up class paths while executing Hive test suites. Please refer to [this thread] [1] for details. A clean build would be even safer, but we only clean `lib_managed` here to save build time.
This PR also takes the chance to clean up some minor typos and formatting issues in the comments.
[1]: https://github.com/apache/spark/pull/3157#issuecomment-67656488
<!-- Reviewable:start -->
[<img src="https://reviewable.io/review_button.png" height=40 alt="Review on Reviewable"/>](https://reviewable.io/reviews/apache/spark/3756)
<!-- Reviewable:end -->
Author: Cheng Lian <li...@databricks.com>
Closes #3756 from liancheng/clean-lib-managed and squashes the following commits:
e2bd21d [Cheng Lian] Adds lib_managed to clean set
c9f2f3e [Cheng Lian] Cleans lib_managed before compiling with Hive 0.13.1
(cherry picked from commit 395b771feed3fc90e5053defbe86dbd673fde582)
Signed-off-by: Josh Rosen <jo...@databricks.com>
commit 6a46cc3c83592a8d8e2ae4e6b44c26e39df1e340
Author: Andrew Or <an...@databricks.com>
Date: 2014-12-23T22:28:36Z
[SPARK-4730][YARN] Warn against deprecated YARN settings
See https://issues.apache.org/jira/browse/SPARK-4730.
Author: Andrew Or <an...@databricks.com>
Closes #3590 from andrewor14/yarn-settings and squashes the following commits:
36e0753 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-settings
dcd1316 [Andrew Or] Warn against deprecated YARN settings
(cherry picked from commit 27c5399f4dd542e36ea579956b8cb0613de25c6d)
Signed-off-by: Josh Rosen <jo...@databricks.com>
commit 01adf45a9b2e0264ee4571dd51c04a57126b666a
Author: Ilayaperumal Gopinathan <ig...@pivotal.io>
Date: 2014-12-23T23:14:54Z
[SPARK-4802] [streaming] Remove receiverInfo once receiver is de-registered
Once the streaming receiver is de-registered at executor, the `ReceiverTrackerActor` needs to
remove the corresponding reveiverInfo from the `receiverInfo` map at `ReceiverTracker`.
Author: Ilayaperumal Gopinathan <ig...@pivotal.io>
Closes #3647 from ilayaperumalg/receiverInfo-RTracker and squashes the following commits:
6eb97d5 [Ilayaperumal Gopinathan] Polishing based on the review
3640c86 [Ilayaperumal Gopinathan] Remove receiverInfo once receiver is de-registered
(cherry picked from commit 10d69e9cbfdabe95d0e513176d5347d7b59da0ee)
Signed-off-by: Tathagata Das <ta...@gmail.com>
commit aa78c23ac809a8014ad47a9047f408c5a5714120
Author: jerryshao <sa...@intel.com>
Date: 2014-12-23T23:45:53Z
[SPARK-4671][Streaming]Do not replicate streaming block when WAL is enabled
Currently streaming block will be replicated when specific storage level is set, since WAL is already fault tolerant, so replication is needless and will hurt the throughput of streaming application.
Hi tdas , as per discussed about this issue, I fixed with this implementation, I'm not is this the way you want, would you mind taking a look at it? Thanks a lot.
Author: jerryshao <sa...@intel.com>
Closes #3534 from jerryshao/SPARK-4671 and squashes the following commits:
500b456 [jerryshao] Do not replicate streaming block when WAL is enabled
(cherry picked from commit 3f5f4cc4e7b3bc458e0579d247a0652dca365853)
Signed-off-by: Tathagata Das <ta...@gmail.com>
commit 1a4e2ba7369b9eb1dab6cb66cdb7b21129e7faf1
Author: Marcelo Vanzin <va...@cloudera.com>
Date: 2014-12-24T00:02:59Z
[SPARK-4606] Send EOF to child JVM when there's no more data to read.
Author: Marcelo Vanzin <va...@cloudera.com>
Closes #3460 from vanzin/SPARK-4606 and squashes the following commits:
031207d [Marcelo Vanzin] [SPARK-4606] Send EOF to child JVM when there's no more data to read.
(cherry picked from commit 7e2deb71c4239564631b19c748e95c3d1aa1c77d)
Signed-off-by: Josh Rosen <jo...@databricks.com>
commit 17d6f547bf3612890f79e331b427e286c5fca730
Author: zsxwing <zs...@gmail.com>
Date: 2014-12-25T03:49:41Z
[SPARK-4873][Streaming] Use `Future.zip` instead of `Future.flatMap`(for-loop) in WriteAheadLogBasedBlockHandler
Use `Future.zip` instead of `Future.flatMap`(for-loop). `zip` implies these two Futures will run concurrently, while `flatMap` usually means one Future depends on the other one.
Author: zsxwing <zs...@gmail.com>
Closes #3721 from zsxwing/SPARK-4873 and squashes the following commits:
46a2cd9 [zsxwing] Use Future.zip instead of Future.flatMap(for-loop)
(cherry picked from commit b4d0db80a0bfba7f1e045d4edb9357b4b2c0a557)
Signed-off-by: Tathagata Das <ta...@gmail.com>
commit 475ab6ec71992aa214284ea3df547163ad27c90c
Author: Denny Lee <de...@gmail.com>
Date: 2014-12-25T22:05:55Z
Fix "Building Spark With Maven" link in README.md
Corrected link to the Building Spark with Maven page from its original (http://spark.apache.org/docs/latest/building-with-maven.html) to the current page (http://spark.apache.org/docs/latest/building-spark.html)
Author: Denny Lee <de...@gmail.com>
Closes #3802 from dennyglee/patch-1 and squashes the following commits:
15f601a [Denny Lee] Update README.md
(cherry picked from commit 08b18c7eb790c65670778eab8a6e32486c5f76e9)
Signed-off-by: Josh Rosen <jo...@databricks.com>
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Branch 1.2
Posted by kayano-shoko <gi...@git.apache.org>.
Github user kayano-shoko closed the pull request at:
https://github.com/apache/spark/pull/10389
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org