You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by f7753 <gi...@git.apache.org> on 2015/11/11 04:47:00 UTC
[GitHub] spark pull request: Branch 1.4
GitHub user f7753 opened a pull request:
https://github.com/apache/spark/pull/9616
Branch 1.4
Has any one meet the following situation:
We observed that during the shuffle period of a sort job, the tasks distributed among several nodes are running after by after, not concurrently. e.g.
task 0~15 belongs to node0
task 16~31 belongs to node1
task 32~47 belongs to node2, they finished in this manner: 16~31(the first wave), 0 (the second wave), 32~33(the third wave) …
Why not 0,16,32, then 1, 17, 33, then…
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/spark branch-1.4
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9616.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9616
----
commit b9c046f6d7b16e56f3f87de95fc8f862ff2c73e5
Author: Reynold Xin <rx...@databricks.com>
Date: 2015-06-07T17:52:02Z
[SPARK-8004][SQL] Quote identifier in JDBC data source.
This is a follow-up patch to #6577 to replace columnEnclosing to quoteIdentifier.
I also did some minor cleanup to the JdbcDialect file.
Author: Reynold Xin <rx...@databricks.com>
Closes #6689 from rxin/jdbc-quote and squashes the following commits:
bad365f [Reynold Xin] Fixed test compilation...
e39e14e [Reynold Xin] Fixed compilation.
db9a8e0 [Reynold Xin] [SPARK-8004][SQL] Quote identifier in JDBC data source.
(cherry picked from commit d6d601a07b17069d41eb4114bd5f7ab2c106720d)
Signed-off-by: Reynold Xin <rx...@databricks.com>
commit 58bfdd62122b9b64ddeb405c565dbd30fc0bf997
Author: Daoyuan Wang <da...@intel.com>
Date: 2015-06-08T08:07:50Z
[SPARK-4761] [DOC] [SQL] kryo default setting in SQL Thrift server
this is a follow up of #3621
/cc liancheng pwendell
Author: Daoyuan Wang <da...@intel.com>
Closes #6639 from adrian-wang/kryodoc and squashes the following commits:
3c4b1cf [Daoyuan Wang] [DOC] kryo default setting in SQL Thrift server
(cherry picked from commit 10fc2f6f51819f263eec941bdc1db22c554f9118)
Signed-off-by: Reynold Xin <rx...@databricks.com>
commit a3afc2cbabf3752ef82973e3ca8214f2ed1dde89
Author: linweizhong <li...@huawei.com>
Date: 2015-06-08T08:34:16Z
[SPARK-7705] [YARN] Cleanup of .sparkStaging directory fails if application is killed
As I have tested, if we cancel or kill the app then the final status may be undefined, killed or succeeded, so clean up staging directory when appMaster exit at any final application status.
Author: linweizhong <li...@huawei.com>
Closes #6409 from Sephiroth-Lin/SPARK-7705 and squashes the following commits:
3a5a0a5 [linweizhong] Update
83dc274 [linweizhong] Update
923d44d [linweizhong] Update
0dd7c2d [linweizhong] Update
b76a102 [linweizhong] Update code style
7846b69 [linweizhong] Update
bd6cf0d [linweizhong] Refactor
aed9f18 [linweizhong] Clean up stagingDir when launch app on yarn
95595c3 [linweizhong] Cleanup of .sparkStaging directory when AppMaster exit at any final application status
(cherry picked from commit eacd4a929bf5d697c33b1b705dcf958651cd20f4)
Signed-off-by: Sean Owen <so...@cloudera.com>
commit 69197c3e382abd477e6eeb87ffbda69bfa68fa14
Author: Cheng Lian <li...@databricks.com>
Date: 2015-06-08T18:35:30Z
[SPARK-8121] [SQL] Fixes InsertIntoHadoopFsRelation job initialization for Hadoop 1.x (branch 1.4 backport based on https://github.com/apache/spark/pull/6669)
commit 99c2a57348ee4fc0f6801dfb91fd5cf5a7dfef12
Author: Marcelo Vanzin <va...@cloudera.com>
Date: 2015-06-08T14:37:28Z
[SPARK-8126] [BUILD] Use custom temp directory during build.
Even with all the efforts to cleanup the temp directories created by
unit tests, Spark leaves a lot of garbage in /tmp after a test run.
This change overrides java.io.tmpdir to place those files under the
build directory instead.
After an sbt full unit test run, I was left with > 400 MB of temp
files. Since they're now under the build dir, it's much easier to
clean them up.
Also make a slight change to a unit test to make it not pollute the
source directory with test data.
Author: Marcelo Vanzin <va...@cloudera.com>
Closes #6674 from vanzin/SPARK-8126 and squashes the following commits:
0f8ad41 [Marcelo Vanzin] Make sure tmp dir exists when tests run.
643e916 [Marcelo Vanzin] [MINOR] [BUILD] Use custom temp directory during build.
commit e9a83723614ead03159b620bf391f38de698423d
Author: Andrew Or <an...@databricks.com>
Date: 2015-06-09T01:09:21Z
[SPARK-8162] [HOTFIX] Fix NPE in spark-shell
This was caused by this commit: f271347
This patch does not attempt to fix the root cause of why the `VisibleForTesting` annotation causes a NPE in the shell. We should find a way to fix that separately.
Author: Andrew Or <an...@databricks.com>
Closes #6711 from andrewor14/fix-spark-shell and squashes the following commits:
bf62ecc [Andrew Or] Prevent NPE in spark-shell
commit 0a9383decbe83034b93ddc1805dfdb2052e3c27b
Author: hqzizania <qi...@intel.com>
Date: 2015-06-09T04:40:12Z
[SPARK-6820] [SPARKR] Convert NAs to null type in SparkR DataFrames
Author: hqzizania <qi...@intel.com>
Closes #6190 from hqzizania/R and squashes the following commits:
1641f9e [hqzizania] fixes and add test units
bb3411a [hqzizania] Convert NAs to null type in SparkR DataFrames
(cherry picked from commit a5c52c1a3488b69bec19e460d2d1fdb0c9ada58d)
Signed-off-by: Shivaram Venkataraman <sh...@cs.berkeley.edu>
commit a7b7a194ac6d3b38b0f3da2b6527bbe3e2a050f1
Author: FavioVazquez <fa...@gmail.com>
Date: 2015-06-09T14:02:18Z
[SPARK-8274] [DOCUMENTATION-MLLIB] Fix wrong URLs in MLlib Frequent Pattern Mining Documentation
There is a mistake in the URLs of the Scala section of FP-Growth in the MLlib Frequent Pattern Mining documentation. The URL points to https://spark.apache.org/docs/latest/api/java/org/apache/spark/mllib/fpm/FPGrowth.html which is the Java's API, the link should point to the Scala API https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.mllib.fpm.FPGrowth
There's another mistake in the FP-GrowthModel in the same section, the link points, again, to the Java's API https://spark.apache.org/docs/latest/api/java/org/apache/spark/mllib/fpm/FPGrowthModel.html, the link should point to the Scala API https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.mllib.fpm.FPGrowthModel
Author: FavioVazquez <fa...@gmail.com>
Closes #6722 from FavioVazquez/fix-wrog-urls-mllib-fpgrowth and squashes the following commits:
e1ca54d [FavioVazquez] - Fixed wrong URLs in MLlib Frequent Pattern Mining, FP-Growth Scala section
ad882a3 [FavioVazquez] Merge remote-tracking branch 'upstream/master'
f27a20b [FavioVazquez] Merge remote-tracking branch 'upstream/master'
9af7074 [FavioVazquez] Merge remote-tracking branch 'upstream/master'
edab1ef [FavioVazquez] Merge remote-tracking branch 'upstream/master'
b2e2f8c [FavioVazquez] Merge remote-tracking branch 'upstream/master'
(cherry picked from commit 490d5a72ec1e5105f030fd7110acf62534e05f5a)
Signed-off-by: Sean Owen <so...@cloudera.com>
commit 1175cfe088c812a2a1b7334ebb7f5113663e3f18
Author: Andrew Or <an...@databricks.com>
Date: 2015-06-09T22:44:02Z
[MINOR] [UI] DAG visualization: trim whitespace from input
Just as a safeguard against DOM rewriting.
Author: Andrew Or <an...@databricks.com>
Closes #6732 from andrewor14/dag-viz-trim and squashes the following commits:
7e9bacb [Andrew Or] [MINOR] [UI] DAG visualization: trim whitespace from input
(cherry picked from commit 0d5892dc723d203e7d892d3beacbaa97aedb1a24)
Signed-off-by: Andrew Or <an...@databricks.com>
commit a0a7f2f92108d73c7db63c718b769d903061bf66
Author: Patrick Wendell <pa...@databricks.com>
Date: 2015-06-09T23:14:21Z
[SPARK-6511] [DOCUMENTATION] Explain how to use Hadoop provided builds
This provides preliminary documentation pointing out how to use the
Hadoop free builds. I am hoping over time this list can grow to
include most of the popular Hadoop distributions.
Getting more people using these builds will help us long term reduce
the number of binaries we build.
Author: Patrick Wendell <pa...@databricks.com>
Closes #6729 from pwendell/hadoop-provided and squashes the following commits:
1113b76 [Patrick Wendell] [SPARK-6511] [Documentation] Explain how to use Hadoop provided builds
(cherry picked from commit 6e4fb0c9e8f03cf068c422777cfce82a89e8e738)
Signed-off-by: Patrick Wendell <pa...@databricks.com>
commit 7b88e6a1e3b8f79cb41842bc21893760dc4b74e6
Author: Cheng Lian <li...@databricks.com>
Date: 2015-06-10T18:48:14Z
[SQL] [MINOR] Fixes a minor Java example error in SQL programming guide
Author: Cheng Lian <li...@databricks.com>
Closes #6749 from liancheng/java-sample-fix and squashes the following commits:
5b44585 [Cheng Lian] Fixes a minor Java example error in SQL programming guide
(cherry picked from commit 8f7308f9c49805b9486aaae5f60e4481e8ba24e8)
Signed-off-by: Reynold Xin <rx...@databricks.com>
commit 28e8a6ea65fd08ab9cefc4d179d5c66ffefd3eb4
Author: Hossein <ho...@databricks.com>
Date: 2015-06-10T20:18:48Z
[SPARK-8282] [SPARKR] Make number of threads used in RBackend configurable
Read number of threads for RBackend from configuration.
[SPARK-8282] #comment Linking with JIRA
Author: Hossein <ho...@databricks.com>
Closes #6730 from falaki/SPARK-8282 and squashes the following commits:
33b3d98 [Hossein] Documented new config parameter
70f2a9c [Hossein] Fixing import
ec44225 [Hossein] Read number of threads for RBackend from configuration
(cherry picked from commit 30ebf1a233295539c2455bd838bae7315711e1e2)
Signed-off-by: Andrew Or <an...@databricks.com>
commit 568d1d51d695bea4389f4470cd98707f3049885a
Author: Adam Roberts <ar...@uk.ibm.com>
Date: 2015-06-10T20:21:01Z
[SPARK-7756] CORE RDDOperationScope fix for IBM Java
IBM Java has an extra method when we do getStackTrace(): this is "getStackTraceImpl", a native method. This causes two tests to fail within "DStreamScopeSuite" when running with IBM Java. Instead of "map" or "filter" being the method names found, "getStackTrace" is returned. This commit addresses such an issue by using dropWhile. Given that our current method is withScope, we look for the next method that isn't ours: we don't care about methods that come before us in the stack trace: e.g. getStackTrace (regardless of how many levels this might go).
IBM:
java.lang.Thread.getStackTraceImpl(Native Method)
java.lang.Thread.getStackTrace(Thread.java:1117)
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:104)
Oracle:
PRINTING STACKTRACE!!!
java.lang.Thread.getStackTrace(Thread.java:1552)
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:106)
I've tested this with Oracle and IBM Java, no side effects for other tests introduced.
Author: Adam Roberts <ar...@uk.ibm.com>
Author: a-roberts <ar...@uk.ibm.com>
Closes #6740 from a-roberts/RDDScopeStackCrawlFix and squashes the following commits:
13ce390 [Adam Roberts] Ensure consistency with String equality checking
a4fc0e0 [a-roberts] Update RDDOperationScope.scala
(cherry picked from commit 19e30b48f3c6d0b72871d3e15b9564c1b2822700)
Signed-off-by: Andrew Or <an...@databricks.com>
commit 2846a357f32bfa129bc37f4d1cbe9e19caaf69c9
Author: WangTaoTheTonic <wa...@huawei.com>
Date: 2015-06-10T20:34:19Z
[SPARK-8273] Driver hangs up when yarn shutdown in client mode
In client mode, if yarn was shut down with spark application running, the application will hang up after several retries(default: 30) because the exception throwed by YarnClientImpl could not be caught by upper level, we should exit in case that user can not be aware that.
The exception we wanna catch is [here](https://github.com/apache/hadoop/blob/branch-2.7.0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryInvocationHandler.java#L122), and I try to fix it refer to [MR](https://github.com/apache/hadoop/blob/branch-2.7.0/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java#L320).
Author: WangTaoTheTonic <wa...@huawei.com>
Closes #6717 from WangTaoTheTonic/SPARK-8273 and squashes the following commits:
28752d6 [WangTaoTheTonic] catch the throwed exception
commit 59fc3f197247c6c8c40ea7479573af023c89d718
Author: Paavo <pp...@gmail.com>
Date: 2015-06-10T22:17:42Z
[SPARK-8200] [MLLIB] Check for empty RDDs in StreamingLinearAlgorithm
Test cases for both StreamingLinearRegression and StreamingLogisticRegression, and code fix.
Edit:
This contribution is my original work and I license the work to the project under the project's open source license.
Author: Paavo <pp...@gmail.com>
Closes #6713 from pparkkin/streamingmodel-empty-rdd and squashes the following commits:
ff5cd78 [Paavo] Update strings to use interpolation.
db234cf [Paavo] Use !rdd.isEmpty.
54ad89e [Paavo] Test case for empty stream.
393e36f [Paavo] Ignore empty RDDs.
0bfc365 [Paavo] Test case for empty stream.
(cherry picked from commit b928f543845ddd39e914a0e8f0b0205fd86100c5)
Signed-off-by: Sean Owen <so...@cloudera.com>
commit 5c05b5c0d25fc902bf95ed7b93ad7b5775631150
Author: navis.ryu <na...@apache.org>
Date: 2015-06-11T01:19:12Z
[SPARK-8285] [SQL] CombineSum should be calculated as unlimited decimal first
case cs CombineSum(expr) =>
val calcType = expr.dataType
expr.dataType match {
case DecimalType.Fixed(_, _) =>
DecimalType.Unlimited
case _ =>
expr.dataType
}
calcType is always expr.dataType. credits are all belong to IntelliJ
Author: navis.ryu <na...@apache.org>
Closes #6736 from navis/SPARK-8285 and squashes the following commits:
20382c1 [navis.ryu] [SPARK-8285] [SQL] CombineSum should be calculated as unlimited decimal first
(cherry picked from commit 6a47114bc297f0bce874e425feb1c24a5c26cef0)
Signed-off-by: Reynold Xin <rx...@databricks.com>
commit b313920abdc2c075ab038bf38beb62425df0908e
Author: Adam Roberts <ar...@uk.ibm.com>
Date: 2015-06-11T07:40:46Z
[SPARK-8289] Specify stack size for consistency with Java tests - resolves test failures
This change is a simple one and specifies a stack size of 4096k instead of the vendor default for Java tests (the defaults vary between Java vendors). This remedies test failures observed with JavaALSSuite with IBM and Oracle Java owing to a lower default size in comparison to the size with OpenJDK. 4096k is a suitable default where the tests pass with each Java vendor tested. The alternative is to reduce the number of iterations in the test (no observed failures with 5 iterations instead of 15).
-Xss works with Oracle's HotSpot VM, IBM's J9 VM and OpenJDK (IcedTea).
I have ensured this does not have any negative implications for other tests.
Author: Adam Roberts <ar...@uk.ibm.com>
Author: a-roberts <ar...@uk.ibm.com>
Closes #6727 from a-roberts/IncJavaStackSize and squashes the following commits:
ab40aea [Adam Roberts] Specify stack size for SBT builds
5032d8d [a-roberts] Update pom.xml
(cherry picked from commit 6b68366df345d4572cf138f9efe17e23d0d1971e)
Signed-off-by: Sean Owen <so...@cloudera.com>
commit 3a62569afb8fcd3d1610b4ede0f2c5e595acb9b9
Author: Shivaram Venkataraman <sh...@cs.berkeley.edu>
Date: 2015-06-11T20:22:08Z
[SPARK-8310] [EC2] Update spark-ec2 branch to 1.4
cc pwendell -- We should probably update our release guidelines to change this when we cut a release branch ?
Author: Shivaram Venkataraman <sh...@cs.berkeley.edu>
Closes #6765 from shivaram/SPARK-8310-14 and squashes the following commits:
066e44e [Shivaram Venkataraman] Update spark-ec2 branch to 1.4
commit 8b25f62bf19b02042675aa1d4e4b58cc4deb3e26
Author: Marcelo Vanzin <va...@cloudera.com>
Date: 2015-06-11T22:29:03Z
[SPARK-6511] [docs] Fix example command in hadoop-provided docs.
Author: Marcelo Vanzin <va...@cloudera.com>
Closes #6766 from vanzin/SPARK-6511 and squashes the following commits:
49f0f67 [Marcelo Vanzin] [SPARK-6511] [docs] Fix example command in hadoop-provided docs.
(cherry picked from commit 9cbdf31ec1399d4d43a1863c15688ce78b6dfd92)
Signed-off-by: Reynold Xin <rx...@databricks.com>
commit 141eab71ee3aa05da899ecfc6bae40b3798a4665
Author: Mark Smith <ma...@bronto.com>
Date: 2015-06-12T17:28:30Z
[SPARK-8322] [EC2] Added spark 1.4.0 into the VALID_SPARK_VERSIONS and…
… SPARK_TACHYON_MAP
Author: Mark Smith <ma...@bronto.com>
Closes #6777 from markmsmith/branch-1.4 and squashes the following commits:
a218cfa [Mark Smith] [SPARK-8322][EC2] Fixed tachyon mapp entry to point to 0.6.4
90d1655 [Mark Smith] [SPARK-8322][EC2] Added spark 1.4.0 into the VALID_SPARK_VERSIONS and SPARK_TACHYON_MAP
commit 76083734196a7571de314df79e88759b650ed1f3
Author: Andrew Or <an...@databricks.com>
Date: 2015-06-12T18:14:55Z
[SPARK-8330] DAG visualization: trim whitespace from input
Safeguard against DOM rewriting.
Author: Andrew Or <an...@databricks.com>
Closes #6787 from andrewor14/dag-viz-trim and squashes the following commits:
0fb4afe [Andrew Or] Trim input metadata from DOM
(cherry picked from commit 88604051511c788d7abb41a49e3eb3a8330c09a9)
Signed-off-by: Andrew Or <an...@databricks.com>
commit 7c11ccf3913ac6a5d178994704d8b0983829b43b
Author: Tathagata Das <ta...@gmail.com>
Date: 2015-06-12T22:22:59Z
[SPARK-7284] [STREAMING] Updated streaming documentation
- Kinesis API updated
- Kafka version updated, and Python API for Direct Kafka added
- Added SQLContext.getOrCreate()
- Added information on how to get partitionId in foreachRDD
Author: Tathagata Das <ta...@gmail.com>
Closes #6781 from tdas/SPARK-7284 and squashes the following commits:
aac7be0 [Tathagata Das] Added information on how to get partition id
a66ec22 [Tathagata Das] Complete the line incomplete line,
a92ca39 [Tathagata Das] Updated streaming documentation
(cherry picked from commit e9471d3414d327c7d0853e18f1844ab1bd09c8ed)
Signed-off-by: Tathagata Das <ta...@gmail.com>
commit 1ca431e83f070f9737b4cc3b7918188ad5dd3d36
Author: Michael Armbrust <mi...@databricks.com>
Date: 2015-06-13T06:11:16Z
[SPARK-8329][SQL] Allow _ in DataSource options
Author: Michael Armbrust <mi...@databricks.com>
Closes #6786 from marmbrus/optionsParser and squashes the following commits:
e7d18ef [Michael Armbrust] add dots
99a3452 [Michael Armbrust] [SPARK-8329][SQL] Allow _ in DataSource options
(cherry picked from commit 4aed66f299a67f5a594da9316b6bf4c345838216)
Signed-off-by: Reynold Xin <rx...@databricks.com>
commit 187a3d5385e778c188d0c1c2adc755ac2d25e8e8
Author: Mike Dusenberry <du...@gmail.com>
Date: 2015-06-14T04:22:46Z
[Spark-8343] [Streaming] [Docs] Improve Spark Streaming Guides.
This improves the Spark Streaming Guides by fixing broken links, rewording confusing sections, fixing typos, adding missing words, etc.
Author: Mike Dusenberry <du...@gmail.com>
Closes #6801 from dusenberrymw/SPARK-8343_Improve_Spark_Streaming_Guides_MERGED and squashes the following commits:
6688090 [Mike Dusenberry] Improvements to the Spark Streaming Custom Receiver Guide, including slight rewording of confusing sections, and fixing typos & missing words.
436fbd8 [Mike Dusenberry] Bunch of improvements to the Spark Streaming Guide, including fixing broken links, slight rewording of confusing sections, fixing typos & missing words, etc.
(cherry picked from commit 35d1267cf8e918032c92a206b22bb301bf0c806e)
Signed-off-by: Reynold Xin <rx...@databricks.com>
commit 4634be5a7db4f2fd82cfb5c602b79129d1d9e246
Author: Josh Rosen <jo...@databricks.com>
Date: 2015-06-14T16:34:35Z
[SPARK-8354] [SQL] Fix off-by-factor-of-8 error when allocating scratch space in UnsafeFixedWidthAggregationMap
UnsafeFixedWidthAggregationMap contains an off-by-factor-of-8 error when allocating row conversion scratch space: we take a size requirement, measured in bytes, then allocate a long array of that size. This means that we end up allocating 8x too much conversion space.
This patch fixes this by allocating a `byte[]` array instead. This doesn't impose any new limitations on the maximum sizes of UnsafeRows, since UnsafeRowConverter already used integers when calculating the size requirements for rows.
Author: Josh Rosen <jo...@databricks.com>
Closes #6809 from JoshRosen/sql-bytes-vs-words-fix and squashes the following commits:
6520339 [Josh Rosen] Updates to reflect fact that UnsafeRow max size is constrained by max byte[] size
(cherry picked from commit ea7fd2ff6454e8d819a39bf49901074e49b5714e)
Signed-off-by: Josh Rosen <jo...@databricks.com>
commit 2805d145e30e4cabd11a7d33c4f80edbc54cc54a
Author: Michael Armbrust <mi...@databricks.com>
Date: 2015-06-14T18:21:42Z
[SPARK-8358] [SQL] Wait for child resolution when resolving generators
Author: Michael Armbrust <mi...@databricks.com>
Closes #6811 from marmbrus/aliasExplodeStar and squashes the following commits:
fbd2065 [Michael Armbrust] more style
806a373 [Michael Armbrust] fix style
7cbb530 [Michael Armbrust] [SPARK-8358][SQL] Wait for child resolution when resolving generatorsa
(cherry picked from commit 9073a426e444e4bc6efa8608e54e0a986f38a270)
Signed-off-by: Michael Armbrust <mi...@databricks.com>
commit 0ffbf085190b9d4dc13a8b6545e4e1022083bd35
Author: Peter Hoffmann <ph...@peter-hoffmann.com>
Date: 2015-06-14T18:41:16Z
fix read/write mixup
Author: Peter Hoffmann <ph...@peter-hoffmann.com>
Closes #6815 from hoffmann/patch-1 and squashes the following commits:
2abb6da [Peter Hoffmann] fix read/write mixup
(cherry picked from commit f3f2a4397da164f0ddfa5d60bf441099296c4346)
Signed-off-by: Reynold Xin <rx...@databricks.com>
commit fff8d7ee6c7e88ed96c29260480e8228e7fb1435
Author: tedyu <yu...@gmail.com>
Date: 2015-06-16T00:00:38Z
SPARK-8336 Fix NullPointerException with functions.rand()
This PR fixes the problem reported by Justin Yip in the thread 'NullPointerException with functions.rand()'
Tested using spark-shell and verified that the following works:
sqlContext.createDataFrame(Seq((1,2), (3, 100))).withColumn("index", rand(30)).show()
Author: tedyu <yu...@gmail.com>
Closes #6793 from tedyu/master and squashes the following commits:
62fd97b [tedyu] Create RandomSuite
750f92c [tedyu] Add test for Rand() with seed
a1d66c5 [tedyu] Fix NullPointerException with functions.rand()
(cherry picked from commit 1a62d61696a0481508d83a07d19ab3701245ac20)
Signed-off-by: Reynold Xin <rx...@databricks.com>
commit f287f7ea141fa7a3e9f8b7d3a2180b63cd77088d
Author: huangzhaowei <ca...@gmail.com>
Date: 2015-06-16T06:16:09Z
[SPARK-8367] [STREAMING] Add a limit for 'spark.streaming.blockInterval` since a data loss bug.
Bug had reported in the jira [SPARK-8367](https://issues.apache.org/jira/browse/SPARK-8367)
The relution is limitting the configuration `spark.streaming.blockInterval` to a positive number.
Author: huangzhaowei <ca...@gmail.com>
Author: huangzhaowei <Sa...@users.noreply.github.com>
Closes #6818 from SaintBacchus/SPARK-8367 and squashes the following commits:
c9d1927 [huangzhaowei] Update BlockGenerator.scala
bd3f71a [huangzhaowei] Use requre instead of if
3d17796 [huangzhaowei] [SPARK_8367][Streaming]Add a limit for 'spark.streaming.blockInterval' since a data loss bug.
(cherry picked from commit ccf010f27bc62f7e7f409c6eef7488ab476de609)
Signed-off-by: Sean Owen <so...@cloudera.com>
commit 1378bdc4a9a974b40c7c509f4af7f07bdc892e14
Author: Moussa Taifi <mo...@gmail.com>
Date: 2015-06-16T19:59:22Z
[SPARK-DOCS] [SPARK-SQL] Update sql-programming-guide.md
Typo in thriftserver section
Author: Moussa Taifi <mo...@gmail.com>
Closes #6847 from moutai/patch-1 and squashes the following commits:
1bd29df [Moussa Taifi] Update sql-programming-guide.md
(cherry picked from commit dc455b88330f79b1181a585277ea9ed3e0763703)
Signed-off-by: Sean Owen <so...@cloudera.com>
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Branch 1.4
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9616#issuecomment-155653719
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Branch 1.4
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/9616#issuecomment-155674840
@f7753 can you please close the pull request?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Branch 1.4
Posted by f7753 <gi...@git.apache.org>.
Github user f7753 closed the pull request at:
https://github.com/apache/spark/pull/9616
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org