You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by guowei2 <gi...@git.apache.org> on 2014/05/28 12:55:59 UTC
[GitHub] spark pull request: 0.9.1
GitHub user guowei2 opened a pull request:
https://github.com/apache/spark/pull/903
0.9.1
receive kafka message with muti-consumers
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/guowei2/spark 0.9.1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/903.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #903
----
commit 1d9c210dcbc0c63893921fdef591edd9ffe4c48e
Author: Patrick Wendell <pw...@gmail.com>
Date: 2014-01-14T08:45:27Z
Version changes for release 0.9.0.
commit b60840e2d5ae6bd6d1c38426e346a414effb5a48
Author: Patrick Wendell <pw...@gmail.com>
Date: 2014-01-14T08:48:34Z
Merge pull request #418 from pwendell/0.9-versions
Version changes for release 0.9.0.
commit 1b4adc21d77f801be795c2814fc0a501f0e6309b
Author: Patrick Wendell <pw...@gmail.com>
Date: 2014-01-14T09:18:34Z
Merge pull request #420 from pwendell/header-files
Add missing header files
(cherry picked from commit fa75e5e1c50da7d1e6c6f41c2d6d591c1e8a025f)
Signed-off-by: Patrick Wendell <pw...@gmail.com>
commit bf3b1506957bd419b6bee4d8ade9fc3c78761dbf
Author: Reynold Xin <rx...@apache.org>
Date: 2014-01-14T17:44:43Z
Merge pull request #423 from jegonzal/GraphXProgrammingGuide
Improving the graphx-programming-guide
This PR will track a few minor improvements to the content and formatting of the graphx-programming-guide.
(cherry picked from commit 3fcc68bfa5e9ef4b7abfd5051b6847a833e1ad2f)
Signed-off-by: Reynold Xin <rx...@apache.org>
commit 119b6c524c659951e6abe791f2559048444b5c22
Author: Reynold Xin <rx...@apache.org>
Date: 2014-01-14T21:28:44Z
Merge pull request #425 from rxin/scaladoc
API doc update & make Broadcast public
In #413 Broadcast was mistakenly made private[spark]. I changed it to public again. Also exposing id in public given the R frontend requires that.
Copied some of the documentation from the programming guide to API Doc for Broadcast and Accumulator.
This should be cherry picked into branch-0.9 as well for 0.9.0 release.
(cherry picked from commit 2ce23a55a3c4033873bb262919d89e5afabb9134)
Signed-off-by: Reynold Xin <rx...@apache.org>
commit a14933dac1e8b866d49a161854453b56a6e1dfcc
Author: Reynold Xin <rx...@apache.org>
Date: 2014-01-14T22:52:24Z
Merge pull request #427 from pwendell/deprecate-aggregator
Deprecate rather than remove old combineValuesByKey function
(cherry picked from commit d601a76d1fdd25b95020b2e32bacde583cf6aa50)
Signed-off-by: Reynold Xin <rx...@apache.org>
commit 329c9df13670871acccd834eb042c59be12bb8f6
Author: Reynold Xin <rx...@apache.org>
Date: 2014-01-14T22:53:24Z
Merge pull request #429 from ankurdave/graphx-examples-pom.xml
Add GraphX dependency to examples/pom.xml
(cherry picked from commit 193a0757c87b717e3b6b4f005ecdbb56b04ad9b4)
Signed-off-by: Reynold Xin <rx...@apache.org>
commit 2f930d5ae10d603370b49b91719259f17dc25628
Author: Reynold Xin <rx...@apache.org>
Date: 2014-01-14T22:59:13Z
Merge pull request #428 from pwendell/writeable-objects
Don't clone records for text files
(cherry picked from commit 74b46acdc57293c103ab5dd5af931d0d0e32c0ed)
Signed-off-by: Reynold Xin <rx...@apache.org>
commit ce66ca78b7c67b8dfc865d2aa32f7c4a71493ca4
Author: Patrick Wendell <pw...@gmail.com>
Date: 2014-01-14T23:16:46Z
Small change to maven build
commit 40c97afce8be3c45590aaf5f789cbff058443892
Author: Patrick Wendell <pw...@gmail.com>
Date: 2014-01-14T23:57:53Z
[maven-release-plugin] prepare release v0.9.0-incubating
commit 51131bf820330bd2475a9676d151f5d488c150a7
Author: Patrick Wendell <pw...@gmail.com>
Date: 2014-01-14T23:57:59Z
[maven-release-plugin] prepare for next development iteration
commit 6fa4e02dd19308c9629fb898061334d554def641
Author: Reynold Xin <rx...@apache.org>
Date: 2014-01-15T05:51:06Z
Merge pull request #431 from ankurdave/graphx-caching-doc
Describe caching and uncaching in GraphX programming guide
(cherry picked from commit ad294db326f57beb98f9734e2b4c45d9da1a4c89)
Signed-off-by: Reynold Xin <rx...@apache.org>
commit a075a452da54f488d178457c87f8109148521a35
Author: Reynold Xin <rx...@apache.org>
Date: 2014-01-15T05:52:13Z
Merge branch 'branch-0.9' of https://git-wip-us.apache.org/repos/asf/incubator-spark into branch-0.9
commit 2c6c07f428079f390901f662d893cd932c90a70a
Author: Reynold Xin <rx...@apache.org>
Date: 2014-01-15T05:52:50Z
Merge pull request #424 from jegonzal/GraphXProgrammingGuide
Additional edits for clarity in the graphx programming guide.
Added an overview of the Graph and GraphOps functions and fixed numerous typos.
(cherry picked from commit 3a386e238984c48a6ac07974b92647beae1199b3)
Signed-off-by: Reynold Xin <rx...@apache.org>
commit 863dd722436598a390d7a0c319c6cf49b488b5f2
Author: Patrick Wendell <pw...@gmail.com>
Date: 2014-01-15T07:05:23Z
Reverting release plugin changes
commit fbfbb331db64dc1c5daef4585c7f85ce0323f5e8
Author: Patrick Wendell <pw...@gmail.com>
Date: 2014-01-15T06:50:36Z
Merge pull request #434 from rxin/graphxmaven
Fixed SVDPlusPlusSuite in Maven build.
This should go into 0.9.0 also.
(cherry picked from commit 087487e90e4d6269d7a027f7cb718120f6c10505)
Signed-off-by: Patrick Wendell <pw...@gmail.com>
commit 2859cab2f50099d1a691aecb5f7e5dfa26dccdb1
Author: Patrick Wendell <pw...@gmail.com>
Date: 2014-01-15T07:07:55Z
Merge pull request #435 from tdas/filestream-fix
Fixed the flaky tests by making SparkConf not serializable
SparkConf was being serialized with CoGroupedRDD and Aggregator, which somehow caused OptionalJavaException while being deserialized as part of a ShuffleMapTask. SparkConf should not even be serializable (according to conversation with Matei). This change fixes that.
@mateiz @pwendell
(cherry picked from commit 139c24ef08e6ffb090975c9808a2cba304eb79e0)
Signed-off-by: Patrick Wendell <pw...@gmail.com>
commit 2f015c2e77f991d8d880c73a241898b370708c56
Author: Reynold Xin <rx...@apache.org>
Date: 2014-01-15T07:17:05Z
Merge pull request #436 from ankurdave/VertexId-case
Rename VertexID -> VertexId in GraphX
(cherry picked from commit 3d9e66d92ada4fa93dd0bd78cb4c80f8169e6393)
Signed-off-by: Reynold Xin <rx...@apache.org>
commit e12c374d223c67f57ac2ec4af55a9e413272dd10
Author: Patrick Wendell <pw...@gmail.com>
Date: 2014-01-15T18:00:50Z
Merge pull request #433 from markhamstra/debFix
Updated Debian packaging
(cherry picked from commit 494d3c077496735e6ebca3217de4f0cc6b6419f2)
Signed-off-by: Patrick Wendell <pw...@gmail.com>
commit aca40aae87179b248da7d96290d19d4c59c01558
Author: Patrick Wendell <pw...@gmail.com>
Date: 2014-01-15T19:15:07Z
Merge pull request #441 from pwendell/graphx-build
GraphX shouldn't list Spark as provided.
I noticed this when building an application against GraphX to audit the released artifacts.
(cherry picked from commit 5fecd2516dc8de28b76fe6e0fbdca7922cc28d1c)
Signed-off-by: Patrick Wendell <pw...@gmail.com>
commit 29c76d96b2489823a7ad4781129b707c73108bf8
Author: Patrick Wendell <pw...@gmail.com>
Date: 2014-01-15T21:54:45Z
Merge pull request #443 from tdas/filestream-fix
Made some classes private[stremaing] and deprecated a method in JavaStreamingContext.
Classes `RawTextHelper`, `RawTextSender` and `RateLimitedOutputStream` are not useful in the streaming API. There are not used by the core functionality and was there as a support classes for an obscure example. One of the classes is RawTextSender has a main function which can be executed using bin/spark-class even if it is made private[streaming]. In future, I will probably completely remove these classes. For the time being, I am just converting them to private[streaming].
Accessing underlying JavaSparkContext in JavaStreamingContext was through `JavaStreamingContext.sc` . This is deprecated and preferred method is `JavaStreamingContext.sparkContext` to keep it consistent with the `StreamingContext.sparkContext`.
(cherry picked from commit 2a05403a7ced4ecf6084c96f582ee3a24f3cc874)
Signed-off-by: Patrick Wendell <pw...@gmail.com>
commit e3fa36f259b7ede73bc148891e2635bf41221660
Author: Patrick Wendell <pw...@gmail.com>
Date: 2014-01-15T21:55:14Z
Merge pull request #442 from pwendell/standalone
Workers should use working directory as spark home if it's not specified
If users don't set SPARK_HOME in their environment file when launching an application, the standalone cluster should default to the spark home of the worker.
(cherry picked from commit 59f475c79fc8fd6d3485e4d0adf6768b6a9225a4)
Signed-off-by: Patrick Wendell <pw...@gmail.com>
commit 4ccedb3d1d8c40d3ec914b2d17be9b43aa4744f4
Author: Patrick Wendell <pw...@gmail.com>
Date: 2014-01-15T22:25:45Z
Merge pull request #444 from mateiz/py-version
Clarify that Python 2.7 is only needed for MLlib
(cherry picked from commit 4f0c361b0e140f5f6879f019b2e1a16c683c705c)
Signed-off-by: Patrick Wendell <pw...@gmail.com>
commit 7749b988dcc6f3abfd1d5b42f07f82eb680010e0
Author: Patrick Wendell <pw...@gmail.com>
Date: 2014-01-15T22:33:37Z
Change log for release 0.9.0-incubating
commit 7348893f0edd96dacce2f00970db1976266f7008
Author: Patrick Wendell <pw...@gmail.com>
Date: 2014-01-15T22:53:02Z
[maven-release-plugin] prepare release v0.9.0-incubating
commit 4ae8a4b39e738daf3f06c38de869b07927a927e1
Author: Patrick Wendell <pw...@gmail.com>
Date: 2014-01-15T22:53:11Z
[maven-release-plugin] prepare for next development iteration
commit 7b0d5a5f306b0adb6590eb551096939d0c617142
Author: Patrick Wendell <pw...@gmail.com>
Date: 2014-01-17T07:18:15Z
Merge pull request #451 from Qiuzhuang/master
Fixed Window spark shell launch script error.
JIRA SPARK-1029:https://spark-project.atlassian.net/browse/SPARK-1029
(cherry picked from commit d749d472b37448edb322bc7208a3db925c9a4fc2)
Signed-off-by: Patrick Wendell <pw...@gmail.com>
commit c8f92730f886fd91a96eb1090ac97cb34cf3709a
Author: Matei Zaharia <ma...@databricks.com>
Date: 2014-01-18T20:48:49Z
Remove Typesafe Config usage and conf files to fix nested property names
With Typesafe Config we had the subtle problem of no longer allowing
nested property names, which are used for a few of our properties:
http://apache-spark-developers-list.1001551.n3.nabble.com/Config-properties-broken-in-master-td208.html
commit ff7201cf9e4f3af6943d6ff5d4b625404a174fec
Author: Patrick Wendell <pw...@gmail.com>
Date: 2014-01-18T20:49:21Z
Merge pull request #461 from pwendell/master
Use renamed shuffle spill config in CoGroupedRDD.scala
This one got missed when it was renamed.
(cherry picked from commit aa981e4e97a11dbd5a4d012bfbdb395982968372)
Signed-off-by: Patrick Wendell <pw...@gmail.com>
commit 34e911ce9a9f91f3259189861779032069257852
Author: Patrick Wendell <pw...@gmail.com>
Date: 2014-01-19T00:17:34Z
Merge pull request #462 from mateiz/conf-file-fix
Remove Typesafe Config usage and conf files to fix nested property names
With Typesafe Config we had the subtle problem of no longer allowing
nested property names, which are used for a few of our properties:
http://apache-spark-developers-list.1001551.n3.nabble.com/Config-properties-broken-in-master-td208.html
This PR is for branch 0.9 but should be added into master too.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: 0.9.1
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/903#issuecomment-50692388
@guowei2 do you mind closing this? I think something is messed up about this patch
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: 0.9.1
Posted by guowei2 <gi...@git.apache.org>.
Github user guowei2 closed the pull request at:
https://github.com/apache/spark/pull/903
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: 0.9.1
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/903#issuecomment-44397035
@guowei2 Yes it looks like you have a load of other commits in this branch. You can rebase on master, and then force-push to your branch to correct it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: 0.9.1
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/903#issuecomment-44390808
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: 0.9.1
Posted by guowei2 <gi...@git.apache.org>.
Github user guowei2 commented on the pull request:
https://github.com/apache/spark/pull/903#issuecomment-44391492
sorry，i pull wrong batch
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---