You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by cfregly <gi...@git.apache.org> on 2014/09/06 22:22:02 UTC
[GitHub] spark pull request: [SPARK-1981][Streaming] Updated kinesis docs a...
GitHub user cfregly opened a pull request:
https://github.com/apache/spark/pull/2306
[SPARK-1981][Streaming] Updated kinesis docs and added architecture diagram
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cfregly/spark kinesis-doc
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2306.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2306
----
commit b3b0ff118cac3c0a5a10f9912b383bb0665c9a1b
Author: Chris Fregly <ch...@fregly.com>
Date: 2014-07-16T07:03:04Z
[SPARK-1981] Add AWS Kinesis streaming support
commit d18e680670dfed3039477941e953b5811adde4ab
Author: Chris Fregly <ch...@fregly.com>
Date: 2014-07-16T22:23:21Z
Merge remote-tracking branch 'upstream/master'
Resolved conflict:
project/SparkBuild.scala
commit cd68c0d7bb0c1ef38e7c92d0cd6eb4a7ccf2ce27
Author: Chris Fregly <ch...@fregly.com>
Date: 2014-07-19T03:16:13Z
fixed typos and backward compatibility
commit e7c8978482e1092cf599a31b402ab1f4ec11e36c
Author: Chris Fregly <ch...@fregly.com>
Date: 2014-07-22T03:09:42Z
Merge remote-tracking branch 'upstream/master'
commit 828f8aeb1081cf7ad9e5386e1cce933ece9c3d62
Author: Chris Fregly <ch...@fregly.com>
Date: 2014-07-22T05:20:42Z
more cleanup
commit 338997e6e750c206bfb50a654b725be5f33beb07
Author: Chris Fregly <ch...@fregly.com>
Date: 2014-07-22T15:54:35Z
improve build docs for kinesis
commit 6c395619dde93a9b8e9137b1150de4ae5129cf4b
Author: Chris Fregly <ch...@fregly.com>
Date: 2014-07-24T03:55:55Z
parameterized the versions of the aws java sdk and kinesis client
commit 21de67fbc5298ce73dd36fe8372dbeee27f69854
Author: Chris Fregly <ch...@fregly.com>
Date: 2014-07-31T00:43:29Z
Merge remote-tracking branch 'upstream/master'
Incorporated feedback from TD. Renamed project to kinesis-asl,
simplified examples, improved docs, fixed bug with AWS
regions outside of us-east-1
commit db3eefd0b845ade8251235698c74f8bdc5d35e5a
Author: Chris Fregly <ch...@fregly.com>
Date: 2014-07-31T00:43:40Z
Merge remote-tracking branch 'upstream/master'
commit 912640cb344c77102e4ca4d884b8b0d0206ed627
Author: Chris Fregly <ch...@fregly.com>
Date: 2014-07-31T01:03:27Z
changed the foundKinesis class to be a publically-avail class
commit d17ca6d6a36ddf0a3030eacae0eace3fdd758cc5
Author: Chris Fregly <ch...@fregly.com>
Date: 2014-07-31T17:00:09Z
per TD's feedback: updated docs, simplified the KinesisUtils api
commit bf614e9ed870a3c23670d3783d574b1e4280bd81
Author: Chris Fregly <ch...@fregly.com>
Date: 2014-07-31T17:33:20Z
per matei's feedback: moved the kinesis examples into the examples/ dir
commit e33cbeb0238be90a878b71bc7354f957dfe45891
Author: Chris Fregly <ch...@fregly.com>
Date: 2014-08-01T06:07:46Z
Merge remote-tracking branch 'upstream/master'
commit 74e5c7c3ce99f5cd30d269d62aca31d2b275288c
Author: Chris Fregly <ch...@fregly.com>
Date: 2014-08-01T21:14:42Z
updated per TD's feedback. simplified examples, updated docs
commit 0e1c67b079b87b12f58557922ea7d4b257ee571b
Author: Chris Fregly <ch...@fregly.com>
Date: 2014-08-01T21:15:41Z
Merge remote-tracking branch 'upstream/master'
commit 691a6be900015358d55a03c046f93d6336297ea2
Author: Chris Fregly <ch...@fregly.com>
Date: 2014-08-01T21:47:00Z
fixed tests and formatting, fixed a bug with JavaKinesisWordCount during
union of streams
commit 0393795b53c2789973c081dba6f7651fd8678adc
Author: Chris Fregly <ch...@fregly.com>
Date: 2014-08-02T02:23:04Z
moved Kinesis examples out of examples/ and back into extras/kinesis-asl
updated the build to only include kinesis-asl inside the examples jar
when -Pkinesis-asl is specified
commit 47745816b21d7d2255a98283e3055a5a2a397a27
Author: Chris Fregly <ch...@fregly.com>
Date: 2014-08-02T07:10:07Z
updated docs, renamed retry to retryRandom to be more clear, removed
retries around store() method
commit 8e1ae2e2174e78bac1b73cb72ceec45adad1c35a
Author: Chris Fregly <ch...@fregly.com>
Date: 2014-08-03T02:02:40Z
Merge remote-tracking branch 'upstream/master'
commit 862df67eddc234e86a047a1ddae7a438d794c280
Author: Chris Fregly <ch...@fregly.com>
Date: 2014-08-03T23:38:56Z
Merge remote-tracking branch 'upstream/master'
commit 0f37061e2e5ac4a73213d2dccc848aa8bc57b56f
Author: Chris Fregly <ch...@fregly.com>
Date: 2014-08-03T23:40:45Z
SPARK-1981: (Kinesis streaming support) updated streaming-kinesis.md
commit 9b1c71afb83856c4f0f68cbf92d2b6b444a0bb73
Author: Chris Fregly <ch...@fregly.com>
Date: 2014-08-03T23:56:06Z
better explained why spark checkpoints are disabled in the example (due
to no stateful operations being used)
commit baefa30c8acd8ad2d26e4a7f54b071cdaab09b58
Author: Chris Fregly <ch...@fregly.com>
Date: 2014-09-06T20:14:18Z
Merge branch 'master' of https://github.com/cfregly/spark
Conflicts:
docs/streaming-kinesis.md
commit fed0ad926658f50a958edb62d4f0c4733f1e34d5
Author: Chris Fregly <ch...@fregly.com>
Date: 2014-09-06T20:20:21Z
updated kinesis docs and added an arch diagram
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1981][Streaming] Updated kinesis docs a...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2306#issuecomment-54729402
I fixed a few formatting issues and merge it in my own PR. Hope that is fine.
https://github.com/apache/spark/pull/2307/files
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1981][Streaming] Updated kinesis docs a...
Posted by cfregly <gi...@git.apache.org>.
Github user cfregly closed the pull request at:
https://github.com/apache/spark/pull/2306
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1981][Streaming] Updated kinesis docs a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2306#issuecomment-54727680
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1981][Streaming] Updated kinesis docs a...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2306#discussion_r17211366
--- Diff: docs/streaming-kinesis-integration.md ---
@@ -48,33 +51,54 @@ A Kinesis stream can be set up at one of the valid Kinesis endpoints with 1 or m
</div>
</div>
+ `streamingContext`: StreamingContext containg an application name used by Kinesis to tie this Kinesis application to the Kinesis stream
--- End diff --
"containing" spelling error.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org