You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/07/24 03:49:00 UTC
[jira] [Commented] (BEAM-1899) JStorm runner
[ https://issues.apache.org/jira/browse/BEAM-1899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16097910#comment-16097910 ]
ASF GitHub Bot commented on BEAM-1899:
--------------------------------------
GitHub user bastiliu opened a pull request:
https://github.com/apache/beam/pull/3623
[BEAM-1899] Merge branch master up to commit '5e3c5c6574bc70320683d6c16fc3b11791a77418' into jstorm-runner
Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes.
- [ ] Each commit in the pull request should have a meaningful subject line and body.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue.
- [ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
- [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
---
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/bastiliu/beam jstorm-runner
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/3623.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3623
----
commit a0da7becae2ce4346f04d2194b5553333e41d0d1
Author: Aljoscha Krettek <al...@gmail.com>
Date: 2017-05-08T18:17:39Z
This closes #2476
commit 44c9385ddbcfbdf904c5d054754dca3ddde88ec2
Author: Dan Halperin <dh...@google.com>
Date: 2017-05-06T00:16:34Z
[BEAM-2190] pom.xml: do a better job of dependency management
Even if Beam appears to have the correct dependencies, we cannot
guarantee that modules that depend on us transitively get the right
dependencies. For example, even though grpc-protobuf-lite has
protobuf-lite excluded, and the Maven Enforcer banned-dependencies
check passes... if a user happens to get a transitive dependency on
grpc-all first, they may pull in grpc-protobuf from that other source
without the exclusion. Thus we need to exclude protobuf-lite from
grpc-all as well.
While we're here, also add guava-jdk5 to the set of banned dependencies,
though (as above) we cannot currently properly identify the places it
might be transitively exposed in a users' pom.xml.
commit 2b0e699b8f6a0aac26ac57c536e8d45cee1c3d2e
Author: Dan Halperin <dh...@google.com>
Date: 2017-05-08T18:35:56Z
This closes #2934
commit 5bac40efcc7bcd9d53183579ec6089c23aeda782
Author: Dan Halperin <dh...@google.com>
Date: 2017-05-06T02:06:03Z
[BEAM-2212] FileBasedSource: improve message when logging.
ValueProvider should not be printed, rather the string instead.
commit 9423babd8f827e843723c218441e9a91aaa7b361
Author: Dan Halperin <dh...@google.com>
Date: 2017-05-08T16:59:16Z
[BEAM-2212] FileBasedSource: refactor to remove uses of fileOrPatternSpec.get()
Makes it less likely to have errors from printing ValueProviders instead of runtime values
commit e1791c3f83a007013625b43f9976e7ffa64e8a49
Author: Dan Halperin <dh...@google.com>
Date: 2017-05-08T18:48:42Z
This closes #2956
commit 13e51c9cd114eec73c47b71f46214dd92ee81048
Author: Dan Halperin <dh...@google.com>
Date: 2017-05-06T02:49:29Z
Convert Executor Services to use Daemon Threads
This will cause the DirectRunner to automatically shut down when the
worker threads are shut down.
commit 63c6bea3bbdb52964ac54073326d4ad99ba7432c
Author: Thomas Groh <tg...@google.com>
Date: 2017-05-08T19:52:09Z
This closes #2954
commit 50a4c56de1d3aa0e51e8b2b08b71715be4c2ef80
Author: Sourabh Bajaj <so...@google.com>
Date: 2017-05-07T20:55:20Z
[BEAM-2206] Move pipelineOptions into options modules
commit 604c2f59e2a928a95106814b0b55b8097dc39faa
Author: Ahmet Altay <al...@google.com>
Date: 2017-05-08T19:56:15Z
This closes #2925
commit d3fc7d18a4a3c394d32d85ef34c08a85712a8b18
Author: Reuven Lax <re...@google.com>
Date: 2017-05-08T16:06:55Z
TableDescription is allowed to be null.
commit c481396edc7a3e906cd79dcc2a356baf9c93895e
Author: Eugene Kirpichov <ki...@google.com>
Date: 2017-05-08T20:41:01Z
This closes #2953
commit 1fd15bd7441e1be72feb3f9535ffe536d730a531
Author: Reuven Lax <re...@google.com>
Date: 2017-05-07T18:50:08Z
Shortens PubsubIO names for reading/writing messages.
commit d1914ed0c3195e1de2f707e2bb7bc57597b834fe
Author: Eugene Kirpichov <ki...@google.com>
Date: 2017-05-08T20:45:25Z
This closes #2950
commit b7635aadc2532b5c3bdc38f070865091ebf901c1
Author: Ahmet Altay <al...@google.com>
Date: 2017-05-08T20:42:02Z
Remove apex README.md. This information is already in the website in
quickstart and apex runner pages.
https://github.com/apache/beam-site/pull/232 moves small bits of missing
content.
commit 962e8796d68f9a81b0ae3d1b30d7a4bcadbdaf60
Author: Ahmet Altay <al...@google.com>
Date: 2017-05-08T21:28:36Z
This closes #2962
commit fba4c010ce90db8b5cdac489d47126ca3ee8db65
Author: Ahmet Altay <al...@google.com>
Date: 2017-05-08T22:00:33Z
Remove readme.md files that are covered in web site
commit 152c5bcc7f6fb53565ed132653384f4244542b59
Author: Ahmet Altay <al...@google.com>
Date: 2017-05-08T22:04:46Z
This closes #2966
commit 78c29238aa471138fb55681d8700b095a3606e0d
Author: Robert Bradshaw <ro...@gmail.com>
Date: 2017-05-08T21:54:17Z
[BEAM-2224] Fix temp file management on Windows.
commit f315cc9ff649ccee9c9aff18f4af30cfbd58e5e5
Author: Robert Bradshaw <ro...@gmail.com>
Date: 2017-05-08T23:49:05Z
Closes #2965
commit fd1dda09fc4db48cfbc21afd6310320533fffe78
Author: Reuven Lax <re...@google.com>
Date: 2017-04-29T14:33:54Z
[BEAM-2154] More scalable dynamic BigQueryIO.Write
If too many tables are generated in a bundle, spill and group the results
before writing files. Generating hundreds or thousands of file write buffers
in a single bundle was causing workers to crash with out of memory.
commit 6a2586a4ee279f66fd4eae816e93fa8e397acf6b
Author: Eugene Kirpichov <ki...@google.com>
Date: 2017-05-09T01:02:14Z
This closes #2883
commit 367fcb28d544934797d25cb34d54136b2d7d6e99
Author: Vikas Kedigehalli <vi...@google.com>
Date: 2017-05-06T02:24:51Z
Remove google api BackOff usage from sdks/core
commit 87578a6a69f15bca502d9b4c15999aa383e539a5
Author: Vikas Kedigehalli <vi...@google.com>
Date: 2017-05-06T02:33:45Z
Use guava Base64 encoding instead of google api client
commit 88e044fda0a7d0ea1372f075a4c7a4bd54af47da
Author: Luke Cwik <lc...@google.com>
Date: 2017-05-09T02:20:01Z
[BEAM-1871] Remove Google API client dependency from sdks/core
This closes #2937
commit 16f355f7e481ebf029c0edb878742f6fea57b6cd
Author: Dan Halperin <dh...@google.com>
Date: 2017-05-08T22:37:29Z
[BEAM-2211] DataflowRunner: remove validation of file read/write paths
Now that users can implement and register custom FileSystems,
we can no longer really effectively validate filesystems they
can read or write files from. They can even register file://
to point to some HDFS path, e.g.,
commit 23731fe7ab12ea5d3ebb7b3f16c788b83d0caf0b
Author: Dan Halperin <dh...@google.com>
Date: 2017-05-09T03:13:38Z
This closes #2968
commit 27e9a060ed593b2b53b88481591f14b1a274c61b
Author: Robert Bradshaw <ro...@gmail.com>
Date: 2017-05-05T23:20:37Z
Remove explicit used of nested contexts.
find . -type f -name '*.java' | xargs sed -i '' 's/\([.]..code[(].*\), *context.nested..[)]/\1)/'
find . -type f -name '*.java' | xargs sed -i '' 's/\([.]..code[(].*\), *nestedContext[)]/\1)/'
find . -type f -name '*.java' | xargs sed -i '' 's/\([.]..code[(].*\), *Context.NESTED[)]/\1)/'
find . -type f -name '*.java' | xargs sed -i '' 's/\([.]..code[(].*\), *[^ ]*.Context.NESTED[)]/\1)/'
Added back explicit context in CoGbkResult.java due to compile error.
commit 996dce37b76b103f104328b7caa65f73a1bcb15a
Author: Robert Bradshaw <ro...@gmail.com>
Date: 2017-05-05T23:36:47Z
Remove contexts from coders where they'll never be used.
commit b7f3341ed4fc073a834a55773bcb4a2c7f821c52
Author: Robert Bradshaw <ro...@gmail.com>
Date: 2017-05-06T00:24:02Z
automated context removal or redirection
----
> JStorm runner
> -------------
>
> Key: BEAM-1899
> URL: https://issues.apache.org/jira/browse/BEAM-1899
> Project: Beam
> Issue Type: New Feature
> Components: runner-ideas
> Reporter: Pei He
> Assignee: Pei He
>
> Implement a runner for JStorm https://github.com/alibaba/jstorm
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)