You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by silviulica <gi...@git.apache.org> on 2016/06/14 19:16:05 UTC
[GitHub] incubator-beam pull request #461: Initial Beam Python SDK
GitHub user silviulica opened a pull request:
https://github.com/apache/incubator-beam/pull/461
Initial Beam Python SDK
Baby Beam Python SDK joins the Beam family.
Code moved from https://github.com/GoogleCloudPlatform/DataflowPythonSDK/
R: @davorbonaci, @francesperry, @robertwb
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/silviulica/incubator-beam beam_python
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-beam/pull/461.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #461
----
commit f18859267f4b1b44ac1bf175c763295d8f4fc6c3
Author: Silviu Calinoiu <si...@google.com>
Date: 2016-02-25T16:10:33Z
first commit
commit 9d0079f4720757d6bc725c2b9bd7711a2ff22e3d
Author: Silviu Calinoiu <si...@google.com>
Date: 2016-02-25T16:15:47Z
Initial push.
commit 5dbf438c501332506eda2c9278cf8340939ece35
Author: Silviu Calinoiu <si...@google.com>
Date: 2016-02-25T21:42:41Z
Several refactorings in preparation for making the repo public.
commit 784a342660b8a875a2b3499ff8202fc50327640f
Author: Silviu Calinoiu <si...@google.com>
Date: 2016-02-25T22:11:42Z
Small fixes in BigQuery snippets and wordcount example.
commit f5d1b52f158abf33ceb03bf29a0c44a126192f3b
Author: robertwb <ro...@google.com>
Date: 2016-02-25T22:12:47Z
Python Dataflow fit-n-finish.
* Updated duplicate label message to be more in line with java. (Also, the issue is more often than not a different transform of the same name, e.g. two Creates.)
* Actually call default_label. Eliminate messy traceback when PTransform.__init__ is not called.
* Add a DeterministicPickleCoder that pickles but fails at runtime if the encoding is not known to be deterministic.
* Get rid of some spurious warnings for incompletely inferred types.
* Remove obsolete TODOs.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115602962
commit 914231c502a30e67b8e6b5a2e7723c797e650c38
Author: gildea <gi...@google.com>
Date: 2016-02-26T01:44:47Z
README: add explicit Table of Contents.
Add new script update-readme-toc.sh to update this section when the
document contents change. Add anchors in the README.md that the
script can use to collect section names and point to sections.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115624954
commit 40b39a4a630f29ded91896347512642d28abbbc7
Author: chamikara <ch...@google.com>
Date: 2016-02-26T23:32:28Z
Some more fixes related to argument passing.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115716125
commit 553936c8775bc9ba4376b0a4cc220b28316e14ec
Author: gildea <gi...@google.com>
Date: 2016-02-26T23:41:32Z
Readme: add a missing section anchor, close all anchors.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115717014
commit 56d13586ef255fd76e9d32b03e996c41e47decab
Author: gildea <gi...@google.com>
Date: 2016-02-27T00:52:22Z
"README" edit from Robert: []
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115723709
commit 347c509e9683ab526bf6c91c77525eb4ce362a3c
Author: chamikara <ch...@google.com>
Date: 2016-02-29T19:47:06Z
Code snippets for Web doc on PipelineOptions.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115880615
commit 62ac247028b6b47afdcd5281e4d765f87cf7e628
Author: gildea <gi...@google.com>
Date: 2016-03-01T16:28:55Z
Depend on google-apitools-dataflow-v1b3 >= 0.4.20160217
New version includes JobState enum value JOB_STATE_DRAINING.
----Release Notes----
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=116002538
commit c25e77f1a160a0f2e84b9d181e50b9193f4b72f1
Author: chamikara <ch...@google.com>
Date: 2016-03-02T04:27:25Z
Performs several updates to doc snippents for PipelineOptions.
----Release Notes----
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=116093263
commit 155bd6dd63f65405c7b9457013f845f3ad0a2c85
Author: altay <al...@google.com>
Date: 2016-03-02T22:50:44Z
Validate pipeline options at the time of pipeline creation.
----Release Notes----
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=116185013
commit 5be011f7c87ff69c18b16b138c5ebc622fbcbe18
Author: altay <al...@google.com>
Date: 2016-03-02T23:02:00Z
Simplify whitelist warning to show warning before every run.
----Release Notes----
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=116186349
commit 6fdc7e21ead964147dbd1fa30ab432f8f96e20cd
Author: ccy <cc...@google.com>
Date: 2016-03-02T23:36:29Z
Support combiner lifting, update batch job major version to 4
Support combiner lifting by supporting combining operations in
PartialGroupByKey and GroupAlsoByWindows. Update
ENVIRONMENT_MAJOR_VERSION of Python SDK Dataflow runner to 4 for batch
jobs.
----Release Notes----
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=116190446
commit c88359abd75f1502508d4a10fa158ada541531de
Author: altay <al...@google.com>
Date: 2016-03-03T01:03:49Z
Adding required options to the remote execution snippet
----Release Notes----
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=116200233
commit f0fd320c15d0562392cdc7b9f7ff1a4842082544
Author: silviuc <si...@google.com>
Date: 2016-03-03T01:23:25Z
Add protobuf as dependency to address namespace sharing issue
----Release Notes----
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=116202522
commit fb4ee4bf6a9a7114108fd1912f3a8316e4585b1a
Author: robertwb <ro...@google.com>
Date: 2016-03-03T01:35:35Z
Improve pickling robustness
Monkey-patch dill to allow both the UnboundMethodType and MethodType
names for types.[Unbound]MethodType. Also don't pickle the compare
operator (and possibly its entire context) with every element in Top.
----Release Notes----
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=116203729
commit 809148419c48d382926f49b563b2c6f2c2397191
Author: robertwb <ro...@google.com>
Date: 2016-03-03T10:02:25Z
Update dill to version 0.2.5
----Release Notes----
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=116230877
commit 6bfb1ba12e29f8e6ccfde5bc4628497df8a5817b
Author: silviuc <si...@google.com>
Date: 2016-03-03T23:58:57Z
Support for staging SDK tarball downloaded from github
----Release Notes----
Changed default behavior to download the SDK tarball from github with
the appropriate version matching the version of the SDK submitting the
workflow. The --sdk-location option can be used to override this
behavior and use an explicit tarball location (GCS path or URL).
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=116304397
commit 11d384df6c350fcbf848bda98f2fd1651cea53b4
Author: altay <al...@google.com>
Date: 2016-03-04T00:17:46Z
Initialize worker logging earlier
Moves common worker logging code from streaming and batch worker to start.py. This is required for log statements in start.py to work.
----Release Notes----
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=116306682
commit 12da8d010cd1dacf20557444ffc615d0e27a100f
Author: altay <al...@google.com>
Date: 2016-03-04T19:31:55Z
Display a warning when pipeline option runner is not specified
----Release Notes----
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=116383057
commit 90ea3714a0cd9c64a864e4ae04a0d2cb973a2680
Author: altay <al...@google.com>
Date: 2016-03-04T19:47:57Z
Apply format string to log message only when there are args
----Release Notes----
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=116385000
commit 94af736d199834bea0f60d263bbe05a58ab01982
Author: ccy <cc...@google.com>
Date: 2016-03-04T22:14:23Z
Change "is_streaming" pipeline option to "streaming"
----Release Notes----
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=116400225
commit 9d361271f36ec63334c4a6c54a2ae7cbc350468b
Author: robertwb <ro...@google.com>
Date: 2016-03-05T01:34:33Z
Many pickling fixes.
We now test pickleability for at pipeline construction,
even for the direct runner. Also better debugability.
----Release Notes----
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=116419809
commit a2c34c05a2c0394a11ba28c65ec9641beb78901e
Author: gildea <gi...@google.com>
Date: 2016-03-06T21:40:08Z
Improve the str() output of various Dataflow classes
----Release Notes----
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=116499527
commit 70a632fd82aa4bac0765d79ac2ee3414f87741c3
Author: robertwb <ro...@google.com>
Date: 2016-03-07T20:47:04Z
Disallow (broken) pickling of generators.
----Release Notes----
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=116577198
commit bd2a89aa3909614e927c15229ca4b612e042b532
Author: altay <al...@google.com>
Date: 2016-03-07T21:13:10Z
Accept arbitrary objects as first input to the logger
----Release Notes----
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=116579883
commit 8c75cf9df164ca11f6a0d47030cc7dd14528b183
Author: ccy <cc...@google.com>
Date: 2016-03-07T23:53:33Z
Support timer-based triggers, watermark holds in streaming jobs
Introduces watermark holds, OutputTimeFns and timer support with time
domains for emulated batch/DirectPipelineRunner and streaming Windmill
remote execution.
----Release Notes----
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=116597580
commit fa34338a6345e6f22b99776758364b1cfcc66f15
Author: altay <al...@google.com>
Date: 2016-03-08T02:03:47Z
Display a warning only when pipeline option runner is not specified
----Release Notes----
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=116609566
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-beam pull request #461: Initial Beam Python SDK
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-beam/pull/461
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---