You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by ke...@apache.org on 2022/06/15 20:40:28 UTC
[beam-starter-python] branch main updated: Initial commit (#1)
This is an automated email from the ASF dual-hosted git repository.
kenn pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/beam-starter-python.git
The following commit(s) were added to refs/heads/main by this push:
new 9e9a97d Initial commit (#1)
9e9a97d is described below
commit 9e9a97d573378b0a36a2545e8def087392bee907
Author: David Cavazos <dc...@google.com>
AuthorDate: Wed Jun 15 13:40:23 2022 -0700
Initial commit (#1)
---
.github/PULL_REQUEST_TEMPLATE.md | 12 +++++++
.github/workflows/test.yaml | 30 ++++++++++++++++
.gitignore | 40 +++++++++++++++++++++
CONTRIBUTING.md | 69 +++++++++++++++++++++++++++++++++++
LICENSE | 37 ++++++++++++++++++-
MANIFEST.in | 1 +
NOTICE | 5 +++
README.md | 78 ++++++++++++++++++++++++++++++++++++++++
main.py | 32 +++++++++++++++++
my_app/__init__.py | 9 +++++
my_app/app.py | 27 ++++++++++++++
requirements.txt | 1 +
setup.py | 22 ++++++++++++
test/__init__.py | 7 ++++
test/test_my_app.py | 34 ++++++++++++++++++
15 files changed, 403 insertions(+), 1 deletion(-)
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
new file mode 100644
index 0000000..50a629d
--- /dev/null
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -0,0 +1,12 @@
+**Please** add a meaningful description for your change here
+
+------------------------
+
+Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
+
+ - [ ] I agree that my contributions are licensed with both [Apache ASL2](../../LICENSE-APACHE) and [MIT](../../LICENSE-MIT).
+ - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
+ - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
+ - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
+
+See the [`CONTRIBUTING.md`](../../CONTRIBUTING.md) guide for more information.
diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml
new file mode 100644
index 0000000..ca7484a
--- /dev/null
+++ b/.github/workflows/test.yaml
@@ -0,0 +1,30 @@
+# Copyright 2022 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
+# https://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+# <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your
+# option. This file may not be copied, modified, or distributed
+# except according to those terms.
+
+name: Test
+
+on: [push]
+
+jobs:
+ tests:
+ runs-on: ubuntu-latest
+ strategy:
+ matrix:
+ python-version: ["3.7", "3.8", "3.9"]
+ steps:
+ - uses: actions/checkout@v3
+ - name: Set up Python ${{ matrix.python-version }}
+ uses: actions/setup-python@v3
+ with:
+ python-version: ${{ matrix.python-version }}
+ - name: Install local package
+ run: |
+ pip install -U pip
+ pip install -e .
+ - name: Run tests
+ run: python -m unittest -v
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..934815f
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,40 @@
+# IDEs
+.vscode/
+.idea/
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# Virtual environments.
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
new file mode 100644
index 0000000..45e824a
--- /dev/null
+++ b/CONTRIBUTING.md
@@ -0,0 +1,69 @@
+# Contributing
+
+đđ Thanks for taking the time to contribute! đđ
+
+There are many ways to contribute, here are some.
+
+## Filing an issue
+
+If there's any issue you encounter or anything that needs to be fixed, feel free to [create a GitHub issue](https://github.com/apache/beam-starter-python/issues).
+
+## Contributing to this starter project
+
+If this is your first time contributing to a GitHub repo,
+we recommmend going through the
+[GitHub quickstart](https://docs.github.com/en/get-started/quickstart/hello-world).
+
+It's a good idea to discuss your plans with the Beam community through the dev@beam.apache.org mailing list before doing any changes.
+
+Here's a small overview of the process.
+
+1. [Fork the repo](https://docs.github.com/en/get-started/quickstart/fork-a-repo).
+
+1. Clone the repo.
+
+ ```sh
+ export GITHUB_USERNAME="my-github-username"
+
+ git clone git@github.com:$GITHUB_USERNAME/beam-starter-python.git
+ ```
+
+1. Set the [upstream remote branch](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/configuring-a-remote-for-a-fork).
+
+ ```sh
+ cd beam-starter-python
+ git remote add upstream git@github.com:apache/beam-starter-python.git
+ ```
+
+1. Create and change to a new branch.
+
+ ```sh
+ git checkout -B my-branch-name
+ ```
+
+1. Modify the code! đą
+
+1. Run the tests. For steps on how to run them see the [`README.md`](README.md).
+
+1. Commit and push your changes to your branch in `origin`.
+
+ ```sh
+ git commit -m "one line description of your changes"
+ git push
+ ```
+
+1. [Create a Pull Request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request).
+
+1. Add reviewers, and [address review comments](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests).
+
+1. Once it's approved, we can merge the Pull Request.
+
+For more information about proposing changes to a GitHub repository, see the
+[Propose changes](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-branches)
+page in the GitHub docs.
+
+## Contributing to Apache Beam
+
+For information on how to contribute to
+[Apache Beam](https://github.com/apache/beam), see the
+[Contribution guide](https://beam.apache.org/contribute/).
diff --git a/LICENSE b/LICENSE
index 261eeb9..c7e6385 100644
--- a/LICENSE
+++ b/LICENSE
@@ -1,3 +1,38 @@
+This software is distributed under the terms of both the MIT license and the
+Apache License (Version 2.0).
+
+
+MIT license
+
+Copyright 2022 Google LLC
+
+Permission is hereby granted, free of charge, to any
+person obtaining a copy of this software and associated
+documentation files (the "Software"), to deal in the
+Software without restriction, including without
+limitation the rights to use, copy, modify, merge,
+publish, distribute, sublicense, and/or sell copies of
+the Software, and to permit persons to whom the Software
+is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice
+shall be included in all copies or substantial portions
+of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF
+ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED
+TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
+PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
+SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR
+IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+DEALINGS IN THE SOFTWARE.
+
+
+Apache 2 license
+
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
@@ -198,4 +233,4 @@
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
- limitations under the License.
+ limitations under the License.
\ No newline at end of file
diff --git a/MANIFEST.in b/MANIFEST.in
new file mode 100644
index 0000000..540b720
--- /dev/null
+++ b/MANIFEST.in
@@ -0,0 +1 @@
+include requirements.txt
\ No newline at end of file
diff --git a/NOTICE b/NOTICE
new file mode 100644
index 0000000..454af2e
--- /dev/null
+++ b/NOTICE
@@ -0,0 +1,5 @@
+Apache Beam
+Copyright [2022-] The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..dcd218e
--- /dev/null
+++ b/README.md
@@ -0,0 +1,78 @@
+# Apache Beam starter for Python
+
+If you want to clone this repository to start your own project,
+you can choose the license you prefer and feel free to delete anything related to the license you are dropping.
+
+## Before you begin
+
+Make sure you have a [Python 3](https://www.python.org/) development environment ready.
+If you don't, you can download and install it from the
+[Python downloads page](https://www.python.org/downloads/).
+
+We recommend using a virtual environment to isolate your project's dependencies.
+
+```sh
+# Create a new Python virtual environment.
+python -m venv env
+
+# Activate the virtual environment.
+source env/bin/activate
+```
+
+While activated, your `python` and `pip` commands will point to the virtual environment,
+so any changes or install dependencies are self-contained.
+
+As a one time setup, let's install the project's dependencies from the [`requirements.txt`](requirements.txt) file.
+
+```py
+# It's always a good idea to update pip before installing dependencies.
+pip install -U pip
+
+# Install the project as a local package, this installs all the dependencies as well.
+pip install -e .
+```
+
+> âšī¸ Once you are done, you can run the `deactivate` command to go back to your global Python installation.
+
+### Running the pipeline
+
+Running your pipeline in Python is as easy as running the script file directly.
+
+```sh
+# You can run the script file directly.
+python main.py
+
+# To run passing command line arguments.
+python main.py --input-text="đ"
+
+# To run the tests.
+python -m unittest -v
+```
+
+## GitHub Actions automated testing
+
+This project already comes with automated testing via [GitHub Actions](https://github.com/features/actions).
+
+To configure it, look at the [`.github/workflows/test.yaml`](.github/workflows/test.yaml) file.
+
+## Using other runners
+
+To keep this template small, it only includes the [Direct Runner](https://beam.apache.org/documentation/runners/direct/).
+
+For a comparison of what each runner currently supports, look at the [Beam Capability Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).
+
+To add a new runner, visit the runner's page for instructions on how to include it.
+
+## Contributing
+
+Thank you for your interest in contributing!
+All contributions are welcome! đđ
+
+Please refer to the [`CONTRIBUTING.md`](CONTRIBUTING.md) file for more information.
+
+# License
+
+This software is distributed under the terms of both the MIT license and the
+Apache License (Version 2.0).
+
+See [LICENSE](LICENSE) for details.
diff --git a/main.py b/main.py
new file mode 100644
index 0000000..2de2540
--- /dev/null
+++ b/main.py
@@ -0,0 +1,32 @@
+# Copyright 2022 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
+# https://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+# <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your
+# option. This file may not be copied, modified, or distributed
+# except according to those terms.
+
+from apache_beam.options.pipeline_options import PipelineOptions
+
+from my_app import app
+
+
+if __name__ == "__main__":
+ import argparse
+ import logging
+
+ logging.getLogger().setLevel(logging.INFO)
+
+ parser = argparse.ArgumentParser()
+ parser.add_argument(
+ "--input-text",
+ default="Default input text",
+ help="Input text to print.",
+ )
+ args, beam_args = parser.parse_known_args()
+
+ beam_options = PipelineOptions(save_main_session=True, setup_file="./setup.py")
+ app.run(
+ input_text=args.input_text,
+ beam_options=beam_options,
+ )
diff --git a/my_app/__init__.py b/my_app/__init__.py
new file mode 100644
index 0000000..b547cc6
--- /dev/null
+++ b/my_app/__init__.py
@@ -0,0 +1,9 @@
+# Copyright 2022 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
+# https://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+# <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your
+# option. This file may not be copied, modified, or distributed
+# except according to those terms.
+
+from .app import *
diff --git a/my_app/app.py b/my_app/app.py
new file mode 100644
index 0000000..7edcd7a
--- /dev/null
+++ b/my_app/app.py
@@ -0,0 +1,27 @@
+# Copyright 2022 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
+# https://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+# <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your
+# option. This file may not be copied, modified, or distributed
+# except according to those terms.
+
+from typing import Callable, Optional
+import apache_beam as beam
+from apache_beam.options.pipeline_options import PipelineOptions
+
+
+def run(
+ input_text: str,
+ beam_options: Optional[PipelineOptions] = None,
+ test: Callable[[beam.PCollection], None] = lambda _: None,
+) -> None:
+ with beam.Pipeline(options=beam_options) as pipeline:
+ elements = (
+ pipeline
+ | "Create elements" >> beam.Create(["Hello", "World!", input_text])
+ | "Print elements" >> beam.Map(print)
+ )
+
+ # Used for testing only.
+ test(elements)
diff --git a/requirements.txt b/requirements.txt
new file mode 100644
index 0000000..34a4d2e
--- /dev/null
+++ b/requirements.txt
@@ -0,0 +1 @@
+apache-beam==2.39.0
diff --git a/setup.py b/setup.py
new file mode 100644
index 0000000..cbee563
--- /dev/null
+++ b/setup.py
@@ -0,0 +1,22 @@
+# Copyright 2022 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
+# https://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+# <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your
+# option. This file may not be copied, modified, or distributed
+# except according to those terms.
+
+from setuptools import setup, find_packages
+
+with open("requirements.txt") as f:
+ requirements = f.readlines()
+
+setup(
+ name="My app",
+ version="1.0",
+ description="Python Apache Beam pipeline.",
+ author="My name",
+ author_email="my@email.com",
+ packages=find_packages(),
+ install_requires=requirements,
+)
diff --git a/test/__init__.py b/test/__init__.py
new file mode 100644
index 0000000..bfe3779
--- /dev/null
+++ b/test/__init__.py
@@ -0,0 +1,7 @@
+# Copyright 2022 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
+# https://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+# <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your
+# option. This file may not be copied, modified, or distributed
+# except according to those terms.
diff --git a/test/test_my_app.py b/test/test_my_app.py
new file mode 100644
index 0000000..c3ecedb
--- /dev/null
+++ b/test/test_my_app.py
@@ -0,0 +1,34 @@
+# Copyright 2022 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
+# https://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+# <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your
+# option. This file may not be copied, modified, or distributed
+# except according to those terms.
+
+# For more information on unittest, see:
+# https://docs.python.org/3/library/unittest.html
+
+import unittest
+from unittest.mock import patch
+
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that, equal_to
+
+import my_app
+
+
+@patch("apache_beam.Pipeline", TestPipeline)
+@patch("builtins.print", lambda x: x)
+class TestApp(unittest.TestCase):
+ def test_run_direct_runner(self):
+ # Note that the order of the elements doesn't matter.
+ expected = ["Test", "Hello", "World!"]
+ my_app.run(
+ input_text="Test",
+ test=lambda elements: assert_that(elements, equal_to(expected)),
+ )
+
+
+if __name__ == "__main__":
+ unittest.main()