You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2022/11/04 05:18:06 UTC
[beam] branch nightly-refs/heads/master updated (e2ea0c76969 -> 7da182a8d1d)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch nightly-refs/heads/master
in repository https://gitbox.apache.org/repos/asf/beam.git
from e2ea0c76969 Update REVIEWERS.yaml (#23955)
add 8d71aae2c61 [Spark dataset runner] Add direct translation of Combine.GroupedValues (related to #23845)
add b7110268ab5 Merge pull request #23846: [Spark dataset runner] Add direct translation of Combine.GroupedValues
add 85b6b643bff Concept guide on orchestrating Beam preprocessing (#23094)
add 05c89ed2fd6 Initial draft of Batched DoFn user guide (#23909)
add 279b2755744 WIP: Dataframe API ML preprocessing notebook (#22587)
add 15297da74bc emit load job IDs as soon as they come up
add 9c3ce5debec style fix
add 4f64c7e547c Merge pull request #23954: Emit job ids via side output in TriggerFileLoads process to keep beam.Flatten() happy for Spark and Flink runners
add ec6da53ba23 [Python] Added none check while accessing active_process_bundle (#23947)
add 77116c5c814 [Tour Of Beam] saving user code (#23938)
add e2463a4c965 Disable flaky fn_api_runner tests (#23971)
add 7da182a8d1d Make BatchConverter inference errors more helpful (#23965)
No new revisions were added by this update.
Summary of changes:
.../workflows/tour_of_beam_backend_integration.yml | 18 +-
build.gradle.kts | 6 +-
.../beam-ml/dataframe_api_preprocessing.ipynb | 3496 +++++++++++++++++++
examples/notebooks/tour-of-beam/dataframes.ipynb | 24 +-
learning/tour-of-beam/backend/README.md | 8 +
learning/tour-of-beam/backend/docker-compose.yml | 13 +
learning/tour-of-beam/backend/function.go | 62 +-
learning/tour-of-beam/backend/go.mod | 2 +
.../tour-of-beam/backend/integration_tests/api.go | 10 +
.../{auth_test.go => auth_emulator.go} | 54 +-
.../backend/integration_tests/auth_test.go | 156 +-
.../backend/integration_tests/client.go | 16 +-
.../backend/integration_tests/function_test.go | 10 +-
.../backend/integration_tests/local.sh | 4 +
learning/tour-of-beam/backend/internal/entity.go | 15 +-
.../backend/internal/service/content.go | 16 +-
.../{storage/iface.go => service/pg_adapter.go} | 33 +-
.../backend/internal/storage/adapter.go | 5 +-
.../backend/internal/storage/datastore.go | 38 +-
.../tour-of-beam/backend/internal/storage/iface.go | 1 +
.../tour-of-beam/backend/internal/storage/mock.go | 4 +
.../backend/internal/storage/schema.go | 1 +
.../tour-of-beam/backend/playground_api/api.pb.go | 3507 ++++++++++++++++++++
.../backend/playground_api/api_grpc.pb.go | 791 +++++
.../storage/iface.go => playground_api/helper.go} | 29 +-
.../tour-of-beam/backend/playground_api/mock.go | 1077 ++++++
.../backend/samples/api/get_user_progress.json | 7 +-
.../translation/TransformTranslator.java | 11 +
.../batch/CombineGroupedValuesTranslatorBatch.java | 75 +
.../translation/batch/PipelineTranslatorBatch.java | 6 +-
.../batch/CombineGroupedValuesTest.java | 70 +
.../examples/ml-orchestration/README.md | 22 +
.../kfp/components/ingestion/Dockerfile | 29 +-
.../kfp/components/ingestion/component.yaml | 41 +-
.../kfp/components/ingestion/requirements.txt | 23 +-
.../kfp/components/ingestion/src/ingest.py | 74 +
.../kfp/components/preprocessing/Dockerfile | 29 +-
.../kfp/components/preprocessing/component.yaml | 64 +
.../kfp/components/preprocessing/requirements.txt | 26 +-
.../kfp/components/preprocessing/src/preprocess.py | 208 ++
.../kfp/components/train/Dockerfile | 27 +-
.../kfp/components/train/component.yaml | 46 +-
.../kfp/components/train/requirements.txt | 23 +-
.../kfp/components/train/src/train.py | 83 +
.../examples/ml-orchestration/kfp/pipeline.json | 247 ++
.../examples/ml-orchestration/kfp/pipeline.py | 132 +
.../examples/ml-orchestration/kfp/requirements.txt | 23 +-
.../ml-orchestration/tfx/coco_captions_local.py | 141 +
.../ml-orchestration/tfx/coco_captions_utils.py | 87 +
.../examples/ml-orchestration/tfx/requirements.txt | 22 +-
.../apache_beam/io/gcp/bigquery_file_loads.py | 43 +-
.../portability/fn_api_runner/fn_runner_test.py | 2 +
.../apache_beam/runners/worker/worker_status.py | 2 +-
.../apache_beam/transforms/batch_dofn_test.py | 49 +-
sdks/python/apache_beam/transforms/core.py | 27 +-
.../content/en/documentation/ml/data-processing.md | 2 +
.../content/en/documentation/ml/orchestration.md | 223 ++
.../site/content/en/documentation/ml/overview.md | 3 +-
.../content/en/documentation/programming-guide.md | 235 ++
.../partials/section-menu/en/documentation.html | 2 +
.../static/images/orchestrated-beam-pipeline.svg | 35 +
.../static/images/standalone-beam-pipeline.svg | 35 +
62 files changed, 11106 insertions(+), 464 deletions(-)
create mode 100644 examples/notebooks/beam-ml/dataframe_api_preprocessing.ipynb
copy learning/tour-of-beam/backend/integration_tests/{auth_test.go => auth_emulator.go} (73%)
copy learning/tour-of-beam/backend/internal/{storage/iface.go => service/pg_adapter.go} (56%)
create mode 100644 learning/tour-of-beam/backend/playground_api/api.pb.go
create mode 100644 learning/tour-of-beam/backend/playground_api/api_grpc.pb.go
copy learning/tour-of-beam/backend/{internal/storage/iface.go => playground_api/helper.go} (53%)
create mode 100644 learning/tour-of-beam/backend/playground_api/mock.go
create mode 100644 runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/CombineGroupedValuesTranslatorBatch.java
create mode 100644 runners/spark/3/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/CombineGroupedValuesTest.java
create mode 100644 sdks/python/apache_beam/examples/ml-orchestration/README.md
copy learning/tour-of-beam/backend/docker-compose.yml => sdks/python/apache_beam/examples/ml-orchestration/kfp/components/ingestion/Dockerfile (62%)
copy learning/tour-of-beam/backend/docker-compose.yml => sdks/python/apache_beam/examples/ml-orchestration/kfp/components/ingestion/component.yaml (56%)
copy learning/tour-of-beam/backend/docker-compose.yml => sdks/python/apache_beam/examples/ml-orchestration/kfp/components/ingestion/requirements.txt (59%)
create mode 100644 sdks/python/apache_beam/examples/ml-orchestration/kfp/components/ingestion/src/ingest.py
copy learning/tour-of-beam/backend/docker-compose.yml => sdks/python/apache_beam/examples/ml-orchestration/kfp/components/preprocessing/Dockerfile (62%)
create mode 100644 sdks/python/apache_beam/examples/ml-orchestration/kfp/components/preprocessing/component.yaml
copy learning/tour-of-beam/backend/docker-compose.yml => sdks/python/apache_beam/examples/ml-orchestration/kfp/components/preprocessing/requirements.txt (62%)
create mode 100644 sdks/python/apache_beam/examples/ml-orchestration/kfp/components/preprocessing/src/preprocess.py
copy learning/tour-of-beam/backend/docker-compose.yml => sdks/python/apache_beam/examples/ml-orchestration/kfp/components/train/Dockerfile (62%)
copy learning/tour-of-beam/backend/docker-compose.yml => sdks/python/apache_beam/examples/ml-orchestration/kfp/components/train/component.yaml (52%)
copy learning/tour-of-beam/backend/docker-compose.yml => sdks/python/apache_beam/examples/ml-orchestration/kfp/components/train/requirements.txt (62%)
create mode 100644 sdks/python/apache_beam/examples/ml-orchestration/kfp/components/train/src/train.py
create mode 100644 sdks/python/apache_beam/examples/ml-orchestration/kfp/pipeline.json
create mode 100644 sdks/python/apache_beam/examples/ml-orchestration/kfp/pipeline.py
copy learning/tour-of-beam/backend/docker-compose.yml => sdks/python/apache_beam/examples/ml-orchestration/kfp/requirements.txt (62%)
create mode 100644 sdks/python/apache_beam/examples/ml-orchestration/tfx/coco_captions_local.py
create mode 100644 sdks/python/apache_beam/examples/ml-orchestration/tfx/coco_captions_utils.py
copy learning/tour-of-beam/backend/docker-compose.yml => sdks/python/apache_beam/examples/ml-orchestration/tfx/requirements.txt (62%)
create mode 100644 website/www/site/content/en/documentation/ml/orchestration.md
mode change 100755 => 100644 website/www/site/content/en/documentation/ml/overview.md
create mode 100644 website/www/site/static/images/orchestrated-beam-pipeline.svg
create mode 100644 website/www/site/static/images/standalone-beam-pipeline.svg