You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "derrickaw (via GitHub)" <gi...@apache.org> on 2025/08/20 00:48:30 UTC
[PR] [YAML] - Jinja example [beam]
derrickaw opened a new pull request, #35914:
URL: https://github.com/apache/beam/pull/35914
#35909
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292084152
##########
sdks/python/apache_beam/yaml/examples/README.md:
##########
@@ -229,6 +233,27 @@ gcloud dataflow yaml run $JOB_NAME \
--region $REGION
```
+### Jinja
+
+Jinja [templatization](https://beam.apache.org/documentation/sdks/yaml/#jinja-templatization)
+can be used to build off of different contexts and/or with different
+configurations.
+
+Several examples will be created based on the already used word count example
+by leveraging Jinja templating engine for dynamic pipeline generation based on
+inputs from the user through `% include`, `% import`, and inheritance
+directives.
+
+% import:
+- [wordCount.yaml](#TODO: pending)
+
+% include:
+- [wordCount.yaml](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/yaml/examples/transforms/jinja/include/wordCount.yaml)
Review Comment:
> Similar pattern that Charles used on some of his ML work with Readme usage more in the applicable folder and not in the main one since multiple folders will be under Jinja.
I missed that Readme entirely when I did my review. SGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292581951
##########
sdks/python/apache_beam/yaml/examples/testing/examples_test.py:
##########
@@ -538,8 +557,64 @@
word = element.split('=')[1].split(',')[0].replace("'", '')
count = int(element.split('=')[2].replace(')', ''))
all_words += [word] * count
- random.shuffle(all_words)
+ return _wordcount_random_shuffler(test_spec, all_words, env)
+
+
+@YamlExamplesTestSuite.register_test_preprocessor(
+ ['test_wordCountInclude_yaml'])
+def _wordcount_test_preprocessor(
+ test_spec: dict, expected: List[str], env: TestEnvironment):
+ """
+ Preprocessor for the wordcount Jinja tests.
+
+ This preprocessor generates a random input file based on the expected output
+ of the wordcount example. This allows the test to verify the pipeline's
+ correctness without relying on a fixed input file.
+
+ Based on this expected output: # Row(output='king - 311')
+
+ Args:
+ test_spec: The dictionary representation of the YAML pipeline specification.
+ expected: A list of strings representing the expected output of the
+ pipeline.
+ env: The TestEnvironment object providing utilities for creating temporary
+ files.
+
+ Returns:
+ The modified test_spec dictionary with the input file path replaced.
+ """
+ all_words = []
+ for element in expected:
+ word = element.split('=')[1].split(' - ')[0].replace("'", '')
+ count = int(element.split('=')[1].split(' - ')[1].replace("')", ''))
+ all_words += [word] * count
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292063929
##########
sdks/python/apache_beam/yaml/examples/transforms/jinja/include/wordCountInclude.yaml:
##########
@@ -0,0 +1,71 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This examples reads from a public file stored on Google Cloud. This
+# requires authenticating with Google Cloud, or setting the file in
+#`ReadFromText` to a local file.
+#
+# To set up Application Default Credentials,
+# see https://cloud.google.com/docs/authentication/external/set-up-adc.
+#
+# This pipeline reads in a text file, counts distinct words found in the text,
+# then logs a row containing each word and its count.
+
+pipeline:
+ transforms:
+ - name: Read from GCS
+ type: ReadFromText
+ config:
+{% include 'apache_beam/yaml/examples/transforms/jinja/include/submodules/readFromText.yaml' %}
Review Comment:
1. We don't want to put the whole transform here because it limits us from changing names or inputs. Wanted to give an example.
2. Ok on names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292587296
##########
sdks/python/apache_beam/yaml/examples/testing/input_data.py:
##########
@@ -33,6 +33,36 @@ def text_data():
])
+def word_count_jinja_parameter_data():
+ return \
+ '{"readFromTextTransform": {"path": ' \
+ '"gs://dataflow-samples/shakespeare/kinglear.txt"}, ' \
+ '"mapToFieldsSplitConfig": {"language": "python", ' \
+ '"fields":{"value":"1"}}, ' \
+ '"explodeTransform":{"fields":"word"}, ' \
+ '"combineTransform":{"group_by":"word", "combine":{"value":"sum"}}, ' \
+ '"mapToFieldsCountConfig":{"language": "python", "fields":' \
+ '{"output": "word + \\" - \\" + str(value)"}}, ' \
+ '"writeToTextTransform":' \
+ '{"path":"gs://apache-beam-testing-derrickaw/wordCounts/"}}'
Review Comment:
This long, concatenated string is hard to read and maintain. For better readability, you could construct a Python dictionary and then dump it to a JSON string using `json.dumps`. This would make the structure much clearer and easier to modify.
Note: This change would require adding `import json` at the top of the file.
```python
def word_count_jinja_parameter_data():
params = {
"readFromTextTransform": {
"path": "gs://dataflow-samples/shakespeare/kinglear.txt"
},
"mapToFieldsSplitConfig": {
"language": "python",
"fields": {
"value": "1"
}
},
"explodeTransform": {
"fields": "word"
},
"combineTransform": {
"group_by": "word",
"combine": {
"value": "sum"
}
},
"mapToFieldsCountConfig": {
"language": "python",
"fields": {
"output": "word + \" - \" + str(value)"
}
},
"writeToTextTransform": {
"path": "gs://apache-beam-testing-derrickaw/wordCounts/"
}
}
return json.dumps(params)
```
##########
sdks/python/apache_beam/yaml/examples/transforms/jinja/include/submodules/combineTransform.yaml:
##########
@@ -0,0 +1,27 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# The Combine transform groups by each word and count the occurrences.
+
+ - name: Count words
+ type: Combine
+ config:
+ group_by:
+ - {{combineTransform.group_by}}
+ combine:
+ value: {{combineTransform.combine.value}}
Review Comment:
It's a common convention to have a newline at the end of a file. This file is missing one. Please consider adding it for consistency with other files in the project. This also applies to several other new YAML files in this pull request.
```yaml
value: {{combineTransform.combine.value}}
```
##########
sdks/python/apache_beam/yaml/examples/transforms/jinja/include/wordCountInclude.yaml:
##########
@@ -0,0 +1,66 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This examples reads from a public file stored on Google Cloud. This
+# requires authenticating with Google Cloud, or setting the file in
+#`ReadFromText` to a local file.
+#
+# To set up Application Default Credentials,
+# see https://cloud.google.com/docs/authentication/external/set-up-adc.
+#
+# This pipeline reads in a text file, counts distinct words found in the text,
+# then logs a row containing each word and its count.
+
+pipeline:
+ type: chain
+ transforms:
+# Read in text file
+{% include 'apache_beam/yaml/examples/transforms/jinja/include/submodules/readFromTextTransform.yaml' %}
+
+# Split words and count occurrences
+ - name: Split words
+ type: MapToFields
+ config:
+{% include 'apache_beam/yaml/examples/transforms/jinja/include/submodules/mapToFieldsSplitConfig.yaml' %}
+
+# Explode into individual words
+{% include 'apache_beam/yaml/examples/transforms/jinja/include/submodules/explodeTransform.yaml' %}
+
+# Group by word
+{% include 'apache_beam/yaml/examples/transforms/jinja/include/submodules/combineTransform.yaml' %}
+
+# Format output to a single string consisting of `word - count`
+ - name: Format output
+ type: MapToFields
+ config:
+{% include 'apache_beam/yaml/examples/transforms/jinja/include/submodules/mapToFieldsCountConfig.yaml' %}
+
+# Write to text file on GCS, locally, etc
+{% include 'apache_beam/yaml/examples/transforms/jinja/include/submodules/writeToTextTransform.yaml' %}
Review Comment:
The use of `{% include %}` at the beginning of the line makes this YAML file syntactically invalid before Jinja processing, and it's hard to understand the structure.
For better readability and maintainability, I suggest indenting the `{% include %}` tags to match the expected YAML structure and removing the corresponding indentation from the included submodule files. This makes both the main pipeline file and the included snippets more intuitive and closer to valid YAML.
For example, you could change this file to:
```yaml
...
transforms:
# Read in text file
{% include 'apache_beam/yaml/examples/transforms/jinja/include/submodules/readFromTextTransform.yaml' %}
# Split words and count occurrences
- name: Split words
type: MapToFields
config:
{% include 'apache_beam/yaml/examples/transforms/jinja/include/submodules/mapToFieldsSplitConfig.yaml' %}
...
```
And then adjust the included files accordingly. For example `readFromTextTransform.yaml` would become:
```yaml
- name: Read from GCS
type: ReadFromText
config:
path: {{readFromTextTransform.path}}
```
(with its leading indentation removed). This is a significant refactoring, but it would greatly improve the clarity of the example.
##########
sdks/python/apache_beam/yaml/examples/transforms/jinja/include/README.md:
##########
@@ -0,0 +1,63 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+## Jinja % include Pipeline
+
+This example leverages the `% include` Jinja directive by having one main
+pipeline and then submodules for each transformed used.
+
+General setup:
+```sh
+export PIPELINE_FILE=apache_beam/yaml/examples/transforms/jinja/include/wordCountInclude.yaml
+export KINGLEAR="gs://dataflow-samples/shakespeare/kinglear.txt"
+export TEMP_LOCATION="gs://MY-BUCKET/wordCounts/"
+
+cd <PATH_TO_BEAM_REPO>/beam/sdks/python
+```
+
+Multiline Run Example:
+```sh
+python -m apache_beam.yaml.main \
+ --yaml_pipeline_file="${PIPELINE_FILE}" \
+ --jinja_variables='{
+ "readFromTextTransform": {"path": "'"${KINGLEAR}"'"},
+ "mapToFieldsSplitConfig": {
+ "language": "python",
+ "fields": {
+ "value": "1"
+ }
+ },
+ "explodeTransform": {"fields": "word"},
+ "combineTransform": {
+ "group_by": "word",
+ "combine": {"value": "sum"}
+ },
+ "mapToFieldsCountConfig": {
+ "language": "python",
+ "fields": {"output": "word + \" - \" + str(value)"}
+ },
+ "writeToTextTransform": {"path": "'"${TEMP_LOCATION}"'"}
+ }'
+```
+
+Single Line Run Example:
+```sh
+python -m apache_beam.yaml.main --yaml_pipeline_file="${PIPELINE_FILE}" --jinja_variables='{"readFromTextTransform": {"path": "gs://dataflow-samples/shakespeare/kinglear.txt"}, "mapToFieldsSplitConfig": {"language": "python", "fields":{"value":"1"}}, "explodeTransform":{"fields":"word"}, "combineTransform":{"group_by":"word", "combine":{"value":"sum"}}, "mapToFieldsCountConfig":{"language": "python", "fields":{"output":"word + \" - \" + str(value)"}}, "writeToTextTransform":{"path":"${TEMP_LOCATION}"}}'
Review Comment:
In the single line run example, the `${TEMP_LOCATION}` variable is inside single quotes, which prevents shell expansion. This means it will be passed as a literal string instead of its value, causing the command to fail. To fix this, you should break out of the single quotes to allow for variable expansion, similar to how it's done in the multiline example.
```suggestion
python -m apache_beam.yaml.main --yaml_pipeline_file="${PIPELINE_FILE}" --jinja_variables='{"readFromTextTransform": {"path": "gs://dataflow-samples/shakespeare/kinglear.txt"}, "mapToFieldsSplitConfig": {"language": "python", "fields":{"value":"1"}}, "explodeTransform":{"fields":"word"}, "combineTransform":{"group_by":"word", "combine":{"value":"sum"}}, "mapToFieldsCountConfig": {"language": "python", "fields":{"output":"word + \" - \" + str(value)"}}, "writeToTextTransform":{"path":"'"${TEMP_LOCATION}"'"}}'
```
##########
sdks/python/apache_beam/yaml/examples/testing/input_data.py:
##########
@@ -33,6 +33,36 @@
])
+def word_count_jinja_parameter_data():
+ return \
+ '{"readFromTextTransform": {"path": ' \
+ '"gs://dataflow-samples/shakespeare/kinglear.txt"}, ' \
+ '"mapToFieldsSplitConfig": {"language": "python", ' \
+ '"fields":{"value":"1"}}, ' \
+ '"explodeTransform":{"fields":"word"}, ' \
+ '"combineTransform":{"group_by":"word", "combine":{"value":"sum"}}, ' \
+ '"mapToFieldsCountConfig":{"language": "python", "fields":' \
+ '{"output": "word + \\" - \\" + str(value)"}}, ' \
+ '"writeToTextTransform":' \
+ '{"path":"gs://apache-beam-testing-derrickaw/wordCounts/"}}'
+
+
+def word_count_jinja_template_data():
+ return \
+[('apache_beam/yaml/examples/transforms/jinja/'
+ 'include/submodules/readFromTextTransform.yaml'),
+ ('apache_beam/yaml/examples/transforms/jinja/'
+ 'include/submodules/mapToFieldsSplitConfig.yaml'),
+ ('apache_beam/yaml/examples/transforms/jinja/'
+ 'include/submodules/explodeTransform.yaml'),
+ ('apache_beam/yaml/examples/transforms/jinja/'
+ 'include/submodules/combineTransform.yaml'),
+ ('apache_beam/yaml/examples/transforms/jinja/'
+ 'include/submodules/mapToFieldsCountConfig.yaml'),
+ ('apache_beam/yaml/examples/transforms/jinja/'
+ 'include/submodules/writeToTextTransform.yaml')]
Review Comment:
The backslash for line continuation is unnecessary here, as Python automatically handles lists that span multiple lines. Removing it and formatting as a standard list would make the code cleaner and more idiomatic.
```python
def word_count_jinja_template_data():
return [
'apache_beam/yaml/examples/transforms/jinja/include/submodules/readFromTextTransform.yaml',
'apache_beam/yaml/examples/transforms/jinja/include/submodules/mapToFieldsSplitConfig.yaml',
'apache_beam/yaml/examples/transforms/jinja/include/submodules/explodeTransform.yaml',
'apache_beam/yaml/examples/transforms/jinja/include/submodules/combineTransform.yaml',
'apache_beam/yaml/examples/transforms/jinja/include/submodules/mapToFieldsCountConfig.yaml',
'apache_beam/yaml/examples/transforms/jinja/include/submodules/writeToTextTransform.yaml'
]
```
##########
sdks/python/apache_beam/yaml/examples/testing/examples_test.py:
##########
@@ -339,8 +343,21 @@ def test_yaml_example(self):
for i, line in enumerate(expected):
expected[i] = line.replace('# ', '').replace('\n', '')
expected = [line for line in expected if line]
+
+ raw_spec_string = ''.join(lines)
+ # Filter for any jinja preprocessor - this has to be done before other
+ # preprocessors.
+ jinja_preprocessor = [
+ preprocessor for preprocessor in custom_preprocessors
+ if 'jinja_preprocessor' in preprocessor.__name__
+ ]
+ if jinja_preprocessor:
+ jinja_preprocessor = jinja_preprocessor[0]
+ raw_spec_string = jinja_preprocessor(raw_spec_string)
+ custom_preprocessors.remove(jinja_preprocessor)
Review Comment:
This logic assumes that at most one `jinja_preprocessor` is found. If multiple preprocessors match the condition, it will silently use the first one, which might hide issues or lead to unexpected behavior. It would be more robust to ensure that only one such preprocessor exists by adding an assertion.
```suggestion
if jinja_preprocessor:
assert len(jinja_preprocessor) == 1, "Expected at most one jinja preprocessor."
jinja_preprocessor = jinja_preprocessor[0]
raw_spec_string = jinja_preprocessor(raw_spec_string)
custom_preprocessors.remove(jinja_preprocessor)
```
##########
sdks/python/apache_beam/yaml/examples/testing/examples_test.py:
##########
@@ -538,8 +558,64 @@
word = element.split('=')[1].split(',')[0].replace("'", '')
count = int(element.split('=')[2].replace(')', ''))
all_words += [word] * count
- random.shuffle(all_words)
+ return _wordcount_random_shuffler(test_spec, all_words, env)
+
+
+@YamlExamplesTestSuite.register_test_preprocessor(
+ ['test_wordCountInclude_yaml'])
+def _wordcount_test_preprocessor(
Review Comment:
The function `_wordcount_test_preprocessor` was recently renamed to `_wordcount_minimal_test_preprocessor` for clarity. Introducing a new function with the old name can be confusing. Consider renaming this function to be more specific, for example, `_wordcount_jinja_test_preprocessor`, to reflect its purpose and avoid ambiguity.
```suggestion
def _wordcount_jinja_test_preprocessor(
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292588025
##########
sdks/python/apache_beam/yaml/examples/transforms/jinja/include/README.md:
##########
@@ -0,0 +1,63 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+## Jinja % include Pipeline
+
+This example leverages the `% include` Jinja directive by having one main
+pipeline and then submodules for each transformed used.
+
+General setup:
+```sh
+export PIPELINE_FILE=apache_beam/yaml/examples/transforms/jinja/include/wordCountInclude.yaml
+export KINGLEAR="gs://dataflow-samples/shakespeare/kinglear.txt"
+export TEMP_LOCATION="gs://MY-BUCKET/wordCounts/"
+
+cd <PATH_TO_BEAM_REPO>/beam/sdks/python
+```
+
+Multiline Run Example:
+```sh
+python -m apache_beam.yaml.main \
+ --yaml_pipeline_file="${PIPELINE_FILE}" \
+ --jinja_variables='{
+ "readFromTextTransform": {"path": "'"${KINGLEAR}"'"},
+ "mapToFieldsSplitConfig": {
+ "language": "python",
+ "fields": {
+ "value": "1"
+ }
+ },
+ "explodeTransform": {"fields": "word"},
+ "combineTransform": {
+ "group_by": "word",
+ "combine": {"value": "sum"}
+ },
+ "mapToFieldsCountConfig": {
+ "language": "python",
+ "fields": {"output": "word + \" - \" + str(value)"}
+ },
+ "writeToTextTransform": {"path": "'"${TEMP_LOCATION}"'"}
+ }'
+```
+
+Single Line Run Example:
+```sh
+python -m apache_beam.yaml.main --yaml_pipeline_file="${PIPELINE_FILE}" --jinja_variables='{"readFromTextTransform": {"path": "gs://dataflow-samples/shakespeare/kinglear.txt"}, "mapToFieldsSplitConfig": {"language": "python", "fields":{"value":"1"}}, "explodeTransform":{"fields":"word"}, "combineTransform":{"group_by":"word", "combine":{"value":"sum"}}, "mapToFieldsCountConfig":{"language": "python", "fields":{"output":"word + \" - \" + str(value)"}}, "writeToTextTransform":{"path":"${TEMP_LOCATION}"}}'
Review Comment:
its fine, it works
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #35914:
URL: https://github.com/apache/beam/pull/35914#issuecomment-3212175152
/gemini review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292590837
##########
sdks/python/apache_beam/yaml/examples/testing/examples_test.py:
##########
@@ -538,8 +558,64 @@
word = element.split('=')[1].split(',')[0].replace("'", '')
count = int(element.split('=')[2].replace(')', ''))
all_words += [word] * count
- random.shuffle(all_words)
+ return _wordcount_random_shuffler(test_spec, all_words, env)
+
+
+@YamlExamplesTestSuite.register_test_preprocessor(
+ ['test_wordCountInclude_yaml'])
+def _wordcount_test_preprocessor(
Review Comment:
sounds reasonable, done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja example [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35914:
URL: https://github.com/apache/beam/pull/35914#issuecomment-3207155184
Assigning reviewers:
R: @liferoad for label python.
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292582922
##########
sdks/python/apache_beam/yaml/examples/testing/examples_test.py:
##########
@@ -1175,6 +1251,51 @@ def _batch_log_analysis_test_preprocessor(
return test_spec
+@YamlExamplesTestSuite.register_test_preprocessor(
+ ['test_wordCountInclude_yaml'])
+def _jinja_preprocessor(raw_spec_string: str):
+ """
+ Preprocessor for Jinja-based YAML tests.
+
+ This function takes a raw YAML string, which is treated as a Jinja2
+ template, and renders it to produce the final pipeline specification.
+ It specifically handles templates that use the `{% include ... %}`
+ directive by manually loading the content of the included files from the
+ filesystem.
+
+ The Jinja variables required for rendering are loaded from a predefined
+ data source.
+
+ Args:
+ raw_spec_string: A string containing the raw YAML content, which is a
+ Jinja2 template.
+
+ Returns:
+ A string containing the fully rendered YAML pipeline specification.
+ """
+
+ jinja_variables = json.loads(input_data.word_count_jinja_parameter_data())
+
+ test_file_dir = os.path.dirname(__file__)
+ sdk_root = os.path.abspath(os.path.join(test_file_dir, '../../../..'))
+
+ include_files = input_data.word_count_jinja_template_data()
+ mock_templates = {'main_template': raw_spec_string}
+ for file_path in include_files:
+ full_path = os.path.join(sdk_root, file_path)
+ with open(full_path, 'r', encoding='utf-8') as f:
+ mock_templates[file_path] = f.read()
+
+ # Can't use the standard expand_jinja method due to it not supporting
+ # `% include` jinja templization.
+ # TODO: Maybe update expand_jinja to handle this case.
Review Comment:
added todo link
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292589161
##########
sdks/python/apache_beam/yaml/examples/testing/examples_test.py:
##########
@@ -339,8 +343,21 @@ def test_yaml_example(self):
for i, line in enumerate(expected):
expected[i] = line.replace('# ', '').replace('\n', '')
expected = [line for line in expected if line]
+
+ raw_spec_string = ''.join(lines)
+ # Filter for any jinja preprocessor - this has to be done before other
+ # preprocessors.
+ jinja_preprocessor = [
+ preprocessor for preprocessor in custom_preprocessors
+ if 'jinja_preprocessor' in preprocessor.__name__
+ ]
+ if jinja_preprocessor:
+ jinja_preprocessor = jinja_preprocessor[0]
+ raw_spec_string = jinja_preprocessor(raw_spec_string)
+ custom_preprocessors.remove(jinja_preprocessor)
Review Comment:
ignoring
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on PR #35914:
URL: https://github.com/apache/beam/pull/35914#issuecomment-3214417750
Run Python_Integration PreCommit 3.9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on PR #35914:
URL: https://github.com/apache/beam/pull/35914#issuecomment-3214430584
Run Python_ML PreCommit 3.10
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292036590
##########
sdks/python/apache_beam/yaml/examples/README.md:
##########
@@ -229,6 +233,27 @@ gcloud dataflow yaml run $JOB_NAME \
--region $REGION
```
+### Jinja
+
+Jinja [templatization](https://beam.apache.org/documentation/sdks/yaml/#jinja-templatization)
+can be used to build off of different contexts and/or with different
+configurations.
+
+Several examples will be created based on the already used word count example
+by leveraging Jinja templating engine for dynamic pipeline generation based on
+inputs from the user through `% include`, `% import`, and inheritance
+directives.
+
+% import:
Review Comment:
Should these be code blocks? Also looks like some pieces are still missing, we should probably not include them for now
##########
sdks/python/apache_beam/yaml/examples/README.md:
##########
@@ -229,6 +233,27 @@ gcloud dataflow yaml run $JOB_NAME \
--region $REGION
```
+### Jinja
+
+Jinja [templatization](https://beam.apache.org/documentation/sdks/yaml/#jinja-templatization)
+can be used to build off of different contexts and/or with different
+configurations.
+
+Several examples will be created based on the already used word count example
+by leveraging Jinja templating engine for dynamic pipeline generation based on
+inputs from the user through `% include`, `% import`, and inheritance
+directives.
+
+% import:
+- [wordCount.yaml](#TODO: pending)
+
+% include:
+- [wordCount.yaml](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/yaml/examples/transforms/jinja/include/wordCount.yaml)
Review Comment:
Should this be `wordCountInclude.yaml`?
##########
sdks/python/apache_beam/yaml/examples/transforms/jinja/include/wordCountInclude.yaml:
##########
@@ -0,0 +1,71 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This examples reads from a public file stored on Google Cloud. This
+# requires authenticating with Google Cloud, or setting the file in
+#`ReadFromText` to a local file.
+#
+# To set up Application Default Credentials,
+# see https://cloud.google.com/docs/authentication/external/set-up-adc.
+#
+# This pipeline reads in a text file, counts distinct words found in the text,
+# then logs a row containing each word and its count.
+
+pipeline:
+ transforms:
+ - name: Read from GCS
+ type: ReadFromText
+ config:
+{% include 'apache_beam/yaml/examples/transforms/jinja/include/submodules/readFromText.yaml' %}
Review Comment:
Practically, I don't think it would be common to include just the config like this, especially if it is very simple (and also parameterized)
##########
sdks/python/apache_beam/yaml/examples/transforms/jinja/include/wordCountInclude.yaml:
##########
@@ -0,0 +1,71 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This examples reads from a public file stored on Google Cloud. This
+# requires authenticating with Google Cloud, or setting the file in
+#`ReadFromText` to a local file.
+#
+# To set up Application Default Credentials,
+# see https://cloud.google.com/docs/authentication/external/set-up-adc.
+#
+# This pipeline reads in a text file, counts distinct words found in the text,
+# then logs a row containing each word and its count.
+
+pipeline:
+ transforms:
+ - name: Read from GCS
+ type: ReadFromText
+ config:
+{% include 'apache_beam/yaml/examples/transforms/jinja/include/submodules/readFromText.yaml' %}
Review Comment:
Rather than just including the config, could we maybe map in whole transforms for simple ones like this?
Since this is an example, we should also explain what we're doing in comments
##########
sdks/python/apache_beam/yaml/examples/README.md:
##########
@@ -229,6 +233,27 @@ gcloud dataflow yaml run $JOB_NAME \
--region $REGION
```
+### Jinja
+
+Jinja [templatization](https://beam.apache.org/documentation/sdks/yaml/#jinja-templatization)
+can be used to build off of different contexts and/or with different
+configurations.
+
+Several examples will be created based on the already used word count example
+by leveraging Jinja templating engine for dynamic pipeline generation based on
+inputs from the user through `% include`, `% import`, and inheritance
+directives.
+
+% import:
+- [wordCount.yaml](#TODO: pending)
+
+% include:
+- [wordCount.yaml](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/yaml/examples/transforms/jinja/include/wordCount.yaml)
Review Comment:
It would also be good to give an example of invoking this (and how you would pass in appropriate parameters)
##########
sdks/python/apache_beam/yaml/examples/transforms/jinja/include/wordCountInclude.yaml:
##########
@@ -0,0 +1,71 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This examples reads from a public file stored on Google Cloud. This
+# requires authenticating with Google Cloud, or setting the file in
+#`ReadFromText` to a local file.
+#
+# To set up Application Default Credentials,
+# see https://cloud.google.com/docs/authentication/external/set-up-adc.
+#
+# This pipeline reads in a text file, counts distinct words found in the text,
+# then logs a row containing each word and its count.
+
+pipeline:
+ transforms:
+ - name: Read from GCS
+ type: ReadFromText
+ config:
+{% include 'apache_beam/yaml/examples/transforms/jinja/include/submodules/readFromText.yaml' %}
Review Comment:
I'd also recommend using more descriptive file names. So for examples:
`readFromTextTransform.yaml` (assuming we put the whole transform there)
or
`mapToFieldsCountConfig.yaml` (assuming we keep that one as config since it is more complicated)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292063929
##########
sdks/python/apache_beam/yaml/examples/transforms/jinja/include/wordCountInclude.yaml:
##########
@@ -0,0 +1,71 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This examples reads from a public file stored on Google Cloud. This
+# requires authenticating with Google Cloud, or setting the file in
+#`ReadFromText` to a local file.
+#
+# To set up Application Default Credentials,
+# see https://cloud.google.com/docs/authentication/external/set-up-adc.
+#
+# This pipeline reads in a text file, counts distinct words found in the text,
+# then logs a row containing each word and its count.
+
+pipeline:
+ transforms:
+ - name: Read from GCS
+ type: ReadFromText
+ config:
+{% include 'apache_beam/yaml/examples/transforms/jinja/include/submodules/readFromText.yaml' %}
Review Comment:
1. We don't want to put the whole transform here because it limits us from changing names or inputs. Wanted to give an example.
2. Ok on better names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292582347
##########
sdks/python/apache_beam/yaml/examples/README.md:
##########
@@ -229,6 +233,27 @@ gcloud dataflow yaml run $JOB_NAME \
--region $REGION
```
+### Jinja
+
+Jinja [templatization](https://beam.apache.org/documentation/sdks/yaml/#jinja-templatization)
+can be used to build off of different contexts and/or with different
+configurations.
+
+Several examples will be created based on the already used word count example
+by leveraging Jinja templating engine for dynamic pipeline generation based on
+inputs from the user through `% include`, `% import`, and inheritance
+directives.
+
+% import:
+- [wordCount.yaml](#TODO: pending)
+
+% include:
+- [wordCount.yaml](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/yaml/examples/transforms/jinja/include/wordCount.yaml)
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292592257
##########
sdks/python/apache_beam/yaml/examples/testing/input_data.py:
##########
@@ -33,6 +33,36 @@ def text_data():
])
+def word_count_jinja_parameter_data():
+ return \
+ '{"readFromTextTransform": {"path": ' \
+ '"gs://dataflow-samples/shakespeare/kinglear.txt"}, ' \
+ '"mapToFieldsSplitConfig": {"language": "python", ' \
+ '"fields":{"value":"1"}}, ' \
+ '"explodeTransform":{"fields":"word"}, ' \
+ '"combineTransform":{"group_by":"word", "combine":{"value":"sum"}}, ' \
+ '"mapToFieldsCountConfig":{"language": "python", "fields":' \
+ '{"output": "word + \\" - \\" + str(value)"}}, ' \
+ '"writeToTextTransform":' \
+ '{"path":"gs://apache-beam-testing-derrickaw/wordCounts/"}}'
Review Comment:
ok, thats reasonable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on PR #35914:
URL: https://github.com/apache/beam/pull/35914#issuecomment-3206844060
Run Python PreCommit 3.11
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on PR #35914:
URL: https://github.com/apache/beam/pull/35914#issuecomment-3214410367
Run Prism_Python PreCommit 3.12
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm merged PR #35914:
URL: https://github.com/apache/beam/pull/35914
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292594851
##########
sdks/python/apache_beam/yaml/examples/transforms/jinja/include/wordCountInclude.yaml:
##########
@@ -0,0 +1,66 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This examples reads from a public file stored on Google Cloud. This
+# requires authenticating with Google Cloud, or setting the file in
+#`ReadFromText` to a local file.
+#
+# To set up Application Default Credentials,
+# see https://cloud.google.com/docs/authentication/external/set-up-adc.
+#
+# This pipeline reads in a text file, counts distinct words found in the text,
+# then logs a row containing each word and its count.
+
+pipeline:
+ type: chain
+ transforms:
+# Read in text file
+{% include 'apache_beam/yaml/examples/transforms/jinja/include/submodules/readFromTextTransform.yaml' %}
+
+# Split words and count occurrences
+ - name: Split words
+ type: MapToFields
+ config:
+{% include 'apache_beam/yaml/examples/transforms/jinja/include/submodules/mapToFieldsSplitConfig.yaml' %}
+
+# Explode into individual words
+{% include 'apache_beam/yaml/examples/transforms/jinja/include/submodules/explodeTransform.yaml' %}
+
+# Group by word
+{% include 'apache_beam/yaml/examples/transforms/jinja/include/submodules/combineTransform.yaml' %}
+
+# Format output to a single string consisting of `word - count`
+ - name: Format output
+ type: MapToFields
+ config:
+{% include 'apache_beam/yaml/examples/transforms/jinja/include/submodules/mapToFieldsCountConfig.yaml' %}
+
+# Write to text file on GCS, locally, etc
+{% include 'apache_beam/yaml/examples/transforms/jinja/include/submodules/writeToTextTransform.yaml' %}
Review Comment:
i tried that, but the spacing gets weird. Its not as simple as you assumed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on PR #35914:
URL: https://github.com/apache/beam/pull/35914#issuecomment-3207147302
assign set of reviewers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292056256
##########
sdks/python/apache_beam/yaml/examples/README.md:
##########
@@ -229,6 +233,27 @@ gcloud dataflow yaml run $JOB_NAME \
--region $REGION
```
+### Jinja
+
+Jinja [templatization](https://beam.apache.org/documentation/sdks/yaml/#jinja-templatization)
+can be used to build off of different contexts and/or with different
+configurations.
+
+Several examples will be created based on the already used word count example
+by leveraging Jinja templating engine for dynamic pipeline generation based on
+inputs from the user through `% include`, `% import`, and inheritance
+directives.
+
+% import:
+- [wordCount.yaml](#TODO: pending)
+
+% include:
+- [wordCount.yaml](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/yaml/examples/transforms/jinja/include/wordCount.yaml)
Review Comment:
I had it here before, but I moved it to the other Readme. Similar pattern that Charles used on some of his ML work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #35914:
URL: https://github.com/apache/beam/pull/35914#issuecomment-3211692830
## [Codecov](https://app.codecov.io/gh/apache/beam/pull/35914?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 56.67%. Comparing base ([`f59a45e`](https://app.codecov.io/gh/apache/beam/commit/f59a45e63ae779101a8ad441a512d4205396cca4?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`d741ff8`](https://app.codecov.io/gh/apache/beam/commit/d741ff8a4a39d977a9044d78dfdb6e9037658d53?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 11 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #35914 +/- ##
=========================================
Coverage 56.67% 56.67%
Complexity 3380 3380
=========================================
Files 1219 1219
Lines 184561 184568 +7
Branches 3507 3507
=========================================
+ Hits 104595 104611 +16
+ Misses 76639 76630 -9
Partials 3327 3327
```
| [Flag](https://app.codecov.io/gh/apache/beam/pull/35914/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [python](https://app.codecov.io/gh/apache/beam/pull/35914/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `80.82% <100.00%> (+0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/beam/pull/35914?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292087497
##########
sdks/python/apache_beam/yaml/examples/README.md:
##########
@@ -229,6 +233,27 @@ gcloud dataflow yaml run $JOB_NAME \
--region $REGION
```
+### Jinja
+
+Jinja [templatization](https://beam.apache.org/documentation/sdks/yaml/#jinja-templatization)
+can be used to build off of different contexts and/or with different
+configurations.
+
+Several examples will be created based on the already used word count example
+by leveraging Jinja templating engine for dynamic pipeline generation based on
+inputs from the user through `% include`, `% import`, and inheritance
+directives.
+
+% import:
+- [wordCount.yaml](#TODO: pending)
+
+% include:
+- [wordCount.yaml](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/yaml/examples/transforms/jinja/include/wordCount.yaml)
Review Comment:
Maybe we could just link to that folder or the Readme here then? I worry that it will be easy to miss if you just follow the link (basically how I reviewed this)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292592673
##########
sdks/python/apache_beam/yaml/examples/testing/input_data.py:
##########
@@ -33,6 +33,36 @@
])
+def word_count_jinja_parameter_data():
+ return \
+ '{"readFromTextTransform": {"path": ' \
+ '"gs://dataflow-samples/shakespeare/kinglear.txt"}, ' \
+ '"mapToFieldsSplitConfig": {"language": "python", ' \
+ '"fields":{"value":"1"}}, ' \
+ '"explodeTransform":{"fields":"word"}, ' \
+ '"combineTransform":{"group_by":"word", "combine":{"value":"sum"}}, ' \
+ '"mapToFieldsCountConfig":{"language": "python", "fields":' \
+ '{"output": "word + \\" - \\" + str(value)"}}, ' \
+ '"writeToTextTransform":' \
+ '{"path":"gs://apache-beam-testing-derrickaw/wordCounts/"}}'
+
+
+def word_count_jinja_template_data():
+ return \
+[('apache_beam/yaml/examples/transforms/jinja/'
+ 'include/submodules/readFromTextTransform.yaml'),
+ ('apache_beam/yaml/examples/transforms/jinja/'
+ 'include/submodules/mapToFieldsSplitConfig.yaml'),
+ ('apache_beam/yaml/examples/transforms/jinja/'
+ 'include/submodules/explodeTransform.yaml'),
+ ('apache_beam/yaml/examples/transforms/jinja/'
+ 'include/submodules/combineTransform.yaml'),
+ ('apache_beam/yaml/examples/transforms/jinja/'
+ 'include/submodules/mapToFieldsCountConfig.yaml'),
+ ('apache_beam/yaml/examples/transforms/jinja/'
+ 'include/submodules/writeToTextTransform.yaml')]
Review Comment:
80 character limit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on PR #35914:
URL: https://github.com/apache/beam/pull/35914#issuecomment-3211640103
Run Python_Transforms PreCommit 3.10
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292056256
##########
sdks/python/apache_beam/yaml/examples/README.md:
##########
@@ -229,6 +233,27 @@ gcloud dataflow yaml run $JOB_NAME \
--region $REGION
```
+### Jinja
+
+Jinja [templatization](https://beam.apache.org/documentation/sdks/yaml/#jinja-templatization)
+can be used to build off of different contexts and/or with different
+configurations.
+
+Several examples will be created based on the already used word count example
+by leveraging Jinja templating engine for dynamic pipeline generation based on
+inputs from the user through `% include`, `% import`, and inheritance
+directives.
+
+% import:
+- [wordCount.yaml](#TODO: pending)
+
+% include:
+- [wordCount.yaml](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/yaml/examples/transforms/jinja/include/wordCount.yaml)
Review Comment:
1. I had it here before, but I moved it to the other Readme. Will update name.
2. Similar pattern that Charles used on some of his ML work with Readme usage more in the applicable folder and not in the main one since multiple folders will be under Jinja.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292205965
##########
sdks/python/apache_beam/yaml/examples/testing/input_data.py:
##########
@@ -33,6 +33,34 @@ def text_data():
])
+def word_count_jinja_parameter_data():
+ return \
+ '{"readFromText": {"path": ' \
+ '"gs://dataflow-samples/shakespeare/kinglear.txt"}, ' \
+ '"mapToFields_split": {"language": "python", "fields":{"value":"1",\
+ "word":{"callable":"apache_beam/yaml/examples/transforms/jinja/\
+ include/submodules/wordCount_mapper.txt"}}}, ' \
+ '"explode":{"fields":"word"}, ' \
+ '"combine":{"group_by":"word", "combine":{"value":"sum"}}, ' \
+ '"mapToFields_count":{"language": "python", "fields":' \
+ '{"output": "word + \\" - \\" + str(value)"}}, ' \
+ '"writeToText":{"path":"gs://apache-beam-testing-derrickaw/wordCounts/"}}'
Review Comment:
This function constructs a JSON string by concatenating multiple string literals, which is hard to read and maintain. It's better to define a Python dictionary and use `json.dumps` to serialize it.
More importantly, the `callable` path `.../wordCount_mapper.txt` points to a file that is not included in this pull request. This will cause test failures and is a critical issue that needs to be resolved.
```python
def word_count_jinja_parameter_data():
import json
data = {
"readFromText": {
"path": "gs://dataflow-samples/shakespeare/kinglear.txt"
},
"mapToFields_split": {
"language": "python",
"fields": {
"value": "1",
"word": {
"callable": "apache_beam/yaml/examples/transforms/jinja/include/submodules/wordCount_mapper.txt"
}
}
},
"explode": {
"fields": "word"
},
"combine": {
"group_by": "word",
"combine": {
"value": "sum"
}
},
"mapToFields_count": {
"language": "python",
"fields": {
"output": "word + \" - \" + str(value)"
}
},
"writeToText": {
"path": "gs://apache-beam-testing-derrickaw/wordCounts/"
}
}
return json.dumps(data)
```
##########
sdks/python/apache_beam/yaml/examples/transforms/jinja/include/submodules/mapToFields_split.yaml:
##########
@@ -0,0 +1,40 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This examples reads from a public file stored on Google Cloud. This
+# requires authenticating with Google Cloud, or setting the file in
+#`ReadFromText` to a local file.
+#
+# To set up Application Default Credentials,
+# see https://cloud.google.com/docs/authentication/external/set-up-adc.
+#
+# This pipeline reads in a text file, counts distinct words found in the text,
+# then logs a row containing each word and its count.
+
+ language: {{mapToFields_split.language}}
+ fields:
+ word:
+ callable: |-
+ # TODO: Including another file here works fine, but if the file
+ # has a license header or other irrevalent comments, it will break
+ # the pipeline. Need to investigate more on Jinja filtering
+ # in the expand_jinja method or some other way.
Review Comment:
The TODO comment raises a valid concern about including files with license headers or other comments. If this can break the pipeline, it's a significant issue with the current approach of handling Jinja includes. This should be investigated and fixed. If a fix is not straightforward, an issue should be filed to track this problem, and the TODO should reference it.
##########
sdks/python/apache_beam/yaml/examples/README.md:
##########
@@ -229,6 +233,27 @@ gcloud dataflow yaml run $JOB_NAME \
--region $REGION
```
+### Jinja
+
+Jinja [templatization](https://beam.apache.org/documentation/sdks/yaml/#jinja-templatization)
+can be used to build off of different contexts and/or with different
+configurations.
+
+Several examples will be created based on the already used word count example
+by leveraging Jinja templating engine for dynamic pipeline generation based on
+inputs from the user through `% include`, `% import`, and inheritance
+directives.
+
+% import:
+- [wordCount.yaml](#TODO: pending)
+
+% include:
+- [wordCount.yaml](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/yaml/examples/transforms/jinja/include/wordCount.yaml)
Review Comment:
The link for the `% include` example appears to be incorrect. It points to `wordCount.yaml`, but the new example file is `wordCountInclude.yaml`. Both the link text and the URL should be updated to reflect the correct filename to avoid confusion.
```suggestion
- [wordCountInclude.yaml](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/yaml/examples/transforms/jinja/include/wordCountInclude.yaml)
```
##########
sdks/python/apache_beam/yaml/examples/transforms/jinja/include/README.md:
##########
@@ -0,0 +1,63 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+## Jinja % include Pipeline
+
+This example leverages the `% include` Jinja directive by having one main
+pipeline and then submodules for each transformed used.
+
+General setup:
+```sh
+export PIPELINE_FILE=apache_beam/yaml/examples/transforms/jinja/wordCount.yaml
Review Comment:
The `PIPELINE_FILE` variable is set to `.../wordCount.yaml`, but the example file is `wordCountInclude.yaml`. This should be corrected to avoid confusion for users following the instructions.
```suggestion
export PIPELINE_FILE=apache_beam/yaml/examples/transforms/jinja/include/wordCountInclude.yaml
```
##########
sdks/python/apache_beam/yaml/examples/transforms/jinja/include/README.md:
##########
@@ -0,0 +1,63 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+## Jinja % include Pipeline
+
+This example leverages the `% include` Jinja directive by having one main
+pipeline and then submodules for each transformed used.
+
+General setup:
+```sh
+export PIPELINE_FILE=apache_beam/yaml/examples/transforms/jinja/wordCount.yaml
+export KINGLEAR="gs://dataflow-samples/shakespeare/kinglear.txt"
+export TEMP_LOCATION="gs://MY-BUCKET/wordCounts/"
+
+cd <PATH_TO_BEAM_REPO>/beam/sdks/python
+```
+
+Multiline Run Example:
+```sh
+python -m apache_beam.yaml.main \
+ --yaml_pipeline_file=apache_beam/yaml/examples/transforms/jinja/include/wordCountInclude.yaml \
+ --jinja_variables='{
+ "readFromText": {"path": "'"${KINGLEAR}"'"},
+ "mapToFields_split": {
+ "language": "python",
+ "fields": {
+ "value": "1"
+ }
+ },
+ "explode": {"fields": "word"},
+ "combine": {
+ "group_by": "word",
+ "combine": {"value": "sum"}
+ },
+ "mapToFields_count": {
+ "language": "python",
+ "fields": {"output": "word + \" - \" + str(value)"}
+ },
+ "writeToText": {"path": "'"${TEMP_LOCATION}"'"}
+ }'
+```
+
+Single Line Run Example:
+```sh
+python -m apache_beam.yaml.main --yaml_pipeline_file=apache_beam/yaml/examples/transforms/jinja/include/wordCountInclude.yaml --jinja_variables='{"readFromText": {"path": "gs://dataflow-samples/shakespeare/kinglear.txt"}, "mapToFields_split": {"language": "python", "fields":{"value":"1"}}, "explode":{"fields":"word"}, "combine":{"group_by":"word", "combine":{"value":"sum"}}, "mapToFields_count":{"language": "python", "fields":{"output":"word + \" - \" + str(value)"}}, "writeToText":{"path":"${TEMP_LOCATION}"}}'
Review Comment:
The single-line run command will fail because the shell variable `${TEMP_LOCATION}` is inside single quotes, which prevents its expansion. To fix this, you can use a mix of single and double quotes as shown in the multiline example.
```suggestion
python -m apache_beam.yaml.main --yaml_pipeline_file=apache_beam/yaml/examples/transforms/jinja/include/wordCountInclude.yaml --jinja_variables='{"readFromText": {"path": "gs://dataflow-samples/shakespeare/kinglear.txt"}, "mapToFields_split": {"language": "python", "fields":{"value":"1"}}, "explode":{"fields":"word"}, "combine":{"group_by":"word", "combine":{"value":"sum"}}, "mapToFields_count":{"language": "python", "fields":{"output":"word + \" - \" + str(value)"}}, "writeToText":{"path":"'"${TEMP_LOCATION}"'"}}'
```
##########
sdks/python/apache_beam/yaml/examples/testing/examples_test.py:
##########
@@ -1175,6 +1251,51 @@ def _batch_log_analysis_test_preprocessor(
return test_spec
+@YamlExamplesTestSuite.register_test_preprocessor(
+ ['test_wordCountInclude_yaml'])
+def _jinja_preprocessor(raw_spec_string: str):
+ """
+ Preprocessor for Jinja-based YAML tests.
+
+ This function takes a raw YAML string, which is treated as a Jinja2
+ template, and renders it to produce the final pipeline specification.
+ It specifically handles templates that use the `{% include ... %}`
+ directive by manually loading the content of the included files from the
+ filesystem.
+
+ The Jinja variables required for rendering are loaded from a predefined
+ data source.
+
+ Args:
+ raw_spec_string: A string containing the raw YAML content, which is a
+ Jinja2 template.
+
+ Returns:
+ A string containing the fully rendered YAML pipeline specification.
+ """
+
+ jinja_variables = json.loads(input_data.word_count_jinja_parameter_data())
+
+ test_file_dir = os.path.dirname(__file__)
+ sdk_root = os.path.abspath(os.path.join(test_file_dir, '../../../..'))
+
+ include_files = input_data.word_count_jinja_template_data()
+ mock_templates = {'main_template': raw_spec_string}
+ for file_path in include_files:
+ full_path = os.path.join(sdk_root, file_path)
+ with open(full_path, 'r', encoding='utf-8') as f:
+ mock_templates[file_path] = f.read()
+
+ # Can't use the standard expand_jinja method due to it not supporting
+ # `% include` jinja templization.
+ # TODO: Maybe update expand_jinja to handle this case.
Review Comment:
The `_jinja_preprocessor` raises a concern about including files with license headers or other comments, stating it can break the pipeline. This is a significant issue with the current approach of handling Jinja includes. This should be investigated and fixed. If a fix is not straightforward, an issue should be filed to track this problem, and the TODO should reference it.
##########
sdks/python/apache_beam/yaml/examples/testing/examples_test.py:
##########
@@ -538,8 +557,64 @@
word = element.split('=')[1].split(',')[0].replace("'", '')
count = int(element.split('=')[2].replace(')', ''))
all_words += [word] * count
- random.shuffle(all_words)
+ return _wordcount_random_shuffler(test_spec, all_words, env)
+
+
+@YamlExamplesTestSuite.register_test_preprocessor(
+ ['test_wordCountInclude_yaml'])
+def _wordcount_test_preprocessor(
Review Comment:
The new preprocessor function is named `_wordcount_test_preprocessor`. This name was previously used by another function that was just renamed to `_wordcount_minimal_test_preprocessor`. To avoid confusion and improve clarity, it would be better to give this new function a more specific name, such as `_wordcount_include_test_preprocessor`.
```suggestion
def _wordcount_include_test_preprocessor(
```
##########
sdks/python/apache_beam/yaml/examples/testing/examples_test.py:
##########
@@ -538,8 +557,64 @@
word = element.split('=')[1].split(',')[0].replace("'", '')
count = int(element.split('=')[2].replace(')', ''))
all_words += [word] * count
- random.shuffle(all_words)
+ return _wordcount_random_shuffler(test_spec, all_words, env)
+
+
+@YamlExamplesTestSuite.register_test_preprocessor(
+ ['test_wordCountInclude_yaml'])
+def _wordcount_test_preprocessor(
+ test_spec: dict, expected: List[str], env: TestEnvironment):
+ """
+ Preprocessor for the wordcount Jinja tests.
+
+ This preprocessor generates a random input file based on the expected output
+ of the wordcount example. This allows the test to verify the pipeline's
+ correctness without relying on a fixed input file.
+
+ Based on this expected output: # Row(output='king - 311')
+
+ Args:
+ test_spec: The dictionary representation of the YAML pipeline specification.
+ expected: A list of strings representing the expected output of the
+ pipeline.
+ env: The TestEnvironment object providing utilities for creating temporary
+ files.
+
+ Returns:
+ The modified test_spec dictionary with the input file path replaced.
+ """
+ all_words = []
+ for element in expected:
+ word = element.split('=')[1].split(' - ')[0].replace("'", '')
+ count = int(element.split('=')[1].split(' - ')[1].replace("')", ''))
+ all_words += [word] * count
Review Comment:
Parsing the expected output string with multiple `split()` calls is fragile and can easily break if the string format changes. Using a regular expression would be more robust and readable.
```suggestion
import re
all_words = []
for element in expected:
match = re.search(r"output='(.*?) - (\\d+)'", element)
if match:
word, count_str = match.groups()
all_words += [word] * int(count_str)
```
##########
sdks/python/apache_beam/yaml/examples/testing/input_data.py:
##########
@@ -33,6 +33,34 @@
])
+def word_count_jinja_parameter_data():
+ return \
+ '{"readFromText": {"path": ' \
+ '"gs://dataflow-samples/shakespeare/kinglear.txt"}, ' \
+ '"mapToFields_split": {"language": "python", "fields":{"value":"1",\
+ "word":{"callable":"apache_beam/yaml/examples/transforms/jinja/\
+ include/submodules/wordCount_mapper.txt"}}}, ' \
+ '"explode":{"fields":"word"}, ' \
+ '"combine":{"group_by":"word", "combine":{"value":"sum"}}, ' \
+ '"mapToFields_count":{"language": "python", "fields":' \
+ '{"output": "word + \\" - \\" + str(value)"}}, ' \
+ '"writeToText":{"path":"gs://apache-beam-testing-derrickaw/wordCounts/"}}'
+
+
+def word_count_jinja_template_data():
+ return \
+[('apache_beam/yaml/examples/transforms/jinja/'
+ 'include/submodules/readFromText.yaml'),
+ ('apache_beam/yaml/examples/transforms/jinja/'
+ 'include/submodules/mapToFields_split.yaml'),
+ 'apache_beam/yaml/examples/transforms/jinja/include/submodules/explode.yaml',
+ 'apache_beam/yaml/examples/transforms/jinja/include/submodules/combine.yaml',
+ ('apache_beam/yaml/examples/transforms/jinja/'
+ 'include/submodules/mapToFields_count.yaml'),
+ ('apache_beam/yaml/examples/transforms/jinja/'
+ 'include/submodules/writeToText.yaml')]
Review Comment:
The list returned by this function is formatted inconsistently, with a mix of strings and strings inside parentheses. For consistency and readability, it should be a simple list of strings.
```python
def word_count_jinja_template_data():
return [
'apache_beam/yaml/examples/transforms/jinja/include/submodules/readFromText.yaml',
'apache_beam/yaml/examples/transforms/jinja/include/submodules/mapToFields_split.yaml',
'apache_beam/yaml/examples/transforms/jinja/include/submodules/explode.yaml',
'apache_beam/yaml/examples/transforms/jinja/include/submodules/combine.yaml',
'apache_beam/yaml/examples/transforms/jinja/include/submodules/mapToFields_count.yaml',
'apache_beam/yaml/examples/transforms/jinja/include/submodules/writeToText.yaml'
]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292582570
##########
sdks/python/apache_beam/yaml/examples/transforms/jinja/include/README.md:
##########
@@ -0,0 +1,63 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+## Jinja % include Pipeline
+
+This example leverages the `% include` Jinja directive by having one main
+pipeline and then submodules for each transformed used.
+
+General setup:
+```sh
+export PIPELINE_FILE=apache_beam/yaml/examples/transforms/jinja/wordCount.yaml
+export KINGLEAR="gs://dataflow-samples/shakespeare/kinglear.txt"
+export TEMP_LOCATION="gs://MY-BUCKET/wordCounts/"
+
+cd <PATH_TO_BEAM_REPO>/beam/sdks/python
+```
+
+Multiline Run Example:
+```sh
+python -m apache_beam.yaml.main \
+ --yaml_pipeline_file=apache_beam/yaml/examples/transforms/jinja/include/wordCountInclude.yaml \
+ --jinja_variables='{
+ "readFromText": {"path": "'"${KINGLEAR}"'"},
+ "mapToFields_split": {
+ "language": "python",
+ "fields": {
+ "value": "1"
+ }
+ },
+ "explode": {"fields": "word"},
+ "combine": {
+ "group_by": "word",
+ "combine": {"value": "sum"}
+ },
+ "mapToFields_count": {
+ "language": "python",
+ "fields": {"output": "word + \" - \" + str(value)"}
+ },
+ "writeToText": {"path": "'"${TEMP_LOCATION}"'"}
+ }'
+```
+
+Single Line Run Example:
+```sh
+python -m apache_beam.yaml.main --yaml_pipeline_file=apache_beam/yaml/examples/transforms/jinja/include/wordCountInclude.yaml --jinja_variables='{"readFromText": {"path": "gs://dataflow-samples/shakespeare/kinglear.txt"}, "mapToFields_split": {"language": "python", "fields":{"value":"1"}}, "explode":{"fields":"word"}, "combine":{"group_by":"word", "combine":{"value":"sum"}}, "mapToFields_count":{"language": "python", "fields":{"output":"word + \" - \" + str(value)"}}, "writeToText":{"path":"${TEMP_LOCATION}"}}'
Review Comment:
it works, not needed
##########
sdks/python/apache_beam/yaml/examples/transforms/jinja/include/submodules/mapToFields_split.yaml:
##########
@@ -0,0 +1,40 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This examples reads from a public file stored on Google Cloud. This
+# requires authenticating with Google Cloud, or setting the file in
+#`ReadFromText` to a local file.
+#
+# To set up Application Default Credentials,
+# see https://cloud.google.com/docs/authentication/external/set-up-adc.
+#
+# This pipeline reads in a text file, counts distinct words found in the text,
+# then logs a row containing each word and its count.
+
+ language: {{mapToFields_split.language}}
+ fields:
+ word:
+ callable: |-
+ # TODO: Including another file here works fine, but if the file
+ # has a license header or other irrevalent comments, it will break
+ # the pipeline. Need to investigate more on Jinja filtering
+ # in the expand_jinja method or some other way.
Review Comment:
added todo link
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292584730
##########
sdks/python/apache_beam/yaml/examples/testing/input_data.py:
##########
@@ -33,6 +33,34 @@ def text_data():
])
+def word_count_jinja_parameter_data():
+ return \
+ '{"readFromText": {"path": ' \
+ '"gs://dataflow-samples/shakespeare/kinglear.txt"}, ' \
+ '"mapToFields_split": {"language": "python", "fields":{"value":"1",\
+ "word":{"callable":"apache_beam/yaml/examples/transforms/jinja/\
+ include/submodules/wordCount_mapper.txt"}}}, ' \
+ '"explode":{"fields":"word"}, ' \
+ '"combine":{"group_by":"word", "combine":{"value":"sum"}}, ' \
+ '"mapToFields_count":{"language": "python", "fields":' \
+ '{"output": "word + \\" - \\" + str(value)"}}, ' \
+ '"writeToText":{"path":"gs://apache-beam-testing-derrickaw/wordCounts/"}}'
Review Comment:
has to be this way and removed the mapper
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292581637
##########
sdks/python/apache_beam/yaml/examples/testing/input_data.py:
##########
@@ -33,6 +33,34 @@
])
+def word_count_jinja_parameter_data():
+ return \
+ '{"readFromText": {"path": ' \
+ '"gs://dataflow-samples/shakespeare/kinglear.txt"}, ' \
+ '"mapToFields_split": {"language": "python", "fields":{"value":"1",\
+ "word":{"callable":"apache_beam/yaml/examples/transforms/jinja/\
+ include/submodules/wordCount_mapper.txt"}}}, ' \
+ '"explode":{"fields":"word"}, ' \
+ '"combine":{"group_by":"word", "combine":{"value":"sum"}}, ' \
+ '"mapToFields_count":{"language": "python", "fields":' \
+ '{"output": "word + \\" - \\" + str(value)"}}, ' \
+ '"writeToText":{"path":"gs://apache-beam-testing-derrickaw/wordCounts/"}}'
+
+
+def word_count_jinja_template_data():
+ return \
+[('apache_beam/yaml/examples/transforms/jinja/'
+ 'include/submodules/readFromText.yaml'),
+ ('apache_beam/yaml/examples/transforms/jinja/'
+ 'include/submodules/mapToFields_split.yaml'),
+ 'apache_beam/yaml/examples/transforms/jinja/include/submodules/explode.yaml',
+ 'apache_beam/yaml/examples/transforms/jinja/include/submodules/combine.yaml',
+ ('apache_beam/yaml/examples/transforms/jinja/'
+ 'include/submodules/mapToFields_count.yaml'),
+ ('apache_beam/yaml/examples/transforms/jinja/'
+ 'include/submodules/writeToText.yaml')]
Review Comment:
it has to be this way due to less than 80 characters wide
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292059002
##########
sdks/python/apache_beam/yaml/examples/README.md:
##########
@@ -229,6 +233,27 @@ gcloud dataflow yaml run $JOB_NAME \
--region $REGION
```
+### Jinja
+
+Jinja [templatization](https://beam.apache.org/documentation/sdks/yaml/#jinja-templatization)
+can be used to build off of different contexts and/or with different
+configurations.
+
+Several examples will be created based on the already used word count example
+by leveraging Jinja templating engine for dynamic pipeline generation based on
+inputs from the user through `% include`, `% import`, and inheritance
+directives.
+
+% import:
Review Comment:
It was more of a heading and not usable code. Will try to rethink this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292082106
##########
sdks/python/apache_beam/yaml/examples/transforms/jinja/include/wordCountInclude.yaml:
##########
@@ -0,0 +1,71 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This examples reads from a public file stored on Google Cloud. This
+# requires authenticating with Google Cloud, or setting the file in
+#`ReadFromText` to a local file.
+#
+# To set up Application Default Credentials,
+# see https://cloud.google.com/docs/authentication/external/set-up-adc.
+#
+# This pipeline reads in a text file, counts distinct words found in the text,
+# then logs a row containing each word and its count.
+
+pipeline:
+ transforms:
+ - name: Read from GCS
+ type: ReadFromText
+ config:
+{% include 'apache_beam/yaml/examples/transforms/jinja/include/submodules/readFromText.yaml' %}
Review Comment:
> We generally don't want to put the whole transform here because it limits us from changing names or inputs.
I don't follow - could you give an example of someething we'd want to change but wouldn't be able to?
> How about a mix? Some transforms, some configs only?
This SGTM - I think the more complicated configs make sense, the one liners seem less useful
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292419646
##########
sdks/python/apache_beam/yaml/examples/testing/examples_test.py:
##########
@@ -538,8 +557,64 @@
word = element.split('=')[1].split(',')[0].replace("'", '')
count = int(element.split('=')[2].replace(')', ''))
all_words += [word] * count
- random.shuffle(all_words)
+ return _wordcount_random_shuffler(test_spec, all_words, env)
+
+
+@YamlExamplesTestSuite.register_test_preprocessor(
+ ['test_wordCountInclude_yaml'])
+def _wordcount_test_preprocessor(
Review Comment:
Leaving as is for future work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on PR #35914:
URL: https://github.com/apache/beam/pull/35914#issuecomment-3212872278
/gemini review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on PR #35914:
URL: https://github.com/apache/beam/pull/35914#issuecomment-3206407212
Run Python PreCommit 3.11
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292063929
##########
sdks/python/apache_beam/yaml/examples/transforms/jinja/include/wordCountInclude.yaml:
##########
@@ -0,0 +1,71 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This examples reads from a public file stored on Google Cloud. This
+# requires authenticating with Google Cloud, or setting the file in
+#`ReadFromText` to a local file.
+#
+# To set up Application Default Credentials,
+# see https://cloud.google.com/docs/authentication/external/set-up-adc.
+#
+# This pipeline reads in a text file, counts distinct words found in the text,
+# then logs a row containing each word and its count.
+
+pipeline:
+ transforms:
+ - name: Read from GCS
+ type: ReadFromText
+ config:
+{% include 'apache_beam/yaml/examples/transforms/jinja/include/submodules/readFromText.yaml' %}
Review Comment:
1. We generally don't want to put the whole transform here because it limits us from changing names or inputs. Wanted to give an example of how to use. How about a mix? Some transforms, some configs only?
2. Ok on better names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292094401
##########
sdks/python/apache_beam/yaml/examples/transforms/jinja/include/wordCountInclude.yaml:
##########
@@ -0,0 +1,71 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This examples reads from a public file stored on Google Cloud. This
+# requires authenticating with Google Cloud, or setting the file in
+#`ReadFromText` to a local file.
+#
+# To set up Application Default Credentials,
+# see https://cloud.google.com/docs/authentication/external/set-up-adc.
+#
+# This pipeline reads in a text file, counts distinct words found in the text,
+# then logs a row containing each word and its count.
+
+pipeline:
+ transforms:
+ - name: Read from GCS
+ type: ReadFromText
+ config:
+{% include 'apache_beam/yaml/examples/transforms/jinja/include/submodules/readFromText.yaml' %}
Review Comment:
> I don't follow - could you give an example of someething we'd want to change but wouldn't be able to?
>
> >
Maybe a mute point now with both options, but if we have the whole transform in the include yaml. We can't change the name or input paths etc unless we pass even more stuff through the jinjia variables. So the option I went with was to give an example where a user could control or orient the pipeline graph from the main yaml file and then include what configs they needed from a library. When we push everything to a library, we lose that control, but maybe that is fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on PR #35914:
URL: https://github.com/apache/beam/pull/35914#issuecomment-3211890296
Run Python PreCommit 3.12
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292099853
##########
sdks/python/apache_beam/yaml/examples/README.md:
##########
@@ -229,6 +233,27 @@ gcloud dataflow yaml run $JOB_NAME \
--region $REGION
```
+### Jinja
+
+Jinja [templatization](https://beam.apache.org/documentation/sdks/yaml/#jinja-templatization)
+can be used to build off of different contexts and/or with different
+configurations.
+
+Several examples will be created based on the already used word count example
+by leveraging Jinja templating engine for dynamic pipeline generation based on
+inputs from the user through `% include`, `% import`, and inheritance
+directives.
+
+% import:
+- [wordCount.yaml](#TODO: pending)
+
+% include:
+- [wordCount.yaml](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/yaml/examples/transforms/jinja/include/wordCount.yaml)
Review Comment:
Will do. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #35914:
URL: https://github.com/apache/beam/pull/35914#issuecomment-3212176767
I am trying to test how good Gemini knows our YAML. The review comments could be skipped. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292581263
##########
sdks/python/apache_beam/yaml/examples/transforms/jinja/include/README.md:
##########
@@ -0,0 +1,63 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+## Jinja % include Pipeline
+
+This example leverages the `% include` Jinja directive by having one main
+pipeline and then submodules for each transformed used.
+
+General setup:
+```sh
+export PIPELINE_FILE=apache_beam/yaml/examples/transforms/jinja/wordCount.yaml
Review Comment:
This is addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2292593047
##########
sdks/python/apache_beam/yaml/examples/transforms/jinja/include/submodules/combineTransform.yaml:
##########
@@ -0,0 +1,27 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# The Combine transform groups by each word and count the occurrences.
+
+ - name: Count words
+ type: Combine
+ config:
+ group_by:
+ - {{combineTransform.group_by}}
+ combine:
+ value: {{combineTransform.combine.value}}
Review Comment:
done, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2293757275
##########
sdks/python/apache_beam/yaml/examples/transforms/jinja/include/wordCountInclude.yaml:
##########
@@ -0,0 +1,71 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This examples reads from a public file stored on Google Cloud. This
+# requires authenticating with Google Cloud, or setting the file in
+#`ReadFromText` to a local file.
+#
+# To set up Application Default Credentials,
+# see https://cloud.google.com/docs/authentication/external/set-up-adc.
+#
+# This pipeline reads in a text file, counts distinct words found in the text,
+# then logs a row containing each word and its count.
+
+pipeline:
+ transforms:
+ - name: Read from GCS
+ type: ReadFromText
+ config:
+{% include 'apache_beam/yaml/examples/transforms/jinja/include/submodules/readFromText.yaml' %}
Review Comment:
Updated, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #35914:
URL: https://github.com/apache/beam/pull/35914#discussion_r2293758266
##########
sdks/python/apache_beam/yaml/examples/README.md:
##########
@@ -229,6 +233,27 @@ gcloud dataflow yaml run $JOB_NAME \
--region $REGION
```
+### Jinja
+
+Jinja [templatization](https://beam.apache.org/documentation/sdks/yaml/#jinja-templatization)
+can be used to build off of different contexts and/or with different
+configurations.
+
+Several examples will be created based on the already used word count example
+by leveraging Jinja templating engine for dynamic pipeline generation based on
+inputs from the user through `% include`, `% import`, and inheritance
+directives.
+
+% import:
Review Comment:
Updated, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja % include example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on PR #35914:
URL: https://github.com/apache/beam/pull/35914#issuecomment-3214428162
Run Python_Transforms PreCommit 3.10
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja example [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on PR #35914:
URL: https://github.com/apache/beam/pull/35914#issuecomment-3206406935
Run Python PreCommit 3.10
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] - Jinja example [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #35914:
URL: https://github.com/apache/beam/pull/35914#issuecomment-3207670002
Love the jinia idea for include. :)
We can add more examples more explicitly here later if you want: https://beam.apache.org/documentation/sdks/yaml/#jinja-templatization
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org