You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "riteshghorse (via GitHub)" <gi...@apache.org> on 2024/03/28 02:36:50 UTC
[PR] [Docs] Update Bigtable enrichment notebook [beam]
riteshghorse opened a new pull request, #30783:
URL: https://github.com/apache/beam/pull/30783
Clearly explain and demonstrate the use of enrichment transform for `BigTableEnrichmentHandler`.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Update Bigtable enrichment notebook [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #30783:
URL: https://github.com/apache/beam/pull/30783#discussion_r1543366519
##########
examples/notebooks/beam-ml/bigtable_enrichment_transform.ipynb:
##########
@@ -509,15 +509,48 @@
"id": "K41xhvmA5yQk"
},
"source": [
- "To establish a client for the Bigtable enrichment handler, replace `<PROJECT_ID>`, `<INSTANCE_ID>`, and `<TABLE_ID>` with the appropriate values for those fields. The `row_key` variable is the field name from the input row that contains the row key to use when querying Bigtable.\n",
+ "Configure the `BigTableEnrichmentHandler` handler with the following required parameters:\n",
+ "\n",
+ "* `project_id`: the Google Cloud project ID for the Bigtable instance\n",
+ "* `instance_id`: the instance name of the Bigtable cluster\n",
+ "* `table_id`: the table ID of table containing relevant data\n",
+ "* `row_key`: The field name from the input row that contains the row key to use when querying Bigtable.\n",
"\n",
- "To convert a `string` type to a `byte` type or a `byte` type to a `string` type from Bigtable, you can configure additional options, such as [`app_profile_id`](https://cloud.google.com/bigtable/docs/app-profiles), [`row_filter`](https://cloud.google.com/python/docs/reference/bigtable/latest/row-filters), and [`encoding`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.bigtable.html#apache_beam.transforms.enrichment_handlers.bigtable.BigTableEnrichmentHandler:~:text=for%20more%20details.-,encoding,-(str)%20%E2%80%93%20encoding) type.\n",
+ "Optionally, to provide additional configuration to the `BigTableEnrichmentHandler` handler, see [module docs](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.bigtable.html#module-apache_beam.transforms.enrichment_handlers.bigtable)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "yFMcaf8i7TbI"
+ },
+ "source": [
+ "**Note:** When exceptions occur, by default, the logging severity is set to warning ([`ExceptionLevel.WARN`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.utils.html#apache_beam.transforms.enrichment_handlers.utils.ExceptionLevel.WARN)). To configure the severity to raise exceptions, set `exception_level` to [`ExceptionLevel.RAISE`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.utils.html#apache_beam.transforms.enrichment_handlers.utils.ExceptionLevel.RAISE). To ignore exceptions, set `exception_level` to [`ExceptionLevel.QUIET`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.utils.html#apache_beam.transforms.enrichment_handlers.utils.ExceptionLevel.QUIET).\n",
"\n",
- "The default `encoding` type is `utf-8`.\n",
+ "The following example demonstrates how to set exception level in `BigTableEnrichmentHandler`:\n",
"\n",
- "\n"
+ "```\n",
+ "bigtable_handler = BigTableEnrichmentHandler(project_id=PROJECT_ID,\n",
+ " instance_id=INSTANCE_ID,\n",
+ " table_id=TABLE_ID,\n",
+ " row_key=row_key,\n",
+ " exception_level=ExceptionLevel.RAISE)\n",
+ "```"
]
},
+ {
+ "cell_type": "markdown",
+ "source": [
+ "The key component of the Bigtable handler is the `row_key` parameter. The `row_key` represent the field in input schema (`beam.Row`) that contains the row key for a row in the table.\n",
+ "\n",
+ "As of Apache Beam version 2.54.0, if the table uses composite row keys, then you can:\n",
Review Comment:
I've phrased this slightly different because users only need to do one of these steps
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Update Bigtable enrichment notebook [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse merged PR #30783:
URL: https://github.com/apache/beam/pull/30783
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Update Bigtable enrichment notebook [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #30783:
URL: https://github.com/apache/beam/pull/30783#issuecomment-2025804076
Thanks again for the help in making this notebook look better!
merging!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Update Bigtable enrichment notebook [beam]
Posted by "rszper (via GitHub)" <gi...@apache.org>.
rszper commented on code in PR #30783:
URL: https://github.com/apache/beam/pull/30783#discussion_r1543389740
##########
examples/notebooks/beam-ml/bigtable_enrichment_transform.ipynb:
##########
@@ -509,15 +509,48 @@
"id": "K41xhvmA5yQk"
},
"source": [
- "To establish a client for the Bigtable enrichment handler, replace `<PROJECT_ID>`, `<INSTANCE_ID>`, and `<TABLE_ID>` with the appropriate values for those fields. The `row_key` variable is the field name from the input row that contains the row key to use when querying Bigtable.\n",
+ "Configure the `BigTableEnrichmentHandler` handler with the following required parameters:\n",
+ "\n",
+ "* `project_id`: the Google Cloud project ID for the Bigtable instance\n",
+ "* `instance_id`: the instance name of the Bigtable cluster\n",
+ "* `table_id`: the table ID of table containing relevant data\n",
+ "* `row_key`: The field name from the input row that contains the row key to use when querying Bigtable.\n",
"\n",
- "To convert a `string` type to a `byte` type or a `byte` type to a `string` type from Bigtable, you can configure additional options, such as [`app_profile_id`](https://cloud.google.com/bigtable/docs/app-profiles), [`row_filter`](https://cloud.google.com/python/docs/reference/bigtable/latest/row-filters), and [`encoding`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.bigtable.html#apache_beam.transforms.enrichment_handlers.bigtable.BigTableEnrichmentHandler:~:text=for%20more%20details.-,encoding,-(str)%20%E2%80%93%20encoding) type.\n",
+ "Optionally, you can use parameters to further configure the `BigTableEnrichmentHandler` handler. For more information about the available parameters, see [enrichment handler module documentation](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.bigtable.html#module-apache_beam.transforms.enrichment_handlers.bigtable)."
Review Comment:
```suggestion
"Optionally, you can use parameters to further configure the `BigTableEnrichmentHandler` handler. For more information about the available parameters, see the [enrichment handler module documentation](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.bigtable.html#module-apache_beam.transforms.enrichment_handlers.bigtable)."
```
##########
examples/notebooks/beam-ml/bigtable_enrichment_transform.ipynb:
##########
@@ -624,6 +642,44 @@
" return beam.Row(**enriched)"
]
},
+ {
+ "cell_type": "markdown",
+ "source": [
+ "To provide a `lambda` function for using a custom join with the enrichment transform, see the following example.\n",
+ "\n",
+ "```\n",
+ "with beam.Pipeline() as p:\n",
+ " output = (p\n",
+ " ...\n",
+ " | \"Enrich with BigTable\" >> Enrichment(bigtable_handler, join_fn=custom_join)\n",
+ " | \"RunInference\" >> RunInference(model_handler)\n",
+ " ...\n",
+ " )\n",
+ "```"
+ ],
+ "metadata": {
+ "id": "fe3bIclV1jZ5"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Because the enrichment transform make API calls to the remote service, use the `timeout` parameter to specify a timeout duration of 10 seconds:\n",
Review Comment:
```suggestion
"Because the enrichment transform makes API calls to the remote service, use the `timeout` parameter to specify a timeout duration of 10 seconds:\n",
```
##########
examples/notebooks/beam-ml/bigtable_enrichment_transform.ipynb:
##########
@@ -601,9 +617,11 @@
"id": "F-xjiP_pHWZr"
},
"source": [
- "To make a prediction, use the following fields: `product_id`, `quantity`, `price`, `customer_id`, and `customer_location`. Retrieve the value of the `customer_location` field from Bigtable.\n",
+ "The enrichment transform performs a [`cross_join`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment.html#apache_beam.transforms.enrichment.cross_join) by default. This will return the enriched row with the following fields: `sale_id`, `customer_id`, `product_id`, `quantity`, `price`, and `customer_location`.\n",
Review Comment:
```suggestion
"By default, the enrichment transform performs a [`cross_join`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment.html#apache_beam.transforms.enrichment.cross_join). This join returns the enriched row with the following fields: `sale_id`, `customer_id`, `product_id`, `quantity`, `price`, and `customer_location`.\n",
```
##########
examples/notebooks/beam-ml/bigtable_enrichment_transform.ipynb:
##########
@@ -601,9 +617,11 @@
"id": "F-xjiP_pHWZr"
},
"source": [
- "To make a prediction, use the following fields: `product_id`, `quantity`, `price`, `customer_id`, and `customer_location`. Retrieve the value of the `customer_location` field from Bigtable.\n",
+ "The enrichment transform performs a [`cross_join`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment.html#apache_beam.transforms.enrichment.cross_join) by default. This will return the enriched row with the following fields: `sale_id`, `customer_id`, `product_id`, `quantity`, `price`, and `customer_location`.\n",
+ "\n",
+ "But for the ecommerce use case, to make a prediction, the trained model needs the following fields: `product_id`, `quantity`, `price`, `customer_id`, and `customer_location`.\n",
Review Comment:
```suggestion
"To make a prediction when running the ecommerce example, however, the trained model needs the following fields: `product_id`, `quantity`, `price`, `customer_id`, and `customer_location`.\n",
```
##########
examples/notebooks/beam-ml/bigtable_enrichment_transform.ipynb:
##########
@@ -601,9 +617,11 @@
"id": "F-xjiP_pHWZr"
},
"source": [
- "To make a prediction, use the following fields: `product_id`, `quantity`, `price`, `customer_id`, and `customer_location`. Retrieve the value of the `customer_location` field from Bigtable.\n",
+ "The enrichment transform performs a [`cross_join`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment.html#apache_beam.transforms.enrichment.cross_join) by default. This will return the enriched row with the following fields: `sale_id`, `customer_id`, `product_id`, `quantity`, `price`, and `customer_location`.\n",
+ "\n",
+ "But for the ecommerce use case, to make a prediction, the trained model needs the following fields: `product_id`, `quantity`, `price`, `customer_id`, and `customer_location`.\n",
"\n",
- "Because the enrichment transform performs a [`cross_join`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment.html#apache_beam.transforms.enrichment.cross_join) by default, design the custom join to enrich the input data. This design ensures that the join includes only the specified fields."
+ "Design a custom join function that takes two dictionaries as input and returns an enriched row that include these fields."
Review Comment:
```suggestion
"Therefore, to get the required fields for the ecommerce example, design a custom join function that takes two dictionaries as input and returns an enriched row that include these fields."
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Update Bigtable enrichment notebook [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #30783:
URL: https://github.com/apache/beam/pull/30783#discussion_r1543368304
##########
examples/notebooks/beam-ml/bigtable_enrichment_transform.ipynb:
##########
@@ -601,9 +617,9 @@
"id": "F-xjiP_pHWZr"
},
"source": [
- "To make a prediction, use the following fields: `product_id`, `quantity`, `price`, `customer_id`, and `customer_location`. Retrieve the value of the `customer_location` field from Bigtable.\n",
+ "The enrichment transform performs a [`cross_join`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment.html#apache_beam.transforms.enrichment.cross_join) by default. To override this behavior, the transform accepts a `join_fn` lambda function. The lambda function takes two dictionaries as input and returns an enriched row.\n",
"\n",
- "Because the enrichment transform performs a [`cross_join`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment.html#apache_beam.transforms.enrichment.cross_join) by default, design the custom join to enrich the input data. This design ensures that the join includes only the specified fields."
+ "For our ecommerce use case, to make a prediction, it needs the following fields: `product_id`, `quantity`, `price`, `customer_id`, and `customer_location`. Design the custom join to enrich the input data such that the enriched row has these fields."
Review Comment:
I've changed the structure for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Update Bigtable enrichment notebook [beam]
Posted by "rszper (via GitHub)" <gi...@apache.org>.
rszper commented on code in PR #30783:
URL: https://github.com/apache/beam/pull/30783#discussion_r1543219708
##########
examples/notebooks/beam-ml/bigtable_enrichment_transform.ipynb:
##########
@@ -509,15 +509,48 @@
"id": "K41xhvmA5yQk"
},
"source": [
- "To establish a client for the Bigtable enrichment handler, replace `<PROJECT_ID>`, `<INSTANCE_ID>`, and `<TABLE_ID>` with the appropriate values for those fields. The `row_key` variable is the field name from the input row that contains the row key to use when querying Bigtable.\n",
+ "Configure the `BigTableEnrichmentHandler` handler with the following required parameters:\n",
+ "\n",
+ "* `project_id`: the Google Cloud project ID for the Bigtable instance\n",
+ "* `instance_id`: the instance name of the Bigtable cluster\n",
+ "* `table_id`: the table ID of table containing relevant data\n",
+ "* `row_key`: The field name from the input row that contains the row key to use when querying Bigtable.\n",
"\n",
- "To convert a `string` type to a `byte` type or a `byte` type to a `string` type from Bigtable, you can configure additional options, such as [`app_profile_id`](https://cloud.google.com/bigtable/docs/app-profiles), [`row_filter`](https://cloud.google.com/python/docs/reference/bigtable/latest/row-filters), and [`encoding`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.bigtable.html#apache_beam.transforms.enrichment_handlers.bigtable.BigTableEnrichmentHandler:~:text=for%20more%20details.-,encoding,-(str)%20%E2%80%93%20encoding) type.\n",
+ "Optionally, to provide additional configuration to the `BigTableEnrichmentHandler` handler, see [module docs](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.bigtable.html#module-apache_beam.transforms.enrichment_handlers.bigtable)."
Review Comment:
```suggestion
"Optionally, you can use parameters to further configure the `BigTableEnrichmentHandler` handler. For more information about the available parameters, see [enrichment handler module documentation](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.bigtable.html#module-apache_beam.transforms.enrichment_handlers.bigtable)."
```
##########
examples/notebooks/beam-ml/bigtable_enrichment_transform.ipynb:
##########
@@ -509,15 +509,48 @@
"id": "K41xhvmA5yQk"
},
"source": [
- "To establish a client for the Bigtable enrichment handler, replace `<PROJECT_ID>`, `<INSTANCE_ID>`, and `<TABLE_ID>` with the appropriate values for those fields. The `row_key` variable is the field name from the input row that contains the row key to use when querying Bigtable.\n",
+ "Configure the `BigTableEnrichmentHandler` handler with the following required parameters:\n",
+ "\n",
+ "* `project_id`: the Google Cloud project ID for the Bigtable instance\n",
+ "* `instance_id`: the instance name of the Bigtable cluster\n",
+ "* `table_id`: the table ID of table containing relevant data\n",
+ "* `row_key`: The field name from the input row that contains the row key to use when querying Bigtable.\n",
"\n",
- "To convert a `string` type to a `byte` type or a `byte` type to a `string` type from Bigtable, you can configure additional options, such as [`app_profile_id`](https://cloud.google.com/bigtable/docs/app-profiles), [`row_filter`](https://cloud.google.com/python/docs/reference/bigtable/latest/row-filters), and [`encoding`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.bigtable.html#apache_beam.transforms.enrichment_handlers.bigtable.BigTableEnrichmentHandler:~:text=for%20more%20details.-,encoding,-(str)%20%E2%80%93%20encoding) type.\n",
+ "Optionally, to provide additional configuration to the `BigTableEnrichmentHandler` handler, see [module docs](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.bigtable.html#module-apache_beam.transforms.enrichment_handlers.bigtable)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "yFMcaf8i7TbI"
+ },
+ "source": [
+ "**Note:** When exceptions occur, by default, the logging severity is set to warning ([`ExceptionLevel.WARN`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.utils.html#apache_beam.transforms.enrichment_handlers.utils.ExceptionLevel.WARN)). To configure the severity to raise exceptions, set `exception_level` to [`ExceptionLevel.RAISE`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.utils.html#apache_beam.transforms.enrichment_handlers.utils.ExceptionLevel.RAISE). To ignore exceptions, set `exception_level` to [`ExceptionLevel.QUIET`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.utils.html#apache_beam.transforms.enrichment_handlers.utils.ExceptionLevel.QUIET).\n",
"\n",
- "The default `encoding` type is `utf-8`.\n",
+ "The following example demonstrates how to set exception level in `BigTableEnrichmentHandler`:\n",
Review Comment:
```suggestion
"The following example demonstrates how to set the exception level in the `BigTableEnrichmentHandler` handler:\n",
```
##########
examples/notebooks/beam-ml/bigtable_enrichment_transform.ipynb:
##########
@@ -509,15 +509,48 @@
"id": "K41xhvmA5yQk"
},
"source": [
- "To establish a client for the Bigtable enrichment handler, replace `<PROJECT_ID>`, `<INSTANCE_ID>`, and `<TABLE_ID>` with the appropriate values for those fields. The `row_key` variable is the field name from the input row that contains the row key to use when querying Bigtable.\n",
+ "Configure the `BigTableEnrichmentHandler` handler with the following required parameters:\n",
+ "\n",
+ "* `project_id`: the Google Cloud project ID for the Bigtable instance\n",
+ "* `instance_id`: the instance name of the Bigtable cluster\n",
+ "* `table_id`: the table ID of table containing relevant data\n",
+ "* `row_key`: The field name from the input row that contains the row key to use when querying Bigtable.\n",
"\n",
- "To convert a `string` type to a `byte` type or a `byte` type to a `string` type from Bigtable, you can configure additional options, such as [`app_profile_id`](https://cloud.google.com/bigtable/docs/app-profiles), [`row_filter`](https://cloud.google.com/python/docs/reference/bigtable/latest/row-filters), and [`encoding`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.bigtable.html#apache_beam.transforms.enrichment_handlers.bigtable.BigTableEnrichmentHandler:~:text=for%20more%20details.-,encoding,-(str)%20%E2%80%93%20encoding) type.\n",
+ "Optionally, to provide additional configuration to the `BigTableEnrichmentHandler` handler, see [module docs](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.bigtable.html#module-apache_beam.transforms.enrichment_handlers.bigtable)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "yFMcaf8i7TbI"
+ },
+ "source": [
+ "**Note:** When exceptions occur, by default, the logging severity is set to warning ([`ExceptionLevel.WARN`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.utils.html#apache_beam.transforms.enrichment_handlers.utils.ExceptionLevel.WARN)). To configure the severity to raise exceptions, set `exception_level` to [`ExceptionLevel.RAISE`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.utils.html#apache_beam.transforms.enrichment_handlers.utils.ExceptionLevel.RAISE). To ignore exceptions, set `exception_level` to [`ExceptionLevel.QUIET`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.utils.html#apache_beam.transforms.enrichment_handlers.utils.ExceptionLevel.QUIET).\n",
"\n",
- "The default `encoding` type is `utf-8`.\n",
+ "The following example demonstrates how to set exception level in `BigTableEnrichmentHandler`:\n",
"\n",
- "\n"
+ "```\n",
+ "bigtable_handler = BigTableEnrichmentHandler(project_id=PROJECT_ID,\n",
+ " instance_id=INSTANCE_ID,\n",
+ " table_id=TABLE_ID,\n",
+ " row_key=row_key,\n",
+ " exception_level=ExceptionLevel.RAISE)\n",
+ "```"
]
},
+ {
+ "cell_type": "markdown",
+ "source": [
+ "The key component of the Bigtable handler is the `row_key` parameter. The `row_key` represent the field in input schema (`beam.Row`) that contains the row key for a row in the table.\n",
Review Comment:
```suggestion
"The `row_key` parameter represents the field in input schema (`beam.Row`) that contains the row key for a row in the table.\n",
```
##########
examples/notebooks/beam-ml/bigtable_enrichment_transform.ipynb:
##########
@@ -624,6 +640,44 @@
" return beam.Row(**enriched)"
]
},
+ {
+ "cell_type": "markdown",
+ "source": [
+ "The following example demonstrates the code needed to provide a lambda function for a custom join to the enrichment transform:\n",
+ "\n",
+ "```\n",
+ "with beam.Pipeline() as p:\n",
+ " output = (p\n",
+ " ...\n",
+ " | \"Enrich with BigTable\" >> Enrichment(bigtable_handler, join_fn=custom_join)\n",
+ " | \"RunInference\" >> RunInference(model_handler)\n",
+ " ...\n",
+ " )\n",
+ "```"
+ ],
+ "metadata": {
+ "id": "fe3bIclV1jZ5"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Since the enrichment transform make API calls to the remote service, specify a timeout duration of 10 seconds using the timeout param:\n",
Review Comment:
```suggestion
"Because the enrichment transform make API calls to the remote service, use the `timeout` parameter to specify a timeout duration of 10 seconds:\n",
```
##########
examples/notebooks/beam-ml/bigtable_enrichment_transform.ipynb:
##########
@@ -509,15 +509,48 @@
"id": "K41xhvmA5yQk"
},
"source": [
- "To establish a client for the Bigtable enrichment handler, replace `<PROJECT_ID>`, `<INSTANCE_ID>`, and `<TABLE_ID>` with the appropriate values for those fields. The `row_key` variable is the field name from the input row that contains the row key to use when querying Bigtable.\n",
+ "Configure the `BigTableEnrichmentHandler` handler with the following required parameters:\n",
+ "\n",
+ "* `project_id`: the Google Cloud project ID for the Bigtable instance\n",
+ "* `instance_id`: the instance name of the Bigtable cluster\n",
+ "* `table_id`: the table ID of table containing relevant data\n",
+ "* `row_key`: The field name from the input row that contains the row key to use when querying Bigtable.\n",
"\n",
- "To convert a `string` type to a `byte` type or a `byte` type to a `string` type from Bigtable, you can configure additional options, such as [`app_profile_id`](https://cloud.google.com/bigtable/docs/app-profiles), [`row_filter`](https://cloud.google.com/python/docs/reference/bigtable/latest/row-filters), and [`encoding`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.bigtable.html#apache_beam.transforms.enrichment_handlers.bigtable.BigTableEnrichmentHandler:~:text=for%20more%20details.-,encoding,-(str)%20%E2%80%93%20encoding) type.\n",
+ "Optionally, to provide additional configuration to the `BigTableEnrichmentHandler` handler, see [module docs](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.bigtable.html#module-apache_beam.transforms.enrichment_handlers.bigtable)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "yFMcaf8i7TbI"
+ },
+ "source": [
+ "**Note:** When exceptions occur, by default, the logging severity is set to warning ([`ExceptionLevel.WARN`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.utils.html#apache_beam.transforms.enrichment_handlers.utils.ExceptionLevel.WARN)). To configure the severity to raise exceptions, set `exception_level` to [`ExceptionLevel.RAISE`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.utils.html#apache_beam.transforms.enrichment_handlers.utils.ExceptionLevel.RAISE). To ignore exceptions, set `exception_level` to [`ExceptionLevel.QUIET`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.utils.html#apache_beam.transforms.enrichment_handlers.utils.ExceptionLevel.QUIET).\n",
"\n",
- "The default `encoding` type is `utf-8`.\n",
+ "The following example demonstrates how to set exception level in `BigTableEnrichmentHandler`:\n",
"\n",
- "\n"
+ "```\n",
+ "bigtable_handler = BigTableEnrichmentHandler(project_id=PROJECT_ID,\n",
+ " instance_id=INSTANCE_ID,\n",
+ " table_id=TABLE_ID,\n",
+ " row_key=row_key,\n",
+ " exception_level=ExceptionLevel.RAISE)\n",
+ "```"
]
},
+ {
+ "cell_type": "markdown",
+ "source": [
+ "The key component of the Bigtable handler is the `row_key` parameter. The `row_key` represent the field in input schema (`beam.Row`) that contains the row key for a row in the table.\n",
+ "\n",
+ "As of Apache Beam version 2.54.0, if the table uses composite row keys, then you can:\n",
+ "* modify the input schema to contain the row key in the format required by Bigtable.\n",
+ "* use a custom enrichment handler ([example handler with composite row key support](https://gist.github.com/riteshghorse/21f4480c1c545ea01166e6bdf4c183e1))."
Review Comment:
```suggestion
"* Use a custom enrichment handler. For more information, see the [example handler with composite row key support](https://gist.github.com/riteshghorse/21f4480c1c545ea01166e6bdf4c183e1)."
```
##########
examples/notebooks/beam-ml/bigtable_enrichment_transform.ipynb:
##########
@@ -509,15 +509,48 @@
"id": "K41xhvmA5yQk"
},
"source": [
- "To establish a client for the Bigtable enrichment handler, replace `<PROJECT_ID>`, `<INSTANCE_ID>`, and `<TABLE_ID>` with the appropriate values for those fields. The `row_key` variable is the field name from the input row that contains the row key to use when querying Bigtable.\n",
+ "Configure the `BigTableEnrichmentHandler` handler with the following required parameters:\n",
+ "\n",
+ "* `project_id`: the Google Cloud project ID for the Bigtable instance\n",
+ "* `instance_id`: the instance name of the Bigtable cluster\n",
+ "* `table_id`: the table ID of table containing relevant data\n",
+ "* `row_key`: The field name from the input row that contains the row key to use when querying Bigtable.\n",
"\n",
- "To convert a `string` type to a `byte` type or a `byte` type to a `string` type from Bigtable, you can configure additional options, such as [`app_profile_id`](https://cloud.google.com/bigtable/docs/app-profiles), [`row_filter`](https://cloud.google.com/python/docs/reference/bigtable/latest/row-filters), and [`encoding`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.bigtable.html#apache_beam.transforms.enrichment_handlers.bigtable.BigTableEnrichmentHandler:~:text=for%20more%20details.-,encoding,-(str)%20%E2%80%93%20encoding) type.\n",
+ "Optionally, to provide additional configuration to the `BigTableEnrichmentHandler` handler, see [module docs](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.bigtable.html#module-apache_beam.transforms.enrichment_handlers.bigtable)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "yFMcaf8i7TbI"
+ },
+ "source": [
+ "**Note:** When exceptions occur, by default, the logging severity is set to warning ([`ExceptionLevel.WARN`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.utils.html#apache_beam.transforms.enrichment_handlers.utils.ExceptionLevel.WARN)). To configure the severity to raise exceptions, set `exception_level` to [`ExceptionLevel.RAISE`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.utils.html#apache_beam.transforms.enrichment_handlers.utils.ExceptionLevel.RAISE). To ignore exceptions, set `exception_level` to [`ExceptionLevel.QUIET`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.utils.html#apache_beam.transforms.enrichment_handlers.utils.ExceptionLevel.QUIET).\n",
"\n",
- "The default `encoding` type is `utf-8`.\n",
+ "The following example demonstrates how to set exception level in `BigTableEnrichmentHandler`:\n",
"\n",
- "\n"
+ "```\n",
+ "bigtable_handler = BigTableEnrichmentHandler(project_id=PROJECT_ID,\n",
+ " instance_id=INSTANCE_ID,\n",
+ " table_id=TABLE_ID,\n",
+ " row_key=row_key,\n",
+ " exception_level=ExceptionLevel.RAISE)\n",
+ "```"
]
},
+ {
+ "cell_type": "markdown",
+ "source": [
+ "The key component of the Bigtable handler is the `row_key` parameter. The `row_key` represent the field in input schema (`beam.Row`) that contains the row key for a row in the table.\n",
+ "\n",
+ "As of Apache Beam version 2.54.0, if the table uses composite row keys, then you can:\n",
Review Comment:
```suggestion
"Starting in Apache Beam version 2.54.0, if the table uses composite row keys, then you can do the following tasks:\n",
```
##########
examples/notebooks/beam-ml/bigtable_enrichment_transform.ipynb:
##########
@@ -562,13 +584,7 @@
"source": [
"## Use the enrichment transform\n",
"\n",
- "To use the [enrichment transform](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment.html#apache_beam.transforms.enrichment.Enrichment), the [`EnrichmentHandler`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment.html#apache_beam.transforms.enrichment.EnrichmentSourceHandler) parameter is required. You can also use a configuration parameter to specify a `lambda` for a join function, a timeout, a throttler, and a repeater (retry strategy).\n",
- "\n",
- "\n",
- "* `join_fn`: A lambda function that takes dictionaries as input and returns an enriched row (`Callable[[Dict[str, Any], Dict[str, Any]], beam.Row]`). The enriched row specifies how to join the data fetched from the API. Defaults to a [cross-join](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment.html#apache_beam.transforms.enrichment.cross_join).\n",
- "* `timeout`: The number of seconds to wait for the request to be completed by the API before timing out. Defaults to 30 seconds.\n",
- "* `throttler`: Specifies the throttling mechanism. The only supported option is default client-side adaptive throttling.\n",
- "* `repeater`: Specifies the retry strategy when errors like `TooManyRequests` and `TimeoutException` occur. Defaults to [`ExponentialBackOffRepeater`](https://beam.apache.org/releases/pydoc/current/apache_beam.io.requestresponse.html#apache_beam.io.requestresponse.ExponentialBackOffRepeater).\n"
+ "To use the [enrichment transform](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment.html#apache_beam.transforms.enrichment.Enrichment), only the enrichment handler parameter is required."
Review Comment:
```suggestion
"To use the [enrichment transform](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment.html#apache_beam.transforms.enrichment.Enrichment), the enrichment handler parameter is the only required parameter."
```
##########
examples/notebooks/beam-ml/bigtable_enrichment_transform.ipynb:
##########
@@ -509,15 +509,48 @@
"id": "K41xhvmA5yQk"
},
"source": [
- "To establish a client for the Bigtable enrichment handler, replace `<PROJECT_ID>`, `<INSTANCE_ID>`, and `<TABLE_ID>` with the appropriate values for those fields. The `row_key` variable is the field name from the input row that contains the row key to use when querying Bigtable.\n",
+ "Configure the `BigTableEnrichmentHandler` handler with the following required parameters:\n",
+ "\n",
+ "* `project_id`: the Google Cloud project ID for the Bigtable instance\n",
+ "* `instance_id`: the instance name of the Bigtable cluster\n",
+ "* `table_id`: the table ID of table containing relevant data\n",
+ "* `row_key`: The field name from the input row that contains the row key to use when querying Bigtable.\n",
"\n",
- "To convert a `string` type to a `byte` type or a `byte` type to a `string` type from Bigtable, you can configure additional options, such as [`app_profile_id`](https://cloud.google.com/bigtable/docs/app-profiles), [`row_filter`](https://cloud.google.com/python/docs/reference/bigtable/latest/row-filters), and [`encoding`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.bigtable.html#apache_beam.transforms.enrichment_handlers.bigtable.BigTableEnrichmentHandler:~:text=for%20more%20details.-,encoding,-(str)%20%E2%80%93%20encoding) type.\n",
+ "Optionally, to provide additional configuration to the `BigTableEnrichmentHandler` handler, see [module docs](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.bigtable.html#module-apache_beam.transforms.enrichment_handlers.bigtable)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "yFMcaf8i7TbI"
+ },
+ "source": [
+ "**Note:** When exceptions occur, by default, the logging severity is set to warning ([`ExceptionLevel.WARN`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.utils.html#apache_beam.transforms.enrichment_handlers.utils.ExceptionLevel.WARN)). To configure the severity to raise exceptions, set `exception_level` to [`ExceptionLevel.RAISE`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.utils.html#apache_beam.transforms.enrichment_handlers.utils.ExceptionLevel.RAISE). To ignore exceptions, set `exception_level` to [`ExceptionLevel.QUIET`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.utils.html#apache_beam.transforms.enrichment_handlers.utils.ExceptionLevel.QUIET).\n",
"\n",
- "The default `encoding` type is `utf-8`.\n",
+ "The following example demonstrates how to set exception level in `BigTableEnrichmentHandler`:\n",
"\n",
- "\n"
+ "```\n",
+ "bigtable_handler = BigTableEnrichmentHandler(project_id=PROJECT_ID,\n",
+ " instance_id=INSTANCE_ID,\n",
+ " table_id=TABLE_ID,\n",
+ " row_key=row_key,\n",
+ " exception_level=ExceptionLevel.RAISE)\n",
+ "```"
]
},
+ {
+ "cell_type": "markdown",
+ "source": [
+ "The key component of the Bigtable handler is the `row_key` parameter. The `row_key` represent the field in input schema (`beam.Row`) that contains the row key for a row in the table.\n",
+ "\n",
+ "As of Apache Beam version 2.54.0, if the table uses composite row keys, then you can:\n",
+ "* modify the input schema to contain the row key in the format required by Bigtable.\n",
Review Comment:
```suggestion
"* Modify the input schema to contain the row key in the format required by Bigtable.\n",
```
##########
examples/notebooks/beam-ml/bigtable_enrichment_transform.ipynb:
##########
@@ -624,6 +640,44 @@
" return beam.Row(**enriched)"
]
},
+ {
+ "cell_type": "markdown",
+ "source": [
+ "The following example demonstrates the code needed to provide a lambda function for a custom join to the enrichment transform:\n",
Review Comment:
```suggestion
"To provide a `lambda` function for using a custom join with the enrichment transform, see the following example.\n",
```
##########
examples/notebooks/beam-ml/bigtable_enrichment_transform.ipynb:
##########
@@ -509,15 +509,48 @@
"id": "K41xhvmA5yQk"
},
"source": [
- "To establish a client for the Bigtable enrichment handler, replace `<PROJECT_ID>`, `<INSTANCE_ID>`, and `<TABLE_ID>` with the appropriate values for those fields. The `row_key` variable is the field name from the input row that contains the row key to use when querying Bigtable.\n",
+ "Configure the `BigTableEnrichmentHandler` handler with the following required parameters:\n",
+ "\n",
+ "* `project_id`: the Google Cloud project ID for the Bigtable instance\n",
+ "* `instance_id`: the instance name of the Bigtable cluster\n",
+ "* `table_id`: the table ID of table containing relevant data\n",
+ "* `row_key`: The field name from the input row that contains the row key to use when querying Bigtable.\n",
"\n",
- "To convert a `string` type to a `byte` type or a `byte` type to a `string` type from Bigtable, you can configure additional options, such as [`app_profile_id`](https://cloud.google.com/bigtable/docs/app-profiles), [`row_filter`](https://cloud.google.com/python/docs/reference/bigtable/latest/row-filters), and [`encoding`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.bigtable.html#apache_beam.transforms.enrichment_handlers.bigtable.BigTableEnrichmentHandler:~:text=for%20more%20details.-,encoding,-(str)%20%E2%80%93%20encoding) type.\n",
+ "Optionally, to provide additional configuration to the `BigTableEnrichmentHandler` handler, see [module docs](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.bigtable.html#module-apache_beam.transforms.enrichment_handlers.bigtable)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "yFMcaf8i7TbI"
+ },
+ "source": [
+ "**Note:** When exceptions occur, by default, the logging severity is set to warning ([`ExceptionLevel.WARN`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.utils.html#apache_beam.transforms.enrichment_handlers.utils.ExceptionLevel.WARN)). To configure the severity to raise exceptions, set `exception_level` to [`ExceptionLevel.RAISE`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.utils.html#apache_beam.transforms.enrichment_handlers.utils.ExceptionLevel.RAISE). To ignore exceptions, set `exception_level` to [`ExceptionLevel.QUIET`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.utils.html#apache_beam.transforms.enrichment_handlers.utils.ExceptionLevel.QUIET).\n",
"\n",
- "The default `encoding` type is `utf-8`.\n",
+ "The following example demonstrates how to set exception level in `BigTableEnrichmentHandler`:\n",
"\n",
- "\n"
+ "```\n",
+ "bigtable_handler = BigTableEnrichmentHandler(project_id=PROJECT_ID,\n",
+ " instance_id=INSTANCE_ID,\n",
+ " table_id=TABLE_ID,\n",
+ " row_key=row_key,\n",
+ " exception_level=ExceptionLevel.RAISE)\n",
+ "```"
]
},
+ {
+ "cell_type": "markdown",
+ "source": [
+ "The key component of the Bigtable handler is the `row_key` parameter. The `row_key` represent the field in input schema (`beam.Row`) that contains the row key for a row in the table.\n",
+ "\n",
+ "As of Apache Beam version 2.54.0, if the table uses composite row keys, then you can:\n",
+ "* modify the input schema to contain the row key in the format required by Bigtable.\n",
+ "* use a custom enrichment handler ([example handler with composite row key support](https://gist.github.com/riteshghorse/21f4480c1c545ea01166e6bdf4c183e1))."
Review Comment:
Should this link be pointing to your repo?
##########
examples/notebooks/beam-ml/bigtable_enrichment_transform.ipynb:
##########
@@ -601,9 +617,9 @@
"id": "F-xjiP_pHWZr"
},
"source": [
- "To make a prediction, use the following fields: `product_id`, `quantity`, `price`, `customer_id`, and `customer_location`. Retrieve the value of the `customer_location` field from Bigtable.\n",
+ "The enrichment transform performs a [`cross_join`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment.html#apache_beam.transforms.enrichment.cross_join) by default. To override this behavior, the transform accepts a `join_fn` lambda function. The lambda function takes two dictionaries as input and returns an enriched row.\n",
"\n",
- "Because the enrichment transform performs a [`cross_join`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment.html#apache_beam.transforms.enrichment.cross_join) by default, design the custom join to enrich the input data. This design ensures that the join includes only the specified fields."
+ "For our ecommerce use case, to make a prediction, it needs the following fields: `product_id`, `quantity`, `price`, `customer_id`, and `customer_location`. Design the custom join to enrich the input data such that the enriched row has these fields."
Review Comment:
```suggestion
"For the ecommerce use case, to make a prediction, provide the following fields: `product_id`, `quantity`, `price`, `customer_id`, and `customer_location`. When you design the custom join to enrich the input data, the enriched row must include these fields."
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Update Bigtable enrichment notebook [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #30783:
URL: https://github.com/apache/beam/pull/30783#discussion_r1543287362
##########
examples/notebooks/beam-ml/bigtable_enrichment_transform.ipynb:
##########
@@ -509,15 +509,48 @@
"id": "K41xhvmA5yQk"
},
"source": [
- "To establish a client for the Bigtable enrichment handler, replace `<PROJECT_ID>`, `<INSTANCE_ID>`, and `<TABLE_ID>` with the appropriate values for those fields. The `row_key` variable is the field name from the input row that contains the row key to use when querying Bigtable.\n",
+ "Configure the `BigTableEnrichmentHandler` handler with the following required parameters:\n",
+ "\n",
+ "* `project_id`: the Google Cloud project ID for the Bigtable instance\n",
+ "* `instance_id`: the instance name of the Bigtable cluster\n",
+ "* `table_id`: the table ID of table containing relevant data\n",
+ "* `row_key`: The field name from the input row that contains the row key to use when querying Bigtable.\n",
"\n",
- "To convert a `string` type to a `byte` type or a `byte` type to a `string` type from Bigtable, you can configure additional options, such as [`app_profile_id`](https://cloud.google.com/bigtable/docs/app-profiles), [`row_filter`](https://cloud.google.com/python/docs/reference/bigtable/latest/row-filters), and [`encoding`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.bigtable.html#apache_beam.transforms.enrichment_handlers.bigtable.BigTableEnrichmentHandler:~:text=for%20more%20details.-,encoding,-(str)%20%E2%80%93%20encoding) type.\n",
+ "Optionally, to provide additional configuration to the `BigTableEnrichmentHandler` handler, see [module docs](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.bigtable.html#module-apache_beam.transforms.enrichment_handlers.bigtable)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "yFMcaf8i7TbI"
+ },
+ "source": [
+ "**Note:** When exceptions occur, by default, the logging severity is set to warning ([`ExceptionLevel.WARN`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.utils.html#apache_beam.transforms.enrichment_handlers.utils.ExceptionLevel.WARN)). To configure the severity to raise exceptions, set `exception_level` to [`ExceptionLevel.RAISE`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.utils.html#apache_beam.transforms.enrichment_handlers.utils.ExceptionLevel.RAISE). To ignore exceptions, set `exception_level` to [`ExceptionLevel.QUIET`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.utils.html#apache_beam.transforms.enrichment_handlers.utils.ExceptionLevel.QUIET).\n",
"\n",
- "The default `encoding` type is `utf-8`.\n",
+ "The following example demonstrates how to set exception level in `BigTableEnrichmentHandler`:\n",
"\n",
- "\n"
+ "```\n",
+ "bigtable_handler = BigTableEnrichmentHandler(project_id=PROJECT_ID,\n",
+ " instance_id=INSTANCE_ID,\n",
+ " table_id=TABLE_ID,\n",
+ " row_key=row_key,\n",
+ " exception_level=ExceptionLevel.RAISE)\n",
+ "```"
]
},
+ {
+ "cell_type": "markdown",
+ "source": [
+ "The key component of the Bigtable handler is the `row_key` parameter. The `row_key` represent the field in input schema (`beam.Row`) that contains the row key for a row in the table.\n",
+ "\n",
+ "As of Apache Beam version 2.54.0, if the table uses composite row keys, then you can:\n",
+ "* modify the input schema to contain the row key in the format required by Bigtable.\n",
+ "* use a custom enrichment handler ([example handler with composite row key support](https://gist.github.com/riteshghorse/21f4480c1c545ea01166e6bdf4c183e1))."
Review Comment:
Actually I can replace this with a online pastebin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Update Bigtable enrichment notebook [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30783:
URL: https://github.com/apache/beam/pull/30783#issuecomment-2025440365
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Update Bigtable enrichment notebook [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #30783:
URL: https://github.com/apache/beam/pull/30783#issuecomment-2025435512
R: @rszper
CC: @rezarokni
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Update Bigtable enrichment notebook [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #30783:
URL: https://github.com/apache/beam/pull/30783#issuecomment-2025759265
Thanks for the review! I've applied all comments except the two places where I've restructured a bit. PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org