You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "EdwardCuiPeacock (via GitHub)" <gi...@apache.org> on 2023/07/12 14:25:47 UTC
[GitHub] [beam] EdwardCuiPeacock opened a new issue, #27469: [Bug]: 16177 has type numpy.int64, but expected one of: bytes
EdwardCuiPeacock opened a new issue, #27469:
URL: https://github.com/apache/beam/issues/27469
### What happened?
When running TFX's Evaluator component (which uses `tensorflow_model_analysis` to create beam jobs), the following error occurs in one of the workers, while all the other workers was able to complete successfully:
Traceback:
```
Error message from worker: Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 287, in _execute
response = task()
File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 360, in
lambda: self.create_worker().do_instruction(request), request)
File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 596, in do_instruction
return getattr(self, request_type)(
File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 635, in process_bundle
monitoring_infos = bundle_processor.monitoring_infos()
File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1139, in monitoring_infos
op.monitoring_infos(transform_id, dict(tag_to_pcollection_id)))
File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/operations.py", line 543, in monitoring_infos
all_monitoring_infos.update(self.user_monitoring_infos(transform_id))
File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/operations.py", line 584, in user_monitoring_infos
return self.metrics_container.to_runner_api_monitoring_infos(transform_id)
File "/usr/local/lib/python3.9/site-packages/apache_beam/metrics/execution.py", line 309, in to_runner_api_monitoring_infos
all_metrics = [
File "/usr/local/lib/python3.9/site-packages/apache_beam/metrics/execution.py", line 310, in
cell.to_runner_api_monitoring_info(key.metric_name, transform_id)
File "/usr/local/lib/python3.9/site-packages/apache_beam/metrics/cells.py", line 76, in to_runner_api_monitoring_info
mi = self.to_runner_api_monitoring_info_impl(name, transform_id)
File "/usr/local/lib/python3.9/site-packages/apache_beam/metrics/cells.py", line 150, in to_runner_api_monitoring_info_impl
return monitoring_infos.int64_user_counter(
File "/usr/local/lib/python3.9/site-packages/apache_beam/metrics/monitoring_infos.py", line 185, in int64_user_counter
return create_monitoring_info(
File "/usr/local/lib/python3.9/site-packages/apache_beam/metrics/monitoring_infos.py", line 302, in create_monitoring_info
return metrics_pb2.MonitoringInfo(
TypeError: 7006 has type numpy.int64, but expected one of: bytes
```
This error also happens only occasionally, but frequent enough to break production pipeline with a recurring schedule. I would like to understand the root cause of this error to prevent issues in production.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [X] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: 16177 has type numpy.int64, but expected one of: bytes [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #27469:
URL: https://github.com/apache/beam/issues/27469#issuecomment-5931861704
This issue has been closed due to lack of activity. If you think that is incorrect, you can reopen the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey commented on issue #27469: [Bug]: 16177 has type numpy.int64, but expected one of: bytes
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on issue #27469:
URL: https://github.com/apache/beam/issues/27469#issuecomment-1638285006
The root of the problem here is that numpy.int64 is not an instance of an int, so the numpy.int64 isn't getting encoded before being dropped into the MonitoringInfo proto. That logic check is here: https://github.com/apache/beam/blob/41e6628498c6ebdfc2932c61b68bcbde4da1a47e/sdks/python/apache_beam/metrics/monitoring_infos.py#L202
It looks like end-to-end we're expecting only int values to be aggregated within the Beam metric, but we don't necessarily encounter a hard check for that until we are trying to encode metric into a proto. It looks like the `inc()` and `dec()` methods get a type hint through their default params but `update()` doesn't enforce a type hint yet. Would you know if that is the method being used to work on the metric here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #27469: [Bug]: 16177 has type numpy.int64, but expected one of: bytes
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #27469:
URL: https://github.com/apache/beam/issues/27469#issuecomment-1638765151
Thanks for taking a look, @jrmccluskey .
It is possible that `update()` calls are happening withing Beam internals, but it sounds like somewhere in the application an incorrect type is passed as a counter value. It may be a bug in TFMA codebase. Perhaps adding logs to print offending counter name could help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: 16177 has type numpy.int64, but expected one of: bytes [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #27469:
URL: https://github.com/apache/beam/issues/27469#issuecomment-5478566267
This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #27469: [Bug]: 16177 has type numpy.int64, but expected one of: bytes
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #27469:
URL: https://github.com/apache/beam/issues/27469#issuecomment-1640317786
let's continue on https://github.com/tensorflow/model-analysis/issues/171
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: 16177 has type numpy.int64, but expected one of: bytes [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #27469: [Bug]: 16177 has type numpy.int64, but expected one of: bytes
URL: https://github.com/apache/beam/issues/27469
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org