You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "ahmedabu98 (via GitHub)" <gi...@apache.org> on 2023/12/05 20:12:28 UTC
[I] [Bug] [BQ Streaming Inserts]: Whole numbers in float field is treated like an int [beam]
ahmedabu98 opened a new issue, #29622:
URL: https://github.com/apache/beam/issues/29622
### What happened?
When writing to BQ with streaming inserts, we do some serializing to JSON and int values can have a maximum value of < 2^64. Writing ints of higher value than this results in a `TypeError: Integer exceeds 64-bit range` (as opposed to floats, which can have much higher value).
When writing a whole number to a float field, the IO treats it as an int during serialization. This becomes a problem when trying to right a large whole number as a float because it results in the error mentioned above.
Reproduce with this code:
```
with beam.Pipeline() as p:
(p | beam.Create([
{
"str": "hello",
"ts": datetime.datetime(1970, 1, 1, tzinfo=pytz.utc),
"float": 18446744073709551616
}]) | beam.io.WriteToBigQuery(
table="google.com:clouddfe:ahmedabualsaud_test.actual_repro",
schema="str:STRING,ts:TIMESTAMP,float:FLOAT",
method="STREAMING_INSERTS"))
```
Instead of that float value, the following works:
`"float": 18446744073709551615` (below the 2^64 max in value)
`"float": 18446744073709551616.0` (explicitly setting as a float)
Stacktrace:
```
File "apache_beam/runners/common.py", line 1493, in apache_beam.runners.common.DoFnRunner._invoke_bundle_method
self._reraise_augmented(exn)
File "apache_beam/runners/common.py", line 574, in apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
def invoke_finish_bundle(self):
File "apache_beam/runners/common.py", line 580, in apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
self.signature.finish_bundle_method.method_value())
File "/Users/ahmedabualsaud/github/apache/beam/sdks/python/apache_beam/io/gcp/bigquery.py", line 1602, in finish_bundle
return self._flush_all_batches()
File "/Users/ahmedabualsaud/github/apache/beam/sdks/python/apache_beam/io/gcp/bigquery.py", line 1610, in _flush_all_batches
*[
File "/Users/ahmedabualsaud/github/apache/beam/sdks/python/apache_beam/io/gcp/bigquery.py", line 1611, in <listcomp>
self._flush_batch(destination)
File "/Users/ahmedabualsaud/github/apache/beam/sdks/python/apache_beam/io/gcp/bigquery.py", line 1639, in _flush_batch
passed, errors = self.bigquery_wrapper.insert_rows(
File "/Users/ahmedabualsaud/github/apache/beam/sdks/python/apache_beam/io/gcp/bigquery_tools.py", line 1264, in insert_rows
rows = [
File "/Users/ahmedabualsaud/github/apache/beam/sdks/python/apache_beam/io/gcp/bigquery_tools.py", line 1265, in <listcomp>
fast_json_loads(fast_json_dumps(r, default=default_encoder))
TypeError: Integer exceeds 64-bit range [while running 'WriteToBigQuery/_StreamToBigQuery/StreamInsertRows/ParDo(BigQueryWriteFn)']
```
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [X] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [BQ Streaming Inserts]: Whole number in float field is treated like an int [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #29622: [Bug] [BQ Streaming Inserts]: Whole number in float field is treated like an int
URL: https://github.com/apache/beam/issues/29622
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [BQ Streaming Inserts]: Whole number in float field is treated like an int [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #29622:
URL: https://github.com/apache/beam/issues/29622#issuecomment-3000724219
This can trigger this error as well:
```
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
import datetime
import pytz
def create_data():
return [
{
"str": "hello",
"ts": datetime.datetime(1970, 1, 1, tzinfo=pytz.utc)
},
{
"str": 999999999999999999999999,
"ts": datetime.datetime(1970, 1, 1, tzinfo=pytz.utc)
}
]
def run():
options = PipelineOptions(
project='cts10-kumarreddyg',
region='us-central1',
temp_location='gs://naveen123/temp',
runner='DataflowRunner',
service_account_email='id-20207358358-compute-dev-589@cts10-kumarreddyg.iam.gserviceaccount.com'
)
with beam.Pipeline(options=options) as p:
(
p
| 'CreateData' >> beam.Create(create_data())
| 'WriteToBigQuery' >> beam.io.WriteToBigQuery(
table='cts10-kumarreddyg.naveen.testerror',
schema='str:STRING,ts:TIMESTAMP',
method='STREAMING_INSERTS',
create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND
)
)
if __name__ == '__main__':
run()
#"str": str(999999999999999999999999)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [BQ Streaming Inserts]: Whole number in float field is treated like an int [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #29622:
URL: https://github.com/apache/beam/issues/29622#issuecomment-4397377949
This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [BQ Streaming Inserts]: Whole number in float field is treated like an int [beam]
Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.
johnjcasey commented on issue #29622:
URL: https://github.com/apache/beam/issues/29622#issuecomment-1843179707
We may want to simply not work on this, as Streaming Inserts should not be used in favor of the storage api
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [BQ Streaming Inserts]: Whole number in float field is treated like an int [beam]
Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.
johnjcasey commented on issue #29622:
URL: https://github.com/apache/beam/issues/29622#issuecomment-1843286585
In general, streaming inserts issues will not be prioritized, as the storage write api is preferred for streaming writes to BQ.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [BQ Streaming Inserts]: Whole number in float field is treated like an int [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #29622:
URL: https://github.com/apache/beam/issues/29622#issuecomment-4642750518
This issue has been closed due to lack of activity. If you think that is incorrect, you can reopen the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org