You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Nick Edwards <nw...@gmail.com> on 2023/03/03 18:26:08 UTC
Error "record is out of upper bound"
Hello! I’m investigating a warning we’re receiving through Google Cloud’s
Monitoring. The warning reads "record is out of upper bound 60000: " and
then includes a 5 or 6 digit integer. I’m having trouble finding any info
on this sort of behavior. Any insight or assistance you can offer is
greatly appreciated!
[image: image.png]
Re: Error "record is out of upper bound"
Posted by Svetak Sundhar via user <us...@beam.apache.org>.
Hi Nick-- Thanks for your email.
Could you file a ticket with GCP Support, providing a job ID and project ID
to look at? We have some tooling that can help us further investigate the
issue.
If not, we could maybe hop on a call and investigate the issue.
Thanks,
Svetak Sundhar
Technical Solutions Engineer, Data
s <ne...@google.com>vetaksundhar@google.com
On Fri, Mar 3, 2023 at 1:26 PM Nick Edwards <nw...@gmail.com>
wrote:
> Hello! I’m investigating a warning we’re receiving through Google Cloud’s
> Monitoring. The warning reads "record is out of upper bound 60000: " and
> then includes a 5 or 6 digit integer. I’m having trouble finding any info
> on this sort of behavior. Any insight or assistance you can offer is
> greatly appreciated!
>
>
> [image: image.png]
>
>
>
>
Re: Error "record is out of upper bound"
Posted by Svetak Sundhar via user <us...@beam.apache.org>.
For posterity here met with Nick over GVC, the issue is coming from the
file Bruno identified above, and is a warning for their custom histogram
logic.
Svetak Sundhar
Technical Solutions Engineer, Data
s <ne...@google.com>vetaksundhar@google.com
On Fri, Mar 3, 2023 at 1:40 PM Bruno Volpato via user <us...@beam.apache.org>
wrote:
> Found some more info, sorry for the chopping.
>
> Are you using *bigqueryio* or *bigquery_tools* somehow?
> If so, biguquery_tools defines a histogram using 20 buckets of 3 seconds
> each to export latencies (see
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery_tools.py#L347-L351
> ).
>
> Besides having requests with latency >60s not properly recorded (and
> warning logs), this shouldn't have any impact.
>
>
> On Fri, Mar 3, 2023 at 1:31 PM Bruno Volpato <bv...@google.com> wrote:
>
>> Hi Nick,
>>
>> This seems to come from utils/histogram.py
>> <https://github.com/apache/beam/blob/master/sdks/python/apache_beam/utils/histogram.py#L75-L89>.
>> Any chance that you are initializing it in a way that defines bounds up to
>> 60,000 but invoking record() with out of bounds value?
>>
>> Best,
>> Bruno
>>
>> On Fri, Mar 3, 2023 at 1:26 PM Nick Edwards <nw...@gmail.com>
>> wrote:
>>
>>> Hello! I’m investigating a warning we’re receiving through Google
>>> Cloud’s Monitoring. The warning reads "record is out of upper bound 60000:
>>> " and then includes a 5 or 6 digit integer. I’m having trouble finding any
>>> info on this sort of behavior. Any insight or assistance you can offer is
>>> greatly appreciated!
>>>
>>>
>>> [image: image.png]
>>>
>>>
>>>
>>>
>>
Re: Error "record is out of upper bound"
Posted by Bruno Volpato via user <us...@beam.apache.org>.
Found some more info, sorry for the chopping.
Are you using *bigqueryio* or *bigquery_tools* somehow?
If so, biguquery_tools defines a histogram using 20 buckets of 3 seconds
each to export latencies (see
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery_tools.py#L347-L351
).
Besides having requests with latency >60s not properly recorded (and
warning logs), this shouldn't have any impact.
On Fri, Mar 3, 2023 at 1:31 PM Bruno Volpato <bv...@google.com> wrote:
> Hi Nick,
>
> This seems to come from utils/histogram.py
> <https://github.com/apache/beam/blob/master/sdks/python/apache_beam/utils/histogram.py#L75-L89>.
> Any chance that you are initializing it in a way that defines bounds up to
> 60,000 but invoking record() with out of bounds value?
>
> Best,
> Bruno
>
> On Fri, Mar 3, 2023 at 1:26 PM Nick Edwards <nw...@gmail.com>
> wrote:
>
>> Hello! I’m investigating a warning we’re receiving through Google Cloud’s
>> Monitoring. The warning reads "record is out of upper bound 60000: " and
>> then includes a 5 or 6 digit integer. I’m having trouble finding any info
>> on this sort of behavior. Any insight or assistance you can offer is
>> greatly appreciated!
>>
>>
>> [image: image.png]
>>
>>
>>
>>
>
Re: Error "record is out of upper bound"
Posted by Bruno Volpato via user <us...@beam.apache.org>.
Hi Nick,
This seems to come from utils/histogram.py
<https://github.com/apache/beam/blob/master/sdks/python/apache_beam/utils/histogram.py#L75-L89>.
Any chance that you are initializing it in a way that defines bounds up to
60,000 but invoking record() with out of bounds value?
Best,
Bruno
On Fri, Mar 3, 2023 at 1:26 PM Nick Edwards <nw...@gmail.com>
wrote:
> Hello! I’m investigating a warning we’re receiving through Google Cloud’s
> Monitoring. The warning reads "record is out of upper bound 60000: " and
> then includes a 5 or 6 digit integer. I’m having trouble finding any info
> on this sort of behavior. Any insight or assistance you can offer is
> greatly appreciated!
>
>
> [image: image.png]
>
>
>
>