You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Matthew Brown <me...@matthewbrown.io> on 2022/04/05 07:42:05 UTC

BigQuery connector debugging

Hi all,

I'm attempting to build a Table API connector for BigQuery using the
BigQuery Storage API (
https://cloud.google.com/bigquery/docs/reference/storage).

I've got a base structure built out at
https://github.com/mnbbrown/bigquery-connector
There's a couple of things I have to do yet like correcting the mapping
between the BigQuery avro schema and flink TypeInformation, and add a test
suite.

I've added it to an internal project I'm working on and "nextRecord" on the
InputFormat is never called. I can see open/close,
openInputFormat/closeInputFormat, etc being called correctly.

Does anybody have any debugging tips?

Thanks,
Matt.

-- 
--
AU: +61 459 493 730
UK: +44 7927 618921
@mnbbrown

Re: BigQuery connector debugging

Posted by Martijn Visser <ma...@apache.org>.

Hey Matthew,

Thanks also for sharing the code that you're working on. What are your
plans with the connector? I could imagine that others would also be
interested, so perhaps you wanted to add it to https://flink-packages.org/
?

Best regards,

Martijn Visser
https://twitter.com/MartijnVisser82
https://github.com/MartijnVisser


On Wed, 6 Apr 2022 at 08:36, Matthew Brown <me...@matthewbrown.io> wrote:

> Thanks Robert!
>
> All working now.
>
> Turns out an incorrect log4j2 configuration was slurping up a warning
> about converting org.apache.avro.util.Utf8 to the internal flink data
> type.
> I've drawn some inspiration from flink-avro and added a converter that
> converts and re-orders the Avro encoded BigQuery results to the correct
> flink RowData.
>
> Cheers,
> Matt.
>
> On Tue, Apr 5, 2022 at 9:28 PM Robert Metzger <me...@gmail.com> wrote:
>
>> Hi Matt,
>>
>> At first glance your code looks fine. I guess you'll need to follow the
>> codepaths more with the debugger.
>> Have you made sure that "reachedEnd()" returns false?
>>
>>
>> On Tue, Apr 5, 2022 at 9:42 AM Matthew Brown <me...@matthewbrown.io> wrote:
>>
>>> Hi all,
>>>
>>> I'm attempting to build a Table API connector for BigQuery using the
>>> BigQuery Storage API (
>>> https://cloud.google.com/bigquery/docs/reference/storage).
>>>
>>> I've got a base structure built out at
>>> https://github.com/mnbbrown/bigquery-connector
>>> There's a couple of things I have to do yet like correcting the mapping
>>> between the BigQuery avro schema and flink TypeInformation, and add a test
>>> suite.
>>>
>>> I've added it to an internal project I'm working on and "nextRecord" on
>>> the InputFormat is never called. I can see open/close,
>>> openInputFormat/closeInputFormat, etc being called correctly.
>>>
>>> Does anybody have any debugging tips?
>>>
>>> Thanks,
>>> Matt.
>>>
>>> --
>>> --
>>> AU: +61 459 493 730
>>> UK: +44 7927 618921
>>> @mnbbrown
>>>
>>
>
> --
> --
> AU: +61 459 493 730
> UK: +44 7927 618921
> @mnbbrown
>

Re: BigQuery connector debugging

Posted by Matthew Brown <me...@matthewbrown.io>.

Thanks Robert!

All working now.

Turns out an incorrect log4j2 configuration was slurping up a warning about
converting org.apache.avro.util.Utf8 to the internal flink data type.
I've drawn some inspiration from flink-avro and added a converter that
converts and re-orders the Avro encoded BigQuery results to the correct
flink RowData.

Cheers,
Matt.

On Tue, Apr 5, 2022 at 9:28 PM Robert Metzger <me...@gmail.com> wrote:

> Hi Matt,
>
> At first glance your code looks fine. I guess you'll need to follow the
> codepaths more with the debugger.
> Have you made sure that "reachedEnd()" returns false?
>
>
> On Tue, Apr 5, 2022 at 9:42 AM Matthew Brown <me...@matthewbrown.io> wrote:
>
>> Hi all,
>>
>> I'm attempting to build a Table API connector for BigQuery using the
>> BigQuery Storage API (
>> https://cloud.google.com/bigquery/docs/reference/storage).
>>
>> I've got a base structure built out at
>> https://github.com/mnbbrown/bigquery-connector
>> There's a couple of things I have to do yet like correcting the mapping
>> between the BigQuery avro schema and flink TypeInformation, and add a test
>> suite.
>>
>> I've added it to an internal project I'm working on and "nextRecord" on
>> the InputFormat is never called. I can see open/close,
>> openInputFormat/closeInputFormat, etc being called correctly.
>>
>> Does anybody have any debugging tips?
>>
>> Thanks,
>> Matt.
>>
>> --
>> --
>> AU: +61 459 493 730
>> UK: +44 7927 618921
>> @mnbbrown
>>
>

-- 
--
AU: +61 459 493 730
UK: +44 7927 618921
@mnbbrown

Re: BigQuery connector debugging

Posted by Robert Metzger <me...@gmail.com>.

Hi Matt,

At first glance your code looks fine. I guess you'll need to follow the
codepaths more with the debugger.
Have you made sure that "reachedEnd()" returns false?


On Tue, Apr 5, 2022 at 9:42 AM Matthew Brown <me...@matthewbrown.io> wrote:

> Hi all,
>
> I'm attempting to build a Table API connector for BigQuery using the
> BigQuery Storage API (
> https://cloud.google.com/bigquery/docs/reference/storage).
>
> I've got a base structure built out at
> https://github.com/mnbbrown/bigquery-connector
> There's a couple of things I have to do yet like correcting the mapping
> between the BigQuery avro schema and flink TypeInformation, and add a test
> suite.
>
> I've added it to an internal project I'm working on and "nextRecord" on
> the InputFormat is never called. I can see open/close,
> openInputFormat/closeInputFormat, etc being called correctly.
>
> Does anybody have any debugging tips?
>
> Thanks,
> Matt.
>
> --
> --
> AU: +61 459 493 730
> UK: +44 7927 618921
> @mnbbrown
>