You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Anton Kedin <ke...@google.com> on 2018/08/13 23:06:15 UTC
[SQL] Create External Schema
Hi,
I am planning to work on implementing a support for external schema
providers for Beam SQL and wanted to share a high level idea how I think
this can work.
*Short Version*
Implement CREATE FOREIGN SCHEMA statement:
CREATE FOREIGN SCHEMA
TYPE 'bigquery'
LOCATION 'dataset_example'
AS bq;
CREATE FOREIGN SCHEMA
TYPE 'hcatalog'
LOCATION 'hive-server:2341'
AS hive;
SELECT *
FROM
bq.table_example_bq AS bq_table1
JOIN
hive.table_example_hive AS hive_table1
ON
bq_table1.some_field = hive_table1.some_other_field;
*A Bit Longer Version: *
https://docs.google.com/document/d/1Ilk3OpDxrp3bHNlcnYDoj29tt9bd1E0EXt8i0WytNmQ
Thoughts, ideas?
Regards,
Anton
Re: [SQL] Create External Schema
Posted by Anton Kedin <ke...@google.com>.
It should be. I briefly looked at it and it seems possible to use
SchemaRegistry for this, although it will need some additional wiring for
named schemas.
Regards,
Anton
On Mon, Aug 13, 2018 at 4:14 PM Reuven Lax <re...@google.com> wrote:
> Is it possible to extend Beam's SchemaRegistry to do this?
>
> On Mon, Aug 13, 2018 at 4:06 PM Anton Kedin <ke...@google.com> wrote:
>
>> Hi,
>>
>> I am planning to work on implementing a support for external schema
>> providers for Beam SQL and wanted to share a high level idea how I think
>> this can work.
>>
>> *Short Version*
>> Implement CREATE FOREIGN SCHEMA statement:
>>
>> CREATE FOREIGN SCHEMA
>>
>> TYPE 'bigquery'
>>
>> LOCATION 'dataset_example'
>>
>> AS bq;
>>
>> CREATE FOREIGN SCHEMA
>>
>> TYPE 'hcatalog'
>>
>> LOCATION 'hive-server:2341'
>>
>> AS hive;
>>
>> SELECT *
>>
>> FROM
>>
>> bq.table_example_bq AS bq_table1
>>
>> JOIN
>>
>> hive.table_example_hive AS hive_table1
>>
>> ON
>> bq_table1.some_field = hive_table1.some_other_field;
>>
>> *A Bit Longer Version: *
>> https://docs.google.com/document/d/1Ilk3OpDxrp3bHNlcnYDoj29tt9bd1E0EXt8i0WytNmQ
>>
>> Thoughts, ideas?
>>
>> Regards,
>> Anton
>>
>
Re: [SQL] Create External Schema
Posted by Mingmin Xu <mi...@gmail.com>.
awesome proposal to integrate with existing external schemas, add some
comments in doc.
On Mon, Aug 13, 2018 at 4:13 PM, Reuven Lax <re...@google.com> wrote:
> Is it possible to extend Beam's SchemaRegistry to do this?
>
> On Mon, Aug 13, 2018 at 4:06 PM Anton Kedin <ke...@google.com> wrote:
>
>> Hi,
>>
>> I am planning to work on implementing a support for external schema
>> providers for Beam SQL and wanted to share a high level idea how I think
>> this can work.
>>
>> *Short Version*
>> Implement CREATE FOREIGN SCHEMA statement:
>>
>> CREATE FOREIGN SCHEMA
>>
>> TYPE 'bigquery'
>>
>> LOCATION 'dataset_example'
>>
>> AS bq;
>>
>> CREATE FOREIGN SCHEMA
>>
>> TYPE 'hcatalog'
>>
>> LOCATION 'hive-server:2341'
>>
>> AS hive;
>>
>> SELECT *
>>
>> FROM
>>
>> bq.table_example_bq AS bq_table1
>>
>> JOIN
>>
>> hive.table_example_hive AS hive_table1
>>
>> ON
>> bq_table1.some_field = hive_table1.some_other_field;
>>
>> *A Bit Longer Version: *https://docs.google.com/document/d/
>> 1Ilk3OpDxrp3bHNlcnYDoj29tt9bd1E0EXt8i0WytNmQ
>>
>> Thoughts, ideas?
>>
>> Regards,
>> Anton
>>
>
--
----
Mingmin
Re: [SQL] Create External Schema
Posted by Reuven Lax <re...@google.com>.
Is it possible to extend Beam's SchemaRegistry to do this?
On Mon, Aug 13, 2018 at 4:06 PM Anton Kedin <ke...@google.com> wrote:
> Hi,
>
> I am planning to work on implementing a support for external schema
> providers for Beam SQL and wanted to share a high level idea how I think
> this can work.
>
> *Short Version*
> Implement CREATE FOREIGN SCHEMA statement:
>
> CREATE FOREIGN SCHEMA
>
> TYPE 'bigquery'
>
> LOCATION 'dataset_example'
>
> AS bq;
>
> CREATE FOREIGN SCHEMA
>
> TYPE 'hcatalog'
>
> LOCATION 'hive-server:2341'
>
> AS hive;
>
> SELECT *
>
> FROM
>
> bq.table_example_bq AS bq_table1
>
> JOIN
>
> hive.table_example_hive AS hive_table1
>
> ON
> bq_table1.some_field = hive_table1.some_other_field;
>
> *A Bit Longer Version: *
> https://docs.google.com/document/d/1Ilk3OpDxrp3bHNlcnYDoj29tt9bd1E0EXt8i0WytNmQ
>
> Thoughts, ideas?
>
> Regards,
> Anton
>