You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by françois lacombe <fr...@dcbrain.com> on 2018/08/31 05:40:35 UTC
Table API: get Schema from TableSchema, TypeInformation[] or Avro
Hi all,
Today I'm looking into derivating an Avro schema json string into a Schema
object.
In the overview of
https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/table/connect.html
Avro is used as a format and never as a schema.
This was a topic in JIRA-9813
I can get a TableSchema with TableSchema schema =
TableSchema.fromTypeInfo(AvroSchemaConverter.convertToTypeInfo(sch_csv.toString()));
but I can't use it with BatchTableDescriptor.withSchema().
How can I get a Schema from TableSchema, TypeInformation<?>[] or even Avro
json string?
A little bridge is missing between TableSchema and
org.apache.flink.table.descriptors.Schema it seems.
Thanks in advance for any useful hint
François
Re: Table API: get Schema from TableSchema, TypeInformation[] or Avro
Posted by françois lacombe <fr...@dcbrain.com>.
Ok, looking forward reading your document.
All the best
François
2018-08-31 9:09 GMT+02:00 Timo Walther <tw...@apache.org>:
> Thanks for your response. I think we won't need this utility in the near
> future. As mentioned, I'm working on a design document that allows for
> better abstraction. I think I will publish it next week.
>
> Regards,
> Timo
>
>
> Am 31.08.18 um 08:36 schrieb françois lacombe:
>
> Hi Timo
>
> Yes it helps, thank you.
> I'll start building such an utility method. Are you interested to get the
> source?
>
> According to mapping here : https://ci.apache.org/
> projects/flink/flink-docs-release-1.6/dev/table/connect.
> html#apache-avro-format
> Is there any way to get corresponding TypeInformation<?> of an Avro type
> or should I hard code a List<String, TypeInformation<?>>?
>
> All the best
>
> François
>
> 2018-08-31 8:12 GMT+02:00 Timo Walther <tw...@apache.org>:
>
>> Hi,
>>
>> thanks for your feedback. I agree that the the current interfaces are not
>> flexible enough to fit to every use case. The unified connector API is a a
>> very recent feature that still needs some polishing. I'm working on a
>> design document to improve the situation there.
>>
>> For now, you can simply implement some utitilty method that just iterates
>> over column names and types of TableSchema and calls `schema.field(name,
>> type)`
>>
>> I hope this helps.
>>
>> Regards,
>> Timo
>>
>> Am 31.08.18 um 07:40 schrieb françois lacombe:
>>
>> Hi all,
>>>
>>> Today I'm looking into derivating an Avro schema json string into a
>>> Schema object.
>>> In the overview of https://ci.apache.org/projects
>>> /flink/flink-docs-release-1.6/dev/table/connect.html Avro is used as a
>>> format and never as a schema.
>>>
>>> This was a topic in JIRA-9813
>>> I can get a TableSchema with TableSchema schema =
>>> TableSchema.fromTypeInfo(AvroSchemaConverter.convertToTypeInfo(sch_csv.toString()));
>>> but I can't use it with BatchTableDescriptor.withSchema().
>>>
>>> How can I get a Schema from TableSchema, TypeInformation<?>[] or even
>>> Avro json string?
>>> A little bridge is missing between TableSchema and
>>> org.apache.flink.table.descriptors.Schema it seems.
>>>
>>> Thanks in advance for any useful hint
>>>
>>> François
>>>
>>
>>
>>
>
>
Re: Table API: get Schema from TableSchema, TypeInformation[] or
Avro
Posted by Timo Walther <tw...@apache.org>.
Thanks for your response. I think we won't need this utility in the near
future. As mentioned, I'm working on a design document that allows for
better abstraction. I think I will publish it next week.
Regards,
Timo
Am 31.08.18 um 08:36 schrieb françois lacombe:
> Hi Timo
>
> Yes it helps, thank you.
> I'll start building such an utility method. Are you interested to get
> the source?
>
> According to mapping here :
> https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/table/connect.html#apache-avro-format
> Is there any way to get corresponding TypeInformation<?> of an Avro
> type or should I hard code a List<String, TypeInformation<?>>?
>
> All the best
>
> François
>
> 2018-08-31 8:12 GMT+02:00 Timo Walther <twalthr@apache.org
> <ma...@apache.org>>:
>
> Hi,
>
> thanks for your feedback. I agree that the the current interfaces
> are not flexible enough to fit to every use case. The unified
> connector API is a a very recent feature that still needs some
> polishing. I'm working on a design document to improve the
> situation there.
>
> For now, you can simply implement some utitilty method that just
> iterates over column names and types of TableSchema and calls
> `schema.field(name, type)`
>
> I hope this helps.
>
> Regards,
> Timo
>
> Am 31.08.18 um 07:40 schrieb françois lacombe:
>
> Hi all,
>
> Today I'm looking into derivating an Avro schema json string
> into a Schema object.
> In the overview of
> https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/table/connect.html
> <https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/table/connect.html>
> Avro is used as a format and never as a schema.
>
> This was a topic in JIRA-9813
> I can get a TableSchema with TableSchema schema =
> TableSchema.fromTypeInfo(AvroSchemaConverter.convertToTypeInfo(sch_csv.toString()));
> but I can't use it with BatchTableDescriptor.withSchema().
>
> How can I get a Schema from TableSchema, TypeInformation<?>[]
> or even Avro json string?
> A little bridge is missing between TableSchema and
> org.apache.flink.table.descriptors.Schema it seems.
>
> Thanks in advance for any useful hint
>
> François
>
>
>
>
Re: Table API: get Schema from TableSchema, TypeInformation[] or Avro
Posted by françois lacombe <fr...@dcbrain.com>.
Hi Timo
Yes it helps, thank you.
I'll start building such an utility method. Are you interested to get the
source?
According to mapping here :
https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/table/connect.html#apache-avro-format
Is there any way to get corresponding TypeInformation<?> of an Avro type or
should I hard code a List<String, TypeInformation<?>>?
All the best
François
2018-08-31 8:12 GMT+02:00 Timo Walther <tw...@apache.org>:
> Hi,
>
> thanks for your feedback. I agree that the the current interfaces are not
> flexible enough to fit to every use case. The unified connector API is a a
> very recent feature that still needs some polishing. I'm working on a
> design document to improve the situation there.
>
> For now, you can simply implement some utitilty method that just iterates
> over column names and types of TableSchema and calls `schema.field(name,
> type)`
>
> I hope this helps.
>
> Regards,
> Timo
>
> Am 31.08.18 um 07:40 schrieb françois lacombe:
>
> Hi all,
>>
>> Today I'm looking into derivating an Avro schema json string into a
>> Schema object.
>> In the overview of https://ci.apache.org/projects
>> /flink/flink-docs-release-1.6/dev/table/connect.html Avro is used as a
>> format and never as a schema.
>>
>> This was a topic in JIRA-9813
>> I can get a TableSchema with TableSchema schema =
>> TableSchema.fromTypeInfo(AvroSchemaConverter.convertToTypeInfo(sch_csv.toString()));
>> but I can't use it with BatchTableDescriptor.withSchema().
>>
>> How can I get a Schema from TableSchema, TypeInformation<?>[] or even
>> Avro json string?
>> A little bridge is missing between TableSchema and
>> org.apache.flink.table.descriptors.Schema it seems.
>>
>> Thanks in advance for any useful hint
>>
>> François
>>
>
>
>
Re: Table API: get Schema from TableSchema, TypeInformation[] or
Avro
Posted by Timo Walther <tw...@apache.org>.
Hi,
thanks for your feedback. I agree that the the current interfaces are
not flexible enough to fit to every use case. The unified connector API
is a a very recent feature that still needs some polishing. I'm working
on a design document to improve the situation there.
For now, you can simply implement some utitilty method that just
iterates over column names and types of TableSchema and calls
`schema.field(name, type)`
I hope this helps.
Regards,
Timo
Am 31.08.18 um 07:40 schrieb françois lacombe:
> Hi all,
>
> Today I'm looking into derivating an Avro schema json string into a
> Schema object.
> In the overview of
> https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/table/connect.html
> Avro is used as a format and never as a schema.
>
> This was a topic in JIRA-9813
> I can get a TableSchema with TableSchema schema =
> TableSchema.fromTypeInfo(AvroSchemaConverter.convertToTypeInfo(sch_csv.toString()));
> but I can't use it with BatchTableDescriptor.withSchema().
>
> How can I get a Schema from TableSchema, TypeInformation<?>[] or even
> Avro json string?
> A little bridge is missing between TableSchema and
> org.apache.flink.table.descriptors.Schema it seems.
>
> Thanks in advance for any useful hint
>
> François