You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Xingcan Cui <xi...@gmail.com> on 2023/03/03 22:55:20 UTC
Large schemas lead to long DataStream-to-table transformation names
Hi all,
We are dealing with some streams with large (nested) schemas. When using `t
ableEnv.createTemporaryView()` to register a DataStream to a table, the
transformation always gets a large name. It's not a big problem, but quite
annoying since the UI and logs are hard to read.
Internally, `ExternalDynamicSource` (and `ExternalDynamicSink`) invokes
`physicalDataType.toString()` to generate an operator name (which will also
be used as the transformation name). I'm thinking to introduce a new table
config to either truncate the name or use a limited level of logicalType to
generate the name (works for nested schemas).
What do you think?
Best,
Xingcan
Re: Large schemas lead to long DataStream-to-table transformation names
Posted by Xingcan Cui <xi...@gmail.com>.
Oh, I just realized that FLIP-195 has already solved this. We'll upgrade
our Flink version to 1.15+. Thanks!
On Mon, Mar 6, 2023 at 10:08 AM Xingcan Cui <xi...@gmail.com> wrote:
> Hi Jark,
>
> Yes. I believe field names of the table would be enough to describe the
> conversion operator. I'll try to improve this.
>
> Best,
> Xingcan
>
> On Sun, Mar 5, 2023 at 9:18 PM Jark Wu <im...@gmail.com> wrote:
>
>> Hi Xingcan,
>>
>> I think `physicalDataType.toString()` is indeed verbose in this case.
>> Normal table scan generates descriptions using field names instead of the
>> full schema.
>> Will that help in your case?
>>
>> Best,
>> Jark
>>
>> On Sat, 4 Mar 2023 at 06:57, Xingcan Cui <xi...@gmail.com> wrote:
>>
>> > Hi all,
>> >
>> > We are dealing with some streams with large (nested) schemas. When
>> using `t
>> > ableEnv.createTemporaryView()` to register a DataStream to a table, the
>> > transformation always gets a large name. It's not a big problem, but
>> quite
>> > annoying since the UI and logs are hard to read.
>> >
>> > Internally, `ExternalDynamicSource` (and `ExternalDynamicSink`) invokes
>> > `physicalDataType.toString()` to generate an operator name (which will
>> also
>> > be used as the transformation name). I'm thinking to introduce a new
>> table
>> > config to either truncate the name or use a limited level of
>> logicalType to
>> > generate the name (works for nested schemas).
>> >
>> > What do you think?
>> >
>> > Best,
>> > Xingcan
>> >
>>
>
Re: Large schemas lead to long DataStream-to-table transformation names
Posted by Xingcan Cui <xi...@gmail.com>.
Hi Jark,
Yes. I believe field names of the table would be enough to describe the
conversion operator. I'll try to improve this.
Best,
Xingcan
On Sun, Mar 5, 2023 at 9:18 PM Jark Wu <im...@gmail.com> wrote:
> Hi Xingcan,
>
> I think `physicalDataType.toString()` is indeed verbose in this case.
> Normal table scan generates descriptions using field names instead of the
> full schema.
> Will that help in your case?
>
> Best,
> Jark
>
> On Sat, 4 Mar 2023 at 06:57, Xingcan Cui <xi...@gmail.com> wrote:
>
> > Hi all,
> >
> > We are dealing with some streams with large (nested) schemas. When using
> `t
> > ableEnv.createTemporaryView()` to register a DataStream to a table, the
> > transformation always gets a large name. It's not a big problem, but
> quite
> > annoying since the UI and logs are hard to read.
> >
> > Internally, `ExternalDynamicSource` (and `ExternalDynamicSink`) invokes
> > `physicalDataType.toString()` to generate an operator name (which will
> also
> > be used as the transformation name). I'm thinking to introduce a new
> table
> > config to either truncate the name or use a limited level of logicalType
> to
> > generate the name (works for nested schemas).
> >
> > What do you think?
> >
> > Best,
> > Xingcan
> >
>
Re: Large schemas lead to long DataStream-to-table transformation names
Posted by Jark Wu <im...@gmail.com>.
Hi Xingcan,
I think `physicalDataType.toString()` is indeed verbose in this case.
Normal table scan generates descriptions using field names instead of the
full schema.
Will that help in your case?
Best,
Jark
On Sat, 4 Mar 2023 at 06:57, Xingcan Cui <xi...@gmail.com> wrote:
> Hi all,
>
> We are dealing with some streams with large (nested) schemas. When using `t
> ableEnv.createTemporaryView()` to register a DataStream to a table, the
> transformation always gets a large name. It's not a big problem, but quite
> annoying since the UI and logs are hard to read.
>
> Internally, `ExternalDynamicSource` (and `ExternalDynamicSink`) invokes
> `physicalDataType.toString()` to generate an operator name (which will also
> be used as the transformation name). I'm thinking to introduce a new table
> config to either truncate the name or use a limited level of logicalType to
> generate the name (works for nested schemas).
>
> What do you think?
>
> Best,
> Xingcan
>