You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "lixin58688@163.com" <li...@163.com> on 2023/11/24 04:02:36 UTC

回复: Re:Re: [DISCUSS][FLINK-32993] Datagen connector handles length-constrained fields according to the schema definition by default

Thank Jane for providing examples to make discussions clearer.
Thank Lincoln and Xuyang for your feedback,I agree with you wholeheartedly that it is better to throw an error instead of ignoring it directly.
Extending datagen to generate variable length values is really an excelent idea, I will create another jira to follow up.

Taking the example provided,

  1.  For fixed-length data types (char, binary), two DDLs which custom length should throw exception like 'User-defined length of the fixed-length field f0 is not supported.'

  1.
CREATE TABLE foo (
f0 CHAR(5)
) WITH ('connector' = 'datagen', 'fields.f0.length' = '10');

CREATE TABLE bar (
f0 CHAR(5)
) WITH ('connector' = 'datagen', 'fields.f0.length' = '1');

  1.  For variable-length data types (varchar, varbinary)，the first DDL can be executed legally, if illegal user-defined length configured, will throw exception like 'User-defined length of the VARCHAR field %s should be shorter than the schema definition.'

  1.
CREATE TABLE meow (
f0 VARCHAR(20)
) WITH ('connector' = 'datagen', 'fields.f0.length' = '10');

  1.  For special variable-length data types, since the length of String and Bytes is very large (2^31 - 1), when users does not specify a smaller field length, Fields that occupy a huge amount of memory (estimated to be more than 2GB) will be generated by default, which can easily lead to "java.lang.OutOfMemoryError: Java heap space", so I recommend that the default length of these two fields is 100 just like before, but the length can be configured to less than 2^31-1.

  1.
CREATE TABLE purr (
f0 STRING
) WITH ('connector' = 'datagen', 'fields.f0.length' = '10');

Updates have been synchronized to the merge request [1]

WDYT?

[1] https://github.com/apache/flink/pull/23678


Best!
Yubin

Re: Re:Re: [DISCUSS][FLINK-32993] Datagen connector handles length-constrained fields according to the schema definition by default

Posted by Lincoln Lee <li...@gmail.com>.
Hi Yubin,

Thanks for the update!  +1 for keeping the default length of String and
Bytes unchange.

Best,
Lincoln Lee


lixin58688@163.com <li...@163.com> 于2023年11月24日周五 12:04写道：

> Thank Jane for providing examples to make discussions clearer.
> Thank Lincoln and Xuyang for your feedback,I agree with you wholeheartedly
> that it is better to throw an error instead of ignoring it directly.
> Extending datagen to generate variable length values is really an excelent
> idea, I will create another jira to follow up.
>
> Taking the example provided,
>
>   1.  For fixed-length data types (char, binary), two DDLs which custom
> length should throw exception like 'User-defined length of the fixed-length
> field f0 is not supported.'
>
>   1.
> CREATE TABLE foo (
> f0 CHAR(5)
> ) WITH ('connector' = 'datagen', 'fields.f0.length' = '10');
>
> CREATE TABLE bar (
> f0 CHAR(5)
> ) WITH ('connector' = 'datagen', 'fields.f0.length' = '1');
>
>   1.  For variable-length data types (varchar, varbinary)，the first DDL
> can be executed legally, if illegal user-defined length configured, will
> throw exception like 'User-defined length of the VARCHAR field %s should be
> shorter than the schema definition.'
>
>   1.
> CREATE TABLE meow (
> f0 VARCHAR(20)
> ) WITH ('connector' = 'datagen', 'fields.f0.length' = '10');
>
>   1.  For special variable-length data types, since the length of String
> and Bytes is very large (2^31 - 1), when users does not specify a smaller
> field length, Fields that occupy a huge amount of memory (estimated to be
> more than 2GB) will be generated by default, which can easily lead to
> "java.lang.OutOfMemoryError: Java heap space", so I recommend that the
> default length of these two fields is 100 just like before, but the length
> can be configured to less than 2^31-1.
>
>   1.
> CREATE TABLE purr (
> f0 STRING
> ) WITH ('connector' = 'datagen', 'fields.f0.length' = '10');
>
> Updates have been synchronized to the merge request [1]
>
> WDYT?
>
> [1] https://github.com/apache/flink/pull/23678
>
>
> Best!
> Yubin
>