You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by yuxia <lu...@alumni.sjtu.edu.cn> on 2023/04/06 08:44:45 UTC
Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
Hi everyone.
If there are no other questions or concerns for the FLIP[1], I'd like to start the vote next Monday (4.10).
[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement
Best regards,
Yuxia
----- 原始邮件 -----
发件人: "yuxia" <lu...@alumni.sjtu.edu.cn>
收件人: "dev" <de...@flink.apache.org>
发送时间: 星期五, 2023年 3 月 24日 上午 11:27:42
主题: Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
Thanks all for your feedback.
@Shammon FY
My gut feeling is that the end user shouldn't care about whether it'll delete direcotry or move to Trash directory with the TRUNCATE TABLE statement. They only need to know it will delete all rows from a table.
To me, I think delete directory or move to trash is more likely to be a behavior of external storage level instead of SQL statement level. In Hive, if user configure Trash, it will then move files to trash for DROP statment.
Also, hardly did I see such usage with TRUNCATE TABLE statement in other engines. What's more, to support it, we have to extend the TRUNCATE TABLE synax which won't then compliant with SQL standard. I really don't want to do that and I believe it'll make user confused if we do so.
@Hang
`TRUNCATE TABLE` is meant to delete all rows of a base table. So, it makes no sense that table source implements it.
If user use TRUNCATE TABLE statement to truncate a table, the planner will only try to
find the DynamicTableSink for the corresponding table.
@Ran Tao
1: Thanks for you reminder. I said it won't support view in the FLIP, but forget to said temporary table is also not supported. Now, I add this part to this FLIP.
2: Yes, I also considered to incldue it in this FLIP before. But as far as I see, I haven't seen much usage of truncate table with partition. It's not as useful as truncate table. So, I tend to keep this FLIP simple in here without supporting truncate table with partition.
Also, seems for `truncate table with partition`, differnet engines may have differernt syntax;
Hive[1]/Spark[2] use the following syntax:
TRUNCATE TABLE table_name [PARTITION partition_spec]
SqlServer[3] use the follwoing syntax:
TRUNCATE TABLE { database_name.schema_name.table_name | schema_name.table_name | table_name } [ WITH ( PARTITIONS ( { <partition_number_expression> | <range> }
So, I'm tend to be cautious about it.
But I'm open to this. If there's any feedback or strong requirement, I don't mind to add it in this FLIP.
If we do need it in some day, I can propose it in a new FLIP. It won't break the current design.
As for concrete syntax in the FLIP, I think the current one is the concrete syntax, we don't allow TABLE keyword to be optional.
3: Thanks for your reminder, I have updadted the FLIP for this.
[1]https://cwiki.apache.org/confluence/display/hive/languagemanual+ddl#LanguageManualDDL-TruncateTable
[2]https://spark.apache.org/docs/3.0.0-preview/sql-ref-syntax-ddl-truncate-table.html
[3]https://learn.microsoft.com/en-us/sql/t-sql/statements/truncate-table-transact-sql?view=sql-server-ver16
Best regards,
Yuxia
----- 原始邮件 -----
发件人: "Ran Tao" <ch...@gmail.com>
收件人: "dev" <de...@flink.apache.org>
发送时间: 星期四, 2023年 3 月 23日 下午 6:28:17
主题: Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
Hi, yuxia.
Thanks for starting the discussion.
I think it's a nice improvement to support TRUNCATE TABLE statement because
many other mature engines supports it.
I have some questions.
1. because table has different types, whether we will support view or
temporary tables?
2. some other engines such as spark and hive support TRUNCATE TABLE with
partition. whether we will support？
btw, i think you need give the TRUNCATE TABLE concrete syntax in the FLIP
because some engines has different syntaxes.
for example, hive allow TRUNCATE TABLE be TRUNCATE [TABLE] which means
TABLE keyword can be optional.
3. The Proposed Changes try to use SqlToOperationConverter and run in
TableEnvironmentImpl#executeInternal.
I think it's out of date, the community is refactoring the conversion logic
from SqlNode to operation[1] and executions in TableEnvironmentImpl[2].
I suggest you can use new way to support it.
[1] https://issues.apache.org/jira/browse/FLINK-31464
[2] https://issues.apache.org/jira/browse/FLINK-31368
Best Regards,
Ran Tao
https://github.com/chucheng92
yuxia <lu...@alumni.sjtu.edu.cn> 于2023年3月22日周三 21:13写道：
> Hi, devs.
>
> I'd like to start a discussion about FLIP-302: Support TRUNCATE TABLE
> statement [1].
>
> The TRUNCATE TABLE statement is a SQL command that allows users to quickly
> and efficiently delete all rows from a table without dropping the table
> itself. This statement is commonly used in data warehouse, where large data
> sets are frequently loaded and unloaded from tables.
> So, this FLIP is meant to support TRUNCATE TABLE statement. M ore exactly,
> this FLIP will bring Flink the TRUNCATE TABLE syntax and an interface with
> which the coresponding connectors can implement their own logic for
> truncating table.
>
> Looking forwards to your feedback.
>
> [1]: [
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement
> |
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement
> ]
>
>
> Best regards,
> Yuxia
>
Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
Posted by yuxia <lu...@alumni.sjtu.edu.cn>.
Hi, all.
I started a vote for this FLIP[1], please vote there[2] or ask additional
questions here[3].
[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement+in+batch+mode
[2] https://lists.apache.org/thread/fosvz0zcyfn6bp6vz2oxl45vq9qhkn2v
[3] https://lists.apache.org/thread/m4r3wrd7p96wdst3nz3ncqzog6kf51cf
Best regards,
Yuxia
----- 原始邮件 -----
发件人: "Jark Wu" <im...@gmail.com>
收件人: "dev" <de...@flink.apache.org>
发送时间: 星期五, 2023年 4 月 14日 下午 11:04:58
主题: Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
Hi Yuxia,
Thank you for the updating. That sounds good to me.
Best,
Jark
> 2023年4月14日 19:00，yuxia <lu...@alumni.sjtu.edu.cn> 写道：
>
> Hi, Jark.
> I'm expecting if the "executeTruncation" returns false, Flink will throw an generic exception like "Fail to execute truncate table statement."
> But the connector implementation can also throw more specific exception like "Fail to execute truncate table statement for it table is been writing by other jobs".
>
> But after think it over, I'm afraid of the connector implementation will always return false to make Flink itself construnct the exception which maybe not very useful for it provides
> much less exception message instead of throwing more specific exception.
> So I decide to change it to `void executeTruncation()` and reminder to throw exception if truncate operation hasn't been executed successfully in the java doc of the method.
> I had updated this FLIP.
>
>
> Best regards,
> Yuxia
>
> ----- 原始邮件 -----
> 发件人: "Jark Wu" <im...@gmail.com>
> 收件人: "dev" <de...@flink.apache.org>
> 发送时间: 星期五, 2023年 4 月 14日 下午 5:10:48
> 主题: Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
>
> The FLIP looks good to me. +1 to start a vote.
>
> I just have a question: what will happen if the "executeTruncation" returns
> false without any exceptions?
>
> Best,
> Jark
>
> On Thu, 13 Apr 2023 at 19:59, Jing Ge <ji...@ververica.com.invalid> wrote:
>
>> Thanks Yuxia for the clarification and FLIP update. The FLIP looks good!
>>
>> Best regards,
>> Jing
>>
>> On Mon, Apr 10, 2023 at 3:51 AM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
>>
>>> 1:
>>> Actaully, considering the Flink's implementation, Flink just provides
>>> Truncate Table syntax to help user simlify data management as said in
>> this
>>> FLIP and push the implementation of Truncate Table to external connector.
>>> Normally, the effect of TRUENCATE TABLE is same as Drop Table + Create
>>> Table. But the real difference/benefit depends on the implementation of
>> the
>>> external connector.
>>> For example, for DROP Table statement, some external connectors may also
>>> drop the view related or other things.
>>> But for Truncate Table, the connectors may just delete all data without
>>> other operations.
>>>
>>>
>>> 2:
>>> At very begining, I'm thinking about in which case user may want to
>>> truncate a temporary table.
>>> I thought users can always create a table in catalog(if the table doesn't
>>> exist in a catalog) and truncate the table. So I tend not to expose it to
>>> user.
>>> But after I think it over again, I think it may be reasonable to support
>>> truncate a temporary table for the case that user just want to delete all
>>> datas from a table in an external storage without storing the metadata of
>>> the table in a catalog so that the other user/session can't see the
>> metada.
>>> I think we can relax to the constraint to support truncate temporary
>>> table. Now, I update it to the FLIP.
>>>
>>>
>>> 3:
>>> Thanks for your input, I agree that we can dicuss it in a different FLIP.
>>>
>>>
>>>
>>> Best regards,
>>> Yuxia
>>>
>>> ----- 原始邮件 -----
>>> 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
>>> 收件人: "dev" <de...@flink.apache.org>
>>> 发送时间: 星期六, 2023年 4 月 08日 上午 3:05:11
>>> 主题: Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
>>>
>>> Hi yuxia,
>>>
>>> Thanks for raising this topic. It is indeed a useful feature. +1 for
>>> having it in Flink. I have some small questions and it would be great if
>>> related information could be described in the FLIP.
>>>
>>> 1. Speaking of data warehouse use cases, what is the benefit of using
>>> TRUNCATE table over DROP table + CREATE table IF NOT EXISTS with the
>>> consideration of concrete Flink implementations? What would be the
>>> suggestion for users to use TRUNCATE instead of DROP + CREATE... and
>>> vise versa?
>>>
>>> 2. Since some engines support it, would you like to describe your
>>> thought about why TRUNCATE table does not support temporary table?
>>>
>>> 3. The partition support is an important feature, afaic. It might
>>> deserve a different FLIP and consider e.g.: TRUNCATE TABLE
>>> tt_dw_usr_exp_xxx PARTITION(dt='20230303') and ALTER TABLE
>>> tt_dw_usr_exp_xxx DROP IF EXISTS PARTITION(dt='20230303').
>>>
>>> Looking forward to your thoughts. Thanks!
>>>
>>> Best regards,
>>>
>>> Jing
>>>
>>> On 4/7/23 05:04, Jingsong Li wrote:
>>>> +1 for voting.
>>>>
>>>> Best,
>>>> Jingsong
>>>>
>>>> On Thu, Apr 6, 2023 at 4:52 PM yuxia <lu...@alumni.sjtu.edu.cn>
>>> wrote:
>>>>> Hi everyone.
>>>>>
>>>>> If there are no other questions or concerns for the FLIP[1], I'd like
>>> to start the vote next Monday (4.10).
>>>>>
>>>>> [1]
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement
>>>>>
>>>>> Best regards,
>>>>> Yuxia
>>>>>
>>>>> ----- 原始邮件 -----
>>>>> 发件人: "yuxia" <lu...@alumni.sjtu.edu.cn>
>>>>> 收件人: "dev" <de...@flink.apache.org>
>>>>> 发送时间: 星期五, 2023年 3 月 24日 上午 11:27:42
>>>>> 主题: Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
>>>>>
>>>>> Thanks all for your feedback.
>>>>>
>>>>> @Shammon FY
>>>>> My gut feeling is that the end user shouldn't care about whether it'll
>>> delete direcotry or move to Trash directory with the TRUNCATE TABLE
>>> statement. They only need to know it will delete all rows from a table.
>>>>> To me, I think delete directory or move to trash is more likely to be
>> a
>>> behavior of external storage level instead of SQL statement level. In
>> Hive,
>>> if user configure Trash, it will then move files to trash for DROP
>> statment.
>>>>> Also, hardly did I see such usage with TRUNCATE TABLE statement in
>>> other engines. What's more, to support it, we have to extend the TRUNCATE
>>> TABLE synax which won't then compliant with SQL standard. I really don't
>>> want to do that and I believe it'll make user confused if we do so.
>>>>>
>>>>> @Hang
>>>>> `TRUNCATE TABLE` is meant to delete all rows of a base table. So, it
>>> makes no sense that table source implements it.
>>>>> If user use TRUNCATE TABLE statement to truncate a table, the planner
>>> will only try to
>>>>> find the DynamicTableSink for the corresponding table.
>>>>>
>>>>> @Ran Tao
>>>>> 1: Thanks for you reminder. I said it won't support view in the FLIP,
>>> but forget to said temporary table is also not supported. Now, I add this
>>> part to this FLIP.
>>>>>
>>>>> 2: Yes, I also considered to incldue it in this FLIP before. But as
>> far
>>> as I see, I haven't seen much usage of truncate table with partition.
>> It's
>>> not as useful as truncate table. So, I tend to keep this FLIP simple in
>>> here without supporting truncate table with partition.
>>>>> Also, seems for `truncate table with partition`, differnet engines may
>>> have differernt syntax;
>>>>> Hive[1]/Spark[2] use the following syntax:
>>>>> TRUNCATE TABLE table_name [PARTITION partition_spec]
>>>>>
>>>>> SqlServer[3] use the follwoing syntax:
>>>>> TRUNCATE TABLE { database_name.schema_name.table_name |
>>> schema_name.table_name | table_name } [ WITH ( PARTITIONS ( {
>>> <partition_number_expression> | <range> }
>>>>> So, I'm tend to be cautious about it.
>>>>>
>>>>> But I'm open to this. If there's any feedback or strong requirement, I
>>> don't mind to add it in this FLIP.
>>>>> If we do need it in some day, I can propose it in a new FLIP. It won't
>>> break the current design.
>>>>>
>>>>> As for concrete syntax in the FLIP, I think the current one is the
>>> concrete syntax, we don't allow TABLE keyword to be optional.
>>>>>
>>>>> 3: Thanks for your reminder, I have updadted the FLIP for this.
>>>>>
>>>>>
>>>>> [1]
>>>
>> https://cwiki.apache.org/confluence/display/hive/languagemanual+ddl#LanguageManualDDL-TruncateTable
>>>>> [2]
>>>
>> https://spark.apache.org/docs/3.0.0-preview/sql-ref-syntax-ddl-truncate-table.html
>>>>> [3]
>>>
>> https://learn.microsoft.com/en-us/sql/t-sql/statements/truncate-table-transact-sql?view=sql-server-ver16
>>>>>
>>>>>
>>>>>
>>>>> Best regards,
>>>>> Yuxia
>>>>>
>>>>> ----- 原始邮件 -----
>>>>> 发件人: "Ran Tao" <ch...@gmail.com>
>>>>> 收件人: "dev" <de...@flink.apache.org>
>>>>> 发送时间: 星期四, 2023年 3 月 23日 下午 6:28:17
>>>>> 主题: Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
>>>>>
>>>>> Hi, yuxia.
>>>>>
>>>>> Thanks for starting the discussion.
>>>>> I think it's a nice improvement to support TRUNCATE TABLE statement
>>> because
>>>>> many other mature engines supports it.
>>>>>
>>>>> I have some questions.
>>>>> 1. because table has different types, whether we will support view or
>>>>> temporary tables?
>>>>>
>>>>> 2. some other engines such as spark and hive support TRUNCATE TABLE
>> with
>>>>> partition. whether we will support？
>>>>> btw, i think you need give the TRUNCATE TABLE concrete syntax in the
>>> FLIP
>>>>> because some engines has different syntaxes.
>>>>> for example, hive allow TRUNCATE TABLE be TRUNCATE [TABLE] which means
>>>>> TABLE keyword can be optional.
>>>>>
>>>>> 3. The Proposed Changes try to use SqlToOperationConverter and run in
>>>>> TableEnvironmentImpl#executeInternal.
>>>>> I think it's out of date, the community is refactoring the conversion
>>> logic
>>>>> from SqlNode to operation[1] and executions in
>> TableEnvironmentImpl[2].
>>>>> I suggest you can use new way to support it.
>>>>>
>>>>> [1] https://issues.apache.org/jira/browse/FLINK-31464
>>>>> [2] https://issues.apache.org/jira/browse/FLINK-31368
>>>>>
>>>>> Best Regards,
>>>>> Ran Tao
>>>>> https://github.com/chucheng92
>>>>>
>>>>>
>>>>> yuxia <lu...@alumni.sjtu.edu.cn> 于2023年3月22日周三 21:13写道：
>>>>>
>>>>>> Hi, devs.
>>>>>>
>>>>>> I'd like to start a discussion about FLIP-302: Support TRUNCATE TABLE
>>>>>> statement [1].
>>>>>>
>>>>>> The TRUNCATE TABLE statement is a SQL command that allows users to
>>> quickly
>>>>>> and efficiently delete all rows from a table without dropping the
>> table
>>>>>> itself. This statement is commonly used in data warehouse, where
>> large
>>> data
>>>>>> sets are frequently loaded and unloaded from tables.
>>>>>> So, this FLIP is meant to support TRUNCATE TABLE statement. M ore
>>> exactly,
>>>>>> this FLIP will bring Flink the TRUNCATE TABLE syntax and an interface
>>> with
>>>>>> which the coresponding connectors can implement their own logic for
>>>>>> truncating table.
>>>>>>
>>>>>> Looking forwards to your feedback.
>>>>>>
>>>>>> [1]: [
>>>>>>
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement
>>>>>> |
>>>>>>
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement
>>>>>> ]
>>>>>>
>>>>>>
>>>>>> Best regards,
>>>>>> Yuxia
>>>>>>
>>>
>>
Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
Posted by Jark Wu <im...@gmail.com>.
Hi Yuxia,
Thank you for the updating. That sounds good to me.
Best,
Jark
> 2023年4月14日 19:00，yuxia <lu...@alumni.sjtu.edu.cn> 写道：
>
> Hi, Jark.
> I'm expecting if the "executeTruncation" returns false, Flink will throw an generic exception like "Fail to execute truncate table statement."
> But the connector implementation can also throw more specific exception like "Fail to execute truncate table statement for it table is been writing by other jobs".
>
> But after think it over, I'm afraid of the connector implementation will always return false to make Flink itself construnct the exception which maybe not very useful for it provides
> much less exception message instead of throwing more specific exception.
> So I decide to change it to `void executeTruncation()` and reminder to throw exception if truncate operation hasn't been executed successfully in the java doc of the method.
> I had updated this FLIP.
>
>
> Best regards,
> Yuxia
>
> ----- 原始邮件 -----
> 发件人: "Jark Wu" <im...@gmail.com>
> 收件人: "dev" <de...@flink.apache.org>
> 发送时间: 星期五, 2023年 4 月 14日 下午 5:10:48
> 主题: Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
>
> The FLIP looks good to me. +1 to start a vote.
>
> I just have a question: what will happen if the "executeTruncation" returns
> false without any exceptions?
>
> Best,
> Jark
>
> On Thu, 13 Apr 2023 at 19:59, Jing Ge <ji...@ververica.com.invalid> wrote:
>
>> Thanks Yuxia for the clarification and FLIP update. The FLIP looks good!
>>
>> Best regards,
>> Jing
>>
>> On Mon, Apr 10, 2023 at 3:51 AM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
>>
>>> 1:
>>> Actaully, considering the Flink's implementation, Flink just provides
>>> Truncate Table syntax to help user simlify data management as said in
>> this
>>> FLIP and push the implementation of Truncate Table to external connector.
>>> Normally, the effect of TRUENCATE TABLE is same as Drop Table + Create
>>> Table. But the real difference/benefit depends on the implementation of
>> the
>>> external connector.
>>> For example, for DROP Table statement, some external connectors may also
>>> drop the view related or other things.
>>> But for Truncate Table, the connectors may just delete all data without
>>> other operations.
>>>
>>>
>>> 2:
>>> At very begining, I'm thinking about in which case user may want to
>>> truncate a temporary table.
>>> I thought users can always create a table in catalog(if the table doesn't
>>> exist in a catalog) and truncate the table. So I tend not to expose it to
>>> user.
>>> But after I think it over again, I think it may be reasonable to support
>>> truncate a temporary table for the case that user just want to delete all
>>> datas from a table in an external storage without storing the metadata of
>>> the table in a catalog so that the other user/session can't see the
>> metada.
>>> I think we can relax to the constraint to support truncate temporary
>>> table. Now, I update it to the FLIP.
>>>
>>>
>>> 3:
>>> Thanks for your input, I agree that we can dicuss it in a different FLIP.
>>>
>>>
>>>
>>> Best regards,
>>> Yuxia
>>>
>>> ----- 原始邮件 -----
>>> 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
>>> 收件人: "dev" <de...@flink.apache.org>
>>> 发送时间: 星期六, 2023年 4 月 08日 上午 3:05:11
>>> 主题: Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
>>>
>>> Hi yuxia,
>>>
>>> Thanks for raising this topic. It is indeed a useful feature. +1 for
>>> having it in Flink. I have some small questions and it would be great if
>>> related information could be described in the FLIP.
>>>
>>> 1. Speaking of data warehouse use cases, what is the benefit of using
>>> TRUNCATE table over DROP table + CREATE table IF NOT EXISTS with the
>>> consideration of concrete Flink implementations? What would be the
>>> suggestion for users to use TRUNCATE instead of DROP + CREATE... and
>>> vise versa?
>>>
>>> 2. Since some engines support it, would you like to describe your
>>> thought about why TRUNCATE table does not support temporary table?
>>>
>>> 3. The partition support is an important feature, afaic. It might
>>> deserve a different FLIP and consider e.g.: TRUNCATE TABLE
>>> tt_dw_usr_exp_xxx PARTITION(dt='20230303') and ALTER TABLE
>>> tt_dw_usr_exp_xxx DROP IF EXISTS PARTITION(dt='20230303').
>>>
>>> Looking forward to your thoughts. Thanks!
>>>
>>> Best regards,
>>>
>>> Jing
>>>
>>> On 4/7/23 05:04, Jingsong Li wrote:
>>>> +1 for voting.
>>>>
>>>> Best,
>>>> Jingsong
>>>>
>>>> On Thu, Apr 6, 2023 at 4:52 PM yuxia <lu...@alumni.sjtu.edu.cn>
>>> wrote:
>>>>> Hi everyone.
>>>>>
>>>>> If there are no other questions or concerns for the FLIP[1], I'd like
>>> to start the vote next Monday (4.10).
>>>>>
>>>>> [1]
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement
>>>>>
>>>>> Best regards,
>>>>> Yuxia
>>>>>
>>>>> ----- 原始邮件 -----
>>>>> 发件人: "yuxia" <lu...@alumni.sjtu.edu.cn>
>>>>> 收件人: "dev" <de...@flink.apache.org>
>>>>> 发送时间: 星期五, 2023年 3 月 24日 上午 11:27:42
>>>>> 主题: Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
>>>>>
>>>>> Thanks all for your feedback.
>>>>>
>>>>> @Shammon FY
>>>>> My gut feeling is that the end user shouldn't care about whether it'll
>>> delete direcotry or move to Trash directory with the TRUNCATE TABLE
>>> statement. They only need to know it will delete all rows from a table.
>>>>> To me, I think delete directory or move to trash is more likely to be
>> a
>>> behavior of external storage level instead of SQL statement level. In
>> Hive,
>>> if user configure Trash, it will then move files to trash for DROP
>> statment.
>>>>> Also, hardly did I see such usage with TRUNCATE TABLE statement in
>>> other engines. What's more, to support it, we have to extend the TRUNCATE
>>> TABLE synax which won't then compliant with SQL standard. I really don't
>>> want to do that and I believe it'll make user confused if we do so.
>>>>>
>>>>> @Hang
>>>>> `TRUNCATE TABLE` is meant to delete all rows of a base table. So, it
>>> makes no sense that table source implements it.
>>>>> If user use TRUNCATE TABLE statement to truncate a table, the planner
>>> will only try to
>>>>> find the DynamicTableSink for the corresponding table.
>>>>>
>>>>> @Ran Tao
>>>>> 1: Thanks for you reminder. I said it won't support view in the FLIP,
>>> but forget to said temporary table is also not supported. Now, I add this
>>> part to this FLIP.
>>>>>
>>>>> 2: Yes, I also considered to incldue it in this FLIP before. But as
>> far
>>> as I see, I haven't seen much usage of truncate table with partition.
>> It's
>>> not as useful as truncate table. So, I tend to keep this FLIP simple in
>>> here without supporting truncate table with partition.
>>>>> Also, seems for `truncate table with partition`, differnet engines may
>>> have differernt syntax;
>>>>> Hive[1]/Spark[2] use the following syntax:
>>>>> TRUNCATE TABLE table_name [PARTITION partition_spec]
>>>>>
>>>>> SqlServer[3] use the follwoing syntax:
>>>>> TRUNCATE TABLE { database_name.schema_name.table_name |
>>> schema_name.table_name | table_name } [ WITH ( PARTITIONS ( {
>>> <partition_number_expression> | <range> }
>>>>> So, I'm tend to be cautious about it.
>>>>>
>>>>> But I'm open to this. If there's any feedback or strong requirement, I
>>> don't mind to add it in this FLIP.
>>>>> If we do need it in some day, I can propose it in a new FLIP. It won't
>>> break the current design.
>>>>>
>>>>> As for concrete syntax in the FLIP, I think the current one is the
>>> concrete syntax, we don't allow TABLE keyword to be optional.
>>>>>
>>>>> 3: Thanks for your reminder, I have updadted the FLIP for this.
>>>>>
>>>>>
>>>>> [1]
>>>
>> https://cwiki.apache.org/confluence/display/hive/languagemanual+ddl#LanguageManualDDL-TruncateTable
>>>>> [2]
>>>
>> https://spark.apache.org/docs/3.0.0-preview/sql-ref-syntax-ddl-truncate-table.html
>>>>> [3]
>>>
>> https://learn.microsoft.com/en-us/sql/t-sql/statements/truncate-table-transact-sql?view=sql-server-ver16
>>>>>
>>>>>
>>>>>
>>>>> Best regards,
>>>>> Yuxia
>>>>>
>>>>> ----- 原始邮件 -----
>>>>> 发件人: "Ran Tao" <ch...@gmail.com>
>>>>> 收件人: "dev" <de...@flink.apache.org>
>>>>> 发送时间: 星期四, 2023年 3 月 23日 下午 6:28:17
>>>>> 主题: Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
>>>>>
>>>>> Hi, yuxia.
>>>>>
>>>>> Thanks for starting the discussion.
>>>>> I think it's a nice improvement to support TRUNCATE TABLE statement
>>> because
>>>>> many other mature engines supports it.
>>>>>
>>>>> I have some questions.
>>>>> 1. because table has different types, whether we will support view or
>>>>> temporary tables?
>>>>>
>>>>> 2. some other engines such as spark and hive support TRUNCATE TABLE
>> with
>>>>> partition. whether we will support？
>>>>> btw, i think you need give the TRUNCATE TABLE concrete syntax in the
>>> FLIP
>>>>> because some engines has different syntaxes.
>>>>> for example, hive allow TRUNCATE TABLE be TRUNCATE [TABLE] which means
>>>>> TABLE keyword can be optional.
>>>>>
>>>>> 3. The Proposed Changes try to use SqlToOperationConverter and run in
>>>>> TableEnvironmentImpl#executeInternal.
>>>>> I think it's out of date, the community is refactoring the conversion
>>> logic
>>>>> from SqlNode to operation[1] and executions in
>> TableEnvironmentImpl[2].
>>>>> I suggest you can use new way to support it.
>>>>>
>>>>> [1] https://issues.apache.org/jira/browse/FLINK-31464
>>>>> [2] https://issues.apache.org/jira/browse/FLINK-31368
>>>>>
>>>>> Best Regards,
>>>>> Ran Tao
>>>>> https://github.com/chucheng92
>>>>>
>>>>>
>>>>> yuxia <lu...@alumni.sjtu.edu.cn> 于2023年3月22日周三 21:13写道：
>>>>>
>>>>>> Hi, devs.
>>>>>>
>>>>>> I'd like to start a discussion about FLIP-302: Support TRUNCATE TABLE
>>>>>> statement [1].
>>>>>>
>>>>>> The TRUNCATE TABLE statement is a SQL command that allows users to
>>> quickly
>>>>>> and efficiently delete all rows from a table without dropping the
>> table
>>>>>> itself. This statement is commonly used in data warehouse, where
>> large
>>> data
>>>>>> sets are frequently loaded and unloaded from tables.
>>>>>> So, this FLIP is meant to support TRUNCATE TABLE statement. M ore
>>> exactly,
>>>>>> this FLIP will bring Flink the TRUNCATE TABLE syntax and an interface
>>> with
>>>>>> which the coresponding connectors can implement their own logic for
>>>>>> truncating table.
>>>>>>
>>>>>> Looking forwards to your feedback.
>>>>>>
>>>>>> [1]: [
>>>>>>
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement
>>>>>> |
>>>>>>
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement
>>>>>> ]
>>>>>>
>>>>>>
>>>>>> Best regards,
>>>>>> Yuxia
>>>>>>
>>>
>>
Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
Posted by yuxia <lu...@alumni.sjtu.edu.cn>.
Hi, Jark.
I'm expecting if the "executeTruncation" returns false, Flink will throw an generic exception like "Fail to execute truncate table statement."
But the connector implementation can also throw more specific exception like "Fail to execute truncate table statement for it table is been writing by other jobs".
But after think it over, I'm afraid of the connector implementation will always return false to make Flink itself construnct the exception which maybe not very useful for it provides
much less exception message instead of throwing more specific exception.
So I decide to change it to `void executeTruncation()` and reminder to throw exception if truncate operation hasn't been executed successfully in the java doc of the method.
I had updated this FLIP.
Best regards,
Yuxia
----- 原始邮件 -----
发件人: "Jark Wu" <im...@gmail.com>
收件人: "dev" <de...@flink.apache.org>
发送时间: 星期五, 2023年 4 月 14日 下午 5:10:48
主题: Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
The FLIP looks good to me. +1 to start a vote.
I just have a question: what will happen if the "executeTruncation" returns
false without any exceptions?
Best,
Jark
On Thu, 13 Apr 2023 at 19:59, Jing Ge <ji...@ververica.com.invalid> wrote:
> Thanks Yuxia for the clarification and FLIP update. The FLIP looks good!
>
> Best regards,
> Jing
>
> On Mon, Apr 10, 2023 at 3:51 AM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
>
> > 1:
> > Actaully, considering the Flink's implementation, Flink just provides
> > Truncate Table syntax to help user simlify data management as said in
> this
> > FLIP and push the implementation of Truncate Table to external connector.
> > Normally, the effect of TRUENCATE TABLE is same as Drop Table + Create
> > Table. But the real difference/benefit depends on the implementation of
> the
> > external connector.
> > For example, for DROP Table statement, some external connectors may also
> > drop the view related or other things.
> > But for Truncate Table, the connectors may just delete all data without
> > other operations.
> >
> >
> > 2:
> > At very begining, I'm thinking about in which case user may want to
> > truncate a temporary table.
> > I thought users can always create a table in catalog(if the table doesn't
> > exist in a catalog) and truncate the table. So I tend not to expose it to
> > user.
> > But after I think it over again, I think it may be reasonable to support
> > truncate a temporary table for the case that user just want to delete all
> > datas from a table in an external storage without storing the metadata of
> > the table in a catalog so that the other user/session can't see the
> metada.
> > I think we can relax to the constraint to support truncate temporary
> > table. Now, I update it to the FLIP.
> >
> >
> > 3:
> > Thanks for your input, I agree that we can dicuss it in a different FLIP.
> >
> >
> >
> > Best regards,
> > Yuxia
> >
> > ----- 原始邮件 -----
> > 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> > 收件人: "dev" <de...@flink.apache.org>
> > 发送时间: 星期六, 2023年 4 月 08日 上午 3:05:11
> > 主题: Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
> >
> > Hi yuxia,
> >
> > Thanks for raising this topic. It is indeed a useful feature. +1 for
> > having it in Flink. I have some small questions and it would be great if
> > related information could be described in the FLIP.
> >
> > 1. Speaking of data warehouse use cases, what is the benefit of using
> > TRUNCATE table over DROP table + CREATE table IF NOT EXISTS with the
> > consideration of concrete Flink implementations? What would be the
> > suggestion for users to use TRUNCATE instead of DROP + CREATE... and
> > vise versa?
> >
> > 2. Since some engines support it, would you like to describe your
> > thought about why TRUNCATE table does not support temporary table?
> >
> > 3. The partition support is an important feature, afaic. It might
> > deserve a different FLIP and consider e.g.: TRUNCATE TABLE
> > tt_dw_usr_exp_xxx PARTITION(dt='20230303') and ALTER TABLE
> > tt_dw_usr_exp_xxx DROP IF EXISTS PARTITION(dt='20230303').
> >
> > Looking forward to your thoughts. Thanks!
> >
> > Best regards,
> >
> > Jing
> >
> > On 4/7/23 05:04, Jingsong Li wrote:
> > > +1 for voting.
> > >
> > > Best,
> > > Jingsong
> > >
> > > On Thu, Apr 6, 2023 at 4:52 PM yuxia <lu...@alumni.sjtu.edu.cn>
> > wrote:
> > >> Hi everyone.
> > >>
> > >> If there are no other questions or concerns for the FLIP[1], I'd like
> > to start the vote next Monday (4.10).
> > >>
> > >> [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement
> > >>
> > >> Best regards,
> > >> Yuxia
> > >>
> > >> ----- 原始邮件 -----
> > >> 发件人: "yuxia" <lu...@alumni.sjtu.edu.cn>
> > >> 收件人: "dev" <de...@flink.apache.org>
> > >> 发送时间: 星期五, 2023年 3 月 24日 上午 11:27:42
> > >> 主题: Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
> > >>
> > >> Thanks all for your feedback.
> > >>
> > >> @Shammon FY
> > >> My gut feeling is that the end user shouldn't care about whether it'll
> > delete direcotry or move to Trash directory with the TRUNCATE TABLE
> > statement. They only need to know it will delete all rows from a table.
> > >> To me, I think delete directory or move to trash is more likely to be
> a
> > behavior of external storage level instead of SQL statement level. In
> Hive,
> > if user configure Trash, it will then move files to trash for DROP
> statment.
> > >> Also, hardly did I see such usage with TRUNCATE TABLE statement in
> > other engines. What's more, to support it, we have to extend the TRUNCATE
> > TABLE synax which won't then compliant with SQL standard. I really don't
> > want to do that and I believe it'll make user confused if we do so.
> > >>
> > >> @Hang
> > >> `TRUNCATE TABLE` is meant to delete all rows of a base table. So, it
> > makes no sense that table source implements it.
> > >> If user use TRUNCATE TABLE statement to truncate a table, the planner
> > will only try to
> > >> find the DynamicTableSink for the corresponding table.
> > >>
> > >> @Ran Tao
> > >> 1: Thanks for you reminder. I said it won't support view in the FLIP,
> > but forget to said temporary table is also not supported. Now, I add this
> > part to this FLIP.
> > >>
> > >> 2: Yes, I also considered to incldue it in this FLIP before. But as
> far
> > as I see, I haven't seen much usage of truncate table with partition.
> It's
> > not as useful as truncate table. So, I tend to keep this FLIP simple in
> > here without supporting truncate table with partition.
> > >> Also, seems for `truncate table with partition`, differnet engines may
> > have differernt syntax;
> > >> Hive[1]/Spark[2] use the following syntax:
> > >> TRUNCATE TABLE table_name [PARTITION partition_spec]
> > >>
> > >> SqlServer[3] use the follwoing syntax:
> > >> TRUNCATE TABLE { database_name.schema_name.table_name |
> > schema_name.table_name | table_name } [ WITH ( PARTITIONS ( {
> > <partition_number_expression> | <range> }
> > >> So, I'm tend to be cautious about it.
> > >>
> > >> But I'm open to this. If there's any feedback or strong requirement, I
> > don't mind to add it in this FLIP.
> > >> If we do need it in some day, I can propose it in a new FLIP. It won't
> > break the current design.
> > >>
> > >> As for concrete syntax in the FLIP, I think the current one is the
> > concrete syntax, we don't allow TABLE keyword to be optional.
> > >>
> > >> 3: Thanks for your reminder, I have updadted the FLIP for this.
> > >>
> > >>
> > >> [1]
> >
> https://cwiki.apache.org/confluence/display/hive/languagemanual+ddl#LanguageManualDDL-TruncateTable
> > >> [2]
> >
> https://spark.apache.org/docs/3.0.0-preview/sql-ref-syntax-ddl-truncate-table.html
> > >> [3]
> >
> https://learn.microsoft.com/en-us/sql/t-sql/statements/truncate-table-transact-sql?view=sql-server-ver16
> > >>
> > >>
> > >>
> > >> Best regards,
> > >> Yuxia
> > >>
> > >> ----- 原始邮件 -----
> > >> 发件人: "Ran Tao" <ch...@gmail.com>
> > >> 收件人: "dev" <de...@flink.apache.org>
> > >> 发送时间: 星期四, 2023年 3 月 23日 下午 6:28:17
> > >> 主题: Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
> > >>
> > >> Hi, yuxia.
> > >>
> > >> Thanks for starting the discussion.
> > >> I think it's a nice improvement to support TRUNCATE TABLE statement
> > because
> > >> many other mature engines supports it.
> > >>
> > >> I have some questions.
> > >> 1. because table has different types, whether we will support view or
> > >> temporary tables?
> > >>
> > >> 2. some other engines such as spark and hive support TRUNCATE TABLE
> with
> > >> partition. whether we will support？
> > >> btw, i think you need give the TRUNCATE TABLE concrete syntax in the
> > FLIP
> > >> because some engines has different syntaxes.
> > >> for example, hive allow TRUNCATE TABLE be TRUNCATE [TABLE] which means
> > >> TABLE keyword can be optional.
> > >>
> > >> 3. The Proposed Changes try to use SqlToOperationConverter and run in
> > >> TableEnvironmentImpl#executeInternal.
> > >> I think it's out of date, the community is refactoring the conversion
> > logic
> > >> from SqlNode to operation[1] and executions in
> TableEnvironmentImpl[2].
> > >> I suggest you can use new way to support it.
> > >>
> > >> [1] https://issues.apache.org/jira/browse/FLINK-31464
> > >> [2] https://issues.apache.org/jira/browse/FLINK-31368
> > >>
> > >> Best Regards,
> > >> Ran Tao
> > >> https://github.com/chucheng92
> > >>
> > >>
> > >> yuxia <lu...@alumni.sjtu.edu.cn> 于2023年3月22日周三 21:13写道：
> > >>
> > >>> Hi, devs.
> > >>>
> > >>> I'd like to start a discussion about FLIP-302: Support TRUNCATE TABLE
> > >>> statement [1].
> > >>>
> > >>> The TRUNCATE TABLE statement is a SQL command that allows users to
> > quickly
> > >>> and efficiently delete all rows from a table without dropping the
> table
> > >>> itself. This statement is commonly used in data warehouse, where
> large
> > data
> > >>> sets are frequently loaded and unloaded from tables.
> > >>> So, this FLIP is meant to support TRUNCATE TABLE statement. M ore
> > exactly,
> > >>> this FLIP will bring Flink the TRUNCATE TABLE syntax and an interface
> > with
> > >>> which the coresponding connectors can implement their own logic for
> > >>> truncating table.
> > >>>
> > >>> Looking forwards to your feedback.
> > >>>
> > >>> [1]: [
> > >>>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement
> > >>> |
> > >>>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement
> > >>> ]
> > >>>
> > >>>
> > >>> Best regards,
> > >>> Yuxia
> > >>>
> >
>
Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
Posted by Jark Wu <im...@gmail.com>.
The FLIP looks good to me. +1 to start a vote.
I just have a question: what will happen if the "executeTruncation" returns
false without any exceptions?
Best,
Jark
On Thu, 13 Apr 2023 at 19:59, Jing Ge <ji...@ververica.com.invalid> wrote:
> Thanks Yuxia for the clarification and FLIP update. The FLIP looks good!
>
> Best regards,
> Jing
>
> On Mon, Apr 10, 2023 at 3:51 AM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
>
> > 1:
> > Actaully, considering the Flink's implementation, Flink just provides
> > Truncate Table syntax to help user simlify data management as said in
> this
> > FLIP and push the implementation of Truncate Table to external connector.
> > Normally, the effect of TRUENCATE TABLE is same as Drop Table + Create
> > Table. But the real difference/benefit depends on the implementation of
> the
> > external connector.
> > For example, for DROP Table statement, some external connectors may also
> > drop the view related or other things.
> > But for Truncate Table, the connectors may just delete all data without
> > other operations.
> >
> >
> > 2:
> > At very begining, I'm thinking about in which case user may want to
> > truncate a temporary table.
> > I thought users can always create a table in catalog(if the table doesn't
> > exist in a catalog) and truncate the table. So I tend not to expose it to
> > user.
> > But after I think it over again, I think it may be reasonable to support
> > truncate a temporary table for the case that user just want to delete all
> > datas from a table in an external storage without storing the metadata of
> > the table in a catalog so that the other user/session can't see the
> metada.
> > I think we can relax to the constraint to support truncate temporary
> > table. Now, I update it to the FLIP.
> >
> >
> > 3:
> > Thanks for your input, I agree that we can dicuss it in a different FLIP.
> >
> >
> >
> > Best regards,
> > Yuxia
> >
> > ----- 原始邮件 -----
> > 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> > 收件人: "dev" <de...@flink.apache.org>
> > 发送时间: 星期六, 2023年 4 月 08日 上午 3:05:11
> > 主题: Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
> >
> > Hi yuxia,
> >
> > Thanks for raising this topic. It is indeed a useful feature. +1 for
> > having it in Flink. I have some small questions and it would be great if
> > related information could be described in the FLIP.
> >
> > 1. Speaking of data warehouse use cases, what is the benefit of using
> > TRUNCATE table over DROP table + CREATE table IF NOT EXISTS with the
> > consideration of concrete Flink implementations? What would be the
> > suggestion for users to use TRUNCATE instead of DROP + CREATE... and
> > vise versa?
> >
> > 2. Since some engines support it, would you like to describe your
> > thought about why TRUNCATE table does not support temporary table?
> >
> > 3. The partition support is an important feature, afaic. It might
> > deserve a different FLIP and consider e.g.: TRUNCATE TABLE
> > tt_dw_usr_exp_xxx PARTITION(dt='20230303') and ALTER TABLE
> > tt_dw_usr_exp_xxx DROP IF EXISTS PARTITION(dt='20230303').
> >
> > Looking forward to your thoughts. Thanks!
> >
> > Best regards,
> >
> > Jing
> >
> > On 4/7/23 05:04, Jingsong Li wrote:
> > > +1 for voting.
> > >
> > > Best,
> > > Jingsong
> > >
> > > On Thu, Apr 6, 2023 at 4:52 PM yuxia <lu...@alumni.sjtu.edu.cn>
> > wrote:
> > >> Hi everyone.
> > >>
> > >> If there are no other questions or concerns for the FLIP[1], I'd like
> > to start the vote next Monday (4.10).
> > >>
> > >> [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement
> > >>
> > >> Best regards,
> > >> Yuxia
> > >>
> > >> ----- 原始邮件 -----
> > >> 发件人: "yuxia" <lu...@alumni.sjtu.edu.cn>
> > >> 收件人: "dev" <de...@flink.apache.org>
> > >> 发送时间: 星期五, 2023年 3 月 24日 上午 11:27:42
> > >> 主题: Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
> > >>
> > >> Thanks all for your feedback.
> > >>
> > >> @Shammon FY
> > >> My gut feeling is that the end user shouldn't care about whether it'll
> > delete direcotry or move to Trash directory with the TRUNCATE TABLE
> > statement. They only need to know it will delete all rows from a table.
> > >> To me, I think delete directory or move to trash is more likely to be
> a
> > behavior of external storage level instead of SQL statement level. In
> Hive,
> > if user configure Trash, it will then move files to trash for DROP
> statment.
> > >> Also, hardly did I see such usage with TRUNCATE TABLE statement in
> > other engines. What's more, to support it, we have to extend the TRUNCATE
> > TABLE synax which won't then compliant with SQL standard. I really don't
> > want to do that and I believe it'll make user confused if we do so.
> > >>
> > >> @Hang
> > >> `TRUNCATE TABLE` is meant to delete all rows of a base table. So, it
> > makes no sense that table source implements it.
> > >> If user use TRUNCATE TABLE statement to truncate a table, the planner
> > will only try to
> > >> find the DynamicTableSink for the corresponding table.
> > >>
> > >> @Ran Tao
> > >> 1: Thanks for you reminder. I said it won't support view in the FLIP,
> > but forget to said temporary table is also not supported. Now, I add this
> > part to this FLIP.
> > >>
> > >> 2: Yes, I also considered to incldue it in this FLIP before. But as
> far
> > as I see, I haven't seen much usage of truncate table with partition.
> It's
> > not as useful as truncate table. So, I tend to keep this FLIP simple in
> > here without supporting truncate table with partition.
> > >> Also, seems for `truncate table with partition`, differnet engines may
> > have differernt syntax;
> > >> Hive[1]/Spark[2] use the following syntax:
> > >> TRUNCATE TABLE table_name [PARTITION partition_spec]
> > >>
> > >> SqlServer[3] use the follwoing syntax:
> > >> TRUNCATE TABLE { database_name.schema_name.table_name |
> > schema_name.table_name | table_name } [ WITH ( PARTITIONS ( {
> > <partition_number_expression> | <range> }
> > >> So, I'm tend to be cautious about it.
> > >>
> > >> But I'm open to this. If there's any feedback or strong requirement, I
> > don't mind to add it in this FLIP.
> > >> If we do need it in some day, I can propose it in a new FLIP. It won't
> > break the current design.
> > >>
> > >> As for concrete syntax in the FLIP, I think the current one is the
> > concrete syntax, we don't allow TABLE keyword to be optional.
> > >>
> > >> 3: Thanks for your reminder, I have updadted the FLIP for this.
> > >>
> > >>
> > >> [1]
> >
> https://cwiki.apache.org/confluence/display/hive/languagemanual+ddl#LanguageManualDDL-TruncateTable
> > >> [2]
> >
> https://spark.apache.org/docs/3.0.0-preview/sql-ref-syntax-ddl-truncate-table.html
> > >> [3]
> >
> https://learn.microsoft.com/en-us/sql/t-sql/statements/truncate-table-transact-sql?view=sql-server-ver16
> > >>
> > >>
> > >>
> > >> Best regards,
> > >> Yuxia
> > >>
> > >> ----- 原始邮件 -----
> > >> 发件人: "Ran Tao" <ch...@gmail.com>
> > >> 收件人: "dev" <de...@flink.apache.org>
> > >> 发送时间: 星期四, 2023年 3 月 23日 下午 6:28:17
> > >> 主题: Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
> > >>
> > >> Hi, yuxia.
> > >>
> > >> Thanks for starting the discussion.
> > >> I think it's a nice improvement to support TRUNCATE TABLE statement
> > because
> > >> many other mature engines supports it.
> > >>
> > >> I have some questions.
> > >> 1. because table has different types, whether we will support view or
> > >> temporary tables?
> > >>
> > >> 2. some other engines such as spark and hive support TRUNCATE TABLE
> with
> > >> partition. whether we will support？
> > >> btw, i think you need give the TRUNCATE TABLE concrete syntax in the
> > FLIP
> > >> because some engines has different syntaxes.
> > >> for example, hive allow TRUNCATE TABLE be TRUNCATE [TABLE] which means
> > >> TABLE keyword can be optional.
> > >>
> > >> 3. The Proposed Changes try to use SqlToOperationConverter and run in
> > >> TableEnvironmentImpl#executeInternal.
> > >> I think it's out of date, the community is refactoring the conversion
> > logic
> > >> from SqlNode to operation[1] and executions in
> TableEnvironmentImpl[2].
> > >> I suggest you can use new way to support it.
> > >>
> > >> [1] https://issues.apache.org/jira/browse/FLINK-31464
> > >> [2] https://issues.apache.org/jira/browse/FLINK-31368
> > >>
> > >> Best Regards,
> > >> Ran Tao
> > >> https://github.com/chucheng92
> > >>
> > >>
> > >> yuxia <lu...@alumni.sjtu.edu.cn> 于2023年3月22日周三 21:13写道：
> > >>
> > >>> Hi, devs.
> > >>>
> > >>> I'd like to start a discussion about FLIP-302: Support TRUNCATE TABLE
> > >>> statement [1].
> > >>>
> > >>> The TRUNCATE TABLE statement is a SQL command that allows users to
> > quickly
> > >>> and efficiently delete all rows from a table without dropping the
> table
> > >>> itself. This statement is commonly used in data warehouse, where
> large
> > data
> > >>> sets are frequently loaded and unloaded from tables.
> > >>> So, this FLIP is meant to support TRUNCATE TABLE statement. M ore
> > exactly,
> > >>> this FLIP will bring Flink the TRUNCATE TABLE syntax and an interface
> > with
> > >>> which the coresponding connectors can implement their own logic for
> > >>> truncating table.
> > >>>
> > >>> Looking forwards to your feedback.
> > >>>
> > >>> [1]: [
> > >>>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement
> > >>> |
> > >>>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement
> > >>> ]
> > >>>
> > >>>
> > >>> Best regards,
> > >>> Yuxia
> > >>>
> >
>
Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
Posted by Jing Ge <ji...@ververica.com.INVALID>.
Thanks Yuxia for the clarification and FLIP update. The FLIP looks good!
Best regards,
Jing
On Mon, Apr 10, 2023 at 3:51 AM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
> 1:
> Actaully, considering the Flink's implementation, Flink just provides
> Truncate Table syntax to help user simlify data management as said in this
> FLIP and push the implementation of Truncate Table to external connector.
> Normally, the effect of TRUENCATE TABLE is same as Drop Table + Create
> Table. But the real difference/benefit depends on the implementation of the
> external connector.
> For example, for DROP Table statement, some external connectors may also
> drop the view related or other things.
> But for Truncate Table, the connectors may just delete all data without
> other operations.
>
>
> 2:
> At very begining, I'm thinking about in which case user may want to
> truncate a temporary table.
> I thought users can always create a table in catalog(if the table doesn't
> exist in a catalog) and truncate the table. So I tend not to expose it to
> user.
> But after I think it over again, I think it may be reasonable to support
> truncate a temporary table for the case that user just want to delete all
> datas from a table in an external storage without storing the metadata of
> the table in a catalog so that the other user/session can't see the metada.
> I think we can relax to the constraint to support truncate temporary
> table. Now, I update it to the FLIP.
>
>
> 3:
> Thanks for your input, I agree that we can dicuss it in a different FLIP.
>
>
>
> Best regards,
> Yuxia
>
> ----- 原始邮件 -----
> 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> 收件人: "dev" <de...@flink.apache.org>
> 发送时间: 星期六, 2023年 4 月 08日 上午 3:05:11
> 主题: Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
>
> Hi yuxia,
>
> Thanks for raising this topic. It is indeed a useful feature. +1 for
> having it in Flink. I have some small questions and it would be great if
> related information could be described in the FLIP.
>
> 1. Speaking of data warehouse use cases, what is the benefit of using
> TRUNCATE table over DROP table + CREATE table IF NOT EXISTS with the
> consideration of concrete Flink implementations? What would be the
> suggestion for users to use TRUNCATE instead of DROP + CREATE... and
> vise versa?
>
> 2. Since some engines support it, would you like to describe your
> thought about why TRUNCATE table does not support temporary table?
>
> 3. The partition support is an important feature, afaic. It might
> deserve a different FLIP and consider e.g.: TRUNCATE TABLE
> tt_dw_usr_exp_xxx PARTITION(dt='20230303') and ALTER TABLE
> tt_dw_usr_exp_xxx DROP IF EXISTS PARTITION(dt='20230303').
>
> Looking forward to your thoughts. Thanks!
>
> Best regards,
>
> Jing
>
> On 4/7/23 05:04, Jingsong Li wrote:
> > +1 for voting.
> >
> > Best,
> > Jingsong
> >
> > On Thu, Apr 6, 2023 at 4:52 PM yuxia <lu...@alumni.sjtu.edu.cn>
> wrote:
> >> Hi everyone.
> >>
> >> If there are no other questions or concerns for the FLIP[1], I'd like
> to start the vote next Monday (4.10).
> >>
> >> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement
> >>
> >> Best regards,
> >> Yuxia
> >>
> >> ----- 原始邮件 -----
> >> 发件人: "yuxia" <lu...@alumni.sjtu.edu.cn>
> >> 收件人: "dev" <de...@flink.apache.org>
> >> 发送时间: 星期五, 2023年 3 月 24日 上午 11:27:42
> >> 主题: Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
> >>
> >> Thanks all for your feedback.
> >>
> >> @Shammon FY
> >> My gut feeling is that the end user shouldn't care about whether it'll
> delete direcotry or move to Trash directory with the TRUNCATE TABLE
> statement. They only need to know it will delete all rows from a table.
> >> To me, I think delete directory or move to trash is more likely to be a
> behavior of external storage level instead of SQL statement level. In Hive,
> if user configure Trash, it will then move files to trash for DROP statment.
> >> Also, hardly did I see such usage with TRUNCATE TABLE statement in
> other engines. What's more, to support it, we have to extend the TRUNCATE
> TABLE synax which won't then compliant with SQL standard. I really don't
> want to do that and I believe it'll make user confused if we do so.
> >>
> >> @Hang
> >> `TRUNCATE TABLE` is meant to delete all rows of a base table. So, it
> makes no sense that table source implements it.
> >> If user use TRUNCATE TABLE statement to truncate a table, the planner
> will only try to
> >> find the DynamicTableSink for the corresponding table.
> >>
> >> @Ran Tao
> >> 1: Thanks for you reminder. I said it won't support view in the FLIP,
> but forget to said temporary table is also not supported. Now, I add this
> part to this FLIP.
> >>
> >> 2: Yes, I also considered to incldue it in this FLIP before. But as far
> as I see, I haven't seen much usage of truncate table with partition. It's
> not as useful as truncate table. So, I tend to keep this FLIP simple in
> here without supporting truncate table with partition.
> >> Also, seems for `truncate table with partition`, differnet engines may
> have differernt syntax;
> >> Hive[1]/Spark[2] use the following syntax:
> >> TRUNCATE TABLE table_name [PARTITION partition_spec]
> >>
> >> SqlServer[3] use the follwoing syntax:
> >> TRUNCATE TABLE { database_name.schema_name.table_name |
> schema_name.table_name | table_name } [ WITH ( PARTITIONS ( {
> <partition_number_expression> | <range> }
> >> So, I'm tend to be cautious about it.
> >>
> >> But I'm open to this. If there's any feedback or strong requirement, I
> don't mind to add it in this FLIP.
> >> If we do need it in some day, I can propose it in a new FLIP. It won't
> break the current design.
> >>
> >> As for concrete syntax in the FLIP, I think the current one is the
> concrete syntax, we don't allow TABLE keyword to be optional.
> >>
> >> 3: Thanks for your reminder, I have updadted the FLIP for this.
> >>
> >>
> >> [1]
> https://cwiki.apache.org/confluence/display/hive/languagemanual+ddl#LanguageManualDDL-TruncateTable
> >> [2]
> https://spark.apache.org/docs/3.0.0-preview/sql-ref-syntax-ddl-truncate-table.html
> >> [3]
> https://learn.microsoft.com/en-us/sql/t-sql/statements/truncate-table-transact-sql?view=sql-server-ver16
> >>
> >>
> >>
> >> Best regards,
> >> Yuxia
> >>
> >> ----- 原始邮件 -----
> >> 发件人: "Ran Tao" <ch...@gmail.com>
> >> 收件人: "dev" <de...@flink.apache.org>
> >> 发送时间: 星期四, 2023年 3 月 23日 下午 6:28:17
> >> 主题: Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
> >>
> >> Hi, yuxia.
> >>
> >> Thanks for starting the discussion.
> >> I think it's a nice improvement to support TRUNCATE TABLE statement
> because
> >> many other mature engines supports it.
> >>
> >> I have some questions.
> >> 1. because table has different types, whether we will support view or
> >> temporary tables?
> >>
> >> 2. some other engines such as spark and hive support TRUNCATE TABLE with
> >> partition. whether we will support？
> >> btw, i think you need give the TRUNCATE TABLE concrete syntax in the
> FLIP
> >> because some engines has different syntaxes.
> >> for example, hive allow TRUNCATE TABLE be TRUNCATE [TABLE] which means
> >> TABLE keyword can be optional.
> >>
> >> 3. The Proposed Changes try to use SqlToOperationConverter and run in
> >> TableEnvironmentImpl#executeInternal.
> >> I think it's out of date, the community is refactoring the conversion
> logic
> >> from SqlNode to operation[1] and executions in TableEnvironmentImpl[2].
> >> I suggest you can use new way to support it.
> >>
> >> [1] https://issues.apache.org/jira/browse/FLINK-31464
> >> [2] https://issues.apache.org/jira/browse/FLINK-31368
> >>
> >> Best Regards,
> >> Ran Tao
> >> https://github.com/chucheng92
> >>
> >>
> >> yuxia <lu...@alumni.sjtu.edu.cn> 于2023年3月22日周三 21:13写道：
> >>
> >>> Hi, devs.
> >>>
> >>> I'd like to start a discussion about FLIP-302: Support TRUNCATE TABLE
> >>> statement [1].
> >>>
> >>> The TRUNCATE TABLE statement is a SQL command that allows users to
> quickly
> >>> and efficiently delete all rows from a table without dropping the table
> >>> itself. This statement is commonly used in data warehouse, where large
> data
> >>> sets are frequently loaded and unloaded from tables.
> >>> So, this FLIP is meant to support TRUNCATE TABLE statement. M ore
> exactly,
> >>> this FLIP will bring Flink the TRUNCATE TABLE syntax and an interface
> with
> >>> which the coresponding connectors can implement their own logic for
> >>> truncating table.
> >>>
> >>> Looking forwards to your feedback.
> >>>
> >>> [1]: [
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement
> >>> |
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement
> >>> ]
> >>>
> >>>
> >>> Best regards,
> >>> Yuxia
> >>>
>
Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
Posted by yuxia <lu...@alumni.sjtu.edu.cn>.
1:
Actaully, considering the Flink's implementation, Flink just provides Truncate Table syntax to help user simlify data management as said in this FLIP and push the implementation of Truncate Table to external connector. Normally, the effect of TRUENCATE TABLE is same as Drop Table + Create Table. But the real difference/benefit depends on the implementation of the external connector.
For example, for DROP Table statement, some external connectors may also drop the view related or other things.
But for Truncate Table, the connectors may just delete all data without other operations.
2:
At very begining, I'm thinking about in which case user may want to truncate a temporary table.
I thought users can always create a table in catalog(if the table doesn't exist in a catalog) and truncate the table. So I tend not to expose it to user.
But after I think it over again, I think it may be reasonable to support truncate a temporary table for the case that user just want to delete all datas from a table in an external storage without storing the metadata of the table in a catalog so that the other user/session can't see the metada.
I think we can relax to the constraint to support truncate temporary table. Now, I update it to the FLIP.
3:
Thanks for your input, I agree that we can dicuss it in a different FLIP.
Best regards,
Yuxia
----- 原始邮件 -----
发件人: "Jing Ge" <ji...@ververica.com.INVALID>
收件人: "dev" <de...@flink.apache.org>
发送时间: 星期六, 2023年 4 月 08日 上午 3:05:11
主题: Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
Hi yuxia,
Thanks for raising this topic. It is indeed a useful feature. +1 for
having it in Flink. I have some small questions and it would be great if
related information could be described in the FLIP.
1. Speaking of data warehouse use cases, what is the benefit of using
TRUNCATE table over DROP table + CREATE table IF NOT EXISTS with the
consideration of concrete Flink implementations? What would be the
suggestion for users to use TRUNCATE instead of DROP + CREATE... and
vise versa?
2. Since some engines support it, would you like to describe your
thought about why TRUNCATE table does not support temporary table?
3. The partition support is an important feature, afaic. It might
deserve a different FLIP and consider e.g.: TRUNCATE TABLE
tt_dw_usr_exp_xxx PARTITION(dt='20230303') and ALTER TABLE
tt_dw_usr_exp_xxx DROP IF EXISTS PARTITION(dt='20230303').
Looking forward to your thoughts. Thanks!
Best regards,
Jing
On 4/7/23 05:04, Jingsong Li wrote:
> +1 for voting.
>
> Best,
> Jingsong
>
> On Thu, Apr 6, 2023 at 4:52 PM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
>> Hi everyone.
>>
>> If there are no other questions or concerns for the FLIP[1], I'd like to start the vote next Monday (4.10).
>>
>> [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement
>>
>> Best regards,
>> Yuxia
>>
>> ----- 原始邮件 -----
>> 发件人: "yuxia" <lu...@alumni.sjtu.edu.cn>
>> 收件人: "dev" <de...@flink.apache.org>
>> 发送时间: 星期五, 2023年 3 月 24日 上午 11:27:42
>> 主题: Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
>>
>> Thanks all for your feedback.
>>
>> @Shammon FY
>> My gut feeling is that the end user shouldn't care about whether it'll delete direcotry or move to Trash directory with the TRUNCATE TABLE statement. They only need to know it will delete all rows from a table.
>> To me, I think delete directory or move to trash is more likely to be a behavior of external storage level instead of SQL statement level. In Hive, if user configure Trash, it will then move files to trash for DROP statment.
>> Also, hardly did I see such usage with TRUNCATE TABLE statement in other engines. What's more, to support it, we have to extend the TRUNCATE TABLE synax which won't then compliant with SQL standard. I really don't want to do that and I believe it'll make user confused if we do so.
>>
>> @Hang
>> `TRUNCATE TABLE` is meant to delete all rows of a base table. So, it makes no sense that table source implements it.
>> If user use TRUNCATE TABLE statement to truncate a table, the planner will only try to
>> find the DynamicTableSink for the corresponding table.
>>
>> @Ran Tao
>> 1: Thanks for you reminder. I said it won't support view in the FLIP, but forget to said temporary table is also not supported. Now, I add this part to this FLIP.
>>
>> 2: Yes, I also considered to incldue it in this FLIP before. But as far as I see, I haven't seen much usage of truncate table with partition. It's not as useful as truncate table. So, I tend to keep this FLIP simple in here without supporting truncate table with partition.
>> Also, seems for `truncate table with partition`, differnet engines may have differernt syntax;
>> Hive[1]/Spark[2] use the following syntax:
>> TRUNCATE TABLE table_name [PARTITION partition_spec]
>>
>> SqlServer[3] use the follwoing syntax:
>> TRUNCATE TABLE { database_name.schema_name.table_name | schema_name.table_name | table_name } [ WITH ( PARTITIONS ( { <partition_number_expression> | <range> }
>> So, I'm tend to be cautious about it.
>>
>> But I'm open to this. If there's any feedback or strong requirement, I don't mind to add it in this FLIP.
>> If we do need it in some day, I can propose it in a new FLIP. It won't break the current design.
>>
>> As for concrete syntax in the FLIP, I think the current one is the concrete syntax, we don't allow TABLE keyword to be optional.
>>
>> 3: Thanks for your reminder, I have updadted the FLIP for this.
>>
>>
>> [1]https://cwiki.apache.org/confluence/display/hive/languagemanual+ddl#LanguageManualDDL-TruncateTable
>> [2]https://spark.apache.org/docs/3.0.0-preview/sql-ref-syntax-ddl-truncate-table.html
>> [3]https://learn.microsoft.com/en-us/sql/t-sql/statements/truncate-table-transact-sql?view=sql-server-ver16
>>
>>
>>
>> Best regards,
>> Yuxia
>>
>> ----- 原始邮件 -----
>> 发件人: "Ran Tao" <ch...@gmail.com>
>> 收件人: "dev" <de...@flink.apache.org>
>> 发送时间: 星期四, 2023年 3 月 23日 下午 6:28:17
>> 主题: Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
>>
>> Hi, yuxia.
>>
>> Thanks for starting the discussion.
>> I think it's a nice improvement to support TRUNCATE TABLE statement because
>> many other mature engines supports it.
>>
>> I have some questions.
>> 1. because table has different types, whether we will support view or
>> temporary tables?
>>
>> 2. some other engines such as spark and hive support TRUNCATE TABLE with
>> partition. whether we will support？
>> btw, i think you need give the TRUNCATE TABLE concrete syntax in the FLIP
>> because some engines has different syntaxes.
>> for example, hive allow TRUNCATE TABLE be TRUNCATE [TABLE] which means
>> TABLE keyword can be optional.
>>
>> 3. The Proposed Changes try to use SqlToOperationConverter and run in
>> TableEnvironmentImpl#executeInternal.
>> I think it's out of date, the community is refactoring the conversion logic
>> from SqlNode to operation[1] and executions in TableEnvironmentImpl[2].
>> I suggest you can use new way to support it.
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-31464
>> [2] https://issues.apache.org/jira/browse/FLINK-31368
>>
>> Best Regards,
>> Ran Tao
>> https://github.com/chucheng92
>>
>>
>> yuxia <lu...@alumni.sjtu.edu.cn> 于2023年3月22日周三 21:13写道：
>>
>>> Hi, devs.
>>>
>>> I'd like to start a discussion about FLIP-302: Support TRUNCATE TABLE
>>> statement [1].
>>>
>>> The TRUNCATE TABLE statement is a SQL command that allows users to quickly
>>> and efficiently delete all rows from a table without dropping the table
>>> itself. This statement is commonly used in data warehouse, where large data
>>> sets are frequently loaded and unloaded from tables.
>>> So, this FLIP is meant to support TRUNCATE TABLE statement. M ore exactly,
>>> this FLIP will bring Flink the TRUNCATE TABLE syntax and an interface with
>>> which the coresponding connectors can implement their own logic for
>>> truncating table.
>>>
>>> Looking forwards to your feedback.
>>>
>>> [1]: [
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement
>>> |
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement
>>> ]
>>>
>>>
>>> Best regards,
>>> Yuxia
>>>
Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
Posted by Jing Ge <ji...@ververica.com.INVALID>.
Hi yuxia,
Thanks for raising this topic. It is indeed a useful feature. +1 for
having it in Flink. I have some small questions and it would be great if
related information could be described in the FLIP.
1. Speaking of data warehouse use cases, what is the benefit of using
TRUNCATE table over DROP table + CREATE table IF NOT EXISTS with the
consideration of concrete Flink implementations? What would be the
suggestion for users to use TRUNCATE instead of DROP + CREATE... and
vise versa?
2. Since some engines support it, would you like to describe your
thought about why TRUNCATE table does not support temporary table?
3. The partition support is an important feature, afaic. It might
deserve a different FLIP and consider e.g.: TRUNCATE TABLE
tt_dw_usr_exp_xxx PARTITION(dt='20230303') and ALTER TABLE
tt_dw_usr_exp_xxx DROP IF EXISTS PARTITION(dt='20230303').
Looking forward to your thoughts. Thanks!
Best regards,
Jing
On 4/7/23 05:04, Jingsong Li wrote:
> +1 for voting.
>
> Best,
> Jingsong
>
> On Thu, Apr 6, 2023 at 4:52 PM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
>> Hi everyone.
>>
>> If there are no other questions or concerns for the FLIP[1], I'd like to start the vote next Monday (4.10).
>>
>> [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement
>>
>> Best regards,
>> Yuxia
>>
>> ----- 原始邮件 -----
>> 发件人: "yuxia" <lu...@alumni.sjtu.edu.cn>
>> 收件人: "dev" <de...@flink.apache.org>
>> 发送时间: 星期五, 2023年 3 月 24日 上午 11:27:42
>> 主题: Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
>>
>> Thanks all for your feedback.
>>
>> @Shammon FY
>> My gut feeling is that the end user shouldn't care about whether it'll delete direcotry or move to Trash directory with the TRUNCATE TABLE statement. They only need to know it will delete all rows from a table.
>> To me, I think delete directory or move to trash is more likely to be a behavior of external storage level instead of SQL statement level. In Hive, if user configure Trash, it will then move files to trash for DROP statment.
>> Also, hardly did I see such usage with TRUNCATE TABLE statement in other engines. What's more, to support it, we have to extend the TRUNCATE TABLE synax which won't then compliant with SQL standard. I really don't want to do that and I believe it'll make user confused if we do so.
>>
>> @Hang
>> `TRUNCATE TABLE` is meant to delete all rows of a base table. So, it makes no sense that table source implements it.
>> If user use TRUNCATE TABLE statement to truncate a table, the planner will only try to
>> find the DynamicTableSink for the corresponding table.
>>
>> @Ran Tao
>> 1: Thanks for you reminder. I said it won't support view in the FLIP, but forget to said temporary table is also not supported. Now, I add this part to this FLIP.
>>
>> 2: Yes, I also considered to incldue it in this FLIP before. But as far as I see, I haven't seen much usage of truncate table with partition. It's not as useful as truncate table. So, I tend to keep this FLIP simple in here without supporting truncate table with partition.
>> Also, seems for `truncate table with partition`, differnet engines may have differernt syntax;
>> Hive[1]/Spark[2] use the following syntax:
>> TRUNCATE TABLE table_name [PARTITION partition_spec]
>>
>> SqlServer[3] use the follwoing syntax:
>> TRUNCATE TABLE { database_name.schema_name.table_name | schema_name.table_name | table_name } [ WITH ( PARTITIONS ( { <partition_number_expression> | <range> }
>> So, I'm tend to be cautious about it.
>>
>> But I'm open to this. If there's any feedback or strong requirement, I don't mind to add it in this FLIP.
>> If we do need it in some day, I can propose it in a new FLIP. It won't break the current design.
>>
>> As for concrete syntax in the FLIP, I think the current one is the concrete syntax, we don't allow TABLE keyword to be optional.
>>
>> 3: Thanks for your reminder, I have updadted the FLIP for this.
>>
>>
>> [1]https://cwiki.apache.org/confluence/display/hive/languagemanual+ddl#LanguageManualDDL-TruncateTable
>> [2]https://spark.apache.org/docs/3.0.0-preview/sql-ref-syntax-ddl-truncate-table.html
>> [3]https://learn.microsoft.com/en-us/sql/t-sql/statements/truncate-table-transact-sql?view=sql-server-ver16
>>
>>
>>
>> Best regards,
>> Yuxia
>>
>> ----- 原始邮件 -----
>> 发件人: "Ran Tao" <ch...@gmail.com>
>> 收件人: "dev" <de...@flink.apache.org>
>> 发送时间: 星期四, 2023年 3 月 23日 下午 6:28:17
>> 主题: Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
>>
>> Hi, yuxia.
>>
>> Thanks for starting the discussion.
>> I think it's a nice improvement to support TRUNCATE TABLE statement because
>> many other mature engines supports it.
>>
>> I have some questions.
>> 1. because table has different types, whether we will support view or
>> temporary tables?
>>
>> 2. some other engines such as spark and hive support TRUNCATE TABLE with
>> partition. whether we will support？
>> btw, i think you need give the TRUNCATE TABLE concrete syntax in the FLIP
>> because some engines has different syntaxes.
>> for example, hive allow TRUNCATE TABLE be TRUNCATE [TABLE] which means
>> TABLE keyword can be optional.
>>
>> 3. The Proposed Changes try to use SqlToOperationConverter and run in
>> TableEnvironmentImpl#executeInternal.
>> I think it's out of date, the community is refactoring the conversion logic
>> from SqlNode to operation[1] and executions in TableEnvironmentImpl[2].
>> I suggest you can use new way to support it.
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-31464
>> [2] https://issues.apache.org/jira/browse/FLINK-31368
>>
>> Best Regards,
>> Ran Tao
>> https://github.com/chucheng92
>>
>>
>> yuxia <lu...@alumni.sjtu.edu.cn> 于2023年3月22日周三 21:13写道：
>>
>>> Hi, devs.
>>>
>>> I'd like to start a discussion about FLIP-302: Support TRUNCATE TABLE
>>> statement [1].
>>>
>>> The TRUNCATE TABLE statement is a SQL command that allows users to quickly
>>> and efficiently delete all rows from a table without dropping the table
>>> itself. This statement is commonly used in data warehouse, where large data
>>> sets are frequently loaded and unloaded from tables.
>>> So, this FLIP is meant to support TRUNCATE TABLE statement. M ore exactly,
>>> this FLIP will bring Flink the TRUNCATE TABLE syntax and an interface with
>>> which the coresponding connectors can implement their own logic for
>>> truncating table.
>>>
>>> Looking forwards to your feedback.
>>>
>>> [1]: [
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement
>>> |
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement
>>> ]
>>>
>>>
>>> Best regards,
>>> Yuxia
>>>
Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
Posted by Jingsong Li <ji...@gmail.com>.
+1 for voting.
Best,
Jingsong
On Thu, Apr 6, 2023 at 4:52 PM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
>
> Hi everyone.
>
> If there are no other questions or concerns for the FLIP[1], I'd like to start the vote next Monday (4.10).
>
> [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement
>
> Best regards,
> Yuxia
>
> ----- 原始邮件 -----
> 发件人: "yuxia" <lu...@alumni.sjtu.edu.cn>
> 收件人: "dev" <de...@flink.apache.org>
> 发送时间: 星期五, 2023年 3 月 24日 上午 11:27:42
> 主题: Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
>
> Thanks all for your feedback.
>
> @Shammon FY
> My gut feeling is that the end user shouldn't care about whether it'll delete direcotry or move to Trash directory with the TRUNCATE TABLE statement. They only need to know it will delete all rows from a table.
> To me, I think delete directory or move to trash is more likely to be a behavior of external storage level instead of SQL statement level. In Hive, if user configure Trash, it will then move files to trash for DROP statment.
> Also, hardly did I see such usage with TRUNCATE TABLE statement in other engines. What's more, to support it, we have to extend the TRUNCATE TABLE synax which won't then compliant with SQL standard. I really don't want to do that and I believe it'll make user confused if we do so.
>
> @Hang
> `TRUNCATE TABLE` is meant to delete all rows of a base table. So, it makes no sense that table source implements it.
> If user use TRUNCATE TABLE statement to truncate a table, the planner will only try to
> find the DynamicTableSink for the corresponding table.
>
> @Ran Tao
> 1: Thanks for you reminder. I said it won't support view in the FLIP, but forget to said temporary table is also not supported. Now, I add this part to this FLIP.
>
> 2: Yes, I also considered to incldue it in this FLIP before. But as far as I see, I haven't seen much usage of truncate table with partition. It's not as useful as truncate table. So, I tend to keep this FLIP simple in here without supporting truncate table with partition.
> Also, seems for `truncate table with partition`, differnet engines may have differernt syntax;
> Hive[1]/Spark[2] use the following syntax:
> TRUNCATE TABLE table_name [PARTITION partition_spec]
>
> SqlServer[3] use the follwoing syntax:
> TRUNCATE TABLE { database_name.schema_name.table_name | schema_name.table_name | table_name } [ WITH ( PARTITIONS ( { <partition_number_expression> | <range> }
> So, I'm tend to be cautious about it.
>
> But I'm open to this. If there's any feedback or strong requirement, I don't mind to add it in this FLIP.
> If we do need it in some day, I can propose it in a new FLIP. It won't break the current design.
>
> As for concrete syntax in the FLIP, I think the current one is the concrete syntax, we don't allow TABLE keyword to be optional.
>
> 3: Thanks for your reminder, I have updadted the FLIP for this.
>
>
> [1]https://cwiki.apache.org/confluence/display/hive/languagemanual+ddl#LanguageManualDDL-TruncateTable
> [2]https://spark.apache.org/docs/3.0.0-preview/sql-ref-syntax-ddl-truncate-table.html
> [3]https://learn.microsoft.com/en-us/sql/t-sql/statements/truncate-table-transact-sql?view=sql-server-ver16
>
>
>
> Best regards,
> Yuxia
>
> ----- 原始邮件 -----
> 发件人: "Ran Tao" <ch...@gmail.com>
> 收件人: "dev" <de...@flink.apache.org>
> 发送时间: 星期四, 2023年 3 月 23日 下午 6:28:17
> 主题: Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement
>
> Hi, yuxia.
>
> Thanks for starting the discussion.
> I think it's a nice improvement to support TRUNCATE TABLE statement because
> many other mature engines supports it.
>
> I have some questions.
> 1. because table has different types, whether we will support view or
> temporary tables?
>
> 2. some other engines such as spark and hive support TRUNCATE TABLE with
> partition. whether we will support？
> btw, i think you need give the TRUNCATE TABLE concrete syntax in the FLIP
> because some engines has different syntaxes.
> for example, hive allow TRUNCATE TABLE be TRUNCATE [TABLE] which means
> TABLE keyword can be optional.
>
> 3. The Proposed Changes try to use SqlToOperationConverter and run in
> TableEnvironmentImpl#executeInternal.
> I think it's out of date, the community is refactoring the conversion logic
> from SqlNode to operation[1] and executions in TableEnvironmentImpl[2].
> I suggest you can use new way to support it.
>
> [1] https://issues.apache.org/jira/browse/FLINK-31464
> [2] https://issues.apache.org/jira/browse/FLINK-31368
>
> Best Regards,
> Ran Tao
> https://github.com/chucheng92
>
>
> yuxia <lu...@alumni.sjtu.edu.cn> 于2023年3月22日周三 21:13写道：
>
> > Hi, devs.
> >
> > I'd like to start a discussion about FLIP-302: Support TRUNCATE TABLE
> > statement [1].
> >
> > The TRUNCATE TABLE statement is a SQL command that allows users to quickly
> > and efficiently delete all rows from a table without dropping the table
> > itself. This statement is commonly used in data warehouse, where large data
> > sets are frequently loaded and unloaded from tables.
> > So, this FLIP is meant to support TRUNCATE TABLE statement. M ore exactly,
> > this FLIP will bring Flink the TRUNCATE TABLE syntax and an interface with
> > which the coresponding connectors can implement their own logic for
> > truncating table.
> >
> > Looking forwards to your feedback.
> >
> > [1]: [
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement
> > |
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement
> > ]
> >
> >
> > Best regards,
> > Yuxia
> >