[I] [Bug] OOM when flink query sql insert into a table from another in streaming mode [incubator-paimon]

["zvand (via GitHub)" <gi...@apache.org>]

zvand opened a new issue, #2684:
URL: https://github.com/apache/incubator-paimon/issues/2684

   ### Search before asking
   
   - [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
   
   
   ### Paimon version
   
   0.6.0
   
   ### Compute Engine
   
   flink 1.17.1
   
   ### Minimal reproduce step
   
   ```sql
   CREATE TABLE db_issue.ods_order_history
   (
       field1 INT,
       field2 STRING,
       field3 STRING,
       field4 STRING,
       field5 STRING,
       field6 STRING,
       field7 STRING,
       field8 STRING,
       field8 INT,
       field10 INT,
       field11 STRING,
       field12 STRING,
       field13 STRING,
       field14 STRING,
       field15 STRING,
       field16 STRING,
       field17 STRING,
       field18 STRING,
       field19 BIGINT,
       payment_time     TIMESTAMP comment 'payment finish time',
       data_time        TIMESTAMP comment 'data report time',
       PRIMARY KEY (field1, field2, field3, field4, field11, field5, field7, field16) NOT ENFORCED
   ) WITH ('bucket'='8')
   ;
   
   CREATE TABLE db_issue.ods_order_daily
   (
       field1 INT,
       field2 STRING,
       field3 STRING,
       field4 STRING,
       field5 STRING,
       field6 STRING,
       field7 STRING,
       field8 STRING,
       field8 INT,
       field10 INT,
       field11 STRING,
       field12 STRING,
       field13 STRING,
       field14 STRING,
       field15 STRING,
       field16 STRING,
       field17 STRING,
       field18 STRING,
       field19 BIGINT,
       payment_time     TIMESTAMP comment 'payment finish time',
       payment_date     DATE comment 'payment finish date',
       data_time        TIMESTAMP comment 'data report time',
       PRIMARY KEY (payment_date, field1, field2, field3, field4, field11, field5, field7, field16) NOT ENFORCED
   )
   PARTITIONED BY (payment_date)
   ;
   
   set parallelism.default=8;
   
   INSERT INTO ods_order_daily
   SELECT
       field1,
       field2,
       field3,
       field4,
       field5,
       field6,
       field7,
       field8,
       field8,
       field10,
       field11,
       field12,
       field13,
       field14,
       field15,
       field16,
       field17,
       field18,
       field19,
       payment_time,
       TO_DATE(DATE_FORMAT(payment_time, 'yyyy-MM-dd')) AS payment_date,
       data_time
   FROM ods_order_history;
   ```
   
   ### What doesn't meet your expectations?
   
   The ods_order_history table has about 7,000,000 rows, when i run the last sql to insert into ods_order_daily in streaming mode, it cause OOM and restart in loop, but when i set the query in batch mode, i work well
   here is the exception logs
   <img width="1391" alt="image" src="https://github.com/apache/incubator-paimon/assets/78157666/57d75d15-1671-478d-ac17-1151501ecd70">
   
   
   ### Anything else?
   
   None
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org