[GitHub] [shardingsphere] sandynz commented on pull request #14978: Simplify MemoryPipelineChannel impl to resolve scaling OOME on large data set

[GitBox <gi...@apache.org>]

sandynz commented on pull request #14978:
URL: https://github.com/apache/shardingsphere/pull/14978#issuecomment-1018253837


   It's found in a simple performance test.
   Table `sbtest1` with 20 millions lines of records that's generated by sysbench.
   Scaling job ran on 4G heap JVM.
   It throw exception on inventory dump:
   ```
   [ERROR] 2022-01-21 11:58:23.078 [ShardingSphere-Scaling-execute-4] o.a.s.d.p.core.task.InventoryTask - get an error when migrating the inventory data
   java.lang.OutOfMemoryError: GC overhead limit exceeded
   	at java.util.LinkedHashMap.newNode(LinkedHashMap.java:256)
   	at java.util.HashMap.putVal(HashMap.java:631)
   	at java.util.HashMap.put(HashMap.java:612)
   	at org.apache.shardingsphere.driver.executor.batch.BatchExecutionUnit.mapAddBatchCount(BatchExecutionUnit.java:56)
   	at org.apache.shardingsphere.driver.executor.batch.BatchPreparedStatementExecutor.reviseBatchExecutionUnit(BatchPreparedStatementExecutor.java:118)
   ```
   
   Most memory was consumed by `toBeAckRecords` which is instanceof `ConcurrentLinkedDeque` in `AbstractBitSetChannel`. It's remove for now, `AckCallback` is enough for now.
   
   On the latest code, it works, test result:
   ```
   mysql> show scaling list;
   +-----------------+---------+----------------------+--------+---------------------+---------------------+
   | id              | tables  | sharding_total_count | active | create_time         | stop_time           |
   +-----------------+---------+----------------------+--------+---------------------+---------------------+
   | 223261682468932 | sbtest1 | 1                    | false  | 2022-01-21 14:51:00 | 2022-01-21 15:04:45 |
   +-----------------+---------+----------------------+--------+---------------------+---------------------+
   1 row in set (0.01 sec)
   
   mysql> show scaling status 223261682468932;
   +------+-------------+----------+--------+-------------------------------+--------------------------+
   | item | data_source | status   | active | inventory_finished_percentage | incremental_idle_minutes |
   +------+-------------+----------+--------+-------------------------------+--------------------------+
   | 0    | ds_0        | FINISHED | false  | 100                           | 4                        |
   +------+-------------+----------+--------+-------------------------------+--------------------------+
   1 row in set (0.01 sec)
   
   mysql> check scaling 223261682468932 by type (name=CRC32_MATCH);
   +------------+----------------------+----------------------+-----------------------+-------------------------+
   | table_name | source_records_count | target_records_count | records_count_matched | records_content_matched |
   +------------+----------------------+----------------------+-----------------------+-------------------------+
   | sbtest1    | 22762887             | 22762887             | true                  | true                    |
   +------------+----------------------+----------------------+-----------------------+-------------------------+
   1 row in set (1 min 47.23 sec)
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org