You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "BsoBird (via GitHub)" <gi...@apache.org> on 2023/04/25 17:16:58 UTC
[GitHub] [iceberg] BsoBird opened a new issue, #7431: iceberg mor table execute merge very very slow
BsoBird opened a new issue, #7431:
URL: https://github.com/apache/iceberg/issues/7431
### Query engine
iceberg 1.2.1
spark 3.3.2
### Question
I created two tables with the same amount of data, one MOR table and one COW table, both with nearly 400 million data and a data size of 200G.
I MERGE INTO 100 million data to each table.
COW table took 9 minutes to complete the execution.
MOR table is currently executing close to 30 minutes has not yet finished.
Why is the MOR table so slow to execute MERGE?
Here is my execution SQL:
```
CREATE TABLE IF NOT EXISTS local.test.b_std_trade_4_iceberg_orc_zstd ( `uni_order_id` string, `data_from` bigint, `partner` string, `plat_code` string, `order_id` string, `uni_shop_id` string, `uni_id` string, `guide_id` string, `shop_id` string, `plat_account` string, `total_fee` double, `item_discount_fee` double, `trade_discount_fee` double, `adjust_fee` double, `post_fee` double, `discount_rate` double, `payment_no_postfee` double, `payment` double, `pay_time` string, `product_num` bigint, `order_status` string, `is_refund` string, `refund_fee` double, `insert_time` string, `created` string, `endtime` string, `modified` string, `trade_type` string, `receiver_name` string, `receiver_country` string, `receiver_state` string, `receiver_city` string, `receiver_district` string, `receiver_town` string, `receiver_address` string, `receiver_mobile` string, `trade_source` string, `delivery_type` string, `consign_time` string, `orders_num` bigint, `is_presale` bigint, `presale_status`
string, `first_fee_paytime` string, `last_fee_paytime` string, `first_paid_fee` double, `tenant` string, `tidb_modified` string, `step_paid_fee` double, `seller_flag` string, `is_used_store_card` BIGINT, `store_card_used` DOUBLE, `store_card_basic_used` DOUBLE, `store_card_expand_used` DOUBLE, `order_promotion_num` BIGINT, `item_promotion_num` BIGINT, `buyer_remark` string, `seller_remark` string, trade_business_type string )TBLPROPERTIES ('read.orc.vectorization.enabled'='true','write.format.default'='orc','write.orc.bloom.filter.columns'='tenant,shop_id,plat_code,order_status,payment,pay_time,created','kyuubi.zorder.enabled'='true','kyuubi.zorder.cols'='tenant,shop_id,plat_code,order_status,payment,pay_time,created','write.orc.compression-codec'='zstd') STORED AS iceberg;
merge into local.test.b_std_trade_4_iceberg_orc_zstd t using( select `uni_order_id`, `data_from`, `partner`, `plat_code`, `order_id`, `uni_shop_id`, `uni_id`, `guide_id`, `shop_id`, `plat_account`, `total_fee`, `item_discount_fee`, `trade_discount_fee`, `adjust_fee`, `post_fee`, `discount_rate`, `payment_no_postfee`, `payment`, `pay_time`, `product_num`, `order_status`, `is_refund`, `refund_fee`, `insert_time`, `created`, `endtime`, `modified`, `trade_type`, `receiver_name`, `receiver_country`, `receiver_state`, `receiver_city`, `receiver_district`, `receiver_town`, `receiver_address`, `receiver_mobile`, `trade_source`, `delivery_type`, `consign_time`, `orders_num`, `is_presale`, `presale_status`, `first_fee_paytime`, `last_fee_paytime`, `first_paid_fee`, `tenant`, `tidb_modified`, `step_paid_fee`, `seller_flag`, `is_used_store_card`, `store_card_used`, `store_card_basic_used`, `store_card_expand_used`, `order_promotion_num`, `item_promotion_num`, `buyer_remark`, `seller_remark`,
trade_business_type from ( select `uni_order_id`, `data_from`, `partner`, `plat_code`, `order_id`, `uni_shop_id`, `uni_id`, `guide_id`, `shop_id`, `plat_account`, `total_fee`, `item_discount_fee`, `trade_discount_fee`, `adjust_fee`, `post_fee`, `discount_rate`, `payment_no_postfee`, `payment`, `pay_time`, `product_num`, `order_status`, `is_refund`, `refund_fee`, `insert_time`, `created`, `endtime`, `modified`, `trade_type`, `receiver_name`, `receiver_country`, `receiver_state`, `receiver_city`, `receiver_district`, `receiver_town`, `receiver_address`, `receiver_mobile`, `trade_source`, `delivery_type`, `consign_time`, `orders_num`, `is_presale`, `presale_status`, `first_fee_paytime`, `last_fee_paytime`, `first_paid_fee`, `tenant`, `tidb_modified`, `step_paid_fee`, `seller_flag`, `is_used_store_card`, `store_card_used`, `store_card_basic_used`, `store_card_expand_used`, `order_promotion_num`, `item_promotion_num`, `buyer_remark`, `seller_remark`, trade_business_type,row_number() over
(partition by uni_order_id order by modified desc,dt desc) as rank from spark_catalog.dw_source.s_std_trade_tidb where dt>='2023-04-20' ) small where rank=1) s ON t.uni_order_id = s.uni_order_id and t.tenant = s.tenant and t.partner = s.partner WHEN MATCHED AND s.modified>=t.modified then UPDATE SET t.uni_order_id = s.uni_order_id, t.data_from = s.data_from, t.partner = s.partner, t.plat_code = s.plat_code, t.order_id = s.order_id, t.uni_shop_id = s.uni_shop_id, t.uni_id = s.uni_id, t.guide_id = s.guide_id, t.shop_id = s.shop_id, t.plat_account = s.plat_account, t.total_fee = s.total_fee, t.item_discount_fee = s.item_discount_fee, t.trade_discount_fee = s.trade_discount_fee, t.adjust_fee = s.adjust_fee, t.post_fee = s.post_fee, t.discount_rate = s.discount_rate, t.payment_no_postfee = s.payment_no_postfee, t.payment = s.payment, t.pay_time = s.pay_time, t.product_num = s.product_num, t.order_status = s.order_status, t.is_refund = s.is_refund, t.refund_fee = s.refund_fee, t.insert_t
ime = s.insert_time, t.created = s.created, t.endtime = s.endtime, t.modified = s.modified, t.trade_type = s.trade_type, t.receiver_name = s.receiver_name, t.receiver_country = s.receiver_country, t.receiver_state = s.receiver_state, t.receiver_city = s.receiver_city, t.receiver_district = s.receiver_district, t.receiver_town = s.receiver_town, t.receiver_address = s.receiver_address, t.receiver_mobile = s.receiver_mobile, t.trade_source = s.trade_source, t.delivery_type = s.delivery_type, t.consign_time = s.consign_time, t.orders_num = s.orders_num, t.is_presale = s.is_presale, t.presale_status = s.presale_status, t.first_fee_paytime = s.first_fee_paytime, t.last_fee_paytime = s.last_fee_paytime, t.first_paid_fee = s.first_paid_fee, t.tenant = s.tenant, t.tidb_modified = s.tidb_modified, t.step_paid_fee = s.step_paid_fee, t.seller_flag = s.seller_flag, t.is_used_store_card = s.is_used_store_card, t.store_card_used = s.store_card_used, t.store_card_basic_used = s.store_card_basic_us
ed, t.store_card_expand_used = s.store_card_expand_used, t.order_promotion_num = s.order_promotion_num, t.item_promotion_num = s.item_promotion_num, t.buyer_remark = s.buyer_remark, t.seller_remark = s.seller_remark, t.trade_business_type = s.trade_business_type WHEN NOT MATCHED THEN INSERT * ;
--9 minutes to execute
CREATE TABLE IF NOT EXISTS local.test.b_std_trade_4_iceberg_orc_zstd_mor ( `uni_order_id` string, `data_from` bigint, `partner` string, `plat_code` string, `order_id` string, `uni_shop_id` string, `uni_id` string, `guide_id` string, `shop_id` string, `plat_account` string, `total_fee` double, `item_discount_fee` double, `trade_discount_fee` double, `adjust_fee` double, `post_fee` double, `discount_rate` double, `payment_no_postfee` double, `payment` double, `pay_time` string, `product_num` bigint, `order_status` string, `is_refund` string, `refund_fee` double, `insert_time` string, `created` string, `endtime` string, `modified` string, `trade_type` string, `receiver_name` string, `receiver_country` string, `receiver_state` string, `receiver_city` string, `receiver_district` string, `receiver_town` string, `receiver_address` string, `receiver_mobile` string, `trade_source` string, `delivery_type` string, `consign_time` string, `orders_num` bigint, `is_presale` bigint, `presale_stat
us` string, `first_fee_paytime` string, `last_fee_paytime` string, `first_paid_fee` double, `tenant` string, `tidb_modified` string, `step_paid_fee` double, `seller_flag` string, `is_used_store_card` BIGINT, `store_card_used` DOUBLE, `store_card_basic_used` DOUBLE, `store_card_expand_used` DOUBLE, `order_promotion_num` BIGINT, `item_promotion_num` BIGINT, `buyer_remark` string, `seller_remark` string, trade_business_type string )TBLPROPERTIES ('read.orc.vectorization.enabled'='true','write.format.default'='orc','write.orc.bloom.filter.columns'='tenant,shop_id,plat_code,order_status,payment,pay_time,created','kyuubi.zorder.enabled'='true','kyuubi.zorder.cols'='tenant,shop_id,plat_code,order_status,payment,pay_time,created','write.orc.compression-codec'='zstd','write.delete.mode'='merge-on-read','write.update.mode'='merge-on-read','write.merge.mode'='merge-on-read') STORED AS iceberg;
merge into local.test.b_std_trade_4_iceberg_orc_zstd_mor t using( select `uni_order_id`, `data_from`, `partner`, `plat_code`, `order_id`, `uni_shop_id`, `uni_id`, `guide_id`, `shop_id`, `plat_account`, `total_fee`, `item_discount_fee`, `trade_discount_fee`, `adjust_fee`, `post_fee`, `discount_rate`, `payment_no_postfee`, `payment`, `pay_time`, `product_num`, `order_status`, `is_refund`, `refund_fee`, `insert_time`, `created`, `endtime`, `modified`, `trade_type`, `receiver_name`, `receiver_country`, `receiver_state`, `receiver_city`, `receiver_district`, `receiver_town`, `receiver_address`, `receiver_mobile`, `trade_source`, `delivery_type`, `consign_time`, `orders_num`, `is_presale`, `presale_status`, `first_fee_paytime`, `last_fee_paytime`, `first_paid_fee`, `tenant`, `tidb_modified`, `step_paid_fee`, `seller_flag`, `is_used_store_card`, `store_card_used`, `store_card_basic_used`, `store_card_expand_used`, `order_promotion_num`, `item_promotion_num`, `buyer_remark`, `seller_remar
k`, trade_business_type from ( select `uni_order_id`, `data_from`, `partner`, `plat_code`, `order_id`, `uni_shop_id`, `uni_id`, `guide_id`, `shop_id`, `plat_account`, `total_fee`, `item_discount_fee`, `trade_discount_fee`, `adjust_fee`, `post_fee`, `discount_rate`, `payment_no_postfee`, `payment`, `pay_time`, `product_num`, `order_status`, `is_refund`, `refund_fee`, `insert_time`, `created`, `endtime`, `modified`, `trade_type`, `receiver_name`, `receiver_country`, `receiver_state`, `receiver_city`, `receiver_district`, `receiver_town`, `receiver_address`, `receiver_mobile`, `trade_source`, `delivery_type`, `consign_time`, `orders_num`, `is_presale`, `presale_status`, `first_fee_paytime`, `last_fee_paytime`, `first_paid_fee`, `tenant`, `tidb_modified`, `step_paid_fee`, `seller_flag`, `is_used_store_card`, `store_card_used`, `store_card_basic_used`, `store_card_expand_used`, `order_promotion_num`, `item_promotion_num`, `buyer_remark`, `seller_remark`, trade_business_type,row_number()
over(partition by uni_order_id order by modified desc,dt desc) as rank from spark_catalog.dw_source.s_std_trade_tidb where dt>='2023-04-20' ) small where rank=1) s ON t.uni_order_id = s.uni_order_id and t.tenant = s.tenant and t.partner = s.partner WHEN MATCHED AND s.modified>=t.modified then UPDATE SET t.uni_order_id = s.uni_order_id, t.data_from = s.data_from, t.partner = s.partner, t.plat_code = s.plat_code, t.order_id = s.order_id, t.uni_shop_id = s.uni_shop_id, t.uni_id = s.uni_id, t.guide_id = s.guide_id, t.shop_id = s.shop_id, t.plat_account = s.plat_account, t.total_fee = s.total_fee, t.item_discount_fee = s.item_discount_fee, t.trade_discount_fee = s.trade_discount_fee, t.adjust_fee = s.adjust_fee, t.post_fee = s.post_fee, t.discount_rate = s.discount_rate, t.payment_no_postfee = s.payment_no_postfee, t.payment = s.payment, t.pay_time = s.pay_time, t.product_num = s.product_num, t.order_status = s.order_status, t.is_refund = s.is_refund, t.refund_fee = s.refund_fee, t.inse
rt_time = s.insert_time, t.created = s.created, t.endtime = s.endtime, t.modified = s.modified, t.trade_type = s.trade_type, t.receiver_name = s.receiver_name, t.receiver_country = s.receiver_country, t.receiver_state = s.receiver_state, t.receiver_city = s.receiver_city, t.receiver_district = s.receiver_district, t.receiver_town = s.receiver_town, t.receiver_address = s.receiver_address, t.receiver_mobile = s.receiver_mobile, t.trade_source = s.trade_source, t.delivery_type = s.delivery_type, t.consign_time = s.consign_time, t.orders_num = s.orders_num, t.is_presale = s.is_presale, t.presale_status = s.presale_status, t.first_fee_paytime = s.first_fee_paytime, t.last_fee_paytime = s.last_fee_paytime, t.first_paid_fee = s.first_paid_fee, t.tenant = s.tenant, t.tidb_modified = s.tidb_modified, t.step_paid_fee = s.step_paid_fee, t.seller_flag = s.seller_flag, t.is_used_store_card = s.is_used_store_card, t.store_card_used = s.store_card_used, t.store_card_basic_used = s.store_card_basi
c_used, t.store_card_expand_used = s.store_card_expand_used, t.order_promotion_num = s.order_promotion_num, t.item_promotion_num = s.item_promotion_num, t.buyer_remark = s.buyer_remark, t.seller_remark = s.seller_remark, t.trade_business_type = s.trade_business_type WHEN NOT MATCHED THEN INSERT * ;
--30 minutes and still not executed
```
This is a screenshot of the current execution status of the MOR table:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on issue #7431: iceberg mor table execute merge very very slow
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on issue #7431:
URL: https://github.com/apache/iceberg/issues/7431#issuecomment-1545209242
I think the COW join stages does not have the detail you have of MOR stages, if you can send the detail of those stages, that'll be nice.
But I'm thinknig one hypothesis I'll check tomorrow with @RussellSpitzer and @aokolnychyi , is perhaps there is skew in both COW and MOR. But in COW we skew only in the first join when calculating the file set, and its smaller as we select much less data in this calculation. The second join benefits from knowing the exact file set, so only read what is necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] chenwyi2 commented on issue #7431: iceberg mor table execute merge very very slow
Posted by "chenwyi2 (via GitHub)" <gi...@apache.org>.
chenwyi2 commented on issue #7431:
URL: https://github.com/apache/iceberg/issues/7431#issuecomment-1644917581
write.orc.bloom.filter.columns can take effect on spark 3.1? when i set write.orc.bloom.filter.columns=xx, wrote data by spark and query table by xx field, but i can not find any query improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] iceberg mor table execute merge very very slow [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7431:
URL: https://github.com/apache/iceberg/issues/7431#issuecomment-2316467485
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] BsoBird commented on issue #7431: iceberg mor table execute merge very very slow
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on issue #7431:
URL: https://github.com/apache/iceberg/issues/7431#issuecomment-1522792056
@singhpk234
Am I right in thinking that MOR tables are not currently available on a large scale? After all, MOR tables take too long to execute. Also do we have a solution or plan to fix this problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] BsoBird commented on issue #7431: iceberg mor table execute merge very very slow
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on issue #7431:
URL: https://github.com/apache/iceberg/issues/7431#issuecomment-1522158235
Can anyone help guide me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] BsoBird commented on issue #7431: iceberg mor table execute merge very very slow
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on issue #7431:
URL: https://github.com/apache/iceberg/issues/7431#issuecomment-1543083207
@szehon-ho
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] BsoBird commented on issue #7431: iceberg mor table execute merge very very slow
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on issue #7431:
URL: https://github.com/apache/iceberg/issues/7431#issuecomment-1569430392
@szehon-ho
I re-tested ICEBERG 1.3.0 on SPARK 3.3. but the problem is still not solved. Is this problem solved in SPARK3.4?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] BsoBird commented on issue #7431: iceberg mor table execute merge very very slow
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on issue #7431:
URL: https://github.com/apache/iceberg/issues/7431#issuecomment-1645123416
@chenwyi2
bloom filter 的作用可能没有想象中那么大. 当基础数据集很大的时候,bloom filter就会出现假阳性的问题.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] BsoBird commented on issue #7431: iceberg mor table execute merge very very slow
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on issue #7431:
URL: https://github.com/apache/iceberg/issues/7431#issuecomment-1550709396
@szehon-ho
Sounds good, I'll test it first after v1.3 is released
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] iceberg mor table execute merge very very slow [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7431:
URL: https://github.com/apache/iceberg/issues/7431#issuecomment-2345010051
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] singhpk234 commented on issue #7431: iceberg mor table execute merge very very slow
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on issue #7431:
URL: https://github.com/apache/iceberg/issues/7431#issuecomment-1522173428
`RowLevelCommandDynamicPruning` is not supported in MOR :
https://github.com/apache/iceberg/blob/50ca63bde82547c42475591455c00a429c854d4b/spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/RowLevelCommandDynamicPruning.scala#L66-L68
hence there is a difference in performance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] BsoBird commented on issue #7431: iceberg mor table execute merge very very slow
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on issue #7431:
URL: https://github.com/apache/iceberg/issues/7431#issuecomment-1535686642
@aokolnychyi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on issue #7431: iceberg mor table execute merge very very slow
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on issue #7431:
URL: https://github.com/apache/iceberg/issues/7431#issuecomment-1545199924
Thanks. Downloaded your files from: [link](https://data-plat.oss-cn-zhangjiakou.aliyuncs.com/shared/MOR%20VS%20COW.zip?OSSAccessKeyId=LTAI4GGcFCPKf6YSoejXNG29&Expires=1683942120&Signature=keJS303TjgG7oZVrVue140U%2Fork%3D)
Looks like there is terrible skew in the MOR join.
Can you check the COW joins and see the details? Is there no comparable skew?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] BsoBird commented on issue #7431: iceberg mor table execute merge very very slow
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on issue #7431:
URL: https://github.com/apache/iceberg/issues/7431#issuecomment-1544213677
@szehon-ho
Hello. I have prepared two datasets with the same amount of data and conducted experiments in the same environment.
Here are the results of the experiments I collected.
From the current point of view, there is a skew job in the processing of the MOR table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on issue #7431: iceberg mor table execute merge very very slow
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on issue #7431:
URL: https://github.com/apache/iceberg/issues/7431#issuecomment-1550413854
Missed your earlier comment as well, https://github.com/apache/iceberg/pull/7520 seems it should help as well. Is it possible to re-test after these two are in?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] iceberg mor table execute merge very very slow [iceberg]
Posted by "wimlewis-amazon (via GitHub)" <gi...@apache.org>.
wimlewis-amazon commented on issue #7431:
URL: https://github.com/apache/iceberg/issues/7431#issuecomment-2471204881
I'm having what looks like the same or similar problem. 1.6.1 seems a little bit faster than 1.0.0 or 1.4.x, but still incredibly slow — far too slow for MOR to be a usable mode in this situation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] iceberg mor table execute merge very very slow [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #7431: iceberg mor table execute merge very very slow
URL: https://github.com/apache/iceberg/issues/7431
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] BsoBird commented on issue #7431: iceberg mor table execute merge very very slow
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on issue #7431:
URL: https://github.com/apache/iceberg/issues/7431#issuecomment-1542339473
@szehon-ho
hi. I uploaded the image.
When I ran the SQL this time, there were some third-party tasks in the cluster that were competing with me for resources, so my SQL execution took a little longer this time. I hope this factor does not affect your analysis too much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on issue #7431: iceberg mor table execute merge very very slow
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on issue #7431:
URL: https://github.com/apache/iceberg/issues/7431#issuecomment-1546395461
Actually I noticed I may be wrong, the screenshot I posted is the last stage (MergeInto), its not Join as I thought. So it may be related to @aokolnychyi fix : https://github.com/apache/iceberg/pull/7558 , and less to do with runtimeFiltering.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on issue #7431: iceberg mor table execute merge very very slow
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on issue #7431:
URL: https://github.com/apache/iceberg/issues/7431#issuecomment-1536598262
It looks like what folks are mentioning here, the noticeable difference in plan is
```
: +- BatchScan[uni_order_id#3586, data_from#3587L, partner#3588, plat_code#3589, order_id#3590, uni_shop_id#3591, uni_id#3592, guide_id#3593, shop_id#3594, plat_account#3595, total_fee#3596, item_discount_fee#3597, trade_discount_fee#3598, adjust_fee#3599, post_fee#3600, discount_rate#3601, payment_no_postfee#3602, payment#3603, pay_time#3604, product_num#3605L, order_status#3606, is_refund#3607, refund_fee#3608, insert_time#3609, ... 35 more fields] local.test.b_std_trade_4_iceberg_orc_zstd (branch=null) [filters=, groupedBy=] RuntimeFilters: [dynamicpruningexpression(_file#3706 IN dynamicpruning#4007)]
: +- SubqueryAdaptiveBroadcast dynamicpruning#4007, 0, false, Project [_file#4006], [_file#4006]
```
Looks like pushed to test.b_std_trade_4_iceberg_orc_zstd in COW only
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on issue #7431: iceberg mor table execute merge very very slow
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on issue #7431:
URL: https://github.com/apache/iceberg/issues/7431#issuecomment-1535156996
There are a few pending changes I am yet to submit to OSS regarding MoR. Runtime filtering is one of the potential causes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on issue #7431: iceberg mor table execute merge very very slow
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on issue #7431:
URL: https://github.com/apache/iceberg/issues/7431#issuecomment-1542897084
Hi, thanks. Yea I am comparing the two stage trees, nothing immediately jumps out to me to say why MOR is 40 mins and the other is 17 mins.
Comparing the two joins,
* MOR join has shuffle bytes: 207.6 GB and 50.3 GB.
* COW has two joins (first for determining list of files, second is the actual join with filter on the file list)
* 1st join has shuflfe bytes: 10.3G and 7.0 G (file projection calcluation)
* 2nd join has 156.9 G and 109.3 G (final join after file filter)
So just based on that, I dont see huge difference here.
Also I notice that the data for the two runs may be different, you have 785 million and 499 million rows for COW, and then 261 million and 498 million for MOR.
I am not a huge expert on Spark UI, but is there some where you can see how long each stage takes? Hope you dont have to re-run both jobs but can just get it from the History.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] BsoBird commented on issue #7431: iceberg mor table execute merge very very slow
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on issue #7431:
URL: https://github.com/apache/iceberg/issues/7431#issuecomment-1545206673
@szehon-ho
The file I provided you with contains the execution details of the COW table. Do you need any additional information from me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on issue #7431: iceberg mor table execute merge very very slow
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on issue #7431:
URL: https://github.com/apache/iceberg/issues/7431#issuecomment-1539191389
If you have time, do you have the Spark History UI of the runtimes of how much time it spends in each stage, in both versions? I see a running time for MOR but would be great to compare to COW as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org