Re: [I] Add encoding + compression metrics to columnar shuffle [datafusion-comet]

["0lai0 (via GitHub)" <gi...@apache.org>]

0lai0 commented on issue #1212:
URL: https://github.com/apache/datafusion-comet/issues/1212#issuecomment-4303487371

   Hi @andygrove , I would like to confirm that I am reproducing issue #1212 correctly.
   
   When I run with `spark.comet.shuffle.mode=jvm`, I get CometColumnarExchange, and the metric shows:
   encoding and compression time: 0 ms (single value) and no total (min, med, max) distribution for that metric.
   
   When I switch to `spark.comet.shuffle.mode=native`, I get CometExchange, and encoding and compression time total (min, med, max) has non-zero values.
   
    Is this is behavior that #1212 is intended to address?
   
   `spark.comet.shuffle.mode=jvm`
   <img width="455" height="699" alt="Image" src="https://github.com/user-attachments/assets/aea024e1-0325-4135-832c-0f95424c6901" />
   `spark.comet.shuffle.mode=native`
   <img width="439" height="758" alt="Image" src="https://github.com/user-attachments/assets/3a548f9d-204d-41ee-818b-734616192990" />


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org