You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "aokolnychyi (via GitHub)" <gi...@apache.org> on 2023/09/20 01:40:17 UTC

[GitHub] [iceberg] aokolnychyi opened a new pull request, #8594: Spark 3.5: Add distibuted planning benchmarks

aokolnychyi opened a new pull request, #8594:
URL: https://github.com/apache/iceberg/pull/8594

   This PR adds local benchmarks for distributed planning. Will post results shortly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on pull request #8594: Spark 3.5: Add distibuted planning benchmarks

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #8594:
URL: https://github.com/apache/iceberg/pull/8594#issuecomment-1728497362

   SGTM, Just want to make sure we don't have this as the only gatekeeping on perf improvements to Dist Planning


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on pull request #8594: Spark 3.5: Add distibuted planning benchmarks

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #8594:
URL: https://github.com/apache/iceberg/pull/8594#issuecomment-1728461956

   If these only test the overhead in a local context, are we worried there may be a time where performance in a local context would become worse with a code change but performance in a distributed context would be better?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on pull request #8594: Spark 3.5: Add distibuted planning benchmarks

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #8594:
URL: https://github.com/apache/iceberg/pull/8594#issuecomment-1728488297

   @RussellSpitzer, these benchmarks can be used to profile local planning too. I was just explaining why distributed planning can't be faster and why the numbers look worse.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on pull request #8594: Spark 3.5: Add distibuted planning benchmarks

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #8594:
URL: https://github.com/apache/iceberg/pull/8594#issuecomment-1728456923

   cc @amogh-jahagirdar @flyrain @RussellSpitzer 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on pull request #8594: Spark 3.5: Add distibuted planning benchmarks

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #8594:
URL: https://github.com/apache/iceberg/pull/8594#issuecomment-1728513947

   @RussellSpitzer, 100%. The biggest issue is usually the speed with which we can fetch the result from other nodes. That can only be tested on a real cluster.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on pull request #8594: Spark 3.5: Add distibuted planning benchmarks

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #8594:
URL: https://github.com/apache/iceberg/pull/8594#issuecomment-1728495414

   @RussellSpitzer, got you. Yeah, we can't benchmark the distributed planning locally for sure. However, it is still beneficial for local profiling. It helped me quite a bit to reduce the overhead. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on pull request #8594: Spark 3.5: Add distibuted planning benchmarks

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #8594:
URL: https://github.com/apache/iceberg/pull/8594#issuecomment-1728489956

   > @RussellSpitzer, these benchmarks can be used to profile local planning too. I was just explaining why distributed planning can't be faster and why the numbers look worse.
   
   Yeah I understand that I was thinking more of the context of comparing changes to Distributed Planning. If Version A gets a score of 10 and Version B gets a score of 9 does that mean something to us if a distributed benchmark of A scores 5 and a distributed benchmark of b scored 15


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on pull request #8594: Spark 3.5: Add distibuted planning benchmarks

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #8594:
URL: https://github.com/apache/iceberg/pull/8594#issuecomment-1728493598

   > Yeah I understand that I was thinking more of the context of comparing changes to Distributed Planning. If Version A gets a score of 10 and Version B gets a score of 9 does that mean something to us if a distributed benchmark of A scores 5 and a distributed benchmark of b scored 15
   
   Probably? However, I would imagine comparing before/after value of the same benchmark to be the primary metric.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi merged pull request #8594: Spark 3.5: Add distibuted planning benchmarks

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi merged PR #8594:
URL: https://github.com/apache/iceberg/pull/8594


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org