You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zeruibao (via GitHub)" <gi...@apache.org> on 2023/10/31 18:45:54 UTC

[PR] [SPARK-43380][SQL][Follow-up] Fix slowdown in Avro read [spark]

zeruibao opened a new pull request, #43606:
URL: https://github.com/apache/spark/pull/43606

   ### What changes were proposed in this pull request?
   Fix slowdown in Avro read. There is a https://github.com/apache/spark/pull/42503 that causes the performance regression. It seems that `SQLConf.get.getConf(confKey)` is very costly. Move it out of `newWriter` function.
   
   ### Why are the changes needed?
   Need to fix the performance regression of Avro read.
   
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   
   ### How was this patch tested?
   Existing UT test
   
   ### Was this patch authored or co-authored using generative AI tooling?
   No
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-43380][SQL][Follow-up] Fix slowdown in Avro read [spark]

Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang closed pull request #43606: [SPARK-43380][SQL][Follow-up] Fix slowdown in Avro read
URL: https://github.com/apache/spark/pull/43606


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-43380][SQL][Follow-up] Fix slowdown in Avro read [spark]

Posted by "zeruibao (via GitHub)" <gi...@apache.org>.
zeruibao commented on PR #43606:
URL: https://github.com/apache/spark/pull/43606#issuecomment-1788111440

   Hi @gengliangwang and @dongjoon-hyun, all avro related tests have passed. Should be good to merge now haha.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-43380][SQL][Follow-up] Fix slowdown in Avro read [spark]

Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #43606:
URL: https://github.com/apache/spark/pull/43606#issuecomment-1788184967

   Thanks,merging to master/branch-3.5


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org