You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/11/30 09:35:58 UTC
[GitHub] [iceberg] rbalamohan opened a new issue, #6320: ArrowBuf boundary checks causing CPU burn and slowness in vectorized parq reading
rbalamohan opened a new issue, #6320:
URL: https://github.com/apache/iceberg/issues/6320
### Apache Iceberg version
1.1.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
When running queries like Q27 in iceberg V2 with vectorized parquet reading, it was observed that it is slower than traditional spark+vectorized_parq_reading. Profile more revealed that cache allocation was causing pressure on JVM which is covered in https://github.com/apache/iceberg/issues/6319.
I added a local patch to disable the cache and profile for CPU. This was done to get past this issue and look for other bottlenecks. This revealed that good amount of CPU was spent on ArrowBuf boundary checks. This can be disabled by having "-Darrow.enable_unsafe_memory_access=true" in the JVM options. I observed 25% improvement in runtime with vectorized processing in q27 with both these issues addressed. Need to check if this option can be enabled in iceberg directly, or it needs to be documented so that users can include it in executor & driver JVM options.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi closed issue #6320: ArrowBuf boundary checks causing CPU burn and slowness in vectorized parq reading
Posted by GitBox <gi...@apache.org>.
aokolnychyi closed issue #6320: ArrowBuf boundary checks causing CPU burn and slowness in vectorized parq reading
URL: https://github.com/apache/iceberg/issues/6320
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hililiwei commented on issue #6320: ArrowBuf boundary checks causing CPU burn and slowness in vectorized parq reading
Posted by GitBox <gi...@apache.org>.
hililiwei commented on issue #6320:
URL: https://github.com/apache/iceberg/issues/6320#issuecomment-1342084322
+1 on documenting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on issue #6320: ArrowBuf boundary checks causing CPU burn and slowness in vectorized parq reading
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on issue #6320:
URL: https://github.com/apache/iceberg/issues/6320#issuecomment-1377927016
Resolved by PR #6550.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] singhpk234 commented on issue #6320: ArrowBuf boundary checks causing CPU burn and slowness in vectorized parq reading
Posted by GitBox <gi...@apache.org>.
singhpk234 commented on issue #6320:
URL: https://github.com/apache/iceberg/issues/6320#issuecomment-1332996731
+1 on documenting considering the speedup it brings. I think having a section like https://spark.apache.org/docs/latest/sql-performance-tuning.html in iceberg could be really helpful.
@kbendick also mentioned the same in https://github.com/apache/iceberg/issues/4217#issuecomment-1159209016 as a recommendation to avoid redundant checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rbalamohan commented on issue #6320: ArrowBuf boundary checks causing CPU burn and slowness in vectorized parq reading
Posted by GitBox <gi...@apache.org>.
rbalamohan commented on issue #6320:
URL: https://github.com/apache/iceberg/issues/6320#issuecomment-1333034477
Looks like this property is document in arrow project itself in perf section. It will be good to have this document it for iceberg for perf (given vecctorized parq uses arrow).
https://github.com/apache/arrow/blob/master/java/README.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org