You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "JkSelf (via GitHub)" <gi...@apache.org> on 2025/02/12 10:04:26 UTC
[I] [VL] Track Jvm based libhdfs performance gap issue [incubator-gluten]
JkSelf opened a new issue, #7746:
URL: https://github.com/apache/incubator-gluten/issues/7746
### Description
Currently, we supported JVM-based libhdfs in both the [Velox ](https://github.com/facebookincubator/velox/pull/9835)and [Gluten ](https://github.com/apache/incubator-gluten/pull/6172)communities. However, the performance in Jenkins tests has not met our expectations.
- Static linking with libhdfs3.so: 340%
- Dynamic linking with JVM: 214.75% (requires further investigation)
- Dynamic linking with libhdfs3: 323.85% (only replacing libhdfs.so with libhdfs3.so)
- Expected performance for static linking in [PR#7697](https://github.com/apache/incubator-gluten/pull/7697): 340%
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Track Jvm based libhdfs performance gap issue [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #7746:
URL: https://github.com/apache/incubator-gluten/issues/7746#issuecomment-2451324216
FYI, static means link at compile time. Dynamic here means loading so library at runtime.
Ratio above means the performance boost compared to Vanilla Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Track Jvm based libhdfs performance gap issue [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #7746:
URL: https://github.com/apache/incubator-gluten/issues/7746#issuecomment-2458146901
Looks the indirect call to jvm is not efficient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Track Jvm based libhdfs performance gap issue [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #7746:
URL: https://github.com/apache/incubator-gluten/issues/7746#issuecomment-2460742356
All above perf test is on single node with enough SSDs. Not sure if the short circuit is enabled or not in JVM test, but even not the perf shouldn't drop so much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Track Jvm based libhdfs performance gap issue [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #7746:
URL: https://github.com/apache/incubator-gluten/issues/7746#issuecomment-2465596845
@JkSelf Can you update above table with current data?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Track Jvm based libhdfs performance gap issue [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #7746:
URL: https://github.com/apache/incubator-gluten/issues/7746#issuecomment-2461131824
More clear. @JkSelf Can you also test the missing config?
perf gain over vanilla | link at compile time | load at runtime
-- | -- | --
libhdfs3 | 3.4 | 3.23
jvm libhdfs | | 2.14
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Track Jvm based libhdfs performance gap issue [incubator-gluten]
Posted by "zhanglistar (via GitHub)" <gi...@apache.org>.
zhanglistar commented on issue #7746:
URL: https://github.com/apache/incubator-gluten/issues/7746#issuecomment-2467204099
So you guys turn to static linking?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Track Jvm based libhdfs performance gap issue [incubator-gluten]
Posted by "Yohahaha (via GitHub)" <gi...@apache.org>.
Yohahaha commented on issue #7746:
URL: https://github.com/apache/incubator-gluten/issues/7746#issuecomment-2653231979
I found this issue more severe in our environment.
TPCDS sf1000 + Spark 3.4 + 192core
date 1024: 494s
date 1025: 632s (1.27x)
date 1119: 580s (1.17x)
@FelixYBW
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Track Jvm based libhdfs performance gap issue [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #7746:
URL: https://github.com/apache/incubator-gluten/issues/7746#issuecomment-2484336647
Thank you Ke. Can you add a readme and describe how to config libhdfs3 and hadoop libhdfs? Is changing executorEnv.LD_LIBRARY_PATH enough?
Going fwd, we will only support run time load of libhdfs library.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Track Jvm based libhdfs performance gap issue [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #7746:
URL: https://github.com/apache/incubator-gluten/issues/7746#issuecomment-2471570913
> So you guys turn to static linking?
No, latest perf test shows libhdfs3 has the same perf. If libhdfs jvm also has the same perf, we can just use the runtime loading way.
perf gain over vanilla | link at compile time | load at runtime
-- | -- | --
libhdfs3 | 3.23 | 3.23
jvm libhdfs | | 2.14
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Track Jvm based libhdfs performance gap issue [incubator-gluten]
Posted by "Yohahaha (via GitHub)" <gi...@apache.org>.
Yohahaha commented on issue #7746:
URL: https://github.com/apache/incubator-gluten/issues/7746#issuecomment-2670235870
> [@Yohahaha](https://github.com/Yohahaha) `date 1024`, data was read using the native libhdfs3 library, which does not take advantage of short-circuit reads. However, `date 1119`, data was accessed using the JVM-based libhdfs, which benefits from short-circuit reads. I enabled short-circuit reads in my local test, resulting in a 1.09x performance improvement as noted above. And you can re-test with enabling short circuit read.
I have set `dfs.client.read.shortcircuit=true`, are there any configurations I may have missed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Track Jvm based libhdfs performance gap issue [incubator-gluten]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on issue #7746:
URL: https://github.com/apache/incubator-gluten/issues/7746#issuecomment-2670553881
@Yohahaha No other configurations need to set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Track Jvm based libhdfs performance gap issue [incubator-gluten]
Posted by "zhanglistar (via GitHub)" <gi...@apache.org>.
zhanglistar commented on issue #7746:
URL: https://github.com/apache/incubator-gluten/issues/7746#issuecomment-2456605626
Didn't think that performance diff so fast.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Track Jvm based libhdfs performance gap issue [incubator-gluten]
Posted by "zhanglistar (via GitHub)" <gi...@apache.org>.
zhanglistar commented on issue #7746:
URL: https://github.com/apache/incubator-gluten/issues/7746#issuecomment-2458521982
I thought IO is the big head, indirect call is roughly the virtual table cost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Track Jvm based libhdfs performance gap issue [incubator-gluten]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on issue #7746:
URL: https://github.com/apache/incubator-gluten/issues/7746#issuecomment-2655472542
@Yohahaha `date 1024`, data was read using the native libhdfs3 library, which does not take advantage of short-circuit reads. However, `date 1119`, data was accessed using the JVM-based libhdfs, which benefits from short-circuit reads. I enabled short-circuit reads in my local test, resulting in a 1.09x performance improvement as noted above. And you can re-test with enabling short circuit read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Track Jvm based libhdfs performance gap issue [incubator-gluten]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on issue #7746:
URL: https://github.com/apache/incubator-gluten/issues/7746#issuecomment-2670533760
@Yohahaha You also need to set the `dfs.domain.socket.path` as below.
```
<property>
<name>dfs.client.read.shortcircuit</name>
<value>true</value>
</property>
<property>
<name>dfs.domain.socket.path</name>
<value>/var/lib/hadoop-hdfs/dn_socket</value>
</property>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Track Jvm based libhdfs performance gap issue [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #7746:
URL: https://github.com/apache/incubator-gluten/issues/7746#issuecomment-2670561599
Did you create the path?
also check the stderr to see if short-circuit does be enable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Track Jvm based libhdfs performance gap issue [incubator-gluten]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on issue #7746:
URL: https://github.com/apache/incubator-gluten/issues/7746#issuecomment-2482406467
@FelixYBW @zhanglistar
After merging the JVM libhdfs patch [PR#6172](https://github.com/apache/incubator-gluten/pull/6172) on 10/25, the Jenkins performance significantly degraded compared to the test results from 10/24. Over the past few days, we identified three performance degradation issues:
1. IO Thread Configuration: The IO thread configuration was different between the tests on 10/24 (disabled) and 10/25 (enabled), causing an increase in peak memory usage. This was fixed by disabling the IO thread configuration.
2. Hadoop Configuration passing when support viewfs: When supporting viewfs, we passed the Hadoop configuration while converting the SplitInfo for each partition, resulting in approximately a 5% performance degradation. We added a configuration option to enable viewfs support and converted the viewfs path to the hdfs path after obtaining all partitions' SplitInfo in [PR#7892](https://github.com/apache/incubator-gluten/pull/7892) .
3. Thread Local Parameter: To address Depeek's comment about forward reference, we moved the thread local parameter from a member parameter to a template parameter in the method. The frequent initialization of the thread local parameter in the method caused about a 5% performance degradation. We reverted the thread local parameter to a member parameter in [PR#7966](https://github.com/apache/incubator-gluten/pull/7966).
After addressing the above three issues, we re-tested the performance of dynamic and static linking of libhdfs and libhdfs3. The conclusions are as follows:
1. The results for static linking and dynamic linking are almost same.
2. The performance of libhdfs3 is 3.28x (1058s vs 3187s).
3. The performance of libhdfs is 3.01x (971s vs 3187s).
4. The performance of libhdfs is about 1.08x(971s vs 1058s) lower compared to libhdfs3, which aligns with our previous tests [here](https://github.com/facebookincubator/velox/discussions/9969#discussioncomment-9760242).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Track Jvm based libhdfs performance gap issue [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW closed issue #7746: [VL] Track Jvm based libhdfs performance gap issue
URL: https://github.com/apache/incubator-gluten/issues/7746
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Track Jvm based libhdfs performance gap issue [incubator-gluten]
Posted by "Yohahaha (via GitHub)" <gi...@apache.org>.
Yohahaha commented on issue #7746:
URL: https://github.com/apache/incubator-gluten/issues/7746#issuecomment-2655225324
> [@Yohahaha](https://github.com/Yohahaha) Have you enable the short-circuit when you running `date 1119` ?
no, the configs are consistent across all tests.
I guess the diff between 1025 and 1119 is hdfs thread-local variable optimize.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Track Jvm based libhdfs performance gap issue [incubator-gluten]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on issue #7746:
URL: https://github.com/apache/incubator-gluten/issues/7746#issuecomment-2655162061
@Yohahaha Have you enable the short-circuit when you running `date 1119` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Track Jvm based libhdfs performance gap issue [incubator-gluten]
Posted by "Yohahaha (via GitHub)" <gi...@apache.org>.
Yohahaha commented on issue #7746:
URL: https://github.com/apache/incubator-gluten/issues/7746#issuecomment-2670540749
> [@Yohahaha](https://github.com/Yohahaha) You also need to set the `dfs.domain.socket.path` as below.
>
> ```
> <property>
> <name>dfs.client.read.shortcircuit</name>
> <value>true</value>
> </property>
> <property>
> <name>dfs.domain.socket.path</name>
> <value>/var/lib/hadoop-hdfs/dn_socket</value>
> </property>
> ```
`dfs.domain.socket.path` was set too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org