You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "ajeyabsfujitsu (via GitHub)" <gi...@apache.org> on 2025/01/03 11:13:45 UTC
[I] [VL] Need Clarification on Java/Scala to Native C++ Handover and Profiling Native C++ Files [incubator-gluten]
ajeyabsfujitsu opened a new issue, #8419:
URL: https://github.com/apache/incubator-gluten/issues/8419
### Description
I am exploring the incubator-gluten repository and trying to understand how Java/Scala code interacts with the native C++ code. However, I am having difficulty identifying where and how this handover happens within the codebase.
An explanation on this process would be extremely helpful.
**Java/Scala to C++ Handover** - I want to know the specific parts of the codebase where Java/Scala calls are handed over to native C++ code. I’m unable to locate the key files or classes where this interaction is defined. If there are specific entry points or patterns used for this handover, please provide details.
**Profiling C++ Code** - I’m attempting to use perf to profile the execution and see which C++ functions or files are being called. However, the perf output only shows functions from jibjvm.so or libc.so, and no Gluten or Velox-specific C++ functions are visible. Are there additional configurations or profiling tools recommended for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Need Clarification on Java/Scala to Native C++ Handover and Profiling Native C++ Files [incubator-gluten]
Posted by "VaibhavFRI (via GitHub)" <gi...@apache.org>.
VaibhavFRI commented on issue #8419:
URL: https://github.com/apache/incubator-gluten/issues/8419#issuecomment-2572333090
@ajeyabsfujitsu you should set `spark.gluten.sql.debug.keepJniWorkspaceDir=<path_to_dir>` conf also for `spark.gluten.sql.debug.keepJniWorkspace=true` to work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Need Clarification on Java/Scala to Native C++ Handover and Profiling Native C++ Files [incubator-gluten]
Posted by "rajatma1993 (via GitHub)" <gi...@apache.org>.
rajatma1993 commented on issue #8419:
URL: https://github.com/apache/incubator-gluten/issues/8419#issuecomment-2575054515
@zhztheplayer @FelixYBW
Based on your suggestions, I analyzed the file "Jniwrapper" to trace the Java calls that callback to C++. However, I noticed that most of the Gluten functions are pointing to Scala files rather than Java, which should ultimately lead to the Jniwrapper.
I would like to understand the flow of Gluten functions that transition from Java to C++. Could you please guide me on which folders and files I should refer to for better clarity?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Need Clarification on Java/Scala to Native C++ Handover and Profiling Native C++ Files [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #8419:
URL: https://github.com/apache/incubator-gluten/issues/8419#issuecomment-2571411789
you may go through all the files in the jni folder. Any JNI call can trigger the native function.
this folder is for Velox backend:
https://github.com/apache/incubator-gluten/tree/main/cpp/velox/jni
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Need Clarification on Java/Scala to Native C++ Handover and Profiling Native C++ Files [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #8419:
URL: https://github.com/apache/incubator-gluten/issues/8419#issuecomment-2571827877
> Java/Scala to C++ Handover
The code used for C++ / Java communication is usually written in files that have "JniWrapper" as names or name suffixes. `VeloxJniWrapper.cc`, `...JniWrapper.scala`, etc.
> In perf call graph everything is just libjvm and libc. Is there something I am missing?
Not sure, but perhaps something like this might happen when the debug symbols are missing from the profiler. Note Gluten deletes the used native library files from `tmp/` while the JVM is exiting. Could set `spark.gluten.sql.debug.keepJniWorkspace=true` to avoid the deletion if it's the reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Need Clarification on Java/Scala to Native C++ Handover and Profiling Native C++ Files [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #8419:
URL: https://github.com/apache/incubator-gluten/issues/8419#issuecomment-2569837979
https://github.com/apache/incubator-gluten/tree/main/cpp/core/jni is the place where java call c++
if your gluten is running, from perf you can see the hotspot of velox functions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Need Clarification on Java/Scala to Native C++ Handover and Profiling Native C++ Files [incubator-gluten]
Posted by "ajeyabsfujitsu (via GitHub)" <gi...@apache.org>.
ajeyabsfujitsu commented on issue #8419:
URL: https://github.com/apache/incubator-gluten/issues/8419#issuecomment-2572292990
@FelixYBW Thanks. I will go through that folder to try finding the calls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Need Clarification on Java/Scala to Native C++ Handover and Profiling Native C++ Files [incubator-gluten]
Posted by "ajeyabsfujitsu (via GitHub)" <gi...@apache.org>.
ajeyabsfujitsu commented on issue #8419:
URL: https://github.com/apache/incubator-gluten/issues/8419#issuecomment-2572306814
@zhztheplayer Thanks for the response.
Debug symbols were installed with this -
```
apt install -yq libc6-dbg libstdc++6-12-dbg openjdk-${JDK_VERSION}-dbg
```
Is there any other command that you are aware of?
I am trying to run tpch workload runs.
spark.gluten.sql.debug.keepJniWorkspace=true
I tried setting this flag in tpch_parquet.sh but getting an error
java.util.NoSuchElementException: spark.gluten.sql.debug.keepJniWorkspaceDir
Should this flag be added somewhere else?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Need Clarification on Java/Scala to Native C++ Handover and Profiling Native C++ Files [incubator-gluten]
Posted by "ajeyabsfujitsu (via GitHub)" <gi...@apache.org>.
ajeyabsfujitsu commented on issue #8419:
URL: https://github.com/apache/incubator-gluten/issues/8419#issuecomment-2572337380
Thanks Vaibhav.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Need Clarification on Java/Scala to Native C++ Handover and Profiling Native C++ Files [incubator-gluten]
Posted by "ajeyabsfujitsu (via GitHub)" <gi...@apache.org>.
ajeyabsfujitsu commented on issue #8419:
URL: https://github.com/apache/incubator-gluten/issues/8419#issuecomment-2570102435
Thanks for the response.
In perf call graph everything is just libjvm and libc. Is there something I am missing?

In async profiler, I get the java/scala functions called, but I wanted to know at what point native functions take over. I will refer the files in cpp/core/jni to see if there is any connection I can find.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Need Clarification on Java/Scala to Native C++ Handover and Profiling Native C++ Files [incubator-gluten]
Posted by "brijrajk (via GitHub)" <gi...@apache.org>.
brijrajk commented on issue #8419:
URL: https://github.com/apache/incubator-gluten/issues/8419#issuecomment-2585119004
@rajatma1993 You should check the $GLUTEN_HOME/cpp/core/jni/JniWrapper.cc to understand where the handover of native calls takes place.
So for example

the functionName : Java_org_apache_gluten_vectorized_PlanEvaluatorJniWrapper_nativeCreateKernelWithIterator
will give you reference from where it is being called. If you split the above function name by "_"
you will get
Java, org, apache, gluten, vectorized, PlanEvaluatorJniWrapper, nativeCreateKernelWithIterator.
so the class name from which it is called :
org.apache.gluten.vectorized.PlanEvaluatorJniWrapper.java and the function name : nativeCreateKernelWithIterator
Now you can search the above class into gluten code

this file is located at : $GLUTEN_HOME/gluten-data/src/main/java/org/apache/gluten/vectorized/PlanEvaluatorJniWrapper.java
Hope this helps
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Need Clarification on Java/Scala to Native C++ Handover and Profiling Native C++ Files [incubator-gluten]
Posted by "VaibhavFRI (via GitHub)" <gi...@apache.org>.
VaibhavFRI commented on issue #8419:
URL: https://github.com/apache/incubator-gluten/issues/8419#issuecomment-2572420496
@zhztheplayer I tried setting the `spark.gluten.sql.debug.keepJniWorkspace=true` conf, but I still get libjvm.so and no gluten function calls. Is anything else needs to be set in perf?

<img width="809" alt="image" src="https://github.com/user-attachments/assets/835c9792-a787-4c55-aee4-a574ecb93dc8" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org