You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "ajeyabsfujitsu (via GitHub)" <gi...@apache.org> on 2025/01/03 11:13:45 UTC

[I] [VL] Need Clarification on Java/Scala to Native C++ Handover and Profiling Native C++ Files [incubator-gluten]

ajeyabsfujitsu opened a new issue, #8419:
URL: https://github.com/apache/incubator-gluten/issues/8419

   ### Description
   
   I am exploring the incubator-gluten repository and trying to understand how Java/Scala code interacts with the native C++ code. However, I am having difficulty identifying where and how this handover happens within the codebase.
   
   An explanation on this process would be extremely helpful.
   
   
   **Java/Scala to C++ Handover** - I want to know the specific parts of the codebase where Java/Scala calls are handed over to native C++ code.  I’m unable to locate the key files or classes where this interaction is defined. If there are specific entry points or patterns used for this handover, please provide details.
   
   **Profiling C++ Code** - I’m attempting to use perf to profile the execution and see which C++ functions or files are being called. However, the perf output only shows functions from jibjvm.so or libc.so, and no Gluten or Velox-specific C++ functions are visible. Are there additional configurations or profiling tools recommended for this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [VL] Need Clarification on Java/Scala to Native C++ Handover and Profiling Native C++ Files [incubator-gluten]

Posted by "VaibhavFRI (via GitHub)" <gi...@apache.org>.
VaibhavFRI commented on issue #8419:
URL: https://github.com/apache/incubator-gluten/issues/8419#issuecomment-2572333090

   @ajeyabsfujitsu you should set `spark.gluten.sql.debug.keepJniWorkspaceDir=<path_to_dir>` conf also for `spark.gluten.sql.debug.keepJniWorkspace=true` to work.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [VL] Need Clarification on Java/Scala to Native C++ Handover and Profiling Native C++ Files [incubator-gluten]

Posted by "rajatma1993 (via GitHub)" <gi...@apache.org>.
rajatma1993 commented on issue #8419:
URL: https://github.com/apache/incubator-gluten/issues/8419#issuecomment-2575054515

   @zhztheplayer @FelixYBW 
   
   Based on your suggestions, I analyzed the file "Jniwrapper" to trace the Java calls that callback to C++. However, I noticed that most of the Gluten functions are pointing to Scala files rather than Java, which should ultimately lead to the Jniwrapper.
   
   I would like to understand the flow of Gluten functions that transition from Java to C++. Could you please guide me on which folders and files I should refer to for better clarity?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [VL] Need Clarification on Java/Scala to Native C++ Handover and Profiling Native C++ Files [incubator-gluten]

Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #8419:
URL: https://github.com/apache/incubator-gluten/issues/8419#issuecomment-2571411789

   you may go through all the files in the jni folder. Any JNI call can trigger the native function.
   
   this folder is for Velox backend:
   https://github.com/apache/incubator-gluten/tree/main/cpp/velox/jni
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [VL] Need Clarification on Java/Scala to Native C++ Handover and Profiling Native C++ Files [incubator-gluten]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #8419:
URL: https://github.com/apache/incubator-gluten/issues/8419#issuecomment-2571827877

   > Java/Scala to C++ Handover
   
   The code used for C++ / Java communication is usually written in files that have "JniWrapper" as names or name suffixes. `VeloxJniWrapper.cc`, `...JniWrapper.scala`, etc.
   
   > In perf call graph everything is just libjvm and libc. Is there something I am missing?
   
   Not sure, but perhaps something like this might happen when the debug symbols are missing from the profiler. Note Gluten deletes the used native library files from `tmp/` while the JVM is exiting. Could set `spark.gluten.sql.debug.keepJniWorkspace=true` to avoid the deletion if it's the reason.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [VL] Need Clarification on Java/Scala to Native C++ Handover and Profiling Native C++ Files [incubator-gluten]

Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #8419:
URL: https://github.com/apache/incubator-gluten/issues/8419#issuecomment-2569837979

   https://github.com/apache/incubator-gluten/tree/main/cpp/core/jni is the place where java call c++
   
   if your gluten is running, from perf you can see the hotspot of velox functions


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [VL] Need Clarification on Java/Scala to Native C++ Handover and Profiling Native C++ Files [incubator-gluten]

Posted by "ajeyabsfujitsu (via GitHub)" <gi...@apache.org>.
ajeyabsfujitsu commented on issue #8419:
URL: https://github.com/apache/incubator-gluten/issues/8419#issuecomment-2572292990

   @FelixYBW Thanks. I will go through that folder to try finding the calls.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [VL] Need Clarification on Java/Scala to Native C++ Handover and Profiling Native C++ Files [incubator-gluten]

Posted by "ajeyabsfujitsu (via GitHub)" <gi...@apache.org>.
ajeyabsfujitsu commented on issue #8419:
URL: https://github.com/apache/incubator-gluten/issues/8419#issuecomment-2572306814

   @zhztheplayer  Thanks for the response.
   
   Debug symbols were installed with this - 
   ```
   apt install -yq libc6-dbg libstdc++6-12-dbg openjdk-${JDK_VERSION}-dbg
   ```
   
   Is there any other command that you are aware of?
   
   I am trying to run tpch workload runs.
   
   spark.gluten.sql.debug.keepJniWorkspace=true 
   
   I tried setting this flag in tpch_parquet.sh but getting an error 
   
   java.util.NoSuchElementException: spark.gluten.sql.debug.keepJniWorkspaceDir 
   
   
   Should this flag be added somewhere else?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [VL] Need Clarification on Java/Scala to Native C++ Handover and Profiling Native C++ Files [incubator-gluten]

Posted by "ajeyabsfujitsu (via GitHub)" <gi...@apache.org>.
ajeyabsfujitsu commented on issue #8419:
URL: https://github.com/apache/incubator-gluten/issues/8419#issuecomment-2572337380

   Thanks Vaibhav.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [VL] Need Clarification on Java/Scala to Native C++ Handover and Profiling Native C++ Files [incubator-gluten]

Posted by "ajeyabsfujitsu (via GitHub)" <gi...@apache.org>.
ajeyabsfujitsu commented on issue #8419:
URL: https://github.com/apache/incubator-gluten/issues/8419#issuecomment-2570102435

   Thanks for the response.
   
   In perf call graph everything is just libjvm and libc. Is there something I am missing?
   ![perf_velox](https://github.com/user-attachments/assets/ffe3f91a-c61e-4713-8daa-a32eff608417)
   
   
   In async profiler, I get the java/scala functions called, but I wanted to know at what point native functions take over. I will refer the files in cpp/core/jni to see if there is any connection I can find.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [VL] Need Clarification on Java/Scala to Native C++ Handover and Profiling Native C++ Files [incubator-gluten]

Posted by "brijrajk (via GitHub)" <gi...@apache.org>.
brijrajk commented on issue #8419:
URL: https://github.com/apache/incubator-gluten/issues/8419#issuecomment-2585119004

   @rajatma1993 You should check the $GLUTEN_HOME/cpp/core/jni/JniWrapper.cc to understand where the handover of native calls takes place.
   
   So for example 
   
   ![image](https://github.com/user-attachments/assets/3daaef0a-099d-48e4-aea9-253d28537a20)
   
   the functionName  : Java_org_apache_gluten_vectorized_PlanEvaluatorJniWrapper_nativeCreateKernelWithIterator
   
   will give you reference from where it is being called. If you split the above function name by "_"
   you will get 
   Java, org, apache, gluten, vectorized, PlanEvaluatorJniWrapper, nativeCreateKernelWithIterator.
   
   so the class name from which it is called : 
   org.apache.gluten.vectorized.PlanEvaluatorJniWrapper.java   and the function name : nativeCreateKernelWithIterator
   
   Now you can search the above class into gluten code 
   ![image](https://github.com/user-attachments/assets/ee9ef8cf-c0db-4b1f-8d29-352592f097b4)
   
   this file is located at : $GLUTEN_HOME/gluten-data/src/main/java/org/apache/gluten/vectorized/PlanEvaluatorJniWrapper.java
   
   
   Hope this helps
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [VL] Need Clarification on Java/Scala to Native C++ Handover and Profiling Native C++ Files [incubator-gluten]

Posted by "VaibhavFRI (via GitHub)" <gi...@apache.org>.
VaibhavFRI commented on issue #8419:
URL: https://github.com/apache/incubator-gluten/issues/8419#issuecomment-2572420496

   @zhztheplayer I tried setting the `spark.gluten.sql.debug.keepJniWorkspace=true` conf, but I still get libjvm.so and no gluten function calls. Is anything else needs to be set in perf?
   ![image](https://github.com/user-attachments/assets/b8e2d896-d516-4300-aa04-da96b2725887)
   <img width="809" alt="image" src="https://github.com/user-attachments/assets/835c9792-a787-4c55-aee4-a574ecb93dc8" />
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org