You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 05:06:55 UTC

[I] Support spark writer running on thrift server [hudi]

hudi-bot opened a new issue, #15284:
URL: https://github.com/apache/hudi/issues/15284

   Spark thrift server manages a sql by running with the thread local spark session. It identifies the jobs belong to the sql by setting the job group Id. If user wants to cancel the sql running, thrift server will call session.cancelJobGroup. However, the implementation of hudi spark writer uses different job group Ids to distinguish the different proccesing part of the writing progress, like workprofile, compaction, rollback and so on. There is an Incompatibility
   
   ## JIRA info
   
   - Link: https://issues.apache.org/jira/browse/HUDI-4415
   - Type: New Feature
   
   
   ---
   
   
   ## Comments
   
   22/Jul/22 06:42;rmahindra;[~minihippo] are you planning to work on it and raise a PR?;;;
   
   ---
   
   25/Jul/22 07:25;minihippo;[~rmahindra] If the problem has high priority according to the feedback, i will raise a pr to solve it. By now, i get a simple implementation as a workaround;;;


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Support spark writer running on thrift server [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #15284: Support spark writer running on thrift server
URL: https://github.com/apache/hudi/issues/15284


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Support spark writer running on thrift server [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #15284:
URL: https://github.com/apache/hudi/issues/15284#issuecomment-4795346489

   This appears to be addressed by [HUDI-8596] / PR #12358 (commit d6d4e8893a). `HoodieSparkEngineContext.setJobStatus` no longer calls `setJobGroup`; it uses `setJobDescription` instead, so the thrift server's job group id is preserved and `session.cancelJobGroup` works correctly. A regression test in `TestHoodieSparkEngineContext.testSetJobStatus` asserts that the job group id set by the caller is not overwritten by Hudi's write path. Closing as fixed — please reopen if you can still reproduce on current master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org