You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "ahshahid (via GitHub)" <gi...@apache.org> on 2025/11/17 20:41:35 UTC
[PR] [SPARK-54386][SQL][CONNECT][CLIENT]. Fix for potential deadlock in the JavaEncoderSuite, caus… [spark]
ahshahid opened a new pull request, #53100:
URL: https://github.com/apache/spark/pull/53100
…ed by buffers of the spawned process, not being consumed.
### What changes were proposed in this pull request?
Added code in the client VM to consume the output and error stream of the connect server process spawned by it, so that spawned process's buffers are cleared..
### Why are the changes needed?
The Connect Server process spawned by the test using the class RemoteSparkSession, by default ( non debug mode) , does not consume the spawned process's outout and error buffers. In certain situations ( extra logging ) it can cause deadlock, as the synchronous thread doing the logging acquires the lock in the Log4J's class, but is not able to write as the buffers are full... which causes other threads in the server vm to block , waiting to acquire the lock.
The stacktraces of the deadlock situation are available in the corresponding ticket.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
All the existing tests ( including the JavaEncoderSuite) passes with the code of cleaning up of the buffers on my local checxkout.
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54386][CONNECT][TESTS] Fix for potential deadlock in the JavaEncoderSuite [spark]
Posted by "ahshahid (via GitHub)" <gi...@apache.org>.
ahshahid commented on PR #53100:
URL: https://github.com/apache/spark/pull/53100#issuecomment-3609787601
Thanks...
This bug and PR both were opened before the later bug ...but mine is still
open and PR still uncommitted ...
I sort of know the reasons...but would like to hear from "horses mouth",
why no action happens on bugs or Prs opend.
On Wed, Dec 3, 2025, 9:33 PM Cheng Pan ***@***.***> wrote:
> *pan3793* left a comment (apache/spark#53100)
> <https://github.com/apache/spark/pull/53100#issuecomment-3609754209>
>
> Seems that SPARK-54551 <https://issues.apache.org/jira/browse/SPARK-54551>
> fixed the exact same issue
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/spark/pull/53100#issuecomment-3609754209>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AC6XG2CFJKKWHXFWNXLDXND376MQHAVCNFSM6AAAAACMMCXZRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMMBZG42TIMRQHE>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54386][CONNECT][TESTS] Fix for potential deadlock in the JavaEncoderSuite [spark]
Posted by "ahshahid (via GitHub)" <gi...@apache.org>.
ahshahid closed pull request #53100: [SPARK-54386][CONNECT][TESTS] Fix for potential deadlock in the JavaEncoderSuite
URL: https://github.com/apache/spark/pull/53100
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54386][SQL][CONNECT][CLIENT]. Fix for potential deadlock in the JavaEncoderSuite, caus… [spark]
Posted by "ahshahid (via GitHub)" <gi...@apache.org>.
ahshahid commented on PR #53100:
URL: https://github.com/apache/spark/pull/53100#issuecomment-3561319186
> Shouldn't we do something with the logs? Like show them to the user?
The reason I just ignored the log lines, is that there is a already a boolean isDebug which if true, redirects the outputs of the child process to the testing VM ( which outputs to console).. So looks like it was intended to print log lines only if debug is on, else intention was to ignore... so keeping the behaviour same... Also please note that the added lines kick in onlt if toDebug is false...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54386][CONNECT][TESTS] Fix for potential deadlock in the JavaEncoderSuite [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on PR #53100:
URL: https://github.com/apache/spark/pull/53100#issuecomment-3613402126
@ahshahid is the problem fixed or not? Is this PR still needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54386][CONNECT][TESTS] Fix for potential deadlock in the JavaEncoderSuite [spark]
Posted by "ahshahid (via GitHub)" <gi...@apache.org>.
ahshahid commented on PR #53100:
URL: https://github.com/apache/spark/pull/53100#issuecomment-3613654597
@hvanhovell and @pan3793 and @holdenk ,
Thank you for your comments.. This PR is no longer needed as its fixed by the PRs mentioned by @pan3793 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54386][SQL][CONNECT][CLIENT]. Fix for potential deadlock in the JavaEncoderSuite, caus… [spark]
Posted by "holdenk (via GitHub)" <gi...@apache.org>.
holdenk commented on PR #53100:
URL: https://github.com/apache/spark/pull/53100#issuecomment-3560768132
Shouldn't we do something with the logs? Like show them to the user?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54386][CONNECT][TESTS] Fix for potential deadlock in the JavaEncoderSuite [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #53100:
URL: https://github.com/apache/spark/pull/53100#issuecomment-3609754209
Seems that SPARK-54551 fixed the exact same issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org