You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "ganeshashree (via GitHub)" <gi...@apache.org> on 2025/09/22 03:24:18 UTC
[PR] [SPARK-53656][SQL] Refactor MemoryStream to use SparkSession instead of SQLContext [spark]
ganeshashree opened a new pull request, #52402:
URL: https://github.com/apache/spark/pull/52402
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Refactor MemoryStream to use SparkSession instead of SQLContext.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
SQLContext is deprecated in newer versions of Spark.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as new features, bug fixes, or other behavior changes. Documentation-only updates are not considered user-facing changes.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
Verified that the affected tests are passing successfully.
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53656][SS] Refactor MemoryStream to use SparkSession instead of SQLContext [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #52402:
URL: https://github.com/apache/spark/pull/52402#issuecomment-3450029953
I understand it's a "hard-to-understand" protocol, but historically, Apache Spark project considers the classes not documented in Scala/Java/Python doc as non-public API. I'm not a part of the discussion/decision, but IIUC there is a protocol with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53656][SQL] Refactor MemoryStream to use SparkSession instead of SQLContext [spark]
Posted by "ganeshashree (via GitHub)" <gi...@apache.org>.
ganeshashree commented on code in PR #52402:
URL: https://github.com/apache/spark/pull/52402#discussion_r2366879652
##########
sql/pipelines/src/test/scala/org/apache/spark/sql/pipelines/graph/MaterializeTablesSuite.scala:
##########
@@ -261,6 +262,7 @@ abstract class MaterializeTablesSuite extends BaseCoreExecutionTest {
test("invalid schema merge") {
val session = spark
+ implicit val sparkSession: SparkSession = spark
import session.implicits._
implicit def sqlContext: org.apache.spark.sql.classic.SQLContext = spark.sqlContext
Review Comment:
No, removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53656][SQL] Refactor MemoryStream to use SparkSession instead of SQLContext [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52402:
URL: https://github.com/apache/spark/pull/52402#discussion_r2366650702
##########
sql/pipelines/src/test/scala/org/apache/spark/sql/pipelines/graph/MaterializeTablesSuite.scala:
##########
@@ -261,6 +262,7 @@ abstract class MaterializeTablesSuite extends BaseCoreExecutionTest {
test("invalid schema merge") {
val session = spark
+ implicit val sparkSession: SparkSession = spark
import session.implicits._
implicit def sqlContext: org.apache.spark.sql.classic.SQLContext = spark.sqlContext
Review Comment:
do we still need this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53656][SS] Refactor MemoryStream to use SparkSession instead of SQLContext [spark]
Posted by "ganeshashree (via GitHub)" <gi...@apache.org>.
ganeshashree commented on PR #52402:
URL: https://github.com/apache/spark/pull/52402#issuecomment-3369878866
> > @ganeshashree Thanks for the proposal. The change looks OK to me.
> > Have we checked the warn (build/log) message when we use SQLContext here? If we weren't providing the message to migrate easily, it might be beneficial to defer replacement of apply() and have intermediate migration step (deprecation of the existing methods and removal of them in Spark 5.0.0).
>
> @HeartSaVioR Thanks for reviewing. Currently, no warning appears in the build log when we use SQLContext. Creating two versions of `MemoryStream.apply` for SparkSession and SQLContext and showing a warning for SQLContext would require resolving ambiguity when both sparkSession and sqlContext are set as implicit variables. Since this is an internal API, please review whether it's acceptable to make this change and update the callers to use MemoryStream with an implicit SparkSession instead of SQLContext, where applicable. I'm exploring further to resolve the ambiguity by preferring SparkSession instead of SQLContext.
Made changes to support two versions of `MemoryStream.apply` for SparkSession and SQLContext, with a warning for SQLContext, and also addressed ambiguity when both sparkSession and sqlContext are set as implicit variables by defining a low-priority trait.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53656][SS] Refactor MemoryStream to use SparkSession instead of SQLContext [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #52402:
URL: https://github.com/apache/spark/pull/52402#issuecomment-3383986138
@HeartSaVioR do you have any other concerns with this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53656][SQL] Refactor MemoryStream to use SparkSession instead of SQLContext [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52402:
URL: https://github.com/apache/spark/pull/52402#discussion_r2366651249
##########
sql/pipelines/src/test/scala/org/apache/spark/sql/pipelines/graph/TriggeredGraphExecutionSuite.scala:
##########
@@ -179,6 +179,7 @@ class TriggeredGraphExecutionSuite extends ExecutionTest with SharedSparkSession
test("three hop pipeline") {
val session = spark
+ implicit val sparkSession: SparkSession = spark
Review Comment:
where was the previous implicit SQLContext defined?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53656][SQL] Refactor MemoryStream to use SparkSession instead of SQLContext [spark]
Posted by "ganeshashree (via GitHub)" <gi...@apache.org>.
ganeshashree commented on code in PR #52402:
URL: https://github.com/apache/spark/pull/52402#discussion_r2366880346
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/AsyncProgressTrackingMicroBatchExecutionSuite.scala:
##########
@@ -67,9 +67,9 @@ class AsyncProgressTrackingMicroBatchExecutionSuite
class MemoryStreamCapture[A: Encoder](
id: Int,
- sqlContext: SQLContext,
+ sparkSession: SparkSession,
numPartitions: Option[Int] = None)
- extends MemoryStream[A](id, sqlContext, numPartitions = numPartitions) {
+ extends MemoryStream[A](id, sparkSession: SparkSession, numPartitions = numPartitions) {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53656][SQL] Refactor MemoryStream to use SparkSession instead of SQLContext [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52402:
URL: https://github.com/apache/spark/pull/52402#discussion_r2366649574
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/ContinuousMemoryStream.scala:
##########
@@ -44,8 +44,9 @@ import org.apache.spark.util.RpcUtils
* ContinuousMemoryStreamInputPartitionReader instances to poll. It returns the record at
* the specified offset within the list, or null if that offset doesn't yet have a record.
*/
-class ContinuousMemoryStream[A : Encoder](id: Int, sqlContext: SQLContext, numPartitions: Int = 2)
- extends MemoryStreamBase[A](sqlContext) with ContinuousStream {
+class ContinuousMemoryStream[A : Encoder](id: Int,
+ sparkSession: SparkSession, numPartitions: Int = 2)
Review Comment:
```suggestion
class ContinuousMemoryStream[A : Encoder](
id: Int,
sparkSession: SparkSession,
numPartitions: Int = 2)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53656][SQL] Refactor MemoryStream to use SparkSession instead of SQLContext [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52402:
URL: https://github.com/apache/spark/pull/52402#discussion_r2366649739
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/AsyncProgressTrackingMicroBatchExecutionSuite.scala:
##########
@@ -67,9 +67,9 @@ class AsyncProgressTrackingMicroBatchExecutionSuite
class MemoryStreamCapture[A: Encoder](
id: Int,
- sqlContext: SQLContext,
+ sparkSession: SparkSession,
numPartitions: Option[Int] = None)
- extends MemoryStream[A](id, sqlContext, numPartitions = numPartitions) {
+ extends MemoryStream[A](id, sparkSession: SparkSession, numPartitions = numPartitions) {
Review Comment:
```suggestion
extends MemoryStream[A](id, sparkSession, numPartitions = numPartitions) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53656][SS] Refactor MemoryStream to use SparkSession instead of SQLContext [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52402:
URL: https://github.com/apache/spark/pull/52402#discussion_r2415490494
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/ContinuousMemoryStream.scala:
##########
@@ -112,11 +115,11 @@ class ContinuousMemoryStream[A : Encoder](id: Int, sqlContext: SQLContext, numPa
object ContinuousMemoryStream {
Review Comment:
shall we do the same low priority implicit trick here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53656][SQL] Refactor MemoryStream to use SparkSession instead of SQLContext [spark]
Posted by "ganeshashree (via GitHub)" <gi...@apache.org>.
ganeshashree commented on code in PR #52402:
URL: https://github.com/apache/spark/pull/52402#discussion_r2366886669
##########
sql/pipelines/src/test/scala/org/apache/spark/sql/pipelines/graph/TriggeredGraphExecutionSuite.scala:
##########
@@ -179,6 +179,7 @@ class TriggeredGraphExecutionSuite extends ExecutionTest with SharedSparkSession
test("three hop pipeline") {
val session = spark
+ implicit val sparkSession: SparkSession = spark
Review Comment:
It seems like it was getting implicit sqlContext defined in [SharedSparkSession](https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/test/SharedSparkSession.scala#L106).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53656][SQL] Refactor MemoryStream to use SparkSession instead of SQLContext [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52402:
URL: https://github.com/apache/spark/pull/52402#discussion_r2366650905
##########
sql/pipelines/src/test/scala/org/apache/spark/sql/pipelines/graph/MaterializeTablesSuite.scala:
##########
@@ -261,6 +262,7 @@ abstract class MaterializeTablesSuite extends BaseCoreExecutionTest {
test("invalid schema merge") {
val session = spark
+ implicit val sparkSession: SparkSession = spark
import session.implicits._
implicit def sqlContext: org.apache.spark.sql.classic.SQLContext = spark.sqlContext
Review Comment:
ditto for others in this file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53656][SS] Refactor MemoryStream to use SparkSession instead of SQLContext [spark]
Posted by "ganeshashree (via GitHub)" <gi...@apache.org>.
ganeshashree commented on PR #52402:
URL: https://github.com/apache/spark/pull/52402#issuecomment-3420475092
> Could you please resolve the conflict?
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53656][SS] Refactor MemoryStream to use SparkSession instead of SQLContext [spark]
Posted by "ganeshashree (via GitHub)" <gi...@apache.org>.
ganeshashree commented on PR #52402:
URL: https://github.com/apache/spark/pull/52402#issuecomment-3344875714
> @ganeshashree Thanks for the proposal. The change looks OK to me.
>
> Have we checked the warn (build/log) message when we use SQLContext here? If we weren't providing the message to migrate easily, it might be beneficial to defer replacement of apply() and have intermediate migration step (deprecation of the existing methods and removal of them in Spark 5.0.0).
Thanks for reviewing. Currently, no warning appears in the build log when we use SQLContext. Creating two versions of `MemoryStream.apply` for SparkSession and SQLContext and showing a warning for SQLContext would require resolving ambiguity when both sparkSession and sqlContext are set as implicit variables. Since this is an internal API, please review whether it's acceptable to make this change and update the callers to use MemoryStream with an implicit SparkSession instead of SQLContext where applicable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53656][SS] Refactor MemoryStream to use SparkSession instead of SQLContext [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #52402:
URL: https://github.com/apache/spark/pull/52402#issuecomment-3420771240
Thanks! Merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53656][SS] Refactor MemoryStream to use SparkSession instead of SQLContext [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #52402:
URL: https://github.com/apache/spark/pull/52402#issuecomment-3450353278
@manuzhang we didn't remove the old method, how does it break iceberg tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53656][SS] Refactor MemoryStream to use SparkSession instead of SQLContext [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #52402:
URL: https://github.com/apache/spark/pull/52402#issuecomment-3455323257
We have a clear definition of public APIs: the APIs listed in the public doc such as https://spark.apache.org/docs/latest/api/scala/org/apache/spark/index.html are public.
Spark does not use modifiers like `private[sql]` to hide all the internal APIs, so that Spark plugins can do powerful things. But it does not mean Spark guarantees backward compatibility for all these compile-time-public internal APIs, which is just not possible. Spark plugins are responsible for updating their code to catch up with Spark's internal APIs changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53656][SQL] Refactor MemoryStream to use SparkSession instead of SQLContext [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #52402:
URL: https://github.com/apache/spark/pull/52402#issuecomment-3317616935
@ganeshashree
Thanks for the proposal. The change looks OK to me.
Have we checked the warn (build/log) message when we use SQLContext here? If we weren't providing the message to migrate easily, it might be beneficial to defer replacement of apply() and have intermediate migration step (deprecation of the existing methods and removal of them in Spark 5.0.0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53656][SQL] Refactor MemoryStream to use SparkSession instead of SQLContext [spark]
Posted by "ganeshashree (via GitHub)" <gi...@apache.org>.
ganeshashree commented on code in PR #52402:
URL: https://github.com/apache/spark/pull/52402#discussion_r2366878634
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/ContinuousMemoryStream.scala:
##########
@@ -44,8 +44,9 @@ import org.apache.spark.util.RpcUtils
* ContinuousMemoryStreamInputPartitionReader instances to poll. It returns the record at
* the specified offset within the list, or null if that offset doesn't yet have a record.
*/
-class ContinuousMemoryStream[A : Encoder](id: Int, sqlContext: SQLContext, numPartitions: Int = 2)
- extends MemoryStreamBase[A](sqlContext) with ContinuousStream {
+class ContinuousMemoryStream[A : Encoder](id: Int,
+ sparkSession: SparkSession, numPartitions: Int = 2)
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53656][SS] Refactor MemoryStream to use SparkSession instead of SQLContext [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR closed pull request #52402: [SPARK-53656][SS] Refactor MemoryStream to use SparkSession instead of SQLContext
URL: https://github.com/apache/spark/pull/52402
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53656][SS] Refactor MemoryStream to use SparkSession instead of SQLContext [spark]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #52402:
URL: https://github.com/apache/spark/pull/52402#issuecomment-3451129856
@ganeshashree Yes, I've already [made the change in 4.1.0 support and the tests passed for 4.1.0-preview3(RC1)](https://github.com/apache/iceberg/pull/14155). I just want to call out this should not be considered an internal API, especially for downstream Java projects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53656][SS] Refactor MemoryStream to use SparkSession instead of SQLContext [spark]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #52402:
URL: https://github.com/apache/spark/pull/52402#issuecomment-3450932000
@cloud-fan which old method do you mean? The constructor has changed and that's breaking for Java code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53656][SS] Refactor MemoryStream to use SparkSession instead of SQLContext [spark]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #52402:
URL: https://github.com/apache/spark/pull/52402#issuecomment-3449967255
Note this is not an internal API as Iceberg uses it in tests. Of course, we can easily change it at the caller side.
https://github.com/apache/iceberg/blob/68e555b94f4706a2af41dcb561c84007230c0bc1/spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java#L223
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53656][SS] Refactor MemoryStream to use SparkSession instead of SQLContext [spark]
Posted by "ganeshashree (via GitHub)" <gi...@apache.org>.
ganeshashree commented on PR #52402:
URL: https://github.com/apache/spark/pull/52402#issuecomment-3451033736
> @cloud-fan which old method do you mean? The constructor has changed and that's breaking for Java code.
@manuzhang Thanks for reporting this. The current changes are backward compatible with Scala but not with Java. I see that [two tests in Iceberg 4.0](https://github.com/search?q=repo%3Aapache%2Ficeberg+return+new+MemoryStream+path%3A%2F%5Espark%5C%2Fv4%5C.0%5C%2Fspark%5C%2Fsrc%5C%2Ftest%5C%2Fjava%5C%2Forg%5C%2Fapache%5C%2Ficeberg%5C%2Fspark%5C%2Fsource%5C%2F%2F&type=code) are breaking due to this change. Is it fine to modify the tests to use SparkSession instead of sqlContext? Please let me know if you rely on the old version of the constructor that takes sqlContext as a parameter. I can consider making this change backward compatible with Java as well. However, since sqlContext is deprecated, it is best practice to use the new version of the constructor and pass sparkSession as a parameter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53656][SS] Refactor MemoryStream to use SparkSession instead of SQLContext [spark]
Posted by "ganeshashree (via GitHub)" <gi...@apache.org>.
ganeshashree commented on code in PR #52402:
URL: https://github.com/apache/spark/pull/52402#discussion_r2443774779
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/ContinuousMemoryStream.scala:
##########
@@ -112,11 +115,11 @@ class ContinuousMemoryStream[A : Encoder](id: Int, sqlContext: SQLContext, numPa
object ContinuousMemoryStream {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53656][SQL] Refactor MemoryStream to use SparkSession instead of SQLContext [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #52402:
URL: https://github.com/apache/spark/pull/52402#issuecomment-3316659462
cc @HeartSaVioR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53656][SS] Refactor MemoryStream to use SparkSession instead of SQLContext [spark]
Posted by "ganeshashree (via GitHub)" <gi...@apache.org>.
ganeshashree commented on code in PR #52402:
URL: https://github.com/apache/spark/pull/52402#discussion_r2366886669
##########
sql/pipelines/src/test/scala/org/apache/spark/sql/pipelines/graph/TriggeredGraphExecutionSuite.scala:
##########
@@ -179,6 +179,7 @@ class TriggeredGraphExecutionSuite extends ExecutionTest with SharedSparkSession
test("three hop pipeline") {
val session = spark
+ implicit val sparkSession: SparkSession = spark
Review Comment:
It seems like it was getting implicit sqlContext defined in [SharedSparkSession](https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/test/SharedSparkSession.scala#L106). Explicitly defining the implicit SparkSession is required because the existing implicit SparkSession was assigned to a non-implicit session variable, and it couldn't locate the implicit SparkSession within the anonymous block.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org