You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "JiaqiWang18 (via GitHub)" <gi...@apache.org> on 2025/07/18 00:06:23 UTC

[PR] [WIP] Spark 52432 session graph registry [spark]

JiaqiWang18 opened a new pull request, #51544:
URL: https://github.com/apache/spark/pull/51544

   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
     8. If you want to add or modify an error type or message, please read the guideline first in
        'common/utils/src/main/resources/error/README.md'.
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as new features, bug fixes, or other behavior changes. Documentation-only updates are not considered user-facing changes.
   
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
   -->
   
   
   ### Was this patch authored or co-authored using generative AI tooling?
   <!--
   If generative AI tooling has been used in the process of authoring this patch, please include the
   phrase: 'Generated-by: ' followed by the name of the tool and its version.
   If no, write 'No'.
   Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-52432][SDP][SQL] Scope DataflowGraphRegistry to Session [spark]

Posted by "sryza (via GitHub)" <gi...@apache.org>.
sryza commented on PR #51544:
URL: https://github.com/apache/spark/pull/51544#issuecomment-3097214846

   Merged to master


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-52432][SDP][SQL] Scope DataflowGraphRegistry to Session [spark]

Posted by "AnishMahto (via GitHub)" <gi...@apache.org>.
AnishMahto commented on code in PR #51544:
URL: https://github.com/apache/spark/pull/51544#discussion_r2216618046


##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SessionHolder.scala:
##########
@@ -486,6 +493,48 @@ case class SessionHolder(userId: String, sessionId: String, session: SparkSessio
     Option(pipelineExecutions.get(graphId))
   }
 
+  private[connect] def createDataflowGraph(
+      defaultCatalog: String,
+      defaultDatabase: String,
+      defaultSqlConf: Map[String, String]): String = {
+    dataflowGraphRegistry.createDataflowGraph(defaultCatalog, defaultDatabase, defaultSqlConf)
+  }
+
+  /**
+   * Retrieves the dataflow graph for the given graph ID.
+   */
+  private[connect] def getDataflowGraph(graphId: String): Option[GraphRegistrationContext] = {
+    dataflowGraphRegistry.getDataflowGraph(graphId)
+  }
+
+  /**
+   * Retrieves the dataflow graph for the given graph ID, throwing if not found.
+   */
+  private[connect] def getDataflowGraphOrThrow(graphId: String): GraphRegistrationContext = {
+    dataflowGraphRegistry.getDataflowGraphOrThrow(graphId)
+  }
+
+  /**
+   * Removes the dataflow graph with the given ID.
+   */
+  private[connect] def dropDataflowGraph(graphId: String): Unit = {
+    dataflowGraphRegistry.dropDataflowGraph(graphId)
+  }
+
+  /**
+   * Returns all dataflow graphs in this session.
+   */
+  private[connect] def getAllDataflowGraphs: Seq[GraphRegistrationContext] = {
+    dataflowGraphRegistry.getAllDataflowGraphs
+  }
+
+  /**
+   * Removes all dataflow graphs from this session. Called during session cleanup.
+   */
+  private[connect] def dropAllDataflowGraphs(): Unit = {
+    dataflowGraphRegistry.dropAllDataflowGraphs()
+  }
+

Review Comment:
   Is there any particular reason why we added these delegator methods, rather than just having callers call `SessionHolder.dataflowGraphRegistry.blah()`?
   
   If its for access modifier reasons, why not just do `private[connect] lazy val dataflowGraphRegistry`?



##########
sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/PythonPipelineSuite.scala:
##########
@@ -78,8 +83,16 @@ class PythonPipelineSuite
       throw new RuntimeException(
         s"Python process failed with exit code $exitCode. Output: ${output.mkString("\n")}")
     }
+    val activateSessions = SparkConnectService.sessionManager.listActiveSessions

Review Comment:
   `val activeSessions`



##########
sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/SparkDeclarativePipelinesServerSuite.scala:
##########
@@ -251,4 +251,243 @@ class SparkDeclarativePipelinesServerSuite
       assert(spark.table("spark_catalog.other.tableD").count() == 5)
     }
   }
+
+  test("dataflow graphs are session-specific") {
+    withRawBlockingStub { implicit stub =>
+      // Create a dataflow graph in the default session
+      val graphId1 = createDataflowGraph
+
+      // Register a dataset in the default session
+      sendPlan(
+        buildPlanFromPipelineCommand(
+          PipelineCommand
+            .newBuilder()
+            .setDefineDataset(
+              DefineDataset
+                .newBuilder()
+                .setDataflowGraphId(graphId1)
+                .setDatasetName("session1_table")
+                .setDatasetType(DatasetType.MATERIALIZED_VIEW))
+            .build()))
+
+      // Verify the graph exists in the default session
+      assert(getDefaultSessionHolder.getAllDataflowGraphs.size == 1)
+    }
+
+    // Create a second session with different user/session ID
+    val newSessionId = UUID.randomUUID().toString
+    val newSessionUserId = "session2_user"
+
+    withRawBlockingStub { implicit stub =>
+      // Override the test context to use different session
+      val newSessionExecuteRequest = buildExecutePlanRequest(
+        buildCreateDataflowGraphPlan(
+          proto.PipelineCommand.CreateDataflowGraph
+            .newBuilder()
+            .setDefaultCatalog("spark_catalog")
+            .setDefaultDatabase("default")
+            .build())).toBuilder
+        .setUserContext(proto.UserContext
+          .newBuilder()
+          .setUserId(newSessionUserId)
+          .build())
+        .setSessionId(newSessionId)
+        .build()
+
+      val response = stub.executePlan(newSessionExecuteRequest)
+      val graphId2 =
+        response.next().getPipelineCommandResult.getCreateDataflowGraphResult.getDataflowGraphId
+
+      // Register a different dataset in second session
+      val session2DefineRequest = buildExecutePlanRequest(
+        buildPlanFromPipelineCommand(
+          PipelineCommand
+            .newBuilder()
+            .setDefineDataset(
+              DefineDataset
+                .newBuilder()
+                .setDataflowGraphId(graphId2)
+                .setDatasetName("session2_table")
+                .setDatasetType(DatasetType.MATERIALIZED_VIEW))
+            .build())).toBuilder
+        .setUserContext(proto.UserContext
+          .newBuilder()
+          .setUserId(newSessionUserId)
+          .build())
+        .setSessionId(newSessionId)
+        .build()
+
+      stub.executePlan(session2DefineRequest).next()
+
+      // Verify session isolation - each session should only see its own graphs
+      val newSessionHolder = SparkConnectService.sessionManager
+        .getIsolatedSessionIfPresent(SessionKey(newSessionUserId, newSessionId))
+        .getOrElse(throw new RuntimeException("New session not found"))
+
+      val defaultSessionGraphs = getDefaultSessionHolder.getAllDataflowGraphs
+      val newSessionGraphs = newSessionHolder.getAllDataflowGraphs
+
+      assert(defaultSessionGraphs.size == 1)
+      assert(newSessionGraphs.size == 1)
+
+      assert(
+        defaultSessionGraphs.head.toDataflowGraph.tables
+          .exists(_.identifier.table == "session1_table"),
+        "Session 1 should have its own table")
+      assert(
+        newSessionGraphs.head.toDataflowGraph.tables
+          .exists(_.identifier.table == "session2_table"),
+        "Session 2 should have its own table")
+    }
+  }
+
+  test("dataflow graphs are cleaned up when session is closed") {
+    val testUserId = "test_user"
+    val testSessionId = UUID.randomUUID().toString
+
+    // Create a session and dataflow graph
+    withRawBlockingStub { implicit stub =>
+      val createGraphRequest = buildExecutePlanRequest(
+        buildCreateDataflowGraphPlan(
+          proto.PipelineCommand.CreateDataflowGraph
+            .newBuilder()
+            .setDefaultCatalog("spark_catalog")
+            .setDefaultDatabase("default")
+            .build())).toBuilder
+        .setUserContext(proto.UserContext
+          .newBuilder()
+          .setUserId(testUserId)
+          .build())
+        .setSessionId(testSessionId)
+        .build()
+
+      val response = stub.executePlan(createGraphRequest)
+      val graphId =
+        response.next().getPipelineCommandResult.getCreateDataflowGraphResult.getDataflowGraphId
+
+      // Register a dataset
+      val defineRequest = buildExecutePlanRequest(
+        buildPlanFromPipelineCommand(
+          PipelineCommand
+            .newBuilder()
+            .setDefineDataset(
+              DefineDataset
+                .newBuilder()
+                .setDataflowGraphId(graphId)
+                .setDatasetName("test_table")
+                .setDatasetType(DatasetType.MATERIALIZED_VIEW))
+            .build())).toBuilder
+        .setUserContext(proto.UserContext
+          .newBuilder()
+          .setUserId(testUserId)
+          .build())
+        .setSessionId(testSessionId)
+        .build()
+
+      stub.executePlan(defineRequest).next()
+
+      // Verify the graph exists
+      val sessionHolder = SparkConnectService.sessionManager
+        .getIsolatedSessionIfPresent(SessionKey(testUserId, testSessionId))
+        .get
+
+      val graphsBefore = sessionHolder.getAllDataflowGraphs
+      assert(graphsBefore.size == 1)
+
+      // Close the session
+      SparkConnectService.sessionManager.closeSession(SessionKey(testUserId, testSessionId))
+
+      // Verify the session is no longer available
+      val sessionAfterClose = SparkConnectService.sessionManager
+        .getIsolatedSessionIfPresent(SessionKey(testUserId, testSessionId))
+
+      assert(sessionAfterClose.isEmpty, "Session should be cleaned up after close")
+      // Verify the graph is removed
+      val graphsAfter = sessionHolder.getAllDataflowGraphs
+      assert(graphsAfter.isEmpty, "Graph should be removed after session close")
+    }
+  }
+
+  test("multiple dataflow graphs can exist in the same session") {
+    withRawBlockingStub { implicit stub =>
+      // Create two dataflow graphs in the same session
+      val graphId1 = createDataflowGraph
+      val graphId2 = createDataflowGraph
+
+      // Register datasets in both graphs
+      sendPlan(
+        buildPlanFromPipelineCommand(
+          PipelineCommand
+            .newBuilder()
+            .setDefineDataset(
+              DefineDataset
+                .newBuilder()
+                .setDataflowGraphId(graphId1)
+                .setDatasetName("graph1_table")
+                .setDatasetType(DatasetType.MATERIALIZED_VIEW))
+            .build()))
+
+      sendPlan(
+        buildPlanFromPipelineCommand(
+          PipelineCommand
+            .newBuilder()
+            .setDefineDataset(
+              DefineDataset
+                .newBuilder()
+                .setDataflowGraphId(graphId2)
+                .setDatasetName("graph2_table")
+                .setDatasetType(DatasetType.MATERIALIZED_VIEW))
+            .build()))
+
+      // Verify both graphs exist in the session
+      val sessionHolder = getDefaultSessionHolder
+      val graph1 = sessionHolder.getDataflowGraph(graphId1).getOrElse {
+        fail(s"Graph with ID $graphId1 not found in session")
+      }
+      val graph2 = sessionHolder.getDataflowGraph(graphId2).getOrElse {
+        fail(s"Graph with ID $graphId2 not found in session")
+      }

Review Comment:
   nit: just call `getDataflowGraphOrThrow`



##########
sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/SparkDeclarativePipelinesServerTest.scala:
##########
@@ -23,20 +23,29 @@ import org.apache.spark.connect.{proto => sc}
 import org.apache.spark.connect.proto.{PipelineCommand, PipelineEvent}
 import org.apache.spark.sql.connect.{SparkConnectServerTest, SparkConnectTestUtils}
 import org.apache.spark.sql.connect.planner.SparkConnectPlanner
-import org.apache.spark.sql.connect.service.{SessionKey, SparkConnectService}
+import org.apache.spark.sql.connect.service.{SessionHolder, SessionKey, SparkConnectService}
 import org.apache.spark.sql.pipelines.utils.PipelineTest
 
 class SparkDeclarativePipelinesServerTest extends SparkConnectServerTest {
 
   override def afterEach(): Unit = {
     SparkConnectService.sessionManager
       .getIsolatedSessionIfPresent(SessionKey(defaultUserId, defaultSessionId))
-      .foreach(_.removeAllPipelineExecutions())
-    DataflowGraphRegistry.dropAllDataflowGraphs()
+      .foreach(s => {
+        s.removeAllPipelineExecutions()
+        s.dropAllDataflowGraphs()
+      })
     PipelineTest.cleanupMetastore(spark)
     super.afterEach()
   }
 
+  // Helper method to get the session holder
+  protected def getDefaultSessionHolder: SessionHolder = {
+    SparkConnectService.sessionManager
+      .getIsolatedSessionIfPresent(SessionKey(defaultUserId, defaultSessionId))
+      .getOrElse(throw new RuntimeException("Session not found"))

Review Comment:
   nit: just call `getIsolatedSession`



##########
sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/PythonPipelineSuite.scala:
##########
@@ -78,8 +83,16 @@ class PythonPipelineSuite
       throw new RuntimeException(
         s"Python process failed with exit code $exitCode. Output: ${output.mkString("\n")}")
     }
+    val activateSessions = SparkConnectService.sessionManager.listActiveSessions
 
-    val dataflowGraphContexts = DataflowGraphRegistry.getAllDataflowGraphs
+    // get the session holder by finding the session with the custom UUID set in the conf
+    val sessionHolder = activateSessions
+      .map(info => SparkConnectService.sessionManager.getIsolatedSessionIfPresent(info.key).get)

Review Comment:
   `getIsolatedSession()` instead of `getIsolatedSessionIfPresent(...).get`



##########
sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/PythonPipelineSuite.scala:
##########
@@ -42,6 +44,8 @@ class PythonPipelineSuite
 
   def buildGraph(pythonText: String): DataflowGraph = {
     val indentedPythonText = pythonText.linesIterator.map("    " + _).mkString("\n")
+    // create a unique identifier to allow identifying the session and dataflow graph
+    val identifier = UUID.randomUUID().toString

Review Comment:
   nit: rename to something more descriptive like `customSessionIdentifier`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-52432][SDP][SQL] Scope DataflowGraphRegistry to Session [spark]

Posted by "sryza (via GitHub)" <gi...@apache.org>.
sryza closed pull request #51544: [SPARK-52432][SDP][SQL] Scope DataflowGraphRegistry to Session
URL: https://github.com/apache/spark/pull/51544


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-52432][SDP][SQL] Scope DataflowGraphRegistry to Session [spark]

Posted by "JiaqiWang18 (via GitHub)" <gi...@apache.org>.
JiaqiWang18 commented on PR #51544:
URL: https://github.com/apache/spark/pull/51544#issuecomment-3090023485

   @AnishMahto @sryza 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org