You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "sryza (via GitHub)" <gi...@apache.org> on 2025/05/21 03:39:59 UTC

[PR] [SPARK-52238] Python client for Declarative Pipelines [spark]

sryza opened a new pull request, #50963:
URL: https://github.com/apache/spark/pull/50963

   ### What changes were proposed in this pull request?
   
   Adds the Python client for Declarative Pipelines. This implements the command line interface and Python APIs described in the [Declarative Pipelines SPIP](https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0#heading=h.9g6a5f8v6xig).
   
   #### Python API for defining pipeline graph elements
   
   The Python API consists of these APIs for defining flows and datasets in a pipeline dataflow graph (see their docstring for more details):
   - `create_streaming_table`
   - `@append_flow`
   - `@materialized_view`
   - `@table`
   - `@temporary_view`
   
   Example file of definitions:
   
   ```python
   from pyspark.sql import SparkSession
   from pyspark.sql import pipelines as sdp
   
   spark = SparkSession.getActiveSession()
   
   @sdp.materialized_view
   def baby_names_raw():
       return (
           spark.read.option("header", "true").csv("babynames.csv")
           .withColumnRenamed("First Name", "First_Name")
       )
   ```
   
   #### Command line interface
   
   The CLI is implemented as a Spark Connect client. It enables launching runs of declarative pipelines. It accepts a YAML spec, which specifies where on the local filesystem to look for the Python and SQL files that contain the definitions of the flows and datasets that make up the pipeline dataflow graph.
   
   Example usage:
   ```
   bin/spark-pipelines run --remote sc://localhost --spec pipeline.yml
   ```
   
   Example output:
   
   ```
   Loading pipeline spec from pipeline.yaml...
   Creating Spark session...
   Creating dataflow graph...
   Registering graph elements...
   Loading definitions. Root directory: ..
   Found 1 files matching glob 'transformations/**/*.py'
   Importing transformations/baby_names_raw.py...
   Found 1 files matching glob 'transformations/**/*.sql'
   Registering SQL file transformations/baby_names_prepared.sql...
   Starting run...
   Starting execution...
   2025-05-20T15:08:01.395Z: Flow `spark_catalog`.`default`.`baby_names_raw` is QUEUED.
   2025-05-20T15:08:01.398Z: Flow `spark_catalog`.`default`.`baby_names_prepared` is QUEUED.
   2025-05-20T15:08:01.402Z: Flow 'spark_catalog.default.baby_names_raw' is PLANNING.
   2025-05-20T15:08:01.403Z: Flow `spark_catalog`.`default`.`baby_names_raw` is STARTING.
   2025-05-20T15:08:01.404Z: Flow `spark_catalog`.`default`.`baby_names_raw` is RUNNING.
   2025-05-20T15:08:03.096Z: Flow 'spark_catalog.default.baby_names_raw' has COMPLETED.
   2025-05-20T15:08:03.422Z: Flow 'spark_catalog.default.baby_names_prepared' is PLANNING.
   2025-05-20T15:08:03.422Z: Flow `spark_catalog`.`default`.`baby_names_prepared` is STARTING.
   2025-05-20T15:08:03.422Z: Flow `spark_catalog`.`default`.`baby_names_prepared` is RUNNING.
   2025-05-20T15:08:03.875Z: Flow 'spark_catalog.default.baby_names_prepared' has COMPLETED.
   2025-05-20T15:08:05.492Z: Run has COMPLETED.
   ```
   
   ### Why are the changes needed?
   
   In order to implement Declarative Pipelines, as described in the SPIP.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No previous behavior is changed, but new behavior is introduced.
   
   ### How was this patch tested?
   
   #### Unit testing
   
   Includes unit tests for:
   - Python API error cases – test_decorators.py
   - Command line functionality – test_cli.py
   - The harness for registering graph elements while evaluating pipeline definition Python files – test_graph_element_registry.py
   - Code for blocking execution and analysis within decorated query functions – test_block_connect_access.py
   
   Note that, once the backend is wired up, we will submit additional unit tests that cover end-to-end pipeline execution with Python.
   
   #### CLI testing
   
   With the Declarative Pipelines Spark Connect backend (coming in a future PR), I ran the CLI and confirmed that it executed a pipeline as expected.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   <!--
   If generative AI tooling has been used in the process of authoring this patch, please include the
   phrase: 'Generated-by: ' followed by the name of the tool and its version.
   If no, write 'No'.
   Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52238][PYTHON] Python client for Declarative Pipelines [spark]

Posted by "sryza (via GitHub)" <gi...@apache.org>.
sryza commented on PR #50963:
URL: https://github.com/apache/spark/pull/50963#issuecomment-2898138585

   @zhengruifeng this initial implementation is just for Connect. Connect is more straightforward to support, because Connect DataFrames are lazier than classic DataFrames. This means we can evaluate the user's decorated query function immediately rather than call back after all upstream datasets have been resolved.
   
   However, it's designed in a way that can support classic in the future – by implementing a `GraphElementRegistry` that registers graph elements over Py4J instead of Connect.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52238][PYTHON] Python client for Declarative Pipelines [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #50963:
URL: https://github.com/apache/spark/pull/50963#discussion_r2134851252


##########
python/pyspark/pipelines/cli.py:
##########
@@ -0,0 +1,276 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Implementation of spark-pipelines CLI.
+
+Example usage:
+    $ bin/spark-pipelines run --spec /path/to/pipeline.yaml
+"""
+from contextlib import contextmanager
+import argparse
+import importlib.util
+import os
+import yaml

Review Comment:
   There are a bunch of scheduled jobs that we don't run for PR builders at https://github.com/apache/spark/actions, e.g., JDK 17, 21, Maven, MacOS etc.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52238][PYTHON] Python client for Declarative Pipelines [spark]

Posted by "sryza (via GitHub)" <gi...@apache.org>.
sryza commented on code in PR #50963:
URL: https://github.com/apache/spark/pull/50963#discussion_r2114904322


##########
dev/sparktestsupport/modules.py:
##########
@@ -1501,6 +1501,18 @@ def __hash__(self):
     ],
 )
 
+pyspark_pipelines = Module(
+    name="pyspark-pipelines",
+    dependencies=[],

Review Comment:
   👍 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52238][PYTHON] Python client for Declarative Pipelines [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #50963:
URL: https://github.com/apache/spark/pull/50963#discussion_r2134845644


##########
python/pyspark/pipelines/cli.py:
##########
@@ -0,0 +1,276 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Implementation of spark-pipelines CLI.
+
+Example usage:
+    $ bin/spark-pipelines run --spec /path/to/pipeline.yaml
+"""
+from contextlib import contextmanager
+import argparse
+import importlib.util
+import os
+import yaml

Review Comment:
   Seems like the PySpark tests fail if `yaml` is not installed (https://github.com/apache/spark/actions/runs/15516895252/job/43685059103). I think we should skip the tests if yaml is not found.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52238][PYTHON] Python client for Declarative Pipelines [spark]

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #50963:
URL: https://github.com/apache/spark/pull/50963#discussion_r2099707879


##########
python/pyspark/sql/tests/pipelines/test_block_connect_access.py:
##########
@@ -0,0 +1,57 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import unittest
+
+from pyspark.errors import PySparkException
+from pyspark.testing.connectutils import ReusedConnectTestCase
+from pyspark.sql.pipelines.block_connect_access import block_spark_connect_execution_and_analysis
+
+
+class BlockSparkConnectAccessTests(ReusedConnectTestCase):

Review Comment:
   Do we plan to also test this file in classic mode?



##########
python/pyspark/sql/tests/pipelines/test_graph_element_registry.py:
##########
@@ -0,0 +1,128 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import unittest
+
+from pyspark.errors import PySparkException
+from pyspark.sql.pipelines.graph_element_registry import graph_element_registration_context
+from pyspark.sql import pipelines as sdp
+from pyspark.sql import SparkSession
+from pyspark.sql.tests.pipelines.local_graph_element_registry import LocalGraphElementRegistry
+
+
+class GraphElementRegistryTest(unittest.TestCase):
+    def test_graph_element_registry(self):
+        spark = SparkSession.builder.getOrCreate()

Review Comment:
   why not reusing the ReusedSQLTestCase (for classic) and ReusedConnectTestCase (for connect)?



##########
python/pyspark/sql/tests/pipelines/test_block_connect_access.py:
##########
@@ -0,0 +1,57 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import unittest
+
+from pyspark.errors import PySparkException
+from pyspark.testing.connectutils import ReusedConnectTestCase
+from pyspark.sql.pipelines.block_connect_access import block_spark_connect_execution_and_analysis
+
+
+class BlockSparkConnectAccessTests(ReusedConnectTestCase):

Review Comment:
   new tests should be registered in `dev/sparktestsupport/modules.py`, otherwise they are skipped



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52238][PYTHON] Python client for Declarative Pipelines [spark]

Posted by "sryza (via GitHub)" <gi...@apache.org>.
sryza commented on code in PR #50963:
URL: https://github.com/apache/spark/pull/50963#discussion_r2100494983


##########
python/pyspark/sql/tests/pipelines/test_block_connect_access.py:
##########
@@ -0,0 +1,57 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import unittest
+
+from pyspark.errors import PySparkException
+from pyspark.testing.connectutils import ReusedConnectTestCase
+from pyspark.sql.pipelines.block_connect_access import block_spark_connect_execution_and_analysis
+
+
+class BlockSparkConnectAccessTests(ReusedConnectTestCase):

Review Comment:
   This should only be tested in connect mode – do I need to add something to the file to set that up?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52238][PYTHON] Python client for Declarative Pipelines [spark]

Posted by "sryza (via GitHub)" <gi...@apache.org>.
sryza commented on code in PR #50963:
URL: https://github.com/apache/spark/pull/50963#discussion_r2134849954


##########
python/pyspark/pipelines/cli.py:
##########
@@ -0,0 +1,276 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Implementation of spark-pipelines CLI.
+
+Example usage:
+    $ bin/spark-pipelines run --spec /path/to/pipeline.yaml
+"""
+from contextlib import contextmanager
+import argparse
+import importlib.util
+import os
+import yaml

Review Comment:
   Is this what we do for other dependencies? I'd be worried that, if we accidentally make a change to Spark CI that avoids installing pyyaml when we'd otherwise expect it to be installed, then the tests could get broken and we wouldn't find out.
   
   I could alternatively help trace down why it isn't installed in that situation and fix it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52238][PYTHON] Python client for Declarative Pipelines [spark]

Posted by "sryza (via GitHub)" <gi...@apache.org>.
sryza commented on code in PR #50963:
URL: https://github.com/apache/spark/pull/50963#discussion_r2110038278


##########
python/pyspark/sql/tests/pipelines/test_graph_element_registry.py:
##########
@@ -0,0 +1,128 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import unittest
+
+from pyspark.errors import PySparkException
+from pyspark.sql.pipelines.graph_element_registry import graph_element_registration_context
+from pyspark.sql import pipelines as sdp
+from pyspark.sql import SparkSession
+from pyspark.sql.tests.pipelines.local_graph_element_registry import LocalGraphElementRegistry
+
+
+class GraphElementRegistryTest(unittest.TestCase):
+    def test_graph_element_registry(self):
+        spark = SparkSession.builder.getOrCreate()

Review Comment:
   Ah – this test actually doesn't need a `SparkSession`. Updating it to take it out.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52238][PYTHON] Python client for Declarative Pipelines [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #50963:
URL: https://github.com/apache/spark/pull/50963#discussion_r2134851078


##########
python/pyspark/pipelines/cli.py:
##########
@@ -0,0 +1,276 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Implementation of spark-pipelines CLI.
+
+Example usage:
+    $ bin/spark-pipelines run --spec /path/to/pipeline.yaml
+"""
+from contextlib import contextmanager
+import argparse
+import importlib.util
+import os
+import yaml

Review Comment:
   Yeah we do .. the tests work without any dependency basically. That build is a scheduled build dedicated for that (to test without any dependencies).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52238][PYTHON] Python client for Declarative Pipelines [spark]

Posted by "sryza (via GitHub)" <gi...@apache.org>.
sryza closed pull request #50963: [SPARK-52238][PYTHON] Python client for Declarative Pipelines
URL: https://github.com/apache/spark/pull/50963


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52238][PYTHON] Python client for Declarative Pipelines [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #50963:
URL: https://github.com/apache/spark/pull/50963#discussion_r2114880598


##########
dev/sparktestsupport/modules.py:
##########
@@ -1501,6 +1501,18 @@ def __hash__(self):
     ],
 )
 
+pyspark_pipelines = Module(
+    name="pyspark-pipelines",
+    dependencies=[],

Review Comment:
   Let's add dependencies same as `pyspark_sql`. If you have a Spark Connect tests, it has to be `pyspark_connect`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-52238][PYTHON] Python client for Declarative Pipelines [spark]

Posted by "sryza (via GitHub)" <gi...@apache.org>.
sryza commented on code in PR #50963:
URL: https://github.com/apache/spark/pull/50963#discussion_r2136103963


##########
python/pyspark/pipelines/cli.py:
##########
@@ -0,0 +1,276 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Implementation of spark-pipelines CLI.
+
+Example usage:
+    $ bin/spark-pipelines run --spec /path/to/pipeline.yaml
+"""
+from contextlib import contextmanager
+import argparse
+import importlib.util
+import os
+import yaml

Review Comment:
   Got it – I'll make this change. Any chance you have a reference to how we skip tests for other missing depndencies, so I can make add something consistent?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org