(spark) branch branch-4.1 updated: [SPARK-54508][PYTHON] Fix `spark-pipelines` to resolve `spec` file path more robustly

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.1 by this push:
     new 4c03da6f5aab [SPARK-54508][PYTHON] Fix `spark-pipelines` to resolve `spec` file path more robustly
4c03da6f5aab is described below

commit 4c03da6f5aab82b5786a7b6acc10f468b098f367
Author: Sandy Ryza <sa...@databricks.com>
AuthorDate: Tue Nov 25 13:40:25 2025 -0800

    [SPARK-54508][PYTHON] Fix `spark-pipelines` to resolve `spec` file path more robustly
    
    ### What changes were proposed in this pull request?
    
    Fixes a bug that causes executing `spark-pipelines` with a spec file in a subdirectory to fail:
    
    ```
    spark-pipelines run --spec subdir/spark-pipelines.yml
    ```
    
    results in
    
    ```
    pyspark.errors.exceptions.connect.AnalysisException: [RUN_EMPTY_PIPELINE] Pipelines are expected to have at least one non-temporary dataset defined (tables, persisted views) but no non-temporary datasets were found in your pipeline.
    ```
    
    ### Why are the changes needed?
    
    Fix a bug.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Fixes a user-facing bug
    
    ### How was this patch tested?
    
    Ran
    ```
    spark-pipelines run --spec subdir/spark-pipelines.yml
    ```
    
    Observed it fail before the change and succeed after.
    
    With our current Pipelines CLI unit testing setup, there isn't a straightforward way to write a test for this, but I'm investigating whether we can augment it.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    No.
    
    Closes #53219 from sryza/cli-relative-path.
    
    Authored-by: Sandy Ryza <sa...@databricks.com>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
    (cherry picked from commit c61d40c88f7bb50e81b7fb98852a96b2e6296b59)
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 python/pyspark/pipelines/cli.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/python/pyspark/pipelines/cli.py b/python/pyspark/pipelines/cli.py
index 3e1c53289724..8c70c8331120 100644
--- a/python/pyspark/pipelines/cli.py
+++ b/python/pyspark/pipelines/cli.py
@@ -228,7 +228,8 @@ def register_definitions(
     - Import Python files matching the glob patterns in the spec.
     - Register SQL files matching the glob patterns in the spec.
     """
-    path = spec_path.parent
+    path = spec_path.parent.resolve()
+
     with change_dir(path):
         with graph_element_registration_context(registry):
             log_with_curr_timestamp(f"Loading definitions. Root directory: '{path}'.")
@@ -260,7 +261,7 @@ def register_definitions(
                         log_with_curr_timestamp(f"Registering SQL file {file}...")
                         with file.open("r") as f:
                             sql = f.read()
-                        file_path_relative_to_spec = file.relative_to(spec_path.parent)
+                        file_path_relative_to_spec = file.relative_to(path)
                         registry.register_sql(sql, file_path_relative_to_spec)
                     else:
                         raise PySparkException(


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org