You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "AnuragKDwivedi (via GitHub)" <gi...@apache.org> on 2026/06/24 08:41:02 UTC

[PR] [SPARK-57295][SQL] Extend whitespace-only path validation consistency to direct file-path queries [spark]

AnuragKDwivedi opened a new pull request, #56732:
URL: https://github.com/apache/spark/pull/56732

   ## Description
   
   This is a follow-up to PR #56356, which improved validation consistency for namespace locations by treating whitespace-only values as invalid locations.
   
   ### What changes were proposed in this pull request?
   
   This PR extends the same validation behavior to direct file-path queries.
   
   Currently, direct file-path validation checks only for empty strings using `isEmpty`. Consequently, whitespace-only paths such as `" "`, `"\t"`, and `"\n"` are not recognized as empty during analysis and may fail later with datasource-specific errors.
   
   This PR updates the validation to use `SparkStringUtils.isBlank(...)`, ensuring that whitespace-only paths are treated as invalid and consistently fail with the standard `INVALID_EMPTY_LOCATION` error.
   
   By doing so, the change aligns direct file-path validation with the existing namespace location validation logic and improves consistency across Spark SQL location handling.
   
   ### Why are the changes needed?
   
   Currently, validation behavior differs depending on the type of location being processed:
   
   * Empty paths (`""`) are rejected during analysis with `INVALID_EMPTY_LOCATION`.
   * Whitespace-only paths (`" "`, `"\t"`, `"\n"`) may bypass analysis-time validation and fail later with datasource-specific errors.
   
   Using `SparkStringUtils.isBlank(...)` ensures consistent handling of all blank path values across Spark SQL.
   
   ### Does this PR introduce any user-facing change?
   
   Yes.
   
   Whitespace-only direct file paths are now rejected during analysis with `INVALID_EMPTY_LOCATION`, providing behavior consistent with namespace location validation.
   
   ### How was this patch tested?
   
   Added regression test coverage for blank path values, including:
   
   * `""`
   * `" "`
   * `"\t"`
   * `"\n"`
   
   and verified that they consistently fail with `INVALID_EMPTY_LOCATION`.
   
   Jira - https://issues.apache.org/jira/browse/SPARK-57295


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57295][SQL] Extend whitespace-only path validation consistency to direct file-path queries [spark]

Posted by "AnuragKDwivedi (via GitHub)" <gi...@apache.org>.
AnuragKDwivedi commented on PR #56732:
URL: https://github.com/apache/spark/pull/56732#issuecomment-4807058461

   Thanks @cloud-fan and @uros-b  for your review comments. I've updated the tests based on your suggestions. Could you please take another look? If everything looks good, I'd appreciate your help in getting this PR merged. Thanks!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57295][SQL] Extend whitespace-only path validation consistency to direct file-path queries [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #56732:
URL: https://github.com/apache/spark/pull/56732#discussion_r3466549519


##########
sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala:
##########
@@ -1697,6 +1697,15 @@ class SQLQuerySuite extends SharedSparkSession with AdaptiveSparkPlanHelper
     }
     assert(e.message.contains("Unsupported data source type for direct query on files: " +
       "org.apache.spark.sql.execution.datasources.jdbc"))
+
+    // Test for empty and whitespace-only paths
+    Seq("", " ", "\t", "\n", "\t\n", " \t ").foreach { location =>
+      val e = intercept[AnalysisException] {
+        sql(s"select id from json.`$location`")
+      }
+      assert(e.message.contains("The location name cannot be empty string"))

Review Comment:
   The new test uses `assert(e.message.contains(...))` instead of the structured `checkError(condition = "INVALID_EMPTY_LOCATION", parameters = Map("location" -> location))` form used by the sibling tests. 
   
   `contains` is brittle and can false-pass; switching to `checkError` would match sibling convention and validate the exact condition code + params. Please update accordingly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57295][SQL] Extend whitespace-only path validation consistency to direct file-path queries [spark]

Posted by "AnuragKDwivedi (via GitHub)" <gi...@apache.org>.
AnuragKDwivedi commented on code in PR #56732:
URL: https://github.com/apache/spark/pull/56732#discussion_r3479625411


##########
sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala:
##########
@@ -1697,6 +1697,15 @@ class SQLQuerySuite extends SharedSparkSession with AdaptiveSparkPlanHelper
     }
     assert(e.message.contains("Unsupported data source type for direct query on files: " +
       "org.apache.spark.sql.execution.datasources.jdbc"))
+
+    // Test for empty and whitespace-only paths
+    Seq("", " ", "\t", "\n", "\t\n", " \t ").foreach { location =>
+      val e = intercept[AnalysisException] {
+        sql(s"select id from json.`$location`")
+      }
+      assert(e.message.contains("The location name cannot be empty string"))

Review Comment:
   Thanks for the suggestion, I have updated the test now. please take a look again



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57295][SQL] Extend whitespace-only path validation consistency to direct file-path queries [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #56732: [SPARK-57295][SQL] Extend whitespace-only path validation consistency to direct file-path queries
URL: https://github.com/apache/spark/pull/56732


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57295][SQL] Extend whitespace-only path validation consistency to direct file-path queries [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #56732:
URL: https://github.com/apache/spark/pull/56732#issuecomment-4843799946

   thanks, merging to master/4.x!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57295][SQL] Extend whitespace-only path validation consistency to direct file-path queries [spark]

Posted by "AnuragKDwivedi (via GitHub)" <gi...@apache.org>.
AnuragKDwivedi commented on PR #56732:
URL: https://github.com/apache/spark/pull/56732#issuecomment-4787456328

   @cloud-fan Could you please review this PR when you get a chance?
   
   This is a follow-up to #56356 that implements the whitespace-only path validation consistency improvement you suggested earlier for direct file-path queries by using `SparkStringUtils.isBlank(...)`.
   
   Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org