[PR] [FLINK-38774][table-common] Skip hidden directories when recursively listing partitions [flink]

["hulincup (via GitHub)" <gi...@apache.org>]

hulincup opened a new pull request, #28913:
URL: https://github.com/apache/flink/pull/28913

   ## What
   
   `PartitionPathUtils.listStatusRecursively` descended into every subdirectory returned by `fs.listStatus`, including hidden ones such as `_temporary`. A non-hidden child of a hidden dir (e.g. `_temporary/job-123`) sitting at partition depth was then collected with an empty partition spec, which later surfaces as a `TableException: incomplete partition spec` in the filesystem connector source.
   
   Reported in FLINK-38774 (supersedes the older FLINK-31975).
   
   ## Fix
   
   Reuse the existing `listStatusWithoutHidden` helper to filter hidden children at **every** recursion level — not only at the leaf, where `searchPartSpecAndPaths` already filtered via `isHiddenFile`. Add a null guard consistent with `listStatusWithoutHidden`'s contract (returns `null` when `fs.listStatus` returns `null`), which also removes a latent NPE in the old for-each over a null list.
   
   ```java
   if (fileStatus.isDir()) {
       FileStatus[] children = listStatusWithoutHidden(fs, fileStatus.getPath());
       if (children == null) {
           return;
       }
       for (FileStatus stat : children) {
           listStatusRecursively(fs, stat, level + 1, expectLevel, results);
       }
   }
   ```
   
   ## Test
   
   Adds a `@TempDir` regression test that creates a real partition (`date=2019-8-30/country=China`) alongside a hidden `_temporary/job-123` subtree and asserts `searchPartSpecAndPaths` returns only the real partition. Without the fix the hidden `job-123` leaf leaks in with an empty spec (2 entries); with the fix only the real partition is returned (1 entry).
   
   ## Notes
   
   - Supersedes the abandoned #27314 (stale-closed after 120 days of inactivity, no technical objection — reviewer only asked for a unit test, which is included here).
   - This is a `flink-table/flink-table-common` utility, not literally under `flink-connectors/`, but the bug manifests via the filesystem connector (tagged `Connectors / FileSystem`).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-38774][table-common] Skip hidden directories when recursively listing partitions [flink]

["flinkbot (via GitHub)" <gi...@apache.org>]

flinkbot commented on PR #28913:
URL: https://github.com/apache/flink/pull/28913#issuecomment-5175505677

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8bb6c431bdca1a3dc1b13970c99b53e0c776ecdb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8bb6c431bdca1a3dc1b13970c99b53e0c776ecdb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8bb6c431bdca1a3dc1b13970c99b53e0c776ecdb UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org