You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "aglinxinyuan (via GitHub)" <gi...@apache.org> on 2026/07/17 07:39:08 UTC

[I] Workflow execution fails on Windows: Iceberg local storage requires winutils.exe [texera]

aglinxinyuan opened a new issue, #6487:
URL: https://github.com/apache/texera/issues/6487

   ## Describe the bug
   
   On a Windows development machine, every workflow execution fails immediately at execution start, before any operator runs:
   
   ```
   java.lang.RuntimeException: java.io.FileNotFoundException: Hadoop bin directory does not exist:
   C:\...\Coursier\cache\...\org\apache\hadoop\bin -see https://wiki.apache.org/hadoop/WindowsProblems
   ```
   
   ## Root cause
   
   Iceberg's `HadoopFileIO` (used for the local warehouse by both the default `postgres` catalog and the `hadoop` catalog) writes through Hadoop's `LocalFileSystem`. On Windows, hadoop-common applies POSIX permissions on **every** file/directory creation by shelling out to `%HADOOP_HOME%\bin\winutils.exe`, and there is no configuration flag to disable this:
   
   ```
   ExecutionStatsService                    (create runtime-stats Iceberg table)
    └─ IcebergUtil.createTable
        └─ JdbcTableOperations.doCommit → HadoopFileIO
            └─ RawLocalFileSystem.mkdirs / create
                └─ RawLocalFileSystem.setPermission     ← on every file/dir creation
                    └─ Shell.getSetPermissionCommand    ← requires winutils.exe chmod
                        └─ FileNotFoundException: Hadoop bin directory does not exist
   ```
   
   Hadoop only skips the winutils path when its native library (`hadoop.dll`) is loadable, which also requires a native Hadoop installation. Neither ships with Texera, so a stock checkout cannot run workflows on Windows.
   
   | | |
   |---|---|
   | Affected storage paths | workflow results, runtime statistics, console messages (all Iceberg local-warehouse writes) |
   | Catalog types affected | `postgres` (default), `hadoop` |
   | Not affected | `rest` catalog (S3FileIO), Linux/macOS hosts |
   | Hadoop version | hadoop-common 3.5.0 (`RawLocalFileSystem.setPermission`) |
   
   ## To Reproduce
   
   1. On Windows, without `HADOOP_HOME` pointing to a native Hadoop installation (the default for a fresh checkout), start Texera and run any workflow.
   2. Execution fails immediately with the stack trace above.
   
   ## Expected behavior
   
   Workflows run on a stock Windows checkout. POSIX permission bits carry no meaning on NTFS, so Hadoop's chmod-on-create can be safely skipped there instead of failing the execution.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Workflow execution fails on Windows: Iceberg local storage requires winutils.exe [texera]

Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan closed issue #6487: Workflow execution fails on Windows: Iceberg local storage requires winutils.exe
URL: https://github.com/apache/texera/issues/6487


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Workflow execution fails on Windows: Iceberg local storage requires winutils.exe [texera]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6487:
URL: https://github.com/apache/texera/issues/6487#issuecomment-5000250150

   <!-- texera:template-compliance -->
   👋 Thanks for opening this issue, @aglinxinyuan!
   
   It looks like the issue description doesn't quite follow our template yet:
   
   - This issue doesn't appear to use one of our templates (Bug, Feature, or Task). Please open it using a template so the required details are captured.
   
   Filling out the template helps reviewers understand and triage your contribution faster. Please edit the description to complete it. This message will disappear automatically once the template is followed.
   
   You can find the template prompts by editing the description, or see [CONTRIBUTING.md](https://github.com/apache/texera/blob/main/CONTRIBUTING.md) for the full contribution flow.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org