You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hop.apache.org by "vbhanuchander-lang (via GitHub)" <gi...@apache.org> on 2026/08/29 16:45:54 UTC
[PR] Issue #8138 : Do not send a path option when writing with file_format=jdbc (hop)
vbhanuchander-lang opened a new pull request, #8176:
URL: https://github.com/apache/hop/pull/8176
Fixes #8138
### The defect
`SparkFileIoSupport.writeDataset` always called the one-argument `DataFrameWriter.save(path)`:
```java
writer.save(path); // SparkFileIoSupport.java:98
```
That overload stores its argument as the `path` writer option before saving. Spark's JDBC provider forwards every option it does not recognise to the driver as a **connection property**, so a JDBC sink — which has no path — receives one anyway.
Postgres and MySQL ignore connection properties they do not know, so this has been invisible against them. Teradata validates them and fails the write:
```
[Teradata JDBC Driver] [Error 1536] [SQLState HY000] Invalid connection parameter name path
```
`SparkFileOutputHandler` compounded it: it rejected the transform outright when no file path was configured, so the user had to invent a path for a sink that has none — and that invented value is precisely what got sent to the driver.
### The fix
`isPathless(format)` names the formats whose sink is addressed entirely through options rather than a path. Writing one of those now calls the no-argument `save()`, so nothing enters the writer options:
```java
if (pathless) {
writer.save();
} else {
writer.save(path);
}
```
Only `jdbc` is in that set today. It is a named set rather than an `equals` check so other option-addressed sinks can join it, but I deliberately did not add `kafka`/`console`/`noop` speculatively — `jdbc` is the one with a reported failure.
Also: the error message for a pathless format no longer quotes a path that played no part in the write, and `SparkFileOutput` no longer demands a file path when the format is pathless.
`SparkLakeTableSupport` shares `writeDataset` but only ever passes `delta` or `iceberg`, both path-based, so it is unaffected.
### Tests
The obvious test — write to an embedded database and see whether it complains — **does not work, and it is worth saying why.** I tried it with H2 first and it passed *with the fix reverted*: H2, like Postgres and MySQL, silently ignores connection properties it does not recognise. Only a strict driver such as Teradata's surfaces this, and depending on one in a unit test is not an option.
So the tests assert the thing that is actually wrong — what Spark hands the driver. `CapturingJdbcDriver` records the `Properties` of every connection and delegates to H2, and the test asserts `path` is not among them. Reverting the fix fails it with exactly the right message:
```
Spark sent 'path' as a connection property: [path] ==> expected: <false> but was: <true>
```
`CapturingJdbcDriver` is top-level rather than nested on purpose: Spark's `DriverRegistry` matches a registered driver by *canonical* name, which for a nested class does not equal the binary name it is loaded by, and the lookup fails with an internal error.
H2 is added in test scope only, with the version imported from `hop-libs-jdbc` the same way `plugins/actions/waitforsql` does it.
Verified locally on JDK 21: `hop-engines-spark` **189 tests, 0 failures**. `spotless:check` and `apache-rat:check` clean.
### Note on scope
I could not reproduce against Teradata itself — no access to that driver or a Vantage instance. What is verified here is that Hop no longer sends `path`, which is the cause identified in the report; @tmortada, confirmation against your Teradata setup would be worth having before this is merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Issue #8138 : Do not send a path option when writing with file_format=jdbc (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva merged PR #8176:
URL: https://github.com/apache/hop/pull/8176
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Issue #8138 : Do not send a path option when writing with file_format=jdbc (hop)
Posted by "tmortada (via GitHub)" <gi...@apache.org>.
tmortada commented on PR #8176:
URL: https://github.com/apache/hop/pull/8176#issuecomment-5464351538
@vbhanuchander-lang
Confirmed against a real Teradata Vantage Express instance 20.00.28.81 Thanks!
### Testing method
Rather than a full build, I patched just the two changed files ( SparkFileIoSupport. java, SparkFileOutputHandler. java ), compiled them with javac against the classpath already present in a apache/hop: 2.19.0 -based container, and injected the two resulting .class files into the running hop-engines-spark-2.19.0. jar with jar -u
I verified with:
* <file_path>/tmp/hop-bug-repro/category_sales_summary_pg</file_path>
* <file_path />
* Removed
The 3 scenarios work
### Log Information
2026/08/29 18:34:02 - General - Handled Spark Lake Table Input : Read Category Sales Summary from Iceberg format=iceberg mode=TABLE table=nessie.retail.category_sales_summary timeTravel=NONE columns=[product_category, total_revenue, transaction_count, avg_sale_value, total_units_sold]
2026/08/29 18:34:04 - General - Handled Spark File Output : Write JDBC Direct to Teradata format=jdbc mode=Overwrite path=/tmp/hop-bug-repro/category_sales_summary
2026/08/29 18:34:05 - bug_repro_iceberg_to_teradata_direct - Spark pipeline finished, result row count=0
HopRun exit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org