You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "psendyk (via GitHub)" <gi...@apache.org> on 2023/08/03 20:01:44 UTC
[GitHub] [hudi] psendyk commented on issue #8890: [SUPPORT] Spark structured streaming ingestion into Hudi fails after an upgrade to 0.12.2
psendyk commented on issue #8890:
URL: https://github.com/apache/hudi/issues/8890#issuecomment-1664577373
I've narrowed down the bug to a specific commit -- I believe it's `218240e` from [this PR](https://github.com/apache/hudi/pull/6516). After I reverted it on the `release-0.12.2` branch, built the JAR, and ran the ingestion again, it did not run into any issues. While I don't fully understand why this was a breaking change for us, this commit touches `AbstractTableFileSystemView.getLatestFileSlicesBeforeOrOn` which is invoked by `SparkUpsertDeltaCommitPartitioner.getSmallFileCandidates` when computing small files. Fwiw, I don't see anything wrong with that commit itself but perhaps it exposed a different bug.
It seems there's already a fix for it though. 0.13.1 still runs into the same issue but when I ran a `master` build it proceeded with no issues. Is there an ETA on the 0.14.0 release? Thanks!
Also, I realize the commit I linked was released in 0.12.1 not 0.12.2... Apologies about that, we manage our JARs in our own artifact repository and it must've been mislabeled as 0.12.1 while we're actually running 0.12.0. To confirm, I've downloaded a 0.12.1 version from the Maven repo and ran into the same issue but 0.12.0 worked fine. It shouldn't have affected the reproducibility though as (as @danny0405 already mentioned) it's not a version incompatibility issue but rather a bug; even when I create the table using 0.12.1, the job still fails after a while.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org