You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Ethan Guo (Jira)" <ji...@apache.org> on 2023/01/24 19:50:00 UTC
[jira] [Updated] (HUDI-5092) Querying Hudi table throws NoSuchMethodError in Databricks runtime
[ https://issues.apache.org/jira/browse/HUDI-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ethan Guo updated HUDI-5092:
----------------------------
Fix Version/s: 0.13.1
(was: 0.12.2)
> Querying Hudi table throws NoSuchMethodError in Databricks runtime
> -------------------------------------------------------------------
>
> Key: HUDI-5092
> URL: https://issues.apache.org/jira/browse/HUDI-5092
> Project: Apache Hudi
> Issue Type: Bug
> Components: spark
> Affects Versions: 0.12.0
> Reporter: Ethan Guo
> Assignee: Ethan Guo
> Priority: Blocker
> Fix For: 0.13.1
>
>
> Originally reported by the user:
> [https://github.com/apache/hudi/issues/6137]
>
> Crux of the issue is that Databricks's DBR runtime diverges from OSS Spark, and in that case `FileStatusCache` API is very clearly divergent b/w the two.
> There are a few approaches we can take:
> # Avoid reliance on Spark's FIleStatusCache implementation altogether and rely on our own one
> # Apply more staggered approach where we first try to use Spark's FileStatusCache and if it doesn't match expected API, we fallback to our own impl
>
> Approach # 1 would actually mean that we're not sharing cache implementation w/ Spark, which in turn would entail that in some cases we might be keeping 2 instances of the same cache. Approach # 2 remediates that and allows us to only fallback in case API is not compatible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)