[jira] [Created] (HUDI-7005) Flink SQL Queries on Hudi Table fail when using the hudi-aws-bundle jar

["Prabhu Joseph (Jira)" <ji...@apache.org>]

Prabhu Joseph created HUDI-7005:
-----------------------------------

             Summary: Flink SQL Queries on Hudi Table fail when using the hudi-aws-bundle jar
                 Key: HUDI-7005
                 URL: https://issues.apache.org/jira/browse/HUDI-7005
             Project: Apache Hudi
          Issue Type: Bug
          Components: flink-sql
    Affects Versions: 0.14.0
            Reporter: Prabhu Joseph


Flink SQL Queries on Hudi Table fail when using the hudi-aws-bundle jar. hudi-aws-bundle jar is needed for metastore sync into AWS Glue.

*Error stack trace from Flink SQL Client Log*
{code}
Caused by: java.lang.NoSuchMethodError: org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchema(Z)Lorg/apache/hudi/org/apache/avro/Schema;
        at org.apache.hudi.util.StreamerUtil.getTableAvroSchema(StreamerUtil.java:431)
        at org.apache.hudi.util.StreamerUtil.getLatestTableSchema(StreamerUtil.java:441)
        at org.apache.hudi.table.catalog.HoodieHiveCatalog.getTable(HoodieHiveCatalog.java:420)
        at org.apache.flink.table.catalog.CatalogManager.getUnresolvedTable(CatalogManager.java:685)
        at org.apache.flink.table.catalog.CatalogManager.dropTableInternal(CatalogManager.java:1236)
        at org.apache.flink.table.catalog.CatalogManager.dropTable(CatalogManager.java:1207)
        at org.apache.flink.table.operations.ddl.DropTableOperation.execute(DropTableOperation.java:72)
        at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1092)
        at org.apache.flink.table.gateway.service.operation.OperationExecutor.callOperation(OperationExecutor.java:556)
        at org.apache.flink.table.gateway.service.operation.OperationExecutor.executeOperation(OperationExecutor.java:444) 
        at org.apache.flink.table.gateway.service.operation.OperationExecutor.executeStatement(OperationExecutor.java:207)
        at org.apache.flink.table.gateway.service.SqlGatewayServiceImpl.lambda$executeStatement$1(SqlGatewayServiceImpl.java:212)
        at org.apache.flink.table.gateway.service.operation.OperationManager.lambda$submitOperation$1(OperationManager.java:119)
        at org.apache.flink.table.gateway.service.operation.OperationManager$Operation.lambda$run$0(OperationManager.java:258)
        ... 7 more
{code}


*Repro*

{code}
cd /usr/lib/flink/lib

wget https://repo1.maven.org/maven2/org/apache/hudi/hudi-flink1.17-bundle/0.14.0/hudi-flink1.17-bundle-0.14.0.jar

wget https://repo1.maven.org/maven2/org/apache/hudi/hudi-aws-bundle/0.14.0/hudi-aws-bundle-0.14.0.jar

flink-yarn-session -d

/usr/lib/flink/bin/sql-client.sh embedded


CREATE CATALOG glue_catalog_for_hudi WITH (
'type' = 'hudi',
'mode' = 'hms',
'table.external' = 'true',
'default-database' = 'default',
'hive.conf.dir' = '/etc/hive/conf.dist',
'catalog.path' = 's3://prabhuflinks3/HUDICDC/warehouse/'
);


USE CATALOG glue_catalog_for_hudi;

CREATE DATABASE IF NOT EXISTS flink_glue_hudi_db;

use flink_glue_hudi_db;

CREATE TABLE `glue_catalog_for_hudi`.`flink_glue_hudi_db`.`Persons_src` (
ID INT NOT NULL,
FirstName STRING,
Age STRING,
PRIMARY KEY (`ID`) NOT ENFORCED
)
WITH (
'connector' = 'hudi',
'write.tasks' = '2', 
'path' = 's3://prabhuflinks3/HUDICDC/warehouse/Persons_src',
'table.type' = 'COPY_ON_WRITE',
'read.streaming.enabled' = 'true',
'read.streaming.check-interval' = '1',
'hoodie.embed.timeline.server' = 'false',
'hive_sync.mode' = 'glue'
);

{code}


*Root Cause*

hudi-flink-bundle has relocated the org.apache.avro. into org.apache.hudi.org.apache.avro. while shading the hudi-common classes (TableSchemaResolver), whereas hudi-aws-bundle has not relocated it while shading hudi-common classes. There are two TableSchemaResolver classes found in the classpath: one from hudi-flink-bundle, which has relocated org.apache.avro, and another from hudi-aws-bundle, which has not relocated org.apache.avro. This causes NoSuchMethodError issues.

*Fix*

The fix is to relocate org.apache.avro. into org.apache.hudi.org.apache.avro. in hudi-aws-bundle. This change has fixed the above issue.





--
This message was sent by Atlassian Jira
(v8.20.10#820010)