You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2022/02/28 09:22:00 UTC
[jira] [Commented] (IMPALA-11112) Impala can't resolve json tables created by Hive
[ https://issues.apache.org/jira/browse/IMPALA-11112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17498792#comment-17498792 ]
ASF subversion and git services commented on IMPALA-11112:
----------------------------------------------------------
Commit 7942a8ca2316357296fe351a40b6a87b9de769c8 in impala's branch refs/heads/master from pranav.lodha
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=7942a8c ]
IMPALA-11112: Impala can't resolve json tables created by Hive
Impala was using wrong json serde string which was causing the
subsequent errors. The error got resolved on correcting the json
serde string. Now impala can resolve json tables created by hive
and also hive can resolve json tables created in impala.
End-to-end tests are also included in
tests/metadata/test_hms_integration.py.
Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5
Reviewed-on: http://gerrit.cloudera.org:8080/18263
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Impala can't resolve json tables created by Hive
> ------------------------------------------------
>
> Key: IMPALA-11112
> URL: https://issues.apache.org/jira/browse/IMPALA-11112
> Project: IMPALA
> Issue Type: Sub-task
> Components: Catalog
> Affects Versions: Impala 4.1.0
> Reporter: Quanlong Huang
> Assignee: Pranav Yogi Lodha
> Priority: Critical
>
> JSON tables created by Hive can't be resolved in Impala. For instance, creating a JSON table in Hive using beeline:
> {code:java}
> $ beeline -u "jdbc:hive2://localhost:11050"
> 0: jdbc:hive2://localhost:11050> create table my_json_tbl (id int, name string) stored as jsonfile;{code}
> Then describe it in Impala using impala-shell:
> {code:java}
> $ bin/impala-shell.sh
> [localhost:21050] default> invalidate metadata my_json_tbl;
> [localhost:21050] default> desc my_json_tbl;
> Query: describe my_json_tbl
> ERROR: AnalysisException: Failed to load metadata for table: default.my_json_tbl
> CAUSED BY: InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported.
> CAUSED BY: TableLoadingException: Failed to load metadata for table: default.my_json_tbl
> CAUSED BY: InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported. {code}
> There are more details in logs
> {code:java}
> I0209 14:59:55.419886 10083 jni-util.cc:286] 914bfb645cab120b:443e5ec500000000] org.apache.impala.common.AnalysisException: Failed to load metadata for table: default.my_json_tbl
> CAUSED BY: InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported.
> at org.apache.impala.analysis.DescribeTableStmt.analyze(DescribeTableStmt.java:112)
> at org.apache.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:497)
> at org.apache.impala.analysis.AnalysisContext.analyzeAndAuthorize(AnalysisContext.java:461)
> at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:1677)
> at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1643)
> at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1613)
> at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:164)
> Caused by: org.apache.impala.catalog.TableLoadingException: Failed to load metadata for table: default.my_json_tbl
> CAUSED BY: InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported.
> at org.apache.impala.catalog.IncompleteTable.loadFromThrift(IncompleteTable.java:123)
> at org.apache.impala.catalog.Table.fromThrift(Table.java:513)
> at org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:469)
> at org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:324)
> at org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:253)
> at org.apache.impala.service.FeCatalogManager$CatalogdImpl.updateCatalogCache(FeCatalogManager.java:114)
> at org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:410)
> at org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:185)
> at ========.<Remote stack trace on catalogd>: org.apache.impala.catalog.TableLoadingException: Failed to load metadata for table: default.my_json_tbl
> at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1265)
> at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1147)
> at org.apache.impala.catalog.TableLoader.load(TableLoader.java:144)
> at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:245)
> at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:242)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.impala.catalog.HdfsStorageDescriptor$InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported.
> at org.apache.impala.catalog.HdfsStorageDescriptor.fromStorageDescriptor(HdfsStorageDescriptor.java:216)
> at org.apache.impala.catalog.HdfsTable.setPrototypePartition(HdfsTable.java:1139)
> at org.apache.impala.catalog.HdfsTable.initializePartitionMetadata(HdfsTable.java:656)
> at org.apache.impala.catalog.HdfsTable.loadAllPartitions(HdfsTable.java:678)
> at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1253)
> ... 8 more {code}
> We also need to fix the issue that Hive can't resolve JSON tables created by Impala.
> {code}
> Error while compiling statement: FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.ddl.DDLTask. java.lang.ClassNotFoundException Class org.apache.hadoop.hive.serde2.lazy.JsonSerDe not found
> {code}
> Probably change a SerDe that Hive recognizes.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org