You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Quanlong Huang (Jira)" <ji...@apache.org> on 2022/02/15 06:06:00 UTC
[jira] [Updated] (IMPALA-11112) Impala can't resolve json tables created by Hive
[ https://issues.apache.org/jira/browse/IMPALA-11112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Quanlong Huang updated IMPALA-11112:
------------------------------------
Description:
JSON tables created by Hive can't be resolved in Impala. For instance, creating a JSON table in Hive using beeline:
{code:java}
$ beeline -u "jdbc:hive2://localhost:11050"
0: jdbc:hive2://localhost:11050> create table my_json_tbl (id int, name string) stored as jsonfile;{code}
Then describe it in Impala using impala-shell:
{code:java}
$ bin/impala-shell.sh
[localhost:21050] default> invalidate metadata my_json_tbl;
[localhost:21050] default> desc my_json_tbl;
Query: describe my_json_tbl
ERROR: AnalysisException: Failed to load metadata for table: default.my_json_tbl
CAUSED BY: InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported.
CAUSED BY: TableLoadingException: Failed to load metadata for table: default.my_json_tbl
CAUSED BY: InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported. {code}
There are more details in logs
{code:java}
I0209 14:59:55.419886 10083 jni-util.cc:286] 914bfb645cab120b:443e5ec500000000] org.apache.impala.common.AnalysisException: Failed to load metadata for table: default.my_json_tbl
CAUSED BY: InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported.
at org.apache.impala.analysis.DescribeTableStmt.analyze(DescribeTableStmt.java:112)
at org.apache.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:497)
at org.apache.impala.analysis.AnalysisContext.analyzeAndAuthorize(AnalysisContext.java:461)
at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:1677)
at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1643)
at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1613)
at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:164)
Caused by: org.apache.impala.catalog.TableLoadingException: Failed to load metadata for table: default.my_json_tbl
CAUSED BY: InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported.
at org.apache.impala.catalog.IncompleteTable.loadFromThrift(IncompleteTable.java:123)
at org.apache.impala.catalog.Table.fromThrift(Table.java:513)
at org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:469)
at org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:324)
at org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:253)
at org.apache.impala.service.FeCatalogManager$CatalogdImpl.updateCatalogCache(FeCatalogManager.java:114)
at org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:410)
at org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:185)
at ========.<Remote stack trace on catalogd>: org.apache.impala.catalog.TableLoadingException: Failed to load metadata for table: default.my_json_tbl
at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1265)
at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1147)
at org.apache.impala.catalog.TableLoader.load(TableLoader.java:144)
at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:245)
at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:242)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.impala.catalog.HdfsStorageDescriptor$InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported.
at org.apache.impala.catalog.HdfsStorageDescriptor.fromStorageDescriptor(HdfsStorageDescriptor.java:216)
at org.apache.impala.catalog.HdfsTable.setPrototypePartition(HdfsTable.java:1139)
at org.apache.impala.catalog.HdfsTable.initializePartitionMetadata(HdfsTable.java:656)
at org.apache.impala.catalog.HdfsTable.loadAllPartitions(HdfsTable.java:678)
at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1253)
... 8 more {code}
We also need to fix the issue that Hive can't resolve JSON tables created by Impala.
{code}
Error while compiling statement: FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.ddl.DDLTask. java.lang.ClassNotFoundException Class org.apache.hadoop.hive.serde2.lazy.JsonSerDe not found
{code}
Probably change a SerDe that Hive recognizes.
was:
JSON tables created by Hive can't be resolved in Impala. For instance, creating a JSON table in Hive using beeline:
{code:java}
$ beeline -u "jdbc:hive2://localhost:11050"
0: jdbc:hive2://localhost:11050> create table my_json_tbl (id int, name string) stored as jsonfile;{code}
Then describe it in Impala using impala-shell:
{code:java}
$ bin/impala-shell.sh
[localhost:21050] default> invalidate metadata my_json_tbl;
[localhost:21050] default> desc my_json_tbl;
Query: describe my_json_tbl
ERROR: AnalysisException: Failed to load metadata for table: default.my_json_tbl
CAUSED BY: InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported.
CAUSED BY: TableLoadingException: Failed to load metadata for table: default.my_json_tbl
CAUSED BY: InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported. {code}
There are more details in logs
{code:java}
I0209 14:59:55.419886 10083 jni-util.cc:286] 914bfb645cab120b:443e5ec500000000] org.apache.impala.common.AnalysisException: Failed to load metadata for table: default.my_json_tbl
CAUSED BY: InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported.
at org.apache.impala.analysis.DescribeTableStmt.analyze(DescribeTableStmt.java:112)
at org.apache.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:497)
at org.apache.impala.analysis.AnalysisContext.analyzeAndAuthorize(AnalysisContext.java:461)
at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:1677)
at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1643)
at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1613)
at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:164)
Caused by: org.apache.impala.catalog.TableLoadingException: Failed to load metadata for table: default.my_json_tbl
CAUSED BY: InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported.
at org.apache.impala.catalog.IncompleteTable.loadFromThrift(IncompleteTable.java:123)
at org.apache.impala.catalog.Table.fromThrift(Table.java:513)
at org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:469)
at org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:324)
at org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:253)
at org.apache.impala.service.FeCatalogManager$CatalogdImpl.updateCatalogCache(FeCatalogManager.java:114)
at org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:410)
at org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:185)
at ========.<Remote stack trace on catalogd>: org.apache.impala.catalog.TableLoadingException: Failed to load metadata for table: default.my_json_tbl
at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1265)
at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1147)
at org.apache.impala.catalog.TableLoader.load(TableLoader.java:144)
at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:245)
at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:242)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.impala.catalog.HdfsStorageDescriptor$InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported.
at org.apache.impala.catalog.HdfsStorageDescriptor.fromStorageDescriptor(HdfsStorageDescriptor.java:216)
at org.apache.impala.catalog.HdfsTable.setPrototypePartition(HdfsTable.java:1139)
at org.apache.impala.catalog.HdfsTable.initializePartitionMetadata(HdfsTable.java:656)
at org.apache.impala.catalog.HdfsTable.loadAllPartitions(HdfsTable.java:678)
at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1253)
... 8 more {code}
> Impala can't resolve json tables created by Hive
> ------------------------------------------------
>
> Key: IMPALA-11112
> URL: https://issues.apache.org/jira/browse/IMPALA-11112
> Project: IMPALA
> Issue Type: Sub-task
> Components: Catalog
> Affects Versions: Impala 4.1.0
> Reporter: Quanlong Huang
> Assignee: Pranav Yogi Lodha
> Priority: Critical
>
> JSON tables created by Hive can't be resolved in Impala. For instance, creating a JSON table in Hive using beeline:
> {code:java}
> $ beeline -u "jdbc:hive2://localhost:11050"
> 0: jdbc:hive2://localhost:11050> create table my_json_tbl (id int, name string) stored as jsonfile;{code}
> Then describe it in Impala using impala-shell:
> {code:java}
> $ bin/impala-shell.sh
> [localhost:21050] default> invalidate metadata my_json_tbl;
> [localhost:21050] default> desc my_json_tbl;
> Query: describe my_json_tbl
> ERROR: AnalysisException: Failed to load metadata for table: default.my_json_tbl
> CAUSED BY: InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported.
> CAUSED BY: TableLoadingException: Failed to load metadata for table: default.my_json_tbl
> CAUSED BY: InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported. {code}
> There are more details in logs
> {code:java}
> I0209 14:59:55.419886 10083 jni-util.cc:286] 914bfb645cab120b:443e5ec500000000] org.apache.impala.common.AnalysisException: Failed to load metadata for table: default.my_json_tbl
> CAUSED BY: InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported.
> at org.apache.impala.analysis.DescribeTableStmt.analyze(DescribeTableStmt.java:112)
> at org.apache.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:497)
> at org.apache.impala.analysis.AnalysisContext.analyzeAndAuthorize(AnalysisContext.java:461)
> at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:1677)
> at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1643)
> at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1613)
> at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:164)
> Caused by: org.apache.impala.catalog.TableLoadingException: Failed to load metadata for table: default.my_json_tbl
> CAUSED BY: InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported.
> at org.apache.impala.catalog.IncompleteTable.loadFromThrift(IncompleteTable.java:123)
> at org.apache.impala.catalog.Table.fromThrift(Table.java:513)
> at org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:469)
> at org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:324)
> at org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:253)
> at org.apache.impala.service.FeCatalogManager$CatalogdImpl.updateCatalogCache(FeCatalogManager.java:114)
> at org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:410)
> at org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:185)
> at ========.<Remote stack trace on catalogd>: org.apache.impala.catalog.TableLoadingException: Failed to load metadata for table: default.my_json_tbl
> at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1265)
> at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1147)
> at org.apache.impala.catalog.TableLoader.load(TableLoader.java:144)
> at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:245)
> at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:242)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.impala.catalog.HdfsStorageDescriptor$InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hadoop.hive.serde2.JsonSerDe' is not supported.
> at org.apache.impala.catalog.HdfsStorageDescriptor.fromStorageDescriptor(HdfsStorageDescriptor.java:216)
> at org.apache.impala.catalog.HdfsTable.setPrototypePartition(HdfsTable.java:1139)
> at org.apache.impala.catalog.HdfsTable.initializePartitionMetadata(HdfsTable.java:656)
> at org.apache.impala.catalog.HdfsTable.loadAllPartitions(HdfsTable.java:678)
> at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1253)
> ... 8 more {code}
> We also need to fix the issue that Hive can't resolve JSON tables created by Impala.
> {code}
> Error while compiling statement: FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.ddl.DDLTask. java.lang.ClassNotFoundException Class org.apache.hadoop.hive.serde2.lazy.JsonSerDe not found
> {code}
> Probably change a SerDe that Hive recognizes.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org