You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Ayush Saxena (Jira)" <ji...@apache.org> on 2023/06/20 04:20:00 UTC
[jira] [Resolved] (HIVE-27447) Iceberg: Queries failing due to FileSystem close errors due to column statistics
[ https://issues.apache.org/jira/browse/HIVE-27447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ayush Saxena resolved HIVE-27447.
---------------------------------
Fix Version/s: 4.0.0-beta-1
Resolution: Fixed
> Iceberg: Queries failing due to FileSystem close errors due to column statistics
> --------------------------------------------------------------------------------
>
> Key: HIVE-27447
> URL: https://issues.apache.org/jira/browse/HIVE-27447
> Project: Hive
> Issue Type: Bug
> Reporter: Ayush Saxena
> Assignee: Ayush Saxena
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-beta-1
>
>
> During fetching column statistics we are closing the filesystem in the HS2, FS is shared and cached, so closing it ain't a good idea.
> HS2 Logs show
> {noformat}
> 2023-06-16T16:57:09,515 DEBUG [45c1a38d-23c0-4e3a-96b7-c398bb215b2245c1a38d-23c0-4e3a-96b7-c398bb215b22 HiveServer2-Handler-Pool: Thread-114] fs.FileSystem: FileSystem.close() by method: org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:1511)); Key: (boroknagyz (auth:SIMPLE))@hdfs://localhost:20500; URI: hdfs://localhost:20500; Object Identity Hash: 14ec8672
> 2023-06-16T16:57:09,515 TRACE [45c1a38d-23c0-4e3a-96b7-c398bb215b2245c1a38d-23c0-4e3a-96b7-c398bb215b22 HiveServer2-Handler-Pool: Thread-114] fs.FileSystem: FileSystem.close() full stack trace:
> java.lang.Throwable: null
> at org.apache.hadoop.fs.FileSystem.debugLogFileSystemClose(FileSystem.java:627) ~[hadoop-common-3.1.1.7.2.16.0-287.jar:?]
> at org.apache.hadoop.fs.FileSystem.close(FileSystem.java:2599) ~[hadoop-common-3.1.1.7.2.16.0-287.jar:?]
> at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:1511) ~[hadoop-hdfs-client-3.1.1.7.2.16.0-287.jar:?]
> at org.apache.iceberg.mr.hive.HiveIcebergStorageHandler.canProvideColStatistics(HiveIcebergStorageHandler.java:444) ~[hive-iceberg-handler-3.1.3000.2023.0.15.0-159.jar:3.1.3000.2023.0.15.0-159]
> at org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStats(StatsUtils.java:1074) ~[hive-exec-3.1.3000.2023.0.15.0-159.jar:3.1.3000.2023.0.15.0-159]
> at org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:298) ~[hive-exec-3.1.3000.2023.0.15.0-159.jar:3.1.3000.2023.0.15.0-159]{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)