You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Jingxuan Fu (Jira)" <ji...@apache.org> on 2022/05/09 14:00:00 UTC
[jira] [Assigned] (HIVE-26213) "hive.limit.pushdown.memory.usage" better not be equal to 1.0, otherwise it will raise an error
[ https://issues.apache.org/jira/browse/HIVE-26213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jingxuan Fu reassigned HIVE-26213:
----------------------------------
> "hive.limit.pushdown.memory.usage" better not be equal to 1.0, otherwise it will raise an error
> -----------------------------------------------------------------------------------------------
>
> Key: HIVE-26213
> URL: https://issues.apache.org/jira/browse/HIVE-26213
> Project: Hive
> Issue Type: Bug
> Affects Versions: 3.1.2
> Environment: Hive 3.1.2
> os.name=Linux
> os.arch=amd64
> os.version=5.4.0-72-generic
> java.version=1.8.0_162
> java.vendor=Oracle Corporation
> Reporter: Jingxuan Fu
> Assignee: Jingxuan Fu
> Priority: Major
>
> In hive-default.xml.template
>
>
>
>
> {code:java}
> <property> <name>hive.limit.pushdown.memory.usage</name> <value>0.1</value> <description> Expects value between 0.0f and 1.0f. The fraction of available memory to be used for buffering rows in Reducesink operator for limit pushdown optimization. </description> </property>
> {code}
>
>
> Based on the description of hive-default.xml.template, hive.limit.pushdown.memory.usage expects a value between 0.0 and 1.0, setting hive.limit.pushdown.memory.usage to 1.0 means that it expects the available memory of all buffered lines for the limit pushdown optimization, and successfully start hiveserver2.
> Then, call the java api to write a program to establish a jdbc connection as a client to access hive, using JDBCDemo as an example.
>
> {code:java}
> import demo.utils.JDBCUtils; public class JDBCDemo{ public static void main(String[] args) throws Exception { JDBCUtils.init(); JDBCUtils.createDatabase(); JDBCUtils.showDatabases(); JDBCUtils.createTable(); JDBCUtils.showTables(); JDBCUtils.descTable(); JDBCUtils.loadData(); JDBCUtils.selectData(); JDBCUtils.countData(); JDBCUtils.dropDatabase(); JDBCUtils.dropTable(); JDBCUtils.destory(); } }
> {code}
> After running the client program, both the client and the hiveserver throw exceptions.
>
> {code:java}
> 2022-05-09 19:05:36: Starting HiveServer2 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Hive Session ID = 67a6db8d-f957-4d5d-ac18-28403adab7f3 Hive Session ID = f9f8772c-5765-4c3e-bcff-ca605c667be7 OK OK OK OK OK OK OK Loading data to table default.emp OK FAILED: SemanticException Invalid memory usage value 1.0 for hive.limit.pushdown.memory.usage{code}
>
>
>
> {code:java}
> liky@ljq1:~/hive_jdbc_test$ ./startJDBC_0.sh SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/liky/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/liky/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Running: drop database if exists hive_jdbc_test Running: create database hive_jdbc_test Running: show databases default hive_jdbc_test Running: drop table if exists emp Running: create table emp( empno int, ename string, job string, mgr int, hiredate string, sal double, comm double, deptno int ) row format delimited fields terminated by '\t' Running: show tables emp Running: desc emp empno int ename string job string mgr int hiredate string sal double comm double deptno int Running: load data local inpath '/home/liky/hiveJDBCTestData/data.txt' overwrite into table emp Running: select * from emp Exception in thread "main" org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException Invalid memory usage value 1.0 for hive.limit.pushdown.memory.usage at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:380) at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:366) at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:354) at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:293) at org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:509) at demo.utils.JDBCUtils.selectData(JDBCUtils.java:98) at demo.test.JDBCDemo.main(JDBCDemo.java:19){code}
>
>
> Setting hive.limit.pushdown.memory.usage to 0.0 has no exception.
> So, setting hive.limit.pushdown.memory.usage to 1.0 is not desirable, *hive-default.xml.template is not clear enough for the description of the boundary of the value, it is better to use the interval to indicate the value that is [0.0,1.0).*
--
This message was sent by Atlassian Jira
(v8.20.7#820007)