[jira] [Updated] (HIVE-26213) "hive.limit.pushdown.memory.usage" better not be equal to 1.0, otherwise it will raise an error

["ECFuzz (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-26213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ECFuzz updated HIVE-26213:
--------------------------
    Component/s: Documentation

> "hive.limit.pushdown.memory.usage" better not be equal to 1.0, otherwise it will raise an error
> -----------------------------------------------------------------------------------------------
>
>                 Key: HIVE-26213
>                 URL: https://issues.apache.org/jira/browse/HIVE-26213
>             Project: Hive
>          Issue Type: Bug
>          Components: Documentation
>    Affects Versions: 3.1.2
>         Environment: Hive 3.1.2
> os.name=Linux
> os.arch=amd64
> os.version=5.4.0-72-generic
> java.version=1.8.0_162
> java.vendor=Oracle Corporation
>            Reporter: ECFuzz
>            Assignee: ECFuzz
>            Priority: Minor
>
> In hive-default.xml.template
> {code:java}
> <property>
>   <name>hive.limit.pushdown.memory.usage</name>
>   <value>0.1</value>
>   <description>
>     Expects value between 0.0f and 1.0f.
>     The fraction of available memory to be used for buffering rows in Reducesink operator for limit pushdown optimization.
>   </description>
> </property>{code}
> Based on the description of hive-default.xml.template, hive.limit.pushdown.memory.usage expects a value between 0.0 and 1.0, setting hive.limit.pushdown.memory.usage to 1.0 means that it expects the available memory of all buffered lines for the limit pushdown optimization, and successfully start hiveserver2.
> Then, call the java api to write a program to establish a jdbc connection as a client to access hive, using JDBCDemo as an example.
> {code:java}
> import demo.utils.JDBCUtils;
> public class JDBCDemo{
> public static void main(String[] args) throws Exception
> {   JDBCUtils.init();   JDBCUtils.createDatabase();   JDBCUtils.showDatabases();   JDBCUtils.createTable();   JDBCUtils.showTables();   JDBCUtils.descTable();   JDBCUtils.loadData();   JDBCUtils.selectData();   JDBCUtils.countData();   JDBCUtils.dropDatabase();   JDBCUtils.dropTable();   JDBCUtils.destory(); }
> }
> {code}
> After running the client program, both the client and the hiveserver throw exceptions.
> {code:java}
> 2022-05-09 19:05:36: Starting HiveServer2
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Hive Session ID = 67a6db8d-f957-4d5d-ac18-28403adab7f3
> Hive Session ID = f9f8772c-5765-4c3e-bcff-ca605c667be7
> OK
> OK
> OK
> OK
> OK
> OK
> OK
> Loading data to table default.emp
> OK
> FAILED: SemanticException Invalid memory usage value 1.0 for hive.limit.pushdown.memory.usage{code}
> {code:java}
> liky@ljq1:~/hive_jdbc_test$ ./startJDBC_0.sh 
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:file:/home/liky/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/home/liky/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Running: drop database if exists hive_jdbc_test
> Running: create database hive_jdbc_test
> Running: show databases
> default
> hive_jdbc_test
> Running: drop table if exists emp
> Running: create table emp(
> empno int,
> ename string,
> job string,
> mgr int,
> hiredate string,
> sal double,
> comm double,
> deptno int
> )
> row format delimited fields terminated by '\t'
> Running: show tables
> emp
> Running: desc emp
> empno   int
> ename   string
> job     string
> mgr     int
> hiredate       string
> sal     double
> comm   double
> deptno int
> Running: load data local inpath '/home/liky/hiveJDBCTestData/data.txt' overwrite into table emp
> Running: select * from emp
> Exception in thread "main" org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException Invalid memory usage value 1.0 for hive.limit.pushdown.memory.usage
>       at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:380)
>       at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:366)
>       at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:354)
>       at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:293)
>       at org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:509)
>       at demo.utils.JDBCUtils.selectData(JDBCUtils.java:98)
>       at demo.test.JDBCDemo.main(JDBCDemo.java:19){code}
> Setting hive.limit.pushdown.memory.usage to 0.0 has no exception.
> So, setting hive.limit.pushdown.memory.usage to 1.0 is not desirable, *hive-default.xml.template is not clear enough for the description of the boundary of the value, it is better to use the interval to indicate the value that is [0.0,1.0).*



--
This message was sent by Atlassian Jira
(v8.20.10#820010)