You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Jingxuan Fu (Jira)" <ji...@apache.org> on 2022/05/09 14:04:00 UTC
[jira] [Updated] (HIVE-26213) "hive.limit.pushdown.memory.usage" better not be equal to 1.0, otherwise it will raise an error
[ https://issues.apache.org/jira/browse/HIVE-26213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jingxuan Fu updated HIVE-26213:
-------------------------------
Description:
In hive-default.xml.template
<property>
<name>hive.limit.pushdown.memory.usage</name>
<value>0.1</value>
<description>
Expects value between 0.0f and 1.0f.
The fraction of available memory to be used for buffering rows in Reducesink operator for limit pushdown optimization.
</description>
</property>
Based on the description of hive-default.xml.template, hive.limit.pushdown.memory.usage expects a value between 0.0 and 1.0, setting hive.limit.pushdown.memory.usage to 1.0 means that it expects the available memory of all buffered lines for the limit pushdown optimization, and successfully start hiveserver2.
Then, call the java api to write a program to establish a jdbc connection as a client to access hive, using JDBCDemo as an example.
import demo.utils.JDBCUtils;
public class JDBCDemo{
public static void main(String[] args) throws Exception {
JDBCUtils.init();
JDBCUtils.createDatabase();
JDBCUtils.showDatabases();
JDBCUtils.createTable();
JDBCUtils.showTables();
JDBCUtils.descTable();
JDBCUtils.loadData();
JDBCUtils.selectData();
JDBCUtils.countData();
JDBCUtils.dropDatabase();
JDBCUtils.dropTable();
JDBCUtils.destory();
}
}
After running the client program, both the client and the hiveserver throw exceptions.
2022-05-09 19:05:36: Starting HiveServer2
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = 67a6db8d-f957-4d5d-ac18-28403adab7f3
Hive Session ID = f9f8772c-5765-4c3e-bcff-ca605c667be7
OK
OK
OK
OK
OK
OK
OK
Loading data to table default.emp
OK
FAILED: SemanticException Invalid memory usage value 1.0 for hive.limit.pushdown.memory.usage
liky@ljq1:~/hive_jdbc_test$ ./startJDBC_0.sh
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/liky/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/liky/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Running: drop database if exists hive_jdbc_test
Running: create database hive_jdbc_test
Running: show databases
default
hive_jdbc_test
Running: drop table if exists emp
Running: create table emp(
empno int,
ename string,
job string,
mgr int,
hiredate string,
sal double,
comm double,
deptno int
)
row format delimited fields terminated by '\t'
Running: show tables
emp
Running: desc emp
empno int
ename string
job string
mgr int
hiredate string
sal double
comm double
deptno int
Running: load data local inpath '/home/liky/hiveJDBCTestData/data.txt' overwrite into table emp
Running: select * from emp
Exception in thread "main" org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException Invalid memory usage value 1.0 for hive.limit.pushdown.memory.usage
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:380)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:366)
at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:354)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:293)
at org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:509)
at demo.utils.JDBCUtils.selectData(JDBCUtils.java:98)
at demo.test.JDBCDemo.main(JDBCDemo.java:19)
Setting hive.limit.pushdown.memory.usage to 0.0 has no exception.
So, setting hive.limit.pushdown.memory.usage to 1.0 is not desirable, *hive-default.xml.template is not clear enough for the description of the boundary of the value, it is better to use the interval to indicate the value that is [0.0,1.0).*
was:
In hive-default.xml.template
{code:java}
<property> <name>hive.limit.pushdown.memory.usage</name> <value>0.1</value> <description> Expects value between 0.0f and 1.0f. The fraction of available memory to be used for buffering rows in Reducesink operator for limit pushdown optimization. </description> </property>
{code}
Based on the description of hive-default.xml.template, hive.limit.pushdown.memory.usage expects a value between 0.0 and 1.0, setting hive.limit.pushdown.memory.usage to 1.0 means that it expects the available memory of all buffered lines for the limit pushdown optimization, and successfully start hiveserver2.
Then, call the java api to write a program to establish a jdbc connection as a client to access hive, using JDBCDemo as an example.
{code:java}
import demo.utils.JDBCUtils; public class JDBCDemo{ public static void main(String[] args) throws Exception { JDBCUtils.init(); JDBCUtils.createDatabase(); JDBCUtils.showDatabases(); JDBCUtils.createTable(); JDBCUtils.showTables(); JDBCUtils.descTable(); JDBCUtils.loadData(); JDBCUtils.selectData(); JDBCUtils.countData(); JDBCUtils.dropDatabase(); JDBCUtils.dropTable(); JDBCUtils.destory(); } }
{code}
After running the client program, both the client and the hiveserver throw exceptions.
{code:java}
2022-05-09 19:05:36: Starting HiveServer2 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Hive Session ID = 67a6db8d-f957-4d5d-ac18-28403adab7f3 Hive Session ID = f9f8772c-5765-4c3e-bcff-ca605c667be7 OK OK OK OK OK OK OK Loading data to table default.emp OK FAILED: SemanticException Invalid memory usage value 1.0 for hive.limit.pushdown.memory.usage{code}
{code:java}
liky@ljq1:~/hive_jdbc_test$ ./startJDBC_0.sh SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/liky/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/liky/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Running: drop database if exists hive_jdbc_test Running: create database hive_jdbc_test Running: show databases default hive_jdbc_test Running: drop table if exists emp Running: create table emp( empno int, ename string, job string, mgr int, hiredate string, sal double, comm double, deptno int ) row format delimited fields terminated by '\t' Running: show tables emp Running: desc emp empno int ename string job string mgr int hiredate string sal double comm double deptno int Running: load data local inpath '/home/liky/hiveJDBCTestData/data.txt' overwrite into table emp Running: select * from emp Exception in thread "main" org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException Invalid memory usage value 1.0 for hive.limit.pushdown.memory.usage at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:380) at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:366) at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:354) at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:293) at org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:509) at demo.utils.JDBCUtils.selectData(JDBCUtils.java:98) at demo.test.JDBCDemo.main(JDBCDemo.java:19){code}
Setting hive.limit.pushdown.memory.usage to 0.0 has no exception.
So, setting hive.limit.pushdown.memory.usage to 1.0 is not desirable, *hive-default.xml.template is not clear enough for the description of the boundary of the value, it is better to use the interval to indicate the value that is [0.0,1.0).*
> "hive.limit.pushdown.memory.usage" better not be equal to 1.0, otherwise it will raise an error
> -----------------------------------------------------------------------------------------------
>
> Key: HIVE-26213
> URL: https://issues.apache.org/jira/browse/HIVE-26213
> Project: Hive
> Issue Type: Bug
> Affects Versions: 3.1.2
> Environment: Hive 3.1.2
> os.name=Linux
> os.arch=amd64
> os.version=5.4.0-72-generic
> java.version=1.8.0_162
> java.vendor=Oracle Corporation
> Reporter: Jingxuan Fu
> Assignee: Jingxuan Fu
> Priority: Major
>
> In hive-default.xml.template
> <property>
> <name>hive.limit.pushdown.memory.usage</name>
> <value>0.1</value>
> <description>
> Expects value between 0.0f and 1.0f.
> The fraction of available memory to be used for buffering rows in Reducesink operator for limit pushdown optimization.
> </description>
> </property>
> Based on the description of hive-default.xml.template, hive.limit.pushdown.memory.usage expects a value between 0.0 and 1.0, setting hive.limit.pushdown.memory.usage to 1.0 means that it expects the available memory of all buffered lines for the limit pushdown optimization, and successfully start hiveserver2.
> Then, call the java api to write a program to establish a jdbc connection as a client to access hive, using JDBCDemo as an example.
> import demo.utils.JDBCUtils;
> public class JDBCDemo{
> public static void main(String[] args) throws Exception {
> JDBCUtils.init();
> JDBCUtils.createDatabase();
> JDBCUtils.showDatabases();
> JDBCUtils.createTable();
> JDBCUtils.showTables();
> JDBCUtils.descTable();
> JDBCUtils.loadData();
> JDBCUtils.selectData();
> JDBCUtils.countData();
> JDBCUtils.dropDatabase();
> JDBCUtils.dropTable();
> JDBCUtils.destory();
> }
> }
> After running the client program, both the client and the hiveserver throw exceptions.
> 2022-05-09 19:05:36: Starting HiveServer2
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Hive Session ID = 67a6db8d-f957-4d5d-ac18-28403adab7f3
> Hive Session ID = f9f8772c-5765-4c3e-bcff-ca605c667be7
> OK
> OK
> OK
> OK
> OK
> OK
> OK
> Loading data to table default.emp
> OK
> FAILED: SemanticException Invalid memory usage value 1.0 for hive.limit.pushdown.memory.usage
> liky@ljq1:~/hive_jdbc_test$ ./startJDBC_0.sh
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:file:/home/liky/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/home/liky/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Running: drop database if exists hive_jdbc_test
> Running: create database hive_jdbc_test
> Running: show databases
> default
> hive_jdbc_test
> Running: drop table if exists emp
> Running: create table emp(
> empno int,
> ename string,
> job string,
> mgr int,
> hiredate string,
> sal double,
> comm double,
> deptno int
> )
> row format delimited fields terminated by '\t'
> Running: show tables
> emp
> Running: desc emp
> empno int
> ename string
> job string
> mgr int
> hiredate string
> sal double
> comm double
> deptno int
> Running: load data local inpath '/home/liky/hiveJDBCTestData/data.txt' overwrite into table emp
> Running: select * from emp
> Exception in thread "main" org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException Invalid memory usage value 1.0 for hive.limit.pushdown.memory.usage
> at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:380)
> at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:366)
> at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:354)
> at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:293)
> at org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:509)
> at demo.utils.JDBCUtils.selectData(JDBCUtils.java:98)
> at demo.test.JDBCDemo.main(JDBCDemo.java:19)
> Setting hive.limit.pushdown.memory.usage to 0.0 has no exception.
> So, setting hive.limit.pushdown.memory.usage to 1.0 is not desirable, *hive-default.xml.template is not clear enough for the description of the boundary of the value, it is better to use the interval to indicate the value that is [0.0,1.0).*
--
This message was sent by Atlassian Jira
(v8.20.7#820007)