You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Kim Chew <kc...@gmail.com> on 2014/08/28 03:12:40 UTC
DBInputFormat.getSplits() results a div by zero when joining two tables.
I have a small table in mysql (Two rows) and I created a Hive version of it
using HQL,
CREATE TABLE IF NOT EXISTS
port_usage (protocolID INT, ports STRING, bytesReceived INT, description
STRING)
STORED BY 'com.foo.MyDatabaseStorageHandler'
TBLPROPERTIES ("mapred.jdbc.driver.class"="com.mysql.jdbc.Driver",
"mapred.jdbc.url"="jdbc:mysql://172.31.7.128:3310/test",
"mapred.jdbc.username"="kim",
"mapred.jdbc.password"="kim",
"mapred.jdbc.input.table.name"="port_usage");
"port_usage" is created successfully and has exactly the same two rows as
in mysql. When I tried to do a join (Unable to do map side join because of
HIVE-6670) so I set "hive.auto.convert.join=false),
select serverip fromvectors v JOIN port_usage u ON v.protocolid =
u.protocolid;
I got the "divided by zero" exception.
java.lang.ArithmeticException: divide by zero
at
org.apache.hadoop.mapred.lib.db.DBInputFormat.getSplits(DBInputFormat.java:303)
at
com.foo.MyDatabaseInputFormat.getSplits(MyDatabaseInputFormat.java:37)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:294)
at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:303)
at
org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:1109)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1101)
at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:188)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:1010)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:963)
Is it because there are some "mapred.jdbc" properties I forget to set?
"MyDatabaseInputFormat" simply is a wrapper around
"org.apache.hadoop.mapred.lib.db.DBInputFormat",
@Override
public InputSplit[] getSplits(JobConf jConf, int chunks) throws
IOException {
return super.getSplits(jConf, chunks);
}
so I am not messing around the "getSplits"
TIA.
Kim