You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Kim Chew <kc...@gmail.com> on 2014/08/28 03:12:40 UTC

DBInputFormat.getSplits() results a div by zero when joining two tables.

I have a small table in mysql (Two rows) and I created a Hive version of it
using HQL,

CREATE TABLE IF NOT EXISTS
port_usage (protocolID INT, ports STRING, bytesReceived INT, description
STRING)
STORED BY 'com.foo.MyDatabaseStorageHandler'
    TBLPROPERTIES ("mapred.jdbc.driver.class"="com.mysql.jdbc.Driver",
                   "mapred.jdbc.url"="jdbc:mysql://172.31.7.128:3310/test",
                   "mapred.jdbc.username"="kim",
                   "mapred.jdbc.password"="kim",
                   "mapred.jdbc.input.table.name"="port_usage");

"port_usage" is created successfully and has exactly the same two rows as
in mysql. When I tried to do a join (Unable to do map side join because of
HIVE-6670) so I set "hive.auto.convert.join=false),

select serverip fromvectors v JOIN port_usage u ON v.protocolid =
u.protocolid;

I got the "divided by zero" exception.

java.lang.ArithmeticException: divide by zero
    at
org.apache.hadoop.mapred.lib.db.DBInputFormat.getSplits(DBInputFormat.java:303)
    at
com.foo.MyDatabaseInputFormat.getSplits(MyDatabaseInputFormat.java:37)
    at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:294)
    at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:303)
    at
org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:1109)
    at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1101)
    at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:188)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:1010)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:963)

Is it because there are some "mapred.jdbc" properties I forget to set?

"MyDatabaseInputFormat" simply is a wrapper around
"org.apache.hadoop.mapred.lib.db.DBInputFormat",

@Override
    public InputSplit[] getSplits(JobConf jConf, int chunks) throws
IOException {
        return super.getSplits(jConf, chunks);
    }

so I am not messing around the "getSplits"

TIA.

Kim