You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by Mars J <xu...@gmail.com> on 2016/04/05 09:14:26 UTC
Starting Kylin need hive-site.xml in the path ?
Hello All,
There is a problem that every time I start kylin, it will read a
hive-site.xml file in the path where I execute the starting action. if
there is no that file, the cube building job where give me an error message
like follow, and the /home/marsj is my kylin start path.
If I copy the hive-site.xml to this path , my job is execute
successfully.
Is this a bug ? No matter kylin version is 1.3 or 1.5, this problem
is alway there.
java.lang.RuntimeException: java.io.IOException:
java.lang.RuntimeException: java.io.FileNotFoundException:
/home/marsj/hive-site.xml (没有那个文件或目录)
at org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:86)
at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:111)
at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:91)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:118)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: java.lang.RuntimeException:
java.io.FileNotFoundException: /home/marsj/hive-site.xml (没有那个文件或目录)
Re: Starting Kylin need hive-site.xml in the path ?
Posted by Mars J <xu...@gmail.com>.
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException:
/home/marsj/hive-site.xml (没有那个文件或目录)
3150 at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2342)
3151 at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2205)
3152 at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2112)
3153 at org.apache.hadoop.conf.Configuration.get(Configuration.java:1078)
3154 at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:2605)
3155 at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:2626)
3156 at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:2696)
3157 at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:2651)
3158 at org.apache.hive.hcatalog.common.HCatUtil.getHiveConf(HCatUtil.java:599)
3159 at org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:100)
3160 at org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:86)
3161 at org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:95)
3162 ... 14 more
3163 Caused by: java.io.FileNotFoundException:
/home/marsj/hive-site.xml (没有那个文件或目录)
3164 at java.io.FileInputStream.open0(Native Method)
3165 at java.io.FileInputStream.open(FileInputStream.java:195)
3166 at java.io.FileInputStream.<init>(FileInputStream.java:138)
3167 at java.io.FileInputStream.<init>(FileInputStream.java:93)
3168 at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
3169 at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
3170 at java.net.URL.openStream(URL.java:1038)
3171 at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2171)
3172 at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2239)
3173 ... 25 more
it's the follow logs, it seems like kylin's starting require hive-conf,when
I execute find-hive-dependency.sh,it will print as follow:
find-hive-dependency.sh Logging initialized using configuration in
jar:file:/opt/hive-1.2.1/lib/hive-common-1.2.1.jar!/hive-log4j.properties
HCAT_HOME is set to: /opt/hive-1.2.1/hcatalog/, use it to find hcatalog
path: hive dependency:
/opt/hive-1.2.1/conf:/opt/hive-1.2.1/lib/stax-api-1.0.1.jar:/opt/hive-1.2.1/lib/bonecp-0.8.0.RELEASE.jar:/opt/hive-1.2.1/lib/maven-scm-api-1.4.jar:/opt/hive-1.2.1/lib/hive-shims-0.20S-1.2.1.jar:/opt/hive-1.2.1/lib/plexus-utils-1.5.6.jar:/opt/hive-1.2.1/lib/libfb303-0.9.2.jar:/opt/hive-1.2.1/lib/super-csv-2.2.0.jar:/opt/hive-1.2.1/lib/ant-1.9.1.jar:/opt/hive-1.2.1/lib/commons-beanutils-1.7.0.jar:/opt/hive-1.2.1/lib/hive-ant-1.2.1.jar:/opt/hive-1.2.1/lib/hive-shims-1.2.1.jar:/opt/hive-1.2.1/lib/antlr-2.7.7.jar:/opt/hive-1.2.1/lib/commons-dbcp-1.4.jar:/opt/hive-1.2.1/lib/guava-14.0.1.jar:/opt/hive-1.2.1/lib/hive-shims-scheduler-1.2.1.jar:/opt/hive-1.2.1/lib/ST4-4.0.4.jar:/opt/hive-1.2.1/lib/jline-2.12.jar:/opt/hive-1.2.1/lib/geronimo-jta_1.1_spec-1.1.1.jar:/opt/hive-1.2.1/lib/hive-hcatalog-streaming-1.2.1.jar:/opt/hive-1.2.1/lib/mail-1.4.1.jar:/opt/hive-1.2.1/lib/commons-compiler-2.7.6.jar:/opt/hive-1.2.1/lib/jdo-api-3.0.1.jar:/opt/hive-1.2.1/lib/datanucleus-rdbms-3.2.9.jar:/opt/hive-1.2.1/lib/commons-digester-1.8.jar:/opt/hive-1.2.1/lib/janino-2.7.6.jar:/opt/hive-1.2.1/lib/apache-log4j-extras-1.2.17.jar:/opt/hive-1.2.1/lib/hive-shims-common-1.2.1.jar:/opt/hive-1.2.1/lib/opencsv-2.3.jar:/opt/hive-1.2.1/lib/commons-httpclient-3.0.1.jar:/opt/hive-1.2.1/lib/mysql-connector-java-5.1.38-bin.jar:/opt/hive-1.2.1/lib/oro-2.0.8.jar:/opt/hive-1.2.1/lib/hive-hcatalog-pig-adapter-1.2.1.jar:/opt/hive-1.2.1/lib/xz-1.0.jar:/opt/hive-1.2.1/lib/commons-logging-1.1.3.jar:/opt/hive-1.2.1/lib/netty-3.7.0.Final.jar:/opt/hive-1.2.1/lib/hive-serde-1.2.1.jar:/opt/hive-1.2.1/lib/jetty-all-server-7.6.0.v20120127.jar:/opt/hive-1.2.1/lib/log4j-1.2.16.jar:/opt/hive-1.2.1/lib/jta-1.1.jar:/opt/hive-1.2.1/lib/ant-launcher-1.9.1.jar:/opt/hive-1.2.1/lib/servlet-api-2.5.jar:/opt/hive-1.2.1/lib/jcommander-1.32.jar:/opt/hive-1.2.1/lib/avro-1.7.5.jar:/opt/hive-1.2.1/lib/zookeeper-3.4.6.jar:/opt/hive-1.2.1/lib/asm-tree-3.1.jar:/opt/hive-1.2.1/lib/commons-math-2.1.jar:/opt/hive-1.2.1/lib/commons-lang-2.6.jar:/opt/hive-1.2.1/lib/commons-configuration-1.6.jar:/opt/hive-1.2.1/lib/hive-beeline-1.2.1.jar:/opt/hive-1.2.1/lib/hive-hcatalog-core-1.2.1.jar:/opt/hive-1.2.1/lib/commons-compress-1.4.1.jar:/opt/hive-1.2.1/lib/hive-exec-1.2.1.jar:/opt/hive-1.2.1/lib/commons-vfs2-2.0.jar:/opt/hive-1.2.1/lib/activation-1.1.jar:/opt/hive-1.2.1/lib/asm-commons-3.1.jar:/opt/hive-1.2.1/lib/hive-jdbc-1.2.1.jar:/opt/hive-1.2.1/lib/accumulo-core-1.6.0.jar:/opt/hive-1.2.1/lib/accumulo-trace-1.6.0.jar:/opt/hive-1.2.1/lib/stringtemplate-3.2.1.jar:/opt/hive-1.2.1/lib/ivy-2.4.0.jar:/opt/hive-1.2.1/lib/geronimo-jaspic_1.0_spec-1.0.jar:/opt/hive-1.2.1/lib/hive-hbase-handler-1.2.1.jar:/opt/hive-1.2.1/lib/commons-codec-1.4.jar:/opt/hive-1.2.1/lib/paranamer-2.3.jar:/opt/hive-1.2.1/lib/snappy-java-1.0.5.jar:/opt/hive-1.2.1/lib/hive-jdbc-1.2.1-standalone.jar:/opt/hive-1.2.1/lib/hive-hwi-1.2.1.jar:/opt/hive-1.2.1/lib/velocity-1.5.jar:/opt/hive-1.2.1/lib/junit-4.11.jar:/opt/hive-1.2.1/lib/json-20090211.jar:/opt/hive-1.2.1/lib/commons-cli-1.2.jar:/opt/hive-1.2.1/lib/datanucleus-api-jdo-3.2.6.jar:/opt/hive-1.2.1/lib/jetty-all-7.6.0.v20120127.jar:/opt/hive-1.2.1/lib/geronimo-annotation_1.0_spec-1.1.1.jar:/opt/hive-1.2.1/lib/commons-collections-3.2.1.jar:/opt/hive-1.2.1/lib/regexp-1.3.jar:/opt/hive-1.2.1/lib/hive-contrib-1.2.1.jar:/opt/hive-1.2.1/lib/curator-recipes-2.6.0.jar:/opt/hive-1.2.1/lib/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar:/opt/hive-1.2.1/lib/hive-metastore-1.2.1.jar:/opt/hive-1.2.1/lib/commons-pool-1.5.4.jar:/opt/hive-1.2.1/lib/accumulo-fate-1.6.0.jar:/opt/hive-1.2.1/lib/derby-10.10.2.0.jar:/opt/hive-1.2.1/lib/httpclient-4.4.jar:/opt/hive-1.2.1/lib/datanucleus-core-3.2.10.jar:/opt/hive-1.2.1/lib/hive-accumulo-handler-1.2.1.jar:/opt/hive-1.2.1/lib/parquet-hadoop-bundle-1.6.0.jar:/opt/hive-1.2.1/lib/hive-testutils-1.2.1.jar:/opt/hive-1.2.1/lib/httpcore-4.4.jar:/opt/hive-1.2.1/lib/jsr305-3.0.0.jar:/opt/hive-1.2.1/lib/hive-hcatalog-server-extensions-1.2.1.jar:/opt/hive-1.2.1/lib/hamcrest-core-1.1.jar:/opt/hive-1.2.1/lib/groovy-all-2.1.6.jar:/opt/hive-1.2.1/lib/hive-parent-auth-hook-1.0.5.jar:/opt/hive-1.2.1/lib/curator-client-2.6.0.jar:/opt/hive-1.2.1/lib/libthrift-0.9.2.jar:/opt/hive-1.2.1/lib/maven-scm-provider-svn-commons-1.4.jar:/opt/hive-1.2.1/lib/tempus-fugit-1.1.jar:/opt/hive-1.2.1/lib/antlr-runtime-3.4.jar:/opt/hive-1.2.1/lib/maven-scm-provider-svnexe-1.4.jar:/opt/hive-1.2.1/lib/joda-time-2.5.jar:/opt/hive-1.2.1/lib/jpam-1.1.jar:/opt/hive-1.2.1/lib/hive-service-1.2.1.jar:/opt/hive-1.2.1/lib/eigenbase-properties-1.1.5.jar:/opt/hive-1.2.1/lib/commons-beanutils-core-1.8.0.jar:/opt/hive-1.2.1/lib/accumulo-start-1.6.0.jar:/opt/hive-1.2.1/lib/hive-common-1.2.1.jar:/opt/hive-1.2.1/lib/commons-io-2.4.jar:/opt/hive-1.2.1/lib/curator-framework-2.6.0.jar:/opt/hive-1.2.1/lib/hive-shims-0.23-1.2.1.jar:/opt/hive-1.2.1/lib/hive-cli-1.2.1.jar:/opt/hive-1.2.1/hcatalog/share/hcatalog/hive-hcatalog-core-1.2.1.jar
it's find the hive conf directory is /opt-hive-1.2.1/conf ...why can't read
the hive-site.xml in this directory ?
Re: Starting Kylin need hive-site.xml in the path ?
Posted by Li Yang <li...@apache.org>.
Any more logs beneath the "Caused by: java.io.IOException"??
Kylin requires hive for sure, and it grabs hive classpath by
'bin/find-hive-dependency.sh' which in turn calls `hive -e set | grep
'env:CLASSPATH'`
You want to analyze these results to see why '/home/marsj/hive-site.xml' is
being insisted.
On Tue, Apr 5, 2016 at 3:14 PM, Mars J <xu...@gmail.com> wrote:
> Hello All,
> There is a problem that every time I start kylin, it will read a
> hive-site.xml file in the path where I execute the starting action. if
> there is no that file, the cube building job where give me an error message
> like follow, and the /home/marsj is my kylin start path.
> If I copy the hive-site.xml to this path , my job is execute
> successfully.
>
> Is this a bug ? No matter kylin version is 1.3 or 1.5, this problem
> is alway there.
>
> java.lang.RuntimeException: java.io.IOException: java.lang.RuntimeException: java.io.FileNotFoundException: /home/marsj/hive-site.xml (没有那个文件或目录)
> at org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:86)
> at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:111)
> at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:91)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:118)
> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: java.lang.RuntimeException: java.io.FileNotFoundException: /home/marsj/hive-site.xml (没有那个文件或目录)
>
>