You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by Diego Pinheiro <di...@gmail.com> on 2015/09/01 01:46:02 UTC
Re: Kylin and Remote Server
After some changes, I am getting the following error:
[pool-5-thread-2]:[2015-08-31
19:42:13,394][ERROR][org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:83)]
- error in FactDistinctColumnsJob
java.io.IOException:
NoSuchObjectException(message:DEFAULT.kylin_intermediate_kylin_sales_cube_desc_20120101000000_20150727000000_a7572e8b_2c6a_4904_a19b_d74933e6d008
table not found)
at org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:97)
at org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:51)
at org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:101)
at org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:74)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:112)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:132)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: NoSuchObjectException(message:DEFAULT.kylin_intermediate_kylin_sales_cube_desc_20120101000000_20150727000000_a7572e8b_2c6a_4904_a19b_d74933e6d008
table not found)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1560)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
at com.sun.proxy.$Proxy45.get_table(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:997)
at org.apache.hive.hcatalog.common.HCatUtil.getTable(HCatUtil.java:191)
at org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:105)
at org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:86)
at org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:95)
... 13 more
Did you already face it?
On Wed, Aug 26, 2015 at 11:20 PM, Diego Pinheiro
<di...@gmail.com> wrote:
> @DroopyHoo, it is good to know that since we are planning to change
> authentication, but this is not the cause of my error. Since hongbin
> ma comments, I changed my kylin properties to use the same as sandbox
> (which is my VM HDP 2.1). So, I am using LDAP auth for now.
>
> @hongbin ma, kylin.job.run.as.remote.cmd was set to true indeed. I
> changed it to false, but I am still getting the same errors:
>
> [pool-5-thread-2]:[2015-08-25
> 19:16:03,679][ERROR][org.apache.kylin.job.tools.HadoopStatusChecker.checkStatus(HadoopStatusChecker.java:91)]
> - error check status
> java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:198)
> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
>
> org.apache.kylin.job.exception.ExecuteException: java.lang.NullPointerException
> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:110)
> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:106)
>
> Just to resume, this is my current environment:
>
> - one machine with HDP 2.1 (which I already ran Kylin 0.7.2 and it worked well);
> - HDP 2.1 is my cluster, my machine is the client;
> - I did not change any configuration in my cluster;
> - Kylin is in my client machine with the default configuration files;
> - I downloaded Hadoop 2.4.0, HBase 0.98.0 and Hive 0.13.1 to use in my
> client as a Hadoop CLI;
> - In the template of these files: hadoop/core-site.xml,
> hbase/hbase-site.xml and hive/hive-site.xml, I just added one or two
> properties to set my cluster IP. All these tools are apparently ok
> since I can access my cluster from them.
>
> Whereas my cluster is ok, I was thinking if my problem is in Hadoop
> CLI configuration files in the client...what changes have you guys
> done in those three configuration files? Did you also change
> yarn-site.xml or hdfs-site.xml?
>
> HDP 2.1 makes Kylin installation really easy. But, since I am new in
> Hadoop, I am facing these problems when setting up the client machine.
>
>
> On Wed, Aug 26, 2015 at 5:40 AM, DroopyHoo <ol...@gmail.com> wrote:
>> Hi Diego
>>
>> We met this error stack when deploying in our Hadoop enviroment(not
>> sandbox). The problem we met is the function for checking MR job status do
>> not support kerberos auth (our hadoop cluster use kerberos service). So we
>> made some changes to this part of source code.
>>
>> I'm not sure whether this case could help you to analyse the problem.
>>
>> 在 15/8/26 上午10:46, Diego Pinheiro 写道:
>>
>>> Hi Bin Mahone,
>>>
>>> sorry for the late reply. Thank you for your support. I didn't know
>>> about Kylin instances. It is really interesting.
>>>
>>> However, let me ask you, I was setting up my hadoop client machine
>>> with Kylin to communicate to my sandbox. But things are not working
>>> well.
>>>
>>> I have installed hadoop 2.4.0, hbase 0.98.0 and hive 0.13.1. All them
>>> are working and I can access my "remote server" from my client machine
>>> (actually, I set kylin as sandbox since all my hadoop cli is pointing
>>> to my sandbox). Then, Kylin was built and everything was ok until I
>>> tried to build the cube.
>>>
>>> I got the following errors always in the second step of cube build:
>>>
>>> [pool-5-thread-2]:[2015-08-25
>>>
>>> 19:16:03,679][ERROR][org.apache.kylin.job.tools.HadoopStatusChecker.checkStatus(HadoopStatusChecker.java:91)]
>>> - error check status
>>> java.net.ConnectException: Connection refused
>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>> at
>>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
>>> at
>>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:198)
>>> at
>>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>> at java.net.Socket.connect(Socket.java:579)
>>> at java.net.Socket.connect(Socket.java:528)
>>> at java.net.Socket.<init>(Socket.java:425)
>>> at java.net.Socket.<init>(Socket.java:280)
>>> at
>>> org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80)
>>> at
>>> org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122)
>>> at
>>> org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
>>> at
>>> org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
>>> at
>>> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
>>> at
>>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
>>> at
>>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
>>> at
>>> org.apache.kylin.job.tools.HadoopStatusGetter.getHttpResponse(HadoopStatusGetter.java:78)
>>> at
>>> org.apache.kylin.job.tools.HadoopStatusGetter.get(HadoopStatusGetter.java:55)
>>> at
>>> org.apache.kylin.job.tools.HadoopStatusChecker.checkStatus(HadoopStatusChecker.java:56)
>>> at
>>> org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:136)
>>> at
>>> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:106)
>>> at
>>> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
>>> at
>>> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:106)
>>> at
>>> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:133)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> org.apache.kylin.job.exception.ExecuteException:
>>> java.lang.NullPointerException
>>> at
>>> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:110)
>>> at
>>> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
>>> at
>>> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:106)
>>> at
>>> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:133)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:745)
>>> Caused by: java.lang.NullPointerException
>>> at
>>> org.apache.kylin.job.common.MapReduceExecutable.onExecuteStart(MapReduceExecutable.java:73)
>>> at
>>> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:105)
>>> ... 6 more
>>>
>>> Do you have any thoughts about these errors? (detailed log is attached)
>>>
>>>
>>> On Fri, Aug 21, 2015 at 3:21 AM, hongbin ma <ma...@apache.org> wrote:
>>>>
>>>> the document is summarized at
>>>> http://kylin.incubator.apache.org/docs/install/kylin_cluster.html
>>>>
>>>> On Fri, Aug 21, 2015 at 1:51 PM, hongbin ma <ma...@apache.org> wrote:
>>>>
>>>>> hi Diego
>>>>>
>>>>> the config "kylin.job.run.as.remote.cmd" is somehow ambiguous , it is
>>>>> enabled when you cannot run Kylin server on the same machine as your
>>>>> hadoop
>>>>> CLI, for example, if you're starting Kylin from you local IDE, and you
>>>>> hadoop CLI is a sandbox in another machine, this is the "remote" case.
>>>>>
>>>>> In most of the production deployments we suggest using '"non-remote"
>>>>> mode,
>>>>> that is, kylin instance is started on the hadoop CLI. The picture
>>>>> depicts
>>>>> the scenario:
>>>>>
>>>>> https://github.com/apache/incubator-kylin/blob/0.7/website/images/install/on_cli_install_scene.png
>>>>>
>>>>> Kylin instances are stateless, the runtime state is saved in its
>>>>> "Metadata Store" in hbase (kylin.metadata.url config in
>>>>> conf/kylin.properties). For load balance considerations it is possible
>>>>> to
>>>>> start multiple Kylin instances sharing the same metadata store (thus
>>>>> sharing the same state on table schemas, job status, cube status, etc.)
>>>>>
>>>>> Each of the kylin instances has a kylin.server.mode entry in
>>>>> conf/kylin.properties specifying the runtime mode, it has three options:
>>>>> 1.
>>>>> "job" for running job engine only 2. "query" for running query engine
>>>>> only
>>>>> and 3 "all" for running both. Notice that only one server can run the
>>>>> job
>>>>> engine("all" mode or "job" mode), the others must all be "query" mode.
>>>>>
>>>>> A typical scenario is depicted in the attachment chart.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>>
>>>>> *Bin Mahone | 马洪宾*
>>>>> Apache Kylin: http://kylin.io
>>>>> Github: https://github.com/binmahone
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>>
>>>> *Bin Mahone | 马洪宾*
>>>> Apache Kylin: http://kylin.io
>>>> Github: https://github.com/binmahone
>>
>>
>> --
>> -------
>> Wei Hu
>>
Re: Kylin and Remote Server
Posted by hongbin ma <ma...@apache.org>.
Well organized!
It's definitely helpful to other community members,
thanks!
--
Regards,
*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone
Re: Kylin and Remote Server
Posted by Diego Pinheiro <di...@gmail.com>.
Hi,
(I hope this can help someone)
after a long time, I could ran Kylin from a client machine and send
the jobs to my cluster (VM HDP 2.1).
Since I am a newbie in Hadoop, It took me a lot to configure my client
machine from scratch. I did not install any package from Hortonworks
or Cloudera. The step by step is quite long, but I would like to share
with you to help others that could be in the same situation that I
was. I describe it here:
https://gist.github.com/diegoValhalla/b54fa0d16a904f80b1f5 (as I do
not have a blog). All configuration files that I used are attached. I
checked that this properties are the minimum to set the environment.
When setting Kylin client machine I faced to major problems:
hive-hcatalog-core.jar and updateJobCounter() function. The first
problem was happening due to different class definitions[1] so I
replace hive-hcatalor-core.jar by HDP 2.1 one. Then, I got the problem
in updateJobCounter() method[2]. I guess this is related to some
misconfiguration in hadoop properties. However, this does not change
cube build. This was how I ran Kylin.
If anyone has ran Kylin in a different way, please, let me know.
[1] http://mail-archives.apache.org/mod_mbox/kylin-dev/201506.mbox/%3CBN1PR02MB0706A02D7C0EE200FC7D9E3CAB60@BN1PR02MB070.namprd02.prod.outlook.com%3E
[2] http://mail-archives.apache.org/mod_mbox/kylin-dev/201509.mbox/%3CCAO44d8EL7=XHc_ZgNpLttz7unur0P14daOu1nb=b4i0DhCiv1g@mail.gmail.com%3E
On Mon, Aug 31, 2015 at 11:19 PM, Shi, Shaofeng <sh...@ebay.com> wrote:
> or you can change “kylin.job.hive.database.for.intermediatetable” back to
> “default” to bypass this issue;
>
> On 9/1/15, 8:59 AM, "Shi, Shaofeng" <sh...@ebay.com> wrote:
>
>>should be related with: https://issues.apache.org/jira/browse/KYLIN-975
>>
>>The patch is available now; You can make a new build by pull 0.7-staging
>>branch and then run scripts/package.sh
>>
>>On 9/1/15, 7:46 AM, "Diego Pinheiro" <di...@gmail.com> wrote:
>>
>>>After some changes, I am getting the following error:
>>>
>>>[pool-5-thread-2]:[2015-08-31
>>>19:42:13,394][ERROR][org.apache.kylin.job.hadoop.cube.FactDistinctColumns
>>>J
>>>ob.run(FactDistinctColumnsJob.java:83)]
>>>- error in FactDistinctColumnsJob
>>>java.io.IOException:
>>>NoSuchObjectException(message:DEFAULT.kylin_intermediate_kylin_sales_cube
>>>_
>>>desc_20120101000000_20150727000000_a7572e8b_2c6a_4904_a19b_d74933e6d008
>>>table not found)
>>> at
>>>org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputForm
>>>a
>>>t.java:97)
>>> at
>>>org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputForm
>>>a
>>>t.java:51)
>>> at
>>>org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.setupMapper(FactD
>>>i
>>>stinctColumnsJob.java:101)
>>> at
>>>org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctC
>>>o
>>>lumnsJob.java:74)
>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>> at
>>>org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutabl
>>>e
>>>.java:112)
>>> at
>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecuta
>>>b
>>>le.java:107)
>>> at
>>>org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultCha
>>>i
>>>nedExecutable.java:50)
>>> at
>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecuta
>>>b
>>>le.java:107)
>>> at
>>>org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(Defau
>>>l
>>>tScheduler.java:132)
>>> at
>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java
>>>:
>>>1145)
>>> at
>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav
>>>a
>>>:615)
>>> at java.lang.Thread.run(Thread.java:745)
>>>Caused by:
>>>NoSuchObjectException(message:DEFAULT.kylin_intermediate_kylin_sales_cube
>>>_
>>>desc_20120101000000_20150727000000_a7572e8b_2c6a_4904_a19b_d74933e6d008
>>>table not found)
>>> at
>>>org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveM
>>>e
>>>taStore.java:1560)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java
>>>:
>>>57)
>>> at
>>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI
>>>m
>>>pl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>> at
>>>org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHan
>>>d
>>>ler.java:105)
>>> at com.sun.proxy.$Proxy45.get_table(Unknown Source)
>>> at
>>>org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaSto
>>>r
>>>eClient.java:997)
>>> at
>>>org.apache.hive.hcatalog.common.HCatUtil.getTable(HCatUtil.java:191)
>>> at
>>>org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(Initia
>>>l
>>>izeInput.java:105)
>>> at
>>>org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInp
>>>u
>>>t.java:86)
>>> at
>>>org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputForm
>>>a
>>>t.java:95)
>>> ... 13 more
>>>
>>>Did you already face it?
>>>
>>>
>>>On Wed, Aug 26, 2015 at 11:20 PM, Diego Pinheiro
>>><di...@gmail.com> wrote:
>>>> @DroopyHoo, it is good to know that since we are planning to change
>>>> authentication, but this is not the cause of my error. Since hongbin
>>>> ma comments, I changed my kylin properties to use the same as sandbox
>>>> (which is my VM HDP 2.1). So, I am using LDAP auth for now.
>>>>
>>>> @hongbin ma, kylin.job.run.as.remote.cmd was set to true indeed. I
>>>> changed it to false, but I am still getting the same errors:
>>>>
>>>> [pool-5-thread-2]:[2015-08-25
>>>>
>>>>19:16:03,679][ERROR][org.apache.kylin.job.tools.HadoopStatusChecker.chec
>>>>k
>>>>Status(HadoopStatusChecker.java:91)]
>>>> - error check status
>>>> java.net.ConnectException: Connection refused
>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>> at
>>>>java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:
>>>>3
>>>>39)
>>>> at
>>>>java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImp
>>>>l
>>>>.java:198)
>>>> at
>>>>java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:18
>>>>2
>>>>)
>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>>> at java.net.Socket.connect(Socket.java:579)
>>>>
>>>> org.apache.kylin.job.exception.ExecuteException:
>>>>java.lang.NullPointerException
>>>> at
>>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecut
>>>>a
>>>>ble.java:110)
>>>> at
>>>>org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultCh
>>>>a
>>>>inedExecutable.java:50)
>>>> at
>>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecut
>>>>a
>>>>ble.java:106)
>>>>
>>>> Just to resume, this is my current environment:
>>>>
>>>> - one machine with HDP 2.1 (which I already ran Kylin 0.7.2 and it
>>>>worked well);
>>>> - HDP 2.1 is my cluster, my machine is the client;
>>>> - I did not change any configuration in my cluster;
>>>> - Kylin is in my client machine with the default configuration files;
>>>> - I downloaded Hadoop 2.4.0, HBase 0.98.0 and Hive 0.13.1 to use in my
>>>> client as a Hadoop CLI;
>>>> - In the template of these files: hadoop/core-site.xml,
>>>> hbase/hbase-site.xml and hive/hive-site.xml, I just added one or two
>>>> properties to set my cluster IP. All these tools are apparently ok
>>>> since I can access my cluster from them.
>>>>
>>>> Whereas my cluster is ok, I was thinking if my problem is in Hadoop
>>>> CLI configuration files in the client...what changes have you guys
>>>> done in those three configuration files? Did you also change
>>>> yarn-site.xml or hdfs-site.xml?
>>>>
>>>> HDP 2.1 makes Kylin installation really easy. But, since I am new in
>>>> Hadoop, I am facing these problems when setting up the client machine.
>>>>
>>>>
>>>> On Wed, Aug 26, 2015 at 5:40 AM, DroopyHoo <ol...@gmail.com> wrote:
>>>>> Hi Diego
>>>>>
>>>>> We met this error stack when deploying in our Hadoop enviroment(not
>>>>> sandbox). The problem we met is the function for checking MR job
>>>>>status do
>>>>> not support kerberos auth (our hadoop cluster use kerberos service).
>>>>>So we
>>>>> made some changes to this part of source code.
>>>>>
>>>>> I'm not sure whether this case could help you to analyse the problem.
>>>>>
>>>>> 在 15/8/26 上午10:46, Diego Pinheiro 写道:
>>>>>
>>>>>> Hi Bin Mahone,
>>>>>>
>>>>>> sorry for the late reply. Thank you for your support. I didn't know
>>>>>> about Kylin instances. It is really interesting.
>>>>>>
>>>>>> However, let me ask you, I was setting up my hadoop client machine
>>>>>> with Kylin to communicate to my sandbox. But things are not working
>>>>>> well.
>>>>>>
>>>>>> I have installed hadoop 2.4.0, hbase 0.98.0 and hive 0.13.1. All them
>>>>>> are working and I can access my "remote server" from my client
>>>>>>machine
>>>>>> (actually, I set kylin as sandbox since all my hadoop cli is pointing
>>>>>> to my sandbox). Then, Kylin was built and everything was ok until I
>>>>>> tried to build the cube.
>>>>>>
>>>>>> I got the following errors always in the second step of cube build:
>>>>>>
>>>>>> [pool-5-thread-2]:[2015-08-25
>>>>>>
>>>>>>
>>>>>>19:16:03,679][ERROR][org.apache.kylin.job.tools.HadoopStatusChecker.ch
>>>>>>e
>>>>>>ckStatus(HadoopStatusChecker.java:91)]
>>>>>> - error check status
>>>>>> java.net.ConnectException: Connection refused
>>>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>>> at
>>>>>>
>>>>>>java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.jav
>>>>>>a
>>>>>>:339)
>>>>>> at
>>>>>>
>>>>>>java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketI
>>>>>>m
>>>>>>pl.java:198)
>>>>>> at
>>>>>>
>>>>>>java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:
>>>>>>1
>>>>>>82)
>>>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>>>>> at java.net.Socket.connect(Socket.java:579)
>>>>>> at java.net.Socket.connect(Socket.java:528)
>>>>>> at java.net.Socket.<init>(Socket.java:425)
>>>>>> at java.net.Socket.<init>(Socket.java:280)
>>>>>> at
>>>>>>
>>>>>>org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.cr
>>>>>>e
>>>>>>ateSocket(DefaultProtocolSocketFactory.java:80)
>>>>>> at
>>>>>>
>>>>>>org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.cr
>>>>>>e
>>>>>>ateSocket(DefaultProtocolSocketFactory.java:122)
>>>>>> at
>>>>>>
>>>>>>org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:
>>>>>>7
>>>>>>07)
>>>>>> at
>>>>>>
>>>>>>org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(Http
>>>>>>M
>>>>>>ethodDirector.java:387)
>>>>>> at
>>>>>>
>>>>>>org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMet
>>>>>>h
>>>>>>odDirector.java:171)
>>>>>> at
>>>>>>
>>>>>>org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java
>>>>>>:
>>>>>>397)
>>>>>> at
>>>>>>
>>>>>>org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java
>>>>>>:
>>>>>>323)
>>>>>> at
>>>>>>
>>>>>>org.apache.kylin.job.tools.HadoopStatusGetter.getHttpResponse(HadoopSt
>>>>>>a
>>>>>>tusGetter.java:78)
>>>>>> at
>>>>>>
>>>>>>org.apache.kylin.job.tools.HadoopStatusGetter.get(HadoopStatusGetter.j
>>>>>>a
>>>>>>va:55)
>>>>>> at
>>>>>>
>>>>>>org.apache.kylin.job.tools.HadoopStatusChecker.checkStatus(HadoopStatu
>>>>>>s
>>>>>>Checker.java:56)
>>>>>> at
>>>>>>
>>>>>>org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecut
>>>>>>a
>>>>>>ble.java:136)
>>>>>> at
>>>>>>
>>>>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExec
>>>>>>u
>>>>>>table.java:106)
>>>>>> at
>>>>>>
>>>>>>org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(Default
>>>>>>C
>>>>>>hainedExecutable.java:50)
>>>>>> at
>>>>>>
>>>>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExec
>>>>>>u
>>>>>>table.java:106)
>>>>>> at
>>>>>>
>>>>>>org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(De
>>>>>>f
>>>>>>aultScheduler.java:133)
>>>>>> at
>>>>>>
>>>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
>>>>>>a
>>>>>>va:1145)
>>>>>> at
>>>>>>
>>>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
>>>>>>j
>>>>>>ava:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>
>>>>>> org.apache.kylin.job.exception.ExecuteException:
>>>>>> java.lang.NullPointerException
>>>>>> at
>>>>>>
>>>>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExec
>>>>>>u
>>>>>>table.java:110)
>>>>>> at
>>>>>>
>>>>>>org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(Default
>>>>>>C
>>>>>>hainedExecutable.java:50)
>>>>>> at
>>>>>>
>>>>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExec
>>>>>>u
>>>>>>table.java:106)
>>>>>> at
>>>>>>
>>>>>>org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(De
>>>>>>f
>>>>>>aultScheduler.java:133)
>>>>>> at
>>>>>>
>>>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
>>>>>>a
>>>>>>va:1145)
>>>>>> at
>>>>>>
>>>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
>>>>>>j
>>>>>>ava:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> Caused by: java.lang.NullPointerException
>>>>>> at
>>>>>>
>>>>>>org.apache.kylin.job.common.MapReduceExecutable.onExecuteStart(MapRedu
>>>>>>c
>>>>>>eExecutable.java:73)
>>>>>> at
>>>>>>
>>>>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExec
>>>>>>u
>>>>>>table.java:105)
>>>>>> ... 6 more
>>>>>>
>>>>>> Do you have any thoughts about these errors? (detailed log is
>>>>>>attached)
>>>>>>
>>>>>>
>>>>>> On Fri, Aug 21, 2015 at 3:21 AM, hongbin ma <ma...@apache.org>
>>>>>>wrote:
>>>>>>>
>>>>>>> the document is summarized at
>>>>>>> http://kylin.incubator.apache.org/docs/install/kylin_cluster.html
>>>>>>>
>>>>>>> On Fri, Aug 21, 2015 at 1:51 PM, hongbin ma <ma...@apache.org>
>>>>>>>wrote:
>>>>>>>
>>>>>>>> hi Diego
>>>>>>>>
>>>>>>>> the config "kylin.job.run.as.remote.cmd" is somehow ambiguous , it
>>>>>>>>is
>>>>>>>> enabled when you cannot run Kylin server on the same machine as
>>>>>>>>your
>>>>>>>> hadoop
>>>>>>>> CLI, for example, if you're starting Kylin from you local IDE, and
>>>>>>>>you
>>>>>>>> hadoop CLI is a sandbox in another machine, this is the "remote"
>>>>>>>>case.
>>>>>>>>
>>>>>>>> In most of the production deployments we suggest using
>>>>>>>>'"non-remote"
>>>>>>>> mode,
>>>>>>>> that is, kylin instance is started on the hadoop CLI. The picture
>>>>>>>> depicts
>>>>>>>> the scenario:
>>>>>>>>
>>>>>>>>
>>>>>>>>https://github.com/apache/incubator-kylin/blob/0.7/website/images/in
>>>>>>>>s
>>>>>>>>tall/on_cli_install_scene.png
>>>>>>>>
>>>>>>>> Kylin instances are stateless, the runtime state is saved in its
>>>>>>>> "Metadata Store" in hbase (kylin.metadata.url config in
>>>>>>>> conf/kylin.properties). For load balance considerations it is
>>>>>>>>possible
>>>>>>>> to
>>>>>>>> start multiple Kylin instances sharing the same metadata store
>>>>>>>>(thus
>>>>>>>> sharing the same state on table schemas, job status, cube status,
>>>>>>>>etc.)
>>>>>>>>
>>>>>>>> Each of the kylin instances has a kylin.server.mode entry in
>>>>>>>> conf/kylin.properties specifying the runtime mode, it has three
>>>>>>>>options:
>>>>>>>> 1.
>>>>>>>> "job" for running job engine only 2. "query" for running query
>>>>>>>>engine
>>>>>>>> only
>>>>>>>> and 3 "all" for running both. Notice that only one server can run
>>>>>>>>the
>>>>>>>> job
>>>>>>>> engine("all" mode or "job" mode), the others must all be "query"
>>>>>>>>mode.
>>>>>>>>
>>>>>>>> A typical scenario is depicted in the attachment chart.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> *Bin Mahone | 马洪宾*
>>>>>>>> Apache Kylin: http://kylin.io
>>>>>>>> Github: https://github.com/binmahone
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>>
>>>>>>> *Bin Mahone | 马洪宾*
>>>>>>> Apache Kylin: http://kylin.io
>>>>>>> Github: https://github.com/binmahone
>>>>>
>>>>>
>>>>> --
>>>>> -------
>>>>> Wei Hu
>>
>
Re: Kylin and Remote Server
Posted by "Shi, Shaofeng" <sh...@ebay.com>.
or you can change “kylin.job.hive.database.for.intermediatetable” back to
“default” to bypass this issue;
On 9/1/15, 8:59 AM, "Shi, Shaofeng" <sh...@ebay.com> wrote:
>should be related with: https://issues.apache.org/jira/browse/KYLIN-975
>
>The patch is available now; You can make a new build by pull 0.7-staging
>branch and then run scripts/package.sh
>
>On 9/1/15, 7:46 AM, "Diego Pinheiro" <di...@gmail.com> wrote:
>
>>After some changes, I am getting the following error:
>>
>>[pool-5-thread-2]:[2015-08-31
>>19:42:13,394][ERROR][org.apache.kylin.job.hadoop.cube.FactDistinctColumns
>>J
>>ob.run(FactDistinctColumnsJob.java:83)]
>>- error in FactDistinctColumnsJob
>>java.io.IOException:
>>NoSuchObjectException(message:DEFAULT.kylin_intermediate_kylin_sales_cube
>>_
>>desc_20120101000000_20150727000000_a7572e8b_2c6a_4904_a19b_d74933e6d008
>>table not found)
>> at
>>org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputForm
>>a
>>t.java:97)
>> at
>>org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputForm
>>a
>>t.java:51)
>> at
>>org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.setupMapper(FactD
>>i
>>stinctColumnsJob.java:101)
>> at
>>org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctC
>>o
>>lumnsJob.java:74)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>> at
>>org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutabl
>>e
>>.java:112)
>> at
>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecuta
>>b
>>le.java:107)
>> at
>>org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultCha
>>i
>>nedExecutable.java:50)
>> at
>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecuta
>>b
>>le.java:107)
>> at
>>org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(Defau
>>l
>>tScheduler.java:132)
>> at
>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java
>>:
>>1145)
>> at
>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav
>>a
>>:615)
>> at java.lang.Thread.run(Thread.java:745)
>>Caused by:
>>NoSuchObjectException(message:DEFAULT.kylin_intermediate_kylin_sales_cube
>>_
>>desc_20120101000000_20150727000000_a7572e8b_2c6a_4904_a19b_d74933e6d008
>>table not found)
>> at
>>org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveM
>>e
>>taStore.java:1560)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java
>>:
>>57)
>> at
>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI
>>m
>>pl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>>org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHan
>>d
>>ler.java:105)
>> at com.sun.proxy.$Proxy45.get_table(Unknown Source)
>> at
>>org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaSto
>>r
>>eClient.java:997)
>> at
>>org.apache.hive.hcatalog.common.HCatUtil.getTable(HCatUtil.java:191)
>> at
>>org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(Initia
>>l
>>izeInput.java:105)
>> at
>>org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInp
>>u
>>t.java:86)
>> at
>>org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputForm
>>a
>>t.java:95)
>> ... 13 more
>>
>>Did you already face it?
>>
>>
>>On Wed, Aug 26, 2015 at 11:20 PM, Diego Pinheiro
>><di...@gmail.com> wrote:
>>> @DroopyHoo, it is good to know that since we are planning to change
>>> authentication, but this is not the cause of my error. Since hongbin
>>> ma comments, I changed my kylin properties to use the same as sandbox
>>> (which is my VM HDP 2.1). So, I am using LDAP auth for now.
>>>
>>> @hongbin ma, kylin.job.run.as.remote.cmd was set to true indeed. I
>>> changed it to false, but I am still getting the same errors:
>>>
>>> [pool-5-thread-2]:[2015-08-25
>>>
>>>19:16:03,679][ERROR][org.apache.kylin.job.tools.HadoopStatusChecker.chec
>>>k
>>>Status(HadoopStatusChecker.java:91)]
>>> - error check status
>>> java.net.ConnectException: Connection refused
>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>> at
>>>java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:
>>>3
>>>39)
>>> at
>>>java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImp
>>>l
>>>.java:198)
>>> at
>>>java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:18
>>>2
>>>)
>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>> at java.net.Socket.connect(Socket.java:579)
>>>
>>> org.apache.kylin.job.exception.ExecuteException:
>>>java.lang.NullPointerException
>>> at
>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecut
>>>a
>>>ble.java:110)
>>> at
>>>org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultCh
>>>a
>>>inedExecutable.java:50)
>>> at
>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecut
>>>a
>>>ble.java:106)
>>>
>>> Just to resume, this is my current environment:
>>>
>>> - one machine with HDP 2.1 (which I already ran Kylin 0.7.2 and it
>>>worked well);
>>> - HDP 2.1 is my cluster, my machine is the client;
>>> - I did not change any configuration in my cluster;
>>> - Kylin is in my client machine with the default configuration files;
>>> - I downloaded Hadoop 2.4.0, HBase 0.98.0 and Hive 0.13.1 to use in my
>>> client as a Hadoop CLI;
>>> - In the template of these files: hadoop/core-site.xml,
>>> hbase/hbase-site.xml and hive/hive-site.xml, I just added one or two
>>> properties to set my cluster IP. All these tools are apparently ok
>>> since I can access my cluster from them.
>>>
>>> Whereas my cluster is ok, I was thinking if my problem is in Hadoop
>>> CLI configuration files in the client...what changes have you guys
>>> done in those three configuration files? Did you also change
>>> yarn-site.xml or hdfs-site.xml?
>>>
>>> HDP 2.1 makes Kylin installation really easy. But, since I am new in
>>> Hadoop, I am facing these problems when setting up the client machine.
>>>
>>>
>>> On Wed, Aug 26, 2015 at 5:40 AM, DroopyHoo <ol...@gmail.com> wrote:
>>>> Hi Diego
>>>>
>>>> We met this error stack when deploying in our Hadoop enviroment(not
>>>> sandbox). The problem we met is the function for checking MR job
>>>>status do
>>>> not support kerberos auth (our hadoop cluster use kerberos service).
>>>>So we
>>>> made some changes to this part of source code.
>>>>
>>>> I'm not sure whether this case could help you to analyse the problem.
>>>>
>>>> 在 15/8/26 上午10:46, Diego Pinheiro 写道:
>>>>
>>>>> Hi Bin Mahone,
>>>>>
>>>>> sorry for the late reply. Thank you for your support. I didn't know
>>>>> about Kylin instances. It is really interesting.
>>>>>
>>>>> However, let me ask you, I was setting up my hadoop client machine
>>>>> with Kylin to communicate to my sandbox. But things are not working
>>>>> well.
>>>>>
>>>>> I have installed hadoop 2.4.0, hbase 0.98.0 and hive 0.13.1. All them
>>>>> are working and I can access my "remote server" from my client
>>>>>machine
>>>>> (actually, I set kylin as sandbox since all my hadoop cli is pointing
>>>>> to my sandbox). Then, Kylin was built and everything was ok until I
>>>>> tried to build the cube.
>>>>>
>>>>> I got the following errors always in the second step of cube build:
>>>>>
>>>>> [pool-5-thread-2]:[2015-08-25
>>>>>
>>>>>
>>>>>19:16:03,679][ERROR][org.apache.kylin.job.tools.HadoopStatusChecker.ch
>>>>>e
>>>>>ckStatus(HadoopStatusChecker.java:91)]
>>>>> - error check status
>>>>> java.net.ConnectException: Connection refused
>>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>> at
>>>>>
>>>>>java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.jav
>>>>>a
>>>>>:339)
>>>>> at
>>>>>
>>>>>java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketI
>>>>>m
>>>>>pl.java:198)
>>>>> at
>>>>>
>>>>>java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:
>>>>>1
>>>>>82)
>>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>>>> at java.net.Socket.connect(Socket.java:579)
>>>>> at java.net.Socket.connect(Socket.java:528)
>>>>> at java.net.Socket.<init>(Socket.java:425)
>>>>> at java.net.Socket.<init>(Socket.java:280)
>>>>> at
>>>>>
>>>>>org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.cr
>>>>>e
>>>>>ateSocket(DefaultProtocolSocketFactory.java:80)
>>>>> at
>>>>>
>>>>>org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.cr
>>>>>e
>>>>>ateSocket(DefaultProtocolSocketFactory.java:122)
>>>>> at
>>>>>
>>>>>org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:
>>>>>7
>>>>>07)
>>>>> at
>>>>>
>>>>>org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(Http
>>>>>M
>>>>>ethodDirector.java:387)
>>>>> at
>>>>>
>>>>>org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMet
>>>>>h
>>>>>odDirector.java:171)
>>>>> at
>>>>>
>>>>>org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java
>>>>>:
>>>>>397)
>>>>> at
>>>>>
>>>>>org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java
>>>>>:
>>>>>323)
>>>>> at
>>>>>
>>>>>org.apache.kylin.job.tools.HadoopStatusGetter.getHttpResponse(HadoopSt
>>>>>a
>>>>>tusGetter.java:78)
>>>>> at
>>>>>
>>>>>org.apache.kylin.job.tools.HadoopStatusGetter.get(HadoopStatusGetter.j
>>>>>a
>>>>>va:55)
>>>>> at
>>>>>
>>>>>org.apache.kylin.job.tools.HadoopStatusChecker.checkStatus(HadoopStatu
>>>>>s
>>>>>Checker.java:56)
>>>>> at
>>>>>
>>>>>org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecut
>>>>>a
>>>>>ble.java:136)
>>>>> at
>>>>>
>>>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExec
>>>>>u
>>>>>table.java:106)
>>>>> at
>>>>>
>>>>>org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(Default
>>>>>C
>>>>>hainedExecutable.java:50)
>>>>> at
>>>>>
>>>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExec
>>>>>u
>>>>>table.java:106)
>>>>> at
>>>>>
>>>>>org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(De
>>>>>f
>>>>>aultScheduler.java:133)
>>>>> at
>>>>>
>>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
>>>>>a
>>>>>va:1145)
>>>>> at
>>>>>
>>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
>>>>>j
>>>>>ava:615)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> org.apache.kylin.job.exception.ExecuteException:
>>>>> java.lang.NullPointerException
>>>>> at
>>>>>
>>>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExec
>>>>>u
>>>>>table.java:110)
>>>>> at
>>>>>
>>>>>org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(Default
>>>>>C
>>>>>hainedExecutable.java:50)
>>>>> at
>>>>>
>>>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExec
>>>>>u
>>>>>table.java:106)
>>>>> at
>>>>>
>>>>>org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(De
>>>>>f
>>>>>aultScheduler.java:133)
>>>>> at
>>>>>
>>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
>>>>>a
>>>>>va:1145)
>>>>> at
>>>>>
>>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
>>>>>j
>>>>>ava:615)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>> Caused by: java.lang.NullPointerException
>>>>> at
>>>>>
>>>>>org.apache.kylin.job.common.MapReduceExecutable.onExecuteStart(MapRedu
>>>>>c
>>>>>eExecutable.java:73)
>>>>> at
>>>>>
>>>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExec
>>>>>u
>>>>>table.java:105)
>>>>> ... 6 more
>>>>>
>>>>> Do you have any thoughts about these errors? (detailed log is
>>>>>attached)
>>>>>
>>>>>
>>>>> On Fri, Aug 21, 2015 at 3:21 AM, hongbin ma <ma...@apache.org>
>>>>>wrote:
>>>>>>
>>>>>> the document is summarized at
>>>>>> http://kylin.incubator.apache.org/docs/install/kylin_cluster.html
>>>>>>
>>>>>> On Fri, Aug 21, 2015 at 1:51 PM, hongbin ma <ma...@apache.org>
>>>>>>wrote:
>>>>>>
>>>>>>> hi Diego
>>>>>>>
>>>>>>> the config "kylin.job.run.as.remote.cmd" is somehow ambiguous , it
>>>>>>>is
>>>>>>> enabled when you cannot run Kylin server on the same machine as
>>>>>>>your
>>>>>>> hadoop
>>>>>>> CLI, for example, if you're starting Kylin from you local IDE, and
>>>>>>>you
>>>>>>> hadoop CLI is a sandbox in another machine, this is the "remote"
>>>>>>>case.
>>>>>>>
>>>>>>> In most of the production deployments we suggest using
>>>>>>>'"non-remote"
>>>>>>> mode,
>>>>>>> that is, kylin instance is started on the hadoop CLI. The picture
>>>>>>> depicts
>>>>>>> the scenario:
>>>>>>>
>>>>>>>
>>>>>>>https://github.com/apache/incubator-kylin/blob/0.7/website/images/in
>>>>>>>s
>>>>>>>tall/on_cli_install_scene.png
>>>>>>>
>>>>>>> Kylin instances are stateless, the runtime state is saved in its
>>>>>>> "Metadata Store" in hbase (kylin.metadata.url config in
>>>>>>> conf/kylin.properties). For load balance considerations it is
>>>>>>>possible
>>>>>>> to
>>>>>>> start multiple Kylin instances sharing the same metadata store
>>>>>>>(thus
>>>>>>> sharing the same state on table schemas, job status, cube status,
>>>>>>>etc.)
>>>>>>>
>>>>>>> Each of the kylin instances has a kylin.server.mode entry in
>>>>>>> conf/kylin.properties specifying the runtime mode, it has three
>>>>>>>options:
>>>>>>> 1.
>>>>>>> "job" for running job engine only 2. "query" for running query
>>>>>>>engine
>>>>>>> only
>>>>>>> and 3 "all" for running both. Notice that only one server can run
>>>>>>>the
>>>>>>> job
>>>>>>> engine("all" mode or "job" mode), the others must all be "query"
>>>>>>>mode.
>>>>>>>
>>>>>>> A typical scenario is depicted in the attachment chart.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>>
>>>>>>> *Bin Mahone | 马洪宾*
>>>>>>> Apache Kylin: http://kylin.io
>>>>>>> Github: https://github.com/binmahone
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>>
>>>>>> *Bin Mahone | 马洪宾*
>>>>>> Apache Kylin: http://kylin.io
>>>>>> Github: https://github.com/binmahone
>>>>
>>>>
>>>> --
>>>> -------
>>>> Wei Hu
>
Re: Kylin and Remote Server
Posted by "Shi, Shaofeng" <sh...@ebay.com>.
should be related with: https://issues.apache.org/jira/browse/KYLIN-975
The patch is available now; You can make a new build by pull 0.7-staging
branch and then run scripts/package.sh
On 9/1/15, 7:46 AM, "Diego Pinheiro" <di...@gmail.com> wrote:
>After some changes, I am getting the following error:
>
>[pool-5-thread-2]:[2015-08-31
>19:42:13,394][ERROR][org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJ
>ob.run(FactDistinctColumnsJob.java:83)]
>- error in FactDistinctColumnsJob
>java.io.IOException:
>NoSuchObjectException(message:DEFAULT.kylin_intermediate_kylin_sales_cube_
>desc_20120101000000_20150727000000_a7572e8b_2c6a_4904_a19b_d74933e6d008
>table not found)
> at
>org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputForma
>t.java:97)
> at
>org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputForma
>t.java:51)
> at
>org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.setupMapper(FactDi
>stinctColumnsJob.java:101)
> at
>org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctCo
>lumnsJob.java:74)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
>org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable
>.java:112)
> at
>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutab
>le.java:107)
> at
>org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChai
>nedExecutable.java:50)
> at
>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutab
>le.java:107)
> at
>org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(Defaul
>tScheduler.java:132)
> at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
>1145)
> at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java
>:615)
> at java.lang.Thread.run(Thread.java:745)
>Caused by:
>NoSuchObjectException(message:DEFAULT.kylin_intermediate_kylin_sales_cube_
>desc_20120101000000_20150727000000_a7572e8b_2c6a_4904_a19b_d74933e6d008
>table not found)
> at
>org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMe
>taStore.java:1560)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>57)
> at
>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>pl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
>org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHand
>ler.java:105)
> at com.sun.proxy.$Proxy45.get_table(Unknown Source)
> at
>org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStor
>eClient.java:997)
> at
>org.apache.hive.hcatalog.common.HCatUtil.getTable(HCatUtil.java:191)
> at
>org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(Initial
>izeInput.java:105)
> at
>org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInpu
>t.java:86)
> at
>org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputForma
>t.java:95)
> ... 13 more
>
>Did you already face it?
>
>
>On Wed, Aug 26, 2015 at 11:20 PM, Diego Pinheiro
><di...@gmail.com> wrote:
>> @DroopyHoo, it is good to know that since we are planning to change
>> authentication, but this is not the cause of my error. Since hongbin
>> ma comments, I changed my kylin properties to use the same as sandbox
>> (which is my VM HDP 2.1). So, I am using LDAP auth for now.
>>
>> @hongbin ma, kylin.job.run.as.remote.cmd was set to true indeed. I
>> changed it to false, but I am still getting the same errors:
>>
>> [pool-5-thread-2]:[2015-08-25
>>
>>19:16:03,679][ERROR][org.apache.kylin.job.tools.HadoopStatusChecker.check
>>Status(HadoopStatusChecker.java:91)]
>> - error check status
>> java.net.ConnectException: Connection refused
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at
>>java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:3
>>39)
>> at
>>java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl
>>.java:198)
>> at
>>java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182
>>)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>> at java.net.Socket.connect(Socket.java:579)
>>
>> org.apache.kylin.job.exception.ExecuteException:
>>java.lang.NullPointerException
>> at
>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecuta
>>ble.java:110)
>> at
>>org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultCha
>>inedExecutable.java:50)
>> at
>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecuta
>>ble.java:106)
>>
>> Just to resume, this is my current environment:
>>
>> - one machine with HDP 2.1 (which I already ran Kylin 0.7.2 and it
>>worked well);
>> - HDP 2.1 is my cluster, my machine is the client;
>> - I did not change any configuration in my cluster;
>> - Kylin is in my client machine with the default configuration files;
>> - I downloaded Hadoop 2.4.0, HBase 0.98.0 and Hive 0.13.1 to use in my
>> client as a Hadoop CLI;
>> - In the template of these files: hadoop/core-site.xml,
>> hbase/hbase-site.xml and hive/hive-site.xml, I just added one or two
>> properties to set my cluster IP. All these tools are apparently ok
>> since I can access my cluster from them.
>>
>> Whereas my cluster is ok, I was thinking if my problem is in Hadoop
>> CLI configuration files in the client...what changes have you guys
>> done in those three configuration files? Did you also change
>> yarn-site.xml or hdfs-site.xml?
>>
>> HDP 2.1 makes Kylin installation really easy. But, since I am new in
>> Hadoop, I am facing these problems when setting up the client machine.
>>
>>
>> On Wed, Aug 26, 2015 at 5:40 AM, DroopyHoo <ol...@gmail.com> wrote:
>>> Hi Diego
>>>
>>> We met this error stack when deploying in our Hadoop enviroment(not
>>> sandbox). The problem we met is the function for checking MR job
>>>status do
>>> not support kerberos auth (our hadoop cluster use kerberos service).
>>>So we
>>> made some changes to this part of source code.
>>>
>>> I'm not sure whether this case could help you to analyse the problem.
>>>
>>> 在 15/8/26 上午10:46, Diego Pinheiro 写道:
>>>
>>>> Hi Bin Mahone,
>>>>
>>>> sorry for the late reply. Thank you for your support. I didn't know
>>>> about Kylin instances. It is really interesting.
>>>>
>>>> However, let me ask you, I was setting up my hadoop client machine
>>>> with Kylin to communicate to my sandbox. But things are not working
>>>> well.
>>>>
>>>> I have installed hadoop 2.4.0, hbase 0.98.0 and hive 0.13.1. All them
>>>> are working and I can access my "remote server" from my client machine
>>>> (actually, I set kylin as sandbox since all my hadoop cli is pointing
>>>> to my sandbox). Then, Kylin was built and everything was ok until I
>>>> tried to build the cube.
>>>>
>>>> I got the following errors always in the second step of cube build:
>>>>
>>>> [pool-5-thread-2]:[2015-08-25
>>>>
>>>>
>>>>19:16:03,679][ERROR][org.apache.kylin.job.tools.HadoopStatusChecker.che
>>>>ckStatus(HadoopStatusChecker.java:91)]
>>>> - error check status
>>>> java.net.ConnectException: Connection refused
>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>> at
>>>>
>>>>java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java
>>>>:339)
>>>> at
>>>>
>>>>java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketIm
>>>>pl.java:198)
>>>> at
>>>>
>>>>java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:1
>>>>82)
>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>>> at java.net.Socket.connect(Socket.java:579)
>>>> at java.net.Socket.connect(Socket.java:528)
>>>> at java.net.Socket.<init>(Socket.java:425)
>>>> at java.net.Socket.<init>(Socket.java:280)
>>>> at
>>>>
>>>>org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.cre
>>>>ateSocket(DefaultProtocolSocketFactory.java:80)
>>>> at
>>>>
>>>>org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.cre
>>>>ateSocket(DefaultProtocolSocketFactory.java:122)
>>>> at
>>>>
>>>>org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:7
>>>>07)
>>>> at
>>>>
>>>>org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpM
>>>>ethodDirector.java:387)
>>>> at
>>>>
>>>>org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMeth
>>>>odDirector.java:171)
>>>> at
>>>>
>>>>org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:
>>>>397)
>>>> at
>>>>
>>>>org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:
>>>>323)
>>>> at
>>>>
>>>>org.apache.kylin.job.tools.HadoopStatusGetter.getHttpResponse(HadoopSta
>>>>tusGetter.java:78)
>>>> at
>>>>
>>>>org.apache.kylin.job.tools.HadoopStatusGetter.get(HadoopStatusGetter.ja
>>>>va:55)
>>>> at
>>>>
>>>>org.apache.kylin.job.tools.HadoopStatusChecker.checkStatus(HadoopStatus
>>>>Checker.java:56)
>>>> at
>>>>
>>>>org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecuta
>>>>ble.java:136)
>>>> at
>>>>
>>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecu
>>>>table.java:106)
>>>> at
>>>>
>>>>org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultC
>>>>hainedExecutable.java:50)
>>>> at
>>>>
>>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecu
>>>>table.java:106)
>>>> at
>>>>
>>>>org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(Def
>>>>aultScheduler.java:133)
>>>> at
>>>>
>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja
>>>>va:1145)
>>>> at
>>>>
>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
>>>>ava:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>>
>>>> org.apache.kylin.job.exception.ExecuteException:
>>>> java.lang.NullPointerException
>>>> at
>>>>
>>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecu
>>>>table.java:110)
>>>> at
>>>>
>>>>org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultC
>>>>hainedExecutable.java:50)
>>>> at
>>>>
>>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecu
>>>>table.java:106)
>>>> at
>>>>
>>>>org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(Def
>>>>aultScheduler.java:133)
>>>> at
>>>>
>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja
>>>>va:1145)
>>>> at
>>>>
>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
>>>>ava:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> Caused by: java.lang.NullPointerException
>>>> at
>>>>
>>>>org.apache.kylin.job.common.MapReduceExecutable.onExecuteStart(MapReduc
>>>>eExecutable.java:73)
>>>> at
>>>>
>>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecu
>>>>table.java:105)
>>>> ... 6 more
>>>>
>>>> Do you have any thoughts about these errors? (detailed log is
>>>>attached)
>>>>
>>>>
>>>> On Fri, Aug 21, 2015 at 3:21 AM, hongbin ma <ma...@apache.org>
>>>>wrote:
>>>>>
>>>>> the document is summarized at
>>>>> http://kylin.incubator.apache.org/docs/install/kylin_cluster.html
>>>>>
>>>>> On Fri, Aug 21, 2015 at 1:51 PM, hongbin ma <ma...@apache.org>
>>>>>wrote:
>>>>>
>>>>>> hi Diego
>>>>>>
>>>>>> the config "kylin.job.run.as.remote.cmd" is somehow ambiguous , it
>>>>>>is
>>>>>> enabled when you cannot run Kylin server on the same machine as your
>>>>>> hadoop
>>>>>> CLI, for example, if you're starting Kylin from you local IDE, and
>>>>>>you
>>>>>> hadoop CLI is a sandbox in another machine, this is the "remote"
>>>>>>case.
>>>>>>
>>>>>> In most of the production deployments we suggest using '"non-remote"
>>>>>> mode,
>>>>>> that is, kylin instance is started on the hadoop CLI. The picture
>>>>>> depicts
>>>>>> the scenario:
>>>>>>
>>>>>>
>>>>>>https://github.com/apache/incubator-kylin/blob/0.7/website/images/ins
>>>>>>tall/on_cli_install_scene.png
>>>>>>
>>>>>> Kylin instances are stateless, the runtime state is saved in its
>>>>>> "Metadata Store" in hbase (kylin.metadata.url config in
>>>>>> conf/kylin.properties). For load balance considerations it is
>>>>>>possible
>>>>>> to
>>>>>> start multiple Kylin instances sharing the same metadata store (thus
>>>>>> sharing the same state on table schemas, job status, cube status,
>>>>>>etc.)
>>>>>>
>>>>>> Each of the kylin instances has a kylin.server.mode entry in
>>>>>> conf/kylin.properties specifying the runtime mode, it has three
>>>>>>options:
>>>>>> 1.
>>>>>> "job" for running job engine only 2. "query" for running query
>>>>>>engine
>>>>>> only
>>>>>> and 3 "all" for running both. Notice that only one server can run
>>>>>>the
>>>>>> job
>>>>>> engine("all" mode or "job" mode), the others must all be "query"
>>>>>>mode.
>>>>>>
>>>>>> A typical scenario is depicted in the attachment chart.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>>
>>>>>> *Bin Mahone | 马洪宾*
>>>>>> Apache Kylin: http://kylin.io
>>>>>> Github: https://github.com/binmahone
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>>
>>>>> *Bin Mahone | 马洪宾*
>>>>> Apache Kylin: http://kylin.io
>>>>> Github: https://github.com/binmahone
>>>
>>>
>>> --
>>> -------
>>> Wei Hu