You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@kylin.apache.org by BELLIER Jean-luc <je...@rte-france.com> on 2018/02/09 17:18:26 UTC

Cannot rebuild cubes

Hello,

I am unable to rebuild cubes. The logs give me the following error :

Exception in thread "Thread-13" java.lang.RuntimeException: java.lang.IllegalStateException: Cannot start job scheduler due to lack of job lock
at org.apache.kylin.rest.service.JobService$1.run(JobService.java:131)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Cannot start job scheduler due to lack of job lock
at org.apache.kylin.job.impl.threadpool.DefaultScheduler.init(DefaultScheduler.java:211)
at org.apache.kylin.rest.service.JobService$1.run(JobService.java:126)
... 1 more

As far as I can see on this topic, I understand that there should be a process that is still running or was not remove correctly.
I tried several 'kylin.sh stop' / 'kylin.sh start' with no effect.

I attach my kylin.out and kylin.log files.

Thank you in advance for your help.

Best regards,
Jean-Luc.

"Ce message est destin? exclusivement aux personnes ou entit?s auxquelles il est adress? et peut contenir des informations privil?gi?es ou confidentielles. Si vous avez re?u ce document par erreur, merci de nous l'indiquer par retour, de ne pas le transmettre et de proc?der ? sa destruction.

This message is solely intended for the use of the individual or entity to which it is addressed and may contain information that is privileged or confidential. If you have received this communication by error, please notify us immediately by electronic mail, do not disclose it and delete the original message."

RE: Cannot rebuild cubes

Posted by BELLIER Jean-luc <je...@rte-france.com>.

Hello Ted,

Apparently, there were modifications on the cluster this morning, so it was not stable.
I still have the same error on the screen, but the reason is different in the log file.

I send it in attachment.

Do you have any idea ?

Thank you in advance for your help.

Best regards,
Jean-Luc.


De : BELLIER Jean-luc
Envoyé : mardi 13 février 2018 09:29
À : 'user@kylin.apache.org' <us...@kylin.apache.org>
Objet : RE: Cannot rebuild cubes

Hello Ted,

It is OK now, it wxas just oa problem of rights on user hive on /kylin/metadata on HDFS.
I succeeded in created my first cube.
Unfortunately, I cannot query it. I selected the project, and the cube then moved to ‘Insight tab’ and typed : select count(*) from kylin_sales. I got the following message :
null while executing SQL: "select count(*) from kylin_sales LIMIT 50000"

the log says : ‘unknown source’

2018-02-13 09:21:16,854 ERROR [http-bio-7070-exec-6] controller.BasicController:57 :
org.apache.kylin.rest.exception.InternalErrorException: null
while executing SQL: "select count(*) from kylin_sales LIMIT 50000"
        at org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:479)
        at org.apache.kylin.rest.controller.QueryController.query(QueryController.java:78)
        at sun.reflect.GeneratedMethodAccessor205.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)

How could I solve this ? By rebuilding the model and then the cube ?
If yes, which commands/process shoud I use ?

Thank you for your help.

Best regards,
Jean-Luc.

De : BELLIER Jean-luc
Envoyé : lundi 12 février 2018 15:20
À : 'user@kylin.apache.org' <us...@kylin.apache.org>>
Objet : RE: Cannot rebuild cubes

Hello again Ted,

I did the following :

·         launched the command ‘beeline’

·         launched the command (under beeline) : !connect jdbc:hive2://<host>:<port>/<db>;principal=<Server_Principal_of_HiveServer2>
And I got the following message :

Driver: Hive JDBC (version 1.1.0-cdh5.14.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ

Could this be the source of the problem, and, if yes, how to correct it ?

Thank you for your help.

Best regards,
Jean-Luc.

De : BELLIER Jean-luc
Envoyé : lundi 12 février 2018 14:39
À : user@kylin.apache.org<ma...@kylin.apache.org>
Objet : RE: Cannot rebuild cubes

Hello again,

My test is not conclusive, because the time slice I chose was before 2012. There is no data before this year. So it is normal.
I did then another test, to get the data between 2010-01-01 and 2014-02-02, but I still get the same result : no data available on step 1. I attach my kylin.log file.

In my kylin.log file, I can find the following error message while trying to create the temp Hive tables :
2018-02-12 14:30:01,574 INFO  [Scheduler 481383589 Job 8a97abb9-69c0-4484-b2e0-ed801dcb845a-161] hive.CreateFlatHiveTableStep:38 : Connecting to jdbc:hive2://cloudera-manager.hades.rte-france.com:10000/
2018-02-12 14:30:01,754 INFO  [Scheduler 481383589 Job 8a97abb9-69c0-4484-b2e0-ed801dcb845a-161] hive.CreateFlatHiveTableStep:38 : Unknown HS2 problem when communicating with Thrift server.
2018-02-12 14:30:01,754 INFO  [Scheduler 481383589 Job 8a97abb9-69c0-4484-b2e0-ed801dcb845a-161] hive.CreateFlatHiveTableStep:38 : Error: Could not open client transport with JDBC Uri: jdbc:hive2://cloudera-manager.hades.rte-france.com:10000/: Peer indicated failure: Unsupported mechanism type PLAIN (state=08S01,code=0)
2018-02-12 14:30:01,754 INFO  [Scheduler 481383589 Job 8a97abb9-69c0-4484-b2e0-ed801dcb845a-161] hive.CreateFlatHiveTableStep:38 : Beeline version 1.1.0-cdh5.14.0 by Apache Hive
2018-02-12 14:30:11,729 DEBUG [localhost-startStop-1-SendThread(node4.hades.rte-france.com:2181)] zookeeper.ClientCnxn:717 : Got ping response for sessionid: 0x261895d4a1c0140 after 0ms

Could you please give me any clue ?

NB : my kylin.properties file remains the same as before (see parameters below).

Thank you for your help.
Have a good day.

Best regards,
Jean-Luc.

De : BELLIER Jean-luc [mailto:jean-luc.bellier@rte-france.com]
Envoyé : lundi 12 février 2018 12:29
À : user@kylin.apache.org<ma...@kylin.apache.org>
Objet : RE: Cannot rebuild cubes

Hello Ted

Thank you for your reply. What is the command to check what you are suggesting ?
In the meanwhile, I changed the way accessing to hive to ‘beeline’, and faced to write rights problems to HDFS.
This is now solved.

Now I try to build the cube ‘kylin_sales_cube’. All the tables are present in Hive, and they contain data. The metadata are also in HBase in ‘kylin_metadata’ table.
What I do not understand is that the job seems to run with no data. In the step #1, the log tells me ‘No Data Available’

[cid:image001.gif@01D3A4A6.AA66DA30][cid:image002.gif@01D3A4A6.AA66DA30]

Could it be a problem of rights ?

Here is my kylin.properties

## Hive client, valid value [cli, beeline]
kylin.source.hive.client=beeline
#
## Parameters for beeline client, only necessary if hive client is beeline
kylin.source.hive.beeline-params=-n root --hiveconf hive.security.authorization.sqlstd.confwhitelist.append='mapreduce.job.*|dfs.*' -u jdbc:hive2://cloudera-manager.hades.rte-france.com:10000/;principal=hive/cloudera-manager.hades.rte-france.com@HADES.RTE-FRANCE.COM
#
#kylin.source.hive.keep-flat-table=false
#
## Hive database name for putting the intermediate flat tables
kylin.source.hive.database-for-flat-table=default

The full log files have been attached in this mail (using ‘Diagnosis’ option) on the job in Web interface.

PS  : I noticed the presence of file ‘health-check.sh’ in /tmp/kylin on HDFS. Is is normal ?

Thank you in advance for your help.

Best regards,
Jean-Luc.

De : Ted Yu [mailto:yuzhihong@gmail.com]
Envoyé : vendredi 9 février 2018 22:28
À : user@kylin.apache.org<ma...@kylin.apache.org>
Objet : Re: Cannot rebuild cubes

In the log, I saw:

2018-02-09 17:39:57,634 ERROR [http-bio-7070-exec-8] controller.CubeController:337 : Segments overlap: kylin_sales_cube[20100101000000_20140102000000] and          kylin_sales_cube[20120101000000_20140102000000]
java.lang.IllegalStateException: Segments overlap: kylin_sales_cube[20100101000000_20140102000000] and kylin_sales_cube[20120101000000_20140102000000]
  at org.apache.kylin.metadata.model.Segments.validate(Segments.java:404)
  at org.apache.kylin.cube.CubeManager.updateCubeWithRetry(CubeManager.java:393)
  at org.apache.kylin.cube.CubeManager.updateCube(CubeManager.java:349)
  at org.apache.kylin.cube.CubeManager.appendSegment(CubeManager.java:485)
  at org.apache.kylin.cube.CubeManager.appendSegment(CubeManager.java:455)
  at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:239)

It seems there was some wrong segment in metadata.
Have you tried dumping out metadata and find the invalid segment ?

Cheers

On Fri, Feb 9, 2018 at 9:18 AM, BELLIER Jean-luc <je...@rte-france.com>> wrote:
Hello,

I am unable to rebuild cubes. The logs give me the following error :

Exception in thread "Thread-13" java.lang.RuntimeException: java.lang.IllegalStateException: Cannot start job scheduler due to lack of job lock
        at org.apache.kylin.rest.service.JobService$1.run(JobService.java:131)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Cannot start job scheduler due to lack of job lock
        at org.apache.kylin.job.impl.threadpool.DefaultScheduler.init(DefaultScheduler.java:211)
        at org.apache.kylin.rest.service.JobService$1.run(JobService.java:126)
        ... 1 more

As far as I can see on this topic, I understand that there should be a process that is still running or was not remove correctly.
I tried several ‘kylin.sh stop’ / ‘kylin.sh start’ with no effect.

I attach my kylin.out and kylin.log files.

Thank you in advance for your help.

Best regards,
Jean-Luc.


"Ce message est destiné exclusivement aux personnes ou entités auxquelles il est adressé et peut contenir des informations privilégiées ou confidentielles. Si vous avez reçu ce document par erreur, merci de nous l'indiquer par retour, de ne pas le transmettre et de procéder à sa destruction.

This message is solely intended for the use of the individual or entity to which it is addressed and may contain information that is privileged or confidential. If you have received this communication by error, please notify us immediately by electronic mail, do not disclose it and delete the original message."



"Ce message est destiné exclusivement aux personnes ou entités auxquelles il est adressé et peut contenir des informations privilégiées ou confidentielles. Si vous avez reçu ce document par erreur, merci de nous l'indiquer par retour, de ne pas le transmettre et de procéder à sa destruction.

This message is solely intended for the use of the individual or entity to which it is addressed and may contain information that is privileged or confidential. If you have received this communication by error, please notify us immediately by electronic mail, do not disclose it and delete the original message."


"Ce message est destiné exclusivement aux personnes ou entités auxquelles il est adressé et peut contenir des informations privilégiées ou confidentielles. Si vous avez reçu ce document par erreur, merci de nous l'indiquer par retour, de ne pas le transmettre et de procéder à sa destruction.

This message is solely intended for the use of the individual or entity to which it is addressed and may contain information that is privileged or confidential. If you have received this communication by error, please notify us immediately by electronic mail, do not disclose it and delete the original message."

RE: Cannot rebuild cubes

Posted by BELLIER Jean-luc <je...@rte-france.com>.

Hello again,

My test is not conclusive, because the time slice I chose was before 2012. There is no data before this year. So it is normal.
I did then another test, to get the data between 2010-01-01 and 2014-02-02, but I still get the same result : no data available on step 1. I attach my kylin.log file.

In my kylin.log file, I can find the following error message while trying to create the temp Hive tables :
2018-02-12 14:30:01,574 INFO [Scheduler 481383589 Job 8a97abb9-69c0-4484-b2e0-ed801dcb845a-161] hive.CreateFlatHiveTableStep:38 : Connecting to jdbc:hive2://cloudera-manager.hades.rte-france.com:10000/
2018-02-12 14:30:01,754 INFO [Scheduler 481383589 Job 8a97abb9-69c0-4484-b2e0-ed801dcb845a-161] hive.CreateFlatHiveTableStep:38 : Unknown HS2 problem when communicating with Thrift server.
2018-02-12 14:30:01,754 INFO [Scheduler 481383589 Job 8a97abb9-69c0-4484-b2e0-ed801dcb845a-161] hive.CreateFlatHiveTableStep:38 : Error: Could not open client transport with JDBC Uri: jdbc:hive2://cloudera-manager.hades.rte-france.com:10000/: Peer indicated failure: Unsupported mechanism type PLAIN (state=08S01,code=0)
2018-02-12 14:30:01,754 INFO [Scheduler 481383589 Job 8a97abb9-69c0-4484-b2e0-ed801dcb845a-161] hive.CreateFlatHiveTableStep:38 : Beeline version 1.1.0-cdh5.14.0 by Apache Hive
2018-02-12 14:30:11,729 DEBUG [localhost-startStop-1-SendThread(node4.hades.rte-france.com:2181)] zookeeper.ClientCnxn:717 : Got ping response for sessionid: 0x261895d4a1c0140 after 0ms

Could you please give me any clue ?

NB : my kylin.properties file remains the same as before (see parameters below).

Thank you for your help.
Have a good day.

Best regards,
Jean-Luc.

De : BELLIER Jean-luc [mailto:jean-luc.bellier@rte-france.com]
Envoyé : lundi 12 février 2018 12:29
À : user@kylin.apache.org
Objet : RE: Cannot rebuild cubes

Hello Ted

Thank you for your reply. What is the command to check what you are suggesting ?
In the meanwhile, I changed the way accessing to hive to ‘beeline’, and faced to write rights problems to HDFS.
This is now solved.

Now I try to build the cube ‘kylin_sales_cube’. All the tables are present in Hive, and they contain data. The metadata are also in HBase in ‘kylin_metadata’ table.
What I do not understand is that the job seems to run with no data. In the step #1, the log tells me ‘No Data Available’

[cid:image001.gif@01D3A40B.E9B5D600][cid:image002.gif@01D3A40B.E9B5D600]

Could it be a problem of rights ?

Here is my kylin.properties

## Hive client, valid value [cli, beeline]
kylin.source.hive.client=beeline
#
## Parameters for beeline client, only necessary if hive client is beeline
kylin.source.hive.beeline-params=-n root --hiveconf hive.security.authorization.sqlstd.confwhitelist.append='mapreduce.job.*|dfs.*' -u jdbc:hive2://cloudera-manager.hades.rte-france.com:10000/;principal=hive/cloudera-manager.hades.rte-france.com@HADES.RTE-FRANCE.COM
#
#kylin.source.hive.keep-flat-table=false
#
## Hive database name for putting the intermediate flat tables
kylin.source.hive.database-for-flat-table=default

The full log files have been attached in this mail (using ‘Diagnosis’ option) on the job in Web interface.

PS : I noticed the presence of file ‘health-check.sh’ in /tmp/kylin on HDFS. Is is normal ?

Thank you in advance for your help.

Best regards,
Jean-Luc.

De : Ted Yu [mailto:yuzhihong@gmail.com]
Envoyé : vendredi 9 février 2018 22:28
À : user@kylin.apache.org<ma...@kylin.apache.org>
Objet : Re: Cannot rebuild cubes

In the log, I saw:

2018-02-09 17:39:57,634 ERROR [http-bio-7070-exec-8] controller.CubeController:337 : Segments overlap: kylin_sales_cube[20100101000000_20140102000000] and kylin_sales_cube[20120101000000_20140102000000]
java.lang.IllegalStateException: Segments overlap: kylin_sales_cube[20100101000000_20140102000000] and kylin_sales_cube[20120101000000_20140102000000]
at org.apache.kylin.metadata.model.Segments.validate(Segments.java:404)
at org.apache.kylin.cube.CubeManager.updateCubeWithRetry(CubeManager.java:393)
at org.apache.kylin.cube.CubeManager.updateCube(CubeManager.java:349)
at org.apache.kylin.cube.CubeManager.appendSegment(CubeManager.java:485)
at org.apache.kylin.cube.CubeManager.appendSegment(CubeManager.java:455)
at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:239)

It seems there was some wrong segment in metadata.
Have you tried dumping out metadata and find the invalid segment ?

Cheers

On Fri, Feb 9, 2018 at 9:18 AM, BELLIER Jean-luc <je...@rte-france.com>> wrote:
Hello,

I am unable to rebuild cubes. The logs give me the following error :

As far as I can see on this topic, I understand that there should be a process that is still running or was not remove correctly.
I tried several ‘kylin.sh stop’ / ‘kylin.sh start’ with no effect.

I attach my kylin.out and kylin.log files.

Thank you in advance for your help.

Best regards,
Jean-Luc.

"Ce message est destiné exclusivement aux personnes ou entités auxquelles il est adressé et peut contenir des informations privilégiées ou confidentielles. Si vous avez reçu ce document par erreur, merci de nous l'indiquer par retour, de ne pas le transmettre et de procéder à sa destruction.

RE: Cannot rebuild cubes

Posted by BELLIER Jean-luc <je...@rte-france.com>.

Hello Ted,

It is OK now, it wxas just oa problem of rights on user hive on /kylin/metadata on HDFS.
I succeeded in created my first cube.
Unfortunately, I cannot query it. I selected the project, and the cube then moved to ‘Insight tab’ and typed : select count(*) from kylin_sales. I got the following message :
null while executing SQL: "select count(*) from kylin_sales LIMIT 50000"

the log says : ‘unknown source’

2018-02-13 09:21:16,854 ERROR [http-bio-7070-exec-6] controller.BasicController:57 :
org.apache.kylin.rest.exception.InternalErrorException: null
while executing SQL: "select count(*) from kylin_sales LIMIT 50000"
        at org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:479)
        at org.apache.kylin.rest.controller.QueryController.query(QueryController.java:78)
        at sun.reflect.GeneratedMethodAccessor205.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)

How could I solve this ? By rebuilding the model and then the cube ?
If yes, which commands/process shoud I use ?

Thank you for your help.

Best regards,
Jean-Luc.

De : BELLIER Jean-luc
Envoyé : lundi 12 février 2018 15:20
À : 'user@kylin.apache.org' <us...@kylin.apache.org>>
Objet : RE: Cannot rebuild cubes

Hello again Ted,

I did the following :

·         launched the command ‘beeline’

·         launched the command (under beeline) : !connect jdbc:hive2://<host>:<port>/<db>;principal=<Server_Principal_of_HiveServer2>
And I got the following message :

Driver: Hive JDBC (version 1.1.0-cdh5.14.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ

Could this be the source of the problem, and, if yes, how to correct it ?

Thank you for your help.

Best regards,
Jean-Luc.

De : BELLIER Jean-luc
Envoyé : lundi 12 février 2018 14:39
À : user@kylin.apache.org<ma...@kylin.apache.org>
Objet : RE: Cannot rebuild cubes

Hello again,

My test is not conclusive, because the time slice I chose was before 2012. There is no data before this year. So it is normal.
I did then another test, to get the data between 2010-01-01 and 2014-02-02, but I still get the same result : no data available on step 1. I attach my kylin.log file.

In my kylin.log file, I can find the following error message while trying to create the temp Hive tables :
2018-02-12 14:30:01,574 INFO  [Scheduler 481383589 Job 8a97abb9-69c0-4484-b2e0-ed801dcb845a-161] hive.CreateFlatHiveTableStep:38 : Connecting to jdbc:hive2://cloudera-manager.hades.rte-france.com:10000/
2018-02-12 14:30:01,754 INFO  [Scheduler 481383589 Job 8a97abb9-69c0-4484-b2e0-ed801dcb845a-161] hive.CreateFlatHiveTableStep:38 : Unknown HS2 problem when communicating with Thrift server.
2018-02-12 14:30:01,754 INFO  [Scheduler 481383589 Job 8a97abb9-69c0-4484-b2e0-ed801dcb845a-161] hive.CreateFlatHiveTableStep:38 : Error: Could not open client transport with JDBC Uri: jdbc:hive2://cloudera-manager.hades.rte-france.com:10000/: Peer indicated failure: Unsupported mechanism type PLAIN (state=08S01,code=0)
2018-02-12 14:30:01,754 INFO  [Scheduler 481383589 Job 8a97abb9-69c0-4484-b2e0-ed801dcb845a-161] hive.CreateFlatHiveTableStep:38 : Beeline version 1.1.0-cdh5.14.0 by Apache Hive
2018-02-12 14:30:11,729 DEBUG [localhost-startStop-1-SendThread(node4.hades.rte-france.com:2181)] zookeeper.ClientCnxn:717 : Got ping response for sessionid: 0x261895d4a1c0140 after 0ms

Could you please give me any clue ?

NB : my kylin.properties file remains the same as before (see parameters below).

Thank you for your help.
Have a good day.

Best regards,
Jean-Luc.

De : BELLIER Jean-luc [mailto:jean-luc.bellier@rte-france.com]
Envoyé : lundi 12 février 2018 12:29
À : user@kylin.apache.org<ma...@kylin.apache.org>
Objet : RE: Cannot rebuild cubes

Hello Ted

Thank you for your reply. What is the command to check what you are suggesting ?
In the meanwhile, I changed the way accessing to hive to ‘beeline’, and faced to write rights problems to HDFS.
This is now solved.

Now I try to build the cube ‘kylin_sales_cube’. All the tables are present in Hive, and they contain data. The metadata are also in HBase in ‘kylin_metadata’ table.
What I do not understand is that the job seems to run with no data. In the step #1, the log tells me ‘No Data Available’

[cid:image001.gif@01D3A4A6.AA66DA30][cid:image002.gif@01D3A4A6.AA66DA30]

Could it be a problem of rights ?

Here is my kylin.properties

## Hive client, valid value [cli, beeline]
kylin.source.hive.client=beeline
#
## Parameters for beeline client, only necessary if hive client is beeline
kylin.source.hive.beeline-params=-n root --hiveconf hive.security.authorization.sqlstd.confwhitelist.append='mapreduce.job.*|dfs.*' -u jdbc:hive2://cloudera-manager.hades.rte-france.com:10000/;principal=hive/cloudera-manager.hades.rte-france.com@HADES.RTE-FRANCE.COM
#
#kylin.source.hive.keep-flat-table=false
#
## Hive database name for putting the intermediate flat tables
kylin.source.hive.database-for-flat-table=default

The full log files have been attached in this mail (using ‘Diagnosis’ option) on the job in Web interface.

PS  : I noticed the presence of file ‘health-check.sh’ in /tmp/kylin on HDFS. Is is normal ?

Thank you in advance for your help.

Best regards,
Jean-Luc.

De : Ted Yu [mailto:yuzhihong@gmail.com]
Envoyé : vendredi 9 février 2018 22:28
À : user@kylin.apache.org<ma...@kylin.apache.org>
Objet : Re: Cannot rebuild cubes

In the log, I saw:

2018-02-09 17:39:57,634 ERROR [http-bio-7070-exec-8] controller.CubeController:337 : Segments overlap: kylin_sales_cube[20100101000000_20140102000000] and          kylin_sales_cube[20120101000000_20140102000000]
java.lang.IllegalStateException: Segments overlap: kylin_sales_cube[20100101000000_20140102000000] and kylin_sales_cube[20120101000000_20140102000000]
  at org.apache.kylin.metadata.model.Segments.validate(Segments.java:404)
  at org.apache.kylin.cube.CubeManager.updateCubeWithRetry(CubeManager.java:393)
  at org.apache.kylin.cube.CubeManager.updateCube(CubeManager.java:349)
  at org.apache.kylin.cube.CubeManager.appendSegment(CubeManager.java:485)
  at org.apache.kylin.cube.CubeManager.appendSegment(CubeManager.java:455)
  at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:239)

It seems there was some wrong segment in metadata.
Have you tried dumping out metadata and find the invalid segment ?

Cheers

On Fri, Feb 9, 2018 at 9:18 AM, BELLIER Jean-luc <je...@rte-france.com>> wrote:
Hello,

I am unable to rebuild cubes. The logs give me the following error :

Exception in thread "Thread-13" java.lang.RuntimeException: java.lang.IllegalStateException: Cannot start job scheduler due to lack of job lock
        at org.apache.kylin.rest.service.JobService$1.run(JobService.java:131)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Cannot start job scheduler due to lack of job lock
        at org.apache.kylin.job.impl.threadpool.DefaultScheduler.init(DefaultScheduler.java:211)
        at org.apache.kylin.rest.service.JobService$1.run(JobService.java:126)
        ... 1 more

As far as I can see on this topic, I understand that there should be a process that is still running or was not remove correctly.
I tried several ‘kylin.sh stop’ / ‘kylin.sh start’ with no effect.

I attach my kylin.out and kylin.log files.

Thank you in advance for your help.

Best regards,
Jean-Luc.


"Ce message est destiné exclusivement aux personnes ou entités auxquelles il est adressé et peut contenir des informations privilégiées ou confidentielles. Si vous avez reçu ce document par erreur, merci de nous l'indiquer par retour, de ne pas le transmettre et de procéder à sa destruction.

This message is solely intended for the use of the individual or entity to which it is addressed and may contain information that is privileged or confidential. If you have received this communication by error, please notify us immediately by electronic mail, do not disclose it and delete the original message."



"Ce message est destiné exclusivement aux personnes ou entités auxquelles il est adressé et peut contenir des informations privilégiées ou confidentielles. Si vous avez reçu ce document par erreur, merci de nous l'indiquer par retour, de ne pas le transmettre et de procéder à sa destruction.

This message is solely intended for the use of the individual or entity to which it is addressed and may contain information that is privileged or confidential. If you have received this communication by error, please notify us immediately by electronic mail, do not disclose it and delete the original message."


"Ce message est destiné exclusivement aux personnes ou entités auxquelles il est adressé et peut contenir des informations privilégiées ou confidentielles. Si vous avez reçu ce document par erreur, merci de nous l'indiquer par retour, de ne pas le transmettre et de procéder à sa destruction.

This message is solely intended for the use of the individual or entity to which it is addressed and may contain information that is privileged or confidential. If you have received this communication by error, please notify us immediately by electronic mail, do not disclose it and delete the original message."

RE: Cannot rebuild cubes

Posted by BELLIER Jean-luc <je...@rte-france.com>.

Hello again Ted,

I did the following :

·         launched the command ‘beeline’

·         launched the command (under beeline) : !connect jdbc:hive2://<host>:<port>/<db>;principal=<Server_Principal_of_HiveServer2>
And I got the following message :

Driver: Hive JDBC (version 1.1.0-cdh5.14.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ

Could this be the source of the problem, and, if yes, how to correct it ?

Thank you for your help.

Best regards,
Jean-Luc.

De : BELLIER Jean-luc
Envoyé : lundi 12 février 2018 14:39
À : user@kylin.apache.org
Objet : RE: Cannot rebuild cubes

Hello again,

My test is not conclusive, because the time slice I chose was before 2012. There is no data before this year. So it is normal.
I did then another test, to get the data between 2010-01-01 and 2014-02-02, but I still get the same result : no data available on step 1. I attach my kylin.log file.

In my kylin.log file, I can find the following error message while trying to create the temp Hive tables :
2018-02-12 14:30:01,574 INFO  [Scheduler 481383589 Job 8a97abb9-69c0-4484-b2e0-ed801dcb845a-161] hive.CreateFlatHiveTableStep:38 : Connecting to jdbc:hive2://cloudera-manager.hades.rte-france.com:10000/
2018-02-12 14:30:01,754 INFO  [Scheduler 481383589 Job 8a97abb9-69c0-4484-b2e0-ed801dcb845a-161] hive.CreateFlatHiveTableStep:38 : Unknown HS2 problem when communicating with Thrift server.
2018-02-12 14:30:01,754 INFO  [Scheduler 481383589 Job 8a97abb9-69c0-4484-b2e0-ed801dcb845a-161] hive.CreateFlatHiveTableStep:38 : Error: Could not open client transport with JDBC Uri: jdbc:hive2://cloudera-manager.hades.rte-france.com:10000/: Peer indicated failure: Unsupported mechanism type PLAIN (state=08S01,code=0)
2018-02-12 14:30:01,754 INFO  [Scheduler 481383589 Job 8a97abb9-69c0-4484-b2e0-ed801dcb845a-161] hive.CreateFlatHiveTableStep:38 : Beeline version 1.1.0-cdh5.14.0 by Apache Hive
2018-02-12 14:30:11,729 DEBUG [localhost-startStop-1-SendThread(node4.hades.rte-france.com:2181)] zookeeper.ClientCnxn:717 : Got ping response for sessionid: 0x261895d4a1c0140 after 0ms

Could you please give me any clue ?

NB : my kylin.properties file remains the same as before (see parameters below).

Thank you for your help.
Have a good day.

Best regards,
Jean-Luc.

De : BELLIER Jean-luc [mailto:jean-luc.bellier@rte-france.com]
Envoyé : lundi 12 février 2018 12:29
À : user@kylin.apache.org<ma...@kylin.apache.org>
Objet : RE: Cannot rebuild cubes

Hello Ted

Thank you for your reply. What is the command to check what you are suggesting ?
In the meanwhile, I changed the way accessing to hive to ‘beeline’, and faced to write rights problems to HDFS.
This is now solved.

Now I try to build the cube ‘kylin_sales_cube’. All the tables are present in Hive, and they contain data. The metadata are also in HBase in ‘kylin_metadata’ table.
What I do not understand is that the job seems to run with no data. In the step #1, the log tells me ‘No Data Available’

[cid:image001.gif@01D3A415.024BD8F0][cid:image002.gif@01D3A415.024BD8F0]

Could it be a problem of rights ?

Here is my kylin.properties

## Hive client, valid value [cli, beeline]
kylin.source.hive.client=beeline
#
## Parameters for beeline client, only necessary if hive client is beeline
kylin.source.hive.beeline-params=-n root --hiveconf hive.security.authorization.sqlstd.confwhitelist.append='mapreduce.job.*|dfs.*' -u jdbc:hive2://cloudera-manager.hades.rte-france.com:10000/;principal=hive/cloudera-manager.hades.rte-france.com@HADES.RTE-FRANCE.COM
#
#kylin.source.hive.keep-flat-table=false
#
## Hive database name for putting the intermediate flat tables
kylin.source.hive.database-for-flat-table=default

The full log files have been attached in this mail (using ‘Diagnosis’ option) on the job in Web interface.

PS  : I noticed the presence of file ‘health-check.sh’ in /tmp/kylin on HDFS. Is is normal ?

Thank you in advance for your help.

Best regards,
Jean-Luc.

De : Ted Yu [mailto:yuzhihong@gmail.com]
Envoyé : vendredi 9 février 2018 22:28
À : user@kylin.apache.org<ma...@kylin.apache.org>
Objet : Re: Cannot rebuild cubes

In the log, I saw:

2018-02-09 17:39:57,634 ERROR [http-bio-7070-exec-8] controller.CubeController:337 : Segments overlap: kylin_sales_cube[20100101000000_20140102000000] and          kylin_sales_cube[20120101000000_20140102000000]
java.lang.IllegalStateException: Segments overlap: kylin_sales_cube[20100101000000_20140102000000] and kylin_sales_cube[20120101000000_20140102000000]
  at org.apache.kylin.metadata.model.Segments.validate(Segments.java:404)
  at org.apache.kylin.cube.CubeManager.updateCubeWithRetry(CubeManager.java:393)
  at org.apache.kylin.cube.CubeManager.updateCube(CubeManager.java:349)
  at org.apache.kylin.cube.CubeManager.appendSegment(CubeManager.java:485)
  at org.apache.kylin.cube.CubeManager.appendSegment(CubeManager.java:455)
  at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:239)

It seems there was some wrong segment in metadata.
Have you tried dumping out metadata and find the invalid segment ?

Cheers

On Fri, Feb 9, 2018 at 9:18 AM, BELLIER Jean-luc <je...@rte-france.com>> wrote:
Hello,

I am unable to rebuild cubes. The logs give me the following error :

Exception in thread "Thread-13" java.lang.RuntimeException: java.lang.IllegalStateException: Cannot start job scheduler due to lack of job lock
        at org.apache.kylin.rest.service.JobService$1.run(JobService.java:131)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Cannot start job scheduler due to lack of job lock
        at org.apache.kylin.job.impl.threadpool.DefaultScheduler.init(DefaultScheduler.java:211)
        at org.apache.kylin.rest.service.JobService$1.run(JobService.java:126)
        ... 1 more

As far as I can see on this topic, I understand that there should be a process that is still running or was not remove correctly.
I tried several ‘kylin.sh stop’ / ‘kylin.sh start’ with no effect.

I attach my kylin.out and kylin.log files.

Thank you in advance for your help.

Best regards,
Jean-Luc.


"Ce message est destiné exclusivement aux personnes ou entités auxquelles il est adressé et peut contenir des informations privilégiées ou confidentielles. Si vous avez reçu ce document par erreur, merci de nous l'indiquer par retour, de ne pas le transmettre et de procéder à sa destruction.

This message is solely intended for the use of the individual or entity to which it is addressed and may contain information that is privileged or confidential. If you have received this communication by error, please notify us immediately by electronic mail, do not disclose it and delete the original message."



"Ce message est destiné exclusivement aux personnes ou entités auxquelles il est adressé et peut contenir des informations privilégiées ou confidentielles. Si vous avez reçu ce document par erreur, merci de nous l'indiquer par retour, de ne pas le transmettre et de procéder à sa destruction.

This message is solely intended for the use of the individual or entity to which it is addressed and may contain information that is privileged or confidential. If you have received this communication by error, please notify us immediately by electronic mail, do not disclose it and delete the original message."


"Ce message est destiné exclusivement aux personnes ou entités auxquelles il est adressé et peut contenir des informations privilégiées ou confidentielles. Si vous avez reçu ce document par erreur, merci de nous l'indiquer par retour, de ne pas le transmettre et de procéder à sa destruction.

This message is solely intended for the use of the individual or entity to which it is addressed and may contain information that is privileged or confidential. If you have received this communication by error, please notify us immediately by electronic mail, do not disclose it and delete the original message."

RE: Cannot rebuild cubes

Posted by BELLIER Jean-luc <je...@rte-france.com>.

Hello Ted

[cid:image001.gif@01D3A3FB.2019D900] [cid:image002.gif@01D3A3FB.2019D900]

Could it be a problem of rights ?

Here is my kylin.properties

The full log files have been attached in this mail (using ‘Diagnosis’ option) on the job in Web interface.

PS : I noticed the presence of file ‘health-check.sh’ in /tmp/kylin on HDFS. Is is normal ?

Thank you in advance for your help.

Best regards,
Jean-Luc.

De : Ted Yu [mailto:yuzhihong@gmail.com]
Envoyé : vendredi 9 février 2018 22:28
À : user@kylin.apache.org<ma...@kylin.apache.org>
Objet : Re: Cannot rebuild cubes

In the log, I saw:

It seems there was some wrong segment in metadata.
Have you tried dumping out metadata and find the invalid segment ?

Cheers

On Fri, Feb 9, 2018 at 9:18 AM, BELLIER Jean-luc <je...@rte-france.com>> wrote:
Hello,

I am unable to rebuild cubes. The logs give me the following error :

I attach my kylin.out and kylin.log files.

Thank you in advance for your help.

Best regards,
Jean-Luc.

Re: Cannot rebuild cubes

Posted by Ted Yu <yu...@gmail.com>.

In the log, I saw:

2018-02-09 17:39:57,634 ERROR [http-bio-7070-exec-8]
controller.CubeController:337 : Segments overlap:
kylin_sales_cube[20100101000000_20140102000000] and
kylin_sales_cube[20120101000000_20140102000000]
java.lang.IllegalStateException: Segments overlap:
kylin_sales_cube[20100101000000_20140102000000] and
kylin_sales_cube[20120101000000_20140102000000]
  at org.apache.kylin.metadata.model.Segments.validate(Segments.java:404)
  at
org.apache.kylin.cube.CubeManager.updateCubeWithRetry(CubeManager.java:393)
  at org.apache.kylin.cube.CubeManager.updateCube(CubeManager.java:349)
  at org.apache.kylin.cube.CubeManager.appendSegment(CubeManager.java:485)
  at org.apache.kylin.cube.CubeManager.appendSegment(CubeManager.java:455)
  at
org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:239)

It seems there was some wrong segment in metadata.
Have you tried dumping out metadata and find the invalid segment ?

Cheers

On Fri, Feb 9, 2018 at 9:18 AM, BELLIER Jean-luc <
jean-luc.bellier@rte-france.com> wrote:

> Hello,
>
>
>
> I am unable to rebuild cubes. The logs give me the following error :
>
>
>
> *Exception in thread "Thread-13" java.lang.RuntimeException:
> java.lang.IllegalStateException: Cannot start job scheduler due to lack of
> job lock*
>
> *        at
> org.apache.kylin.rest.service.JobService$1.run(JobService.java:131)*
>
> *        at java.lang.Thread.run(Thread.java:748)*
>
> *Caused by: java.lang.IllegalStateException: Cannot start job scheduler
> due to lack of job lock*
>
> *        at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler.init(DefaultScheduler.java:211)*
>
> *        at
> org.apache.kylin.rest.service.JobService$1.run(JobService.java:126)*
>
> *        ... 1 more*
>
>
>
> As far as I can see on this topic, I understand that there should be a
> process that is still running or was not remove correctly.
>
> I tried several ‘kylin.sh stop’ / ‘kylin.sh start’ with no effect.
>
>
>
> I attach my kylin.out and kylin.log files.
>
>
>
> Thank you in advance for your help.
>
>
>
> Best regards,
>
> Jean-Luc.
>
>
> "Ce message est destiné exclusivement aux personnes ou entités auxquelles
> il est adressé et peut contenir des informations privilégiées ou
> confidentielles. Si vous avez reçu ce document par erreur, merci de nous
> l'indiquer par retour, de ne pas le transmettre et de procéder à sa
> destruction.
>
> This message is solely intended for the use of the individual or entity to
> which it is addressed and may contain information that is privileged or
> confidential. If you have received this communication by error, please
> notify us immediately by electronic mail, do not disclose it and delete the
> original message."
>