You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@griffin.apache.org by jo...@boehringer-ingelheim.com on 2019/08/21 14:52:51 UTC
Connect Griffin to Hive secured metastore
Hello,
We are trying to connect Griffin to a secured Hive metastore, does anybody know how to configure Griffin for this connection? We are getting authorization errors in the metastore.
Waiting for your answer.
Regards
Re: Connect Griffin to Hive secured metastore
Posted by William Guo <gu...@apache.org>.
hi,
we are using livy to connect with spark, could you check livy related
documents,
Otherwise file a ticket for us.
Thanks,
William
On Fri, Aug 23, 2019 at 2:11 PM <
jose.martin_santacruz.ext@boehringer-ingelheim.com> wrote:
> Hello,
>
> No the authentication in the cluster is MapR-SASL.
>
> Regards
>
> -----Original Message-----
> From: Qian Wang <qw...@gmail.com>
> Sent: Thursday, August 22, 2019 7:04 PM
> To: dev@griffin.apache.org
> Cc: zxBCN De_La_Fuente_Diaz,Alvaro (IT EDS) EXTERNAL <
> alvaro.de_la_fuente_diaz.ext@boehringer-ingelheim.com>
> Subject: RE: Connect Griffin to Hive secured metastore
>
> Hi,
>
> Do you have kerberos auth? If you have, you need config the
>
> livy.need.kerberos=true
> #if livy need kerberos is false then don't need set following two
> properties livy.server.auth.kerberos.principal=livy/kerberos.principal
> livy.server.auth.kerberos.keytab=/path/to/livy/keytab/file
> Best,
> Qian
>
> On Aug 22, 2019, 4:04 AM -0700,
> jose.martin_santacruz.ext@boehringer-ingelheim.com, wrote:
> > Hi Qian,
> >
> > Thank you very much for your help, we changed the connection to Hive
> Metadata to Hive JDBC and now we are able to get Hive Metadata.
> > But now we have a problem with Livy authorization, the problem is that
> we do not know how to configure user and password for Livy in Griffin, does
> anybody know how to do it.
> > The error we are getting is the following:
> >
> > 2019-08-22 10:50:00.830 INFO 83698 --- [ryBean_Worker-2]
> > o.a.g.c.j.LivyTaskSubmitHelper [230] : Post To Livy URI is:
> > https://inhas68625.eu.boehringer.com:8998/batches
> > 2019-08-22 10:50:00.830 INFO 83698 --- [ryBean_Worker-2]
> > o.a.g.c.j.LivyTaskSubmitHelper [232] : Need Kerberos:false
> > 2019-08-22 10:50:00.830 INFO 83698 --- [ryBean_Worker-2]
> > o.a.g.c.j.LivyTaskSubmitHelper [244] : The livy server doesn't need
> > Kerberos Authentication
> > 2019-08-22 10:50:01.462 ERROR 83698 --- [ryBean_Worker-2]
> o.a.g.c.j.SparkSubmitJob [116] : Post spark task ERROR.
> >
> > org.springframework.web.client.HttpClientErrorException: 401
> > Authentication required at
> > org.springframework.web.client.DefaultResponseErrorHandler.handleError
> > (DefaultResponseErrorHandler.java:91)
> > ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
> > at
> > org.springframework.web.client.RestTemplate.handleResponse(RestTemplat
> > e.java:700) ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
> > at
> > org.springframework.web.client.RestTemplate.doExecute(RestTemplate.jav
> > a:653) ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
> > at
> > org.springframework.web.client.RestTemplate.execute(RestTemplate.java:
> > 613) ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
> > at
> > org.springframework.web.client.RestTemplate.postForObject(RestTemplate
> > .java:380) ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
> > at
> > org.apache.griffin.core.job.LivyTaskSubmitHelper.postToLivy(LivyTaskSu
> > bmitHelper.java:248) ~[classes!/:0.6.0-SNAPSHOT] at
> > org.apache.griffin.core.job.SparkSubmitJob.post2Livy(SparkSubmitJob.ja
> > va:131) ~[classes!/:0.6.0-SNAPSHOT] at
> > org.apache.griffin.core.job.SparkSubmitJob.post2LivyWithRetry(SparkSub
> > mitJob.java:224) ~[classes!/:0.6.0-SNAPSHOT] at
> > org.apache.griffin.core.job.SparkSubmitJob.saveJobInstance(SparkSubmit
> > Job.java:213) ~[classes!/:0.6.0-SNAPSHOT] at
> > org.apache.griffin.core.job.SparkSubmitJob.execute(SparkSubmitJob.java
> > :113) [classes!/:0.6.0-SNAPSHOT] at
> > org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> > [quartz-2.2.2.jar!/:?] at
> > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.ja
> > va:573) [quartz-2.2.2.jar!/:?]
> >
> > Waiting for your answer.
> >
> > Regards
> >
> > -----Original Message-----
> > From: Qian Wang <qw...@gmail.com>
> > Sent: Wednesday, August 21, 2019 7:17 PM
> > To: dev@griffin.apache.org
> > Subject: Re: Connect Griffin to Hive secured metastore
> >
> > Hi,
> >
> > You have an alternative method to get Hive Metadata by using Hive JDBC.
> If you want to use JDBC, you need change
> org.apache.griffin.core.metastore.hive.HiveMetaStoreController:
> > @Autowired
> > @Qualifier(value = "jdbcSvc")
> > private HiveMetaStoreService hiveMetaStoreService; Also, if your Hive is
> Authenticated by Kerberos, you need setup application.properties:
> > #Hive jdbc
> > hive.jdbc.className=org.apache.hive.jdbc.HiveDriver
> > hive.jdbc.url=jdbc:hive2://localhost:10000/ #your Hive url
> hive.need.kerberos=true # if you need Kerberos Auth hive.keytab.user=
> xxx@xx.com hive.keytab.path=/path/to/keytab/file #here is absolute path
> Hopefully can answer your question.
> >
> > Best,
> > Eric
> > On Aug 21, 2019, 7:52 AM -0700,
> jose.martin_santacruz.ext@boehringer-ingelheim.com, wrote:
> > > Hello,
> > >
> > > We are trying to connect Griffin to a secured Hive metastore, does
> anybody know how to configure Griffin for this connection? We are getting
> authorization errors in the metastore.
> > >
> > > Waiting for your answer.
> > >
> > > Regards
>
RE: Connect Griffin to Hive secured metastore
Posted by jo...@boehringer-ingelheim.com.
Hello,
No the authentication in the cluster is MapR-SASL.
Regards
-----Original Message-----
From: Qian Wang <qw...@gmail.com>
Sent: Thursday, August 22, 2019 7:04 PM
To: dev@griffin.apache.org
Cc: zxBCN De_La_Fuente_Diaz,Alvaro (IT EDS) EXTERNAL <al...@boehringer-ingelheim.com>
Subject: RE: Connect Griffin to Hive secured metastore
Hi,
Do you have kerberos auth? If you have, you need config the
livy.need.kerberos=true
#if livy need kerberos is false then don't need set following two properties livy.server.auth.kerberos.principal=livy/kerberos.principal
livy.server.auth.kerberos.keytab=/path/to/livy/keytab/file
Best,
Qian
On Aug 22, 2019, 4:04 AM -0700, jose.martin_santacruz.ext@boehringer-ingelheim.com, wrote:
> Hi Qian,
>
> Thank you very much for your help, we changed the connection to Hive Metadata to Hive JDBC and now we are able to get Hive Metadata.
> But now we have a problem with Livy authorization, the problem is that we do not know how to configure user and password for Livy in Griffin, does anybody know how to do it.
> The error we are getting is the following:
>
> 2019-08-22 10:50:00.830 INFO 83698 --- [ryBean_Worker-2]
> o.a.g.c.j.LivyTaskSubmitHelper [230] : Post To Livy URI is:
> https://inhas68625.eu.boehringer.com:8998/batches
> 2019-08-22 10:50:00.830 INFO 83698 --- [ryBean_Worker-2]
> o.a.g.c.j.LivyTaskSubmitHelper [232] : Need Kerberos:false
> 2019-08-22 10:50:00.830 INFO 83698 --- [ryBean_Worker-2]
> o.a.g.c.j.LivyTaskSubmitHelper [244] : The livy server doesn't need
> Kerberos Authentication
> 2019-08-22 10:50:01.462 ERROR 83698 --- [ryBean_Worker-2] o.a.g.c.j.SparkSubmitJob [116] : Post spark task ERROR.
>
> org.springframework.web.client.HttpClientErrorException: 401
> Authentication required at
> org.springframework.web.client.DefaultResponseErrorHandler.handleError
> (DefaultResponseErrorHandler.java:91)
> ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
> at
> org.springframework.web.client.RestTemplate.handleResponse(RestTemplat
> e.java:700) ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
> at
> org.springframework.web.client.RestTemplate.doExecute(RestTemplate.jav
> a:653) ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
> at
> org.springframework.web.client.RestTemplate.execute(RestTemplate.java:
> 613) ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
> at
> org.springframework.web.client.RestTemplate.postForObject(RestTemplate
> .java:380) ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
> at
> org.apache.griffin.core.job.LivyTaskSubmitHelper.postToLivy(LivyTaskSu
> bmitHelper.java:248) ~[classes!/:0.6.0-SNAPSHOT] at
> org.apache.griffin.core.job.SparkSubmitJob.post2Livy(SparkSubmitJob.ja
> va:131) ~[classes!/:0.6.0-SNAPSHOT] at
> org.apache.griffin.core.job.SparkSubmitJob.post2LivyWithRetry(SparkSub
> mitJob.java:224) ~[classes!/:0.6.0-SNAPSHOT] at
> org.apache.griffin.core.job.SparkSubmitJob.saveJobInstance(SparkSubmit
> Job.java:213) ~[classes!/:0.6.0-SNAPSHOT] at
> org.apache.griffin.core.job.SparkSubmitJob.execute(SparkSubmitJob.java
> :113) [classes!/:0.6.0-SNAPSHOT] at
> org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> [quartz-2.2.2.jar!/:?] at
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.ja
> va:573) [quartz-2.2.2.jar!/:?]
>
> Waiting for your answer.
>
> Regards
>
> -----Original Message-----
> From: Qian Wang <qw...@gmail.com>
> Sent: Wednesday, August 21, 2019 7:17 PM
> To: dev@griffin.apache.org
> Subject: Re: Connect Griffin to Hive secured metastore
>
> Hi,
>
> You have an alternative method to get Hive Metadata by using Hive JDBC. If you want to use JDBC, you need change org.apache.griffin.core.metastore.hive.HiveMetaStoreController:
> @Autowired
> @Qualifier(value = "jdbcSvc")
> private HiveMetaStoreService hiveMetaStoreService; Also, if your Hive is Authenticated by Kerberos, you need setup application.properties:
> #Hive jdbc
> hive.jdbc.className=org.apache.hive.jdbc.HiveDriver
> hive.jdbc.url=jdbc:hive2://localhost:10000/ #your Hive url hive.need.kerberos=true # if you need Kerberos Auth hive.keytab.user=xxx@xx.com hive.keytab.path=/path/to/keytab/file #here is absolute path Hopefully can answer your question.
>
> Best,
> Eric
> On Aug 21, 2019, 7:52 AM -0700, jose.martin_santacruz.ext@boehringer-ingelheim.com, wrote:
> > Hello,
> >
> > We are trying to connect Griffin to a secured Hive metastore, does anybody know how to configure Griffin for this connection? We are getting authorization errors in the metastore.
> >
> > Waiting for your answer.
> >
> > Regards
RE: Connect Griffin to Hive secured metastore
Posted by Qian Wang <qw...@gmail.com>.
Hi,
Do you have kerberos auth? If you have, you need config the
livy.need.kerberos=true
#if livy need kerberos is false then don't need set following two properties
livy.server.auth.kerberos.principal=livy/kerberos.principal
livy.server.auth.kerberos.keytab=/path/to/livy/keytab/file
Best,
Qian
On Aug 22, 2019, 4:04 AM -0700, jose.martin_santacruz.ext@boehringer-ingelheim.com, wrote:
> Hi Qian,
>
> Thank you very much for your help, we changed the connection to Hive Metadata to Hive JDBC and now we are able to get Hive Metadata.
> But now we have a problem with Livy authorization, the problem is that we do not know how to configure user and password for Livy in Griffin, does anybody know how to do it.
> The error we are getting is the following:
>
> 2019-08-22 10:50:00.830 INFO 83698 --- [ryBean_Worker-2] o.a.g.c.j.LivyTaskSubmitHelper [230] : Post To Livy URI is: https://inhas68625.eu.boehringer.com:8998/batches
> 2019-08-22 10:50:00.830 INFO 83698 --- [ryBean_Worker-2] o.a.g.c.j.LivyTaskSubmitHelper [232] : Need Kerberos:false
> 2019-08-22 10:50:00.830 INFO 83698 --- [ryBean_Worker-2] o.a.g.c.j.LivyTaskSubmitHelper [244] : The livy server doesn't need Kerberos Authentication
> 2019-08-22 10:50:01.462 ERROR 83698 --- [ryBean_Worker-2] o.a.g.c.j.SparkSubmitJob [116] : Post spark task ERROR.
>
> org.springframework.web.client.HttpClientErrorException: 401 Authentication required
> at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91) ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
> at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:700) ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
> at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:653) ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
> at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:613) ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
> at org.springframework.web.client.RestTemplate.postForObject(RestTemplate.java:380) ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
> at org.apache.griffin.core.job.LivyTaskSubmitHelper.postToLivy(LivyTaskSubmitHelper.java:248) ~[classes!/:0.6.0-SNAPSHOT]
> at org.apache.griffin.core.job.SparkSubmitJob.post2Livy(SparkSubmitJob.java:131) ~[classes!/:0.6.0-SNAPSHOT]
> at org.apache.griffin.core.job.SparkSubmitJob.post2LivyWithRetry(SparkSubmitJob.java:224) ~[classes!/:0.6.0-SNAPSHOT]
> at org.apache.griffin.core.job.SparkSubmitJob.saveJobInstance(SparkSubmitJob.java:213) ~[classes!/:0.6.0-SNAPSHOT]
> at org.apache.griffin.core.job.SparkSubmitJob.execute(SparkSubmitJob.java:113) [classes!/:0.6.0-SNAPSHOT]
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-2.2.2.jar!/:?]
> at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) [quartz-2.2.2.jar!/:?]
>
> Waiting for your answer.
>
> Regards
>
> -----Original Message-----
> From: Qian Wang <qw...@gmail.com>
> Sent: Wednesday, August 21, 2019 7:17 PM
> To: dev@griffin.apache.org
> Subject: Re: Connect Griffin to Hive secured metastore
>
> Hi,
>
> You have an alternative method to get Hive Metadata by using Hive JDBC. If you want to use JDBC, you need change org.apache.griffin.core.metastore.hive.HiveMetaStoreController:
> @Autowired
> @Qualifier(value = "jdbcSvc")
> private HiveMetaStoreService hiveMetaStoreService; Also, if your Hive is Authenticated by Kerberos, you need setup application.properties:
> #Hive jdbc
> hive.jdbc.className=org.apache.hive.jdbc.HiveDriver
> hive.jdbc.url=jdbc:hive2://localhost:10000/ #your Hive url hive.need.kerberos=true # if you need Kerberos Auth hive.keytab.user=xxx@xx.com hive.keytab.path=/path/to/keytab/file #here is absolute path Hopefully can answer your question.
>
> Best,
> Eric
> On Aug 21, 2019, 7:52 AM -0700, jose.martin_santacruz.ext@boehringer-ingelheim.com, wrote:
> > Hello,
> >
> > We are trying to connect Griffin to a secured Hive metastore, does anybody know how to configure Griffin for this connection? We are getting authorization errors in the metastore.
> >
> > Waiting for your answer.
> >
> > Regards
RE: Connect Griffin to Hive secured metastore
Posted by jo...@boehringer-ingelheim.com.
Hi Qian,
Thank you very much for your help, we changed the connection to Hive Metadata to Hive JDBC and now we are able to get Hive Metadata.
But now we have a problem with Livy authorization, the problem is that we do not know how to configure user and password for Livy in Griffin, does anybody know how to do it.
The error we are getting is the following:
2019-08-22 10:50:00.830 INFO 83698 --- [ryBean_Worker-2] o.a.g.c.j.LivyTaskSubmitHelper [230] : Post To Livy URI is: https://inhas68625.eu.boehringer.com:8998/batches
2019-08-22 10:50:00.830 INFO 83698 --- [ryBean_Worker-2] o.a.g.c.j.LivyTaskSubmitHelper [232] : Need Kerberos:false
2019-08-22 10:50:00.830 INFO 83698 --- [ryBean_Worker-2] o.a.g.c.j.LivyTaskSubmitHelper [244] : The livy server doesn't need Kerberos Authentication
2019-08-22 10:50:01.462 ERROR 83698 --- [ryBean_Worker-2] o.a.g.c.j.SparkSubmitJob [116] : Post spark task ERROR.
org.springframework.web.client.HttpClientErrorException: 401 Authentication required
at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91) ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:700) ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:653) ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:613) ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
at org.springframework.web.client.RestTemplate.postForObject(RestTemplate.java:380) ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
at org.apache.griffin.core.job.LivyTaskSubmitHelper.postToLivy(LivyTaskSubmitHelper.java:248) ~[classes!/:0.6.0-SNAPSHOT]
at org.apache.griffin.core.job.SparkSubmitJob.post2Livy(SparkSubmitJob.java:131) ~[classes!/:0.6.0-SNAPSHOT]
at org.apache.griffin.core.job.SparkSubmitJob.post2LivyWithRetry(SparkSubmitJob.java:224) ~[classes!/:0.6.0-SNAPSHOT]
at org.apache.griffin.core.job.SparkSubmitJob.saveJobInstance(SparkSubmitJob.java:213) ~[classes!/:0.6.0-SNAPSHOT]
at org.apache.griffin.core.job.SparkSubmitJob.execute(SparkSubmitJob.java:113) [classes!/:0.6.0-SNAPSHOT]
at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-2.2.2.jar!/:?]
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) [quartz-2.2.2.jar!/:?]
Waiting for your answer.
Regards
-----Original Message-----
From: Qian Wang <qw...@gmail.com>
Sent: Wednesday, August 21, 2019 7:17 PM
To: dev@griffin.apache.org
Subject: Re: Connect Griffin to Hive secured metastore
Hi,
You have an alternative method to get Hive Metadata by using Hive JDBC. If you want to use JDBC, you need change org.apache.griffin.core.metastore.hive.HiveMetaStoreController:
@Autowired
@Qualifier(value = "jdbcSvc")
private HiveMetaStoreService hiveMetaStoreService; Also, if your Hive is Authenticated by Kerberos, you need setup application.properties:
#Hive jdbc
hive.jdbc.className=org.apache.hive.jdbc.HiveDriver
hive.jdbc.url=jdbc:hive2://localhost:10000/ #your Hive url hive.need.kerberos=true # if you need Kerberos Auth hive.keytab.user=xxx@xx.com hive.keytab.path=/path/to/keytab/file #here is absolute path Hopefully can answer your question.
Best,
Eric
On Aug 21, 2019, 7:52 AM -0700, jose.martin_santacruz.ext@boehringer-ingelheim.com, wrote:
> Hello,
>
> We are trying to connect Griffin to a secured Hive metastore, does anybody know how to configure Griffin for this connection? We are getting authorization errors in the metastore.
>
> Waiting for your answer.
>
> Regards
Re: Connect Griffin to Hive secured metastore
Posted by Qian Wang <qw...@gmail.com>.
Hi,
You have an alternative method to get Hive Metadata by using Hive JDBC. If you want to use JDBC, you need change org.apache.griffin.core.metastore.hive.HiveMetaStoreController:
@Autowired
@Qualifier(value = "jdbcSvc")
private HiveMetaStoreService hiveMetaStoreService;
Also, if your Hive is Authenticated by Kerberos, you need setup application.properties:
#Hive jdbc
hive.jdbc.className=org.apache.hive.jdbc.HiveDriver
hive.jdbc.url=jdbc:hive2://localhost:10000/ #your Hive url
hive.need.kerberos=true # if you need Kerberos Auth
hive.keytab.user=xxx@xx.com
hive.keytab.path=/path/to/keytab/file #here is absolute path
Hopefully can answer your question.
Best,
Eric
On Aug 21, 2019, 7:52 AM -0700, jose.martin_santacruz.ext@boehringer-ingelheim.com, wrote:
> Hello,
>
> We are trying to connect Griffin to a secured Hive metastore, does anybody know how to configure Griffin for this connection? We are getting authorization errors in the metastore.
>
> Waiting for your answer.
>
> Regards