You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by 王杰 <ja...@Outlook.com> on 2021/01/12 15:12:22 UTC

[DISCUSS] Support obtaining Hive delegation tokens when submitting application to Yarn

Hi everyone,

Currently, Hive delegation token is not obtained when Flink submits the application in Yarn mode using kinit way. The ticket is https://issues.apache.org/jira/browse/FLINK-20714. I'd like to start a discussion about how to support this feature.

Maybe we have two options:
1. Using a reflection way to construct a Hive client to obtain the token, just same as the org.apache.flink.yarn.Utils.obtainTokenForHBase implementation.
2. Introduce a pluggable delegation provider via SPI. Delegation provider could be placed in connector related code, so reflection is not needed and is more extendable.



Both options have to handle how to specify the HiveConf to use. In Hive connector, user could specify both hiveConfDir and hadoopConfDir when creating HiveCatalog. The hadoopConfDir may not the same as the Hadoop configuration in HadoopModule.

Looking forward to your suggestions.

--
Best regards!
Jie Wang

Re: [DISCUSS] Support obtaining Hive delegation tokens when submitting application to Yarn

Posted by Rui Li <li...@gmail.com>.

Thanks Jie for driving this discussion. I also prefer a pluggable
delegation token provider. And I think users can use configurations to
specify the hive conf path, similar to how users specify a hive catalog.

On Wed, Jan 13, 2021 at 4:51 PM Till Rohrmann <tr...@apache.org> wrote:

> Hi Jie Wang,
>
> thanks for starting this discussion. To me the SPI approach sounds better
> because it is not as brittle as using reflection. Concerning the
> configuration, we could think about introducing some Hive specific
> configuration options which allow us to specify these paths. How are other
> projects which integrate with Hive are solving this problem?
>
> Cheers,
> Till
>
> On Tue, Jan 12, 2021 at 4:13 PM 王 杰 <ja...@outlook.com> wrote:
>
>> Hi everyone,
>>
>> Currently, Hive delegation token is not obtained when Flink submits the
>> application in Yarn mode using kinit way. The ticket is
>> https://issues.apache.org/jira/browse/FLINK-20714. I'd like to start a
>> discussion about how to support this feature.
>>
>> Maybe we have two options:
>> 1. Using a reflection way to construct a Hive client to obtain the token,
>> just same as the org.apache.flink.yarn.Utils.obtainTokenForHBase
>> implementation.
>> 2. Introduce a pluggable delegation provider via SPI. Delegation provider
>> could be placed in connector related code, so reflection is not needed and
>> is more extendable.
>>
>>
>>
>> Both options have to handle how to specify the HiveConf to use. In Hive
>> connector, user could specify both hiveConfDir and hadoopConfDir when
>> creating HiveCatalog. The hadoopConfDir may not the same as the Hadoop
>> configuration in HadoopModule.
>>
>> Looking forward to your suggestions.
>>
>> --
>> Best regards!
>> Jie Wang
>>
>>

-- 
Best regards!
Rui Li

Re: [DISCUSS] Support obtaining Hive delegation tokens when submitting application to Yarn

Posted by Till Rohrmann <tr...@apache.org>.

Hi Jie Wang,

thanks for starting this discussion. To me the SPI approach sounds better
because it is not as brittle as using reflection. Concerning the
configuration, we could think about introducing some Hive specific
configuration options which allow us to specify these paths. How are other
projects which integrate with Hive are solving this problem?

Cheers,
Till

On Tue, Jan 12, 2021 at 4:13 PM 王 杰 <ja...@outlook.com> wrote:

> Hi everyone,
>
> Currently, Hive delegation token is not obtained when Flink submits the
> application in Yarn mode using kinit way. The ticket is
> https://issues.apache.org/jira/browse/FLINK-20714. I'd like to start a
> discussion about how to support this feature.
>
> Maybe we have two options:
> 1. Using a reflection way to construct a Hive client to obtain the token,
> just same as the org.apache.flink.yarn.Utils.obtainTokenForHBase
> implementation.
> 2. Introduce a pluggable delegation provider via SPI. Delegation provider
> could be placed in connector related code, so reflection is not needed and
> is more extendable.
>
>
>
> Both options have to handle how to specify the HiveConf to use. In Hive
> connector, user could specify both hiveConfDir and hadoopConfDir when
> creating HiveCatalog. The hadoopConfDir may not the same as the Hadoop
> configuration in HadoopModule.
>
> Looking forward to your suggestions.
>
> --
> Best regards!
> Jie Wang
>
>