You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Kidong Lee <my...@gmail.com> on 2016/06/27 08:11:52 UTC
Problem with HDFS Encryption.
Hi,
I got some problem using drill with HDFS Encryption.
With Hive, DFS Storage, I got the errors like this:
Error: SYSTEM ERROR: IOException: No KeyProvider is configured, cannot
access an encrypted file
Even if I have added some confs below to drill storage plugin, the result
is the same:
in dfs storage:
"config": {
"hadoop.security.key.provider.path": "kms://http@xxxx.com;
xxxx.com:16000/kms",
"dfs.encryption.key.provider.uri": "kms://http@xxxx.com;
xxxx.com:16000/kms"
}
in hive storage:
"configProps": {
...
"hadoop.security.key.provider.path": "kms://http@xxxx.com;
xxxx.com:16000/kms",
"dfs.encryption.key.provider.uri": "kms://http@xxxx.com;
xxxx.com:16000/kms"
}
I have tested with hive for the tables of the encrypted files on hdfs, it
works fine.
Any idea.
- Kidong Lee.
Re: Problem with HDFS Encryption.
Posted by Parth Chandra <pc...@maprtech.com>.
Glad you figured it out, and thanks for posting the solution!
On Mon, Jun 27, 2016 at 7:31 PM, Kidong Lee <my...@gmail.com> wrote:
> Hi Parth,
>
> thanks for your advice.
>
> I solved the problem.
>
> The problem was that the drill user which run drill did not know
> HADOOP_HOME env, thus drill could not read my hadoop site.xml confs.
> I have exported HADOOP_HOME explicitly with drill user and run drill, it
> works fine!!!
>
> - Kidong.
>
>
>
>
> 2016-06-28 9:58 GMT+09:00 Parth Chandra <pc...@maprtech.com>:
>
> > You might need to make sure these files are in the Drill classpath. You
> > could create a link to these files (or a copy) in your DRILL_HOME/conf
> > directory and try.
> >
> > On Mon, Jun 27, 2016 at 4:45 PM, Kidong Lee <my...@gmail.com> wrote:
> >
> > > Yes, I did. I have also tested my parquet file which reside in the
> > > encryption zone and can be read with hive and parquet tool.
> > >
> > > - kidong
> > >
> > > 2016년 6월 28일 화요일, Parth Chandra<pc...@maprtech.com>님이 작성한 메시지:
> > >
> > > > Hi Kidong,
> > > >
> > > > I haven't tried this myself, but my guess is that the KMS settings
> > need
> > > > to be provided at the HDFS layer not in the drill storage plugin.
> > > >
> > > > Specify hadoop.security.key.provider.path in core-site
> > > >
> > > > Specify dfs.encryption.key.provider.uri in hdfs-site
> > > >
> > > > Or did you already do that?
> > > >
> > > > Parth
> > > >
> > > >
> > > > On Mon, Jun 27, 2016 at 1:11 AM, Kidong Lee <mykidong@gmail.com
> > > > <javascript:;>> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I got some problem using drill with HDFS Encryption.
> > > > >
> > > > > With Hive, DFS Storage, I got the errors like this:
> > > > > Error: SYSTEM ERROR: IOException: No KeyProvider is configured,
> > cannot
> > > > > access an encrypted file
> > > > >
> > > > > Even if I have added some confs below to drill storage plugin, the
> > > result
> > > > > is the same:
> > > > >
> > > > > in dfs storage:
> > > > > "config": {
> > > > > "hadoop.security.key.provider.path": "kms://http@xxxx.com
> > > > <javascript:;>;
> > > > > xxxx.com:16000/kms",
> > > > > "dfs.encryption.key.provider.uri": "kms://http@xxxx.com
> > > > <javascript:;>;
> > > > > xxxx.com:16000/kms"
> > > > > }
> > > > >
> > > > > in hive storage:
> > > > > "configProps": {
> > > > > ...
> > > > > "hadoop.security.key.provider.path": "kms://http@xxxx.com
> > > > <javascript:;>;
> > > > > xxxx.com:16000/kms",
> > > > > "dfs.encryption.key.provider.uri": "kms://http@xxxx.com
> > > > <javascript:;>;
> > > > > xxxx.com:16000/kms"
> > > > > }
> > > > >
> > > > > I have tested with hive for the tables of the encrypted files on
> > hdfs,
> > > it
> > > > > works fine.
> > > > >
> > > > > Any idea.
> > > > >
> > > > > - Kidong Lee.
> > > > >
> > > >
> > >
> >
>
Re: Problem with HDFS Encryption.
Posted by Parth Chandra <pc...@maprtech.com>.
You might need to make sure these files are in the Drill classpath. You
could create a link to these files (or a copy) in your DRILL_HOME/conf
directory and try.
On Mon, Jun 27, 2016 at 4:45 PM, Kidong Lee <my...@gmail.com> wrote:
> Yes, I did. I have also tested my parquet file which reside in the
> encryption zone and can be read with hive and parquet tool.
>
> - kidong
>
> 2016년 6월 28일 화요일, Parth Chandra<pc...@maprtech.com>님이 작성한 메시지:
>
> > Hi Kidong,
> >
> > I haven't tried this myself, but my guess is that the KMS settings need
> > to be provided at the HDFS layer not in the drill storage plugin.
> >
> > Specify hadoop.security.key.provider.path in core-site
> >
> > Specify dfs.encryption.key.provider.uri in hdfs-site
> >
> > Or did you already do that?
> >
> > Parth
> >
> >
> > On Mon, Jun 27, 2016 at 1:11 AM, Kidong Lee <mykidong@gmail.com
> > <javascript:;>> wrote:
> >
> > > Hi,
> > >
> > > I got some problem using drill with HDFS Encryption.
> > >
> > > With Hive, DFS Storage, I got the errors like this:
> > > Error: SYSTEM ERROR: IOException: No KeyProvider is configured, cannot
> > > access an encrypted file
> > >
> > > Even if I have added some confs below to drill storage plugin, the
> result
> > > is the same:
> > >
> > > in dfs storage:
> > > "config": {
> > > "hadoop.security.key.provider.path": "kms://http@xxxx.com
> > <javascript:;>;
> > > xxxx.com:16000/kms",
> > > "dfs.encryption.key.provider.uri": "kms://http@xxxx.com
> > <javascript:;>;
> > > xxxx.com:16000/kms"
> > > }
> > >
> > > in hive storage:
> > > "configProps": {
> > > ...
> > > "hadoop.security.key.provider.path": "kms://http@xxxx.com
> > <javascript:;>;
> > > xxxx.com:16000/kms",
> > > "dfs.encryption.key.provider.uri": "kms://http@xxxx.com
> > <javascript:;>;
> > > xxxx.com:16000/kms"
> > > }
> > >
> > > I have tested with hive for the tables of the encrypted files on hdfs,
> it
> > > works fine.
> > >
> > > Any idea.
> > >
> > > - Kidong Lee.
> > >
> >
>
Re: Problem with HDFS Encryption.
Posted by Kidong Lee <my...@gmail.com>.
Yes, I did. I have also tested my parquet file which reside in the
encryption zone and can be read with hive and parquet tool.
- kidong
2016년 6월 28일 화요일, Parth Chandra<pc...@maprtech.com>님이 작성한 메시지:
> Hi Kidong,
>
> I haven't tried this myself, but my guess is that the KMS settings need
> to be provided at the HDFS layer not in the drill storage plugin.
>
> Specify hadoop.security.key.provider.path in core-site
>
> Specify dfs.encryption.key.provider.uri in hdfs-site
>
> Or did you already do that?
>
> Parth
>
>
> On Mon, Jun 27, 2016 at 1:11 AM, Kidong Lee <mykidong@gmail.com
> <javascript:;>> wrote:
>
> > Hi,
> >
> > I got some problem using drill with HDFS Encryption.
> >
> > With Hive, DFS Storage, I got the errors like this:
> > Error: SYSTEM ERROR: IOException: No KeyProvider is configured, cannot
> > access an encrypted file
> >
> > Even if I have added some confs below to drill storage plugin, the result
> > is the same:
> >
> > in dfs storage:
> > "config": {
> > "hadoop.security.key.provider.path": "kms://http@xxxx.com
> <javascript:;>;
> > xxxx.com:16000/kms",
> > "dfs.encryption.key.provider.uri": "kms://http@xxxx.com
> <javascript:;>;
> > xxxx.com:16000/kms"
> > }
> >
> > in hive storage:
> > "configProps": {
> > ...
> > "hadoop.security.key.provider.path": "kms://http@xxxx.com
> <javascript:;>;
> > xxxx.com:16000/kms",
> > "dfs.encryption.key.provider.uri": "kms://http@xxxx.com
> <javascript:;>;
> > xxxx.com:16000/kms"
> > }
> >
> > I have tested with hive for the tables of the encrypted files on hdfs, it
> > works fine.
> >
> > Any idea.
> >
> > - Kidong Lee.
> >
>
Re: Problem with HDFS Encryption.
Posted by Parth Chandra <pc...@maprtech.com>.
Hi Kidong,
I haven't tried this myself, but my guess is that the KMS settings need
to be provided at the HDFS layer not in the drill storage plugin.
Specify hadoop.security.key.provider.path in core-site
Specify dfs.encryption.key.provider.uri in hdfs-site
Or did you already do that?
Parth
On Mon, Jun 27, 2016 at 1:11 AM, Kidong Lee <my...@gmail.com> wrote:
> Hi,
>
> I got some problem using drill with HDFS Encryption.
>
> With Hive, DFS Storage, I got the errors like this:
> Error: SYSTEM ERROR: IOException: No KeyProvider is configured, cannot
> access an encrypted file
>
> Even if I have added some confs below to drill storage plugin, the result
> is the same:
>
> in dfs storage:
> "config": {
> "hadoop.security.key.provider.path": "kms://http@xxxx.com;
> xxxx.com:16000/kms",
> "dfs.encryption.key.provider.uri": "kms://http@xxxx.com;
> xxxx.com:16000/kms"
> }
>
> in hive storage:
> "configProps": {
> ...
> "hadoop.security.key.provider.path": "kms://http@xxxx.com;
> xxxx.com:16000/kms",
> "dfs.encryption.key.provider.uri": "kms://http@xxxx.com;
> xxxx.com:16000/kms"
> }
>
> I have tested with hive for the tables of the encrypted files on hdfs, it
> works fine.
>
> Any idea.
>
> - Kidong Lee.
>