You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by Sanooj Padmakumar <p....@gmail.com> on 2016/04/26 05:23:49 UTC
Re: Kerberos and bulkload
Apologies if its against the convention to re-open an old discussion.
The exception was resolved after I added
"conf.set("fs.permissions.umask-mode", "000");" as per the suggestion
provided by Gabriel.
However the same exception has re-surfaced again in a new kerberized
cluster setup and I am unable to get it going. Some differences between the
earlier try and the current one is that earlier I used 4.3.0 version of
Phoenix and now I am on 4.5.2. I tried the same program in a non-kerberized
cluster with the 4.5.2 of Phoenix and that works.
I already tried setting file permission as shown above, and also running as
hbase user, but nothing helped.
Pasting the exception below.
16/04/25 11:25:02 INFO mapreduce.LoadIncrementalHFiles: Trying to load
hfile=hdfs://nameservice1/tmp/output/account7/ACCOUNT/0/f26e809c916a45198e19cd63a4971d65
first=<<masked>> last=<<masked>>
16/04/25 11:26:11 INFO client.RpcRetryingCaller: Call exception, tries=10,
retries=35, started=68213 ms ago, cancelled=false, msg=row '' on table
'ACCOUNT' at
region=ACCOUNT,,1461124051363.cbb334b331f00e23a6b83b85685cf793.,
hostname=<<masked>>,60020,1461243982079, seqNum=10
16/04/25 11:26:31 INFO client.RpcRetryingCaller: Call exception, tries=11,
retries=35, started=88415 ms ago, cancelled=false, msg=row '' on table
'ACCOUNT' at
region=ACCOUNT,,1461124051363.cbb334b331f00e23a6b83b85685cf793.,
hostname=<<masked>>,60020,1461243982079, seqNum=10
Any points to debug this will help.
Thanks
Sanooj
On Wed, Nov 18, 2015 at 8:26 PM, Gabriel Reid <ga...@gmail.com>
wrote:
> Re-adding the user list, which I accidentally left off.
>
> On Wed, Nov 18, 2015 at 3:55 PM, Gabriel Reid <ga...@gmail.com>
> wrote:
> > Yes, I believe that's correct, if you change the umask you make the
> > HFiles readable to all during creation.
> >
> > I believe that the alternate solutions listed on the jira ticket
> > (running the tool as the hbase user or using the alternate HBase
> > coprocessor for loading HFiles) won't have this drawback.
> >
> > - Gabriel
> >
> >
> > On Wed, Nov 18, 2015 at 3:49 PM, Sanooj Padmakumar <p....@gmail.com>
> wrote:
> >> Thank you Gabriel..
> >>
> >> Does it mean that the generated hfile be read/modified by any user other
> >> than "hbase" user?
> >>
> >> Regards
> >> Sanooj
> >>
> >> On 18 Nov 2015 02:39, "Gabriel Reid" <ga...@gmail.com> wrote:
> >>>
> >>> Hi Sanooj,
> >>>
> >>> Yes, I think that should do it, or you can pass that config parameter
> >>> as a command line parameter.
> >>>
> >>> - Gabriel
> >>>
> >>> On Tue, Nov 17, 2015 at 8:16 PM, Sanooj Padmakumar <p.sanooj@gmail.com
> >
> >>> wrote:
> >>> > Hi Gabriel
> >>> >
> >>> > Thank you so much
> >>> >
> >>> > I set the below property and it worked now.. I hope this is the
> correct
> >>> > thing to do ?
> >>> >
> >>> > conf.set("fs.permissions.umask-mode", "000");
> >>> >
> >>> >
> >>> > Thanks Again
> >>> >
> >>> > Sanooj
> >>> >
> >>> > On Wed, Nov 18, 2015 at 12:29 AM, Gabriel Reid <
> gabriel.reid@gmail.com>
> >>> > wrote:
> >>> >>
> >>> >> Hi Sanooj,
> >>> >>
> >>> >> I believe that this is related to the issue described in PHOENIX-976
> >>> >> [1]. In that case, it's not strictly related to Kerberos, but
> instead
> >>> >> to file permissions (could it be that your dev environment also
> >>> >> doesn't have file permissions turned on?)
> >>> >>
> >>> >> If you look at the comments on that jira ticket, there are a couple
> of
> >>> >> things that you could try doing to resolve this (running the import
> >>> >> job as the hbase user, or using custom file permissions, or using an
> >>> >> alternate incremental load coprocessor).
> >>> >>
> >>> >> - Gabriel
> >>> >>
> >>> >>
> >>> >> 1. https://issues.apache.org/jira/browse/PHOENIX-976
> >>> >>
> >>> >> On Tue, Nov 17, 2015 at 7:14 PM, Sanooj Padmakumar <
> p.sanooj@gmail.com>
> >>> >> wrote:
> >>> >> > Hello -
> >>> >> >
> >>> >> > I am using the bulkload of Phoenix on a cluster secured with
> >>> >> > Kerberos.
> >>> >> > The
> >>> >> > mapper runs fine, reducer runs fine .. and then the counters are
> >>> >> > printed
> >>> >> > fine.. finally the LoadIncrementalHFiles steps fails.. A portion
> of
> >>> >> > the
> >>> >> > log
> >>> >> > is given below..
> >>> >> >
> >>> >> >
> >>> >> > 15/11/17 09:44:48 INFO mapreduce.LoadIncrementalHFiles: Trying to
> >>> >> > load
> >>> >> > hfile=hdfs://..........<<masked>>>
> >>> >> > 15/11/17 09:45:56 INFO client.RpcRetryingCaller: Call exception,
> >>> >> > tries=10,
> >>> >> > retries=35, started=68220 ms ago, cancelled=false, msg=row '' on
> >>> >> > table
> >>> >> > 'TABLE1' at region=TABLE1,<<<masked>>>>, seqNum=26
> >>> >> > 15/11/17 09:46:16 INFO client.RpcRetryingCaller: Call exception,
> >>> >> > tries=11,
> >>> >> > retries=35, started=88315 ms ago, cancelled=false, msg=row '' on
> >>> >> > table
> >>> >> > 'TABLE1' at region=TABLE1,<<<masked>>>>, seqNum=26
> >>> >> >
> >>> >> > Is there any setting I should make inorder to make the program
> work
> >>> >> > on
> >>> >> > Kerberos secured environment ?
> >>> >> >
> >>> >> > Please note , our DEV environment doesnt use Kerberos and things
> are
> >>> >> > working
> >>> >> > just fine
> >>> >> >
> >>> >> > --
> >>> >> > Thanks in advance,
> >>> >> > Sanooj Padmakumar
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Thanks,
> >>> > Sanooj Padmakumar
>
--
Thanks,
Sanooj Padmakumar
Re: Kerberos and bulkload
Posted by Sanooj Padmakumar <p....@gmail.com>.
Setting the below configuration in the MR made it work
conf.set("hbase.coprocessor.region.classes", "");
Thanks everyone!
On Tue, Apr 26, 2016 at 8:53 AM, Sanooj Padmakumar <p....@gmail.com>
wrote:
> Apologies if its against the convention to re-open an old discussion.
>
> The exception was resolved after I added
> "conf.set("fs.permissions.umask-mode", "000");" as per the suggestion
> provided by Gabriel.
>
> However the same exception has re-surfaced again in a new kerberized
> cluster setup and I am unable to get it going. Some differences between the
> earlier try and the current one is that earlier I used 4.3.0 version of
> Phoenix and now I am on 4.5.2. I tried the same program in a non-kerberized
> cluster with the 4.5.2 of Phoenix and that works.
>
> I already tried setting file permission as shown above, and also running
> as hbase user, but nothing helped.
>
> Pasting the exception below.
>
> 16/04/25 11:25:02 INFO mapreduce.LoadIncrementalHFiles: Trying to load
> hfile=hdfs://nameservice1/tmp/output/account7/ACCOUNT/0/f26e809c916a45198e19cd63a4971d65
> first=<<masked>> last=<<masked>>
> 16/04/25 11:26:11 INFO client.RpcRetryingCaller: Call exception, tries=10,
> retries=35, started=68213 ms ago, cancelled=false, msg=row '' on table
> 'ACCOUNT' at
> region=ACCOUNT,,1461124051363.cbb334b331f00e23a6b83b85685cf793.,
> hostname=<<masked>>,60020,1461243982079, seqNum=10
> 16/04/25 11:26:31 INFO client.RpcRetryingCaller: Call exception, tries=11,
> retries=35, started=88415 ms ago, cancelled=false, msg=row '' on table
> 'ACCOUNT' at
> region=ACCOUNT,,1461124051363.cbb334b331f00e23a6b83b85685cf793.,
> hostname=<<masked>>,60020,1461243982079, seqNum=10
>
>
> Any points to debug this will help.
>
> Thanks
> Sanooj
>
> On Wed, Nov 18, 2015 at 8:26 PM, Gabriel Reid <ga...@gmail.com>
> wrote:
>
>> Re-adding the user list, which I accidentally left off.
>>
>> On Wed, Nov 18, 2015 at 3:55 PM, Gabriel Reid <ga...@gmail.com>
>> wrote:
>> > Yes, I believe that's correct, if you change the umask you make the
>> > HFiles readable to all during creation.
>> >
>> > I believe that the alternate solutions listed on the jira ticket
>> > (running the tool as the hbase user or using the alternate HBase
>> > coprocessor for loading HFiles) won't have this drawback.
>> >
>> > - Gabriel
>> >
>> >
>> > On Wed, Nov 18, 2015 at 3:49 PM, Sanooj Padmakumar <p....@gmail.com>
>> wrote:
>> >> Thank you Gabriel..
>> >>
>> >> Does it mean that the generated hfile be read/modified by any user
>> other
>> >> than "hbase" user?
>> >>
>> >> Regards
>> >> Sanooj
>> >>
>> >> On 18 Nov 2015 02:39, "Gabriel Reid" <ga...@gmail.com> wrote:
>> >>>
>> >>> Hi Sanooj,
>> >>>
>> >>> Yes, I think that should do it, or you can pass that config parameter
>> >>> as a command line parameter.
>> >>>
>> >>> - Gabriel
>> >>>
>> >>> On Tue, Nov 17, 2015 at 8:16 PM, Sanooj Padmakumar <
>> p.sanooj@gmail.com>
>> >>> wrote:
>> >>> > Hi Gabriel
>> >>> >
>> >>> > Thank you so much
>> >>> >
>> >>> > I set the below property and it worked now.. I hope this is the
>> correct
>> >>> > thing to do ?
>> >>> >
>> >>> > conf.set("fs.permissions.umask-mode", "000");
>> >>> >
>> >>> >
>> >>> > Thanks Again
>> >>> >
>> >>> > Sanooj
>> >>> >
>> >>> > On Wed, Nov 18, 2015 at 12:29 AM, Gabriel Reid <
>> gabriel.reid@gmail.com>
>> >>> > wrote:
>> >>> >>
>> >>> >> Hi Sanooj,
>> >>> >>
>> >>> >> I believe that this is related to the issue described in
>> PHOENIX-976
>> >>> >> [1]. In that case, it's not strictly related to Kerberos, but
>> instead
>> >>> >> to file permissions (could it be that your dev environment also
>> >>> >> doesn't have file permissions turned on?)
>> >>> >>
>> >>> >> If you look at the comments on that jira ticket, there are a
>> couple of
>> >>> >> things that you could try doing to resolve this (running the import
>> >>> >> job as the hbase user, or using custom file permissions, or using
>> an
>> >>> >> alternate incremental load coprocessor).
>> >>> >>
>> >>> >> - Gabriel
>> >>> >>
>> >>> >>
>> >>> >> 1. https://issues.apache.org/jira/browse/PHOENIX-976
>> >>> >>
>> >>> >> On Tue, Nov 17, 2015 at 7:14 PM, Sanooj Padmakumar <
>> p.sanooj@gmail.com>
>> >>> >> wrote:
>> >>> >> > Hello -
>> >>> >> >
>> >>> >> > I am using the bulkload of Phoenix on a cluster secured with
>> >>> >> > Kerberos.
>> >>> >> > The
>> >>> >> > mapper runs fine, reducer runs fine .. and then the counters are
>> >>> >> > printed
>> >>> >> > fine.. finally the LoadIncrementalHFiles steps fails.. A portion
>> of
>> >>> >> > the
>> >>> >> > log
>> >>> >> > is given below..
>> >>> >> >
>> >>> >> >
>> >>> >> > 15/11/17 09:44:48 INFO mapreduce.LoadIncrementalHFiles: Trying to
>> >>> >> > load
>> >>> >> > hfile=hdfs://..........<<masked>>>
>> >>> >> > 15/11/17 09:45:56 INFO client.RpcRetryingCaller: Call exception,
>> >>> >> > tries=10,
>> >>> >> > retries=35, started=68220 ms ago, cancelled=false, msg=row '' on
>> >>> >> > table
>> >>> >> > 'TABLE1' at region=TABLE1,<<<masked>>>>, seqNum=26
>> >>> >> > 15/11/17 09:46:16 INFO client.RpcRetryingCaller: Call exception,
>> >>> >> > tries=11,
>> >>> >> > retries=35, started=88315 ms ago, cancelled=false, msg=row '' on
>> >>> >> > table
>> >>> >> > 'TABLE1' at region=TABLE1,<<<masked>>>>, seqNum=26
>> >>> >> >
>> >>> >> > Is there any setting I should make inorder to make the program
>> work
>> >>> >> > on
>> >>> >> > Kerberos secured environment ?
>> >>> >> >
>> >>> >> > Please note , our DEV environment doesnt use Kerberos and things
>> are
>> >>> >> > working
>> >>> >> > just fine
>> >>> >> >
>> >>> >> > --
>> >>> >> > Thanks in advance,
>> >>> >> > Sanooj Padmakumar
>> >>> >
>> >>> >
>> >>> >
>> >>> >
>> >>> > --
>> >>> > Thanks,
>> >>> > Sanooj Padmakumar
>>
>
>
>
> --
> Thanks,
> Sanooj Padmakumar
>
--
Thanks,
Sanooj Padmakumar