You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by jamal sasha <ja...@gmail.com> on 2012/12/27 21:03:17 UTC
setting hadoop for pseudo distributed mode.
Hi,
So I am still in process of learning hadoop.
I tried to run wordcount.java (by writing my own mapper reducer.. creating
jar and then running it in a pseudo distributed mode).
At that time I got an error, something like
ERROR security.UserGroupInformation: PriviledgedActionException as:mhduser
cause:org.apache.hadoop.mapred.InvalidInputException: Input path does not
exist: hdfs://localhost:54310/user/hduser/wiki
So I googled around.. and found that I should put the following two lines
on my driver code:
conf.addResource(new Path("/usr/local/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/usr/local/hadoop/conf/hdfs-site.xml file"));
(path of where my core-site and hdfs-site are) and after that it ran just
fine.
Q1) How did putting those two lines solved the issue??
Q2)
I am now using third party libraries which are taking input from hdfs and
writing output to hdfs...
But in an intermediatory step, it creates a raw output.. I am again getting
the error:
ERROR security.UserGroupInformation: PriviledgedActionException as:mhduser
cause:org.apache.hadoop.mapred.InvalidInputException: Input path does not
exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
How do I resolve this?
Any suggestions.
THanks
Jamal.
Re: setting hadoop for pseudo distributed mode.
Posted by Mohammad Tariq <do...@gmail.com>.
what are those libraries and how are they reading data from HDFS? you were
trying with MR jobs if i'm not wrong? in order to perform read/write on
HDFS we need HDFS API with a Configuration object. how are you doing it
here?
Best Regards,
Tariq
+91-9741563634
https://mtariq.jux.com/
On Fri, Dec 28, 2012 at 2:38 AM, jamal sasha <ja...@gmail.com> wrote:
> Hi,
> Thanks for throwing insight.
>
> So the code snippet looks like this
>
> String interout = final_output + "/intermediate";
>
> try {
> new CreateInterOutput().main(new String[] { input, interout });
> } catch (Exception e) {
> e.printStackTrace();
> return;
> }
>
> try {
> new CreateFinalOutput().main(new String[] { interout, final_output });
>
> } catch (Exception e) {
> e.printStackTrace();
> return;
> }
>
> Any suggestions where it might be faltering?
>
>
> On Thu, Dec 27, 2012 at 12:49 PM, Mohammad Tariq <do...@gmail.com>wrote:
>
>> Hello Jamal,
>>
>> Please find my commands embedded below :
>>
>> Q1) How did putting those two lines solved the issue??
>> >> By adding those two resources you make sure that your code looks for
>> the input path inside HDFS, which would otherwise look for it in the local
>> FS by default. The files core-site.xml and hdfs-site.xml tell your code
>> where to go for NN and DN.
>>
>> Q2)
>> I am now using third party libraries which are taking input from hdfs
>> and writing output to hdfs...
>> But in an intermediatory step, it creates a raw output.. I am again
>> getting the error:
>> ERROR security.UserGroupInformation: PriviledgedActionException
>> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
>> does not exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
>> How do I resolve this?
>> >>If you are able to compile your code properly then there is no problem
>> with the third party libraries which you are using. It looks like to me
>> that your code doesn't have the proper info about the intermediate path.
>> Please make sure you have told your code the exact location of intermediate
>> output.
>>
>>
>> Best Regards,
>> Tariq
>> +91-9741563634
>> https://mtariq.jux.com/
>>
>>
>> On Fri, Dec 28, 2012 at 1:33 AM, jamal sasha <ja...@gmail.com>wrote:
>>
>>> Hi,
>>> So I am still in process of learning hadoop.
>>> I tried to run wordcount.java (by writing my own mapper reducer..
>>> creating jar and then running it in a pseudo distributed mode).
>>>
>>> At that time I got an error, something like
>>> ERROR security.UserGroupInformation: PriviledgedActionException
>>> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
>>> does not exist: hdfs://localhost:54310/user/hduser/wiki
>>> So I googled around.. and found that I should put the following two
>>> lines on my driver code:
>>>
>>> conf.addResource(new Path("/usr/local/hadoop/conf/core-site.xml"));
>>> conf.addResource(new Path("/usr/local/hadoop/conf/hdfs-site.xml
>>> file"));
>>>
>>> (path of where my core-site and hdfs-site are) and after that it ran
>>> just fine.
>>>
>>> Q1) How did putting those two lines solved the issue??
>>> Q2)
>>> I am now using third party libraries which are taking input from hdfs
>>> and writing output to hdfs...
>>> But in an intermediatory step, it creates a raw output.. I am again
>>> getting the error:
>>> ERROR security.UserGroupInformation: PriviledgedActionException
>>> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
>>> does not exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
>>> How do I resolve this?
>>>
>>> Any suggestions.
>>> THanks
>>> Jamal.
>>>
>>
>>
>
Re: setting hadoop for pseudo distributed mode.
Posted by Mohammad Tariq <do...@gmail.com>.
what are those libraries and how are they reading data from HDFS? you were
trying with MR jobs if i'm not wrong? in order to perform read/write on
HDFS we need HDFS API with a Configuration object. how are you doing it
here?
Best Regards,
Tariq
+91-9741563634
https://mtariq.jux.com/
On Fri, Dec 28, 2012 at 2:38 AM, jamal sasha <ja...@gmail.com> wrote:
> Hi,
> Thanks for throwing insight.
>
> So the code snippet looks like this
>
> String interout = final_output + "/intermediate";
>
> try {
> new CreateInterOutput().main(new String[] { input, interout });
> } catch (Exception e) {
> e.printStackTrace();
> return;
> }
>
> try {
> new CreateFinalOutput().main(new String[] { interout, final_output });
>
> } catch (Exception e) {
> e.printStackTrace();
> return;
> }
>
> Any suggestions where it might be faltering?
>
>
> On Thu, Dec 27, 2012 at 12:49 PM, Mohammad Tariq <do...@gmail.com>wrote:
>
>> Hello Jamal,
>>
>> Please find my commands embedded below :
>>
>> Q1) How did putting those two lines solved the issue??
>> >> By adding those two resources you make sure that your code looks for
>> the input path inside HDFS, which would otherwise look for it in the local
>> FS by default. The files core-site.xml and hdfs-site.xml tell your code
>> where to go for NN and DN.
>>
>> Q2)
>> I am now using third party libraries which are taking input from hdfs
>> and writing output to hdfs...
>> But in an intermediatory step, it creates a raw output.. I am again
>> getting the error:
>> ERROR security.UserGroupInformation: PriviledgedActionException
>> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
>> does not exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
>> How do I resolve this?
>> >>If you are able to compile your code properly then there is no problem
>> with the third party libraries which you are using. It looks like to me
>> that your code doesn't have the proper info about the intermediate path.
>> Please make sure you have told your code the exact location of intermediate
>> output.
>>
>>
>> Best Regards,
>> Tariq
>> +91-9741563634
>> https://mtariq.jux.com/
>>
>>
>> On Fri, Dec 28, 2012 at 1:33 AM, jamal sasha <ja...@gmail.com>wrote:
>>
>>> Hi,
>>> So I am still in process of learning hadoop.
>>> I tried to run wordcount.java (by writing my own mapper reducer..
>>> creating jar and then running it in a pseudo distributed mode).
>>>
>>> At that time I got an error, something like
>>> ERROR security.UserGroupInformation: PriviledgedActionException
>>> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
>>> does not exist: hdfs://localhost:54310/user/hduser/wiki
>>> So I googled around.. and found that I should put the following two
>>> lines on my driver code:
>>>
>>> conf.addResource(new Path("/usr/local/hadoop/conf/core-site.xml"));
>>> conf.addResource(new Path("/usr/local/hadoop/conf/hdfs-site.xml
>>> file"));
>>>
>>> (path of where my core-site and hdfs-site are) and after that it ran
>>> just fine.
>>>
>>> Q1) How did putting those two lines solved the issue??
>>> Q2)
>>> I am now using third party libraries which are taking input from hdfs
>>> and writing output to hdfs...
>>> But in an intermediatory step, it creates a raw output.. I am again
>>> getting the error:
>>> ERROR security.UserGroupInformation: PriviledgedActionException
>>> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
>>> does not exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
>>> How do I resolve this?
>>>
>>> Any suggestions.
>>> THanks
>>> Jamal.
>>>
>>
>>
>
Re: setting hadoop for pseudo distributed mode.
Posted by Mohammad Tariq <do...@gmail.com>.
what are those libraries and how are they reading data from HDFS? you were
trying with MR jobs if i'm not wrong? in order to perform read/write on
HDFS we need HDFS API with a Configuration object. how are you doing it
here?
Best Regards,
Tariq
+91-9741563634
https://mtariq.jux.com/
On Fri, Dec 28, 2012 at 2:38 AM, jamal sasha <ja...@gmail.com> wrote:
> Hi,
> Thanks for throwing insight.
>
> So the code snippet looks like this
>
> String interout = final_output + "/intermediate";
>
> try {
> new CreateInterOutput().main(new String[] { input, interout });
> } catch (Exception e) {
> e.printStackTrace();
> return;
> }
>
> try {
> new CreateFinalOutput().main(new String[] { interout, final_output });
>
> } catch (Exception e) {
> e.printStackTrace();
> return;
> }
>
> Any suggestions where it might be faltering?
>
>
> On Thu, Dec 27, 2012 at 12:49 PM, Mohammad Tariq <do...@gmail.com>wrote:
>
>> Hello Jamal,
>>
>> Please find my commands embedded below :
>>
>> Q1) How did putting those two lines solved the issue??
>> >> By adding those two resources you make sure that your code looks for
>> the input path inside HDFS, which would otherwise look for it in the local
>> FS by default. The files core-site.xml and hdfs-site.xml tell your code
>> where to go for NN and DN.
>>
>> Q2)
>> I am now using third party libraries which are taking input from hdfs
>> and writing output to hdfs...
>> But in an intermediatory step, it creates a raw output.. I am again
>> getting the error:
>> ERROR security.UserGroupInformation: PriviledgedActionException
>> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
>> does not exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
>> How do I resolve this?
>> >>If you are able to compile your code properly then there is no problem
>> with the third party libraries which you are using. It looks like to me
>> that your code doesn't have the proper info about the intermediate path.
>> Please make sure you have told your code the exact location of intermediate
>> output.
>>
>>
>> Best Regards,
>> Tariq
>> +91-9741563634
>> https://mtariq.jux.com/
>>
>>
>> On Fri, Dec 28, 2012 at 1:33 AM, jamal sasha <ja...@gmail.com>wrote:
>>
>>> Hi,
>>> So I am still in process of learning hadoop.
>>> I tried to run wordcount.java (by writing my own mapper reducer..
>>> creating jar and then running it in a pseudo distributed mode).
>>>
>>> At that time I got an error, something like
>>> ERROR security.UserGroupInformation: PriviledgedActionException
>>> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
>>> does not exist: hdfs://localhost:54310/user/hduser/wiki
>>> So I googled around.. and found that I should put the following two
>>> lines on my driver code:
>>>
>>> conf.addResource(new Path("/usr/local/hadoop/conf/core-site.xml"));
>>> conf.addResource(new Path("/usr/local/hadoop/conf/hdfs-site.xml
>>> file"));
>>>
>>> (path of where my core-site and hdfs-site are) and after that it ran
>>> just fine.
>>>
>>> Q1) How did putting those two lines solved the issue??
>>> Q2)
>>> I am now using third party libraries which are taking input from hdfs
>>> and writing output to hdfs...
>>> But in an intermediatory step, it creates a raw output.. I am again
>>> getting the error:
>>> ERROR security.UserGroupInformation: PriviledgedActionException
>>> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
>>> does not exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
>>> How do I resolve this?
>>>
>>> Any suggestions.
>>> THanks
>>> Jamal.
>>>
>>
>>
>
Re: setting hadoop for pseudo distributed mode.
Posted by Mohammad Tariq <do...@gmail.com>.
what are those libraries and how are they reading data from HDFS? you were
trying with MR jobs if i'm not wrong? in order to perform read/write on
HDFS we need HDFS API with a Configuration object. how are you doing it
here?
Best Regards,
Tariq
+91-9741563634
https://mtariq.jux.com/
On Fri, Dec 28, 2012 at 2:38 AM, jamal sasha <ja...@gmail.com> wrote:
> Hi,
> Thanks for throwing insight.
>
> So the code snippet looks like this
>
> String interout = final_output + "/intermediate";
>
> try {
> new CreateInterOutput().main(new String[] { input, interout });
> } catch (Exception e) {
> e.printStackTrace();
> return;
> }
>
> try {
> new CreateFinalOutput().main(new String[] { interout, final_output });
>
> } catch (Exception e) {
> e.printStackTrace();
> return;
> }
>
> Any suggestions where it might be faltering?
>
>
> On Thu, Dec 27, 2012 at 12:49 PM, Mohammad Tariq <do...@gmail.com>wrote:
>
>> Hello Jamal,
>>
>> Please find my commands embedded below :
>>
>> Q1) How did putting those two lines solved the issue??
>> >> By adding those two resources you make sure that your code looks for
>> the input path inside HDFS, which would otherwise look for it in the local
>> FS by default. The files core-site.xml and hdfs-site.xml tell your code
>> where to go for NN and DN.
>>
>> Q2)
>> I am now using third party libraries which are taking input from hdfs
>> and writing output to hdfs...
>> But in an intermediatory step, it creates a raw output.. I am again
>> getting the error:
>> ERROR security.UserGroupInformation: PriviledgedActionException
>> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
>> does not exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
>> How do I resolve this?
>> >>If you are able to compile your code properly then there is no problem
>> with the third party libraries which you are using. It looks like to me
>> that your code doesn't have the proper info about the intermediate path.
>> Please make sure you have told your code the exact location of intermediate
>> output.
>>
>>
>> Best Regards,
>> Tariq
>> +91-9741563634
>> https://mtariq.jux.com/
>>
>>
>> On Fri, Dec 28, 2012 at 1:33 AM, jamal sasha <ja...@gmail.com>wrote:
>>
>>> Hi,
>>> So I am still in process of learning hadoop.
>>> I tried to run wordcount.java (by writing my own mapper reducer..
>>> creating jar and then running it in a pseudo distributed mode).
>>>
>>> At that time I got an error, something like
>>> ERROR security.UserGroupInformation: PriviledgedActionException
>>> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
>>> does not exist: hdfs://localhost:54310/user/hduser/wiki
>>> So I googled around.. and found that I should put the following two
>>> lines on my driver code:
>>>
>>> conf.addResource(new Path("/usr/local/hadoop/conf/core-site.xml"));
>>> conf.addResource(new Path("/usr/local/hadoop/conf/hdfs-site.xml
>>> file"));
>>>
>>> (path of where my core-site and hdfs-site are) and after that it ran
>>> just fine.
>>>
>>> Q1) How did putting those two lines solved the issue??
>>> Q2)
>>> I am now using third party libraries which are taking input from hdfs
>>> and writing output to hdfs...
>>> But in an intermediatory step, it creates a raw output.. I am again
>>> getting the error:
>>> ERROR security.UserGroupInformation: PriviledgedActionException
>>> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
>>> does not exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
>>> How do I resolve this?
>>>
>>> Any suggestions.
>>> THanks
>>> Jamal.
>>>
>>
>>
>
Re: setting hadoop for pseudo distributed mode.
Posted by jamal sasha <ja...@gmail.com>.
Hi,
Thanks for throwing insight.
So the code snippet looks like this
String interout = final_output + "/intermediate";
try {
new CreateInterOutput().main(new String[] { input, interout });
} catch (Exception e) {
e.printStackTrace();
return;
}
try {
new CreateFinalOutput().main(new String[] { interout, final_output });
} catch (Exception e) {
e.printStackTrace();
return;
}
Any suggestions where it might be faltering?
On Thu, Dec 27, 2012 at 12:49 PM, Mohammad Tariq <do...@gmail.com> wrote:
> Hello Jamal,
>
> Please find my commands embedded below :
>
> Q1) How did putting those two lines solved the issue??
> >> By adding those two resources you make sure that your code looks for
> the input path inside HDFS, which would otherwise look for it in the local
> FS by default. The files core-site.xml and hdfs-site.xml tell your code
> where to go for NN and DN.
>
> Q2)
> I am now using third party libraries which are taking input from hdfs
> and writing output to hdfs...
> But in an intermediatory step, it creates a raw output.. I am again
> getting the error:
> ERROR security.UserGroupInformation: PriviledgedActionException
> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
> does not exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
> How do I resolve this?
> >>If you are able to compile your code properly then there is no problem
> with the third party libraries which you are using. It looks like to me
> that your code doesn't have the proper info about the intermediate path.
> Please make sure you have told your code the exact location of intermediate
> output.
>
>
> Best Regards,
> Tariq
> +91-9741563634
> https://mtariq.jux.com/
>
>
> On Fri, Dec 28, 2012 at 1:33 AM, jamal sasha <ja...@gmail.com>wrote:
>
>> Hi,
>> So I am still in process of learning hadoop.
>> I tried to run wordcount.java (by writing my own mapper reducer..
>> creating jar and then running it in a pseudo distributed mode).
>>
>> At that time I got an error, something like
>> ERROR security.UserGroupInformation: PriviledgedActionException
>> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
>> does not exist: hdfs://localhost:54310/user/hduser/wiki
>> So I googled around.. and found that I should put the following two lines
>> on my driver code:
>>
>> conf.addResource(new Path("/usr/local/hadoop/conf/core-site.xml"));
>> conf.addResource(new Path("/usr/local/hadoop/conf/hdfs-site.xml
>> file"));
>>
>> (path of where my core-site and hdfs-site are) and after that it ran just
>> fine.
>>
>> Q1) How did putting those two lines solved the issue??
>> Q2)
>> I am now using third party libraries which are taking input from hdfs
>> and writing output to hdfs...
>> But in an intermediatory step, it creates a raw output.. I am again
>> getting the error:
>> ERROR security.UserGroupInformation: PriviledgedActionException
>> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
>> does not exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
>> How do I resolve this?
>>
>> Any suggestions.
>> THanks
>> Jamal.
>>
>
>
Re: setting hadoop for pseudo distributed mode.
Posted by jamal sasha <ja...@gmail.com>.
Hi,
Thanks for throwing insight.
So the code snippet looks like this
String interout = final_output + "/intermediate";
try {
new CreateInterOutput().main(new String[] { input, interout });
} catch (Exception e) {
e.printStackTrace();
return;
}
try {
new CreateFinalOutput().main(new String[] { interout, final_output });
} catch (Exception e) {
e.printStackTrace();
return;
}
Any suggestions where it might be faltering?
On Thu, Dec 27, 2012 at 12:49 PM, Mohammad Tariq <do...@gmail.com> wrote:
> Hello Jamal,
>
> Please find my commands embedded below :
>
> Q1) How did putting those two lines solved the issue??
> >> By adding those two resources you make sure that your code looks for
> the input path inside HDFS, which would otherwise look for it in the local
> FS by default. The files core-site.xml and hdfs-site.xml tell your code
> where to go for NN and DN.
>
> Q2)
> I am now using third party libraries which are taking input from hdfs
> and writing output to hdfs...
> But in an intermediatory step, it creates a raw output.. I am again
> getting the error:
> ERROR security.UserGroupInformation: PriviledgedActionException
> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
> does not exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
> How do I resolve this?
> >>If you are able to compile your code properly then there is no problem
> with the third party libraries which you are using. It looks like to me
> that your code doesn't have the proper info about the intermediate path.
> Please make sure you have told your code the exact location of intermediate
> output.
>
>
> Best Regards,
> Tariq
> +91-9741563634
> https://mtariq.jux.com/
>
>
> On Fri, Dec 28, 2012 at 1:33 AM, jamal sasha <ja...@gmail.com>wrote:
>
>> Hi,
>> So I am still in process of learning hadoop.
>> I tried to run wordcount.java (by writing my own mapper reducer..
>> creating jar and then running it in a pseudo distributed mode).
>>
>> At that time I got an error, something like
>> ERROR security.UserGroupInformation: PriviledgedActionException
>> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
>> does not exist: hdfs://localhost:54310/user/hduser/wiki
>> So I googled around.. and found that I should put the following two lines
>> on my driver code:
>>
>> conf.addResource(new Path("/usr/local/hadoop/conf/core-site.xml"));
>> conf.addResource(new Path("/usr/local/hadoop/conf/hdfs-site.xml
>> file"));
>>
>> (path of where my core-site and hdfs-site are) and after that it ran just
>> fine.
>>
>> Q1) How did putting those two lines solved the issue??
>> Q2)
>> I am now using third party libraries which are taking input from hdfs
>> and writing output to hdfs...
>> But in an intermediatory step, it creates a raw output.. I am again
>> getting the error:
>> ERROR security.UserGroupInformation: PriviledgedActionException
>> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
>> does not exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
>> How do I resolve this?
>>
>> Any suggestions.
>> THanks
>> Jamal.
>>
>
>
Re: setting hadoop for pseudo distributed mode.
Posted by jamal sasha <ja...@gmail.com>.
Hi,
Thanks for throwing insight.
So the code snippet looks like this
String interout = final_output + "/intermediate";
try {
new CreateInterOutput().main(new String[] { input, interout });
} catch (Exception e) {
e.printStackTrace();
return;
}
try {
new CreateFinalOutput().main(new String[] { interout, final_output });
} catch (Exception e) {
e.printStackTrace();
return;
}
Any suggestions where it might be faltering?
On Thu, Dec 27, 2012 at 12:49 PM, Mohammad Tariq <do...@gmail.com> wrote:
> Hello Jamal,
>
> Please find my commands embedded below :
>
> Q1) How did putting those two lines solved the issue??
> >> By adding those two resources you make sure that your code looks for
> the input path inside HDFS, which would otherwise look for it in the local
> FS by default. The files core-site.xml and hdfs-site.xml tell your code
> where to go for NN and DN.
>
> Q2)
> I am now using third party libraries which are taking input from hdfs
> and writing output to hdfs...
> But in an intermediatory step, it creates a raw output.. I am again
> getting the error:
> ERROR security.UserGroupInformation: PriviledgedActionException
> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
> does not exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
> How do I resolve this?
> >>If you are able to compile your code properly then there is no problem
> with the third party libraries which you are using. It looks like to me
> that your code doesn't have the proper info about the intermediate path.
> Please make sure you have told your code the exact location of intermediate
> output.
>
>
> Best Regards,
> Tariq
> +91-9741563634
> https://mtariq.jux.com/
>
>
> On Fri, Dec 28, 2012 at 1:33 AM, jamal sasha <ja...@gmail.com>wrote:
>
>> Hi,
>> So I am still in process of learning hadoop.
>> I tried to run wordcount.java (by writing my own mapper reducer..
>> creating jar and then running it in a pseudo distributed mode).
>>
>> At that time I got an error, something like
>> ERROR security.UserGroupInformation: PriviledgedActionException
>> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
>> does not exist: hdfs://localhost:54310/user/hduser/wiki
>> So I googled around.. and found that I should put the following two lines
>> on my driver code:
>>
>> conf.addResource(new Path("/usr/local/hadoop/conf/core-site.xml"));
>> conf.addResource(new Path("/usr/local/hadoop/conf/hdfs-site.xml
>> file"));
>>
>> (path of where my core-site and hdfs-site are) and after that it ran just
>> fine.
>>
>> Q1) How did putting those two lines solved the issue??
>> Q2)
>> I am now using third party libraries which are taking input from hdfs
>> and writing output to hdfs...
>> But in an intermediatory step, it creates a raw output.. I am again
>> getting the error:
>> ERROR security.UserGroupInformation: PriviledgedActionException
>> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
>> does not exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
>> How do I resolve this?
>>
>> Any suggestions.
>> THanks
>> Jamal.
>>
>
>
Re: setting hadoop for pseudo distributed mode.
Posted by jamal sasha <ja...@gmail.com>.
Hi,
Thanks for throwing insight.
So the code snippet looks like this
String interout = final_output + "/intermediate";
try {
new CreateInterOutput().main(new String[] { input, interout });
} catch (Exception e) {
e.printStackTrace();
return;
}
try {
new CreateFinalOutput().main(new String[] { interout, final_output });
} catch (Exception e) {
e.printStackTrace();
return;
}
Any suggestions where it might be faltering?
On Thu, Dec 27, 2012 at 12:49 PM, Mohammad Tariq <do...@gmail.com> wrote:
> Hello Jamal,
>
> Please find my commands embedded below :
>
> Q1) How did putting those two lines solved the issue??
> >> By adding those two resources you make sure that your code looks for
> the input path inside HDFS, which would otherwise look for it in the local
> FS by default. The files core-site.xml and hdfs-site.xml tell your code
> where to go for NN and DN.
>
> Q2)
> I am now using third party libraries which are taking input from hdfs
> and writing output to hdfs...
> But in an intermediatory step, it creates a raw output.. I am again
> getting the error:
> ERROR security.UserGroupInformation: PriviledgedActionException
> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
> does not exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
> How do I resolve this?
> >>If you are able to compile your code properly then there is no problem
> with the third party libraries which you are using. It looks like to me
> that your code doesn't have the proper info about the intermediate path.
> Please make sure you have told your code the exact location of intermediate
> output.
>
>
> Best Regards,
> Tariq
> +91-9741563634
> https://mtariq.jux.com/
>
>
> On Fri, Dec 28, 2012 at 1:33 AM, jamal sasha <ja...@gmail.com>wrote:
>
>> Hi,
>> So I am still in process of learning hadoop.
>> I tried to run wordcount.java (by writing my own mapper reducer..
>> creating jar and then running it in a pseudo distributed mode).
>>
>> At that time I got an error, something like
>> ERROR security.UserGroupInformation: PriviledgedActionException
>> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
>> does not exist: hdfs://localhost:54310/user/hduser/wiki
>> So I googled around.. and found that I should put the following two lines
>> on my driver code:
>>
>> conf.addResource(new Path("/usr/local/hadoop/conf/core-site.xml"));
>> conf.addResource(new Path("/usr/local/hadoop/conf/hdfs-site.xml
>> file"));
>>
>> (path of where my core-site and hdfs-site are) and after that it ran just
>> fine.
>>
>> Q1) How did putting those two lines solved the issue??
>> Q2)
>> I am now using third party libraries which are taking input from hdfs
>> and writing output to hdfs...
>> But in an intermediatory step, it creates a raw output.. I am again
>> getting the error:
>> ERROR security.UserGroupInformation: PriviledgedActionException
>> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
>> does not exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
>> How do I resolve this?
>>
>> Any suggestions.
>> THanks
>> Jamal.
>>
>
>
Re: setting hadoop for pseudo distributed mode.
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Jamal,
Please find my commands embedded below :
Q1) How did putting those two lines solved the issue??
>> By adding those two resources you make sure that your code looks for the
input path inside HDFS, which would otherwise look for it in the local FS
by default. The files core-site.xml and hdfs-site.xml tell your code where
to go for NN and DN.
Q2)
I am now using third party libraries which are taking input from hdfs and
writing output to hdfs...
But in an intermediatory step, it creates a raw output.. I am again getting
the error:
ERROR security.UserGroupInformation: PriviledgedActionException as:mhduser
cause:org.apache.hadoop.mapred.InvalidInputException: Input path does not
exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
How do I resolve this?
>>If you are able to compile your code properly then there is no problem
with the third party libraries which you are using. It looks like to me
that your code doesn't have the proper info about the intermediate path.
Please make sure you have told your code the exact location of intermediate
output.
Best Regards,
Tariq
+91-9741563634
https://mtariq.jux.com/
On Fri, Dec 28, 2012 at 1:33 AM, jamal sasha <ja...@gmail.com> wrote:
> Hi,
> So I am still in process of learning hadoop.
> I tried to run wordcount.java (by writing my own mapper reducer.. creating
> jar and then running it in a pseudo distributed mode).
>
> At that time I got an error, something like
> ERROR security.UserGroupInformation: PriviledgedActionException
> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
> does not exist: hdfs://localhost:54310/user/hduser/wiki
> So I googled around.. and found that I should put the following two lines
> on my driver code:
>
> conf.addResource(new Path("/usr/local/hadoop/conf/core-site.xml"));
> conf.addResource(new Path("/usr/local/hadoop/conf/hdfs-site.xml
> file"));
>
> (path of where my core-site and hdfs-site are) and after that it ran just
> fine.
>
> Q1) How did putting those two lines solved the issue??
> Q2)
> I am now using third party libraries which are taking input from hdfs
> and writing output to hdfs...
> But in an intermediatory step, it creates a raw output.. I am again
> getting the error:
> ERROR security.UserGroupInformation: PriviledgedActionException
> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
> does not exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
> How do I resolve this?
>
> Any suggestions.
> THanks
> Jamal.
>
Re: setting hadoop for pseudo distributed mode.
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Jamal,
Please find my commands embedded below :
Q1) How did putting those two lines solved the issue??
>> By adding those two resources you make sure that your code looks for the
input path inside HDFS, which would otherwise look for it in the local FS
by default. The files core-site.xml and hdfs-site.xml tell your code where
to go for NN and DN.
Q2)
I am now using third party libraries which are taking input from hdfs and
writing output to hdfs...
But in an intermediatory step, it creates a raw output.. I am again getting
the error:
ERROR security.UserGroupInformation: PriviledgedActionException as:mhduser
cause:org.apache.hadoop.mapred.InvalidInputException: Input path does not
exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
How do I resolve this?
>>If you are able to compile your code properly then there is no problem
with the third party libraries which you are using. It looks like to me
that your code doesn't have the proper info about the intermediate path.
Please make sure you have told your code the exact location of intermediate
output.
Best Regards,
Tariq
+91-9741563634
https://mtariq.jux.com/
On Fri, Dec 28, 2012 at 1:33 AM, jamal sasha <ja...@gmail.com> wrote:
> Hi,
> So I am still in process of learning hadoop.
> I tried to run wordcount.java (by writing my own mapper reducer.. creating
> jar and then running it in a pseudo distributed mode).
>
> At that time I got an error, something like
> ERROR security.UserGroupInformation: PriviledgedActionException
> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
> does not exist: hdfs://localhost:54310/user/hduser/wiki
> So I googled around.. and found that I should put the following two lines
> on my driver code:
>
> conf.addResource(new Path("/usr/local/hadoop/conf/core-site.xml"));
> conf.addResource(new Path("/usr/local/hadoop/conf/hdfs-site.xml
> file"));
>
> (path of where my core-site and hdfs-site are) and after that it ran just
> fine.
>
> Q1) How did putting those two lines solved the issue??
> Q2)
> I am now using third party libraries which are taking input from hdfs
> and writing output to hdfs...
> But in an intermediatory step, it creates a raw output.. I am again
> getting the error:
> ERROR security.UserGroupInformation: PriviledgedActionException
> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
> does not exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
> How do I resolve this?
>
> Any suggestions.
> THanks
> Jamal.
>
Re: setting hadoop for pseudo distributed mode.
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Jamal,
Please find my commands embedded below :
Q1) How did putting those two lines solved the issue??
>> By adding those two resources you make sure that your code looks for the
input path inside HDFS, which would otherwise look for it in the local FS
by default. The files core-site.xml and hdfs-site.xml tell your code where
to go for NN and DN.
Q2)
I am now using third party libraries which are taking input from hdfs and
writing output to hdfs...
But in an intermediatory step, it creates a raw output.. I am again getting
the error:
ERROR security.UserGroupInformation: PriviledgedActionException as:mhduser
cause:org.apache.hadoop.mapred.InvalidInputException: Input path does not
exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
How do I resolve this?
>>If you are able to compile your code properly then there is no problem
with the third party libraries which you are using. It looks like to me
that your code doesn't have the proper info about the intermediate path.
Please make sure you have told your code the exact location of intermediate
output.
Best Regards,
Tariq
+91-9741563634
https://mtariq.jux.com/
On Fri, Dec 28, 2012 at 1:33 AM, jamal sasha <ja...@gmail.com> wrote:
> Hi,
> So I am still in process of learning hadoop.
> I tried to run wordcount.java (by writing my own mapper reducer.. creating
> jar and then running it in a pseudo distributed mode).
>
> At that time I got an error, something like
> ERROR security.UserGroupInformation: PriviledgedActionException
> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
> does not exist: hdfs://localhost:54310/user/hduser/wiki
> So I googled around.. and found that I should put the following two lines
> on my driver code:
>
> conf.addResource(new Path("/usr/local/hadoop/conf/core-site.xml"));
> conf.addResource(new Path("/usr/local/hadoop/conf/hdfs-site.xml
> file"));
>
> (path of where my core-site and hdfs-site are) and after that it ran just
> fine.
>
> Q1) How did putting those two lines solved the issue??
> Q2)
> I am now using third party libraries which are taking input from hdfs
> and writing output to hdfs...
> But in an intermediatory step, it creates a raw output.. I am again
> getting the error:
> ERROR security.UserGroupInformation: PriviledgedActionException
> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
> does not exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
> How do I resolve this?
>
> Any suggestions.
> THanks
> Jamal.
>
Re: setting hadoop for pseudo distributed mode.
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Jamal,
Please find my commands embedded below :
Q1) How did putting those two lines solved the issue??
>> By adding those two resources you make sure that your code looks for the
input path inside HDFS, which would otherwise look for it in the local FS
by default. The files core-site.xml and hdfs-site.xml tell your code where
to go for NN and DN.
Q2)
I am now using third party libraries which are taking input from hdfs and
writing output to hdfs...
But in an intermediatory step, it creates a raw output.. I am again getting
the error:
ERROR security.UserGroupInformation: PriviledgedActionException as:mhduser
cause:org.apache.hadoop.mapred.InvalidInputException: Input path does not
exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
How do I resolve this?
>>If you are able to compile your code properly then there is no problem
with the third party libraries which you are using. It looks like to me
that your code doesn't have the proper info about the intermediate path.
Please make sure you have told your code the exact location of intermediate
output.
Best Regards,
Tariq
+91-9741563634
https://mtariq.jux.com/
On Fri, Dec 28, 2012 at 1:33 AM, jamal sasha <ja...@gmail.com> wrote:
> Hi,
> So I am still in process of learning hadoop.
> I tried to run wordcount.java (by writing my own mapper reducer.. creating
> jar and then running it in a pseudo distributed mode).
>
> At that time I got an error, something like
> ERROR security.UserGroupInformation: PriviledgedActionException
> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
> does not exist: hdfs://localhost:54310/user/hduser/wiki
> So I googled around.. and found that I should put the following two lines
> on my driver code:
>
> conf.addResource(new Path("/usr/local/hadoop/conf/core-site.xml"));
> conf.addResource(new Path("/usr/local/hadoop/conf/hdfs-site.xml
> file"));
>
> (path of where my core-site and hdfs-site are) and after that it ran just
> fine.
>
> Q1) How did putting those two lines solved the issue??
> Q2)
> I am now using third party libraries which are taking input from hdfs
> and writing output to hdfs...
> But in an intermediatory step, it creates a raw output.. I am again
> getting the error:
> ERROR security.UserGroupInformation: PriviledgedActionException
> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
> does not exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
> How do I resolve this?
>
> Any suggestions.
> THanks
> Jamal.
>