You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Huy Pham <ph...@yahoo-inc.com> on 2013/08/03 03:38:04 UTC
How to construct path to a HDFS dir in local host
So I am trying to get the content of a HDFS dir. The instruction gives an example like this:
FileStatus[] status = fs.listStatus(new Path("hdfs://jp.seka.com:9000/user/jeka/in"));
But now I need to apply this example to my local host, which has HDFS dir "/data". I tried putting "/data" and "hdfs://localhost/data" but it still said "file not found" in the output.
The code I use can be found here: https://sites.google.com/site/hadoopandhive/home/how-to-read-all-files-in-a-directory-in-hdfs-using-hadoop-filesystem-api
Can someone explain how to put the right string for the path?
Thanks
Re: How to construct path to a HDFS dir in local host
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Huy,
Try adding the port as well and see if it helps :
hdfs://localhost:9000/data.(as
per your config)
And if you don't want to give the complete URI and do it by giving just the
directory Path add the following lines in your client to tell it that you
want to operate on HDFS and not on your local FS
Configuration conf = new Configuration();
conf.addResource(new Path(
"/Users/miqbal1/hadoop-eco/hadoop-1.1.2/conf/core-site.xml"));
conf.addResource(new Path(
"/Users/miqbal1/hadoop-eco/hadoop-1.1.2/conf/hdfs-site.xml"));
HTH
Warm Regards,
Tariq
cloudfront.blogspot.com
On Sat, Aug 3, 2013 at 11:08 PM, Harsh J <ha...@cloudera.com> wrote:
> Ensure you also initialize the FileSystem object (fs) against the
> right FS URI. It may be trying to resolve /data as a LocalFS path.
>
> On Sat, Aug 3, 2013 at 7:08 AM, Huy Pham <ph...@yahoo-inc.com> wrote:
> > So I am trying to get the content of a HDFS dir. The instruction gives an
> > example like this:
> > FileStatus[] status = fs.listStatus(new
> > Path("hdfs://jp.seka.com:9000/user/jeka/in"));
> > But now I need to apply this example to my local host, which has HDFS dir
> > "/data". I tried putting "/data" and "hdfs://localhost/data" but it still
> > said "file not found" in the output.
> > The code I use can be found here:
> >
> https://sites.google.com/site/hadoopandhive/home/how-to-read-all-files-in-a-directory-in-hdfs-using-hadoop-filesystem-api
> > Can someone explain how to put the right string for the path?
> > Thanks
>
>
>
> --
> Harsh J
>
Re: How to construct path to a HDFS dir in local host
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Huy,
Try adding the port as well and see if it helps :
hdfs://localhost:9000/data.(as
per your config)
And if you don't want to give the complete URI and do it by giving just the
directory Path add the following lines in your client to tell it that you
want to operate on HDFS and not on your local FS
Configuration conf = new Configuration();
conf.addResource(new Path(
"/Users/miqbal1/hadoop-eco/hadoop-1.1.2/conf/core-site.xml"));
conf.addResource(new Path(
"/Users/miqbal1/hadoop-eco/hadoop-1.1.2/conf/hdfs-site.xml"));
HTH
Warm Regards,
Tariq
cloudfront.blogspot.com
On Sat, Aug 3, 2013 at 11:08 PM, Harsh J <ha...@cloudera.com> wrote:
> Ensure you also initialize the FileSystem object (fs) against the
> right FS URI. It may be trying to resolve /data as a LocalFS path.
>
> On Sat, Aug 3, 2013 at 7:08 AM, Huy Pham <ph...@yahoo-inc.com> wrote:
> > So I am trying to get the content of a HDFS dir. The instruction gives an
> > example like this:
> > FileStatus[] status = fs.listStatus(new
> > Path("hdfs://jp.seka.com:9000/user/jeka/in"));
> > But now I need to apply this example to my local host, which has HDFS dir
> > "/data". I tried putting "/data" and "hdfs://localhost/data" but it still
> > said "file not found" in the output.
> > The code I use can be found here:
> >
> https://sites.google.com/site/hadoopandhive/home/how-to-read-all-files-in-a-directory-in-hdfs-using-hadoop-filesystem-api
> > Can someone explain how to put the right string for the path?
> > Thanks
>
>
>
> --
> Harsh J
>
Re: How to construct path to a HDFS dir in local host
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Huy,
Try adding the port as well and see if it helps :
hdfs://localhost:9000/data.(as
per your config)
And if you don't want to give the complete URI and do it by giving just the
directory Path add the following lines in your client to tell it that you
want to operate on HDFS and not on your local FS
Configuration conf = new Configuration();
conf.addResource(new Path(
"/Users/miqbal1/hadoop-eco/hadoop-1.1.2/conf/core-site.xml"));
conf.addResource(new Path(
"/Users/miqbal1/hadoop-eco/hadoop-1.1.2/conf/hdfs-site.xml"));
HTH
Warm Regards,
Tariq
cloudfront.blogspot.com
On Sat, Aug 3, 2013 at 11:08 PM, Harsh J <ha...@cloudera.com> wrote:
> Ensure you also initialize the FileSystem object (fs) against the
> right FS URI. It may be trying to resolve /data as a LocalFS path.
>
> On Sat, Aug 3, 2013 at 7:08 AM, Huy Pham <ph...@yahoo-inc.com> wrote:
> > So I am trying to get the content of a HDFS dir. The instruction gives an
> > example like this:
> > FileStatus[] status = fs.listStatus(new
> > Path("hdfs://jp.seka.com:9000/user/jeka/in"));
> > But now I need to apply this example to my local host, which has HDFS dir
> > "/data". I tried putting "/data" and "hdfs://localhost/data" but it still
> > said "file not found" in the output.
> > The code I use can be found here:
> >
> https://sites.google.com/site/hadoopandhive/home/how-to-read-all-files-in-a-directory-in-hdfs-using-hadoop-filesystem-api
> > Can someone explain how to put the right string for the path?
> > Thanks
>
>
>
> --
> Harsh J
>
Re: How to construct path to a HDFS dir in local host
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Huy,
Try adding the port as well and see if it helps :
hdfs://localhost:9000/data.(as
per your config)
And if you don't want to give the complete URI and do it by giving just the
directory Path add the following lines in your client to tell it that you
want to operate on HDFS and not on your local FS
Configuration conf = new Configuration();
conf.addResource(new Path(
"/Users/miqbal1/hadoop-eco/hadoop-1.1.2/conf/core-site.xml"));
conf.addResource(new Path(
"/Users/miqbal1/hadoop-eco/hadoop-1.1.2/conf/hdfs-site.xml"));
HTH
Warm Regards,
Tariq
cloudfront.blogspot.com
On Sat, Aug 3, 2013 at 11:08 PM, Harsh J <ha...@cloudera.com> wrote:
> Ensure you also initialize the FileSystem object (fs) against the
> right FS URI. It may be trying to resolve /data as a LocalFS path.
>
> On Sat, Aug 3, 2013 at 7:08 AM, Huy Pham <ph...@yahoo-inc.com> wrote:
> > So I am trying to get the content of a HDFS dir. The instruction gives an
> > example like this:
> > FileStatus[] status = fs.listStatus(new
> > Path("hdfs://jp.seka.com:9000/user/jeka/in"));
> > But now I need to apply this example to my local host, which has HDFS dir
> > "/data". I tried putting "/data" and "hdfs://localhost/data" but it still
> > said "file not found" in the output.
> > The code I use can be found here:
> >
> https://sites.google.com/site/hadoopandhive/home/how-to-read-all-files-in-a-directory-in-hdfs-using-hadoop-filesystem-api
> > Can someone explain how to put the right string for the path?
> > Thanks
>
>
>
> --
> Harsh J
>
Re: How to construct path to a HDFS dir in local host
Posted by Harsh J <ha...@cloudera.com>.
Ensure you also initialize the FileSystem object (fs) against the
right FS URI. It may be trying to resolve /data as a LocalFS path.
On Sat, Aug 3, 2013 at 7:08 AM, Huy Pham <ph...@yahoo-inc.com> wrote:
> So I am trying to get the content of a HDFS dir. The instruction gives an
> example like this:
> FileStatus[] status = fs.listStatus(new
> Path("hdfs://jp.seka.com:9000/user/jeka/in"));
> But now I need to apply this example to my local host, which has HDFS dir
> "/data". I tried putting "/data" and "hdfs://localhost/data" but it still
> said "file not found" in the output.
> The code I use can be found here:
> https://sites.google.com/site/hadoopandhive/home/how-to-read-all-files-in-a-directory-in-hdfs-using-hadoop-filesystem-api
> Can someone explain how to put the right string for the path?
> Thanks
--
Harsh J
Re: How to construct path to a HDFS dir in local host
Posted by Harsh J <ha...@cloudera.com>.
Ensure you also initialize the FileSystem object (fs) against the
right FS URI. It may be trying to resolve /data as a LocalFS path.
On Sat, Aug 3, 2013 at 7:08 AM, Huy Pham <ph...@yahoo-inc.com> wrote:
> So I am trying to get the content of a HDFS dir. The instruction gives an
> example like this:
> FileStatus[] status = fs.listStatus(new
> Path("hdfs://jp.seka.com:9000/user/jeka/in"));
> But now I need to apply this example to my local host, which has HDFS dir
> "/data". I tried putting "/data" and "hdfs://localhost/data" but it still
> said "file not found" in the output.
> The code I use can be found here:
> https://sites.google.com/site/hadoopandhive/home/how-to-read-all-files-in-a-directory-in-hdfs-using-hadoop-filesystem-api
> Can someone explain how to put the right string for the path?
> Thanks
--
Harsh J
Re: How to construct path to a HDFS dir in local host
Posted by Shahab Yunus <sh...@gmail.com>.
I think you need to have a user folder (with whatever user you are running
hadoop) and put your files there. In fact that folder should be already
created there. The hdfsr data dir property is for setting the path to the
data files.
Regards,
Shahab
On Fri, Aug 2, 2013 at 9:38 PM, Huy Pham <ph...@yahoo-inc.com> wrote:
> So I am trying to get the content of a HDFS dir. The instruction gives
> an example like this:
> FileStatus[] status = fs.listStatus(new Path("hdfs://
> jp.seka.com:9000/user/jeka/in"));
> But now I need to apply this example to my local host, which has HDFS dir
> "/data". I tried putting "/data" and "hdfs://localhost/data" but it still
> said "file not found" in the output.
> The code I use can be found here:
> https://sites.google.com/site/hadoopandhive/home/how-to-read-all-files-in-a-directory-in-hdfs-using-hadoop-filesystem-api
> Can someone explain how to put the right string for the path?
> Thanks
>
Re: How to construct path to a HDFS dir in local host
Posted by Harsh J <ha...@cloudera.com>.
Ensure you also initialize the FileSystem object (fs) against the
right FS URI. It may be trying to resolve /data as a LocalFS path.
On Sat, Aug 3, 2013 at 7:08 AM, Huy Pham <ph...@yahoo-inc.com> wrote:
> So I am trying to get the content of a HDFS dir. The instruction gives an
> example like this:
> FileStatus[] status = fs.listStatus(new
> Path("hdfs://jp.seka.com:9000/user/jeka/in"));
> But now I need to apply this example to my local host, which has HDFS dir
> "/data". I tried putting "/data" and "hdfs://localhost/data" but it still
> said "file not found" in the output.
> The code I use can be found here:
> https://sites.google.com/site/hadoopandhive/home/how-to-read-all-files-in-a-directory-in-hdfs-using-hadoop-filesystem-api
> Can someone explain how to put the right string for the path?
> Thanks
--
Harsh J
Re: How to construct path to a HDFS dir in local host
Posted by Harsh J <ha...@cloudera.com>.
Ensure you also initialize the FileSystem object (fs) against the
right FS URI. It may be trying to resolve /data as a LocalFS path.
On Sat, Aug 3, 2013 at 7:08 AM, Huy Pham <ph...@yahoo-inc.com> wrote:
> So I am trying to get the content of a HDFS dir. The instruction gives an
> example like this:
> FileStatus[] status = fs.listStatus(new
> Path("hdfs://jp.seka.com:9000/user/jeka/in"));
> But now I need to apply this example to my local host, which has HDFS dir
> "/data". I tried putting "/data" and "hdfs://localhost/data" but it still
> said "file not found" in the output.
> The code I use can be found here:
> https://sites.google.com/site/hadoopandhive/home/how-to-read-all-files-in-a-directory-in-hdfs-using-hadoop-filesystem-api
> Can someone explain how to put the right string for the path?
> Thanks
--
Harsh J
Re: How to construct path to a HDFS dir in local host
Posted by Shahab Yunus <sh...@gmail.com>.
I think you need to have a user folder (with whatever user you are running
hadoop) and put your files there. In fact that folder should be already
created there. The hdfsr data dir property is for setting the path to the
data files.
Regards,
Shahab
On Fri, Aug 2, 2013 at 9:38 PM, Huy Pham <ph...@yahoo-inc.com> wrote:
> So I am trying to get the content of a HDFS dir. The instruction gives
> an example like this:
> FileStatus[] status = fs.listStatus(new Path("hdfs://
> jp.seka.com:9000/user/jeka/in"));
> But now I need to apply this example to my local host, which has HDFS dir
> "/data". I tried putting "/data" and "hdfs://localhost/data" but it still
> said "file not found" in the output.
> The code I use can be found here:
> https://sites.google.com/site/hadoopandhive/home/how-to-read-all-files-in-a-directory-in-hdfs-using-hadoop-filesystem-api
> Can someone explain how to put the right string for the path?
> Thanks
>
Re: How to construct path to a HDFS dir in local host
Posted by Shahab Yunus <sh...@gmail.com>.
I think you need to have a user folder (with whatever user you are running
hadoop) and put your files there. In fact that folder should be already
created there. The hdfsr data dir property is for setting the path to the
data files.
Regards,
Shahab
On Fri, Aug 2, 2013 at 9:38 PM, Huy Pham <ph...@yahoo-inc.com> wrote:
> So I am trying to get the content of a HDFS dir. The instruction gives
> an example like this:
> FileStatus[] status = fs.listStatus(new Path("hdfs://
> jp.seka.com:9000/user/jeka/in"));
> But now I need to apply this example to my local host, which has HDFS dir
> "/data". I tried putting "/data" and "hdfs://localhost/data" but it still
> said "file not found" in the output.
> The code I use can be found here:
> https://sites.google.com/site/hadoopandhive/home/how-to-read-all-files-in-a-directory-in-hdfs-using-hadoop-filesystem-api
> Can someone explain how to put the right string for the path?
> Thanks
>
Re: How to construct path to a HDFS dir in local host
Posted by Shahab Yunus <sh...@gmail.com>.
I think you need to have a user folder (with whatever user you are running
hadoop) and put your files there. In fact that folder should be already
created there. The hdfsr data dir property is for setting the path to the
data files.
Regards,
Shahab
On Fri, Aug 2, 2013 at 9:38 PM, Huy Pham <ph...@yahoo-inc.com> wrote:
> So I am trying to get the content of a HDFS dir. The instruction gives
> an example like this:
> FileStatus[] status = fs.listStatus(new Path("hdfs://
> jp.seka.com:9000/user/jeka/in"));
> But now I need to apply this example to my local host, which has HDFS dir
> "/data". I tried putting "/data" and "hdfs://localhost/data" but it still
> said "file not found" in the output.
> The code I use can be found here:
> https://sites.google.com/site/hadoopandhive/home/how-to-read-all-files-in-a-directory-in-hdfs-using-hadoop-filesystem-api
> Can someone explain how to put the right string for the path?
> Thanks
>