You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Kevin <kl...@gmail.com> on 2008/08/04 22:53:37 UTC
Examples of using DFS without MapReduce
Hi there,
I am trying to use the DFS of hadoop in other applications. It is not
clear to me how that could be carried out easily. Could any one give a
direction to go or examples? Thank you.
-Kevin
Re: Examples of using DFS without MapReduce
Posted by Michael Bieniosek <mi...@powerset.com>.
To be honest, I have permissions turned off on my DFS (set the config variable "dfs.permissions" to be "false").
Poking around in the source code, from hadoop/core/trunk/src/core/org/apache/hadoop/security/UnixUserGroupInformation.java it looks like you can set the config variable "hadoop.job.ugi" to be "user,group" (or "user,group1,group2,..."). I haven't actually tried it though.
-Michael
On 8/4/08 3:36 PM, "Kevin" <kl...@gmail.com> wrote:
Thank you! The java code is exactly what I want.
Following your code, I encounter the user permission issue when trying
to write to a file. I wonder if the user id could be manipulated in
the protocol.
-Kevin
On Mon, Aug 4, 2008 at 2:27 PM, Michael Bieniosek <mi...@powerset.com> wrote:
> You can make shell calls:
>
> hadoop/bin/hadoop fs -fs namenode.example.com:10000 -ls /
>
> If you're in java, you can use the org.apache.hadoop.fs.FileSystem class:
>
> Configuration config = new Configuration();
> config.set("fs.default.name", "namenode.example.com:10000")
> FileSystem fs = FileSystem.get(config);
> fs.listStatus(new Path("/"))
>
> -Michael
>
> On 8/4/08 1:53 PM, "Kevin" <kl...@gmail.com> wrote:
>
> Hi there,
>
> I am trying to use the DFS of hadoop in other applications. It is not
> clear to me how that could be carried out easily. Could any one give a
> direction to go or examples? Thank you.
>
> -Kevin
>
>
Re: Examples of using DFS without MapReduce
Posted by Kevin <kl...@gmail.com>.
Thank you! The java code is exactly what I want.
Following your code, I encounter the user permission issue when trying
to write to a file. I wonder if the user id could be manipulated in
the protocol.
-Kevin
On Mon, Aug 4, 2008 at 2:27 PM, Michael Bieniosek <mi...@powerset.com> wrote:
> You can make shell calls:
>
> hadoop/bin/hadoop fs -fs namenode.example.com:10000 -ls /
>
> If you're in java, you can use the org.apache.hadoop.fs.FileSystem class:
>
> Configuration config = new Configuration();
> config.set("fs.default.name", "namenode.example.com:10000")
> FileSystem fs = FileSystem.get(config);
> fs.listStatus(new Path("/"))
>
> -Michael
>
> On 8/4/08 1:53 PM, "Kevin" <kl...@gmail.com> wrote:
>
> Hi there,
>
> I am trying to use the DFS of hadoop in other applications. It is not
> clear to me how that could be carried out easily. Could any one give a
> direction to go or examples? Thank you.
>
> -Kevin
>
>
Re: Examples of using DFS without MapReduce
Posted by Michael Bieniosek <mi...@powerset.com>.
You can make shell calls:
hadoop/bin/hadoop fs -fs namenode.example.com:10000 -ls /
If you're in java, you can use the org.apache.hadoop.fs.FileSystem class:
Configuration config = new Configuration();
config.set("fs.default.name", "namenode.example.com:10000")
FileSystem fs = FileSystem.get(config);
fs.listStatus(new Path("/"))
-Michael
On 8/4/08 1:53 PM, "Kevin" <kl...@gmail.com> wrote:
Hi there,
I am trying to use the DFS of hadoop in other applications. It is not
clear to me how that could be carried out easily. Could any one give a
direction to go or examples? Thank you.
-Kevin