You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Chris Nauroth <cn...@apache.org> on 2022/10/04 03:35:40 UTC
Re: hdfs commands available in Spark?
I expect you'd be able to fork a separate process and run any hadoop/hdfs
commands that you'd like.
If you're coding in Java, then I expect you can code to APIs to accomplish
this goal, basically doing the equivalent of what the HDFS CLI does now:
https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tools/GetGroupsBase.java0
Chris Nauroth
On Wed, Sep 7, 2022 at 6:15 AM <ph...@free.fr> wrote:
> Hello,
> does anyone know if "hdfs" commands can be run with Spark?
> For instance, I would like to run "hdfs groups <username>" in Spark. Is it
> possible?
> Many thanks
> Philippe
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org
>
>
Re: hdfs commands available in Spark?
Posted by Chris Nauroth <cn...@apache.org>.
I should further clarify. First, here is a correction to the link in my
last email:
https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tools/GetGroupsBase.java
Additionally, there is a nuance in that "hdfs groups" is really contacting
the NameNode to ask for its opinion of which groups the user belongs to. If
you want to match that behavior completely, then it's relevant to look at
the GetGroups class in HDFS:
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetGroups.java
Chris Nauroth
On Mon, Oct 3, 2022 at 8:35 PM Chris Nauroth <cn...@apache.org> wrote:
> I expect you'd be able to fork a separate process and run any hadoop/hdfs
> commands that you'd like.
>
> If you're coding in Java, then I expect you can code to APIs to accomplish
> this goal, basically doing the equivalent of what the HDFS CLI does now:
>
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tools/GetGroupsBase.java0
>
> Chris Nauroth
>
>
> On Wed, Sep 7, 2022 at 6:15 AM <ph...@free.fr> wrote:
>
>> Hello,
>> does anyone know if "hdfs" commands can be run with Spark?
>> For instance, I would like to run "hdfs groups <username>" in Spark. Is
>> it possible?
>> Many thanks
>> Philippe
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: user-help@hadoop.apache.org
>>
>>