You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Alexander Striffeler <a....@students.unibe.ch> on 2015/07/22 09:41:29 UTC
NNBench on external HDFS
Hi all
I'm pretty new to the Hadoop environment and I'm about performing some
micro benchmarks. In particular, I'm struggling with executing NNBench
against an external File System:
hadoop jar
/usr/hdp/2.2.6.0-2800/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar
nnbench -Dfs.defaultFS='hfds://<external.file.system>' -operation
create_write -bytesToWrite 10 -maps 2 -reduces 1 -numberOfFiles 100
-baseDir hdfs://dapsilon.daplab.ch/user/username/nnbench-`hostname -s`
yields in
java.lang.IllegalArgumentException: Wrong FS:
hdfs://<external.file.system>/user/username/nnbench-hostname/data,
expected: hdfs://<native fs>
If I neglect the ext FS prefix in the baseDir, NNBench simply ignores
the -D option and writes the files to the native DFS. Does anyone have
an idea how to solve this and nnbench an external DFS?
Thanks a lot, any hints are very appreciated!
Regards,
Alex
RE: NNBench on external HDFS
Posted by Brahma Reddy Battula <br...@huawei.com>.
HI Alex
HDFS-7651 is raised for same..Please have a look at once..
Thanks & Regards
Brahma Reddy Battula
_______________________________________
From: Alexander Striffeler [a.striffeler@students.unibe.ch]
Sent: Thursday, July 23, 2015 1:00 PM
To: user@hadoop.apache.org
Subject: Re: NNBench on external HDFS
Hi Chris,
Wow, thanks a lot for your swift and extensive response! I'll try your
suggestion with the local copy and in a second step I'll open a jira
request...
Have a good day,
--Alex
On 22.07.2015 20:14, Chris Nauroth wrote:
> Hi Alexander,
>
> Your NNBench usage looks basically correct, but NNBench is not a standard
> Hadoop tool. It does not implement the org.apache.hadoop.util.Tool
> interface, it does not execute through org.apache.hadoop.util.ToolRunner,
> and therefore it does not support the command line arguments that a lot of
> other Hadoop tools like the FsShell support. Specifically, it does not
> support passing -D arguments to override fs.defaultFS or any other
> configuration properties.
>
> An alternative way to handle this would be to get a local copy of the
> configuration directory from the remote cluster that you want to test. I
> expect those configuration files would have fs.defaultFS set to the URL of
> that remote cluster in core-site.xml. Before launching NNBench, run
> "export HADOOP_CONF_DIR=<path to local copy of configuration files>".
> After exporting that environment variable, you can run "hadoop classpath"
> to print the classpath that will be used by all "hadoop" commands and
> confirm that the correct configuration directory for the target cluster is
> on the classpath. Then, you can run NNBench again, but drop the -D
> argument, since it's going to get ignored anyway.
>
> I don't see any reason why NNBench shouldn't implement the standard Tool
> interface and thus support the command line arguments that you were
> expecting. If you'd like to request that as an enhancement, please go
> ahead and file an HDFS jira to request it. Feel free to post a patch too
> if you're inclined. Otherwise, someone else in the community can pick it
> up.
>
> I hope this helps.
>
> --Chris Nauroth
>
>
>
>
> On 7/22/15, 12:41 AM, "Alexander Striffeler"
> <a....@students.unibe.ch> wrote:
>
>> Hi all
>>
>> I'm pretty new to the Hadoop environment and I'm about performing some
>> micro benchmarks. In particular, I'm struggling with executing NNBench
>> against an external File System:
>>
>> hadoop jar
>> /usr/hdp/2.2.6.0-2800/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-t
>> ests.jar
>> nnbench -Dfs.defaultFS='hfds://<external.file.system>' -operation
>> create_write -bytesToWrite 10 -maps 2 -reduces 1 -numberOfFiles 100
>> -baseDir hdfs://dapsilon.daplab.ch/user/username/nnbench-`hostname -s`
>>
>> yields in
>> java.lang.IllegalArgumentException: Wrong FS:
>> hdfs://<external.file.system>/user/username/nnbench-hostname/data,
>> expected: hdfs://<native fs>
>>
>> If I neglect the ext FS prefix in the baseDir, NNBench simply ignores
>> the -D option and writes the files to the native DFS. Does anyone have
>> an idea how to solve this and nnbench an external DFS?
>>
>> Thanks a lot, any hints are very appreciated!
>> Regards,
>> Alex
RE: NNBench on external HDFS
Posted by Brahma Reddy Battula <br...@huawei.com>.
HI Alex
HDFS-7651 is raised for same..Please have a look at once..
Thanks & Regards
Brahma Reddy Battula
_______________________________________
From: Alexander Striffeler [a.striffeler@students.unibe.ch]
Sent: Thursday, July 23, 2015 1:00 PM
To: user@hadoop.apache.org
Subject: Re: NNBench on external HDFS
Hi Chris,
Wow, thanks a lot for your swift and extensive response! I'll try your
suggestion with the local copy and in a second step I'll open a jira
request...
Have a good day,
--Alex
On 22.07.2015 20:14, Chris Nauroth wrote:
> Hi Alexander,
>
> Your NNBench usage looks basically correct, but NNBench is not a standard
> Hadoop tool. It does not implement the org.apache.hadoop.util.Tool
> interface, it does not execute through org.apache.hadoop.util.ToolRunner,
> and therefore it does not support the command line arguments that a lot of
> other Hadoop tools like the FsShell support. Specifically, it does not
> support passing -D arguments to override fs.defaultFS or any other
> configuration properties.
>
> An alternative way to handle this would be to get a local copy of the
> configuration directory from the remote cluster that you want to test. I
> expect those configuration files would have fs.defaultFS set to the URL of
> that remote cluster in core-site.xml. Before launching NNBench, run
> "export HADOOP_CONF_DIR=<path to local copy of configuration files>".
> After exporting that environment variable, you can run "hadoop classpath"
> to print the classpath that will be used by all "hadoop" commands and
> confirm that the correct configuration directory for the target cluster is
> on the classpath. Then, you can run NNBench again, but drop the -D
> argument, since it's going to get ignored anyway.
>
> I don't see any reason why NNBench shouldn't implement the standard Tool
> interface and thus support the command line arguments that you were
> expecting. If you'd like to request that as an enhancement, please go
> ahead and file an HDFS jira to request it. Feel free to post a patch too
> if you're inclined. Otherwise, someone else in the community can pick it
> up.
>
> I hope this helps.
>
> --Chris Nauroth
>
>
>
>
> On 7/22/15, 12:41 AM, "Alexander Striffeler"
> <a....@students.unibe.ch> wrote:
>
>> Hi all
>>
>> I'm pretty new to the Hadoop environment and I'm about performing some
>> micro benchmarks. In particular, I'm struggling with executing NNBench
>> against an external File System:
>>
>> hadoop jar
>> /usr/hdp/2.2.6.0-2800/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-t
>> ests.jar
>> nnbench -Dfs.defaultFS='hfds://<external.file.system>' -operation
>> create_write -bytesToWrite 10 -maps 2 -reduces 1 -numberOfFiles 100
>> -baseDir hdfs://dapsilon.daplab.ch/user/username/nnbench-`hostname -s`
>>
>> yields in
>> java.lang.IllegalArgumentException: Wrong FS:
>> hdfs://<external.file.system>/user/username/nnbench-hostname/data,
>> expected: hdfs://<native fs>
>>
>> If I neglect the ext FS prefix in the baseDir, NNBench simply ignores
>> the -D option and writes the files to the native DFS. Does anyone have
>> an idea how to solve this and nnbench an external DFS?
>>
>> Thanks a lot, any hints are very appreciated!
>> Regards,
>> Alex
RE: NNBench on external HDFS
Posted by Brahma Reddy Battula <br...@huawei.com>.
HI Alex
HDFS-7651 is raised for same..Please have a look at once..
Thanks & Regards
Brahma Reddy Battula
_______________________________________
From: Alexander Striffeler [a.striffeler@students.unibe.ch]
Sent: Thursday, July 23, 2015 1:00 PM
To: user@hadoop.apache.org
Subject: Re: NNBench on external HDFS
Hi Chris,
Wow, thanks a lot for your swift and extensive response! I'll try your
suggestion with the local copy and in a second step I'll open a jira
request...
Have a good day,
--Alex
On 22.07.2015 20:14, Chris Nauroth wrote:
> Hi Alexander,
>
> Your NNBench usage looks basically correct, but NNBench is not a standard
> Hadoop tool. It does not implement the org.apache.hadoop.util.Tool
> interface, it does not execute through org.apache.hadoop.util.ToolRunner,
> and therefore it does not support the command line arguments that a lot of
> other Hadoop tools like the FsShell support. Specifically, it does not
> support passing -D arguments to override fs.defaultFS or any other
> configuration properties.
>
> An alternative way to handle this would be to get a local copy of the
> configuration directory from the remote cluster that you want to test. I
> expect those configuration files would have fs.defaultFS set to the URL of
> that remote cluster in core-site.xml. Before launching NNBench, run
> "export HADOOP_CONF_DIR=<path to local copy of configuration files>".
> After exporting that environment variable, you can run "hadoop classpath"
> to print the classpath that will be used by all "hadoop" commands and
> confirm that the correct configuration directory for the target cluster is
> on the classpath. Then, you can run NNBench again, but drop the -D
> argument, since it's going to get ignored anyway.
>
> I don't see any reason why NNBench shouldn't implement the standard Tool
> interface and thus support the command line arguments that you were
> expecting. If you'd like to request that as an enhancement, please go
> ahead and file an HDFS jira to request it. Feel free to post a patch too
> if you're inclined. Otherwise, someone else in the community can pick it
> up.
>
> I hope this helps.
>
> --Chris Nauroth
>
>
>
>
> On 7/22/15, 12:41 AM, "Alexander Striffeler"
> <a....@students.unibe.ch> wrote:
>
>> Hi all
>>
>> I'm pretty new to the Hadoop environment and I'm about performing some
>> micro benchmarks. In particular, I'm struggling with executing NNBench
>> against an external File System:
>>
>> hadoop jar
>> /usr/hdp/2.2.6.0-2800/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-t
>> ests.jar
>> nnbench -Dfs.defaultFS='hfds://<external.file.system>' -operation
>> create_write -bytesToWrite 10 -maps 2 -reduces 1 -numberOfFiles 100
>> -baseDir hdfs://dapsilon.daplab.ch/user/username/nnbench-`hostname -s`
>>
>> yields in
>> java.lang.IllegalArgumentException: Wrong FS:
>> hdfs://<external.file.system>/user/username/nnbench-hostname/data,
>> expected: hdfs://<native fs>
>>
>> If I neglect the ext FS prefix in the baseDir, NNBench simply ignores
>> the -D option and writes the files to the native DFS. Does anyone have
>> an idea how to solve this and nnbench an external DFS?
>>
>> Thanks a lot, any hints are very appreciated!
>> Regards,
>> Alex
RE: NNBench on external HDFS
Posted by Brahma Reddy Battula <br...@huawei.com>.
HI Alex
HDFS-7651 is raised for same..Please have a look at once..
Thanks & Regards
Brahma Reddy Battula
_______________________________________
From: Alexander Striffeler [a.striffeler@students.unibe.ch]
Sent: Thursday, July 23, 2015 1:00 PM
To: user@hadoop.apache.org
Subject: Re: NNBench on external HDFS
Hi Chris,
Wow, thanks a lot for your swift and extensive response! I'll try your
suggestion with the local copy and in a second step I'll open a jira
request...
Have a good day,
--Alex
On 22.07.2015 20:14, Chris Nauroth wrote:
> Hi Alexander,
>
> Your NNBench usage looks basically correct, but NNBench is not a standard
> Hadoop tool. It does not implement the org.apache.hadoop.util.Tool
> interface, it does not execute through org.apache.hadoop.util.ToolRunner,
> and therefore it does not support the command line arguments that a lot of
> other Hadoop tools like the FsShell support. Specifically, it does not
> support passing -D arguments to override fs.defaultFS or any other
> configuration properties.
>
> An alternative way to handle this would be to get a local copy of the
> configuration directory from the remote cluster that you want to test. I
> expect those configuration files would have fs.defaultFS set to the URL of
> that remote cluster in core-site.xml. Before launching NNBench, run
> "export HADOOP_CONF_DIR=<path to local copy of configuration files>".
> After exporting that environment variable, you can run "hadoop classpath"
> to print the classpath that will be used by all "hadoop" commands and
> confirm that the correct configuration directory for the target cluster is
> on the classpath. Then, you can run NNBench again, but drop the -D
> argument, since it's going to get ignored anyway.
>
> I don't see any reason why NNBench shouldn't implement the standard Tool
> interface and thus support the command line arguments that you were
> expecting. If you'd like to request that as an enhancement, please go
> ahead and file an HDFS jira to request it. Feel free to post a patch too
> if you're inclined. Otherwise, someone else in the community can pick it
> up.
>
> I hope this helps.
>
> --Chris Nauroth
>
>
>
>
> On 7/22/15, 12:41 AM, "Alexander Striffeler"
> <a....@students.unibe.ch> wrote:
>
>> Hi all
>>
>> I'm pretty new to the Hadoop environment and I'm about performing some
>> micro benchmarks. In particular, I'm struggling with executing NNBench
>> against an external File System:
>>
>> hadoop jar
>> /usr/hdp/2.2.6.0-2800/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-t
>> ests.jar
>> nnbench -Dfs.defaultFS='hfds://<external.file.system>' -operation
>> create_write -bytesToWrite 10 -maps 2 -reduces 1 -numberOfFiles 100
>> -baseDir hdfs://dapsilon.daplab.ch/user/username/nnbench-`hostname -s`
>>
>> yields in
>> java.lang.IllegalArgumentException: Wrong FS:
>> hdfs://<external.file.system>/user/username/nnbench-hostname/data,
>> expected: hdfs://<native fs>
>>
>> If I neglect the ext FS prefix in the baseDir, NNBench simply ignores
>> the -D option and writes the files to the native DFS. Does anyone have
>> an idea how to solve this and nnbench an external DFS?
>>
>> Thanks a lot, any hints are very appreciated!
>> Regards,
>> Alex
Re: NNBench on external HDFS
Posted by Alexander Striffeler <a....@students.unibe.ch>.
Hi Chris,
Wow, thanks a lot for your swift and extensive response! I'll try your
suggestion with the local copy and in a second step I'll open a jira
request...
Have a good day,
--Alex
On 22.07.2015 20:14, Chris Nauroth wrote:
> Hi Alexander,
>
> Your NNBench usage looks basically correct, but NNBench is not a standard
> Hadoop tool. It does not implement the org.apache.hadoop.util.Tool
> interface, it does not execute through org.apache.hadoop.util.ToolRunner,
> and therefore it does not support the command line arguments that a lot of
> other Hadoop tools like the FsShell support. Specifically, it does not
> support passing -D arguments to override fs.defaultFS or any other
> configuration properties.
>
> An alternative way to handle this would be to get a local copy of the
> configuration directory from the remote cluster that you want to test. I
> expect those configuration files would have fs.defaultFS set to the URL of
> that remote cluster in core-site.xml. Before launching NNBench, run
> "export HADOOP_CONF_DIR=<path to local copy of configuration files>".
> After exporting that environment variable, you can run "hadoop classpath"
> to print the classpath that will be used by all "hadoop" commands and
> confirm that the correct configuration directory for the target cluster is
> on the classpath. Then, you can run NNBench again, but drop the -D
> argument, since it's going to get ignored anyway.
>
> I don't see any reason why NNBench shouldn't implement the standard Tool
> interface and thus support the command line arguments that you were
> expecting. If you'd like to request that as an enhancement, please go
> ahead and file an HDFS jira to request it. Feel free to post a patch too
> if you're inclined. Otherwise, someone else in the community can pick it
> up.
>
> I hope this helps.
>
> --Chris Nauroth
>
>
>
>
> On 7/22/15, 12:41 AM, "Alexander Striffeler"
> <a....@students.unibe.ch> wrote:
>
>> Hi all
>>
>> I'm pretty new to the Hadoop environment and I'm about performing some
>> micro benchmarks. In particular, I'm struggling with executing NNBench
>> against an external File System:
>>
>> hadoop jar
>> /usr/hdp/2.2.6.0-2800/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-t
>> ests.jar
>> nnbench -Dfs.defaultFS='hfds://<external.file.system>' -operation
>> create_write -bytesToWrite 10 -maps 2 -reduces 1 -numberOfFiles 100
>> -baseDir hdfs://dapsilon.daplab.ch/user/username/nnbench-`hostname -s`
>>
>> yields in
>> java.lang.IllegalArgumentException: Wrong FS:
>> hdfs://<external.file.system>/user/username/nnbench-hostname/data,
>> expected: hdfs://<native fs>
>>
>> If I neglect the ext FS prefix in the baseDir, NNBench simply ignores
>> the -D option and writes the files to the native DFS. Does anyone have
>> an idea how to solve this and nnbench an external DFS?
>>
>> Thanks a lot, any hints are very appreciated!
>> Regards,
>> Alex
Re: NNBench on external HDFS
Posted by Alexander Striffeler <a....@students.unibe.ch>.
Hi Chris,
Wow, thanks a lot for your swift and extensive response! I'll try your
suggestion with the local copy and in a second step I'll open a jira
request...
Have a good day,
--Alex
On 22.07.2015 20:14, Chris Nauroth wrote:
> Hi Alexander,
>
> Your NNBench usage looks basically correct, but NNBench is not a standard
> Hadoop tool. It does not implement the org.apache.hadoop.util.Tool
> interface, it does not execute through org.apache.hadoop.util.ToolRunner,
> and therefore it does not support the command line arguments that a lot of
> other Hadoop tools like the FsShell support. Specifically, it does not
> support passing -D arguments to override fs.defaultFS or any other
> configuration properties.
>
> An alternative way to handle this would be to get a local copy of the
> configuration directory from the remote cluster that you want to test. I
> expect those configuration files would have fs.defaultFS set to the URL of
> that remote cluster in core-site.xml. Before launching NNBench, run
> "export HADOOP_CONF_DIR=<path to local copy of configuration files>".
> After exporting that environment variable, you can run "hadoop classpath"
> to print the classpath that will be used by all "hadoop" commands and
> confirm that the correct configuration directory for the target cluster is
> on the classpath. Then, you can run NNBench again, but drop the -D
> argument, since it's going to get ignored anyway.
>
> I don't see any reason why NNBench shouldn't implement the standard Tool
> interface and thus support the command line arguments that you were
> expecting. If you'd like to request that as an enhancement, please go
> ahead and file an HDFS jira to request it. Feel free to post a patch too
> if you're inclined. Otherwise, someone else in the community can pick it
> up.
>
> I hope this helps.
>
> --Chris Nauroth
>
>
>
>
> On 7/22/15, 12:41 AM, "Alexander Striffeler"
> <a....@students.unibe.ch> wrote:
>
>> Hi all
>>
>> I'm pretty new to the Hadoop environment and I'm about performing some
>> micro benchmarks. In particular, I'm struggling with executing NNBench
>> against an external File System:
>>
>> hadoop jar
>> /usr/hdp/2.2.6.0-2800/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-t
>> ests.jar
>> nnbench -Dfs.defaultFS='hfds://<external.file.system>' -operation
>> create_write -bytesToWrite 10 -maps 2 -reduces 1 -numberOfFiles 100
>> -baseDir hdfs://dapsilon.daplab.ch/user/username/nnbench-`hostname -s`
>>
>> yields in
>> java.lang.IllegalArgumentException: Wrong FS:
>> hdfs://<external.file.system>/user/username/nnbench-hostname/data,
>> expected: hdfs://<native fs>
>>
>> If I neglect the ext FS prefix in the baseDir, NNBench simply ignores
>> the -D option and writes the files to the native DFS. Does anyone have
>> an idea how to solve this and nnbench an external DFS?
>>
>> Thanks a lot, any hints are very appreciated!
>> Regards,
>> Alex
Re: NNBench on external HDFS
Posted by Alexander Striffeler <a....@students.unibe.ch>.
Hi Chris,
Wow, thanks a lot for your swift and extensive response! I'll try your
suggestion with the local copy and in a second step I'll open a jira
request...
Have a good day,
--Alex
On 22.07.2015 20:14, Chris Nauroth wrote:
> Hi Alexander,
>
> Your NNBench usage looks basically correct, but NNBench is not a standard
> Hadoop tool. It does not implement the org.apache.hadoop.util.Tool
> interface, it does not execute through org.apache.hadoop.util.ToolRunner,
> and therefore it does not support the command line arguments that a lot of
> other Hadoop tools like the FsShell support. Specifically, it does not
> support passing -D arguments to override fs.defaultFS or any other
> configuration properties.
>
> An alternative way to handle this would be to get a local copy of the
> configuration directory from the remote cluster that you want to test. I
> expect those configuration files would have fs.defaultFS set to the URL of
> that remote cluster in core-site.xml. Before launching NNBench, run
> "export HADOOP_CONF_DIR=<path to local copy of configuration files>".
> After exporting that environment variable, you can run "hadoop classpath"
> to print the classpath that will be used by all "hadoop" commands and
> confirm that the correct configuration directory for the target cluster is
> on the classpath. Then, you can run NNBench again, but drop the -D
> argument, since it's going to get ignored anyway.
>
> I don't see any reason why NNBench shouldn't implement the standard Tool
> interface and thus support the command line arguments that you were
> expecting. If you'd like to request that as an enhancement, please go
> ahead and file an HDFS jira to request it. Feel free to post a patch too
> if you're inclined. Otherwise, someone else in the community can pick it
> up.
>
> I hope this helps.
>
> --Chris Nauroth
>
>
>
>
> On 7/22/15, 12:41 AM, "Alexander Striffeler"
> <a....@students.unibe.ch> wrote:
>
>> Hi all
>>
>> I'm pretty new to the Hadoop environment and I'm about performing some
>> micro benchmarks. In particular, I'm struggling with executing NNBench
>> against an external File System:
>>
>> hadoop jar
>> /usr/hdp/2.2.6.0-2800/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-t
>> ests.jar
>> nnbench -Dfs.defaultFS='hfds://<external.file.system>' -operation
>> create_write -bytesToWrite 10 -maps 2 -reduces 1 -numberOfFiles 100
>> -baseDir hdfs://dapsilon.daplab.ch/user/username/nnbench-`hostname -s`
>>
>> yields in
>> java.lang.IllegalArgumentException: Wrong FS:
>> hdfs://<external.file.system>/user/username/nnbench-hostname/data,
>> expected: hdfs://<native fs>
>>
>> If I neglect the ext FS prefix in the baseDir, NNBench simply ignores
>> the -D option and writes the files to the native DFS. Does anyone have
>> an idea how to solve this and nnbench an external DFS?
>>
>> Thanks a lot, any hints are very appreciated!
>> Regards,
>> Alex
Re: NNBench on external HDFS
Posted by Alexander Striffeler <a....@students.unibe.ch>.
Hi Chris,
Wow, thanks a lot for your swift and extensive response! I'll try your
suggestion with the local copy and in a second step I'll open a jira
request...
Have a good day,
--Alex
On 22.07.2015 20:14, Chris Nauroth wrote:
> Hi Alexander,
>
> Your NNBench usage looks basically correct, but NNBench is not a standard
> Hadoop tool. It does not implement the org.apache.hadoop.util.Tool
> interface, it does not execute through org.apache.hadoop.util.ToolRunner,
> and therefore it does not support the command line arguments that a lot of
> other Hadoop tools like the FsShell support. Specifically, it does not
> support passing -D arguments to override fs.defaultFS or any other
> configuration properties.
>
> An alternative way to handle this would be to get a local copy of the
> configuration directory from the remote cluster that you want to test. I
> expect those configuration files would have fs.defaultFS set to the URL of
> that remote cluster in core-site.xml. Before launching NNBench, run
> "export HADOOP_CONF_DIR=<path to local copy of configuration files>".
> After exporting that environment variable, you can run "hadoop classpath"
> to print the classpath that will be used by all "hadoop" commands and
> confirm that the correct configuration directory for the target cluster is
> on the classpath. Then, you can run NNBench again, but drop the -D
> argument, since it's going to get ignored anyway.
>
> I don't see any reason why NNBench shouldn't implement the standard Tool
> interface and thus support the command line arguments that you were
> expecting. If you'd like to request that as an enhancement, please go
> ahead and file an HDFS jira to request it. Feel free to post a patch too
> if you're inclined. Otherwise, someone else in the community can pick it
> up.
>
> I hope this helps.
>
> --Chris Nauroth
>
>
>
>
> On 7/22/15, 12:41 AM, "Alexander Striffeler"
> <a....@students.unibe.ch> wrote:
>
>> Hi all
>>
>> I'm pretty new to the Hadoop environment and I'm about performing some
>> micro benchmarks. In particular, I'm struggling with executing NNBench
>> against an external File System:
>>
>> hadoop jar
>> /usr/hdp/2.2.6.0-2800/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-t
>> ests.jar
>> nnbench -Dfs.defaultFS='hfds://<external.file.system>' -operation
>> create_write -bytesToWrite 10 -maps 2 -reduces 1 -numberOfFiles 100
>> -baseDir hdfs://dapsilon.daplab.ch/user/username/nnbench-`hostname -s`
>>
>> yields in
>> java.lang.IllegalArgumentException: Wrong FS:
>> hdfs://<external.file.system>/user/username/nnbench-hostname/data,
>> expected: hdfs://<native fs>
>>
>> If I neglect the ext FS prefix in the baseDir, NNBench simply ignores
>> the -D option and writes the files to the native DFS. Does anyone have
>> an idea how to solve this and nnbench an external DFS?
>>
>> Thanks a lot, any hints are very appreciated!
>> Regards,
>> Alex
Re: NNBench on external HDFS
Posted by Chris Nauroth <cn...@hortonworks.com>.
Hi Alexander,
Your NNBench usage looks basically correct, but NNBench is not a standard
Hadoop tool. It does not implement the org.apache.hadoop.util.Tool
interface, it does not execute through org.apache.hadoop.util.ToolRunner,
and therefore it does not support the command line arguments that a lot of
other Hadoop tools like the FsShell support. Specifically, it does not
support passing -D arguments to override fs.defaultFS or any other
configuration properties.
An alternative way to handle this would be to get a local copy of the
configuration directory from the remote cluster that you want to test. I
expect those configuration files would have fs.defaultFS set to the URL of
that remote cluster in core-site.xml. Before launching NNBench, run
"export HADOOP_CONF_DIR=<path to local copy of configuration files>".
After exporting that environment variable, you can run "hadoop classpath"
to print the classpath that will be used by all "hadoop" commands and
confirm that the correct configuration directory for the target cluster is
on the classpath. Then, you can run NNBench again, but drop the -D
argument, since it's going to get ignored anyway.
I don't see any reason why NNBench shouldn't implement the standard Tool
interface and thus support the command line arguments that you were
expecting. If you'd like to request that as an enhancement, please go
ahead and file an HDFS jira to request it. Feel free to post a patch too
if you're inclined. Otherwise, someone else in the community can pick it
up.
I hope this helps.
--Chris Nauroth
On 7/22/15, 12:41 AM, "Alexander Striffeler"
<a....@students.unibe.ch> wrote:
>Hi all
>
>I'm pretty new to the Hadoop environment and I'm about performing some
>micro benchmarks. In particular, I'm struggling with executing NNBench
>against an external File System:
>
>hadoop jar
>/usr/hdp/2.2.6.0-2800/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-t
>ests.jar
>nnbench -Dfs.defaultFS='hfds://<external.file.system>' -operation
>create_write -bytesToWrite 10 -maps 2 -reduces 1 -numberOfFiles 100
>-baseDir hdfs://dapsilon.daplab.ch/user/username/nnbench-`hostname -s`
>
>yields in
>java.lang.IllegalArgumentException: Wrong FS:
>hdfs://<external.file.system>/user/username/nnbench-hostname/data,
>expected: hdfs://<native fs>
>
>If I neglect the ext FS prefix in the baseDir, NNBench simply ignores
>the -D option and writes the files to the native DFS. Does anyone have
>an idea how to solve this and nnbench an external DFS?
>
>Thanks a lot, any hints are very appreciated!
>Regards,
>Alex
Re: NNBench on external HDFS
Posted by Chris Nauroth <cn...@hortonworks.com>.
Hi Alexander,
Your NNBench usage looks basically correct, but NNBench is not a standard
Hadoop tool. It does not implement the org.apache.hadoop.util.Tool
interface, it does not execute through org.apache.hadoop.util.ToolRunner,
and therefore it does not support the command line arguments that a lot of
other Hadoop tools like the FsShell support. Specifically, it does not
support passing -D arguments to override fs.defaultFS or any other
configuration properties.
An alternative way to handle this would be to get a local copy of the
configuration directory from the remote cluster that you want to test. I
expect those configuration files would have fs.defaultFS set to the URL of
that remote cluster in core-site.xml. Before launching NNBench, run
"export HADOOP_CONF_DIR=<path to local copy of configuration files>".
After exporting that environment variable, you can run "hadoop classpath"
to print the classpath that will be used by all "hadoop" commands and
confirm that the correct configuration directory for the target cluster is
on the classpath. Then, you can run NNBench again, but drop the -D
argument, since it's going to get ignored anyway.
I don't see any reason why NNBench shouldn't implement the standard Tool
interface and thus support the command line arguments that you were
expecting. If you'd like to request that as an enhancement, please go
ahead and file an HDFS jira to request it. Feel free to post a patch too
if you're inclined. Otherwise, someone else in the community can pick it
up.
I hope this helps.
--Chris Nauroth
On 7/22/15, 12:41 AM, "Alexander Striffeler"
<a....@students.unibe.ch> wrote:
>Hi all
>
>I'm pretty new to the Hadoop environment and I'm about performing some
>micro benchmarks. In particular, I'm struggling with executing NNBench
>against an external File System:
>
>hadoop jar
>/usr/hdp/2.2.6.0-2800/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-t
>ests.jar
>nnbench -Dfs.defaultFS='hfds://<external.file.system>' -operation
>create_write -bytesToWrite 10 -maps 2 -reduces 1 -numberOfFiles 100
>-baseDir hdfs://dapsilon.daplab.ch/user/username/nnbench-`hostname -s`
>
>yields in
>java.lang.IllegalArgumentException: Wrong FS:
>hdfs://<external.file.system>/user/username/nnbench-hostname/data,
>expected: hdfs://<native fs>
>
>If I neglect the ext FS prefix in the baseDir, NNBench simply ignores
>the -D option and writes the files to the native DFS. Does anyone have
>an idea how to solve this and nnbench an external DFS?
>
>Thanks a lot, any hints are very appreciated!
>Regards,
>Alex
Re: NNBench on external HDFS
Posted by Chris Nauroth <cn...@hortonworks.com>.
Hi Alexander,
Your NNBench usage looks basically correct, but NNBench is not a standard
Hadoop tool. It does not implement the org.apache.hadoop.util.Tool
interface, it does not execute through org.apache.hadoop.util.ToolRunner,
and therefore it does not support the command line arguments that a lot of
other Hadoop tools like the FsShell support. Specifically, it does not
support passing -D arguments to override fs.defaultFS or any other
configuration properties.
An alternative way to handle this would be to get a local copy of the
configuration directory from the remote cluster that you want to test. I
expect those configuration files would have fs.defaultFS set to the URL of
that remote cluster in core-site.xml. Before launching NNBench, run
"export HADOOP_CONF_DIR=<path to local copy of configuration files>".
After exporting that environment variable, you can run "hadoop classpath"
to print the classpath that will be used by all "hadoop" commands and
confirm that the correct configuration directory for the target cluster is
on the classpath. Then, you can run NNBench again, but drop the -D
argument, since it's going to get ignored anyway.
I don't see any reason why NNBench shouldn't implement the standard Tool
interface and thus support the command line arguments that you were
expecting. If you'd like to request that as an enhancement, please go
ahead and file an HDFS jira to request it. Feel free to post a patch too
if you're inclined. Otherwise, someone else in the community can pick it
up.
I hope this helps.
--Chris Nauroth
On 7/22/15, 12:41 AM, "Alexander Striffeler"
<a....@students.unibe.ch> wrote:
>Hi all
>
>I'm pretty new to the Hadoop environment and I'm about performing some
>micro benchmarks. In particular, I'm struggling with executing NNBench
>against an external File System:
>
>hadoop jar
>/usr/hdp/2.2.6.0-2800/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-t
>ests.jar
>nnbench -Dfs.defaultFS='hfds://<external.file.system>' -operation
>create_write -bytesToWrite 10 -maps 2 -reduces 1 -numberOfFiles 100
>-baseDir hdfs://dapsilon.daplab.ch/user/username/nnbench-`hostname -s`
>
>yields in
>java.lang.IllegalArgumentException: Wrong FS:
>hdfs://<external.file.system>/user/username/nnbench-hostname/data,
>expected: hdfs://<native fs>
>
>If I neglect the ext FS prefix in the baseDir, NNBench simply ignores
>the -D option and writes the files to the native DFS. Does anyone have
>an idea how to solve this and nnbench an external DFS?
>
>Thanks a lot, any hints are very appreciated!
>Regards,
>Alex
Re: NNBench on external HDFS
Posted by Chris Nauroth <cn...@hortonworks.com>.
Hi Alexander,
Your NNBench usage looks basically correct, but NNBench is not a standard
Hadoop tool. It does not implement the org.apache.hadoop.util.Tool
interface, it does not execute through org.apache.hadoop.util.ToolRunner,
and therefore it does not support the command line arguments that a lot of
other Hadoop tools like the FsShell support. Specifically, it does not
support passing -D arguments to override fs.defaultFS or any other
configuration properties.
An alternative way to handle this would be to get a local copy of the
configuration directory from the remote cluster that you want to test. I
expect those configuration files would have fs.defaultFS set to the URL of
that remote cluster in core-site.xml. Before launching NNBench, run
"export HADOOP_CONF_DIR=<path to local copy of configuration files>".
After exporting that environment variable, you can run "hadoop classpath"
to print the classpath that will be used by all "hadoop" commands and
confirm that the correct configuration directory for the target cluster is
on the classpath. Then, you can run NNBench again, but drop the -D
argument, since it's going to get ignored anyway.
I don't see any reason why NNBench shouldn't implement the standard Tool
interface and thus support the command line arguments that you were
expecting. If you'd like to request that as an enhancement, please go
ahead and file an HDFS jira to request it. Feel free to post a patch too
if you're inclined. Otherwise, someone else in the community can pick it
up.
I hope this helps.
--Chris Nauroth
On 7/22/15, 12:41 AM, "Alexander Striffeler"
<a....@students.unibe.ch> wrote:
>Hi all
>
>I'm pretty new to the Hadoop environment and I'm about performing some
>micro benchmarks. In particular, I'm struggling with executing NNBench
>against an external File System:
>
>hadoop jar
>/usr/hdp/2.2.6.0-2800/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-t
>ests.jar
>nnbench -Dfs.defaultFS='hfds://<external.file.system>' -operation
>create_write -bytesToWrite 10 -maps 2 -reduces 1 -numberOfFiles 100
>-baseDir hdfs://dapsilon.daplab.ch/user/username/nnbench-`hostname -s`
>
>yields in
>java.lang.IllegalArgumentException: Wrong FS:
>hdfs://<external.file.system>/user/username/nnbench-hostname/data,
>expected: hdfs://<native fs>
>
>If I neglect the ext FS prefix in the baseDir, NNBench simply ignores
>the -D option and writes the files to the native DFS. Does anyone have
>an idea how to solve this and nnbench an external DFS?
>
>Thanks a lot, any hints are very appreciated!
>Regards,
>Alex