You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Karim Awara <ka...@kaust.edu.sa> on 2013/12/14 23:43:06 UTC
MiniDFSCluster setup
Hi,
I read I can use MiniDFSCluster to set my own tests in case I modify hadop
source code. I have built hadoop 2.2. However, I can't find any source on
how to get the MiniDFSCluster working. Can someone point a link to me that
helps?
--
Best Regards,
Karim Ahmed Awara
--
------------------------------
This message and its contents, including attachments are intended solely
for the original recipient. If you are not the intended recipient or have
received this message in error, please notify me immediately and delete
this message from your computer system. Any unauthorized use or
distribution is prohibited. Please consider the environment before printing
this email.
Re: MiniDFSCluster setup
Posted by Karim Awara <ka...@kaust.edu.sa>.
You were right on both of them! It is now working. Thanks a lot.
--
Best Regards,
Karim Ahmed Awara
On Mon, Dec 16, 2013 at 4:53 AM, Vinayakumar B <vi...@huawei.com>wrote:
> Hi Kareem,
>
> Hi See that issue you are facing is
> https://issues.apache.org/jira/browse/HADOOP-9908,
>
>
>
> It expectes common-version-info.properties to be present under classes
> directory, which will be present while running tests from maven. But when
> eclipse cleans the output and compiles again this will be cleaned.
>
>
>
> So I suggest merge the changes to your VersionInfo.java from HADOOP-9908
> and run the test. It just eats the NPE and unblock you from running tests.
>
>
>
> BTW, You may face another problem saying, “webapps/hdfs not found” after
> version file problem is solved.
>
> Well I have a solution for that too. If yes, follow below steps to
> resolve that.
>
> This kind of workaround only, I didnt see the fix available.
>
> 1. Create a folder in *hadoop-hdfs, *ex: *web*
>
> 2. Copy full *hadoop-hdfs/target/webapps *directory to
> *hadoop-hdfs/web*
>
> 3. Now add *hadoop-hdfs/web *as source directory in eclipse project.
>
> 4. Rebuild hadoop-hdfs and run the test.
>
>
>
> If any more problems let me know.
>
>
>
> Cheers,
>
> Vinayakumar B
>
>
>
> *From:* Karim Awara [mailto:karim.awara@kaust.edu.sa]
> *Sent:* 15 December 2013 22:26
> *To:* user
> *Subject:* Re: MiniDFSCluster setup
>
>
>
>
>
> I imported all the projects under the root dir "hadoop-2.2.0-src". And
> still the same error. I havenot touched any conf files after I build with
> maven. Is that correct? because it seems eclipse is not aware of some
> sources that command-line maven is aware of.
>
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
>
> On Sun, Dec 15, 2013 at 7:10 PM, Ted Yu <yu...@gmail.com> wrote:
>
> Can you import from root of your workspace so that all sub-projects are
> imported ?
>
>
>
> Cheers
>
>
>
> On Sun, Dec 15, 2013 at 7:45 AM, Karim Awara <ka...@kaust.edu.sa>
> wrote:
>
>
>
> Attached below. So What I did was after downloading hadoop src. I ran
>
> $ mvn install -DskipTests
>
> $ mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true
>
>
>
> And imported both Hadoop Common and HDFS projects into eclipse.
>
>
>
>
>
>
>
>
> java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hdfs.server.common.Storage.getBuildVersion(Storage.java:1026)
> at
> org.apache.hadoop.hdfs.server.protocol.NamespaceInfo.<init>(NamespaceInfo.java:58)
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:266)
> at
> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:122)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:775)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:642)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:334)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:316)
> at
> org.apache.hadoop.hdfs.TestWriteRead.initJunitModeTest(TestWriteRead.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> Caused by: java.lang.NullPointerException
> at java.util.Properties$LineReader.readLine(Properties.java:435)
> at java.util.Properties.load0(Properties.java:354)
> at java.util.Properties.load(Properties.java:342)
> at org.apache.hadoop.util.VersionInfo.<init>(VersionInfo.java:51)
> at org.apache.hadoop.util.VersionInfo.<clinit>(VersionInfo.java:97)
> ... 36 more
>
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.TestWriteRead.shutdown(TestWriteRead.java:89)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
>
>
>
>
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
>
> On Sun, Dec 15, 2013 at 5:59 PM, Ted Yu <yu...@gmail.com> wrote:
>
> Can you show us the full stack trace ?
>
> In Eclipse, was there any project shown with a red bang or red cross ?
>
>
>
> Cheers
>
>
>
> On Sun, Dec 15, 2013 at 2:26 AM, Karim Awara <ka...@kaust.edu.sa>
> wrote:
>
>
>
> It tells me java.lang.ExceptionInitializerError at
> org.apache.....hdfs.server.common.Storage.getBuildVersion!
>
> I feel I am missing some parameters?
>
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
>
> On Sun, Dec 15, 2013 at 5:08 AM, Ted Yu <yu...@gmail.com> wrote:
>
> You can use the following command to generate .project files for Eclipse
> (at the root of your workspace):
>
> mvn clean package -DskipTests eclipse:eclipse
>
>
>
> When you import hadoop, call sub-projects would be imported.
>
>
>
> I was able to run TestWriteRead in Eclipse successfully.
>
>
>
> Cheers
>
>
>
> On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <ka...@kaust.edu.sa>
> wrote:
>
>
>
> Running the test from Maven through commandline works fine. But I am
> using eclipse. And it generates problem if I try to run the test as Junit,
> as if eclipse is not aware of any of the conf parameters or args. Can
> someone point to me a detailed source where it explains how to run Junit
> through Eclipse for hadoop 2.2.x?
>
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
>
> On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
>
> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you would
> see a lot of tests which use MiniDFSCluster
>
> e.g.
>
> cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>
>
> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>
>
>
> Cheers
>
>
>
> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <ka...@kaust.edu.sa>
> wrote:
>
> Hi,
>
> I read I can use MiniDFSCluster to set my own tests in case I modify hadop
> source code. I have built hadoop 2.2. However, I can't find any source on
> how to get the MiniDFSCluster working. Can someone point a link to me that
> helps?
>
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
> ------------------------------
>
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
>
>
>
>
>
> ------------------------------
>
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
>
>
>
>
>
> ------------------------------
>
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
>
>
>
>
>
> ------------------------------
>
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
>
>
>
>
>
> ------------------------------
>
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
--
------------------------------
This message and its contents, including attachments are intended solely
for the original recipient. If you are not the intended recipient or have
received this message in error, please notify me immediately and delete
this message from your computer system. Any unauthorized use or
distribution is prohibited. Please consider the environment before printing
this email.
Re: MiniDFSCluster setup
Posted by Karim Awara <ka...@kaust.edu.sa>.
You were right on both of them! It is now working. Thanks a lot.
--
Best Regards,
Karim Ahmed Awara
On Mon, Dec 16, 2013 at 4:53 AM, Vinayakumar B <vi...@huawei.com>wrote:
> Hi Kareem,
>
> Hi See that issue you are facing is
> https://issues.apache.org/jira/browse/HADOOP-9908,
>
>
>
> It expectes common-version-info.properties to be present under classes
> directory, which will be present while running tests from maven. But when
> eclipse cleans the output and compiles again this will be cleaned.
>
>
>
> So I suggest merge the changes to your VersionInfo.java from HADOOP-9908
> and run the test. It just eats the NPE and unblock you from running tests.
>
>
>
> BTW, You may face another problem saying, “webapps/hdfs not found” after
> version file problem is solved.
>
> Well I have a solution for that too. If yes, follow below steps to
> resolve that.
>
> This kind of workaround only, I didnt see the fix available.
>
> 1. Create a folder in *hadoop-hdfs, *ex: *web*
>
> 2. Copy full *hadoop-hdfs/target/webapps *directory to
> *hadoop-hdfs/web*
>
> 3. Now add *hadoop-hdfs/web *as source directory in eclipse project.
>
> 4. Rebuild hadoop-hdfs and run the test.
>
>
>
> If any more problems let me know.
>
>
>
> Cheers,
>
> Vinayakumar B
>
>
>
> *From:* Karim Awara [mailto:karim.awara@kaust.edu.sa]
> *Sent:* 15 December 2013 22:26
> *To:* user
> *Subject:* Re: MiniDFSCluster setup
>
>
>
>
>
> I imported all the projects under the root dir "hadoop-2.2.0-src". And
> still the same error. I havenot touched any conf files after I build with
> maven. Is that correct? because it seems eclipse is not aware of some
> sources that command-line maven is aware of.
>
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
>
> On Sun, Dec 15, 2013 at 7:10 PM, Ted Yu <yu...@gmail.com> wrote:
>
> Can you import from root of your workspace so that all sub-projects are
> imported ?
>
>
>
> Cheers
>
>
>
> On Sun, Dec 15, 2013 at 7:45 AM, Karim Awara <ka...@kaust.edu.sa>
> wrote:
>
>
>
> Attached below. So What I did was after downloading hadoop src. I ran
>
> $ mvn install -DskipTests
>
> $ mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true
>
>
>
> And imported both Hadoop Common and HDFS projects into eclipse.
>
>
>
>
>
>
>
>
> java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hdfs.server.common.Storage.getBuildVersion(Storage.java:1026)
> at
> org.apache.hadoop.hdfs.server.protocol.NamespaceInfo.<init>(NamespaceInfo.java:58)
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:266)
> at
> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:122)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:775)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:642)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:334)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:316)
> at
> org.apache.hadoop.hdfs.TestWriteRead.initJunitModeTest(TestWriteRead.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> Caused by: java.lang.NullPointerException
> at java.util.Properties$LineReader.readLine(Properties.java:435)
> at java.util.Properties.load0(Properties.java:354)
> at java.util.Properties.load(Properties.java:342)
> at org.apache.hadoop.util.VersionInfo.<init>(VersionInfo.java:51)
> at org.apache.hadoop.util.VersionInfo.<clinit>(VersionInfo.java:97)
> ... 36 more
>
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.TestWriteRead.shutdown(TestWriteRead.java:89)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
>
>
>
>
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
>
> On Sun, Dec 15, 2013 at 5:59 PM, Ted Yu <yu...@gmail.com> wrote:
>
> Can you show us the full stack trace ?
>
> In Eclipse, was there any project shown with a red bang or red cross ?
>
>
>
> Cheers
>
>
>
> On Sun, Dec 15, 2013 at 2:26 AM, Karim Awara <ka...@kaust.edu.sa>
> wrote:
>
>
>
> It tells me java.lang.ExceptionInitializerError at
> org.apache.....hdfs.server.common.Storage.getBuildVersion!
>
> I feel I am missing some parameters?
>
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
>
> On Sun, Dec 15, 2013 at 5:08 AM, Ted Yu <yu...@gmail.com> wrote:
>
> You can use the following command to generate .project files for Eclipse
> (at the root of your workspace):
>
> mvn clean package -DskipTests eclipse:eclipse
>
>
>
> When you import hadoop, call sub-projects would be imported.
>
>
>
> I was able to run TestWriteRead in Eclipse successfully.
>
>
>
> Cheers
>
>
>
> On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <ka...@kaust.edu.sa>
> wrote:
>
>
>
> Running the test from Maven through commandline works fine. But I am
> using eclipse. And it generates problem if I try to run the test as Junit,
> as if eclipse is not aware of any of the conf parameters or args. Can
> someone point to me a detailed source where it explains how to run Junit
> through Eclipse for hadoop 2.2.x?
>
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
>
> On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
>
> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you would
> see a lot of tests which use MiniDFSCluster
>
> e.g.
>
> cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>
>
> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>
>
>
> Cheers
>
>
>
> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <ka...@kaust.edu.sa>
> wrote:
>
> Hi,
>
> I read I can use MiniDFSCluster to set my own tests in case I modify hadop
> source code. I have built hadoop 2.2. However, I can't find any source on
> how to get the MiniDFSCluster working. Can someone point a link to me that
> helps?
>
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
> ------------------------------
>
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
>
>
>
>
>
> ------------------------------
>
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
>
>
>
>
>
> ------------------------------
>
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
>
>
>
>
>
> ------------------------------
>
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
>
>
>
>
>
> ------------------------------
>
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
--
------------------------------
This message and its contents, including attachments are intended solely
for the original recipient. If you are not the intended recipient or have
received this message in error, please notify me immediately and delete
this message from your computer system. Any unauthorized use or
distribution is prohibited. Please consider the environment before printing
this email.
Re: MiniDFSCluster setup
Posted by Karim Awara <ka...@kaust.edu.sa>.
You were right on both of them! It is now working. Thanks a lot.
--
Best Regards,
Karim Ahmed Awara
On Mon, Dec 16, 2013 at 4:53 AM, Vinayakumar B <vi...@huawei.com>wrote:
> Hi Kareem,
>
> Hi See that issue you are facing is
> https://issues.apache.org/jira/browse/HADOOP-9908,
>
>
>
> It expectes common-version-info.properties to be present under classes
> directory, which will be present while running tests from maven. But when
> eclipse cleans the output and compiles again this will be cleaned.
>
>
>
> So I suggest merge the changes to your VersionInfo.java from HADOOP-9908
> and run the test. It just eats the NPE and unblock you from running tests.
>
>
>
> BTW, You may face another problem saying, “webapps/hdfs not found” after
> version file problem is solved.
>
> Well I have a solution for that too. If yes, follow below steps to
> resolve that.
>
> This kind of workaround only, I didnt see the fix available.
>
> 1. Create a folder in *hadoop-hdfs, *ex: *web*
>
> 2. Copy full *hadoop-hdfs/target/webapps *directory to
> *hadoop-hdfs/web*
>
> 3. Now add *hadoop-hdfs/web *as source directory in eclipse project.
>
> 4. Rebuild hadoop-hdfs and run the test.
>
>
>
> If any more problems let me know.
>
>
>
> Cheers,
>
> Vinayakumar B
>
>
>
> *From:* Karim Awara [mailto:karim.awara@kaust.edu.sa]
> *Sent:* 15 December 2013 22:26
> *To:* user
> *Subject:* Re: MiniDFSCluster setup
>
>
>
>
>
> I imported all the projects under the root dir "hadoop-2.2.0-src". And
> still the same error. I havenot touched any conf files after I build with
> maven. Is that correct? because it seems eclipse is not aware of some
> sources that command-line maven is aware of.
>
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
>
> On Sun, Dec 15, 2013 at 7:10 PM, Ted Yu <yu...@gmail.com> wrote:
>
> Can you import from root of your workspace so that all sub-projects are
> imported ?
>
>
>
> Cheers
>
>
>
> On Sun, Dec 15, 2013 at 7:45 AM, Karim Awara <ka...@kaust.edu.sa>
> wrote:
>
>
>
> Attached below. So What I did was after downloading hadoop src. I ran
>
> $ mvn install -DskipTests
>
> $ mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true
>
>
>
> And imported both Hadoop Common and HDFS projects into eclipse.
>
>
>
>
>
>
>
>
> java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hdfs.server.common.Storage.getBuildVersion(Storage.java:1026)
> at
> org.apache.hadoop.hdfs.server.protocol.NamespaceInfo.<init>(NamespaceInfo.java:58)
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:266)
> at
> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:122)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:775)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:642)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:334)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:316)
> at
> org.apache.hadoop.hdfs.TestWriteRead.initJunitModeTest(TestWriteRead.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> Caused by: java.lang.NullPointerException
> at java.util.Properties$LineReader.readLine(Properties.java:435)
> at java.util.Properties.load0(Properties.java:354)
> at java.util.Properties.load(Properties.java:342)
> at org.apache.hadoop.util.VersionInfo.<init>(VersionInfo.java:51)
> at org.apache.hadoop.util.VersionInfo.<clinit>(VersionInfo.java:97)
> ... 36 more
>
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.TestWriteRead.shutdown(TestWriteRead.java:89)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
>
>
>
>
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
>
> On Sun, Dec 15, 2013 at 5:59 PM, Ted Yu <yu...@gmail.com> wrote:
>
> Can you show us the full stack trace ?
>
> In Eclipse, was there any project shown with a red bang or red cross ?
>
>
>
> Cheers
>
>
>
> On Sun, Dec 15, 2013 at 2:26 AM, Karim Awara <ka...@kaust.edu.sa>
> wrote:
>
>
>
> It tells me java.lang.ExceptionInitializerError at
> org.apache.....hdfs.server.common.Storage.getBuildVersion!
>
> I feel I am missing some parameters?
>
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
>
> On Sun, Dec 15, 2013 at 5:08 AM, Ted Yu <yu...@gmail.com> wrote:
>
> You can use the following command to generate .project files for Eclipse
> (at the root of your workspace):
>
> mvn clean package -DskipTests eclipse:eclipse
>
>
>
> When you import hadoop, call sub-projects would be imported.
>
>
>
> I was able to run TestWriteRead in Eclipse successfully.
>
>
>
> Cheers
>
>
>
> On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <ka...@kaust.edu.sa>
> wrote:
>
>
>
> Running the test from Maven through commandline works fine. But I am
> using eclipse. And it generates problem if I try to run the test as Junit,
> as if eclipse is not aware of any of the conf parameters or args. Can
> someone point to me a detailed source where it explains how to run Junit
> through Eclipse for hadoop 2.2.x?
>
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
>
> On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
>
> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you would
> see a lot of tests which use MiniDFSCluster
>
> e.g.
>
> cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>
>
> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>
>
>
> Cheers
>
>
>
> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <ka...@kaust.edu.sa>
> wrote:
>
> Hi,
>
> I read I can use MiniDFSCluster to set my own tests in case I modify hadop
> source code. I have built hadoop 2.2. However, I can't find any source on
> how to get the MiniDFSCluster working. Can someone point a link to me that
> helps?
>
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
> ------------------------------
>
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
>
>
>
>
>
> ------------------------------
>
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
>
>
>
>
>
> ------------------------------
>
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
>
>
>
>
>
> ------------------------------
>
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
>
>
>
>
>
> ------------------------------
>
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
--
------------------------------
This message and its contents, including attachments are intended solely
for the original recipient. If you are not the intended recipient or have
received this message in error, please notify me immediately and delete
this message from your computer system. Any unauthorized use or
distribution is prohibited. Please consider the environment before printing
this email.
Re: MiniDFSCluster setup
Posted by Karim Awara <ka...@kaust.edu.sa>.
You were right on both of them! It is now working. Thanks a lot.
--
Best Regards,
Karim Ahmed Awara
On Mon, Dec 16, 2013 at 4:53 AM, Vinayakumar B <vi...@huawei.com>wrote:
> Hi Kareem,
>
> Hi See that issue you are facing is
> https://issues.apache.org/jira/browse/HADOOP-9908,
>
>
>
> It expectes common-version-info.properties to be present under classes
> directory, which will be present while running tests from maven. But when
> eclipse cleans the output and compiles again this will be cleaned.
>
>
>
> So I suggest merge the changes to your VersionInfo.java from HADOOP-9908
> and run the test. It just eats the NPE and unblock you from running tests.
>
>
>
> BTW, You may face another problem saying, “webapps/hdfs not found” after
> version file problem is solved.
>
> Well I have a solution for that too. If yes, follow below steps to
> resolve that.
>
> This kind of workaround only, I didnt see the fix available.
>
> 1. Create a folder in *hadoop-hdfs, *ex: *web*
>
> 2. Copy full *hadoop-hdfs/target/webapps *directory to
> *hadoop-hdfs/web*
>
> 3. Now add *hadoop-hdfs/web *as source directory in eclipse project.
>
> 4. Rebuild hadoop-hdfs and run the test.
>
>
>
> If any more problems let me know.
>
>
>
> Cheers,
>
> Vinayakumar B
>
>
>
> *From:* Karim Awara [mailto:karim.awara@kaust.edu.sa]
> *Sent:* 15 December 2013 22:26
> *To:* user
> *Subject:* Re: MiniDFSCluster setup
>
>
>
>
>
> I imported all the projects under the root dir "hadoop-2.2.0-src". And
> still the same error. I havenot touched any conf files after I build with
> maven. Is that correct? because it seems eclipse is not aware of some
> sources that command-line maven is aware of.
>
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
>
> On Sun, Dec 15, 2013 at 7:10 PM, Ted Yu <yu...@gmail.com> wrote:
>
> Can you import from root of your workspace so that all sub-projects are
> imported ?
>
>
>
> Cheers
>
>
>
> On Sun, Dec 15, 2013 at 7:45 AM, Karim Awara <ka...@kaust.edu.sa>
> wrote:
>
>
>
> Attached below. So What I did was after downloading hadoop src. I ran
>
> $ mvn install -DskipTests
>
> $ mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true
>
>
>
> And imported both Hadoop Common and HDFS projects into eclipse.
>
>
>
>
>
>
>
>
> java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hdfs.server.common.Storage.getBuildVersion(Storage.java:1026)
> at
> org.apache.hadoop.hdfs.server.protocol.NamespaceInfo.<init>(NamespaceInfo.java:58)
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:266)
> at
> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:122)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:775)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:642)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:334)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:316)
> at
> org.apache.hadoop.hdfs.TestWriteRead.initJunitModeTest(TestWriteRead.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> Caused by: java.lang.NullPointerException
> at java.util.Properties$LineReader.readLine(Properties.java:435)
> at java.util.Properties.load0(Properties.java:354)
> at java.util.Properties.load(Properties.java:342)
> at org.apache.hadoop.util.VersionInfo.<init>(VersionInfo.java:51)
> at org.apache.hadoop.util.VersionInfo.<clinit>(VersionInfo.java:97)
> ... 36 more
>
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.TestWriteRead.shutdown(TestWriteRead.java:89)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
>
>
>
>
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
>
> On Sun, Dec 15, 2013 at 5:59 PM, Ted Yu <yu...@gmail.com> wrote:
>
> Can you show us the full stack trace ?
>
> In Eclipse, was there any project shown with a red bang or red cross ?
>
>
>
> Cheers
>
>
>
> On Sun, Dec 15, 2013 at 2:26 AM, Karim Awara <ka...@kaust.edu.sa>
> wrote:
>
>
>
> It tells me java.lang.ExceptionInitializerError at
> org.apache.....hdfs.server.common.Storage.getBuildVersion!
>
> I feel I am missing some parameters?
>
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
>
> On Sun, Dec 15, 2013 at 5:08 AM, Ted Yu <yu...@gmail.com> wrote:
>
> You can use the following command to generate .project files for Eclipse
> (at the root of your workspace):
>
> mvn clean package -DskipTests eclipse:eclipse
>
>
>
> When you import hadoop, call sub-projects would be imported.
>
>
>
> I was able to run TestWriteRead in Eclipse successfully.
>
>
>
> Cheers
>
>
>
> On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <ka...@kaust.edu.sa>
> wrote:
>
>
>
> Running the test from Maven through commandline works fine. But I am
> using eclipse. And it generates problem if I try to run the test as Junit,
> as if eclipse is not aware of any of the conf parameters or args. Can
> someone point to me a detailed source where it explains how to run Junit
> through Eclipse for hadoop 2.2.x?
>
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
>
> On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
>
> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you would
> see a lot of tests which use MiniDFSCluster
>
> e.g.
>
> cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>
>
> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>
>
>
> Cheers
>
>
>
> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <ka...@kaust.edu.sa>
> wrote:
>
> Hi,
>
> I read I can use MiniDFSCluster to set my own tests in case I modify hadop
> source code. I have built hadoop 2.2. However, I can't find any source on
> how to get the MiniDFSCluster working. Can someone point a link to me that
> helps?
>
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
> ------------------------------
>
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
>
>
>
>
>
> ------------------------------
>
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
>
>
>
>
>
> ------------------------------
>
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
>
>
>
>
>
> ------------------------------
>
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
>
>
>
>
>
> ------------------------------
>
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
--
------------------------------
This message and its contents, including attachments are intended solely
for the original recipient. If you are not the intended recipient or have
received this message in error, please notify me immediately and delete
this message from your computer system. Any unauthorized use or
distribution is prohibited. Please consider the environment before printing
this email.
RE: MiniDFSCluster setup
Posted by Vinayakumar B <vi...@huawei.com>.
Hi Kareem,
Hi See that issue you are facing is https://issues.apache.org/jira/browse/HADOOP-9908,
It expectes common-version-info.properties to be present under classes directory, which will be present while running tests from maven. But when eclipse cleans the output and compiles again this will be cleaned.
So I suggest merge the changes to your VersionInfo.java from HADOOP-9908 and run the test. It just eats the NPE and unblock you from running tests.
BTW, You may face another problem saying, "webapps/hdfs not found" after version file problem is solved.
Well I have a solution for that too. If yes, follow below steps to resolve that.
This kind of workaround only, I didnt see the fix available.
1. Create a folder in hadoop-hdfs, ex: web
2. Copy full hadoop-hdfs/target/webapps directory to hadoop-hdfs/web
3. Now add hadoop-hdfs/web as source directory in eclipse project.
4. Rebuild hadoop-hdfs and run the test.
If any more problems let me know.
Cheers,
Vinayakumar B
From: Karim Awara [mailto:karim.awara@kaust.edu.sa]
Sent: 15 December 2013 22:26
To: user
Subject: Re: MiniDFSCluster setup
I imported all the projects under the root dir "hadoop-2.2.0-src". And still the same error. I havenot touched any conf files after I build with maven. Is that correct? because it seems eclipse is not aware of some sources that command-line maven is aware of.
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 7:10 PM, Ted Yu <yu...@gmail.com>> wrote:
Can you import from root of your workspace so that all sub-projects are imported ?
Cheers
On Sun, Dec 15, 2013 at 7:45 AM, Karim Awara <ka...@kaust.edu.sa>> wrote:
Attached below. So What I did was after downloading hadoop src. I ran
$ mvn install -DskipTests
$ mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true
And imported both Hadoop Common and HDFS projects into eclipse.
java.lang.ExceptionInInitializerError
at org.apache.hadoop.hdfs.server.common.Storage.getBuildVersion(Storage.java:1026)
at org.apache.hadoop.hdfs.server.protocol.NamespaceInfo.<init>(NamespaceInfo.java:58)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:266)
at org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:122)
at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:775)
at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:642)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:334)
at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:316)
at org.apache.hadoop.hdfs.TestWriteRead.initJunitModeTest(TestWriteRead.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Caused by: java.lang.NullPointerException
at java.util.Properties$LineReader.readLine(Properties.java:435)
at java.util.Properties.load0(Properties.java:354)
at java.util.Properties.load(Properties.java:342)
at org.apache.hadoop.util.VersionInfo.<init>(VersionInfo.java:51)
at org.apache.hadoop.util.VersionInfo.<clinit>(VersionInfo.java:97)
... 36 more
java.lang.NullPointerException
at org.apache.hadoop.hdfs.TestWriteRead.shutdown(TestWriteRead.java:89)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 5:59 PM, Ted Yu <yu...@gmail.com>> wrote:
Can you show us the full stack trace ?
In Eclipse, was there any project shown with a red bang or red cross ?
Cheers
On Sun, Dec 15, 2013 at 2:26 AM, Karim Awara <ka...@kaust.edu.sa>> wrote:
It tells me java.lang.ExceptionInitializerError at org.apache.....hdfs.server.common.Storage.getBuildVersion!
I feel I am missing some parameters?
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 5:08 AM, Ted Yu <yu...@gmail.com>> wrote:
You can use the following command to generate .project files for Eclipse (at the root of your workspace):
mvn clean package -DskipTests eclipse:eclipse
When you import hadoop, call sub-projects would be imported.
I was able to run TestWriteRead in Eclipse successfully.
Cheers
On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <ka...@kaust.edu.sa>> wrote:
Running the test from Maven through commandline works fine. But I am using eclipse. And it generates problem if I try to run the test as Junit, as if eclipse is not aware of any of the conf parameters or args. Can someone point to me a detailed source where it explains how to run Junit through Eclipse for hadoop 2.2.x?
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com>> wrote:
If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you would see a lot of tests which use MiniDFSCluster
e.g.
cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
Cheers
On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <ka...@kaust.edu.sa>> wrote:
Hi,
I read I can use MiniDFSCluster to set my own tests in case I modify hadop source code. I have built hadoop 2.2. However, I can't find any source on how to get the MiniDFSCluster working. Can someone point a link to me that helps?
--
Best Regards,
Karim Ahmed Awara
________________________________
This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.
________________________________
This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.
________________________________
This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.
________________________________
This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.
________________________________
This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.
RE: MiniDFSCluster setup
Posted by Vinayakumar B <vi...@huawei.com>.
Hi Kareem,
Hi See that issue you are facing is https://issues.apache.org/jira/browse/HADOOP-9908,
It expectes common-version-info.properties to be present under classes directory, which will be present while running tests from maven. But when eclipse cleans the output and compiles again this will be cleaned.
So I suggest merge the changes to your VersionInfo.java from HADOOP-9908 and run the test. It just eats the NPE and unblock you from running tests.
BTW, You may face another problem saying, "webapps/hdfs not found" after version file problem is solved.
Well I have a solution for that too. If yes, follow below steps to resolve that.
This kind of workaround only, I didnt see the fix available.
1. Create a folder in hadoop-hdfs, ex: web
2. Copy full hadoop-hdfs/target/webapps directory to hadoop-hdfs/web
3. Now add hadoop-hdfs/web as source directory in eclipse project.
4. Rebuild hadoop-hdfs and run the test.
If any more problems let me know.
Cheers,
Vinayakumar B
From: Karim Awara [mailto:karim.awara@kaust.edu.sa]
Sent: 15 December 2013 22:26
To: user
Subject: Re: MiniDFSCluster setup
I imported all the projects under the root dir "hadoop-2.2.0-src". And still the same error. I havenot touched any conf files after I build with maven. Is that correct? because it seems eclipse is not aware of some sources that command-line maven is aware of.
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 7:10 PM, Ted Yu <yu...@gmail.com>> wrote:
Can you import from root of your workspace so that all sub-projects are imported ?
Cheers
On Sun, Dec 15, 2013 at 7:45 AM, Karim Awara <ka...@kaust.edu.sa>> wrote:
Attached below. So What I did was after downloading hadoop src. I ran
$ mvn install -DskipTests
$ mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true
And imported both Hadoop Common and HDFS projects into eclipse.
java.lang.ExceptionInInitializerError
at org.apache.hadoop.hdfs.server.common.Storage.getBuildVersion(Storage.java:1026)
at org.apache.hadoop.hdfs.server.protocol.NamespaceInfo.<init>(NamespaceInfo.java:58)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:266)
at org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:122)
at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:775)
at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:642)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:334)
at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:316)
at org.apache.hadoop.hdfs.TestWriteRead.initJunitModeTest(TestWriteRead.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Caused by: java.lang.NullPointerException
at java.util.Properties$LineReader.readLine(Properties.java:435)
at java.util.Properties.load0(Properties.java:354)
at java.util.Properties.load(Properties.java:342)
at org.apache.hadoop.util.VersionInfo.<init>(VersionInfo.java:51)
at org.apache.hadoop.util.VersionInfo.<clinit>(VersionInfo.java:97)
... 36 more
java.lang.NullPointerException
at org.apache.hadoop.hdfs.TestWriteRead.shutdown(TestWriteRead.java:89)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 5:59 PM, Ted Yu <yu...@gmail.com>> wrote:
Can you show us the full stack trace ?
In Eclipse, was there any project shown with a red bang or red cross ?
Cheers
On Sun, Dec 15, 2013 at 2:26 AM, Karim Awara <ka...@kaust.edu.sa>> wrote:
It tells me java.lang.ExceptionInitializerError at org.apache.....hdfs.server.common.Storage.getBuildVersion!
I feel I am missing some parameters?
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 5:08 AM, Ted Yu <yu...@gmail.com>> wrote:
You can use the following command to generate .project files for Eclipse (at the root of your workspace):
mvn clean package -DskipTests eclipse:eclipse
When you import hadoop, call sub-projects would be imported.
I was able to run TestWriteRead in Eclipse successfully.
Cheers
On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <ka...@kaust.edu.sa>> wrote:
Running the test from Maven through commandline works fine. But I am using eclipse. And it generates problem if I try to run the test as Junit, as if eclipse is not aware of any of the conf parameters or args. Can someone point to me a detailed source where it explains how to run Junit through Eclipse for hadoop 2.2.x?
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com>> wrote:
If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you would see a lot of tests which use MiniDFSCluster
e.g.
cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
Cheers
On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <ka...@kaust.edu.sa>> wrote:
Hi,
I read I can use MiniDFSCluster to set my own tests in case I modify hadop source code. I have built hadoop 2.2. However, I can't find any source on how to get the MiniDFSCluster working. Can someone point a link to me that helps?
--
Best Regards,
Karim Ahmed Awara
________________________________
This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.
________________________________
This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.
________________________________
This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.
________________________________
This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.
________________________________
This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.
RE: MiniDFSCluster setup
Posted by Vinayakumar B <vi...@huawei.com>.
Hi Kareem,
Hi See that issue you are facing is https://issues.apache.org/jira/browse/HADOOP-9908,
It expectes common-version-info.properties to be present under classes directory, which will be present while running tests from maven. But when eclipse cleans the output and compiles again this will be cleaned.
So I suggest merge the changes to your VersionInfo.java from HADOOP-9908 and run the test. It just eats the NPE and unblock you from running tests.
BTW, You may face another problem saying, "webapps/hdfs not found" after version file problem is solved.
Well I have a solution for that too. If yes, follow below steps to resolve that.
This kind of workaround only, I didnt see the fix available.
1. Create a folder in hadoop-hdfs, ex: web
2. Copy full hadoop-hdfs/target/webapps directory to hadoop-hdfs/web
3. Now add hadoop-hdfs/web as source directory in eclipse project.
4. Rebuild hadoop-hdfs and run the test.
If any more problems let me know.
Cheers,
Vinayakumar B
From: Karim Awara [mailto:karim.awara@kaust.edu.sa]
Sent: 15 December 2013 22:26
To: user
Subject: Re: MiniDFSCluster setup
I imported all the projects under the root dir "hadoop-2.2.0-src". And still the same error. I havenot touched any conf files after I build with maven. Is that correct? because it seems eclipse is not aware of some sources that command-line maven is aware of.
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 7:10 PM, Ted Yu <yu...@gmail.com>> wrote:
Can you import from root of your workspace so that all sub-projects are imported ?
Cheers
On Sun, Dec 15, 2013 at 7:45 AM, Karim Awara <ka...@kaust.edu.sa>> wrote:
Attached below. So What I did was after downloading hadoop src. I ran
$ mvn install -DskipTests
$ mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true
And imported both Hadoop Common and HDFS projects into eclipse.
java.lang.ExceptionInInitializerError
at org.apache.hadoop.hdfs.server.common.Storage.getBuildVersion(Storage.java:1026)
at org.apache.hadoop.hdfs.server.protocol.NamespaceInfo.<init>(NamespaceInfo.java:58)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:266)
at org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:122)
at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:775)
at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:642)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:334)
at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:316)
at org.apache.hadoop.hdfs.TestWriteRead.initJunitModeTest(TestWriteRead.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Caused by: java.lang.NullPointerException
at java.util.Properties$LineReader.readLine(Properties.java:435)
at java.util.Properties.load0(Properties.java:354)
at java.util.Properties.load(Properties.java:342)
at org.apache.hadoop.util.VersionInfo.<init>(VersionInfo.java:51)
at org.apache.hadoop.util.VersionInfo.<clinit>(VersionInfo.java:97)
... 36 more
java.lang.NullPointerException
at org.apache.hadoop.hdfs.TestWriteRead.shutdown(TestWriteRead.java:89)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 5:59 PM, Ted Yu <yu...@gmail.com>> wrote:
Can you show us the full stack trace ?
In Eclipse, was there any project shown with a red bang or red cross ?
Cheers
On Sun, Dec 15, 2013 at 2:26 AM, Karim Awara <ka...@kaust.edu.sa>> wrote:
It tells me java.lang.ExceptionInitializerError at org.apache.....hdfs.server.common.Storage.getBuildVersion!
I feel I am missing some parameters?
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 5:08 AM, Ted Yu <yu...@gmail.com>> wrote:
You can use the following command to generate .project files for Eclipse (at the root of your workspace):
mvn clean package -DskipTests eclipse:eclipse
When you import hadoop, call sub-projects would be imported.
I was able to run TestWriteRead in Eclipse successfully.
Cheers
On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <ka...@kaust.edu.sa>> wrote:
Running the test from Maven through commandline works fine. But I am using eclipse. And it generates problem if I try to run the test as Junit, as if eclipse is not aware of any of the conf parameters or args. Can someone point to me a detailed source where it explains how to run Junit through Eclipse for hadoop 2.2.x?
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com>> wrote:
If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you would see a lot of tests which use MiniDFSCluster
e.g.
cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
Cheers
On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <ka...@kaust.edu.sa>> wrote:
Hi,
I read I can use MiniDFSCluster to set my own tests in case I modify hadop source code. I have built hadoop 2.2. However, I can't find any source on how to get the MiniDFSCluster working. Can someone point a link to me that helps?
--
Best Regards,
Karim Ahmed Awara
________________________________
This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.
________________________________
This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.
________________________________
This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.
________________________________
This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.
________________________________
This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.
RE: MiniDFSCluster setup
Posted by Vinayakumar B <vi...@huawei.com>.
Hi Kareem,
Hi See that issue you are facing is https://issues.apache.org/jira/browse/HADOOP-9908,
It expectes common-version-info.properties to be present under classes directory, which will be present while running tests from maven. But when eclipse cleans the output and compiles again this will be cleaned.
So I suggest merge the changes to your VersionInfo.java from HADOOP-9908 and run the test. It just eats the NPE and unblock you from running tests.
BTW, You may face another problem saying, "webapps/hdfs not found" after version file problem is solved.
Well I have a solution for that too. If yes, follow below steps to resolve that.
This kind of workaround only, I didnt see the fix available.
1. Create a folder in hadoop-hdfs, ex: web
2. Copy full hadoop-hdfs/target/webapps directory to hadoop-hdfs/web
3. Now add hadoop-hdfs/web as source directory in eclipse project.
4. Rebuild hadoop-hdfs and run the test.
If any more problems let me know.
Cheers,
Vinayakumar B
From: Karim Awara [mailto:karim.awara@kaust.edu.sa]
Sent: 15 December 2013 22:26
To: user
Subject: Re: MiniDFSCluster setup
I imported all the projects under the root dir "hadoop-2.2.0-src". And still the same error. I havenot touched any conf files after I build with maven. Is that correct? because it seems eclipse is not aware of some sources that command-line maven is aware of.
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 7:10 PM, Ted Yu <yu...@gmail.com>> wrote:
Can you import from root of your workspace so that all sub-projects are imported ?
Cheers
On Sun, Dec 15, 2013 at 7:45 AM, Karim Awara <ka...@kaust.edu.sa>> wrote:
Attached below. So What I did was after downloading hadoop src. I ran
$ mvn install -DskipTests
$ mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true
And imported both Hadoop Common and HDFS projects into eclipse.
java.lang.ExceptionInInitializerError
at org.apache.hadoop.hdfs.server.common.Storage.getBuildVersion(Storage.java:1026)
at org.apache.hadoop.hdfs.server.protocol.NamespaceInfo.<init>(NamespaceInfo.java:58)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:266)
at org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:122)
at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:775)
at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:642)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:334)
at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:316)
at org.apache.hadoop.hdfs.TestWriteRead.initJunitModeTest(TestWriteRead.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Caused by: java.lang.NullPointerException
at java.util.Properties$LineReader.readLine(Properties.java:435)
at java.util.Properties.load0(Properties.java:354)
at java.util.Properties.load(Properties.java:342)
at org.apache.hadoop.util.VersionInfo.<init>(VersionInfo.java:51)
at org.apache.hadoop.util.VersionInfo.<clinit>(VersionInfo.java:97)
... 36 more
java.lang.NullPointerException
at org.apache.hadoop.hdfs.TestWriteRead.shutdown(TestWriteRead.java:89)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 5:59 PM, Ted Yu <yu...@gmail.com>> wrote:
Can you show us the full stack trace ?
In Eclipse, was there any project shown with a red bang or red cross ?
Cheers
On Sun, Dec 15, 2013 at 2:26 AM, Karim Awara <ka...@kaust.edu.sa>> wrote:
It tells me java.lang.ExceptionInitializerError at org.apache.....hdfs.server.common.Storage.getBuildVersion!
I feel I am missing some parameters?
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 5:08 AM, Ted Yu <yu...@gmail.com>> wrote:
You can use the following command to generate .project files for Eclipse (at the root of your workspace):
mvn clean package -DskipTests eclipse:eclipse
When you import hadoop, call sub-projects would be imported.
I was able to run TestWriteRead in Eclipse successfully.
Cheers
On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <ka...@kaust.edu.sa>> wrote:
Running the test from Maven through commandline works fine. But I am using eclipse. And it generates problem if I try to run the test as Junit, as if eclipse is not aware of any of the conf parameters or args. Can someone point to me a detailed source where it explains how to run Junit through Eclipse for hadoop 2.2.x?
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com>> wrote:
If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you would see a lot of tests which use MiniDFSCluster
e.g.
cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
Cheers
On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <ka...@kaust.edu.sa>> wrote:
Hi,
I read I can use MiniDFSCluster to set my own tests in case I modify hadop source code. I have built hadoop 2.2. However, I can't find any source on how to get the MiniDFSCluster working. Can someone point a link to me that helps?
--
Best Regards,
Karim Ahmed Awara
________________________________
This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.
________________________________
This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.
________________________________
This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.
________________________________
This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.
________________________________
This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.
Re: MiniDFSCluster setup
Posted by Karim Awara <ka...@kaust.edu.sa>.
I imported all the projects under the root dir "hadoop-2.2.0-src". And
still the same error. I havenot touched any conf files after I build with
maven. Is that correct? because it seems eclipse is not aware of some
sources that command-line maven is aware of.
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 7:10 PM, Ted Yu <yu...@gmail.com> wrote:
> Can you import from root of your workspace so that all sub-projects are
> imported ?
>
> Cheers
>
>
> On Sun, Dec 15, 2013 at 7:45 AM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
>>
>> Attached below. So What I did was after downloading hadoop src. I ran
>>
>> $ mvn install -DskipTests$ mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true
>>
>>
>> And imported both Hadoop Common and HDFS projects into eclipse.
>>
>>
>>
>>
>>
>>
>>
>>
>> java.lang.ExceptionInInitializerError
>> at
>> org.apache.hadoop.hdfs.server.common.Storage.getBuildVersion(Storage.java:1026)
>> at
>> org.apache.hadoop.hdfs.server.protocol.NamespaceInfo.<init>(NamespaceInfo.java:58)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:266)
>> at
>> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:122)
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:775)
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:642)
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:334)
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:316)
>> at
>> org.apache.hadoop.hdfs.TestWriteRead.initJunitModeTest(TestWriteRead.java:77)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
>> at
>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>> at
>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
>> at
>> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
>> at
>> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
>> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
>> at
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
>> at
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
>> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
>> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
>> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
>> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
>> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
>> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
>> at
>> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
>> at
>> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
>> Caused by: java.lang.NullPointerException
>> at java.util.Properties$LineReader.readLine(Properties.java:435)
>> at java.util.Properties.load0(Properties.java:354)
>> at java.util.Properties.load(Properties.java:342)
>> at org.apache.hadoop.util.VersionInfo.<init>(VersionInfo.java:51)
>> at org.apache.hadoop.util.VersionInfo.<clinit>(VersionInfo.java:97)
>> ... 36 more
>>
>> java.lang.NullPointerException
>> at
>> org.apache.hadoop.hdfs.TestWriteRead.shutdown(TestWriteRead.java:89)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
>> at
>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>> at
>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
>> at
>> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
>> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
>> at
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
>> at
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
>> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
>> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
>> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
>> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
>> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
>> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
>> at
>> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
>> at
>> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
>>
>>
>>
>>
>>
>> --
>> Best Regards,
>> Karim Ahmed Awara
>>
>>
>> On Sun, Dec 15, 2013 at 5:59 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> Can you show us the full stack trace ?
>>> In Eclipse, was there any project shown with a red bang or red cross ?
>>>
>>> Cheers
>>>
>>>
>>> On Sun, Dec 15, 2013 at 2:26 AM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>>
>>>>
>>>> It tells me java.lang.ExceptionInitializerError at
>>>> org.apache.....hdfs.server.common.Storage.getBuildVersion!
>>>>
>>>> I feel I am missing some parameters?
>>>>
>>>> --
>>>> Best Regards,
>>>> Karim Ahmed Awara
>>>>
>>>>
>>>> On Sun, Dec 15, 2013 at 5:08 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> You can use the following command to generate .project files for
>>>>> Eclipse (at the root of your workspace):
>>>>> mvn clean package -DskipTests eclipse:eclipse
>>>>>
>>>>> When you import hadoop, call sub-projects would be imported.
>>>>>
>>>>> I was able to run TestWriteRead in Eclipse successfully.
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <karim.awara@kaust.edu.sa
>>>>> > wrote:
>>>>>
>>>>>>
>>>>>> Running the test from Maven through commandline works fine. But I am
>>>>>> using eclipse. And it generates problem if I try to run the test as Junit,
>>>>>> as if eclipse is not aware of any of the conf parameters or args. Can
>>>>>> someone point to me a detailed source where it explains how to run Junit
>>>>>> through Eclipse for hadoop 2.2.x?
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards,
>>>>>> Karim Ahmed Awara
>>>>>>
>>>>>>
>>>>>> On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>
>>>>>>> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you
>>>>>>> would see a lot of tests which use MiniDFSCluster
>>>>>>> e.g.
>>>>>>> cluster = new
>>>>>>> MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>>>>>>>
>>>>>>> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <
>>>>>>> karim.awara@kaust.edu.sa> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I read I can use MiniDFSCluster to set my own tests in case I
>>>>>>>> modify hadop source code. I have built hadoop 2.2. However, I can't find
>>>>>>>> any source on how to get the MiniDFSCluster working. Can someone point a
>>>>>>>> link to me that helps?
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best Regards,
>>>>>>>> Karim Ahmed Awara
>>>>>>>>
>>>>>>>> ------------------------------
>>>>>>>> This message and its contents, including attachments are intended
>>>>>>>> solely for the original recipient. If you are not the intended recipient or
>>>>>>>> have received this message in error, please notify me immediately and
>>>>>>>> delete this message from your computer system. Any unauthorized use or
>>>>>>>> distribution is prohibited. Please consider the environment before printing
>>>>>>>> this email.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ------------------------------
>>>>>> This message and its contents, including attachments are intended
>>>>>> solely for the original recipient. If you are not the intended recipient or
>>>>>> have received this message in error, please notify me immediately and
>>>>>> delete this message from your computer system. Any unauthorized use or
>>>>>> distribution is prohibited. Please consider the environment before printing
>>>>>> this email.
>>>>>>
>>>>>
>>>>>
>>>>
>>>> ------------------------------
>>>> This message and its contents, including attachments are intended
>>>> solely for the original recipient. If you are not the intended recipient or
>>>> have received this message in error, please notify me immediately and
>>>> delete this message from your computer system. Any unauthorized use or
>>>> distribution is prohibited. Please consider the environment before printing
>>>> this email.
>>>>
>>>
>>>
>>
>> ------------------------------
>> This message and its contents, including attachments are intended solely
>> for the original recipient. If you are not the intended recipient or have
>> received this message in error, please notify me immediately and delete
>> this message from your computer system. Any unauthorized use or
>> distribution is prohibited. Please consider the environment before printing
>> this email.
>>
>
>
--
------------------------------
This message and its contents, including attachments are intended solely
for the original recipient. If you are not the intended recipient or have
received this message in error, please notify me immediately and delete
this message from your computer system. Any unauthorized use or
distribution is prohibited. Please consider the environment before printing
this email.
Re: MiniDFSCluster setup
Posted by Karim Awara <ka...@kaust.edu.sa>.
I imported all the projects under the root dir "hadoop-2.2.0-src". And
still the same error. I havenot touched any conf files after I build with
maven. Is that correct? because it seems eclipse is not aware of some
sources that command-line maven is aware of.
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 7:10 PM, Ted Yu <yu...@gmail.com> wrote:
> Can you import from root of your workspace so that all sub-projects are
> imported ?
>
> Cheers
>
>
> On Sun, Dec 15, 2013 at 7:45 AM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
>>
>> Attached below. So What I did was after downloading hadoop src. I ran
>>
>> $ mvn install -DskipTests$ mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true
>>
>>
>> And imported both Hadoop Common and HDFS projects into eclipse.
>>
>>
>>
>>
>>
>>
>>
>>
>> java.lang.ExceptionInInitializerError
>> at
>> org.apache.hadoop.hdfs.server.common.Storage.getBuildVersion(Storage.java:1026)
>> at
>> org.apache.hadoop.hdfs.server.protocol.NamespaceInfo.<init>(NamespaceInfo.java:58)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:266)
>> at
>> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:122)
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:775)
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:642)
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:334)
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:316)
>> at
>> org.apache.hadoop.hdfs.TestWriteRead.initJunitModeTest(TestWriteRead.java:77)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
>> at
>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>> at
>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
>> at
>> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
>> at
>> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
>> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
>> at
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
>> at
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
>> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
>> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
>> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
>> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
>> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
>> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
>> at
>> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
>> at
>> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
>> Caused by: java.lang.NullPointerException
>> at java.util.Properties$LineReader.readLine(Properties.java:435)
>> at java.util.Properties.load0(Properties.java:354)
>> at java.util.Properties.load(Properties.java:342)
>> at org.apache.hadoop.util.VersionInfo.<init>(VersionInfo.java:51)
>> at org.apache.hadoop.util.VersionInfo.<clinit>(VersionInfo.java:97)
>> ... 36 more
>>
>> java.lang.NullPointerException
>> at
>> org.apache.hadoop.hdfs.TestWriteRead.shutdown(TestWriteRead.java:89)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
>> at
>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>> at
>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
>> at
>> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
>> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
>> at
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
>> at
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
>> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
>> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
>> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
>> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
>> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
>> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
>> at
>> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
>> at
>> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
>>
>>
>>
>>
>>
>> --
>> Best Regards,
>> Karim Ahmed Awara
>>
>>
>> On Sun, Dec 15, 2013 at 5:59 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> Can you show us the full stack trace ?
>>> In Eclipse, was there any project shown with a red bang or red cross ?
>>>
>>> Cheers
>>>
>>>
>>> On Sun, Dec 15, 2013 at 2:26 AM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>>
>>>>
>>>> It tells me java.lang.ExceptionInitializerError at
>>>> org.apache.....hdfs.server.common.Storage.getBuildVersion!
>>>>
>>>> I feel I am missing some parameters?
>>>>
>>>> --
>>>> Best Regards,
>>>> Karim Ahmed Awara
>>>>
>>>>
>>>> On Sun, Dec 15, 2013 at 5:08 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> You can use the following command to generate .project files for
>>>>> Eclipse (at the root of your workspace):
>>>>> mvn clean package -DskipTests eclipse:eclipse
>>>>>
>>>>> When you import hadoop, call sub-projects would be imported.
>>>>>
>>>>> I was able to run TestWriteRead in Eclipse successfully.
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <karim.awara@kaust.edu.sa
>>>>> > wrote:
>>>>>
>>>>>>
>>>>>> Running the test from Maven through commandline works fine. But I am
>>>>>> using eclipse. And it generates problem if I try to run the test as Junit,
>>>>>> as if eclipse is not aware of any of the conf parameters or args. Can
>>>>>> someone point to me a detailed source where it explains how to run Junit
>>>>>> through Eclipse for hadoop 2.2.x?
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards,
>>>>>> Karim Ahmed Awara
>>>>>>
>>>>>>
>>>>>> On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>
>>>>>>> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you
>>>>>>> would see a lot of tests which use MiniDFSCluster
>>>>>>> e.g.
>>>>>>> cluster = new
>>>>>>> MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>>>>>>>
>>>>>>> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <
>>>>>>> karim.awara@kaust.edu.sa> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I read I can use MiniDFSCluster to set my own tests in case I
>>>>>>>> modify hadop source code. I have built hadoop 2.2. However, I can't find
>>>>>>>> any source on how to get the MiniDFSCluster working. Can someone point a
>>>>>>>> link to me that helps?
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best Regards,
>>>>>>>> Karim Ahmed Awara
>>>>>>>>
>>>>>>>> ------------------------------
>>>>>>>> This message and its contents, including attachments are intended
>>>>>>>> solely for the original recipient. If you are not the intended recipient or
>>>>>>>> have received this message in error, please notify me immediately and
>>>>>>>> delete this message from your computer system. Any unauthorized use or
>>>>>>>> distribution is prohibited. Please consider the environment before printing
>>>>>>>> this email.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ------------------------------
>>>>>> This message and its contents, including attachments are intended
>>>>>> solely for the original recipient. If you are not the intended recipient or
>>>>>> have received this message in error, please notify me immediately and
>>>>>> delete this message from your computer system. Any unauthorized use or
>>>>>> distribution is prohibited. Please consider the environment before printing
>>>>>> this email.
>>>>>>
>>>>>
>>>>>
>>>>
>>>> ------------------------------
>>>> This message and its contents, including attachments are intended
>>>> solely for the original recipient. If you are not the intended recipient or
>>>> have received this message in error, please notify me immediately and
>>>> delete this message from your computer system. Any unauthorized use or
>>>> distribution is prohibited. Please consider the environment before printing
>>>> this email.
>>>>
>>>
>>>
>>
>> ------------------------------
>> This message and its contents, including attachments are intended solely
>> for the original recipient. If you are not the intended recipient or have
>> received this message in error, please notify me immediately and delete
>> this message from your computer system. Any unauthorized use or
>> distribution is prohibited. Please consider the environment before printing
>> this email.
>>
>
>
--
------------------------------
This message and its contents, including attachments are intended solely
for the original recipient. If you are not the intended recipient or have
received this message in error, please notify me immediately and delete
this message from your computer system. Any unauthorized use or
distribution is prohibited. Please consider the environment before printing
this email.
Re: MiniDFSCluster setup
Posted by Karim Awara <ka...@kaust.edu.sa>.
I imported all the projects under the root dir "hadoop-2.2.0-src". And
still the same error. I havenot touched any conf files after I build with
maven. Is that correct? because it seems eclipse is not aware of some
sources that command-line maven is aware of.
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 7:10 PM, Ted Yu <yu...@gmail.com> wrote:
> Can you import from root of your workspace so that all sub-projects are
> imported ?
>
> Cheers
>
>
> On Sun, Dec 15, 2013 at 7:45 AM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
>>
>> Attached below. So What I did was after downloading hadoop src. I ran
>>
>> $ mvn install -DskipTests$ mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true
>>
>>
>> And imported both Hadoop Common and HDFS projects into eclipse.
>>
>>
>>
>>
>>
>>
>>
>>
>> java.lang.ExceptionInInitializerError
>> at
>> org.apache.hadoop.hdfs.server.common.Storage.getBuildVersion(Storage.java:1026)
>> at
>> org.apache.hadoop.hdfs.server.protocol.NamespaceInfo.<init>(NamespaceInfo.java:58)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:266)
>> at
>> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:122)
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:775)
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:642)
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:334)
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:316)
>> at
>> org.apache.hadoop.hdfs.TestWriteRead.initJunitModeTest(TestWriteRead.java:77)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
>> at
>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>> at
>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
>> at
>> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
>> at
>> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
>> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
>> at
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
>> at
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
>> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
>> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
>> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
>> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
>> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
>> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
>> at
>> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
>> at
>> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
>> Caused by: java.lang.NullPointerException
>> at java.util.Properties$LineReader.readLine(Properties.java:435)
>> at java.util.Properties.load0(Properties.java:354)
>> at java.util.Properties.load(Properties.java:342)
>> at org.apache.hadoop.util.VersionInfo.<init>(VersionInfo.java:51)
>> at org.apache.hadoop.util.VersionInfo.<clinit>(VersionInfo.java:97)
>> ... 36 more
>>
>> java.lang.NullPointerException
>> at
>> org.apache.hadoop.hdfs.TestWriteRead.shutdown(TestWriteRead.java:89)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
>> at
>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>> at
>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
>> at
>> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
>> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
>> at
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
>> at
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
>> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
>> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
>> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
>> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
>> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
>> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
>> at
>> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
>> at
>> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
>>
>>
>>
>>
>>
>> --
>> Best Regards,
>> Karim Ahmed Awara
>>
>>
>> On Sun, Dec 15, 2013 at 5:59 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> Can you show us the full stack trace ?
>>> In Eclipse, was there any project shown with a red bang or red cross ?
>>>
>>> Cheers
>>>
>>>
>>> On Sun, Dec 15, 2013 at 2:26 AM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>>
>>>>
>>>> It tells me java.lang.ExceptionInitializerError at
>>>> org.apache.....hdfs.server.common.Storage.getBuildVersion!
>>>>
>>>> I feel I am missing some parameters?
>>>>
>>>> --
>>>> Best Regards,
>>>> Karim Ahmed Awara
>>>>
>>>>
>>>> On Sun, Dec 15, 2013 at 5:08 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> You can use the following command to generate .project files for
>>>>> Eclipse (at the root of your workspace):
>>>>> mvn clean package -DskipTests eclipse:eclipse
>>>>>
>>>>> When you import hadoop, call sub-projects would be imported.
>>>>>
>>>>> I was able to run TestWriteRead in Eclipse successfully.
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <karim.awara@kaust.edu.sa
>>>>> > wrote:
>>>>>
>>>>>>
>>>>>> Running the test from Maven through commandline works fine. But I am
>>>>>> using eclipse. And it generates problem if I try to run the test as Junit,
>>>>>> as if eclipse is not aware of any of the conf parameters or args. Can
>>>>>> someone point to me a detailed source where it explains how to run Junit
>>>>>> through Eclipse for hadoop 2.2.x?
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards,
>>>>>> Karim Ahmed Awara
>>>>>>
>>>>>>
>>>>>> On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>
>>>>>>> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you
>>>>>>> would see a lot of tests which use MiniDFSCluster
>>>>>>> e.g.
>>>>>>> cluster = new
>>>>>>> MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>>>>>>>
>>>>>>> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <
>>>>>>> karim.awara@kaust.edu.sa> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I read I can use MiniDFSCluster to set my own tests in case I
>>>>>>>> modify hadop source code. I have built hadoop 2.2. However, I can't find
>>>>>>>> any source on how to get the MiniDFSCluster working. Can someone point a
>>>>>>>> link to me that helps?
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best Regards,
>>>>>>>> Karim Ahmed Awara
>>>>>>>>
>>>>>>>> ------------------------------
>>>>>>>> This message and its contents, including attachments are intended
>>>>>>>> solely for the original recipient. If you are not the intended recipient or
>>>>>>>> have received this message in error, please notify me immediately and
>>>>>>>> delete this message from your computer system. Any unauthorized use or
>>>>>>>> distribution is prohibited. Please consider the environment before printing
>>>>>>>> this email.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ------------------------------
>>>>>> This message and its contents, including attachments are intended
>>>>>> solely for the original recipient. If you are not the intended recipient or
>>>>>> have received this message in error, please notify me immediately and
>>>>>> delete this message from your computer system. Any unauthorized use or
>>>>>> distribution is prohibited. Please consider the environment before printing
>>>>>> this email.
>>>>>>
>>>>>
>>>>>
>>>>
>>>> ------------------------------
>>>> This message and its contents, including attachments are intended
>>>> solely for the original recipient. If you are not the intended recipient or
>>>> have received this message in error, please notify me immediately and
>>>> delete this message from your computer system. Any unauthorized use or
>>>> distribution is prohibited. Please consider the environment before printing
>>>> this email.
>>>>
>>>
>>>
>>
>> ------------------------------
>> This message and its contents, including attachments are intended solely
>> for the original recipient. If you are not the intended recipient or have
>> received this message in error, please notify me immediately and delete
>> this message from your computer system. Any unauthorized use or
>> distribution is prohibited. Please consider the environment before printing
>> this email.
>>
>
>
--
------------------------------
This message and its contents, including attachments are intended solely
for the original recipient. If you are not the intended recipient or have
received this message in error, please notify me immediately and delete
this message from your computer system. Any unauthorized use or
distribution is prohibited. Please consider the environment before printing
this email.
Re: MiniDFSCluster setup
Posted by Karim Awara <ka...@kaust.edu.sa>.
I imported all the projects under the root dir "hadoop-2.2.0-src". And
still the same error. I havenot touched any conf files after I build with
maven. Is that correct? because it seems eclipse is not aware of some
sources that command-line maven is aware of.
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 7:10 PM, Ted Yu <yu...@gmail.com> wrote:
> Can you import from root of your workspace so that all sub-projects are
> imported ?
>
> Cheers
>
>
> On Sun, Dec 15, 2013 at 7:45 AM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
>>
>> Attached below. So What I did was after downloading hadoop src. I ran
>>
>> $ mvn install -DskipTests$ mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true
>>
>>
>> And imported both Hadoop Common and HDFS projects into eclipse.
>>
>>
>>
>>
>>
>>
>>
>>
>> java.lang.ExceptionInInitializerError
>> at
>> org.apache.hadoop.hdfs.server.common.Storage.getBuildVersion(Storage.java:1026)
>> at
>> org.apache.hadoop.hdfs.server.protocol.NamespaceInfo.<init>(NamespaceInfo.java:58)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:266)
>> at
>> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:122)
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:775)
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:642)
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:334)
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:316)
>> at
>> org.apache.hadoop.hdfs.TestWriteRead.initJunitModeTest(TestWriteRead.java:77)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
>> at
>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>> at
>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
>> at
>> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
>> at
>> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
>> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
>> at
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
>> at
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
>> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
>> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
>> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
>> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
>> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
>> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
>> at
>> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
>> at
>> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
>> Caused by: java.lang.NullPointerException
>> at java.util.Properties$LineReader.readLine(Properties.java:435)
>> at java.util.Properties.load0(Properties.java:354)
>> at java.util.Properties.load(Properties.java:342)
>> at org.apache.hadoop.util.VersionInfo.<init>(VersionInfo.java:51)
>> at org.apache.hadoop.util.VersionInfo.<clinit>(VersionInfo.java:97)
>> ... 36 more
>>
>> java.lang.NullPointerException
>> at
>> org.apache.hadoop.hdfs.TestWriteRead.shutdown(TestWriteRead.java:89)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
>> at
>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>> at
>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
>> at
>> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
>> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
>> at
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
>> at
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
>> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
>> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
>> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
>> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
>> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
>> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
>> at
>> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
>> at
>> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
>>
>>
>>
>>
>>
>> --
>> Best Regards,
>> Karim Ahmed Awara
>>
>>
>> On Sun, Dec 15, 2013 at 5:59 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> Can you show us the full stack trace ?
>>> In Eclipse, was there any project shown with a red bang or red cross ?
>>>
>>> Cheers
>>>
>>>
>>> On Sun, Dec 15, 2013 at 2:26 AM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>>
>>>>
>>>> It tells me java.lang.ExceptionInitializerError at
>>>> org.apache.....hdfs.server.common.Storage.getBuildVersion!
>>>>
>>>> I feel I am missing some parameters?
>>>>
>>>> --
>>>> Best Regards,
>>>> Karim Ahmed Awara
>>>>
>>>>
>>>> On Sun, Dec 15, 2013 at 5:08 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> You can use the following command to generate .project files for
>>>>> Eclipse (at the root of your workspace):
>>>>> mvn clean package -DskipTests eclipse:eclipse
>>>>>
>>>>> When you import hadoop, call sub-projects would be imported.
>>>>>
>>>>> I was able to run TestWriteRead in Eclipse successfully.
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <karim.awara@kaust.edu.sa
>>>>> > wrote:
>>>>>
>>>>>>
>>>>>> Running the test from Maven through commandline works fine. But I am
>>>>>> using eclipse. And it generates problem if I try to run the test as Junit,
>>>>>> as if eclipse is not aware of any of the conf parameters or args. Can
>>>>>> someone point to me a detailed source where it explains how to run Junit
>>>>>> through Eclipse for hadoop 2.2.x?
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards,
>>>>>> Karim Ahmed Awara
>>>>>>
>>>>>>
>>>>>> On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>
>>>>>>> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you
>>>>>>> would see a lot of tests which use MiniDFSCluster
>>>>>>> e.g.
>>>>>>> cluster = new
>>>>>>> MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>>>>>>>
>>>>>>> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <
>>>>>>> karim.awara@kaust.edu.sa> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I read I can use MiniDFSCluster to set my own tests in case I
>>>>>>>> modify hadop source code. I have built hadoop 2.2. However, I can't find
>>>>>>>> any source on how to get the MiniDFSCluster working. Can someone point a
>>>>>>>> link to me that helps?
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best Regards,
>>>>>>>> Karim Ahmed Awara
>>>>>>>>
>>>>>>>> ------------------------------
>>>>>>>> This message and its contents, including attachments are intended
>>>>>>>> solely for the original recipient. If you are not the intended recipient or
>>>>>>>> have received this message in error, please notify me immediately and
>>>>>>>> delete this message from your computer system. Any unauthorized use or
>>>>>>>> distribution is prohibited. Please consider the environment before printing
>>>>>>>> this email.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ------------------------------
>>>>>> This message and its contents, including attachments are intended
>>>>>> solely for the original recipient. If you are not the intended recipient or
>>>>>> have received this message in error, please notify me immediately and
>>>>>> delete this message from your computer system. Any unauthorized use or
>>>>>> distribution is prohibited. Please consider the environment before printing
>>>>>> this email.
>>>>>>
>>>>>
>>>>>
>>>>
>>>> ------------------------------
>>>> This message and its contents, including attachments are intended
>>>> solely for the original recipient. If you are not the intended recipient or
>>>> have received this message in error, please notify me immediately and
>>>> delete this message from your computer system. Any unauthorized use or
>>>> distribution is prohibited. Please consider the environment before printing
>>>> this email.
>>>>
>>>
>>>
>>
>> ------------------------------
>> This message and its contents, including attachments are intended solely
>> for the original recipient. If you are not the intended recipient or have
>> received this message in error, please notify me immediately and delete
>> this message from your computer system. Any unauthorized use or
>> distribution is prohibited. Please consider the environment before printing
>> this email.
>>
>
>
--
------------------------------
This message and its contents, including attachments are intended solely
for the original recipient. If you are not the intended recipient or have
received this message in error, please notify me immediately and delete
this message from your computer system. Any unauthorized use or
distribution is prohibited. Please consider the environment before printing
this email.
Re: MiniDFSCluster setup
Posted by Ted Yu <yu...@gmail.com>.
Can you import from root of your workspace so that all sub-projects are
imported ?
Cheers
On Sun, Dec 15, 2013 at 7:45 AM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
> Attached below. So What I did was after downloading hadoop src. I ran
>
> $ mvn install -DskipTests$ mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true
>
>
> And imported both Hadoop Common and HDFS projects into eclipse.
>
>
>
>
>
>
>
>
> java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hdfs.server.common.Storage.getBuildVersion(Storage.java:1026)
> at
> org.apache.hadoop.hdfs.server.protocol.NamespaceInfo.<init>(NamespaceInfo.java:58)
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:266)
> at
> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:122)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:775)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:642)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:334)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:316)
> at
> org.apache.hadoop.hdfs.TestWriteRead.initJunitModeTest(TestWriteRead.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> Caused by: java.lang.NullPointerException
> at java.util.Properties$LineReader.readLine(Properties.java:435)
> at java.util.Properties.load0(Properties.java:354)
> at java.util.Properties.load(Properties.java:342)
> at org.apache.hadoop.util.VersionInfo.<init>(VersionInfo.java:51)
> at org.apache.hadoop.util.VersionInfo.<clinit>(VersionInfo.java:97)
> ... 36 more
>
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.TestWriteRead.shutdown(TestWriteRead.java:89)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
>
>
>
>
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
> On Sun, Dec 15, 2013 at 5:59 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> Can you show us the full stack trace ?
>> In Eclipse, was there any project shown with a red bang or red cross ?
>>
>> Cheers
>>
>>
>> On Sun, Dec 15, 2013 at 2:26 AM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>
>>>
>>> It tells me java.lang.ExceptionInitializerError at
>>> org.apache.....hdfs.server.common.Storage.getBuildVersion!
>>>
>>> I feel I am missing some parameters?
>>>
>>> --
>>> Best Regards,
>>> Karim Ahmed Awara
>>>
>>>
>>> On Sun, Dec 15, 2013 at 5:08 AM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> You can use the following command to generate .project files for
>>>> Eclipse (at the root of your workspace):
>>>> mvn clean package -DskipTests eclipse:eclipse
>>>>
>>>> When you import hadoop, call sub-projects would be imported.
>>>>
>>>> I was able to run TestWriteRead in Eclipse successfully.
>>>>
>>>> Cheers
>>>>
>>>>
>>>> On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>>>
>>>>>
>>>>> Running the test from Maven through commandline works fine. But I am
>>>>> using eclipse. And it generates problem if I try to run the test as Junit,
>>>>> as if eclipse is not aware of any of the conf parameters or args. Can
>>>>> someone point to me a detailed source where it explains how to run Junit
>>>>> through Eclipse for hadoop 2.2.x?
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Karim Ahmed Awara
>>>>>
>>>>>
>>>>> On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>>
>>>>>> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you
>>>>>> would see a lot of tests which use MiniDFSCluster
>>>>>> e.g.
>>>>>> cluster = new
>>>>>> MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>>>>>>
>>>>>> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>>
>>>>>> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <
>>>>>> karim.awara@kaust.edu.sa> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I read I can use MiniDFSCluster to set my own tests in case I modify
>>>>>>> hadop source code. I have built hadoop 2.2. However, I can't find any
>>>>>>> source on how to get the MiniDFSCluster working. Can someone point a link
>>>>>>> to me that helps?
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards,
>>>>>>> Karim Ahmed Awara
>>>>>>>
>>>>>>> ------------------------------
>>>>>>> This message and its contents, including attachments are intended
>>>>>>> solely for the original recipient. If you are not the intended recipient or
>>>>>>> have received this message in error, please notify me immediately and
>>>>>>> delete this message from your computer system. Any unauthorized use or
>>>>>>> distribution is prohibited. Please consider the environment before printing
>>>>>>> this email.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> ------------------------------
>>>>> This message and its contents, including attachments are intended
>>>>> solely for the original recipient. If you are not the intended recipient or
>>>>> have received this message in error, please notify me immediately and
>>>>> delete this message from your computer system. Any unauthorized use or
>>>>> distribution is prohibited. Please consider the environment before printing
>>>>> this email.
>>>>>
>>>>
>>>>
>>>
>>> ------------------------------
>>> This message and its contents, including attachments are intended solely
>>> for the original recipient. If you are not the intended recipient or have
>>> received this message in error, please notify me immediately and delete
>>> this message from your computer system. Any unauthorized use or
>>> distribution is prohibited. Please consider the environment before printing
>>> this email.
>>>
>>
>>
>
> ------------------------------
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
Re: MiniDFSCluster setup
Posted by Ted Yu <yu...@gmail.com>.
Can you import from root of your workspace so that all sub-projects are
imported ?
Cheers
On Sun, Dec 15, 2013 at 7:45 AM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
> Attached below. So What I did was after downloading hadoop src. I ran
>
> $ mvn install -DskipTests$ mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true
>
>
> And imported both Hadoop Common and HDFS projects into eclipse.
>
>
>
>
>
>
>
>
> java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hdfs.server.common.Storage.getBuildVersion(Storage.java:1026)
> at
> org.apache.hadoop.hdfs.server.protocol.NamespaceInfo.<init>(NamespaceInfo.java:58)
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:266)
> at
> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:122)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:775)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:642)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:334)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:316)
> at
> org.apache.hadoop.hdfs.TestWriteRead.initJunitModeTest(TestWriteRead.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> Caused by: java.lang.NullPointerException
> at java.util.Properties$LineReader.readLine(Properties.java:435)
> at java.util.Properties.load0(Properties.java:354)
> at java.util.Properties.load(Properties.java:342)
> at org.apache.hadoop.util.VersionInfo.<init>(VersionInfo.java:51)
> at org.apache.hadoop.util.VersionInfo.<clinit>(VersionInfo.java:97)
> ... 36 more
>
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.TestWriteRead.shutdown(TestWriteRead.java:89)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
>
>
>
>
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
> On Sun, Dec 15, 2013 at 5:59 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> Can you show us the full stack trace ?
>> In Eclipse, was there any project shown with a red bang or red cross ?
>>
>> Cheers
>>
>>
>> On Sun, Dec 15, 2013 at 2:26 AM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>
>>>
>>> It tells me java.lang.ExceptionInitializerError at
>>> org.apache.....hdfs.server.common.Storage.getBuildVersion!
>>>
>>> I feel I am missing some parameters?
>>>
>>> --
>>> Best Regards,
>>> Karim Ahmed Awara
>>>
>>>
>>> On Sun, Dec 15, 2013 at 5:08 AM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> You can use the following command to generate .project files for
>>>> Eclipse (at the root of your workspace):
>>>> mvn clean package -DskipTests eclipse:eclipse
>>>>
>>>> When you import hadoop, call sub-projects would be imported.
>>>>
>>>> I was able to run TestWriteRead in Eclipse successfully.
>>>>
>>>> Cheers
>>>>
>>>>
>>>> On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>>>
>>>>>
>>>>> Running the test from Maven through commandline works fine. But I am
>>>>> using eclipse. And it generates problem if I try to run the test as Junit,
>>>>> as if eclipse is not aware of any of the conf parameters or args. Can
>>>>> someone point to me a detailed source where it explains how to run Junit
>>>>> through Eclipse for hadoop 2.2.x?
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Karim Ahmed Awara
>>>>>
>>>>>
>>>>> On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>>
>>>>>> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you
>>>>>> would see a lot of tests which use MiniDFSCluster
>>>>>> e.g.
>>>>>> cluster = new
>>>>>> MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>>>>>>
>>>>>> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>>
>>>>>> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <
>>>>>> karim.awara@kaust.edu.sa> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I read I can use MiniDFSCluster to set my own tests in case I modify
>>>>>>> hadop source code. I have built hadoop 2.2. However, I can't find any
>>>>>>> source on how to get the MiniDFSCluster working. Can someone point a link
>>>>>>> to me that helps?
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards,
>>>>>>> Karim Ahmed Awara
>>>>>>>
>>>>>>> ------------------------------
>>>>>>> This message and its contents, including attachments are intended
>>>>>>> solely for the original recipient. If you are not the intended recipient or
>>>>>>> have received this message in error, please notify me immediately and
>>>>>>> delete this message from your computer system. Any unauthorized use or
>>>>>>> distribution is prohibited. Please consider the environment before printing
>>>>>>> this email.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> ------------------------------
>>>>> This message and its contents, including attachments are intended
>>>>> solely for the original recipient. If you are not the intended recipient or
>>>>> have received this message in error, please notify me immediately and
>>>>> delete this message from your computer system. Any unauthorized use or
>>>>> distribution is prohibited. Please consider the environment before printing
>>>>> this email.
>>>>>
>>>>
>>>>
>>>
>>> ------------------------------
>>> This message and its contents, including attachments are intended solely
>>> for the original recipient. If you are not the intended recipient or have
>>> received this message in error, please notify me immediately and delete
>>> this message from your computer system. Any unauthorized use or
>>> distribution is prohibited. Please consider the environment before printing
>>> this email.
>>>
>>
>>
>
> ------------------------------
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
Re: MiniDFSCluster setup
Posted by Ted Yu <yu...@gmail.com>.
Can you import from root of your workspace so that all sub-projects are
imported ?
Cheers
On Sun, Dec 15, 2013 at 7:45 AM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
> Attached below. So What I did was after downloading hadoop src. I ran
>
> $ mvn install -DskipTests$ mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true
>
>
> And imported both Hadoop Common and HDFS projects into eclipse.
>
>
>
>
>
>
>
>
> java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hdfs.server.common.Storage.getBuildVersion(Storage.java:1026)
> at
> org.apache.hadoop.hdfs.server.protocol.NamespaceInfo.<init>(NamespaceInfo.java:58)
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:266)
> at
> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:122)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:775)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:642)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:334)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:316)
> at
> org.apache.hadoop.hdfs.TestWriteRead.initJunitModeTest(TestWriteRead.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> Caused by: java.lang.NullPointerException
> at java.util.Properties$LineReader.readLine(Properties.java:435)
> at java.util.Properties.load0(Properties.java:354)
> at java.util.Properties.load(Properties.java:342)
> at org.apache.hadoop.util.VersionInfo.<init>(VersionInfo.java:51)
> at org.apache.hadoop.util.VersionInfo.<clinit>(VersionInfo.java:97)
> ... 36 more
>
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.TestWriteRead.shutdown(TestWriteRead.java:89)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
>
>
>
>
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
> On Sun, Dec 15, 2013 at 5:59 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> Can you show us the full stack trace ?
>> In Eclipse, was there any project shown with a red bang or red cross ?
>>
>> Cheers
>>
>>
>> On Sun, Dec 15, 2013 at 2:26 AM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>
>>>
>>> It tells me java.lang.ExceptionInitializerError at
>>> org.apache.....hdfs.server.common.Storage.getBuildVersion!
>>>
>>> I feel I am missing some parameters?
>>>
>>> --
>>> Best Regards,
>>> Karim Ahmed Awara
>>>
>>>
>>> On Sun, Dec 15, 2013 at 5:08 AM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> You can use the following command to generate .project files for
>>>> Eclipse (at the root of your workspace):
>>>> mvn clean package -DskipTests eclipse:eclipse
>>>>
>>>> When you import hadoop, call sub-projects would be imported.
>>>>
>>>> I was able to run TestWriteRead in Eclipse successfully.
>>>>
>>>> Cheers
>>>>
>>>>
>>>> On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>>>
>>>>>
>>>>> Running the test from Maven through commandline works fine. But I am
>>>>> using eclipse. And it generates problem if I try to run the test as Junit,
>>>>> as if eclipse is not aware of any of the conf parameters or args. Can
>>>>> someone point to me a detailed source where it explains how to run Junit
>>>>> through Eclipse for hadoop 2.2.x?
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Karim Ahmed Awara
>>>>>
>>>>>
>>>>> On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>>
>>>>>> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you
>>>>>> would see a lot of tests which use MiniDFSCluster
>>>>>> e.g.
>>>>>> cluster = new
>>>>>> MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>>>>>>
>>>>>> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>>
>>>>>> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <
>>>>>> karim.awara@kaust.edu.sa> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I read I can use MiniDFSCluster to set my own tests in case I modify
>>>>>>> hadop source code. I have built hadoop 2.2. However, I can't find any
>>>>>>> source on how to get the MiniDFSCluster working. Can someone point a link
>>>>>>> to me that helps?
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards,
>>>>>>> Karim Ahmed Awara
>>>>>>>
>>>>>>> ------------------------------
>>>>>>> This message and its contents, including attachments are intended
>>>>>>> solely for the original recipient. If you are not the intended recipient or
>>>>>>> have received this message in error, please notify me immediately and
>>>>>>> delete this message from your computer system. Any unauthorized use or
>>>>>>> distribution is prohibited. Please consider the environment before printing
>>>>>>> this email.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> ------------------------------
>>>>> This message and its contents, including attachments are intended
>>>>> solely for the original recipient. If you are not the intended recipient or
>>>>> have received this message in error, please notify me immediately and
>>>>> delete this message from your computer system. Any unauthorized use or
>>>>> distribution is prohibited. Please consider the environment before printing
>>>>> this email.
>>>>>
>>>>
>>>>
>>>
>>> ------------------------------
>>> This message and its contents, including attachments are intended solely
>>> for the original recipient. If you are not the intended recipient or have
>>> received this message in error, please notify me immediately and delete
>>> this message from your computer system. Any unauthorized use or
>>> distribution is prohibited. Please consider the environment before printing
>>> this email.
>>>
>>
>>
>
> ------------------------------
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
Re: MiniDFSCluster setup
Posted by Ted Yu <yu...@gmail.com>.
Can you import from root of your workspace so that all sub-projects are
imported ?
Cheers
On Sun, Dec 15, 2013 at 7:45 AM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
> Attached below. So What I did was after downloading hadoop src. I ran
>
> $ mvn install -DskipTests$ mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true
>
>
> And imported both Hadoop Common and HDFS projects into eclipse.
>
>
>
>
>
>
>
>
> java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hdfs.server.common.Storage.getBuildVersion(Storage.java:1026)
> at
> org.apache.hadoop.hdfs.server.protocol.NamespaceInfo.<init>(NamespaceInfo.java:58)
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:266)
> at
> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:122)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:775)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:642)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:334)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:316)
> at
> org.apache.hadoop.hdfs.TestWriteRead.initJunitModeTest(TestWriteRead.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> Caused by: java.lang.NullPointerException
> at java.util.Properties$LineReader.readLine(Properties.java:435)
> at java.util.Properties.load0(Properties.java:354)
> at java.util.Properties.load(Properties.java:342)
> at org.apache.hadoop.util.VersionInfo.<init>(VersionInfo.java:51)
> at org.apache.hadoop.util.VersionInfo.<clinit>(VersionInfo.java:97)
> ... 36 more
>
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.TestWriteRead.shutdown(TestWriteRead.java:89)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
>
>
>
>
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
> On Sun, Dec 15, 2013 at 5:59 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> Can you show us the full stack trace ?
>> In Eclipse, was there any project shown with a red bang or red cross ?
>>
>> Cheers
>>
>>
>> On Sun, Dec 15, 2013 at 2:26 AM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>
>>>
>>> It tells me java.lang.ExceptionInitializerError at
>>> org.apache.....hdfs.server.common.Storage.getBuildVersion!
>>>
>>> I feel I am missing some parameters?
>>>
>>> --
>>> Best Regards,
>>> Karim Ahmed Awara
>>>
>>>
>>> On Sun, Dec 15, 2013 at 5:08 AM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> You can use the following command to generate .project files for
>>>> Eclipse (at the root of your workspace):
>>>> mvn clean package -DskipTests eclipse:eclipse
>>>>
>>>> When you import hadoop, call sub-projects would be imported.
>>>>
>>>> I was able to run TestWriteRead in Eclipse successfully.
>>>>
>>>> Cheers
>>>>
>>>>
>>>> On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>>>
>>>>>
>>>>> Running the test from Maven through commandline works fine. But I am
>>>>> using eclipse. And it generates problem if I try to run the test as Junit,
>>>>> as if eclipse is not aware of any of the conf parameters or args. Can
>>>>> someone point to me a detailed source where it explains how to run Junit
>>>>> through Eclipse for hadoop 2.2.x?
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Karim Ahmed Awara
>>>>>
>>>>>
>>>>> On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>>
>>>>>> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you
>>>>>> would see a lot of tests which use MiniDFSCluster
>>>>>> e.g.
>>>>>> cluster = new
>>>>>> MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>>>>>>
>>>>>> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>>
>>>>>> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <
>>>>>> karim.awara@kaust.edu.sa> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I read I can use MiniDFSCluster to set my own tests in case I modify
>>>>>>> hadop source code. I have built hadoop 2.2. However, I can't find any
>>>>>>> source on how to get the MiniDFSCluster working. Can someone point a link
>>>>>>> to me that helps?
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards,
>>>>>>> Karim Ahmed Awara
>>>>>>>
>>>>>>> ------------------------------
>>>>>>> This message and its contents, including attachments are intended
>>>>>>> solely for the original recipient. If you are not the intended recipient or
>>>>>>> have received this message in error, please notify me immediately and
>>>>>>> delete this message from your computer system. Any unauthorized use or
>>>>>>> distribution is prohibited. Please consider the environment before printing
>>>>>>> this email.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> ------------------------------
>>>>> This message and its contents, including attachments are intended
>>>>> solely for the original recipient. If you are not the intended recipient or
>>>>> have received this message in error, please notify me immediately and
>>>>> delete this message from your computer system. Any unauthorized use or
>>>>> distribution is prohibited. Please consider the environment before printing
>>>>> this email.
>>>>>
>>>>
>>>>
>>>
>>> ------------------------------
>>> This message and its contents, including attachments are intended solely
>>> for the original recipient. If you are not the intended recipient or have
>>> received this message in error, please notify me immediately and delete
>>> this message from your computer system. Any unauthorized use or
>>> distribution is prohibited. Please consider the environment before printing
>>> this email.
>>>
>>
>>
>
> ------------------------------
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
Re: MiniDFSCluster setup
Posted by Karim Awara <ka...@kaust.edu.sa>.
Attached below. So What I did was after downloading hadoop src. I ran
$ mvn install -DskipTests$ mvn eclipse:eclipse -DdownloadSources=true
-DdownloadJavadocs=true
And imported both Hadoop Common and HDFS projects into eclipse.
java.lang.ExceptionInInitializerError
at
org.apache.hadoop.hdfs.server.common.Storage.getBuildVersion(Storage.java:1026)
at
org.apache.hadoop.hdfs.server.protocol.NamespaceInfo.<init>(NamespaceInfo.java:58)
at
org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:266)
at
org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:122)
at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:775)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:642)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:334)
at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:316)
at
org.apache.hadoop.hdfs.TestWriteRead.initJunitModeTest(TestWriteRead.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Caused by: java.lang.NullPointerException
at java.util.Properties$LineReader.readLine(Properties.java:435)
at java.util.Properties.load0(Properties.java:354)
at java.util.Properties.load(Properties.java:342)
at org.apache.hadoop.util.VersionInfo.<init>(VersionInfo.java:51)
at org.apache.hadoop.util.VersionInfo.<clinit>(VersionInfo.java:97)
... 36 more
java.lang.NullPointerException
at org.apache.hadoop.hdfs.TestWriteRead.shutdown(TestWriteRead.java:89)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 5:59 PM, Ted Yu <yu...@gmail.com> wrote:
> Can you show us the full stack trace ?
> In Eclipse, was there any project shown with a red bang or red cross ?
>
> Cheers
>
>
> On Sun, Dec 15, 2013 at 2:26 AM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
>>
>> It tells me java.lang.ExceptionInitializerError at
>> org.apache.....hdfs.server.common.Storage.getBuildVersion!
>>
>> I feel I am missing some parameters?
>>
>> --
>> Best Regards,
>> Karim Ahmed Awara
>>
>>
>> On Sun, Dec 15, 2013 at 5:08 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> You can use the following command to generate .project files for Eclipse
>>> (at the root of your workspace):
>>> mvn clean package -DskipTests eclipse:eclipse
>>>
>>> When you import hadoop, call sub-projects would be imported.
>>>
>>> I was able to run TestWriteRead in Eclipse successfully.
>>>
>>> Cheers
>>>
>>>
>>> On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>>
>>>>
>>>> Running the test from Maven through commandline works fine. But I am
>>>> using eclipse. And it generates problem if I try to run the test as Junit,
>>>> as if eclipse is not aware of any of the conf parameters or args. Can
>>>> someone point to me a detailed source where it explains how to run Junit
>>>> through Eclipse for hadoop 2.2.x?
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>> Karim Ahmed Awara
>>>>
>>>>
>>>> On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you
>>>>> would see a lot of tests which use MiniDFSCluster
>>>>> e.g.
>>>>> cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>>>>>
>>>>> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <karim.awara@kaust.edu.sa
>>>>> > wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I read I can use MiniDFSCluster to set my own tests in case I modify
>>>>>> hadop source code. I have built hadoop 2.2. However, I can't find any
>>>>>> source on how to get the MiniDFSCluster working. Can someone point a link
>>>>>> to me that helps?
>>>>>>
>>>>>> --
>>>>>> Best Regards,
>>>>>> Karim Ahmed Awara
>>>>>>
>>>>>> ------------------------------
>>>>>> This message and its contents, including attachments are intended
>>>>>> solely for the original recipient. If you are not the intended recipient or
>>>>>> have received this message in error, please notify me immediately and
>>>>>> delete this message from your computer system. Any unauthorized use or
>>>>>> distribution is prohibited. Please consider the environment before printing
>>>>>> this email.
>>>>>
>>>>>
>>>>>
>>>>
>>>> ------------------------------
>>>> This message and its contents, including attachments are intended
>>>> solely for the original recipient. If you are not the intended recipient or
>>>> have received this message in error, please notify me immediately and
>>>> delete this message from your computer system. Any unauthorized use or
>>>> distribution is prohibited. Please consider the environment before printing
>>>> this email.
>>>>
>>>
>>>
>>
>> ------------------------------
>> This message and its contents, including attachments are intended solely
>> for the original recipient. If you are not the intended recipient or have
>> received this message in error, please notify me immediately and delete
>> this message from your computer system. Any unauthorized use or
>> distribution is prohibited. Please consider the environment before printing
>> this email.
>>
>
>
--
------------------------------
This message and its contents, including attachments are intended solely
for the original recipient. If you are not the intended recipient or have
received this message in error, please notify me immediately and delete
this message from your computer system. Any unauthorized use or
distribution is prohibited. Please consider the environment before printing
this email.
Re: MiniDFSCluster setup
Posted by Karim Awara <ka...@kaust.edu.sa>.
Attached below. So What I did was after downloading hadoop src. I ran
$ mvn install -DskipTests$ mvn eclipse:eclipse -DdownloadSources=true
-DdownloadJavadocs=true
And imported both Hadoop Common and HDFS projects into eclipse.
java.lang.ExceptionInInitializerError
at
org.apache.hadoop.hdfs.server.common.Storage.getBuildVersion(Storage.java:1026)
at
org.apache.hadoop.hdfs.server.protocol.NamespaceInfo.<init>(NamespaceInfo.java:58)
at
org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:266)
at
org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:122)
at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:775)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:642)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:334)
at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:316)
at
org.apache.hadoop.hdfs.TestWriteRead.initJunitModeTest(TestWriteRead.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Caused by: java.lang.NullPointerException
at java.util.Properties$LineReader.readLine(Properties.java:435)
at java.util.Properties.load0(Properties.java:354)
at java.util.Properties.load(Properties.java:342)
at org.apache.hadoop.util.VersionInfo.<init>(VersionInfo.java:51)
at org.apache.hadoop.util.VersionInfo.<clinit>(VersionInfo.java:97)
... 36 more
java.lang.NullPointerException
at org.apache.hadoop.hdfs.TestWriteRead.shutdown(TestWriteRead.java:89)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 5:59 PM, Ted Yu <yu...@gmail.com> wrote:
> Can you show us the full stack trace ?
> In Eclipse, was there any project shown with a red bang or red cross ?
>
> Cheers
>
>
> On Sun, Dec 15, 2013 at 2:26 AM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
>>
>> It tells me java.lang.ExceptionInitializerError at
>> org.apache.....hdfs.server.common.Storage.getBuildVersion!
>>
>> I feel I am missing some parameters?
>>
>> --
>> Best Regards,
>> Karim Ahmed Awara
>>
>>
>> On Sun, Dec 15, 2013 at 5:08 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> You can use the following command to generate .project files for Eclipse
>>> (at the root of your workspace):
>>> mvn clean package -DskipTests eclipse:eclipse
>>>
>>> When you import hadoop, call sub-projects would be imported.
>>>
>>> I was able to run TestWriteRead in Eclipse successfully.
>>>
>>> Cheers
>>>
>>>
>>> On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>>
>>>>
>>>> Running the test from Maven through commandline works fine. But I am
>>>> using eclipse. And it generates problem if I try to run the test as Junit,
>>>> as if eclipse is not aware of any of the conf parameters or args. Can
>>>> someone point to me a detailed source where it explains how to run Junit
>>>> through Eclipse for hadoop 2.2.x?
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>> Karim Ahmed Awara
>>>>
>>>>
>>>> On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you
>>>>> would see a lot of tests which use MiniDFSCluster
>>>>> e.g.
>>>>> cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>>>>>
>>>>> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <karim.awara@kaust.edu.sa
>>>>> > wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I read I can use MiniDFSCluster to set my own tests in case I modify
>>>>>> hadop source code. I have built hadoop 2.2. However, I can't find any
>>>>>> source on how to get the MiniDFSCluster working. Can someone point a link
>>>>>> to me that helps?
>>>>>>
>>>>>> --
>>>>>> Best Regards,
>>>>>> Karim Ahmed Awara
>>>>>>
>>>>>> ------------------------------
>>>>>> This message and its contents, including attachments are intended
>>>>>> solely for the original recipient. If you are not the intended recipient or
>>>>>> have received this message in error, please notify me immediately and
>>>>>> delete this message from your computer system. Any unauthorized use or
>>>>>> distribution is prohibited. Please consider the environment before printing
>>>>>> this email.
>>>>>
>>>>>
>>>>>
>>>>
>>>> ------------------------------
>>>> This message and its contents, including attachments are intended
>>>> solely for the original recipient. If you are not the intended recipient or
>>>> have received this message in error, please notify me immediately and
>>>> delete this message from your computer system. Any unauthorized use or
>>>> distribution is prohibited. Please consider the environment before printing
>>>> this email.
>>>>
>>>
>>>
>>
>> ------------------------------
>> This message and its contents, including attachments are intended solely
>> for the original recipient. If you are not the intended recipient or have
>> received this message in error, please notify me immediately and delete
>> this message from your computer system. Any unauthorized use or
>> distribution is prohibited. Please consider the environment before printing
>> this email.
>>
>
>
--
------------------------------
This message and its contents, including attachments are intended solely
for the original recipient. If you are not the intended recipient or have
received this message in error, please notify me immediately and delete
this message from your computer system. Any unauthorized use or
distribution is prohibited. Please consider the environment before printing
this email.
Re: MiniDFSCluster setup
Posted by Karim Awara <ka...@kaust.edu.sa>.
Attached below. So What I did was after downloading hadoop src. I ran
$ mvn install -DskipTests$ mvn eclipse:eclipse -DdownloadSources=true
-DdownloadJavadocs=true
And imported both Hadoop Common and HDFS projects into eclipse.
java.lang.ExceptionInInitializerError
at
org.apache.hadoop.hdfs.server.common.Storage.getBuildVersion(Storage.java:1026)
at
org.apache.hadoop.hdfs.server.protocol.NamespaceInfo.<init>(NamespaceInfo.java:58)
at
org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:266)
at
org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:122)
at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:775)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:642)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:334)
at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:316)
at
org.apache.hadoop.hdfs.TestWriteRead.initJunitModeTest(TestWriteRead.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Caused by: java.lang.NullPointerException
at java.util.Properties$LineReader.readLine(Properties.java:435)
at java.util.Properties.load0(Properties.java:354)
at java.util.Properties.load(Properties.java:342)
at org.apache.hadoop.util.VersionInfo.<init>(VersionInfo.java:51)
at org.apache.hadoop.util.VersionInfo.<clinit>(VersionInfo.java:97)
... 36 more
java.lang.NullPointerException
at org.apache.hadoop.hdfs.TestWriteRead.shutdown(TestWriteRead.java:89)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 5:59 PM, Ted Yu <yu...@gmail.com> wrote:
> Can you show us the full stack trace ?
> In Eclipse, was there any project shown with a red bang or red cross ?
>
> Cheers
>
>
> On Sun, Dec 15, 2013 at 2:26 AM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
>>
>> It tells me java.lang.ExceptionInitializerError at
>> org.apache.....hdfs.server.common.Storage.getBuildVersion!
>>
>> I feel I am missing some parameters?
>>
>> --
>> Best Regards,
>> Karim Ahmed Awara
>>
>>
>> On Sun, Dec 15, 2013 at 5:08 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> You can use the following command to generate .project files for Eclipse
>>> (at the root of your workspace):
>>> mvn clean package -DskipTests eclipse:eclipse
>>>
>>> When you import hadoop, call sub-projects would be imported.
>>>
>>> I was able to run TestWriteRead in Eclipse successfully.
>>>
>>> Cheers
>>>
>>>
>>> On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>>
>>>>
>>>> Running the test from Maven through commandline works fine. But I am
>>>> using eclipse. And it generates problem if I try to run the test as Junit,
>>>> as if eclipse is not aware of any of the conf parameters or args. Can
>>>> someone point to me a detailed source where it explains how to run Junit
>>>> through Eclipse for hadoop 2.2.x?
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>> Karim Ahmed Awara
>>>>
>>>>
>>>> On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you
>>>>> would see a lot of tests which use MiniDFSCluster
>>>>> e.g.
>>>>> cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>>>>>
>>>>> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <karim.awara@kaust.edu.sa
>>>>> > wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I read I can use MiniDFSCluster to set my own tests in case I modify
>>>>>> hadop source code. I have built hadoop 2.2. However, I can't find any
>>>>>> source on how to get the MiniDFSCluster working. Can someone point a link
>>>>>> to me that helps?
>>>>>>
>>>>>> --
>>>>>> Best Regards,
>>>>>> Karim Ahmed Awara
>>>>>>
>>>>>> ------------------------------
>>>>>> This message and its contents, including attachments are intended
>>>>>> solely for the original recipient. If you are not the intended recipient or
>>>>>> have received this message in error, please notify me immediately and
>>>>>> delete this message from your computer system. Any unauthorized use or
>>>>>> distribution is prohibited. Please consider the environment before printing
>>>>>> this email.
>>>>>
>>>>>
>>>>>
>>>>
>>>> ------------------------------
>>>> This message and its contents, including attachments are intended
>>>> solely for the original recipient. If you are not the intended recipient or
>>>> have received this message in error, please notify me immediately and
>>>> delete this message from your computer system. Any unauthorized use or
>>>> distribution is prohibited. Please consider the environment before printing
>>>> this email.
>>>>
>>>
>>>
>>
>> ------------------------------
>> This message and its contents, including attachments are intended solely
>> for the original recipient. If you are not the intended recipient or have
>> received this message in error, please notify me immediately and delete
>> this message from your computer system. Any unauthorized use or
>> distribution is prohibited. Please consider the environment before printing
>> this email.
>>
>
>
--
------------------------------
This message and its contents, including attachments are intended solely
for the original recipient. If you are not the intended recipient or have
received this message in error, please notify me immediately and delete
this message from your computer system. Any unauthorized use or
distribution is prohibited. Please consider the environment before printing
this email.
Re: MiniDFSCluster setup
Posted by Karim Awara <ka...@kaust.edu.sa>.
Attached below. So What I did was after downloading hadoop src. I ran
$ mvn install -DskipTests$ mvn eclipse:eclipse -DdownloadSources=true
-DdownloadJavadocs=true
And imported both Hadoop Common and HDFS projects into eclipse.
java.lang.ExceptionInInitializerError
at
org.apache.hadoop.hdfs.server.common.Storage.getBuildVersion(Storage.java:1026)
at
org.apache.hadoop.hdfs.server.protocol.NamespaceInfo.<init>(NamespaceInfo.java:58)
at
org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:266)
at
org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:122)
at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:775)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:642)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:334)
at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:316)
at
org.apache.hadoop.hdfs.TestWriteRead.initJunitModeTest(TestWriteRead.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Caused by: java.lang.NullPointerException
at java.util.Properties$LineReader.readLine(Properties.java:435)
at java.util.Properties.load0(Properties.java:354)
at java.util.Properties.load(Properties.java:342)
at org.apache.hadoop.util.VersionInfo.<init>(VersionInfo.java:51)
at org.apache.hadoop.util.VersionInfo.<clinit>(VersionInfo.java:97)
... 36 more
java.lang.NullPointerException
at org.apache.hadoop.hdfs.TestWriteRead.shutdown(TestWriteRead.java:89)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 5:59 PM, Ted Yu <yu...@gmail.com> wrote:
> Can you show us the full stack trace ?
> In Eclipse, was there any project shown with a red bang or red cross ?
>
> Cheers
>
>
> On Sun, Dec 15, 2013 at 2:26 AM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
>>
>> It tells me java.lang.ExceptionInitializerError at
>> org.apache.....hdfs.server.common.Storage.getBuildVersion!
>>
>> I feel I am missing some parameters?
>>
>> --
>> Best Regards,
>> Karim Ahmed Awara
>>
>>
>> On Sun, Dec 15, 2013 at 5:08 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> You can use the following command to generate .project files for Eclipse
>>> (at the root of your workspace):
>>> mvn clean package -DskipTests eclipse:eclipse
>>>
>>> When you import hadoop, call sub-projects would be imported.
>>>
>>> I was able to run TestWriteRead in Eclipse successfully.
>>>
>>> Cheers
>>>
>>>
>>> On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>>
>>>>
>>>> Running the test from Maven through commandline works fine. But I am
>>>> using eclipse. And it generates problem if I try to run the test as Junit,
>>>> as if eclipse is not aware of any of the conf parameters or args. Can
>>>> someone point to me a detailed source where it explains how to run Junit
>>>> through Eclipse for hadoop 2.2.x?
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>> Karim Ahmed Awara
>>>>
>>>>
>>>> On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you
>>>>> would see a lot of tests which use MiniDFSCluster
>>>>> e.g.
>>>>> cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>>>>>
>>>>> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <karim.awara@kaust.edu.sa
>>>>> > wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I read I can use MiniDFSCluster to set my own tests in case I modify
>>>>>> hadop source code. I have built hadoop 2.2. However, I can't find any
>>>>>> source on how to get the MiniDFSCluster working. Can someone point a link
>>>>>> to me that helps?
>>>>>>
>>>>>> --
>>>>>> Best Regards,
>>>>>> Karim Ahmed Awara
>>>>>>
>>>>>> ------------------------------
>>>>>> This message and its contents, including attachments are intended
>>>>>> solely for the original recipient. If you are not the intended recipient or
>>>>>> have received this message in error, please notify me immediately and
>>>>>> delete this message from your computer system. Any unauthorized use or
>>>>>> distribution is prohibited. Please consider the environment before printing
>>>>>> this email.
>>>>>
>>>>>
>>>>>
>>>>
>>>> ------------------------------
>>>> This message and its contents, including attachments are intended
>>>> solely for the original recipient. If you are not the intended recipient or
>>>> have received this message in error, please notify me immediately and
>>>> delete this message from your computer system. Any unauthorized use or
>>>> distribution is prohibited. Please consider the environment before printing
>>>> this email.
>>>>
>>>
>>>
>>
>> ------------------------------
>> This message and its contents, including attachments are intended solely
>> for the original recipient. If you are not the intended recipient or have
>> received this message in error, please notify me immediately and delete
>> this message from your computer system. Any unauthorized use or
>> distribution is prohibited. Please consider the environment before printing
>> this email.
>>
>
>
--
------------------------------
This message and its contents, including attachments are intended solely
for the original recipient. If you are not the intended recipient or have
received this message in error, please notify me immediately and delete
this message from your computer system. Any unauthorized use or
distribution is prohibited. Please consider the environment before printing
this email.
Re: MiniDFSCluster setup
Posted by Ted Yu <yu...@gmail.com>.
Can you show us the full stack trace ?
In Eclipse, was there any project shown with a red bang or red cross ?
Cheers
On Sun, Dec 15, 2013 at 2:26 AM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
> It tells me java.lang.ExceptionInitializerError at
> org.apache.....hdfs.server.common.Storage.getBuildVersion!
>
> I feel I am missing some parameters?
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
> On Sun, Dec 15, 2013 at 5:08 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> You can use the following command to generate .project files for Eclipse
>> (at the root of your workspace):
>> mvn clean package -DskipTests eclipse:eclipse
>>
>> When you import hadoop, call sub-projects would be imported.
>>
>> I was able to run TestWriteRead in Eclipse successfully.
>>
>> Cheers
>>
>>
>> On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>
>>>
>>> Running the test from Maven through commandline works fine. But I am
>>> using eclipse. And it generates problem if I try to run the test as Junit,
>>> as if eclipse is not aware of any of the conf parameters or args. Can
>>> someone point to me a detailed source where it explains how to run Junit
>>> through Eclipse for hadoop 2.2.x?
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Karim Ahmed Awara
>>>
>>>
>>> On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you would
>>>> see a lot of tests which use MiniDFSCluster
>>>> e.g.
>>>> cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>>>>
>>>> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>>>>
>>>> Cheers
>>>>
>>>>
>>>> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I read I can use MiniDFSCluster to set my own tests in case I modify
>>>>> hadop source code. I have built hadoop 2.2. However, I can't find any
>>>>> source on how to get the MiniDFSCluster working. Can someone point a link
>>>>> to me that helps?
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Karim Ahmed Awara
>>>>>
>>>>> ------------------------------
>>>>> This message and its contents, including attachments are intended
>>>>> solely for the original recipient. If you are not the intended recipient or
>>>>> have received this message in error, please notify me immediately and
>>>>> delete this message from your computer system. Any unauthorized use or
>>>>> distribution is prohibited. Please consider the environment before printing
>>>>> this email.
>>>>
>>>>
>>>>
>>>
>>> ------------------------------
>>> This message and its contents, including attachments are intended solely
>>> for the original recipient. If you are not the intended recipient or have
>>> received this message in error, please notify me immediately and delete
>>> this message from your computer system. Any unauthorized use or
>>> distribution is prohibited. Please consider the environment before printing
>>> this email.
>>>
>>
>>
>
> ------------------------------
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
Re: MiniDFSCluster setup
Posted by Ted Yu <yu...@gmail.com>.
Can you show us the full stack trace ?
In Eclipse, was there any project shown with a red bang or red cross ?
Cheers
On Sun, Dec 15, 2013 at 2:26 AM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
> It tells me java.lang.ExceptionInitializerError at
> org.apache.....hdfs.server.common.Storage.getBuildVersion!
>
> I feel I am missing some parameters?
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
> On Sun, Dec 15, 2013 at 5:08 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> You can use the following command to generate .project files for Eclipse
>> (at the root of your workspace):
>> mvn clean package -DskipTests eclipse:eclipse
>>
>> When you import hadoop, call sub-projects would be imported.
>>
>> I was able to run TestWriteRead in Eclipse successfully.
>>
>> Cheers
>>
>>
>> On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>
>>>
>>> Running the test from Maven through commandline works fine. But I am
>>> using eclipse. And it generates problem if I try to run the test as Junit,
>>> as if eclipse is not aware of any of the conf parameters or args. Can
>>> someone point to me a detailed source where it explains how to run Junit
>>> through Eclipse for hadoop 2.2.x?
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Karim Ahmed Awara
>>>
>>>
>>> On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you would
>>>> see a lot of tests which use MiniDFSCluster
>>>> e.g.
>>>> cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>>>>
>>>> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>>>>
>>>> Cheers
>>>>
>>>>
>>>> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I read I can use MiniDFSCluster to set my own tests in case I modify
>>>>> hadop source code. I have built hadoop 2.2. However, I can't find any
>>>>> source on how to get the MiniDFSCluster working. Can someone point a link
>>>>> to me that helps?
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Karim Ahmed Awara
>>>>>
>>>>> ------------------------------
>>>>> This message and its contents, including attachments are intended
>>>>> solely for the original recipient. If you are not the intended recipient or
>>>>> have received this message in error, please notify me immediately and
>>>>> delete this message from your computer system. Any unauthorized use or
>>>>> distribution is prohibited. Please consider the environment before printing
>>>>> this email.
>>>>
>>>>
>>>>
>>>
>>> ------------------------------
>>> This message and its contents, including attachments are intended solely
>>> for the original recipient. If you are not the intended recipient or have
>>> received this message in error, please notify me immediately and delete
>>> this message from your computer system. Any unauthorized use or
>>> distribution is prohibited. Please consider the environment before printing
>>> this email.
>>>
>>
>>
>
> ------------------------------
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
Re: MiniDFSCluster setup
Posted by Ted Yu <yu...@gmail.com>.
Can you show us the full stack trace ?
In Eclipse, was there any project shown with a red bang or red cross ?
Cheers
On Sun, Dec 15, 2013 at 2:26 AM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
> It tells me java.lang.ExceptionInitializerError at
> org.apache.....hdfs.server.common.Storage.getBuildVersion!
>
> I feel I am missing some parameters?
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
> On Sun, Dec 15, 2013 at 5:08 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> You can use the following command to generate .project files for Eclipse
>> (at the root of your workspace):
>> mvn clean package -DskipTests eclipse:eclipse
>>
>> When you import hadoop, call sub-projects would be imported.
>>
>> I was able to run TestWriteRead in Eclipse successfully.
>>
>> Cheers
>>
>>
>> On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>
>>>
>>> Running the test from Maven through commandline works fine. But I am
>>> using eclipse. And it generates problem if I try to run the test as Junit,
>>> as if eclipse is not aware of any of the conf parameters or args. Can
>>> someone point to me a detailed source where it explains how to run Junit
>>> through Eclipse for hadoop 2.2.x?
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Karim Ahmed Awara
>>>
>>>
>>> On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you would
>>>> see a lot of tests which use MiniDFSCluster
>>>> e.g.
>>>> cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>>>>
>>>> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>>>>
>>>> Cheers
>>>>
>>>>
>>>> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I read I can use MiniDFSCluster to set my own tests in case I modify
>>>>> hadop source code. I have built hadoop 2.2. However, I can't find any
>>>>> source on how to get the MiniDFSCluster working. Can someone point a link
>>>>> to me that helps?
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Karim Ahmed Awara
>>>>>
>>>>> ------------------------------
>>>>> This message and its contents, including attachments are intended
>>>>> solely for the original recipient. If you are not the intended recipient or
>>>>> have received this message in error, please notify me immediately and
>>>>> delete this message from your computer system. Any unauthorized use or
>>>>> distribution is prohibited. Please consider the environment before printing
>>>>> this email.
>>>>
>>>>
>>>>
>>>
>>> ------------------------------
>>> This message and its contents, including attachments are intended solely
>>> for the original recipient. If you are not the intended recipient or have
>>> received this message in error, please notify me immediately and delete
>>> this message from your computer system. Any unauthorized use or
>>> distribution is prohibited. Please consider the environment before printing
>>> this email.
>>>
>>
>>
>
> ------------------------------
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
Re: MiniDFSCluster setup
Posted by Ted Yu <yu...@gmail.com>.
Can you show us the full stack trace ?
In Eclipse, was there any project shown with a red bang or red cross ?
Cheers
On Sun, Dec 15, 2013 at 2:26 AM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
> It tells me java.lang.ExceptionInitializerError at
> org.apache.....hdfs.server.common.Storage.getBuildVersion!
>
> I feel I am missing some parameters?
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
> On Sun, Dec 15, 2013 at 5:08 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> You can use the following command to generate .project files for Eclipse
>> (at the root of your workspace):
>> mvn clean package -DskipTests eclipse:eclipse
>>
>> When you import hadoop, call sub-projects would be imported.
>>
>> I was able to run TestWriteRead in Eclipse successfully.
>>
>> Cheers
>>
>>
>> On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>
>>>
>>> Running the test from Maven through commandline works fine. But I am
>>> using eclipse. And it generates problem if I try to run the test as Junit,
>>> as if eclipse is not aware of any of the conf parameters or args. Can
>>> someone point to me a detailed source where it explains how to run Junit
>>> through Eclipse for hadoop 2.2.x?
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Karim Ahmed Awara
>>>
>>>
>>> On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you would
>>>> see a lot of tests which use MiniDFSCluster
>>>> e.g.
>>>> cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>>>>
>>>> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>>>>
>>>> Cheers
>>>>
>>>>
>>>> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I read I can use MiniDFSCluster to set my own tests in case I modify
>>>>> hadop source code. I have built hadoop 2.2. However, I can't find any
>>>>> source on how to get the MiniDFSCluster working. Can someone point a link
>>>>> to me that helps?
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Karim Ahmed Awara
>>>>>
>>>>> ------------------------------
>>>>> This message and its contents, including attachments are intended
>>>>> solely for the original recipient. If you are not the intended recipient or
>>>>> have received this message in error, please notify me immediately and
>>>>> delete this message from your computer system. Any unauthorized use or
>>>>> distribution is prohibited. Please consider the environment before printing
>>>>> this email.
>>>>
>>>>
>>>>
>>>
>>> ------------------------------
>>> This message and its contents, including attachments are intended solely
>>> for the original recipient. If you are not the intended recipient or have
>>> received this message in error, please notify me immediately and delete
>>> this message from your computer system. Any unauthorized use or
>>> distribution is prohibited. Please consider the environment before printing
>>> this email.
>>>
>>
>>
>
> ------------------------------
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
Re: MiniDFSCluster setup
Posted by Karim Awara <ka...@kaust.edu.sa>.
It tells me java.lang.ExceptionInitializerError at
org.apache.....hdfs.server.common.Storage.getBuildVersion!
I feel I am missing some parameters?
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 5:08 AM, Ted Yu <yu...@gmail.com> wrote:
> You can use the following command to generate .project files for Eclipse
> (at the root of your workspace):
> mvn clean package -DskipTests eclipse:eclipse
>
> When you import hadoop, call sub-projects would be imported.
>
> I was able to run TestWriteRead in Eclipse successfully.
>
> Cheers
>
>
> On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
>>
>> Running the test from Maven through commandline works fine. But I am
>> using eclipse. And it generates problem if I try to run the test as Junit,
>> as if eclipse is not aware of any of the conf parameters or args. Can
>> someone point to me a detailed source where it explains how to run Junit
>> through Eclipse for hadoop 2.2.x?
>>
>>
>>
>> --
>> Best Regards,
>> Karim Ahmed Awara
>>
>>
>> On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you would
>>> see a lot of tests which use MiniDFSCluster
>>> e.g.
>>> cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>>>
>>> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>>>
>>> Cheers
>>>
>>>
>>> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>>
>>>> Hi,
>>>>
>>>> I read I can use MiniDFSCluster to set my own tests in case I modify
>>>> hadop source code. I have built hadoop 2.2. However, I can't find any
>>>> source on how to get the MiniDFSCluster working. Can someone point a link
>>>> to me that helps?
>>>>
>>>> --
>>>> Best Regards,
>>>> Karim Ahmed Awara
>>>>
>>>> ------------------------------
>>>> This message and its contents, including attachments are intended
>>>> solely for the original recipient. If you are not the intended recipient or
>>>> have received this message in error, please notify me immediately and
>>>> delete this message from your computer system. Any unauthorized use or
>>>> distribution is prohibited. Please consider the environment before printing
>>>> this email.
>>>
>>>
>>>
>>
>> ------------------------------
>> This message and its contents, including attachments are intended solely
>> for the original recipient. If you are not the intended recipient or have
>> received this message in error, please notify me immediately and delete
>> this message from your computer system. Any unauthorized use or
>> distribution is prohibited. Please consider the environment before printing
>> this email.
>>
>
>
--
------------------------------
This message and its contents, including attachments are intended solely
for the original recipient. If you are not the intended recipient or have
received this message in error, please notify me immediately and delete
this message from your computer system. Any unauthorized use or
distribution is prohibited. Please consider the environment before printing
this email.
Re: MiniDFSCluster setup
Posted by Karim Awara <ka...@kaust.edu.sa>.
It tells me java.lang.ExceptionInitializerError at
org.apache.....hdfs.server.common.Storage.getBuildVersion!
I feel I am missing some parameters?
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 5:08 AM, Ted Yu <yu...@gmail.com> wrote:
> You can use the following command to generate .project files for Eclipse
> (at the root of your workspace):
> mvn clean package -DskipTests eclipse:eclipse
>
> When you import hadoop, call sub-projects would be imported.
>
> I was able to run TestWriteRead in Eclipse successfully.
>
> Cheers
>
>
> On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
>>
>> Running the test from Maven through commandline works fine. But I am
>> using eclipse. And it generates problem if I try to run the test as Junit,
>> as if eclipse is not aware of any of the conf parameters or args. Can
>> someone point to me a detailed source where it explains how to run Junit
>> through Eclipse for hadoop 2.2.x?
>>
>>
>>
>> --
>> Best Regards,
>> Karim Ahmed Awara
>>
>>
>> On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you would
>>> see a lot of tests which use MiniDFSCluster
>>> e.g.
>>> cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>>>
>>> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>>>
>>> Cheers
>>>
>>>
>>> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>>
>>>> Hi,
>>>>
>>>> I read I can use MiniDFSCluster to set my own tests in case I modify
>>>> hadop source code. I have built hadoop 2.2. However, I can't find any
>>>> source on how to get the MiniDFSCluster working. Can someone point a link
>>>> to me that helps?
>>>>
>>>> --
>>>> Best Regards,
>>>> Karim Ahmed Awara
>>>>
>>>> ------------------------------
>>>> This message and its contents, including attachments are intended
>>>> solely for the original recipient. If you are not the intended recipient or
>>>> have received this message in error, please notify me immediately and
>>>> delete this message from your computer system. Any unauthorized use or
>>>> distribution is prohibited. Please consider the environment before printing
>>>> this email.
>>>
>>>
>>>
>>
>> ------------------------------
>> This message and its contents, including attachments are intended solely
>> for the original recipient. If you are not the intended recipient or have
>> received this message in error, please notify me immediately and delete
>> this message from your computer system. Any unauthorized use or
>> distribution is prohibited. Please consider the environment before printing
>> this email.
>>
>
>
--
------------------------------
This message and its contents, including attachments are intended solely
for the original recipient. If you are not the intended recipient or have
received this message in error, please notify me immediately and delete
this message from your computer system. Any unauthorized use or
distribution is prohibited. Please consider the environment before printing
this email.
Re: MiniDFSCluster setup
Posted by Karim Awara <ka...@kaust.edu.sa>.
It tells me java.lang.ExceptionInitializerError at
org.apache.....hdfs.server.common.Storage.getBuildVersion!
I feel I am missing some parameters?
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 5:08 AM, Ted Yu <yu...@gmail.com> wrote:
> You can use the following command to generate .project files for Eclipse
> (at the root of your workspace):
> mvn clean package -DskipTests eclipse:eclipse
>
> When you import hadoop, call sub-projects would be imported.
>
> I was able to run TestWriteRead in Eclipse successfully.
>
> Cheers
>
>
> On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
>>
>> Running the test from Maven through commandline works fine. But I am
>> using eclipse. And it generates problem if I try to run the test as Junit,
>> as if eclipse is not aware of any of the conf parameters or args. Can
>> someone point to me a detailed source where it explains how to run Junit
>> through Eclipse for hadoop 2.2.x?
>>
>>
>>
>> --
>> Best Regards,
>> Karim Ahmed Awara
>>
>>
>> On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you would
>>> see a lot of tests which use MiniDFSCluster
>>> e.g.
>>> cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>>>
>>> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>>>
>>> Cheers
>>>
>>>
>>> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>>
>>>> Hi,
>>>>
>>>> I read I can use MiniDFSCluster to set my own tests in case I modify
>>>> hadop source code. I have built hadoop 2.2. However, I can't find any
>>>> source on how to get the MiniDFSCluster working. Can someone point a link
>>>> to me that helps?
>>>>
>>>> --
>>>> Best Regards,
>>>> Karim Ahmed Awara
>>>>
>>>> ------------------------------
>>>> This message and its contents, including attachments are intended
>>>> solely for the original recipient. If you are not the intended recipient or
>>>> have received this message in error, please notify me immediately and
>>>> delete this message from your computer system. Any unauthorized use or
>>>> distribution is prohibited. Please consider the environment before printing
>>>> this email.
>>>
>>>
>>>
>>
>> ------------------------------
>> This message and its contents, including attachments are intended solely
>> for the original recipient. If you are not the intended recipient or have
>> received this message in error, please notify me immediately and delete
>> this message from your computer system. Any unauthorized use or
>> distribution is prohibited. Please consider the environment before printing
>> this email.
>>
>
>
--
------------------------------
This message and its contents, including attachments are intended solely
for the original recipient. If you are not the intended recipient or have
received this message in error, please notify me immediately and delete
this message from your computer system. Any unauthorized use or
distribution is prohibited. Please consider the environment before printing
this email.
Re: MiniDFSCluster setup
Posted by Karim Awara <ka...@kaust.edu.sa>.
It tells me java.lang.ExceptionInitializerError at
org.apache.....hdfs.server.common.Storage.getBuildVersion!
I feel I am missing some parameters?
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 5:08 AM, Ted Yu <yu...@gmail.com> wrote:
> You can use the following command to generate .project files for Eclipse
> (at the root of your workspace):
> mvn clean package -DskipTests eclipse:eclipse
>
> When you import hadoop, call sub-projects would be imported.
>
> I was able to run TestWriteRead in Eclipse successfully.
>
> Cheers
>
>
> On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
>>
>> Running the test from Maven through commandline works fine. But I am
>> using eclipse. And it generates problem if I try to run the test as Junit,
>> as if eclipse is not aware of any of the conf parameters or args. Can
>> someone point to me a detailed source where it explains how to run Junit
>> through Eclipse for hadoop 2.2.x?
>>
>>
>>
>> --
>> Best Regards,
>> Karim Ahmed Awara
>>
>>
>> On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you would
>>> see a lot of tests which use MiniDFSCluster
>>> e.g.
>>> cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>>>
>>> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>>>
>>> Cheers
>>>
>>>
>>> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>>
>>>> Hi,
>>>>
>>>> I read I can use MiniDFSCluster to set my own tests in case I modify
>>>> hadop source code. I have built hadoop 2.2. However, I can't find any
>>>> source on how to get the MiniDFSCluster working. Can someone point a link
>>>> to me that helps?
>>>>
>>>> --
>>>> Best Regards,
>>>> Karim Ahmed Awara
>>>>
>>>> ------------------------------
>>>> This message and its contents, including attachments are intended
>>>> solely for the original recipient. If you are not the intended recipient or
>>>> have received this message in error, please notify me immediately and
>>>> delete this message from your computer system. Any unauthorized use or
>>>> distribution is prohibited. Please consider the environment before printing
>>>> this email.
>>>
>>>
>>>
>>
>> ------------------------------
>> This message and its contents, including attachments are intended solely
>> for the original recipient. If you are not the intended recipient or have
>> received this message in error, please notify me immediately and delete
>> this message from your computer system. Any unauthorized use or
>> distribution is prohibited. Please consider the environment before printing
>> this email.
>>
>
>
--
------------------------------
This message and its contents, including attachments are intended solely
for the original recipient. If you are not the intended recipient or have
received this message in error, please notify me immediately and delete
this message from your computer system. Any unauthorized use or
distribution is prohibited. Please consider the environment before printing
this email.
Re: MiniDFSCluster setup
Posted by Ted Yu <yu...@gmail.com>.
You can use the following command to generate .project files for Eclipse
(at the root of your workspace):
mvn clean package -DskipTests eclipse:eclipse
When you import hadoop, call sub-projects would be imported.
I was able to run TestWriteRead in Eclipse successfully.
Cheers
On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
> Running the test from Maven through commandline works fine. But I am
> using eclipse. And it generates problem if I try to run the test as Junit,
> as if eclipse is not aware of any of the conf parameters or args. Can
> someone point to me a detailed source where it explains how to run Junit
> through Eclipse for hadoop 2.2.x?
>
>
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
> On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you would
>> see a lot of tests which use MiniDFSCluster
>> e.g.
>> cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>>
>> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>>
>> Cheers
>>
>>
>> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>
>>> Hi,
>>>
>>> I read I can use MiniDFSCluster to set my own tests in case I modify
>>> hadop source code. I have built hadoop 2.2. However, I can't find any
>>> source on how to get the MiniDFSCluster working. Can someone point a link
>>> to me that helps?
>>>
>>> --
>>> Best Regards,
>>> Karim Ahmed Awara
>>>
>>> ------------------------------
>>> This message and its contents, including attachments are intended solely
>>> for the original recipient. If you are not the intended recipient or have
>>> received this message in error, please notify me immediately and delete
>>> this message from your computer system. Any unauthorized use or
>>> distribution is prohibited. Please consider the environment before printing
>>> this email.
>>
>>
>>
>
> ------------------------------
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
Re: MiniDFSCluster setup
Posted by Ted Yu <yu...@gmail.com>.
You can use the following command to generate .project files for Eclipse
(at the root of your workspace):
mvn clean package -DskipTests eclipse:eclipse
When you import hadoop, call sub-projects would be imported.
I was able to run TestWriteRead in Eclipse successfully.
Cheers
On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
> Running the test from Maven through commandline works fine. But I am
> using eclipse. And it generates problem if I try to run the test as Junit,
> as if eclipse is not aware of any of the conf parameters or args. Can
> someone point to me a detailed source where it explains how to run Junit
> through Eclipse for hadoop 2.2.x?
>
>
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
> On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you would
>> see a lot of tests which use MiniDFSCluster
>> e.g.
>> cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>>
>> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>>
>> Cheers
>>
>>
>> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>
>>> Hi,
>>>
>>> I read I can use MiniDFSCluster to set my own tests in case I modify
>>> hadop source code. I have built hadoop 2.2. However, I can't find any
>>> source on how to get the MiniDFSCluster working. Can someone point a link
>>> to me that helps?
>>>
>>> --
>>> Best Regards,
>>> Karim Ahmed Awara
>>>
>>> ------------------------------
>>> This message and its contents, including attachments are intended solely
>>> for the original recipient. If you are not the intended recipient or have
>>> received this message in error, please notify me immediately and delete
>>> this message from your computer system. Any unauthorized use or
>>> distribution is prohibited. Please consider the environment before printing
>>> this email.
>>
>>
>>
>
> ------------------------------
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
Re: MiniDFSCluster setup
Posted by Ted Yu <yu...@gmail.com>.
You can use the following command to generate .project files for Eclipse
(at the root of your workspace):
mvn clean package -DskipTests eclipse:eclipse
When you import hadoop, call sub-projects would be imported.
I was able to run TestWriteRead in Eclipse successfully.
Cheers
On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
> Running the test from Maven through commandline works fine. But I am
> using eclipse. And it generates problem if I try to run the test as Junit,
> as if eclipse is not aware of any of the conf parameters or args. Can
> someone point to me a detailed source where it explains how to run Junit
> through Eclipse for hadoop 2.2.x?
>
>
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
> On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you would
>> see a lot of tests which use MiniDFSCluster
>> e.g.
>> cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>>
>> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>>
>> Cheers
>>
>>
>> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>
>>> Hi,
>>>
>>> I read I can use MiniDFSCluster to set my own tests in case I modify
>>> hadop source code. I have built hadoop 2.2. However, I can't find any
>>> source on how to get the MiniDFSCluster working. Can someone point a link
>>> to me that helps?
>>>
>>> --
>>> Best Regards,
>>> Karim Ahmed Awara
>>>
>>> ------------------------------
>>> This message and its contents, including attachments are intended solely
>>> for the original recipient. If you are not the intended recipient or have
>>> received this message in error, please notify me immediately and delete
>>> this message from your computer system. Any unauthorized use or
>>> distribution is prohibited. Please consider the environment before printing
>>> this email.
>>
>>
>>
>
> ------------------------------
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
Re: MiniDFSCluster setup
Posted by Ted Yu <yu...@gmail.com>.
You can use the following command to generate .project files for Eclipse
(at the root of your workspace):
mvn clean package -DskipTests eclipse:eclipse
When you import hadoop, call sub-projects would be imported.
I was able to run TestWriteRead in Eclipse successfully.
Cheers
On Sat, Dec 14, 2013 at 4:56 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
> Running the test from Maven through commandline works fine. But I am
> using eclipse. And it generates problem if I try to run the test as Junit,
> as if eclipse is not aware of any of the conf parameters or args. Can
> someone point to me a detailed source where it explains how to run Junit
> through Eclipse for hadoop 2.2.x?
>
>
>
> --
> Best Regards,
> Karim Ahmed Awara
>
>
> On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you would
>> see a lot of tests which use MiniDFSCluster
>> e.g.
>> cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>>
>> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>>
>> Cheers
>>
>>
>> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>>
>>> Hi,
>>>
>>> I read I can use MiniDFSCluster to set my own tests in case I modify
>>> hadop source code. I have built hadoop 2.2. However, I can't find any
>>> source on how to get the MiniDFSCluster working. Can someone point a link
>>> to me that helps?
>>>
>>> --
>>> Best Regards,
>>> Karim Ahmed Awara
>>>
>>> ------------------------------
>>> This message and its contents, including attachments are intended solely
>>> for the original recipient. If you are not the intended recipient or have
>>> received this message in error, please notify me immediately and delete
>>> this message from your computer system. Any unauthorized use or
>>> distribution is prohibited. Please consider the environment before printing
>>> this email.
>>
>>
>>
>
> ------------------------------
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
>
Re: MiniDFSCluster setup
Posted by Karim Awara <ka...@kaust.edu.sa>.
Running the test from Maven through commandline works fine. But I am using
eclipse. And it generates problem if I try to run the test as Junit, as if
eclipse is not aware of any of the conf parameters or args. Can someone
point to me a detailed source where it explains how to run Junit through
Eclipse for hadoop 2.2.x?
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you would
> see a lot of tests which use MiniDFSCluster
> e.g.
> cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>
> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>
> Cheers
>
>
> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
>> Hi,
>>
>> I read I can use MiniDFSCluster to set my own tests in case I modify
>> hadop source code. I have built hadoop 2.2. However, I can't find any
>> source on how to get the MiniDFSCluster working. Can someone point a link
>> to me that helps?
>>
>> --
>> Best Regards,
>> Karim Ahmed Awara
>>
>> ------------------------------
>> This message and its contents, including attachments are intended solely
>> for the original recipient. If you are not the intended recipient or have
>> received this message in error, please notify me immediately and delete
>> this message from your computer system. Any unauthorized use or
>> distribution is prohibited. Please consider the environment before printing
>> this email.
>
>
>
--
------------------------------
This message and its contents, including attachments are intended solely
for the original recipient. If you are not the intended recipient or have
received this message in error, please notify me immediately and delete
this message from your computer system. Any unauthorized use or
distribution is prohibited. Please consider the environment before printing
this email.
Re: MiniDFSCluster setup
Posted by Karim Awara <ka...@kaust.edu.sa>.
Running the test from Maven through commandline works fine. But I am using
eclipse. And it generates problem if I try to run the test as Junit, as if
eclipse is not aware of any of the conf parameters or args. Can someone
point to me a detailed source where it explains how to run Junit through
Eclipse for hadoop 2.2.x?
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you would
> see a lot of tests which use MiniDFSCluster
> e.g.
> cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>
> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>
> Cheers
>
>
> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
>> Hi,
>>
>> I read I can use MiniDFSCluster to set my own tests in case I modify
>> hadop source code. I have built hadoop 2.2. However, I can't find any
>> source on how to get the MiniDFSCluster working. Can someone point a link
>> to me that helps?
>>
>> --
>> Best Regards,
>> Karim Ahmed Awara
>>
>> ------------------------------
>> This message and its contents, including attachments are intended solely
>> for the original recipient. If you are not the intended recipient or have
>> received this message in error, please notify me immediately and delete
>> this message from your computer system. Any unauthorized use or
>> distribution is prohibited. Please consider the environment before printing
>> this email.
>
>
>
--
------------------------------
This message and its contents, including attachments are intended solely
for the original recipient. If you are not the intended recipient or have
received this message in error, please notify me immediately and delete
this message from your computer system. Any unauthorized use or
distribution is prohibited. Please consider the environment before printing
this email.
Re: MiniDFSCluster setup
Posted by Karim Awara <ka...@kaust.edu.sa>.
Running the test from Maven through commandline works fine. But I am using
eclipse. And it generates problem if I try to run the test as Junit, as if
eclipse is not aware of any of the conf parameters or args. Can someone
point to me a detailed source where it explains how to run Junit through
Eclipse for hadoop 2.2.x?
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you would
> see a lot of tests which use MiniDFSCluster
> e.g.
> cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>
> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>
> Cheers
>
>
> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
>> Hi,
>>
>> I read I can use MiniDFSCluster to set my own tests in case I modify
>> hadop source code. I have built hadoop 2.2. However, I can't find any
>> source on how to get the MiniDFSCluster working. Can someone point a link
>> to me that helps?
>>
>> --
>> Best Regards,
>> Karim Ahmed Awara
>>
>> ------------------------------
>> This message and its contents, including attachments are intended solely
>> for the original recipient. If you are not the intended recipient or have
>> received this message in error, please notify me immediately and delete
>> this message from your computer system. Any unauthorized use or
>> distribution is prohibited. Please consider the environment before printing
>> this email.
>
>
>
--
------------------------------
This message and its contents, including attachments are intended solely
for the original recipient. If you are not the intended recipient or have
received this message in error, please notify me immediately and delete
this message from your computer system. Any unauthorized use or
distribution is prohibited. Please consider the environment before printing
this email.
Re: MiniDFSCluster setup
Posted by Karim Awara <ka...@kaust.edu.sa>.
Running the test from Maven through commandline works fine. But I am using
eclipse. And it generates problem if I try to run the test as Junit, as if
eclipse is not aware of any of the conf parameters or args. Can someone
point to me a detailed source where it explains how to run Junit through
Eclipse for hadoop 2.2.x?
--
Best Regards,
Karim Ahmed Awara
On Sun, Dec 15, 2013 at 2:55 AM, Ted Yu <yu...@gmail.com> wrote:
> If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you would
> see a lot of tests which use MiniDFSCluster
> e.g.
> cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
>
> hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
>
> Cheers
>
>
> On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
>> Hi,
>>
>> I read I can use MiniDFSCluster to set my own tests in case I modify
>> hadop source code. I have built hadoop 2.2. However, I can't find any
>> source on how to get the MiniDFSCluster working. Can someone point a link
>> to me that helps?
>>
>> --
>> Best Regards,
>> Karim Ahmed Awara
>>
>> ------------------------------
>> This message and its contents, including attachments are intended solely
>> for the original recipient. If you are not the intended recipient or have
>> received this message in error, please notify me immediately and delete
>> this message from your computer system. Any unauthorized use or
>> distribution is prohibited. Please consider the environment before printing
>> this email.
>
>
>
--
------------------------------
This message and its contents, including attachments are intended solely
for the original recipient. If you are not the intended recipient or have
received this message in error, please notify me immediately and delete
this message from your computer system. Any unauthorized use or
distribution is prohibited. Please consider the environment before printing
this email.
Re: MiniDFSCluster setup
Posted by Ted Yu <yu...@gmail.com>.
If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you would see
a lot of tests which use MiniDFSCluster
e.g.
cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
Cheers
On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
> Hi,
>
> I read I can use MiniDFSCluster to set my own tests in case I modify hadop
> source code. I have built hadoop 2.2. However, I can't find any source on
> how to get the MiniDFSCluster working. Can someone point a link to me that
> helps?
>
> --
> Best Regards,
> Karim Ahmed Awara
>
> ------------------------------
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
Re: MiniDFSCluster setup
Posted by Ted Yu <yu...@gmail.com>.
If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you would see
a lot of tests which use MiniDFSCluster
e.g.
cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
Cheers
On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
> Hi,
>
> I read I can use MiniDFSCluster to set my own tests in case I modify hadop
> source code. I have built hadoop 2.2. However, I can't find any source on
> how to get the MiniDFSCluster working. Can someone point a link to me that
> helps?
>
> --
> Best Regards,
> Karim Ahmed Awara
>
> ------------------------------
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
Re: MiniDFSCluster setup
Posted by Ted Yu <yu...@gmail.com>.
If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you would see
a lot of tests which use MiniDFSCluster
e.g.
cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
Cheers
On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
> Hi,
>
> I read I can use MiniDFSCluster to set my own tests in case I modify hadop
> source code. I have built hadoop 2.2. However, I can't find any source on
> how to get the MiniDFSCluster working. Can someone point a link to me that
> helps?
>
> --
> Best Regards,
> Karim Ahmed Awara
>
> ------------------------------
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.
Re: MiniDFSCluster setup
Posted by Ted Yu <yu...@gmail.com>.
If you search under hadoop-hdfs-project/hadoop-hdfs/src/test, you would see
a lot of tests which use MiniDFSCluster
e.g.
cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
hadoop-hdfs-project/hadoop-hdfs/src/test//java/org/apache/hadoop/hdfs/TestWriteRead.java
Cheers
On Sat, Dec 14, 2013 at 2:43 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
> Hi,
>
> I read I can use MiniDFSCluster to set my own tests in case I modify hadop
> source code. I have built hadoop 2.2. However, I can't find any source on
> how to get the MiniDFSCluster working. Can someone point a link to me that
> helps?
>
> --
> Best Regards,
> Karim Ahmed Awara
>
> ------------------------------
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.