You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Connor Woodson <cw...@gmail.com> on 2013/01/19 23:56:30 UTC
HDFS Test Failure
Running "mvn test" on the latest Flume code, I get a test failure in
TestHDFSEventSinkOnMiniCluster.
I'm using a fresh build of Ubuntu - is there a package I'm supposed to
install for it to work?
<testcase time="2.092"
classname="org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster"
name="org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster">
<error
type="java.lang.NullPointerException">java.lang.NullPointerException
at
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422)
at
org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
at
org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:124)
at
org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster.setup(TestHDFSEventSinkOnMiniCluster.java:73)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
</error>
<system-out>2013-01-19 22:46:11,966 (main) [WARN -
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:137)]
Metrics system not started: Cannot locate configuration: tried
hadoop-metrics2-namenode.properties, hadoop-metrics2.properties
Starting DataNode 0 with dfs.data.dir:
target/test/dfs/dfs/data/data1,target/test/dfs/dfs/data/data2
2013-01-19 22:46:12,950 (main) [WARN -
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:137)]
Metrics system not started: Cannot locate configuration: tried
hadoop-metrics2-datanode.properties, hadoop-metrics2.properties
2013-01-19 22:46:12,970 (main) [WARN -
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1577)]
Invalid directory in dfs.data.dir: Incorrect permission for
target/test/dfs/dfs/data/data1, expected: rwxr-xr-x, while actual: rwxrwxr-x
2013-01-19 22:46:12,978 (main) [WARN -
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1577)]
Invalid directory in dfs.data.dir: Incorrect permission for
target/test/dfs/dfs/data/data2, expected: rwxr-xr-x, while actual: rwxrwxr-x
2013-01-19 22:46:12,978 (main) [ERROR -
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1583)]
All directories in dfs.data.dir are invalid.
</system-out>
</testcase>
Re: HDFS Test Failure
Posted by Connor Woodson <cw...@gmail.com>.
Alright, fixed it. So if the HDFS tests fail, it is (likely) a file
permissions issue.
And if the HBase tests timeout, it is (again, likely) a DNS issue as
documented here<http://akbarahmed.com/2012/08/11/debugging-hbase-unable-to-find-a-viable-location-to-assign-region-root/>
.
- Connor
On Sun, Jan 27, 2013 at 6:24 PM, Connor Woodson <cw...@gmail.com>wrote:
> I ran it in Sudo mode to try and get around that, which worked; I changed
> the default umask to 0022 and it works without sudo. However, I am still
> getting the timeout for the HBase sink tests. Is there something else
> that's not set correctly?
>
> Thanks,
>
> - Connor
>
>
> On Sun, Jan 27, 2013 at 4:01 PM, Brock Noland <br...@cloudera.com> wrote:
>
>> Have you tried my umask suggestion? More detail here:
>> http://s.apache.org/9or
>>
>> On Sun, Jan 27, 2013 at 5:44 PM, Connor Woodson <cw...@gmail.com>
>> wrote:
>> > Nope, doesn't work. Doing 'mvn clean install' still gives me errors on
>> the
>> > HDFS mini cluster test; doing an install without tests and then running
>> 'mvn
>> > test' doesn't work either.
>> >
>> > Running the tests with sudo generates the same results as before: HDFS
>> tests
>> > work, but TestAsyncHBaseSink / TestHBaseSink end up timing out.
>> >
>> > - Connor
>> >
>> >
>> > On Fri, Jan 25, 2013 at 6:20 PM, Mike Percy <mp...@apache.org> wrote:
>> >>
>> >> Seems strange. Connor have you tried running "mvn clean install" and do
>> >> you get the same results?
>> >>
>> >> Flume is weird because we push SNAPSHOT builds per commit so you have
>> to
>> >> install to avoid strange dependency issues sometimes. It's especially
>> >> insidious to do mvn clean package.
>> >>
>> >> I don't know if it's related to this problem but I'd be +1 for
>> disabling
>> >> pushing SNAPSHOT builds to Maven, unless anyone sees the benefit of
>> keeping
>> >> it this way.
>> >>
>> >> Regards,
>> >> Mike
>> >>
>> >>
>> >> On Fri, Jan 25, 2013 at 5:38 PM, Connor Woodson <
>> cwoodson.dev@gmail.com>
>> >> wrote:
>> >>>
>> >>> Running "mvn clean test" as root, the HDFS test doesn't crash.
>> >>> TestAsyncHBaseSink takes a long time but succeeds. TestHBaseSink,
>> however,
>> >>> fails after a while when it times out.
>> >>>
>> >>> How can I get this to work without running in 'sudo' mode, and why
>> might
>> >>> the TestHBaseSink be hanging for just me?
>> >>>
>> >>> - Connor
>> >>>
>> >>>
>> >>>
>> >>> On Sat, Jan 19, 2013 at 3:06 PM, Brock Noland <br...@cloudera.com>
>> wrote:
>> >>>>
>> >>>> I think there is/was a bug in HDFS which caused a NPE due to umask.
>> >>>>
>> >>>> My guess is it's 0002 where as it needs to be 0022.
>> >>>>
>> >>>> On Sat, Jan 19, 2013 at 2:56 PM, Connor Woodson <
>> cwoodson.dev@gmail.com>
>> >>>> wrote:
>> >>>> > Running "mvn test" on the latest Flume code, I get a test failure
>> in
>> >>>> > TestHDFSEventSinkOnMiniCluster.
>> >>>> >
>> >>>> > I'm using a fresh build of Ubuntu - is there a package I'm
>> supposed to
>> >>>> > install for it to work?
>> >>>> >
>> >>>> > <testcase time="2.092"
>> >>>> >
>> classname="org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster"
>> >>>> > name="org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster">
>> >>>> > <error
>> >>>> >
>> type="java.lang.NullPointerException">java.lang.NullPointerException
>> >>>> > at
>> >>>> >
>> >>>> >
>> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422)
>> >>>> > at
>> >>>> >
>> >>>> >
>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
>> >>>> > at
>> >>>> >
>> >>>> >
>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:124)
>> >>>> > at
>> >>>> >
>> >>>> >
>> org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster.setup(TestHDFSEventSinkOnMiniCluster.java:73)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >>>> > at
>> >>>> >
>> >>>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> >>>> > at
>> >>>> >
>> >>>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:616)
>> >>>> > at
>> >>>> >
>> >>>> >
>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
>> >>>> > at
>> >>>> >
>> >>>> >
>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>> >>>> > at
>> >>>> >
>> >>>> >
>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
>> >>>> > at
>> >>>> >
>> >>>> >
>> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
>> >>>> > at
>> >>>> >
>> >>>> >
>> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
>> >>>> > at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
>> >>>> > at
>> >>>> >
>> >>>> >
>> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
>> >>>> > at
>> >>>> >
>> >>>> >
>> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
>> >>>> > at
>> >>>> >
>> >>>> >
>> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >>>> > at
>> >>>> >
>> >>>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> >>>> > at
>> >>>> >
>> >>>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:616)
>> >>>> > at
>> >>>> >
>> >>>> >
>> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
>> >>>> > at
>> >>>> >
>> >>>> >
>> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
>> >>>> > at
>> >>>> >
>> >>>> >
>> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
>> >>>> > at
>> >>>> >
>> >>>> >
>> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
>> >>>> > at
>> >>>> >
>> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
>> >>>> > </error>
>> >>>> > <system-out>2013-01-19 22:46:11,966 (main) [WARN -
>> >>>> >
>> >>>> >
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:137)]
>> >>>> > Metrics system not started: Cannot locate configuration: tried
>> >>>> > hadoop-metrics2-namenode.properties, hadoop-metrics2.properties
>> >>>> > Starting DataNode 0 with dfs.data.dir:
>> >>>> > target/test/dfs/dfs/data/data1,target/test/dfs/dfs/data/data2
>> >>>> > 2013-01-19 22:46:12,950 (main) [WARN -
>> >>>> >
>> >>>> >
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:137)]
>> >>>> > Metrics system not started: Cannot locate configuration: tried
>> >>>> > hadoop-metrics2-datanode.properties, hadoop-metrics2.properties
>> >>>> > 2013-01-19 22:46:12,970 (main) [WARN -
>> >>>> >
>> >>>> >
>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1577)]
>> >>>> > Invalid directory in dfs.data.dir: Incorrect permission for
>> >>>> > target/test/dfs/dfs/data/data1, expected: rwxr-xr-x, while actual:
>> >>>> > rwxrwxr-x
>> >>>> > 2013-01-19 22:46:12,978 (main) [WARN -
>> >>>> >
>> >>>> >
>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1577)]
>> >>>> > Invalid directory in dfs.data.dir: Incorrect permission for
>> >>>> > target/test/dfs/dfs/data/data2, expected: rwxr-xr-x, while actual:
>> >>>> > rwxrwxr-x
>> >>>> > 2013-01-19 22:46:12,978 (main) [ERROR -
>> >>>> >
>> >>>> >
>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1583)]
>> >>>> > All directories in dfs.data.dir are invalid.
>> >>>> > </system-out>
>> >>>> > </testcase>
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Apache MRUnit - Unit testing MapReduce -
>> >>>> http://incubator.apache.org/mrunit/
>> >>>
>> >>>
>> >>
>> >
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce -
>> http://incubator.apache.org/mrunit/
>>
>
>
Re: HDFS Test Failure
Posted by Connor Woodson <cw...@gmail.com>.
I ran it in Sudo mode to try and get around that, which worked; I changed
the default umask to 0022 and it works without sudo. However, I am still
getting the timeout for the HBase sink tests. Is there something else
that's not set correctly?
Thanks,
- Connor
On Sun, Jan 27, 2013 at 4:01 PM, Brock Noland <br...@cloudera.com> wrote:
> Have you tried my umask suggestion? More detail here:
> http://s.apache.org/9or
>
> On Sun, Jan 27, 2013 at 5:44 PM, Connor Woodson <cw...@gmail.com>
> wrote:
> > Nope, doesn't work. Doing 'mvn clean install' still gives me errors on
> the
> > HDFS mini cluster test; doing an install without tests and then running
> 'mvn
> > test' doesn't work either.
> >
> > Running the tests with sudo generates the same results as before: HDFS
> tests
> > work, but TestAsyncHBaseSink / TestHBaseSink end up timing out.
> >
> > - Connor
> >
> >
> > On Fri, Jan 25, 2013 at 6:20 PM, Mike Percy <mp...@apache.org> wrote:
> >>
> >> Seems strange. Connor have you tried running "mvn clean install" and do
> >> you get the same results?
> >>
> >> Flume is weird because we push SNAPSHOT builds per commit so you have to
> >> install to avoid strange dependency issues sometimes. It's especially
> >> insidious to do mvn clean package.
> >>
> >> I don't know if it's related to this problem but I'd be +1 for disabling
> >> pushing SNAPSHOT builds to Maven, unless anyone sees the benefit of
> keeping
> >> it this way.
> >>
> >> Regards,
> >> Mike
> >>
> >>
> >> On Fri, Jan 25, 2013 at 5:38 PM, Connor Woodson <cwoodson.dev@gmail.com
> >
> >> wrote:
> >>>
> >>> Running "mvn clean test" as root, the HDFS test doesn't crash.
> >>> TestAsyncHBaseSink takes a long time but succeeds. TestHBaseSink,
> however,
> >>> fails after a while when it times out.
> >>>
> >>> How can I get this to work without running in 'sudo' mode, and why
> might
> >>> the TestHBaseSink be hanging for just me?
> >>>
> >>> - Connor
> >>>
> >>>
> >>>
> >>> On Sat, Jan 19, 2013 at 3:06 PM, Brock Noland <br...@cloudera.com>
> wrote:
> >>>>
> >>>> I think there is/was a bug in HDFS which caused a NPE due to umask.
> >>>>
> >>>> My guess is it's 0002 where as it needs to be 0022.
> >>>>
> >>>> On Sat, Jan 19, 2013 at 2:56 PM, Connor Woodson <
> cwoodson.dev@gmail.com>
> >>>> wrote:
> >>>> > Running "mvn test" on the latest Flume code, I get a test failure in
> >>>> > TestHDFSEventSinkOnMiniCluster.
> >>>> >
> >>>> > I'm using a fresh build of Ubuntu - is there a package I'm supposed
> to
> >>>> > install for it to work?
> >>>> >
> >>>> > <testcase time="2.092"
> >>>> >
> classname="org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster"
> >>>> > name="org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster">
> >>>> > <error
> >>>> > type="java.lang.NullPointerException">java.lang.NullPointerException
> >>>> > at
> >>>> >
> >>>> >
> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422)
> >>>> > at
> >>>> >
> >>>> >
> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
> >>>> > at
> >>>> >
> >>>> >
> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:124)
> >>>> > at
> >>>> >
> >>>> >
> org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster.setup(TestHDFSEventSinkOnMiniCluster.java:73)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>> > at
> >>>> >
> >>>> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >>>> > at
> >>>> >
> >>>> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:616)
> >>>> > at
> >>>> >
> >>>> >
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> >>>> > at
> >>>> >
> >>>> >
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> >>>> > at
> >>>> >
> >>>> >
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> >>>> > at
> >>>> >
> >>>> >
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
> >>>> > at
> >>>> >
> >>>> >
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
> >>>> > at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> >>>> > at
> >>>> >
> >>>> >
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
> >>>> > at
> >>>> >
> >>>> >
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
> >>>> > at
> >>>> >
> >>>> >
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>> > at
> >>>> >
> >>>> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >>>> > at
> >>>> >
> >>>> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:616)
> >>>> > at
> >>>> >
> >>>> >
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> >>>> > at
> >>>> >
> >>>> >
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> >>>> > at
> >>>> >
> >>>> >
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> >>>> > at
> >>>> >
> >>>> >
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
> >>>> > at
> >>>> >
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
> >>>> > </error>
> >>>> > <system-out>2013-01-19 22:46:11,966 (main) [WARN -
> >>>> >
> >>>> >
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:137)]
> >>>> > Metrics system not started: Cannot locate configuration: tried
> >>>> > hadoop-metrics2-namenode.properties, hadoop-metrics2.properties
> >>>> > Starting DataNode 0 with dfs.data.dir:
> >>>> > target/test/dfs/dfs/data/data1,target/test/dfs/dfs/data/data2
> >>>> > 2013-01-19 22:46:12,950 (main) [WARN -
> >>>> >
> >>>> >
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:137)]
> >>>> > Metrics system not started: Cannot locate configuration: tried
> >>>> > hadoop-metrics2-datanode.properties, hadoop-metrics2.properties
> >>>> > 2013-01-19 22:46:12,970 (main) [WARN -
> >>>> >
> >>>> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1577)]
> >>>> > Invalid directory in dfs.data.dir: Incorrect permission for
> >>>> > target/test/dfs/dfs/data/data1, expected: rwxr-xr-x, while actual:
> >>>> > rwxrwxr-x
> >>>> > 2013-01-19 22:46:12,978 (main) [WARN -
> >>>> >
> >>>> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1577)]
> >>>> > Invalid directory in dfs.data.dir: Incorrect permission for
> >>>> > target/test/dfs/dfs/data/data2, expected: rwxr-xr-x, while actual:
> >>>> > rwxrwxr-x
> >>>> > 2013-01-19 22:46:12,978 (main) [ERROR -
> >>>> >
> >>>> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1583)]
> >>>> > All directories in dfs.data.dir are invalid.
> >>>> > </system-out>
> >>>> > </testcase>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Apache MRUnit - Unit testing MapReduce -
> >>>> http://incubator.apache.org/mrunit/
> >>>
> >>>
> >>
> >
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>
Re: HDFS Test Failure
Posted by Brock Noland <br...@cloudera.com>.
Have you tried my umask suggestion? More detail here: http://s.apache.org/9or
On Sun, Jan 27, 2013 at 5:44 PM, Connor Woodson <cw...@gmail.com> wrote:
> Nope, doesn't work. Doing 'mvn clean install' still gives me errors on the
> HDFS mini cluster test; doing an install without tests and then running 'mvn
> test' doesn't work either.
>
> Running the tests with sudo generates the same results as before: HDFS tests
> work, but TestAsyncHBaseSink / TestHBaseSink end up timing out.
>
> - Connor
>
>
> On Fri, Jan 25, 2013 at 6:20 PM, Mike Percy <mp...@apache.org> wrote:
>>
>> Seems strange. Connor have you tried running "mvn clean install" and do
>> you get the same results?
>>
>> Flume is weird because we push SNAPSHOT builds per commit so you have to
>> install to avoid strange dependency issues sometimes. It's especially
>> insidious to do mvn clean package.
>>
>> I don't know if it's related to this problem but I'd be +1 for disabling
>> pushing SNAPSHOT builds to Maven, unless anyone sees the benefit of keeping
>> it this way.
>>
>> Regards,
>> Mike
>>
>>
>> On Fri, Jan 25, 2013 at 5:38 PM, Connor Woodson <cw...@gmail.com>
>> wrote:
>>>
>>> Running "mvn clean test" as root, the HDFS test doesn't crash.
>>> TestAsyncHBaseSink takes a long time but succeeds. TestHBaseSink, however,
>>> fails after a while when it times out.
>>>
>>> How can I get this to work without running in 'sudo' mode, and why might
>>> the TestHBaseSink be hanging for just me?
>>>
>>> - Connor
>>>
>>>
>>>
>>> On Sat, Jan 19, 2013 at 3:06 PM, Brock Noland <br...@cloudera.com> wrote:
>>>>
>>>> I think there is/was a bug in HDFS which caused a NPE due to umask.
>>>>
>>>> My guess is it's 0002 where as it needs to be 0022.
>>>>
>>>> On Sat, Jan 19, 2013 at 2:56 PM, Connor Woodson <cw...@gmail.com>
>>>> wrote:
>>>> > Running "mvn test" on the latest Flume code, I get a test failure in
>>>> > TestHDFSEventSinkOnMiniCluster.
>>>> >
>>>> > I'm using a fresh build of Ubuntu - is there a package I'm supposed to
>>>> > install for it to work?
>>>> >
>>>> > <testcase time="2.092"
>>>> > classname="org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster"
>>>> > name="org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster">
>>>> > <error
>>>> > type="java.lang.NullPointerException">java.lang.NullPointerException
>>>> > at
>>>> >
>>>> > org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422)
>>>> > at
>>>> >
>>>> > org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
>>>> > at
>>>> >
>>>> > org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:124)
>>>> > at
>>>> >
>>>> > org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster.setup(TestHDFSEventSinkOnMiniCluster.java:73)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> > at
>>>> >
>>>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>> > at
>>>> >
>>>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> > at java.lang.reflect.Method.invoke(Method.java:616)
>>>> > at
>>>> >
>>>> > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
>>>> > at
>>>> >
>>>> > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>>>> > at
>>>> >
>>>> > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
>>>> > at
>>>> >
>>>> > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
>>>> > at
>>>> >
>>>> > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
>>>> > at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
>>>> > at
>>>> >
>>>> > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
>>>> > at
>>>> >
>>>> > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
>>>> > at
>>>> >
>>>> > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> > at
>>>> >
>>>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>> > at
>>>> >
>>>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> > at java.lang.reflect.Method.invoke(Method.java:616)
>>>> > at
>>>> >
>>>> > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
>>>> > at
>>>> >
>>>> > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
>>>> > at
>>>> >
>>>> > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
>>>> > at
>>>> >
>>>> > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
>>>> > at
>>>> > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
>>>> > </error>
>>>> > <system-out>2013-01-19 22:46:11,966 (main) [WARN -
>>>> >
>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:137)]
>>>> > Metrics system not started: Cannot locate configuration: tried
>>>> > hadoop-metrics2-namenode.properties, hadoop-metrics2.properties
>>>> > Starting DataNode 0 with dfs.data.dir:
>>>> > target/test/dfs/dfs/data/data1,target/test/dfs/dfs/data/data2
>>>> > 2013-01-19 22:46:12,950 (main) [WARN -
>>>> >
>>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:137)]
>>>> > Metrics system not started: Cannot locate configuration: tried
>>>> > hadoop-metrics2-datanode.properties, hadoop-metrics2.properties
>>>> > 2013-01-19 22:46:12,970 (main) [WARN -
>>>> >
>>>> > org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1577)]
>>>> > Invalid directory in dfs.data.dir: Incorrect permission for
>>>> > target/test/dfs/dfs/data/data1, expected: rwxr-xr-x, while actual:
>>>> > rwxrwxr-x
>>>> > 2013-01-19 22:46:12,978 (main) [WARN -
>>>> >
>>>> > org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1577)]
>>>> > Invalid directory in dfs.data.dir: Incorrect permission for
>>>> > target/test/dfs/dfs/data/data2, expected: rwxr-xr-x, while actual:
>>>> > rwxrwxr-x
>>>> > 2013-01-19 22:46:12,978 (main) [ERROR -
>>>> >
>>>> > org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1583)]
>>>> > All directories in dfs.data.dir are invalid.
>>>> > </system-out>
>>>> > </testcase>
>>>>
>>>>
>>>>
>>>> --
>>>> Apache MRUnit - Unit testing MapReduce -
>>>> http://incubator.apache.org/mrunit/
>>>
>>>
>>
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: HDFS Test Failure
Posted by Connor Woodson <cw...@gmail.com>.
Nope, doesn't work. Doing 'mvn clean install' still gives me errors on the
HDFS mini cluster test; doing an install without tests and then running
'mvn test' doesn't work either.
Running the tests with sudo generates the same results as before: HDFS
tests work, but TestAsyncHBaseSink / TestHBaseSink end up timing out.
- Connor
On Fri, Jan 25, 2013 at 6:20 PM, Mike Percy <mp...@apache.org> wrote:
> Seems strange. Connor have you tried running "mvn clean install" and do
> you get the same results?
>
> Flume is weird because we push SNAPSHOT builds per commit so you have to
> install to avoid strange dependency issues sometimes. It's especially
> insidious to do mvn clean package.
>
> I don't know if it's related to this problem but I'd be +1 for disabling
> pushing SNAPSHOT builds to Maven, unless anyone sees the benefit of keeping
> it this way.
>
> Regards,
> Mike
>
>
> On Fri, Jan 25, 2013 at 5:38 PM, Connor Woodson <cw...@gmail.com>wrote:
>
>> Running "mvn clean test" as root, the HDFS test doesn't crash.
>> TestAsyncHBaseSink takes a long time but succeeds. TestHBaseSink, however,
>> fails after a while when it times out.
>>
>> How can I get this to work without running in 'sudo' mode, and why might
>> the TestHBaseSink be hanging for just me?
>>
>> - Connor
>>
>>
>>
>> On Sat, Jan 19, 2013 at 3:06 PM, Brock Noland <br...@cloudera.com> wrote:
>>
>>> I think there is/was a bug in HDFS which caused a NPE due to umask.
>>>
>>> My guess is it's 0002 where as it needs to be 0022.
>>>
>>> On Sat, Jan 19, 2013 at 2:56 PM, Connor Woodson <cw...@gmail.com>
>>> wrote:
>>> > Running "mvn test" on the latest Flume code, I get a test failure in
>>> > TestHDFSEventSinkOnMiniCluster.
>>> >
>>> > I'm using a fresh build of Ubuntu - is there a package I'm supposed to
>>> > install for it to work?
>>> >
>>> > <testcase time="2.092"
>>> > classname="org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster"
>>> > name="org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster">
>>> > <error
>>> > type="java.lang.NullPointerException">java.lang.NullPointerException
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
>>> > at
>>> >
>>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:124)
>>> > at
>>> >
>>> org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster.setup(TestHDFSEventSinkOnMiniCluster.java:73)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> > at java.lang.reflect.Method.invoke(Method.java:616)
>>> > at
>>> >
>>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
>>> > at
>>> >
>>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>>> > at
>>> >
>>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
>>> > at
>>> >
>>> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
>>> > at
>>> >
>>> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
>>> > at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
>>> > at
>>> >
>>> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
>>> > at
>>> >
>>> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
>>> > at
>>> >
>>> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> > at java.lang.reflect.Method.invoke(Method.java:616)
>>> > at
>>> >
>>> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
>>> > at
>>> >
>>> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
>>> > at
>>> >
>>> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
>>> > at
>>> >
>>> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
>>> > at
>>> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
>>> > </error>
>>> > <system-out>2013-01-19 22:46:11,966 (main) [WARN -
>>> >
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:137)]
>>> > Metrics system not started: Cannot locate configuration: tried
>>> > hadoop-metrics2-namenode.properties, hadoop-metrics2.properties
>>> > Starting DataNode 0 with dfs.data.dir:
>>> > target/test/dfs/dfs/data/data1,target/test/dfs/dfs/data/data2
>>> > 2013-01-19 22:46:12,950 (main) [WARN -
>>> >
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:137)]
>>> > Metrics system not started: Cannot locate configuration: tried
>>> > hadoop-metrics2-datanode.properties, hadoop-metrics2.properties
>>> > 2013-01-19 22:46:12,970 (main) [WARN -
>>> >
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1577)]
>>> > Invalid directory in dfs.data.dir: Incorrect permission for
>>> > target/test/dfs/dfs/data/data1, expected: rwxr-xr-x, while actual:
>>> rwxrwxr-x
>>> > 2013-01-19 22:46:12,978 (main) [WARN -
>>> >
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1577)]
>>> > Invalid directory in dfs.data.dir: Incorrect permission for
>>> > target/test/dfs/dfs/data/data2, expected: rwxr-xr-x, while actual:
>>> rwxrwxr-x
>>> > 2013-01-19 22:46:12,978 (main) [ERROR -
>>> >
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1583)]
>>> > All directories in dfs.data.dir are invalid.
>>> > </system-out>
>>> > </testcase>
>>>
>>>
>>>
>>> --
>>> Apache MRUnit - Unit testing MapReduce -
>>> http://incubator.apache.org/mrunit/
>>>
>>
>>
>
Re: HDFS Test Failure
Posted by Mike Percy <mp...@apache.org>.
Seems strange. Connor have you tried running "mvn clean install" and do you
get the same results?
Flume is weird because we push SNAPSHOT builds per commit so you have to
install to avoid strange dependency issues sometimes. It's especially
insidious to do mvn clean package.
I don't know if it's related to this problem but I'd be +1 for disabling
pushing SNAPSHOT builds to Maven, unless anyone sees the benefit of keeping
it this way.
Regards,
Mike
On Fri, Jan 25, 2013 at 5:38 PM, Connor Woodson <cw...@gmail.com>wrote:
> Running "mvn clean test" as root, the HDFS test doesn't crash.
> TestAsyncHBaseSink takes a long time but succeeds. TestHBaseSink, however,
> fails after a while when it times out.
>
> How can I get this to work without running in 'sudo' mode, and why might
> the TestHBaseSink be hanging for just me?
>
> - Connor
>
>
>
> On Sat, Jan 19, 2013 at 3:06 PM, Brock Noland <br...@cloudera.com> wrote:
>
>> I think there is/was a bug in HDFS which caused a NPE due to umask.
>>
>> My guess is it's 0002 where as it needs to be 0022.
>>
>> On Sat, Jan 19, 2013 at 2:56 PM, Connor Woodson <cw...@gmail.com>
>> wrote:
>> > Running "mvn test" on the latest Flume code, I get a test failure in
>> > TestHDFSEventSinkOnMiniCluster.
>> >
>> > I'm using a fresh build of Ubuntu - is there a package I'm supposed to
>> > install for it to work?
>> >
>> > <testcase time="2.092"
>> > classname="org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster"
>> > name="org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster">
>> > <error
>> > type="java.lang.NullPointerException">java.lang.NullPointerException
>> > at
>> >
>> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422)
>> > at
>> >
>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
>> > at
>> >
>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:124)
>> > at
>> >
>> org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster.setup(TestHDFSEventSinkOnMiniCluster.java:73)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > at java.lang.reflect.Method.invoke(Method.java:616)
>> > at
>> >
>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
>> > at
>> >
>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>> > at
>> >
>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
>> > at
>> >
>> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
>> > at
>> >
>> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
>> > at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
>> > at
>> >
>> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
>> > at
>> >
>> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
>> > at
>> >
>> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > at java.lang.reflect.Method.invoke(Method.java:616)
>> > at
>> >
>> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
>> > at
>> >
>> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
>> > at
>> >
>> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
>> > at
>> >
>> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
>> > at
>> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
>> > </error>
>> > <system-out>2013-01-19 22:46:11,966 (main) [WARN -
>> >
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:137)]
>> > Metrics system not started: Cannot locate configuration: tried
>> > hadoop-metrics2-namenode.properties, hadoop-metrics2.properties
>> > Starting DataNode 0 with dfs.data.dir:
>> > target/test/dfs/dfs/data/data1,target/test/dfs/dfs/data/data2
>> > 2013-01-19 22:46:12,950 (main) [WARN -
>> >
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:137)]
>> > Metrics system not started: Cannot locate configuration: tried
>> > hadoop-metrics2-datanode.properties, hadoop-metrics2.properties
>> > 2013-01-19 22:46:12,970 (main) [WARN -
>> >
>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1577)]
>> > Invalid directory in dfs.data.dir: Incorrect permission for
>> > target/test/dfs/dfs/data/data1, expected: rwxr-xr-x, while actual:
>> rwxrwxr-x
>> > 2013-01-19 22:46:12,978 (main) [WARN -
>> >
>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1577)]
>> > Invalid directory in dfs.data.dir: Incorrect permission for
>> > target/test/dfs/dfs/data/data2, expected: rwxr-xr-x, while actual:
>> rwxrwxr-x
>> > 2013-01-19 22:46:12,978 (main) [ERROR -
>> >
>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1583)]
>> > All directories in dfs.data.dir are invalid.
>> > </system-out>
>> > </testcase>
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce -
>> http://incubator.apache.org/mrunit/
>>
>
>
Re: HDFS Test Failure
Posted by Connor Woodson <cw...@gmail.com>.
Running "mvn clean test" as root, the HDFS test doesn't crash.
TestAsyncHBaseSink takes a long time but succeeds. TestHBaseSink, however,
fails after a while when it times out.
How can I get this to work without running in 'sudo' mode, and why might
the TestHBaseSink be hanging for just me?
- Connor
On Sat, Jan 19, 2013 at 3:06 PM, Brock Noland <br...@cloudera.com> wrote:
> I think there is/was a bug in HDFS which caused a NPE due to umask.
>
> My guess is it's 0002 where as it needs to be 0022.
>
> On Sat, Jan 19, 2013 at 2:56 PM, Connor Woodson <cw...@gmail.com>
> wrote:
> > Running "mvn test" on the latest Flume code, I get a test failure in
> > TestHDFSEventSinkOnMiniCluster.
> >
> > I'm using a fresh build of Ubuntu - is there a package I'm supposed to
> > install for it to work?
> >
> > <testcase time="2.092"
> > classname="org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster"
> > name="org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster">
> > <error
> > type="java.lang.NullPointerException">java.lang.NullPointerException
> > at
> >
> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422)
> > at
> >
> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
> > at
> >
> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:124)
> > at
> >
> org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster.setup(TestHDFSEventSinkOnMiniCluster.java:73)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:616)
> > at
> >
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> > at
> >
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> > at
> >
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> > at
> >
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
> > at
> >
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
> > at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> > at
> >
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
> > at
> >
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
> > at
> >
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:616)
> > at
> >
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> > at
> >
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> > at
> >
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> > at
> >
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
> > at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
> > </error>
> > <system-out>2013-01-19 22:46:11,966 (main) [WARN -
> >
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:137)]
> > Metrics system not started: Cannot locate configuration: tried
> > hadoop-metrics2-namenode.properties, hadoop-metrics2.properties
> > Starting DataNode 0 with dfs.data.dir:
> > target/test/dfs/dfs/data/data1,target/test/dfs/dfs/data/data2
> > 2013-01-19 22:46:12,950 (main) [WARN -
> >
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:137)]
> > Metrics system not started: Cannot locate configuration: tried
> > hadoop-metrics2-datanode.properties, hadoop-metrics2.properties
> > 2013-01-19 22:46:12,970 (main) [WARN -
> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1577)]
> > Invalid directory in dfs.data.dir: Incorrect permission for
> > target/test/dfs/dfs/data/data1, expected: rwxr-xr-x, while actual:
> rwxrwxr-x
> > 2013-01-19 22:46:12,978 (main) [WARN -
> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1577)]
> > Invalid directory in dfs.data.dir: Incorrect permission for
> > target/test/dfs/dfs/data/data2, expected: rwxr-xr-x, while actual:
> rwxrwxr-x
> > 2013-01-19 22:46:12,978 (main) [ERROR -
> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1583)]
> > All directories in dfs.data.dir are invalid.
> > </system-out>
> > </testcase>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>
Re: HDFS Test Failure
Posted by Brock Noland <br...@cloudera.com>.
I think there is/was a bug in HDFS which caused a NPE due to umask.
My guess is it's 0002 where as it needs to be 0022.
On Sat, Jan 19, 2013 at 2:56 PM, Connor Woodson <cw...@gmail.com> wrote:
> Running "mvn test" on the latest Flume code, I get a test failure in
> TestHDFSEventSinkOnMiniCluster.
>
> I'm using a fresh build of Ubuntu - is there a package I'm supposed to
> install for it to work?
>
> <testcase time="2.092"
> classname="org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster"
> name="org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster">
> <error
> type="java.lang.NullPointerException">java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:124)
> at
> org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster.setup(TestHDFSEventSinkOnMiniCluster.java:73)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
> </error>
> <system-out>2013-01-19 22:46:11,966 (main) [WARN -
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:137)]
> Metrics system not started: Cannot locate configuration: tried
> hadoop-metrics2-namenode.properties, hadoop-metrics2.properties
> Starting DataNode 0 with dfs.data.dir:
> target/test/dfs/dfs/data/data1,target/test/dfs/dfs/data/data2
> 2013-01-19 22:46:12,950 (main) [WARN -
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:137)]
> Metrics system not started: Cannot locate configuration: tried
> hadoop-metrics2-datanode.properties, hadoop-metrics2.properties
> 2013-01-19 22:46:12,970 (main) [WARN -
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1577)]
> Invalid directory in dfs.data.dir: Incorrect permission for
> target/test/dfs/dfs/data/data1, expected: rwxr-xr-x, while actual: rwxrwxr-x
> 2013-01-19 22:46:12,978 (main) [WARN -
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1577)]
> Invalid directory in dfs.data.dir: Incorrect permission for
> target/test/dfs/dfs/data/data2, expected: rwxr-xr-x, while actual: rwxrwxr-x
> 2013-01-19 22:46:12,978 (main) [ERROR -
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1583)]
> All directories in dfs.data.dir are invalid.
> </system-out>
> </testcase>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/