You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by lars hofhansl <lh...@yahoo.com> on 2011/10/27 23:53:28 UTC
startMiniDFSCluster and file permissions
I just noticed today that I could not run any test that starts a MiniDFSCluster.
The exception I got was this:
java.lang.NullPointerException
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:350)
at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:519)
at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:475)
at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:462)
In the logs I had:
2011-10-27 14:17:48,238 WARN [main] datanode.DataNode(1540): Invalid directory in dfs.data.dir: Incorrect permission for /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data1, expected: rwxr-xr-x, while actual: rwxrwxr-x
2011-10-27 14:17:48,260 WARN [main] datanode.DataNode(1540): Invalid directory in dfs.data.dir: Incorrect permission for /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data2, expected: rwxr-xr-x, while actual: rwxrwxr-x
2011-10-27 14:17:48,261 ERROR [main] datanode.DataNode(1546): All directories in dfs.data.dir are invalid.
And indeed I see this in org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(...):
FsPermission dataDirPermission =
new FsPermission(conf.get(DATA_DIR_PERMISSION_KEY,
DEFAULT_DATA_DIR_PERMISSION));
for (String dir : dataDirs) {
try {
DiskChecker.checkDir(localFS, new Path(dir), dataDirPermission);
dirs.add(new File(dir));
} catch(IOException e) {
LOG.warn("Invalid directory in " + DATA_DIR_KEY + ": " +
e.getMessage());
}
}
(where DEFAULT_DATA_DIR_PERMISSION is 755)
The default umask on my machine is 0002, so that would seem to explain the discrepancy.
Changing my umask to 0022 fixed the problem!
I cannot be the only one seeing this. This is just a heads for anyone who runs into this, as I wasted over an hour on this.
I assume this is due to the switch to hadoop 0.20.205.
As I am fairly ignorant about Maven... Is there a way to set the default umask automatically for the test processes?
-- Lars
Re: startMiniDFSCluster and file permissions
Posted by Stack <st...@duboce.net>.
On Thu, Oct 27, 2011 at 4:16 PM, lars hofhansl <lh...@yahoo.com> wrote:
> Hmm... checkDir eventually calls checkPermission and that does an equals check on the expected and actual permissions.
>
> So we'd need to set DATA_DIR_PERMISSION_KEY to (777 XOR umask). Ugh.
>
We should do it?
St.Ack
Re: startMiniDFSCluster and file permissions
Posted by Gary Helmling <gh...@gmail.com>.
No, the HDFS-1560 change actually sets the permissions on the
directories if they don't match the expected. The current behavior is
just to bail out if they don't match, so you have to orchestrate your
umask so the default permissions wind up right.
On Thu, Oct 27, 2011 at 4:34 PM, lars hofhansl <lh...@yahoo.com> wrote:
> Wouldn't that still break in the same way only that you'd need to set umask to 077 in order to avoid it?
>
>
>
> ----- Original Message -----
> From: Gary Helmling <gh...@gmail.com>
> To: dev@hbase.apache.org; lars hofhansl <lh...@yahoo.com>
> Cc:
> Sent: Thursday, October 27, 2011 4:20 PM
> Subject: Re: startMiniDFSCluster and file permissions
>
> This is fixed by HDFS-1560, though unfortunately it's not in
> 0.20.205.0. I've just been running with umask set to 022.
>
>
> On Thu, Oct 27, 2011 at 4:16 PM, lars hofhansl <lh...@yahoo.com> wrote:
>> Hmm... checkDir eventually calls checkPermission and that does an equals check on the expected and actual permissions.
>>
>> So we'd need to set DATA_DIR_PERMISSION_KEY to (777 XOR umask). Ugh.
>>
>>
>> -- Lars
>>
>>
>> ----- Original Message -----
>> From: Stack <st...@duboce.net>
>> To: dev@hbase.apache.org
>> Cc: lars hofhansl <lh...@yahoo.com>
>> Sent: Thursday, October 27, 2011 3:41 PM
>> Subject: Re: startMiniDFSCluster and file permissions
>>
>> Why don't we set DATA_DIR_PERMISSION_KEY to be permissive just before
>> we spin up minidfscluster?
>> St.Ack
>>
>>
>>
>> On Thu, Oct 27, 2011 at 3:03 PM, Ted Yu <yu...@gmail.com> wrote:
>>> I think Apache Jenkins doesn't have this problem - otherwise we should have
>>> seen it by now.
>>>
>>> FYI:
>>> http://www.avajava.com/tutorials/lessons/how-do-i-set-the-default-file-and-directory-permissions.html
>>>
>>> On Thu, Oct 27, 2011 at 2:53 PM, lars hofhansl <lh...@yahoo.com> wrote:
>>>
>>>> I just noticed today that I could not run any test that starts a
>>>> MiniDFSCluster.
>>>>
>>>> The exception I got was this:
>>>> java.lang.NullPointerException
>>>> at
>>>> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422)
>>>> at
>>>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
>>>> at
>>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:350)
>>>> at
>>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:519)
>>>> at
>>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:475)
>>>> at
>>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:462)
>>>>
>>>> In the logs I had:
>>>> 2011-10-27 14:17:48,238 WARN [main] datanode.DataNode(1540): Invalid
>>>> directory in dfs.data.dir: Incorrect permission for
>>>> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data1,
>>>> expected: rwxr-xr-x, while actual: rwxrwxr-x
>>>> 2011-10-27 14:17:48,260 WARN [main] datanode.DataNode(1540): Invalid
>>>> directory in dfs.data.dir: Incorrect permission for
>>>> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data2,
>>>> expected: rwxr-xr-x, while actual: rwxrwxr-x
>>>> 2011-10-27 14:17:48,261 ERROR [main] datanode.DataNode(1546): All
>>>> directories in dfs.data.dir are invalid.
>>>>
>>>>
>>>> And indeed I see this in
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(...):
>>>>
>>>> FsPermission dataDirPermission =
>>>> new FsPermission(conf.get(DATA_DIR_PERMISSION_KEY,
>>>> DEFAULT_DATA_DIR_PERMISSION));
>>>> for (String dir : dataDirs) {
>>>> try {
>>>> DiskChecker.checkDir(localFS, new Path(dir), dataDirPermission);
>>>> dirs.add(new File(dir));
>>>> } catch(IOException e) {
>>>> LOG.warn("Invalid directory in " + DATA_DIR_KEY + ": " +
>>>> e.getMessage());
>>>> }
>>>> }
>>>>
>>>>
>>>> (where DEFAULT_DATA_DIR_PERMISSION is 755)
>>>>
>>>>
>>>> The default umask on my machine is 0002, so that would seem to explain the
>>>> discrepancy.
>>>>
>>>> Changing my umask to 0022 fixed the problem!
>>>> I cannot be the only one seeing this. This is just a heads for anyone who
>>>> runs into this, as I wasted over an hour on this.
>>>>
>>>> I assume this is due to the switch to hadoop 0.20.205.
>>>>
>>>> As I am fairly ignorant about Maven... Is there a way to set the default
>>>> umask automatically for the test processes?
>>>>
>>>> -- Lars
>>>>
>>>>
>>>
>>
>>
>
>
Re: startMiniDFSCluster and file permissions
Posted by lars hofhansl <lh...@yahoo.com>.
Wouldn't that still break in the same way only that you'd need to set umask to 077 in order to avoid it?
----- Original Message -----
From: Gary Helmling <gh...@gmail.com>
To: dev@hbase.apache.org; lars hofhansl <lh...@yahoo.com>
Cc:
Sent: Thursday, October 27, 2011 4:20 PM
Subject: Re: startMiniDFSCluster and file permissions
This is fixed by HDFS-1560, though unfortunately it's not in
0.20.205.0. I've just been running with umask set to 022.
On Thu, Oct 27, 2011 at 4:16 PM, lars hofhansl <lh...@yahoo.com> wrote:
> Hmm... checkDir eventually calls checkPermission and that does an equals check on the expected and actual permissions.
>
> So we'd need to set DATA_DIR_PERMISSION_KEY to (777 XOR umask). Ugh.
>
>
> -- Lars
>
>
> ----- Original Message -----
> From: Stack <st...@duboce.net>
> To: dev@hbase.apache.org
> Cc: lars hofhansl <lh...@yahoo.com>
> Sent: Thursday, October 27, 2011 3:41 PM
> Subject: Re: startMiniDFSCluster and file permissions
>
> Why don't we set DATA_DIR_PERMISSION_KEY to be permissive just before
> we spin up minidfscluster?
> St.Ack
>
>
>
> On Thu, Oct 27, 2011 at 3:03 PM, Ted Yu <yu...@gmail.com> wrote:
>> I think Apache Jenkins doesn't have this problem - otherwise we should have
>> seen it by now.
>>
>> FYI:
>> http://www.avajava.com/tutorials/lessons/how-do-i-set-the-default-file-and-directory-permissions.html
>>
>> On Thu, Oct 27, 2011 at 2:53 PM, lars hofhansl <lh...@yahoo.com> wrote:
>>
>>> I just noticed today that I could not run any test that starts a
>>> MiniDFSCluster.
>>>
>>> The exception I got was this:
>>> java.lang.NullPointerException
>>> at
>>> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422)
>>> at
>>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
>>> at
>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:350)
>>> at
>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:519)
>>> at
>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:475)
>>> at
>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:462)
>>>
>>> In the logs I had:
>>> 2011-10-27 14:17:48,238 WARN [main] datanode.DataNode(1540): Invalid
>>> directory in dfs.data.dir: Incorrect permission for
>>> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data1,
>>> expected: rwxr-xr-x, while actual: rwxrwxr-x
>>> 2011-10-27 14:17:48,260 WARN [main] datanode.DataNode(1540): Invalid
>>> directory in dfs.data.dir: Incorrect permission for
>>> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data2,
>>> expected: rwxr-xr-x, while actual: rwxrwxr-x
>>> 2011-10-27 14:17:48,261 ERROR [main] datanode.DataNode(1546): All
>>> directories in dfs.data.dir are invalid.
>>>
>>>
>>> And indeed I see this in
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(...):
>>>
>>> FsPermission dataDirPermission =
>>> new FsPermission(conf.get(DATA_DIR_PERMISSION_KEY,
>>> DEFAULT_DATA_DIR_PERMISSION));
>>> for (String dir : dataDirs) {
>>> try {
>>> DiskChecker.checkDir(localFS, new Path(dir), dataDirPermission);
>>> dirs.add(new File(dir));
>>> } catch(IOException e) {
>>> LOG.warn("Invalid directory in " + DATA_DIR_KEY + ": " +
>>> e.getMessage());
>>> }
>>> }
>>>
>>>
>>> (where DEFAULT_DATA_DIR_PERMISSION is 755)
>>>
>>>
>>> The default umask on my machine is 0002, so that would seem to explain the
>>> discrepancy.
>>>
>>> Changing my umask to 0022 fixed the problem!
>>> I cannot be the only one seeing this. This is just a heads for anyone who
>>> runs into this, as I wasted over an hour on this.
>>>
>>> I assume this is due to the switch to hadoop 0.20.205.
>>>
>>> As I am fairly ignorant about Maven... Is there a way to set the default
>>> umask automatically for the test processes?
>>>
>>> -- Lars
>>>
>>>
>>
>
>
Re: startMiniDFSCluster and file permissions
Posted by lars hofhansl <lh...@yahoo.com>.
Thanks Andrew. This is very useful information for us.
----- Original Message -----
From: Andrew Purtell <ap...@apache.org>
To: "dev@hbase.apache.org" <de...@hbase.apache.org>; lars hofhansl <lh...@yahoo.com>; Matt Foley <mf...@hortonworks.com>
Cc:
Sent: Thursday, October 27, 2011 4:38 PM
Subject: Re: startMiniDFSCluster and file permissions
IMHO, HDFS-1560 should go into 0.20.205.1.
There are a couple other patches we've found useful and have kept floating on 205: https://github.com/trendmicro/hadoop-common/commits/work/
- HDFS-611 improves performance of a HBase cluster
HDFS-1205 names the threads introduced by HDFS-611 for use in troubleshooting
- HADOOP-5124 / HDFS-1257 are a more direct way to deal with delete-related performance issues on a HBase cluster, by addressing the underlying delete work distribution issue in HDFS 0.20.x. Actually without this patch I've seen DNs in small clusters become extremely unbalanced when HBase is under heavy write load (triggering a high rate of compactions). The commit in the above referenced tree is a port of the combination of the two jiras. HDFS-611 is still good to have.
- HADOOP-6522 fixes decoding of codepoint zero in UTF8. Without this change TestUTF8 often fails for me.
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>________________________________
>From: Gary Helmling <gh...@gmail.com>
>To: dev@hbase.apache.org; lars hofhansl <lh...@yahoo.com>
>Sent: Thursday, October 27, 2011 4:20 PM
>Subject: Re: startMiniDFSCluster and file permissions
>
>This is fixed by HDFS-1560, though unfortunately it's not in
>0.20.205.0. I've just been running with umask set to 022.
>
>
>On Thu, Oct 27, 2011 at 4:16 PM, lars hofhansl <lh...@yahoo.com> wrote:
>> Hmm... checkDir eventually calls checkPermission and that does an equals check on the expected and actual permissions.
>>
>> So we'd need to set DATA_DIR_PERMISSION_KEY to (777 XOR umask). Ugh.
>>
>>
>> -- Lars
>>
>>
>> ----- Original Message -----
>> From: Stack <st...@duboce.net>
>> To: dev@hbase.apache.org
>> Cc: lars hofhansl <lh...@yahoo.com>
>> Sent: Thursday, October 27, 2011 3:41 PM
>> Subject: Re: startMiniDFSCluster and file permissions
>>
>> Why don't we set DATA_DIR_PERMISSION_KEY to be permissive just before
>> we spin up minidfscluster?
>> St.Ack
>>
>>
>>
>> On Thu, Oct 27, 2011 at 3:03 PM, Ted Yu <yu...@gmail.com> wrote:
>>> I think Apache Jenkins doesn't have this problem - otherwise we should have
>>> seen it by now.
>>>
>>> FYI:
>>> http://www.avajava.com/tutorials/lessons/how-do-i-set-the-default-file-and-directory-permissions.html
>>>
>>> On Thu, Oct 27, 2011 at 2:53 PM, lars hofhansl <lh...@yahoo.com> wrote:
>>>
>>>> I just noticed today that I could not run any test that starts a
>>>> MiniDFSCluster.
>>>>
>>>> The exception I got was this:
>>>> java.lang.NullPointerException
>>>> at
>>>> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422)
>>>> at
>>>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
>>>> at
>>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:350)
>>>> at
>>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:519)
>>>> at
>>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:475)
>>>> at
>>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:462)
>>>>
>>>> In the logs I had:
>>>> 2011-10-27 14:17:48,238 WARN [main] datanode.DataNode(1540): Invalid
>>>> directory in dfs.data.dir: Incorrect permission for
>>>> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data1,
>>>> expected: rwxr-xr-x, while actual: rwxrwxr-x
>>>> 2011-10-27 14:17:48,260 WARN [main] datanode.DataNode(1540): Invalid
>>>> directory in dfs.data.dir: Incorrect permission for
>>>> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data2,
>>>> expected: rwxr-xr-x, while actual: rwxrwxr-x
>>>> 2011-10-27 14:17:48,261 ERROR [main] datanode.DataNode(1546): All
>>>> directories in dfs.data.dir are invalid.
>>>>
>>>>
>>>> And indeed I see this in
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(...):
>>>>
>>>> FsPermission dataDirPermission =
>>>> new FsPermission(conf.get(DATA_DIR_PERMISSION_KEY,
>>>> DEFAULT_DATA_DIR_PERMISSION));
>>>> for (String dir : dataDirs) {
>>>> try {
>>>> DiskChecker.checkDir(localFS, new Path(dir), dataDirPermission);
>>>> dirs.add(new File(dir));
>>>> } catch(IOException e) {
>>>> LOG.warn("Invalid directory in " + DATA_DIR_KEY + ": " +
>>>> e.getMessage());
>>>> }
>>>> }
>>>>
>>>>
>>>> (where DEFAULT_DATA_DIR_PERMISSION is 755)
>>>>
>>>>
>>>> The default umask on my machine is 0002, so that would seem to explain the
>>>> discrepancy.
>>>>
>>>> Changing my umask to 0022 fixed the problem!
>>>> I cannot be the only one seeing this. This is just a heads for anyone who
>>>> runs into this, as I wasted over an hour on this.
>>>>
>>>> I assume this is due to the switch to hadoop 0.20.205.
>>>>
>>>> As I am fairly ignorant about Maven... Is there a way to set the default
>>>> umask automatically for the test processes?
>>>>
>>>> -- Lars
>>>>
>>>>
>>>
>>
>>
>
>
>
Re: startMiniDFSCluster and file permissions
Posted by Andrew Purtell <ap...@apache.org>.
IMHO, HDFS-1560 should go into 0.20.205.1.
There are a couple other patches we've found useful and have kept floating on 205: https://github.com/trendmicro/hadoop-common/commits/work/
- HDFS-611 improves performance of a HBase cluster
HDFS-1205 names the threads introduced by HDFS-611 for use in troubleshooting
- HADOOP-5124 / HDFS-1257 are a more direct way to deal with delete-related performance issues on a HBase cluster, by addressing the underlying delete work distribution issue in HDFS 0.20.x. Actually without this patch I've seen DNs in small clusters become extremely unbalanced when HBase is under heavy write load (triggering a high rate of compactions). The commit in the above referenced tree is a port of the combination of the two jiras. HDFS-611 is still good to have.
- HADOOP-6522 fixes decoding of codepoint zero in UTF8. Without this change TestUTF8 often fails for me.
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>________________________________
>From: Gary Helmling <gh...@gmail.com>
>To: dev@hbase.apache.org; lars hofhansl <lh...@yahoo.com>
>Sent: Thursday, October 27, 2011 4:20 PM
>Subject: Re: startMiniDFSCluster and file permissions
>
>This is fixed by HDFS-1560, though unfortunately it's not in
>0.20.205.0. I've just been running with umask set to 022.
>
>
>On Thu, Oct 27, 2011 at 4:16 PM, lars hofhansl <lh...@yahoo.com> wrote:
>> Hmm... checkDir eventually calls checkPermission and that does an equals check on the expected and actual permissions.
>>
>> So we'd need to set DATA_DIR_PERMISSION_KEY to (777 XOR umask). Ugh.
>>
>>
>> -- Lars
>>
>>
>> ----- Original Message -----
>> From: Stack <st...@duboce.net>
>> To: dev@hbase.apache.org
>> Cc: lars hofhansl <lh...@yahoo.com>
>> Sent: Thursday, October 27, 2011 3:41 PM
>> Subject: Re: startMiniDFSCluster and file permissions
>>
>> Why don't we set DATA_DIR_PERMISSION_KEY to be permissive just before
>> we spin up minidfscluster?
>> St.Ack
>>
>>
>>
>> On Thu, Oct 27, 2011 at 3:03 PM, Ted Yu <yu...@gmail.com> wrote:
>>> I think Apache Jenkins doesn't have this problem - otherwise we should have
>>> seen it by now.
>>>
>>> FYI:
>>> http://www.avajava.com/tutorials/lessons/how-do-i-set-the-default-file-and-directory-permissions.html
>>>
>>> On Thu, Oct 27, 2011 at 2:53 PM, lars hofhansl <lh...@yahoo.com> wrote:
>>>
>>>> I just noticed today that I could not run any test that starts a
>>>> MiniDFSCluster.
>>>>
>>>> The exception I got was this:
>>>> java.lang.NullPointerException
>>>> at
>>>> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422)
>>>> at
>>>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
>>>> at
>>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:350)
>>>> at
>>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:519)
>>>> at
>>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:475)
>>>> at
>>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:462)
>>>>
>>>> In the logs I had:
>>>> 2011-10-27 14:17:48,238 WARN [main] datanode.DataNode(1540): Invalid
>>>> directory in dfs.data.dir: Incorrect permission for
>>>> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data1,
>>>> expected: rwxr-xr-x, while actual: rwxrwxr-x
>>>> 2011-10-27 14:17:48,260 WARN [main] datanode.DataNode(1540): Invalid
>>>> directory in dfs.data.dir: Incorrect permission for
>>>> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data2,
>>>> expected: rwxr-xr-x, while actual: rwxrwxr-x
>>>> 2011-10-27 14:17:48,261 ERROR [main] datanode.DataNode(1546): All
>>>> directories in dfs.data.dir are invalid.
>>>>
>>>>
>>>> And indeed I see this in
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(...):
>>>>
>>>> FsPermission dataDirPermission =
>>>> new FsPermission(conf.get(DATA_DIR_PERMISSION_KEY,
>>>> DEFAULT_DATA_DIR_PERMISSION));
>>>> for (String dir : dataDirs) {
>>>> try {
>>>> DiskChecker.checkDir(localFS, new Path(dir), dataDirPermission);
>>>> dirs.add(new File(dir));
>>>> } catch(IOException e) {
>>>> LOG.warn("Invalid directory in " + DATA_DIR_KEY + ": " +
>>>> e.getMessage());
>>>> }
>>>> }
>>>>
>>>>
>>>> (where DEFAULT_DATA_DIR_PERMISSION is 755)
>>>>
>>>>
>>>> The default umask on my machine is 0002, so that would seem to explain the
>>>> discrepancy.
>>>>
>>>> Changing my umask to 0022 fixed the problem!
>>>> I cannot be the only one seeing this. This is just a heads for anyone who
>>>> runs into this, as I wasted over an hour on this.
>>>>
>>>> I assume this is due to the switch to hadoop 0.20.205.
>>>>
>>>> As I am fairly ignorant about Maven... Is there a way to set the default
>>>> umask automatically for the test processes?
>>>>
>>>> -- Lars
>>>>
>>>>
>>>
>>
>>
>
>
>
Re: startMiniDFSCluster and file permissions
Posted by Gary Helmling <gh...@gmail.com>.
This is fixed by HDFS-1560, though unfortunately it's not in
0.20.205.0. I've just been running with umask set to 022.
On Thu, Oct 27, 2011 at 4:16 PM, lars hofhansl <lh...@yahoo.com> wrote:
> Hmm... checkDir eventually calls checkPermission and that does an equals check on the expected and actual permissions.
>
> So we'd need to set DATA_DIR_PERMISSION_KEY to (777 XOR umask). Ugh.
>
>
> -- Lars
>
>
> ----- Original Message -----
> From: Stack <st...@duboce.net>
> To: dev@hbase.apache.org
> Cc: lars hofhansl <lh...@yahoo.com>
> Sent: Thursday, October 27, 2011 3:41 PM
> Subject: Re: startMiniDFSCluster and file permissions
>
> Why don't we set DATA_DIR_PERMISSION_KEY to be permissive just before
> we spin up minidfscluster?
> St.Ack
>
>
>
> On Thu, Oct 27, 2011 at 3:03 PM, Ted Yu <yu...@gmail.com> wrote:
>> I think Apache Jenkins doesn't have this problem - otherwise we should have
>> seen it by now.
>>
>> FYI:
>> http://www.avajava.com/tutorials/lessons/how-do-i-set-the-default-file-and-directory-permissions.html
>>
>> On Thu, Oct 27, 2011 at 2:53 PM, lars hofhansl <lh...@yahoo.com> wrote:
>>
>>> I just noticed today that I could not run any test that starts a
>>> MiniDFSCluster.
>>>
>>> The exception I got was this:
>>> java.lang.NullPointerException
>>> at
>>> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422)
>>> at
>>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
>>> at
>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:350)
>>> at
>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:519)
>>> at
>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:475)
>>> at
>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:462)
>>>
>>> In the logs I had:
>>> 2011-10-27 14:17:48,238 WARN [main] datanode.DataNode(1540): Invalid
>>> directory in dfs.data.dir: Incorrect permission for
>>> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data1,
>>> expected: rwxr-xr-x, while actual: rwxrwxr-x
>>> 2011-10-27 14:17:48,260 WARN [main] datanode.DataNode(1540): Invalid
>>> directory in dfs.data.dir: Incorrect permission for
>>> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data2,
>>> expected: rwxr-xr-x, while actual: rwxrwxr-x
>>> 2011-10-27 14:17:48,261 ERROR [main] datanode.DataNode(1546): All
>>> directories in dfs.data.dir are invalid.
>>>
>>>
>>> And indeed I see this in
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(...):
>>>
>>> FsPermission dataDirPermission =
>>> new FsPermission(conf.get(DATA_DIR_PERMISSION_KEY,
>>> DEFAULT_DATA_DIR_PERMISSION));
>>> for (String dir : dataDirs) {
>>> try {
>>> DiskChecker.checkDir(localFS, new Path(dir), dataDirPermission);
>>> dirs.add(new File(dir));
>>> } catch(IOException e) {
>>> LOG.warn("Invalid directory in " + DATA_DIR_KEY + ": " +
>>> e.getMessage());
>>> }
>>> }
>>>
>>>
>>> (where DEFAULT_DATA_DIR_PERMISSION is 755)
>>>
>>>
>>> The default umask on my machine is 0002, so that would seem to explain the
>>> discrepancy.
>>>
>>> Changing my umask to 0022 fixed the problem!
>>> I cannot be the only one seeing this. This is just a heads for anyone who
>>> runs into this, as I wasted over an hour on this.
>>>
>>> I assume this is due to the switch to hadoop 0.20.205.
>>>
>>> As I am fairly ignorant about Maven... Is there a way to set the default
>>> umask automatically for the test processes?
>>>
>>> -- Lars
>>>
>>>
>>
>
>
Re: startMiniDFSCluster and file permissions
Posted by lars hofhansl <lh...@yahoo.com>.
Hmm... checkDir eventually calls checkPermission and that does an equals check on the expected and actual permissions.
So we'd need to set DATA_DIR_PERMISSION_KEY to (777 XOR umask). Ugh.
-- Lars
----- Original Message -----
From: Stack <st...@duboce.net>
To: dev@hbase.apache.org
Cc: lars hofhansl <lh...@yahoo.com>
Sent: Thursday, October 27, 2011 3:41 PM
Subject: Re: startMiniDFSCluster and file permissions
Why don't we set DATA_DIR_PERMISSION_KEY to be permissive just before
we spin up minidfscluster?
St.Ack
On Thu, Oct 27, 2011 at 3:03 PM, Ted Yu <yu...@gmail.com> wrote:
> I think Apache Jenkins doesn't have this problem - otherwise we should have
> seen it by now.
>
> FYI:
> http://www.avajava.com/tutorials/lessons/how-do-i-set-the-default-file-and-directory-permissions.html
>
> On Thu, Oct 27, 2011 at 2:53 PM, lars hofhansl <lh...@yahoo.com> wrote:
>
>> I just noticed today that I could not run any test that starts a
>> MiniDFSCluster.
>>
>> The exception I got was this:
>> java.lang.NullPointerException
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422)
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
>> at
>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:350)
>> at
>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:519)
>> at
>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:475)
>> at
>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:462)
>>
>> In the logs I had:
>> 2011-10-27 14:17:48,238 WARN [main] datanode.DataNode(1540): Invalid
>> directory in dfs.data.dir: Incorrect permission for
>> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data1,
>> expected: rwxr-xr-x, while actual: rwxrwxr-x
>> 2011-10-27 14:17:48,260 WARN [main] datanode.DataNode(1540): Invalid
>> directory in dfs.data.dir: Incorrect permission for
>> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data2,
>> expected: rwxr-xr-x, while actual: rwxrwxr-x
>> 2011-10-27 14:17:48,261 ERROR [main] datanode.DataNode(1546): All
>> directories in dfs.data.dir are invalid.
>>
>>
>> And indeed I see this in
>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(...):
>>
>> FsPermission dataDirPermission =
>> new FsPermission(conf.get(DATA_DIR_PERMISSION_KEY,
>> DEFAULT_DATA_DIR_PERMISSION));
>> for (String dir : dataDirs) {
>> try {
>> DiskChecker.checkDir(localFS, new Path(dir), dataDirPermission);
>> dirs.add(new File(dir));
>> } catch(IOException e) {
>> LOG.warn("Invalid directory in " + DATA_DIR_KEY + ": " +
>> e.getMessage());
>> }
>> }
>>
>>
>> (where DEFAULT_DATA_DIR_PERMISSION is 755)
>>
>>
>> The default umask on my machine is 0002, so that would seem to explain the
>> discrepancy.
>>
>> Changing my umask to 0022 fixed the problem!
>> I cannot be the only one seeing this. This is just a heads for anyone who
>> runs into this, as I wasted over an hour on this.
>>
>> I assume this is due to the switch to hadoop 0.20.205.
>>
>> As I am fairly ignorant about Maven... Is there a way to set the default
>> umask automatically for the test processes?
>>
>> -- Lars
>>
>>
>
Re: startMiniDFSCluster and file permissions
Posted by Stack <st...@duboce.net>.
Why don't we set DATA_DIR_PERMISSION_KEY to be permissive just before
we spin up minidfscluster?
St.Ack
On Thu, Oct 27, 2011 at 3:03 PM, Ted Yu <yu...@gmail.com> wrote:
> I think Apache Jenkins doesn't have this problem - otherwise we should have
> seen it by now.
>
> FYI:
> http://www.avajava.com/tutorials/lessons/how-do-i-set-the-default-file-and-directory-permissions.html
>
> On Thu, Oct 27, 2011 at 2:53 PM, lars hofhansl <lh...@yahoo.com> wrote:
>
>> I just noticed today that I could not run any test that starts a
>> MiniDFSCluster.
>>
>> The exception I got was this:
>> java.lang.NullPointerException
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422)
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
>> at
>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:350)
>> at
>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:519)
>> at
>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:475)
>> at
>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:462)
>>
>> In the logs I had:
>> 2011-10-27 14:17:48,238 WARN [main] datanode.DataNode(1540): Invalid
>> directory in dfs.data.dir: Incorrect permission for
>> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data1,
>> expected: rwxr-xr-x, while actual: rwxrwxr-x
>> 2011-10-27 14:17:48,260 WARN [main] datanode.DataNode(1540): Invalid
>> directory in dfs.data.dir: Incorrect permission for
>> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data2,
>> expected: rwxr-xr-x, while actual: rwxrwxr-x
>> 2011-10-27 14:17:48,261 ERROR [main] datanode.DataNode(1546): All
>> directories in dfs.data.dir are invalid.
>>
>>
>> And indeed I see this in
>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(...):
>>
>> FsPermission dataDirPermission =
>> new FsPermission(conf.get(DATA_DIR_PERMISSION_KEY,
>> DEFAULT_DATA_DIR_PERMISSION));
>> for (String dir : dataDirs) {
>> try {
>> DiskChecker.checkDir(localFS, new Path(dir), dataDirPermission);
>> dirs.add(new File(dir));
>> } catch(IOException e) {
>> LOG.warn("Invalid directory in " + DATA_DIR_KEY + ": " +
>> e.getMessage());
>> }
>> }
>>
>>
>> (where DEFAULT_DATA_DIR_PERMISSION is 755)
>>
>>
>> The default umask on my machine is 0002, so that would seem to explain the
>> discrepancy.
>>
>> Changing my umask to 0022 fixed the problem!
>> I cannot be the only one seeing this. This is just a heads for anyone who
>> runs into this, as I wasted over an hour on this.
>>
>> I assume this is due to the switch to hadoop 0.20.205.
>>
>> As I am fairly ignorant about Maven... Is there a way to set the default
>> umask automatically for the test processes?
>>
>> -- Lars
>>
>>
>
Re: startMiniDFSCluster and file permissions
Posted by lars hofhansl <lh...@yahoo.com>.
I know how to change the default permissions (as mentioned in my email below)...
The point is that I like my umask to be 0002. All users here are in their own group,
so 0775 as default permission makes sense.
Now I have to remember to set my umask to 0022 every time I want to run the tests,
and since 0002 is the default on most Linux distributions I can't be the only one seeing this.
On jenkins there must be a different default umask, or something else must be different.
-- Lars
----- Original Message -----
From: Ted Yu <yu...@gmail.com>
To: dev@hbase.apache.org; lars hofhansl <lh...@yahoo.com>
Cc:
Sent: Thursday, October 27, 2011 3:03 PM
Subject: Re: startMiniDFSCluster and file permissions
I think Apache Jenkins doesn't have this problem - otherwise we should have
seen it by now.
FYI:
http://www.avajava.com/tutorials/lessons/how-do-i-set-the-default-file-and-directory-permissions.html
On Thu, Oct 27, 2011 at 2:53 PM, lars hofhansl <lh...@yahoo.com> wrote:
> I just noticed today that I could not run any test that starts a
> MiniDFSCluster.
>
> The exception I got was this:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:350)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:519)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:475)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:462)
>
> In the logs I had:
> 2011-10-27 14:17:48,238 WARN [main] datanode.DataNode(1540): Invalid
> directory in dfs.data.dir: Incorrect permission for
> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data1,
> expected: rwxr-xr-x, while actual: rwxrwxr-x
> 2011-10-27 14:17:48,260 WARN [main] datanode.DataNode(1540): Invalid
> directory in dfs.data.dir: Incorrect permission for
> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data2,
> expected: rwxr-xr-x, while actual: rwxrwxr-x
> 2011-10-27 14:17:48,261 ERROR [main] datanode.DataNode(1546): All
> directories in dfs.data.dir are invalid.
>
>
> And indeed I see this in
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(...):
>
> FsPermission dataDirPermission =
> new FsPermission(conf.get(DATA_DIR_PERMISSION_KEY,
> DEFAULT_DATA_DIR_PERMISSION));
> for (String dir : dataDirs) {
> try {
> DiskChecker.checkDir(localFS, new Path(dir), dataDirPermission);
> dirs.add(new File(dir));
> } catch(IOException e) {
> LOG.warn("Invalid directory in " + DATA_DIR_KEY + ": " +
> e.getMessage());
> }
> }
>
>
> (where DEFAULT_DATA_DIR_PERMISSION is 755)
>
>
> The default umask on my machine is 0002, so that would seem to explain the
> discrepancy.
>
> Changing my umask to 0022 fixed the problem!
> I cannot be the only one seeing this. This is just a heads for anyone who
> runs into this, as I wasted over an hour on this.
>
> I assume this is due to the switch to hadoop 0.20.205.
>
> As I am fairly ignorant about Maven... Is there a way to set the default
> umask automatically for the test processes?
>
> -- Lars
>
>
Re: startMiniDFSCluster and file permissions
Posted by Ted Yu <yu...@gmail.com>.
I think Apache Jenkins doesn't have this problem - otherwise we should have
seen it by now.
FYI:
http://www.avajava.com/tutorials/lessons/how-do-i-set-the-default-file-and-directory-permissions.html
On Thu, Oct 27, 2011 at 2:53 PM, lars hofhansl <lh...@yahoo.com> wrote:
> I just noticed today that I could not run any test that starts a
> MiniDFSCluster.
>
> The exception I got was this:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:350)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:519)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:475)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:462)
>
> In the logs I had:
> 2011-10-27 14:17:48,238 WARN [main] datanode.DataNode(1540): Invalid
> directory in dfs.data.dir: Incorrect permission for
> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data1,
> expected: rwxr-xr-x, while actual: rwxrwxr-x
> 2011-10-27 14:17:48,260 WARN [main] datanode.DataNode(1540): Invalid
> directory in dfs.data.dir: Incorrect permission for
> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data2,
> expected: rwxr-xr-x, while actual: rwxrwxr-x
> 2011-10-27 14:17:48,261 ERROR [main] datanode.DataNode(1546): All
> directories in dfs.data.dir are invalid.
>
>
> And indeed I see this in
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(...):
>
> FsPermission dataDirPermission =
> new FsPermission(conf.get(DATA_DIR_PERMISSION_KEY,
> DEFAULT_DATA_DIR_PERMISSION));
> for (String dir : dataDirs) {
> try {
> DiskChecker.checkDir(localFS, new Path(dir), dataDirPermission);
> dirs.add(new File(dir));
> } catch(IOException e) {
> LOG.warn("Invalid directory in " + DATA_DIR_KEY + ": " +
> e.getMessage());
> }
> }
>
>
> (where DEFAULT_DATA_DIR_PERMISSION is 755)
>
>
> The default umask on my machine is 0002, so that would seem to explain the
> discrepancy.
>
> Changing my umask to 0022 fixed the problem!
> I cannot be the only one seeing this. This is just a heads for anyone who
> runs into this, as I wasted over an hour on this.
>
> I assume this is due to the switch to hadoop 0.20.205.
>
> As I am fairly ignorant about Maven... Is there a way to set the default
> umask automatically for the test processes?
>
> -- Lars
>
>