You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Vladimir Rozov <v....@comcast.net> on 2012/09/05 19:42:10 UTC
MiniDFSCluster
There are few methods on MiniDFSCluster class that are declared as static (getBlockFile, getStorageDirPath), though as long as MiniDFSCluster is not a singleton they should be instance methods not class methods. In my tests I see that starting second instance of MiniDFSCluster invalidates the first instance if I don’t change cluster base directory (existing data directory is fully deleted), but at the same time static declaration of getBlockFile and getStorageDirPath does not allow base directory to be changed without affecting functionality.
Thank you,
Vlad
Re: MiniDFSCluster
Posted by Vladimir Rozov <v....@comcast.net>.
Agree. It is a regression in 2.x, though even in 1.x such method as
corruptBlockOnDataNode() uses "test.build.data" property instead of instance
data_dir to locate data directory and will possibly corrupt blocks on wrong
mini cluster if "test.build.data" property is changed after the cluster is
instantiated.
> This is a regression. for 1.0 you can change dir by setting the relevant
> data directory "test.build.dir".
In 2.0 it is also possible to use "hdfs.minidfs.basedir" configuration
option to change base directory. It still does not prevent two clusters to
be started at the same time that use the same base directory effectively
invalidating one of them. The best option is to change data directory (for
example "data", "data1", "data2") for every instance that share the same
MiniDFSCluster base directory or at minimum fail the second instance if the
first instance is up and running using the same base directory. I'll file
new JIRA for voting once I fix HDFS-3892.
Vlad
-----Original Message-----
From: Steve Loughran
Sent: Friday, September 07, 2012 6:15 AM
To: hdfs-dev@hadoop.apache.org
Subject: Re: MiniDFSCluster
On 5 September 2012 18:42, Vladimir Rozov <v....@comcast.net> wrote:
> There are few methods on MiniDFSCluster class that are declared as static
> (getBlockFile, getStorageDirPath), though as long as MiniDFSCluster is not
> a singleton they should be instance methods not class methods.
These aren't in 1.x, but new stuff in 2.x, which means that this behaviour
is a regression.
> In my tests I see that starting second instance of MiniDFSCluster
> invalidates the first instance if I don’t change cluster base directory
> (existing data directory is fully deleted), but at the same time static
> declaration of getBlockFile and getStorageDirPath does not allow base
> directory to be changed without affecting functionality.
>
This is a regression. for 1.0 you can change dir by setting the relevant
data directory "test.build.dir".
I don't see any reason why the static stuff is really needed, it's used in
various tests, but that could be changed -especially as the static methods
aren't in 1.x.
Why not file a JIRA -and perhaps a patch?
Re: MiniDFSCluster
Posted by Steve Loughran <st...@hortonworks.com>.
On 5 September 2012 18:42, Vladimir Rozov <v....@comcast.net> wrote:
> There are few methods on MiniDFSCluster class that are declared as static
> (getBlockFile, getStorageDirPath), though as long as MiniDFSCluster is not
> a singleton they should be instance methods not class methods.
These aren't in 1.x, but new stuff in 2.x, which means that this behaviour
is a regression.
> In my tests I see that starting second instance of MiniDFSCluster
> invalidates the first instance if I don’t change cluster base directory
> (existing data directory is fully deleted), but at the same time static
> declaration of getBlockFile and getStorageDirPath does not allow base
> directory to be changed without affecting functionality.
>
This is a regression. for 1.0 you can change dir by setting the relevant
data directory "test.build.dir".
I don't see any reason why the static stuff is really needed, it's used in
various tests, but that could be changed -especially as the static methods
aren't in 1.x.
Why not file a JIRA -and perhaps a patch?
Re: MiniDFSCluster
Posted by v....@comcast.net.
Thanks Colin. I filed https://issues.apache.org/jira/browse/HDFS-3892 . I'll look into it.
Thank you,
Vlad
----- Original Message -----
From: "Colin McCabe" <cm...@alumni.cmu.edu>
To: hdfs-dev@hadoop.apache.org
Sent: Wednesday, September 5, 2012 1:19:37 PM
Subject: Re: MiniDFSCluster
Hi Vlad,
I think you might be on to something. File a JIRA?
It should be a simple improvement, I think.
cheers,
Colin
On Wed, Sep 5, 2012 at 10:42 AM, Vladimir Rozov <v....@comcast.net> wrote:
> There are few methods on MiniDFSCluster class that are declared as static (getBlockFile, getStorageDirPath), though as long as MiniDFSCluster is not a singleton they should be instance methods not class methods. In my tests I see that starting second instance of MiniDFSCluster invalidates the first instance if I don’t change cluster base directory (existing data directory is fully deleted), but at the same time static declaration of getBlockFile and getStorageDirPath does not allow base directory to be changed without affecting functionality.
>
>
>
> Thank you,
>
>
>
> Vlad
Re: MiniDFSCluster
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
Hi Vlad,
I think you might be on to something. File a JIRA?
It should be a simple improvement, I think.
cheers,
Colin
On Wed, Sep 5, 2012 at 10:42 AM, Vladimir Rozov <v....@comcast.net> wrote:
> There are few methods on MiniDFSCluster class that are declared as static (getBlockFile, getStorageDirPath), though as long as MiniDFSCluster is not a singleton they should be instance methods not class methods. In my tests I see that starting second instance of MiniDFSCluster invalidates the first instance if I don’t change cluster base directory (existing data directory is fully deleted), but at the same time static declaration of getBlockFile and getStorageDirPath does not allow base directory to be changed without affecting functionality.
>
>
>
> Thank you,
>
>
>
> Vlad