You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Avinash Kujur <av...@gmail.com> on 2014/05/13 11:33:30 UTC
enable regular expression on which parameter?
mapreduce-5851
i can see many parameters in Distcp class. in which parameter do we need to
enable regular expressions?
private static final String usage = NAME
+ " [OPTIONS] <srcurl>* <desturl>" +
"\n\nOPTIONS:" +
"\n-p[rbugp] Preserve status" +
"\n r: replication number" +
"\n b: block size" +
"\n u: user" +
"\n g: group" +
"\n p: permission" +
"\n -p alone is equivalent to -prbugp" +
"\n-i Ignore failures" +
"\n-log <logdir> Write logs to <logdir>" +
"\n-m <num_maps> Maximum number of simultaneous copies" +
"\n-overwrite Overwrite destination" +
"\n-update Overwrite if src size different from dst
size" +
"\n-f <urilist_uri> Use list at <urilist_uri> as src list" +
"\n-filelimit <n> Limit the total number of files to be <= n"
+
"\n-sizelimit <n> Limit the total size to be <= n bytes" +
"\n-delete Delete the files existing in the dst but
not in src" +
"\n-mapredSslConf <f> Filename of SSL configuration for mapper
task" +
"\n\nNOTE 1: if -overwrite or -update are set, each source URI is " +
"\n interpreted as an isomorphic update to an existing
directory." +
"\nFor example:" +
"\nhadoop " + NAME + " -p -update \"hdfs://A:8020/user/foo/bar\" " +
"\"hdfs://B:8020/user/foo/baz\"\n" +
"\n would update all descendants of 'baz' also in 'bar'; it would
" +
"\n *not* update /user/foo/baz/bar" +
"\n\nNOTE 2: The parameter <n> in -filelimit and -sizelimit can be " +
"\n specified with symbolic representation. For examples," +
"\n 1230k = 1230 * 1024 = 1259520" +
"\n 891g = 891 * 1024^3 = 956703965184" +
"\n";
Re: enable regular expression on which parameter?
Posted by Ravi Prakash <ra...@ymail.com>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Avinash!
That JIRA is still open and does not seem to have been fixed. There are
a lot of issues with providing regexes though. A long standing issue has
been https://issues.apache.org/jira/browse/HDFS-13 which makes it even
harder
HTH
Ravi
On 05/13/14 02:33, Avinash Kujur wrote:
> mapreduce-5851
> i can see many parameters in Distcp class. in which parameter do we
need to
> enable regular expressions?
>
> private static final String usage = NAME
> + " [OPTIONS] <srcurl>* <desturl>" +
> "\n\nOPTIONS:" +
> "\n-p[rbugp] Preserve status" +
> "\n r: replication number" +
> "\n b: block size" +
> "\n u: user" +
> "\n g: group" +
> "\n p: permission" +
> "\n -p alone is equivalent to -prbugp" +
> "\n-i Ignore failures" +
> "\n-log <logdir> Write logs to <logdir>" +
> "\n-m <num_maps> Maximum number of simultaneous copies" +
> "\n-overwrite Overwrite destination" +
> "\n-update Overwrite if src size different from dst
> size" +
> "\n-f <urilist_uri> Use list at <urilist_uri> as src list" +
> "\n-filelimit <n> Limit the total number of files to be
<= n"
> +
> "\n-sizelimit <n> Limit the total size to be <= n bytes" +
> "\n-delete Delete the files existing in the dst but
> not in src" +
> "\n-mapredSslConf <f> Filename of SSL configuration for mapper
> task" +
>
> "\n\nNOTE 1: if -overwrite or -update are set, each source URI
is " +
> "\n interpreted as an isomorphic update to an existing
> directory." +
> "\nFor example:" +
> "\nhadoop " + NAME + " -p -update \"hdfs://A:8020/user/foo/bar\" " +
> "\"hdfs://B:8020/user/foo/baz\"\n" +
> "\n would update all descendants of 'baz' also in 'bar'; it
would
> " +
> "\n *not* update /user/foo/baz/bar" +
>
> "\n\nNOTE 2: The parameter <n> in -filelimit and -sizelimit can
be " +
> "\n specified with symbolic representation. For examples," +
> "\n 1230k = 1230 * 1024 = 1259520" +
> "\n 891g = 891 * 1024^3 = 956703965184" +
>
> "\n";
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBAgAGBQJTcvnBAAoJEGunL/HJl4XeacEQAIW/V14C9XZXKhWenEAALDQs
lFir6u0sdeelKYPCzqHGt41CWAMdSWl1YNl0gBXd1+o32U/y/T4Rb9vyZ6U5yG8I
OKpEEWx9ckiOke/jdpe0fxt2LiVyXhq/W3GckSinga5obZYtq1GWT+DsSMsXIU4b
EGwGe1prOs9o1wRQv00dWFskP3CifocZUYX7RKePfrNsHmlobonGl1gyjOpBHgoP
bXDsatQm5JQINDI8JjyBmXfqtQGWSSuSh7k/y8vfSBRDVwLeQNF5E6XrJcavFVeV
Anzst1eP0IsKbSFh3wnxPpEeOhhhYAv3mNbvtYu3c5/PzUmE5gFBQIgMTTMbBH16
xPT2btTIGueQTUQY6MTmmBaIH149s0opVKpLLizaFyqm/VJiUDgeiMDLpZhXYtHM
fC1swGBrK4IAmrHFGVbZs1ZfO5abntDkPlZJTbHvNX7CKTOR+CFmiYeBQ5buIcFU
bFgTKH6b9TQL4yHbVwpxEzgCId4YlheCiiDslXjLW5rPfHwtUGUkbXJQjGDHYafE
siJ2VYK6fI6E7Jq8GU+Ktw6z3gVZ2DFToPkudBNWGTsbHih6ARTW6fsY8w/RVRiI
IqSniJ103lKXZ+LGe3E2JyHkP5trjl5QnQFp4d9i7JiUXVanVAP93/h74emzJbkK
ctwR9R87n7ipXrGzMWtP
=3EnT
-----END PGP SIGNATURE-----
Re: enable regular expression on which parameter?
Posted by Ravi Prakash <ra...@ymail.com>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Avinash!
That JIRA is still open and does not seem to have been fixed. There are
a lot of issues with providing regexes though. A long standing issue has
been https://issues.apache.org/jira/browse/HDFS-13 which makes it even
harder
HTH
Ravi
On 05/13/14 02:33, Avinash Kujur wrote:
> mapreduce-5851
> i can see many parameters in Distcp class. in which parameter do we
need to
> enable regular expressions?
>
> private static final String usage = NAME
> + " [OPTIONS] <srcurl>* <desturl>" +
> "\n\nOPTIONS:" +
> "\n-p[rbugp] Preserve status" +
> "\n r: replication number" +
> "\n b: block size" +
> "\n u: user" +
> "\n g: group" +
> "\n p: permission" +
> "\n -p alone is equivalent to -prbugp" +
> "\n-i Ignore failures" +
> "\n-log <logdir> Write logs to <logdir>" +
> "\n-m <num_maps> Maximum number of simultaneous copies" +
> "\n-overwrite Overwrite destination" +
> "\n-update Overwrite if src size different from dst
> size" +
> "\n-f <urilist_uri> Use list at <urilist_uri> as src list" +
> "\n-filelimit <n> Limit the total number of files to be
<= n"
> +
> "\n-sizelimit <n> Limit the total size to be <= n bytes" +
> "\n-delete Delete the files existing in the dst but
> not in src" +
> "\n-mapredSslConf <f> Filename of SSL configuration for mapper
> task" +
>
> "\n\nNOTE 1: if -overwrite or -update are set, each source URI
is " +
> "\n interpreted as an isomorphic update to an existing
> directory." +
> "\nFor example:" +
> "\nhadoop " + NAME + " -p -update \"hdfs://A:8020/user/foo/bar\" " +
> "\"hdfs://B:8020/user/foo/baz\"\n" +
> "\n would update all descendants of 'baz' also in 'bar'; it
would
> " +
> "\n *not* update /user/foo/baz/bar" +
>
> "\n\nNOTE 2: The parameter <n> in -filelimit and -sizelimit can
be " +
> "\n specified with symbolic representation. For examples," +
> "\n 1230k = 1230 * 1024 = 1259520" +
> "\n 891g = 891 * 1024^3 = 956703965184" +
>
> "\n";
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBAgAGBQJTcvnBAAoJEGunL/HJl4XeacEQAIW/V14C9XZXKhWenEAALDQs
lFir6u0sdeelKYPCzqHGt41CWAMdSWl1YNl0gBXd1+o32U/y/T4Rb9vyZ6U5yG8I
OKpEEWx9ckiOke/jdpe0fxt2LiVyXhq/W3GckSinga5obZYtq1GWT+DsSMsXIU4b
EGwGe1prOs9o1wRQv00dWFskP3CifocZUYX7RKePfrNsHmlobonGl1gyjOpBHgoP
bXDsatQm5JQINDI8JjyBmXfqtQGWSSuSh7k/y8vfSBRDVwLeQNF5E6XrJcavFVeV
Anzst1eP0IsKbSFh3wnxPpEeOhhhYAv3mNbvtYu3c5/PzUmE5gFBQIgMTTMbBH16
xPT2btTIGueQTUQY6MTmmBaIH149s0opVKpLLizaFyqm/VJiUDgeiMDLpZhXYtHM
fC1swGBrK4IAmrHFGVbZs1ZfO5abntDkPlZJTbHvNX7CKTOR+CFmiYeBQ5buIcFU
bFgTKH6b9TQL4yHbVwpxEzgCId4YlheCiiDslXjLW5rPfHwtUGUkbXJQjGDHYafE
siJ2VYK6fI6E7Jq8GU+Ktw6z3gVZ2DFToPkudBNWGTsbHih6ARTW6fsY8w/RVRiI
IqSniJ103lKXZ+LGe3E2JyHkP5trjl5QnQFp4d9i7JiUXVanVAP93/h74emzJbkK
ctwR9R87n7ipXrGzMWtP
=3EnT
-----END PGP SIGNATURE-----
Re: enable regular expression on which parameter?
Posted by Ravi Prakash <ra...@ymail.com>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Avinash!
That JIRA is still open and does not seem to have been fixed. There are
a lot of issues with providing regexes though. A long standing issue has
been https://issues.apache.org/jira/browse/HDFS-13 which makes it even
harder
HTH
Ravi
On 05/13/14 02:33, Avinash Kujur wrote:
> mapreduce-5851
> i can see many parameters in Distcp class. in which parameter do we
need to
> enable regular expressions?
>
> private static final String usage = NAME
> + " [OPTIONS] <srcurl>* <desturl>" +
> "\n\nOPTIONS:" +
> "\n-p[rbugp] Preserve status" +
> "\n r: replication number" +
> "\n b: block size" +
> "\n u: user" +
> "\n g: group" +
> "\n p: permission" +
> "\n -p alone is equivalent to -prbugp" +
> "\n-i Ignore failures" +
> "\n-log <logdir> Write logs to <logdir>" +
> "\n-m <num_maps> Maximum number of simultaneous copies" +
> "\n-overwrite Overwrite destination" +
> "\n-update Overwrite if src size different from dst
> size" +
> "\n-f <urilist_uri> Use list at <urilist_uri> as src list" +
> "\n-filelimit <n> Limit the total number of files to be
<= n"
> +
> "\n-sizelimit <n> Limit the total size to be <= n bytes" +
> "\n-delete Delete the files existing in the dst but
> not in src" +
> "\n-mapredSslConf <f> Filename of SSL configuration for mapper
> task" +
>
> "\n\nNOTE 1: if -overwrite or -update are set, each source URI
is " +
> "\n interpreted as an isomorphic update to an existing
> directory." +
> "\nFor example:" +
> "\nhadoop " + NAME + " -p -update \"hdfs://A:8020/user/foo/bar\" " +
> "\"hdfs://B:8020/user/foo/baz\"\n" +
> "\n would update all descendants of 'baz' also in 'bar'; it
would
> " +
> "\n *not* update /user/foo/baz/bar" +
>
> "\n\nNOTE 2: The parameter <n> in -filelimit and -sizelimit can
be " +
> "\n specified with symbolic representation. For examples," +
> "\n 1230k = 1230 * 1024 = 1259520" +
> "\n 891g = 891 * 1024^3 = 956703965184" +
>
> "\n";
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBAgAGBQJTcvnBAAoJEGunL/HJl4XeacEQAIW/V14C9XZXKhWenEAALDQs
lFir6u0sdeelKYPCzqHGt41CWAMdSWl1YNl0gBXd1+o32U/y/T4Rb9vyZ6U5yG8I
OKpEEWx9ckiOke/jdpe0fxt2LiVyXhq/W3GckSinga5obZYtq1GWT+DsSMsXIU4b
EGwGe1prOs9o1wRQv00dWFskP3CifocZUYX7RKePfrNsHmlobonGl1gyjOpBHgoP
bXDsatQm5JQINDI8JjyBmXfqtQGWSSuSh7k/y8vfSBRDVwLeQNF5E6XrJcavFVeV
Anzst1eP0IsKbSFh3wnxPpEeOhhhYAv3mNbvtYu3c5/PzUmE5gFBQIgMTTMbBH16
xPT2btTIGueQTUQY6MTmmBaIH149s0opVKpLLizaFyqm/VJiUDgeiMDLpZhXYtHM
fC1swGBrK4IAmrHFGVbZs1ZfO5abntDkPlZJTbHvNX7CKTOR+CFmiYeBQ5buIcFU
bFgTKH6b9TQL4yHbVwpxEzgCId4YlheCiiDslXjLW5rPfHwtUGUkbXJQjGDHYafE
siJ2VYK6fI6E7Jq8GU+Ktw6z3gVZ2DFToPkudBNWGTsbHih6ARTW6fsY8w/RVRiI
IqSniJ103lKXZ+LGe3E2JyHkP5trjl5QnQFp4d9i7JiUXVanVAP93/h74emzJbkK
ctwR9R87n7ipXrGzMWtP
=3EnT
-----END PGP SIGNATURE-----
Re: enable regular expression on which parameter?
Posted by Ravi Prakash <ra...@ymail.com>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Avinash!
That JIRA is still open and does not seem to have been fixed. There are
a lot of issues with providing regexes though. A long standing issue has
been https://issues.apache.org/jira/browse/HDFS-13 which makes it even
harder
HTH
Ravi
On 05/13/14 02:33, Avinash Kujur wrote:
> mapreduce-5851
> i can see many parameters in Distcp class. in which parameter do we
need to
> enable regular expressions?
>
> private static final String usage = NAME
> + " [OPTIONS] <srcurl>* <desturl>" +
> "\n\nOPTIONS:" +
> "\n-p[rbugp] Preserve status" +
> "\n r: replication number" +
> "\n b: block size" +
> "\n u: user" +
> "\n g: group" +
> "\n p: permission" +
> "\n -p alone is equivalent to -prbugp" +
> "\n-i Ignore failures" +
> "\n-log <logdir> Write logs to <logdir>" +
> "\n-m <num_maps> Maximum number of simultaneous copies" +
> "\n-overwrite Overwrite destination" +
> "\n-update Overwrite if src size different from dst
> size" +
> "\n-f <urilist_uri> Use list at <urilist_uri> as src list" +
> "\n-filelimit <n> Limit the total number of files to be
<= n"
> +
> "\n-sizelimit <n> Limit the total size to be <= n bytes" +
> "\n-delete Delete the files existing in the dst but
> not in src" +
> "\n-mapredSslConf <f> Filename of SSL configuration for mapper
> task" +
>
> "\n\nNOTE 1: if -overwrite or -update are set, each source URI
is " +
> "\n interpreted as an isomorphic update to an existing
> directory." +
> "\nFor example:" +
> "\nhadoop " + NAME + " -p -update \"hdfs://A:8020/user/foo/bar\" " +
> "\"hdfs://B:8020/user/foo/baz\"\n" +
> "\n would update all descendants of 'baz' also in 'bar'; it
would
> " +
> "\n *not* update /user/foo/baz/bar" +
>
> "\n\nNOTE 2: The parameter <n> in -filelimit and -sizelimit can
be " +
> "\n specified with symbolic representation. For examples," +
> "\n 1230k = 1230 * 1024 = 1259520" +
> "\n 891g = 891 * 1024^3 = 956703965184" +
>
> "\n";
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBAgAGBQJTcvnBAAoJEGunL/HJl4XeacEQAIW/V14C9XZXKhWenEAALDQs
lFir6u0sdeelKYPCzqHGt41CWAMdSWl1YNl0gBXd1+o32U/y/T4Rb9vyZ6U5yG8I
OKpEEWx9ckiOke/jdpe0fxt2LiVyXhq/W3GckSinga5obZYtq1GWT+DsSMsXIU4b
EGwGe1prOs9o1wRQv00dWFskP3CifocZUYX7RKePfrNsHmlobonGl1gyjOpBHgoP
bXDsatQm5JQINDI8JjyBmXfqtQGWSSuSh7k/y8vfSBRDVwLeQNF5E6XrJcavFVeV
Anzst1eP0IsKbSFh3wnxPpEeOhhhYAv3mNbvtYu3c5/PzUmE5gFBQIgMTTMbBH16
xPT2btTIGueQTUQY6MTmmBaIH149s0opVKpLLizaFyqm/VJiUDgeiMDLpZhXYtHM
fC1swGBrK4IAmrHFGVbZs1ZfO5abntDkPlZJTbHvNX7CKTOR+CFmiYeBQ5buIcFU
bFgTKH6b9TQL4yHbVwpxEzgCId4YlheCiiDslXjLW5rPfHwtUGUkbXJQjGDHYafE
siJ2VYK6fI6E7Jq8GU+Ktw6z3gVZ2DFToPkudBNWGTsbHih6ARTW6fsY8w/RVRiI
IqSniJ103lKXZ+LGe3E2JyHkP5trjl5QnQFp4d9i7JiUXVanVAP93/h74emzJbkK
ctwR9R87n7ipXrGzMWtP
=3EnT
-----END PGP SIGNATURE-----