You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Doug Judd (JIRA)" <ji...@apache.org> on 2007/01/17 00:51:27 UTC
[jira] Created: (HADOOP-895) Reduce hang on small set of
SequenceFiles (reproducible)
Reduce hang on small set of SequenceFiles (reproducible)
--------------------------------------------------------
Key: HADOOP-895
URL: https://issues.apache.org/jira/browse/HADOOP-895
Project: Hadoop
Issue Type: Bug
Components: mapred
Affects Versions: 0.10.1
Environment: This happens both on a mac and on a Linux x86_64 machine.
Reporter: Doug Judd
I'm running into a problem where a Map/Reduce job hangs in the reduce phase. It happens on a simple program to dump the keys of a small set of SequenceFiles. It can be reproduced as follows:
1. Pull a vanilla hadoop-0.10.1 release and untar it
2. Apply the attached patch
3. Modify conf/hadoop-env.sh to have the correct JAVA_HOME
4. Run the following script:
#!/bin/sh
ant zvents
./bin/hadoop namenode -format
mkdir fs-hadoop/dfs/tmp
mkdir fs-hadoop/dfs/data
mkdir -p fs-hadoop/mapred/local
./bin/start-all.sh
sleep 5
./bin/hadoop dfs -mkdir /heritrix
./bin/hadoop dfs -mkdir /heritrix/crawls
./bin/hadoop dfs -mkdir /heritrix/crawls/test-4-20070116061502956
./bin/hadoop dfs -copyFromLocal data/IAH-20070116061532-00000-doug-judds-computer.local /heritrix/crawls/test-4-20070116061502956
./bin/hadoop dfs -copyFromLocal data/IAH-20070116061539-00001-doug-judds-computer.local /heritrix/crawls/test-4-20070116061502956
./bin/hadoop dfs -copyFromLocal data/IAH-20070116061539-00002-doug-judds-computer.local /heritrix/crawls/test-4-20070116061502956
./bin/hadoop com.zvents.hadoop.DumpUrls /heritrix/crawls/test-4-20070116061502956 /foo
- Doug
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-895) Reduce hang on small set of
SequenceFiles (reproducible)
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12511797 ]
Arun C Murthy commented on HADOOP-895:
--------------------------------------
Doug, do you still see this with recent versions of hadoop?
Also HADOOP-1158 addresses parts of this problem...
> Reduce hang on small set of SequenceFiles (reproducible)
> --------------------------------------------------------
>
> Key: HADOOP-895
> URL: https://issues.apache.org/jira/browse/HADOOP-895
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.1
> Environment: This happens both on a mac and on a Linux x86_64 machine.
> Reporter: Doug Judd
> Priority: Blocker
> Attachments: hangmod-patchfile
>
>
> I'm running into a problem where a Map/Reduce job hangs in the reduce phase. It happens on a simple program to dump the keys of a small set of SequenceFiles. It can be reproduced as follows:
> 1. Pull a vanilla hadoop-0.10.1 release and untar it
> 2. Apply the attached patch
> 3. Modify conf/hadoop-env.sh to have the correct JAVA_HOME
> 4. Run the following script:
> #!/bin/sh
> ant zvents
> ./bin/hadoop namenode -format
> mkdir fs-hadoop/dfs/tmp
> mkdir fs-hadoop/dfs/data
> mkdir -p fs-hadoop/mapred/local
> ./bin/start-all.sh
> sleep 5
> ./bin/hadoop dfs -mkdir /heritrix
> ./bin/hadoop dfs -mkdir /heritrix/crawls
> ./bin/hadoop dfs -mkdir /heritrix/crawls/test-4-20070116061502956
> ./bin/hadoop dfs -copyFromLocal data/IAH-20070116061532-00000-doug-judds-computer.local /heritrix/crawls/test-4-20070116061502956
> ./bin/hadoop dfs -copyFromLocal data/IAH-20070116061539-00001-doug-judds-computer.local /heritrix/crawls/test-4-20070116061502956
> ./bin/hadoop dfs -copyFromLocal data/IAH-20070116061539-00002-doug-judds-computer.local /heritrix/crawls/test-4-20070116061502956
> ./bin/hadoop com.zvents.hadoop.DumpUrls /heritrix/crawls/test-4-20070116061502956 /foo
> - Doug
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-895) Reduce hang on small set of
SequenceFiles (reproducible)
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12496523 ]
Devaraj Das commented on HADOOP-895:
------------------------------------
Did you get a chance to test your app with the newer releases? Does this problem still exist?
> Reduce hang on small set of SequenceFiles (reproducible)
> --------------------------------------------------------
>
> Key: HADOOP-895
> URL: https://issues.apache.org/jira/browse/HADOOP-895
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.1
> Environment: This happens both on a mac and on a Linux x86_64 machine.
> Reporter: Doug Judd
> Priority: Blocker
> Attachments: hangmod-patchfile
>
>
> I'm running into a problem where a Map/Reduce job hangs in the reduce phase. It happens on a simple program to dump the keys of a small set of SequenceFiles. It can be reproduced as follows:
> 1. Pull a vanilla hadoop-0.10.1 release and untar it
> 2. Apply the attached patch
> 3. Modify conf/hadoop-env.sh to have the correct JAVA_HOME
> 4. Run the following script:
> #!/bin/sh
> ant zvents
> ./bin/hadoop namenode -format
> mkdir fs-hadoop/dfs/tmp
> mkdir fs-hadoop/dfs/data
> mkdir -p fs-hadoop/mapred/local
> ./bin/start-all.sh
> sleep 5
> ./bin/hadoop dfs -mkdir /heritrix
> ./bin/hadoop dfs -mkdir /heritrix/crawls
> ./bin/hadoop dfs -mkdir /heritrix/crawls/test-4-20070116061502956
> ./bin/hadoop dfs -copyFromLocal data/IAH-20070116061532-00000-doug-judds-computer.local /heritrix/crawls/test-4-20070116061502956
> ./bin/hadoop dfs -copyFromLocal data/IAH-20070116061539-00001-doug-judds-computer.local /heritrix/crawls/test-4-20070116061502956
> ./bin/hadoop dfs -copyFromLocal data/IAH-20070116061539-00002-doug-judds-computer.local /heritrix/crawls/test-4-20070116061502956
> ./bin/hadoop com.zvents.hadoop.DumpUrls /heritrix/crawls/test-4-20070116061502956 /foo
> - Doug
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-895) Reduce hang on small set of
SequenceFiles (reproducible)
Posted by "Doug Judd (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465677 ]
Doug Judd commented on HADOOP-895:
----------------------------------
I'm seeing the following messages (and Exception) in the tasktracker log file ...
2007-01-17 00:03:36,489 INFO org.apache.hadoop.mapred.TaskTracker: task_0001_r_000000_0 0.0% reduce > copy >
2007-01-17 00:03:41,493 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000000_0 Need 3 map output(s)
2007-01-17 00:03:41,494 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000000_0 Got 3 known map output location(s); scheduling...
2007-01-17 00:03:41,494 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000000_0 Scheduled 0 of 3 known outputs (3 slow hosts and 0 dup hosts)
2007-01-17 00:03:41,494 INFO org.apache.hadoop.mapred.TaskTracker: task_0001_r_000000_0 0.0% reduce > copy >
2007-01-17 00:03:46,498 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000000_0 Need 3 map output(s)
2007-01-17 00:03:46,498 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000000_0 Got 3 known map output location(s); scheduling...
2007-01-17 00:03:46,498 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000000_0 Scheduled 1 of 3 known outputs (0 slow hosts and 2 dup hosts)
2007-01-17 00:03:46,498 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000000_0 Copying task_0001_m_000001_0 output from motherlode000.zvents.com.
2007-01-17 00:03:46,501 WARN org.apache.hadoop.mapred.TaskRunner: task_0001_r_000000_0 copy failed: task_0001_m_000001_0 from motherlode000.zvents.com
2007-01-17 00:03:46,501 WARN org.apache.hadoop.mapred.TaskRunner: java.io.IOException: Server returned HTTP response code: 500 for URL: http://motherlode000.zvents.com:50060/mapOutput?map=task_000\
1_m_000001_0&reduce=0
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1174)
at org.apache.hadoop.mapred.MapOutputLocation.getFile(MapOutputLocation.java:121)
at org.apache.hadoop.mapred.ReduceTaskRunner$MapOutputCopier.copyOutput(ReduceTaskRunner.java:236)
at org.apache.hadoop.mapred.ReduceTaskRunner$MapOutputCopier.run(ReduceTaskRunner.java:199)
2007-01-17 00:03:46,501 WARN org.apache.hadoop.mapred.TaskRunner: task_0001_r_000000_0 adding host motherlode000.zvents.com to penalty box, next contact in 278 seconds
2007-01-17 00:03:46,501 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000000_0 Need 3 map output(s)
2007-01-17 00:03:46,501 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000000_0 Need 3 map output location(s)
2007-01-17 00:03:46,502 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000000_0 Got 3 map outputs from jobtracker
2007-01-17 00:03:46,502 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000000_0 Got 3 known map output location(s); scheduling...
2007-01-17 00:03:46,502 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000000_0 Scheduled 0 of 3 known outputs (3 slow hosts and 0 dup hosts)
2007-01-17 00:03:46,502 INFO org.apache.hadoop.mapred.TaskTracker: task_0001_r_000000_0 0.0% reduce > copy >
2007-01-17 00:03:51,506 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000000_0 Need 3 map output(s)
2007-01-17 00:03:51,506 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000000_0 Got 3 known map output location(s); scheduling...
2007-01-17 00:03:51,506 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000000_0 Scheduled 0 of 3 known outputs (3 slow hosts and 0 dup hosts)
2007-01-17 00:03:51,506 INFO org.apache.hadoop.mapred.TaskTracker: task_0001_r_000000_0 0.0% reduce > copy >
2007-01-17 00:03:56,510 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000000_0 Need 3 map output(s)
2007-01-17 00:03:56,510 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000000_0 Got 3 known map output location(s); scheduling...
2007-01-17 00:03:56,510 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000000_0 Scheduled 0 of 3 known outputs (3 slow hosts and 0 dup hosts)
2007-01-17 00:03:56,510 INFO org.apache.hadoop.mapred.TaskTracker: task_0001_r_000000_0 0.0% reduce > copy >
2007-01-17 00:04:01,514 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000000_0 Need 3 map output(s)
2007-01-17 00:04:01,514 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000000_0 Got 3 known map output location(s); scheduling...
2007-01-17 00:04:01,514 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000000_0 Scheduled 0 of 3 known outputs (3 slow hosts and 0 dup hosts)
2007-01-17 00:04:01,514 INFO org.apache.hadoop.mapred.TaskTracker: task_0001_r_000000_0 0.0% reduce > copy >
> Reduce hang on small set of SequenceFiles (reproducible)
> --------------------------------------------------------
>
> Key: HADOOP-895
> URL: https://issues.apache.org/jira/browse/HADOOP-895
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.1
> Environment: This happens both on a mac and on a Linux x86_64 machine.
> Reporter: Doug Judd
> Attachments: hangmod-patchfile
>
>
> I'm running into a problem where a Map/Reduce job hangs in the reduce phase. It happens on a simple program to dump the keys of a small set of SequenceFiles. It can be reproduced as follows:
> 1. Pull a vanilla hadoop-0.10.1 release and untar it
> 2. Apply the attached patch
> 3. Modify conf/hadoop-env.sh to have the correct JAVA_HOME
> 4. Run the following script:
> #!/bin/sh
> ant zvents
> ./bin/hadoop namenode -format
> mkdir fs-hadoop/dfs/tmp
> mkdir fs-hadoop/dfs/data
> mkdir -p fs-hadoop/mapred/local
> ./bin/start-all.sh
> sleep 5
> ./bin/hadoop dfs -mkdir /heritrix
> ./bin/hadoop dfs -mkdir /heritrix/crawls
> ./bin/hadoop dfs -mkdir /heritrix/crawls/test-4-20070116061502956
> ./bin/hadoop dfs -copyFromLocal data/IAH-20070116061532-00000-doug-judds-computer.local /heritrix/crawls/test-4-20070116061502956
> ./bin/hadoop dfs -copyFromLocal data/IAH-20070116061539-00001-doug-judds-computer.local /heritrix/crawls/test-4-20070116061502956
> ./bin/hadoop dfs -copyFromLocal data/IAH-20070116061539-00002-doug-judds-computer.local /heritrix/crawls/test-4-20070116061502956
> ./bin/hadoop com.zvents.hadoop.DumpUrls /heritrix/crawls/test-4-20070116061502956 /foo
> - Doug
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-895) Reduce hang on small set of
SequenceFiles (reproducible)
Posted by "Doug Judd (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Judd updated HADOOP-895:
-----------------------------
Priority: Blocker (was: Major)
> Reduce hang on small set of SequenceFiles (reproducible)
> --------------------------------------------------------
>
> Key: HADOOP-895
> URL: https://issues.apache.org/jira/browse/HADOOP-895
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.1
> Environment: This happens both on a mac and on a Linux x86_64 machine.
> Reporter: Doug Judd
> Priority: Blocker
> Attachments: hangmod-patchfile
>
>
> I'm running into a problem where a Map/Reduce job hangs in the reduce phase. It happens on a simple program to dump the keys of a small set of SequenceFiles. It can be reproduced as follows:
> 1. Pull a vanilla hadoop-0.10.1 release and untar it
> 2. Apply the attached patch
> 3. Modify conf/hadoop-env.sh to have the correct JAVA_HOME
> 4. Run the following script:
> #!/bin/sh
> ant zvents
> ./bin/hadoop namenode -format
> mkdir fs-hadoop/dfs/tmp
> mkdir fs-hadoop/dfs/data
> mkdir -p fs-hadoop/mapred/local
> ./bin/start-all.sh
> sleep 5
> ./bin/hadoop dfs -mkdir /heritrix
> ./bin/hadoop dfs -mkdir /heritrix/crawls
> ./bin/hadoop dfs -mkdir /heritrix/crawls/test-4-20070116061502956
> ./bin/hadoop dfs -copyFromLocal data/IAH-20070116061532-00000-doug-judds-computer.local /heritrix/crawls/test-4-20070116061502956
> ./bin/hadoop dfs -copyFromLocal data/IAH-20070116061539-00001-doug-judds-computer.local /heritrix/crawls/test-4-20070116061502956
> ./bin/hadoop dfs -copyFromLocal data/IAH-20070116061539-00002-doug-judds-computer.local /heritrix/crawls/test-4-20070116061502956
> ./bin/hadoop com.zvents.hadoop.DumpUrls /heritrix/crawls/test-4-20070116061502956 /foo
> - Doug
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-895) Reduce hang on small set of
SequenceFiles (reproducible)
Posted by "Doug Judd (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465686 ]
Doug Judd commented on HADOOP-895:
----------------------------------
This test program works fine on 0.9.2.
> Reduce hang on small set of SequenceFiles (reproducible)
> --------------------------------------------------------
>
> Key: HADOOP-895
> URL: https://issues.apache.org/jira/browse/HADOOP-895
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.1
> Environment: This happens both on a mac and on a Linux x86_64 machine.
> Reporter: Doug Judd
> Attachments: hangmod-patchfile
>
>
> I'm running into a problem where a Map/Reduce job hangs in the reduce phase. It happens on a simple program to dump the keys of a small set of SequenceFiles. It can be reproduced as follows:
> 1. Pull a vanilla hadoop-0.10.1 release and untar it
> 2. Apply the attached patch
> 3. Modify conf/hadoop-env.sh to have the correct JAVA_HOME
> 4. Run the following script:
> #!/bin/sh
> ant zvents
> ./bin/hadoop namenode -format
> mkdir fs-hadoop/dfs/tmp
> mkdir fs-hadoop/dfs/data
> mkdir -p fs-hadoop/mapred/local
> ./bin/start-all.sh
> sleep 5
> ./bin/hadoop dfs -mkdir /heritrix
> ./bin/hadoop dfs -mkdir /heritrix/crawls
> ./bin/hadoop dfs -mkdir /heritrix/crawls/test-4-20070116061502956
> ./bin/hadoop dfs -copyFromLocal data/IAH-20070116061532-00000-doug-judds-computer.local /heritrix/crawls/test-4-20070116061502956
> ./bin/hadoop dfs -copyFromLocal data/IAH-20070116061539-00001-doug-judds-computer.local /heritrix/crawls/test-4-20070116061502956
> ./bin/hadoop dfs -copyFromLocal data/IAH-20070116061539-00002-doug-judds-computer.local /heritrix/crawls/test-4-20070116061502956
> ./bin/hadoop com.zvents.hadoop.DumpUrls /heritrix/crawls/test-4-20070116061502956 /foo
> - Doug
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-895) Reduce hang on small set of
SequenceFiles (reproducible)
Posted by "Doug Judd (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Judd updated HADOOP-895:
-----------------------------
Attachment: hangmod-patchfile
> Reduce hang on small set of SequenceFiles (reproducible)
> --------------------------------------------------------
>
> Key: HADOOP-895
> URL: https://issues.apache.org/jira/browse/HADOOP-895
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.1
> Environment: This happens both on a mac and on a Linux x86_64 machine.
> Reporter: Doug Judd
> Attachments: hangmod-patchfile
>
>
> I'm running into a problem where a Map/Reduce job hangs in the reduce phase. It happens on a simple program to dump the keys of a small set of SequenceFiles. It can be reproduced as follows:
> 1. Pull a vanilla hadoop-0.10.1 release and untar it
> 2. Apply the attached patch
> 3. Modify conf/hadoop-env.sh to have the correct JAVA_HOME
> 4. Run the following script:
> #!/bin/sh
> ant zvents
> ./bin/hadoop namenode -format
> mkdir fs-hadoop/dfs/tmp
> mkdir fs-hadoop/dfs/data
> mkdir -p fs-hadoop/mapred/local
> ./bin/start-all.sh
> sleep 5
> ./bin/hadoop dfs -mkdir /heritrix
> ./bin/hadoop dfs -mkdir /heritrix/crawls
> ./bin/hadoop dfs -mkdir /heritrix/crawls/test-4-20070116061502956
> ./bin/hadoop dfs -copyFromLocal data/IAH-20070116061532-00000-doug-judds-computer.local /heritrix/crawls/test-4-20070116061502956
> ./bin/hadoop dfs -copyFromLocal data/IAH-20070116061539-00001-doug-judds-computer.local /heritrix/crawls/test-4-20070116061502956
> ./bin/hadoop dfs -copyFromLocal data/IAH-20070116061539-00002-doug-judds-computer.local /heritrix/crawls/test-4-20070116061502956
> ./bin/hadoop com.zvents.hadoop.DumpUrls /heritrix/crawls/test-4-20070116061502956 /foo
> - Doug
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (HADOOP-895) Reduce hang on small set of
SequenceFiles (reproducible)
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sameer Paranjpye resolved HADOOP-895.
-------------------------------------
Resolution: Cannot Reproduce
Doug, are you still seeing this? If so, I'll reopen the issue.
> Reduce hang on small set of SequenceFiles (reproducible)
> --------------------------------------------------------
>
> Key: HADOOP-895
> URL: https://issues.apache.org/jira/browse/HADOOP-895
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.1
> Environment: This happens both on a mac and on a Linux x86_64 machine.
> Reporter: Doug Judd
> Priority: Blocker
> Attachments: hangmod-patchfile
>
>
> I'm running into a problem where a Map/Reduce job hangs in the reduce phase. It happens on a simple program to dump the keys of a small set of SequenceFiles. It can be reproduced as follows:
> 1. Pull a vanilla hadoop-0.10.1 release and untar it
> 2. Apply the attached patch
> 3. Modify conf/hadoop-env.sh to have the correct JAVA_HOME
> 4. Run the following script:
> #!/bin/sh
> ant zvents
> ./bin/hadoop namenode -format
> mkdir fs-hadoop/dfs/tmp
> mkdir fs-hadoop/dfs/data
> mkdir -p fs-hadoop/mapred/local
> ./bin/start-all.sh
> sleep 5
> ./bin/hadoop dfs -mkdir /heritrix
> ./bin/hadoop dfs -mkdir /heritrix/crawls
> ./bin/hadoop dfs -mkdir /heritrix/crawls/test-4-20070116061502956
> ./bin/hadoop dfs -copyFromLocal data/IAH-20070116061532-00000-doug-judds-computer.local /heritrix/crawls/test-4-20070116061502956
> ./bin/hadoop dfs -copyFromLocal data/IAH-20070116061539-00001-doug-judds-computer.local /heritrix/crawls/test-4-20070116061502956
> ./bin/hadoop dfs -copyFromLocal data/IAH-20070116061539-00002-doug-judds-computer.local /heritrix/crawls/test-4-20070116061502956
> ./bin/hadoop com.zvents.hadoop.DumpUrls /heritrix/crawls/test-4-20070116061502956 /foo
> - Doug
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.