hadoop 0.16.3 problems to submit job

[Andreas Kostyrka <an...@kostyrka.org>]

Hi!

When I'm trying to submit a streaming job to my EC2/S3 cluster, I'm
getting the following errors:

additionalConfSpec_:null
null=@@@userJobConfProps_.get(stream.shipped.hadoopstreaming
packageJobJar: [/tmp/hadoop-hadoop/hadoop-unjar53992/] [] /tmp/streamjob53993.jar tmpDir=null
^[[1;5A08/04/21 14:26:11 WARN httpclient.RestS3Service: Retrying request - attempt 1 of 5
08/04/21 14:26:11 INFO httpclient.HttpMethodDirector: I/O exception (java.net.SocketException) caught when processing request: Connection reset
08/04/21 14:26:11 INFO httpclient.HttpMethodDirector: Retrying request
08/04/21 14:27:44 WARN httpclient.RestS3Service: Retrying request - attempt 2 of 5
08/04/21 14:27:44 INFO httpclient.HttpMethodDirector: I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out
08/04/21 14:27:44 INFO httpclient.HttpMethodDirector: Retrying request
08/04/21 14:32:12 WARN service.S3Service: Encountered 1 S3 Internal Server error(s), will retry in 50ms
08/04/21 14:32:21 WARN service.S3Service: Encountered 1 S3 Internal Server error(s), will retry in 50ms
08/04/21 14:32:26 WARN service.S3Service: Encountered 1 S3 Internal Server error(s), will retry in 50ms
08/04/21 14:32:31 WARN httpclient.RestS3Service: Retrying request - attempt 1 of 5
08/04/21 14:32:31 INFO httpclient.HttpMethodDirector: I/O exception (java.net.SocketException) caught when processing request: Connection reset
08/04/21 14:32:31 INFO httpclient.HttpMethodDirector: Retrying request
08/04/21 14:32:35 WARN httpclient.RestS3Service: Retrying request - attempt 1 of 5
08/04/21 14:32:35 INFO httpclient.HttpMethodDirector: I/O exception (java.net.SocketException) caught when processing request: Connection reset
08/04/21 14:32:35 INFO httpclient.HttpMethodDirector: Retrying request
08/04/21 14:36:06 WARN httpclient.RestS3Service: Retrying request - attempt 1 of 5
08/04/21 14:36:06 INFO httpclient.HttpMethodDirector: I/O exception (org.apache.commons.httpclient.NoHttpResponseException) caught when processing request: The server s3.amazonaws.com failed to respond
08/04/21 14:36:06 INFO httpclient.HttpMethodDirector: Retrying request
08/04/21 14:36:34 WARN httpclient.RestS3Service: Retrying request - attempt 2 of 5
08/04/21 14:36:34 INFO httpclient.HttpMethodDirector: I/O exception (java.net.SocketException) caught when processing request: Connection reset
08/04/21 14:36:34 INFO httpclient.HttpMethodDirector: Retrying request
08/04/21 14:41:47 WARN httpclient.RestS3Service: Retrying request - attempt 1 of 5
08/04/21 14:41:47 INFO httpclient.HttpMethodDirector: I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out
08/04/21 14:41:47 INFO httpclient.HttpMethodDirector: Retrying request
08/04/21 14:54:00 WARN httpclient.RestS3Service: Retrying request - attempt 1 of 5
08/04/21 14:54:00 INFO httpclient.HttpMethodDirector: I/O exception (java.net.SocketException) caught when processing request: Connection reset
08/04/21 14:54:00 INFO httpclient.HttpMethodDirector: Retrying request
08/04/21 15:04:48 INFO mapred.FileInputFormat: Total input paths to process : 18611
08/04/21 15:06:43 WARN httpclient.RestS3Service: Retrying request - attempt 1 of 5
08/04/21 15:06:43 INFO httpclient.HttpMethodDirector: I/O exception (org.apache.commons.httpclient.NoHttpResponseException) caught when processing request: The server s3.amazonaws.com failed to respond
08/04/21 15:06:43 INFO httpclient.HttpMethodDirector: Retrying request
08/04/21 15:14:33 WARN service.S3Service: Encountered 1 S3 Internal Server error(s), will retry in 50ms
08/04/21 15:14:34 WARN httpclient.RestS3Service: Retrying request - attempt 1 of 5
08/04/21 15:14:34 INFO httpclient.HttpMethodDirector: I/O exception (java.net.SocketException) caught when processing request: Connection reset
08/04/21 15:14:34 INFO httpclient.HttpMethodDirector: Retrying request
08/04/21 15:15:02 WARN httpclient.RestS3Service: Retrying request - attempt 1 of 5
08/04/21 15:15:02 INFO httpclient.HttpMethodDirector: I/O exception (java.net.SocketException) caught when processing request: Connection reset
08/04/21 15:15:02 INFO httpclient.HttpMethodDirector: Retrying request

Now I wonder if it's normal? Notice that it took hadoop 38 minutes to
figure out the number of input files.

Now this has been running for around an hour and I still have no idea if
the job will be submitted or will hang forever (I had such cases with
0.16.2 where I stopped the submission after half a day).

Any ideas?

TIA,

Andreas Kostyrka