You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Wouter de Bie <wo...@spotify.com> on 2011/07/21 12:10:14 UTC
jets3t 0.7.4
Hi guys,
I've just trying to upgrade to jets3t 0.7.4 from 0.6.1, because the connection pool gets depleted after 20 requests. Now, I'm getting the following stack trace when trying to access s3. Does anyone have a clue how to solve this?
FAILED: Hive Internal Error: java.lang.RuntimeException(org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: Request Error. HEAD '/tmp' on Host MYBUCKET.s3.amazonaws.com' @ 'Thu, 21 Jul 2011 09:10:20 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId: 37E8E1DD4E8FC3D8, HostId: abkdf12EFCJBY9FyBwZrC9SEcSibtW7moeZHnSBdUoyKwIO8btedy0Q8AjdNQqop)
java.lang.RuntimeException: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: Request Error. HEAD '/tmp' on Host 'MYBUCKET.s3.amazonaws.com' @ 'Thu, 21 Jul 2011 09:10:20 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId: 37E8E1DD4E8FC3D8, HostId: abkdf12EFCJBY9FyBwZrC9SEcSibtW7moeZHnSBdUoyKwIO8btedy0Q8AjdNQqop
at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:151)
at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:190)
at org.apache.hadoop.hive.ql.Context.getMRTmpFileURI(Context.java:247)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:900)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:6594)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:238)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:340)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:736)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:456)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
Caused by: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: Request Error. HEAD '/tmp' on Host 'MYBUCKET.s3.amazonaws.com' @ 'Thu, 21 Jul 2011 09:10:20 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId: 37E8E1DD4E8FC3D8, HostId: abkdf12EFCJBY9FyBwZrC9SEcSibtW7moeZHnSBdUoyKwIO8btedy0Q8AjdNQqop
at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleServiceException(Jets3tNativeFileSystemStore.java:229)
at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:111)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at org.apache.hadoop.fs.s3native.$Proxy4.retrieveMetadata(Unknown Source)
at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:394)
at org.apache.hadoop.fs.s3native.NativeS3FileSystem.mkdir(NativeS3FileSystem.java:507)
at org.apache.hadoop.fs.s3native.NativeS3FileSystem.mkdirs(NativeS3FileSystem.java:500)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1226)
at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:147)
... 15 more
Caused by: org.jets3t.service.S3ServiceException: Request Error. HEAD '/tmp' on Host 'MYBUCKET.s3.amazonaws.com' @ 'Thu, 21 Jul 2011 09:10:20 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId: 37E8E1DD4E8FC3D8, HostId: abkdf12EFCJBY9FyBwZrC9SEcSibtW7moeZHnSBdUoyKwIO8btedy0Q8AjdNQqop
at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:520)
at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestHead(RestS3Service.java:868)
at org.jets3t.service.impl.rest.httpclient.RestS3Service.getObjectImpl(RestS3Service.java:2016)
at org.jets3t.service.impl.rest.httpclient.RestS3Service.getObjectDetailsImpl(RestS3Service.java:1944)
at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:3059)
at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:1940)
at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:103)
... 27 more
Caused by: org.jets3t.service.impl.rest.HttpException
at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:518)
... 33 more
--Wouter de Bie
Developer Business Intelligence, Spotify
wouter@spotify.com (mailto:wouter@spotify.com)
+46 72 018 0777
Re: jets3t 0.7.4
Posted by Wouter de Bie <wo...@spotify.com>.
And setting httpclient.max-connections=100 doesn't seem to be picked up.
--Wouter de Bie
Developer Business Intelligence, Spotify
wouter@spotify.com (mailto:wouter@spotify.com)
+46 72 018 0777
On Friday, July 22, 2011 at 4:54 PM, Wouter de Bie wrote:
> Hi,
>
> When I use 0.6.1, hive gets stuck at the following point after about 6 "select * from table limit 10":
>
> 2011-07-22 14:50:44,153 DEBUG httpclient.HttpClient (HttpClient.java:executeMethod(322)) - enter HttpClient.executeMethod(HttpMethod)
> 2011-07-22 14:50:44,153 DEBUG httpclient.HttpClient (HttpClient.java:executeMethod(373)) - enter HttpClient.executeMethod(HostConfiguration,HttpMethod,HttpState)
> 2011-07-22 14:50:44,153 DEBUG httpclient.MultiThreadedHttpConnectionManager (MultiThreadedHttpConnectionManager.java:getConnectionWithTimeout(383)) - enter HttpConnectionManager.getConnectionWithTimeout(HostConfiguration, long)
> 2011-07-22 14:50:44,154 DEBUG httpclient.MultiThreadedHttpConnectionManager (MultiThreadedHttpConnectionManager.java:getConnectionWithTimeout(390)) - HttpConnectionManager.getConnection: config = HostConfiguration[host=https://MYBUCKET.s3.amazonaws.com], timeout = 0
> 2011-07-22 14:50:44,154 DEBUG httpclient.MultiThreadedHttpConnectionManager (MultiThreadedHttpConnectionManager.java:getHostPool(775)) - enter HttpConnectionManager.ConnectionPool.getHostPool(HostConfiguration)
> 2011-07-22 14:50:44,155 DEBUG httpclient.MultiThreadedHttpConnectionManager (MultiThreadedHttpConnectionManager.java:doGetConnection(494)) - Unable to get a connection, waiting..., hostConfig=HostConfiguration[host=https://MYBUCKET.s3.amazonaws.com]
>
>
> --Wouter de Bie
> Developer Business Intelligence, Spotify
> wouter@spotify.com (mailto:wouter@spotify.com)
> +46 72 018 0777
>
> On Friday, July 22, 2011 at 11:30 AM, Florin Diaconeasa wrote:
>
> > Hi,
> >
> > First of all i think jets3t is called by hadoop and not by hive. Hive, from my knowledge, doesn't know about the storage system and i see that the jets3t jar is actually inside the lib directory of hadoop.
> >
> > Apparently there is a way to configure jets3t: http://www.jets3t.org/toolkit/configuration.html
> >
> > I tried putting a jets3t.properties file with httpclient.max-connections=100 in the conf dir of hadoop, but not sure it took into consideration.
> >
> > Also, another question: How did you reach the conclusion that the pool got depleted?
> >
> > On 21 July 2011 23:45, Wouter de Bie <wouter@spotify.com (mailto:wouter@spotify.com)> wrote:
> > > hive 0.7.0+27.1-2~maverick-cdh3 and hadoop 0.20.2+923.21-1
> > >
> > > --Wouter de Bie
> > > Developer Business Intelligence, Spotify
> > > wouter@spotify.com (mailto:wouter@spotify.com)
> > > +46 72 018 0777
> > >
> > > On Thursday, July 21, 2011 at 9:05 PM, Florin Diaconeasa wrote:
> > >
> > > > What hive version are you using?
> > > >
> > > > On Jul 21, 2011, at 1:10 PM, Wouter de Bie wrote:
> > > > > Hi guys,
> > > > >
> > > > > I've just trying to upgrade to jets3t 0.7.4 from 0.6.1, because the connection pool gets depleted after 20 requests. Now, I'm getting the following stack trace when trying to access s3. Does anyone have a clue how to solve this?
> > > > >
> > > > > FAILED: Hive Internal Error: java.lang.RuntimeException(org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: Request Error. HEAD '/tmp' on Host MYBUCKET.s3.amazonaws.com (http://MYBUCKET.s3.amazonaws.com)' @ 'Thu, 21 Jul 2011 09:10:20 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId: 37E8E1DD4E8FC3D8, HostId: abkdf12EFCJBY9FyBwZrC9SEcSibtW7moeZHnSBdUoyKwIO8btedy0Q8AjdNQqop)
> > > > > java.lang.RuntimeException: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: Request Error. HEAD '/tmp' on Host 'MYBUCKET.s3.amazonaws.com (http://MYBUCKET.s3.amazonaws.com)' @ 'Thu, 21 Jul 2011 09:10:20 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId: 37E8E1DD4E8FC3D8, HostId: abkdf12EFCJBY9FyBwZrC9SEcSibtW7moeZHnSBdUoyKwIO8btedy0Q8AjdNQqop
> > > > > at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:151)
> > > > > at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:190)
> > > > > at org.apache.hadoop.hive.ql.Context.getMRTmpFileURI(Context.java:247)
> > > > > at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:900)
> > > > > at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:6594)
> > > > > at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:238)
> > > > > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:340)
> > > > > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:736)
> > > > > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164)
> > > > > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
> > > > > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:456)
> > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > > > at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
> > > > > Caused by: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: Request Error. HEAD '/tmp' on Host 'MYBUCKET.s3.amazonaws.com (http://MYBUCKET.s3.amazonaws.com)' @ 'Thu, 21 Jul 2011 09:10:20 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId: 37E8E1DD4E8FC3D8, HostId: abkdf12EFCJBY9FyBwZrC9SEcSibtW7moeZHnSBdUoyKwIO8btedy0Q8AjdNQqop
> > > > > at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleServiceException(Jets3tNativeFileSystemStore.java:229)
> > > > > at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:111)
> > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > > > at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > > > > at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > > > > at org.apache.hadoop.fs.s3native.$Proxy4.retrieveMetadata(Unknown Source)
> > > > > at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:394)
> > > > > at org.apache.hadoop.fs.s3native.NativeS3FileSystem.mkdir(NativeS3FileSystem.java:507)
> > > > > at org.apache.hadoop.fs.s3native.NativeS3FileSystem.mkdirs(NativeS3FileSystem.java:500)
> > > > > at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1226)
> > > > > at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:147)
> > > > > ... 15 more
> > > > > Caused by: org.jets3t.service.S3ServiceException: Request Error. HEAD '/tmp' on Host 'MYBUCKET.s3.amazonaws.com (http://MYBUCKET.s3.amazonaws.com)' @ 'Thu, 21 Jul 2011 09:10:20 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId: 37E8E1DD4E8FC3D8, HostId: abkdf12EFCJBY9FyBwZrC9SEcSibtW7moeZHnSBdUoyKwIO8btedy0Q8AjdNQqop
> > > > > at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:520)
> > > > > at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestHead(RestS3Service.java:868)
> > > > > at org.jets3t.service.impl.rest.httpclient.RestS3Service.getObjectImpl(RestS3Service.java:2016)
> > > > > at org.jets3t.service.impl.rest.httpclient.RestS3Service.getObjectDetailsImpl(RestS3Service.java:1944)
> > > > > at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:3059)
> > > > > at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:1940)
> > > > > at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:103)
> > > > > ... 27 more
> > > > > Caused by: org.jets3t.service.impl.rest.HttpException
> > > > > at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:518)
> > > > > ... 33 more
> > > > >
> > > > >
> > > > > --Wouter de Bie
> > > > > Developer Business Intelligence, Spotify
> > > > > wouter@spotify.com (mailto:wouter@spotify.com)
> > > > > +46 72 018 0777
> > > >
> > >
> >
> >
> >
> > --
> >
> >
> > Florin
>
Re: jets3t 0.7.4
Posted by Wouter de Bie <wo...@spotify.com>.
Hi,
When I use 0.6.1, hive gets stuck at the following point after about 6 "select * from table limit 10":
2011-07-22 14:50:44,153 DEBUG httpclient.HttpClient (HttpClient.java:executeMethod(322)) - enter HttpClient.executeMethod(HttpMethod)
2011-07-22 14:50:44,153 DEBUG httpclient.HttpClient (HttpClient.java:executeMethod(373)) - enter HttpClient.executeMethod(HostConfiguration,HttpMethod,HttpState)
2011-07-22 14:50:44,153 DEBUG httpclient.MultiThreadedHttpConnectionManager (MultiThreadedHttpConnectionManager.java:getConnectionWithTimeout(383)) - enter HttpConnectionManager.getConnectionWithTimeout(HostConfiguration, long)
2011-07-22 14:50:44,154 DEBUG httpclient.MultiThreadedHttpConnectionManager (MultiThreadedHttpConnectionManager.java:getConnectionWithTimeout(390)) - HttpConnectionManager.getConnection: config = HostConfiguration[host=https://MYBUCKET.s3.amazonaws.com], timeout = 0
2011-07-22 14:50:44,154 DEBUG httpclient.MultiThreadedHttpConnectionManager (MultiThreadedHttpConnectionManager.java:getHostPool(775)) - enter HttpConnectionManager.ConnectionPool.getHostPool(HostConfiguration)
2011-07-22 14:50:44,155 DEBUG httpclient.MultiThreadedHttpConnectionManager (MultiThreadedHttpConnectionManager.java:doGetConnection(494)) - Unable to get a connection, waiting..., hostConfig=HostConfiguration[host=https://MYBUCKET.s3.amazonaws.com]
--Wouter de Bie
Developer Business Intelligence, Spotify
wouter@spotify.com (mailto:wouter@spotify.com)
+46 72 018 0777
On Friday, July 22, 2011 at 11:30 AM, Florin Diaconeasa wrote:
> Hi,
>
> First of all i think jets3t is called by hadoop and not by hive. Hive, from my knowledge, doesn't know about the storage system and i see that the jets3t jar is actually inside the lib directory of hadoop.
>
> Apparently there is a way to configure jets3t: http://www.jets3t.org/toolkit/configuration.html
>
> I tried putting a jets3t.properties file with httpclient.max-connections=100 in the conf dir of hadoop, but not sure it took into consideration.
>
> Also, another question: How did you reach the conclusion that the pool got depleted?
>
> On 21 July 2011 23:45, Wouter de Bie <wouter@spotify.com (mailto:wouter@spotify.com)> wrote:
> > hive 0.7.0+27.1-2~maverick-cdh3 and hadoop 0.20.2+923.21-1
> >
> > --Wouter de Bie
> > Developer Business Intelligence, Spotify
> > wouter@spotify.com (mailto:wouter@spotify.com)
> > +46 72 018 0777
> >
> > On Thursday, July 21, 2011 at 9:05 PM, Florin Diaconeasa wrote:
> >
> > > What hive version are you using?
> > >
> > > On Jul 21, 2011, at 1:10 PM, Wouter de Bie wrote:
> > > > Hi guys,
> > > >
> > > > I've just trying to upgrade to jets3t 0.7.4 from 0.6.1, because the connection pool gets depleted after 20 requests. Now, I'm getting the following stack trace when trying to access s3. Does anyone have a clue how to solve this?
> > > >
> > > > FAILED: Hive Internal Error: java.lang.RuntimeException(org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: Request Error. HEAD '/tmp' on Host MYBUCKET.s3.amazonaws.com (http://MYBUCKET.s3.amazonaws.com)' @ 'Thu, 21 Jul 2011 09:10:20 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId: 37E8E1DD4E8FC3D8, HostId: abkdf12EFCJBY9FyBwZrC9SEcSibtW7moeZHnSBdUoyKwIO8btedy0Q8AjdNQqop)
> > > > java.lang.RuntimeException: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: Request Error. HEAD '/tmp' on Host 'MYBUCKET.s3.amazonaws.com (http://MYBUCKET.s3.amazonaws.com)' @ 'Thu, 21 Jul 2011 09:10:20 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId: 37E8E1DD4E8FC3D8, HostId: abkdf12EFCJBY9FyBwZrC9SEcSibtW7moeZHnSBdUoyKwIO8btedy0Q8AjdNQqop
> > > > at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:151)
> > > > at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:190)
> > > > at org.apache.hadoop.hive.ql.Context.getMRTmpFileURI(Context.java:247)
> > > > at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:900)
> > > > at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:6594)
> > > > at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:238)
> > > > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:340)
> > > > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:736)
> > > > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164)
> > > > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
> > > > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:456)
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > > at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
> > > > Caused by: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: Request Error. HEAD '/tmp' on Host 'MYBUCKET.s3.amazonaws.com (http://MYBUCKET.s3.amazonaws.com)' @ 'Thu, 21 Jul 2011 09:10:20 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId: 37E8E1DD4E8FC3D8, HostId: abkdf12EFCJBY9FyBwZrC9SEcSibtW7moeZHnSBdUoyKwIO8btedy0Q8AjdNQqop
> > > > at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleServiceException(Jets3tNativeFileSystemStore.java:229)
> > > > at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:111)
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > > at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > > > at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > > > at org.apache.hadoop.fs.s3native.$Proxy4.retrieveMetadata(Unknown Source)
> > > > at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:394)
> > > > at org.apache.hadoop.fs.s3native.NativeS3FileSystem.mkdir(NativeS3FileSystem.java:507)
> > > > at org.apache.hadoop.fs.s3native.NativeS3FileSystem.mkdirs(NativeS3FileSystem.java:500)
> > > > at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1226)
> > > > at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:147)
> > > > ... 15 more
> > > > Caused by: org.jets3t.service.S3ServiceException: Request Error. HEAD '/tmp' on Host 'MYBUCKET.s3.amazonaws.com (http://MYBUCKET.s3.amazonaws.com)' @ 'Thu, 21 Jul 2011 09:10:20 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId: 37E8E1DD4E8FC3D8, HostId: abkdf12EFCJBY9FyBwZrC9SEcSibtW7moeZHnSBdUoyKwIO8btedy0Q8AjdNQqop
> > > > at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:520)
> > > > at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestHead(RestS3Service.java:868)
> > > > at org.jets3t.service.impl.rest.httpclient.RestS3Service.getObjectImpl(RestS3Service.java:2016)
> > > > at org.jets3t.service.impl.rest.httpclient.RestS3Service.getObjectDetailsImpl(RestS3Service.java:1944)
> > > > at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:3059)
> > > > at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:1940)
> > > > at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:103)
> > > > ... 27 more
> > > > Caused by: org.jets3t.service.impl.rest.HttpException
> > > > at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:518)
> > > > ... 33 more
> > > >
> > > >
> > > > --Wouter de Bie
> > > > Developer Business Intelligence, Spotify
> > > > wouter@spotify.com (mailto:wouter@spotify.com)
> > > > +46 72 018 0777
> > >
> >
>
>
>
> --
>
>
> Florin
Re: jets3t 0.7.4
Posted by Florin Diaconeasa <fl...@gmail.com>.
Hi,
First of all i think jets3t is called by hadoop and not by hive. Hive, from
my knowledge, doesn't know about the storage system and i see that the
jets3t jar is actually inside the lib directory of hadoop.
Apparently there is a way to configure jets3t:
http://www.jets3t.org/toolkit/configuration.html
I tried putting a jets3t.properties file with *
httpclient.max-connections=100* in the conf dir of hadoop, but not sure it
took into consideration.
Also, another question: How did you reach the conclusion that the pool got
depleted?
On 21 July 2011 23:45, Wouter de Bie <wo...@spotify.com> wrote:
> hive 0.7.0+27.1-2~maverick-cdh3 and hadoop 0.20.2+923.21-1
>
> --
> Wouter de Bie
> Developer Business Intelligence, Spotify
> wouter@spotify.com
> <pe...@spotify.com>+46 72 018 0777
>
> On Thursday, July 21, 2011 at 9:05 PM, Florin Diaconeasa wrote:
>
> What hive version are you using?
>
> On Jul 21, 2011, at 1:10 PM, Wouter de Bie wrote:
>
> Hi guys,
>
> I've just trying to upgrade to jets3t 0.7.4 from 0.6.1, because the
> connection pool gets depleted after 20 requests. Now, I'm getting the
> following stack trace when trying to access s3. Does anyone have a clue how
> to solve this?
>
> FAILED: Hive Internal Error:
> java.lang.RuntimeException(org.apache.hadoop.fs.s3.S3Exception:
> org.jets3t.service.S3ServiceException: Request Error. HEAD '/tmp' on Host
> MYBUCKET.s3.amazonaws.com' @ 'Thu, 21 Jul 2011 09:10:20 GMT' --
> ResponseCode: 404, ResponseStatus: Not Found, RequestId: 37E8E1DD4E8FC3D8,
> HostId: abkdf12EFCJBY9FyBwZrC9SEcSibtW7moeZHnSBdUoyKwIO8btedy0Q8AjdNQqop)
> java.lang.RuntimeException: org.apache.hadoop.fs.s3.S3Exception:
> org.jets3t.service.S3ServiceException: Request Error. HEAD '/tmp' on Host '
> MYBUCKET.s3.amazonaws.com' @ 'Thu, 21 Jul 2011 09:10:20 GMT' --
> ResponseCode: 404, ResponseStatus: Not Found, RequestId: 37E8E1DD4E8FC3D8,
> HostId: abkdf12EFCJBY9FyBwZrC9SEcSibtW7moeZHnSBdUoyKwIO8btedy0Q8AjdNQqop
> at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:151)
> at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:190)
> at org.apache.hadoop.hive.ql.Context.getMRTmpFileURI(Context.java:247)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:900)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:6594)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:238)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:340)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:736)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:456)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
> Caused by: org.apache.hadoop.fs.s3.S3Exception:
> org.jets3t.service.S3ServiceException: Request Error. HEAD '/tmp' on Host '
> MYBUCKET.s3.amazonaws.com' @ 'Thu, 21 Jul 2011 09:10:20 GMT' --
> ResponseCode: 404, ResponseStatus: Not Found, RequestId: 37E8E1DD4E8FC3D8,
> HostId: abkdf12EFCJBY9FyBwZrC9SEcSibtW7moeZHnSBdUoyKwIO8btedy0Q8AjdNQqop
> at
> org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleServiceException(Jets3tNativeFileSystemStore.java:229)
> at
> org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:111)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.fs.s3native.$Proxy4.retrieveMetadata(Unknown Source)
> at
> org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:394)
> at
> org.apache.hadoop.fs.s3native.NativeS3FileSystem.mkdir(NativeS3FileSystem.java:507)
> at
> org.apache.hadoop.fs.s3native.NativeS3FileSystem.mkdirs(NativeS3FileSystem.java:500)
> at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1226)
> at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:147)
> ... 15 more
> Caused by: org.jets3t.service.S3ServiceException: Request Error. HEAD
> '/tmp' on Host 'MYBUCKET.s3.amazonaws.com' @ 'Thu, 21 Jul 2011 09:10:20
> GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId:
> 37E8E1DD4E8FC3D8, HostId:
> abkdf12EFCJBY9FyBwZrC9SEcSibtW7moeZHnSBdUoyKwIO8btedy0Q8AjdNQqop
> at
> org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:520)
> at
> org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestHead(RestS3Service.java:868)
> at
> org.jets3t.service.impl.rest.httpclient.RestS3Service.getObjectImpl(RestS3Service.java:2016)
> at
> org.jets3t.service.impl.rest.httpclient.RestS3Service.getObjectDetailsImpl(RestS3Service.java:1944)
> at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:3059)
> at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:1940)
> at
> org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:103)
> ... 27 more
> Caused by: org.jets3t.service.impl.rest.HttpException
> at
> org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:518)
> ... 33 more
>
>
> --
> Wouter de Bie
> Developer Business Intelligence, Spotify
> wouter@spotify.com
> <pe...@spotify.com>+46 72 018 0777
>
>
>
>
--
Florin
Re: jets3t 0.7.4
Posted by Wouter de Bie <wo...@spotify.com>.
hive 0.7.0+27.1-2~maverick-cdh3 and hadoop 0.20.2+923.21-1
--Wouter de Bie
Developer Business Intelligence, Spotify
wouter@spotify.com (mailto:wouter@spotify.com)
+46 72 018 0777
On Thursday, July 21, 2011 at 9:05 PM, Florin Diaconeasa wrote:
> What hive version are you using?
>
> On Jul 21, 2011, at 1:10 PM, Wouter de Bie wrote:
> > Hi guys,
> >
> > I've just trying to upgrade to jets3t 0.7.4 from 0.6.1, because the connection pool gets depleted after 20 requests. Now, I'm getting the following stack trace when trying to access s3. Does anyone have a clue how to solve this?
> >
> > FAILED: Hive Internal Error: java.lang.RuntimeException(org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: Request Error. HEAD '/tmp' on Host MYBUCKET.s3.amazonaws.com (http://MYBUCKET.s3.amazonaws.com)' @ 'Thu, 21 Jul 2011 09:10:20 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId: 37E8E1DD4E8FC3D8, HostId: abkdf12EFCJBY9FyBwZrC9SEcSibtW7moeZHnSBdUoyKwIO8btedy0Q8AjdNQqop)
> > java.lang.RuntimeException: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: Request Error. HEAD '/tmp' on Host 'MYBUCKET.s3.amazonaws.com (http://MYBUCKET.s3.amazonaws.com)' @ 'Thu, 21 Jul 2011 09:10:20 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId: 37E8E1DD4E8FC3D8, HostId: abkdf12EFCJBY9FyBwZrC9SEcSibtW7moeZHnSBdUoyKwIO8btedy0Q8AjdNQqop
> > at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:151)
> > at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:190)
> > at org.apache.hadoop.hive.ql.Context.getMRTmpFileURI(Context.java:247)
> > at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:900)
> > at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:6594)
> > at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:238)
> > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:340)
> > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:736)
> > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164)
> > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
> > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:456)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
> > Caused by: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: Request Error. HEAD '/tmp' on Host 'MYBUCKET.s3.amazonaws.com (http://MYBUCKET.s3.amazonaws.com)' @ 'Thu, 21 Jul 2011 09:10:20 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId: 37E8E1DD4E8FC3D8, HostId: abkdf12EFCJBY9FyBwZrC9SEcSibtW7moeZHnSBdUoyKwIO8btedy0Q8AjdNQqop
> > at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleServiceException(Jets3tNativeFileSystemStore.java:229)
> > at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:111)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.fs.s3native.$Proxy4.retrieveMetadata(Unknown Source)
> > at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:394)
> > at org.apache.hadoop.fs.s3native.NativeS3FileSystem.mkdir(NativeS3FileSystem.java:507)
> > at org.apache.hadoop.fs.s3native.NativeS3FileSystem.mkdirs(NativeS3FileSystem.java:500)
> > at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1226)
> > at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:147)
> > ... 15 more
> > Caused by: org.jets3t.service.S3ServiceException: Request Error. HEAD '/tmp' on Host 'MYBUCKET.s3.amazonaws.com (http://MYBUCKET.s3.amazonaws.com)' @ 'Thu, 21 Jul 2011 09:10:20 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId: 37E8E1DD4E8FC3D8, HostId: abkdf12EFCJBY9FyBwZrC9SEcSibtW7moeZHnSBdUoyKwIO8btedy0Q8AjdNQqop
> > at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:520)
> > at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestHead(RestS3Service.java:868)
> > at org.jets3t.service.impl.rest.httpclient.RestS3Service.getObjectImpl(RestS3Service.java:2016)
> > at org.jets3t.service.impl.rest.httpclient.RestS3Service.getObjectDetailsImpl(RestS3Service.java:1944)
> > at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:3059)
> > at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:1940)
> > at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:103)
> > ... 27 more
> > Caused by: org.jets3t.service.impl.rest.HttpException
> > at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:518)
> > ... 33 more
> >
> >
> > --Wouter de Bie
> > Developer Business Intelligence, Spotify
> > wouter@spotify.com (mailto:wouter@spotify.com)
> > +46 72 018 0777
>
Re: jets3t 0.7.4
Posted by Florin Diaconeasa <fl...@gmail.com>.
What hive version are you using?
On Jul 21, 2011, at 1:10 PM, Wouter de Bie wrote:
> Hi guys,
>
> I've just trying to upgrade to jets3t 0.7.4 from 0.6.1, because the connection pool gets depleted after 20 requests. Now, I'm getting the following stack trace when trying to access s3. Does anyone have a clue how to solve this?
>
> FAILED: Hive Internal Error: java.lang.RuntimeException(org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: Request Error. HEAD '/tmp' on Host MYBUCKET.s3.amazonaws.com' @ 'Thu, 21 Jul 2011 09:10:20 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId: 37E8E1DD4E8FC3D8, HostId: abkdf12EFCJBY9FyBwZrC9SEcSibtW7moeZHnSBdUoyKwIO8btedy0Q8AjdNQqop)
> java.lang.RuntimeException: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: Request Error. HEAD '/tmp' on Host 'MYBUCKET.s3.amazonaws.com' @ 'Thu, 21 Jul 2011 09:10:20 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId: 37E8E1DD4E8FC3D8, HostId: abkdf12EFCJBY9FyBwZrC9SEcSibtW7moeZHnSBdUoyKwIO8btedy0Q8AjdNQqop
> at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:151)
> at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:190)
> at org.apache.hadoop.hive.ql.Context.getMRTmpFileURI(Context.java:247)
> at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:900)
> at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:6594)
> at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:238)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:340)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:736)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:456)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
> Caused by: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: Request Error. HEAD '/tmp' on Host 'MYBUCKET.s3.amazonaws.com' @ 'Thu, 21 Jul 2011 09:10:20 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId: 37E8E1DD4E8FC3D8, HostId: abkdf12EFCJBY9FyBwZrC9SEcSibtW7moeZHnSBdUoyKwIO8btedy0Q8AjdNQqop
> at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleServiceException(Jets3tNativeFileSystemStore.java:229)
> at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:111)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.fs.s3native.$Proxy4.retrieveMetadata(Unknown Source)
> at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:394)
> at org.apache.hadoop.fs.s3native.NativeS3FileSystem.mkdir(NativeS3FileSystem.java:507)
> at org.apache.hadoop.fs.s3native.NativeS3FileSystem.mkdirs(NativeS3FileSystem.java:500)
> at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1226)
> at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:147)
> ... 15 more
> Caused by: org.jets3t.service.S3ServiceException: Request Error. HEAD '/tmp' on Host 'MYBUCKET.s3.amazonaws.com' @ 'Thu, 21 Jul 2011 09:10:20 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId: 37E8E1DD4E8FC3D8, HostId: abkdf12EFCJBY9FyBwZrC9SEcSibtW7moeZHnSBdUoyKwIO8btedy0Q8AjdNQqop
> at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:520)
> at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestHead(RestS3Service.java:868)
> at org.jets3t.service.impl.rest.httpclient.RestS3Service.getObjectImpl(RestS3Service.java:2016)
> at org.jets3t.service.impl.rest.httpclient.RestS3Service.getObjectDetailsImpl(RestS3Service.java:1944)
> at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:3059)
> at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:1940)
> at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:103)
> ... 27 more
> Caused by: org.jets3t.service.impl.rest.HttpException
> at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:518)
> ... 33 more
>
>
> --
> Wouter de Bie
> Developer Business Intelligence, Spotify
> wouter@spotify.com
> +46 72 018 0777
>