HIVE and S3

[Balaji Rao <sb...@gmail.com>]

Hi,
   I had to post this question to this list because I feel there might
be a bug here.

I'm having problems with HIVE- EC2 reading files on S3 written by other tools

 I have a lot of files and folders on S3 created by s3cmd and utilized
by Elastic MapReduce (HIVE) and they work interchangeably, files
created by HIVE-EMR can be read by s3cmd and vice versa. However, I'm
having problems with HIVE/Hadoop running on EC2. Both Hive 0.7 and 0.8
seem to create an additional folder "/" on S3

 For example, if I have a file s3://bucket/path/00000 created by s3cmd
or HIVE-EMR and I try to create an external table on HIVE- EC2

 create external table wc(site string, cnt int) row format delimited
fields terminated by '\t' stored as textfile location
's3://bucket/path'

This does not recognize the EMR created s3 folders, instead I see a
new folder "/"

 <bucket> / "/" / path


When I look at the debug information, HIVE seems to be sending an
extra "/" when creating a table

Here is a debug message and if you see the path, there is a "/" and a
"%2f". Probably a bug in the code ?

hive> create external table wc(site string, cnt int) .... location
's3://masked/wcoverlay/';

      <StringToSign>GETWed, 07 Mar 2012 18:26:03
GMT/masked/%2Fwcoverlay</StringToSign><AWSAccessKeyId>.....


Am I missing something?

Thanks,
Balaji