hive error when trying to write data to s3n

[Pun Intended <pu...@gmail.com>]

Hello,

I have a query that used to work fine previously. I am testing on hive 1.1
now and it is failing. The AWS access and secret key have permissions to
read and write data to this directory. The directory exists.

hive -e " insert overwrite directory
's3n://<access_key>:<secret_key>@my_bucket/a/b/2015-07-30' SELECT * FROM
my_table WHERE date='2015-07-30';"

The errors that I get are:

1) I believe this is a non-critical error by reading the hive archives.
Logging initialized using configuration in
jar:file:/opt/cloudera/parcels/CDH-5.4.3-1.cdh5.4.3.p0.6/jars/hive-common-1.1.0-cdh5.4.3.jar!/hive-log4j.properties
-chgrp: '' does not match expected pattern for group
Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...


2) This, I believe is critical and the query fails:
Moving data to:
s3n://<access_key>:<secret_key>@my_bucket/a/b/2015-07-30/.hive-staging_hive_2015-08-04_18-38-47_649_1476668515119011800-1/-ext-10000
Failed with exception Wrong FS:
s3n://<access_key>:<secret_key>@my_bucket/a/b/2015-07-30/.hive-staging_hive_2015-08-04_18-38-47_649_1476668515119011800-1/-ext-10002,
expected: hdfs://somemachine.qe1.blah.com:8020
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask


Do you have any advice how I can fix this? Is this a known issue?

Thanks!
--py

Re: hive error when trying to write data to s3n

[Pun Intended <pu...@gmail.com>]

Yes, the explain plan definitely only has Move Operators (no Copy
Operators). With that though, this definitely looks like a hive bug? Does
anyone know if there is corresponding HIVE ticket or a workaround for the
issue? Thanks!

  Stage: Stage-3
    Move Operator
      files:
          hdfs directory: true
          destination:
s3n://<access_key>:<secret_key>@my_bucket/a/b/2015-07-30
/.hive-staging_hive_2015-08-04_20-38-31_599_7368878475721182549-1/-ext-10000

  Stage: Stage-0
    Move Operator
      files:
          hdfs directory: true
          destination:
s3n://<access_key>:<secret_key>@my_bucket/a/b/2015-07-30

  Stage: Stage-5
    Move Operator
      files:
          hdfs directory: true
          destination:
s3n://<access_key>:<secret_key>@my_bucket/a/b/2015-07-30
/.hive-staging_hive_2015-08-04_20-38-31_599_7368878475721182549-1/-ext-10000

On Tue, Aug 4, 2015 at 4:08 PM, Gopal Vijayaraghavan <go...@apache.org>
wrote:

>
> > Moving data to:
> >s3n://<access_key>:<secret_key>@my_bucket/a/b/2015-07-30/.hive-staging_hiv
> >e_2015-08-04_18-38-47_649_1476668515119011800-1/-ext-10000
> > Failed with exception Wrong FS:
> >s3n://<access_key>:<secret_key>@my_bucket/a/b/2015-07-30/.hive-staging_hiv
> >e_2015-08-04_18-38-47_649_1476668515119011800-1/-ext-10002, expected:
> >hdfs://s
> ..
> > FAILED: Execution Error, return code 1 from
> >org.apache.hadoop.hive.ql.exec.MoveTask
>
> Looks like the insert overwrite isn¹t triggering the CopyTask - MoveTask
> is the wrong operation for cross-FS operations.
>
> You can try looking at the Explain plan and confirm that.
>
> Cheers,
> Gopal
>
>
>

Re: hive error when trying to write data to s3n

[Gopal Vijayaraghavan <go...@apache.org>]

> Moving data to: 
>s3n://<access_key>:<secret_key>@my_bucket/a/b/2015-07-30/.hive-staging_hiv
>e_2015-08-04_18-38-47_649_1476668515119011800-1/-ext-10000
> Failed with exception Wrong FS:
>s3n://<access_key>:<secret_key>@my_bucket/a/b/2015-07-30/.hive-staging_hiv
>e_2015-08-04_18-38-47_649_1476668515119011800-1/-ext-10002, expected:
>hdfs://s
..
> FAILED: Execution Error, return code 1 from
>org.apache.hadoop.hive.ql.exec.MoveTask

Looks like the insert overwrite isn¹t triggering the CopyTask - MoveTask
is the wrong operation for cross-FS operations.

You can try looking at the Explain plan and confirm that.

Cheers,
Gopal