You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Mich Talebzadeh <mi...@peridale.co.uk> on 2015/11/12 22:24:22 UTC
export/import in hive failing with nested directory exception!
Hi,
I am doing a simple export and import test from one database in Hive to
another database in the same Hive instance. I thought this would have been
straight forward.
hive> use asehadoop;
OK
Time taken: 0.35 seconds
hive> export table t to '/export/t';
Copying data from
file:/tmp/hive/af07f3eb-e2d2-4393-9aac-c3d1a83f2398/hive_2015-11-12_21-19-44
_375_659453881906265295-1/-local-10000/_metadata
Copying file:
file:/tmp/hive/af07f3eb-e2d2-4393-9aac-c3d1a83f2398/hive_2015-11-12_21-19-44
_375_659453881906265295-1/-local-10000/_metadata
Copying data from hdfs://rhes564:9000/user/hive/warehouse/asehadoop.db/t
Copying file:
hdfs://rhes564:9000/user/hive/warehouse/asehadoop.db/t/delta_0000055_0000055
Copying file:
hdfs://rhes564:9000/user/hive/warehouse/asehadoop.db/t/delta_0000056_0000056
Copying file:
hdfs://rhes564:9000/user/hive/warehouse/asehadoop.db/t/delta_0000057_0000057
Copying file:
hdfs://rhes564:9000/user/hive/warehouse/asehadoop.db/t/delta_0000058_0000058
Copying file:
hdfs://rhes564:9000/user/hive/warehouse/asehadoop.db/t/delta_0000059_0000059
Copying file:
hdfs://rhes564:9000/user/hive/warehouse/asehadoop.db/t/delta_0000060_0000060
Copying file:
hdfs://rhes564:9000/user/hive/warehouse/asehadoop.db/t/delta_0000061_0000061
Copying file:
hdfs://rhes564:9000/user/hive/warehouse/asehadoop.db/t/delta_0000062_0000062
OK
Time taken: 22.529 seconds
hive> use test;
OK
Time taken: 0.011 seconds
hive> import from '/export/t';
Copying data from hdfs://rhes564:9000/export/t/data
Copying file: hdfs://rhes564:9000/export/t/data/delta_0000055_0000055
Copying file: hdfs://rhes564:9000/export/t/data/delta_0000056_0000056
Copying file: hdfs://rhes564:9000/export/t/data/delta_0000057_0000057
Copying file: hdfs://rhes564:9000/export/t/data/delta_0000058_0000058
Copying file: hdfs://rhes564:9000/export/t/data/delta_0000059_0000059
Copying file: hdfs://rhes564:9000/export/t/data/delta_0000060_0000060
Copying file: hdfs://rhes564:9000/export/t/data/delta_0000061_0000061
Copying file: hdfs://rhes564:9000/export/t/data/delta_0000062_0000062
Loading data to table test.t
Failed with exception checkPaths:
hdfs://rhes564:9000/tmp/hive/hduser/af07f3eb-e2d2-4393-9aac-c3d1a83f2398/hiv
e_2015-11-12_21-20-43_538_674567030614000538-1/-ext-10000 has nested
directoryhdfs://rhes564:9000/tmp/hive/hduser/af07f3eb-e2d2-4393-9aac-c3d1a83
f2398/hive_2015-11-12_21-20-43_538_674567030614000538-1/-ext-10000/delta_000
0055_0000055
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask
Mich Talebzadeh
Sybase ASE 15 Gold Medal Award 2008
A Winning Strategy: Running the most Critical Financial Data on ASE 15
<http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908
.pdf>
http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.
pdf
Author of the books "A Practitioner's Guide to Upgrading to Sybase ASE 15",
ISBN 978-0-9563693-0-7.
co-author "Sybase Transact SQL Guidelines Best Practices", ISBN
978-0-9759693-0-4
Publications due shortly:
Complex Event Processing in Heterogeneous Environments, ISBN:
978-0-9563693-3-8
Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume
one out shortly
<http://talebzadehmich.wordpress.com/> http://talebzadehmich.wordpress.com
NOTE: The information in this email is proprietary and confidential. This
message is for the designated recipient only, if you are not the intended
recipient, you should destroy it immediately. Any information in this
message shall not be understood as given or endorsed by Peridale Technology
Ltd, its subsidiaries or their employees, unless expressly so stated. It is
the responsibility of the recipient to ensure that this email is virus free,
therefore neither Peridale Ltd, its subsidiaries nor their employees accept
any responsibility.
Re: export/import in hive failing with nested directory exception!
Posted by Lefty Leverenz <le...@gmail.com>.
This needs to be documented here: Limitations
<https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Limitations>
in
the Hive Transactions wikidoc.
-- Lefty
On Fri, Nov 13, 2015 at 12:39 AM, sreebalineni . <sr...@gmail.com>
wrote:
> Hello Gopal,
> Are there any plans for fixing this? Any idea?
>
> On Fri, Nov 13, 2015 at 6:31 AM, Gopal Vijayaraghavan <go...@apache.org>
> wrote:
>
>> Hi,
>>
>> >Thanks Gopal. Indeed table t is defined as ORC and transactional.
>> >
>> >Any reason why this should not work for transactional tables?
>>
>> The committed transactions list is actually missing from the exported
>> metadata.
>>
>> So the EXPORT as it exists today is a dirty read snapshot, which is not a
>> good thing when the data is continously being streamed in.
>>
>> I don't think IMPORT likes that (and why should it?).
>>
>> Try with a fully compacted table.
>>
>> Cheers,
>> Gopal
>>
>>
>>
>
RE: export/import in hive failing with nested directory exception!
Posted by Mich Talebzadeh <mi...@peridale.co.uk>.
This potentially breaks the ACID properties of Hive.
My own take is that export/import functionality was added before ACID properties and transactional tables were added to Hive and as such did not cater for this type of work.
There are two options IMO:
1. Update documentation for import/export to highlight the limitations with regard to transactional tables
2. Provide an alternative mechanism on how to migrate transactional tables.
HTH,
Mich Talebzadeh
Sybase ASE 15 Gold Medal Award 2008
A Winning Strategy: Running the most Critical Financial Data on ASE 15
http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf
Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15", ISBN 978-0-9563693-0-7.
co-author "Sybase Transact SQL Guidelines Best Practices", ISBN 978-0-9759693-0-4
Publications due shortly:
Complex Event Processing in Heterogeneous Environments, ISBN: 978-0-9563693-3-8
Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one out shortly
http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Technology Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.
From: sreebalineni . [mailto:sreebalineni@gmail.com]
Sent: 13 November 2015 05:39
To: user@hive.apache.org
Subject: Re: export/import in hive failing with nested directory exception!
Hello Gopal,
Are there any plans for fixing this? Any idea?
On Fri, Nov 13, 2015 at 6:31 AM, Gopal Vijayaraghavan <gopalv@apache.org <ma...@apache.org> > wrote:
Hi,
>Thanks Gopal. Indeed table t is defined as ORC and transactional.
>
>Any reason why this should not work for transactional tables?
The committed transactions list is actually missing from the exported
metadata.
So the EXPORT as it exists today is a dirty read snapshot, which is not a
good thing when the data is continously being streamed in.
I don't think IMPORT likes that (and why should it?).
Try with a fully compacted table.
Cheers,
Gopal
Re: export/import in hive failing with nested directory exception!
Posted by "sreebalineni ." <sr...@gmail.com>.
Hello Gopal,
Are there any plans for fixing this? Any idea?
On Fri, Nov 13, 2015 at 6:31 AM, Gopal Vijayaraghavan <go...@apache.org>
wrote:
> Hi,
>
> >Thanks Gopal. Indeed table t is defined as ORC and transactional.
> >
> >Any reason why this should not work for transactional tables?
>
> The committed transactions list is actually missing from the exported
> metadata.
>
> So the EXPORT as it exists today is a dirty read snapshot, which is not a
> good thing when the data is continously being streamed in.
>
> I don't think IMPORT likes that (and why should it?).
>
> Try with a fully compacted table.
>
> Cheers,
> Gopal
>
>
>
Re: export/import in hive failing with nested directory exception!
Posted by Gopal Vijayaraghavan <go...@apache.org>.
Hi,
>Thanks Gopal. Indeed table t is defined as ORC and transactional.
>
>Any reason why this should not work for transactional tables?
The committed transactions list is actually missing from the exported
metadata.
So the EXPORT as it exists today is a dirty read snapshot, which is not a
good thing when the data is continously being streamed in.
I don't think IMPORT likes that (and why should it?).
Try with a fully compacted table.
Cheers,
Gopal
RE: export/import in hive failing with nested directory exception!
Posted by Mich Talebzadeh <mi...@peridale.co.uk>.
Thanks Gopal. Indeed table t is defined as ORC and transactional.
Any reason why this should not work for transactional tables?
Regards,
Mich
NOTE: The information in this email is proprietary and confidential. This
message is for the designated recipient only, if you are not the intended
recipient, you should destroy it immediately. Any information in this
message shall not be understood as given or endorsed by Peridale Technology
Ltd, its subsidiaries or their employees, unless expressly so stated. It is
the responsibility of the recipient to ensure that this email is virus free,
therefore neither Peridale Ltd, its subsidiaries nor their employees accept
any responsibility.
-----Original Message-----
From: Gopal Vijayaraghavan [mailto:gopal@hortonworks.com] On Behalf Of Gopal
Vijayaraghavan
Sent: 13 November 2015 00:17
To: user@hive.apache.org
Subject: Re: export/import in hive failing with nested directory exception!
> I am doing a simple export and import test from one database in Hive
>to another database in the same Hive instance. I thought this would
>have been straight forward.
Not if ACID transactions are involved.
> Copying file:
>hdfs://rhes564:9000/user/hive/warehouse/asehadoop.db/t/delta_0000056_00
>000
>56
This should be fine for non transactional tables.
Cheers,
Gopal
Re: export/import in hive failing with nested directory exception!
Posted by Gopal Vijayaraghavan <go...@apache.org>.
> I am doing a simple export and import test from one database in Hive to
>another database in the same Hive instance. I thought this would have
>been straight forward.
Not if ACID transactions are involved.
> Copying file:
>hdfs://rhes564:9000/user/hive/warehouse/asehadoop.db/t/delta_0000056_00000
>56
This should be fine for non transactional tables.
Cheers,
Gopal