You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@sqoop.apache.org by Marcin Cylke <mc...@touk.pl> on 2012/03/07 16:54:57 UTC
Re: [ sqoopusers ] Re: import to hive hangs
On 07/03/12 15:28, Jarek Jarcec Cecho wrote:
> I'm not sure whether it's connected to your problem, but sqoop is designed to firstly import data into HDFS to a temporary directory and than using hive command "LOAD DATA" move them to target location in HIVE warehouse directory. You seems to be importing data directly into default HIVE warehouse directory (--targer-dir /user/hive/warehouse/sample_rm2). I would firstly try to change --target-dir parameter to simple "--target-dir sample_rm2" (directory with this name will be created in your HDFS home directory) or any other HDFS path outside /user/hive/warehouse.
This helped quite a lot I've again forgot about not writting into
hive warehouse directly.
Now, when I execute it with --target-dir sample_rm , and not full path,
sqoop finishes execution. However Hive is not aware of this table. The
appropriate directory is in /user/hive/warehouse, but is not visible via
'show tables' in hive shell, nor I can select any data from it.
Is there an easy fix for that?
Marcin
Re: import to hive hangs
Posted by Marcin Cylke <mc...@touk.pl>.
On 09/03/12 16:28, Arvind Prabhakar wrote:
> Hi Marcin,
>
> Thanks a lot for fixing the logging issue. In order for us to commit
> your patch, you would need to attach it to the Jira and grant license
> to it for inclusion. I have created the Jira for you at:
>
> https://issues.apache.org/jira/browse/SQOOP-460
Thanks, I've attached the patch. Mind that it's quick and dirty and is
rather intended as a heads-up - might require more general approach.
Marcin
Re: import to hive hangs
Posted by Arvind Prabhakar <ar...@apache.org>.
Hi Marcin,
Thanks a lot for fixing the logging issue. In order for us to commit
your patch, you would need to attach it to the Jira and grant license
to it for inclusion. I have created the Jira for you at:
https://issues.apache.org/jira/browse/SQOOP-460
Thanks,
Arvind Prabhakar
On Fri, Mar 9, 2012 at 5:58 AM, Marcin Cylke <mc...@touk.pl> wrote:
> On 08/03/12 08:29, Marcin Cylke wrote:
>>
>> On 07/03/12 18:01, Jarek Jarcec Cecho wrote:
>>>
>>> It's hard for me to guess what happened. Would you mind providing log
>>> file with --verbose flag set?
>>
>>
>> Here - I've attached sqoop verbose log as well as hive job output log
>> (the one present in the sqoop logs)
>
>
> I've realized that adding --verbose to sqoop does not change log category to
> DEBUG.
>
> I've applied this simple patch (sqoop.patch) against trunk and now the
> output of --verbose is a bit longer. Hope this helps.
>
> Marcin
>
Re: import to hive hangs
Posted by Marcin Cylke <mc...@touk.pl>.
On 08/03/12 08:29, Marcin Cylke wrote:
> On 07/03/12 18:01, Jarek Jarcec Cecho wrote:
>> It's hard for me to guess what happened. Would you mind providing log
>> file with --verbose flag set?
>
> Here - I've attached sqoop verbose log as well as hive job output log
> (the one present in the sqoop logs)
I've realized that adding --verbose to sqoop does not change log
category to DEBUG.
I've applied this simple patch (sqoop.patch) against trunk and now the
output of --verbose is a bit longer. Hope this helps.
Marcin
Re: import to hive hangs
Posted by Marcin Cylke <mc...@touk.pl>.
On 07/03/12 18:01, Jarek Jarcec Cecho wrote:
> It's hard for me to guess what happened. Would you mind providing log file with --verbose flag set?
Here - I've attached sqoop verbose log as well as hive job output log
(the one present in the sqoop logs)
Hope this helps with the problem.
Marcin
Re: [ sqoopusers ] Re: import to hive hangs
Posted by Jarek Jarcec Cecho <ja...@apache.org>.
It's hard for me to guess what happened. Would you mind providing log file with --verbose flag set?
Jarcec
On Mar 7, 2012, at 4:54 PM, Marcin Cylke wrote:
> On 07/03/12 15:28, Jarek Jarcec Cecho wrote:
>> I'm not sure whether it's connected to your problem, but sqoop is designed to firstly import data into HDFS to a temporary directory and than using hive command "LOAD DATA" move them to target location in HIVE warehouse directory. You seems to be importing data directly into default HIVE warehouse directory (--targer-dir /user/hive/warehouse/sample_rm2). I would firstly try to change --target-dir parameter to simple "--target-dir sample_rm2" (directory with this name will be created in your HDFS home directory) or any other HDFS path outside /user/hive/warehouse.
>
> This helped quite a lot I've again forgot about not writting into
> hive warehouse directly.
>
> Now, when I execute it with --target-dir sample_rm , and not full path,
> sqoop finishes execution. However Hive is not aware of this table. The
> appropriate directory is in /user/hive/warehouse, but is not visible via
> 'show tables' in hive shell, nor I can select any data from it.
>
> Is there an easy fix for that?
>
> Marcin
>