You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by hadoopman <ha...@gmail.com> on 2011/04/14 00:16:38 UTC
Hive regex SerDe issue?
Is there an issue with using the regex SerDe with loading into Hive text
files above 2 gigs in size? I've been experiencing out of memory errors
with a select group of logs when running a hive job. I have been able
to load the data if I use split to cut it in half or thirds. No problem.
Googling around and figured I'd ask while I'm digging into this.
Thanks!
Re: Hive regex SerDe issue?
Posted by hadoopman <ha...@gmail.com>.
I appreciate the feedback. I'll check in the morning. We're running
64bit Ubuntu 10.04 LTS server and I 'believe' it should all be 64 bit
but I'll verify that.
Thanks !!
On 04/13/2011 08:31 PM, Lance Norskog wrote:
> . Usually this is a positive-integer problem somewhere. If you use a
> 32-bit Java this would be a problem.
>
> On Wed, Apr 13, 2011 at 3:16 PM, hadoopman<ha...@gmail.com> wrote:
>
>> Is there an issue with using the regex SerDe with loading into Hive text
>> files above 2 gigs in size? I've been experiencing out of memory errors
>> with a select group of logs when running a hive job. I have been able to
>> load the data if I use split to cut it in half or thirds. No problem.
>>
>> Googling around and figured I'd ask while I'm digging into this.
>>
>> Thanks!
>>
>>
>
>
>
Re: Hive regex SerDe issue?
Posted by Lance Norskog <go...@gmail.com>.
. Usually this is a positive-integer problem somewhere. If you use a
32-bit Java this would be a problem.
On Wed, Apr 13, 2011 at 3:16 PM, hadoopman <ha...@gmail.com> wrote:
> Is there an issue with using the regex SerDe with loading into Hive text
> files above 2 gigs in size? I've been experiencing out of memory errors
> with a select group of logs when running a hive job. I have been able to
> load the data if I use split to cut it in half or thirds. No problem.
>
> Googling around and figured I'd ask while I'm digging into this.
>
> Thanks!
>
--
Lance Norskog
goksron@gmail.com