You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Guy Doulberg <Gu...@conduit.com> on 2011/05/22 10:38:22 UTC
EOFException
Hey,
I am dealing with a problem I just don't understand, maybe you could help me...
I have an external table, that is partitioned by date (for example '20110511').
I assembled a hive query that contains multiple insertions.
When I ran this query on small scale of partitions, lets say 5 days or a week, the query ends up successfully.
However, I need to run this query over 6 months of partitions, so I tried to run the query, the query has failed.
In the cluster log I could see that one mapper got EOFException, the location of the file was not specified.
I looked for a partition that points to an empty file.... all the locations of the partitions are bind to a directory that contain data.
I looked at the hadoop configuration xml.... I saw that in the mapred.input.dir configuration one of the input files looks like this:
hdfs://master001/tmp/hive-hive/hive_2011-05-22_12-16-46_680_69545572900013773/-mr-10009/1
I looked what there is under this location and found, a file called an emptyFile with 0kb.
When running the short query (over 5 days) this location doesn't exist in the mapred.input.dir configuration.
What do you think?
Thanks,
Guy Doulberg
Re: EOFException
Posted by Tom Hall <th...@gmail.com>.
Check out the xcievers
http://tech.backtype.com/the-dark-side-of-hadoop
Tom
On Sun, May 22, 2011 at 9:38 AM, Guy Doulberg <Gu...@conduit.com> wrote:
> Hey,
>
>
>
> I am dealing with a problem I just don't understand, maybe you could help
> me…
>
>
>
> I have an external table, that is partitioned by date (for example
> '20110511').
>
>
>
> I assembled a hive query that contains multiple insertions.
>
>
>
> When I ran this query on small scale of partitions, lets say 5 days or a
> week, the query ends up successfully.
>
>
>
> However, I need to run this query over 6 months of partitions, so I tried to
> run the query, the query has failed.
>
>
>
> In the cluster log I could see that one mapper got EOFException, the
> location of the file was not specified.
>
>
>
> I looked for a partition that points to an empty file…. all the locations of
> the partitions are bind to a directory that contain data.
>
>
>
> I looked at the hadoop configuration xml…. I saw that in the
> mapred.input.dir configuration one of the input files looks like this:
>
> hdfs://master001/tmp/hive-hive/hive_2011-05-22_12-16-46_680_69545572900013773/-mr-10009/1
>
>
>
> I looked what there is under this location and found, a file called an
> emptyFile with 0kb.
>
>
>
> When running the short query (over 5 days) this location doesn't exist in
> the mapred.input.dir configuration.
>
>
>
> What do you think?
>
>
>
>
>
> Thanks,
>
> Guy Doulberg
>
>