You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Matt Pestritto <ma...@pestritto.com> on 2009/10/01 16:52:47 UTC
Re: Hive-74
There were errors in the hive.log
2009-10-01 10:40:53,631 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.core.resources" but it cannot be resolved.
2009-10-01 10:40:53,631 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.core.resources" but it cannot be resolved.
2009-10-01 10:40:53,633 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.core.runtime" but it cannot be resolved.
2009-10-01 10:40:53,633 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.core.runtime" but it cannot be resolved.
2009-10-01 10:40:53,634 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.text" but it cannot be resolved.
2009-10-01 10:40:53,634 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.text" but it cannot be resolved.
2009-10-01 10:40:57,143 WARN mapred.JobClient
(JobClient.java:configureCommandLineOptions(539)) - Use GenericOptionsParser
for parsing the arguments. Applications should implement Tool for the same.
2009-10-01 10:40:58,609 ERROR exec.ExecDriver
(SessionState.java:printError(248)) - Ended Job = job_200909301537_0068 with
errors
2009-10-01 10:40:58,622 ERROR ql.Driver (SessionState.java:printError(248))
- FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.ExecDriver
On Wed, Sep 30, 2009 at 5:26 PM, Namit Jain <nj...@facebook.com> wrote:
> What you are doing seems OK ?
> Can you get the stack trace from /tmp/<username>/hive.log ?
>
>
>
>
>
> -----Original Message-----
> From: Matt Pestritto [mailto:matt@pestritto.com]
> Sent: Wednesday, September 30, 2009 6:51 AM
> To: hive-dev@hadoop.apache.org; hive-user@hadoop.apache.org
> Subject: Fwd: Hive-74
>
> Including hive-user in case someone has any experience with this..
> Thanks
> -Matt
>
> ---------- Forwarded message ----------
> From: Matt Pestritto <ma...@pestritto.com>
> Date: Tue, Sep 29, 2009 at 5:26 PM
> Subject: Hive-74
> To: hive-dev@hadoop.apache.org
>
>
> Hi-
>
> I'm having a problem using CombineHiveInputSplit. I believe this was
> patched in http://issues.apache.org/jira/browse/HIVE-74
>
> I'm currently running hadoop 20.1 using hive trunk.
>
> hive-default.xml has the following property:
> <property>
> <name>hive.input.format</name>
> <value></value>
> <description>The default input format, if it is not specified, the system
> assigns it. It is set to HiveInputFormat for hadoop versions 17, 18 and 19,
> whereas it is set to CombinedHiveInputFormat for hadoop 20. The user can
> always overwrite it - if there is a bug in CombinedHiveInputFormat, it can
> always be manually set to HiveInputFormat. </description>
> </property>
>
> I added the following to hive-site.xml: ( Notice, the description in
> hive-default.xml has CombinedHiveInputFormat which does not work for me -
> the property value seems to be Combine(-d) )
> <property>
> <name>hive.input.format</name>
> <value>org.apache.hadoop.hive.ql.io.CombineHiveInputFormat</value>
> <description>The default input format, if it is not specified, the system
> assigns it. It is set to HiveInputFormat for hadoop versions 17, 18 and 19,
> whereas it is set to CombinedHiveInputFormat for hadoop 20. The user can
> always overwrite it - if there is a bug in CombinedHiveInputFormat, it can
> always be manually set to HiveInputFormat. </description>
> </property>
>
> When I launch a job the cli exits immediately:
> hive> select count(1) from my_table;
> Total MapReduce jobs = 1
> Number of reduce tasks determined at compile time: 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=<number>
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=<number>
> In order to set a constant number of reducers:
> set mapred.reduce.tasks=<number>
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.ExecDriver
> hive> exit ;
>
> If I set the property value to
> org.apache.hadoop.hive.ql.io.HiveInputFormat,
> the job runs fine.
>
> Suggestions ? Is there something that I am missing ?
>
> Thanks
> -Matt
>
Re: Hive-74
Posted by Matt Pestritto <ma...@pestritto.com>.
Namit -
I have tried hive-trunk as of this afternoon and hive release 814942 (
revision with CombineHiveInputFormat commit ) .
Also - there are no logs that get generated on the tasktrackers for the
hadoop job that fails. The only log that is generated on the jobtracker is
the jobconf.
Thanks
-Matt
On Thu, Oct 8, 2009 at 1:26 AM, Namit Jain <nj...@facebook.com> wrote:
> Hi Matt,
>
> Sorry for the late reply.
>
> hive> set
> hive.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat;
>
> I tried it running on hadoop 20 and it ran fine for me.
>
> Which hive release are you using ?
>
> Also, you got a runtime error – can you see the stderr logs on the tracker
> ?
>
> Thanks,
> -namit
>
>
>
> On 10/1/09 5:01 PM, "Matt Pestritto" <ma...@pestritto.com> wrote:
>
> Namit -
> Any idea on how to resolve ?
> Thanks
>
> On Thu, Oct 1, 2009 at 10:52 AM, Matt Pestritto <ma...@pestritto.com>
> wrote:
>
> > There were errors in the hive.log
> >
> > 2009-10-01 10:40:53,631 ERROR DataNucleus.Plugin
> > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> > "org.eclipse.core.resources" but it cannot be resolved.
> > 2009-10-01 10:40:53,631 ERROR DataNucleus.Plugin
> > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> > "org.eclipse.core.resources" but it cannot be resolved.
> > 2009-10-01 10:40:53,633 ERROR DataNucleus.Plugin
> > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> > "org.eclipse.core.runtime" but it cannot be resolved.
> > 2009-10-01 10:40:53,633 ERROR DataNucleus.Plugin
> > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> > "org.eclipse.core.runtime" but it cannot be resolved.
> > 2009-10-01 10:40:53,634 ERROR DataNucleus.Plugin
> > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> > "org.eclipse.text" but it cannot be resolved.
> > 2009-10-01 10:40:53,634 ERROR DataNucleus.Plugin
> > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> > "org.eclipse.text" but it cannot be resolved.
> > 2009-10-01 10:40:57,143 WARN mapred.JobClient
> > (JobClient.java:configureCommandLineOptions(539)) - Use
> GenericOptionsParser
> > for parsing the arguments. Applications should implement Tool for the
> same.
> > 2009-10-01 10:40:58,609 ERROR exec.ExecDriver
> > (SessionState.java:printError(248)) - Ended Job = job_200909301537_0068
> with
> > errors
> > 2009-10-01 10:40:58,622 ERROR ql.Driver
> (SessionState.java:printError(248))
> > - FAILED: Execution Error, return code 2 from
> > org.apache.hadoop.hive.ql.exec.ExecDriver
> >
> >
> >
> > On Wed, Sep 30, 2009 at 5:26 PM, Namit Jain <nj...@facebook.com> wrote:
> >
> >> What you are doing seems OK ?
> >> Can you get the stack trace from /tmp/<username>/hive.log ?
> >>
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Matt Pestritto [mailto:matt@pestritto.com] <matt@pestritto.com]>
> >> Sent: Wednesday, September 30, 2009 6:51 AM
> >> To: hive-dev@hadoop.apache.org; hive-user@hadoop.apache.org
> >> Subject: Fwd: Hive-74
> >>
> >> Including hive-user in case someone has any experience with this..
> >> Thanks
> >> -Matt
> >>
> >> ---------- Forwarded message ----------
> >> From: Matt Pestritto <ma...@pestritto.com>
> >> Date: Tue, Sep 29, 2009 at 5:26 PM
> >> Subject: Hive-74
> >> To: hive-dev@hadoop.apache.org
> >>
> >>
> >> Hi-
> >>
> >> I'm having a problem using CombineHiveInputSplit. I believe this was
> >> patched in http://issues.apache.org/jira/browse/HIVE-74
> >>
> >> I'm currently running hadoop 20.1 using hive trunk.
> >>
> >> hive-default.xml has the following property:
> >> <property>
> >> <name>hive.input.format</name>
> >> <value></value>
> >> <description>The default input format, if it is not specified, the
> system
> >> assigns it. It is set to HiveInputFormat for hadoop versions 17, 18 and
> >> 19,
> >> whereas it is set to CombinedHiveInputFormat for hadoop 20. The user can
> >> always overwrite it - if there is a bug in CombinedHiveInputFormat, it
> can
> >> always be manually set to HiveInputFormat. </description>
> >> </property>
> >>
> >> I added the following to hive-site.xml: ( Notice, the description in
> >> hive-default.xml has CombinedHiveInputFormat which does not work for me
> -
> >> the property value seems to be Combine(-d) )
> >> <property>
> >> <name>hive.input.format</name>
> >> <value>org.apache.hadoop.hive.ql.io.CombineHiveInputFormat</value>
> >> <description>The default input format, if it is not specified, the
> system
> >> assigns it. It is set to HiveInputFormat for hadoop versions 17, 18 and
> >> 19,
> >> whereas it is set to CombinedHiveInputFormat for hadoop 20. The user can
> >> always overwrite it - if there is a bug in CombinedHiveInputFormat, it
> can
> >> always be manually set to HiveInputFormat. </description>
> >> </property>
> >>
> >> When I launch a job the cli exits immediately:
> >> hive> select count(1) from my_table;
> >> Total MapReduce jobs = 1
> >> Number of reduce tasks determined at compile time: 1
> >> In order to change the average load for a reducer (in bytes):
> >> set hive.exec.reducers.bytes.per.reducer=<number>
> >> In order to limit the maximum number of reducers:
> >> set hive.exec.reducers.max=<number>
> >> In order to set a constant number of reducers:
> >> set mapred.reduce.tasks=<number>
> >> FAILED: Execution Error, return code 2 from
> >> org.apache.hadoop.hive.ql.exec.ExecDriver
> >> hive> exit ;
> >>
> >> If I set the property value to
> >> org.apache.hadoop.hive.ql.io.HiveInputFormat,
> >> the job runs fine.
> >>
> >> Suggestions ? Is there something that I am missing ?
> >>
> >> Thanks
> >> -Matt
> >>
> >
> >
>
>
Re: Hive-74
Posted by Matt Pestritto <ma...@pestritto.com>.
Namit -
I have tried hive-trunk as of this afternoon and hive release 814942 (
revision with CombineHiveInputFormat commit ) .
Also - there are no logs that get generated on the tasktrackers for the
hadoop job that fails. The only log that is generated on the jobtracker is
the jobconf.
Thanks
-Matt
On Thu, Oct 8, 2009 at 1:26 AM, Namit Jain <nj...@facebook.com> wrote:
> Hi Matt,
>
> Sorry for the late reply.
>
> hive> set
> hive.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat;
>
> I tried it running on hadoop 20 and it ran fine for me.
>
> Which hive release are you using ?
>
> Also, you got a runtime error – can you see the stderr logs on the tracker
> ?
>
> Thanks,
> -namit
>
>
>
> On 10/1/09 5:01 PM, "Matt Pestritto" <ma...@pestritto.com> wrote:
>
> Namit -
> Any idea on how to resolve ?
> Thanks
>
> On Thu, Oct 1, 2009 at 10:52 AM, Matt Pestritto <ma...@pestritto.com>
> wrote:
>
> > There were errors in the hive.log
> >
> > 2009-10-01 10:40:53,631 ERROR DataNucleus.Plugin
> > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> > "org.eclipse.core.resources" but it cannot be resolved.
> > 2009-10-01 10:40:53,631 ERROR DataNucleus.Plugin
> > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> > "org.eclipse.core.resources" but it cannot be resolved.
> > 2009-10-01 10:40:53,633 ERROR DataNucleus.Plugin
> > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> > "org.eclipse.core.runtime" but it cannot be resolved.
> > 2009-10-01 10:40:53,633 ERROR DataNucleus.Plugin
> > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> > "org.eclipse.core.runtime" but it cannot be resolved.
> > 2009-10-01 10:40:53,634 ERROR DataNucleus.Plugin
> > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> > "org.eclipse.text" but it cannot be resolved.
> > 2009-10-01 10:40:53,634 ERROR DataNucleus.Plugin
> > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> > "org.eclipse.text" but it cannot be resolved.
> > 2009-10-01 10:40:57,143 WARN mapred.JobClient
> > (JobClient.java:configureCommandLineOptions(539)) - Use
> GenericOptionsParser
> > for parsing the arguments. Applications should implement Tool for the
> same.
> > 2009-10-01 10:40:58,609 ERROR exec.ExecDriver
> > (SessionState.java:printError(248)) - Ended Job = job_200909301537_0068
> with
> > errors
> > 2009-10-01 10:40:58,622 ERROR ql.Driver
> (SessionState.java:printError(248))
> > - FAILED: Execution Error, return code 2 from
> > org.apache.hadoop.hive.ql.exec.ExecDriver
> >
> >
> >
> > On Wed, Sep 30, 2009 at 5:26 PM, Namit Jain <nj...@facebook.com> wrote:
> >
> >> What you are doing seems OK ?
> >> Can you get the stack trace from /tmp/<username>/hive.log ?
> >>
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Matt Pestritto [mailto:matt@pestritto.com] <matt@pestritto.com]>
> >> Sent: Wednesday, September 30, 2009 6:51 AM
> >> To: hive-dev@hadoop.apache.org; hive-user@hadoop.apache.org
> >> Subject: Fwd: Hive-74
> >>
> >> Including hive-user in case someone has any experience with this..
> >> Thanks
> >> -Matt
> >>
> >> ---------- Forwarded message ----------
> >> From: Matt Pestritto <ma...@pestritto.com>
> >> Date: Tue, Sep 29, 2009 at 5:26 PM
> >> Subject: Hive-74
> >> To: hive-dev@hadoop.apache.org
> >>
> >>
> >> Hi-
> >>
> >> I'm having a problem using CombineHiveInputSplit. I believe this was
> >> patched in http://issues.apache.org/jira/browse/HIVE-74
> >>
> >> I'm currently running hadoop 20.1 using hive trunk.
> >>
> >> hive-default.xml has the following property:
> >> <property>
> >> <name>hive.input.format</name>
> >> <value></value>
> >> <description>The default input format, if it is not specified, the
> system
> >> assigns it. It is set to HiveInputFormat for hadoop versions 17, 18 and
> >> 19,
> >> whereas it is set to CombinedHiveInputFormat for hadoop 20. The user can
> >> always overwrite it - if there is a bug in CombinedHiveInputFormat, it
> can
> >> always be manually set to HiveInputFormat. </description>
> >> </property>
> >>
> >> I added the following to hive-site.xml: ( Notice, the description in
> >> hive-default.xml has CombinedHiveInputFormat which does not work for me
> -
> >> the property value seems to be Combine(-d) )
> >> <property>
> >> <name>hive.input.format</name>
> >> <value>org.apache.hadoop.hive.ql.io.CombineHiveInputFormat</value>
> >> <description>The default input format, if it is not specified, the
> system
> >> assigns it. It is set to HiveInputFormat for hadoop versions 17, 18 and
> >> 19,
> >> whereas it is set to CombinedHiveInputFormat for hadoop 20. The user can
> >> always overwrite it - if there is a bug in CombinedHiveInputFormat, it
> can
> >> always be manually set to HiveInputFormat. </description>
> >> </property>
> >>
> >> When I launch a job the cli exits immediately:
> >> hive> select count(1) from my_table;
> >> Total MapReduce jobs = 1
> >> Number of reduce tasks determined at compile time: 1
> >> In order to change the average load for a reducer (in bytes):
> >> set hive.exec.reducers.bytes.per.reducer=<number>
> >> In order to limit the maximum number of reducers:
> >> set hive.exec.reducers.max=<number>
> >> In order to set a constant number of reducers:
> >> set mapred.reduce.tasks=<number>
> >> FAILED: Execution Error, return code 2 from
> >> org.apache.hadoop.hive.ql.exec.ExecDriver
> >> hive> exit ;
> >>
> >> If I set the property value to
> >> org.apache.hadoop.hive.ql.io.HiveInputFormat,
> >> the job runs fine.
> >>
> >> Suggestions ? Is there something that I am missing ?
> >>
> >> Thanks
> >> -Matt
> >>
> >
> >
>
>
Re: Hive-74
Posted by Matt Pestritto <ma...@pestritto.com>.
Thanks Namit -
The patch worked for me.
On Mon, Oct 19, 2009 at 1:23 PM, Namit Jain <nj...@facebook.com> wrote:
> Yes, we also ran into this problem, and Zheng has a patch for this.
>
> Either you can apply the patch on hadoop and get it to work, or copy the
> code in
>
> Hive and have a hive patch. I am not sure if that is the best approach,
> since that will
>
> lead to code duplication.
>
>
>
>
>
> Thanks,
>
> -namit
>
>
>
>
>
> *From:* Matt Pestritto [mailto:matt@pestritto.com]
> *Sent:* Monday, October 19, 2009 10:20 AM
> *To:* hive-user@hadoop.apache.org
> *Subject:* Re: Hive-74
>
>
>
> Namit -
>
> I finally had a chance to look at this again. I am running hadoop 20.1 and
> hive trunk.
>
> I'm still having a problem with combine input. I found an error in my
> jobtracker logs:
>
> 12:48:08,792 INFO [JobInProgress] Input size for job job_200910160957_0003 = 13. Number of splits = 1
>
> 12:48:08,794 ERROR [JobTracker] Job initialization failed:
>
> java.lang.IllegalArgumentException: Network location name contains /: /default-rack
>
> at org.apache.hadoop.net.NodeBase.set(NodeBase.java:75)
>
> at org.apache.hadoop.net.NodeBase.<init>(NodeBase.java:57)
>
> at org.apache.hadoop.mapred.JobTracker.addHostToNodeMapping(JobTracker.java:2390)
>
> at org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2384)
>
> at org.apache.hadoop.mapred.JobInProgress.createCache(JobInProgress.java:349)
>
> at org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:450)
>
> at org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:3147)
>
> at org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:79)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:619)
>
> 12:48:08,794 INFO [JobTracker] Failing job job_200910160957_0003
>
> 12:48:09,560 INFO [JobTracker] Killing job job_200910160957_0003
>
> This seems to be related to hadoop-5759 but it looks like it was patched
> coincidently today by Zheng.
> https://issues.apache.org/jira/browse/HADOOP-5759
>
> I'm guessing I need to build hadoop from 20 branch for this to work ?
>
> Thanks
> -Matt
>
> On Thu, Oct 8, 2009 at 1:26 AM, Namit Jain <nj...@facebook.com> wrote:
>
> Hi Matt,
>
> Sorry for the late reply.
>
> hive> set
> hive.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat;
>
> I tried it running on hadoop 20 and it ran fine for me.
>
> Which hive release are you using ?
>
> Also, you got a runtime error – can you see the stderr logs on the tracker
> ?
>
> Thanks,
> -namit
>
>
>
>
> On 10/1/09 5:01 PM, "Matt Pestritto" <ma...@pestritto.com> wrote:
>
> Namit -
> Any idea on how to resolve ?
> Thanks
>
> On Thu, Oct 1, 2009 at 10:52 AM, Matt Pestritto <ma...@pestritto.com>
> wrote:
>
> > There were errors in the hive.log
> >
> > 2009-10-01 10:40:53,631 ERROR DataNucleus.Plugin
> > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> > "org.eclipse.core.resources" but it cannot be resolved.
> > 2009-10-01 10:40:53,631 ERROR DataNucleus.Plugin
> > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> > "org.eclipse.core.resources" but it cannot be resolved.
> > 2009-10-01 10:40:53,633 ERROR DataNucleus.Plugin
> > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> > "org.eclipse.core.runtime" but it cannot be resolved.
> > 2009-10-01 10:40:53,633 ERROR DataNucleus.Plugin
> > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> > "org.eclipse.core.runtime" but it cannot be resolved.
> > 2009-10-01 10:40:53,634 ERROR DataNucleus.Plugin
> > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> > "org.eclipse.text" but it cannot be resolved.
> > 2009-10-01 10:40:53,634 ERROR DataNucleus.Plugin
> > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> > "org.eclipse.text" but it cannot be resolved.
> > 2009-10-01 10:40:57,143 WARN mapred.JobClient
> > (JobClient.java:configureCommandLineOptions(539)) - Use
> GenericOptionsParser
> > for parsing the arguments. Applications should implement Tool for the
> same.
> > 2009-10-01 10:40:58,609 ERROR exec.ExecDriver
> > (SessionState.java:printError(248)) - Ended Job = job_200909301537_0068
> with
> > errors
> > 2009-10-01 10:40:58,622 ERROR ql.Driver
> (SessionState.java:printError(248))
> > - FAILED: Execution Error, return code 2 from
> > org.apache.hadoop.hive.ql.exec.ExecDriver
> >
> >
> >
> > On Wed, Sep 30, 2009 at 5:26 PM, Namit Jain <nj...@facebook.com> wrote:
> >
> >> What you are doing seems OK ?
> >> Can you get the stack trace from /tmp/<username>/hive.log ?
> >>
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Matt Pestritto [mailto:matt@pestritto.com] <matt@pestritto.com]>
> >> Sent: Wednesday, September 30, 2009 6:51 AM
> >> To: hive-dev@hadoop.apache.org; hive-user@hadoop.apache.org
> >> Subject: Fwd: Hive-74
> >>
> >> Including hive-user in case someone has any experience with this..
> >> Thanks
> >> -Matt
> >>
> >> ---------- Forwarded message ----------
> >> From: Matt Pestritto <ma...@pestritto.com>
> >> Date: Tue, Sep 29, 2009 at 5:26 PM
> >> Subject: Hive-74
> >> To: hive-dev@hadoop.apache.org
> >>
> >>
> >> Hi-
> >>
> >> I'm having a problem using CombineHiveInputSplit. I believe this was
> >> patched in http://issues.apache.org/jira/browse/HIVE-74
> >>
> >> I'm currently running hadoop 20.1 using hive trunk.
> >>
> >> hive-default.xml has the following property:
> >> <property>
> >> <name>hive.input.format</name>
> >> <value></value>
> >> <description>The default input format, if it is not specified, the
> system
> >> assigns it. It is set to HiveInputFormat for hadoop versions 17, 18 and
> >> 19,
> >> whereas it is set to CombinedHiveInputFormat for hadoop 20. The user can
> >> always overwrite it - if there is a bug in CombinedHiveInputFormat, it
> can
> >> always be manually set to HiveInputFormat. </description>
> >> </property>
> >>
> >> I added the following to hive-site.xml: ( Notice, the description in
> >> hive-default.xml has CombinedHiveInputFormat which does not work for me
> -
> >> the property value seems to be Combine(-d) )
> >> <property>
> >> <name>hive.input.format</name>
> >> <value>org.apache.hadoop.hive.ql.io.CombineHiveInputFormat</value>
> >> <description>The default input format, if it is not specified, the
> system
> >> assigns it. It is set to HiveInputFormat for hadoop versions 17, 18 and
> >> 19,
> >> whereas it is set to CombinedHiveInputFormat for hadoop 20. The user can
> >> always overwrite it - if there is a bug in CombinedHiveInputFormat, it
> can
> >> always be manually set to HiveInputFormat. </description>
> >> </property>
> >>
> >> When I launch a job the cli exits immediately:
> >> hive> select count(1) from my_table;
> >> Total MapReduce jobs = 1
> >> Number of reduce tasks determined at compile time: 1
> >> In order to change the average load for a reducer (in bytes):
> >> set hive.exec.reducers.bytes.per.reducer=<number>
> >> In order to limit the maximum number of reducers:
> >> set hive.exec.reducers.max=<number>
> >> In order to set a constant number of reducers:
> >> set mapred.reduce.tasks=<number>
> >> FAILED: Execution Error, return code 2 from
> >> org.apache.hadoop.hive.ql.exec.ExecDriver
> >> hive> exit ;
> >>
> >> If I set the property value to
> >> org.apache.hadoop.hive.ql.io.HiveInputFormat,
> >> the job runs fine.
> >>
> >> Suggestions ? Is there something that I am missing ?
> >>
> >> Thanks
> >> -Matt
> >>
> >
> >
>
>
>
RE: Hive-74
Posted by Namit Jain <nj...@facebook.com>.
Yes, we also ran into this problem, and Zheng has a patch for this.
Either you can apply the patch on hadoop and get it to work, or copy the code in
Hive and have a hive patch. I am not sure if that is the best approach, since that will
lead to code duplication.
Thanks,
-namit
From: Matt Pestritto [mailto:matt@pestritto.com]
Sent: Monday, October 19, 2009 10:20 AM
To: hive-user@hadoop.apache.org
Subject: Re: Hive-74
Namit -
I finally had a chance to look at this again. I am running hadoop 20.1 and hive trunk.
I'm still having a problem with combine input. I found an error in my jobtracker logs:
12:48:08,792 INFO [JobInProgress] Input size for job job_200910160957_0003 = 13. Number of splits = 1
12:48:08,794 ERROR [JobTracker] Job initialization failed:
java.lang.IllegalArgumentException: Network location name contains /: /default-rack
at org.apache.hadoop.net.NodeBase.set(NodeBase.java:75)
at org.apache.hadoop.net.NodeBase.<init>(NodeBase.java:57)
at org.apache.hadoop.mapred.JobTracker.addHostToNodeMapping(JobTracker.java:2390)
at org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2384)
at org.apache.hadoop.mapred.JobInProgress.createCache(JobInProgress.java:349)
at org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:450)
at org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:3147)
at org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:79)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
12:48:08,794 INFO [JobTracker] Failing job job_200910160957_0003
12:48:09,560 INFO [JobTracker] Killing job job_200910160957_0003
This seems to be related to hadoop-5759 but it looks like it was patched coincidently today by Zheng. https://issues.apache.org/jira/browse/HADOOP-5759
I'm guessing I need to build hadoop from 20 branch for this to work ?
Thanks
-Matt
On Thu, Oct 8, 2009 at 1:26 AM, Namit Jain <nj...@facebook.com>> wrote:
Hi Matt,
Sorry for the late reply.
hive> set hive.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat;
I tried it running on hadoop 20 and it ran fine for me.
Which hive release are you using ?
Also, you got a runtime error - can you see the stderr logs on the tracker ?
Thanks,
-namit
On 10/1/09 5:01 PM, "Matt Pestritto" <ma...@pestritto.com>> wrote:
Namit -
Any idea on how to resolve ?
Thanks
On Thu, Oct 1, 2009 at 10:52 AM, Matt Pestritto <ma...@pestritto.com>> wrote:
> There were errors in the hive.log
>
> 2009-10-01 10:40:53,631 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.core.resources" but it cannot be resolved.
> 2009-10-01 10:40:53,631 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.core.resources" but it cannot be resolved.
> 2009-10-01 10:40:53,633 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.core.runtime" but it cannot be resolved.
> 2009-10-01 10:40:53,633 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.core.runtime" but it cannot be resolved.
> 2009-10-01 10:40:53,634 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.text" but it cannot be resolved.
> 2009-10-01 10:40:53,634 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.text" but it cannot be resolved.
> 2009-10-01 10:40:57,143 WARN mapred.JobClient
> (JobClient.java:configureCommandLineOptions(539)) - Use GenericOptionsParser
> for parsing the arguments. Applications should implement Tool for the same.
> 2009-10-01 10:40:58,609 ERROR exec.ExecDriver
> (SessionState.java:printError(248)) - Ended Job = job_200909301537_0068 with
> errors
> 2009-10-01 10:40:58,622 ERROR ql.Driver (SessionState.java:printError(248))
> - FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.ExecDriver
>
>
>
> On Wed, Sep 30, 2009 at 5:26 PM, Namit Jain <nj...@facebook.com>> wrote:
>
>> What you are doing seems OK ?
>> Can you get the stack trace from /tmp/<username>/hive.log ?
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Matt Pestritto [mailto:matt@pestritto.com]
>> Sent: Wednesday, September 30, 2009 6:51 AM
>> To: hive-dev@hadoop.apache.org<ma...@hadoop.apache.org>; hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
>> Subject: Fwd: Hive-74
>>
>> Including hive-user in case someone has any experience with this..
>> Thanks
>> -Matt
>>
>> ---------- Forwarded message ----------
>> From: Matt Pestritto <ma...@pestritto.com>>
>> Date: Tue, Sep 29, 2009 at 5:26 PM
>> Subject: Hive-74
>> To: hive-dev@hadoop.apache.org<ma...@hadoop.apache.org>
>>
>>
>> Hi-
>>
>> I'm having a problem using CombineHiveInputSplit. I believe this was
>> patched in http://issues.apache.org/jira/browse/HIVE-74
>>
>> I'm currently running hadoop 20.1 using hive trunk.
>>
>> hive-default.xml has the following property:
>> <property>
>> <name>hive.input.format</name>
>> <value></value>
>> <description>The default input format, if it is not specified, the system
>> assigns it. It is set to HiveInputFormat for hadoop versions 17, 18 and
>> 19,
>> whereas it is set to CombinedHiveInputFormat for hadoop 20. The user can
>> always overwrite it - if there is a bug in CombinedHiveInputFormat, it can
>> always be manually set to HiveInputFormat. </description>
>> </property>
>>
>> I added the following to hive-site.xml: ( Notice, the description in
>> hive-default.xml has CombinedHiveInputFormat which does not work for me -
>> the property value seems to be Combine(-d) )
>> <property>
>> <name>hive.input.format</name>
>> <value>org.apache.hadoop.hive.ql.io.CombineHiveInputFormat</value>
>> <description>The default input format, if it is not specified, the system
>> assigns it. It is set to HiveInputFormat for hadoop versions 17, 18 and
>> 19,
>> whereas it is set to CombinedHiveInputFormat for hadoop 20. The user can
>> always overwrite it - if there is a bug in CombinedHiveInputFormat, it can
>> always be manually set to HiveInputFormat. </description>
>> </property>
>>
>> When I launch a job the cli exits immediately:
>> hive> select count(1) from my_table;
>> Total MapReduce jobs = 1
>> Number of reduce tasks determined at compile time: 1
>> In order to change the average load for a reducer (in bytes):
>> set hive.exec.reducers.bytes.per.reducer=<number>
>> In order to limit the maximum number of reducers:
>> set hive.exec.reducers.max=<number>
>> In order to set a constant number of reducers:
>> set mapred.reduce.tasks=<number>
>> FAILED: Execution Error, return code 2 from
>> org.apache.hadoop.hive.ql.exec.ExecDriver
>> hive> exit ;
>>
>> If I set the property value to
>> org.apache.hadoop.hive.ql.io.HiveInputFormat,
>> the job runs fine.
>>
>> Suggestions ? Is there something that I am missing ?
>>
>> Thanks
>> -Matt
>>
>
>
Re: Hive-74
Posted by Matt Pestritto <ma...@pestritto.com>.
Namit -
I finally had a chance to look at this again. I am running hadoop 20.1 and
hive trunk.
I'm still having a problem with combine input. I found an error in my
jobtracker logs:
12:48:08,792 INFO [JobInProgress] Input size for job
job_200910160957_0003 = 13. Number of splits = 1
12:48:08,794 ERROR [JobTracker] Job initialization failed:
java.lang.IllegalArgumentException: Network location name contains /:
/default-rack
at org.apache.hadoop.net.NodeBase.set(NodeBase.java:75)
at org.apache.hadoop.net.NodeBase.<init>(NodeBase.java:57)
at org.apache.hadoop.mapred.JobTracker.addHostToNodeMapping(JobTracker.java:2390)
at org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2384)
at org.apache.hadoop.mapred.JobInProgress.createCache(JobInProgress.java:349)
at org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:450)
at org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:3147)
at org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:79)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
12:48:08,794 INFO [JobTracker] Failing job job_200910160957_0003
12:48:09,560 INFO [JobTracker] Killing job job_200910160957_0003
This seems to be related to hadoop-5759 but it looks like it was patched
coincidently today by Zheng.
https://issues.apache.org/jira/browse/HADOOP-5759
I'm guessing I need to build hadoop from 20 branch for this to work ?
Thanks
-Matt
On Thu, Oct 8, 2009 at 1:26 AM, Namit Jain <nj...@facebook.com> wrote:
> Hi Matt,
>
> Sorry for the late reply.
>
> hive> set
> hive.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat;
>
> I tried it running on hadoop 20 and it ran fine for me.
>
> Which hive release are you using ?
>
> Also, you got a runtime error – can you see the stderr logs on the tracker
> ?
>
> Thanks,
> -namit
>
>
>
> On 10/1/09 5:01 PM, "Matt Pestritto" <ma...@pestritto.com> wrote:
>
> Namit -
> Any idea on how to resolve ?
> Thanks
>
> On Thu, Oct 1, 2009 at 10:52 AM, Matt Pestritto <ma...@pestritto.com>
> wrote:
>
> > There were errors in the hive.log
> >
> > 2009-10-01 10:40:53,631 ERROR DataNucleus.Plugin
> > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> > "org.eclipse.core.resources" but it cannot be resolved.
> > 2009-10-01 10:40:53,631 ERROR DataNucleus.Plugin
> > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> > "org.eclipse.core.resources" but it cannot be resolved.
> > 2009-10-01 10:40:53,633 ERROR DataNucleus.Plugin
> > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> > "org.eclipse.core.runtime" but it cannot be resolved.
> > 2009-10-01 10:40:53,633 ERROR DataNucleus.Plugin
> > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> > "org.eclipse.core.runtime" but it cannot be resolved.
> > 2009-10-01 10:40:53,634 ERROR DataNucleus.Plugin
> > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> > "org.eclipse.text" but it cannot be resolved.
> > 2009-10-01 10:40:53,634 ERROR DataNucleus.Plugin
> > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> > "org.eclipse.text" but it cannot be resolved.
> > 2009-10-01 10:40:57,143 WARN mapred.JobClient
> > (JobClient.java:configureCommandLineOptions(539)) - Use
> GenericOptionsParser
> > for parsing the arguments. Applications should implement Tool for the
> same.
> > 2009-10-01 10:40:58,609 ERROR exec.ExecDriver
> > (SessionState.java:printError(248)) - Ended Job = job_200909301537_0068
> with
> > errors
> > 2009-10-01 10:40:58,622 ERROR ql.Driver
> (SessionState.java:printError(248))
> > - FAILED: Execution Error, return code 2 from
> > org.apache.hadoop.hive.ql.exec.ExecDriver
> >
> >
> >
> > On Wed, Sep 30, 2009 at 5:26 PM, Namit Jain <nj...@facebook.com> wrote:
> >
> >> What you are doing seems OK ?
> >> Can you get the stack trace from /tmp/<username>/hive.log ?
> >>
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Matt Pestritto [mailto:matt@pestritto.com] <matt@pestritto.com]>
> >> Sent: Wednesday, September 30, 2009 6:51 AM
> >> To: hive-dev@hadoop.apache.org; hive-user@hadoop.apache.org
> >> Subject: Fwd: Hive-74
> >>
> >> Including hive-user in case someone has any experience with this..
> >> Thanks
> >> -Matt
> >>
> >> ---------- Forwarded message ----------
> >> From: Matt Pestritto <ma...@pestritto.com>
> >> Date: Tue, Sep 29, 2009 at 5:26 PM
> >> Subject: Hive-74
> >> To: hive-dev@hadoop.apache.org
> >>
> >>
> >> Hi-
> >>
> >> I'm having a problem using CombineHiveInputSplit. I believe this was
> >> patched in http://issues.apache.org/jira/browse/HIVE-74
> >>
> >> I'm currently running hadoop 20.1 using hive trunk.
> >>
> >> hive-default.xml has the following property:
> >> <property>
> >> <name>hive.input.format</name>
> >> <value></value>
> >> <description>The default input format, if it is not specified, the
> system
> >> assigns it. It is set to HiveInputFormat for hadoop versions 17, 18 and
> >> 19,
> >> whereas it is set to CombinedHiveInputFormat for hadoop 20. The user can
> >> always overwrite it - if there is a bug in CombinedHiveInputFormat, it
> can
> >> always be manually set to HiveInputFormat. </description>
> >> </property>
> >>
> >> I added the following to hive-site.xml: ( Notice, the description in
> >> hive-default.xml has CombinedHiveInputFormat which does not work for me
> -
> >> the property value seems to be Combine(-d) )
> >> <property>
> >> <name>hive.input.format</name>
> >> <value>org.apache.hadoop.hive.ql.io.CombineHiveInputFormat</value>
> >> <description>The default input format, if it is not specified, the
> system
> >> assigns it. It is set to HiveInputFormat for hadoop versions 17, 18 and
> >> 19,
> >> whereas it is set to CombinedHiveInputFormat for hadoop 20. The user can
> >> always overwrite it - if there is a bug in CombinedHiveInputFormat, it
> can
> >> always be manually set to HiveInputFormat. </description>
> >> </property>
> >>
> >> When I launch a job the cli exits immediately:
> >> hive> select count(1) from my_table;
> >> Total MapReduce jobs = 1
> >> Number of reduce tasks determined at compile time: 1
> >> In order to change the average load for a reducer (in bytes):
> >> set hive.exec.reducers.bytes.per.reducer=<number>
> >> In order to limit the maximum number of reducers:
> >> set hive.exec.reducers.max=<number>
> >> In order to set a constant number of reducers:
> >> set mapred.reduce.tasks=<number>
> >> FAILED: Execution Error, return code 2 from
> >> org.apache.hadoop.hive.ql.exec.ExecDriver
> >> hive> exit ;
> >>
> >> If I set the property value to
> >> org.apache.hadoop.hive.ql.io.HiveInputFormat,
> >> the job runs fine.
> >>
> >> Suggestions ? Is there something that I am missing ?
> >>
> >> Thanks
> >> -Matt
> >>
> >
> >
>
>
Re: Hive-74
Posted by Namit Jain <nj...@facebook.com>.
Hi Matt,
Sorry for the late reply.
hive> set hive.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat;
I tried it running on hadoop 20 and it ran fine for me.
Which hive release are you using ?
Also, you got a runtime error - can you see the stderr logs on the tracker ?
Thanks,
-namit
On 10/1/09 5:01 PM, "Matt Pestritto" <ma...@pestritto.com> wrote:
Namit -
Any idea on how to resolve ?
Thanks
On Thu, Oct 1, 2009 at 10:52 AM, Matt Pestritto <ma...@pestritto.com> wrote:
> There were errors in the hive.log
>
> 2009-10-01 10:40:53,631 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.core.resources" but it cannot be resolved.
> 2009-10-01 10:40:53,631 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.core.resources" but it cannot be resolved.
> 2009-10-01 10:40:53,633 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.core.runtime" but it cannot be resolved.
> 2009-10-01 10:40:53,633 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.core.runtime" but it cannot be resolved.
> 2009-10-01 10:40:53,634 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.text" but it cannot be resolved.
> 2009-10-01 10:40:53,634 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.text" but it cannot be resolved.
> 2009-10-01 10:40:57,143 WARN mapred.JobClient
> (JobClient.java:configureCommandLineOptions(539)) - Use GenericOptionsParser
> for parsing the arguments. Applications should implement Tool for the same.
> 2009-10-01 10:40:58,609 ERROR exec.ExecDriver
> (SessionState.java:printError(248)) - Ended Job = job_200909301537_0068 with
> errors
> 2009-10-01 10:40:58,622 ERROR ql.Driver (SessionState.java:printError(248))
> - FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.ExecDriver
>
>
>
> On Wed, Sep 30, 2009 at 5:26 PM, Namit Jain <nj...@facebook.com> wrote:
>
>> What you are doing seems OK ?
>> Can you get the stack trace from /tmp/<username>/hive.log ?
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Matt Pestritto [mailto:matt@pestritto.com]
>> Sent: Wednesday, September 30, 2009 6:51 AM
>> To: hive-dev@hadoop.apache.org; hive-user@hadoop.apache.org
>> Subject: Fwd: Hive-74
>>
>> Including hive-user in case someone has any experience with this..
>> Thanks
>> -Matt
>>
>> ---------- Forwarded message ----------
>> From: Matt Pestritto <ma...@pestritto.com>
>> Date: Tue, Sep 29, 2009 at 5:26 PM
>> Subject: Hive-74
>> To: hive-dev@hadoop.apache.org
>>
>>
>> Hi-
>>
>> I'm having a problem using CombineHiveInputSplit. I believe this was
>> patched in http://issues.apache.org/jira/browse/HIVE-74
>>
>> I'm currently running hadoop 20.1 using hive trunk.
>>
>> hive-default.xml has the following property:
>> <property>
>> <name>hive.input.format</name>
>> <value></value>
>> <description>The default input format, if it is not specified, the system
>> assigns it. It is set to HiveInputFormat for hadoop versions 17, 18 and
>> 19,
>> whereas it is set to CombinedHiveInputFormat for hadoop 20. The user can
>> always overwrite it - if there is a bug in CombinedHiveInputFormat, it can
>> always be manually set to HiveInputFormat. </description>
>> </property>
>>
>> I added the following to hive-site.xml: ( Notice, the description in
>> hive-default.xml has CombinedHiveInputFormat which does not work for me -
>> the property value seems to be Combine(-d) )
>> <property>
>> <name>hive.input.format</name>
>> <value>org.apache.hadoop.hive.ql.io.CombineHiveInputFormat</value>
>> <description>The default input format, if it is not specified, the system
>> assigns it. It is set to HiveInputFormat for hadoop versions 17, 18 and
>> 19,
>> whereas it is set to CombinedHiveInputFormat for hadoop 20. The user can
>> always overwrite it - if there is a bug in CombinedHiveInputFormat, it can
>> always be manually set to HiveInputFormat. </description>
>> </property>
>>
>> When I launch a job the cli exits immediately:
>> hive> select count(1) from my_table;
>> Total MapReduce jobs = 1
>> Number of reduce tasks determined at compile time: 1
>> In order to change the average load for a reducer (in bytes):
>> set hive.exec.reducers.bytes.per.reducer=<number>
>> In order to limit the maximum number of reducers:
>> set hive.exec.reducers.max=<number>
>> In order to set a constant number of reducers:
>> set mapred.reduce.tasks=<number>
>> FAILED: Execution Error, return code 2 from
>> org.apache.hadoop.hive.ql.exec.ExecDriver
>> hive> exit ;
>>
>> If I set the property value to
>> org.apache.hadoop.hive.ql.io.HiveInputFormat,
>> the job runs fine.
>>
>> Suggestions ? Is there something that I am missing ?
>>
>> Thanks
>> -Matt
>>
>
>
Re: Hive-74
Posted by Namit Jain <nj...@facebook.com>.
Hi Matt,
Sorry for the late reply.
hive> set hive.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat;
I tried it running on hadoop 20 and it ran fine for me.
Which hive release are you using ?
Also, you got a runtime error - can you see the stderr logs on the tracker ?
Thanks,
-namit
On 10/1/09 5:01 PM, "Matt Pestritto" <ma...@pestritto.com> wrote:
Namit -
Any idea on how to resolve ?
Thanks
On Thu, Oct 1, 2009 at 10:52 AM, Matt Pestritto <ma...@pestritto.com> wrote:
> There were errors in the hive.log
>
> 2009-10-01 10:40:53,631 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.core.resources" but it cannot be resolved.
> 2009-10-01 10:40:53,631 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.core.resources" but it cannot be resolved.
> 2009-10-01 10:40:53,633 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.core.runtime" but it cannot be resolved.
> 2009-10-01 10:40:53,633 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.core.runtime" but it cannot be resolved.
> 2009-10-01 10:40:53,634 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.text" but it cannot be resolved.
> 2009-10-01 10:40:53,634 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.text" but it cannot be resolved.
> 2009-10-01 10:40:57,143 WARN mapred.JobClient
> (JobClient.java:configureCommandLineOptions(539)) - Use GenericOptionsParser
> for parsing the arguments. Applications should implement Tool for the same.
> 2009-10-01 10:40:58,609 ERROR exec.ExecDriver
> (SessionState.java:printError(248)) - Ended Job = job_200909301537_0068 with
> errors
> 2009-10-01 10:40:58,622 ERROR ql.Driver (SessionState.java:printError(248))
> - FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.ExecDriver
>
>
>
> On Wed, Sep 30, 2009 at 5:26 PM, Namit Jain <nj...@facebook.com> wrote:
>
>> What you are doing seems OK ?
>> Can you get the stack trace from /tmp/<username>/hive.log ?
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Matt Pestritto [mailto:matt@pestritto.com]
>> Sent: Wednesday, September 30, 2009 6:51 AM
>> To: hive-dev@hadoop.apache.org; hive-user@hadoop.apache.org
>> Subject: Fwd: Hive-74
>>
>> Including hive-user in case someone has any experience with this..
>> Thanks
>> -Matt
>>
>> ---------- Forwarded message ----------
>> From: Matt Pestritto <ma...@pestritto.com>
>> Date: Tue, Sep 29, 2009 at 5:26 PM
>> Subject: Hive-74
>> To: hive-dev@hadoop.apache.org
>>
>>
>> Hi-
>>
>> I'm having a problem using CombineHiveInputSplit. I believe this was
>> patched in http://issues.apache.org/jira/browse/HIVE-74
>>
>> I'm currently running hadoop 20.1 using hive trunk.
>>
>> hive-default.xml has the following property:
>> <property>
>> <name>hive.input.format</name>
>> <value></value>
>> <description>The default input format, if it is not specified, the system
>> assigns it. It is set to HiveInputFormat for hadoop versions 17, 18 and
>> 19,
>> whereas it is set to CombinedHiveInputFormat for hadoop 20. The user can
>> always overwrite it - if there is a bug in CombinedHiveInputFormat, it can
>> always be manually set to HiveInputFormat. </description>
>> </property>
>>
>> I added the following to hive-site.xml: ( Notice, the description in
>> hive-default.xml has CombinedHiveInputFormat which does not work for me -
>> the property value seems to be Combine(-d) )
>> <property>
>> <name>hive.input.format</name>
>> <value>org.apache.hadoop.hive.ql.io.CombineHiveInputFormat</value>
>> <description>The default input format, if it is not specified, the system
>> assigns it. It is set to HiveInputFormat for hadoop versions 17, 18 and
>> 19,
>> whereas it is set to CombinedHiveInputFormat for hadoop 20. The user can
>> always overwrite it - if there is a bug in CombinedHiveInputFormat, it can
>> always be manually set to HiveInputFormat. </description>
>> </property>
>>
>> When I launch a job the cli exits immediately:
>> hive> select count(1) from my_table;
>> Total MapReduce jobs = 1
>> Number of reduce tasks determined at compile time: 1
>> In order to change the average load for a reducer (in bytes):
>> set hive.exec.reducers.bytes.per.reducer=<number>
>> In order to limit the maximum number of reducers:
>> set hive.exec.reducers.max=<number>
>> In order to set a constant number of reducers:
>> set mapred.reduce.tasks=<number>
>> FAILED: Execution Error, return code 2 from
>> org.apache.hadoop.hive.ql.exec.ExecDriver
>> hive> exit ;
>>
>> If I set the property value to
>> org.apache.hadoop.hive.ql.io.HiveInputFormat,
>> the job runs fine.
>>
>> Suggestions ? Is there something that I am missing ?
>>
>> Thanks
>> -Matt
>>
>
>
Re: Hive-74
Posted by Matt Pestritto <ma...@pestritto.com>.
Namit -
Any idea on how to resolve ?
Thanks
On Thu, Oct 1, 2009 at 10:52 AM, Matt Pestritto <ma...@pestritto.com> wrote:
> There were errors in the hive.log
>
> 2009-10-01 10:40:53,631 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.core.resources" but it cannot be resolved.
> 2009-10-01 10:40:53,631 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.core.resources" but it cannot be resolved.
> 2009-10-01 10:40:53,633 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.core.runtime" but it cannot be resolved.
> 2009-10-01 10:40:53,633 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.core.runtime" but it cannot be resolved.
> 2009-10-01 10:40:53,634 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.text" but it cannot be resolved.
> 2009-10-01 10:40:53,634 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.text" but it cannot be resolved.
> 2009-10-01 10:40:57,143 WARN mapred.JobClient
> (JobClient.java:configureCommandLineOptions(539)) - Use GenericOptionsParser
> for parsing the arguments. Applications should implement Tool for the same.
> 2009-10-01 10:40:58,609 ERROR exec.ExecDriver
> (SessionState.java:printError(248)) - Ended Job = job_200909301537_0068 with
> errors
> 2009-10-01 10:40:58,622 ERROR ql.Driver (SessionState.java:printError(248))
> - FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.ExecDriver
>
>
>
> On Wed, Sep 30, 2009 at 5:26 PM, Namit Jain <nj...@facebook.com> wrote:
>
>> What you are doing seems OK ?
>> Can you get the stack trace from /tmp/<username>/hive.log ?
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Matt Pestritto [mailto:matt@pestritto.com]
>> Sent: Wednesday, September 30, 2009 6:51 AM
>> To: hive-dev@hadoop.apache.org; hive-user@hadoop.apache.org
>> Subject: Fwd: Hive-74
>>
>> Including hive-user in case someone has any experience with this..
>> Thanks
>> -Matt
>>
>> ---------- Forwarded message ----------
>> From: Matt Pestritto <ma...@pestritto.com>
>> Date: Tue, Sep 29, 2009 at 5:26 PM
>> Subject: Hive-74
>> To: hive-dev@hadoop.apache.org
>>
>>
>> Hi-
>>
>> I'm having a problem using CombineHiveInputSplit. I believe this was
>> patched in http://issues.apache.org/jira/browse/HIVE-74
>>
>> I'm currently running hadoop 20.1 using hive trunk.
>>
>> hive-default.xml has the following property:
>> <property>
>> <name>hive.input.format</name>
>> <value></value>
>> <description>The default input format, if it is not specified, the system
>> assigns it. It is set to HiveInputFormat for hadoop versions 17, 18 and
>> 19,
>> whereas it is set to CombinedHiveInputFormat for hadoop 20. The user can
>> always overwrite it - if there is a bug in CombinedHiveInputFormat, it can
>> always be manually set to HiveInputFormat. </description>
>> </property>
>>
>> I added the following to hive-site.xml: ( Notice, the description in
>> hive-default.xml has CombinedHiveInputFormat which does not work for me -
>> the property value seems to be Combine(-d) )
>> <property>
>> <name>hive.input.format</name>
>> <value>org.apache.hadoop.hive.ql.io.CombineHiveInputFormat</value>
>> <description>The default input format, if it is not specified, the system
>> assigns it. It is set to HiveInputFormat for hadoop versions 17, 18 and
>> 19,
>> whereas it is set to CombinedHiveInputFormat for hadoop 20. The user can
>> always overwrite it - if there is a bug in CombinedHiveInputFormat, it can
>> always be manually set to HiveInputFormat. </description>
>> </property>
>>
>> When I launch a job the cli exits immediately:
>> hive> select count(1) from my_table;
>> Total MapReduce jobs = 1
>> Number of reduce tasks determined at compile time: 1
>> In order to change the average load for a reducer (in bytes):
>> set hive.exec.reducers.bytes.per.reducer=<number>
>> In order to limit the maximum number of reducers:
>> set hive.exec.reducers.max=<number>
>> In order to set a constant number of reducers:
>> set mapred.reduce.tasks=<number>
>> FAILED: Execution Error, return code 2 from
>> org.apache.hadoop.hive.ql.exec.ExecDriver
>> hive> exit ;
>>
>> If I set the property value to
>> org.apache.hadoop.hive.ql.io.HiveInputFormat,
>> the job runs fine.
>>
>> Suggestions ? Is there something that I am missing ?
>>
>> Thanks
>> -Matt
>>
>
>
Re: Hive-74
Posted by Matt Pestritto <ma...@pestritto.com>.
Namit -
Any idea on how to resolve ?
Thanks
On Thu, Oct 1, 2009 at 10:52 AM, Matt Pestritto <ma...@pestritto.com> wrote:
> There were errors in the hive.log
>
> 2009-10-01 10:40:53,631 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.core.resources" but it cannot be resolved.
> 2009-10-01 10:40:53,631 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.core.resources" but it cannot be resolved.
> 2009-10-01 10:40:53,633 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.core.runtime" but it cannot be resolved.
> 2009-10-01 10:40:53,633 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.core.runtime" but it cannot be resolved.
> 2009-10-01 10:40:53,634 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.text" but it cannot be resolved.
> 2009-10-01 10:40:53,634 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.text" but it cannot be resolved.
> 2009-10-01 10:40:57,143 WARN mapred.JobClient
> (JobClient.java:configureCommandLineOptions(539)) - Use GenericOptionsParser
> for parsing the arguments. Applications should implement Tool for the same.
> 2009-10-01 10:40:58,609 ERROR exec.ExecDriver
> (SessionState.java:printError(248)) - Ended Job = job_200909301537_0068 with
> errors
> 2009-10-01 10:40:58,622 ERROR ql.Driver (SessionState.java:printError(248))
> - FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.ExecDriver
>
>
>
> On Wed, Sep 30, 2009 at 5:26 PM, Namit Jain <nj...@facebook.com> wrote:
>
>> What you are doing seems OK ?
>> Can you get the stack trace from /tmp/<username>/hive.log ?
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Matt Pestritto [mailto:matt@pestritto.com]
>> Sent: Wednesday, September 30, 2009 6:51 AM
>> To: hive-dev@hadoop.apache.org; hive-user@hadoop.apache.org
>> Subject: Fwd: Hive-74
>>
>> Including hive-user in case someone has any experience with this..
>> Thanks
>> -Matt
>>
>> ---------- Forwarded message ----------
>> From: Matt Pestritto <ma...@pestritto.com>
>> Date: Tue, Sep 29, 2009 at 5:26 PM
>> Subject: Hive-74
>> To: hive-dev@hadoop.apache.org
>>
>>
>> Hi-
>>
>> I'm having a problem using CombineHiveInputSplit. I believe this was
>> patched in http://issues.apache.org/jira/browse/HIVE-74
>>
>> I'm currently running hadoop 20.1 using hive trunk.
>>
>> hive-default.xml has the following property:
>> <property>
>> <name>hive.input.format</name>
>> <value></value>
>> <description>The default input format, if it is not specified, the system
>> assigns it. It is set to HiveInputFormat for hadoop versions 17, 18 and
>> 19,
>> whereas it is set to CombinedHiveInputFormat for hadoop 20. The user can
>> always overwrite it - if there is a bug in CombinedHiveInputFormat, it can
>> always be manually set to HiveInputFormat. </description>
>> </property>
>>
>> I added the following to hive-site.xml: ( Notice, the description in
>> hive-default.xml has CombinedHiveInputFormat which does not work for me -
>> the property value seems to be Combine(-d) )
>> <property>
>> <name>hive.input.format</name>
>> <value>org.apache.hadoop.hive.ql.io.CombineHiveInputFormat</value>
>> <description>The default input format, if it is not specified, the system
>> assigns it. It is set to HiveInputFormat for hadoop versions 17, 18 and
>> 19,
>> whereas it is set to CombinedHiveInputFormat for hadoop 20. The user can
>> always overwrite it - if there is a bug in CombinedHiveInputFormat, it can
>> always be manually set to HiveInputFormat. </description>
>> </property>
>>
>> When I launch a job the cli exits immediately:
>> hive> select count(1) from my_table;
>> Total MapReduce jobs = 1
>> Number of reduce tasks determined at compile time: 1
>> In order to change the average load for a reducer (in bytes):
>> set hive.exec.reducers.bytes.per.reducer=<number>
>> In order to limit the maximum number of reducers:
>> set hive.exec.reducers.max=<number>
>> In order to set a constant number of reducers:
>> set mapred.reduce.tasks=<number>
>> FAILED: Execution Error, return code 2 from
>> org.apache.hadoop.hive.ql.exec.ExecDriver
>> hive> exit ;
>>
>> If I set the property value to
>> org.apache.hadoop.hive.ql.io.HiveInputFormat,
>> the job runs fine.
>>
>> Suggestions ? Is there something that I am missing ?
>>
>> Thanks
>> -Matt
>>
>
>