You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Jonathan Hodges <ho...@gmail.com> on 2013/12/02 15:21:25 UTC
Re: Using Hive with WebHCat
Hi,
I have setup WebHCat that is bundled with Hive 0.11.0. I am able to kick
of map reduce jobs with the REST API successfully. However I am having
some issues with Hive commands over REST. The following is my
$TEMPLETON_HOME/webhcat-site.xml.
<?xml version="1.0" encoding="UTF-8"?>
<!-- The default settings for Templeton. -->
<!-- Edit templeton-site.xml to change settings for your local -->
<!-- install. -->
<configuration>
<property>
<name>templeton.pig.archive</name>
<value>s3n://pearson-alto-hadoop/apps/webhcat/archives/pig-0.11.1.tar.gz</value>
<description>The path to the Pig archive.</description>
</property>
<property>
<name>templeton.pig.path</name>
<value>pig-0.11.1.tar.gz/pig-0.11.1/bin/pig</value>
<description>The path to the Pig executable.</description>
</property>
<property>
<name>templeton.hive.archive</name>
<value>s3n://pearson-alto-hadoop/apps/webhcat/archives/hive-0.11.0.tar.gz</value>
<description>The path to the Hive archive.</description>
</property>
<property>
<name>templeton.hive.path</name>
<value>hive-0.11.0.tar.gz/hive-0.11.0-bin/bin/hive</value>
<description>The path to the Hive executable.</description>
</property>
</configuration>
curl -s -d user.name=hadoop \
> -d execute="show+tables;" \
> -d statusdir="s3n://pearson-alto-hadoop/webhcat/hive" \
> 'http://10.201.5.28:50111/templeton/v1/hive'
{"id":"job_201311281741_0020"}
When I check the statusdir and jobs folders I see the job had an exit
status of 1 so it wasn't successful.
hadoop fs -ls s3n://pearson-alto-hadoop/webhcat/hive
Found 1 items
-rwxrwxrwx 1 2 2013-11-29 15:15 /webhcat/hive/exit
hadoop fs -ls /templeton-hadoop/jobs/job_201311281741_0020
Found 3 items
-rw-r--r-- 1 hadoop supergroup 4 2013-11-29 15:15
/templeton-hadoop/jobs/job_201311281741_0020/completed
-rw-r--r-- 1 hadoop supergroup 1 2013-11-29 15:15
/templeton-hadoop/jobs/job_201311281741_0020/exitValue
-rw-r--r-- 1 hadoop supergroup 6 2013-11-29 15:15
/templeton-hadoop/jobs/job_201311281741_0020/user
Here is what I see in the logs.
DEBUG | 29 Nov 2013 15:15:36,133 | org.apache.hcatalog.templeton.Server |
queued job job_201311281741_0020 in 13403 ms
DEBUG | 29 Nov 2013 15:16:09,583 |
org.apache.hcatalog.templeton.tool.HDFSStorage | Couldn't find
/templeton-hadoop/jobs/job_201311281741_0020/notified: File does not exist:
/templeton-hadoop/jobs/job_201311281741_0020/notified
DEBUG | 29 Nov 2013 15:16:09,584 |
org.apache.hcatalog.templeton.tool.HDFSStorage | Couldn't find
/templeton-hadoop/jobs/job_201311281741_0020/callback: File does not exist:
/templeton-hadoop/jobs/job_201311281741_0020/callback
How do I figure out the reason for failure?
Thanks,
Jonathan
Re: Using Hive with WebHCat
Posted by Eugene Koifman <ek...@hortonworks.com>.
Sorry, missed this mail earlier. The fact that these files are missing is
OK. When you ask for status info via REST it tries to return a list of
fields (each field like 'exitValue' is a file when using HDFStorage, which
is the default) but it doesn't know which ones have never been written.
For example, if you didn't register a callback URL when submitting the
job, the 'notified' will never be created. That's why these are DEBUG
level messages.
On Sat, Dec 21, 2013 at 1:07 PM, Jonathan Hodges <ho...@gmail.com> wrote:
> Sorry forgot to mention the job tracker UI shows a TempletonControllerJob
> completing successfully.
>
>
> On Sat, Dec 21, 2013 at 9:37 AM, Jonathan Hodges <ho...@gmail.com>wrote:
>
>> Hi Eugene,
>>
>> The few lines I included above are from webhcat.log
>>
>> DEBUG | 29 Nov 2013 15:15:36,133 | org.apache.hcatalog.templeton.Server |
>> queued job job_201311281741_0020 in 13403 ms
>> DEBUG | 29 Nov 2013 15:16:09,583 |
>> org.apache.hcatalog.templeton.tool.HDFSStorage | Couldn't find
>> /templeton-hadoop/jobs/job_201311281741_0020/notified: File does not exist:
>> /templeton-hadoop/jobs/job_201311281741_0020/notified
>> DEBUG | 29 Nov 2013 15:16:09,584 |
>> org.apache.hcatalog.templeton.tool.HDFSStorage | Couldn't find
>> /templeton-hadoop/jobs/job_201311281741_0020/callback: File does not exist:
>> /templeton-hadoop/jobs/job_201311281741_0020/callback
>>
>> Those files indeed don't exist. The job_201311281741_0020 directory was
>> created after starting the Hive job, but only has 'completed', 'exitValue',
>> and 'user' files. Could these missing files be the issue? If so, how do I
>> ensure they are created?
>>
>> Thanks in advance for the assistance.
>> -Jonathan
>>
>>
>> On Wed, Dec 18, 2013 at 5:15 PM, Eugene Koifman <ekoifman@hortonworks.com
>> > wrote:
>>
>>> It may be worth looking in webhcat.log and using job tracker UI
>>>
>>>
>>> On Mon, Dec 2, 2013 at 6:21 AM, Jonathan Hodges <ho...@gmail.com>wrote:
>>>
>>>> Hi,
>>>>
>>>> I have setup WebHCat that is bundled with Hive 0.11.0. I am able to
>>>> kick of map reduce jobs with the REST API successfully. However I am
>>>> having some issues with Hive commands over REST. The following is my
>>>> $TEMPLETON_HOME/webhcat-site.xml.
>>>>
>>>>
>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>> <!-- The default settings for Templeton. -->
>>>> <!-- Edit templeton-site.xml to change settings for your local -->
>>>> <!-- install. -->
>>>>
>>>> <configuration>
>>>>
>>>> <property>
>>>> <name>templeton.pig.archive</name>
>>>>
>>>> <value>s3n://pearson-alto-hadoop/apps/webhcat/archives/pig-0.11.1.tar.gz</value>
>>>> <description>The path to the Pig archive.</description>
>>>> </property>
>>>>
>>>> <property>
>>>> <name>templeton.pig.path</name>
>>>> <value>pig-0.11.1.tar.gz/pig-0.11.1/bin/pig</value>
>>>> <description>The path to the Pig executable.</description>
>>>> </property>
>>>>
>>>> <property>
>>>> <name>templeton.hive.archive</name>
>>>>
>>>> <value>s3n://pearson-alto-hadoop/apps/webhcat/archives/hive-0.11.0.tar.gz</value>
>>>> <description>The path to the Hive archive.</description>
>>>> </property>
>>>>
>>>> <property>
>>>> <name>templeton.hive.path</name>
>>>> <value>hive-0.11.0.tar.gz/hive-0.11.0-bin/bin/hive</value>
>>>> <description>The path to the Hive executable.</description>
>>>> </property>
>>>>
>>>> </configuration>
>>>>
>>>>
>>>> curl -s -d user.name=hadoop \
>>>> > -d execute="show+tables;" \
>>>> > -d statusdir="s3n://pearson-alto-hadoop/webhcat/hive" \
>>>> > 'http://10.201.5.28:50111/templeton/v1/hive'
>>>> {"id":"job_201311281741_0020"}
>>>>
>>>>
>>>> When I check the statusdir and jobs folders I see the job had an exit
>>>> status of 1 so it wasn't successful.
>>>>
>>>> hadoop fs -ls s3n://pearson-alto-hadoop/webhcat/hive
>>>> Found 1 items
>>>> -rwxrwxrwx 1 2 2013-11-29 15:15 /webhcat/hive/exit
>>>>
>>>>
>>>> hadoop fs -ls /templeton-hadoop/jobs/job_201311281741_0020
>>>> Found 3 items
>>>> -rw-r--r-- 1 hadoop supergroup 4 2013-11-29 15:15
>>>> /templeton-hadoop/jobs/job_201311281741_0020/completed
>>>> -rw-r--r-- 1 hadoop supergroup 1 2013-11-29 15:15
>>>> /templeton-hadoop/jobs/job_201311281741_0020/exitValue
>>>> -rw-r--r-- 1 hadoop supergroup 6 2013-11-29 15:15
>>>> /templeton-hadoop/jobs/job_201311281741_0020/user
>>>>
>>>> Here is what I see in the logs.
>>>>
>>>> DEBUG | 29 Nov 2013 15:15:36,133 | org.apache.hcatalog.templeton.Server
>>>> | queued job job_201311281741_0020 in 13403 ms
>>>> DEBUG | 29 Nov 2013 15:16:09,583 |
>>>> org.apache.hcatalog.templeton.tool.HDFSStorage | Couldn't find
>>>> /templeton-hadoop/jobs/job_201311281741_0020/notified: File does not exist:
>>>> /templeton-hadoop/jobs/job_201311281741_0020/notified
>>>> DEBUG | 29 Nov 2013 15:16:09,584 |
>>>> org.apache.hcatalog.templeton.tool.HDFSStorage | Couldn't find
>>>> /templeton-hadoop/jobs/job_201311281741_0020/callback: File does not exist:
>>>> /templeton-hadoop/jobs/job_201311281741_0020/callback
>>>>
>>>>
>>>> How do I figure out the reason for failure?
>>>>
>>>> Thanks,
>>>> Jonathan
>>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>>
>>
>>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Using Hive with WebHCat
Posted by Jonathan Hodges <ho...@gmail.com>.
Sorry forgot to mention the job tracker UI shows a TempletonControllerJob
completing successfully.
On Sat, Dec 21, 2013 at 9:37 AM, Jonathan Hodges <ho...@gmail.com> wrote:
> Hi Eugene,
>
> The few lines I included above are from webhcat.log
>
> DEBUG | 29 Nov 2013 15:15:36,133 | org.apache.hcatalog.templeton.Server |
> queued job job_201311281741_0020 in 13403 ms
> DEBUG | 29 Nov 2013 15:16:09,583 |
> org.apache.hcatalog.templeton.tool.HDFSStorage | Couldn't find
> /templeton-hadoop/jobs/job_201311281741_0020/notified: File does not exist:
> /templeton-hadoop/jobs/job_201311281741_0020/notified
> DEBUG | 29 Nov 2013 15:16:09,584 |
> org.apache.hcatalog.templeton.tool.HDFSStorage | Couldn't find
> /templeton-hadoop/jobs/job_201311281741_0020/callback: File does not exist:
> /templeton-hadoop/jobs/job_201311281741_0020/callback
>
> Those files indeed don't exist. The job_201311281741_0020 directory was
> created after starting the Hive job, but only has 'completed', 'exitValue',
> and 'user' files. Could these missing files be the issue? If so, how do I
> ensure they are created?
>
> Thanks in advance for the assistance.
> -Jonathan
>
>
> On Wed, Dec 18, 2013 at 5:15 PM, Eugene Koifman <ek...@hortonworks.com>wrote:
>
>> It may be worth looking in webhcat.log and using job tracker UI
>>
>>
>> On Mon, Dec 2, 2013 at 6:21 AM, Jonathan Hodges <ho...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> I have setup WebHCat that is bundled with Hive 0.11.0. I am able to
>>> kick of map reduce jobs with the REST API successfully. However I am
>>> having some issues with Hive commands over REST. The following is my
>>> $TEMPLETON_HOME/webhcat-site.xml.
>>>
>>>
>>> <?xml version="1.0" encoding="UTF-8"?>
>>> <!-- The default settings for Templeton. -->
>>> <!-- Edit templeton-site.xml to change settings for your local -->
>>> <!-- install. -->
>>>
>>> <configuration>
>>>
>>> <property>
>>> <name>templeton.pig.archive</name>
>>>
>>> <value>s3n://pearson-alto-hadoop/apps/webhcat/archives/pig-0.11.1.tar.gz</value>
>>> <description>The path to the Pig archive.</description>
>>> </property>
>>>
>>> <property>
>>> <name>templeton.pig.path</name>
>>> <value>pig-0.11.1.tar.gz/pig-0.11.1/bin/pig</value>
>>> <description>The path to the Pig executable.</description>
>>> </property>
>>>
>>> <property>
>>> <name>templeton.hive.archive</name>
>>>
>>> <value>s3n://pearson-alto-hadoop/apps/webhcat/archives/hive-0.11.0.tar.gz</value>
>>> <description>The path to the Hive archive.</description>
>>> </property>
>>>
>>> <property>
>>> <name>templeton.hive.path</name>
>>> <value>hive-0.11.0.tar.gz/hive-0.11.0-bin/bin/hive</value>
>>> <description>The path to the Hive executable.</description>
>>> </property>
>>>
>>> </configuration>
>>>
>>>
>>> curl -s -d user.name=hadoop \
>>> > -d execute="show+tables;" \
>>> > -d statusdir="s3n://pearson-alto-hadoop/webhcat/hive" \
>>> > 'http://10.201.5.28:50111/templeton/v1/hive'
>>> {"id":"job_201311281741_0020"}
>>>
>>>
>>> When I check the statusdir and jobs folders I see the job had an exit
>>> status of 1 so it wasn't successful.
>>>
>>> hadoop fs -ls s3n://pearson-alto-hadoop/webhcat/hive
>>> Found 1 items
>>> -rwxrwxrwx 1 2 2013-11-29 15:15 /webhcat/hive/exit
>>>
>>>
>>> hadoop fs -ls /templeton-hadoop/jobs/job_201311281741_0020
>>> Found 3 items
>>> -rw-r--r-- 1 hadoop supergroup 4 2013-11-29 15:15
>>> /templeton-hadoop/jobs/job_201311281741_0020/completed
>>> -rw-r--r-- 1 hadoop supergroup 1 2013-11-29 15:15
>>> /templeton-hadoop/jobs/job_201311281741_0020/exitValue
>>> -rw-r--r-- 1 hadoop supergroup 6 2013-11-29 15:15
>>> /templeton-hadoop/jobs/job_201311281741_0020/user
>>>
>>> Here is what I see in the logs.
>>>
>>> DEBUG | 29 Nov 2013 15:15:36,133 | org.apache.hcatalog.templeton.Server
>>> | queued job job_201311281741_0020 in 13403 ms
>>> DEBUG | 29 Nov 2013 15:16:09,583 |
>>> org.apache.hcatalog.templeton.tool.HDFSStorage | Couldn't find
>>> /templeton-hadoop/jobs/job_201311281741_0020/notified: File does not exist:
>>> /templeton-hadoop/jobs/job_201311281741_0020/notified
>>> DEBUG | 29 Nov 2013 15:16:09,584 |
>>> org.apache.hcatalog.templeton.tool.HDFSStorage | Couldn't find
>>> /templeton-hadoop/jobs/job_201311281741_0020/callback: File does not exist:
>>> /templeton-hadoop/jobs/job_201311281741_0020/callback
>>>
>>>
>>> How do I figure out the reason for failure?
>>>
>>> Thanks,
>>> Jonathan
>>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>
>
Re: Using Hive with WebHCat
Posted by Jonathan Hodges <ho...@gmail.com>.
Hi Eugene,
The few lines I included above are from webhcat.log
DEBUG | 29 Nov 2013 15:15:36,133 | org.apache.hcatalog.templeton.Server |
queued job job_201311281741_0020 in 13403 ms
DEBUG | 29 Nov 2013 15:16:09,583 |
org.apache.hcatalog.templeton.tool.HDFSStorage | Couldn't find
/templeton-hadoop/jobs/job_201311281741_0020/notified: File does not exist:
/templeton-hadoop/jobs/job_201311281741_0020/notified
DEBUG | 29 Nov 2013 15:16:09,584 |
org.apache.hcatalog.templeton.tool.HDFSStorage | Couldn't find
/templeton-hadoop/jobs/job_201311281741_0020/callback: File does not exist:
/templeton-hadoop/jobs/job_201311281741_0020/callback
Those files indeed don't exist. The job_201311281741_0020 directory was
created after starting the Hive job, but only has 'completed', 'exitValue',
and 'user' files. Could these missing files be the issue? If so, how do I
ensure they are created?
Thanks in advance for the assistance.
-Jonathan
On Wed, Dec 18, 2013 at 5:15 PM, Eugene Koifman <ek...@hortonworks.com>wrote:
> It may be worth looking in webhcat.log and using job tracker UI
>
>
> On Mon, Dec 2, 2013 at 6:21 AM, Jonathan Hodges <ho...@gmail.com> wrote:
>
>> Hi,
>>
>> I have setup WebHCat that is bundled with Hive 0.11.0. I am able to kick
>> of map reduce jobs with the REST API successfully. However I am having
>> some issues with Hive commands over REST. The following is my
>> $TEMPLETON_HOME/webhcat-site.xml.
>>
>>
>> <?xml version="1.0" encoding="UTF-8"?>
>> <!-- The default settings for Templeton. -->
>> <!-- Edit templeton-site.xml to change settings for your local -->
>> <!-- install. -->
>>
>> <configuration>
>>
>> <property>
>> <name>templeton.pig.archive</name>
>>
>> <value>s3n://pearson-alto-hadoop/apps/webhcat/archives/pig-0.11.1.tar.gz</value>
>> <description>The path to the Pig archive.</description>
>> </property>
>>
>> <property>
>> <name>templeton.pig.path</name>
>> <value>pig-0.11.1.tar.gz/pig-0.11.1/bin/pig</value>
>> <description>The path to the Pig executable.</description>
>> </property>
>>
>> <property>
>> <name>templeton.hive.archive</name>
>>
>> <value>s3n://pearson-alto-hadoop/apps/webhcat/archives/hive-0.11.0.tar.gz</value>
>> <description>The path to the Hive archive.</description>
>> </property>
>>
>> <property>
>> <name>templeton.hive.path</name>
>> <value>hive-0.11.0.tar.gz/hive-0.11.0-bin/bin/hive</value>
>> <description>The path to the Hive executable.</description>
>> </property>
>>
>> </configuration>
>>
>>
>> curl -s -d user.name=hadoop \
>> > -d execute="show+tables;" \
>> > -d statusdir="s3n://pearson-alto-hadoop/webhcat/hive" \
>> > 'http://10.201.5.28:50111/templeton/v1/hive'
>> {"id":"job_201311281741_0020"}
>>
>>
>> When I check the statusdir and jobs folders I see the job had an exit
>> status of 1 so it wasn't successful.
>>
>> hadoop fs -ls s3n://pearson-alto-hadoop/webhcat/hive
>> Found 1 items
>> -rwxrwxrwx 1 2 2013-11-29 15:15 /webhcat/hive/exit
>>
>>
>> hadoop fs -ls /templeton-hadoop/jobs/job_201311281741_0020
>> Found 3 items
>> -rw-r--r-- 1 hadoop supergroup 4 2013-11-29 15:15
>> /templeton-hadoop/jobs/job_201311281741_0020/completed
>> -rw-r--r-- 1 hadoop supergroup 1 2013-11-29 15:15
>> /templeton-hadoop/jobs/job_201311281741_0020/exitValue
>> -rw-r--r-- 1 hadoop supergroup 6 2013-11-29 15:15
>> /templeton-hadoop/jobs/job_201311281741_0020/user
>>
>> Here is what I see in the logs.
>>
>> DEBUG | 29 Nov 2013 15:15:36,133 | org.apache.hcatalog.templeton.Server |
>> queued job job_201311281741_0020 in 13403 ms
>> DEBUG | 29 Nov 2013 15:16:09,583 |
>> org.apache.hcatalog.templeton.tool.HDFSStorage | Couldn't find
>> /templeton-hadoop/jobs/job_201311281741_0020/notified: File does not exist:
>> /templeton-hadoop/jobs/job_201311281741_0020/notified
>> DEBUG | 29 Nov 2013 15:16:09,584 |
>> org.apache.hcatalog.templeton.tool.HDFSStorage | Couldn't find
>> /templeton-hadoop/jobs/job_201311281741_0020/callback: File does not exist:
>> /templeton-hadoop/jobs/job_201311281741_0020/callback
>>
>>
>> How do I figure out the reason for failure?
>>
>> Thanks,
>> Jonathan
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Using Hive with WebHCat
Posted by Eugene Koifman <ek...@hortonworks.com>.
It may be worth looking in webhcat.log and using job tracker UI
On Mon, Dec 2, 2013 at 6:21 AM, Jonathan Hodges <ho...@gmail.com> wrote:
> Hi,
>
> I have setup WebHCat that is bundled with Hive 0.11.0. I am able to kick
> of map reduce jobs with the REST API successfully. However I am having
> some issues with Hive commands over REST. The following is my
> $TEMPLETON_HOME/webhcat-site.xml.
>
>
> <?xml version="1.0" encoding="UTF-8"?>
> <!-- The default settings for Templeton. -->
> <!-- Edit templeton-site.xml to change settings for your local -->
> <!-- install. -->
>
> <configuration>
>
> <property>
> <name>templeton.pig.archive</name>
>
> <value>s3n://pearson-alto-hadoop/apps/webhcat/archives/pig-0.11.1.tar.gz</value>
> <description>The path to the Pig archive.</description>
> </property>
>
> <property>
> <name>templeton.pig.path</name>
> <value>pig-0.11.1.tar.gz/pig-0.11.1/bin/pig</value>
> <description>The path to the Pig executable.</description>
> </property>
>
> <property>
> <name>templeton.hive.archive</name>
>
> <value>s3n://pearson-alto-hadoop/apps/webhcat/archives/hive-0.11.0.tar.gz</value>
> <description>The path to the Hive archive.</description>
> </property>
>
> <property>
> <name>templeton.hive.path</name>
> <value>hive-0.11.0.tar.gz/hive-0.11.0-bin/bin/hive</value>
> <description>The path to the Hive executable.</description>
> </property>
>
> </configuration>
>
>
> curl -s -d user.name=hadoop \
> > -d execute="show+tables;" \
> > -d statusdir="s3n://pearson-alto-hadoop/webhcat/hive" \
> > 'http://10.201.5.28:50111/templeton/v1/hive'
> {"id":"job_201311281741_0020"}
>
>
> When I check the statusdir and jobs folders I see the job had an exit
> status of 1 so it wasn't successful.
>
> hadoop fs -ls s3n://pearson-alto-hadoop/webhcat/hive
> Found 1 items
> -rwxrwxrwx 1 2 2013-11-29 15:15 /webhcat/hive/exit
>
>
> hadoop fs -ls /templeton-hadoop/jobs/job_201311281741_0020
> Found 3 items
> -rw-r--r-- 1 hadoop supergroup 4 2013-11-29 15:15
> /templeton-hadoop/jobs/job_201311281741_0020/completed
> -rw-r--r-- 1 hadoop supergroup 1 2013-11-29 15:15
> /templeton-hadoop/jobs/job_201311281741_0020/exitValue
> -rw-r--r-- 1 hadoop supergroup 6 2013-11-29 15:15
> /templeton-hadoop/jobs/job_201311281741_0020/user
>
> Here is what I see in the logs.
>
> DEBUG | 29 Nov 2013 15:15:36,133 | org.apache.hcatalog.templeton.Server |
> queued job job_201311281741_0020 in 13403 ms
> DEBUG | 29 Nov 2013 15:16:09,583 |
> org.apache.hcatalog.templeton.tool.HDFSStorage | Couldn't find
> /templeton-hadoop/jobs/job_201311281741_0020/notified: File does not exist:
> /templeton-hadoop/jobs/job_201311281741_0020/notified
> DEBUG | 29 Nov 2013 15:16:09,584 |
> org.apache.hcatalog.templeton.tool.HDFSStorage | Couldn't find
> /templeton-hadoop/jobs/job_201311281741_0020/callback: File does not exist:
> /templeton-hadoop/jobs/job_201311281741_0020/callback
>
>
> How do I figure out the reason for failure?
>
> Thanks,
> Jonathan
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Using Hive with WebHCat
Posted by Jonathan Hodges <ho...@gmail.com>.
Would it be advisable to try 0.12, maybe this issue is resolved?
On Wed, Dec 4, 2013 at 6:17 PM, Jonathan Hodges <ho...@gmail.com> wrote:
> Hi Thejas,
>
> Thanks for your reply. The 'templeton.storage.root' property is set to
> the default value, '/templeton-hadoop'. Sorry I wasn't clear above but the
> directory listing is the 'templeton.storage.root' directory on HDFS.
>
> hadoop fs -ls /templeton-hadoop/jobs/job_201311281741_0020
> Found 3 items
> -rw-r--r-- 1 hadoop supergroup 4 2013-11-29 15:15
> /templeton-hadoop/jobs/job_201311281741_0020/completed
> -rw-r--r-- 1 hadoop supergroup 1 2013-11-29 15:15
> /templeton-hadoop/jobs/job_201311281741_0020/exitValue
> -rw-r--r-- 1 hadoop supergroup 6 2013-11-29 15:15
> /templeton-hadoop/jobs/job_201311281741_0020/user
>
> Any other ideas? Could using S3 instead of HDFS for the Pig and Hive
> archives be a problem? Based on the logs it seems to find the archives
> just fine and fails somewhere in the Hive execution.
>
> -Jonathan
>
>
>
> On Tue, Dec 3, 2013 at 6:23 PM, Thejas Nair <th...@hortonworks.com>wrote:
>
>> Can you try setting "templeton.storage.root" in webhcat-site.xml to a
>> directory that exists ?
>>
>>
>> On Mon, Dec 2, 2013 at 6:21 AM, Jonathan Hodges <ho...@gmail.com>
>> wrote:
>> > Hi,
>> >
>> > I have setup WebHCat that is bundled with Hive 0.11.0. I am able to
>> kick of
>> > map reduce jobs with the REST API successfully. However I am having
>> some
>> > issues with Hive commands over REST. The following is my
>> > $TEMPLETON_HOME/webhcat-site.xml.
>> >
>> >
>> > <?xml version="1.0" encoding="UTF-8"?>
>> > <!-- The default settings for Templeton. -->
>> > <!-- Edit templeton-site.xml to change settings for your local -->
>> > <!-- install. -->
>> >
>> > <configuration>
>> >
>> > <property>
>> > <name>templeton.pig.archive</name>
>> >
>> >
>> <value>s3n://pearson-alto-hadoop/apps/webhcat/archives/pig-0.11.1.tar.gz</value>
>> > <description>The path to the Pig archive.</description>
>> > </property>
>> >
>> > <property>
>> > <name>templeton.pig.path</name>
>> > <value>pig-0.11.1.tar.gz/pig-0.11.1/bin/pig</value>
>> > <description>The path to the Pig executable.</description>
>> > </property>
>> >
>> > <property>
>> > <name>templeton.hive.archive</name>
>> >
>> >
>> <value>s3n://pearson-alto-hadoop/apps/webhcat/archives/hive-0.11.0.tar.gz</value>
>> > <description>The path to the Hive archive.</description>
>> > </property>
>> >
>> > <property>
>> > <name>templeton.hive.path</name>
>> > <value>hive-0.11.0.tar.gz/hive-0.11.0-bin/bin/hive</value>
>> > <description>The path to the Hive executable.</description>
>> > </property>
>> >
>> > </configuration>
>> >
>> >
>> > curl -s -d user.name=hadoop \
>> >> -d execute="show+tables;" \
>> >> -d statusdir="s3n://pearson-alto-hadoop/webhcat/hive" \
>> >> 'http://10.201.5.28:50111/templeton/v1/hive'
>> > {"id":"job_201311281741_0020"}
>> >
>> >
>> > When I check the statusdir and jobs folders I see the job had an exit
>> status
>> > of 1 so it wasn't successful.
>> >
>> > hadoop fs -ls s3n://pearson-alto-hadoop/webhcat/hive
>> > Found 1 items
>> > -rwxrwxrwx 1 2 2013-11-29 15:15 /webhcat/hive/exit
>> >
>> >
>> > hadoop fs -ls /templeton-hadoop/jobs/job_201311281741_0020
>> > Found 3 items
>> > -rw-r--r-- 1 hadoop supergroup 4 2013-11-29 15:15
>> > /templeton-hadoop/jobs/job_201311281741_0020/completed
>> > -rw-r--r-- 1 hadoop supergroup 1 2013-11-29 15:15
>> > /templeton-hadoop/jobs/job_201311281741_0020/exitValue
>> > -rw-r--r-- 1 hadoop supergroup 6 2013-11-29 15:15
>> > /templeton-hadoop/jobs/job_201311281741_0020/user
>> >
>> > Here is what I see in the logs.
>> >
>> > DEBUG | 29 Nov 2013 15:15:36,133 | org.apache.hcatalog.templeton.Server
>> |
>> > queued job job_201311281741_0020 in 13403 ms
>> > DEBUG | 29 Nov 2013 15:16:09,583 |
>> > org.apache.hcatalog.templeton.tool.HDFSStorage | Couldn't find
>> > /templeton-hadoop/jobs/job_201311281741_0020/notified: File does not
>> exist:
>> > /templeton-hadoop/jobs/job_201311281741_0020/notified
>> > DEBUG | 29 Nov 2013 15:16:09,584 |
>> > org.apache.hcatalog.templeton.tool.HDFSStorage | Couldn't find
>> > /templeton-hadoop/jobs/job_201311281741_0020/callback: File does not
>> exist:
>> > /templeton-hadoop/jobs/job_201311281741_0020/callback
>> >
>> >
>> > How do I figure out the reason for failure?
>> >
>> > Thanks,
>> > Jonathan
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified
>> that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender
>> immediately
>> and delete it from your system. Thank You.
>>
>
>
Re: Using Hive with WebHCat
Posted by Jonathan Hodges <ho...@gmail.com>.
Hi Thejas,
Thanks for your reply. The 'templeton.storage.root' property is set to the
default value, '/templeton-hadoop'. Sorry I wasn't clear above but the
directory listing is the 'templeton.storage.root' directory on HDFS.
hadoop fs -ls /templeton-hadoop/jobs/job_201311281741_0020
Found 3 items
-rw-r--r-- 1 hadoop supergroup 4 2013-11-29 15:15
/templeton-hadoop/jobs/job_201311281741_0020/completed
-rw-r--r-- 1 hadoop supergroup 1 2013-11-29 15:15
/templeton-hadoop/jobs/job_201311281741_0020/exitValue
-rw-r--r-- 1 hadoop supergroup 6 2013-11-29 15:15
/templeton-hadoop/jobs/job_201311281741_0020/user
Any other ideas? Could using S3 instead of HDFS for the Pig and Hive
archives be a problem? Based on the logs it seems to find the archives
just fine and fails somewhere in the Hive execution.
-Jonathan
On Tue, Dec 3, 2013 at 6:23 PM, Thejas Nair <th...@hortonworks.com> wrote:
> Can you try setting "templeton.storage.root" in webhcat-site.xml to a
> directory that exists ?
>
>
> On Mon, Dec 2, 2013 at 6:21 AM, Jonathan Hodges <ho...@gmail.com> wrote:
> > Hi,
> >
> > I have setup WebHCat that is bundled with Hive 0.11.0. I am able to
> kick of
> > map reduce jobs with the REST API successfully. However I am having some
> > issues with Hive commands over REST. The following is my
> > $TEMPLETON_HOME/webhcat-site.xml.
> >
> >
> > <?xml version="1.0" encoding="UTF-8"?>
> > <!-- The default settings for Templeton. -->
> > <!-- Edit templeton-site.xml to change settings for your local -->
> > <!-- install. -->
> >
> > <configuration>
> >
> > <property>
> > <name>templeton.pig.archive</name>
> >
> >
> <value>s3n://pearson-alto-hadoop/apps/webhcat/archives/pig-0.11.1.tar.gz</value>
> > <description>The path to the Pig archive.</description>
> > </property>
> >
> > <property>
> > <name>templeton.pig.path</name>
> > <value>pig-0.11.1.tar.gz/pig-0.11.1/bin/pig</value>
> > <description>The path to the Pig executable.</description>
> > </property>
> >
> > <property>
> > <name>templeton.hive.archive</name>
> >
> >
> <value>s3n://pearson-alto-hadoop/apps/webhcat/archives/hive-0.11.0.tar.gz</value>
> > <description>The path to the Hive archive.</description>
> > </property>
> >
> > <property>
> > <name>templeton.hive.path</name>
> > <value>hive-0.11.0.tar.gz/hive-0.11.0-bin/bin/hive</value>
> > <description>The path to the Hive executable.</description>
> > </property>
> >
> > </configuration>
> >
> >
> > curl -s -d user.name=hadoop \
> >> -d execute="show+tables;" \
> >> -d statusdir="s3n://pearson-alto-hadoop/webhcat/hive" \
> >> 'http://10.201.5.28:50111/templeton/v1/hive'
> > {"id":"job_201311281741_0020"}
> >
> >
> > When I check the statusdir and jobs folders I see the job had an exit
> status
> > of 1 so it wasn't successful.
> >
> > hadoop fs -ls s3n://pearson-alto-hadoop/webhcat/hive
> > Found 1 items
> > -rwxrwxrwx 1 2 2013-11-29 15:15 /webhcat/hive/exit
> >
> >
> > hadoop fs -ls /templeton-hadoop/jobs/job_201311281741_0020
> > Found 3 items
> > -rw-r--r-- 1 hadoop supergroup 4 2013-11-29 15:15
> > /templeton-hadoop/jobs/job_201311281741_0020/completed
> > -rw-r--r-- 1 hadoop supergroup 1 2013-11-29 15:15
> > /templeton-hadoop/jobs/job_201311281741_0020/exitValue
> > -rw-r--r-- 1 hadoop supergroup 6 2013-11-29 15:15
> > /templeton-hadoop/jobs/job_201311281741_0020/user
> >
> > Here is what I see in the logs.
> >
> > DEBUG | 29 Nov 2013 15:15:36,133 | org.apache.hcatalog.templeton.Server |
> > queued job job_201311281741_0020 in 13403 ms
> > DEBUG | 29 Nov 2013 15:16:09,583 |
> > org.apache.hcatalog.templeton.tool.HDFSStorage | Couldn't find
> > /templeton-hadoop/jobs/job_201311281741_0020/notified: File does not
> exist:
> > /templeton-hadoop/jobs/job_201311281741_0020/notified
> > DEBUG | 29 Nov 2013 15:16:09,584 |
> > org.apache.hcatalog.templeton.tool.HDFSStorage | Couldn't find
> > /templeton-hadoop/jobs/job_201311281741_0020/callback: File does not
> exist:
> > /templeton-hadoop/jobs/job_201311281741_0020/callback
> >
> >
> > How do I figure out the reason for failure?
> >
> > Thanks,
> > Jonathan
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Using Hive with WebHCat
Posted by Thejas Nair <th...@hortonworks.com>.
Can you try setting "templeton.storage.root" in webhcat-site.xml to a
directory that exists ?
On Mon, Dec 2, 2013 at 6:21 AM, Jonathan Hodges <ho...@gmail.com> wrote:
> Hi,
>
> I have setup WebHCat that is bundled with Hive 0.11.0. I am able to kick of
> map reduce jobs with the REST API successfully. However I am having some
> issues with Hive commands over REST. The following is my
> $TEMPLETON_HOME/webhcat-site.xml.
>
>
> <?xml version="1.0" encoding="UTF-8"?>
> <!-- The default settings for Templeton. -->
> <!-- Edit templeton-site.xml to change settings for your local -->
> <!-- install. -->
>
> <configuration>
>
> <property>
> <name>templeton.pig.archive</name>
>
> <value>s3n://pearson-alto-hadoop/apps/webhcat/archives/pig-0.11.1.tar.gz</value>
> <description>The path to the Pig archive.</description>
> </property>
>
> <property>
> <name>templeton.pig.path</name>
> <value>pig-0.11.1.tar.gz/pig-0.11.1/bin/pig</value>
> <description>The path to the Pig executable.</description>
> </property>
>
> <property>
> <name>templeton.hive.archive</name>
>
> <value>s3n://pearson-alto-hadoop/apps/webhcat/archives/hive-0.11.0.tar.gz</value>
> <description>The path to the Hive archive.</description>
> </property>
>
> <property>
> <name>templeton.hive.path</name>
> <value>hive-0.11.0.tar.gz/hive-0.11.0-bin/bin/hive</value>
> <description>The path to the Hive executable.</description>
> </property>
>
> </configuration>
>
>
> curl -s -d user.name=hadoop \
>> -d execute="show+tables;" \
>> -d statusdir="s3n://pearson-alto-hadoop/webhcat/hive" \
>> 'http://10.201.5.28:50111/templeton/v1/hive'
> {"id":"job_201311281741_0020"}
>
>
> When I check the statusdir and jobs folders I see the job had an exit status
> of 1 so it wasn't successful.
>
> hadoop fs -ls s3n://pearson-alto-hadoop/webhcat/hive
> Found 1 items
> -rwxrwxrwx 1 2 2013-11-29 15:15 /webhcat/hive/exit
>
>
> hadoop fs -ls /templeton-hadoop/jobs/job_201311281741_0020
> Found 3 items
> -rw-r--r-- 1 hadoop supergroup 4 2013-11-29 15:15
> /templeton-hadoop/jobs/job_201311281741_0020/completed
> -rw-r--r-- 1 hadoop supergroup 1 2013-11-29 15:15
> /templeton-hadoop/jobs/job_201311281741_0020/exitValue
> -rw-r--r-- 1 hadoop supergroup 6 2013-11-29 15:15
> /templeton-hadoop/jobs/job_201311281741_0020/user
>
> Here is what I see in the logs.
>
> DEBUG | 29 Nov 2013 15:15:36,133 | org.apache.hcatalog.templeton.Server |
> queued job job_201311281741_0020 in 13403 ms
> DEBUG | 29 Nov 2013 15:16:09,583 |
> org.apache.hcatalog.templeton.tool.HDFSStorage | Couldn't find
> /templeton-hadoop/jobs/job_201311281741_0020/notified: File does not exist:
> /templeton-hadoop/jobs/job_201311281741_0020/notified
> DEBUG | 29 Nov 2013 15:16:09,584 |
> org.apache.hcatalog.templeton.tool.HDFSStorage | Couldn't find
> /templeton-hadoop/jobs/job_201311281741_0020/callback: File does not exist:
> /templeton-hadoop/jobs/job_201311281741_0020/callback
>
>
> How do I figure out the reason for failure?
>
> Thanks,
> Jonathan
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.