You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Harshal Vora <ha...@gmail.com> on 2014/10/14 06:47:04 UTC
oozie hadoop callback notification failure
Logs on Jobtracker machine:
ERROR org.apache.hadoop.mapred.JobEndNotifier: Notification failure
[URL:
http://:11000/oozie/callback?id=0000443-141012054231695-oozie-oozi-W@visitInfoProcessor-cascading&status=SUCCEEDED&
remaining retries: 0 interval: 30000]
~/.bashrc on oozie machine.
export OOZIE_URL=http://localhost:11000/oozie
Is the hostname empty in the callback URL because it is configured to
localhost on oozie machine?
For some reason the transition between oozie actions and job status
change is very very slow.
Regards,
--
Regards,
Harshal Vora
www.radiolocus.com
Re: oozie hadoop callback notification failure
Posted by Shwetha GS <sh...@inmobi.com>.
Check the status of coordinator actions. If the number of running actions
is not same as coord concurrency, you can check if the enough actions are
not materialised. You can look at materialisation configs to tune it
On Sat, Oct 18, 2014 at 4:33 PM, Harshal Vora <ha...@gmail.com> wrote:
> Dear Swetha,
>
> We fixed the OOZIE_HTTP_HOSTNAME issue temporarily by hard-coding the
> machine name and adding dns entries on every hadoop TT machine.
> But the jobs still seem slower than expected. Almost all the time we have
> free mapper/reducer slots available on hadoop but enough jobs are not
> running.
>
> Apart from setting the concurrency of the coordinator jobs to a higher
> number, is there any other configuration that can be changed so that idle
> time is minimum?
> I read on a blog regarding maximum number of threads for jobs in each
> state and even in the oozie configuration I can see a lot of threadpools
> defined with max thread limits.
>
> Is there any particular config that we need to change?
> We have a backlog of jobs for one month and there are a lot of jobs
> available that can be submitted on the cluster, so according to me all the
> hadoop slots should be completely utilized.
>
> Regards,
>
>
> On 10/14/2014 11:56 AM, Shwetha GS wrote:
>
>> This is what oozie server does:
>> export OOZIE_HTTP_HOSTNAME=`hostname -f`
>> export OOZIE_BASE_URL="http://
>> ${OOZIE_HTTP_HOSTNAME}:${OOZIE_HTTP_PORT}/oozie"
>> catalina_opts="${catalina_opts} -Doozie.base.url=${OOZIE_BASE_URL}";
>>
>> Callback url is picked up from oozie.base.url config which is set to
>> http://:11000/oozie
>> in your case which means oozie failed to get hostname
>>
>> The jobs run on TT, so the host resolution should work on TTs
>>
>> -Shwetha
>>
>>
>> On Tue, Oct 14, 2014 at 11:45 AM, Harshal Vora <ha...@gmail.com>
>> wrote:
>>
>> Re-sending the shell output
>>>
>>> ------------------------------------------------------------
>>> ------------------------------
>>>
>>> [xxx@server-lp-4 ~]$ hostname -f
>>> server-lp-4
>>> [xxx@server-lp-4 ~]$ sudo su yyy
>>> [sudo] password for xxx:
>>> bash-4.1$ hostname -f
>>> server-lp-4
>>> ------------------------------------------------------------
>>> ------------------------------
>>>
>>>
>>>
>>> On 10/14/2014 11:43 AM, Harshal Vora wrote:
>>>
>>> Dear Shwetha,
>>>>
>>>> Below is the output of shell on oozie machine for both regular user
>>>> (xxx)
>>>> as well as oozie user (yyy) which is used to submit the jobs.
>>>> For security purpose I have changed the user names to xxx and yyy.
>>>>
>>>> We use /etc/hosts for DNS mapping as we do not have a central DNS
>>>> server.
>>>> So I believe we will have to put the oozie machines mapping i.e.
>>>> "server-lp-4" on jobtracker machine.
>>>>
>>>> ------------------------------------------------------------
>>>> ------------------------------
>>>>
>>>> [xxx@server-lp-4 ~]$ hostname -f
>>>> server-lp-4
>>>> [xxx@packet-server-lp-4 ~]$ sudo su yyy
>>>> [sudo] password for xxx:
>>>> bash-4.1$ hostname -f
>>>> server-lp-4
>>>> ------------------------------------------------------------
>>>> ------------------------------
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 10/14/2014 11:36 AM, Shwetha GS wrote:
>>>>
>>>> If the command 'hostname -f' was resolving correctly, callback url
>>>>> would
>>>>> have the host. So, I don't think the command is returning the host.
>>>>>
>>>>> Once you resolve this, hadoop machine where the jobs runs should
>>>>> resolve
>>>>> the host.
>>>>>
>>>>> On Tue, Oct 14, 2014 at 11:14 AM, Harshal Vora <ha...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Dear Swetha,
>>>>>
>>>>>> On oozie machine, hostname points to "server-lp-4" which is the name
>>>>>> we
>>>>>> have given to our machine.
>>>>>> Does this mean we need a mapping in DNS for this hostname on the
>>>>>> hadoop
>>>>>> machine?
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>>
>>>>>> On 10/14/2014 10:49 AM, Shwetha GS wrote:
>>>>>>
>>>>>> Oozie uses 'hostname -f' to get the hostname which is used in
>>>>>> setting
>>>>>>
>>>>>>> callback url. Check if its resolving correctly. This is the reason
>>>>>>> for
>>>>>>> delay between job completion and oozie status update
>>>>>>>
>>>>>>> -Shwetha
>>>>>>>
>>>>>>> On Tue, Oct 14, 2014 at 10:17 AM, Harshal Vora <harshal022@gmail.com
>>>>>>> >
>>>>>>> wrote:
>>>>>>>
>>>>>>> Logs on Jobtracker machine:
>>>>>>>
>>>>>>> ERROR org.apache.hadoop.mapred.JobEndNotifier: Notification failure
>>>>>>>> [URL:
>>>>>>>> http://:11000/oozie/callback?id=0000443-141012054231695-
>>>>>>>> oozie-oozi-W@
>>>>>>>> visitInfoProcessor-cascading&status=SUCCEEDED& remaining retries: 0
>>>>>>>> interval: 30000]
>>>>>>>>
>>>>>>>> ~/.bashrc on oozie machine.
>>>>>>>> export OOZIE_URL=http://localhost:11000/oozie
>>>>>>>>
>>>>>>>> Is the hostname empty in the callback URL because it is configured
>>>>>>>> to
>>>>>>>> localhost on oozie machine?
>>>>>>>>
>>>>>>>> For some reason the transition between oozie actions and job status
>>>>>>>> change
>>>>>>>> is very very slow.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards,
>>>>>>>> Harshal Vora
>>>>>>>> www.radiolocus.com
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>> Regards,
>>>>>> Harshal Vora
>>>>>> www.radiolocus.com
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> --
>>> Regards,
>>> Harshal Vora
>>> www.radiolocus.com
>>>
>>>
>>>
>
> --
> Regards,
> Harshal Vora
> www.radiolocus.com
>
>
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.
Re: oozie hadoop callback notification failure
Posted by Harshal Vora <ha...@gmail.com>.
Dear Swetha,
We fixed the OOZIE_HTTP_HOSTNAME issue temporarily by hard-coding the
machine name and adding dns entries on every hadoop TT machine.
But the jobs still seem slower than expected. Almost all the time we
have free mapper/reducer slots available on hadoop but enough jobs are
not running.
Apart from setting the concurrency of the coordinator jobs to a higher
number, is there any other configuration that can be changed so that
idle time is minimum?
I read on a blog regarding maximum number of threads for jobs in each
state and even in the oozie configuration I can see a lot of threadpools
defined with max thread limits.
Is there any particular config that we need to change?
We have a backlog of jobs for one month and there are a lot of jobs
available that can be submitted on the cluster, so according to me all
the hadoop slots should be completely utilized.
Regards,
On 10/14/2014 11:56 AM, Shwetha GS wrote:
> This is what oozie server does:
> export OOZIE_HTTP_HOSTNAME=`hostname -f`
> export OOZIE_BASE_URL="http://
> ${OOZIE_HTTP_HOSTNAME}:${OOZIE_HTTP_PORT}/oozie"
> catalina_opts="${catalina_opts} -Doozie.base.url=${OOZIE_BASE_URL}";
>
> Callback url is picked up from oozie.base.url config which is set to
> http://:11000/oozie
> in your case which means oozie failed to get hostname
>
> The jobs run on TT, so the host resolution should work on TTs
>
> -Shwetha
>
>
> On Tue, Oct 14, 2014 at 11:45 AM, Harshal Vora <ha...@gmail.com> wrote:
>
>> Re-sending the shell output
>>
>> ------------------------------------------------------------------------------------------
>>
>> [xxx@server-lp-4 ~]$ hostname -f
>> server-lp-4
>> [xxx@server-lp-4 ~]$ sudo su yyy
>> [sudo] password for xxx:
>> bash-4.1$ hostname -f
>> server-lp-4
>> ------------------------------------------------------------------------------------------
>>
>>
>>
>> On 10/14/2014 11:43 AM, Harshal Vora wrote:
>>
>>> Dear Shwetha,
>>>
>>> Below is the output of shell on oozie machine for both regular user (xxx)
>>> as well as oozie user (yyy) which is used to submit the jobs.
>>> For security purpose I have changed the user names to xxx and yyy.
>>>
>>> We use /etc/hosts for DNS mapping as we do not have a central DNS server.
>>> So I believe we will have to put the oozie machines mapping i.e.
>>> "server-lp-4" on jobtracker machine.
>>>
>>> ------------------------------------------------------------------------------------------
>>>
>>> [xxx@server-lp-4 ~]$ hostname -f
>>> server-lp-4
>>> [xxx@packet-server-lp-4 ~]$ sudo su yyy
>>> [sudo] password for xxx:
>>> bash-4.1$ hostname -f
>>> server-lp-4
>>> ------------------------------------------------------------------------------------------
>>>
>>>
>>>
>>>
>>>
>>> On 10/14/2014 11:36 AM, Shwetha GS wrote:
>>>
>>>> If the command 'hostname -f' was resolving correctly, callback url would
>>>> have the host. So, I don't think the command is returning the host.
>>>>
>>>> Once you resolve this, hadoop machine where the jobs runs should resolve
>>>> the host.
>>>>
>>>> On Tue, Oct 14, 2014 at 11:14 AM, Harshal Vora <ha...@gmail.com>
>>>> wrote:
>>>>
>>>> Dear Swetha,
>>>>> On oozie machine, hostname points to "server-lp-4" which is the name we
>>>>> have given to our machine.
>>>>> Does this mean we need a mapping in DNS for this hostname on the hadoop
>>>>> machine?
>>>>>
>>>>> Regards,
>>>>>
>>>>>
>>>>> On 10/14/2014 10:49 AM, Shwetha GS wrote:
>>>>>
>>>>> Oozie uses 'hostname -f' to get the hostname which is used in setting
>>>>>> callback url. Check if its resolving correctly. This is the reason for
>>>>>> delay between job completion and oozie status update
>>>>>>
>>>>>> -Shwetha
>>>>>>
>>>>>> On Tue, Oct 14, 2014 at 10:17 AM, Harshal Vora <ha...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Logs on Jobtracker machine:
>>>>>>
>>>>>>> ERROR org.apache.hadoop.mapred.JobEndNotifier: Notification failure
>>>>>>> [URL:
>>>>>>> http://:11000/oozie/callback?id=0000443-141012054231695-oozie-oozi-W@
>>>>>>> visitInfoProcessor-cascading&status=SUCCEEDED& remaining retries: 0
>>>>>>> interval: 30000]
>>>>>>>
>>>>>>> ~/.bashrc on oozie machine.
>>>>>>> export OOZIE_URL=http://localhost:11000/oozie
>>>>>>>
>>>>>>> Is the hostname empty in the callback URL because it is configured to
>>>>>>> localhost on oozie machine?
>>>>>>>
>>>>>>> For some reason the transition between oozie actions and job status
>>>>>>> change
>>>>>>> is very very slow.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Harshal Vora
>>>>>>> www.radiolocus.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>> Regards,
>>>>> Harshal Vora
>>>>> www.radiolocus.com
>>>>>
>>>>>
>>>>>
>>>
>> --
>> Regards,
>> Harshal Vora
>> www.radiolocus.com
>>
>>
--
Regards,
Harshal Vora
www.radiolocus.com
Re: oozie hadoop callback notification failure
Posted by Shwetha GS <sh...@inmobi.com>.
This is what oozie server does:
export OOZIE_HTTP_HOSTNAME=`hostname -f`
export OOZIE_BASE_URL="http://
${OOZIE_HTTP_HOSTNAME}:${OOZIE_HTTP_PORT}/oozie"
catalina_opts="${catalina_opts} -Doozie.base.url=${OOZIE_BASE_URL}";
Callback url is picked up from oozie.base.url config which is set to
http://:11000/oozie
in your case which means oozie failed to get hostname
The jobs run on TT, so the host resolution should work on TTs
-Shwetha
On Tue, Oct 14, 2014 at 11:45 AM, Harshal Vora <ha...@gmail.com> wrote:
> Re-sending the shell output
>
> ------------------------------------------------------------------------------------------
>
> [xxx@server-lp-4 ~]$ hostname -f
> server-lp-4
> [xxx@server-lp-4 ~]$ sudo su yyy
> [sudo] password for xxx:
> bash-4.1$ hostname -f
> server-lp-4
> ------------------------------------------------------------------------------------------
>
>
>
> On 10/14/2014 11:43 AM, Harshal Vora wrote:
>
>> Dear Shwetha,
>>
>> Below is the output of shell on oozie machine for both regular user (xxx)
>> as well as oozie user (yyy) which is used to submit the jobs.
>> For security purpose I have changed the user names to xxx and yyy.
>>
>> We use /etc/hosts for DNS mapping as we do not have a central DNS server.
>> So I believe we will have to put the oozie machines mapping i.e.
>> "server-lp-4" on jobtracker machine.
>>
>> ------------------------------------------------------------------------------------------
>>
>> [xxx@server-lp-4 ~]$ hostname -f
>> server-lp-4
>> [xxx@packet-server-lp-4 ~]$ sudo su yyy
>> [sudo] password for xxx:
>> bash-4.1$ hostname -f
>> server-lp-4
>> ------------------------------------------------------------------------------------------
>>
>>
>>
>>
>>
>> On 10/14/2014 11:36 AM, Shwetha GS wrote:
>>
>>> If the command 'hostname -f' was resolving correctly, callback url would
>>> have the host. So, I don't think the command is returning the host.
>>>
>>> Once you resolve this, hadoop machine where the jobs runs should resolve
>>> the host.
>>>
>>> On Tue, Oct 14, 2014 at 11:14 AM, Harshal Vora <ha...@gmail.com>
>>> wrote:
>>>
>>> Dear Swetha,
>>>>
>>>> On oozie machine, hostname points to "server-lp-4" which is the name we
>>>> have given to our machine.
>>>> Does this mean we need a mapping in DNS for this hostname on the hadoop
>>>> machine?
>>>>
>>>> Regards,
>>>>
>>>>
>>>> On 10/14/2014 10:49 AM, Shwetha GS wrote:
>>>>
>>>> Oozie uses 'hostname -f' to get the hostname which is used in setting
>>>>> callback url. Check if its resolving correctly. This is the reason for
>>>>> delay between job completion and oozie status update
>>>>>
>>>>> -Shwetha
>>>>>
>>>>> On Tue, Oct 14, 2014 at 10:17 AM, Harshal Vora <ha...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Logs on Jobtracker machine:
>>>>>
>>>>>> ERROR org.apache.hadoop.mapred.JobEndNotifier: Notification failure
>>>>>> [URL:
>>>>>> http://:11000/oozie/callback?id=0000443-141012054231695-oozie-oozi-W@
>>>>>> visitInfoProcessor-cascading&status=SUCCEEDED& remaining retries: 0
>>>>>> interval: 30000]
>>>>>>
>>>>>> ~/.bashrc on oozie machine.
>>>>>> export OOZIE_URL=http://localhost:11000/oozie
>>>>>>
>>>>>> Is the hostname empty in the callback URL because it is configured to
>>>>>> localhost on oozie machine?
>>>>>>
>>>>>> For some reason the transition between oozie actions and job status
>>>>>> change
>>>>>> is very very slow.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Harshal Vora
>>>>>> www.radiolocus.com
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>> Regards,
>>>> Harshal Vora
>>>> www.radiolocus.com
>>>>
>>>>
>>>>
>>
>>
>
> --
> Regards,
> Harshal Vora
> www.radiolocus.com
>
>
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.
Re: oozie hadoop callback notification failure
Posted by Harshal Vora <ha...@gmail.com>.
Re-sending the shell output
------------------------------------------------------------------------------------------
[xxx@server-lp-4 ~]$ hostname -f
server-lp-4
[xxx@server-lp-4 ~]$ sudo su yyy
[sudo] password for xxx:
bash-4.1$ hostname -f
server-lp-4
------------------------------------------------------------------------------------------
On 10/14/2014 11:43 AM, Harshal Vora wrote:
> Dear Shwetha,
>
> Below is the output of shell on oozie machine for both regular user
> (xxx) as well as oozie user (yyy) which is used to submit the jobs.
> For security purpose I have changed the user names to xxx and yyy.
>
> We use /etc/hosts for DNS mapping as we do not have a central DNS server.
> So I believe we will have to put the oozie machines mapping i.e.
> "server-lp-4" on jobtracker machine.
>
> ------------------------------------------------------------------------------------------
>
> [xxx@server-lp-4 ~]$ hostname -f
> server-lp-4
> [xxx@packet-server-lp-4 ~]$ sudo su yyy
> [sudo] password for xxx:
> bash-4.1$ hostname -f
> server-lp-4
> ------------------------------------------------------------------------------------------
>
>
>
>
>
> On 10/14/2014 11:36 AM, Shwetha GS wrote:
>> If the command 'hostname -f' was resolving correctly, callback url would
>> have the host. So, I don't think the command is returning the host.
>>
>> Once you resolve this, hadoop machine where the jobs runs should resolve
>> the host.
>>
>> On Tue, Oct 14, 2014 at 11:14 AM, Harshal Vora <ha...@gmail.com>
>> wrote:
>>
>>> Dear Swetha,
>>>
>>> On oozie machine, hostname points to "server-lp-4" which is the name we
>>> have given to our machine.
>>> Does this mean we need a mapping in DNS for this hostname on the hadoop
>>> machine?
>>>
>>> Regards,
>>>
>>>
>>> On 10/14/2014 10:49 AM, Shwetha GS wrote:
>>>
>>>> Oozie uses 'hostname -f' to get the hostname which is used in setting
>>>> callback url. Check if its resolving correctly. This is the reason for
>>>> delay between job completion and oozie status update
>>>>
>>>> -Shwetha
>>>>
>>>> On Tue, Oct 14, 2014 at 10:17 AM, Harshal Vora <ha...@gmail.com>
>>>> wrote:
>>>>
>>>> Logs on Jobtracker machine:
>>>>> ERROR org.apache.hadoop.mapred.JobEndNotifier: Notification failure
>>>>> [URL:
>>>>> http://:11000/oozie/callback?id=0000443-141012054231695-oozie-oozi-W@
>>>>> visitInfoProcessor-cascading&status=SUCCEEDED& remaining retries: 0
>>>>> interval: 30000]
>>>>>
>>>>> ~/.bashrc on oozie machine.
>>>>> export OOZIE_URL=http://localhost:11000/oozie
>>>>>
>>>>> Is the hostname empty in the callback URL because it is configured to
>>>>> localhost on oozie machine?
>>>>>
>>>>> For some reason the transition between oozie actions and job status
>>>>> change
>>>>> is very very slow.
>>>>>
>>>>> Regards,
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Harshal Vora
>>>>> www.radiolocus.com
>>>>>
>>>>>
>>>>>
>>> --
>>> Regards,
>>> Harshal Vora
>>> www.radiolocus.com
>>>
>>>
>
>
--
Regards,
Harshal Vora
www.radiolocus.com
Re: oozie hadoop callback notification failure
Posted by Harshal Vora <ha...@gmail.com>.
Dear Shwetha,
Below is the output of shell on oozie machine for both regular user
(xxx) as well as oozie user (yyy) which is used to submit the jobs.
For security purpose I have changed the user names to xxx and yyy.
We use /etc/hosts for DNS mapping as we do not have a central DNS server.
So I believe we will have to put the oozie machines mapping i.e.
"server-lp-4" on jobtracker machine.
------------------------------------------------------------------------------------------
[xxx@server-lp-4 ~]$ hostname -f
server-lp-4
[xxx@packet-server-lp-4 ~]$ sudo su yyy
[sudo] password for xxx:
bash-4.1$ hostname -f
server-lp-4
------------------------------------------------------------------------------------------
On 10/14/2014 11:36 AM, Shwetha GS wrote:
> If the command 'hostname -f' was resolving correctly, callback url would
> have the host. So, I don't think the command is returning the host.
>
> Once you resolve this, hadoop machine where the jobs runs should resolve
> the host.
>
> On Tue, Oct 14, 2014 at 11:14 AM, Harshal Vora <ha...@gmail.com> wrote:
>
>> Dear Swetha,
>>
>> On oozie machine, hostname points to "server-lp-4" which is the name we
>> have given to our machine.
>> Does this mean we need a mapping in DNS for this hostname on the hadoop
>> machine?
>>
>> Regards,
>>
>>
>> On 10/14/2014 10:49 AM, Shwetha GS wrote:
>>
>>> Oozie uses 'hostname -f' to get the hostname which is used in setting
>>> callback url. Check if its resolving correctly. This is the reason for
>>> delay between job completion and oozie status update
>>>
>>> -Shwetha
>>>
>>> On Tue, Oct 14, 2014 at 10:17 AM, Harshal Vora <ha...@gmail.com>
>>> wrote:
>>>
>>> Logs on Jobtracker machine:
>>>> ERROR org.apache.hadoop.mapred.JobEndNotifier: Notification failure
>>>> [URL:
>>>> http://:11000/oozie/callback?id=0000443-141012054231695-oozie-oozi-W@
>>>> visitInfoProcessor-cascading&status=SUCCEEDED& remaining retries: 0
>>>> interval: 30000]
>>>>
>>>> ~/.bashrc on oozie machine.
>>>> export OOZIE_URL=http://localhost:11000/oozie
>>>>
>>>> Is the hostname empty in the callback URL because it is configured to
>>>> localhost on oozie machine?
>>>>
>>>> For some reason the transition between oozie actions and job status
>>>> change
>>>> is very very slow.
>>>>
>>>> Regards,
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Harshal Vora
>>>> www.radiolocus.com
>>>>
>>>>
>>>>
>> --
>> Regards,
>> Harshal Vora
>> www.radiolocus.com
>>
>>
--
Regards,
Harshal Vora
www.radiolocus.com
Re: oozie hadoop callback notification failure
Posted by Shwetha GS <sh...@inmobi.com>.
If the command 'hostname -f' was resolving correctly, callback url would
have the host. So, I don't think the command is returning the host.
Once you resolve this, hadoop machine where the jobs runs should resolve
the host.
On Tue, Oct 14, 2014 at 11:14 AM, Harshal Vora <ha...@gmail.com> wrote:
> Dear Swetha,
>
> On oozie machine, hostname points to "server-lp-4" which is the name we
> have given to our machine.
> Does this mean we need a mapping in DNS for this hostname on the hadoop
> machine?
>
> Regards,
>
>
> On 10/14/2014 10:49 AM, Shwetha GS wrote:
>
>> Oozie uses 'hostname -f' to get the hostname which is used in setting
>> callback url. Check if its resolving correctly. This is the reason for
>> delay between job completion and oozie status update
>>
>> -Shwetha
>>
>> On Tue, Oct 14, 2014 at 10:17 AM, Harshal Vora <ha...@gmail.com>
>> wrote:
>>
>> Logs on Jobtracker machine:
>>> ERROR org.apache.hadoop.mapred.JobEndNotifier: Notification failure
>>> [URL:
>>> http://:11000/oozie/callback?id=0000443-141012054231695-oozie-oozi-W@
>>> visitInfoProcessor-cascading&status=SUCCEEDED& remaining retries: 0
>>> interval: 30000]
>>>
>>> ~/.bashrc on oozie machine.
>>> export OOZIE_URL=http://localhost:11000/oozie
>>>
>>> Is the hostname empty in the callback URL because it is configured to
>>> localhost on oozie machine?
>>>
>>> For some reason the transition between oozie actions and job status
>>> change
>>> is very very slow.
>>>
>>> Regards,
>>>
>>>
>>> --
>>> Regards,
>>> Harshal Vora
>>> www.radiolocus.com
>>>
>>>
>>>
>
> --
> Regards,
> Harshal Vora
> www.radiolocus.com
>
>
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.
Re: oozie hadoop callback notification failure
Posted by Harshal Vora <ha...@gmail.com>.
Dear Swetha,
On oozie machine, hostname points to "server-lp-4" which is the name we
have given to our machine.
Does this mean we need a mapping in DNS for this hostname on the hadoop
machine?
Regards,
On 10/14/2014 10:49 AM, Shwetha GS wrote:
> Oozie uses 'hostname -f' to get the hostname which is used in setting
> callback url. Check if its resolving correctly. This is the reason for
> delay between job completion and oozie status update
>
> -Shwetha
>
> On Tue, Oct 14, 2014 at 10:17 AM, Harshal Vora <ha...@gmail.com> wrote:
>
>> Logs on Jobtracker machine:
>> ERROR org.apache.hadoop.mapred.JobEndNotifier: Notification failure [URL:
>> http://:11000/oozie/callback?id=0000443-141012054231695-oozie-oozi-W@
>> visitInfoProcessor-cascading&status=SUCCEEDED& remaining retries: 0
>> interval: 30000]
>>
>> ~/.bashrc on oozie machine.
>> export OOZIE_URL=http://localhost:11000/oozie
>>
>> Is the hostname empty in the callback URL because it is configured to
>> localhost on oozie machine?
>>
>> For some reason the transition between oozie actions and job status change
>> is very very slow.
>>
>> Regards,
>>
>>
>> --
>> Regards,
>> Harshal Vora
>> www.radiolocus.com
>>
>>
--
Regards,
Harshal Vora
www.radiolocus.com
Re: oozie hadoop callback notification failure
Posted by Shwetha GS <sh...@inmobi.com>.
Oozie uses 'hostname -f' to get the hostname which is used in setting
callback url. Check if its resolving correctly. This is the reason for
delay between job completion and oozie status update
-Shwetha
On Tue, Oct 14, 2014 at 10:17 AM, Harshal Vora <ha...@gmail.com> wrote:
> Logs on Jobtracker machine:
> ERROR org.apache.hadoop.mapred.JobEndNotifier: Notification failure [URL:
> http://:11000/oozie/callback?id=0000443-141012054231695-oozie-oozi-W@
> visitInfoProcessor-cascading&status=SUCCEEDED& remaining retries: 0
> interval: 30000]
>
> ~/.bashrc on oozie machine.
> export OOZIE_URL=http://localhost:11000/oozie
>
> Is the hostname empty in the callback URL because it is configured to
> localhost on oozie machine?
>
> For some reason the transition between oozie actions and job status change
> is very very slow.
>
> Regards,
>
>
> --
> Regards,
> Harshal Vora
> www.radiolocus.com
>
>
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.