You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@tez.apache.org by Jonathan Eagles <je...@gmail.com> on 2019/08/01 03:53:58 UTC

Re: Adapter operation failed.Timeline server(AST) is out of reach. Either it's down, or CROS is not enabled

Maria,

I'll try to help. I helped to contribute the Cross Origin Filter to the
Hadoop as well as the contributing the feature that allows the timeline
server to enable the Cross Origin Filter. In my own Tez UI setup there are
some differences, as my Tez UI is based on 0.9.2 and Hadoop stack is based
on 2.8.5. I'm using an older version of hive but not using LLAP in my setup.

A good description of someone else with a similar problem can be found here
https://issues.apache.org/jira/browse/TEZ-3488

You seem to have a good setup, as the history url was able to download the
static javascript Tez UI. For reference I will put some comprehensive
documentation here which will help others if they encounter similar issues.

Setup of the timeline server can be found here
http://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/TimelineServer.html

Setup of the Tez history logging service
http://tez.apache.org/tez-ui.html
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61331897

Specifically something like this is needed.
<property>
   <name>tez.history.logging.service.class</name>
   <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>
</property>

Ok. Now to the issue.

"Either it's down, or CORS is not enabled"

This can mean two things. Either 1) The Tez UI has fetched the from the
timeline server's REST API but was unable to display it since CORS headers
weren't sent along with the request or 2) The Tez UI wasn't able to reach
the timeline server's REST API. Often, this means the Tez UI is configured
to request the data from the wrong port or URL or that the REST API service
isn't running.

Debugging: To debug CORS related issues, I will often open the developer
tools console of my browser (chrome or firefox), reload the Tez UI while
developer tools are open, select the timeline server REST API  (something
that looks like http://localhost:8188/ws/v1/timeline/TEZ_DAG_ID) and look
at the Headers and Response panels on the network tab. This tells me the
address timeline server web address the Tez UI is configured to get data
from as well as see if the CORS headers and data are returned in the
response.
You can copy the URL and try it in your browser to see if it works
correctly as the browser only blocks the data on javascript pages.
[image: Screen Shot 2019-07-31 at 10.45.09 PM.png]

On Wed, Jul 31, 2019 at 7:30 AM Maria <li...@126.com> wrote:

> Hi,all:
>     My env. :   hive-2.3.0+tez-0.8.5+hadoop-2.7.1, and  I want to use hive
> llap in  my project. while it is failed in loading Application
> Dtails/DAGs/Configurations when redirecting url to tez ui  by click
> "Tracking URL ApplicationMaster".
> And the whole error in tez-ui(
> https://xxx.xxx.xxx.xxx:4433/tez-ui2-0.8.5/#/tez-app/application_1564573868628_004) as
> follow:
>
> "Adapter operation failed. Timeline server(AST) is out of reach. Either
> it's down, or CORS is not enabled.
> Stack:
> Error:Adapter operation failed
> at ember$data$lib$adapters$errors$$AdapterError:EmberError(
> https://10.121.70.27:4433/tez-ui2-0.8.5/assets/vendor.js:2486:21)
> at new ember$data$lib$adapters$errors$$AdapterError(
> https://10.121.70.27:4433/tez-ui2-0.8.5/assets/vendor.js:
> <https://10.121.70.27:4433/tez-ui2-0.8.5/assets/vendor.js:2486:21)>
> 80222:50)
> at Class.handleResponse(
> https://10.121.70.27:4433/tez-ui2-0.8.5/assets/vendor.js:
> <https://10.121.70.27:4433/tez-ui2-0.8.5/assets/vendor.js:2486:21)>
> 81517:16)
> ....  ......
> at done(https://10.121.70.27:4433/tez-ui2-0.8.5/assets/vendor.js
> <https://10.121.70.27:4433/tez-ui2-0.8.5/assets/vendor.js:2486:21)>
> :8487:14)
> at XMLHttpRequest.<anonymous>(
> https://10.121.70.27:4433/tez-ui2-0.8.5/assets/vendor.js
> <https://10.121.70.27:4433/tez-ui2-0.8.5/assets/vendor.js:2486:21)>
> :8820:9)
> "
> I have configed "*yarn.timeline-service.http-cross-origin.enabled=true*"
> in yarn-site.xml.
> Reference ideas in
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61331897 doesn't
> seem to help.
>
> Does someone can help? Thanks .
>
>
> Maria.
>
>
>
>
>

Re:Re: Re: Adapter operation failed.Timeline server(AST) is out of reach. Either it's down, or CROS is not enabled

Posted by Maria <li...@126.com>.

Once again, thank you very much for your attention and sincere help. Jonathan.
I do have another problem. What a weird question for me：
I execute "select count(*) from llap_test" in hive2 cli . while the situation like this：

--------------------------------------------------------------------------------------------------------------------------------------------------
VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------------------------------------------------------------------------
Map 1 llap KILLED 1 0 0 1 0 1
Reducer 2 llap KILLED 1 0 0 1 0 0
--------------------------------------------------------------------------------------------------------------------------------------------------
VERTICES: 00/02 [>>-------------------------------------------------------------]0% ELAPSED TIME: 506.91 s

I am sure LLAP daemon and slider-appmaster daemon are normal.
I look into tez am log in tez-ui and didn't even found any Exception. I am sooooooooooo confused where are the Exceptions or error infos about this?

P.S.
My env.

Hive: 2.3.0
hadoop: 2.7.1
tez: 0.9.2
slider: 0.92.0

Maria.

在 2019-08-01 21:24:48，"Jonathan Eagles" <je...@gmail.com> 写道：

I so glad you were able to debug and fix. Keep reaching out to this group for future issues. Can't wait to hear about your success stories and problems so we can help to fix.

Jonathan

On Thu, Aug 1, 2019 at 1:45 AM Maria <li...@126.com> wrote:

Jonathan ,
Thank you sooooo much for your Detailed reply. I resolved the problem, I am so embarrassed ....the real reason in my case is because my tez-ui port is usin the HTTPS protocol, while the timeline server is using the HTTP protocol, communication cannot be established. tez-ui shows well after configuration item about tez ui port in ${TOMCAT_HOME}/conf/server.xml is modified to use unencrypted protocol.

Maria.

在 2019-08-01 10:53:58，"Jonathan Eagles" <je...@gmail.com> 写道：

Maria,

I'll try to help. I helped to contribute the Cross Origin Filter to the Hadoop as well as the contributing the feature that allows the timeline server to enable the Cross Origin Filter. In my own Tez UI setup there are some differences, as my Tez UI is based on 0.9.2 and Hadoop stack is based on 2.8.5. I'm using an older version of hive but not using LLAP in my setup.

A good description of someone else with a similar problem can be found here https://issues.apache.org/jira/browse/TEZ-3488

You seem to have a good setup, as the history url was able to download the static javascript Tez UI. For reference I will put some comprehensive documentation here which will help others if they encounter similar issues.

Setup of the timeline server can be found here
http://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/TimelineServer.html

Setup of the Tez history logging service
http://tez.apache.org/tez-ui.html

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61331897

Specifically something like this is needed.
<property>
<name>tez.history.logging.service.class</name>
<value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>
</property>

Ok. Now to the issue.

"Either it's down, or CORS is not enabled"

This can mean two things. Either 1) The Tez UI has fetched the from the timeline server's REST API but was unable to display it since CORS headers weren't sent along with the request or 2) The Tez UI wasn't able to reach the timeline server's REST API. Often, this means the Tez UI is configured to request the data from the wrong port or URL or that the REST API service isn't running.

Debugging: To debug CORS related issues, I will often open the developer tools console of my browser (chrome or firefox), reload the Tez UI while developer tools are open, select the timeline server REST API (something that looks like http://localhost:8188/ws/v1/timeline/TEZ_DAG_ID) and look at the Headers and Response panels on the network tab. This tells me the address timeline server web address the Tez UI is configured to get data from as well as see if the CORS headers and data are returned in the response.
You can copy the URL and try it in your browser to see if it works correctly as the browser only blocks the data on javascript pages.

On Wed, Jul 31, 2019 at 7:30 AM Maria <li...@126.com> wrote:

Hi,all:
My env. : hive-2.3.0+tez-0.8.5+hadoop-2.7.1, and I want to use hive llap in my project. while it is failed in loading Application Dtails/DAGs/Configurations when redirecting url to tez ui by click "Tracking URL ApplicationMaster".
And the whole error in tez-ui(https://xxx.xxx.xxx.xxx:4433/tez-ui2-0.8.5/#/tez-app/application_1564573868628_004) as follow:

"Adapter operation failed. Timeline server(AST) is out of reach. Either it's down, or CORS is not enabled.
Stack:
Error:Adapter operation failed
at ember$data$lib$adapters$errors$$AdapterError:EmberError(https://10.121.70.27:4433/tez-ui2-0.8.5/assets/vendor.js:2486:21)
at new ember$data$lib$adapters$errors$$AdapterError(https://10.121.70.27:4433/tez-ui2-0.8.5/assets/vendor.js:80222:50)
at Class.handleResponse(https://10.121.70.27:4433/tez-ui2-0.8.5/assets/vendor.js:81517:16)
.... ......
at done(https://10.121.70.27:4433/tez-ui2-0.8.5/assets/vendor.js:8487:14)
at XMLHttpRequest.<anonymous>(https://10.121.70.27:4433/tez-ui2-0.8.5/assets/vendor.js:8820:9)
"
I have configed "yarn.timeline-service.http-cross-origin.enabled=true" in yarn-site.xml.
Reference ideas in https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61331897 doesn't seem to help.

Does someone can help? Thanks .

Maria.

Re: Re: Adapter operation failed.Timeline server(AST) is out of reach. Either it's down, or CROS is not enabled

Posted by Jonathan Eagles <je...@gmail.com>.

I so glad you were able to debug and fix. Keep reaching out to this group
for future issues.  Can't wait to hear about your success stories and
problems so we can help to fix.

Jonathan

On Thu, Aug 1, 2019 at 1:45 AM Maria <li...@126.com> wrote:

> Jonathan ,
> Thank you  sooooo much for your Detailed reply. I resolved the problem, I
> am so embarrassed ....the real reason in my case is because my tez-ui
> port is usin the HTTPS protocol, while the timeline server is using the
> HTTP protocol, communication cannot be established.  tez-ui shows well
> after configuration item about tez ui port in
> ${TOMCAT_HOME}/conf/server.xml is modified to use unencrypted protocol.
>
> Maria.
>
> 在 2019-08-01 10:53:58，"Jonathan Eagles" <je...@gmail.com> 写道：
>
> Maria,
>
> I'll try to help. I helped to contribute the Cross Origin Filter to the
> Hadoop as well as the contributing the feature that allows the timeline
> server to enable the Cross Origin Filter. In my own Tez UI setup there are
> some differences, as my Tez UI is based on 0.9.2 and Hadoop stack is based
> on 2.8.5. I'm using an older version of hive but not using LLAP in my setup.
>
> A good description of someone else with a similar problem can be found
> here https://issues.apache.org/jira/browse/TEZ-3488
>
> You seem to have a good setup, as the history url was able to download the
> static javascript Tez UI. For reference I will put some comprehensive
> documentation here which will help others if they encounter similar issues.
>
> Setup of the timeline server can be found here
>
> http://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/TimelineServer.html
>
> Setup of the Tez history logging service
> http://tez.apache.org/tez-ui.html
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61331897
>
> Specifically something like this is needed.
> <property>
>    <name>tez.history.logging.service.class</name>
>
>    <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>
> </property>
>
> Ok. Now to the issue.
>
> "Either it's down, or CORS is not enabled"
>
> This can mean two things. Either 1) The Tez UI has fetched the from the
> timeline server's REST API but was unable to display it since CORS headers
> weren't sent along with the request or 2) The Tez UI wasn't able to reach
> the timeline server's REST API. Often, this means the Tez UI is configured
> to request the data from the wrong port or URL or that the REST API service
> isn't running.
>
> Debugging: To debug CORS related issues, I will often open the developer
> tools console of my browser (chrome or firefox), reload the Tez UI while
> developer tools are open, select the timeline server REST API  (something
> that looks like http://localhost:8188/ws/v1/timeline/TEZ_DAG_ID) and look
> at the Headers and Response panels on the network tab. This tells me the
> address timeline server web address the Tez UI is configured to get data
> from as well as see if the CORS headers and data are returned in the
> response.
> You can copy the URL and try it in your browser to see if it works
> correctly as the browser only blocks the data on javascript pages.
> [image: Screen Shot 2019-07-31 at 10.45.09 PM.png]
>
> On Wed, Jul 31, 2019 at 7:30 AM Maria <li...@126.com> wrote:
>
>> Hi,all:
>>     My env. :   hive-2.3.0+tez-0.8.5+hadoop-2.7.1, and  I want to use
>> hive llap in  my project. while it is failed in loading Application
>> Dtails/DAGs/Configurations when redirecting url to tez ui  by click
>> "Tracking URL ApplicationMaster".
>> And the whole error in tez-ui(
>> https://xxx.xxx.xxx.xxx:4433/tez-ui2-0.8.5/#/tez-app/application_1564573868628_004) as
>> follow:
>>
>> "Adapter operation failed. Timeline server(AST) is out of reach. Either
>> it's down, or CORS is not enabled.
>> Stack:
>> Error:Adapter operation failed
>> at ember$data$lib$adapters$errors$$AdapterError:EmberError(
>> https://10.121.70.27:4433/tez-ui2-0.8.5/assets/vendor.js:2486:21)
>> at new ember$data$lib$adapters$errors$$AdapterError(
>> https://10.121.70.27:4433/tez-ui2-0.8.5/assets/vendor.js:
>> <https://10.121.70.27:4433/tez-ui2-0.8.5/assets/vendor.js:2486:21)>
>> 80222:50)
>> at Class.handleResponse(
>> https://10.121.70.27:4433/tez-ui2-0.8.5/assets/vendor.js:
>> <https://10.121.70.27:4433/tez-ui2-0.8.5/assets/vendor.js:2486:21)>
>> 81517:16)
>> ....  ......
>> at done(https://10.121.70.27:4433/tez-ui2-0.8.5/assets/vendor.js
>> <https://10.121.70.27:4433/tez-ui2-0.8.5/assets/vendor.js:2486:21)>
>> :8487:14)
>> at XMLHttpRequest.<anonymous>(
>> https://10.121.70.27:4433/tez-ui2-0.8.5/assets/vendor.js
>> <https://10.121.70.27:4433/tez-ui2-0.8.5/assets/vendor.js:2486:21)>
>> :8820:9)
>> "
>> I have configed "*yarn.timeline-service.http-cross-origin.enabled=true*"
>> in yarn-site.xml.
>> Reference ideas in
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61331897
>>  doesn't seem to help.
>>
>> Does someone can help? Thanks .
>>
>>
>> Maria.
>>
>>
>>
>>
>>
>
>
>
>

Re:Re: Adapter operation failed.Timeline server(AST) is out of reach. Either it's down, or CROS is not enabled

Posted by Maria <li...@126.com>.

Maria.

在 2019-08-01 10:53:58，"Jonathan Eagles" <je...@gmail.com> 写道：

Maria,

A good description of someone else with a similar problem can be found here https://issues.apache.org/jira/browse/TEZ-3488

Setup of the timeline server can be found here
http://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/TimelineServer.html

Setup of the Tez history logging service
http://tez.apache.org/tez-ui.html

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61331897

Specifically something like this is needed.
<property>
<name>tez.history.logging.service.class</name>
<value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>
</property>

Ok. Now to the issue.

"Either it's down, or CORS is not enabled"

On Wed, Jul 31, 2019 at 7:30 AM Maria <li...@126.com> wrote:

Does someone can help? Thanks .

Maria.