You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tez.apache.org by Jyotsna G <jy...@gmail.com> on 2018/03/19 11:47:17 UTC
Tez not returning .json results inspite of request completing sucessfully
Hi,
We are making a rest URI request which hits Hive server 2 and translates in
to a hive query on TEZ.
Sample uri:
slc00yeb.us.oracle.com:8989/thirdeye-services/v1.0/OSCS/query/OSCSRecords.json?COMP_FAMILY=FMW&COMP_NAME=WLS&COMPTYPE1_FIRST=RC1&COMPTYPE1_SECOND=RC2&COMPTYPE2_FIRST=manifest.publish.label.WLS_GENERIC:carb-manifest:12.2.1.3.0-170722.1880245:pom&COMPTYPE2_SECOND=manifest.publish.label.WLS_GENERIC:carb-manifest:12.2.1.3.0-170726.1880744:pom&RELEASE_FIRST=12.2.1.3.0&RELEASE_SECOND=12.2.1.3.0&COMPTYPE=MANIFEST
Hadoop version : 2.4.0
Hive:1.2.2
Tez : 5.4.0
All my queries run fine in HIVE CLI , also via standalone program
connecting to hive via JDBC ODBC connection and returns hive query results,
but only when I hit the rest URL, results are not returned.
I have enabled the below settings for tez
tez.session.am.dag.submit.timeout.secs 90
tez.am.mode.session : true
tez.am.container.reuse.enable: true
tez.allow.disabled.timeline-domains : true
tez.yarn.ats.enabled : true
tez.use.cluster.hadoop-libs : true
The url on the browser keeps spinning without the json response, even after
the tez job on yarn cluster has finished successfully. Attached Tez job
diagnosis screenshot.
From the dag log :
2018-03-19 11:34:36,367 INFO [AsyncDispatcher event handler] impl.DAGImpl:
dag_1521448362533_0019_1 transitioned from RUNNING to SUCCEEDED
2018-03-19 11:34:36,368 INFO [AsyncDispatcher event handler]
app.DAGAppMaster: DAG completed, dagId=dag_1521448362533_0019_1,
dagState=SUCCEEDED
{"entity":"dag_1521448362533_0019_1","entitytype":"TEZ_DAG_ID","events":[{"ts":1521459276335,"eventtype":"DAG_FINISHED"}],"otherinfo":{"startTime":1521459269619,"endTime":1521459276335,"timeTaken":6716,"status":"SUCCEEDED","diagnostics":"","counters":{"counterGroups":[{"counterGroupName":"org.apache.tez.common.counters.DAGCounter","counterGroupDisplayName":"org.apache.tez.common.counters.DAGCounter","counters":[{"counterName":"NUM_SUCCEEDED_TASKS","counterDisplayName":"NUM_SUCCEEDED_TASKS","counterValue":2},{"counterName":"TOTAL_LAUNCHED_TASKS","counterDisplayName":"TOTAL_LAUNCHED_TASKS","counterValue":2},{"counterName":"DATA_LOCAL_TASKS","counterDisplayName":"DATA_LOCAL_TASKS","counterValue":1}]},{"counterGroupName":"org.apache.tez.common.counters.FileSystemCounter","counterGroupDisplayName":"File
System
Counters","counters":[{"counterName":"FILE_BYTES_READ","counterDisplayName":"FILE_BYTES_READ","counterValue":41993},{"counterName":"FILE_BYTES_WRITTEN","counterDisplayName":"FILE_BYTES_WRITTEN","counterValue":83954},{"counterName":"FILE_READ_OPS","counterDisplayName":"FILE_READ_OPS","counterValue":0},{"counterName":"FILE_LARGE_READ_OPS","counterDisplayName":"FILE_LARGE_READ_OPS","counterValue":0},{"counterName":"FILE_WRITE_OPS","counterDisplayName":"FILE_WRITE_OPS","counterValue":0},{"counterName":"HDFS_BYTES_READ","counterDisplayName":"HDFS_BYTES_READ","counterValue":59231},{"counterName":"HDFS_BYTES_WRITTEN","counterDisplayName":"HDFS_BYTES_WRITTEN","counterValue":39631},{"counterName":"HDFS_READ_OPS","counterDisplayName":"HDFS_READ_OPS","counterValue":5},{"counterName":"HDFS_LARGE_READ_OPS","counterDisplayName":"HDFS_LARGE_READ_OPS","counterValue":0},{"counterName":"HDFS_WRITE_OPS","counterDisplayName":"HDFS_WRITE_OPS","counterValue":2}]},{"counterGroupName":"org.apache.tez.common.counters.TaskCounter","counterGroupDisplayName":"org.apache.tez.common.counters.TaskCounter","counters":[{"counterName":"REDUCE_INPUT_GROUPS","counterDisplayName":"REDUCE_INPUT_GROUPS","counterValue":190},{"counterName":"REDUCE_INPUT_RECORDS","counterDisplayName":"REDUCE_INPUT_RECORDS","counterValue":191},{"counterName":"COMBINE_INPUT_RECORDS","counterDisplayName":"COMBINE_INPUT_RECORDS","counterValue":0},{"counterName":"SPILLED_RECORDS","counterDisplayName":"SPILLED_RECORDS","counterValue":382},{"counterName":"NUM_SHUFFLED_INPUTS","counterDisplayName":"NUM_SHUFFLED_INPUTS","counterValue":1},{"counterName":"NUM_SKIPPED_INPUTS","counterDisplayName":"NUM_SKIPPED_INPUTS","counterValue":0},{"counterName":"NUM_FAILED_SHUFFLE_INPUTS","counterDisplayName":"NUM_FAILED_SHUFFLE_INPUTS","counterValue":0},{"counterName":"MERGED_MAP_OUTPUTS","counterDisplayName":"MERGED_MAP_OUTPUTS","counterValue":1},{"counterName":"GC_TIME_MILLIS","counterDisplayName":"GC_TIME_MILLIS","counterValue":84},{"counterName":"CPU_MILLISECONDS","counterDisplayName":"CPU_MILLISECONDS","counterValue":5520},{"counterName":"PHYSICAL_MEMORY_BYTES","counterDisplayName":"PHYSICAL_MEMORY_BYTES","counterValue":661254144},{"counterName":"VIRTUAL_MEMORY_BYTES","counterDisplayName":"VIRTUAL_MEMORY_BYTES","counterValue":4294270976},{"counterName":"COMMITTED_HEAP_BYTES","counterDisplayName":"COMMITTED_HEAP_BYTES","counterValue":402653184},{"counterName":"INPUT_RECORDS_PROCESSED","counterDisplayName":"INPUT_RECORDS_PROCESSED","counterValue":8445},{"counterName":"OUTPUT_RECORDS","counterDisplayName":"OUTPUT_RECORDS","counterValue":191},{"counterName":"OUTPUT_BYTES","counterDisplayName":"OUTPUT_BYTES","counterValue":41350},{"counterName":"OUTPUT_BYTES_WITH_OVERHEAD","counterDisplayName":"OUTPUT_BYTES_WITH_OVERHEAD","counterValue":41957},{"counterName":"OUTPUT_BYTES_PHYSICAL","counterDisplayName":"OUTPUT_BYTES_PHYSICAL","counterValue":41961},{"counterName":"ADDITIONAL_SPILLS_BYTES_WRITTEN","counterDisplayName":"ADDITIONAL_SPILLS_BYTES_WRITTEN","counterValue":41961},{"counterName":"ADDITIONAL_SPILLS_BYTES_READ","counterDisplayName":"ADDITIONAL_SPILLS_BYTES_READ","counterValue":41961},{"counterName":"ADDITIONAL_SPILL_COUNT","counterDisplayName":"ADDITIONAL_SPILL_COUNT","counterValue":0},{"counterName":"SHUFFLE_BYTES","counterDisplayName":"SHUFFLE_BYTES","counterValue":41961},{"counterName":"SHUFFLE_BYTES_DECOMPRESSED","counterDisplayName":"SHUFFLE_BYTES_DECOMPRESSED","counterValue":41957},{"counterName":"SHUFFLE_BYTES_TO_MEM","counterDisplayName":"SHUFFLE_BYTES_TO_MEM","counterValue":41961},{"counterName":"SHUFFLE_BYTES_TO_DISK","counterDisplayName":"SHUFFLE_BYTES_TO_DISK","counterValue":0},{"counterName":"SHUFFLE_BYTES_DISK_DIRECT","counterDisplayName":"SHUFFLE_BYTES_DISK_DIRECT","counterValue":0},{"counterName":"NUM_MEM_TO_DISK_MERGES","counterDisplayName":"NUM_MEM_TO_DISK_MERGES","counterValue":0},{"counterName":"NUM_DISK_TO_DISK_MERGES","counterDisplayName":"NUM_DISK_TO_DISK_MERGES","counterValue":0}]},{"counterGroupName":"HIVE","counterGroupDisplayName":"HIVE","counters":[{"counterName":"CREATED_FILES","counterDisplayName":"CREATED_FILES","counterValue":1},{"counterName":"DESERIALIZE_ERRORS","counterDisplayName":"DESERIALIZE_ERRORS","counterValue":0},{"counterName":"RECORDS_IN_Map_1","counterDisplayName":"RECORDS_I
But the json response isnt coming. What am I missing here?
Thanks,
Jyotsna
Re: Tez not returning .json results inspite of request completing sucessfully
Posted by Jonathan Eagles <je...@gmail.com>.
Thank you for the update, Jyotsna. You have taught me a lot regarding
HiveServer2 by reporting this issue.
Regards,
jeagles
On Tue, Mar 20, 2018 at 3:33 AM, Jyotsna G <jy...@gmail.com> wrote:
> Fixed the issue by making the following config changes :
>
> tez.am.container.idle.release-timeout-min.millis=30000
>
> tez.am.container.idle.release-timeout-max.millis=90000
>
> hive.prewarm.enabled = true
>
> hive.prewarm.numcontainers=2
> hive.server2.tez.initialize.default.sessions=true
>
> Thank you so much! great relief!
>
> On Tue, Mar 20, 2018 at 12:45 PM, Jyotsna G <jy...@gmail.com> wrote:
>
>> Thank you so much Jonathan for your response. I could point my finger at
>> the issue a bit more in detail now. Its with the way connection is obtained
>> to Hive (on Tez) and closing it. I need to tweak my Tez configurations with
>> respect to sessions.
>>
>> When executed with a single connection, it works perfectly fine and
>> returns the Json object to browser.
>>
>> *SINGLE CONNECTION*
>>
>>
>>
>> private static String driverName = "org.apache.hive.jdbc.HiveDriver";
>>
>> private static String CONNECTION_URL= "jdbc:hive2://slc06ffs.us.orac
>> le.com:10000";
>>
>>
>>
>> try {
>>
>> Class.forName(driverName);
>>
>> } catch (ClassNotFoundException e) {
>>
>> e.printStackTrace();
>>
>> System.exit(1);
>>
>> }
>>
>> Connection con = DriverManager.getConnection(CONNECTION_URL, "", "");
>>
>> Statement stmt = con.createStatement();
>>
>> //EXECUTE ALL YOUR STATEMENTS HERE
>>
>> con.close();
>>
>>
>>
>>
>>
>>
>>
>> The issue is with when they use pool of connections as below. Once
>> connection is obtained and query executed, results are not shown back on
>> browser and also hive server 2 refuses any further connections after that
>> single hung request (for which tez dag executes successfully)
>>
>>
>>
>> *POOL *
>>
>>
>>
>> And using the pool is like so:
>>
>>
>>
>> Connection con = HiveDatasource.getConnectionFromPool();
>>
>> Statement stmt = con.createStatement();
>>
>> // EXECUTE ALL YOUR QUERIES HERE
>>
>> con.close();
>>
>>
>> Hive server 2 log when using a connection pool on closing the
>> connection(Attached the entire log of debug mode):
>>
>> *18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO
>> thrift.ThriftCLIService: Session disconnected without closing properly,
>> close it now*
>> 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG
>> session.SessionState: Removing resource dir /tmp/38896527-5822-4453-9918-a
>> 5f36db09d1a_resources
>> *18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO
>> tez.TezSessionPoolManager: Closing tez session default? false*
>> *18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO
>> tez.TezSessionState: Closing Tez Session*
>> *18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO
>> client.TezClient: Shutting down Tez Session,
>> sessionName=HIVE-12d75efd-ec5d-4e9a-bfc7-3e3dbb48ee6b,
>> applicationId=application_1521448362533_0101*
>> 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG
>> ipc.Client: The ping interval is 60000 ms.
>> 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG
>> ipc.Client: Connecting to slc03qzg/10.240.178.10:9032
>> 18/03/19 23:48:33 [IPC Client (368802354) connection to slc03qzg/
>> 10.240.178.10:9032 from thirdeye]: DEBUG ipc.Client: IPC Client
>> (368802354) connection to slc03qzg/10.240.178.10:9032 from thirdeye:
>> starting, having connections 1
>> 18/03/19 23:48:33 [IPC Parameter Sending Thread #2]: DEBUG ipc.Client:
>> IPC Client (368802354) connection to slc03qzg/10.240.178.10:9032 from
>> thirdeye sending #1811
>> 18/03/19 23:48:33 [IPC Client (368802354) connection to slc03qzg/
>> 10.240.178.10:9032 from thirdeye]: DEBUG ipc.Client: IPC Client
>> (368802354) connection to slc03qzg/10.240.178.10:9032 from thirdeye got
>> value #1811
>> 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG
>> ipc.ProtobufRpcEngine: Call: getApplicationReport took 5ms
>> 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO
>> client.TezClientUtils: Application not running,
>> applicationId=application_1521448362533_0101,
>> yarnApplicationState=FINISHED, finalApplicationStatus=SUCCEEDED,
>> trackingUrl=slc03qzg:9088/proxy/application_1521448362533_0101/A,
>> diagnostics=Session stats:submittedDAGs=2, successfulDAGs=2, failedDAGs=0,
>> killedDAGs=0
>>
>> *18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO
>> client.TezClient: Failed to shutdown Tez Session via proxy*
>> *org.apache.tez.dag.api.SessionNotRunning: Application not running,
>> applicationId=application_1521448362533_0101,
>> yarnApplicationState=FINISHED, finalApplicationStatus=SUCCEEDED,
>> trackingUrl=slc03qzg:9088/proxy/application_1521448362533_0101/A,
>> diagnostics=Session stats:submittedDAGs=2, successfulDAGs=2, failedDAGs=0,
>> killedDAGs=0*
>>
>> at org.apache.tez.client.TezClientUtils.getSessionAMProxy(TezCl
>> ientUtils.java:798)
>> at org.apache.tez.client.TezClient.getSessionAMProxy(TezClient.java:633)
>> at org.apache.tez.client.TezClient.stop(TezClient.java:440)
>> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.close(
>> TezSessionState.java:257)
>> at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.
>> close(TezSessionPoolManager.java:185)
>> at org.apache.hadoop.hive.ql.session.SessionState.close(Session
>> State.java:1482)
>> at org.apache.hive.service.cli.session.HiveSessionImpl.close(Hi
>> veSessionImpl.java:567)
>> at org.apache.hive.service.cli.session.SessionManager.closeSess
>> ion(SessionManager.java:292)
>> at org.apache.hive.service.cli.CLIService.closeSession(CLIServi
>> ce.java:237)
>> at org.apache.hive.service.cli.thrift.ThriftCLIService$1.delete
>> Context(ThriftCLIService.java:122)
>> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.
>> run(TThreadPoolServer.java:300)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1142)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO
>> client.TezClient: Could not connect to AM, killing session via YARN,
>> sessionName=HIVE-12d75efd-ec5d-4e9a-bfc7-3e3dbb48ee6b,
>> applicationId=application_1521448362533_0101
>> 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO
>> impl.YarnClientImpl: Killing application application_1521448362533_0101
>> 18/03/19 23:48:33 [IPC Parameter Sending Thread #2]: DEBUG ipc.Client:
>> IPC Client (368802354) connection to slc03qzg/10.240.178.10:9032 from
>> thirdeye sending #1812
>> 18/03/19 23:48:33 [IPC Client (368802354) connection to slc03qzg/
>> 10.240.178.10:9032 from thirdeye]: DEBUG ipc.Client: IPC Client
>> (368802354) connection to slc03qzg/10.240.178.10:9032 from thirdeye got
>> value #1812
>> 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG
>> ipc.ProtobufRpcEngine: Call: forceKillApplication took 2ms
>> 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG
>> service.AbstractService: Service: org.apache.hadoop.yarn.client.api.impl.YarnClientImpl
>> entered state STOPPED
>> 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG
>> ipc.Client: The ping interval is 60000 ms.
>> 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG
>> ipc.Client: Connecting to slc06ffs/10.245.175.89:55310
>> 18/03/19 23:48:33 [IPC Client (368802354) connection to slc06ffs/
>> 10.245.175.89:55310 from thirdeye]: DEBUG ipc.Client: IPC Client
>> (368802354) connection to slc06ffs/10.245.175.89:55310 from thirdeye:
>> starting, having connections 2
>> 18/03/19 23:48:33 [IPC Parameter Sending Thread #2]: DEBUG ipc.Client:
>> IPC Client (368802354) connection to slc06ffs/10.245.175.89:55310 from
>> thirdeye sending #1813
>> 18/03/19 23:48:33 [IPC Client (368802354) connection to slc06ffs/
>> 10.245.175.89:55310 from thirdeye]: DEBUG ipc.Client: IPC Client
>> (368802354) connection to slc06ffs/10.245.175.89:55310 from thirdeye got
>> value #1813
>> 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG
>> ipc.ProtobufRpcEngine: Call: delete took 2ms
>> 18/03/19 23:48:33 [IPC Parameter Sending Thread #2]: DEBUG ipc.Client:
>> IPC Client (368802354) connection to slc06ffs/10.245.175.89:55310 from
>> thirdeye sending #1814
>> 18/03/19 23:48:33 [IPC Client (368802354) connection to slc06ffs/
>> 10.245.175.89:55310 from thirdeye]: DEBUG ipc.Client: IPC Client
>> (368802354) connection to slc06ffs/10.245.175.89:55310 from thirdeye got
>> value #1814
>> 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG
>> ipc.ProtobufRpcEngine: Call: delete took 0ms
>> 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG
>> cli.CLIService: SessionHandle [38896527-5822-4453-9918-a5f36db09d1a]:
>> closeSession()
>> 18/03/19 23:48:43 [IPC Client (368802354) connection to slc03qzg/
>> 10.240.178.10:9032 from thirdeye]: DEBUG ipc.Client: IPC Client
>> (368802354) connection to slc03qzg/10.240.178.10:9032 from thirdeye:
>> closed
>> 18/03/19 23:48:43 [IPC Client (368802354) connection to slc03qzg/
>> 10.240.178.10:9032 from thirdeye]: DEBUG ipc.Client: IPC Client
>> (368802354) connection to slc03qzg/10.240.178.10:9032 from thirdeye:
>> stopped, remaining connections 1
>> 18/03/19 23:48:43 [IPC Client (368802354) connection to slc06ffs/
>> 10.245.175.89:55310 from thirdeye]: DEBUG ipc.Client: IPC Client
>> (368802354) connection to slc06ffs/10.245.175.89:55310 from thirdeye:
>> closed
>> 18/03/19 23:48:43 [IPC Client (368802354) connection to slc06ffs/
>> 10.245.175.89:55310 from thirdeye]: DEBUG ipc.Client: IPC Client
>> (368802354) connection to slc06ffs/10.245.175.89:55310 from thirdeye:
>> stopped, remaining connections 0
>>
>>
>>
>> On Mon, Mar 19, 2018 at 10:22 PM, Jonathan Eagles <je...@gmail.com>
>> wrote:
>>
>>> I'm glad you checked this with an MR run. That help clarify the issue,
>>> ruling out the browser compatibility issue. As Tez is an application
>>> framework, it is not responsible for returning results to the browser.
>>> Using Hive CLI, it is the Hive CLI that is responsible for displaying the
>>> results of the query. For HiveServer2, it is HiveServer2's responsibility
>>> of displaying the results. It is true that the yarn and tez configurations
>>> for Hive CLI and HiveServer2 can be different, but it is not the
>>> responsibility of those settings to change how Tez returns results, as it
>>> is Hive CLI and HiveServer2 that returns results.
>>>
>>> Do you have access to the support logs or to a support team that can
>>> retrieve the relevant parts of those logs that will help to investigate
>>> this further? The relevant Tez logs you have provided are helpful, but
>>> only to show that Tez is working correctly. Without having access to
>>> the the HiveServer2 or thirdeye-services logs, I'm afraid it will be very
>>> difficult to debug this issue further.
>>>
>>> On Mon, Mar 19, 2018 at 11:22 AM, Jyotsna G <jy...@gmail.com> wrote:
>>>
>>>> Thank you for your response Jonathan.
>>>>
>>>> When I change the execution engine to MR, results are fetched perfectly
>>>> for the rest query, it's only when I change the execution engine to tez
>>>> that I see the issue occurring.
>>>>
>>>> To rephrase my question, are there any further sessions specific
>>>> configurations for tez that I might have missed? Any yarn related
>>>> configurations for tez that I might have to look for?
>>>>
>>>>
>>>>
>>>> On Mon, Mar 19, 2018, 21:37 Jonathan Eagles <je...@gmail.com> wrote:
>>>>
>>>>> Thanks for reaching out to us, Jyotsna.
>>>>>
>>>>> Let me make sure I understand what your setup and the issue you are
>>>>> experiencing. It sounds like the setup looks like this.
>>>>>
>>>>> Browser -> Third party service "thirdeye-services" -> HiveServer2 ->
>>>>> Tez
>>>>>
>>>>> Since you are showing logs and screenshots of the successful Tez Job.
>>>>> My initial thought is that HiveServer2 was able to successfully and
>>>>> correctly get the results from Tez. That would make me believe that either
>>>>> the connection between the browser to the third party service
>>>>> "thirdeye-services" isn't working (perhaps browser compatibility issue,
>>>>> security issue, or service issue) or the connection between the third party
>>>>> service "thirdeye-services" to HiveServer2 issue (issue retrieving results
>>>>> from HiveServer2).
>>>>>
>>>>> Since our expertise is only with the Tez portion of the setup and that
>>>>> the Tez portion of the setup seems to be working, the Tez community won't
>>>>> be the best at addressing this issue. My recommendation would be try the
>>>>> query in a different browser as a simple test (Chrome, Firefox) checking
>>>>> with third party "thirdeye-services" browser compatibility. If you have
>>>>> access to the thirdparty-services logs or logs to the HiveServer2 logs
>>>>> those will be helpful in debugging. When reaching out to the
>>>>> thirdeye-services support or HiveServer2 please reference these logs to get
>>>>> the best support from the community. The hive user list is
>>>>> user@hive.apache.org and I have don't have a support contact for the
>>>>> thirdparty software thirdeye-services that seems to be part of this setup.
>>>>>
>>>>> Thanks again for reaching to the Tez Community.
>>>>>
>>>>> Regards,
>>>>> jeagles
>>>>>
>>>>> On Mon, Mar 19, 2018 at 6:47 AM, Jyotsna G <jy...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>> We are making a rest URI request which hits Hive server 2 and
>>>>>> translates in to a hive query on TEZ.
>>>>>>
>>>>>> Sample uri:
>>>>>> slc00yeb.us.oracle.com:8989/thirdeye-services/v1.0/OSCS/quer
>>>>>> y/OSCSRecords.json?COMP_FAMILY=FMW&COMP_NAME=WLS&COMPTYPE1_F
>>>>>> IRST=RC1&COMPTYPE1_SECOND=RC2&COMPTYPE2_FIRST=manifest.publi
>>>>>> sh.label.WLS_GENERIC:carb-manifest:12.2.1.3.0-170722.1880245
>>>>>> :pom&COMPTYPE2_SECOND=manifest.publish.label.WLS_GENERIC:
>>>>>> carb-manifest:12.2.1.3.0-170726.1880744:pom&RELEASE_
>>>>>> FIRST=12.2.1.3.0&RELEASE_SECOND=12.2.1.3.0&COMPTYPE=MANIFEST
>>>>>>
>>>>>> Hadoop version : 2.4.0
>>>>>> Hive:1.2.2
>>>>>> Tez : 5.4.0
>>>>>>
>>>>>> All my queries run fine in HIVE CLI , also via standalone program
>>>>>> connecting to hive via JDBC ODBC connection and returns hive query results,
>>>>>> but only when I hit the rest URL, results are not returned.
>>>>>>
>>>>>> I have enabled the below settings for tez
>>>>>>
>>>>>> tez.session.am.dag.submit.timeout.secs 90
>>>>>>
>>>>>> tez.am.mode.session : true
>>>>>> tez.am.container.reuse.enable: true
>>>>>> tez.allow.disabled.timeline-domains : true
>>>>>> tez.yarn.ats.enabled : true
>>>>>> tez.use.cluster.hadoop-libs : true
>>>>>>
>>>>>> The url on the browser keeps spinning without the json response, even
>>>>>> after the tez job on yarn cluster has finished successfully. Attached Tez
>>>>>> job diagnosis screenshot.
>>>>>>
>>>>>> From the dag log :
>>>>>>
>>>>>> 2018-03-19 11:34:36,367 INFO [AsyncDispatcher event handler]
>>>>>> impl.DAGImpl: dag_1521448362533_0019_1 transitioned from RUNNING to
>>>>>> SUCCEEDED
>>>>>> 2018-03-19 11:34:36,368 INFO [AsyncDispatcher event handler]
>>>>>> app.DAGAppMaster: DAG completed, dagId=dag_1521448362533_0019_1,
>>>>>> dagState=SUCCEEDED
>>>>>>
>>>>>> {"entity":"dag_1521448362533_0019_1","entitytype":"TEZ_DAG_I
>>>>>> D","events":[{"ts":1521459276335,"eventtype":"DAG_FINISHED"}
>>>>>> ],"otherinfo":{"startTime":1521459269619,"endTime":152145927
>>>>>> 6335,"timeTaken":6716,"status":"SUCCEEDED","diagnostics":"",
>>>>>> "counters":{"counterGroups":[{"counterGroupName":"org.apache
>>>>>> .tez.common.counters.DAGCounter","counterGroupDisplayName":"
>>>>>> org.apache.tez.common.counters.DAGCounter","counters
>>>>>> ":[{"counterName":"NUM_SUCCEEDED_TASKS","counterDispl
>>>>>> ayName":"NUM_SUCCEEDED_TASKS","counterValue":2},{"counterNam
>>>>>> e":"TOTAL_LAUNCHED_TASKS","counterDisplayName":"TOTAL_
>>>>>> LAUNCHED_TASKS","counterValue":2},{"counterName":"DATA_LOCAL_TASKS","
>>>>>> counterDisplayName":"DATA_LOCAL_TASKS","counterValue":1}
>>>>>> ]},{"counterGroupName":"org.apache.tez.common.counters.
>>>>>> FileSystemCounter","counterGroupDisplayName":"File System
>>>>>> Counters","counters":[{"counterName":"FILE_BYTES_READ","coun
>>>>>> terDisplayName":"FILE_BYTES_READ","counterValue":41993},{"co
>>>>>> unterName":"FILE_BYTES_WRITTEN","counterDisplayName":"FILE_
>>>>>> BYTES_WRITTEN","counterValue":83954},{"counterName":"FILE_
>>>>>> READ_OPS","counterDisplayName":"FILE_READ_OPS","counterValue
>>>>>> ":0},{"counterName":"FILE_LARGE_READ_OPS","counterDispla
>>>>>> yName":"FILE_LARGE_READ_OPS","counterValue":0},{"
>>>>>> counterName":"FILE_WRITE_OPS","counterDisplayName":"FILE_
>>>>>> WRITE_OPS","counterValue":0},{"counterName":"HDFS_BYTES_
>>>>>> READ","counterDisplayName":"HDFS_BYTES_READ","counterValue":59231},{"
>>>>>> counterName":"HDFS_BYTES_WRITTEN","counterDisplayName":"
>>>>>> HDFS_BYTES_WRITTEN","counterValue":39631},{"counterName":"
>>>>>> HDFS_READ_OPS","counterDisplayName":"HDFS_READ_OPS","
>>>>>> counterValue":5},{"counterName":"HDFS_LARGE_READ_OPS","
>>>>>> counterDisplayName":"HDFS_LARGE_READ_OPS","counterValue"
>>>>>> :0},{"counterName":"HDFS_WRITE_OPS","counterDisplayName":"HDFS_WRIT
>>>>>> E_OPS","counterValue":2}]},{"counterGroupName":"org.apache.
>>>>>> tez.common.counters.TaskCounter","counterGroupDisplayName":"
>>>>>> org.apache.tez.common.counters.TaskCounter","
>>>>>> counters":[{"counterName":"REDUCE_INPUT_GROUPS","
>>>>>> counterDisplayName":"REDUCE_INPUT_GROUPS","counterValue":
>>>>>> 190},{"counterName":"REDUCE_INPUT_RECORDS","
>>>>>> counterDisplayName":"REDUCE_INPUT_RECORDS","counterValue":
>>>>>> 191},{"counterName":"COMBINE_INPUT_RECORDS","
>>>>>> counterDisplayName":"COMBINE_INPUT_RECORDS","counterValue":
>>>>>> 0},{"counterName":"SPILLED_RECORDS","counterDisplayName":
>>>>>> "SPILLED_RECORDS","counterValue":382},{"counterName":"NUM_SHUFFLED_
>>>>>> INPUTS","counterDisplayName":"NUM_SHUFFLED_INPUTS","counterV
>>>>>> alue":1},{"counterName":"NUM_SKIPPED_INPUTS","
>>>>>> counterDisplayName":"NUM_SKIPPED_INPUTS","counterValue"
>>>>>> :0},{"counterName":"NUM_FAILED_SHUFFLE_INPUTS","
>>>>>> counterDisplayName":"NUM_FAILED_SHUFFLE_INPUTS","
>>>>>> counterValue":0},{"counterName":"MERGED_MAP_OUTPUTS","
>>>>>> counterDisplayName":"MERGED_MAP_OUTPUTS","counterValue":1}
>>>>>> ,{"counterName":"GC_TIME_MILLIS","counterDisplayName":"
>>>>>> GC_TIME_MILLIS","counterValue":84},{"counterName":"CPU_
>>>>>> MILLISECONDS","counterDisplayName":"CPU_MILLI
>>>>>> SECONDS","counterValue":5520},{"counterName":"PHYSICAL_
>>>>>> MEMORY_BYTES","counterDisplayName":"PHYSICAL_MEMORY_BYTES","
>>>>>> counterValue":661254144},{"counterName":"VIRTUAL_MEMORY_
>>>>>> BYTES","counterDisplayName":"VIRTUAL_MEMORY_BYTES","
>>>>>> counterValue":4294270976},{"counterName":"COMMITTED_HEAP_
>>>>>> BYTES","counterDisplayName":"COMMITTED_HEAP_BYTES","
>>>>>> counterValue":402653184},{"counterName":"INPUT_RECORDS_
>>>>>> PROCESSED","counterDisplayName":"INPUT_RECORDS_PROCESSED","
>>>>>> counterValue":8445},{"counterName":"OUTPUT_RECORDS",
>>>>>> "counterDisplayName":"OUTPUT_RECORDS","counterValue":191},{
>>>>>> "counterName":"OUTPUT_BYTES","counterDisplayName":"OUTPUT_
>>>>>> BYTES","counterValue":41350},{"counterName":"OUTPUT_BYTES_
>>>>>> WITH_OVERHEAD","counterDisplayName":"OUTPUT_BYTES_WITH_
>>>>>> OVERHEAD","counterValue":41957},{"counterName":"OUTPUT_
>>>>>> BYTES_PHYSICAL","counterDisplayName":"OUTPUT_
>>>>>> BYTES_PHYSICAL","counterValue":41961},{"counterName":"
>>>>>> ADDITIONAL_SPILLS_BYTES_WRITTEN","counterDisplayName":
>>>>>> "ADDITIONAL_SPILLS_BYTES_WRITTEN","counterValue":41961}
>>>>>> ,{"counterName":"ADDITIONAL_SPILLS_BYTES_READ","counterDisplayName":"
>>>>>> ADDITIONAL_SPILLS_BYTES_READ","counterValue":41961},{"counte
>>>>>> rName":"ADDITIONAL_SPILL_COUNT","counterDisplayName":"ADDITI
>>>>>> ONAL_SPILL_COUNT","counterValue":0},{"counterName":"SHUFFLE_
>>>>>> BYTES","counterDisplayName":"SHUFFLE_BYTES","counterValue":
>>>>>> 41961},{"counterName":"SHUFFLE_BYTES_DECOMPRESSED","counterD
>>>>>> isplayName":"SHUFFLE_BYTES_DECOMPRESSED","counterValue":
>>>>>> 41957},{"counterName":"SHUFFLE_BYTES_TO_MEM","
>>>>>> counterDisplayName":"SHUFFLE_BYTES_TO_MEM","counterValue":
>>>>>> 41961},{"counterName":"SHUFFLE_BYTES_TO_DISK","
>>>>>> counterDisplayName":"SHUFFLE_BYTES_TO_DISK","counterValue":
>>>>>> 0},{"counterName":"SHUFFLE_BYTES_DISK_DIRECT","
>>>>>> counterDisplayName":"SHUFFLE_BYTES_DISK_DIRECT","
>>>>>> counterValue":0},{"counterName":"NUM_MEM_TO_DISK_MERGES","
>>>>>> counterDisplayName":"NUM_MEM_TO_DISK_MERGES","counterValue":
>>>>>> 0},{"counterName":"NUM_DISK_TO_DISK_MERGES","counterDisplayN
>>>>>> ame":"NUM_DISK_TO_DISK_MERGES","counterValue":0}]},{"counter
>>>>>> GroupName":"HIVE","counterGroupDisplayName":"HIVE","counters
>>>>>> ":[{"counterName":"CREATED_FILES","counterDisplayName":"CREA
>>>>>> TED_FILES","counterValue":1},{"counterName":"DESERIALIZE_
>>>>>> ERRORS","counterDisplayName":"DESERIALIZE_ERRORS","
>>>>>> counterValue":0},{"counterName":"RECORDS_IN_Map_1","
>>>>>> counterDisplayName":"RECORDS_I
>>>>>>
>>>>>>
>>>>>> But the json response isnt coming. What am I missing here?
>>>>>>
>>>>>> Thanks,
>>>>>> Jyotsna
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>>
>
Re: Tez not returning .json results inspite of request completing sucessfully
Posted by Jyotsna G <jy...@gmail.com>.
Fixed the issue by making the following config changes :
tez.am.container.idle.release-timeout-min.millis=30000
tez.am.container.idle.release-timeout-max.millis=90000
hive.prewarm.enabled = true
hive.prewarm.numcontainers=2
hive.server2.tez.initialize.default.sessions=true
Thank you so much! great relief!
On Tue, Mar 20, 2018 at 12:45 PM, Jyotsna G <jy...@gmail.com> wrote:
> Thank you so much Jonathan for your response. I could point my finger at
> the issue a bit more in detail now. Its with the way connection is obtained
> to Hive (on Tez) and closing it. I need to tweak my Tez configurations with
> respect to sessions.
>
> When executed with a single connection, it works perfectly fine and
> returns the Json object to browser.
>
> *SINGLE CONNECTION*
>
>
>
> private static String driverName = "org.apache.hive.jdbc.HiveDriver";
>
> private static String CONNECTION_URL= "jdbc:hive2://slc06ffs.us.
> oracle.com:10000";
>
>
>
> try {
>
> Class.forName(driverName);
>
> } catch (ClassNotFoundException e) {
>
> e.printStackTrace();
>
> System.exit(1);
>
> }
>
> Connection con = DriverManager.getConnection(CONNECTION_URL, "", "");
>
> Statement stmt = con.createStatement();
>
> //EXECUTE ALL YOUR STATEMENTS HERE
>
> con.close();
>
>
>
>
>
>
>
> The issue is with when they use pool of connections as below. Once
> connection is obtained and query executed, results are not shown back on
> browser and also hive server 2 refuses any further connections after that
> single hung request (for which tez dag executes successfully)
>
>
>
> *POOL *
>
>
>
> And using the pool is like so:
>
>
>
> Connection con = HiveDatasource.getConnectionFromPool();
>
> Statement stmt = con.createStatement();
>
> // EXECUTE ALL YOUR QUERIES HERE
>
> con.close();
>
>
> Hive server 2 log when using a connection pool on closing the
> connection(Attached the entire log of debug mode):
>
> *18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO
> thrift.ThriftCLIService: Session disconnected without closing properly,
> close it now*
> 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG
> session.SessionState: Removing resource dir /tmp/38896527-5822-4453-9918-
> a5f36db09d1a_resources
> *18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO
> tez.TezSessionPoolManager: Closing tez session default? false*
> *18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO
> tez.TezSessionState: Closing Tez Session*
> *18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO
> client.TezClient: Shutting down Tez Session,
> sessionName=HIVE-12d75efd-ec5d-4e9a-bfc7-3e3dbb48ee6b,
> applicationId=application_1521448362533_0101*
> 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG ipc.Client:
> The ping interval is 60000 ms.
> 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG ipc.Client:
> Connecting to slc03qzg/10.240.178.10:9032
> 18/03/19 23:48:33 [IPC Client (368802354) connection to slc03qzg/
> 10.240.178.10:9032 from thirdeye]: DEBUG ipc.Client: IPC Client
> (368802354) connection to slc03qzg/10.240.178.10:9032 from thirdeye:
> starting, having connections 1
> 18/03/19 23:48:33 [IPC Parameter Sending Thread #2]: DEBUG ipc.Client: IPC
> Client (368802354) connection to slc03qzg/10.240.178.10:9032 from
> thirdeye sending #1811
> 18/03/19 23:48:33 [IPC Client (368802354) connection to slc03qzg/
> 10.240.178.10:9032 from thirdeye]: DEBUG ipc.Client: IPC Client
> (368802354) connection to slc03qzg/10.240.178.10:9032 from thirdeye got
> value #1811
> 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG
> ipc.ProtobufRpcEngine: Call: getApplicationReport took 5ms
> 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO
> client.TezClientUtils: Application not running, applicationId=application_1521448362533_0101,
> yarnApplicationState=FINISHED, finalApplicationStatus=SUCCEEDED,
> trackingUrl=slc03qzg:9088/proxy/application_1521448362533_0101/A,
> diagnostics=Session stats:submittedDAGs=2, successfulDAGs=2, failedDAGs=0,
> killedDAGs=0
>
> *18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO
> client.TezClient: Failed to shutdown Tez Session via proxy*
> *org.apache.tez.dag.api.SessionNotRunning: Application not running,
> applicationId=application_1521448362533_0101,
> yarnApplicationState=FINISHED, finalApplicationStatus=SUCCEEDED,
> trackingUrl=slc03qzg:9088/proxy/application_1521448362533_0101/A,
> diagnostics=Session stats:submittedDAGs=2, successfulDAGs=2, failedDAGs=0,
> killedDAGs=0*
>
> at org.apache.tez.client.TezClientUtils.getSessionAMProxy(
> TezClientUtils.java:798)
> at org.apache.tez.client.TezClient.getSessionAMProxy(TezClient.java:633)
> at org.apache.tez.client.TezClient.stop(TezClient.java:440)
> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.
> close(TezSessionState.java:257)
> at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.close(
> TezSessionPoolManager.java:185)
> at org.apache.hadoop.hive.ql.session.SessionState.close(
> SessionState.java:1482)
> at org.apache.hive.service.cli.session.HiveSessionImpl.close(
> HiveSessionImpl.java:567)
> at org.apache.hive.service.cli.session.SessionManager.
> closeSession(SessionManager.java:292)
> at org.apache.hive.service.cli.CLIService.closeSession(
> CLIService.java:237)
> at org.apache.hive.service.cli.thrift.ThriftCLIService$1.deleteContext(
> ThriftCLIService.java:122)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(
> TThreadPoolServer.java:300)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO
> client.TezClient: Could not connect to AM, killing session via YARN,
> sessionName=HIVE-12d75efd-ec5d-4e9a-bfc7-3e3dbb48ee6b,
> applicationId=application_1521448362533_0101
> 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO
> impl.YarnClientImpl: Killing application application_1521448362533_0101
> 18/03/19 23:48:33 [IPC Parameter Sending Thread #2]: DEBUG ipc.Client: IPC
> Client (368802354) connection to slc03qzg/10.240.178.10:9032 from
> thirdeye sending #1812
> 18/03/19 23:48:33 [IPC Client (368802354) connection to slc03qzg/
> 10.240.178.10:9032 from thirdeye]: DEBUG ipc.Client: IPC Client
> (368802354) connection to slc03qzg/10.240.178.10:9032 from thirdeye got
> value #1812
> 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG
> ipc.ProtobufRpcEngine: Call: forceKillApplication took 2ms
> 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG
> service.AbstractService: Service: org.apache.hadoop.yarn.client.api.impl.YarnClientImpl
> entered state STOPPED
> 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG ipc.Client:
> The ping interval is 60000 ms.
> 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG ipc.Client:
> Connecting to slc06ffs/10.245.175.89:55310
> 18/03/19 23:48:33 [IPC Client (368802354) connection to slc06ffs/
> 10.245.175.89:55310 from thirdeye]: DEBUG ipc.Client: IPC Client
> (368802354) connection to slc06ffs/10.245.175.89:55310 from thirdeye:
> starting, having connections 2
> 18/03/19 23:48:33 [IPC Parameter Sending Thread #2]: DEBUG ipc.Client: IPC
> Client (368802354) connection to slc06ffs/10.245.175.89:55310 from
> thirdeye sending #1813
> 18/03/19 23:48:33 [IPC Client (368802354) connection to slc06ffs/
> 10.245.175.89:55310 from thirdeye]: DEBUG ipc.Client: IPC Client
> (368802354) connection to slc06ffs/10.245.175.89:55310 from thirdeye got
> value #1813
> 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG
> ipc.ProtobufRpcEngine: Call: delete took 2ms
> 18/03/19 23:48:33 [IPC Parameter Sending Thread #2]: DEBUG ipc.Client: IPC
> Client (368802354) connection to slc06ffs/10.245.175.89:55310 from
> thirdeye sending #1814
> 18/03/19 23:48:33 [IPC Client (368802354) connection to slc06ffs/
> 10.245.175.89:55310 from thirdeye]: DEBUG ipc.Client: IPC Client
> (368802354) connection to slc06ffs/10.245.175.89:55310 from thirdeye got
> value #1814
> 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG
> ipc.ProtobufRpcEngine: Call: delete took 0ms
> 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG
> cli.CLIService: SessionHandle [38896527-5822-4453-9918-a5f36db09d1a]:
> closeSession()
> 18/03/19 23:48:43 [IPC Client (368802354) connection to slc03qzg/
> 10.240.178.10:9032 from thirdeye]: DEBUG ipc.Client: IPC Client
> (368802354) connection to slc03qzg/10.240.178.10:9032 from thirdeye:
> closed
> 18/03/19 23:48:43 [IPC Client (368802354) connection to slc03qzg/
> 10.240.178.10:9032 from thirdeye]: DEBUG ipc.Client: IPC Client
> (368802354) connection to slc03qzg/10.240.178.10:9032 from thirdeye:
> stopped, remaining connections 1
> 18/03/19 23:48:43 [IPC Client (368802354) connection to slc06ffs/
> 10.245.175.89:55310 from thirdeye]: DEBUG ipc.Client: IPC Client
> (368802354) connection to slc06ffs/10.245.175.89:55310 from thirdeye:
> closed
> 18/03/19 23:48:43 [IPC Client (368802354) connection to slc06ffs/
> 10.245.175.89:55310 from thirdeye]: DEBUG ipc.Client: IPC Client
> (368802354) connection to slc06ffs/10.245.175.89:55310 from thirdeye:
> stopped, remaining connections 0
>
>
>
> On Mon, Mar 19, 2018 at 10:22 PM, Jonathan Eagles <je...@gmail.com>
> wrote:
>
>> I'm glad you checked this with an MR run. That help clarify the issue,
>> ruling out the browser compatibility issue. As Tez is an application
>> framework, it is not responsible for returning results to the browser.
>> Using Hive CLI, it is the Hive CLI that is responsible for displaying the
>> results of the query. For HiveServer2, it is HiveServer2's responsibility
>> of displaying the results. It is true that the yarn and tez configurations
>> for Hive CLI and HiveServer2 can be different, but it is not the
>> responsibility of those settings to change how Tez returns results, as it
>> is Hive CLI and HiveServer2 that returns results.
>>
>> Do you have access to the support logs or to a support team that can
>> retrieve the relevant parts of those logs that will help to investigate
>> this further? The relevant Tez logs you have provided are helpful, but
>> only to show that Tez is working correctly. Without having access to the
>> the HiveServer2 or thirdeye-services logs, I'm afraid it will be very
>> difficult to debug this issue further.
>>
>> On Mon, Mar 19, 2018 at 11:22 AM, Jyotsna G <jy...@gmail.com> wrote:
>>
>>> Thank you for your response Jonathan.
>>>
>>> When I change the execution engine to MR, results are fetched perfectly
>>> for the rest query, it's only when I change the execution engine to tez
>>> that I see the issue occurring.
>>>
>>> To rephrase my question, are there any further sessions specific
>>> configurations for tez that I might have missed? Any yarn related
>>> configurations for tez that I might have to look for?
>>>
>>>
>>>
>>> On Mon, Mar 19, 2018, 21:37 Jonathan Eagles <je...@gmail.com> wrote:
>>>
>>>> Thanks for reaching out to us, Jyotsna.
>>>>
>>>> Let me make sure I understand what your setup and the issue you are
>>>> experiencing. It sounds like the setup looks like this.
>>>>
>>>> Browser -> Third party service "thirdeye-services" -> HiveServer2 ->
>>>> Tez
>>>>
>>>> Since you are showing logs and screenshots of the successful Tez Job.
>>>> My initial thought is that HiveServer2 was able to successfully and
>>>> correctly get the results from Tez. That would make me believe that either
>>>> the connection between the browser to the third party service
>>>> "thirdeye-services" isn't working (perhaps browser compatibility issue,
>>>> security issue, or service issue) or the connection between the third party
>>>> service "thirdeye-services" to HiveServer2 issue (issue retrieving results
>>>> from HiveServer2).
>>>>
>>>> Since our expertise is only with the Tez portion of the setup and that
>>>> the Tez portion of the setup seems to be working, the Tez community won't
>>>> be the best at addressing this issue. My recommendation would be try the
>>>> query in a different browser as a simple test (Chrome, Firefox) checking
>>>> with third party "thirdeye-services" browser compatibility. If you have
>>>> access to the thirdparty-services logs or logs to the HiveServer2 logs
>>>> those will be helpful in debugging. When reaching out to the
>>>> thirdeye-services support or HiveServer2 please reference these logs to get
>>>> the best support from the community. The hive user list is
>>>> user@hive.apache.org and I have don't have a support contact for the
>>>> thirdparty software thirdeye-services that seems to be part of this setup.
>>>>
>>>> Thanks again for reaching to the Tez Community.
>>>>
>>>> Regards,
>>>> jeagles
>>>>
>>>> On Mon, Mar 19, 2018 at 6:47 AM, Jyotsna G <jy...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>> We are making a rest URI request which hits Hive server 2 and
>>>>> translates in to a hive query on TEZ.
>>>>>
>>>>> Sample uri:
>>>>> slc00yeb.us.oracle.com:8989/thirdeye-services/v1.0/OSCS/quer
>>>>> y/OSCSRecords.json?COMP_FAMILY=FMW&COMP_NAME=WLS&COMPTYPE1_F
>>>>> IRST=RC1&COMPTYPE1_SECOND=RC2&COMPTYPE2_FIRST=manifest.publi
>>>>> sh.label.WLS_GENERIC:carb-manifest:12.2.1.3.0-170722.
>>>>> 1880245:pom&COMPTYPE2_SECOND=manifest.publish.label.WLS_
>>>>> GENERIC:carb-manifest:12.2.1.3.0-170726.1880744:pom&RELEASE
>>>>> _FIRST=12.2.1.3.0&RELEASE_SECOND=12.2.1.3.0&COMPTYPE=MANIFEST
>>>>>
>>>>> Hadoop version : 2.4.0
>>>>> Hive:1.2.2
>>>>> Tez : 5.4.0
>>>>>
>>>>> All my queries run fine in HIVE CLI , also via standalone program
>>>>> connecting to hive via JDBC ODBC connection and returns hive query results,
>>>>> but only when I hit the rest URL, results are not returned.
>>>>>
>>>>> I have enabled the below settings for tez
>>>>>
>>>>> tez.session.am.dag.submit.timeout.secs 90
>>>>>
>>>>> tez.am.mode.session : true
>>>>> tez.am.container.reuse.enable: true
>>>>> tez.allow.disabled.timeline-domains : true
>>>>> tez.yarn.ats.enabled : true
>>>>> tez.use.cluster.hadoop-libs : true
>>>>>
>>>>> The url on the browser keeps spinning without the json response, even
>>>>> after the tez job on yarn cluster has finished successfully. Attached Tez
>>>>> job diagnosis screenshot.
>>>>>
>>>>> From the dag log :
>>>>>
>>>>> 2018-03-19 11:34:36,367 INFO [AsyncDispatcher event handler]
>>>>> impl.DAGImpl: dag_1521448362533_0019_1 transitioned from RUNNING to
>>>>> SUCCEEDED
>>>>> 2018-03-19 11:34:36,368 INFO [AsyncDispatcher event handler]
>>>>> app.DAGAppMaster: DAG completed, dagId=dag_1521448362533_0019_1,
>>>>> dagState=SUCCEEDED
>>>>>
>>>>> {"entity":"dag_1521448362533_0019_1","entitytype":"TEZ_DAG_I
>>>>> D","events":[{"ts":1521459276335,"eventtype":"DAG_FINISHED"}
>>>>> ],"otherinfo":{"startTime":1521459269619,"endTime":152145927
>>>>> 6335,"timeTaken":6716,"status":"SUCCEEDED","diagnostics":"",
>>>>> "counters":{"counterGroups":[{"counterGroupName":"org.
>>>>> apache.tez.common.counters.DAGCounter","counterGroupDispl
>>>>> ayName":"org.apache.tez.common.counters.DAGCounter","
>>>>> counters":[{"counterName":"NUM_SUCCEEDED_TASKS","counterDisp
>>>>> layName":"NUM_SUCCEEDED_TASKS","counterValue":2},{"
>>>>> counterName":"TOTAL_LAUNCHED_TASKS","counterDisplayName":"
>>>>> TOTAL_LAUNCHED_TASKS","counterValue":2},{"counterName":"DATA_LOCAL_
>>>>> TASKS","counterDisplayName":"DATA_LOCAL_TASKS","counterValue":1}]},{"
>>>>> counterGroupName":"org.apache.tez.common.counters.FileSystem
>>>>> Counter","counterGroupDisplayName":"File System
>>>>> Counters","counters":[{"counterName":"FILE_BYTES_READ","coun
>>>>> terDisplayName":"FILE_BYTES_READ","counterValue":41993},{"
>>>>> counterName":"FILE_BYTES_WRITTEN","counterDisplayName":"
>>>>> FILE_BYTES_WRITTEN","counterValue":83954},{"counterName":"
>>>>> FILE_READ_OPS","counterDisplayName":"FILE_READ_OPS","
>>>>> counterValue":0},{"counterName":"FILE_LARGE_READ_OPS","
>>>>> counterDisplayName":"FILE_LARGE_READ_OPS","counterValue"
>>>>> :0},{"counterName":"FILE_WRITE_OPS","counterDisplayName":"
>>>>> FILE_WRITE_OPS","counterValue":0},{"counterName":"HDFS_BYTES_READ","
>>>>> counterDisplayName":"HDFS_BYTES_READ","counterValue":59231},
>>>>> {"counterName":"HDFS_BYTES_WRITTEN","counterDisplayName":
>>>>> "HDFS_BYTES_WRITTEN","counterValue":39631},{"
>>>>> counterName":"HDFS_READ_OPS","counterDisplayName":"HDFS_
>>>>> READ_OPS","counterValue":5},{"counterName":"HDFS_LARGE_READ_
>>>>> OPS","counterDisplayName":"HDFS_LARGE_READ_OPS","counterValue":0},{"
>>>>> counterName":"HDFS_WRITE_OPS","counterDisplayName":"HDFS_
>>>>> WRITE_OPS","counterValue":2}]},{"counterGroupName":"org.
>>>>> apache.tez.common.counters.TaskCounter","
>>>>> counterGroupDisplayName":"org.apache.tez.common.counters.
>>>>> TaskCounter","counters":[{"counterName":"REDUCE_INPUT_
>>>>> GROUPS","counterDisplayName":"REDUCE_INPUT_GROUPS","
>>>>> counterValue":190},{"counterName":"REDUCE_INPUT_
>>>>> RECORDS","counterDisplayName":"REDUCE_INPUT_RECORDS","
>>>>> counterValue":191},{"counterName":"COMBINE_INPUT_
>>>>> RECORDS","counterDisplayName":"COMBINE_INPUT_RECORDS","
>>>>> counterValue":0},{"counterName":"SPILLED_RECORDS"
>>>>> ,"counterDisplayName":"SPILLED_RECORDS","counterValue
>>>>> ":382},{"counterName":"NUM_SHUFFLED_INPUTS","counterDisplayN
>>>>> ame":"NUM_SHUFFLED_INPUTS","counterValue":1},{"
>>>>> counterName":"NUM_SKIPPED_INPUTS","counterDisplayName":"
>>>>> NUM_SKIPPED_INPUTS","counterValue":0},{"counterName":"NUM_FAILED_
>>>>> SHUFFLE_INPUTS","counterDisplayName":"NUM_FAILED_SHUFFLE_
>>>>> INPUTS","counterValue":0},{"counterName":"MERGED_MAP_
>>>>> OUTPUTS","counterDisplayName":"MERGED_MAP_OUTPUTS","
>>>>> counterValue":1},{"counterName":"GC_TIME_MILLIS",
>>>>> "counterDisplayName":"GC_TIME_MILLIS","counterValue":84},{"
>>>>> counterName":"CPU_MILLISECONDS","counterDisplayName":"CPU_
>>>>> MILLISECONDS","counterValue":5520},{"counterName":"
>>>>> PHYSICAL_MEMORY_BYTES","counterDisplayName":"PHYSICAL_
>>>>> MEMORY_BYTES","counterValue":661254144},{"counterName":"
>>>>> VIRTUAL_MEMORY_BYTES","counterDisplayName":"VIRTUAL_
>>>>> MEMORY_BYTES","counterValue":4294270976},{"counterName":"
>>>>> COMMITTED_HEAP_BYTES","counterDisplayName":"COMMITTED
>>>>> _HEAP_BYTES","counterValue":402653184},{"counterName":"
>>>>> INPUT_RECORDS_PROCESSED","counterDisplayName":"INPUT_RECORDS
>>>>> _PROCESSED","counterValue":8445},{"counterName":"OUTPUT_RECO
>>>>> RDS","counterDisplayName":"OUTPUT_RECORDS","counterValue"
>>>>> :191},{"counterName":"OUTPUT_BYTES","counterDisplayName":"
>>>>> OUTPUT_BYTES","counterValue":41350},{"counterName":"OUTPUT_
>>>>> BYTES_WITH_OVERHEAD","counterDisplayName":"OUTPUT_
>>>>> BYTES_WITH_OVERHEAD","counterValue":41957},{"
>>>>> counterName":"OUTPUT_BYTES_PHYSICAL","counterDisplayName"
>>>>> :"OUTPUT_BYTES_PHYSICAL","counterValue":41961},{"
>>>>> counterName":"ADDITIONAL_SPILLS_BYTES_WRITTEN","counterDisplayName":"
>>>>> ADDITIONAL_SPILLS_BYTES_WRITTEN","counterValue":41961},{"cou
>>>>> nterName":"ADDITIONAL_SPILLS_BYTES_READ","counterDisplayNam
>>>>> e":"ADDITIONAL_SPILLS_BYTES_READ","counterValue":41961},{"co
>>>>> unterName":"ADDITIONAL_SPILL_COUNT","counterDisplayName":"A
>>>>> DDITIONAL_SPILL_COUNT","counterValue":0},{"counterName":"SHU
>>>>> FFLE_BYTES","counterDisplayName":"SHUFFLE_BYTES","counterVal
>>>>> ue":41961},{"counterName":"SHUFFLE_BYTES_DECOMPRESSED","
>>>>> counterDisplayName":"SHUFFLE_BYTES_DECOMPRESSED","
>>>>> counterValue":41957},{"counterName":"SHUFFLE_BYTES_
>>>>> TO_MEM","counterDisplayName":"SHUFFLE_BYTES_TO_MEM","
>>>>> counterValue":41961},{"counterName":"SHUFFLE_BYTES_
>>>>> TO_DISK","counterDisplayName":"SHUFFLE_BYTES_TO_DISK","
>>>>> counterValue":0},{"counterName":"SHUFFLE_BYTES_
>>>>> DISK_DIRECT","counterDisplayName":"SHUFFLE_BYTES_DISK_DIRECT
>>>>> ","counterValue":0},{"counterName":"NUM_MEM_TO_DISK_MERGES",
>>>>> "counterDisplayName":"NUM_MEM_TO_DISK_MERGES","counterValue"
>>>>> :0},{"counterName":"NUM_DISK_TO_DISK_MERGES","counterDispla
>>>>> yName":"NUM_DISK_TO_DISK_MERGES","counterValue":0}]},{"count
>>>>> erGroupName":"HIVE","counterGroupDisplayName":"HIVE","counte
>>>>> rs":[{"counterName":"CREATED_FILES","counterDisplayName":"
>>>>> CREATED_FILES","counterValue":1},{"counterName":"
>>>>> DESERIALIZE_ERRORS","counterDisplayName":"DESERIALIZE_
>>>>> ERRORS","counterValue":0},{"counterName":"RECORDS_IN_Map_1
>>>>> ","counterDisplayName":"RECORDS_I
>>>>>
>>>>>
>>>>> But the json response isnt coming. What am I missing here?
>>>>>
>>>>> Thanks,
>>>>> Jyotsna
>>>>>
>>>>>
>>>>>
>>>>
>>
>
Re: Tez not returning .json results inspite of request completing sucessfully
Posted by Jyotsna G <jy...@gmail.com>.
Thank you so much Jonathan for your response. I could point my finger at
the issue a bit more in detail now. Its with the way connection is obtained
to Hive (on Tez) and closing it. I need to tweak my Tez configurations with
respect to sessions.
When executed with a single connection, it works perfectly fine and returns
the Json object to browser.
*SINGLE CONNECTION*
private static String driverName = "org.apache.hive.jdbc.HiveDriver";
private static String CONNECTION_URL= "jdbc:hive2://
slc06ffs.us.oracle.com:10000";
try {
Class.forName(driverName);
} catch (ClassNotFoundException e) {
e.printStackTrace();
System.exit(1);
}
Connection con = DriverManager.getConnection(CONNECTION_URL, "", "");
Statement stmt = con.createStatement();
//EXECUTE ALL YOUR STATEMENTS HERE
con.close();
The issue is with when they use pool of connections as below. Once
connection is obtained and query executed, results are not shown back on
browser and also hive server 2 refuses any further connections after that
single hung request (for which tez dag executes successfully)
*POOL *
And using the pool is like so:
Connection con = HiveDatasource.getConnectionFromPool();
Statement stmt = con.createStatement();
// EXECUTE ALL YOUR QUERIES HERE
con.close();
Hive server 2 log when using a connection pool on closing the
connection(Attached the entire log of debug mode):
*18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO
thrift.ThriftCLIService: Session disconnected without closing properly,
close it now*
18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG
session.SessionState: Removing resource dir
/tmp/38896527-5822-4453-9918-a5f36db09d1a_resources
*18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO
tez.TezSessionPoolManager: Closing tez session default? false*
*18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO
tez.TezSessionState: Closing Tez Session*
*18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO
client.TezClient: Shutting down Tez Session,
sessionName=HIVE-12d75efd-ec5d-4e9a-bfc7-3e3dbb48ee6b,
applicationId=application_1521448362533_0101*
18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG ipc.Client:
The ping interval is 60000 ms.
18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG ipc.Client:
Connecting to slc03qzg/10.240.178.10:9032
18/03/19 23:48:33 [IPC Client (368802354) connection to slc03qzg/
10.240.178.10:9032 from thirdeye]: DEBUG ipc.Client: IPC Client (368802354)
connection to slc03qzg/10.240.178.10:9032 from thirdeye: starting, having
connections 1
18/03/19 23:48:33 [IPC Parameter Sending Thread #2]: DEBUG ipc.Client: IPC
Client (368802354) connection to slc03qzg/10.240.178.10:9032 from thirdeye
sending #1811
18/03/19 23:48:33 [IPC Client (368802354) connection to slc03qzg/
10.240.178.10:9032 from thirdeye]: DEBUG ipc.Client: IPC Client (368802354)
connection to slc03qzg/10.240.178.10:9032 from thirdeye got value #1811
18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG
ipc.ProtobufRpcEngine: Call: getApplicationReport took 5ms
18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO
client.TezClientUtils: Application not running,
applicationId=application_1521448362533_0101,
yarnApplicationState=FINISHED, finalApplicationStatus=SUCCEEDED,
trackingUrl=slc03qzg:9088/proxy/application_1521448362533_0101/A,
diagnostics=Session stats:submittedDAGs=2, successfulDAGs=2, failedDAGs=0,
killedDAGs=0
*18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO
client.TezClient: Failed to shutdown Tez Session via proxy*
*org.apache.tez.dag.api.SessionNotRunning: Application not running,
applicationId=application_1521448362533_0101,
yarnApplicationState=FINISHED, finalApplicationStatus=SUCCEEDED,
trackingUrl=slc03qzg:9088/proxy/application_1521448362533_0101/A,
diagnostics=Session stats:submittedDAGs=2, successfulDAGs=2, failedDAGs=0,
killedDAGs=0*
at
org.apache.tez.client.TezClientUtils.getSessionAMProxy(TezClientUtils.java:798)
at org.apache.tez.client.TezClient.getSessionAMProxy(TezClient.java:633)
at org.apache.tez.client.TezClient.stop(TezClient.java:440)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.close(TezSessionState.java:257)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.close(TezSessionPoolManager.java:185)
at
org.apache.hadoop.hive.ql.session.SessionState.close(SessionState.java:1482)
at
org.apache.hive.service.cli.session.HiveSessionImpl.close(HiveSessionImpl.java:567)
at
org.apache.hive.service.cli.session.SessionManager.closeSession(SessionManager.java:292)
at org.apache.hive.service.cli.CLIService.closeSession(CLIService.java:237)
at
org.apache.hive.service.cli.thrift.ThriftCLIService$1.deleteContext(ThriftCLIService.java:122)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:300)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO
client.TezClient: Could not connect to AM, killing session via YARN,
sessionName=HIVE-12d75efd-ec5d-4e9a-bfc7-3e3dbb48ee6b,
applicationId=application_1521448362533_0101
18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO
impl.YarnClientImpl: Killing application application_1521448362533_0101
18/03/19 23:48:33 [IPC Parameter Sending Thread #2]: DEBUG ipc.Client: IPC
Client (368802354) connection to slc03qzg/10.240.178.10:9032 from thirdeye
sending #1812
18/03/19 23:48:33 [IPC Client (368802354) connection to slc03qzg/
10.240.178.10:9032 from thirdeye]: DEBUG ipc.Client: IPC Client (368802354)
connection to slc03qzg/10.240.178.10:9032 from thirdeye got value #1812
18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG
ipc.ProtobufRpcEngine: Call: forceKillApplication took 2ms
18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG
service.AbstractService: Service:
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl entered state STOPPED
18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG ipc.Client:
The ping interval is 60000 ms.
18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG ipc.Client:
Connecting to slc06ffs/10.245.175.89:55310
18/03/19 23:48:33 [IPC Client (368802354) connection to slc06ffs/
10.245.175.89:55310 from thirdeye]: DEBUG ipc.Client: IPC Client
(368802354) connection to slc06ffs/10.245.175.89:55310 from thirdeye:
starting, having connections 2
18/03/19 23:48:33 [IPC Parameter Sending Thread #2]: DEBUG ipc.Client: IPC
Client (368802354) connection to slc06ffs/10.245.175.89:55310 from thirdeye
sending #1813
18/03/19 23:48:33 [IPC Client (368802354) connection to slc06ffs/
10.245.175.89:55310 from thirdeye]: DEBUG ipc.Client: IPC Client
(368802354) connection to slc06ffs/10.245.175.89:55310 from thirdeye got
value #1813
18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG
ipc.ProtobufRpcEngine: Call: delete took 2ms
18/03/19 23:48:33 [IPC Parameter Sending Thread #2]: DEBUG ipc.Client: IPC
Client (368802354) connection to slc06ffs/10.245.175.89:55310 from thirdeye
sending #1814
18/03/19 23:48:33 [IPC Client (368802354) connection to slc06ffs/
10.245.175.89:55310 from thirdeye]: DEBUG ipc.Client: IPC Client
(368802354) connection to slc06ffs/10.245.175.89:55310 from thirdeye got
value #1814
18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG
ipc.ProtobufRpcEngine: Call: delete took 0ms
18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG
cli.CLIService: SessionHandle [38896527-5822-4453-9918-a5f36db09d1a]:
closeSession()
18/03/19 23:48:43 [IPC Client (368802354) connection to slc03qzg/
10.240.178.10:9032 from thirdeye]: DEBUG ipc.Client: IPC Client (368802354)
connection to slc03qzg/10.240.178.10:9032 from thirdeye: closed
18/03/19 23:48:43 [IPC Client (368802354) connection to slc03qzg/
10.240.178.10:9032 from thirdeye]: DEBUG ipc.Client: IPC Client (368802354)
connection to slc03qzg/10.240.178.10:9032 from thirdeye: stopped, remaining
connections 1
18/03/19 23:48:43 [IPC Client (368802354) connection to slc06ffs/
10.245.175.89:55310 from thirdeye]: DEBUG ipc.Client: IPC Client
(368802354) connection to slc06ffs/10.245.175.89:55310 from thirdeye: closed
18/03/19 23:48:43 [IPC Client (368802354) connection to slc06ffs/
10.245.175.89:55310 from thirdeye]: DEBUG ipc.Client: IPC Client
(368802354) connection to slc06ffs/10.245.175.89:55310 from thirdeye:
stopped, remaining connections 0
On Mon, Mar 19, 2018 at 10:22 PM, Jonathan Eagles <je...@gmail.com> wrote:
> I'm glad you checked this with an MR run. That help clarify the issue,
> ruling out the browser compatibility issue. As Tez is an application
> framework, it is not responsible for returning results to the browser.
> Using Hive CLI, it is the Hive CLI that is responsible for displaying the
> results of the query. For HiveServer2, it is HiveServer2's responsibility
> of displaying the results. It is true that the yarn and tez configurations
> for Hive CLI and HiveServer2 can be different, but it is not the
> responsibility of those settings to change how Tez returns results, as it
> is Hive CLI and HiveServer2 that returns results.
>
> Do you have access to the support logs or to a support team that can
> retrieve the relevant parts of those logs that will help to investigate
> this further? The relevant Tez logs you have provided are helpful, but
> only to show that Tez is working correctly. Without having access to the
> the HiveServer2 or thirdeye-services logs, I'm afraid it will be very
> difficult to debug this issue further.
>
> On Mon, Mar 19, 2018 at 11:22 AM, Jyotsna G <jy...@gmail.com> wrote:
>
>> Thank you for your response Jonathan.
>>
>> When I change the execution engine to MR, results are fetched perfectly
>> for the rest query, it's only when I change the execution engine to tez
>> that I see the issue occurring.
>>
>> To rephrase my question, are there any further sessions specific
>> configurations for tez that I might have missed? Any yarn related
>> configurations for tez that I might have to look for?
>>
>>
>>
>> On Mon, Mar 19, 2018, 21:37 Jonathan Eagles <je...@gmail.com> wrote:
>>
>>> Thanks for reaching out to us, Jyotsna.
>>>
>>> Let me make sure I understand what your setup and the issue you are
>>> experiencing. It sounds like the setup looks like this.
>>>
>>> Browser -> Third party service "thirdeye-services" -> HiveServer2 ->
>>> Tez
>>>
>>> Since you are showing logs and screenshots of the successful Tez Job. My
>>> initial thought is that HiveServer2 was able to successfully and correctly
>>> get the results from Tez. That would make me believe that either the
>>> connection between the browser to the third party service
>>> "thirdeye-services" isn't working (perhaps browser compatibility issue,
>>> security issue, or service issue) or the connection between the third party
>>> service "thirdeye-services" to HiveServer2 issue (issue retrieving results
>>> from HiveServer2).
>>>
>>> Since our expertise is only with the Tez portion of the setup and that
>>> the Tez portion of the setup seems to be working, the Tez community won't
>>> be the best at addressing this issue. My recommendation would be try the
>>> query in a different browser as a simple test (Chrome, Firefox) checking
>>> with third party "thirdeye-services" browser compatibility. If you have
>>> access to the thirdparty-services logs or logs to the HiveServer2 logs
>>> those will be helpful in debugging. When reaching out to the
>>> thirdeye-services support or HiveServer2 please reference these logs to get
>>> the best support from the community. The hive user list is
>>> user@hive.apache.org and I have don't have a support contact for the
>>> thirdparty software thirdeye-services that seems to be part of this setup.
>>>
>>> Thanks again for reaching to the Tez Community.
>>>
>>> Regards,
>>> jeagles
>>>
>>> On Mon, Mar 19, 2018 at 6:47 AM, Jyotsna G <jy...@gmail.com> wrote:
>>>
>>>> Hi,
>>>> We are making a rest URI request which hits Hive server 2 and
>>>> translates in to a hive query on TEZ.
>>>>
>>>> Sample uri:
>>>> slc00yeb.us.oracle.com:8989/thirdeye-services/v1.0/OSCS/quer
>>>> y/OSCSRecords.json?COMP_FAMILY=FMW&COMP_NAME=WLS&COMPTYPE1_
>>>> FIRST=RC1&COMPTYPE1_SECOND=RC2&COMPTYPE2_FIRST=manifest.
>>>> publish.label.WLS_GENERIC:carb-manifest:12.2.1.3.0-
>>>> 170722.1880245:pom&COMPTYPE2_SECOND=manifest.publish.label.
>>>> WLS_GENERIC:carb-manifest:12.2.1.3.0-170726.1880744:pom&
>>>> RELEASE_FIRST=12.2.1.3.0&RELEASE_SECOND=12.2.1.3.0&COMPTYPE=MANIFEST
>>>>
>>>> Hadoop version : 2.4.0
>>>> Hive:1.2.2
>>>> Tez : 5.4.0
>>>>
>>>> All my queries run fine in HIVE CLI , also via standalone program
>>>> connecting to hive via JDBC ODBC connection and returns hive query results,
>>>> but only when I hit the rest URL, results are not returned.
>>>>
>>>> I have enabled the below settings for tez
>>>>
>>>> tez.session.am.dag.submit.timeout.secs 90
>>>>
>>>> tez.am.mode.session : true
>>>> tez.am.container.reuse.enable: true
>>>> tez.allow.disabled.timeline-domains : true
>>>> tez.yarn.ats.enabled : true
>>>> tez.use.cluster.hadoop-libs : true
>>>>
>>>> The url on the browser keeps spinning without the json response, even
>>>> after the tez job on yarn cluster has finished successfully. Attached Tez
>>>> job diagnosis screenshot.
>>>>
>>>> From the dag log :
>>>>
>>>> 2018-03-19 11:34:36,367 INFO [AsyncDispatcher event handler]
>>>> impl.DAGImpl: dag_1521448362533_0019_1 transitioned from RUNNING to
>>>> SUCCEEDED
>>>> 2018-03-19 11:34:36,368 INFO [AsyncDispatcher event handler]
>>>> app.DAGAppMaster: DAG completed, dagId=dag_1521448362533_0019_1,
>>>> dagState=SUCCEEDED
>>>>
>>>> {"entity":"dag_1521448362533_0019_1","entitytype":"TEZ_DAG_I
>>>> D","events":[{"ts":1521459276335,"eventtype":"DAG_FINISHED"}
>>>> ],"otherinfo":{"startTime":1521459269619,"endTime":
>>>> 1521459276335,"timeTaken":6716,"status":"SUCCEEDED","
>>>> diagnostics":"","counters":{"counterGroups":[{"counterGroup
>>>> Name":"org.apache.tez.common.counters.DAGCounter","counterG
>>>> roupDisplayName":"org.apache.tez.common.counters.DAGCounter
>>>> ","counters":[{"counterName":"NUM_SUCCEEDED_TASKS","
>>>> counterDisplayName":"NUM_SUCCEEDED_TASKS","counterValue
>>>> ":2},{"counterName":"TOTAL_LAUNCHED_TASKS","counterDisplayName":"TOTAL_
>>>> LAUNCHED_TASKS","counterValue":2},{"counterName":"DATA_LOCAL_TASKS","
>>>> counterDisplayName":"DATA_LOCAL_TASKS","counterValue":1}
>>>> ]},{"counterGroupName":"org.apache.tez.common.counters.Fil
>>>> eSystemCounter","counterGroupDisplayName":"File System
>>>> Counters","counters":[{"counterName":"FILE_BYTES_READ","
>>>> counterDisplayName":"FILE_BYTES_READ","counterValue":41993},
>>>> {"counterName":"FILE_BYTES_WRITTEN","counterDisplayName":
>>>> "FILE_BYTES_WRITTEN","counterValue":83954},{"
>>>> counterName":"FILE_READ_OPS","counterDisplayName":"FILE_
>>>> READ_OPS","counterValue":0},{"counterName":"FILE_LARGE_READ_
>>>> OPS","counterDisplayName":"FILE_LARGE_READ_OPS","counterValu
>>>> e":0},{"counterName":"FILE_WRITE_OPS","counterDisplayName":"FILE_WRIT
>>>> E_OPS","counterValue":0},{"counterName":"HDFS_BYTES_READ"
>>>> ,"counterDisplayName":"HDFS_BYTES_READ","counterValue":
>>>> 59231},{"counterName":"HDFS_BYTES_WRITTEN","counterDisplayName":"HDFS_
>>>> BYTES_WRITTEN","counterValue":39631},{"counterName":"HDFS_
>>>> READ_OPS","counterDisplayName":"HDFS_READ_OPS","
>>>> counterValue":5},{"counterName":"HDFS_LARGE_READ_OPS","
>>>> counterDisplayName":"HDFS_LARGE_READ_OPS","counterValue"
>>>> :0},{"counterName":"HDFS_WRITE_OPS","counterDisplayName":"HDFS_WRIT
>>>> E_OPS","counterValue":2}]},{"counterGroupName":"org.apache.
>>>> tez.common.counters.TaskCounter","counterGroupDisplayName":"
>>>> org.apache.tez.common.counters.TaskCounter","
>>>> counters":[{"counterName":"REDUCE_INPUT_GROUPS","
>>>> counterDisplayName":"REDUCE_INPUT_GROUPS","counterValue":
>>>> 190},{"counterName":"REDUCE_INPUT_RECORDS","
>>>> counterDisplayName":"REDUCE_INPUT_RECORDS","counterValue":
>>>> 191},{"counterName":"COMBINE_INPUT_RECORDS","
>>>> counterDisplayName":"COMBINE_INPUT_RECORDS","counterValue":
>>>> 0},{"counterName":"SPILLED_RECORDS","counterDisplayName":
>>>> "SPILLED_RECORDS","counterValue":382},{"counterName":"NUM_
>>>> SHUFFLED_INPUTS","counterDisplayName":"NUM_
>>>> SHUFFLED_INPUTS","counterValue":1},{"counterName":"NUM_
>>>> SKIPPED_INPUTS","counterDisplayName":"NUM_
>>>> SKIPPED_INPUTS","counterValue":0},{"counterName":"NUM_
>>>> FAILED_SHUFFLE_INPUTS","counterDisplayName":"NUM_FAILED_
>>>> SHUFFLE_INPUTS","counterValue":0},{"counterName":"MERGED_MAP_OUTPUTS","
>>>> counterDisplayName":"MERGED_MAP_OUTPUTS","counterValue":1}
>>>> ,{"counterName":"GC_TIME_MILLIS","counterDisplayName":"
>>>> GC_TIME_MILLIS","counterValue":84},{"counterName":"CPU_MILLI
>>>> SECONDS","counterDisplayName":"CPU_MILLISECONDS","
>>>> counterValue":5520},{"counterName":"PHYSICAL_MEMORY_
>>>> BYTES","counterDisplayName":"PHYSICAL_MEMORY_BYTES","
>>>> counterValue":661254144},{"counterName":"VIRTUAL_MEMORY_
>>>> BYTES","counterDisplayName":"VIRTUAL_MEMORY_BYTES","
>>>> counterValue":4294270976},{"counterName":"COMMITTED_HEAP_
>>>> BYTES","counterDisplayName":"COMMITTED_HEAP_BYTES","counterV
>>>> alue":402653184},{"counterName":"INPUT_RECORDS_PROCESSED","c
>>>> ounterDisplayName":"INPUT_RECORDS_PROCESSED","counterValue":
>>>> 8445},{"counterName":"OUTPUT_RECORDS","counterDisplayName":
>>>> "OUTPUT_RECORDS","counterValue":191},{"counterName":"OUTPUT_BYTES","c
>>>> ounterDisplayName":"OUTPUT_BYTES","counterValue":41350},{"
>>>> counterName":"OUTPUT_BYTES_WITH_OVERHEAD","counterDisplayNam
>>>> e":"OUTPUT_BYTES_WITH_OVERHEAD","counterValue":
>>>> 41957},{"counterName":"OUTPUT_BYTES_PHYSICAL","
>>>> counterDisplayName":"OUTPUT_BYTES_PHYSICAL","counterValue"
>>>> :41961},{"counterName":"ADDITIONAL_SPILLS_BYTES_
>>>> WRITTEN","counterDisplayName":"ADDITIONAL_SPILLS_BYTES_WRITT
>>>> EN","counterValue":41961},{"counterName":"ADDITIONAL_SPILL
>>>> S_BYTES_READ","counterDisplayName":"ADDITIONAL_SPILLS_BYTES_
>>>> READ","counterValue":41961},{"counterName":"ADDITIONAL_SPILL
>>>> _COUNT","counterDisplayName":"ADDITIONAL_SPILL_COUNT","count
>>>> erValue":0},{"counterName":"SHUFFLE_BYTES","counterDisplay
>>>> Name":"SHUFFLE_BYTES","counterValue":41961},{"
>>>> counterName":"SHUFFLE_BYTES_DECOMPRESSED","counterDisplayNam
>>>> e":"SHUFFLE_BYTES_DECOMPRESSED","counterValue":41957},{"counterName":"
>>>> SHUFFLE_BYTES_TO_MEM","counterDisplayName":"SHUFFLE_
>>>> BYTES_TO_MEM","counterValue":41961},{"counterName":"
>>>> SHUFFLE_BYTES_TO_DISK","counterDisplayName":"SHUFFLE_
>>>> BYTES_TO_DISK","counterValue":0},{"counterName":"SHUFFLE_
>>>> BYTES_DISK_DIRECT","counterDisplayName":"SHUFFLE_BYTES_DISK_
>>>> DIRECT","counterValue":0},{"counterName":"NUM_MEM_TO_DISK_ME
>>>> RGES","counterDisplayName":"NUM_MEM_TO_DISK_MERGES","counter
>>>> Value":0},{"counterName":"NUM_DISK_TO_DISK_MERGES","counterD
>>>> isplayName":"NUM_DISK_TO_DISK_MERGES","counterValue":0}]},{"
>>>> counterGroupName":"HIVE","counterGroupDisplayName":"HIVE","
>>>> counters":[{"counterName":"CREATED_FILES","counterDisplay
>>>> Name":"CREATED_FILES","counterValue":1},{"counterName
>>>> ":"DESERIALIZE_ERRORS","counterDisplayName":"DESERIALI
>>>> ZE_ERRORS","counterValue":0},{"counterName":"RECORDS_IN_Map_
>>>> 1","counterDisplayName":"RECORDS_I
>>>>
>>>>
>>>> But the json response isnt coming. What am I missing here?
>>>>
>>>> Thanks,
>>>> Jyotsna
>>>>
>>>>
>>>>
>>>
>
Re: Tez not returning .json results inspite of request completing sucessfully
Posted by Jonathan Eagles <je...@gmail.com>.
I'm glad you checked this with an MR run. That help clarify the issue,
ruling out the browser compatibility issue. As Tez is an application
framework, it is not responsible for returning results to the browser.
Using Hive CLI, it is the Hive CLI that is responsible for displaying the
results of the query. For HiveServer2, it is HiveServer2's responsibility
of displaying the results. It is true that the yarn and tez configurations
for Hive CLI and HiveServer2 can be different, but it is not the
responsibility of those settings to change how Tez returns results, as it
is Hive CLI and HiveServer2 that returns results.
Do you have access to the support logs or to a support team that can
retrieve the relevant parts of those logs that will help to investigate
this further? The relevant Tez logs you have provided are helpful, but
only to show that Tez is working correctly. Without having access to the
the HiveServer2 or thirdeye-services logs, I'm afraid it will be very
difficult to debug this issue further.
On Mon, Mar 19, 2018 at 11:22 AM, Jyotsna G <jy...@gmail.com> wrote:
> Thank you for your response Jonathan.
>
> When I change the execution engine to MR, results are fetched perfectly
> for the rest query, it's only when I change the execution engine to tez
> that I see the issue occurring.
>
> To rephrase my question, are there any further sessions specific
> configurations for tez that I might have missed? Any yarn related
> configurations for tez that I might have to look for?
>
>
>
> On Mon, Mar 19, 2018, 21:37 Jonathan Eagles <je...@gmail.com> wrote:
>
>> Thanks for reaching out to us, Jyotsna.
>>
>> Let me make sure I understand what your setup and the issue you are
>> experiencing. It sounds like the setup looks like this.
>>
>> Browser -> Third party service "thirdeye-services" -> HiveServer2 -> Tez
>>
>> Since you are showing logs and screenshots of the successful Tez Job. My
>> initial thought is that HiveServer2 was able to successfully and correctly
>> get the results from Tez. That would make me believe that either the
>> connection between the browser to the third party service
>> "thirdeye-services" isn't working (perhaps browser compatibility issue,
>> security issue, or service issue) or the connection between the third party
>> service "thirdeye-services" to HiveServer2 issue (issue retrieving results
>> from HiveServer2).
>>
>> Since our expertise is only with the Tez portion of the setup and that
>> the Tez portion of the setup seems to be working, the Tez community won't
>> be the best at addressing this issue. My recommendation would be try the
>> query in a different browser as a simple test (Chrome, Firefox) checking
>> with third party "thirdeye-services" browser compatibility. If you have
>> access to the thirdparty-services logs or logs to the HiveServer2 logs
>> those will be helpful in debugging. When reaching out to the
>> thirdeye-services support or HiveServer2 please reference these logs to get
>> the best support from the community. The hive user list is
>> user@hive.apache.org and I have don't have a support contact for the
>> thirdparty software thirdeye-services that seems to be part of this setup.
>>
>> Thanks again for reaching to the Tez Community.
>>
>> Regards,
>> jeagles
>>
>> On Mon, Mar 19, 2018 at 6:47 AM, Jyotsna G <jy...@gmail.com> wrote:
>>
>>> Hi,
>>> We are making a rest URI request which hits Hive server 2 and translates
>>> in to a hive query on TEZ.
>>>
>>> Sample uri:
>>> slc00yeb.us.oracle.com:8989/thirdeye-services/v1.0/OSCS/
>>> query/OSCSRecords.json?COMP_FAMILY=FMW&COMP_NAME=WLS&
>>> COMPTYPE1_FIRST=RC1&COMPTYPE1_SECOND=RC2&COMPTYPE2_FIRST=
>>> manifest.publish.label.WLS_GENERIC:carb-manifest:12.2.1.
>>> 3.0-170722.1880245:pom&COMPTYPE2_SECOND=manifest.
>>> publish.label.WLS_GENERIC:carb-manifest:12.2.1.3.0-
>>> 170726.1880744:pom&RELEASE_FIRST=12.2.1.3.0&RELEASE_
>>> SECOND=12.2.1.3.0&COMPTYPE=MANIFEST
>>>
>>> Hadoop version : 2.4.0
>>> Hive:1.2.2
>>> Tez : 5.4.0
>>>
>>> All my queries run fine in HIVE CLI , also via standalone program
>>> connecting to hive via JDBC ODBC connection and returns hive query results,
>>> but only when I hit the rest URL, results are not returned.
>>>
>>> I have enabled the below settings for tez
>>>
>>> tez.session.am.dag.submit.timeout.secs 90
>>>
>>> tez.am.mode.session : true
>>> tez.am.container.reuse.enable: true
>>> tez.allow.disabled.timeline-domains : true
>>> tez.yarn.ats.enabled : true
>>> tez.use.cluster.hadoop-libs : true
>>>
>>> The url on the browser keeps spinning without the json response, even
>>> after the tez job on yarn cluster has finished successfully. Attached Tez
>>> job diagnosis screenshot.
>>>
>>> From the dag log :
>>>
>>> 2018-03-19 11:34:36,367 INFO [AsyncDispatcher event handler]
>>> impl.DAGImpl: dag_1521448362533_0019_1 transitioned from RUNNING to
>>> SUCCEEDED
>>> 2018-03-19 11:34:36,368 INFO [AsyncDispatcher event handler]
>>> app.DAGAppMaster: DAG completed, dagId=dag_1521448362533_0019_1,
>>> dagState=SUCCEEDED
>>>
>>> {"entity":"dag_1521448362533_0019_1","entitytype":"TEZ_DAG_
>>> ID","events":[{"ts":1521459276335,"eventtype":"
>>> DAG_FINISHED"}],"otherinfo":{"startTime":1521459269619,"
>>> endTime":1521459276335,"timeTaken":6716,"status":"
>>> SUCCEEDED","diagnostics":"","counters":{"counterGroups":[{"
>>> counterGroupName":"org.apache.tez.common.counters.DAGCounter","
>>> counterGroupDisplayName":"org.apache.tez.common.counters.
>>> DAGCounter","counters":[{"counterName":"NUM_SUCCEEDED_
>>> TASKS","counterDisplayName":"NUM_SUCCEEDED_TASKS","counterValue":2},{"
>>> counterName":"TOTAL_LAUNCHED_TASKS","counterDisplayName":"
>>> TOTAL_LAUNCHED_TASKS","counterValue":2},{"counterName":"DATA_LOCAL_
>>> TASKS","counterDisplayName":"DATA_LOCAL_TASKS","counterValue":1}]},{"
>>> counterGroupName":"org.apache.tez.common.counters.FileSystemCounter","counterGroupDisplayName":"File
>>> System Counters","counters":[{"counterName":"FILE_BYTES_READ"
>>> ,"counterDisplayName":"FILE_BYTES_READ","counterValue":
>>> 41993},{"counterName":"FILE_BYTES_WRITTEN","counterDisplayName":"FILE_
>>> BYTES_WRITTEN","counterValue":83954},{"counterName":"FILE_
>>> READ_OPS","counterDisplayName":"FILE_READ_OPS","counterValue":0},{"
>>> counterName":"FILE_LARGE_READ_OPS","counterDisplayName":"
>>> FILE_LARGE_READ_OPS","counterValue":0},{"counterName":"FILE_WRITE_OPS",
>>> "counterDisplayName":"FILE_WRITE_OPS","counterValue":0},{
>>> "counterName":"HDFS_BYTES_READ","counterDisplayName":"HDFS_BYTES_READ","
>>> counterValue":59231},{"counterName":"HDFS_BYTES_
>>> WRITTEN","counterDisplayName":"HDFS_BYTES_WRITTEN","
>>> counterValue":39631},{"counterName":"HDFS_READ_OPS","
>>> counterDisplayName":"HDFS_READ_OPS","counterValue":5},{"
>>> counterName":"HDFS_LARGE_READ_OPS","counterDisplayName":"
>>> HDFS_LARGE_READ_OPS","counterValue":0},{"counterName":"HDFS_WRITE_OPS",
>>> "counterDisplayName":"HDFS_WRITE_OPS","counterValue":2}]}
>>> ,{"counterGroupName":"org.apache.tez.common.counters.TaskCounter","
>>> counterGroupDisplayName":"org.apache.tez.common.counters.
>>> TaskCounter","counters":[{"counterName":"REDUCE_INPUT_
>>> GROUPS","counterDisplayName":"REDUCE_INPUT_GROUPS","
>>> counterValue":190},{"counterName":"REDUCE_INPUT_
>>> RECORDS","counterDisplayName":"REDUCE_INPUT_RECORDS","
>>> counterValue":191},{"counterName":"COMBINE_INPUT_
>>> RECORDS","counterDisplayName":"COMBINE_INPUT_RECORDS","
>>> counterValue":0},{"counterName":"SPILLED_RECORDS","counterDisplayName":"
>>> SPILLED_RECORDS","counterValue":382},{"counterName":"NUM_SHUFFLED_
>>> INPUTS","counterDisplayName":"NUM_SHUFFLED_INPUTS","counterValue":1},{"
>>> counterName":"NUM_SKIPPED_INPUTS","counterDisplayName":"
>>> NUM_SKIPPED_INPUTS","counterValue":0},{"counterName":"NUM_FAILED_
>>> SHUFFLE_INPUTS","counterDisplayName":"NUM_FAILED_SHUFFLE_INPUTS","
>>> counterValue":0},{"counterName":"MERGED_MAP_
>>> OUTPUTS","counterDisplayName":"MERGED_MAP_OUTPUTS","counterValue":1},{"
>>> counterName":"GC_TIME_MILLIS","counterDisplayName":"GC_TIME_
>>> MILLIS","counterValue":84},{"counterName":"CPU_MILLISECONDS","
>>> counterDisplayName":"CPU_MILLISECONDS","counterValue":
>>> 5520},{"counterName":"PHYSICAL_MEMORY_BYTES","
>>> counterDisplayName":"PHYSICAL_MEMORY_BYTES","counterValue":
>>> 661254144},{"counterName":"VIRTUAL_MEMORY_BYTES","
>>> counterDisplayName":"VIRTUAL_MEMORY_BYTES","counterValue":
>>> 4294270976},{"counterName":"COMMITTED_HEAP_BYTES","counterDisplayName":"
>>> COMMITTED_HEAP_BYTES","counterValue":402653184},{"
>>> counterName":"INPUT_RECORDS_PROCESSED","counterDisplayName":"INPUT_
>>> RECORDS_PROCESSED","counterValue":8445},{"counterName":"OUTPUT_RECORDS",
>>> "counterDisplayName":"OUTPUT_RECORDS","counterValue":191},{
>>> "counterName":"OUTPUT_BYTES","counterDisplayName":"OUTPUT_
>>> BYTES","counterValue":41350},{"counterName":"OUTPUT_BYTES_
>>> WITH_OVERHEAD","counterDisplayName":"OUTPUT_BYTES_WITH_OVERHEAD","
>>> counterValue":41957},{"counterName":"OUTPUT_BYTES_
>>> PHYSICAL","counterDisplayName":"OUTPUT_BYTES_PHYSICAL","
>>> counterValue":41961},{"counterName":"ADDITIONAL_SPILLS_BYTES_WRITTEN","
>>> counterDisplayName":"ADDITIONAL_SPILLS_BYTES_
>>> WRITTEN","counterValue":41961},{"counterName":"ADDITIONAL_
>>> SPILLS_BYTES_READ","counterDisplayName":"ADDITIONAL_SPILLS_BYTES_READ",
>>> "counterValue":41961},{"counterName":"ADDITIONAL_SPILL_COUNT","
>>> counterDisplayName":"ADDITIONAL_SPILL_COUNT","counterValue":0},{"
>>> counterName":"SHUFFLE_BYTES","counterDisplayName":"SHUFFLE_
>>> BYTES","counterValue":41961},{"counterName":"SHUFFLE_BYTES_
>>> DECOMPRESSED","counterDisplayName":"SHUFFLE_BYTES_DECOMPRESSED","
>>> counterValue":41957},{"counterName":"SHUFFLE_BYTES_
>>> TO_MEM","counterDisplayName":"SHUFFLE_BYTES_TO_MEM","
>>> counterValue":41961},{"counterName":"SHUFFLE_BYTES_
>>> TO_DISK","counterDisplayName":"SHUFFLE_BYTES_TO_DISK","
>>> counterValue":0},{"counterName":"SHUFFLE_BYTES_DISK_DIRECT","
>>> counterDisplayName":"SHUFFLE_BYTES_DISK_DIRECT","counterValue":0},{"
>>> counterName":"NUM_MEM_TO_DISK_MERGES","counterDisplayName":"
>>> NUM_MEM_TO_DISK_MERGES","counterValue":0},{"counterName":"NUM_DISK_TO_
>>> DISK_MERGES","counterDisplayName":"NUM_DISK_
>>> TO_DISK_MERGES","counterValue":0}]},{"counterGroupName":"HIVE","
>>> counterGroupDisplayName":"HIVE","counters":[{"
>>> counterName":"CREATED_FILES","counterDisplayName":"CREATED_
>>> FILES","counterValue":1},{"counterName":"DESERIALIZE_
>>> ERRORS","counterDisplayName":"DESERIALIZE_ERRORS","counterValue":0},{"
>>> counterName":"RECORDS_IN_Map_1","counterDisplayName":"RECORDS_I
>>>
>>>
>>> But the json response isnt coming. What am I missing here?
>>>
>>> Thanks,
>>> Jyotsna
>>>
>>>
>>>
>>
Re: Tez not returning .json results inspite of request completing sucessfully
Posted by Jyotsna G <jy...@gmail.com>.
Thank you for your response Jonathan.
When I change the execution engine to MR, results are fetched perfectly for
the rest query, it's only when I change the execution engine to tez that I
see the issue occurring.
To rephrase my question, are there any further sessions specific
configurations for tez that I might have missed? Any yarn related
configurations for tez that I might have to look for?
On Mon, Mar 19, 2018, 21:37 Jonathan Eagles <je...@gmail.com> wrote:
> Thanks for reaching out to us, Jyotsna.
>
> Let me make sure I understand what your setup and the issue you are
> experiencing. It sounds like the setup looks like this.
>
> Browser -> Third party service "thirdeye-services" -> HiveServer2 -> Tez
>
> Since you are showing logs and screenshots of the successful Tez Job. My
> initial thought is that HiveServer2 was able to successfully and correctly
> get the results from Tez. That would make me believe that either the
> connection between the browser to the third party service
> "thirdeye-services" isn't working (perhaps browser compatibility issue,
> security issue, or service issue) or the connection between the third party
> service "thirdeye-services" to HiveServer2 issue (issue retrieving results
> from HiveServer2).
>
> Since our expertise is only with the Tez portion of the setup and that the
> Tez portion of the setup seems to be working, the Tez community won't be
> the best at addressing this issue. My recommendation would be try the query
> in a different browser as a simple test (Chrome, Firefox) checking with
> third party "thirdeye-services" browser compatibility. If you have access
> to the thirdparty-services logs or logs to the HiveServer2 logs those will
> be helpful in debugging. When reaching out to the thirdeye-services support
> or HiveServer2 please reference these logs to get the best support from the
> community. The hive user list is user@hive.apache.org and I have don't
> have a support contact for the thirdparty software thirdeye-services that
> seems to be part of this setup.
>
> Thanks again for reaching to the Tez Community.
>
> Regards,
> jeagles
>
> On Mon, Mar 19, 2018 at 6:47 AM, Jyotsna G <jy...@gmail.com> wrote:
>
>> Hi,
>> We are making a rest URI request which hits Hive server 2 and translates
>> in to a hive query on TEZ.
>>
>> Sample uri:
>>
>> slc00yeb.us.oracle.com:8989/thirdeye-services/v1.0/OSCS/query/OSCSRecords.json?COMP_FAMILY=FMW&COMP_NAME=WLS&COMPTYPE1_FIRST=RC1&COMPTYPE1_SECOND=RC2&COMPTYPE2_FIRST=manifest.publish.label.WLS_GENERIC:carb-manifest:12.2.1.3.0-170722.1880245:pom&COMPTYPE2_SECOND=manifest.publish.label.WLS_GENERIC:carb-manifest:12.2.1.3.0-170726.1880744:pom&RELEASE_FIRST=12.2.1.3.0&RELEASE_SECOND=12.2.1.3.0&COMPTYPE=MANIFEST
>>
>> Hadoop version : 2.4.0
>> Hive:1.2.2
>> Tez : 5.4.0
>>
>> All my queries run fine in HIVE CLI , also via standalone program
>> connecting to hive via JDBC ODBC connection and returns hive query results,
>> but only when I hit the rest URL, results are not returned.
>>
>> I have enabled the below settings for tez
>>
>> tez.session.am.dag.submit.timeout.secs 90
>>
>> tez.am.mode.session : true
>> tez.am.container.reuse.enable: true
>> tez.allow.disabled.timeline-domains : true
>> tez.yarn.ats.enabled : true
>> tez.use.cluster.hadoop-libs : true
>>
>> The url on the browser keeps spinning without the json response, even
>> after the tez job on yarn cluster has finished successfully. Attached Tez
>> job diagnosis screenshot.
>>
>> From the dag log :
>>
>> 2018-03-19 11:34:36,367 INFO [AsyncDispatcher event handler]
>> impl.DAGImpl: dag_1521448362533_0019_1 transitioned from RUNNING to
>> SUCCEEDED
>> 2018-03-19 11:34:36,368 INFO [AsyncDispatcher event handler]
>> app.DAGAppMaster: DAG completed, dagId=dag_1521448362533_0019_1,
>> dagState=SUCCEEDED
>>
>> {"entity":"dag_1521448362533_0019_1","entitytype":"TEZ_DAG_ID","events":[{"ts":1521459276335,"eventtype":"DAG_FINISHED"}],"otherinfo":{"startTime":1521459269619,"endTime":1521459276335,"timeTaken":6716,"status":"SUCCEEDED","diagnostics":"","counters":{"counterGroups":[{"counterGroupName":"org.apache.tez.common.counters.DAGCounter","counterGroupDisplayName":"org.apache.tez.common.counters.DAGCounter","counters":[{"counterName":"NUM_SUCCEEDED_TASKS","counterDisplayName":"NUM_SUCCEEDED_TASKS","counterValue":2},{"counterName":"TOTAL_LAUNCHED_TASKS","counterDisplayName":"TOTAL_LAUNCHED_TASKS","counterValue":2},{"counterName":"DATA_LOCAL_TASKS","counterDisplayName":"DATA_LOCAL_TASKS","counterValue":1}]},{"counterGroupName":"org.apache.tez.common.counters.FileSystemCounter","counterGroupDisplayName":"File
>> System
>> Counters","counters":[{"counterName":"FILE_BYTES_READ","counterDisplayName":"FILE_BYTES_READ","counterValue":41993},{"counterName":"FILE_BYTES_WRITTEN","counterDisplayName":"FILE_BYTES_WRITTEN","counterValue":83954},{"counterName":"FILE_READ_OPS","counterDisplayName":"FILE_READ_OPS","counterValue":0},{"counterName":"FILE_LARGE_READ_OPS","counterDisplayName":"FILE_LARGE_READ_OPS","counterValue":0},{"counterName":"FILE_WRITE_OPS","counterDisplayName":"FILE_WRITE_OPS","counterValue":0},{"counterName":"HDFS_BYTES_READ","counterDisplayName":"HDFS_BYTES_READ","counterValue":59231},{"counterName":"HDFS_BYTES_WRITTEN","counterDisplayName":"HDFS_BYTES_WRITTEN","counterValue":39631},{"counterName":"HDFS_READ_OPS","counterDisplayName":"HDFS_READ_OPS","counterValue":5},{"counterName":"HDFS_LARGE_READ_OPS","counterDisplayName":"HDFS_LARGE_READ_OPS","counterValue":0},{"counterName":"HDFS_WRITE_OPS","counterDisplayName":"HDFS_WRITE_OPS","counterValue":2}]},{"counterGroupName":"org.apache.tez.common.counters.TaskCounter","counterGroupDisplayName":"org.apache.tez.common.counters.TaskCounter","counters":[{"counterName":"REDUCE_INPUT_GROUPS","counterDisplayName":"REDUCE_INPUT_GROUPS","counterValue":190},{"counterName":"REDUCE_INPUT_RECORDS","counterDisplayName":"REDUCE_INPUT_RECORDS","counterValue":191},{"counterName":"COMBINE_INPUT_RECORDS","counterDisplayName":"COMBINE_INPUT_RECORDS","counterValue":0},{"counterName":"SPILLED_RECORDS","counterDisplayName":"SPILLED_RECORDS","counterValue":382},{"counterName":"NUM_SHUFFLED_INPUTS","counterDisplayName":"NUM_SHUFFLED_INPUTS","counterValue":1},{"counterName":"NUM_SKIPPED_INPUTS","counterDisplayName":"NUM_SKIPPED_INPUTS","counterValue":0},{"counterName":"NUM_FAILED_SHUFFLE_INPUTS","counterDisplayName":"NUM_FAILED_SHUFFLE_INPUTS","counterValue":0},{"counterName":"MERGED_MAP_OUTPUTS","counterDisplayName":"MERGED_MAP_OUTPUTS","counterValue":1},{"counterName":"GC_TIME_MILLIS","counterDisplayName":"GC_TIME_MILLIS","counterValue":84},{"counterName":"CPU_MILLISECONDS","counterDisplayName":"CPU_MILLISECONDS","counterValue":5520},{"counterName":"PHYSICAL_MEMORY_BYTES","counterDisplayName":"PHYSICAL_MEMORY_BYTES","counterValue":661254144},{"counterName":"VIRTUAL_MEMORY_BYTES","counterDisplayName":"VIRTUAL_MEMORY_BYTES","counterValue":4294270976},{"counterName":"COMMITTED_HEAP_BYTES","counterDisplayName":"COMMITTED_HEAP_BYTES","counterValue":402653184},{"counterName":"INPUT_RECORDS_PROCESSED","counterDisplayName":"INPUT_RECORDS_PROCESSED","counterValue":8445},{"counterName":"OUTPUT_RECORDS","counterDisplayName":"OUTPUT_RECORDS","counterValue":191},{"counterName":"OUTPUT_BYTES","counterDisplayName":"OUTPUT_BYTES","counterValue":41350},{"counterName":"OUTPUT_BYTES_WITH_OVERHEAD","counterDisplayName":"OUTPUT_BYTES_WITH_OVERHEAD","counterValue":41957},{"counterName":"OUTPUT_BYTES_PHYSICAL","counterDisplayName":"OUTPUT_BYTES_PHYSICAL","counterValue":41961},{"counterName":"ADDITIONAL_SPILLS_BYTES_WRITTEN","counterDisplayName":"ADDITIONAL_SPILLS_BYTES_WRITTEN","counterValue":41961},{"counterName":"ADDITIONAL_SPILLS_BYTES_READ","counterDisplayName":"ADDITIONAL_SPILLS_BYTES_READ","counterValue":41961},{"counterName":"ADDITIONAL_SPILL_COUNT","counterDisplayName":"ADDITIONAL_SPILL_COUNT","counterValue":0},{"counterName":"SHUFFLE_BYTES","counterDisplayName":"SHUFFLE_BYTES","counterValue":41961},{"counterName":"SHUFFLE_BYTES_DECOMPRESSED","counterDisplayName":"SHUFFLE_BYTES_DECOMPRESSED","counterValue":41957},{"counterName":"SHUFFLE_BYTES_TO_MEM","counterDisplayName":"SHUFFLE_BYTES_TO_MEM","counterValue":41961},{"counterName":"SHUFFLE_BYTES_TO_DISK","counterDisplayName":"SHUFFLE_BYTES_TO_DISK","counterValue":0},{"counterName":"SHUFFLE_BYTES_DISK_DIRECT","counterDisplayName":"SHUFFLE_BYTES_DISK_DIRECT","counterValue":0},{"counterName":"NUM_MEM_TO_DISK_MERGES","counterDisplayName":"NUM_MEM_TO_DISK_MERGES","counterValue":0},{"counterName":"NUM_DISK_TO_DISK_MERGES","counterDisplayName":"NUM_DISK_TO_DISK_MERGES","counterValue":0}]},{"counterGroupName":"HIVE","counterGroupDisplayName":"HIVE","counters":[{"counterName":"CREATED_FILES","counterDisplayName":"CREATED_FILES","counterValue":1},{"counterName":"DESERIALIZE_ERRORS","counterDisplayName":"DESERIALIZE_ERRORS","counterValue":0},{"counterName":"RECORDS_IN_Map_1","counterDisplayName":"RECORDS_I
>>
>>
>> But the json response isnt coming. What am I missing here?
>>
>> Thanks,
>> Jyotsna
>>
>>
>>
>
Re: Tez not returning .json results inspite of request completing sucessfully
Posted by Jonathan Eagles <je...@gmail.com>.
Thanks for reaching out to us, Jyotsna.
Let me make sure I understand what your setup and the issue you are
experiencing. It sounds like the setup looks like this.
Browser -> Third party service "thirdeye-services" -> HiveServer2 -> Tez
Since you are showing logs and screenshots of the successful Tez Job. My
initial thought is that HiveServer2 was able to successfully and correctly
get the results from Tez. That would make me believe that either the
connection between the browser to the third party service
"thirdeye-services" isn't working (perhaps browser compatibility issue,
security issue, or service issue) or the connection between the third party
service "thirdeye-services" to HiveServer2 issue (issue retrieving results
from HiveServer2).
Since our expertise is only with the Tez portion of the setup and that the
Tez portion of the setup seems to be working, the Tez community won't be
the best at addressing this issue. My recommendation would be try the query
in a different browser as a simple test (Chrome, Firefox) checking with
third party "thirdeye-services" browser compatibility. If you have access
to the thirdparty-services logs or logs to the HiveServer2 logs those will
be helpful in debugging. When reaching out to the thirdeye-services support
or HiveServer2 please reference these logs to get the best support from the
community. The hive user list is user@hive.apache.org and I have don't have
a support contact for the thirdparty software thirdeye-services that seems
to be part of this setup.
Thanks again for reaching to the Tez Community.
Regards,
jeagles
On Mon, Mar 19, 2018 at 6:47 AM, Jyotsna G <jy...@gmail.com> wrote:
> Hi,
> We are making a rest URI request which hits Hive server 2 and translates
> in to a hive query on TEZ.
>
> Sample uri:
> slc00yeb.us.oracle.com:8989/thirdeye-services/v1.0/OSCS/quer
> y/OSCSRecords.json?COMP_FAMILY=FMW&COMP_NAME=WLS&COMPTYPE1_
> FIRST=RC1&COMPTYPE1_SECOND=RC2&COMPTYPE2_FIRST=manifest.
> publish.label.WLS_GENERIC:carb-manifest:12.2.1.3.0-
> 170722.1880245:pom&COMPTYPE2_SECOND=manifest.publish.label.
> WLS_GENERIC:carb-manifest:12.2.1.3.0-170726.1880744:pom&
> RELEASE_FIRST=12.2.1.3.0&RELEASE_SECOND=12.2.1.3.0&COMPTYPE=MANIFEST
>
> Hadoop version : 2.4.0
> Hive:1.2.2
> Tez : 5.4.0
>
> All my queries run fine in HIVE CLI , also via standalone program
> connecting to hive via JDBC ODBC connection and returns hive query results,
> but only when I hit the rest URL, results are not returned.
>
> I have enabled the below settings for tez
>
> tez.session.am.dag.submit.timeout.secs 90
>
> tez.am.mode.session : true
> tez.am.container.reuse.enable: true
> tez.allow.disabled.timeline-domains : true
> tez.yarn.ats.enabled : true
> tez.use.cluster.hadoop-libs : true
>
> The url on the browser keeps spinning without the json response, even
> after the tez job on yarn cluster has finished successfully. Attached Tez
> job diagnosis screenshot.
>
> From the dag log :
>
> 2018-03-19 11:34:36,367 INFO [AsyncDispatcher event handler] impl.DAGImpl:
> dag_1521448362533_0019_1 transitioned from RUNNING to SUCCEEDED
> 2018-03-19 11:34:36,368 INFO [AsyncDispatcher event handler]
> app.DAGAppMaster: DAG completed, dagId=dag_1521448362533_0019_1,
> dagState=SUCCEEDED
>
> {"entity":"dag_1521448362533_0019_1","entitytype":"TEZ_DAG_I
> D","events":[{"ts":1521459276335,"eventtype":"DAG_FINISHED"}
> ],"otherinfo":{"startTime":1521459269619,"endTime":
> 1521459276335,"timeTaken":6716,"status":"SUCCEEDED","
> diagnostics":"","counters":{"counterGroups":[{"counterGroup
> Name":"org.apache.tez.common.counters.DAGCounter","counterG
> roupDisplayName":"org.apache.tez.common.counters.DAGCounter
> ","counters":[{"counterName":"NUM_SUCCEEDED_TASKS","
> counterDisplayName":"NUM_SUCCEEDED_TASKS","counterValue
> ":2},{"counterName":"TOTAL_LAUNCHED_TASKS","counterDisplayName":"TOTAL_
> LAUNCHED_TASKS","counterValue":2},{"counterName":"DATA_LOCAL_TASKS","
> counterDisplayName":"DATA_LOCAL_TASKS","counterValue":1}
> ]},{"counterGroupName":"org.apache.tez.common.counters.Fil
> eSystemCounter","counterGroupDisplayName":"File System
> Counters","counters":[{"counterName":"FILE_BYTES_READ","
> counterDisplayName":"FILE_BYTES_READ","counterValue":41993},
> {"counterName":"FILE_BYTES_WRITTEN","counterDisplayName":
> "FILE_BYTES_WRITTEN","counterValue":83954},{"
> counterName":"FILE_READ_OPS","counterDisplayName":"FILE_
> READ_OPS","counterValue":0},{"counterName":"FILE_LARGE_READ_
> OPS","counterDisplayName":"FILE_LARGE_READ_OPS","counterValu
> e":0},{"counterName":"FILE_WRITE_OPS","counterDisplayName":"FILE_WRIT
> E_OPS","counterValue":0},{"counterName":"HDFS_BYTES_READ"
> ,"counterDisplayName":"HDFS_BYTES_READ","counterValue":
> 59231},{"counterName":"HDFS_BYTES_WRITTEN","counterDisplayName":"HDFS_
> BYTES_WRITTEN","counterValue":39631},{"counterName":"HDFS_
> READ_OPS","counterDisplayName":"HDFS_READ_OPS","
> counterValue":5},{"counterName":"HDFS_LARGE_READ_OPS","
> counterDisplayName":"HDFS_LARGE_READ_OPS","counterValue"
> :0},{"counterName":"HDFS_WRITE_OPS","counterDisplayName":"HDFS_WRIT
> E_OPS","counterValue":2}]},{"counterGroupName":"org.apache.
> tez.common.counters.TaskCounter","counterGroupDisplayName":"
> org.apache.tez.common.counters.TaskCounter","counters":[{"counterName":"
> REDUCE_INPUT_GROUPS","counterDisplayName":"REDUCE_
> INPUT_GROUPS","counterValue":190},{"counterName":"REDUCE_INPUT_RECORDS","
> counterDisplayName":"REDUCE_INPUT_RECORDS","counterValue":
> 191},{"counterName":"COMBINE_INPUT_RECORDS","counterDisplayName":"COMBINE_
> INPUT_RECORDS","counterValue":0},{"counterName":"SPILLED_
> RECORDS","counterDisplayName":"SPILLED_RECORDS","counterValu
> e":382},{"counterName":"NUM_SHUFFLED_INPUTS","counterDisplayName":"NUM_
> SHUFFLED_INPUTS","counterValue":1},{"counterName":"NUM_SKIPPED_INPUTS","
> counterDisplayName":"NUM_SKIPPED_INPUTS","counterValue"
> :0},{"counterName":"NUM_FAILED_SHUFFLE_INPUTS","counte
> rDisplayName":"NUM_FAILED_SHUFFLE_INPUTS","counterValue"
> :0},{"counterName":"MERGED_MAP_OUTPUTS","counterDisplayName":"MERGED_
> MAP_OUTPUTS","counterValue":1},{"counterName":"GC_TIME_
> MILLIS","counterDisplayName":"GC_TIME_MILLIS","counterValue"
> :84},{"counterName":"CPU_MILLISECONDS","counterDisplayName":
> "CPU_MILLISECONDS","counterValue":5520},{"counterName":"PHYSICAL_MEMORY_
> BYTES","counterDisplayName":"PHYSICAL_MEMORY_BYTES","
> counterValue":661254144},{"counterName":"VIRTUAL_MEMORY_
> BYTES","counterDisplayName":"VIRTUAL_MEMORY_BYTES","
> counterValue":4294270976},{"counterName":"COMMITTED_HEAP_
> BYTES","counterDisplayName":"COMMITTED_HEAP_BYTES","counterV
> alue":402653184},{"counterName":"INPUT_RECORDS_PROCESSED","c
> ounterDisplayName":"INPUT_RECORDS_PROCESSED","counterValue":
> 8445},{"counterName":"OUTPUT_RECORDS","counterDisplayName":
> "OUTPUT_RECORDS","counterValue":191},{"counterName":"OUTPUT_BYTES","c
> ounterDisplayName":"OUTPUT_BYTES","counterValue":41350},{"
> counterName":"OUTPUT_BYTES_WITH_OVERHEAD","counterDisplayNam
> e":"OUTPUT_BYTES_WITH_OVERHEAD","counterValue":
> 41957},{"counterName":"OUTPUT_BYTES_PHYSICAL","
> counterDisplayName":"OUTPUT_BYTES_PHYSICAL","counterValue"
> :41961},{"counterName":"ADDITIONAL_SPILLS_BYTES_
> WRITTEN","counterDisplayName":"ADDITIONAL_SPILLS_BYTES_WRITT
> EN","counterValue":41961},{"counterName":"ADDITIONAL_SPILL
> S_BYTES_READ","counterDisplayName":"ADDITIONAL_SPILLS_BYTES_
> READ","counterValue":41961},{"counterName":"ADDITIONAL_SPILL
> _COUNT","counterDisplayName":"ADDITIONAL_SPILL_COUNT","count
> erValue":0},{"counterName":"SHUFFLE_BYTES","counterDisplay
> Name":"SHUFFLE_BYTES","counterValue":41961},{"
> counterName":"SHUFFLE_BYTES_DECOMPRESSED","counterDisplayNam
> e":"SHUFFLE_BYTES_DECOMPRESSED","counterValue":41957},{"counterName":"
> SHUFFLE_BYTES_TO_MEM","counterDisplayName":"SHUFFLE_
> BYTES_TO_MEM","counterValue":41961},{"counterName":"
> SHUFFLE_BYTES_TO_DISK","counterDisplayName":"SHUFFLE_
> BYTES_TO_DISK","counterValue":0},{"counterName":"SHUFFLE_
> BYTES_DISK_DIRECT","counterDisplayName":"SHUFFLE_BYTES_DISK_
> DIRECT","counterValue":0},{"counterName":"NUM_MEM_TO_DISK_ME
> RGES","counterDisplayName":"NUM_MEM_TO_DISK_MERGES","counter
> Value":0},{"counterName":"NUM_DISK_TO_DISK_MERGES","counterD
> isplayName":"NUM_DISK_TO_DISK_MERGES","counterValue":0}]},{"
> counterGroupName":"HIVE","counterGroupDisplayName":"HIVE","
> counters":[{"counterName":"CREATED_FILES","counterDisplay
> Name":"CREATED_FILES","counterValue":1},{"counterName
> ":"DESERIALIZE_ERRORS","counterDisplayName":"DESERIALI
> ZE_ERRORS","counterValue":0},{"counterName":"RECORDS_IN_Map_
> 1","counterDisplayName":"RECORDS_I
>
>
> But the json response isnt coming. What am I missing here?
>
> Thanks,
> Jyotsna
>
>
>