You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Attila Magyar (Jira)" <ji...@apache.org> on 2020/06/03 13:32:00 UTC
[jira] [Comment Edited] (HIVE-23277) HiveProtoLogger should carry
out JSON conversion in its own thread
[ https://issues.apache.org/jira/browse/HIVE-23277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17124953#comment-17124953 ]
Attila Magyar edited comment on HIVE-23277 at 6/3/20, 1:31 PM:
---------------------------------------------------------------
Hey [~rajesh.balamohan], I made a patch for this, where json serialization happens on the logWriter's thread. The event is only built partially up front with a json object (not the serialized string) and the conversion happens right before writing out the event.
However events like this take up more space in memory as before. About twice as much. The queue has a default max capacity of 64 so this might not be a problem.
{code:java}
./app=hiveserver2/2020-06-03-13-05_0.log.gz:<14>1 2020-06-03T13:07:39.838Z hiveserver2-0.hiveserver2-service.compute-1591188147-6npj.svc.cluster.local hiveserver2 1 7e79dde9-4ac7-4df6-932f-1be75ec58e73 [mdc@18060 class="hooks.HiveProtoLoggingHook" level="INFO" thread="Hive Hook Proto Log Writer 0"] XXX size with serialized JSON: 392288
./app=hiveserver2/2020-06-03-13-05_0.log.gz:<14>1 2020-06-03T13:07:39.833Z hiveserver2-0.hiveserver2-service.compute-1591188147-6npj.svc.cluster.local hiveserver2 1 7e79dde9-4ac7-4df6-932f-1be75ec58e73 [mdc@18060 class="hooks.HiveProtoLoggingHook" level="INFO" thread="Hive Hook Proto Log Writer 0"] XXX with JSON object: 779536{code}
How significant do you think the speed improvements is? Is it worth it? Based on my own measurements the JSON serialization wasn't that slow with the queries I used (about 10-15 ms).
was (Author: amagyar):
Hey [~rajesh.balamohan], I made a patch for this, where json serialization happens on the logWriter's thread. The event is only built partially up front with a json object (not the serialized string) and the conversion happens right before writing out the event.
However events like this takes up more space in memory as before. About twice as much. The queue has a default max capacity of 64 so this might not be a problem.
{code:java}
./app=hiveserver2/2020-06-03-13-05_0.log.gz:<14>1 2020-06-03T13:07:39.838Z hiveserver2-0.hiveserver2-service.compute-1591188147-6npj.svc.cluster.local hiveserver2 1 7e79dde9-4ac7-4df6-932f-1be75ec58e73 [mdc@18060 class="hooks.HiveProtoLoggingHook" level="INFO" thread="Hive Hook Proto Log Writer 0"] XXX size with serialized JSON: 392288
./app=hiveserver2/2020-06-03-13-05_0.log.gz:<14>1 2020-06-03T13:07:39.833Z hiveserver2-0.hiveserver2-service.compute-1591188147-6npj.svc.cluster.local hiveserver2 1 7e79dde9-4ac7-4df6-932f-1be75ec58e73 [mdc@18060 class="hooks.HiveProtoLoggingHook" level="INFO" thread="Hive Hook Proto Log Writer 0"] XXX with JSON object: 779536{code}
> HiveProtoLogger should carry out JSON conversion in its own thread
> ------------------------------------------------------------------
>
> Key: HIVE-23277
> URL: https://issues.apache.org/jira/browse/HIVE-23277
> Project: Hive
> Issue Type: Improvement
> Reporter: Rajesh Balamohan
> Assignee: Attila Magyar
> Priority: Minor
> Attachments: HIVE-23277.1.patch, Screenshot 2020-04-23 at 11.27.42 AM.png
>
>
> !Screenshot 2020-04-23 at 11.27.42 AM.png|width=623,height=423!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)