You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Joe McDonnell (JIRA)" <ji...@apache.org> on 2019/01/10 02:23:00 UTC
[jira] [Commented] (IMPALA-8063) Excessive logging from
BeeswaxConnection::get_state() bloats JUnitXML output
[ https://issues.apache.org/jira/browse/IMPALA-8063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16738915#comment-16738915 ]
Joe McDonnell commented on IMPALA-8063:
---------------------------------------
Unless I'm missing something, wait_for_state is missing a sleep. It spins in a busy loop.
> Excessive logging from BeeswaxConnection::get_state() bloats JUnitXML output
> ----------------------------------------------------------------------------
>
> Key: IMPALA-8063
> URL: https://issues.apache.org/jira/browse/IMPALA-8063
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 3.2.0
> Reporter: Joe McDonnell
> Priority: Blocker
> Labels: broken-build
>
> BeeswaxConnection has logging for each call of get_state:
>
> {code:java}
> def get_state(self, operation_handle):
> LOG.info("-- getting state for operation: %s" % operation_handle)
> return self.__beeswax_client.get_state(operation_handle.get_handle())
> {code}
> With IMPALA-7625, ImpalaTestSuite::wait_for_state() calls this more frequently:
>
>
> {code:java}
> def wait_for_state(self, handle, expected_state, timeout):
> """Waits for the given 'query_handle' to reach the 'expected_state'. If it does not
> reach the given state within 'timeout' seconds, the method throws an AssertionError.
> """
> start_time = time.time()
> actual_state = self.client.get_state(handle)
> while actual_state != expected_state and time.time() - start_time < timeout:
> actual_state = self.client.get_state(handle)
> if actual_state != expected_state:
> raise Timeout("query '%s' did not reach expected state '%s', last known state '%s'"
> % (handle.get_handle().id, expected_state, actual_state))
> {code}
> When running our tests in exhaustive mode, that increases the size of the logging significantly. For example:
>
> {noformat}
> Before this change:
> $ ls -l TEST-impala-parallel.xml
> -rw-rw-r-- 1 joe joe 34254745 Jan 6 23:23 TEST-impala-parallel.xml
> $ grep "getting state for operation" TEST-impala-parallel.xml | wc -
> l
> 1044
> After this change:
> $ ls -l TEST-impala-parallel.xml
> -rw-rw-r-- 1 joe joe 159187682 Jan 9 14:51 TEST-impala-parallel.xml
> $ grep "getting state for operation" TEST-impala-parallel.xml | wc -
> l
> 1167084
> {noformat}
>
>
> We should reduce this logging. Bloated JUnitXML files add burden to developer workstations and any Jenkins infrastructure parsing them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org