You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Joe McDonnell (JIRA)" <ji...@apache.org> on 2019/01/10 02:26:00 UTC
[jira] [Updated] (IMPALA-8063) Excessive logging from
BeeswaxConnection::get_state() bloats JUnitXML output
[ https://issues.apache.org/jira/browse/IMPALA-8063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joe McDonnell updated IMPALA-8063:
----------------------------------
Description:
BeeswaxConnection has logging for each call of get_state:
{code:java}
def get_state(self, operation_handle):
LOG.info("-- getting state for operation: %s" % operation_handle)
return self.__beeswax_client.get_state(operation_handle.get_handle())
{code}
With IMPALA-7625, ImpalaTestSuite::wait_for_state() calls this more frequently:
{code:java}
def wait_for_state(self, handle, expected_state, timeout):
"""Waits for the given 'query_handle' to reach the 'expected_state'. If it does not
reach the given state within 'timeout' seconds, the method throws an AssertionError.
"""
start_time = time.time()
actual_state = self.client.get_state(handle)
while actual_state != expected_state and time.time() - start_time < timeout:
actual_state = self.client.get_state(handle)
if actual_state != expected_state:
raise Timeout("query '%s' did not reach expected state '%s', last known state '%s'"
% (handle.get_handle().id, expected_state, actual_state))
{code}
When running our tests in exhaustive mode, that increases the size of the logging significantly. For example:
{noformat}
Before this change:
$ ls -l TEST-impala-parallel.xml
-rw-rw-r-- 1 joe joe 34254745 Jan 6 23:23 TEST-impala-parallel.xml
$ grep "getting state for operation" TEST-impala-parallel.xml | wc -
l
1044
After this change:
$ ls -l TEST-impala-parallel.xml
-rw-rw-r-- 1 joe joe 159187682 Jan 9 14:51 TEST-impala-parallel.xml
$ grep "getting state for operation" TEST-impala-parallel.xml | wc -
l
1167084
{noformat}
We should reduce this logging. Bloated JUnitXML files add burden to developer workstations and any Jenkins infrastructure parsing them.
was:
BeeswaxConnection has logging for each call of get_state:
{code:java}
def get_state(self, operation_handle):
LOG.info("-- getting state for operation: %s" % operation_handle)
return self.__beeswax_client.get_state(operation_handle.get_handle())
{code}
With IMPALA-7625, ImpalaTestSuite::wait_for_state() calls this more frequently:
{code:java}
def wait_for_state(self, handle, expected_state, timeout):
"""Waits for the given 'query_handle' to reach the 'expected_state'. If it does not
reach the given state within 'timeout' seconds, the method throws an AssertionError.
"""
start_time = time.time()
actual_state = self.client.get_state(handle)
while actual_state != expected_state and time.time() - start_time < timeout:
actual_state = self.client.get_state(handle)
if actual_state != expected_state:
raise Timeout("query '%s' did not reach expected state '%s', last known state '%s'"
% (handle.get_handle().id, expected_state, actual_state))
{code}
When running our tests in exhaustive mode, that increases the size of the logging significantly. For example:
{noformat}
Before this change:
$ ls -l TEST-impala-parallel.xml
-rw-rw-r-- 1 joe joe 34254745 Jan 6 23:23 TEST-impala-parallel.xml
$ grep "getting state for operation" TEST-impala-parallel.xml | wc -
l
1044
After this change:
$ ls -l TEST-impala-parallel.xml
-rw-rw-r-- 1 joe joe 159187682 Jan 9 14:51 TEST-impala-parallel.xml
$ grep "getting state for operation" TEST-impala-parallel.xml | wc -
l
1167084
{noformat}
We should reduce this logging. Bloated JUnitXML files add burden to developer workstations and any Jenkins infrastructure parsing them.
> Excessive logging from BeeswaxConnection::get_state() bloats JUnitXML output
> ----------------------------------------------------------------------------
>
> Key: IMPALA-8063
> URL: https://issues.apache.org/jira/browse/IMPALA-8063
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 3.2.0
> Reporter: Joe McDonnell
> Priority: Blocker
> Labels: broken-build
>
> BeeswaxConnection has logging for each call of get_state:
>
> {code:java}
> def get_state(self, operation_handle):
> LOG.info("-- getting state for operation: %s" % operation_handle)
> return self.__beeswax_client.get_state(operation_handle.get_handle())
> {code}
> With IMPALA-7625, ImpalaTestSuite::wait_for_state() calls this more frequently:
>
>
> {code:java}
> def wait_for_state(self, handle, expected_state, timeout):
> """Waits for the given 'query_handle' to reach the 'expected_state'. If it does not
> reach the given state within 'timeout' seconds, the method throws an AssertionError.
> """
> start_time = time.time()
> actual_state = self.client.get_state(handle)
> while actual_state != expected_state and time.time() - start_time < timeout:
> actual_state = self.client.get_state(handle)
> if actual_state != expected_state:
> raise Timeout("query '%s' did not reach expected state '%s', last known state '%s'"
> % (handle.get_handle().id, expected_state, actual_state))
> {code}
> When running our tests in exhaustive mode, that increases the size of the logging significantly. For example:
>
> {noformat}
> Before this change:
> $ ls -l TEST-impala-parallel.xml
> -rw-rw-r-- 1 joe joe 34254745 Jan 6 23:23 TEST-impala-parallel.xml
> $ grep "getting state for operation" TEST-impala-parallel.xml | wc -
> l
> 1044
> After this change:
> $ ls -l TEST-impala-parallel.xml
> -rw-rw-r-- 1 joe joe 159187682 Jan 9 14:51 TEST-impala-parallel.xml
> $ grep "getting state for operation" TEST-impala-parallel.xml | wc -
> l
> 1167084
> {noformat}
>
>
> We should reduce this logging. Bloated JUnitXML files add burden to developer workstations and any Jenkins infrastructure parsing them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org