[jira] [Created] (AURORA-513) Errors during thrift RPC calls cause ugly stack dumps

["Mark Chu-Carroll (JIRA)" <ji...@apache.org>]

Mark Chu-Carroll created AURORA-513:
---------------------------------------

             Summary: Errors during thrift RPC calls cause ugly stack dumps
                 Key: AURORA-513
                 URL: https://issues.apache.org/jira/browse/AURORA-513
             Project: Aurora
          Issue Type: Bug
          Components: Client
            Reporter: Mark Chu-Carroll
            Assignee: Mark Chu-Carroll


Anonymized error message below. The user did an update; during the course of the update, what appears to be a transient network glitch caused a thrift socket to become None, which led to an invalid attempt to dereference None, which caused a stack dump. The actual error is in thrift-generated code.

 INFO] Examining instances: [360, 361, 362, 363, 364, 365]
 INFO] Adding instances: [360, 361, 362, 363, 364, 365]
 INFO] Instances added
 INFO] Watching instances: [360, 361, 362, 363, 364, 365]
Traceback (most recent call last):
  File "...aurora-client/twitter/common/app/application.py", line 738, in _wrap_method
    return_code = method()
  File "...aurora-client/twitter/common/app/application.py", line 760, in <lambda>
    main = lambda: main_method(*args, **kwargs)
  File "...aurora-client/.deps/apache.aurora.clientv2-0.5.1_DEV1401741199-py2.6.egg/apache/aurora/client/base.py", line 72, in wrapped_function
    return fn(*args)
  File "...aurora-client/.deps/apache.aurora.clientv2-0.5.1_DEV1401741199-py2.6.egg/apache/aurora/client/commands/core.py", line 614, in update
    resp = api.update_job(config, options.health_check_interval_seconds, options.shards)
  File "...aurora-client/.deps/apache.aurora.clientv2-0.5.1_DEV1401741199-py2.6.egg/apache/aurora/client/hooks/hooked_api.py", line 184, in update_job
    config, health_check_interval_seconds=health_check_interval_seconds, instances=instances))
  File "...aurora-client/.deps/apache.aurora.clientv2-0.5.1_DEV1401741199-py2.6.egg/apache/aurora/client/hooks/hooked_api.py", line 145, in _hooked_call
    resp = api_call()
  File "...aurora-client/.deps/apache.aurora.clientv2-0.5.1_DEV1401741199-py2.6.egg/apache/aurora/client/api/__init__.py", line 128, in update_job
    return updater.update(instances)
  File "...aurora-client/.deps/apache.aurora.clientv2-0.5.1_DEV1401741199-py2.6.egg/apache/aurora/client/api/updater.py", line 455, in update
    if not self._update(instance_configs):
  File "...aurora-client/.deps/apache.aurora.clientv2-0.5.1_DEV1401741199-py2.6.egg/apache/aurora/client/api/updater.py", line 143, in _update
    failed_instances = self._watcher.watch(instances_to_watch) if instances_to_watch else set()
  File "...aurora-client/.deps/apache.aurora.clientv2-0.5.1_DEV1401741199-py2.6.egg/apache/aurora/client/api/instance_watcher.py", line 95, in watch
    running_tasks = self._get_tasks_by_instance_id(instance_ids)
  File "...aurora-client/.deps/apache.aurora.clientv2-0.5.1_DEV1401741199-py2.6.egg/apache/aurora/client/api/instance_watcher.py", line 134, in _get_tasks_by_instance_id
    resp = self._scheduler.getTasksStatus(query)
  File "...aurora-client/.deps/apache.aurora.clientv2-0.5.1_DEV1401741199-py2.6.egg/apache/aurora/client/api/scheduler_client.py", line 264, in method_wrapper
    return method(*(args + auth_args))
  File "...aurora-client/.deps/apache.gen.aurora-0.5.1_DEV1401741199-py2.6.egg/gen/apache/aurora/api/ReadOnlyScheduler.py", line 129, in getTasksStatus
    return self.recv_getTasksStatus()
  File "...aurora-client/.deps/apache.gen.aurora-0.5.1_DEV1401741199-py2.6.egg/gen/apache/aurora/api/ReadOnlyScheduler.py", line 140, in recv_getTasksStatus
    (fname, mtype, rseqid) = self._iprot.readMessageBegin()
  File "...aurora-client/.deps/thrift-0.9.1-py2.6-macosx-10.4-x86_64.egg/thrift/protocol/TJSONProtocol.py", line 317, in readMessageBegin
    self.readJSONArrayStart()
  File "...aurora-client/.deps/thrift-0.9.1-py2.6-macosx-10.4-x86_64.egg/thrift/protocol/TJSONProtocol.py", line 305, in readJSONArrayStart
    self.readJSONSyntaxChar(LBRACKET)
  File "...aurora-client/.deps/thrift-0.9.1-py2.6-macosx-10.4-x86_64.egg/thrift/protocol/TJSONProtocol.py", line 210, in readJSONSyntaxChar
    current = self.reader.read()
  File "...aurora-client/.deps/thrift-0.9.1-py2.6-macosx-10.4-x86_64.egg/thrift/protocol/TJSONProtocol.py", line 139, in read
    self.data = self.protocol.trans.read(1)
  File "...aurora-client/.deps/thrift-0.9.1-py2.6-macosx-10.4-x86_64.egg/thrift/transport/THttpClient.py", line 97, in read
    return self.__http.file.read(sz)
AttributeError: 'NoneType' object has no attribute 'read'




--
This message was sent by Atlassian JIRA
(v6.2#6252)