[jira] [Created] (KUDU-2830) Improve RPC traces on RPC timeout

["Grant Henke (JIRA)" <ji...@apache.org>]

Grant Henke created KUDU-2830:
---------------------------------

             Summary: Improve RPC traces on RPC timeout
                 Key: KUDU-2830
                 URL: https://issues.apache.org/jira/browse/KUDU-2830
             Project: Kudu
          Issue Type: Improvement
            Reporter: Grant Henke


When RPCs are retried repeatedly and then eventually a timeout is caused, the logged RPC trace includes the entire nested RPC call stack. This is often way to much to read and usually includes a lot of repeated information. Perhaps we could use various levels of detail and summarization based on the log level to make things more useful. 

Below is a sample RPC trace:
{noformat}
19/01/22 06:01:56 WARN scheduler.TaskSetManager: Lost task 1125.0 in stage 1.0 (TID 1152, vc1308.halxg.cloudera.com, executor 9): java.lang.RuntimeException: failed to write 516 rows from DataFrame to Kudu; sample errors: Timed out: can not complete before timeout: Batch{operations=516, tablet="c31055f8d3f74ee3b52687e7aff5b68a" [0x0000000300000003, <end>), ignoreAllDuplicateRows=false, rpc=KuduRpc(method=Write, tablet=c31055f8d3f74ee3b52687e7aff5b68a, attempt=35, DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms] querying master, [0ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [60ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [60ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [97ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [97ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [119ms] querying master, [119ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [120ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [120ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [160ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [160ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [218ms] querying master, [218ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [218ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [219ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [262ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [262ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [479ms] querying master, [479ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [480ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [516ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [516ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [719ms] querying master, [719ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [720ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [720ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [758ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [758ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [879ms] querying master, [879ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [879ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [879ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [916ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [916ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [1060ms] querying master, [1060ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [1060ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [1060ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of capacity), [1103ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of capacity), [4480ms] querying master, [4480ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [4480ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [4480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of capacity), [4524ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of capacity), [7020ms] querying master, [7020ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [7020ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [7020ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of capacity), [7061ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of capacity), [9820ms] querying master, [9820ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [9821ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [9821ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of capacity), [9860ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of capacity), [13100ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.63% of capacity), [13138ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.63% of capacity), [16920ms] querying master, [16920ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [16920ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [16920ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of capacity), [16957ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of capacity), [18959ms] querying master, [18959ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [18960ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [18960ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of capacity), [19002ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of capacity), [22439ms] querying master, [22439ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [22440ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [22440ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of capacity), [22476ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of capacity), [24520ms] querying master, [24520ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [24520ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [24520ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of capacity), [24562ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of capacity), [25079ms] querying master, [25079ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [25080ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [25080ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of capacity), [25117ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of capacity), [28739ms] querying master, [28740ms] trace too long, truncated)}Timed out: can not complete before timeout: Batch{operations=516, tablet="c31055f8d3f74ee3b52687e7aff5b68a" [0x0000000300000003, <end>), ignoreAllDuplicateRows=false, rpc=KuduRpc(method=Write, tablet=c31055f8d3f74ee3b52687e7aff5b68a, attempt=35, DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms] querying master, [0ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [60ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [60ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [97ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [97ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [119ms] querying master, [119ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [120ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [120ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [160ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [160ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [218ms] querying master, [218ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [218ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [219ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [262ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [262ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [479ms] querying master, [479ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [480ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [516ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [516ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [719ms] querying master, [719ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [720ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [720ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [758ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [758ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [879ms] querying master, [879ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [879ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [879ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [916ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [916ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [1060ms] querying master, [1060ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [1060ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [1060ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of capacity), [1103ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of capacity), [4480ms] querying master, [4480ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [4480ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [4480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of capacity), [4524ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of capacity), [7020ms] querying master, [7020ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [7020ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [7020ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of capacity), [7061ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of capacity), [9820ms] querying master, [9820ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [9821ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [9821ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of capacity), [9860ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of capacity), [13100ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.63% of capacity), [13138ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.63% of capacity), [16920ms] querying master, [16920ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [16920ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [16920ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of capacity), [16957ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of capacity), [18959ms] querying master, [18959ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [18960ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [18960ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of capacity), [19002ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of capacity), [22439ms] querying master, [22439ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [22440ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [22440ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of capacity), [22476ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of capacity), [24520ms] querying master, [24520ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [24520ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [24520ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of capacity), [24562ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of capacity), [25079ms] querying master, [25079ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [25080ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [25080ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of capacity), [25117ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of capacity), [28739ms] querying master, [28740ms] trace too long, truncated)}Timed out: can not complete before timeout: Batch{operations=516, tablet="c31055f8d3f74ee3b52687e7aff5b68a" [0x0000000300000003, <end>), ignoreAllDuplicateRows=false, rpc=KuduRpc(method=Write, tablet=c31055f8d3f74ee3b52687e7aff5b68a, attempt=35, DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms] querying master, [0ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [60ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [60ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [97ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [97ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [119ms] querying master, [119ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [120ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [120ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [160ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [160ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [218ms] querying master, [218ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [218ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [219ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [262ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [262ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [479ms] querying master, [479ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [480ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [516ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [516ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [719ms] querying master, [719ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [720ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [720ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [758ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [758ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [879ms] querying master, [879ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [879ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [879ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [916ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [916ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [1060ms] querying master, [1060ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [1060ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [1060ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of capacity), [1103ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of capacity), [4480ms] querying master, [4480ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [4480ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [4480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of capacity), [4524ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of capacity), [7020ms] querying master, [7020ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [7020ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [7020ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of capacity), [7061ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of capacity), [9820ms] querying master, [9820ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [9821ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [9821ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of capacity), [9860ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of capacity), [13100ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.63% of capacity), [13138ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.63% of capacity), [16920ms] querying master, [16920ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [16920ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [16920ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of capacity), [16957ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of capacity), [18959ms] querying master, [18959ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [18960ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [18960ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of capacity), [19002ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of capacity), [22439ms] querying master, [22439ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [22440ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [22440ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of capacity), [22476ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of capacity), [24520ms] querying master, [24520ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [24520ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [24520ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of capacity), [24562ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of capacity), [25079ms] querying master, [25079ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [25080ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [25080ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of capacity), [25117ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of capacity), [28739ms] querying master, [28740ms] trace too long, truncated)}Timed out: can not complete before timeout: Batch{operations=516, tablet="c31055f8d3f74ee3b52687e7aff5b68a" [0x0000000300000003, <end>), ignoreAllDuplicateRows=false, rpc=KuduRpc(method=Write, tablet=c31055f8d3f74ee3b52687e7aff5b68a, attempt=35, DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms] querying master, [0ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [60ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [60ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [97ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [97ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [119ms] querying master, [119ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [120ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [120ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [160ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [160ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [218ms] querying master, [218ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [218ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [219ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [262ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [262ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [479ms] querying master, [479ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [480ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [516ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [516ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [719ms] querying master, [719ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [720ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [720ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [758ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [758ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [879ms] querying master, [879ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [879ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [879ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [916ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [916ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [1060ms] querying master, [1060ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [1060ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [1060ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of capacity), [1103ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of capacity), [4480ms] querying master, [4480ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [4480ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [4480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of capacity), [4524ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of capacity), [7020ms] querying master, [7020ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [7020ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [7020ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of capacity), [7061ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of capacity), [9820ms] querying master, [9820ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [9821ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [9821ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of capacity), [9860ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of capacity), [13100ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.63% of capacity), [13138ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.63% of capacity), [16920ms] querying master, [16920ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [16920ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [16920ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of capacity), [16957ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of capacity), [18959ms] querying master, [18959ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [18960ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [18960ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of capacity), [19002ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of capacity), [22439ms] querying master, [22439ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [22440ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [22440ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of capacity), [22476ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of capacity), [24520ms] querying master, [24520ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [24520ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [24520ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of capacity), [24562ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of capacity), [25079ms] querying master, [25079ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [25080ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [25080ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of capacity), [25117ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of capacity), [28739ms] querying master, [28740ms] trace too long, truncated)}Timed out: can not complete before timeout: Batch{operations=516, tablet="c31055f8d3f74ee3b52687e7aff5b68a" [0x0000000300000003, <end>), ignoreAllDuplicateRows=false, rpc=KuduRpc(method=Write, tablet=c31055f8d3f74ee3b52687e7aff5b68a, attempt=35, DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms] querying master, [0ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [60ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [60ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [97ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [97ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [119ms] querying master, [119ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [120ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [120ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [160ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [160ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [218ms] querying master, [218ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [218ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [219ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [262ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [262ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [479ms] querying master, [479ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [480ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [516ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [516ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [719ms] querying master, [719ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [720ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [720ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [758ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [758ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [879ms] querying master, [879ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [879ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [879ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [916ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [916ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of capacity), [1060ms] querying master, [1060ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [1060ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [1060ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of capacity), [1103ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of capacity), [4480ms] querying master, [4480ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [4480ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [4480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of capacity), [4524ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of capacity), [7020ms] querying master, [7020ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [7020ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [7020ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of capacity), [7061ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of capacity), [9820ms] querying master, [9820ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [9821ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [9821ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of capacity), [9860ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of capacity), [13100ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.63% of capacity), [13138ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.63% of capacity), [16920ms] querying master, [16920ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [16920ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [16920ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of capacity), [16957ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of capacity), [18959ms] querying master, [18959ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [18960ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [18960ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of capacity), [19002ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of capacity), [22439ms] querying master, [22439ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [22440ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [22440ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of capacity), [22476ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of capacity), [24520ms] querying master, [24520ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [24520ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [24520ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of capacity), [24562ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of capacity), [25079ms] querying master, [25079ms] Sub rpc: GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051, [25080ms] Sub rpc: GetTableLocations received from server master-vc1320.halxg.cloudera.com:7051 response OK, [25080ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of capacity), [25117ms] delaying RPC due to Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of capacity), [28739ms] querying master, [28740ms] trace too long, truncated)}

at org.apache.kudu.spark.kudu.KuduContext$$anonfun$writeRows$1.apply(KuduContext.scala:311)

at org.apache.kudu.spark.kudu.KuduContext$$anonfun$writeRows$1.apply(KuduContext.scala:299)

at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:929)

at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:929)

at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)

at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)

at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)

at org.apache.spark.scheduler.Task.run(Task.scala:109)

at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748){noformat}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)