You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@superset.apache.org by gi...@git.apache.org on 2017/09/11 09:06:50 UTC
[GitHub] rumbin opened a new issue #3442: [SQLLab + Celery worker] MemoryError with large result sets
rumbin opened a new issue #3442: [SQLLab + Celery worker] MemoryError with large result sets
URL: https://github.com/apache/incubator-superset/issues/3442
Make sure these boxes are checked before submitting your issue - thank you!
- [x] I have checked the superset logs for python stacktraces and included it here as text if any
- [x] I have reproduced the issue with at least the latest released version of superset
- [x] I have checked the issue tracker for the same issue and I haven't found one similar
### Version information
* 0.19.1, being compared with 0.15.4
* Backend: PostgreSQL 9.6.4.
### Expected results
Queries that run flawlessly in 0.15.4 should also succeed in 0.19.1.
The query that I take as an example takes 4-5 sec in 0.15.4, and results in a table of ~670k rows, 21 columns and ~98 MB as CSV file.
### Actual results
On 1st run of the query after restarting superset worker:
```
[2017-09-11 10:24:15,974: ERROR/MainProcess] Process 'ForkPoolWorker-7' pid:50228 exited with 'signal 9 (SIGKILL)'
```
The quere results are delivered as expected and are downloadable as CSV.
Running the same query another time right afterwards will not succeed, though:
```
[2017-09-11 10:27:58,399: ERROR/ForkPoolWorker-29] Can't allocate memory to compress data
Traceback (most recent call last):
File "/opt/apps/superset/venv_py3_superset0.19.1/lib64/python3.4/site-packages/superset/sql_lab.py", line 91, in get_sql_results
ctask, query_id, return_results, store_results)
File "/opt/apps/superset/venv_py3_superset0.19.1/lib64/python3.4/site-packages/superset/sql_lab.py", line 240, in execute_sql
results_backend.set(key, utils.zlib_compress(json_payload))
File "/opt/apps/superset/venv_py3_superset0.19.1/lib64/python3.4/site-packages/superset/utils.py", line 633, in zlib_compress
return zlib.compress(bytes(data, "utf-8"))
MemoryError: Can't allocate memory to compress data
```
SQLLab still thinks that the query is running. Cancelling the query and re-running it results in this output of the worker console:
```
[2017-09-11 10:30:19,056: ERROR/MainProcess] Process 'ForkPoolWorker-29' pid:50252 exited with 'signal 9 (SIGKILL)'
[2017-09-11 10:30:19,564: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 9 (SIGKILL).',)
Traceback (most recent call last):
File "/opt/apps/superset/venv_py3_superset0.19.1/lib64/python3.4/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
human_status(exitcode)),
billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL).
### Note: SQLLab now shows "Query was stopped"
Cancelling [2017-09-11 10:31:25,333: ERROR/ForkPoolWorker-6] Can't allocate memory to compress data
Traceback (most recent call last):
File "/opt/apps/superset/venv_py3_superset0.19.1/lib64/python3.4/site-packages/superset/sql_lab.py", line 91, in get_sql_results
ctask, query_id, return_results, store_results)
File "/opt/apps/superset/venv_py3_superset0.19.1/lib64/python3.4/site-packages/superset/sql_lab.py", line 240, in execute_sql
results_backend.set(key, utils.zlib_compress(json_payload))
File "/opt/apps/superset/venv_py3_superset0.19.1/lib64/python3.4/site-packages/superset/utils.py", line 633, in zlib_compress
return zlib.compress(bytes(data, "utf-8"))
MemoryError: Can't allocate memory to compress data
```
Cancelling the _superset worker_ process requires hitting Ctrl-C three times then?
### Steps to reproduce
Run a query that results in a table that returns a large table (here: 21 cols, 670k rows, 98 MB) repeatedly.
### Console output
The console logs I present here is quite lengthy. In the _superset worker_ output, I added some comments (marked with "###") with rough timestamps in order to show what my interaction was.l
**superset worker**
```
-------------- celery@XXX v4.1.0 (latentcall)
---- **** -----
--- * *** * -- Linux-3.12.59-60.45-default-x86_64-with-SuSE-12-x86_64 2017-09-11 10:10:14
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: __main__:0x7fb2254bb358
- ** ---------- .> transport: redis://localhost:6379/0
- ** ---------- .> results: redis://localhost:6379/0
- *** --- * --- .> concurrency: 32 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[2017-09-11 10:24:15,974: ERROR/MainProcess] Process 'ForkPoolWorker-7' pid:50228 exited with 'signal 9 (SIGKILL)'
### result arrived at 10:24:20
### requesting CSV at 10:25:40
### downloading CVS at 10:26:10
### then re-running query
[2017-09-11 10:27:58,399: ERROR/ForkPoolWorker-29] Can't allocate memory to compress data
Traceback (most recent call last):
File "/opt/apps/superset/venv_py3_superset0.19.1/lib64/python3.4/site-packages/superset/sql_lab.py", line 91, in get_sql_results
ctask, query_id, return_results, store_results)
File "/opt/apps/superset/venv_py3_superset0.19.1/lib64/python3.4/site-packages/superset/sql_lab.py", line 240, in execute_sql
results_backend.set(key, utils.zlib_compress(json_payload))
File "/opt/apps/superset/venv_py3_superset0.19.1/lib64/python3.4/site-packages/superset/utils.py", line 633, in zlib_compress
return zlib.compress(bytes(data, "utf-8"))
MemoryError: Can't allocate memory to compress data
### SQLLab still claims that the query is running
### Cancelling query at 10:29:40
### Re-running it at 10:30:00
[2017-09-11 10:30:19,056: ERROR/MainProcess] Process 'ForkPoolWorker-29' pid:50252 exited with 'signal 9 (SIGKILL)'
[2017-09-11 10:30:19,564: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 9 (SIGKILL).',)
Traceback (most recent call last):
File "/opt/apps/superset/venv_py3_superset0.19.1/lib64/python3.4/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
human_status(exitcode)),
billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL).
### SQLLab shows "Query was stopped"
### Cancelling
[2017-09-11 10:31:25,333: ERROR/ForkPoolWorker-6] Can't allocate memory to compress data
Traceback (most recent call last):
File "/opt/apps/superset/venv_py3_superset0.19.1/lib64/python3.4/site-packages/superset/sql_lab.py", line 91, in get_sql_results
ctask, query_id, return_results, store_results)
File "/opt/apps/superset/venv_py3_superset0.19.1/lib64/python3.4/site-packages/superset/sql_lab.py", line 240, in execute_sql
results_backend.set(key, utils.zlib_compress(json_payload))
File "/opt/apps/superset/venv_py3_superset0.19.1/lib64/python3.4/site-packages/superset/utils.py", line 633, in zlib_compress
return zlib.compress(bytes(data, "utf-8"))
MemoryError: Can't allocate memory to compress data
### Cancelling query at 10:31:40
### Hitting Ctrl-C once
### ^C
worker: Hitting Ctrl+C again will terminate all running tasks!
worker: Warm shutdown (MainProcess)
### Hitting Ctrl-C again
### ^C
worker: Cold shutdown (MainProcess)
### once more
### ^C
worker: Cold shutdown (MainProcess)
[2017-09-11 10:32:50,396: WARNING/MainProcess] Error in atexit._run_exitfuncs:
[2017-09-11 10:32:50,414: WARNING/MainProcess] Traceback (most recent call last):
[2017-09-11 10:32:50,415: WARNING/MainProcess] File "/usr/lib64/python3.4/multiprocessing/util.py", line 252, in _run_finalizers
[2017-09-11 10:32:50,455: WARNING/MainProcess] finalizer()
[2017-09-11 10:32:50,455: WARNING/MainProcess] File "/usr/lib64/python3.4/multiprocessing/util.py", line 185, in __call__
[2017-09-11 10:32:50,455: WARNING/MainProcess] res = self._callback(*self._args, **self._kwargs)
[2017-09-11 10:32:50,456: WARNING/MainProcess] File "/opt/apps/superset/venv_py3_superset0.19.1/lib64/python3.4/site-packages/billiard/pool.py", line 1650, in _terminate_pool
[2017-09-11 10:32:50,458: WARNING/MainProcess] p.join()
[2017-09-11 10:32:50,458: WARNING/MainProcess] File "/opt/apps/superset/venv_py3_superset0.19.1/lib64/python3.4/site-packages/billiard/process.py", line 148, in join
[2017-09-11 10:32:50,466: WARNING/MainProcess] res = self._popen.wait(timeout)
[2017-09-11 10:32:50,467: WARNING/MainProcess] File "/opt/apps/superset/venv_py3_superset0.19.1/lib64/python3.4/site-packages/billiard/popen_fork.py", line 57, in wait
[2017-09-11 10:32:50,468: WARNING/MainProcess] return self.poll(os.WNOHANG if timeout == 0.0 else 0)
[2017-09-11 10:32:50,468: WARNING/MainProcess] File "/opt/apps/superset/venv_py3_superset0.19.1/lib64/python3.4/site-packages/billiard/popen_fork.py", line 33, in poll
[2017-09-11 10:32:50,468: WARNING/MainProcess] pid, sts = os.waitpid(self.pid, flag)
[2017-09-11 10:32:50,468: WARNING/MainProcess] File "/opt/apps/superset/venv_py3_superset0.19.1/lib64/python3.4/site-packages/celery/apps/worker.py", line 288, in _handle_request
[2017-09-11 10:32:50,479: WARNING/MainProcess] raise exc(exitcode)
[2017-09-11 10:32:50,481: WARNING/MainProcess] celery.exceptions
[2017-09-11 10:32:50,481: WARNING/MainProcess] .
[2017-09-11 10:32:50,481: WARNING/MainProcess] WorkerTerminate
[2017-09-11 10:32:50,481: WARNING/MainProcess] :
[2017-09-11 10:32:50,481: WARNING/MainProcess] 1
```
**superset runserver**
```
2017-09-11 10:22:53,100:INFO:root:[stats_logger] (incr) sql_json
2017-09-11 10:22:53,102:INFO:root:Parsing with sqlparse statement select
XXX
2017-09-11 10:22:53,123:INFO:root:Triggering query_id: 1635
2017-09-11 10:22:53,123:INFO:root:Running query on a Celery worker
2017-09-11 10:22:53,551:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:22:55,552:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:22:57,636:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:22:59,804:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:01,560:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:04,069:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:05,550:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:07,930:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:09,550:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:11,742:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:13,733:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:15,552:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:17,999:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:19,552:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:21,556:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:23,563:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:25,554:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:27,552:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:29,557:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:31,555:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:33,550:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:35,548:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:37,551:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:39,551:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:41,551:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:43,555:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:45,551:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:47,550:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:49,549:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:51,552:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:53,551:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:55,551:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:57,548:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:23:59,549:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:24:01,549:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:24:03,549:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:24:05,558:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:24:07,557:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:24:09,550:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:24:09,831:INFO:root:[stats_logger] (incr) results
2017-09-11 10:24:09,899:INFO:root:Parsing with sqlparse statement select
XXX
2017-09-11 10:24:11,581:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:25:43,703:INFO:root:[stats_logger] (incr) csv
2017-09-11 10:25:43,740:INFO:root:Exporting CSV file [BJ7k_675b]
2017-09-11 10:25:43,750:INFO:root:Parsing with sqlparse statement select
XXX
2017-09-11 10:25:43,763:INFO:root:Fetching CSV from results backend [bc2a2585-5c96-48df-8d1e-dcfe3b7030d9]
2017-09-11 10:25:43,786:INFO:root:Decompressing
2017-09-11 10:25:58,466:INFO:root:Using pandas to convert to CSV
2017-09-11 10:26:06,166:INFO:root:Ready to return response
2017-09-11 10:26:40,600:INFO:root:[stats_logger] (incr) sql_json
2017-09-11 10:26:40,695:INFO:root:Parsing with sqlparse statement select
XXX
2017-09-11 10:26:41,351:INFO:root:Triggering query_id: 1636
2017-09-11 10:26:41,351:INFO:root:Running query on a Celery worker
2017-09-11 10:26:42,417:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:26:43,561:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:26:45,559:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:26:47,555:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:26:49,557:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:26:51,567:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:26:53,556:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:26:55,632:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:26:57,557:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:26:59,558:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:01,619:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:03,648:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:05,608:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:07,567:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:09,562:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:11,563:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:13,579:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:15,559:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:17,572:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:19,561:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:21,574:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:23,555:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:25,559:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:27,556:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:29,559:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:31,555:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:33,559:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:35,556:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:37,554:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:39,554:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:41,555:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:43,554:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:45,563:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:47,555:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:49,556:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:51,561:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:53,558:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:55,565:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:57,557:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:27:59,556:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:01,559:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:03,555:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:05,556:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:07,560:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:09,558:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:11,553:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:13,557:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:15,557:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:17,556:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:19,557:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:21,558:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:23,558:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:25,557:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:27,556:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:29,556:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:31,558:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:33,555:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:35,555:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:37,557:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:39,558:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:41,556:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:43,555:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:45,555:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:47,562:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:49,555:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:51,557:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:53,556:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:55,555:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:57,559:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:28:59,563:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:29:01,557:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:29:03,557:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:29:05,556:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:29:07,556:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:29:09,556:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:29:11,556:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:29:13,559:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:29:15,560:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:29:17,557:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:29:19,556:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:29:21,556:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:29:23,560:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:29:25,556:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:29:27,557:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:29:29,556:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:29:31,556:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:29:33,556:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:29:35,563:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:29:37,559:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:29:39,557:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:29:41,130:INFO:root:[stats_logger] (incr) stop_query
2017-09-11 10:30:00,563:INFO:root:[stats_logger] (incr) sql_json
2017-09-11 10:30:00,615:INFO:root:Parsing with sqlparse statement select
XXX
2017-09-11 10:30:01,406:INFO:root:Triggering query_id: 1637
2017-09-11 10:30:01,406:INFO:root:Running query on a Celery worker
2017-09-11 10:30:01,567:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:03,562:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:05,558:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:07,558:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:09,591:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:16,384:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:18,988:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:19,005:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:19,022:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:19,557:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:21,559:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:23,558:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:25,557:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:27,560:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:29,558:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:31,560:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:33,559:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:35,558:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:37,559:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:39,558:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:41,560:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:43,559:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:45,564:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:47,563:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:49,561:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:51,559:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:53,560:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:55,558:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:57,559:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:30:59,559:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:31:01,564:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:31:03,559:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:31:05,559:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:31:07,558:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:31:09,558:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:31:11,575:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:31:13,558:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:31:15,558:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:31:17,561:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:31:19,860:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:31:21,558:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:31:23,571:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:31:25,563:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:31:27,558:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:31:29,557:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:31:31,561:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:31:33,558:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:31:35,567:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:31:37,559:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:31:39,559:INFO:root:[stats_logger] (incr) queries
2017-09-11 10:31:41,032:INFO:root:[stats_logger] (incr) stop_query
[2017-09-11 10:36:41 +0200] [50103] [CRITICAL] WORKER TIMEOUT (pid:50207)
2017-09-11 10:36:42,610:INFO:root:[stats_logger] (incr) stop_query
[2017-09-11 10:36:42 +0200] [61025] [INFO] Booting worker with pid: 61025
Loaded your LOCAL configuration at [/opt/apps/config/superset_config.py]
/usr/lib64/python3.4/importlib/_bootstrap.py:1161: ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
spec.loader.load_module(spec.name)
2017-09-11 10:40:20,182:WARNING:root:Access is Denied for: can_sqllab on: Superset
[2017-09-11 10:41:42 +0200] [50103] [CRITICAL] WORKER TIMEOUT (pid:50206)
[2017-09-11 10:41:43 +0200] [62654] [INFO] Booting worker with pid: 62654
Loaded your LOCAL configuration at [/opt/apps/config/superset_config.py]
/usr/lib64/python3.4/importlib/_bootstrap.py:1161: ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
spec.loader.load_module(spec.name)
[2017-09-11 10:56:19 +0200] [50103] [INFO] Handling signal: winch
```
**memory usage**
```
prodb@ideweiiss8068:~> paste <(date) <(free | head -1)
Mon Sep 11 10:22:30 CEST 2017 total used free shared buffers cached
prodb@ideweiiss8068:~> while true; do paste <(date) <(free | grep Mem); sleep 2; done
Mon Sep 11 10:22:51 CEST 2017 Mem: 2896136 2170628 725508 3524 13320 188572
Mon Sep 11 10:22:53 CEST 2017 Mem: 2896136 2184668 711468 3524 13328 188560
Mon Sep 11 10:22:55 CEST 2017 Mem: 2896136 2438628 457508 3524 13328 188628
Mon Sep 11 10:22:57 CEST 2017 Mem: 2896136 2788240 107896 3524 152 41276
Mon Sep 11 10:22:59 CEST 2017 Mem: 2896136 2782688 113448 0 200 53692
Mon Sep 11 10:23:01 CEST 2017 Mem: 2896136 2795844 100292 0 128 48712
Mon Sep 11 10:23:03 CEST 2017 Mem: 2896136 2689532 206604 1772 556 115264
Mon Sep 11 10:23:05 CEST 2017 Mem: 2896136 2788908 107228 1772 600 143276
Mon Sep 11 10:23:07 CEST 2017 Mem: 2896136 2788264 107872 1772 196 101092
Mon Sep 11 10:23:09 CEST 2017 Mem: 2896136 2764632 131504 1772 316 136308
Mon Sep 11 10:23:11 CEST 2017 Mem: 2896136 2790252 105884 1772 156 115176
Mon Sep 11 10:23:14 CEST 2017 Mem: 2896136 2785336 110800 1772 192 108188
Mon Sep 11 10:23:16 CEST 2017 Mem: 2896136 2779704 116432 1772 132 110612
Mon Sep 11 10:23:18 CEST 2017 Mem: 2896136 2741060 155076 1772 192 120840
Mon Sep 11 10:23:20 CEST 2017 Mem: 2896136 2756328 139808 1772 152 128324
Mon Sep 11 10:23:22 CEST 2017 Mem: 2896136 2777660 118476 1808 352 136104
Mon Sep 11 10:23:24 CEST 2017 Mem: 2896136 2794356 101780 1808 140 121536
Mon Sep 11 10:23:26 CEST 2017 Mem: 2896136 2740316 155820 444 160 95668
Mon Sep 11 10:23:28 CEST 2017 Mem: 2896136 2785252 110884 436 180 86672
Mon Sep 11 10:23:30 CEST 2017 Mem: 2896136 2126052 770084 436 372 115188
Mon Sep 11 10:23:32 CEST 2017 Mem: 2896136 2237540 658596 436 372 116788
Mon Sep 11 10:23:34 CEST 2017 Mem: 2896136 2256388 639748 436 392 117552
Mon Sep 11 10:23:36 CEST 2017 Mem: 2896136 2279624 616512 436 396 117560
Mon Sep 11 10:23:38 CEST 2017 Mem: 2896136 2301124 595012 436 416 119516
Mon Sep 11 10:23:40 CEST 2017 Mem: 2896136 2377732 518404 436 416 119388
Mon Sep 11 10:23:42 CEST 2017 Mem: 2896136 2442596 453540 436 420 119560
Mon Sep 11 10:23:44 CEST 2017 Mem: 2896136 2540368 355768 436 460 134640
Mon Sep 11 10:23:46 CEST 2017 Mem: 2896136 2625352 270784 436 464 135144
Mon Sep 11 10:23:48 CEST 2017 Mem: 2896136 2685484 210652 436 464 135144
Mon Sep 11 10:23:50 CEST 2017 Mem: 2896136 2770244 125892 436 476 136496
Mon Sep 11 10:23:52 CEST 2017 Mem: 2896136 2528388 367748 436 300 96736
Mon Sep 11 10:23:54 CEST 2017 Mem: 2896136 2574440 321696 436 304 100116
Mon Sep 11 10:23:56 CEST 2017 Mem: 2896136 2625604 270532 436 312 100812
Mon Sep 11 10:23:58 CEST 2017 Mem: 2896136 2679456 216680 436 316 103552
Mon Sep 11 10:24:00 CEST 2017 Mem: 2896136 2730836 165300 436 320 105184
Mon Sep 11 10:24:02 CEST 2017 Mem: 2896136 2794928 101208 1888 588 118644
Mon Sep 11 10:24:04 CEST 2017 Mem: 2896136 2787948 108188 1888 168 98616
Mon Sep 11 10:24:06 CEST 2017 Mem: 2896136 2778308 117828 216 224 71288
Mon Sep 11 10:24:08 CEST 2017 Mem: 2896136 2482512 413624 216 284 72440
Mon Sep 11 10:24:11 CEST 2017 Mem: 2896136 2600844 295292 184 172 88588
Mon Sep 11 10:24:13 CEST 2017 Mem: 2896136 2771312 124824 16 148 70892
Mon Sep 11 10:24:16 CEST 2017 Mem: 2896136 1238952 1657184 1136 2108 69632
Mon Sep 11 10:24:18 CEST 2017 Mem: 2896136 985384 1910752 1156 3144 161044
Mon Sep 11 10:24:20 CEST 2017 Mem: 2896136 998996 1897140 1156 3300 163852
Mon Sep 11 10:24:22 CEST 2017 Mem: 2896136 1009600 1886536 1280 4236 174356
Mon Sep 11 10:24:24 CEST 2017 Mem: 2896136 1009812 1886324 1268 4256 174696
Mon Sep 11 10:24:26 CEST 2017 Mem: 2896136 1009732 1886404 1268 4256 174696
Mon Sep 11 10:24:28 CEST 2017 Mem: 2896136 1009452 1886684 1268 4264 174648
Mon Sep 11 10:24:30 CEST 2017 Mem: 2896136 1009404 1886732 1268 4412 175000
Mon Sep 11 10:24:32 CEST 2017 Mem: 2896136 1009736 1886400 1268 4412 174944
Mon Sep 11 10:24:34 CEST 2017 Mem: 2896136 1010228 1885908 1268 4424 175304
Mon Sep 11 10:24:36 CEST 2017 Mem: 2896136 1010444 1885692 1268 4428 175396
Mon Sep 11 10:24:38 CEST 2017 Mem: 2896136 1010252 1885884 1272 4436 175688
Mon Sep 11 10:24:40 CEST 2017 Mem: 2896136 1010288 1885848 1272 4436 175720
Mon Sep 11 10:24:42 CEST 2017 Mem: 2896136 1010228 1885908 1272 4436 175712
Mon Sep 11 10:24:44 CEST 2017 Mem: 2896136 1010472 1885664 1272 4436 175708
Mon Sep 11 10:24:46 CEST 2017 Mem: 2896136 1010760 1885376 1272 4440 176160
Mon Sep 11 10:24:48 CEST 2017 Mem: 2896136 1010904 1885232 1272 4440 176160
Mon Sep 11 10:24:50 CEST 2017 Mem: 2896136 1011420 1884716 1272 4440 176160
Mon Sep 11 10:24:52 CEST 2017 Mem: 2896136 1011020 1885116 1272 4440 176164
Mon Sep 11 10:24:54 CEST 2017 Mem: 2896136 1011032 1885104 1272 4440 176160
Mon Sep 11 10:24:56 CEST 2017 Mem: 2896136 1011412 1884724 1272 4452 176156
Mon Sep 11 10:24:58 CEST 2017 Mem: 2896136 1010832 1885304 1272 4452 176224
Mon Sep 11 10:25:00 CEST 2017 Mem: 2896136 1011504 1884632 1272 4452 176268
Mon Sep 11 10:25:02 CEST 2017 Mem: 2896136 1021512 1874624 2276 5680 181196
Mon Sep 11 10:25:04 CEST 2017 Mem: 2896136 1021664 1874472 2308 5680 181252
Mon Sep 11 10:25:06 CEST 2017 Mem: 2896136 1035672 1860464 2308 5756 193828
Mon Sep 11 10:25:08 CEST 2017 Mem: 2896136 1054616 1841520 2308 5784 213732
Mon Sep 11 10:25:10 CEST 2017 Mem: 2896136 1054980 1841156 2308 5784 213740
Mon Sep 11 10:25:12 CEST 2017 Mem: 2896136 1054516 1841620 2308 5788 213752
Mon Sep 11 10:25:14 CEST 2017 Mem: 2896136 1054860 1841276 2308 5788 213756
Mon Sep 11 10:25:16 CEST 2017 Mem: 2896136 1057024 1839112 2432 7360 217124
Mon Sep 11 10:25:18 CEST 2017 Mem: 2896136 1056980 1839156 2500 7360 217212
Mon Sep 11 10:25:20 CEST 2017 Mem: 2896136 1057252 1838884 2500 7360 217208
Mon Sep 11 10:25:22 CEST 2017 Mem: 2896136 1056828 1839308 2504 7368 217224
Mon Sep 11 10:25:24 CEST 2017 Mem: 2896136 1056704 1839432 2528 7368 217256
Mon Sep 11 10:25:26 CEST 2017 Mem: 2896136 1056628 1839508 2528 7368 217252
Mon Sep 11 10:25:28 CEST 2017 Mem: 2896136 1056448 1839688 2528 7368 217248
Mon Sep 11 10:25:30 CEST 2017 Mem: 2896136 1056444 1839692 2528 7376 217248
Mon Sep 11 10:25:32 CEST 2017 Mem: 2896136 1056276 1839860 2528 7376 217236
Mon Sep 11 10:25:34 CEST 2017 Mem: 2896136 1056380 1839756 2528 7376 217236
Mon Sep 11 10:25:36 CEST 2017 Mem: 2896136 1056580 1839556 2528 7376 217228
Mon Sep 11 10:25:38 CEST 2017 Mem: 2896136 1056340 1839796 2528 7376 217228
Mon Sep 11 10:25:40 CEST 2017 Mem: 2896136 1056332 1839804 2528 7376 217228
Mon Sep 11 10:25:42 CEST 2017 Mem: 2896136 1056324 1839812 2528 7376 217228
Mon Sep 11 10:25:44 CEST 2017 Mem: 2896136 1063572 1832564 2528 7384 215124
Mon Sep 11 10:25:46 CEST 2017 Mem: 2896136 1076156 1819980 2528 7384 216768
Mon Sep 11 10:25:48 CEST 2017 Mem: 2896136 1089288 1806848 2528 7384 219052
Mon Sep 11 10:25:50 CEST 2017 Mem: 2896136 1102888 1793248 2528 7388 222396
Mon Sep 11 10:25:52 CEST 2017 Mem: 2896136 1134748 1761388 2528 7388 221268
Mon Sep 11 10:25:54 CEST 2017 Mem: 2896136 1465536 1430600 2528 7540 222064
Mon Sep 11 10:25:56 CEST 2017 Mem: 2896136 2034504 861632 2528 7680 224672
Mon Sep 11 10:25:58 CEST 2017 Mem: 2896136 2060168 835968 2528 7684 227336
Mon Sep 11 10:26:00 CEST 2017 Mem: 2896136 2076424 819712 2528 7692 227272
Mon Sep 11 10:26:02 CEST 2017 Mem: 2896136 2100696 795440 2640 7708 228256
Mon Sep 11 10:26:04 CEST 2017 Mem: 2896136 2126920 769216 2640 7708 228256
Mon Sep 11 10:26:06 CEST 2017 Mem: 2896136 1552528 1343608 2640 7724 228592
Mon Sep 11 10:26:08 CEST 2017 Mem: 2896136 1552796 1343340 2640 7732 228592
Mon Sep 11 10:26:10 CEST 2017 Mem: 2896136 1554008 1342128 2640 7732 228596
Mon Sep 11 10:26:12 CEST 2017 Mem: 2896136 1553908 1342228 2640 7736 228592
Mon Sep 11 10:26:14 CEST 2017 Mem: 2896136 1554008 1342128 2640 7736 228596
Mon Sep 11 10:26:16 CEST 2017 Mem: 2896136 1553860 1342276 2640 7736 228596
Mon Sep 11 10:26:18 CEST 2017 Mem: 2896136 1554264 1341872 2640 7736 228596
Mon Sep 11 10:26:20 CEST 2017 Mem: 2896136 1453052 1443084 2640 7736 228584
Mon Sep 11 10:26:22 CEST 2017 Mem: 2896136 1452956 1443180 2644 7736 228592
Mon Sep 11 10:26:24 CEST 2017 Mem: 2896136 1452768 1443368 2644 7736 228592
Mon Sep 11 10:26:26 CEST 2017 Mem: 2896136 1452856 1443280 2644 7744 228588
Mon Sep 11 10:26:28 CEST 2017 Mem: 2896136 1452928 1443208 2644 7744 228588
Mon Sep 11 10:26:30 CEST 2017 Mem: 2896136 1452656 1443480 2644 7744 228584
Mon Sep 11 10:26:32 CEST 2017 Mem: 2896136 1452796 1443340 2644 7744 228584
Mon Sep 11 10:26:34 CEST 2017 Mem: 2896136 1452568 1443568 2644 7748 229020
Mon Sep 11 10:26:36 CEST 2017 Mem: 2896136 1452720 1443416 2644 7756 229020
Mon Sep 11 10:26:38 CEST 2017 Mem: 2896136 1453256 1442880 2644 7756 229068
Mon Sep 11 10:26:40 CEST 2017 Mem: 2896136 1458456 1437680 2644 7756 228720
Mon Sep 11 10:26:42 CEST 2017 Mem: 2896136 1475628 1420508 2644 9124 231384
Mon Sep 11 10:26:44 CEST 2017 Mem: 2896136 1494480 1401656 2644 9980 233040
Mon Sep 11 10:26:46 CEST 2017 Mem: 2896136 1517792 1378344 2644 11940 242252
Mon Sep 11 10:26:48 CEST 2017 Mem: 2896136 1625488 1270648 2644 12532 246612
Mon Sep 11 10:26:50 CEST 2017 Mem: 2896136 1942596 953540 2644 12536 246716
Mon Sep 11 10:26:52 CEST 2017 Mem: 2896136 2534832 361304 2644 12536 246180
Mon Sep 11 10:26:54 CEST 2017 Mem: 2896136 2649124 247012 2644 232 140420
Mon Sep 11 10:26:56 CEST 2017 Mem: 2896136 2634916 261220 208 308 87108
Mon Sep 11 10:26:59 CEST 2017 Mem: 2896136 2715508 180628 208 348 98444
Mon Sep 11 10:27:01 CEST 2017 Mem: 2896136 2782704 113432 208 228 71652
Mon Sep 11 10:27:03 CEST 2017 Mem: 2896136 2784320 111816 1808 516 115784
Mon Sep 11 10:27:05 CEST 2017 Mem: 2896136 2791588 104548 1808 140 73764
Mon Sep 11 10:27:07 CEST 2017 Mem: 2896136 2762016 134120 1808 128 77360
Mon Sep 11 10:27:09 CEST 2017 Mem: 2896136 2787368 108768 1808 156 85032
Mon Sep 11 10:27:11 CEST 2017 Mem: 2896136 2747376 148760 1808 144 75664
Mon Sep 11 10:27:13 CEST 2017 Mem: 2896136 2787588 108548 1808 212 78136
Mon Sep 11 10:27:15 CEST 2017 Mem: 2896136 2755408 140728 1804 204 95572
Mon Sep 11 10:27:17 CEST 2017 Mem: 2896136 2741008 155128 1804 172 94692
Mon Sep 11 10:27:19 CEST 2017 Mem: 2896136 2795728 100408 1804 376 82144
Mon Sep 11 10:27:21 CEST 2017 Mem: 2896136 2069872 826264 1792 232 110668
Mon Sep 11 10:27:23 CEST 2017 Mem: 2896136 2174300 721836 1820 280 124756
Mon Sep 11 10:27:25 CEST 2017 Mem: 2896136 2264700 631436 1820 316 126448
Mon Sep 11 10:27:27 CEST 2017 Mem: 2896136 2285036 611100 1820 316 126480
Mon Sep 11 10:27:29 CEST 2017 Mem: 2896136 2308040 588096 1820 460 126828
Mon Sep 11 10:27:31 CEST 2017 Mem: 2896136 2334304 561832 1820 464 126904
Mon Sep 11 10:27:33 CEST 2017 Mem: 2896136 2417044 479092 1820 604 127152
Mon Sep 11 10:27:35 CEST 2017 Mem: 2896136 2483420 412716 1820 620 127592
Mon Sep 11 10:27:37 CEST 2017 Mem: 2896136 2566028 330108 1820 628 127800
Mon Sep 11 10:27:39 CEST 2017 Mem: 2896136 2651212 244924 1820 780 130352
Mon Sep 11 10:27:41 CEST 2017 Mem: 2896136 2719784 176352 1820 780 130352
Mon Sep 11 10:27:43 CEST 2017 Mem: 2896136 2794424 101712 1820 784 130348
Mon Sep 11 10:27:45 CEST 2017 Mem: 2896136 2546452 349684 1820 560 90468
Mon Sep 11 10:27:47 CEST 2017 Mem: 2896136 2596676 299460 1820 560 91344
Mon Sep 11 10:27:49 CEST 2017 Mem: 2896136 2649484 246652 1820 724 93236
Mon Sep 11 10:27:51 CEST 2017 Mem: 2896136 2700640 195496 1820 724 93232
Mon Sep 11 10:27:53 CEST 2017 Mem: 2896136 2751680 144456 1820 724 93228
Mon Sep 11 10:27:55 CEST 2017 Mem: 2896136 2781996 114140 1820 668 94444
Mon Sep 11 10:27:57 CEST 2017 Mem: 2896136 2813764 82372 540 672 79960
Mon Sep 11 10:28:00 CEST 2017 Mem: 2896136 2499748 396388 516 1048 86976
Mon Sep 11 10:28:02 CEST 2017 Mem: 2896136 2513824 382312 1772 1352 101020
Mon Sep 11 10:28:04 CEST 2017 Mem: 2896136 2527772 368364 1924 1520 114572
Mon Sep 11 10:28:06 CEST 2017 Mem: 2896136 2527632 368504 1924 1520 115312
Mon Sep 11 10:28:08 CEST 2017 Mem: 2896136 2567452 328684 1924 1680 154940
Mon Sep 11 10:28:10 CEST 2017 Mem: 2896136 2568020 328116 1924 1680 154936
Mon Sep 11 10:28:12 CEST 2017 Mem: 2896136 2567664 328472 1924 1680 154952
Mon Sep 11 10:28:14 CEST 2017 Mem: 2896136 2567716 328420 1924 1684 154936
Mon Sep 11 10:28:16 CEST 2017 Mem: 2896136 2567404 328732 1924 1684 154936
Mon Sep 11 10:28:18 CEST 2017 Mem: 2896136 2567396 328740 1924 1684 154912
Mon Sep 11 10:28:20 CEST 2017 Mem: 2896136 2568900 327236 1924 1696 155944
Mon Sep 11 10:28:22 CEST 2017 Mem: 2896136 2568612 327524 1924 1696 155948
Mon Sep 11 10:28:24 CEST 2017 Mem: 2896136 2570652 325484 1968 1716 158488
Mon Sep 11 10:28:26 CEST 2017 Mem: 2896136 2570768 325368 1968 1728 158476
Mon Sep 11 10:28:28 CEST 2017 Mem: 2896136 2570848 325288 1968 1728 158480
Mon Sep 11 10:28:30 CEST 2017 Mem: 2896136 2570916 325220 1968 1868 158520
Mon Sep 11 10:28:32 CEST 2017 Mem: 2896136 2571024 325112 1968 1872 158520
Mon Sep 11 10:28:34 CEST 2017 Mem: 2896136 2571052 325084 1968 1872 158520
Mon Sep 11 10:28:36 CEST 2017 Mem: 2896136 2571244 324892 1968 1880 158720
Mon Sep 11 10:28:38 CEST 2017 Mem: 2896136 2571228 324908 1968 1884 158720
Mon Sep 11 10:28:40 CEST 2017 Mem: 2896136 2571328 324808 1968 1884 158748
Mon Sep 11 10:28:42 CEST 2017 Mem: 2896136 2571272 324864 1968 1884 158748
Mon Sep 11 10:28:44 CEST 2017 Mem: 2896136 2571164 324972 1968 1884 158748
Mon Sep 11 10:28:46 CEST 2017 Mem: 2896136 2571340 324796 1968 1884 158772
Mon Sep 11 10:28:48 CEST 2017 Mem: 2896136 2571272 324864 1968 1884 158784
Mon Sep 11 10:28:50 CEST 2017 Mem: 2896136 2571480 324656 1968 1888 158780
Mon Sep 11 10:28:52 CEST 2017 Mem: 2896136 2571708 324428 1968 1888 158804
Mon Sep 11 10:28:54 CEST 2017 Mem: 2896136 2571628 324508 1968 1888 158804
Mon Sep 11 10:28:56 CEST 2017 Mem: 2896136 2571660 324476 1968 1900 158804
Mon Sep 11 10:28:58 CEST 2017 Mem: 2896136 2571484 324652 1968 1900 158804
Mon Sep 11 10:29:00 CEST 2017 Mem: 2896136 2571612 324524 1968 1900 158868
Mon Sep 11 10:29:02 CEST 2017 Mem: 2896136 2571884 324252 2092 1904 159112
Mon Sep 11 10:29:04 CEST 2017 Mem: 2896136 2571924 324212 2124 1904 159292
Mon Sep 11 10:29:06 CEST 2017 Mem: 2896136 2572072 324064 2124 1904 159300
Mon Sep 11 10:29:08 CEST 2017 Mem: 2896136 2572416 323720 2124 1920 159292
Mon Sep 11 10:29:10 CEST 2017 Mem: 2896136 2572368 323768 2124 1920 159296
Mon Sep 11 10:29:12 CEST 2017 Mem: 2896136 2571816 324320 2124 1920 159288
Mon Sep 11 10:29:14 CEST 2017 Mem: 2896136 2571932 324204 2124 1924 159308
Mon Sep 11 10:29:16 CEST 2017 Mem: 2896136 2571884 324252 2124 1924 159308
Mon Sep 11 10:29:18 CEST 2017 Mem: 2896136 2571856 324280 2124 1924 159308
Mon Sep 11 10:29:20 CEST 2017 Mem: 2896136 2572328 323808 2124 1928 159288
Mon Sep 11 10:29:22 CEST 2017 Mem: 2896136 2572048 324088 2132 1928 159360
Mon Sep 11 10:29:24 CEST 2017 Mem: 2896136 2572240 323896 2132 1928 159356
Mon Sep 11 10:29:26 CEST 2017 Mem: 2896136 2572124 324012 2132 1940 159360
Mon Sep 11 10:29:28 CEST 2017 Mem: 2896136 2572300 323836 2132 1940 159376
Mon Sep 11 10:29:30 CEST 2017 Mem: 2896136 2572796 323340 2132 1944 159744
Mon Sep 11 10:29:32 CEST 2017 Mem: 2896136 2572696 323440 2132 1948 159812
Mon Sep 11 10:29:34 CEST 2017 Mem: 2896136 2572780 323356 2132 1948 159812
Mon Sep 11 10:29:36 CEST 2017 Mem: 2896136 2573620 322516 2132 1960 160076
Mon Sep 11 10:29:38 CEST 2017 Mem: 2896136 2573436 322700 2132 1964 160076
Mon Sep 11 10:29:40 CEST 2017 Mem: 2896136 2573444 322692 2132 1964 160068
Mon Sep 11 10:29:42 CEST 2017 Mem: 2896136 2576788 319348 2132 1964 160464
Mon Sep 11 10:29:44 CEST 2017 Mem: 2896136 2576668 319468 2132 1968 160464
Mon Sep 11 10:29:46 CEST 2017 Mem: 2896136 2576644 319492 2132 1968 160484
Mon Sep 11 10:29:48 CEST 2017 Mem: 2896136 2576568 319568 2132 1968 160484
Mon Sep 11 10:29:50 CEST 2017 Mem: 2896136 2577268 318868 2132 1968 160484
Mon Sep 11 10:29:52 CEST 2017 Mem: 2896136 2577268 318868 2132 1968 160484
Mon Sep 11 10:29:54 CEST 2017 Mem: 2896136 2577144 318992 2132 1968 160484
Mon Sep 11 10:29:56 CEST 2017 Mem: 2896136 2577400 318736 2132 1976 160484
Mon Sep 11 10:29:58 CEST 2017 Mem: 2896136 2577316 318820 2132 1976 160484
Mon Sep 11 10:30:00 CEST 2017 Mem: 2896136 2578260 317876 2132 1976 160488
Mon Sep 11 10:30:02 CEST 2017 Mem: 2896136 2591584 304552 2256 2900 162920
Mon Sep 11 10:30:04 CEST 2017 Mem: 2896136 2600808 295328 2328 2904 165324
Mon Sep 11 10:30:06 CEST 2017 Mem: 2896136 2620672 275464 2328 2944 172604
Mon Sep 11 10:30:08 CEST 2017 Mem: 2896136 2770732 125404 2328 2972 179292
Mon Sep 11 10:30:11 CEST 2017 Mem: 2896136 2807560 88576 2328 304 76196
Mon Sep 11 10:30:15 CEST 2017 Mem: 2896136 2810784 85352 2328 308 76700
Mon Sep 11 10:30:20 CEST 2017 Mem: 2896136 1417628 1478508 2932 3608 176892
Mon Sep 11 10:30:22 CEST 2017 Mem: 2896136 1912828 983308 2940 4292 196444
Mon Sep 11 10:30:24 CEST 2017 Mem: 2896136 2254356 641780 2944 4300 199700
Mon Sep 11 10:30:26 CEST 2017 Mem: 2896136 2150284 745852 2944 4324 197096
Mon Sep 11 10:30:28 CEST 2017 Mem: 2896136 2240400 655736 2944 4324 197088
Mon Sep 11 10:30:30 CEST 2017 Mem: 2896136 2301036 595100 2944 4472 197428
Mon Sep 11 10:30:32 CEST 2017 Mem: 2896136 2335808 560328 2944 4476 197428
Mon Sep 11 10:30:34 CEST 2017 Mem: 2896136 2408472 487664 2944 4480 197808
Mon Sep 11 10:30:36 CEST 2017 Mem: 2896136 2491768 404368 2944 4488 197812
Mon Sep 11 10:30:38 CEST 2017 Mem: 2896136 2578684 317452 2944 6208 201296
Mon Sep 11 10:30:40 CEST 2017 Mem: 2896136 2648612 247524 2932 6828 203772
Mon Sep 11 10:30:42 CEST 2017 Mem: 2896136 2732180 163956 2932 6972 204012
Mon Sep 11 10:30:44 CEST 2017 Mem: 2896136 2795708 100428 2932 6984 204004
Mon Sep 11 10:30:46 CEST 2017 Mem: 2896136 2779028 117108 2532 180 110220
Mon Sep 11 10:30:48 CEST 2017 Mem: 2896136 2712536 183600 1316 164 104452
Mon Sep 11 10:30:50 CEST 2017 Mem: 2896136 2112380 783756 1316 216 127748
Mon Sep 11 10:30:52 CEST 2017 Mem: 2896136 2222804 673332 1316 216 129280
Mon Sep 11 10:30:54 CEST 2017 Mem: 2896136 2247032 649104 1316 664 134148
Mon Sep 11 10:30:56 CEST 2017 Mem: 2896136 2272508 623628 1316 976 136372
Mon Sep 11 10:30:58 CEST 2017 Mem: 2896136 2292300 603836 1316 976 136836
Mon Sep 11 10:31:00 CEST 2017 Mem: 2896136 2371484 524652 1316 984 136832
Mon Sep 11 10:31:02 CEST 2017 Mem: 2896136 2450084 446052 2608 2988 148304
Mon Sep 11 10:31:04 CEST 2017 Mem: 2896136 2547640 348496 2608 3156 162892
Mon Sep 11 10:31:06 CEST 2017 Mem: 2896136 2661628 234508 2608 3316 193600
Mon Sep 11 10:31:08 CEST 2017 Mem: 2896136 2726244 169892 2608 3328 193928
Mon Sep 11 10:31:10 CEST 2017 Mem: 2896136 2769176 126960 2608 3112 155388
Mon Sep 11 10:31:12 CEST 2017 Mem: 2896136 2558116 338020 2608 932 145408
Mon Sep 11 10:31:14 CEST 2017 Mem: 2896136 2599240 296896 2608 996 145756
Mon Sep 11 10:31:16 CEST 2017 Mem: 2896136 2652820 243316 2608 1000 147080
Mon Sep 11 10:31:18 CEST 2017 Mem: 2896136 2717032 179104 2608 1020 161512
Mon Sep 11 10:31:20 CEST 2017 Mem: 2896136 2769104 127032 2608 1028 161520
Mon Sep 11 10:31:22 CEST 2017 Mem: 2896136 2785564 110572 2648 1032 128132
Mon Sep 11 10:31:24 CEST 2017 Mem: 2896136 2813480 82656 2256 188 84332
Mon Sep 11 10:31:26 CEST 2017 Mem: 2896136 2480436 415700 1556 504 80488
Mon Sep 11 10:31:28 CEST 2017 Mem: 2896136 2484628 411508 1556 520 84920
Mon Sep 11 10:31:30 CEST 2017 Mem: 2896136 2486076 410060 1556 684 85988
Mon Sep 11 10:31:32 CEST 2017 Mem: 2896136 2486280 409856 1556 684 86024
Mon Sep 11 10:31:34 CEST 2017 Mem: 2896136 2486684 409452 1556 700 86508
Mon Sep 11 10:31:36 CEST 2017 Mem: 2896136 2488516 407620 1556 716 88200
Mon Sep 11 10:31:38 CEST 2017 Mem: 2896136 2488828 407308 1556 724 88416
Mon Sep 11 10:31:40 CEST 2017 Mem: 2896136 2489736 406400 1556 724 88648
Mon Sep 11 10:31:42 CEST 2017 Mem: 2896136 2493116 403020 1556 728 89564
Mon Sep 11 10:31:44 CEST 2017 Mem: 2896136 2492652 403484 1556 728 89560
Mon Sep 11 10:31:46 CEST 2017 Mem: 2896136 2493184 402952 1556 732 90064
Mon Sep 11 10:31:48 CEST 2017 Mem: 2896136 2493012 403124 1556 732 90064
Mon Sep 11 10:31:50 CEST 2017 Mem: 2896136 2496988 399148 1556 760 93340
Mon Sep 11 10:31:52 CEST 2017 Mem: 2896136 2498592 397544 1556 776 94748
Mon Sep 11 10:31:54 CEST 2017 Mem: 2896136 2498928 397208 1556 788 95120
Mon Sep 11 10:31:56 CEST 2017 Mem: 2896136 2499212 396924 1556 932 95760
Mon Sep 11 10:31:58 CEST 2017 Mem: 2896136 2499312 396824 1556 932 95760
Mon Sep 11 10:32:00 CEST 2017 Mem: 2896136 2499236 396900 1556 932 95760
Mon Sep 11 10:32:02 CEST 2017 Mem: 2896136 2513320 382816 2060 1376 109492
Mon Sep 11 10:32:04 CEST 2017 Mem: 2896136 2526808 369328 2060 1388 123012
Mon Sep 11 10:32:06 CEST 2017 Mem: 2896136 2558004 338132 2060 1436 153812
Mon Sep 11 10:32:08 CEST 2017 Mem: 2896136 2558296 337840 2060 1436 153812
Mon Sep 11 10:32:10 CEST 2017 Mem: 2896136 2558148 337988 2060 1436 153816
Mon Sep 11 10:32:12 CEST 2017 Mem: 2896136 2558272 337864 2060 1440 153816
Mon Sep 11 10:32:14 CEST 2017 Mem: 2896136 2558040 338096 2060 1440 153816
Mon Sep 11 10:32:16 CEST 2017 Mem: 2896136 2558204 337932 2060 1440 153800
Mon Sep 11 10:32:18 CEST 2017 Mem: 2896136 2558064 338072 2060 1440 153804
Mon Sep 11 10:32:20 CEST 2017 Mem: 2896136 2582132 314004 2060 1448 154104
Mon Sep 11 10:32:22 CEST 2017 Mem: 2896136 2529612 366524 2060 1452 119500
Mon Sep 11 10:32:24 CEST 2017 Mem: 2896136 2529344 366792 2060 1460 119484
Mon Sep 11 10:32:26 CEST 2017 Mem: 2896136 2529364 366772 2060 1460 119480
Mon Sep 11 10:32:28 CEST 2017 Mem: 2896136 2529488 366648 2060 1460 119476
Mon Sep 11 10:32:30 CEST 2017 Mem: 2896136 2529440 366696 2060 1460 119476
Mon Sep 11 10:32:32 CEST 2017 Mem: 2896136 2529152 366984 2060 1460 119452
Mon Sep 11 10:32:34 CEST 2017 Mem: 2896136 2529136 367000 2060 1588 119472
Mon Sep 11 10:32:37 CEST 2017 Mem: 2896136 2529424 366712 2060 1588 119472
Mon Sep 11 10:32:39 CEST 2017 Mem: 2896136 2529764 366372 2060 1588 119488
Mon Sep 11 10:32:41 CEST 2017 Mem: 2896136 2529772 366364 2060 1588 119476
Mon Sep 11 10:32:43 CEST 2017 Mem: 2896136 2529904 366232 2060 1588 119460
Mon Sep 11 10:32:45 CEST 2017 Mem: 2896136 2529788 366348 2060 1588 119460
Mon Sep 11 10:32:47 CEST 2017 Mem: 2896136 2529940 366196 2060 1588 119448
Mon Sep 11 10:32:49 CEST 2017 Mem: 2896136 2529928 366208 2060 1588 119448
Mon Sep 11 10:32:51 CEST 2017 Mem: 2896136 2544992 351144 2060 2004 125128
Mon Sep 11 10:32:53 CEST 2017 Mem: 2896136 2507544 388592 2060 2292 145348
Mon Sep 11 10:32:55 CEST 2017 Mem: 2896136 2508300 387836 2060 2316 145476
^C
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services