You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@trafficserver.apache.org by GitBox <gi...@apache.org> on 2021/11/21 03:21:50 UTC
[GitHub] [trafficserver] cukiernik opened a new pull request #8534: TS_USE_MMAP: mmap+memcopy instead pread+pwrite
cukiernik opened a new pull request #8534:
URL: https://github.com/apache/trafficserver/pull/8534
apt install libaio-dev
./configure --enable-mmap --enable-experimental-linux-native-aio
./configure --enable-mmap
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] cukiernik edited a comment on pull request #8534: TS_USE_MMAP: mmap+memcopy instead pread+pwrite
Posted by GitBox <gi...@apache.org>.
cukiernik edited a comment on pull request #8534:
URL: https://github.com/apache/trafficserver/pull/8534#issuecomment-1011014464
Hello, after my long holidays.
@ywkaras: I didn't run the _au test_, because _pip_ and _PyPi_ seem to endanger for my network security.
@keesspoelstra: Yes, run it without a ram cache. I was also wondering whether to force _msync_ after each modification, using _MS_ASYNC_, a add dedicated thread to _MS_SYNC_. but I couldn't decide on any of the solutions, I will propose a solution without it. **It is not needed to a correctly working**. As I wrote above, I check it with a gigantic amount of RAM, and after modifying the whole cache, _msync MS_SYNC_ takes more than a few minutes.
In my task, I don't need to store content cache after _shutdown_. In this case, _MAP_ANONYMOUS_ start faster. It is also suggested by @SolidWallOfCode, but with _MAP_SHARED_ it's easier to trace and debug for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] cukiernik edited a comment on pull request #8534: TS_USE_MMAP: mmap+memcopy instead pread+pwrite
Posted by GitBox <gi...@apache.org>.
cukiernik edited a comment on pull request #8534:
URL: https://github.com/apache/trafficserver/pull/8534#issuecomment-1011014464
Hello, after my long holidays.
@ywkaras: I didn't run the _au test_, because _pip_ and _PyPi_ seem to endanger for my system security.
@keesspoelstra: Yes, run it without a ram cache. I was also wondering whether to force _msync_ after each modyfication, using _MS_ASYNC_, a add dedicated thread to _MS_SYNC_. but I couldn't decide on any of the solutions, I will propose a solution without it. **It is not needet to a correcty working**. As I wrote above, I check it with a gigantic amount of RAM, and after modifying the whole cache, _msync MS_SYNC_ takes more than a few minutes.
In my task, I don't need to store content cache after _shutdown_. In this case, _MAP_ANONYMOUS_ start faster. It is also suggested @SolidWallOfCode, but with _MAP_SHARED_ it's easier to trace and debug for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] cukiernik edited a comment on pull request #8534: TS_USE_MMAP: mmap+memcopy instead pread+pwrite
Posted by GitBox <gi...@apache.org>.
cukiernik edited a comment on pull request #8534:
URL: https://github.com/apache/trafficserver/pull/8534#issuecomment-1011014464
Hello, after my long holidays.
@ywkaras: I didn't run the _au test_, because _pip_ and _PyPi_ seem to endanger for my network security.
@keesspoelstra: Yes, run it without a ram cache. I was also wondering whether to force _msync_ after each modification, using _MS_ASYNC_, a add dedicated thread to _MS_SYNC_. but I couldn't decide on any of the solutions, I will propose a solution without it. **It is not needed to a correctly working**. As I wrote above, I check it with a gigantic amount of RAM, and after modifying the whole cache, _msync MS_SYNC_ takes more than a few minutes.
In my task, I don't need to store content cache after _shutdown_. In this case, _MAP_ANONYMOUS_ start faster. It is also suggested @SolidWallOfCode, but with _MAP_SHARED_ it's easier to trace and debug for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] cukiernik edited a comment on pull request #8534: TS_USE_MMAP: mmap+memcopy instead pread+pwrite
Posted by GitBox <gi...@apache.org>.
cukiernik edited a comment on pull request #8534:
URL: https://github.com/apache/trafficserver/pull/8534#issuecomment-976973657
The improvement is not significant, single percentages. Speed up is due to reduce a system calls, but at the cost of increased page faults. I am testing it on a host with huge RAM.
The reading from the file is done only on the missing page (instead of a system call each time). Memory writes to a file when the system is low on free pages. So performance of IO thread's is very close to that of a swap file.
Further acceleration will be achieved by optimizing the _agg_copy_. _Mmap_ allows aggregation into target memory, which will eliminate _memcpy_.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] cukiernik edited a comment on pull request #8534: TS_USE_MMAP: mmap+memcopy instead pread+pwrite
Posted by GitBox <gi...@apache.org>.
cukiernik edited a comment on pull request #8534:
URL: https://github.com/apache/trafficserver/pull/8534#issuecomment-1011014464
Hello, after my long holidays.
@ywkaras: I didn't run the _au test_, because _pip_ and _PyPi_ seem to endanger for my network security.
@keesspoelstra: Yes, run it without a ram cache. I was also wondering whether to force _msync_ after each modyfication, using _MS_ASYNC_, a add dedicated thread to _MS_SYNC_. but I couldn't decide on any of the solutions, I will propose a solution without it. **It is not needed to a correcty working**. As I wrote above, I check it with a gigantic amount of RAM, and after modifying the whole cache, _msync MS_SYNC_ takes more than a few minutes.
In my task, I don't need to store content cache after _shutdown_. In this case, _MAP_ANONYMOUS_ start faster. It is also suggested @SolidWallOfCode, but with _MAP_SHARED_ it's easier to trace and debug for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] keesspoelstra commented on pull request #8534: TS_USE_MMAP: mmap+memcopy instead pread+pwrite
Posted by GitBox <gi...@apache.org>.
keesspoelstra commented on pull request #8534:
URL: https://github.com/apache/trafficserver/pull/8534#issuecomment-1008724337
@cukiernik , A few percent sounds significant already 👍 this could be very useful. You have run it without RAM cache?
I think that for a write into a memory mapped file it has to page the data into memory first (a read), so that adds some overhead. We may be able to get some more performance here by writing with pwrite instead of memcpy.
This also keeps write behaviour predictable, which is better if ATS crashes (which it never does of course).
With memory mapped files it is not guaranteed that a valid index will have been written after a crash. A msync could solve this, but the order in which the msync writes the dirty pages is not guaranteed, so a bit more orchestration would be needed for msyncing the index + data. Worst case a lot of data will be written at a very inconvenient moment.
Linux provides a consistent view between pwrites and memory mapped files, for other systems we'd have to check the behaviour.
You might want to take a look at this:
https://github.com/apache/trafficserver/blob/3012ca0f94cf7202e75422757d7498bc7e4f9295/iocore/aio/AIO.cc#L598
I think we're hitting sync I/O here if the pages are not in memory, blocking the thread instead of using the original async nature of the function. I don't know if passing the read to a number of specialized threads would yield better performance, I suspect it will all depend on the workload.
Mincore and msync can check whether pages are in memory, so that could be used to set up a fast (serve from memory) path, but the drawback would be the overhead of these functions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] cukiernik commented on pull request #8534: TS_USE_MMAP: mmap+memcopy instead pread+pwrite
Posted by GitBox <gi...@apache.org>.
cukiernik commented on pull request #8534:
URL: https://github.com/apache/trafficserver/pull/8534#issuecomment-1011014464
Hello, after my long holidays.
@ywkaras: I didn't run the _au test_, because _pip_ and _PyPi_ seem to endanger my system security.
@keesspoelstra: I was also wondering whether to force _msync_ after each modyfication, using _MS_ASYNC_, a add dedicated thread to _MS_SYNC_. but I couldn't decide on any of the solutions, I will propose a solution without it. **It is not needet to a correcty working**. As I wrote above, I check it with a gigantic amount of RAM, and after modifying the whole cache, _msync MS_SYNC_ takes more than a few minutes.
In my task, I don't need to store content cache after _shutdown_. In this case, _MAP_ANONYMOUS_ works faster. It is also suggested @SolidWallOfCode, but with _MAP_SHARED_ it's easier to trace and debug for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] ezelkow1 commented on pull request #8534: TS_USE_MMAP: mmap+memcopy instead pread+pwrite
Posted by GitBox <gi...@apache.org>.
ezelkow1 commented on pull request #8534:
URL: https://github.com/apache/trafficserver/pull/8534#issuecomment-979475453
[approve ci]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] ywkaras commented on pull request #8534: TS_USE_MMAP: mmap+memcopy instead pread+pwrite
Posted by GitBox <gi...@apache.org>.
ywkaras commented on pull request #8534:
URL: https://github.com/apache/trafficserver/pull/8534#issuecomment-983191056
@cukiernik Have you run all the Au tests on a build with this configured?
Are you currently at Intel, can you talk about the applications you all are using ATS for?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] cukiernik edited a comment on pull request #8534: TS_USE_MMAP: mmap+memcopy instead pread+pwrite
Posted by GitBox <gi...@apache.org>.
cukiernik edited a comment on pull request #8534:
URL: https://github.com/apache/trafficserver/pull/8534#issuecomment-1011014464
Hello, after my long holidays.
@ywkaras: I didn't run the _au test_, because _pip_ and _PyPi_ seem to endanger for my system security.
@keesspoelstra: I was also wondering whether to force _msync_ after each modyfication, using _MS_ASYNC_, a add dedicated thread to _MS_SYNC_. but I couldn't decide on any of the solutions, I will propose a solution without it. **It is not needet to a correcty working**. As I wrote above, I check it with a gigantic amount of RAM, and after modifying the whole cache, _msync MS_SYNC_ takes more than a few minutes.
In my task, I don't need to store content cache after _shutdown_. In this case, _MAP_ANONYMOUS_ works faster. It is also suggested @SolidWallOfCode, but with _MAP_SHARED_ it's easier to trace and debug for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] bryancall commented on pull request #8534: TS_USE_MMAP: mmap+memcopy instead pread+pwrite
Posted by GitBox <gi...@apache.org>.
bryancall commented on pull request #8534:
URL: https://github.com/apache/trafficserver/pull/8534#issuecomment-976021788
@cukiernik Is this supposed to be faster than pread/pwrite? Do you have any benchmarks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] cukiernik commented on pull request #8534: TS_USE_MMAP: mmap+memcopy instead pread+pwrite
Posted by GitBox <gi...@apache.org>.
cukiernik commented on pull request #8534:
URL: https://github.com/apache/trafficserver/pull/8534#issuecomment-979450523
Should I redo "push" to start "**expected checks**" processes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] cukiernik commented on pull request #8534: TS_USE_MMAP: mmap+memcopy instead pread+pwrite
Posted by GitBox <gi...@apache.org>.
cukiernik commented on pull request #8534:
URL: https://github.com/apache/trafficserver/pull/8534#issuecomment-986773279
@ywkaras : Yes, I ran a tests, but I don't know what's on your mind when you write _all_?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] ezelkow1 commented on pull request #8534: TS_USE_MMAP: mmap+memcopy instead pread+pwrite
Posted by GitBox <gi...@apache.org>.
ezelkow1 commented on pull request #8534:
URL: https://github.com/apache/trafficserver/pull/8534#issuecomment-1045223777
[approve ci rocky]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] cukiernik edited a comment on pull request #8534: TS_USE_MMAP: mmap+memcopy instead pread+pwrite
Posted by GitBox <gi...@apache.org>.
cukiernik edited a comment on pull request #8534:
URL: https://github.com/apache/trafficserver/pull/8534#issuecomment-976973657
The improvement is not significant, single percentages. Speed up is due to reduce a system calls, but at the cost of increased page faults.
The reading from the file is done only on the missing page (instead of a system call each time). Memory writes to a file when the system is low on free pages. So performance of IO thread's is very close to that of a swap file.
Further acceleration will be achieved by optimizing the agg_copy. Mmap allows aggregation into target memory, which will eliminate memcpy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] bryancall edited a comment on pull request #8534: TS_USE_MMAP: mmap+memcopy instead pread+pwrite
Posted by GitBox <gi...@apache.org>.
bryancall edited a comment on pull request #8534:
URL: https://github.com/apache/trafficserver/pull/8534#issuecomment-976021788
@cukiernik Is this supposed to be faster than pread/pwrite? Do you have any benchmarks?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] keesspoelstra commented on pull request #8534: TS_USE_MMAP: mmap+memcopy instead pread+pwrite
Posted by GitBox <gi...@apache.org>.
keesspoelstra commented on pull request #8534:
URL: https://github.com/apache/trafficserver/pull/8534#issuecomment-1009307011
Just checked whether ink_aio_read and ink_aio_write are used by anything else. In the C API we have TSAIOWrite + TSAIORead. This feature will break plugins which expect to be able to write past the end of the file with TSAIOWrite.
As the mmap stuff is experimental it I guess it is not a problem now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] cukiernik edited a comment on pull request #8534: TS_USE_MMAP: mmap+memcopy instead pread+pwrite
Posted by GitBox <gi...@apache.org>.
cukiernik edited a comment on pull request #8534:
URL: https://github.com/apache/trafficserver/pull/8534#issuecomment-1011014464
Hello, after my long holidays.
@ywkaras: I didn't run the _au test_, because _pip_ and _PyPi_ seem to endanger for my network security.
@keesspoelstra: Yes, run it without a ram cache. I was also wondering whether to force _msync_ after each modification, using _MS_ASYNC_, a add dedicated thread to _MS_SYNC_. As a last resort, it possible is parallel _mmap_ in _traffic_crashlog_, and _msync_ from it on fail _TS_MAIN_. I couldn't decide on any of the solutions, I will propose a solution without it. **It is not needed to a correctly working**. As I wrote above, I check it with a gigantic amount of RAM, and after modifying the whole cache, _msync MS_SYNC_ takes more than a few minutes.
In my task, I don't need to store content cache after _shutdown_. In this case, _MAP_ANONYMOUS_ start faster. It is also suggested by @SolidWallOfCode, but with _MAP_SHARED_ it's easier to trace and debug for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] cukiernik commented on pull request #8534: TS_USE_MMAP: mmap+memcopy instead pread+pwrite
Posted by GitBox <gi...@apache.org>.
cukiernik commented on pull request #8534:
URL: https://github.com/apache/trafficserver/pull/8534#issuecomment-976973657
The improvement is not significant, single percentages. Speed up is due to reduce a system calls, but at the cost of increased page faults.
The reading from the file is done only on the missing page (instead of a system call each time). Memory writes to a file when the system is low on free pages. So this performance of IO thread's is very close to that of a swap file.
Further acceleration will be achieved by optimizing the agg_copy. Mmap allows aggregation into target memory, which will eliminate memcpy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] cukiernik edited a comment on pull request #8534: TS_USE_MMAP: mmap+memcopy instead pread+pwrite
Posted by GitBox <gi...@apache.org>.
cukiernik edited a comment on pull request #8534:
URL: https://github.com/apache/trafficserver/pull/8534#issuecomment-976973657
The improvement is not significant, single percentages. Speed up is due to reduce a system calls, but at the cost of increased page faults. I am testing it on a host with huge RAM.
The reading from the file is done only on the missing page (instead of a system call each time). Memory writes to a file when the system is low on free pages. So performance of IO thread's is very close to that of a swap file.
Further acceleration will be achieved by optimizing the _agg_copy_. _Mmap_ allows aggregation into target memory, which will eliminate _memcpy_. I expect this to be a bigger benefit than just switching to mmap
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] ywkaras commented on pull request #8534: TS_USE_MMAP: mmap+memcopy instead pread+pwrite
Posted by GitBox <gi...@apache.org>.
ywkaras commented on pull request #8534:
URL: https://github.com/apache/trafficserver/pull/8534#issuecomment-986843726
To run all the Au tests, you cd to the root directory of your clone of the trafficserver repo, build it with your branch, install it, and then do:
```
cd tests
./autests.sh --ats-bin <install-root-directory>/bin
```
It will take 20 - 40 minutes to run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] ywkaras edited a comment on pull request #8534: TS_USE_MMAP: mmap+memcopy instead pread+pwrite
Posted by GitBox <gi...@apache.org>.
ywkaras edited a comment on pull request #8534:
URL: https://github.com/apache/trafficserver/pull/8534#issuecomment-986843726
To run all the Au tests, you cd to the root directory of your clone of the trafficserver repo, build it with your branch and the new **configure** option, install it, and then do:
```
cd tests
./autests.sh --ats-bin <install-root-directory>/bin
```
It will take 20 - 40 minutes to run.
Would be good to do this twice, with and without the --enable-experimental-linux-native-aio configure option for **configure**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] cukiernik edited a comment on pull request #8534: TS_USE_MMAP: mmap+memcopy instead pread+pwrite
Posted by GitBox <gi...@apache.org>.
cukiernik edited a comment on pull request #8534:
URL: https://github.com/apache/trafficserver/pull/8534#issuecomment-1011014464
Hello, after my long holidays.
@ywkaras: I didn't run the _au test_, because _pip_ and _PyPi_ seem to endanger for my system security.
@keesspoelstra: I was also wondering whether to force _msync_ after each modyfication, using _MS_ASYNC_, a add dedicated thread to _MS_SYNC_. but I couldn't decide on any of the solutions, I will propose a solution without it. **It is not needet to a correcty working**. As I wrote above, I check it with a gigantic amount of RAM, and after modifying the whole cache, _msync MS_SYNC_ takes more than a few minutes.
In my task, I don't need to store content cache after _shutdown_. In this case, _MAP_ANONYMOUS_ start faster. It is also suggested @SolidWallOfCode, but with _MAP_SHARED_ it's easier to trace and debug for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org