You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/04/10 13:47:00 UTC
[jira] [Commented] (ARROW-2222) [C++] Add option to validate
Flatbuffers messages
[ https://issues.apache.org/jira/browse/ARROW-2222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16432282#comment-16432282 ]
ASF GitHub Bot commented on ARROW-2222:
---------------------------------------
crepererum commented on issue #1763: ARROW-2222: handle untrusted inputs (POC)
URL: https://github.com/apache/arrow/pull/1763#issuecomment-380103831
## Environment
### Build Commands
```
env CC=clang CXX=clang++ cmake -DCMAKE_BUILD_TYPE=Release -DARROW_BUILD_BENCHMARKS=ON -GNinja ..
env CC=clang CXX=clang++ ninja
```
### Conda
```
# Name Version Build Channel
boost 1.66.0 py36_1 conda-forge
boost-cpp 1.66.0 1 conda-forge
bzip2 1.0.6 1 conda-forge
ca-certificates 2018.1.18 0 conda-forge
certifi 2018.1.18 py36_0 conda-forge
clangdev 6.0.0 default_0 conda-forge
cmake 3.11.0 0 conda-forge
curl 7.59.0 1 conda-forge
expat 2.2.5 0 conda-forge
icu 58.2 0 conda-forge
intel-openmp 2018.0.0 8
krb5 1.14.6 0 conda-forge
libcxx 6.0.0 0 conda-forge
libgfortran 3.0.1 h93005f0_2
libiconv 1.15 0 conda-forge
libssh2 1.8.0 2 conda-forge
libuv 1.20.0 0 conda-forge
libxml2 2.9.8 0 conda-forge
llvm-meta 6.0.0 0 conda-forge
llvmdev 6.0.0 default_0 conda-forge
mkl 2018.0.2 1
mkl_fft 1.0.1 py36_1 conda-forge
mkl_random 1.0.1 py36_0 conda-forge
ncurses 5.9 10 conda-forge
ninja 1.8.2 1 conda-forge
numpy 1.14.2 py36ha9ae307_1
openssl 1.0.2n 0 conda-forge
pip 9.0.3 py36_0 conda-forge
python 3.6.5 1 conda-forge
readline 7.0 0 conda-forge
rhash 1.3.4 0 conda-forge
setuptools 39.0.1 py36_0 conda-forge
sqlite 3.20.1 2 conda-forge
tk 8.6.7 0 conda-forge
wheel 0.31.0 py36_0 conda-forge
xz 5.2.3 0 conda-forge
zlib 1.2.11 0 conda-forge
```
## Test Results
### complete PR
```
Run on (4 X 2700 MHz CPU s)
2018-04-10 15:33:15
Benchmark Time CPU Iterations
----------------------------------------------------------------------------------------
BM_WriteRecordBatch/1/min_time:1.000/real_time 76462 ns 76204 ns 16080 12.7719GB/s
BM_WriteRecordBatch/4/min_time:1.000/real_time 63077 ns 62791 ns 21229 15.482GB/s
BM_WriteRecordBatch/16/min_time:1.000/real_time 56850 ns 56584 ns 24134 17.1778GB/s
BM_WriteRecordBatch/64/min_time:1.000/real_time 58209 ns 58003 ns 22891 16.7768GB/s
BM_WriteRecordBatch/256/min_time:1.000/real_time 105328 ns 104930 ns 12450 9.27162GB/s
BM_WriteRecordBatch/1024/min_time:1.000/real_time 247453 ns 245516 ns 5383 3.94645GB/s
BM_WriteRecordBatch/4k/min_time:1.000/real_time 904931 ns 900426 ns 1384 1105.06MB/s
BM_WriteRecordBatch/8k/min_time:1.000/real_time 1869873 ns 1861388 ns 704 534.796MB/s
BM_ReadRecordBatch/1/min_time:1.000/real_time 2409 ns 2402 ns 584482 405.366GB/s
BM_ReadRecordBatch/4/min_time:1.000/real_time 4919 ns 4888 ns 284403 198.544GB/s
BM_ReadRecordBatch/16/min_time:1.000/real_time 14176 ns 14123 ns 73649 68.8866GB/s
BM_ReadRecordBatch/64/min_time:1.000/real_time 51842 ns 51373 ns 26951 18.8372GB/s
BM_ReadRecordBatch/256/min_time:1.000/real_time 201006 ns 200181 ns 6691 4.85837GB/s
BM_ReadRecordBatch/1024/min_time:1.000/real_time 845037 ns 835158 ns 1228 1.15564GB/s
BM_ReadRecordBatch/4k/min_time:1.000/real_time 3340162 ns 3328564 ns 422 299.387MB/s
BM_ReadRecordBatch/8k/min_time:1.000/real_time 7270398 ns 7240046 ns 195 137.544MB/s
```
### PR w/o flatbuffer validation
```
Run on (4 X 2700 MHz CPU s)
2018-04-10 15:35:33
Benchmark Time CPU Iterations
----------------------------------------------------------------------------------------
BM_WriteRecordBatch/1/min_time:1.000/real_time 76915 ns 76548 ns 18341 12.6967GB/s
BM_WriteRecordBatch/4/min_time:1.000/real_time 65802 ns 65135 ns 21929 14.841GB/s
BM_WriteRecordBatch/16/min_time:1.000/real_time 56544 ns 56359 ns 24133 17.271GB/s
BM_WriteRecordBatch/64/min_time:1.000/real_time 62888 ns 62205 ns 23517 15.5287GB/s
BM_WriteRecordBatch/256/min_time:1.000/real_time 106749 ns 105842 ns 13172 9.14824GB/s
BM_WriteRecordBatch/1024/min_time:1.000/real_time 255753 ns 253039 ns 5469 3.81838GB/s
BM_WriteRecordBatch/4k/min_time:1.000/real_time 990244 ns 969574 ns 1428 1009.85MB/s
BM_WriteRecordBatch/8k/min_time:1.000/real_time 1881824 ns 1870245 ns 754 531.399MB/s
BM_ReadRecordBatch/1/min_time:1.000/real_time 2408 ns 2393 ns 585709 405.624GB/s
BM_ReadRecordBatch/4/min_time:1.000/real_time 4815 ns 4790 ns 293507 202.816GB/s
BM_ReadRecordBatch/16/min_time:1.000/real_time 14351 ns 14294 ns 95709 68.047GB/s
BM_ReadRecordBatch/64/min_time:1.000/real_time 51902 ns 51532 ns 26885 18.8155GB/s
BM_ReadRecordBatch/256/min_time:1.000/real_time 214495 ns 209335 ns 7029 4.55284GB/s
BM_ReadRecordBatch/1024/min_time:1.000/real_time 817379 ns 809841 ns 1781 1.19475GB/s
BM_ReadRecordBatch/4k/min_time:1.000/real_time 3305725 ns 3296526 ns 401 302.506MB/s
BM_ReadRecordBatch/8k/min_time:1.000/real_time 7218564 ns 7187446 ns 193 138.532MB/s
```
### baseline (`6633cc94c84f9ecc516f03175da56361cf4bd67f`)
```
Run on (4 X 2700 MHz CPU s)
2018-04-10 15:36:57
Benchmark Time CPU Iterations
----------------------------------------------------------------------------------------
BM_WriteRecordBatch/1/min_time:1.000/real_time 77609 ns 77412 ns 16827 12.5831GB/s
BM_WriteRecordBatch/4/min_time:1.000/real_time 62939 ns 62705 ns 21531 15.516GB/s
BM_WriteRecordBatch/16/min_time:1.000/real_time 58021 ns 57736 ns 22232 16.8311GB/s
BM_WriteRecordBatch/64/min_time:1.000/real_time 58180 ns 57996 ns 24262 16.7851GB/s
BM_WriteRecordBatch/256/min_time:1.000/real_time 109448 ns 108065 ns 11190 8.92262GB/s
BM_WriteRecordBatch/1024/min_time:1.000/real_time 256618 ns 254950 ns 4017 3.80551GB/s
BM_WriteRecordBatch/4k/min_time:1.000/real_time 907773 ns 904273 ns 1553 1101.6MB/s
BM_WriteRecordBatch/8k/min_time:1.000/real_time 1864168 ns 1859038 ns 732 536.432MB/s
BM_ReadRecordBatch/1/min_time:1.000/real_time 2414 ns 2405 ns 579529 404.6GB/s
BM_ReadRecordBatch/4/min_time:1.000/real_time 4828 ns 4807 ns 292168 202.264GB/s
BM_ReadRecordBatch/16/min_time:1.000/real_time 14357 ns 14312 ns 96203 68.0212GB/s
BM_ReadRecordBatch/64/min_time:1.000/real_time 51445 ns 51189 ns 26539 18.9828GB/s
BM_ReadRecordBatch/256/min_time:1.000/real_time 196948 ns 196421 ns 6995 4.95849GB/s
BM_ReadRecordBatch/1024/min_time:1.000/real_time 799547 ns 797232 ns 1773 1.22139GB/s
BM_ReadRecordBatch/4k/min_time:1.000/real_time 3374307 ns 3362444 ns 423 296.357MB/s
BM_ReadRecordBatch/8k/min_time:1.000/real_time 7205776 ns 7186692 ns 195 138.778MB/s
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
> [C++] Add option to validate Flatbuffers messages
> -------------------------------------------------
>
> Key: ARROW-2222
> URL: https://issues.apache.org/jira/browse/ARROW-2222
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Wes McKinney
> Assignee: Marco Neumann
> Priority: Major
> Labels: pull-request-available
>
> This is follow up work to ARROW-1589, ARROW-2023, and can be validated by the {{ipc-fuzzer-test}}. Users receiving untrusted input streams can prevent segfaults this way
> As part of this, we should quantify the overhead associated with message validation in regular use
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)