You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by 高立周 <li...@zilliz.com> on 2018/03/30 05:40:47 UTC
poor write performance with optional fields
Hi guys,
I'm testing parquet write performance for optional fields with attached cc file.
The throughput is 125MB/s, but the nvme disks limits is 570MB/s.
So, my question is that whether it is expected performance ?
Is there anything wrong with the test program ?
Thanks,
Lizhou
lizgao@lizgao-ubuntu:~/CLionProjects/parquet-test/cmake-build-debug$ ./parquet_test
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0330 13:28:57.599980 17640 reader-writer-performance.cc:72] BEGIN performance_test
I0330 13:29:13.640228 17640 reader-writer-performance.cc:112] END performance_test
PROFILE: interrupts/evictions/bytes = 1551/857/61200
lizgao@lizgao-ubuntu:~/CLionProjects/parquet-test/cmake-build-debug$ ls -lh parquet_cpp_example.parquet
-rw-r--r-- 1 lizgao lizgao 2.0G Mar 30 13:29 parquet_cpp_example.parquet
lizgao@lizgao-ubuntu:~/CLionProjects/parquet-test/cmake-build-debug$ dd if=/dev/zero of=test bs=1M count=4096 oflag=direct
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 7.48397 s, 574 MB/s
lizgao@lizgao-ubuntu:~/CLionProjects/parquet-test/cmake-build-debug$ dd if=/dev/zero of=test bs=1M count=4096 oflag=direct
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 7.98757 s, 538 MB/s
lizgao@lizgao-ubuntu:~/CLionProjects/parquet-test/cmake-build-debug$ dd if=/dev/zero of=test bs=10M count=400 oflag=direct
400+0 records in
400+0 records out
4194304000 bytes (4.2 GB, 3.9 GiB) copied, 8.03284 s, 522 MB/s
lizgao@lizgao-ubuntu:~/CLionProjects/parquet-test/cmake-build-debug$ dd if=/dev/zero of=test bs=1G count=4 oflag=direct
4+0 records in
4+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 8.30123 s, 517 MB/s
Re:poor write performance with optional fields
Posted by 高立周 <li...@zilliz.com>.
Is there anybody able to help on this ?
Regards,
Lizhou
------------------ Original ------------------
From: "高立周"<li...@zilliz.com>;
Date: Fri, Mar 30, 2018 01:40 PM
To: "dev"<de...@parquet.apache.org>;
Subject: poor write performance with optional fields
Hi guys,
I'm testing parquet write performance for optional fields with attached cc file.
The throughput is 125MB/s, but the nvme disks limits is 570MB/s.
So, my question is that whether it is expected performance ?
Is there anything wrong with the test program ?
Thanks,
Lizhou
lizgao@lizgao-ubuntu:~/CLionProjects/parquet-test/cmake-build-debug$ ./parquet_test
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0330 13:28:57.599980 17640 reader-writer-performance.cc:72] BEGIN performance_test
I0330 13:29:13.640228 17640 reader-writer-performance.cc:112] END performance_test
PROFILE: interrupts/evictions/bytes = 1551/857/61200
lizgao@lizgao-ubuntu:~/CLionProjects/parquet-test/cmake-build-debug$ ls -lh parquet_cpp_example.parquet
-rw-r--r-- 1 lizgao lizgao 2.0G Mar 30 13:29 parquet_cpp_example.parquet
lizgao@lizgao-ubuntu:~/CLionProjects/parquet-test/cmake-build-debug$ dd if=/dev/zero of=test bs=1M count=4096 oflag=direct
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 7.48397 s, 574 MB/s
lizgao@lizgao-ubuntu:~/CLionProjects/parquet-test/cmake-build-debug$ dd if=/dev/zero of=test bs=1M count=4096 oflag=direct
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 7.98757 s, 538 MB/s
lizgao@lizgao-ubuntu:~/CLionProjects/parquet-test/cmake-build-debug$ dd if=/dev/zero of=test bs=10M count=400 oflag=direct
400+0 records in
400+0 records out
4194304000 bytes (4.2 GB, 3.9 GiB) copied, 8.03284 s, 522 MB/s
lizgao@lizgao-ubuntu:~/CLionProjects/parquet-test/cmake-build-debug$ dd if=/dev/zero of=test bs=1G count=4 oflag=direct
4+0 records in
4+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 8.30123 s, 517 MB/s