You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brpc.apache.org by GitBox <gi...@apache.org> on 2021/11/20 18:01:07 UTC
[GitHub] [incubator-brpc] emmettwu opened a new pull request #1611: Add streaming lz4 compession
emmettwu opened a new pull request #1611:
URL: https://github.com/apache/incubator-brpc/pull/1611
Compress method Compress size(B) Compress time(us) Decompress time(us) Compress throughput(MB/s) Decompress throughput(MB/s) Compress ratio
Snappy 128 0.630420 0.676840 193.633312 180.353278 37.500000%
Gzip 128 4.886340 0.804860 24.981952 151.666517 47.656250%
Zlib 128 4.376520 0.665340 27.892095 183.470575 38.281250%
Lz4 128 0.614140 0.359200 198.766263 339.839400 58.115935%
Snappy 1024 0.690060 0.697920 1415.184911 1399.247048 8.789062%
Gzip 1024 7.144860 1.752040 136.680425 557.385962 6.640625%
Zlib 1024 6.703100 1.304000 145.688189 748.897623 5.468750%
Lz4 1024 0.781000 0.520440 1250.400128 1876.417070 7.810039%
Snappy 16384 4.155420 4.253955 3760.149466 3673.052421 4.962158%
Gzip 16384 55.165820 19.621680 283.236974 796.313070 0.781250%
Zlib 16384 48.379395 13.014209 322.968077 1200.610811 0.708008%
Lz4 16384 3.494336 3.105762 4471.521994 5030.971921 1.227403%
Snappy 32768 7.633301 8.515039 4093.903921 3669.977292 4.824829%
Gzip 32768 105.522363 37.225879 296.145756 839.469770 0.537109%
Zlib 32768 92.468164 24.807813 337.954152 1259.683819 0.500488%
Lz4 32768 6.346191 4.902539 4924.213280 6374.248038 1.037690%
Snappy 524288 121.535938 143.765625 4114.009488 3477.882839 4.747772%
Gzip 524288 2128.425000 564.970312 234.915489 885.002254 0.305939%
Zlib 524288 1917.210938 367.104688 260.795508 1362.009304 0.303650%
Lz4 524288 105.067188 66.967187 4758.859658 7466.343125 0.857538%
LZ4 compress/decompress throughput outperform snappy, gzip and zlib.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] emmettwu closed pull request #1611: Implement streaming lz4 compession
Posted by GitBox <gi...@apache.org>.
emmettwu closed pull request #1611:
URL: https://github.com/apache/incubator-brpc/pull/1611
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] emmettwu commented on pull request #1611: Implement streaming lz4 compession
Posted by GitBox <gi...@apache.org>.
emmettwu commented on pull request #1611:
URL: https://github.com/apache/incubator-brpc/pull/1611#issuecomment-978815237
本地Makefile编译没问题,travis-ci编译不过。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] emmettwu closed pull request #1611: Implement streaming lz4 compession
Posted by GitBox <gi...@apache.org>.
emmettwu closed pull request #1611:
URL: https://github.com/apache/incubator-brpc/pull/1611
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] emmettwu commented on a change in pull request #1611: Implement streaming lz4 compession
Posted by GitBox <gi...@apache.org>.
emmettwu commented on a change in pull request #1611:
URL: https://github.com/apache/incubator-brpc/pull/1611#discussion_r757737953
##########
File path: src/butil/third_party/lz4/lz4.c
##########
@@ -0,0 +1,2495 @@
+/*
Review comment:
LICENSE更新完成了。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] chenzhangyi commented on a change in pull request #1611: Implement streaming lz4 compession
Posted by GitBox <gi...@apache.org>.
chenzhangyi commented on a change in pull request #1611:
URL: https://github.com/apache/incubator-brpc/pull/1611#discussion_r757882734
##########
File path: src/butil/third_party/lz4/lz4.c
##########
@@ -0,0 +1,2495 @@
+/*
Review comment:
这个文件需要加上额外的namespace,不然如果应用方也使用了lz4的话,会造成链接冲突
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] chenzhangyi commented on a change in pull request #1611: Implement streaming lz4 compession
Posted by GitBox <gi...@apache.org>.
chenzhangyi commented on a change in pull request #1611:
URL: https://github.com/apache/incubator-brpc/pull/1611#discussion_r756563100
##########
File path: src/brpc/policy/lz4_compress.cpp
##########
@@ -0,0 +1,141 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <algorithm>
+
+#include "butil/logging.h"
+#include "butil/third_party/lz4/lz4.h"
+#include "brpc/policy/lz4_compress.h"
+#include "brpc/protocol.h"
+
+
+namespace brpc {
+namespace policy {
+
+bool Lz4Compress(const google::protobuf::Message& res, butil::IOBuf* buf) {
+ butil::IOBuf serialized_pb;
+ butil::IOBufAsZeroCopyOutputStream wrapper(&serialized_pb);
+ if (res.SerializeToZeroCopyStream(&wrapper)) {
+ return Lz4Compress(serialized_pb, buf);
+ }
+ LOG(WARNING) << "Fail to serialize input pb";
+ return false;
+}
+
+bool Lz4Decompress(const butil::IOBuf& data, google::protobuf::Message* req) {
+ butil::IOBuf binary_pb;
+ if (Lz4Decompress(data, &binary_pb)) {
+ return ParsePbFromIOBuf(req, binary_pb);
+ }
+ LOG(WARNING) << "Fail to lz4 decompress, size=" << data.size();
+ return false;
+}
+
+bool Lz4Compress(const butil::IOBuf& in, butil::IOBuf* out) {
+ size_t ref_cnt = in.backing_block_num();
+ LZ4_stream_t* lz4_stream = LZ4_createStream();
+ butil::IOBuf block_buf;
+ std::vector<size_t> block_metas;
Review comment:
不要用size_t, 这个跨架构大小不是确定的。 这里得用int32或者int16.
##########
File path: src/butil/third_party/lz4/lz4.c
##########
@@ -0,0 +1,2495 @@
+/*
Review comment:
需要更新下LICENSE
##########
File path: src/brpc/policy/lz4_compress.cpp
##########
@@ -0,0 +1,141 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <algorithm>
+
+#include "butil/logging.h"
+#include "butil/third_party/lz4/lz4.h"
+#include "brpc/policy/lz4_compress.h"
+#include "brpc/protocol.h"
+
+
+namespace brpc {
+namespace policy {
+
+bool Lz4Compress(const google::protobuf::Message& res, butil::IOBuf* buf) {
+ butil::IOBuf serialized_pb;
+ butil::IOBufAsZeroCopyOutputStream wrapper(&serialized_pb);
+ if (res.SerializeToZeroCopyStream(&wrapper)) {
+ return Lz4Compress(serialized_pb, buf);
+ }
+ LOG(WARNING) << "Fail to serialize input pb";
+ return false;
+}
+
+bool Lz4Decompress(const butil::IOBuf& data, google::protobuf::Message* req) {
+ butil::IOBuf binary_pb;
+ if (Lz4Decompress(data, &binary_pb)) {
+ return ParsePbFromIOBuf(req, binary_pb);
+ }
+ LOG(WARNING) << "Fail to lz4 decompress, size=" << data.size();
+ return false;
+}
+
+bool Lz4Compress(const butil::IOBuf& in, butil::IOBuf* out) {
+ size_t ref_cnt = in.backing_block_num();
+ LZ4_stream_t* lz4_stream = LZ4_createStream();
+ butil::IOBuf block_buf;
+ std::vector<size_t> block_metas;
+ for (size_t i = 0; i < ref_cnt; ++i) {
+ butil::StringPiece block_view = in.backing_block(i);
+ size_t src_block_size = block_view.size();
+ size_t dst_block_bound = LZ4_compressBound(src_block_size);
+ char* dst = new char[dst_block_bound];
+ size_t dst_block_size =
+ LZ4_compress_fast_continue(lz4_stream, block_view.data(), dst,
+ src_block_size, dst_block_bound, 1);
+ block_buf.append_user_data(reinterpret_cast<void *>(dst), dst_block_size,
+ [](void *d) {
+ char *nd = reinterpret_cast<char *>(d);
+ delete[] nd;
+ });
+ block_metas.emplace_back(dst_block_size);
+ block_metas.emplace_back(src_block_size);
+ }
+ size_t nblocks = block_metas.size() / 2;
+ out->append(&nblocks, sizeof(size_t));
Review comment:
不能直接这么写,需要转换成网络序。另外同上,这里不能用size_t
##########
File path: src/brpc/policy/lz4_compress.cpp
##########
@@ -0,0 +1,141 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <algorithm>
+
+#include "butil/logging.h"
+#include "butil/third_party/lz4/lz4.h"
+#include "brpc/policy/lz4_compress.h"
+#include "brpc/protocol.h"
+
+
+namespace brpc {
+namespace policy {
+
+bool Lz4Compress(const google::protobuf::Message& res, butil::IOBuf* buf) {
+ butil::IOBuf serialized_pb;
+ butil::IOBufAsZeroCopyOutputStream wrapper(&serialized_pb);
+ if (res.SerializeToZeroCopyStream(&wrapper)) {
+ return Lz4Compress(serialized_pb, buf);
+ }
Review comment:
建议写一下文档先讨论下对应压缩的wire format, 这里似乎并不是效率最高的。 结合zero copy stream, 这里应该能做到一边序列化一边压缩(至少传输格式上需要能保留这种实现可能)。 现在还是多构造了一次中间数据。
##########
File path: src/brpc/policy/lz4_compress.cpp
##########
@@ -0,0 +1,141 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <algorithm>
+
+#include "butil/logging.h"
+#include "butil/third_party/lz4/lz4.h"
+#include "brpc/policy/lz4_compress.h"
+#include "brpc/protocol.h"
+
+
+namespace brpc {
+namespace policy {
+
+bool Lz4Compress(const google::protobuf::Message& res, butil::IOBuf* buf) {
+ butil::IOBuf serialized_pb;
+ butil::IOBufAsZeroCopyOutputStream wrapper(&serialized_pb);
+ if (res.SerializeToZeroCopyStream(&wrapper)) {
+ return Lz4Compress(serialized_pb, buf);
+ }
+ LOG(WARNING) << "Fail to serialize input pb";
+ return false;
+}
+
+bool Lz4Decompress(const butil::IOBuf& data, google::protobuf::Message* req) {
+ butil::IOBuf binary_pb;
+ if (Lz4Decompress(data, &binary_pb)) {
+ return ParsePbFromIOBuf(req, binary_pb);
+ }
+ LOG(WARNING) << "Fail to lz4 decompress, size=" << data.size();
+ return false;
+}
+
+bool Lz4Compress(const butil::IOBuf& in, butil::IOBuf* out) {
+ size_t ref_cnt = in.backing_block_num();
+ LZ4_stream_t* lz4_stream = LZ4_createStream();
+ butil::IOBuf block_buf;
+ std::vector<size_t> block_metas;
+ for (size_t i = 0; i < ref_cnt; ++i) {
+ butil::StringPiece block_view = in.backing_block(i);
+ size_t src_block_size = block_view.size();
+ size_t dst_block_bound = LZ4_compressBound(src_block_size);
+ char* dst = new char[dst_block_bound];
+ size_t dst_block_size =
+ LZ4_compress_fast_continue(lz4_stream, block_view.data(), dst,
+ src_block_size, dst_block_bound, 1);
+ block_buf.append_user_data(reinterpret_cast<void *>(dst), dst_block_size,
+ [](void *d) {
+ char *nd = reinterpret_cast<char *>(d);
+ delete[] nd;
+ });
+ block_metas.emplace_back(dst_block_size);
+ block_metas.emplace_back(src_block_size);
+ }
+ size_t nblocks = block_metas.size() / 2;
+ out->append(&nblocks, sizeof(size_t));
+ out->append(block_metas.data(), sizeof(size_t) * block_metas.size());
+ out->append(block_buf);
+ LZ4_freeStream(lz4_stream);
+ return true;
+}
+
+bool Lz4Decompress(const butil::IOBuf& in, butil::IOBuf* out) {
+ butil::IOBufBytesIterator buf_iter(in);
+ // nblocks
+ size_t in_size = in.size();
+ if (buf_iter.bytes_left() < sizeof(size_t)) {
+ LOG(ERROR) << "Invalid lz4 decompress buf, size=" << in_size;
+ return false;
+ }
+ size_t nblocks = 0;
+ buf_iter.copy_and_forward(&nblocks, sizeof(size_t));
+ if (nblocks <= 0) {
+ LOG(ERROR) << "Invalid nblocks=" << nblocks;
+ return false;
+ }
+
+ // block_metas
+ if (buf_iter.bytes_left() < nblocks * 2 * sizeof(size_t)) {
+ LOG(ERROR) << "Invalid nblocks=" << nblocks
+ << " bytes_left=" << buf_iter.bytes_left();
+ return false;
+ }
+ std::vector<size_t> block_metas(nblocks * 2, 0);
+ buf_iter.copy_and_forward(block_metas.data(), nblocks * 2 * sizeof(size_t));
Review comment:
这些都得实现为序列化,这里如果考虑压缩的话,可以用varint encoding(protobuf应该有类似的接口)
##########
File path: src/brpc/policy/lz4_compress.cpp
##########
@@ -0,0 +1,141 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <algorithm>
+
+#include "butil/logging.h"
+#include "butil/third_party/lz4/lz4.h"
+#include "brpc/policy/lz4_compress.h"
+#include "brpc/protocol.h"
+
+
+namespace brpc {
+namespace policy {
+
+bool Lz4Compress(const google::protobuf::Message& res, butil::IOBuf* buf) {
+ butil::IOBuf serialized_pb;
+ butil::IOBufAsZeroCopyOutputStream wrapper(&serialized_pb);
+ if (res.SerializeToZeroCopyStream(&wrapper)) {
+ return Lz4Compress(serialized_pb, buf);
+ }
+ LOG(WARNING) << "Fail to serialize input pb";
+ return false;
+}
+
+bool Lz4Decompress(const butil::IOBuf& data, google::protobuf::Message* req) {
+ butil::IOBuf binary_pb;
+ if (Lz4Decompress(data, &binary_pb)) {
+ return ParsePbFromIOBuf(req, binary_pb);
+ }
+ LOG(WARNING) << "Fail to lz4 decompress, size=" << data.size();
+ return false;
+}
+
+bool Lz4Compress(const butil::IOBuf& in, butil::IOBuf* out) {
+ size_t ref_cnt = in.backing_block_num();
+ LZ4_stream_t* lz4_stream = LZ4_createStream();
+ butil::IOBuf block_buf;
+ std::vector<size_t> block_metas;
+ for (size_t i = 0; i < ref_cnt; ++i) {
+ butil::StringPiece block_view = in.backing_block(i);
+ size_t src_block_size = block_view.size();
+ size_t dst_block_bound = LZ4_compressBound(src_block_size);
+ char* dst = new char[dst_block_bound];
+ size_t dst_block_size =
+ LZ4_compress_fast_continue(lz4_stream, block_view.data(), dst,
+ src_block_size, dst_block_bound, 1);
+ block_buf.append_user_data(reinterpret_cast<void *>(dst), dst_block_size,
+ [](void *d) {
+ char *nd = reinterpret_cast<char *>(d);
+ delete[] nd;
+ });
+ block_metas.emplace_back(dst_block_size);
+ block_metas.emplace_back(src_block_size);
+ }
+ size_t nblocks = block_metas.size() / 2;
+ out->append(&nblocks, sizeof(size_t));
+ out->append(block_metas.data(), sizeof(size_t) * block_metas.size());
+ out->append(block_buf);
+ LZ4_freeStream(lz4_stream);
+ return true;
+}
+
+bool Lz4Decompress(const butil::IOBuf& in, butil::IOBuf* out) {
+ butil::IOBufBytesIterator buf_iter(in);
+ // nblocks
+ size_t in_size = in.size();
+ if (buf_iter.bytes_left() < sizeof(size_t)) {
+ LOG(ERROR) << "Invalid lz4 decompress buf, size=" << in_size;
+ return false;
+ }
+ size_t nblocks = 0;
+ buf_iter.copy_and_forward(&nblocks, sizeof(size_t));
+ if (nblocks <= 0) {
+ LOG(ERROR) << "Invalid nblocks=" << nblocks;
+ return false;
+ }
+
+ // block_metas
+ if (buf_iter.bytes_left() < nblocks * 2 * sizeof(size_t)) {
+ LOG(ERROR) << "Invalid nblocks=" << nblocks
+ << " bytes_left=" << buf_iter.bytes_left();
+ return false;
+ }
+ std::vector<size_t> block_metas(nblocks * 2, 0);
+ buf_iter.copy_and_forward(block_metas.data(), nblocks * 2 * sizeof(size_t));
+ size_t block_nbytes = 0;
+ size_t max_block = 0;
+ for (size_t i = 0; i < nblocks; ++i) {
+ block_nbytes += block_metas[2 * i];
+ max_block = std::max(max_block, block_metas[2 * i]);
+ }
+
+ // block_buf
+ if (buf_iter.bytes_left() != block_nbytes) {
+ LOG(ERROR) << "Invalid block_nbytes=" << block_nbytes
+ << " bytes_left=" << buf_iter.bytes_left();
+ return false;
+ }
+ LZ4_streamDecode_t* lz4_stream_decode = LZ4_createStreamDecode();
+ char* in_scratch = new char[max_block];
Review comment:
用DEFNIE_SMALL_ARRAY, 大部分情况下应该不会很大.
##########
File path: src/brpc/global.cpp
##########
@@ -386,6 +387,11 @@ static void GlobalInitializeOrDieImpl() {
if (RegisterCompressHandler(COMPRESS_TYPE_SNAPPY, snappy_compress) != 0) {
exit(1);
}
+ const CompressHandler lz4_compress =
+ { Lz4Compress, Lz4Decompress, "lz4" };
Review comment:
lz4有 frame, block, stream不同模式(记忆中block似乎是压缩比最高的). 这里要么就改成lz4s. 而不是直接用lz4.
##########
File path: src/brpc/policy/lz4_compress.cpp
##########
@@ -0,0 +1,141 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <algorithm>
+
+#include "butil/logging.h"
+#include "butil/third_party/lz4/lz4.h"
+#include "brpc/policy/lz4_compress.h"
+#include "brpc/protocol.h"
+
+
+namespace brpc {
+namespace policy {
+
+bool Lz4Compress(const google::protobuf::Message& res, butil::IOBuf* buf) {
+ butil::IOBuf serialized_pb;
+ butil::IOBufAsZeroCopyOutputStream wrapper(&serialized_pb);
+ if (res.SerializeToZeroCopyStream(&wrapper)) {
+ return Lz4Compress(serialized_pb, buf);
+ }
+ LOG(WARNING) << "Fail to serialize input pb";
+ return false;
+}
+
+bool Lz4Decompress(const butil::IOBuf& data, google::protobuf::Message* req) {
+ butil::IOBuf binary_pb;
+ if (Lz4Decompress(data, &binary_pb)) {
+ return ParsePbFromIOBuf(req, binary_pb);
+ }
+ LOG(WARNING) << "Fail to lz4 decompress, size=" << data.size();
+ return false;
+}
+
+bool Lz4Compress(const butil::IOBuf& in, butil::IOBuf* out) {
+ size_t ref_cnt = in.backing_block_num();
+ LZ4_stream_t* lz4_stream = LZ4_createStream();
+ butil::IOBuf block_buf;
+ std::vector<size_t> block_metas;
+ for (size_t i = 0; i < ref_cnt; ++i) {
+ butil::StringPiece block_view = in.backing_block(i);
+ size_t src_block_size = block_view.size();
+ size_t dst_block_bound = LZ4_compressBound(src_block_size);
+ char* dst = new char[dst_block_bound];
+ size_t dst_block_size =
+ LZ4_compress_fast_continue(lz4_stream, block_view.data(), dst,
+ src_block_size, dst_block_bound, 1);
+ block_buf.append_user_data(reinterpret_cast<void *>(dst), dst_block_size,
+ [](void *d) {
+ char *nd = reinterpret_cast<char *>(d);
+ delete[] nd;
+ });
+ block_metas.emplace_back(dst_block_size);
+ block_metas.emplace_back(src_block_size);
+ }
+ size_t nblocks = block_metas.size() / 2;
+ out->append(&nblocks, sizeof(size_t));
+ out->append(block_metas.data(), sizeof(size_t) * block_metas.size());
+ out->append(block_buf);
+ LZ4_freeStream(lz4_stream);
+ return true;
+}
+
+bool Lz4Decompress(const butil::IOBuf& in, butil::IOBuf* out) {
+ butil::IOBufBytesIterator buf_iter(in);
+ // nblocks
+ size_t in_size = in.size();
+ if (buf_iter.bytes_left() < sizeof(size_t)) {
+ LOG(ERROR) << "Invalid lz4 decompress buf, size=" << in_size;
+ return false;
+ }
+ size_t nblocks = 0;
+ buf_iter.copy_and_forward(&nblocks, sizeof(size_t));
+ if (nblocks <= 0) {
+ LOG(ERROR) << "Invalid nblocks=" << nblocks;
+ return false;
+ }
+
+ // block_metas
+ if (buf_iter.bytes_left() < nblocks * 2 * sizeof(size_t)) {
+ LOG(ERROR) << "Invalid nblocks=" << nblocks
+ << " bytes_left=" << buf_iter.bytes_left();
+ return false;
+ }
+ std::vector<size_t> block_metas(nblocks * 2, 0);
+ buf_iter.copy_and_forward(block_metas.data(), nblocks * 2 * sizeof(size_t));
+ size_t block_nbytes = 0;
+ size_t max_block = 0;
+ for (size_t i = 0; i < nblocks; ++i) {
+ block_nbytes += block_metas[2 * i];
+ max_block = std::max(max_block, block_metas[2 * i]);
+ }
+
+ // block_buf
+ if (buf_iter.bytes_left() != block_nbytes) {
+ LOG(ERROR) << "Invalid block_nbytes=" << block_nbytes
+ << " bytes_left=" << buf_iter.bytes_left();
+ return false;
+ }
+ LZ4_streamDecode_t* lz4_stream_decode = LZ4_createStreamDecode();
+ char* in_scratch = new char[max_block];
+ for (size_t i = 0; i < nblocks; ++i) {
+ size_t src_block_size = block_metas[i * 2];
+ size_t dst_block_size = block_metas[i * 2 + 1];
+ char* out_buf = new char[dst_block_size];
+ buf_iter.copy_and_forward(in_scratch, src_block_size);
+ size_t dcp_size = LZ4_decompress_safe_continue(lz4_stream_decode, in_scratch, out_buf, src_block_size, dst_block_size);
+ if (dcp_size != dst_block_size) {
+ LOG(ERROR) << "Fail to lz4 decompress block, dst_block_size="
+ << dst_block_size << " decompress_size=" << dcp_size;
+ delete[] out_buf;
+ delete[] in_scratch;
+ LZ4_freeStreamDecode(lz4_stream_decode);
+ return false;
+ }
+ out->append_user_data(out_buf, dst_block_size, [](void *d) {
Review comment:
iobuf as zero copy stream这里无法使用么, 必须依赖user_data?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org