You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2023/01/05 16:03:02 UTC

[GitHub] [iotdb] panagiotisl opened a new pull request, #8766: Chimp compression

panagiotisl opened a new pull request, #8766:
URL: https://github.com/apache/iotdb/pull/8766

   ## Description
   This PR adds the Chimp compression algorithm for double and single precision floating point data
   ![chimp-gorilla](https://user-images.githubusercontent.com/1543082/210823173-c833ae79-74ac-4e74-b67d-3ae7baaa7c60.png)
   . 
   
   ### Algorithm.
   Chimp was recently presented in VLDB 2022 (https://www.vldb.org/pvldb/vol15/p3058-liakos.pdf):
   Panagiotis Liakos, Katia Papakonstantinopoulou, Yannis Kotidis:
   Chimp: Efficient Lossless Floating Point Compression for Time Series Databases. Proc. VLDB Endow. 15(11): 3058-3070 (2022)
   The algorithm focuses exclusively on floating point data and takes advantage of more than one earlier values encountered to significantly outperform the state-of-the-art Gorilla algorithm in terms of compression ratio, while preserving its speed.
   The implementations provided here focus on the Chimp128 variation for double precision, that uses 128 previous earlier values, and Chimp64 for single precision, that uses 64 earlier values.
   
   Indicative results for many different time series datasets, that highlight the significant benefits expected with the adoption of Chimp in terms of space savings:
   ![chimp-gorilla](https://user-images.githubusercontent.com/1543082/210823287-c2fd8746-a866-465c-a9bd-c58ed35b0aae.png)
   
   
   
   ### Adoption
   Chimp is already used in the latest releases of DuckDB (https://github.com/duckdb/duckdb/pull/4878)
   
   ### Implementation
   
   The algorithm is implemented to reuse code from GorillaEncoderV2.java and GorillaDecoderV2.java, and Long, DoublePrecision, Int and Float versions have been built on top of it. Method organization, design, and naming has been based on the respective Gorilla classes.
   
   ### Testing
   
   A new class named ChimpDecoderTest executes all tests implemented in the GorillaDecoderV2Test class, ensuring the threshold for code coverage
   
   <hr>
   
   This PR has:
   - [X ] been self-reviewed.
       - [ ] concurrent read
       - [ ] concurrent write
       - [ ] concurrent read and write 
   - [ ] added documentation for new or modified features or behaviors.
   - [ ] added Javadocs for most classes and all non-trivial methods. 
   - [ ] added or updated version, __license__, or notice information
   - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious 
     for an unfamiliar reader.
   - [X ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold 
     for code coverage.
   - [ ] added integration tests.
   - [ ] been tested in a test IoTDB cluster.
   
   
   <hr>
   
   ##### Key changed/added classes (or packages if there are too many classes) in this PR
   tsfile/src/main/java/org/apache/iotdb/tsfile/encoding/encoder/IntChimpEncoder.java
   tsfile/src/main/java/org/apache/iotdb/tsfile/encoding/encoder/LongChimpEncoder.java
   tsfile/src/main/java/org/apache/iotdb/tsfile/encoding/encoder/SinglePrecisionChimpEncoder.java
   tsfile/src/main/java/org/apache/iotdb/tsfile/encoding/encoder/DoublePrecisionChimpEncoder.java
   
   tsfile/src/main/java/org/apache/iotdb/tsfile/encoding/encoder/IntChimpDecoder.java
   tsfile/src/main/java/org/apache/iotdb/tsfile/encoding/encoder/LongChimpDecoder.java
   tsfile/src/main/java/org/apache/iotdb/tsfile/encoding/encoder/SinglePrecisionChimpDecoder.java
   tsfile/src/main/java/org/apache/iotdb/tsfile/encoding/encoder/DoublePrecisionChimpDecoder.java


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [iotdb] panagiotisl commented on pull request #8766: Chimp compression

Posted by GitBox <gi...@apache.org>.
panagiotisl commented on PR #8766:
URL: https://github.com/apache/iotdb/pull/8766#issuecomment-1377136003

   Also some timing comparisons/experiments with three different datasets:
   
   # 5,000,000 values (Stocks-Germany)
   GORILLA Encoding time: 1163
   GORILLA Decoding time: 158
   CHIMP Encoding time: 553
   CHIMP Decoding time: 114
   
   # 2,905,887 values (city-temperature)
   GORILLA Encoding time: 855
   GORILLA Decoding time: 97
   CHIMP Encoding time: 412
   CHIMP Decoding time: 99
   
   # 8,927 values (SSD-benchmark)
   GORILLA Encoding time: 28
   GORILLA Decoding time: 7
   CHIMP Encoding time: 7
   CHIMP Decoding time: 6
   
   These timings refer to encoding the values to a byte array and decoding them using the Java code only. If writing the data to disk is involved the speed up that Chimp can offer will be more evident, as it usually needs to write much less bytes to the disk.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [iotdb] HTHou commented on pull request #8766: Chimp compression

Posted by GitBox <gi...@apache.org>.
HTHou commented on PR #8766:
URL: https://github.com/apache/iotdb/pull/8766#issuecomment-1376635141

   I did a simple test. After executed the following sql.
   
   ![136561673316878_ pic](https://user-images.githubusercontent.com/25913899/211446856-f7787c4f-9ec3-42fb-9e84-4ccebfe88f9c.jpg)
   
   The tsfile size with Chimp is 243 bytes, the tsfile size with Gorilla is 248 bytes. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [iotdb] qiaojialin commented on pull request #8766: Chimp compression

Posted by GitBox <gi...@apache.org>.
qiaojialin commented on PR #8766:
URL: https://github.com/apache/iotdb/pull/8766#issuecomment-1374414511

   Hi Panagiotis, thanks for contribution!  Please update the UserGuide, then users could see this encoding: 
   
   docs/UserGuide/Data-Concept/Encoding.md  
   docs/zh/UserGuide/Data-Concept/Encoding.md
   
   You could just copy the English version into docs/zh/UserGuide/Data-Concept/Encoding.md, then we could translate in the review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [iotdb] panagiotisl commented on a diff in pull request #8766: Chimp compression

Posted by GitBox <gi...@apache.org>.
panagiotisl commented on code in PR #8766:
URL: https://github.com/apache/iotdb/pull/8766#discussion_r1063355884


##########
tsfile/src/main/java/org/apache/iotdb/tsfile/encoding/decoder/DoublePrecisionChimpDecoder.java:
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.tsfile.encoding.decoder;
+
+import java.nio.ByteBuffer;
+
+/**
+ * This class includes code modified from Panagiotis Liakos chimp project.
+ *
+ * <p>Copyright: 2022- Panagiotis Liakos, Katia Papakonstantinopoulou and Yannis Kotidis
+ *
+ * <p>Project page: https://github.com/panagiotisl/chimp
+ *
+ * <p>License: https://github.com/panagiotisl/chimp/blob/main/LICENCE.md

Review Comment:
   Thanks a lot! I've update the license to the code in the PR, as well as the original github repo.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [iotdb] HTHou commented on a diff in pull request #8766: Chimp compression

Posted by GitBox <gi...@apache.org>.
HTHou commented on code in PR #8766:
URL: https://github.com/apache/iotdb/pull/8766#discussion_r1063335356


##########
tsfile/src/main/java/org/apache/iotdb/tsfile/encoding/decoder/DoublePrecisionChimpDecoder.java:
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.tsfile.encoding.decoder;
+
+import java.nio.ByteBuffer;
+
+/**
+ * This class includes code modified from Panagiotis Liakos chimp project.
+ *
+ * <p>Copyright: 2022- Panagiotis Liakos, Katia Papakonstantinopoulou and Yannis Kotidis
+ *
+ * <p>Project page: https://github.com/panagiotisl/chimp
+ *
+ * <p>License: https://github.com/panagiotisl/chimp/blob/main/LICENCE.md

Review Comment:
   BSD, MIT, or Apache 2.0 is ok.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [iotdb] HTHou commented on pull request #8766: Chimp compression

Posted by GitBox <gi...@apache.org>.
HTHou commented on PR #8766:
URL: https://github.com/apache/iotdb/pull/8766#issuecomment-1374428776

   Hi, there are more code to be update, in order to make Chimp can be used in IoTDB.
   
   server/src/main/java/org/apache/iotdb/db/utils/SchemaUtils.java
   tsfile/src/main/java/org/apache/iotdb/tsfile/encoding/encoder/TSEncodingBuilder.java
   tsfile/src/main/java/org/apache/iotdb/tsfile/encoding/decoder/Decoder.java
   tsfile/src/main/java/org/apache/iotdb/tsfile/file/metadata/enums/TSEncoding.java getTsEncoding method
   client-cpp/src/main/Session.h
   client-py/iotdb/utils/IoTDBConstants.py
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [iotdb] panagiotisl commented on pull request #8766: Chimp compression

Posted by GitBox <gi...@apache.org>.
panagiotisl commented on PR #8766:
URL: https://github.com/apache/iotdb/pull/8766#issuecomment-1373356054

   Some checks are failing but I don't think my PR has anything to do with the failures.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [iotdb] HTHou commented on a diff in pull request #8766: Chimp compression

Posted by GitBox <gi...@apache.org>.
HTHou commented on code in PR #8766:
URL: https://github.com/apache/iotdb/pull/8766#discussion_r1063284219


##########
tsfile/src/main/java/org/apache/iotdb/tsfile/encoding/decoder/DoublePrecisionChimpDecoder.java:
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.tsfile.encoding.decoder;
+
+import java.nio.ByteBuffer;
+
+/**
+ * This class includes code modified from Panagiotis Liakos chimp project.
+ *
+ * <p>Copyright: 2022- Panagiotis Liakos, Katia Papakonstantinopoulou and Yannis Kotidis
+ *
+ * <p>Project page: https://github.com/panagiotisl/chimp
+ *
+ * <p>License: https://github.com/panagiotisl/chimp/blob/main/LICENCE.md

Review Comment:
   Hi, thanks so much for your contribution!
   
   I have a question about the license. Is the license of your code GPLv3? GPLv3 is actually not allowed to be imported in an Apache 2.0 project. Any particular reason to choose it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [iotdb] panagiotisl commented on a diff in pull request #8766: Chimp compression

Posted by GitBox <gi...@apache.org>.
panagiotisl commented on code in PR #8766:
URL: https://github.com/apache/iotdb/pull/8766#discussion_r1063355884


##########
tsfile/src/main/java/org/apache/iotdb/tsfile/encoding/decoder/DoublePrecisionChimpDecoder.java:
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.tsfile.encoding.decoder;
+
+import java.nio.ByteBuffer;
+
+/**
+ * This class includes code modified from Panagiotis Liakos chimp project.
+ *
+ * <p>Copyright: 2022- Panagiotis Liakos, Katia Papakonstantinopoulou and Yannis Kotidis
+ *
+ * <p>Project page: https://github.com/panagiotisl/chimp
+ *
+ * <p>License: https://github.com/panagiotisl/chimp/blob/main/LICENCE.md

Review Comment:
   Thanks a lot! I've update the license to Apache 2.0, in the code in the PR, as well as the original github repo.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [iotdb] panagiotisl commented on pull request #8766: Chimp compression

Posted by GitBox <gi...@apache.org>.
panagiotisl commented on PR #8766:
URL: https://github.com/apache/iotdb/pull/8766#issuecomment-1377079167

   > 
   
   I have used the above example to make tests with 4 different configurations using the first 1,000 values of the basel-wind dataset (see Figure in the PR text). The results are:
   
   Chimp (with snappy): 4695 bytes
   Gorilla (with snappy): 8527 bytes
   Chimp (uncompressed): 4812 bytes
   Gorilla (uncompressed): 8653 bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [iotdb] HTHou commented on a diff in pull request #8766: Chimp compression

Posted by "HTHou (via GitHub)" <gi...@apache.org>.
HTHou commented on code in PR #8766:
URL: https://github.com/apache/iotdb/pull/8766#discussion_r1090423603


##########
docs/zh/UserGuide/Data-Concept/Encoding.md:
##########
@@ -63,6 +63,12 @@ GORILLA 编码是一种无损编码，它比较适合编码前后值比较接近
 
 ZigZag编码将有符号整型映射到无符号整型，适合比较小的整数。
 
+* CHIMP
+
+CHIMP encoding is lossless. It is the state-of-the-art compression algorithm for streaming floating point data, providing impressive savings compared to earlier approaches. It is suitable for any numerical sequence with similar values and works best for sequence data without large fluctuations and/or random noise.
+
+Usage restrictions: When using CHIMP to encode INT32 data, you need to ensure that there is no data point with the value `Integer.MIN_VALUE` in the sequence. When using CHIMP to encode INT64 data, you need to ensure that there is no data point with the value `Long.MIN_VALUE` in the sequence.
+

Review Comment:
   ```suggestion
   CHIMP 是一种无损编码。它是一种新的流式浮点数据压缩算法，可以节省存储空间。这个编码适用于前后值比较接近的数值序列，对波动小和随机噪声少的序列数据更加友好。
   
   使用限制：如果对 INT32 类型数据使用 CHIMP 编码，需要确保数据点中没有 `Integer.MIN_VALUE`。 如果对 INT64 类型数据使用 CHIMP 编码，需要确保数据点中没有 `Long.MIN_VALUE`。
   
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [iotdb] HTHou merged pull request #8766: Chimp compression

Posted by "HTHou (via GitHub)" <gi...@apache.org>.
HTHou merged PR #8766:
URL: https://github.com/apache/iotdb/pull/8766


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [iotdb] panagiotisl commented on a diff in pull request #8766: Chimp compression

Posted by GitBox <gi...@apache.org>.
panagiotisl commented on code in PR #8766:
URL: https://github.com/apache/iotdb/pull/8766#discussion_r1063293330


##########
tsfile/src/main/java/org/apache/iotdb/tsfile/encoding/decoder/DoublePrecisionChimpDecoder.java:
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.tsfile.encoding.decoder;
+
+import java.nio.ByteBuffer;
+
+/**
+ * This class includes code modified from Panagiotis Liakos chimp project.
+ *
+ * <p>Copyright: 2022- Panagiotis Liakos, Katia Papakonstantinopoulou and Yannis Kotidis
+ *
+ * <p>Project page: https://github.com/panagiotisl/chimp
+ *
+ * <p>License: https://github.com/panagiotisl/chimp/blob/main/LICENCE.md

Review Comment:
   We can update the license,  can you please let me know which are the available options?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org