You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "tsreaper (via GitHub)" <gi...@apache.org> on 2025/09/15 08:48:52 UTC
[PR] [avro] Optimize map in avro [paimon]
tsreaper opened a new pull request, #6256:
URL: https://github.com/apache/paimon/pull/6256
### Purpose
`map<string, string>` is a useful data type in many tables. However, current `map<string, string>` in Paimon avro is slow, because Paimon reader decodes it into `HashMap`s.
This PR optimizes `map<string, string>` by storing the original bytes from avro, so in scenarios like compacting, where we don't care about the exact values of a row, we don't need to pay the decoding and encoding costs. When the exact value is needed, these bytes will be decoded into `BinaryArray`s.
### Tests
* `FormatReadWriteTest`
### API and Format
No format changes.
### Documentation
No new feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [avro] Optimize map in avro [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #6256:
URL: https://github.com/apache/paimon/pull/6256
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [avro] Optimize map in avro [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #6256:
URL: https://github.com/apache/paimon/pull/6256#discussion_r2348976912
##########
paimon-common/src/main/java/org/apache/paimon/data/BinaryWriter.java:
##########
@@ -62,7 +62,11 @@ public interface BinaryWriter {
void writeString(int pos, BinaryString value);
- void writeBinary(int pos, byte[] bytes);
+ default void writeBinary(int pos, byte[] bytes) {
Review Comment:
Do not introduce to performance critical part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [avro] Optimize map in avro [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #6256:
URL: https://github.com/apache/paimon/pull/6256#discussion_r2354699773
##########
paimon-format/src/main/java/org/apache/paimon/format/avro/AvroBytesStringMap.java:
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.format.avro;
+
+import org.apache.paimon.data.BinaryArray;
+import org.apache.paimon.data.BinaryArrayWriter;
+import org.apache.paimon.data.InternalArray;
+import org.apache.paimon.data.InternalMap;
+import org.apache.paimon.types.DataTypes;
+import org.apache.paimon.utils.IntArrayList;
+
+import org.apache.avro.io.BinaryData;
+import org.apache.avro.io.BinaryDecoder;
+
+import java.io.IOException;
+
+/** An {@link InternalMap} stored in avro format bytes. */
+public class AvroBytesStringMap implements InternalMap {
+
+ private static final int ELEMENT_SIZE =
+ BinaryArray.calculateFixLengthPartSize(DataTypes.STRING());
+
+ // map size
+ private int size;
+ // stores original bytes from avro files
+ private byte[] bytes;
+ // total bytes length
+ private int lengthInBytes;
+ // offset of each string
+ private final IntArrayList off;
+ // length of each string
+ private final IntArrayList len;
+
+ private BinaryArray keyArray;
+ private BinaryArray valueArray;
+
+ public AvroBytesStringMap(BinaryDecoder decoder, boolean valueNullable) throws IOException {
Review Comment:
Can you create a static method for this? And just pass all fields to this constructor.
In this way, `AvroBytesStringMap` can be an immutable class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org