You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "steFaiz (via GitHub)" <gi...@apache.org> on 2026/07/15 03:27:57 UTC

[PR] [core] Support Map type for Blob File Format [paimon]

steFaiz opened a new pull request, #8635:
URL: https://github.com/apache/paimon/pull/8635

   ### Purpose
   
   ### Tests
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [format] Support Map type for Blob File Format [paimon]

Posted by "steFaiz (via GitHub)" <gi...@apache.org>.
steFaiz commented on PR #8635:
URL: https://github.com/apache/paimon/pull/8635#issuecomment-5030805856

   @JingsongLi  This PR is ready to review! PTAL if you have some time! Thanks!
   As I said, this PR is very big...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [format] Support Map type for Blob File Format [paimon]

Posted by "steFaiz (via GitHub)" <gi...@apache.org>.
steFaiz commented on code in PR #8635:
URL: https://github.com/apache/paimon/pull/8635#discussion_r3620128794


##########
paimon-api/src/main/java/org/apache/paimon/types/BlobType.java:
##########
@@ -118,6 +118,25 @@ public static boolean isBlobFileField(DataType type) {
         if (type.getTypeRoot() == DataTypeRoot.ARRAY) {
             return ((ArrayType) type).getElementType().getTypeRoot() == DataTypeRoot.BLOB;
         }
+        if (type.getTypeRoot() == DataTypeRoot.MAP) {
+            MapType mapType = (MapType) type;
+            return mapType.getValueType().getTypeRoot() == DataTypeRoot.BLOB
+                    && isSupportedMapBlobKeyType(mapType.getKeyType());
+        }
         return false;
     }
+
+    private static boolean isSupportedMapBlobKeyType(DataType keyType) {

Review Comment:
   I was meant to fast-fail on creating table. Do you think we do not have to fast fail here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [format] Support Map type for Blob File Format [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8635:
URL: https://github.com/apache/paimon/pull/8635#issuecomment-5009843140

   > @JingsongLi Of course! But this PR only focus on file format layer modification. Wiring to Table & Engine layer will be brought in subsequent PRs.
   > 
   > I'm afraid that modifying all layers in one PR is too big to review.
   
   @steFaiz It is OK in a single PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [format] Support Map type for Blob File Format [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8635:
URL: https://github.com/apache/paimon/pull/8635#discussion_r3620099244


##########
paimon-format/src/main/java/org/apache/paimon/format/blob/BlobFileFormat.java:
##########
@@ -60,6 +58,14 @@ public class BlobFileFormat extends FileFormat {
 
     @Nullable public BlobConsumer writeConsumer;
 
+    public BlobFileFormat() {

Review Comment:
   Do not add constructors for tests. Please check whole PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [format] Support Map type for Blob File Format [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8635:
URL: https://github.com/apache/paimon/pull/8635#discussion_r3620041661


##########
paimon-api/src/main/java/org/apache/paimon/types/BlobType.java:
##########
@@ -118,6 +118,25 @@ public static boolean isBlobFileField(DataType type) {
         if (type.getTypeRoot() == DataTypeRoot.ARRAY) {
             return ((ArrayType) type).getElementType().getTypeRoot() == DataTypeRoot.BLOB;
         }
+        if (type.getTypeRoot() == DataTypeRoot.MAP) {
+            MapType mapType = (MapType) type;
+            return mapType.getValueType().getTypeRoot() == DataTypeRoot.BLOB
+                    && isSupportedMapBlobKeyType(mapType.getKeyType());
+        }
         return false;
     }
+
+    private static boolean isSupportedMapBlobKeyType(DataType keyType) {

Review Comment:
   Why need to limit this? Just let runtime to check this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [format] Support Map type for Blob File Format [paimon]

Posted by "steFaiz (via GitHub)" <gi...@apache.org>.
steFaiz commented on code in PR #8635:
URL: https://github.com/apache/paimon/pull/8635#discussion_r3620125382


##########
paimon-format/src/main/java/org/apache/paimon/format/blob/BlobFileFormat.java:
##########
@@ -60,6 +58,14 @@ public class BlobFileFormat extends FileFormat {
 
     @Nullable public BlobConsumer writeConsumer;
 
+    public BlobFileFormat() {

Review Comment:
   Thanks! My Bad, fixing it!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [format] Support Map type for Blob File Format [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8635:
URL: https://github.com/apache/paimon/pull/8635


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [format] Support Map type for Blob File Format [paimon]

Posted by "steFaiz (via GitHub)" <gi...@apache.org>.
steFaiz commented on PR #8635:
URL: https://github.com/apache/paimon/pull/8635#issuecomment-5033675916

   @JingsongLi  Thanks for your review! All reviews addressed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [format] Support Map type for Blob File Format [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8635:
URL: https://github.com/apache/paimon/pull/8635#discussion_r3620332207


##########
paimon-api/src/main/java/org/apache/paimon/types/BlobType.java:
##########
@@ -118,6 +118,25 @@ public static boolean isBlobFileField(DataType type) {
         if (type.getTypeRoot() == DataTypeRoot.ARRAY) {
             return ((ArrayType) type).getElementType().getTypeRoot() == DataTypeRoot.BLOB;
         }
+        if (type.getTypeRoot() == DataTypeRoot.MAP) {
+            MapType mapType = (MapType) type;
+            return mapType.getValueType().getTypeRoot() == DataTypeRoot.BLOB
+                    && isSupportedMapBlobKeyType(mapType.getKeyType());
+        }
         return false;
     }
+
+    private static boolean isSupportedMapBlobKeyType(DataType keyType) {

Review Comment:
   Yes, Ultimately, we should support all key types supported by Paimon.



##########
paimon-api/src/main/java/org/apache/paimon/types/BlobType.java:
##########
@@ -118,6 +118,25 @@ public static boolean isBlobFileField(DataType type) {
         if (type.getTypeRoot() == DataTypeRoot.ARRAY) {
             return ((ArrayType) type).getElementType().getTypeRoot() == DataTypeRoot.BLOB;
         }
+        if (type.getTypeRoot() == DataTypeRoot.MAP) {
+            MapType mapType = (MapType) type;
+            return mapType.getValueType().getTypeRoot() == DataTypeRoot.BLOB
+                    && isSupportedMapBlobKeyType(mapType.getKeyType());
+        }
         return false;
     }
+
+    private static boolean isSupportedMapBlobKeyType(DataType keyType) {

Review Comment:
   Ultimately, we should support all key types supported by Paimon.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [format] Support Map type for Blob File Format [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8635:
URL: https://github.com/apache/paimon/pull/8635#issuecomment-4988540364

   cc @leaves12138 to take a look.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [format] Support Map type for Blob File Format [paimon]

Posted by "steFaiz (via GitHub)" <gi...@apache.org>.
steFaiz commented on PR #8635:
URL: https://github.com/apache/paimon/pull/8635#issuecomment-4989192632

   @JingsongLi  Of course! But this PR only focus on file format layer modification. Wiring to Table & Engine layer will be brought in subsequent PRs.
   
   I'm afraid that modifying all layers in one PR is too big to review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [format] Support Map type for Blob File Format [paimon]

Posted by "steFaiz (via GitHub)" <gi...@apache.org>.
steFaiz closed pull request #8635: [format] Support Map<X, Blob> type for Blob File Format
URL: https://github.com/apache/paimon/pull/8635


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [format] Support Map type for Blob File Format [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8635:
URL: https://github.com/apache/paimon/pull/8635#discussion_r3620049969


##########
paimon-core/src/main/java/org/apache/paimon/blob/PrimaryKeyBlobExternalizer.java:
##########
@@ -32,31 +34,52 @@
 import org.apache.paimon.io.DataFilePathFactory;
 import org.apache.paimon.types.ArrayType;
 import org.apache.paimon.types.DataField;
+import org.apache.paimon.types.DataType;
 import org.apache.paimon.types.DataTypeRoot;
+import org.apache.paimon.types.DataTypes;
+import org.apache.paimon.types.MapType;
 import org.apache.paimon.types.RowKind;
 import org.apache.paimon.types.RowType;
 import org.apache.paimon.utils.IOUtils;
+import org.apache.paimon.utils.InternalRowUtils;
 import org.apache.paimon.utils.RowDataToObjectArrayConverter;
 
 import java.io.IOException;
 import java.util.ArrayList;
-import java.util.Collections;
+import java.util.LinkedHashMap;
 import java.util.List;
+import java.util.Map;
 import java.util.Set;
 import java.util.TreeSet;
 
+import static org.apache.paimon.types.BlobType.isBlobFileField;
 import static org.apache.paimon.utils.Preconditions.checkArgument;
 
 /** Externalizes primary-key BLOB values before they enter the MergeTree write buffer. */
 public class PrimaryKeyBlobExternalizer {
 
+    private static final RowType BLOB_ROW_TYPE = RowType.of(DataTypes.BLOB());
+
     private final FileIO fileIO;
     private final RowDataToObjectArrayConverter rowConverter;
-    private final int[] blobFieldIndexes;
-    private final boolean[] blobArrayFields;
-    private final ManagedBlobPackWriter[] packWriters;
+    private final ManagedBlobField[] blobFields;
     private final List<Path> uncommittedPacks;
 
+    public PrimaryKeyBlobExternalizer(

Review Comment:
   Do not add constructor for test codes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org