You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "tsreaper (via GitHub)" <gi...@apache.org> on 2024/08/28 09:34:48 UTC
[PR] [avro] Support map with non-string key type [paimon]
tsreaper opened a new pull request, #4088:
URL: https://github.com/apache/paimon/pull/4088
### Purpose
Currently, Paimon avro format only supports map with string key type. This PR supports map with non-string key type by storing map as an array of entries, just like how Iceberg stores avro maps. This PR is also a pre-requisite of Iceberg compatible statistics, because Iceberg stores statistics as maps in metadata files.
### Tests
Unit tests.
### API and Format
No storage format changes.
### Documentation
No need for document.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [avro] Support map with non-string key type [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #4088:
URL: https://github.com/apache/paimon/pull/4088
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [avro] Support map with non-string key type [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4088:
URL: https://github.com/apache/paimon/pull/4088#discussion_r1735477425
##########
paimon-format/src/main/java/org/apache/paimon/format/avro/AvroSchemaConverter.java:
##########
@@ -188,14 +188,38 @@ public static Schema convertToSchema(
return nullable ? nullableSchema(record) : record;
case MULTISET:
case MAP:
- Schema map =
- SchemaBuilder.builder()
- .map()
- .values(
- convertToSchema(
- extractValueTypeToAvroMap(dataType),
- rowName,
- rowNameMapping));
+ DataType keyType = extractKeyTypeToAvroMap(dataType);
+ DataType valueType = extractValueTypeToAvroMap(dataType);
+ Schema map;
+ if (isArrayMap(dataType)) {
Review Comment:
isStringKeyMap?
##########
paimon-format/src/main/java/org/apache/paimon/format/avro/AvroSchemaConverter.java:
##########
@@ -188,14 +188,38 @@ public static Schema convertToSchema(
return nullable ? nullableSchema(record) : record;
case MULTISET:
case MAP:
- Schema map =
- SchemaBuilder.builder()
- .map()
- .values(
- convertToSchema(
- extractValueTypeToAvroMap(dataType),
- rowName,
- rowNameMapping));
+ DataType keyType = extractKeyTypeToAvroMap(dataType);
+ DataType valueType = extractValueTypeToAvroMap(dataType);
+ Schema map;
+ if (isArrayMap(dataType)) {
+ map =
+ SchemaBuilder.builder()
+ .map()
+ .values(convertToSchema(valueType, rowName, rowNameMapping));
+ } else {
+ String mapRowName = rowName + "_kv";
Review Comment:
add a comment to explain using `array<struct>` to represent a map.
##########
paimon-format/src/main/java/org/apache/paimon/format/avro/AvroSchemaConverter.java:
##########
@@ -188,14 +188,38 @@ public static Schema convertToSchema(
return nullable ? nullableSchema(record) : record;
case MULTISET:
case MAP:
- Schema map =
- SchemaBuilder.builder()
- .map()
- .values(
- convertToSchema(
- extractValueTypeToAvroMap(dataType),
- rowName,
- rowNameMapping));
+ DataType keyType = extractKeyTypeToAvroMap(dataType);
+ DataType valueType = extractValueTypeToAvroMap(dataType);
+ Schema map;
+ if (isArrayMap(dataType)) {
+ map =
+ SchemaBuilder.builder()
+ .map()
+ .values(convertToSchema(valueType, rowName, rowNameMapping));
+ } else {
+ String mapRowName = rowName + "_kv";
+ SchemaBuilder.GenericDefault<Schema> kvBuilder =
+ SchemaBuilder.builder()
+ .record(mapRowName)
+ .fields()
+ .name("key")
+ .type(convertToSchema(keyType, mapRowName, rowNameMapping))
+ .noDefault()
+ .name("value")
+ .type(convertToSchema(valueType, mapRowName, rowNameMapping));
+ if (valueType.isNullable()) {
+ map =
+ SchemaBuilder.builder()
+ .array()
+ .items(kvBuilder.withDefault(null).endRecord());
Review Comment:
just:
`FieldAssembler<Schema> assembler = valueType.isNullable() ? kvBuilder.withDefault(null) : kvBuilder.noDefault();`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [avro] Support map with non-string key type [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4088:
URL: https://github.com/apache/paimon/pull/4088#discussion_r1735482383
##########
paimon-format/src/test/java/org/apache/paimon/format/avro/AvroFormatReadWriteTest.java:
##########
@@ -34,4 +53,38 @@ protected AvroFormatReadWriteTest() {
protected FileFormat fileFormat() {
return new AvroFileFormat(new FileFormatFactory.FormatContext(new Options(), 1024));
}
+
+ @Test
+ public void testMapTypeNonStringKey() throws Exception {
Review Comment:
Also add test in `FormatReadWriteTest`, maybe you can just add `testFullTypes`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [avro] Support map with non-string key type [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #4088:
URL: https://github.com/apache/paimon/pull/4088#issuecomment-2316682054
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org