You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "maltesander (via GitHub)" <gi...@apache.org> on 2026/07/03 06:19:53 UTC
[PR] NIFI-16069 - PutIcebergRecord fails with ClassCastException when writing complex types (arrays, maps, nested records) [nifi]
maltesander opened a new pull request, #11391:
URL: https://github.com/apache/nifi/pull/11391
# Summary
[NIFI-16069](https://issues.apache.org/jira/browse/NIFI-16069) - PutIcebergRecord fails with ClassCastException when writing complex types (arrays, maps, nested records)
PutIcebergRecord fails to write FlowFiles whose schema contains complex/nested types like Iceberg list, map, or struct columns. RecordConverter only translates top-level scalar values (java.sql timestamp/date/time -> java.time) and passes complex values through unchanged.
As a result, values reach Iceberg's Parquet writer in NiFi's native representation, which is incompatible with what Iceberg expects:
- Nested records arrive as org.apache.nifi.serialization.record.MapRecord but Iceberg requires org.apache.iceberg.StructLike.
- Array fields arrive as Object[] but Iceberg's writer requires a java.util.Collection.
- Maps and elements/values nested inside these types are likewise not converted (e.g. a date inside an array or map value).
Because conversion is gated on scalar field types only, records consisting solely of complex fields skip conversion entirely.
# Tracking
Please complete the following tracking steps prior to pull request creation.
### Issue Tracking
- [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created
### Pull Request Tracking
- Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000`
- Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000`
- Pull request contains [commits signed](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits) with a registered key indicating `Verified` status
### Pull Request Formatting
- Pull Request based on current revision of the `main` branch
- Pull Request refers to a feature branch with one commit containing changes
# Verification
Please indicate the verification steps performed prior to pull request creation.
### Build
- [ ] Build completed using `./mvnw clean install -P contrib-check`
- [ ] JDK 21
- [ ] JDK 25
### Licensing
- [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html)
- [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files
### Documentation
- [ ] Documentation formatting appears as expected in rendered files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-16069 - PutIcebergRecord fails with ClassCastException when writing complex types (arrays, maps, nested records) [nifi]
Posted by "maltesander (via GitHub)" <gi...@apache.org>.
maltesander commented on PR #11391:
URL: https://github.com/apache/nifi/pull/11391#issuecomment-5435062952
> Thanks for your patience @maltesander. I'm open to addressing the multiple timestamp types in this pull request, but deferring it to a separate issue also works.
>
> I noted a few remaining minor recommendations.
No worries. Tried to adapt to the review:
- removed early returns
- added missing method header comments for parameters and return values
I pushed the timestamp changes here https://github.com/apache/nifi/pull/11391/commits/270dc35ba3b33a3da6e4fe684986b1c9952a1a67 and added it to the PR description (not the jira ticket).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-16069 - PutIcebergRecord fails with ClassCastException when writing complex types (arrays, maps, nested records) [nifi]
Posted by "maltesander (via GitHub)" <gi...@apache.org>.
maltesander commented on code in PR #11391:
URL: https://github.com/apache/nifi/pull/11391#discussion_r3563489879
##########
nifi-extension-bundles/nifi-iceberg-bundle/nifi-iceberg-processors/src/main/java/org/apache/nifi/processors/iceberg/record/RecordConverter.java:
##########
@@ -39,63 +43,107 @@ class RecordConverter {
private static final Set<RecordFieldType> CONVERSION_REQUIRED_FIELD_TYPES = Set.of(
RecordFieldType.TIMESTAMP,
RecordFieldType.DATE,
- RecordFieldType.TIME
+ RecordFieldType.TIME,
+ RecordFieldType.ARRAY,
+ RecordFieldType.RECORD,
+ RecordFieldType.MAP
);
/**
- * Get Converted Record with conditional handling for field values requiring translation
+ * Get Converted Record with recursive, schema-aware handling for field values requiring translation
*
* @param inputRecord Input Record to be converted
+ * @param struct Iceberg Struct Type describing the target field types (may be null for scalar-only conversion)
* @return Input Record or new Record with converted field values
*/
- static Record getConvertedRecord(final Record inputRecord) {
- final Record convertedRecord;
-
+ static Record getConvertedRecord(final Record inputRecord, final Types.StructType struct) {
final RecordSchema recordSchema = inputRecord.getSchema();
- if (isConversionRequired(recordSchema)) {
- final Map<String, Object> values = inputRecord.toMap();
- convertedRecord = getConvertedRecord(recordSchema, values);
- } else {
- convertedRecord = inputRecord;
+ if (!isConversionRequired(recordSchema)) {
+ return inputRecord;
}
- return convertedRecord;
- }
-
- private static Record getConvertedRecord(final RecordSchema recordSchema, final Map<String, Object> values) {
+ final Map<String, Object> values = inputRecord.toMap();
final Map<String, Object> convertedValues = new LinkedHashMap<>();
-
for (final Map.Entry<String, Object> entry : values.entrySet()) {
final String field = entry.getKey();
- final Object value = entry.getValue();
- final Object converted = getConvertedValue(value);
- convertedValues.put(field, converted);
+ final Type fieldType = fieldType(struct, field);
+ convertedValues.put(field, convertValue(entry.getValue(), fieldType));
}
return new MapRecord(recordSchema, convertedValues);
}
- private static Object getConvertedValue(final Object value) {
+ static Object convertValue(final Object value, final Type icebergType) {
return switch (value) {
// Convert java.sql types to corresponding java.time types for Apache Iceberg
case Timestamp timestamp -> timestamp.toLocalDateTime();
case Date date -> date.toLocalDate();
case Time time -> time.toLocalTime();
- case null, default -> value;
+ // Recursively convert complex types against the matching Iceberg type
+ case null, default -> convertComplexValue(value, icebergType);
};
}
- private static boolean isConversionRequired(final RecordSchema recordSchema) {
- final List<RecordField> fields = recordSchema.getFields();
+ /**
+ * Recursively convert array, collection, nested record, and map values against the matching Iceberg type.
+ * The value is returned unchanged when the target Iceberg type is unknown or does not describe a complex type
+ * matching the value.
+ */
+ private static Object convertComplexValue(final Object value, final Type icebergType) {
+ if (icebergType == null) {
+ return value;
+ }
+
+ if (icebergType.isListType()) {
+ final Type elementType = icebergType.asListType().elementType();
+ if (value instanceof Object[] array) {
+ return convertList(Arrays.asList(array), elementType);
+ }
+ if (value instanceof Collection<?> collection) {
+ return convertList(collection, elementType);
+ }
+ } else if (icebergType.isStructType() && value instanceof Record nestedRecord) {
+ return new DelegatedRecord(nestedRecord, icebergType.asStructType());
+ } else if (icebergType.isMapType() && value instanceof Map<?, ?> map) {
+ return convertMap(map, icebergType.asMapType());
+ }
- for (final RecordField field : fields) {
- final DataType dataType = field.getDataType();
- final RecordFieldType recordFieldType = dataType.getFieldType();
+ return value;
+ }
+
+ private static List<Object> convertList(final Collection<?> collection, final Type elementType) {
+ final List<Object> converted = new ArrayList<>(collection.size());
+ for (final Object element : collection) {
+ converted.add(convertValue(element, elementType));
+ }
+ return converted;
+ }
+
+ private static Map<Object, Object> convertMap(final Map<?, ?> map, final Types.MapType mapType) {
+ final Map<Object, Object> converted = new LinkedHashMap<>();
Review Comment:
LinkedHashMap vs HashMap might be debateable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-16069 - PutIcebergRecord fails with ClassCastException when writing complex types (arrays, maps, nested records) [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #11391:
URL: https://github.com/apache/nifi/pull/11391#discussion_r3561527508
##########
nifi-extension-bundles/nifi-iceberg-bundle/nifi-iceberg-parquet-writer/src/test/java/org/apache/nifi/services/iceberg/parquet/ParquetIcebergWriterTest.java:
##########
@@ -194,6 +196,43 @@ void testWriteDataFilesPartitionedTimestamp() throws IOException {
assertEquals(microsecondsExpected, partitionField);
}
+ @Test
+ void testWriteDataFilesComplexTypes() throws IOException {
+ runner.enableControllerService(parquetIcebergWriter);
+
+ final Types.StructType nestedStruct = Types.StructType.of(
+ Types.NestedField.optional(10, "city", Types.StringType.get())
+ );
+ final Schema schema = new Schema(
+ Types.NestedField.required(1, "id", Types.StringType.get()),
+ Types.NestedField.optional(2, "tags",
+ Types.ListType.ofOptional(3, Types.StringType.get())),
+ Types.NestedField.optional(4, "address", nestedStruct),
+ Types.NestedField.optional(5, "attributes",
+ Types.MapType.ofOptional(6, 7, Types.StringType.get(), Types.StringType.get()))
+ );
Review Comment:
The field names should be declared statically and reused in both the input Schema and output row.
##########
nifi-extension-bundles/nifi-iceberg-bundle/nifi-iceberg-processors/src/main/java/org/apache/nifi/processors/iceberg/record/RecordConverter.java:
##########
@@ -39,63 +43,87 @@ class RecordConverter {
private static final Set<RecordFieldType> CONVERSION_REQUIRED_FIELD_TYPES = Set.of(
RecordFieldType.TIMESTAMP,
RecordFieldType.DATE,
- RecordFieldType.TIME
+ RecordFieldType.TIME,
+ RecordFieldType.ARRAY,
+ RecordFieldType.RECORD,
+ RecordFieldType.MAP
);
/**
- * Get Converted Record with conditional handling for field values requiring translation
+ * Get Converted Record with recursive, schema-aware handling for field values requiring translation
*
* @param inputRecord Input Record to be converted
+ * @param struct Iceberg Struct Type describing the target field types (may be null for scalar-only conversion)
* @return Input Record or new Record with converted field values
*/
- static Record getConvertedRecord(final Record inputRecord) {
- final Record convertedRecord;
-
+ static Record getConvertedRecord(final Record inputRecord, final Types.StructType struct) {
final RecordSchema recordSchema = inputRecord.getSchema();
- if (isConversionRequired(recordSchema)) {
- final Map<String, Object> values = inputRecord.toMap();
- convertedRecord = getConvertedRecord(recordSchema, values);
- } else {
- convertedRecord = inputRecord;
+ if (!isConversionRequired(recordSchema)) {
+ return inputRecord;
}
- return convertedRecord;
- }
-
- private static Record getConvertedRecord(final RecordSchema recordSchema, final Map<String, Object> values) {
+ final Map<String, Object> values = inputRecord.toMap();
final Map<String, Object> convertedValues = new LinkedHashMap<>();
-
for (final Map.Entry<String, Object> entry : values.entrySet()) {
final String field = entry.getKey();
- final Object value = entry.getValue();
- final Object converted = getConvertedValue(value);
- convertedValues.put(field, converted);
+ final Type fieldType = fieldType(struct, field);
+ convertedValues.put(field, convertValue(entry.getValue(), fieldType));
}
return new MapRecord(recordSchema, convertedValues);
}
- private static Object getConvertedValue(final Object value) {
+ static Object convertValue(final Object value, final Type icebergType) {
return switch (value) {
// Convert java.sql types to corresponding java.time types for Apache Iceberg
case Timestamp timestamp -> timestamp.toLocalDateTime();
case Date date -> date.toLocalDate();
case Time time -> time.toLocalTime();
+ case Object[] array when icebergType != null && icebergType.isListType() ->
+ convertList(Arrays.asList(array), icebergType.asListType().elementType());
+ case Collection<?> collection when icebergType != null && icebergType.isListType() ->
+ convertList(collection, icebergType.asListType().elementType());
+ case Record nestedRecord when icebergType != null && icebergType.isStructType() ->
+ new DelegatedRecord(nestedRecord, icebergType.asStructType());
+ case Map<?, ?> map when icebergType != null && icebergType.isMapType() ->
+ convertMap(map, icebergType.asMapType());
Review Comment:
The nested conditionals are difficult to read, I recommend breaking this out in more detail to make it easier to follow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-16069 - PutIcebergRecord fails with ClassCastException when writing complex types (arrays, maps, nested records) [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory merged PR #11391:
URL: https://github.com/apache/nifi/pull/11391
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-16069 - PutIcebergRecord fails with ClassCastException when writing complex types (arrays, maps, nested records) [nifi]
Posted by "maltesander (via GitHub)" <gi...@apache.org>.
maltesander commented on code in PR #11391:
URL: https://github.com/apache/nifi/pull/11391#discussion_r3869117796
##########
nifi-extension-bundles/nifi-iceberg-bundle/nifi-iceberg-processors/src/main/java/org/apache/nifi/processors/iceberg/record/RecordConverter.java:
##########
@@ -39,63 +43,110 @@ class RecordConverter {
private static final Set<RecordFieldType> CONVERSION_REQUIRED_FIELD_TYPES = Set.of(
RecordFieldType.TIMESTAMP,
RecordFieldType.DATE,
- RecordFieldType.TIME
+ RecordFieldType.TIME,
+ RecordFieldType.ARRAY,
+ RecordFieldType.RECORD,
+ RecordFieldType.MAP,
+ // CHOICE can wrap any of the above, so it must also trigger conversion.
+ RecordFieldType.CHOICE
);
/**
- * Get Converted Record with conditional handling for field values requiring translation
+ * Get Converted Record with recursive, schema-aware handling for field values requiring translation
*
* @param inputRecord Input Record to be converted
+ * @param struct Iceberg Struct Type describing the target field types (may be null for scalar-only conversion)
* @return Input Record or new Record with converted field values
*/
- static Record getConvertedRecord(final Record inputRecord) {
- final Record convertedRecord;
-
+ static Record getConvertedRecord(final Record inputRecord, final Types.StructType struct) {
final RecordSchema recordSchema = inputRecord.getSchema();
- if (isConversionRequired(recordSchema)) {
- final Map<String, Object> values = inputRecord.toMap();
- convertedRecord = getConvertedRecord(recordSchema, values);
- } else {
- convertedRecord = inputRecord;
+ if (!isConversionRequired(recordSchema)) {
+ return inputRecord;
Review Comment:
fixed https://github.com/apache/nifi/pull/11391/changes/c3be398ad1b3adef3909390a2deaf269f817484a
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-16069 - PutIcebergRecord fails with ClassCastException when writing complex types (arrays, maps, nested records) [nifi]
Posted by "maltesander (via GitHub)" <gi...@apache.org>.
maltesander commented on PR #11391:
URL: https://github.com/apache/nifi/pull/11391#issuecomment-5100881777
I pushed some improvements @exceptionfactory:
- pre-allocated LinkedHashMap in convertMap https://github.com/apache/nifi/pull/11391/commits/48a9f1e74498f97f17b26161b5fbc31e606839ca
- added `RecordFieldType.CHOICE` to the required type conversions list + test https://github.com/apache/nifi/pull/11391/commits/c5ade0837f1f20ca68c489dec4405e48b06456af
And one more thing / question: Iceberg supports TimestampTz which we cannot differ via `RecordFieldType`:
```
TZPROBE timestamptz <- LocalDateTime => ClassCastException: class java.time.LocalDateTime
cannot be cast to class java.time.OffsetDateTime
TZPROBE timestamptz <- OffsetDateTime => WROTE OK
```
Without something like (timezones vary, needs to be decided):
```java
private static Object convertTimestamp(final Timestamp timestamp, final Type icebergType) {
return shouldAdjustToUtc(icebergType) ? timestamp.toInstant().atOffset(ZoneOffset.UTC) : timestamp.toLocalDateTime();
}
private static boolean shouldAdjustToUtc(final Type icebergType) {
return switch (icebergType) {
case Types.TimestampType timestampType -> timestampType.shouldAdjustToUTC();
case Types.TimestampNanoType timestampNanoType -> timestampNanoType.shouldAdjustToUTC();
case null, default -> false;
};
}
```
The first 3 tests fail without explicit conversion, so we are still missing (at least) one more fix:
```java
@Test
void testConvertTimestampWithoutZone() {
final Timestamp timestamp = Timestamp.valueOf(CREATED_LOCAL_DATE_TIME);
final Object converted = RecordConverter.convertValue(timestamp, Types.TimestampType.withoutZone());
assertEquals(CREATED_LOCAL_DATE_TIME, converted);
}
/**
* Iceberg timestamptz columns require an OffsetDateTime rather than a LocalDateTime. A Timestamp identifies an
* instant, so the converted value must describe that same instant expressed at UTC.
*/
@Test
void testConvertTimestampWithZone() {
final Timestamp timestamp = Timestamp.valueOf(CREATED_LOCAL_DATE_TIME);
final Object converted = RecordConverter.convertValue(timestamp, Types.TimestampType.withZone());
final OffsetDateTime offsetDateTime = assertInstanceOf(OffsetDateTime.class, converted);
assertEquals(ZoneOffset.UTC, offsetDateTime.getOffset());
assertEquals(timestamp.toInstant(), offsetDateTime.toInstant());
}
@Test
void testConvertTimestampNanoWithZone() {
final Timestamp timestamp = Timestamp.valueOf(CREATED_LOCAL_DATE_TIME);
final Object converted = RecordConverter.convertValue(timestamp, Types.TimestampNanoType.withZone());
final OffsetDateTime offsetDateTime = assertInstanceOf(OffsetDateTime.class, converted);
assertEquals(timestamp.toInstant(), offsetDateTime.toInstant());
}
/**
* The Iceberg type is not known for every field, so an unresolved type must retain the LocalDateTime conversion.
*/
@Test
void testConvertTimestampUnknownIcebergType() {
final Timestamp timestamp = Timestamp.valueOf(CREATED_LOCAL_DATE_TIME);
final Object converted = RecordConverter.convertValue(timestamp, null);
assertEquals(CREATED_LOCAL_DATE_TIME, converted);
}
/**
* A timestamptz column nested inside a struct must be converted through the recursive path, matching the
* positional access Iceberg uses when writing.
*/
@Test
void testGetConvertedRecordNestedTimestampWithZone() {
final Types.StructType structType = Types.StructType.of(
Types.NestedField.optional(1, CREATED_FIELD_NAME, Types.TimestampType.withZone())
);
final RecordSchema nestedSchema = new SimpleRecordSchema(List.of(
new RecordField(CREATED_FIELD_NAME, RecordFieldType.TIMESTAMP.getDataType())
));
final Timestamp timestamp = Timestamp.valueOf(CREATED_LOCAL_DATE_TIME);
final Map<String, Object> nestedValues = new LinkedHashMap<>();
nestedValues.put(CREATED_FIELD_NAME, timestamp);
final Record nestedRecord = new MapRecord(nestedSchema, nestedValues);
final Object converted = RecordConverter.convertValue(nestedRecord, structType);
final StructLike struct = assertInstanceOf(StructLike.class, converted);
assertEquals(timestamp.toInstant(), struct.get(0, OffsetDateTime.class).toInstant());
}
```
This is not mentioned in the Jira ticket or this PR so i would defer fixing this in this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-16069 - PutIcebergRecord fails with ClassCastException when writing complex types (arrays, maps, nested records) [nifi]
Posted by "maltesander (via GitHub)" <gi...@apache.org>.
maltesander commented on code in PR #11391:
URL: https://github.com/apache/nifi/pull/11391#discussion_r3563236113
##########
nifi-extension-bundles/nifi-iceberg-bundle/nifi-iceberg-parquet-writer/src/test/java/org/apache/nifi/services/iceberg/parquet/ParquetIcebergWriterTest.java:
##########
@@ -194,6 +196,43 @@ void testWriteDataFilesPartitionedTimestamp() throws IOException {
assertEquals(microsecondsExpected, partitionField);
}
+ @Test
+ void testWriteDataFilesComplexTypes() throws IOException {
+ runner.enableControllerService(parquetIcebergWriter);
+
+ final Types.StructType nestedStruct = Types.StructType.of(
+ Types.NestedField.optional(10, "city", Types.StringType.get())
+ );
+ final Schema schema = new Schema(
+ Types.NestedField.required(1, "id", Types.StringType.get()),
+ Types.NestedField.optional(2, "tags",
+ Types.ListType.ofOptional(3, Types.StringType.get())),
+ Types.NestedField.optional(4, "address", nestedStruct),
+ Types.NestedField.optional(5, "attributes",
+ Types.MapType.ofOptional(6, 7, Types.StringType.get(), Types.StringType.get()))
+ );
Review Comment:
Moved hardcoded constants to static ones: https://github.com/apache/nifi/pull/11391/commits/ef3e162af7aa479ecba4c318428a8ba3c8ff5704
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-16069 - PutIcebergRecord fails with ClassCastException when writing complex types (arrays, maps, nested records) [nifi]
Posted by "maltesander (via GitHub)" <gi...@apache.org>.
maltesander commented on code in PR #11391:
URL: https://github.com/apache/nifi/pull/11391#discussion_r3563235349
##########
nifi-extension-bundles/nifi-iceberg-bundle/nifi-iceberg-processors/src/main/java/org/apache/nifi/processors/iceberg/record/RecordConverter.java:
##########
@@ -39,63 +43,87 @@ class RecordConverter {
private static final Set<RecordFieldType> CONVERSION_REQUIRED_FIELD_TYPES = Set.of(
RecordFieldType.TIMESTAMP,
RecordFieldType.DATE,
- RecordFieldType.TIME
+ RecordFieldType.TIME,
+ RecordFieldType.ARRAY,
+ RecordFieldType.RECORD,
+ RecordFieldType.MAP
);
/**
- * Get Converted Record with conditional handling for field values requiring translation
+ * Get Converted Record with recursive, schema-aware handling for field values requiring translation
*
* @param inputRecord Input Record to be converted
+ * @param struct Iceberg Struct Type describing the target field types (may be null for scalar-only conversion)
* @return Input Record or new Record with converted field values
*/
- static Record getConvertedRecord(final Record inputRecord) {
- final Record convertedRecord;
-
+ static Record getConvertedRecord(final Record inputRecord, final Types.StructType struct) {
final RecordSchema recordSchema = inputRecord.getSchema();
- if (isConversionRequired(recordSchema)) {
- final Map<String, Object> values = inputRecord.toMap();
- convertedRecord = getConvertedRecord(recordSchema, values);
- } else {
- convertedRecord = inputRecord;
+ if (!isConversionRequired(recordSchema)) {
+ return inputRecord;
}
- return convertedRecord;
- }
-
- private static Record getConvertedRecord(final RecordSchema recordSchema, final Map<String, Object> values) {
+ final Map<String, Object> values = inputRecord.toMap();
final Map<String, Object> convertedValues = new LinkedHashMap<>();
-
for (final Map.Entry<String, Object> entry : values.entrySet()) {
final String field = entry.getKey();
- final Object value = entry.getValue();
- final Object converted = getConvertedValue(value);
- convertedValues.put(field, converted);
+ final Type fieldType = fieldType(struct, field);
+ convertedValues.put(field, convertValue(entry.getValue(), fieldType));
}
return new MapRecord(recordSchema, convertedValues);
}
- private static Object getConvertedValue(final Object value) {
+ static Object convertValue(final Object value, final Type icebergType) {
return switch (value) {
// Convert java.sql types to corresponding java.time types for Apache Iceberg
case Timestamp timestamp -> timestamp.toLocalDateTime();
case Date date -> date.toLocalDate();
case Time time -> time.toLocalTime();
+ case Object[] array when icebergType != null && icebergType.isListType() ->
+ convertList(Arrays.asList(array), icebergType.asListType().elementType());
+ case Collection<?> collection when icebergType != null && icebergType.isListType() ->
+ convertList(collection, icebergType.asListType().elementType());
+ case Record nestedRecord when icebergType != null && icebergType.isStructType() ->
+ new DelegatedRecord(nestedRecord, icebergType.asStructType());
+ case Map<?, ?> map when icebergType != null && icebergType.isMapType() ->
+ convertMap(map, icebergType.asMapType());
Review Comment:
Thank you for the feedback @exceptionfactory. I tried to simplify here https://github.com/apache/nifi/pull/11391/commits/d174fd39accfa55c4106128f3bc6277c769fcd88.
I introduced a `convertComplexValue` method to handle this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-16069 - PutIcebergRecord fails with ClassCastException when writing complex types (arrays, maps, nested records) [nifi]
Posted by "maltesander (via GitHub)" <gi...@apache.org>.
maltesander commented on code in PR #11391:
URL: https://github.com/apache/nifi/pull/11391#discussion_r3869127809
##########
nifi-extension-bundles/nifi-iceberg-bundle/nifi-iceberg-processors/src/main/java/org/apache/nifi/processors/iceberg/record/RecordConverter.java:
##########
@@ -39,63 +43,110 @@ class RecordConverter {
private static final Set<RecordFieldType> CONVERSION_REQUIRED_FIELD_TYPES = Set.of(
RecordFieldType.TIMESTAMP,
RecordFieldType.DATE,
- RecordFieldType.TIME
+ RecordFieldType.TIME,
+ RecordFieldType.ARRAY,
+ RecordFieldType.RECORD,
+ RecordFieldType.MAP,
+ // CHOICE can wrap any of the above, so it must also trigger conversion.
+ RecordFieldType.CHOICE
);
/**
- * Get Converted Record with conditional handling for field values requiring translation
+ * Get Converted Record with recursive, schema-aware handling for field values requiring translation
*
* @param inputRecord Input Record to be converted
+ * @param struct Iceberg Struct Type describing the target field types (may be null for scalar-only conversion)
* @return Input Record or new Record with converted field values
*/
- static Record getConvertedRecord(final Record inputRecord) {
- final Record convertedRecord;
-
+ static Record getConvertedRecord(final Record inputRecord, final Types.StructType struct) {
final RecordSchema recordSchema = inputRecord.getSchema();
- if (isConversionRequired(recordSchema)) {
- final Map<String, Object> values = inputRecord.toMap();
- convertedRecord = getConvertedRecord(recordSchema, values);
- } else {
- convertedRecord = inputRecord;
+ if (!isConversionRequired(recordSchema)) {
+ return inputRecord;
}
- return convertedRecord;
- }
-
- private static Record getConvertedRecord(final RecordSchema recordSchema, final Map<String, Object> values) {
+ final Map<String, Object> values = inputRecord.toMap();
final Map<String, Object> convertedValues = new LinkedHashMap<>();
-
for (final Map.Entry<String, Object> entry : values.entrySet()) {
final String field = entry.getKey();
- final Object value = entry.getValue();
- final Object converted = getConvertedValue(value);
- convertedValues.put(field, converted);
+ final Type fieldType = fieldType(struct, field);
+ convertedValues.put(field, convertValue(entry.getValue(), fieldType));
}
return new MapRecord(recordSchema, convertedValues);
}
- private static Object getConvertedValue(final Object value) {
+ static Object convertValue(final Object value, final Type icebergType) {
return switch (value) {
// Convert java.sql types to corresponding java.time types for Apache Iceberg
case Timestamp timestamp -> timestamp.toLocalDateTime();
case Date date -> date.toLocalDate();
case Time time -> time.toLocalTime();
- case null, default -> value;
+ // Recursively convert complex types against the matching Iceberg type
+ case null, default -> convertComplexValue(value, icebergType);
};
}
- private static boolean isConversionRequired(final RecordSchema recordSchema) {
- final List<RecordField> fields = recordSchema.getFields();
+ /**
+ * Recursively convert array, collection, nested record, and map values against the matching Iceberg type.
+ * The value is returned unchanged when the target Iceberg type is unknown or does not describe a complex type
+ * matching the value.
+ */
+ private static Object convertComplexValue(final Object value, final Type icebergType) {
Review Comment:
fixed https://github.com/apache/nifi/pull/11391/changes/10dac695d1d21a0a9a1bd4b5533449b00ad665d3
##########
nifi-extension-bundles/nifi-iceberg-bundle/nifi-iceberg-processors/src/main/java/org/apache/nifi/processors/iceberg/record/RecordConverter.java:
##########
@@ -39,63 +43,110 @@ class RecordConverter {
private static final Set<RecordFieldType> CONVERSION_REQUIRED_FIELD_TYPES = Set.of(
RecordFieldType.TIMESTAMP,
RecordFieldType.DATE,
- RecordFieldType.TIME
+ RecordFieldType.TIME,
+ RecordFieldType.ARRAY,
+ RecordFieldType.RECORD,
+ RecordFieldType.MAP,
+ // CHOICE can wrap any of the above, so it must also trigger conversion.
+ RecordFieldType.CHOICE
);
/**
- * Get Converted Record with conditional handling for field values requiring translation
+ * Get Converted Record with recursive, schema-aware handling for field values requiring translation
*
* @param inputRecord Input Record to be converted
+ * @param struct Iceberg Struct Type describing the target field types (may be null for scalar-only conversion)
* @return Input Record or new Record with converted field values
*/
- static Record getConvertedRecord(final Record inputRecord) {
- final Record convertedRecord;
-
+ static Record getConvertedRecord(final Record inputRecord, final Types.StructType struct) {
final RecordSchema recordSchema = inputRecord.getSchema();
- if (isConversionRequired(recordSchema)) {
- final Map<String, Object> values = inputRecord.toMap();
- convertedRecord = getConvertedRecord(recordSchema, values);
- } else {
- convertedRecord = inputRecord;
+ if (!isConversionRequired(recordSchema)) {
+ return inputRecord;
}
- return convertedRecord;
- }
-
- private static Record getConvertedRecord(final RecordSchema recordSchema, final Map<String, Object> values) {
+ final Map<String, Object> values = inputRecord.toMap();
final Map<String, Object> convertedValues = new LinkedHashMap<>();
-
for (final Map.Entry<String, Object> entry : values.entrySet()) {
final String field = entry.getKey();
- final Object value = entry.getValue();
- final Object converted = getConvertedValue(value);
- convertedValues.put(field, converted);
+ final Type fieldType = fieldType(struct, field);
+ convertedValues.put(field, convertValue(entry.getValue(), fieldType));
}
return new MapRecord(recordSchema, convertedValues);
}
- private static Object getConvertedValue(final Object value) {
+ static Object convertValue(final Object value, final Type icebergType) {
return switch (value) {
// Convert java.sql types to corresponding java.time types for Apache Iceberg
case Timestamp timestamp -> timestamp.toLocalDateTime();
case Date date -> date.toLocalDate();
case Time time -> time.toLocalTime();
- case null, default -> value;
+ // Recursively convert complex types against the matching Iceberg type
+ case null, default -> convertComplexValue(value, icebergType);
};
}
- private static boolean isConversionRequired(final RecordSchema recordSchema) {
- final List<RecordField> fields = recordSchema.getFields();
+ /**
+ * Recursively convert array, collection, nested record, and map values against the matching Iceberg type.
+ * The value is returned unchanged when the target Iceberg type is unknown or does not describe a complex type
+ * matching the value.
+ */
Review Comment:
fixed https://github.com/apache/nifi/pull/11391/changes/10dac695d1d21a0a9a1bd4b5533449b00ad665d3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-16069 - PutIcebergRecord fails with ClassCastException when writing complex types (arrays, maps, nested records) [nifi]
Posted by "maltesander (via GitHub)" <gi...@apache.org>.
maltesander commented on code in PR #11391:
URL: https://github.com/apache/nifi/pull/11391#discussion_r3563489879
##########
nifi-extension-bundles/nifi-iceberg-bundle/nifi-iceberg-processors/src/main/java/org/apache/nifi/processors/iceberg/record/RecordConverter.java:
##########
@@ -39,63 +43,107 @@ class RecordConverter {
private static final Set<RecordFieldType> CONVERSION_REQUIRED_FIELD_TYPES = Set.of(
RecordFieldType.TIMESTAMP,
RecordFieldType.DATE,
- RecordFieldType.TIME
+ RecordFieldType.TIME,
+ RecordFieldType.ARRAY,
+ RecordFieldType.RECORD,
+ RecordFieldType.MAP
);
/**
- * Get Converted Record with conditional handling for field values requiring translation
+ * Get Converted Record with recursive, schema-aware handling for field values requiring translation
*
* @param inputRecord Input Record to be converted
+ * @param struct Iceberg Struct Type describing the target field types (may be null for scalar-only conversion)
* @return Input Record or new Record with converted field values
*/
- static Record getConvertedRecord(final Record inputRecord) {
- final Record convertedRecord;
-
+ static Record getConvertedRecord(final Record inputRecord, final Types.StructType struct) {
final RecordSchema recordSchema = inputRecord.getSchema();
- if (isConversionRequired(recordSchema)) {
- final Map<String, Object> values = inputRecord.toMap();
- convertedRecord = getConvertedRecord(recordSchema, values);
- } else {
- convertedRecord = inputRecord;
+ if (!isConversionRequired(recordSchema)) {
+ return inputRecord;
}
- return convertedRecord;
- }
-
- private static Record getConvertedRecord(final RecordSchema recordSchema, final Map<String, Object> values) {
+ final Map<String, Object> values = inputRecord.toMap();
final Map<String, Object> convertedValues = new LinkedHashMap<>();
-
for (final Map.Entry<String, Object> entry : values.entrySet()) {
final String field = entry.getKey();
- final Object value = entry.getValue();
- final Object converted = getConvertedValue(value);
- convertedValues.put(field, converted);
+ final Type fieldType = fieldType(struct, field);
+ convertedValues.put(field, convertValue(entry.getValue(), fieldType));
}
return new MapRecord(recordSchema, convertedValues);
}
- private static Object getConvertedValue(final Object value) {
+ static Object convertValue(final Object value, final Type icebergType) {
return switch (value) {
// Convert java.sql types to corresponding java.time types for Apache Iceberg
case Timestamp timestamp -> timestamp.toLocalDateTime();
case Date date -> date.toLocalDate();
case Time time -> time.toLocalTime();
- case null, default -> value;
+ // Recursively convert complex types against the matching Iceberg type
+ case null, default -> convertComplexValue(value, icebergType);
};
}
- private static boolean isConversionRequired(final RecordSchema recordSchema) {
- final List<RecordField> fields = recordSchema.getFields();
+ /**
+ * Recursively convert array, collection, nested record, and map values against the matching Iceberg type.
+ * The value is returned unchanged when the target Iceberg type is unknown or does not describe a complex type
+ * matching the value.
+ */
+ private static Object convertComplexValue(final Object value, final Type icebergType) {
+ if (icebergType == null) {
+ return value;
+ }
+
+ if (icebergType.isListType()) {
+ final Type elementType = icebergType.asListType().elementType();
+ if (value instanceof Object[] array) {
+ return convertList(Arrays.asList(array), elementType);
+ }
+ if (value instanceof Collection<?> collection) {
+ return convertList(collection, elementType);
+ }
+ } else if (icebergType.isStructType() && value instanceof Record nestedRecord) {
+ return new DelegatedRecord(nestedRecord, icebergType.asStructType());
+ } else if (icebergType.isMapType() && value instanceof Map<?, ?> map) {
+ return convertMap(map, icebergType.asMapType());
+ }
- for (final RecordField field : fields) {
- final DataType dataType = field.getDataType();
- final RecordFieldType recordFieldType = dataType.getFieldType();
+ return value;
+ }
+
+ private static List<Object> convertList(final Collection<?> collection, final Type elementType) {
+ final List<Object> converted = new ArrayList<>(collection.size());
+ for (final Object element : collection) {
+ converted.add(convertValue(element, elementType));
+ }
+ return converted;
+ }
+
+ private static Map<Object, Object> convertMap(final Map<?, ?> map, final Types.MapType mapType) {
+ final Map<Object, Object> converted = new LinkedHashMap<>();
Review Comment:
LinkedHashMap vs HashMap might be debateable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-16069 - PutIcebergRecord fails with ClassCastException when writing complex types (arrays, maps, nested records) [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #11391:
URL: https://github.com/apache/nifi/pull/11391#discussion_r3866325503
##########
nifi-extension-bundles/nifi-iceberg-bundle/nifi-iceberg-processors/src/main/java/org/apache/nifi/processors/iceberg/record/RecordConverter.java:
##########
@@ -39,63 +43,110 @@ class RecordConverter {
private static final Set<RecordFieldType> CONVERSION_REQUIRED_FIELD_TYPES = Set.of(
RecordFieldType.TIMESTAMP,
RecordFieldType.DATE,
- RecordFieldType.TIME
+ RecordFieldType.TIME,
+ RecordFieldType.ARRAY,
+ RecordFieldType.RECORD,
+ RecordFieldType.MAP,
+ // CHOICE can wrap any of the above, so it must also trigger conversion.
+ RecordFieldType.CHOICE
);
/**
- * Get Converted Record with conditional handling for field values requiring translation
+ * Get Converted Record with recursive, schema-aware handling for field values requiring translation
*
* @param inputRecord Input Record to be converted
+ * @param struct Iceberg Struct Type describing the target field types (may be null for scalar-only conversion)
* @return Input Record or new Record with converted field values
*/
- static Record getConvertedRecord(final Record inputRecord) {
- final Record convertedRecord;
-
+ static Record getConvertedRecord(final Record inputRecord, final Types.StructType struct) {
final RecordSchema recordSchema = inputRecord.getSchema();
- if (isConversionRequired(recordSchema)) {
- final Map<String, Object> values = inputRecord.toMap();
- convertedRecord = getConvertedRecord(recordSchema, values);
- } else {
- convertedRecord = inputRecord;
+ if (!isConversionRequired(recordSchema)) {
+ return inputRecord;
Review Comment:
I recommend adjusting the approach to have a single return, instead of a short-circuit return
##########
nifi-extension-bundles/nifi-iceberg-bundle/nifi-iceberg-processors/src/main/java/org/apache/nifi/processors/iceberg/record/RecordConverter.java:
##########
@@ -39,63 +43,110 @@ class RecordConverter {
private static final Set<RecordFieldType> CONVERSION_REQUIRED_FIELD_TYPES = Set.of(
RecordFieldType.TIMESTAMP,
RecordFieldType.DATE,
- RecordFieldType.TIME
+ RecordFieldType.TIME,
+ RecordFieldType.ARRAY,
+ RecordFieldType.RECORD,
+ RecordFieldType.MAP,
+ // CHOICE can wrap any of the above, so it must also trigger conversion.
+ RecordFieldType.CHOICE
);
/**
- * Get Converted Record with conditional handling for field values requiring translation
+ * Get Converted Record with recursive, schema-aware handling for field values requiring translation
*
* @param inputRecord Input Record to be converted
+ * @param struct Iceberg Struct Type describing the target field types (may be null for scalar-only conversion)
* @return Input Record or new Record with converted field values
*/
- static Record getConvertedRecord(final Record inputRecord) {
- final Record convertedRecord;
-
+ static Record getConvertedRecord(final Record inputRecord, final Types.StructType struct) {
final RecordSchema recordSchema = inputRecord.getSchema();
- if (isConversionRequired(recordSchema)) {
- final Map<String, Object> values = inputRecord.toMap();
- convertedRecord = getConvertedRecord(recordSchema, values);
- } else {
- convertedRecord = inputRecord;
+ if (!isConversionRequired(recordSchema)) {
+ return inputRecord;
}
- return convertedRecord;
- }
-
- private static Record getConvertedRecord(final RecordSchema recordSchema, final Map<String, Object> values) {
+ final Map<String, Object> values = inputRecord.toMap();
final Map<String, Object> convertedValues = new LinkedHashMap<>();
-
for (final Map.Entry<String, Object> entry : values.entrySet()) {
final String field = entry.getKey();
- final Object value = entry.getValue();
- final Object converted = getConvertedValue(value);
- convertedValues.put(field, converted);
+ final Type fieldType = fieldType(struct, field);
+ convertedValues.put(field, convertValue(entry.getValue(), fieldType));
}
return new MapRecord(recordSchema, convertedValues);
}
- private static Object getConvertedValue(final Object value) {
+ static Object convertValue(final Object value, final Type icebergType) {
return switch (value) {
// Convert java.sql types to corresponding java.time types for Apache Iceberg
case Timestamp timestamp -> timestamp.toLocalDateTime();
case Date date -> date.toLocalDate();
case Time time -> time.toLocalTime();
- case null, default -> value;
+ // Recursively convert complex types against the matching Iceberg type
+ case null, default -> convertComplexValue(value, icebergType);
};
}
- private static boolean isConversionRequired(final RecordSchema recordSchema) {
- final List<RecordField> fields = recordSchema.getFields();
+ /**
+ * Recursively convert array, collection, nested record, and map values against the matching Iceberg type.
+ * The value is returned unchanged when the target Iceberg type is unknown or does not describe a complex type
+ * matching the value.
+ */
+ private static Object convertComplexValue(final Object value, final Type icebergType) {
Review Comment:
Managing multiple returns can become difficult, I recommend refactoring to a single return
##########
nifi-extension-bundles/nifi-iceberg-bundle/nifi-iceberg-processors/src/main/java/org/apache/nifi/processors/iceberg/record/RecordConverter.java:
##########
@@ -39,63 +43,110 @@ class RecordConverter {
private static final Set<RecordFieldType> CONVERSION_REQUIRED_FIELD_TYPES = Set.of(
RecordFieldType.TIMESTAMP,
RecordFieldType.DATE,
- RecordFieldType.TIME
+ RecordFieldType.TIME,
+ RecordFieldType.ARRAY,
+ RecordFieldType.RECORD,
+ RecordFieldType.MAP,
+ // CHOICE can wrap any of the above, so it must also trigger conversion.
+ RecordFieldType.CHOICE
);
/**
- * Get Converted Record with conditional handling for field values requiring translation
+ * Get Converted Record with recursive, schema-aware handling for field values requiring translation
*
* @param inputRecord Input Record to be converted
+ * @param struct Iceberg Struct Type describing the target field types (may be null for scalar-only conversion)
* @return Input Record or new Record with converted field values
*/
- static Record getConvertedRecord(final Record inputRecord) {
- final Record convertedRecord;
-
+ static Record getConvertedRecord(final Record inputRecord, final Types.StructType struct) {
final RecordSchema recordSchema = inputRecord.getSchema();
- if (isConversionRequired(recordSchema)) {
- final Map<String, Object> values = inputRecord.toMap();
- convertedRecord = getConvertedRecord(recordSchema, values);
- } else {
- convertedRecord = inputRecord;
+ if (!isConversionRequired(recordSchema)) {
+ return inputRecord;
}
- return convertedRecord;
- }
-
- private static Record getConvertedRecord(final RecordSchema recordSchema, final Map<String, Object> values) {
+ final Map<String, Object> values = inputRecord.toMap();
final Map<String, Object> convertedValues = new LinkedHashMap<>();
-
for (final Map.Entry<String, Object> entry : values.entrySet()) {
final String field = entry.getKey();
- final Object value = entry.getValue();
- final Object converted = getConvertedValue(value);
- convertedValues.put(field, converted);
+ final Type fieldType = fieldType(struct, field);
+ convertedValues.put(field, convertValue(entry.getValue(), fieldType));
}
return new MapRecord(recordSchema, convertedValues);
}
- private static Object getConvertedValue(final Object value) {
+ static Object convertValue(final Object value, final Type icebergType) {
return switch (value) {
// Convert java.sql types to corresponding java.time types for Apache Iceberg
case Timestamp timestamp -> timestamp.toLocalDateTime();
case Date date -> date.toLocalDate();
case Time time -> time.toLocalTime();
- case null, default -> value;
+ // Recursively convert complex types against the matching Iceberg type
+ case null, default -> convertComplexValue(value, icebergType);
};
}
- private static boolean isConversionRequired(final RecordSchema recordSchema) {
- final List<RecordField> fields = recordSchema.getFields();
+ /**
+ * Recursively convert array, collection, nested record, and map values against the matching Iceberg type.
+ * The value is returned unchanged when the target Iceberg type is unknown or does not describe a complex type
+ * matching the value.
+ */
Review Comment:
When adding a method-level comment, the parameters and return should be included.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org