You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "the-other-tim-brown (via GitHub)" <gi...@apache.org> on 2025/09/11 19:42:44 UTC
[PR] Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
the-other-tim-brown opened a new pull request, #13882:
URL: https://github.com/apache/hudi/pull/13882
### Change Logs
The provided spark writer support for parquet has an option for using the legacy writer format that will write Decimals in the legacy format and lists in the legacy format. On the other hand, the avro based writer always writes the Decimal in the legacy format and has a flag for writing lists in the legacy format.
The adoption of 3-level lists is more common and required when syncing Hudi tables to Iceberg so we need to have control on how we write lists that is independent of the decimal writing.
In this PR, the row write support is adapted to always persists the decimals in the legacy format to ensure it is compatible with other Hudi readers. The avro flag along with the hudi flag for writing in the legacy format will apply to how we write out the lists. If legacy format is enabled, 2-level lists are written. Otherwise it is 3-level lists.
On top of this, the spark writer also had a restriction where timestamps within the same writer were always written out with the same precision causing issues if users had specified different granularities in their schema. This PR also addresses this by passing in the avro schema to the write support class to determine the precision of the timestamp and convert it accordingly.
### Impact
Ensures better consistency between writes when using spark rows and parquet-avro.
### Risk level (write none, low medium or high below)
Medium
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345680041
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,47 +18,149 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
+import org.apache.hudi.SparkAdapterSupport;
+import org.apache.hudi.avro.HoodieAvroUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
- * Hoodie Write Support for directly writing Row to Parquet.
+ * Hoodie Write Support for directly writing Row to Parquet and adding the Hudi bloom index to the file metadata.
+ * The implementation is adapted from Spark's {@link org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport} but has the following major differences:
+ * <ul>
+ * <li>Decimals are always written with the legacy format to ensure compatibility with parquet-avro and other readers</li>
+ * <li>Writing 2-level or 3-level lists is configurable with either the Hudi option, hoodie.parquet.writelegacyformat.enabled, or the parquet-avro option, parquet.avro.write-old-list-structure
+ * to ensure consistency across writer paths.</li>
+ * <li>The scale of the timestamps is determined by the Hudi writer schema instead of relying on Spark configuration</li>
+ * </ul>
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
+ private static final Schema MAP_KEY_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final String MAP_REPEATED_NAME = "key_value";
+ private static final String MAP_KEY_NAME = "key";
+ private static final String MAP_VALUE_NAME = "value";
private final Configuration hadoopConf;
private final Option<HoodieBloomFilterWriteSupport<UTF8String>> bloomFilterWriteSupportOpt;
+ private final byte[] decimalBuffer = new byte[Decimal.minBytesForPrecision()[DecimalType.MAX_PRECISION()]];
+ private final Enumeration.Value datetimeRebaseMode = (Enumeration.Value) SparkAdapterSupport.getSparkAdapter().getDateTimeRebaseMode();
+ private final Function1<Object, Object> dateRebaseFunction = DataSourceUtils.createDateRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private final Function1<Object, Object> timestampRebaseFunction = DataSourceUtils.createTimestampRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private RecordConsumer recordConsumer;
+ private final boolean writeLegacyListFormat;
+ private final ValueWriter[] rootFieldWriters;
+ private final Schema avroSchema;
- public HoodieRowParquetWriteSupport(Configuration conf, StructType structType, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
+ public HoodieRowParquetWriteSupport(Configuration conf, StructType schema, Option<BloomFilter> bloomFilterOpt, HoodieWriteConfig config) {
Configuration hadoopConf = new Configuration(conf);
- hadoopConf.set("spark.sql.parquet.writeLegacyFormat", config.getStringOrDefault(HoodieStorageConfig.PARQUET_WRITE_LEGACY_FORMAT_ENABLED, "false"));
+ String writeLegacyFormatEnabled = config.getStringOrDefault(HoodieStorageConfig.PARQUET_WRITE_LEGACY_FORMAT_ENABLED, "false");
+ hadoopConf.set("spark.sql.parquet.writeLegacyFormat", writeLegacyFormatEnabled);
hadoopConf.set("spark.sql.parquet.outputTimestampType", config.getStringOrDefault(HoodieStorageConfig.PARQUET_OUTPUT_TIMESTAMP_TYPE));
hadoopConf.set("spark.sql.parquet.fieldId.write.enabled", config.getStringOrDefault(PARQUET_FIELD_ID_WRITE_ENABLED));
- setSchema(structType, hadoopConf);
-
+ this.writeLegacyListFormat = Boolean.parseBoolean(writeLegacyFormatEnabled)
+ || Boolean.parseBoolean(config.getStringOrDefault(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE, "false"));
+ this.avroSchema = SerDeHelper.fromJson(config.getInternalSchema()).map(internalSchema -> AvroInternalSchemaConverter.convert(internalSchema, "spark_schema"))
+ .orElseGet(() -> {
+ Schema parsedSchema = new Schema.Parser().parse(config.getWriteSchema());
+ if (config.populateMetaFields()) {
+ return HoodieAvroUtils.addMetadataFields(parsedSchema, config.allowOperationMetadataField());
+ }
+ return parsedSchema;
+ });
+ ParquetWriteSupport.setSchema(HoodieInternalRowUtils.getCachedSchema(avroSchema), hadoopConf);
+ this.rootFieldWriters = getFieldWriters(schema, avroSchema);
this.hadoopConf = hadoopConf;
this.bloomFilterWriteSupportOpt = bloomFilterOpt.map(HoodieBloomFilterRowWriteSupport::new);
}
+ private ValueWriter[] getFieldWriters(StructType schema, Schema avroSchema) {
+ Map<String, Integer> fieldNameToIndex = new HashMap<>();
+ for (int i = 0; i < schema.fields().length; i++) {
+ fieldNameToIndex.put(schema.fields()[i].name(), i);
+ }
+ return IntStream.range(0, avroSchema.getFields().size()).mapToObj(i -> {
+ Schema.Field field = avroSchema.getFields().get(i);
+ Integer structIndex = fieldNameToIndex.get(field.name());
+ return makeWriter(field.schema(), schema.fields()[structIndex].dataType());
+ }).toArray(ValueWriter[]::new);
+ }
+
public Configuration getHadoopConf() {
return hadoopConf;
}
+ @Override
+ public WriteContext init(Configuration configuration) {
+ Map<String, String> metadata = new HashMap<>();
+ metadata.put("org.apache.spark.version", VersionUtils.shortVersion(HoodieSparkUtils.getSparkVersion()));
Review Comment:
`org.apache.hudi.parquet.write.support` to have both version and class name? @vinothchandar do you think this is OK?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345640112
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,47 +18,149 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
+import org.apache.hudi.SparkAdapterSupport;
+import org.apache.hudi.avro.HoodieAvroUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
- * Hoodie Write Support for directly writing Row to Parquet.
+ * Hoodie Write Support for directly writing Row to Parquet and adding the Hudi bloom index to the file metadata.
+ * The implementation is adapted from Spark's {@link org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport} but has the following major differences:
+ * <ul>
+ * <li>Decimals are always written with the legacy format to ensure compatibility with parquet-avro and other readers</li>
+ * <li>Writing 2-level or 3-level lists is configurable with either the Hudi option, hoodie.parquet.writelegacyformat.enabled, or the parquet-avro option, parquet.avro.write-old-list-structure
+ * to ensure consistency across writer paths.</li>
+ * <li>The scale of the timestamps is determined by the Hudi writer schema instead of relying on Spark configuration</li>
+ * </ul>
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
+ private static final Schema MAP_KEY_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final String MAP_REPEATED_NAME = "key_value";
+ private static final String MAP_KEY_NAME = "key";
+ private static final String MAP_VALUE_NAME = "value";
private final Configuration hadoopConf;
private final Option<HoodieBloomFilterWriteSupport<UTF8String>> bloomFilterWriteSupportOpt;
+ private final byte[] decimalBuffer = new byte[Decimal.minBytesForPrecision()[DecimalType.MAX_PRECISION()]];
+ private final Enumeration.Value datetimeRebaseMode = (Enumeration.Value) SparkAdapterSupport.getSparkAdapter().getDateTimeRebaseMode();
+ private final Function1<Object, Object> dateRebaseFunction = DataSourceUtils.createDateRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private final Function1<Object, Object> timestampRebaseFunction = DataSourceUtils.createTimestampRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private RecordConsumer recordConsumer;
+ private final boolean writeLegacyListFormat;
+ private final ValueWriter[] rootFieldWriters;
+ private final Schema avroSchema;
- public HoodieRowParquetWriteSupport(Configuration conf, StructType structType, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
+ public HoodieRowParquetWriteSupport(Configuration conf, StructType schema, Option<BloomFilter> bloomFilterOpt, HoodieWriteConfig config) {
Configuration hadoopConf = new Configuration(conf);
- hadoopConf.set("spark.sql.parquet.writeLegacyFormat", config.getStringOrDefault(HoodieStorageConfig.PARQUET_WRITE_LEGACY_FORMAT_ENABLED, "false"));
+ String writeLegacyFormatEnabled = config.getStringOrDefault(HoodieStorageConfig.PARQUET_WRITE_LEGACY_FORMAT_ENABLED, "false");
+ hadoopConf.set("spark.sql.parquet.writeLegacyFormat", writeLegacyFormatEnabled);
hadoopConf.set("spark.sql.parquet.outputTimestampType", config.getStringOrDefault(HoodieStorageConfig.PARQUET_OUTPUT_TIMESTAMP_TYPE));
hadoopConf.set("spark.sql.parquet.fieldId.write.enabled", config.getStringOrDefault(PARQUET_FIELD_ID_WRITE_ENABLED));
- setSchema(structType, hadoopConf);
-
+ this.writeLegacyListFormat = Boolean.parseBoolean(writeLegacyFormatEnabled)
+ || Boolean.parseBoolean(config.getStringOrDefault(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE, "false"));
+ this.avroSchema = SerDeHelper.fromJson(config.getInternalSchema()).map(internalSchema -> AvroInternalSchemaConverter.convert(internalSchema, "spark_schema"))
+ .orElseGet(() -> {
+ Schema parsedSchema = new Schema.Parser().parse(config.getWriteSchema());
+ if (config.populateMetaFields()) {
+ return HoodieAvroUtils.addMetadataFields(parsedSchema, config.allowOperationMetadataField());
+ }
+ return parsedSchema;
+ });
+ ParquetWriteSupport.setSchema(HoodieInternalRowUtils.getCachedSchema(avroSchema), hadoopConf);
+ this.rootFieldWriters = getFieldWriters(schema, avroSchema);
this.hadoopConf = hadoopConf;
this.bloomFilterWriteSupportOpt = bloomFilterOpt.map(HoodieBloomFilterRowWriteSupport::new);
}
+ private ValueWriter[] getFieldWriters(StructType schema, Schema avroSchema) {
+ Map<String, Integer> fieldNameToIndex = new HashMap<>();
+ for (int i = 0; i < schema.fields().length; i++) {
+ fieldNameToIndex.put(schema.fields()[i].name(), i);
+ }
+ return IntStream.range(0, avroSchema.getFields().size()).mapToObj(i -> {
+ Schema.Field field = avroSchema.getFields().get(i);
+ Integer structIndex = fieldNameToIndex.get(field.name());
+ return makeWriter(field.schema(), schema.fields()[structIndex].dataType());
+ }).toArray(ValueWriter[]::new);
+ }
+
public Configuration getHadoopConf() {
return hadoopConf;
}
+ @Override
+ public WriteContext init(Configuration configuration) {
+ Map<String, String> metadata = new HashMap<>();
+ metadata.put("org.apache.spark.version", VersionUtils.shortVersion(HoodieSparkUtils.getSparkVersion()));
Review Comment:
We can. What should the key be?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345738287
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,47 +18,149 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
+import org.apache.hudi.SparkAdapterSupport;
+import org.apache.hudi.avro.HoodieAvroUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
- * Hoodie Write Support for directly writing Row to Parquet.
+ * Hoodie Write Support for directly writing Row to Parquet and adding the Hudi bloom index to the file metadata.
+ * The implementation is adapted from Spark's {@link org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport} but has the following major differences:
+ * <ul>
+ * <li>Decimals are always written with the legacy format to ensure compatibility with parquet-avro and other readers</li>
+ * <li>Writing 2-level or 3-level lists is configurable with either the Hudi option, hoodie.parquet.writelegacyformat.enabled, or the parquet-avro option, parquet.avro.write-old-list-structure
+ * to ensure consistency across writer paths.</li>
+ * <li>The scale of the timestamps is determined by the Hudi writer schema instead of relying on Spark configuration</li>
+ * </ul>
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
+ private static final Schema MAP_KEY_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final String MAP_REPEATED_NAME = "key_value";
+ private static final String MAP_KEY_NAME = "key";
+ private static final String MAP_VALUE_NAME = "value";
private final Configuration hadoopConf;
private final Option<HoodieBloomFilterWriteSupport<UTF8String>> bloomFilterWriteSupportOpt;
+ private final byte[] decimalBuffer = new byte[Decimal.minBytesForPrecision()[DecimalType.MAX_PRECISION()]];
+ private final Enumeration.Value datetimeRebaseMode = (Enumeration.Value) SparkAdapterSupport.getSparkAdapter().getDateTimeRebaseMode();
+ private final Function1<Object, Object> dateRebaseFunction = DataSourceUtils.createDateRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private final Function1<Object, Object> timestampRebaseFunction = DataSourceUtils.createTimestampRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private RecordConsumer recordConsumer;
+ private final boolean writeLegacyListFormat;
+ private final ValueWriter[] rootFieldWriters;
+ private final Schema avroSchema;
- public HoodieRowParquetWriteSupport(Configuration conf, StructType structType, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
+ public HoodieRowParquetWriteSupport(Configuration conf, StructType schema, Option<BloomFilter> bloomFilterOpt, HoodieWriteConfig config) {
Configuration hadoopConf = new Configuration(conf);
- hadoopConf.set("spark.sql.parquet.writeLegacyFormat", config.getStringOrDefault(HoodieStorageConfig.PARQUET_WRITE_LEGACY_FORMAT_ENABLED, "false"));
+ String writeLegacyFormatEnabled = config.getStringOrDefault(HoodieStorageConfig.PARQUET_WRITE_LEGACY_FORMAT_ENABLED, "false");
+ hadoopConf.set("spark.sql.parquet.writeLegacyFormat", writeLegacyFormatEnabled);
hadoopConf.set("spark.sql.parquet.outputTimestampType", config.getStringOrDefault(HoodieStorageConfig.PARQUET_OUTPUT_TIMESTAMP_TYPE));
hadoopConf.set("spark.sql.parquet.fieldId.write.enabled", config.getStringOrDefault(PARQUET_FIELD_ID_WRITE_ENABLED));
- setSchema(structType, hadoopConf);
-
+ this.writeLegacyListFormat = Boolean.parseBoolean(writeLegacyFormatEnabled)
+ || Boolean.parseBoolean(config.getStringOrDefault(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE, "false"));
+ this.avroSchema = SerDeHelper.fromJson(config.getInternalSchema()).map(internalSchema -> AvroInternalSchemaConverter.convert(internalSchema, "spark_schema"))
+ .orElseGet(() -> {
+ Schema parsedSchema = new Schema.Parser().parse(config.getWriteSchema());
+ if (config.populateMetaFields()) {
+ return HoodieAvroUtils.addMetadataFields(parsedSchema, config.allowOperationMetadataField());
+ }
+ return parsedSchema;
+ });
+ ParquetWriteSupport.setSchema(HoodieInternalRowUtils.getCachedSchema(avroSchema), hadoopConf);
+ this.rootFieldWriters = getFieldWriters(schema, avroSchema);
this.hadoopConf = hadoopConf;
this.bloomFilterWriteSupportOpt = bloomFilterOpt.map(HoodieBloomFilterRowWriteSupport::new);
}
+ private ValueWriter[] getFieldWriters(StructType schema, Schema avroSchema) {
+ Map<String, Integer> fieldNameToIndex = new HashMap<>();
+ for (int i = 0; i < schema.fields().length; i++) {
+ fieldNameToIndex.put(schema.fields()[i].name(), i);
+ }
+ return IntStream.range(0, avroSchema.getFields().size()).mapToObj(i -> {
+ Schema.Field field = avroSchema.getFields().get(i);
+ Integer structIndex = fieldNameToIndex.get(field.name());
+ return makeWriter(field.schema(), schema.fields()[structIndex].dataType());
+ }).toArray(ValueWriter[]::new);
+ }
+
public Configuration getHadoopConf() {
return hadoopConf;
}
+ @Override
+ public WriteContext init(Configuration configuration) {
+ Map<String, String> metadata = new HashMap<>();
+ metadata.put("org.apache.spark.version", VersionUtils.shortVersion(HoodieSparkUtils.getSparkVersion()));
Review Comment:
I didn't see the necessaties, just write enough UTs to ensure the correctness, like covering all the data types we consentrated and fixed here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345695170
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -73,6 +175,173 @@ public void add(UTF8String recordKey) {
bloomFilterWriteSupport.addKey(recordKey));
}
+ @FunctionalInterface
+ private interface ValueWriter {
+ void write(SpecializedGetters row, int ordinal);
+ }
+
+ private void consumeMessage(Runnable writer) {
+ recordConsumer.startMessage();
+ writer.run();
+ recordConsumer.endMessage();
+ }
+
+ private void consumeGroup(Runnable writer) {
+ recordConsumer.startGroup();
+ writer.run();
+ recordConsumer.endGroup();
+ }
+
+ private void consumeField(String field, int index, Runnable writer) {
+ recordConsumer.startField(field, index);
+ writer.run();
+ recordConsumer.endField(field, index);
+ }
+
+ private void writeFields(InternalRow row, Schema schema, ValueWriter[] fieldWriters) {
+ for (int i = 0; i < fieldWriters.length; i++) {
+ int index = i;
+ if (!row.isNullAt(i)) {
+ Schema.Field field = schema.getFields().get(index);
+ consumeField(field.name(), index, () -> fieldWriters[index].write(row, index));
+ }
+ }
+ }
+
+ private ValueWriter makeWriter(Schema avroSchema, DataType dataType) {
+ Schema resolvedSchema = resolveNullableSchema(avroSchema);
+ Schema.Type type = resolvedSchema.getType();
+ LogicalType logicalType = resolvedSchema.getLogicalType();
+ switch (type) {
+ case BOOLEAN:
+ return (row, ordinal) -> recordConsumer.addBoolean(row.getBoolean(ordinal));
+ case INT:
+ if (logicalType != null) {
+ if (logicalType.getName().equals(LogicalTypes.date().getName())) {
+ return (row, ordinal) -> recordConsumer.addInteger((Integer) dateRebaseFunction.apply(row.getInt(ordinal)));
+ }
+ }
+ return (row, ordinal) -> recordConsumer.addInteger(row.getInt(ordinal));
+ case LONG:
+ if (logicalType != null) {
+ if (logicalType.getName().equals(LogicalTypes.timestampMillis().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong(DateTimeUtils.microsToMillis((long) timestampRebaseFunction.apply(row.getLong(ordinal))));
Review Comment:
Should `dataType instanceof TimestampType` be added as an additional check to make sure Spark uses `TimestampType` for the Avro field to be defensive?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345020642
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,47 +18,149 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
+import org.apache.hudi.SparkAdapterSupport;
+import org.apache.hudi.avro.HoodieAvroUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
- * Hoodie Write Support for directly writing Row to Parquet.
+ * Hoodie Write Support for directly writing Row to Parquet and adding the Hudi bloom index to the file metadata.
+ * The implementation is adapted from Spark's {@link org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport} but has the following major differences:
+ * <ul>
+ * <li>Decimals are always written with the legacy format to ensure compatibility with parquet-avro and other readers</li>
+ * <li>Writing 2-level or 3-level lists is configurable with either the Hudi option, hoodie.parquet.writelegacyformat.enabled, or the parquet-avro option, parquet.avro.write-old-list-structure
Review Comment:
Wonder why 2-level, 3-level? No need to support higher level of lists?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3287387215
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
}, {
"hash" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491",
"triggerID" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"triggerType" : "PUSH"
}, {
"hash" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8503",
"triggerID" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"triggerType" : "PUSH"
}, {
"hash" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8506",
"triggerID" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"triggerType" : "PUSH"
}, {
"hash" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8507",
"triggerID" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"triggerType" : "PUSH"
}, {
"hash" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8510",
"triggerID" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"triggerType" : "PUSH"
}, {
"hash" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8511",
"triggerID" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"triggerType" : "PUSH"
}, {
"hash" : "3ff5acf174c775406ffb54be92650b03718bf48b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8513",
"triggerID" : "3ff5acf174c775406ffb54be92650b03718bf48b",
"triggerType" : "PUSH"
}, {
"hash" : "25a0be4048eb5d9e8c082ad7ff1026053a20747c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "25a0be4048eb5d9e8c082ad7ff1026053a20747c",
"triggerType" : "PUSH"
}, {
"hash" : "27b7921b67ae40d374478457fa51b7ffc7336f50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27b7921b67ae40d374478457fa51b7ffc7336f50",
"triggerType" : "PUSH"
}, {
"hash" : "55213c73a0d42e87f0132b15eb793ef33e3561d3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "55213c73a0d42e87f0132b15eb793ef33e3561d3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 73dd1c38e7cc54233a051cb85222f56dd4cf22b3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8511)
* 3ff5acf174c775406ffb54be92650b03718bf48b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8513)
* 25a0be4048eb5d9e8c082ad7ff1026053a20747c UNKNOWN
* 27b7921b67ae40d374478457fa51b7ffc7336f50 UNKNOWN
* 55213c73a0d42e87f0132b15eb793ef33e3561d3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345938497
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,47 +18,149 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
+import org.apache.hudi.SparkAdapterSupport;
+import org.apache.hudi.avro.HoodieAvroUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
- * Hoodie Write Support for directly writing Row to Parquet.
+ * Hoodie Write Support for directly writing Row to Parquet and adding the Hudi bloom index to the file metadata.
+ * The implementation is adapted from Spark's {@link org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport} but has the following major differences:
+ * <ul>
+ * <li>Decimals are always written with the legacy format to ensure compatibility with parquet-avro and other readers</li>
+ * <li>Writing 2-level or 3-level lists is configurable with either the Hudi option, hoodie.parquet.writelegacyformat.enabled, or the parquet-avro option, parquet.avro.write-old-list-structure
+ * to ensure consistency across writer paths.</li>
+ * <li>The scale of the timestamps is determined by the Hudi writer schema instead of relying on Spark configuration</li>
+ * </ul>
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
+ private static final Schema MAP_KEY_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final String MAP_REPEATED_NAME = "key_value";
+ private static final String MAP_KEY_NAME = "key";
+ private static final String MAP_VALUE_NAME = "value";
private final Configuration hadoopConf;
private final Option<HoodieBloomFilterWriteSupport<UTF8String>> bloomFilterWriteSupportOpt;
+ private final byte[] decimalBuffer = new byte[Decimal.minBytesForPrecision()[DecimalType.MAX_PRECISION()]];
+ private final Enumeration.Value datetimeRebaseMode = (Enumeration.Value) SparkAdapterSupport.getSparkAdapter().getDateTimeRebaseMode();
+ private final Function1<Object, Object> dateRebaseFunction = DataSourceUtils.createDateRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private final Function1<Object, Object> timestampRebaseFunction = DataSourceUtils.createTimestampRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private RecordConsumer recordConsumer;
+ private final boolean writeLegacyListFormat;
+ private final ValueWriter[] rootFieldWriters;
+ private final Schema avroSchema;
- public HoodieRowParquetWriteSupport(Configuration conf, StructType structType, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
+ public HoodieRowParquetWriteSupport(Configuration conf, StructType schema, Option<BloomFilter> bloomFilterOpt, HoodieWriteConfig config) {
Configuration hadoopConf = new Configuration(conf);
- hadoopConf.set("spark.sql.parquet.writeLegacyFormat", config.getStringOrDefault(HoodieStorageConfig.PARQUET_WRITE_LEGACY_FORMAT_ENABLED, "false"));
+ String writeLegacyFormatEnabled = config.getStringOrDefault(HoodieStorageConfig.PARQUET_WRITE_LEGACY_FORMAT_ENABLED, "false");
+ hadoopConf.set("spark.sql.parquet.writeLegacyFormat", writeLegacyFormatEnabled);
hadoopConf.set("spark.sql.parquet.outputTimestampType", config.getStringOrDefault(HoodieStorageConfig.PARQUET_OUTPUT_TIMESTAMP_TYPE));
hadoopConf.set("spark.sql.parquet.fieldId.write.enabled", config.getStringOrDefault(PARQUET_FIELD_ID_WRITE_ENABLED));
- setSchema(structType, hadoopConf);
-
+ this.writeLegacyListFormat = Boolean.parseBoolean(writeLegacyFormatEnabled)
+ || Boolean.parseBoolean(config.getStringOrDefault(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE, "false"));
+ this.avroSchema = SerDeHelper.fromJson(config.getInternalSchema()).map(internalSchema -> AvroInternalSchemaConverter.convert(internalSchema, "spark_schema"))
+ .orElseGet(() -> {
+ Schema parsedSchema = new Schema.Parser().parse(config.getWriteSchema());
+ if (config.populateMetaFields()) {
+ return HoodieAvroUtils.addMetadataFields(parsedSchema, config.allowOperationMetadataField());
+ }
+ return parsedSchema;
+ });
+ ParquetWriteSupport.setSchema(HoodieInternalRowUtils.getCachedSchema(avroSchema), hadoopConf);
+ this.rootFieldWriters = getFieldWriters(schema, avroSchema);
this.hadoopConf = hadoopConf;
this.bloomFilterWriteSupportOpt = bloomFilterOpt.map(HoodieBloomFilterRowWriteSupport::new);
}
+ private ValueWriter[] getFieldWriters(StructType schema, Schema avroSchema) {
+ Map<String, Integer> fieldNameToIndex = new HashMap<>();
+ for (int i = 0; i < schema.fields().length; i++) {
+ fieldNameToIndex.put(schema.fields()[i].name(), i);
+ }
+ return IntStream.range(0, avroSchema.getFields().size()).mapToObj(i -> {
+ Schema.Field field = avroSchema.getFields().get(i);
+ Integer structIndex = fieldNameToIndex.get(field.name());
+ return makeWriter(field.schema(), schema.fields()[structIndex].dataType());
+ }).toArray(ValueWriter[]::new);
+ }
+
public Configuration getHadoopConf() {
return hadoopConf;
}
+ @Override
+ public WriteContext init(Configuration configuration) {
+ Map<String, String> metadata = new HashMap<>();
+ metadata.put("org.apache.spark.version", VersionUtils.shortVersion(HoodieSparkUtils.getSparkVersion()));
Review Comment:
I don't have a strong opinion. This is going to be handy for debugging parquet files in production in case we see data type issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345631899
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,47 +18,152 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
+import org.apache.hudi.SparkAdapterSupport$;
+import org.apache.hudi.avro.HoodieAvroUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
+import org.apache.hudi.common.table.HoodieTableConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
+import static org.apache.hudi.config.HoodieWriteConfig.ALLOW_OPERATION_METADATA_FIELD;
+import static org.apache.hudi.config.HoodieWriteConfig.AVRO_SCHEMA_STRING;
+import static org.apache.hudi.config.HoodieWriteConfig.INTERNAL_SCHEMA_STRING;
/**
- * Hoodie Write Support for directly writing Row to Parquet.
+ * Hoodie Write Support for directly writing Row to Parquet and adding the Hudi bloom index to the file metadata.
+ * The implementation is adapted from Spark's {@link org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport} but has the following major differences:
+ * <ul>
+ * <li>Decimals are always written with the legacy format to ensure compatibility with parquet-avro and other readers</li>
+ * <li>Writing 2-level or 3-level lists is configurable with either the Hudi option, hoodie.parquet.writelegacyformat.enabled, or the parquet-avro option, parquet.avro.write-old-list-structure
+ * to ensure consistency across writer paths.</li>
+ * <li>The scale of the timestamps is determined by the Hudi writer schema instead of relying on Spark configuration</li>
+ * </ul>
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
+ private static final Schema MAP_KEY_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final String MAP_REPEATED_NAME = "key_value";
+ private static final String MAP_KEY_NAME = "key";
+ private static final String MAP_VALUE_NAME = "value";
private final Configuration hadoopConf;
private final Option<HoodieBloomFilterWriteSupport<UTF8String>> bloomFilterWriteSupportOpt;
+ private final byte[] decimalBuffer = new byte[Decimal.minBytesForPrecision()[DecimalType.MAX_PRECISION()]];
+ private final Enumeration.Value datetimeRebaseMode = (Enumeration.Value) SparkAdapterSupport$.MODULE$.sparkAdapter().getDateTimeRebaseMode();
+ private final Function1<Object, Object> dateRebaseFunction = DataSourceUtils.createDateRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private final Function1<Object, Object> timestampRebaseFunction = DataSourceUtils.createTimestampRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private RecordConsumer recordConsumer;
+ private final boolean writeLegacyListFormat;
+ private final ValueWriter[] rootFieldWriters;
+ private final Schema avroSchema;
- public HoodieRowParquetWriteSupport(Configuration conf, StructType structType, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
+ public HoodieRowParquetWriteSupport(Configuration conf, StructType schema, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
Configuration hadoopConf = new Configuration(conf);
- hadoopConf.set("spark.sql.parquet.writeLegacyFormat", config.getStringOrDefault(HoodieStorageConfig.PARQUET_WRITE_LEGACY_FORMAT_ENABLED, "false"));
+ String writeLegacyFormatEnabled = config.getStringOrDefault(HoodieStorageConfig.PARQUET_WRITE_LEGACY_FORMAT_ENABLED, "false");
+ hadoopConf.set("spark.sql.parquet.writeLegacyFormat", writeLegacyFormatEnabled);
hadoopConf.set("spark.sql.parquet.outputTimestampType", config.getStringOrDefault(HoodieStorageConfig.PARQUET_OUTPUT_TIMESTAMP_TYPE));
hadoopConf.set("spark.sql.parquet.fieldId.write.enabled", config.getStringOrDefault(PARQUET_FIELD_ID_WRITE_ENABLED));
- setSchema(structType, hadoopConf);
-
+ this.writeLegacyListFormat = Boolean.parseBoolean(writeLegacyFormatEnabled)
+ || Boolean.parseBoolean(config.getStringOrDefault(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE, "false"));
+ this.avroSchema = SerDeHelper.fromJson(config.getString(INTERNAL_SCHEMA_STRING)).map(internalSchema -> AvroInternalSchemaConverter.convert(internalSchema, "spark_schema"))
+ .orElseGet(() -> {
+ Schema parsedSchema = new Schema.Parser().parse(config.getString(AVRO_SCHEMA_STRING));
Review Comment:
I think the schema here should also consider `WRITE_SCHEMA_OVERRIDE` to be aligned with `getWriteSchema()`.
```
HoodieWriteConfig:
public String getWriteSchema() {
if (props.containsKey(WRITE_SCHEMA_OVERRIDE.key())) {
return getString(WRITE_SCHEMA_OVERRIDE);
}
return getSchema();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345625363
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieInternalRowFileWriterFactory.java:
##########
@@ -54,24 +55,26 @@ public class HoodieInternalRowFileWriterFactory {
public static HoodieInternalRowFileWriter getInternalRowFileWriter(StoragePath path,
HoodieTable hoodieTable,
HoodieWriteConfig writeConfig,
- StructType schema)
+ StructType schema,
+ Schema avroSchema)
Review Comment:
Got it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345504416
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,47 +18,149 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
+import org.apache.hudi.SparkAdapterSupport;
+import org.apache.hudi.avro.HoodieAvroUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
- * Hoodie Write Support for directly writing Row to Parquet.
+ * Hoodie Write Support for directly writing Row to Parquet and adding the Hudi bloom index to the file metadata.
+ * The implementation is adapted from Spark's {@link org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport} but has the following major differences:
+ * <ul>
+ * <li>Decimals are always written with the legacy format to ensure compatibility with parquet-avro and other readers</li>
+ * <li>Writing 2-level or 3-level lists is configurable with either the Hudi option, hoodie.parquet.writelegacyformat.enabled, or the parquet-avro option, parquet.avro.write-old-list-structure
+ * to ensure consistency across writer paths.</li>
+ * <li>The scale of the timestamps is determined by the Hudi writer schema instead of relying on Spark configuration</li>
+ * </ul>
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
+ private static final Schema MAP_KEY_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final String MAP_REPEATED_NAME = "key_value";
+ private static final String MAP_KEY_NAME = "key";
+ private static final String MAP_VALUE_NAME = "value";
private final Configuration hadoopConf;
private final Option<HoodieBloomFilterWriteSupport<UTF8String>> bloomFilterWriteSupportOpt;
+ private final byte[] decimalBuffer = new byte[Decimal.minBytesForPrecision()[DecimalType.MAX_PRECISION()]];
+ private final Enumeration.Value datetimeRebaseMode = (Enumeration.Value) SparkAdapterSupport.getSparkAdapter().getDateTimeRebaseMode();
+ private final Function1<Object, Object> dateRebaseFunction = DataSourceUtils.createDateRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private final Function1<Object, Object> timestampRebaseFunction = DataSourceUtils.createTimestampRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
Review Comment:
Do we care about these which are in the `ParquetWriteSupport`:
```
private val int96RebaseMode = LegacyBehaviorPolicy.withName(
SQLConf.get.getConf(SQLConf.PARQUET_INT96_REBASE_MODE_IN_WRITE))
private val int96RebaseFunc = DataSourceUtils.createTimestampRebaseFuncInWrite(
int96RebaseMode, "Parquet INT96")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345033572
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,47 +18,149 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
+import org.apache.hudi.SparkAdapterSupport;
+import org.apache.hudi.avro.HoodieAvroUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
- * Hoodie Write Support for directly writing Row to Parquet.
+ * Hoodie Write Support for directly writing Row to Parquet and adding the Hudi bloom index to the file metadata.
+ * The implementation is adapted from Spark's {@link org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport} but has the following major differences:
+ * <ul>
+ * <li>Decimals are always written with the legacy format to ensure compatibility with parquet-avro and other readers</li>
+ * <li>Writing 2-level or 3-level lists is configurable with either the Hudi option, hoodie.parquet.writelegacyformat.enabled, or the parquet-avro option, parquet.avro.write-old-list-structure
Review Comment:
ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3283307104
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
}, {
"hash" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5bca64047318b420b21f7dadd1f274d1330319e1 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488)
* 00f82b3c501021529d2fa67b4fb5860eef21d634 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3282401988
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e5ea200c5cea9f5c2dda1554b765bfe11e276ae7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2342357576
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,39 +18,90 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.LegacyBehaviorPolicy;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
* Hoodie Write Support for directly writing Row to Parquet.
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
+ private static final Schema MAP_KEY_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final String MAP_REPEATED_NAME = "key_value";
+ private static final String MAP_KEY_NAME = "key";
+ private static final String MAP_VALUE_NAME = "value";
private final Configuration hadoopConf;
private final Option<HoodieBloomFilterWriteSupport<UTF8String>> bloomFilterWriteSupportOpt;
+ private final byte[] decimalBuffer = new byte[Decimal.minBytesForPrecision()[DecimalType.MAX_PRECISION()]];
+ private final Enumeration.Value datetimeRebaseMode = LegacyBehaviorPolicy.withName(SQLConf.get().getConf(SQLConf.PARQUET_REBASE_MODE_IN_WRITE()));
+ private final Function1<Object, Object> dateRebaseFunction = DataSourceUtils.createDateRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private final Function1<Object, Object> timestampRebaseFunction = DataSourceUtils.createTimestampRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private RecordConsumer recordConsumer;
+ private final boolean writeLegacyListFormat;
+ private final ValueWriter[] rootFieldWriters;
+ private final Schema avroSchema;
- public HoodieRowParquetWriteSupport(Configuration conf, StructType structType, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
+ public HoodieRowParquetWriteSupport(Configuration conf, StructType schema, Schema avroSchema, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
Review Comment:
This should be possible. Right now the constructor takes in a `HoodieConfig` should it be updated to `HoodieWriteConfig`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2342559536
##########
hudi-spark-datasource/hudi-spark/src/test/resources/schema_with_logical_types.avsc:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+ "type": "record",
+ "name": "LogicalTypesSample",
+ "namespace": "test",
+ "fields": [
+ {
+ "name": "_row_key",
+ "type": "string"
+ },
+ {
+ "name": "partition_path",
+ "type": "string"
+ },
+ {
+ "name": "ts",
+ "type": ["long", "null"]
+ },
+ {
+ "name": "_hoodie_is_deleted",
+ "type": ["boolean", "null"],
+ "default" : false
+ },
+ {
+ "name": "double_field",
+ "type": "double",
+ "default": 0.0
+ },
+ {
+ "name": "float_field",
+ "type": "float",
+ "default": 0.0
+ },
+ {
+ "name": "int_field",
+ "type": "int",
+ "default": 0
+ },
+ {
+ "name": "long_field",
+ "type": "long",
+ "default": 0
+ },
+ {
+ "name": "string_field",
+ "type": {
+ "type": "string",
+ "avro.java.string": "String"
+ },
+ "default": ""
+ },
+ {
+ "name": "bytes_field",
+ "type": "bytes",
+ "default": ""
+ },
+ {
+ "name": "decimal_field",
+ "type": {
+ "type": "fixed",
+ "name": "decimal_field_testing",
+ "size": 5,
+ "logicalType": "decimal",
+ "precision": 10,
+ "scale": 6
+ },
+ "default": "\u0000"
+ },
+ {
+ "name": "nested_record",
+ "type": [
+ "null",
+ {
+ "type": "record",
+ "name": "Nested",
+ "namespace": "test",
+ "fields": [
+ {
+ "name": "nested_int",
+ "type": "int",
+ "default": 0
+ },
+ {
+ "name": "nested_string",
+ "type": "string"
+ }
+ ]
+ }
+ ],
+ "default": null
+ },
+ {
+ "name":"nullable_map_field",
+ "type":["null", {
+ "type":"map",
+ "values": "Nested"
+ }],
+ "default": null
+ },
+ {
+ "name": "array_field",
+ "type": {
+ "type": "array",
+ "items": "Nested"
+ },
+ "default": []
+ },
+ {
+ "name": "date_nullable_field",
+ "type": [
+ "null",
+ {
+ "type": "int",
+ "logicalType": "date"
+ }
+ ],
+ "default": null
+ },
+ {
+ "name": "timestamp_millis_field",
+ "type":
+ {
+ "type": "long",
+ "logicalType": "timestamp-millis"
+ }
+ },
+ {
+ "name": "timestamp_micros_nullable_field",
+ "type": [
+ "null",
+ {
+ "type": "long",
+ "logicalType": "timestamp-micros"
+ }
+ ],
+ "default": null
+ },
+ {
+ "name": "timestamp_local_millis_nullable_field",
+ "type": [
+ "null",
+ {
+ "type": "long",
+ "logicalType": "local-timestamp-millis"
+ }
+ ],
+ "default": null
+ },
+ {
+ "name": "timestamp_local_micros_field",
+ "type":
+ {
+ "type": "long",
+ "logicalType": "local-timestamp-micros"
+ }
+ }
+ ]
+}
Review Comment:
nit: need an extra empty line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3283310563
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
}, {
"hash" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491",
"triggerID" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5bca64047318b420b21f7dadd1f274d1330319e1 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488)
* 00f82b3c501021529d2fa67b4fb5860eef21d634 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345459119
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,47 +18,149 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
+import org.apache.hudi.SparkAdapterSupport;
+import org.apache.hudi.avro.HoodieAvroUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
- * Hoodie Write Support for directly writing Row to Parquet.
+ * Hoodie Write Support for directly writing Row to Parquet and adding the Hudi bloom index to the file metadata.
+ * The implementation is adapted from Spark's {@link org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport} but has the following major differences:
+ * <ul>
+ * <li>Decimals are always written with the legacy format to ensure compatibility with parquet-avro and other readers</li>
+ * <li>Writing 2-level or 3-level lists is configurable with either the Hudi option, hoodie.parquet.writelegacyformat.enabled, or the parquet-avro option, parquet.avro.write-old-list-structure
+ * to ensure consistency across writer paths.</li>
+ * <li>The scale of the timestamps is determined by the Hudi writer schema instead of relying on Spark configuration</li>
+ * </ul>
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
+ private static final Schema MAP_KEY_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final String MAP_REPEATED_NAME = "key_value";
+ private static final String MAP_KEY_NAME = "key";
+ private static final String MAP_VALUE_NAME = "value";
private final Configuration hadoopConf;
private final Option<HoodieBloomFilterWriteSupport<UTF8String>> bloomFilterWriteSupportOpt;
+ private final byte[] decimalBuffer = new byte[Decimal.minBytesForPrecision()[DecimalType.MAX_PRECISION()]];
+ private final Enumeration.Value datetimeRebaseMode = (Enumeration.Value) SparkAdapterSupport.getSparkAdapter().getDateTimeRebaseMode();
+ private final Function1<Object, Object> dateRebaseFunction = DataSourceUtils.createDateRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private final Function1<Object, Object> timestampRebaseFunction = DataSourceUtils.createTimestampRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private RecordConsumer recordConsumer;
+ private final boolean writeLegacyListFormat;
+ private final ValueWriter[] rootFieldWriters;
+ private final Schema avroSchema;
- public HoodieRowParquetWriteSupport(Configuration conf, StructType structType, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
+ public HoodieRowParquetWriteSupport(Configuration conf, StructType schema, Option<BloomFilter> bloomFilterOpt, HoodieWriteConfig config) {
Review Comment:
Now I see clustering tests have covered the type validation, which should be good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3287367941
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
}, {
"hash" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491",
"triggerID" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"triggerType" : "PUSH"
}, {
"hash" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8503",
"triggerID" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"triggerType" : "PUSH"
}, {
"hash" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8506",
"triggerID" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"triggerType" : "PUSH"
}, {
"hash" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8507",
"triggerID" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"triggerType" : "PUSH"
}, {
"hash" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8510",
"triggerID" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"triggerType" : "PUSH"
}, {
"hash" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8511",
"triggerID" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"triggerType" : "PUSH"
}, {
"hash" : "3ff5acf174c775406ffb54be92650b03718bf48b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3ff5acf174c775406ffb54be92650b03718bf48b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 73dd1c38e7cc54233a051cb85222f56dd4cf22b3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8511)
* 3ff5acf174c775406ffb54be92650b03718bf48b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2344169729
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -73,6 +148,177 @@ public void add(UTF8String recordKey) {
bloomFilterWriteSupport.addKey(recordKey));
}
+ @FunctionalInterface
+ private interface ValueWriter {
+ void write(SpecializedGetters row, int ordinal);
+ }
+
+ private void consumeMessage(Runnable writer) {
+ recordConsumer.startMessage();
Review Comment:
Yes, this is all borrowed from there. Let me update the java doc to do a link to the class and explain the differences.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3282458100
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e5ea200c5cea9f5c2dda1554b765bfe11e276ae7 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474)
* 190cbc827b787e7f347a76add30a94f7621e750d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2344392798
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,39 +18,90 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.LegacyBehaviorPolicy;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
* Hoodie Write Support for directly writing Row to Parquet.
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
+ private static final Schema MAP_KEY_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final String MAP_REPEATED_NAME = "key_value";
+ private static final String MAP_KEY_NAME = "key";
+ private static final String MAP_VALUE_NAME = "value";
private final Configuration hadoopConf;
private final Option<HoodieBloomFilterWriteSupport<UTF8String>> bloomFilterWriteSupportOpt;
+ private final byte[] decimalBuffer = new byte[Decimal.minBytesForPrecision()[DecimalType.MAX_PRECISION()]];
+ private final Enumeration.Value datetimeRebaseMode = LegacyBehaviorPolicy.withName(SQLConf.get().getConf(SQLConf.PARQUET_REBASE_MODE_IN_WRITE()));
+ private final Function1<Object, Object> dateRebaseFunction = DataSourceUtils.createDateRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private final Function1<Object, Object> timestampRebaseFunction = DataSourceUtils.createTimestampRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private RecordConsumer recordConsumer;
+ private final boolean writeLegacyListFormat;
+ private final ValueWriter[] rootFieldWriters;
+ private final Schema avroSchema;
- public HoodieRowParquetWriteSupport(Configuration conf, StructType structType, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
+ public HoodieRowParquetWriteSupport(Configuration conf, StructType schema, Schema avroSchema, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
Review Comment:
@yihua another thing I'm curious about is where does the schema get resolved against the existing table schema? Does that update the schema in the write config?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3287377037
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
}, {
"hash" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491",
"triggerID" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"triggerType" : "PUSH"
}, {
"hash" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8503",
"triggerID" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"triggerType" : "PUSH"
}, {
"hash" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8506",
"triggerID" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"triggerType" : "PUSH"
}, {
"hash" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8507",
"triggerID" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"triggerType" : "PUSH"
}, {
"hash" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8510",
"triggerID" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"triggerType" : "PUSH"
}, {
"hash" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8511",
"triggerID" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"triggerType" : "PUSH"
}, {
"hash" : "3ff5acf174c775406ffb54be92650b03718bf48b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8513",
"triggerID" : "3ff5acf174c775406ffb54be92650b03718bf48b",
"triggerType" : "PUSH"
}, {
"hash" : "25a0be4048eb5d9e8c082ad7ff1026053a20747c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "25a0be4048eb5d9e8c082ad7ff1026053a20747c",
"triggerType" : "PUSH"
}, {
"hash" : "27b7921b67ae40d374478457fa51b7ffc7336f50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27b7921b67ae40d374478457fa51b7ffc7336f50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 73dd1c38e7cc54233a051cb85222f56dd4cf22b3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8511)
* 3ff5acf174c775406ffb54be92650b03718bf48b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8513)
* 25a0be4048eb5d9e8c082ad7ff1026053a20747c UNKNOWN
* 27b7921b67ae40d374478457fa51b7ffc7336f50 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2342353579
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/HoodieDataTypeUtils.scala:
##########
@@ -37,72 +29,4 @@ object HoodieDataTypeUtils {
*/
def parseStructTypeFromJson(jsonSchema: String): StructType =
StructType.fromString(jsonSchema)
-
- def canUseRowWriter(schema: Schema, conf: Configuration): Boolean = {
- if (conf.getBoolean(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE, true)) {
- // if we can write lists with the old list structure, we can use row writer regardless of decimal precision
- true
- } else if (!HoodieAvroUtils.hasSmallPrecisionDecimalField(schema)) {
- true
- } else {
- // small precision decimals require the legacy write mode but lists and maps require the new write mode when
- // WRITE_OLD_LIST_STRUCTURE is false so we can only use row writer if one is present and the other is not
- if (HoodieAvroUtils.hasListOrMapField(schema)) {
- log.warn("Cannot use row writer due to presence of list or map with a small precision decimal field")
- false
- } else {
- true
- }
- }
- }
Review Comment:
That is correct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3282670108
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 190cbc827b787e7f347a76add30a94f7621e750d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3283386498
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
}, {
"hash" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491",
"triggerID" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 00f82b3c501021529d2fa67b4fb5860eef21d634 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3287154226
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
}, {
"hash" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491",
"triggerID" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"triggerType" : "PUSH"
}, {
"hash" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8503",
"triggerID" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"triggerType" : "PUSH"
}, {
"hash" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8506",
"triggerID" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"triggerType" : "PUSH"
}, {
"hash" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8507",
"triggerID" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"triggerType" : "PUSH"
}, {
"hash" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8510",
"triggerID" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"triggerType" : "PUSH"
}, {
"hash" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8511",
"triggerID" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d4438bc21cf60af1858e4f0602d91115cc41fced Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8510)
* 73dd1c38e7cc54233a051cb85222f56dd4cf22b3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8511)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2346862242
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -73,6 +189,180 @@ public void add(UTF8String recordKey) {
bloomFilterWriteSupport.addKey(recordKey));
}
+ @FunctionalInterface
+ private interface ValueWriter {
+ void write(SpecializedGetters row, int ordinal);
+ }
+
+ private void consumeMessage(Runnable writer) {
+ recordConsumer.startMessage();
+ writer.run();
+ recordConsumer.endMessage();
+ }
+
+ private void consumeGroup(Runnable writer) {
+ recordConsumer.startGroup();
+ writer.run();
+ recordConsumer.endGroup();
+ }
+
+ private void consumeField(String field, int index, Runnable writer) {
+ recordConsumer.startField(field, index);
+ writer.run();
+ recordConsumer.endField(field, index);
+ }
+
+ private void writeFields(InternalRow row, StructType schema, ValueWriter[] fieldWriters) {
+ for (int i = 0; i < row.numFields(); i++) {
+ int index = i;
+ if (!row.isNullAt(i)) {
+ StructField field = schema.fields()[i];
+ consumeField(field.name(), index, () -> fieldWriters[index].write(row, index));
+ }
Review Comment:
nit: reminder to remove `index`; `field = schema.fields()[i]` can also be inlined on L220
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -73,6 +189,180 @@ public void add(UTF8String recordKey) {
bloomFilterWriteSupport.addKey(recordKey));
}
+ @FunctionalInterface
+ private interface ValueWriter {
+ void write(SpecializedGetters row, int ordinal);
+ }
+
+ private void consumeMessage(Runnable writer) {
+ recordConsumer.startMessage();
+ writer.run();
+ recordConsumer.endMessage();
+ }
+
+ private void consumeGroup(Runnable writer) {
+ recordConsumer.startGroup();
+ writer.run();
+ recordConsumer.endGroup();
+ }
+
+ private void consumeField(String field, int index, Runnable writer) {
+ recordConsumer.startField(field, index);
+ writer.run();
+ recordConsumer.endField(field, index);
+ }
+
+ private void writeFields(InternalRow row, StructType schema, ValueWriter[] fieldWriters) {
+ for (int i = 0; i < row.numFields(); i++) {
+ int index = i;
+ if (!row.isNullAt(i)) {
+ StructField field = schema.fields()[i];
+ consumeField(field.name(), index, () -> fieldWriters[index].write(row, index));
+ }
+ }
+ }
+
+ private ValueWriter makeWriter(Schema avroSchema, DataType dataType) {
+ Schema resolvedSchema = avroSchema == null ? null : resolveNullableSchema(avroSchema);
+ LogicalType logicalType = resolvedSchema != null ? resolvedSchema.getLogicalType() : null;
+
+ if (dataType == DataTypes.BooleanType) {
+ return (row, ordinal) -> recordConsumer.addBoolean(row.getBoolean(ordinal));
+ } else if (dataType == DataTypes.DateType) {
+ return (row, ordinal) -> recordConsumer.addInteger((Integer) dateRebaseFunction.apply(row.getInt(ordinal)));
+ } else if (dataType == DataTypes.ShortType) {
+ return (row, ordinal) -> recordConsumer.addInteger(row.getShort(ordinal));
+ } else if (dataType == DataTypes.IntegerType) {
Review Comment:
I see `ParquetWriteSupport` also handles `YearMonthIntervalType` for integers. Not sure if this class need to do that.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -73,6 +189,180 @@ public void add(UTF8String recordKey) {
bloomFilterWriteSupport.addKey(recordKey));
}
+ @FunctionalInterface
+ private interface ValueWriter {
+ void write(SpecializedGetters row, int ordinal);
+ }
+
+ private void consumeMessage(Runnable writer) {
+ recordConsumer.startMessage();
+ writer.run();
+ recordConsumer.endMessage();
+ }
+
+ private void consumeGroup(Runnable writer) {
+ recordConsumer.startGroup();
+ writer.run();
+ recordConsumer.endGroup();
+ }
+
+ private void consumeField(String field, int index, Runnable writer) {
+ recordConsumer.startField(field, index);
+ writer.run();
+ recordConsumer.endField(field, index);
+ }
+
+ private void writeFields(InternalRow row, StructType schema, ValueWriter[] fieldWriters) {
+ for (int i = 0; i < row.numFields(); i++) {
+ int index = i;
+ if (!row.isNullAt(i)) {
+ StructField field = schema.fields()[i];
+ consumeField(field.name(), index, () -> fieldWriters[index].write(row, index));
+ }
+ }
+ }
+
+ private ValueWriter makeWriter(Schema avroSchema, DataType dataType) {
+ Schema resolvedSchema = avroSchema == null ? null : resolveNullableSchema(avroSchema);
+ LogicalType logicalType = resolvedSchema != null ? resolvedSchema.getLogicalType() : null;
+
+ if (dataType == DataTypes.BooleanType) {
+ return (row, ordinal) -> recordConsumer.addBoolean(row.getBoolean(ordinal));
+ } else if (dataType == DataTypes.DateType) {
+ return (row, ordinal) -> recordConsumer.addInteger((Integer) dateRebaseFunction.apply(row.getInt(ordinal)));
+ } else if (dataType == DataTypes.ShortType) {
+ return (row, ordinal) -> recordConsumer.addInteger(row.getShort(ordinal));
+ } else if (dataType == DataTypes.IntegerType) {
+ return (row, ordinal) -> recordConsumer.addInteger(row.getInt(ordinal));
+ } else if (dataType == DataTypes.LongType) {
+ return (row, ordinal) -> recordConsumer.addLong(row.getLong(ordinal));
Review Comment:
Similarly, `DayTimeIntervalType` is handled by `ParquetWriteSupport`. We can intentionally skip it if not supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3282582438
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e5ea200c5cea9f5c2dda1554b765bfe11e276ae7 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474)
* 190cbc827b787e7f347a76add30a94f7621e750d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3286016679
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
}, {
"hash" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491",
"triggerID" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"triggerType" : "PUSH"
}, {
"hash" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8503",
"triggerID" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"triggerType" : "PUSH"
}, {
"hash" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2380d2b0bc91be8f86fe87911f7c503d938a072f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8503)
* 1406c146e1fa7c3e449bfe9e99fbceaafb1ad209 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345512027
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,47 +18,149 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
+import org.apache.hudi.SparkAdapterSupport;
+import org.apache.hudi.avro.HoodieAvroUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
- * Hoodie Write Support for directly writing Row to Parquet.
+ * Hoodie Write Support for directly writing Row to Parquet and adding the Hudi bloom index to the file metadata.
+ * The implementation is adapted from Spark's {@link org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport} but has the following major differences:
+ * <ul>
+ * <li>Decimals are always written with the legacy format to ensure compatibility with parquet-avro and other readers</li>
+ * <li>Writing 2-level or 3-level lists is configurable with either the Hudi option, hoodie.parquet.writelegacyformat.enabled, or the parquet-avro option, parquet.avro.write-old-list-structure
+ * to ensure consistency across writer paths.</li>
+ * <li>The scale of the timestamps is determined by the Hudi writer schema instead of relying on Spark configuration</li>
+ * </ul>
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
+ private static final Schema MAP_KEY_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final String MAP_REPEATED_NAME = "key_value";
+ private static final String MAP_KEY_NAME = "key";
+ private static final String MAP_VALUE_NAME = "value";
private final Configuration hadoopConf;
private final Option<HoodieBloomFilterWriteSupport<UTF8String>> bloomFilterWriteSupportOpt;
+ private final byte[] decimalBuffer = new byte[Decimal.minBytesForPrecision()[DecimalType.MAX_PRECISION()]];
+ private final Enumeration.Value datetimeRebaseMode = (Enumeration.Value) SparkAdapterSupport.getSparkAdapter().getDateTimeRebaseMode();
+ private final Function1<Object, Object> dateRebaseFunction = DataSourceUtils.createDateRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private final Function1<Object, Object> timestampRebaseFunction = DataSourceUtils.createTimestampRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
Review Comment:
We don't support int96 in the avro schema so I left them out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345550720
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -73,6 +175,173 @@ public void add(UTF8String recordKey) {
bloomFilterWriteSupport.addKey(recordKey));
}
+ @FunctionalInterface
+ private interface ValueWriter {
+ void write(SpecializedGetters row, int ordinal);
+ }
+
+ private void consumeMessage(Runnable writer) {
+ recordConsumer.startMessage();
+ writer.run();
+ recordConsumer.endMessage();
+ }
+
+ private void consumeGroup(Runnable writer) {
+ recordConsumer.startGroup();
+ writer.run();
+ recordConsumer.endGroup();
+ }
+
+ private void consumeField(String field, int index, Runnable writer) {
+ recordConsumer.startField(field, index);
+ writer.run();
+ recordConsumer.endField(field, index);
+ }
+
+ private void writeFields(InternalRow row, Schema schema, ValueWriter[] fieldWriters) {
+ for (int i = 0; i < fieldWriters.length; i++) {
+ int index = i;
+ if (!row.isNullAt(i)) {
+ Schema.Field field = schema.getFields().get(index);
+ consumeField(field.name(), index, () -> fieldWriters[index].write(row, index));
+ }
+ }
+ }
+
+ private ValueWriter makeWriter(Schema avroSchema, DataType dataType) {
+ Schema resolvedSchema = resolveNullableSchema(avroSchema);
+ Schema.Type type = resolvedSchema.getType();
+ LogicalType logicalType = resolvedSchema.getLogicalType();
+ switch (type) {
+ case BOOLEAN:
+ return (row, ordinal) -> recordConsumer.addBoolean(row.getBoolean(ordinal));
+ case INT:
+ if (logicalType != null) {
+ if (logicalType.getName().equals(LogicalTypes.date().getName())) {
+ return (row, ordinal) -> recordConsumer.addInteger((Integer) dateRebaseFunction.apply(row.getInt(ordinal)));
+ }
+ }
+ return (row, ordinal) -> recordConsumer.addInteger(row.getInt(ordinal));
+ case LONG:
+ if (logicalType != null) {
+ if (logicalType.getName().equals(LogicalTypes.timestampMillis().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong(DateTimeUtils.microsToMillis((long) timestampRebaseFunction.apply(row.getLong(ordinal))));
Review Comment:
Yes, the timestamp in spark is always micros under the hood based on my understanding
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3282406497
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e5ea200c5cea9f5c2dda1554b765bfe11e276ae7 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345407541
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,47 +18,149 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
+import org.apache.hudi.SparkAdapterSupport;
+import org.apache.hudi.avro.HoodieAvroUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
- * Hoodie Write Support for directly writing Row to Parquet.
+ * Hoodie Write Support for directly writing Row to Parquet and adding the Hudi bloom index to the file metadata.
+ * The implementation is adapted from Spark's {@link org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport} but has the following major differences:
+ * <ul>
+ * <li>Decimals are always written with the legacy format to ensure compatibility with parquet-avro and other readers</li>
+ * <li>Writing 2-level or 3-level lists is configurable with either the Hudi option, hoodie.parquet.writelegacyformat.enabled, or the parquet-avro option, parquet.avro.write-old-list-structure
+ * to ensure consistency across writer paths.</li>
+ * <li>The scale of the timestamps is determined by the Hudi writer schema instead of relying on Spark configuration</li>
+ * </ul>
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
+ private static final Schema MAP_KEY_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final String MAP_REPEATED_NAME = "key_value";
+ private static final String MAP_KEY_NAME = "key";
+ private static final String MAP_VALUE_NAME = "value";
private final Configuration hadoopConf;
private final Option<HoodieBloomFilterWriteSupport<UTF8String>> bloomFilterWriteSupportOpt;
+ private final byte[] decimalBuffer = new byte[Decimal.minBytesForPrecision()[DecimalType.MAX_PRECISION()]];
+ private final Enumeration.Value datetimeRebaseMode = (Enumeration.Value) SparkAdapterSupport.getSparkAdapter().getDateTimeRebaseMode();
+ private final Function1<Object, Object> dateRebaseFunction = DataSourceUtils.createDateRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private final Function1<Object, Object> timestampRebaseFunction = DataSourceUtils.createTimestampRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private RecordConsumer recordConsumer;
+ private final boolean writeLegacyListFormat;
+ private final ValueWriter[] rootFieldWriters;
+ private final Schema avroSchema;
- public HoodieRowParquetWriteSupport(Configuration conf, StructType structType, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
+ public HoodieRowParquetWriteSupport(Configuration conf, StructType schema, Option<BloomFilter> bloomFilterOpt, HoodieWriteConfig config) {
Review Comment:
Shall we add functional tests to write parquet files from this write support vs Avro writer and validate that the parquet files use the same format for timestamp, list, etc.? Or is this already covered by other functional tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345622464
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,39 +18,90 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
+import org.apache.hudi.SparkAdapterSupport;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
* Hoodie Write Support for directly writing Row to Parquet.
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
Review Comment:
@danny0405 are `RowDataParquetWriteSupport` and `ParquetRowDataWriter`'s write behavior on decimal, list, and timestamps consistent with `HoodieAvroWriteSupport` and `HoodieRowParquetWriteSupport`, so there's interoperability between Spark and Flink on a Hudi table?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345640089
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,39 +18,90 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.LegacyBehaviorPolicy;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
* Hoodie Write Support for directly writing Row to Parquet.
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
+ private static final Schema MAP_KEY_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final String MAP_REPEATED_NAME = "key_value";
+ private static final String MAP_KEY_NAME = "key";
+ private static final String MAP_VALUE_NAME = "value";
private final Configuration hadoopConf;
private final Option<HoodieBloomFilterWriteSupport<UTF8String>> bloomFilterWriteSupportOpt;
+ private final byte[] decimalBuffer = new byte[Decimal.minBytesForPrecision()[DecimalType.MAX_PRECISION()]];
+ private final Enumeration.Value datetimeRebaseMode = LegacyBehaviorPolicy.withName(SQLConf.get().getConf(SQLConf.PARQUET_REBASE_MODE_IN_WRITE()));
+ private final Function1<Object, Object> dateRebaseFunction = DataSourceUtils.createDateRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private final Function1<Object, Object> timestampRebaseFunction = DataSourceUtils.createTimestampRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private RecordConsumer recordConsumer;
+ private final boolean writeLegacyListFormat;
+ private final ValueWriter[] rootFieldWriters;
+ private final Schema avroSchema;
- public HoodieRowParquetWriteSupport(Configuration conf, StructType structType, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
+ public HoodieRowParquetWriteSupport(Configuration conf, StructType schema, Schema avroSchema, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
Configuration hadoopConf = new Configuration(conf);
- hadoopConf.set("spark.sql.parquet.writeLegacyFormat", config.getStringOrDefault(HoodieStorageConfig.PARQUET_WRITE_LEGACY_FORMAT_ENABLED, "false"));
+ String writeLegacyFormatEnabled = config.getStringOrDefault(HoodieStorageConfig.PARQUET_WRITE_LEGACY_FORMAT_ENABLED, "false");
+ hadoopConf.set("spark.sql.parquet.writeLegacyFormat", writeLegacyFormatEnabled);
Review Comment:
@jonvex @linliu-code should the unncessary parquet write config settings be removed from `SparkReaderContextFactory` to avoid confusion?
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,39 +18,90 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.LegacyBehaviorPolicy;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
* Hoodie Write Support for directly writing Row to Parquet.
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
+ private static final Schema MAP_KEY_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final String MAP_REPEATED_NAME = "key_value";
+ private static final String MAP_KEY_NAME = "key";
+ private static final String MAP_VALUE_NAME = "value";
private final Configuration hadoopConf;
private final Option<HoodieBloomFilterWriteSupport<UTF8String>> bloomFilterWriteSupportOpt;
+ private final byte[] decimalBuffer = new byte[Decimal.minBytesForPrecision()[DecimalType.MAX_PRECISION()]];
+ private final Enumeration.Value datetimeRebaseMode = LegacyBehaviorPolicy.withName(SQLConf.get().getConf(SQLConf.PARQUET_REBASE_MODE_IN_WRITE()));
+ private final Function1<Object, Object> dateRebaseFunction = DataSourceUtils.createDateRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private final Function1<Object, Object> timestampRebaseFunction = DataSourceUtils.createTimestampRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private RecordConsumer recordConsumer;
+ private final boolean writeLegacyListFormat;
+ private final ValueWriter[] rootFieldWriters;
+ private final Schema avroSchema;
- public HoodieRowParquetWriteSupport(Configuration conf, StructType structType, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
+ public HoodieRowParquetWriteSupport(Configuration conf, StructType schema, Schema avroSchema, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
Configuration hadoopConf = new Configuration(conf);
- hadoopConf.set("spark.sql.parquet.writeLegacyFormat", config.getStringOrDefault(HoodieStorageConfig.PARQUET_WRITE_LEGACY_FORMAT_ENABLED, "false"));
+ String writeLegacyFormatEnabled = config.getStringOrDefault(HoodieStorageConfig.PARQUET_WRITE_LEGACY_FORMAT_ENABLED, "false");
+ hadoopConf.set("spark.sql.parquet.writeLegacyFormat", writeLegacyFormatEnabled);
Review Comment:
@jonvex @linliu-code should the unnecessary parquet write config settings be removed from `SparkReaderContextFactory` to avoid confusion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2342735951
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,39 +18,90 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.LegacyBehaviorPolicy;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
* Hoodie Write Support for directly writing Row to Parquet.
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
+ private static final Schema MAP_KEY_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final String MAP_REPEATED_NAME = "key_value";
+ private static final String MAP_KEY_NAME = "key";
+ private static final String MAP_VALUE_NAME = "value";
private final Configuration hadoopConf;
private final Option<HoodieBloomFilterWriteSupport<UTF8String>> bloomFilterWriteSupportOpt;
+ private final byte[] decimalBuffer = new byte[Decimal.minBytesForPrecision()[DecimalType.MAX_PRECISION()]];
+ private final Enumeration.Value datetimeRebaseMode = LegacyBehaviorPolicy.withName(SQLConf.get().getConf(SQLConf.PARQUET_REBASE_MODE_IN_WRITE()));
+ private final Function1<Object, Object> dateRebaseFunction = DataSourceUtils.createDateRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private final Function1<Object, Object> timestampRebaseFunction = DataSourceUtils.createTimestampRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private RecordConsumer recordConsumer;
+ private final boolean writeLegacyListFormat;
+ private final ValueWriter[] rootFieldWriters;
+ private final Schema avroSchema;
- public HoodieRowParquetWriteSupport(Configuration conf, StructType structType, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
+ public HoodieRowParquetWriteSupport(Configuration conf, StructType schema, Schema avroSchema, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
Review Comment:
`HoodieConfig` is ambiguous, +1 to `HoodieWriteConfig`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2342736470
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,39 +18,90 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
+import org.apache.hudi.SparkAdapterSupport;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
* Hoodie Write Support for directly writing Row to Parquet.
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
Review Comment:
yeah, Flink did this from the very beginning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345444460
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkAdapterSupport.scala:
##########
@@ -43,4 +43,6 @@ object SparkAdapterSupport {
getClass.getClassLoader.loadClass(adapterClass)
.newInstance().asInstanceOf[SparkAdapter]
}
+
+ def getSparkAdapter: SparkAdapter = sparkAdapter
Review Comment:
In Java code, this is used to get the spark adapter `SparkAdapterSupport$.MODULE$.sparkAdapter()` in other classes
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,47 +18,149 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
+import org.apache.hudi.SparkAdapterSupport;
+import org.apache.hudi.avro.HoodieAvroUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
- * Hoodie Write Support for directly writing Row to Parquet.
+ * Hoodie Write Support for directly writing Row to Parquet and adding the Hudi bloom index to the file metadata.
+ * The implementation is adapted from Spark's {@link org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport} but has the following major differences:
+ * <ul>
+ * <li>Decimals are always written with the legacy format to ensure compatibility with parquet-avro and other readers</li>
+ * <li>Writing 2-level or 3-level lists is configurable with either the Hudi option, hoodie.parquet.writelegacyformat.enabled, or the parquet-avro option, parquet.avro.write-old-list-structure
+ * to ensure consistency across writer paths.</li>
+ * <li>The scale of the timestamps is determined by the Hudi writer schema instead of relying on Spark configuration</li>
+ * </ul>
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
+ private static final Schema MAP_KEY_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final String MAP_REPEATED_NAME = "key_value";
+ private static final String MAP_KEY_NAME = "key";
+ private static final String MAP_VALUE_NAME = "value";
private final Configuration hadoopConf;
private final Option<HoodieBloomFilterWriteSupport<UTF8String>> bloomFilterWriteSupportOpt;
+ private final byte[] decimalBuffer = new byte[Decimal.minBytesForPrecision()[DecimalType.MAX_PRECISION()]];
+ private final Enumeration.Value datetimeRebaseMode = (Enumeration.Value) SparkAdapterSupport.getSparkAdapter().getDateTimeRebaseMode();
Review Comment:
nit: use `SparkAdapterSupport$.MODULE$.sparkAdapter()` so there is no need to add a new API in `SparkAdapterSupport`, to be consistent with the usage in other Java classes
##########
hudi-spark-datasource/hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/adapter/Spark3_5Adapter.scala:
##########
@@ -38,7 +38,7 @@ import org.apache.spark.sql.execution.datasources.orc.Spark35OrcReader
import org.apache.spark.sql.execution.datasources.parquet.{ParquetFileFormat, Spark35LegacyHoodieParquetFileFormat, Spark35ParquetReader}
import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
import org.apache.spark.sql.hudi.analysis.TableValuedFunctions
-import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.internal.{LegacyBehaviorPolicy, SQLConf}
Review Comment:
So the class is moved in Spark thus the adapter pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3287176891
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
}, {
"hash" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491",
"triggerID" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"triggerType" : "PUSH"
}, {
"hash" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8503",
"triggerID" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"triggerType" : "PUSH"
}, {
"hash" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8506",
"triggerID" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"triggerType" : "PUSH"
}, {
"hash" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8507",
"triggerID" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"triggerType" : "PUSH"
}, {
"hash" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8510",
"triggerID" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"triggerType" : "PUSH"
}, {
"hash" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8511",
"triggerID" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 73dd1c38e7cc54233a051cb85222f56dd4cf22b3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8511)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345339675
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,39 +18,90 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.LegacyBehaviorPolicy;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
* Hoodie Write Support for directly writing Row to Parquet.
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
+ private static final Schema MAP_KEY_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final String MAP_REPEATED_NAME = "key_value";
+ private static final String MAP_KEY_NAME = "key";
+ private static final String MAP_VALUE_NAME = "value";
private final Configuration hadoopConf;
private final Option<HoodieBloomFilterWriteSupport<UTF8String>> bloomFilterWriteSupportOpt;
+ private final byte[] decimalBuffer = new byte[Decimal.minBytesForPrecision()[DecimalType.MAX_PRECISION()]];
+ private final Enumeration.Value datetimeRebaseMode = LegacyBehaviorPolicy.withName(SQLConf.get().getConf(SQLConf.PARQUET_REBASE_MODE_IN_WRITE()));
+ private final Function1<Object, Object> dateRebaseFunction = DataSourceUtils.createDateRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private final Function1<Object, Object> timestampRebaseFunction = DataSourceUtils.createTimestampRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private RecordConsumer recordConsumer;
+ private final boolean writeLegacyListFormat;
+ private final ValueWriter[] rootFieldWriters;
+ private final Schema avroSchema;
- public HoodieRowParquetWriteSupport(Configuration conf, StructType structType, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
+ public HoodieRowParquetWriteSupport(Configuration conf, StructType schema, Schema avroSchema, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
Review Comment:
@the-other-tim-brown I think the reason of using `HoodieConfig` here is that the file writer factory (`HoodieFileWriterFactory`) is in `hudi-common` module which does not have access to `HoodieWriteConfig`, though the config object should be write config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2342600831
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,39 +18,90 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.LegacyBehaviorPolicy;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
* Hoodie Write Support for directly writing Row to Parquet.
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
+ private static final Schema MAP_KEY_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final String MAP_REPEATED_NAME = "key_value";
+ private static final String MAP_KEY_NAME = "key";
+ private static final String MAP_VALUE_NAME = "value";
private final Configuration hadoopConf;
private final Option<HoodieBloomFilterWriteSupport<UTF8String>> bloomFilterWriteSupportOpt;
+ private final byte[] decimalBuffer = new byte[Decimal.minBytesForPrecision()[DecimalType.MAX_PRECISION()]];
+ private final Enumeration.Value datetimeRebaseMode = LegacyBehaviorPolicy.withName(SQLConf.get().getConf(SQLConf.PARQUET_REBASE_MODE_IN_WRITE()));
+ private final Function1<Object, Object> dateRebaseFunction = DataSourceUtils.createDateRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private final Function1<Object, Object> timestampRebaseFunction = DataSourceUtils.createTimestampRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private RecordConsumer recordConsumer;
+ private final boolean writeLegacyListFormat;
+ private final ValueWriter[] rootFieldWriters;
+ private final Schema avroSchema;
- public HoodieRowParquetWriteSupport(Configuration conf, StructType structType, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
+ public HoodieRowParquetWriteSupport(Configuration conf, StructType schema, Schema avroSchema, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
Configuration hadoopConf = new Configuration(conf);
- hadoopConf.set("spark.sql.parquet.writeLegacyFormat", config.getStringOrDefault(HoodieStorageConfig.PARQUET_WRITE_LEGACY_FORMAT_ENABLED, "false"));
+ String writeLegacyFormatEnabled = config.getStringOrDefault(HoodieStorageConfig.PARQUET_WRITE_LEGACY_FORMAT_ENABLED, "false");
+ hadoopConf.set("spark.sql.parquet.writeLegacyFormat", writeLegacyFormatEnabled);
Review Comment:
They should not impact the reader. The reader uses the parquet schema to figure out the scale of the timestamps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3286070374
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
}, {
"hash" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491",
"triggerID" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"triggerType" : "PUSH"
}, {
"hash" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8503",
"triggerID" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"triggerType" : "PUSH"
}, {
"hash" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8506",
"triggerID" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"triggerType" : "PUSH"
}, {
"hash" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1406c146e1fa7c3e449bfe9e99fbceaafb1ad209 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8506)
* d71d18af39bc98ed0eb85f0565967aef8e3fe305 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345681039
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -73,6 +175,173 @@ public void add(UTF8String recordKey) {
bloomFilterWriteSupport.addKey(recordKey));
}
+ @FunctionalInterface
+ private interface ValueWriter {
+ void write(SpecializedGetters row, int ordinal);
+ }
+
+ private void consumeMessage(Runnable writer) {
+ recordConsumer.startMessage();
+ writer.run();
+ recordConsumer.endMessage();
+ }
+
+ private void consumeGroup(Runnable writer) {
+ recordConsumer.startGroup();
+ writer.run();
+ recordConsumer.endGroup();
+ }
+
+ private void consumeField(String field, int index, Runnable writer) {
+ recordConsumer.startField(field, index);
+ writer.run();
+ recordConsumer.endField(field, index);
+ }
+
+ private void writeFields(InternalRow row, Schema schema, ValueWriter[] fieldWriters) {
+ for (int i = 0; i < fieldWriters.length; i++) {
+ int index = i;
+ if (!row.isNullAt(i)) {
+ Schema.Field field = schema.getFields().get(index);
+ consumeField(field.name(), index, () -> fieldWriters[index].write(row, index));
+ }
+ }
+ }
+
+ private ValueWriter makeWriter(Schema avroSchema, DataType dataType) {
+ Schema resolvedSchema = resolveNullableSchema(avroSchema);
+ Schema.Type type = resolvedSchema.getType();
+ LogicalType logicalType = resolvedSchema.getLogicalType();
+ switch (type) {
+ case BOOLEAN:
+ return (row, ordinal) -> recordConsumer.addBoolean(row.getBoolean(ordinal));
+ case INT:
Review Comment:
Got it. Then we don't have to add it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345741708
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -73,6 +178,173 @@ public void add(UTF8String recordKey) {
bloomFilterWriteSupport.addKey(recordKey));
}
+ @FunctionalInterface
+ private interface ValueWriter {
+ void write(SpecializedGetters row, int ordinal);
+ }
+
+ private void consumeMessage(Runnable writer) {
+ recordConsumer.startMessage();
+ writer.run();
+ recordConsumer.endMessage();
+ }
+
+ private void consumeGroup(Runnable writer) {
+ recordConsumer.startGroup();
+ writer.run();
+ recordConsumer.endGroup();
+ }
+
+ private void consumeField(String field, int index, Runnable writer) {
+ recordConsumer.startField(field, index);
+ writer.run();
+ recordConsumer.endField(field, index);
+ }
+
+ private void writeFields(InternalRow row, Schema schema, ValueWriter[] fieldWriters) {
+ for (int i = 0; i < fieldWriters.length; i++) {
+ int index = i;
+ if (!row.isNullAt(i)) {
+ Schema.Field field = schema.getFields().get(index);
+ consumeField(field.name(), index, () -> fieldWriters[index].write(row, index));
+ }
+ }
+ }
+
+ private ValueWriter makeWriter(Schema avroSchema, DataType dataType) {
+ Schema resolvedSchema = resolveNullableSchema(avroSchema);
+ Schema.Type type = resolvedSchema.getType();
+ LogicalType logicalType = resolvedSchema.getLogicalType();
+ switch (type) {
+ case BOOLEAN:
+ return (row, ordinal) -> recordConsumer.addBoolean(row.getBoolean(ordinal));
+ case INT:
+ if (logicalType != null) {
+ if (logicalType.getName().equals(LogicalTypes.date().getName())) {
+ return (row, ordinal) -> recordConsumer.addInteger((Integer) dateRebaseFunction.apply(row.getInt(ordinal)));
+ }
+ }
+ return (row, ordinal) -> recordConsumer.addInteger(row.getInt(ordinal));
+ case LONG:
+ if (logicalType != null) {
+ if (logicalType.getName().equals(LogicalTypes.timestampMillis().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong(DateTimeUtils.microsToMillis((long) timestampRebaseFunction.apply(row.getLong(ordinal))));
+ } else if (logicalType.getName().equals(LogicalTypes.timestampMicros().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong((long) timestampRebaseFunction.apply(row.getLong(ordinal)));
+ } else if (logicalType.getName().equals(LogicalTypes.localTimestampMicros().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong(row.getLong(ordinal));
+ } else if (logicalType.getName().equals(LogicalTypes.localTimestampMillis().getName()) && dataType instanceof TimestampNTZType) {
+ // only go through conversion if spark data type is TimestampNTZType
+ return (row, ordinal) -> recordConsumer.addLong(DateTimeUtils.microsToMillis(row.getLong(ordinal)));
+ }
+ }
+ return (row, ordinal) -> recordConsumer.addLong(row.getLong(ordinal));
+ case FLOAT:
+ return (row, ordinal) -> recordConsumer.addFloat(row.getFloat(ordinal));
+ case DOUBLE:
+ return (row, ordinal) -> recordConsumer.addDouble(row.getDouble(ordinal));
+ case STRING:
+ case ENUM:
+ return (row, ordinal) -> recordConsumer.addBinary(
+ Binary.fromReusedByteArray(row.getUTF8String(ordinal).getBytes()));
+ case BYTES:
+ case FIXED:
+ if (logicalType != null && logicalType.getName().equals("decimal")) {
+ return (row, ordinal) -> {
+ int precision = ((LogicalTypes.Decimal) logicalType).getPrecision();
+ int scale = ((LogicalTypes.Decimal) logicalType).getScale();
+ long unscaled = row.getDecimal(ordinal, precision, scale).toUnscaledLong();
+ int i = 0;
+ int numBytes = Decimal.minBytesForPrecision()[precision];
+ int shift = 8 * (numBytes - 1);
+ while (i < numBytes) {
+ decimalBuffer[i] = (byte) ((unscaled >> shift) & 0xFF);
+ i += 1;
+ shift -= 8;
+ }
+ recordConsumer.addBinary(Binary.fromReusedByteArray(decimalBuffer, 0, numBytes));
+ };
+ }
+ return (row, ordinal) -> recordConsumer.addBinary(
+ Binary.fromReusedByteArray(row.getBinary(ordinal)));
+ case RECORD:
+ ValueWriter[] fieldWriters = getFieldWriters(((StructType) dataType), resolvedSchema);
+ return (row, ordinal) ->
+ consumeGroup(() -> writeFields(row.getStruct(ordinal, resolvedSchema.getFields().size()), resolvedSchema, fieldWriters));
+ case ARRAY:
+ ValueWriter elementWriter = makeWriter(resolvedSchema.getElementType(), ((ArrayType) dataType).elementType());
+ if (!writeLegacyListFormat) {
+ return threeLevelArrayWriter("list", "element", elementWriter);
+ } else if (resolvedSchema.getElementType().isNullable()) {
+ return threeLevelArrayWriter("bag", "array", elementWriter);
+ } else {
+ return twoLevelArrayWriter("array", elementWriter);
+ }
+ case MAP:
+ ValueWriter keyWriter = makeWriter(MAP_KEY_SCHEMA, DataTypes.StringType);
+ ValueWriter valueWriter = makeWriter(resolvedSchema.getValueType(), ((MapType) dataType).valueType());
+ return (row, ordinal) -> {
+ MapData mapData = row.getMap(ordinal);
+ ArrayData keyArray = mapData.keyArray();
+ ArrayData valueArray = mapData.valueArray();
+ consumeGroup(() -> {
+ if (mapData.numElements() > 0) {
+ consumeField(MAP_REPEATED_NAME, 0, () -> {
+ for (int i = 0; i < mapData.numElements(); i++) {
+ int index = i;
Review Comment:
True, will clean this up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3288745781
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
}, {
"hash" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491",
"triggerID" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"triggerType" : "PUSH"
}, {
"hash" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8503",
"triggerID" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"triggerType" : "PUSH"
}, {
"hash" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8506",
"triggerID" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"triggerType" : "PUSH"
}, {
"hash" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8507",
"triggerID" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"triggerType" : "PUSH"
}, {
"hash" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8510",
"triggerID" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"triggerType" : "PUSH"
}, {
"hash" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8511",
"triggerID" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"triggerType" : "PUSH"
}, {
"hash" : "3ff5acf174c775406ffb54be92650b03718bf48b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8513",
"triggerID" : "3ff5acf174c775406ffb54be92650b03718bf48b",
"triggerType" : "PUSH"
}, {
"hash" : "25a0be4048eb5d9e8c082ad7ff1026053a20747c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "25a0be4048eb5d9e8c082ad7ff1026053a20747c",
"triggerType" : "PUSH"
}, {
"hash" : "27b7921b67ae40d374478457fa51b7ffc7336f50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27b7921b67ae40d374478457fa51b7ffc7336f50",
"triggerType" : "PUSH"
}, {
"hash" : "55213c73a0d42e87f0132b15eb793ef33e3561d3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8514",
"triggerID" : "55213c73a0d42e87f0132b15eb793ef33e3561d3",
"triggerType" : "PUSH"
}, {
"hash" : "a822d5e52a76ec069ed389b279744219ede6ad93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a822d5e52a76ec069ed389b279744219ede6ad93",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25a0be4048eb5d9e8c082ad7ff1026053a20747c UNKNOWN
* 27b7921b67ae40d374478457fa51b7ffc7336f50 UNKNOWN
* 55213c73a0d42e87f0132b15eb793ef33e3561d3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8514)
* a822d5e52a76ec069ed389b279744219ede6ad93 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3282494551
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e5ea200c5cea9f5c2dda1554b765bfe11e276ae7 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474)
* 190cbc827b787e7f347a76add30a94f7621e750d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2342302277
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieInternalRowFileWriterFactory.java:
##########
@@ -54,24 +55,26 @@ public class HoodieInternalRowFileWriterFactory {
public static HoodieInternalRowFileWriter getInternalRowFileWriter(StoragePath path,
HoodieTable hoodieTable,
HoodieWriteConfig writeConfig,
- StructType schema)
+ StructType schema,
+ Schema avroSchema)
Review Comment:
Is it possible to only pass in `Schema avroSchema` and derive `StructType schema` here internally, or that will incur another large refactoring? Conceptually `StructType schema` is derived somewhere else from the Avro write or table schema.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/MultipleSparkJobExecutionStrategy.java:
##########
@@ -109,11 +108,7 @@ public HoodieWriteMetadata<HoodieData<WriteStatus>> performClustering(final Hood
Math.min(clusteringPlan.getInputGroups().size(), writeConfig.getClusteringMaxParallelism()),
new CustomizedThreadFactory("clustering-job-group", true));
try {
- boolean canUseRowWriter = getWriteConfig().getBooleanOrDefault("hoodie.datasource.write.row.writer.enable", true)
- && HoodieDataTypeUtils.canUseRowWriter(schema, engineContext.hadoopConfiguration());
- if (canUseRowWriter) {
- HoodieDataTypeUtils.tryOverrideParquetWriteLegacyFormatProperty(writeConfig.getProps(), schema);
- }
Review Comment:
So are all such tweaks of parquet writer behavior through write configs controlled in a central place in the write support class (`HoodieRowParquetWriteSupport`) so it's easier to maintain?
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,39 +18,90 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.LegacyBehaviorPolicy;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
* Hoodie Write Support for directly writing Row to Parquet.
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
+ private static final Schema MAP_KEY_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final String MAP_REPEATED_NAME = "key_value";
+ private static final String MAP_KEY_NAME = "key";
+ private static final String MAP_VALUE_NAME = "value";
private final Configuration hadoopConf;
private final Option<HoodieBloomFilterWriteSupport<UTF8String>> bloomFilterWriteSupportOpt;
+ private final byte[] decimalBuffer = new byte[Decimal.minBytesForPrecision()[DecimalType.MAX_PRECISION()]];
+ private final Enumeration.Value datetimeRebaseMode = LegacyBehaviorPolicy.withName(SQLConf.get().getConf(SQLConf.PARQUET_REBASE_MODE_IN_WRITE()));
+ private final Function1<Object, Object> dateRebaseFunction = DataSourceUtils.createDateRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private final Function1<Object, Object> timestampRebaseFunction = DataSourceUtils.createTimestampRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private RecordConsumer recordConsumer;
+ private final boolean writeLegacyListFormat;
+ private final ValueWriter[] rootFieldWriters;
+ private final Schema avroSchema;
- public HoodieRowParquetWriteSupport(Configuration conf, StructType structType, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
+ public HoodieRowParquetWriteSupport(Configuration conf, StructType schema, Schema avroSchema, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
Review Comment:
Could the `Schema avroSchema` be derived within the constructor give the write config `config` instance is available (like `BulkInsertDataInternalWriterHelper`) so there is no need to pass in the `avroSchema` from different place and the logic is consistent?
```
this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(writeConfig.getWriteSchema()), writeConfig.allowOperationMetadataField());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2342396113
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,39 +18,90 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.LegacyBehaviorPolicy;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
* Hoodie Write Support for directly writing Row to Parquet.
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
+ private static final Schema MAP_KEY_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final String MAP_REPEATED_NAME = "key_value";
+ private static final String MAP_KEY_NAME = "key";
+ private static final String MAP_VALUE_NAME = "value";
private final Configuration hadoopConf;
private final Option<HoodieBloomFilterWriteSupport<UTF8String>> bloomFilterWriteSupportOpt;
+ private final byte[] decimalBuffer = new byte[Decimal.minBytesForPrecision()[DecimalType.MAX_PRECISION()]];
+ private final Enumeration.Value datetimeRebaseMode = LegacyBehaviorPolicy.withName(SQLConf.get().getConf(SQLConf.PARQUET_REBASE_MODE_IN_WRITE()));
+ private final Function1<Object, Object> dateRebaseFunction = DataSourceUtils.createDateRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private final Function1<Object, Object> timestampRebaseFunction = DataSourceUtils.createTimestampRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private RecordConsumer recordConsumer;
+ private final boolean writeLegacyListFormat;
+ private final ValueWriter[] rootFieldWriters;
+ private final Schema avroSchema;
- public HoodieRowParquetWriteSupport(Configuration conf, StructType structType, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
+ public HoodieRowParquetWriteSupport(Configuration conf, StructType schema, Schema avroSchema, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
Configuration hadoopConf = new Configuration(conf);
- hadoopConf.set("spark.sql.parquet.writeLegacyFormat", config.getStringOrDefault(HoodieStorageConfig.PARQUET_WRITE_LEGACY_FORMAT_ENABLED, "false"));
+ String writeLegacyFormatEnabled = config.getStringOrDefault(HoodieStorageConfig.PARQUET_WRITE_LEGACY_FORMAT_ENABLED, "false");
+ hadoopConf.set("spark.sql.parquet.writeLegacyFormat", writeLegacyFormatEnabled);
Review Comment:
I see that `SparkReaderContextFactory` sets the `"spark.sql.parquet.outputTimestampType"` and `"spark.sql.parquet.writeLegacyFormat"` based on the sqlConf, not based on the `HoodieStorageConfig.PARQUET_WRITE_LEGACY_FORMAT_ENABLED`. Should that be fixed? Also do these configs control Spark parquet reader behavior (I'm wondering why these Spark parquet writer configs are need in the reader context)?
```
// Broadcast: SQLConf.
// Broadcast: Configuration.
Configuration configs = getHadoopConfiguration(jsc.hadoopConfiguration());
schemaEvolutionConfigs.forEach(configs::set);
configs.set(SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE().key(), sqlConf.getConfString(SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE().key()));
configs.set(SQLConf.PARQUET_WRITE_LEGACY_FORMAT().key(), sqlConf.getConfString(SQLConf.PARQUET_WRITE_LEGACY_FORMAT().key()));
configurationBroadcast = jsc.broadcast(new SerializableConfiguration(configs));
// Broadcast: BaseFilereader.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3283072087
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 190cbc827b787e7f347a76add30a94f7621e750d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477)
* 5bca64047318b420b21f7dadd1f274d1330319e1 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345379467
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,39 +18,90 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.LegacyBehaviorPolicy;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
* Hoodie Write Support for directly writing Row to Parquet.
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
+ private static final Schema MAP_KEY_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final String MAP_REPEATED_NAME = "key_value";
+ private static final String MAP_KEY_NAME = "key";
+ private static final String MAP_VALUE_NAME = "value";
private final Configuration hadoopConf;
private final Option<HoodieBloomFilterWriteSupport<UTF8String>> bloomFilterWriteSupportOpt;
+ private final byte[] decimalBuffer = new byte[Decimal.minBytesForPrecision()[DecimalType.MAX_PRECISION()]];
+ private final Enumeration.Value datetimeRebaseMode = LegacyBehaviorPolicy.withName(SQLConf.get().getConf(SQLConf.PARQUET_REBASE_MODE_IN_WRITE()));
+ private final Function1<Object, Object> dateRebaseFunction = DataSourceUtils.createDateRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private final Function1<Object, Object> timestampRebaseFunction = DataSourceUtils.createTimestampRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private RecordConsumer recordConsumer;
+ private final boolean writeLegacyListFormat;
+ private final ValueWriter[] rootFieldWriters;
+ private final Schema avroSchema;
- public HoodieRowParquetWriteSupport(Configuration conf, StructType structType, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
+ public HoodieRowParquetWriteSupport(Configuration conf, StructType schema, Schema avroSchema, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
Review Comment:
Here are the details of how writer schema is determined on two core write flows:
(1) Spark datasource/SQL writer: the source schema is derived from the Dataframe's StructType and converted to Avro schema (`val sourceSchema = convertStructTypeToAvroSchema(df.schema, avroRecordName, avroRecordNamespace)` in `HoodieSparkSqlWriter#writeInternal`). Then in the same method, the writer schema is derived from the source schema and table schema, and optionally the internal schema for schema on read.
```
// NOTE: Target writer's schema is deduced based on
// - Source's schema
// - Existing table's schema (including its Hudi's [[InternalSchema]] representation)
val writerSchema = HoodieSchemaUtils.deduceWriterSchema(sourceSchema, latestTableSchemaOpt, internalSchemaOpt, parameters)
```
Subsequently, this writer schema is adjusted based on the partition column and meta field handing; and eventually used to set `hoodie.avro.schema` in the write config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345493650
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,47 +18,149 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
+import org.apache.hudi.SparkAdapterSupport;
+import org.apache.hudi.avro.HoodieAvroUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
- * Hoodie Write Support for directly writing Row to Parquet.
+ * Hoodie Write Support for directly writing Row to Parquet and adding the Hudi bloom index to the file metadata.
+ * The implementation is adapted from Spark's {@link org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport} but has the following major differences:
+ * <ul>
+ * <li>Decimals are always written with the legacy format to ensure compatibility with parquet-avro and other readers</li>
+ * <li>Writing 2-level or 3-level lists is configurable with either the Hudi option, hoodie.parquet.writelegacyformat.enabled, or the parquet-avro option, parquet.avro.write-old-list-structure
+ * to ensure consistency across writer paths.</li>
+ * <li>The scale of the timestamps is determined by the Hudi writer schema instead of relying on Spark configuration</li>
+ * </ul>
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
Review Comment:
Do we need to adjust and customize the configurations of the `ParquetReadSupport` when reading the parquet files on Spark?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345512578
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/functional/TestSparkSortAndSizeClustering.java:
##########
@@ -130,24 +155,55 @@ public void writeAndClustering(boolean isRow) throws IOException {
List<Row> rows = readRecords();
assertEquals(numRecords, rows.size());
- validateDecimalTypeAfterClustering(writeStats);
+ validateTypesAfterClustering(writeStats);
+ validateDateAndTimestampFields(rows, ts);
}
- // Validate that clustering produces decimals in legacy format
- private void validateDecimalTypeAfterClustering(List<HoodieWriteStat> writeStats) {
+ private void validateDateAndTimestampFields(List<Row> rows, long ts) {
+ Schema schema = HoodieAvroUtils.addMetadataFields(getSchema(), false);
+ Timestamp timestamp = new Timestamp(ts);
+ int dateFieldIndex = schema.getField("date_nullable_field").pos();
+ int tsMillisFieldIndex = schema.getField("timestamp_millis_field").pos();
+ int tsMicrosNullableFieldIndex = schema.getField("timestamp_micros_nullable_field").pos();
+ int tsLocalMillisFieldIndex = schema.getField("timestamp_local_millis_nullable_field").pos();
+ int tsLocalMicrosFieldIndex = schema.getField("timestamp_local_micros_field").pos();
+ for (Row row : rows) {
+ assertEquals(timestamp, row.get(tsMillisFieldIndex));
+ if (!row.isNullAt(tsMicrosNullableFieldIndex)) {
+ assertEquals(timestamp, row.get(tsMicrosNullableFieldIndex));
+ }
+ if (!row.isNullAt(dateFieldIndex)) {
+ assertTrue(row.get(dateFieldIndex) instanceof Date);
+ }
+ if (!row.isNullAt(tsLocalMillisFieldIndex)) {
+ assertEquals(ts, row.get(tsLocalMillisFieldIndex));
+ }
+ assertEquals(ts * 1000L, row.get(tsLocalMicrosFieldIndex));
+ }
+ }
+
+ // Validate that clustering produces decimals in legacy format and lists in newer format. Assert that the unit is correct on the timestamps
+ private void validateTypesAfterClustering(List<HoodieWriteStat> writeStats) {
Review Comment:
There is a test that does not use row writer which covers the avro path but it doesn't hurt to validate in both spots
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2346902291
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -73,6 +189,180 @@ public void add(UTF8String recordKey) {
bloomFilterWriteSupport.addKey(recordKey));
}
+ @FunctionalInterface
+ private interface ValueWriter {
+ void write(SpecializedGetters row, int ordinal);
+ }
+
+ private void consumeMessage(Runnable writer) {
+ recordConsumer.startMessage();
+ writer.run();
+ recordConsumer.endMessage();
+ }
+
+ private void consumeGroup(Runnable writer) {
+ recordConsumer.startGroup();
+ writer.run();
+ recordConsumer.endGroup();
+ }
+
+ private void consumeField(String field, int index, Runnable writer) {
+ recordConsumer.startField(field, index);
+ writer.run();
+ recordConsumer.endField(field, index);
+ }
+
+ private void writeFields(InternalRow row, StructType schema, ValueWriter[] fieldWriters) {
+ for (int i = 0; i < row.numFields(); i++) {
+ int index = i;
+ if (!row.isNullAt(i)) {
+ StructField field = schema.fields()[i];
+ consumeField(field.name(), index, () -> fieldWriters[index].write(row, index));
+ }
+ }
+ }
+
+ private ValueWriter makeWriter(Schema avroSchema, DataType dataType) {
+ Schema resolvedSchema = avroSchema == null ? null : resolveNullableSchema(avroSchema);
+ LogicalType logicalType = resolvedSchema != null ? resolvedSchema.getLogicalType() : null;
+
+ if (dataType == DataTypes.BooleanType) {
+ return (row, ordinal) -> recordConsumer.addBoolean(row.getBoolean(ordinal));
+ } else if (dataType == DataTypes.DateType) {
+ return (row, ordinal) -> recordConsumer.addInteger((Integer) dateRebaseFunction.apply(row.getInt(ordinal)));
+ } else if (dataType == DataTypes.ShortType) {
+ return (row, ordinal) -> recordConsumer.addInteger(row.getShort(ordinal));
+ } else if (dataType == DataTypes.IntegerType) {
+ return (row, ordinal) -> recordConsumer.addInteger(row.getInt(ordinal));
+ } else if (dataType == DataTypes.LongType) {
+ return (row, ordinal) -> recordConsumer.addLong(row.getLong(ordinal));
+ } else if (dataType == DataTypes.TimestampType) {
+ if (logicalType == null || logicalType.getName().equals(LogicalTypes.timestampMicros().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong((long) timestampRebaseFunction.apply(row.getLong(ordinal)));
+ } else if (logicalType.getName().equals(LogicalTypes.timestampMillis().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong(DateTimeUtils.microsToMillis((long) timestampRebaseFunction.apply(row.getLong(ordinal))));
+ } else {
+ throw new UnsupportedOperationException("Unsupported timestamp type: " + logicalType);
Review Comment:
Updated
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -73,6 +189,180 @@ public void add(UTF8String recordKey) {
bloomFilterWriteSupport.addKey(recordKey));
}
+ @FunctionalInterface
+ private interface ValueWriter {
+ void write(SpecializedGetters row, int ordinal);
+ }
+
+ private void consumeMessage(Runnable writer) {
+ recordConsumer.startMessage();
+ writer.run();
+ recordConsumer.endMessage();
+ }
+
+ private void consumeGroup(Runnable writer) {
+ recordConsumer.startGroup();
+ writer.run();
+ recordConsumer.endGroup();
+ }
+
+ private void consumeField(String field, int index, Runnable writer) {
+ recordConsumer.startField(field, index);
+ writer.run();
+ recordConsumer.endField(field, index);
+ }
+
+ private void writeFields(InternalRow row, StructType schema, ValueWriter[] fieldWriters) {
+ for (int i = 0; i < row.numFields(); i++) {
+ int index = i;
+ if (!row.isNullAt(i)) {
+ StructField field = schema.fields()[i];
+ consumeField(field.name(), index, () -> fieldWriters[index].write(row, index));
+ }
+ }
+ }
+
+ private ValueWriter makeWriter(Schema avroSchema, DataType dataType) {
+ Schema resolvedSchema = avroSchema == null ? null : resolveNullableSchema(avroSchema);
+ LogicalType logicalType = resolvedSchema != null ? resolvedSchema.getLogicalType() : null;
+
+ if (dataType == DataTypes.BooleanType) {
+ return (row, ordinal) -> recordConsumer.addBoolean(row.getBoolean(ordinal));
+ } else if (dataType == DataTypes.DateType) {
+ return (row, ordinal) -> recordConsumer.addInteger((Integer) dateRebaseFunction.apply(row.getInt(ordinal)));
+ } else if (dataType == DataTypes.ShortType) {
+ return (row, ordinal) -> recordConsumer.addInteger(row.getShort(ordinal));
+ } else if (dataType == DataTypes.IntegerType) {
+ return (row, ordinal) -> recordConsumer.addInteger(row.getInt(ordinal));
+ } else if (dataType == DataTypes.LongType) {
+ return (row, ordinal) -> recordConsumer.addLong(row.getLong(ordinal));
+ } else if (dataType == DataTypes.TimestampType) {
+ if (logicalType == null || logicalType.getName().equals(LogicalTypes.timestampMicros().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong((long) timestampRebaseFunction.apply(row.getLong(ordinal)));
+ } else if (logicalType.getName().equals(LogicalTypes.timestampMillis().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong(DateTimeUtils.microsToMillis((long) timestampRebaseFunction.apply(row.getLong(ordinal))));
+ } else {
+ throw new UnsupportedOperationException("Unsupported timestamp type: " + logicalType);
+ }
+ } else if (SparkAdapterSupport$.MODULE$.sparkAdapter().isTimestampNTZType(dataType)) {
+ if (logicalType == null || logicalType.getName().equals(LogicalTypes.localTimestampMicros().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong(row.getLong(ordinal));
+ } else if (logicalType.getName().equals(LogicalTypes.localTimestampMillis().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong(DateTimeUtils.microsToMillis(row.getLong(ordinal)));
+ } else {
+ throw new UnsupportedOperationException("Unsupported timestamp type: " + logicalType);
Review Comment:
Updated
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -73,6 +189,180 @@ public void add(UTF8String recordKey) {
bloomFilterWriteSupport.addKey(recordKey));
}
+ @FunctionalInterface
+ private interface ValueWriter {
+ void write(SpecializedGetters row, int ordinal);
+ }
+
+ private void consumeMessage(Runnable writer) {
+ recordConsumer.startMessage();
+ writer.run();
+ recordConsumer.endMessage();
+ }
+
+ private void consumeGroup(Runnable writer) {
+ recordConsumer.startGroup();
+ writer.run();
+ recordConsumer.endGroup();
+ }
+
+ private void consumeField(String field, int index, Runnable writer) {
+ recordConsumer.startField(field, index);
+ writer.run();
+ recordConsumer.endField(field, index);
+ }
+
+ private void writeFields(InternalRow row, StructType schema, ValueWriter[] fieldWriters) {
+ for (int i = 0; i < row.numFields(); i++) {
+ int index = i;
+ if (!row.isNullAt(i)) {
+ StructField field = schema.fields()[i];
+ consumeField(field.name(), index, () -> fieldWriters[index].write(row, index));
+ }
+ }
+ }
+
+ private ValueWriter makeWriter(Schema avroSchema, DataType dataType) {
+ Schema resolvedSchema = avroSchema == null ? null : resolveNullableSchema(avroSchema);
+ LogicalType logicalType = resolvedSchema != null ? resolvedSchema.getLogicalType() : null;
+
+ if (dataType == DataTypes.BooleanType) {
+ return (row, ordinal) -> recordConsumer.addBoolean(row.getBoolean(ordinal));
+ } else if (dataType == DataTypes.DateType) {
+ return (row, ordinal) -> recordConsumer.addInteger((Integer) dateRebaseFunction.apply(row.getInt(ordinal)));
+ } else if (dataType == DataTypes.ShortType) {
+ return (row, ordinal) -> recordConsumer.addInteger(row.getShort(ordinal));
+ } else if (dataType == DataTypes.IntegerType) {
+ return (row, ordinal) -> recordConsumer.addInteger(row.getInt(ordinal));
+ } else if (dataType == DataTypes.LongType) {
+ return (row, ordinal) -> recordConsumer.addLong(row.getLong(ordinal));
+ } else if (dataType == DataTypes.TimestampType) {
+ if (logicalType == null || logicalType.getName().equals(LogicalTypes.timestampMicros().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong((long) timestampRebaseFunction.apply(row.getLong(ordinal)));
+ } else if (logicalType.getName().equals(LogicalTypes.timestampMillis().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong(DateTimeUtils.microsToMillis((long) timestampRebaseFunction.apply(row.getLong(ordinal))));
+ } else {
+ throw new UnsupportedOperationException("Unsupported timestamp type: " + logicalType);
+ }
+ } else if (SparkAdapterSupport$.MODULE$.sparkAdapter().isTimestampNTZType(dataType)) {
+ if (logicalType == null || logicalType.getName().equals(LogicalTypes.localTimestampMicros().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong(row.getLong(ordinal));
+ } else if (logicalType.getName().equals(LogicalTypes.localTimestampMillis().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong(DateTimeUtils.microsToMillis(row.getLong(ordinal)));
+ } else {
+ throw new UnsupportedOperationException("Unsupported timestamp type: " + logicalType);
+ }
+ } else if (dataType == DataTypes.FloatType) {
+ return (row, ordinal) -> recordConsumer.addFloat(row.getFloat(ordinal));
+ } else if (dataType == DataTypes.DoubleType) {
+ return (row, ordinal) -> recordConsumer.addDouble(row.getDouble(ordinal));
+ } else if (dataType == DataTypes.StringType) {
+ return (row, ordinal) -> recordConsumer.addBinary(
+ Binary.fromReusedByteArray(row.getUTF8String(ordinal).getBytes()));
+ } else if (dataType == DataTypes.BinaryType) {
+ return (row, ordinal) -> recordConsumer.addBinary(
+ Binary.fromReusedByteArray(row.getBinary(ordinal)));
+ } else if (dataType instanceof DecimalType) {
+ return (row, ordinal) -> {
+ int precision = ((DecimalType) dataType).precision();
+ int scale = ((DecimalType) dataType).scale();
+ byte[] bytes = row.getDecimal(ordinal, precision, scale).toJavaBigDecimal().unscaledValue().toByteArray();
Review Comment:
Added a check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3288791264
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
}, {
"hash" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491",
"triggerID" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"triggerType" : "PUSH"
}, {
"hash" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8503",
"triggerID" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"triggerType" : "PUSH"
}, {
"hash" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8506",
"triggerID" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"triggerType" : "PUSH"
}, {
"hash" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8507",
"triggerID" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"triggerType" : "PUSH"
}, {
"hash" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8510",
"triggerID" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"triggerType" : "PUSH"
}, {
"hash" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8511",
"triggerID" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"triggerType" : "PUSH"
}, {
"hash" : "3ff5acf174c775406ffb54be92650b03718bf48b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8513",
"triggerID" : "3ff5acf174c775406ffb54be92650b03718bf48b",
"triggerType" : "PUSH"
}, {
"hash" : "25a0be4048eb5d9e8c082ad7ff1026053a20747c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "25a0be4048eb5d9e8c082ad7ff1026053a20747c",
"triggerType" : "PUSH"
}, {
"hash" : "27b7921b67ae40d374478457fa51b7ffc7336f50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27b7921b67ae40d374478457fa51b7ffc7336f50",
"triggerType" : "PUSH"
}, {
"hash" : "55213c73a0d42e87f0132b15eb793ef33e3561d3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8514",
"triggerID" : "55213c73a0d42e87f0132b15eb793ef33e3561d3",
"triggerType" : "PUSH"
}, {
"hash" : "a822d5e52a76ec069ed389b279744219ede6ad93",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8522",
"triggerID" : "a822d5e52a76ec069ed389b279744219ede6ad93",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25a0be4048eb5d9e8c082ad7ff1026053a20747c UNKNOWN
* 27b7921b67ae40d374478457fa51b7ffc7336f50 UNKNOWN
* 55213c73a0d42e87f0132b15eb793ef33e3561d3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8514)
* a822d5e52a76ec069ed389b279744219ede6ad93 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8522)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3288858630
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
}, {
"hash" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491",
"triggerID" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"triggerType" : "PUSH"
}, {
"hash" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8503",
"triggerID" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"triggerType" : "PUSH"
}, {
"hash" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8506",
"triggerID" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"triggerType" : "PUSH"
}, {
"hash" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8507",
"triggerID" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"triggerType" : "PUSH"
}, {
"hash" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8510",
"triggerID" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"triggerType" : "PUSH"
}, {
"hash" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8511",
"triggerID" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"triggerType" : "PUSH"
}, {
"hash" : "3ff5acf174c775406ffb54be92650b03718bf48b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8513",
"triggerID" : "3ff5acf174c775406ffb54be92650b03718bf48b",
"triggerType" : "PUSH"
}, {
"hash" : "25a0be4048eb5d9e8c082ad7ff1026053a20747c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "25a0be4048eb5d9e8c082ad7ff1026053a20747c",
"triggerType" : "PUSH"
}, {
"hash" : "27b7921b67ae40d374478457fa51b7ffc7336f50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27b7921b67ae40d374478457fa51b7ffc7336f50",
"triggerType" : "PUSH"
}, {
"hash" : "55213c73a0d42e87f0132b15eb793ef33e3561d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8514",
"triggerID" : "55213c73a0d42e87f0132b15eb793ef33e3561d3",
"triggerType" : "PUSH"
}, {
"hash" : "a822d5e52a76ec069ed389b279744219ede6ad93",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8522",
"triggerID" : "a822d5e52a76ec069ed389b279744219ede6ad93",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25a0be4048eb5d9e8c082ad7ff1026053a20747c UNKNOWN
* 27b7921b67ae40d374478457fa51b7ffc7336f50 UNKNOWN
* a822d5e52a76ec069ed389b279744219ede6ad93 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8522)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2344112223
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -73,6 +148,177 @@ public void add(UTF8String recordKey) {
bloomFilterWriteSupport.addKey(recordKey));
}
+ @FunctionalInterface
+ private interface ValueWriter {
+ void write(SpecializedGetters row, int ordinal);
+ }
+
+ private void consumeMessage(Runnable writer) {
+ recordConsumer.startMessage();
Review Comment:
did we borrow some impls from the Spark native parquet writers, they should have some efficient codes there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345017343
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,47 +18,149 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
+import org.apache.hudi.SparkAdapterSupport;
+import org.apache.hudi.avro.HoodieAvroUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
- * Hoodie Write Support for directly writing Row to Parquet.
+ * Hoodie Write Support for directly writing Row to Parquet and adding the Hudi bloom index to the file metadata.
+ * The implementation is adapted from Spark's {@link org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport} but has the following major differences:
+ * <ul>
+ * <li>Decimals are always written with the legacy format to ensure compatibility with parquet-avro and other readers</li>
+ * <li>Writing 2-level or 3-level lists is configurable with either the Hudi option, hoodie.parquet.writelegacyformat.enabled, or the parquet-avro option, parquet.avro.write-old-list-structure
+ * to ensure consistency across writer paths.</li>
Review Comment:
nit: put line 81 to 80.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345352734
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,39 +18,90 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.LegacyBehaviorPolicy;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
* Hoodie Write Support for directly writing Row to Parquet.
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
+ private static final Schema MAP_KEY_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final String MAP_REPEATED_NAME = "key_value";
+ private static final String MAP_KEY_NAME = "key";
+ private static final String MAP_VALUE_NAME = "value";
private final Configuration hadoopConf;
private final Option<HoodieBloomFilterWriteSupport<UTF8String>> bloomFilterWriteSupportOpt;
+ private final byte[] decimalBuffer = new byte[Decimal.minBytesForPrecision()[DecimalType.MAX_PRECISION()]];
+ private final Enumeration.Value datetimeRebaseMode = LegacyBehaviorPolicy.withName(SQLConf.get().getConf(SQLConf.PARQUET_REBASE_MODE_IN_WRITE()));
+ private final Function1<Object, Object> dateRebaseFunction = DataSourceUtils.createDateRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private final Function1<Object, Object> timestampRebaseFunction = DataSourceUtils.createTimestampRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private RecordConsumer recordConsumer;
+ private final boolean writeLegacyListFormat;
+ private final ValueWriter[] rootFieldWriters;
+ private final Schema avroSchema;
- public HoodieRowParquetWriteSupport(Configuration conf, StructType structType, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
+ public HoodieRowParquetWriteSupport(Configuration conf, StructType schema, Schema avroSchema, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
Review Comment:
It's not always a HoodieWriteConfig, check out `ParquetUtils#serializeRecordsToLogBlock` for an example
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2342327656
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,39 +18,90 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.LegacyBehaviorPolicy;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
* Hoodie Write Support for directly writing Row to Parquet.
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
+ private static final Schema MAP_KEY_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final String MAP_REPEATED_NAME = "key_value";
+ private static final String MAP_KEY_NAME = "key";
+ private static final String MAP_VALUE_NAME = "value";
private final Configuration hadoopConf;
private final Option<HoodieBloomFilterWriteSupport<UTF8String>> bloomFilterWriteSupportOpt;
+ private final byte[] decimalBuffer = new byte[Decimal.minBytesForPrecision()[DecimalType.MAX_PRECISION()]];
+ private final Enumeration.Value datetimeRebaseMode = LegacyBehaviorPolicy.withName(SQLConf.get().getConf(SQLConf.PARQUET_REBASE_MODE_IN_WRITE()));
+ private final Function1<Object, Object> dateRebaseFunction = DataSourceUtils.createDateRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private final Function1<Object, Object> timestampRebaseFunction = DataSourceUtils.createTimestampRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private RecordConsumer recordConsumer;
+ private final boolean writeLegacyListFormat;
+ private final ValueWriter[] rootFieldWriters;
+ private final Schema avroSchema;
- public HoodieRowParquetWriteSupport(Configuration conf, StructType structType, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
+ public HoodieRowParquetWriteSupport(Configuration conf, StructType schema, Schema avroSchema, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
Review Comment:
In the future, when `HoodieSchema` abstraction is introduced, it's easier to swap the current schema representation with `HoodieSchema` in a limited number of places.
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/HoodieDataTypeUtils.scala:
##########
@@ -37,72 +29,4 @@ object HoodieDataTypeUtils {
*/
def parseStructTypeFromJson(jsonSchema: String): StructType =
StructType.fromString(jsonSchema)
-
- def canUseRowWriter(schema: Schema, conf: Configuration): Boolean = {
- if (conf.getBoolean(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE, true)) {
- // if we can write lists with the old list structure, we can use row writer regardless of decimal precision
- true
- } else if (!HoodieAvroUtils.hasSmallPrecisionDecimalField(schema)) {
- true
- } else {
- // small precision decimals require the legacy write mode but lists and maps require the new write mode when
- // WRITE_OLD_LIST_STRUCTURE is false so we can only use row writer if one is present and the other is not
- if (HoodieAvroUtils.hasListOrMapField(schema)) {
- log.warn("Cannot use row writer due to presence of list or map with a small precision decimal field")
- false
- } else {
- true
- }
- }
- }
Review Comment:
To clarify, there is no longer cases that row writer should be not used and fallback should be incurred, correct? And the timestamp precision is also properly handled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3283198348
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5bca64047318b420b21f7dadd1f274d1330319e1 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345737737
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -73,6 +175,173 @@ public void add(UTF8String recordKey) {
bloomFilterWriteSupport.addKey(recordKey));
}
+ @FunctionalInterface
+ private interface ValueWriter {
+ void write(SpecializedGetters row, int ordinal);
+ }
+
+ private void consumeMessage(Runnable writer) {
+ recordConsumer.startMessage();
+ writer.run();
+ recordConsumer.endMessage();
+ }
+
+ private void consumeGroup(Runnable writer) {
+ recordConsumer.startGroup();
+ writer.run();
+ recordConsumer.endGroup();
+ }
+
+ private void consumeField(String field, int index, Runnable writer) {
+ recordConsumer.startField(field, index);
+ writer.run();
+ recordConsumer.endField(field, index);
+ }
+
+ private void writeFields(InternalRow row, Schema schema, ValueWriter[] fieldWriters) {
+ for (int i = 0; i < fieldWriters.length; i++) {
+ int index = i;
+ if (!row.isNullAt(i)) {
+ Schema.Field field = schema.getFields().get(index);
+ consumeField(field.name(), index, () -> fieldWriters[index].write(row, index));
+ }
+ }
+ }
+
+ private ValueWriter makeWriter(Schema avroSchema, DataType dataType) {
+ Schema resolvedSchema = resolveNullableSchema(avroSchema);
+ Schema.Type type = resolvedSchema.getType();
+ LogicalType logicalType = resolvedSchema.getLogicalType();
+ switch (type) {
+ case BOOLEAN:
+ return (row, ordinal) -> recordConsumer.addBoolean(row.getBoolean(ordinal));
+ case INT:
+ if (logicalType != null) {
+ if (logicalType.getName().equals(LogicalTypes.date().getName())) {
+ return (row, ordinal) -> recordConsumer.addInteger((Integer) dateRebaseFunction.apply(row.getInt(ordinal)));
+ }
+ }
+ return (row, ordinal) -> recordConsumer.addInteger(row.getInt(ordinal));
+ case LONG:
+ if (logicalType != null) {
+ if (logicalType.getName().equals(LogicalTypes.timestampMillis().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong(DateTimeUtils.microsToMillis((long) timestampRebaseFunction.apply(row.getLong(ordinal))));
Review Comment:
Yes, I'm going to flip this so the struct is the source of truth for the incoming type and then the avro just augments the writer by providing the scale of the timestamps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3287395366
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
}, {
"hash" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491",
"triggerID" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"triggerType" : "PUSH"
}, {
"hash" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8503",
"triggerID" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"triggerType" : "PUSH"
}, {
"hash" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8506",
"triggerID" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"triggerType" : "PUSH"
}, {
"hash" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8507",
"triggerID" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"triggerType" : "PUSH"
}, {
"hash" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8510",
"triggerID" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"triggerType" : "PUSH"
}, {
"hash" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8511",
"triggerID" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"triggerType" : "PUSH"
}, {
"hash" : "3ff5acf174c775406ffb54be92650b03718bf48b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8513",
"triggerID" : "3ff5acf174c775406ffb54be92650b03718bf48b",
"triggerType" : "PUSH"
}, {
"hash" : "25a0be4048eb5d9e8c082ad7ff1026053a20747c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "25a0be4048eb5d9e8c082ad7ff1026053a20747c",
"triggerType" : "PUSH"
}, {
"hash" : "27b7921b67ae40d374478457fa51b7ffc7336f50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27b7921b67ae40d374478457fa51b7ffc7336f50",
"triggerType" : "PUSH"
}, {
"hash" : "55213c73a0d42e87f0132b15eb793ef33e3561d3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "55213c73a0d42e87f0132b15eb793ef33e3561d3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3ff5acf174c775406ffb54be92650b03718bf48b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8513)
* 25a0be4048eb5d9e8c082ad7ff1026053a20747c UNKNOWN
* 27b7921b67ae40d374478457fa51b7ffc7336f50 UNKNOWN
* 55213c73a0d42e87f0132b15eb793ef33e3561d3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3287504334
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
}, {
"hash" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491",
"triggerID" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"triggerType" : "PUSH"
}, {
"hash" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8503",
"triggerID" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"triggerType" : "PUSH"
}, {
"hash" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8506",
"triggerID" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"triggerType" : "PUSH"
}, {
"hash" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8507",
"triggerID" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"triggerType" : "PUSH"
}, {
"hash" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8510",
"triggerID" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"triggerType" : "PUSH"
}, {
"hash" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8511",
"triggerID" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"triggerType" : "PUSH"
}, {
"hash" : "3ff5acf174c775406ffb54be92650b03718bf48b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8513",
"triggerID" : "3ff5acf174c775406ffb54be92650b03718bf48b",
"triggerType" : "PUSH"
}, {
"hash" : "25a0be4048eb5d9e8c082ad7ff1026053a20747c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "25a0be4048eb5d9e8c082ad7ff1026053a20747c",
"triggerType" : "PUSH"
}, {
"hash" : "27b7921b67ae40d374478457fa51b7ffc7336f50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27b7921b67ae40d374478457fa51b7ffc7336f50",
"triggerType" : "PUSH"
}, {
"hash" : "55213c73a0d42e87f0132b15eb793ef33e3561d3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8514",
"triggerID" : "55213c73a0d42e87f0132b15eb793ef33e3561d3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25a0be4048eb5d9e8c082ad7ff1026053a20747c UNKNOWN
* 27b7921b67ae40d374478457fa51b7ffc7336f50 UNKNOWN
* 55213c73a0d42e87f0132b15eb793ef33e3561d3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8514)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2346902129
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -73,6 +189,180 @@ public void add(UTF8String recordKey) {
bloomFilterWriteSupport.addKey(recordKey));
}
+ @FunctionalInterface
+ private interface ValueWriter {
+ void write(SpecializedGetters row, int ordinal);
+ }
+
+ private void consumeMessage(Runnable writer) {
+ recordConsumer.startMessage();
+ writer.run();
+ recordConsumer.endMessage();
+ }
+
+ private void consumeGroup(Runnable writer) {
+ recordConsumer.startGroup();
+ writer.run();
+ recordConsumer.endGroup();
+ }
+
+ private void consumeField(String field, int index, Runnable writer) {
+ recordConsumer.startField(field, index);
+ writer.run();
+ recordConsumer.endField(field, index);
+ }
+
+ private void writeFields(InternalRow row, StructType schema, ValueWriter[] fieldWriters) {
+ for (int i = 0; i < row.numFields(); i++) {
+ int index = i;
+ if (!row.isNullAt(i)) {
+ StructField field = schema.fields()[i];
+ consumeField(field.name(), index, () -> fieldWriters[index].write(row, index));
+ }
Review Comment:
Updated
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -73,6 +189,180 @@ public void add(UTF8String recordKey) {
bloomFilterWriteSupport.addKey(recordKey));
}
+ @FunctionalInterface
+ private interface ValueWriter {
+ void write(SpecializedGetters row, int ordinal);
+ }
+
+ private void consumeMessage(Runnable writer) {
+ recordConsumer.startMessage();
+ writer.run();
+ recordConsumer.endMessage();
+ }
+
+ private void consumeGroup(Runnable writer) {
+ recordConsumer.startGroup();
+ writer.run();
+ recordConsumer.endGroup();
+ }
+
+ private void consumeField(String field, int index, Runnable writer) {
+ recordConsumer.startField(field, index);
+ writer.run();
+ recordConsumer.endField(field, index);
+ }
+
+ private void writeFields(InternalRow row, StructType schema, ValueWriter[] fieldWriters) {
+ for (int i = 0; i < row.numFields(); i++) {
+ int index = i;
+ if (!row.isNullAt(i)) {
+ StructField field = schema.fields()[i];
+ consumeField(field.name(), index, () -> fieldWriters[index].write(row, index));
+ }
+ }
+ }
+
+ private ValueWriter makeWriter(Schema avroSchema, DataType dataType) {
+ Schema resolvedSchema = avroSchema == null ? null : resolveNullableSchema(avroSchema);
+ LogicalType logicalType = resolvedSchema != null ? resolvedSchema.getLogicalType() : null;
+
+ if (dataType == DataTypes.BooleanType) {
+ return (row, ordinal) -> recordConsumer.addBoolean(row.getBoolean(ordinal));
+ } else if (dataType == DataTypes.DateType) {
+ return (row, ordinal) -> recordConsumer.addInteger((Integer) dateRebaseFunction.apply(row.getInt(ordinal)));
+ } else if (dataType == DataTypes.ShortType) {
+ return (row, ordinal) -> recordConsumer.addInteger(row.getShort(ordinal));
+ } else if (dataType == DataTypes.IntegerType) {
Review Comment:
Added this and the other one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3288887374
Ignoring the PR validation as the validation is outdated and to be updated by #13883. The PR title conforms to the new guideline.
Merging this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345589709
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -73,6 +178,173 @@ public void add(UTF8String recordKey) {
bloomFilterWriteSupport.addKey(recordKey));
}
+ @FunctionalInterface
+ private interface ValueWriter {
+ void write(SpecializedGetters row, int ordinal);
+ }
+
+ private void consumeMessage(Runnable writer) {
+ recordConsumer.startMessage();
+ writer.run();
+ recordConsumer.endMessage();
+ }
+
+ private void consumeGroup(Runnable writer) {
+ recordConsumer.startGroup();
+ writer.run();
+ recordConsumer.endGroup();
+ }
+
+ private void consumeField(String field, int index, Runnable writer) {
+ recordConsumer.startField(field, index);
+ writer.run();
+ recordConsumer.endField(field, index);
+ }
+
+ private void writeFields(InternalRow row, Schema schema, ValueWriter[] fieldWriters) {
+ for (int i = 0; i < fieldWriters.length; i++) {
+ int index = i;
+ if (!row.isNullAt(i)) {
+ Schema.Field field = schema.getFields().get(index);
+ consumeField(field.name(), index, () -> fieldWriters[index].write(row, index));
+ }
+ }
+ }
+
+ private ValueWriter makeWriter(Schema avroSchema, DataType dataType) {
+ Schema resolvedSchema = resolveNullableSchema(avroSchema);
+ Schema.Type type = resolvedSchema.getType();
+ LogicalType logicalType = resolvedSchema.getLogicalType();
+ switch (type) {
+ case BOOLEAN:
+ return (row, ordinal) -> recordConsumer.addBoolean(row.getBoolean(ordinal));
+ case INT:
+ if (logicalType != null) {
+ if (logicalType.getName().equals(LogicalTypes.date().getName())) {
+ return (row, ordinal) -> recordConsumer.addInteger((Integer) dateRebaseFunction.apply(row.getInt(ordinal)));
+ }
+ }
+ return (row, ordinal) -> recordConsumer.addInteger(row.getInt(ordinal));
+ case LONG:
+ if (logicalType != null) {
+ if (logicalType.getName().equals(LogicalTypes.timestampMillis().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong(DateTimeUtils.microsToMillis((long) timestampRebaseFunction.apply(row.getLong(ordinal))));
+ } else if (logicalType.getName().equals(LogicalTypes.timestampMicros().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong((long) timestampRebaseFunction.apply(row.getLong(ordinal)));
+ } else if (logicalType.getName().equals(LogicalTypes.localTimestampMicros().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong(row.getLong(ordinal));
+ } else if (logicalType.getName().equals(LogicalTypes.localTimestampMillis().getName()) && dataType instanceof TimestampNTZType) {
+ // only go through conversion if spark data type is TimestampNTZType
+ return (row, ordinal) -> recordConsumer.addLong(DateTimeUtils.microsToMillis(row.getLong(ordinal)));
+ }
+ }
+ return (row, ordinal) -> recordConsumer.addLong(row.getLong(ordinal));
+ case FLOAT:
+ return (row, ordinal) -> recordConsumer.addFloat(row.getFloat(ordinal));
+ case DOUBLE:
+ return (row, ordinal) -> recordConsumer.addDouble(row.getDouble(ordinal));
+ case STRING:
+ case ENUM:
+ return (row, ordinal) -> recordConsumer.addBinary(
+ Binary.fromReusedByteArray(row.getUTF8String(ordinal).getBytes()));
+ case BYTES:
+ case FIXED:
+ if (logicalType != null && logicalType.getName().equals("decimal")) {
+ return (row, ordinal) -> {
+ int precision = ((LogicalTypes.Decimal) logicalType).getPrecision();
+ int scale = ((LogicalTypes.Decimal) logicalType).getScale();
+ long unscaled = row.getDecimal(ordinal, precision, scale).toUnscaledLong();
+ int i = 0;
+ int numBytes = Decimal.minBytesForPrecision()[precision];
+ int shift = 8 * (numBytes - 1);
+ while (i < numBytes) {
+ decimalBuffer[i] = (byte) ((unscaled >> shift) & 0xFF);
+ i += 1;
+ shift -= 8;
+ }
+ recordConsumer.addBinary(Binary.fromReusedByteArray(decimalBuffer, 0, numBytes));
+ };
+ }
+ return (row, ordinal) -> recordConsumer.addBinary(
+ Binary.fromReusedByteArray(row.getBinary(ordinal)));
+ case RECORD:
+ ValueWriter[] fieldWriters = getFieldWriters(((StructType) dataType), resolvedSchema);
+ return (row, ordinal) ->
+ consumeGroup(() -> writeFields(row.getStruct(ordinal, resolvedSchema.getFields().size()), resolvedSchema, fieldWriters));
+ case ARRAY:
+ ValueWriter elementWriter = makeWriter(resolvedSchema.getElementType(), ((ArrayType) dataType).elementType());
+ if (!writeLegacyListFormat) {
+ return threeLevelArrayWriter("list", "element", elementWriter);
+ } else if (resolvedSchema.getElementType().isNullable()) {
+ return threeLevelArrayWriter("bag", "array", elementWriter);
+ } else {
+ return twoLevelArrayWriter("array", elementWriter);
+ }
+ case MAP:
+ ValueWriter keyWriter = makeWriter(MAP_KEY_SCHEMA, DataTypes.StringType);
+ ValueWriter valueWriter = makeWriter(resolvedSchema.getValueType(), ((MapType) dataType).valueType());
+ return (row, ordinal) -> {
+ MapData mapData = row.getMap(ordinal);
+ ArrayData keyArray = mapData.keyArray();
+ ArrayData valueArray = mapData.valueArray();
+ consumeGroup(() -> {
+ if (mapData.numElements() > 0) {
+ consumeField(MAP_REPEATED_NAME, 0, () -> {
+ for (int i = 0; i < mapData.numElements(); i++) {
+ int index = i;
+ consumeGroup(() -> {
+ if (!keyArray.isNullAt(index)) {
Review Comment:
`ParquetWriteSupport` does not check null for key. Does Avro allow null so we need this?
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -73,6 +178,173 @@ public void add(UTF8String recordKey) {
bloomFilterWriteSupport.addKey(recordKey));
}
+ @FunctionalInterface
+ private interface ValueWriter {
+ void write(SpecializedGetters row, int ordinal);
+ }
+
+ private void consumeMessage(Runnable writer) {
+ recordConsumer.startMessage();
+ writer.run();
+ recordConsumer.endMessage();
+ }
+
+ private void consumeGroup(Runnable writer) {
+ recordConsumer.startGroup();
+ writer.run();
+ recordConsumer.endGroup();
+ }
+
+ private void consumeField(String field, int index, Runnable writer) {
+ recordConsumer.startField(field, index);
+ writer.run();
+ recordConsumer.endField(field, index);
+ }
+
+ private void writeFields(InternalRow row, Schema schema, ValueWriter[] fieldWriters) {
+ for (int i = 0; i < fieldWriters.length; i++) {
+ int index = i;
+ if (!row.isNullAt(i)) {
+ Schema.Field field = schema.getFields().get(index);
+ consumeField(field.name(), index, () -> fieldWriters[index].write(row, index));
+ }
+ }
+ }
+
+ private ValueWriter makeWriter(Schema avroSchema, DataType dataType) {
+ Schema resolvedSchema = resolveNullableSchema(avroSchema);
+ Schema.Type type = resolvedSchema.getType();
+ LogicalType logicalType = resolvedSchema.getLogicalType();
+ switch (type) {
+ case BOOLEAN:
+ return (row, ordinal) -> recordConsumer.addBoolean(row.getBoolean(ordinal));
+ case INT:
+ if (logicalType != null) {
+ if (logicalType.getName().equals(LogicalTypes.date().getName())) {
+ return (row, ordinal) -> recordConsumer.addInteger((Integer) dateRebaseFunction.apply(row.getInt(ordinal)));
+ }
+ }
+ return (row, ordinal) -> recordConsumer.addInteger(row.getInt(ordinal));
+ case LONG:
+ if (logicalType != null) {
+ if (logicalType.getName().equals(LogicalTypes.timestampMillis().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong(DateTimeUtils.microsToMillis((long) timestampRebaseFunction.apply(row.getLong(ordinal))));
+ } else if (logicalType.getName().equals(LogicalTypes.timestampMicros().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong((long) timestampRebaseFunction.apply(row.getLong(ordinal)));
+ } else if (logicalType.getName().equals(LogicalTypes.localTimestampMicros().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong(row.getLong(ordinal));
+ } else if (logicalType.getName().equals(LogicalTypes.localTimestampMillis().getName()) && dataType instanceof TimestampNTZType) {
+ // only go through conversion if spark data type is TimestampNTZType
+ return (row, ordinal) -> recordConsumer.addLong(DateTimeUtils.microsToMillis(row.getLong(ordinal)));
+ }
+ }
+ return (row, ordinal) -> recordConsumer.addLong(row.getLong(ordinal));
+ case FLOAT:
+ return (row, ordinal) -> recordConsumer.addFloat(row.getFloat(ordinal));
+ case DOUBLE:
+ return (row, ordinal) -> recordConsumer.addDouble(row.getDouble(ordinal));
+ case STRING:
+ case ENUM:
+ return (row, ordinal) -> recordConsumer.addBinary(
+ Binary.fromReusedByteArray(row.getUTF8String(ordinal).getBytes()));
+ case BYTES:
+ case FIXED:
+ if (logicalType != null && logicalType.getName().equals("decimal")) {
+ return (row, ordinal) -> {
+ int precision = ((LogicalTypes.Decimal) logicalType).getPrecision();
+ int scale = ((LogicalTypes.Decimal) logicalType).getScale();
+ long unscaled = row.getDecimal(ordinal, precision, scale).toUnscaledLong();
+ int i = 0;
+ int numBytes = Decimal.minBytesForPrecision()[precision];
+ int shift = 8 * (numBytes - 1);
+ while (i < numBytes) {
+ decimalBuffer[i] = (byte) ((unscaled >> shift) & 0xFF);
+ i += 1;
+ shift -= 8;
+ }
+ recordConsumer.addBinary(Binary.fromReusedByteArray(decimalBuffer, 0, numBytes));
+ };
+ }
+ return (row, ordinal) -> recordConsumer.addBinary(
+ Binary.fromReusedByteArray(row.getBinary(ordinal)));
+ case RECORD:
+ ValueWriter[] fieldWriters = getFieldWriters(((StructType) dataType), resolvedSchema);
+ return (row, ordinal) ->
+ consumeGroup(() -> writeFields(row.getStruct(ordinal, resolvedSchema.getFields().size()), resolvedSchema, fieldWriters));
+ case ARRAY:
+ ValueWriter elementWriter = makeWriter(resolvedSchema.getElementType(), ((ArrayType) dataType).elementType());
+ if (!writeLegacyListFormat) {
+ return threeLevelArrayWriter("list", "element", elementWriter);
+ } else if (resolvedSchema.getElementType().isNullable()) {
+ return threeLevelArrayWriter("bag", "array", elementWriter);
+ } else {
+ return twoLevelArrayWriter("array", elementWriter);
+ }
+ case MAP:
+ ValueWriter keyWriter = makeWriter(MAP_KEY_SCHEMA, DataTypes.StringType);
+ ValueWriter valueWriter = makeWriter(resolvedSchema.getValueType(), ((MapType) dataType).valueType());
+ return (row, ordinal) -> {
+ MapData mapData = row.getMap(ordinal);
+ ArrayData keyArray = mapData.keyArray();
+ ArrayData valueArray = mapData.valueArray();
+ consumeGroup(() -> {
+ if (mapData.numElements() > 0) {
+ consumeField(MAP_REPEATED_NAME, 0, () -> {
+ for (int i = 0; i < mapData.numElements(); i++) {
+ int index = i;
Review Comment:
nit: no need for a new variable `index`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345640239
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -73,6 +178,173 @@ public void add(UTF8String recordKey) {
bloomFilterWriteSupport.addKey(recordKey));
}
+ @FunctionalInterface
+ private interface ValueWriter {
+ void write(SpecializedGetters row, int ordinal);
+ }
+
+ private void consumeMessage(Runnable writer) {
+ recordConsumer.startMessage();
+ writer.run();
+ recordConsumer.endMessage();
+ }
+
+ private void consumeGroup(Runnable writer) {
+ recordConsumer.startGroup();
+ writer.run();
+ recordConsumer.endGroup();
+ }
+
+ private void consumeField(String field, int index, Runnable writer) {
+ recordConsumer.startField(field, index);
+ writer.run();
+ recordConsumer.endField(field, index);
+ }
+
+ private void writeFields(InternalRow row, Schema schema, ValueWriter[] fieldWriters) {
+ for (int i = 0; i < fieldWriters.length; i++) {
+ int index = i;
+ if (!row.isNullAt(i)) {
+ Schema.Field field = schema.getFields().get(index);
+ consumeField(field.name(), index, () -> fieldWriters[index].write(row, index));
+ }
+ }
+ }
+
+ private ValueWriter makeWriter(Schema avroSchema, DataType dataType) {
+ Schema resolvedSchema = resolveNullableSchema(avroSchema);
+ Schema.Type type = resolvedSchema.getType();
+ LogicalType logicalType = resolvedSchema.getLogicalType();
+ switch (type) {
+ case BOOLEAN:
+ return (row, ordinal) -> recordConsumer.addBoolean(row.getBoolean(ordinal));
+ case INT:
+ if (logicalType != null) {
+ if (logicalType.getName().equals(LogicalTypes.date().getName())) {
+ return (row, ordinal) -> recordConsumer.addInteger((Integer) dateRebaseFunction.apply(row.getInt(ordinal)));
+ }
+ }
+ return (row, ordinal) -> recordConsumer.addInteger(row.getInt(ordinal));
+ case LONG:
+ if (logicalType != null) {
+ if (logicalType.getName().equals(LogicalTypes.timestampMillis().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong(DateTimeUtils.microsToMillis((long) timestampRebaseFunction.apply(row.getLong(ordinal))));
+ } else if (logicalType.getName().equals(LogicalTypes.timestampMicros().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong((long) timestampRebaseFunction.apply(row.getLong(ordinal)));
+ } else if (logicalType.getName().equals(LogicalTypes.localTimestampMicros().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong(row.getLong(ordinal));
+ } else if (logicalType.getName().equals(LogicalTypes.localTimestampMillis().getName()) && dataType instanceof TimestampNTZType) {
+ // only go through conversion if spark data type is TimestampNTZType
+ return (row, ordinal) -> recordConsumer.addLong(DateTimeUtils.microsToMillis(row.getLong(ordinal)));
+ }
+ }
+ return (row, ordinal) -> recordConsumer.addLong(row.getLong(ordinal));
+ case FLOAT:
+ return (row, ordinal) -> recordConsumer.addFloat(row.getFloat(ordinal));
+ case DOUBLE:
+ return (row, ordinal) -> recordConsumer.addDouble(row.getDouble(ordinal));
+ case STRING:
+ case ENUM:
+ return (row, ordinal) -> recordConsumer.addBinary(
+ Binary.fromReusedByteArray(row.getUTF8String(ordinal).getBytes()));
+ case BYTES:
+ case FIXED:
+ if (logicalType != null && logicalType.getName().equals("decimal")) {
+ return (row, ordinal) -> {
+ int precision = ((LogicalTypes.Decimal) logicalType).getPrecision();
+ int scale = ((LogicalTypes.Decimal) logicalType).getScale();
+ long unscaled = row.getDecimal(ordinal, precision, scale).toUnscaledLong();
+ int i = 0;
+ int numBytes = Decimal.minBytesForPrecision()[precision];
+ int shift = 8 * (numBytes - 1);
+ while (i < numBytes) {
+ decimalBuffer[i] = (byte) ((unscaled >> shift) & 0xFF);
+ i += 1;
+ shift -= 8;
+ }
+ recordConsumer.addBinary(Binary.fromReusedByteArray(decimalBuffer, 0, numBytes));
+ };
+ }
+ return (row, ordinal) -> recordConsumer.addBinary(
+ Binary.fromReusedByteArray(row.getBinary(ordinal)));
+ case RECORD:
+ ValueWriter[] fieldWriters = getFieldWriters(((StructType) dataType), resolvedSchema);
+ return (row, ordinal) ->
+ consumeGroup(() -> writeFields(row.getStruct(ordinal, resolvedSchema.getFields().size()), resolvedSchema, fieldWriters));
+ case ARRAY:
+ ValueWriter elementWriter = makeWriter(resolvedSchema.getElementType(), ((ArrayType) dataType).elementType());
+ if (!writeLegacyListFormat) {
+ return threeLevelArrayWriter("list", "element", elementWriter);
+ } else if (resolvedSchema.getElementType().isNullable()) {
+ return threeLevelArrayWriter("bag", "array", elementWriter);
+ } else {
+ return twoLevelArrayWriter("array", elementWriter);
+ }
+ case MAP:
+ ValueWriter keyWriter = makeWriter(MAP_KEY_SCHEMA, DataTypes.StringType);
+ ValueWriter valueWriter = makeWriter(resolvedSchema.getValueType(), ((MapType) dataType).valueType());
+ return (row, ordinal) -> {
+ MapData mapData = row.getMap(ordinal);
+ ArrayData keyArray = mapData.keyArray();
+ ArrayData valueArray = mapData.valueArray();
+ consumeGroup(() -> {
+ if (mapData.numElements() > 0) {
+ consumeField(MAP_REPEATED_NAME, 0, () -> {
+ for (int i = 0; i < mapData.numElements(); i++) {
+ int index = i;
+ consumeGroup(() -> {
+ if (!keyArray.isNullAt(index)) {
Review Comment:
It does not based on my understanding, removing this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2346866775
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -73,6 +189,180 @@ public void add(UTF8String recordKey) {
bloomFilterWriteSupport.addKey(recordKey));
}
+ @FunctionalInterface
+ private interface ValueWriter {
+ void write(SpecializedGetters row, int ordinal);
+ }
+
+ private void consumeMessage(Runnable writer) {
+ recordConsumer.startMessage();
+ writer.run();
+ recordConsumer.endMessage();
+ }
+
+ private void consumeGroup(Runnable writer) {
+ recordConsumer.startGroup();
+ writer.run();
+ recordConsumer.endGroup();
+ }
+
+ private void consumeField(String field, int index, Runnable writer) {
+ recordConsumer.startField(field, index);
+ writer.run();
+ recordConsumer.endField(field, index);
+ }
+
+ private void writeFields(InternalRow row, StructType schema, ValueWriter[] fieldWriters) {
+ for (int i = 0; i < row.numFields(); i++) {
+ int index = i;
+ if (!row.isNullAt(i)) {
+ StructField field = schema.fields()[i];
+ consumeField(field.name(), index, () -> fieldWriters[index].write(row, index));
+ }
+ }
+ }
+
+ private ValueWriter makeWriter(Schema avroSchema, DataType dataType) {
+ Schema resolvedSchema = avroSchema == null ? null : resolveNullableSchema(avroSchema);
+ LogicalType logicalType = resolvedSchema != null ? resolvedSchema.getLogicalType() : null;
+
+ if (dataType == DataTypes.BooleanType) {
+ return (row, ordinal) -> recordConsumer.addBoolean(row.getBoolean(ordinal));
+ } else if (dataType == DataTypes.DateType) {
+ return (row, ordinal) -> recordConsumer.addInteger((Integer) dateRebaseFunction.apply(row.getInt(ordinal)));
+ } else if (dataType == DataTypes.ShortType) {
+ return (row, ordinal) -> recordConsumer.addInteger(row.getShort(ordinal));
+ } else if (dataType == DataTypes.IntegerType) {
+ return (row, ordinal) -> recordConsumer.addInteger(row.getInt(ordinal));
+ } else if (dataType == DataTypes.LongType) {
+ return (row, ordinal) -> recordConsumer.addLong(row.getLong(ordinal));
+ } else if (dataType == DataTypes.TimestampType) {
+ if (logicalType == null || logicalType.getName().equals(LogicalTypes.timestampMicros().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong((long) timestampRebaseFunction.apply(row.getLong(ordinal)));
+ } else if (logicalType.getName().equals(LogicalTypes.timestampMillis().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong(DateTimeUtils.microsToMillis((long) timestampRebaseFunction.apply(row.getLong(ordinal))));
+ } else {
+ throw new UnsupportedOperationException("Unsupported timestamp type: " + logicalType);
Review Comment:
```suggestion
throw new UnsupportedOperationException("Unsupported Avro logical type for TimestampType: " + logicalType);
```
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -73,6 +189,180 @@ public void add(UTF8String recordKey) {
bloomFilterWriteSupport.addKey(recordKey));
}
+ @FunctionalInterface
+ private interface ValueWriter {
+ void write(SpecializedGetters row, int ordinal);
+ }
+
+ private void consumeMessage(Runnable writer) {
+ recordConsumer.startMessage();
+ writer.run();
+ recordConsumer.endMessage();
+ }
+
+ private void consumeGroup(Runnable writer) {
+ recordConsumer.startGroup();
+ writer.run();
+ recordConsumer.endGroup();
+ }
+
+ private void consumeField(String field, int index, Runnable writer) {
+ recordConsumer.startField(field, index);
+ writer.run();
+ recordConsumer.endField(field, index);
+ }
+
+ private void writeFields(InternalRow row, StructType schema, ValueWriter[] fieldWriters) {
+ for (int i = 0; i < row.numFields(); i++) {
+ int index = i;
+ if (!row.isNullAt(i)) {
+ StructField field = schema.fields()[i];
+ consumeField(field.name(), index, () -> fieldWriters[index].write(row, index));
+ }
+ }
+ }
+
+ private ValueWriter makeWriter(Schema avroSchema, DataType dataType) {
+ Schema resolvedSchema = avroSchema == null ? null : resolveNullableSchema(avroSchema);
+ LogicalType logicalType = resolvedSchema != null ? resolvedSchema.getLogicalType() : null;
+
+ if (dataType == DataTypes.BooleanType) {
+ return (row, ordinal) -> recordConsumer.addBoolean(row.getBoolean(ordinal));
+ } else if (dataType == DataTypes.DateType) {
+ return (row, ordinal) -> recordConsumer.addInteger((Integer) dateRebaseFunction.apply(row.getInt(ordinal)));
+ } else if (dataType == DataTypes.ShortType) {
+ return (row, ordinal) -> recordConsumer.addInteger(row.getShort(ordinal));
+ } else if (dataType == DataTypes.IntegerType) {
+ return (row, ordinal) -> recordConsumer.addInteger(row.getInt(ordinal));
+ } else if (dataType == DataTypes.LongType) {
+ return (row, ordinal) -> recordConsumer.addLong(row.getLong(ordinal));
+ } else if (dataType == DataTypes.TimestampType) {
+ if (logicalType == null || logicalType.getName().equals(LogicalTypes.timestampMicros().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong((long) timestampRebaseFunction.apply(row.getLong(ordinal)));
+ } else if (logicalType.getName().equals(LogicalTypes.timestampMillis().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong(DateTimeUtils.microsToMillis((long) timestampRebaseFunction.apply(row.getLong(ordinal))));
+ } else {
+ throw new UnsupportedOperationException("Unsupported timestamp type: " + logicalType);
+ }
+ } else if (SparkAdapterSupport$.MODULE$.sparkAdapter().isTimestampNTZType(dataType)) {
+ if (logicalType == null || logicalType.getName().equals(LogicalTypes.localTimestampMicros().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong(row.getLong(ordinal));
+ } else if (logicalType.getName().equals(LogicalTypes.localTimestampMillis().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong(DateTimeUtils.microsToMillis(row.getLong(ordinal)));
+ } else {
+ throw new UnsupportedOperationException("Unsupported timestamp type: " + logicalType);
Review Comment:
```suggestion
throw new UnsupportedOperationException("Unsupported Avro logical type for TimestampNTZType: " + logicalType);
```
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -73,6 +189,180 @@ public void add(UTF8String recordKey) {
bloomFilterWriteSupport.addKey(recordKey));
}
+ @FunctionalInterface
+ private interface ValueWriter {
+ void write(SpecializedGetters row, int ordinal);
+ }
+
+ private void consumeMessage(Runnable writer) {
+ recordConsumer.startMessage();
+ writer.run();
+ recordConsumer.endMessage();
+ }
+
+ private void consumeGroup(Runnable writer) {
+ recordConsumer.startGroup();
+ writer.run();
+ recordConsumer.endGroup();
+ }
+
+ private void consumeField(String field, int index, Runnable writer) {
+ recordConsumer.startField(field, index);
+ writer.run();
+ recordConsumer.endField(field, index);
+ }
+
+ private void writeFields(InternalRow row, StructType schema, ValueWriter[] fieldWriters) {
+ for (int i = 0; i < row.numFields(); i++) {
+ int index = i;
+ if (!row.isNullAt(i)) {
+ StructField field = schema.fields()[i];
+ consumeField(field.name(), index, () -> fieldWriters[index].write(row, index));
+ }
+ }
+ }
+
+ private ValueWriter makeWriter(Schema avroSchema, DataType dataType) {
+ Schema resolvedSchema = avroSchema == null ? null : resolveNullableSchema(avroSchema);
+ LogicalType logicalType = resolvedSchema != null ? resolvedSchema.getLogicalType() : null;
+
+ if (dataType == DataTypes.BooleanType) {
+ return (row, ordinal) -> recordConsumer.addBoolean(row.getBoolean(ordinal));
+ } else if (dataType == DataTypes.DateType) {
+ return (row, ordinal) -> recordConsumer.addInteger((Integer) dateRebaseFunction.apply(row.getInt(ordinal)));
+ } else if (dataType == DataTypes.ShortType) {
+ return (row, ordinal) -> recordConsumer.addInteger(row.getShort(ordinal));
+ } else if (dataType == DataTypes.IntegerType) {
+ return (row, ordinal) -> recordConsumer.addInteger(row.getInt(ordinal));
+ } else if (dataType == DataTypes.LongType) {
+ return (row, ordinal) -> recordConsumer.addLong(row.getLong(ordinal));
+ } else if (dataType == DataTypes.TimestampType) {
+ if (logicalType == null || logicalType.getName().equals(LogicalTypes.timestampMicros().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong((long) timestampRebaseFunction.apply(row.getLong(ordinal)));
+ } else if (logicalType.getName().equals(LogicalTypes.timestampMillis().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong(DateTimeUtils.microsToMillis((long) timestampRebaseFunction.apply(row.getLong(ordinal))));
+ } else {
+ throw new UnsupportedOperationException("Unsupported timestamp type: " + logicalType);
+ }
+ } else if (SparkAdapterSupport$.MODULE$.sparkAdapter().isTimestampNTZType(dataType)) {
+ if (logicalType == null || logicalType.getName().equals(LogicalTypes.localTimestampMicros().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong(row.getLong(ordinal));
+ } else if (logicalType.getName().equals(LogicalTypes.localTimestampMillis().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong(DateTimeUtils.microsToMillis(row.getLong(ordinal)));
+ } else {
+ throw new UnsupportedOperationException("Unsupported timestamp type: " + logicalType);
+ }
+ } else if (dataType == DataTypes.FloatType) {
+ return (row, ordinal) -> recordConsumer.addFloat(row.getFloat(ordinal));
+ } else if (dataType == DataTypes.DoubleType) {
+ return (row, ordinal) -> recordConsumer.addDouble(row.getDouble(ordinal));
+ } else if (dataType == DataTypes.StringType) {
+ return (row, ordinal) -> recordConsumer.addBinary(
+ Binary.fromReusedByteArray(row.getUTF8String(ordinal).getBytes()));
+ } else if (dataType == DataTypes.BinaryType) {
+ return (row, ordinal) -> recordConsumer.addBinary(
+ Binary.fromReusedByteArray(row.getBinary(ordinal)));
+ } else if (dataType instanceof DecimalType) {
+ return (row, ordinal) -> {
+ int precision = ((DecimalType) dataType).precision();
+ int scale = ((DecimalType) dataType).scale();
+ byte[] bytes = row.getDecimal(ordinal, precision, scale).toJavaBigDecimal().unscaledValue().toByteArray();
Review Comment:
Check `ValidationUtils.checkArgument(precision <= DecimalType.MAX_PRECISION)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345032403
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,47 +18,149 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
+import org.apache.hudi.SparkAdapterSupport;
+import org.apache.hudi.avro.HoodieAvroUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
- * Hoodie Write Support for directly writing Row to Parquet.
+ * Hoodie Write Support for directly writing Row to Parquet and adding the Hudi bloom index to the file metadata.
+ * The implementation is adapted from Spark's {@link org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport} but has the following major differences:
+ * <ul>
+ * <li>Decimals are always written with the legacy format to ensure compatibility with parquet-avro and other readers</li>
+ * <li>Writing 2-level or 3-level lists is configurable with either the Hudi option, hoodie.parquet.writelegacyformat.enabled, or the parquet-avro option, parquet.avro.write-old-list-structure
+ * to ensure consistency across writer paths.</li>
Review Comment:
it will exceed the line length limit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345401003
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,39 +18,90 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.LegacyBehaviorPolicy;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
* Hoodie Write Support for directly writing Row to Parquet.
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
+ private static final Schema MAP_KEY_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final String MAP_REPEATED_NAME = "key_value";
+ private static final String MAP_KEY_NAME = "key";
+ private static final String MAP_VALUE_NAME = "value";
private final Configuration hadoopConf;
private final Option<HoodieBloomFilterWriteSupport<UTF8String>> bloomFilterWriteSupportOpt;
+ private final byte[] decimalBuffer = new byte[Decimal.minBytesForPrecision()[DecimalType.MAX_PRECISION()]];
+ private final Enumeration.Value datetimeRebaseMode = LegacyBehaviorPolicy.withName(SQLConf.get().getConf(SQLConf.PARQUET_REBASE_MODE_IN_WRITE()));
+ private final Function1<Object, Object> dateRebaseFunction = DataSourceUtils.createDateRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private final Function1<Object, Object> timestampRebaseFunction = DataSourceUtils.createTimestampRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private RecordConsumer recordConsumer;
+ private final boolean writeLegacyListFormat;
+ private final ValueWriter[] rootFieldWriters;
+ private final Schema avroSchema;
- public HoodieRowParquetWriteSupport(Configuration conf, StructType structType, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
+ public HoodieRowParquetWriteSupport(Configuration conf, StructType schema, Schema avroSchema, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
Review Comment:
(2) Hudi Streamer: the source provides the input batch which is provided with the schema provider to include the schema of the source data; and this part has schema resolution between incoming and table schema
```
SchemaProvider getDeducedSchemaProvider(Schema incomingSchema, SchemaProvider sourceSchemaProvider, HoodieTableMetaClient metaClient) {
Option<Schema> latestTableSchemaOpt = UtilHelpers.getLatestTableSchema(hoodieSparkContext.jsc(), storage, cfg.targetBasePath, metaClient);
Option<InternalSchema> internalSchemaOpt = HoodieConversionUtils.toJavaOption(
HoodieSchemaUtils.getLatestTableInternalSchema(
HoodieStreamer.Config.getProps(conf, cfg), metaClient));
// Deduce proper target (writer's) schema for the input dataset, reconciling its
// schema w/ the table's one
Schema targetSchema = HoodieSchemaUtils.deduceWriterSchema(
HoodieAvroUtils.removeMetadataFields(incomingSchema),
latestTableSchemaOpt, internalSchemaOpt, props);
// Override schema provider with the reconciled target schema
return new DelegatingSchemaProvider(props, hoodieSparkContext.jsc(), sourceSchemaProvider,
new SimpleSchemaProvider(hoodieSparkContext.jsc(), targetSchema, props));
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3285483200
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
}, {
"hash" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491",
"triggerID" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"triggerType" : "PUSH"
}, {
"hash" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8503",
"triggerID" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 00f82b3c501021529d2fa67b4fb5860eef21d634 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491)
* 2380d2b0bc91be8f86fe87911f7c503d938a072f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8503)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3285613214
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
}, {
"hash" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491",
"triggerID" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"triggerType" : "PUSH"
}, {
"hash" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8503",
"triggerID" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2380d2b0bc91be8f86fe87911f7c503d938a072f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8503)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345478282
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/functional/TestSparkSortAndSizeClustering.java:
##########
@@ -130,24 +155,55 @@ public void writeAndClustering(boolean isRow) throws IOException {
List<Row> rows = readRecords();
assertEquals(numRecords, rows.size());
- validateDecimalTypeAfterClustering(writeStats);
+ validateTypesAfterClustering(writeStats);
+ validateDateAndTimestampFields(rows, ts);
}
- // Validate that clustering produces decimals in legacy format
- private void validateDecimalTypeAfterClustering(List<HoodieWriteStat> writeStats) {
+ private void validateDateAndTimestampFields(List<Row> rows, long ts) {
+ Schema schema = HoodieAvroUtils.addMetadataFields(getSchema(), false);
+ Timestamp timestamp = new Timestamp(ts);
+ int dateFieldIndex = schema.getField("date_nullable_field").pos();
+ int tsMillisFieldIndex = schema.getField("timestamp_millis_field").pos();
+ int tsMicrosNullableFieldIndex = schema.getField("timestamp_micros_nullable_field").pos();
+ int tsLocalMillisFieldIndex = schema.getField("timestamp_local_millis_nullable_field").pos();
+ int tsLocalMicrosFieldIndex = schema.getField("timestamp_local_micros_field").pos();
+ for (Row row : rows) {
+ assertEquals(timestamp, row.get(tsMillisFieldIndex));
+ if (!row.isNullAt(tsMicrosNullableFieldIndex)) {
+ assertEquals(timestamp, row.get(tsMicrosNullableFieldIndex));
+ }
+ if (!row.isNullAt(dateFieldIndex)) {
+ assertTrue(row.get(dateFieldIndex) instanceof Date);
+ }
+ if (!row.isNullAt(tsLocalMillisFieldIndex)) {
+ assertEquals(ts, row.get(tsLocalMillisFieldIndex));
+ }
+ assertEquals(ts * 1000L, row.get(tsLocalMicrosFieldIndex));
+ }
+ }
+
+ // Validate that clustering produces decimals in legacy format and lists in newer format. Assert that the unit is correct on the timestamps
+ private void validateTypesAfterClustering(List<HoodieWriteStat> writeStats) {
writeStats.stream().map(writeStat -> new StoragePath(metaClient.getBasePath(), writeStat.getPath())).forEach(writtenPath -> {
MessageType schema = ParquetUtils.readMetadata(storage, writtenPath)
.getFileMetaData().getSchema();
- int index = schema.getFieldIndex("height");
- Type decimalType = schema.getFields().get(index);
+ int decimalFieldIndex = schema.getFieldIndex("decimal_field");
+ Type decimalType = schema.getFields().get(decimalFieldIndex);
Review Comment:
Should the precision and scale of decimal type be validated too? Wondering if this PR fixes #11335 already.
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/functional/TestSparkSortAndSizeClustering.java:
##########
@@ -130,24 +155,55 @@ public void writeAndClustering(boolean isRow) throws IOException {
List<Row> rows = readRecords();
assertEquals(numRecords, rows.size());
- validateDecimalTypeAfterClustering(writeStats);
+ validateTypesAfterClustering(writeStats);
+ validateDateAndTimestampFields(rows, ts);
}
- // Validate that clustering produces decimals in legacy format
- private void validateDecimalTypeAfterClustering(List<HoodieWriteStat> writeStats) {
+ private void validateDateAndTimestampFields(List<Row> rows, long ts) {
+ Schema schema = HoodieAvroUtils.addMetadataFields(getSchema(), false);
+ Timestamp timestamp = new Timestamp(ts);
+ int dateFieldIndex = schema.getField("date_nullable_field").pos();
+ int tsMillisFieldIndex = schema.getField("timestamp_millis_field").pos();
+ int tsMicrosNullableFieldIndex = schema.getField("timestamp_micros_nullable_field").pos();
+ int tsLocalMillisFieldIndex = schema.getField("timestamp_local_millis_nullable_field").pos();
+ int tsLocalMicrosFieldIndex = schema.getField("timestamp_local_micros_field").pos();
+ for (Row row : rows) {
+ assertEquals(timestamp, row.get(tsMillisFieldIndex));
+ if (!row.isNullAt(tsMicrosNullableFieldIndex)) {
+ assertEquals(timestamp, row.get(tsMicrosNullableFieldIndex));
+ }
+ if (!row.isNullAt(dateFieldIndex)) {
+ assertTrue(row.get(dateFieldIndex) instanceof Date);
Review Comment:
Also validate the exact value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3283069962
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 190cbc827b787e7f347a76add30a94f7621e750d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477)
* 5bca64047318b420b21f7dadd1f274d1330319e1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345493650
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,47 +18,149 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
+import org.apache.hudi.SparkAdapterSupport;
+import org.apache.hudi.avro.HoodieAvroUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
- * Hoodie Write Support for directly writing Row to Parquet.
+ * Hoodie Write Support for directly writing Row to Parquet and adding the Hudi bloom index to the file metadata.
+ * The implementation is adapted from Spark's {@link org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport} but has the following major differences:
+ * <ul>
+ * <li>Decimals are always written with the legacy format to ensure compatibility with parquet-avro and other readers</li>
+ * <li>Writing 2-level or 3-level lists is configurable with either the Hudi option, hoodie.parquet.writelegacyformat.enabled, or the parquet-avro option, parquet.avro.write-old-list-structure
+ * to ensure consistency across writer paths.</li>
+ * <li>The scale of the timestamps is determined by the Hudi writer schema instead of relying on Spark configuration</li>
+ * </ul>
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
Review Comment:
Do we need to adjust and customize the configurations of the `ParquetReadSupport` when reading the parquet files on Spark based on the new write behavior, or are the changes to `WriteSupport` reflected in the Parquet data type in the files so standard parquer reader should work as expected?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345551438
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/functional/TestSparkSortAndSizeClustering.java:
##########
@@ -130,24 +155,55 @@ public void writeAndClustering(boolean isRow) throws IOException {
List<Row> rows = readRecords();
assertEquals(numRecords, rows.size());
- validateDecimalTypeAfterClustering(writeStats);
+ validateTypesAfterClustering(writeStats);
+ validateDateAndTimestampFields(rows, ts);
}
- // Validate that clustering produces decimals in legacy format
- private void validateDecimalTypeAfterClustering(List<HoodieWriteStat> writeStats) {
+ private void validateDateAndTimestampFields(List<Row> rows, long ts) {
+ Schema schema = HoodieAvroUtils.addMetadataFields(getSchema(), false);
+ Timestamp timestamp = new Timestamp(ts);
+ int dateFieldIndex = schema.getField("date_nullable_field").pos();
+ int tsMillisFieldIndex = schema.getField("timestamp_millis_field").pos();
+ int tsMicrosNullableFieldIndex = schema.getField("timestamp_micros_nullable_field").pos();
+ int tsLocalMillisFieldIndex = schema.getField("timestamp_local_millis_nullable_field").pos();
+ int tsLocalMicrosFieldIndex = schema.getField("timestamp_local_micros_field").pos();
+ for (Row row : rows) {
+ assertEquals(timestamp, row.get(tsMillisFieldIndex));
+ if (!row.isNullAt(tsMicrosNullableFieldIndex)) {
+ assertEquals(timestamp, row.get(tsMicrosNullableFieldIndex));
+ }
+ if (!row.isNullAt(dateFieldIndex)) {
+ assertTrue(row.get(dateFieldIndex) instanceof Date);
Review Comment:
Updated the test
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/functional/TestSparkSortAndSizeClustering.java:
##########
@@ -130,24 +155,55 @@ public void writeAndClustering(boolean isRow) throws IOException {
List<Row> rows = readRecords();
assertEquals(numRecords, rows.size());
- validateDecimalTypeAfterClustering(writeStats);
+ validateTypesAfterClustering(writeStats);
+ validateDateAndTimestampFields(rows, ts);
}
- // Validate that clustering produces decimals in legacy format
- private void validateDecimalTypeAfterClustering(List<HoodieWriteStat> writeStats) {
+ private void validateDateAndTimestampFields(List<Row> rows, long ts) {
+ Schema schema = HoodieAvroUtils.addMetadataFields(getSchema(), false);
+ Timestamp timestamp = new Timestamp(ts);
+ int dateFieldIndex = schema.getField("date_nullable_field").pos();
+ int tsMillisFieldIndex = schema.getField("timestamp_millis_field").pos();
+ int tsMicrosNullableFieldIndex = schema.getField("timestamp_micros_nullable_field").pos();
+ int tsLocalMillisFieldIndex = schema.getField("timestamp_local_millis_nullable_field").pos();
+ int tsLocalMicrosFieldIndex = schema.getField("timestamp_local_micros_field").pos();
+ for (Row row : rows) {
+ assertEquals(timestamp, row.get(tsMillisFieldIndex));
+ if (!row.isNullAt(tsMicrosNullableFieldIndex)) {
+ assertEquals(timestamp, row.get(tsMicrosNullableFieldIndex));
+ }
+ if (!row.isNullAt(dateFieldIndex)) {
+ assertTrue(row.get(dateFieldIndex) instanceof Date);
+ }
+ if (!row.isNullAt(tsLocalMillisFieldIndex)) {
+ assertEquals(ts, row.get(tsLocalMillisFieldIndex));
+ }
+ assertEquals(ts * 1000L, row.get(tsLocalMicrosFieldIndex));
+ }
+ }
+
+ // Validate that clustering produces decimals in legacy format and lists in newer format. Assert that the unit is correct on the timestamps
+ private void validateTypesAfterClustering(List<HoodieWriteStat> writeStats) {
writeStats.stream().map(writeStat -> new StoragePath(metaClient.getBasePath(), writeStat.getPath())).forEach(writtenPath -> {
MessageType schema = ParquetUtils.readMetadata(storage, writtenPath)
.getFileMetaData().getSchema();
- int index = schema.getFieldIndex("height");
- Type decimalType = schema.getFields().get(index);
+ int decimalFieldIndex = schema.getFieldIndex("decimal_field");
+ Type decimalType = schema.getFields().get(decimalFieldIndex);
Review Comment:
Updated the test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3287046988
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
}, {
"hash" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491",
"triggerID" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"triggerType" : "PUSH"
}, {
"hash" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8503",
"triggerID" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"triggerType" : "PUSH"
}, {
"hash" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8506",
"triggerID" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"triggerType" : "PUSH"
}, {
"hash" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8507",
"triggerID" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"triggerType" : "PUSH"
}, {
"hash" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8510",
"triggerID" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"triggerType" : "PUSH"
}, {
"hash" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d4438bc21cf60af1858e4f0602d91115cc41fced Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8510)
* 73dd1c38e7cc54233a051cb85222f56dd4cf22b3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345030944
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,47 +18,149 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
+import org.apache.hudi.SparkAdapterSupport;
+import org.apache.hudi.avro.HoodieAvroUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
- * Hoodie Write Support for directly writing Row to Parquet.
+ * Hoodie Write Support for directly writing Row to Parquet and adding the Hudi bloom index to the file metadata.
+ * The implementation is adapted from Spark's {@link org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport} but has the following major differences:
+ * <ul>
+ * <li>Decimals are always written with the legacy format to ensure compatibility with parquet-avro and other readers</li>
+ * <li>Writing 2-level or 3-level lists is configurable with either the Hudi option, hoodie.parquet.writelegacyformat.enabled, or the parquet-avro option, parquet.avro.write-old-list-structure
Review Comment:
These are how parquet represents the lists in the schema
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3286325606
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
}, {
"hash" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491",
"triggerID" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"triggerType" : "PUSH"
}, {
"hash" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8503",
"triggerID" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"triggerType" : "PUSH"
}, {
"hash" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8506",
"triggerID" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"triggerType" : "PUSH"
}, {
"hash" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8507",
"triggerID" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d71d18af39bc98ed0eb85f0565967aef8e3fe305 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8507)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345522286
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,47 +18,149 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
+import org.apache.hudi.SparkAdapterSupport;
+import org.apache.hudi.avro.HoodieAvroUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
- * Hoodie Write Support for directly writing Row to Parquet.
+ * Hoodie Write Support for directly writing Row to Parquet and adding the Hudi bloom index to the file metadata.
+ * The implementation is adapted from Spark's {@link org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport} but has the following major differences:
+ * <ul>
+ * <li>Decimals are always written with the legacy format to ensure compatibility with parquet-avro and other readers</li>
+ * <li>Writing 2-level or 3-level lists is configurable with either the Hudi option, hoodie.parquet.writelegacyformat.enabled, or the parquet-avro option, parquet.avro.write-old-list-structure
+ * to ensure consistency across writer paths.</li>
+ * <li>The scale of the timestamps is determined by the Hudi writer schema instead of relying on Spark configuration</li>
+ * </ul>
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
+ private static final Schema MAP_KEY_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final String MAP_REPEATED_NAME = "key_value";
+ private static final String MAP_KEY_NAME = "key";
+ private static final String MAP_VALUE_NAME = "value";
private final Configuration hadoopConf;
private final Option<HoodieBloomFilterWriteSupport<UTF8String>> bloomFilterWriteSupportOpt;
+ private final byte[] decimalBuffer = new byte[Decimal.minBytesForPrecision()[DecimalType.MAX_PRECISION()]];
+ private final Enumeration.Value datetimeRebaseMode = (Enumeration.Value) SparkAdapterSupport.getSparkAdapter().getDateTimeRebaseMode();
+ private final Function1<Object, Object> dateRebaseFunction = DataSourceUtils.createDateRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private final Function1<Object, Object> timestampRebaseFunction = DataSourceUtils.createTimestampRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private RecordConsumer recordConsumer;
+ private final boolean writeLegacyListFormat;
+ private final ValueWriter[] rootFieldWriters;
+ private final Schema avroSchema;
- public HoodieRowParquetWriteSupport(Configuration conf, StructType structType, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
+ public HoodieRowParquetWriteSupport(Configuration conf, StructType schema, Option<BloomFilter> bloomFilterOpt, HoodieWriteConfig config) {
Configuration hadoopConf = new Configuration(conf);
- hadoopConf.set("spark.sql.parquet.writeLegacyFormat", config.getStringOrDefault(HoodieStorageConfig.PARQUET_WRITE_LEGACY_FORMAT_ENABLED, "false"));
+ String writeLegacyFormatEnabled = config.getStringOrDefault(HoodieStorageConfig.PARQUET_WRITE_LEGACY_FORMAT_ENABLED, "false");
+ hadoopConf.set("spark.sql.parquet.writeLegacyFormat", writeLegacyFormatEnabled);
hadoopConf.set("spark.sql.parquet.outputTimestampType", config.getStringOrDefault(HoodieStorageConfig.PARQUET_OUTPUT_TIMESTAMP_TYPE));
hadoopConf.set("spark.sql.parquet.fieldId.write.enabled", config.getStringOrDefault(PARQUET_FIELD_ID_WRITE_ENABLED));
- setSchema(structType, hadoopConf);
-
+ this.writeLegacyListFormat = Boolean.parseBoolean(writeLegacyFormatEnabled)
+ || Boolean.parseBoolean(config.getStringOrDefault(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE, "false"));
+ this.avroSchema = SerDeHelper.fromJson(config.getInternalSchema()).map(internalSchema -> AvroInternalSchemaConverter.convert(internalSchema, "spark_schema"))
+ .orElseGet(() -> {
+ Schema parsedSchema = new Schema.Parser().parse(config.getWriteSchema());
+ if (config.populateMetaFields()) {
+ return HoodieAvroUtils.addMetadataFields(parsedSchema, config.allowOperationMetadataField());
+ }
+ return parsedSchema;
+ });
+ ParquetWriteSupport.setSchema(HoodieInternalRowUtils.getCachedSchema(avroSchema), hadoopConf);
+ this.rootFieldWriters = getFieldWriters(schema, avroSchema);
this.hadoopConf = hadoopConf;
this.bloomFilterWriteSupportOpt = bloomFilterOpt.map(HoodieBloomFilterRowWriteSupport::new);
}
+ private ValueWriter[] getFieldWriters(StructType schema, Schema avroSchema) {
+ Map<String, Integer> fieldNameToIndex = new HashMap<>();
+ for (int i = 0; i < schema.fields().length; i++) {
+ fieldNameToIndex.put(schema.fields()[i].name(), i);
+ }
+ return IntStream.range(0, avroSchema.getFields().size()).mapToObj(i -> {
+ Schema.Field field = avroSchema.getFields().get(i);
+ Integer structIndex = fieldNameToIndex.get(field.name());
+ return makeWriter(field.schema(), schema.fields()[structIndex].dataType());
+ }).toArray(ValueWriter[]::new);
+ }
+
public Configuration getHadoopConf() {
return hadoopConf;
}
+ @Override
+ public WriteContext init(Configuration configuration) {
+ Map<String, String> metadata = new HashMap<>();
+ metadata.put("org.apache.spark.version", VersionUtils.shortVersion(HoodieSparkUtils.getSparkVersion()));
Review Comment:
Do we want to add a custom metadata field here to indicate the Hudi version/write support with which the parquet is written so we know the parquet file is written with `HoodieRowParquetWriteSupport`?
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -73,6 +175,173 @@ public void add(UTF8String recordKey) {
bloomFilterWriteSupport.addKey(recordKey));
}
+ @FunctionalInterface
+ private interface ValueWriter {
+ void write(SpecializedGetters row, int ordinal);
+ }
+
+ private void consumeMessage(Runnable writer) {
+ recordConsumer.startMessage();
+ writer.run();
+ recordConsumer.endMessage();
+ }
+
+ private void consumeGroup(Runnable writer) {
+ recordConsumer.startGroup();
+ writer.run();
+ recordConsumer.endGroup();
+ }
+
+ private void consumeField(String field, int index, Runnable writer) {
+ recordConsumer.startField(field, index);
+ writer.run();
+ recordConsumer.endField(field, index);
+ }
+
+ private void writeFields(InternalRow row, Schema schema, ValueWriter[] fieldWriters) {
+ for (int i = 0; i < fieldWriters.length; i++) {
+ int index = i;
+ if (!row.isNullAt(i)) {
+ Schema.Field field = schema.getFields().get(index);
+ consumeField(field.name(), index, () -> fieldWriters[index].write(row, index));
+ }
+ }
+ }
+
+ private ValueWriter makeWriter(Schema avroSchema, DataType dataType) {
+ Schema resolvedSchema = resolveNullableSchema(avroSchema);
+ Schema.Type type = resolvedSchema.getType();
+ LogicalType logicalType = resolvedSchema.getLogicalType();
+ switch (type) {
+ case BOOLEAN:
+ return (row, ordinal) -> recordConsumer.addBoolean(row.getBoolean(ordinal));
+ case INT:
+ if (logicalType != null) {
+ if (logicalType.getName().equals(LogicalTypes.date().getName())) {
+ return (row, ordinal) -> recordConsumer.addInteger((Integer) dateRebaseFunction.apply(row.getInt(ordinal)));
+ }
+ }
+ return (row, ordinal) -> recordConsumer.addInteger(row.getInt(ordinal));
+ case LONG:
+ if (logicalType != null) {
+ if (logicalType.getName().equals(LogicalTypes.timestampMillis().getName())) {
+ return (row, ordinal) -> recordConsumer.addLong(DateTimeUtils.microsToMillis((long) timestampRebaseFunction.apply(row.getLong(ordinal))));
Review Comment:
Does this assume that the input row uses micros for `TimestampType`?
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -73,6 +175,173 @@ public void add(UTF8String recordKey) {
bloomFilterWriteSupport.addKey(recordKey));
}
+ @FunctionalInterface
+ private interface ValueWriter {
+ void write(SpecializedGetters row, int ordinal);
+ }
+
+ private void consumeMessage(Runnable writer) {
+ recordConsumer.startMessage();
+ writer.run();
+ recordConsumer.endMessage();
+ }
+
+ private void consumeGroup(Runnable writer) {
+ recordConsumer.startGroup();
+ writer.run();
+ recordConsumer.endGroup();
+ }
+
+ private void consumeField(String field, int index, Runnable writer) {
+ recordConsumer.startField(field, index);
+ writer.run();
+ recordConsumer.endField(field, index);
+ }
+
+ private void writeFields(InternalRow row, Schema schema, ValueWriter[] fieldWriters) {
+ for (int i = 0; i < fieldWriters.length; i++) {
+ int index = i;
+ if (!row.isNullAt(i)) {
+ Schema.Field field = schema.getFields().get(index);
+ consumeField(field.name(), index, () -> fieldWriters[index].write(row, index));
+ }
+ }
+ }
+
+ private ValueWriter makeWriter(Schema avroSchema, DataType dataType) {
+ Schema resolvedSchema = resolveNullableSchema(avroSchema);
+ Schema.Type type = resolvedSchema.getType();
+ LogicalType logicalType = resolvedSchema.getLogicalType();
+ switch (type) {
+ case BOOLEAN:
+ return (row, ordinal) -> recordConsumer.addBoolean(row.getBoolean(ordinal));
+ case INT:
Review Comment:
Should `time-micros` be handled too?
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -73,6 +175,173 @@ public void add(UTF8String recordKey) {
bloomFilterWriteSupport.addKey(recordKey));
}
+ @FunctionalInterface
+ private interface ValueWriter {
+ void write(SpecializedGetters row, int ordinal);
+ }
+
+ private void consumeMessage(Runnable writer) {
+ recordConsumer.startMessage();
+ writer.run();
+ recordConsumer.endMessage();
+ }
+
+ private void consumeGroup(Runnable writer) {
+ recordConsumer.startGroup();
+ writer.run();
+ recordConsumer.endGroup();
+ }
+
+ private void consumeField(String field, int index, Runnable writer) {
+ recordConsumer.startField(field, index);
+ writer.run();
+ recordConsumer.endField(field, index);
+ }
+
+ private void writeFields(InternalRow row, Schema schema, ValueWriter[] fieldWriters) {
+ for (int i = 0; i < fieldWriters.length; i++) {
+ int index = i;
+ if (!row.isNullAt(i)) {
+ Schema.Field field = schema.getFields().get(index);
+ consumeField(field.name(), index, () -> fieldWriters[index].write(row, index));
+ }
+ }
+ }
+
+ private ValueWriter makeWriter(Schema avroSchema, DataType dataType) {
+ Schema resolvedSchema = resolveNullableSchema(avroSchema);
+ Schema.Type type = resolvedSchema.getType();
+ LogicalType logicalType = resolvedSchema.getLogicalType();
+ switch (type) {
+ case BOOLEAN:
+ return (row, ordinal) -> recordConsumer.addBoolean(row.getBoolean(ordinal));
+ case INT:
Review Comment:
Based on Avro spec, `time-millis` logical type annotates an Avro `int`, so `time-millis` should be handled here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3287033425
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
}, {
"hash" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491",
"triggerID" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"triggerType" : "PUSH"
}, {
"hash" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8503",
"triggerID" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"triggerType" : "PUSH"
}, {
"hash" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8506",
"triggerID" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"triggerType" : "PUSH"
}, {
"hash" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8507",
"triggerID" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"triggerType" : "PUSH"
}, {
"hash" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d71d18af39bc98ed0eb85f0565967aef8e3fe305 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8507)
* d4438bc21cf60af1858e4f0602d91115cc41fced UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345550062
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -73,6 +175,173 @@ public void add(UTF8String recordKey) {
bloomFilterWriteSupport.addKey(recordKey));
}
+ @FunctionalInterface
+ private interface ValueWriter {
+ void write(SpecializedGetters row, int ordinal);
+ }
+
+ private void consumeMessage(Runnable writer) {
+ recordConsumer.startMessage();
+ writer.run();
+ recordConsumer.endMessage();
+ }
+
+ private void consumeGroup(Runnable writer) {
+ recordConsumer.startGroup();
+ writer.run();
+ recordConsumer.endGroup();
+ }
+
+ private void consumeField(String field, int index, Runnable writer) {
+ recordConsumer.startField(field, index);
+ writer.run();
+ recordConsumer.endField(field, index);
+ }
+
+ private void writeFields(InternalRow row, Schema schema, ValueWriter[] fieldWriters) {
+ for (int i = 0; i < fieldWriters.length; i++) {
+ int index = i;
+ if (!row.isNullAt(i)) {
+ Schema.Field field = schema.getFields().get(index);
+ consumeField(field.name(), index, () -> fieldWriters[index].write(row, index));
+ }
+ }
+ }
+
+ private ValueWriter makeWriter(Schema avroSchema, DataType dataType) {
+ Schema resolvedSchema = resolveNullableSchema(avroSchema);
+ Schema.Type type = resolvedSchema.getType();
+ LogicalType logicalType = resolvedSchema.getLogicalType();
+ switch (type) {
+ case BOOLEAN:
+ return (row, ordinal) -> recordConsumer.addBoolean(row.getBoolean(ordinal));
+ case INT:
Review Comment:
I don't see an equivalent spark type though so I don't think there is a required conversion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3287407508
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
}, {
"hash" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491",
"triggerID" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"triggerType" : "PUSH"
}, {
"hash" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8503",
"triggerID" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"triggerType" : "PUSH"
}, {
"hash" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8506",
"triggerID" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"triggerType" : "PUSH"
}, {
"hash" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8507",
"triggerID" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"triggerType" : "PUSH"
}, {
"hash" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8510",
"triggerID" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"triggerType" : "PUSH"
}, {
"hash" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8511",
"triggerID" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"triggerType" : "PUSH"
}, {
"hash" : "3ff5acf174c775406ffb54be92650b03718bf48b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8513",
"triggerID" : "3ff5acf174c775406ffb54be92650b03718bf48b",
"triggerType" : "PUSH"
}, {
"hash" : "25a0be4048eb5d9e8c082ad7ff1026053a20747c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "25a0be4048eb5d9e8c082ad7ff1026053a20747c",
"triggerType" : "PUSH"
}, {
"hash" : "27b7921b67ae40d374478457fa51b7ffc7336f50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27b7921b67ae40d374478457fa51b7ffc7336f50",
"triggerType" : "PUSH"
}, {
"hash" : "55213c73a0d42e87f0132b15eb793ef33e3561d3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8514",
"triggerID" : "55213c73a0d42e87f0132b15eb793ef33e3561d3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3ff5acf174c775406ffb54be92650b03718bf48b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8513)
* 25a0be4048eb5d9e8c082ad7ff1026053a20747c UNKNOWN
* 27b7921b67ae40d374478457fa51b7ffc7336f50 UNKNOWN
* 55213c73a0d42e87f0132b15eb793ef33e3561d3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8514)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2342351606
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/MultipleSparkJobExecutionStrategy.java:
##########
@@ -109,11 +108,7 @@ public HoodieWriteMetadata<HoodieData<WriteStatus>> performClustering(final Hood
Math.min(clusteringPlan.getInputGroups().size(), writeConfig.getClusteringMaxParallelism()),
new CustomizedThreadFactory("clustering-job-group", true));
try {
- boolean canUseRowWriter = getWriteConfig().getBooleanOrDefault("hoodie.datasource.write.row.writer.enable", true)
- && HoodieDataTypeUtils.canUseRowWriter(schema, engineContext.hadoopConfiguration());
- if (canUseRowWriter) {
- HoodieDataTypeUtils.tryOverrideParquetWriteLegacyFormatProperty(writeConfig.getProps(), schema);
- }
Review Comment:
Yes, this check was mainly for the decimal support in the past and that is all handled in `HoodieRowParquetWriteSupport` now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2342350693
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieInternalRowFileWriterFactory.java:
##########
@@ -54,24 +55,26 @@ public class HoodieInternalRowFileWriterFactory {
public static HoodieInternalRowFileWriter getInternalRowFileWriter(StoragePath path,
HoodieTable hoodieTable,
HoodieWriteConfig writeConfig,
- StructType schema)
+ StructType schema,
+ Schema avroSchema)
Review Comment:
There may be differences on TimestampNTZ fields. Right now these are coming through as `long` instead of a timestamp in the StructType so it was necessary to pass in struct to get this context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3287374298
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
}, {
"hash" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491",
"triggerID" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"triggerType" : "PUSH"
}, {
"hash" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8503",
"triggerID" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"triggerType" : "PUSH"
}, {
"hash" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8506",
"triggerID" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"triggerType" : "PUSH"
}, {
"hash" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8507",
"triggerID" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"triggerType" : "PUSH"
}, {
"hash" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8510",
"triggerID" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"triggerType" : "PUSH"
}, {
"hash" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8511",
"triggerID" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"triggerType" : "PUSH"
}, {
"hash" : "3ff5acf174c775406ffb54be92650b03718bf48b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8513",
"triggerID" : "3ff5acf174c775406ffb54be92650b03718bf48b",
"triggerType" : "PUSH"
}, {
"hash" : "25a0be4048eb5d9e8c082ad7ff1026053a20747c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "25a0be4048eb5d9e8c082ad7ff1026053a20747c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 73dd1c38e7cc54233a051cb85222f56dd4cf22b3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8511)
* 3ff5acf174c775406ffb54be92650b03718bf48b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8513)
* 25a0be4048eb5d9e8c082ad7ff1026053a20747c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3287369590
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
}, {
"hash" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491",
"triggerID" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"triggerType" : "PUSH"
}, {
"hash" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8503",
"triggerID" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"triggerType" : "PUSH"
}, {
"hash" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8506",
"triggerID" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"triggerType" : "PUSH"
}, {
"hash" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8507",
"triggerID" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"triggerType" : "PUSH"
}, {
"hash" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8510",
"triggerID" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"triggerType" : "PUSH"
}, {
"hash" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8511",
"triggerID" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"triggerType" : "PUSH"
}, {
"hash" : "3ff5acf174c775406ffb54be92650b03718bf48b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8513",
"triggerID" : "3ff5acf174c775406ffb54be92650b03718bf48b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 73dd1c38e7cc54233a051cb85222f56dd4cf22b3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8511)
* 3ff5acf174c775406ffb54be92650b03718bf48b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8513)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3285478010
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
}, {
"hash" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491",
"triggerID" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"triggerType" : "PUSH"
}, {
"hash" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 00f82b3c501021529d2fa67b4fb5860eef21d634 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491)
* 2380d2b0bc91be8f86fe87911f7c503d938a072f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3286020699
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
}, {
"hash" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491",
"triggerID" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"triggerType" : "PUSH"
}, {
"hash" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8503",
"triggerID" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"triggerType" : "PUSH"
}, {
"hash" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8506",
"triggerID" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2380d2b0bc91be8f86fe87911f7c503d938a072f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8503)
* 1406c146e1fa7c3e449bfe9e99fbceaafb1ad209 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8506)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #13882:
URL: https://github.com/apache/hudi/pull/13882
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345511529
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,47 +18,149 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
+import org.apache.hudi.SparkAdapterSupport;
+import org.apache.hudi.avro.HoodieAvroUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
- * Hoodie Write Support for directly writing Row to Parquet.
+ * Hoodie Write Support for directly writing Row to Parquet and adding the Hudi bloom index to the file metadata.
+ * The implementation is adapted from Spark's {@link org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport} but has the following major differences:
+ * <ul>
+ * <li>Decimals are always written with the legacy format to ensure compatibility with parquet-avro and other readers</li>
+ * <li>Writing 2-level or 3-level lists is configurable with either the Hudi option, hoodie.parquet.writelegacyformat.enabled, or the parquet-avro option, parquet.avro.write-old-list-structure
+ * to ensure consistency across writer paths.</li>
+ * <li>The scale of the timestamps is determined by the Hudi writer schema instead of relying on Spark configuration</li>
+ * </ul>
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
Review Comment:
The reader will use the parquet schema directly to parse out the values so no changes are required
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345510741
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkAdapterSupport.scala:
##########
@@ -43,4 +43,6 @@ object SparkAdapterSupport {
getClass.getClassLoader.loadClass(adapterClass)
.newInstance().asInstanceOf[SparkAdapter]
}
+
+ def getSparkAdapter: SparkAdapter = sparkAdapter
Review Comment:
Ah! I knew there was a way but couldn't remember
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345511616
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/functional/TestSparkSortAndSizeClustering.java:
##########
@@ -130,24 +155,55 @@ public void writeAndClustering(boolean isRow) throws IOException {
List<Row> rows = readRecords();
assertEquals(numRecords, rows.size());
- validateDecimalTypeAfterClustering(writeStats);
+ validateTypesAfterClustering(writeStats);
+ validateDateAndTimestampFields(rows, ts);
}
- // Validate that clustering produces decimals in legacy format
- private void validateDecimalTypeAfterClustering(List<HoodieWriteStat> writeStats) {
+ private void validateDateAndTimestampFields(List<Row> rows, long ts) {
+ Schema schema = HoodieAvroUtils.addMetadataFields(getSchema(), false);
+ Timestamp timestamp = new Timestamp(ts);
+ int dateFieldIndex = schema.getField("date_nullable_field").pos();
+ int tsMillisFieldIndex = schema.getField("timestamp_millis_field").pos();
+ int tsMicrosNullableFieldIndex = schema.getField("timestamp_micros_nullable_field").pos();
+ int tsLocalMillisFieldIndex = schema.getField("timestamp_local_millis_nullable_field").pos();
+ int tsLocalMicrosFieldIndex = schema.getField("timestamp_local_micros_field").pos();
+ for (Row row : rows) {
+ assertEquals(timestamp, row.get(tsMillisFieldIndex));
+ if (!row.isNullAt(tsMicrosNullableFieldIndex)) {
+ assertEquals(timestamp, row.get(tsMicrosNullableFieldIndex));
+ }
+ if (!row.isNullAt(dateFieldIndex)) {
+ assertTrue(row.get(dateFieldIndex) instanceof Date);
+ }
+ if (!row.isNullAt(tsLocalMillisFieldIndex)) {
+ assertEquals(ts, row.get(tsLocalMillisFieldIndex));
+ }
+ assertEquals(ts * 1000L, row.get(tsLocalMicrosFieldIndex));
+ }
+ }
+
+ // Validate that clustering produces decimals in legacy format and lists in newer format. Assert that the unit is correct on the timestamps
+ private void validateTypesAfterClustering(List<HoodieWriteStat> writeStats) {
Review Comment:
Do we want to validate the types before clustering to make sure the write behavior of Avro and Row writer are aligned, which serves as another validation to make sure any future change in either Avro or Row writer will be caught by this test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345510524
##########
hudi-spark-datasource/hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/adapter/Spark3_5Adapter.scala:
##########
@@ -38,7 +38,7 @@ import org.apache.spark.sql.execution.datasources.orc.Spark35OrcReader
import org.apache.spark.sql.execution.datasources.parquet.{ParquetFileFormat, Spark35LegacyHoodieParquetFileFormat, Spark35ParquetReader}
import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
import org.apache.spark.sql.hudi.analysis.TableValuedFunctions
-import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.internal.{LegacyBehaviorPolicy, SQLConf}
Review Comment:
Yes, the import is different unfortunately
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3286301777
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
}, {
"hash" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491",
"triggerID" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"triggerType" : "PUSH"
}, {
"hash" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8503",
"triggerID" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"triggerType" : "PUSH"
}, {
"hash" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8506",
"triggerID" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"triggerType" : "PUSH"
}, {
"hash" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8507",
"triggerID" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1406c146e1fa7c3e449bfe9e99fbceaafb1ad209 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8506)
* d71d18af39bc98ed0eb85f0565967aef8e3fe305 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8507)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13882:
URL: https://github.com/apache/hudi/pull/13882#discussion_r2345133537
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -18,39 +18,90 @@
package org.apache.hudi.io.storage.row;
+import org.apache.hudi.HoodieSparkUtils;
import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
import org.apache.hudi.common.bloom.BloomFilter;
import org.apache.hudi.common.config.HoodieConfig;
import org.apache.hudi.common.config.HoodieStorageConfig;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.sql.HoodieInternalRowUtils;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.SpecializedGetters;
+import org.apache.spark.sql.catalyst.util.ArrayData;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.catalyst.util.MapData;
+import org.apache.spark.sql.execution.datasources.DataSourceUtils;
import org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport;
+import org.apache.spark.sql.internal.LegacyBehaviorPolicy;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Decimal;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.MapType;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.TimestampNTZType;
import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.VersionUtils;
import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
+import java.util.stream.IntStream;
+import scala.Enumeration;
+import scala.Function1;
+
+import static org.apache.hudi.avro.AvroSchemaUtils.resolveNullableSchema;
import static org.apache.hudi.common.config.HoodieStorageConfig.PARQUET_FIELD_ID_WRITE_ENABLED;
/**
* Hoodie Write Support for directly writing Row to Parquet.
*/
-public class HoodieRowParquetWriteSupport extends ParquetWriteSupport {
+public class HoodieRowParquetWriteSupport extends WriteSupport<InternalRow> {
+ private static final Schema MAP_KEY_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final String MAP_REPEATED_NAME = "key_value";
+ private static final String MAP_KEY_NAME = "key";
+ private static final String MAP_VALUE_NAME = "value";
private final Configuration hadoopConf;
private final Option<HoodieBloomFilterWriteSupport<UTF8String>> bloomFilterWriteSupportOpt;
+ private final byte[] decimalBuffer = new byte[Decimal.minBytesForPrecision()[DecimalType.MAX_PRECISION()]];
+ private final Enumeration.Value datetimeRebaseMode = LegacyBehaviorPolicy.withName(SQLConf.get().getConf(SQLConf.PARQUET_REBASE_MODE_IN_WRITE()));
+ private final Function1<Object, Object> dateRebaseFunction = DataSourceUtils.createDateRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private final Function1<Object, Object> timestampRebaseFunction = DataSourceUtils.createTimestampRebaseFuncInWrite(datetimeRebaseMode, "Parquet");
+ private RecordConsumer recordConsumer;
+ private final boolean writeLegacyListFormat;
+ private final ValueWriter[] rootFieldWriters;
+ private final Schema avroSchema;
- public HoodieRowParquetWriteSupport(Configuration conf, StructType structType, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
+ public HoodieRowParquetWriteSupport(Configuration conf, StructType schema, Schema avroSchema, Option<BloomFilter> bloomFilterOpt, HoodieConfig config) {
Review Comment:
The HoodieWriteConfig class is not available on all of the paths that call this code currently so this is causing some issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Update ParquetWriteSupport for Rows to match Avro writer behavior [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13882:
URL: https://github.com/apache/hudi/pull/13882#issuecomment-3287035047
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8474",
"triggerID" : "e5ea200c5cea9f5c2dda1554b765bfe11e276ae7",
"triggerType" : "PUSH"
}, {
"hash" : "190cbc827b787e7f347a76add30a94f7621e750d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8477",
"triggerID" : "190cbc827b787e7f347a76add30a94f7621e750d",
"triggerType" : "PUSH"
}, {
"hash" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8488",
"triggerID" : "5bca64047318b420b21f7dadd1f274d1330319e1",
"triggerType" : "PUSH"
}, {
"hash" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8491",
"triggerID" : "00f82b3c501021529d2fa67b4fb5860eef21d634",
"triggerType" : "PUSH"
}, {
"hash" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8503",
"triggerID" : "2380d2b0bc91be8f86fe87911f7c503d938a072f",
"triggerType" : "PUSH"
}, {
"hash" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8506",
"triggerID" : "1406c146e1fa7c3e449bfe9e99fbceaafb1ad209",
"triggerType" : "PUSH"
}, {
"hash" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8507",
"triggerID" : "d71d18af39bc98ed0eb85f0565967aef8e3fe305",
"triggerType" : "PUSH"
}, {
"hash" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8510",
"triggerID" : "d4438bc21cf60af1858e4f0602d91115cc41fced",
"triggerType" : "PUSH"
}, {
"hash" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "73dd1c38e7cc54233a051cb85222f56dd4cf22b3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d71d18af39bc98ed0eb85f0565967aef8e3fe305 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8507)
* d4438bc21cf60af1858e4f0602d91115cc41fced Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8510)
* 73dd1c38e7cc54233a051cb85222f56dd4cf22b3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org