[jira] [Commented] (HUDI-6824) Make sure serialization of log blocks is language and architecture independent

["Ethan Guo (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HUDI-6824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17764444#comment-17764444 ] 

Ethan Guo commented on HUDI-6824:
---------------------------------

Scope of Concerns
 - Endianness - the order in which the bytes of a multi-byte value are stored
 - Serialized bytes that may change across platforms, language, architecture, machine, etc.

Log Format
 - Serialization are done by the HoodieLogFormatWriter , which uses Hadoop's FSDataOutputStream extending Java's DataOutputStream , using big-endian only
 - The following calls are used
 -- outputStream.writeLong : Writes an int to the underlying output stream as four bytes, high byte first.
 -- outputStream.writeInt : Writes a long to the underlying output stream as eight bytes, high byte first.
- Log block header and footer are turned into byte array from String value of each type
-- HoodieLogBlock.getLogMetadataBytes 
-- String#getBytes() is used for encoding, using the platform's default charset. The default character encoding scheme on Windows is ANSI, while the default character encoding scheme on Linux is UTF-8. This should be fixed to use a predetermined charset.
- Content bytes of each log block type
-- HoodieDataBlock
--- Avro/CDC: serialized using Avro's BinaryEncoder and GenericDatumWriter<IndexedRecord> . Version, size info are written with outputStream.writeInt
--- Parquet: serialized using parquet writer HoodieParquetStreamWriter or HoodieSparkParquetStreamWriter (using HoodieRowParquetWriteSupport )
--- HFile: serialized using HFile writer HFile.Writer . Each record value is serialized by Avro (`HoodieAvroUtils.indexedRecordToBytes`). Schema is serialized by String.getBytes() 
-- HoodieDeleteBlock
--- delete records are serialized using Avro's DatumWriter<HoodieDeleteRecordList> and BinaryEncoder 
-- HoodieCommandBlock
--- no content
- Log Block metadata
-- INSTANT_TIME : String in instant time format
-- TARGET_INSTANT_TIME : String in instant time format
-- SCHEMA : Avro schema in JSON String
-- COMMAND_BLOCK_TYPE : Integer/id in String, representing the type in HoodieCommandBlockTypeEnum 
-- COMPACTED_BLOCK_TIMES : Comma-separated list of instant times
-- RECORD_POSITIONS : String of Base64-encoded bytes after using portable serialization on Roaring64NavigableMap 
-- BLOCK_SEQUENCE_NUMBER : writeAttemptNumber,blockSequenceNumber

Metadata in Base File
- Bloom Filter in parquet footer
 -- Record key in Java String is encoded to bytes using UTF_8
 -- Simple bloom filter: Base64 encoded String from byte array; the byte array is generated by using Hadoop BloomFilter's serialization (`SimpleBloomFilter#serializeToString`)
 -- Dynamic bounded bloom filter: a few meta info encoded in integers followed by multiple bloom filters, each of which is a Base64 encoded String from byte array; the byte array is generated by using Hadoop BloomFilter's serialization (`HoodieDynamicBoundedBloomFilter#serializeToString`)
 -- The Hadoop BloomFilter is merely using Java data structures and we can copy the code out if we want to remove the "dependency" on Hadoop BloomFilter

MDT
- Bloom Filter (`HoodieMetadataPayload.createBloomFilterMetadataRecord`)
-- Bloom filters are read out to ByteBuffer and used to create metadata payload, storing byte array serialized by Avro
- Column Stats (`HoodieMetadataPayload.createColumnStatsRecords`)
-- Min and max values in Java type are wrapped by Avro Wrapper and serialized with Avro

> Make sure serialization of log blocks is language and architecture independent
> ------------------------------------------------------------------------------
>
>                 Key: HUDI-6824
>                 URL: https://issues.apache.org/jira/browse/HUDI-6824
>             Project: Apache Hudi
>          Issue Type: Task
>            Reporter: Ethan Guo
>            Assignee: Ethan Guo
>            Priority: Blocker
>             Fix For: 1.0.0
>
>
> Serialization of log blocks and other information to bytes should be language and architecture independent, e.g., there should be no issue around big-endian and little-endian.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)