You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@orc.apache.org by "cxzl25 (via GitHub)" <gi...@apache.org> on 2024/04/24 11:07:19 UTC
[PR] ORC-1700: Write parquet decimal type data in Benchmark using `FIXED_LEN_BYTE_ARRAY` type [orc]
cxzl25 opened a new pull request, #1910:
URL: https://github.com/apache/orc/pull/1910
### What changes were proposed in this pull request?
This PR aims to write parquet decimal type data in Benchmark using `FIXED_LEN_BYTE_ARRAY` type.
### Why are the changes needed?
Because the decimal type of the parquet file generated now corresponds to the binary type of parquet, but Spark3.5.1 does not support reading.
Spark 3.5.1 supports reading if using the `FIXED_LEN_BYTE_ARRAY` type.
main
```
optional binary fare_amount (DECIMAL(8,2));
```
PR
```
optional fixed_len_byte_array(5) fare_amount (DECIMAL(10,2));
```
```bash
java -jar spark/target/orc-benchmarks-spark-2.1.0-SNAPSHOT.jar spark data -format=parquet -compress zstd -data taxi
```
```java
org.apache.spark.sql.execution.datasources.SchemaColumnConvertNotSupportedException: column: [fare_amount], physicalType: BINARY, logicalType: decimal(8,2)
at org.apache.spark.sql.execution.datasources.parquet.ParquetVectorUpdaterFactory.constructConvertNotSupportedException(ParquetVectorUpdaterFactory.java:1136)
at org.apache.spark.sql.execution.datasources.parquet.ParquetVectorUpdaterFactory.getUpdater(ParquetVectorUpdaterFactory.java:199)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readBatch(VectorizedColumnReader.java:175)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:342)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextKeyValue(VectorizedParquetRecordReader.java:233)
at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
at org.apache.orc.bench.spark.SparkBenchmark.processReader(SparkBenchmark.java:170)
at org.apache.orc.bench.spark.SparkBenchmark.fullRead(SparkBenchmark.java:216)
at org.apache.orc.bench.spark.jmh_generated.SparkBenchmark_fullRead_jmhTest.fullRead_avgt_jmhStub(SparkBenchmark_fullRead_jmhTest.java:219)
```
### How was this patch tested?
local test
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1700: Write parquet decimal type data in Benchmark using `FIXED_LEN_BYTE_ARRAY` type [orc]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #1910:
URL: https://github.com/apache/orc/pull/1910#issuecomment-2109612201
Could you please check if an email sent from private@orc.apache.org is accidently moved to the spam folder? @cxzl25
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1700: Write parquet decimal type data in Benchmark using `FIXED_LEN_BYTE_ARRAY` type [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #1910:
URL: https://github.com/apache/orc/pull/1910#discussion_r1581243641
##########
java/bench/core/src/resources/taxi.schema:
##########
@@ -9,13 +9,13 @@ struct<
PULocationID: bigint,
DOLocationID: bigint,
payment_type: bigint,
- fare_amount: decimal(8,2),
- extra: decimal(8,2),
- mta_tax: decimal(8,2),
- tip_amount: decimal(8,2),
- tolls_amount: decimal(8,2),
- improvement_surcharge: decimal(8,2),
- total_amount: decimal(8,2),
+ fare_amount: decimal(10,2),
Review Comment:
Good to know. Do we happen to know which SPARK JIRA is related to that change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1700: Write parquet decimal type data in Benchmark using `FIXED_LEN_BYTE_ARRAY` type [orc]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on code in PR #1910:
URL: https://github.com/apache/orc/pull/1910#discussion_r1582038081
##########
java/bench/core/src/resources/taxi.schema:
##########
@@ -9,13 +9,13 @@ struct<
PULocationID: bigint,
DOLocationID: bigint,
payment_type: bigint,
- fare_amount: decimal(8,2),
- extra: decimal(8,2),
- mta_tax: decimal(8,2),
- tip_amount: decimal(8,2),
- tolls_amount: decimal(8,2),
- improvement_surcharge: decimal(8,2),
- total_amount: decimal(8,2),
+ fare_amount: decimal(10,2),
Review Comment:
The logic written in this may not actually be related to Spark, because Spark will convert out-of-range values into NULL.
```sql
SELECT CAST(82599861 AS decimal(8, 2))
```
```
NULL
```
If in this PR, the taxi.schema is not modified, the implementation is written directly, and there is no cast involved at this time, so it will fail to write.
```
java -jar core/target/orc-benchmarks-core-*-uber.jar generate data -format parquet -data taxi -compress snappy
```
```java
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: arraycopy: destination index -1 out of bounds for byte[4]
at java.base/java.lang.System.arraycopy(Native Method)
at org.apache.orc.bench.core.convert.avro.AvroWriter.decimalToBinary(AvroWriter.java:381)
at org.apache.orc.bench.core.convert.avro.AvroWriter$DecimalConverter.convert(AvroWriter.java:182)
at org.apache.orc.bench.core.convert.parquet.ParquetWriter.writeBatch(ParquetWriter.java:78)
at org.apache.orc.bench.core.convert.GenerateVariants.run(GenerateVariants.java:157)
at org.apache.orc.bench.core.Driver.main(Driver.java:64)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1700: Write parquet decimal type data in Benchmark using `FIXED_LEN_BYTE_ARRAY` type [orc]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #1910:
URL: https://github.com/apache/orc/pull/1910#issuecomment-2080342459
Should we use INT32 and INT64 for decimals where applicable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1700: Write parquet decimal type data in Benchmark using `FIXED_LEN_BYTE_ARRAY` type [orc]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on PR #1910:
URL: https://github.com/apache/orc/pull/1910#issuecomment-2109637488
> Could you please check if an email sent from [private@orc.apache.org](mailto:private@orc.apache.org) is accidently moved to the spam folder
Wow, I did miss this email, thank you @wgtmac so much for inviting me, and thank you @dongjoon-hyun so much to for commenting and merging multiple times, and to the entire ORC community!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1700: Write parquet decimal type data in Benchmark using `FIXED_LEN_BYTE_ARRAY` type [orc]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on PR #1910:
URL: https://github.com/apache/orc/pull/1910#issuecomment-2081354990
> Should we use INT32 and INT64 for decimals where applicable?
Yes, Spark does this by default. It provides an option `spark.sql.parquet.writeLegacyFormat=true` to achieve alignment with Hive writing decimal method.
https://github.com/apache/spark/blob/8b8ea60bd4f22ea5763a77bac2d51f25d2479be9/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetWriteSupport.scala#L328-L339
```java
writeLegacyParquetFormat match {
// Standard mode, 1 <= precision <= 9, writes as INT32
case false if precision <= Decimal.MAX_INT_DIGITS => int32Writer
// Standard mode, 10 <= precision <= 18, writes as INT64
case false if precision <= Decimal.MAX_LONG_DIGITS => int64Writer
// Legacy mode, 1 <= precision <= 18, writes as FIXED_LEN_BYTE_ARRAY
case true if precision <= Decimal.MAX_LONG_DIGITS => binaryWriterUsingUnscaledLong
// Either standard or legacy mode, 19 <= precision <= 38, writes as FIXED_LEN_BYTE_ARRAY
case _ => binaryWriterUsingUnscaledBytes
```
https://github.com/apache/hive/blob/4614ce72a7f366674d89a3a78f687e419400cb89/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/DataWritableWriter.java#L568-L578
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1700: Write parquet decimal type data in Benchmark using `FIXED_LEN_BYTE_ARRAY` type [orc]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on PR #1910:
URL: https://github.com/apache/orc/pull/1910#issuecomment-2074820310
BINARY type promotion can be supported in Spark 4.0.0, and the test using 4.0.0 SNAPSHOT can work in #1909
[[SPARK-40876](https://issues.apache.org/jira/browse/SPARK-40876)][SQL] Widening type promotions in Parquet readers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1700: Write parquet decimal type data in Benchmark using `FIXED_LEN_BYTE_ARRAY` type [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #1910: ORC-1700: Write parquet decimal type data in Benchmark using `FIXED_LEN_BYTE_ARRAY` type
URL: https://github.com/apache/orc/pull/1910
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1700: Write parquet decimal type data in Benchmark using `FIXED_LEN_BYTE_ARRAY` type [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1910:
URL: https://github.com/apache/orc/pull/1910#issuecomment-2218456637
Sorry for the long delay. Thank you, @cxzl25 and @wgtmac . :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1700: Write parquet decimal type data in Benchmark using `FIXED_LEN_BYTE_ARRAY` type [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1910:
URL: https://github.com/apache/orc/pull/1910#issuecomment-2218457205
Merged to main/2.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1700: Write parquet decimal type data in Benchmark using `FIXED_LEN_BYTE_ARRAY` type [orc]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on code in PR #1910:
URL: https://github.com/apache/orc/pull/1910#discussion_r1577706424
##########
java/bench/core/src/resources/taxi.schema:
##########
@@ -9,13 +9,13 @@ struct<
PULocationID: bigint,
DOLocationID: bigint,
payment_type: bigint,
- fare_amount: decimal(8,2),
- extra: decimal(8,2),
- mta_tax: decimal(8,2),
- tip_amount: decimal(8,2),
- tolls_amount: decimal(8,2),
- improvement_surcharge: decimal(8,2),
- total_amount: decimal(8,2),
+ fare_amount: decimal(10,2),
Review Comment:
Testing locally I found that there is such data as 82599861, which causes the decimal write to fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org