You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2024/07/17 20:26:58 UTC
[I] Investigate options for improving performance when reading decimals from Parquet [datafusion-comet]
andygrove opened a new issue, #679:
URL: https://github.com/apache/datafusion-comet/issues/679
### What is the problem the feature request solves?
This issue is for discussing the comment from @parthchandra in https://github.com/apache/datafusion-comet/pull/671#issuecomment-2234180553.
> Just looking at this one case, with decimal fields and only scan enabled, we are much slower. This is consistent with something I saw when working on the parallel reader.
> From a profiling run I saw that a potential bottleneck was [BosonVector.getDecimal](https://github.com/apache/datafusion-comet/blob/7ac2fb9f6672fbb29c2e5de7e62b457efc0bfebf/common/src/main/java/org/apache/comet/vector/CometVector.java#L74) which has an expensive creation of a BigInteger followed by an expensive creation of a BigDecimal.
> However, this path would be hit only for precision > 18 or if spark.comet.use.decimal128 was set to true (it is false by default).
> Also, I'm not sure if there is a way to eliminate this though.
### Describe the potential solution
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate options for improving performance when reading decimals from Parquet [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #679:
URL: https://github.com/apache/datafusion-comet/issues/679#issuecomment-2234272725

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate options for improving performance when reading decimals from Parquet [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on issue #679:
URL: https://github.com/apache/datafusion-comet/issues/679#issuecomment-2240962098
It looks like pure scan is not a problem based on profiling
### Baseline

### Scan-only enabled

### Scan comparison (the name says add_many_decimals, but I was just lazy to change the name)

Now I remember that we need to increase the minimum number of iteration to get a stable results. We used to use a remote Linux machine that was stable so 2-3 iterations were fine. But for local testing, we need to increase it. For the third screen shot, I used 33 iterations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate options for improving performance when reading decimals from Parquet [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on issue #679:
URL: https://github.com/apache/datafusion-comet/issues/679#issuecomment-2234421589
This seemingly points the finger at `Decimal.createUnsafe`. But I fail to see how that is taking so much time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate options for improving performance when reading decimals from Parquet [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on issue #679:
URL: https://github.com/apache/datafusion-comet/issues/679#issuecomment-2249216481
I see different results in profiling. I ran a simple query - `select ss_net_profit from store_sales` for a **100** iterations with `useDecimal128` enabled and see the following -
<img width="1591" alt="Screenshot 2024-07-24 at 5 48 23 PM" src="https://github.com/user-attachments/assets/dc6633db-eef6-487c-a8dc-869348ce10e9">
What stands out is that the bulk of the time is being spent in the `comet::parquet::read::values::<impl comet::parquet::read::PlainDecoding for comet::parquet::data_type::Int32DecimalType>::decode`
Within this method the main time consumers (as a percentage of time spent in cpu) are
`core::slice::<impl [T]>::fill` - 16.76%
`comet::common::bit::memcpy` - 7.07%
`core::slice::<impl [T]>::fill` - 5.18% (second code path)
Overall Comet is 0.4x of Spark.
I made a change to `comet::common::bit::memcpy` to use `copy_nonoverlapped` which is unsafe and see a 25% improvement. (After the change, Comet is 0.5x of Spark)
However I don't know the best way to avoid the `slice.fill` calls without voiding the warranty. I'm looking at [MaybeUninit](https://doc.rust-lang.org/std/mem/union.MaybeUninit.html), but the documentation quite rightly warns of there being dragons.
Also, with `useDecimal128` disabled, we are slower than Spark because we treat the value a Decimal irrespective of precision. Spark reads and processes the value as `Int`
A minor change to Comet results in Comet being 1.2x of Spark for this query with `useDecimal128` disabled.
I'll post a PR after some testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate options for improving performance when reading decimals from Parquet [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #679:
URL: https://github.com/apache/datafusion-comet/issues/679#issuecomment-2237341435
`createUnsafe` takes `Long` object. Maybe boxing also takes some time?
`def createUnsafe(unscaled: Long, precision: Int, scale: Int): Decimal`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate options for improving performance when reading decimals from Parquet [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #679:
URL: https://github.com/apache/datafusion-comet/issues/679#issuecomment-2234367885
Here is the full flamegraph. The hash aggregate is running in Spark, so maybe it is not surprising that so much time is in calls to getDecimal

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate options for improving performance when reading decimals from Parquet [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #679:
URL: https://github.com/apache/datafusion-comet/issues/679#issuecomment-2237336876
> It might be that the assignment to `Decimal.longVal` (or `Decimal.intVal`) in `Decimal.createUnsafe` is copying data from native to Jvm memory and that has a performance issue. But there may be no way to overcome this other than to use off-heap memory in the JVM side.
Comet first reads the integer value by calling `getInt`. If any data copying is significant between off-heap and heap memory, it should be in this call. But you can see from the flame graph that `getInt` uses much less time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate options for improving performance when reading decimals from Parquet [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #679:
URL: https://github.com/apache/datafusion-comet/issues/679#issuecomment-2234380204
This is the flamegraph where Comet is performing scan + exec. Because the hash aggregate is operating on Arrow data, we no longer see lots of decimals being accessed from JVM code.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate options for improving performance when reading decimals from Parquet [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura closed issue #679: Investigate options for improving performance when reading decimals from Parquet
URL: https://github.com/apache/datafusion-comet/issues/679
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate options for improving performance when reading decimals from Parquet [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on issue #679:
URL: https://github.com/apache/datafusion-comet/issues/679#issuecomment-2294414453
We created many fixes. closing for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Investigate options for improving performance when reading decimals from Parquet [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on issue #679:
URL: https://github.com/apache/datafusion-comet/issues/679#issuecomment-2234431447
It might be that the assignment to `Decimal.longVal` (or `Decimal.intVal`) in `Decimal.createUnsafe` is copying data from native to Jvm memory and that has a performance issue.
But there may be no way to overcome this other than to use off-heap memory in the JVM side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org