You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2024/07/17 20:26:58 UTC

[I] Investigate options for improving performance when reading decimals from Parquet [datafusion-comet]

andygrove opened a new issue, #679:
URL: https://github.com/apache/datafusion-comet/issues/679

   ### What is the problem the feature request solves?
   
   This issue is for discussing the comment from @parthchandra in https://github.com/apache/datafusion-comet/pull/671#issuecomment-2234180553.
   
   > Just looking at this one case, with decimal fields and only scan enabled, we are much slower. This is consistent with something I saw when working on the parallel reader.
   > From a profiling run I saw that a potential bottleneck was [BosonVector.getDecimal](https://github.com/apache/datafusion-comet/blob/7ac2fb9f6672fbb29c2e5de7e62b457efc0bfebf/common/src/main/java/org/apache/comet/vector/CometVector.java#L74) which has an expensive creation of a BigInteger followed by an expensive creation of a BigDecimal.
   > However, this path would be hit only for precision > 18 or if spark.comet.use.decimal128 was set to true (it is false by default).
   > Also, I'm not sure if there is a way to eliminate this though.
   
   
   ### Describe the potential solution
   
   _No response_
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate options for improving performance when reading decimals from Parquet [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #679:
URL: https://github.com/apache/datafusion-comet/issues/679#issuecomment-2234272725

   ![Screenshot from 2024-07-17 14-59-18](https://github.com/user-attachments/assets/85af2838-98c0-452f-a9c7-e3bda4887160)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate options for improving performance when reading decimals from Parquet [datafusion-comet]

Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on issue #679:
URL: https://github.com/apache/datafusion-comet/issues/679#issuecomment-2240962098

   It looks like pure scan is not a problem based on profiling
   ### Baseline
   ![Screenshot 2024-07-19 at 10 49 16 PM](https://github.com/user-attachments/assets/1f2bd34b-fe3c-453c-a023-ed087f6368de)
   ### Scan-only enabled
   ![Screenshot 2024-07-19 at 10 48 59 PM](https://github.com/user-attachments/assets/059a529d-4aae-4d3a-a69b-a23c5aef45c2)
   
   ### Scan comparison (the name says add_many_decimals, but I was just lazy to change the name)
   ![Screenshot 2024-07-19 at 11 28 02 PM](https://github.com/user-attachments/assets/8e389460-4c07-4c8b-a7fa-e5ffc157e8a9)
   
   Now I remember that we need to increase the minimum number of iteration to get a stable results. We used to use a remote Linux machine that was stable so 2-3 iterations were fine. But for local testing, we need to increase it. For the third screen shot, I used 33 iterations.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate options for improving performance when reading decimals from Parquet [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on issue #679:
URL: https://github.com/apache/datafusion-comet/issues/679#issuecomment-2234421589

   This seemingly points the finger at `Decimal.createUnsafe`. But I fail to see  how that is taking so much time. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate options for improving performance when reading decimals from Parquet [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on issue #679:
URL: https://github.com/apache/datafusion-comet/issues/679#issuecomment-2249216481

   I see different results in profiling. I ran a simple query - `select ss_net_profit from store_sales` for a **100** iterations with `useDecimal128` enabled and see the following -
   <img width="1591" alt="Screenshot 2024-07-24 at 5 48 23 PM" src="https://github.com/user-attachments/assets/dc6633db-eef6-487c-a8dc-869348ce10e9">
   
   What stands out is that the bulk of the time is being spent in the `comet::parquet::read::values::<impl comet::parquet::read::PlainDecoding for comet::parquet::data_type::Int32DecimalType>::decode` 
   Within this method the main time consumers (as a percentage of time spent in cpu) are 
     `core::slice::<impl [T]>::fill`   - 16.76%
     `comet::common::bit::memcpy`  - 7.07%
     `core::slice::<impl [T]>::fill` - 5.18% (second code path)
   
   Overall Comet is 0.4x of Spark.
     
     I made a change to `comet::common::bit::memcpy` to use `copy_nonoverlapped` which is unsafe and see a 25% improvement.  (After the change, Comet is 0.5x of Spark)
     
     However I don't know the best way to avoid the `slice.fill` calls without voiding the warranty. I'm looking at [MaybeUninit](https://doc.rust-lang.org/std/mem/union.MaybeUninit.html), but the documentation quite rightly warns of there being dragons. 
   
    Also, with `useDecimal128` disabled, we are slower than Spark because we treat the value a Decimal irrespective of precision. Spark reads and processes the value as `Int` 
    A minor change to Comet results in Comet being 1.2x of Spark for this query with `useDecimal128` disabled.
    I'll post a PR after some testing.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate options for improving performance when reading decimals from Parquet [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #679:
URL: https://github.com/apache/datafusion-comet/issues/679#issuecomment-2237341435

   `createUnsafe` takes `Long` object. Maybe boxing also takes some time?
   
   `def createUnsafe(unscaled: Long, precision: Int, scale: Int): Decimal`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate options for improving performance when reading decimals from Parquet [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #679:
URL: https://github.com/apache/datafusion-comet/issues/679#issuecomment-2234367885

   Here is the full flamegraph. The hash aggregate is running in Spark, so maybe it is not surprising that so much time is in calls to getDecimal
   
   ![Screenshot from 2024-07-17 15-39-08](https://github.com/user-attachments/assets/2eecfba6-727a-46ad-9e10-1db234691eb8)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate options for improving performance when reading decimals from Parquet [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #679:
URL: https://github.com/apache/datafusion-comet/issues/679#issuecomment-2237336876

   > It might be that the assignment to `Decimal.longVal` (or `Decimal.intVal`) in `Decimal.createUnsafe` is copying data from native to Jvm memory and that has a performance issue. But there may be no way to overcome this other than to use off-heap memory in the JVM side.
   
   Comet first reads the integer value by calling `getInt`. If any data copying is significant between off-heap and heap memory, it should be in this call. But you can see from the flame graph that `getInt` uses much less time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate options for improving performance when reading decimals from Parquet [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #679:
URL: https://github.com/apache/datafusion-comet/issues/679#issuecomment-2234380204

   This is the flamegraph where Comet is performing scan + exec. Because the hash aggregate is operating on Arrow data, we no longer see lots of decimals being accessed from JVM code.
   
   ![Screenshot from 2024-07-17 15-47-29](https://github.com/user-attachments/assets/1e9c8b87-be0f-4e0e-9ab3-7c53f6130f75)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate options for improving performance when reading decimals from Parquet [datafusion-comet]

Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura closed issue #679: Investigate options for improving performance when reading decimals from Parquet
URL: https://github.com/apache/datafusion-comet/issues/679


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate options for improving performance when reading decimals from Parquet [datafusion-comet]

Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on issue #679:
URL: https://github.com/apache/datafusion-comet/issues/679#issuecomment-2294414453

   We created many fixes. closing for now


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Investigate options for improving performance when reading decimals from Parquet [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on issue #679:
URL: https://github.com/apache/datafusion-comet/issues/679#issuecomment-2234431447

   It might be that the assignment to `Decimal.longVal` (or `Decimal.intVal`) in `Decimal.createUnsafe` is copying data from native to Jvm memory and that has a performance issue. 
   But there may be no way to overcome this other than to use off-heap memory in the JVM side. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org