You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "karuppuchamysuresh (via GitHub)" <gi...@apache.org> on 2026/01/30 07:16:17 UTC

[PR] docs: update data_types.md to reflect current Arrow type mappings [datafusion]

karuppuchamysuresh opened a new pull request, #20072:
URL: https://github.com/apache/datafusion/pull/20072

   ## Which issue does this PR close?                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                        
     - Closes #18314                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                        
     ## Rationale for this change                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                        
     The documentation in `data_types.md` was outdated and showed `Utf8` as the default mapping for character types (CHAR, VARCHAR, TEXT, STRING), but the current implementation defaults to `Utf8View`. This caused confusion for users reading the   
     documentation as it didn't match the actual behavior.                                                                                                                                                                                              
                                                                                                                                                                                                                                                        
     Additionally, the "Supported Arrow Types" section at the end was redundant since `arrow_typeof` now supports all Arrow types, making the comprehensive list unnecessary.                                                                           
                                                                                                                                                                                                                                                        
     ## What changes are included in this PR?                                                                                                                                                                                                           
                                                                                                                                                                                                                                                        
     1. **Updated Character Types table**: Changed the Arrow DataType column from `Utf8` to `Utf8View` for CHAR, VARCHAR, TEXT, and STRING types                                                                                                        
     2. **Added configuration note**: Documented the `datafusion.sql_parser.map_string_types_to_utf8view` setting that allows users to switch back to `Utf8` if needed                                                                                  
     3. **Removed outdated section**: Deleted the "Supported Arrow Types" section (39 lines) as it's no longer necessary                                                                                                                                
                                                                                                                                                                                                                                                        
     ## Are these changes tested?                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                        
     This is a documentation-only change. The documentation accurately reflects the current behavior of DataFusion:                                                                                                                                     
     - The default mapping to `Utf8View` is the current implementation behavior                                                                                                                                                                         
     - The `datafusion.sql_parser.map_string_types_to_utf8view` configuration option exists and works as documented                                                                                                                                     
                                                                                                                                                                                                                                                        
     ## Are there any user-facing changes?                                                                                                                                                                                                              
                                                                                                                                                                                                                                                        
     Yes, documentation changes only. Users will now see accurate information about:                                                                                                                                                                    
     - The correct default Arrow type mappings for character types                                                                                                                                                                                      
     - How to configure the string type mapping behavior if they need the old `Utf8` behavior 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] docs: update data_types.md to reflect current Arrow type mappings [datafusion]

Posted by "karuppuchamysuresh (via GitHub)" <gi...@apache.org>.
karuppuchamysuresh commented on PR #20072:
URL: https://github.com/apache/datafusion/pull/20072#issuecomment-3824245175

   Thanks for the review, @Jefffrey! I've addressed your feedback about Decimal256 and reviewed the entire page for accuracy.                                                                                                                         
                                                                                                                                                                                                                                                        
     Changes Made                                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                        
     Added explicit documentation for Decimal256 in the Numeric Types table:                                                                                                                                                              
                                                                                                                                                                                                                                                        
     I've split the DECIMAL mapping into two separate table rows to make it clear that both Decimal128 and Decimal256 are supported:                                                                                                                    
                                                                                                                                                                                                                                                        
     | `DECIMAL(precision, scale)` where precision ≤ 38 | `Decimal128(precision, scale)` |                                                                                                                                                              
     | `DECIMAL(precision, scale)` where precision > 38 | `Decimal256(precision, scale)` |                                                                                                                                                              
                                                                                                                                                                                                                                                        
     The maximum supported precision for `DECIMAL` types is 76.                                                                                                                                                                                         
                                                                                                                                                                                                                                                        
     This is based on the implementation in datafusion/sql/src/utils.rs:310-316, which uses:                                                                                                                                                            
     - Decimal128 for precision ≤ 38 (DECIMAL128_MAX_PRECISION)                                                                                                                                                                                         
     - Decimal256 for precision 39-76 (DECIMAL256_MAX_PRECISION)                                                                                                                                                                                        
                                                                                                                                                                                                                                                        
     The split into two rows makes it immediately visible to users that both types are supported, rather than requiring them to read a note below the table.                                                                                            
                                                                                                                                                                                                                                                        
     Verification of Other Type Mappings                                                                                                                                                                                                                
                                                                                                                                                                                                                                                        
     I also verified the accuracy of all other type mappings in the document:                                                                                                                                                                           
                                                                                                                                                                                                                                                        
     ✅ Numeric Types: All integer and float mappings are correct                                                                                                                                                                                       
     ✅ Date/Time Types: Verified against datafusion/sql/src/planner.rs:                                                                                                                                                                                
     - DATE → Date32 (line 711)                                                                                                                                                                                                                         
     - TIME → Time64(Nanosecond) (lines 712-721)                                                                                                                                                                                                        
     - TIMESTAMP → Timestamp(Nanosecond, None) (lines 688-710)                                                                                                                                                                                          
     - INTERVAL → Interval(MonthDayNano) (line 738)                                                                                                                                                                                                     
                                                                                                                                                                                                                                                        
     ✅ Binary Types: BYTEA → Binary (line 733)                                                                                                                                                                                                         
     ✅ Boolean Types: BOOLEAN → Boolean (line 637)                                                                                                                                                                                                     
     ✅ Character Types: Already updated in this PR to Utf8View                                                                                                                                                                                         
                                                                                                                                                                                                                                                        
     All mappings now accurately reflect the current implementation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] docs: update data_types.md to reflect current Arrow type mappings [datafusion]

Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead merged PR #20072:
URL: https://github.com/apache/datafusion/pull/20072


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org