You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/08/20 00:59:57 UTC
(datafusion) branch main updated: docs: Document missing SQL data types (#24471)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion.git
The following commit(s) were added to refs/heads/main by this push:
new 733b195726 docs: Document missing SQL data types (#24471)
733b195726 is described below
commit 733b195726e0c19007893e8a543aa7c8e7367026
Author: Nuno Faria <nu...@gmail.com>
AuthorDate: Thu Aug 20 00:50:40 2026 +0000
docs: Document missing SQL data types (#24471)
## Which issue does this PR close?
- N/A.
## Rationale for this change
The data types page
(https://datafusion.apache.org/user-guide/sql/data_types.html) is quite
outdated, e.g., still claiming that `ARRAY` is not supported. This PR
updates the missing types and also adds missing aliases.
This should only be merged after #24398.
## What changes are included in this PR?
- Add missing types, like `ARRAY`, `STRUCT`, and `MAP`, as well as alias
for existing types.
- Updated the sqlparser-rs link, which was pointing to the wrong line.
## Are these changes tested?
Documentation only.
## Are there any user-facing changes?
No.
---
docs/source/user-guide/sql/data_types.md | 81 ++++++++++++++++++++------------
1 file changed, 51 insertions(+), 30 deletions(-)
diff --git a/docs/source/user-guide/sql/data_types.md b/docs/source/user-guide/sql/data_types.md
index 502193df41..2e75fce6c7 100644
--- a/docs/source/user-guide/sql/data_types.md
+++ b/docs/source/user-guide/sql/data_types.md
@@ -21,7 +21,7 @@
DataFusion uses Arrow, and thus the Arrow type system, for query
execution. The SQL types from
-[sqlparser-rs](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/src/ast/data_type.rs#L27)
+[sqlparser-rs](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/src/ast/data_type.rs#L49)
are mapped to [Arrow data types](https://docs.rs/arrow/latest/arrow/datatypes/enum.DataType.html) according to the following table.
This mapping occurs when defining the schema in a `CREATE EXTERNAL TABLE` command or when performing a SQL `CAST` operation.
@@ -67,32 +67,39 @@ select arrow_cast(now(), 'Timestamp(Second, None)') as "now()";
## Character Types
-| SQL DataType | Arrow DataType |
-| ------------ | -------------- |
-| `CHAR` | `Utf8View` |
-| `VARCHAR` | `Utf8View` |
-| `TEXT` | `Utf8View` |
-| `STRING` | `Utf8View` |
+| SQL DataType | Arrow DataType |
+| ------------------- | -------------- |
+| `CHAR` | `Utf8View` |
+| `VARCHAR` | `Utf8View` |
+| `TEXT` | `Utf8View` |
+| `STRING` | `Utf8View` |
+| `CHARACTER VARYING` | `Utf8View` |
+| `CHAR VARYING` | `Utf8View` |
+| `NVARCHAR` | `Utf8View` |
By default, string types are mapped to `Utf8View`. This can be configured using the `datafusion.sql_parser.map_string_types_to_utf8view` setting. When set to `false`, string types are mapped to `Utf8` instead.
+Note that `CHAR`, `VARCHAR`, `STRING`, `CHARACTER VARYING`, `CHAR VARYING`, and `NVARCHAR` can
+optionally receive the maximum length (e.g., `VARCHAR(n)`). However, this limit is ignored. To raise
+an error when the length is passed, set `datafusion.sql_parser.support_varchar_with_length` to
+`false`.
+
## Numeric Types
-| SQL DataType | Arrow DataType |
-| ------------------------------------------------ | :----------------------------- |
-| `TINYINT` | `Int8` |
-| `SMALLINT` | `Int16` |
-| `INT` or `INTEGER` | `Int32` |
-| `BIGINT` | `Int64` |
-| `TINYINT UNSIGNED` | `UInt8` |
-| `SMALLINT UNSIGNED` | `UInt16` |
-| `INT UNSIGNED` or `INTEGER UNSIGNED` | `UInt32` |
-| `BIGINT UNSIGNED` | `UInt64` |
-| `FLOAT` | `Float32` |
-| `REAL` | `Float32` |
-| `DOUBLE` | `Float64` |
-| `DECIMAL(precision, scale)` where precision ≤ 38 | `Decimal128(precision, scale)` |
-| `DECIMAL(precision, scale)` where precision > 38 | `Decimal256(precision, scale)` |
+| SQL DataType | Arrow DataType |
+| ----------------------------------------------------------------------------- | :----------------------------- |
+| `TINYINT` | `Int8` |
+| `SMALLINT` or `INT2` | `Int16` |
+| `INT` or `INTEGER` or `INT4` | `Int32` |
+| `BIGINT` or `INT8` | `Int64` |
+| `TINYINT UNSIGNED` or `UTINYINT` | `UInt8` |
+| `SMALLINT UNSIGNED` or `INT2 UNSIGNED` or `USMALLINT` | `UInt16` |
+| `INT UNSIGNED` or `INTEGER UNSIGNED` or `INT4 UNSIGNED` | `UInt32` |
+| `BIGINT UNSIGNED` or `INT8 UNSIGNED` or `UBIGINT` | `UInt64` |
+| `FLOAT` or `FLOAT4` or `REAL` | `Float32` |
+| `DOUBLE` or `DOUBLE PRECISION` or `FLOAT8` | `Float64` |
+| `DECIMAL(precision, scale) or NUMERIC(precision, scale)` where precision ≤ 38 | `Decimal128(precision, scale)` |
+| `DECIMAL(precision, scale) or NUMERIC(precision, scale)` where precision > 38 | `Decimal256(precision, scale)` |
The maximum supported precision for `DECIMAL` types is 76.
@@ -107,32 +114,46 @@ The maximum supported precision for `DECIMAL` types is 76.
## Boolean Types
-| SQL DataType | Arrow DataType |
-| ------------ | :------------- |
-| `BOOLEAN` | `Boolean` |
+| SQL DataType | Arrow DataType |
+| ------------------- | :------------- |
+| `BOOLEAN` or `BOOL` | `Boolean` |
## Binary Types
| SQL DataType | Arrow DataType |
| ------------ | :------------- |
| `BYTEA` | `Binary` |
+| `BLOB` | `Binary` |
+| `BINARY` | `Binary` |
+| `VARBINARY` | `Binary` |
+| `BYTES` | `Binary` |
You can create binary literals using a hex string literal such as
`X'1234'` to create a `Binary` value of two bytes, `0x12` and `0x34`.
+## Nested Types
+
+| SQL DataType | Arrow DataType |
+| --------------------------------- | :------------------------------ |
+| `ARRAY<T>` or `T[]` | `List(T)` |
+| `T[n]` | `FixedSizeList(n x T)` |
+| `MAP(k, v)` | `Map(k, v)` |
+| `STRUCT<[c0] T0[, ..., [cn] Tn]>` | `Struct(c0: T0[, ..., cn: Tn])` |
+
+While columns defined with `STRUCT` require all rows to have the same schema, `MAP` imposes no such
+restriction.
+
+Also note that `STRUCT` can be declared without defining the field names. In that case, they will be
+set to `c0`, `c1`, ..., `cn`.
+
## Unsupported SQL Types
| SQL Data Type | Arrow DataType |
| ------------- | :------------------ |
| `UUID` | _Not yet supported_ |
-| `BLOB` | _Not yet supported_ |
| `CLOB` | _Not yet supported_ |
-| `BINARY` | _Not yet supported_ |
-| `VARBINARY` | _Not yet supported_ |
| `REGCLASS` | _Not yet supported_ |
-| `NVARCHAR` | _Not yet supported_ |
| `CUSTOM` | _Not yet supported_ |
-| `ARRAY` | _Not yet supported_ |
| `ENUM` | _Not yet supported_ |
| `SET` | _Not yet supported_ |
| `DATETIME` | _Not yet supported_ |
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org