You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/08/20 00:50:49 UTC
(datafusion) branch gh-readonly-queue/main/pr-24471-861cd7d7164a3653f4a47801b4e95e773f4c78b8 created (now 733b195726)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-24471-861cd7d7164a3653f4a47801b4e95e773f4c78b8
in repository https://gitbox.apache.org/repos/asf/datafusion.git
at 733b195726 docs: Document missing SQL data types (#24471)
This branch includes the following new commits:
new 733b195726 docs: Document missing SQL data types (#24471)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion) 01/01: docs: Document missing SQL data types (#24471)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-24471-861cd7d7164a3653f4a47801b4e95e773f4c78b8
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit 733b195726e0c19007893e8a543aa7c8e7367026
Author: Nuno Faria <nu...@gmail.com>
AuthorDate: Thu Aug 20 00:50:40 2026 +0000
docs: Document missing SQL data types (#24471)
## Which issue does this PR close?
- N/A.
## Rationale for this change
The data types page
(https://datafusion.apache.org/user-guide/sql/data_types.html) is quite
outdated, e.g., still claiming that `ARRAY` is not supported. This PR
updates the missing types and also adds missing aliases.
This should only be merged after #24398.
## What changes are included in this PR?
- Add missing types, like `ARRAY`, `STRUCT`, and `MAP`, as well as alias
for existing types.
- Updated the sqlparser-rs link, which was pointing to the wrong line.
## Are these changes tested?
Documentation only.
## Are there any user-facing changes?
No.
---
docs/source/user-guide/sql/data_types.md | 81 ++++++++++++++++++++------------
1 file changed, 51 insertions(+), 30 deletions(-)
diff --git a/docs/source/user-guide/sql/data_types.md b/docs/source/user-guide/sql/data_types.md
index 502193df41..2e75fce6c7 100644
--- a/docs/source/user-guide/sql/data_types.md
+++ b/docs/source/user-guide/sql/data_types.md
@@ -21,7 +21,7 @@
DataFusion uses Arrow, and thus the Arrow type system, for query
execution. The SQL types from
-[sqlparser-rs](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/src/ast/data_type.rs#L27)
+[sqlparser-rs](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/src/ast/data_type.rs#L49)
are mapped to [Arrow data types](https://docs.rs/arrow/latest/arrow/datatypes/enum.DataType.html) according to the following table.
This mapping occurs when defining the schema in a `CREATE EXTERNAL TABLE` command or when performing a SQL `CAST` operation.
@@ -67,32 +67,39 @@ select arrow_cast(now(), 'Timestamp(Second, None)') as "now()";
## Character Types
-| SQL DataType | Arrow DataType |
-| ------------ | -------------- |
-| `CHAR` | `Utf8View` |
-| `VARCHAR` | `Utf8View` |
-| `TEXT` | `Utf8View` |
-| `STRING` | `Utf8View` |
+| SQL DataType | Arrow DataType |
+| ------------------- | -------------- |
+| `CHAR` | `Utf8View` |
+| `VARCHAR` | `Utf8View` |
+| `TEXT` | `Utf8View` |
+| `STRING` | `Utf8View` |
+| `CHARACTER VARYING` | `Utf8View` |
+| `CHAR VARYING` | `Utf8View` |
+| `NVARCHAR` | `Utf8View` |
By default, string types are mapped to `Utf8View`. This can be configured using the `datafusion.sql_parser.map_string_types_to_utf8view` setting. When set to `false`, string types are mapped to `Utf8` instead.
+Note that `CHAR`, `VARCHAR`, `STRING`, `CHARACTER VARYING`, `CHAR VARYING`, and `NVARCHAR` can
+optionally receive the maximum length (e.g., `VARCHAR(n)`). However, this limit is ignored. To raise
+an error when the length is passed, set `datafusion.sql_parser.support_varchar_with_length` to
+`false`.
+
## Numeric Types
-| SQL DataType | Arrow DataType |
-| ------------------------------------------------ | :----------------------------- |
-| `TINYINT` | `Int8` |
-| `SMALLINT` | `Int16` |
-| `INT` or `INTEGER` | `Int32` |
-| `BIGINT` | `Int64` |
-| `TINYINT UNSIGNED` | `UInt8` |
-| `SMALLINT UNSIGNED` | `UInt16` |
-| `INT UNSIGNED` or `INTEGER UNSIGNED` | `UInt32` |
-| `BIGINT UNSIGNED` | `UInt64` |
-| `FLOAT` | `Float32` |
-| `REAL` | `Float32` |
-| `DOUBLE` | `Float64` |
-| `DECIMAL(precision, scale)` where precision ≤ 38 | `Decimal128(precision, scale)` |
-| `DECIMAL(precision, scale)` where precision > 38 | `Decimal256(precision, scale)` |
+| SQL DataType | Arrow DataType |
+| ----------------------------------------------------------------------------- | :----------------------------- |
+| `TINYINT` | `Int8` |
+| `SMALLINT` or `INT2` | `Int16` |
+| `INT` or `INTEGER` or `INT4` | `Int32` |
+| `BIGINT` or `INT8` | `Int64` |
+| `TINYINT UNSIGNED` or `UTINYINT` | `UInt8` |
+| `SMALLINT UNSIGNED` or `INT2 UNSIGNED` or `USMALLINT` | `UInt16` |
+| `INT UNSIGNED` or `INTEGER UNSIGNED` or `INT4 UNSIGNED` | `UInt32` |
+| `BIGINT UNSIGNED` or `INT8 UNSIGNED` or `UBIGINT` | `UInt64` |
+| `FLOAT` or `FLOAT4` or `REAL` | `Float32` |
+| `DOUBLE` or `DOUBLE PRECISION` or `FLOAT8` | `Float64` |
+| `DECIMAL(precision, scale) or NUMERIC(precision, scale)` where precision ≤ 38 | `Decimal128(precision, scale)` |
+| `DECIMAL(precision, scale) or NUMERIC(precision, scale)` where precision > 38 | `Decimal256(precision, scale)` |
The maximum supported precision for `DECIMAL` types is 76.
@@ -107,32 +114,46 @@ The maximum supported precision for `DECIMAL` types is 76.
## Boolean Types
-| SQL DataType | Arrow DataType |
-| ------------ | :------------- |
-| `BOOLEAN` | `Boolean` |
+| SQL DataType | Arrow DataType |
+| ------------------- | :------------- |
+| `BOOLEAN` or `BOOL` | `Boolean` |
## Binary Types
| SQL DataType | Arrow DataType |
| ------------ | :------------- |
| `BYTEA` | `Binary` |
+| `BLOB` | `Binary` |
+| `BINARY` | `Binary` |
+| `VARBINARY` | `Binary` |
+| `BYTES` | `Binary` |
You can create binary literals using a hex string literal such as
`X'1234'` to create a `Binary` value of two bytes, `0x12` and `0x34`.
+## Nested Types
+
+| SQL DataType | Arrow DataType |
+| --------------------------------- | :------------------------------ |
+| `ARRAY<T>` or `T[]` | `List(T)` |
+| `T[n]` | `FixedSizeList(n x T)` |
+| `MAP(k, v)` | `Map(k, v)` |
+| `STRUCT<[c0] T0[, ..., [cn] Tn]>` | `Struct(c0: T0[, ..., cn: Tn])` |
+
+While columns defined with `STRUCT` require all rows to have the same schema, `MAP` imposes no such
+restriction.
+
+Also note that `STRUCT` can be declared without defining the field names. In that case, they will be
+set to `c0`, `c1`, ..., `cn`.
+
## Unsupported SQL Types
| SQL Data Type | Arrow DataType |
| ------------- | :------------------ |
| `UUID` | _Not yet supported_ |
-| `BLOB` | _Not yet supported_ |
| `CLOB` | _Not yet supported_ |
-| `BINARY` | _Not yet supported_ |
-| `VARBINARY` | _Not yet supported_ |
| `REGCLASS` | _Not yet supported_ |
-| `NVARCHAR` | _Not yet supported_ |
| `CUSTOM` | _Not yet supported_ |
-| `ARRAY` | _Not yet supported_ |
| `ENUM` | _Not yet supported_ |
| `SET` | _Not yet supported_ |
| `DATETIME` | _Not yet supported_ |
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org