You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "gyang94 (via GitHub)" <gi...@apache.org> on 2026/08/21 09:32:21 UTC
[I] [Bug] Flink Catalog loses nested NOT NULL constraints in complex types [fluss]
gyang94 opened a new issue, #4062:
URL: https://github.com/apache/fluss/issues/4062
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Fluss version
main (development)
### Please describe the bug 🐞
### Environment
- Flink 1.20.3
- Fluss Flink connector built from the current development branch
### Description
When a Fluss table is created through Flink SQL/Fluss Catalog, `NOT NULL` declared on fields nested inside `ROW`, `ARRAY<ROW<...>>`, or similar complex types is lost.
For example:
```sql
CREATE TABLE nested_nullability_repro (
headers ARRAY<ROW<name STRING NOT NULL, header_value BYTES>>
) WITH (
'bucket.num' = '1'
);
```
Expected:
```text
headers ARRAY<ROW<name STRING NOT NULL, header_value BYTES>>
```
Actual schema retrieved from Fluss metadata:
```text
headers ARRAY<ROW<name STRING, header_value BYTES>>
```
Top-level `NOT NULL` constraints are retained; the problem affects nested fields.
### Minimal reproduction without Fluss
The same loss can be reproduced in Flink 1.20.3 with the default in-memory Catalog, before the schema reaches `FlinkCatalog`:
```java
TableEnvironment tableEnvironment =
TableEnvironment.create(EnvironmentSettings.newInstance().inBatchMode().build());
tableEnvironment.executeSql(
"CREATE TABLE nested_nullability_repro ("
+ "headers ARRAY<ROW<name STRING NOT NULL, header_value BYTES>>) "
+ "WITH ('connector' = 'blackhole')");
CatalogBaseTable table =
tableEnvironment
.getCatalog(tableEnvironment.getCurrentCatalog())
.get()
.getTable(
new ObjectPath(
tableEnvironment.getCurrentDatabase(),
"nested_nullability_repro"));
System.out.println(table.getUnresolvedSchema());
```
Output:
```text
(
`headers` ARRAY<ROW<`name` STRING, `header_value` BYTES>>
)
```
This indicates that nested nullability has already been lost in Flink's SQL/Catalog schema representation before the Fluss connector converts the type.
### Why direct Kafka CreateTopics does not show the problem
The Kafka compatibility smoke test creates a topic through Kafka `AdminClient.createTopics`. The Fluss Kafka backend builds a `TableDescriptor` directly with:
```java
DataTypes.FIELD("name", DataTypes.STRING().copy(false))
```
That path bypasses Flink SQL/Catalog conversion. The programmatically constructed Fluss type and the `TableDescriptor` JSON round trip preserve nested `NOT NULL`, so smoke-test tables can contain `ROW<name STRING NOT NULL, value BYTES>`.
### Impact
- The persisted Fluss schema differs from the user's Flink DDL.
- Nested data-integrity constraints are silently weakened.
- Components that validate exact nested nullability can reject an otherwise correctly declared table.
- Flink SQL-created tables and programmatically created Fluss tables behave differently.
### Solution
1. Add a regression test to the Fluss Flink Catalog integration tests.
2. Determine which supported Flink versions are affected.
3. Track/fix the behavior upstream in Apache Flink if it is still present in current Flink releases.
4. Decide whether the Fluss connector should document or defensively handle affected Flink versions.
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Flink Catalog loses nested NOT NULL constraints in complex types [fluss]
Posted by "leekeiabstraction (via GitHub)" <gi...@apache.org>.
leekeiabstraction commented on issue #4062:
URL: https://github.com/apache/fluss/issues/4062#issuecomment-5449916526
Assigned to you @Rango-2017
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Flink Catalog loses nested NOT NULL constraints in complex types [fluss]
Posted by "leekeiabstraction (via GitHub)" <gi...@apache.org>.
leekeiabstraction closed issue #4062: [Bug] Flink Catalog loses nested NOT NULL constraints in complex types
URL: https://github.com/apache/fluss/issues/4062
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Flink Catalog loses nested NOT NULL constraints in complex types [fluss]
Posted by "leekeiabstraction (via GitHub)" <gi...@apache.org>.
leekeiabstraction commented on issue #4062:
URL: https://github.com/apache/fluss/issues/4062#issuecomment-5499791785
@Rango-2017 Checking in, is there any progress on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Flink Catalog loses nested NOT NULL constraints in complex types [fluss]
Posted by "leekeiabstraction (via GitHub)" <gi...@apache.org>.
leekeiabstraction commented on issue #4062:
URL: https://github.com/apache/fluss/issues/4062#issuecomment-5505757424
Taking over to unblock release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Flink Catalog loses nested NOT NULL constraints in complex types [fluss]
Posted by "leekeiabstraction (via GitHub)" <gi...@apache.org>.
leekeiabstraction commented on issue #4062:
URL: https://github.com/apache/fluss/issues/4062#issuecomment-5506033367
Looked deeper into this.
This is actually a bug in Flink 1.x and 2.0, 2.1.
https://issues.apache.org/jira/browse/FLINK-20539
Removing blocker priority on this, raised PR for docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Flink Catalog loses nested NOT NULL constraints in complex types [fluss]
Posted by "Rango-2017 (via GitHub)" <gi...@apache.org>.
Rango-2017 commented on issue #4062:
URL: https://github.com/apache/fluss/issues/4062#issuecomment-5403959963
I'd like to take this.
**Understanding:** Top-level `NOT NULL` is kept, but `NOT NULL` on fields inside `ROW` / `ARRAY<ROW<...>>` is lost after `CREATE TABLE` through the Flink Catalog. `FlinkCatalog#createTable` already goes through `ResolvedCatalogTable` and `FlinkConversions.toFlussTable` → `FlinkTypeToFlussType`, which recurses into nested types and passes `isNullable()`. The GenericInMemoryCatalog repro in this issue shows that Flink's *unresolved* schema printer drops nested nullability, so I want to split two cases before changing code:
1. If `ResolvedSchema` / `Column.getDataType()` still carries nested `NOT NULL`, persist it via `toFlussType` and assert it on `Admin#getTableInfo` (raw Fluss metadata), plus a Catalog ITCase. If `DESCRIBE` / `getUnresolvedSchema()` still looks wrong after that, the remaining gap is likely `toFlinkTable` → Flink `Schema.Builder`, and I will fix that path if we control it.
2. If Flink has already dropped nested `NOT NULL` on the resolved type before `createTable`, that is an upstream Flink representation issue. I will report that here instead of inventing a Fluss-only workaround.
**Scope:** Flink Catalog type conversion and a regression ITCase for `ARRAY<ROW<... NOT NULL>>` / `ROW<... NOT NULL>`. No table format or RPC change.
Could this be assigned to me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org