You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "RexLyc (via GitHub)" <gi...@apache.org> on 2026/01/15 11:33:10 UTC
[I] [Bug] [Connector-V2] nullable option not work for some connector [seatunnel]
RexLyc opened a new issue, #10350:
URL: https://github.com/apache/seatunnel/issues/10350
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
I'm using seatunnel these days. And I'm wondering if there is a problem that some connectors didn't do nullable check, suck as S3/Sftp/LocalFs. These connectors are designed to read&write from&to a file. In many cases, they work well. But when I misdefine a column name in schema, which I use a wrong name. Things are getting wired.
For example, my csv file seems like
```
a,b
1,2
```
and my schema seems like
```
schema {
columns = [
{
name = b
type = int
},
{
name = c
type = int
}
]
}
```
I think I should get an error, because c doens't exist. But instead, I get an empty value
```
b,c
2,
```
After I got this, I think I might get error if i set 'nullable' to 'c'.
```
schema {
columns = [
{
name = b
type = int
},
{
name = c
type = int
nullable = false
}
]
}
```
But I still got the result above.
So there might be two problems.
1. At least csv format file deserializeSchema class didn't check whether schema is out of fields in real data file.
2. When user provide a wrong schema (p.s. column won't exist) , we should check and report an error.
In fact, I'm not sure if this is a feature or a bug.
### SeaTunnel Version
2.3.12
### SeaTunnel Config
```conf
env {
job.mode = "BATCH"
job.name = "seatunnel_job"
parallelism = 1
}
source {
LocalFile {
path = "/home/kpad/connector/data/localfs"
file_format_type = "csv"
file_filter_pattern = "alice-greater-than"
csv_use_header_line = true
field_delimiter = ","
schema {
columns = [
{
name = greater_than
type = bigint
nullable = false
},
{
name = x
type = double
nullable = true
}
]
}
}
}
sink {
LocalFile {
path = "/home/kpad/var/storage/data"
file_format_type = "csv"
single_file_mode = true
custom_filename = true
is_enable_transaction = false
enable_header_write = true
create_empty_file_when_no_data = true
file_name_expression = "greater-than-dataset"
filename_extension = ".csv"
}
}
```
### Running Command
```shell
// I'm using golang to run seatunnel (locally in my docker)
s.cmd = exec.Command("seatunnel.sh", "-c", "seatunnel.conf", "-m", "local")
```
### Error Exception
```log
I'm wondering why nullable error didn't appear.
```
### Zeta or Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
<img width="3420" height="1904" alt="Image" src="https://github.com/user-attachments/assets/44cf312f-87f3-41bf-9719-7b68582b52ce" />
This place didn't check whether this column SHOULD exists. As I said above, user might give a wrong column name that will never appear in real data file.
I'm still looking for the `SeaTunnelRow` processing code that should be designed to do nullable check.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Connector-V2] nullable option not work for some connector [seatunnel]
Posted by "RexLyc (via GitHub)" <gi...@apache.org>.
RexLyc commented on issue #10350:
URL: https://github.com/apache/seatunnel/issues/10350#issuecomment-3804580451
I remembered at least LocalFile/S3File has this problem. Further more, as I said above, this problem seems can also be appeared under others situations. Such as user give a column that doesn't exist, and this column required nullable check. In this case, we also get null value without any error thrown.
The configuration and example data below has been tested, and you should get same error under seatunnel 2.3.12 version.
```conf
env {
job.mode = "BATCH"
job.name = "seatunnel_job"
parallelism = 1
}
source {
LocalFile {
path = "/home/kpad/connector/data/localfs"
file_format_type = "csv"
file_filter_pattern = "alice-local-table.csv"
csv_use_header_line = true
field_delimiter = ","
schema {
columns = [
{
name = id
type = bigint
nullable = false
},
{
name = x
type = double
nullable = true
}
]
}
}
}
sink {
LocalFile {
path = "/home/kpad/var/storage/data"
file_format_type = "csv"
single_file_mode = true
custom_filename = true
is_enable_transaction = false
enable_header_write = true
create_empty_file_when_no_data = true
file_name_expression = "alice-local-import"
filename_extension = ".csv"
}
}
```
This is my input csv file, which placed in /home/kpad/connector/data/localfs/alice-local-table.csv
```csv
id,x
,9.9
```
And I got the result csv placed in /home/kpad/var/storage/data/alice-local-import.csv
```csv
id,x
,9.9
```
In this case, nullable check doesn't work.
And even if I set my input csv like this.
```csv
x
9.9
10.0
```
In this case, I totally removed column id. And I still got the result.
```
id,x
,9.9
,10.0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Connector-V2] nullable option not work for some connector [seatunnel]
Posted by "RexLyc (via GitHub)" <gi...@apache.org>.
RexLyc commented on issue #10350:
URL: https://github.com/apache/seatunnel/issues/10350#issuecomment-3754342763
By the way, I also found that S3File/LocalFS/Sftp connectors, won't throw 'file doesn't exist' exception.
I know that we are using name to filter a group of files. But when we got 0 file, I think it would be better we throw an error.
Or we give a option to make user decide whether throw error or we ignore it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Connector-V2] nullable option not work for some connector [seatunnel]
Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on issue #10350:
URL: https://github.com/apache/seatunnel/issues/10350#issuecomment-3756857459
<!-- code-pr-reviewer -->
**Thank you for reporting this.**
I've checked the codebase and can confirm the inconsistency in `nullable` option handling across connectors.
- `CsvReadStrategy.readProcess` validates nullability when missing columns are encountered (in `seatunnel-connectors-v2/connector-file/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/reader/CsvReadStrategy.java`)
- S3 and SFTP connectors reuse the same base class, so they should behave similarly
- Other connectors (e.g., JDBC, Kafka) don't enforce nullable constraints at read time, only during schema initialization
To help us reproduce and fix this efficiently, could you clarify:
1. Which specific connectors did you observe ignoring the `nullable` option?
2. What is the expected vs actual behavior (e.g., error on null vs null values accepted)?
3. Could you provide a minimal sample config and data file that demonstrates the issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Connector-V2] nullable option not work for some connector [seatunnel]
Posted by "DanielCarter-stack (via GitHub)" <gi...@apache.org>.
DanielCarter-stack commented on issue #10350:
URL: https://github.com/apache/seatunnel/issues/10350#issuecomment-3759069890
<!-- code-pr-reviewer -->
Thanks for reporting this. To help investigate effectively, could you please provide:
1. Which specific connectors are affected?
2. Exact connector configuration showing the `nullable` option usage?
3. Expected vs actual behavior (e.g., schema mismatch, runtime error, or silent ignore)?
The `nullable` option is typically processed in source table schema parsing, so connector-specific configs and SeaTunnel version would help identify whether this is a framework-level or connector-specific issue.
Please share a minimal reproducible example with connector type, config, and the error/log output.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Connector-V2] nullable option not work for some connector [seatunnel]
Posted by "RexLyc (via GitHub)" <gi...@apache.org>.
RexLyc commented on issue #10350:
URL: https://github.com/apache/seatunnel/issues/10350#issuecomment-3804594821
> By the way, I also found that connectors such as S3File/LocalFS/Sftp, won't throw 'file doesn't exist' exception. I know that we are using name to filter a group of files. But when we got 0 file, I think it would be better we throw an error. Or we give a option to make user decide whether throw error or we ignore it.
Also, I have to say, some times I'm not sure whether this is a feature or a bug.😂
These two problems make me really comfuse. Cause I will get download/upload successfully finished, but get nothing in target data source.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Connector-V2] nullable option not work for some connector [seatunnel]
Posted by "RexLyc (via GitHub)" <gi...@apache.org>.
RexLyc commented on issue #10350:
URL: https://github.com/apache/seatunnel/issues/10350#issuecomment-3766169840
Sure, I'll get back to this shortly. Work has been a bit busy these past couple of days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org