You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "JeremyXin (via GitHub)" <gi...@apache.org> on 2025/07/21 02:54:18 UTC
[I] [BUG][connector-file-hadoop] When reading parquet file field value as BINARY type, write downstream data exception [seatunnel]
JeremyXin opened a new issue, #9140:
URL: https://github.com/apache/seatunnel/issues/9140
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
When I synchronized the upstream Hive table (actually it was synchronizing the HDFS file) in the Doris scenario, I found that the data written to Doris was abnormal.
The SQL statement of creating the Hive table is as below:
```
CREATE TABLE `xxx`.`xxx`(
`org_openid` string COMMENT 'xxx',
`is_admin` string COMMENT 'xxx',
`function_type` string COMMENT 'xxx',
`mail_active_day_count` string COMMENT 'xxx',
`mail_active_week_count` string COMMENT 'xxx',
`mail_active_last_week_count` string COMMENT 'xxx',
`mail_next_week_retention` string COMMENT 'xxx',
`mail_active_month_count` string COMMENT 'xxx',
`create_time` string COMMENT 'Creating Time',
`update_time` string COMMENT 'Updating Time',
`mail_active_web_day_count` bigint COMMENT 'xxx',
`mail_active_notweb_day_count` bigint COMMENT 'xxx',
`mail_active_web_week_count` bigint COMMENT 'xxx',
`mail_active_notweb_week_count` bigint COMMENT 'xxx')
PARTITIONED BY (
`dt` string)
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION
'hdfs://xxx'
```
The format of HDFS file:
```
ce4b677d9f44dc4b-421324f20000000a_2037610321_data.0.parq
```
After my local test analysis, I found that when using 'ParquetReadStragegy' to obtain metadata schema according to files, the above org_openid, is_amdin and other string fields were identified as BINARY, and the original logical type (OriginType) was null. As a result, those fields are considered as Byte Array (`PrimitiveByteArrayType`).
During the reading file process, the above fields were read into a byte array (`HeapByteBuffer` type). Even after applying the `resolveObject` method, they are still presented as a byte array. As a result, the data written into the Doris table is abnormal.
I'm not sure why the original logical type (`OriginType`) of the fields in HDFS files created from the above Hive table is null, and currently, I cannot alter the structure of the upstream Hive table.
For this situation, the possible solution might be that the parquet file supports user-defined schema so that these fields can be defined as SeaTunnelRowType of String. Then under the `case STRING` branch of the `resolveObject` method, if the field value type is ByteBuffer, additional processing is done to convert it to String. In fact, I have already implemented the above changes locally and written the correct data.
Actually, I noticed that `getSeaTunnelRowTypeInfoWithUserConfigRowType` method in `OrcReadStrategy` class is implemented to allow user-defined schema, but it isn't yet implemented in `ParquetReadStrategy` class. Maybe it is also worth implementing?
If this is a issues, I can submit the pr and fix it using the above design logic.
### SeaTunnel Version
2.3.8
### SeaTunnel Config
```conf
env {
job.mode = "BATCH"
parallelism = 10
}
source {
HdfsFile {
path = "hdfs://cluster1/xxx"
file_format_type = "parquet"
fs.defaultFS = "hdfs://cluster1"
hdfs_site_path = "xxx/hdfs-site.xml"
krb5_path = "xxx/krb5.conf"
kerberos_principal = "xxx"
kerberos_keytab_path = "xxx/kerberos.keytab"
}
}
sink {
Doris {
fenodes = "xxx:8030"
username = "xxx"
password = "xxx"
database = "database"
table = "table"
doris.config {
format = "json"
read_json_by_line = "true"
}
}
}
```
### Running Command
```shell
sh bin/seatunnel.sh --config config/v2.batch.hdfs.template -m local
```
### Error Exception
```log
parquet data is incorrectly written to the downstream doris table. For details, see the screenshot
```
### Zeta or Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
The schema parsed from the parquet file


The record value when reading the parquet file

Upstream hive table data sampling:

Write downstream doris table data sampling. The written data is abnormal.

### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [BUG][connector-file-hadoop] When reading parquet file field value as BINARY type, write downstream data exception [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #9140: [BUG][connector-file-hadoop] When reading parquet file field value as BINARY type, write downstream data exception
URL: https://github.com/apache/seatunnel/issues/9140
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [BUG][connector-file-hadoop] When reading parquet file field value as BINARY type, write downstream data exception [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #9140:
URL: https://github.com/apache/seatunnel/issues/9140#issuecomment-2869287228
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [BUG][connector-file-hadoop] When reading parquet file field value as BINARY type, write downstream data exception [seatunnel]
Posted by "corgy-w (via GitHub)" <gi...@apache.org>.
corgy-w closed issue #9140: [BUG][connector-file-hadoop] When reading parquet file field value as BINARY type, write downstream data exception
URL: https://github.com/apache/seatunnel/issues/9140
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [BUG][connector-file-hadoop] When reading parquet file field value as BINARY type, write downstream data exception [seatunnel]
Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on issue #9140:
URL: https://github.com/apache/seatunnel/issues/9140#issuecomment-2792493145
> Actually, I noticed that getSeaTunnelRowTypeInfoWithUserConfigRowType method in OrcReadStrategy class is implemented to allow user-defined schema, but it isn't yet implemented in ParquetReadStrategy class. Maybe it is also worth implementing?
Yes, we should implement same logic in `ParquetReadStrategy`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [BUG][connector-file-hadoop] When reading parquet file field value as BINARY type, write downstream data exception [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #9140:
URL: https://github.com/apache/seatunnel/issues/9140#issuecomment-2896134506
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org