You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/02/24 02:12:14 UTC
[GitHub] [incubator-seatunnel] kalencaya opened a new issue #1324: [Bug] [seatunnel-connector-flink-file] can't sync jdbc type eg date、datetime column by flink JdbcSource and FileSink
kalencaya opened a new issue #1324:
URL: https://github.com/apache/incubator-seatunnel/issues/1324
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
As title described, when using flink connector to sync data between JdbcSource and FileSink plugin, I got a strange failure.
FileSink plugin parameter format is json, any date jdbc type such as date, time, datetime, timesamp column failed to sync, but csv and text format work well.
### SeaTunnel Version
dev branch
### SeaTunnel Config
```conf
not need
```
### Running Command
```shell
not need
```
### Error Exception
```log
not need
```
### Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] kalencaya commented on issue #1324: [Bug] [seatunnel-connector-flink-file] can't sync jdbc type eg date、datetime column by flink JdbcSource and FileSink
Posted by GitBox <gi...@apache.org>.
kalencaya commented on issue #1324:
URL: https://github.com/apache/incubator-seatunnel/issues/1324#issuecomment-1050611529
I had found out why FileSink can't output some date columns produced by JdbcSource.
JdbcSource maps date jdbc type to java.sql.Date and so on automatically, then FileSource uses JsonRowOutputFormat output data to file.
Unfortunately, JsonRowOutputFormat ignores java.sql.Date、java.sql.Time or java.sql.Timestamp for SqlTimeTypeInfo not behaves as a basic type but a atomic type.
I think changing if condition from `type.isBasicType()` to `type instanceof AtomicType` is a good idea.
```
public class JsonRowOutputFormat extends FileOutputFormat<Row> {
@Override
public void writeRecord(Row record) throws IOException {
final JSONObject json = getJson(record, rowTypeInfo);
byte[] bytes = json.toString().getBytes(charset);
this.stream.write(bytes);
this.stream.write(NEWLINE);
}
private JSONObject getJson(Row record, RowTypeInfo rowTypeInfo) {
String[] fieldNames = rowTypeInfo.getFieldNames();
int i = 0;
JSONObject json = new JSONObject();
for (String name : fieldNames) {
Object field = record.getField(i);
final TypeInformation type = rowTypeInfo.getTypeAt(i);
// SqlTimeTypeInfo is not a basic type
if (type.isBasicType())) {
json.put(name, field);
} else if (type instanceof ObjectArrayTypeInfo) {
ObjectArrayTypeInfo arrayTypeInfo = (ObjectArrayTypeInfo) type;
TypeInformation componentInfo = arrayTypeInfo.getComponentInfo();
JSONArray jsonArray = new JSONArray();
if (componentInfo instanceof RowTypeInfo) {
final Row[] rows = (Row[]) field;
for (Row r : rows) {
jsonArray.add(getJson(r, (RowTypeInfo) componentInfo));
}
} else {
jsonArray.addAll(Arrays.asList((Object[]) field));
}
json.put(name, jsonArray);
} else if (type instanceof RowTypeInfo) {
RowTypeInfo typeInfo = (RowTypeInfo) type;
json.put(name, getJson((Row) field, typeInfo));
}
i++;
}
return json;
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] leo65535 closed issue #1324: [Bug] [seatunnel-connector-flink-file] can't sync jdbc type eg date、datetime column by flink JdbcSource and FileSink
Posted by GitBox <gi...@apache.org>.
leo65535 closed issue #1324:
URL: https://github.com/apache/incubator-seatunnel/issues/1324
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org