You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/10/12 10:03:10 UTC
[GitHub] [incubator-seatunnel] gitfortian opened a new issue, #3077: [Bug] [Module Name] doris insert error caused by data that include \n
gitfortian opened a new issue, #3077:
URL: https://github.com/apache/incubator-seatunnel/issues/3077
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
Reason: actual column number is less than schema column number.actual number: 1, column separator: [\t], line delimiter: [
], schema number: 57;
and when i set
doris.strip_outer_array="true"
doris.format="json"
in seatunnel conf it is not work either
### SeaTunnel Version
2.1.3
### SeaTunnel Config
```conf
env {
# You can set spark configuration here
# see available properties defined by spark: https://spark.apache.org/docs/latest/configuration.html#available-properties
spark.app.name = "SeaTunnel"
spark.executor.instances = 2
spark.executor.cores = 8
spark.executor.memory = "4g"
}
source {
jdbc {
driver = "com.mysql.jdbc.Driver"
url = "jdbc:mysql://127.0.0.1:3306/test"
user = "test"
password = "xxx"
table="test",
result_table_name="test"
}
transform {
}
sink {
Doris {
fenodes="127.0.0.1:8031"
database="test"
table="test"
user="test"
password="xxx"
batch_size=1000
doris.column_separator="\t"
或者下面两个配置
doris.strip_outer_array=true
doris.format="json"
}
}
```
### Running Command
```shell
./bin/start-seatunnel-spark.sh \
--master local[4] \
--deploy-mode client \
--config ./config/mysql2dorisspark.conf
```
### Error Exception
```log
Reason: actual column number is less than schema column number.actual number: 1, column separator: [\t], line delimiter: [
], schema number: 57; . src line https://miniprogtest.xxx.chinaunicom.cn/wechat-server/\x01\N\x01\N\x01\N\x01\N\x01\N\x01\N\x01\N\x01\N\x01\N\x01\N\x01false\x01\N\x01admin\x01import\x012022-06-08
```
### Flink or Spark Version
spark 2.4
### Java or Scala Version
java 1.8 scala 2.11.12
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CallMeKingsley97 commented on issue #3077: [Bug] [Module Name] doris insert error caused by data that include \n
Posted by GitBox <gi...@apache.org>.
CallMeKingsley97 commented on issue #3077:
URL: https://github.com/apache/incubator-seatunnel/issues/3077#issuecomment-1296423777
Use your record
`https://miniprogtest.xxx.chinaunicom.cn/wechat-server/\x01\N\x01\N\x01\N\x01\N\x01\N\x01\N\x01\N\x01\N\x01\N\x01\N\x01false\x01\N\x01admin\x01import\x012022-06-08)` , **failed.**
And return the info: **Reason: null value for not null column, column=name. src line: [];**
So maybe the error caused by `\x01`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] gitfortian commented on issue #3077: [Bug] [Module Name] doris insert error caused by data that include \n
Posted by GitBox <gi...@apache.org>.
gitfortian commented on issue #3077:
URL: https://github.com/apache/incubator-seatunnel/issues/3077#issuecomment-1288485461
> Could you give some records in the mysql? @gitfortian
i viewed the spark connector code and found that problem cased by u split content by \n default this hard coding is not reasonable,can u support that insert data by setting properties like flink 'sink.properties.format' = 'json',
'sink.properties.read_json_by_line' = 'true'
> Could you give some records in the mysql? @gitfortian
<img width="470" alt="doris" src="https://user-images.githubusercontent.com/52306466/197461889-3ea7382a-bf31-4d7f-9fde-c1bcdd280f4f.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] gitfortian commented on issue #3077: [Bug] [Module Name] doris insert error caused by data that include \n
Posted by GitBox <gi...@apache.org>.
gitfortian commented on issue #3077:
URL: https://github.com/apache/incubator-seatunnel/issues/3077#issuecomment-1288465702
u can insert into a record that which filed is string type and that content include \n would case this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] gitfortian commented on issue #3077: [Bug] [Module Name] doris insert error caused by data that include \n
Posted by GitBox <gi...@apache.org>.
gitfortian commented on issue #3077:
URL: https://github.com/apache/incubator-seatunnel/issues/3077#issuecomment-1288486223
mkString("\n") this cause the issue happens
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #3077: [Bug] [Module Name] doris insert error caused by data that include \n
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #3077:
URL: https://github.com/apache/incubator-seatunnel/issues/3077#issuecomment-1350157536
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] gitfortian commented on issue #3077: [Bug] [Module Name] doris insert error caused by data that include \n
Posted by GitBox <gi...@apache.org>.
gitfortian commented on issue #3077:
URL: https://github.com/apache/incubator-seatunnel/issues/3077#issuecomment-1288470340
> Could you give some records in the mysql? @gitfortian
i viewed the spark connector code and found that problem cased by u split content by \n default this hard coding is not reasonable,can u support that insert data by setting properties like flink 'sink.properties.format' = 'json',
'sink.properties.read_json_by_line' = 'true'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] gitfortian commented on issue #3077: [Bug] [Module Name] doris insert error caused by data that include \n
Posted by GitBox <gi...@apache.org>.
gitfortian commented on issue #3077:
URL: https://github.com/apache/incubator-seatunnel/issues/3077#issuecomment-1288466885
>
mysql? idont unstand
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CallMeKingsley97 commented on issue #3077: [Bug] [Module Name] doris insert error caused by data that include \n
Posted by GitBox <gi...@apache.org>.
CallMeKingsley97 commented on issue #3077:
URL: https://github.com/apache/incubator-seatunnel/issues/3077#issuecomment-1294822500
> mkString("\n") this cause the issue happens
Thanks, I'll try to fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CallMeKingsley97 commented on issue #3077: [Bug] [Module Name] doris insert error caused by data that include \n
Posted by GitBox <gi...@apache.org>.
CallMeKingsley97 commented on issue #3077:
URL: https://github.com/apache/incubator-seatunnel/issues/3077#issuecomment-1278713570
Could you give some records in the mysql? @gitfortian
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CallMeKingsley97 commented on issue #3077: [Bug] [Module Name] doris insert error caused by data that include \n
Posted by GitBox <gi...@apache.org>.
CallMeKingsley97 commented on issue #3077:
URL: https://github.com/apache/incubator-seatunnel/issues/3077#issuecomment-1296407659
> mkString("\n") this cause the issue happens
but when I try to insert into doris with the records that contains "\n", the result is succeed. (Spark)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] closed issue #3077: [Bug] [Module Name] doris insert error caused by data that include \n
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #3077: [Bug] [Module Name] doris insert error caused by data that include \n
URL: https://github.com/apache/incubator-seatunnel/issues/3077
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #3077: [Bug] [Module Name] doris insert error caused by data that include \n
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #3077:
URL: https://github.com/apache/incubator-seatunnel/issues/3077#issuecomment-1332936318
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org