You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/04/28 12:23:26 UTC
[GitHub] [hudi] zhilinli123 opened a new issue, #5460: org.apache.hudi.exception.HoodieRemoteException: status code: 500, reason phrase: Server Error
zhilinli123 opened a new issue, #5460:
URL: https://github.com/apache/hudi/issues/5460
<img width="1862" alt="image" src="https://user-images.githubusercontent.com/76689593/165749665-28cb958d-f2e7-4591-88b7-49d60b729b5d.png">
<img width="1826" alt="image" src="https://user-images.githubusercontent.com/76689593/165749892-9d34219e-dbba-4f25-949f-b85d850a7d8d.png">
<img width="1094" alt="image" src="https://user-images.githubusercontent.com/76689593/165750140-38577dc3-6e24-4d2b-aa85-3d962230953f.png">
<img width="1741" alt="image" src="https://user-images.githubusercontent.com/76689593/165750705-ae8c8f35-c219-4b73-b1bf-4d9e9dcc9978.png">
flink:1.14.3
hudi: newest master
hadoop: 3.2.1(cdh)
I am currently consuming kafka data to write multiple HUDi tables at the same time, but I found error log information on TM, causing compression failure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhilinli123 commented on issue #5460: org.apache.hudi.exception.HoodieRemoteException: status code: 500, reason phrase: Server Error
Posted by GitBox <gi...@apache.org>.
zhilinli123 commented on issue #5460:
URL: https://github.com/apache/hudi/issues/5460#issuecomment-1112829289
> @
> @zhilinli123为了帮助您确定根本原因，您能否提供重现问题的步骤/命令，包括如何编写多个表？您能否还显示`.hoodie`文件夹中的完整时间线，包括`20220428191757755`引发错误的时间线？您是否在同一张表上运行了任何并发编写器或 aysnc 表服务？
我当前是将Mysql cdc数据通过flink dataStream发送到kafka的一个topic中,下游使用侧输出流(outputTag)进行分流消费到各个hudi表中,当前我只有一个flink程序向hdfs目录去写不涉及到两个程序并行写入情况 如果需要我复现问题请随时联系我
<img width="1932" alt="image" src="https://user-images.githubusercontent.com/76689593/165878202-69f5e084-21ee-4289-bb7a-4534509132c2.png">
[errorLog.log](https://github.com/apache/hudi/files/8587801/errorLog.log)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #5460: org.apache.hudi.exception.HoodieRemoteException: status code: 500, reason phrase: Server Error
Posted by GitBox <gi...@apache.org>.
yihua commented on issue #5460:
URL: https://github.com/apache/hudi/issues/5460#issuecomment-1113620953
@zhilinli123 Definitely. Could you also show `20220428191757755*` files and their content and the full timeline in `.hoodie` folder directly? (`.hoodie` 文件夹的所有内容，包括`20220428191757755*`，以及这个 instant 文件里的内容)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhilinli123 commented on issue #5460: org.apache.hudi.exception.HoodieRemoteException: status code: 500, reason phrase: Server Error
Posted by GitBox <gi...@apache.org>.
zhilinli123 commented on issue #5460:
URL: https://github.com/apache/hudi/issues/5460#issuecomment-1171983234
> @zhilinli123 I checked markers you uploaded and they look fine. I don't see the timeline content from `.hoodie`. Could you paste the screenshot again?
>
> There are a couple of fixes around markers in Flink since then: #5604, #5611. Could you check if the latest master resolves your issue?
<img width="1562" alt="image" src="https://user-images.githubusercontent.com/76689593/176836563-f5f1940c-b6f2-45aa-b39b-df7ccc65b586.png">
The same thing happens again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #5460: org.apache.hudi.exception.HoodieRemoteException: status code: 500, reason phrase: Server Error
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #5460:
URL: https://github.com/apache/hudi/issues/5460#issuecomment-1172846821
We have created a retry for remote request recently: maybe it can help you https://github.com/apache/hudi/pull/5884#issuecomment-1171983624
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #5460: org.apache.hudi.exception.HoodieRemoteException: status code: 500, reason phrase: Server Error
Posted by GitBox <gi...@apache.org>.
yihua commented on issue #5460:
URL: https://github.com/apache/hudi/issues/5460#issuecomment-1112633524
Based on the current information, it's likely that the same marker is created twice and the second attempt fails. @danny0405 do you know if Flink has special handling around creating direct markers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhilinli123 commented on issue #5460: org.apache.hudi.exception.HoodieRemoteException: status code: 500, reason phrase: Server Error
Posted by GitBox <gi...@apache.org>.
zhilinli123 commented on issue #5460:
URL: https://github.com/apache/hudi/issues/5460#issuecomment-1171129279
>
I have not found this problem in the latest version for the time being. I will contact you if there is a problem. Now you can turn it off
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #5460: org.apache.hudi.exception.HoodieRemoteException: status code: 500, reason phrase: Server Error
Posted by GitBox <gi...@apache.org>.
yihua commented on issue #5460:
URL: https://github.com/apache/hudi/issues/5460#issuecomment-1169419551
@zhilinli123 I checked markers you uploaded and they look fine. I don't see the timeline content from `.hoodie`. Could you paste the screenshot again?
There are a couple of fixes around markers in Flink since then: #5604, #5611. Could you check if the latest master resolves your issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #5460: org.apache.hudi.exception.HoodieRemoteException: status code: 500, reason phrase: Server Error
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #5460:
URL: https://github.com/apache/hudi/issues/5460#issuecomment-1229542045
thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhilinli123 commented on issue #5460: org.apache.hudi.exception.HoodieRemoteException: status code: 500, reason phrase: Server Error
Posted by GitBox <gi...@apache.org>.
zhilinli123 commented on issue #5460:
URL: https://github.com/apache/hudi/issues/5460#issuecomment-1112830605
> @zhilinli123 by "compression failure", do you mean the Hudi compaction fails and `20220428191757755` is a scheduled compaction? Have you tried to restart and retry the Flink job and see the ingestion can proceed?
@yihua Could you please help to locate the problem? Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #5460: org.apache.hudi.exception.HoodieRemoteException: status code: 500, reason phrase: Server Error
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #5460: org.apache.hudi.exception.HoodieRemoteException: status code: 500, reason phrase: Server Error
URL: https://github.com/apache/hudi/issues/5460
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #5460: org.apache.hudi.exception.HoodieRemoteException: status code: 500, reason phrase: Server Error
Posted by GitBox <gi...@apache.org>.
codope commented on issue #5460:
URL: https://github.com/apache/hudi/issues/5460#issuecomment-1207201666
@zhilinli123 Could you please try the patch suggested above by Danny? This will go in the upcoming release 0.12.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #5460: org.apache.hudi.exception.HoodieRemoteException: status code: 500, reason phrase: Server Error
Posted by GitBox <gi...@apache.org>.
yihua commented on issue #5460:
URL: https://github.com/apache/hudi/issues/5460#issuecomment-1112631217
@zhilinli123 To help you identify the root cause, could you provide the steps/commands to reproduce the issue, including how multiple tables are written? Could you also show the full timeline in `.hoodie` folder, including the `20220428191757755` which throws the error? Did you run any concurrent writers or aysnc table services on the same table?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #5460: org.apache.hudi.exception.HoodieRemoteException: status code: 500, reason phrase: Server Error
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #5460:
URL: https://github.com/apache/hudi/issues/5460#issuecomment-1112799467
> Based on the current information, it's likely that the same marker is created twice and the second attempt fails. @danny0405 do you know if Flink has special handling around creating direct markers?
The flink write handle all check the existence of the MARKER files before creating new, not sure which step throws this error ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhilinli123 commented on issue #5460: org.apache.hudi.exception.HoodieRemoteException: status code: 500, reason phrase: Server Error
Posted by GitBox <gi...@apache.org>.
zhilinli123 commented on issue #5460:
URL: https://github.com/apache/hudi/issues/5460#issuecomment-1112830030
> @zhilinli123 by "compression failure", do you mean the Hudi compaction fails and `20220428191757755` is a scheduled compaction? Have you tried to restart and retry the Flink job and see the ingestion can proceed?
It is now understood that failure during compression causes the rollback program to be repeatedly restarted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #5460: org.apache.hudi.exception.HoodieRemoteException: status code: 500, reason phrase: Server Error
Posted by GitBox <gi...@apache.org>.
yihua commented on issue #5460:
URL: https://github.com/apache/hudi/issues/5460#issuecomment-1112635290
@zhilinli123 by "compression failure", do you mean the Hudi compaction fails and `20220428191757755` is a scheduled compaction? Have you tried to restart and retry the Flink job and see the ingestion can proceed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhilinli123 commented on issue #5460: org.apache.hudi.exception.HoodieRemoteException: status code: 500, reason phrase: Server Error
Posted by GitBox <gi...@apache.org>.
zhilinli123 commented on issue #5460:
URL: https://github.com/apache/hudi/issues/5460#issuecomment-1118217639
<img width="1884" alt="image" src="https://user-images.githubusercontent.com/76689593/166872145-73004006-f46d-4b93-a646-2b234604cbb5.png">
[20220505133151119.tar.gz](https://github.com/apache/hudi/files/8629071/20220505133151119.tar.gz)
[temp.tar.gz](https://github.com/apache/hudi/files/8629072/temp.tar.gz)
@yihua @danny0405 You guys, I've downloaded the timeline 20220505133151119. Does that help?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhilinli123 commented on issue #5460: org.apache.hudi.exception.HoodieRemoteException: status code: 500, reason phrase: Server Error
Posted by GitBox <gi...@apache.org>.
zhilinli123 commented on issue #5460:
URL: https://github.com/apache/hudi/issues/5460#issuecomment-1219631899
Thank you guys, I did not reappear in the new version of this problem should have been fixed ~
@yihua @danny0405 @codope
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org