You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user-zh@flink.apache.org by 欧阳苗 <a9...@qq.com> on 2020/04/06 07:29:40 UTC
回复： flink 内存溢出 如何排查
你好，
我看了下这篇文档，但是只给出了加大jvm的内存，但是我更想知道的是该如何排查哪一部分代码造成的内存溢出，请问有什么方法吗。
其次我这里的环境是，flink on k8s, 单机版。有一个服务是专门远程flink job,这个服务也是部署在k8s中
------------------ 原始邮件 ------------------
发件人: "Xintong Song"<tonysong820@gmail.com>;
发送时间: 2020年4月6日(星期一) 下午2:56
收件人: "user-zh"<user-zh@flink.apache.org>;
主题: Re: flink 内存溢出 如何排查
你好，
请参考官方文档：部署与运维/内存配置/常见问题/OutOfMemoryError: Java heap space
https://ci.apache.org/projects/flink/flink-docs-release-1.10/zh/ops/memory/mem_trouble.html#outofmemoryerror-java-heap-space
Thank you~
Xintong Song
On Mon, Apr 6, 2020 at 12:33 AM 欧阳苗 <a952064610@qq.com> wrote:
> 2020-04-04 18:58:41.515 ERROR 1 --- [ent-IO-thread-4]
> org.apache.flink.runtime.rest.RestClient.parseResponse:393 : Received
> response was neither of the expected type ([simple type, class
> org.apache.flink.runtime.rest.messages.job.JobExecutionResultResponseBody])
> nor an error.
> Response=JsonResponse{json={"status":{"id":"COMPLETED"},"job-execution-result":{"id":"95bfafbc9b8e31565d44ae0018a0af7d","application-status":"FAILED","accumulator-results":{},"net-runtime":3098,"failure-cause":{"class":"java.lang.OutOfMemoryError","stack-trace":"java.lang.OutOfMemoryError:
> Java heap
> space\n","serialized-throwable":"rO0ABXNyAClvcmcuYXBhY2hlLmZsaW5rLnV0aWwuU2VyaWFsaXplZFRocm93YWJsZWUWnfUfpxPzAgADTAAZZnVsbFN0cmluZ2lmaWVkU3RhY2tUcmFjZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAFm9yaWdpbmFsRXJyb3JDbGFzc05hbWVxAH4AAVsAE3NlcmlhbGl6ZWRFeGNlcHRpb250AAJbQnhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdlcQB+AAFbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBwdAAPSmF2YSBoZWFwIHNwYWNldXIAHltMamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50OwJGKjw8/SI5AgAAeHAAAAAAc3IAJmphdmEudXRpbC5Db2xsZWN0aW9ucyRVbm1vZGlmaWFibGVMaXN0/A8lMbXsjhACAAFMAARsaXN0cQB+AAd4cgAsamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZUNvbGxlY3Rpb24ZQgCAy173HgIAAUwAAWN0ABZMamF2YS91dGlsL0NvbGxlY3Rpb247eHBzcgATamF2YS51dGlsLkFycmF5TGlzdHiB0h2Zx2GdAwABSQAEc2l6ZXhwAAAAAHcEAAAAAHhxAH4AEXh0ACxqYXZhLmxhbmcuT3V0T2ZNZW1vcnlFcnJvcjogSmF2YSBoZWFwIHNwYWNlCnQAGmphdmEubGFuZy5PdXRPZk1lbW9yeUVycm9ydXIAAltCrPMX+AYIVOACAAB4cAAAAf6s7QAFc3IAGmphdmEubGFuZy5PdXRPZk1lbW9yeUVycm9ycjG7cIiI4xUCAAB4cgAdamF2YS5sYW5nLlZpcnR1YWxNYWNoaW5lRXJyb3I5wlZUgC8OHgIAAHhyAA9qYXZhLmxhbmcuRXJyb3JFHTZWi4IOVgIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARMAAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9sYW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBwdAAPSmF2YSBoZWFwIHNwYWNldXIAHltMamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50OwJGKjw8/SI5AgAAeHAAAAABc3IAG2phdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudGEJxZomNt2FAgAESQAKbGluZU51bWJlckwADmRlY2xhcmluZ0NsYXNzcQB+AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFeHCAAAAAdAAAcHEAfgAOcHg="}}},
> httpResponseStatus=200 OK}
>
>
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException:
> Unrecognized field "status" (class
> org.apache.flink.runtime.rest.messages.ErrorResponseBody), not marked as
> ignorable (one known property: "errors"])
> &nbsp;at [Source: N/A; line: -1, column: -1] (through reference chain:
> org.apache.flink.runtime.rest.messages.ErrorResponseBody["status"])
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:62)
> ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.DeserializationContext.reportUnknownProperty(DeserializationContext.java:851)
> ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:1085)
> ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1392)
> ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperties(BeanDeserializerBase.java:1346)
> ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:455)
> ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1127)
> ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:298)
> ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:133)
> ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3779)
> ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2050)
> ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.treeToValue(ObjectMapper.java:2547)
> ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0]
> at
> org.apache.flink.runtime.rest.RestClient.parseResponse(RestClient.java:388)
> ~[flink-runtime_2.12-1.8.0.jar!/:1.8.0]
> at
> org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$3(RestClient.java:373)
> ~[flink-runtime_2.12-1.8.0.jar!/:1.8.0]
> at
> java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:952)
> [?:1.8.0_192]
> at
> java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926)
> [?:1.8.0_192]
> at
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
> [?:1.8.0_192]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0_192]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:1.8.0_192]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]
Re: flink 内存溢出 如何排查
Posted by Xintong Song <to...@gmail.com>.
你说的这种情况，只能是分析 jvm heap dump 看下内存都用在哪里了。
可以试下配置 env.java.opts 添加 -XX:+HeapDumpOnOutOfMemoryError。
Thank you~
Xintong Song
On Mon, Apr 6, 2020 at 3:30 PM 欧阳苗 <a9...@qq.com> wrote:
> 你好，
> 我看了下这篇文档，但是只给出了加大jvm的内存，但是我更想知道的是该如何排查哪一部分代码造成的内存溢出，请问有什么方法吗。
> 其次我这里的环境是，flink on k8s, 单机版。有一个服务是专门远程flink job,这个服务也是部署在k8s中
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "Xintong Song"<tonysong820@gmail.com>;
> 发送时间: 2020年4月6日(星期一) 下午2:56
> 收件人: "user-zh"<user-zh@flink.apache.org>;
>
> 主题: Re: flink 内存溢出 如何排查
>
>
>
> 你好，
>
> 请参考官方文档：部署与运维/内存配置/常见问题/OutOfMemoryError: Java heap space
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.10/zh/ops/memory/mem_trouble.html#outofmemoryerror-java-heap-space
>
> Thank you~
>
> Xintong Song
>
>
>
> On Mon, Apr 6, 2020 at 12:33 AM 欧阳苗 <a952064610@qq.com> wrote:
>
> > 2020-04-04 18:58:41.515 ERROR 1 --- [ent-IO-thread-4]
> > org.apache.flink.runtime.rest.RestClient.parseResponse:393 : Received
> > response was neither of the expected type ([simple type, class
> >
> org.apache.flink.runtime.rest.messages.job.JobExecutionResultResponseBody])
> > nor an error.
> >
> Response=JsonResponse{json={"status":{"id":"COMPLETED"},"job-execution-result":{"id":"95bfafbc9b8e31565d44ae0018a0af7d","application-status":"FAILED","accumulator-results":{},"net-runtime":3098,"failure-cause":{"class":"java.lang.OutOfMemoryError","stack-trace":"java.lang.OutOfMemoryError:
> > Java heap
> >
> space\n","serialized-throwable":"rO0ABXNyAClvcmcuYXBhY2hlLmZsaW5rLnV0aWwuU2VyaWFsaXplZFRocm93YWJsZWUWnfUfpxPzAgADTAAZZnVsbFN0cmluZ2lmaWVkU3RhY2tUcmFjZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAFm9yaWdpbmFsRXJyb3JDbGFzc05hbWVxAH4AAVsAE3NlcmlhbGl6ZWRFeGNlcHRpb250AAJbQnhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdlcQB+AAFbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBwdAAPSmF2YSBoZWFwIHNwYWNldXIAHltMamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50OwJGKjw8/SI5AgAAeHAAAAAAc3IAJmphdmEudXRpbC5Db2xsZWN0aW9ucyRVbm1vZGlmaWFibGVMaXN0/A8lMbXsjhACAAFMAARsaXN0cQB+AAd4cgAsamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZUNvbGxlY3Rpb24ZQgCAy173HgIAAUwAAWN0ABZMamF2YS91dGlsL0NvbGxlY3Rpb247eHBzcgATamF2YS51dGlsLkFycmF5TGlzdHiB0h2Zx2GdAwABSQAEc2l6ZXhwAAAAAHcEAAAAAHhxAH4AEXh0ACxqYXZhLmxhbmcuT3V0T2ZNZW1vcnlFcnJvcjogSmF2YSBoZWFwIHNwYWNlCnQAGmphdmEubGFuZy5PdXRPZk1lbW9yeUVycm9ydXIAAltCrPMX+AYIVOACAAB4cAAAAf6s7QAFc3IAGmphdmEubGFuZy5PdXRPZk1lbW9yeUVycm9ycjG7cIiI4xUCAAB4cgAdamF2YS5sYW5nLlZpcnR1YWxNYWNoaW5lRXJyb3I5wlZUgC8OHgIAAHhyAA9qYXZhLmxhbmcuRXJyb3JFHTZWi4IOVgIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARMAAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9sYW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBwdAAPSmF2YSBoZWFwIHNwYWNldXIAHltMamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50OwJGKjw8/SI5AgAAeHAAAAABc3IAG2phdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudGEJxZomNt2FAgAESQAKbGluZU51bWJlckwADmRlY2xhcmluZ0NsYXNzcQB+AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFeHCAAAAAdAAAcHEAfgAOcHg="}}},
> > httpResponseStatus=200 OK}
> >
> >
> >
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException:
> > Unrecognized field "status" (class
> > org.apache.flink.runtime.rest.messages.ErrorResponseBody), not marked
> as
> > ignorable (one known property: "errors"])
> > &nbsp;at [Source: N/A; line: -1, column: -1] (through reference
> chain:
> > org.apache.flink.runtime.rest.messages.ErrorResponseBody["status"])
> > at
> >
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:62)
> > ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0]
> > at
> >
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.DeserializationContext.reportUnknownProperty(DeserializationContext.java:851)
> > ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0]
> > at
> >
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:1085)
> > ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0]
> > at
> >
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1392)
> > ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0]
> > at
> >
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperties(BeanDeserializerBase.java:1346)
> > ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0]
> > at
> >
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:455)
> > ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0]
> > at
> >
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1127)
> > ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0]
> > at
> >
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:298)
> > ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0]
> > at
> >
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:133)
> > ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0]
> > at
> >
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3779)
> > ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0]
> > at
> >
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2050)
> > ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0]
> > at
> >
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.treeToValue(ObjectMapper.java:2547)
> > ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0]
> > at
> >
> org.apache.flink.runtime.rest.RestClient.parseResponse(RestClient.java:388)
> > ~[flink-runtime_2.12-1.8.0.jar!/:1.8.0]
> > at
> >
> org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$3(RestClient.java:373)
> > ~[flink-runtime_2.12-1.8.0.jar!/:1.8.0]
> > at
> >
> java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:952)
> > [?:1.8.0_192]
> > at
> >
> java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926)
> > [?:1.8.0_192]
> > at
> >
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
> > [?:1.8.0_192]
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > [?:1.8.0_192]
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > [?:1.8.0_192]
> > at
> java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]