回复：1.11.1 报OutOfMemoryError: Metaspace. 错误

[Asahi Lee <97...@qq.com>]

你好！
       我是用的是flink 1.11.2版本，在MySQL的驱动包中有一下的代码：
/*
  Copyright (c) 2013, 2015, Oracle and/or its affiliates. All rights reserved.

  The MySQL Connector/J is licensed under the terms of the GPLv2
  <http://www.gnu.org/licenses/old-licenses/gpl-2.0.html&gt;, like most MySQL Connectors.
  There are special exceptions to the terms and conditions of the GPLv2 as it is applied to
  this software, see the FOSS License Exception
  <http://www.mysql.com/about/legal/licensing/foss-exception.html&gt;.

  This program is free software; you can redistribute it and/or modify it under the terms
  of the GNU General Public License as published by the Free Software Foundation; version 2
  of the License.

  This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
  without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
  See the GNU General Public License for more details.

  You should have received a copy of the GNU General Public License along with this
  program; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth
  Floor, Boston, MA 02110-1301  USA

 */

package com.mysql.cj.jdbc;

import java.lang.ref.Reference;

import com.mysql.cj.jdbc.NonRegisteringDriver.ConnectionPhantomReference;

public class AbandonedConnectionCleanupThread extends Thread {
    private static boolean running = true;
    private static Thread threadRef = null;

    public AbandonedConnectionCleanupThread() {
        super("Abandoned connection cleanup thread");
    }

    @Override
    public void run() {
        threadRef = this;
        while (running) {
            try {
                Reference<? extends ConnectionImpl&gt; ref = NonRegisteringDriver.refQueue.remove(100);
                if (ref != null) {
                    try {
                        ((ConnectionPhantomReference) ref).cleanup();
                    } finally {
                        NonRegisteringDriver.connectionPhantomRefs.remove(ref);
                    }
                }

            } catch (Exception ex) {
                // no where to really log this if we're static
            }
        }
    }

    public static void shutdown() throws InterruptedException {
        running = false;
        if (threadRef != null) {
            threadRef.interrupt();
            threadRef.join();
            threadRef = null;
        }
    }

}
而上面的shutdown没有被调用，导致有线程停止不了，造成OutOfMemoryError: Metaspace.


------------------&nbsp;原始邮件&nbsp;------------------
发件人:                                                                                                                        "user-zh"                                                                                    <18868816710@163.com&gt;;
发送时间:&nbsp;2020年11月4日(星期三) 晚上9:32
收件人:&nbsp;"user-zh"<user-zh@flink.apache.org&gt;;

主题:&nbsp;Re:1.11.1 报OutOfMemoryError: Metaspace. 错误



Hi Asahi,


你用的 Flink 版本是哪个呢？
对于 Metaspace OOM 的话，如果是稳定在某些值，确实需要加大这个内存大小的话，可以设置 参数 `taskmanager-memory-jvm-metaspace-size` [1]。
对于根本原因的话，需要看下为什么有那么多的无效 Conection，是不是长时间没有数据发送导致 connection invalid，然后再次使用时候就会重启，导致进程级别的缓存无效 Connection对象，这样的话，有线程一直在检测这个对象，就无法 GC。对于 connection 是否 invalid 的处理目前有个 issue[2] 已经解决了。


[1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/config.html#taskmanager-memory-jvm-metaspace-size
[2] https://issues.apache.org/jira/browse/FLINK-16681


Best,
Hailong Wang
在 2020-11-04 19:08:37，"Asahi Lee" <978466273@qq.com&gt; 写道：
&gt;你好！
&gt;&amp;nbsp; &amp;nbsp; &amp;nbsp; 我是用flink sql,通过JDBC源读取mysql表数据，多次提交后，报OutOfMemoryError: Metaspace.错误，最后分析是mysql驱动包中有个守护线程一直在检测清理失效的connection，导致有线程一直存在，class资源无法释放导致的，请问这个如何处理？

Re: 回复：1.11.1 报OutOfMemoryError: Metaspace. 错误

[YueKun <ku...@qingping.co>]

你好，请问这个问题解决了吗？我目前也有遇到这个情况



--
Sent from: http://apache-flink.147419.n8.nabble.com/