You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user-zh@flink.apache.org by Asahi Lee <97...@qq.com.INVALID> on 2021/11/19 02:47:48 UTC
回复： 哪里可以下载到downloads/setup-pyflink-virtual-env.sh脚本
我通过source my_env/bin/activate和指定环境变量PYFLINK_CLIENT_EXECUTABLE均未提交成功，错误未变化；
我的错误中，jobmanager的日志中显示No module named pyflink，而jobmanager是运行在yarn集群上；
请问，还有还有什么未配置？
> LogType:jobmanager.out
> Log Upload Time:星期四 十一月 18 20:48:45 +0800 2021
> LogLength:37
> Log Contents:
> /bin/python: No module named pyflink
------------------ 原始邮件 ------------------
发件人: "user-zh" <dian0511.fu@gmail.com>;
发送时间: 2021年11月19日(星期五) 上午9:38
收件人: "user-zh"<user-zh@flink.apache.org>;
主题: Re: 哪里可以下载到downloads/setup-pyflink-virtual-env.sh脚本
-pyexec 指定的是集群端所用的Python环境，客户端需要编译Flink作业，也会依赖Python环境。可以看一下这个文档:
https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/dev/python/dependency_management/#python-interpreter-of-client
On Thu, Nov 18, 2021 at 9:00 PM Asahi Lee <978466273@qq.com.invalid> wrote:
> Hi !
> &nbsp; &nbsp;我在java Table api中使用python udf
> 函数，通过下面的命令提交应用，报无法启动python服务错误，请问我的提交方式对吗？jm日志为/bin/python: No module named
> pyflink。
>
>
> ./flink-1.13.2/bin/flink&nbsp;
> run-application -t yarn-application&nbsp;
> -Dyarn.provided.lib.dirs="hdfs://nameservice1/user/flink/flinklib"&nbsp;
> -Dyarn.application.queue=d
> -p 1&nbsp;
> -pyarch /opt/venv.zip
> -pyexec venv.zip/venv/bin/python&nbsp;
> -pyfs /opt/test.py&nbsp;
> -c test.PyUDFTest&nbsp;
> /opt/flink-python-test-1.0-SNAPSHOT.jar
>
>
>
> 错误：
> Caused by: java.lang.RuntimeException: Python callback server start failed!
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.flink.client.python.PythonFunctionFactory.createPythonFunctionFactory(PythonFunctionFactory.java:167)
> ~[flink-python_2.11-1.13.2.jar:1.13.2]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.flink.client.python.PythonFunctionFactory$1.load(PythonFunctionFactory.java:88)
> ~[flink-python_2.11-1.13.2.jar:1.13.2]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.flink.client.python.PythonFunctionFactory$1.load(PythonFunctionFactory.java:84)
> ~[flink-python_2.11-1.13.2.jar:1.13.2]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
> ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
> ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
> ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
> ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
> ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
> ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.flink.client.python.PythonFunctionFactory.getPythonFunction(PythonFunctionFactory.java:129)
> ~[flink-python_2.11-1.13.2.jar:1.13.2]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_111]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_111]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_111]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.flink.table.functions.python.utils.PythonFunctionUtils.getPythonFunction(PythonFunctionUtils.java:45)
> ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.flink.table.functions.UserDefinedFunctionHelper.instantiateFunction(UserDefinedFunctionHelper.java:206)
> ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.flink.table.catalog.FunctionCatalog.getFunctionDefinition(FunctionCatalog.java:659)
> ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.flink.table.catalog.FunctionCatalog.resolveAmbiguousFunctionReference(FunctionCatalog.java:606)
> ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.flink.table.catalog.FunctionCatalog.lookupFunction(FunctionCatalog.java:362)
> ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.flink.table.planner.catalog.FunctionCatalogOperatorTable.lookupOperatorOverloads(FunctionCatalogOperatorTable.java:97)
> ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.calcite.sql.util.ChainedSqlOperatorTable.lookupOperatorOverloads(ChainedSqlOperatorTable.java:67)
> ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1183)
> ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1169)
> ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1200)
> ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1169)
> ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:945)
> ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:704)
> ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:152)
> ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:110)
> ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:201)
> ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:101)
> ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:704)
> ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; at test.PyUDFTest.main(PyUDFTest.java:22)
> ~[flink-python-test-1.0-SNAPSHOT.jar:?]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_111]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_111]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_111]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
> ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
> ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; at
> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:242)
> ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &nbsp; &nbsp; &nbsp; &nbsp; ... 10 more
> 2021-11-18 20:48:44,458 INFO&nbsp;
> org.apache.flink.runtime.blob.BlobServer&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;[] - Stopped BLOB server at
> 0.0.0.0:45070
> 2021-11-18 20:48:44,458 INFO&nbsp;
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint&nbsp; &nbsp; &nbsp;
> &nbsp; [] - Shutting YarnApplicationClusterEntryPoint down with application
> status UNKNOWN. Diagnostics Cluster entrypoint has been closed externally..
> 2021-11-18 20:48:44,458 INFO&nbsp;
> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Shutting
> down rest endpoint.
> 2021-11-18 20:48:44,474 INFO&nbsp;
> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Removing
> cache directory
> /tmp/flink-web-dd82c3c0-f457-492d-8e64-5ae74fe9abbd/flink-web-ui
> 2021-11-18 20:48:44,474 INFO&nbsp;
> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] -
> http://cdh5node3:40216 lost leadership
> 2021-11-18 20:48:44,474 INFO&nbsp;
> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Shut
> down complete.
> 2021-11-18 20:48:44,474 INFO&nbsp;
> org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent
> [] - Closing components.
> 2021-11-18 20:48:44,475 INFO&nbsp;
> org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess
> [] - Stopping SessionDispatcherLeaderProcess.
> 2021-11-18 20:48:44,475 INFO&nbsp;
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher&nbsp; &nbsp;
> &nbsp;[] - Stopping dispatcher akka.tcp://flink@cdh5node3
> :34697/user/rpc/dispatcher_1.
> 2021-11-18 20:48:44,476 INFO&nbsp;
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher&nbsp; &nbsp;
> &nbsp;[] - Stopping all currently running jobs of dispatcher
> akka.tcp://flink@cdh5node3:34697/user/rpc/dispatcher_1.
>
>
> LogType:jobmanager.out
> Log Upload Time:星期四 十一月 18 20:48:45 +0800 2021
> LogLength:37
> Log Contents:
> /bin/python: No module named pyflink
>
>
>
>
>
>
>
>
>
>
>
>
>
> ------------------&nbsp;原始邮件&nbsp;------------------
> 发件人:
> "user-zh"
> <
> xyzhong131@163.com&gt;;
> 发送时间:&nbsp;2021年11月18日(星期四) 下午3:34
> 收件人:&nbsp;"user-zh"<user-zh@flink.apache.org&gt;;
>
> 主题:&nbsp;Re:哪里可以下载到downloads/setup-pyflink-virtual-env.sh脚本
>
>
>
> Hi!<br/&gt;我在下面的旧版本中找到了，你试下能不能用：<br/&gt;
> https://nightlies.apache.org/flink/flink-docs-release-1.12/downloads/setup-pyflink-virtual-env.sh
> 在 2021-11-18 15:05:03，"Asahi Lee" <978466273@qq.com.INVALID&gt; 写道：
> &gt;Hi!
> &gt;&amp;nbsp; &amp;nbsp;
> 我在flink官方文档中看到通过&amp;nbsp;&amp;nbsp;setup-pyflink-virtual-env.sh
> 脚本制作python虚拟环境，请问再哪里可以下载到？文档地址如下：
> &gt;
> https://nightlies.apache.org/flink/flink-docs-release-1.14/zh/docs/dev/python/faq/
回复： 哪里可以下载到downloads/setup-pyflink-virtual-env.sh脚本
Posted by Asahi Lee <as...@qq.com.INVALID>.
我忘记重新构建venv.zip了，我重新构建1.14.0版本后，程序正确运行了，谢谢！
------------------ 原始邮件 ------------------
发件人: "user-zh" <dian0511.fu@gmail.com>;
发送时间: 2021年11月22日(星期一) 上午9:24
收件人: "user-zh"<user-zh@flink.apache.org>;
主题: Re: 哪里可以下载到downloads/setup-pyflink-virtual-env.sh脚本
Flink用的1.14.0，venv.zip中的PyFlink版本是多少？
On Sun, Nov 21, 2021 at 7:59 PM Asahi Lee <asahi.lee@qq.com.invalid> wrote:
> Hi!
> &nbsp; &nbsp;我通过如下命令提交成功了，python的参数需求-D方式传入，-py方式传入不生效：
> ./flink-1.14.0/bin/flink
> run-application -t yarn-application
> -Dyarn.provided.lib.dirs="hdfs://nameservice1/user/flink/flinklib"
> -Dyarn.application.queue=d
> -Dpython.archives="/opt/venv.zip"
> -Dpython.client.executable="venv.zip/venv/bin/python"
> -Dpython.executable="venv.zip/venv/bin/python"
> -Dpython.files="/opt/test.py"
> -p 1
> -c test.PyUDFTest
> /opt/flink-python-test-1.0-SNAPSHOT.jar
>
>
>
> 但是我应用提交成功后，程序运行发生错误，如下：
>
>
> taskmanager.log
>
>
> 2021-11-19 13:59:24,030 ERROR
> /yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py:296
> [] - Error processing instruction 1. Original traceback is
> Traceback (most recent call last):
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 289, in _execute
> &nbsp; &nbsp; response = task()
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 362, in <lambda&gt;
> &nbsp; &nbsp; lambda: self.create_worker().do_instruction(request),
> request)
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 607, in do_instruction
> &nbsp; &nbsp; getattr(request, request_type), request.instruction_id)
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 638, in process_bundle
> &nbsp; &nbsp; instruction_id, request.process_bundle_descriptor_id)
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 467, in get
> &nbsp; &nbsp; self.data_channel_factory)
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 868, in __init__
> &nbsp; &nbsp; self.ops =
> self.create_execution_tree(self.process_bundle_descriptor)
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 925, in create_execution_tree
> &nbsp; &nbsp; descriptor.transforms, key=topological_height,
> reverse=True)])
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 924, in <listcomp&gt;
> &nbsp; &nbsp; get_operation(transform_id))) for transform_id in sorted(
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 812, in wrapper
> &nbsp; &nbsp; result = cache[args] = func(*args)
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 906, in get_operation
> &nbsp; &nbsp; pcoll_id in
> descriptor.transforms[transform_id].outputs.items()
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 906, in <dictcomp&gt;
> &nbsp; &nbsp; pcoll_id in
> descriptor.transforms[transform_id].outputs.items()
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 904, in <listcomp&gt;
> &nbsp; &nbsp; tag: [get_operation(op) for op in pcoll_consumers[pcoll_id]]
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 812, in wrapper
> &nbsp; &nbsp; result = cache[args] = func(*args)
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 906, in get_operation
> &nbsp; &nbsp; pcoll_id in
> descriptor.transforms[transform_id].outputs.items()
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 906, in <dictcomp&gt;
> &nbsp; &nbsp; pcoll_id in
> descriptor.transforms[transform_id].outputs.items()
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 904, in <listcomp&gt;
> &nbsp; &nbsp; tag: [get_operation(op) for op in pcoll_consumers[pcoll_id]]
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 812, in wrapper
> &nbsp; &nbsp; result = cache[args] = func(*args)
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 909, in get_operation
> &nbsp; &nbsp; transform_id, transform_consumers)
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 1198, in create_operation
> &nbsp; &nbsp; return creator(self, transform_id, transform_proto, payload,
> consumers)
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 1320, in create_sink_runner
> &nbsp; &nbsp; output_coder = factory.get_coder(grpc_port.coder_id)
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 1222, in get_coder
> &nbsp; &nbsp; return self.context.coders.get_by_id(coder_id)
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/pipeline_context.py",
> line 116, in get_by_id
> &nbsp; &nbsp; self._id_to_proto[id], self._pipeline_context)
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/coders/coders.py",
> line 370, in from_runner_api
> &nbsp; &nbsp; [context.coders.get_by_id(c) for c in
> coder_proto.component_coder_ids],
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/coders/coders.py",
> line 370, in <listcomp&gt;
> &nbsp; &nbsp; [context.coders.get_by_id(c) for c in
> coder_proto.component_coder_ids],
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/pipeline_context.py",
> line 116, in get_by_id
> &nbsp; &nbsp; self._id_to_proto[id], self._pipeline_context)
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/coders/coders.py",
> line 370, in from_runner_api
> &nbsp; &nbsp; [context.coders.get_by_id(c) for c in
> coder_proto.component_coder_ids],
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/coders/coders.py",
> line 370, in <listcomp&gt;
> &nbsp; &nbsp; [context.coders.get_by_id(c) for c in
> coder_proto.component_coder_ids],
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/pipeline_context.py",
> line 116, in get_by_id
> &nbsp; &nbsp; self._id_to_proto[id], self._pipeline_context)
> &nbsp; File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/coders/coders.py",
> line 367, in from_runner_api
> &nbsp; &nbsp; parameter_type, constructor =
> cls._known_urns[coder_proto.spec.urn]
> KeyError: 'flink:coder:v1'
>
>
> &nbsp;
> 2021-11-19 14:02:24,031 WARN&nbsp;
> org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory [] -
> Expiring environment urn: "beam:env:process:v1"
> payload:
> "\032\321\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/pyflink/bin/pyflink-udf-runner.sh\"%\n\004PATH\022\035/sbin:/usr/sbin:/bin:/usr/bin\"V\n\017HADOOP_CONF_DIR\022C/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER\"\031\n\vHISTCONTROL\022\nignoredups\"\027\n\022SUPERVISOR_ENABLED\022\0011\"\031\n\021CONDA_DEFAULT_ENV\022\004base\"#\n\026jobmanager.rpc.address\022\tcdh5node1\"4\n\026KEYTRUSTEE_SERVER_HOME\022\032/usr/lib/keytrustee-server\"-\n\020CONDA_PYTHON_EXE\022\031/opt/anaconda3/bin/python\"\027\n\023CGROUP_GROUP_MEMORY\022\000\"$\n\023table.exec.timezone\022\rAsia/Shanghai\"Z\n\026CDH_HBASE_INDEXER_HOME\022@/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hbase-solr\"\036\n\fCONDA_PREFIX\022\016/opt/anaconda3\"\233\001\n\vio.tmp.dirs\022\213\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150,/hdfs/yarn/usercache/tempodata/appcache/application_1637293573159_0150\"\222\b\n\tYARN_OPTS\022\204\b-Djava.net.preferIPv4Stack=true&nbsp;
> -server -Xms1073741824 -Xmx1073741824 -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
> -XX:+CMSParallelRemarkEnabled
> -Dlibrary.leveldbjni.path=/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER
> -Dhadoop.event.appender=,EventCatcher -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=/tmp/yarn_yarn-NODEMANAGER-bef961adb705d3d90452581c3b4991c4_pid31718.hprof
> -XX:OnOutOfMemoryError=/opt/cm-5.16.2/lib64/cmf/service/common/killparent.sh
> -Dhadoop.log.dir=/var/log/hadoop-yarn -Dyarn.log.dir=/var/log/hadoop-yarn
> -Dhadoop.log.file=hadoop-cmf-yarn-NODEMANAGER-cdh5node1.log.out
> -Dyarn.log.file=hadoop-cmf-yarn-NODEMANAGER-cdh5node1.log.out
> -Dyarn.home.dir=/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn
> -Dhadoop.home.dir=/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn
> -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA
> -Djava.library.path=/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/lib/native\"!\n\031HADOOP_HOME_WARN_SUPPRESS\022\004true\"\017\n\aJVM_PID\022\0045524\"O\n\tJSVC_HOME\022B/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/bigtop-utils\"w\n\003PWD\022p/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/container_e176_1637293573159_0150_01_000002\"V\n\017CDH_HTTPFS_HOME\022C/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-httpfs\"P\n\016CDH_FLUME_HOME\022&gt;/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/flume-ng\"Q\n\020CDH_PARQUET_HOME\022=/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/parquet\"\"\n\006python\022\030venv.zip/venv/bin/python\"%\n\bLESSOPEN\022\031||/usr/bin/lesspipe.sh
> %s\"\035\n\031SELINUX_USE_CURRENT_RANGE\022\000\"\034\n\fpython.files\022\f/opt/test.py\"O\n\017CDH_HADOOP_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop\"A\n\nweb.tmpdir\0223/tmp/flink-web-e35b2953-cfce-4655-bbac-bbacf1dbba5e\"\235\001\n\023_PYTHON_WORKING_DIR\022\205\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives\"\035\n\vPYTHON_HOME\022\016/opt/anaconda3\"H\n\034CLOUDERA_MYSQL_CONNECTOR_JAR\022(/usr/share/java/mysql-connector-java.jar\"I\n\fCDH_HUE_HOME\0229/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hue\"
> \n\034NM_AUX_SERVICE_spark_shuffle\022\000\"1\n
> __table.query-start.local-time__\022\r1637330289401\"8\n\020HIVE_DEFAULT_XML\022$/etc/hive/conf.dist/hive-default.xml\"\031\n\025CONDA_PROMPT_MODIFIER\022\000\"\n\n\005SHLVL\022\0014\"I\n\036$internal.yarn.log-config-file\022\'/opt/flink-1.14.0/conf/log4j.properties\"u\n\035python.internal.files-key-map\022Tpython_file_13c795ab581a80efa6a9947d919503287f51d055172132c4457dc7e58ad114cc:test.py\"3\n\017CMF_PACKAGE_DIR\022
> /opt/cm-5.16.2/lib64/cmf/service\"-\n\017CGROUP_ROOT_CPU\022\032/sys/fs/cgroup/cpu,cpuacct\"%\n\tCONDA_EXE\022\030/opt/anaconda3/bin/conda\"\020\n\bHISTSIZE\022\0041000\"\036\n\tJAVA_HOME\022\021/usr/java/default\"5\n\fCMF_CONF_DIR\022%/opt/cm-5.16.2/etc/cloudera-scm-agent\"0\n\"jobmanager.memory.jvm-overhead.min\022\n201326592b\"\r\n\004TERM\022\005xterm\"R\n\rCDH_HDFS_HOME\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-hdfs\"\023\n\016XDG_SESSION_ID\022\0017\"\033\n\027SELINUX_LEVEL_REQUESTED\022\000\"=\n\fYARN_LOGFILE\022-hadoop-cmf-yarn-NODEMANAGER-cdh5node1.log.out\"+\n\020HADOOP_USER_NAME\022\
> 027tempodata@MERITDATA.COM\"3\n*execution.savepoint.ignore-unclaimed-state\022\005false\"X\n\023SCM_DEFINES_SCRIPTS\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/meta/cdh_env.sh\"(\n\033_PYTHON_WORKER_MEMORY_LIMIT\022\t536870920\"U\n\020HADOOP_HDFS_HOME\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-hdfs\"\\\n\022HADOOP_MAPRED_HOME\022F/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-mapreduce\"1\n\023CGROUP_ROOT_CPUACCT\022\032/sys/fs/cgroup/cpu,cpuacct\"/\n\023CMF_AGENT_MGMT_HOME\022\030/opt/cm-5.16.2/share/cmf\"\037\n\001_\022\032/usr/java/default/bin/java\"%\n\tMGMT_HOME\022\030/opt/cm-5.16.2/share/cmf\"\026\n\022CGROUP_GROUP_BLKIO\022\000\"\025\n\aSSH_TTY\022\n/dev/pts/2\"#\n\nSSH_CLIENT\022\025191.168.1.34
> 61192 22\"
> \n\030PYFLINK_GATEWAY_DISABLED\022\004true\"K\n\rCDH_AVRO_HOME\022:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/avro\"
> \n\016CDH_SQOOP_HOME\022\016/usr/lib/sqoop\"T\n\024CDH_HUE_PLUGINS_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop\"\'\n\036jobmanager.memory.process.size\022\0051600m\"\020\n\vCDH_VERSION\022\0015\"!\n\035HADOOP_JOB_HISTORYSERVER_OPTS\022\000\"
> \n\016CDH_KAFKA_HOME\022\016/usr/lib/kafka\"J\n\035CLOUDERA_ORACLE_CONNECTOR_JAR\022)/usr/share/java/oracle-connector-java.jar\"4\n\016SSH_CONNECTION\022\"191.168.1.34
> 61192 191.168.1.24
> 22\")\n\037internal.cluster.execution-mode\022\006NORMAL\"(\n\aNLSPATH\022\035/usr/dt/lib/nls/msg/%L/%N.cat\"7\n\022KEYTRUSTEE_KP_HOME\022!/usr/share/keytrustee-keyprovider\"+\n\017CMF_SERVER_ROOT\022\030/opt/cm-5.16.2/share/cmf\"\035\n\aCMF_ETC\022\022/opt/cm-5.16.2/etc\"2\n$jobmanager.memory.jvm-metaspace.size\022\n268435456b\"M\n\016CDH_SPARK_HOME\022;/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/spark\"\030\n\024CGROUP_GROUP_CPUACCT\022\000\";\n\fCONTAINER_ID\022+container_e176_1637293573159_0150_01_000002\"M\n\016CDH_LLAMA_HOME\022;/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/llama\"t\n\rXDG_DATA_DIRS\022c/root/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share\"K\n\rCDH_KUDU_HOME\022:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/kudu\"%\n\fPARCELS_ROOT\022\025/opt/cloudera/parcels\"&gt;\n\034high-availability.cluster-id\022\036application_1637293573159_0150\"\035\n\031YARN_RESOURCEMANAGER_OPTS\022\000\"4\n\030python.client.executable\022\030venv.zip/venv/bin/python\"\026\n\rtable.planner\022\005BLINK\"\034\n\004MAIL\022\024/var/spool/mail/root\"1\n
> __table.query-start.epoch-time__\022\r1637301489401\"\030\n\023parallelism.default\022\0011\"\027\n\023pipeline.classpaths\022\000\"M\n\016CDH_HBASE_HOME\022;/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hbase\"\024\n\aLOGNAME\022\ttempodata\"V\n\fCDH_MR2_HOME\022F/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-mapreduce\"R\n\vTOMCAT_HOME\022C/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/bigtop-tomcat\"\033\n\016_FLINK_NODE_ID\022\tcdh5node1\"I\n\fCDH_PIG_HOME\0229/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/pig\"\034\n\026yarn.application.queue\022\002df\"M\n\rHADOOP_PREFIX\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop\"\232\001\n\nLOCAL_DIRS\022\213\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150,/hdfs/yarn/usercache/tempodata/appcache/application_1637293573159_0150\"\342\001\n\nPYTHONPATH\022\323\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-files/blob_p-30842d5cb48fe53fd9f94c2a4e364225246729cc-b3cef870d37841a25e243a9ca6b64863\"(\n\037taskmanager.memory.process.size\022\0051728m\"\022\n\005SHELL\022\t/bin/bash\"\r\n\bweb.port\022\0010\"8\n\026HADOOP_CLIENT_CONF_DIR\022\036/etc/hadoop/conf.cloudera.yarn\"U\n\022CDH_ZOOKEEPER_HOME\022?/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/zookeeper\"T\n\rYARN_CONF_DIR\022C/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER\"-\n\037jobmanager.memory.off-heap.size\022\n134217728b\"|\n
> python.internal.archives-key-map\022Xpython_archive_bfe925aa9bea9ad79f3e1b595cb3e39a3acfa4656b3a5870173959195400aabb:venv.zip\"\321\001\n\bLOG_DIRS\022\304\001/yarn/container-logs/application_1637293573159_0150/container_e176_1637293573159_0150_01_000002,/hdfs/yarn-container-logs/application_1637293573159_0150/container_e176_1637293573159_0150_01_000002\"R\n
> NM_AUX_SERVICE_mapreduce_shuffle\022.AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=\r\n\"W\n\034CLOUDERA_POSTGRESQL_JDBC_JAR\0227/opt/cm-5.16.2/share/cmf/lib/postgresql-42.1.4.jre7.jar\"U\n\020HADOOP_YARN_HOME\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn\".\n\017PARCEL_DIRNAMES\022\033CDH-5.16.2-1.cdh5.16.2.p0.8\"a\n\025SUPERVISOR_SERVER_URL\022Hunix:///opt/cm-5.16.2/run/cloudera-scm-agent/supervisor/supervisord.sock\"\034\n\023jobmanager.rpc.port\022\00539719\"\"\n\vCMF_SBINDIR\022\023/opt/cm-5.16.2/sbin\"w\n\023WEBHCAT_DEFAULT_XML\022`/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/etc/hive-webhcat/conf.dist/webhcat-default.xml\"\"\n\017CDH_CRUNCH_HOME\022\017/usr/lib/crunch\"0\n\027SUPERVISOR_PROCESS_NAME\022\0253224-yarn-NODEMANAGER\".\n\025SUPERVISOR_GROUP_NAME\022\0253224-yarn-NODEMANAGER\"-\n\021python.executable\022\030venv.zip/venv/bin/python\"\373\004\n\020_FLINK_CLASSPATH\022\346\004:flink-python-test-1.0-SNAPSHOT.jar:lib/flink-csv-1.14.0.jar:lib/flink-json-1.14.0.jar:lib/flink-python_2.11-1.14.0.jar:lib/flink-shaded-zookeeper-3.4.14.jar:lib/flink-table_2.11-1.14.0.jar:lib/lib/flink-csv-1.14.0.jar:lib/lib/flink-json-1.14.0.jar:lib/lib/flink-shaded-zookeeper-3.4.14.jar:lib/lib/flink-table_2.11-1.14.0.jar:lib/lib/log4j-1.2-api-2.14.1.jar:lib/lib/log4j-api-2.14.1.jar:lib/lib/log4j-core-2.14.1.jar:lib/lib/log4j-slf4j-impl-2.14.1.jar:lib/log4j-1.2-api-2.14.1.jar:lib/log4j-api-2.14.1.jar:lib/log4j-core-2.14.1.jar:lib/log4j-slf4j-impl-2.14.1.jar:lib/flink-dist_2.11-1.14.0.jar:flink-conf.yaml:\")\n\021CGROUP_ROOT_BLKIO\022\024/sys/fs/cgroup/blkio\"\031\n\frest.address\022\tcdh5node1\"K\n\rCDH_SOLR_HOME\022:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/solr\"\237\v\n\tLS_COLORS\022\221\vrs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:\"T\n\rCDH_HCAT_HOME\022C/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hive-hcatalog\"9\n\037$internal.deployment.config-dir\022\026/opt/flink-1.14.0/conf\"0\n\"jobmanager.memory.jvm-overhead.max\022\n201326592b\"M\n\016CDH_OOZIE_HOME\022;/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/oozie\"O\n\017CDH_SENTRY_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/sentry\"
> \n\004ROOT\022\030/opt/cm-5.16.2/lib64/cmf\"\034\n\020YARN_ROOT_LOGGER\022\bINFO,RFA\"0\n&amp;jobmanager.execution.failover-strategy\022\006region\"\031\n\aCMF_VAR\022\016/opt/cm-5.16.2\"O\n\bCONF_DIR\022C/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER\"-\n\017XFILESEARCHPATH\022\032/usr/dt/app-defaults/%L/Dt\"\037\n\026execution.runtime-mode\022\005BATCH\"\023\n\004LANG\022\vzh_CN.UTF-8\"\021\n\fCMF_SUDO_CMD\022\001
> \"\024\n\aNM_HOST\022\tcdh5node1\"\205\001\n\fBOOT_LOG_DIR\022u/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69\"\031\n\aDISPLAY\022\016localhost:12.0\"Z\n\022HADOOP_LIBEXEC_DIR\022D/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec\"\221\a\n\020HADOOP_CLASSPATH\022\374\006/opt/cm-5.16.2/share/cmf/lib/plugins/event-publish-5.16.2-shaded.jar:/opt/cm-5.16.2/share/cmf/lib/plugins/tt-instrumentation-5.16.2.jar:/etc/hadoop/conf:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-yarn/.//*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*\"\032\n\026SELINUX_ROLE_REQUESTED\022\000\"R\n\026yarn.provided.lib.dirs\0228hdfs://nameservice1/user/tempodata/lixu/flink-1.14.0/lib\"\025\n\rIS_KERBERIZED\022\004true\"+\n\022CGROUP_ROOT_MEMORY\022\025/sys/fs/cgroup/memory\"$\n\fYARN_LOG_DIR\022\024/var/log/hadoop-yarn\"
> \n\017python.archives\022\r/opt/venv.zip\"\"\n\035taskmanager.numberOfTaskSlots\022\0011\"\024\n\020CGROUP_GROUP_CPU\022\000\".\n\vORACLE_HOME\022\037/usr/share/oracle/instantclient\"O\n\017CDH_SQOOP2_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/sqoop2\"R\n\022HADOOP_COMMON_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop\"\020\n\vCONDA_SHLVL\022\0011\"M\n\032$internal.application.main\022/com.meritdata.cloud.flink.python.test.PyUDFTest\"K\n\rCDH_HIVE_HOME\022:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hive\"P\n\fCDH_KMS_HOME\022@/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-kms\"\024\n\fNM_HTTP_PORT\022\0048042\"O\n\017CDH_IMPALA_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/impala\"*\n\016CMF_AGENT_ROOT\022\030/opt/cm-5.16.2/lib64/cmf\"$\n\rNM_LOCAL_DIRS\022\023/yarn/nm
> /hdfs/yarn\"\017\n\aNM_PORT\022\0048041\"\361\003\n\025YARN_NODEMANAGER_OPTS\022\327\003-Xms1073741824
> -Xmx1073741824 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled
> -Dlibrary.leveldbjni.path=/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER
> -Dhadoop.event.appender=,EventCatcher -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=/tmp/yarn_yarn-NODEMANAGER-bef961adb705d3d90452581c3b4991c4_pid31718.hprof
> -XX:OnOutOfMemoryError=/opt/cm-5.16.2/lib64/cmf/service/common/killparent.sh\"\034\n\020execution.target\022\bembedded\"\021\n\004USER\022\ttempodata\"\361\b\n\tCLASSPATH\022\343\b:flink-python-test-1.0-SNAPSHOT.jar:lib/flink-csv-1.14.0.jar:lib/flink-json-1.14.0.jar:lib/flink-python_2.11-1.14.0.jar:lib/flink-shaded-zookeeper-3.4.14.jar:lib/flink-table_2.11-1.14.0.jar:lib/lib/flink-csv-1.14.0.jar:lib/lib/flink-json-1.14.0.jar:lib/lib/flink-shaded-zookeeper-3.4.14.jar:lib/lib/flink-table_2.11-1.14.0.jar:lib/lib/log4j-1.2-api-2.14.1.jar:lib/lib/log4j-api-2.14.1.jar:lib/lib/log4j-core-2.14.1.jar:lib/lib/log4j-slf4j-impl-2.14.1.jar:lib/log4j-1.2-api-2.14.1.jar:lib/log4j-api-2.14.1.jar:lib/log4j-core-2.14.1.jar:lib/log4j-slf4j-impl-2.14.1.jar:lib/flink-dist_2.11-1.14.0.jar:flink-conf.yaml::/etc/hadoop/conf.cloudera.yarn:/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-hdfs/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn/lib/*\"-\n%
> internal.io.tmpdirs.use-local-default\022\004true\"Y\n\016CDH_HADOOP_BIN\022G/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/bin/hadoop\"[\n\fCDH_MR1_HOME\022K/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-0.20-mapreduce\"\240\001\n\032HADOOP_TOKEN_FILE_LOCATION\022\201\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/container_e176_1637293573159_0150_01_000002/container_tokens\"\025\n\bHOSTNAME\022\tcdh5node1\"\032\n\022execution.attached\022\004true\",\n#execution.shutdown-on-attached-exit\022\005false\"\252\001\n\rpipeline.jars\022\230\001file:/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/container_e176_1637293573159_0150_01_000001/flink-python-test-1.0-SNAPSHOT.jar\"\221\001\n\017CM_STATUS_CODES\022~STATUS_NONE
> HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED
> JOBTRACKER_IN_STANDBY_MODE
> YARN_RM_IN_STANDBY_MODE\"R\n\rCDH_YARN_HOME\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn\"K\n\vSEARCH_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/search\"\036\n\017XDG_RUNTIME_DIR\022\v/run/user/0\"\016\n\004HOME\022\006/home/\"*\n\033jobmanager.memory.heap.size\022\v1073741824b\"\025\n\020MALLOC_ARENA_MAX\022\0014"
> &nbsp;with 1 remaining bundle references. Taking note to clean it up
> during shutdown if the references are not removed by then.
> 2021-11-19 14:02:24,046 INFO&nbsp;
> org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory [] -
> Closing environment urn: "beam:env:process:v1"
> payload:
> "\032\321\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/pyflink/bin/pyflink-udf-runner.sh\"%\n\004PATH\022\035/sbin:/usr/sbin:/bin:/usr/bin\"V\n\017HADOOP_CONF_DIR\022C/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER\"\031\n\vHISTCONTROL\022\nignoredups\"\027\n\022SUPERVISOR_ENABLED\022\0011\"\031\n\021CONDA_DEFAULT_ENV\022\004base\"#\n\026jobmanager.rpc.address\022\tcdh5node1\"4\n\026KEYTRUSTEE_SERVER_HOME\022\032/usr/lib/keytrustee-server\"-\n\020CONDA_PYTHON_EXE\022\031/opt/anaconda3/bin/python\"\027\n\023CGROUP_GROUP_MEMORY\022\000\"$\n\023table.exec.timezone\022\rAsia/Shanghai\"Z\n\026CDH_HBASE_INDEXER_HOME\022@/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hbase-solr\"\036\n\fCONDA_PREFIX\022\016/opt/anaconda3\"\233\001\n\vio.tmp.dirs\022\213\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150,/hdfs/yarn/usercache/tempodata/appcache/application_1637293573159_0150\"\222\b\n\tYARN_OPTS\022\204\b-Djava.net.preferIPv4Stack=true&nbsp;
> -server -Xms1073741824 -Xmx1073741824 -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
> -XX:+CMSParallelRemarkEnabled
> -Dlibrary.leveldbjni.path=/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER
> -Dhadoop.event.appender=,EventCatcher -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=/tmp/yarn_yarn-NODEMANAGER-bef961adb705d3d90452581c3b4991c4_pid31718.hprof
> -XX:OnOutOfMemoryError=/opt/cm-5.16.2/lib64/cmf/service/common/killparent.sh
> -Dhadoop.log.dir=/var/log/hadoop-yarn -Dyarn.log.dir=/var/log/hadoop-yarn
> -Dhadoop.log.file=hadoop-cmf-yarn-NODEMANAGER-cdh5node1.log.out
> -Dyarn.log.file=hadoop-cmf-yarn-NODEMANAGER-cdh5node1.log.out
> -Dyarn.home.dir=/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn
> -Dhadoop.home.dir=/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn
> -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA
> -Djava.library.path=/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/lib/native\"!\n\031HADOOP_HOME_WARN_SUPPRESS\022\004true\"\017\n\aJVM_PID\022\0045524\"O\n\tJSVC_HOME\022B/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/bigtop-utils\"w\n\003PWD\022p/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/container_e176_1637293573159_0150_01_000002\"V\n\017CDH_HTTPFS_HOME\022C/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-httpfs\"P\n\016CDH_FLUME_HOME\022&gt;/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/flume-ng\"Q\n\020CDH_PARQUET_HOME\022=/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/parquet\"\"\n\006python\022\030venv.zip/venv/bin/python\"%\n\bLESSOPEN\022\031||/usr/bin/lesspipe.sh
> %s\"\035\n\031SELINUX_USE_CURRENT_RANGE\022\000\"\034\n\fpython.files\022\f/opt/test.py\"O\n\017CDH_HADOOP_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop\"A\n\nweb.tmpdir\0223/tmp/flink-web-e35b2953-cfce-4655-bbac-bbacf1dbba5e\"\235\001\n\023_PYTHON_WORKING_DIR\022\205\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives\"\035\n\vPYTHON_HOME\022\016/opt/anaconda3\"H\n\034CLOUDERA_MYSQL_CONNECTOR_JAR\022(/usr/share/java/mysql-connector-java.jar\"I\n\fCDH_HUE_HOME\0229/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hue\"
> \n\034NM_AUX_SERVICE_spark_shuffle\022\000\"1\n
> __table.query-start.local-time__\022\r1637330289401\"8\n\020HIVE_DEFAULT_XML\022$/etc/hive/conf.dist/hive-default.xml\"\031\n\025CONDA_PROMPT_MODIFIER\022\000\"\n\n\005SHLVL\022\0014\"I\n\036$internal.yarn.log-config-file\022\'/opt/flink-1.14.0/conf/log4j.properties\"u\n\035python.internal.files-key-map\022Tpython_file_13c795ab581a80efa6a9947d919503287f51d055172132c4457dc7e58ad114cc:test.py\"3\n\017CMF_PACKAGE_DIR\022
> /opt/cm-5.16.2/lib64/cmf/service\"-\n\017CGROUP_ROOT_CPU\022\032/sys/fs/cgroup/cpu,cpuacct\"%\n\tCONDA_EXE\022\030/opt/anaconda3/bin/conda\"\020\n\bHISTSIZE\022\0041000\"\036\n\tJAVA_HOME\022\021/usr/java/default\"5\n\fCMF_CONF_DIR\022%/opt/cm-5.16.2/etc/cloudera-scm-agent\"0\n\"jobmanager.memory.jvm-overhead.min\022\n201326592b\"\r\n\004TERM\022\005xterm\"R\n\rCDH_HDFS_HOME\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-hdfs\"\023\n\016XDG_SESSION_ID\022\0017\"\033\n\027SELINUX_LEVEL_REQUESTED\022\000\"=\n\fYARN_LOGFILE\022-hadoop-cmf-yarn-NODEMANAGER-cdh5node1.log.out\"+\n\020HADOOP_USER_NAME\022\
> 027tempodata@MERITDATA.COM\"3\n*execution.savepoint.ignore-unclaimed-state\022\005false\"X\n\023SCM_DEFINES_SCRIPTS\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/meta/cdh_env.sh\"(\n\033_PYTHON_WORKER_MEMORY_LIMIT\022\t536870920\"U\n\020HADOOP_HDFS_HOME\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-hdfs\"\\\n\022HADOOP_MAPRED_HOME\022F/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-mapreduce\"1\n\023CGROUP_ROOT_CPUACCT\022\032/sys/fs/cgroup/cpu,cpuacct\"/\n\023CMF_AGENT_MGMT_HOME\022\030/opt/cm-5.16.2/share/cmf\"\037\n\001_\022\032/usr/java/default/bin/java\"%\n\tMGMT_HOME\022\030/opt/cm-5.16.2/share/cmf\"\026\n\022CGROUP_GROUP_BLKIO\022\000\"\025\n\aSSH_TTY\022\n/dev/pts/2\"#\n\nSSH_CLIENT\022\025191.168.1.34
> 61192 22\"
> \n\030PYFLINK_GATEWAY_DISABLED\022\004true\"K\n\rCDH_AVRO_HOME\022:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/avro\"
> \n\016CDH_SQOOP_HOME\022\016/usr/lib/sqoop\"T\n\024CDH_HUE_PLUGINS_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop\"\'\n\036jobmanager.memory.process.size\022\0051600m\"\020\n\vCDH_VERSION\022\0015\"!\n\035HADOOP_JOB_HISTORYSERVER_OPTS\022\000\"
> \n\016CDH_KAFKA_HOME\022\016/usr/lib/kafka\"J\n\035CLOUDERA_ORACLE_CONNECTOR_JAR\022)/usr/share/java/oracle-connector-java.jar\"4\n\016SSH_CONNECTION\022\"191.168.1.34
> 61192 191.168.1.24
> 22\")\n\037internal.cluster.execution-mode\022\006NORMAL\"(\n\aNLSPATH\022\035/usr/dt/lib/nls/msg/%L/%N.cat\"7\n\022KEYTRUSTEE_KP_HOME\022!/usr/share/keytrustee-keyprovider\"+\n\017CMF_SERVER_ROOT\022\030/opt/cm-5.16.2/share/cmf\"\035\n\aCMF_ETC\022\022/opt/cm-5.16.2/etc\"2\n$jobmanager.memory.jvm-metaspace.size\022\n268435456b\"M\n\016CDH_SPARK_HOME\022;/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/spark\"\030\n\024CGROUP_GROUP_CPUACCT\022\000\";\n\fCONTAINER_ID\022+container_e176_1637293573159_0150_01_000002\"M\n\016CDH_LLAMA_HOME\022;/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/llama\"t\n\rXDG_DATA_DIRS\022c/root/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share\"K\n\rCDH_KUDU_HOME\022:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/kudu\"%\n\fPARCELS_ROOT\022\025/opt/cloudera/parcels\"&gt;\n\034high-availability.cluster-id\022\036application_1637293573159_0150\"\035\n\031YARN_RESOURCEMANAGER_OPTS\022\000\"4\n\030python.client.executable\022\030venv.zip/venv/bin/python\"\026\n\rtable.planner\022\005BLINK\"\034\n\004MAIL\022\024/var/spool/mail/root\"1\n
> __table.query-start.epoch-time__\022\r1637301489401\"\030\n\023parallelism.default\022\0011\"\027\n\023pipeline.classpaths\022\000\"M\n\016CDH_HBASE_HOME\022;/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hbase\"\024\n\aLOGNAME\022\ttempodata\"V\n\fCDH_MR2_HOME\022F/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-mapreduce\"R\n\vTOMCAT_HOME\022C/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/bigtop-tomcat\"\033\n\016_FLINK_NODE_ID\022\tcdh5node1\"I\n\fCDH_PIG_HOME\0229/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/pig\"\034\n\026yarn.application.queue\022\002df\"M\n\rHADOOP_PREFIX\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop\"\232\001\n\nLOCAL_DIRS\022\213\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150,/hdfs/yarn/usercache/tempodata/appcache/application_1637293573159_0150\"\342\001\n\nPYTHONPATH\022\323\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-files/blob_p-30842d5cb48fe53fd9f94c2a4e364225246729cc-b3cef870d37841a25e243a9ca6b64863\"(\n\037taskmanager.memory.process.size\022\0051728m\"\022\n\005SHELL\022\t/bin/bash\"\r\n\bweb.port\022\0010\"8\n\026HADOOP_CLIENT_CONF_DIR\022\036/etc/hadoop/conf.cloudera.yarn\"U\n\022CDH_ZOOKEEPER_HOME\022?/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/zookeeper\"T\n\rYARN_CONF_DIR\022C/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER\"-\n\037jobmanager.memory.off-heap.size\022\n134217728b\"|\n
> python.internal.archives-key-map\022Xpython_archive_bfe925aa9bea9ad79f3e1b595cb3e39a3acfa4656b3a5870173959195400aabb:venv.zip\"\321\001\n\bLOG_DIRS\022\304\001/yarn/container-logs/application_1637293573159_0150/container_e176_1637293573159_0150_01_000002,/hdfs/yarn-container-logs/application_1637293573159_0150/container_e176_1637293573159_0150_01_000002\"R\n
> NM_AUX_SERVICE_mapreduce_shuffle\022.AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=\r\n\"W\n\034CLOUDERA_POSTGRESQL_JDBC_JAR\0227/opt/cm-5.16.2/share/cmf/lib/postgresql-42.1.4.jre7.jar\"U\n\020HADOOP_YARN_HOME\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn\".\n\017PARCEL_DIRNAMES\022\033CDH-5.16.2-1.cdh5.16.2.p0.8\"a\n\025SUPERVISOR_SERVER_URL\022Hunix:///opt/cm-5.16.2/run/cloudera-scm-agent/supervisor/supervisord.sock\"\034\n\023jobmanager.rpc.port\022\00539719\"\"\n\vCMF_SBINDIR\022\023/opt/cm-5.16.2/sbin\"w\n\023WEBHCAT_DEFAULT_XML\022`/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/etc/hive-webhcat/conf.dist/webhcat-default.xml\"\"\n\017CDH_CRUNCH_HOME\022\017/usr/lib/crunch\"0\n\027SUPERVISOR_PROCESS_NAME\022\0253224-yarn-NODEMANAGER\".\n\025SUPERVISOR_GROUP_NAME\022\0253224-yarn-NODEMANAGER\"-\n\021python.executable\022\030venv.zip/venv/bin/python\"\373\004\n\020_FLINK_CLASSPATH\022\346\004:flink-python-test-1.0-SNAPSHOT.jar:lib/flink-csv-1.14.0.jar:lib/flink-json-1.14.0.jar:lib/flink-python_2.11-1.14.0.jar:lib/flink-shaded-zookeeper-3.4.14.jar:lib/flink-table_2.11-1.14.0.jar:lib/lib/flink-csv-1.14.0.jar:lib/lib/flink-json-1.14.0.jar:lib/lib/flink-shaded-zookeeper-3.4.14.jar:lib/lib/flink-table_2.11-1.14.0.jar:lib/lib/log4j-1.2-api-2.14.1.jar:lib/lib/log4j-api-2.14.1.jar:lib/lib/log4j-core-2.14.1.jar:lib/lib/log4j-slf4j-impl-2.14.1.jar:lib/log4j-1.2-api-2.14.1.jar:lib/log4j-api-2.14.1.jar:lib/log4j-core-2.14.1.jar:lib/log4j-slf4j-impl-2.14.1.jar:lib/flink-dist_2.11-1.14.0.jar:flink-conf.yaml:\")\n\021CGROUP_ROOT_BLKIO\022\024/sys/fs/cgroup/blkio\"\031\n\frest.address\022\tcdh5node1\"K\n\rCDH_SOLR_HOME\022:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/solr\"\237\v\n\tLS_COLORS\022\221\vrs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:\"T\n\rCDH_HCAT_HOME\022C/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hive-hcatalog\"9\n\037$internal.deployment.config-dir\022\026/opt/flink-1.14.0/conf\"0\n\"jobmanager.memory.jvm-overhead.max\022\n201326592b\"M\n\016CDH_OOZIE_HOME\022;/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/oozie\"O\n\017CDH_SENTRY_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/sentry\"
> \n\004ROOT\022\030/opt/cm-5.16.2/lib64/cmf\"\034\n\020YARN_ROOT_LOGGER\022\bINFO,RFA\"0\n&amp;jobmanager.execution.failover-strategy\022\006region\"\031\n\aCMF_VAR\022\016/opt/cm-5.16.2\"O\n\bCONF_DIR\022C/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER\"-\n\017XFILESEARCHPATH\022\032/usr/dt/app-defaults/%L/Dt\"\037\n\026execution.runtime-mode\022\005BATCH\"\023\n\004LANG\022\vzh_CN.UTF-8\"\021\n\fCMF_SUDO_CMD\022\001
> \"\024\n\aNM_HOST\022\tcdh5node1\"\205\001\n\fBOOT_LOG_DIR\022u/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69\"\031\n\aDISPLAY\022\016localhost:12.0\"Z\n\022HADOOP_LIBEXEC_DIR\022D/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec\"\221\a\n\020HADOOP_CLASSPATH\022\374\006/opt/cm-5.16.2/share/cmf/lib/plugins/event-publish-5.16.2-shaded.jar:/opt/cm-5.16.2/share/cmf/lib/plugins/tt-instrumentation-5.16.2.jar:/etc/hadoop/conf:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-yarn/.//*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*\"\032\n\026SELINUX_ROLE_REQUESTED\022\000\"R\n\026yarn.provided.lib.dirs\0228hdfs://nameservice1/user/tempodata/lixu/flink-1.14.0/lib\"\025\n\rIS_KERBERIZED\022\004true\"+\n\022CGROUP_ROOT_MEMORY\022\025/sys/fs/cgroup/memory\"$\n\fYARN_LOG_DIR\022\024/var/log/hadoop-yarn\"
> \n\017python.archives\022\r/opt/venv.zip\"\"\n\035taskmanager.numberOfTaskSlots\022\0011\"\024\n\020CGROUP_GROUP_CPU\022\000\".\n\vORACLE_HOME\022\037/usr/share/oracle/instantclient\"O\n\017CDH_SQOOP2_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/sqoop2\"R\n\022HADOOP_COMMON_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop\"\020\n\vCONDA_SHLVL\022\0011\"M\n\032$internal.application.main\022/com.meritdata.cloud.flink.python.test.PyUDFTest\"K\n\rCDH_HIVE_HOME\022:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hive\"P\n\fCDH_KMS_HOME\022@/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-kms\"\024\n\fNM_HTTP_PORT\022\0048042\"O\n\017CDH_IMPALA_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/impala\"*\n\016CMF_AGENT_ROOT\022\030/opt/cm-5.16.2/lib64/cmf\"$\n\rNM_LOCAL_DIRS\022\023/yarn/nm
> /hdfs/yarn\"\017\n\aNM_PORT\022\0048041\"\361\003\n\025YARN_NODEMANAGER_OPTS\022\327\003-Xms1073741824
> -Xmx1073741824 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled
> -Dlibrary.leveldbjni.path=/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER
> -Dhadoop.event.appender=,EventCatcher -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=/tmp/yarn_yarn-NODEMANAGER-bef961adb705d3d90452581c3b4991c4_pid31718.hprof
> -XX:OnOutOfMemoryError=/opt/cm-5.16.2/lib64/cmf/service/common/killparent.sh\"\034\n\020execution.target\022\bembedded\"\021\n\004USER\022\ttempodata\"\361\b\n\tCLASSPATH\022\343\b:flink-python-test-1.0-SNAPSHOT.jar:lib/flink-csv-1.14.0.jar:lib/flink-json-1.14.0.jar:lib/flink-python_2.11-1.14.0.jar:lib/flink-shaded-zookeeper-3.4.14.jar:lib/flink-table_2.11-1.14.0.jar:lib/lib/flink-csv-1.14.0.jar:lib/lib/flink-json-1.14.0.jar:lib/lib/flink-shaded-zookeeper-3.4.14.jar:lib/lib/flink-table_2.11-1.14.0.jar:lib/lib/log4j-1.2-api-2.14.1.jar:lib/lib/log4j-api-2.14.1.jar:lib/lib/log4j-core-2.14.1.jar:lib/lib/log4j-slf4j-impl-2.14.1.jar:lib/log4j-1.2-api-2.14.1.jar:lib/log4j-api-2.14.1.jar:lib/log4j-core-2.14.1.jar:lib/log4j-slf4j-impl-2.14.1.jar:lib/flink-dist_2.11-1.14.0.jar:flink-conf.yaml::/etc/hadoop/conf.cloudera.yarn:/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-hdfs/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn/lib/*\"-\n%
> internal.io.tmpdirs.use-local-default\022\004true\"Y\n\016CDH_HADOOP_BIN\022G/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/bin/hadoop\"[\n\fCDH_MR1_HOME\022K/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-0.20-mapreduce\"\240\001\n\032HADOOP_TOKEN_FILE_LOCATION\022\201\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/container_e176_1637293573159_0150_01_000002/container_tokens\"\025\n\bHOSTNAME\022\tcdh5node1\"\032\n\022execution.attached\022\004true\",\n#execution.shutdown-on-attached-exit\022\005false\"\252\001\n\rpipeline.jars\022\230\001file:/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/container_e176_1637293573159_0150_01_000001/flink-python-test-1.0-SNAPSHOT.jar\"\221\001\n\017CM_STATUS_CODES\022~STATUS_NONE
> HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED
> JOBTRACKER_IN_STANDBY_MODE
> YARN_RM_IN_STANDBY_MODE\"R\n\rCDH_YARN_HOME\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn\"K\n\vSEARCH_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/search\"\036\n\017XDG_RUNTIME_DIR\022\v/run/user/0\"\016\n\004HOME\022\006/home/\"*\n\033jobmanager.memory.heap.size\022\v1073741824b\"\025\n\020MALLOC_ARENA_MAX\022\0014"
>
>
> 2021-11-19 14:02:24,050 INFO&nbsp;
> org.apache.beam.runners.fnexecution.logging.GrpcLoggingService [] - 1 Beam
> Fn Logging clients still connected during shutdown.
> 2021-11-19 14:02:25,332 WARN&nbsp;
> org.apache.flink.runtime.taskmanager.Task&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; [] - Source: Values(tuples=[[{
> _UTF-16LE'1' }, { _UTF-16LE'2' }, { _UTF-16LE'3' }]], values=[f0]) -&gt;
> Calc(select=[f0]) -&gt; PythonCalc(select=[func1(f0) AS a]) -&gt; Sink:
> Collect table sink (1/1)#0 (bef195197949f0a6c3716ff2e6e8b17d) switched from
> RUNNING to FAILED with failure cause: java.lang.RuntimeException: Failed to
> start remote bundle
> at
> org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.startBundle(BeamPythonFunctionRunner.java:333)
> at
> org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.checkInvokeStartBundle(BeamPythonFunctionRunner.java:314)
> at
> org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.process(BeamPythonFunctionRunner.java:293)
> at
> org.apache.flink.table.runtime.operators.python.scalar.PythonScalarFunctionOperator.processElementInternal(PythonScalarFunctionOperator.java:89)
> at
> org.apache.flink.table.runtime.operators.python.scalar.PythonScalarFunctionOperator.processElementInternal(PythonScalarFunctionOperator.java:38)
> at
> org.apache.flink.table.runtime.operators.python.AbstractStatelessFunctionOperator.processElement(AbstractStatelessFunctionOperator.java:117)
> at
> org.apache.flink.streaming.runtime.tasks.ChainingOutput.pushToOperator(ChainingOutput.java:99)
> at
> org.apache.flink.streaming.runtime.tasks.ChainingOutput.collect(ChainingOutput.java:80)
> at
> org.apache.flink.streaming.runtime.tasks.ChainingOutput.collect(ChainingOutput.java:39)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:29)
> at BatchExecCalc$8.processElement(Unknown Source)
> at
> org.apache.flink.streaming.runtime.tasks.ChainingOutput.pushToOperator(ChainingOutput.java:99)
> at
> org.apache.flink.streaming.runtime.tasks.ChainingOutput.collect(ChainingOutput.java:80)
> at
> org.apache.flink.streaming.runtime.tasks.ChainingOutput.collect(ChainingOutput.java:39)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:29)
> at
> org.apache.flink.streaming.api.operators.StreamSourceContexts$ManualWatermarkContext.processAndCollect(StreamSourceContexts.java:418)
> at
> org.apache.flink.streaming.api.operators.StreamSourceContexts$WatermarkContext.collect(StreamSourceContexts.java:513)
> at
> org.apache.flink.streaming.api.operators.StreamSourceContexts$SwitchingOnClose.collect(StreamSourceContexts.java:103)
> at
> org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:92)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:116)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:73)
> at
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:323)
> Caused by: java.lang.RuntimeException: No client connected within timeout
> at
> org.apache.beam.runners.fnexecution.data.GrpcDataService.send(GrpcDataService.java:192)
> at
> org.apache.beam.runners.fnexecution.control.SdkHarnessClient$BundleProcessor.newBundle(SdkHarnessClient.java:287)
> at
> org.apache.beam.runners.fnexecution.control.SdkHarnessClient$BundleProcessor.newBundle(SdkHarnessClient.java:197)
> at
> org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$SimpleStageBundleFactory.getBundle(DefaultJobBundleFactory.java:519)
> at
> org.apache.beam.runners.fnexecution.control.StageBundleFactory.getBundle(StageBundleFactory.java:87)
> at
> org.apache.beam.runners.fnexecution.control.StageBundleFactory.getBundle(StageBundleFactory.java:76)
> at
> org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.startBundle(BeamPythonFunctionRunner.java:323)
> ... 23 more
> Caused by: java.util.concurrent.TimeoutException: Waited 3 minutes for
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.SettableFuture@533506fe
> [status=PENDING]
> at
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:471)
> at
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:90)
> at
> org.apache.beam.runners.fnexecution.data.GrpcDataService.send(GrpcDataService.java:187)
> ... 29 more
>
>
>
>
>
> ------------------&nbsp;原始邮件&nbsp;------------------
> 发件人:
> "user-zh"
> <
> dian0511.fu@gmail.com&gt;;
> 发送时间:&nbsp;2021年11月19日(星期五) 中午11:03
> 收件人:&nbsp;"user-zh"<user-zh@flink.apache.org&gt;;
>
> 主题:&nbsp;Re: 哪里可以下载到downloads/setup-pyflink-virtual-env.sh脚本
>
>
>
> 刚注意到你用的YARN application模式，PyFlink 1.14.0才支持YARN application模式，主要是新增了命令行选项“
> -pyclientexec” 和配置“python.client.executable”：
>
> https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/python/python_config/#python-client-executable
>
> 对于你这个作业来说，你需要通过使用1.14.0版本，同时添加命令行选项：-pyclientexec venv.zip/venv/bin/python
>
>
> On Fri, Nov 19, 2021 at 10:48 AM Asahi Lee <978466273@qq.com.invalid&gt;
> wrote:
>
> &gt; 我通过source
> my_env/bin/activate和指定环境变量PYFLINK_CLIENT_EXECUTABLE均未提交成功，错误未变化；
> &gt; 我的错误中，jobmanager的日志中显示No module named pyflink，而jobmanager是运行在yarn集群上；
> &gt; 请问，还有还有什么未配置？
> &gt;
> &gt;
> &gt; &amp;gt; LogType:jobmanager.out
> &gt; &amp;gt; Log Upload Time:星期四 十一月 18 20:48:45 +0800 2021
> &gt; &amp;gt; LogLength:37
> &gt; &amp;gt; Log Contents:
> &gt; &amp;gt; /bin/python: No module named pyflink
> &gt;
> &gt;
> &gt;
> &gt;
> &gt; ------------------&amp;nbsp;原始邮件&amp;nbsp;------------------
> &gt; 发件人:
> &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> "user-zh"
> &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> <
> &gt; dian0511.fu@gmail.com&amp;gt;;
> &gt; 发送时间:&amp;nbsp;2021年11月19日(星期五) 上午9:38
> &gt; 收件人:&amp;nbsp;"user-zh"<user-zh@flink.apache.org&amp;gt;;
> &gt;
> &gt; 主题:&amp;nbsp;Re: 哪里可以下载到downloads/setup-pyflink-virtual-env.sh脚本
> &gt;
> &gt;
> &gt;
> &gt; -pyexec 指定的是集群端所用的Python环境，客户端需要编译Flink作业，也会依赖Python环境。可以看一下这个文档:
> &gt;
> &gt;
> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/dev/python/dependency_management/#python-interpreter-of-client
> &gt
> <https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/dev/python/dependency_management/#python-interpreter-of-client&gt>
> ;
> &gt; On Thu, Nov 18, 2021 at 9:00 PM Asahi Lee <978466273@qq.com.invalid
> &amp;gt;
> &gt; wrote:
> &gt;
> &gt; &amp;gt; Hi !
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp;我在java Table api中使用python udf
> &gt; &amp;gt; 函数，通过下面的命令提交应用，报无法启动python服务错误，请问我的提交方式对吗？jm日志为/bin/python:
> No module
> &gt; named
> &gt; &amp;gt; pyflink。
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt; ./flink-1.13.2/bin/flink&amp;amp;nbsp;
> &gt; &amp;gt; run-application -t yarn-application&amp;amp;nbsp;
> &gt; &amp;gt;
> &gt;
> -Dyarn.provided.lib.dirs="hdfs://nameservice1/user/flink/flinklib"&amp;amp;nbsp;
> &gt; &amp;gt; -Dyarn.application.queue=d
> &gt; &amp;gt; -p 1&amp;amp;nbsp;
> &gt; &amp;gt; -pyarch /opt/venv.zip
> &gt; &amp;gt; -pyexec venv.zip/venv/bin/python&amp;amp;nbsp;
> &gt; &amp;gt; -pyfs /opt/test.py&amp;amp;nbsp;
> &gt; &amp;gt; -c test.PyUDFTest&amp;amp;nbsp;
> &gt; &amp;gt; /opt/flink-python-test-1.0-SNAPSHOT.jar
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt; 错误：
> &gt; &amp;gt; Caused by: java.lang.RuntimeException: Python callback
> server start
> &gt; failed!
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.flink.client.python.PythonFunctionFactory.createPythonFunctionFactory(PythonFunctionFactory.java:167)
> &gt; &amp;gt; ~[flink-python_2.11-1.13.2.jar:1.13.2]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.flink.client.python.PythonFunctionFactory$1.load(PythonFunctionFactory.java:88)
> &gt; &amp;gt; ~[flink-python_2.11-1.13.2.jar:1.13.2]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.flink.client.python.PythonFunctionFactory$1.load(PythonFunctionFactory.java:84)
> &gt; &amp;gt; ~[flink-python_2.11-1.13.2.jar:1.13.2]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
> &gt; &amp;gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
> &gt; &amp;gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
> &gt; &amp;gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
> &gt; &amp;gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> &gt; &amp;gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
> &gt; &amp;gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
> &gt; &amp;gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.flink.client.python.PythonFunctionFactory.getPythonFunction(PythonFunctionFactory.java:129)
> &gt; &amp;gt; ~[flink-python_2.11-1.13.2.jar:1.13.2]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt; sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> &gt; ~[?:1.8.0_111]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> &gt; &amp;gt; ~[?:1.8.0_111]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> &gt; &amp;gt; ~[?:1.8.0_111]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt; java.lang.reflect.Method.invoke(Method.java:498)
> ~[?:1.8.0_111]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.flink.table.functions.python.utils.PythonFunctionUtils.getPythonFunction(PythonFunctionUtils.java:45)
> &gt; &amp;gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.flink.table.functions.UserDefinedFunctionHelper.instantiateFunction(UserDefinedFunctionHelper.java:206)
> &gt; &amp;gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.flink.table.catalog.FunctionCatalog.getFunctionDefinition(FunctionCatalog.java:659)
> &gt; &amp;gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.flink.table.catalog.FunctionCatalog.resolveAmbiguousFunctionReference(FunctionCatalog.java:606)
> &gt; &amp;gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.flink.table.catalog.FunctionCatalog.lookupFunction(FunctionCatalog.java:362)
> &gt; &amp;gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.flink.table.planner.catalog.FunctionCatalogOperatorTable.lookupOperatorOverloads(FunctionCatalogOperatorTable.java:97)
> &gt; &amp;gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.calcite.sql.util.ChainedSqlOperatorTable.lookupOperatorOverloads(ChainedSqlOperatorTable.java:67)
> &gt; &amp;gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1183)
> &gt; &amp;gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1169)
> &gt; &amp;gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1200)
> &gt; &amp;gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1169)
> &gt; &amp;gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:945)
> &gt; &amp;gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:704)
> &gt; &amp;gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt; org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org
> &gt;
> $apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:152)
> &gt; &amp;gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:110)
> &gt; &amp;gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:201)
> &gt; &amp;gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:101)
> &gt; &amp;gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:704)
> &gt; &amp;gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; test.PyUDFTest.main(PyUDFTest.java:22)
> &gt; &amp;gt; ~[flink-python-test-1.0-SNAPSHOT.jar:?]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt; sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> &gt; ~[?:1.8.0_111]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> &gt; &amp;gt; ~[?:1.8.0_111]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> &gt; &amp;gt; ~[?:1.8.0_111]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt; java.lang.reflect.Method.invoke(Method.java:498)
> ~[?:1.8.0_111]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
> &gt; &amp;gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
> &gt; &amp;gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> &gt; &amp;gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> at
> &gt; &amp;gt;
> &gt;
> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:242)
> &gt; &amp;gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> ... 10 more
> &gt; &amp;gt; 2021-11-18 20:48:44,458 INFO&amp;amp;nbsp;
> &gt; &amp;gt; org.apache.flink.runtime.blob.BlobServer&amp;amp;nbsp;
> &amp;amp;nbsp;
> &gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> &gt; &amp;gt; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp; &amp;amp;nbsp;
> &amp;amp;nbsp; &amp;amp;nbsp;[] -
> &gt; Stopped BLOB server at
> &gt; &amp;gt; 0.0.0.0:45070
> &gt; &amp;gt; 2021-11-18 20:48:44,458 INFO&amp;amp;nbsp;
> &gt; &amp;gt;
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint&amp;amp;nbsp;
> &gt; &amp;amp;nbsp; &amp;amp;nbsp;
> &gt; &amp;gt; &amp;amp;nbsp; [] - Shutting
> YarnApplicationClusterEntryPoint down with
> &gt; application
> &gt; &amp;gt; status UNKNOWN. Diagnostics Cluster entrypoint has been
> closed
> &gt; externally..
> &gt; &amp;gt; 2021-11-18 20:48:44,458 INFO&amp;amp;nbsp;
> &gt; &amp;gt;
> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] -
> &gt; Shutting
> &gt; &amp;gt; down rest endpoint.
> &gt; &amp;gt; 2021-11-18 20:48:44,474 INFO&amp;amp;nbsp;
> &gt; &amp;gt;
> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] -
> &gt; Removing
> &gt; &amp;gt; cache directory
> &gt; &amp;gt;
> /tmp/flink-web-dd82c3c0-f457-492d-8e64-5ae74fe9abbd/flink-web-ui
> &gt; &amp;gt; 2021-11-18 20:48:44,474 INFO&amp;amp;nbsp;
> &gt; &amp;gt;
> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] -
> &gt; &amp;gt; http://cdh5node3:40216 lost leadership
> &gt; &amp;gt; 2021-11-18 20:48:44,474 INFO&amp;amp;nbsp;
> &gt; &amp;gt;
> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] -
> &gt; Shut
> &gt; &amp;gt; down complete.
> &gt; &amp;gt; 2021-11-18 20:48:44,474 INFO&amp;amp;nbsp;
> &gt; &amp;gt;
> &gt;
> org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent
> &gt; &amp;gt; [] - Closing components.
> &gt; &amp;gt; 2021-11-18 20:48:44,475 INFO&amp;amp;nbsp;
> &gt; &amp;gt;
> &gt;
> org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess
> &gt; &amp;gt; [] - Stopping SessionDispatcherLeaderProcess.
> &gt; &amp;gt; 2021-11-18 20:48:44,475 INFO&amp;amp;nbsp;
> &gt; &amp;gt;
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher&amp;amp;nbsp;
> &gt; &amp;amp;nbsp;
> &gt; &amp;gt; &amp;amp;nbsp;[] - Stopping dispatcher
> akka.tcp://flink@cdh5node3
> &gt; &amp;gt; :34697/user/rpc/dispatcher_1.
> &gt; &amp;gt; 2021-11-18 20:48:44,476 INFO&amp;amp;nbsp;
> &gt; &amp;gt;
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher&amp;amp;nbsp;
> &gt; &amp;amp;nbsp;
> &gt; &amp;gt; &amp;amp;nbsp;[] - Stopping all currently running jobs of
> dispatcher
> &gt; &amp;gt; akka.tcp://flink@cdh5node3:34697/user/rpc/dispatcher_1.
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt; LogType:jobmanager.out
> &gt; &amp;gt; Log Upload Time:星期四 十一月 18 20:48:45 +0800 2021
> &gt; &amp;gt; LogLength:37
> &gt; &amp;gt; Log Contents:
> &gt; &amp;gt; /bin/python: No module named pyflink
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt;
> ------------------&amp;amp;nbsp;原始邮件&amp;amp;nbsp;------------------
> &gt; &amp;gt; 发件人:
> &gt;
> &amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> &gt; "user-zh"
> &gt;
> &amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;
> &gt; <
> &gt; &amp;gt; xyzhong131@163.com&amp;amp;gt;;
> &gt; &amp;gt; 发送时间:&amp;amp;nbsp;2021年11月18日(星期四) 下午3:34
> &gt; &amp;gt; 收件人:&amp;amp;nbsp;"user-zh"<user-zh@flink.apache.org
> &amp;amp;gt;;
> &gt; &amp;gt;
> &gt; &amp;gt;
> 主题:&amp;amp;nbsp;Re:哪里可以下载到downloads/setup-pyflink-virtual-env.sh脚本
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt;
> &gt; &amp;gt; Hi!<br/&amp;amp;gt;我在下面的旧版本中找到了，你试下能不能用：<br/&amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> https://nightlies.apache.org/flink/flink-docs-release-1.12/downloads/setup-pyflink-virtual-env.sh
> &gt
> <https://nightlies.apache.org/flink/flink-docs-release-1.12/downloads/setup-pyflink-virtual-env.sh&gt>;
> &amp;gt
> &gt; <
> https://nightlies.apache.org/flink/flink-docs-release-1.12/downloads/setup-pyflink-virtual-env.sh&amp;gt&gt
> ;;
> &gt; 在 2021-11-18 15:05:03，"Asahi Lee" <978466273@qq.com.INVALID&amp;amp;gt;
> 写道：
> &gt; &amp;gt; &amp;amp;gt;Hi!
> &gt; &amp;gt; &amp;amp;gt;&amp;amp;amp;nbsp; &amp;amp;amp;nbsp;
> &gt; &amp;gt;
> &gt;
> 我在flink官方文档中看到通过&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;setup-pyflink-virtual-env.sh
> &gt; &amp;gt; 脚本制作python虚拟环境，请问再哪里可以下载到？文档地址如下：
> &gt; &amp;gt; &amp;amp;gt;
> &gt; &amp;gt;
> &gt;
> https://nightlies.apache.org/flink/flink-docs-release-1.14/zh/docs/dev/python/faq/
Re: 哪里可以下载到downloads/setup-pyflink-virtual-env.sh脚本
Posted by Dian Fu <di...@gmail.com>.
Flink用的1.14.0，venv.zip中的PyFlink版本是多少？
On Sun, Nov 21, 2021 at 7:59 PM Asahi Lee <as...@qq.com.invalid> wrote:
> Hi!
> 我通过如下命令提交成功了，python的参数需求-D方式传入，-py方式传入不生效：
> ./flink-1.14.0/bin/flink
> run-application -t yarn-application
> -Dyarn.provided.lib.dirs="hdfs://nameservice1/user/flink/flinklib"
> -Dyarn.application.queue=d
> -Dpython.archives="/opt/venv.zip"
> -Dpython.client.executable="venv.zip/venv/bin/python"
> -Dpython.executable="venv.zip/venv/bin/python"
> -Dpython.files="/opt/test.py"
> -p 1
> -c test.PyUDFTest
> /opt/flink-python-test-1.0-SNAPSHOT.jar
>
>
>
> 但是我应用提交成功后，程序运行发生错误，如下：
>
>
> taskmanager.log
>
>
> 2021-11-19 13:59:24,030 ERROR
> /yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py:296
> [] - Error processing instruction 1. Original traceback is
> Traceback (most recent call last):
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 289, in _execute
> response = task()
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 362, in <lambda>
> lambda: self.create_worker().do_instruction(request),
> request)
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 607, in do_instruction
> getattr(request, request_type), request.instruction_id)
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 638, in process_bundle
> instruction_id, request.process_bundle_descriptor_id)
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 467, in get
> self.data_channel_factory)
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 868, in __init__
> self.ops =
> self.create_execution_tree(self.process_bundle_descriptor)
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 925, in create_execution_tree
> descriptor.transforms, key=topological_height,
> reverse=True)])
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 924, in <listcomp>
> get_operation(transform_id))) for transform_id in sorted(
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 812, in wrapper
> result = cache[args] = func(*args)
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 906, in get_operation
> pcoll_id in
> descriptor.transforms[transform_id].outputs.items()
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 906, in <dictcomp>
> pcoll_id in
> descriptor.transforms[transform_id].outputs.items()
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 904, in <listcomp>
> tag: [get_operation(op) for op in pcoll_consumers[pcoll_id]]
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 812, in wrapper
> result = cache[args] = func(*args)
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 906, in get_operation
> pcoll_id in
> descriptor.transforms[transform_id].outputs.items()
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 906, in <dictcomp>
> pcoll_id in
> descriptor.transforms[transform_id].outputs.items()
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 904, in <listcomp>
> tag: [get_operation(op) for op in pcoll_consumers[pcoll_id]]
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 812, in wrapper
> result = cache[args] = func(*args)
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 909, in get_operation
> transform_id, transform_consumers)
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 1198, in create_operation
> return creator(self, transform_id, transform_proto, payload,
> consumers)
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 1320, in create_sink_runner
> output_coder = factory.get_coder(grpc_port.coder_id)
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 1222, in get_coder
> return self.context.coders.get_by_id(coder_id)
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/pipeline_context.py",
> line 116, in get_by_id
> self._id_to_proto[id], self._pipeline_context)
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/coders/coders.py",
> line 370, in from_runner_api
> [context.coders.get_by_id(c) for c in
> coder_proto.component_coder_ids],
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/coders/coders.py",
> line 370, in <listcomp>
> [context.coders.get_by_id(c) for c in
> coder_proto.component_coder_ids],
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/pipeline_context.py",
> line 116, in get_by_id
> self._id_to_proto[id], self._pipeline_context)
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/coders/coders.py",
> line 370, in from_runner_api
> [context.coders.get_by_id(c) for c in
> coder_proto.component_coder_ids],
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/coders/coders.py",
> line 370, in <listcomp>
> [context.coders.get_by_id(c) for c in
> coder_proto.component_coder_ids],
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/pipeline_context.py",
> line 116, in get_by_id
> self._id_to_proto[id], self._pipeline_context)
> File
> "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/coders/coders.py",
> line 367, in from_runner_api
> parameter_type, constructor =
> cls._known_urns[coder_proto.spec.urn]
> KeyError: 'flink:coder:v1'
>
>
>
> 2021-11-19 14:02:24,031 WARN
> org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory [] -
> Expiring environment urn: "beam:env:process:v1"
> payload:
> "\032\321\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/pyflink/bin/pyflink-udf-runner.sh\"%\n\004PATH\022\035/sbin:/usr/sbin:/bin:/usr/bin\"V\n\017HADOOP_CONF_DIR\022C/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER\"\031\n\vHISTCONTROL\022\nignoredups\"\027\n\022SUPERVISOR_ENABLED\022\0011\"\031\n\021CONDA_DEFAULT_ENV\022\004base\"#\n\026jobmanager.rpc.address\022\tcdh5node1\"4\n\026KEYTRUSTEE_SERVER_HOME\022\032/usr/lib/keytrustee-server\"-\n\020CONDA_PYTHON_EXE\022\031/opt/anaconda3/bin/python\"\027\n\023CGROUP_GROUP_MEMORY\022\000\"$\n\023table.exec.timezone\022\rAsia/Shanghai\"Z\n\026CDH_HBASE_INDEXER_HOME\022@/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hbase-solr\"\036\n\fCONDA_PREFIX\022\016/opt/anaconda3\"\233\001\n\vio.tmp.dirs\022\213\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150,/hdfs/yarn/usercache/tempodata/appcache/application_1637293573159_0150\"\222\b\n\tYARN_OPTS\022\204\b-Djava.net.preferIPv4Stack=true
> -server -Xms1073741824 -Xmx1073741824 -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
> -XX:+CMSParallelRemarkEnabled
> -Dlibrary.leveldbjni.path=/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER
> -Dhadoop.event.appender=,EventCatcher -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=/tmp/yarn_yarn-NODEMANAGER-bef961adb705d3d90452581c3b4991c4_pid31718.hprof
> -XX:OnOutOfMemoryError=/opt/cm-5.16.2/lib64/cmf/service/common/killparent.sh
> -Dhadoop.log.dir=/var/log/hadoop-yarn -Dyarn.log.dir=/var/log/hadoop-yarn
> -Dhadoop.log.file=hadoop-cmf-yarn-NODEMANAGER-cdh5node1.log.out
> -Dyarn.log.file=hadoop-cmf-yarn-NODEMANAGER-cdh5node1.log.out
> -Dyarn.home.dir=/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn
> -Dhadoop.home.dir=/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn
> -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA
> -Djava.library.path=/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/lib/native\"!\n\031HADOOP_HOME_WARN_SUPPRESS\022\004true\"\017\n\aJVM_PID\022\0045524\"O\n\tJSVC_HOME\022B/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/bigtop-utils\"w\n\003PWD\022p/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/container_e176_1637293573159_0150_01_000002\"V\n\017CDH_HTTPFS_HOME\022C/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-httpfs\"P\n\016CDH_FLUME_HOME\022>/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/flume-ng\"Q\n\020CDH_PARQUET_HOME\022=/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/parquet\"\"\n\006python\022\030venv.zip/venv/bin/python\"%\n\bLESSOPEN\022\031||/usr/bin/lesspipe.sh
> %s\"\035\n\031SELINUX_USE_CURRENT_RANGE\022\000\"\034\n\fpython.files\022\f/opt/test.py\"O\n\017CDH_HADOOP_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop\"A\n\nweb.tmpdir\0223/tmp/flink-web-e35b2953-cfce-4655-bbac-bbacf1dbba5e\"\235\001\n\023_PYTHON_WORKING_DIR\022\205\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives\"\035\n\vPYTHON_HOME\022\016/opt/anaconda3\"H\n\034CLOUDERA_MYSQL_CONNECTOR_JAR\022(/usr/share/java/mysql-connector-java.jar\"I\n\fCDH_HUE_HOME\0229/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hue\"
> \n\034NM_AUX_SERVICE_spark_shuffle\022\000\"1\n
> __table.query-start.local-time__\022\r1637330289401\"8\n\020HIVE_DEFAULT_XML\022$/etc/hive/conf.dist/hive-default.xml\"\031\n\025CONDA_PROMPT_MODIFIER\022\000\"\n\n\005SHLVL\022\0014\"I\n\036$internal.yarn.log-config-file\022\'/opt/flink-1.14.0/conf/log4j.properties\"u\n\035python.internal.files-key-map\022Tpython_file_13c795ab581a80efa6a9947d919503287f51d055172132c4457dc7e58ad114cc:test.py\"3\n\017CMF_PACKAGE_DIR\022
> /opt/cm-5.16.2/lib64/cmf/service\"-\n\017CGROUP_ROOT_CPU\022\032/sys/fs/cgroup/cpu,cpuacct\"%\n\tCONDA_EXE\022\030/opt/anaconda3/bin/conda\"\020\n\bHISTSIZE\022\0041000\"\036\n\tJAVA_HOME\022\021/usr/java/default\"5\n\fCMF_CONF_DIR\022%/opt/cm-5.16.2/etc/cloudera-scm-agent\"0\n\"jobmanager.memory.jvm-overhead.min\022\n201326592b\"\r\n\004TERM\022\005xterm\"R\n\rCDH_HDFS_HOME\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-hdfs\"\023\n\016XDG_SESSION_ID\022\0017\"\033\n\027SELINUX_LEVEL_REQUESTED\022\000\"=\n\fYARN_LOGFILE\022-hadoop-cmf-yarn-NODEMANAGER-cdh5node1.log.out\"+\n\020HADOOP_USER_NAME\022\
> 027tempodata@MERITDATA.COM\"3\n*execution.savepoint.ignore-unclaimed-state\022\005false\"X\n\023SCM_DEFINES_SCRIPTS\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/meta/cdh_env.sh\"(\n\033_PYTHON_WORKER_MEMORY_LIMIT\022\t536870920\"U\n\020HADOOP_HDFS_HOME\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-hdfs\"\\\n\022HADOOP_MAPRED_HOME\022F/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-mapreduce\"1\n\023CGROUP_ROOT_CPUACCT\022\032/sys/fs/cgroup/cpu,cpuacct\"/\n\023CMF_AGENT_MGMT_HOME\022\030/opt/cm-5.16.2/share/cmf\"\037\n\001_\022\032/usr/java/default/bin/java\"%\n\tMGMT_HOME\022\030/opt/cm-5.16.2/share/cmf\"\026\n\022CGROUP_GROUP_BLKIO\022\000\"\025\n\aSSH_TTY\022\n/dev/pts/2\"#\n\nSSH_CLIENT\022\025191.168.1.34
> 61192 22\"
> \n\030PYFLINK_GATEWAY_DISABLED\022\004true\"K\n\rCDH_AVRO_HOME\022:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/avro\"
> \n\016CDH_SQOOP_HOME\022\016/usr/lib/sqoop\"T\n\024CDH_HUE_PLUGINS_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop\"\'\n\036jobmanager.memory.process.size\022\0051600m\"\020\n\vCDH_VERSION\022\0015\"!\n\035HADOOP_JOB_HISTORYSERVER_OPTS\022\000\"
> \n\016CDH_KAFKA_HOME\022\016/usr/lib/kafka\"J\n\035CLOUDERA_ORACLE_CONNECTOR_JAR\022)/usr/share/java/oracle-connector-java.jar\"4\n\016SSH_CONNECTION\022\"191.168.1.34
> 61192 191.168.1.24
> 22\")\n\037internal.cluster.execution-mode\022\006NORMAL\"(\n\aNLSPATH\022\035/usr/dt/lib/nls/msg/%L/%N.cat\"7\n\022KEYTRUSTEE_KP_HOME\022!/usr/share/keytrustee-keyprovider\"+\n\017CMF_SERVER_ROOT\022\030/opt/cm-5.16.2/share/cmf\"\035\n\aCMF_ETC\022\022/opt/cm-5.16.2/etc\"2\n$jobmanager.memory.jvm-metaspace.size\022\n268435456b\"M\n\016CDH_SPARK_HOME\022;/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/spark\"\030\n\024CGROUP_GROUP_CPUACCT\022\000\";\n\fCONTAINER_ID\022+container_e176_1637293573159_0150_01_000002\"M\n\016CDH_LLAMA_HOME\022;/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/llama\"t\n\rXDG_DATA_DIRS\022c/root/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share\"K\n\rCDH_KUDU_HOME\022:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/kudu\"%\n\fPARCELS_ROOT\022\025/opt/cloudera/parcels\">\n\034high-availability.cluster-id\022\036application_1637293573159_0150\"\035\n\031YARN_RESOURCEMANAGER_OPTS\022\000\"4\n\030python.client.executable\022\030venv.zip/venv/bin/python\"\026\n\rtable.planner\022\005BLINK\"\034\n\004MAIL\022\024/var/spool/mail/root\"1\n
> __table.query-start.epoch-time__\022\r1637301489401\"\030\n\023parallelism.default\022\0011\"\027\n\023pipeline.classpaths\022\000\"M\n\016CDH_HBASE_HOME\022;/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hbase\"\024\n\aLOGNAME\022\ttempodata\"V\n\fCDH_MR2_HOME\022F/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-mapreduce\"R\n\vTOMCAT_HOME\022C/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/bigtop-tomcat\"\033\n\016_FLINK_NODE_ID\022\tcdh5node1\"I\n\fCDH_PIG_HOME\0229/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/pig\"\034\n\026yarn.application.queue\022\002df\"M\n\rHADOOP_PREFIX\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop\"\232\001\n\nLOCAL_DIRS\022\213\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150,/hdfs/yarn/usercache/tempodata/appcache/application_1637293573159_0150\"\342\001\n\nPYTHONPATH\022\323\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-files/blob_p-30842d5cb48fe53fd9f94c2a4e364225246729cc-b3cef870d37841a25e243a9ca6b64863\"(\n\037taskmanager.memory.process.size\022\0051728m\"\022\n\005SHELL\022\t/bin/bash\"\r\n\bweb.port\022\0010\"8\n\026HADOOP_CLIENT_CONF_DIR\022\036/etc/hadoop/conf.cloudera.yarn\"U\n\022CDH_ZOOKEEPER_HOME\022?/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/zookeeper\"T\n\rYARN_CONF_DIR\022C/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER\"-\n\037jobmanager.memory.off-heap.size\022\n134217728b\"|\n
> python.internal.archives-key-map\022Xpython_archive_bfe925aa9bea9ad79f3e1b595cb3e39a3acfa4656b3a5870173959195400aabb:venv.zip\"\321\001\n\bLOG_DIRS\022\304\001/yarn/container-logs/application_1637293573159_0150/container_e176_1637293573159_0150_01_000002,/hdfs/yarn-container-logs/application_1637293573159_0150/container_e176_1637293573159_0150_01_000002\"R\n
> NM_AUX_SERVICE_mapreduce_shuffle\022.AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=\r\n\"W\n\034CLOUDERA_POSTGRESQL_JDBC_JAR\0227/opt/cm-5.16.2/share/cmf/lib/postgresql-42.1.4.jre7.jar\"U\n\020HADOOP_YARN_HOME\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn\".\n\017PARCEL_DIRNAMES\022\033CDH-5.16.2-1.cdh5.16.2.p0.8\"a\n\025SUPERVISOR_SERVER_URL\022Hunix:///opt/cm-5.16.2/run/cloudera-scm-agent/supervisor/supervisord.sock\"\034\n\023jobmanager.rpc.port\022\00539719\"\"\n\vCMF_SBINDIR\022\023/opt/cm-5.16.2/sbin\"w\n\023WEBHCAT_DEFAULT_XML\022`/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/etc/hive-webhcat/conf.dist/webhcat-default.xml\"\"\n\017CDH_CRUNCH_HOME\022\017/usr/lib/crunch\"0\n\027SUPERVISOR_PROCESS_NAME\022\0253224-yarn-NODEMANAGER\".\n\025SUPERVISOR_GROUP_NAME\022\0253224-yarn-NODEMANAGER\"-\n\021python.executable\022\030venv.zip/venv/bin/python\"\373\004\n\020_FLINK_CLASSPATH\022\346\004:flink-python-test-1.0-SNAPSHOT.jar:lib/flink-csv-1.14.0.jar:lib/flink-json-1.14.0.jar:lib/flink-python_2.11-1.14.0.jar:lib/flink-shaded-zookeeper-3.4.14.jar:lib/flink-table_2.11-1.14.0.jar:lib/lib/flink-csv-1.14.0.jar:lib/lib/flink-json-1.14.0.jar:lib/lib/flink-shaded-zookeeper-3.4.14.jar:lib/lib/flink-table_2.11-1.14.0.jar:lib/lib/log4j-1.2-api-2.14.1.jar:lib/lib/log4j-api-2.14.1.jar:lib/lib/log4j-core-2.14.1.jar:lib/lib/log4j-slf4j-impl-2.14.1.jar:lib/log4j-1.2-api-2.14.1.jar:lib/log4j-api-2.14.1.jar:lib/log4j-core-2.14.1.jar:lib/log4j-slf4j-impl-2.14.1.jar:lib/flink-dist_2.11-1.14.0.jar:flink-conf.yaml:\")\n\021CGROUP_ROOT_BLKIO\022\024/sys/fs/cgroup/blkio\"\031\n\frest.address\022\tcdh5node1\"K\n\rCDH_SOLR_HOME\022:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/solr\"\237\v\n\tLS_COLORS\022\221\vrs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:\"T\n\rCDH_HCAT_HOME\022C/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hive-hcatalog\"9\n\037$internal.deployment.config-dir\022\026/opt/flink-1.14.0/conf\"0\n\"jobmanager.memory.jvm-overhead.max\022\n201326592b\"M\n\016CDH_OOZIE_HOME\022;/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/oozie\"O\n\017CDH_SENTRY_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/sentry\"
> \n\004ROOT\022\030/opt/cm-5.16.2/lib64/cmf\"\034\n\020YARN_ROOT_LOGGER\022\bINFO,RFA\"0\n&jobmanager.execution.failover-strategy\022\006region\"\031\n\aCMF_VAR\022\016/opt/cm-5.16.2\"O\n\bCONF_DIR\022C/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER\"-\n\017XFILESEARCHPATH\022\032/usr/dt/app-defaults/%L/Dt\"\037\n\026execution.runtime-mode\022\005BATCH\"\023\n\004LANG\022\vzh_CN.UTF-8\"\021\n\fCMF_SUDO_CMD\022\001
> \"\024\n\aNM_HOST\022\tcdh5node1\"\205\001\n\fBOOT_LOG_DIR\022u/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69\"\031\n\aDISPLAY\022\016localhost:12.0\"Z\n\022HADOOP_LIBEXEC_DIR\022D/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec\"\221\a\n\020HADOOP_CLASSPATH\022\374\006/opt/cm-5.16.2/share/cmf/lib/plugins/event-publish-5.16.2-shaded.jar:/opt/cm-5.16.2/share/cmf/lib/plugins/tt-instrumentation-5.16.2.jar:/etc/hadoop/conf:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-yarn/.//*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*\"\032\n\026SELINUX_ROLE_REQUESTED\022\000\"R\n\026yarn.provided.lib.dirs\0228hdfs://nameservice1/user/tempodata/lixu/flink-1.14.0/lib\"\025\n\rIS_KERBERIZED\022\004true\"+\n\022CGROUP_ROOT_MEMORY\022\025/sys/fs/cgroup/memory\"$\n\fYARN_LOG_DIR\022\024/var/log/hadoop-yarn\"
> \n\017python.archives\022\r/opt/venv.zip\"\"\n\035taskmanager.numberOfTaskSlots\022\0011\"\024\n\020CGROUP_GROUP_CPU\022\000\".\n\vORACLE_HOME\022\037/usr/share/oracle/instantclient\"O\n\017CDH_SQOOP2_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/sqoop2\"R\n\022HADOOP_COMMON_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop\"\020\n\vCONDA_SHLVL\022\0011\"M\n\032$internal.application.main\022/com.meritdata.cloud.flink.python.test.PyUDFTest\"K\n\rCDH_HIVE_HOME\022:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hive\"P\n\fCDH_KMS_HOME\022@/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-kms\"\024\n\fNM_HTTP_PORT\022\0048042\"O\n\017CDH_IMPALA_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/impala\"*\n\016CMF_AGENT_ROOT\022\030/opt/cm-5.16.2/lib64/cmf\"$\n\rNM_LOCAL_DIRS\022\023/yarn/nm
> /hdfs/yarn\"\017\n\aNM_PORT\022\0048041\"\361\003\n\025YARN_NODEMANAGER_OPTS\022\327\003-Xms1073741824
> -Xmx1073741824 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled
> -Dlibrary.leveldbjni.path=/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER
> -Dhadoop.event.appender=,EventCatcher -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=/tmp/yarn_yarn-NODEMANAGER-bef961adb705d3d90452581c3b4991c4_pid31718.hprof
> -XX:OnOutOfMemoryError=/opt/cm-5.16.2/lib64/cmf/service/common/killparent.sh\"\034\n\020execution.target\022\bembedded\"\021\n\004USER\022\ttempodata\"\361\b\n\tCLASSPATH\022\343\b:flink-python-test-1.0-SNAPSHOT.jar:lib/flink-csv-1.14.0.jar:lib/flink-json-1.14.0.jar:lib/flink-python_2.11-1.14.0.jar:lib/flink-shaded-zookeeper-3.4.14.jar:lib/flink-table_2.11-1.14.0.jar:lib/lib/flink-csv-1.14.0.jar:lib/lib/flink-json-1.14.0.jar:lib/lib/flink-shaded-zookeeper-3.4.14.jar:lib/lib/flink-table_2.11-1.14.0.jar:lib/lib/log4j-1.2-api-2.14.1.jar:lib/lib/log4j-api-2.14.1.jar:lib/lib/log4j-core-2.14.1.jar:lib/lib/log4j-slf4j-impl-2.14.1.jar:lib/log4j-1.2-api-2.14.1.jar:lib/log4j-api-2.14.1.jar:lib/log4j-core-2.14.1.jar:lib/log4j-slf4j-impl-2.14.1.jar:lib/flink-dist_2.11-1.14.0.jar:flink-conf.yaml::/etc/hadoop/conf.cloudera.yarn:/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-hdfs/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn/lib/*\"-\n%
> internal.io.tmpdirs.use-local-default\022\004true\"Y\n\016CDH_HADOOP_BIN\022G/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/bin/hadoop\"[\n\fCDH_MR1_HOME\022K/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-0.20-mapreduce\"\240\001\n\032HADOOP_TOKEN_FILE_LOCATION\022\201\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/container_e176_1637293573159_0150_01_000002/container_tokens\"\025\n\bHOSTNAME\022\tcdh5node1\"\032\n\022execution.attached\022\004true\",\n#execution.shutdown-on-attached-exit\022\005false\"\252\001\n\rpipeline.jars\022\230\001file:/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/container_e176_1637293573159_0150_01_000001/flink-python-test-1.0-SNAPSHOT.jar\"\221\001\n\017CM_STATUS_CODES\022~STATUS_NONE
> HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED
> JOBTRACKER_IN_STANDBY_MODE
> YARN_RM_IN_STANDBY_MODE\"R\n\rCDH_YARN_HOME\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn\"K\n\vSEARCH_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/search\"\036\n\017XDG_RUNTIME_DIR\022\v/run/user/0\"\016\n\004HOME\022\006/home/\"*\n\033jobmanager.memory.heap.size\022\v1073741824b\"\025\n\020MALLOC_ARENA_MAX\022\0014"
> with 1 remaining bundle references. Taking note to clean it up
> during shutdown if the references are not removed by then.
> 2021-11-19 14:02:24,046 INFO
> org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory [] -
> Closing environment urn: "beam:env:process:v1"
> payload:
> "\032\321\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/pyflink/bin/pyflink-udf-runner.sh\"%\n\004PATH\022\035/sbin:/usr/sbin:/bin:/usr/bin\"V\n\017HADOOP_CONF_DIR\022C/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER\"\031\n\vHISTCONTROL\022\nignoredups\"\027\n\022SUPERVISOR_ENABLED\022\0011\"\031\n\021CONDA_DEFAULT_ENV\022\004base\"#\n\026jobmanager.rpc.address\022\tcdh5node1\"4\n\026KEYTRUSTEE_SERVER_HOME\022\032/usr/lib/keytrustee-server\"-\n\020CONDA_PYTHON_EXE\022\031/opt/anaconda3/bin/python\"\027\n\023CGROUP_GROUP_MEMORY\022\000\"$\n\023table.exec.timezone\022\rAsia/Shanghai\"Z\n\026CDH_HBASE_INDEXER_HOME\022@/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hbase-solr\"\036\n\fCONDA_PREFIX\022\016/opt/anaconda3\"\233\001\n\vio.tmp.dirs\022\213\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150,/hdfs/yarn/usercache/tempodata/appcache/application_1637293573159_0150\"\222\b\n\tYARN_OPTS\022\204\b-Djava.net.preferIPv4Stack=true
> -server -Xms1073741824 -Xmx1073741824 -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
> -XX:+CMSParallelRemarkEnabled
> -Dlibrary.leveldbjni.path=/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER
> -Dhadoop.event.appender=,EventCatcher -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=/tmp/yarn_yarn-NODEMANAGER-bef961adb705d3d90452581c3b4991c4_pid31718.hprof
> -XX:OnOutOfMemoryError=/opt/cm-5.16.2/lib64/cmf/service/common/killparent.sh
> -Dhadoop.log.dir=/var/log/hadoop-yarn -Dyarn.log.dir=/var/log/hadoop-yarn
> -Dhadoop.log.file=hadoop-cmf-yarn-NODEMANAGER-cdh5node1.log.out
> -Dyarn.log.file=hadoop-cmf-yarn-NODEMANAGER-cdh5node1.log.out
> -Dyarn.home.dir=/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn
> -Dhadoop.home.dir=/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn
> -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA
> -Djava.library.path=/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/lib/native\"!\n\031HADOOP_HOME_WARN_SUPPRESS\022\004true\"\017\n\aJVM_PID\022\0045524\"O\n\tJSVC_HOME\022B/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/bigtop-utils\"w\n\003PWD\022p/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/container_e176_1637293573159_0150_01_000002\"V\n\017CDH_HTTPFS_HOME\022C/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-httpfs\"P\n\016CDH_FLUME_HOME\022>/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/flume-ng\"Q\n\020CDH_PARQUET_HOME\022=/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/parquet\"\"\n\006python\022\030venv.zip/venv/bin/python\"%\n\bLESSOPEN\022\031||/usr/bin/lesspipe.sh
> %s\"\035\n\031SELINUX_USE_CURRENT_RANGE\022\000\"\034\n\fpython.files\022\f/opt/test.py\"O\n\017CDH_HADOOP_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop\"A\n\nweb.tmpdir\0223/tmp/flink-web-e35b2953-cfce-4655-bbac-bbacf1dbba5e\"\235\001\n\023_PYTHON_WORKING_DIR\022\205\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives\"\035\n\vPYTHON_HOME\022\016/opt/anaconda3\"H\n\034CLOUDERA_MYSQL_CONNECTOR_JAR\022(/usr/share/java/mysql-connector-java.jar\"I\n\fCDH_HUE_HOME\0229/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hue\"
> \n\034NM_AUX_SERVICE_spark_shuffle\022\000\"1\n
> __table.query-start.local-time__\022\r1637330289401\"8\n\020HIVE_DEFAULT_XML\022$/etc/hive/conf.dist/hive-default.xml\"\031\n\025CONDA_PROMPT_MODIFIER\022\000\"\n\n\005SHLVL\022\0014\"I\n\036$internal.yarn.log-config-file\022\'/opt/flink-1.14.0/conf/log4j.properties\"u\n\035python.internal.files-key-map\022Tpython_file_13c795ab581a80efa6a9947d919503287f51d055172132c4457dc7e58ad114cc:test.py\"3\n\017CMF_PACKAGE_DIR\022
> /opt/cm-5.16.2/lib64/cmf/service\"-\n\017CGROUP_ROOT_CPU\022\032/sys/fs/cgroup/cpu,cpuacct\"%\n\tCONDA_EXE\022\030/opt/anaconda3/bin/conda\"\020\n\bHISTSIZE\022\0041000\"\036\n\tJAVA_HOME\022\021/usr/java/default\"5\n\fCMF_CONF_DIR\022%/opt/cm-5.16.2/etc/cloudera-scm-agent\"0\n\"jobmanager.memory.jvm-overhead.min\022\n201326592b\"\r\n\004TERM\022\005xterm\"R\n\rCDH_HDFS_HOME\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-hdfs\"\023\n\016XDG_SESSION_ID\022\0017\"\033\n\027SELINUX_LEVEL_REQUESTED\022\000\"=\n\fYARN_LOGFILE\022-hadoop-cmf-yarn-NODEMANAGER-cdh5node1.log.out\"+\n\020HADOOP_USER_NAME\022\
> 027tempodata@MERITDATA.COM\"3\n*execution.savepoint.ignore-unclaimed-state\022\005false\"X\n\023SCM_DEFINES_SCRIPTS\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/meta/cdh_env.sh\"(\n\033_PYTHON_WORKER_MEMORY_LIMIT\022\t536870920\"U\n\020HADOOP_HDFS_HOME\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-hdfs\"\\\n\022HADOOP_MAPRED_HOME\022F/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-mapreduce\"1\n\023CGROUP_ROOT_CPUACCT\022\032/sys/fs/cgroup/cpu,cpuacct\"/\n\023CMF_AGENT_MGMT_HOME\022\030/opt/cm-5.16.2/share/cmf\"\037\n\001_\022\032/usr/java/default/bin/java\"%\n\tMGMT_HOME\022\030/opt/cm-5.16.2/share/cmf\"\026\n\022CGROUP_GROUP_BLKIO\022\000\"\025\n\aSSH_TTY\022\n/dev/pts/2\"#\n\nSSH_CLIENT\022\025191.168.1.34
> 61192 22\"
> \n\030PYFLINK_GATEWAY_DISABLED\022\004true\"K\n\rCDH_AVRO_HOME\022:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/avro\"
> \n\016CDH_SQOOP_HOME\022\016/usr/lib/sqoop\"T\n\024CDH_HUE_PLUGINS_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop\"\'\n\036jobmanager.memory.process.size\022\0051600m\"\020\n\vCDH_VERSION\022\0015\"!\n\035HADOOP_JOB_HISTORYSERVER_OPTS\022\000\"
> \n\016CDH_KAFKA_HOME\022\016/usr/lib/kafka\"J\n\035CLOUDERA_ORACLE_CONNECTOR_JAR\022)/usr/share/java/oracle-connector-java.jar\"4\n\016SSH_CONNECTION\022\"191.168.1.34
> 61192 191.168.1.24
> 22\")\n\037internal.cluster.execution-mode\022\006NORMAL\"(\n\aNLSPATH\022\035/usr/dt/lib/nls/msg/%L/%N.cat\"7\n\022KEYTRUSTEE_KP_HOME\022!/usr/share/keytrustee-keyprovider\"+\n\017CMF_SERVER_ROOT\022\030/opt/cm-5.16.2/share/cmf\"\035\n\aCMF_ETC\022\022/opt/cm-5.16.2/etc\"2\n$jobmanager.memory.jvm-metaspace.size\022\n268435456b\"M\n\016CDH_SPARK_HOME\022;/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/spark\"\030\n\024CGROUP_GROUP_CPUACCT\022\000\";\n\fCONTAINER_ID\022+container_e176_1637293573159_0150_01_000002\"M\n\016CDH_LLAMA_HOME\022;/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/llama\"t\n\rXDG_DATA_DIRS\022c/root/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share\"K\n\rCDH_KUDU_HOME\022:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/kudu\"%\n\fPARCELS_ROOT\022\025/opt/cloudera/parcels\">\n\034high-availability.cluster-id\022\036application_1637293573159_0150\"\035\n\031YARN_RESOURCEMANAGER_OPTS\022\000\"4\n\030python.client.executable\022\030venv.zip/venv/bin/python\"\026\n\rtable.planner\022\005BLINK\"\034\n\004MAIL\022\024/var/spool/mail/root\"1\n
> __table.query-start.epoch-time__\022\r1637301489401\"\030\n\023parallelism.default\022\0011\"\027\n\023pipeline.classpaths\022\000\"M\n\016CDH_HBASE_HOME\022;/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hbase\"\024\n\aLOGNAME\022\ttempodata\"V\n\fCDH_MR2_HOME\022F/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-mapreduce\"R\n\vTOMCAT_HOME\022C/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/bigtop-tomcat\"\033\n\016_FLINK_NODE_ID\022\tcdh5node1\"I\n\fCDH_PIG_HOME\0229/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/pig\"\034\n\026yarn.application.queue\022\002df\"M\n\rHADOOP_PREFIX\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop\"\232\001\n\nLOCAL_DIRS\022\213\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150,/hdfs/yarn/usercache/tempodata/appcache/application_1637293573159_0150\"\342\001\n\nPYTHONPATH\022\323\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-files/blob_p-30842d5cb48fe53fd9f94c2a4e364225246729cc-b3cef870d37841a25e243a9ca6b64863\"(\n\037taskmanager.memory.process.size\022\0051728m\"\022\n\005SHELL\022\t/bin/bash\"\r\n\bweb.port\022\0010\"8\n\026HADOOP_CLIENT_CONF_DIR\022\036/etc/hadoop/conf.cloudera.yarn\"U\n\022CDH_ZOOKEEPER_HOME\022?/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/zookeeper\"T\n\rYARN_CONF_DIR\022C/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER\"-\n\037jobmanager.memory.off-heap.size\022\n134217728b\"|\n
> python.internal.archives-key-map\022Xpython_archive_bfe925aa9bea9ad79f3e1b595cb3e39a3acfa4656b3a5870173959195400aabb:venv.zip\"\321\001\n\bLOG_DIRS\022\304\001/yarn/container-logs/application_1637293573159_0150/container_e176_1637293573159_0150_01_000002,/hdfs/yarn-container-logs/application_1637293573159_0150/container_e176_1637293573159_0150_01_000002\"R\n
> NM_AUX_SERVICE_mapreduce_shuffle\022.AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=\r\n\"W\n\034CLOUDERA_POSTGRESQL_JDBC_JAR\0227/opt/cm-5.16.2/share/cmf/lib/postgresql-42.1.4.jre7.jar\"U\n\020HADOOP_YARN_HOME\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn\".\n\017PARCEL_DIRNAMES\022\033CDH-5.16.2-1.cdh5.16.2.p0.8\"a\n\025SUPERVISOR_SERVER_URL\022Hunix:///opt/cm-5.16.2/run/cloudera-scm-agent/supervisor/supervisord.sock\"\034\n\023jobmanager.rpc.port\022\00539719\"\"\n\vCMF_SBINDIR\022\023/opt/cm-5.16.2/sbin\"w\n\023WEBHCAT_DEFAULT_XML\022`/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/etc/hive-webhcat/conf.dist/webhcat-default.xml\"\"\n\017CDH_CRUNCH_HOME\022\017/usr/lib/crunch\"0\n\027SUPERVISOR_PROCESS_NAME\022\0253224-yarn-NODEMANAGER\".\n\025SUPERVISOR_GROUP_NAME\022\0253224-yarn-NODEMANAGER\"-\n\021python.executable\022\030venv.zip/venv/bin/python\"\373\004\n\020_FLINK_CLASSPATH\022\346\004:flink-python-test-1.0-SNAPSHOT.jar:lib/flink-csv-1.14.0.jar:lib/flink-json-1.14.0.jar:lib/flink-python_2.11-1.14.0.jar:lib/flink-shaded-zookeeper-3.4.14.jar:lib/flink-table_2.11-1.14.0.jar:lib/lib/flink-csv-1.14.0.jar:lib/lib/flink-json-1.14.0.jar:lib/lib/flink-shaded-zookeeper-3.4.14.jar:lib/lib/flink-table_2.11-1.14.0.jar:lib/lib/log4j-1.2-api-2.14.1.jar:lib/lib/log4j-api-2.14.1.jar:lib/lib/log4j-core-2.14.1.jar:lib/lib/log4j-slf4j-impl-2.14.1.jar:lib/log4j-1.2-api-2.14.1.jar:lib/log4j-api-2.14.1.jar:lib/log4j-core-2.14.1.jar:lib/log4j-slf4j-impl-2.14.1.jar:lib/flink-dist_2.11-1.14.0.jar:flink-conf.yaml:\")\n\021CGROUP_ROOT_BLKIO\022\024/sys/fs/cgroup/blkio\"\031\n\frest.address\022\tcdh5node1\"K\n\rCDH_SOLR_HOME\022:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/solr\"\237\v\n\tLS_COLORS\022\221\vrs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:\"T\n\rCDH_HCAT_HOME\022C/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hive-hcatalog\"9\n\037$internal.deployment.config-dir\022\026/opt/flink-1.14.0/conf\"0\n\"jobmanager.memory.jvm-overhead.max\022\n201326592b\"M\n\016CDH_OOZIE_HOME\022;/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/oozie\"O\n\017CDH_SENTRY_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/sentry\"
> \n\004ROOT\022\030/opt/cm-5.16.2/lib64/cmf\"\034\n\020YARN_ROOT_LOGGER\022\bINFO,RFA\"0\n&jobmanager.execution.failover-strategy\022\006region\"\031\n\aCMF_VAR\022\016/opt/cm-5.16.2\"O\n\bCONF_DIR\022C/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER\"-\n\017XFILESEARCHPATH\022\032/usr/dt/app-defaults/%L/Dt\"\037\n\026execution.runtime-mode\022\005BATCH\"\023\n\004LANG\022\vzh_CN.UTF-8\"\021\n\fCMF_SUDO_CMD\022\001
> \"\024\n\aNM_HOST\022\tcdh5node1\"\205\001\n\fBOOT_LOG_DIR\022u/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69\"\031\n\aDISPLAY\022\016localhost:12.0\"Z\n\022HADOOP_LIBEXEC_DIR\022D/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec\"\221\a\n\020HADOOP_CLASSPATH\022\374\006/opt/cm-5.16.2/share/cmf/lib/plugins/event-publish-5.16.2-shaded.jar:/opt/cm-5.16.2/share/cmf/lib/plugins/tt-instrumentation-5.16.2.jar:/etc/hadoop/conf:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-yarn/.//*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*\"\032\n\026SELINUX_ROLE_REQUESTED\022\000\"R\n\026yarn.provided.lib.dirs\0228hdfs://nameservice1/user/tempodata/lixu/flink-1.14.0/lib\"\025\n\rIS_KERBERIZED\022\004true\"+\n\022CGROUP_ROOT_MEMORY\022\025/sys/fs/cgroup/memory\"$\n\fYARN_LOG_DIR\022\024/var/log/hadoop-yarn\"
> \n\017python.archives\022\r/opt/venv.zip\"\"\n\035taskmanager.numberOfTaskSlots\022\0011\"\024\n\020CGROUP_GROUP_CPU\022\000\".\n\vORACLE_HOME\022\037/usr/share/oracle/instantclient\"O\n\017CDH_SQOOP2_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/sqoop2\"R\n\022HADOOP_COMMON_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop\"\020\n\vCONDA_SHLVL\022\0011\"M\n\032$internal.application.main\022/com.meritdata.cloud.flink.python.test.PyUDFTest\"K\n\rCDH_HIVE_HOME\022:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hive\"P\n\fCDH_KMS_HOME\022@/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-kms\"\024\n\fNM_HTTP_PORT\022\0048042\"O\n\017CDH_IMPALA_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/impala\"*\n\016CMF_AGENT_ROOT\022\030/opt/cm-5.16.2/lib64/cmf\"$\n\rNM_LOCAL_DIRS\022\023/yarn/nm
> /hdfs/yarn\"\017\n\aNM_PORT\022\0048041\"\361\003\n\025YARN_NODEMANAGER_OPTS\022\327\003-Xms1073741824
> -Xmx1073741824 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled
> -Dlibrary.leveldbjni.path=/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER
> -Dhadoop.event.appender=,EventCatcher -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=/tmp/yarn_yarn-NODEMANAGER-bef961adb705d3d90452581c3b4991c4_pid31718.hprof
> -XX:OnOutOfMemoryError=/opt/cm-5.16.2/lib64/cmf/service/common/killparent.sh\"\034\n\020execution.target\022\bembedded\"\021\n\004USER\022\ttempodata\"\361\b\n\tCLASSPATH\022\343\b:flink-python-test-1.0-SNAPSHOT.jar:lib/flink-csv-1.14.0.jar:lib/flink-json-1.14.0.jar:lib/flink-python_2.11-1.14.0.jar:lib/flink-shaded-zookeeper-3.4.14.jar:lib/flink-table_2.11-1.14.0.jar:lib/lib/flink-csv-1.14.0.jar:lib/lib/flink-json-1.14.0.jar:lib/lib/flink-shaded-zookeeper-3.4.14.jar:lib/lib/flink-table_2.11-1.14.0.jar:lib/lib/log4j-1.2-api-2.14.1.jar:lib/lib/log4j-api-2.14.1.jar:lib/lib/log4j-core-2.14.1.jar:lib/lib/log4j-slf4j-impl-2.14.1.jar:lib/log4j-1.2-api-2.14.1.jar:lib/log4j-api-2.14.1.jar:lib/log4j-core-2.14.1.jar:lib/log4j-slf4j-impl-2.14.1.jar:lib/flink-dist_2.11-1.14.0.jar:flink-conf.yaml::/etc/hadoop/conf.cloudera.yarn:/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-hdfs/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn/lib/*\"-\n%
> internal.io.tmpdirs.use-local-default\022\004true\"Y\n\016CDH_HADOOP_BIN\022G/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/bin/hadoop\"[\n\fCDH_MR1_HOME\022K/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-0.20-mapreduce\"\240\001\n\032HADOOP_TOKEN_FILE_LOCATION\022\201\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/container_e176_1637293573159_0150_01_000002/container_tokens\"\025\n\bHOSTNAME\022\tcdh5node1\"\032\n\022execution.attached\022\004true\",\n#execution.shutdown-on-attached-exit\022\005false\"\252\001\n\rpipeline.jars\022\230\001file:/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/container_e176_1637293573159_0150_01_000001/flink-python-test-1.0-SNAPSHOT.jar\"\221\001\n\017CM_STATUS_CODES\022~STATUS_NONE
> HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED
> JOBTRACKER_IN_STANDBY_MODE
> YARN_RM_IN_STANDBY_MODE\"R\n\rCDH_YARN_HOME\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn\"K\n\vSEARCH_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/search\"\036\n\017XDG_RUNTIME_DIR\022\v/run/user/0\"\016\n\004HOME\022\006/home/\"*\n\033jobmanager.memory.heap.size\022\v1073741824b\"\025\n\020MALLOC_ARENA_MAX\022\0014"
>
>
> 2021-11-19 14:02:24,050 INFO
> org.apache.beam.runners.fnexecution.logging.GrpcLoggingService [] - 1 Beam
> Fn Logging clients still connected during shutdown.
> 2021-11-19 14:02:25,332 WARN
> org.apache.flink.runtime.taskmanager.Task
> [] - Source: Values(tuples=[[{
> _UTF-16LE'1' }, { _UTF-16LE'2' }, { _UTF-16LE'3' }]], values=[f0]) ->
> Calc(select=[f0]) -> PythonCalc(select=[func1(f0) AS a]) -> Sink:
> Collect table sink (1/1)#0 (bef195197949f0a6c3716ff2e6e8b17d) switched from
> RUNNING to FAILED with failure cause: java.lang.RuntimeException: Failed to
> start remote bundle
> at
> org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.startBundle(BeamPythonFunctionRunner.java:333)
> at
> org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.checkInvokeStartBundle(BeamPythonFunctionRunner.java:314)
> at
> org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.process(BeamPythonFunctionRunner.java:293)
> at
> org.apache.flink.table.runtime.operators.python.scalar.PythonScalarFunctionOperator.processElementInternal(PythonScalarFunctionOperator.java:89)
> at
> org.apache.flink.table.runtime.operators.python.scalar.PythonScalarFunctionOperator.processElementInternal(PythonScalarFunctionOperator.java:38)
> at
> org.apache.flink.table.runtime.operators.python.AbstractStatelessFunctionOperator.processElement(AbstractStatelessFunctionOperator.java:117)
> at
> org.apache.flink.streaming.runtime.tasks.ChainingOutput.pushToOperator(ChainingOutput.java:99)
> at
> org.apache.flink.streaming.runtime.tasks.ChainingOutput.collect(ChainingOutput.java:80)
> at
> org.apache.flink.streaming.runtime.tasks.ChainingOutput.collect(ChainingOutput.java:39)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:29)
> at BatchExecCalc$8.processElement(Unknown Source)
> at
> org.apache.flink.streaming.runtime.tasks.ChainingOutput.pushToOperator(ChainingOutput.java:99)
> at
> org.apache.flink.streaming.runtime.tasks.ChainingOutput.collect(ChainingOutput.java:80)
> at
> org.apache.flink.streaming.runtime.tasks.ChainingOutput.collect(ChainingOutput.java:39)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:29)
> at
> org.apache.flink.streaming.api.operators.StreamSourceContexts$ManualWatermarkContext.processAndCollect(StreamSourceContexts.java:418)
> at
> org.apache.flink.streaming.api.operators.StreamSourceContexts$WatermarkContext.collect(StreamSourceContexts.java:513)
> at
> org.apache.flink.streaming.api.operators.StreamSourceContexts$SwitchingOnClose.collect(StreamSourceContexts.java:103)
> at
> org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:92)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:116)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:73)
> at
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:323)
> Caused by: java.lang.RuntimeException: No client connected within timeout
> at
> org.apache.beam.runners.fnexecution.data.GrpcDataService.send(GrpcDataService.java:192)
> at
> org.apache.beam.runners.fnexecution.control.SdkHarnessClient$BundleProcessor.newBundle(SdkHarnessClient.java:287)
> at
> org.apache.beam.runners.fnexecution.control.SdkHarnessClient$BundleProcessor.newBundle(SdkHarnessClient.java:197)
> at
> org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$SimpleStageBundleFactory.getBundle(DefaultJobBundleFactory.java:519)
> at
> org.apache.beam.runners.fnexecution.control.StageBundleFactory.getBundle(StageBundleFactory.java:87)
> at
> org.apache.beam.runners.fnexecution.control.StageBundleFactory.getBundle(StageBundleFactory.java:76)
> at
> org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.startBundle(BeamPythonFunctionRunner.java:323)
> ... 23 more
> Caused by: java.util.concurrent.TimeoutException: Waited 3 minutes for
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.SettableFuture@533506fe
> [status=PENDING]
> at
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:471)
> at
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:90)
> at
> org.apache.beam.runners.fnexecution.data.GrpcDataService.send(GrpcDataService.java:187)
> ... 29 more
>
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人:
> "user-zh"
> <
> dian0511.fu@gmail.com>;
> 发送时间: 2021年11月19日(星期五) 中午11:03
> 收件人: "user-zh"<user-zh@flink.apache.org>;
>
> 主题: Re: 哪里可以下载到downloads/setup-pyflink-virtual-env.sh脚本
>
>
>
> 刚注意到你用的YARN application模式，PyFlink 1.14.0才支持YARN application模式，主要是新增了命令行选项“
> -pyclientexec” 和配置“python.client.executable”：
>
> https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/python/python_config/#python-client-executable
>
> 对于你这个作业来说，你需要通过使用1.14.0版本，同时添加命令行选项：-pyclientexec venv.zip/venv/bin/python
>
>
> On Fri, Nov 19, 2021 at 10:48 AM Asahi Lee <978466273@qq.com.invalid>
> wrote:
>
> > 我通过source
> my_env/bin/activate和指定环境变量PYFLINK_CLIENT_EXECUTABLE均未提交成功，错误未变化；
> > 我的错误中，jobmanager的日志中显示No module named pyflink，而jobmanager是运行在yarn集群上；
> > 请问，还有还有什么未配置？
> >
> >
> > &gt; LogType:jobmanager.out
> > &gt; Log Upload Time:星期四 十一月 18 20:48:45 +0800 2021
> > &gt; LogLength:37
> > &gt; Log Contents:
> > &gt; /bin/python: No module named pyflink
> >
> >
> >
> >
> > ------------------&nbsp;原始邮件&nbsp;------------------
> > 发件人:
> >
> "user-zh"
> >
> <
> > dian0511.fu@gmail.com&gt;;
> > 发送时间:&nbsp;2021年11月19日(星期五) 上午9:38
> > 收件人:&nbsp;"user-zh"<user-zh@flink.apache.org&gt;;
> >
> > 主题:&nbsp;Re: 哪里可以下载到downloads/setup-pyflink-virtual-env.sh脚本
> >
> >
> >
> > -pyexec 指定的是集群端所用的Python环境，客户端需要编译Flink作业，也会依赖Python环境。可以看一下这个文档:
> >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/dev/python/dependency_management/#python-interpreter-of-client
> >
> <https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/dev/python/dependency_management/#python-interpreter-of-client>>
> ;
> > On Thu, Nov 18, 2021 at 9:00 PM Asahi Lee <978466273@qq.com.invalid
> &gt;
> > wrote:
> >
> > &gt; Hi !
> > &gt; &amp;nbsp; &amp;nbsp;我在java Table api中使用python udf
> > &gt; 函数，通过下面的命令提交应用，报无法启动python服务错误，请问我的提交方式对吗？jm日志为/bin/python:
> No module
> > named
> > &gt; pyflink。
> > &gt;
> > &gt;
> > &gt; ./flink-1.13.2/bin/flink&amp;nbsp;
> > &gt; run-application -t yarn-application&amp;nbsp;
> > &gt;
> >
> -Dyarn.provided.lib.dirs="hdfs://nameservice1/user/flink/flinklib"&amp;nbsp;
> > &gt; -Dyarn.application.queue=d
> > &gt; -p 1&amp;nbsp;
> > &gt; -pyarch /opt/venv.zip
> > &gt; -pyexec venv.zip/venv/bin/python&amp;nbsp;
> > &gt; -pyfs /opt/test.py&amp;nbsp;
> > &gt; -c test.PyUDFTest&amp;nbsp;
> > &gt; /opt/flink-python-test-1.0-SNAPSHOT.jar
> > &gt;
> > &gt;
> > &gt;
> > &gt; 错误：
> > &gt; Caused by: java.lang.RuntimeException: Python callback
> server start
> > failed!
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.flink.client.python.PythonFunctionFactory.createPythonFunctionFactory(PythonFunctionFactory.java:167)
> > &gt; ~[flink-python_2.11-1.13.2.jar:1.13.2]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.flink.client.python.PythonFunctionFactory$1.load(PythonFunctionFactory.java:88)
> > &gt; ~[flink-python_2.11-1.13.2.jar:1.13.2]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.flink.client.python.PythonFunctionFactory$1.load(PythonFunctionFactory.java:84)
> > &gt; ~[flink-python_2.11-1.13.2.jar:1.13.2]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
> > &gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
> > &gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
> > &gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
> > &gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> > &gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
> > &gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
> > &gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.flink.client.python.PythonFunctionFactory.getPythonFunction(PythonFunctionFactory.java:129)
> > &gt; ~[flink-python_2.11-1.13.2.jar:1.13.2]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt; sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > ~[?:1.8.0_111]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > &gt; ~[?:1.8.0_111]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > &gt; ~[?:1.8.0_111]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt; java.lang.reflect.Method.invoke(Method.java:498)
> ~[?:1.8.0_111]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.flink.table.functions.python.utils.PythonFunctionUtils.getPythonFunction(PythonFunctionUtils.java:45)
> > &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.flink.table.functions.UserDefinedFunctionHelper.instantiateFunction(UserDefinedFunctionHelper.java:206)
> > &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.flink.table.catalog.FunctionCatalog.getFunctionDefinition(FunctionCatalog.java:659)
> > &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.flink.table.catalog.FunctionCatalog.resolveAmbiguousFunctionReference(FunctionCatalog.java:606)
> > &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.flink.table.catalog.FunctionCatalog.lookupFunction(FunctionCatalog.java:362)
> > &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.flink.table.planner.catalog.FunctionCatalogOperatorTable.lookupOperatorOverloads(FunctionCatalogOperatorTable.java:97)
> > &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.calcite.sql.util.ChainedSqlOperatorTable.lookupOperatorOverloads(ChainedSqlOperatorTable.java:67)
> > &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1183)
> > &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1169)
> > &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1200)
> > &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1169)
> > &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:945)
> > &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:704)
> > &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt; org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org
> >
> $apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:152)
> > &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:110)
> > &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:201)
> > &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:101)
> > &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:704)
> > &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > test.PyUDFTest.main(PyUDFTest.java:22)
> > &gt; ~[flink-python-test-1.0-SNAPSHOT.jar:?]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt; sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > ~[?:1.8.0_111]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > &gt; ~[?:1.8.0_111]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > &gt; ~[?:1.8.0_111]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt; java.lang.reflect.Method.invoke(Method.java:498)
> ~[?:1.8.0_111]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
> > &gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
> > &gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> > &gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> at
> > &gt;
> >
> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:242)
> > &gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> ... 10 more
> > &gt; 2021-11-18 20:48:44,458 INFO&amp;nbsp;
> > &gt; org.apache.flink.runtime.blob.BlobServer&amp;nbsp;
> &amp;nbsp;
> > &amp;nbsp; &amp;nbsp; &amp;nbsp;
> > &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;
> &amp;nbsp; &amp;nbsp;[] -
> > Stopped BLOB server at
> > &gt; 0.0.0.0:45070
> > &gt; 2021-11-18 20:48:44,458 INFO&amp;nbsp;
> > &gt;
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint&amp;nbsp;
> > &amp;nbsp; &amp;nbsp;
> > &gt; &amp;nbsp; [] - Shutting
> YarnApplicationClusterEntryPoint down with
> > application
> > &gt; status UNKNOWN. Diagnostics Cluster entrypoint has been
> closed
> > externally..
> > &gt; 2021-11-18 20:48:44,458 INFO&amp;nbsp;
> > &gt;
> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] -
> > Shutting
> > &gt; down rest endpoint.
> > &gt; 2021-11-18 20:48:44,474 INFO&amp;nbsp;
> > &gt;
> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] -
> > Removing
> > &gt; cache directory
> > &gt;
> /tmp/flink-web-dd82c3c0-f457-492d-8e64-5ae74fe9abbd/flink-web-ui
> > &gt; 2021-11-18 20:48:44,474 INFO&amp;nbsp;
> > &gt;
> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] -
> > &gt; http://cdh5node3:40216 lost leadership
> > &gt; 2021-11-18 20:48:44,474 INFO&amp;nbsp;
> > &gt;
> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] -
> > Shut
> > &gt; down complete.
> > &gt; 2021-11-18 20:48:44,474 INFO&amp;nbsp;
> > &gt;
> >
> org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent
> > &gt; [] - Closing components.
> > &gt; 2021-11-18 20:48:44,475 INFO&amp;nbsp;
> > &gt;
> >
> org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess
> > &gt; [] - Stopping SessionDispatcherLeaderProcess.
> > &gt; 2021-11-18 20:48:44,475 INFO&amp;nbsp;
> > &gt;
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher&amp;nbsp;
> > &amp;nbsp;
> > &gt; &amp;nbsp;[] - Stopping dispatcher
> akka.tcp://flink@cdh5node3
> > &gt; :34697/user/rpc/dispatcher_1.
> > &gt; 2021-11-18 20:48:44,476 INFO&amp;nbsp;
> > &gt;
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher&amp;nbsp;
> > &amp;nbsp;
> > &gt; &amp;nbsp;[] - Stopping all currently running jobs of
> dispatcher
> > &gt; akka.tcp://flink@cdh5node3:34697/user/rpc/dispatcher_1.
> > &gt;
> > &gt;
> > &gt; LogType:jobmanager.out
> > &gt; Log Upload Time:星期四 十一月 18 20:48:45 +0800 2021
> > &gt; LogLength:37
> > &gt; Log Contents:
> > &gt; /bin/python: No module named pyflink
> > &gt;
> > &gt;
> > &gt;
> > &gt;
> > &gt;
> > &gt;
> > &gt;
> > &gt;
> > &gt;
> > &gt;
> > &gt;
> > &gt;
> > &gt;
> > &gt;
> ------------------&amp;nbsp;原始邮件&amp;nbsp;------------------
> > &gt; 发件人:
> >
> &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> > "user-zh"
> >
> &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> > <
> > &gt; xyzhong131@163.com&amp;gt;;
> > &gt; 发送时间:&amp;nbsp;2021年11月18日(星期四) 下午3:34
> > &gt; 收件人:&amp;nbsp;"user-zh"<user-zh@flink.apache.org
> &amp;gt;;
> > &gt;
> > &gt;
> 主题:&amp;nbsp;Re:哪里可以下载到downloads/setup-pyflink-virtual-env.sh脚本
> > &gt;
> > &gt;
> > &gt;
> > &gt; Hi!<br/&amp;gt;我在下面的旧版本中找到了，你试下能不能用：<br/&amp;gt;
> > &gt;
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.12/downloads/setup-pyflink-virtual-env.sh
> >
> <https://nightlies.apache.org/flink/flink-docs-release-1.12/downloads/setup-pyflink-virtual-env.sh>>;
> &gt
> > <
> https://nightlies.apache.org/flink/flink-docs-release-1.12/downloads/setup-pyflink-virtual-env.sh&gt>
> ;;
> > 在 2021-11-18 15:05:03，"Asahi Lee" <978466273@qq.com.INVALID&amp;gt;
> 写道：
> > &gt; &amp;gt;Hi!
> > &gt; &amp;gt;&amp;amp;nbsp; &amp;amp;nbsp;
> > &gt;
> >
> 我在flink官方文档中看到通过&amp;amp;nbsp;&amp;amp;nbsp;setup-pyflink-virtual-env.sh
> > &gt; 脚本制作python虚拟环境，请问再哪里可以下载到？文档地址如下：
> > &gt; &amp;gt;
> > &gt;
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.14/zh/docs/dev/python/faq/
回复： 哪里可以下载到downloads/setup-pyflink-virtual-env.sh脚本
Posted by Asahi Lee <as...@qq.com.INVALID>.
Hi!
我通过如下命令提交成功了，python的参数需求-D方式传入，-py方式传入不生效：
./flink-1.14.0/bin/flink
run-application -t yarn-application
-Dyarn.provided.lib.dirs="hdfs://nameservice1/user/flink/flinklib"
-Dyarn.application.queue=d
-Dpython.archives="/opt/venv.zip"
-Dpython.client.executable="venv.zip/venv/bin/python"
-Dpython.executable="venv.zip/venv/bin/python"
-Dpython.files="/opt/test.py"
-p 1
-c test.PyUDFTest
/opt/flink-python-test-1.0-SNAPSHOT.jar
但是我应用提交成功后，程序运行发生错误，如下：
taskmanager.log
2021-11-19 13:59:24,030 ERROR /yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py:296 [] - Error processing instruction 1. Original traceback is
Traceback (most recent call last):
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 289, in _execute
response = task()
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 362, in <lambda>
lambda: self.create_worker().do_instruction(request), request)
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 607, in do_instruction
getattr(request, request_type), request.instruction_id)
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 638, in process_bundle
instruction_id, request.process_bundle_descriptor_id)
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 467, in get
self.data_channel_factory)
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 868, in __init__
self.ops = self.create_execution_tree(self.process_bundle_descriptor)
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 925, in create_execution_tree
descriptor.transforms, key=topological_height, reverse=True)])
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 924, in <listcomp>
get_operation(transform_id))) for transform_id in sorted(
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 812, in wrapper
result = cache[args] = func(*args)
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 906, in get_operation
pcoll_id in descriptor.transforms[transform_id].outputs.items()
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 906, in <dictcomp>
pcoll_id in descriptor.transforms[transform_id].outputs.items()
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 904, in <listcomp>
tag: [get_operation(op) for op in pcoll_consumers[pcoll_id]]
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 812, in wrapper
result = cache[args] = func(*args)
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 906, in get_operation
pcoll_id in descriptor.transforms[transform_id].outputs.items()
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 906, in <dictcomp>
pcoll_id in descriptor.transforms[transform_id].outputs.items()
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 904, in <listcomp>
tag: [get_operation(op) for op in pcoll_consumers[pcoll_id]]
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 812, in wrapper
result = cache[args] = func(*args)
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 909, in get_operation
transform_id, transform_consumers)
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1198, in create_operation
return creator(self, transform_id, transform_proto, payload, consumers)
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1320, in create_sink_runner
output_coder = factory.get_coder(grpc_port.coder_id)
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1222, in get_coder
return self.context.coders.get_by_id(coder_id)
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/pipeline_context.py", line 116, in get_by_id
self._id_to_proto[id], self._pipeline_context)
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/coders/coders.py", line 370, in from_runner_api
[context.coders.get_by_id(c) for c in coder_proto.component_coder_ids],
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/coders/coders.py", line 370, in <listcomp>
[context.coders.get_by_id(c) for c in coder_proto.component_coder_ids],
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/pipeline_context.py", line 116, in get_by_id
self._id_to_proto[id], self._pipeline_context)
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/coders/coders.py", line 370, in from_runner_api
[context.coders.get_by_id(c) for c in coder_proto.component_coder_ids],
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/coders/coders.py", line 370, in <listcomp>
[context.coders.get_by_id(c) for c in coder_proto.component_coder_ids],
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/runners/pipeline_context.py", line 116, in get_by_id
self._id_to_proto[id], self._pipeline_context)
File "/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/apache_beam/coders/coders.py", line 367, in from_runner_api
parameter_type, constructor = cls._known_urns[coder_proto.spec.urn]
KeyError: 'flink:coder:v1'
2021-11-19 14:02:24,031 WARN org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory [] - Expiring environment urn: "beam:env:process:v1"
payload: "\032\321\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/pyflink/bin/pyflink-udf-runner.sh\"%\n\004PATH\022\035/sbin:/usr/sbin:/bin:/usr/bin\"V\n\017HADOOP_CONF_DIR\022C/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER\"\031\n\vHISTCONTROL\022\nignoredups\"\027\n\022SUPERVISOR_ENABLED\022\0011\"\031\n\021CONDA_DEFAULT_ENV\022\004base\"#\n\026jobmanager.rpc.address\022\tcdh5node1\"4\n\026KEYTRUSTEE_SERVER_HOME\022\032/usr/lib/keytrustee-server\"-\n\020CONDA_PYTHON_EXE\022\031/opt/anaconda3/bin/python\"\027\n\023CGROUP_GROUP_MEMORY\022\000\"$\n\023table.exec.timezone\022\rAsia/Shanghai\"Z\n\026CDH_HBASE_INDEXER_HOME\022@/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hbase-solr\"\036\n\fCONDA_PREFIX\022\016/opt/anaconda3\"\233\001\n\vio.tmp.dirs\022\213\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150,/hdfs/yarn/usercache/tempodata/appcache/application_1637293573159_0150\"\222\b\n\tYARN_OPTS\022\204\b-Djava.net.preferIPv4Stack=true -server -Xms1073741824 -Xmx1073741824 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -Dlibrary.leveldbjni.path=/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER -Dhadoop.event.appender=,EventCatcher -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/yarn_yarn-NODEMANAGER-bef961adb705d3d90452581c3b4991c4_pid31718.hprof -XX:OnOutOfMemoryError=/opt/cm-5.16.2/lib64/cmf/service/common/killparent.sh -Dhadoop.log.dir=/var/log/hadoop-yarn -Dyarn.log.dir=/var/log/hadoop-yarn -Dhadoop.log.file=hadoop-cmf-yarn-NODEMANAGER-cdh5node1.log.out -Dyarn.log.file=hadoop-cmf-yarn-NODEMANAGER-cdh5node1.log.out -Dyarn.home.dir=/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn -Dhadoop.home.dir=/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/lib/native\"!\n\031HADOOP_HOME_WARN_SUPPRESS\022\004true\"\017\n\aJVM_PID\022\0045524\"O\n\tJSVC_HOME\022B/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/bigtop-utils\"w\n\003PWD\022p/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/container_e176_1637293573159_0150_01_000002\"V\n\017CDH_HTTPFS_HOME\022C/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-httpfs\"P\n\016CDH_FLUME_HOME\022>/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/flume-ng\"Q\n\020CDH_PARQUET_HOME\022=/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/parquet\"\"\n\006python\022\030venv.zip/venv/bin/python\"%\n\bLESSOPEN\022\031||/usr/bin/lesspipe.sh %s\"\035\n\031SELINUX_USE_CURRENT_RANGE\022\000\"\034\n\fpython.files\022\f/opt/test.py\"O\n\017CDH_HADOOP_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop\"A\n\nweb.tmpdir\0223/tmp/flink-web-e35b2953-cfce-4655-bbac-bbacf1dbba5e\"\235\001\n\023_PYTHON_WORKING_DIR\022\205\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives\"\035\n\vPYTHON_HOME\022\016/opt/anaconda3\"H\n\034CLOUDERA_MYSQL_CONNECTOR_JAR\022(/usr/share/java/mysql-connector-java.jar\"I\n\fCDH_HUE_HOME\0229/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hue\" \n\034NM_AUX_SERVICE_spark_shuffle\022\000\"1\n __table.query-start.local-time__\022\r1637330289401\"8\n\020HIVE_DEFAULT_XML\022$/etc/hive/conf.dist/hive-default.xml\"\031\n\025CONDA_PROMPT_MODIFIER\022\000\"\n\n\005SHLVL\022\0014\"I\n\036$internal.yarn.log-config-file\022\'/opt/flink-1.14.0/conf/log4j.properties\"u\n\035python.internal.files-key-map\022Tpython_file_13c795ab581a80efa6a9947d919503287f51d055172132c4457dc7e58ad114cc:test.py\"3\n\017CMF_PACKAGE_DIR\022 /opt/cm-5.16.2/lib64/cmf/service\"-\n\017CGROUP_ROOT_CPU\022\032/sys/fs/cgroup/cpu,cpuacct\"%\n\tCONDA_EXE\022\030/opt/anaconda3/bin/conda\"\020\n\bHISTSIZE\022\0041000\"\036\n\tJAVA_HOME\022\021/usr/java/default\"5\n\fCMF_CONF_DIR\022%/opt/cm-5.16.2/etc/cloudera-scm-agent\"0\n\"jobmanager.memory.jvm-overhead.min\022\n201326592b\"\r\n\004TERM\022\005xterm\"R\n\rCDH_HDFS_HOME\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-hdfs\"\023\n\016XDG_SESSION_ID\022\0017\"\033\n\027SELINUX_LEVEL_REQUESTED\022\000\"=\n\fYARN_LOGFILE\022-hadoop-cmf-yarn-NODEMANAGER-cdh5node1.log.out\"+\n\020HADOOP_USER_NAME\022\027tempodata@MERITDATA.COM\"3\n*execution.savepoint.ignore-unclaimed-state\022\005false\"X\n\023SCM_DEFINES_SCRIPTS\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/meta/cdh_env.sh\"(\n\033_PYTHON_WORKER_MEMORY_LIMIT\022\t536870920\"U\n\020HADOOP_HDFS_HOME\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-hdfs\"\\\n\022HADOOP_MAPRED_HOME\022F/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-mapreduce\"1\n\023CGROUP_ROOT_CPUACCT\022\032/sys/fs/cgroup/cpu,cpuacct\"/\n\023CMF_AGENT_MGMT_HOME\022\030/opt/cm-5.16.2/share/cmf\"\037\n\001_\022\032/usr/java/default/bin/java\"%\n\tMGMT_HOME\022\030/opt/cm-5.16.2/share/cmf\"\026\n\022CGROUP_GROUP_BLKIO\022\000\"\025\n\aSSH_TTY\022\n/dev/pts/2\"#\n\nSSH_CLIENT\022\025191.168.1.34 61192 22\" \n\030PYFLINK_GATEWAY_DISABLED\022\004true\"K\n\rCDH_AVRO_HOME\022:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/avro\" \n\016CDH_SQOOP_HOME\022\016/usr/lib/sqoop\"T\n\024CDH_HUE_PLUGINS_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop\"\'\n\036jobmanager.memory.process.size\022\0051600m\"\020\n\vCDH_VERSION\022\0015\"!\n\035HADOOP_JOB_HISTORYSERVER_OPTS\022\000\" \n\016CDH_KAFKA_HOME\022\016/usr/lib/kafka\"J\n\035CLOUDERA_ORACLE_CONNECTOR_JAR\022)/usr/share/java/oracle-connector-java.jar\"4\n\016SSH_CONNECTION\022\"191.168.1.34 61192 191.168.1.24 22\")\n\037internal.cluster.execution-mode\022\006NORMAL\"(\n\aNLSPATH\022\035/usr/dt/lib/nls/msg/%L/%N.cat\"7\n\022KEYTRUSTEE_KP_HOME\022!/usr/share/keytrustee-keyprovider\"+\n\017CMF_SERVER_ROOT\022\030/opt/cm-5.16.2/share/cmf\"\035\n\aCMF_ETC\022\022/opt/cm-5.16.2/etc\"2\n$jobmanager.memory.jvm-metaspace.size\022\n268435456b\"M\n\016CDH_SPARK_HOME\022;/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/spark\"\030\n\024CGROUP_GROUP_CPUACCT\022\000\";\n\fCONTAINER_ID\022+container_e176_1637293573159_0150_01_000002\"M\n\016CDH_LLAMA_HOME\022;/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/llama\"t\n\rXDG_DATA_DIRS\022c/root/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share\"K\n\rCDH_KUDU_HOME\022:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/kudu\"%\n\fPARCELS_ROOT\022\025/opt/cloudera/parcels\">\n\034high-availability.cluster-id\022\036application_1637293573159_0150\"\035\n\031YARN_RESOURCEMANAGER_OPTS\022\000\"4\n\030python.client.executable\022\030venv.zip/venv/bin/python\"\026\n\rtable.planner\022\005BLINK\"\034\n\004MAIL\022\024/var/spool/mail/root\"1\n __table.query-start.epoch-time__\022\r1637301489401\"\030\n\023parallelism.default\022\0011\"\027\n\023pipeline.classpaths\022\000\"M\n\016CDH_HBASE_HOME\022;/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hbase\"\024\n\aLOGNAME\022\ttempodata\"V\n\fCDH_MR2_HOME\022F/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-mapreduce\"R\n\vTOMCAT_HOME\022C/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/bigtop-tomcat\"\033\n\016_FLINK_NODE_ID\022\tcdh5node1\"I\n\fCDH_PIG_HOME\0229/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/pig\"\034\n\026yarn.application.queue\022\002df\"M\n\rHADOOP_PREFIX\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop\"\232\001\n\nLOCAL_DIRS\022\213\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150,/hdfs/yarn/usercache/tempodata/appcache/application_1637293573159_0150\"\342\001\n\nPYTHONPATH\022\323\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-files/blob_p-30842d5cb48fe53fd9f94c2a4e364225246729cc-b3cef870d37841a25e243a9ca6b64863\"(\n\037taskmanager.memory.process.size\022\0051728m\"\022\n\005SHELL\022\t/bin/bash\"\r\n\bweb.port\022\0010\"8\n\026HADOOP_CLIENT_CONF_DIR\022\036/etc/hadoop/conf.cloudera.yarn\"U\n\022CDH_ZOOKEEPER_HOME\022?/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/zookeeper\"T\n\rYARN_CONF_DIR\022C/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER\"-\n\037jobmanager.memory.off-heap.size\022\n134217728b\"|\n python.internal.archives-key-map\022Xpython_archive_bfe925aa9bea9ad79f3e1b595cb3e39a3acfa4656b3a5870173959195400aabb:venv.zip\"\321\001\n\bLOG_DIRS\022\304\001/yarn/container-logs/application_1637293573159_0150/container_e176_1637293573159_0150_01_000002,/hdfs/yarn-container-logs/application_1637293573159_0150/container_e176_1637293573159_0150_01_000002\"R\n NM_AUX_SERVICE_mapreduce_shuffle\022.AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=\r\n\"W\n\034CLOUDERA_POSTGRESQL_JDBC_JAR\0227/opt/cm-5.16.2/share/cmf/lib/postgresql-42.1.4.jre7.jar\"U\n\020HADOOP_YARN_HOME\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn\".\n\017PARCEL_DIRNAMES\022\033CDH-5.16.2-1.cdh5.16.2.p0.8\"a\n\025SUPERVISOR_SERVER_URL\022Hunix:///opt/cm-5.16.2/run/cloudera-scm-agent/supervisor/supervisord.sock\"\034\n\023jobmanager.rpc.port\022\00539719\"\"\n\vCMF_SBINDIR\022\023/opt/cm-5.16.2/sbin\"w\n\023WEBHCAT_DEFAULT_XML\022`/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/etc/hive-webhcat/conf.dist/webhcat-default.xml\"\"\n\017CDH_CRUNCH_HOME\022\017/usr/lib/crunch\"0\n\027SUPERVISOR_PROCESS_NAME\022\0253224-yarn-NODEMANAGER\".\n\025SUPERVISOR_GROUP_NAME\022\0253224-yarn-NODEMANAGER\"-\n\021python.executable\022\030venv.zip/venv/bin/python\"\373\004\n\020_FLINK_CLASSPATH\022\346\004:flink-python-test-1.0-SNAPSHOT.jar:lib/flink-csv-1.14.0.jar:lib/flink-json-1.14.0.jar:lib/flink-python_2.11-1.14.0.jar:lib/flink-shaded-zookeeper-3.4.14.jar:lib/flink-table_2.11-1.14.0.jar:lib/lib/flink-csv-1.14.0.jar:lib/lib/flink-json-1.14.0.jar:lib/lib/flink-shaded-zookeeper-3.4.14.jar:lib/lib/flink-table_2.11-1.14.0.jar:lib/lib/log4j-1.2-api-2.14.1.jar:lib/lib/log4j-api-2.14.1.jar:lib/lib/log4j-core-2.14.1.jar:lib/lib/log4j-slf4j-impl-2.14.1.jar:lib/log4j-1.2-api-2.14.1.jar:lib/log4j-api-2.14.1.jar:lib/log4j-core-2.14.1.jar:lib/log4j-slf4j-impl-2.14.1.jar:lib/flink-dist_2.11-1.14.0.jar:flink-conf.yaml:\")\n\021CGROUP_ROOT_BLKIO\022\024/sys/fs/cgroup/blkio\"\031\n\frest.address\022\tcdh5node1\"K\n\rCDH_SOLR_HOME\022:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/solr\"\237\v\n\tLS_COLORS\022\221\vrs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:\"T\n\rCDH_HCAT_HOME\022C/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hive-hcatalog\"9\n\037$internal.deployment.config-dir\022\026/opt/flink-1.14.0/conf\"0\n\"jobmanager.memory.jvm-overhead.max\022\n201326592b\"M\n\016CDH_OOZIE_HOME\022;/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/oozie\"O\n\017CDH_SENTRY_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/sentry\" \n\004ROOT\022\030/opt/cm-5.16.2/lib64/cmf\"\034\n\020YARN_ROOT_LOGGER\022\bINFO,RFA\"0\n&jobmanager.execution.failover-strategy\022\006region\"\031\n\aCMF_VAR\022\016/opt/cm-5.16.2\"O\n\bCONF_DIR\022C/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER\"-\n\017XFILESEARCHPATH\022\032/usr/dt/app-defaults/%L/Dt\"\037\n\026execution.runtime-mode\022\005BATCH\"\023\n\004LANG\022\vzh_CN.UTF-8\"\021\n\fCMF_SUDO_CMD\022\001 \"\024\n\aNM_HOST\022\tcdh5node1\"\205\001\n\fBOOT_LOG_DIR\022u/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69\"\031\n\aDISPLAY\022\016localhost:12.0\"Z\n\022HADOOP_LIBEXEC_DIR\022D/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec\"\221\a\n\020HADOOP_CLASSPATH\022\374\006/opt/cm-5.16.2/share/cmf/lib/plugins/event-publish-5.16.2-shaded.jar:/opt/cm-5.16.2/share/cmf/lib/plugins/tt-instrumentation-5.16.2.jar:/etc/hadoop/conf:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-yarn/.//*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*\"\032\n\026SELINUX_ROLE_REQUESTED\022\000\"R\n\026yarn.provided.lib.dirs\0228hdfs://nameservice1/user/tempodata/lixu/flink-1.14.0/lib\"\025\n\rIS_KERBERIZED\022\004true\"+\n\022CGROUP_ROOT_MEMORY\022\025/sys/fs/cgroup/memory\"$\n\fYARN_LOG_DIR\022\024/var/log/hadoop-yarn\" \n\017python.archives\022\r/opt/venv.zip\"\"\n\035taskmanager.numberOfTaskSlots\022\0011\"\024\n\020CGROUP_GROUP_CPU\022\000\".\n\vORACLE_HOME\022\037/usr/share/oracle/instantclient\"O\n\017CDH_SQOOP2_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/sqoop2\"R\n\022HADOOP_COMMON_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop\"\020\n\vCONDA_SHLVL\022\0011\"M\n\032$internal.application.main\022/com.meritdata.cloud.flink.python.test.PyUDFTest\"K\n\rCDH_HIVE_HOME\022:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hive\"P\n\fCDH_KMS_HOME\022@/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-kms\"\024\n\fNM_HTTP_PORT\022\0048042\"O\n\017CDH_IMPALA_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/impala\"*\n\016CMF_AGENT_ROOT\022\030/opt/cm-5.16.2/lib64/cmf\"$\n\rNM_LOCAL_DIRS\022\023/yarn/nm /hdfs/yarn\"\017\n\aNM_PORT\022\0048041\"\361\003\n\025YARN_NODEMANAGER_OPTS\022\327\003-Xms1073741824 -Xmx1073741824 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -Dlibrary.leveldbjni.path=/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER -Dhadoop.event.appender=,EventCatcher -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/yarn_yarn-NODEMANAGER-bef961adb705d3d90452581c3b4991c4_pid31718.hprof -XX:OnOutOfMemoryError=/opt/cm-5.16.2/lib64/cmf/service/common/killparent.sh\"\034\n\020execution.target\022\bembedded\"\021\n\004USER\022\ttempodata\"\361\b\n\tCLASSPATH\022\343\b:flink-python-test-1.0-SNAPSHOT.jar:lib/flink-csv-1.14.0.jar:lib/flink-json-1.14.0.jar:lib/flink-python_2.11-1.14.0.jar:lib/flink-shaded-zookeeper-3.4.14.jar:lib/flink-table_2.11-1.14.0.jar:lib/lib/flink-csv-1.14.0.jar:lib/lib/flink-json-1.14.0.jar:lib/lib/flink-shaded-zookeeper-3.4.14.jar:lib/lib/flink-table_2.11-1.14.0.jar:lib/lib/log4j-1.2-api-2.14.1.jar:lib/lib/log4j-api-2.14.1.jar:lib/lib/log4j-core-2.14.1.jar:lib/lib/log4j-slf4j-impl-2.14.1.jar:lib/log4j-1.2-api-2.14.1.jar:lib/log4j-api-2.14.1.jar:lib/log4j-core-2.14.1.jar:lib/log4j-slf4j-impl-2.14.1.jar:lib/flink-dist_2.11-1.14.0.jar:flink-conf.yaml::/etc/hadoop/conf.cloudera.yarn:/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-hdfs/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn/lib/*\"-\n%internal.io.tmpdirs.use-local-default\022\004true\"Y\n\016CDH_HADOOP_BIN\022G/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/bin/hadoop\"[\n\fCDH_MR1_HOME\022K/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-0.20-mapreduce\"\240\001\n\032HADOOP_TOKEN_FILE_LOCATION\022\201\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/container_e176_1637293573159_0150_01_000002/container_tokens\"\025\n\bHOSTNAME\022\tcdh5node1\"\032\n\022execution.attached\022\004true\",\n#execution.shutdown-on-attached-exit\022\005false\"\252\001\n\rpipeline.jars\022\230\001file:/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/container_e176_1637293573159_0150_01_000001/flink-python-test-1.0-SNAPSHOT.jar\"\221\001\n\017CM_STATUS_CODES\022~STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE\"R\n\rCDH_YARN_HOME\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn\"K\n\vSEARCH_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/search\"\036\n\017XDG_RUNTIME_DIR\022\v/run/user/0\"\016\n\004HOME\022\006/home/\"*\n\033jobmanager.memory.heap.size\022\v1073741824b\"\025\n\020MALLOC_ARENA_MAX\022\0014"
with 1 remaining bundle references. Taking note to clean it up during shutdown if the references are not removed by then.
2021-11-19 14:02:24,046 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory [] - Closing environment urn: "beam:env:process:v1"
payload: "\032\321\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives/venv.zip/venv/lib/python3.7/site-packages/pyflink/bin/pyflink-udf-runner.sh\"%\n\004PATH\022\035/sbin:/usr/sbin:/bin:/usr/bin\"V\n\017HADOOP_CONF_DIR\022C/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER\"\031\n\vHISTCONTROL\022\nignoredups\"\027\n\022SUPERVISOR_ENABLED\022\0011\"\031\n\021CONDA_DEFAULT_ENV\022\004base\"#\n\026jobmanager.rpc.address\022\tcdh5node1\"4\n\026KEYTRUSTEE_SERVER_HOME\022\032/usr/lib/keytrustee-server\"-\n\020CONDA_PYTHON_EXE\022\031/opt/anaconda3/bin/python\"\027\n\023CGROUP_GROUP_MEMORY\022\000\"$\n\023table.exec.timezone\022\rAsia/Shanghai\"Z\n\026CDH_HBASE_INDEXER_HOME\022@/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hbase-solr\"\036\n\fCONDA_PREFIX\022\016/opt/anaconda3\"\233\001\n\vio.tmp.dirs\022\213\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150,/hdfs/yarn/usercache/tempodata/appcache/application_1637293573159_0150\"\222\b\n\tYARN_OPTS\022\204\b-Djava.net.preferIPv4Stack=true -server -Xms1073741824 -Xmx1073741824 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -Dlibrary.leveldbjni.path=/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER -Dhadoop.event.appender=,EventCatcher -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/yarn_yarn-NODEMANAGER-bef961adb705d3d90452581c3b4991c4_pid31718.hprof -XX:OnOutOfMemoryError=/opt/cm-5.16.2/lib64/cmf/service/common/killparent.sh -Dhadoop.log.dir=/var/log/hadoop-yarn -Dyarn.log.dir=/var/log/hadoop-yarn -Dhadoop.log.file=hadoop-cmf-yarn-NODEMANAGER-cdh5node1.log.out -Dyarn.log.file=hadoop-cmf-yarn-NODEMANAGER-cdh5node1.log.out -Dyarn.home.dir=/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn -Dhadoop.home.dir=/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/lib/native\"!\n\031HADOOP_HOME_WARN_SUPPRESS\022\004true\"\017\n\aJVM_PID\022\0045524\"O\n\tJSVC_HOME\022B/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/bigtop-utils\"w\n\003PWD\022p/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/container_e176_1637293573159_0150_01_000002\"V\n\017CDH_HTTPFS_HOME\022C/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-httpfs\"P\n\016CDH_FLUME_HOME\022>/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/flume-ng\"Q\n\020CDH_PARQUET_HOME\022=/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/parquet\"\"\n\006python\022\030venv.zip/venv/bin/python\"%\n\bLESSOPEN\022\031||/usr/bin/lesspipe.sh %s\"\035\n\031SELINUX_USE_CURRENT_RANGE\022\000\"\034\n\fpython.files\022\f/opt/test.py\"O\n\017CDH_HADOOP_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop\"A\n\nweb.tmpdir\0223/tmp/flink-web-e35b2953-cfce-4655-bbac-bbacf1dbba5e\"\235\001\n\023_PYTHON_WORKING_DIR\022\205\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-archives\"\035\n\vPYTHON_HOME\022\016/opt/anaconda3\"H\n\034CLOUDERA_MYSQL_CONNECTOR_JAR\022(/usr/share/java/mysql-connector-java.jar\"I\n\fCDH_HUE_HOME\0229/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hue\" \n\034NM_AUX_SERVICE_spark_shuffle\022\000\"1\n __table.query-start.local-time__\022\r1637330289401\"8\n\020HIVE_DEFAULT_XML\022$/etc/hive/conf.dist/hive-default.xml\"\031\n\025CONDA_PROMPT_MODIFIER\022\000\"\n\n\005SHLVL\022\0014\"I\n\036$internal.yarn.log-config-file\022\'/opt/flink-1.14.0/conf/log4j.properties\"u\n\035python.internal.files-key-map\022Tpython_file_13c795ab581a80efa6a9947d919503287f51d055172132c4457dc7e58ad114cc:test.py\"3\n\017CMF_PACKAGE_DIR\022 /opt/cm-5.16.2/lib64/cmf/service\"-\n\017CGROUP_ROOT_CPU\022\032/sys/fs/cgroup/cpu,cpuacct\"%\n\tCONDA_EXE\022\030/opt/anaconda3/bin/conda\"\020\n\bHISTSIZE\022\0041000\"\036\n\tJAVA_HOME\022\021/usr/java/default\"5\n\fCMF_CONF_DIR\022%/opt/cm-5.16.2/etc/cloudera-scm-agent\"0\n\"jobmanager.memory.jvm-overhead.min\022\n201326592b\"\r\n\004TERM\022\005xterm\"R\n\rCDH_HDFS_HOME\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-hdfs\"\023\n\016XDG_SESSION_ID\022\0017\"\033\n\027SELINUX_LEVEL_REQUESTED\022\000\"=\n\fYARN_LOGFILE\022-hadoop-cmf-yarn-NODEMANAGER-cdh5node1.log.out\"+\n\020HADOOP_USER_NAME\022\027tempodata@MERITDATA.COM\"3\n*execution.savepoint.ignore-unclaimed-state\022\005false\"X\n\023SCM_DEFINES_SCRIPTS\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/meta/cdh_env.sh\"(\n\033_PYTHON_WORKER_MEMORY_LIMIT\022\t536870920\"U\n\020HADOOP_HDFS_HOME\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-hdfs\"\\\n\022HADOOP_MAPRED_HOME\022F/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-mapreduce\"1\n\023CGROUP_ROOT_CPUACCT\022\032/sys/fs/cgroup/cpu,cpuacct\"/\n\023CMF_AGENT_MGMT_HOME\022\030/opt/cm-5.16.2/share/cmf\"\037\n\001_\022\032/usr/java/default/bin/java\"%\n\tMGMT_HOME\022\030/opt/cm-5.16.2/share/cmf\"\026\n\022CGROUP_GROUP_BLKIO\022\000\"\025\n\aSSH_TTY\022\n/dev/pts/2\"#\n\nSSH_CLIENT\022\025191.168.1.34 61192 22\" \n\030PYFLINK_GATEWAY_DISABLED\022\004true\"K\n\rCDH_AVRO_HOME\022:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/avro\" \n\016CDH_SQOOP_HOME\022\016/usr/lib/sqoop\"T\n\024CDH_HUE_PLUGINS_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop\"\'\n\036jobmanager.memory.process.size\022\0051600m\"\020\n\vCDH_VERSION\022\0015\"!\n\035HADOOP_JOB_HISTORYSERVER_OPTS\022\000\" \n\016CDH_KAFKA_HOME\022\016/usr/lib/kafka\"J\n\035CLOUDERA_ORACLE_CONNECTOR_JAR\022)/usr/share/java/oracle-connector-java.jar\"4\n\016SSH_CONNECTION\022\"191.168.1.34 61192 191.168.1.24 22\")\n\037internal.cluster.execution-mode\022\006NORMAL\"(\n\aNLSPATH\022\035/usr/dt/lib/nls/msg/%L/%N.cat\"7\n\022KEYTRUSTEE_KP_HOME\022!/usr/share/keytrustee-keyprovider\"+\n\017CMF_SERVER_ROOT\022\030/opt/cm-5.16.2/share/cmf\"\035\n\aCMF_ETC\022\022/opt/cm-5.16.2/etc\"2\n$jobmanager.memory.jvm-metaspace.size\022\n268435456b\"M\n\016CDH_SPARK_HOME\022;/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/spark\"\030\n\024CGROUP_GROUP_CPUACCT\022\000\";\n\fCONTAINER_ID\022+container_e176_1637293573159_0150_01_000002\"M\n\016CDH_LLAMA_HOME\022;/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/llama\"t\n\rXDG_DATA_DIRS\022c/root/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share\"K\n\rCDH_KUDU_HOME\022:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/kudu\"%\n\fPARCELS_ROOT\022\025/opt/cloudera/parcels\">\n\034high-availability.cluster-id\022\036application_1637293573159_0150\"\035\n\031YARN_RESOURCEMANAGER_OPTS\022\000\"4\n\030python.client.executable\022\030venv.zip/venv/bin/python\"\026\n\rtable.planner\022\005BLINK\"\034\n\004MAIL\022\024/var/spool/mail/root\"1\n __table.query-start.epoch-time__\022\r1637301489401\"\030\n\023parallelism.default\022\0011\"\027\n\023pipeline.classpaths\022\000\"M\n\016CDH_HBASE_HOME\022;/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hbase\"\024\n\aLOGNAME\022\ttempodata\"V\n\fCDH_MR2_HOME\022F/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-mapreduce\"R\n\vTOMCAT_HOME\022C/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/bigtop-tomcat\"\033\n\016_FLINK_NODE_ID\022\tcdh5node1\"I\n\fCDH_PIG_HOME\0229/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/pig\"\034\n\026yarn.application.queue\022\002df\"M\n\rHADOOP_PREFIX\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop\"\232\001\n\nLOCAL_DIRS\022\213\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150,/hdfs/yarn/usercache/tempodata/appcache/application_1637293573159_0150\"\342\001\n\nPYTHONPATH\022\323\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69/python-files/blob_p-30842d5cb48fe53fd9f94c2a4e364225246729cc-b3cef870d37841a25e243a9ca6b64863\"(\n\037taskmanager.memory.process.size\022\0051728m\"\022\n\005SHELL\022\t/bin/bash\"\r\n\bweb.port\022\0010\"8\n\026HADOOP_CLIENT_CONF_DIR\022\036/etc/hadoop/conf.cloudera.yarn\"U\n\022CDH_ZOOKEEPER_HOME\022?/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/zookeeper\"T\n\rYARN_CONF_DIR\022C/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER\"-\n\037jobmanager.memory.off-heap.size\022\n134217728b\"|\n python.internal.archives-key-map\022Xpython_archive_bfe925aa9bea9ad79f3e1b595cb3e39a3acfa4656b3a5870173959195400aabb:venv.zip\"\321\001\n\bLOG_DIRS\022\304\001/yarn/container-logs/application_1637293573159_0150/container_e176_1637293573159_0150_01_000002,/hdfs/yarn-container-logs/application_1637293573159_0150/container_e176_1637293573159_0150_01_000002\"R\n NM_AUX_SERVICE_mapreduce_shuffle\022.AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=\r\n\"W\n\034CLOUDERA_POSTGRESQL_JDBC_JAR\0227/opt/cm-5.16.2/share/cmf/lib/postgresql-42.1.4.jre7.jar\"U\n\020HADOOP_YARN_HOME\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn\".\n\017PARCEL_DIRNAMES\022\033CDH-5.16.2-1.cdh5.16.2.p0.8\"a\n\025SUPERVISOR_SERVER_URL\022Hunix:///opt/cm-5.16.2/run/cloudera-scm-agent/supervisor/supervisord.sock\"\034\n\023jobmanager.rpc.port\022\00539719\"\"\n\vCMF_SBINDIR\022\023/opt/cm-5.16.2/sbin\"w\n\023WEBHCAT_DEFAULT_XML\022`/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/etc/hive-webhcat/conf.dist/webhcat-default.xml\"\"\n\017CDH_CRUNCH_HOME\022\017/usr/lib/crunch\"0\n\027SUPERVISOR_PROCESS_NAME\022\0253224-yarn-NODEMANAGER\".\n\025SUPERVISOR_GROUP_NAME\022\0253224-yarn-NODEMANAGER\"-\n\021python.executable\022\030venv.zip/venv/bin/python\"\373\004\n\020_FLINK_CLASSPATH\022\346\004:flink-python-test-1.0-SNAPSHOT.jar:lib/flink-csv-1.14.0.jar:lib/flink-json-1.14.0.jar:lib/flink-python_2.11-1.14.0.jar:lib/flink-shaded-zookeeper-3.4.14.jar:lib/flink-table_2.11-1.14.0.jar:lib/lib/flink-csv-1.14.0.jar:lib/lib/flink-json-1.14.0.jar:lib/lib/flink-shaded-zookeeper-3.4.14.jar:lib/lib/flink-table_2.11-1.14.0.jar:lib/lib/log4j-1.2-api-2.14.1.jar:lib/lib/log4j-api-2.14.1.jar:lib/lib/log4j-core-2.14.1.jar:lib/lib/log4j-slf4j-impl-2.14.1.jar:lib/log4j-1.2-api-2.14.1.jar:lib/log4j-api-2.14.1.jar:lib/log4j-core-2.14.1.jar:lib/log4j-slf4j-impl-2.14.1.jar:lib/flink-dist_2.11-1.14.0.jar:flink-conf.yaml:\")\n\021CGROUP_ROOT_BLKIO\022\024/sys/fs/cgroup/blkio\"\031\n\frest.address\022\tcdh5node1\"K\n\rCDH_SOLR_HOME\022:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/solr\"\237\v\n\tLS_COLORS\022\221\vrs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:\"T\n\rCDH_HCAT_HOME\022C/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hive-hcatalog\"9\n\037$internal.deployment.config-dir\022\026/opt/flink-1.14.0/conf\"0\n\"jobmanager.memory.jvm-overhead.max\022\n201326592b\"M\n\016CDH_OOZIE_HOME\022;/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/oozie\"O\n\017CDH_SENTRY_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/sentry\" \n\004ROOT\022\030/opt/cm-5.16.2/lib64/cmf\"\034\n\020YARN_ROOT_LOGGER\022\bINFO,RFA\"0\n&jobmanager.execution.failover-strategy\022\006region\"\031\n\aCMF_VAR\022\016/opt/cm-5.16.2\"O\n\bCONF_DIR\022C/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER\"-\n\017XFILESEARCHPATH\022\032/usr/dt/app-defaults/%L/Dt\"\037\n\026execution.runtime-mode\022\005BATCH\"\023\n\004LANG\022\vzh_CN.UTF-8\"\021\n\fCMF_SUDO_CMD\022\001 \"\024\n\aNM_HOST\022\tcdh5node1\"\205\001\n\fBOOT_LOG_DIR\022u/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/python-dist-6a5f3e17-4ecf-4782-b9ac-08888e944d69\"\031\n\aDISPLAY\022\016localhost:12.0\"Z\n\022HADOOP_LIBEXEC_DIR\022D/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec\"\221\a\n\020HADOOP_CLASSPATH\022\374\006/opt/cm-5.16.2/share/cmf/lib/plugins/event-publish-5.16.2-shaded.jar:/opt/cm-5.16.2/share/cmf/lib/plugins/tt-instrumentation-5.16.2.jar:/etc/hadoop/conf:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-yarn/.//*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*\"\032\n\026SELINUX_ROLE_REQUESTED\022\000\"R\n\026yarn.provided.lib.dirs\0228hdfs://nameservice1/user/tempodata/lixu/flink-1.14.0/lib\"\025\n\rIS_KERBERIZED\022\004true\"+\n\022CGROUP_ROOT_MEMORY\022\025/sys/fs/cgroup/memory\"$\n\fYARN_LOG_DIR\022\024/var/log/hadoop-yarn\" \n\017python.archives\022\r/opt/venv.zip\"\"\n\035taskmanager.numberOfTaskSlots\022\0011\"\024\n\020CGROUP_GROUP_CPU\022\000\".\n\vORACLE_HOME\022\037/usr/share/oracle/instantclient\"O\n\017CDH_SQOOP2_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/sqoop2\"R\n\022HADOOP_COMMON_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop\"\020\n\vCONDA_SHLVL\022\0011\"M\n\032$internal.application.main\022/com.meritdata.cloud.flink.python.test.PyUDFTest\"K\n\rCDH_HIVE_HOME\022:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hive\"P\n\fCDH_KMS_HOME\022@/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-kms\"\024\n\fNM_HTTP_PORT\022\0048042\"O\n\017CDH_IMPALA_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/impala\"*\n\016CMF_AGENT_ROOT\022\030/opt/cm-5.16.2/lib64/cmf\"$\n\rNM_LOCAL_DIRS\022\023/yarn/nm /hdfs/yarn\"\017\n\aNM_PORT\022\0048041\"\361\003\n\025YARN_NODEMANAGER_OPTS\022\327\003-Xms1073741824 -Xmx1073741824 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -Dlibrary.leveldbjni.path=/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER -Dhadoop.event.appender=,EventCatcher -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/yarn_yarn-NODEMANAGER-bef961adb705d3d90452581c3b4991c4_pid31718.hprof -XX:OnOutOfMemoryError=/opt/cm-5.16.2/lib64/cmf/service/common/killparent.sh\"\034\n\020execution.target\022\bembedded\"\021\n\004USER\022\ttempodata\"\361\b\n\tCLASSPATH\022\343\b:flink-python-test-1.0-SNAPSHOT.jar:lib/flink-csv-1.14.0.jar:lib/flink-json-1.14.0.jar:lib/flink-python_2.11-1.14.0.jar:lib/flink-shaded-zookeeper-3.4.14.jar:lib/flink-table_2.11-1.14.0.jar:lib/lib/flink-csv-1.14.0.jar:lib/lib/flink-json-1.14.0.jar:lib/lib/flink-shaded-zookeeper-3.4.14.jar:lib/lib/flink-table_2.11-1.14.0.jar:lib/lib/log4j-1.2-api-2.14.1.jar:lib/lib/log4j-api-2.14.1.jar:lib/lib/log4j-core-2.14.1.jar:lib/lib/log4j-slf4j-impl-2.14.1.jar:lib/log4j-1.2-api-2.14.1.jar:lib/log4j-api-2.14.1.jar:lib/log4j-core-2.14.1.jar:lib/log4j-slf4j-impl-2.14.1.jar:lib/flink-dist_2.11-1.14.0.jar:flink-conf.yaml::/etc/hadoop/conf.cloudera.yarn:/opt/cm-5.16.2/run/cloudera-scm-agent/process/3224-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-hdfs/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn/lib/*\"-\n%internal.io.tmpdirs.use-local-default\022\004true\"Y\n\016CDH_HADOOP_BIN\022G/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/bin/hadoop\"[\n\fCDH_MR1_HOME\022K/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-0.20-mapreduce\"\240\001\n\032HADOOP_TOKEN_FILE_LOCATION\022\201\001/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/container_e176_1637293573159_0150_01_000002/container_tokens\"\025\n\bHOSTNAME\022\tcdh5node1\"\032\n\022execution.attached\022\004true\",\n#execution.shutdown-on-attached-exit\022\005false\"\252\001\n\rpipeline.jars\022\230\001file:/yarn/nm/usercache/tempodata/appcache/application_1637293573159_0150/container_e176_1637293573159_0150_01_000001/flink-python-test-1.0-SNAPSHOT.jar\"\221\001\n\017CM_STATUS_CODES\022~STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE\"R\n\rCDH_YARN_HOME\022A/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn\"K\n\vSEARCH_HOME\022</opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/search\"\036\n\017XDG_RUNTIME_DIR\022\v/run/user/0\"\016\n\004HOME\022\006/home/\"*\n\033jobmanager.memory.heap.size\022\v1073741824b\"\025\n\020MALLOC_ARENA_MAX\022\0014"
2021-11-19 14:02:24,050 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService [] - 1 Beam Fn Logging clients still connected during shutdown.
2021-11-19 14:02:25,332 WARN org.apache.flink.runtime.taskmanager.Task [] - Source: Values(tuples=[[{ _UTF-16LE'1' }, { _UTF-16LE'2' }, { _UTF-16LE'3' }]], values=[f0]) -> Calc(select=[f0]) -> PythonCalc(select=[func1(f0) AS a]) -> Sink: Collect table sink (1/1)#0 (bef195197949f0a6c3716ff2e6e8b17d) switched from RUNNING to FAILED with failure cause: java.lang.RuntimeException: Failed to start remote bundle
at org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.startBundle(BeamPythonFunctionRunner.java:333)
at org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.checkInvokeStartBundle(BeamPythonFunctionRunner.java:314)
at org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.process(BeamPythonFunctionRunner.java:293)
at org.apache.flink.table.runtime.operators.python.scalar.PythonScalarFunctionOperator.processElementInternal(PythonScalarFunctionOperator.java:89)
at org.apache.flink.table.runtime.operators.python.scalar.PythonScalarFunctionOperator.processElementInternal(PythonScalarFunctionOperator.java:38)
at org.apache.flink.table.runtime.operators.python.AbstractStatelessFunctionOperator.processElement(AbstractStatelessFunctionOperator.java:117)
at org.apache.flink.streaming.runtime.tasks.ChainingOutput.pushToOperator(ChainingOutput.java:99)
at org.apache.flink.streaming.runtime.tasks.ChainingOutput.collect(ChainingOutput.java:80)
at org.apache.flink.streaming.runtime.tasks.ChainingOutput.collect(ChainingOutput.java:39)
at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56)
at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:29)
at BatchExecCalc$8.processElement(Unknown Source)
at org.apache.flink.streaming.runtime.tasks.ChainingOutput.pushToOperator(ChainingOutput.java:99)
at org.apache.flink.streaming.runtime.tasks.ChainingOutput.collect(ChainingOutput.java:80)
at org.apache.flink.streaming.runtime.tasks.ChainingOutput.collect(ChainingOutput.java:39)
at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56)
at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:29)
at org.apache.flink.streaming.api.operators.StreamSourceContexts$ManualWatermarkContext.processAndCollect(StreamSourceContexts.java:418)
at org.apache.flink.streaming.api.operators.StreamSourceContexts$WatermarkContext.collect(StreamSourceContexts.java:513)
at org.apache.flink.streaming.api.operators.StreamSourceContexts$SwitchingOnClose.collect(StreamSourceContexts.java:103)
at org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:92)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:116)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:73)
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:323)
Caused by: java.lang.RuntimeException: No client connected within timeout
at org.apache.beam.runners.fnexecution.data.GrpcDataService.send(GrpcDataService.java:192)
at org.apache.beam.runners.fnexecution.control.SdkHarnessClient$BundleProcessor.newBundle(SdkHarnessClient.java:287)
at org.apache.beam.runners.fnexecution.control.SdkHarnessClient$BundleProcessor.newBundle(SdkHarnessClient.java:197)
at org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$SimpleStageBundleFactory.getBundle(DefaultJobBundleFactory.java:519)
at org.apache.beam.runners.fnexecution.control.StageBundleFactory.getBundle(StageBundleFactory.java:87)
at org.apache.beam.runners.fnexecution.control.StageBundleFactory.getBundle(StageBundleFactory.java:76)
at org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.startBundle(BeamPythonFunctionRunner.java:323)
... 23 more
Caused by: java.util.concurrent.TimeoutException: Waited 3 minutes for org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.SettableFuture@533506fe[status=PENDING]
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:471)
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:90)
at org.apache.beam.runners.fnexecution.data.GrpcDataService.send(GrpcDataService.java:187)
... 29 more
------------------ 原始邮件 ------------------
发件人: "user-zh" <dian0511.fu@gmail.com>;
发送时间: 2021年11月19日(星期五) 中午11:03
收件人: "user-zh"<user-zh@flink.apache.org>;
主题: Re: 哪里可以下载到downloads/setup-pyflink-virtual-env.sh脚本
刚注意到你用的YARN application模式，PyFlink 1.14.0才支持YARN application模式，主要是新增了命令行选项“
-pyclientexec” 和配置“python.client.executable”：
https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/python/python_config/#python-client-executable
对于你这个作业来说，你需要通过使用1.14.0版本，同时添加命令行选项：-pyclientexec venv.zip/venv/bin/python
On Fri, Nov 19, 2021 at 10:48 AM Asahi Lee <978466273@qq.com.invalid> wrote:
> 我通过source my_env/bin/activate和指定环境变量PYFLINK_CLIENT_EXECUTABLE均未提交成功，错误未变化；
> 我的错误中，jobmanager的日志中显示No module named pyflink，而jobmanager是运行在yarn集群上；
> 请问，还有还有什么未配置？
>
>
> &gt; LogType:jobmanager.out
> &gt; Log Upload Time:星期四 十一月 18 20:48:45 +0800 2021
> &gt; LogLength:37
> &gt; Log Contents:
> &gt; /bin/python: No module named pyflink
>
>
>
>
> ------------------&nbsp;原始邮件&nbsp;------------------
> 发件人:
> "user-zh"
> <
> dian0511.fu@gmail.com&gt;;
> 发送时间:&nbsp;2021年11月19日(星期五) 上午9:38
> 收件人:&nbsp;"user-zh"<user-zh@flink.apache.org&gt;;
>
> 主题:&nbsp;Re: 哪里可以下载到downloads/setup-pyflink-virtual-env.sh脚本
>
>
>
> -pyexec 指定的是集群端所用的Python环境，客户端需要编译Flink作业，也会依赖Python环境。可以看一下这个文档:
>
> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/dev/python/dependency_management/#python-interpreter-of-client
>
> On Thu, Nov 18, 2021 at 9:00 PM Asahi Lee <978466273@qq.com.invalid&gt;
> wrote:
>
> &gt; Hi !
> &gt; &amp;nbsp; &amp;nbsp;我在java Table api中使用python udf
> &gt; 函数，通过下面的命令提交应用，报无法启动python服务错误，请问我的提交方式对吗？jm日志为/bin/python: No module
> named
> &gt; pyflink。
> &gt;
> &gt;
> &gt; ./flink-1.13.2/bin/flink&amp;nbsp;
> &gt; run-application -t yarn-application&amp;nbsp;
> &gt;
> -Dyarn.provided.lib.dirs="hdfs://nameservice1/user/flink/flinklib"&amp;nbsp;
> &gt; -Dyarn.application.queue=d
> &gt; -p 1&amp;nbsp;
> &gt; -pyarch /opt/venv.zip
> &gt; -pyexec venv.zip/venv/bin/python&amp;nbsp;
> &gt; -pyfs /opt/test.py&amp;nbsp;
> &gt; -c test.PyUDFTest&amp;nbsp;
> &gt; /opt/flink-python-test-1.0-SNAPSHOT.jar
> &gt;
> &gt;
> &gt;
> &gt; 错误：
> &gt; Caused by: java.lang.RuntimeException: Python callback server start
> failed!
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.flink.client.python.PythonFunctionFactory.createPythonFunctionFactory(PythonFunctionFactory.java:167)
> &gt; ~[flink-python_2.11-1.13.2.jar:1.13.2]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.flink.client.python.PythonFunctionFactory$1.load(PythonFunctionFactory.java:88)
> &gt; ~[flink-python_2.11-1.13.2.jar:1.13.2]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.flink.client.python.PythonFunctionFactory$1.load(PythonFunctionFactory.java:84)
> &gt; ~[flink-python_2.11-1.13.2.jar:1.13.2]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
> &gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
> &gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
> &gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
> &gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> &gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
> &gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
> &gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.flink.client.python.PythonFunctionFactory.getPythonFunction(PythonFunctionFactory.java:129)
> &gt; ~[flink-python_2.11-1.13.2.jar:1.13.2]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt; sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_111]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> &gt; ~[?:1.8.0_111]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> &gt; ~[?:1.8.0_111]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt; java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.flink.table.functions.python.utils.PythonFunctionUtils.getPythonFunction(PythonFunctionUtils.java:45)
> &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.flink.table.functions.UserDefinedFunctionHelper.instantiateFunction(UserDefinedFunctionHelper.java:206)
> &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.flink.table.catalog.FunctionCatalog.getFunctionDefinition(FunctionCatalog.java:659)
> &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.flink.table.catalog.FunctionCatalog.resolveAmbiguousFunctionReference(FunctionCatalog.java:606)
> &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.flink.table.catalog.FunctionCatalog.lookupFunction(FunctionCatalog.java:362)
> &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.flink.table.planner.catalog.FunctionCatalogOperatorTable.lookupOperatorOverloads(FunctionCatalogOperatorTable.java:97)
> &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.calcite.sql.util.ChainedSqlOperatorTable.lookupOperatorOverloads(ChainedSqlOperatorTable.java:67)
> &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1183)
> &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1169)
> &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1200)
> &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1169)
> &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:945)
> &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:704)
> &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt; org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org
> $apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:152)
> &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:110)
> &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:201)
> &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:101)
> &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:704)
> &gt; ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> test.PyUDFTest.main(PyUDFTest.java:22)
> &gt; ~[flink-python-test-1.0-SNAPSHOT.jar:?]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt; sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_111]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> &gt; ~[?:1.8.0_111]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> &gt; ~[?:1.8.0_111]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt; java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
> &gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
> &gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> &gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; at
> &gt;
> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:242)
> &gt; ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; ... 10 more
> &gt; 2021-11-18 20:48:44,458 INFO&amp;nbsp;
> &gt; org.apache.flink.runtime.blob.BlobServer&amp;nbsp; &amp;nbsp;
> &amp;nbsp; &amp;nbsp; &amp;nbsp;
> &gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;[] -
> Stopped BLOB server at
> &gt; 0.0.0.0:45070
> &gt; 2021-11-18 20:48:44,458 INFO&amp;nbsp;
> &gt; org.apache.flink.runtime.entrypoint.ClusterEntrypoint&amp;nbsp;
> &amp;nbsp; &amp;nbsp;
> &gt; &amp;nbsp; [] - Shutting YarnApplicationClusterEntryPoint down with
> application
> &gt; status UNKNOWN. Diagnostics Cluster entrypoint has been closed
> externally..
> &gt; 2021-11-18 20:48:44,458 INFO&amp;nbsp;
> &gt; org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] -
> Shutting
> &gt; down rest endpoint.
> &gt; 2021-11-18 20:48:44,474 INFO&amp;nbsp;
> &gt; org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] -
> Removing
> &gt; cache directory
> &gt; /tmp/flink-web-dd82c3c0-f457-492d-8e64-5ae74fe9abbd/flink-web-ui
> &gt; 2021-11-18 20:48:44,474 INFO&amp;nbsp;
> &gt; org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] -
> &gt; http://cdh5node3:40216 lost leadership
> &gt; 2021-11-18 20:48:44,474 INFO&amp;nbsp;
> &gt; org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] -
> Shut
> &gt; down complete.
> &gt; 2021-11-18 20:48:44,474 INFO&amp;nbsp;
> &gt;
> org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent
> &gt; [] - Closing components.
> &gt; 2021-11-18 20:48:44,475 INFO&amp;nbsp;
> &gt;
> org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess
> &gt; [] - Stopping SessionDispatcherLeaderProcess.
> &gt; 2021-11-18 20:48:44,475 INFO&amp;nbsp;
> &gt; org.apache.flink.runtime.dispatcher.StandaloneDispatcher&amp;nbsp;
> &amp;nbsp;
> &gt; &amp;nbsp;[] - Stopping dispatcher akka.tcp://flink@cdh5node3
> &gt; :34697/user/rpc/dispatcher_1.
> &gt; 2021-11-18 20:48:44,476 INFO&amp;nbsp;
> &gt; org.apache.flink.runtime.dispatcher.StandaloneDispatcher&amp;nbsp;
> &amp;nbsp;
> &gt; &amp;nbsp;[] - Stopping all currently running jobs of dispatcher
> &gt; akka.tcp://flink@cdh5node3:34697/user/rpc/dispatcher_1.
> &gt;
> &gt;
> &gt; LogType:jobmanager.out
> &gt; Log Upload Time:星期四 十一月 18 20:48:45 +0800 2021
> &gt; LogLength:37
> &gt; Log Contents:
> &gt; /bin/python: No module named pyflink
> &gt;
> &gt;
> &gt;
> &gt;
> &gt;
> &gt;
> &gt;
> &gt;
> &gt;
> &gt;
> &gt;
> &gt;
> &gt;
> &gt; ------------------&amp;nbsp;原始邮件&amp;nbsp;------------------
> &gt; 发件人:
> &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> "user-zh"
> &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
> <
> &gt; xyzhong131@163.com&amp;gt;;
> &gt; 发送时间:&amp;nbsp;2021年11月18日(星期四) 下午3:34
> &gt; 收件人:&amp;nbsp;"user-zh"<user-zh@flink.apache.org&amp;gt;;
> &gt;
> &gt; 主题:&amp;nbsp;Re:哪里可以下载到downloads/setup-pyflink-virtual-env.sh脚本
> &gt;
> &gt;
> &gt;
> &gt; Hi!<br/&amp;gt;我在下面的旧版本中找到了，你试下能不能用：<br/&amp;gt;
> &gt;
> https://nightlies.apache.org/flink/flink-docs-release-1.12/downloads/setup-pyflink-virtual-env.sh
> &gt
> <https://nightlies.apache.org/flink/flink-docs-release-1.12/downloads/setup-pyflink-virtual-env.sh&gt>;
> 在 2021-11-18 15:05:03，"Asahi Lee" <978466273@qq.com.INVALID&amp;gt; 写道：
> &gt; &amp;gt;Hi!
> &gt; &amp;gt;&amp;amp;nbsp; &amp;amp;nbsp;
> &gt;
> 我在flink官方文档中看到通过&amp;amp;nbsp;&amp;amp;nbsp;setup-pyflink-virtual-env.sh
> &gt; 脚本制作python虚拟环境，请问再哪里可以下载到？文档地址如下：
> &gt; &amp;gt;
> &gt;
> https://nightlies.apache.org/flink/flink-docs-release-1.14/zh/docs/dev/python/faq/
Re: 哪里可以下载到downloads/setup-pyflink-virtual-env.sh脚本
Posted by Dian Fu <di...@gmail.com>.
刚注意到你用的YARN application模式，PyFlink 1.14.0才支持YARN application模式，主要是新增了命令行选项“
-pyclientexec” 和配置“python.client.executable”：
https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/python/python_config/#python-client-executable
对于你这个作业来说，你需要通过使用1.14.0版本，同时添加命令行选项：-pyclientexec venv.zip/venv/bin/python
On Fri, Nov 19, 2021 at 10:48 AM Asahi Lee <97...@qq.com.invalid> wrote:
> 我通过source my_env/bin/activate和指定环境变量PYFLINK_CLIENT_EXECUTABLE均未提交成功，错误未变化；
> 我的错误中，jobmanager的日志中显示No module named pyflink，而jobmanager是运行在yarn集群上；
> 请问，还有还有什么未配置？
>
>
> > LogType:jobmanager.out
> > Log Upload Time:星期四 十一月 18 20:48:45 +0800 2021
> > LogLength:37
> > Log Contents:
> > /bin/python: No module named pyflink
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人:
> "user-zh"
> <
> dian0511.fu@gmail.com>;
> 发送时间: 2021年11月19日(星期五) 上午9:38
> 收件人: "user-zh"<user-zh@flink.apache.org>;
>
> 主题: Re: 哪里可以下载到downloads/setup-pyflink-virtual-env.sh脚本
>
>
>
> -pyexec 指定的是集群端所用的Python环境，客户端需要编译Flink作业，也会依赖Python环境。可以看一下这个文档:
>
> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/dev/python/dependency_management/#python-interpreter-of-client
>
> On Thu, Nov 18, 2021 at 9:00 PM Asahi Lee <978466273@qq.com.invalid>
> wrote:
>
> > Hi !
> > &nbsp; &nbsp;我在java Table api中使用python udf
> > 函数，通过下面的命令提交应用，报无法启动python服务错误，请问我的提交方式对吗？jm日志为/bin/python: No module
> named
> > pyflink。
> >
> >
> > ./flink-1.13.2/bin/flink&nbsp;
> > run-application -t yarn-application&nbsp;
> >
> -Dyarn.provided.lib.dirs="hdfs://nameservice1/user/flink/flinklib"&nbsp;
> > -Dyarn.application.queue=d
> > -p 1&nbsp;
> > -pyarch /opt/venv.zip
> > -pyexec venv.zip/venv/bin/python&nbsp;
> > -pyfs /opt/test.py&nbsp;
> > -c test.PyUDFTest&nbsp;
> > /opt/flink-python-test-1.0-SNAPSHOT.jar
> >
> >
> >
> > 错误：
> > Caused by: java.lang.RuntimeException: Python callback server start
> failed!
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.flink.client.python.PythonFunctionFactory.createPythonFunctionFactory(PythonFunctionFactory.java:167)
> > ~[flink-python_2.11-1.13.2.jar:1.13.2]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.flink.client.python.PythonFunctionFactory$1.load(PythonFunctionFactory.java:88)
> > ~[flink-python_2.11-1.13.2.jar:1.13.2]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.flink.client.python.PythonFunctionFactory$1.load(PythonFunctionFactory.java:84)
> > ~[flink-python_2.11-1.13.2.jar:1.13.2]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
> > ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
> > ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
> > ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
> > ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> > ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
> > ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
> > ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.flink.client.python.PythonFunctionFactory.getPythonFunction(PythonFunctionFactory.java:129)
> > ~[flink-python_2.11-1.13.2.jar:1.13.2]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_111]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > ~[?:1.8.0_111]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > ~[?:1.8.0_111]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> > java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.flink.table.functions.python.utils.PythonFunctionUtils.getPythonFunction(PythonFunctionUtils.java:45)
> > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.flink.table.functions.UserDefinedFunctionHelper.instantiateFunction(UserDefinedFunctionHelper.java:206)
> > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.flink.table.catalog.FunctionCatalog.getFunctionDefinition(FunctionCatalog.java:659)
> > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.flink.table.catalog.FunctionCatalog.resolveAmbiguousFunctionReference(FunctionCatalog.java:606)
> > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.flink.table.catalog.FunctionCatalog.lookupFunction(FunctionCatalog.java:362)
> > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.flink.table.planner.catalog.FunctionCatalogOperatorTable.lookupOperatorOverloads(FunctionCatalogOperatorTable.java:97)
> > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.calcite.sql.util.ChainedSqlOperatorTable.lookupOperatorOverloads(ChainedSqlOperatorTable.java:67)
> > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1183)
> > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1169)
> > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1200)
> > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1169)
> > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:945)
> > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:704)
> > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> > org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org
> $apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:152)
> > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:110)
> > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:201)
> > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:101)
> > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:704)
> > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> test.PyUDFTest.main(PyUDFTest.java:22)
> > ~[flink-python-test-1.0-SNAPSHOT.jar:?]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_111]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > ~[?:1.8.0_111]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > ~[?:1.8.0_111]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> > java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
> > ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
> > ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> > ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; at
> >
> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:242)
> > ~[flink-dist_2.11-1.13.1.jar:1.13.1]
> > &nbsp; &nbsp; &nbsp; &nbsp; ... 10 more
> > 2021-11-18 20:48:44,458 INFO&nbsp;
> > org.apache.flink.runtime.blob.BlobServer&nbsp; &nbsp;
> &nbsp; &nbsp; &nbsp;
> > &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;[] -
> Stopped BLOB server at
> > 0.0.0.0:45070
> > 2021-11-18 20:48:44,458 INFO&nbsp;
> > org.apache.flink.runtime.entrypoint.ClusterEntrypoint&nbsp;
> &nbsp; &nbsp;
> > &nbsp; [] - Shutting YarnApplicationClusterEntryPoint down with
> application
> > status UNKNOWN. Diagnostics Cluster entrypoint has been closed
> externally..
> > 2021-11-18 20:48:44,458 INFO&nbsp;
> > org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] -
> Shutting
> > down rest endpoint.
> > 2021-11-18 20:48:44,474 INFO&nbsp;
> > org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] -
> Removing
> > cache directory
> > /tmp/flink-web-dd82c3c0-f457-492d-8e64-5ae74fe9abbd/flink-web-ui
> > 2021-11-18 20:48:44,474 INFO&nbsp;
> > org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] -
> > http://cdh5node3:40216 lost leadership
> > 2021-11-18 20:48:44,474 INFO&nbsp;
> > org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] -
> Shut
> > down complete.
> > 2021-11-18 20:48:44,474 INFO&nbsp;
> >
> org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent
> > [] - Closing components.
> > 2021-11-18 20:48:44,475 INFO&nbsp;
> >
> org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess
> > [] - Stopping SessionDispatcherLeaderProcess.
> > 2021-11-18 20:48:44,475 INFO&nbsp;
> > org.apache.flink.runtime.dispatcher.StandaloneDispatcher&nbsp;
> &nbsp;
> > &nbsp;[] - Stopping dispatcher akka.tcp://flink@cdh5node3
> > :34697/user/rpc/dispatcher_1.
> > 2021-11-18 20:48:44,476 INFO&nbsp;
> > org.apache.flink.runtime.dispatcher.StandaloneDispatcher&nbsp;
> &nbsp;
> > &nbsp;[] - Stopping all currently running jobs of dispatcher
> > akka.tcp://flink@cdh5node3:34697/user/rpc/dispatcher_1.
> >
> >
> > LogType:jobmanager.out
> > Log Upload Time:星期四 十一月 18 20:48:45 +0800 2021
> > LogLength:37
> > Log Contents:
> > /bin/python: No module named pyflink
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > ------------------&nbsp;原始邮件&nbsp;------------------
> > 发件人:
> >
> "user-zh"
> >
> <
> > xyzhong131@163.com&gt;;
> > 发送时间:&nbsp;2021年11月18日(星期四) 下午3:34
> > 收件人:&nbsp;"user-zh"<user-zh@flink.apache.org&gt;;
> >
> > 主题:&nbsp;Re:哪里可以下载到downloads/setup-pyflink-virtual-env.sh脚本
> >
> >
> >
> > Hi!<br/&gt;我在下面的旧版本中找到了，你试下能不能用：<br/&gt;
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.12/downloads/setup-pyflink-virtual-env.sh
> >
> <https://nightlies.apache.org/flink/flink-docs-release-1.12/downloads/setup-pyflink-virtual-env.sh>>;
> 在 2021-11-18 15:05:03，"Asahi Lee" <978466273@qq.com.INVALID&gt; 写道：
> > &gt;Hi!
> > &gt;&amp;nbsp; &amp;nbsp;
> >
> 我在flink官方文档中看到通过&amp;nbsp;&amp;nbsp;setup-pyflink-virtual-env.sh
> > 脚本制作python虚拟环境，请问再哪里可以下载到？文档地址如下：
> > &gt;
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.14/zh/docs/dev/python/faq/