You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "Light-Towers (via GitHub)" <gi...@apache.org> on 2025/01/02 09:31:11 UTC
[I] [Bug] [Hive source connector] FileConnectorException: ErrorCode:[FILE-08], ErrorDescription:[File read failed] [seatunnel]
Light-Towers opened a new issue, #8427:
URL: https://github.com/apache/seatunnel/issues/8427
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
Hive source connector 出现文件读取异常,但文件确实存在(见截图)。业务表将近有4亿条数据,共200个文件。请问为什么会出现这个问题?

### SeaTunnel Version
2.3.8
### SeaTunnel Config
```conf
env {
execution.parallelism = 4
job.mode = "BATCH"
read_limit.rows_per_second = 50000
}
source {
Hive {
table_name = "dw.app_enterprise_info"
metastore_uri = "thrift://master02:9083"
hive_site_path = "/etc/hive/conf/hive-site.xml"
hdfs_site_path = "/etc/hadoop/conf/hdfs-site.xml"
read_partitions = ["dt=2024-12-31"]
compress_codec = "orc"
}
}
sink {
Console { }
}
```
### Running Command
```shell
/opt/seatunnel/apache-seatunnel-2.3.8/bin/seatunnel.sh --config ./hive2mongo -m local
```
### Error Exception
```log
2025-01-02 16:47:04,888 INFO [.a.s.c.s.c.s.ConsoleSinkWriter] [st-multi-table-sink-writer-1] - subtaskIndex=0 rowIndex=1806171: SeaTunnelRow#tableId=dw.app_enterprise_info SeaTunnelRow#kind=INSERT : d783b7bc34975a, 银川市建材销售部, , , , , , xxx, 一般项目:, 银川市商住楼105, 银川市商住楼105, 2022-02-24, 926xxxxNW4Y, 001, 注销, [106.30129, 38.494615], 640000, 宁夏回族自治区, 640100, 银川市, , , , , , , , , , , , , , , , , , , [1], 18100000066, , , [{"category":"状态","tagName":"续"}], 6, 2022-02-24, 9999-12-31, , , 人民币, , MAxxxxx4, 2024-04-29, 1, 0, 2024-07-18, 2024-12-31
2025-01-02 16:47:04,890 INFO [c.h.i.s.t.TcpServerConnection ] [hz.main.IO.thread-in-1] - [localhost]:5801 [seatunnel-930048] [5.1] Connection[id=1, /127.0.0.1:5801->/127.0.0.1:55002, qualifier=null, endpoint=[127.0.0.1]:55002, remoteUuid=412e8da3-b9de-4db3-b558-bf654a01e6af, alive=false, connectionType=JVM, planeIndex=-1] closed. Reason: Connection closed by the other side
2025-01-02 16:47:04,890 INFO [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel-930048] [5.1] Removed connection to endpoint: [localhost]:5801:a3575d69-8d06-49d7-b52e-c16135fdeb48, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/127.0.0.1:55002->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2025-01-02 16:47:04.622, lastWriteTime=2025-01-02 16:47:01.020, closedTime=2025-01-02 16:47:04.886, connected server version=5.1}
2025-01-02 16:47:04,890 INFO [c.h.c.LifecycleService ] [main] - hz.client_1 [seatunnel-930048] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
2025-01-02 16:47:04,892 INFO [c.h.c.i.ClientEndpointManager ] [hz.main.event-3] - [localhost]:5801 [seatunnel-930048] [5.1] Destroying ClientEndpoint{connection=Connection[id=1, /127.0.0.1:5801->/127.0.0.1:55002, qualifier=null, endpoint=[127.0.0.1]:55002, remoteUuid=412e8da3-b9de-4db3-b558-bf654a01e6af, alive=false, connectionType=JVM, planeIndex=-1], clientUuid=412e8da3-b9de-4db3-b558-bf654a01e6af, clientName=hz.client_1, authenticated=true, clientVersion=5.1, creationTime=1735805435983, latest clientAttributes=lastStatisticsCollectionTime=1735807621018,enterprise=false,clientType=JVM,clientVersion=5.1,clusterConnectionTimestamp=1735805435971,clientAddress=127.0.0.1,clientName=hz.client_1,credentials.principal=null,os.committedVirtualMemorySize=24853250048,os.freePhysicalMemorySize=113494515712,os.freeSwapSpaceSize=0,os.maxFileDescriptorCount=100000,os.openFileDescriptorCount=69,os.processCpuTime=1177350000000,os.systemLoadAverage=0.64,os.totalPhysicalMemorySize=270185119744,os
.totalSwapSpaceSize=0,runtime.availableProcessors=88,runtime.freeMemory=186469144,runtime.maxMemory=513802240,runtime.totalMemory=513802240,runtime.uptime=2188449,runtime.usedMemory=327335416, labels=[]}
2025-01-02 16:47:04,893 INFO [c.h.c.LifecycleService ] [main] - hz.client_1 [seatunnel-930048] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
2025-01-02 16:47:04,894 INFO [s.c.s.s.c.ClientExecuteCommand] [main] - Closed SeaTunnel client......
2025-01-02 16:47:04,894 INFO [c.h.c.LifecycleService ] [main] - [localhost]:5801 [seatunnel-930048] [5.1] [localhost]:5801 is SHUTTING_DOWN
2025-01-02 16:47:04,898 INFO [c.h.i.p.i.MigrationManager ] [hz.main.cached.thread-16] - [localhost]:5801 [seatunnel-930048] [5.1] Shutdown request of Member [localhost]:5801 - a3575d69-8d06-49d7-b52e-c16135fdeb48 this master is handled
2025-01-02 16:47:04,904 INFO [c.h.i.i.Node ] [main] - [localhost]:5801 [seatunnel-930048] [5.1] Shutting down connection manager...
2025-01-02 16:47:04,906 INFO [c.h.i.i.Node ] [main] - [localhost]:5801 [seatunnel-930048] [5.1] Shutting down node engine...
2025-01-02 16:47:04,919 INFO [.c.c.DefaultClassLoaderService] [main] - close classloader service
2025-01-02 16:47:04,919 INFO [o.a.s.c.u.RetryUtils ] [event-forwarder-0] - Failed to execute due to java.lang.NullPointerException: Target cannot be null!
at com.hazelcast.internal.util.Preconditions.checkNotNull(Preconditions.java:59)
at com.hazelcast.spi.impl.operationservice.impl.OperationServiceImpl.createInvocationBuilder(OperationServiceImpl.java:300)
at org.apache.seatunnel.engine.server.utils.NodeEngineUtil.sendOperationToMasterNode(NodeEngineUtil.java:37)
at org.apache.seatunnel.engine.server.EventService.lambda$initEventForwardService$1(EventService.java:72)
at org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:48)
at org.apache.seatunnel.engine.server.EventService.lambda$initEventForwardService$2(EventService.java:70)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
. Retrying attempt (1/2) after backoff of 0 ms
2025-01-02 16:47:04,920 INFO [o.a.s.c.u.RetryUtils ] [event-forwarder-0] - Failed to execute due to java.lang.NullPointerException: Target cannot be null!
at com.hazelcast.internal.util.Preconditions.checkNotNull(Preconditions.java:59)
at com.hazelcast.spi.impl.operationservice.impl.OperationServiceImpl.createInvocationBuilder(OperationServiceImpl.java:300)
at org.apache.seatunnel.engine.server.utils.NodeEngineUtil.sendOperationToMasterNode(NodeEngineUtil.java:37)
at org.apache.seatunnel.engine.server.EventService.lambda$initEventForwardService$1(EventService.java:72)
at org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:48)
at org.apache.seatunnel.engine.server.EventService.lambda$initEventForwardService$2(EventService.java:70)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
. Retrying attempt (2/2) after backoff of 0 ms
2025-01-02 16:47:04,920 WARN [o.a.s.e.s.EventService ] [event-forwarder-0] - Event forward failed, discard events 1
java.lang.RuntimeException: Execute given execution failed after retry 2 times
at org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:75) ~[seatunnel-starter.jar:2.3.8]
at org.apache.seatunnel.engine.server.EventService.lambda$initEventForwardService$2(EventService.java:70) ~[seatunnel-starter.jar:2.3.8]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: java.lang.NullPointerException: Target cannot be null!
at com.hazelcast.internal.util.Preconditions.checkNotNull(Preconditions.java:59) ~[seatunnel-starter.jar:2.3.8]
at com.hazelcast.spi.impl.operationservice.impl.OperationServiceImpl.createInvocationBuilder(OperationServiceImpl.java:300) ~[seatunnel-starter.jar:2.3.8]
at org.apache.seatunnel.engine.server.utils.NodeEngineUtil.sendOperationToMasterNode(NodeEngineUtil.java:37) ~[seatunnel-starter.jar:2.3.8]
at org.apache.seatunnel.engine.server.EventService.lambda$initEventForwardService$1(EventService.java:72) ~[seatunnel-starter.jar:2.3.8]
at org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:48) ~[seatunnel-starter.jar:2.3.8]
... 6 more
2025-01-02 16:47:04,931 INFO [c.h.i.i.NodeExtension ] [main] - [localhost]:5801 [seatunnel-930048] [5.1] Destroying node NodeExtension.
2025-01-02 16:47:04,931 INFO [c.h.i.i.Node ] [main] - [localhost]:5801 [seatunnel-930048] [5.1] Hazelcast Shutdown is completed in 34 ms.
2025-01-02 16:47:04,932 INFO [c.h.c.LifecycleService ] [main] - [localhost]:5801 [seatunnel-930048] [5.1] [localhost]:5801 is SHUTDOWN
2025-01-02 16:47:04,932 INFO [s.c.s.s.c.ClientExecuteCommand] [main] - Closed HazelcastInstance ......
2025-01-02 16:47:04,932 INFO [s.c.s.s.c.ClientExecuteCommand] [main] - Closed metrics executor service ......
2025-01-02 16:47:04,932 ERROR [o.a.s.c.s.SeaTunnel ] [main] -
===============================================================================
2025-01-02 16:47:04,932 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Fatal Error,
2025-01-02 16:47:04,932 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Please submit bug report in https://github.com/apache/seatunnel/issues
2025-01-02 16:47:04,932 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Reason:SeaTunnel job executed failed
2025-01-02 16:47:04,933 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:213)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.connectors.seatunnel.file.exception.FileConnectorException: ErrorCode:[FILE-08], ErrorDescription:[File read failed] - Read data from this file [dw.app_enterprise_info_hdfs://nameservice1/user/hive/warehouse/app/app_enterprise_info/dt=2024-12-31/part-00009-7f77f045-c85d-4e50-9f98-51d2874ecce2.c000] failed
at org.apache.seatunnel.connectors.seatunnel.hive.source.reader.MultipleTableHiveSourceReader.pollNext(MultipleTableHiveSourceReader.java:87)
at org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.collect(SourceFlowLifeCycle.java:159)
at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.collect(SourceSeaTunnelTask.java:127)
at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:168)
at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:132)
at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:693)
at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1018)
at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:39)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:205)
... 2 more
2025-01-02 16:47:04,934 ERROR [o.a.s.c.s.SeaTunnel ] [main] -
===============================================================================
Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:213)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.connectors.seatunnel.file.exception.FileConnectorException: ErrorCode:[FILE-08], ErrorDescription:[File read failed] - Read data from this file [dw.app_enterprise_info_hdfs://nameservice1/user/hive/warehouse/app/app_enterprise_info/dt=2024-12-31/part-00009-7f77f045-c85d-4e50-9f98-51d2874ecce2.c000] failed
at org.apache.seatunnel.connectors.seatunnel.hive.source.reader.MultipleTableHiveSourceReader.pollNext(MultipleTableHiveSourceReader.java:87)
at org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.collect(SourceFlowLifeCycle.java:159)
at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.collect(SourceSeaTunnelTask.java:127)
at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:168)
at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:132)
at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:693)
at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1018)
at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:39)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:205)
... 2 more
2025-01-02 16:47:04,935 INFO [s.c.s.s.c.ClientExecuteCommand] [ForkJoinPool.commonPool-worker-100] - run shutdown hook because get close signal
```
### Zeta or Flink or Spark Version
Zeta
### Java or Scala Version
java version "1.8.0_181"
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Hive source connector] OrcReadStrategy 读取文件出现 NPE [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8427:
URL: https://github.com/apache/seatunnel/issues/8427#issuecomment-2691774445
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Hive source connector] OrcReadStrategy 读取文件出现 NPE [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #8427: [Bug] [Hive source connector] OrcReadStrategy 读取文件出现 NPE
URL: https://github.com/apache/seatunnel/issues/8427
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Hive source connector] OrcReadStrategy 读取文件出现 NLP [seatunnel]
Posted by "Light-Towers (via GitHub)" <gi...@apache.org>.
Light-Towers commented on issue #8427:
URL: https://github.com/apache/seatunnel/issues/8427#issuecomment-2614270615
> Are you saying that if a field in the Hive table is an array and there are two records with the same array value, this bug will be triggered?
Yes, if nonull records in a rowBatch is same,like:
```
['欧元'],null,['欧元']
```
Here are my testing steps:
1. sql:
```sql
CREATE TABLE `test.orc_array_2`(
`finance_currency` array<string>
) PARTITIONED BY (`dt` string) STORED AS ORC;
insert into test.orc_array_2 partition (dt="2025-01-06") values (array('欧元')),(array('欧元'));
```
2. copy file `/user/hive/warehouse/test/orc_array_2/dt=2025-01-06/000000_0` into the project `connector-file-base/src/test/resources/000000_0` , change testOrcRead()
3. change `OrcReadStrategyTest.testOrcRead()` content "/test.orc" into "/000000_0", the error will happen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Hive source connector] OrcReadStrategy 读取文件出现 NPE [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8427:
URL: https://github.com/apache/seatunnel/issues/8427#issuecomment-2708584584
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Hive source connector] OrcReadStrategy 读取文件出现 NLP [seatunnel]
Posted by "Light-Towers (via GitHub)" <gi...@apache.org>.
Light-Towers commented on issue #8427:
URL: https://github.com/apache/seatunnel/issues/8427#issuecomment-2612032870
调试后发现,当一批次某个字段数据都一样时,rowBatch.cols[0].child.vector中只会剩下一条数据,而对应的

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Hive source connector] OrcReadStrategy 读取文件出现 NLP [seatunnel]
Posted by "Light-Towers (via GitHub)" <gi...@apache.org>.
Light-Towers commented on issue #8427:
URL: https://github.com/apache/seatunnel/issues/8427#issuecomment-2611883429
我发现当数组字段中如果数据完全一致就会异常,像这样,

而只要有1条数据不同便会成功。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Hive source connector] OrcReadStrategy 读取文件出现 NLP [seatunnel]
Posted by "TyrantLucifer (via GitHub)" <gi...@apache.org>.
TyrantLucifer commented on issue #8427:
URL: https://github.com/apache/seatunnel/issues/8427#issuecomment-2614216800
Are you saying that if a field in the Hive table is an array and there are two records with the same array value, this bug will be triggered?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org