You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "datagic (via GitHub)" <gi...@apache.org> on 2023/01/30 06:17:32 UTC
[GitHub] [incubator-seatunnel] datagic opened a new issue, #3999: [Bug] [Connector-V2] cannot assign instance of com.clickhouse.client.ClickHouseNode
datagic opened a new issue, #3999:
URL: https://github.com/apache/incubator-seatunnel/issues/3999
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
Synchronizing the hdfs data to the clickhouse has been difficult and has never been successful so far.
### SeaTunnel Version
seatunnel ver: 2.3.0
clickhouse ver: 21.8.4.51
### SeaTunnel Config
```conf
env {
execution.parallelism = 10
job.mode = "BATCH"
}
source {
HdfsFile {
path = "/user/hive/warehouse/data.db/xxxxx/"
type = "text"
fs.defaultFS = "hdfs://xxx:8020"
}
}
transform {
}
sink {
Clickhouse {
host = "1.1.1.1:8123"
database = "dname"
table = "tname"
fields = ["a", "b", "c", "d", "e", "f"]
username = "default"
password = "xxxxx"
}
}
```
### Running Command
```shell
./seatunnel.sh --config ../config/xxx.conf -e local
```
### Error Exception
```log
2023-01-30 14:00:13,869 ERROR org.apache.seatunnel.core.starter.Seatunnel - Exception StackTrace:java.util.concurrent.CompletionException: com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.ClassCastException: cannot assign instance of com.clickhouse.client.ClickHouseNode to field org.apache.seatunnel.connectors.seatunnel.clickhouse.shard.Shard.node of type com.clickhouse.client.ClickHouseNode in instance of org.apache.seatunnel.connectors.seatunnel.clickhouse.shard.Shard
at com.hazelcast.spi.impl.AbstractInvocationFuture.wrapInCompletionException(AbstractInvocationFuture.java:1347)
at com.hazelcast.spi.impl.AbstractInvocationFuture.cascadeException(AbstractInvocationFuture.java:1340)
at com.hazelcast.spi.impl.AbstractInvocationFuture.access$200(AbstractInvocationFuture.java:65)
at com.hazelcast.spi.impl.AbstractInvocationFuture$ApplyNode.execute(AbstractInvocationFuture.java:1478)
at com.hazelcast.spi.impl.AbstractInvocationFuture.unblockOtherNode(AbstractInvocationFuture.java:797)
at com.hazelcast.spi.impl.AbstractInvocationFuture.unblockAll(AbstractInvocationFuture.java:759)
at com.hazelcast.spi.impl.AbstractInvocationFuture.complete0(AbstractInvocationFuture.java:1235)
at com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionallyInternal(AbstractInvocationFuture.java:1223)
at com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionally(AbstractInvocationFuture.java:709)
at com.hazelcast.client.impl.spi.impl.ClientInvocation.completeExceptionally(ClientInvocation.java:294)
at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyExceptionWithOwnedPermission(ClientInvocation.java:321)
at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyException(ClientInvocation.java:304)
at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.handleResponse(ClientResponseHandlerSupplier.java:164)
at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.process(ClientResponseHandlerSupplier.java:141)
at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.access$300(ClientResponseHandlerSupplier.java:60)
at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier$DynamicResponseHandler.accept(ClientResponseHandlerSupplier.java:251)
at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier$DynamicResponseHandler.accept(ClientResponseHandlerSupplier.java:243)
at com.hazelcast.client.impl.connection.tcp.TcpClientConnection.handleClientMessage(TcpClientConnection.java:245)
at com.hazelcast.client.impl.protocol.util.ClientMessageDecoder.handleMessage(ClientMessageDecoder.java:135)
at com.hazelcast.client.impl.protocol.util.ClientMessageDecoder.onRead(ClientMessageDecoder.java:89)
at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:136)
at com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:383)
at com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:368)
at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:294)
at com.hazelcast.internal.networking.nio.NioThread.executeRun(NioThread.java:249)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
Caused by: com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.ClassCastException: cannot assign instance of com.clickhouse.client.ClickHouseNode to field org.apache.seatunnel.connectors.seatunnel.clickhouse.shard.Shard.node of type com.clickhouse.client.ClickHouseNode in instance of org.apache.seatunnel.connectors.seatunnel.clickhouse.shard.Shard
at com.hazelcast.internal.serialization.impl.SerializationUtil.handleException(SerializationUtil.java:111)
at com.hazelcast.client.impl.protocol.util.ClientMessageDecoder.onRead(ClientMessageDecoder.java:89)
at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:136)
at com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:383)
at com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:368)
at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:294)
at com.hazelcast.internal.networking.nio.NioThread.executeRun(NioThread.java:249)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
Caused by: com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.ClassCastException: cannot assign instance of com.clickhouse.client.ClickHouseNode to field org.apache.seatunnel.connectors.seatunnel.clickhouse.shard.Shard.node of type com.clickhouse.client.ClickHouseNode in instance of org.apache.seatunnel.connectors.seatunnel.clickhouse.shard.Shard
at com.hazelcast.internal.serialization.impl.SerializationUtil.handleException(SerializationUtil.java:111)
at com.hazelcast.internal.serialization.impl.AbstractSerializationService.readObject(AbstractSerializationService.java:355)
at com.hazelcast.internal.serialization.impl.ByteArrayObjectDataInput.readObject(ByteArrayObjectDataInput.java:600)
at org.apache.seatunnel.engine.core.dag.logical.LogicalVertex.readData(LogicalVertex.java:100)
at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.readInternal(DataSerializableSerializer.java:160)
at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:106)
at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:51)
at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:44)
at com.hazelcast.internal.serialization.impl.AbstractSerializationService.readObject(AbstractSerializationService.java:349)
at com.hazelcast.internal.serialization.impl.ByteArrayObjectDataInput.readObject(ByteArrayObjectDataInput.java:600)
at org.apache.seatunnel.engine.core.dag.logical.LogicalDag.readData(LogicalDag.java:154)
at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.readInternal(DataSerializableSerializer.java:160)
at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:106)
at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:51)
at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:44)
at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:268)
at com.hazelcast.jet.impl.execution.init.CustomClassLoadedObject.deserializeWithCustomClassLoader(CustomClassLoadedObject.java:66)
at org.apache.seatunnel.engine.server.master.JobMaster.init(JobMaster.java:159)
at org.apache.seatunnel.engine.server.CoordinatorService.lambda$submitJob$4(CoordinatorService.java:338)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassCastException: cannot assign instance of com.clickhouse.client.ClickHouseNode to field org.apache.seatunnel.connectors.seatunnel.clickhouse.shard.Shard.node of type com.clickhouse.client.ClickHouseNode in instance of org.apache.seatunnel.connectors.seatunnel.clickhouse.shard.Shard
at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2287)
at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1417)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2293)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
at com.hazelcast.internal.serialization.impl.defaultserializers.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:92)
at com.hazelcast.internal.serialization.impl.defaultserializers.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:85)
at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:44)
at com.hazelcast.internal.serialization.impl.AbstractSerializationService.readObject(AbstractSerializationService.java:349)
... 22 more
```
### Flink or Spark Version
Not used
### Java or Scala Version
jdk 1.8
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Ackyo commented on issue #3999: [Bug] [Connector-V2] cannot assign instance of com.clickhouse.client.ClickHouseNode
Posted by "Ackyo (via GitHub)" <gi...@apache.org>.
Ackyo commented on issue #3999:
URL: https://github.com/apache/incubator-seatunnel/issues/3999#issuecomment-1463168876
I have the same problem too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Connector-V2] cannot assign instance of com.clickhouse.client.ClickHouseNode [seatunnel]
Posted by "taojun610-chris (via GitHub)" <gi...@apache.org>.
taojun610-chris commented on issue #3999:
URL: https://github.com/apache/seatunnel/issues/3999#issuecomment-2008608109
I have the same problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Tianxingdao commented on issue #3999: [Bug] [Connector-V2] cannot assign instance of com.clickhouse.client.ClickHouseNode
Posted by "Tianxingdao (via GitHub)" <gi...@apache.org>.
Tianxingdao commented on issue #3999:
URL: https://github.com/apache/incubator-seatunnel/issues/3999#issuecomment-1427564737
I have the same problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] qianmoQ commented on issue #3999: [Bug] [Connector-V2] cannot assign instance of com.clickhouse.client.ClickHouseNode
Posted by "qianmoQ (via GitHub)" <gi...@apache.org>.
qianmoQ commented on issue #3999:
URL: https://github.com/apache/incubator-seatunnel/issues/3999#issuecomment-1420133273
> > Please provide the Click House environment?
>
> clickhouse ver: 21.8.4.51
>
> I can execute it through the earlier version of waterdrop-1.5.0，However, the above error reports were encountered in Seatunnel-2.3.0.
Unfortunately, I did not reproduce the problem described in multiple tests with hadoop HA and non-HA clusters
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] qianmoQ commented on issue #3999: [Bug] [Connector-V2] cannot assign instance of com.clickhouse.client.ClickHouseNode
Posted by "qianmoQ (via GitHub)" <gi...@apache.org>.
qianmoQ commented on issue #3999:
URL: https://github.com/apache/incubator-seatunnel/issues/3999#issuecomment-1408160878
Please provide the Click House environment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ditgittube commented on issue #3999: [Bug] [Connector-V2] cannot assign instance of com.clickhouse.client.ClickHouseNode
Posted by "ditgittube (via GitHub)" <gi...@apache.org>.
ditgittube commented on issue #3999:
URL: https://github.com/apache/incubator-seatunnel/issues/3999#issuecomment-1476057526
I have the same problem too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Connector-V2] cannot assign instance of com.clickhouse.client.ClickHouseNode [seatunnel]
Posted by "BigTailMonkey (via GitHub)" <gi...@apache.org>.
BigTailMonkey commented on issue #3999:
URL: https://github.com/apache/seatunnel/issues/3999#issuecomment-1916252880
> > > Please provide the Click House environment?
> >
> >
> > clickhouse ver: 21.8.4.51
> > I can execute it through the earlier version of waterdrop-1.5.0，However, the above error reports were encountered in Seatunnel-2.3.0.
>
> Unfortunately, I did not reproduce the problem described in multiple tests with hadoop HA and non-HA clusters
I have the same problem too.This is my config file:
`env {
execution.parallelism = 1
job.mode = "BATCH"
checkpoint.interval = 10000
}
source {
FakeSource {
parallelism = 1
result_table_name = "fake"
row.num = 1
schema = {
fields {
id = "int"
a = "string"
b = "string"
}
}
}
}
sink {
Clickhouse {
source_table_name = "fake"
host = "192.168.1.110:8123"
database = "test_seatunnel"
table = "table_a"
username = "default"
password = ""
}
}`
But I succeeded twice and failed once with same ClickHouse Database and ClickHouse Driver.
The one success was on my own macbook pro with seatunnel 2.3.3+jdk 1.8.0_321.
And the other was also using seatunnel 2.3.3+jdk 11.0.21 on my own macbook pro
The one failure was on centos 7 with seatunnel 2.3.3+jdk 11.0.21.
So I'm guessing it's the OS env? That's a bad guess.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] datagic commented on issue #3999: [Bug] [Connector-V2] cannot assign instance of com.clickhouse.client.ClickHouseNode
Posted by "datagic (via GitHub)" <gi...@apache.org>.
datagic commented on issue #3999:
URL: https://github.com/apache/incubator-seatunnel/issues/3999#issuecomment-1409694340
> Please provide the Click House environment?
clickhouse ver: 21.8.4.51
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Connector-V2] cannot assign instance of com.clickhouse.client.ClickHouseNode [seatunnel]
Posted by "BigTailMonkey (via GitHub)" <gi...@apache.org>.
BigTailMonkey commented on issue #3999:
URL: https://github.com/apache/seatunnel/issues/3999#issuecomment-1916403887
I solved the problem: typed a zip archive of seatunnel from my own computer, transferred it to the machine that was reporting the error, and ran it again and it worked. But I still don't know what's causing it.
Both downloads of seatunnel are version 2.3.3 and the MD5 is the same when compared.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Connector-V2] cannot assign instance of com.clickhouse.client.ClickHouseNode [seatunnel]
Posted by "qianmoQ (via GitHub)" <gi...@apache.org>.
qianmoQ commented on issue #3999:
URL: https://github.com/apache/seatunnel/issues/3999#issuecomment-1916961816
> I solved the problem: typed a zip archive of seatunnel from my own computer, transferred it to the machine that was reporting the error, and ran it again and it worked. But I still don't know what's causing it. Both downloads of seatunnel are version 2.3.3 and the MD5 is the same when compared.
Please provide complete diary？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Connector-V2] cannot assign instance of com.clickhouse.client.ClickHouseNode [seatunnel]
Posted by "WuJiY (via GitHub)" <gi...@apache.org>.
WuJiY commented on issue #3999:
URL: https://github.com/apache/seatunnel/issues/3999#issuecomment-1951642685
解决了吗
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org