You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/08/31 11:05:53 UTC
[GitHub] [incubator-seatunnel] somnmos opened a new issue, #2583: [Bug] [kafka-source] Multiple KafkaSource failed with Only one StreamingContext may be started in this JVM.
somnmos opened a new issue, #2583:
URL: https://github.com/apache/incubator-seatunnel/issues/2583
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
I add seatunnel as maven dependencies to start my service, when I create two kafka-source task, I found only one task works, another one failed.
### SeaTunnel Version
V1.2.2
### SeaTunnel Config
```conf
env {
spark.app.name = "SeaTunnel"
spark.executor.instances = 2
spark.executor.cores = 1
spark.executor.memory = "1g"
spark.master = "local"
spark.serializer = "org.apache.spark.serializer.KryoSerializer"
spark.yarn.archive = "hdfs:///app-logs/spark-2.4.4-1.0.2.zip"
spark.proxy-user = "spark"
spark.driver.allowMultipleContexts = true
}
source {
KafkaStream {
topics = "test1"
consumer.bootstrap.servers = "xxx.xxx.xxx.xxx:9092"
consumer.group.id = "consumer0001"
result_table_name = "test"
}
}
transform {
Sql {
sql = "SELECT * FROM test"
}
}
sink {
Console {
}
}
```
### Running Command
```shell
import org.apache.seatunnel.common.config.DeployMode;
import org.apache.seatunnel.core.base.Seatunnel;
import org.apache.seatunnel.core.base.command.Command;
import org.apache.seatunnel.core.spark.args.SparkCommandArgs;
import org.apache.seatunnel.core.spark.command.SparkCommandBuilder;
import java.io.FileNotFoundException;
import java.net.URISyntaxException;
import java.net.URL;
import java.nio.file.Paths;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public class SparkEngineExample {
static class SparkTask implements Runnable {
private String confFile;
public SparkTask(String confFile) {
this.confFile = confFile;
}
public String getTestConfigFile(String configFile) throws URISyntaxException, FileNotFoundException {
URL resource = SparkEngineExample.class.getResource(configFile);
if (resource == null) {
throw new FileNotFoundException("Could not find config file: " + configFile);
}
return Paths.get(resource.toURI()).toString();
}
@Override
public void run() {
String configFile = null;
try {
configFile = getTestConfigFile(this.confFile);
} catch (URISyntaxException e) {
e.printStackTrace();
} catch (FileNotFoundException e) {
e.printStackTrace();
}
SparkCommandArgs sparkArgs = new SparkCommandArgs();
sparkArgs.setConfigFile(configFile);
sparkArgs.setCheckConfig(false);
sparkArgs.setVariables(null);
sparkArgs.setDeployMode(DeployMode.CLIENT);
Command<SparkCommandArgs> sparkCommand = new SparkCommandBuilder().buildCommand(sparkArgs);
Seatunnel.run(sparkCommand);
}
}
public static void main(String[] args) {
ExecutorService service = Executors.newCachedThreadPool();
service.execute(new SparkTask("/examples/test1.conf"));
service.execute(new SparkTask("/examples/test2.conf"));
service.shutdown();
}
}
```
### Error Exception
```log
22/08/31 18:53:41 ERROR Seatunnel:
===============================================================================
22/08/31 18:53:41 ERROR Seatunnel: Fatal Error,
22/08/31 18:53:41 ERROR Seatunnel: Please submit bug report in https://github.com/apache/incubator-seatunnel/issues
22/08/31 18:53:41 ERROR Seatunnel: Reason:Execute Spark task error
22/08/31 18:53:41 ERROR Seatunnel: Exception StackTrace:java.lang.RuntimeException: Execute Spark task error
at org.apache.seatunnel.core.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:79)
at org.apache.seatunnel.core.base.Seatunnel.run(Seatunnel.java:39)
at com.somnus.SparkEngineExample$SparkTask.run(SparkEngineExample.java:67)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Only one StreamingContext may be started in this JVM. Currently running StreamingContext was started atorg.apache.spark.streaming.StreamingContext.start(StreamingContext.scala:569)
org.apache.seatunnel.spark.stream.SparkStreamingExecution.start(SparkStreamingExecution.scala:68)
org.apache.seatunnel.core.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:76)
org.apache.seatunnel.core.base.Seatunnel.run(Seatunnel.java:39)
com.somnus.SparkEngineExample$SparkTask.run(SparkEngineExample.java:67)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
at org.apache.spark.streaming.StreamingContext$.org$apache$spark$streaming$StreamingContext$$assertNoOtherContextIsActive(StreamingContext.scala:738)
at org.apache.spark.streaming.StreamingContext.start(StreamingContext.scala:571)
at org.apache.seatunnel.spark.stream.SparkStreamingExecution.start(SparkStreamingExecution.scala:68)
at org.apache.seatunnel.core.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:76)
... 5 more
22/08/31 18:53:41 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@16c74877{/streaming,null,AVAILABLE,@Spark}
```
### Flink or Spark Version
Spark2.4.4
### Java or Scala Version
Scala2.11.12
### Screenshots
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] somnmos commented on issue #2583: [Bug] [kafka-source] Multiple KafkaSource failed with Only one StreamingContext may be started in this JVM.
Posted by GitBox <gi...@apache.org>.
somnmos commented on issue #2583:
URL: https://github.com/apache/incubator-seatunnel/issues/2583#issuecomment-1245234842
I am Looking forward to your further information.
------------------ 原始邮件 ------------------
发件人: "apache/incubator-seatunnel" ***@***.***>;
发送时间: 2022年9月7日(星期三) 中午1:54
***@***.***>;
***@***.******@***.***>;
主题: Re: [apache/incubator-seatunnel] [Bug] [kafka-source] Multiple KafkaSource failed with Only one StreamingContext may be started in this JVM. (Issue #2583)
I wanna to quote the seatunnel maven dependency directly in my microservice module, rather than install seatunnel binary on my virtual machine.
It's hard to do that at now. But maybe we can implement this feature in our new engine.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #2583: [Bug] [kafka-source] Multiple KafkaSource failed with Only one StreamingContext may be started in this JVM.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #2583:
URL: https://github.com/apache/incubator-seatunnel/issues/2583#issuecomment-1331490913
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] somnmos commented on issue #2583: [Bug] [kafka-source] Multiple KafkaSource failed with Only one StreamingContext may be started in this JVM.
Posted by GitBox <gi...@apache.org>.
somnmos commented on issue #2583:
URL: https://github.com/apache/incubator-seatunnel/issues/2583#issuecomment-1238909904
I wanna to quote the seatunnel maven dependency directly in my microservice module, rather than install seatunnel binary on my virtual machine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on issue #2583: [Bug] [kafka-source] Multiple KafkaSource failed with Only one StreamingContext may be started in this JVM.
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on issue #2583:
URL: https://github.com/apache/incubator-seatunnel/issues/2583#issuecomment-1232810279
The error very clear, can't run two spark job in one jvm. Why do you want to break this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on issue #2583: [Bug] [kafka-source] Multiple KafkaSource failed with Only one StreamingContext may be started in this JVM.
Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on issue #2583:
URL: https://github.com/apache/incubator-seatunnel/issues/2583#issuecomment-1504867587
> > > Don't this bug fixed by #2706 ?
> >
> >
> > No, can't run two job in one jvm on spark/flink. We can support it on our own eninge if user want to.
>
> now was it supported on the zeta engine?
Yes, The zeta supports this feature architecturally. But the code should do some change on client side. If you want this feature. Please create an issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on issue #2583: [Bug] [kafka-source] Multiple KafkaSource failed with Only one StreamingContext may be started in this JVM.
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on issue #2583:
URL: https://github.com/apache/incubator-seatunnel/issues/2583#issuecomment-1238937902
> I wanna to quote the seatunnel maven dependency directly in my microservice module, rather than install seatunnel binary on my virtual machine.
It's hard to do that at now. But maybe we can implement this feature in our new engine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X closed issue #2583: [Bug] [kafka-source] Multiple KafkaSource failed with Only one StreamingContext may be started in this JVM.
Posted by GitBox <gi...@apache.org>.
Hisoka-X closed issue #2583: [Bug] [kafka-source] Multiple KafkaSource failed with Only one StreamingContext may be started in this JVM.
URL: https://github.com/apache/incubator-seatunnel/issues/2583
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on issue #2583: [Bug] [kafka-source] Multiple KafkaSource failed with Only one StreamingContext may be started in this JVM.
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on issue #2583:
URL: https://github.com/apache/incubator-seatunnel/issues/2583#issuecomment-1257519473
Don't this bug fixed by https://github.com/apache/incubator-seatunnel/pull/2706 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on issue #2583: [Bug] [kafka-source] Multiple KafkaSource failed with Only one StreamingContext may be started in this JVM.
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on issue #2583:
URL: https://github.com/apache/incubator-seatunnel/issues/2583#issuecomment-1257526091
> Don't this bug fixed by #2706 ?
No, can't run two job in one jvm on spark/flink. We can support it on our own eninge if user want to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] somnmos commented on issue #2583: [Bug] [kafka-source] Multiple KafkaSource failed with Only one StreamingContext may be started in this JVM.
Posted by "somnmos (via GitHub)" <gi...@apache.org>.
somnmos commented on issue #2583:
URL: https://github.com/apache/incubator-seatunnel/issues/2583#issuecomment-1504844861
> > Don't this bug fixed by #2706 ?
>
> No, can't run two job in one jvm on spark/flink. We can support it on our own eninge if user want to.
now it support it on the zeta engine?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org