You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by GitBox <gi...@apache.org> on 2022/08/02 19:37:43 UTC
[GitHub] [bigtop] vivostar opened a new pull request, #973: BIGTOP-3767. Add Spark Thrift Server Puppet Deploy Code
vivostar opened a new pull request, #973:
URL: https://github.com/apache/bigtop/pull/973
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'BIGTOP-3638: Your PR title ...'.
-->
### Description of PR
Add Spark Thrift Server Puppet Deploy Code
### How was this patch tested?
1. #972 first should resolve the pr
2. edit the docker-hadoop.sh
```yaml
bigtop::hadoop_head_node: $1
hadoop::hadoop_storage_dirs:
- /data/1
bigtop::roles_enabled: true
bigtop::roles:
- namenode
- resourcemanager
- hive-client
- hive-metastore
- datanode
- nodemanager
- mapred-app
- spark-on-yarn
- spark-thriftserver
bigtop::bigtop_repo_uri: ["http://repos.bigtop.apache.org/releases/3.1.0/centos/7/\$basearch"]
hadoop_hive::common_config::hive_execution_engine: "tez"
hadoop::common::use_tez: true
hadoop::common_yarn::yarn_resourcemanager_scheduler_class: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler
hadoop::common_yarn::yarn_scheduler_capacity_maximum_am_resource_percent: 0.5
hadoop_hive::common_config::hive_tez_container_size: 1024
hadoop_hive::common_config::hive_tez_cpu_vcores: 1
hadoop_hive::common_config::hive_server2_enable_doAs: false
```
3. create a docker containner
```
./docker-hadoop.sh -d -c 1
```
4. now you can use beeline to test the Thrift JDBC/ODBC server:
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'BIGTOP-3638. Your PR title ...')?
- [x] Make sure that newly added files do not have any licensing issues. When in doubt refer to https://www.apache.org/licenses/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #973: BIGTOP-3767. Add Spark Thrift Server Puppet Deploy Code
Posted by GitBox <gi...@apache.org>.
iwasakims commented on PR #973:
URL: https://github.com/apache/bigtop/pull/973#issuecomment-1203483822
You can update the `roles_map` in bigtop-deploy/puppet/modules/spark/templates/spark-defaults.conf and add smoke-tests in follow-up JIRAs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] vivostar commented on pull request #973: BIGTOP-3767. Add Spark Thrift Server Puppet Deploy Code
Posted by GitBox <gi...@apache.org>.
vivostar commented on PR #973:
URL: https://github.com/apache/bigtop/pull/973#issuecomment-1210002338
hi, @iwasakims, I have updated the pr(we can bind port manually).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] vivostar commented on pull request #973: BIGTOP-3767. Add Spark Thrift Server Puppet Deploy Code
Posted by GitBox <gi...@apache.org>.
vivostar commented on PR #973:
URL: https://github.com/apache/bigtop/pull/973#issuecomment-1210074439
tested container debian-10 x86_64, there is an error when start thrift server.
```shell
22/08/10 01:56:11 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://2d5a19e00c2e.bigtop.apache.org:7077...
22/08/10 01:56:11 WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master 2d5a19e00c2e.bigtop.apache.org:7077
org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)
at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:109)
at org.apache.spark.deploy.client.StandaloneAppClient$ClientEndpoint$$anon$1.run(StandaloneAppClient.scala:107)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Failed to connect to 2d5a19e00c2e.bigtop.apache.org/172.21.0.3:7077
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:287)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:218)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:230)
at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:204)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:202)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:198)
... 4 more
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 2d5a19e00c2e.bigtop.apache.org/172.21.0.3:7077
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
```
That's because the defalt spark cluster manager type is standalone instead of yarn. So I should add another config to contral spark cluster manager type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] vivostar commented on pull request #973: BIGTOP-3767. Add Spark Thrift Server Puppet Deploy Code
Posted by GitBox <gi...@apache.org>.
vivostar commented on PR #973:
URL: https://github.com/apache/bigtop/pull/973#issuecomment-1210588112
tested on container debain-10 x86_64
```shell
./docker-hadoop.sh \
-d \
--create 2 \
--image bigtop/puppet:trunk-debian-10 \
--memory 8g \
--repo http://repos.bigtop.apache.org/releases/3.1.1/debian/10/\$\(ARCH\) \
--disable-gpg-check \
--stack hdfs,yarn,mapreduce,spark,hive
root@0fc48ca83bcd:/# sudo -u hdfs hdfs dfs -mkdir /user/spark
root@0fc48ca83bcd:/# sudo -u hdfs hdfs dfs -chmod 777 /user/spark
root@0fc48ca83bcd:/# sudo -u hdfs hdfs dfs -chown spark:spark /user/spark
root@0fc48ca83bcd:/# future=""
root@0fc48ca83bcd:/# bash -c "puppet apply --detailed-exitcodes $future --hiera_config=/etc/puppet/hiera.yaml --modulepath=/bigtop-home/bigtop-deploy/puppet/modules:/etc/puppet/modules:/usr/share/puppet/modules:/etc/puppetlabs/code/modules:/etc/puppet/code/modules /bigtop-home/bigtop-deploy/puppet/manifests"
root@0fc48ca83bcd:/# beeline
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #973: BIGTOP-3767. Add Spark Thrift Server Puppet Deploy Code
Posted by GitBox <gi...@apache.org>.
iwasakims commented on PR #973:
URL: https://github.com/apache/bigtop/pull/973#issuecomment-1203478250
spark-thriftserver stopped after emitting following log to /var/log/spark/spark-thriftserver.out when I ran smoke-tests of spark. @vivostar
```
22/08/03 04:35:48 INFO HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
22/08/03 04:35:48 INFO ObjectStore: ObjectStore, initialize called
22/08/03 04:35:49 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
22/08/03 04:35:49 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
Wed Aug 03 04:35:49 UTC 2022 Thread[main,5,main] java.io.FileNotFoundException: derby.log (Permission denied)
Wed Aug 03 04:35:49 UTC 2022 Thread[main,5,main] Cleanup action starting
ERROR XBM0H: Directory /metastore_db cannot be created.
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.impl.services.monitor.StorageFactoryService$10.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.derby.impl.services.monitor.StorageFactoryService.createServiceRoot(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.createPersistentService(Unknown Source)
at org.apache.derby.impl.services.monitor.FileMonitor.createPersistentService(Unknown Source)
at org.apache.derby.iapi.services.monitor.Monitor.createPersistentService(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection$5.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.derby.impl.jdbc.EmbedConnection.createPersistentService(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.createDatabase(Unknown Source)
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] vivostar commented on pull request #973: BIGTOP-3767. Add Spark Thrift Server Puppet Deploy Code
Posted by GitBox <gi...@apache.org>.
vivostar commented on PR #973:
URL: https://github.com/apache/bigtop/pull/973#issuecomment-1211507355
tested on debian-10, centos-7, ubuntu-20.04
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] vivostar commented on pull request #973: BIGTOP-3767. Add Spark Thrift Server Puppet Deploy Code
Posted by GitBox <gi...@apache.org>.
vivostar commented on PR #973:
URL: https://github.com/apache/bigtop/pull/973#issuecomment-1211577314
tested on ubtuntu-18.04 and fedora-35
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims merged pull request #973: BIGTOP-3767. Add Spark Thrift Server Puppet Deploy Code
Posted by GitBox <gi...@apache.org>.
iwasakims merged PR #973:
URL: https://github.com/apache/bigtop/pull/973
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #973: BIGTOP-3767. Add Spark Thrift Server Puppet Deploy Code
Posted by GitBox <gi...@apache.org>.
iwasakims commented on PR #973:
URL: https://github.com/apache/bigtop/pull/973#issuecomment-1206006802
I tried this again. spark-thriftserver just stopped even without running the smoke-test. My previous comment was misleading. It looks insufficient configuration for the service.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on a diff in pull request #973: BIGTOP-3767. Add Spark Thrift Server Puppet Deploy Code
Posted by GitBox <gi...@apache.org>.
iwasakims commented on code in PR #973:
URL: https://github.com/apache/bigtop/pull/973#discussion_r939560607
##########
bigtop-packages/src/common/spark/spark-thriftserver.svc:
##########
@@ -45,7 +45,8 @@ start() {
LOG_FILE=/var/log/spark/${DAEMON}.out
runuser -s /bin/bash $SVC_USER -c "nohup nice -n 0 \
- ${EXEC_PATH} --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 1 $DAEMON_FLAGS \
+ ${EXEC_PATH} --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 \
+ --hiveconf hive.server2.thrift.port=12000 1 $DAEMON_FLAGS \
Review Comment:
The port number should not be hardcoded. We usually use service default file (/etc/default/spark-thriftserver) for this. init script generated from template [loads the service default file.](https://github.com/apache/bigtop/blob/60509a922055072223639b268feb0e212666ec52/bigtop-packages/src/templates/init.d.tmpl#L190-L193)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] vivostar commented on pull request #973: BIGTOP-3767. Add Spark Thrift Server Puppet Deploy Code
Posted by GitBox <gi...@apache.org>.
vivostar commented on PR #973:
URL: https://github.com/apache/bigtop/pull/973#issuecomment-1211557492
tested on debian-11, rockylinux-8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] vivostar commented on pull request #973: BIGTOP-3767. Add Spark Thrift Server Puppet Deploy Code
Posted by GitBox <gi...@apache.org>.
vivostar commented on PR #973:
URL: https://github.com/apache/bigtop/pull/973#issuecomment-1206599721
> > could you give me your deploy yaml so that I can reproduce the problem?
>
> I modified cluster.pp for local testing. The gateway_server is the same node of hadoop_head_node in the following deployment.
>
> ```
> diff --git a/bigtop-deploy/puppet/manifests/cluster.pp b/bigtop-deploy/puppet/manifests/cluster.pp
> index b016d6f..c432036 100644
> --- a/bigtop-deploy/puppet/manifests/cluster.pp
> +++ b/bigtop-deploy/puppet/manifests/cluster.pp
> @@ -69,6 +69,7 @@ $roles_map = {
> worker => ["spark-on-yarn"],
> client => ["spark-client"],
> library => ["spark-yarn-slave"],
> + gateway_server => ["spark-thriftserver"],
> },
> spark-standalone => {
> master => ["spark-master"],
> ```
>
> ```
> $ cd provisioner/docker
> $ ./docker-hadoop.sh \
> --create 3 \
> --image bigtop/puppet:trunk-centos-7 \
> --memory 8g \
> --repo file:///bigtop-home/output \
> --disable-gpg-check \
> --stack hdfs,yarn,mapreduce,spark
> ```
Hi, @iwasakims , the spark thrift server depends on hive. That's the reason of that error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] vivostar commented on pull request #973: BIGTOP-3767. Add Spark Thrift Server Puppet Deploy Code
Posted by GitBox <gi...@apache.org>.
vivostar commented on PR #973:
URL: https://github.com/apache/bigtop/pull/973#issuecomment-1204164125
> > /var/log/spark/spark-thriftserver.out
>
> The log destination should be /var/log/spark/spark-thriftserver.log rather than spark-thriftserver.out.
Thanks for your advice, I will fix these problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #973: BIGTOP-3767. Add Spark Thrift Server Puppet Deploy Code
Posted by GitBox <gi...@apache.org>.
iwasakims commented on PR #973:
URL: https://github.com/apache/bigtop/pull/973#issuecomment-1206009379
> could you give me your deploy yaml so that I can reproduce the problem?
I modified cluster.pp for local testing. The gateway_server is the same node of hadoop_head_node in the following deployment.
```
diff --git a/bigtop-deploy/puppet/manifests/cluster.pp b/bigtop-deploy/puppet/manifests/cluster.pp
index b016d6f..c432036 100644
--- a/bigtop-deploy/puppet/manifests/cluster.pp
+++ b/bigtop-deploy/puppet/manifests/cluster.pp
@@ -69,6 +69,7 @@ $roles_map = {
worker => ["spark-on-yarn"],
client => ["spark-client"],
library => ["spark-yarn-slave"],
+ gateway_server => ["spark-thriftserver"],
},
spark-standalone => {
master => ["spark-master"],
```
```
$ cd provisioner/docker
$ ./docker-hadoop.sh \
--create 3 \
--image bigtop/puppet:trunk-centos-7 \
--memory 8g \
--repo file:///bigtop-home/output \
--disable-gpg-check \
--stack hdfs,yarn,mapreduce,spark
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #973: BIGTOP-3767. Add Spark Thrift Server Puppet Deploy Code
Posted by GitBox <gi...@apache.org>.
iwasakims commented on PR #973:
URL: https://github.com/apache/bigtop/pull/973#issuecomment-1214292836
@vivostar I filed [BIGTOP-3776](https://issues.apache.org/jira/browse/BIGTOP-3776) as a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] vivostar commented on pull request #973: BIGTOP-3767. Add Spark Thrift Server Puppet Deploy Code
Posted by GitBox <gi...@apache.org>.
vivostar commented on PR #973:
URL: https://github.com/apache/bigtop/pull/973#issuecomment-1205286438
> Cleanup
hi, @iwasakims , could you give me your deploy yaml so that I can reproduce the problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] vivostar commented on pull request #973: BIGTOP-3767. Add Spark Thrift Server Puppet Deploy Code
Posted by GitBox <gi...@apache.org>.
vivostar commented on PR #973:
URL: https://github.com/apache/bigtop/pull/973#issuecomment-1210187500
When testing on Debian-10, the puppet verion is puppet5, There is verion puppet gap, and it will make default spark master url be spark standalone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] vivostar commented on a diff in pull request #973: BIGTOP-3767. Add Spark Thrift Server Puppet Deploy Code
Posted by GitBox <gi...@apache.org>.
vivostar commented on code in PR #973:
URL: https://github.com/apache/bigtop/pull/973#discussion_r939596924
##########
bigtop-packages/src/common/spark/spark-thriftserver.svc:
##########
@@ -45,7 +45,8 @@ start() {
LOG_FILE=/var/log/spark/${DAEMON}.out
runuser -s /bin/bash $SVC_USER -c "nohup nice -n 0 \
- ${EXEC_PATH} --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 1 $DAEMON_FLAGS \
+ ${EXEC_PATH} --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 \
+ --hiveconf hive.server2.thrift.port=12000 1 $DAEMON_FLAGS \
Review Comment:
thanks, I fix this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] vivostar commented on pull request #973: BIGTOP-3767. Add Spark Thrift Server Puppet Deploy Code
Posted by GitBox <gi...@apache.org>.
vivostar commented on PR #973:
URL: https://github.com/apache/bigtop/pull/973#issuecomment-1205375234
The current spark smoke test doesn't correlate to the spark thrift server, so I think the smoke test shouldn't effect the the spark thrift server.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] vivostar commented on pull request #973: BIGTOP-3767. Add Spark Thrift Server Puppet Deploy Code
Posted by GitBox <gi...@apache.org>.
vivostar commented on PR #973:
URL: https://github.com/apache/bigtop/pull/973#issuecomment-1206624823
And the stacks should exclude the **hive server2**, that's because spark thrift server default port is the same with hive server2 defalt port.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] vivostar commented on pull request #973: BIGTOP-3767. Add Spark Thrift Server Puppet Deploy Code
Posted by GitBox <gi...@apache.org>.
vivostar commented on PR #973:
URL: https://github.com/apache/bigtop/pull/973#issuecomment-1206003910
After running spark smoke test, the thrift server was killed(maybe we should use other database for hive metastore rather than defalt deby). But there is no such log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] vivostar commented on pull request #973: BIGTOP-3767. Add Spark Thrift Server Puppet Deploy Code
Posted by GitBox <gi...@apache.org>.
vivostar commented on PR #973:
URL: https://github.com/apache/bigtop/pull/973#issuecomment-1206775050
According to the dicussion above, I update the pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #973: BIGTOP-3767. Add Spark Thrift Server Puppet Deploy Code
Posted by GitBox <gi...@apache.org>.
iwasakims commented on PR #973:
URL: https://github.com/apache/bigtop/pull/973#issuecomment-1203493503
> /var/log/spark/spark-thriftserver.out
The log destination should be /var/log/spark/spark-thriftserver.log rather than spark-thriftserver.out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] vivostar commented on pull request #973: BIGTOP-3767. Add Spark Thrift Server Puppet Deploy Code
Posted by GitBox <gi...@apache.org>.
vivostar commented on PR #973:
URL: https://github.com/apache/bigtop/pull/973#issuecomment-1210022987
tested on centos-7 x86_64(you can define spark thrift server port by editting the site.yaml)
```shell
./docker-hadoop.sh \
-d \
--create 2 \
--image bigtop/puppet:trunk-centos-7 \
--memory 8g \
--repo http://repos.bigtop.apache.org/releases/3.1.0/centos/7/\$basearch \
--disable-gpg-check \
--stack hdfs,yarn,mapreduce,spark,hive
[root@37a8654d8c5f /]# netstat -an | grep 12000
tcp 0 0 0.0.0.0:12000 0.0.0.0:* LISTEN
[root@37a8654d8c5f /]# beeline
beeline> !connect jdbc:hive2://localhost:12000
0: jdbc:hive2://localhost:12000> show databases;
+------------+
| namespace |
+------------+
| default |
+------------+
1 row selected (0.666 seconds)
0: jdbc:hive2://localhost:12000> create database tmp;
+---------+
| Result |
+---------+
+---------+
No rows selected (0.284 seconds)
0: jdbc:hive2://localhost:12000> create table tmp.person(id string, name string);
+---------+
| Result |
+---------+
+---------+
No rows selected (0.442 seconds)
0: jdbc:hive2://localhost:12000> insert overwrite table tmp.person select '001', 'Don';
+---------+
| Result |
+---------+
+---------+
No rows selected (2.69 seconds)
0: jdbc:hive2://localhost:12000> select * from tmp.person;
+------+-------+
| id | name |
+------+-------+
| 001 | Don |
+------+-------+
1 row selected (1.201 seconds)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org