You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by GitBox <gi...@apache.org> on 2020/04/22 13:49:10 UTC
[GitHub] [kylin] hit-lacus opened a new pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
hit-lacus opened a new pull request #1182:
URL: https://github.com/apache/kylin/pull/1182
## Proposed changes
Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue.
## Types of changes
What types of changes does your code introduce to Kylin?
_Put an `x` in the boxes that apply_
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] Documentation Update (if none of the other choices apply)
## Checklist
_Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._
- [ ] I have create an issue on [Kylin's jira](https://issues.apache.org/jira/browse/KYLIN), and have described the bug/feature there in detail
- [ ] Commit messages in my PR start with the related jira ID, like "KYLIN-0000 Make Kylin project open-source"
- [ ] Compiling and unit tests pass locally with my changes
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] If this change need a document change, I will prepare another pr against the `document` branch
- [ ] Any dependent changes have been merged
## Further comments
If this is a relatively large or complex change, kick off the discussion at user@kylin or dev@kylin by explaining why you chose the solution you did and what alternatives you considered, etc...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619862739
## Scale Out/In Job Service
- https://kubernetes.io/docs/tasks/run-application/scale-stateful-set/
- Modify replica(from 1 to 3)
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# vim kylin-job/kylin-job-statefulset.yaml
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl apply -f kylin-job/
```
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get pod -n kylin-example
NAME READY STATUS RESTARTS AGE
kylin-job-0 2/2 Running 0 27s
kylin-job-1 2/2 Running 0 69s
kylin-job-2 2/2 Running 0 67s
kylin-memcached-0 1/1 Running 0 131m
kylin-query-0 2/2 Running 0 66m
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get pod -n kylin-example
NAME READY STATUS RESTARTS AGE
kylin-job-0 2/2 Running 0 34s
kylin-job-1 2/2 Running 0 76s
kylin-job-2 2/2 Running 0 74s
kylin-memcached-0 1/1 Running 0 131m
kylin-query-0 2/2 Running 0 66m
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get statefulset -n kylin-example
NAME READY AGE
kylin-job 3/3 115m
kylin-memcached 1/1 131m
kylin-query 1/1 66m
```
<img width="1189" alt="image" src="https://user-images.githubusercontent.com/14030549/80381674-dcaca380-88d3-11ea-9b43-96ffc452b319.png">
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619861990
## Create Kylin Job Service
- Change configuration of memcached related(part-1)
```shell
# kylin.cache.memcached.hosts=cache-svc.kylin-example.svc.cluster.local:11211
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# cd ../config/
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/config# vim kylin-job/kylin.properties
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/config# vim kylin-query/kylin.properties
```
- Change configuration of memcached related(part-2)
```
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/config# vim tomcat/context.xml
```
```xml
<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
memcachedNodes="n1:cache-svc.kylin-example.svc.cluster.local:11211"
storageKeyPrefix="context"
requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
/>
```
- Create related configmap
```
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example hadoop-config \
> --from-file=../config/hadoop/core-site.xml \
> --from-file=../config/hadoop/hdfs-site.xml \
> --from-file=../config/hadoop/yarn-site.xml \
> --from-file=../config/hadoop/mapred-site.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:28.055160 4398 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/hadoop-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example hive-config \
> --from-file=../config/hadoop/hive-site.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:35.449192 4525 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/hive-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example hbase-config \
> --from-file=../config/hadoop/hbase-site.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:36.222142 4583 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/hbase-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example kylin-more-config \
> --from-file=../config/kylin-more/applicationContext.xml \
> --from-file=../config/kylin-more/ehcache.xml \
> --from-file=../config/kylin-more/ehcache-test.xml \
> --from-file=../config/kylin-more/kylinMetrics.xml \
> --from-file=../config/kylin-more/kylinSecurity.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:43.678445 4720 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/kylin-more-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example filebeat-config \
> --from-file=../config/filebeat/filebeat.yml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:51.686495 4901 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/filebeat-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example tomcat-config \
> --from-file=../config/tomcat/server.xml \
> --from-file=../config/tomcat/context.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:52.431775 4926 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/tomcat-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example kylin-job-config \
> --from-file=../config/kylin-job/kylin-kafka-consumer.xml \
> --from-file=../config/kylin-job/kylin_hive_conf.xml \
> --from-file=../config/kylin-job/kylin_job_conf.xml \
> --from-file=../config/kylin-job/kylin_job_conf_inmem.xml \
> --from-file=../config/kylin-job/kylin-server-log4j.properties \
> --from-file=../config/kylin-job/kylin-spark-log4j.properties \
> --from-file=../config/kylin-job/kylin-tools-log4j.properties \
> --from-file=../config/kylin-job/kylin.properties \
> --from-file=../config/kylin-job/setenv.sh \
> --from-file=../config/kylin-job/setenv-tool.sh \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:10:03.250415 5141 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/kylin-job-config configured
```
- Start job service
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl apply -f kylin-job
service/kylin-job-svc created
statefulset.apps/kylin-job created
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get statefulset -n kylin-example
NAME READY AGE
kylin-job 1/1 12s
kylin-memcached 1/1 16m
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get pod -n kylin-example
NAME READY STATUS RESTARTS AGE
kylin-job-0 2/2 Running 0 23s
kylin-memcached-0 1/1 Running 0 16m
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# docker ps | grep kylin-job-
11517f6f98d6 a9acac7280fe "/usr/local/bin/dock…" 2 minutes ago Up 2 minutes k8s_filebeat_kylin-job-0_kylin-example_284fa3ea-5bce-437e-b769-92d6c938c628_0
fc0095a9f2d3 7ea27faa653f "sh -c 'cp $KYLIN_HO…" 2 minutes ago Up 2 minutes k8s_kylin_kylin-job-0_kylin-example_284fa3ea-5bce-437e-b769-92d6c938c628_0
58aa2566d7fb k8s.gcr.io/pause:3.2 "/pause" 2 minutes ago Up 2 minutes k8s_POD_kylin-job-0_kylin-example_284fa3ea-5bce-437e-b769-92d6c938c628_0
```
- Visit Web UI
http://cloud:30010/kylin/models
- Attach pod if you need check failure reason
```shell
root@open-source:/home/ubuntu/example/deployment# kubectl exec -it kylin-job-0 -n kylin-example -- bash
Defaulting container name to kylin.
Use 'kubectl describe pod/kylin-job-0 -n kylin-example' to see all of the containers in this pod.
[root@kylin-job-0 tmp]# cd $KYLIN_HOME/logs
[root@kylin-job-0 logs]# ll
total 196
-rw-r--r-- 1 root root 1875 Apr 27 12:13 kylin.gc.6188.0.current
-rw-r--r-- 1 root root 4794 Apr 27 12:21 kylin.gc.pid6427.0.current
-rw-r--r-- 1 root root 179233 Apr 27 12:22 kylin.log
-rw-r--r-- 1 root root 4651 Apr 27 12:14 kylin.out
[root@kylin-job-0 logs]# tail -f kylin.*
==> kylin.gc.6188.0.current <==
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus commented on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus commented on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619862166
## Create Kylin Query Service
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619861990
## Create Kylin Job Service
- Change configuration of memcached related(part-1)
```shell
# kylin.cache.memcached.hosts=cache-svc.kylin-example.svc.cluster.local:11211
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# cd ../config/
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/config# vim kylin-job/kylin.properties
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/config# vim kylin-query/kylin.properties
```
- Change configuration of memcached related(part-2)
```
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/config# vim tomcat/context.xml
```
```xml
<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
memcachedNodes="n1:cache-svc.kylin-example.svc.cluster.local:11211"
storageKeyPrefix="context"
requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
/>
```
- Create related configmap
```
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example hadoop-config \
> --from-file=../config/hadoop/core-site.xml \
> --from-file=../config/hadoop/hdfs-site.xml \
> --from-file=../config/hadoop/yarn-site.xml \
> --from-file=../config/hadoop/mapred-site.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:28.055160 4398 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/hadoop-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example hive-config \
> --from-file=../config/hadoop/hive-site.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:35.449192 4525 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/hive-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example hbase-config \
> --from-file=../config/hadoop/hbase-site.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:36.222142 4583 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/hbase-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example kylin-more-config \
> --from-file=../config/kylin-more/applicationContext.xml \
> --from-file=../config/kylin-more/ehcache.xml \
> --from-file=../config/kylin-more/ehcache-test.xml \
> --from-file=../config/kylin-more/kylinMetrics.xml \
> --from-file=../config/kylin-more/kylinSecurity.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:43.678445 4720 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/kylin-more-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example filebeat-config \
> --from-file=../config/filebeat/filebeat.yml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:51.686495 4901 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/filebeat-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example tomcat-config \
> --from-file=../config/tomcat/server.xml \
> --from-file=../config/tomcat/context.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:52.431775 4926 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/tomcat-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example kylin-job-config \
> --from-file=../config/kylin-job/kylin-kafka-consumer.xml \
> --from-file=../config/kylin-job/kylin_hive_conf.xml \
> --from-file=../config/kylin-job/kylin_job_conf.xml \
> --from-file=../config/kylin-job/kylin_job_conf_inmem.xml \
> --from-file=../config/kylin-job/kylin-server-log4j.properties \
> --from-file=../config/kylin-job/kylin-spark-log4j.properties \
> --from-file=../config/kylin-job/kylin-tools-log4j.properties \
> --from-file=../config/kylin-job/kylin.properties \
> --from-file=../config/kylin-job/setenv.sh \
> --from-file=../config/kylin-job/setenv-tool.sh \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:10:03.250415 5141 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/kylin-job-config configured
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619833208
## Check Env
#### Hadoop Env
```shell
[root@cdh-master ~]# hadoop version
Hadoop 2.6.0-cdh5.7.6
Subversion http://github.com/cloudera/hadoop -r a3edabab129caaada2958ab32061aa0e52a3f7df
Compiled by jenkins on 2017-02-21T23:08Z
Compiled with protoc 2.5.0
From source with checksum 3de3a6f2894b326919cd7a9b4e573e
This command was run using /opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/jars/hadoop-common-2.6.0-cdh5.7.6.jar
```
#### Elasticsearch Env
```shell
[root@cdh-master elasticsearch-7.6.0]# bin/elasticsearch -V
future versions of Elasticsearch will require Java 11; your Java version from [/usr/java/jdk1.8.0_171/jre] does not meet this requirement
Version: 7.6.0, Build: default/tar/7f634e9f44834fbc12724506cc1da681b0c3b1e3/2020-02-06T00:09:00.449973Z, JVM: 1.8.0_171
```
#### Kubernetes Env
```shell
root@open-source:/home/ubuntu# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.4 LTS
Release: 18.04
Codename: bionic
root@open-source:/home/ubuntu# kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.1", GitCommit:"7879fc12a63337efff607952a323df90cdc7a335", GitTreeState:"clean", BuildDate:"2020-04-08T17:38:50Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.1", GitCommit:"7879fc12a63337efff607952a323df90cdc7a335", GitTreeState:"clean", BuildDate:"2020-04-08T17:30:47Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
root@open-source:/home/ubuntu# free -h
total used free shared buff/cache available
Mem: 19G 3.6G 11G 2.2M 4.5G 15G
Swap: 0B 0B 0B
root@open-source:/home/ubuntu# cat /proc/cpuinfo| grep "cpu cores"| uniq
cpu cores : 8
root@open-source:/home/ubuntu# uname -a
Linux open-source 4.15.0-96-generic #97-Ubuntu SMP Wed Apr 1 03:25:46 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
root@open-source:/home/ubuntu# kubectl get node -o=wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
open-source Ready master 11d v1.18.1 10.1.2.41 <none> Ubuntu 18.04.4 LTS 4.15.0-96-generic docker://19.3.8
root@open-source:/home/ubuntu# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-555fc8cc5c-jbt4h 1/1 Running 1 11d
calico-node-xzrbs 1/1 Running 1 11d
coredns-66bff467f8-q5j2q 1/1 Running 1 11d
coredns-66bff467f8-sxxfx 1/1 Running 1 11d
etcd-open-source 1/1 Running 1 11d
kube-apiserver-open-source 1/1 Running 1 11d
kube-controller-manager-open-source 1/1 Running 3 11d
kube-proxy-mnzq4 1/1 Running 1 11d
kube-scheduler-open-source 1/1 Running 3 11d
root@open-source:/home/ubuntu# kubectl cluster-info
Kubernetes master is running at https://10.1.2.41:6443
KubeDNS is running at https://10.1.2.41:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] coveralls edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-617824003
## Pull Request Test Coverage Report for [Build 5857](https://coveralls.io/builds/30367072)
* **0** of **0** changed or added relevant lines in **0** files are covered.
* **7** unchanged lines in **2** files lost coverage.
* Overall coverage remained the same at **27.466%**
---
| Files with Coverage Reduction | New Missed Lines | % |
| :-----|--------------|--: |
| [core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DefaultScheduler.java](https://coveralls.io/builds/30367072/source?filename=core-job%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fjob%2Fimpl%2Fthreadpool%2FDefaultScheduler.java#L194) | 2 | 80.23% |
| [core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/30367072/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L439) | 5 | 77.81% |
<!-- | **Total:** | **7** | | -->
| Totals | [](https://coveralls.io/builds/30367072) |
| :-- | --: |
| Change from base [Build 5853](https://coveralls.io/builds/30348545): | 0.0% |
| Covered Lines: | 24316 |
| Relevant Lines: | 88532 |
---
##### 💛 - [Coveralls](https://coveralls.io)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619833363
## Prepare Docker Image
- Downloads required binary
```shell
xiaoxiang.yu:~/ $ ll Downloads/apache-kylin-3.0.1-bin-cdh57.tar [18:39:26]
-rw-r--r--@ 1 xiaoxiang.yu INC\Domain Users 253M 4 15 16:49 Downloads/apache-kylin-3.0.1-bin-cdh57.tar
xiaoxiang.yu:~/ $ ll Downloads/spark-2.3.2-bin-hadoop2.7.tgz [18:39:52]
-rwxrwxrwx 1 xiaoxiang.yu INC\Domain Users 215M 3 23 01:36 Downloads/spark-2.3.2-bin-hadoop2.7.tgz
```
- Downloads required jars
```shell
xiaoxiang.yu:~/ $ ll Downloads/mem* [18:40:05]
-rw-r--r--@ 1 xiaoxiang.yu INC\Domain Users 163K 4 27 18:45 Downloads/memcached-session-manager-2.1.1.jar
-rw-r--r--@ 1 xiaoxiang.yu INC\Domain Users 11K 4 27 18:45 Downloads/memcached-session-manager-tc7-2.1.1.jar
```
- Remove old image
```shell
xiaoxiang.yu:CDH57/ (kubernetes*) $ docker rmi hadoop-client:cdh57 [18:46:33]
Untagged: hadoop-client:cdh57
xiaoxiang.yu:CDH57/ (kubernetes*) $ docker images |grep hadoop-client | grep wc -l [18:47:31]
xiaoxiang.yu:CDH57/ (kubernetes*) $
xiaoxiang.yu:kylin-client/ (kubernetes*) $ docker rmi kylin-client:3.0.1-cdh57 [18:51:58]
Untagged: kylin-client:3.0.1-cdh57
Deleted: sha256:eb16424fc7cae2e4ca105e778290feec0ef370da2cc19e7df8b37ddd7166de57
Deleted: sha256:5c5d43dc5f482ea5e60f58902f577bbfe3bc22b37175d6a5c28b3f66646f9626
Deleted: sha256:9e509fdbf7d47b4cc28c45060809fecb888fab4c79b94f27f6826e78ed012431
Deleted: sha256:3c97b6cb2781d7a594f18cccf357ce90214d7e1317fabf28edfccf45896af502
Deleted: sha256:64b5601d1cb411c481382e6fe022f18dda4c5d070476a922bc4fd7531bad0846
Deleted: sha256:fe814862cb919a47a9c162506921c0ee93edca19c5d36006f92153487c4429ef
Deleted: sha256:1badb3b00a3f56ba66627e8c802b0a8f6a394bc2bc0dfa04d15a3085de9e34b9
Deleted: sha256:afdb1ccabfabd7f375a184c6a8aa3b0aeb54195e92fd67d5a225a9e6fc501469
Deleted: sha256:515eaf5fcbabf292b0e39d7fffd2d43f6c02d4b13af4f99be79027d120ce792d
Deleted: sha256:f88e58969b5f908502fbddf2e058a97d31ed31134c8020b0762eea1224a86383
Deleted: sha256:24b707ed4fe0ac52d4f3b32f9eac13fcd3698071a521f02ba1494d8cd7ec7646
Deleted: sha256:664b07d4d73e4a1b587983f0b80fd7500d93f1c1eedbf925adf5b94391583389
Deleted: sha256:75cdc77ae2aa8a0be849cd09961fc80a4d3856964fe91e265da2580b2d7dbd10
Deleted: sha256:78e1a1873e763dd81d65d0249869d546e39806fc9ae415d508bff7fac3a1c1eb
```
- Build Hadoop-Client image
```shell
xiaoxiang.yu:CDH57/ (kubernetes*) $ pwd [18:50:57]
/Users/xiaoxiang.yu/IntelliJ_IDEA_Project/MyKylin/kubernetes/docker/hadoop-client/CDH57
xiaoxiang.yu:CDH57/ (kubernetes*) $ cp ~/Downloads/spark-2.3.2-bin-hadoop2.7.tgz provided-binary [18:48:25]
xiaoxiang.yu:CDH57/ (kubernetes*) $ [18:48:37]
xiaoxiang.yu:CDH57/ (kubernetes*) $ [18:49:02]
xiaoxiang.yu:CDH57/ (kubernetes*) $ sh build-image.sh [18:49:02]
Sending build context to Docker daemon 225.9MB
Step 1/26 : FROM centos:7.3.1611
---> c5d48e81b986
... ... ...
Step 26/26 : ADD provided-binary/spark-${SPARK_VERSION}-bin-hadoop2.7.tgz .
---> ee576395a7f4
Successfully built ee576395a7f4
Successfully tagged hadoop-client:cdh57
```
- Build kylin-client image
```shell
xiaoxiang.yu:kylin-client/ (kubernetes*) $ cp /Users/xiaoxiang.yu/Downloads/memcached-session-manager-2.1.1.jar apache-kylin-3.0.1-bin-cdh57/tomcat/lib
xiaoxiang.yu:kylin-client/ (kubernetes*) $ cp /Users/xiaoxiang.yu/Downloads/memcached-session-manager-tc7-2.1.1.jar apache-kylin-3.0.1-bin-cdh57/tomcat/lib
xiaoxiang.yu:kylin-client/ (kubernetes*) $ sh build-image.sh [19:06:35]
Sending build context to Docker daemon 531.4MB
Step 1/23 : FROM hadoop-client:cdh57
---> ee576395a7f4
Step 2/23 : MAINTAINER Apache Kylin
---> Using cache
---> bd37b4be904f
Step 3/23 : ARG USER=apache_kylin
---> Using cache
---> 7e860f60cd6b
Step 4/23 : ARG USER_HOME=/home/${USER}
---> Using cache
---> 81d3ff20f301
Step 5/23 : ARG KYLIN_VERSION=apache-kylin-3.0.1-bin-cdh57
---> Using cache
---> 9d9622c9fab7
Step 6/23 : ARG HADOOP_CONF_HOME=/etc/hadoop/conf
---> Using cache
---> 7fa3676c6199
Step 7/23 : ARG HIVE_CONF_HOME=/etc/hive/conf
---> Using cache
---> 9e4679f4f582
Step 8/23 : ARG HBASE_CONF_HOME=/etc/hbase/conf
---> Using cache
---> 491b4dbe0cec
Step 9/23 : ARG SPARK_CONF_HOME=$KYLIN_HOME/hadoop-conf
---> Using cache
---> 1c2e39c268f4
Step 10/23 : ENV SPARK_HOME /opt/spark-2.3.2-bin-hadoop2.7
---> Using cache
---> db65c266f986
Step 11/23 : ENV KYLIN_HOME=$USER_HOME/kylin
---> Using cache
---> f8a17081a4a5
Step 12/23 : WORKDIR /tmp
---> Using cache
---> 5af436187fd4
Step 13/23 : RUN set -x && yum install -y openssh-clients cronie unzip sudo net-tools iftop tcpdump perf telnet bind-utils && yum clean all
---> Using cache
---> ef53f3155c7e
Step 14/23 : RUN set -x && groupadd -r $USER && useradd -r -m -g $USER $USER -d $USER_HOME && echo '$USER ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
---> Using cache
---> 1d0ee15484f7
Step 15/23 : COPY --chown=$USER:$USER $KYLIN_VERSION $KYLIN_HOME
---> Using cache
---> debe0d8920d5
Step 16/23 : RUN set -x && unzip -qq $KYLIN_HOME/tomcat/webapps/kylin.war -d $KYLIN_HOME/tomcat/webapps/kylin && chown -R $USER:$USER $KYLIN_HOME/tomcat/webapps/kylin && rm $KYLIN_HOME/tomcat/webapps/kylin.war && mkdir $SPARK_CONF_HOME && ln -s $HADOOP_CONF_HOME/core-site.xml $SPARK_CONF_HOME/core-site.xml && ln -s $HADOOP_CONF_HOME/hdfs-site.xml $SPARK_CONF_HOME/hdfs-site.xml && ln -s $HADOOP_CONF_HOME/yarn-site.xml $SPARK_CONF_HOME/yarn-site.xml && ln -s $HADOOP_CONF_HOME/hdfs-variable.xml $SPARK_CONF_HOME/hdfs-variable.xml && ln -s $HADOOP_CONF_HOME/yarn-variable.xml $SPARK_CONF_HOME/yarn-variable.xml && ln -s $HADOOP_CONF_HOME/federation-mapping.xml $SPARK_CONF_HOME/federation-mapping.xml && ln -s $HIVE_CONF_HOME/hive-site.xml $SPARK_CONF_HOME/hive-site.xml && ln -s $HBASE_CONF_HOME/hbase-site.xml $SPARK_CONF_HOME/hbase-site.xml && chown -R $USER:$USER $SPARK_CONF_HOME
---> Using cache
---> 40913e8eae5c
Step 17/23 : ENV TOOL_HOME=$USER_HOME/bin
---> Using cache
---> 889756023204
Step 18/23 : COPY bin $TOOL_HOME
---> Using cache
---> 01e1041cc7cb
Step 19/23 : COPY crontab.txt /tmp/crontab.txt
---> 9e084a6fd4b5
Step 20/23 : RUN /usr/bin/crontab -u $USER /tmp/crontab.txt && rm -rf /tmp/* && chmod 755 $TOOL_HOME/*
---> Running in 023036a3dff3
Removing intermediate container 023036a3dff3
---> 2279b966c55b
Step 21/23 : EXPOSE 7070
---> Running in ee0cc6d75bc6
Removing intermediate container ee0cc6d75bc6
---> 13400f20c95b
Step 22/23 : USER root
---> Running in 57b588a270c2
Removing intermediate container 57b588a270c2
---> 6d41c69a36a4
Step 23/23 : CMD ["sh", "-c", "$TOOL_HOME/bootstrap.sh server -d"]
---> Running in c786eea2ce0f
Removing intermediate container c786eea2ce0f
---> 7ea27faa653f
Successfully built 7ea27faa653f
Successfully tagged kylin-client:3.0.1-cdh57
```
- Load docker image
```shell
root@open-source:/home/ubuntu# docker rmi kylin-client:3.0.1-cdh57
Untagged: kylin-client:3.0.1-cdh57
Deleted: sha256:7ea27faa653fe85616092245d090f7bab3fb9667415992b8cf9ce7c073869d12
Deleted: sha256:186299dc4f8a000fcb6ace77700402a25aba127c20a97d830d7f7774fb7f7acf
Deleted: sha256:9d8a3a9b3a48dd060d8776996f05b4ff53e3931af4a1c982496ef905a8f1d199
Deleted: sha256:06930b02da59551eb04af6f19440f6634ab8e702685d39db52853ccd9c2e7895
Deleted: sha256:120143be03767e0dcf8afdd6ed2c6dde4384abf88e660e8ce3112bc2c2c35e65
Deleted: sha256:6a793a7d385ad6fb128035a74802be0b9872926717b0f1acec4d41d687d1f7fa
Deleted: sha256:80e715ae1252910da1c6a14909a0934ece73a79af8412e3afebeb22fe774d502
Deleted: sha256:fcca76dbf8b63155051771490d21ccbaf63999fe0e20a12615cb670945a05b6b
Deleted: sha256:b66102dad1d107c4be08fdcb86663acfac34528db6d26a8c9e029f5b9e2b59ba
root@open-source:/home/ubuntu# docker load --input kylin-client-3.0.1-cdh57.tar
3b0f0cce5376: Loading layer [==================================================>] 256.6MB/256.6MB
cc5bfaff0dfd: Loading layer [==================================================>] 86.18MB/86.18MB
a66a1b16493a: Loading layer [==================================================>] 316.9kB/316.9kB
9e92f4570e21: Loading layer [==================================================>] 265.8MB/265.8MB
739690f8c9bc: Loading layer [==================================================>] 102.2MB/102.2MB
cf03738f4a83: Loading layer [==================================================>] 10.24kB/10.24kB
560aa175075b: Loading layer [==================================================>] 2.56kB/2.56kB
a03bb6647762: Loading layer [==================================================>] 13.82kB/13.82kB
Loaded image: kylin-client:3.0.1-cdh57
root@open-source:/home/ubuntu#
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619861773
## Create Memcached Service
- Create namespace
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create namespace kylin-example
Error from server (AlreadyExists): namespaces "kylin-example" already exists
```
- Create memcached service
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl apply -f memcached
service/cache-svc created
statefulset.apps/kylin-memcached created
```
- Check service and pod state
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get pod -n kylin-example
NAME READY STATUS RESTARTS AGE
kylin-memcached-0 1/1 Running 0 60s
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get svc -n kylin-example
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cache-svc ClusterIP None <none> 11211/TCP 69s
```
- Check hostname and port listen
```
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl run -it --image=busybox:1.28.4 --rm --restart=Never sh -n test-dnsIf you don't see a command prompt, try pressing enter.
/ # nslookup cache-svc.kylin-example.svc.cluster.local
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: cache-svc.kylin-example.svc.cluster.local
Address 1: 192.168.11.9 kylin-memcached-0.cache-svc.kylin-example.svc.cluster.local
/ # telnet cache-svc.kylin-example.svc.cluster.local 11211
^C
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619833363
## Prepare Docker Image
- Downloads required binary
```shell
xiaoxiang.yu:~/ $ ll Downloads/apache-kylin-3.0.1-bin-cdh57.tar [18:39:26]
-rw-r--r--@ 1 xiaoxiang.yu INC\Domain Users 253M 4 15 16:49 Downloads/apache-kylin-3.0.1-bin-cdh57.tar
xiaoxiang.yu:~/ $ ll Downloads/spark-2.3.2-bin-hadoop2.7.tgz [18:39:52]
-rwxrwxrwx 1 xiaoxiang.yu INC\Domain Users 215M 3 23 01:36 Downloads/spark-2.3.2-bin-hadoop2.7.tgz
```
- Downloads required jars
```shell
xiaoxiang.yu:~/ $ ll Downloads/mem* [18:40:05]
-rw-r--r--@ 1 xiaoxiang.yu INC\Domain Users 163K 4 27 18:45 Downloads/memcached-session-manager-2.1.1.jar
-rw-r--r--@ 1 xiaoxiang.yu INC\Domain Users 11K 4 27 18:45 Downloads/memcached-session-manager-tc7-2.1.1.jar
```
- Remove old image
```shell
xiaoxiang.yu:CDH57/ (kubernetes*) $ docker rmi hadoop-client:cdh57 [18:46:33]
Untagged: hadoop-client:cdh57
xiaoxiang.yu:CDH57/ (kubernetes*) $ docker images |grep hadoop-client | grep wc -l [18:47:31]
xiaoxiang.yu:CDH57/ (kubernetes*) $
```
- Build Hadoop-Client image
```shell
xiaoxiang.yu:CDH57/ (kubernetes*) $ pwd [18:50:57]
/Users/xiaoxiang.yu/IntelliJ_IDEA_Project/MyKylin/kubernetes/docker/hadoop-client/CDH57
xiaoxiang.yu:CDH57/ (kubernetes*) $ cp ~/Downloads/spark-2.3.2-bin-hadoop2.7.tgz provided-binary [18:48:25]
xiaoxiang.yu:CDH57/ (kubernetes*) $ [18:48:37]
xiaoxiang.yu:CDH57/ (kubernetes*) $ [18:49:02]
xiaoxiang.yu:CDH57/ (kubernetes*) $ sh build-image.sh [18:49:02]
Sending build context to Docker daemon 225.9MB
Step 1/26 : FROM centos:7.3.1611
---> c5d48e81b986
Step 2/26 : MAINTAINER Apache Kylin
---> Using cache
---> 7d31909dd19a
Step 3/26 : ENV JAVA_VERSION 1.8.0
---> Using cache
---> 596fd0b28b00
Step 4/26 : ENV SPARK_VERSION 2.3.2
---> Using cache
---> ec0263221e5b
Step 5/26 : ENV JAVA_HOME /usr/lib/jvm/java-${JAVA_VERSION}
---> Using cache
---> fcb50630b39a
Step 6/26 : ENV HADOOP_HOME /usr/lib/hadoop
---> Using cache
---> 729ba7b92158
Step 7/26 : ENV HIVE_HOME /usr/lib/hive
---> Using cache
---> 720cd72e9904
Step 8/26 : ENV HCAT_HOME /usr/lib/hive-hcatalog
---> Using cache
---> 7ef4bfdbc4cd
Step 9/26 : ENV HBASE_HOME /usr/lib/hbase
---> Using cache
---> 818e685192a5
Step 10/26 : ENV SPARK_HOME /opt/spark-${SPARK_VERSION}-bin-hadoop2.7
---> Using cache
---> ec3429565158
Step 11/26 : ENV PATH $PATH:$SPARK_HOME/bin:
---> Using cache
---> f2b052643286
Step 12/26 : ENV HADOOP_CONF_DIR /etc/hadoop/conf
---> Using cache
---> 0d76e82d405c
Step 13/26 : ENV HIVE_CONF_DIR /etc/hive/conf
---> Using cache
---> 1ad66c55a1de
Step 14/26 : ENV HBASE_CONF_DIR /etc/hbase/conf
---> Using cache
---> 91e260f7d120
Step 15/26 : ENV HIVE_CONF ${HIVE_CONF_DIR}
---> Using cache
---> 4321f4fe59f4
Step 16/26 : ENV HIVE_LIB ${HIVE_HOME}/lib
---> Using cache
---> 6b6b4e2a5ff4
Step 17/26 : RUN echo $'[cloudera-cdh5] \nname=Cloudera\'s Distribution for Hadoop, Version 5 \nbaseurl=https://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.7.6/ \ngpgkey =https://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/RPM-GPG-KEY-cloudera \ngpgcheck = 1' > /etc/yum.repos.d/cloudera-cdh5.repo
---> Using cache
---> ca7957895170
Step 18/26 : WORKDIR /opt
---> Using cache
---> c85839b823ef
Step 19/26 : RUN set -xeu && yum -y install java-1.8.0-openjdk-devel
---> Using cache
---> 70de99c7c287
Step 20/26 : RUN set -xeu && yum -y install krb5-workstation
---> Using cache
---> ee2d6cf2c56c
Step 21/26 : RUN set -xeu && yum -y install hadoop-client
---> Using cache
---> 7c332c100a62
Step 22/26 : RUN set -xeu && yum -y install hive hive-hcatalog
---> Using cache
---> 54d0c23e2781
Step 23/26 : RUN set -xeu && yum -y install hbase
---> Using cache
---> 57aa98165f79
Step 24/26 : RUN set -xeu && curl -o ${HIVE_HOME}/lib/hadoop-lzo-0.4.15.jar https://clojars.org/repo/hadoop-lzo/hadoop-lzo/0.4.15/hadoop-lzo-0.4.15.jar && curl -o ${HIVE_HOME}/lib/mysql-connector-java-5.1.24.jar https://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.24/mysql-connector-java-5.1.24.jar
---> Using cache
---> c9494c618436
Step 25/26 : RUN set -xeu && yum -q clean all && rm -rf /var/cache/yum && rm -rf /tmp/* /var/tmp/*
---> Using cache
---> 4a2d6490c98d
Step 26/26 : ADD provided-binary/spark-${SPARK_VERSION}-bin-hadoop2.7.tgz .
---> ee576395a7f4
Successfully built ee576395a7f4
Successfully tagged hadoop-client:cdh57
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619861990
## Create Kylin Job Service
- Change configuration of memcached related(part-1)
```shell
# kylin.cache.memcached.hosts=cache-svc.kylin-example.svc.cluster.local:11211
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# cd ../config/
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/config# vim kylin-job/kylin.properties
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/config# vim kylin-query/kylin.properties
```
- Change configuration of memcached related(part-2)
```
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/config# vim tomcat/context.xml
```
```xml
<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
memcachedNodes="n1:cache-svc.kylin-example.svc.cluster.local:11211"
storageKeyPrefix="context"
requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
/>
```
- Create related configmap
```
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example hadoop-config \
> --from-file=../config/hadoop/core-site.xml \
> --from-file=../config/hadoop/hdfs-site.xml \
> --from-file=../config/hadoop/yarn-site.xml \
> --from-file=../config/hadoop/mapred-site.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:28.055160 4398 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/hadoop-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example hive-config \
> --from-file=../config/hadoop/hive-site.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:35.449192 4525 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/hive-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example hbase-config \
> --from-file=../config/hadoop/hbase-site.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:36.222142 4583 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/hbase-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example kylin-more-config \
> --from-file=../config/kylin-more/applicationContext.xml \
> --from-file=../config/kylin-more/ehcache.xml \
> --from-file=../config/kylin-more/ehcache-test.xml \
> --from-file=../config/kylin-more/kylinMetrics.xml \
> --from-file=../config/kylin-more/kylinSecurity.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:43.678445 4720 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/kylin-more-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example filebeat-config \
> --from-file=../config/filebeat/filebeat.yml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:51.686495 4901 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/filebeat-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example tomcat-config \
> --from-file=../config/tomcat/server.xml \
> --from-file=../config/tomcat/context.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:52.431775 4926 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/tomcat-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example kylin-job-config \
> --from-file=../config/kylin-job/kylin-kafka-consumer.xml \
> --from-file=../config/kylin-job/kylin_hive_conf.xml \
> --from-file=../config/kylin-job/kylin_job_conf.xml \
> --from-file=../config/kylin-job/kylin_job_conf_inmem.xml \
> --from-file=../config/kylin-job/kylin-server-log4j.properties \
> --from-file=../config/kylin-job/kylin-spark-log4j.properties \
> --from-file=../config/kylin-job/kylin-tools-log4j.properties \
> --from-file=../config/kylin-job/kylin.properties \
> --from-file=../config/kylin-job/setenv.sh \
> --from-file=../config/kylin-job/setenv-tool.sh \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:10:03.250415 5141 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/kylin-job-config configured
```
- Start job service
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl apply -f kylin-job
service/kylin-job-svc created
statefulset.apps/kylin-job created
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get statefulset -n kylin-example
NAME READY AGE
kylin-job 1/1 12s
kylin-memcached 1/1 16m
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get pod -n kylin-example
NAME READY STATUS RESTARTS AGE
kylin-job-0 2/2 Running 0 23s
kylin-memcached-0 1/1 Running 0 16m
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# docker ps | grep kylin-job-
11517f6f98d6 a9acac7280fe "/usr/local/bin/dock…" 2 minutes ago Up 2 minutes k8s_filebeat_kylin-job-0_kylin-example_284fa3ea-5bce-437e-b769-92d6c938c628_0
fc0095a9f2d3 7ea27faa653f "sh -c 'cp $KYLIN_HO…" 2 minutes ago Up 2 minutes k8s_kylin_kylin-job-0_kylin-example_284fa3ea-5bce-437e-b769-92d6c938c628_0
58aa2566d7fb k8s.gcr.io/pause:3.2 "/pause" 2 minutes ago Up 2 minutes k8s_POD_kylin-job-0_kylin-example_284fa3ea-5bce-437e-b769-92d6c938c628_0
```
- Visit Web UI
http://cloud:30010/kylin/models
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] coveralls edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-617824003
## Pull Request Test Coverage Report for [Build 5855](https://coveralls.io/builds/30361859)
* **0** of **0** changed or added relevant lines in **0** files are covered.
* **3** unchanged lines in **3** files lost coverage.
* Overall coverage remained the same at **27.466%**
---
| Files with Coverage Reduction | New Missed Lines | % |
| :-----|--------------|--: |
| [core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/30361859/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L553) | 1 | 78.12% |
| [server-base/src/main/java/org/apache/kylin/rest/util/QueryRequestLimits.java](https://coveralls.io/builds/30361859/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Futil%2FQueryRequestLimits.java#L72) | 1 | 47.62% |
| [stream-core/src/main/java/org/apache/kylin/stream/core/storage/columnar/ColumnarStoreCache.java](https://coveralls.io/builds/30361859/source?filename=stream-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstream%2Fcore%2Fstorage%2Fcolumnar%2FColumnarStoreCache.java#L146) | 1 | 56.82% |
<!-- | **Total:** | **3** | | -->
| Totals | [](https://coveralls.io/builds/30361859) |
| :-- | --: |
| Change from base [Build 5853](https://coveralls.io/builds/30348545): | 0.0% |
| Covered Lines: | 24316 |
| Relevant Lines: | 88532 |
---
##### 💛 - [Coveralls](https://coveralls.io)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619862275
## Test Kylin Query Service
- Instance List
<img width="1189" alt="image" src="https://user-images.githubusercontent.com/14030549/80373561-d4e70200-88c7-11ea-9387-55f8125c69b9.png">
- Test Query 1
<img width="1052" alt="image" src="https://user-images.githubusercontent.com/14030549/80374180-d8c75400-88c8-11ea-8c15-dccdb335fdb4.png">
- Test Query 2
<img width="1046" alt="image" src="https://user-images.githubusercontent.com/14030549/80380959-d0741680-88d2-11ea-8a94-963e1f0f2d13.png">
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619862739
## Scale Out/In Job Service
- https://kubernetes.io/docs/tasks/run-application/scale-stateful-set/
- Modify replica(from 1 to 3) and scale out job server
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# vim kylin-job/kylin-job-statefulset.yaml
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl apply -f kylin-job/
```
- Check state of pods
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get pod -n kylin-example
NAME READY STATUS RESTARTS AGE
kylin-job-0 2/2 Running 0 27s
kylin-job-1 2/2 Running 0 69s
kylin-job-2 2/2 Running 0 67s
kylin-memcached-0 1/1 Running 0 131m
kylin-query-0 2/2 Running 0 66m
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get pod -n kylin-example
NAME READY STATUS RESTARTS AGE
kylin-job-0 2/2 Running 0 34s
kylin-job-1 2/2 Running 0 76s
kylin-job-2 2/2 Running 0 74s
kylin-memcached-0 1/1 Running 0 131m
kylin-query-0 2/2 Running 0 66m
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get statefulset -n kylin-example
NAME READY AGE
kylin-job 3/3 115m
kylin-memcached 1/1 131m
kylin-query 1/1 66m
root@open-source:/home/ubuntu/example/deployment# docker ps | grep kylin-job-
da9c0cf239e4 a9acac7280fe "/usr/local/bin/dock…" 8 minutes ago Up 8 minutes k8s_filebeat_kylin-job-0_kylin-example_26c08037-0543-4e15-9ac2-54021266a52b_0
06ff2e6a2d3e 7ea27faa653f "sh -c 'cp $KYLIN_HO…" 8 minutes ago Up 8 minutes k8s_kylin_kylin-job-0_kylin-example_26c08037-0543-4e15-9ac2-54021266a52b_0
87062bb80c1b k8s.gcr.io/pause:3.2 "/pause" 8 minutes ago Up 8 minutes k8s_POD_kylin-job-0_kylin-example_26c08037-0543-4e15-9ac2-54021266a52b_0
91d96d1bee0f a9acac7280fe "/usr/local/bin/dock…" 8 minutes ago Up 8 minutes k8s_filebeat_kylin-job-2_kylin-example_93cc142c-0dc8-4e5e-886d-4408fac76c5d_0
d5d352acbb8d 7ea27faa653f "sh -c 'cp $KYLIN_HO…" 8 minutes ago Up 8 minutes k8s_kylin_kylin-job-2_kylin-example_93cc142c-0dc8-4e5e-886d-4408fac76c5d_0
ad05de976d8a k8s.gcr.io/pause:3.2 "/pause" 8 minutes ago Up 8 minutes k8s_POD_kylin-job-2_kylin-example_93cc142c-0dc8-4e5e-886d-4408fac76c5d_0
41de3f26e592 a9acac7280fe "/usr/local/bin/dock…" 8 minutes ago Up 8 minutes k8s_filebeat_kylin-job-1_kylin-example_b83623a1-c8a4-456f-95e3-fce19d78afb1_0
b3666e72bc7b 7ea27faa653f "sh -c 'cp $KYLIN_HO…" 8 minutes ago Up 8 minutes k8s_kylin_kylin-job-1_kylin-example_b83623a1-c8a4-456f-95e3-fce19d78afb1_0
1489fd6e04db k8s.gcr.io/pause:3.2 "/pause" 8 minutes ago Up 8 minutes k8s_POD_kylin-job-1_kylin-example_b83623a1-c8a4-456f-95e3-fce19d78afb1_0
root@open-source:/home/ubuntu/example/deployment#
```
- Instance List
<img width="1189" alt="image" src="https://user-images.githubusercontent.com/14030549/80381674-dcaca380-88d3-11ea-9b43-96ffc452b319.png">
- Test another build
<img width="1375" alt="image" src="https://user-images.githubusercontent.com/14030549/80383050-aa03aa80-88d5-11ea-8424-52825dd0d8cd.png">
- Force kill Active JobServer(Failover Test)
```shell
root@open-source:/home/ubuntu/example/deployment# docker kill b3666e72bc7b
b3666e72bc7b
```
- Check current active job server
<img width="1190" alt="image" src="https://user-images.githubusercontent.com/14030549/80383200-df0ffd00-88d5-11ea-8474-f21de4d0a6e3.png">
- Test a third build
<img width="1376" alt="image" src="https://user-images.githubusercontent.com/14030549/80384382-6447e180-88d7-11ea-9732-263d13d76115.png">
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] sanjulian commented on a change in pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
sanjulian commented on a change in pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#discussion_r419336226
##########
File path: kubernetes/example/deployment/kylin-query/kylin-query-statefulset.yaml
##########
@@ -0,0 +1,143 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+ name: kylin-query
+ namespace: kylin-example
+ labels:
+ app: kylin
+ role: query
+ environment: poc
+ version: 3.0.1
+ hadoop.version: cdh5.7
+spec:
+ serviceName: kylin-query-svc
+ replicas: 1
+ selector:
+ matchLabels:
+ app: kylin
+ role: query
+ template:
+ metadata:
+ labels:
+ app: kylin
+ role: query
+ spec:
+ hostAliases:
+ - ip: "10.1.3.90"
+ hostnames:
+ - "cdh-master"
+ - ip: "10.1.3.93"
+ hostnames:
+ - "cdh-worker-1"
+ - ip: "10.1.3.94"
+ hostnames:
+ - "cdh-worker-2"
+ - ip: "10.1.3.91"
+ hostnames:
+ - "cdh-client"
+ containers:
+ - name: kylin
+ image: kylin-client:3.0.1-cdh57
+ imagePullPolicy: IfNotPresent
+ command:
+ - sh
+ - -c
+ args:
+ - cp $KYLIN_HOME/tomcat-config/* $KYLIN_HOME/tomcat/conf;
+ cp $KYLIN_HOME/kylin-more-config $KYLIN_HOME/tomcat/webapps/kylin/WEB-INF/classes/;
+ $TOOL_HOME/bootstrap.sh server -d;
+ ports:
+ - containerPort: 7070
+ - containerPort: 7443
+ volumeMounts:
+ - name: kylin-query-config
+ mountPath: /home/apache_kylin/kylin/conf
+ - name: tomcat-config
+ mountPath: /home/apache_kylin/kylin/tomcat-config
+ - name: kylin-more-config
+ mountPath: /home/apache_kylin/kylin/kylin-more-config/
+ - name: hadoop-config
+ mountPath: /etc/hadoop/conf
+ - name: hive-config
+ mountPath: /etc/hive/conf
+ - name: hbase-config
+ mountPath: /etc/hbase/conf
+ - name: kylin-logs
+ mountPath: /home/apache_kylin/kylin/logs
+ - name: tomcat-logs
+ mountPath: /home/apache_kylin/kylin/tomcat/logs
+ resources:
+ requests:
+ memory: 2Gi
+ cpu: 700m
+ limits:
+ memory: 2Gi
+ cpu: 700m
+ - name: filebeat
+ image: docker.elastic.co/beats/filebeat:6.4.3
+ args:
+ - -c
+ - /usr/share/filebeat/config/filebeat.yml
+ - -e
+ volumeMounts:
+ - name: kylin-logs
+ mountPath: /home/apache_kylin/kylin/logs
+ - name: tomcat-logs
+ mountPath: /home/apache_kylin/kylin/tomcat/logs
+ - name: filebeat-config
+ mountPath: /usr/share/filebeat/config
+ readOnly: true
+ resources:
+ requests:
+ memory: 1Gi
+ cpu: 300m
+ limits:
+ memory: 1Gi
+ cpu: 300m
+ volumes:
+ - name: kylin-logs
+ emptyDir:
+ sizeLimit: 3Gi
+ - name: tomcat-logs
+ emptyDir:
+ sizeLimit: 2Gi
+ - configMap:
Review comment:
Sorry, my bad. I did not notice the configMap is mounted.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus commented on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus commented on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619861990
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619833208
## Check Env
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619833363
## Prepare Docker Image
- Downloads required binary
```shell
xiaoxiang.yu:~/ $ ll Downloads/apache-kylin-3.0.1-bin-cdh57.tar [18:39:26]
-rw-r--r--@ 1 xiaoxiang.yu INC\Domain Users 253M 4 15 16:49 Downloads/apache-kylin-3.0.1-bin-cdh57.tar
xiaoxiang.yu:~/ $ ll Downloads/spark-2.3.2-bin-hadoop2.7.tgz [18:39:52]
-rwxrwxrwx 1 xiaoxiang.yu INC\Domain Users 215M 3 23 01:36 Downloads/spark-2.3.2-bin-hadoop2.7.tgz
```
- Downloads required jars
```shell
xiaoxiang.yu:~/ $ ll Downloads/mem* [18:40:05]
-rw-r--r--@ 1 xiaoxiang.yu INC\Domain Users 163K 4 27 18:45 Downloads/memcached-session-manager-2.1.1.jar
-rw-r--r--@ 1 xiaoxiang.yu INC\Domain Users 11K 4 27 18:45 Downloads/memcached-session-manager-tc7-2.1.1.jar
```
- Remove old image
```shell
xiaoxiang.yu:CDH57/ (kubernetes*) $ docker rmi hadoop-client:cdh57 [18:46:33]
Untagged: hadoop-client:cdh57
xiaoxiang.yu:CDH57/ (kubernetes*) $ docker images |grep hadoop-client | grep wc -l [18:47:31]
xiaoxiang.yu:CDH57/ (kubernetes*) $
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus commented on a change in pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus commented on a change in pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#discussion_r420218979
##########
File path: kubernetes/example/deployment/kylin-job/kylin-job-service.yaml
##########
@@ -0,0 +1,36 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+apiVersion: v1
+kind: Service
+metadata:
+ name: kylin-job-svc
Review comment:
OK, I finally understand what you mean, thanks for this suggestion, it do make sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus commented on a change in pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus commented on a change in pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#discussion_r419249812
##########
File path: kubernetes/example/deployment/kylin-query/kylin-query-statefulset.yaml
##########
@@ -0,0 +1,143 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+ name: kylin-query
+ namespace: kylin-example
+ labels:
+ app: kylin
+ role: query
+ environment: poc
+ version: 3.0.1
+ hadoop.version: cdh5.7
+spec:
+ serviceName: kylin-query-svc
+ replicas: 1
+ selector:
+ matchLabels:
+ app: kylin
+ role: query
+ template:
+ metadata:
+ labels:
+ app: kylin
+ role: query
+ spec:
+ hostAliases:
+ - ip: "10.1.3.90"
+ hostnames:
+ - "cdh-master"
+ - ip: "10.1.3.93"
+ hostnames:
+ - "cdh-worker-1"
+ - ip: "10.1.3.94"
+ hostnames:
+ - "cdh-worker-2"
+ - ip: "10.1.3.91"
+ hostnames:
+ - "cdh-client"
+ containers:
+ - name: kylin
+ image: kylin-client:3.0.1-cdh57
+ imagePullPolicy: IfNotPresent
+ command:
+ - sh
+ - -c
+ args:
+ - cp $KYLIN_HOME/tomcat-config/* $KYLIN_HOME/tomcat/conf;
+ cp $KYLIN_HOME/kylin-more-config $KYLIN_HOME/tomcat/webapps/kylin/WEB-INF/classes/;
+ $TOOL_HOME/bootstrap.sh server -d;
+ ports:
+ - containerPort: 7070
+ - containerPort: 7443
+ volumeMounts:
+ - name: kylin-query-config
+ mountPath: /home/apache_kylin/kylin/conf
+ - name: tomcat-config
+ mountPath: /home/apache_kylin/kylin/tomcat-config
+ - name: kylin-more-config
+ mountPath: /home/apache_kylin/kylin/kylin-more-config/
+ - name: hadoop-config
+ mountPath: /etc/hadoop/conf
+ - name: hive-config
+ mountPath: /etc/hive/conf
+ - name: hbase-config
+ mountPath: /etc/hbase/conf
+ - name: kylin-logs
+ mountPath: /home/apache_kylin/kylin/logs
+ - name: tomcat-logs
+ mountPath: /home/apache_kylin/kylin/tomcat/logs
+ resources:
+ requests:
+ memory: 2Gi
+ cpu: 700m
+ limits:
+ memory: 2Gi
+ cpu: 700m
+ - name: filebeat
+ image: docker.elastic.co/beats/filebeat:6.4.3
+ args:
+ - -c
+ - /usr/share/filebeat/config/filebeat.yml
+ - -e
+ volumeMounts:
+ - name: kylin-logs
+ mountPath: /home/apache_kylin/kylin/logs
+ - name: tomcat-logs
+ mountPath: /home/apache_kylin/kylin/tomcat/logs
+ - name: filebeat-config
+ mountPath: /usr/share/filebeat/config
+ readOnly: true
+ resources:
+ requests:
+ memory: 1Gi
+ cpu: 300m
+ limits:
+ memory: 1Gi
+ cpu: 300m
+ volumes:
+ - name: kylin-logs
+ emptyDir:
+ sizeLimit: 3Gi
+ - name: tomcat-logs
+ emptyDir:
+ sizeLimit: 2Gi
+ - configMap:
Review comment:
I am not quite understand . Query server need hadoop configuration file such as `hbase-site.xml`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] sanjulian commented on a change in pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
sanjulian commented on a change in pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#discussion_r417748474
##########
File path: kubernetes/docker/hadoop-client/README.md
##########
@@ -0,0 +1,18 @@
+## Background
+### What is hadoop-client docker image and why we need this?
+As we all know, the node you want to deploy Kylin, should has provided Hadoop
+dependency(jars and configuration files), these dependency let you have access
+ to Hadoop Service, such as HDFS, HBase, Hive, which are needed by Apache Kylin.
+Unfortunately, each Hadoop distribution(CHD or HDP etc.) has its own specific jars. So, we
+can provided specific image for specific Hadoop distribution, which will make image management task
+more easier. This will have following two benefits:
+1. Someone who has better knowledge on Hadoop can do this work, and let kylin
+ user build their Kylin image base on provided Hadoop-Client image.
+2. Upgrade Kylin will be much easier.
+
+### Build Step for CDH5.7
+1. Place Spark binary(*spark-2.3.2-bin-hadoop2.7.tgz*) into dir `provided-binary`.
+2. Run `build-image.sh` to build image.
+
+> If you are using other hadoop distribution, please consider build image yourself.
+
Review comment:
Could we add the some build step for apache hadoop client?
Such as download hadoop client first, and update version in dockerfile.
##########
File path: kubernetes/example/deployment/kylin-job/kylin-job-service.yaml
##########
@@ -0,0 +1,36 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+apiVersion: v1
+kind: Service
+metadata:
+ name: kylin-job-svc
Review comment:
I suppose we do not need the services for job, since the job pod only for building cube instead of query.
##########
File path: kubernetes/example/deployment/kylin-query/kylin-query-statefulset.yaml
##########
@@ -0,0 +1,143 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+ name: kylin-query
+ namespace: kylin-example
+ labels:
+ app: kylin
+ role: query
+ environment: poc
+ version: 3.0.1
+ hadoop.version: cdh5.7
+spec:
+ serviceName: kylin-query-svc
+ replicas: 1
+ selector:
+ matchLabels:
+ app: kylin
+ role: query
+ template:
+ metadata:
+ labels:
+ app: kylin
+ role: query
+ spec:
+ hostAliases:
+ - ip: "10.1.3.90"
+ hostnames:
+ - "cdh-master"
+ - ip: "10.1.3.93"
+ hostnames:
+ - "cdh-worker-1"
+ - ip: "10.1.3.94"
+ hostnames:
+ - "cdh-worker-2"
+ - ip: "10.1.3.91"
+ hostnames:
+ - "cdh-client"
+ containers:
+ - name: kylin
+ image: kylin-client:3.0.1-cdh57
+ imagePullPolicy: IfNotPresent
+ command:
+ - sh
+ - -c
+ args:
+ - cp $KYLIN_HOME/tomcat-config/* $KYLIN_HOME/tomcat/conf;
Review comment:
Remove the copy before the init.
Please use subPath to mount configMap or secret files for tomcat and kylin configuration.
Here is the reference: https://kubernetes.io/docs/concepts/storage/volumes/#using-subpath
##########
File path: kubernetes/docker/hadoop-client/apache-hadoop2.7/Dockerfile
##########
@@ -0,0 +1,85 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+FROM centos:7.3.1611
+
+MAINTAINER Apache Kylin
+
+WORKDIR /tmp
+
+# install jdk and other commands
+RUN set -x \
+ && yum install -y which \
+ java-1.8.0-openjdk \
+ java-1.8.0-openjdk-devel \
+ krb5-workstation \
+ && yum clean all
+
+
+# version variables
+ARG HADOOP_VERSION=2.7.4
+ARG HIVE_VERSION=1.2.2
+ARG HBASE_VERSION=1.3.1
+ARG SPARK_VERSION=2.2.1
+ARG ZK_VERSION=3.4.6
+
+ARG APACHE_HOME=/apache
+
+RUN set -x \
+ && mkdir -p $APACHE_HOME
+
+ENV JAVA_HOME /etc/alternatives/jre
+
+# install hadoop
+ENV HADOOP_HOME=$APACHE_HOME/hadoop
+ADD hdp-clients/hadoop-${HADOOP_VERSION}.tar.gz $APACHE_HOME
+RUN set -x \
+ && ln -s $APACHE_HOME/hadoop-${HADOOP_VERSION} $HADOOP_HOME
+
+#install hive
+ENV HIVE_HOME=$APACHE_HOME/hive
+ADD hdp-clients/apache-hive-${HIVE_VERSION}.tar.gz $APACHE_HOME
+RUN set -x \
+ && ln -s $APACHE_HOME/apache-hive-${HIVE_VERSION} $HIVE_HOME
+
+#install hbase
+ENV HBASE_HOME=$APACHE_HOME/hbase
+ADD hdp-clients/hbase-${HBASE_VERSION}.tar.gz $APACHE_HOME
+RUN set -x \
+ && ln -s $APACHE_HOME/hbase-${HBASE_VERSION} $HBASE_HOME
+
+#install spark
+ENV SPARK_HOME=$APACHE_HOME/spark
+ADD hdp-clients/spark-${SPARK_VERSION}.tar.gz $APACHE_HOME
+RUN set -x \
+ && ln -s $APACHE_HOME/spark-${SPARK_VERSION} $SPARK_HOME
+
+#install zk
+ENV ZK_HOME=$APACHE_HOME/zookeeper
+ADD hdp-clients/zookeeper-${ZK_VERSION}.tar.gz $APACHE_HOME
+RUN set -x \
+ && ln -s $APACHE_HOME/zookeeper-${ZK_VERSION} $ZK_HOME
+
+ENV PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin:$HBASE_HOME/bin:$ZK_HOME/bin
+
+COPY conf/hadoop $HADOOP_HOME/etc/hadoop
Review comment:
I suppose we should remove the hadoop config. There are hadoop config for configmap in development files.
##########
File path: kubernetes/example/deployment/kylin-query/kylin-query-statefulset.yaml
##########
@@ -0,0 +1,143 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+ name: kylin-query
+ namespace: kylin-example
+ labels:
+ app: kylin
+ role: query
+ environment: poc
+ version: 3.0.1
+ hadoop.version: cdh5.7
+spec:
+ serviceName: kylin-query-svc
+ replicas: 1
+ selector:
+ matchLabels:
+ app: kylin
+ role: query
+ template:
+ metadata:
+ labels:
+ app: kylin
+ role: query
+ spec:
+ hostAliases:
+ - ip: "10.1.3.90"
+ hostnames:
+ - "cdh-master"
+ - ip: "10.1.3.93"
+ hostnames:
+ - "cdh-worker-1"
+ - ip: "10.1.3.94"
+ hostnames:
+ - "cdh-worker-2"
+ - ip: "10.1.3.91"
+ hostnames:
+ - "cdh-client"
+ containers:
+ - name: kylin
+ image: kylin-client:3.0.1-cdh57
+ imagePullPolicy: IfNotPresent
+ command:
+ - sh
+ - -c
+ args:
+ - cp $KYLIN_HOME/tomcat-config/* $KYLIN_HOME/tomcat/conf;
+ cp $KYLIN_HOME/kylin-more-config $KYLIN_HOME/tomcat/webapps/kylin/WEB-INF/classes/;
+ $TOOL_HOME/bootstrap.sh server -d;
+ ports:
+ - containerPort: 7070
+ - containerPort: 7443
+ volumeMounts:
+ - name: kylin-query-config
+ mountPath: /home/apache_kylin/kylin/conf
+ - name: tomcat-config
+ mountPath: /home/apache_kylin/kylin/tomcat-config
+ - name: kylin-more-config
+ mountPath: /home/apache_kylin/kylin/kylin-more-config/
+ - name: hadoop-config
+ mountPath: /etc/hadoop/conf
+ - name: hive-config
+ mountPath: /etc/hive/conf
+ - name: hbase-config
+ mountPath: /etc/hbase/conf
+ - name: kylin-logs
+ mountPath: /home/apache_kylin/kylin/logs
+ - name: tomcat-logs
+ mountPath: /home/apache_kylin/kylin/tomcat/logs
+ resources:
+ requests:
+ memory: 2Gi
+ cpu: 700m
+ limits:
+ memory: 2Gi
+ cpu: 700m
+ - name: filebeat
+ image: docker.elastic.co/beats/filebeat:6.4.3
+ args:
+ - -c
+ - /usr/share/filebeat/config/filebeat.yml
+ - -e
+ volumeMounts:
+ - name: kylin-logs
+ mountPath: /home/apache_kylin/kylin/logs
+ - name: tomcat-logs
+ mountPath: /home/apache_kylin/kylin/tomcat/logs
+ - name: filebeat-config
+ mountPath: /usr/share/filebeat/config
+ readOnly: true
+ resources:
+ requests:
+ memory: 1Gi
+ cpu: 300m
+ limits:
+ memory: 1Gi
+ cpu: 300m
+ volumes:
+ - name: kylin-logs
+ emptyDir:
+ sizeLimit: 3Gi
+ - name: tomcat-logs
+ emptyDir:
+ sizeLimit: 2Gi
+ - configMap:
+ name: hadoop-config
+ name: hadoop-config
+ - configMap:
+ name: hive-config
+ name: hive-config
+ - configMap:
+ name: hbase-config
+ name: hbase-config
+ - configMap:
Review comment:
As I know, there are some secret files in kylin.properties, such as kylin.job.notification-mail-password, kylin.security.ldap.connection-password... Could we change the kylin-***-config as secrets instead of configMap?
##########
File path: kubernetes/example/deployment/kylin-query/kylin-query-statefulset.yaml
##########
@@ -0,0 +1,143 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+ name: kylin-query
+ namespace: kylin-example
+ labels:
+ app: kylin
+ role: query
+ environment: poc
+ version: 3.0.1
+ hadoop.version: cdh5.7
+spec:
+ serviceName: kylin-query-svc
+ replicas: 1
+ selector:
+ matchLabels:
+ app: kylin
+ role: query
+ template:
+ metadata:
+ labels:
+ app: kylin
+ role: query
+ spec:
+ hostAliases:
+ - ip: "10.1.3.90"
+ hostnames:
+ - "cdh-master"
+ - ip: "10.1.3.93"
+ hostnames:
+ - "cdh-worker-1"
+ - ip: "10.1.3.94"
+ hostnames:
+ - "cdh-worker-2"
+ - ip: "10.1.3.91"
+ hostnames:
+ - "cdh-client"
+ containers:
+ - name: kylin
+ image: kylin-client:3.0.1-cdh57
+ imagePullPolicy: IfNotPresent
+ command:
+ - sh
+ - -c
+ args:
+ - cp $KYLIN_HOME/tomcat-config/* $KYLIN_HOME/tomcat/conf;
+ cp $KYLIN_HOME/kylin-more-config $KYLIN_HOME/tomcat/webapps/kylin/WEB-INF/classes/;
+ $TOOL_HOME/bootstrap.sh server -d;
+ ports:
+ - containerPort: 7070
+ - containerPort: 7443
+ volumeMounts:
+ - name: kylin-query-config
+ mountPath: /home/apache_kylin/kylin/conf
+ - name: tomcat-config
+ mountPath: /home/apache_kylin/kylin/tomcat-config
+ - name: kylin-more-config
+ mountPath: /home/apache_kylin/kylin/kylin-more-config/
+ - name: hadoop-config
+ mountPath: /etc/hadoop/conf
+ - name: hive-config
+ mountPath: /etc/hive/conf
+ - name: hbase-config
+ mountPath: /etc/hbase/conf
+ - name: kylin-logs
+ mountPath: /home/apache_kylin/kylin/logs
+ - name: tomcat-logs
+ mountPath: /home/apache_kylin/kylin/tomcat/logs
+ resources:
+ requests:
+ memory: 2Gi
+ cpu: 700m
+ limits:
+ memory: 2Gi
+ cpu: 700m
+ - name: filebeat
+ image: docker.elastic.co/beats/filebeat:6.4.3
+ args:
+ - -c
+ - /usr/share/filebeat/config/filebeat.yml
+ - -e
+ volumeMounts:
+ - name: kylin-logs
+ mountPath: /home/apache_kylin/kylin/logs
+ - name: tomcat-logs
+ mountPath: /home/apache_kylin/kylin/tomcat/logs
+ - name: filebeat-config
+ mountPath: /usr/share/filebeat/config
+ readOnly: true
+ resources:
+ requests:
+ memory: 1Gi
+ cpu: 300m
+ limits:
+ memory: 1Gi
+ cpu: 300m
+ volumes:
+ - name: kylin-logs
+ emptyDir:
+ sizeLimit: 3Gi
+ - name: tomcat-logs
+ emptyDir:
+ sizeLimit: 2Gi
+ - configMap:
Review comment:
The hadoop configmap is not used. Should we delete them or add the volume mount for hdp configuration?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619833363
## Prepare Docker Image
- Downloads required binary
```shell
xiaoxiang.yu:~/ $ ll Downloads/apache-kylin-3.0.1-bin-cdh57.tar [18:39:26]
-rw-r--r--@ 1 xiaoxiang.yu INC\Domain Users 253M 4 15 16:49 Downloads/apache-kylin-3.0.1-bin-cdh57.tar
xiaoxiang.yu:~/ $ ll Downloads/spark-2.3.2-bin-hadoop2.7.tgz [18:39:52]
-rwxrwxrwx 1 xiaoxiang.yu INC\Domain Users 215M 3 23 01:36 Downloads/spark-2.3.2-bin-hadoop2.7.tgz
```
- Downloads required jars
```shell
xiaoxiang.yu:~/ $ ll Downloads/mem* [18:40:05]
-rw-r--r--@ 1 xiaoxiang.yu INC\Domain Users 163K 4 27 18:45 Downloads/memcached-session-manager-2.1.1.jar
-rw-r--r--@ 1 xiaoxiang.yu INC\Domain Users 11K 4 27 18:45 Downloads/memcached-session-manager-tc7-2.1.1.jar
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] sanjulian commented on a change in pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
sanjulian commented on a change in pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#discussion_r419336226
##########
File path: kubernetes/example/deployment/kylin-query/kylin-query-statefulset.yaml
##########
@@ -0,0 +1,143 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+ name: kylin-query
+ namespace: kylin-example
+ labels:
+ app: kylin
+ role: query
+ environment: poc
+ version: 3.0.1
+ hadoop.version: cdh5.7
+spec:
+ serviceName: kylin-query-svc
+ replicas: 1
+ selector:
+ matchLabels:
+ app: kylin
+ role: query
+ template:
+ metadata:
+ labels:
+ app: kylin
+ role: query
+ spec:
+ hostAliases:
+ - ip: "10.1.3.90"
+ hostnames:
+ - "cdh-master"
+ - ip: "10.1.3.93"
+ hostnames:
+ - "cdh-worker-1"
+ - ip: "10.1.3.94"
+ hostnames:
+ - "cdh-worker-2"
+ - ip: "10.1.3.91"
+ hostnames:
+ - "cdh-client"
+ containers:
+ - name: kylin
+ image: kylin-client:3.0.1-cdh57
+ imagePullPolicy: IfNotPresent
+ command:
+ - sh
+ - -c
+ args:
+ - cp $KYLIN_HOME/tomcat-config/* $KYLIN_HOME/tomcat/conf;
+ cp $KYLIN_HOME/kylin-more-config $KYLIN_HOME/tomcat/webapps/kylin/WEB-INF/classes/;
+ $TOOL_HOME/bootstrap.sh server -d;
+ ports:
+ - containerPort: 7070
+ - containerPort: 7443
+ volumeMounts:
+ - name: kylin-query-config
+ mountPath: /home/apache_kylin/kylin/conf
+ - name: tomcat-config
+ mountPath: /home/apache_kylin/kylin/tomcat-config
+ - name: kylin-more-config
+ mountPath: /home/apache_kylin/kylin/kylin-more-config/
+ - name: hadoop-config
+ mountPath: /etc/hadoop/conf
+ - name: hive-config
+ mountPath: /etc/hive/conf
+ - name: hbase-config
+ mountPath: /etc/hbase/conf
+ - name: kylin-logs
+ mountPath: /home/apache_kylin/kylin/logs
+ - name: tomcat-logs
+ mountPath: /home/apache_kylin/kylin/tomcat/logs
+ resources:
+ requests:
+ memory: 2Gi
+ cpu: 700m
+ limits:
+ memory: 2Gi
+ cpu: 700m
+ - name: filebeat
+ image: docker.elastic.co/beats/filebeat:6.4.3
+ args:
+ - -c
+ - /usr/share/filebeat/config/filebeat.yml
+ - -e
+ volumeMounts:
+ - name: kylin-logs
+ mountPath: /home/apache_kylin/kylin/logs
+ - name: tomcat-logs
+ mountPath: /home/apache_kylin/kylin/tomcat/logs
+ - name: filebeat-config
+ mountPath: /usr/share/filebeat/config
+ readOnly: true
+ resources:
+ requests:
+ memory: 1Gi
+ cpu: 300m
+ limits:
+ memory: 1Gi
+ cpu: 300m
+ volumes:
+ - name: kylin-logs
+ emptyDir:
+ sizeLimit: 3Gi
+ - name: tomcat-logs
+ emptyDir:
+ sizeLimit: 2Gi
+ - configMap:
Review comment:
Sorry, my bad. I did not notice the configMap is mounted. Please ignore the previous comment I wrote.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus commented on a change in pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus commented on a change in pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#discussion_r419249019
##########
File path: kubernetes/docker/hadoop-client/apache-hadoop2.7/Dockerfile
##########
@@ -0,0 +1,85 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+FROM centos:7.3.1611
+
+MAINTAINER Apache Kylin
+
+WORKDIR /tmp
+
+# install jdk and other commands
+RUN set -x \
+ && yum install -y which \
+ java-1.8.0-openjdk \
+ java-1.8.0-openjdk-devel \
+ krb5-workstation \
+ && yum clean all
+
+
+# version variables
+ARG HADOOP_VERSION=2.7.4
+ARG HIVE_VERSION=1.2.2
+ARG HBASE_VERSION=1.3.1
+ARG SPARK_VERSION=2.2.1
+ARG ZK_VERSION=3.4.6
+
+ARG APACHE_HOME=/apache
+
+RUN set -x \
+ && mkdir -p $APACHE_HOME
+
+ENV JAVA_HOME /etc/alternatives/jre
+
+# install hadoop
+ENV HADOOP_HOME=$APACHE_HOME/hadoop
+ADD hdp-clients/hadoop-${HADOOP_VERSION}.tar.gz $APACHE_HOME
+RUN set -x \
+ && ln -s $APACHE_HOME/hadoop-${HADOOP_VERSION} $HADOOP_HOME
+
+#install hive
+ENV HIVE_HOME=$APACHE_HOME/hive
+ADD hdp-clients/apache-hive-${HIVE_VERSION}.tar.gz $APACHE_HOME
+RUN set -x \
+ && ln -s $APACHE_HOME/apache-hive-${HIVE_VERSION} $HIVE_HOME
+
+#install hbase
+ENV HBASE_HOME=$APACHE_HOME/hbase
+ADD hdp-clients/hbase-${HBASE_VERSION}.tar.gz $APACHE_HOME
+RUN set -x \
+ && ln -s $APACHE_HOME/hbase-${HBASE_VERSION} $HBASE_HOME
+
+#install spark
+ENV SPARK_HOME=$APACHE_HOME/spark
+ADD hdp-clients/spark-${SPARK_VERSION}.tar.gz $APACHE_HOME
+RUN set -x \
+ && ln -s $APACHE_HOME/spark-${SPARK_VERSION} $SPARK_HOME
+
+#install zk
+ENV ZK_HOME=$APACHE_HOME/zookeeper
+ADD hdp-clients/zookeeper-${ZK_VERSION}.tar.gz $APACHE_HOME
+RUN set -x \
+ && ln -s $APACHE_HOME/zookeeper-${ZK_VERSION} $ZK_HOME
+
+ENV PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin:$HBASE_HOME/bin:$ZK_HOME/bin
+
+COPY conf/hadoop $HADOOP_HOME/etc/hadoop
Review comment:
OK, I will do it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] sanjulian commented on a change in pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
sanjulian commented on a change in pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#discussion_r419333808
##########
File path: kubernetes/example/deployment/kylin-job/kylin-job-service.yaml
##########
@@ -0,0 +1,36 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+apiVersion: v1
+kind: Service
+metadata:
+ name: kylin-job-svc
Review comment:
Yes, that’s why I suggested to remove the service. It is unnecessary. And user talk with Kylin by query service, even call build job api.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619862275
## Test Kylin Query Service
- Instance List
<img width="1189" alt="image" src="https://user-images.githubusercontent.com/14030549/80373561-d4e70200-88c7-11ea-9387-55f8125c69b9.png">
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus commented on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus commented on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619861191
## Prepare Hadoop Configuration
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619861773
## Create Memcached Service
- Create namespace
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create namespace kylin-example
Error from server (AlreadyExists): namespaces "kylin-example" already exists
```
- Create memcached service
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl apply -f memcached
service/cache-svc created
statefulset.apps/kylin-memcached created
```
- Check service and pod state
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get pod -n kylin-example
NAME READY STATUS RESTARTS AGE
kylin-memcached-0 1/1 Running 0 60s
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get svc -n kylin-example
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cache-svc ClusterIP None <none> 11211/TCP 69s
```
- Check hostname
```
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl run -it --image=busybox:1.28.4 --rm --restart=Never sh -n test-dnsIf you don't see a command prompt, try pressing enter.
/ # nslookup cache-svc.kylin-example.svc.cluster.local
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: cache-svc.kylin-example.svc.cluster.local
Address 1: 192.168.11.9 kylin-memcached-0.cache-svc.kylin-example.svc.cluster.local
/ # telnet cache-svc.kylin-example.svc.cluster.local 11211
^C
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619862064
## Test Kylin Job Service
- Instance List
<img width="1190" alt="image" src="https://user-images.githubusercontent.com/14030549/80372342-fcd56600-88c5-11ea-9552-a06878129169.png">
- Create sample model/cube
<img width="1437" alt="image" src="https://user-images.githubusercontent.com/14030549/80372032-6e60e480-88c5-11ea-8000-c6bffd328073.png">
- Submit job and build succeed
<img width="1417" alt="image" src="https://user-images.githubusercontent.com/14030549/80372793-a74d8900-88c6-11ea-93da-3360cf4bd2f1.png">
<img width="1414" alt="image" src="https://user-images.githubusercontent.com/14030549/80372979-f5628c80-88c6-11ea-95cd-b0f49521c178.png">
<img width="1071" alt="image" src="https://user-images.githubusercontent.com/14030549/80373078-19be6900-88c7-11ea-96fa-61fd64917a16.png">
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619861990
## Create Kylin Job Service
- Change configuration of memcached related(part-1)
```shell
# kylin.cache.memcached.hosts=cache-svc.kylin-example.svc.cluster.local:11211
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# cd ../config/
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/config# vim kylin-job/kylin.properties
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/config# vim kylin-query/kylin.properties
```
- Change configuration of memcached related(part-2)
```
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/config# vim tomcat/context.xml
```
```xml
<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
memcachedNodes="n1:cache-svc.kylin-example.svc.cluster.local:11211"
storageKeyPrefix="context"
requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
/>
```
- Create related configmap
```
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example hadoop-config \
> --from-file=../config/hadoop/core-site.xml \
> --from-file=../config/hadoop/hdfs-site.xml \
> --from-file=../config/hadoop/yarn-site.xml \
> --from-file=../config/hadoop/mapred-site.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:28.055160 4398 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/hadoop-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example hive-config \
> --from-file=../config/hadoop/hive-site.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:35.449192 4525 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/hive-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example hbase-config \
> --from-file=../config/hadoop/hbase-site.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:36.222142 4583 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/hbase-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example kylin-more-config \
> --from-file=../config/kylin-more/applicationContext.xml \
> --from-file=../config/kylin-more/ehcache.xml \
> --from-file=../config/kylin-more/ehcache-test.xml \
> --from-file=../config/kylin-more/kylinMetrics.xml \
> --from-file=../config/kylin-more/kylinSecurity.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:43.678445 4720 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/kylin-more-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example filebeat-config \
> --from-file=../config/filebeat/filebeat.yml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:51.686495 4901 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/filebeat-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example tomcat-config \
> --from-file=../config/tomcat/server.xml \
> --from-file=../config/tomcat/context.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:52.431775 4926 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/tomcat-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example kylin-job-config \
> --from-file=../config/kylin-job/kylin-kafka-consumer.xml \
> --from-file=../config/kylin-job/kylin_hive_conf.xml \
> --from-file=../config/kylin-job/kylin_job_conf.xml \
> --from-file=../config/kylin-job/kylin_job_conf_inmem.xml \
> --from-file=../config/kylin-job/kylin-server-log4j.properties \
> --from-file=../config/kylin-job/kylin-spark-log4j.properties \
> --from-file=../config/kylin-job/kylin-tools-log4j.properties \
> --from-file=../config/kylin-job/kylin.properties \
> --from-file=../config/kylin-job/setenv.sh \
> --from-file=../config/kylin-job/setenv-tool.sh \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:10:03.250415 5141 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/kylin-job-config configured
```
- Start job service
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl apply -f kylin-job
service/kylin-job-svc created
statefulset.apps/kylin-job created
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get statefulset -n kylin-example
NAME READY AGE
kylin-job 1/1 12s
kylin-memcached 1/1 16m
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get pod -n kylin-example
NAME READY STATUS RESTARTS AGE
kylin-job-0 2/2 Running 0 23s
kylin-memcached-0 1/1 Running 0 16m
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# docker ps | grep kylin-job-
11517f6f98d6 a9acac7280fe "/usr/local/bin/dock…" 2 minutes ago Up 2 minutes k8s_filebeat_kylin-job-0_kylin-example_284fa3ea-5bce-437e-b769-92d6c938c628_0
fc0095a9f2d3 7ea27faa653f "sh -c 'cp $KYLIN_HO…" 2 minutes ago Up 2 minutes k8s_kylin_kylin-job-0_kylin-example_284fa3ea-5bce-437e-b769-92d6c938c628_0
58aa2566d7fb k8s.gcr.io/pause:3.2 "/pause" 2 minutes ago Up 2 minutes k8s_POD_kylin-job-0_kylin-example_284fa3ea-5bce-437e-b769-92d6c938c628_0
```
- Visit Web UI
http://cloud:30010/kylin/models
- Attach pod
```shell
root@open-source:/home/ubuntu/example/deployment# kubectl exec -it kylin-job-0 -n kylin-example -- bash
Defaulting container name to kylin.
Use 'kubectl describe pod/kylin-job-0 -n kylin-example' to see all of the containers in this pod.
[root@kylin-job-0 tmp]# cd $KYLIN_HOME/logs
[root@kylin-job-0 logs]# ll
total 196
-rw-r--r-- 1 root root 1875 Apr 27 12:13 kylin.gc.6188.0.current
-rw-r--r-- 1 root root 4794 Apr 27 12:21 kylin.gc.pid6427.0.current
-rw-r--r-- 1 root root 179233 Apr 27 12:22 kylin.log
-rw-r--r-- 1 root root 4651 Apr 27 12:14 kylin.out
[root@kylin-job-0 logs]# tail -f kylin.*
==> kylin.gc.6188.0.current <==
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619862739
## Scale Out/In Query Service
- https://kubernetes.io/docs/tasks/run-application/scale-stateful-set/
- Modify replica(from 1 to 3)
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# vim kylin-job/kylin-job-statefulset.yaml
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl apply -f kylin-job/
```
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get pod -n kylin-example
NAME READY STATUS RESTARTS AGE
kylin-job-0 2/2 Running 0 27s
kylin-job-1 2/2 Running 0 69s
kylin-job-2 2/2 Running 0 67s
kylin-memcached-0 1/1 Running 0 131m
kylin-query-0 2/2 Running 0 66m
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get pod -n kylin-example
NAME READY STATUS RESTARTS AGE
kylin-job-0 2/2 Running 0 34s
kylin-job-1 2/2 Running 0 76s
kylin-job-2 2/2 Running 0 74s
kylin-memcached-0 1/1 Running 0 131m
kylin-query-0 2/2 Running 0 66m
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get statefulset -n kylin-example
NAME READY AGE
kylin-job 3/3 115m
kylin-memcached 1/1 131m
kylin-query 1/1 66m
```
<img width="1189" alt="image" src="https://user-images.githubusercontent.com/14030549/80381674-dcaca380-88d3-11ea-9b43-96ffc452b319.png">
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619861990
## Create Kylin Job Service
- Change configuration of memcached related(part-1)
```shell
# kylin.cache.memcached.hosts=cache-svc.kylin-example.svc.cluster.local:11211
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# cd ../config/
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/config# vim kylin-job/kylin.properties
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/config# vim kylin-query/kylin.properties
```
- Change configuration of memcached related(part-2)
```
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/config# vim tomcat/context.xml
```
```xml
<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
memcachedNodes="n1:cache-svc.kylin-example.svc.cluster.local:11211"
storageKeyPrefix="context"
requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
/>
```
- Create related configmap
```
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example hadoop-config \
> --from-file=../config/hadoop/core-site.xml \
> --from-file=../config/hadoop/hdfs-site.xml \
> --from-file=../config/hadoop/yarn-site.xml \
> --from-file=../config/hadoop/mapred-site.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:28.055160 4398 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/hadoop-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example hive-config \
> --from-file=../config/hadoop/hive-site.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:35.449192 4525 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/hive-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example hbase-config \
> --from-file=../config/hadoop/hbase-site.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:36.222142 4583 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/hbase-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example kylin-more-config \
> --from-file=../config/kylin-more/applicationContext.xml \
> --from-file=../config/kylin-more/ehcache.xml \
> --from-file=../config/kylin-more/ehcache-test.xml \
> --from-file=../config/kylin-more/kylinMetrics.xml \
> --from-file=../config/kylin-more/kylinSecurity.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:43.678445 4720 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/kylin-more-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example filebeat-config \
> --from-file=../config/filebeat/filebeat.yml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:51.686495 4901 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/filebeat-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example tomcat-config \
> --from-file=../config/tomcat/server.xml \
> --from-file=../config/tomcat/context.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:52.431775 4926 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/tomcat-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example kylin-job-config \
> --from-file=../config/kylin-job/kylin-kafka-consumer.xml \
> --from-file=../config/kylin-job/kylin_hive_conf.xml \
> --from-file=../config/kylin-job/kylin_job_conf.xml \
> --from-file=../config/kylin-job/kylin_job_conf_inmem.xml \
> --from-file=../config/kylin-job/kylin-server-log4j.properties \
> --from-file=../config/kylin-job/kylin-spark-log4j.properties \
> --from-file=../config/kylin-job/kylin-tools-log4j.properties \
> --from-file=../config/kylin-job/kylin.properties \
> --from-file=../config/kylin-job/setenv.sh \
> --from-file=../config/kylin-job/setenv-tool.sh \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:10:03.250415 5141 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/kylin-job-config configured
```
- Start job service
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl apply -f kylin-job
service/kylin-job-svc created
statefulset.apps/kylin-job created
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get statefulset -n kylin-example
NAME READY AGE
kylin-job 1/1 12s
kylin-memcached 1/1 16m
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get pod -n kylin-example
NAME READY STATUS RESTARTS AGE
kylin-job-0 2/2 Running 0 23s
kylin-memcached-0 1/1 Running 0 16m
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] coveralls edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-617824003
## Pull Request Test Coverage Report for [Build 5878](https://coveralls.io/builds/30575046)
* **0** of **0** changed or added relevant lines in **0** files are covered.
* **1** unchanged line in **1** file lost coverage.
* Overall coverage decreased (**-0.001%**) to **27.319%**
---
| Files with Coverage Reduction | New Missed Lines | % |
| :-----|--------------|--: |
| [core-dictionary/src/main/java/org/apache/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://coveralls.io/builds/30575046/source?filename=core-dictionary%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fdict%2Flookup%2Fcache%2FRocksDBLookupTable.java#L62) | 1 | 81.08% |
<!-- | **Total:** | **1** | | -->
| Totals | [](https://coveralls.io/builds/30575046) |
| :-- | --: |
| Change from base [Build 5865](https://coveralls.io/builds/30432594): | -0.001% |
| Covered Lines: | 24324 |
| Relevant Lines: | 89036 |
---
##### 💛 - [Coveralls](https://coveralls.io)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus commented on a change in pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus commented on a change in pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#discussion_r419298338
##########
File path: kubernetes/example/deployment/kylin-query/kylin-query-statefulset.yaml
##########
@@ -0,0 +1,143 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+ name: kylin-query
+ namespace: kylin-example
+ labels:
+ app: kylin
+ role: query
+ environment: poc
+ version: 3.0.1
+ hadoop.version: cdh5.7
+spec:
+ serviceName: kylin-query-svc
+ replicas: 1
+ selector:
+ matchLabels:
+ app: kylin
+ role: query
+ template:
+ metadata:
+ labels:
+ app: kylin
+ role: query
+ spec:
+ hostAliases:
+ - ip: "10.1.3.90"
+ hostnames:
+ - "cdh-master"
+ - ip: "10.1.3.93"
+ hostnames:
+ - "cdh-worker-1"
+ - ip: "10.1.3.94"
+ hostnames:
+ - "cdh-worker-2"
+ - ip: "10.1.3.91"
+ hostnames:
+ - "cdh-client"
+ containers:
+ - name: kylin
+ image: kylin-client:3.0.1-cdh57
+ imagePullPolicy: IfNotPresent
+ command:
+ - sh
+ - -c
+ args:
+ - cp $KYLIN_HOME/tomcat-config/* $KYLIN_HOME/tomcat/conf;
+ cp $KYLIN_HOME/kylin-more-config $KYLIN_HOME/tomcat/webapps/kylin/WEB-INF/classes/;
+ $TOOL_HOME/bootstrap.sh server -d;
+ ports:
+ - containerPort: 7070
+ - containerPort: 7443
+ volumeMounts:
+ - name: kylin-query-config
+ mountPath: /home/apache_kylin/kylin/conf
+ - name: tomcat-config
+ mountPath: /home/apache_kylin/kylin/tomcat-config
+ - name: kylin-more-config
+ mountPath: /home/apache_kylin/kylin/kylin-more-config/
+ - name: hadoop-config
+ mountPath: /etc/hadoop/conf
+ - name: hive-config
+ mountPath: /etc/hive/conf
+ - name: hbase-config
+ mountPath: /etc/hbase/conf
+ - name: kylin-logs
+ mountPath: /home/apache_kylin/kylin/logs
+ - name: tomcat-logs
+ mountPath: /home/apache_kylin/kylin/tomcat/logs
+ resources:
+ requests:
+ memory: 2Gi
+ cpu: 700m
+ limits:
+ memory: 2Gi
+ cpu: 700m
+ - name: filebeat
+ image: docker.elastic.co/beats/filebeat:6.4.3
+ args:
+ - -c
+ - /usr/share/filebeat/config/filebeat.yml
+ - -e
+ volumeMounts:
+ - name: kylin-logs
+ mountPath: /home/apache_kylin/kylin/logs
+ - name: tomcat-logs
+ mountPath: /home/apache_kylin/kylin/tomcat/logs
+ - name: filebeat-config
+ mountPath: /usr/share/filebeat/config
+ readOnly: true
+ resources:
+ requests:
+ memory: 1Gi
+ cpu: 300m
+ limits:
+ memory: 1Gi
+ cpu: 300m
+ volumes:
+ - name: kylin-logs
+ emptyDir:
+ sizeLimit: 3Gi
+ - name: tomcat-logs
+ emptyDir:
+ sizeLimit: 2Gi
+ - configMap:
+ name: hadoop-config
+ name: hadoop-config
+ - configMap:
+ name: hive-config
+ name: hive-config
+ - configMap:
+ name: hbase-config
+ name: hbase-config
+ - configMap:
Review comment:
Good suggestion! I think we can add this to documentation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus commented on a change in pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus commented on a change in pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#discussion_r419248811
##########
File path: kubernetes/docker/hadoop-client/README.md
##########
@@ -0,0 +1,18 @@
+## Background
+### What is hadoop-client docker image and why we need this?
+As we all know, the node you want to deploy Kylin, should has provided Hadoop
+dependency(jars and configuration files), these dependency let you have access
+ to Hadoop Service, such as HDFS, HBase, Hive, which are needed by Apache Kylin.
+Unfortunately, each Hadoop distribution(CHD or HDP etc.) has its own specific jars. So, we
+can provided specific image for specific Hadoop distribution, which will make image management task
+more easier. This will have following two benefits:
+1. Someone who has better knowledge on Hadoop can do this work, and let kylin
+ user build their Kylin image base on provided Hadoop-Client image.
+2. Upgrade Kylin will be much easier.
+
+### Build Step for CDH5.7
+1. Place Spark binary(*spark-2.3.2-bin-hadoop2.7.tgz*) into dir `provided-binary`.
+2. Run `build-image.sh` to build image.
+
+> If you are using other hadoop distribution, please consider build image yourself.
+
Review comment:
OK, I will add it to readme file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619862739
## Scale Out/In Job Service
- https://kubernetes.io/docs/tasks/run-application/scale-stateful-set/
- Modify replica(from 1 to 3) and scale out job server
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# vim kylin-job/kylin-job-statefulset.yaml
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl apply -f kylin-job/
```
- Check state of pods
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get pod -n kylin-example
NAME READY STATUS RESTARTS AGE
kylin-job-0 2/2 Running 0 27s
kylin-job-1 2/2 Running 0 69s
kylin-job-2 2/2 Running 0 67s
kylin-memcached-0 1/1 Running 0 131m
kylin-query-0 2/2 Running 0 66m
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get pod -n kylin-example
NAME READY STATUS RESTARTS AGE
kylin-job-0 2/2 Running 0 34s
kylin-job-1 2/2 Running 0 76s
kylin-job-2 2/2 Running 0 74s
kylin-memcached-0 1/1 Running 0 131m
kylin-query-0 2/2 Running 0 66m
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get statefulset -n kylin-example
NAME READY AGE
kylin-job 3/3 115m
kylin-memcached 1/1 131m
kylin-query 1/1 66m
root@open-source:/home/ubuntu/example/deployment# docker ps | grep kylin-job-
da9c0cf239e4 a9acac7280fe "/usr/local/bin/dock…" 8 minutes ago Up 8 minutes k8s_filebeat_kylin-job-0_kylin-example_26c08037-0543-4e15-9ac2-54021266a52b_0
06ff2e6a2d3e 7ea27faa653f "sh -c 'cp $KYLIN_HO…" 8 minutes ago Up 8 minutes k8s_kylin_kylin-job-0_kylin-example_26c08037-0543-4e15-9ac2-54021266a52b_0
87062bb80c1b k8s.gcr.io/pause:3.2 "/pause" 8 minutes ago Up 8 minutes k8s_POD_kylin-job-0_kylin-example_26c08037-0543-4e15-9ac2-54021266a52b_0
91d96d1bee0f a9acac7280fe "/usr/local/bin/dock…" 8 minutes ago Up 8 minutes k8s_filebeat_kylin-job-2_kylin-example_93cc142c-0dc8-4e5e-886d-4408fac76c5d_0
d5d352acbb8d 7ea27faa653f "sh -c 'cp $KYLIN_HO…" 8 minutes ago Up 8 minutes k8s_kylin_kylin-job-2_kylin-example_93cc142c-0dc8-4e5e-886d-4408fac76c5d_0
ad05de976d8a k8s.gcr.io/pause:3.2 "/pause" 8 minutes ago Up 8 minutes k8s_POD_kylin-job-2_kylin-example_93cc142c-0dc8-4e5e-886d-4408fac76c5d_0
41de3f26e592 a9acac7280fe "/usr/local/bin/dock…" 8 minutes ago Up 8 minutes k8s_filebeat_kylin-job-1_kylin-example_b83623a1-c8a4-456f-95e3-fce19d78afb1_0
b3666e72bc7b 7ea27faa653f "sh -c 'cp $KYLIN_HO…" 8 minutes ago Up 8 minutes k8s_kylin_kylin-job-1_kylin-example_b83623a1-c8a4-456f-95e3-fce19d78afb1_0
1489fd6e04db k8s.gcr.io/pause:3.2 "/pause" 8 minutes ago Up 8 minutes k8s_POD_kylin-job-1_kylin-example_b83623a1-c8a4-456f-95e3-fce19d78afb1_0
root@open-source:/home/ubuntu/example/deployment#
```
- Instance List
<img width="1189" alt="image" src="https://user-images.githubusercontent.com/14030549/80381674-dcaca380-88d3-11ea-9b43-96ffc452b319.png">
- Test another build
<img width="1375" alt="image" src="https://user-images.githubusercontent.com/14030549/80383050-aa03aa80-88d5-11ea-8424-52825dd0d8cd.png">
- Force kill Active JobServer(Failover Test)
```shell
root@open-source:/home/ubuntu/example/deployment# docker kill b3666e72bc7b
b3666e72bc7b
```
- Check current active job server
<img width="1190" alt="image" src="https://user-images.githubusercontent.com/14030549/80383200-df0ffd00-88d5-11ea-8474-f21de4d0a6e3.png">
- Test a third build
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619862166
## Create Kylin Query Service
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example kylin-query-config \
> --from-file=../config/kylin-query/kylin-kafka-consumer.xml \
> --from-file=../config/kylin-query/kylin_hive_conf.xml \
> --from-file=../config/kylin-query/kylin_job_conf.xml \
> --from-file=../config/kylin-query/kylin_job_conf_inmem.xml \
> --from-file=../config/kylin-query/kylin-server-log4j.properties \
> --from-file=../config/kylin-query/kylin-spark-log4j.properties \
> --from-file=../config/kylin-query/kylin-tools-log4j.properties \
> --from-file=../config/kylin-query/kylin.properties \
> --from-file=../config/kylin-query/setenv.sh \
> --from-file=../config/kylin-query/setenv-tool.sh \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:36:38.484197 10902 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/kylin-query-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl apply -f kylin-query
service/kylin-query-svc created
statefulset.apps/kylin-query created
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get pod -n kylin-example
NAME READY STATUS RESTARTS AGE
kylin-job-0 2/2 Running 0 28m
kylin-memcached-0 1/1 Running 0 44m
kylin-query-0 2/2 Running 0 11s
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get statefulset -n kylin-example
NAME READY AGE
kylin-job 1/1 28m
kylin-memcached 1/1 44m
kylin-query 1/1 19s
```
- Go to Web UI http://cloud:30012/kylin/admin
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619833363
## Prepare Docker Image
- Downloads required binary
```shell
xiaoxiang.yu:~/ $ ll Downloads/apache-kylin-3.0.1-bin-cdh57.tar [18:39:26]
-rw-r--r--@ 1 xiaoxiang.yu INC\Domain Users 253M 4 15 16:49 Downloads/apache-kylin-3.0.1-bin-cdh57.tar
xiaoxiang.yu:~/ $ ll Downloads/spark-2.3.2-bin-hadoop2.7.tgz [18:39:52]
-rwxrwxrwx 1 xiaoxiang.yu INC\Domain Users 215M 3 23 01:36 Downloads/spark-2.3.2-bin-hadoop2.7.tgz
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] coveralls edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-617824003
## Pull Request Test Coverage Report for [Build 5877](https://coveralls.io/builds/30574948)
* **0** of **0** changed or added relevant lines in **0** files are covered.
* **3** unchanged lines in **2** files lost coverage.
* Overall coverage decreased (**-0.003%**) to **27.317%**
---
| Files with Coverage Reduction | New Missed Lines | % |
| :-----|--------------|--: |
| [core-dictionary/src/main/java/org/apache/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://coveralls.io/builds/30574948/source?filename=core-dictionary%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fdict%2Flookup%2Fcache%2FRocksDBLookupTable.java#L62) | 1 | 81.08% |
| [core-cube/src/main/java/org/apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://coveralls.io/builds/30574948/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Fcuboid%2FTreeCuboidScheduler.java#L124) | 2 | 68.46% |
<!-- | **Total:** | **3** | | -->
| Totals | [](https://coveralls.io/builds/30574948) |
| :-- | --: |
| Change from base [Build 5865](https://coveralls.io/builds/30432594): | -0.003% |
| Covered Lines: | 24322 |
| Relevant Lines: | 89036 |
---
##### 💛 - [Coveralls](https://coveralls.io)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus commented on a change in pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus commented on a change in pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#discussion_r419248580
##########
File path: kubernetes/example/deployment/kylin-job/kylin-job-service.yaml
##########
@@ -0,0 +1,36 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+apiVersion: v1
+kind: Service
+metadata:
+ name: kylin-job-svc
Review comment:
In my test, if I use only one service, the SQL query request will be sent to both job server and query server, thus sometimes the request will be rejected, because job server won't answer such request.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619861773
## Create Memcached Service
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create namespace kylin-example
Error from server (AlreadyExists): namespaces "kylin-example" already exists
```
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl apply -f memcached
service/cache-svc created
statefulset.apps/kylin-memcached created
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] coveralls edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-617824003
## Pull Request Test Coverage Report for [Build 5860](https://coveralls.io/builds/30388872)
* **0** of **0** changed or added relevant lines in **0** files are covered.
* **6** unchanged lines in **2** files lost coverage.
* Overall coverage increased (+**0.001%**) to **27.467%**
---
| Files with Coverage Reduction | New Missed Lines | % |
| :-----|--------------|--: |
| [server-base/src/main/java/org/apache/kylin/rest/util/QueryRequestLimits.java](https://coveralls.io/builds/30388872/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Futil%2FQueryRequestLimits.java#L72) | 1 | 47.62% |
| [core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/30388872/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L439) | 5 | 77.81% |
<!-- | **Total:** | **6** | | -->
| Totals | [](https://coveralls.io/builds/30388872) |
| :-- | --: |
| Change from base [Build 5853](https://coveralls.io/builds/30348545): | 0.001% |
| Covered Lines: | 24317 |
| Relevant Lines: | 88532 |
---
##### 💛 - [Coveralls](https://coveralls.io)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus commented on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus commented on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619833363
## Cluster Env
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619862739
## Scale up Job Service
- https://kubernetes.io/docs/tasks/run-application/scale-stateful-set/
- Modify replica(from 1 to 3) and scale out job server
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# vim kylin-job/kylin-job-statefulset.yaml
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl apply -f kylin-job/
```
- Check state of pods
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get pod -n kylin-example
NAME READY STATUS RESTARTS AGE
kylin-job-0 2/2 Running 0 27s
kylin-job-1 2/2 Running 0 69s
kylin-job-2 2/2 Running 0 67s
kylin-memcached-0 1/1 Running 0 131m
kylin-query-0 2/2 Running 0 66m
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get pod -n kylin-example
NAME READY STATUS RESTARTS AGE
kylin-job-0 2/2 Running 0 34s
kylin-job-1 2/2 Running 0 76s
kylin-job-2 2/2 Running 0 74s
kylin-memcached-0 1/1 Running 0 131m
kylin-query-0 2/2 Running 0 66m
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get statefulset -n kylin-example
NAME READY AGE
kylin-job 3/3 115m
kylin-memcached 1/1 131m
kylin-query 1/1 66m
root@open-source:/home/ubuntu/example/deployment# docker ps | grep kylin-job-
da9c0cf239e4 a9acac7280fe "/usr/local/bin/dock…" 8 minutes ago Up 8 minutes k8s_filebeat_kylin-job-0_kylin-example_26c08037-0543-4e15-9ac2-54021266a52b_0
06ff2e6a2d3e 7ea27faa653f "sh -c 'cp $KYLIN_HO…" 8 minutes ago Up 8 minutes k8s_kylin_kylin-job-0_kylin-example_26c08037-0543-4e15-9ac2-54021266a52b_0
87062bb80c1b k8s.gcr.io/pause:3.2 "/pause" 8 minutes ago Up 8 minutes k8s_POD_kylin-job-0_kylin-example_26c08037-0543-4e15-9ac2-54021266a52b_0
91d96d1bee0f a9acac7280fe "/usr/local/bin/dock…" 8 minutes ago Up 8 minutes k8s_filebeat_kylin-job-2_kylin-example_93cc142c-0dc8-4e5e-886d-4408fac76c5d_0
d5d352acbb8d 7ea27faa653f "sh -c 'cp $KYLIN_HO…" 8 minutes ago Up 8 minutes k8s_kylin_kylin-job-2_kylin-example_93cc142c-0dc8-4e5e-886d-4408fac76c5d_0
ad05de976d8a k8s.gcr.io/pause:3.2 "/pause" 8 minutes ago Up 8 minutes k8s_POD_kylin-job-2_kylin-example_93cc142c-0dc8-4e5e-886d-4408fac76c5d_0
41de3f26e592 a9acac7280fe "/usr/local/bin/dock…" 8 minutes ago Up 8 minutes k8s_filebeat_kylin-job-1_kylin-example_b83623a1-c8a4-456f-95e3-fce19d78afb1_0
b3666e72bc7b 7ea27faa653f "sh -c 'cp $KYLIN_HO…" 8 minutes ago Up 8 minutes k8s_kylin_kylin-job-1_kylin-example_b83623a1-c8a4-456f-95e3-fce19d78afb1_0
1489fd6e04db k8s.gcr.io/pause:3.2 "/pause" 8 minutes ago Up 8 minutes k8s_POD_kylin-job-1_kylin-example_b83623a1-c8a4-456f-95e3-fce19d78afb1_0
root@open-source:/home/ubuntu/example/deployment#
```
- Instance List
<img width="1189" alt="image" src="https://user-images.githubusercontent.com/14030549/80381674-dcaca380-88d3-11ea-9b43-96ffc452b319.png">
- Test another build
<img width="1375" alt="image" src="https://user-images.githubusercontent.com/14030549/80383050-aa03aa80-88d5-11ea-8424-52825dd0d8cd.png">
- Force kill Active JobServer(Failover Test)
```shell
root@open-source:/home/ubuntu/example/deployment# docker kill b3666e72bc7b
b3666e72bc7b
```
- Check current active job server
<img width="1190" alt="image" src="https://user-images.githubusercontent.com/14030549/80383200-df0ffd00-88d5-11ea-8474-f21de4d0a6e3.png">
- Test a third build
<img width="1376" alt="image" src="https://user-images.githubusercontent.com/14030549/80384382-6447e180-88d7-11ea-9732-263d13d76115.png">
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus commented on a change in pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus commented on a change in pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#discussion_r419248580
##########
File path: kubernetes/example/deployment/kylin-job/kylin-job-service.yaml
##########
@@ -0,0 +1,36 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+apiVersion: v1
+kind: Service
+metadata:
+ name: kylin-job-svc
Review comment:
In my test, if I use only one service, the SQL query request will sent to job server, and the request will be rejected because job server won't answer such request.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619833363
## Prepare Configuration and Script Files
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619833363
## Prepare Docker Image
- Downloads required binary
```shell
xiaoxiang.yu:~/ $ ll Downloads/apache-kylin-3.0.1-bin-cdh57.tar [18:39:26]
-rw-r--r--@ 1 xiaoxiang.yu INC\Domain Users 253M 4 15 16:49 Downloads/apache-kylin-3.0.1-bin-cdh57.tar
xiaoxiang.yu:~/ $ ll Downloads/spark-2.3.2-bin-hadoop2.7.tgz [18:39:52]
-rwxrwxrwx 1 xiaoxiang.yu INC\Domain Users 215M 3 23 01:36 Downloads/spark-2.3.2-bin-hadoop2.7.tgz
```
- Downloads required jars
```shell
xiaoxiang.yu:~/ $ ll Downloads/mem* [18:40:05]
-rw-r--r--@ 1 xiaoxiang.yu INC\Domain Users 163K 4 27 18:45 Downloads/memcached-session-manager-2.1.1.jar
-rw-r--r--@ 1 xiaoxiang.yu INC\Domain Users 11K 4 27 18:45 Downloads/memcached-session-manager-tc7-2.1.1.jar
```
- Remove old image
```shell
xiaoxiang.yu:CDH57/ (kubernetes*) $ docker rmi hadoop-client:cdh57 [18:46:33]
Untagged: hadoop-client:cdh57
xiaoxiang.yu:CDH57/ (kubernetes*) $ docker images |grep hadoop-client | grep wc -l [18:47:31]
xiaoxiang.yu:CDH57/ (kubernetes*) $
xiaoxiang.yu:kylin-client/ (kubernetes*) $ docker rmi kylin-client:3.0.1-cdh57 [18:51:58]
Untagged: kylin-client:3.0.1-cdh57
Deleted: sha256:eb16424fc7cae2e4ca105e778290feec0ef370da2cc19e7df8b37ddd7166de57
Deleted: sha256:5c5d43dc5f482ea5e60f58902f577bbfe3bc22b37175d6a5c28b3f66646f9626
Deleted: sha256:9e509fdbf7d47b4cc28c45060809fecb888fab4c79b94f27f6826e78ed012431
Deleted: sha256:3c97b6cb2781d7a594f18cccf357ce90214d7e1317fabf28edfccf45896af502
Deleted: sha256:64b5601d1cb411c481382e6fe022f18dda4c5d070476a922bc4fd7531bad0846
Deleted: sha256:fe814862cb919a47a9c162506921c0ee93edca19c5d36006f92153487c4429ef
Deleted: sha256:1badb3b00a3f56ba66627e8c802b0a8f6a394bc2bc0dfa04d15a3085de9e34b9
Deleted: sha256:afdb1ccabfabd7f375a184c6a8aa3b0aeb54195e92fd67d5a225a9e6fc501469
Deleted: sha256:515eaf5fcbabf292b0e39d7fffd2d43f6c02d4b13af4f99be79027d120ce792d
Deleted: sha256:f88e58969b5f908502fbddf2e058a97d31ed31134c8020b0762eea1224a86383
Deleted: sha256:24b707ed4fe0ac52d4f3b32f9eac13fcd3698071a521f02ba1494d8cd7ec7646
Deleted: sha256:664b07d4d73e4a1b587983f0b80fd7500d93f1c1eedbf925adf5b94391583389
Deleted: sha256:75cdc77ae2aa8a0be849cd09961fc80a4d3856964fe91e265da2580b2d7dbd10
Deleted: sha256:78e1a1873e763dd81d65d0249869d546e39806fc9ae415d508bff7fac3a1c1eb
```
- Build Hadoop-Client image
```shell
xiaoxiang.yu:CDH57/ (kubernetes*) $ pwd [18:50:57]
/Users/xiaoxiang.yu/IntelliJ_IDEA_Project/MyKylin/kubernetes/docker/hadoop-client/CDH57
xiaoxiang.yu:CDH57/ (kubernetes*) $ cp ~/Downloads/spark-2.3.2-bin-hadoop2.7.tgz provided-binary [18:48:25]
xiaoxiang.yu:CDH57/ (kubernetes*) $ [18:48:37]
xiaoxiang.yu:CDH57/ (kubernetes*) $ [18:49:02]
xiaoxiang.yu:CDH57/ (kubernetes*) $ sh build-image.sh [18:49:02]
Sending build context to Docker daemon 225.9MB
Step 1/26 : FROM centos:7.3.1611
---> c5d48e81b986
... ... ...
Step 26/26 : ADD provided-binary/spark-${SPARK_VERSION}-bin-hadoop2.7.tgz .
---> ee576395a7f4
Successfully built ee576395a7f4
Successfully tagged hadoop-client:cdh57
```
- Build kylin-client image
```shell
xiaoxiang.yu:kylin-client/ (kubernetes*) $ cp /Users/xiaoxiang.yu/Downloads/memcached-session-manager-2.1.1.jar apache-kylin-3.0.1-bin-cdh57/tomcat/lib
xiaoxiang.yu:kylin-client/ (kubernetes*) $ cp /Users/xiaoxiang.yu/Downloads/memcached-session-manager-tc7-2.1.1.jar apache-kylin-3.0.1-bin-cdh57/tomcat/lib
xiaoxiang.yu:kylin-client/ (kubernetes*) $ sh build-image.sh [19:06:35]
Sending build context to Docker daemon 531.4MB
Step 1/23 : FROM hadoop-client:cdh57
---> ee576395a7f4
Step 2/23 : MAINTAINER Apache Kylin
---> Using cache
---> bd37b4be904f
Step 3/23 : ARG USER=apache_kylin
---> Using cache
---> 7e860f60cd6b
Step 4/23 : ARG USER_HOME=/home/${USER}
---> Using cache
---> 81d3ff20f301
Step 5/23 : ARG KYLIN_VERSION=apache-kylin-3.0.1-bin-cdh57
---> Using cache
---> 9d9622c9fab7
Step 6/23 : ARG HADOOP_CONF_HOME=/etc/hadoop/conf
---> Using cache
---> 7fa3676c6199
Step 7/23 : ARG HIVE_CONF_HOME=/etc/hive/conf
---> Using cache
---> 9e4679f4f582
Step 8/23 : ARG HBASE_CONF_HOME=/etc/hbase/conf
---> Using cache
---> 491b4dbe0cec
Step 9/23 : ARG SPARK_CONF_HOME=$KYLIN_HOME/hadoop-conf
---> Using cache
---> 1c2e39c268f4
Step 10/23 : ENV SPARK_HOME /opt/spark-2.3.2-bin-hadoop2.7
---> Using cache
---> db65c266f986
Step 11/23 : ENV KYLIN_HOME=$USER_HOME/kylin
---> Using cache
---> f8a17081a4a5
Step 12/23 : WORKDIR /tmp
---> Using cache
---> 5af436187fd4
Step 13/23 : RUN set -x && yum install -y openssh-clients cronie unzip sudo net-tools iftop tcpdump perf telnet bind-utils && yum clean all
---> Using cache
---> ef53f3155c7e
Step 14/23 : RUN set -x && groupadd -r $USER && useradd -r -m -g $USER $USER -d $USER_HOME && echo '$USER ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
---> Using cache
---> 1d0ee15484f7
Step 15/23 : COPY --chown=$USER:$USER $KYLIN_VERSION $KYLIN_HOME
---> Using cache
---> debe0d8920d5
Step 16/23 : RUN set -x && unzip -qq $KYLIN_HOME/tomcat/webapps/kylin.war -d $KYLIN_HOME/tomcat/webapps/kylin && chown -R $USER:$USER $KYLIN_HOME/tomcat/webapps/kylin && rm $KYLIN_HOME/tomcat/webapps/kylin.war && mkdir $SPARK_CONF_HOME && ln -s $HADOOP_CONF_HOME/core-site.xml $SPARK_CONF_HOME/core-site.xml && ln -s $HADOOP_CONF_HOME/hdfs-site.xml $SPARK_CONF_HOME/hdfs-site.xml && ln -s $HADOOP_CONF_HOME/yarn-site.xml $SPARK_CONF_HOME/yarn-site.xml && ln -s $HADOOP_CONF_HOME/hdfs-variable.xml $SPARK_CONF_HOME/hdfs-variable.xml && ln -s $HADOOP_CONF_HOME/yarn-variable.xml $SPARK_CONF_HOME/yarn-variable.xml && ln -s $HADOOP_CONF_HOME/federation-mapping.xml $SPARK_CONF_HOME/federation-mapping.xml && ln -s $HIVE_CONF_HOME/hive-site.xml $SPARK_CONF_HOME/hive-site.xml && ln -s $HBASE_CONF_HOME/hbase-site.xml $SPARK_CONF_HOME/hbase-site.xml && chown -R $USER:$USER $SPARK_CONF_HOME
---> Using cache
---> 40913e8eae5c
Step 17/23 : ENV TOOL_HOME=$USER_HOME/bin
---> Using cache
---> 889756023204
Step 18/23 : COPY bin $TOOL_HOME
---> Using cache
---> 01e1041cc7cb
Step 19/23 : COPY crontab.txt /tmp/crontab.txt
---> 9e084a6fd4b5
Step 20/23 : RUN /usr/bin/crontab -u $USER /tmp/crontab.txt && rm -rf /tmp/* && chmod 755 $TOOL_HOME/*
---> Running in 023036a3dff3
Removing intermediate container 023036a3dff3
---> 2279b966c55b
Step 21/23 : EXPOSE 7070
---> Running in ee0cc6d75bc6
Removing intermediate container ee0cc6d75bc6
---> 13400f20c95b
Step 22/23 : USER root
---> Running in 57b588a270c2
Removing intermediate container 57b588a270c2
---> 6d41c69a36a4
Step 23/23 : CMD ["sh", "-c", "$TOOL_HOME/bootstrap.sh server -d"]
---> Running in c786eea2ce0f
Removing intermediate container c786eea2ce0f
---> 7ea27faa653f
Successfully built 7ea27faa653f
Successfully tagged kylin-client:3.0.1-cdh57
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619833363
## Prepare Docker Image
- Downloads required binary
```shell
xiaoxiang.yu:~/ $ ll Downloads/apache-kylin-3.0.1-bin-cdh57.tar [18:39:26]
-rw-r--r--@ 1 xiaoxiang.yu INC\Domain Users 253M 4 15 16:49 Downloads/apache-kylin-3.0.1-bin-cdh57.tar
xiaoxiang.yu:~/ $ ll Downloads/spark-2.3.2-bin-hadoop2.7.tgz [18:39:52]
-rwxrwxrwx 1 xiaoxiang.yu INC\Domain Users 215M 3 23 01:36 Downloads/spark-2.3.2-bin-hadoop2.7.tgz
```
- Downloads required jars
```shell
xiaoxiang.yu:~/ $ ll Downloads/mem* [18:40:05]
-rw-r--r--@ 1 xiaoxiang.yu INC\Domain Users 163K 4 27 18:45 Downloads/memcached-session-manager-2.1.1.jar
-rw-r--r--@ 1 xiaoxiang.yu INC\Domain Users 11K 4 27 18:45 Downloads/memcached-session-manager-tc7-2.1.1.jar
```
- Remove old image
```shell
xiaoxiang.yu:CDH57/ (kubernetes*) $ docker rmi hadoop-client:cdh57 [18:46:33]
Untagged: hadoop-client:cdh57
xiaoxiang.yu:CDH57/ (kubernetes*) $ docker images |grep hadoop-client | grep wc -l [18:47:31]
xiaoxiang.yu:CDH57/ (kubernetes*) $
xiaoxiang.yu:kylin-client/ (kubernetes*) $ docker rmi kylin-client:3.0.1-cdh57 [18:51:58]
Untagged: kylin-client:3.0.1-cdh57
Deleted: sha256:eb16424fc7cae2e4ca105e778290feec0ef370da2cc19e7df8b37ddd7166de57
Deleted: sha256:5c5d43dc5f482ea5e60f58902f577bbfe3bc22b37175d6a5c28b3f66646f9626
Deleted: sha256:9e509fdbf7d47b4cc28c45060809fecb888fab4c79b94f27f6826e78ed012431
Deleted: sha256:3c97b6cb2781d7a594f18cccf357ce90214d7e1317fabf28edfccf45896af502
Deleted: sha256:64b5601d1cb411c481382e6fe022f18dda4c5d070476a922bc4fd7531bad0846
Deleted: sha256:fe814862cb919a47a9c162506921c0ee93edca19c5d36006f92153487c4429ef
Deleted: sha256:1badb3b00a3f56ba66627e8c802b0a8f6a394bc2bc0dfa04d15a3085de9e34b9
Deleted: sha256:afdb1ccabfabd7f375a184c6a8aa3b0aeb54195e92fd67d5a225a9e6fc501469
Deleted: sha256:515eaf5fcbabf292b0e39d7fffd2d43f6c02d4b13af4f99be79027d120ce792d
Deleted: sha256:f88e58969b5f908502fbddf2e058a97d31ed31134c8020b0762eea1224a86383
Deleted: sha256:24b707ed4fe0ac52d4f3b32f9eac13fcd3698071a521f02ba1494d8cd7ec7646
Deleted: sha256:664b07d4d73e4a1b587983f0b80fd7500d93f1c1eedbf925adf5b94391583389
Deleted: sha256:75cdc77ae2aa8a0be849cd09961fc80a4d3856964fe91e265da2580b2d7dbd10
Deleted: sha256:78e1a1873e763dd81d65d0249869d546e39806fc9ae415d508bff7fac3a1c1eb
```
- Build Hadoop-Client image
```shell
xiaoxiang.yu:CDH57/ (kubernetes*) $ pwd [18:50:57]
/Users/xiaoxiang.yu/IntelliJ_IDEA_Project/MyKylin/kubernetes/docker/hadoop-client/CDH57
xiaoxiang.yu:CDH57/ (kubernetes*) $ cp ~/Downloads/spark-2.3.2-bin-hadoop2.7.tgz provided-binary [18:48:25]
xiaoxiang.yu:CDH57/ (kubernetes*) $ [18:48:37]
xiaoxiang.yu:CDH57/ (kubernetes*) $ [18:49:02]
xiaoxiang.yu:CDH57/ (kubernetes*) $ sh build-image.sh [18:49:02]
Sending build context to Docker daemon 225.9MB
Step 1/26 : FROM centos:7.3.1611
---> c5d48e81b986
Step 2/26 : MAINTAINER Apache Kylin
---> Using cache
---> 7d31909dd19a
Step 3/26 : ENV JAVA_VERSION 1.8.0
---> Using cache
---> 596fd0b28b00
Step 4/26 : ENV SPARK_VERSION 2.3.2
---> Using cache
---> ec0263221e5b
Step 5/26 : ENV JAVA_HOME /usr/lib/jvm/java-${JAVA_VERSION}
---> Using cache
---> fcb50630b39a
Step 6/26 : ENV HADOOP_HOME /usr/lib/hadoop
---> Using cache
---> 729ba7b92158
Step 7/26 : ENV HIVE_HOME /usr/lib/hive
---> Using cache
---> 720cd72e9904
Step 8/26 : ENV HCAT_HOME /usr/lib/hive-hcatalog
---> Using cache
---> 7ef4bfdbc4cd
Step 9/26 : ENV HBASE_HOME /usr/lib/hbase
---> Using cache
---> 818e685192a5
Step 10/26 : ENV SPARK_HOME /opt/spark-${SPARK_VERSION}-bin-hadoop2.7
---> Using cache
---> ec3429565158
Step 11/26 : ENV PATH $PATH:$SPARK_HOME/bin:
---> Using cache
---> f2b052643286
Step 12/26 : ENV HADOOP_CONF_DIR /etc/hadoop/conf
---> Using cache
---> 0d76e82d405c
Step 13/26 : ENV HIVE_CONF_DIR /etc/hive/conf
---> Using cache
---> 1ad66c55a1de
Step 14/26 : ENV HBASE_CONF_DIR /etc/hbase/conf
---> Using cache
---> 91e260f7d120
Step 15/26 : ENV HIVE_CONF ${HIVE_CONF_DIR}
---> Using cache
---> 4321f4fe59f4
Step 16/26 : ENV HIVE_LIB ${HIVE_HOME}/lib
---> Using cache
---> 6b6b4e2a5ff4
Step 17/26 : RUN echo $'[cloudera-cdh5] \nname=Cloudera\'s Distribution for Hadoop, Version 5 \nbaseurl=https://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.7.6/ \ngpgkey =https://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/RPM-GPG-KEY-cloudera \ngpgcheck = 1' > /etc/yum.repos.d/cloudera-cdh5.repo
---> Using cache
---> ca7957895170
Step 18/26 : WORKDIR /opt
---> Using cache
---> c85839b823ef
Step 19/26 : RUN set -xeu && yum -y install java-1.8.0-openjdk-devel
---> Using cache
---> 70de99c7c287
Step 20/26 : RUN set -xeu && yum -y install krb5-workstation
---> Using cache
---> ee2d6cf2c56c
Step 21/26 : RUN set -xeu && yum -y install hadoop-client
---> Using cache
---> 7c332c100a62
Step 22/26 : RUN set -xeu && yum -y install hive hive-hcatalog
---> Using cache
---> 54d0c23e2781
Step 23/26 : RUN set -xeu && yum -y install hbase
---> Using cache
---> 57aa98165f79
Step 24/26 : RUN set -xeu && curl -o ${HIVE_HOME}/lib/hadoop-lzo-0.4.15.jar https://clojars.org/repo/hadoop-lzo/hadoop-lzo/0.4.15/hadoop-lzo-0.4.15.jar && curl -o ${HIVE_HOME}/lib/mysql-connector-java-5.1.24.jar https://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.24/mysql-connector-java-5.1.24.jar
---> Using cache
---> c9494c618436
Step 25/26 : RUN set -xeu && yum -q clean all && rm -rf /var/cache/yum && rm -rf /tmp/* /var/tmp/*
---> Using cache
---> 4a2d6490c98d
Step 26/26 : ADD provided-binary/spark-${SPARK_VERSION}-bin-hadoop2.7.tgz .
---> ee576395a7f4
Successfully built ee576395a7f4
Successfully tagged hadoop-client:cdh57
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619862166
## Create Kylin Query Service
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example kylin-query-config \
> --from-file=../config/kylin-query/kylin-kafka-consumer.xml \
> --from-file=../config/kylin-query/kylin_hive_conf.xml \
> --from-file=../config/kylin-query/kylin_job_conf.xml \
> --from-file=../config/kylin-query/kylin_job_conf_inmem.xml \
> --from-file=../config/kylin-query/kylin-server-log4j.properties \
> --from-file=../config/kylin-query/kylin-spark-log4j.properties \
> --from-file=../config/kylin-query/kylin-tools-log4j.properties \
> --from-file=../config/kylin-query/kylin.properties \
> --from-file=../config/kylin-query/setenv.sh \
> --from-file=../config/kylin-query/setenv-tool.sh \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:36:38.484197 10902 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/kylin-query-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl apply -f kylin-query
service/kylin-query-svc created
statefulset.apps/kylin-query created
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get pod -n kylin-example
NAME READY STATUS RESTARTS AGE
kylin-job-0 2/2 Running 0 28m
kylin-memcached-0 1/1 Running 0 44m
kylin-query-0 2/2 Running 0 11s
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get statefulset -n kylin-example
NAME READY AGE
kylin-job 1/1 28m
kylin-memcached 1/1 44m
kylin-query 1/1 19s
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] coveralls edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-617824003
## Pull Request Test Coverage Report for [Build 5858](https://coveralls.io/builds/30370682)
* **0** of **0** changed or added relevant lines in **0** files are covered.
* **6** unchanged lines in **2** files lost coverage.
* Overall coverage decreased (**-0.001%**) to **27.465%**
---
| Files with Coverage Reduction | New Missed Lines | % |
| :-----|--------------|--: |
| [stream-core/src/main/java/org/apache/kylin/stream/core/storage/columnar/ColumnarStoreCache.java](https://coveralls.io/builds/30370682/source?filename=stream-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstream%2Fcore%2Fstorage%2Fcolumnar%2FColumnarStoreCache.java#L146) | 1 | 56.82% |
| [core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/30370682/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L439) | 5 | 77.81% |
<!-- | **Total:** | **6** | | -->
| Totals | [](https://coveralls.io/builds/30370682) |
| :-- | --: |
| Change from base [Build 5853](https://coveralls.io/builds/30348545): | -0.001% |
| Covered Lines: | 24315 |
| Relevant Lines: | 88532 |
---
##### 💛 - [Coveralls](https://coveralls.io)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus commented on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus commented on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619833208
## Cluster Env
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619862064
## Test Kylin Job Service
- Instance List
<img width="1190" alt="image" src="https://user-images.githubusercontent.com/14030549/80372342-fcd56600-88c5-11ea-9552-a06878129169.png">
- Create sample model/cube
<img width="1437" alt="image" src="https://user-images.githubusercontent.com/14030549/80372032-6e60e480-88c5-11ea-8000-c6bffd328073.png">
- Submit job and build succeed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619861191
## Prepare Hadoop Configuration
- Download https://github.com/hit-lacus/kylin/tree/kubernetes
- Go to `kubernetes/example` dir.
```shell
root@open-source:/home/ubuntu/kylin-kubernetes# cd kubernetes/example/
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example# pwd
/home/ubuntu/kylin-kubernetes/kubernetes/example
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example# ll
total 24
drwxr-xr-x 4 ubuntu ubuntu 4096 Apr 27 11:40 ./
drwxr-xr-x 6 ubuntu ubuntu 4096 Apr 27 11:40 ../
drwxr-xr-x 7 ubuntu ubuntu 4096 Apr 27 11:40 config/
drwxr-xr-x 5 ubuntu ubuntu 4096 Apr 27 11:40 deployment/
-rwxr-xr-x 1 ubuntu ubuntu 7400 Apr 27 11:40 README.md*
```
- Replace hadoop configuration files with yours.
```
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/config/hadoop# ll
total 48
drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 27 11:40 ./
drwxr-xr-x 7 ubuntu ubuntu 4096 Apr 27 11:40 ../
-rwxr-xr-x 1 ubuntu ubuntu 4286 Apr 27 11:40 core-site.xml*
-rwxr-xr-x 1 ubuntu ubuntu 3492 Apr 27 11:40 hbase-site.xml*
-rwxr-xr-x 1 ubuntu ubuntu 2458 Apr 27 11:40 hdfs-site.xml*
-rwxr-xr-x 1 ubuntu ubuntu 6336 Apr 27 11:40 hive-site.xml*
-rwxr-xr-x 1 ubuntu ubuntu 5574 Apr 27 11:40 mapred-site.xml*
-rwxr-xr-x 1 ubuntu ubuntu 4305 Apr 27 11:40 yarn-site.xml*
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619862739
## Scale Out/In Job Service
- https://kubernetes.io/docs/tasks/run-application/scale-stateful-set/
- Modify replica(from 1 to 3)
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# vim kylin-job/kylin-job-statefulset.yaml
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl apply -f kylin-job/
```
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get pod -n kylin-example
NAME READY STATUS RESTARTS AGE
kylin-job-0 2/2 Running 0 27s
kylin-job-1 2/2 Running 0 69s
kylin-job-2 2/2 Running 0 67s
kylin-memcached-0 1/1 Running 0 131m
kylin-query-0 2/2 Running 0 66m
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get pod -n kylin-example
NAME READY STATUS RESTARTS AGE
kylin-job-0 2/2 Running 0 34s
kylin-job-1 2/2 Running 0 76s
kylin-job-2 2/2 Running 0 74s
kylin-memcached-0 1/1 Running 0 131m
kylin-query-0 2/2 Running 0 66m
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get statefulset -n kylin-example
NAME READY AGE
kylin-job 3/3 115m
kylin-memcached 1/1 131m
kylin-query 1/1 66m
root@open-source:/home/ubuntu/example/deployment# docker ps | grep kylin-job-
da9c0cf239e4 a9acac7280fe "/usr/local/bin/dock…" 8 minutes ago Up 8 minutes k8s_filebeat_kylin-job-0_kylin-example_26c08037-0543-4e15-9ac2-54021266a52b_0
06ff2e6a2d3e 7ea27faa653f "sh -c 'cp $KYLIN_HO…" 8 minutes ago Up 8 minutes k8s_kylin_kylin-job-0_kylin-example_26c08037-0543-4e15-9ac2-54021266a52b_0
87062bb80c1b k8s.gcr.io/pause:3.2 "/pause" 8 minutes ago Up 8 minutes k8s_POD_kylin-job-0_kylin-example_26c08037-0543-4e15-9ac2-54021266a52b_0
91d96d1bee0f a9acac7280fe "/usr/local/bin/dock…" 8 minutes ago Up 8 minutes k8s_filebeat_kylin-job-2_kylin-example_93cc142c-0dc8-4e5e-886d-4408fac76c5d_0
d5d352acbb8d 7ea27faa653f "sh -c 'cp $KYLIN_HO…" 8 minutes ago Up 8 minutes k8s_kylin_kylin-job-2_kylin-example_93cc142c-0dc8-4e5e-886d-4408fac76c5d_0
ad05de976d8a k8s.gcr.io/pause:3.2 "/pause" 8 minutes ago Up 8 minutes k8s_POD_kylin-job-2_kylin-example_93cc142c-0dc8-4e5e-886d-4408fac76c5d_0
41de3f26e592 a9acac7280fe "/usr/local/bin/dock…" 8 minutes ago Up 8 minutes k8s_filebeat_kylin-job-1_kylin-example_b83623a1-c8a4-456f-95e3-fce19d78afb1_0
b3666e72bc7b 7ea27faa653f "sh -c 'cp $KYLIN_HO…" 8 minutes ago Up 8 minutes k8s_kylin_kylin-job-1_kylin-example_b83623a1-c8a4-456f-95e3-fce19d78afb1_0
1489fd6e04db k8s.gcr.io/pause:3.2 "/pause" 8 minutes ago Up 8 minutes k8s_POD_kylin-job-1_kylin-example_b83623a1-c8a4-456f-95e3-fce19d78afb1_0
root@open-source:/home/ubuntu/example/deployment#
```
<img width="1189" alt="image" src="https://user-images.githubusercontent.com/14030549/80381674-dcaca380-88d3-11ea-9b43-96ffc452b319.png">
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619862064
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619861990
## Create Kylin Job Service
- Change configuration of memcached related(part-1)
```shell
# kylin.cache.memcached.hosts=cache-svc.kylin-example.svc.cluster.local:11211
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# cd ../config/
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/config# vim kylin-job/kylin.properties
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/config# vim kylin-query/kylin.properties
```
- Change configuration of memcached related(part-2)
```
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/config# vim tomcat/context.xml
```
```xml
<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
memcachedNodes="n1:cache-svc.kylin-example.svc.cluster.local:11211"
storageKeyPrefix="context"
requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
/>
```
- Create related configmap
```
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example hadoop-config \
> --from-file=../config/hadoop/core-site.xml \
> --from-file=../config/hadoop/hdfs-site.xml \
> --from-file=../config/hadoop/yarn-site.xml \
> --from-file=../config/hadoop/mapred-site.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:28.055160 4398 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/hadoop-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example hive-config \
> --from-file=../config/hadoop/hive-site.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:35.449192 4525 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/hive-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example hbase-config \
> --from-file=../config/hadoop/hbase-site.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:36.222142 4583 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/hbase-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example kylin-more-config \
> --from-file=../config/kylin-more/applicationContext.xml \
> --from-file=../config/kylin-more/ehcache.xml \
> --from-file=../config/kylin-more/ehcache-test.xml \
> --from-file=../config/kylin-more/kylinMetrics.xml \
> --from-file=../config/kylin-more/kylinSecurity.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:43.678445 4720 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/kylin-more-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example filebeat-config \
> --from-file=../config/filebeat/filebeat.yml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:51.686495 4901 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/filebeat-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example tomcat-config \
> --from-file=../config/tomcat/server.xml \
> --from-file=../config/tomcat/context.xml \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:09:52.431775 4926 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/tomcat-config configured
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl create configmap -n kylin-example kylin-job-config \
> --from-file=../config/kylin-job/kylin-kafka-consumer.xml \
> --from-file=../config/kylin-job/kylin_hive_conf.xml \
> --from-file=../config/kylin-job/kylin_job_conf.xml \
> --from-file=../config/kylin-job/kylin_job_conf_inmem.xml \
> --from-file=../config/kylin-job/kylin-server-log4j.properties \
> --from-file=../config/kylin-job/kylin-spark-log4j.properties \
> --from-file=../config/kylin-job/kylin-tools-log4j.properties \
> --from-file=../config/kylin-job/kylin.properties \
> --from-file=../config/kylin-job/setenv.sh \
> --from-file=../config/kylin-job/setenv-tool.sh \
> --dry-run -o yaml | kubectl apply -f -
W0427 12:10:03.250415 5141 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/kylin-job-config configured
```
- Start job service
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl apply -f kylin-job
service/kylin-job-svc created
statefulset.apps/kylin-job created
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get statefulset -n kylin-example
NAME READY AGE
kylin-job 1/1 12s
kylin-memcached 1/1 16m
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get pod -n kylin-example
NAME READY STATUS RESTARTS AGE
kylin-job-0 2/2 Running 0 23s
kylin-memcached-0 1/1 Running 0 16m
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# docker ps | grep kylin-job-
11517f6f98d6 a9acac7280fe "/usr/local/bin/dock…" 2 minutes ago Up 2 minutes k8s_filebeat_kylin-job-0_kylin-example_284fa3ea-5bce-437e-b769-92d6c938c628_0
fc0095a9f2d3 7ea27faa653f "sh -c 'cp $KYLIN_HO…" 2 minutes ago Up 2 minutes k8s_kylin_kylin-job-0_kylin-example_284fa3ea-5bce-437e-b769-92d6c938c628_0
58aa2566d7fb k8s.gcr.io/pause:3.2 "/pause" 2 minutes ago Up 2 minutes k8s_POD_kylin-job-0_kylin-example_284fa3ea-5bce-437e-b769-92d6c938c628_0
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619862275
## Test Kylin Query Service
- Instance List
<img width="1189" alt="image" src="https://user-images.githubusercontent.com/14030549/80373561-d4e70200-88c7-11ea-9387-55f8125c69b9.png">
- Test Query 1
<img width="1052" alt="image" src="https://user-images.githubusercontent.com/14030549/80374180-d8c75400-88c8-11ea-8c15-dccdb335fdb4.png">
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] coveralls edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-617824003
## Pull Request Test Coverage Report for [Build 5879](https://coveralls.io/builds/30575668)
* **0** of **0** changed or added relevant lines in **0** files are covered.
* **5** unchanged lines in **2** files lost coverage.
* Overall coverage remained the same at **27.32%**
---
| Files with Coverage Reduction | New Missed Lines | % |
| :-----|--------------|--: |
| [core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DefaultScheduler.java](https://coveralls.io/builds/30575668/source?filename=core-job%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fjob%2Fimpl%2Fthreadpool%2FDefaultScheduler.java#L194) | 2 | 80.23% |
| [core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/30575668/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L449) | 3 | 78.42% |
<!-- | **Total:** | **5** | | -->
| Totals | [](https://coveralls.io/builds/30575668) |
| :-- | --: |
| Change from base [Build 5865](https://coveralls.io/builds/30432594): | 0.0% |
| Covered Lines: | 24325 |
| Relevant Lines: | 89036 |
---
##### 💛 - [Coveralls](https://coveralls.io)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619862739
## Scale Out/In Query Service
- https://kubernetes.io/docs/tasks/run-application/scale-stateful-set/
- Modify replica
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# vim kylin-job/kylin-job-statefulset.yaml
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl apply -f kylin-job/
```
```shell
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get pod -n kylin-example
NAME READY STATUS RESTARTS AGE
kylin-job-0 2/2 Running 0 27s
kylin-job-1 2/2 Running 0 69s
kylin-job-2 2/2 Running 0 67s
kylin-memcached-0 1/1 Running 0 131m
kylin-query-0 2/2 Running 0 66m
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment#
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get pod -n kylin-example
NAME READY STATUS RESTARTS AGE
kylin-job-0 2/2 Running 0 34s
kylin-job-1 2/2 Running 0 76s
kylin-job-2 2/2 Running 0 74s
kylin-memcached-0 1/1 Running 0 131m
kylin-query-0 2/2 Running 0 66m
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# kubectl get statefulset -n kylin-example
NAME READY AGE
kylin-job 3/3 115m
kylin-memcached 1/1 131m
kylin-query 1/1 66m
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] coveralls commented on issue #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
coveralls commented on issue #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-617824003
## Pull Request Test Coverage Report for [Build 5840](https://coveralls.io/builds/30260676)
* **0** of **41** **(0.0%)** changed or added relevant lines in **1** file are covered.
* **42** unchanged lines in **3** files lost coverage.
* Overall coverage decreased (**-0.001%**) to **27.464%**
---
| Changes Missing Coverage | Covered Lines | Changed/Added Lines | % |
| :-----|--------------|--------|---: |
| [metrics-reporter-hive/src/main/java/org/apache/kylin/metrics/lib/impl/hive/HiveProducer.java](https://coveralls.io/builds/30260676/source?filename=metrics-reporter-hive%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetrics%2Flib%2Fimpl%2Fhive%2FHiveProducer.java#L104) | 0 | 41 | 0.0%
<!-- | **Total:** | **0** | **41** | **0.0%** | -->
| Files with Coverage Reduction | New Missed Lines | % |
| :-----|--------------|--: |
| [server-base/src/main/java/org/apache/kylin/rest/util/QueryRequestLimits.java](https://coveralls.io/builds/30260676/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Futil%2FQueryRequestLimits.java#L72) | 1 | 47.62% |
| [core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DefaultScheduler.java](https://coveralls.io/builds/30260676/source?filename=core-job%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fjob%2Fimpl%2Fthreadpool%2FDefaultScheduler.java#L194) | 2 | 80.23% |
| [metrics-reporter-hive/src/main/java/org/apache/kylin/metrics/lib/impl/hive/HiveProducer.java](https://coveralls.io/builds/30260676/source?filename=metrics-reporter-hive%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetrics%2Flib%2Fimpl%2Fhive%2FHiveProducer.java#L76) | 39 | 0% |
<!-- | **Total:** | **42** | | -->
| Totals | [](https://coveralls.io/builds/30260676) |
| :-- | --: |
| Change from base [Build 5830](https://coveralls.io/builds/30067502): | -0.001% |
| Covered Lines: | 24315 |
| Relevant Lines: | 88533 |
---
##### 💛 - [Coveralls](https://coveralls.io)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus commented on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus commented on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619862275
## Test Kylin Query Service
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619863981
## Clean up All Service
- https://kubernetes.io/docs/tasks/run-application/delete-stateful-set/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619833363
## Prepare Docker Image
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus commented on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus commented on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619862739
## Scale Out Query Service
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619833363
## Prepare Docker Image
- Downloads required binary
```shell
xiaoxiang.yu:~/ $ ll Downloads/apache-kylin-3.0.1-bin-cdh57.tar [18:39:26]
-rw-r--r--@ 1 xiaoxiang.yu INC\Domain Users 253M 4 15 16:49 Downloads/apache-kylin-3.0.1-bin-cdh57.tar
xiaoxiang.yu:~/ $ ll Downloads/spark-2.3.2-bin-hadoop2.7.tgz [18:39:52]
-rwxrwxrwx 1 xiaoxiang.yu INC\Domain Users 215M 3 23 01:36 Downloads/spark-2.3.2-bin-hadoop2.7.tgz
```
- Downloads required jars
```shell
xiaoxiang.yu:~/ $ ll Downloads/mem* [18:40:05]
-rw-r--r--@ 1 xiaoxiang.yu INC\Domain Users 163K 4 27 18:45 Downloads/memcached-session-manager-2.1.1.jar
-rw-r--r--@ 1 xiaoxiang.yu INC\Domain Users 11K 4 27 18:45 Downloads/memcached-session-manager-tc7-2.1.1.jar
```
- Remove old image
```shell
xiaoxiang.yu:CDH57/ (kubernetes*) $ docker rmi hadoop-client:cdh57 [18:46:33]
Untagged: hadoop-client:cdh57
xiaoxiang.yu:CDH57/ (kubernetes*) $ docker images |grep hadoop-client | grep wc -l [18:47:31]
xiaoxiang.yu:CDH57/ (kubernetes*) $
xiaoxiang.yu:kylin-client/ (kubernetes*) $ docker rmi kylin-client:3.0.1-cdh57 [18:51:58]
Untagged: kylin-client:3.0.1-cdh57
Deleted: sha256:eb16424fc7cae2e4ca105e778290feec0ef370da2cc19e7df8b37ddd7166de57
Deleted: sha256:5c5d43dc5f482ea5e60f58902f577bbfe3bc22b37175d6a5c28b3f66646f9626
Deleted: sha256:9e509fdbf7d47b4cc28c45060809fecb888fab4c79b94f27f6826e78ed012431
Deleted: sha256:3c97b6cb2781d7a594f18cccf357ce90214d7e1317fabf28edfccf45896af502
Deleted: sha256:64b5601d1cb411c481382e6fe022f18dda4c5d070476a922bc4fd7531bad0846
Deleted: sha256:fe814862cb919a47a9c162506921c0ee93edca19c5d36006f92153487c4429ef
Deleted: sha256:1badb3b00a3f56ba66627e8c802b0a8f6a394bc2bc0dfa04d15a3085de9e34b9
Deleted: sha256:afdb1ccabfabd7f375a184c6a8aa3b0aeb54195e92fd67d5a225a9e6fc501469
Deleted: sha256:515eaf5fcbabf292b0e39d7fffd2d43f6c02d4b13af4f99be79027d120ce792d
Deleted: sha256:f88e58969b5f908502fbddf2e058a97d31ed31134c8020b0762eea1224a86383
Deleted: sha256:24b707ed4fe0ac52d4f3b32f9eac13fcd3698071a521f02ba1494d8cd7ec7646
Deleted: sha256:664b07d4d73e4a1b587983f0b80fd7500d93f1c1eedbf925adf5b94391583389
Deleted: sha256:75cdc77ae2aa8a0be849cd09961fc80a4d3856964fe91e265da2580b2d7dbd10
Deleted: sha256:78e1a1873e763dd81d65d0249869d546e39806fc9ae415d508bff7fac3a1c1eb
```
- Build Hadoop-Client image
```shell
xiaoxiang.yu:CDH57/ (kubernetes*) $ pwd [18:50:57]
/Users/xiaoxiang.yu/IntelliJ_IDEA_Project/MyKylin/kubernetes/docker/hadoop-client/CDH57
xiaoxiang.yu:CDH57/ (kubernetes*) $ cp ~/Downloads/spark-2.3.2-bin-hadoop2.7.tgz provided-binary [18:48:25]
xiaoxiang.yu:CDH57/ (kubernetes*) $ [18:48:37]
xiaoxiang.yu:CDH57/ (kubernetes*) $ [18:49:02]
xiaoxiang.yu:CDH57/ (kubernetes*) $ sh build-image.sh [18:49:02]
Sending build context to Docker daemon 225.9MB
Step 1/26 : FROM centos:7.3.1611
---> c5d48e81b986
... ... ...
Step 26/26 : ADD provided-binary/spark-${SPARK_VERSION}-bin-hadoop2.7.tgz .
---> ee576395a7f4
Successfully built ee576395a7f4
Successfully tagged hadoop-client:cdh57
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus commented on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus commented on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619861773
## Create Memcached Service
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619833208
## Check Env
#### Hadoop Env
```shell
[root@cdh-master ~]# hadoop version
Hadoop 2.6.0-cdh5.7.6
Subversion http://github.com/cloudera/hadoop -r a3edabab129caaada2958ab32061aa0e52a3f7df
Compiled by jenkins on 2017-02-21T23:08Z
Compiled with protoc 2.5.0
From source with checksum 3de3a6f2894b326919cd7a9b4e573e
This command was run using /opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/jars/hadoop-common-2.6.0-cdh5.7.6.jar
```
#### Elasticsearch Env
```shell
[root@cdh-master elasticsearch-7.6.0]# bin/elasticsearch -V
future versions of Elasticsearch will require Java 11; your Java version from [/usr/java/jdk1.8.0_171/jre] does not meet this requirement
Version: 7.6.0, Build: default/tar/7f634e9f44834fbc12724506cc1da681b0c3b1e3/2020-02-06T00:09:00.449973Z, JVM: 1.8.0_171
```
#### Kubernetes Env
```shell
root@open-source:/home/ubuntu# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.4 LTS
Release: 18.04
Codename: bionic
root@open-source:/home/ubuntu# kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.1", GitCommit:"7879fc12a63337efff607952a323df90cdc7a335", GitTreeState:"clean", BuildDate:"2020-04-08T17:38:50Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.1", GitCommit:"7879fc12a63337efff607952a323df90cdc7a335", GitTreeState:"clean", BuildDate:"2020-04-08T17:30:47Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
root@open-source:/home/ubuntu# free -h
total used free shared buff/cache available
Mem: 19G 3.6G 11G 2.2M 4.5G 15G
Swap: 0B 0B 0B
root@open-source:/home/ubuntu# cat /proc/cpuinfo| grep "cpu cores"| uniq
cpu cores : 8
root@open-source:/home/ubuntu# uname -a
Linux open-source 4.15.0-96-generic #97-Ubuntu SMP Wed Apr 1 03:25:46 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
root@open-source:/home/ubuntu# kubectl get node -o=wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
open-source Ready master 11d v1.18.1 10.1.2.41 <none> Ubuntu 18.04.4 LTS 4.15.0-96-generic docker://19.3.8
root@open-source:/home/ubuntu# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-555fc8cc5c-jbt4h 1/1 Running 1 11d
calico-node-xzrbs 1/1 Running 1 11d
coredns-66bff467f8-q5j2q 1/1 Running 1 11d
coredns-66bff467f8-sxxfx 1/1 Running 1 11d
etcd-open-source 1/1 Running 1 11d
kube-apiserver-open-source 1/1 Running 1 11d
kube-controller-manager-open-source 1/1 Running 3 11d
kube-proxy-mnzq4 1/1 Running 1 11d
kube-scheduler-open-source 1/1 Running 3 11d
root@open-source:/home/ubuntu# kubectl cluster-info
Kubernetes master is running at https://10.1.2.41:6443
KubeDNS is running at https://10.1.2.41:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus commented on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus commented on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619863981
## Clean up All Service
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619861990
## Create Kylin Job Service
- Change configuration of memcached related(part-1)
```shell
# kylin.cache.memcached.hosts=cache-svc.kylin-example.svc.cluster.local:11211
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/deployment# cd ../config/
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/config# vim kylin-job/kylin.properties
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/config# vim kylin-query/kylin.properties
```
- Change configuration of memcached related(part-2)
```
root@open-source:/home/ubuntu/kylin-kubernetes/kubernetes/example/config# vim tomcat/context.xml
```
```xml
<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
memcachedNodes="n1:cache-svc.kylin-example.svc.cluster.local:11211"
storageKeyPrefix="context"
requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
/>
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#issuecomment-619862739
## Scale Out/In Query Service
- https://kubernetes.io/docs/tasks/run-application/scale-stateful-set/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org