You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by xi...@apache.org on 2019/11/05 06:51:18 UTC
[incubator-pinot] branch aws-quickstart created (now 84c76da)
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a change to branch aws-quickstart
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
at 84c76da Adding example for aws
This branch includes the following new commits:
new 84c76da Adding example for aws
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[incubator-pinot] 01/01: Adding example for aws
Posted by xi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch aws-quickstart
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 84c76dafb4a34dd07ed952ee90d72075c9502a4f
Author: Xiang Fu <fx...@gmail.com>
AuthorDate: Mon Nov 4 21:02:16 2019 -0800
Adding example for aws
---
kubernetes/examples/helm/README.md | 81 ++++++++++++++++++++--
.../examples/helm/pinot-realtime-quickstart.yml | 36 ++++++++--
2 files changed, 108 insertions(+), 9 deletions(-)
diff --git a/kubernetes/examples/helm/README.md b/kubernetes/examples/helm/README.md
index 2c8c280..f1648cf 100644
--- a/kubernetes/examples/helm/README.md
+++ b/kubernetes/examples/helm/README.md
@@ -28,6 +28,69 @@
- Configure kubectl to connect to the Kubernetes cluster.
- Skip to [Section: How to setup a Pinot cluster for demo](#How to setup a Pinot cluster for demo) if a k8s cluster is already setup.
+
+## (Optional) Setup a Kubernetes cluster on Amazon Elastic Kubernetes Service (Amazon EKS)
+
+### (Optional) Create a new k8s cluster on AWS EKS
+
+- Install AWS CLI (<https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html#install-tool-bundled>)
+- Install AWS-IAM-AUTHENTICATOR (<https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html>)
+- Install eksctl (<https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html#installing-eksctl>)
+
+- Login to your AWS account.
+
+```bash
+aws configure
+```
+
+Note that environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` will override the aws configuration in file `~/.aws/credentials`.
+
+- Amazon EKS Prerequisites
+
+It requires Role ARN, VPC, Subnets, Security Groups to create an EKS cluster.
+
+Please follow wiki [Amazon EKS Prerequisites](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-console.html#eks-prereqs) to
+create Amazon Resource Name (ARN) and Amazon EKS Cluster VPC
+
+- Create an EKS cluster
+
+Please modify the parameters in the example command below:
+
+```bash
+eksctl create cluster \
+--name pinot-quickstart \
+--version 1.14 \
+--region us-west-2 \
+--nodegroup-name standard-workers \
+--node-type t3.small \
+--nodes 3 \
+--nodes-min 3 \
+--nodes-max 4 \
+--node-ami auto
+```
+You can monitor cluster status by command:
+
+```bash
+EKS_CLUSTER_NAME=pinot-quickstart
+aws eks describe-cluster --name ${EKS_CLUSTER_NAME}
+```
+
+Once the cluster is in `ACTIVE` status, it's ready to be used.
+
+### (Optional) How to connect to an existing cluster
+
+Simply run below command to get the credential for the cluster you just created or your existing cluster.
+
+```bash
+EKS_CLUSTER_NAME=pinot-quickstart
+aws eks update-kubeconfig --name ${EKS_CLUSTER_NAME}
+```
+
+To verify the connection, you can run
+```bash
+kubectl get nodes
+```
+
## (Optional) Setup a Kubernetes cluster on Google Kubernetes Engine(GKE)
### (Optional) Create a new k8s cluster on GKE
@@ -129,18 +192,26 @@ helm dependency update
### Start Pinot with Helm
+If cluster is just initialized, ensure helm is initialized by running:
+
+```bash
+helm init --service-account tiller
+```
+
+Then deploy pinot cluster by:
+
```bash
helm install --namespace "pinot-quickstart" --name "pinot" .
```
#### Troubleshooting
-Error: Please run below command if encountering issue:
+- Error: Please run below command if encountering issue:
```
Error: could not find tiller".
```
-Resolution:
+- Resolution:
```bash
kubectl -n kube-system delete deployment tiller-deploy
@@ -148,11 +219,11 @@ kubectl -n kube-system delete service/tiller-deploy
helm init --service-account tiller
```
-Error: Please run below command if encountering permission issue:
+- Error: Please run below command if encountering permission issue:
```Error: release pinot failed: namespaces "pinot-quickstart" is forbidden: User "system:serviceaccount:kube-system:default" cannot get resource "namespaces" in API group "" in the namespace "pinot-quickstart"```
-Resolution:
+- Resolution:
```bash
kubectl apply -f helm-rbac.yaml
@@ -196,7 +267,7 @@ Please use below script to do local port-forwarding and open Pinot query console
### How to clean up Pinot deployment
```bash
-kubectl delete -f pinot-example-loader.yml
+kubectl delete -f pinot-realtime-quickstart.yml
helm del --purge kafka
helm del --purge pinot
```
diff --git a/kubernetes/examples/helm/pinot-realtime-quickstart.yml b/kubernetes/examples/helm/pinot-realtime-quickstart.yml
index 7b47bb2..a121b11 100644
--- a/kubernetes/examples/helm/pinot-realtime-quickstart.yml
+++ b/kubernetes/examples/helm/pinot-realtime-quickstart.yml
@@ -414,21 +414,34 @@ data:
apiVersion: batch/v1
kind: Job
metadata:
- name: pinot-realtime-quickstart
+ name: pinot-realtime-quickstart-pinot-schema-creation
namespace: pinot-quickstart
spec:
template:
spec:
containers:
- - name: loading-data-to-kafka
- image: winedepot/pinot:0.1.13-SNAPSHOT
- args: [ "StreamAvroIntoKafka", "-avroFile", "sample_data/airlineStats_data.avro", "-kafkaTopic", "flights-realtime", "-kafkaBrokerList", "kafka:9092", "-zkAddress", "kafka-zookeeper:2181" ]
- name: pinot-add-example-schema
image: winedepot/pinot:0.1.13-SNAPSHOT
args: [ "AddSchema", "-schemaFile", "/var/pinot/examples/airlineStats_schema.json", "-controllerHost", "pinot-controller", "-controllerPort", "9000", "-exec" ]
volumeMounts:
- name: examples
mountPath: /var/pinot/examples
+ restartPolicy: OnFailure
+ volumes:
+ - name: examples
+ configMap:
+ name: examples
+ backoffLimit: 100
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+ name: pinot-realtime-quickstart-pinot-table-creation
+ namespace: pinot-quickstart
+spec:
+ template:
+ spec:
+ containers:
- name: pinot-add-example-realtime-table
image: winedepot/pinot:0.1.13-SNAPSHOT
args: [ "AddTable", "-filePath", "/var/pinot/examples/airlineStats_realtime_table_config.json", "-controllerHost", "pinot-controller", "-controllerPort", "9000", "-exec" ]
@@ -440,6 +453,21 @@ spec:
- name: examples
configMap:
name: examples
+ backoffLimit: 100
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+ name: pinot-realtime-quickstart-load-data-into-kafka
+ namespace: pinot-quickstart
+spec:
+ template:
+ spec:
+ containers:
+ - name: loading-data-to-kafka
+ image: winedepot/pinot:0.1.13-SNAPSHOT
+ args: [ "StreamAvroIntoKafka", "-avroFile", "sample_data/airlineStats_data.avro", "-kafkaTopic", "flights-realtime", "-kafkaBrokerList", "kafka:9092", "-zkAddress", "kafka-zookeeper:2181" ]
+ restartPolicy: OnFailure
backoffLimit: 3
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org