How to create a Kubernetes cluster for Testground

Search…

Testground is not coupled with AWS, but the infrastructure playbooks described in this section are targeted at AWS.

Requirements
First and foremost, you need an AWS account with API access.
Next, download and install all required software:
1.​kops >= 1.17.0
2.​terraform >= 0.12.21
3.​AWS CLI​
4.​helm >= 3.0
Setup AWS cloud credentials
1.​Generate your AWS IAM credentials.
2.​Configure the aws-cli tool with your credentials.
Generate a Testground SSH key for kops
It is used for the Kubernetes master and worker nodes
1
# generate ~/.ssh/testground_rsa    
2
#          ~/.ssh/testground_rsa.pub
3
​
4
$ ssh-keygen -t rsa -b 4096 -C "[email protected]" \
5
                            -f ~/.ssh/testground_rsa -q -P ""
Copied!
Create a bucket for kops state
This is similar to Terraform state bucket.
1
$ aws s3api create-bucket \
2
      --bucket <bucket_name> \
3
      --region <region> --create-bucket-configuration LocationConstraint=<region>
Copied!
Where:
<bucket_name> is an AWS account-wide unique bucket name to store this cluster's kops state, e.g. kops-backend-bucket-<your_username>.
<region> is an AWS region like eu-central-1 or us-west-2.
Configure cluster variables
a cluster name (for example name.k8s.local)
set AWS region
set AWS availability zone A (not region; for example us-west-2a [availability zone]) - used for master node and worker nodes
set AWS availability zone B (not region; for example us-west-2b [availability zone]) - used for more worker nodes
set kops state store bucket (the bucket we created in the section above)
set number of worker nodes
set master node instance type (read on best practices at https://kubernetes.io/docs/setup/best-practices/cluster-large/#size-of-master-and-master-components)
set worker node instance type
set location of your cluster SSH public key (for example ~/.ssh/testground_rsa.pub generated above)
set team and project name - these values are used as tags in AWS for cost allocation purposes
You might want to add them to your rc file (.zshrc, .bashrc, etc.), or to an .env.sh file that you source.
1
export CLUSTER_NAME=<desired kubernetes cluster name (e.g. mycluster.k8s.local)>
2
export DEPLOYMENT_NAME=<desired kubernetes deployment name>
3
export KOPS_STATE_STORE=s3://<kops state s3 bucket>
4
export AWS_REGION=<aws region, for example eu-central-1>
5
export ZONE_A=<aws availability zone, for example eu-central-1a>
6
export ZONE_B=<aws availability zone, for example eu-central-1b>
7
export WORKER_NODES=4
8
export MASTER_NODE_TYPE=c5.2xlarge
9
export WORKER_NODE_TYPE=c5.2xlarge
10
export PUBKEY=$HOME/.ssh/testground_rsa.pub
11
export TEAM=<your team name ; tag is used for cost allocation purposes>
12
export PROJECT=<your project name ; tag is used for cost allocation purposes>
Copied!
Setup required Helm chart repositories
1
$ helm repo add stable https://charts.helm.sh/stable
2
$ helm repo add bitnami https://charts.bitnami.com/bitnami
3
$ helm repo add influxdata https://helm.influxdata.com/
4
$ helm repo update
Copied!
Configure the Testground client
Create a .env.toml file in your $TESTGROUND_HOME and add your AWS region to the ["aws"] section.
Create cloud resources for the Kubernetes cluster
This will take about 10-15 minutes to complete.
Once you run this command, take some time to walk the dog, clean up around the office, or go get yourself some coffee! When you return, your shiny new Kubernetes cluster will be ready to run Testground plans.
1
$ git clone https://github.com/testground/infra
2
​
3
$ cd infra
4
​
5
$ ./k8s/install.sh ./k8s/cluster.yaml
Copied!
Destroy the cluster when you're done working on it
Do not forget to delete the cluster once you are done running test plans.
1
$ ./k8s/delete.sh
Copied!
Resizing the cluster
Edit the cluster state and change number of nodes.
1
$ kops edit ig nodes
Copied!
Apply the new configuration
1
$ kops update cluster $NAME --yes
Copied!
Wait for nodes to come up and for DaemonSets to be Running on all new nodes
1
$ watch 'kubectl get pods'
Copied!
Use a Kubernetes context for another cluster
kops lets you download the entire Kubernetes context config.
If you want to let other people on your team connect to your Kubernetes cluster, you need to give them the information.
1
$ kops export kubecfg --state $KOPS_STATE_STORE --name=$NAME
Copied!