Testground is still in early stage of development, so it is possible that:
Testground crashes
One of the underlying systems that Testground uses crashes (Kubernetes
, weave
, redis
, etc.)
Testground doesn't properly clean-up after a test run
etc.
Here are a few commands that could be helpful for you to inspect the state of your Kubernetes cluster and clean up after Testground:
Delete all pods that have the testground.plan=dht
label. This is useful in case you used the --run-cfg keep_service=true
setting on Testground.
$ kubectl delete pods -l testground.plan=dht --grace-period=0 --force
Restart the sidecar
daemon which manages networks for all testplans
$ kubectl delete pods -l name=testground-sidecar --grace-period=0 --force
You can check all running pods with
$ kubectl get pods -o wide
Another useful combination is watching
for pods that are not in Running
state or that are failing their health checks, with:
# watch all non-running podswatch 'kubectl get pods --all-namespaces -o wide | grep -v Running'​# watch all not-ready podswatch 'kubectl get pods --all-namespaces -o wide | grep "0\/1"'
$ kubectl logs <pod-id, e.g. tg-dht-c95b5>
$ kubectl port-forward svc/testground-infra-redis-master 6379:6379 &$ redis-cli -h localhost -p 6379