Search for a command to run...
ElasticSearch H/A setup using ECK operator in EKS
I love to grease knots and bolts of SDLC, nurture the underlying infra, rightly automate, monitor systems and enable the dev teams to achieve more with less.
No comments yet. Be the first to comment.
Command Cheat sheet https://helm.sh/docs/intro/cheatsheet/ Add/Search/Install/Uninstall repo helm repo add bitnami https://charts.bitnami.com/bitnami helm repo update bitnami helm repo list helm install wordpress bitnami/wordpress helm list helm sta...
https://kops.sigs.k8s.io/getting_started/aws/ Pre-requisite DOMAIN=kops.balmanrawat.com.np aws route53 create-hosted-zone \ --name ${DOMAIN} \ --caller-reference kops-expriment ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --ou...
Containers have become the default way to package, ship, and run modern applications. They make deployments repeatable, portable, and scalable. But containers are not magic security boxes. A container
Details: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/ Pre-requisite A compatible Linux host. Verify the MAC address and product_uuid are unique for every node. Kubernetes uses these values to uniquely ident...
Docs Link1 Link2 Install Session Manger Plugin AWS SSM Proxy. Optional Start basic ssh connection aws ssm start-session \ --target i-askfsa8asld Port Forwarding aws ssm start-session \ --target i-00e68c055d1c0b087 \ --document-na...
This guide provides the information for deploying/managing highly available Elastic search setup in EKS using Elastic Cloud on Kubernetes Operator. In this, we will see how we can deploy the Elasticsearch version 8.8.1 using the ECK Operator.
The major source of information for this guide is the official ECK docs.
We will be deploying/managing Elasticsearch using the Elastic Cloud on Kubernetes Operator which is the latest recommended approach and managed by the Elastic team, the community itself. Please find more details here.
The current setup will create an elastic search with 3 master nodes and 2 data nodes and make them accessible via LoadBalancer in AWS EKS. It tries to distribute the elastic search nodes evenly among the existing k8s worker nodes.
The setup assumes that EKS cluster is up and running and has enough resources to provision the ES Cluster. The current setup creates 3 master nodes and 2 data nodes which require approx 2GB memory and 1 vCPU per ElasticSearch node.
Install custom resources for provisioning the kind: ElasticSearch resource. It creates other Custom Resources supported by ECK as well but can be ignored. Command
kubectl create -f https://download.elastic.co/downloads/eck/2.8.0/crds.yaml
Output
customresourcedefinition.apiextensions.k8s.io/agents.agent.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/apmservers.apm.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/beats.beat.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/elasticmapsservers.maps.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/elasticsearchautoscalers.autoscaling.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/elasticsearches.elasticsearch.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/enterprisesearches.enterprisesearch.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/kibanas.kibana.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/logstashes.logstash.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/stackconfigpolicies.stackconfigpolicy.k8s.elastic.co created
The below command installs operator and required roles to manage various operations. By default, it creates an elastic-system namespace and deploys resources under it. Command
kubectl apply -f https://download.elastic.co/downloads/eck/2.8.0/operator.yaml
Output
namespace/elastic-system created
serviceaccount/elastic-operator created
secret/elastic-webhook-server-cert created
configmap/elastic-operator created
clusterrole.rbac.authorization.k8s.io/elastic-operator created
clusterrole.rbac.authorization.k8s.io/elastic-operator-view created
clusterrole.rbac.authorization.k8s.io/elastic-operator-edit created
clusterrolebinding.rbac.authorization.k8s.io/elastic-operator created
service/elastic-webhook-server created
statefulset.apps/elastic-operator created
validatingwebhookconfiguration.admissionregistration.k8s.io/elastic-webhook.k8s.elastic.co created
Verify if the operator is up and running by looking at the logs. Command
kubectl -n elastic-system logs -f statefulset.apps/elastic-operator
Output
{"log.level":"info","@timestamp":"2023-06-19T04:38:33.355Z","log.logger":"manager.eck-operator","message":"Starting EventSource","service.version":"2.8.0+3940cf4d","service.type":"eck","ecs.version":"1.4.0","controller":"beat-controller","source":"kind source: *v1.Secret"}
{"log.level":"info","@timestamp":"2023-06-19T04:38:33.355Z","log.logger":"manager.eck-operator","message":"Starting EventSource","service.version":"2.8.0+3940cf4d","service.type":"eck","ecs.version":"1.4.0","controller":"beat-controller","source":"kind source: *v1.Secret"}
{"log.level":"info","@timestamp":"2023-06-19T04:38:33.355Z","log.logger":"manager.eck-operator","message":"Starting Controller","service.version":"2.8.0+3940cf4d","service.type":"eck","ecs.version":"1.4.0","controller":"beat-controller"}
...
This is an optional step as the elasticsearch.yaml manifest is configured to use the default gp2 storage class. If you want to override this behavior you can create your custom storage class and update its name in the elasticsearch.yaml file.
NOTE: Make sure the EBS CSI Plugin is active and the EKS Nodes have permission to manage volumes on behalf of the provisioner. Please include the following policy if missing.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:AttachVolume",
"ec2:CreateSnapshot",
"ec2:CreateTags",
"ec2:CreateVolume",
"ec2:DeleteSnapshot",
"ec2:DeleteTags",
"ec2:DeleteVolume",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeInstances",
"ec2:DescribeSnapshots",
"ec2:DescribeTags",
"ec2:DescribeVolumes",
"ec2:DescribeVolumesModifications",
"ec2:DetachVolume",
"ec2:ModifyVolume"
],
"Resource": "*"
}
]
}
Verify if the EBS CSI controller is actively running. Command
kubectl get pods -n kube-system -lapp=ebs-csi-controller
``
Output
```shell
NAME READY STATUS RESTARTS AGE
ebs-csi-controller-6876d9b86-d88kq 6/6 Running 0 5m49s
ebs-csi-controller-6876d9b86-t47wn 6/6 Running 0 5m49s
Now as our pre-requisite is met we can deploy the elastic search in the cluster using elasticsearch.yaml template
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: esearch
spec:
version: 8.8.1
http:
service:
spec:
type: LoadBalancer
tls:
selfSignedCertificate:
subjectAltNames:
- dns: localhost
- dns: es.esearch.app
#link: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-update-strategy.html
updateStrategy:
changeBudget:
maxSurge: 1
maxUnavailable: 1
#link: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-pod-disruption-budget.html
podDisruptionBudget:
spec:
minAvailable: 4
selector:
matchLabels:
elasticsearch.k8s.elastic.co/cluster-name: esearch
#Behind the scenes, ECK translates each NodeSet specified in the Elasticsearch resource into a StatefulSet in Kubernetes
#Upgrade Patterns: #Behind the scenes, ECK translates each NodeSet specified in the Elasticsearch resource into a StatefulSet in Kubernetes
nodeSets:
- name: masters
count: 3
config:
node.roles: ["master"]
# node.store.allow_mmap: false
xpack.ml.enabled: true
volumeClaimTemplates:
- metadata:
name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
storageClassName: gp2
podTemplate:
metadata:
labels:
app: elasticsearch
spec:
terminationGracePeriodSeconds: 150
#link: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-advanced-node-scheduling.html#k8s-affinity-options
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
elasticsearch.k8s.elastic.co/cluster-name: esearch
topologyKey: kubernetes.io/hostname
#https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-virtual-memory.html#k8s_using_an_init_container_to_set_virtual_memory
initContainers:
- name: sysctl
securityContext:
privileged: true
runAsUser: 0
command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
containers:
- name: elasticsearch
resources:
limits:
memory: 3Gi
requests:
memory: 2Gi
env:
- name: ES_JAVA_OPTS
value: "-Xms1g -Xmx1g"
- name: data
count: 2
config:
node.roles: ["data"]
# node.store.allow_mmap: false
xpack.ml.enabled: true
volumeClaimTemplates:
- metadata:
name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: gp2
podTemplate:
metadata:
labels:
app: elasticsearch
spec:
terminationGracePeriodSeconds: 150
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
elasticsearch.k8s.elastic.co/cluster-name: esearch
topologyKey: kubernetes.io/hostname
initContainers:
- name: sysctl
securityContext:
privileged: true
runAsUser: 0
command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
containers:
- name: elasticsearch
resources:
limits:
memory: 3Gi
requests:
memory: 2Gi
env:
- name: ES_JAVA_OPTS
value: "-Xms1g -Xmx1g"
k apply -f elasticsearch.yaml
Verify by listing the Elasticsearch resources
Initial Status
#When you create the cluster, there is unknown HEALTH status and the PHASE is ApplyingChanges. After a while, the PHASE turns into Ready, and HEALTH becomes green.
➜ kubectl get es
NAME HEALTH NODES VERSION PHASE AGE
esearch unknown 8.8.1 ApplyingChanges 22s
➜ kubectl get pods
NAME READY STATUS RESTARTS AGE
esearch-es-data-0 0/1 Init:0/3 0 23s
esearch-es-data-1 0/1 Init:0/3 0 23s
esearch-es-masters-0 0/1 Init:0/3 0 23s
esearch-es-masters-1 0/1 Init:0/3 0 23s
esearch-es-masters-2 0/1 Init:0/3 0 23s
➜ kubectl get sts
NAME READY AGE
esearch-es-data 0/2 28s
esearch-es-masters 0/3 29s
➜ elastic-search git:(main) ✗
Eventual Status
➜ kubectl get es
NAME HEALTH NODES VERSION PHASE AGE
esearch green 5 8.8.1 Ready 3m5s
➜ kubectl get pods
NAME READY STATUS RESTARTS AGE
esearch-es-data-0 1/1 Running 0 3m10s
esearch-es-data-1 1/1 Running 0 3m10s
esearch-es-masters-0 1/1 Running 0 3m10s
esearch-es-masters-1 1/1 Running 0 3m10s
esearch-es-masters-2 1/1 Running 0 3m10s
➜ kubectl get sts
NAME READY AGE
esearch-es-data 2/2 3m16s
esearch-es-masters 3/3 3m17s
To access the cluster properly you will need es endpoint, es password and the ca certificate. Please use the below commands to get the information:
#Set Elasticsearch name
export ES_NAME=esearch
#Get Service Public endpoint
ENDPOINT=$(kubectl get svc ${ES_NAME}-es-http -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
#Get ElasticSearch User Password
PASSWORD=$(kubectl get secret ${ES_NAME}-es-elastic-user -o go-template='{{.data.elastic | base64decode}}')
#Get ES CA Certificate
kubectl get secret ${ES_NAME}-es-http-certs-public -o go-template='{{index .data "tls.crt" | base64decode }}' > ca.cert
Once we have the endpoint, certificates and password we can not connect with Elasticsearch cluster using https. Please use the below verification command to see if everything is set properly. Command
curl -X GET --cacert ca.cert -u "elastic:${PASSWORD}" "https://${ENDPOINT}:9200"
Output
{
"name" : "esearch-es-masters-2",
"cluster_name" : "esearch",
"cluster_uuid" : "jygZP_KOS_6ELM5WjSe9cQ",
"version" : {
"number" : "8.8.1",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "f8edfccba429b6477927a7c1ce1bc6729521305e",
"build_date" : "2023-06-05T21:32:25.188464208Z",
"build_snapshot" : false,
"lucene_version" : "9.6.0",
"minimum_wire_compatibility_version" : "7.17.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "You Know, for Search"
}
If we need to revert all the changes we can do it in the following order.
kubectl delete -f elasticsearch.yaml
kubectl delete -f https://download.elastic.co/downloads/eck/2.8.0/operator.yaml
kubectl delete -f https://download.elastic.co/downloads/eck/2.8.0/crds.yaml
Hurray 🎉 This concludes the setup of the elastic search using the ECK operator. For any further details and customization please follow the details section below.
There are multiple choices and configuration settings that we made when defining the elasticsearch.yaml manifest. Please find details about them in the section below:
While running any kind of updates/upgrades we want to make sure we are not under the capacity or overcapacity. maxsurge specifies a maximum addition number of nodes than our existing pool of es nodes, this will have us prevent excessive utilization of resources during the upgrades. maxUnavailable specifies a minimum number of nodes that can be available at a time.
...
updateStrategy:
changeBudget:
maxSurge: 1 #creates only one new node at a time.
maxUnavailable: 1 #using 1 because we only have 2 data nodes. Can be increased if we have more data/master nodes. Low value increases stability but increases deployment time i.e we need to find the right balance.
...
During the upgrades, maintenance and failure of any nodes we still want to maintain the pool of the nodes such that the es cluster continues to serve the traffic. We are using podDistributionBudget where minAvailable is set to 4 and the selector matches the es nodes.
...
podDisruptionBudget:
spec:
minAvailable: 4 #using 4 as we have 3 master nodes and 2 data nodes.
selector:
matchLabels:
elasticsearch.k8s.elastic.co/cluster-name: "{{.metadata.name}}"
...
The config tries to ensure the es nodes are distributed uniformly across the worker nodes for better availability and resiliency using topologyKey: kubernetes.io/hostname.
...
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution: #using preferred instead of required as there always won't be unique k8s node per es node and using required might make es nodes unschedulable
- weight: 100 #give high priority for this rule
podAffinityTerm:
labelSelector:
matchLabels:
elasticsearch.k8s.elastic.co/cluster-name: "{{.metadata.name}}"
topologyKey: kubernetes.io/hostname #this ensures distribution of es nodes based on the hostname
...
For this scenario the configuration specifies 3 master node(for minimal H/A) and 2 data nodes(for minimal H/A). For the sake of simplicity other roles are ignored.
Default values for virtual address space on Linux distributions are too low for Elasticsearch to work properly, which may result in out-of-memory exceptions. For production workloads, it is strongly recommended to increase the kernel setting vm.max_map_count to 262144.
The storage class used in the volumeClaimTemplates template is storageClassName: gp2 default storage class created when EKS cluster is created which can be changed if you have different requirements for different roles.
JVM flags can be overridden using the environment variables and this can largely impact the performance of the es cluster overall. It should be different for different roles for now the basic configuration is JVM start memory: 1GB and JVM max: 1GB.
...
env:
- name: ES_JAVA_OPTS
value: "-Xms1g -Xmx1g"
...
The termination grace period for pod is used to give enough time to gracefully exit the es node during any kinds of upgrades and maintenance tasks. Current scenario specific 150 which should be enough but can highly differ based on the node type and size of traffic/data.
...
spec:
terminationGracePeriodSeconds: 300
...
Many more settings can be configured to tune the cluster setup to our needs. Please find more details in this API reference.