https://helm.sh/docs/intro/cheatsheet/

Add/Search/Install/Uninstall repo

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update bitnami
helm repo list

helm install wordpress bitnami/wordpress
helm list
helm status wordpress
helm get all wordpress
helm get values wordpress
helm uninstall wordpress #--keep-history uninstall while keeping history

Search Repositories

helm search repo
helm search hub
helm search bitnami
helm search repo wordpress
helm search hub wordpress

Installation Order Helm

• namespace

• NetworkPolicy

• ResourceQuota

• LimitRange

• PodSecurityPolicy

• PodDisruptionBudget

• ServiceAccount

• Secret

• SecretList

• ConfigMap

• StorageClass

• PersistentVolume

• PersistentVolumeClaim

• CustomResourceDefinition

• ClusterRole

• ClusterRoleList

• ClusterRoleBinding

• ClusterRoleBindingList

• Role

• RoleList

• RoleBinding

• RoleBindingList

• Service

• DaemonSet

• Pod

• ReplicationController

• ReplicaSet

• Deployment

• HorizontalPodAutoscaler

• StatefulSet

• Job

• CronJob

• Ingress

• APIService

Customizing Chart before Installing

helm show values bitnami/wordpress

More Installation Methods

The helm install command can install from several sources:

• A chart repository (as we've seen above)

• A local chart archive (helm install foo foo-0.1.1.tgz)

• An unpacked chart directory (helm install foo path/to/foo)

• A full URL (helm install foo https://example.com/charts/foo-1.2.3.tgz)

'helm upgrade' and 'helm rollback':

An upgrade takes an existing release and upgrades it according to the information you provide. Because Kubernetes charts can be large and complex, Helm tries to perform the least invasive upgrade.

NOTE: It will only update things that have changed since the last release.

To see new upgrade too place:

helm get values wordpress

To rollback:

helm rollback wordpress 1

Helpful Options for Install/Upgrade/Rollback

• --timeout

• --wait It will wait for as long as the --timeout value. If timeout is reached, the release will be marked as FAILED.

• --no-hooks: This skips running hooks for the command

• --recreate-pods (only available for upgrade and rollback)

Creating your own chart

helm create webhotel
helm package webhotel
helm install my-hotel ./webhotel

## debugging chart
helm install --debug --dry-run goodly-guppy ./webhotel

NOTES:

• By virtue of the fact that this file is in the mychart/templates/ directory, it will be sent through the template engine.

• We recommend using the extension .yaml for YAML files and .tpl for helpers.

• The template directive {{ .Release.Name }} injects the release name into the template.

• The Release object is one of the built-in objects for Helm

Built-in Objects

Objects are passed into a template from the template engine. And your code can pass objects around. There are even a few ways to create new objects within your templates, like with the tuple .

Objects can be simple, and have just one value. Or they can contain other objects or functions. For example, the Release object contains several objects (like Release.Name) and the Files object has a few functions.

NOTE: The built-in values always begin with a capital letter.

Release Object

This object describes the release itself. It has several objects inside of it:

• Release.Name

• Release.Namespace

• Release.IsUpgrade: true if the current operation is an upgrade or rollback.

• Release.IsInstall: true if the current operation is an install.

• Release.Revision: The revision number for this release. On install, this is 1, and it is incremented with each upgrade and rollback.

• Release.Service: The service that is rendering the present template. On Helm, this is always Helm.

Values Object

Values passed into the template from the values.yaml file and from user-supplied files. By default, Values is empty.

Chart Object

The contents of the Chart.yaml file. Any data in Chart.yaml will be accessible here. For example {{ .Chart.Name }}-{{ .Chart.Version }} will print out the mychart-0.1.0.

The available fields are listed in the Charts Guide

Subchart Object

This provides access to the scope (.Values, .Charts, .Releases etc.) of subcharts to the parent. For example .Subcharts.mySubChart.myValue to access the myValue in the mySubChart chart.

Files Object

This provides access to all non-special files in a chart. While you cannot use it to access templates, you can use it to access other files in the chart. See the section Accessing Files for more.

• • Files.Get is a function for getting a file by name (.Files.Get config.ini)

    • Files.GetBytes is a function for getting the contents of a file as an array of bytes instead of as a string. This is useful for things like images.

    • Files.Glob is a function that returns a list of files whose names match the given shell glob pattern.

    • Files.Lines is a function that reads a file line-by-line. This is useful for iterating over each line in a file.

    • Files.AsSecrets is a function that returns the file bodies as Base 64 encoded strings.

    • Files.AsConfig is a function that returns file bodies as a YAML map.

Capabilities Object

This provides information about what capabilities the Kubernetes cluster supports.

• • Capabilities.APIVersions is a set of versions.

    • Capabilities.APIVersions.Has $version indicates whether a version (e.g., batch/v1) or resource (e.g., apps/v1/Deployment) is available on the cluster.

    • Capabilities.KubeVersion and Capabilities.KubeVersion.Version is the Kubernetes version.

    • Capabilities.KubeVersion.Major is the Kubernetes major version.

    • Capabilities.KubeVersion.Minor is the Kubernetes minor version.

    • Capabilities.HelmVersion is the object containing the Helm Version details, it is the same output of helm version.

    • Capabilities.HelmVersion.Version is the current Helm version in semver format.

    • Capabilities.HelmVersion.GitCommit is the Helm git sha1.

    • Capabilities.HelmVersion.GitTreeState is the state of the Helm git tree.

    • Capabilities.HelmVersion.GoVersion is the version of the Go compiler used.

Template Object

Contains information about the current template that is being executed

• • Template.Name: A namespaced file path to the current template (e.g. mychart/templates/mytemplate.yaml)

    • Template.BasePath: The namespaced path to the templates directory of the current chart (e.g. mychart/templates).

Values Files

This object provides access to values passed into the chart. Its contents come from multiple sources:

• The values.yaml file in the chart

• If this is a subchart, the values.yaml file of a parent chart

• A values file if passed into helm install or helm upgrade with the -f flag (helm install -f myvals.yaml ./mychart)

• Individual parameters passed with --set (such as helm install --set foo=bar ./mychart)

NOTE: values.yaml is the default, which can be overridden by a parent chart's values.yaml, which can in turn be overridden by a user-supplied values file, which can in turn be overridden by --set parameters.

Deleting a default:

helm install stable/drupal --set livenessProbe.exec.command=[cat,docroot/CHANGELOG.txt] --set livenessProbe.httpGet=null

Template Functions and Pipelines

So far, we've seen how to place information into a template. But that information is placed into the template unmodified. Sometimes we want to transform the supplied data in a way that makes it more useable to us.

Template Function

List of available functions: https://helm.sh/docs/chart_template_guide/function_list/

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-configmap
data:
  myvalue: "Hello World"
  drink: {{ quote .Values.favorite.drink }}
  food: {{ quote .Values.favorite.food }}

Template Function syntax

functionName arg1 arg2...

Pipelines

Pipelines are an efficient way of getting several things done in sequence. Let's rewrite the above example using a pipeline.

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-configmap
data:
  myvalue: "Hello World"
  drink: {{ .Values.favorite.drink | repeat 5 | quote }}
  food: {{ .Values.favorite.food | upper | quote }}

### above snippet will be translated to below👇 yaml
# Source: mychart/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: melting-porcup-configmap
data:
  myvalue: "Hello World"
  drink: "coffeecoffeecoffeecoffeecoffee"
  food: "PIZZA"

Using Default Function

drink: {{ .Values.favorite.drink | default "tea" | quote }}
drink: {{ .Values.favorite.drink | default (printf "%s-tea" (include "fullname" .)) }}

In some places, an if conditional guard may be better suited than default.\

Using the lookup function

The lookup function can be used to look up resources in a running cluster. The synopsis of the lookup function is lookup apiVersion, kind, namespace, name -> resource or resource list

Example:

(lookup "v1" "Namespace" "" "mynamespace").metadata.annotations

NOTE: To test lookup against a running cluster, helm template|install|upgrade|delete|rollback --dry-run=server should be used

Operators are functions

For templates, the operators (eq, ne, lt, gt, and, or and so on) are all implemented as functions. In pipelines, operations can be grouped with parentheses ((, and )).

Flow Control

Helm's template language provides the following control structures:

• if/else for creating conditional blocks

• with to specify a scope

• range, which provides a "for each"-style loop

If/Else

{{ if PIPELINE }}
  # Do something
{{ else if OTHER PIPELINE }}
  # Do something else
{{ else }}
  # Default case
{{ end }}

## Example
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-configmap
data:
  myvalue: "Hello World"
  drink: {{ .Values.favorite.drink | default "tea" | quote }}
  food: {{ .Values.favorite.food | upper | quote }}
  {{ if eq .Values.favorite.drink "coffee" }}mug: "true"{{ end }}

### Values
apiVersion: v1
kind: ConfigMap
metadata:
  name: eyewitness-elk-configmap
data:
  myvalue: "Hello World"
  drink: "coffee"
  food: "PIZZA"
  mug: "true"

Modifying scope using with

The next control structure to look at is the with action. This controls variable scoping. Recall that . is a reference to the current scope. So .Values tells the template to find the Values object in the current scope.

###Syntax:
{{ with PIPELINE }}
  # restricted scope
{{ end }}

### Example
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-configmap
data:
  myvalue: "Hello World"
  {{- with .Values.favorite }}
  drink: {{ .drink | default "tea" | quote }}
  food: {{ .food | upper | quote }}
  {{- end }}
  release: {{ .Release.Name }}

Looping with the range action

##Example
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-configmap
data:
  myvalue: "Hello World"
  {{- with .Values.favorite }}
  drink: {{ .drink | default "tea" | quote }}
  food: {{ .food | upper | quote }}
  {{- end }}
  toppings: |-
    {{- range .Values.pizzaToppings }}
    - {{ . | title | quote }}
    {{- end }}   

## Tuple example
sizes: |-
    {{- range tuple "small" "medium" "large" }}
    - {{ . }}
    {{- end }}

In addition to lists and tuples, range can be used to iterate over collections that have a key and a value (like a map or dict).

Variables

In Helm templates, a variable is a named reference to another object. It follows the form $name. Variables are assigned with a special assignment operator: :=. We can rewrite the above to use a variable for Release.Name.

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-configmap
data:
  myvalue: "Hello World"
  {{- $relname := .Release.Name -}}
  {{- with .Values.favorite }}
  drink: {{ .drink | default "tea" | quote }}
  food: {{ .food | upper | quote }}
  release: {{ $relname }}
  {{- end }}

ariables are particularly useful in range loops. They can be used on list-like objects to capture both the index and the value:

  toppings: |-
    {{- range $index, $topping := .Values.pizzaToppings }}
      {{ $index }}: {{ $topping }}
    {{- end }}

#result
toppings: |-
      0: mushrooms
      1: cheese
      2: peppers
      3: onions

For data structures that have both a key and a value, we can use range to get both. For example, we can loop through .Values.favorite like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-configmap
data:
  myvalue: "Hello World"
  {{- range $key, $val := .Values.favorite }}
  {{ $key }}: {{ $val | quote }}
  {{- end }}

#result
apiVersion: v1
kind: ConfigMap
metadata:
  name: eager-rabbit-configmap
data:
  myvalue: "Hello World"
  drink: "coffee"
  food: "pizza"

Variables are normally not "global". They are scoped to the block in which they are declared. Earlier, we assigned $relname in the top level of the template. That variable will be in scope for the entire template. But in our last example, $key and $val will only be in scope inside of the {{ range... }}{{ end }} block.

However, there is one variable that is always global - $ -this variable will always point to the root context.

Named Template

It is time to move beyond one template, and begin to create others. In this section, we will see how to define named templates in one file, and then use them elsewhere. A named template (sometimes called a partial or a subtemplate) is simply a template defined inside of a file, and given a name. We'll see two ways to create them, and a few different ways to use them.

NOTE: An important detail to keep in mind when naming templates: template names are global. If you declare two templates with the same name, whichever one is loaded last will be the one used. Because templates in subcharts are compiled together with top-level templates, you should be careful to name your templates with chart-specific names.

Partials and _ files

• Most files in templates/ are treated as if they contain Kubernetes manifests

• The NOTES.txt is one exception

• But files whose name begins with an underscore (_) are assumed to not have a manifest inside. These files are not rendered to Kubernetes object definitions, but are available everywhere within other chart templates for use.

Declaring and using templates with define and template

{{- define "MY.NAME" }}
  # body of template here
{{- end }}

### Example
{{- define "mychart.labels" }}
  labels:
    generator: helm
    date: {{ now | htmlDate }}
{{- end }}

Now we can embed this template inside of our existing ConfigMap, and then include it with the template action:

{{- define "mychart.labels" }}
  labels:
    generator: helm
    date: {{ now | htmlDate }}
{{- end }}
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-configmap
  {{- template "mychart.labels" }}
data:
  myvalue: "Hello World"
  {{- range $key, $val := .Values.favorite }}
  {{ $key }}: {{ $val | quote }}
  {{- end }}

## Result
apiVersion: v1
kind: ConfigMap
metadata:
  name: running-panda-configmap
  labels:
    generator: helm
    date: 2016-11-02
data:
  myvalue: "Hello World"
  drink: "coffee"
  food: "pizza"

NOTE: Note: a define does not produce output unless it is called with a template, as in this example.

Conventionally, Helm charts put these templates inside of a partials file, usually _helpers.tpl. Let's move this function there:

The include function

Because template is an action, and not a function, there is no way to pass the output of a template call to other functions; the data is simply inserted inline.

To work around this case, Helm provides an alternative to template that will import the contents of a template into the present pipeline where it can be passed along to other functions in the pipeline.

Here's the example above, corrected to use indent to indent the mychart.app template correctly

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-configmap
  labels:
{{ include "mychart.app" . | indent 4 }}
data:
  myvalue: "Hello World"
  {{- range $key, $val := .Values.favorite }}
  {{ $key }}: {{ $val | quote }}
  {{- end }}
{{ include "mychart.app" . | indent 2 }}

NOTE: It is considered preferable to use include over template in Helm templates simply so that the output formatting can be handled better for YAML documents.

Accessing Files Inside Templates

https://helm.sh/docs/chart_template_guide/accessing_files/

Sometimes it is desirable to import a file that is not a template and inject its contents without sending the contents through the template renderer. Helm provides access to files through the .Files object.

NOTES

• It is okay to add extra files to your Helm chart. These files will be bundled. Be careful, though. Charts must be smaller than 1M because of the storage limitations of Kubernetes objects.

• Some files cannot be accessed through the .Files object, usually for security reasons.

  • Files in templates/ cannot be accessed.

  • Files excluded using .helmignore cannot be accessed.

  • Files outside of a Helm application subchart, including those of the parent, cannot be access

Basic Examples

onfig1.toml:

message = Hello from config 1

config2.toml:

message = This is config 2

config3.toml:

message = Goodbye from config 3

Each of these is a simple TOML file (think old-school Windows INI files). We know the names of these files, so we can use a range function to loop through them and inject their contents into our ConfigMap.

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-configmap
data:
  {{- $files := .Files }}
  {{- range tuple "config1.toml" "config2.toml" "config3.toml" }}
  {{ . }}: |-
        {{ $files.Get . }}
  {{- end }}

### Result
apiVersion: v1
kind: ConfigMap
metadata:
  name: quieting-giraf-configmap
data:
  config1.toml: |-
        message = Hello from config 1

  config2.toml: |-
        message = This is config 2

  config3.toml: |-
        message = Goodbye from config 3

Creating a NOTES.txt File

https://helm.sh/docs/chart_template_guide/notes_files/

Subcharts and Global Values

Before we dive into the code, there are a few important details to learn about application subcharts.

1. A subchart is considered "stand-alone", which means a subchart can never explicitly depend on its parent chart.

2. For that reason, a subchart cannot access the values of its parent.

3. A parent chart can override values for subcharts.

4. Helm has a concept of global values that can be accessed by all charts.

Creating a Subchart

For these exercises, we'll start with the mychart/ chart we created at the beginning of this guide, and we'll add a new chart inside of it.

cd mychart/charts
helm create mysubchart
Creating mysubchart
rm -rf mysubchart/templates/*

Overriding Values from a Parent Values.yaml

favorite:
  drink: coffee
  food: pizza
pizzaToppings:
  - mushrooms
  - cheese
  - peppers
  - onions

##Chart name
mysubchart:
  dessert: ice cream

Global Chart Values

Global values are values that can be accessed from any chart or subchart by exactly the same name. Globals require explicit declaration. You can't use an existing non-global as if it were a global.

The Values data type has a reserved section called Values.global where global values can be set. Let's set one in our mychart/values.yaml file.

favorite:
  drink: coffee
  food: pizza
pizzaToppings:
  - mushrooms
  - cheese
  - peppers
  - onions

mysubchart:
  dessert: ice cream

global:
  salad: caesar

Example Config Map

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-configmap
data:
  salad: {{ .Values.global.salad }}

Sharing Templates with Subcharts

Parent charts and subcharts can share templates. Any defined block in any chart is available to other charts.

For example, we can define a simple template like this:

{{- define "labels" }}from: mychart{{ end }}

Recall how the labels on templates are globally shared. Thus, the labels chart can be included from any other chart.

While chart developers have a choice between include and template, one advantage of using include is that include can dynamically reference templates:

{{ include $mytemplate }}

The above will dereference $mytemplate. The template function, in contrast, will only accept a string literal.

Pre-requisite

DOMAIN=kops.balmanrawat.com.np

aws route53 create-hosted-zone \
    --name ${DOMAIN} \
    --caller-reference kops-expriment

ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text)

aws s3api create-bucket \
    --bucket kops-${ACCOUNT_ID} \
    --region us-east-1

aws s3api put-bucket-versioning \
    --bucket kops-${ACCOUNT_ID} \
    --versioning-configuration Status=Enabled

OIDC

aws s3api create-bucket \
    --bucket kops-oidc-${ACCOUNT_ID} \
    --region us-east-1 \
    --object-ownership BucketOwnerPreferred

aws s3api put-public-access-block \
    --bucket kops-oidc-${ACCOUNT_ID} \
    --public-access-block-configuration BlockPublicAcls=false,IgnorePublicAcls=false,BlockPublicPolicy=false,RestrictPublicBuckets=false

aws s3api put-bucket-acl \
    --bucket kops-oidc-${ACCOUNT_ID} \
    --acl public-read

Creating Cluster

export NAME=${DOMAIN}
export KOPS_STATE_STORE=s3://kops-${ACCOUNT_ID}

# aws ec2 describe-availability-zones --region us-east-1

kops create cluster \
    --name=${NAME} \
    --cloud=aws \
    --zones=us-east-1a \
    --dns-zone=kubernetes.${DOMAIN} \
    --discovery-store=s3://kops-oidc-${ACCOUNT_ID}/${NAME}/discovery

kops update cluster --name ${NAME} --yes --admin

Validate

Suggestions:
 * validate cluster: kops validate cluster --wait 10m
 * list nodes: kubectl get nodes --show-labels
 * ssh to a control-plane node: ssh -i ~/.ssh/id_rsa ubuntu@
 * the ubuntu user is specific to Ubuntu. If not using Ubuntu please use the appropriate user based on your OS.
 * read about installing addons at: https://kops.sigs.k8s.io/addons.

Cleanup

kops delete cluster --name ${NAME} --yes

aws s3 rm s3://kops-${ACCOUNT_ID} --recursive
aws s3api delete-bucket --bucket kops-${ACCOUNT_ID} --region us-east-1

aws s3 rm s3://kops-oidc-${ACCOUNT_ID} --recursive
aws s3api delete-bucket --bucket kops-oidc-${ACCOUNT_ID} --region us-east-1

HOSTEDZONE_ID=$(aws route53 list-hosted-zones-by-name --dns-name ${DOMAIN} --query 'HostedZones[0].Id' --output text | sed 's/\/hostedzone\///')
aws route53 list-resource-record-sets --hosted-zone-id ${HOSTEDZONE_ID} > delete-record-set.json
#delete ns/soa record form the file and run the delete
aws route53 change-resource-record-sets --hosted-zone-id ${HOSTEDZONE_ID} --change-batch file://delete-record-sets.json
aws route53 delete-hosted-zone --id ${HOSTEDZONE_ID}

Chapter 1 ( Container Security Threats )

Risks, Threats, and Mitigations

A risk is a potential problem, and the effects of that problem if it were to occur.

A threat is a path to that risk occurring.

A mitigation is a countermeasure against a threat—something you can do to prevent the threat or at least reduce the likelihood of its success.

Security Principles

• Least Privilege

• Defense in Depth

  • have multiple layers of defense

• Reducing the Attack Surface

  • less code and less dependency

  • system with less components

  • Limiting the users and components who can access a service

• Limiting the Blast Radius

• Segregation of Duties

  • give only subset of privilege to different components or people

Chapter 2 ( Linux System Calls, Permissions, and Capabilities )

System Calls

The programmatic interface that the user space code uses to make these requests of the kernel is known as the system call or syscall interface.

There are some 300+ different system calls, with the number varying according to the version of Linux kernel.

File Permissions

• setuid

• setgid

Linux Capabilities: there is another way to give ping sufficient privileges to open the socket without the executable having all the privileges associated with root.

Because setuid provides a dangerous pathway to privilege escalation, some container image scanners (covered in Chapter 7) will report on the presence of files with the setuid bit set. You can also prevent it from being used with the --no-new-privileges flag on a docker run command.

Linux Capabilities

There are over 30 different capabilities in today’s Linux kernel. Capabilities can be assigned to a thread to determine whether that thread can perform certain actions. For example, a thread needs the CAP_NET_BIND_SERVICE capability in order to bind to a low-numbered (below 1024) port. CAP_SYS_BOOT exists so that arbitrary executables don’t have permission to reboot the system. CAP_SYS_MODULE is needed to load or unload kernel modules.

You can see the capabilities assigned to a process by using the getpcaps command. For example, a process run by a non-root user typically won’t have capabilities:

getpcaps <pid>
setcap 'cap_net_raw+p' <executable>

Chapter 3 ( Control Groups )

Cgroup Hierarchies

There is a hierarchy of control groups for each type of resource being managed, and each hierarchy is managed by a cgroup controller. Any Linux process is a member of one cgroup of each type, and when it is first created, a process inherits the cgroups of its parent.

The Linux kernel communicates information about cgroups through a set of pseudo‐ filesystems that typically reside at /sys/fs/cgroup.

Assigning a Process to a Cgroup

echo 100000 > memory.limit_in_bytes
echo 29903 > cgroup.procs

Chapter 4 (Container Isolation)

Linux Namespaces

If cgroups control the resources that a process can use, namespaces control what it can see. By putting a process in a namespace, you can restrict the resources that are visible to that process.

PID Namespace:

• Isolates the process ID number space, allowing processes inside the namespace to have the same PID as processes outside the namespace.

• Enables the creation of containers with their own init process, providing a private process tree.

NET Namespace:

• Provides isolation of network interfaces, IP addresses, routing tables, and port numbers.

• Allows containers to have their own network stack, separate from the host and other containers.

MNT Namespace:

• Isolates the filesystem mount points, so changes to mounts (e.g., adding or removing filesystems) in one namespace do not affect others.

• Ensures each container can have its own file system layout.

IPC Namespace:

• Isolates System V IPC objects and POSIX message queues, allowing processes to communicate within the same namespace without interfering with other namespaces.

• Useful for containers that need shared memory or semaphores.

UTS Namespace:

• Isolates the hostname and NIS domain name, allowing containers to have a different hostname and domain name from the host.

• Useful for setting custom hostnames within containers.

USER Namespace:

• Isolates user and group IDs, enabling processes to have different user IDs and privileges inside the namespace compared to outside.

• Allows running containers as non-root users while providing root privileges within the container.

CGROUP Namespace:

• Isolates the view of the cgroups, so each namespace sees its own cgroups.

• Helps in managing and limiting the resources (CPU, memory, etc.) available to the processes within the namespace.

TIME Namespace:

• Isolates the system and monotonic clocks, allowing processes within the namespace to see and manipulate different time values.

• Useful for testing software behavior at different times or handling time-sensitive applications in containers without affecting the host system.

Chapter 5 ( Virtual Machines )

....

Capter 6 ( Container Images )

Dockerfile Best Practices for Security

Base image

• Refer to an image from a trusted registry

• The smaller the base image, the less likely that it includes unnecessary code, and hence the smaller the attack surface.

• https://github.com/GoogleContainerTools/distroless

• Be thoughtful about using a tag or a digest to reference the base image.

Use multi-stage builds

Non-root USER

RUN commands

Volume mounts

• Be careful while running volume mounts of system directories like /etc

Don’t include sensitive data in the Dockerfile

Avoid setuid binaries

Avoid unnecessary code

Include everything that your container needs

Chapter 7 ( Software Vulnerabilities in Images )

https://github.com/aquasecurity/trivy

https://aws.amazon.com/blogs/containers/scanning-images-with-trivy-in-an-aws-codepipeline/

https://bitnami.com/stack/nginx

Chapter 8 ( Strengthening Container Isolation )

https://blog.jessfraz.com/post/how-to-use-new-docker-seccomp-profiles/

https://github.com/nevins-b/falco2seccomp

https://github.com/aquasecurity/tracee

In AppArmor, a profile can be associated with an executable file, determining what that file is allowed to do in terms of capabilities and file access permissions.

/sys/module/apparmor/parameters/enabled

AppArmor and other LSMs implement mandatory access controls. A mandatory access control is set by a central administrator, and once set, other users do not have any ability to modify the control or pass it on to another user.

https://github.com/genuinetools/bane (Custom & better AppArmor profile generator for Docker containers.)

gVisor

Google’s gVisor sandboxes a container by intercepting system calls in much the same way that a hypervisor intercepts the system calls of a guest virtual machine.

According to the gVisor documentation, gVisor is a “user-space kernel,” which strikes me as a contradiction in terms but is meant to describe how a number of Linux sys‐ tem calls are implemented in user space through paravirtualization. As you saw in Chapter 5, paravirtualization means reimplementing instructions that would other‐ wise be run by the host kernel.

Kata Containers

As you’ve seen in Chapter 4, when you run a regular container, the container runtime starts a new process within the host. The idea with Kata Containers is to run contain‐ ers within a separate virtual machine. This approach gives the ability to run applica‐ tions from regular OCI format container images, with all the isolation of a virtual machine.

Kata uses a proxy between the container runtime and a separate target host where the application code runs. The runtime proxy creates a separate virtual machine using QEMU to run the container on its behalf.

One criticism of Kata Containers is that you have to wait for a virtual machine to boot up. The folks at AWS have created a lightweight virtual machine that is specifically designed for running containers, with much faster startup times than a normal VM: Firecracker.

Firecracker

As you saw in “Disadvantages of Virtual Machines” on page 62, virtual machines are slow to start, making them unsuitable for the ephemeral workloads that typically run in containers. But what if you had a virtual machine that boots extremely quickly? Firecracker is a virtual machine offering the benefits of secure isolation through a hypervisor and no shared kernel, but with startup times around 100ms, it is much more suitable for containers. It has the benefit of becoming field-hardened due to its (as I understand it, gradual) adoption by AWS for its Lambda and Fargate services.

Unikernels

The idea of Unikernels is to create a dedicated machine image consisting of the appli‐ cation and the parts of the operating system that the app needs. This machine image can run directly on the hypervisor, giving the same levels of isolation as regular virtual machines, but with a lightweight startup time similar to what we see in Firecracker.

Every application has to be compiled into a Unikernel image complete with every‐ thing it needs to operate. The hypervisor can boot up this machine in just the same way that it would boot a standard Linux virtual machine image.

Chapter 9 ( Breaking Container Isolation )

Containers Run as Root by Default

Rootless Containers

The --privileged Flag and Capabilities

Mounting Sensitive Directories

Mounting the Docker Socket

Sharing Namespaces Between a Container and Its Host

Sidecar Containers

Chapter 10 ( Container Network Security )

https://bitnami.com/stack/nginx

https://rootlesscontaine.rs/

docker run -it --rm nginx
./tracee.py -c -e cap_capable
docker run --cap-drop=all --cap-add=<cap1> --cap-add=<cap2> <image>

Layer 3/4 Routing and Rules

As you already know, routing at Layer 3 is concerned about deciding the next hop for an IP packet. This decision is based on a set of rules about which addresses are reached over which interface. But this is just a subset of things you can do with Layer 3 rules: there are also some fun things that can go on at this level to drop packets or manipulate IP addresses, for example, to implement load balancing, NAT, firewalls, and network security policies. Rules can also act at layer 4 to take into account the port number. These rules rely on a kernel feature called netfilter.

netfilter is a packet-filtering framework that was first introduced into the Linux kernel in version 2.4. It uses a set of rules that define what to do with a packet based on its source and destination addresses.

There are a few different ways that netfilter rules can be configured in user space. Let’s look at the two most common options: iptables and IPVS.

iptables

The iptables tool is one way of configuring IP packet–handling rules that are dealt with in the kernel using netfilter. There are several different table types; the two most interesting types in the context of container networking are filter and nat:

• filter—for deciding whether to drop or forward packets

• nat—for translating addresses

  As a root user, you can see the current set of rules of a particular type by running iptables -t <table type> -L.

IPVS

IP Virtual Server (IPVS) is sometimes referred to as Layer 4 load balancing or Layer 4 LAN switching. It is another rules implementation similar to iptables, but it’s opti‐ mized for load balancing by storing the forwarding rules in hash tables.

This optimization makes it very performant for kube-proxy’s use case, but it doesn’t necessarily mean you can draw conclusions about the performance of network plug- ins implementing network policies.

Whether it’s IPVS or iptables that manages netfilter rules, they act within the ker‐ nel.

Chapter 11 ( Securely Connecting Components with TLS )

....

Chapter 12 ( Passing Secrets to Containers )

1. Storing the Secret in the Container Image

2. Passing the Secret Over the Network

3. Passing Secrets in Environment Variables

4. Passing Secrets Through Files

Chapter 13 ( Container Runtime Protection )

Container Image Profiles

Network Traffic Profiles

Executable Profiles

• Observing executables with eBPF

File Access Profiles

User ID Profiles

Other Runtime Profiles

Container Security Tools

• AppArmor, SELinux, or seccomp profile.

• Network traffic can be policed at runtime using network policy or a service mesh, as described in

• CNCF project Falco.

Prevention or alerting

Chapter 14 ( Containers and the OWASP Top 10 )

....

Tools to play with Container Internals

• lscgroup

• lib-cgroup

• setns

• nsenter

• unshare

• chroot

• lsns

Links

https://www.openwall.com/lists/oss-security/2019/02/11/2

https://github.com/0xAX/linux-insides

https://github.com/opencontainers/umoci

https://github.com/containers/skopeo

https://medium.com/microscaling-systems/spot-the-docker-difference-9f99adcc4aaf

https://github.com/GoogleContainerTools/distroless

https://github.com/aquasecurity/trivy

https://aws.amazon.com/blogs/containers/scanning-images-with-trivy-in-an-aws-codepipeline/

https://www.schneier.com/blog/archives/2016/08/the_nsa_is_hoar.html

https://blog.jessfraz.com/post/how-to-use-new-docker-seccomp-profiles/

https://bitnami.com/stack/nginx

Pre-requisite

• A compatible Linux host.

• Verify the MAC address and product_uuid are unique for every node. Kubernetes uses these values to uniquely identify the nodes in the cluster.

    # Verify the mac-address
    ip link
  
    # The product_uuid can verfied using following commands. 
    # If these values are not unique to each node, the installation process may fail.
    sudo cat /sys/class/dmi/id/product_uuid
  

• Check network adapters

• Check required ports
  These required ports need to be open in order for Kubernetes components to communicate with each other. The pod network plugin you use may also require certain ports to be open. Since this differs with each pod network plugin

nc 127.0.0.1 6443 -v

• 2 GB or more of RAM per machine (any less will leave little room for your apps).

• 2 CPUs or more.

• Swap configuration. The default behavior of a kubelet was to fail to start if swap memory was detected on a node. See Swap memory management for more details.

  • You MUST disable swap if the kubelet is not properly configured to use swap. For example, sudo swapoff -a will disable swapping temporarily. To make this change persistent across reboots, make sure swap is disabled in config files like /etc/fstab, systemd.swap, depending how it was configured on your system.

    sudo swapoff -a

Network Configuration

Enable IPv4 packet forwarding:

# sysctl params required by setup, params persist across reboots
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
EOF

# Apply sysctl params without reboot
sudo sysctl --system

#verify
sysctl net.ipv4.ip_forward

Configure cgroup drivers

NOTE: Starting with v1.22 and later, when creating a cluster with kubeadm, if the user does not set the cgroupDriver field under KubeletConfiguration, kubeadm defaults it to systemd.

Install Containerd

## Containerd
curl -LO https://github.com/containerd/containerd/releases/download/v1.7.18/containerd-1.7.18-linux-amd64.tar.gz
sudo tar Cxzvf /usr/local containerd-1.7.18-linux-amd64.tar.gz

## Systemd
curl -O https://raw.githubusercontent.com/containerd/containerd/main/containerd.service
sudo mv containerd.service /lib/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now containerd

Runc

curl -LO  https://github.com/opencontainers/runc/releases/download/v1.1.12/runc.amd64
sudo install -m 755 runc.amd64 /usr/local/sbin/runc

Install CNI plugins

~> curl -LO https://github.com/containernetworking/plugins/releases/download/v1.5.0/cni-plugins-linux-amd64-v1.5.0.tgz
~> mkdir -p /opt/cni/bin
~> sudo tar Cxzvf /opt/cni/bin cni-plugins-linux-amd64-v1.5.0.tgz

~> ls /opt/cni/bin/
LICENSE    bandwidth  dhcp   firewall     host-local  loopback  portmap  sbr     tap     vlan
README.md  bridge     dummy  host-device  ipvlan      macvlan   ptp      static  tuning  vrf

Configure containerd

sudo mkdir -p /etc/containerd/
containerd config default | sudo tee  /etc/containerd/config.toml
# Update the SystemdCgroup = false line to SystemdCgroup = true

# You need CRI support enabled to use containerd with Kubernetes. 
#Make sure that cri is not included in thedisabled_plugins list
# within /etc/containerd/config.toml; if you made changes to that 
# file, also restart containerd.
sudo vim /etc/containerd/config.toml
sudo systemctl restart containerd

• Installing kubeadm, kubelet and kubectl

kubeadm will not install or manage kubelet or kubectl for you, so you will need to ensure they match the version of the Kubernetes control plane you want kubeadm to install for you.

• Network setup

ip route show

Optional: Preparing the required container images
This step is optional and only applies in case you wish kubeadm init and kubeadm join to not download the default container images which are hosted at registry.k8s.io.

Initializing your control-plane node

• (Recommended) If you have plans to upgrade this single control-plane kubeadm cluster to high availability you should specify the --control-plane-endpoint to set the shared endpoint for all control-plane nodes. Such an endpoint can be either a DNS name or an IP address of a load-balancer.

• Choose a Pod network add-on, and verify whether it requires any arguments to be passed to kubeadm init. Depending on which third-party provider you choose, you might need to set the --pod-network-cidr to a provider-specific value. See Installing a Pod network add-on.

• (Optional) kubeadm tries to detect the container runtime by using a list of well known endpoints. To use different container runtime or if there are more than one installed on the provisioned node, specify the --cri-socket argument to kubeadm. See Installing a runtime.

Run init

~> sudo kubeadm init --pod-network-cidr 10.33.0.0/16
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.0.8.15:6443 --token vajf1q.ahw188f33xklo8um \
        --discovery-token-ca-cert-hash sha256:58e837184a19c648286f21ccac308b2894fc0592ceee7c24f1ce0e616885bc52 

### Sanity Check
kubect get nodes
kubect get pods -n kube-system

Deploy CNI Plugins

#for single node cluster
kubectl taint nodes vagrant node-role.kubernetes.io/control-plane-
curl https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml -O
kubectl apply -f calico.yaml

Untaint Nodes

kubectl taint nodes vagrant node.cilium.io/agent-not-ready-

• Link1

• Link2

Install

Session Manger Plugin

AWS SSM Proxy. Optional

Start basic ssh connection

    aws ssm start-session \
    --target i-askfsa8asld

Port Forwarding

aws ssm start-session \
    --target i-00e68c055d1c0b087 \
    --document-name AWS-StartPortForwardingSession \
    --parameters '{"portNumber":["remote-host-port"], "localPortNumber":["local-port"]}'

Port Forwarding to Remote host

aws ssm start-session \
    --target i-00e68c055d1c0b087 \
    --document-name AWS-StartPortForwardingSessionToRemoteHost \
    --parameters '{"host":["remote-host-ip"],"portNumber":["remote-host-port"], "localPortNumber":["local-port"]}'

kubectl get pods
kubectl get pods --show-labels
kubectl run nginx --image=nginx
k run nginx --image=nginx --labels app=nginx
kubectl describe pod nginx
kubectl delete pod nginx
kubectl delete pod -lapp=nginx
k delete pod nginx --force --grace-period=0
k delete pod nginx --now #similar to --grace-period=1
kubectl run nginx --image=nginx --dry-run=client -o yaml > nginx-pod.yaml
kubectl edit pod nginx
kubectl get pods -o json
k run static-busybox --image=busybox -- sh -c "sleep 1000"
k run busybox --image=busybox --restart=Never -- sh -c "echo sleeping.... && sleep 5"
kubectl run busybox-pod --image=busybox --restart=Never --command -- /bin/sh -c "echo 'sleeping...'; sleep 5"
kubectl replace nginx --force -f nginx.yaml

You cannot edit pod spec other than
- spec.containers[*].image
- spec.initContainers[*].image
- spec.activeDeadlineSeconds
- spec.tolerations

Replicaset

k get rs nginx
kubectl describe rs nginx
k explain rs nginx
k edit rs nginx
k scale rs nginx --replicas 2
k scale --replicas 2 -f manifest.yaml
k delete rs nginx
k delete rs nginx --force --grace-period=0

Deployments

k get deploy
k create deploy nginx --image=nginx --replicas=3
# With Deployments you can easily edit any field/property of the POD template. Since the pod template is a child of the deployment specification, since it deletes the pod and recreats it

Namespace

k get ns
k create ns myns
k get pods -n myns
k run nginx --image=nginx -n myns
k get pods -A
k get pods --all-namespaces

Domain Names

<service-name>.<namespace>.svc.<cluster-endpoint>

Service

k get svc
k get svc --show-labels
k describe svc
k get ep
k run nginx --image=nginx --expose --port 8080
k expose pod nginx-pod --name nginx-service --port 80
k expose deployment webapp --type NodePort --port 30082 --target-port 8080
#port range => (30000 - 32767)

kube-proxy --proxy-mode iptables/ipvs/userspace #default is iptables
kube-api-server --service-cluster-ip-range ## is the option to specify the ip range of the service
iptables -L -t nat | grep <service-name> to list the iptables rules
tail -f /var/log/kube-proxy.log

DNS

• before k8s 1.12 DNS was called kube-dns and now the recommended is core-dns

• configuration file for core-dns can be found in /etc/coredns/corefile

• Service FQDN: <service-name>.<namespace>.svc.<cluster-endpoint>

  • eg: web-service.default.svc.cluster.local

• By default pod DNS is disabled.

• FQDN: <pod-ip>.<namespace>.pod.<cluster-endpoint>

  • eg: 10-122-5-1.default.pod.cluster.local

• core-dns is accessible using the service with name kube-dns

• pod's DNS configurations are automatically set by the Kubernetes using kubelet process. kubelet gets the cluster domain and the DNS IP using its config.

Sample Core File

Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }

## core file
kubectl get cm coredns -n kube-system -o yaml | less

##

Nodes

k get nodes
k get nodes -o jsonpath='{.items[*].status.nodeInfo.osImage}'
k taint node node01 color=blue:NoExecute
k taint node node01 color=blue:NoSchedule
k taint node node01 color=blue:PreferNoSchedule
k taint node controlplane node-role.kubernetes.io/control-plane:NoSchedule-
k label node node01 color=blue
kubectl label node node01 color-

Ip Commands

#see the list of interfaces
ip link

#see ip address
ip addr
ip addr show

## add ip address
ip addr add 192.168.1.5/24 dev eth0

# list the routes
ip route
#ip route add <network-addr> via <gateway-ip>
ip route add 192.168.1.0/24 via 192.168.2.1
#if gateway is directly linked with interface then you can 
ip route add 192.168.1.0/24 dev eth0

#verify if the forward is enabled or not. if 0 disabled/if 1 enabled
cat /proc/sys/net/ipv4/ip_forward
#Modify /etc/systcl.conf to forward the request from one interface to others

## modify /etc/nsswitch.conf to update the priority of the dns server
cat /etc/nsswitch.conf

#you can add extra nameservers in resolv.conf by adding
nameserver 8.8.8.8
#you can append domain alias as well by using
search mycompany.com, prod.mycompany.com

This guide provides the information for deploying/managing highly available Elastic search setup in EKS using Elastic Cloud on Kubernetes Operator. In this, we will see how we can deploy the Elasticsearch version 8.8.1 using the ECK Operator.

The major source of information for this guide is the official ECK docs.

Why ECK Operator?

We will be deploying/managing Elasticsearch using the Elastic Cloud on Kubernetes Operator which is the latest recommended approach and managed by the Elastic team, the community itself. Please find more details here.

Architecture

The current setup will create an elastic search with 3 master nodes and 2 data nodes and make them accessible via LoadBalancer in AWS EKS. It tries to distribute the elastic search nodes evenly among the existing k8s worker nodes.

Pre-requisite

EKS Cluster ☸️

The setup assumes that EKS cluster is up and running and has enough resources to provision the ES Cluster. The current setup creates 3 master nodes and 2 data nodes which require approx 2GB memory and 1 vCPU per ElasticSearch node.

Install ECK CRDs ☸️

Install custom resources for provisioning the kind: ElasticSearch resource. It creates other Custom Resources supported by ECK as well but can be ignored. Command

kubectl create -f https://download.elastic.co/downloads/eck/2.8.0/crds.yaml

Output

customresourcedefinition.apiextensions.k8s.io/agents.agent.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/apmservers.apm.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/beats.beat.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/elasticmapsservers.maps.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/elasticsearchautoscalers.autoscaling.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/elasticsearches.elasticsearch.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/enterprisesearches.enterprisesearch.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/kibanas.kibana.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/logstashes.logstash.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/stackconfigpolicies.stackconfigpolicy.k8s.elastic.co created

Install Operator with its RBAC rules ☸️

The below command installs operator and required roles to manage various operations. By default, it creates an elastic-system namespace and deploys resources under it. Command

kubectl apply -f https://download.elastic.co/downloads/eck/2.8.0/operator.yaml

Output

namespace/elastic-system created
serviceaccount/elastic-operator created
secret/elastic-webhook-server-cert created
configmap/elastic-operator created
clusterrole.rbac.authorization.k8s.io/elastic-operator created
clusterrole.rbac.authorization.k8s.io/elastic-operator-view created
clusterrole.rbac.authorization.k8s.io/elastic-operator-edit created
clusterrolebinding.rbac.authorization.k8s.io/elastic-operator created
service/elastic-webhook-server created
statefulset.apps/elastic-operator created
validatingwebhookconfiguration.admissionregistration.k8s.io/elastic-webhook.k8s.elastic.co created

Verify ☑️

Verify if the operator is up and running by looking at the logs. Command

kubectl -n elastic-system logs -f statefulset.apps/elastic-operator

Output

{"log.level":"info","@timestamp":"2023-06-19T04:38:33.355Z","log.logger":"manager.eck-operator","message":"Starting EventSource","service.version":"2.8.0+3940cf4d","service.type":"eck","ecs.version":"1.4.0","controller":"beat-controller","source":"kind source: *v1.Secret"}
{"log.level":"info","@timestamp":"2023-06-19T04:38:33.355Z","log.logger":"manager.eck-operator","message":"Starting EventSource","service.version":"2.8.0+3940cf4d","service.type":"eck","ecs.version":"1.4.0","controller":"beat-controller","source":"kind source: *v1.Secret"}
{"log.level":"info","@timestamp":"2023-06-19T04:38:33.355Z","log.logger":"manager.eck-operator","message":"Starting Controller","service.version":"2.8.0+3940cf4d","service.type":"eck","ecs.version":"1.4.0","controller":"beat-controller"}
...

Storage Class (Optional) ☸️

This is an optional step as the elasticsearch.yaml manifest is configured to use the default gp2 storage class. If you want to override this behavior you can create your custom storage class and update its name in the elasticsearch.yaml file.

NOTE: Make sure the EBS CSI Plugin is active and the EKS Nodes have permission to manage volumes on behalf of the provisioner. Please include the following policy if missing.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:AttachVolume",
        "ec2:CreateSnapshot",
        "ec2:CreateTags",
        "ec2:CreateVolume",
        "ec2:DeleteSnapshot",
        "ec2:DeleteTags",
        "ec2:DeleteVolume",
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeInstances",
        "ec2:DescribeSnapshots",
        "ec2:DescribeTags",
        "ec2:DescribeVolumes",
        "ec2:DescribeVolumesModifications",
        "ec2:DetachVolume",
        "ec2:ModifyVolume"
      ],
      "Resource": "*"
    }
  ]
}

Verify ☑️

Verify if the EBS CSI controller is actively running. Command

kubectl get pods -n kube-system -lapp=ebs-csi-controller
``
Output
```shell
NAME                                 READY   STATUS    RESTARTS   AGE
ebs-csi-controller-6876d9b86-d88kq   6/6     Running   0          5m49s
ebs-csi-controller-6876d9b86-t47wn   6/6     Running   0          5m49s

Deploy Elasticsearch ☸️

Now as our pre-requisite is met we can deploy the elastic search in the cluster using elasticsearch.yaml template

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: esearch
spec:
  version: 8.8.1
  http:
    service:
      spec:
        type: LoadBalancer
    tls:
      selfSignedCertificate:
        subjectAltNames:
        - dns: localhost
        - dns: es.esearch.app
  #link: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-update-strategy.html
  updateStrategy:
    changeBudget:
      maxSurge: 1
      maxUnavailable: 1
  #link: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-pod-disruption-budget.html
  podDisruptionBudget:
    spec:
      minAvailable: 4
      selector:
        matchLabels:
          elasticsearch.k8s.elastic.co/cluster-name: esearch
  #Behind the scenes, ECK translates each NodeSet specified in the Elasticsearch resource into a StatefulSet in Kubernetes
  #Upgrade Patterns:  #Behind the scenes, ECK translates each NodeSet specified in the Elasticsearch resource into a StatefulSet in Kubernetes
  nodeSets:
  - name: masters
    count: 3
    config:
      node.roles: ["master"]
      # node.store.allow_mmap: false
      xpack.ml.enabled: true
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
        storageClassName: gp2
    podTemplate:
      metadata:
        labels:
          app: elasticsearch
      spec:
        terminationGracePeriodSeconds: 150
      #link: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-advanced-node-scheduling.html#k8s-affinity-options
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    elasticsearch.k8s.elastic.co/cluster-name: esearch
                topologyKey: kubernetes.io/hostname
        #https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-virtual-memory.html#k8s_using_an_init_container_to_set_virtual_memory
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
            runAsUser: 0
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        containers:
        - name: elasticsearch
          resources:
            limits:
              memory: 3Gi
            requests:
              memory: 2Gi
          env:
          - name: ES_JAVA_OPTS
            value: "-Xms1g -Xmx1g"
  - name: data
    count: 2
    config:
      node.roles: ["data"]
      # node.store.allow_mmap: false
      xpack.ml.enabled: true
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        storageClassName: gp2
    podTemplate:
      metadata:
        labels:
          app: elasticsearch
      spec:
        terminationGracePeriodSeconds: 150
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    elasticsearch.k8s.elastic.co/cluster-name: esearch
                topologyKey: kubernetes.io/hostname
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
            runAsUser: 0
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        containers:
        - name: elasticsearch
          resources:
            limits:
              memory: 3Gi
            requests:
              memory: 2Gi
          env:
          - name: ES_JAVA_OPTS
            value: "-Xms1g -Xmx1g"

Apply ⤵️

k apply -f elasticsearch.yaml

Verify ☑️

Verify by listing the Elasticsearch resources

Initial Status

#When you create the cluster, there is unknown HEALTH status and the PHASE is ApplyingChanges. After a while, the PHASE turns into Ready, and HEALTH becomes green.
➜ kubectl get es  
NAME     HEALTH    NODES   VERSION   PHASE             AGE
esearch   unknown           8.8.1     ApplyingChanges   22s

➜ kubectl get pods
NAME                  READY   STATUS     RESTARTS   AGE
esearch-es-data-0      0/1     Init:0/3   0          23s
esearch-es-data-1      0/1     Init:0/3   0          23s
esearch-es-masters-0   0/1     Init:0/3   0          23s
esearch-es-masters-1   0/1     Init:0/3   0          23s
esearch-es-masters-2   0/1     Init:0/3   0          23s

➜ kubectl get sts 
NAME                READY   AGE
esearch-es-data      0/2     28s
esearch-es-masters   0/3     29s
➜  elastic-search git:(main) ✗

Eventual Status

➜ kubectl get es
NAME     HEALTH   NODES   VERSION   PHASE   AGE
esearch   green    5       8.8.1     Ready   3m5s

➜ kubectl get pods
NAME                  READY   STATUS    RESTARTS   AGE
esearch-es-data-0      1/1     Running   0          3m10s
esearch-es-data-1      1/1     Running   0          3m10s
esearch-es-masters-0   1/1     Running   0          3m10s
esearch-es-masters-1   1/1     Running   0          3m10s
esearch-es-masters-2   1/1     Running   0          3m10s

➜ kubectl get sts
NAME                READY   AGE
esearch-es-data      2/2     3m16s
esearch-es-masters   3/3     3m17s

Accessing 🌍

To access the cluster properly you will need es endpoint, es password and the ca certificate. Please use the below commands to get the information:

#Set Elasticsearch name
export ES_NAME=esearch

#Get Service Public endpoint
ENDPOINT=$(kubectl get svc ${ES_NAME}-es-http -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

#Get ElasticSearch User Password
PASSWORD=$(kubectl get secret ${ES_NAME}-es-elastic-user -o go-template='{{.data.elastic | base64decode}}')

#Get ES CA Certificate
kubectl get secret ${ES_NAME}-es-http-certs-public -o go-template='{{index .data "tls.crt" | base64decode }}' > ca.cert

Verify ☑️

Once we have the endpoint, certificates and password we can not connect with Elasticsearch cluster using https. Please use the below verification command to see if everything is set properly. Command

curl -X GET --cacert ca.cert -u "elastic:${PASSWORD}" "https://${ENDPOINT}:9200"

Output

{
  "name" : "esearch-es-masters-2",
  "cluster_name" : "esearch",
  "cluster_uuid" : "jygZP_KOS_6ELM5WjSe9cQ",
  "version" : {
    "number" : "8.8.1",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "f8edfccba429b6477927a7c1ce1bc6729521305e",
    "build_date" : "2023-06-05T21:32:25.188464208Z",
    "build_snapshot" : false,
    "lucene_version" : "9.6.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

Cleanup ❌

If we need to revert all the changes we can do it in the following order.

kubectl delete -f elasticsearch.yaml
kubectl delete -f https://download.elastic.co/downloads/eck/2.8.0/operator.yaml
kubectl delete -f https://download.elastic.co/downloads/eck/2.8.0/crds.yaml

Hurray 🎉 This concludes the setup of the elastic search using the ECK operator. For any further details and customization please follow the details section below.

Details on elasticsearch manifest

There are multiple choices and configuration settings that we made when defining the elasticsearch.yaml manifest. Please find details about them in the section below:

1. Update/Upgrade Resiliency

While running any kind of updates/upgrades we want to make sure we are not under the capacity or overcapacity. maxsurge specifies a maximum addition number of nodes than our existing pool of es nodes, this will have us prevent excessive utilization of resources during the upgrades. maxUnavailable specifies a minimum number of nodes that can be available at a time.

...
updateStrategy:
    changeBudget:
      maxSurge: 1 #creates only one new node at a time.
      maxUnavailable: 1 #using 1 because we only have 2 data nodes. Can be increased if we have more data/master nodes. Low value increases stability but increases deployment time i.e we need to find the right balance.
...

2. Pod Disruption Budget

During the upgrades, maintenance and failure of any nodes we still want to maintain the pool of the nodes such that the es cluster continues to serve the traffic. We are using podDistributionBudget where minAvailable is set to 4 and the selector matches the es nodes.

...
podDisruptionBudget:
  spec:
    minAvailable: 4 #using 4 as we have 3 master nodes and 2 data nodes.
    selector:
      matchLabels:
        elasticsearch.k8s.elastic.co/cluster-name: "{{.metadata.name}}"
...

3. Uniform ES nodes distribution

The config tries to ensure the es nodes are distributed uniformly across the worker nodes for better availability and resiliency using topologyKey: kubernetes.io/hostname.

...
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution: #using preferred instead of required as there always won't be unique k8s node per es node and using required might make es nodes unschedulable
    - weight: 100 #give high priority for this rule
      podAffinityTerm:
        labelSelector:
          matchLabels:
            elasticsearch.k8s.elastic.co/cluster-name: "{{.metadata.name}}"
        topologyKey: kubernetes.io/hostname #this ensures distribution of es nodes based on the hostname
...

4. ES Node Roles/Counts

For this scenario the configuration specifies 3 master node(for minimal H/A) and 2 data nodes(for minimal H/A). For the sake of simplicity other roles are ignored.

5. Memory mapping configuration

Default values for virtual address space on Linux distributions are too low for Elasticsearch to work properly, which may result in out-of-memory exceptions. For production workloads, it is strongly recommended to increase the kernel setting vm.max_map_count to 262144.

6. Storage Class

The storage class used in the volumeClaimTemplates template is storageClassName: gp2 default storage class created when EKS cluster is created which can be changed if you have different requirements for different roles.

7. JVM flags

JVM flags can be overridden using the environment variables and this can largely impact the performance of the es cluster overall. It should be different for different roles for now the basic configuration is JVM start memory: 1GB and JVM max: 1GB.

...
env:
  - name: ES_JAVA_OPTS
    value: "-Xms1g -Xmx1g"
...

8. Termination Period

The termination grace period for pod is used to give enough time to gracefully exit the es node during any kinds of upgrades and maintenance tasks. Current scenario specific 150 which should be enough but can highly differ based on the node type and size of traffic/data.

...
spec:
  terminationGracePeriodSeconds: 300
...

Further configuration and details 📖

Many more settings can be configured to tune the cluster setup to our needs. Please find more details in this API reference.

https://docs.aws.amazon.com/autoscaling/ec2/userguide/what-is-amazon-ec2-auto-scaling.html

Autoscaling Beyond EC2 Autoscaling

https://docs.aws.amazon.com/autoscaling/application/userguide/what-is-application-auto-scaling.html

Components of Autoscaling

Scaling Options

Autoscaling Group

Configuration Options

Benefits

• Adjust the workload with the varied demand

• Distribute Instances across AZ for HA

  • instance distribution

  • AZ rebalance

  • Capacity rebalancing for Spot instance type

• Optimum utilization of the resources

Lifecycle

Scale Out

• ASG launches the required number of EC2 instances, using its assigned launch template. instances start in the Pending state.

• When each instance is fully configured and passes the Amazon EC2 health checks, it is attached to the Auto Scaling group and enters the InService state.

• If your ASG is configured with ELB, it will automatically register your instance with the load balancer before it marks the instance as InService.

• The instance is counted against the desired capacity after it gets to InService stage.

Scale In

• The ASG uses its termination policy to determine which instances to terminate.

• Instances that are in the process of terminating from the ASG and shutting down enter the Terminating state, and can't be put back into service.

• If you add a lifecycle hook to your Auto Scaling group, you can perform a custom action here.

• If your ASG is configured to receive traffic from an ELB, ASG automatically deregisters the terminating instance from the load balancer before running lifecycle hooks.

Attach Instance

You can attach existing EC2 Instance which meets certain criteria to be managed by the ASG.

For an instance to be attached, it must meet the following criteria:

• The instance is in the running state with Amazon EC2.

• The AMI used to launch the instance must still exist.

• The instance is not a member of another Auto Scaling group.

• The instance is launched into one of the Availability Zones defined in your Auto Scaling group.

• If the Auto Scaling group has an attached load balancer target group or Classic Load Balancer, the instance and the load balancer must both be in the same VPC.

When you attach instances, consider the following:

• The desired capacity of the group increases by the number of instances being attached. If the number of instances being attached plus the desired capacity exceeds the maximum size of the group, the request fails.

• If you attach an instance to an Auto Scaling group that has an attached load balancer target group or Classic Load Balancer, the instance is registered with the load balancer.

Detach Instance

You can detach EC2 Instance from ASG and the instance will no longer be managed by the ASG.

When you detach instances, consider the following:

• The desired capacity of the group decreases by the number of instances being detached. If the number of instances being detached makes the desired count lower than the minimum size of the group, the request fails. To skip this behavior you can select replace flag to maintain the same desired capacity where ASG adds another EC2 Instance in replacement.

• If detach multiple instances from the same Availability Zone, Amazon EC2 Auto Scaling can rebalance the Availability Zones unless you suspend the AZRebalance process. For more information, see Suspend and resume a process for an Auto Scaling group.

• If you detach an instance from an Auto Scaling group that has an attached load balancer target group or Classic Load Balancer, the instance is deregistered from the load balancer. If connection draining (deregistration delay) is enabled for your load balancer, Amazon EC2 Auto Scaling waits for in-flight requests to complete.

Enter and Exit Standby

You can change the state of the EC2 Instance from temporary to standby mode to debug and put it back to the Inservice stage. ASG still manages the Instance even in the StandBy stage except the fact is that it doesn't receive live traffic.

Things to know when dealing with Standby:

• You put the instance into the standby state. The instance remains in this state until you exit the standby state.

• By default, the value that you specified as your desired capacity is decremented when you put an instance on standby. This prevents the launch of an additional instance while you have this instance on standby. Alternatively, you can specify that your desired capacity is not decremented.

• After you put an instance back in service, the desired capacity is incremented. If you did not decrement the capacity when you put the instance on standby, the Auto Scaling group detects that you have more instances than you need. It applies the termination policy in effect to reduce the size of the group.

• Amazon EC2 Auto Scaling does not perform health checks on instances that are in a standby state. For example, if you put a healthy instance on standby and then terminate it, Amazon EC2 Auto Scaling continues to report the instance as healthy.

Lifecycle hooks

• If you added an autoscaling:EC2_INSTANCE_LAUNCHING lifecycle hook to your Auto Scaling group, the instances move from the Pending state to the Pending:Wait state.

• After completion, the instances enter the Pending:Proceed state.

• When the instances are fully configured, they are attached to the Auto Scaling group and they enter the InService state.

• If you added an autoscaling:EC2_INSTANCE_TERMINATING lifecycle hook to your Auto Scaling group, the instances move from the Terminating state to the Terminating:Wait state.

• After completion, the instances enter the Terminating:Proceed state.

Launch Templates

Not all Amazon EC2 Auto Scaling features are available when you use launch configurations. For example, you cannot create an Auto Scaling group that launches both Spot and On-Demand Instances or that specifies multiple instance types. You must use a launch template to configure these features.

With launch templates, you can also use newer features of Amazon EC2. This includes

• Systems Manager parameters (AMI ID),

• the current generation of EBS Provisioned IOPS volumes (io2),

• EBS volume tagging,

• T2 Unlimited instances,

• Elastic Inference, and

• Dedicated Hosts

When you create a launch template, all parameters are optional. However, if a launch template does not specify an AMI, you cannot add the AMI when you create your Auto Scaling group. If you specify an AMI but no instance type, you can add one or more instance types when you create your Auto Scaling group.

Replacing Instances

Instance refresh

An instance refresh can be helpful when you have a new Amazon Machine Image (AMI) or a new user data script. To use an instance refresh, first, create a new launch template that specifies the new AMI or user data script. Then, start an instance refresh to begin updating the instances in the group immediately.

NOTE: Can cause downtime if there is only 1 EC2 Instance in the ASG, because it terminates the EC2 Instance first and then only adds after that

Replace based on maximum instance lifetime

The maximum instance lifetime specifies the maximum amount of time (in seconds) that an instance can be in service before it is terminated and replaced. A common use case might be a requirement to replace your instances on a schedule because of internal security policies or external compliance controls.

NOTE: Can cause downtime if there is only 1 EC2 Instance in the ASG, because it terminates the EC2 Instance first and then only adds after that

Scaling

Manual Scaling

You can scale in/out by using the following methods

• Update min/desired/max capacity as per the need and scale your EC2 Instances.

• Attach/Detach EC2 Instance

Dynamic Scaling

Target Tracking Autoscaling

Increase and decrease the current capacity of the group based on a Amazon CloudWatch metric and a target value.

• It will scale out the Auto Scaling group if any of the target tracking policies are ready for scale out, but will scale in only if all of the target tracking policies (with the scale-in portion enabled) are ready to scale in.

• Scaling might determine that adding 1.5 instances will decrease the CPU utilization to close to 50 percent. Because it is not possible to add 1.5 instances, AWS rounds up and adds two instances. This might decrease the CPU utilization to a value below 50 percent, but it ensures that your application has enough resources to support it. Similarly, if Scaling determines that removing 1.5 instances increases your CPU utilization to above 50 percent, it removes just one instance.

• Per-defined metrics to choose from: ALBRequestCountPerTarget, ASGAverageNetworkOut, ASGAverageNetworkIn, ASGAverageCPUUtilization.

• AWS recommends that you only use metrics that are available at one-minute intervals to help you scale faster in response to utilization changes.

Simple Autoscaling

Increase and decrease the current capacity of the group based on a single scaling adjustment, with a cooldown period between each scaling activity.

Step Autoscaling

It is similar to simple autoscaling but we can work with multiple slices of the alarm values. Increase and decrease the current capacity of the group based on a set of scaling adjustments, known as step adjustments, that vary based on the size of the alarm breach.

Scheduled Based Autoscaling

Predictive Autoscaling

Termination Policies:

Default Termination Policy

Determines which instance to terminate on the scale-in events, instance refresh and AZ balance.

• The instance in the AZ with most instances

• The instance with the older version of the launch template

• If instances have the same launch template, terminate the instance with the closed billing hour

Allocation Strategy

Terminate instance based on the lowest price, lowest priority for OnDemand Instance

Oldest Launch Template

Terminate instances that have the oldest launch template.

Oldest Launch Configurations

Terminate instances that have the oldest launch configuration.

Closest to Next Launch Hour

Terminate instances that are closest to the next billing hour.

Newest Instance

Terminate the newest instance in the group. This policy is useful when you're testing a new launch configuration but don't want to keep it in production.

Oldest Instances

Terminate the oldest instance in the group. This option is useful when you're upgrading the instances in the Auto Scaling group to a new EC2 instance type.

Custom Policy

This can be defined by using the lambda backed custom policy.

NOTE: Amazon EC2 Auto Scaling always balances instances across Availability Zones first, regardless of which termination policy is used. As a result, you might encounter situations in which some newer instances are terminated before older instances.

Suspending the ASG Process

There are certain times when we want to suspend various automatic processes that ASG conducts. For example: when we are deploying the changes to EC2 Instance we don't want to Launch a new instance and Terminate existing ones. We can achieve this by suspending the Launch and Terminate Process.

• Launch—Adds instances to the Auto Scaling group when the group scales out, or when Amazon EC2 Auto Scaling chooses to launch instances for other reasons, such as when it adds instances to a warm pool.

• Terminate—Removes instances from the Auto Scaling group when the group scales in, or when Amazon EC2 Auto Scaling chooses to terminate instances for other reasons, such as when an instance is terminated for exceeding its maximum lifetime duration or failing a health check.

• AddToLoadBalancer—Adds instances to the attached load balancer target group or Classic Load Balancer when they are launched.

• AlarmNotification—Accepts notifications from CloudWatch alarms that are associated with dynamic scaling policies.

• AZRebalance—Balances the number of EC2 instances in the group evenly across all of the specified Availability Zones when the group becomes unbalanced.

• HealthCheck—Checks the health of the instances and marks an instance as unhealthy if Amazon EC2 or Elastic Load Balancing tells Amazon EC2 Auto Scaling that the instance is unhealthy.

• InstanceRefresh—Terminates and replaces instances using the instance refresh feature.

• ReplaceUnhealthy—Terminates instances that are marked as unhealthy and then create new instances to replace them.

• ScheduledActions—Performs the scheduled scaling actions that you create or that are created for you when you create an AWS Auto Scaling scaling plan and turn on predictive scaling.

• Create ASG, SQS Queue, Target Tracking Policy

• Use bash scripts to publish/consume messages to/from the SQS Queue

• Use bash scripts to publish the custom metrics

• Finally, autoscale based on the message count on the queue

Scenario

Let's suppose a scenario where users publish a data processing request in an AWS SQS. The SLA for the end user is that they should get the results within 2 seconds of the request being sent. Let's suppose each request takes 0.1 seconds to be processed by an EC2 Instance.

Calculation backlog per Instance:

Now let's calculate the message backlog count per instance to determine when to run the autoscaling.

Backlog Per Instance = Time User Can Wait/Time it takes to serve a Request
Backlog Per Instance = 2/0.1 = 20

This means the number of messages in the SQS queue/Instance count shouldn't be greater than 20.

Now we have found Backlog Per Instance=20 as our target for autoscaling. Enough theory now let's get started with creating the necessary setup in the AWS Account.

NOTE: Please be careful with this tutorial as it can add extra costs to your AWS bills

Pre-requisite

• EC2 Autoscaling Group

• Amazon SQS

• Basic Knowledge of AWS CLI and CloudWatch Metrics

Create Autoscaling Group

I already have an Autoscaling group with the name CustomMetricsASG and please verify your min/max/desired capacity is properly set to allow the autoscaling to happen.

Create Queue

Create a queue using the below command and verify

➜ aws sqs create-queue --queue-name myasgqueue
{
    "QueueUrl": "https://sqs.us-east-1.amazonaws.com/922726392568/myasgqueue"
}

Publish Messages to SQS

Now you can use the below commands to publish the messages in the SQS queue which acts like the requests sent to our system in real-world scenario.

##Producer

##Replace with your queue endpoint
QUEUE_URL=https://sqs.us-east-1.amazonaws.com/922726392568/myasgqueue
while true
do
  echo "Publishing messages..."
  sleep 3
  aws sqs send-message --queue-url ${QUEUE_URL} \
   --message-body "aws asg mock messages to increase the load..." \
   --no-cli-pager
done

If the above script ran successfully you will see outputs like below:

Consume Messages from SQS(Optional...)

This is optional but if you want to make it more realistic what you can do is increase the sleep time below such that publishing is done fast and the consuming is slower which increases the backlog in the Queue.

## Conumer
##Replace with your queue endpoint
QUEUE_URL=https://sqs.us-east-1.amazonaws.com/922726392568/myasgqueue
while true
do
  echo "consuming messages..."
  sleep 3
  aws sqs receive-message --queue-url ${${QUEUE_URL}} \
  --no-cli-pager
done

Publishing Custom Metrics

The below script does the following things:

• Fetches the number of available messages in the SQS Queue

• Fetches the number of the EC2 Instances in InService state

• Divides Message Count/InstanceCount to get the message BacklogPerInstance

• Then publish the custom metrics every 60 seconds setting:

  • MetricName: MyBacklogPerInstance

  • Namespace: MyCustomASGMetrics

  • Dimension as: QueueName=${QUEUE_NAME}

NOTE: Before running the script, please replace the values of the below variables.

#!/bin/bash
##Replace these values, before running the script
QUEUE_URL=https://sqs.us-east-1.amazonaws.com/922726392568/myasgqueue
ASG_NAME=CustomMetricsASG
QUEUE_NAME=myasgqueue

while true
do
  echo "....starting...."
  sleep 60
  printf "[INFO] Querying.. Available Queue Message\n"
  APPROX_AVAILABLE_MESSAGES=$(aws sqs get-queue-attributes --queue-url ${QUEUE_URL} --attribute-names ApproximateNumberOfMessages --query Attributes.ApproximateNumberOfMessages --output text | tr -d '[:space:]')
  STATUS_CODE=$?
  if [[ ${STATUS_CODE} -ne 0 ]]
  then
    printf "[WARN] APPROX_AVAILABLE_MESSAGES retrival failed with status code: %s ...\n" ${STATUS_CODE}
    continue
  fi
  printf "[INFO] The Number of Available Message: %s\n" "${APPROX_AVAILABLE_MESSAGES}"

  printf "[INFO] Querying.. Number of Instance in Service\n"
  IN_SERVICE_COUNT=$(aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names ${ASG_NAME} --query "AutoScalingGroups[].Instances[?LifecycleState=='InService'].[InstanceId]"\
    --output text | wc -l | tr -d '[:space:]')
  STATUS_CODE=$?
  if [[ ${STATUS_CODE} -ne 0 ]]
  then
    printf "[WARN] IN_SERVICE_COUNT retrival failed with status code: %s ...\n" ${STATUS_CODE}
    continue
  fi
  printf "[INFO] The Number of Instance in Service: %s\n" "${IN_SERVICE_COUNT}"

  BACKLOG_PER_INSTANCE=$((APPROX_AVAILABLE_MESSAGES / IN_SERVICE_COUNT))
  STATUS_CODE=$?
  if [[ ${STATUS_CODE} -ne 0 ]]
  then
    printf "[WARN] BACKLOG_PER_INSTANCE calculation failed with status code: %s ...\n" ${STATUS_CODE}
    continue
  fi
  printf "[INFO] Calculated Backlog per instance %s/%s: %s\n" "${APPROX_AVAILABLE_MESSAGES}" "${IN_SERVICE_COUNT}" "${BACKLOG_PER_INSTANCE}"

  aws cloudwatch put-metric-data --metric-name MyBacklogPerInstance --namespace MyCustomASGMetrics \
  --unit None --value ${BACKLOG_PER_INSTANCE} --dimensions QueueName=${QUEUE_NAME}
  STATUS_CODE=$?
  if [[ ${STATUS_CODE} -ne 0 ]]
  then
    printf "[WARN] put-metrics-data failed with status code: %s ...\n" ${STATUS_CODE}
    continue
  fi
  printf "[INFO] Successfully published custom metrics with value: %s ...\n" ${BACKLOG_PER_INSTANCE}
done

Once you run the above script, if everything goes fine you should be able to see the custom metrics in the AWS CloudWatch Metrics

Creating TargetTracking Autoscaling

Until now you can't create target tracking autoscaling Policy with custom metrics from AWS Console so, you need to create it from AWS CLI.

Policy Config

Please copy the below config and update with your relevant values:

{
   "TargetValue":20,
   "CustomizedMetricSpecification":{
      "MetricName":"MyBacklogPerInstance",
      "Namespace":"MyCustomASGMetrics",
      "Dimensions":[
         {
            "Name":"QueueName",
            "Value":"myasgqueue"
         }
      ],
      "Statistic":"Average",
      "Unit":"None"
   }
}

Create the Policy

Replace the values in the below variables and run the command

##Replace the below ASG_NAME with your values
ASG_NAME=CustomMetricsASG
aws autoscaling put-scaling-policy --policy-name sqs20-target-tracking-scaling-policy \
  --auto-scaling-group-name ${ASG_NAME} --policy-type TargetTrackingScaling \
  --target-tracking-configuration file://config.json

Once created successfully you will be able to see it in the Autoscaling section of the ASG

Scaling in Action

If there are enough points in the custom metrics and the value is greater than the defined threshold you will quickly see the scaling happening in the action.

Checking Activity in ASG

Conclusion

This is how you can setup autoscaling for any custom metrics and with any other autoscaling policies. You can even mix it with other autoscaling policies like: simple, step autoscaling.

Thanks for reading! Catch you next time!

• eksctl. details

• AWS CLI. details

• Helm. details

Create EKS Cluster

This will create an EKS cluster in us-east-1 region with 2 nodes in the node-group.

export CLUSTER_NAME=demo-cluster
export AWS_REGION=us-east-1
eksctl create cluster --name ${CLUSTER_NAME} --version 1.26 --node-type t3.medium --nodes 2 --managed --region ${AWS_REGION}
.....
.....
2023-05-28 18:47:47 [✔]  EKS cluster "demo-cluster" in "us-east-1" region is ready.

verify

When running the following command kubectl get nodes you should see the nodes in a ready state.

➜ kubectl get nodes   
NAME                             STATUS   ROLES    AGE    VERSION
ip-192-168-17-231.ec2.internal   Ready    <none>   114m   v1.26.4-eks-0a21954
ip-192-168-61-195.ec2.internal   Ready    <none>   114m   v1.26.4-eks-0a21954

Deploy External Secrets Operator

Use below helm commands to:

• add/update the repo

• install the operator

helm repo add external-secrets https://charts.external-secrets.io
helm repo update

helm install external-secrets \
  external-secrets/external-secrets \
    --namespace external-secrets \
    --create-namespace \
    --set installCRDs=true \
    --wait
.....
external-secrets has been deployed successfully!
....

verify

When you run k get all -n external-secrets you should be able to see all the pods pod/external-secrets-* , pod/external-secrets-cert-controller-*, pod/external-secrets-webhook-* running successfully.

➜ k get all -n external-secrets
NAME                                                    READY   STATUS    RESTARTS   AGE
pod/external-secrets-5997dc48b4-drncs                   1/1     Running   0          2m33s
pod/external-secrets-cert-controller-587977f796-mdk9p   1/1     Running   0          2m33s
pod/external-secrets-webhook-674f79cb47-jz7ds           1/1     Running   0          2m33s

NAME                               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/external-secrets-webhook   ClusterIP   10.100.84.243   <none>        443/TCP   2m33s

NAME                                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/external-secrets                   1/1     1            1           2m34s
deployment.apps/external-secrets-cert-controller   1/1     1            1           2m34s
deployment.apps/external-secrets-webhook           1/1     1            1           2m34s

NAME                                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/external-secrets-5997dc48b4                   1         1         1       2m34s
replicaset.apps/external-secrets-cert-controller-587977f796   1         1         1       2m34s
replicaset.apps/external-secrets-webhook-674f79cb47

Configuring IAM roles for service accounts (IRSA) and Secrets Manager

Add OIDC Provider

➜ eksctl utils associate-iam-oidc-provider --cluster=${CLUSTER_NAME} --approve

2023-05-28 13:25:07 [✔]  created IAM Open ID Connect provider for cluster "demo-cluster" in "us-east-1"

Create Secret in Secrets Manager

SECRET_ARN=$(aws secretsmanager create-secret --name user-creds \
    --secret-string "{\"user\":\"admin\",\"password\":\"topsecret\"}" \
    --region ${AWS_REGION} | jq -r .ARN)

Create IAM Policy

IAM Policy to Read/Describe the user-creds secret in the secrets manager

➜ IAM_POLICY_ARN=$(aws iam create-policy --policy-name eso-sm-reader-policy --policy-document '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:DescribeSecret",
        "secretsmanager:GetSecretValue"
      ],
      "Resource": ["'${SECRET_ARN}'"]
    }
  ]
}' | jq -r .Policy.Arn)

Create a Service Account and Association

➜ eksctl create iamserviceaccount \
    --name eso-secret-irsa \
    --namespace default \
    --cluster ${CLUSTER_NAME} \
    --role-name "eso-sm-reader-role" \
    --attach-policy-arn $IAM_POLICY_ARN \
    --approve

verify

k describe sa eso-secret-irsa                      
Name:                eso-secret-irsa
Namespace:           default
Labels:              app.kubernetes.io/managed-by=eksctl
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::808053714438:role/eso-sm-reader-role
Image pull secrets:  <none>
Mountable secrets:   <none>
Tokens:              <none>
Events:              <none>

Deploy External Secret Store/Secret

Create SecretStore

cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: secret-store
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: eso-secret-irsa
EOF

Verify

kubectl get SecretStore                
NAME           AGE   STATUS   CAPABILITIES   READY
secret-store   58s   Valid    ReadWrite      True

Create ExternalSecret

cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: user-creds
spec:
  refreshInterval: 5m
  secretStoreRef:
    name: secret-store
    kind: SecretStore
  target:
    name: user-creds
  data:
  - secretKey: user
    remoteRef:
      key: user-creds
      property: user
  - secretKey: password
    remoteRef:
      key: user-creds
      property: password
EOF

Verify

➜ kubectl get ExternalSecret           
NAME         STORE          REFRESH INTERVAL   STATUS         READY
user-creds   secret-store   5m                 SecretSynced   True
➜ kubectl get secret
NAME         TYPE     DATA   AGE
user-creds   Opaque   2      30s

Deploy Application

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: creds-app
spec:
  containers:
    - name: app
      image: k8s.gcr.io/busybox
      command: [ "/bin/sh", "-c", "env" ]
      envFrom:
      - secretRef:
          name: user-creds
EOF

Verify

➜ kubectl logs pod/creds-app     
....   
user=admin
password=topsecret
....

A Chart is a helm package.

A Repo is a place where these charts get collected and shared.

A Release is an instance of a chart running in a Kubernetes cluster.

Directory Structure

wordpress/
  Chart.yaml      # A YAML file containing information about the chart
  LICENSE         # OPTIONAL: A plain text file containing the license for the chart
  README.md       # OPTIONAL: A human-readable README file
  values.yaml     # The default configuration values for this chart
  values.schema.json  # OPTIONAL: A JSON Schema for imposing a structure on the values.yaml file
  charts/         # A directory containing any charts upon which this chart depends.
  crds/           # Custom Resource Definitions
  templates/      # A directory of templates that, when combined with values,
                  # will generate valid Kubernetes manifest files.
  templates/NOTES.txt # OPTIONAL: A plain text file containing short usage notes

Manage Repo

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo list
helm repo remove <repo-name>
helm repo update
helm repo remove
helm repo index

Manage Chart

#create helm chart
helm create <chart-name>

#Searching
helm search repo <chart-name>
helm search repo <repo-name> #to see all the charts in a repo
helm search hub <chart-name> #to search on artifcathub

#See information on charts
helm show/inspect all/chart/crds/readme/values <chart-name>
helm show all bitnami/wordpress
helm show values bitnami/wordpress
helm show chart bitnami/wordpress
helm show readme bitnami/wordpress

#generate the manifest from chart
helm template <chart-name>

Managing Releases

#install
helm install <release-name> <chart-name>/<local-chart-path>
#list installed release
helm list
#uninstall
helm uinstall <release-name>
#history
helm history <release-name>
#rollback
helm rollback <release-name> <revision> 

#See information of the release
helm get all/hooks/manifest/notes/values <release-name>
helm get manifest happy-panda

#Install dry run
helm install --dry-run --disable-openapi-validation <release-name> <chart-name>

Helm Built-in Objects

Objects are passed into a template from the template engine. The built-in values always begin with a capital letter. This is in keeping with Go's naming convention.

Release: This object describes the release itself. It has several objects inside of it:
    Release.Name
    Release.Namespace
    Release.IsUpgrade
    Release.IsInstall
    Release.Revision
    Release.Service

Values: Values passed into the template from the values.yaml file and from user-supplied files. By default, Values is empty.

Chart: The contents of the Chart.yaml file. Any data in Chart.yaml  will be accessible here.

Files:
    Files.Get is a function for getting a file by name (.Files.Get config.ini)
    Files.GetBytes is a function for getting the contents of a file as an array of bytes instead of as a string. This is useful for things like images.
    Files.Glob is a function that returns a list of files whose names match the given shell glob pattern.
    Files.Lines is a function that reads a file line-by-line. This is useful for iterating over each line in a file.
    Files.AsSecrets is a function that returns the file bodies as Base 64 encoded strings.
    Files.AsConfig is a function that returns file bodies as a YAML map.

Capabilities: This provides information about what capabilities the Kubernetes cluster supports.
    Capabilities.APIVersions is a set of versions.
    Capabilities.APIVersions.Has $version indicates whether a version (e.g., batch/v1) or resource (e.g., apps/v1/Deployment) is available on the cluster.
    Capabilities.KubeVersion and Capabilities.KubeVersion.Version is the Kubernetes version.
    Capabilities.KubeVersion.Major is the Kubernetes major version.
    Capabilities.KubeVersion.Minor is the Kubernetes minor version.
    Capabilities.HelmVersion is the object containing the Helm Version details, it is the same output of helm version
    Capabilities.HelmVersion.Version is the current Helm version in semver format.
    Capabilities.HelmVersion.GitCommit is the Helm git sha1.
    Capabilities.HelmVersion.GitTreeState is the state of the Helm git tree.
    Capabilities.HelmVersion.GoVersion is the version of the Go compiler used.

Template: Contains information about the current template that is being executed
    Template.Name: A namespaced file path to the current template (e.g. mychart/templates/mytemplate.yaml)
    Template.BasePath: The namespaced path to the templates directory of the current chart (e.g. mychart/templates).

Helm Lookup functions

template functions list: https://helm.sh/docs/chart_template_guide/function_list/

<function-name> arg1 arg2...
default "default value" .Values.<value>
lookup "apiVersion", "kind", "namespace", "name" -> resource or resource list
eg: lookup "v1" "Pod" "mynamespace" "mypod" => equivalent to: kubectl get pod mypod -n mynamespace

Operators Functions
eq, ne, lt, gt, and, or

Named Templates
{{- define "MY.NAME" }}
  # body of template here
{{- end }}

{{- template "MY.NAME" }}

{{- template "MY.NAME" . }} ##using the context

{{ include "MY.NAME" . | indent 4 }} ##include function can be used to pass values so, it is recommend to use include. Because template is an action, and not a function, there is no way to pass the output of a template call to other functions; the data is simply inserted inline.

Helm Passing the values

There are two ways to pass values one is using values.yml file and the other is –set flag. If both are used, --set values are merged into --values with higher precedence. Overrides specified with --set are persisted in a ConfigMap. Values that have been --set can be viewed for a given release with helm get values <release-name>. Values that have been --set can be cleared by running helm upgrade with --reset-values specified.

helm install -f values.yaml local-mysql bitnami/mysql -set db.auth.username="sth", db.auth.password="supersecret"

As infrastructure grows more complex, managing multiple CloudFormation Stacks becomes a challenge. Typically, actions such as creating, updating, or deleting stacks are performed on a single stack at a time. Inaddition deleting stacks in a development or testing environment can be cumbersome because we usually want to destroy the whole environment and to do that stacks must be deleted in the reverse order of creation.

cfn-compose offers a solution to this problem by providing a way to manage multiple, related stacks using a declarative yaml language, making the process easier and more streamlined. For more details please go through the rest of the Readme.

cfn-compose(cfnc in short)

A command-line tool for managing CloudFormation Stacks at scale.
Github Repo: https://github.com/rbalman/cfn-compose

Features

• Create/Update/Delete multiple CloudFormation stacks in parallel or sequentially

• Customize the CloudFormation stacks dependency using yaml config

• Delete multiple CloudFormation stacks respecting the creation sequence

• DryRun mode to plan the change

• Generate/Validate/visualize configuration with ease

• Supports Go Template for dynamic value substitution

Limitations

• Supports limited CFN attributes

• No Retry Mechanism

• No Configurable concurrency. One Go routine is spun for every flow.

• One compose file can have maximum 50 flows and each flow can have up to 50 stacks. This is by design, to limit stacks in a compose file.

Installation

Binary is available for Linux, Windows and Mac OS (amd64 and arm64). Download the binary for your respective platform from the releases page.

Using go cli

go install github.com/rbalman/cfn-compose@latest

Usage

➜ cfnc --help
Manage cloudformation stacks at scale. Design and deploy multiple cloudformation stacks either in sequence or in parallel using declarative configuration

Usage:
  cfnc [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  config      Generate, validate and visualize the compose configuration
  deploy      Deploys the stacks based on the sequence specified in the compose configuration
  destroy     Destroys all the stacks in the reverse order of creation
  help        Help about any command

Flags:
  -c, --config string     File path to compose file (default "cfn-compose.yml")
  -d, --dry-run           Run commands in dry run mode
  -h, --help              help for cfnc
  -l, --loglevel string   Specify Log Levels. Valid Levels are: DEBUG, INFO, WARN, ERROR (default "INFO")
  -v, --version           version for cfnc

Use "cfnc [command] --help" for more information about a command.

Examples

## Deploy
cfnc deploy
## Deploy in dry run mode
cfnc deploy -d

## Destroy
cfnc destroy
## Destroy in dry run mode
cfnc destroy -d

## Generate Validate and Visualize compose configuration
cfnc config generate
cfnc config validate
cfnc config visualize

Man

Documentation

Sample Config File:

Description: Sample CloudFormation Compose file
Vars:
  Key1: Value1
  Key2: Value2
Flows:
  Flow1:
    Order: 0
    Description: Flow1 Description
    Stacks:
      - Stack1
      - Stack2
  Flow2:
    Order: 1
    Description: Flow2 description
    Stacks:
      - Stack1
      - Stack2

A typical compose configuration contains:

• Optional Description

• Optional Vars section to define variables in Key: Value mapping. Only static variables are supported at the moment. eg:

Vars:
  ENV_TYPE: 'nonproduction'
  ENV_NAME: 'demo'
  AWS_PROFILE: 'demo'

• Mandatory Flows: section Flow is a collection of CloudFormation stacks that are deployed sequentially. Flows is the collection of flow which can be ordered using Order property. Flows can run in parallel or sequentially based on the Order property.

  • Optional Order can be any unsigned integer. Default Order is set to 0. Flow with the lowest orders are deployed first.

  • Optional Description

  • Mandatory Stacks which is the collection of CFN stack. Below are the supported attributes of the stack object

    • mandatory template_file or template_url (only s3 url)

    • mandatory stack_name

    • optional capabilities

    • optional parameters

    • optional tags

    • optional tags

Sample:

Description: Sample CloudFormation Compose file
Vars:
  ENV_NAME: cfnc
  ENV_TYPE: nonproduction
Flows:
  SecurityGroup:
    Order: 0
    Description: Creates SecurityGroup
    Stacks:
      - template_file: <cfn-template-path>
        stack_name: stack-name1
        parameters:
          EnvironmentName: '{{ .ENV_NAME }}'
          EnvironmentType: '{{ .ENV_TYPE }}'
        tags:
          EnvironmentName: '{{ .ENV_NAME }}'
          EnvironmentType: '{{ .ENV_TYPE }}'

  EC2Instance:
    Order: 1
    Description: Deploying EC2 Instance
    Stacks:
      - template_file: <cfn-template-path>
        stack_name: stack-name2
        parameters:
          EnvironmentName: '{{ .ENV_NAME }}'
          EnvironmentType: '{{ .ENV_TYPE }}'
        tags:
          EnvironmentName: '{{ .ENV_NAME }}'
          EnvironmentType: '{{ .ENV_TYPE }}'

Please consult examples for quick start ec2-sg example and demo ec2-sqs-rds example

Contributors

There exists ample opportunity for enhancement and you are welcome to make a valuable contribution. If you have any concerns, recommendations, ideas feel free to create issues or create PR. Details Example

During the early stage of computing IT Infrastructures were completely physical and they were managed completely physically by the experts. Then slowly with the rapid growth of high performance computing hardwares, virtualization and Internet raised the adoption of virtualization. These virtulized Infra were managed with various scripting languages.

Then came the era of the Cloud, driving the innovation entirely to the next level where everything possible was virtualized be it Servers, Queues, Storage, Databases, Networks, Caches, CDNs, Firewalls and more.

All these infrastructure management were abstracted with a simple API call, for eg. We can create a bucket using this create bucket API. The challenges with the APIs are that we need to deal with low level details like headers, authentication information, retries, dependencies, payloads, failure handling, logging, state management and many more.

Then comes the IAC where all these low level complexities are orchestrated through the human friendly abstractions.

Infrastructure as a Code

Infrastructure as a code abbreviated as IAC is the management of IT Infrastructures like Servers, Networks, Storage, Databases, Queues either in the Cloud or the on-prem data centers through the use of some coding pattern or some form of templating language. IAC is basically a representation of Infrastructure in the form of a Code/template such that it can completely go through a similar SDLC life cycle. IAC can be the part of a repository, reviewed similar to the code-base and be the part of the CI/CD pipeline to build and deploy it through automation.

CloudFormation

Verbatim: AWS CloudFormation gives you an easy way to model a collection of related AWS and third-party resources, provision them quickly and consistently, and manage them throughout their lifecycles, by treating infrastructure as code.

Beauty of CloudFormation is that it is not a tool unlike other IACs like terraform rather it is a AWS Service which provides IAC solution, which means no additional tool required to be configured. All we need is the permission to use the CloudFormation Service, then use AWS CLI or AWS Console to do rest of the operations.

Working

The working of the CloudFormation can be explained pretty well with the simple architecture diagram. First you will need to create a template where you will declare all the resources you want to create in a .json or .yaml file. Then either use the AWS SDK, AWS CLI or the AWS Console to provide your template to the CloudFormation Service and then CloudFormation takes care of provisioning and configuring the resources for you.

Hello World

Now let’s create a S3 Bucket using CloudFormation. We will use the template below and provide it to the CloudFormation Service to provision the S3 Bucket. The template can be provided using AWS CLI or through AWS Console as below.

Sample Template Please create a file name s3.yaml and add the following content.

---
Resources:
 S3Bucket:
   Type: 'AWS::S3::Bucket'
   Properties:
     BucketName: 'hello-world-demo-bucket-balman'

Deploy with AWS CLI Fire the command below to create the stack.

cd <template-directory>
aws cloudformation create-stack --stack-name hello-world-stack --template-body file://s3.yaml --profile sandbox --region sandbox

{
"StackId":"arn:aws:cloudformation:us-east-1:856960422202:stack/hello-world-stack/ae626820-1504-11ec-86e1-0e5ae768be0f"
}

OR Use AWS Console

Anatomy of Hello World Template

---
# This section defines the resources
Resources:
# Your preferred name for this resource. This is called 
# logical Id and should be unique within the template.
 MyS3Bucket:
# Resource type of the resource you want to create. eg. 
# AWS::EC2::SecurityGroup is the type for creating security group.
   Type: 'AWS::S3::Bucket'
# Set of properties that is supported by this particular service
   Properties:
     BucketName: 'hello-world-demo-bucket-balman'

CloudFormation Template

CloudFormation template is a plain text file where all the required resources and their configuration are written in the pre-defined format. CloudFormation template supports both .yml and .json. I prefer to use yaml because it is more human readable.

Structure

CloudFormation template can only have sections that are mentioned below. They can be in any order but regarding the convention and readability following ordering is done usually. Please find the respective description of each section below.

Note: Resources is the only mandatory section of template and you should define at least one resource to create.

---
AWSTemplateFormatVersion: "AWS Template versions. If not provided the latest version will be used"

Description: "Description of the template"

Metadata: "Collection of metadata objects that provides additional information about the template."

Parameters: "Values that can be passed at runtime to change the behavior of the template. It is similar to passing argument to a function"

Rules: "Validates a parameter or a combination of parameters passed to a template during a stack creation or stack update."

Mappings: "A collection of static keys and values that can be used to get values based on the conditions. eg. when you want to use different AMIs for different regions you can use region and key and AmiId as value"

Conditions: "List of conditions that can be used to create certain resources based on the conditions. eg. you might not want to create all resources in your non production environment"

Transform: "This is a more advanced topic of CloudFormation where CloudFormation templates can be transformed from one form to another form. eg. SAM template, AWS::Include can dynamically include the CloudFormation template snippets hosted in your s3 bucket"

Resources: "This is where you define all of your resources and their configurations."

Outputs: "List of certain attributes of the resource that you want to expose for eg. bucket-name for s3 bucket resource. Outputs can also be exported which can be imported in another stack for use. eg. export helps when you could have a VPC stack and another ec2 instance stack needs the VPCId"

CloudFormation Stack

During the resource creation process CloudFormation creates a separate resource called CloudFormation stack which bundles all the created resources and preserve their state, so that they can be managed later.

Analogy: CloudFormation template is like a Class whereas the CloudFormation Stack is like an object where you can create as many stacks as you want from a single template.

Authoring Template from Scratch

1. Know the Resource

The very first step is to have knowledge about the service or the resource you want to create. You should know the various attributes of the resources and know what happens when these attributes are set to default or overridden by certain values. For eg. you should be able to know what does these attributes instance_type, ami_id mean in context to creating EC2 Instance.

2. Find the documentation

All the resources supported by the CloudFormation are documented in the AWS CloudFormation docs, see the the list of supported resources. Every CloudFormation has the same format as described below(Sample docs)

Syntax:

This section provides the syntax to define particular resource in the template. Syntax must have Type(what resource to create) and its Properties(how to create).

Type: <AWS Resource Type>
Properties:
   Key: Value

Properties :

This section contains the details of individual property.

• Description: Describes the essence of the property
• Required: Shows whether property is required or not
• Type: data type of the property. eg. string, number, list
• Update requires: Specifies behavior when this property is updated. Behavior can be No Interruptions, Some interruptions, Replacement

Return Values :

Shows what value is return when referenced to the resource. It is usually name, Id, Arn of the resource.

Examples :

This section shows the sample examples of the given resource and can be helpful to get started with the template.

3. Create the template file

Let's create a minimal template to create an EC2 Instance and save it as ec2.yaml. I have only specified ImageId and SubnetId which are required to launch the Instance.

  Resources:
    MyEC2Instance: 
      Type: AWS::EC2::Instance
      Properties:
        ImageId: ami-09e67e426f25ce0d7 # replace with your own values
        SubnetId: subnet-066a074e239e6a0a3 # replace with your own values

4. Create Stack

AWS CLI

aws cloudformation create-stack --stack-name 'balman-ec2-instance' --template-body file://ec2.yaml --profile sandbox --region us-east-1

Once the stack has been created successfully you should able to see it in CloudFormation dashboard. Also in the EC2 Instances list you will be able to see the newly created EC2 Instance.

5. Managing Stack: AWS CLI

create-stack

aws cloudformation create-stack --stack-name <stack-name> --template-body file://<file-path>

update-stack

aws cloudformation update-stack --stack-name <stack-name> --template-body file://<file-path>

delete-stack

aws cloudformation delete-stack --stack-name <stack-name> --template-body file://<file-path>

wait-stack Wait commands are mostly useful when you want create your own scripts that needs to wait for certain conditions to happen.

aws cloudformation wait (stack-create-complete OR stack-delete-complete OR stack-update-complete) --stack-name <stack-name>

create-change-set Creating change-set is equivalent to creating a dry-run before executing the actual change to the running resources. This is very important when you are working with production resources.

aws cloudformation create-change-set --change-set-name <your-change-set-name> --stack-name '<stack-name>' --template-body file://<file-path>

To look for the newly created change set go through Change sets > Change-set .

execute-change-set Change set can be executed from AWS Console or AWS CLI to apply the change.

aws cloudformation create-change-set --change-set-name <your-change-set-name> --stack-name '<stack-name>' --template-body file://<file-path>

6. Managing Stack: AWS Console

One of the beauty of the AWS CloudFormation is that it provides super friendly UI to manage and see all the necessary information. Following diagram shows high level overview of what can be seen/done from the Console.

Other Concepts

1. Parameters

Parameters are like the variables of the template, they can be modified at runtime to change the attributes without needing to create a separate template. They are defined under the Parameters: section of the template. Parameter gives us various options to customize them to build reliable template like:

• Allowed Pattern(where only string with specific regex are allowed.
• Max Length, Min Length to allow input string length within the range.
• Min Value, Max Value to allow Numeric value within the range.
• Date type to allow only specific data types.
• Default values to use when no values are provided.
• Noecho option for secrets values that shouldn't be shown in the dashboard. For more details please find the AWS Docs .

Syntax:

Parameters:
  ParameterLogicalID:
    Type: DataType
    ParameterProperty: value

Example Template Now Let's evolve our previous ec2.yaml template to use the parameters.

Parameters:
  MyInstanceType:
    Type: String
    Default: t2.micro
    AllowedValues:
      - t2.micro
      - m1.small
      - t2.small
    Description: Enter t2.micro, m1.small, or t2.small. Default is t2.micro.
Resources:
  MyEC2Instance: 
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: !Ref MyInstanceType
      ImageId: ami-09e67e426f25ce0d7
      SubnetId: subnet-066a074e239e6a0a3

If you do not specify parameters it will take the default value, but if you want other than default value you can pass the parameters while update/create operation. There are two ways you can specify parameters as show below:

a. Inline

aws cloudformation update-stack --stack-name 'balman-ec2-instance' --template-body file://ec2.yaml --parameters ParameterKey=MyInstanceType,ParameterValue=t2.small --profile sandbox --region us-east-1

b. Using parameters file

create a parameter file and save as .json.

[
  {
    "ParameterKey": "MyInstanceType",
    "ParameterValue": "t2.small"
  }
]

update stack

aws cloudformation update-stack --stack-name 'balman-ec2-instance' --template-body file://ec2.yaml --parameters file://params.json --profile sandbox --region us-east-1

Any of above update will update the parameter and update the respective stack as shown below

2. Pseudo Parameters

Pseudo Parameters are built-in parameters which can be used similar to the user defined parameters. It can be accessed using Ref function. eg. AWS::Region, AWS::AccountId. Please find more details in this AWS docs .

3. Built-in Functions

CloudFormation provides us various kinds of built-in functions that we can use to make our template dynamic. Functions can be used for various kinds of cases like joining, splitting the strings, getting values from the parameters, Importing outputs from another stack, Enforcing certain conditions and many more. Please find more details in this AWS Docs.

eg. FinInMap function can be used to find the specific values from the map created in the Mappings section.

syntax: !FindInMap [ MapName, TopLevelKey, SecondLevelKey ]

sample This template supports multi-region creation of the Ec2 Instance because the AMIId will be selected by the FindInMap functions dynamically based on the region.

Parameters:
  MyInstanceType:
    Type: String
    Default: t2.micro
    AllowedValues:
      - t2.micro
      - m1.small
      - t2.small
    Description: Enter t2.micro, m1.small, or t2.small. Default is t2.micro.
  AMITYpe:
    Type: String
    Default: Ubuntu
    AllowedValues:
      - Ubuntu
      - AmazonLinux
    Description: Enter Ubuntu or AmazonLinux. Default is Ubuntu.
Mappings:
  AMIIdMap: #Use the latest values
    "us-east-1":
      "Ubuntu": "ami-09e67e426f25ce0d7"
      "AmazonLinux": "ami-087c17d1fe0178315"
    "us-east-1":
      "Ubuntu": "ami-097297e426f25ce0d7"
      "AmazonLinux": "ami-a7297e426f25ce0d7"
    "us-east-1":
      "Ubuntu": "ami-097297e426f25820d7"
      "AmazonLinux": "ami-397297e426f25820d7"
Resources:
  MyEC2Instance: 
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: !Ref MyInstanceType
      ImageId: !FindInMap [ AMIIdMap, !Ref 'AWS::Region', !Ref AMITYpe ]
      SubnetId: subnet-066a074e239e6a0a3

4. Conditions

Conditions are used to specify conditions in templates to define certain properties or resource only when conditions are met. eg. you might want to use different instance type for different environments or even want to prevent certain resource from provisioning. Please find more details in this AWS docs.

sample Template below creates condition called IsProduction and use it to change instance type based on the EnvType value and provision ProdSecurityGroup resource only when the EnvType is production.

Parameters:
  EnvType:
    Type: String
    Default: nonproduction
    AllowedValues:
      - nonproduction
      - production
    Description: Enter production, nonproduction. Default is nonproduction.
Conditions:
  IsProduction: !Equals 
    - !Ref EnvType
    - production
Resources:
  MyEC2Instance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: !If
        - IsProduction
        - 't2.micro'
        - 't2.small'
      ImageId: ami-09e67e426f25ce0d7
      SubnetId: subnet-066a074e239e6a0a3
  ProdSecurityGroup:
    Condition: IsProduction
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: create sg only if the env type is production
      SecurityGroupEgress:
      - CidrIp: 127.0.0.1/32
        IpProtocol: "-1"
      VpcId: vpc-0be6d92acf7a792d4

5. Outputs

Outputs in the CloudFormation can be used to show or exchange important resources attributes created by the template like resource Id, Name, Arn across different stacks. For eg. you might have a separate template that creates VPC and another template that creates EC2 Instance/RDS. In that case you will have to export the VPCId from the VPC template and Import VPCId into EC2Instance template. Please find more details in this AWS docs .

Syntax:

Outputs:
  Logical ID:
    Description: Information about the value
    Value: Value to return
    Export:
      Name: Value to export

Example For the sake of simplicity I have two templates one creates security group and exports its Id and the other template Imports the Id and use it in the EC2 Instance.

sg-outputs.yaml

Resources: 
  ProdSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: create sg only if the env type is production
      SecurityGroupEgress:
      - CidrIp: 127.0.0.1/32
        IpProtocol: "-1"
      VpcId: vpc-0be6d92acf7a792d4

Outputs:
  SecurityGroupId:
    Description: Id of the SecurityGroup
    Value: !Ref ProdSecurityGroup #This will get Id of the Security Group. Please look at Return values section to find more details.
    Export:
      Name: demoSecurityGroupId

outputs section

Now let's use the template below to import the security group id ec2-outputs.yaml

Resources:
  MyEC2Instance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: 't2.micro'
      ImageId: ami-09e67e426f25ce0d7
      SubnetId: subnet-066a074e239e6a0a3
      SecurityGroupIds: 
        - !ImportValue demoSecurityGroupId

Final Thoughts

This is no way the exhaustive details of the CloudFormation but a mere high level overview of various components that are needed to get anyone started with it. There are other advanced topics that are completely left untouched in this blog: nested CloudFormation, custom CloudFormation, CloudFormation modules, Failure Handling, CloudFormation Registry, Transform, Macros which can be resources for other blogs. Please find the sample CloudFormation files in this Github Repo.

Any comments, feedback are welcome. Thank You.