Project 15 → Deploying -10 Microservices on Kubernetes cluster via Jenkins CI/CD Pipeline

Project Overview

In this project we deploy 10 microservices in kubernetes cluster using jenkins pipeline….first we launch a Base server in which all the task is performed….then we take ssh of that EC2….next we install docker , jenkins , configure java ,eksctl , kubectl etc… moving forward to setup k8s cluster in aws that is EKS and atlast we run a multibranch pipeline that fetch code from github and deploy it on EKS cluster that is all about the project let’s begin

Completion steps →

Step 1 : Set a Base Instance

Step 2 : create an IAM user with specific permission

Step 3: Kubernetes

Step 4: Install Essential tools

Step 5: Setup jenkins

Step 6: create jenkins pipeline

Step 1 : Set a Base Instance

1. Go to aws console and launch a instance with given configuration

a. AMI = ubuntu 24

b. Type = t2.large

c. Storage = 30GB

d. ports to be opened = 22 , 80 ,443 ,8080

2. Take a ssh of your ec2 >>click on connect>>ssh client

3. copy the command and open your cmd and locate the download folder where your key pair (.pem file) is stored

run the above command

Step 2 : create an IAM user with specific permission

1. go to aws console and search for IAM and then click on user>>create user

follow as shown above

attach specific policies directly as shown above

• AmazonEC2FullAccess
• AmazonEKSClusterPolicy
• AmazonEKSWorkerNodePolicy
• AWSCloudFormationFullAccess
• IAMFullAccess

download username and password

2. create an Inline policy for eks

click on create inline policy>>json and enter the following script

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "eks:*",
            "Resource": "*"
        }
    ]
}

3. After this create a access keys and secret keys to work with awscli in base ec2

4. Download your secret and acess keys that we need to use later in the project

Step 3: Kubernetes

1. go to your ec2 and run

aws configure

open .csv file to provide access keys and secret keys

2. Now you are log into aws via cli with limited permission provided in earliar steps

3. Create an Eks cluster using the following command

eksctl create cluster --name=EKS-1 --region=ap-south-1 --zones=ap-south-1a,ap-south-1b --without-nodegroup

4. Associate OIDC Provider →An OIDC provider (OpenID Connect provider) is like an identity verification service for applications. It ensures that people or systems accessing your application or cloud resources are who they claim to be.

Imagine you want to enter a secure building. At the entrance:

1. You show your ID card to prove your identity.
2. The security guard checks your ID against a trusted database.

In this case:

• Your ID card is the identity token.
• The OIDC provider is the trusted system (the database) that confirms your ID is valid.

eksctl utils associate-iam-oidc-provider --region ap-south-1 --cluster EKS-1 --approve

5. Create Node Group:

eksctl create nodegroup --cluster=EKS-1 --region=ap-south-1 --name=node2 --node-type=t3.medium --nodes=3 --nodes-min=2 --nodes-max=4 --node-volume-size=20 --ssh-access --ssh-public-key=DevOps --managed --asg-access --external-dns-access --full-ecr-access --appmesh-access --alb-ingress-access

6. update kubeconfig →This command aws eks update-kubeconfig is used to configure your local Kubernetes kubectl command-line tool to interact with an Amazon EKS (Elastic Kubernetes Service) cluster.

aws eks update-kubeconfig --region ap-south-1 --name EKS-1

cluster created by aws-cli

Node created by cluster

a. Create a namesapce:

A namespace in Kubernetes is like a folder for organizing resources. It helps you group and manage Kubernetes objects (like pods, services, or secrets) that belong to the same team, project, or environment.

kubectl create namespace webapps
kubectl get namespaces

b. Creating Service Account:

A Service Account in Kubernetes is like a special user for applications running in your cluster. It allows pods (your running applications) to interact with the Kubernetes API or other systems securely.

vim svc-acc.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
  namespace: webapps

kubectl apply -f svc-acc.yaml

c. Create role :

It is used to define a Role or RoleBinding for an application in your cluster. It specifies what actions the application (or a service account used by the application) is allowed to perform on specific resources within a namespace.

• A Role in Kubernetes grants permissions within a single namespace.
• It specifies what actions (like get, list, create, delete) are allowed on certain resources (like Pods, Services, ConfigMaps).

vim app-role.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: app-role
  namespace: webapps
rules:
  - apiGroups:
        - ""
        - apps
        - autoscaling
        - batch
        - extensions
        - policy
        - rbac.authorization.k8s.io
    resources:
      - pods
      - componentstatuses
      - configmaps
      - daemonsets
      - deployments
      - events
      - endpoints
      - horizontalpodautoscalers
      - ingress
      - jobs
      - limitranges
      - namespaces
      - nodes
      - pods
      - persistentvolumes
      - persistentvolumeclaims
      - resourcequotas
      - replicasets
      - replicationcontrollers
      - serviceaccounts
      - services
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

d. Bind the role to service account:

• A RoleBinding binds a Role to a user, group, or service account, allowing them to use the permissions defined in the Role.

vim role-bind.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: app-rolebinding
  namespace: webapps 
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: app-role 
subjects:
- namespace: webapps 
  kind: ServiceAccount
  name: jenkins 

e. Create a token by utilizing a service account in the desired namespace

A secret token in Kubernetes is used to allow a pod or application to securely prove its identity and access resources it has permission for. It’s like a password or keycard for your application.

vim secret.yaml

apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
  name: mysecretname
  annotations:
    kubernetes.io/service-account.name: jenkins

kubectl apply -f secret.yaml -n webapps

Now run →

kubectl describe secret mysecretname -n webapps

copy the token and paste it to your notepad for future refrence

Step 4: Install Essential tools

back to EC2 and run the following commands first

sudo su
apt update

1. Now make a shell script file to install the docker , jenkins , awscli etc….

vim install.sh

#!/bin/bash
sudo apt update -y
wget -O - https://packages.adoptium.net/artifactory/api/gpg/key/public | tee /etc/apt/keyrings/adoptium.asc
echo "deb [signed-by=/etc/apt/keyrings/adoptium.asc] https://packages.adoptium.net/artifactory/deb $(awk -F= '/^VERSION_CODENAME/{print$2}' /etc/os-release) main" | tee /etc/apt/sources.list.d/adoptium.list
sudo apt update -y
sudo apt install temurin-17-jdk -y
/usr/bin/java --version

#install jenkins
curl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key | sudo tee /usr/share/keyrings/jenkins-keyring.asc > /dev/null
echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] https://pkg.jenkins.io/debian-stable binary/ | sudo tee /etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt-get update -y
sudo apt-get install jenkins -y
sudo systemctl start jenkins
sudo systemctl status jenkins

#install docker
sudo apt-get update
sudo apt-get install docker.io -y
sudo usermod -aG docker ubuntu
sudo usermod -aG docker jenkins
newgrp docker
sudo chmod 777 /var/run/docker.sock
sudo systemctl restart jenkins


# Install AWS CLI 
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
sudo apt-get install unzip -y
unzip awscliv2.zip
sudo ./aws/install


# Install kubectl
sudo apt update
sudo apt install curl -y
curl -LO https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
kubectl version --client

# Install eksctl
curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin
eksctl version

2. Above script will install jenkins , docker ,kubectl ,eksctl etc….. tools

press →esc+:wq for exit

run the above command to execute the script

Step 5: Setup jenkins

1. Go to your browser and type>> public_ip:8080 (to access jenkins console)

2. for this go to your ec2 and connect it

3. run the below commands

sudo su
cat /var/lib/jenkins/secrets/initialAdminPassword

output is your password and paste it to your jenkins

4. Install the suggested plugins

5. Setup your jenkins user

Welcome to jenkins dashboard

a. Install Plugins listed below

1. docker (all plugins)
2. kubernetes (all plugins)
3. Multibranch scan webhook trigger

b. Setup docker credentials

1. go to your jenkins →manage jenkins →credentials →global →add credentials
2. provide your username and password of your dockerhub
3. id==docker

c. setup git credentials →click on me to see how to generate token in github

1. Now copy that token from github and
2. go to your jenkins →manage jenkins →credentials →global →add credentials
3. provide your username and password of your github
4. id==github

click on create

d. set k8s token

remember the token we generated above

1. go to credentials section chose secret text in kind section and paste your token in secret id==k8-token and then click on create

all the credentials are setup let’s move to pipeline script

Step 6: create jenkins pipeline

1. click on New item and chose multibranch pipeline

click ok

2. Now chose git from add source option and paste the project repo url there

chose git credentials from dropdown that we setup in above steps

3. Now we need to setup webhook in github to trigger the pipeline

4. go to your project repo in github and click on setting>>webhooks>>Add a webhook

in Url section type → http://public_ip:8080/multibranch-webhook-trigger/invoke?token=microservice

your webhook is ready and now the pipeline will automatically triggers whenever we make a change in the project repo

5. go back to your jenkins pipeline dashboard

go to your ec2 and run kubectl get svc -n webapps

copy the Load-balancer url and paste it on browser !!! make sure using http not https

Here is website having 10 microservice running in

Usual Error in Jenkins Pipeline →make sure to change api server endpoint in jenkins file

copy this endpoint and replace in jenkinsfile

paste your endpoint here

Now this project is completed here !!!

follow me on linkedin

follow me on github