April 12, 2023

CREATING CUSTOMIZED DOCKER IMAGES – PART5

AWSDevOpsKubernetes
By admin 0 Comments
Docker 1

 

Part 5

 

In Part 5 of this 5-part tutorial on creating Customized Docker images, we will complete the integration of our “remote-host” container with Jenkins. The purpose of creating the “remote-host” container was created to take an automatic backup of the WordPress MySQL database deployed on the Kubernetes cluster.

In Part 4 of this 5-part tutorial, we configured the AWS Secret ID,  AWS Secret Key and MySQL password credentials in Jenkins.

In Part 3 of this 5-part tutorial, we logged into the “remote-host” pod and took a manual backup of the WordPress MySQL database. 

We will now modify the setup for our “remote-host” pod’s integration with Jenkins and automate the entire backup process. 

 

 

Step 1: Understanding the need for mounting the shell script on the pod. 

 

 

In Part 3 of this 5-part tutorial, specifically in Step 2, we had created directories on the Worker Node. We created the “remote-host-data-files” directory. Inside the “remote-host-data-files” directory, we created another “tmp” directory. 

We will store the shell script “my-wp-db-backup-s3.sh” we created in Part 4.

 
Important: Why are we storing the “my-wp-db-backup-s3.sh” in the “tmp” directory?

When automating the process through Jenkins, we need to use the shell script created in Step 4 (mentioned above). We can copy this script from the host machine (VM) onto the “remote-host”  pod. This will work as long as the “remote-host” pod is active.

However, if the “remote-host” pod gets killed or restarts, then this script is no longer available on the pod, which effectively means our automatic mechanism to take backups will fail as the shell script that defines all the necessary parameters and how and where the backup should be taken. 

By storing the “my-wp-db-backup-s3.sh” on the “tmp” directory on the Worker Node, creating a PV (Persistent Volume) and PVC (Persistent Volume Claim) and mounting them on the pod in the deployment file, we ensure that the “my-wp-db-backup-s3.sh” is permanently mounted on the “remote-host” pod. Even if the pod restarts, the “my-wp-db-backup-s3.sh” script will automatically get mounted on the pods. This ensures that the automated Jenkins “job” to take a backup never fails, due to the non-presence of the backup script. 

Below is an extract of the “remote-host-deploy.yaml” file

spec:
  containers:
    - image: 123456789012.dkr.ecr.ap-south-1.amazonaws.com/remote-host:1.1 
      name: remote-host
      ports:
        - containerPort: 22
      volumeMounts:
        - name: remote-host-script
          mountPath: "/home/jenkins/remote-host-data-files/my-wp-db-backup-s3"
          subPath: my-wp-db-backup-s3.sh
        - name: remote-host-tmp
          mountPath: "/tmp"

  volumes:
    - name: remote-host-script
      persistentVolumeClaim:
            claimName: remote-host-pvc
    - name: remote-host-tmp
      persistentVolumeClaim:
            claimName: remote-host-pvc-tmp

 

 

 

Step 2: Create a new Jenkins job to take backups. 

 

 

 
What is a Jenkins job?

Jenkins jobs are a specified set of tasks that runs sequentially as defined by the user. Any automation implemented in Jenkins is a Jenkins Job. These jobs are a significant part of Jenkins’s build process. We can create and build Jenkins jobs to test our application or project. 

Before creating the Jenkins job, let’s look at our “my-wp-db-backup.sh” shell script again.

#!/bin/bash
DB_HOST=$1
PASSWD=$2
DB_NAME=$3
AWS_ACCESS_ID=$4
AWS_SECRET=$5
AWS_DIR=$6

DATE=$(date +%d%m%y-%H%M%S)
BACKUP=my-db-backup-$DATE.sql

mysqldump -u root -h $DB_HOST -p$PASSWD $DB_NAME > /tmp/$BACKUP &&\
export AWS_ACCESS_KEY_ID=$AWS_ACCESS_ID &&\
export AWS_SECRET_ACCESS_KEY=$AWS_SECRET &&\
export AWS_DEFAULT_REGION=ap-south-1
echo “Uploading ….”

aws s3 mv /tmp/$BACKUP s3://$AWS_DIR/$BACKUP ### This moves the sql file directly to s3 ###

As you can see there are 3 Parameters and 3 secret credentials that have been defined in this script.  The 3 secret credentials have already been specified in Step 3: Part 4 of the tutorial. We will now specify the other 3 parameters. 

 

Open the Jenkins dashboard

  • Dashboard –> New Item –>  “my-wp-db-backup-s3” –> Free style project –> OK

 

Once the “job” or “project” is created it opens a “Configure” page.

 

 

Step 3: Configuring the Parameters in Jenkins

 

 

We will now configure the 3 Parameters in Jenkins. These are the parameters which we have defined in 

Note: The parameter name defined in the script and the parameter name defined in Jenkins DO NOT have to be the same. ONLY the order of the parameters has to be the SAME. We look into this when we define the parameters. 

  • Configure — General –> “This project is parameterised”

    • Add Parameter –> String Parameter –> Name –> “MYSQL_DATABASE_HOST” (You can provide any name) –> default value –> 10.96.0.100” (Provide the ClusterIP of the MySQL service as hostname may not work) 

    • Add Parameter –> String Parameter –> Name –> “DATABASE_NAME” –> default value –> “wordpress” (provide the name of the database to be backed up) 

    • Add Parameter –> String Parameter à Name –> “AWS_S3_BUCKET” –> default value –> “atpl-mysql-db-backup” (provide the name of the S3 bucket where backups are to be stored. This S3 bucket was created in Step 4: Part3)       

 

 

Step 4: Configuring the secret credentials in Jenkins

 

 

Next, we will configure the 3 secret credentials in Jenkins. These are the secret credentials that we have defined in our shell script. 

Note: The credential name defined in the script and the credential name defined in Jenkins DO NOT have to be the same. ONLY the order of the parameters has to be the SAME. We look into this when we define the parameters. 

Scroll down the page till you see the “Build Environment“

  • Build Environment –> Use secret text(s) or file(s) –> Bindings –> 

    • Add –>  Secret text –> Variable –>  “MYSQL_PASSWORD” –> Credentials –> (select the secret MySQL database password created in Step3: Part4)

    • Add –> Secret text –> Variable –> “AWS_SECRET_KEY” –> Credentials –> (select the secret AWS secret key created in Step3: Part4)

    • Add –> Secret text –> Variable –> “AWS_ACCESS_ID” –> Credentials –> (select the secret AWS access ID created in Step3: Part4)

Below is a screenshot of the secret credentials.

With this, we have defined the 3 parameters and 3 secret credentials in Jenkins. Now we can execute the shell script from Jenkins.

Just below the credentials, click on “Build Steps“

  • Build Steps –> 

    • Execute shell script on remote host using ssh –> SSH site–> remote_user@10.97.129.185:22  –> Command (enter the below text) –> 

    • /tmp/my-wp-db-backup.sh $MYSQL_DATABASE_HOST $MYSQL_PASSWD $DATABASE_NAME $AWS_ACCESS_ID $AWS_SECRET_KEY $AWS_S3_BUCKET

The command mentioned above with the parameters is based on our shell script “my-wp-db-backup.sh”. Each parameter that we are entering is highlighted below.

Note: This is just for your reference.

#!/bin/bash
DB_HOST=$1                   –> $MYSQL_DATABASE_HOST
PASSWD=$2                    –> $MYSQL_PASSWD
DB_NAME=$3                  –> $DATABASE_NAME
AWS_ACCESS_ID=$4     –> $AWS_ACCESS_ID
AWS_SECRET=$5.           –> $AWS_SECRET_KEY
AWS_DIR=$6.                   –> $AWS_S3_BUCKET

 

Below is a screenshot of the Build step

Click “Save“

 

 

Step 5: Build the Jenkins job

 

 

In this step, we will build our Jenkins “project” or “job” and see if everything works fine end-to-end.

Go to the Jenkins home page. You will see the “project” or  “job”. The job we have created “my-wp-db-backup-s3” should show, click on it.

It will open the page, where we will build the project or job with our predefined parameters. 

  • Dashboard –> my-wp-db-backup-s3 –> Build with parameters –> (This will show the default parameters that we have entered, you can change these values if required) –> Build

Note:  The parameters are automatically populated, as we have pre-defined them in the previous steps. You can change these parameters if you want to take a backup of a different database or store the backup in a different S3 bucket.  

 

Below is a screenshot of the project with build parameters.

Once you click on the “Build” button, it will bring you to the page, where you can see the build number. Click on the build number. 

 

On this page, we can view the console output and check if the build is successful. If there is any “Failure”, it will mention the reason for the failure, so that it can be rectified.

  • Dashboard –> “my-wp-db-backup-s3” (project or job) –>  build number –> Console Output

Once you click on the “Console Output” you can view the status of the build. If there are any errors, then they must be resolved. 

If the build is successful, then the “console output” will show the below message:

[SSH] completed
[SSH] exit-status: 0
Finished: SUCCESS

If the build is “SUCCESSFUL“, only then should go to the final step of the tutorial.

 

Step 6: Automate the backup process using corn

 

 

So far we have manually triggered the job using Jenkins. However, this is not an ideal solution, as backups should be automated and backups should happen without any intervention by a user. 

Jenkins offers an option to configure a job to be completed at a specific date and time, using “cron” like features. This feature is primarily for using Jenkins as a cron replacement, and it is not ideal for continuously building software projects.

For the syntax of using cron, please refer to the below link for getting the syntax as per your backup requirements. 

https://crontab.guru/

In our setup, we will configure the “cron” to t take a daily backup at 01:30 am, of our WordPress MySQL database. The below syntax will take perform a task at 01:30 AM, every day (month), every week, every day (week)

30 1 * * *

Now let’s enter this value in our “cron” 

  • Dashboard –> “my-wp-db-backup-s3” –> Configure –> Build Triggers –> Build periodically –> (Enter the above value) –> Save

However, Jenkins recommends not to use the value “30 1 * * *” as there could be multiple loads scheduled at the same time. So Jenkins recommends, modifying this value, so that it can spread the load evenly. Jenkins recommends using the above value as mentioned below:

“H 1 * * *“

 

Below is a screenshot of the configuration. 

Once saved, Jenkins will take an automated backup of the MySQL database at the specified date and time without any intervention from the user. 

You can use this setup to take a backup of multiple databases or any other database. You just need to install the appropriate client in your “remote-host” container and rebuild the container, if you are taking a backing of any another RDS. 

 

CONGRATULATIONS !!! You have created your own container, deployed that container on a Kubernetes cluster where WordPress and Jenkins are deployed and integrated the container with Jenkins to take an automated backup of the MySQL database. 

 

Part 4 –> Integrating the pod with Jenkins