Check the Integrity of Data in Amazon S3 with Additional Checksums

Check the Integrity of Data in Amazon S3 with Additional Checksums

·

4 min read

Ever wondered if your data has been messed with? Well with AWS S3, I can show you how!

Once you have created an AWS account, the lab starts with creating an S3 bucket and ends with comparing our checksum values.

TLDR

The goal of this lab is to upload a file to Amazon S3, enable checksum on S3, calculate additional checksums, and compare the checksum on Amazon S3 and your local file to verify data integrity.

Some links below detailing important concepts:

How to create an AWS account

What is S3

What is a Checksum

What is Encryption

What is SHA 256

Create S3 Bucket

To create a S3 bucket, you must create an AWS account first. Simply go to https://aws.amazon.com/console/ and click "Create an AWS account". From there, the instructions should be straightforward to set up an account.

After creating an account, type "S3" and under services, click "S3".

From there, click "Create bucket"

After this, enter a unique name for your bucket. You may have to add numbers or special characters to get a unique name. Spaces and capital letters cannot be used. Then select an AWS Region you would like your bucket to be created in. I usually choose the us-east-1, N'Virginia region. For the rest of the options, you can leave them as is.

Upload File with Additional Checksums enabled

On the bucket that was created, click on the bucket and select upload. You can upload any file you desire like a pdf, word doc etc. I chose a simple png file that shows the picture I used as a cover photo for this blog. After adding the file, scroll down to the bottom of the upload section and under the additional checksum section, click enable.

Then click SHA 256 as the function. Then click upload.

Compare Checksum from S3 to that on your local file

Navigate to the properties tab of the file you uploaded: Click on the bucket > Click on the file uploaded> Click on properties> Scroll down to additional checksums.

Here you will find the base64 encoded checksum that Amazon S3 calculated and verified at the time of upload.

Screenshot showing Checksum from AWS S3 bucket

To compare both checksum values, there is a specific command that was instructed to be used from the Amazon instructions :

"The following command performs a sha256 calculation on the same file and converts the hex output to base64: shasum -a 256 image.jpg | cut -f1 -d\ | xxd -r -p | base64"

Before using this, however, it is wise to know what each statement/variable represents.

  • shasum: This command is used to compute SHA message digests. A message digest is a fixed-size hash value

  • a -256: Hash function

  • cut: This Linux command can take out sections of a specified file or piped data. It then prints the result

  • -f1: Used to select a specific field when using the cut command.

  • -d: Used to specify a delimiter when using the cut command.

  • xxd: Used to create a hex dump from a file

  • -r: Used to list files/directories in a reverse order

  • -p: Used to prevent shell reading user-controlled files

  • base 64: This is a group of binary-to-text encoding schemes that represent binary data in an ASCII string format.

  • |: The " | " command (known as Pipe command), allows an individual to use two or more commands such that an output of one command is used as the input for the next command.

Screenshot showing checksum on local computer

Screenshot showing checksum on AWS S3 bucket

As you can see above, when you compare the values from your local computer to that in the AWS console, you can see they both match. Thus the file was not tampered with and the integrity was maintained.

Voila

With the completion of this project, it's time to clean up to avoid surprise costs. Simply delete the test object and the test bucket by clicking on the bucket and selecting delete. Sometimes, you may have to empty the bucket first before deleting the bucket. In my own experience recently, I've incurred these costs but thankfully I set some billing alarms. Due to these alarms, I was able to delete the services I had running and save myself from further costs. You can find more on alarms here. Comparing and contrasting checksums is important in cybersecurity so that one can verify if a file was tampered with or not.

Resources:

https://aws.amazon.com/getting-started/hands-on/amazon-s3-with-additional-checksums/?ref=gsrchandson

https://aws.amazon.com/s3/