Securely Storing Images on Amazon S3 with Presigned URLs in Golang

Several weeks ago, I was mentoring a capstone team at Bangkit Academy when I received a question about how to store images in the back-end. That’s why I decided to write this article.

When building a REST API, you might need an upload feature for files such as images, videos, audio, or documents. You can save the data locally. However, the problem with this approach is that you need a large amount of disk space to store files like images or videos.

So, What is the Solution?

Amazon S3.

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance.

Some terminology in Amazon S3 is a object and bucket. An object is a file and any metadata that describes the file. A bucket is a container for objects.

To save object in Amazon S3, you need create a bucket if you don’t have. Then, specify a bucket name and AWS Region. Each object has a key (or key name), which is the unique identifier for the object within the bucket.

Feature of Amazon S3

• Storage Classes
  By default, objects in S3 are stored in the S3 Standard storage class. Amazon S3 offers storage classes that match your use case, such as S3 Standard, S3 Standard-IA, S3 Intelligent-Tiering, and more.
• Storage Management
  Amazon S3 offers storage management features to help manage costs, meet compliance requirements, and reduce latency. With storage management, you can utilize features such as S3 Lifecycle, S3 Object Lock, S3 Replication, and S3 Batch Operations.
• Access Management and Security
  Amazon S3 has features to manage access to your objects. These features include S3 Block Public Access, Access Control Lists, and more.
• Data Processing
  To process data, you can use feature S3 Object Lambda and Event notifications.
• Storage Logging and Monitoring
  You can monitor your object on Bucket with Amazon CloudTrail and Amazon CloudWatch.
• Analytics and insights
  To understand, analyze, and optimize your usage in S3, AWS have a feature Amazon S3 Storage Lens. Or, you can use Storage Class Analysis.
• Strong consistency
  Don’t worry about your data consistency because S3 have a strong consistency for PUT and DELETE object. For more information, see Amazon S3 data consistency model.

That’s a brief summary of some Amazon S3 features. Let’s look at the implementation of S3 in the case study.

Before we start, make sure you have the following requirements:

• An AWS Account
• Access to Amazon S3
• AWS Secret Key and Access Key

Case Study

I have a simple REST API built with Go. The REST API includes a feature to support uploading user profile photos. Let’s say we have an endpoint v1/image with the POST method. In the request body, we include a form-data field with the name file.

So, lets go to the code 😄

We will build a REST API using the Echo Framework. In this project, we will have a handler, repository, and use case.

In handler.go, i wrote code like this.

func (h *handlerUser) UploadImage(c echo.Context) error {

 _, err := h.jwtAccess.GetUserIdFromToken(c)

 if err != nil {
  return c.JSON(http.StatusUnauthorized, "Unauthorized")
 }

 file, err := c.FormFile("file")
 if err != nil {
  return c.JSON(http.StatusBadRequest, "bad request")
 }
 
 // Validate content type
 if file.Header.Get("Content-Type") != "image/jpeg" && file.Header.Get("Content-Type") != "image/jpg" {
  return c.JSON(http.StatusBadRequest, "Invalid file format. Must be in JPEG or JPG format.")
 }

 // Validate file size
 if file.Size > 2*1024*1024 {
  return c.JSON(http.StatusBadRequest, "File size exceeds the limit. Maximum file size is 2MB.")
 }

 // Validate minimum file size
 if file.Size < 10*1024 {
  return c.JSON(http.StatusBadRequest, "File size is too small. Minimum file size is 10KB.")
 }

 src, err := file.Open()
 if err != nil {
  return c.JSON(http.StatusBadRequest, "bad request")
 }
 defer src.Close()

 bucket, err := h.usecase.UploadImage(src, file)

 if err != nil {
  return c.JSON(http.StatusInternalServerError, "failed to upload image")
 }

 return h.formatResponse.FormatJson(c, http.StatusOK, "success", map[string]interface{}{
  "imageUrl": bucket,
 })
}

The UploadImage function handles an image upload, validate the content type and file size. The file will be process through usecase.UploadImage.

So, let’s jump to usecase.go.

func (u *usecase) UploadImage(file multipart.File, fileHeader *multipart.FileHeader) (url string, err error) {

 buf := bytes.NewBuffer(nil)
 if _, err := buf.ReadFrom(file); err != nil {
  return "", fmt.Errorf("failed to read file: %v", err)
 }

 fileType := fileHeader.Header.Get("Content-Type")

 key := utils.GenerateUUID()
 key += ".jpg"

 cfg, err := config.LoadDefaultConfig(context.TODO())
 if err != nil {
  log.Fatalf("failed to load configuration, %v", err)
 }

 svc := v2.NewFromConfig(cfg)

 _, err = svc.PutObject(context.TODO(), &v2.PutObjectInput{
  Bucket: awsv2.String(os.Getenv("AWS_BUCKET")),
  Key:    awsv2.String(key),
  Body:          bytes.NewReader(buf.Bytes()),
  ContentLength: awsv2.Int64(fileHeader.Size),
  ContentType:   awsv2.String(fileType),
 })

 if err != nil {
  log.Fatalf("failed to put object, %v", err)
 }

 sess, err := session.NewSession(&aws.Config{
  Region: aws.String(os.Getenv("AWS_REGION"))},
 )

 if err != nil {
  log.Fatalf("failed to create session, %v", err)
 }

 client := s3.New(sess)

 urlBucket, _ := client.GetObjectRequest(&s3.GetObjectInput{
  Bucket: aws.String(os.Getenv("AWS_BUCKET")),
  Key:    aws.String(key),
 })

 urlStr, err := urlBucket.Presign(15 * time.Minute)

 if err != nil {
  log.Println("Failed to sign request", err)
 }

 url = urlStr

 return
}

The UploadImage function in usecase handles the process of uploading an image to an AWS S3 bucket. It reads the file into a buffer, generates a unique key for the image, and uses AWS SDK for Go v2 to upload the file to S3.

Look at this code.

_, err = svc.PutObject(context.TODO(), &v2.PutObjectInput{
  Bucket: awsv2.String(os.Getenv("AWS_BUCKET")),
  Key:    awsv2.String(key),
  Body:          bytes.NewReader(buf.Bytes()),
  ContentLength: awsv2.Int64(fileHeader.Size),
  ContentType:   awsv2.String(fileType),
 })

We use the AWS SDK v2 and the PutObjectInput method with parameters such as Bucket (retrieved from the environment) and Key (a unique key for the file name).

Next, look at to this block code.

urlBucket, _ := client.GetObjectRequest(&s3.GetObjectInput{
  Bucket: aws.String(os.Getenv("AWS_BUCKET")),
  Key:    aws.String(key),
 })

 urlStr, err := urlBucket.Presign(15 * time.Minute)

After the file is uploaded, we create a presigned URL for accessing the image with a 15-minute expiration time, which will be returned as the result of the function.

By default, all Amazon S3 objects are private, only the object owner has permission to access them. You can use presigned URLs to grant time-limited access to objects in Amazon S3 without updating your bucket policy. A presigned URL can be entered in a browser or used by a program to download an object. The credentials used by the presigned URL are those of the AWS user who generated the URL.

Testing

We can test uploading the image to the endpoint. If successful, you will get a result like this

If you want to access the image you uploaded, you can copy the link from the response and paste it into your browser. You will then see the image you uploaded.

Voila! We success upload the image to the Amazon S3.

Thanks for reading ~~

Reference:

Getting started with Amazon S3

Getting Started with the AWS SDK for Go V2