Using CDN to Prevent Broken Website During Deployment

Feb 22, 2019

Using CDN to Prevent Broken Website During Deployment

We recently launched our new website 🍻 and during the development of this new website I have learnt a lot of things. One of them is how CDN help us to prevent broken during deployment phase on our continuous integration (CI). As the background, we just implemented CI on our services and I have a plan to use CDN for our website but never had a time to implement that until this problem happen.

We are using combination of Gitlab CI and AWS Elastic Container Service (ECS) for all our docker container. Currently we have around 10 services running for all our system environment. One of those service is our new front-end website that build using Express.js and Nuxt.js. This new front-end website is heavily using javascript and have Server-Side Rendering (SSR) functionality. Initially we want to have fully Single-Page Application (SPA) but this kind of the application will make the search engine bots hard to crawl our website (Except Google bot) and this is the reason we have SSR functionality on our new website.

During the deployment process, ECS will launch new container (or they called Task) depending the number desired that has been set up on the Service configuration. If desired task number is 2, then ECS will launch 2 new Task based on new Task Definition. This 2 new Task will add to the Service alongside of the 2 old Task that has been there from previous deployment.

Then if all the new Task has reach RUNNING state, ECS will drain connection to each old Task before shutting the Task down. This process usually takes minutes before all of old Task deleted. During this transition, if the service is a front-end website service which serves HTTP connection to user browser, there will be a chance this transition make website broken to the user.

Why?

Before we reach deploy process, we have a “build” process on our CI. In the build process, we build all of Vue templates (since we are using Nuxt.js) into javascript bundles and compile Typescripts files into plain javascript (we are using Typescript for our Express.js application). This process will generate new production-ready javascript bundle which contain hash-versioning in the filename. Example: app.072413282a677463d5a9.js. Then this files is registered to internal manifest application so later when the website visited by the user it can serve correct file version to the user . All of the compiled files is saved on the container locally and here comes the problem.

Lets start some illustration. As I explained before, during the deployment ECS will launch new Task before removing old Task. So there is a moment both old and new task served to the load balancer.

User A then visit the site during the deployment process and load balancer redirect the HTTP connection to a container (let’s ignore algorithm that used by load balancer to determine the target container for this moment). In this case the container that receive the HTTP request is Task A, which is the old container. Task A generate HTML response to the user browser, and the HTML response is contain script or style tag that load old version that stored locally on manifest file on Task A, i.e. app.old.js.

After HTML response loaded to user browser, browser then try to load script that has been defined on the HTML which is app.old.js. This request then passed to the load balancer, and the load balancer decided to send the traffic to Task B which is new container. Since all the static resources is stored locally, so there is no app.old.js on Task B. There is only app.new.js on it. Browser then cannot load the resources and finally the website is broken.

The problem looks simple, because this only happen during the deployment process which last for couple minutes. But what if the user that have this problem is new user that coming to the website for very first time. User saw broken page or functionality during checkout process and then decided not to continue with the payment.

Of course there is a helpful user that try to contact the customer service to make sure they can make a payment, but that type of the user is pretty rare. Mostly if user see the page is broken, they just leave the website and never coming back again. This problem is bad for the business and we need to fix the problem.

How?

The solution is simple. All we need to do is move all the static resources to third party storage service. Since we are using Amazon Web Service, we can use their S3 service to store the files. Then I thought, why just use S3 and why not we use CloudFront (CDN) to sit in front of S3 bucket.

Honestly, I’ve never have experience to setting up CDN at all. So this is quite interesting for me. Since the technology is very specific I will explain technical details how we manage to create automated CDN deployment during our build process on CI. As the background we are using Nuxt.js as the wrapper of Vue.js to allow us easily configured Server-Side Rendering, then we wrap again Nuxt.js inside of Express.js, so we can serve both Nuxt.js SSR page and the website API in the same application.

Before we able to serve the static file on the CloudFront, we need to upload all the files to S3 as the Origin resources. You can find a lot of resources how to make S3 bucket as your Origin resources on CloudFront. In our new site there is 2 type of resources, static and dynamic resources.

Dynamic Resources is a file like photos or videos that uploaded by our user. This resources is already stored on S3 so we don’t really do much to make this resources can be serve through CloudFront. It is just a matter changing the URL from S3 URL to CloudFront URL. For example we set our DYNAMIC_PUBLIC_RESOURCE_PREFIX from https://s3-us-west-2.amazonaws.com/bucket-name/static to https://random-subdomain.cloudfront.net/static.

Static Resources is the file that generated during build process, it usually a form of javascript files, CSS file and small image files like icons or background image. When the static resource is built, all the url like images or font will refer to the local one, so we need to setup additional configuration on nuxt.config.js to make sure the URL is converted to CloudFront endpoint when the resource is compiled during CI build process.

After I search through the code base, there is 3 cases where static resource url is used on the application, background-image CSS property and fonts asset on our SCSS file (we are using SCSS as our CSS pre-processor), background-image CSS property on Vue templates, and Image src attribute HTML on Vue template.

Asset URL on SCSS files

Changing asset URL on SCSS files is simple. All you need to do is putting CloudFront CDN URL on publicPath of nuxt.config.js build configuration. Since we store the URL on environment variable during build we can just use CLOUDFRONT_ENDPOINT environment variable to retrieve the URL. Also we have to make sure this is only applied on production or staging build, so we depends on the NODE_ENV, to make sure we are using default /build/ directory only on development. But we have to make sure that it already imported on Nuxt.js environment variable, because if not you cannot use CLOUDFRONT_ENDPOINT and NODE_ENV on the configuration.

If we summed up all of this configuration, the result pretty much like

this:module.exports = {
  env: {
    NODE_ENV: process.env.CLOUDFRONT_ENDPOINT,
    CLOUDFRONT_ENDPOINT: process.env.CLOUDFRONT_ENDPOINT
  },
  build: {
    publicPath:
      process.env.NODE_ENV !== "development"
        ? process.env.CLOUDFRONT_ENDPOINT
        : "/build/",
  }
}

background-image CSS property on Vue templates

Next is how to convert background-image CSS property that contain url() to use CloudFront Endpoint instead using the local one. In Vue templates, we have <style> sections which contain CSS property for the component. In our website, we only have one general CSS file which formatted as SCSS file, then all of the style should placed in the Vue component, since it make us more easier to organize the code. By setting public path configuration seems doesn't automatically changing URL of this background-image property. Finding a solution of this problem is quite challenging, since we haven't found any proper example on the internet.

We found that Nuxt.js is using Post CSS by default, and we can change the configuration of Post CSS in the nuxt.config.js. So all we need to do is changing postcss-url plugin configuration to convert asset url from local to CloudFront endpoint. So we ended up having configuration like this.

module.exports = {
  env: {
    NODE_ENV: process.env.CLOUDFRONT_ENDPOINT,
    CLOUDFRONT_ENDPOINT: process.env.CLOUDFRONT_ENDPOINT
  },
  build: {
    publicPath:
      process.env.NODE_ENV !== "development"
        ? process.env.CLOUDFRONT_ENDPOINT
        : "/build/",
    postcss: {
      plugins: {
        "postcss-import": {},
        "postcss-url": {
          url: asset => {
            // Exclude development mode
            if (process.env.NODE_ENV === "development") {
              return asset.url;
            }

            // Exclude data-url resources
            if (asset.url.substr(0, 4) === "data") {
              return asset.url;
            }

            // Exclude non image asset
            if (asset.url.substr(0, 8) !== "/img/") {
              return asset.url;
            }

            // Return cdn url
            return process.env.CLOUDFRONT_ENDPOINT + asset.url;
          }
        }
      }
  }
}

The configuration basically says that we are still using local url on development environment, keep the asset URL if it is data-url format, skip all non images asset url and do some formatting on the URL if we found the image asset URL on the style.

Image `src` attribute HTML on `Vue` template

The final one is changing static image URL on the Vue template. In Vue files there is a section called <template> where we put our HTML code there and there always be a chance we are using static file image on it. Like this:

<template>
  <div id="header">
    <img src="/img/icon.png">
  </div>
</template>

How we changing the url on this code from the local url to CloudFront endpoint? After trying various method, we ended up to use custom plugin in our Nuxt.js application to swap this URL by using injected

module.export default ({ store }, inject) => {
  inject("image", path => {
    if (process.env.NODE_ENV === "development") {
      return `/img/${path}`;
    }

    return `${process.env.CLOUDFRONT_ENDPOINT}/img/${path}`;
  });
};
Then we attach this plugin to the nuxt.config.js.module.exports = {
  plugins: [
    { src: "~/plugins/image.js" }
  ]
}

After that we are able to use $image module on Vue component like this:

<template>
  <div id="header">
    <img :src="$image('/icon.png')">
  </div>
</template>

By using this simple plugin we are able swap the static image url based on the environment of the machine. Of course I have to find and replace all of the url, resulting a huge changes in our git commit.

Ok, so we are able to change the URL of the static and dynamic resources on or code base. Then we also need to change our CI configuration to make sure all of those files. Luckily we can find that configuration on Nuxt.js documentation. On that documentation we are able to upload all of compiled script in dist folder, and we need to change that configuration to make all our static images also uploaded to AWS S3. This changes is pretty simple, we just need to change gulp.src('./' + config.distDir + '/**'); to gulp.src(['./' + config.distDir + '/**', './static']);.

We also need to include this gulp task to our Dockerfile via npm script in order to be triggered when the CI reach build process. In our package.json we add this gulp script.

{
  ...
  "scripts": {
    "cdn:deploy": "gulp deploy"
  },
  ...
}

Then we add the npm task to Dockerfile.

# Install application
RUN npm install \
    && npm run build \
    && npm run cdn:deploy

After all of this changes, now we are able to relax because user will no see our website broken anymore during our CI deployment. Even though the process seems complete, we still haven’t yet implement a functionality to purge old files from S3 bucket. So we still need clean up old files from the bucket manually once a while. If I somehow find how to do that automatically, I will update this post to reflect that functionality.

That’s it, I hope this post can help you if you currently have problem related CDN deployment using Nuxt.js. See you in my next post.

UPDATE:

Hey, I found the way to remove the old files that generated by Nuxt.js. You can see in this post: https://code-chasm.ghost.io/how-we-manage-to-remove-unused-compiled-nuxt-js-files-on-aws-s3/

Code Chasm

Discussion about this post