Running Cypress Tests on AWS CodeBuild

Cypress is a relatively new web testing tool that is easier to use than Selenium, and it’s gaining in popularity. But using it in a continuous integration environment like AWS CodeBuild requires some additional steps compared to running it directly on your own computer.

This blog post contains helpful information to configure CodeBuild on AWS to run Cypress.

Requirements for running Cypress on CodeBuild

You’ll need several things to run Cypress on AWS:

1. An AWS Account with permissions to create S3 Buckets, run CloudFormation, and manage AWS CodeBuild jobs.
2. A working local Cypress testing project

Things not covered here:

• How to reset the environment before testing
• Security via Cognito or other security systems
• Additional security such as VPC configurations

Proper permissions for running these samples

An AWS administrator will need to grant access to the following permissions:

• AWSCodeBuildAdminAccess
• AmazonS3FullAccess
• AWSCloudFormationFullAccess
• CloudWatchLogsFullAccess

Be careful to work with an AWS account with credits. Whatever you create and whatever bills you generate are your responsibility.

A sample application

For this post, you’ll deploy a simple Angular JavaScript application to a public S3 bucket for testing.
Here’s the beginning of the CloudFormation script that sets up the S3 bucket, named based on the stack name you pass when you create it.

AWSTemplateFormatVersion:                           "2010-09-09"
Description:                                        "Self hosted AWS application"

# See example at https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/quickref-s3.html
# Create an S3 bucket and using it to host a website written in Angular
Resources:

  S3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      # Key setting: without this, the bucket cannot be accessed via an URL
      AccessControl: PublicRead
      # Give the bucket a predictable name based on the stack name passed 
      # when creating the CloudFormation stack
      BucketName: !Sub "${AWS::StackName}-hosting-bucket"
      WebsiteConfiguration:
        IndexDocument: index.html
        ErrorDocument: error.html
    DeletionPolicy: Retain

In order to expose the bucket for web access, you need to assign a bucket policy:

  BucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      PolicyDocument:
        Id: MyPolicy
        Version: 2012-10-17
        Statement:
          - Sid: PublicReadForGetBucketObjects
            Effect: Allow
            Principal: '*'
            Action: 's3:GetObject'
            Resource: !Sub "arn:aws:s3:::${S3Bucket}/*"
      Bucket: !Ref S3Bucket

This policy allows all principals (everyone) to get objects from the bucket, allowing read access to the
objects inside of it. Remember to remove this stack when you’re done with this example or you’ll pay for
additional usage.

Next, you need to expose the URL and bucket name:

Outputs:
  WebsiteURL:
    Value:                      !GetAtt "S3Bucket.WebsiteURL"
    Description: URL for website hosted on S3
  S3BucketName:
    Value: !Ref S3Bucket

A simple shell script installs this stack (you provide a stack name as the argument):

#!/bin/bash

# This script deploys the Angular app to an S3 bucket for hosting

aws cloudformation deploy
  --stack-name ${1} \
  --capabilities CAPABILITY_IAM --template-file \
  ./angular-app/cloudformation.yml \
  || { echo 'cloudformation deploy failed'; exit -1; }
WEBSITE_URL=`aws cloudformation describe-stacks \
               --stack-name ${1} --output text \
               --query "Stacks[*].Outputs[?OutputKey=='WebsiteURL'].OutputValue"`
BUCKET_NAME=`aws cloudformation describe-stacks \
               --stack-name ${1} --output text \
		   	   --query "Stacks[*].Outputs[?OutputKey=='S3BucketName'].OutputValue"`
pushd ./angular-app
npm install || { echo 'npm install failed'; exit -2; }
npm run build || { echo 'build failed.'; exit -3; }
aws s3 cp dist/angular-app/ \
    s3://${BUCKET_NAME} \
    --recursive || { echo 'cp failed'; exit -4; }
echo "The website is now available at ${WEBSITE_URL}"
echo "The S3 bucket name is ${BUCKET_NAME}"
popd

Now your webapp can be deployed to a stack with ./setup-angular-stack.sh my-stack.

What are we deploying, anyway?

Here’s a super simple Angular application component that implements a little todo list. Well, at least one action of the todo list, adding an entry via a form. Because we have to test something…

The Component

import { Component, OnInit } from '@angular/core';
import { ListItem } from '../support/list-item';
import { FormBuilder, FormGroup } from '@angular/forms';
import { ItemType } from '../support/item-type.enum';

@Component({
  selector: 'app-shopping-list',
  templateUrl: './shopping-list.component.html',
  styleUrls: ['./shopping-list.component.scss']
})
export class ShoppingListComponent implements OnInit {
  shoppingList: Array<ListItem> = [];
  shoppingItemForm: FormGroup;
   itemTypes = [ItemType.BAKERY, ItemType.DRINK, ItemType.FOOD, ItemType.PHARMACY,
                ItemType.SALAD_BAR, ItemType.SNACKS, ItemType.TOILETRIES];
  constructor(private builder: FormBuilder) { }

  ngOnInit(): void {
     this.shoppingItemForm = this.builder.group({
       'purchased': [false, [], []],
       'name': ['', [], []],
       'category': ['', [], []]
     }, []);
  }

  addItem() {
    const formData = this.shoppingItemForm.value as ListItem;
    this.shoppingList.push(formData);
    this.shoppingItemForm.reset({purchased: false});
  }
}

The component’s view

<h3>Add an Item</h3>
<form [formGroup]=shoppingItemForm
     (ngSubmit)="addItem()">
    <div class="form-group">
        <label for="name">Item Name</label>
        <input id="name" class="form-control"
                         type="text"
                         formControlName="name">
    </div>
    <div class="form-group">
        <label for="category">Category</label>
        <select id="category" class="form-control" formControlName="category">
            <option [value]="itemTypes[0]">{{ itemTypes[0] }}</option>
            <option [value]="itemTypes[1]">{{ itemTypes[1] }}</option>
            <option [value]="itemTypes[2]">{{ itemTypes[2] }}</option>
            <option [value]="itemTypes[3]">{{ itemTypes[3] }}</option>
            <option [value]="itemTypes[4]">{{ itemTypes[4] }}</option>
            <option [value]="itemTypes[5]">{{ itemTypes[5] }}</option>
        </select>
    </div>
    <div class="button-group">
      <button class="btn-large btn-primary">Add...</button>
    </div>
</form>
<div *ngIf="shoppingList && shoppingList.length > 0">
    <h3>Shopping List</h3>
    <table class="table table-bordered table-striped">
        <thead>
            <th>X</th>
            <th>Name</th>
            <th>Category</th>
        </thead>
        <tbody>
            <tr *ngFor="let item of shoppingList">
                <td><input type="checkbox"></td>
                <td>{{ item.name }} </td>
                <td>{{ item.category }}</td>
            </tr>
        </tbody>
    </table>
</div>

Testing with Cypress

Cypress is a web testing tool that Chariot has begun to use instead of Selenium for a number of web applications. It is relatively easy to set up, has a good developer tool with live reloading that makes it easy for developers to iterate with, and it can run headless in the cloud.

A simple test against the app above would look like this:

describe('Adding a list item', () => {
    beforeEach(() => {
        // the visit is based on a "baseUrl" config
        // setting that establishes the root 
        // website address
        cy.visit('/');
    });
    it('adds an item', () => {
        cy.get('input#name')
          .type('Soap Flakes')
          .should('have.value', 'Soap Flakes')

        cy.contains('Add...').click();

        cy.get('table').should('contain', 'Soap Flakes');

        cy.get('input#name')
        .should('have.value', '');
    });
});

Cypress has a relatively easy to manage configuration system. Normally, with local development,
the cypress.json configuration file can point to your local web server (so you
can iterate on the application and re-run tests as you change the code).

For that purpose, the local repository points to the local Angular stack:

{
    "baseUrl": "http://localhost:4200",
    "viewportWidth": 1024,
    "viewportHieght": 800
	...
}

To test the app locally, you fire up the app in one terminal, and Cypress in another.

Creating the Cypress AWS CodeBuild project via CloudFormation

There are a variety of ways to host Cypress tests. CircleCI has its own Cypress “ORB”,
there is an official Cypress Docker container, and you can
also install and run Cypress directly from a test platform supporting NodeJS and Chromium.
It is that method that we are going to use to run Cypress in an AWS CodeBuild project.

The CloudFormation configuration for the CodeBuild project has several major parts. It begins with two required parameters, GitHubProjectUrl and CypressBaseUrl,
to fetch the project source code and point to the right website for testing:

AWSTemplateFormatVersion:               "2010-09-09"
Description:                            "Cypress Test"

Parameters:
  GitHubProjectUrl:
    Description:                        "The GitHub Project URL"
    Type:                               "String"

  CypressBaseUrl:
    Description:                        "The base URL for the application under test"
    Type:                               "String"

You need to provision an S3 Bucket to hold artifacts/reports (by default, it will
be a generated S3 bucket, so this gives you more control over where the assets will
live):

Resources:

  CypressReportBucket:
    Type:                               AWS::S3::Bucket
    Properties:
      BucketName:                       !Sub "${AWS::StackName}-cypress-reports"

Next, you need to configure an IAM Role to run CodeBuild projects. This role needs to
have access to the newly created reporting bucket, have access to the EC2 AMI registry
to launch the CodeBuild VM, write to CloudWatch logs, and have the ability to read
CloudFormation stack settings to get its configuration.

  CypressRole:
    Type:                               "AWS::IAM::Role"
    Properties:
      RoleName:                         !Sub "${AWS::StackName}-CypressRole"
      AssumeRolePolicyDocument:
        Version:                        "2012-10-17"
        Statement:
          - Effect:                     "Allow"
            Principal:
              Service:
                - "codebuild.amazonaws.com"
            Action:
              - "sts:AssumeRole"
      ManagedPolicyArns:
        - "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPowerUser"
        - "arn:aws:iam::aws:policy/AWSCloudFormationReadOnlyAccess"
      Policies:
        - # for writing to CloudWatch
          PolicyName:                   CodeBuildLoggingPolicy
          PolicyDocument:
            Version:                    "2012-10-17"
            Statement:
              - Effect:                 "Allow"
                Action:
                  - "logs:CreateLogGroup"
                  - "logs:CreateLogStream"
                  - "logs:PutLogEvents"
                Resource:                "*"
        - # For uploading artifacts
          PolicyName:                   CodeBuildS3ArtifactBucketAccess
          PolicyDocument:
            Version:                  "2012-10-17"
            Statement:
              - # Static Hosting Bucket upload...
                Effect:                 "Allow"
                Action:
                  - "s3:*"
                Resource:
                  - !Sub "arn:aws:s3:::${CypressReportBucket}"
                  - !Sub "arn:aws:s3:::${CypressReportBucket}/*"

Next, you define the Cypress runner as an AWS CodeBuild Build Project. We’ll break
this down piece by piece, as it is a long configuration.

First, the overall runner settings:

  CypressRunner:
    Type: AWS::CodeBuild::Project
    Properties:
      # Define the codebuild project with the stack name
      Name:                             !Sub "${AWS::StackName}-cypress-ui"
      Description:                      Test the application via Cypress
      # Tie in the codebuild role
      ServiceRole:                      !GetAtt CypressRole.Arn
      # set a reasonable time here
      TimeoutInMinutes:                 5
      # for now, we're storing the report as an artifact
      # the actual CodeBuild reports are trickier to configure
      # and perhaps are a story for another day
      Artifacts:
        Type:                           S3
        NamespaceType:                  BUILD_ID
        Name:                           "reports.zip"
        # Refer to the S3 bucket we just defined
        Location:                       !Ref CypressReportBucket
        Packaging:                      ZIP
      # Remove this if you want to avoid storing build logs in CloudWatch
      LogsConfig:
        CloudWatchLogs:
          GroupName:                    !Sub "/codebuild/${AWS::StackName}-cypress-logs"
          Status:                       ENABLED
          StreamName:                   "ci-log"

Continue with defining the environment of the CodeBuild job. This passes the
input S3 bucket hosting URL so you can override it in the Cypress build execution.

      Environment:
        Type:                           LINUX_CONTAINER
        PrivilegedMode:                 true
        ComputeType:                    BUILD_GENERAL1_SMALL
        Image:                          aws/codebuild/amazonlinux2-x86_64-standard:3.0
        EnvironmentVariables:
          - Name:                       CYPRESS_BASE_URL
            Value:                      !Ref CypressBaseUrl

Now you have to configure a source code repository to download the test artifacts.
In this example, we#8217;ll use GitHub as our source, passed as the CloudFormation Stack input parameter
GitHubProjectUrl above. Once you configure AWS to see your GitHub projects,
you can execute this CodeBuild project job which will fetch the source from GitHub:

      Source:
        Auth:
          Type:                         OAUTH
        Location:                       !Ref GitHubProjectUrl
        Type:                           GITHUB
        GitCloneDepth:                  1
 

Cypress Execution Options in CodeBuild

To run the build, Cypress must be installed in a CodeBuild virtual machine. We have two options here:
running in the machine natively, and running as a Docker container. This post will focus on installing
CodeBuild directly in NodeJS in our VM.

Finally, let’s review the build itself. This is a CodeBuild BuildSpec file, embedded in
the CloudFormation definition. It will be uploaded to an S3 bucket backing the CodeBuild code.
The key challenges of this build were resolved with some good ole’ Stack Overflowment:

• running Chromium instead of Chrome (as it is provided as part of the CodeBuild amazon-linux instance),
• running Chromium headless (without a graphics card) to avoid installing a desktop environment,
• properly generating the reports (we used Mochawesome as the report generator) and uploading them as build artifacts (not 100% the CodeBuild way, but the CodeBuild reporting engine didn’t work with our reports and this is good enough for our purposes),
• and passing the baseUrl override into the Cypress launch command.

        BuildSpec: |
          # important:  0.2 enables some very useful features
          version: 0.2
          phases:
            install:
              runtime-versions:
                nodejs: 12
            # Pre-build - set up the Cypress runtime
            pre_build:
              commands:
                - cd cypress-tests
                - npm install
            build:
              commands:
                - export CONFIG="baseUrl=$CYPRESS_BASE_URL"
                - echo $CONFIG
                - # NO_COLOR=1 disables ANSI special chars so you can read the output
		- # Also: --headless is essential here. We MUST run headless
		- # in order to avoid installing an X11 desktop and virtual X graphics
		- # context...
                - NO_COLOR=1 ./node_modules/.bin/cypress run \
                             --browser chromium \
                             --headless \
                             --config "$CONFIG"
              finally:
                - # You can grab this from the Cypress docs on running Cypress reports
                - npx mochawesome-merge "cypress/reports/separate-reports/*.json" > mochawesome.json
                - # yes, "marge"
                - npx marge mochawesome.json
                - # this one is stupid but somehow I can't get codebuild to copy multiple top level paths
                - # so I just moved the report dir into place
                - mv mochawesome-report cypress/
          # these are available in the Build Details tab as the reports.zip archive
          artifacts:
            files:
              - 'reports/**/*'
              - 'screenshots/**/*'
              - 'videos/**/*'
              - 'mochawesome-report/**/*'
            base-directory: 'cypress-tests/cypress'

My advice for those who dare: expect to tweak the options until you get it right. Try running the build
script locally with the CYPRESS_BASE_URL set to http://localhost:4200 until it works, then deploy it to CodeBuild.

CodeBuild can be tricky, so we have put this project up on a GitHub repo for you to try. You can get
all of the code above at https://github.com/chariotsolutions/aws-codebuild-cypress-example.