Skip to content

Testing Guideline

Jump to bottom
Menna Tullah Magdy Taha edited this page Jun 5, 2024 · 6 revisions

Testing Strategy Overview

This document serves as the primary reference for the current testing state of the CODE internal developer platform, detailing our implemented user stories, functional requirements, and a comprehensive test plan to ensure product reliability and maintainability.

Generally when testing we'd like to see:

  • Meets the design and development requirements
  • Correctly handles various input types
  • Performs functions within a reasonable timeframe
  • Remains usable and stable under different conditions.
  • Can be successfully installed and run in its intended environments.
  • Effectively sustains performance under high load conditions.

User Stories

This platform is designed for CODE students who are new to web development who need experience working with the cloud or deploying their applications. It streamlines the deployment of public GitHub repositories using Express or Flask frameworks, providing a deployment URL accessible during the assessment phase.

Here are the key user stories the platform addresses:

  1. As a beginner in web development, I want to deploy my Express project during development or assessment processes.
  2. As a beginner in web development, I want to deploy my Flask project during development or assessment processes.
  3. As a beginner in web development, I want to update my deployed Express project to integrate the changes done in the local development environment.
  4. As a beginner in web development, I want to update my deployed Flask project to integrate the changes done in the local development environment.
  5. As a beginner in web development, I want to access my deployed Express application through a public URL and share it for my assessment evaluation.
  6. As a beginner in web development, I want to access my deployed Flask application through a public URL and share it for my assessment evaluation.
  7. As a beginner in web development, I want to see descriptive error messages upon failure in the deployment process of an Express application.
  8. As a beginner in web development, I want to see descriptive error messages upon failure in the deployment process of a Flask application.
  9. As a beginner in web development, I want to have the option to provide environment variables and their values, which are used in the local .env file in the development environment, to ensure consistency between the application under development and the deployed one.
  10. As a beginner in web development, I want to opt out of providing environment variables when deemed unnecessary to have flexibility in the development process.

Functional Requirements

Derived from the above user stories, the system is designed to:

  1. Restrict login to users with a CODE email address.
  2. Provide a deployment trigger template accessible at any time.
  3. Display error messages for invalid URLs and email inputs.
  4. Manage new and update deployment actions.
  5. When the user adds environment variables, use them when running the container of the user application.
  6. Terminate the deployment when the user provides a GitHub repository that doesn’t include Flask or Express dependencies.
  7. Terminate the deployment process when the user chooses the update action on a repository that wasn’t deployed before.
  8. Terminate the deployment process when the user chooses the create action on a repository that was deployed before.
  9. Always be accessible.
  10. The students’ deployments need to be easily removable when the students no longer need to access them.
  11. Handle variable traffic levels at the different phases of the semester.
  12. Create a cluster for hosting student's applications.

Test Plan

The test plan is tailored to verify that all functionalities meet their intended specifications and perform under expected real-world conditions.

Deployment Destination:

  • Code-idp Repository Changes: Tests run on the existing production cluster to conserve resources, as these changes do not affect deployment logic.
  • Idp-hosted-projects Repository Changes: A dedicated testing cluster is spun up for isolated testing of deployment logic changes.

Test Cases

E2E Tests

Each test should be written twice, based on the deployment destination requirement.

  1. First-time Deployment of an Express Application
  • User logs in, uses the initiate-deployment template, provides a valid GitHub repository URL and email, and selects 'create action'.
  • Expected Outcome: User receives an email with the deployment details for the Express application.
  1. First-time Deployment of a Flask Application
  • User logs in, uses the initiate-deployment template, provides a valid GitHub repository URL and email, and selects 'create action'.
  • Expected Outcome: User receives an email with the deployment details for the Flask application.
  1. Updating an Existing Express Application
  • User logs in, uses the initiate-deployment template with a previously deployed Express application's GitHub URL and a valid email, and selects 'update action'.
  • Expected Outcome: User receives an email with the updated deployment details for the Express application.
  1. Updating an Existing Flask Application
  • User logs in, uses the initiate-deployment template with a previously deployed Flask application's GitHub URL and a valid email, and selects 'update action'.
  • Expected Outcome: User receives an email with the updated deployment details for the Flask application.

Integration Tests

  1. Cluster Creation
    • The create-gke-cluster workflow successfully creates a Google Cloud Kubernetes cluster.
  2. Application Removal
    • The delete-kubernetes-default-services workflow removes all deployed applications in the cluster.
  3. Cluster Teardown
    • The delete-gke-cluster workflow successfully tears down the Kubernetes cluster.
  4. Express Image Deployment
    • The dockerise-image workflow pushes an Express application image to the Google Artifact Registry.
  5. Flask Image Deployment
    • The dockerise-image workflow pushes a Flask application image to the Google Artifact Registry.
  6. Docker Image Deployment
    • The deploy-image workflow deploys Docker images to the Kubernetes Engine Cluster.
  7. Express Image Update
    • The deploy-image workflow updates Kubernetes configurations with a new Express Docker image for previously deployed repositories.
  8. Flask Image Update
    • The deploy-image workflow updates Kubernetes configurations with a new Flask Docker image for previously deployed repositories.
  9. Submodule Creation
    • The create-image workflow successfully creates a new submodule for a given repository.
  10. Submodule Update
    • The update-image workflow successfully updates an existing submodule to reflect changes in the corresponding repository.

Unit Tests

  1. Google OAuth Email Validations

    • Login is denied when the email address field is empty.
    • Login is denied for an invalid email address.
    • Login is denied for a valid non-CODE email address.
    • User with a valid CODE email address can log in successfully.

  2. Docker Container Integrity Checks

    • Docker container for an Express application, built using the dedicated Dockerfile, contains the correct directories and files as in the student's repository.
    • Docker container for a Flask application, built using the dedicated Dockerfile, accurately reflects the directories and files of the student's repository.

What has Been Tested

To gain confidence in receiving traffic from students using the application during the assessment phase, we decided that the system should be covered with the previous tests and tested by at least two previous students who had done the web development beginner module before. The CODE internal developer platform would be ready for release upon the confirmation of the previous students that the application meets their needs in terms of deployments and usage.

The following tests are the already implemented ones:

  1. The Google OAuth denies login of an empty email address field.
  2. The Google OAuth denies login with an invalid email address.
  3. The Google OAuth denies login of a valid None CODE email address.
  4. The Google OAuth successfully logs in a user with a valid CODE email address.

What is Missing

Current Stage

The stage we are in right now is implementing end-to-end tests, specifically the first test case:

When the User is logged in successfully, interacts with the initiate-deployment template, provides a public GitHub repository URL with a valid email address, and then chooses the create action to deploy an Express web application for the first time, they should receive an email containing their deployment.

Road Blockers

We only managed to successfully create an open pull request for setting up a a Logged-in test user account for the CODE Internal developer platform. We did not merge it to Main as the E2E test is still unfinished due to multiple issues we faced while creating a user test account:

  1. Using real user accounts in testing is not feasible. This can cause issues if Google blocks these accounts or adds extra checks to test whether they are real users.
  2. Creating a dedicated testing account would be challenging as CODE accounts enforce two-factor authentication for all emails, and two-factor authentications aren’t suitable for automated testing.
  3. Using dedicated libraries for testing Google OAuth such as cypress-social-logins, does not work with github workflows because it will trigger Google validations.

Considering those issues, we focused on completing the programmatic login and manually sending the required parameters to the OAuth callback URL for our application in the test code. We separated testing the Google Sign-in from a UI and programmatic perspective.

Then, we were presented with a challenge on how Backstage handles Single Sign-On and how to get Backstage to accept that the user is logged in upon intercepting the calls done to the Google API. Researching and understanding the internal logic of Backstage APIs consumed a lot of time and was quite a challenge till we deduced how the flow of OAuth works in Backstage and how to get Backstage to deem a user logged in.

Framework Choice

We investigated multiple frameworks, such as Playwright and Cypress, to get through the challenge of creating a logged-in user test account. Upon implementing the login using both frameworks, we noticed it wasn’t feasible to do it alone using Playwright.

It was more applicable to work with Cypress as it provided more flexibility when it came to intercepting API calls, changing the response, and setting the local storage with necessary information before accessing BackStage itself, such as setting a valid JWT token for login.

Further Research Needed & Implications of Current Test State

Tests should be independent and not rely on any third-party APIs. However, our logic relies on many calls to third-party APIs, such as deploying into a cluster or pushing a docker image to the Artifact Registry.

We needed more time to investigate the best approach for having an E2E test that is self-sufficient when our core logic relies on utilizing those APIs to achieve a successful deployment.

Without E2E tests, we cannot rely on our CODE Internal developer platform, which means risks of sudden bugs facing the students. In addition to that, scaling the system isn’t applicable as we don’t have current confidence in the load the system can handle, as load testing hasn't been performed yet. We need to ensure that the configurations are working as intended.

Having unfinished E2E tests also surprises us with the costs of running the cluster as we can’t fully correlate the reasoning behind some expenses that could increase with failures. It also impacts security; the system isn’t ready to handle actual user data or to be used reliably during assessments.

Migration from the current implementation to a new one would be challenging as we can’t map the bugs to their sources without extensive debugging.

Testing our own Platform

yarn test

Or to run the tests without the e2e running:

yarn backstage-cli repo test

If it works

git add --all
git commit -m "ADD: <What has been added>"

git push

And then at your GitHub for https://github.com/codeuniversity/code-idp click Create pull request.

Clone this wiki locally