Automating the Release Process

Problem?

Here at FloQast, one of the most important responsibilities for an SDET is handling the release process. Broadly, this means taking developer changes and pushing them through to production in a timely manner, all the while ensuring that changes function properly and do not introduce any defects. The process typically involves the following manual steps:

1. Verify your release is next by checking the release dashboard.
2. Make announcement in the appropriate channel.
3. Update ticket status from Ready for QA to QA Review.
4. Deploy each updated repository, one by one.
5. Execute the relevant regression suites, one by one.
6. Analyze regression results, one by one.
7. Manually test changes as necessary.
8. Add approval comment to ticket.
9. Update ticket status from QA Review to QA Approved.
10. Update ticket status from QA Approved to Staging Review.
11. Update ticket status from Staging Review to Staging Approved
12. Make announcement in the appropriate channel.
13. Update ITSM status from TODO to In Progress.
14. Add staging-approved labels to relevant PRs.
15. Merge relevant PRs.
16. Wait for production deployments to complete.
17. Update ITSM status from In Progress to Under Review.
18. Execute the relevant smoke suites, one by one.
19. Analyze smoke results, one by one.
20. Add approval comment to ticket.
21. Update ticket status from Staging Approved to Done.
22. Update ticket status from Under Review to Approved.
23. Update ticket status from Approved to Done.

Over the past few months, we’ve taken a look at these manual steps and looked for ways to automate this process as much as possible – with great success. We’ve managed to create a script capable of (mostly) automating the release process, around 70% of the steps outlined above. Depending on the size of the release, this can help shave off about 2 – 3 hours, as well as reduce heavy context switching. This automation should allow SDETs to focus on more important things in the future.

Solution!

The script handles releases using a queue structure, and release files push a variety of predefined actions onto the queue. Release files are defined per engineering team, giving the script some flexibility. It flows something like this:

// Initialize queue

const queue = []

// Load queue w/ actions

queue.push({
    type: ['JIRACOMMENT', 'approve changes on production'],
    payload: {
        // ...
    }
})

queue.push({
    type: ['JIRATRANSITION', 'move jira ticket from [STAGING APPROVED] -> [DONE]'],
    payload: {
        // ...
    }
})

// Execute action(s)

while (!_.isEmpty(queue)) {
    // Pull action off the queue and execute it
}

These individual release files only need to be defined once, so SDETs only have to focus on updating the relevant config files during their day-to-day release duties. There are currently two levels of config files: one for common config required by all releases, and one that individual teams can use to add custom config values. Both of these config files are merged into a single object that gets passed into the release file. This process looks something like this:

// Common config

module.exports = {
    POD: process.env.POD,
    JIRA_USERNAME: process.env.JIRA_USERNAME,
    JIRA_API_TOKEN: process.env.JIRA_API_TOKEN,
    JENKINS_USERNAME: process.env.JENKINS_USERNAME,
    JENKINS_API_TOKEN: process.env.JENKINS_API_TOKEN,
    JIRA: 'https://floqast.atlassian.net/browse/FQ-32950',
    ITSM: 'https://floqast.atlassian.net/browse/ITSM-12819',
    COMPONENTS: [
        'www-close',
        'gl_sap'
    ],
    BRANCH: 'FQ-32950-setup-lambda-placeholder-routes'
}


// Team config

module.exports = {
    BUSINESS_CENTRAL_API_REGRESSION: false,
    GL_SDK_REGRESSION: false,
    CLOUD_CONNECT_REGRESSION: false,
    FILE_PARSING_UI_REGRESSION: false,
    MAIN_REGRESSION: true,
    SMOKE_REGRESSION: true
}


// Merging the two configs

const store = {
    ...config.COMMON, 
    ...config[`${_.toUpper(process.env.POD)}`]
}

After defining a release file and filling out the relevant config files, an SDET executes the following command:

foo@bar:~$ make release

The output will ultimately look very similar to this:

Next Steps

Since we’ve decided to take an opt-in approach to sharing this script, we’re slowly spreading usage throughout FloQast. We’re hoping to gain valuable feedback from the small group of teams that are currently part of the pilot process before getting the rest of QE onboard.

Daniel Vargas

Daniel is a Senior SDET at FloQast with a passion for automating everything! Outside of work, he enjoys spending time with his family and playing retro video games.