Skip to main content

Overview

BuildPulse Flaky Tests is a CI observability platform that helps you find and fix flaky tests. We provide an end-to-end solution to detect flaky tests, give you metrics on productivity and stability, and help with root-cause analysis to fix them.

Your product is only as stable and predictable as your tests.

Product​

We provide an end-to-end solution to detect flaky tests, give you metrics on productivity and stability, and track progress over time:

  • Flaky Test Detection & History
  • Flaky Test Impact Metrics
  • Reports, Notifications, and Integrations
  • Granular Code Coverage
  • PR Bots
  • Test Quarantining
  • API Access
  • Enterprise Features that help you fix flaky tests

Getting Started​

There are 2 steps to enabling BuildPulse for your product.

  1. Exporting a JUnit XML report from your testing framework

  2. Enable BuildPulse CI integrations or the test-reporter to upload results to BuildPulse.

See the Getting Started guide for more info.

What is a flaky test?​

A flaky test is a test that exhibits both a passing and a failing result with the same code. Flaky tests usually remain tribal knowledge and are often ignored, until enough of them accumulate. This can create delays in merging code, delays in releases, bugs in production, and a general lack of confidence in the test suite.

Almost every engineering team eventually experiences flaky tests. Martin Fowler sums up the problem nicely:

"[Flaky tests] are a virulent infection that can completely ruin your entire test suite. As a result they need to be dealt with as soon as you can, before your entire deployment pipeline is compromised." - Martin Fowler

How do flaky tests occur?​

As your product grows in complexity, so too does testing complexity. This complexity can cause initial assumptions in test to break, causing test unstability and flakienss.

Our Approach​

BuildPulse determines that a tests is unstable when we observe a test failure and success for the same code. We determine if code has changed by comparing the git tree SHA against previous builds.

Security​

We only require your test results and various metadata such as git commit and tree SHAs - we currently don't require access to source code or any PII.

BuildPulse is SOC 2 Type 2 certified - please contact sales@buildpulse.io for more information.