Sign-up, password reset, receipts and invitations all depend on email, yet many pipelines never test them, because email feels hard to automate. With a mailbox API it is not. This article shows how to run real end-to-end email tests on every push, in a way that stays fast and does not flake.
The design in one picture
For each test, your code creates a fresh mailbox, uses its address in the flow under test, waits for the message, extracts what it needs, asserts, and deletes the mailbox. Because every test has its own address, tests cannot interfere with each other, and they can run in parallel.
Step 1: store the key as a secret
Create a key just for CI, named ci-github, in your dashboard. In your repository open Settings, Secrets and variables, Actions and add it as BOLTMAIL_KEY. If your runners use fixed addresses, add them to the key's IP allow-list. Hosted runners change addresses, so do not set one for them. Never expose the secret to pull requests from forks.
Step 2: a tiny helper library
Put the API calls in one module so that tests stay readable. This JavaScript version suits Playwright, Cypress or Jest:
// test/mail.js
const BASE = 'https://boltmail.us/api/v1';
const H = { Authorization: 'Bearer ' + process.env.BOLTMAIL_KEY, 'Content-Type': 'application/json' };
const call = (m, p, b) => fetch(BASE + p, { method: m, headers: H, body: b ? JSON.stringify(b) : undefined });
export async function newInbox(prefix = 'ci') {
const r = await call('POST', '/mailboxes', { username: `${prefix}-${Date.now()}-${Math.random().toString(36).slice(2, 7)}` });
if (r.status !== 201) throw new Error('mailbox failed: ' + await r.text());
return (await r.json()).address;
}
export async function nextMail(address, { match = () => true, seconds = 60 } = {}) {
const end = Date.now() + seconds * 1000; let after = 0;
while (Date.now() < end) {
const left = Math.max(1, Math.min(25, Math.floor((end - Date.now()) / 1000)));
const d = await (await call('GET', `/mailboxes/${address}/wait?timeout=${left}&after_id=${after}`)).json();
after = d.last_id;
const hit = d.messages.find(match);
if (hit) return (await (await call('GET', `/mailboxes/${address}/messages/${hit.id}`)).json());
}
throw new Error('no email within ' + seconds + 's');
}
export const dropInbox = address => call('DELETE', `/mailboxes/${address}`);
Notice the unique username built from a timestamp and a random suffix. It prevents address_taken conflicts when jobs run in parallel.
Step 3: write the test
// test/signup.spec.js (Playwright)
import { test, expect } from '@playwright/test';
import { newInbox, nextMail, dropInbox } from './mail.js';
test('new user can sign up and confirm email', async ({ page }) => {
const address = await newInbox('signup');
try {
await page.goto('/register');
await page.fill('[name=email]', address);
await page.fill('[name=password]', 'Correct-Horse-9');
await page.click('button[type=submit]');
const mail = await nextMail(address, { match: m => /confirm/i.test(m.subject) });
expect(mail.from.address).toContain('yourapp');
const code = mail.codes[0];
await page.fill('[name=code]', code);
await page.click('text=Verify');
await expect(page.locator('text=Welcome')).toBeVisible();
} finally {
await dropInbox(address);
}
});
The finally block deletes the mailbox even when an assertion fails, so failed runs do not leak addresses and eat into your mailbox limit.
Step 4: the workflow file
name: e2e
on: [push, pull_request]
jobs:
e2e:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version: 20, cache: npm }
- run: npm ci
- run: npx playwright install --with-deps chromium
- run: npx playwright test
env:
BOLTMAIL_KEY: ${{ secrets.BOLTMAIL_KEY }}
BASE_URL: ${{ vars.STAGING_URL }}
Run against a staging environment that sends real email through your normal provider, not against a mock. The point is to prove that the whole chain, including your provider, works.
Keeping the suite stable
- Give each test its own mailbox. Shared inboxes cause tests to read each other's mail.
- Wait with long polling, never with sleep. It is faster on good days and more tolerant on slow ones.
- Match the message you want by subject or sender, and ignore the rest.
- Set sensible timeouts: 60 seconds for email, and a job-level timeout so a hang cannot block your pipeline.
- Limit parallelism to what your plan's per-minute rate allows. A test uses roughly four to six requests.
- Retry only the infrastructure, not the assertion. Wrap API calls in backoff, as described in the article on rate limits and retries, but let a wrong subject fail loudly.
Estimating your quota
A typical email test makes one create call, one wait call, one read call and one delete call. Say five requests. A suite with 40 email tests running 30 times a day uses 40 times 5 times 30, which is 6,000 requests a day, or about 180,000 a month. That tells you whether the free plan, a mid plan or a larger one fits. Count your own numbers before choosing.
Other CI systems
Nothing here is specific to GitHub. In GitLab CI use masked variables, in Jenkins use credentials bindings, in CircleCI use contexts. The helper only needs the environment variable and outbound HTTPS.
A note on production monitoring
The same helper can run on a schedule as a canary: every ten minutes, trigger a password-reset on production for a throwaway address, wait for the email and alert if it does not arrive within a minute. Email failures are silent, and a canary tells you before your customers do.











