Skip to content

Real Apple CI Needs a Mac Runner and an iPhone

4 min read
GitHub ActionsmacOSiOSTestingFoil

Apple apps have a testing problem that web apps mostly do not: eventually, the thing you need to prove is attached to a real machine.

For Foil for macOS, that means the menu bar, microphone permissions, local audio routing, keychain state, the clipboard, and the GUI session. For Foil iOS, it means a custom keyboard extension, App Group state, TestFlight install behavior, secure-field rules, and a physical iPhone.

So the CI setup is not one magic workflow. It is a claim boundary.

The macOS shape

For app-impacting PRs and merge groups, the hosted macOS CI gates run in GitHub Actions: build with warnings as errors, unit tests with coverage summaries/reports when the result bundle is available, focused UI smoke, and xcresult summaries with failure/cancellation artifacts.

Then there are the jobs that should not pretend to be portable. The Local macOS E2E and Live Microphone QA workflows target a trusted self-hosted Apple Silicon Mac with explicit labels:

self-hosted, macOS, ARM64, trusted-macos-e2e, e2e, gui, browser.

Those jobs include host proof, health checks, workspace cleanup, stale-process cleanup, app builds, live transcription or microphone QA, artifact collection where configured, and cleanup after. The point is not just "run it on my Mac." The point is to make the runner itself part of the test contract.

Diagram of Foil macOS GitHub Actions routing hosted CI jobs and live GUI proof jobs to a self-hosted Mac runner

The distinction matters. Hosted macOS CI can prove that the project builds and the deterministic test suite is healthy. A local GUI runner can exercise selected live transcription paths and scripted microphone/UI smoke behavior against a real logged-in macOS environment.

The iOS shape

Foil iOS is stricter in a different way. It should grow a small hosted GitHub Actions lane for simulator sanity, but the current repo still keeps that as a local script plus physical-device proof helpers, docs, and receipt trails rather than a main-branch .github/workflows directory. The important boundary is that a hosted simulator badge would not replace the physical iPhone receipts.

The automated layer covers the Swift pieces that can be made deterministic: keyboard bridge behavior, exact-once transcript insertion, provider failure presentation, transcript quality filtering, and transcription client request/failure behavior. The simulator sanity script runs xcodebuild test and an unsigned generic iOS build, while explicitly not claiming physical keyboard insertion or host-app behavior.

But the interesting iOS claims are physical:

  • Can the preview phone follow the Foil-specific TestFlight Install -> Replace -> Open flow?
  • Did the custom keyboard check in through shared App Group state?
  • Does Insert latest insert once, then clear state?
  • Does a sterile Safari secure/password fixture keep Foil Keyboard and Insert latest absent?
  • Can the evidence be stored without leaking private transcript text?

That is where the physical harness and WDA evidence scripts come in. They are designed to produce sanitized JSON receipts: hashes, phases, accessibility checks, text counts, App Group summaries. The final audit then says what is true, what is not true yet, and what remains blocked.

Diagram of Foil iOS proof flow from unit tests and simulator sanity to TestFlight rehearsal, physical iPhone automation, sanitized receipts, and final audit

My favorite part is the restraint. In the current Foil iOS README, the safe claim is narrow: build 0.1.0 (13) has receipts for route-first readiness, Foil-specific TestFlight install/replace/open, keyboard-health refresh, and exactly-once Insert latest into a sterile Safari normal-field fixture on iPhone-preview. It explicitly does not claim arbitrary app support.

That is the whole philosophy.

The pattern

For Apple apps, I want the pipeline to answer one question:

What can this environment actually prove?

Hosted macOS can prove compile health, unit behavior, and repeatable smoke tests. A self-hosted Mac can provide evidence for selected GUI and microphone behavior. A simulator can prove fast iOS regressions. Preview iPhone receipts can prove named keyboard-extension paths. A TestFlight rehearsal can provide evidence for release plumbing. A final audit can prove that the words around the build did not outrun the evidence.

The practical rule is simple: let each layer make the strongest claim it can honestly support, then stop there.

That is how the setup stays useful. Not because every test runs everywhere, but because every claim has a runner, receipt, or audit behind it.