Test Automation with agent-device (Part 3)
---
title: "Test Automation with agent-device (Part 3)"
published: true
description: "Save test flows as agent-device .ad scripts and run them again."
tags: claude, ai, flutter, testing
canonical_url: krenin88.github.io
---
This post covers saving test flows as agent-device .ad scripts and running them again, using a Flutter app. It goes through writing an .ad file, getting the app into the right state before a replay, reading the report when a replay stops and fixing it, and the token cost of agent-driven QA. It is based on agent-device 0.21.31 (2026-10-25).
Installation and the basic loop are in Part 1, and real-device setup is in Part 2.
What an .ad file is
When an agent checks an app, it reads the screen, picks the next thing to tap, and reads the result, over and over. Exploring the same flow this way every time keeps costing time and tokens. An .ad file pins down the result of that exploration as a list of commands, and replay runs the list from top to bottom.
Each line has the form <command> [positional...] [flag...], the same as an agent-device command in the terminal. Quoting follows shell rules too, so a selector containing spaces is wrapped like 'label="..."'. Below is the start of a flow that opens the app fresh, taps a button on the first screen, types into an input, and checks the result screen.
# First-run flow — the app ID comes in via -e APP=
open ${APP} --relaunch
wait 'label="<first-screen button>"' 30000
press 'label="<first-screen button>"'
fill 'id="find-input"' "<text to type>"
press 'label="<search button>" role=button'
wait 'label="<result title>"' 60000
wait text "<text the result should show>" 10000
back
close
A few notes:
- # comments work. The docs do not say so, so the first line was a comment in a test replay, and it passed.
- Do not hard-code the app ID. Leave it as ${APP} and pass -e APP=<bundle id> at replay time, and the same file works for builds with different IDs, such as a dev build and a release build.
- The number after wait is a timeout in milliseconds. Give a long one to steps that wait on the network or on model inference.
- Device-selection flags stay out of the file. --platform, --serial, --device, and --settle are not part of a script step; they go on the replay command.
There are two ways to make an .ad file. You can start recording with open <bundle id> --relaunch --save-script=<file>.ad, go through the flow, and save with session save-script, or you can write it by hand. The 24-step flow in this post was written by hand, based on a flow the agent had already passed manually.
Getting to a "first launch" state before replaying
replay only runs the commands written down; it does not reset the app. If the flow starts by checking the first-launch consent dialog, an app that still remembers an earlier consent stops right at step 2. So a separate runner script deletes and reinstalls the app before each replay.
# Android emulator
adb -s <serial> uninstall <bundle id>
adb -s <serial> install -r <apk path>
adb -s <serial> shell cmd locale set-app-locales <bundle id> --locales en-US
agent-device replay <file>.ad --platform android --serial <serial> -e APP=<bundle id> --timeout 240000
# iOS simulator
xcrun simctl uninstall <UDID> <bundle id>
xcrun simctl install <UDID> <Runner.app path>
agent-device replay <file>.ad --platform ios --udid <UDID> -e APP=<bundle id> --timeout 300000
On Android, set-app-locales opens just this app in English while leaving the device language alone. On the iOS simulator no per-app language command was used, so the app follows the simulator's language setting. replay has a default work budget of 60 seconds, so raise it with --timeout for longer flows.
Deleting the app also deletes its data. So the runner script accepts only the designated test emulators (checked by name with adb -s <serial> emu avd name) and simulator UDIDs, and stops with exit code 2 if it gets any other device or an app ID outside your own. An emulator serial such as emulator-5554 is assigned in boot order and does not tell you which AVD it is, which is why the check uses the name.
With that in place, the 24-step flow finished with Replayed 24 steps on both runs on each platform: 8.8 and 9.4 seconds on the Android emulator, 13.9 and 14.0 seconds on the iOS simulator.
When a replay stops
When a replay gets stuck partway, it fails with REPLAY_DIVERGENCE. The report includes the step number and its line, the actionable elements on screen at that moment, and selector suggestions for that screen. So you do not have to start over after a fix, it also tells you how to resume with replay --from <step> --plan-digest <value>. When no automatic repair is possible, it says Repair hint: manual.
A first-launch system dialog. The flow that passed on Android stopped at step 2 the first time it ran on the iOS simulator.
Error (REPLAY_DIVERGENCE): Replay failed at step 2 (wait "label=\"<first-screen button>\"" 30000): wait timed out for selector: ...
Current surface: Allow "<app name>" to track your activity across other companies' apps ...
The report's Current surface showed the App Tracking Transparency (ATT) dialog. It is a system dialog that appears on first launch only on iOS, so the flow written against Android had no step for it. After closing it with agent-device alert dismiss and replaying again, the run got past this step. The runner script does not handle it, so on builds that show this dialog, close it before replaying.
Input role names that differ by platform. Next, the replay stopped at the fill in step 5. The first selector was role=text-field, which matched on Android, but the iOS simulator exposed the same input as an other element with a label. To find one selector that works on both platforms, these were tried in order:
1. editable=true — passed on Android, Selector did not match on iOS
2. role=textfield || role=text-field — || is the "or" syntax that appeared in the report's selector suggestions. It still did not match on iOS.
3. Adding Semantics(identifier:) to the Flutter widget — passed on both platforms
The last fix is one line of app code: identifier: 'find-input' added to the existing Semantics around the input, with no visible change on screen. The selector becomes id="find-input". As noted in Part 1, a Flutter Key does not show up in the accessibility tree, so adding an identifier is the reliable way to run the same script on both platforms.
Token cost and the rules for cutting it
Token cost was the reason for using .ad in the first place. Most of the cost of agent-driven QA came not from output but from cache reads. A cache read is the earlier conversation the agent re-reads on every response. The first real-device check of two apps took about 40 minutes and came to 121 responses, 77,985 output tokens, and 30,225,404 cache-read tokens, roughly 250,000 tokens re-read per response.
The QA agent was already on Sonnet, so the model was not the cause. In the same stretch, 11 screenshots were read into the conversation, and as the context grew, every response re-read more. So five rules went into the QA agent's instructions:
1. Save a flow as .ad the first time it passes, and use replay from then on. When it gets stuck, fix only that step using the divergence report.
2. Act with --settle and look only at the returned diff. Do not read the full screen tree.
3. Save screenshots to files only, and read at most one per item into the conversation.
4. End the session per item: one app, one thing to check. Past 40 responses, it is time to stop.
5. Read logs and build output with grep or tail, only the lines you need.
The batch after the rules (three emulator checks, an iPhone attempt, an Android real device, about 30 minutes) came to 105 responses, 44,113 output tokens, and 19,171,162 cache-read tokens, about 180,000 cache-read tokens per response. The items checked were different, so the two sets of numbers cannot be compared directly, and the 40-response limit was not met. Tokens after adopting .ad have not been measured separately yet, and neither replay on a real device nor running it in CI has been tried.
References
- agent-device repository: <callstack/agent-device>
- agent-device agent setup docs: <oss.callstack.com>
- Flutter SemanticsProperties.identifier: <api.flutter.dev>
Fallback for federated remotes: the copy in the binary and the net in the session
> 📚 React Native Module Federation series — read it in full on warrendeleon.com, where new parts land first.
Post 15 ended its break-it section on its own limit: "None of this gives the binary anywhere else to load from: a dead tab is honest, and it is not a working app." This post gives the binary somewhere else. Every release build now carries a copy of each remote, and the app runs from those copies when the content delivery network (CDN) cannot be reached, or when one remote fails to load while the other keeps loading from it.
Post 1 said who builds that: "Module Federation gives you none of the offline story by itself: shipping a copy of each remote inside the binary, so the reviewed app works on its own with no network, is an architecture you build." It also set the requirement for a remote that will not load: "the app has to degrade to something safe instead of showing a blank screen." Post 15's build meets the requirement for a remote that will not load, and stops there. With the CDN out of reach it launches unresolved, with an error state in either tab. When one remote fails, its tab shows the error state, and Try again never brings the tab back.
The first half builds the copy: a build step that stages it, a phase on each platform that puts it in the binary, and a new launch mode, bundled, that runs every remote from it. The second half builds two nets for a CDN launch. One catches a manifest the CDN cannot serve and drops that one remote to its copy; the other sits around each tab and catches a load that fails later. A Try again that actually retries comes with the second net.
One rule shapes the copy: it is the version already known to run on this binary, frozen on the day the binary was built. It is the guaranteed minimum: new versions still reach installed apps through the CDN, and the host uses the copy only when the CDN cannot deliver a version that loads.
Start from post 15's finished state, the tag post-15-cdn-flip; this post finishes at post-16-fallbacks. As before, you type the configuration changes and the smaller edits, and copy the rest from the end tag: the build tool, the host's code and tests, the native files and one design-system test.
> 📊 Diagram: view it on warrendeleon.com
The launch decides the mode once, before anything federated loads. Using the CDN's versions means a usable map arrived and its versions registered; when they cannot be used, the launch registers the copies instead. It ends unresolved only when that fails too. The two nets act inside a CDN launch, one remote at a time.
Put a copy of each remote in the binary
A copy is chosen by the same map that picks versions from the CDN. The version a binary carries is the one its own app version's map names, because that is the version known to run on that binary: for the 2.0.0 build here, list 1.2.0 and party 1.0.0.
The copy holds that version's signed bundles and its manifest, byte for byte as the CDN serves them. The bytes stay as they are because every bundle is still verified against the signing key when it loads, and a change to its code fails the check.
tools/build-cdn.mjs already builds every version into cdn-root/, and it now stages the copies as well. Fetch the end tag once, since every copy command in this post takes its files from it, and take the tool first:
npx degit@3.8.0 --force warrendeleon/react-native-module-federation#post-16-fallbacks /tmp/pokedex-ref-16
cp /tmp/pokedex-ref-16/tools/build-cdn.mjs tools/
The new stage runs after the CDN tree is built. For each platform it takes every version the map names and copies its bundles and manifest into embed-root/<platform>/<remote>/<version>/:
for (const [remote, version] of Object.entries(embeddedVersions)) {
const published = join(cdnDir, remote, version);
const embedded = join(embedDir, remote, version);
mkdirSync(embedded, { recursive: true });
// Every script of the version sits at the top of its directory with its manifest, and the copy
// is all of them, byte for byte: the container, its chunks, index.bundle, which the manifest
// names among the shared modules' assets, and mf-manifest.json, which the federation runtime
// reads from the disk the way it reads it from the CDN. The assets/ folder beside them holds
// images the host's shared copies of those libraries already carry.
for (const file of readdirSync(published)) {
if (file.endsWith('.bundle') || file === 'mf-manifest.json') {
copyFileSync(join(published, file), join(embedded, file));
}
}
bundledVersions[platform][remote] = version;
The copies keep the <remote>/<version>/ directories instead of sharing one folder, because two remotes can ship vendor chunks with the same file name. In one folder, the second would overwrite the first.
The build tool also records the copied versions in apps/host/src/shell/embedded-versions.ts, a generated file the host compiles in, so the host knows which copies it carries before it reads anything from the disk.
embed-root/ is build output, like cdn-root/, so add it to .gitignore:
# The copies build-cdn stages for the native embed phases.
embed-root/
Run the key generator first, then the tool for both platforms. The generator keeps any keys the checkout already has, and creates them on a fresh checkout of the tag, which has none because they never enter Git. After the CDN tree, the tool reports the copies and the generated file:
node tools/gen-signing-keys.mjs && node tools/build-cdn.mjs
embedded -> embed-root/ios/listApp/1.2.0
embedded -> embed-root/ios/partyApp/1.0.0
embedded -> embed-root/android/listApp/1.2.0
embedded -> embed-root/android/partyApp/1.0.0
wrote -> apps/host/src/shell/embedded-versions.ts
The generated file records the version of each copy, per platform:
export const BUNDLED_VERSIONS: Record<string, Record<string, string>> = {
ios: { listApp: '1.2.0', partyApp: '1.0.0' },
android: { listApp: '1.2.0', partyApp: '1.0.0' },
};
The build tool stages the copy for the app version in MF_APP_VERSION, or for the newest app version with a map when that is unset, so a build for another app version runs the tool with the same variable first.
iOS: the last build phase
On iOS the copy goes inside the .app, next to main.jsbundle. A script does the copying, and a Run Script phase at the end of the Host target runs it. Copy the script from the end tag:
mkdir -p apps/host/scripts && cp /tmp/pokedex-ref-16/apps/host/scripts/embed-remotes-ios.sh apps/host/scripts/
#!/usr/bin/env bash
# --- The copy in the binary, iOS's half. The last Run Script phase of the Host target copies each
# remote's embedded version from embed-root/ios into the app, at cdn/ios/<remote>/<version>/ next
# to main.jsbundle, where the host reads it through absolute file:// URLs: the signed bundles and
# the version's mf-manifest.json.
#
# The <remote>/<version>/ directories are kept rather than flattened: two remotes can ship vendor
# chunks with the same file name, and a flat copy would let one overwrite the other.
#
# It runs in every configuration. A Debug build loads the host's own bundle over http, so it cannot
# read the copy, and there the copy only costs the time it takes. Its remotes come from the dev
# servers when no CDN is configured, or from the CDN when one is.
#
# With embed-root missing it removes any copy an earlier build left behind, prints a warning and
# exits 0, and the build succeeds with nothing embedded. That is the trap: run
# `node tools/build-cdn.mjs` before a release build. ---
set -euo pipefail
REPO_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)
SOURCE="$REPO_DIR/embed-root/ios"
: "${CONFIGURATION_BUILD_DIR:?run this from an Xcode build phase}"
: "${UNLOCALIZED_RESOURCES_FOLDER_PATH:?run this from an Xcode build phase}"
DEST="$CONFIGURATION_BUILD_DIR/$UNLOCALIZED_RESOURCES_FOLDER_PATH/cdn/ios"
if [ ! -d "$SOURCE" ]; then
rm -rf "$DEST"
rmdir "$(dirname "$DEST")" 2>/dev/null || true
echo "warning: $SOURCE does not exist, so no remotes are embedded. Run 'node tools/build-cdn.mjs' before a release build."
exit 0
fi
mkdir -p "$DEST"
# --delete keeps the app to exactly what embed-root holds, so a version an earlier build embedded
# does not linger. The bytes are copied as they are: each bundle is signed over them.
rsync -a --delete --include='*/' --include='*.bundle' --include='mf-manifest.json' --exclude='*' "$SOURCE/" "$DEST/"
echo "embedded $(find "$DEST" -name '*.bundle' | wc -l | tr -d ' ') remote bundles and $(find "$DEST" -name 'mf-manifest.json' | wc -l | tr -d ' ') manifests into $DEST"
The phase is one change to the project file. In Xcode it is a new Run Script phase on the Host target, named "Embed federated remotes (offline fallback)", running "$SRCROOT/../scripts/embed-remotes-ios.sh", placed after every other phase, with "Based on dependency analysis" unchecked. It goes last so that the .app it writes into is already assembled. With the box unchecked it runs on every build, which a phase that declares no outputs does anyway: Xcode skips a script phase only when it has outputs to check. Declaring the copied files under the .app's cdn/ios as outputs, with embed-root/'s files as inputs, is the alternative. The end tag's project file carries the phase with the box unchecked, so copying it makes the same change:
cp /tmp/pokedex-ref-16/apps/host/ios/Host.xcodeproj/project.pbxproj apps/host/ios/Host.xcodeproj/
Android: a Gradle task and a native module
On Android the copy goes into the assets of the APK, the Android package an app is installed from. At the end of apps/host/android/app/build.gradle, after post 15's changes, add:
// --- The copy in the binary, Android's half. build-cdn stages each remote's embedded version under
// embed-root/android, the signed bundles and the version's mf-manifest.json, and this task mirrors
// that tree into the APK's assets at cdn/android, where EmbeddedRemotesModule copies it out at
// launch. Sync rather than Copy, so a file an earlier build embedded and this one does not is
// removed instead of shipped.
//
// With embed-root missing, this task embeds nothing, warnIfNothingToEmbed prints a warning, and the
// build still succeeds. That is the trap: a successful release build that carries no copies. Run
// build-cdn before a release build. ---
def embedSource = file("$rootDir/../../../embed-root/android")
def embeddedAssetsDir = layout.buildDirectory.dir("generated/embedded-remotes").get().asFile
def embedRemotes = tasks.register('embedRemotes', Sync) {
from(embedSource) { include '**/*.bundle', '**/mf-manifest.json' }
into(file("$embeddedAssetsDir/cdn/android"))
}
// --- A Sync with no source skips its actions, so a warning inside it would never print. Gradle
// still clears what the task embedded before when its source goes, so no stale copy ships. This
// task has no inputs or outputs, so it runs on every build, and it prints the warning when
// embed-root/android is missing. ---
def warnIfNothingToEmbed = tasks.register('warnIfNothingToEmbed') {
doLast {
if (!embedSource.exists()) {
logger.warn("warning: ${embedSource} does not exist, so no remotes are embedded. Run 'node tools/build-cdn.mjs' before a release build.")
}
}
}
android {
sourceSets {
main {
assets.srcDirs += embeddedAssetsDir
}
}
}
tasks.named('preBuild').configure { dependsOn embedRemotes, warnIfNothingToEmbed }
An asset is not a file on disk, and Re.Pack reaches a copy's <remote>/<version>/ directory only through a real file path. So a small TurboModule copies assets/cdn out to the app's files directory and resolves with the directory it used. Its spec is short enough to read whole:
import type { TurboModule } from 'react-native';
import { TurboModuleRegistry } from 'react-native';
// --- Android's half of the copy in the binary. A Gradle task packs each remote's embedded version
// into the APK's assets, and assets are not files on disk: Re.Pack's file:// loader needs a real
// path. prepare() copies them out to the app's files directory once per installed build, keeping
// the <remote>/<version>/ layout, and resolves with the directory the host builds file:// URLs
// against.
//
// There is no iOS implementation, because iOS reads the copies straight from the .app. That is why
// this uses TurboModuleRegistry.get, which returns null where the module does not exist, rather
// than getEnforcing, which would throw on iOS the moment this file loads. ---
export interface Spec extends TurboModule {
/** Copy the embedded remotes out, once per installed build, and return the directory. */
prepare(appVersion: string): Promise<string>;
}
export default TurboModuleRegistry.get<Spec>('EmbeddedRemotesModule');
The Kotlin side copies the tree once per install. A marker file holds the app version and the time the APK was installed or updated, so a relaunch reuses the extracted tree, and any new install replaces it, including a rebuilt APK that kept its version number. It copies the bytes as they are, since each bundle's signature covers them. Take the module, its spec and the updated HostNativePackage, which registers it next to post 13's navigation module, from the end tag:
cp /tmp/pokedex-ref-16/apps/host/specs/NativeEmbeddedRemotesModule.ts apps/host/specs/
cp /tmp/pokedex-ref-16/apps/host/android/app/src/main/java/com/host/{EmbeddedRemotesModule,HostNativePackage}.kt apps/host/android/app/src/main/java/com/host/
| | iOS | Android |
|---|---|---|
| Where the copy goes | the .app, at cdn/ios/<remote>/<version>/ | the APK's assets, at cdn/android/<remote>/<version>/, copied out to the files directory at launch |
| What puts it there | the last Run Script phase, embed-remotes-ios.sh | the embedRemotes Sync task, before preBuild |
| How the host finds it | the directory of main.jsbundle, read from React Native's SourceCode module | the directory EmbeddedRemotesModule.prepare() resolves with |
| Without embed-root/ | a warning, a successful build, no copy | a warning, a successful build, no copy |
Load a copy from the disk
The host changes in five files and their tests. Take them from the end tag. mf-modules.d.ts goes, because nothing loads a remote with import() any more (the Try again section says why), and the design system gains a contrast check for the banner's new colour:
cp /tmp/pokedex-ref-16/apps/host/src/shell/{remoteLocator,scriptManager,federationErrors}.ts /tmp/pokedex-ref-16/apps/host/src/shell/FederationBanner.tsx apps/host/src/shell/
cp /tmp/pokedex-ref-16/apps/host/App.tsx /tmp/pokedex-ref-16/apps/host/jest.config.js apps/host/
cp /tmp/pokedex-ref-16/apps/host/__mocks__/{module-federation-runtime,partyApp-partySlice}.js apps/host/__mocks__/
cp /tmp/pokedex-ref-16/apps/host/__tests__/{remoteLocator,scriptManager}.test.ts /tmp/pokedex-ref-16/apps/host/__tests__/{App,RemoteBoundary,FederationBanner}.test.tsx apps/host/__tests__/
cp /tmp/pokedex-ref-16/packages/ui/src/tokens/__tests__/contrast.accessibility.ts packages/ui/src/tokens/__tests__/
rm apps/host/mf-modules.d.ts
The host first has to know where the copies are on the device. On iOS the answer comes from React Native's own SourceCode module: a release build loads file:///…/Host.app/main.jsbundle, and the directory above that file is the .app. A development build loads its bundle from the dev server over http, so there is no directory to derive and no copy to use. On iOS, everything that runs from a copy needs a release build. On Android the directory is whatever prepare() resolved with:
const sourceCode = NativeModules.SourceCode as
| { scriptURL?: string; getConstants?: () => { scriptURL?: string } }
| undefined;
const SCRIPT_URL = sourceCode?.scriptURL ?? sourceCode?.getConstants?.().scriptURL;
const APP_PATH = SCRIPT_URL?.startsWith('file://')
? SCRIPT_URL.replace(/^file:\/\//, '').replace(/\/[^/]+$/, '')
: undefined;
let embeddedRoot: string | undefined = Platform.OS === 'ios' ? APP_PATH : undefined;
Every script of a remote still goes through post 15's resolver. It gains one branch: a remote that runs from its copy, in a launch that could not use the CDN's versions, or after that one remote failed to load from the CDN, resolves to a file inside its copy:
return {
kind: 'locate',
locator: {
url: `file://${input.embeddedRoot}/cdn/${input.platform}/${remoteName}/${version}/${filename}`,
cache: true,
absolute: true,
verifyScriptSignature: input.verify,
},
};
absolute: true is the detail that is easy to miss. Without it, Re.Pack keeps only the script's file name and looks for it at the top level of the app: iOS asks the app bundle's resources through URLForResource:withExtension:, and Android opens an asset of that name. Neither reaches a <remote>/<version>/ directory. With it, Re.Pack reads the file at the path as given.
Verification stays on. Re.Pack's file loader on both platforms runs the same signature check as its download path, before the code runs, on every load. That makes the copy stricter than a cached download, which post 15 noted is run from the disk without a second check. It is also why the copy is never rewritten on its way into the app: change one byte of an installed copy's code and the tab refuses it, failing the same hash check as a tampered download. On iOS the file loader reports the error's full description, so the reason arrives behind the system's generic prefix:
[Error: The operation couldn’t be completed. The bundle verification failed because the bundle hash is invalid.]
On Android the message reads as it does for a tampered download, with no prefix.
The manifest needs no special handling. The copy carries mf-manifest.json beside its bundles, and the federation runtime reads it from a file:// URL the way it reads one from the CDN, because React Native's fetch opens local files: through its file request handler on iOS, and on Android through the blob module's handler, which takes any URL that is not http or https when the response type is blob, the type React Native's fetch asks for.
Bundled mode: the day the CDN is unreachable
The launch now prepares the copies alongside the probe, since neither needs the other, and a launch with no usable map runs from them instead of giving up:
const [versions] = await Promise.all([
CDN_CONFIGURED ? fetchVersionMap() : Promise.resolve(null),
prepareEmbeddedCopies(),
]);
if (!versions) {
return runFromCopies(CDN_CONFIGURED ? 'no usable version map' : 'no CDN configured');
}
// --- The launch that could not use the CDN: no usable map, no CDN configured, or a registration
// that failed. With copies in the binary it runs from them, every remote registered at its copy's
// manifest, which the runtime reads from the disk. With none, or when registering them fails too,
// there is nothing to run. ---
function runFromCopies(reason: string): FederationStatus {
const embedded = REMOTE_NAMES.filter(hasEmbeddedCopy);
if (embedded.length === 0) {
setStatus({ mode: 'unresolved', source: reason, versions: {}, embedded: [] });
return status;
}
const versions = Object.fromEntries(embedded.map(name => [name, bundledVersions[name]]));
setStatus({ mode: 'bundled', source: 'the copy in the binary', versions, embedded });
try {
registerRemotes(
embedded.map(name => ({ name, entry: embeddedManifestUrlFor(name) })),
{ force: true },
);
} catch (error) {
console.warn('[federation] the copies could not be registered', error);
setStatus({ mode: 'unresolved', source: 'remotes could not be registered', versions: {}, embedded: [] });
}
return status;
}
Registering each remote at its copy's manifest, with post 15's force, sends the runtime to the disk for everything about that remote. From there nothing else is special: the runtime fetches the manifest, the resolver sends every script to the copy, and each one is verified as it loads. unresolved remains for a launch whose binary records no copy with a directory to load from, or where registering the copies fails.
Try it by taking the CDN away. Start the server as post 15 did, then make the Release build:
npx http-server@14.1.1 cdn-root -p 8000 -c-1 --cors
( cd apps/host && MF_CDN_BASE=http://localhost:8000 MF_APP_VERSION=2.0.0 npm run ios -- --mode Release )
The banner reads cdn · listApp 1.2.0 · partyApp 1.0.0. Stop the server and check that the map is gone:
curl -s -o /dev/null http://localhost:8000/ios/maps/2.0.0/version-map.json; echo $?
7 means curl could not connect. Close the app and open it again. The probe fails within its 1.5 seconds, the banner turns purple and reads bundled · listApp 1.2.0 · partyApp 1.0.0, and both tabs open from the copies, with nothing asked of the network except PokéAPI for the Pokémon and GitHub for their artwork.
!The Release build with the CDN stopped: the Pokédex tab lists Pokémon from Bulbasaur onwards under the chip listApp 1.2.0, and the purple banner above the tab bar reads bundled, listApp 1.2.0, partyApp 1.0.0
The copy is frozen at build time, and that is the trade-off. A binary built today and launched without the CDN next year runs today's versions, whatever the CDN has shipped since. That is the part of post 1's requirement a copy can meet: the reviewed code runs without the CDN. The Pokémon still come from PokéAPI, so with no network at all the list's request fails and the screen shows its own error, "Couldn't reach PokéAPI."
Nothing about the failure is remembered either. The next launch asks the CDN again, and if a usable map arrives and its versions register, the app is back on the versions the map names.
A remote that fails in a CDN launch
Bundled mode covers a launch that could not use the CDN's versions. The other failure is narrower: the launch uses the CDN's versions, and then one remote fails. The cause might be a version retired from the CDN while a map still names it, a chunk cut off halfway, or a release that throws while its module initialises, which was post 15's third break. In post 15's build each of those leaves one dead tab. Two nets turn them into a tab that runs from its copy, while the other remote stays on the CDN.
The manifest net
Module Federation fetches a remote's mf-manifest.json before any of its code loads. A tab's boundary would hear of that fetch failing, but only once the runtime's own fetch gives up, and the runtime sets no time limit on it. And not every load runs inside a boundary: the party's state and styles modules ask for its manifest at boot, from an effect outside every tab. So the first net sits where the runtime asks for every manifest: a runtime plugin's fetch hook, which the runtime calls before its own fetch, and which may answer with a Response or hand the request back by returning nothing:
const embeddedFallback: ModuleFederationRuntimePlugin = {
name: 'embedded-fallback',
fetch(url: string) {
if (status.mode !== 'cdn') {
return undefined;
}
const remote = manifestRemote(url, REMOTE_NAMES);
if (!remote || !hasEmbeddedCopy(remote)) {
return undefined;
}
if (fallbackRemotes.has(remote)) {
return fetch(embeddedManifestUrlFor(remote));
}
return cdnManifestOrEmbedded(url, remote);
},
};
registerPlugins([embeddedFallback]);
A CDN remote's manifest is fetched from the CDN, within the probe's 1.5-second limit. Any failed request (an error status such as a 404, a timeout or a dropped connection) drops that one remote to its copy and answers with the copy's manifest instead:
async function cdnManifestOrEmbedded(url: string, remote: string): Promise<Response> {
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), PROBE_TIMEOUT_MS);
try {
const response = await fetch(url, { signal: controller.signal });
if (response.ok) {
return response;
}
console.warn(`[federation] ${remote} manifest returned ${response.status}`);
} catch (error) {
console.warn(`[federation] ${remote} manifest could not be fetched`, error);
} finally {
clearTimeout(timer);
}
fallBack(remote);
return fetch(embeddedManifestUrlFor(remote));
}
A manifest that arrives with a success status goes back to the runtime as it is, even one that cannot be read. The runtime fails on that afterwards, and for a tab's load the boundary net catches the failure.
fallBack adds the remote to a set and puts it on the banner, as listApp 1.2.0 embedded. The resolver reads that same set for every script, so the remote's very next script resolves to its copy, while every other remote keeps its CDN URL. The set lives in memory on purpose: a remote drops to its copy only until the next launch, which asks the CDN again, so a failure that was only the network does not keep it there. Remembering failures across launches, and rolling a bad version back permanently, belongs to post 17.
Other tools already offer parts of this, and each would work at its own layer. Module Federation's errorLoadRemote hook can return a fallback when a load fails, the manifest included. Its official retry plugin retries a failed load and can rotate between backup domains, and Re.Pack's script locator takes retry and retryDelay. The fetch hook was the fit here for two reasons. errorLoadRemote hears of a failure only after the runtime's fetch gives up, as the boundary does, while the fetch hook makes the request itself and can put the probe's time limit on it. And retrying is a different policy: it spends time asking the same CDN again, where this net answers at once from a copy the binary already has and leaves the next attempt to the next launch. Retries could sit in front of the net, at the cost of a longer wait before the copy.
Try the net with a version the CDN no longer holds. Start the server again, move the list's version out of it, and cold-start the app:
mv cdn-root/ios/listApp/1.2.0 /tmp/listApp-1.2.0
The list's manifest comes back 404, the list runs from its copy, and the party still loads from the CDN, which the server's log shows:
[2026-09-29T14:40:27.102Z] "GET /ios/listApp/1.2.0/mf-manifest.json" "Host/1 CFNetwork/3860.500.112 Darwin/27.0.0"
[2026-09-29T14:40:27.103Z] "GET /ios/listApp/1.2.0/mf-manifest.json" Error (404): "Not found"
[2026-09-29T14:40:27.120Z] "GET /ios/partyApp/1.0.0/mf-manifest.json" "Host/1 CFNetwork/3860.500.112 Darwin/27.0.0"
[2026-09-29T14:40:27.159Z] "GET /ios/partyApp/1.0.0/partyApp.container.js.bundle" "Host/1 CFNetwork/3860.500.112 Darwin/27.0.0"
!The Release build with the list's version missing from the CDN: the Pokédex tab lists Pokémon from Bulbasaur onwards under the chip listApp 1.2.0, and the blue banner reads cdn, listApp 1.2.0 embedded, partyApp 1.0.0
Move the directory back before going on:
mv /tmp/listApp-1.2.0 cdn-root/ios/listApp/1.2.0
<blockquote class="callout callout-tip">
<p><strong>If every remote drops to its copy, check that the app can reach the server.</strong> A banner that marks both remotes embedded, or reads <code>bundled</code>, while the server is running usually means the app could not reach it: iOS's App Transport Security and Android's cleartext rules refuse plain http if the build does not allow it. This host allows local networking on iOS and, on Android, cleartext only to <code>localhost</code>, <code>127.0.0.1</code> and <code>10.0.2.2</code>, with their subdomains.</p>
</blockquote>
The boundary net
Some failures come after a manifest that arrived fine: a container or a chunk that does not arrive or does not verify, or a module that throws while it is evaluated. They fail the tab's load, and the tab's load runs inside the RemoteBoundary that post 11 put around each tab. The boundary is the second net, and it stays a class component, because React has no way yet to write an error boundary as a function component. It now asks one question before it shows the error state: did the load fail, so the remote never produced a component, and is there a copy this remote has not dropped to yet?
componentDidCatch(error: unknown) {
console.warn(`${this.props.remote} failed`, error);
if (this.dropsToCopy()) {
fallBackAndReload(this.props.remote);
this.nextAttempt();
}
}
// A failure the copy in the binary can answer: the load failed, so the remote never produced a
// component, and this remote has a copy it has not dropped to yet.
dropsToCopy() {
return !this.loaded() && canFallBack(this.props.remote);
}
When it can, the boundary drops the remote to its copy and loads the tab again, rendering the loading state in the meantime, so the error state never appears. canFallBack allows it once per remote, only in a CDN launch, and only for a remote that has a copy. When it cannot, because the copy failed too, or there is none, the tab shows the error state as before.
A remote whose load returned a component keeps its error state when that component throws. That is a decision, and it rests on whether the remote produced a component. Before it does, the tab is still showing its loading state, so a swap to the copy is invisible. Once the load has returned a component, the boundary keeps that version, even when the component throws on its very first render. It records the attempt whose load arrived with a component, not whether the screen was ever shown, so a failure in that attempt counts as thrown by the remote's own code, which has already run in this session. Until the map names a different version, the copy is that same version anyway. So the boundary shows "This tab stopped working" and offers Try again, which renders the remote again and recovers an error that does not repeat. An error that repeats leaves the tab on its error state for the rest of the session, and after a relaunch for as long as the map names that version, even with a working copy in the binary.
The boundary net does not cover the party's two boot modules: they load from an effect outside every tab, so the host only logs a failure the manifest net does not answer. When the state module fails, adding to the party stays disabled until the Party tab loads, because that tab injects the state itself; a failed styles module does not affect adding.
Post 15's third break is now a failed load the copy can answer. Its release throws while its module initialises, and post 15's evaluation window turns that throw into a failed load instead of a fatal report. Build it again. Add the same line below the imports of apps/list/src/PokedexScreen.tsx:
throw new Error('PokedexScreen failed to initialise');
Then ship it as 1.4.0, the way post 15 did:
( cd apps/list && MF_REMOTE_VERSION=1.4.0 npm run bundle:ios:prod ) && rsync -a --exclude '*.map' --exclude mf-stats.json apps/list/cdn/ios/listApp/1.4.0/ cdn-root/ios/listApp/1.4.0/
Delete the line, point the 2.0.0 map at 1.4.0 and relaunch. The tab drops to the 1.2.0 copy: the chip says 1.2.0, the banner marks the list embedded, and the simulator's log has no RCTFatal in it. Put the map back to 1.2.0.
The boundary net also catches a CDN that disappears mid-session. Launch with the server running and leave the Party tab unopened. Add a Pokémon from the Pokédex, stop the server, then open Party for the first time. Only the party's state and styles modules loaded at boot, so its screens are not on the device yet: the download fails, and the tab shows a spinner for a moment and then the party from its copy, with the Pokémon you added and the banner reading partyApp 1.0.0 embedded. Removing the Pokémon there works too, because the copy's screens and the state module that loaded from the CDN share the host's one store.
A Try again that retries
In post 15's build, Try again on a tab whose load failed never brings the tab back. Three records of the failed load stand between Try again and a fresh attempt.
The first is React's. A lazy component calls its load function once: the React documentation says "Both the returned Promise and the Promise's resolved value will be cached, so React will not call load more than once", and a rejection goes to the nearest error boundary. The boundary already handled that, by mounting a new lazy component under a new key for every attempt.
The second is the host bundle's, and it is the one that kept post 15's tabs dead. An import('listApp/ListStack') compiles into a module of the host's own bundle, and the bundle keeps each module it has evaluated for the rest of the session. After a failed load it keeps what the failure left behind, a module with nothing in it, and every later import() settles with that, even when the retry downloads the remote successfully. Measured on an iOS Release build: the retry fetched the manifest and both chunks, and the import still resolved without a component. So the tabs, and the two modules the party loads at boot, no longer use import(). They ask the federation runtime directly, every time:
export async function loadRemoteModule<T>(id: string): Promise<T | undefined> {
const factory = await loadRemote<() => T>(id, { loadFactory: false, from: 'runtime' });
return factory ? factory() : undefined;
}
It asks for the module's factory rather than its exports, with loadFactory: false, which is how post 15's evaluation window gets to wrap it: the runtime hands back the window's wrapper, and calling it here evaluates the module inside the window.
The third is the federation runtime's. In @module-federation/runtime-core 2.9.0, the version this series installs, its record of a remote outlives a failed load: it keeps the remote's entry, the manifest it read, the container it loaded, and a container load that failed, which it hands back to every later request. The remote's chunk registry, a global array named after the remote's uniqueName, also keeps every chunk its last container fetched, and a new container installs all of them before fetching anything. Try again clears both before it loads again:
function reloadRemote(remote: RemoteName, entry: string): void {
try {
registerRemotes([{ name: remote, entry }], { force: true });
} catch (error) {
console.warn(`[federation] ${remote} could not be registered again`, error);
}
delete (globalThis as Record<string, unknown>)[CHUNK_REGISTRY[remote]];
}
Registering the remote again with force removes the runtime's record, the container's global included, and points the next load at entry. Try again reads entry back from the runtime instead of building it, which works in every mode, including development, where only the build knows the dev server's URL. A remote that dropped to its copy still retries its copy, because the manifest net and the resolver both send any remote in the set to its copy.
Re.Pack's script cache needs no clearing for a failed load: once a resolved script's load settles, Re.Pack drops its promise, and a download that fails verification is never written to its cache, on either platform. A refusal from the resolver is the exception. It comes before that cleanup, so Re.Pack keeps the rejection and hands it back to every later request for the same script. This host's resolver refuses only a remote it has no version or copy for.
One gap stays open: Try again cannot cancel a download already in flight. A chunk the failed container was still fetching lands in whichever registry exists when it arrives, and when a script is still loading, Re.Pack answers a new request for it with the promise already outstanding. When the copy is the version the CDN was serving, the default, those are the same bytes either way; when the map names a different version, a late chunk from the CDN can land beside the copy's container.
A render error shows Try again recovering a tab whose code has already loaded. The boundary net's mid-session demo stopped the server, so start it again and leave it running:
npx http-server@14.1.1 cdn-root -p 8000 -c-1 --cors
Then make the list's screen throw while it renders, for its first eight seconds. Below the imports of apps/list/src/PokedexScreen.tsx add:
const LOADED_AT = Date.now();
and as the first lines of PokedexScreen():
if (Date.now() - LOADED_AT < 8000) {
throw new Error('PokedexScreen render failed');
}
The throw has to last: React retries a render that fails, so an error that stops too soon recovers before the boundary's error state is ever shown. Ship it as its own version, the way 1.4.0 shipped, then delete the lines and point the 2.0.0 map at 1.5.0:
( cd apps/list && MF_REMOTE_VERSION=1.5.0 npm run bundle:ios:prod ) && rsync -a --exclude '*.map' --exclude mf-stats.json apps/list/cdn/ios/listApp/1.5.0/ cdn-root/ios/listApp/1.5.0/
Relaunch and the tab shows "This tab stopped working". Wait out the eight seconds and press Try again: the list renders on 1.5.0, and the server's log shows no new request, because the code had loaded and only needed rendering again. Put the map back to 1.2.0 afterwards.
> 🎞️ Animated demo: watch it on warrendeleon.com
The failures and what each one costs, side by side:
| Failure | Caught by | What the user sees |
|---|---|---|
| The CDN cannot be reached at launch | the launch, in bundled mode | every tab from its copy, and a purple banner |
| A remote's manifest fails: a 404, a timeout, a dropped connection | the manifest net | the tab from its copy, marked embedded on the banner |
| A tab's remote code does not arrive, does not verify, or throws as its module initialises | the boundary net | a moment of loading, then the tab from its copy |
| The party's state module fails at boot, once its manifest has arrived | nothing: the host logs it | adding to the party stays disabled until the Party tab loads |
| The party's styles module fails at boot, once its manifest has arrived | nothing: the host logs it | adding still works |
| The copy fails as well, or there is no copy | the boundary | "This tab could not load", with Try again |
| The remote's component throws as it renders | the boundary | "This tab stopped working", with Try again |
Now break it
Both embed steps succeed without embed-root/, and that is the trap. Move it out and rebuild the Release build, this time with --verbose:
mv embed-root /tmp/embed-root && ( cd apps/host && MF_CDN_BASE=http://localhost:8000 MF_APP_VERSION=2.0.0 npm run ios -- --mode Release --verbose )
The build succeeds, and the phase's warning is the only sign that anything is missing. Without --verbose, npm run ios shows a spinner in place of the build's output (or passes the output through xcbeautify or xcpretty, when one is installed), then reports success Successfully built the app and launches. With --verbose, the warning sits among the build's own lines:
debug warning: …/embed-root/ios does not exist, so no remotes are embedded. Run 'node tools/build-cdn.mjs' before a release build.
debug ** BUILD SUCCEEDED ** [11.716 sec]
success Successfully built the app
Stop the server and cold-start the app. The banner still says bundled · listApp 1.2.0 · partyApp 1.0.0, because the host compiled in the versions it was told it carries, and both tabs say "The app's own copy of this remote could not be loaded".
<blockquote class="callout callout-danger">
<p><strong>Build the CDN tree, then the release build, every time.</strong> The embed phases copy whatever <code>embed-root/</code> holds when they run, and when it is missing they let the build succeed with nothing more than a warning in the build's output. A release build made before <code>node tools/build-cdn.mjs</code>, or after <code>embed-root/</code> was cleaned, builds successfully with no copy in it, and nothing fails until the app needs a copy: when the launch cannot use the CDN's versions, or when one remote fails.</p>
</blockquote>
Move it back and rebuild before going on:
mv /tmp/embed-root embed-root && ( cd apps/host && MF_CDN_BASE=http://localhost:8000 MF_APP_VERSION=2.0.0 npm run ios -- --mode Release )
Run it
The copies and the CDN tree, then the server:
node tools/gen-signing-keys.mjs && node tools/build-cdn.mjs
npx http-server@14.1.1 cdn-root -p 8000 -c-1 --cors
The Release build on iOS, then Android with the emulator's address for the machine:
( cd apps/host && MF_CDN_BASE=http://localhost:8000 MF_APP_VERSION=2.0.0 npm run ios -- --mode Release )
( cd apps/host && MF_CDN_BASE=http://10.0.2.2:8000 MF_APP_VERSION=2.0.0 npm run android -- --mode release )
With either running, stop the server and cold-start for bundled mode, or move one version out of cdn-root for the manifest net. On Android the extracted copies sit in the app's files directory, under files/cdn/android/, with the .installed marker beside them in files/cdn/.
The suites, the smoke test and the key generator's tests:
( for d in apps/host apps/list apps/party packages/ui; do ( cd "$d" && npx jest --silent ) || exit 1; done ) && sh scripts/federation-smoke.sh && node --test tools/gen-signing-keys.test.mjs
What you built, and what's next
The demos in this post ran on an iOS Release build, and the copies, their extraction and bundled mode on an Android release build as well.
A remote that fell back is forgotten at the next launch, so a version that keeps failing fails again on every launch before its copy takes over. Remembering failures belongs with the other open problem: the map every binary trusts is still a plain text file anyone who can write to the bucket can change.
Next: signing that map, a counter that makes the app reject an older map, and an app that rolls a failing version back by itself.
Sources
- React: lazy — the load function called once, its promise and resolved value cached, and a rejection thrown to the nearest error boundary
- React source: ReactLazy at 19.2.3 — a rejected load stored on the lazy component and thrown again on every later render
- React: Component, catching rendering errors with an error boundary — "There is currently no way to write an Error Boundary as a function component"
- Module Federation: runtime hooks — the fetch hook for manifest requests, returning Promise<Response> | void | false, and errorLoadRemote
- Module Federation source: runtime-core SnapshotHandler at 2.9.0 — a manifest asked of the fetch hook first, then of the global fetch, then of errorLoadRemote on failure
- Module Federation source: runtime-core remote/index.ts and utils/load.ts at 2.9.0 — registerRemotes with force calling removeRemote, which deletes the remote's entry, manifest, container global and container-loading promise; a failed container load's promise kept and returned to later loads until then
- Module Federation: retry plugin — retries and backup domains for failed loads
- Re.Pack source: ScriptManager types at 5.2.5 — the locator's absolute, retry and retryDelay
- Re.Pack source: ScriptManager.ts at 5.2.5 — a resolved script's promise deleted once its load settles, failed or not; a resolver's rejection comes before that cleanup and is kept
- Re.Pack source: ScriptManager.mm at 5.2.5 — a file:// script read at its absolute path, or by name through URLForResource:withExtension:, and verified before it runs; a download written to the cache only after it verifies
- Re.Pack source: FileSystemScriptLoader.kt and RemoteScriptLoader.kt at 5.2.5 — the same on Android, with an asset opened by its file name when the path is not absolute, and a download verified before it is written
- Apple: URLForResource:withExtension: — a resource looked up by name in the bundle, beside the variant that takes a subdirectory
- swift-build source: ShellScriptTaskProducer.swift at swift-6.2.1-RELEASE — a Run Script phase that declares no outputs, or is set to run always, runs on every build
- React Native CLI source: buildProject.ts at 20.2.0 — without --verbose, the build's output piped through xcbeautify or xcpretty when one is installed, and a spinner otherwise
- React Native source: RCTSourceCode.mm at 0.85.3 — scriptURL, the URL the JavaScript bundle loaded from
- React Native source: RCTFileRequestHandler.mm and RCTAppSetupUtils.mm at 0.85.3 — file:// requests answered on iOS, and the handler registered with the app's networking
- React Native source: BlobModule.kt at 0.85.3 — any URL that is not http or https, read on Android when the response type is blob
- Android: AssetManager — assets opened and listed from the APK, not as files
- Gradle: Sync — a copy that also removes files the source no longer has
- react-native-module-federation — the companion repo, the build at the tag post-16-fallbacks
Mentions
later in November we'll make agent-device leaner and move some of the functionality, such as remote providers, to plugins. I guess every project will have its inflation and shrinking phase and we're approaching that now that core functionality is locked github.com/callstack/agent-device/milestone/1 t.co
Test Automation with agent-device (Part 3) --- title: "Test Automation with agent-device (Part 3)" published: true description: "Save test flows as agent-device .ad scripts and run them again." tags: claude, ai, flutter, testing canonical_url: krenin88.github.io --- This post covers saving test flows as agent-device .ad scripts and running them again, using a Flutter app. It goes through writing an .ad file, getting the app into the right state before a replay, reading the report when a replay stops and fixing it, and the token cost of agent-driven QA. It is based on agent-device 0.21.31 (2026-10-25). Installation and the basic loop are in Part 1, and real-device setup is in Part 2. What an .ad file is When an agent checks an app, it reads the screen, picks the next thing to tap, and reads the result, over and over. Exploring the same flow this way every time keeps costing time and tokens. An .ad file pins down the result of that exploration as a list of commands, and replay runs the list from top to bottom. Each line has the form <command> [positional...] [flag...], the same as an agent-device command in the terminal. Quoting follows shell rules too, so a selector containing spaces is wrapped like 'label="..."'. Below is the start of a flow that opens the app fresh, taps a button on the first screen, types into an input, and checks the result screen. # First-run flow — the app ID comes in via -e APP= open ${APP} --relaunch wait 'label="<first-screen button>"' 30000 press 'label="<first-screen button>"' fill 'id="find-input"' "<text to type>" press 'label="<search button>" role=button' wait 'label="<result title>"' 60000 wait text "<text the result should show>" 10000 back close A few notes: - # comments work. The docs do not say so, so the first line was a comment in a test replay, and it passed. - Do not hard-code the app ID. Leave it as ${APP} and pass -e APP=<bundle id> at replay time, and the same file works for builds with different IDs, such as a dev build and a release build. - The number after wait is a timeout in milliseconds. Give a long one to steps that wait on the network or on model inference. - Device-selection flags stay out of the file. --platform, --serial, --device, and --settle are not part of a script step; they go on the replay command. There are two ways to make an .ad file. You can start recording with open <bundle id> --relaunch --save-script=<file>.ad, go through the flow, and save with session save-script, or you can write it by hand. The 24-step flow in this post was written by hand, based on a flow the agent had already passed manually. Getting to a "first launch" state before replaying replay only runs the commands written down; it does not reset the app. If the flow starts by checking the first-launch consent dialog, an app that still remembers an earlier consent stops right at step 2. So a separate runner script deletes and reinstalls the app before each replay. # Android emulator adb -s <serial> uninstall <bundle id> adb -s <serial> install -r <apk path> adb -s <serial> shell cmd locale set-app-locales <bundle id> --locales en-US agent-device replay <file>.ad --platform android --serial <serial> -e APP=<bundle id> --timeout 240000 # iOS simulator xcrun simctl uninstall <UDID> <bundle id> xcrun simctl install <UDID> <Runner.app path> agent-device replay <file>.ad --platform ios --udid <UDID> -e APP=<bundle id> --timeout 300000 On Android, set-app-locales opens just this app in English while leaving the device language alone. On the iOS simulator no per-app language command was used, so the app follows the simulator's language setting. replay has a default work budget of 60 seconds, so raise it with --timeout for longer flows. Deleting the app also deletes its data. So the runner script accepts only the designated test emulators (checked by name with adb -s <serial> emu avd name) and simulator UDIDs, and stops with exit code 2 if it gets any other device or an app ID outside your own. An emulator serial such as emulator-5554 is assigned in boot order and does not tell you which AVD it is, which is why the check uses the name. With that in place, the 24-step flow finished with Replayed 24 steps on both runs on each platform: 8.8 and 9.4 seconds on the Android emulator, 13.9 and 14.0 seconds on the iOS simulator. When a replay stops When a replay gets stuck partway, it fails with REPLAY_DIVERGENCE. The report includes the step number and its line, the actionable elements on screen at that moment, and selector suggestions for that screen. So you do not have to start over after a fix, it also tells you how to resume with replay --from <step> --plan-digest <value>. When no automatic repair is possible, it says Repair hint: manual. A first-launch system dialog. The flow that passed on Android stopped at step 2 the first time it ran on the iOS simulator. Error (REPLAY_DIVERGENCE): Replay failed at step 2 (wait "label=\"<first-screen button>\"" 30000): wait timed out for selector: ... Current surface: Allow "<app name>" to track your activity across other companies' apps ... The report's Current surface showed the App Tracking Transparency (ATT) dialog. It is a system dialog that appears on first launch only on iOS, so the flow written against Android had no step for it. After closing it with agent-device alert dismiss and replaying again, the run got past this step. The runner script does not handle it, so on builds that show this dialog, close it before replaying. Input role names that differ by platform. Next, the replay stopped at the fill in step 5. The first selector was role=text-field, which matched on Android, but the iOS simulator exposed the same input as an other element with a label. To find one selector that works on both platforms, these were tried in order: 1. editable=true — passed on Android, Selector did not match on iOS 2. role=textfield || role=text-field — || is the "or" syntax that appeared in the report's selector suggestions. It still did not match on iOS. 3. Adding Semantics(identifier:) to the Flutter widget — passed on both platforms The last fix is one line of app code: identifier: 'find-input' added to the existing Semantics around the input, with no visible change on screen. The selector becomes id="find-input". As noted in Part 1, a Flutter Key does not show up in the accessibility tree, so adding an identifier is the reliable way to run the same script on both platforms. Token cost and the rules for cutting it Token cost was the reason for using .ad in the first place. Most of the cost of agent-driven QA came not from output but from cache reads. A cache read is the earlier conversation the agent re-reads on every response. The first real-device check of two apps took about 40 minutes and came to 121 responses, 77,985 output tokens, and 30,225,404 cache-read tokens, roughly 250,000 tokens re-read per response. The QA agent was already on Sonnet, so the model was not the cause. In the same stretch, 11 screenshots were read into the conversation, and as the context grew, every response re-read more. So five rules went into the QA agent's instructions: 1. Save a flow as .ad the first time it passes, and use replay from then on. When it gets stuck, fix only that step using the divergence report. 2. Act with --settle and look only at the returned diff. Do not read the full screen tree. 3. Save screenshots to files only, and read at most one per item into the conversation. 4. End the session per item: one app, one thing to check. Past 40 responses, it is time to stop. 5. Read logs and build output with grep or tail, only the lines you need. The batch after the rules (three emulator checks, an iPhone attempt, an Android real device, about 30 minutes) came to 105 responses, 44,113 output tokens, and 19,171,162 cache-read tokens, about 180,000 cache-read tokens per response. The items checked were different, so the two sets of numbers cannot be compared directly, and the 40-response limit was not met. Tokens after adopting .ad have not been measured separately yet, and neither replay on a real device nor running it in CI has been tried. References - agent-device repository: <callstack/agent-device> - agent-device agent setup docs: <oss.callstack.com> - Flutter SemanticsProperties.identifier: <api.flutter.dev>
Claude Code に React Native アプリを触らせるなら agent-device の CLI はじめに AI に React Native のバグを直させると、コードはそれらしく直してくれます。ただ、シミュレータで本当に直ったかを確かめさせようとすると、何かツールを入れる必要があります。React Native の有名どころの会社からも、Callstack が agent-device、Software Mansion が Argent を出していて、ほかに mobile-mcp もあります。 どれを入れればいいのか。比べた記事を探したら、英語でも日本語でも機能表の比較しか見つかりませんでした。Expo 自身もブログで「検証ツールの比較は今後の実験」と書いています。そこで、Bluesky の公式アプリで実際に直されたバグを使って、180回測りました。 ! 検証環境は Claude Code 2.1.310、Opus 5.5、Xcode 27.1、iOS 26.5 のシミュレータ (iPhone 17) です。アプリは Bluesky の公式アプリ (コミット 61c7cad、Expo SDK 57) を使い、計測は2026年10月に行いました。ほかのアプリやエージェント、Android では試していません。 React Native なら agent-device の CLI を入れればいい 比べたのは、Callstack の agent-device、Software Mansion の Argent、Mobile Next の mobile-mcp の3つです。3つとも CLI と MCP の両方の入口があるので、6通りで測り、1回あたりの費用と、画面で再現から確認までやり切れたかを見ました。作業時間も参考に測っています。 早速結論ですが、自分のアプリに入れるなら agent-device の CLI を選びます。理由は2つです。 一番安い。1回あたり $0.70 で、mobilecli (mobile-mcp の CLI) と並んで最安。同じ agent-device の MCP より3割近く安い React Native 向けの機能がある。コンポーネントの props や state を調べる React DevTools や、Metro の再読み込みをコマンドで使える。mobilecli には無い 費用だけなら mobilecli も同じくらい安く済みます。React Native のアプリなら、機能の分だけ agent-device の CLI が上だと判断しました。 MCP しか使えない環境なら、mobile-mcp を選びます。タップや入力をまとめて1回で送れ、画面を読んだ結果も短いので、MCP の3つの中で一番安く済みました。 速さを重視する場合は、Argent の MCP が選択肢になります。6通りの中で作業時間が一番短く済みました。ただし費用は一番高く、agent-device の CLI の1.6倍かかります。タップするたびにスクショが返ってくるので、やり取りは少なく済む一方、受け取る画像が多くなるためです。 Bluesky の実バグで180回測った 比べた3つのツール どれも、エージェントにシミュレータの画面を読ませ、タップや入力をさせるためのツールです。違いは、画面をエージェントにどう渡すかにあります。 ツール 開発元 画面の渡し方 agent-device Callstack ボタンや文字を @e2 [button] "Add" のような参照の一覧で返す。操作のあとは、変わった部分だけを差分で返す Argent Software Mansion タップなどの操作のたびに、結果のスクショを自動で返す。画面の要素の一覧も別に取れる mobile-mcp Mobile Next 画面の要素を1行1個の一覧で返し、座標を指定してタップする。複数の操作をまとめて1回で送れる CLI はエージェントがシェルでコマンドを打って使い、MCP は Claude Code にツールとして登録して使います。mobile-mcp の CLI は、中で動いている mobilecli というエンジンをそのまま使う形です。この画面の渡し方の違いが、あとで見る費用の差になって表れます。 比べた6つの条件 3つのツールの CLI と MCP で6条件です。ツールはどれも公式ドキュメントが勧める入れ方にしました。 条件 入れたもの agent-device CLI CLI 0.21.31、公式の skill 4つ、公式が勧めるプロジェクトのルール文 agent-device MCP agent-device mcp、同じルール文 Argent MCP argent init が入れるもの一式 (MCP、skill 18個、rules) 0.30.1 Argent CLI 上から MCP の設定だけを外したもの mobile-mcp MCP @mobilenext/mobile-mcp 1.0.13 mobile-mcp CLI 中のエンジンの mobilecli 1.0.21 と公式の skill xcrun simctl の起動・スクショ・URL を開く操作は、どの条件でも使えるようにしました。テレメトリは全ツールで切っています。 この6条件に、次のお題5問 × testID とアクセシビリティのラベルの有無 × 3回を掛けて、全部で180回測りました。 お題は Bluesky で実際に直された5件 Bluesky の公式アプリは Expo SDK 57 のアプリで、E2E テスト用のモックサーバーが付いています。毎回同じデータでアプリを立ち上げられるので、計測に向いています。 お題は、Bluesky で実際にマージされた PR を戻して作りました。選んだのは、モックサーバーで症状を出せて、直ったかを Maestro で判定できる PR です。比べるツールを一度も動かす前に決めたので、結果を見てお題を選び直してはいません。 お題 症状 症状を出すのに必要な操作 #10132 画像に ALT を付けても案内が消えない 画像の添付、ALT の入力と保存 #10163 モデレーションリストが開けない URL でリストを開く #11503 存在しないユーザーのプロフィールが読み込み中のまま止まる URL でプロフィールを開く #8502 引用を禁止した投稿でも、リポストの長押しで引用の作成画面が開く 別のアカウントで引用禁止の投稿を作り、リポストを長押し #11820 (機能追加) 新規登録の生年月日を入力必須にする 新規登録のフォームに入力して Next を押す エージェントには、バグの症状と環境を伝えたうえで「シミュレータで再現して、直して、直ったことを確かめて」と頼みました。条件ごとに変えたのは、使うツールを指定する1文だけです。ALT の案内のバグだけは、Bluesky の issue の英文をそのまま貼っています。「このバグを直して」とだけ頼んだときに、エージェントが自分から画面で確かめに行くかは測っていません。 実際に渡した指示の例このリポジトリは Bluesky の公式アプリです。次のバグに対応してください。 存在しないハンドルのプロフィール (例: bluesky://profile/nobody.test) を開くと、エラー画面にならず、読み込み中の表示のまま止まってしまう。 アプリは iOS シミュレータ「BlueskyLab iPhone 17」(UDID 2B6D50B9-95B8-4FD8-8EFB-37FFE605ACF2) で起動していて、Metro (localhost:8081) とローカルのテストサーバー (localhost) につながっています。alice でサインインしています。このシミュレータ以外は別の作業で使っているので、触らないでください。ファイルを保存すると、Metro がアプリを自動で再読み込みします。 シミュレータの操作には agent-device の CLI を使ってください。 シミュレータでバグを再現し、コードを直して、直ったことをシミュレータで確かめてください。Web 検索はしないでください。終わったら、原因と直した内容を短く報告してください。 バグが直ったかどうかは、エージェントの作業が終わったあとに、アプリが Bluesky の修正と同じ動きになるかを Maestro のフローで確かめました。エージェントが画面で確かめたかどうかは、ログを Sonnet 5.5 に読ませて判定し、食い違った箇所はコードとログを自分で読んで確認しています。それでも見落としはありえます。 エージェントに答えを見せない仕掛け Bluesky は公開リポジトリなので、直し方がエージェントの手の届くところに残っています。答えを見て直せてしまうと、画面で再現する前に作業が終わり、ツールの差を測れません。比べたいツールを通らずに画面を操作できる道も同じです。そこで、次のように塞ぎました。 git の履歴と差分には、修正そのものが残っている。バグを入れた状態を1コミットだけのリポジトリにして、回ごとに作り直した 公式の E2E テストには、どの testID を押せばいいかが書いてある。テストのフォルダを外した アプリには E2E テスト用に、ログインや画面の移動を1回で済ませる隠しボタンがある。描画しないようにした Maestro が入っていると、比べたいツールを使わずに画面を操作できる。Maestro をコマンドとして呼べなくした Web で検索すると、元の PR が見つかる。Web 検索を禁止した 実際の開発ではエージェントが git の履歴や E2E テストも読めるので、費用や時間はこの記事の数字より軽く済むかもしれません。外したものはどの条件でも同じなので、ツールどうしの順位には響かないと見ています。 agent-device の CLI を選ぶ2つの理由 再現から確認までは、6条件すべてが30回中30回やり切りました。差が出たのは費用です。agent-device の CLI を選ぶ理由は、一番安いことと、React Native 向けの機能があることの2つです。 CLI は画面を読む量が少ないので一番安い 条件 1回の費用 (平均) 作業時間 (平均) 再現・確認まで終えた agent-device CLI $0.70 2分05秒 30 mobile-mcp CLI (mobilecli) $0.70 2分08秒 30 mobile-mcp MCP $0.80 1分53秒 30 Argent CLI $0.91 3分04秒 30 agent-device MCP $0.95 1分53秒 30 Argent MCP $1.11 1分41秒 30 agent-device の CLI は、mobilecli と並んで一番安く、Argent の MCP の6割ほどで済みました。同じお題・ラベル・回の組で比べると、agent-device の MCP には30組中29組、Argent の MCP には30組すべてで安くなっています。mobilecli とは30組中18組で、差はありません。 費用は API の料金に換算した値です。トークンの96%はキャッシュの読み込みで、1回平均142万トークンでも費用は $1 前後に収まりました。計測の順番によっては前の回のキャッシュを使えて安く出るので、どの回もキャッシュが空の状態から始めたとして計算し直しています。同じ組み合わせ3回の間のばらつきは、どの条件も6〜14%でした。 作業時間は参考にとどめてください。計測の途中に、同じ Mac で別の作業をしていた時間帯があります。 安さのわけは、画面を読む量にあります。CLI はシェルで動くので、エージェントが | grep や | head で必要な部分だけを読めます。同じツールの CLI と MCP を比べると、3つとも CLI の方が安くなりました。agent-device で3割近く、Argent で2割弱、mobile-mcp で1割強です。 画面を読む量の内訳 agent-device の2つをログで比べると、差は画面を読み取る量から来ていました。 条件 やり取り 画面を1回読んだ結果の長さ 受け取った画像 agent-device CLI 26回 約580文字 3.5枚 agent-device MCP 26回 約5,300文字 2.5枚 やり取りの回数は同じで、1回に読む画面の長さが9倍ほど違います。MCP は、ツールが返したものを丸ごと受け取るしかありません。mobile-mcp でも、CLI の約620文字に対して MCP は約2,300文字でした。 Argent の MCP が1回の作業で受け取った画像は平均9.5枚で、agent-device の CLI (3.5枚) の3倍近くでした。画像は文字よりトークンが重いので、そのぶん費用がかさみます。 なお、「画面を読み取った回」はツールの名前やコマンドの文字で拾う簡易な数え方です。傾向は見えますが、厳密な内訳ではありません。 React Native 向けの機能がある 費用で並んだ mobilecli との違いは、React Native 向けの機能です。agent-device には、React Native のアプリを調べるためのコマンドがあります。 できること コマンドの例 コンポーネントの木、props、state、hooks を見る agent-device react-devtools tree / inspect 遅い描画や再描画の原因を調べる agent-device react-devtools profile start JS だけの変更を Metro で読み込み直す agent-device metro reload LogBox や RedBox の表示を閉じる agent-device react-native dismiss-overlay React DevTools は、Metro につながった開発ビルド (React Native 0.74 以上) なら、アプリに手を入れずに使えます。mobilecli のコマンドは画面の操作と端末の管理が中心で、React Native に特化したものはありません。 ただし、今回の計測ではこの差は測れていません。お題はどれも画面に症状が出るバグで、agent-device の CLI の回でも、ログや通信を見るコマンドが数回使われただけでした。再描画が多すぎて重い、のように画面だけでは原因が分からないバグでどれだけ効くかは、試していません。 作りから予想できたことと、外れたこと ここまでの結果のうち2つは、ツールの作りから予想がつきます。Argent の MCP はタップのたびにスクショを返すので、受け取る画像が増えて高くなります。CLI はエージェントが要る部分だけを読めるので、ツールの返すものを丸ごと受け取る MCP より安くなります。予想と違ったのは、次の2つでした。 testID やラベルを外しても、費用も時間も変わらない testID とアクセシビリティのラベルをアプリから全部外した版でも測りました。アイコンだけのボタンが、名前の無いただの「Button」になる状態です。エージェントがボタンを探しにくくなり、費用がかさむと予想していました。 条件 1回の費用 (ラベルあり → 無し) 無しの方が安かった組 agent-device CLI $0.71 → $0.69 8/15 mobile-mcp CLI $0.71 → $0.69 8/15 mobile-mcp MCP $0.83 → $0.76 8/15 Argent CLI $1.01 → $0.82 11/15 agent-device MCP $0.94 → $0.97 7/15 Argent MCP $1.15 → $1.07 8/15 どの条件も、ラベル無しの方が安かった組は半分前後で、差は3回のばらつきの範囲に収まりました。Argent の CLI だけ2割安くなっていますが、ばらつきと区別しきれません。全条件を合わせた90組で見ても、トークンが少なかったのは41組、時間が短かったのは39組でした。 アイコンだけのリポストのボタンを長押しする #8502 でも、差は出ていません。エージェントは、名前の無いボタンをスクショや並び順から見つけていました。 なお、今回のお題はどれも、たどる画面が少ないものです。何画面もたどる操作では、結果が変わるかもしれません。また、Bluesky は testID とラベルが丁寧に付いたアプリで、ここで測ったのはそれを「外した」状態です。最初から付いていないアプリでは、画面の作りが違うかもしれません。 一番高い Argent の MCP が一番速い 作業時間は平均1分41秒で、6条件の中で一番短くなりました。同じお題どうしで並べると、30組中25組で agent-device の CLI より早く、平均で24秒の差です。 速さのわけは、やり取りの少なさです。タップのたびにスクショが返ってくるので、画面を見るためだけの呼び出しがほとんど要りません。やり取りは平均22回で、agent-device の CLI (26回) より4回少なく済みました。 その代わり、1回のやり取りの費用は agent-device の CLI の倍近く ($0.053 と $0.028) かかります。 まとめ 差が出たのは、バグを直せるかどうかではなく、画面をエージェントにどう渡すかでした。CLI で画面を短く読める agent-device と mobilecli は安く、操作のたびにスクショを返す Argent の MCP は高くつきました。新しいツールを入れるときも、CLI があるか、画面を何で返すか (文字の一覧か、スクショか) を見れば、費用の当たりはつけられます。testID やラベルは、少なくとも費用のためには足さなくて構いません。 まだエージェントの操作ツールを使ったことのない React Native の開発者は、まずは agent-device の CLI を試してみてください。 参考リンク callstack/agent-device (GitHub) software-mansion/argent (GitHub) mobile-next/mobile-mcp (GitHub) bluesky-social/social-app (GitHub)
Test Automation with agent-device (Part 2) --- title: "Test Automation with agent-device (Part 2)" published: true description: "Signing the iPhone runner (automatic and manual) and wireless Android debugging: how to put agent-device on real devices, and the traps." tags: claude, ai, flutter, testing canonical_url: krenin88.github.io --- agent-device drives real devices with the same commands it uses for simulators and emulators. A real iPhone, though, needs a signed test app installed on the device to drive the screen, and a real Android device has to be selected explicitly depending on how it is connected. This post covers why real devices were needed, how to prepare an iPhone and an Android device, and what to watch out for when handling apps on them. It is based on agent-device 0.21.31 (2026-10-24). Installation and the basic loop are in Part 1. Why real devices Simulators and emulators are enough to check app screens, but they cannot reproduce how a build shipped through the store behaves on a real user's device. Three things had to be checked on real devices. Store-build updates and reinstalls. Installing a TestFlight build over the existing App Store version (the update path), and deleting and reinstalling the app and then restoring a backup, can only be checked on a real device. Whether the backup file was overwritten was checked in the device's Files app under iCloud Drive, by comparing file size and modification time before and after the install. System dialogs on first launch. The order in which system dialogs such as App Tracking Transparency (ATT) appear relative to the app's own screens has to be seen on a real install. A reinstall also resets that consent history, so first work out whether the state you want to test can actually be produced on the device. Real ads. The Google Mobile Ads SDK automatically treats an emulator as a test device. logcat prints This request is sent from a test device., and even a release build with production ad unit IDs shows only test ads. That means a no-fill (the ad server finds no ad to serve) cannot be reproduced on an emulator either. On a real device that is not registered as a test device, logcat prints a setTestDeviceIds hint and real ads are requested. In that case, never tap the ad (invalid clicks), watch a production ad to the end at most once per session, and judge the rest from the ad-load results in logcat. Preparing a real iPhone On a real iPhone, agent-device installs an XCTest runner on the device. The XCTest runner is a test app that reads the screen and taps on your behalf, and it has to be signed with a development certificate to install. The first run takes a while because the runner gets built, and the cable has to stay connected throughout. Automatic signing. According to the agent-device docs, if an Apple developer account is signed in to Xcode, the team ID is all you need. To change the runner's bundle ID, also set AGENT_DEVICE_IOS_BUNDLE_ID. export AGENT_DEVICE_IOS_TEAM_ID=<team id> Manual signing. If Xcode has no account, or the existing development profile is Xcode-managed, automatic signing will not work. In that case, create a development profile for the runner yourself with an App Store Connect API key. fastlane's sigh creates a profile for a wildcard app ID (*) containing the registered devices. # key.json: key_id, issuer_id, key (contents of the .p8 file), in_house: false fastlane sigh --development --app_identifier "*" --team_id <team id> \ --api_key_path key.json --provisioning_name "<profile name>" \ --output_path <folder> --skip_install --skip_certificate_verification Putting the .p8 file path in key.json (key_filepath) failed; putting the file contents in key worked. Copy the generated profile to ~/Library/MobileDevice/Provisioning Profiles/<UUID>.mobileprovision, and delete the key file once you are done. Then pass the signing details as environment variables. The signing identity is the SHA-1 value shown by security find-identity -v -p codesigning, and the profile was specified by the name given to sigh. export AGENT_DEVICE_IOS_TEAM_ID=<team id> export AGENT_DEVICE_IOS_SIGNING_IDENTITY=<certificate SHA-1> export AGENT_DEVICE_IOS_PROVISIONING_PROFILE="<profile name>" export AGENT_DEVICE_STATE_DIR=<separate state folder> AGENT_DEVICE_STATE_DIR is the agent-device daemon's state folder. A separate one was set to keep this work apart from the default daemon used by other work. Bundling these variables and the working directory (cd) into one wrapper script runs every command in the same environment. Mac and device settings. Besides signing, three things have to be right for the runner to work. 1. Developer tools security on the Mac must be enabled. If it is off, the runner stops at Developer mode is disabled for Apple development tools. This is a Mac-side setting, separate from Developer Mode on the device; run sudo DevToolsSecurity -enable with admin rights. 2. iPhone Settings → Developer → "Enable UI Automation" must be on. If it is off, the runner installs and the test starts, but nothing responds. 3. Keep the screen on and unlocked. When the runner timed out at "Starting XCTest runner", the device was locked. A retry after unlocking never happened, so the cause is only circumstantial, but checking the lock state first with xcrun devicectl device info lockState --device <UDID> is the safer habit. Handling apps on an iPhone Listing and deleting installed apps on a real device is done with Xcode's devicectl. xcrun devicectl list devices xcrun devicectl device info apps --device <UDID> xcrun devicectl device uninstall app --device <UDID> <bundle-id> The runner is removed the same way. Its bundle ID is com.callstack.agentdevice.runner.uitests.xctrunner, and running open after deleting it builds and installs the runner again. When installing a test build from TestFlight while the App Store version of the same app is already installed, a warning says "You may lose the app's data." The Install button in that warning overwrites the app. It is safer to have the agent stop at this warning and ask a human. Deleting an app wipes its on-device data. If the app has a backup, take a screenshot showing that backup is on in the app's settings and that the last backup time is just now before deleting; otherwise, stop instead of deleting. One more screen-handling note: tapping a toggle in a settings screen by ref sometimes hits the middle of the row and does not flip it. In that case, use the element's rect from snapshot --json and tap the right edge of the toggle by coordinates. Connecting a real Android device A real Android device connected over USB debugging behaves exactly like an emulator. With wireless debugging, the device may not show up in adb devices right after connecting. It appears only after the _adb-tls-connect service shows up in adb mdns services, so do not conclude it is missing; check again a few seconds later. A wireless serial is a long mDNS name and easy to mix up with an emulator. Add --serial to every adb and agent-device command. agent-device sessions are created per working directory, so if an emulator was already in use from the same directory, you get an "already bound to android device" error. In that case, open a separate session with --session <name>. adb mdns services agent-device open <bundle-id> --platform android --serial <serial> --session <name> --foreground agent-device close --session <name> adb disconnect <serial> Three things commonly go wrong on Android: - Quote adb shell commands as one string. Unless the whole thing is wrapped, as in adb shell "cmd notification post -t 'title' <tag> 'body'", adb splits the arguments on spaces. - Buttons hidden by the keyboard are not found. Close the keyboard with adb shell input keyevent KEYCODE_BACK and snapshot again. - Quote selectors that contain spaces. Use the label="..." form; passing flags as one variable blob gives "Unknown flag". When something other than the app, such as an interstitial ad, is on top, a snapshot alone makes it hard to tell what state the app is in. Check the system state with adb as well. adb -s <serial> shell dumpsys activity activities | grep topResumedActivity # what is in front right now adb -s <serial> shell pidof <package> # is the app process alive adb -s <serial> logcat -d | grep -E "FATAL|AndroidRuntime" # traces of a crash Match the snapshot against whether the front screen is the ad or the app, whether the app is still alive behind it, and whether a crash left a record. Ad screens put many webview and link nodes into the accessibility tree, so filter those out when reading a snapshot. References - agent-device repository: <callstack/agent-device> - agent-device agent setup docs: <oss.callstack.com> - fastlane sigh docs: <docs.fastlane.tools> - Android wireless debugging: <developer.android.com>
新時代のE2Eテスト、TesterArmyのe2eをReact Nativeアプリに導入してみた こんにちは！テラーノベルでiOS/Android/Webとフロントエンド周りを担当している@kazutoyoです！ 以前、Coding AgentにiOS/Androidアプリの「目と手」を与えるツールとしてagent-deviceを紹介しました。 zenn.dev 今回は、そのagent-deviceをエンジンに使っている、TesterArmyのE2Eテストフレームワーク e2e を紹介します。 AIキャラクターとチャットするアプリのめろ（Expo / React Native）に最近導入して、PRでシナリオテストを回しています。 mero.ist e2eは、ゴールを自然言語で書くとエージェントがアプリを操作してくれる、いわゆるAIテストツールです。 ただ、全部をAIに任せるツールではなく、AIに任せる部分と、従来どおりロケーターで確かめる部分を1本のテストの中で書き分けられるのが特徴です。 この記事では、e2eの紹介と既存ツールとの比較、めろでの使い方をまとめます。 e2eとは e2eはTesterArmyが開発している、Webとモバイルアプリ向けのE2Eテストフレームワークです。 npmのパッケージ名はそのまま e2e（SDK・ランナー・CLI） エンジンは別パッケージになっている @e2e-dev/web: Playwright経由でChromium/Firefox/WebKitを動かす @e2e-dev/mobile: agent-device経由でiOS Simulator/Android Emulatorを動かす ほかに、結果をPRにコメントするレポーター（@e2e-dev/github）や、クラウドの端末・ブラウザのプロバイダー（EAS Simulators、Limrun、Kernelなど）がある テストはこのように書きます（公式のREADMEより、コメントは筆者が追加）。 import { test, expect } from 'e2e'; // テスト名: 「メンバーがProプランにアップグレードできる」 test('a member upgrades to Pro', async ({ app, agent, screen }) => { // 支払い設定の画面を開く await app.open('/settings/billing'); // 「ワークスペースをProプランにアップグレードする」をゴールとして渡し、操作はエージェントに任せる await agent.act('upgrade the workspace to the Pro plan'); // 「請求のプレビューに日割りの金額が出ている」かをAIに判定させる await agent.assert('the invoice preview shows a prorated amount'); // 最後はロケーターで、ステータスに「Pro」と出ていることを確かめる await expect(screen.getByRole('status')).toContainText('Pro'); }); 画面を開いたら、プランを変える操作はAIに任せ、結果をAIとロケーターの両方で確かめる、という流れです。 e2e.tester.army 3種類のステップ e2eのテストは、次の3種類のステップを組み合わせて書きます。 ステップ API 何をするか モデル呼び出し ゴール agent.act 「ストーリーを1つ開く」のようなゴールを渡すと、エージェントが画面を読んでタップや入力をする あり（キャッシュがあれば無し） AIによる判定 agent.assert / agent.waitFor / agent.extract 「入力欄が画面の下端にある」のような画面の状態をAIが判定する あり ロケーター screen.getByRole(...) / expect(...) Playwrightと同じ感覚で、決まった仕様を確かめる なし エージェントは画面のアクセシビリティツリーを読んで操作します。 agent.assert に { vision: true } を付けると、画面のピクセルも見せて判定させられます。 e2e.tester.army リプレイキャッシュ agent.act の操作は、そのあとのロケーターの検証や agent.assert が通ったときだけ記録されます。 次の実行からはその記録を再生するだけなので、モデルを呼びません。 UIが変わって記録どおりに操作できなくなったときは、その場面からエージェントが引き継ぎます。 AI 4.1k tokens · 2 model calls · openai/gpt-6-luna-fast Cache 4 replayed · 1 handed off · 1 missed 実行の最後には、このようにキャッシュの結果が出ます。 replayed は記録の再生だけで終わったステップ、handed off は再生の途中からエージェントが引き継いだステップ、missed は記録が無く最初からエージェントが操作したステップです。 引き継いだ場合も、エージェントがゴールを達成できれば、テストはそのまま続きます。 AIテストは毎回トークンを使い、毎回動きが変わるのが気になるところですが、検証が通った操作だけを記録しておく仕組みでそこを抑えています。 ただし、キャッシュされるのは agent.act の操作だけです。 agent.assert などのAIによる判定は毎回モデルを呼ぶので、その分のコストは実行のたびにかかります。 e2e.tester.army ChatGPTなどのサブスクでも動かせる エージェントのモデルには、AI SDKのプロバイダーをそのまま渡せます。 APIキーのほかに、ChatGPTやGitHub Copilotなどのサブスクに npx e2e login openai でログインして使うこともできます。 めろでは、OpenAIの低価格なモデルのGPT-6 Lunaを使っていて、手元ではChatGPTのサブスク、CIではOpenRouter経由のAPIキーで呼んでいます。 e2e.tester.army 他のE2Eテストツールとの比較 代表的なE2Eテストのツールと比べると、次のようになります。 Maestro Detox Appium e2e 対象 iOS/Android/Web React Native iOS/Androidほか Web + iOS/Android 書き方 YAML JS/TS 各言語 TS + 自然言語 アプリへの組み込み 不要 必要 不要 不要 要素の指定 テキスト / id testIDなど セレクタ ロケーター or 自然言語 AIによる操作 なし なし なし あり（agent.act） AIによる判定 あり（assertWithAI など） なし なし あり（agent.assert など） 大きな違いは、アプリへの組み込みが要らないことと、判定だけでなく操作までAIに任せられることかなと思います。 Maestroとの違い MaestroはYAMLのフローで手軽に書けます。 repeat や条件付きの runFlow があり、runScript / evalScript でJavaScript（HTTPクライアントも使えます）を実行できるので、ロジックも書けます。 AIについても、assertWithAI や extractTextWithAI で画面の判定や読み取りができます。 e2eとの違いは2つあります。 1つは、テスト全体をTypeScriptで書く点です。ヘルパー関数や型、Vitestでのユニットテストなど、アプリと同じ道具がそのまま使えます。 もう1つは、AIに判定だけでなく操作も任せられる点です。Maestroでは操作はコマンドで書きますが、e2eでは agent.act にゴールを渡すと、エージェントが操作を決めます。 Detoxとの違い Detoxはアプリの内部と同期するグレーボックスのテストなので安定しますが、アプリ側にDetoxを組み込む必要があります。 e2eはアクセシビリティツリーを外から読むので、いつものDevClientのビルドをそのままテストできます。 React Nativeの testID は、iOS/Androidのどちらでも getByTestId で引けます。 自然言語だけで書くAIテストツールとの違い 自然言語だけでシナリオを書くツールは、書くのは楽な一方で、何を確かめたのかが曖昧になりやすいです。 e2eのドキュメントでは、各ゴールの結果を expect で確かめることや、最後の判定をロケーターで決定的に書くことを勧めていて、AIは変わりやすい操作の部分を受け持つ位置づけです。 e2eの弱いところ AIのステップにはコストと待ち時間がかかるので、キャッシュが効く書き方にする必要がある アクセシビリティツリーに出ないもの（OSのFace IDのダイアログや、Androidでツリーに出ない要素など）は、visionに頼るか工夫が要る まだ0.xで、APIの変化が速い めろでの活用例 ここからは、めろでの使い方を、どんなテストをしたいときにどう書いているかでまとめます。 めろは生成AIのチャットアプリなので、おすすめの一覧やサーバーが返すフォームなど、毎回中身が変わる画面が多いです。 そうした画面の操作や判定はAIに任せ、仕様として決まっていることはロケーターで確かめる、という書き分けを基本にしています。 アプリ固有の用語をエージェントに伝えたいとき エージェントの設定の context に、アプリの説明や用語を書いておけます。 export default { targets: [ { name: "ios", engine: mobile({ platform: "ios" }), app: { bundleId: "..." } }, { name: "android", engine: mobile({ platform: "android" }), app: { bundleId: "..." } }, ], agents: { default: { // 手元では ChatGPT のサブスク、CI は API キー model: process.env.CI ? openrouter("openai/gpt-6-luna") : chatgpt("gpt-6-luna"), system: "You are a thorough QA agent. Verify every outcome on screen.", context: "mero は AI キャラクターとチャットする日本語のアプリ。画面の文言は日本語。「ストーリー」はキャラクターとの会話の題材、「トーク」はキャラクターとのチャット。", }, }, } satisfies E2EConfig; アプリの中だけで通じる言葉をここで教えておくと、ゴールの文を短く書けるようになりました。 並びが毎回変わる一覧から、どれか1つを開きたいとき おすすめの一覧のように並びが毎回変わる画面では、「何番目をタップする」とは書けません。 こういう操作は agent.act に任せて、開いた先の画面はロケーターで確かめています。 // どれを開くかはエージェントに任せる await agent.act("ホームの一覧からストーリーを 1 つ開く"); // 開いた先の画面は、ボタンの表示で確かめる await expect(screen.getByRole("button", /^トークを(はじめる|続ける)$/)).toBeVisible(); 一覧からagent.actで1つ開き、会話を始める レイアウトの崩れのように、ロケーターでは確かめにくい見た目を確かめたいとき チャットの画面で、キーボードを開いたままAIの返信が届き、そのあとキーボードを閉じると、入力欄が浮いたまま、一覧の下が真っ黒、ヘッダーが隠れたまま、という不具合が以前ありました。 こうした見た目の崩れは、要素がツリーにあるかどうかでは分かりません。 入力欄が下に戻ったことは位置をロケーターで確かめ、画面全体の見た目は agent.assert に { vision: true } を付けて、画面を見て判定させています。 // 入力欄の位置を測る const inputTop = async () => (await screen.getByTestId("talk-input").boundingBox())?.y ?? 0; const openedTop = await inputTop(); // 一覧をタップしてキーボードを閉じると、入力欄が下に戻る await screen.getByTestId("talk-message-list").tap(); await expect.poll(inputTop).toBeGreaterThan(openedTop); // 画面全体が崩れていないかは、画面を見て AI に判定させる await agent.assert( "キーボードは閉じていて、上部に戻るボタンとストーリー名のヘッダーが見え、入力欄は画面の下端にあり、一覧に真っ黒な領域は無い", { vision: true }, ); サーバー側で中身が変わるフォームを、最後まで進めたいとき めろには、質問に答えていくだけでAIキャラクターを作れる「かんたん作成」という機能があります。 相手の性別、キャラクター、言葉づかい、舞台といった質問に、候補から選んで答えていく形です。 この質問や選択肢はサーバーが返していて、運営側の設定で変わります。 そのため、質問ごとにテストを書くとすぐに壊れてしまいます。 回答はエージェントにまとめて任せて、確認画面に着いたことだけをロケーターで確かめています。 await agent.act( "各ステップでは、すべての質問について候補のボタンを 1 つずつ押して答えてから「次へ」を押す。確認の画面（「これで作る」ボタンがある）まで進む。「これで作る」は押さない", ); await expect(screen.getByRole("button", "これで作る")).toBeVisible({ timeout: 30_000 }); await agent.assert("確認の画面に、回答が表示されている項目が 1 つ以上ある", { vision: true }); そのあとの「編集して保存すると確認画面に戻る」「一時保存して閉じると、続きから再開できる」といった遷移は、ロケーターで書いています。 フォームの中身が変わってもテストは壊れず、確かめたい仕様は固定できます。 agent.actが質問に答えて辿り着いた確認画面 生体認証やバックグラウンドからの復帰を試したいとき @e2e-dev/mobile の device フィクスチャで、生体認証や権限、ホームボタン、アプリの切り替えも操作できます。 Face IDでのアプリロックは、次の流れで確かめています。 device.enrollBiometrics("faceid", true) でFace IDを登録する 設定からロックを有効にする device.home() から device.openApp() でアプリに戻り、ロックをかける device.setBiometrics("faceid", "nonmatch") で認証に失敗させ、ロックが外れないことを確かめる 再試行で "match" を返して解除する OSのダイアログはアプリのツリーに出ないので、Face IDの結果がアプリに反映されるまで送り直すヘルパーを用意しています。 送り直すのは20秒までで、それを過ぎても反映されなければテストを失敗にしているので、失敗を隠すことはありません。 アナリティクスのイベントが正しく送られたかも確かめたいとき これまでは、アナリティクスのイベントが送られているかを、実装のたびに手で確かめていました。 e2eでテストを書くついでに、イベントも同じテストで確かめられないかと思って試したのがこの仕組みです。 e2eのテストは普通のTypeScriptなので、テストの外の仕組みとも組み合わせられます。 めろでは、開発中にJSのバンドルを配信しているReact NativeのMetroサーバーに、ミドルウェアで /__e2e/events という受け口を足して、開発ビルドのアプリがアナリティクスのイベント（Firebase Analyticsなどに送るユーザーの行動ログ）を送るたびにそこへPOSTするようにしています。 テストからはそれを読んで、操作のあとに期待したイベントが送られたか、イベントのパラメータが正しいかを確かめています。 Metroサーバーと同じポートを使うので、CIでポートを変えても、Androidエミュレータ（10.0.2.2）からでもそのまま届きます。 アプリを一通り見て回って、画面の一覧を作りたいとき e2eには、ゴールだけを渡してアプリを探索させる e2e explore があります。 めろでは、この e2e explore を使って、画面マップを自作しています。 画面マップはe2eの機能ではなく、e2eの実行結果を元に自分たちで作っている社内のドキュメントです。 ホームや検索、詳細画面といった領域ごとに e2e explore で探索させて、画面ごとのスクリーンショットと送られたアナリティクスのイベントを集め、自作のスクリプトで画面と遷移をつないでいます。 社内のドキュメントサイトで画面を選ぶと、その画面の説明や仕様テストの内容、探索で見たことが横に出ます。 stagingのデータを変えないよう、探索には別のエージェントを定義して、context で行動を絞っています。 見るだけにすること。してよいのは、画面を開く・戻る・タブの切り替え・スクロール… してはいけないこと: メッセージの送信、購入や課金のボタンを押す… e2e.tester.army PRごとに回して、結果をレビューで見たいとき GitHub Actionsでは、次の流れで回しています。 @expo/fingerprint でネイティブ部分のハッシュを取り、同じハッシュのDevClientのビルドがキャッシュにあればダウンロード、なければビルドしてアップロードする Metroサーバーを起動して pnpm test:e2e を実行する @e2e-dev/github のレポーターが結果をPRに1件のコメントとして貼る（再実行すると同じコメントを更新する） レポートやスクリーンショットをアーティファクトとして保存する JSだけの変更ならネイティブのビルドを飛ばせるので、実行時間のほとんどはテストそのものです。 PRに付くコメントは次のようになります。 この回は全体で7分22秒かかり、AIを使ったのは5ステップ、モデルの呼び出しは12回、費用は$0.0041でした（モデルはGPT-6 Luna）。 agent.assert の判定は毎回モデルを呼ぶので、費用はキャッシュが効いても0にはならず、記録が使えない回はもう少し増えます。 e2e.tester.army 今はすべてのテストを回していますが、今後はe2eのテストフィルターも取り入れたいと考えています。 PRで --changed-since を付けて実行すると、Jevのような決定モデルがPRの差分とテストを読んで、その変更で影響を受けそうにないテストを外してくれます。 テストが増えても、PRで回すテストの数と時間を抑えられそうです。 e2e.tester.army Coding Agent向けのスキル e2eには、Claude CodeやCodex、CursorなどのCoding Agent向けのスキルが、現在3つ用意されています。 スキル 何をするか e2e テストを書く・実行する・失敗の原因を調べる。e2e mcp で起動中のアプリを操作して、画面を確かめたりロケーターを試したりもできる create-verification-skill プロジェクト専用の verify-<アプリ名> スキルと機能の一覧を作り、Coding Agentが自分の変更をユーザーと同じ操作で確かめられるようにする migrate-to-e2e Playwright・Cypress・Selenium・Detox・Maestroのテストを少しずつe2eに移す。進み具合は MIGRATION.md に残るので、次のセッションで続きから再開できる e2e のスキルは npx e2e init で .agents/skills/ と .claude/skills/ に入ります。めろでもこれで入れています。 3つまとめて入れたい場合は、Claude Code・Codex・Cursor向けのプラグインがあります（Claude Codeの例）。 claude plugin marketplace add tester-army/e2e claude plugin install e2e@e2e e2e.tester.army バグバッシュ e2e のスキルには、バグバッシュの手順も入っています。 「このブランチのチェックアウトとアカウント設定をバグバッシュして」のように頼むと、Coding Agentが次の流れで進めてくれます。 アプリの画面とブランチの差分を読んで、探索の目標を5〜10個立てる 目標ごとに e2e explore を走らせる（4つずつ並列） 見つかった問題のうち、探索の限界や意図した仕様によるものを外す 残った問題ごとに再現テストを書いて実行し、テストが失敗したものだけをバグとして報告する AIが見つけた問題をそのまま報告するのではなく、再現テストで確かめてから報告してくれるのが良いところです。 めろではまだ試せていませんが、リリース前の確認などで使ってみたいと思っています。 e2e.tester.army 導入してみて よかった点 変わる部分と変わらない仕様を分けて書ける。おすすめの一覧やサーバー駆動のフォーム、ロケーターでは確かめにくい見た目など、めろには従来のE2Eと相性の悪い部分が多いですが、そこをAIに任せつつ、確かめたい仕様はロケーターで固定できます 普通のTypeScriptで書ける。ヘルパーをVitestでテストしたり、アナリティクスのイベントの検証や画面マップの生成を自分たちで足したりと、アプリの開発と同じ感覚で広げられます アプリに何も組み込まなくていい。いつものDevClientのビルドをそのまま使えます Coding Agentに書かせやすい。e2eはパッケージにドキュメントとエージェント向けのスキルを同梱しているので、Claude Codeに「この機能のe2eを書いて」と頼むと、規約に沿ったテストを書いてくれます 気をつけている点 1つのゴールに1つの結果。agent.act に複数のことを頼まず、ゴールのあとには必ず expect を置きます（キャッシュもこれが前提です） 状態を残さない。下書きやロックの設定など、前の実行の残りで次のテストが変わらないよう、テストの最初と最後で片付けます プラットフォームの差は書いておく。Androidではツリーに出ない要素があるので、確かめられない部分はコメントに書いてiOSだけで確かめています まとめ e2eは、AIを使うところを人が選び、最後の判定はロケーターで決定的に行う設計なので、実際の運用に乗せやすいと感じています。 npx e2e init このコマンドでWeb/モバイルのどちらかのエンジンとモデルを選ぶだけで導入でき、設定とサンプルのテストもできあがります。 Maestroなどからの移行用のスキル（migrate-to-e2e）も用意されているので、既存のMaestroのフローがあれば、それを元に試すこともできます。 e2e.tester.army 生成AIを使ったアプリのように、出力が毎回変わる画面のE2Eに悩んでいる方は、まずは触ってみることをおすすめします！ それでは良いReact Nativeライフを！ 参考リンク e2e（GitHub） e2eドキュメント Core concepts Replay cache Subscriptions Test filters Coding agents Bug bash Maestroからの移行ガイド agent-device（GitHub） Rozeniteとagent-deviceを紹介した過去記事
Test Automation with agent-device (Part 1) --- title: "Test Automation with agent-device (Part 1)" published: true description: "Why agent-device for Flutter app testing, how to install it for Claude Code, the basic command loop, and Flutter gotchas." tags: claude, ai, flutter, testing canonical_url: krenin88.github.io --- agent-device is an open-source tool from Callstack that lets an AI agent drive iOS and Android apps and check what is on screen. This post covers why it was picked for testing a Flutter app, how to install it for Claude Code, the basic command loop, and what to watch out for with Flutter. It is based on version 0.21.31 (2026-10-24). Why agent-device Flutter ships Android and iOS apps from one codebase, so the device-testing tool has to cover iOS too. Google Artemis, the first candidate, is a natural-language testing agent, but it supports Android only; iOS is still just a roadmap item. The other candidates looked like this: - mobile-mcp exposes taps, screenshots, and the accessibility tree on iOS and Android over MCP (the protocol an AI uses to call external tools). It has no sessions, replay, or evidence storage, so a verified flow is hard to hand off to CI. - Maestro is an E2E framework that pins flows in YAML, and it has an MCP server. It suits regression runs of fixed flows better than an agent exploring the screen. - Arbigent is an AI agent built on top of Maestro. Its author rates it 1/5 for speed and 3/5 for reliability, and its YAML format was still changing. - Commercial services (Drizz, Minitap, and others) bundle a real-device farm with a natural-language agent, but the only public material was vendor material, so cost and quality could not be checked. agent-device handles iOS and Android simulators, emulators, and real devices with one tool, and its docs state Flutter support explicitly. It ships both a CLI and an MCP server, can save a verified flow as an .ad script to replay later, and can export to Maestro YAML. It is MIT-licensed. Installation Install the CLI globally and check the environment. npm i -g agent-device@latest agent-device doctor doctor checks connected devices, toolchains, and dev-server status. In this setup it passed with only warnings for unused platforms (HarmonyOS, Vega). Register it with Claude Code as an MCP server. With -s user, the same tool is available in every project. claude mcp add -s user agent-device -- agent-device mcp The agent-device repository includes three agent skills: agent-device, ios-simulator, and android-emulator. The official instruction is npx skills add callstack/agent-device, but Claude Code's auto mode safety check blocked it. In that case, clone the repository and copy the three folders under skills/ into ~/.claude/skills/. To undo everything, run claude mcp remove agent-device -s user and npm uninstall -g agent-device, then delete the three skill folders. The basic loop The basic loop is: prepare a device, open the app, act, verify, close. Pick the device with --platform and --device. agent-device boot --platform android --device "<emulator name>" --headless agent-device install /absolute/path/app-debug.apk --platform android --device "<emulator name>" agent-device open <bundle-id> --platform android --device "<emulator name>" --foreground agent-device press @e5 --settle agent-device fill @e11 "text" --settle agent-device wait text "<expected text>" agent-device screenshot /absolute/path/home.png agent-device close open reads the screen and returns a ref such as @e5 for each element, like buttons and text fields. With --settle, press and fill wait until the screen goes quiet and return only what changed, so the agent can pick its next target without rereading the whole screen. Three things commonly go wrong here: - Refs go stale when the screen changes. Pick a new ref from the diff returned by each action. - A screenshot is not verification. Verify with wait text or wait <selector>, and keep screenshots as evidence only. - Give install an absolute path. A relative path resolves against the agent-device daemon's working directory and fails with "App binary not found". For a Flutter app, build the iOS simulator app with flutter build ios --simulator --debug and the Android app with flutter build apk --debug. Flutter gotchas agent-device reads the screen through the accessibility tree. In Flutter, that means only widgets with text show up by default; icon buttons and containers do not. Add Semantics(identifier:) or semanticLabel to those widgets so a selector can find them. Flutter's Key does not appear in the accessibility tree. Labels in the accessibility tree follow the language the device is set to, so the same app reads with different label text depending on that setting. A text-based selector changes with the device language, which makes identifier the more stable choice. On iOS, AdMob test banners show up as platform_view, webview, and link nodes. Selectors in replay scripts should avoid those nodes. References - agent-device repository: <callstack/agent-device> - agent-device agent setup docs: <oss.callstack.com> - Google Artemis: <google/artemis> - Maestro Flutter support docs: <docs.maestro.dev>
Agents that can see and tap your app are a big part of what we cover in The Agentic Rewind 👇 Join The Agentic Rewind: trnrw.com agent-device: callstack/agent-device
𝐀𝐧𝐝𝐫𝐨𝐢𝐝: 𝚌𝚛𝚘𝚙𝙸𝚖𝚊𝚐𝚎 now shares Fresco's disk cache with Image.prefetch and <Image>. Prefetched? No second download. Cache miss? Saved to the cache, so a later <Image> skips the network. New example app on RN 0.87 too. Release notes: github.com/callstack/react-native-image-editor/re…
Fallback for federated remotes: the copy in the binary and the net in the session > 📚 React Native Module Federation series — read it in full on warrendeleon.com, where new parts land first. Post 15 ended its break-it section on its own limit: "None of this gives the binary anywhere else to load from: a dead tab is honest, and it is not a working app." This post gives the binary somewhere else. Every release build now carries a copy of each remote, and the app runs from those copies when the content delivery network (CDN) cannot be reached, or when one remote fails to load while the other keeps loading from it. Post 1 said who builds that: "Module Federation gives you none of the offline story by itself: shipping a copy of each remote inside the binary, so the reviewed app works on its own with no network, is an architecture you build." It also set the requirement for a remote that will not load: "the app has to degrade to something safe instead of showing a blank screen." Post 15's build meets the requirement for a remote that will not load, and stops there. With the CDN out of reach it launches unresolved, with an error state in either tab. When one remote fails, its tab shows the error state, and Try again never brings the tab back. The first half builds the copy: a build step that stages it, a phase on each platform that puts it in the binary, and a new launch mode, bundled, that runs every remote from it. The second half builds two nets for a CDN launch. One catches a manifest the CDN cannot serve and drops that one remote to its copy; the other sits around each tab and catches a load that fails later. A Try again that actually retries comes with the second net. One rule shapes the copy: it is the version already known to run on this binary, frozen on the day the binary was built. It is the guaranteed minimum: new versions still reach installed apps through the CDN, and the host uses the copy only when the CDN cannot deliver a version that loads. Start from post 15's finished state, the tag post-15-cdn-flip; this post finishes at post-16-fallbacks. As before, you type the configuration changes and the smaller edits, and copy the rest from the end tag: the build tool, the host's code and tests, the native files and one design-system test. > 📊 Diagram: view it on warrendeleon.com The launch decides the mode once, before anything federated loads. Using the CDN's versions means a usable map arrived and its versions registered; when they cannot be used, the launch registers the copies instead. It ends unresolved only when that fails too. The two nets act inside a CDN launch, one remote at a time. Put a copy of each remote in the binary A copy is chosen by the same map that picks versions from the CDN. The version a binary carries is the one its own app version's map names, because that is the version known to run on that binary: for the 2.0.0 build here, list 1.2.0 and party 1.0.0. The copy holds that version's signed bundles and its manifest, byte for byte as the CDN serves them. The bytes stay as they are because every bundle is still verified against the signing key when it loads, and a change to its code fails the check. tools/build-cdn.mjs already builds every version into cdn-root/, and it now stages the copies as well. Fetch the end tag once, since every copy command in this post takes its files from it, and take the tool first: npx degit@3.8.0 --force warrendeleon/react-native-module-federation#post-16-fallbacks /tmp/pokedex-ref-16 cp /tmp/pokedex-ref-16/tools/build-cdn.mjs tools/ The new stage runs after the CDN tree is built. For each platform it takes every version the map names and copies its bundles and manifest into embed-root/<platform>/<remote>/<version>/: for (const [remote, version] of Object.entries(embeddedVersions)) { const published = join(cdnDir, remote, version); const embedded = join(embedDir, remote, version); mkdirSync(embedded, { recursive: true }); // Every script of the version sits at the top of its directory with its manifest, and the copy // is all of them, byte for byte: the container, its chunks, index.bundle, which the manifest // names among the shared modules' assets, and mf-manifest.json, which the federation runtime // reads from the disk the way it reads it from the CDN. The assets/ folder beside them holds // images the host's shared copies of those libraries already carry. for (const file of readdirSync(published)) { if (file.endsWith('.bundle') || file === 'mf-manifest.json') { copyFileSync(join(published, file), join(embedded, file)); } } bundledVersions[platform][remote] = version; The copies keep the <remote>/<version>/ directories instead of sharing one folder, because two remotes can ship vendor chunks with the same file name. In one folder, the second would overwrite the first. The build tool also records the copied versions in apps/host/src/shell/embedded-versions.ts, a generated file the host compiles in, so the host knows which copies it carries before it reads anything from the disk. embed-root/ is build output, like cdn-root/, so add it to .gitignore: # The copies build-cdn stages for the native embed phases. embed-root/ Run the key generator first, then the tool for both platforms. The generator keeps any keys the checkout already has, and creates them on a fresh checkout of the tag, which has none because they never enter Git. After the CDN tree, the tool reports the copies and the generated file: node tools/gen-signing-keys.mjs && node tools/build-cdn.mjs embedded -> embed-root/ios/listApp/1.2.0 embedded -> embed-root/ios/partyApp/1.0.0 embedded -> embed-root/android/listApp/1.2.0 embedded -> embed-root/android/partyApp/1.0.0 wrote -> apps/host/src/shell/embedded-versions.ts The generated file records the version of each copy, per platform: export const BUNDLED_VERSIONS: Record<string, Record<string, string>> = { ios: { listApp: '1.2.0', partyApp: '1.0.0' }, android: { listApp: '1.2.0', partyApp: '1.0.0' }, }; The build tool stages the copy for the app version in MF_APP_VERSION, or for the newest app version with a map when that is unset, so a build for another app version runs the tool with the same variable first. iOS: the last build phase On iOS the copy goes inside the .app, next to main.jsbundle. A script does the copying, and a Run Script phase at the end of the Host target runs it. Copy the script from the end tag: mkdir -p apps/host/scripts && cp /tmp/pokedex-ref-16/apps/host/scripts/embed-remotes-ios.sh apps/host/scripts/ #!/usr/bin/env bash # --- The copy in the binary, iOS's half. The last Run Script phase of the Host target copies each # remote's embedded version from embed-root/ios into the app, at cdn/ios/<remote>/<version>/ next # to main.jsbundle, where the host reads it through absolute file:// URLs: the signed bundles and # the version's mf-manifest.json. # # The <remote>/<version>/ directories are kept rather than flattened: two remotes can ship vendor # chunks with the same file name, and a flat copy would let one overwrite the other. # # It runs in every configuration. A Debug build loads the host's own bundle over http, so it cannot # read the copy, and there the copy only costs the time it takes. Its remotes come from the dev # servers when no CDN is configured, or from the CDN when one is. # # With embed-root missing it removes any copy an earlier build left behind, prints a warning and # exits 0, and the build succeeds with nothing embedded. That is the trap: run # `node tools/build-cdn.mjs` before a release build. --- set -euo pipefail REPO_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd) SOURCE="$REPO_DIR/embed-root/ios" : "${CONFIGURATION_BUILD_DIR:?run this from an Xcode build phase}" : "${UNLOCALIZED_RESOURCES_FOLDER_PATH:?run this from an Xcode build phase}" DEST="$CONFIGURATION_BUILD_DIR/$UNLOCALIZED_RESOURCES_FOLDER_PATH/cdn/ios" if [ ! -d "$SOURCE" ]; then rm -rf "$DEST" rmdir "$(dirname "$DEST")" 2>/dev/null || true echo "warning: $SOURCE does not exist, so no remotes are embedded. Run 'node tools/build-cdn.mjs' before a release build." exit 0 fi mkdir -p "$DEST" # --delete keeps the app to exactly what embed-root holds, so a version an earlier build embedded # does not linger. The bytes are copied as they are: each bundle is signed over them. rsync -a --delete --include='*/' --include='*.bundle' --include='mf-manifest.json' --exclude='*' "$SOURCE/" "$DEST/" echo "embedded $(find "$DEST" -name '*.bundle' | wc -l | tr -d ' ') remote bundles and $(find "$DEST" -name 'mf-manifest.json' | wc -l | tr -d ' ') manifests into $DEST" The phase is one change to the project file. In Xcode it is a new Run Script phase on the Host target, named "Embed federated remotes (offline fallback)", running "$SRCROOT/../scripts/embed-remotes-ios.sh", placed after every other phase, with "Based on dependency analysis" unchecked. It goes last so that the .app it writes into is already assembled. With the box unchecked it runs on every build, which a phase that declares no outputs does anyway: Xcode skips a script phase only when it has outputs to check. Declaring the copied files under the .app's cdn/ios as outputs, with embed-root/'s files as inputs, is the alternative. The end tag's project file carries the phase with the box unchecked, so copying it makes the same change: cp /tmp/pokedex-ref-16/apps/host/ios/Host.xcodeproj/project.pbxproj apps/host/ios/Host.xcodeproj/ Android: a Gradle task and a native module On Android the copy goes into the assets of the APK, the Android package an app is installed from. At the end of apps/host/android/app/build.gradle, after post 15's changes, add: // --- The copy in the binary, Android's half. build-cdn stages each remote's embedded version under // embed-root/android, the signed bundles and the version's mf-manifest.json, and this task mirrors // that tree into the APK's assets at cdn/android, where EmbeddedRemotesModule copies it out at // launch. Sync rather than Copy, so a file an earlier build embedded and this one does not is // removed instead of shipped. // // With embed-root missing, this task embeds nothing, warnIfNothingToEmbed prints a warning, and the // build still succeeds. That is the trap: a successful release build that carries no copies. Run // build-cdn before a release build. --- def embedSource = file("$rootDir/../../../embed-root/android") def embeddedAssetsDir = layout.buildDirectory.dir("generated/embedded-remotes").get().asFile def embedRemotes = tasks.register('embedRemotes', Sync) { from(embedSource) { include '**/*.bundle', '**/mf-manifest.json' } into(file("$embeddedAssetsDir/cdn/android")) } // --- A Sync with no source skips its actions, so a warning inside it would never print. Gradle // still clears what the task embedded before when its source goes, so no stale copy ships. This // task has no inputs or outputs, so it runs on every build, and it prints the warning when // embed-root/android is missing. --- def warnIfNothingToEmbed = tasks.register('warnIfNothingToEmbed') { doLast { if (!embedSource.exists()) { logger.warn("warning: ${embedSource} does not exist, so no remotes are embedded. Run 'node tools/build-cdn.mjs' before a release build.") } } } android { sourceSets { main { assets.srcDirs += embeddedAssetsDir } } } tasks.named('preBuild').configure { dependsOn embedRemotes, warnIfNothingToEmbed } An asset is not a file on disk, and Re.Pack reaches a copy's <remote>/<version>/ directory only through a real file path. So a small TurboModule copies assets/cdn out to the app's files directory and resolves with the directory it used. Its spec is short enough to read whole: import type { TurboModule } from 'react-native'; import { TurboModuleRegistry } from 'react-native'; // --- Android's half of the copy in the binary. A Gradle task packs each remote's embedded version // into the APK's assets, and assets are not files on disk: Re.Pack's file:// loader needs a real // path. prepare() copies them out to the app's files directory once per installed build, keeping // the <remote>/<version>/ layout, and resolves with the directory the host builds file:// URLs // against. // // There is no iOS implementation, because iOS reads the copies straight from the .app. That is why // this uses TurboModuleRegistry.get, which returns null where the module does not exist, rather // than getEnforcing, which would throw on iOS the moment this file loads. --- export interface Spec extends TurboModule { /** Copy the embedded remotes out, once per installed build, and return the directory. */ prepare(appVersion: string): Promise<string>; } export default TurboModuleRegistry.get<Spec>('EmbeddedRemotesModule'); The Kotlin side copies the tree once per install. A marker file holds the app version and the time the APK was installed or updated, so a relaunch reuses the extracted tree, and any new install replaces it, including a rebuilt APK that kept its version number. It copies the bytes as they are, since each bundle's signature covers them. Take the module, its spec and the updated HostNativePackage, which registers it next to post 13's navigation module, from the end tag: cp /tmp/pokedex-ref-16/apps/host/specs/NativeEmbeddedRemotesModule.ts apps/host/specs/ cp /tmp/pokedex-ref-16/apps/host/android/app/src/main/java/com/host/{EmbeddedRemotesModule,HostNativePackage}.kt apps/host/android/app/src/main/java/com/host/ | | iOS | Android | |---|---|---| | Where the copy goes | the .app, at cdn/ios/<remote>/<version>/ | the APK's assets, at cdn/android/<remote>/<version>/, copied out to the files directory at launch | | What puts it there | the last Run Script phase, embed-remotes-ios.sh | the embedRemotes Sync task, before preBuild | | How the host finds it | the directory of main.jsbundle, read from React Native's SourceCode module | the directory EmbeddedRemotesModule.prepare() resolves with | | Without embed-root/ | a warning, a successful build, no copy | a warning, a successful build, no copy | Load a copy from the disk The host changes in five files and their tests. Take them from the end tag. mf-modules.d.ts goes, because nothing loads a remote with import() any more (the Try again section says why), and the design system gains a contrast check for the banner's new colour: cp /tmp/pokedex-ref-16/apps/host/src/shell/{remoteLocator,scriptManager,federationErrors}.ts /tmp/pokedex-ref-16/apps/host/src/shell/FederationBanner.tsx apps/host/src/shell/ cp /tmp/pokedex-ref-16/apps/host/App.tsx /tmp/pokedex-ref-16/apps/host/jest.config.js apps/host/ cp /tmp/pokedex-ref-16/apps/host/__mocks__/{module-federation-runtime,partyApp-partySlice}.js apps/host/__mocks__/ cp /tmp/pokedex-ref-16/apps/host/__tests__/{remoteLocator,scriptManager}.test.ts /tmp/pokedex-ref-16/apps/host/__tests__/{App,RemoteBoundary,FederationBanner}.test.tsx apps/host/__tests__/ cp /tmp/pokedex-ref-16/packages/ui/src/tokens/__tests__/contrast.accessibility.ts packages/ui/src/tokens/__tests__/ rm apps/host/mf-modules.d.ts The host first has to know where the copies are on the device. On iOS the answer comes from React Native's own SourceCode module: a release build loads file:///…/Host.app/main.jsbundle, and the directory above that file is the .app. A development build loads its bundle from the dev server over http, so there is no directory to derive and no copy to use. On iOS, everything that runs from a copy needs a release build. On Android the directory is whatever prepare() resolved with: const sourceCode = NativeModules.SourceCode as | { scriptURL?: string; getConstants?: () => { scriptURL?: string } } | undefined; const SCRIPT_URL = sourceCode?.scriptURL ?? sourceCode?.getConstants?.().scriptURL; const APP_PATH = SCRIPT_URL?.startsWith('file://') ? SCRIPT_URL.replace(/^file:\/\//, '').replace(/\/[^/]+$/, '') : undefined; let embeddedRoot: string | undefined = Platform.OS === 'ios' ? APP_PATH : undefined; Every script of a remote still goes through post 15's resolver. It gains one branch: a remote that runs from its copy, in a launch that could not use the CDN's versions, or after that one remote failed to load from the CDN, resolves to a file inside its copy: return { kind: 'locate', locator: { url: `file://${input.embeddedRoot}/cdn/${input.platform}/${remoteName}/${version}/${filename}`, cache: true, absolute: true, verifyScriptSignature: input.verify, }, }; absolute: true is the detail that is easy to miss. Without it, Re.Pack keeps only the script's file name and looks for it at the top level of the app: iOS asks the app bundle's resources through URLForResource:withExtension:, and Android opens an asset of that name. Neither reaches a <remote>/<version>/ directory. With it, Re.Pack reads the file at the path as given. Verification stays on. Re.Pack's file loader on both platforms runs the same signature check as its download path, before the code runs, on every load. That makes the copy stricter than a cached download, which post 15 noted is run from the disk without a second check. It is also why the copy is never rewritten on its way into the app: change one byte of an installed copy's code and the tab refuses it, failing the same hash check as a tampered download. On iOS the file loader reports the error's full description, so the reason arrives behind the system's generic prefix: [Error: The operation couldn’t be completed. The bundle verification failed because the bundle hash is invalid.] On Android the message reads as it does for a tampered download, with no prefix. The manifest needs no special handling. The copy carries mf-manifest.json beside its bundles, and the federation runtime reads it from a file:// URL the way it reads one from the CDN, because React Native's fetch opens local files: through its file request handler on iOS, and on Android through the blob module's handler, which takes any URL that is not http or https when the response type is blob, the type React Native's fetch asks for. Bundled mode: the day the CDN is unreachable The launch now prepares the copies alongside the probe, since neither needs the other, and a launch with no usable map runs from them instead of giving up: const [versions] = await Promise.all([ CDN_CONFIGURED ? fetchVersionMap() : Promise.resolve(null), prepareEmbeddedCopies(), ]); if (!versions) { return runFromCopies(CDN_CONFIGURED ? 'no usable version map' : 'no CDN configured'); } // --- The launch that could not use the CDN: no usable map, no CDN configured, or a registration // that failed. With copies in the binary it runs from them, every remote registered at its copy's // manifest, which the runtime reads from the disk. With none, or when registering them fails too, // there is nothing to run. --- function runFromCopies(reason: string): FederationStatus { const embedded = REMOTE_NAMES.filter(hasEmbeddedCopy); if (embedded.length === 0) { setStatus({ mode: 'unresolved', source: reason, versions: {}, embedded: [] }); return status; } const versions = Object.fromEntries(embedded.map(name => [name, bundledVersions[name]])); setStatus({ mode: 'bundled', source: 'the copy in the binary', versions, embedded }); try { registerRemotes( embedded.map(name => ({ name, entry: embeddedManifestUrlFor(name) })), { force: true }, ); } catch (error) { console.warn('[federation] the copies could not be registered', error); setStatus({ mode: 'unresolved', source: 'remotes could not be registered', versions: {}, embedded: [] }); } return status; } Registering each remote at its copy's manifest, with post 15's force, sends the runtime to the disk for everything about that remote. From there nothing else is special: the runtime fetches the manifest, the resolver sends every script to the copy, and each one is verified as it loads. unresolved remains for a launch whose binary records no copy with a directory to load from, or where registering the copies fails. Try it by taking the CDN away. Start the server as post 15 did, then make the Release build: npx http-server@14.1.1 cdn-root -p 8000 -c-1 --cors ( cd apps/host && MF_CDN_BASE=http://localhost:8000 MF_APP_VERSION=2.0.0 npm run ios -- --mode Release ) The banner reads cdn · listApp 1.2.0 · partyApp 1.0.0. Stop the server and check that the map is gone: curl -s -o /dev/null http://localhost:8000/ios/maps/2.0.0/version-map.json; echo $? 7 means curl could not connect. Close the app and open it again. The probe fails within its 1.5 seconds, the banner turns purple and reads bundled · listApp 1.2.0 · partyApp 1.0.0, and both tabs open from the copies, with nothing asked of the network except PokéAPI for the Pokémon and GitHub for their artwork. !The Release build with the CDN stopped: the Pokédex tab lists Pokémon from Bulbasaur onwards under the chip listApp 1.2.0, and the purple banner above the tab bar reads bundled, listApp 1.2.0, partyApp 1.0.0 The copy is frozen at build time, and that is the trade-off. A binary built today and launched without the CDN next year runs today's versions, whatever the CDN has shipped since. That is the part of post 1's requirement a copy can meet: the reviewed code runs without the CDN. The Pokémon still come from PokéAPI, so with no network at all the list's request fails and the screen shows its own error, "Couldn't reach PokéAPI." Nothing about the failure is remembered either. The next launch asks the CDN again, and if a usable map arrives and its versions register, the app is back on the versions the map names. A remote that fails in a CDN launch Bundled mode covers a launch that could not use the CDN's versions. The other failure is narrower: the launch uses the CDN's versions, and then one remote fails. The cause might be a version retired from the CDN while a map still names it, a chunk cut off halfway, or a release that throws while its module initialises, which was post 15's third break. In post 15's build each of those leaves one dead tab. Two nets turn them into a tab that runs from its copy, while the other remote stays on the CDN. The manifest net Module Federation fetches a remote's mf-manifest.json before any of its code loads. A tab's boundary would hear of that fetch failing, but only once the runtime's own fetch gives up, and the runtime sets no time limit on it. And not every load runs inside a boundary: the party's state and styles modules ask for its manifest at boot, from an effect outside every tab. So the first net sits where the runtime asks for every manifest: a runtime plugin's fetch hook, which the runtime calls before its own fetch, and which may answer with a Response or hand the request back by returning nothing: const embeddedFallback: ModuleFederationRuntimePlugin = { name: 'embedded-fallback', fetch(url: string) { if (status.mode !== 'cdn') { return undefined; } const remote = manifestRemote(url, REMOTE_NAMES); if (!remote || !hasEmbeddedCopy(remote)) { return undefined; } if (fallbackRemotes.has(remote)) { return fetch(embeddedManifestUrlFor(remote)); } return cdnManifestOrEmbedded(url, remote); }, }; registerPlugins([embeddedFallback]); A CDN remote's manifest is fetched from the CDN, within the probe's 1.5-second limit. Any failed request (an error status such as a 404, a timeout or a dropped connection) drops that one remote to its copy and answers with the copy's manifest instead: async function cdnManifestOrEmbedded(url: string, remote: string): Promise<Response> { const controller = new AbortController(); const timer = setTimeout(() => controller.abort(), PROBE_TIMEOUT_MS); try { const response = await fetch(url, { signal: controller.signal }); if (response.ok) { return response; } console.warn(`[federation] ${remote} manifest returned ${response.status}`); } catch (error) { console.warn(`[federation] ${remote} manifest could not be fetched`, error); } finally { clearTimeout(timer); } fallBack(remote); return fetch(embeddedManifestUrlFor(remote)); } A manifest that arrives with a success status goes back to the runtime as it is, even one that cannot be read. The runtime fails on that afterwards, and for a tab's load the boundary net catches the failure. fallBack adds the remote to a set and puts it on the banner, as listApp 1.2.0 embedded. The resolver reads that same set for every script, so the remote's very next script resolves to its copy, while every other remote keeps its CDN URL. The set lives in memory on purpose: a remote drops to its copy only until the next launch, which asks the CDN again, so a failure that was only the network does not keep it there. Remembering failures across launches, and rolling a bad version back permanently, belongs to post 17. Other tools already offer parts of this, and each would work at its own layer. Module Federation's errorLoadRemote hook can return a fallback when a load fails, the manifest included. Its official retry plugin retries a failed load and can rotate between backup domains, and Re.Pack's script locator takes retry and retryDelay. The fetch hook was the fit here for two reasons. errorLoadRemote hears of a failure only after the runtime's fetch gives up, as the boundary does, while the fetch hook makes the request itself and can put the probe's time limit on it. And retrying is a different policy: it spends time asking the same CDN again, where this net answers at once from a copy the binary already has and leaves the next attempt to the next launch. Retries could sit in front of the net, at the cost of a longer wait before the copy. Try the net with a version the CDN no longer holds. Start the server again, move the list's version out of it, and cold-start the app: mv cdn-root/ios/listApp/1.2.0 /tmp/listApp-1.2.0 The list's manifest comes back 404, the list runs from its copy, and the party still loads from the CDN, which the server's log shows: [2026-09-29T14:40:27.102Z] "GET /ios/listApp/1.2.0/mf-manifest.json" "Host/1 CFNetwork/3860.500.112 Darwin/27.0.0" [2026-09-29T14:40:27.103Z] "GET /ios/listApp/1.2.0/mf-manifest.json" Error (404): "Not found" [2026-09-29T14:40:27.120Z] "GET /ios/partyApp/1.0.0/mf-manifest.json" "Host/1 CFNetwork/3860.500.112 Darwin/27.0.0" [2026-09-29T14:40:27.159Z] "GET /ios/partyApp/1.0.0/partyApp.container.js.bundle" "Host/1 CFNetwork/3860.500.112 Darwin/27.0.0" !The Release build with the list's version missing from the CDN: the Pokédex tab lists Pokémon from Bulbasaur onwards under the chip listApp 1.2.0, and the blue banner reads cdn, listApp 1.2.0 embedded, partyApp 1.0.0 Move the directory back before going on: mv /tmp/listApp-1.2.0 cdn-root/ios/listApp/1.2.0 <blockquote class="callout callout-tip"> <p><strong>If every remote drops to its copy, check that the app can reach the server.</strong> A banner that marks both remotes embedded, or reads <code>bundled</code>, while the server is running usually means the app could not reach it: iOS's App Transport Security and Android's cleartext rules refuse plain http if the build does not allow it. This host allows local networking on iOS and, on Android, cleartext only to <code>localhost</code>, <code>127.0.0.1</code> and <code>10.0.2.2</code>, with their subdomains.</p> </blockquote> The boundary net Some failures come after a manifest that arrived fine: a container or a chunk that does not arrive or does not verify, or a module that throws while it is evaluated. They fail the tab's load, and the tab's load runs inside the RemoteBoundary that post 11 put around each tab. The boundary is the second net, and it stays a class component, because React has no way yet to write an error boundary as a function component. It now asks one question before it shows the error state: did the load fail, so the remote never produced a component, and is there a copy this remote has not dropped to yet? componentDidCatch(error: unknown) { console.warn(`${this.props.remote} failed`, error); if (this.dropsToCopy()) { fallBackAndReload(this.props.remote); this.nextAttempt(); } } // A failure the copy in the binary can answer: the load failed, so the remote never produced a // component, and this remote has a copy it has not dropped to yet. dropsToCopy() { return !this.loaded() && canFallBack(this.props.remote); } When it can, the boundary drops the remote to its copy and loads the tab again, rendering the loading state in the meantime, so the error state never appears. canFallBack allows it once per remote, only in a CDN launch, and only for a remote that has a copy. When it cannot, because the copy failed too, or there is none, the tab shows the error state as before. A remote whose load returned a component keeps its error state when that component throws. That is a decision, and it rests on whether the remote produced a component. Before it does, the tab is still showing its loading state, so a swap to the copy is invisible. Once the load has returned a component, the boundary keeps that version, even when the component throws on its very first render. It records the attempt whose load arrived with a component, not whether the screen was ever shown, so a failure in that attempt counts as thrown by the remote's own code, which has already run in this session. Until the map names a different version, the copy is that same version anyway. So the boundary shows "This tab stopped working" and offers Try again, which renders the remote again and recovers an error that does not repeat. An error that repeats leaves the tab on its error state for the rest of the session, and after a relaunch for as long as the map names that version, even with a working copy in the binary. The boundary net does not cover the party's two boot modules: they load from an effect outside every tab, so the host only logs a failure the manifest net does not answer. When the state module fails, adding to the party stays disabled until the Party tab loads, because that tab injects the state itself; a failed styles module does not affect adding. Post 15's third break is now a failed load the copy can answer. Its release throws while its module initialises, and post 15's evaluation window turns that throw into a failed load instead of a fatal report. Build it again. Add the same line below the imports of apps/list/src/PokedexScreen.tsx: throw new Error('PokedexScreen failed to initialise'); Then ship it as 1.4.0, the way post 15 did: ( cd apps/list && MF_REMOTE_VERSION=1.4.0 npm run bundle:ios:prod ) && rsync -a --exclude '*.map' --exclude mf-stats.json apps/list/cdn/ios/listApp/1.4.0/ cdn-root/ios/listApp/1.4.0/ Delete the line, point the 2.0.0 map at 1.4.0 and relaunch. The tab drops to the 1.2.0 copy: the chip says 1.2.0, the banner marks the list embedded, and the simulator's log has no RCTFatal in it. Put the map back to 1.2.0. The boundary net also catches a CDN that disappears mid-session. Launch with the server running and leave the Party tab unopened. Add a Pokémon from the Pokédex, stop the server, then open Party for the first time. Only the party's state and styles modules loaded at boot, so its screens are not on the device yet: the download fails, and the tab shows a spinner for a moment and then the party from its copy, with the Pokémon you added and the banner reading partyApp 1.0.0 embedded. Removing the Pokémon there works too, because the copy's screens and the state module that loaded from the CDN share the host's one store. A Try again that retries In post 15's build, Try again on a tab whose load failed never brings the tab back. Three records of the failed load stand between Try again and a fresh attempt. The first is React's. A lazy component calls its load function once: the React documentation says "Both the returned Promise and the Promise's resolved value will be cached, so React will not call load more than once", and a rejection goes to the nearest error boundary. The boundary already handled that, by mounting a new lazy component under a new key for every attempt. The second is the host bundle's, and it is the one that kept post 15's tabs dead. An import('listApp/ListStack') compiles into a module of the host's own bundle, and the bundle keeps each module it has evaluated for the rest of the session. After a failed load it keeps what the failure left behind, a module with nothing in it, and every later import() settles with that, even when the retry downloads the remote successfully. Measured on an iOS Release build: the retry fetched the manifest and both chunks, and the import still resolved without a component. So the tabs, and the two modules the party loads at boot, no longer use import(). They ask the federation runtime directly, every time: export async function loadRemoteModule<T>(id: string): Promise<T | undefined> { const factory = await loadRemote<() => T>(id, { loadFactory: false, from: 'runtime' }); return factory ? factory() : undefined; } It asks for the module's factory rather than its exports, with loadFactory: false, which is how post 15's evaluation window gets to wrap it: the runtime hands back the window's wrapper, and calling it here evaluates the module inside the window. The third is the federation runtime's. In @module-federation/runtime-core 2.9.0, the version this series installs, its record of a remote outlives a failed load: it keeps the remote's entry, the manifest it read, the container it loaded, and a container load that failed, which it hands back to every later request. The remote's chunk registry, a global array named after the remote's uniqueName, also keeps every chunk its last container fetched, and a new container installs all of them before fetching anything. Try again clears both before it loads again: function reloadRemote(remote: RemoteName, entry: string): void { try { registerRemotes([{ name: remote, entry }], { force: true }); } catch (error) { console.warn(`[federation] ${remote} could not be registered again`, error); } delete (globalThis as Record<string, unknown>)[CHUNK_REGISTRY[remote]]; } Registering the remote again with force removes the runtime's record, the container's global included, and points the next load at entry. Try again reads entry back from the runtime instead of building it, which works in every mode, including development, where only the build knows the dev server's URL. A remote that dropped to its copy still retries its copy, because the manifest net and the resolver both send any remote in the set to its copy. Re.Pack's script cache needs no clearing for a failed load: once a resolved script's load settles, Re.Pack drops its promise, and a download that fails verification is never written to its cache, on either platform. A refusal from the resolver is the exception. It comes before that cleanup, so Re.Pack keeps the rejection and hands it back to every later request for the same script. This host's resolver refuses only a remote it has no version or copy for. One gap stays open: Try again cannot cancel a download already in flight. A chunk the failed container was still fetching lands in whichever registry exists when it arrives, and when a script is still loading, Re.Pack answers a new request for it with the promise already outstanding. When the copy is the version the CDN was serving, the default, those are the same bytes either way; when the map names a different version, a late chunk from the CDN can land beside the copy's container. A render error shows Try again recovering a tab whose code has already loaded. The boundary net's mid-session demo stopped the server, so start it again and leave it running: npx http-server@14.1.1 cdn-root -p 8000 -c-1 --cors Then make the list's screen throw while it renders, for its first eight seconds. Below the imports of apps/list/src/PokedexScreen.tsx add: const LOADED_AT = Date.now(); and as the first lines of PokedexScreen(): if (Date.now() - LOADED_AT < 8000) { throw new Error('PokedexScreen render failed'); } The throw has to last: React retries a render that fails, so an error that stops too soon recovers before the boundary's error state is ever shown. Ship it as its own version, the way 1.4.0 shipped, then delete the lines and point the 2.0.0 map at 1.5.0: ( cd apps/list && MF_REMOTE_VERSION=1.5.0 npm run bundle:ios:prod ) && rsync -a --exclude '*.map' --exclude mf-stats.json apps/list/cdn/ios/listApp/1.5.0/ cdn-root/ios/listApp/1.5.0/ Relaunch and the tab shows "This tab stopped working". Wait out the eight seconds and press Try again: the list renders on 1.5.0, and the server's log shows no new request, because the code had loaded and only needed rendering again. Put the map back to 1.2.0 afterwards. > 🎞️ Animated demo: watch it on warrendeleon.com The failures and what each one costs, side by side: | Failure | Caught by | What the user sees | |---|---|---| | The CDN cannot be reached at launch | the launch, in bundled mode | every tab from its copy, and a purple banner | | A remote's manifest fails: a 404, a timeout, a dropped connection | the manifest net | the tab from its copy, marked embedded on the banner | | A tab's remote code does not arrive, does not verify, or throws as its module initialises | the boundary net | a moment of loading, then the tab from its copy | | The party's state module fails at boot, once its manifest has arrived | nothing: the host logs it | adding to the party stays disabled until the Party tab loads | | The party's styles module fails at boot, once its manifest has arrived | nothing: the host logs it | adding still works | | The copy fails as well, or there is no copy | the boundary | "This tab could not load", with Try again | | The remote's component throws as it renders | the boundary | "This tab stopped working", with Try again | Now break it Both embed steps succeed without embed-root/, and that is the trap. Move it out and rebuild the Release build, this time with --verbose: mv embed-root /tmp/embed-root && ( cd apps/host && MF_CDN_BASE=http://localhost:8000 MF_APP_VERSION=2.0.0 npm run ios -- --mode Release --verbose ) The build succeeds, and the phase's warning is the only sign that anything is missing. Without --verbose, npm run ios shows a spinner in place of the build's output (or passes the output through xcbeautify or xcpretty, when one is installed), then reports success Successfully built the app and launches. With --verbose, the warning sits among the build's own lines: debug warning: …/embed-root/ios does not exist, so no remotes are embedded. Run 'node tools/build-cdn.mjs' before a release build. debug ** BUILD SUCCEEDED ** [11.716 sec] success Successfully built the app Stop the server and cold-start the app. The banner still says bundled · listApp 1.2.0 · partyApp 1.0.0, because the host compiled in the versions it was told it carries, and both tabs say "The app's own copy of this remote could not be loaded". <blockquote class="callout callout-danger"> <p><strong>Build the CDN tree, then the release build, every time.</strong> The embed phases copy whatever <code>embed-root/</code> holds when they run, and when it is missing they let the build succeed with nothing more than a warning in the build's output. A release build made before <code>node tools/build-cdn.mjs</code>, or after <code>embed-root/</code> was cleaned, builds successfully with no copy in it, and nothing fails until the app needs a copy: when the launch cannot use the CDN's versions, or when one remote fails.</p> </blockquote> Move it back and rebuild before going on: mv /tmp/embed-root embed-root && ( cd apps/host && MF_CDN_BASE=http://localhost:8000 MF_APP_VERSION=2.0.0 npm run ios -- --mode Release ) Run it The copies and the CDN tree, then the server: node tools/gen-signing-keys.mjs && node tools/build-cdn.mjs npx http-server@14.1.1 cdn-root -p 8000 -c-1 --cors The Release build on iOS, then Android with the emulator's address for the machine: ( cd apps/host && MF_CDN_BASE=http://localhost:8000 MF_APP_VERSION=2.0.0 npm run ios -- --mode Release ) ( cd apps/host && MF_CDN_BASE=http://10.0.2.2:8000 MF_APP_VERSION=2.0.0 npm run android -- --mode release ) With either running, stop the server and cold-start for bundled mode, or move one version out of cdn-root for the manifest net. On Android the extracted copies sit in the app's files directory, under files/cdn/android/, with the .installed marker beside them in files/cdn/. The suites, the smoke test and the key generator's tests: ( for d in apps/host apps/list apps/party packages/ui; do ( cd "$d" && npx jest --silent ) || exit 1; done ) && sh scripts/federation-smoke.sh && node --test tools/gen-signing-keys.test.mjs What you built, and what's next The demos in this post ran on an iOS Release build, and the copies, their extraction and bundled mode on an Android release build as well. A remote that fell back is forgotten at the next launch, so a version that keeps failing fails again on every launch before its copy takes over. Remembering failures belongs with the other open problem: the map every binary trusts is still a plain text file anyone who can write to the bucket can change. Next: signing that map, a counter that makes the app reject an older map, and an app that rolls a failing version back by itself. Sources - React: lazy — the load function called once, its promise and resolved value cached, and a rejection thrown to the nearest error boundary - React source: ReactLazy at 19.2.3 — a rejected load stored on the lazy component and thrown again on every later render - React: Component, catching rendering errors with an error boundary — "There is currently no way to write an Error Boundary as a function component" - Module Federation: runtime hooks — the fetch hook for manifest requests, returning Promise<Response> | void | false, and errorLoadRemote - Module Federation source: runtime-core SnapshotHandler at 2.9.0 — a manifest asked of the fetch hook first, then of the global fetch, then of errorLoadRemote on failure - Module Federation source: runtime-core remote/index.ts and utils/load.ts at 2.9.0 — registerRemotes with force calling removeRemote, which deletes the remote's entry, manifest, container global and container-loading promise; a failed container load's promise kept and returned to later loads until then - Module Federation: retry plugin — retries and backup domains for failed loads - Re.Pack source: ScriptManager types at 5.2.5 — the locator's absolute, retry and retryDelay - Re.Pack source: ScriptManager.ts at 5.2.5 — a resolved script's promise deleted once its load settles, failed or not; a resolver's rejection comes before that cleanup and is kept - Re.Pack source: ScriptManager.mm at 5.2.5 — a file:// script read at its absolute path, or by name through URLForResource:withExtension:, and verified before it runs; a download written to the cache only after it verifies - Re.Pack source: FileSystemScriptLoader.kt and RemoteScriptLoader.kt at 5.2.5 — the same on Android, with an asset opened by its file name when the path is not absolute, and a download verified before it is written - Apple: URLForResource:withExtension: — a resource looked up by name in the bundle, beside the variant that takes a subdirectory - swift-build source: ShellScriptTaskProducer.swift at swift-6.2.1-RELEASE — a Run Script phase that declares no outputs, or is set to run always, runs on every build - React Native CLI source: buildProject.ts at 20.2.0 — without --verbose, the build's output piped through xcbeautify or xcpretty when one is installed, and a spinner otherwise - React Native source: RCTSourceCode.mm at 0.85.3 — scriptURL, the URL the JavaScript bundle loaded from - React Native source: RCTFileRequestHandler.mm and RCTAppSetupUtils.mm at 0.85.3 — file:// requests answered on iOS, and the handler registered with the app's networking - React Native source: BlobModule.kt at 0.85.3 — any URL that is not http or https, read on Android when the response type is blob - Android: AssetManager — assets opened and listed from the APK, not as files - Gradle: Sync — a copy that also removes files the source no longer has - react-native-module-federation — the companion repo, the build at the tag post-16-fallbacks
use agent-device for agent verification locally or remotely, whatever, it just works callstack/agent-device