AI & Local Models
FlutterProbe can use a language model for a few things. All of it is optional and none of it is on the
path that decides whether a test passes. With no ai: block in probe.yaml, no AI code runs and nothing
is sent anywhere.
The guarantees
Section titled “The guarantees”- Off by default. No
ai:block, no flag, no AI. - Tests never depend on it.
probe testruns, reports and exits the same with or without a model. The only AI that can affect a result is a step you wrote:see "..." with ai. - Fails open. If the model is down, slow or answers garbage, triage, selector proposals and doctor print a one-line note. Results, report files and the exit code are untouched.
- Bounded. Every call has a timeout, there are no retries, at most 5 failures are triaged, and triage stops after the first provider error instead of calling a dead endpoint once per failure. Selector proposals make at most 3 requests per run.
- Private by choice. With
provider: local, nothing leaves your machine.redact:rules black out screenshot regions before any provider sees them.
Use a local model
Section titled “Use a local model”Any OpenAI-compatible server works: Ollama, LM Studio, llama.cpp’s server.
ai: provider: local endpoint: http://localhost:11434/v1 # Ollama; LM Studio is http://localhost:1234/v1 model: gemma4:31b timeout: 120s # local models can be slow # vision: false # set for a text-only model, see belowNo API key is needed. Check it:
probe ai doctor✓ config: provider local, model gemma4:31b✓ endpoint: reachable, model "gemma4:31b" is available✓ text: answered in 9.6s: "pong"✓ vision: the model accepts images — `with ai` screenshot assertions will workThe doctor reports whether the model accepts images. A text-only model is a supported setup.
In Studio
Section titled “In Studio”Studio’s AI chat reads the same ai: block as the CLI, and these rules apply to it:
- Keys. Save the key in the keychain from the AI panel, or write
api_key: ${NAME}and set that variable in the environment that starts Studio. A plain key inprobe.yamlis not used, and Studio says so. A saved keychain key goes only to the Anthropic provider, never to a local endpoint. - Local endpoints.
provider: localmust namelocalhostor a loopback address. Any other address is refused unlessstudio.ai_allow_remote_endpoint: trueis set. An address with a user name or password is refused, and onlyhttpandhttpsare accepted. - Settings. Workspace settings has an AI provider section for these options, with a model list and Test connection; what it reports never contains the key. It never takes a key: use the key button in the AI panel, or name an environment variable there. Save keeps a plain
api_keyalready inprobe.yaml, because the CLI reads it, and does not use it; naming an environment variable replaces it. A form you did not change is not validated or written, and a refused AI section shows its reason while the other settings still save. For OpenAI and a local endpoint, a saved keychain key is never sent, so the note tells you to name a variable or remove the line. - Find local models. In Settings, a button asks this machine’s own model server which models it has. It runs only on your click, asks only loopback addresses, sends a read-only request for the model list with no key, and sets the result as text. It fills only empty fields, picks a model only when exactly one is listed, and saves nothing until you press Save.
- Preview. Before a message goes to a destination you have not confirmed in this window, Studio shows what it would send now, redacted, and waits for Send. This check is in the Studio window only.
- Explain failures. In Studio, a button under Results asks the provider for advice on the failed tests of the last run (at most 5, one request each, no retries, with a time limit). It shows its preview first, on every Explain for a hosted or remote destination, and sends only after Send. It sends the test name, the file relative to the workspace and the error text, each with hidden characters dropped, the exact key removed, emails and home folders replaced, credentials redacted and a size limit, and nothing else. The advice is shown as text in a card marked Advisory and never changes a result. Each note has a small
L12button (the file’s name as its hover text, never its folder) that opens the line where its test failed, in the same way as a failing row in Results; the place is read from Studio’s own record of the run and goes to the window only: it is not part of what is sent. IP addresses, host names and other personal text in an error are not replaced. - Suggest a fix. Next to Explain failures, a button asks the provider for one ProbeScript line to replace the failed step of the last run, from the failed step, the last run, the open file and, when the workspace setting allows it, the device’s widget tree. Studio builds the request first and shows it; it sends only after Send, and what it sends is what it showed, with credentials redacted (the redaction can only remove more on the wire, never less). A hosted or remote destination shows the preview before every suggestion; one on this machine the first time and again after its model, endpoint, key, workspace or settings change. The answer is text on a card that names the failed step’s line (
L12, a button that opens it, with the same rules as a failing row): it is not applied to the test file, not verified against the app, and it changes no result (the same advisory as the CLI’sselector_proposal). Replace the line (on the card, only for a suggestion that cleaning did not change) puts the suggestion into the editor in place of the failed step’s line: one undo step, never saved and never run. Studio checks it first and says what did not hold: the open file is the file the failed step ran from, its line still has the text it had when the suggestion was asked for, and the file parses with the line replaced into exactly the same steps at the same lines. The step keeps what it does: the same verb and the same flags (optional,if visible,with ai, …), from a closed list of what a suggested fix may be and may replace:tap,long press,double tap,type ... into,see,don't seeandwait until ... appearsordisappears. Only what the step looks for may change: atypekeeps its text and its target (type "bob" into "Email"never becomestype "bob"ortype "pw" into "Email") and acontains "b"keeps its value. So atapbecomes atapthat looks for something else, neveropen link,clear app data,kill,restart,store, a native or webview action,add media, asee ... with ai, a second step, a recipe call or a block, and never atap ... optionalwhere the line was not optional. The suggestion is one line of plain characters (no line break, control, hidden or bidirectional character, no unusual space) and brings nothing the line did not have: no comment, no;, no${ENV}or<variable>. A line with a comment, a step that holds others (repeat,if) and a step in a composite test are not replaced. After a replacement the card keeps a Re-run the failed test button: it runs the test that holds the replaced line, once per click, and shows the verdict on the card (passed, or still failing with the new error). It never saves for you: it stays disabled until you save the file (the run reads the file on disk), and while a run is going, with no device connected, in another file, or after the replaced line was edited, and it says which. If the file changes while the test runs, the verdict says it may not be the replaced line’s result. When the re-run still fails, the card offers Suggest a fix again: the same preview and the same confirmation as the first suggestion (every time for a destination that is not this machine, once per session for one that is), one request per press, and never automatic. Studio asks a provider at most three times for the same failed step (file, line and test) in a session, counts an attempt when its request goes out whatever the provider answers, numbers the card from the second suggestion (Suggested fix (2 of 3)), and refuses a fourth before anything is sent; opening another workspace starts the count again. - Off. With no
ai:block and no saved key, AI is off, and the chat sends nothing.provider: noneturns AI off even when a key is saved. - Answers. What a model answers, and the text of a provider’s error, is cleaned like what is sent: escapes of hidden characters are decoded and hidden characters dropped, the key and credential shapes are removed, an email address becomes
[email]and this account’s own home folder becomes~(the folder your system names,/Users/youand/home/you, also percent-encoded), and only then is the text cut, so a cut cannot leave the start of an address. Other people’s folders are not guessed at in an answer:/users/42and/home/settingsare shown as written. The text sent with a failure still has every/Users/x,/home/x,C:\Users\xand/rootreplaced, which also catches text that only looks like a folder. This applies to the advice of Explain failures, to a suggested fix and its reason, to the text of a provider’s error and to the model names and the connection check details a server returns. A suggested fix that this changed shows a Changed row above the line (type "[email]" into "Email"is not what the model meant), also when only its reason changed: check it before you run it. - Redirects. Provider requests do not follow a redirect, so a prompt is never sent to a second host.
- Redaction. Every system prompt and chat message is redacted before it leaves. The families are API keys of common providers (OpenAI and Anthropic style
sk-, GitHub, AWS, Slack, Google and Stripe), PEM private keys,Authorization: Basicvalues, assignments whose name says secret, token, password or key, secret query parameters such as?token=, a JWT, short ones included (three dot-separated parts; the first starts witheyJand has at least 7 characters, the second at least 4, the third any or missing, so a JWT cut inside its payload is redacted too, as ineyJhbGci.eyJzdWIi.xandeyJhbGci.eyJzdWIi; letters, digits,-and_), and a card number of 13 to 19 digits that passes the Luhn check, written solid or with single spaces or hyphens (4111 1111 1111 1111). A card number glued to letters or an underscore (card_4111...) is not recognised. About one long number in ten passes that check by chance, so a long number such as a millisecond timestamp is sometimes redacted too. Every field of a chat turn (the system prompt with the open file, and each message) takes the same steps as an Explain: hidden characters and no-break spaces are dropped first, so a token split by one is whole again before it is read; the field is bounded (150,000 characters, and a cut field ends with[truncated: N more characters not sent]); the exact saved key is removed; emails are replaced ([email]) and so is this account’s own home folder (~: the folder your system names, and/Users/you,/home/youand\Users\youfor your account name in any case, wherever they appear, also behind a mount prefix such as/host_mnt/Users/youor/mnt/c/Users/you, in a URL, and written with escaped slashes in JSON; a name that continues, such as/home/users/42for/home/user, is not replaced); the families above are redacted; the field is cut to its cap. Other people’s folders are not guessed at, because/home/settingsand/Users/listare routes in a test file. A tab becomes a space. A zero-width non-joiner or joiner is kept between two non-ASCII letters or symbols (Persian words, emoji sequences) and dropped anywhere else. Full-width digits and typographic dashes are read as ASCII ones only inside a run that looks like a card number; other text is sent as it is. The open file is cut back to a space, a line break, a quote or a punctuation mark, so a key or token is never sent in part (a long line with none of them, such as one line of minified JSON, CJK text or a data URI over 64 KB, sends only the[truncated]marker), and only up to the newest 40 messages of a conversation are sent, starting at one of yours (the preview and a notice with the reply say how many were left out). A token split across two messages is not found. The preview shows exactly what the request carries. The reply is cleaned the same way before it is shown (hidden characters dropped, the exact key removed, the families redacted, emails and home folders replaced), so a provider that echoes your text back shows none of it. When that changed the reply, a notice says so (atype "[email]" into "Email"line is not what the model meant).
What you can do with it
Section titled “What you can do with it”| Feature | Command | Needs vision |
|---|---|---|
| Explain failures | probe test --ai-triage, probe triage --input reports/results.json, MCP triage_failure |
no |
| Propose a selector for a failed step | probe test --ai-heal --format json |
no |
| Generate tests | probe generate prompt -p "...", probe generate from-recording ... |
no |
| Screen assertions | see "the total looks correct" with ai |
yes, or text mode |
| Visual smoke check | assert no visual defects with ai |
yes |
| Read text off screen | read "the OTP" with ai into otp |
yes |
Failure triage
Section titled “Failure triage”probe test tests/ --ai-triageAfter the results are final, the model gets each failure’s error text and writes a short probable cause and
a next step, printed and saved to <reports>/triage.md. Since 0.15.0 failure text names the line, the step
and the visible texts/keys on screen, which is what the model reasons over. Triage is advice. It does not
re-run anything or change a result.
Quality depends on the model: a 0.5B model will guess wrong; a 7B-30B model is useful. Treat the output as a hint.
When failures come from a CI run, triage the saved report later:
probe test tests/ --format json -o reports/results.jsonprobe triage --input reports/results.json -o reports/triage.mdSelector proposals for failed steps
Section titled “Selector proposals for failed steps”probe test tests/ --ai-heal --format json -o reports/results.jsonWhen a step runs into its timeout on a text or key selector (tap "Save", see #total), probe records
the texts and widget keys that were visible on screen at that moment. The model is asked for one replacement
from them, and the failed result in the JSON report gets a selector_proposal:
"selector_proposal": { "advisory": "Proposed by a language model from the texts and keys that were on screen when the step timed out. Not applied to the test file and not verified against the app. It does not change this result.", "line": 9, "failed_selector": "\"Save\"", "proposed_selector": "\"Save changes\"", "applied": false, "verified": false, "model": "qwen2.5-coder:7b", "provider": "local", "location": "local"}What it does and does not do:
- The step stays failed. The results, the summary and the exit code are the same when the model
answers, fails or times out. Nothing is written to the
.probefile, and the proposal is not tried against the app. You decide whether to edit the test. - Only what was recorded as on screen. When a step runs into its timeout,
probekeeps the visible texts and widget keys as two lists of data. A proposal is kept only when it is exactly one of those texts, written as"text", or exactly one of those keys, written as#key. A part of a listed text, a text offered as a key, a key offered as a text, or any other word of the failure text is not kept, and the entry gets anunavailablereason with noproposed_selector. - Only step timeouts, and no list is read from the failure text. The lists are recorded when the step
runs into its timeout. They are never read back out of the failure text, because the text on screen is
part of it and could be written to look like a list. The app’s own “Widget not found” message also lists
the visible texts, but it is plain text that a widget can shape, so it is not used: a failure that is not
a step timeout is not sent to the model, and its entry says so in
unavailable. The failure text keeps at most 40 characters of each listed text, and a text over that length is never proposed, even though the full text was on screen. When every listed item is that long, theunavailablereason says so. - Bounded. At most one request per failed step, at most 3 requests per run, each limited by
ai.timeout, no retries, none after a failed request, and two minutes for all of them together. Model text is cut to 200 bytes, the API key is removed from it, and control characters, line and paragraph separators and bidirectional controls in it are replaced by spaces before it is printed or written to the report. The file name and the selector in the terminal line get the same treatment. - What is sent. The failed step, its selector, the failure text and the two lists, with resolved
environment values masked. The failure text is what
--ai-triagesends. No screenshot is sent. - Where it shows. The JSON report and one line per proposal in the terminal.
locationislocalforprovider: localon this machine,networkforprovider: localon another host, andhostedforopenaiandanthropic. Proposals are left out of an uploaded report. JUnit and HTML reports do not show them. - Off by default. Without
--ai-heal, or without anai:block, no request is made.
Text-only models (vision: false)
Section titled “Text-only models (vision: false)”Most small local models cannot see images. With ai.vision: false, see "..." with ai is judged from the
visible texts and widget keys the agent reports instead of a screenshot. Pixel-based steps
(assert no visual defects, read ... with ai) fail with a clear message rather than guessing.
Text mode is refused when ai.redact rules exist, because redaction works on screenshot regions and cannot
be applied to a list of texts. It needs flutter_probe_agent 0.15.0 or newer.
Test generation
Section titled “Test generation”probe generate uses the ai: provider when one is set (including local). With no ai.provider it keeps
its original behavior: Claude with --api-key / ai.api_key.
Cloud providers and privacy
Section titled “Cloud providers and privacy”provider: openai and provider: anthropic send the request to that vendor directly, never through a
FlutterProbe service. Triage and text mode send error text and on-screen text, which can contain user data;
use provider: local if that matters. In Studio’s chat, provider: openai sends the open file with each message, as provider: anthropic already does.
See Configuration for every ai: key.