Skip to content

AI & Local Models

FlutterProbe can use a language model for a few things. All of it is optional and none of it is on the path that decides whether a test passes. With no ai: block in probe.yaml, no AI code runs and nothing is sent anywhere.

  • Off by default. No ai: block, no flag, no AI.
  • Tests never depend on it. probe test runs, reports and exits the same with or without a model. The only AI that can affect a result is a step you wrote: see "..." with ai.
  • Fails open. If the model is down, slow or answers garbage, triage, selector proposals and doctor print a one-line note. Results, report files and the exit code are untouched.
  • Bounded. Every call has a timeout, there are no retries, at most 5 failures are triaged, and triage stops after the first provider error instead of calling a dead endpoint once per failure. Selector proposals make at most 3 requests per run.
  • Private by choice. With provider: local, nothing leaves your machine. redact: rules black out screenshot regions before any provider sees them.

Any OpenAI-compatible server works: Ollama, LM Studio, llama.cpp’s server.

ai:
provider: local
endpoint: http://localhost:11434/v1 # Ollama; LM Studio is http://localhost:1234/v1
model: gemma4:31b
timeout: 120s # local models can be slow
# vision: false # set for a text-only model, see below

No API key is needed. Check it:

Terminal window
probe ai doctor
✓ config: provider local, model gemma4:31b
✓ endpoint: reachable, model "gemma4:31b" is available
✓ text: answered in 9.6s: "pong"
✓ vision: the model accepts images — `with ai` screenshot assertions will work

The doctor reports whether the model accepts images. A text-only model is a supported setup.

Studio’s AI chat reads the same ai: block as the CLI, and these rules apply to it:

  • Keys. Save the key in the keychain from the AI panel, or write api_key: ${NAME} and set that variable in the environment that starts Studio. A plain key in probe.yaml is not used, and Studio says so. A saved keychain key goes only to the Anthropic provider, never to a local endpoint.
  • Local endpoints. provider: local must name localhost or a loopback address. Any other address is refused unless studio.ai_allow_remote_endpoint: true is set. An address with a user name or password is refused, and only http and https are accepted.
  • Settings. Workspace settings has an AI provider section for these options, with a model list and Test connection; what it reports never contains the key. It never takes a key: use the key button in the AI panel, or name an environment variable there. Save keeps a plain api_key already in probe.yaml, because the CLI reads it, and does not use it; naming an environment variable replaces it. A form you did not change is not validated or written, and a refused AI section shows its reason while the other settings still save. For OpenAI and a local endpoint, a saved keychain key is never sent, so the note tells you to name a variable or remove the line.
  • Find local models. In Settings, a button asks this machine’s own model server which models it has. It runs only on your click, asks only loopback addresses, sends a read-only request for the model list with no key, and sets the result as text. It fills only empty fields, picks a model only when exactly one is listed, and saves nothing until you press Save.
  • Preview. Before a message goes to a destination you have not confirmed in this window, Studio shows what it would send now, redacted, and waits for Send. This check is in the Studio window only.
  • Explain failures. In Studio, a button under Results asks the provider for advice on the failed tests of the last run (at most 5, one request each, no retries, with a time limit). It shows its preview first, on every Explain for a hosted or remote destination, and sends only after Send. It sends the test name, the file relative to the workspace and the error text, each with hidden characters dropped, the exact key removed, emails and home folders replaced, credentials redacted and a size limit, and nothing else. The advice is shown as text in a card marked Advisory and never changes a result. Each note has a small L12 button (the file’s name as its hover text, never its folder) that opens the line where its test failed, in the same way as a failing row in Results; the place is read from Studio’s own record of the run and goes to the window only: it is not part of what is sent. IP addresses, host names and other personal text in an error are not replaced.
  • Suggest a fix. Next to Explain failures, a button asks the provider for one ProbeScript line to replace the failed step of the last run, from the failed step, the last run, the open file and, when the workspace setting allows it, the device’s widget tree. Studio builds the request first and shows it; it sends only after Send, and what it sends is what it showed, with credentials redacted (the redaction can only remove more on the wire, never less). A hosted or remote destination shows the preview before every suggestion; one on this machine the first time and again after its model, endpoint, key, workspace or settings change. The answer is text on a card that names the failed step’s line (L12, a button that opens it, with the same rules as a failing row): it is not applied to the test file, not verified against the app, and it changes no result (the same advisory as the CLI’s selector_proposal). Replace the line (on the card, only for a suggestion that cleaning did not change) puts the suggestion into the editor in place of the failed step’s line: one undo step, never saved and never run. Studio checks it first and says what did not hold: the open file is the file the failed step ran from, its line still has the text it had when the suggestion was asked for, and the file parses with the line replaced into exactly the same steps at the same lines. The step keeps what it does: the same verb and the same flags (optional, if visible, with ai, …), from a closed list of what a suggested fix may be and may replace: tap, long press, double tap, type ... into, see, don't see and wait until ... appears or disappears. Only what the step looks for may change: a type keeps its text and its target (type "bob" into "Email" never becomes type "bob" or type "pw" into "Email") and a contains "b" keeps its value. So a tap becomes a tap that looks for something else, never open link, clear app data, kill, restart, store, a native or webview action, add media, a see ... with ai, a second step, a recipe call or a block, and never a tap ... optional where the line was not optional. The suggestion is one line of plain characters (no line break, control, hidden or bidirectional character, no unusual space) and brings nothing the line did not have: no comment, no ;, no ${ENV} or <variable>. A line with a comment, a step that holds others (repeat, if) and a step in a composite test are not replaced. After a replacement the card keeps a Re-run the failed test button: it runs the test that holds the replaced line, once per click, and shows the verdict on the card (passed, or still failing with the new error). It never saves for you: it stays disabled until you save the file (the run reads the file on disk), and while a run is going, with no device connected, in another file, or after the replaced line was edited, and it says which. If the file changes while the test runs, the verdict says it may not be the replaced line’s result. When the re-run still fails, the card offers Suggest a fix again: the same preview and the same confirmation as the first suggestion (every time for a destination that is not this machine, once per session for one that is), one request per press, and never automatic. Studio asks a provider at most three times for the same failed step (file, line and test) in a session, counts an attempt when its request goes out whatever the provider answers, numbers the card from the second suggestion (Suggested fix (2 of 3)), and refuses a fourth before anything is sent; opening another workspace starts the count again.
  • Off. With no ai: block and no saved key, AI is off, and the chat sends nothing. provider: none turns AI off even when a key is saved.
  • Answers. What a model answers, and the text of a provider’s error, is cleaned like what is sent: escapes of hidden characters are decoded and hidden characters dropped, the key and credential shapes are removed, an email address becomes [email] and this account’s own home folder becomes ~ (the folder your system names, /Users/you and /home/you, also percent-encoded), and only then is the text cut, so a cut cannot leave the start of an address. Other people’s folders are not guessed at in an answer: /users/42 and /home/settings are shown as written. The text sent with a failure still has every /Users/x, /home/x, C:\Users\x and /root replaced, which also catches text that only looks like a folder. This applies to the advice of Explain failures, to a suggested fix and its reason, to the text of a provider’s error and to the model names and the connection check details a server returns. A suggested fix that this changed shows a Changed row above the line (type "[email]" into "Email" is not what the model meant), also when only its reason changed: check it before you run it.
  • Redirects. Provider requests do not follow a redirect, so a prompt is never sent to a second host.
  • Redaction. Every system prompt and chat message is redacted before it leaves. The families are API keys of common providers (OpenAI and Anthropic style sk-, GitHub, AWS, Slack, Google and Stripe), PEM private keys, Authorization: Basic values, assignments whose name says secret, token, password or key, secret query parameters such as ?token=, a JWT, short ones included (three dot-separated parts; the first starts with eyJ and has at least 7 characters, the second at least 4, the third any or missing, so a JWT cut inside its payload is redacted too, as in eyJhbGci.eyJzdWIi.x and eyJhbGci.eyJzdWIi; letters, digits, - and _), and a card number of 13 to 19 digits that passes the Luhn check, written solid or with single spaces or hyphens (4111 1111 1111 1111). A card number glued to letters or an underscore (card_4111...) is not recognised. About one long number in ten passes that check by chance, so a long number such as a millisecond timestamp is sometimes redacted too. Every field of a chat turn (the system prompt with the open file, and each message) takes the same steps as an Explain: hidden characters and no-break spaces are dropped first, so a token split by one is whole again before it is read; the field is bounded (150,000 characters, and a cut field ends with [truncated: N more characters not sent]); the exact saved key is removed; emails are replaced ([email]) and so is this account’s own home folder (~: the folder your system names, and /Users/you, /home/you and \Users\you for your account name in any case, wherever they appear, also behind a mount prefix such as /host_mnt/Users/you or /mnt/c/Users/you, in a URL, and written with escaped slashes in JSON; a name that continues, such as /home/users/42 for /home/user, is not replaced); the families above are redacted; the field is cut to its cap. Other people’s folders are not guessed at, because /home/settings and /Users/list are routes in a test file. A tab becomes a space. A zero-width non-joiner or joiner is kept between two non-ASCII letters or symbols (Persian words, emoji sequences) and dropped anywhere else. Full-width digits and typographic dashes are read as ASCII ones only inside a run that looks like a card number; other text is sent as it is. The open file is cut back to a space, a line break, a quote or a punctuation mark, so a key or token is never sent in part (a long line with none of them, such as one line of minified JSON, CJK text or a data URI over 64 KB, sends only the [truncated] marker), and only up to the newest 40 messages of a conversation are sent, starting at one of yours (the preview and a notice with the reply say how many were left out). A token split across two messages is not found. The preview shows exactly what the request carries. The reply is cleaned the same way before it is shown (hidden characters dropped, the exact key removed, the families redacted, emails and home folders replaced), so a provider that echoes your text back shows none of it. When that changed the reply, a notice says so (a type "[email]" into "Email" line is not what the model meant).
Feature Command Needs vision
Explain failures probe test --ai-triage, probe triage --input reports/results.json, MCP triage_failure no
Propose a selector for a failed step probe test --ai-heal --format json no
Generate tests probe generate prompt -p "...", probe generate from-recording ... no
Screen assertions see "the total looks correct" with ai yes, or text mode
Visual smoke check assert no visual defects with ai yes
Read text off screen read "the OTP" with ai into otp yes
Terminal window
probe test tests/ --ai-triage

After the results are final, the model gets each failure’s error text and writes a short probable cause and a next step, printed and saved to <reports>/triage.md. Since 0.15.0 failure text names the line, the step and the visible texts/keys on screen, which is what the model reasons over. Triage is advice. It does not re-run anything or change a result.

Quality depends on the model: a 0.5B model will guess wrong; a 7B-30B model is useful. Treat the output as a hint.

When failures come from a CI run, triage the saved report later:

Terminal window
probe test tests/ --format json -o reports/results.json
probe triage --input reports/results.json -o reports/triage.md
Terminal window
probe test tests/ --ai-heal --format json -o reports/results.json

When a step runs into its timeout on a text or key selector (tap "Save", see #total), probe records the texts and widget keys that were visible on screen at that moment. The model is asked for one replacement from them, and the failed result in the JSON report gets a selector_proposal:

"selector_proposal": {
"advisory": "Proposed by a language model from the texts and keys that were on screen when the step timed out. Not applied to the test file and not verified against the app. It does not change this result.",
"line": 9,
"failed_selector": "\"Save\"",
"proposed_selector": "\"Save changes\"",
"applied": false,
"verified": false,
"model": "qwen2.5-coder:7b",
"provider": "local",
"location": "local"
}

What it does and does not do:

  • The step stays failed. The results, the summary and the exit code are the same when the model answers, fails or times out. Nothing is written to the .probe file, and the proposal is not tried against the app. You decide whether to edit the test.
  • Only what was recorded as on screen. When a step runs into its timeout, probe keeps the visible texts and widget keys as two lists of data. A proposal is kept only when it is exactly one of those texts, written as "text", or exactly one of those keys, written as #key. A part of a listed text, a text offered as a key, a key offered as a text, or any other word of the failure text is not kept, and the entry gets an unavailable reason with no proposed_selector.
  • Only step timeouts, and no list is read from the failure text. The lists are recorded when the step runs into its timeout. They are never read back out of the failure text, because the text on screen is part of it and could be written to look like a list. The app’s own “Widget not found” message also lists the visible texts, but it is plain text that a widget can shape, so it is not used: a failure that is not a step timeout is not sent to the model, and its entry says so in unavailable. The failure text keeps at most 40 characters of each listed text, and a text over that length is never proposed, even though the full text was on screen. When every listed item is that long, the unavailable reason says so.
  • Bounded. At most one request per failed step, at most 3 requests per run, each limited by ai.timeout, no retries, none after a failed request, and two minutes for all of them together. Model text is cut to 200 bytes, the API key is removed from it, and control characters, line and paragraph separators and bidirectional controls in it are replaced by spaces before it is printed or written to the report. The file name and the selector in the terminal line get the same treatment.
  • What is sent. The failed step, its selector, the failure text and the two lists, with resolved environment values masked. The failure text is what --ai-triage sends. No screenshot is sent.
  • Where it shows. The JSON report and one line per proposal in the terminal. location is local for provider: local on this machine, network for provider: local on another host, and hosted for openai and anthropic. Proposals are left out of an uploaded report. JUnit and HTML reports do not show them.
  • Off by default. Without --ai-heal, or without an ai: block, no request is made.

Most small local models cannot see images. With ai.vision: false, see "..." with ai is judged from the visible texts and widget keys the agent reports instead of a screenshot. Pixel-based steps (assert no visual defects, read ... with ai) fail with a clear message rather than guessing.

Text mode is refused when ai.redact rules exist, because redaction works on screenshot regions and cannot be applied to a list of texts. It needs flutter_probe_agent 0.15.0 or newer.

probe generate uses the ai: provider when one is set (including local). With no ai.provider it keeps its original behavior: Claude with --api-key / ai.api_key.

provider: openai and provider: anthropic send the request to that vendor directly, never through a FlutterProbe service. Triage and text mode send error text and on-screen text, which can contain user data; use provider: local if that matters. In Studio’s chat, provider: openai sends the open file with each message, as provider: anthropic already does.

See Configuration for every ai: key.