Check model backend health and recover GPU work

Check the affected server

Open AI → Models and check the selected server. Read Backend readiness separately from the model's memory status.

What you see	Meaning
In memory	The model is loaded. This alone does not prove that a GPU worker is ready.
Worker ready	The backend worker reports ready. Confirm the actual workload with safe input.
Unavailable with successful model tests required	A saved health hold prevents reuse of this backend until the required manual tests succeed.

Review Saved backend health

1. Open platform Configuration and its runtime settings.
2. Find Saved backend health. It appears when there is relevant failure history, a storage problem, or a compatibility notice.
3. Check the displayed server identity. The panel describes the local server, not every node at once.
4. Read the model, backend, failure category, time and current status. Separate active holds from historical failures.
5. Expand Diagnostic details when you need a safe request reference or model, runtime and hardware identity for investigation.

A failure record is evidence that work failed. It does not by itself identify a driver defect. Keep prompts, document contents, passwords and private files out of support messages.

Understand the saved hold

Two separate failure waves within 24 hours can quarantine a particular model, GPU backend, hardware and runtime combination. A quarantine is a saved hold on using that combination.

Time passing, restarting the service, or changing settings does not clear it. Loading the model again does not prove the held backend is healthy.

Automatic processing retains CPU recovery where policy permits it. A manually selected GPU only policy remains strict. It waits or reports the backend unavailable instead of silently using CPU.

Recover and retest

1. Preserve the diagnostic details and investigate the first failure before repeatedly retrying work.
2. If suitable for your workload, use permitted CPU processing while investigating. Check the observed backend and response on an actual request.
3. Choose Open model testing from the health panel, or open Operations → Model testing.
4. Request a manual test for the affected model, server and workloads. Do this when the server has capacity for testing.
5. Each profile runs in three fresh worker processes. Every required pass must succeed. The reported result keeps the slowest throughput and highest memory use.
6. Check the final result and health state. Clearing a hold requires successful GPU tests covering every affected workload with matching identities and no newer serving failure.
7. Run a small real generation or embedding request, then check the observed backend before restoring normal workload.

Automatic testing cannot clear a saved quarantine. The first failed GPU task stops further tests on that backend in the same run. Investigate the new result before requesting another test.

A changed model, driver or runtime is a different combination and needs its own evidence. Restart the service after a driver change, during an agreed maintenance window.

If health storage is unavailable

An unreadable, corrupt, full or unwritable health record can hold GPU work. Follow the displayed message and involve the installation operator.

Preserve the original record. Repair the storage problem while the service is stopped. Do not delete the record or clear saved failures to bypass testing.

Intel Vulkan compatibility notices

This rebuild includes a compatibility rule for one precisely matched Windows Intel GPU, driver and runtime combination. The configuration page reports when that rule is active.

The rule does not apply to every Intel GPU, multiple adapter configuration, or driver version. Do not copy internal environment settings to other machines. Check the notice and test the actual workload.

Next step

Read model scheduling and model and knowledge troubleshooting. Keep your package's acceptance limits separate from saved settings.