Introduction
One of our key objectives with PATCHS is to build world-leading Artificial Intelligence (AI) to relieve GP practice workload, and get patients the help they need quicker. Each time GP practice staff use PATCHS, they are teaching it how to deal with patient requests. So far, we have built two AIs that are currently being used in my practice:
- PATCHS AI Triage, which makes suggestions about the urgency of patients' requests.
- PATCHS AI Mental Health, which identifies when patients requests relate to anxiety and/or depression, and asks the patient to complete a GAD-7 or PHQ-9 questionnaire automatically.
The AIs are working well in my practice, and appear to be safe and save us time - but before we release them to other practices more widely we want to test this scientifically.
The question we're trying to answer is: are these AIs as good as an experienced GP at their particular task? (So, for PATCHS AI Triage it's about triaging patient requests, and for PATCHS AI Mental Health it's about identifying requests relating to anxiety and/or depression).
Methods
- The 'correct' decision for each e.g. the correct triage decision, and whether the request relates to anxiety and/or depression.
- How well an 'experienced' GP performs when they make the same decisions.
We can then compare how well the AIs perform in being 'correct' vs how well the GP performs. The challenge is that:
- There is no 'gold standard' test to determine the 'correct' decision for patient requests.
- We need more than one experienced GP to compare against the AI for the test to be convincing.

Stage 2: Reconsideration
Stage 3: Resolution
Payment
Stage 1: Labeling
This section explains how to do labeling stage 1 in more detail.
To access the labeling page you should go to www.patchs.ai and log in. If you use PATCHS during your normal clinical job, then you can access the labeling page by choosing 'PATCHS AI' in your practice dropdown menu.
Here's what the labeling screen looks like:
And here's a brief instructional video of how to do the labeling (this video doesn't include any real patient information):
The rest of the information below is specific to each of the labels you will apply during labeling.
What is the most appropriate initial triage decision for this request?
Non-clinical (High-level)
- Medication: any request solely asking for medication - either acute or repeat.
- Administrative clinical: admin request that requires input from a GP. Examples:
- Fit / sick note
- TWIMC letter
- Insurance forms
- Administrative non-clinical: admin request that would be best directed to a non-clinician (e.g. receptionist, secretary). Examples:
- Booking pre-arranged appointments (e.g. blood tests, blood pressure measurement)
- Requesting information about opening hours
- Inappropriate requests for help (e.g. facemask exemption letters, shielding letters)
Marking a request as 'non-clinical' would send it to the bottom of the inbox in PATCHS and would be dealt with last:
Signpost (High-level)
Low-level triage decisions are:
- Self-care: A minor illness that does not initially need a health professionals' input to treat. The patient should initially care for themselves with self-help guidance. Examples:
- Viral sore throat
- Gastroenteritis
- Sunburn
- Pharmacy: A minor illness that does not necessarily need a clinician’s input to treat. Patients may initially consider going to a pharmacy for advice or over-the-counter medicines. Examples:
- Mouth ulcers
- Conjunctivitis
- Head lice
- Alternative provider: A request that is more appropriate to be addressed by an alternative care provider rather than a GP practice. Examples:
- Dental problems
- Open access physiotherapy
- Ear syringing
Marking a request as 'signpost' would divert the patient away from the practice so the request wouldn't appear in the practice's inbox and wouldn't be dealt with by the practice at all.
Routine clinical (High-level)
Low-level triage decisions are:
- Routine clinical doctor: A health problem that would traditionally require input from a GP i.e. requires a diagnosis and prescription. In some practices this may be performed by a specially trained nurse or ANP. Patient is unlikely to be harmed if the problem is not addressed within the next 48 hours. Examples:
- Migraine
- Hair loss
- Follow-up from a previous consultation
- Chronic condition management
- Routine clinical non-doctor: A health problem that could be dealt with by a clinician who is not a GP (e.g. a nurse or HCA) i.e. that doesn't necessarily require a diagnosis or prescription. Patient is unlikely to be clinically harmed if the problem is not addressed in the next 48 hours. Examples:
- Contraception management
- Lifestyle counselling
- Travel advice
- Chronic condition monitoring
Marking a request as 'routine clinical' would send it straight to a clinician in the middle of their inbox in PATCHS:
Priority clinical (High-level)
Low-level triage decisions are:
- Urgent clinical doctor: Health problems that require input from GPs where the patient could be harmed if the problem is not addressed in the next 48 hours. Examples:
- Urinary tract infections
- Cellulitis.
- Emergency clinical doctor: Health problems that require input from GPs where the patient could be harmed if the problem is not addressed on the same day. Examples:
- Suicidality
- Asthma exacerbations
- Suspected cancer.
- Emergency services: Health problems that require an ambulance or attendance at the Emergency Department (ED) because the patient could be harmed if the problem is not addressed immediately. Examples:
- Suspected heart attacks
- Suspected strokes
Marking a request as 'priority clinical' would send it straight to a GP and put it at the top of their inbox in PATCHS and would be dealt with first:
What is the main topic of this request?
The purpose of this question to identify requests that could be further evaluated by PATCHS asking the patient more relevant questions (e.g. red flags in back pain) or questionnaires (e.g. the PHQ-9 in patients presenting with depression). Therefore, it's likely most helpful for health problems, rather than medication or admin requests - though there are exceptions of course!This request was difficult to understand
Check this box if the patients' request is written in a non-English language.
Notes
This section is for you to write notes on your rationale for choosing a particular label, and is optional. We would recommend writing notes for any labels you have applied where the decision hasn't been straightforward. You should only write notes that you would be comfortable with other GPs in the project reading because they will be anonymously shared with GPs 1-3 during stage 2 (reconsideration) and potentially also with the expert panel during stage 3 (resolution).