What are we trying to do?
We have built world-leading Artificial Intelligence (AI) into PATCHS to decrease GP practice workload and improve patient safety by triaging patient requests automatically. As part of our ongoing clinical evaluations for our MHRA registration and UKCA marking, we want to directly compare the performance of PATCHS AI and experienced GPs when reviewing the same patient queries. We are trying to answer the question: is PATCHS AI as good as an experienced GP?
This project is testing the following PATCHS AI modules:
How are we doing this?
- There is no 'gold standard' test to determine the 'correct' decision for patient queries in primary care. We will therefore consider a triage decision as 'correct' when 3 different GPs all agree on the same triage decision. This is called a 'reference standard'.
- We need more than one experienced GP to compare against PATCHS AI for the test to be convincing. We will therefore combine the decisions of lots of different experienced GPs at random to simulate the performance of one single GP.
Our approach is summarised in this diagram:
Stage 1: Initial labelling
Stage 2: Reconsideration
Stage 3: Resolution
To access the labeling page go to www.patchs.ai and log in. If you use PATCHS in your own practice, you can access the labeling page by choosing 'PATCHS AI' in the practice dropdown menu. Below is a screenshot of the labelling screen, and a brief instructional video (it does not include any real patient information).
Imagine you work in a typical GP practice
- You do not have...
- (Advanced) Nurse Practitioners that can diagnose and treat minor illnesses
- Open access specialist services e.g. Open Access Physiotherapists
- Special arrangements with outside services e.g. opticians that deal with acute eye issues and refer to ophthalmology if necessary
- An in-house pharmacist
- You do have...
- Nurses who are skilled at LTC management like asthma reviews and pill checks etc
- Health Care Assistants (HCAs) that can do things like measure blood pressure and take blood
- Typical services nearby e.g. GUM clinic, opticians, dentist
- A local pharmacy who is equipped to deal with minor illnesses like: aches and pains, sore throat, coughs, colds, flu, earache, cystitis, skin rashes, teething, red eye. See this link for more information. After assessing the patient, if the pharmacist thinks the patient needs to see a GP, they will direct the patient to you.
- Services you do provide...
- Blood tests
- Services you don't provide...
- Ear syringing
- You are a very efficient practice and always adhere to the following timescales when dealing with patient queries...
- Emergencies - same day
- Urgent - within 48 hours (i.e. either today or tomorrow)
- Routine - after 48 hours but no longer than 1 week; you can usually accommodate continuity of care and requests for specific clinicians within this timeframe too
- Medication requests - within 48 hours (i.e. either today or tomorrow)
- Admin clinical requests (e.g. fit / sick notes, TWMIC letters) - within 1 week
- Admin non-clinical requests (e.g. receptionist needs to book a blood test or smear) - response on same day from receptionists
What is the most appropriate initial triage decision for this request?
|High-level triage decisions||Low-level triage decisions|
|Routine clinical||Routine clinical doctor|
|Routine clinical non-doctor|
|Priority clinical||Urgent clinical doctor|
|Emergency clinical doctor|
- Medication: Requests solely asking for medication - either acute or repeat.
- Assume that patients do not need a medication review by a clinician (including pill checks) unless explicitly stated. If they do - this will be determined by the receptionist or script clerk after receiving the request.
- Requests asking for medication but also mentioning health problems should be triaged as health problems e.g. new or worsening symptoms, antibiotics for UTI, or steroids for asthma exacerbations etc.
- What happens to 'Medication' requests in your fictional 'typical' GP practice? They are initially sent to a receptionist or script clerk first to issue the medication if they can (e.g. it's on repeat). This happens within 48 hours. If they can't issue it (e.g. needs medication review or is acute medication not had for a while), it will be passed to a clinician - timeframes are still adhered to.
- Admin clinical: Admin requests that require input from a GP. Examples:
- Fit / sick note - unless mentions new or worsening symptoms that need clinician review
- Reviewing letters from consultants
- Adding information to medical records
- TWIMC letter
- Insurance forms
- What happens to 'Admin clinical' requests in your fictional 'typical' GP practice? They are sent to a clinician who completes the request within 1 week.
- Admin non-clinical: Requests that should be directed to a non-clinician initially (e.g. receptionist, secretary) i.e. anything a clinician shouldn't be dealing with first. Examples:
- Requests where it isn't clear what the patient needs initially because there isn't enough information
- Informing patients of normal test results
- NB patients requesting test results can be presumed normal unless they state they have been asked to contact the practice to discuss the test results
- Incorrect phone numbers
Confirming or re-arranging appointments
- Booking pre-arranged appointments for things that need an in-person visit (e.g. blood test, smear test, coil fitting, implant fitting, suture removal, flu jabs etc)
Chasing secondary care or other appointments e.g. hospital or scan appointments - unless mentions new or worsening symptoms that need clinician review
Vaccination appointment booking and chasing
- Questions about opening hours
- Inappropriate requests for help (e.g. facemask exemption letters, shielding letters)
- What happens to 'Admin clinical' requests in your fictional 'typical' GP practice? They are sent to a receptionist who responds on the same day.
What happens to 'Non-clinical' requests in PATCHS?
They are ordered to the bottom of the PATCHS inbox with a purple flag.
- Self-care: A minor illness that does not initially need a health professionals' input to treat. The patient should initially care for themselves with self-help guidance first e.g.
https://www.nhs.uk/conditions/sore-throat/. Patients always have the option of declining this advice or coming back to the GP if this initial treatment doesn't work. Suggested examples:
- Viral sore throat
- What happens to 'Self-care' requests in your fictional 'typical' GP practice? Patients are directed to self-care by your receptionists first, and to contact you again if this doesn't work.
- Pharmacy: A minor illness that does not necessarily need a clinician’s input to treat. See the pharmacy described in the typical GP practice above for more detail. Patients may initially consider going to a pharmacy for advice or over-the-counter medicines first. Patients always have the option of declining this advice or coming back to the GP if this initial treatment doesn't work. The pharmacy may also direct the patient to you after assessing the patient. Suggested examples:
- Alternative provider: A non-emergency clinical request that is more appropriate to be addressed by an alternative care provider rather than a GP practice. Suggested examples:
- Midwives / antenatal clinics e.g. someone newly pregnant that requires booking in
- Dental problems (dentist)
- Eye tests (opticians)
- What happens to 'Alternative provider' requests in your fictional 'typical' GP practice? Patients are directed to contact the appropriate local provider for help first by your receptionists.
What happens to 'Signpost' requests in PATCHS?
PATCHS shows patients information links from NHS.uk relevant to their request and offers them the opportunity to cancel their request and not submit it to your GP practice. You can read more about how this happens in this article.
- Routine clinical doctor: A health problem that would traditionally require input from a GP i.e. requires a diagnosis, review, and/or new prescription. Patient is unlikely to be harmed if the problem is not addressed within the next 48 hours to 1 week. Suggested examples:
- Hair loss
- Discussion of abnormal test results - unless results should be dealt with quicker
- Medication reviews
- Follow-up from a previous consultation
- Chronic condition management
- What happens to 'Routine clinical doctor' requests in your fictional 'typical' GP practice? They are sent to a GP who contacts the patient after 48 hours but no longer than 1 week. This includes a named GP for continuity of care.
- Routine clinical non-doctor: A health problem that could be dealt with by a clinician who is not a GP (e.g. a nurse or HCA) i.e. that doesn't require a diagnosis or prescription. Patient is unlikely to be harmed if the problem is not addressed in the next 48 hours to 1 week. Suggested examples:
- Lifestyle counselling
- Travel advice
- Chronic condition monitoring
- What happens to 'Routine clinical non-doctor' requests in your fictional 'typical' GP practice? They are sent to a relevant clinician (not a GP) who contacts the patient after 48 hours but no longer than 1 week.
What happens to 'Routine clinical' requests in PATCHS?
They are sent to a 'Clinical' inbox by Assign AI to help them get reviewed by a clinician sooner. They do not receive a priority order or flag - so remain in the middle of the PATCHS inbox.
- Urgent clinical doctor: Health problems that require input from GPs where the patient could be harmed if the problem is not addressed in the next 48 hours. Suggested examples:
- Suspected cancer
- Acute pain
- What happens to 'Urgent clinical doctor' requests in your fictional 'typical' GP practice? They are sent to a GP who contacts the patient within 48 hours.
- Emergency clinical doctor: Health problems that require input from GPs where the patient could be harmed if the problem is not addressed on the same day. Examples:
- Suicidal thoughts with intention
- Asthma exacerbations
- Infections with the potential for sepsis e.g. cellulitis
- What happens to 'Emergency clinical doctor' requests in your fictional 'typical' GP practice? They are sent to a GP who contacts the patient on the same day.
- Emergency services: Health problems that require an ambulance or attendance at the Emergency Department (ED) because the patient could be harmed if the problem is not addressed immediately. Examples:
- Suspected heart attacks
- Suspected strokes
- What happens to 'Emergency services' requests in your fictional 'typical' GP practice?Patients are directed to call 999 or go to the ED immediately - either by your receptionist or a clinician.
What happens to 'Priority' requests in PATCHS?
For 'Urgent clinical doctor' and 'Emergency clinical doctor' requests:
- During opening hours: They are sent to a 'Clinical' inbox by Assign AI, and are marked with an orange flag and ordered to the top of the PATCHS inbox by Urgency AI. This is to help them get reviewed by a clinician sooner.
- Out of hours: PATCHS advises patients to call 111 in order to get help sooner. Patients can cancel their request and not submit it to your GP practice, and call 111 instead. You can read more about how this happens in this article.
PATCHS shows patients information links from NHS.uk relevant to their request, and gives them the opportunity to cancel their request and not submit it to your GP practice. You can read more about how this happens in this article.
For 'Emergency services', PATCHS advises patients to call 999 or go to the ED immediately at any time of day. Patients can cancel their request and not submit it to your GP practice, and call 999 or go to the ED instead. You can read more about how this happens in this article.
- Photos: Always assume the patient has submitted a photo e.g. if complaining of a rash.
Multiple requests: If the message contains multiple requests from the patient - use the one with the highest clinical priority to make your triage decision. For example, requests for fit / sick notes that mention new or worsening symptoms should be triaged for clinician review not admin.
- Medication requests: Requests solely asking for medication are most likely the patient requesting a repeat prescription e.g. contraceptive pill, antidepressants. They can therefore be issued without clinical review and triaged as 'Medication'. Only assume the patient needs a medication review if explicitly stated. Medication requests should only be triaged for review by a clinician if they explicitly state they need a medication review or mention health problems e.g. new or worsening symptoms, antibiotics for UTI, or steroids for asthma exacerbations.
- Fit / sick notes: should be triaged as 'Admin clinical' unless the request mentions new or worsening symptoms that you think requires review by a clinician.
- Alternative provider: Should only be used for non-urgent, non-emergency conditions only. They should not be used for same-day assessments e.g. A&E for an X-ray for suspected fracture, or optician for sudden onset eye symptoms.
- Triage anything that needs a pre-arranged in-person visit as 'Admin non-clinical'. For example, blood test, smear test, coil fitting, implant fitting, suture removal, flu jabs etc. This is because in practice these would not be dealt with via PATCHS, and would typically just be booked straight in by a receptionist.
- Requests where it isn't clear what the patient needs initially because there isn't enough information. For example, the patient only writes a word or two (as long as those words aren't 'chest pain' or similar!). These should generally be triaged as 'Admin non-clinical' so a receptionist can ask for more information first - they can then pass the request onto a clinician if needed afterwards.
What are the main topics of this request?The purpose of this question is to identify requests where it would be helpful for PATCHS to ask the more questions (e.g. red flags in back pain) or questionnaires (e.g. the PHQ-9 in patients with depression). Therefore, it's likely most helpful for health problems, rather than medication or admin requests - though there are exceptions of course! At the moment, this question is testing the Mental Health AI.
- Only label topics that are substantially mentioned. For example, if mentioned in a way that would make you want to discuss it further with the patient. Don't label topics only mentioned in passing.
- Select more than one topic if more than one is mentioned. If multiple clinical topics are mentioned, please select all relevant labels.
- 'Other' is a helpful answer! Knowing a request mentions other topics helps us identify negative examples. You can select 'Other' on its own where no 'named' topics are mentioned at all: for example, a request about hair loss. You can also select 'Other' if another topic is mentioned in addition to other 'named' clinical topics: for example, if depression, back pain, and eczema are all mentioned in one request you should select 'Depression', 'Back pain', and 'Other' (for eczema).
- Do not label requests solely asking for medication as the condition they treat. Only label them with a clinical topic if they also mention health problems that require review by a clinician. E.g. Do not label requests solely asking for antidepressants as 'depression' unless they mention symptoms like worsening low mood or suicidal thoughts.
Does this patient need a face-to-face appointment?
This question is trying to find out whether you think the patient needs an in-person appointment to resolve their query. It is testing the Face-to-Face (F2F) AI.
The question should only apply to requests that would be passed onto a clinician to resolve i.e. requests you have triaged as Routine clinical or Priority clinical.
Always answer 'No' if you have triaged the request as either Non-clinical or Signpost - even if the request does technically end up in as an in-person appointment (e.g. smear test, flu jab etc). This is because only clinicians who use PATCHS can say whether the patient needs an in-person appointment - not receptionists or other non-clinical staff.
This request was not written in English
Tick this box if the patients' request is written in a non-English language.
This section is for you to write notes on your rationale for choosing a particular label, and is optional. We would recommend writing notes for any labels you have applied where the decision hasn't been straightforward. You should only write notes that you would be comfortable with other GPs in the project reading because they will be anonymously shared with GPs 1-3 during stage 2 (reconsideration) and potentially also with the expert panel during stage 3 (resolution).