What are we trying to do?
We have built world-leading Artificial Intelligence (AI) into PATCHS to decrease GP practice workload and improve patient safety by triaging patient requests automatically. As part of our ongoing clinical evaluations for our MHRA registration and UKCA marking, we want to directly compare the performance of PATCHS AI against experienced GPs when reviewing the same patient queries. We are trying to answer the question: is PATCHS AI as good as an experienced GP at triaging patient requests? This is a research project that we're conducting in collaboration with The University of Manchester.
There are three types of triage decision we want to test:
- Urgency: How urgent a request is, and whether it needs input from a clinician. To test Urgency AI, Signpost AI, Assign AI.
- Clinical topics: What the clinical topics are of a request. To test Topic AI.
- Face-to-face: Whether a request needs a face-to-face appointment. To test F2F AI.
How are we doing this?
- There is no 'gold standard' test to determine the 'correct' decision for patient queries in primary care. We will therefore consider a triage decision as 'correct' when 3 different GPs all agree on the same triage decision. This is called a 'reference standard'.
- We need more than one experienced GP to compare against PATCHS AI for the test to be convincing. We will therefore combine the decisions of lots of different experienced GPs at random to simulate the performance of one single GP.
For each of the three triage decision tests, our approach can be summarised in this diagram:
Stage 1: Initial labelling
Stage 2: Reconsideration
Stage 3: Resolution
- Urgency. If two of three GPs agree, we will accept that label as the reference standard. If there aren’t 2 out of 3 GPs that agree, we will take the highest (most urgent) triage label in that has been applied by any of the GPs in Stage 2 as the reference standard. The rationale for this approach is to obtain the safest triage label for the request, in order to align with the priorities of GP staff users from our empirical research and workshops.
- Clinical topics. If two of three GPs agree, we will accept that label as the reference standard. If there aren’t 2 out of 3 GPs that agree, we will remove those queries from the study as they won't be clear cut and may refer to lots of different clinical topics.
- Face-to-face. Because there are just two possible labels (face-to-face appointment needed or not), there will always be 2 out of 3 GPs that agree, so we will accept that label as the reference standard.
|1||Initial labelling for all triage decisions (Stage 1)||14th Nov 2022 - 1st Dec 2022|
|2||Urgency reconsideration (Stage 2)||Mid-Dec 2022 - Late Dec 2022|
|3||Clinical topics reconsideration (Stage 2)||Early Jan 2023 - Mid Jan 2023|
|4||Face-to-face reconsideration (Stage 2)||Mid Jan 2023 - Late Jan 2023|
- You will be paid £1 for each query you label in Stage 1.
- You will receive a bonus payment of £1 if any of your labels for any of the three triage decisions (urgency, clinical topics, face-to-face) applied either in Stages 1 or 2 match the final 'correct' label (reference standard). So if your 3 triage labels all match the reference standard in Stage 1 you will be paid £4 for labelling that query just once.
- No bonus payments are paid for requests that go to Stage 3.
- Payments will only be made once you have completed all your allocated labels for all four labelling rounds within the timeframes set.
To access the labelling page go to www.patchs.ai and log in. If you use PATCHS in your own practice, you can access the labeling page by choosing 'PATCHS AI' in the practice dropdown menu. Below is a screenshot of the labelling screen, and a brief instructional video (it doesn't include any real patient information).
The guidance below isn't supposed to be prescriptive and is intended to draw on your implicit knowledge as GPs rather than tell you exactly what to do!
Imagine you work in a typical GP practice
- You do not have...
- (Advanced) Nurse Practitioners that can diagnose and treat minor illnesses
- Open access specialist services e.g. Open Access Physiotherapists
- Special arrangements with outside services e.g. opticians that deal with acute eye issues and refer to ophthalmology if necessary
- An in-house pharmacist that does things like reviews or changes medication
- You do have...
- Nurses who are skilled at LTC management like asthma reviews and pill checks etc
- Health Care Assistants (HCAs) that can do things like measure blood pressure and take blood
- Typical services nearby e.g. GUM clinic, opticians, dentist
- A local pharmacy who is equipped to deal with minor illnesses like: aches and pains, sore throat, coughs, colds, flu, earache, cystitis, skin rashes, teething, red eye. See this link for more information. After assessing the patient, if the pharmacist thinks the patient needs to see a GP, they will direct the patient to you.
- Services you do provide...
- Blood tests
- Services you don't provide...
- Ear syringing
- You are a very efficient practice and always adhere to the following timescales when dealing with patient queries...
- Emergencies - same day
- Urgent - within 48 hours (i.e. either today or tomorrow)
- Routine - after 48 hours but no longer than 1 week; you can usually accommodate continuity of care and requests for specific clinicians within this timeframe too
- Medication requests - within 48 hours (i.e. either today or tomorrow)
- Clinical admin requests (e.g. fit / sick notes, TWMIC letters) - within 1 week
- Non-clinical admin requests (e.g. receptionist needs to book a blood test or smear) - response on same day from receptionists
Urgency: What is the most appropriate initial triage decision for this request?
|High-level triage labels||Low-level triage labels|
|Non-clinical||Admin - non-clinical|
|Routine||Admin - clinical|
|Routine nurse or HCA|
Non-clinical requests are requests that should be directed to a non-clinician initially (e.g. receptionist, secretary).
What happens to 'Non-clinical' requests in PATCHS?
Signpost AI shows patients information links from NHS.uk relevant to their request and offers them the opportunity to cancel their request and not submit it to your GP practice if they have addressed their query. If the patient chooses to submit their request to their GP practice anyway, they are highlighted by a purple flag and ordered to the bottom of the PATCHS inbox.
Admin - non-clinical
These are administrative requests that should be dealt with by non-clinicians such as receptionists, secretaries or practice managers. Examples may include:
- Incorrect phone numbers
Confirming or re-arranging appointments
- Booking pre-arranged appointments for things that need an in-person visit (e.g. blood test, smear test, coil fitting, implant fitting, suture removal, flu jabs etc)
Chasing secondary care or other appointments e.g. hospital or scan appointments - unless mentions new or worsening symptoms that need clinician review
Vaccination appointment booking and chasing
- Questions about opening hours
- Inappropriate requests for help (e.g. facemask exemption letters, shielding letters)
What happens to 'Admin clinical' requests in your fictional 'typical' GP practice? They are sent to a receptionist who responds on the same day.
These are requests solely asking for medication - either acute or repeat. Tips:
- Only include requests for medication that you'd be happy to issue without a review from a clinician.
- For repeat medication, assume that patients do not need a medication review by a clinician unless explicitly stated.
- Requests asking for medication but also mentioning health problems should be triaged as something else e.g. new or worsening symptoms, antibiotics for UTI, or steroids for asthma exacerbations etc.
What happens to 'Medication' requests in your fictional 'typical' GP practice? They are initially sent to a receptionist or script clerk first to issue the medication if they can (e.g. it's on repeat). This happens within 48 hours. If they can't issue it (e.g. the patient requires a medication review or is acute medication not had for a while), it will be passed to a clinician - timeframes are still adhered to.
Routine requests are requests that should be non-urgently directed to a clinician initially (e.g. GP, nurse, or HCA).
What happens to 'Routine clinical' requests in PATCHS?
Signpost AI shows patients information links from NHS.uk relevant to their request and offers them the opportunity to cancel their request and not submit it to your GP practice if they have addressed their query. If the patient chooses to submit their request to their GP practice anyway, Assign AI sends the request to a 'Clinical' inbox to help it get reviewed by a clinician sooner. These requests do not receive a priority order or flag - so remain in the middle of the PATCHS inbox.
Admin - clinical
These are administrative requests that require input from a clinician - typically a GP. Examples may include:
- Fit / sick notes - unless the request mentions new or worsening symptoms that need a clinician review
- Reviewing letters from consultants
- Adding information to medical records
- TWIMC letters
- Insurance forms
What happens to 'Admin - clinical' requests in your fictional 'typical' GP practice? They are sent to a clinician who completes the request within 1 week.
These are minor illnesses that can initially either be dealt with by the patients themselves with self-help guidance (e.g. https://www.nhs.uk/conditions/sore-throat/) or by going to a pharmacy for advice and/or over-the-counter medication. See the pharmacy described in the typical GP practice above for more detail. As with traditional ways of requesting help - patients have the option of declining this advice or coming back to the GP if this initial treatment doesn't work. The pharmacy may also direct the patient to you after assessing the patient. Examples may include:
- Viral sore throat
- Mouth ulcers
- Head lice
What happens to 'Pharmacy' requests in your fictional 'typical' GP practice? Patients are directed to self-care or to contact your local pharmacy for help first by your receptionists, and to contact you again if this doesn't work.
These are non-emergency clinical issues that are more appropriate to be addressed by an alternative care provider rather than a GP practice. Examples may include:
- Midwife appointments / antenatal clinics e.g. someone newly pregnant that requires booking in
- Dental problems (dentist)
- Eye tests (opticians)
- STI tests (GUM clinic)
What happens to 'Alternative provider' requests in your fictional 'typical' GP practice? Patients are directed to contact the appropriate local provider for help first by your receptionists.
These are health problems that would traditionally require input from a GP i.e. requires a diagnosis, review, and/or new prescription. The patient is unlikely to be harmed if the problem is not addressed within the next 48 hours to 1 week. Examples may include:
- Hair loss
- Longstanding non-serious conditions
- Medication reviews
- Follow-up from a previous consultation
- Chronic condition management
What happens to 'Routine GP' requests in your fictional 'typical' GP practice? They are sent to a GP who contacts the patient after 48 hours but no longer than 1 week. This includes a named GP for continuity of care.
Routine nurse or HCA
These are health problems that could be dealt with by a clinician who is not a GP (e.g. a nurse or HCA) i.e. that doesn't require a diagnosis or new prescription. The patient is unlikely to be harmed if the problem is not addressed in the next 48 hours to 1 week. Examples may include:
- Lifestyle counselling
- Travel advice
- Smoking cessation
- Chronic condition monitoring
What happens to 'Routine nurse or HCA' requests in your fictional 'typical' GP practice? They are sent to a relevant clinician (not a GP) who contacts the patient after 48 hours but no longer than 1 week.
Priority requests are requests that should be urgently directed to a GP (including out of hours [OOH]) or to emergency services (999 or A+E).
What happens to 'Priority' requests in PATCHS?
Signpost AI signposts patients to contact 111 if they submit an urgent request OOH, and signposts them to call 999 or go to A+E if they submit an emergency request at any time. Patients can override this advice and submit their request to their GP practice at their own risk. If they do this, Assign AI sends the request to a 'Clinical' inbox to help it get reviewed by a clinician sooner. Urgency AI also highlights emergency requests with a red flag and orders them to the top of the PATCHS inbox. Urgency AI highlights urgent requests with an orange flag and orders them to the top of the PATCHS inbox below emergencies.
These are health problems that require input from GPs where the patient could be harmed if the problem is not addressed in the next 48 hours. Examples may include:
- Suspected cancer
- Acute pain
- Infections without the potential for sepsis e.g. ear piercing infections
What happens to 'Urgent GP' requests in your fictional 'typical' GP practice? They are sent to a GP who contacts the patient within 48 hours.
These are health problems that require input from GPs where the patient could be harmed if the problem is not addressed on the same day. Examples may include:
- Suicidal thoughts
- Asthma exacerbations
- Infections with the potential for sepsis e.g. cellulitis
What happens to 'Emergency GP' requests in your fictional 'typical' GP practice? They are sent to a GP who contacts the patient on the same day.
These are health problems that require same-day hospital treatment because the patient could be harmed if the problem is not addressed immediately. This includes calling 999 or attending A+E, and 'direct to specialty' hospital admissions if you referred the patient.
999 or A+E examples may include:
- Heart attacks
- Pulmonary embolism
- Suspected fractures
'Direct to specialty' hospital admissions may include:
- Visual loss (ophthalmology)
- Appendicitis (surgeons)
- Cauda equina (spinal surgeon)
- Paraphimosis (urology)
What happens to '999 / A+E' requests in your fictional 'typical' GP practice? Patients are directed to call 999 or go to A+E immediately - either by your receptionist or a clinician - or are referred 'direct to speciality' by a GP.
- Photos. Always assume the patient has submitted a photo (although you won't be able to see it) e.g. if complaining of a rash.
Multiple requests. If the message contains multiple requests from the patient - use the one with the highest clinical priority to make your triage decision. For example, requests for fit / sick notes that mention new or worsening symptoms should be triaged for clinician review not admin.
- A patient mentions a health problem of someone else. Then that should be taken into account and treated in the same way as a request with multiple problems in above (i.e. the highest clinical priority problem should determine your decision). E.g. if the patient has largely written a request about their depression, but mentions they think their neighbour is having a heart attack, we should triage it as a heart attack because the AI shouldn’t be relied upon to distinguish between who has written the request
- Medication requests. Requests solely asking for medication are most likely the patient requesting a repeat prescription e.g. contraceptive pill, antidepressants. They can therefore be issued without clinical review and triaged as 'Medication'. Only assume the patient needs a medication review if explicitly stated. Medication requests should only be triaged for review by a clinician if they explicitly state they need a medication review or mention health problems e.g. new or worsening symptoms, antibiotics for UTI, or steroids for asthma exacerbations.
- Fit / sick notes. should be triaged as 'Admin - clinical' unless the request mentions new or worsening symptoms that you think requires review by a clinician.
- Alternative provider. Should only be used for non-urgent, non-emergency conditions only. They should not be used for same-day assessments e.g. A&E for an X-ray for suspected fracture, or optician for sudden onset eye symptoms.
- Triage anything that needs a pre-arranged in-person visit as 'Admin non-clinical'. For example, blood test, smear test, coil fitting, implant fitting, suture removal, flu jabs etc. This is because in practice these would not be dealt with via PATCHS, and would typically just be booked in by a receptionist.
- The patient hasn't put enough detail. There are requests where it's unclear what the patient wants or what their problem is. These are really difficult to deal with and difficult to give advice on. What we suggest in this situation is to use your best judgment as to what you would do in your own practice. Sorry if this sounds like a cop out!
- Urgent nurse. In your own practice you may send patients to see a nurse urgently, for example to dress a wound. We don’t have an ‘urgent nurse’ category because not all nurses do urgent things like dress wounds, and arguably anything urgent probably needs a GP review first before. However, either way the patient needs urgent help, so please label them as ‘Urgent GP’.
Clinical topics: What are the main topics of this request?The purpose of this question is to test Topic AI, which asks patients specific questions (e.g. red flags in back pain) or questionnaires (e.g. the PHQ-9 in patients with depression) based on the clinical topics of their query. Therefore, it's likely most helpful for health problems, rather than medication or admin requests - though there are exceptions of course!
- Only label topics that are substantially mentioned. For example, if mentioned in a way that would make you want to discuss it further with the patient. Don't label topics only mentioned in passing.
- Select more than one topic if more than one is mentioned. If multiple clinical topics are mentioned, please select all relevant labels.
- 'Other' is a helpful answer! Knowing a request mentions other topics helps us identify negative examples. You can select 'Other' on its own where no 'named' topics are mentioned at all: for example, a request about hair loss. You can also select 'Other' if another topic is mentioned in addition to other 'named' clinical topics: for example, if depression, back pain, and eczema are all mentioned in one request you should select 'Depression', 'Back pain', and 'Other' (for eczema).
- Do not label requests solely asking for medication as the condition they treat. Only label them with a clinical topic if they also mention health problems that require review by a clinician. E.g. Do not label requests solely asking for antidepressants as 'depression' unless they mention symptoms like worsening low mood or suicidal thoughts. Otherwise, please label medication requests as 'Other'.
Face-to-Face: Does this patient need a face-to-face appointment?
The purpose of this question is to test F2F AI. It's trying to find out whether you think the patient needs an in-person appointment to resolve their request.
The question should only apply to requests that would be passed onto a clinician to resolve i.e. requests you have triaged as Routine or Priority.
Always answer 'No' if you have triaged the request as a Non-clinical triage decision - even if the request does technically end up in as an in-person appointment (e.g. receptionist books the patient in for a smear test, flu jab etc). This is because only clinicians who use PATCHS can say whether the patient needs an in-person appointment - not receptionists or other non-clinical staff.
- Requests that you may be able to deal with remotely (e.g. via telephone or using photos). Even though many requests can now be dealt with remotely that would have previously been automatically dealt with face-to-face (e.g. rashes, tonsillitis), it's probably still best to mark these as needing a face-to-face appointment, as sometimes a telephone consultation +/- photos aren't sufficient and often you have to bring these patients in anyway. This can duplicate patient contacts / workload for the same request, which is something we want to avoid.
Patients requesting face-to-face appointments. This should be a judgment call as to what you would do in your normal clinical practice. If the patient wants a face-to-face appointment but you don’t think it’s necessary then don't label as needing one. But if you think it would be reasonable - go ahead.
- Requests where you think the patient should go elsewhere e.g. pharmacy, 999, A&E. They should still be marked as needing a face-to-face appointment if you think they need one. It's just that it ideally it wouldn't be the GP practice doing giving them one.
This request was not written in English
Tick this box if the request is written in a non-English language. You will still be asked for a triage decision. Don’t worry about what triage decision you put - just put anything down as we’ll remove that request from the database.
This section is for you to write notes on your rationale for choosing a particular label, and is optional. We would recommend writing notes for any labels you have applied where the decision hasn't been straightforward. You will be able to review your notes during the Stage 2 labelling rounds to remind yourself of why you made a decision, and add further notes if you wish. In Stage 2 the other GPs that labeled the same requests as you will also be able see the notes you've written (and you'll be able to see theirs too). Your notes will also be visible to the research team. So please don't write anything you wouldn't be happy being shared with others!