If you're a GP on the PATCHS AI Testing Team who doesn't use PATCHS in their own practice, you can read more about PATCHS in general in these articles: What is PATCHS? and Key principles of using PATCHS. For more background reading, you can access all help articles about PATCHS (for both patients and GP practices) here: help.patchs.ai 

 

Contents

What are we trying to do?

How are we doing this?

Stage 1: Initial labelling

Stage 2: Reconsideration

Stage 3: Resolution

Payment

Guidance

Imagine you work in a typical GP practice...

What is the most appropriate initial triage decision for this request?

Non-clinical

Signpost

Routine clinical

Priority clinical

Special cases

What are the main topics of this request?

Does this patient need a face-to-face appointment?

This request was difficult to understand

Notes

To answer the question 'is PATCHS AI as good as an experienced GP?' we need a sample of patient queries where we know: 1) The 'correct' triage decision, and 2) How well an 'experienced' GP performs when making triage decisions on them. We can then compare PATCHS AI's performance against the experienced GP. However, there are two challenges:

Contents

Stage 1: Initial labelling

We have recruited a team of ~30 experienced GPs from around the UK (you!) to 'label' a sample of anonymised patient queries collected from PATCHS - by labelling we mean making a clinical decision about how that query should be triaged. The section below provides guidance on how to label the patient requests. For each query there is also a section for you to write anonymous notes on why you made a particular decision (for example, if the decision wasn't a straightforward). These notes are important for stage 2 because it will remind you what your thought processes were when you made a labeling decision. Each GP is allocated the same number of queries to label at random within the PATCHS website - if you are a PATCHS user you will have a 'PATCHS AI' GP practice that you can access from a dropdown in the top menu. Each query is labeled by four GPs independently (i.e. each GP does not know what the other GPs put as their labels). If GPs 1-3 all agree we accept this as the 'correct' label for the query - the 'reference standard'.* If there is any disagreement, the query proceeds to stage 2 for 'reconsideration'. GP 4 is chosen at random and is the GP we compare against PATCHS AI. Because requests are allocated and GP 4 is chosen at random, all GPs will fairly contribute to GP 4's performance.

 

* Currently we only count agreement on high-level triage decisions described here - not the other labels.

Queries where there is any disagreement between GPs 1-3 on high-level triage decisions are allocated back to those GPs for review. The purpose is for them to reconsider their initial labels, and give them the opportunity to change them if they want. The aim is to get as many queries as possible with complete agreement between GPs 1-3. To help, they will be able to see what labels they originally put along with any notes they made. Each GP can either keep their original labels, or change them. They can also add further notes in the notes section to explain their decision - to add new notes, please entirely delete your old comment and type a completely new one. If, after reconsideration, GPs 1-3 now all agree, we accept this as the 'correct' label. If there is still any disagreement, the request proceeds to stage 3 for 'resolution'.

In stage 3, we take the highest triage label in terms of clinical acuity or importance that has been applied by at least one of the GPs in stage 2 as the reference standard. The rationale for this approach is to obtain the safest triage label for the request, in order to align with the priorities of GP staff users from our empirical research and workshops.

Payments are structured to reflect both the work undertaken by GPs and to incentivise high quality labeling. All GPs are paid one unit for each query labelled in Stage 1: Initial labelling and Stage 2: Reconsideration. So GPs 1-3 are paid the same amount for a query where they all initially agree, vs a query where there is disagreement. All GPs also receive a bonus unit of payment (so two units in total) if their final label - either applied in stage 1 or 2 - matches the final 'correct' label chosen. No bonus payments are paid for requests that go to stage 3. GPs will only be paid once they have completed labeling all their allocated queries for both stages 1 and 2 within the timeframes of the project.

 

When undertaking the labelling, please imagine you work in a fictional 'typical' GP practice. Please do not label based on what services and staff you have at your own practice. This is because we need to make sure PATCHS AI works for all GP practices. In this fictional GP practice...

There are four high-level triage decisions, each with their own low-level triage decisions. Please note that you should put your initial triage decision - which could be changed later if appropriate - for example, if you choose to send the patient to the pharmacy or direct a medication request to the script clerk, the request could always be returned to you if they disagreed. We primarily measure agreement between GPs 1-3 on their high-level decisions, and analyse agreement on the low-level decisions as a secondary measure.

 

High-level triage decisions	Low-level triage decisions
Non-clinical	Medication
	Admin clinical
	Admin non-clinical
Signpost	Self-care
	Pharmacy
	Alternative provider
Routine clinical	Routine clinical doctor
	Routine clinical non-doctor
Priority clinical	Urgent clinical doctor
	Emergency clinical doctor
	Emergency services

 

1. Medication: Requests solely asking for medication - either acute or repeat.
   • Assume that patients do not need a medication review by a clinician (including pill checks) unless explicitly stated. If they do - this will be determined by the receptionist or script clerk after receiving the request.
   • Requests asking for medication but also mentioning health problems should be triaged as health problems e.g. new or worsening symptoms, antibiotics for UTI, or steroids for asthma exacerbations etc.
   • What happens to 'Medication' requests in your fictional 'typical' GP practice? They are initially sent to a receptionist or script clerk first to issue the medication if they can (e.g. it's on repeat). This happens within 48 hours. If they can't issue it (e.g. needs medication review or is acute medication not had for a while), it will be passed to a clinician - timeframes are still adhered to.
2. Admin clinical: Admin requests that require input from a GP. Examples:
   • Fit / sick note - unless mentions new or worsening symptoms that need clinician review
   • Reviewing letters from consultants
   • Adding information to medical records
   • TWIMC letter
   • Insurance forms
   • What happens to 'Admin clinical' requests in your fictional 'typical' GP practice? They are sent to a clinician who completes the request within 1 week.
3. Admin non-clinical: Requests that should be directed to a non-clinician initially (e.g. receptionist, secretary) i.e. anything a clinician shouldn't be dealing with first. Examples:
   • Requests where it isn't clear what the patient needs initially because there isn't enough information
   • Informing patients of normal test results
     • NB patients requesting test results can be presumed normal unless they state they have been asked to contact the practice to discuss the test results
   • Incorrect phone numbers
   • Confirming or re-arranging appointments
   • Booking pre-arranged appointments for things that need an in-person visit (e.g. blood test, smear test, coil fitting, implant fitting, suture removal, flu jabs etc)
   • Chasing secondary care or other appointments e.g. hospital or scan appointments - unless mentions new or worsening symptoms that need clinician review
   • Vaccination appointment booking and chasing
   • Questions about opening hours
   • Inappropriate requests for help (e.g. facemask exemption letters, shielding letters)
   • What happens to 'Admin clinical' requests in your fictional 'typical' GP practice? They are sent to a receptionist who responds on the same day.

What happens to 'Non-clinical' requests in PATCHS?

They are ordered to the bottom of the PATCHS inbox with a purple flag.

Contents

Signpost

1. Self-care: A minor illness that does not initially need a health professionals' input to treat. The patient should initially care for themselves with self-help guidance first e.g.
   https://www.nhs.uk/conditions/sore-throat/. Patients always have the option of declining this advice or coming back to the GP if this initial treatment doesn't work. Suggested examples:
   • Viral sore throat
   • Gastroenteritis
   • Sunburn
   • What happens to 'Self-care' requests in your fictional 'typical' GP practice? Patients are directed to self-care by your receptionists first, and to contact you again if this doesn't work.
2. Pharmacy: A minor illness that does not necessarily need a clinician’s input to treat. See the pharmacy described in the typical GP practice above for more detail. Patients may initially consider going to a pharmacy for advice or over-the-counter medicines first. Patients always have the option of declining this advice or coming back to the GP if this initial treatment doesn't work. The pharmacy may also direct the patient to you after assessing the patient. Suggested examples:
   • Mouth ulcers
   • Conjunctivitis
   • Head lice
   • What happens to 'Pharmacy' requests in your fictional 'typical' GP practice? Patients are directed to contact your local pharmacy for help first by your receptionists, and to contact you again if this doesn't work.
3. Alternative provider: A non-emergency clinical request that is more appropriate to be addressed by an alternative care provider rather than a GP practice. Suggested examples:
   • Midwives / antenatal clinics e.g. someone newly pregnant that requires booking in
   • Dental problems (dentist)
   • Eye tests (opticians)
   • What happens to 'Alternative provider' requests in your fictional 'typical' GP practice? Patients are directed to contact the appropriate local provider for help first by your receptionists.

What happens to 'Signpost' requests in PATCHS?

PATCHS shows patients information links from NHS.uk relevant to their request and offers them the opportunity to cancel their request and not submit it to your GP practice. You can read more about how this happens in this article.

Contents

Routine clinical

1. Routine clinical doctor: A health problem that would traditionally require input from a GP i.e. requires a diagnosis, review, and/or new prescription. Patient is unlikely to be harmed if the problem is not addressed within the next 48 hours to 1 week. Suggested examples:
   • Infertility
   • Hair loss
   • Discussion of abnormal test results - unless results should be dealt with quicker
   • Medication reviews
   • Follow-up from a previous consultation
   • Chronic condition management
   • What happens to 'Routine clinical doctor' requests in your fictional 'typical' GP practice? They are sent to a GP who contacts the patient after 48 hours but no longer than 1 week. This includes a named GP for continuity of care.
2. Routine clinical non-doctor: A health problem that could be dealt with by a clinician who is not a GP (e.g. a nurse or HCA) i.e. that doesn't require a diagnosis or prescription. Patient is unlikely to be harmed if the problem is not addressed in the next 48 hours to 1 week. Suggested examples:
   • Contraception
   • Lifestyle counselling
   • Travel advice
   • Chronic condition monitoring
   • What happens to 'Routine clinical non-doctor' requests in your fictional 'typical' GP practice? They are sent to a relevant clinician (not a GP) who contacts the patient after 48 hours but no longer than 1 week.

What happens to 'Routine clinical' requests in PATCHS?

They are sent to a 'Clinical' inbox by Assign AI to help them get reviewed by a clinician sooner. They do not receive a priority order or flag - so remain in the middle of the PATCHS inbox.

Contents

Priority clinical

1. Urgent clinical doctor: Health problems that require input from GPs where the patient could be harmed if the problem is not addressed in the next 48 hours. Suggested examples:
   • Suspected cancer
   • Acute pain
   • What happens to 'Urgent clinical doctor' requests in your fictional 'typical' GP practice? They are sent to a GP who contacts the patient within 48 hours.
2. Emergency clinical doctor: Health problems that require input from GPs where the patient could be harmed if the problem is not addressed on the same day. Examples:
   • Suicidal thoughts with intention 
   • Asthma exacerbations
   • Infections with the potential for sepsis e.g. cellulitis
   • What happens to 'Emergency clinical doctor' requests in your fictional 'typical' GP practice? They are sent to a GP who contacts the patient on the same day.
3. Emergency services: Health problems that require an ambulance or attendance at the Emergency Department (ED) because the patient could be harmed if the problem is not addressed immediately. Examples:
   • Suspected heart attacks
   • Suspected strokes
   • What happens to 'Emergency services' requests in your fictional 'typical' GP practice?Patients are directed to call 999 or go to the ED immediately - either by your receptionist or a clinician.

What happens to 'Priority' requests in PATCHS?

For 'Urgent clinical doctor' and 'Emergency clinical doctor' requests:

• During opening hours: They are sent to a 'Clinical' inbox by Assign AI, and are marked with an orange flag and ordered to the top of the PATCHS inbox by Urgency AI. This is to help them get reviewed by a clinician sooner.
• Out of hours: PATCHS advises patients to call 111 in order to get help sooner. Patients can cancel their request and not submit it to your GP practice, and call 111 instead. You can read more about how this happens in this article.

PATCHS shows patients information links from NHS.uk relevant to their request, and gives them the opportunity to cancel their request and not submit it to your GP practice. You can read more about how this happens in this article.

For 'Emergency services', PATCHS advises patients to call 999 or go to the ED immediately at any time of day. Patients can cancel their request and not submit it to your GP practice, and call 999 or go to the ED instead. You can read more about how this happens in this article.

 

Contents

Special cases

• Photos: Always assume the patient has submitted a photo e.g. if complaining of a rash.
• Multiple requests: If the message contains multiple requests from the patient - use the one with the highest clinical priority to make your triage decision. For example, requests for fit / sick notes that mention new or worsening symptoms should be triaged for clinician review not admin.
• Medication requests: Requests solely asking for medication are most likely the patient requesting a repeat prescription e.g. contraceptive pill, antidepressants. They can therefore be issued without clinical review and triaged as 'Medication'. Only assume the patient needs a medication review if explicitly stated. Medication requests should only be triaged for review by a clinician if they explicitly state they need a medication review or mention health problems e.g. new or worsening symptoms, antibiotics for UTI, or steroids for asthma exacerbations.
• Fit / sick notes: should be triaged as 'Admin clinical' unless the request mentions new or worsening symptoms that you think requires review by a clinician.
• Alternative provider: Should only be used for non-urgent, non-emergency conditions only. They should not be used for same-day assessments e.g. A&E for an X-ray for suspected fracture, or optician for sudden onset eye symptoms.
• Triage anything that needs a pre-arranged in-person visit as 'Admin non-clinical'. For example, blood test, smear test, coil fitting, implant fitting, suture removal, flu jabs etc. This is because in practice these would not be dealt with via PATCHS, and would typically just be booked straight in by a receptionist.
• Requests where it isn't clear what the patient needs initially because there isn't enough information. For example, the patient only writes a word or two (as long as those words aren't 'chest pain' or similar!). These should generally be triaged as 'Admin non-clinical' so a receptionist can ask for more information first - they can then pass the request onto a clinician if needed afterwards.

Contents

What are the main topics of this request?

The purpose of this question is to identify requests where it would be helpful for PATCHS to ask the more questions (e.g. red flags in back pain) or questionnaires (e.g. the PHQ-9 in patients with depression). Therefore, it's likely most helpful for health problems, rather than medication or admin requests - though there are exceptions of course! At the moment, this question is testing the Mental Health AI.