If you're a GP on the PATCHS AI Testing Team who doesn't use PATCHS in your own practice, you can read more about PATCHS in these articles: What is PATCHS? and Key principles of using PATCHS. For more background reading, you can access all help articles about PATCHS (for both patients and GP practices) here: help.patchs.ai 

 

Contents

What are we trying to do?

How are we doing this?

Stage 1: Initial labelling

Stage 2: Reconsideration

Stage 3: Resolution

Payment

Guidance

Imagine you work in a typical GP practice...

Urgency: What is the most appropriate initial triage decision for this request?

Non-clinical

Routine

Priority

Special cases

Clinical topics: What are the main topics of this request?

Face-to-face: Does this patient need a face-to-face appointment?

This request was not written in English

Notes

To answer the question 'is PATCHS AI as good as an experienced GP?' we need a sample of patient queries where we know: 1) The 'correct' triage decision, and 2) How well an 'experienced' GP performs when making triage decisions on them. We can then compare PATCHS AI's performance against the experienced GP. However, there are two challenges:

Contents

Stage 1: Initial labelling

We have recruited a team of ~30 experienced GPs from around the UK (you!) to 'label' a sample of anonymised patient queries collected from PATCHS - by labelling we mean making a clinical decision about how that query should be triaged. The section below provides guidance on how to label the patient requests. For each query there is also a section for you to write notes on why you made a particular decision (for example, if the decision wasn't a straightforward). These notes are important for Stage 2 because they will remind you and inform the other GPs what your thought processes were when you made a labeling decision. Each GP will be allocated the same number of queries to label at random within PATCHS - if you are already a PATCHS user you can access your queries in the 'PATCHS AI' practice from in the top menu dropdown. Each query will be labeled by four GPs independently (i.e. each GP will not know what the other GPs put). One GP (GP4) will be chosen at random as the GP we compare against PATCHS AI. Because requests are allocated at random and GP4 is chosen at random, all GPs will equally contribute to GP4's performance. The other three GPs (GP 1-3) will determine the 'correct' label for the query (the 'reference standard'). If the labels put by GPs 1-3 all agree we accept this as the reference standard (for urgency triage decisions, we measure agreement on the 'high-level' categories only - see below for further information). If there is any disagreement, the query proceeds to Stage 2 for 'reconsideration'. 

 

Video instructions for stage 1 (please just use this video for general information on how to do the labelling - please use the information written in this article for specific guidance on how to make the triage decisions as some of the information in the video is incorrect e.g. 39.50 marking a request as not requiring a face-to-face that has been triaged to the pharmacy):

Queries where there is any disagreement between GPs1-3 are allocated back to those GPs for reconsideration. The purpose is for you to review your initial labels, and change them if you want, in order to achieve agreement with the other GPs. You will be able to see what labels you originally put along with any notes you made in Stage 1.

 

You will also be able to see what the other GPs put who looked at the same request as you. This will be given to you in a separate text file that looks like this:

 

 

The 'number' refers to the number at the end of the webpage address (URL) for that request in PATCHS. It's the way to check you're looking at the right request in the file (along with checking the other details too e.g. date submitted, age, gender, submitted by):

 

 

The file will only show triage decisions where there currently isn't full agreement between all 3 GPs. Therefore, if a label isn't shown - there's no need to reconsider / change it. For example in the screenshots above, number 62219 only shows 'Urgency' and 'F2F' - so you would only have to look at those two labels - there's already complete agreement on 'Clinical topics' so that can be left alone. For number 62341, only 'Urgency' is shown, so 'F2F' and 'Clinical topics' can be ignored. 

 

The aim is to get complete agreement across all triage labels. Full agreement can only be achieved by people changing their labels. All 3 GPs get the file that shows what the other two GPs put - not just the ones who are in the minority. So please do look at what the other GPs have put and let that help you make your choice. Anyone can, and should, change their labels if they think they got it wrong in Stage 1 after reading what the other two GPs put. That includes those who are in the majority, though understandably it often makes most sense for those in the majority to not change their answers unless they have a strong reason to do so. We’re trying to make the process as fair and unbiased as possible - we're not just trying to coerce whoever’s in the minority to change their decision just to get agreement.

 

A special mention regarding the 'Clinical topics' triage decision. As you know, multiple topics can be added to the 'Clinical topics' triage decision. We treat each topic individually, so as long as there’s agreement between all 3 GPs on one label then we will accept that request as fully agreed. For example, if one GP puts 'depression' and 'back pain', and the other two GPs only put 'depression' - this would count as agreement and would not have to proceed to Stage 3 (resolution). Agreement would only be on ‘depression’ though. To get agreement on ‘back pain’ too the first GP would have to remove their ‘back pain' label or the other two would have to add a ‘back pain’ label - though this wouldn’t be necessary to avoid going to Stage 3.

 

You can still add notes in the notes section to explain your decision if you change or keep your label (though they’re not mandatory) - to add new notes, please entirely delete your old comment and type a completely new one. New notes won’t be seen by anyone else except the research team - we will look at the notes to see why certain decisions were made in Stage 3.

 

If, after reconsideration, GPs 1-3 now all agree, we accept this as the reference standard. If there is still any disagreement, the request proceeds to Stage 3 for 'resolution'.

 

Video instructions for stage 2 (please just use this video for general information on how to do the labelling - please use the information written in this article for specific guidance on how to make the triage decisions):

 

To get the reference standard for queries where there is still disagreement after Stage 2 are resolved in different ways depending on the triage decision:

1. Urgency. If two of three GPs agree in Stage 2, we will accept that label as the reference standard. If there aren’t 2 out of 3 GPs that agree in Stage 2, we will take the highest (most urgent) triage label in that has been applied by any of the GPs in Stage 2 as the reference standard. The rationale for this approach is to obtain the safest triage label for the request, in order to align with the priorities of GP staff users from our empirical research and workshops.
2. Clinical topics. If two of three GPs put the same label in Stage 2, we will accept that label as the reference standard. If there aren’t 2 out of 3 GPs that put the same label in Stage 2, we will assign it the label 'Other'.
3. Face-to-face. Because there are just two possible labels (face-to-face appointment needed or not), there will always be 2 out of 3 GPs that agree in Stage 2, so we will accept that label as the reference standard.

Payments are structured to reflect the work undertaken and to incentivise high quality labelling:

The exact number of queries you will be allocated depends on how many GPs participate in total (because they will be divided up equally between you). Though we estimate each GP could be allocated between 533 and 1066 queries in Stage 1. That translates to a minimum guaranteed payment of £533-£1066 (if none of your labels agree with the reference standard) and a maximum potential payment of £2132-£4264 (if all your labels are eventually 'correct'). Both these scenarios are very unlikely, so the amount of money you will likely be paid will be somewhere between those figures - though the more 'accurate' your labels are the more you will be paid. Translating this into rates of pay, if you triage 1 request per minute (which is likely a conservative estimate) and you get all your triages ‘correct’ in stage 1 you will be paid £240 per hour. The likelihood is you won’t get all your triages correct though - so it will probably closer to £100-£120 per hour.

 

Payments are administered by The University of Manchester. If you pay tax through self-assessment and have a Unique Tax Reference number (e.g. if you're a locum GP or partner) then the university will pay you the full amount and you will be responsible for paying the tax yourself. If you don't have a Unique Tax Reference number (e.g. you only do salaried work) then the university will deduct tax on your behalf via PAYE when you get paid.

 

When undertaking the labelling, please imagine you work in a fictional 'typical' GP practice. Please do not label based on what services and staff you have at your own practice. This is because we need to make sure PATCHS AI works for as many GP practices as possible - and we've based this 'typical' GP practice on services that most GP practices have. In this fictional GP practice...

The purpose of this question is to test Urgency AI, Signpost AI, Assign AI. 

To answer this question there are four possible high-level triage labels you can choose, each with their own low-level triage labels. We will measure agreement on the high-level triage labels - so although choosing the most appropriate low-level triage label is still important, we are most interested in what you put as the high-level triage label - so spend most of your time and effort on that.

Please note that you should put your initial triage decision - which could be changed later if appropriate - for example, if you choose to send the patient to the pharmacy or direct a medication request to the script clerk, the patient can always be directed to a clinician later if they could not deal with it. 

 

High-level triage labels	Low-level triage labels
Non-clinical	Admin - non-clinical
	Medication
Routine	Admin - clinical
	Pharmacy
	Alternative provider
	Routine GP
	Routine nurse or HCA
Priority	Urgent GP
	Emergency GP
	Emergency hospital

 

Admin - non-clinical

These are administrative requests that should be dealt with by non-clinicians such as receptionists, secretaries or practice managers. Examples may include:

• Incorrect phone numbers
• Confirming or re-arranging appointments
• Booking pre-arranged appointments for things that need an in-person visit (e.g. blood test, smear test, coil fitting, implant fitting, suture removal, flu jabs etc)
• Chasing secondary care or other appointments e.g. hospital or scan appointments - unless mentions new or worsening symptoms that need clinician review
• Vaccination appointment booking and chasing
• Questions about opening hours
• Inappropriate requests for help (e.g. facemask exemption letters, shielding letters)

What happens to 'Admin clinical' requests in your fictional 'typical' GP practice? They are sent to a receptionist who responds on the same day.

Medication

These are requests solely asking for medication - either acute or repeat. Tips:

• Only include requests for medication that you'd be happy to issue without a review from a clinician.
• For repeat medication, assume that patients do not need a medication review by a clinician unless explicitly stated.
• Requests asking for medication but also mentioning health problems should be triaged as something else e.g. new or worsening symptoms, antibiotics for UTI, or steroids for asthma exacerbations etc.

What happens to 'Medication' requests in your fictional 'typical' GP practice? They are initially sent to a receptionist or script clerk first to issue the medication if they can (e.g. it's on repeat). This happens within 48 hours. If they can't issue it (e.g. the patient requires a medication review or is acute medication not had for a while), it will be passed to a clinician - timeframes are still adhered to.

Contents

Routine

Routine requests are requests that should be non-urgently directed to a clinician initially (e.g. GP, nurse, or HCA).

What happens to 'Routine clinical' requests in PATCHS?

Signpost AI shows patients information links from NHS.uk relevant to their request and offers them the opportunity to cancel their request and not submit it to your GP practice if they have addressed their query. If the patient chooses to submit their request to their GP practice anyway, Assign AI sends the request to a 'Clinical' inbox to help it get reviewed by a clinician sooner. These requests do not receive a priority order or flag - so remain in the middle of the PATCHS inbox.

Admin - clinical

These are administrative requests that require input from a clinician - typically a GP. Examples may include:

• Fit / sick notes - unless the request mentions new or worsening symptoms that need a clinician review
• Reviewing letters from consultants
• Adding information to medical records
• TWIMC letters
• Insurance forms

What happens to 'Admin - clinical' requests in your fictional 'typical' GP practice? They are sent to a clinician who completes the request within 1 week.

Pharmacy

These are minor illnesses that can initially either be dealt with by the patients themselves with self-help guidance (e.g. https://www.nhs.uk/conditions/sore-throat/) or by going to a pharmacy for advice and/or over-the-counter medication. See the pharmacy described in the typical GP practice above for more detail. As with traditional ways of requesting help - patients have the option of declining this advice or coming back to the GP if this initial treatment doesn't work. The pharmacy may also direct the patient to you after assessing the patient. Examples may include:

• Viral sore throat
• Gastroenteritis
• Sunburn
• Mouth ulcers
• Conjunctivitis
• Head lice

What happens to 'Pharmacy' requests in your fictional 'typical' GP practice? Patients are directed to self-care or to contact your local pharmacy for help first by your receptionists, and to contact you again if this doesn't work.

Alternative provider

These are non-emergency clinical issues that are more appropriate to be addressed by an alternative care provider rather than a GP practice. Examples may include:

• Midwife appointments / antenatal clinics e.g. someone newly pregnant that requires booking in
• Dental problems (dentist)
• Eye tests (opticians)
• STI tests (GUM clinic)
• Podiatrist
• Physiotherapist

What happens to 'Alternative provider' requests in your fictional 'typical' GP practice? Patients are directed to contact the appropriate local provider for help first by your receptionists.

Routine GP

These are health problems that would traditionally require input from a GP i.e. requires a diagnosis, review, and/or new prescription. The patient is unlikely to be harmed if the problem is not addressed within the next 48 hours to 1 week. Examples may include:

• Infertility
• Hair loss
• Longstanding non-serious conditions
• Medication reviews
• Follow-up from a previous consultation
• Chronic condition management

What happens to 'Routine GP' requests in your fictional 'typical' GP practice? They are sent to a GP who contacts the patient after 48 hours but no longer than 1 week. This includes a named GP for continuity of care.

Routine nurse or HCA

These are health problems that could be dealt with by a clinician who is not a GP (e.g. a nurse or HCA) i.e. that doesn't require a diagnosis or new prescription. The patient is unlikely to be harmed if the problem is not addressed in the next 48 hours to 1 week. Examples may include:

• Contraception
• Lifestyle counselling
• Travel advice
• Smoking cessation
• Chronic condition monitoring

What happens to 'Routine nurse or HCA' requests in your fictional 'typical' GP practice? They are sent to a relevant clinician (not a GP) who contacts the patient after 48 hours but no longer than 1 week.

Contents

Priority

Priority requests are requests that should be urgently directed to a GP (including out of hours [OOH]) or to emergency services (999 or A+E).

What happens to 'Priority' requests in PATCHS?

Signpost AI signposts patients to contact 111 if they submit an urgent request OOH, and signposts them to call 999 or go to A+E if they submit an emergency request at any time. Patients can override this advice and submit their request to their GP practice at their own risk. If they do this, Assign AI sends the request to a 'Clinical' inbox to help it get reviewed by a clinician sooner. Urgency AI also highlights emergency requests with a red flag and orders them to the top of the PATCHS inbox. Urgency AI highlights urgent requests with an orange flag and orders them to the top of the PATCHS inbox below emergencies. 

 

Urgent GP

These are health problems that require input from GPs where the patient could be harmed if the problem is not addressed in the next 48 hours. Examples may include:

• Suspected cancer
• Acute pain
• Infections without the potential for sepsis e.g. ear piercing infections

What happens to 'Urgent GP' requests in your fictional 'typical' GP practice? They are sent to a GP who contacts the patient within 48 hours.

Emergency GP

These are health problems that require input from GPs where the patient could be harmed if the problem is not addressed on the same day. Examples may include:

• Suicidal thoughts 
• Asthma exacerbations
• Infections with the potential for sepsis e.g. cellulitis

What happens to 'Emergency GP' requests in your fictional 'typical' GP practice? They are sent to a GP who contacts the patient on the same day.

Emergency hospital

These are health problems that require same-day hospital treatment because the patient could be harmed if the problem is not addressed immediately. This includes calling 999 or attending A+E, and 'direct to specialty' hospital admissions if you referred the patient. 

999 or A+E examples may include:

• Heart attacks
• Strokes
• Pulmonary embolism
• Suspected fractures

'Direct to specialty' hospital admissions may include:

• Visual loss (ophthalmology)
• Appendicitis (surgeons)
• Cauda equina (spinal surgeon)
• Paraphimosis (urology)

What happens to '999 / A+E' requests in your fictional 'typical' GP practice? Patients are directed to call 999 or go to A+E immediately - either by your receptionist or a clinician - or are referred 'direct to speciality' by a GP.

Contents

Special cases

• Photos. Always assume the patient has submitted a photo (although you won't be able to see it) e.g. if complaining of a rash.
• Multiple requests. If the message contains multiple requests from the patient - use the one with the highest clinical priority to make your triage decision. For example, requests for fit / sick notes that mention new or worsening symptoms should be triaged for clinician review not admin.
• A patient mentions a health problem of someone else. Then that should be taken into account and treated in the same way as a request with multiple problems in above (i.e. the highest clinical priority problem should determine your decision). E.g. if the patient has largely written a request about their depression, but mentions they think their neighbour is having a heart attack, we should triage it as a heart attack because the AI shouldn’t be relied upon to distinguish between who has written the request
• Medication requests. Requests solely asking for medication are most likely the patient requesting a repeat prescription e.g. contraceptive pill, antidepressants. They can therefore be issued without clinical review and triaged as 'Medication'. Only assume the patient needs a medication review if explicitly stated. Medication requests should only be triaged for review by a clinician if they explicitly state they need a medication review or mention health problems e.g. new or worsening symptoms, antibiotics for UTI, or steroids for asthma exacerbations.
• Fit / sick notes. should be triaged as 'Admin - clinical' unless the request mentions new or worsening symptoms that you think requires review by a clinician.
• Alternative provider. Should only be used for non-urgent, non-emergency conditions only. They should not be used for same-day assessments e.g. A&E for an X-ray for suspected fracture, or optician for sudden onset eye symptoms.
• Triage anything that needs a pre-arranged in-person visit as 'Admin non-clinical'. For example, blood test, smear test, coil fitting, implant fitting, suture removal, flu jabs etc. This is because in practice these would not be dealt with via PATCHS, and would typically just be booked in by a receptionist.
• The patient hasn't put enough detail or a 'full history'. There are requests where it's unclear what the patient wants or hasn't given a full description of their problem. We  recommend that you base your decision on what the patient has written - not on what they haven't written or may have left out. That may sound obvious, but what it means is if the patient hasn't written anything that sounds urgent then don't triage it as urgent because they may have missed information out. For example, if a patient says they have back pain and they haven't put a lot of information, but they haven't mentioned any red flag symptoms or anything else urgent, then don't triage it as 'Priority' just because you don't have a 'full history'.
• Urgent nurse. In your own practice you may send patients to see a nurse urgently, for example to dress a wound. We don’t have an ‘urgent nurse’ category because not all nurses do urgent things like dress wounds, and arguably anything urgent probably needs a GP review first before. However, either way the patient needs urgent help, so please label them as ‘Urgent GP’.

Contents

Clinical topics: What are the main topics of this request?

The purpose of this question is to test Topic AI, which asks patients specific questions (e.g. red flags in back pain) or questionnaires (e.g. the PHQ-9 in patients with depression) based on the clinical topics of their query. Therefore, it's likely most helpful for health problems, rather than medication or admin requests - though there are exceptions of course!

 

Depression - only label if specifically mentions 'depression' or low mood or symptoms that could only be related to someone with depression e.g. suicidal thoughts. Don't use this label if the patient only mentions anxiety or 'mental health'. DSH should be labeled as all patients with DSH should be screened for depression.

 

Back pain - only label if back pain is explicitly mentioned. This can include phrases like 'sciatica' or 'slipped disc'.

 

Tiredness - only label if the patient explicitly mentions being tired. This can include related phrases like 'exhausted' or 'no energy'.

 

Dermatology - includes anything to do with the skin e.g. bruises, wounds, skin infections, abscesses, nails, hair, lumps and bumps. Not just specific named dermatological diagnoses like psoriasis or eczema.

 

Cough - only label if cough is explicitly mentioned or inferred e.g. 'bringing up sputum'.

 

Sick note - only label if a sick note is explicitly mentioned (not necessarily asked for). Other phrases may include 'fit note', 'letter for work', 'doctor's note' etc.

 

Other - this is a helpful answer! Knowing a request mentions other topics helps us identify negative examples. You can select 'Other' on its own where no 'named' topics are mentioned at all: for example, a request about hair loss. You can also select 'Other' if another topic is mentioned in addition to other 'named' clinical topics: for example, if depression, back pain, and eczema are all mentioned in one request you should select 'Depression', 'Back pain', and 'Other' (for eczema).