PATCHS AI Testing

This article is intended for GPs in the PATCHS AI Testing Team. If you would like to join the team, please contact us to assess your suitability by using the 'Give Feedback' button when logged into PATCHS.
 
If you're a GP on the PATCHS AI Testing Team who doesn't use PATCHS in your own practice, you can read more about PATCHS in these articles: What is PATCHS? and Key principles of using PATCHS. For more background reading, you can access all help articles about PATCHS (for both patients and GP practices) here: help.patchs.ai 
 

What are we trying to do?

We have built world-leading Artificial Intelligence (AI) into PATCHS to decrease GP practice workload and improve patient safety by triaging patient requests automatically. As part of our ongoing clinical evaluations for our MHRA registration and UKCA marking, we want to directly compare the performance of PATCHS AI against experienced GPs when reviewing the same patient queries. We are trying to answer the question: is PATCHS AI as good as an experienced GP at triaging patient requests? This is a research project that we're conducting in collaboration with The University of Manchester.

There are three types of triage decision we want to test:

  1. Urgency: How urgent a request is, and whether it needs input from a clinician. To test Urgency AISignpost AIAssign AI.
  2. Clinical topics: What the clinical topics are of a request. To test Topic AI.
  3. Face-to-face: Whether a request needs a face-to-face appointment. To test F2F AI.

Contents

How are we doing this?

To answer the question 'is PATCHS AI as good as an experienced GP?' we need a sample of patient queries where we know: 1) The 'correct' triage decision, and 2) How well an 'experienced' GP performs when making triage decisions on them. We can then compare PATCHS AI's performance against the experienced GP. However, there are two challenges:
  1. There is no 'gold standard' test to determine the 'correct' decision for patient queries in primary care. We will therefore consider a triage decision as 'correct' when 3 different GPs all agree on the same triage decision. This is called a 'reference standard'.
  2. We need more than one experienced GP to compare against PATCHS AI for the test to be convincing. We will therefore combine the decisions of lots of different experienced GPs at random to simulate the performance of one single GP.

For each of the three triage decision tests, our approach can be summarised in this diagram:

Figure.png

 

Contents

Stage 1: Initial labelling

We have recruited a team of ~30 experienced GPs from around the UK (you!) to 'label' a sample of anonymised patient queries collected from PATCHS - by labelling we mean making a clinical decision about how that query should be triaged. The section below provides guidance on how to label the patient requests. For each query there is also a section for you to write notes on why you made a particular decision (for example, if the decision wasn't a straightforward). These notes are important for Stage 2 because they will remind you and inform the other GPs what your thought processes were when you made a labeling decision. Each GP will be allocated the same number of queries to label at random within PATCHS - if you are already a PATCHS user you can access your queries in the 'PATCHS AI' practice from in the top menu dropdown. Each query will be labeled by four GPs independently (i.e. each GP will not know what the other GPs put). One GP (GP4) will be chosen at random as the GP we compare against PATCHS AI. Because requests are allocated at random and GP4 is chosen at random, all GPs will equally contribute to GP4's performance. The other three GPs (GP1-3) will determine the 'correct' label for the query (the 'reference standard'). If the labels put by GPs 1-3 all agree we accept this as the reference standard (for urgency triage decisions, we measure agreement on the 'high-level' categories only - see below for further information). If there is any disagreement, the query proceeds to Stage 2 for 'reconsideration'. 

Stage 2: Reconsideration

Queries where there is any disagreement between GPs1-3 are allocated back to those GPs for reconsideration. The purpose is for you to review your initial labels, and change them if you want in order to achieve agreement with the other GPs. You will be able to see what labels you originally put along with any notes you made in Stage 1. You will also be able to see what the other GPs put who looked at the same request as you. You can add further notes in the notes section to explain your decision - to add new notes, please entirely delete your old comment and type a completely new one. If, after reconsideration, GPs 1-3 now all agree, we accept this as the reference standard. If there is still any disagreement, the request proceeds to Stage 3 for 'resolution'.

Stage 3: Resolution

To get the reference standard for queries where there is still disagreement after Stage 2 are resolved in different ways depending on the triage decision:
  1. Urgency. If two of three GPs agree, we will accept that label as the reference standard. If there aren’t 2 out of 3 GPs that agree, we will take the highest (most urgent) triage label in that has been applied by any of the GPs in Stage 2 as the reference standard. The rationale for this approach is to obtain the safest triage label for the request, in order to align with the priorities of GP staff users from our empirical research and workshops.
  2. Clinical topics. If two of three GPs agree, we will accept that label as the reference standard. If there aren’t 2 out of 3 GPs that agree, we will remove those queries from the study as they won't be clear cut and may refer to lots of different clinical topics.
  3. Face-to-face. Because there are just two possible labels (face-to-face appointment needed or not), there will always be 2 out of 3 GPs that agree, so we will accept that label as the reference standard.

Timetable

To make the process as efficient as possible, we will use the same group of queries to test all three types of triage decision (urgency, clinical topics, face-to-face). So in Stage 1 each GP will make three triage decisions about each query. We'll then do three different rounds of Stage 2, where each GP will only reconsider queries on one of the triage decisions where there's disagreement. So in total there will be four 'rounds' of labelling:
 
Round Description Dates
1 Initial labelling for all triage decisions (Stage 1) 14th Nov 2022 - 1st Dec 2022
2 Urgency reconsideration (Stage 2) Mid-Dec 2022 - Late Dec 2022
3 Clinical topics reconsideration (Stage 2) Early Jan 2023 - Mid Jan 2023
4 Face-to-face reconsideration (Stage 2) Mid Jan 2023 - Late Jan 2023
 

Payment

Payments are structured to reflect the work undertaken and to incentivise high quality labelling:
  • You will be paid £1 for each query you label in Stage 1.
  • You will receive a bonus payment of £1 if any of your labels for any of the three triage decisions (urgency, clinical topics, face-to-face) applied either in Stages 1 or 2 match the final 'correct' label (reference standard). So if your 3 triage labels all match the reference standard in Stage 1 you will be paid £4 for labelling that query just once.
  • No bonus payments are paid for requests that go to Stage 3.
  • Payments will only be made once you have completed all your allocated labels for all four labelling rounds within the timeframes set.
The exact number of queries you will be allocated depends on how many GPs participate in total (because they will be divided up equally between you). Though we estimate each GP could be allocated between 533 and 1066 queries in Stage 1. That translates to a minimum guaranteed payment of £533-£1066 (if none of your labels agree with the reference standard) and a maximum potential payment of £2132-£4264 (if all your labels are eventually 'correct'). Both these scenarios are very unlikely, so the amount of money you will likely be paid will be somewhere between those figures - though the more 'accurate' your labels are the more you will be paid. Translating this into rates of pay, if you triage 1 request per minute (which is likely a conservative estimate) and you get all your triages ‘correct’ in stage 1 you will be paid £240 per hour. The likelihood is you won’t get all your triages correct though - so it will probably closer to £100-£120 per hour.
 
Payments are administered by The University of Manchester. If you pay tax through self-assessment and have a Unique Tax Reference number (e.g. if you're a locum GP or partner) then the university will pay you the full amount and you will be responsible for paying the tax yourself. If you don't have a Unique Tax Reference number (e.g. you only do salaried work) then the university will deduct tax on your behalf via PAYE when you get paid.
 

Guidance

To access the labelling page go to www.patchs.ai and log in. If you use PATCHS in your own practice, you can access the labeling page by choosing 'PATCHS AI' in the practice dropdown menu. Below is a screenshot of the labelling screen, and a brief instructional video (it doesn't include any real patient information).

The guidance below isn't supposed to be prescriptive and is intended to draw on your implicit knowledge as GPs rather than tell you exactly what to do!

 

Screenshot_2022-11-11_at_15.50.39.png

 

 

Contents

Imagine you work in a typical GP practice

When undertaking the labelling, please imagine you work in a fictional 'typical' GP practice. Please do not label based on what services and staff you have at your own practice. This is because we need to make sure PATCHS AI works for as many GP practices as possible - and we've based this 'typical' GP practice on services that most GP practices have. In this fictional GP practice...
  • You do not have...
    • (Advanced) Nurse Practitioners that can diagnose and treat minor illnesses
    • Open access specialist services e.g. Open Access Physiotherapists
    • Special arrangements with outside services e.g. opticians that deal with acute eye issues and refer to ophthalmology if necessary
    • An in-house pharmacist that does things like reviews or changes medication
  • You do have...
    • GPs
    • Nurses who are skilled at LTC management like asthma reviews and pill checks etc
    • Health Care Assistants (HCAs) that can do things like measure blood pressure and take blood
    • Receptionists
    • Secretaries
    • Typical services nearby e.g. GUM clinic, opticians, dentist
    • A local pharmacy who is equipped to deal with minor illnesses like: aches and pains, sore throat, coughs, colds, flu, earache, cystitis, skin rashes, teething, red eye. See this link for more informationAfter assessing the patient, if the pharmacist thinks the patient needs to see a GP, they will direct the patient to you.
  • Services you do provide...
    • Smears
    • Blood tests
  • Services you don't provide...
    • Ear syringing
  • You are a very efficient practice and always adhere to the following timescales when dealing with patient queries...
    • Emergencies - same day
    • Urgent - within 48 hours (i.e. either today or tomorrow)
    • Routine - after 48 hours but no longer than 1 week; you can usually accommodate continuity of care and requests for specific clinicians within this timeframe too
    • Medication requests - within 48 hours (i.e. either today or tomorrow)
    • Clinical admin requests (e.g. fit / sick notes, TWMIC letters) - within 1 week
    • Non-clinical admin requests (e.g. receptionist needs to book a blood test or smear) - response on same day from receptionists

Contents

Urgency: What is the most appropriate initial triage decision for this request?

The purpose of this question is to test Urgency AISignpost AIAssign AI
To answer this question there are four possible high-level triage labels you can choose, each with their own low-level triage labels. We will measure agreement on the high-level triage labels - so although choosing the most appropriate low-level triage label is still important, we are most interested in what you put as the high-level triage label - so spend most of your time and effort on that.
Please note that you should put your initial triage decision - which could be changed later if appropriate - for example, if you choose to send the patient to the pharmacy or direct a medication request to the script clerk, the patient can always be directed to a clinician later if they could not deal with it. 
 
 

Contents

Non-clinical

Non-clinical requests are requests that should be directed to a non-clinician initially (e.g. receptionist, secretary).

What happens to 'Non-clinical' requests in PATCHS?

Signpost AI shows patients information links from NHS.uk relevant to their request and offers them the opportunity to cancel their request and not submit it to your GP practice if they have addressed their query. If the patient chooses to submit their request to their GP practice anyway, they are highlighted by a purple flag and ordered to the bottom of the PATCHS inbox.

Screenshot_2021-03-19_at_09.29.22_2.png

Admin - non-clinical

These are administrative requests that should be dealt with by non-clinicians such as receptionists, secretaries or practice managers. Examples may include:

  • Incorrect phone numbers
  • Confirming or re-arranging appointments
  • Booking pre-arranged appointments for things that need an in-person visit (e.g. blood test, smear test, coil fitting, implant fitting, suture removal, flu jabs etc)
  • Chasing secondary care or other appointments e.g. hospital or scan appointments - unless mentions new or worsening symptoms that need clinician review
  • Vaccination appointment booking and chasing
  • Questions about opening hours
  • Inappropriate requests for help (e.g. facemask exemption letters, shielding letters)

What happens to 'Admin clinical' requests in your fictional 'typical' GP practiceThey are sent to a receptionist who responds on the same day.

Medication

These are requests solely asking for medication - either acute or repeat. Tips:

  • Only include requests for medication that you'd be happy to issue without a review from a clinician.
  • For repeat medication, assume that patients do not need a medication review by a clinician unless explicitly stated.
  • Requests asking for medication but also mentioning health problems should be triaged as something else e.g. new or worsening symptoms, antibiotics for UTI, or steroids for asthma exacerbations etc.

What happens to 'Medication' requests in your fictional 'typical' GP practice? They are initially sent to a receptionist or script clerk first to issue the medication if they can (e.g. it's on repeat). This happens within 48 hours. If they can't issue it (e.g. the patient requires a medication review or is acute medication not had for a while), it will be passed to a clinician - timeframes are still adhered to.

Contents

Routine

Routine requests are requests that should be non-urgently directed to a clinician initially (e.g. GP, nurse, or HCA).

What happens to 'Routine clinical' requests in PATCHS?

Signpost AI shows patients information links from NHS.uk relevant to their request and offers them the opportunity to cancel their request and not submit it to your GP practice if they have addressed their query. If the patient chooses to submit their request to their GP practice anyway, Assign AI sends the request to a 'Clinical' inbox to help it get reviewed by a clinician sooner. These requests do not receive a priority order or flag - so remain in the middle of the PATCHS inbox.

Screenshot_2021-03-19_at_09.29.22_3.png

Admin - clinical

These are administrative requests that require input from a clinician - typically a GP. Examples may include:

  • Fit / sick notes - unless the request mentions new or worsening symptoms that need a clinician review
  • Reviewing letters from consultants
  • Adding information to medical records
  • TWIMC letters
  • Insurance forms

What happens to 'Admin - clinical' requests in your fictional 'typical' GP practiceThey are sent to a clinician who completes the request within 1 week.

Pharmacy

These are minor illnesses that can initially either be dealt with by the patients themselves with self-help guidance (e.g. https://www.nhs.uk/conditions/sore-throat/) or by going to a pharmacy for advice and/or over-the-counter medication. See the pharmacy described in the typical GP practice above for more detail. As with traditional ways of requesting help - patients have the option of declining this advice or coming back to the GP if this initial treatment doesn't work. The pharmacy may also direct the patient to you after assessing the patient. Examples may include:

  • Viral sore throat
  • Gastroenteritis
  • Sunburn
  • Mouth ulcers
  • Conjunctivitis
  • Head lice

What happens to 'Pharmacy' requests in your fictional 'typical' GP practicePatients are directed to self-care or to contact your local pharmacy for help first by your receptionists, and to contact you again if this doesn't work.

Alternative provider

These are non-emergency clinical issues that are more appropriate to be addressed by an alternative care provider rather than a GP practice. Examples may include:

  • Midwife appointments / antenatal clinics e.g. someone newly pregnant that requires booking in
  • Dental problems (dentist)
  • Eye tests (opticians)
  • STI tests (GUM clinic)

What happens to 'Alternative provider' requests in your fictional 'typical' GP practicePatients are directed to contact the appropriate local provider for help first by your receptionists.

Routine GP

These are health problems that would traditionally require input from a GP i.e. requires a diagnosis, review, and/or new prescription. The patient is unlikely to be harmed if the problem is not addressed within the next 48 hours to 1 week. Examples may include:

  • Infertility
  • Hair loss
  • Longstanding non-serious conditions
  • Medication reviews
  • Follow-up from a previous consultation
  • Chronic condition management

What happens to 'Routine GP' requests in your fictional 'typical' GP practiceThey are sent to a GP who contacts the patient after 48 hours but no longer than 1 week. This includes a named GP for continuity of care.

Routine nurse or HCA

These are health problems that could be dealt with by a clinician who is not a GP (e.g. a nurse or HCA) i.e. that doesn't require a diagnosis or new prescription. The patient is unlikely to be harmed if the problem is not addressed in the next 48 hours to 1 week. Examples may include:

  • Contraception
  • Lifestyle counselling
  • Travel advice
  • Smoking cessation
  • Chronic condition monitoring

What happens to 'Routine nurse or HCA' requests in your fictional 'typical' GP practiceThey are sent to a relevant clinician (not a GP) who contacts the patient after 48 hours but no longer than 1 week.

Contents

Priority

Priority requests are requests that should be urgently directed to a GP (including out of hours [OOH]) or to emergency services (999 or A+E).

What happens to 'Priority' requests in PATCHS?

Signpost AI signposts patients to contact 111 if they submit an urgent request OOH, and signposts them to call 999 or go to A+E if they submit an emergency request at any time. Patients can override this advice and submit their request to their GP practice at their own risk. If they do this, Assign AI sends the request to a 'Clinical' inbox to help it get reviewed by a clinician sooner. Urgency AI also highlights emergency requests with a red flag and orders them to the top of the PATCHS inbox. Urgency AI highlights urgent requests with an orange flag and orders them to the top of the PATCHS inbox below emergencies. 

 

Screenshot_2021-03-19_at_09.29.22.png

Urgent GP

These are health problems that require input from GPs where the patient could be harmed if the problem is not addressed in the next 48 hours. Examples may include:

  • Suspected cancer
  • Acute pain
  • Infections without the potential for sepsis e.g. ear piercing infections

What happens to 'Urgent GP' requests in your fictional 'typical' GP practiceThey are sent to a GP who contacts the patient within 48 hours.

Emergency GP

These are health problems that require input from GPs where the patient could be harmed if the problem is not addressed on the same day. Examples may include:

  • Suicidal thoughts 
  • Asthma exacerbations
  • Infections with the potential for sepsis e.g. cellulitis

What happens to 'Emergency GP' requests in your fictional 'typical' GP practiceThey are sent to a GP who contacts the patient on the same day.

Emergency hospital

These are health problems that require same-day hospital treatment because the patient could be harmed if the problem is not addressed immediately. This includes calling 999 or attending A+E, and 'direct to specialty' hospital admissions if you referred the patient. 

999 or A+E examples may include:

  • Heart attacks
  • Strokes
  • Pulmonary embolism
  • Suspected fractures

'Direct to specialty' hospital admissions may include:

  • Visual loss (ophthalmology)
  • Appendicitis (surgeons)
  • Cauda equina (spinal surgeon)
  • Paraphimosis (urology)

What happens to '999 / A+E' requests in your fictional 'typical' GP practicePatients are directed to call 999 or go to A+E immediately - either by your receptionist or a clinician - or are referred 'direct to speciality' by a GP.

Contents

Special cases

  • Photos. Always assume the patient has submitted a photo (although you won't be able to see it) e.g. if complaining of a rash.
  • Multiple requests. If the message contains multiple requests from the patient - use the one with the highest clinical priority to make your triage decision. For example, requests for fit / sick notes that mention new or worsening symptoms should be triaged for clinician review not admin.
  • A patient mentions a health problem of someone else. Then that should be taken into account and treated in the same way as a request with multiple problems in above (i.e. the highest clinical priority problem should determine your decision). E.g. if the patient has largely written a request about their depression, but mentions they think their neighbour is having a heart attack, we should triage it as a heart attack because the AI shouldn’t be relied upon to distinguish between who has written the request
  • Medication requests. Requests solely asking for medication are most likely the patient requesting a repeat prescription e.g. contraceptive pill, antidepressants. They can therefore be issued without clinical review and triaged as 'Medication'. Only assume the patient needs a medication review if explicitly stated. Medication requests should only be triaged for review by a clinician if they explicitly state they need a medication review or mention health problems e.g. new or worsening symptoms, antibiotics for UTI, or steroids for asthma exacerbations.
  • Fit / sick notes. should be triaged as 'Admin - clinical' unless the request mentions new or worsening symptoms that you think requires review by a clinician.
  • Alternative provider. Should only be used for non-urgent, non-emergency conditions only. They should not be used for same-day assessments e.g. A&E for an X-ray for suspected fracture, or optician for sudden onset eye symptoms.
  • Triage anything that needs a pre-arranged in-person visit as 'Admin non-clinical'. For example, blood test, smear test, coil fitting, implant fitting, suture removal, flu jabs etc. This is because in practice these would not be dealt with via PATCHS, and would typically just be booked in by a receptionist.
  • The patient hasn't put enough detail. There are requests where it's unclear what the patient wants or what their problem is. These are really difficult to deal with and difficult to give advice on. What we suggest in this situation is to use your best judgment as to what you would do in your own practice. Sorry if this sounds like a cop out!
  • Urgent nurse. In your own practice you may send patients to see a nurse urgently, for example to dress a wound. We don’t have an ‘urgent nurse’ category because not all nurses do urgent things like dress wounds, and arguably anything urgent probably needs a GP review first before. However, either way the patient needs urgent help, so please label them as ‘Urgent GP’.

Contents

Clinical topics: What are the main topics of this request?

The purpose of this question is to test Topic AI, which asks patients specific questions (e.g. red flags in back pain) or questionnaires (e.g. the PHQ-9 in patients with depression) based on the clinical topics of their query. Therefore, it's likely most helpful for health problems, rather than medication or admin requests - though there are exceptions of course!
 
Dermatology - includes anything to do with the skin e.g. wounds, infections, abscesses, nails, hair. Not just specific named dermatological diagnoses like psoriasis or eczema.
 
Tips
  • Only label topics that are substantially mentioned. For example, if mentioned in a way that would make you want to discuss it further with the patient. Don't label topics only mentioned in passing.
  • Select more than one topic if more than one is mentioned. If multiple clinical topics are mentioned, please select all relevant labels.
  • 'Other' is a helpful answer! Knowing a request mentions other topics helps us identify negative examples. You can select 'Other' on its own where no 'named' topics are mentioned at all: for example, a request about hair loss. You can also select 'Other' if another topic is mentioned in addition to other 'named' clinical topics: for example, if depression, back pain, and eczema are all mentioned in one request you should select 'Depression', 'Back pain', and 'Other' (for eczema).
  • Do not label requests solely asking for medication as the condition they treat. Only label them with a clinical topic if they also mention health problems that require review by a clinician. E.g. Do not label requests solely asking for antidepressants as 'depression' unless they mention symptoms like worsening low mood or suicidal thoughts. Otherwise, please label medication requests as 'Other'.
 

Face-to-Face: Does this patient need a face-to-face appointment?

The purpose of this question is to test F2F AI. It's trying to find out whether you think the patient needs an in-person appointment to resolve their request.

The question should only apply to requests that would be passed onto a clinician to resolve i.e. requests you have triaged as Routine or Priority.

Always answer 'No' if you have triaged the request as a Non-clinical triage decision - even if the request does technically end up in as an in-person appointment (e.g. receptionist books the patient in for a smear test, flu jab etc). This is because only clinicians who use PATCHS can say whether the patient needs an in-person appointment - not receptionists or other non-clinical staff.

Special cases:

  • Requests that you may be able to deal with remotely (e.g. via telephone or using photos). Even though many requests can now be dealt with remotely that would have previously been automatically dealt with face-to-face (e.g. rashes, tonsillitis), it's probably still best to mark these as needing a face-to-face appointment, as sometimes a telephone consultation +/- photos aren't sufficient and often you have to bring these patients in anyway. This can duplicate patient contacts / workload for the same request, which is something we want to avoid.
  • Patients requesting face-to-face appointments. This should be a judgment call as to what you would do in your normal clinical practice. If the patient wants a face-to-face appointment but you don’t think it’s necessary then don't label as needing one. But if you think it would be reasonable - go ahead.
  • Requests where you think the patient should go elsewhere e.g. pharmacy, 999, A&E. They should still be marked as needing a face-to-face appointment if you think they need one. It's just that it ideally it wouldn't be the GP practice doing giving them one.

 

Contents

This request was not written in English

Tick this box if the request is written in a non-English language. You will still be asked for a triage decision. Don’t worry about what triage decision you put - just put anything down as we’ll remove that request from the database.

Contents

Notes

This section is for you to write notes on your rationale for choosing a particular label, and is optional. We would recommend writing notes for any labels you have applied where the decision hasn't been straightforward. You will be able to review your notes during the Stage 2 labelling rounds to remind yourself of why you made a decision, and add further notes if you wish. In Stage 2 the other GPs that labeled the same requests as you will also be able see the notes you've written (and you'll be able to see theirs too). Your notes will also be visible to the research team. So please don't write anything you wouldn't be happy being shared with others!

Contents