PATCHS AI Testing

This article is intended for GPs in the PATCHS AI Testing Team. If you would like to join the team, please contact us to assess your suitability by using the 'Give Feedback' button when logged into PATCHS.
 
If you're a GP on the PATCHS AI Testing Team who doesn't use PATCHS in their own practice, you can read more about PATCHS in general in these articles: What is PATCHS? and Key principles of using PATCHS. For more background reading, you can access all help articles about PATCHS (for both patients and GP practices) here: help.patchs.ai 
 

What are we trying to do?

We have built world-leading Artificial Intelligence (AI) into PATCHS to decrease GP practice workload and improve patient safety by triaging patient requests automatically. As part of our ongoing clinical evaluations for our MHRA registration and UKCA marking, we want to directly compare the performance of PATCHS AI and experienced GPs when reviewing the same patient queries. We are trying to answer the question: is PATCHS AI as good as an experienced GP?

This project is testing the following PATCHS AI modules:

Contents

How are we doing this?

To answer the question 'is PATCHS AI as good as an experienced GP?' we need a sample of patient queries where we know: 1) The 'correct' triage decision, and 2) How well an 'experienced' GP performs when making triage decisions on them. We can then compare PATCHS AI's performance against the experienced GP. However, there are two challenges:
  1. There is no 'gold standard' test to determine the 'correct' decision for patient queries in primary care. We will therefore consider a triage decision as 'correct' when 3 different GPs all agree on the same triage decision. This is called a 'reference standard'.
  2. We need more than one experienced GP to compare against PATCHS AI for the test to be convincing. We will therefore combine the decisions of lots of different experienced GPs at random to simulate the performance of one single GP.

Our approach is summarised in this diagram:

figures_v7.png

Contents

Stage 1: Initial labelling

We have recruited a team of ~30 experienced GPs from around the UK (you!) to 'label' a sample of anonymised patient queries collected from PATCHS - by labelling we mean making a clinical decision about how that query should be triaged. The section below provides guidance on how to label the patient requests. For each query there is also a section for you to write anonymous notes on why you made a particular decision (for example, if the decision wasn't a straightforward). These notes are important for stage 2 because it will remind you what your thought processes were when you made a labeling decision. Each GP is allocated the same number of queries to label at random within the PATCHS website - if you are a PATCHS user you will have a 'PATCHS AI' GP practice that you can access from a dropdown in the top menu. Each query is labeled by four GPs independently (i.e. each GP does not know what the other GPs put as their labels). If GPs 1-3 all agree we accept this as the 'correct' label for the query - the 'reference standard'.* If there is any disagreement, the query proceeds to stage 2 for 'reconsideration'. GP 4 is chosen at random and is the GP we compare against PATCHS AI. Because requests are allocated and GP 4 is chosen at random, all GPs will fairly contribute to GP 4's performance.
 
* Currently we only count agreement on high-level triage decisions described here - not the other labels.

Stage 2: Reconsideration

Queries where there is any disagreement between GPs 1-3 on high-level triage decisions are allocated back to those GPs for review. The purpose is for them to reconsider their initial labels, and give them the opportunity to change them if they want. The aim is to get as many queries as possible with complete agreement between GPs 1-3. To help, they will be able to see what labels they originally put along with any notes they made. Each GP can either keep their original labels, or change them. They can also add further notes in the notes section to explain their decision - to add new notes, please entirely delete your old comment and type a completely new one. If, after reconsideration, GPs 1-3 now all agree, we accept this as the 'correct' label. If there is still any disagreement, the request proceeds to stage 3 for 'resolution'.

Stage 3: Resolution

In stage 3, we take the highest triage label in terms of clinical acuity or importance that has been applied by at least one of the GPs in stage 2 as the reference standard. The rationale for this approach is to obtain the safest triage label for the request, in order to align with the priorities of GP staff users from our empirical research and workshops.

Payment

Payments are structured to reflect both the work undertaken by GPs and to incentivise high quality labeling. All GPs are paid one unit for each query labelled in Stage 1: Initial labelling and Stage 2: Reconsideration. So GPs 1-3 are paid the same amount for a query where they all initially agree, vs a query where there is disagreement. All GPs also receive a bonus unit of payment (so two units in total) if their final label - either applied in stage 1 or 2 - matches the final 'correct' label chosen. No bonus payments are paid for requests that go to stage 3. GPs will only be paid once they have completed labeling all their allocated queries for both stages 1 and 2 within the timeframes of the project.
 

Guidance

To access the labeling page go to www.patchs.ai and log in. If you use PATCHS in your own practice, you can access the labeling page by choosing 'PATCHS AI' in the practice dropdown menu. Below is a screenshot of the labelling screen, and  a brief instructional video (it does not include any real patient information).

Screenshot_2021-03-15_at_13.57.50.png

 

Contents

Imagine you work in a typical GP practice

When undertaking the labelling, please imagine you work in a fictional 'typical' GP practice. Please do not label based on what services and staff you have at your own practice. This is because we need to make sure PATCHS AI works for all GP practices. In this fictional GP practice...
  • You do not have...
    • (Advanced) Nurse Practitioners that can diagnose and treat minor illnesses
    • Open access specialist services e.g. Open Access Physiotherapists
    • Special arrangements with outside services e.g. opticians that deal with acute eye issues and refer to ophthalmology if necessary
    • An in-house pharmacist
  • You do have...
    • GPs
    • Nurses who are skilled at LTC management like asthma reviews and pill checks etc
    • Health Care Assistants (HCAs) that can do things like measure blood pressure and take blood
    • Receptionists
    • Secretaries
    • Typical services nearby e.g. GUM clinic, opticians, dentist
    • A local pharmacy who is equipped to deal with minor illnesses like: aches and pains, sore throat, coughs, colds, flu, earache, cystitis, skin rashes, teething, red eye. See this link for more informationAfter assessing the patient, if the pharmacist thinks the patient needs to see a GP, they will direct the patient to you.
  • Services you do provide...
    • Smears
    • Blood tests
  • Services you don't provide...
    • Ear syringing
  • You are a very efficient practice and always adhere to the following timescales when dealing with patient queries...
    • Emergencies - same day
    • Urgent - within 48 hours (i.e. either today or tomorrow)
    • Routine - after 48 hours but no longer than 1 week; you can usually accommodate continuity of care and requests for specific clinicians within this timeframe too
    • Medication requests - within 48 hours (i.e. either today or tomorrow)
    • Admin clinical requests (e.g. fit / sick notes, TWMIC letters) - within 1 week
    • Admin non-clinical requests (e.g. receptionist needs to book a blood test or smear) - response on same day from receptionists

Contents

What is the most appropriate initial triage decision for this request?

There are four high-level triage decisions, each with their own low-level triage decisions. Please note that you should put your initial triage decision - which could be changed later if appropriate - for example, if you choose to send the patient to the pharmacy or direct a medication request to the script clerk, the request could always be returned to you if they disagreed. We primarily measure agreement between GPs 1-3 on their high-level decisions, and analyse agreement on the low-level decisions as a secondary measure.
 
 

Contents

Non-clinical

  1. Medication: Requests solely asking for medication - either acute or repeat.
    • Assume that patients do not need a medication review by a clinician (including pill checks) unless explicitly stated. If they do - this will be determined by the receptionist or script clerk after receiving the request.
    • Requests asking for medication but also mentioning health problems should be triaged as health problems e.g. new or worsening symptoms, antibiotics for UTI, or steroids for asthma exacerbations etc.
    • What happens to 'Medication' requests in your fictional 'typical' GP practice? They are initially sent to a receptionist or script clerk first to issue the medication if they can (e.g. it's on repeat). This happens within 48 hours. If they can't issue it (e.g. needs medication review or is acute medication not had for a while), it will be passed to a clinician - timeframes are still adhered to.
  2. Admin clinical: Admin requests that require input from a GP. Examples:
    • Fit / sick note - unless mentions new or worsening symptoms that need clinician review
    • Reviewing letters from consultants
    • Adding information to medical records
    • TWIMC letter
    • Insurance forms
    • What happens to 'Admin clinical' requests in your fictional 'typical' GP practiceThey are sent to a clinician who completes the request within 1 week.
  3. Admin non-clinical: Requests that should be directed to a non-clinician initially (e.g. receptionist, secretary) i.e. anything a clinician shouldn't be dealing with first. Examples:
    • Requests where it isn't clear what the patient needs initially because there isn't enough information
    • Informing patients of normal test results
      • NB patients requesting test results can be presumed normal unless they state they have been asked to contact the practice to discuss the test results
    • Incorrect phone numbers
    • Confirming or re-arranging appointments
    • Booking pre-arranged appointments for things that need an in-person visit (e.g. blood test, smear test, coil fitting, implant fitting, suture removal, flu jabs etc)
    • Chasing secondary care or other appointments e.g. hospital or scan appointments - unless mentions new or worsening symptoms that need clinician review
    • Vaccination appointment booking and chasing
    • Questions about opening hours
    • Inappropriate requests for help (e.g. facemask exemption letters, shielding letters)
    • What happens to 'Admin clinical' requests in your fictional 'typical' GP practiceThey are sent to a receptionist who responds on the same day.

What happens to 'Non-clinical' requests in PATCHS?

They are ordered to the bottom of the PATCHS inbox with a purple flag.

Screenshot_2021-03-19_at_09.29.22_2.png

Contents

Signpost

  1. Self-care: A minor illness that does not initially need a health professionals' input to treat. The patient should initially care for themselves with self-help guidance first e.g.
    https://www.nhs.uk/conditions/sore-throat/. Patients always have the option of declining this advice or coming back to the GP if this initial treatment doesn't work. Suggested examples:
    • Viral sore throat
    • Gastroenteritis
    • Sunburn
    • What happens to 'Self-care' requests in your fictional 'typical' GP practicePatients are directed to self-care by your receptionists first, and to contact you again if this doesn't work.
  2. Pharmacy: A minor illness that does not necessarily need a clinician’s input to treat. See the pharmacy described in the typical GP practice above for more detail. Patients may initially consider going to a pharmacy for advice or over-the-counter medicines first. Patients always have the option of declining this advice or coming back to the GP if this initial treatment doesn't work. The pharmacy may also direct the patient to you after assessing the patient. Suggested examples:
    • Mouth ulcers
    • Conjunctivitis
    • Head lice
    • What happens to 'Pharmacy' requests in your fictional 'typical' GP practicePatients are directed to contact your local pharmacy for help first by your receptionists, and to contact you again if this doesn't work.
  3. Alternative provider: A non-emergency clinical request that is more appropriate to be addressed by an alternative care provider rather than a GP practice. Suggested examples:
    • Midwives / antenatal clinics e.g. someone newly pregnant that requires booking in
    • Dental problems (dentist)
    • Eye tests (opticians)
    • What happens to 'Alternative provider' requests in your fictional 'typical' GP practicePatients are directed to contact the appropriate local provider for help first by your receptionists.

What happens to 'Signpost' requests in PATCHS?

PATCHS shows patients information links from NHS.uk relevant to their request and offers them the opportunity to cancel their request and not submit it to your GP practice. You can read more about how this happens in this article.

Contents

Routine clinical

  1. Routine clinical doctor: A health problem that would traditionally require input from a GP i.e. requires a diagnosis, review, and/or new prescription. Patient is unlikely to be harmed if the problem is not addressed within the next 48 hours to 1 week. Suggested examples:
    • Infertility
    • Hair loss
    • Discussion of abnormal test results - unless results should be dealt with quicker
    • Medication reviews
    • Follow-up from a previous consultation
    • Chronic condition management
    • What happens to 'Routine clinical doctor' requests in your fictional 'typical' GP practiceThey are sent to a GP who contacts the patient after 48 hours but no longer than 1 week. This includes a named GP for continuity of care.
  2. Routine clinical non-doctor: A health problem that could be dealt with by a clinician who is not a GP (e.g. a nurse or HCA) i.e. that doesn't require a diagnosis or prescription. Patient is unlikely to be harmed if the problem is not addressed in the next 48 hours to 1 week. Suggested examples:
    • Contraception
    • Lifestyle counselling
    • Travel advice
    • Chronic condition monitoring
    • What happens to 'Routine clinical non-doctor' requests in your fictional 'typical' GP practiceThey are sent to a relevant clinician (not a GP) who contacts the patient after 48 hours but no longer than 1 week.

What happens to 'Routine clinical' requests in PATCHS?

They are sent to a 'Clinical' inbox by Assign AI to help them get reviewed by a clinician sooner. They do not receive a priority order or flag - so remain in the middle of the PATCHS inbox.

Screenshot_2021-03-19_at_09.29.22_3.png

Contents

Priority clinical

  1. Urgent clinical doctor: Health problems that require input from GPs where the patient could be harmed if the problem is not addressed in the next 48 hours. Suggested examples:
    • Suspected cancer
    • Acute pain
    • What happens to 'Urgent clinical doctor' requests in your fictional 'typical' GP practiceThey are sent to a GP who contacts the patient within 48 hours.
  2. Emergency clinical doctor: Health problems that require input from GPs where the patient could be harmed if the problem is not addressed on the same day. Examples:
    • Suicidal thoughts with intention 
    • Asthma exacerbations
    • Infections with the potential for sepsis e.g. cellulitis
    • What happens to 'Emergency clinical doctor' requests in your fictional 'typical' GP practiceThey are sent to a GP who contacts the patient on the same day.
  3. Emergency services: Health problems that require an ambulance or attendance at the Emergency Department (ED) because the patient could be harmed if the problem is not addressed immediately. Examples:
    • Suspected heart attacks
    • Suspected strokes
    • What happens to 'Emergency services' requests in your fictional 'typical' GP practice?Patients are directed to call 999 or go to the ED immediately - either by your receptionist or a clinician.

What happens to 'Priority' requests in PATCHS?

For 'Urgent clinical doctor' and 'Emergency clinical doctor' requests:

  • During opening hours: They are sent to a 'Clinical' inbox by Assign AI, and are marked with an orange flag and ordered to the top of the PATCHS inbox by Urgency AI. This is to help them get reviewed by a clinician sooner.
  • Out of hours: PATCHS advises patients to call 111 in order to get help sooner. Patients can cancel their request and not submit it to your GP practice, and call 111 instead. You can read more about how this happens in this article.

PATCHS shows patients information links from NHS.uk relevant to their request, and gives them the opportunity to cancel their request and not submit it to your GP practice. You can read more about how this happens in this article.

For 'Emergency services', PATCHS advises patients to call 999 or go to the ED immediately at any time of day. Patients can cancel their request and not submit it to your GP practice, and call 999 or go to the ED instead. You can read more about how this happens in this article.

 

Screenshot_2021-03-19_at_09.29.22.png

Contents

Special cases

  • Photos: Always assume the patient has submitted a photo e.g. if complaining of a rash.
  • Multiple requests: If the message contains multiple requests from the patient - use the one with the highest clinical priority to make your triage decision. For example, requests for fit / sick notes that mention new or worsening symptoms should be triaged for clinician review not admin.
  • Medication requests: Requests solely asking for medication are most likely the patient requesting a repeat prescription e.g. contraceptive pill, antidepressants. They can therefore be issued without clinical review and triaged as 'Medication'. Only assume the patient needs a medication review if explicitly stated. Medication requests should only be triaged for review by a clinician if they explicitly state they need a medication review or mention health problems e.g. new or worsening symptoms, antibiotics for UTI, or steroids for asthma exacerbations.
  • Fit / sick notes: should be triaged as 'Admin clinical' unless the request mentions new or worsening symptoms that you think requires review by a clinician.
  • Alternative provider: Should only be used for non-urgent, non-emergency conditions only. They should not be used for same-day assessments e.g. A&E for an X-ray for suspected fracture, or optician for sudden onset eye symptoms.
  • Triage anything that needs a pre-arranged in-person visit as 'Admin non-clinical'. For example, blood test, smear test, coil fitting, implant fitting, suture removal, flu jabs etc. This is because in practice these would not be dealt with via PATCHS, and would typically just be booked straight in by a receptionist.
  • Requests where it isn't clear what the patient needs initially because there isn't enough information. For example, the patient only writes a word or two (as long as those words aren't 'chest pain' or similar!). These should generally be triaged as 'Admin non-clinical' so a receptionist can ask for more information first - they can then pass the request onto a clinician if needed afterwards.

Contents

What are the main topics of this request?

The purpose of this question is to identify requests where it would be helpful for PATCHS to ask the more questions (e.g. red flags in back pain) or questionnaires (e.g. the PHQ-9 in patients with depression). Therefore, it's likely most helpful for health problems, rather than medication or admin requests - though there are exceptions of course! At the moment, this question is testing the Mental Health AI.
 
Tips
  • Only label topics that are substantially mentioned. For example, if mentioned in a way that would make you want to discuss it further with the patient. Don't label topics only mentioned in passing.
  • Select more than one topic if more than one is mentioned. If multiple clinical topics are mentioned, please select all relevant labels.
  • 'Other' is a helpful answer! Knowing a request mentions other topics helps us identify negative examples. You can select 'Other' on its own where no 'named' topics are mentioned at all: for example, a request about hair loss. You can also select 'Other' if another topic is mentioned in addition to other 'named' clinical topics: for example, if depression, back pain, and eczema are all mentioned in one request you should select 'Depression', 'Back pain', and 'Other' (for eczema).
  • Do not label requests solely asking for medication as the condition they treat. Only label them with a clinical topic if they also mention health problems that require review by a clinician. E.g. Do not label requests solely asking for antidepressants as 'depression' unless they mention symptoms like worsening low mood or suicidal thoughts.
 

Does this patient need a face-to-face appointment?

This question is trying to find out whether you think the patient needs an in-person appointment to resolve their query. It is testing the Face-to-Face (F2F) AI.

The question should only apply to requests that would be passed onto a clinician to resolve i.e. requests you have triaged as Routine clinical or Priority clinical.

Always answer 'No' if you have triaged the request as either Non-clinical or Signpost - even if the request does technically end up in as an in-person appointment (e.g. smear test, flu jab etc). This is because only clinicians who use PATCHS can say whether the patient needs an in-person appointment - not receptionists or other non-clinical staff.

Contents

This request was not written in English

Tick this box if the patients' request is written in a non-English language.

Contents

Notes

This section is for you to write notes on your rationale for choosing a particular label, and is optional. We would recommend writing notes for any labels you have applied where the decision hasn't been straightforward. You should only write notes that you would be comfortable with other GPs in the project reading because they will be anonymously shared with GPs 1-3 during stage 2 (reconsideration) and potentially also with the expert panel during stage 3 (resolution).

Contents