To answer our question, we need a sample of patient requests where we know:

We have therefore devised the following approach, which happens in three stages. The whole process is summarised in this diagram:

We have recruited a team of ~30 experienced GPs from around the UK (you!) to 'label' a sample of anonymised patient requests collected from PATCHS - by labeling we mean putting a triage decision and stating whether the request relates to anxiety and/or depression. Each member of the team will be allocated the same number of requests to label at random, and each request will be labeled by four GPs independently (so you won't know what the other GPs put as their labels). If the labels put by GPs 1-3 all agree we will accept this as the 'correct' label for the request. If they don't all agree, the request will proceed to stage 2 for 'reconsideration'. GP 4 will be the GP against which we will will compare the performance of the AIs - because the requests will be allocated at random between ~30 GPs, all GPs in the team will contribute to the performance of GP 4. The section below describes in more detail how to conduct the labeling.

 

Any patient requests where there isn't full agreement between the labels of GPs 1-3 will be reviewed by those GPs to ask them whether they would reconsider their decisions. Initially we will only do this with the triage labels - not the other labels. They will be shown what labels they initially put, and also anonymously what the other GPs put that made up GPs 1-3 (not GP 4). They will also be presented with any notes they made, along with any notes the other GPs also made. Each GP could either keep their original labels, or change them. They can also add further notes to justify their decision if they wish. If, after reconsideration, all GPs 1-3 now agree, we will accept accept this as the 'correct' label. If they don't all agree, the request will proceed to stage 3 for 'resolution'.

 

Any requests that have failed to get full agreement between GPs 1-3 after the initial labeling and reconsideration rounds will be sent to an 'expert' panel for them to make a decision on the labels. The panel will consist of one GP from the health technology industry, one academic GP with an interest in patient safety or informatics, and one GP who has extensive experience in remote patient triage. They will meet once to discuss these requests and will be presented with each one's 'history' i.e.

GPs 1-4 will be paid one unit for each request they label - this includes payment for reviewing requests in stage 2. So you will be paid the same amount for a request where GPs 1-3 all initially agree, vs a request where they don't and have to review it again. GPs 1-4 will receive a bonus unit of payment (so two units in total) if their final label - either applied in stage 1 or 2 - matches the final 'correct' label chosen. We've structured payment this way to encourage high quality labeling. The expert panel will be paid for their time in the meeting.

 

The options here relate to high-level triage decisions in colour and low-level triage decisions in black. We are primarily measuring agreement between GPs 1-3 on the high-level decisions only. We will still analyse agreement on the low-level decisions, so It's still important to put the most appropriate low-level decision, however if you're struggling with a particular triage decision remember it's the high-level decision we're most interested in.

 

Non-clinical (High-level)

1. Medication: any request solely asking for medication - either acute or repeat.
2. Admin clinical: admin request that requires input from a GP. Examples:
   • Fit / sick note
   • TWIMC letter
   • Insurance forms
   • Reviewing letters from consultants
3. Admin non-clinical: admin request that would be best directed to a non-clinician (e.g. receptionist, secretary). Examples:
   • Booking pre-arranged appointments (e.g. blood tests, blood pressure measurement)
   • Requesting information about opening hours
   • Inappropriate requests for help (e.g. facemask exemption letters, shielding letters)

Marking a request as 'non-clinical' would send it to the bottom of the inbox in PATCHS and would be dealt with last:

 

Signpost (High-level)

Low-level triage decisions are:

1. Self-care: A minor illness that does not initially need a health professionals' input to treat. The patient should initially care for themselves with self-help guidance. Examples:
   • Viral sore throat
   • Gastroenteritis
   • Sunburn
2. Pharmacy: A minor illness that does not necessarily need a clinician’s input to treat. Patients may initially consider going to a pharmacy for advice or over-the-counter medicines. Examples:
   • Mouth ulcers
   • Conjunctivitis
   • Head lice
3. Alternative provider: A non-emergency request that is more appropriate to be addressed by an alternative care provider rather than a GP practice. Examples:
   • Dental problems
   • Open access physiotherapy
   • Ear syringing

Marking a request as 'signpost' would divert the patient away from the practice so the request wouldn't appear in the practice's inbox and wouldn't be dealt with by the practice at all.

Routine clinical (High-level)

Low-level triage decisions are:

1. Routine clinical doctor: A health problem that would traditionally require input from a GP i.e. requires a diagnosis and/or prescription. In some practices this may be performed by a specially trained nurse or ANP. Patient is unlikely to be harmed if the problem is not addressed within the next 48 hours. Examples:
   • Migraine
   • Hair loss
   • Follow-up from a previous consultation
   • Chronic condition management
2. Routine clinical non-doctor: A health problem that could be dealt with by a clinician who is not a GP (e.g. a nurse or HCA) i.e. that doesn't necessarily require a diagnosis or prescription. Patient is unlikely to be clinically harmed if the problem is not addressed in the next 48 hours. Examples:
   • Contraception management
   • Lifestyle counselling
   • Travel advice
   • Chronic condition monitoring

Marking a request as 'routine clinical' would send it straight to a clinician in the middle of their inbox in PATCHS:

 

Priority clinical (High-level)

Low-level triage decisions are:

1. Urgent clinical doctor: Health problems that require input from GPs where the patient could be harmed if the problem is not addressed in the next 48 hours. Examples:
   • Urinary tract infections
   • Cellulitis.
2. Emergency clinical doctor: Health problems that require input from GPs where the patient could be harmed if the problem is not addressed on the same day. Examples:
   • Suicidality
   • Asthma exacerbations
   • Suspected cancer.
3. Emergency services: Health problems that require an ambulance or attendance at the Emergency Department (ED) because the patient could be harmed if the problem is not addressed immediately. Examples:
   • Suspected heart attacks
   • Suspected strokes

Marking a request as 'priority clinical' would send it straight to a GP and put it at the top of their inbox in PATCHS and would be dealt with first:

 

What is the main topic of this request?

The purpose of this question to identify requests that could be further evaluated by PATCHS asking the patient more relevant questions (e.g. red flags in back pain) or questionnaires (e.g. the PHQ-9 in patients presenting with depression). Therefore, it's likely most helpful for health problems, rather than medication or admin requests - though there are exceptions of course!