Did the tool make this task easier?

What we found in the news

An October 2024 JAMA Network Open study (opens in a new tab) examined AI-generated draft replies to patient messages in one health system. Staff views varied across professional roles. The authors offered a possible explanation:

“the specific messages each group sees are different”

— Eden English and colleagues

This quality-improvement study concerned particular clinical workflows and self-reported views. It does not establish time savings for every role or another service. It does suggest a useful evaluation question: whose task are we measuring when we say a tool helps?

Why this matters to us

A draft can arrive quickly while the work around it remains substantial. Someone still checks the details, adjusts the tone, finds missing information, and makes sure the result reaches the intended person.

Imagine a community team trying assistance with activity invitations. One colleague finds it useful for a first draft. Another spends time correcting assumptions about transport. A participant wants the final invitation to answer a simple question about bringing a friend.

Those experiences all belong in the review. Counting generated invitations would tell us that the tool was used. It would not tell us whether the invitation became more useful.

Follow the whole task

Start with the question the invitation needs to answer and continue through the person receiving it. Identify the points where the work changes hands.

In our example, the activity coordinator supplies confirmed details. A colleague prepares the draft, a reviewer checks it, and the participant tests whether the final version makes the next step clear.

Consider the ordinary effort at each stage. Did someone have to search for information the draft should not have guessed? Did a reviewer rewrite a paragraph? Did the participant need to call because a key detail was absent?

Choose a manageable way to record those observations. A few well-described examples can help a team improve an early trial, provided they are not presented as a representative outcome for everyone.

Ask people in different roles what helped

The colleague preparing invitations and the colleague answering subsequent calls may judge usefulness differently. Both have information about the same task.

Invite specific examples. Which part became easier? Where did the tool add checking work? Was there a type of invitation for which the usual approach still worked better?

Include the people receiving the communication. They can tell the team whether the invitation answered the question they had, used a format they could access, and made the next step understandable.

Ask about non-use too. Someone may have chosen another route because the task was unusual, the tool was unavailable, or the result required too much revision. Understanding that choice is more helpful than assuming reluctance.

Decide what is worth keeping

Bring the observations together before widening the trial. The team might retain assistance for a narrow drafting task, change its instructions, or keep particular situations with an experienced colleague.

Be explicit about the basis for that decision. Positive comments from a small group can guide the next test; they do not yet establish a general productivity gain.

Keep quality alongside effort. A faster invitation that leaves out an access detail may simply move the work to the person trying to attend.

Agree when the team will look again. A workflow can change as people gain experience or the tool is updated. The review should remain connected to the task and the people involved, rather than becoming a permanent verdict based on the first demonstration.

What this means for the people you serve

People should be able to judge the result by whether it helps them take the next step. Their contribution may be a question the team forgot to answer or a suggestion that makes the invitation more usable. Those observations belong beside staff feedback when deciding what to continue.

The genus connection

Today, genus partner-portal workflows can support configured content preparation, staff review, and operational information. AI assistance has a bounded role within the enabled workflow, with people responsible for the material they approve.

We would begin a review with a real task and an agreed purpose. Look at the work across preparation, checking, and use, then decide what the evidence supports. An organization can learn something valuable from a modest trial without turning it into a claim about every service.

One question to take away

After the draft appeared, whose work became easier—and whose experience have we not yet asked about?