Skip to content
Back to blog
Introducing Validity & Meaningful Change Indicators
Company

Introducing Validity & Meaningful Change Indicators

Published: 20/11/2018

Written by: Creyos

Edit: These updates are now available in your account. If you have any questions or comments, please do not hesitate to contact us.

Mike here, staff scientist at Creyos (formerly Cambridge Brain Sciences). Based on feedback from practitioners such as yourself, there are several new improvements to the Creyos Health report that will be available in your account on Tuesday, November 27th. These updates are meant to answer two important questions that will help make interpreting assessment results even easier:

1. How do I know if a patient’s performance on a task is likely to be valid?
2. How can I tell if a change in a patient's results is potentially significant?

Keep reading for additional details and if you have any questions, or would like a personalized walkthrough of any of these updates in advance , please let us know and I’d be happy to assist.

Overview of the New Creyos Health Report

The new features of the upcoming Creyos Health report are highlighted below:

new-feature-report-1542817894

Each of the new features above are detailed in the following sections.

Task Validity Overview

To date, there has been no simple or efficient way to verify that performance on a task is truly reflective of a patient’s cognition at that moment, as there may be rare instances in which a patient may not have fully understood the task instructions, or was distracted, leading to an inaccurate result.

That's why Creyos Health reports will soon include a Validity Indicator—a new and easy-to-read section that conveys whether performance on a task is within reasonable bounds and therefore likely to be valid, helping you gain confidence that task scores reflect a patient's true cognitive state.

How does the Validity Indicator work

When a task score meets the validity criteria, a green box will appear within the "Validity Indicator" section of the report. If there is reason to believe based on the validity criteria that a task score may be invalid, a red box will appear instead, as shown below:

meaningful-change


Validity indicators will also appear within data points on the Raw Score Trends graph as shown below:

trends-validity

How do we determine if a task score is likely to be valid?

Each task has a set of parameters that must all be met for the score to be considered likely valid. For instance, the Digit Span task has the following validity conditions:

  • Number of attempts: must be greater than zero
  • *Number of correct answers: must have one or more correct answers, and fewer than 11 correct answers
  • *Test duration: must be longer than 40 seconds, but shorter than (or equal to) 362 seconds
  • *Max score: must be 3 or higher, and less than or equal to 12
  • *Average score: must be 3 or higher, and less than or equal to 8.4

*99% of task scores in the Cambridge Brain Sciences normative database fall within these bounds.

Other tests may have some additional criteria. For instance, Double Trouble has 14 different validity conditions, including an analysis of reaction times for each of the different types of problems a patient may encounter during the task.

What should be done if a task score is likely to be invalid?

If a report shows a potentially invalid result, it is up to you to evaluate all sources of information at your disposal to decide the appropriate course of action. For instance, you may have a patient complete an assessment that includes three memory tasks, with only one of those memory tasks showing as potentially being invalid. In this case, you may choose to forego re-assessing the patient. However, if all of the memory tasks are flagged as potentially being invalid, you may ask the patient to complete the assessment a second time.

As another example, perhaps you are conducting a standard four-task screen as part of your intake process, in which each of the four tasks targets a different cognitive domain. In this case, even if only one of the tasks returns an "Invalid" result, you may wish to ask your patient to repeat either the full assessment, or the one task that was determined likely to be invalid.

Note that the above scenarios are simply examples and every situation should be evaluated individually. Ultimately, it is up to you to determine the appropriate course of action to take in the event of a potentially invalid task result.

I pre-purchased a package of assessments—will I lose a report if I reassess my patient due to an invalid score?

You will not lose a report if you reassess a patient due to an invalid score. Simply issue another assessment, then contact us to let us know, and we will add an extra report to your account.

Note that if you're currently on our new subscription pricing structure, which allows you to reassess a set of patients an unlimited number of times during their treatment plans without additional costs, you do not have to worry about this step. For those of you who may not know about the new pricing structure and would like to learn more, book a quick 5 minute call with our team.

Meaningful Change Overview

As you likely know, Creyos Health can be a valuable tool to assess changes in cognition over time. However, distinguishing between noise and a meaningful change (e.g., a deterioration resulting from a clinical condition, or perhaps an improvement due to a successful treatment protocol) can be difficult. That's why starting today, you'll be able to objectively determine whether a change in an individual's cognitive performance is significant relative to their latest valid score and first valid score (i.e., their baseline).

How does the Meaningful Change Indicator work on Creyos Health reports?

There are two specific questions the Creyos Health report will now answer as it relates to meaningful change:

  • Has there been a potentially significant change relative to the patient's baseline assessment?
  • Has there been a potentially significant change since the last time the patient was assessed?

On the report, you'll see a new block dedicated to the meaningful change indicator, as shown below:


meaningful-change

Note—it is possible that only one of the two conditions for meaningful change are met. If that is the case, you will see only the one relevant sentence, instead of the two shown above.

What are the conditions for determining if a change is likely meaningful?

The vast Creyos normative database allows us to characterize how performance on every task fluctuates over time across a range of testing intervals. Using this data, the bounds of a meaningful change for every test have been quantified and implemented into Creyos Health.

Further details regarding what constitutes a meaningful change for each task will be provided on Tuesday, Nov. 27th as part of the Creyos Health Results Interpretation Guide.

Other General Improvements

Further to the updates outlined above, we have implemented several other usability enhancements based on your feedback. They include the following:

  • Updated raw score trends graph, which now includes dates and percentile ranks for each data point.
  • New classification criteria terminology, utilizing more objective language ("High" or "Low"), as opposed to the subjective terms used previously (i.e., "Poor" or "Superior").
  • Updated task descriptions, which now include how the task outcome measures may relate to common everyday activities (e.g., example activities related to Number Ladder include planning the day’s errands, then carrying out those errands in the correct order by memory).

 

Let Us Know Your Comments

These enhancements are a direct result of the feedback from, and discussions with, the Creyos Health community. As always, if you have any other comments or questions, we are more than happy to assist and look forward to hearing from you.

Join thousands of healthcare professionals who receive a monthly update on the latest research and news in brain health.