A few years ago, I found myself having a similar conversation about digital measures time and time again with partners at pharma and biotech companies:
Study team: “We want to measure activity using a sensor at home.”
Me: “Activity is broad. HumanFirst’s Atlas platform catalogs 50+ ways activity could be measured — what’s most interesting to you?”
Study team: “Let’s do moderate to vigorous activity ”
Me: “Oh excellent. There are over 1000 ways that measure has been reported in the literature with different time domains, accelerometer cut points or METs .”
Study team: “Yikes. How do I narrow it down?”
Me: “Well it should be patient-focused and the industry doesn’t have any standards, yet.”
Study team: “We don’t have time to interview patients. But our clinical team is curious about this new time active calculation — perhaps we try that one?”
Me: <Realizing we are not going to be aligning measures to what matters to patients in this engagement.>
In my past career as a Registered Dietitian, I encountered oncology patients whose tumors had shrunk on MRIs (a successful outcome!), but were still suffering from debilitating fatigue preventing them from living life how they wanted (not a successful outcome in my view). I realized that many treatments weren’t optimizing for what mattered most to patients.
In the clinical setting, we get just a small snapshot of data. Remote monitoring increases that aperture and uncovers the previously unknown to better prevent and treat chronic disease. This data collected in the home can be much more aligned to patient preferences. According to the Digital Medicine Society (DiMe) Library of Digital Endpoints, over 96 sponsors are currently using digital endpoints and a new problem has emerged. Our community was starting down a path of developing thousands of meaningless digital measures that lacked standardization.
“Remember how many reported outcome measures we saw at CTTI? I think it’s getting worse,” I said to Jen Goldsack in the early days of DiMe in 2019.
Jen, the CEO of the Digital Medicine Society (DiMe), had been the lead researcher on a systematic review and database of studies using mobile technology that I worked on at Clinical Trial Transformative Initiative (CTTI) back in 2018.
Jen agreed. Without a common framework and standardized language it would be impossible. We knew we wanted to make a shift in the industry. So, we outlined a plan.
Step 1: develop the underlying digital measure framework.
We brought in Bray Patrick-Lake, an expert in patient advocacy, and co-authored the Measures That Matter paper, later incorporated into The Playbook: Digital Clinical Measures Edition, and has gleaned 40 citations to date.
With this framework, the next time a team said they wanted to measure activity, I was ready to point them to a publication and discuss how the Meaningful Aspect of Health (MAH) should relate to a Concept of Interest (COI) and then the outcome to be measured.
Step 2: address the lack of standardization.
To support the industry in categorizing the ever-growing list of digital measures and deciding which measures we should be capturing, I led the HumanFirst Applied Sciences team in tagging and grouping the measures we were seeing in the literature in highly detailed categorical format.
The ultimate intention was to one day bring the industry to consensus on the measures that matter.
In September 2022, that goal was reached for atopic dermatitis with the first pre-competitive digital measures development from DiMe. The group did the important work of interviewing and surveying actual patients and caregivers to develop this resource.
The DiMe working group developed an impressive terminology and ontology for nocturnal scratch to define how the measure should be assessed and reported across studies and in different patient populations.
The HumanFirst team knows how difficult this ontology work is, because we’ve released it across 25 different ontologies spanning 800 medical conditions. The industry may recognize HumanFirst’s Atlas as a place to go to find and evaluate sensor technologies.
But here’s what you may not realize: over the past three years, that highly detailed categorical format of tagging that we undertook has evolved into an ontology for digital measures, covering scratch, activity, sleep, cardiovascular assessments, pulmonary assessments and many more.
Notably, this ontological work at HumanFirst goes hand-in-hand with DiMe’s development of measures for nocturnal scratch.
The Measure Ontologies by HumanFirst were constructed from review of thousands of studies — if a digital measure has been captured and reported in a peer-reviewed study or clinical trial, it’s been tagged in Atlas — and hundreds of thousands of hours making meticulous decisions about the appropriate categorization.
- Imagine — for example — does the Clock Drawing or Finger Tapping Test count as ”cognition,” even though it can also involve capturing hand movements, reaction times and dexterity?
- Is it important to distinguish between technologies that can capture just blood pressure, versus ones that can also capture change in blood pressure over time?
These kinds of questions don’t have easy answers, but need to be discussed and thought through.
Nowhere else in the world can you find a data platform that enables targeted literature searchers for an exact outcome in the exact context of use. For example, total sleep time in patients with depression or steps per day in patients with cardiomyopathy. This allows for an understanding of how often that measure has been assessed, in what conditions and using what technology — and all in a matter of not hours, or minutes, but seconds. I’m proud to release it for use by the research community.
To whittle down the number of measures and reach true standards, the industry needs more pre-competitive projects from DiMe, who is well positioned to bring together the necessary stakeholders. HumanFirst is pleased to be participating in the core measures project for activity next year and is looking forward to the outputs of the Alzheimer’s Disease & Related Dementia project that is actively underway.
We plan to update this ontological framework— if you’re working on new measures please let us know about your outcomes, and we’ll continuously ship updates to this ontology and track what’s happening across the industry.
Special thanks to the DiMe team: Jen Goldsack, Lucy Cesnakova, Laura Shannon and Gabrielle Dell’Aquilo and the HumanFirst Applied Sciences team: Manny Fanarjian, Caprice Sassano, Catherine Adans-Dester, Max Gaitán and Nathan Coss for making this work possible.
