A Behind-the-Scenes Look at Healthcare IT Analyst Rankings and Reports: What You Should Know

When it comes to healthcare IT analysts (e.g., KLAS Research) and how they rank technology vendors, what’s important for healthcare leaders, IT professionals, and clinicians to know? What methodologies do these analysts use to evaluate and rank vendors? How do they gather data—through customer feedback or other mechanisms? Do they offer their opinions or let the data speak for itself? How are new categories, such as machine learning, determined and defined?

From healthcare-centric IT analysts, such as KLAS and Advisory Board, to cross-industry analysts that cast a much wider net, such as Gartner and International Data Corporation (IDC), this article will turn healthcare leaders into educated consumers of these analysts’ rankings and reports by explaining the key differences between their approaches and methodologies.

Getting to Know the Healthcare IT Analyst Landscape—Ranking vs. Consulting; Unbiased vs. Opinionated

There are several prominent healthcare-specific IT analysts: KLAS, Black Book Rankings, Chilmark Research, and Advisory Board. With various areas of focus, from ranking to consulting (or both), and dramatically diverse research methodologies, understanding the different IT analyst approaches helps healthcare leaders not only find the information they’re looking for, but also develop a keen awareness of how the information was collected. Having a high-level awareness of varying analyst methodologies—knowing, for example, which analysts solicit feedback from a list of customers supplied by a vendor versus customers they find on their own—can help leaders more effectively interpret rankings.

KLAS Research (Ranking Focus)

Focused primarily on ranking vendors, KLAS is a healthcare research and insights firm. Every year, KLAS recognizes top-performing vendors in a variety of segments (e.g., population health and patient access) in its Best in KLAS report. Many health systems purchase these reports to help them make key technology decisions. In early 2017, KLAS released a comprehensive overview of the population health market to help healthcare leaders answer a pressing question: Which vendors are meeting the needs (or are poised to meet the needs) of health systems?

The KLAS research model is customer focused. It’s designed to give healthcare leaders a boots-on-the-ground, customer-centric perspective, and allow vendors to see what customers say about them when they’re not in the room. The model is based on the KLAS belief that healthcare leaders and IT professionals rely on the transparency of their peers to uncover hard-to-get facts that reveal the true story. Its research team conducts in-depth interviews with clinical, financial, IT, and C-Suite leaders within a range of organizations, from single-physician practices to health plans to get candid, closed-door feedback from real users—key decision makers engaged with the solutions they’re evaluating.

What differentiates KLAS from other healthcare-specific vendors is the KLAS “Konfidence” score, which reveals how many customers KLAS spoke with to determine its ranking. For example, if KLAS speaks with fewer than five customers, then it won’t include that vendor in its ranking. If KLAS speaks with between five and 14 customers, then it gives that vendor a provisional ranking—enough customers to receive a report ranking, but not enough to receive a Konfidence score, which requires at least 15 customers (the minimum required to deliver a truly “Konfident” ranking).

Black Book Rankings (Ranking Focus)

Black Book Rankings, the healthcare-specific segment of Black Book Market Research, focuses on healthcare technology and services, client experience polling, and research. Like KLAS, Black Book is in the business of ranking to meet health systems’ research needs in a variety of areas (e.g., patient satisfaction and physician performance). Unlike KLAS, Black Book doesn’t ask vendors for a list of their customers—it finds customers on its own. Black Book is recognized as an unbiased source for polling, surveys, market research, opinion mining, and client experience results in healthcare.

Although this investigative, self-sufficient approach limits the quantity of vendors it can rank, it touts a more neutral, unbiased perspective. Black Book’s website emphasizes this uniquely unbiased stance: “Black Book is the only unbiased rating system. Honestly vendor agnostic. They independently survey all user groups, not just CIO’s staff and they don’t engage vendors until results are in.”

Chilmark Research (Ranking and Consulting Focus)

Chilmark, which prioritizes both ranking and consulting services, covers several HIT domains—those it believes have the greatest promise to transform and improve patient care:

  • Analytics and BI (e.g., 2017 Healthcare Analytics Market Trends Report).
  • Cloud computing models.
  • Health information exchange.
  • IT strategies for patient engagement.
  • mHealth technology and adoption.
  • Remote monitoring and telemedicine.

Chilmark’s mission centers around objectivity: “Based on our core belief that HIT plays a crucial role in improving the quality and efficiency of care, we foster the effective adoption, deployment, and use of HIT by providing objective, high-quality research into technologies with the greatest potential to improve care.” Chilmark strives to achieve objectivity throughout its research philosophy, which states, “vendors have the opportunity to review their company narratives for accuracy, but not the rankings. This can at times be awkward, but independence is essential to our ability to deliver a clear, unbiased snapshot of the market and its offerings, and we go to great lengths to maintain it.”

Chilmark uses primary and secondary research to understand what areas in healthcare IT are working, what areas require caution, and how certain sectors will develop over time:

  • Primary research: solicits qualitative and quantitative data about the market from targeted stakeholders (e.g., questionnaires and in-depth interviews).
  • Secondary research: monitors industry reports, regulations, and news for relevant trends and insights into a technology’s strengths and weaknesses.

In addition to its published reports, Chilmark conducts targeted research for organizations looking for the independent analyst’s perspective (e.g., market opportunity assessments and competitive analyses). Unlike KLAS and Black Book, which rely exclusively on customer data to inform rankings, Chilmark is willing to offer its opinion of a vendor—a strong editorial stand—along with its rankings. This added layer of assessment can be helpful to healthcare leaders looking for more than just a ranking. 

Advisory Board (Consulting Focus)

Like Chilmark, Advisory Board willingly shares its editorial opinion of vendors. Although its business model is centered on consulting, not ranking, Advisory Board emphasizes its trio of services—not only research, but also technology and hands-on consulting support.:

  1. Research: shares its role-specific research (health system executive teams, service line leaders, and industry service providers/suppliers) in many ways, from on-call expert consultations to live web conference presentations.
  2. Technology.
  3. Consulting (growth, care variation reduction, and revenue cycle management): a team of 2,600+ healthcare professionals and network of 4,400+ member health systems provides healthcare leaders with best practices and the guidance, tools, and resources to implement them.

So, while ranking-oriented KLAS and Black Book let customer feedback dictate rankings in their attempts to be truly impartial, consulting-oriented Chilmark and Advisory Board offer their evaluations of vendors based on (but not limited to) what the data reveals.

Cross-Industry Analysts that Apply a Healthcare-Specific Lens

Several cross-industry, non-healthcare specific IT analysts, including Gartner, IDC, and Frost & Sullivan, apply a healthcare-specific lens to their findings (e.g., a healthcare-specific assessment of IT security vendors) to assist healthcare stakeholders in their health IT vendor decision making.


Gartner is known for its independent, objective, cross-industry technology reports. According to its website, Gartner “helps business leaders across all major functions in every industry and enterprise size with the objective insights they need to make the right decisions.” Gartner’s research process map includes five steps:

  1. Scenarios: analysts develop five-year forecasts guided by the question, “What revolutions or major discontinuities will take place over the next five to 10 years?”
  2. Surveys: analysts gather information through formal and informal surveys.
  3. Pattern Recognition: analysts analyze collected data in search of patterns.
  4. Stalking Horse: analysts use a “stalking horse” (an idea examined from different viewpoints) to answer the question, “What position should Gartner take?”
  5. Search and Verify: the final step in verifying Gartner’s position is searching written material from all channels (e.g., financial reports and newswires). Coupled with internal debate over stalking horses, Gartner reaches consensus on its position (and its probability of occurring) or it’s discarded.

Gartner’s research process concludes with a vendor and peer review, in which all ideas and projections are evaluated and challenged. Vendors review any research containing information about them.

Gartner strives for objectivity with a strict code of conduct, which establishes boundaries for its analysts and associates (e.g., analysts/associates can’t own stock in companies/sectors they cover). And because Gartner doesn’t sell or implement technology solutions, it claims no vested interest in the success or failure of any one technology. 

International Data Corporation

IDC offers industry-focused advice for IT buyers in several sectors, including healthcare, and touts an unbiased stance on its website, stating “our most fundamental research mission is to accurately assess and forecast the size and shape of the key IT market segments.” IDC pursues objectivity with a research approach based on several standards:

  • Documentation of methodologies used and forecast assumptions.
  • Projections from only surveys that are statistically sound.
  • Validation and “sanity checks” to help provide boundary conditions for major forecast.
  • Standard definitions across geographies sand coverage areas.
  • Common analytical tools to ensure consistency.

IDC’s market sizing and forecasting processes rely on collecting data using primary and secondary sources (e.g., interviews with IT vendors and user surveys) and validating preliminary values in the market model. 

Frost & Sullivan

Frost & Sullivan is a global research and consulting firm with a Transformation Health practice. Although its primary focus is helping organizations develop growth strategies, its research arm, 360° Research, integrates seven perspectives into a big-picture view of factors that impact every industry, including healthcare:

  1. Competitive intensity
  2. Customer dynamics
  3. Industry convergence
  4. Disruptive technologies
  5. Global mega trends
  6. New business models
  7. Emerging markets

Healthcare leaders with an understanding of the key IT analyst players, healthcare-specific or not, will make better decisions because they know where to look for the information they need, and understand how that information, be it a rank or recommendation, was determined. Some analysts pride themselves on fact-based vendor evaluations; others pride themselves on decades of industry expertise layered on top of data-driven assessments. Neither is superior, but it helps to know the methodology and the motivation behind rankings and reports.

New Market Categories and Why They’re So Challenging for Everyone

New categories (e.g., machine learning) are challenging for IT analysts, health systems, and vendors alike. It’s difficult for IT analysts, like KLAS, to understand the market’s needs, define the category, and group vendors effectively and fairly. They try to group vendors with common capabilities, but these capabilities evolve quickly and there aren’t many customers to solicit feedback from. Health systems don’t always know what they’re looking for or what the precise problem is. And vendors don’t know how to respond to this ambiguity, so there’s variation from one vendor to the next. IT analysts are in the middle trying to make sense of it.

Defining new categories is a hurdle; take machine learning for instance. Many existing machine learning solution vendors provide academically-appealing, standalone models that don’t translate into meaningful, scalable outcomes. Useful, guiding machine learning definitions are hard to find. Arguments about what constitutes the “learning” in machine learning abound. At the end of the day, the machine learning question can be as straightforward as, “What machine learning model is going to reduce sepsis risk in my ED?”

Healthcare machine learning vendors want IT analysts to ask these kinds of questions when they talk to vendors in this new category. For many vendors, all the machine learning in the world is useless unless health systems can use it to improve patient outcomes. The question isn’t, “who has the biggest machine learning capability?”; It’s “Who is effecting the most positive change with It?”

The bottom line: new categories are difficult to define because it’s hard for healthcare leaders to know what they need, hard for vendors in an emerging segment to achieve common functionality, and hard for IT analysts to identify common elements across different vendors in the same segment. Fortunately, defining new capability categories and grouping vendors that promise that capability will evolve and refine over time into more mature, reliable categories with a robust group of customers to base rankings on.

Healthcare Leaders Must be Educated Consumers of IT Analyst Rankings and Reports

It takes a significantly more nuanced interpretation of IT analyst rankings and reports to extract the information healthcare leaders really need. By understanding healthcare-specific and cross-industry IT analysts’ diverse approaches and goals (and new categories, like machine learning), healthcare leaders can better understand the (often expensive) output (rankings and reports), that comes from these analysts.

Additional Reading

Would you like to learn more about this topic? Here are some articles we suggest:

  1. Ten Great Behaviors of the Best Healthcare IT Vendors
  2. Three Must-Haves for Generation Innovation in Healthcare IT
  3. Eight Reasons Why Chief Data Officers Will Help Healthcare Organizations Thrive in the Future
  4. An Inside Look at Building Machine Learning for Healthcare
  5. How Machine Learning in Healthcare Saves Lives