How to Evaluate a Clinical Analytics Vendor: A Checklist


The clinical analytics market is crowded and chaotic, with many vendors lining up to claim they can help providers use their data to improve care and lower costs. Making a wrong decision in the current market will paint you into a corner that could last four to five years, at least, while the market rapidly moves past your organizational payment model and towards accountable care and quality-based models. You cannot afford the time to recover from a bad decision. So, how do you cut through the noise to find a solution that makes the most sense for your organization?

I designed this paper to help you with that decision. I outline general criteria to help you assess a clinical analytics vendor and discuss the technology and change management an effective analytics solution should support. In addition, I introduce a Healthcare Analytics Adoption Model that will help you analyze vendors and evaluate your own plans for analytics adoption. 

Six Criteria for Assessing a Clinical Analytics Company

Embarking on an assessment with the knowledge of key, general criteria can help you determine whether a vendor has the philosophy, experience, and viability to lead to a successful outcome for your organization. By evaluating vendors according to the following, you can narrow the list considerably: 

  1. Completeness of VisionWhat lessons does the vendor bring from the past healthcare analytics market, and how have they adjusted their current strategy and products accordingly? Can they bring lessons from other industries that are more advanced in their adoption of analytics? What is the vendor’s understanding of the present market and industry requirements? What is the vendor’s vision of the future for clinical analytics? Look for vendors who can clearly outline how they have evolved to meet—and anticipate—industry needs. 
  1. Culture and Values of Senior LeadershipIt’s no cliché, but rather the precise truth: the overall culture of a company starts at the top. Get to know the senior leadership of the vendors you are evaluating. Insist on meeting several members of their executive team. Simply put, do the culture and values of a vendor’s senior leadership align with yours? When interacting with individual members of the vendor’s team, ask yourself, “Would I be excited to hire this person into our company?” If the answer is consistently “no,” find another vendor. More than technology is required to leverage clinical analytics to drive real, sustainable change; cultural transformation will be required throughout your organization. If your culture and values don’t mesh with those of your vendor, you will encounter significant roadblocks to success. 
  1. Ability to ExecuteDoes this vendor have a track record for delivering value and satisfaction to their clients? What do KLAS, Gartner, Chilmark, and the Advisory Board have to say about this vendor? Find at least three, preferably five, referenced accounts. Do not accept referenced accounts that are pre-screened and selected by the vendor—every client of the vendor should be open to serve as a reference. Do the vendors you are considering have solid referenced accounts that are similar in size and demographics to your own company? Ask these references very simply: How satisfied are you with the vendor’s products, services, and overall value? Would you hire them again?
  1. Technology, Adaptability, and SupportabilityThe reality is, in today’s connected world, all businesses, including healthcare, move at the speed their software can adapt (either fast or slow) to new processes and business models. Therefore, the underlying engineering and architecture of that software is critically important. You must peel back the covers of the vendor’s products and evaluate their software engineering for modern design patterns (such as object-oriented programming, service-oriented architectures, loose coupling, late binding, and balanced granularity of software services).Glossing over this assessment is akin to buying a multimillion dollar office building without assessing the modularity of the walls, soundness of the foundation, plumbing, and electrical systems. How fast can the system adapt to the market and your unique needs for differentiation? Data standards, vocabularies, and clinical analytics use cases are changing rapidly in the healthcare industry, literally every day, with no signs of slowing down. Find a vendor whose software engineering can keep up. Analytic agility is critical. Executives in your organization can’t wait weeks and months anymore for a new report to inform a critical decision. The industry is changing too fast.
  1. Total Cost of OwnershipThe best solution in the world is of no value if it’s not affordable. To assess affordability, you must understand the total cost of the vendor’s solution. Measuring total cost of ownership (TCO) is easy. You simply add up the three-year labor costs, licensing fees (including third-party), support fees, and hardware costs associated with a vendor’s solution. Many old-school analytics vendors require a significant upfront investment with no guarantee of value for two years or more. Your TCO over three years should be evenly distributed, not front-end loaded, and your contract should be structured with escape clauses if the vendor’s solution cannot prove value in the first year. In today’s market, clients should expect initial value from clinical analytics vendors in fewer than six months, and preferably only three months. If a vendor cannot or will not commit in their contract to this timeframe for delivering value, look for another vendor.
  1. Company ViabilityWill the vendor be around in nine years (the average life span of a significant IT investment)? If not, can you live without them? Take advantage of evaluations by neutral third-party analysts, such as Gartner, Chilmark, KLAS, and The Advisory Board. What are these analysts saying about the vendor’s prospects in the market? Is the vendor in solid financial shape? What’s their monthly burn rate versus income? How many days cash on hand do they maintain? What does their sales pipeline look like? Does the vendor’s executive leadership team have a track record for jumping from one company to another, or do they have a track record of longevity and success? How much is the vendor spending on sales staff in comparison to engineering and product development staff? The best products are supported by a very lean sales staff. That’s because great products sell themselves.

Technology and Cultural Change—Key Considerations for Clinical Analytics Success

Technology is vital to the success of an analytics initiative, but it is only one part of the solution. The meaningful use of analytics is one of the most difficult things for organizations to achieve, culturally. A successful clinical analytics implementation establishes the technology as well as the sustainable cultural changes required to turn the insights from data into improvements in patient care and reductions in cost. 


When evaluating a vendor’s technology, be sure to look at the following: 

Data Modeling and Analytic Logic

Different vendors’ analytics solutions feature different data models. Which data model they use can have a significant effect on the cost, scalability and, especially, adaptability of your clinical analytics solution to support new use cases. Rapidly adaptable and very flexible, a bus architecture is the best data-modeling option for healthcare. Most vendors utilize a healthcare-specific enterprise data model at the heart of their solution, but these enterprise data models are difficult to load and map initially, and slow to evolve subsequently—particularly when faced with new use cases and source system data content. These enterprise data models come in the following three basic flavors: dimensional star schema; enterprise first, second, and third normal form; and I2B2.

Over-modeling data is the single most significant contributor to data warehouse and analytics failure in healthcare. My advice is simple: Stop modeling your data and start relating it. Relating data is what analytics is all about.

In addition to the issue of data modeling, the analytic logic associated with the content of data marts and reporting is critically important. To learn more about the role of data modeling and “binding” data to business and clinical logic in healthcare analytics, read this white paper: The Late Binding Data Warehouse.

Master Reference/Master Data Management

The ability to incorporate data from new and disparate sources into your analytics solution requires significant expertise in master data management. Ask the following about vendor candidates:

  • What is the vendor’s strategy for managing unique patient and provider identifiers?
  • How does the vendor accommodate international, national, regional, and local master data types and naming conventions?
  • Do they support mappings to RxNorm, LOINC, SNOMED, ICD, CPT, and HCPCs?
  • How tightly does the vendor bind your data to the vocabularies that change regularly? (The tighter the binding, the less flexible the analytic design will be to accommodate changes in the vocabulary and analytic use cases based on those vocabularies.)

Metadata Repository

An effective metadata repository is the single most important tool for the widespread utilization and democratization of data in an organization. Look for a vendor that provides a tightly integrated, affordable, simple repository with their overall clinical analytics solution. The most valuable content in a metadata repository is not computable; the most valuable content is subjective data that comes from the data stewards and analysts who have interacted most with the data.

Look for vendors with the ability to maintain this subjective data through a wiki-style, wisdom-of-crowds contribution model. A web-searchable metadata repository should provide information such as the source of the data, how often it is updated, examples of the data, natural language descriptions of the physical data tables and columns, any known data quality issues, and the contact information for the associated data steward. The ability to quickly establish the origins and lineage of data in a data warehouse is also a critical component to an effective repository. Analytic vendors tend to operate in one of two extremes:

  1. They either oversell very complicated and expensive metadata repositories that require an overwhelming level of support and maintenance in return for a declining return on investment.
  2. They offer no solution for metadata management, which is disastrous to a long-term analytic strategy.

Find a vendor that offers a simple, low-cost, pragmatic solution between these extremes.

Managing “White Space” Data

Does your analytics solution offer a data collection alternative to the proliferation of desktop spreadsheets and databases that contain analytically important data? White space data is the data that is collected and stored in desktop spreadsheets and databases that it is not being collected and managed in primary source systems, especially EMRs, or is being collected in clinical notes and must be manually abstracted for reporting and analysis. This desktop data fills in the missing white space of analytic information that is important to the organization. For example, these desktop data sets are commonly found in support of Joint Commission reporting, internal KPIs, finance analytics, and clinical researchers.

It is not unusual for healthcare organizations to have hundreds of these desktop data sources that are critically important to the analytic success of the organization. However, because the data resides on desktop computers and shared drives, it cannot be integrated with other mission critical analytic data that is being stored in the Enterprise Data Warehouse (EDW) from the primary source systems. Data synergy suffers as a result.

White space data also poses information security risks. Analytics vendors must provide a tool for attracting the management of white space data into the content of the EDW. Look for a white space data management tool that is web based, as easy to use as a spreadsheet or desktop database for the collection of data, and makes it easy for end users to convert and upload their existing desktop data sets. Also, look for a security model in the EDW that allows for the isolation and stewardship of these white space data sets.

Visualization Layer

The best analytics solutions include a bundled visualization tool–one that is both affordable and extensible if licensed for the entire organization. However, the analytics visualization layer is very volatile. The leading visualization solution today will not be the leader tomorrow.

Therefore, look for an analytics vendor that can quickly and easily decouple the underlying data model and data content in the data warehouse from the visualization layer and swap the visualization tool with a better alternative when necessary. Also, realize that a single visualization tool will not solve all of your organization’s needs. Data analysts will want to use a variety of tools to access and manipulate data in the enterprise data warehouse. The underlying data models in the data warehouse must be capable of simultaneously supporting multiple visualization tools. Ask vendors if their data model is decoupled from the visualization tool. Does the data model support multiple visualization tools and delivery of data content?


As always in healthcare IT, the privacy and security of patient data is paramount. Ask these important questions of a potential clinical analytics vendor about security:

  • Are there fewer than 20 roles in the initial deployment? Contrary to popular belief, more roles can actually lead to lower security and will definitely lead to higher overhead administrative support costs.
  • Does the solution employ database-level security, visualization-layer security, or some combination of both? The vendor’s solution should support both.
  • What is the vendor’s model for protecting patient-identifiable (protected health information (PHI]) data and more sensitive subsets of PHI that are typically defined at the local state-level (i.e., mental health data, HIV data, and genomic/familial data)?
  • What type of tools and reports are available for managing security and auditing access to patient identifiable data?

Extract, Transform, Load (ETL)

A robust extract, transform, load (ETL) process—how analytics technology extracts data from source systems, applies the required transformations, and writes data into the target database—is fundamental to the success of your chosen solution. Ask vendors to demonstrate how their ETL measures up in terms of reliability, supportability, and reuse. At present, Microsoft’s ETL tool (SQL Server Integration Services [SSIS]) is by far the most cost-effective ETL tool in the market, offering the highest value per dollar.

Performance and Utilization Metrics

As you implement and continue to use an analytics solution, you will need to generate metrics about who is using the system, how are they using it, and how well the system operates. Can the vendor’s solution track basic data about the environment (i.e., user access patterns, query response times, data access patterns, volumes of data, and data objects? This kind of information will be essential as you refine and organize the data content and analytics services you provide from the data warehouse.

Hardware and Software Infrastructure

Does the vendor use Oracle, Microsoft, or IBM for its hardware and software infrastructure? These three are the only viable options in today’s healthcare market and data ecosystem. Hadoop and its associated open source tools is not an appropriate analytic and data warehousing infrastructure option at this time in healthcare (accept for gene sequencing).

Microsoft is the most integrated, affordable, and easiest to manage of these technology platforms. It now makes up 70 percent of all new sales in the clinical analytics and data warehousing market, across all industries, far outpacing Oracle, its closest competitor, in new sales. Rumors about Microsoft’s ability to scale up to large, multi-terabyte data warehouses are totally unfounded. Microsoft’s parallel data warehouse platform can scale to the petabyte level, far beyond the largest data warehouse needs in the healthcare provider space. Ask any data engineer who has worked extensively on either Microsoft or Oracle which platform they believe is the easiest to use and most efficient for delivering quick, adaptable clinical analytics solutions—the answer will almost certainly be Microsoft.

Cultural Change Management

As mentioned previously, technology is important, but it is only a part of the equation in creating a successful clinical analytics program. A vendor’s solution must also include processes and real-world experience for helping manage sustainable, analytics-driven change in your organization. Nothing is more politically or culturally disruptive than the spotlight of clinical analytics, not even the deployment of an EMR. You want a vendor that has been in the trenches of cultural transformation, driven by the enlightenment of data.

Your Clinical Analytics Roadmap: The Healthcare Analytics Adoption Model

Health Catalyst joined with other thought leaders in the analytics industry to lead the development of a Healthcare Analytics Adoption Model. The model outlines eight levels of analytics adoption an organization passes through as it gains sophistication in using data to drive improvement. Like a course curriculum for college studies, following this model with discipline will lead to the successful adoption of clinical analytics in your organization, both culturally and technically. Use this model for evaluating vendors’ capabilities in each level, and have the vendor demonstrate its products and services for each level.

The model also provides a roadmap for your organization to measure progress of clinical analytics adoption. Ask yourself, “How fast do we want to achieve the highest levels of adoption in this model?” With the right analytics vendor as a partner, organizations can achieve Level 5 within 18 months of implementing and following the model, and some organizations can make it in 12 months. Level 7 is easily achievable within 24 to 30 months.This model provides a standard to help you move beyond a patchwork of single-purpose, point solutions to a robust, data-driven health delivery system capable of tailoring care while optimizing efficiency and cost. Does a vendor’s solution support each level in the model? Ask them to prove it.

Choosing the Best Clinical Analytics Company

Healthcare is at the threshold of the next revolution in data management, in which organizations will be able to analyze and make informed decisions based on the data that they’ve been collecting and sharing. This is a critical time to set your organization on a path to data-driven improvement. The criteria outlined in this paper will help you choose the best clinical analytics partner with the expertise, processes, and technology to help you achieve this objective.

Powerpoint Slides

Would you like to use or share these concepts?  Download this Clinical Analytics presentation highlighting the key main points.

Click Here to Download the Slides 

Loading next article...