How to Evaluate a Clinical Analytics Vendor: A Checklist


understand the total cost of the vendor’s solution. Measuring Total Cost of Ownership (TCO) is easy. You simply add up the three-year labor costs, licensing fees (including third-party), support fees and hardware costs associated with a vendor’s solution. Many old-school analytics vendors require a significant upfront investment with no guarantee of value for two years or more.  Your TCO over three years should be evenly distributed, not front-end loaded, and your contract should be structured with escape clauses if the vendor’s solution cannot prove value in the first year. In today’s market, clients should expect initial value from clinical analytics vendors in less than six months, and preferably only three months. If a vendor cannot or will not commit in their contract to this timeframe for delivering value, look for another vendor.

6.              Company Viability

Will the vendor be around in nine years (the average life span of a significant IT investment)?  If not, can you live without them? Take advantage of evaluations by neutral third-party analysts like Gartner, Chilmark, KLAS, and The Advisory Board. What are these analysts saying about the vendor’s prospects in the market?  Is the vendor in solid financial shape?  What’s their monthly burn rate vs. income? How many days cash-on-hand do they maintain?  What does their sales pipeline look like? Does the vendor’s executive leadership team have a track record for jumping from one company to another or do they have a track record of longevity and success?  How much is the vendor spending on sales staff in comparison to engineering and product development staff?  The best products are supported by a very lean sales staff. That’s because great products sell themselves.

Technology and Cultural Change—Key Considerations for Clinical Analytics Success

Clinical Analytics in HealthcareTechnology is vital to the success of an analytics initiative, but it is only one part of the solution. The meaningful use of analytics is one of the most difficult things for organizations to achieve, culturally.  A successful clinical analytics implementation establishes the technology as well as the sustainable cultural changes required to turn the insights from data into improvements in patient care and reductions in cost.


When evaluating a vendor’s technology, be sure to look at the following:

Data Modeling and Analytic Logic

Different vendors’ analytics solutions feature different data models. Which data model they use can have a significant effect on the cost, scalability and – especially – the adaptability of your clinical analytics solution to support new use cases. Rapidly adaptable and very flexible, a bus architecture is the best data-modeling option for healthcare. Most vendors utilize a healthcare-specific enterprise data model at the heart of their solution, but these enterprise data models are difficult to load and map initially, and slow to evolve subsequently, particularly when faced with new use cases and source system data content. These enterprise data models come in the following three basic flavors, so be aware of them:

  1. Dimensional Star Schema
  2. Enterprise 1st, 2nd, 3rd normal form
  3. I2B2

Over-modeling data is the single most significant contributor to data warehouse and analytics failure in healthcare.  My advice is simple:  Stop modeling your data and start relating it.  Relating data is what analytics is all about.

In addition to the issue of data modeling, the analytic logic associated with the content of data marts and reporting is critically important.  To learn more about the role of data modeling and “binding” data to business and clinical logic in healthcare analytics, read this white paper: The Late Binding Data Warehouse.

Master Reference/Master Data Management

The ability to incorporate data from new and disparate sources into your analytics solution requires significant expertise in master data management. What is the vendor’s strategy for managing unique patient and provider identifiers? How does the vendor accommodate international, national, regional and local master data types and naming conventions? Do they support mappings to RxNorm, LOINC, SNOMED, ICD, CPT, and HCPCs?  How tightly does the vendor bind your data to the vocabularies that change regularly?  The tighter the binding, the less flexible the analytic design will be to accommodating changes in the vocabulary and analytic use cases based on those vocabularies.

Metadata Repository

An effective metadata repository is the single most important tool for the widespread utilization and democratization of data in an organization.  Look for a vendor that provides a tightly integrated, affordable, simple repository with their overall clinical analytics solution.  The most valuable content in a metadata repository is not computable – the most valuable content is subjective data that comes from the data stewards and analysts who have interacted most with the data. Look for vendors that have the ability to maintain this subjective data through a wiki-style, wisdom-of-crowds contribution model. A web-searchable metadata repository should provide information such as the source of the data, how often it is updated, examples of the data, natural language descriptions of the physical data tables and columns, any known data quality issues, and the contact information for the associated data steward. The ability to quickly establish the origins and lineage of data in a data warehouse is also a critical component to an effective repository.   Analytic vendors tend to operate in one of two extremes: (1) they either oversell very complicated and expensive metadata repositories that require an overwhelming level of support and maintenance in return for a declining return on investment; or (2) they offer no solution for metadata management, which is disastrous to a long-term analytic strategy.  Find a vendor that offers a simple, low-cost, pragmatic solution between these extremes.

Managing “White Space” Data

Does your analytics solution offer a data collection alternative to the proliferation of desktop spreadsheets and databases that contain analytically important data?

White space data is the data that is collected and stored in desktop spreadsheets and databases that it is not being collected and managed in primary source systems, especially EMRs, or it is being collected in clinical notes and must be manually abstracted for reporting and analysis.  This desktop data fills in the missing “white space” of analytic information that is important to the organization.  For example, these desktop data sets are commonly found in support of Joint Commission reporting, internal KPIs, finance analytics, and clinical researchers.  It is not unusual for healthcare organizations to have hundreds of these desktop data sources that are critically important to the analytic success of the organization.  However, because the data resides on desktop computers and shared drives, it cannot be integrated with other mission critical analytic data that is being stored in the Enterprise Data Warehouse from the primary source systems.  Data synergy suffers as a result.  White space data also poses information security risks.  Analytics vendors must provide a tool for attracting the management of white space data into the content of the EDW.  Look for a white space data management tool that is web-based, as easy to use as a spreadsheet or desktop database for the collection of data, and makes is easy for end users to convert and upload their existing desktop data sets.  Also, look for a security model in the EDW that allows for the isolation and stewardship of these white space data sets.

Visualization Layer

cauti-visualizationThe best analytics solutions include a bundled visualization tool – one that is both affordable and extensible if licensed for the entire organization.  However, the analytics visualization layer is very volatile.  The leading visualization solution today will not be the leader tomorrow.  Therefore, look for an analytics vendor that can quickly and easily decouple the underlying data model and data content in the data warehouse from the visualization layer and swap the visualization tool with a better alternative when necessary. Also realize that a single visualization tool will not solve all of your organization’s needs. Data analysts will want to use a variety of tools to access and manipulate data in the enterprise data warehouse.  The underlying data models in the data warehouse must be capable of supporting multiple visualization tools at the same time. Ask vendors if their data model is decoupled from the visualization tool. Does the data model support multiple visualization tools and delivery of data content?


As always in healthcare IT, the privacy and security of patient data is paramount. Ask these important questions of a potential clinical analytics vendor about security:

  • Are there fewer than 20 roles in the initial deployment?  Contrary to popular belief, more roles can actually lead to lower security and will definitely lead to higher overhead administrative support costs.
  • Does the solution employ database-level security, visualization-layer security or some combination of both?  The vendor’s solution should support both.
  • What is the vendor’s model for protecting patient-identifiable (protected health information (PHI)) data and the more sensitive subsets of PHI that are typically defined at the local state-level, such as mental health data, HIV data, and genomic/familial data?
  • What type of tools and reports are available for managing security and auditing access to patient identifiable data?


A robust ETL process – how analytics technology extracts data from source systems, applies the required transformations and writes data into the target database – is fundamental to the success of your chosen solution. Ask vendors to demonstrate how their ETL measures up in terms of reliability, supportability and reuse.  At present, Microsoft’s ETL tool – SQL Server Integration Services (SSIS) – is by far the most cost-effective ETL tool in the market, offering the highest value per dollar.

Performance and Utilization Metrics

As you implement and continue to use an analytics solution, you will need to generate metrics about who is using the system, how are they using it, and how well the system operates. Can the vendors’ solution track basic data about the environment, such as user access patterns, query response times, data access patterns, volumes of data and data objects? This kind of information will be essential to you as you refine and organize the data content and analytics services you provide from the data warehouse.

Hardware and Software Infrastructure

Does the vendor use Oracle, Microsoft or IBM for its hardware and software infrastructure? These three are the only viable options in today’s healthcare market and data ecosystem.  Hadoop and its

Loading next article...