Build vs. Buy a Healthcare Enterprise Data Warehouse: Which is Best for You?
Chances are, if you are reading this blog, you have heard some flavor of the “build vs. buy” question in the context of data warehousing. For example, here are two conflicting ways that I’ve personally heard this question posed:
- “Do we need to buy [a data warehouse], or can we build it?”
- “Are there any vendors we can buy this from, or will we have to build this?”
As you can imagine, both approaches resonate differently with different people, cultures, and strategies, and the same basic questions sound very different depending on who is asking it.
When we consider a simplified version of the software development life cycle (see figure below), an organization will likely assess their current state, and ask “do we have the tools to use data to help transform care delivery and streamline operations?” If this organization finds they lack that capability—if it’s harder than expected to be able to use the data in the medical record systems, patient experience systems, surveys, and financial-decisions support systems—they may decide to move to the next step and start the design process. Of course, not all processes are going to be as formalized as shown in the figure, but, like the organization in this example, you might end up going through these stages as you think about what your organization’s future state should look like. Between the design and implementation stages, you’ll select an approach. For the purposes of this paper, we’re going to assume you’ve selected an enterprise data warehouse or a data warehousing approach to integrating your data and creating that single source of truth from your disparate systems. You’re likely interested in facilitating Triple Aim reporting, and you’ll proceed either to implement a vendor solution or to build from scratch a data warehouse.
The topic for this paper is really that inflection point/decision point between the design and implement phases. Where you and your organization evaluate the options and ask: should we try to build it from scratch? Should we look toward a vendor to partner with us and buy that solution? Are those our only options? Is there a third option where we can do a little bit of both?
Ultimately, the decision is up to you. We want to provide some of the things that can help factor into your decision-making process.
Historically, Why Were Healthcare Data Warehouses Built, Not Bought?
To give some context to this question, it may help to draw a parallel with the evolution of the EMR. Early adopters of EMRs had to build their own, as there were few commercial vendor options. Then, the first generation of rudimentary commercial EMRs came along. Companies like Siemens, Cerner, and Epic evolved early concepts into second and third generation systems, prompted by the EMR adoption model from HIMSS that gave vendors and customers a benchmark—a roadmap—for product development and acquisition.
Similarly, until recently no single vendor offered an enterprise data warehouse (EDW) solution for healthcare that could deliver quantifiable results and return on investment. Innovative organizations with the resources to support an extensive internal development effort really had only one potential path: build it themselves.
Health systems without the staff, budget, or experience to build a centralized EDW themselves were left with two main options:
- “Data analyst heroism” – where a small number of savvy analysts used whatever reporting or analysis tools they had at their disposal. Great people in these roles achieved excellent results, but their potential value was often underutilized because they needed to spend too much time extracting data instead of analyzing it.
- Implemented best-of-breed analytics solutions to help address specific, siloed, reporting, and analytic needs.
The Case for Building a Healthcare Data Warehouse
As of today, the healthcare analytics market has matured to the point where there is now a compelling buy-option for companies who aren’t interested or able to develop an EDW internally.
But the reality is, there will always exist a small number of healthcare organizations that prefer to leverage internal software developers wherever and whenever possible. These organizations may perceive functionality shortcomings in the commercial offerings and have some great ideas about how they could close those gaps through internal software development.
For some organizations, however, the Build vs. Buy question can become a heated, political battle; dividing parts of the organization (typically IT vs. everyone else) in unhealthy ways, often leading to byzantine system-selection processes, where the internal staff are asked to represent themselves as a vendor, and present themselves as being in competition with external offerings.
But it doesn’t have to be this way. Let’s look a little more closely at some of the pros and cons of building an enterprise data warehouse from scratch:
Pros of Building Your Own EDW:
- The possibility of a “perfect fit” – custom development, in the hands of skilled software engineers, can yield excellent, tailored solutions.
- Potentially lower initial cost – organizations with a few cycles to spare can often get “something” in the way of healthcare analytics stood up very rapidly. In a small number of very specific cases, “something” is often “good enough for now.”
- Pride of ownership – several healthcare organizations, including members of the Healthcare Data Warehousing Association have built successful data warehousing programs from scratch, starting in the late 1990’s. Many of those organizations are now among a short list of organizations that are well-positioned to address the needs of healthcare reform. Viewed by other systems nationwide as now possessing a vital knowledge asset, they are justifiably proud of their accomplishments, vision, and forethought. However, several of those early pioneers now realize that their home-grown solutions are neither sustainable nor adaptable to new analytic use cases in the industry… and the perceived value of these first generation EDWs is in decline.
Cons of Building Your Own EDW:
- Staffing: plan to ramp up significantly, or at the very least, be prepared to dedicate existing software engineering professionals to the data warehouse build for many months if not years. Internal data warehouse projects are notoriously under-scoped, under-resourced, and delivered much later than planned. Only a small number of healthcare organizations with very agile IT governance can make this happen smoothly.
- There is an enormous shortage of experienced data and software engineers in healthcare—in fact, across all industries. There simply aren’t enough skilled people available in the labor market to sufficiently lower the risk of building your own EDW. Across all industries, 60% of internally developed EDW projects fail to meet expectations. The number will be higher in healthcare if we make the mistake of believing we can build and sustain these systems with internal resources alone.
- From a technical perspective, one of the most significant “cons” to building your own EDW is you will be learning lessons (i.e. making mistakes) that have previously been addressed by an experienced vendor. Without a starting point, including coding standards, naming conventions, and a proven approach to data architecture, your developers will soon find themselves refactoring or re-engineering key parts of their data warehousing solution.
- Many of these projects often start “under the radar” and in response to a lack of agility in IT to meet the reporting and analytic needs of the organization. As such, their success can be culturally divisive, and create silos of data that need to eventually be re-incorporated into an enterprise-wise strategy for data management.
- IT teams accustomed to strict waterfall project management approaches may also lack the agility necessary to adapt to rapidly changing vocabularies, standards, and new healthcare analytics use cases. Inability to deliver continuing value can lead to project delays, unmet expectations, and overall frustration with the costs that have gone in to an internal development effort.
The Case for Buying a Healthcare Data Warehouse
On the other hand, buying a data warehouse solution for healthcare is now a viable option. For many organizations without a deep bench of software developers, the possibility of rapidly implementing a