As hospitals and health systems continue exploring AI opportunities, many health IT leaders face a common challenge: determining where to invest, what to prioritize, and how to separate meaningful opportunities from unnecessary risk.
During a recent webinar hosted by Harmony Healthcare IT, Benjamin Cassity, Director of Research & Strategy at KLAS Research; Bill Russell, Founder and CEO of This Week Health and The 229 Project; and Jim Hammer, PMP, FACHDM, Chief Operating Officer of Harmony Healthcare IT, shared guidance for healthcare organizations looking to make more strategic AI investment decisions.
1. Begin With Self-Assessment
Before evaluating vendors or use cases, Hammer recommends taking stock of your current data environment. Start by inventorying your applications, identifying where critical information resides, and assessing how legacy data is accessed, governed, and managed across the organization.
Organizations should also consider consolidating legacy data into a secure, centralized archive. This can reduce fragmentation, preserve longitudinal records, and improve access to historical data that often powers advanced analytics, operational improvement, and AI initiatives.
“For heavy-duty use cases around AI, you want the longitudinal record,” Hammer said. “Legacy data should absolutely be at the table when exploring AI.”
2. Define Your Objectives Before Evaluating Solutions
Before meeting with vendors, Cassity recommends defining the operational, clinical, and financial objectives you hope to achieve. Organizations should have a clear understanding of the problem they are trying to solve, the data available to support it, who owns that data, and how success will ultimately be measured.
Often, healthcare organizations start with a technology search before defining the business need, he said. “We have countless calls from providers who get on the phone and tell us, ‘I just got some money. My CFO told me I need to use it in AI. What should I buy?'”
Establishing goals and expected outcomes upfront can help leaders evaluate opportunities more strategically and avoid pursuing solutions without a clearly defined use case.
3. Start Small and Learn as You Go
Rather than pursuing enterprise-wide transformation from day one, Russell, former CIO of St. Joseph Health, recommends focusing on a specific use case that can deliver meaningful business value. A narrower scope makes it easier to establish governance, measure outcomes, incorporate lessons learned, and refine workflows before expanding to broader initiatives.
“I wouldn’t try to boil the ocean,” Russell said. “Most of the most successful organizations are narrowing the scope. They learn as they go and they put the right resources and workflows behind it.”
4. Prioritize Measurable ROI
When evaluating opportunities, Russell encourages healthcare leaders to focus on initiatives that can produce measurable outcomes, such as cost reductions, avoided expenses, increased productivity, or the replacement of existing investments.
While time savings can be valuable, they do not always translate into meaningful financial impact. Demonstrating tangible results early can help organizations build credibility, secure stakeholder buy-in, and create momentum for future initiatives. As Russell put it, “Trust begets trust.”
5. Ask Sharper Questions
When evaluating vendors, Cassity recommends looking beyond product demonstrations and marketing claims. Instead, focus on questions that help assess a vendor’s maturity, reliability, and ability to deliver results in real-world healthcare environments.
During evaluations, questions such as, “Where does your model get its data?”, “How do you respond when things break?”, and “Can you show outcomes achieved by organizations similar to ours?” can provide valuable insight into how a solution will perform after implementation.
“If a vendor partner can only tell you what the software can do and not what life looks like with it built into the workflow, you need to keep asking more questions,” Cassity said.
6. Understand the Full Cost Structure
Before investing, Hammer recommends developing a clear understanding of the total cost of ownership. In addition to software licensing, organizations should consider model usage fees, token consumption, infrastructure requirements, and ongoing operational expenses.
Without appropriate oversight, costs can escalate quickly as utilization increases. Evaluating both expected benefits and long-term operating costs early can help organizations make more informed investment decisions and avoid budget surprises later.
“Definitely make sure as you’re looking at solutions that you understand what those billing structures are, what limits exist, and what controls you have in place,” Hammer said.
Final Takeaways
The panelists agreed that successful AI initiatives rarely begin with the technology itself. Instead, they start with a clear understanding of organizational goals, trusted data, realistic expectations, and a willingness to learn along the way.
Whether the objective is improving operational efficiency, reducing costs, strengthening clinical workflows, or enhancing the patient experience, organizations are more likely to succeed when they take a disciplined approach to decision-making. That means understanding the data they already have, identifying specific problems worth solving, measuring outcomes rigorously, and scaling only after demonstrating value.
As Russell noted, the most successful organizations are “narrowing the scope” and learning as they go. In a healthcare environment filled with new AI capabilities and evolving vendor offerings, that practical mindset may be one of the most valuable strategies of all.
This is the final article in a series drawn from a recent Harmony Healthcare IT webinar on AI-data readiness. Read the other articles here:
Industry Perspective: AI Investment Is Increasing
The pressure to make smart AI investment decisions is growing. According to the 229 Project’s “Signal Index Spring 2026 Report,” based on a survey of 146 healthcare IT leaders, more than half (60%) reported growing budgets for AI initiatives.
As investment increases, healthcare leaders face greater pressure to prioritize the right opportunities, demonstrate measurable value, and avoid unnecessary complexity. The panelists’ recommendations reflect that reality: start with the data, define the problem, focus on measurable outcomes, and scale deliberately.
