Multi-model Markov Decision Processes
Lauren N. Steimle (steimleumich.edu)
Abstract: Markov decision processes (MDPs) have found success in many application areas that involve sequential decision making under uncertainty, including the evaluation and design of treatment and screening protocols for medical decision making. However, the usefulness of these models is only as good as the data used to parameterize them, and multiple competing data sources are common in many application areas, including medicine. In this article, we introduce the Multi-model MDP (MMDP) which generalizes a standard MDP by allowing for multiple models of the rewards and transition probabilities. Solution of the MMDP generates a single policy that maximizes the weighted performance over all models. This approach allows for the decision maker to explicitly trade-off conflicting sources of data while generating a policy of the same level of complexity for models that only consider a single source of data. We study the structural properties of this problem and show that this problem is at least NP-hard. We develop exact methods and fast approximation methods supported by error bounds. Finally, we illustrate the effectiveness and the scalability of our approach using a case study in preventative blood pressure and cholesterol management that accounts for conflicting published cardiovascular risk models.
Keywords: Robust dynamic programming; medical decision making; Markov decision processes; parameter ambiguity; healthcare applications
Category 1: Robust Optimization
Citation: Steimle, L. N., Kaufman, D.L., and Denton B.T. Multi-model Markov Decision Processes. Optimization-online, Updated on July 27, 2018.
Entry Submitted: 01/25/2018
Modify/Update this entry
|Visitors||Authors||More about us||Links|
Search, Browse the Repository
Give us feedback
|Optimization Journals, Sites, Societies|