Optimization Online


Multi-model Markov Decision Processes: A New Method for Mitigating Parameter Ambiguity

Lauren N. Steimle (steimle***at***umich.edu)
David L. Kaufman (davidlk***at***umich.edu)
Brian T. Denton (btdenton***at***umich.edu)

Abstract: Markov decision processes (MDPs) have found success in many areas including the evaluation and design of treatment and screening protocols for medical decision making problems. However, the usefulness of these models is only as good as the data used to parameterize them, and multiple competing data sources are common in medicine. In this paper, we introduce the Multi-model Markov Decision Process (MMDP) which generalizes a standard MDP by allowing for multiple models of the rewards and transition probabilities. Solution of the MMDP generates a single policy that maximizes the weighted performance over all models. This approach allows for the decision maker to explicitly trade off conflicting sources of data while at the same time relaxing strong assumptions made in the prior literature. We study the structural properties of this problem and show that this problem is at least NP-hard. We develop exact methods and fast approximation methods supported by error bounds for solving the weighted value problem. Finally, we illustrate the effectiveness and the scalability of our approach using a case study in preventative blood pressure and cholesterol management that accounts for conflicting published cardiovascular risk models.

Keywords: Robust dynamic programming; medical decision making; Markov decision processes; parameter ambiguity; healthcare applications

Category 1: Robust Optimization

Citation: Steimle, L. N., Kaufman, D.L., and Denton B.T. Multi-model Markov Decision Processes: A New Method for Mitigating Parameter Ambiguity. Optimization-online, Updated on January 25, 2018

Download: [PDF]

Entry Submitted: 01/25/2018
Entry Accepted: 01/25/2018
Entry Last Modified: 01/29/2018

Modify/Update this entry

  Visitors Authors More about us Links
  Subscribe, Unsubscribe
Digest Archive
Search, Browse the Repository


Coordinator's Board
Classification Scheme
Give us feedback
Optimization Journals, Sites, Societies
Mathematical Optimization Society