Optimization Online


Data-Driven Ranges of Near-Optimal Actions for Finite Markov Decision Processes

Wesley J. Marrero(wmarrerocolon***at***mgh.harvard.edu)
Mariel S. Lavieri(lavieri***at***umich.edu)
Ambuj Tewari(tewaria***at***umich.edu)
Jeremy B. Sussman (jeremysu***at***med.umich.edu)
Rodney A. Hayward(rhayward***at***med.umich.edu)

Abstract: Markov decision process (MDP) models have been used to obtain non-stationary optimal decision rules in various applications, such as treatment planning in medical decision making. However, in practice, decision makers may prefer other strategies that are not statistically different from the optimal decision rules. To benefit from the decision makersí expertise and provide flexibility in implementing decision strategies, we introduce a new framework for identifying sets of near-optimal actions for finite MDP models. We present a simulation-based dynamic programming algorithm that can be executed using parallel computing and show that it converges to the optimal solutions exponentially fast under fairly mild conditions. The sets of near-optimal actions are modeled as nonparametric simultaneous confidence intervals on the difference between an approximately optimal action and the remaining alternatives. By analyzing the structure of the sets, we characterize their behavior with respect to the modeling data and identify when they can be ordered as a range. Lastly, we show the scalability of our approach by finding ranges of near-optimal antihypertensive treatment choices for 16.72 million adults in the US.

Keywords: Markov decision processes, simulation, statistical multiple comparisons, medical decision making, health policy, cardiovascular diseases

Category 1: Applications -- OR and Management Sciences

Category 2: Other Topics (Dynamic Programming )

Category 3: Applications -- Science and Engineering

Citation: Institution Address: MGH Institute for Technology Assessment, Harvard Medical School, Boston, MA 02114 Month/Year: June/2021

Download: [PDF]

Entry Submitted: 10/05/2021
Entry Accepted: 10/05/2021
Entry Last Modified: 10/05/2021

Modify/Update this entry

  Visitors Authors More about us Links
  Subscribe, Unsubscribe
Digest Archive
Search, Browse the Repository


Coordinator's Board
Classification Scheme
Give us feedback
Optimization Journals, Sites, Societies
Mathematical Optimization Society