Decision(Making,Models( Definition( action

1 decision Making, Models Definition Models of decision making attempt to describe, using stochastic differential equations which represent either neural activity or more abstract psychological variables, the dynamical process that produces a commitment to a single action /outcome as a result of incoming evidence that can be ambiguous as to the action it supports. Background decision making can be separated into four processes (Doya, 2008): 1) Acquisition of sensory information to determine the state of the environment and the organism within it.

2 2) Evaluation of potential actions (options) in terms of the cost and benefit to the organism given its belief about the current state. 3) Selection of an action based on, ideally, an optimal tradeoff between the costs and benefits. 4) Use of the outcome of the action to update the costs and benefits associated with it. Models of the dynamics of decision making have focused on perceptual decisions with only two possible responses available.

3 The term two- alternative forced choice (TAFC) applies to such tasks when two stimuli are provided, but the term is now generally used for any binary choice discrimination task. In a perceptual decision , the response, or action , is directly determined by the current percept. Thus the decision in these tasks is essentially one of perceptual categorization, namely process (1) above, though the same models can be used for action selection given ambiguous information of the current state (process 3).

4 Evaluation of the possible responses in terms of their value or the resulting state s utility (process 2) (Sugrue et al., 2005) given both uncertainty in the current state, and uncertainty in the outcomes of an action given the state, is the subject of expected utility theory and prospect theory. The necessary learning and updating of the values of different actions given the actual outcomes they produce (process 4) is the subject of instrumental conditioning and reinforcement learning, for example via temporal- difference learning (Seymour et al.)

5 , 2004) and actor- critic models (Joel et al., 2002). This article is primarily concerned with the dynamics of the production of either a single percept given unreliable sensory evidence (1), or a single action given uncertainty in the outcomes (3). General features of discrimination tasks or TAFC tasks. In a TAFC task, a single decision variable can be defined, representing the likelihood ratio the probability that evidence to date favors one alternative over the other.

6 While TAFC tasks (Figure 1) have provided the dominant paradigm for analysis of choice behavior, the restriction to only two choices is lifted in many of the more recent models of decision making based on multiple variables, allowing for the fitting of a wider range of data sets. The tasks can either be based on a free response paradigm, in which a subject responds after as much or little time as she wants, or an interrogation (forced response) paradigm, in which the stimulus duration is limited and the subject must make a response within a given time interval.

7 The free response paradigm is perhaps more powerful, since each trial produces two types of information: accuracy (correct or incorrect) and response time. However, by variation of the time allowed when responses are forced, both paradigms are valuable for constraining models, since they can provide a distribution of response times for both correct and incorrect trials, as well as the proportion of trials that are correct or incorrect with a given stimulus.

8 These behavioral data can be modified by task difficulty, task instructions, such as ( respond rapidly versus respond accurately ) or reward schedules and inter- trial intervals. Most models of the dynamics of decision making focus on tasks where the time from stimulus onset to response is no more than one to two seconds, a timescale over which neural spiking can be maintained. Choices requiring much more time than this are likely to depend upon multiple memory stores, neural circuits and strategies, which become difficult to identify, extract and model in a dynamical systems framework (a state- based framework is more appropriate).

9 Figure 1. Scheme of the two- alternative forced choice (TAFC) task. Two streams of sensory input, each containing stimulus information, or a signal (S1 and S2) combined with noise ( ! ! and ! ! ), are compared in a decision - making circuit. The circuit must produce one of two responses (A or B) indicating which of the two signals is the stronger. The optimal method for achieving this discrimination is via the sequential probability ratio test (SPRT) which requires the decision making circuit to integrate inputs over time.

10 In the standard setup of the models, two parallel streams of noisy sensory input are available, with each stream supplying evidence in support of one of the two allowed actions (see Figure 1). The sensory inputs can be of either discrete or continuous quantities and can arrive discretely or continuously in time. The majority of models focus on continuous update in continuous time so can be formulated as stochastic differential equations (Gillespie, 1992, Lawler, 2006).