., reach-
ing with an arm to various target points. After suitable training of these
readouts on a small number of target points; reaching movements to
nearby points can also be generated. Sensory or proprioceptive feed-
back turns out to improve the performance of the neural microcircuit
model, if it arrives with a signicant delay of 25 to 100 ms. Further-
more, additional feedbacks of prediction of sensory variables are shown
to improve the performance signicantly. Existing control methods in
robotics that take the particular dynamics of sensors and actuators into
account(embodiment of robot control) are taken one step further with
this approach which provides methods for also using the embodiment of
computation, i.e. the inherent dynamics and spatial structure of neural
circuits, for the design of robot movement controllers.
1
Introduction
This article demonstrates that simple linear readouts from generic neural mi-
crocircuit models consisting of spiking neurons and dynamic synapses can be
trained to generate and control rather complex movements. Using biologically
realistic neural circuit models to generate and control movements is not so easy,
since these models are made of spiking neurons and dynamic synapses which
exhibit a rich inherent dynamics on several temporal scales. This tends to be in
conict with movement control tasks that require focusing on a relatively slow
time scale.
Preceding work on movement control, has drawn attention to the need of
taking the embodiment of motor systems, i.e. the inherent dynamics of sensors
and actuators into account. This approach is taken one step further in this
article, as it provides a method for also taking into account the embodiment
of neural computation, i.e. the inherent dynamics and spatial arrangement of
neural circuits that control the movements. Hence it may be seen as a rst step in
The work was partially supported by the Austrian Science Fond FWF, project #
P15386.
A.J. Ijspeert et al. (Eds.): BioADIT 2004, LNCS 3141, pp. 258273, 2004.
c Springer-Verlag Berlin Heidelberg 2004 Movement Generation and Control
259
a long range program where abstract control principles for biological movement
control and related models developed for articial neural networks [Tani, 2003]
can be implemented and tested on arbitrarily realistic models for the underlying
neural circuitry.
The feasibility of our approach is demonstrated in this article by showing
that simple linear readouts from a generic neural microcircuit model can be
trained to control a 2-joint robot arm, which is a common benchmark task
for testing methods for nonlinear control [Slotine and Li, 1991]. It turns out
that both the spatial organization of information streams, especially the spatial
encoding of slowly varying input variables, and the inherent dynamics of the
generic neural microcircuit model have a signicant impact on its capability
to control movements. In particular it is shown that the inherent dynamics of
neural microcircuits allows these circuits to cope with rather large delays for
proprioceptive and sensory feedback. In fact it turns out that their performance
is optimal for delays that lie in the range of 25 to 100 ms. Additionally it is
shown that the generic neural microcircuit models used by us, possess signicant
amount of temporal integration capabilities. It is also demonstrated that this new
paradigm of motor control provides generalization capabilities to the readouts.
Furthermore, it is shown that the same neural microcircuit model can be trained
simultaneously to predict the results of such feedbacks, and by using the results
of these predicted feedbacks it can improve its performance signicantly in cases
where feedback arrives with other delays, or not at all.
This work complements preceding work where generic neural microcir-
cuit models were used in an open loop for a variety of simulated sen-
sory processing tasks ([Buonomano and Merzenich, 1995], [Maass et al., 2002],
[Maass et al., 2003]). It turns out that the demands on the precision of real-
time computations carried out by such circuit models are substantially higher
for closed-loop applications such as those considered in this article. Somewhat
similar paradigms for neural control based on articial neural network models
have been independently explored by Herbert Jaeger [J¨
ager, 2002].
The neural microcircuit model and the control tasks considered in this article
are specied in the subsequent two sections. Results of computer simulations are
presented in sections 4, 5, 6 and relations to theoretical results are discussed in
section 7.
2
Generic Neural Microcircuit Models
In contrast to common articial neural network models, neural microcircuits
in biological organisms consist of diverse components such as dierent types of
spiking neurons and dynamic synapses, that are each endowed with an inher-
ently complex dynamics of its own. This makes it dicult to construct out of
biologically realistic computational units, implementations of boolean or analog
circuits that have turned out to be useful in the context of computer science or
articial neural networks. On the other hand, it opens the path towards alterna-
tive computational paradigms based on emergent computations in sparsely and 260
P. Joshi and W. Maass
recurrently connected neural microcircuits, composed of diverse dynamic com-
ponents [Maass et al., 2002]. In [Maass et al., 2002] a new computational model,
the liquid state machine, has been proposed that can be used to explain and an-
alyze the capabilities of such neural microcircuits for real-time computing. Con-
sequences of this analysis for applications to closed loop control are discussed
in section 7 of this article. Instead of constructing circuits for specic tasks,
one considers here various probability distributions for neural connectivity. Such
circuits have inherent capabilities for temporal integration of information from
several segments of incoming input streams, and relevant computations that re-
combine pieces of this information in a nonlinear manner emerge automatically.
To be exact, the information about the outputs of a very large class of compu-
tations on information contained in the input stream is automatically present in
the liquid state of the dynamical system in the sense of [Maass et al., 2002],
see [Natschl¨
ager and Maass, 2004].
The liquid state x(
t) models that part of the current circuit state that is
in principle visible to a readout neuron (see Fig. 1, a) that receives synaptic
inputs from all neurons in the circuit. Each component of x(
t) models the impact
that a particular neuron
v may have on the membrane potential of a generic
readout neuron (see Fig. 2). Thus each spike of neuron
v is replaced by a pulse
whose amplitude decays exponentially with a time constant of 30 ms. In other
words: x(
t) is obtained by applying a low-pass lter to the spike trains emitted
by the neurons in the generic neural microcircuit model. We will only consider
information that can be extracted from the liquid state x(
t) of the generic neural
microcircuit model by a simple weighted sum
1
w
× x(t). The weight vector w
will be xed for all
t and all circuit inputs once the training of the (symbolic)
readout neurons has been completed.
In principle one can of course also view various parameters within the circuit
as being subject to learning or adaptation, for example in order to optimize the
dynamics of the circuit for a particular range of control tasks. However this has
turned out to be not necessary for the applications described in this article. One
advantage of just viewing the weight vector w as being plastic is that learning
is quite simple and robust, since it just amounts to linear regression in spite of
the highly nonlinear nature of the control tasks to which this set-up is applied.
Another advantage is that the same neural microcircuit could potentially be
used for various other information processing tasks (e.g. prediction of sensory
feedback, see section 6) that may be desirable for the same or other tasks.
The generic microcircuit models used for the closed loop control tasks de-
scribed in this article were similar in structure to those that were earlier used
for various sensory processing tasks. More precisely, we considered circuits con-
sisting of 600 leaky-integrate-and-re neurons arranged on the grid points of a
20
× 5 × 6 cube in 3D (see Fig. 1, b). 20 % of these neurons were randomly cho-
sen to be inhibitory. Synaptic connections were chosen according to a probability
1
One constant component is added to x(
t) to facilitate the implementation of a con-
stant bias in terms of the form w
× x(t). Movement Generation and Control
261
Fig. 1. a) Information ow diagram for a neural microcircuit model applied to a control
task. The plant may be a motor system or some part of the environment. b) Spatial
layout of neurons in the models considered in this article. The 6 layers on the left hand
side are used for spatial coding of inputs to the circuit (
x
dest
, y
dest
,
1
(
t ),
2
(
t
),
1
(
t),
2
(
t)). Connections between these 6 input layers, as well as between neurons
in the subsequent 6 processing layers are chosen randomly according to a probability
distribution discussed in the text. c) Standard model of a 2-joint robot arm d) Initial
position A and end position B of the robot arm for one of the movements. The target
trajectory of the tip of the robot arm and of the elbow are indicated by dashed lines.
Fig. 2. Snapshots of the liquid state of a neural microcircuit model at 3 dierent ti