Reward-modulated spike-timing-dependent plasticity (STDP) has recently emerged as a candidate for a learning rule that could explain how local learning rules at single synapses support adaptive changes in complex networks of spiking neurons. However the potential and limitations of this learning rule could so far only been tested through computer simulations. This article provides tools for an analytic treatment of reward-modulated STDP, which allow us to derive concrete conditions under which the convergence of reward-modulated STDP can be predicted. In particular, we can produce in this way a theoretical explanation and a computer model for a fundamental experimental finding on reinforcement learning in monkeys by Fetz and Baker. We also report results of computer simulations that have tested further predictions of this theory.