The use of guidance in diffusion models was originally motivated by the fact that the guidance-modified score corresponds to the data density scaled by the conditional likelihood of a label or prompt raised to a power (a tilted distribution). To the contrary, we show that guidance applied even with the exact score function does not sample from the intended tilted distribution. We establish a fine-grained characterization of the dynamics of the probability flow ODE in two cases: (1) mixtures of two products of linearly separable, compactly supported distributions and (2) mixtures of two Gaussians. In both cases, a corollary of our characterization is that increasing the guidance parameter leads to sampling towards the edges of the supports of the class-conditional distributions. In addition to verifying these results empirically in synthetic settings, we also show how our theoretical insights can be applied to practice.