A pivotal brain computation relies on the ability to sustain perception-action loops. Stochastic optimal control theory offers a mathematical framework to explain this process at the algorithmic level through optimality principles. However, incorporating a realistic noise model of the sensorimotor system — accounting for multiplicative noise in feedback and motor output, as well as internal noise in estimation — makes the problem challenging. Currently, the algorithm that is commonly used is the one proposed in (Todorov, 2005). After discovering some pitfalls in the original derivation, i.e., unbiased estimation does not hold in the presence of internal noise, we improve the algorithm by proposing an efficient gradient descent-based optimization that minimizes the cost-to-go while only imposing linearity of the control law. The optimal solution is obtained by iteratively propagating in closed form the sufficient statistics to compute the expected cost and then minimizing this cost with respect to the filter and control gains. We show that this approach features an overall lower cost than current state-of-the-art solutions when internal noise is present. Providing the optimal control law in these cases is crucial for problems such as inverse control inference under rationality assumptions.