Stability in recurrent neural models poses a significant challenge, particularly in developing biologically plausible neurodynamical models that can be seamlessly fit to experimental data. Traditional cortical circuit models are notoriously difficult to train due to expansive nonlinearities in the dynamical system, leading to an optimization problem with nonlinear constraints. These constraints are typically manageable only for small-dimensional systems, but as we scale to higher-dimensional models, stability analysis becomes increasingly difficult, often requiring restrictive mean-field approaches. Moreover, few neurodynamical models implement divisive normalization, and those that do either approximate it or execute it exactly for only specific parameter regimes. Conversely, recurrent neural networks (RNNs) excel in tasks involving sequential data but lack biological plausibility and interpretability. In this work, we bridge these gaps by considering ``oscillatory recurrent gated neural integrator circuits'' (ORGaNICs), a biologically plausible recurrent cortical circuit that implements divisive normalization exactly and has been shown to simulate key neurophysiological phenomena. We prove the remarkable property of unconditional local stability for an arbitrary-dimensional ORGaNICs circuit when the recurrent weight matrix is the identity, by using the indirect method of Lyapunov. Further, for a generic recurrent weight matrix, we prove the stability of the 2D model and demonstrate stability empirically for higher dimensional circuits and useful generalizations. Finally, we evaluate the model's performance on benchmarks for RNNs trained on static and dynamic (sequential) classification tasks. Our results show that ORGaNICs trained by simple backpropagation through time (without gradient clipping/scaling and without specialized training strategies) outperforms alternative neurodynamical models on static data, and it performs comparably to LSTMs, thanks to its intrinsic stability property, which effectively addresses the challenges of learning long-term dependencies in RNNs by mitigating the problem of vanishing and exploding gradients.