Disease grading is a crucial task in medical image analysis, but due to the continuous progression of diseases, the variability within the same level and the similarity between adjacent stages make accurate grading highly challenging.Furthermore, in real-world scenarios, models trained on limited source domain datasets should also be capable of handling data from unseen target domains.Due to the cross-domain variants, the feature distribution between source and unseen target domains can be dramatically different, leading to a substantial decrease in model performance.To address these challenges in cross-domain disease grading, we propose a Severity-aware Recurrent Modeling (Samba) in this paper.As the core objective of most staging tasks is to identify the most severe lesions, which may only occupy a small portion of the image, we propose to encode image patches in a sequential and recurrent manner.Specifically, state space model is tailored to store and transport the severity information by hidden states.Moreover, to mitigate the impact of cross-domain variants, an EM-based state recalibration method is designed to map the patch embeddings into a more compact space.We model the feature distributions of different lesions through Gaussian Mixture Model (GMM) and reconstruct the intermediate features based on learnable severity bases.Extensive experiments show the cross-domain grading ability of the proposed method across three medical modalities. Source code will be made publicly available.