This workshop aims to contribute to the recent radical paradigm shift for fine-tuning in modern machine learning, both theoretically, computationally, and systematically. It encourages researchers to push forward the frontiers of theoretical understanding of fine-tuning, devising expeditious and resource-efficient inference and fine-tuning methods in machine learning systems, enabling their deployment within constrained computational resources.