While the role of model architecture and hardware system on training performance is well-understood and appreciated, the role of data quality and quantity is often overlooked. In this talk, I will highlight the performance implications of data quality, particularly on speed of training and scaling efficiency, and argue that it is time to shift our paradigm from HW/SW co-design to HW/SW/Data tri-design.