When modeling atmospheric chemistry, concentrations are determined by numerically solving large systems of ordinary differential equations that represent a set of chemical reactions. These solvers can be very computationally intensive, particularly those with the thousands or tens of thousands of chemical species and reactions that make up the most accurate models. We demonstrate the application of a deep learning transformer architecture to emulate an atmospheric chemistry box model, and show that this attention-based model outperforms LSTM and autoencoder baselines while providing interpretable predictions that are more than 2 orders of magnitude faster than a numerical solver. This work is part of a larger study to replace the numerical solver in a 3D global chemical model with a machine learned emulator and achieve significant speedups for global climate simulations.