Diffusion models, rivaling GANs for high-quality sample generation, leverage score matching to learn the score function. Despite empirical success, the provable accuracy of gradient-based algorithms for this task remains unclear. As a first step toward answering this question, this paper establishes a mathematical framework for analyzing score estimation using neural networks trained by gradient descent. The analysis covers both the optimization and the generalization aspects of the training procedure. We propose a parametric form to formulate the denoising score-matching problem as a regression with noisy labels. Compared to standard supervised learning, the score-matching problem introduces distinct challenges, including unbounded inputs, vector-valued outputs, and an additional time variable, preventing existing techniques from being applied directly. With a properly designed neural network architecture, we demonstrate an accurate approximation of the score function using a reproducing kernel Hilbert space induced by neural tangent kernels. We establish the first generalization error bound for learning the score function by applying early stopping and coupling arguments.