We propose a novel spectral tensor layer for model-parallel deep neural networks, which can be trainedin the spectrum domain. Spectral tensor neu-ral network is trained by training a number of small,weak subnetworks which can be trained in asyn-chronous parallel. The output of the spectral tensorlayer neural network is the ensemble of the subnet-works’ outputs. Compared with conventional neuralnetworks, spectral tensor neural network has intrin-sical data parallelism as well as model parallelism,which is very suitable for distributed training, and hascompetitive ensemble results.