Solar irradiance forecasting is essential for the integration of the solar power into the power grid system while maintaining its stability. This paper focuses on short-term solar irradiance forecasting (upto 4-hour ahead-of-time prediction) from a past sky image sequence. Most existing work aims for the prediction of the most likely future of the solar irradiance. While it is likely deterministic for intra-hourly prediction, the future solar irradiance is naturally diverse over a relatively long-term horizon (>1h). We therefore introduce approaches to deterministic and stochastic predictions to capture the most likely as well as the diverse future of the solar irradiance. To enable the autoregressive prediction capability of the model, we proposed deep neural networks to predict the future sky images in a deterministic as well as stochastic manner. We evaluate our approaches on benchmark datasets and demonstrate that our approaches achieve superior performance.