Point process encoding models provide powerful statistical methods for understanding the responses of neurons to sensory stimuli. Although these models have been successfully applied to responses of neurons in the early sensory pathway, they have fared less well as a models of responses in deeper brain areas, as they do not easily take into account multiple stages of processing. Here we introduce a new twist on this approach: we include unobserved as well as observed spike trains. This provides us with a more powerful model, and thus more flexibility in fitting data. More importantly, it allows us to estimate connectivity patterns among neurons (both observed and unobserved), and so should give insight into how networks process sensory input. We demonstrate the model on a simple toy network consisting of two neurons. The formalism, based on variational EM, can be easily extended to larger networks.