Magenta and deeplearn.js: Real-time Control of DeepGenerative Music Models in the Browser

There has recently been increased interest in generating music using deep learning techniques, leading to remarkable improvements in the quality and expressiveness of sequence-based models.

Beyond unconditional generation, we aim to explore the ability of the generative models to augment the creativity of musicians and novices alike. To be successful, both the model and the user interface must expose high-level and expressive controls that empower users to explore novel musical possibilities. Furthermore, the interface must be easy both for casual users to access and for professional users to integrate into existing creative workflows. This is key to new directions in adaptive feedback and training of models based on user preferences.

To this end, we train state-of-the-art generative models with conditional controls for several musical domains — virtuosic piano performances, looping melodies and drum beats — and demonstrate user interfaces to control generation from these models in real time using only code running in a browser-based JavaScript environment via deeplearn.js.