Timezone: »
In the last year, generative machine learning and machine creativity have gotten a lot of attention in the non-research world. At the same time there have been significant advances in generative models for media creation and for design. This one-day workshop explores several issues in the domain of generative models for creativity and design. First, we will look at algorithms for generation and creation of new media and new designs, engaging researchers building the next generation of generative models (GANs, RL, etc) and also from a more information-theoretic view of creativity (compression, entropy, etc). Second, we will investigate the social and cultural impact of these new models, engaging researchers from HCI/UX communities. Finally, we’ll hear from some of the artists and musicians who are adopting machine learning approaches like deep learning and reinforcement learning as part of their artistic process. We’ll leave ample time for discussing both the important technical challenges of generative models for creativity and design, as well as the philosophical and cultural issues that surround this area of research.
Background
In 2016, DeepMind’s AlphaGo made two moves against Lee Sedol that were described by the Go community as “brilliant,” “surprising,” “beautiful,” and so forth. Moreover, there was little discussion surrounding the fact that these very creative moves were actually made by a machine (Wired); it was enough that they were great examples of go playing. At the same time, the general public showed more concern for other applications of generative models. Algorithms that allow for convincing voice style transfer (Lyrebird) or puppet-like video face control (Face2Face) have raised concerns that generative ML will be used to make convincing forms of fake news (FastCompany).
Balancing this, the arts and music worlds have positively embraced generative models. Starting with DeepDream and expanding with image and video generation advances (e.g. GANs) we’ve seen lots of new and interesting art and music [citations] technologies provided by the machine learning community. We’ve seen research projects like Google Brain’s Magenta, Sony CSL’s FlowMachines and IBM’s Watson undertake collaborations and attempt to build tools and ML models for use by these communities.
Research
Recent advances in generative models enable new possibilities in art and music production. Language models can be used to write science fiction film scripts (Sunspring) and even replicate the style of individual authors (Deep Tingle). Generative models for image and video allow us to create visions of people, places and things that resemble the distribution of actual images (GANs etc). Sequence modelling techniques have opened up the possibility of generating realistic musical scores (MIDI generation etc) and even raw audio that resembles human speech and physical instruments (DeepMind’s WaveNet, MILA’s Char2Wav and Google’s NSynth). In addition, sequence modelling allows us to model vector images to construct stroke-based drawings of common objects according to human doodles (sketch-rnn).
In addition to field-specific research, a number of papers have come out that are directly applicable to the challenges of generation and evaluation such as learning from human preferences (Christiano et al., 2017) and CycleGAN. The application of Novelty Search (Stanley), evolutionary complexification (Stanley - CPPN, NEAT, Nguyen et al - Plug&Play GANs, Innovation Engine) and intrinsic motivation (Oudeyer et al 2007, Schmidhuber on Fun and Creativity) techniques, where objective functions are constantly evolving, is still not common practice in art and music generation using machine learning.
Another focus of the workshop is how to better enable human influence over generative models. This could include learning from human preferences, exposing model parameters in ways that are understandable and relevant to users in a given application domain (e.g., similar to Morris et al. 2008), enabling users to manipulate models through changes to training data (Fiebrink et al. 2011), allowing users to dynamically mix between multiple generative models (Akten & Grierson 2016), or other techniques. Although questions of how to make learning algorithms controllable and understandable to users are relatively nacesent in the modern context of deep learning and reinforcement learning, such questions have been a growing focus of work within the human-computer interaction community (e.g., examined in a CHI 2016 workshop on Human-Centred Machine Learning), and the AI Safety community (e.g. Christiano et al. 2017, using human preferences to train deep reinforcement learning systems). Such considerations also underpin the new Google “People + AI Research” (PAIR) initiative.
Artists and Musicians
All the above techniques improve our capabilities of producing text, sound and images. Art and music that stands the test of time however requires more than that. Recent research includes a focus on novelty in creative adversarial networks (Elgammal et al., 2017) and considers how generative algorithms can integrate into human creative processes, supporting exploration of new ideas as well as human influence over generated content (Atken & Grierson 2016a, 2016b). Artists including Mario Klingemann, Gene Kogan, Mike Tyka, and Memo Akten have further contributed to this space of work by creating artwork that compellingly demonstrates capabilities of generative algorithms, and by publicly reflecting on the artistic affordances of these new tools.
The goal of this workshop is to bring together researchers interested in advancing art and music generation to present new work, foster collaborations and build networks.
In this workshop, we are particularly interested in how the following can be used in art and music generation: reinforcement learning, generative adversarial networks, novelty search and evaluation as well as learning from user preferences. We welcome submissions of short papers, demos and extended abstracts related to the above.
There will also be an open call for a display of artworks incorporating machine learning techniques.
Fri 8:30 a.m. - 8:45 a.m.
|
Welcome and Introduction
(
Introduction
)
|
🔗 |
Fri 8:45 a.m. - 9:15 a.m.
|
Invited Talk
(
Talk
)
|
Jürgen Schmidhuber 🔗 |
Fri 9:15 a.m. - 9:45 a.m.
|
Invited Talk
(
Talk
)
|
Emily Denton 🔗 |
Fri 9:45 a.m. - 10:15 a.m.
|
Invited Talk
(
Talk
)
|
Rebecca Fiebrink 🔗 |
Fri 10:15 a.m. - 10:20 a.m.
|
GANosaic - Mosaic Creation with Generative Texture Manifolds
(
Spotlight talk
)
|
Nikolay Jetchev · Urs Bergmann · Calvin Seward 🔗 |
Fri 10:20 a.m. - 10:25 a.m.
|
TopoSketch: Drawing in Latent Space
(
Spotlight talk
)
|
🔗 |
Fri 10:25 a.m. - 10:30 a.m.
|
Input parameterization for DeepDream
(
Spotlight talk
)
|
🔗 |
Fri 11:00 a.m. - 11:30 a.m.
|
Invited Talk
(
Talk
)
|
Ian Goodfellow 🔗 |
Fri 11:30 a.m. - 12:00 p.m.
|
Improvised Comedy as a Turing Test
(
Contributed Talk
)
|
🔗 |
Fri 12:00 p.m. - 1:00 p.m.
|
Lunch
|
🔗 |
Fri 1:00 p.m. - 1:30 p.m.
|
Invited Talk
(
Talk
)
|
Ahmed Elgammal 🔗 |
Fri 1:30 p.m. - 2:00 p.m.
|
Hierarchical Variational Autoencoders for Music
(
Contributed Talk
)
|
🔗 |
Fri 2:00 p.m. - 2:30 p.m.
|
Lexical preferences in an automated story writing system
(
Contributed Talk
)
|
🔗 |
Fri 2:30 p.m. - 3:00 p.m.
|
ObamaNet: Photo-realistic lip-sync from text
(
Contributed Talk
)
|
Rithesh Kumar · Jose Sotelo · Kundan Kumar · Alexandre de Brébisson 🔗 |
Fri 3:00 p.m. - 3:30 p.m.
|
Art / Coffee Break
|
🔗 |
Fri 3:30 p.m. - 3:35 p.m.
|
Towards the High-quality Anime Characters Generation with Generative Adversarial Networks
(
Spotlight talk
)
|
🔗 |
Fri 3:35 p.m. - 3:40 p.m.
|
Crowd Sourcing Clothes Design Directed by Adversarial Neural Networks
(
Spotlight talk
)
|
Hiroyuki Osone · Natsumi Kato · Daitetsu Sato · Naoya Muramatsu 🔗 |
Fri 3:40 p.m. - 3:45 p.m.
|
Paper Cubes: Evolving 3D characters in Augmented Reality using Recurrent Neural Networks
(
Spotlight talk
)
|
Anna Fuste · Jonas Jongejan 🔗 |
Fri 3:45 p.m. - 4:15 p.m.
|
Open discussion
(
Discussion
)
|
🔗 |
Fri 4:15 p.m. - 5:00 p.m.
|
Combinatorial Meta Search
(
Poster
)
|
🔗 |
Fri 4:15 p.m. - 5:00 p.m.
|
Exploring Audio Style Transfer
(
Poster
)
|
🔗 |
Fri 4:15 p.m. - 5:00 p.m.
|
Deep Interactive Evolutionary Computation
(
Poster
)
|
Philip Bontrager 🔗 |
Fri 4:15 p.m. - 5:00 p.m.
|
Disentangled representations of style and content for visual art with generative adversarial networks
(
Poster
)
|
🔗 |
Fri 4:15 p.m. - 5:00 p.m.
|
Sequential Line Search for Generative Adversarial Networks
(
Poster
)
|
🔗 |
Fri 4:15 p.m. - 5:00 p.m.
|
ASCII Art Synthesis with Convolutional Networks
(
Poster
)
|
🔗 |
Fri 4:15 p.m. - 5:00 p.m.
|
Compositional Pattern Producing GAN
(
Poster
)
|
🔗 |
Fri 4:15 p.m. - 5:00 p.m.
|
Generative Embedded Mapping Systems for Design
(
Poster
)
|
🔗 |
Fri 4:15 p.m. - 5:00 p.m.
|
Consistent Comic Colorization with Pixel-wise Background Classification
(
Poster
)
|
Jaegul Choo · Sungmin Kang 🔗 |
Fri 4:15 p.m. - 5:00 p.m.
|
Deep Learning for Identifying Potential Conceptual Shifts for Co-creative Drawing
(
Poster
)
|
Pegah Karimi 🔗 |
Fri 4:15 p.m. - 5:00 p.m.
|
Imaginary Soundscape : Cross-Modal Approach to Generate Pseudo Sound Environments
(
Poster
)
|
Yuma Kajihara · Nao Tokui 🔗 |
Fri 4:15 p.m. - 5:00 p.m.
|
Repeating and Remembering: GANs in an art context
(
Poster
)
|
Anna Ridler 🔗 |
Fri 4:15 p.m. - 5:00 p.m.
|
Improvisational Storytelling Agents
(
Poster
)
|
🔗 |
Fri 4:15 p.m. - 5:00 p.m.
|
Learning to Create Piano Performances
(
Poster
)
|
🔗 |
Fri 4:15 p.m. - 5:00 p.m.
|
AI for Fragrance Design
(
Poster
)
|
Richard Segal 🔗 |
Fri 4:15 p.m. - 5:00 p.m.
|
Neural Style Transfer for Audio Spectograms
(
Poster
)
|
Prateek Verma · Julius O Smith 🔗 |
Fri 4:15 p.m. - 5:00 p.m.
|
SocialML: machine learning for social media video creators
(
Poster
)
|
🔗 |
Fri 4:15 p.m. - 5:00 p.m.
|
The Emotional GAN: Priming Adversarial Generation of Art with Emotion
(
Poster
)
|
Judith Amores Fernandez 🔗 |
Fri 4:15 p.m. - 5:00 p.m.
|
Algorithmic composition of polyphonic music with the WaveCRF
(
Poster
)
|
🔗 |
Fri 4:15 p.m. - 5:00 p.m.
|
SOMNIA: Self-Organizing Maps as Neural Interactive Art
(
Poster
)
|
🔗 |
Fri 4:15 p.m. - 5:00 p.m.
|
Generating Black Metal and Math Rock: Beyond Bach, Beethoven, and Beatles
(
Poster
)
|
🔗 |
Fri 4:15 p.m. - 5:00 p.m.
|
Artwork
|
Daniel Ambrosi · Michel Erler · Jason Salavon · Theresa Reimann-Dubbers · Robbie Barrat 🔗 |
Author Information
Douglas Eck (Google Brain)
I’m a research scientist working on Magenta, an effort to generate music, video, images and text using machine intelligence. Magenta is part of the Google Brain team and is using TensorFlow (www.tensorflow.org), an open-source library for machine learning. The question Magenta asks is, “Can machines make music and art? If so, how? If not, why not?” The goal if Magenta is to produce open-source tools and models that help creative people be even more creative. I’m primarily looking at how to use so-called “generative” machine learning models to create engaging media. Additionally, I’m working on how to bring other aspects of the creative process into play. For example, art and music is not just about generating new pieces. It’s also about drawing one’s attention, being surprising, telling an interesting story, knowing what’s interesting in a scene, and so on. Before starting the Magenta project, I worked on music search and recommendation for Google Play Music. My research goal in this area was to use machine learning and audio signal processing to help listeners find the music they want when they want it. This involves both learning from audio and learning from how users consume music. In the audio domain, the main goal is to transform the ones and zeros in a digital audio file into something where musically-similar songs are also numerically similar, making it easier to do music recommendation. This is (a) user-dependent: my idea of similar is not the same as yours and (b) changes with context: my idea of similarity changes when I make a playlist for jogging versus making a playlist for a dinner party. I might choose the same song (say "Taxman" by the Beatles) but perhaps it would be the tempo for jogging that drove the selection of that specific song versus "I like the album Revolver and want to add it to the dinner party mix" for a dinner party playlist. I joined Google in 2003. Before then, I was an Associate Professor in Computer Science at University of Montreal. I helped found the BRAMS research center (Brain Music and Sound; www.brams.org) and was involved at the McGill CIRMMT center (Centre for Interdisciplinary Research in Music Media and Technology; www.cirmmt.org). Aside from audio signal processing and machine learning, I worked on music performance modeling. What exactly does a good music performer add to what is already in the score? I treated this as a machine learning question: Hypothetically, if we showed a piano-playing robot a huge collection of Chopin performances--- from the best in the world all the way down to that of a struggling teenage pianist---could it learn to play well by analyzing all of these examples? If so, what’s the right way to perform that analysis? In the end I learned a lot about the complexity and beauty of human music performance, and how performance relates to and extends composition.
David Ha (Google Brain)
S. M. Ali Eslami (DeepMind)
Sander Dieleman (DeepMind)
Rebecca Fiebrink (Goldsmiths University of London)
Luba Elliott (independent AI Curator)
Luba Elliott is a curator, artist and researcher specialising in artificial intelligence in the creative industries. She is currently working to educate and engage the broader public about the latest developments in creative AI through monthly meetups, talks and tech demonstrations. As curator, she organised workshops and exhibitions on art and AI for The Photographers’ Gallery, the Leverhulme Centre for the Future of Intelligence and Google. Prior to that, she worked in start-ups, including the art collector database Larry’s List. She obtained her undergraduate degree in Modern Languages at the University of Cambridge and has a certificate in Design Thinking from the Hasso-Plattner-Institute D-school in Potsdam.
More from the Same Authors
-
2021 : Inferring a Continuous Distribution of Atom Coordinates from Cryo-EM Images using VAEs »
Dan Rosenbaum · Marta Garnelo · Michal Zielinski · Charles Beattie · Ellen Clancy · Andrea Huber · Pushmeet Kohli · Andrew Senior · John Jumper · Carl Doersch · S. M. Ali Eslami · Olaf Ronneberger · Jonas Adler -
2023 Poster: Self-supervised video pretraining yields human-aligned visual representations »
Nikhil Parthasarathy · S. M. Ali Eslami · Joao Carreira · Olivier Henaff -
2023 Workshop: Machine Learning for Audio »
Brian Kulis · Sadie Allen · Sander Dieleman · Shrikanth Narayanan · Rachel Manzelli · Alice Baird · Alan Cowen -
2023 Affinity Workshop: Muslims in ML »
Sanae Lotfi · Hammaad Adam · Marzyeh Ghassemi · Shakir Mohamed · S. M. Ali Eslami -
2021 : Inferring a Continuous Distribution of Atom Coordinates from Cryo-EM Images using VAEs »
Dan Rosenbaum · Marta Garnelo · Michal Zielinski · Charles Beattie · Ellen Clancy · Andrea Huber · Pushmeet Kohli · Andrew Senior · John Jumper · Carl Doersch · S. M. Ali Eslami · Olaf Ronneberger · Jonas Adler -
2021 Poster: Multimodal Few-Shot Learning with Frozen Language Models »
Maria Tsimpoukelli · Jacob L Menick · Serkan Cabi · S. M. Ali Eslami · Oriol Vinyals · Felix Hill -
2020 : Art Showcase 1 »
Luba Elliott -
2020 : Panel Discussion 1 »
Luba Elliott · Janelle Shane · Sofia Crespo · Scott Eaton · Adam Roberts · Angela Fan -
2020 : Art Showcase 1 »
Luba Elliott -
2020 Workshop: Machine Learning for Creativity and Design 4.0 »
Luba Elliott · Sander Dieleman · Adam Roberts · Tom White · Daphne Ippolito · Holly Grimm · Mattie Tesfaldet · Samaneh Azadi -
2020 : Introduction and Art Gallery Overview »
Luba Elliott -
2019 : Innate Bodies, Innate Brains, and Innate World Models »
David Ha -
2019 : AI Art Gallery Overview »
Luba Elliott -
2019 Workshop: NeurIPS Workshop on Machine Learning for Creativity and Design 3.0 »
Luba Elliott · Sander Dieleman · Adam Roberts · Jesse Engel · Tom White · Rebecca Fiebrink · Parag Mital · Christine McLeavey · Nao Tokui -
2019 Poster: Weight Agnostic Neural Networks »
Adam Gaier · David Ha -
2019 Poster: Learning to Predict Without Looking Ahead: World Models Without Forward Prediction »
Daniel Freeman · David Ha · Luke Metz -
2019 Spotlight: Weight Agnostic Neural Networks »
Adam Gaier · David Ha -
2018 : AI art gallery overview »
Luba Elliott -
2018 : David Ha »
David Ha -
2018 Workshop: Second Workshop on Machine Learning for Creativity and Design »
Luba Elliott · Sander Dieleman · Rebecca Fiebrink · Jesse Engel · Adam Roberts · Tom White -
2018 : Poster Session 1 + Coffee »
Tom Van de Wiele · Rui Zhao · J. Fernando Hernandez-Garcia · Fabio Pardo · Xian Yeow Lee · Xiaolin Andy Li · Marcin Andrychowicz · Jie Tang · Suraj Nair · Juhyeon Lee · Cédric Colas · S. M. Ali Eslami · Yen-Chen Wu · Stephen McAleer · Ryan Julian · Yang Xue · Matthia Sabatelli · Pranav Shyam · Alexandros Kalousis · Giovanni Montana · Emanuele Pesce · Felix Leibfried · Zhanpeng He · Chunxiao Liu · Yanjun Li · Yoshihide Sawada · Alexander Pashevich · Tejas Kulkarni · Keiran Paster · Luca Rigazio · Quan Vuong · Hyunggon Park · Minhae Kwon · Rivindu Weerasekera · Shamane Siriwardhanaa · Rui Wang · Ozsel Kilinc · Keith Ross · Yizhou Wang · Simon Schmitt · Thomas Anthony · Evan Cater · Forest Agostinelli · Tegg Sung · Shirou Maruyama · Alexander Shmakov · Devin Schwab · Mohammad Firouzi · Glen Berseth · Denis Osipychev · Jesse Farebrother · Jianlan Luo · William Agnew · Peter Vrancx · Jonathan Heek · Catalin Ionescu · Haiyan Yin · Megumi Miyashita · Nathan Jay · Noga H. Rotman · Sam Leroux · Shaileshh Bojja Venkatakrishnan · Henri Schmidt · Jack Terwilliger · Ishan Durugkar · Jonathan Sauder · David Kas · Arash Tavakoli · Alain-Sam Cohen · Philip Bontrager · Adam Lerer · Thomas Paine · Ahmed Khalifa · Ruben Rodriguez · Avi Singh · Yiming Zhang -
2018 Poster: Recurrent World Models Facilitate Policy Evolution »
David Ha · Jürgen Schmidhuber -
2018 Oral: Recurrent World Models Facilitate Policy Evolution »
David Ha · Jürgen Schmidhuber -
2018 Poster: A Probabilistic U-Net for Segmentation of Ambiguous Images »
Simon Kohl · Bernardino Romera-Paredes · Clemens Meyer · Jeffrey De Fauw · Joseph R. Ledsam · Klaus Maier-Hein · S. M. Ali Eslami · Danilo Jimenez Rezende · Olaf Ronneberger -
2018 Spotlight: A Probabilistic U-Net for Segmentation of Ambiguous Images »
Simon Kohl · Bernardino Romera-Paredes · Clemens Meyer · Jeffrey De Fauw · Joseph R. Ledsam · Klaus Maier-Hein · S. M. Ali Eslami · Danilo Jimenez Rezende · Olaf Ronneberger -
2017 : Deep learning for music recommendation and generation »
Sander Dieleman -
2017 : Polyphonic piano transcription using deep neural networks »
Douglas Eck -
2017 : Invited Talk »
Rebecca Fiebrink -
2017 Demonstration: Magenta and deeplearn.js: Real-time Control of DeepGenerative Music Models in the Browser »
Curtis Hawthorne · Ian Simon · Adam Roberts · Jesse Engel · Daniel Smilkov · Nikhil Thorat · Douglas Eck -
2016 Demonstration: Interactive musical improvisation with Magenta »
Adam Roberts · Jesse Engel · Curtis Hawthorne · Ian Simon · Elliot Waite · Sageev Oore · Natasha Jaques · Cinjon Resnick · Douglas Eck -
2016 Poster: Unsupervised Learning of 3D Structure from Images »
Danilo Jimenez Rezende · S. M. Ali Eslami · Shakir Mohamed · Peter Battaglia · Max Jaderberg · Nicolas Heess -
2016 Poster: Attend, Infer, Repeat: Fast Scene Understanding with Generative Models »
S. M. Ali Eslami · Nicolas Heess · Theophane Weber · Yuval Tassa · David Szepesvari · koray kavukcuoglu · Geoffrey E Hinton -
2015 Workshop: Black box learning and inference »
Josh Tenenbaum · Jan-Willem van de Meent · Tejas Kulkarni · S. M. Ali Eslami · Brooks Paige · Frank Wood · Zoubin Ghahramani -
2011 Workshop: The 4th International Workshop on Music and Machine Learning: Learning from Musical Structure »
Rafael Ramirez · Darrell Conklin · Douglas Eck · Rif A. Saurous -
2009 Poster: An Infinite Factor Model Hierarchy Via a Noisy-Or Mechanism »
Aaron Courville · Douglas Eck · Yoshua Bengio -
2007 Poster: Automatic Generation of Social Tags for Music Recommendation »
Douglas Eck · Paul Lamere · Thierry Bertin-Mahieux · Stephen J Green