Machine co-creativity continues to grow and attract a wider audience to machine learning. Generative models, for example, have enabled new types of media creation across language, images, and music--including recent advances such as CLIP, VQGAN, and DALL·E. This one-day workshop will broadly explore topics in the applications of machine learning to creativity and design, which includes:
State-of-the-art algorithms for the creation of new media. Machine learning models achieving state-of-the-art in traditional media creation tasks (e.g., image, audio, or video synthesis) that are also being used by the artist community will be showcased.
Artist accessibility of machine learning models. Researchers building the next generation of machine learning models for media creation will be challenged in understanding the accessibility needs of artists. Artists and Human Computer interaction / User Experience community members will be encouraged to engage in the conversation.
The sociocultural and political impact of these new models. With the increased popularity of generative machine learning models, we are witnessing these models start to impact our everyday surroundings, ranging from racial and gender bias in algorithms and datasets used for media creation to how new media manipulation tools may erode our collective trust in media content.
Artistic applications. We will hear directly from some of the artists who are adopting machine learning--including deep learning and reinforcement learning--as part of their own artistic process as well as showcasing their work.
The goal of this workshop is to bring together researchers and artists interested in exploring the intersection of human creativity and machine learning and foster collaboration between them, as well as promote the sharing of ideas, perspectives, new research, artwork, and artistic needs.
Discord invite --> https://bit.ly/3puzVuM
Mon 8:15 a.m. - 8:30 a.m.
|
Welcome
(
Short Live Talk
)
SlidesLive Video » Welcome and introduction presented by Mattie Tesfaldet. |
🔗 |
Mon 8:30 a.m. - 9:30 a.m.
|
Poster Session 1
(
Poster Session
)
link »
Will be held in Discord. |
🔗 |
Mon 9:30 a.m. - 10:00 a.m.
|
Computers, Creativity, and Lovelace
(
Speaker Presentation
)
SlidesLive Video » |
Mark Riedl 🔗 |
Mon 10:00 a.m. - 10:30 a.m.
|
AI for Augmenting Human Creativity
(
Speaker Presentation
)
SlidesLive Video » |
Devi Parikh 🔗 |
Mon 10:30 a.m. - 11:00 a.m.
|
Interspecies Intelligence in Pharmako-AI
(
Speaker Presentation
)
SlidesLive Video » |
Kenric McDowell 🔗 |
Mon 11:00 a.m. - 11:30 a.m.
|
Art Show
(
Video
)
SlidesLive Video » Pre-recorded slide show of the art submissions |
🔗 |
Mon 11:30 a.m. - 12:00 p.m.
|
Panel Discussion: Mark Riedl, Devi Parikh, Kenric Allado-McDowell
(
Q&A Panel
)
SlidesLive Video » Mark, Devi, and Kenric on live panel answering questions, moderated by Mattie Tesfaldet. Please ask questions on Rocket.chat, not Discord. |
🔗 |
Mon 12:00 p.m. - 12:30 p.m.
|
Social 1
(
Informal Discussion
)
link »
Will be held in Discord. |
🔗 |
Mon 12:30 p.m. - 12:40 p.m.
|
StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Synthesis
(
Paper Oral
)
SlidesLive Video » Generating images that fit a given text description using machine learning has improved greatly with the release of technologies such as the CLIP image-text encoder model; however, current methods lack artistic control of the style of image to be generated. We introduce StyleCLIPDraw which adds a style loss to the CLIPDraw text-to-drawing synthesis model to allow artistic control of the synthesized drawings in addition to control of the content via text. Whereas performing decoupled style transfer on a generated image only affects the texture, our proposed coupled approach is able to capture a style in both texture and shape, suggesting that the style of the drawing is coupled with the drawing process itself. More results and our code are available at https://github.com/pschaldenbrand/StyleCLIPDraw |
Peter Schaldenbrand · Jean Oh · Zhixuan Liu 🔗 |
Mon 12:40 p.m. - 12:50 p.m.
|
Soundify: Matching Sound Effects to Video
(
Paper Oral
)
SlidesLive Video » In the art of video editing, sound is really half the story. A skilled video editor overlays sounds, such as effects and ambients, over footage to add character to an object or immerse the viewer within a space. However, through formative interviews with professional video editors, we found that this process can be extremely tedious and time-consuming. We introduce Soundify, a system that matches sound effects to video. By leveraging labeled, studio-quality sound effects libraries and extending CLIP, a neural network with impressive zero-shot image classification capabilities, into a "zero-shot detector", we are able to produce high-quality results without resource-intensive correspondence learning or audio generation. We encourage you to have a look at, or better yet, have a listen to the results at https://chuanenlin.com/soundify. |
David Chuan-En Lin 🔗 |
Mon 12:50 p.m. - 1:00 p.m.
|
Controllable and Interpretable Singing Voice Decomposition via Assem-VC
(
Paper Oral
)
SlidesLive Video » We propose a singing decomposition system that encodes time-aligned linguistic content, pitch, and source speaker identity via Assem-VC. With decomposed speaker-independent information and the target speaker's embedding, we could synthesize the singing voice of the target speaker. In conclusion, we made a perfectly synced duet with the user's singing voice and the target singer's converted singing voice. |
Kangwook Kim · Junhyeok Lee 🔗 |
Mon 1:00 p.m. - 1:10 p.m.
|
Extending the Vocabulary of Fictional Languages using Neural Networks
(
Paper Oral
)
SlidesLive Video » Fictional languages have become increasingly popular over the recent years appearing in novels, movies, TV shows, comics, and video games. While some of these fictional languages have a complete vocabulary, most do not. We propose a deep learning solution to the problem. Using style transfer and machine translation tools, we generate new words for a given target fictional language, while maintaining the style of its creator, hence extending this language vocabulary. |
Thomas Zacharias · Raja Giryes 🔗 |
Mon 1:10 p.m. - 1:15 p.m.
|
Jabberwocky
(
Artwork Spotlight
)
SlidesLive Video » Famous poem, illustrated with the automated text-to-video synthesis tool. Rendered with CLIP-FFT method, optimizing frequencies of Fourier inverse transformation. voiceover: Benedict Cumberbatch pretrained ML model: github.com/openai/CLIP |
Vadim Epstein 🔗 |
Mon 1:15 p.m. - 1:20 p.m.
|
Iterative Iterative
(
Artwork Spotlight
)
SlidesLive Video » I began by feeding thousands of images of my past artwork into a GAN. As a ceramic artist, the source imagery were photographs of sculptures and pottery I made between 2015-2019. The GAN required me to augment the data in multiple ways, including photographing my work from all sides. My vision was that the GAN would "imagine" my future work based on past work. However, the GAN was limited in ways that made this unfeasible. It was trapped within my prior work and could not imagine the new. I developed a collaboration with the machine where the shortcomings of the GAN become generative of new form. I prompted the GAN with my past work, and I interpreted its output as a prompt to either create the unknown half or interpret the output in 3 dimensions. It was within this unknown space that a void was created which offered the opportunity for novelty. As the GAN navigated this void, attempting to create new form from old, I also navigated this void. I embrace and utilize these digital shortcomings as a marking of time within technology and my own art practice. |
Erin Smith 🔗 |
Mon 1:20 p.m. - 1:25 p.m.
|
Artificial Intelligence for High Heel Shoe Design
(
Artwork Spotlight
)
SlidesLive Video » The images generated in this project use the VQGAN+CLIP Script created by Katherine Crowson (2021), licensed under the MIT license. The model has been modified to create new designs of high heel shoes using a text prompt. The text prompt can be as simple as one describing word to redesign the entire shoe. The text prompt is manipulating an image of a high heel shoe , which is applied as an ‘init-image’. This tool can be used to create hundreds of original designs, whether you are a designer or not. Find more of Sophia's work at; https://www.sophianeilldesign.com |
Sophia Neill 🔗 |
Mon 1:25 p.m. - 1:30 p.m.
|
text2pixelart
(
Artwork Spotlight
)
SlidesLive Video » This project explores opportunities for an unsupervised generation of animated pixel-art based on the given text prompt. As the base, I used PixelDraw by @dribnet, based on the ClipDraw library, which in turn is based on the diffvg library. First, I've implemented some handy options, such as smoothness enforcement, palette enforcement, and saturation enforcement. Next, I've exploited the CLIP's "healing" ability and a couple of "demoscene" tricks to keep the coherence between adjacent frames to make several different animations: the infinite panorama, the parallax effect, the 3d swirl around the object, the moving background, the endless looping, and so on. I've tried to give a tribute to old-school classics (using prompts like [fallout], [r-type], and [Guybrush Threepwood]) and to explore the fractal and space thematics, trying to find the best match between the prompt and the effect in each case. The code of some tricks is published already; other hacks are on their way to be publicly available. Web link: https://altsoph.github.io/text2pixelart/ Full video can be downloaded here: https://www.dropbox.com/s/r88x7ze1wy8wjo5/text2pixelartclipv2.mp4?dl=0 |
Alexey Tikhonov 🔗 |
Mon 1:30 p.m. - 2:30 p.m.
|
Poster Session 2
(
Poster Session
)
link »
Will be held in Discord. |
🔗 |
Mon 2:30 p.m. - 3:00 p.m.
|
From Technologies to Tools
(
Speaker Presentation
)
SlidesLive Video » |
Joel Simon 🔗 |
Mon 3:00 p.m. - 3:30 p.m.
|
Okachihuali: Strategies for New Future Building with AI Art
(
Speaker Presentation
)
SlidesLive Video » |
Moisés Horta Valenzuela 🔗 |
Mon 3:30 p.m. - 4:00 p.m.
|
Imaginatory Processes with VQGANxCLIP
(
Speaker Presentation
)
SlidesLive Video » |
Sebastián Rojas · María Constanza Lobos 🔗 |
Mon 4:00 p.m. - 4:30 p.m.
|
Art Show (repeated)
(
Video
)
SlidesLive Video » |
🔗 |
Mon 4:30 p.m. - 5:00 p.m.
|
Panel Discussion: Joel Simon, Hypereikon
(
Q&A Panel
)
SlidesLive Video » Joel and Hypereikon on live panel answering questions, moderated by David Ha. Please ask questions on Rocket.chat, not Discord. |
🔗 |
Mon 5:00 p.m. - 5:15 p.m.
|
Closing Remarks
(
Short Live Talk
)
SlidesLive Video » Closing remarks presented by Tom White. |
🔗 |
Mon 5:15 p.m. - 6:00 p.m.
|
Social 2
(
Informal Discussion
)
link »
Will be held in Discord. |
🔗 |