On the Science of “Alien Intelligences”: Evaluating Cognitive Capabilities in Babies, Animals, and AI

Today’s generative AI systems—termed by some researchers as “alien intelligences”—have exceeded human performance on many benchmarks meant to test humanlike cognitive capabilities. However, these systems still struggle in unhumanlike ways on real-world tasks requiring these capabilities. This disconnect may be due in part to neglect in the AI community of well-founded experimental protocols for evaluating cognition. In this talk I will summarize several recommendations on experimental methods from developmental and comparative psychology—fields that study the “alien intelligences” of babies and non-human animals—and demonstrate the application of such methods in two case studies of cognitive abilities in LLMs: analogical reasoning and visual abstraction.