Ever since the 1950s AI scientists have been experimenting with the prospect of using computer technology to emulate human intelligence. While universal function approximation theorems promised success in this pursuit provided the kinds of tasks human intelligence was solving could be formulated as continuous function approximation problems, and provided enough scale was available to train MLPs of arbitrary width or depth, we find ourselves in the age of billion parameter models, and yet still far away from being able to replicate all aspects of human intelligence. Also, our models are not MLPs, but convolutional, recurrent, or otherwise structured neural networks. In this talk we will discuss why that is, and consider the general principles that can guide us towards building a new generation of neural networks with the kinds of structure that can solve the full spectrum of tasks that human intelligence can solve.

Modern neural networks are powerful function approximators which manipulate complex input data to extract information useful for one or more tasks. Extracting information is a process which transforms data to make it simpler and simpler to use. For example, binary classification might start from high dimensional images, and progressively simplify this data to shrink it down to a single number between 0 and 1: the probability that the original image contained a dog. Many transformations of the input image should lead to the same output. For example, changing background colour, rotating images, and so on, should not affect the answer provided by the neural network. When the transformations applied to the input are invertible, we are dealing with symmetries of the inputs. And a question arises of whether knowledge of such symmetries can help researchers devise better neural networks. In this section, we will visit the subject of symmetries, their benefits, and see examples of their usage.

Deep learning has led to some incredible successes in a very broad range of applications within AI. However, deep learning models remain black boxes, often unable to explain how they reach their final answers with no clear signals as to “what went wrong?” when models fail. Further, they typically require huge amounts of data during training and often do not generalize well beyond the data they have been trained on. But AI has not always been this way. In the “Good-Old days”, GOFAI did not require any data at all and the final solutions were interpretable, but the AI’s were not grounded in the real world. Further, unlike deep learning where a single general algorithm could be used to learn to solve many different problems, a single GOFAI algorithm can only be applied to a single task. So can we have our cake and eat it too? Is there a solution to AI out there that requires a limited amount of data, is interpretable, generalises well to new problems and can be applied to a wide variety of tasks? One interesting, developing area of AI that could answer this question is NeuroSymbolic AI which combines deep learning and logical reasoning in a single model. In this tutorial we will explore these models in the context of “structure” identifying how varying degrees of structure in a model affects its interpretability, how well it generalises to new data as the generality of the algorithm and the variety of tasks it can be applied to.