Spaun performing several tasks. Neural activity and decoding of that activity (thought bubbles) is shown. The model is running at about 1/6 real time. The tasks it performs are: A1 (recognition) three times, A0 (copy drawing) twice, A4 (counting), A5 (question answering), A3 (serial working memory). The tasks are described in the videos of each separate task.
input: 28x28 images (on the right)
output: movements of a physically modeled arm (top down view in the top right corner)
In this video, we see not only the activity of the various brain areas as in the previous video, but also decodings of the neural representations in those areas. The first task we are watching is a recognition task. Spaun must indicate with its arm what input image it saw. The images are randomly picked from a publicly available database of ~60,000 examples of human hand written digits. The classification it thinks it saw is indicated by the decoding of the neural activity in the thought bubble at the back of the brain. A subset of the voltage spikes generated by individual neurons is shown scrolling through the bubble. The decoded value itself is displayed on top of that neural activity.
The thought bubble at the top of the brain is a decoding of the representation in motor cortex. When this activity is decoded, a series of target points to move the arm to is generated. Movement of the arm through these points is controlled by the rest of the motor hierarchy below this level of representation. The resulting motion is evident in the movement of the arm itself.
Here the model is viewing a 2, which we can see is represented in the activity in visual areas, and which drives motor areas to generate a series of targets that move the arm in Spaun's default 'handwriting'.
The next task is slightly different. It is called Copy Drawing. Instead of just recognizing the digit, the model must attempt to reproduce the visual properties of the image it saw. As you can see, the first example it is shown is a two with a looped bottom. The model then produces a two with a similar characteristic loop. In the next example it is shown, it must perform the same task, but the two that is displayed has a straight bottom. As we can see from the motor activity and ultimately the arm motion, the representations in Spaun are able to capture this subtle visual difference. These representations are thus useful not only for categorization, but also for encoding fine perceptual properties.
The next task is a counting task. This task demonstrates that Spaun can not only represent and categorize numbers, it also has an understanding of numerical concepts, and particularly their progression relations. Spaun is first shown a number to start counting from (4), and then it is shown how much to count by (3).
Several new areas of the brain are shown in this example. In particular, three working memory areas are displayed right behind the front most area. If you pay attention to the thought bubbles above the working memory areas, you will see how Spaun performs this task. The topmost working memory stores the starting number. The second stores the number of counts, and the third stores how many counts have been performed. You can watch as Spaun increases the starting number until it has counted the correct number of times. It then generates the appropriate motor response.
Next we have a question answering task. Here, Spaun memorizes a list of numbers and then answers a query about the list. You can watch the topmost working memory as it stores the items in the order they are shown. The frontmost thought bubble shows what task Spaun thinks it is performing (A means answering a question), and what its current goals are. Essentially, this is a memory of the current task context. Once the list of numbers is complete, Spaun is asked a question. The kind of question is indicated by a letter (here a P, which indicates a position question). The next digit indicates which position is being queried. So here Spaun is asked "what is in position 5". Spaun answer the question by decoding its own working memory, and using the result to drive the arm. You will notice that the front bubble indicates 'dec' because Spaun is decoding memories to generate arm movements.
This final task is a simple serial working memory task in which Spaun must memorize a list of items in order, and then repeat it back. We have seen several tasks that Spaun performs correctly, but the errors it makes are equally important to determining if it is a good model of human cognition. As you may know, people make more errors while recalling longer lists. In fact, the pattern of errors in human memory is quite stereotypical - we tend to remember the beginnings and ends of lists better than the middle. As we watch Spaun encode this list, I should note that the darkness of the letter in the thought bubble indicates how well that item can be decoded from the current neural representation. As you can see, the 8 in the middle of this list is beginning to fade, indicating that it is being forgotten by the model. Spaun is now drawing its memory of the list, but when it gets to the 8, it is no longer confident in its own decoding of that memory, so it draws a horizontal line, indicating it has no answer. It then proceeds to complete the list, as it does remember subsequent items. If we look at the pattern of errors over many such runs of Spaun, we will notice that it matches the details of human error patterns.
I should note that, like humans, Spaun does not change it's brain in between performing tasks. Perhaps surprisingly, this is not true of many contemporary cognitive models. We have now seen about half of the tasks Spaun can perform. The rest are in other videos on the Nengo.ca website.