How the Brain ‘Constructs’ the Outside World

How the Brain 'Constructs' the Outside World thumbnail

As a young course instructor in seminars for medical students, I faithfully taught neurophysiology by the book, enthusiastically explaining how the brain perceives the world and controls the body. Sensory stimuli such as those from the eyes, ears, or other sensory organs are converted into electrical signals, which are then transmitted to the appropriate parts of the sensory cortex, which process these inputs and instigate perception. Motor cortex impulses instruct the spinal cord neurons in the motor cortex to initiate a movement.

Most students were satisfied with my textbook explanations about the brain’s input and output mechanisms. However, a few students were more curious and asked awkward questions. “Where in the brain does perception occur?” “What initiates a finger movement before cells in the motor cortex fire?” I would always dispatch their queries with a simple answer: “That all happens in the neocortex.” Then I would skillfully change the subject or use a few obscure Latin terms that my students did not really understand but that seemed scientific enough so that my authoritative-sounding accounts temporarily satisfied them.

I started my research on the brain like many young researchers. I didn’t care if this perception-action theoretical framework was correct or not. I was happy for many years with my own progress and the spectacular discoveries that gradually evolved into what became known in the 1960s as the field of “neuroscience.” Yet my inability to give satisfactory answers to the legitimate questions of my smartest students has haunted me ever since. I had to struggle with the difficulty of explaining something I didn’t understand.

Over the years, I realized that my frustration wasn’t unique. I was not the only one feeling this way. These frustrations were a source of my professional energy and a bright side. These frustrations pushed me over the years towards a different perspective on how the brain interacts to the outside world.

The challenge for me and other neuroscientists is the big question of what exactly is the mind. Since the time of Aristotle thinkers have believed that the soul or mind is a blank slate. A tabula rasa upon which experiences are painted. This view has influenced the thinking of British empiricism, Marxist doctrine, Christian and Persian philosophies, and British empiricism. It has also been a part of psychology and cognitive science over the past century. This “outsider-in” view depicts the mind as a tool to learn about the true nature and purpose of the world. My research has led me to a different view. It states that brain networks’ primary concern is to maintain their internal dynamics and generate myriad patterns of neural activity. The neuronal pattern that led to the action gains meaning when it seems like an unplanned action is beneficial to the organism’s survival. The meaning of the Teddy bear is given to the sound “tete” when an infant says “tee-te”. This framework has been supported by neuroscience advances.

Does “represent” the brain?

Neuroscience was born from the blank slate framework of the millennia, after early thinkers gave mental operations names like tabula rasa. We still look for neural mechanisms that may relate to their ideas today. The dominance of the outside-in framework is illustrated by the outstanding discoveries of the legendary scientific duo David Hubel and Torsten Wiesel, who introduced single-neuronal recordings to study the visual system and were awarded the Nobel Prize in Physiology or Medicine in 1981. They used their signature experiment to record neural activity in animals and show them images of different shapes. Different physical properties, such as edges, light and dark areas, elicited different firing patterns in different sets of neurons. The idea was that neuronal computation begins with simple patterns that are then synthesized into more complex ones. These features are then linked together in the brain to form an object. You don’t need to actively participate. This exercise is performed automatically by the brain.

The outside-in framework assumes that the brain’s primary function is to interpret “signals” received from the world. If this assumption is true, then an additional operation is required to respond to these signals. A central processor, which is a hypothetical device that takes in sensory representations and makes decisions about how to use them to take the correct action, is located between the perceptual inputs.

What is the central processor of this outside-in paradigm model? This mysterious and poorly understood entity is known by many names, including free will, homunculus and decision maker. It can also be called a “black box” depending on whether it is applied to human brains, brains of other animals, or computer models. All these terms refer to the same thing.

The implicit practical implication from the outside-in framework suggests that the next frontier in contemporary neuroscience should be to discover the location of the brain’s central processor and to systematically develop the neuronal mechanisms for decision-making. The physiology and process of decision-making is a popular topic in neuroscience today. Higher-order brain regions such as the prefrontal cortex have been postulated to be the place where “all the things come together” or “all outputs are started.” However, if we look closer, the outside-in framework doesn’t hold together.

This approach cannot explain why photons that hit the retina transform into a memory of a summer trip. This outside-in framework requires artificially inserting a human experimenter to observe this event [see graphic below].. The experimenter-in-the-middle is needed because even if neurons change their firing patterns when receptors on sensory organs are stimulated–by light or sound, for instance–these changes do not intrinsically “represent” anything that can be absorbed and integrated by the brain. The visual cortex neurons that respond to the image, for example, of a rose, have no clue. They don’t “see” the appearance or form of a flower. They generate electrical oscillations as a result of inputs from other brain parts, including complex pathways that lead to the retina.

In other words, neurons in the sensory cortical areas as well as the hypothetical central processor can’t “see” events happening in the world. These changes in neuronal firing patterns are not interpreted by the brain. The neurons that take in these events are unaware of what caused them. Fluctuations of neuronal activity can only be understood by scientists who have the privilege of observing both the brain and the outside world, and then comparing them.

Graphic explains outside-in versus inside-out framework using the example of how neurons in the brain might “see” a flower.
Credit: Brown Bird Design

Perception is what we do

Because neurons don’t have direct access to the outside world they need a way for them to “ground” or compare their firing patterns with something else. Grounding refers to the brain’s ability to assign meaning to changes that occur in neuronal firing patterns as a result of sensory inputs. This is done by linking the activity to another. When a Morse code pattern is linked to the letter “G”, it becomes meaningful.

We learn that even if sticks appear bent in water, they can still be moved. Similar to the distance between two trees or two mountain peaks, we can learn the difference by shifting our perspective and moving around.

The outside-in framework describes a series of events that take place from perception to decision to action. This model shows that neurons in the sensory areas are “driven” (or controlled) by environmental signals. They cannot therefore relate their activity to another. The brain is not a serial processing unit. It does not go through each step sequentially. Instead, any action taken by a person involves the brain’s motor regions informing the rest the cerebral cortex about the action initiated. This is known as a “corollary discharge”.

Neuronal Circuits that initiate an action have two tasks. The first task is to send a command the muscles controlling the eyes and other bodily sensor (the fingers, tongue, and others). These circuits direct bodily sensors in the best direction to enable in-depth investigation and analysis of the source of an input. They also enhance the brain’s ability for identifying the nature and location initial ambiguous signals from the senses.

The second task of the same action circuits is to send notifications–the corollary charges–to sensory or higher-order brain regions. These notifications can be thought of as registered mail receipts. The neurons that initiate eye movements also notify the visual sensory areas of cortex about what is occurring and disambiguate whether a flower is being handled by someone or moving in the wind.

This corollary message provides grounding for the second opinion sensory circuits–a confirmation of “my own actions are the agent of change.” When a person investigates the relationship between the flower and oneself, it sends a similar message to the rest. Without this exploration, stimuli from a flower–the photons arriving on a retina connected to an inexperienced mind–would not become signals that provide a meaningful description about the flower’s size or shape. Perception then can be defined as what we do–not what we passively take in through our senses.

This is a simplified version of the corollary-discharge mechanism. While reading this text, cover one eye with your other hand. Then move the other eye gently to the side with your tip of the finger. You’ll immediately notice that the page is moving. Compared to this, nothing seems to be moving when you’re reading or looking around the room. This constancy is due to the fact that neurons that initiate eye movements to scan sentences also send a signal to the visual system to indicate if the world or eyeball are moving.

Learning with matching

When used to explain learning mechanisms, the contrast between inside-out and outside-in approaches is most striking. The assumption that the brain becomes more complex with experience is implicit in the blank slate model. As we learn, the interconnected brain circuits should become more complex. However, in the inside-out framework, experience is not the only source of brain complexity.

Instead, the brain organizes itself into a vast array of preformed firing patterns known as neuronal trajectory. This self-organized brain model could be compared to a dictionary that is filled with nonsensical words. The way these networks work and their activity level does not change with new experience. Learning occurs rather by matching preexisting neuronal trajectories with events in the world.

To understand the matching process we must examine the benefits and constraints that brain dynamics imposes on experience. Models of blank slate neuronal network assume that there are a number of randomly connected neurons. The assumption is that brain circuits can be altered by any input. An example from artificial intelligence shows us the flaws of this approach. Classical AI research–particularly the branch known as connectionism, the basis for artificial neural networks–adheres to the outside-in, tabula rasa model. This prevailing view was perhaps most explicitly promoted in the 20th century by Alan Turing, the great pioneer of mind modeling: “Presumably the child brain is something like a notebook as one buys it from the stationer’s,” he wrote.

Artificial neural network designed to “write” inputs into a neural circuit are often ineffective because each new input changes the circuit’s dynamics and connections. Plasticity is a characteristic of the circuit. There is one problem. The AI system can alter the connections in its networks to learn, but at any moment, it can wipe all stored memories. This bug is known as catastrophic interference. It is an event that a real brain never experiences.

The inside-out model suggests that self-organized brain network should resist such perturbations. They should still show plasticity when necessary. This balance is achieved by the large differences in the strength of the connections between different groups of neurons. The continuum of connections between neurons is the continuum. While most neurons are only weakly connected to each other, a small subset of neurons has strong connections. The small, but highly connected, minority is always alert. It is quick to fire off information, shares it quickly within its own group, resists any changes to the neuronal circuitry, and does not hesitate to share it with others. These elite subnetworks, often called a “rich club”, are well-informed about neuronal events throughout brain because of their many connections and high communication speeds.

The hard-working rich club makes up roughly 20 percent of the overall population of neurons, but it is in charge of nearly half of the brain’s activity. Contrary to the rich club, the majority of brain’s neurons–the “poor club”, or neural “rich club”–fire slowly and are not connected to other neurons. They are highly plastic and can physically alter the connections between neurons. This is known as synapses.

Both rich and poor clubs are important to maintain brain dynamics. The members of the ever-ready rich club respond similarly to different experiences. They can provide quick, efficient solutions for most situations. Our brains are able to make accurate predictions about the unknown. This is not because we can recall it, but because they always make a guess about a new event. Because it always connects the new to the past, nothing is entirely new to the brain. It is general. Even an inexperienced brain can have a large number of neuronal trajectories, which allows it to match events in the world with preexisting brain patterns. This is possible without the need for extensive reconfiguration of connections. A brain that constantly remakes itself would not be able to adapt to rapidly changing events in the outside world.

But the role of plastic, slow-firing-rate neuron is also important. These neurons are activated when there is an important event in the organism and need to be recorded for future reference. They then use their vast reserves to detect subtle differences between things and alter the strength of connections to other neurons. After seeing different types of canines, children learn the meaning of “dog.” When a youngster sees a sheep for the first time, they may say “dog.” Only when the distinction matters–understanding the difference between a pet and livestock–will they learn to differentiate.

Graphic shows an experiment where a rat’s neuronal firing pattern indicates the direction of its planned route to a reward.
Credit: Brown Bird Design

Cognition as Internalized Action

I was not attempting to create a theory that would be in contradiction to the outside-in framework. My work on the self-organization and rhythmic firing of neuronal population in the hippocampus was decades ago. It was only then that I realized that the brain is more concerned with itself than what is going on around it. This realization prompted me to create a new research agenda in my lab. Together with other research, our experiments revealed that neurons spend most of their activity sustaining the brain’s constantly changing internal states, rather than being controlled through stimuli.

During natural selection, organisms adapt and predict the outcomes of their actions in ecological niches. Complex connections and neuronal computations become more interconnected between sensory inputs and motor outputs as brain complexity increases. This investment allows for the prediction of planned actions in complex and changing environments, and at long time scales in the future. Higher-level brains can also be organized to allow computations to continue even if sensory inputs are temporarily lost and animals’ actions stop. Because brain activity is so important to “seeing”, even if you close your eyes, it’s still possible to know where you are. This mode of neuronal activity is disengaged and allows access to a virtual world that you have created or experienced, which can be used as a gateway to many cognitive processes.

Let’s take an example of a disengaged mode in brain operation, based on our research on the brain’s temporal region, which includes the hippocampus and the nearby entorhinal cortex. These structures are involved with multiple aspects navigation (the tracking and tracking of direction, speed and distance traveled, environmental boundaries and so forth).

Our research is based on the most advanced theories about the functions of the hippocampal systems, such as John O’Keefe’s Nobel-winning discovery. O’Keefe discovered that the spatial location of an animal and the firing of hippocampal neurons during navigation coincide with the location of the animal. These neurons are also known as place cells.

As a rat moves through a maze of cells, distinct groups of place cells activate in a sequence chain that corresponds to its current location. This observation suggests that constantly changing sensory inputs from the environment control the firing of neurons. This is consistent with the outside-in model.

But other experiments, including those in humans, have shown that these networks are also used to manage our internal worlds. They keep track of personal memories and plan future actions. It becomes apparent that cognition can be viewed from the inside out. Navigation through a physical space or a landscape is handled by identical neural mechanisms.

15 years ago, my lab began to investigate the mechanisms of spatial navigation in the hippocampus. This was to contrast the inside-out and outside-in frameworks. In 2008 Eva Pastalkova, a postdoctoral fellow, and I trained rats to alternate between the left and right arms of a maze to find water. At the beginning of each traversal of the maze, the rat was required to run in a wheel for 15 seconds, which helped to ensure that memory alone of the maze routes, and not environmental and body-derived cues, allowed it to choose a particular arm of the maze. As predicted by O’Keefe’s spatial navigation theory (hippocampal neurons “represent” maze corridors and wheel), a few neurons should fire at each spot regardless of whether the rat is in the corridors, or the wheel. However, if neurons’ firing is generated internally by brain mechanisms that support navigation and memory, then the duration of neuronal firing should not vary at any location, including inside the wheel.

These experiments were beyond the reach of outside-in explanations. None of the hundreds of neurons recorded were fired continuously during the wheel’s running. Instead, many neurons fired in a transient sequence.

These neurons cannot be called place cells because the animal’s body did not move while running at the same location as the wheel. Furthermore, it was impossible to distinguish between the firing patterns of individual neurons and neurons active while the rat was moving through the maze.

When we sorted individual trials based on the future choice of left or right arm for the rat, the neuronal tracks were unique. These distinct trajectories eliminated any possibility that these neuronal sequences could have arisen from counting steps, estimating muscular effort, or other unrecognized feedback stimuli. The unique neuronal trajectories made it possible to predict an animal’s maze arm choice starting at the moment it entered the wheel. This was in addition to keeping the previously visited arm in mind throughout wheel running. To receive their rewards [see graphic above]., the animals had to choose the correct maze arm each time.

These experiments led us to the conclusion that the neuronal algorithms we use to walk to the grocery store govern internalized mental travel. Disengaged navigation guides us through the sequence of events that make up our personal recollections, also known as episodic memories.

In truth, episodic memories go beyond recollections of past events. They allow us to look ahead and plan for the future. They act as a “search engine”, allowing us to explore both the past and the future. This realization also suggests a broadening of nomenclature. These experiments have shown that place cell activity progressions are internally generated by preconfigured sequences for each maze corridor. Multiple designations, same mechanism – they can be called memory cells, planning cells, or place cells depending on the circumstances.

Further support for the importance disengaged circuit operations is from “offline” brain activity when an animal is doing nothing, consuming reward or sleeping. After a maze-running adventure, a rat rests in its home cage and generates short, self-organized neuronal trajectories. These sharp wave ripples, as they are known, occur in 100-millisecond time windows and reactivate the same neurons that were firing during several seconds of maze running, recapitulating the neuronal sequences that occurred during maze traversals. Sharp wave-ripple sequences are vital for normal brain function and help to form long-term memories. In fact, severe memory impairment can be caused by altering sharp wave-ripple events through experimental manipulations or diseases.

Clever experiments have been conducted in humans and animals over the past ten years. They show that time-compressed ripples events are a subconscious process that creates new or real alternatives to making decisions about the optimal strategy, building novel inferences, and planning ahead for future action without the need to test them with a real exploit. Our thoughts and plans are deferred actions and disengaged brain activity, which is an essential, active brain operation, are in this sense. The outside-in theory, however, does not attempt to assign a role for the disengaged brain at rest or in the midst sleep.

Graphic shows neuron firing pattern when a rat runs on a track. A preview fires before the run and a reversed pattern after.
Credit: Brown Bird Design

The Meaning of Inside Out

The inside-out approach has many practical applications, in addition to its theoretical implications. It could be used to help develop better diagnostic tools for brain diseases. The current terminology is often inadequate to accurately describe the biological mechanisms behind mental and neurological disorders. Although psychiatrists are aware of this problem, they have struggled to understand the pathological mechanisms and how they relate to symptoms and drug responses.

The inside-out theory is another alternative to the most common connectionist models for conducting AI Research. They could be replaced by models that are self-organized and learn by “matching” their circuitry, rather than constantly changing it. This could allow machines to be disengaged from electronic sensors, creating new forms of computation that are similar to internal cognitive processes.

In real brains, neural mechanisms that operate through disengagement with the senses go hand-in-hand with mechanisms that promote interaction with the environment. All brains, no matter how complex or simple, follow the same basic principles. The essence of cognition is disengaged neural activity, calibrated simultaneously with outside experience. This knowledge would have been invaluable to me when my medical students asked me legitimate questions that I didn’t answer quickly enough.

This article was originally published with the title “Constructing the World from Inside Out” in Scientific American 326, 6, 36-43 (June 2022)

doi: 10. 1038/scientificamerican0622-36



    Gyorgy Buzsaki is a systems neuroscientist whose work has focused on the ways memories form and how brain rhythms segment neural information to support cognition. He was a co-recipient of the 2011 Brain Prize from the Lundbeck Foundation. Buzsak

    Read More