In neuroscience research, we spend a lot of time trying to make sense of brain images. I spend a good part of most of my days answering the question, “what space am I in?” That’s the cool-kid lingo for talking about what kind of image you are working with. There’s “native space”, “corrected space”, “standard space”, “surface space”, “anatomical space”, “functional space”, …, the list goes on and on. For most students that work with brain imaging data, figuring out how to think clearly about all these spaces is difficult, headache-inducing work. But over time you begin to build an intuition about how to deal with spaces, and soon you are moving effortlessly between them.
An image space consists of dimensions broken up into units of size. Within that space, the image itself is composed by adding pixel intensity at the location of each unit. Therefore, an anatomical brain image might be a volume consisting of 3-dimensions.
We can make different types of tissue have different brightness by being clever with our measurement techniques. For example, the anatomical image is sensitive to differences in the fat and water content of the tissue. Other images may be more sensitive to certain kinds of damage, or blood flow, or oxygen content, or many other things.
One of the trickiest bits of analyzing these images is getting all these spaces to line up. For instance, you might be interested in seeing where some brain activity is occurring. Images that are sensitive to brain function are not very sensitive to brain anatomy, making finding the specific location of the activity difficult.
To know exactly where the activity is occurring in an individual, we need to acquire a good anatomical image along with the functional image, and then try our best to line up the two. Another common challenge is that researchers want to analyze multiple research subjects so that our statistics are valid and generalizable to the population. But people have different- looking brains! We need to find some good way of lining up the images of everyone in our sample. Finally, we want our results to have some correspondence to other results in our field of study, so we need some way of making our images understandable with reference to a standard. This makes it possible to do things like meta-analysis across studies.
The strategies and tools that help us take on these challenges are legion. They don’t always work well, and they aren’t always easy to understand. A Guitar Hero aficionado, going to bed after a sick sesh of jamming, may close their eyes and find, to their amazement, that the colored notes are still scrolling at them in an endless stream. My experience learning how to manipulate and move between image spaces was similar. I would close my eyes at the end of a long day and see images flipping, rotating, expanding, warping, and changing contrast. Image spaces would enter my dreams and permeate my thoughts.
Somewhere in this process, I started thinking of myself — all the weird constellation of traits that I think of as me — as one of these spaces. Let’s call it my “native space,” since this is the term imagers use to refer to an individual’s own brain image. I realized that I transform my own native space in all sorts of ways to fit in different contexts. My brain calculates all the same “image transformations” that I do at work all the time. Except my brain does it intuitively with everything happening under the hood. By thinking about myself in terms of image spaces, I’ve gained some deep insights about myself and my relationships. I believe that these insights have made me a more thoughtful and compassionate person.
One of the first lessons that I took away was thinking about my identity as linear transformations between different types of contexts. A linear transformation is a function that preserves the structure of a vector space. That means that lines that were parallel stay parallel. In neuroscience research, it is very common to use a simple kind of linear transformation called a “rigid body transformation” to align two brain images from a single participant. In 3D images, rigid body transformations allow for 3 “translations”, which are shifts up or down, side to side, or forward and backwards, and 3 “rotations”, which are turns around the three axes.
For metaphor building, let’s also reduce my personality to three dimensions. Since it’s convenient, I’ll use the three dimensions of personality that the largely discredited and dastardly psychologist, Hans Eysenck, came up with using factor analysis. To be perfectly clear, I’m not advocating for the reality of these three personality dimensions; they’re just useful here for demonstration. His personality factors are dimensions of 1) extraversion-introversion, 2) neuroticism-emotional stability, and 3) psychoticism-social normality.
Now imagine that on a given day, my personality falls within some small shape of these three dimensions. The “intensities” might represent the likelihood of me representing a particular set of dimensional weights. Day to day, this shape stays relatively constant, but my average extraversion, neuroticism, and psychoticism move around. I identify myself based on this shape.
One area where I use these rigid body transformations is when I am correcting images from misalignments caused by a research participant’s motion over time. In some types of scans, we collect an image of the whole brain about every 2 seconds for 10 minutes or longer. When we put a participant in a scanner, we have them lay on a moving tray that will go into a tight bore surrounded by the powerful and loud magnets. Even though we do our best to make participants comfortable, even going so far as to tuck them in with a warm heated blanket, the experience can still be disconcerting for first-timers. Ideally, participants would not move at all during the scan. Even though we place pads around the participant’s head in the MRI scanner, it is still likely that they will move a few millimeters here and there over a long scan. That is problematic for doing statistics on activity over time because a few millimeters are sometimes the difference of “inside where the brain was” or “outside where the brain was.” To deal with this problem, before I do any statistics, I try to do a rigid body transformation of every frame to line up all the images to where the brain was at the beginning. Sometimes, the subject sneezes. That kind of motion is too great and likely to terrorize my analysis. In those cases, we simply throw out the bad frames in a procedure called “scrubbing.”
I have a very similar self-referential process to track myself from day to day. Even though my personality-shape changes slightly over time, most of the time I remain convinced that I am basically the same person. Sometimes, I don’t recognize myself because I fall outside of this shape. Maybe one day, I’m feeling especially “psychotic”, way outside of my normal shape. These days may be filled with accusations from my partner (“who are you?”) and apologies (“I wasn’t being myself!”).
If someone I know and am close to is acting a lot differently than I expect in a negative way, I think it’s good to provide that feedback, but I’ve also learned to trust that it will pass, just like a sneeze. Huge, sudden, permanent translations in personality are unlikely. Instead of worrying about whether I’ve missed some latent, nasty trait for the past ten years, I just scrub.
These small perturbations in my own home are easy to keep track of and continue to feel “Ben-ly.” But I’ve also discovered that I tend to transform my personality in systematic ways to meet the needs of other contexts. For instance, I try extra hard to be extraverted in professional and social events, and I find myself naturally less psychotic around Christmas. These types of transformations may need more degrees-of-freedom than the small or random shifts in my “native” space.
In brain imaging, a situation arises frequently where you have different types of brain images, but all coming from the same individual. Sometimes the MRI sequence slightly deforms parts of certain images or the sizes of the images differ.
I talked before about translating and rotating to fit two shapes together. There are also two other types of linear transformations: shearing and stretching (also called scaling).
These are useful tools when two brain images are from the same individual. They do represent the same brain, after all, so we don’t expect the differences between the images to be too extreme. It takes some work to find an optimal “affine transformation matrix” that aligns one image to another image space, with its different dimensions and contrasts. There are lots of tricks out there to help with this, which work with varying degrees of success.
It is no different in my own life. Even though I’m still me, it helps to apply small transformations appropriate to the various “spaces” in my own life. It takes some time to learn a personality transformation that is optimal for a certain kind of space. However, one cool thing about finding a transformation between two spaces is that once you find it, you can just use that exact same matrix if you ever need to bring something from one space to the other space again in the future. It’s helpful to solve a transformation for a space once rather than needing to recalculate it every time. Let’s say I’m trying to transform my “native” personality to “party” personality. Parties will vary slightly, and so some tweaks might be ideal, rather than relying on the exact same transformation each time. Even so, you save a lot of work by applying a known transformation. For example, I might have learned that to best transform myself to “party space” I must try to be a lot more extroverted and pro-social (e.g. shearing and scaling myself).
In imaging, we can take a hybrid approach. Find the best transformation once, and then start there before making any more image-specific adjustments. One benefit to finding the transformation matrix is that you can take the inverse of that matrix in order to move backwards from the new space (e.g. “party” space) to the original space (“native” space). This is an important concept because our alignment often works better if we go from the lower resolution image to a higher resolution image at first. Taking the inverse of that transformation that maps a low-resolution to a high-resolution image is often a better strategy than trying to find the transformation from high-res to low-res using fitting algorithms.
This principle felt unintuitive to me when I learned it, and it feels unintuitive when I apply it to my personality. When I think of what I need to do to transform into party-Ben, it makes sense that I should think about what I need to do differently than normal to fit in at the party. But my “native” space is high-resolution. I know it pretty darn well. Party space is a little hazier. What if I try to go from low-res to high-res and ask “how does the party need to change to perfectly fit my native self?” Asking this question feels good because it affirms my sense of identity and that I don’t need to change anything about myself for the party. It asks hypothetically how the party can change for me, which is validating and takes me out of the natural tendency to feel like I am the one who automatically needs to change for the world. It also helps me see the differences a little more clearly. The concept of a general party is kind of vague and undetailed. But a party space just for me?! I can give you lots of details about how I would change any given party. If I want, I can use that information as a clear indicator about how to best immerse myself in the party environment. For example, at a recent party, all of my friends decided that it would be extremely fun to start singing together. I am kind of shy, especially about singing, and I don’t really like doing it, especially in public. But by focusing specifically on my default personality’s mismatch with the current environment, I identified a specific way that I could better immerse myself in the party. I ramped up the extraversion, tapped into my love of music, tried to find some courage, and started singing.
In imaging, finding these linear transformations is usually not too difficult, but there are more difficult situations. These usually involve generalization. For example, I often want to aggregate brain imaging data in a study so that I can do group-level statistics. This poses a significant challenge. Everyone has a different brain; How can I say that one person’s brain activity is more or less in the same spot as someone else’s brain activity? One common approach is to bring everyone’s brain image into a “standard space.”
One of the really useful things about a standard space is that a lot of scientists use only a handful of the same standard brains and coordinate systems to describe scientific results. One common standard space is called Talairach space, named after the neurosurgeon, Jean Talairach, who originally developed the coordinate system. His atlas was based on a single post-mortem dissection of a human brain. In our social world, it is also useful to have a “standard space” of the cultural expectations for personality and behavior. Sometimes we look to “influencers,” which, like the Talairach atlas, are based on a single individual that we want to conform to. But using single exemplars for cultural expectations can be dangerous. Often single examples have weird quirks. For the same reason, in imaging, the field has mostly moved to using brain templates made by averaging the brains of many humans. The most common standard space in this category is a set of templates made by the Montreal Neurological Institute.
Just like in brain imaging, there may be different pockets of people that use different standard spaces. When we meet a stranger, we start from our standard space and try our best to get back to their native space by going in the opposite direction. We start from our idea of what is normal and use that to try to find deviations from our expectations in the stranger. Since these kinds of transformations are non-linear, in order to do this, we need an accurate “warp field.” We can think of this warp field as our map of how the individual differs from the standard space.
The first lesson here is that your image of others and their image of you starts in a standard space. I think it is useful to remember that everyone’s model of everyone else is tenuous, unless you know them well. I also have the intuition that some people build accurate representations of strangers much faster than other people. Just like understanding a good algorithm, explicitly defining and practicing these skills could really benefit our social relationships.
In imaging, one analysis step that often helps with improving transformations is to do “bias field correction.” When you acquire an image in a not perfectly homogenous MRI environment, your images can be corrupted by a low-frequency, smooth signal that overlays the actual image. Like an inhomogeneous MRI environment, my own brain is full of distorting effects that can lead to systemic biases in how I view other people. A good first step, when I am trying to judge someone new, is to examine my picture of this person for these sorts of low-frequency artifacts. If you are a very careful imager, you might actually measure the bias field inhomogeneity itself, rather than just trying to identify and remove the low-frequency bias in your image. One interesting way to measure your own biases in judging others is through the implicit association test (IAT) [https://implicit.harvard.edu/implicit/takeatest.html]. This test is useful for identifying unconscious implicit beliefs that you may have and not be aware of that influence how you judge others.
Another effective strategy for finding good non-linear warping is to “smooth” the images that you are trying to align. You can think of smoothing as blurring or smearing the intensities of an image, which makes edges less distinct. I like to keep in mind that when I’m first trying to understand someone, it’s good to try to focus on the broad patterns, rather than single events, behaviors, or single sentences they say. Emotionally charged events, which can be either positive or negative, are like points of high contrast in an image. Your attention is automatically drawn to them to find information, but they are often not the most useful features for non-linear warping and can often lead to mistakes.
Another major insight for me was that, given the same amount of time, I will do a worse job at understanding somebody the further they are away from my standard space. In fact, a major problem with using standard spaces in brain imaging is that images that are further away from the standard brain will inevitably have a worse transformation. They simply don’t match up well.
Instead of forcing every brain into the same standard space, some researchers now advocate for a more nuanced approach. An alternative to transforming between volumes is to parcellate the brain into smaller regions on an individual basis, and then doing statistics on those parcels. This requires more work and is also dependent on having good parcels. Neuroscientists like to argue about which parcellations are the best. As computation becomes faster and cheaper, more scientists are doing analyses using multiple different kinds of parcellations.
The interesting thing about parcellations is that it reduces an individual to a small number of regions-of-interest (maybe 120 points instead of 100,000 points in the original image), and yet if we have a good system for parcellating the brain in native space, this can often be more useful than analyzing brains in standard space. The approaches are inherently different. Transforming images to a standard space boldly asserts that we can identify small details about your own brain in that standard space. The parcellation approach is humbler, giving up on small details in favor of learning a structure. One of the advantages is that you don’t see the standard space image and accidentally start believing that it is real. It can be tempting, when you see a very brain-looking brain, to forget that it is highly modified and wasn’t actually the brain that you imaged.
The parcellation approach is akin to memorizing a list of some number of questions that you think represents a person’s personality and asking the list of questions to everyone you meet. This is more work but is probably helpful for getting some useful information about an individual that can help understand a person within a group context.
These strategies for generalization are very useful for finding broad patterns. However, sometimes our methods are problematic when we have dynamic or longitudinal data. When we have timeseries data, we sometimes want to apply a single transformation to each of the images collected over time for consistency and to save time. This seems principled enough. If you find a good transformation between the first image in the time series and the image that you want to align it to, you should be able to apply that same transformation to all the images in the time series. A similar principle holds for longitudinal or multi-session experiments. For example, I might have two imaging sessions, separated by a week or so, where I collect functional MRI data, but I only collect an anatomical image in one of the sessions. If I find a transform to the first session and keep all the imaging parameters the same, it might be tempting to just use the same transform for the second dataset.
In practice, this doesn’t always work very well. There are small changes over time that can make the transform you found earlier obsolete. This is usually less of an issue from images in the same session and more of an issue when imaging over multiple sessions. One thing that I often notice here is that sometimes using a transform that you previously found for dynamic data will almost always work. Because of that, when it doesn’t work, it can take you by surprise! It can be easy to miss.
In social life, this happens most often when tracking the personality of the people you have relationships with over long periods of time. Some of the most famous personality researchers, Costa and McCrae, did a series of experiments on lifespan changes in dispositional personality traits. They found extremely high reliability in personality scores across even relatively long periods of time and famously concluded that personality becomes “set like plaster” by the age of 30. Just like in imaging, the non-linear transformations you use to understand someone’s personality relative to some cultural standard almost always work. But if you look at Costa and McCrae’s data [https://doi.org/10.1037/10143-002], you’ll notice that while this pattern is true on average, some folks’ personalities change dramatically. In a few cases, there is even a negative correlation with their previous personality traits. I think it’s useful to keep in mind that while the transformations you use to understand the people in your life are mostly consistent, it’s still possible for them to change.
Thinking about the principles of transformations in imaging has changed the way I think about my own personality and how I make judgements of others. Overall, I think that these changes have been positive. I avoid treating individuals as good examples of standard spaces. I try to smooth over the traits I think I see in new acquaintances and try to first focus on the broad strokes. I am more careful about my understanding of someone in cases where someone is a lot different than the norm. I use specific questions as a tool to know someone better. I keep a lookout for when my mental models of other people in my life are no longer working well. I do my best to know my own biases. I don’t get too worried when the people I know well (or I) am acting or feeling a lot different than normal. And I’ve gotten into the habit of thinking explicitly about the ways I want to adapt myself to different kinds of environments without feeling like it’s an obligation by thinking about transforming the environments around me to my own native space.