Saturday, June 24, 2017

Please Redirect Your Questions to AndysBrainBlog.com!



Starting today, any questions that are posted here I will redirect to andysbrainblog.com. What this means is that I'll reply to your comment by saying "Please post this at andysbrainblog.com," along with a link to the page related to your question.

The comment section of the new website has Disqus, which allows you to post pictures and video. This gives me a better idea of whatever issue you're having with your data. Disqus also makes it easier for me to moderate the comments and to create threads, which I hope will make the whole question-and-answer process easier.

This is also part of my effort to transition everyone to the new website. I've been thinking about automatically redirecting any URL request for this site over to the new site, so be prepared if that happens sometime in the near future.

Tuesday, April 4, 2017

New Website: AndysBrainBlog.com

Le style c'est l'homme. No doubt. But the converse is also partly true: L'homme c'est le style. Because we have a gift for writing in a certain way, we find ourselves, in some sort, becoming our way of writing. We mould ourselves in the likeness of our particular brand of eloquence.

-Aldous Huxley, The Perennial Philosophy



Last summer I made an abortive attempt to create another website which would serve as my professional front, while continuing to post here as I saw fit. Evidently I saw fit to post here virtually all of the time, to the neglect and eventual death of the other website. Partly this was because I was more comfortable with the Blogger interface here, and partly because I didn't like the Wordpress interface there. And so the new website fell stillborn from the press, and everything on this site continued much as before.

I've made another attempt, now with the platform Squarespace, and I believe that this time the change is permanent. The new address is: www.andysbrainblog.com. If you are already following me on Google Plus, the posts will still show up in your feed; comments and posts have been transferred over, so that nothing has been lost - save for the pruning and deletion of some older posts which now bring me little nostalgia and much embarrassment.

I have been together with this blog for five years, far longer than any other relationship in my life. There will always be something oddly touching about its black and orange color scheme, the Arial font, the haphazard way of cobbling certain posts together. Reading the earlier entries, one can see my mind's gradual development over the past half decade, the progression of what I thought about and what I cared about, and how I arrived at this moment, the child the father of the man.

But it will not be deleted. I will not push a button and have it vanish into the electricity. Rather, it will remain embalmed here for all eternity, its noble corpse propped up into a regal position altogether fitting of its legacy of grandeur and loyalty. And on its pedestal, these words appear:

No friend has served me, and no enemy wronged me, whom I have not repaid in full.


In pace requiescat!

Friday, March 31, 2017

NIH Talk: Distinct Regions of mPFC Process Pain and Cognition



Grown-ups like numbers. When you tell them about a new friend, they never ask: "What does his voice sound like?" "What games does he like best?" "Does he collect butterflies?" They ask: "How old is he?" "How many brothers does he have?" "How much does he weigh?" "How much does his father make?" Only then do they think they know him. If you tell grown-ups, "I saw a beautiful red brick house, with geraniums at the windows and doves on the roof..." they won't be able to imagine such a house. You have to tell them, "I saw a house worth a hundred thousand francs." Then they exclaim, "What a pretty house!"

--Saint-Exupery, The Little Prince


I often have vivid fantasies about how my talks will be received: The audience will laugh at my jokes; listen in attentive silence about the obstacles I overcame to carry out my research; gasp in astonishment as I reveal my big finding which will change the field forever. And, at the end of my talk - concluded with a germane and heartfelt anecdote which ties everything together - an ocean-like roar of applause and yells as I am lifted up high on a chair and carried through the streets with great honor. The men shake my hand vigorously and the ladies kiss me on the cheek. "Hats off, gentlemen!" says the town crier, "A genius!"

For some reason, and much to my dismay, reality fails to match my heart's desires. The jokes and asides feel flat and fall stillborn from my mouth. The background of my study feels less like an epic and more like reciting a laundry list. (I swear it sounded much more interesting when I was rehearsing it to myself.) Any small issue with the projector cutting out or with my Powerpoint animations failing to work, in the moment feels as embarrassing and indecent as being caught with my fly unzipped.

But I keep going nonetheless, holding out hope to someday achieve that perfect talk combined with the perfect moment. The ultimate trade awaiting the ultimate practitioner.

In any case, something to strive for.

Tuesday, March 28, 2017

Video Introduction to ROI Analyses



About forty years ago certain persons went up to Laputa, either upon business or diversion, and upon their return began to dislike the management of everything below, and fell into schemes of putting all arts, sciences, languages, and mechanics upon a new foot. To this end they procured a royal patent for erecting an Academy of Projectors in Lagado. Every room hath in it one or more projectors. The first man I saw had been eight years extracting sunbeams out of cucumbers.

--Swift: Gulliver's Travels, Part III, chs. 4-5


I'm updating my videos on fMRI basics, starting with ROI analysis. This is low-hanging fruit, yes, but delicious fruit, fruit packed with nutrients and sugars and vitamins and knowledge, fruit that will cure the scurvy of ignorance and halt the spreading gangrene of frustration.

In these videos you will observe a greater emphasis on illustration and analogy, two of the most effective ways to have concepts like ROI analysis take root inside your mind; to make them have a real, visceral presence when you think about them, and not to exist merely as words that happened to impinge on your retina. These videos take longer to make, but are all the more rewarding. And if they help you to think differently than you did before, if they help you, even without my knowing it, to see the world as I understand it, then I will have taken a significant step toward fulfilling my purpose here on this earth.

Sunday, January 29, 2017

Are All FMRI Results Wrong?




False positive rates in science have been an issue recently; and although we all had a good laugh when it happened to the social psychologists two years ago, now that it's happening to us, it's not so funny.

Anders Eklund and colleagues published a paper last summer showing that cluster correction - one method that FMRI researchers use to test whether their results are statistically significant or not - can lead to high false positive rates, or saying that a result is real, when actually it is a random occurrence that looks like a real result.

Their calculations showed that about 10% of FMRI studies are affected by this error (http://tinyurl.com/jaomsgs). However, keep in mind that even if a study is at risk for reporting a false positive, doesn't mean that their result is necessarily spurious. As with all results, one must go to the original study and take into account the rigor of the experimental design and whether the result looks legitimate.

These flaws have been addressed in recent versions of AFNI, an FMRI software package. The steps to use these updated programs can be found on the blog here: http://tinyurl.com/j5vafsb

Thursday, January 19, 2017

Commentary on Cluster Failure: Inflated False Positives in FMRI



Why did the old Folly end now, and no later? Why did the modern Wisdom begin now, and no sooner?

-Rabelais, Prologue to Book V


=======================

Academia, like our nation's morals, seems to be forever in peril. When I first heard of the replication crisis - about how standards are so loose that a scientist can't even replicate what he ate for breakfast three days ago, much less reproduce another scientist's experiment - I was reminded of my grandpa. He once complained to me that colleges today are fleshpots of hedonism and easy sex. Nonsense, I said. Each year literally tens of students graduate with their virtue intact.

Nonetheless, the more I learned about the replication crisis, the more alarmed I became. You know a crisis is serious when it has its own Wikipedia page, and when its sister phrase "questionable research practices" gets its own acronym: QRP (pronounced "Quirp"). These practices range from data "peeking" - ogling a p-value before she's half-dressed - to fabricating data, one of academia's gravest sins. (Apparently grant agencies have gotten very picky about researchers lying about their results and wasting millions of dollars.) One of my friends suggested campaigning for a War on QRPs, with scientists in white labcoats stamping out QRPs like cockroaches. Who knows - someday QRP Exterminator may become as respected and honorable a position as Lab Manager.

One such QRP was recently flushed out of hiding in an article by Eklund and colleagues published in the journal PNAS (pronounced "P-NAS"), with the klieg lights thrown on cluster correction, the most common FMRI multiple comparison technique. If you're in a neuroimaging lab, or if you've read about FMRI in a magazine, you've come across these clusters - splotches of reds and yellows, color-coded to show the strength of the result. They look attractive, especially on glossy magazine pages, and they make for good copy.


Relative popularity of different correction methods. The majority use cluster correction; a smaller percentage use voxel-based thresholding (i.e., correcting for the total number of voxels in the brain); and the remainder do not use correction.  Men who don't use correction claim that it feels more spontaneous and natural. Adapted from Woo et al. (2014)




There is another reason for the popularity of clusters: As each FMRI dataset contains tens or hundreds of thousands of voxels, and as each voxel is similar to its neighbor, it is useful to test whether a cluster of voxels is significant instead of each voxel individually. This increases the sensitivity of detecting activation, although at the cost of reduced spatial sensitivity. If I point to a cloud of smoke in the distance you will assume there is a fire somewhere, although seeing the cloud itself gives you only a vague idea of where or how big the fire is - or, for that matter, whether its the result of a dozen individual campfires, or of a single inferno.


Problems with Cluster Correction

The authors don't take issue with cluster correction per se; rather, they claim that the assumptions behind the technique, and how the correction is used, can lead to inflated false positives. In other words, using cluster correction makes one more likely to say that a cluster represents a true activation, when the cluster is actually noise. The assumptions they question are:

1) That spatial smoothness is constant over the brain; and

2) That spatial autocorrelation looks like a Gaussian distribution.


Spatial smoothness is in fact not constant; it varies over the brain. Midline regions such as the precuneus, anterior cingulate cortex, and prefrontal areas show more spatial smoothness than the the outer regions - possibly because the midline areas of cortex, with the gray matter of the hemispheres in close proximity to each other, have similar structure and similar timecourses of activity which are averaged together. The underestimation of spatial smoothness in these areas can lead to inflated false positives, which may be why midline areas (such as the anterior cingulate cortex) turn up as a significant result much more often than other areas.

Effects of spatial smoothing at different smoothing sizes, or kernels. The larger the smoothing kernel, the more that adjacent voxels are blurred together, resulting in greater similarity of spatial structure and also of the timecourse of activity.


It is the second assumption, however, which is the main cause behind the increase in false positives demonstrated in the Eklund paper. The idea behind spatial autocorrelation is simple: A voxel is more similar to its immediate neighbors and less similar to voxels farther away, the similarity falling off in concentric spheres around a given voxel. This correlation between each voxel and its neighbors is traditionally modeled with a Gaussian distribution - the same bell-shaped curve that models most distributions, such as IQ or height - when in reality the autocorrelation distribution has much thicker tails:

Comparison of the Gaussian distribution (green) with the empirical ACF (black) and mixed model of a Gaussian and exponential distribution (red). Note the increasing discrepancy between the Gaussian distribution and the empirical distribution around 5mm, with the thicker tails of the empirical distribution suggesting that more distant voxels are more correlated than is assumed under traditional clustering methods.

When the authors ran simulations on large datasets under different experimental conditions (e.g., block design and event-related), they found that this assumption of a Gaussian shape led to false positives far above the expected 5% rate across all the major software packages (SPM, AFNI, and FSL). The commonly used p = 0.01 threshold for the individual voxels of the cluster led to much higher false positive rates than a more stringent p = 0.001 threshold. (Notably, permutation testing, a nonparametric method that does not rely on assumptions of normality, did not show any substantial difference in false positives under any conditions.) These inflated false positive rates led to the following figure, which was picked up by the popular media and discussed in a fair, evenhanded way which emphasized that although false positive rates could be increased in some cases, it would be hasty to conclude that all neuroimaging results are wrong.*


Figure 1 from Eklund et al (2016). Simulations on an open-access dataset for different block and event-related designs. Upper left: Cluster-defining threshold (CDT) of p = 0.01 uncorrected; upper right: CDT of p = 0.001 uncorrected; bottom: voxel-level inference. Black line: Expected false positive rate of 5%. A threshold of p = 0.01 leads to inflated false positives across all of the major software packages (SPM, AFNI, and FSL), while a threshold of p = 0.001 leads to clusters closer to the expected 5% false positive rate. Voxel-level inference is slightly conservative in all cases. Permutation testing (circled in red), a nonparametric method, does as expected under all conditions.

*Or not.

Solutions

There are three solutions to this problem; solutions to make neuroimaging respectable again, to cease being laughed at as cranks, charlatans, raiders of the public fisc, and attention-seeking, unscrupulous bloggers.

One solution is to use voxel-wise thresholds; although, as shown in the Eklund paper, these are often far too conservative, and can lead one to conclude there are no results when they actually do exist. I, for one, cannot stand the thought of those poor results slowly suffocating in their little gray coffins, alive but unable to signal for help. With voxel-wise thresholds we stop up our ears with wax; we think we hear the dim scratching of nails, the whimpers for succor, but dismiss them as figments of a fevered imagination; and we leave those tiny blobs and their tiny blob families to know that their last moments will be horror.

The second, more humane solution is to use more conservative cluster-forming thresholds, such as p = 0.001 uncorrected, in conjunction with more accurate modeling of the spatial autocorrelation. I recommend updating your AFNI binaries to the latest version (i.e., the beginning of 2017 or later), and using the  -acf option in 3dFWHMx. For example:

3dFWHMx -mask mask+tlrc.HEAD -acf tmp.txt errts+tlrc.HEAD

Which will generate parameters for a more accurate estimation of the empirical ACF; you will see something like this:

++start ACF calculations out to radius = 37.68mm
+ACF done (0.74 CPU s thus far)
11.8132 13.748 12.207 12.5629
0.260939 6.36019 16.6387 19.1419 

The first line of numbers are the traditional smoothness estimates in the x-, y-, and z-directions, as well as a weighted average of them (bolded). The second line of numbers are the a, b, and c parameters for the ACF model,

a^(-r*r/(2*b*b))+(1-a)^(-r/c)

With the last number being an updated estimate of the smoothness. These are then input into 3dClustSim:

3dClustSim -acf 0.26 6.36 16.64 -mask mask+tlrc.HEAD -athr 0.05 -pthr 0.001

The difference between the two smoothness estimates leads to substantial differences in significant cluster thresholds. In this case, the traditional estimates lead to a cluster threshold of 668 contiguous voxels, while the ACF version gives a threshold of 1513. This was using an initial cluster-defining threshold of p = 0.01, which can lead to such large differences; more stringent thresholds (e.g., p = 0.001) have less of a difference, but still a significant gap between them.

The third solution is to use non-parametric methods, such as permutation tests; which, since they do not rely on assumptions of normality, are immune to the autocorrelation assumptions listed above. Some popular tools include SPM's SnPM (Statistical non-Parametric Mapping); FSL's randomise; and Eklund's BROCCOLI. Out of the three, I've only used randomise, and found it straightforward to use (see the tutorial here).


Conclusions

Have we been profiting too long from dubious methods in neuroimaging? Have we become lax and complacent in our position as a period science, knowing that no matter how poorly designed the experiment is, no matter how ludicrous the question we are investigating - such as where the word YOLO is in the brain, or how the consumption of Four Loko affects delayed discounting - no matter whether our subjects are technically alive or not, we will probably still get a significant result, most likely in the precuneus?**

The answer to all of these, I would say, is yes. But just because I was able to get the results of my first paper to squeak by with an outdated version of AlphaSim, and consequently have an easy time in graduate school, make tons of money, and be envied by all of my friends, doesn't mean that you should have the same privilege. The way forward will be a hard one for the neuroimagers of today; standards will be higher, methods tighter, reviews harsher. But if you put in enough work, if you exercise enough diligence, and if you live long enough into the ninth year of your graduate program, you may just be able to take advantage of the next major statistical loophole that comes along. Have faith.***



**For the record, I still don't know where the precuneus is. My best guess is: "Somewhere right before the cuneus."

***In all seriousness, AFNI has been responsive, helpful, and, given the circumstances, even gracious with any issues that have come up with their programs - specifically, 3dFWHMx (the smoothness estimator) and 3dClustSim (the cluster threshold estimator). The problems are not so much with AFNI, as with how it is used; and even though it is being patched, it is worth keeping in mind that there will likely be other issues discovered in the future, as with all methods - even non-parametric ones. (Their response to the Eklund paper can be found here.) The responsibility of the user is to make sure that the results that are reported follow the most up-to-date correction methods; that suspicious-looking activations that don't make sense in the experimental context, or consist of several blobs connected by threads of voxels, should be interpreted with caveats; and that even results which are weak and do not pass correction with the current methods, may still be real but underpowered effects. As with any result, it's up to the researcher to use their best judgment.

Thursday, November 17, 2016

On Academic Talks


I am beginning to suspect that most academics do not want to be heard. This is a paradox, given the number and regularity of academic talks. Indeed, some experts estimate that at least eighty thousand academic talks occur at any given moment, and that some three quarters of these are designed to fill up a time slot during the week which simply must be filled, lest we begin to think about what we used to do before meetings were invented. Most of the remainder are to distract the attendees from distressing events such as grant rejections, increases in conference fees, or foreign armies parachuting onto campus. Academic talks, like most variations of committee work, are becoming a polite form of debauch.

No matter. Regardless of how many talks they give, I still think these students, these professors, these men and women, do not want to be heard. Which is too bad. A talk, for me at least, is a chance to see, hear, and possibly smell the person behind the research. A talk injects red-blooded humanity into an enterprise which would rapidly become sterile without it. Journal articles and book chapters at best radiate a kind of cool elegance. Academic writing only tells me about what they think; I want to know more about what they feel. Where their passions lie. And when someone is talking to me in the same room they can't help but show some of those emotions, giving me a sense of why they are doing what they do in the first place.

Yet some of them would deny me even that pleasure. They talk to their laptop, or to the slides, but not to me. They use small font I can't read; complicated diagrams I can't follow; color schemes and animations that are in bad taste. I would be willing to forgive most of this if I could actually hear what they were saying. Usually I can't.

That has led me to outline six simple ways to improve a talk, any of which can be used profitably:

1. Suspense! If it is to have any worth at all, if it is to function as a stimulant and not a narcotic, a talk needs to have that essential ingredient of any effective story, which is suspense. When hearing a good story, the listener always wants to know what happens next: Who is this character, and what are his motivations? Why is there a note on the table and what does it say? What's the deal with those last two or three pages at the end of a book that are intentionally blank? So it is with good talks.

2. Dress well! A well-dressed speaker is a confident speaker, and a confident speaker is an engaging speaker. When in doubt, follow these "quick tips" on how to dress:

  • For Men:
    • Right way: Sport coat, open collar dress shirt, loafers, good pair of jeans.
    • Wrong way: Graphic tee with the words, "Legalize It."
  • For Ladies:
    • Right way: Pantsuit
    • Wrong way: Jeggings


3. Take notes! To be specific, have a friend take notes about you during your talk. Ideally, this should be a friend who is "brutally honest," also known as "being insufferable and pretentious when pointing out your flaws, and having it hurt even more because, deep down, you know he's right." Some people also refer to this person as "Dan."* No matter how much it hurts, remember that if you want to improve, you need to learn how to take criticism graciously! Also remember that when it's all over you can cut "Dan" out of your life completely.

4. Practice! The speaker putting together his slides at the last minute is guaranteed to embarrass himself and waste the audience's time. Since one of the major goals in life is to not be that one person everybody hates, avoid this by having your slides ready days in advance and doing at least two or three run-throughs ahead of time - preferably, in front of a live audience that includes Dan.

5. Eye contact! Try making eye contact with different people during your talk, but be careful of making eye contact for too little or too long. When in doubt, consult the following chart to make sure you hit the "sweet spot":

  • 1-2 seconds: Too little!
  • 2-5 seconds: Just right!
  • 5-10 seconds: Too much!
  • 10-15 seconds: Restraining order!
  • More than 15 seconds: This might be normal for, say, a serial killer

6. Lastly, find speakers you admire - and then imitate them. Really. Nothing helps your style more. Note their posture, the cadence of their voice, the way they carry themselves. Some rely more on humor, others have more of a quiet confidence; some tell stories that move the talk along, others tell you the point of the talk right away and never let you lose focus of it. But they all keep you wanting to know what happens next. They keep you in suspense.

Do not mistake this as recommending the substitution of humor for substance, of histrionics for actual thought. (The results are usually awful.) All I ask is that a speaker think about how he would want to be talked to, and then think about how seriously he has to prepare to meet that standard. Only then he begins to realize the gravity of his situation, no matter how small the talk or how low the stakes. In fact the stakes, when he thinks about it, are quite high. A typical one-hour talk attended by twenty of his colleagues can be either satisfyingly filled or grossly wasted, and a few simple calculations reveals a spread of forty attention-hours - a tremendous responsibility - all hinging on one man's decision about whether he actually wants to tell something and be heard. His choice is one between a successful talk and a crashing bore.

Now, I've witnessed more crashing bores than I would like to remember, and I've delivered more than I would like to admit. The fact is that there are far, far too many, and each one inflicts a serious loss on the audience - the loss of an hour they won't get back. (Were we not so complacent about what to expect out of a talk, we might feel more indignation.) Would they have wasted that hour anyway? Maybe. But a bad talk guarantees it.


*I specifically refer here to my labmate.