Hourglass by borabora on Flickr

Science moves slowly.

That may seem like an odd statement when the pace of press releases and breaking news seems to imply that new discoveries are flying fast and furious, with labs making discoveries and publishing them almost before they’re ready for prime time. What is missing from this perspective are the realities of the funding cycle and publishing cycle.

I just had to fill out a form that required me to list all my recent publications, and I realized that every publication I had submitted, first author or not, in 2011 has to be resubmitted to another journal because either no rational referee could be found (one paper included in the referee report “But I’m a biologist, not a social scientist, so maybe I just don’t get this.” ) or because after the review was complete they determined the paper best fit in another publication. This means, an entire year’s worth or research is delayed in seeing the light of day. In one case, the paper was originally submitted in May, and I only got the rejection letter a couple weeks ago. This is a bit horrifying.

Science moves slowly, and the major reason relates to the checks and balances that are meant to make this a meritocracy.

Here are the time scales we’re dealing with. Let’s say you get an awesome idea January for something you want to do. If the idea is an astronomy research idea, the grant deadline is November, so hurry up and wait. If the idea is education or computation related, you might get lucky and have a May or March deadline. It takes a solid 2 weeks fulltime (unpaid) effort for someone like me (e.g. with experience) to write a kickass proposal with a solid chance of being funded, and 4 days to whip together something with a wing and a prayer. Did I mention this is unpaid effort? So this is something that gets done instead of sleeping, spending time with spouses, or otherwise living. (Admittedly, lots of people ignore the fact that they aren’t supposed to write grants while getting paid off grants, but… I’m in Illinois and don’t want to join the governor in jail). The idea here is to get all the people with similar ideas to pitch their ideas at the same time so the limited funds can be fairly distributed to the best ideas. It’s a good idea, if a delaying one.

Once a grant is submitted, it typically takes 6 – 12 months to find out if you won the grant (this is a competitive processes), and another several months to get the money in hand.

So, for that idea you had in January and submitted a proposal for in November, you might get the funding in August of the following year. That’s 20 months from idea to funding.

Assuming what you’re doing produces results at an zippy pace, you might be able to submit your first paper 10 months later. You are now at 30 months – 2.5 years – from idea to submitting a result to peer review.

It is supposed to take 6-10 weeks to get a paper back from a journal. Often a set of changes are requested, and this is another 4 weeks or so of effort, you then resubmit, and maybe a month later have a publication date. This is assuming all goes well. My personal experiences is this process is more like 6 months than 4.5months.

So, maybe 3 years after the idea, you have something to show for it.

This is if you are lucky.

If your idea requires building an instrument, it might be 2 or 3 years before you start collecting data, and you’re first paper gets submitted 3 years after you are funded. If you (like me) have journal hell, it might be 2 years from writing the paper to seeing it published. If you have grant hell (which one team saw during IYA), you might only get your grant’s funding 2 years after you submitted the grant! This means it might be 3 years from idea to funding, 3 years of research, and 2 years to publication – a total of 7 years from idea to published result.

The two largest delays in going from idea to publication are both related to the peer review process.

Most grants require three steps (greatly simplified): 1) the agency makes sure the grant conforms to requirements, 2) a panel of peers is brought together to review and rank the grants, and 3) the agency figures out which grants it has enough funding to award based on that ranked list, and sends out the money. The review process is about 4 weeks by itself – As panelists, we need time to read, think, discuss, and write up our reviews. Agencies are always alert to the changing tides of congress, and sometimes (often?) funding is held up as agencies hope for a little extra money. (And sometimes, only after peer-review, do they realize the grant didn’t conform to requirements, and it gets rejected on a technicality).

With journals, it’s almost the same set of steps. 1) The journal makes sure the paper is appropriate to the journal, 2) A lone reviewer reads and critiques the paper, and 3) the journal figures out when it can fit the paper into the publication or rejects it. (And sometimes, only after peer-review, they realize it doesn’t fit their journal.)

In a perfect world, the slowness of this process is forgiven by the idealized way that peer-review should lead to constant improvement of our work. Comments on grants should lead grant writers to be able to improve their projects, and always do better (like a good teacher’s comments on an essay). Comments on papers should help researchers better distill their ideas, and should ask critical questions that help make sure the science is valid. The reality is every researcher has that stack of comments that makes no sense. (My most insane: I got a grant back with comments that included being hyper-critical of us for not having our podcast in iTunes, but our grant included a screenshot of our podcast not only in iTunes, but ranked in the top 10 and listed under new and notable). As near as I can tell, 20-40% of peer-reviewed comments in some arenas are useless – they are either arbitrarily harsh and hyper-critical of sometimes imagined details, or they are meaninglessly trivial/missing the point. Having sat on the review side of things, I know that sometimes the desire to not be too harsh of the politically powerful (because we want to keep our jobs) gets in the way of honest reviews that hit on real issues. I know that sometimes the “this researcher knows better” notion, coupled with exhaustion, leads to harsher wording on paper reviews then is necessary.  But if this was all that was happening, we wouldn’t constantly debate axing the peer-review system. The reality is, papers sometimes don’t get published because the reviewer you had has an agenda your research isn’t part of. The reality is, good researchers get black listed because they are nasty (or politically incompetent) humans. The reality is, people are just tired and are sometimes having a bad day and take it out on the paper they are asked to review.

While the peer-review system often works, it also often (probably not just as often, but still often) doesn’t work. I’m starting to struggle with this question: How do we justify the long delays in the progress of science that are introduced by relying on peer review, when peer review introduces chaos into the system? How do we justify all the delays when so many published papers are crap, and so many good papers linger in publication purgatory and get bounced between journals for a year or even years at a time?

I don’t know what the answer is. I’m making the decision to only publish in free-access and embargo free journals, but even they have peer-review. Since I am not a tenured professor I can’t simply share my research by posting it online – universities require peer-reviewed papers for promotion and tenure. The only solution I know is for the entire community to decide the system is broken and make a break away from peer-review at the same time. But is this throwing out the baby with the bath water?

I don’t know the answer, but I’m frustrated that the speed of technological progress causes papers and ideas to become stale before they can make it through peer-review.

There has got to be a better way. I just don’t know what it is, and how to make it the reality.

(One oft suggested substitute system is we all submit to websites that allow a thumb up, thumb down ranking and comments, and people earn authority points through good papers and good reviews. Over time, the top research would rise and the bad research would sink. This is a huge change though, and there is little incentive to change the system so radically, but… This is the best idea I’ve heard so far. Unfortunately, to work, it requires everyone to participate, and that just isn’t going to happen without institutional carrots to motivate faculty and researchers).