This makes me really, really sad. More after the jump.

For the purposes of this post, let me be clear about something: I’m not talking about “crowdfunded” projects that essentially involve people paying to be a data point in a study. I’m talking about using small donations from interested people to fund research that takes place in universities, research institutions and the like, and would otherwise be something that could be funded by traditional sources.

Maybe this would come in the form of something like Gittip to say “Yeah, your cool science is worth a couple bucks a week to me”. Maybe its in a more formalized form, like the folks at SciFund Challenge or the handful of startups chasing the idea.

Regardless, it’s an idea I would really like to see exist. It’s right up there with some benevolent soul dropping such a staggering pile of cash in my lap that I can just conduct research off the interest. Some of the reasons I in particular have a fondness for the concept:

• It allows someone to participate, even indirectly, in the doing of science. Which is neat – Kickstarter lets people fund random ideas in technology, board games and performance art, why not science?
• It’s another funding stream – and really, we could use another one.
• It aligns particularly well with some of the work I do – crowdfunding can act as a good way to get equipment and the like that it’s normally somewhat hard to get grant funding for. Computer hardware, I’m looking at you. Computer hardware is, I’m pretty sure, the great funding frustration of theoretical labs everywhere. Thousands of dollars in Amazon AWS fees? No biggie. A $500 box to sit in the corner and chug through scripts night and day? Out of the question.
• It aligns the interests of the public and the interests of scientists in a very direct way. I’ve been lamenting the lack of motivation to write really good code in science – as long as your code can run after the postdoc who wrote it leaves, what does it matter if it’s poorly commented, slow, and uses a ton of opaque variables?  Code, and reproducible data, and all the other things we say scientists should strive for aren’t the things scientists are paid for. What if they were?

Like the title says, crowdfunding for science is an idea I’d love to see come to fruition. I’ve gone as far as gearing up for the last round of the SciFund challenge, putting together an application…and then withdrawing, having decided it was more trouble than it was worth. Here are my reasons why:

Science is expensive. This is the one I’m having a really hard time getting over. Lets look at the last SciFund challenge:

45% of its projects met their funding goals. That’s a decent payline as far as these things go (better than the 10% some government grants are seeing these days), but its far from a sure thing. But beyond that, how much money were these goals for?

The median payout was $1440 in the last round, the mean a little under $2200. And as much as I hate to say it, that’s a drop in the bucket. That – and another $700 – would get wiped out with a single submission to an open access journal like PLoS Biology. Or a single bit of conference travel. Or an alarmingly small amount of computing time.

That amount is about the same as a pilot funding grant at my institution, one I’ve applied for and won, and I can tell you $2000 just doesn’t go very far. It’s good for establishing a very basic proof of concept for a study. Maybe pay for a single spurt of field work to establish the all important “Preliminary Findings”. Get a graduate somewhere for a week to get some face-time with a piece of equipment/person/animal, etc. The term I’ve taken to refer to this is “grant spackle” (as in, to fill in small holes…). Another term could be “found money”. And like $5 in your pocket that you didn’t know was there, found money is awesome.

But you don’t plan on found money. You don’t hire grad students on found money. You don’t establish a productive lab on found money.

Time is Money. DrugMonkey points to institutional overhead as a problem – for readers who don’t know what that is, often a large portion of a grant (often upwards of 50%) goes to the institution for “indirect costs” – things like keeping the lights on and the copy machines full of toner, and other ancillary costs of doing research. At least in theory. I actually don’t think this is as big a problem as DM does – my limited foray into crowdfunding was met with “Meh, its not enough to worry about”.

My problem is time. The aforementioned pilot grant took an evening to fill out, and requires a single web-form report. Crowdfunded projects require videos, incentives, blog posts, Twitter and Facebook activity…and navigating ones way through the byzantine halls of the university funding office, which is set up for much larger grants, and likely hasn’t heard of your crowdfunding source before. Spending a week playing phone tag isn’t much of a problem if you’re talking about a quarter of a million dollars. For $2000? You’re going to eat through that money in a graduate student’s time rather quickly. This was really what killed it for me – the effort to reward ratio was way off.

Is It Repeatable? Even if you had a friendly grants office, or had figured out some clever way like a separately held foundation to handle the money in a fast and streamlined fashion, and even if we were talking about considerably more money, I have concerns about repeatability. There’s been a lot of talk about the importance of social media in crowdfunding efforts for science, and that’s where my concern lies. Is there a pool of sustainable, repeatable donors – donors who don’t know the person trying to get funding? Or is this essentially passing the hat around a potential fundee’s social network, and banking on the novelty of the idea to generate a little cash? If its the latter, there’s only so many times that can be done before it runs out of steam. You might fund one project a year – will you fund two? Seven? What if they don’t pan out – we’ve seen anger centered around Kickstarter projects that have failed, what about SciFund projects that generate a whole lot of null results (something science does on occasion)? Again, that dumps crowdfunding into the realm of “found money” and something you might do in your spare time, or if you really need to fund that field work in Montana or wherever and can’t scrape together the cash for the ticket and food.

Which are noble goals, but not something I think will change the face of science. But I could be wrong – that’s what the comments section is for.

And now, I’m shamelessly stealing it to look at how I used R in 2012. I figure I should probably let the #2012pythonmeme stay Python-only, so I’m going to try #Rin2012.

1. What’s the coolest R application, package, or library you have discovered in 2012?

This one is like choosing between your favorite children: RStudio and plyr are both new discoveries to me that have massively altered the way I work with R, for the better. If I had to pick one, I’d probably go with RStudio just because of the amount of work that got done this year thanks to a R-specific IDE. Pretty excited about RStudio Server too, I just need a system to run it on. Anyone want to donate one?

2. What new programming technique did you learn in 2012

I’m going to cheat, and say since plyr didn’t get the coolest R package award, we’re going to give it credit here. Because parallelism in R was the new programming technique I learned this year, and for simple “Apply this function by a grouping variable” tasks, plyr and it’s connection to doMC saved some serious time. And it’s always gratifying to see all the cores on your machine go to 100%.

3. Which open source project did you contribute to the most in 2012? What did you do?

I didn’t specifically contribute to any open source projects. However, I did publish the code needed for a workshop I taught on mathematical epidemiology on GitHub, and one of the papers I published has a freely available electronic appendix hosted there as well. I’ve got some plans for next year – stay tuned.

4. Which R blog or website did you read the most in 2012

CrossValidated and StackOverflow for websites. The amazingly useful R-Bloggers lets me cheat in terms of blogs and say “All of them”.

5. What are the top things you want to learn in 2013

I’m interested in doing some social network analysis with R, and there’s a project that I’ve got an extensive codebase for in SAS that I want to translate over to R, but haven’t the faintest clue on how to get started.

6. What is the top software, application, or library you wish someone would write in 2013

Besides the magical R faeries leaving me a copy of the above mentioned SAS code in R? I’d really like to see a replacement for GillespieSSA designed for intensive, research grade projects.

Want to do your own list? here’s how:

• copy-paste the questions and answer them in your blog
• tweet it with #Rin2012 hashtag

1. What’s the coolest Python application, framework, or library you have discovered in 2012?

PiCloud. I’ve been following their growth and expansion, and while for the moment I’ve used them mostly for toy examples and experiments, I think they represent a really promising platform for the kind of on-demand, embarrassingly parallel simulation work I’ve been doing recently. Some of my Python programming got shelved this year while other stuff got done, so I’m looking forward to revisiting the platform and seeing how its developed. The PiCloud Notebook has me pretty excited.

2. What new programming technique did you learn in 2012

Honestly, 2012 was the first year I really pushed myself in terms of programming, so I could probably say “All of them” and be right. Generally speaking, just thinking “I could write a function for that” was a huge step.

3. Which open source project did you contribute to the most in 2012? What did you do?

Nothing really. That isn’t strictly true for R (see the post to follow this one), but my Python skills are pretty dubious. I do have some ideas bouncing around in my head however, so hopefully 2013 will be the year of a contribution or two.

4. Which Python blog or website did you read the most in 2012

StackOverflow and Python Weekly.

5. What are the top things you want to learn in 2013

I’d like to push myself a little more in terms of object-oriented programming. It’s something that I can see the utility of, and would make some of the “book keeping” tasks I need to do in a project or two vastly easier, but it’s still not a technique I’m comfortable with.

I’d also like to dabble in Django, moving from “This result comes out on the command line” into something useful to the world at large.

6. What is the top software, application, or library you wish someone would write in 2013

What’s left of my dissertation. Hopefully that someone is me, or I’m screwed.

Want to do your own list? here’s how:

• copy-paste the questions and answer them in your blog
• tweet it with #2012pythonmeme hashtag

The text of one of the retractions reads as follows:

This article has been retracted: please see Elsevier Policy on Article Withdrawal (http://www.elsevier.com/locate/withdrawalpolicy).

This article has been retracted at the request of the Editor-in-Chief.

A referee’s report on which the editorial decision was made was found to be falsified. The referee’s report was submitted under the name of an established scientist who was not aware of the paper or the report, via a fictitious EES account. Because of the submission of a fake, but well-written and positive referee’s report, the Editor was misled into accepting the paper based upon the positive advice of what he assumed was a well-known expert in the field. This represents a clear violation of the fundamentals of the peer-review process, our publishing policies, and publishing ethics standards. The authors of this paper have been offered the option to re-submit their paper for legitimate peer review.

While we live in an age where all websites are vulnerable, I’ve found journal submission/editorial/review sites to have a uniquely lax approach to security. As just one example, I’ve had my password sent to me, in unencrypted plain text by more than one journal’s system.

Plain text. That’s…pretty basic security fail. Right up there with your administrator password being “password”.

I assume it hasn’t been a bigger problem because peer reviewed journals are kind of obscure targets. But apparently not obscure enough.

I’m back, hopefully, and should be updating content again regularly soon – hoping to post some more coding-related posts, some science posts, and I’ve joined the O’Reilly Books review network, so there’s probably some reviews of their books from the perspective of an Epidemiologist coming down the pipeline.

Or at least that’s the plan.

So when press releases and news reports about the Reproducibility Initiative started making the rounds on various blogs I read, I looked it over with interest. The concept is simple: Reproducible results are good, and should be rewarded. Validate your study through the initiative, and you’ll not only get a ‘Certificate of Reproducibility’ (of whatever worth that might be to you) and more importantly for most career scientists, the replicated results can be published as an independent paper in the PLOS Reproducibility Collection, and the original study will be marked as reproduced in the parent journal if it’s one of the Initiative’s supporters.

That all sounds great…but as with all things, there’s a “but…” coming. Or, to my mind, several. More after the jump.

Cost: How the Reproducibility Initiative (which I’m going to shorten to RI from here on out) works as I understand it is you submit your study and some supporting information to RI – which is an offshoot of the commercial Science Exchange linking up providers of expertise with those who need that expertise – and they hand it off to a blindly matched provider who can validate your results. There’s a key phrase on their site that’s important: “You’ll be responsible for payment of services rendered unless sponsored by a partner organization.”

That means you’re footing the bill, or your grant is. That means you’re essentially paying for a second publication for your work (who knows how valuable those will end up being on a CV) out of progressively more and more stretched research funds. For a big lab this might not be a big thing, but for a small early-stage investigator lab running on the skin of its teeth, or a doctoral student’s project that managed to swing some funding? Yeah right. Without a funding mechanism, all the RI does is look at a small subset of science – both “well heeled” and “reproducible”.

Feasibility: This is the one that actually worries me the most. What, exactly, can Science Exchange reproduce? For the most part, it appears their strength is in wet-lab life science work. Searches for ‘Biostatistics’ and ‘High performance computing’ yielded at best one or two relevant results. ‘Mathematical modeling’ produced none.

For many laboratory-based publications, this might work swimmingly. But how do you propose replicating a mathematical model? The best I can envision is what I’ve termed “Click Run” reproducibility, which means your code works the way you say it works and puts out what you said it puts out. I’ve discussed this a bit with a question on CrossValidated, but as far as I’m concerned, that level of “does your code do what we think it should” reproducibility is a low threshold.

If we’re talking about what you need to really “reproduce” an epidemiology study? It’s not working off the same cohort data. It’s not rerunning the code. It’s conducting an entirely new study of the same effect on a different population. That is what’s needed in the field, it’s what’s needed to address the concerns of someone like Ioannidis and it’s entirely outside the scope of what RI can do.

It seems that science often struggles once it steps away from the bench, and two-group controlled experiments. Statistics becomes murky, causation turns ephemeral, and when people talk about reproducibility, you have to start outlining exactly what you mean. I might have a post on that last one in a few days. In my mind, the goals of the RI are laudable, but between not being able to serve small labs out of budget, and entire fields out of practicality, I wonder if it will do more than double the paper count of a few select labs. That would be a shame.

One problem: That’s not how the Law of Large Numbers works. More after the jump.

First, a definition of the Law of Large Numbers. From the article itself:

the law states that a variable will revert to a mean over a large sample of results.

From Wolfram Alpha:

A “law of large numbers” is one of several theorems expressing the idea that as the number of trials of a random process increases, the percentage difference between the expected and actual values goes to zero.

So both definitions are in agreement. So it’s not the definition of the Law that the article has gotten wrong, just its application. The most useful way (for me at least) to see the Law in action is to visualize it. Lets use a very simple simulation of a fair coin flip. We know that the average of a fair coin should be 0.50. Using some simple R code, we can simulate flipping a coin 10,000 times:

set.seed(807060)
x <- sample(0:1, 10000, repl=T)
s <- cumsum(x); r <- s/(1:n)
plot(r, ylim=c(.01, .60), type=”l”)
lines(c(0,n), c(.50,.50),col=”red”)
round(cbind(x,s,r), 5)[1:10,]; r[n]

This yields a plot that looks like this:

The black line is the cumulative average, with the red line being the known mean the simulation should converge to over time. Notice how after 1, 2, 10 or even 1000 coin tosses, the average isn’t 0.5. After 10,000, it’s considerably closer, but still not necessarily there. After many, many more? That is how the Law of Large Numbers works, and its the underpinning behind most simulation work – that with enough simulations, you should converge on the actual expected value, even if it isn’t known – or (unlike our coin toss example) its hard or impossible to arrive at analytically.

But this has nothing to do with Apple’s performance, past, present, or future. The company’s earnings, share price, or profits aren’t a random process. That it becomes progressively harder to deliver the same % increase as a company grows larger isn’t wrong, but it isn’t the Law of Large Numbers. Just because 500 billion is a large number (and in dollar terms, it is a very large number) doesn’t mean that’s what operating. There are many, many reasons to wonder if Apple’s trajectory can continue. Has it lost its small, lean operating patterns now that it’s the largest company in the U.S.? Can it continue without Steve Job’s driving personality? Can it keep its track record going? After all, eggs aren’t the only thing that comes out of a golden goose.

But it’s not that random processes converge toward the expectation after a progressively larger number of trials. I’d also like to note that, years ago, when Mr. Dell was calling for Apple to be scrapped and the proceeds given to shareholders, and it was trading for very low double digits instead of middling-high triple digits, noone was going “You know, the Law of Large Numbers will eventually drag Apple up. It’s a sure thing!”

Disclosure: I own stock in Apple. And am rather fond of simulation.

It’s actually a better list than most. I find myself getting irked when “Science” is taken to invariably mean either “Physics” or more commonly in life science blogs and the like I read, wet-lab biology/biochemistry. What about us poor theorists? Or population-level empiricists? Do we really need a list dominated by timers to make sure you take your samples out of the water bath in time?

After the jump are my Top 5 apps, hopefully not terribly biased toward my own research. And absolutely not featuring my own (nonexistent) app.

For the record, there aren’t in any particular order.

#1: Dropbox. Dropbox is seriously amazing. It’s essentially a shared, virtual drive sitting out in cyberspace, accessible from your desktop, another computer if you need it, your iPad, iPhone, Android devices, the internet…and sync’d between them all. It’s essentially replaced flash drives for me, which are both unreliable and prone to wander. It’s got a second copy of all my important files. I have it automatically set to be symbolically linked with certain work folders on my desktop so anything I put in those folders will be automatically added to Dropbox – and any files I make on my iPad will show up on my desktop as well. And it’s free for students, and they give you a little extra space (and double the bonus space for referrals).

#2: Papers. This one started as a Mac-only app, but has migrated to iOS, and now it’s on Windows as well. It’s the single most pleasant “paper repository” software I’ve ever used. The citation manager features don’t quite seem ready for primetime (I tried it with my proposal and they required some editing), but in terms of organizing your papers, keeping PDF files corralled, sharing and searching them? Second to none. And with the iOS application, you can keep your libraries sync’d, so as long as you have your phone or iPad, you’re never without that one paper you really need.

#3. iSSH. iSSH may be the data science equivalent of the “time your experiments” app for the wet-lab folks. Simply put, its an easy to use, remarkably clean SSH terminal application for your iPad/iPhone. Which means if you’ve got a PC set up at home that you can SSH into, or access to the friendly local university cluster, etc. you can do it from an amazingly portable device. It extends the capabilities of the iPad and keyboard combination massively – you can now program in R, maraud around Python, even compile and run C applications remotely. Or remotely administer a server. Or check on a long-running job to see if you can come back from your coffee break.

My most dubious use of it was when the East Coast got hit by the earthquake/hurricane combination and I was out of town, SSHing into my machine and assuming that if I could still connect, the damage couldn’t be that bad. I’ve also done far more productive things on it.

#4. Wolfram Alpha. Wolfram Alpha is just a general purpose utility “I don’t know something and I need to” program. Like Google, except it knows you’re using this for science. A search for “Clostridium” gives me the taxonomy of that genus. And other related members of the family. And members of that genus. And its taxonomic network. And it shows its sources.

“San Francisco” gets you a map. The current local time. And weather. And weather history. And the cost of living index, and median home price and unemployment rate and nearby airports and hospitals, etc. etc.

“Odds Ratio” gets you the formula for…you guessed it. And a quick calculator.

“Ethanol” gives you the formula, compound formula and structure diagram for ethanol. And the molecular weight, melting point, boiling point, density, solubility…are we getting the idea?

It’ll even do math for you. Math that’s way more complicated than most calculator programs.

#5. Some Sort of Text Editor. This is a matter of personal preference. I’ve got a copy of Pages, Apple’s word processing app, and while it’s powerful, there continue to be issues hopping back and forth between Pages and Word when you have EndNote citations, and it proved to be more trouble than it’s worth. But some people will want clear, plain, “White screen, cursor, and my words” UIs, while others will prefer some of the more familiar text editor features.

Personally, I use “Textastic“, which has word and character counts, syntax highlighting for programming languages, and can either send files via email or (among other destinations) save directly to DropBox.

So there you have it. My top 5 must have iPad/iPhone apps.

Since that time, Revolution has added support for RHEL 6.0. I figured what the heck, why not give it another go, see if things will be more cooperative this time. Lo and behold, they work! That’s right, it appears that the newly updated version of Revolution R – at least the academic version which I was using – will indeed install and work on the latest version of Fedora.

Now, some caveats:

• I have no idea if it will work for everyone. I’m running a brand new download of Fedora 16 via Parallels on my Mac, installed with essentially every developer related package I could find in hopes of covering all my bases. And only the 64-bit version of Fedora works. Tried it on the 32-bit version, and it failed spectacularly.
• It could break at any time. What with Fedora not being supported.
• If it does break, you’re SOL. My experience with Revolution’s customer service/tech support folks to date suggests that if you’re not running RHEL (or maybe CentOS), you’re SOL if something stops working. So I’d still be a touch skittish about putting any major analysis into Revolution R running on a Fedora machine, or if you do and get it working, touch nothing.
• I don’t actually know that it works. I mean, it installs. ‘$ Revo64′ works as a command. It comes up, and I can do my usual “R, are you working?” procedure, which involves nothing more than creating two vectors of random numbers, plotting them against each other, and running a quick and dirty lm(). All of that worked, but I sincerely doubt that really tested the guts of Revolution’s distribution of R. If someone wants to give me some heavier code to put it through its paces and see if it crashes, please feel free to drop me a line or comment.

But there you have it. Hope for the future.

Now Revolution, if you’re listening? My kingdom for an OS X native version.

I ran across this post at The Tree of Life entitled ‘Interesting new metagenomics paper w/ one big big big caveat – critical software not available”.

The long and short of it? Paper appears in Science, has fancy new methodology, lacks the software for someone else to use their methodology. Blog author understandably annoyed. But I have some sympathy with the authors of the paper itself, as much as I prefer the code for an analysis to be available for publication. My thoughts after the jump.

First, this post was born as a comment on the OP blog. For some reason, blogger hates my WordPress login.

Second, I understand the spirit of the post. In an ideal world, the software would be available. From a stable URL. For every single platform under the sun. Or maybe even a web interface, if the project was feeling particularly fancy. This is How It Should Be ™.

That being said, there are reasons that isn’t true. I’ve had one collaborator essentially decide not to give out code because further analysis – that would appear in future papers – was already baked in, and they weren’t going to go through their code line by line to make sure some poor grad student’s project didn’t get scooped by someone reading the code carefully, or that in removing that stuff, they didn’t manage to otherwise break the software.

But lets leave that aside for the moment, and say – as the original published paper did – that the purpose of all this is a new technique, with new software, that we’re hoping people use. There’s still reasons for the paper to come out and the software not be available yet – legitimate reasons. The development of software and its use in science, while very closely linked, are actually disjoint processes that need not progress at the same pace. Some issues that have happened to me:

Beautiful software, ready to go, sitting idle for months waiting for the right numbers to come in to make it usable.

Output from software that is interesting science on its own, but the software isn’t ready for primetime. Maybe it’s got a hideous command line interface with a dozen opaque arguments that appear in no logical order. Maybe the quick and dirty solution to something that produced interesting results for several datasets is inefficient enough that it needs more memory than something like it should. Maybe the documentation is a series of ad-hoc scribbles on a white board. Or maybe it works on your machine, works on a student’s machine, but utterly breaks the first time you try it on a colleagues. Or perhaps you simply want to make it better, and while the science is ready to go and won’t improve by festering for six months, in six months you could have a GUI. Or better performance. Or cross-platform software. Or all of the above.

I can understand the bloggers frustration. And papers that do this should absolutely both provide enough methods detail that you could write your own software if you had the inclination, and focus on those methods, not mysterious code you don’t have access to. But when I read a paper where there’s clearly software to be had, but it’s not available yet, my first thought is “What went wrong?”