On the 16th, TIOBE Software released the Tiobe Index of the most popular programming languages. For the first time ever, R is in the Top 20. The top spots are, no surprise, occupied by Java and C respectively. More after the jump.

The way this index is assembled is…interesting. The full way the index is assembled is here, but generally its based on a weighted counts of the number of hits on a basket of popular search engines. This does open up the possibility that these are simply the most “Googleable” – or in need of searching – but it seems a decent aggregate measure of these things.

The big story would be Objective-C, which jumped up 3 spots in a dramatic gain in market share, presumably fueled by app development for the iPad and iPhone.

But for stats nerds? R jumped up 6 spots, from 25 to 19, making it into the Top 20. My other favorite, Python, dropped three places from 5 to 8. R is still undeniably a niche language, with 0.609% of the ratings (compared to 17.479% for Java or 3.218% for Python). But its a pretty significant achievement, in my mind, for The R Project for Statistical Computing to be near far more general purpose languages, and topping some heavy hitters, like MATLAB and Fortran.

For reference, SAS is #32 on the list, with a rating of 0.339%

More info is here: http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html

Unfortunately, long term trends are only easily finable for the Top 10 languages, but I’ve sent an email requesting it for R. If it works out, I’ll post it here.

So now I use R maybe…35% of the time?

But this is something of a side note. This review is on Revolution R, a commercial version of R that has some promising stuff in it. Or this review would have been on it, but for some…problems. More after the jump.

Revolution R seems, essentially, to be a distribution of R being billed as “Faster, Better, Stronger!” by Revolution analytics. Some of the underlying math libraries have been tweaked to make it faster in benchmarks, it’s got its own ritzy IDE, and its supposed to handle “Big Data” better – R sometimes struggles with very large data sets.

All of this was interesting to me, and lo, there’s a free academic version.

First problem: No Mac version. This is part of what sold me on the whole R thing in the first place, not having a partition on my machine that might as well be named “SAS and Mass Effect 2″. When asked over Twitter about Mac development, I was told this:

@epigrad We’d love to, but our roadmap is focused on Windows and RedHat for now. But it’s on our list for the future.

Fair enough says I. And I’ve got Parallels, I can get around this! Not being a huge fan of Windows, and the whole point of this being to get away from something that needs a Windows license as well, I decide to check out the Linux version. I admittedly do not have Red Hat Enterprise Linux 5. What I do have is its extremely closely intertwined cousin, Fedora. Sponsored by the same company. Both use Yum as a software package manager. Fedora, according to Wikipedia, “Fedora serves as upstream for future versions of RHEL. RHEL trees are forked off the Fedora repository, and released after a substantial stabilization and quality assurance effort.”

Outstanding. And things designed for “Red Hat Linux” have worked on my Fedora install before – namely SAS and the Enthought Python Distribution.

Not Revolution R my friends. Three attempts to install, a few walls of failed dependency installations, and I gave up and emailed for help, detailing that I was using Fedora 15, and what was happening. What I got back was this:

The Linux version of Revolution R Enterprise is officially only supported on Redhat and CentOS. Our installer requires ‘yum’ to install all of the components. It also relies on ome third party libraries and will attempt to use ‘yum’ to download and install those dependencies.

We list all of these dependencies in our Installation Guide on page 7, in the section ‘Package Dependencies’.  The error message you received is indicating that the installation could not complete successfully, because some of these dependencies were not found. All of the dependent packages need to be installed first, for the main part of the install to proceed.

I would suggest that you attempt to install the software on a Redhat system.

Otherwise, you will need to see if you can find versions of the dependent packages for Fedora 15 in their official repository.

To be blunt…no. I tried, I really did, but R installed like a charm, and I’m not buying another commercial operating system to get your thing to work, or going hunting for my own packages. The whole point of commercial software on Linux OS’s is that you shouldn’t have to do that. For a product that charges $1,000 for a single user workstation license to not work on a nearly identical OS? If your product really is only compatible with Red Hat Enterprise Linux, that’s…a pretty narrow slice of the market. I’d also note that CentOS isn’t listed on the site as one of the supported operating systems – apparently one similar-to-RHEL Os works, but the other doesn’t.

I give up. Back to regular R it is. I’m sure Revolution R is an excellent product, and if you’re using Windows or RHEL, you’re set. But honestly, at that point, I’m back to the things I don’t like about SAS, and away from the things that pulled me to R.

Weirdly, for some other parts of my life, that’s actually true. I’ve found I keep up better with things when they’re busy enough to always occupy a little bit of my attention. One thing, and I can always say “Meh, I can put it off until tomorrow”. But many things? They demand attention.

So lets see if that works. I’m moving most of my Apple/OS X centric musings, along with some projects and the like, over to The Academic Mac. See you there, if you’re interested. Keep reading here if you’re not, I’ll have some new posts coming in the next few days.

Hence the blog suddenly going dark, as it seems to do when the leaves start to fall from the trees and the temperature drops. Hopefully it’ll pick up shortly.

“Is there evidence that evidence-based medicine outperforms other approaches?”

But it’s also stupidly useful for customizing your set-up in ways not necessarily supported in the UI, without the use of a third-party utilities, exactly the way you like it. Two of the most useful things I’ve found recently are described after the break.

First, a warning – you are fiddling about with the roots of your OS. Please at least backup your system before you try this. Also, while these have all worked for me, if you somehow blow up your machine, it’s not my fault.

Two other notes: These were all run on an administrator user account. $ is used to denote the end of the prompt in your terminal shell. Don’t type that bit.

Trick 1: Automatic Backup to DropBox

DropBox is stupid handy for keeping your files “in the cloud” and having access to them anywhere. But what if you want to be able to  be able to backup and use files outside your main Dropbox folder? One line of Terminal code and you’re done. Exploiting “Symbolic Links”, which essentially tell your machine to treat one directory like its in another place, you can link files outside your DropBox folder to one inside it, like so:

$ln -s /Volumes/HardDriveName/username/Documents/ImportantFiles ~/Dropbox/

That takes the folder “Important Files” under your Documents folder and places it in the main directory of your Dropbox. Edit the paths as necessary or, once you’ve typed “ln -s ” simply drag the folder you want to make the link from to the terminal window – the system will automatically fill in the appropriate path. And there you go – instant backup of any directory you want onto your DropBox. Which gives you 2 GB of space for free.

Trick 2: Color Changing Terminal Windows when using SSH

This one is a little more advanced, but in my mind, probably more useful. I often have several Terminal windows up, some of which are running tasks on my own machine, some of which are connected to one of several servers. This makes it occasionally difficult to keep track of what exactly I’m doing. While I can manually use a different Terminal theme for each task with my own personal coloring scheme – but its nicer if the system will do it for me.

Step 1. At your main user directory (the one you can get to by typing cd ~), check and see if you have a file called .bashrc . If you don’t, make a blank document using a text editor like TextWrangler or BBEdit.

Step 2. Open your .bash_profile file in the same directory using a text editor and type the following:

if [ -f ~/.bashrc ]; then         
     . ~/.bashrc 
fi

This lets functions put into your .bashrc file be used.

Step 3. In the Terminal application, make a color scheme you’d like to use when SSHing – this can be a general scheme, or one for each server. In this example, lets assume I made one for SSH tasks generally. Name it something useful, like “SSH-Theme”.

Step 4. In your .bashrc file using a text editor, put in the following:

function tabc() {
NAME=$1; if [ -z "$NAME" ]; then NAME="Pro"; fi
osascript -e "tell application \"Terminal\" to set current settings of front window to settings set \"$NAME\""
}
alias sshiron="tabc SSH-Theme; ssh -X confounding@bigiron.university.edu; tabc"

What’s this do? Generally speaking, it means the Terminal uses the “Pro” theme. But, if I type $ sshiron , the Terminal changes to the new SSH-Theme and logs into the SSH server – in this case, a pretend server at bigiron.university.edu. Edit things accordingly for your own setup. When you exit, your window will return to the Pro theme.

But thankfully, Nancy Pelosi’s Flickr page has provided me with this:

Bad graph is bad. One reason is graphical, the other philosophical.

First, the graphical reason: The percentages on this graph are nonsense. They’re not independent – a massive increase in the national debt in one president means the following president can spend more on a per-percent basis. Clinton’s 37% is not twice (roughly) of what Obama’s 16% is. Its actually less. Considerably less. This is one of my rants about relative measures – they’re often extremely useful, but they’re also very dangerous. Choosing the right denominator is harder than people think it is. And interpreting relative measures is somewhat more difficult than absolute values. So lets try this in absolute dollar terms. Now, a disclaimer: This graph is, by itself, flawed. I’m pulling the numbers from Wikipedia, and I cannot for the life of me find (at 2 AM) whether or not the figures are inflation adjusted. And they’re not adjusted for GDP. But its as close an analog to the graph posted above as I can manage:

And there we go. Note this tells a very different story than the graph above. Admittedly, not one that yields a nice per-party trend. Sorry.

The second issue is a philosophical one. Either graph makes a fundamental assumption – that all debt is the same debt. And that this debt is inherently bad. Lets consider a scenario:

President Jones increases the national debt by a trillion dollars to construct a massive golden likeness of himself, visible not only from space by the surface of Mars.

President Smith increases the national debt by a trillion and a half dollars. To fund research that manages to cure AIDS, Cancer and morning breath.

Is President Smith a worse president? Has he done more harm to the country by putting us more deeply into debt? By making no attempt to distinguish good debt (say, the kind spent getting the country out of a recession, curing disease or saving puppies) from bad debt (unnecessary wars in Somewherestan, massive golden statues) and going by head-to-head amounts, the Left has essentially ceded the argument that debt may sometimes be useful to the Right. Which, if many of your platforms are based on debt and spending – or you believe Keynes may have had a point – seems to be a failing position to start from.

First, how I got there in the first place. I’ve been essentially learning Python and C as I go, working on my dissertation research. This, predictably, has resulted in some problems – and I happened across StackOverflow as a decidedly decent place to get answers to what were admittedly pretty rookie questions in a hurry – and at all random hours of the night. From there, I drifted over to their statistics site, CrossValidated, where I hoped I might do a little more good than just the asking of random programming questions.

More after the jump…

Some thoughts, both good and bad, about CrossValidated, and the community driven question-and-answer concept:

The Good:

First, like it says on the tin – have a question, ask it, get an answer. One the community, hopefully a knowledgable community, can help screen the good answers from the bad. And for most statistics questions, it does a decent job of it. Some more obscure ones don’t get answered, and the farther afield you get from straight-up statistics and into applied work, the more likely it is to go unanswered. Part of the reason I may have accumulated the reputation I have on the site in a month or so of time is that there were a lot of…low hanging fruit Epidemiology questions that had gone unanswered. But answers when they come are clear, well-sourced, and in my experience pretty damned good.

The sites in general are great for the kind of nagging questions about computing that come up and don’t get covered in classes and the like. Lingering questions about programming, hardware, interacting with big, scary cluster computers using command line interfaces…StackExchange is a decent place to get those kinds of questions sorted.

The…Fair:

Some of them are a little off-kilter for the specific state-of-the-art in Epidemiology, but this is a pretty common phenomena. Each field has its own conventions, its own rules and its own way of doing things. Cross-field advice is often a little off. This, in my experience, is especially true when talking to dedicated statisticians – their answers are very, very good, but they require a little bit of processing to get to a usable form.

Software questions are common on the site, and tend to get answered with answers in R. Which is great…if you’re intending to use R. If you’re not – say, again, you’re in a more applied field where there’s a different standard, like SPSS or SAS, you’ll probably get decent theoretical answers, but software specific questions on platforms outside R may struggle to find answers.

CrossValidated is also undeniably a somewhat more obscure site than the bigger components of the StackExchange system – which means the answers are more easily dominated by a single user or small group of them. That does suggest some vulnerability to pushing a particular agenda or two – what if all the contributing users are dirty, filthy frequentists or some such!? I haven’t seen evidence of this problem particularly, but there is a decidedly small population on the site relative to some others.

The Bad:

Some of the questions – to be blunt – will put in you a profound fear of the state of research. But contributing to the site hopefully helps with that in its own, small way.

Stop by, check it out, consider posting on CrossValidated – or any of the other number of StackExchange sites. Programming, server administration, photography, bicycles…there’s options for whole slew of possible interests – and a site to suggest and build support for others, though that process is admittedly a long and hard one.

First a disclaimer: I have been fortunate (unfortunate?) enough to have escaped ever having to directly recruit subjects for a study. All of my research involves either already collected data being re-used for further research purposes, or mathematical modeling/simulation studies, which use entirely virtual subjects. So this is the musing of a random epidemiologist who thinks about study design on occasion, not a battle-worn veteran of the subject recruitment wars.

Online communities represent a really great way to drum up interest for something, and access concentrated groups of individuals. One only needs to consider my post on the traffic on this blog. Two posts on message boards for a relatively small hobby have driven 8,765 pageviews to this site.

Its not hard to see how patient-driven social communities online could represent a spectacular resource for studying rare diseases. It’s extremely common to over-sample individuals with a disease using other types of resources (registries, hospital records, recruiting specialists to enroll their patients), and there’s ways to handle that kind of oversampling statistically. Heck, the entire case-control design is devoted to oversampling cases. So yes, there is some awesome potential for social sites to be used for study recruitment. As a bonus, its the internet, so by and large reaching these people is at a much lower cost than say mailing recruitment information, or advertising. And in this day and age, anything that’s easier on a study budget is a good thing.

I do however have two major concerns about its use, neither one of which is inherently fatal, but do warrant some caution:

1. Social sites over-sample cases, to be sure. But they also over-sample other covariates in a study – covariates like race, income, and gender. It is entirely possible to bias a study by sampling cases from a population with one demographic structure, and controls from a population with another (using non-social media recruitment methods). I’ve done a little research on the reverse problem, perfectly sampling controls and then sampling cases from a population with an unusual demographic structure (in this case, people with landline phones), and it can substantially bias your results. This kind of problem is something that should absolutely be approached with caution.
2. “The Echo Chamber”. Some communities…don’t have the best relationship with the medical community. The Autism-Vaccine link people come to mind immediately. A recent survey, properly eviscerated on Respectful Insolence has generated 7,825 responses. The study is biased, terrible, and has about as much scientific rigor as a badly conceived middle school science project. But numbers of participants like that give it weight, let those looking not for empirically backed truth but a confirmation of their own beliefs say “A study with nearly *8000* people…”.

To be honest, the first problem concerns me more, as its far more subtle. The second problem…is a long-running one, and while its enabled by internet communities, its hardly going to go away by everyone being a little more careful with their methodology.

Which is why I’m pleased to say that Confounded by Confounding posts with the ‘SAS’ category tag will now be added to SAS-X.

For anyone visiting from that site, welcome! Feel free to look around