Wednesday, April 15, 2009

Another workaround for Memory Issues in R / Bioconductor

This post could be entitled "Error: cannot allocate vector of size 256.0 Mb". R provides this maddening response for even the most trivial seeming tasks. I have a 4 Gb linux system, and I have encountered this error message for "vector of size" in the 10s of Mb. And system wide, there appears to be plenty of memory still available, and the swap space hasn't been touched.

What's going on with R memory management??

There are many issues and many variables -- the overall amount of memory in your system, whether you have a 32bit or 64bit systems, how much memory is allocated to the R process (especially under windows), and of course, the size of the "vector" that R is trying to allocate.

But for me, the most salient topic is memory fragmentation, aka the swiss-cheese effect. Matthew Keller has an excellent discussion of the topic here, and I won't try to go into all the details. But basically, even if you have plenty of "free memory," you can run into this memory allocation error if the "free" memory is discontinuous. The above link suggests some workarounds, but here is one that I have not seen posted anywhere:

Quit R. Yes, just quit! But don't worry we will come back again!

> q()
Save workspace image? [y/n/c]: y

Then, immediately start R again, and allow it to load in all the objects you had before. I have been delighted to see that this actually gets me around the vector allocation error. I guess what is happening, is that the objects are all being read back in in a neatly packed manner, making more continuous free memory.

Thursday, April 9, 2009

The Girke R Bioconductor manual

I came across Thomas Girke's wonderful R/Bioconductor manual today. If I had had this UC Riverside professors notes when I started, I would have saved a lot of time.

Wednesday, March 25, 2009

CRAN Task Views: Your guide to R packages

If you're like me, you've at times been frustrated by the looseness of the R package system. These are some of the facts of life in the R world:
  • There are often many different functions with overlapping goals [for example, princomp() and prcomp()].
  • The names of a packages may have little or no relationship to the functions in that package (for example, package "e1071" provides the tune() function for parameter tuning).
  • There doesn't seem to be any organized, curated guide to the packages.
Well, at least the last point on this list of frustrations has been remedied. Be sure to check out the CRAN Task Views pages, which has exactly that: lists of packages with a brief blurb about each one, organized by topics (more than 20 at this writing), curated by (I assume) expert R users/developers.

My favorite guides at the moment are Machine Learning, and Multivariate Statistics.

Keep in mind, the above guides do not include BioConductor packages -- for that, you will still go to the BioConductor website.

R variable class (data types)

R does some amazing things, and often seems to know just want you want to do. Data type "coercion" can also backfire.

I just discovered a great way to check to see how R has interpreted the columns of a file you have just loaded.

> sapply(dataframeobject, class)
gender type score
"factor" "factor" "numeric"

Integrating BioMart queries into BioConductor with the biomaRt package

I've just realized the considerable power and convenience of a web-based bioinformatic resource I have overlooked in the past: biomart.org.

What gets my attention now is the ability to integrate BioMart queries right into your BioConductor pipeline, with a package called biomaRt

The Seattle 2009 BioConductor workshop looks like it was great -- sadly, I missed it. But you can still access some presentation materials on biomaRt.

Installation of biomaRt is accomplished like any other BioConductor pacakge:

    source("http://bioconductor.org/biocLite.R")
biocLite("biomaRt")
I am looking forward to learning more about these resources.

Thursday, March 19, 2009

Pulling data from GEO into R Bioconductor expression sets (eSet)

I just discovered how easy Bioconductor makes it to import data from the Gene Expression Omnibus (GEO).

First, be sure you have the GEOquery package from Bioconductor. You will probably need to install the curl4 devel library package using apt-get or whatever you use to do such things. On my Ubuntu 8.10 distro, the requisite package is called libcurl4-gnutls-dev. After you have this, you should be able to install GEOquery like any other Bioconductor package, ie:

source("http://bioconductor.org/biocLite.R")
biocLite("GEOquery")

Then, pulling a GEO series (GSE) into an eSet object is as simple as:

gseObj <- getGEO("GSE10667")
eSet <- gseObj[[1]]

Note the [[1]] -- necessary because the GSE comes in as a list of eSets, apparently.

See the GEOquery vignette for more details, and for conversions from GDS format.