#picklebox

Here is the R script regarding quality: http://rpubs.com/jfmeadow/27882

Mostly it came down to really low reverse read quality that tanked the total output number, as seen here:

Here is the workflow I am currently using to process sequence data using a combination of QIIME and UPARSE. Ann W put this together based on Mike Robeson's post here.

The only step that takes lots of time is the OTU table python script. No idea why it is so slow, and it might be worth rewriting that script to speed things up.

Github repository here: https://github.com/jfmeadow/ReproducibleDemo

Slides here: https://dl.dropboxusercontent.com/u/62653716/ReproDemo.pdf

This is a topic I've also been asked to talk about during my upcoming visit to San Luis Potosi, Mexico. So I'll get to reuse this potentially several times.

We published the Lillis Dust paper in PLOS ONE this week, so I've also been fielding press for that. It has shown up in Gizmodo, Popular Science, Quartz, Fast Company, Futurity, and lots of other news sites that just printed the press release. So that's fun.

The surface paper was also accepted in Microbiome Journal so that will finally be published soon.

As for manuscript writing in markdown, it is going really well. Except tables are a real bummer. Converting tables between markdown, latex, and word is a hassle, and editing tables in anything other than excel or R is also a hassle. So for now, I'm keeping tables out (in their own excel file) until the final version. Then they can be plugged in pretty easily. Other than that, manuscript writing in markdown is wonderful and pretty much hassle free!

I have also been re-preparing all raw Lillis sequence data to be put into the QIIME DB. The previous efforts to get the data in the shape they wanted didn't work, so instead they have added dual-indexing capability to the DB. Adam and I prepared a full document detailing all sequence files, metadata, barcodes, and sequence format. If they are able to get these datasets into the DB, it will be a much better solution. For now, the indoor evol meta-analysis is put on hold until the situation is more amicable. Rachel, Holly, Ashley and I agreed that we should hold off until about January and reassess. Holly has some great ideas about how to approach the situation without the standard QIIME analysis pipeline which might help us combine all of these disparate datasets.

After resubmitting the Lillis dust and surface manuscripts in the last 2 weeks, I've been spending most of my time working on the pickle box manuscript. This ideally needs to be submitted before the next couple of weeks, but plenty of work remains. Since analysis for this project is so complex, and the datasets are so big, I have broken R markdown scripts into:

analysis where raw data are processed and heavy stats routines are conducted

results where stat outputs are finalized and tables, tests, are output

figures where figures are created.

However, this becomes unwieldy in a hurry, so I think in the future, for similarly complex projects, it would be best to break into:

processing where raw data are formatted

several different results scripts to answer different questions

several different figures scripts for each individual figure.

The results scripts should be compartmentalized because workspaces become really big in a hurry when all analyses are combined. So some routines, such as indicator analysis, should be computed separately so as not to bog down one big analysis script.

The figures scripts really suck to work on inside of RStudio (where knitr works best), because of the crappy way that RStudio handles figure windows. Harder to output to pdf, harder to tweak on the fly, etc. So now I've taken to creating a standalone script for each figure so that the whole script can be sourced every time a change is made. Then the final version can be put into R markdown format to be curated.