On Electronic Lab books

Digital labbooks (or electronic Labbooks as some like to call them) are not a novelty. They have been around in one form or another for a while. We have played with many options over the past years but none were really fully satisfactory.

Word/Pages/LibreOffice

For instance, take Word/Pages/LibreOffice documents. We know that doing a few pages is fine. But when books get big (over 10 pg) and with many figures, the files tend to get corrupted, are difficult to share (mostly due to version tracking), and none of these applications really handle well images: they get often misplaced and corrupted when you change application, format or operating system. Also, formats tend to be proprietary... But worst of all: collaboration is really tough. You surely had, at some point or another, to copy corrections made by 3 collaborators onto your offline labbook. It is no fun. So these are none starters.

Wiki

Our first attempt at an electronic labbook was actually a self hosted wiki. This was nice for sharing protocols, booking equipment, etc. But, people need quite a lot of training to learn how to use it properly. It is difficult to organize content and the maintenance of the site was tricky. For instance, once you upload an image you have no idea of where it will be stored in your filesystem in case you need to retrieve it. Then, when your server is down, your whole lab looses access to their labbooks! Finally, collaboration and tracking changes or commenting is almost impossible.

Google Docs

For many years now we used google docs for writing papers. They are great for collaborating, tracking changes, comments, etc. You can even look, comment, or correct content that lab members have added to their labbooks when you are in the train! Very nice. But again, for labbooks this system still has several pitfalls:

• when you write > 10 pages with images, loading and browsing is very slow. 
• your content has to live online. I am OK with that but many people are uneasy. Mostly, our government hates having all the data we produced being stored by US companies. Maybe they have a point there.
• Then, if you want to share across platforms, this is not ideal either as Google Drive does not play game with Linux (which is our daily driver).
• But, my worst nightmare is their proprietary format. If Google decides tomorrow to stop letting us use Docs for free - or worse- they discontinue the service altogether, we may find ourselves with years of Google Documents in their format that we can only partially convert.... Nightmare for long term storage and accessibility as needed for labbooks. Did you ever find yourself loving 'Inbox' or 'Picassa' or 'Google+' just to find from one day to the other that Google is dropping service in 3 months!

Markdown + Github

Finally, this summer, after looking for several possible options we settled for Markdown. This is a programming language with almost trivial syntax: It is a lightweight Latex. You don't need to know full fledged HTML. You can add images, videos, tables, equations, very very easily. And, there are many applications - online and offline- that support it. Typora, for instance, is a great one. It is so much nicer writing a paper in Typora than in Word or Google Docs! It is fast, reactive, and let't you concentrate on your writing. You can even add references!

But, most critically, it is simple text. This means that it can be stored in your filesystem or online. You own the file, not Google, and it will be readable for ever!

Here is an example screenshot of one of our Markdown Labbooks:

You may be wondering about collaboration. Well, our solution was simply to make our markdown files + figures + movies a Repository followed by git. You can then store it in Github or if you don't want Microsoft to look at your last week results, you can just have a self-hosted GitLab server (quite easy to setup using Docker).

This means that each lab member can have a local copy of their repository and can push it to github every time there are substantive changes (or this can be done automatically using cron!). You can then just pull the repository, make comments, and push them again.

Thus, (Markdown+Git) is perfect for version control (e.g. for tracking changes), for making suggestions, etc.

The only thing I am not yet happy about is the revision of changes (i.e merges). There are some awesome tools (e.g. offline: meld, diffuse, but also online) for merging changes. These work great for code. Unfortunately, merging changes works on lines, not on words. This means you need to either accept or reject changes in full paragraphs, not on a word-by-word basis. I am sure there will be soon tools to do this, but currently I have been unable to find any.

We also tested other online options that use Markdown and are starting to build tools for collaboration. For instance: HackMD / CodiMD, Authorea, StackEdit or Draft. All these applications keep your files online and usually in database format, and this makes it difficult to export them back into your filesystem to keep your data locally. Also, they are still not as good as professional word editors (Word/LibreOffice/Google Docs) at handling merges from multiple collaborators.

If you want to give it a try, checkout our GitHub repository describing how to make your own labbook using Markdown here.

'Project 1.md' in the root directory provides an example of a dummy Labbook with tables, links, headings and sub-headings. If you click in the link under the 'Create Labbook' heading you will access a second Markdown file 'Create_labbook.md' that describes how to actually proceed to create a Markdown Labbook, including basic syntax, applications, how to insert tables, images, etc.