WorkingWiki/Writing Makefiles in WorkingWiki

From Projects
Jump to: navigation, search

Makefiles are the heart of WorkingWiki

WorkingWiki is a tool that transforms a standard wiki, from a website that you can edit into an environment for processing digital data and documents. WorkingWiki is flexible and general-purpose: it transforms arbitrary input files into arbitrary output files. It handles LaTeX documents by transforming .tex files into .pdf files using latex. It handles R programs by transforming .R files into their text and image outputs, by running R, and it can transform other inputs into whatever outputs they produce, by running whatever programs are appropriate.

It's able to be this general because it relies on the strength of a general-purpose process-management tool called make. Make is controlled by files called makefiles, which tell it what programs to run to make inputs into outputs, and when to run them.

You can make use of the full power of WorkingWiki by understanding how to use the makefiles it provides, and how to customize it by adding your own makefiles.

What is a makefile?

A makefile is a text file that contains recipes, in a simple format, for how and when output files are to be made from input files. These recipes, called make rules, have three components: targets, commands, and dependencies. In a nutshell, a target (an output file) is made from its dependencies (input files) by running commands. These three components are arranged in the following structure:

targets : dependencies
	commands

We'll discuss how to write makefiles, and provide some examples, in later sections. For now, suffice it to say that you can, for instance, write a makefile saying how to run R programs and collect their output, or how to run latex to typeset .tex files into finished .pdf files.

We've written makefiles for those things, so you don't need to, but what if you need to use, say, a Python script instead? You can write a makefile saying how to run a python script and collect its output, and WorkingWiki will use it to run your script and display the output on a wiki page.

The built-in makefiles handle some common needs

WorkingWiki comes with centrally-located makefiles, which provide some general-purpose make rules that can be used in all projects. [It's possible to disable them for a given project: in our experience, this option is very rarely needed, if ever.]

WorkingWiki's built-in makefiles provide:

  • Rules for making .pdf files from .tex files (and the .bib and other files that work with them).
  • Rules for making .tex files and their accoutrements into HTML for display in wiki pages.
    • Unless instructed otherwise, when a .tex file appears in a wiki page, WorkingWiki will make it into HTML and display the HTML version in the page, and provide a link that produces the .pdf version.
  • Rules for making .png images from .eps, .pdf, .ps, and .svg files, which can't be displayed within web pages directly otherwise (at least in some browsers).
    • By default, when a .eps file appears in a wiki page, WorkingWiki will display it by using these rules to convert it to .png and displaying that. (Those other types are displayed in other ways, but the rules are there to be used as needed.)
  • a special target, wmd_export.tgz, which is a gzipped tar file containing the current project's source files, together with the central makefile resources and an auto-generated makefile that ensures that when you unpack the working directory and run make commands in it, they will do the right thing. [This is used by the Working Markup tools, but not by WorkingWiki, which currently uses PHP code to make .tgz files for export in more extensive ways.]

There are also general-purpose makefiles written for the lalashan.mcmaster.ca cluster of wikis, which provide extra features. Those makefiles don't come packaged with WorkingWiki, which means that if you're using WorkingWiki on some other site, you may not have access to these features. (All these things are free software, though, so you're welcome to use these make rules on outside sites if you copy and install them from https://github.com/worden-lee/site.git.) Since this documentation is for WorkingWiki in general, and we want it to be useful to both local and outside users, we'll document the extra features in indented gray text, to set it apart from the rest of the documentation.

The lalashan "site makefiles" provide:

  • An extensive system of rules for making text output, output images, and output data files from R scripts, and for chaining a series of R scripts together to do computations in steps.
  • Rules for running Gnuplot to produce plot images from .gpl files containing Gnuplot commands.
  • Rules to make particular pages of a PDF file into bitmap images for viewing in wiki pages.
  • A rule to display a .tex file in an alternative way, by making its .pdf output into a series of bitmap images, one for each page, and displaying them in the wiki page.
  • Rules to run Maxima (a computer math package) and capture its output.
  • Rules for a markup language called "DMU", which can be processed into wiki text (and displayed directly) and into LaTeX (and displayed/downloaded as PDF and HTML).
  • A rule for converting a .csv (Comma-Separated Values) database into a wikitext table, for human-friendly display.
  • A citation-markup language called "RMU", in which a simple format for recording scientific papers gets transformed into .bib files for use in LaTeX documents, as well as into good-looking formats for human viewing.
  • Rules for processing Knitr, Sweave, and R markdown files (typeset documents with R code embedded) into .tex files, which can then be displayed in the usual ways.
  • Utility rules for extracting the word count and line count of a text file, and extracting the error messages from an R output file.

They also instruct make to read in all source files with names ending in ".mk", which gives you a flexible way to add make rules as you go, by adding .mk files as you develop your project.

Controlling your project with custom makefiles

Here are some real-life examples of things people are doing by writing makefiles within WorkingWiki pages.

  • Writing Python scripts, displaying their output text and images, and using data they produce as input to other processing steps.
  • Writing C and C++ programs, compiling and running them within WorkingWiki, and using their output as input to other processes.
  • Writing a system for automatically retrieving social-science data sets from a government website, pre-processing them by doing merges and data-cleaning steps, and making them available as inputs to a range of data-processing projects.
  • Using third-party software to do DNA sequence analysis, maintain a large database of sequences, and output results into the wiki for use in manuscripts.
  • Writing Sage math programs that chain together in a sequence of modular computing steps, and displaying their output as text and images.
    • Writing Sage math programs that output math expressions in .tex form, and using WorkingWiki's LaTeX rules to display the math in the wiki page.
  • Developing a system for modular program steps that self-document their own dependencies, and are automatically transformed into both makefiles and the programs that the makefiles run.
  • Testing complex processes to make sure they perform as expected.
  • [add more here, please!]

Anatomy of a makefile

As seen above, the fundamental structure of a makefile is the rule:

targets : dependencies
	commands

The simplest case of a makefile is simply a file called Makefile, in the same directory as the files to be processed. It can contain any number of rules like this. (It can also include lines that set variables and do various other things.)

When asked to make a file, make looks for a rule that has that file as a target, checks the dependencies, and if needed, runs the commands to update the target.

For example, to make a .pdf file from a .tex file, you would provide, say, a file called myfile.tex file and a suitable makefile, and then run the command

make myfile.pdf

This would cause make to search its makefile for a suitable rule and execute it. If the makefile were to contain a rule

myfile.pdf : myfile.tex
	pdflatex myfile.tex

it would recognize that myfile.tex is there - that is, the dependencies for this rule are present - and it would run the command latex myfile.tex, which processes that file and produces the appropriate PDF output file.

This is the simplest form of a make rule: to make one thing from another thing, run a command.

Important: the long space before each command has to be a tab character - it can't be a series of regular spaces, or the makefile won't work. Because the Tab key doesn't work to insert tab characters into the edit form on a wiki, there's a shortcut button provided to insert tab characters. It looks like this: tab.

If you have multiple commands, put them on separate lines and they'll be run one after the other. If running the commands will produce multiple targets that you care about, put them together before the colon, with spaces between. If the process requires multiple input files, put them together after the colon with spaces between.

The role of dependencies

The dependencies are the files that are needed to make the targets. Make won't run the commands to make the target if it can't provide the dependencies.

But more interestingly, make also keeps track of which files are updated in what order. If a target file already exists, but any of its dependencies are newer than the target, make will rerun the commands to remake the target, making sure it's up to date.

It also does dependency chaining: if any of the dependency files can itself be made or remade, make will do that first, and then update the target using the updated inputs.

This is why you can, for instance, provide a make rule that generates a .tex file from a script, and then ask WorkingWiki to display it as a PDF file. Make will run your script to create the .tex file, and then use WorkingWiki's rules to make that into a PDF.

How to write a pattern rule.

Unlike the above examples, very often we want to provide a way to make particular kinds of outputs from particular kinds of inputs, rather than writing a separate rule for each output file. For example, a rule declaring how to run a Python script and collect its output, which can then be used with several different Python scripts.

This is done using pattern rules, which have the same overall form as the above rules, but with the special character % functioning as a wildcard. In the Python example, the rule might be

%.py.out : %.py
	python $< > $@

This functions as a near-infinite family of make rules, one for each target ending in ".py.out", to be made from a file of the same name but ending only in ".py".

This pattern rule illustrates two important make variables: $< stands for the name of the input file (a filename ending in .py), and $@ stands for the name of the target file (a filename ending in .py.out).

So this rule tells make the following: to make a .py.out file from a .py file, run python with the source filename as a command-line argument, and collect its output (the Unix ">" punctuation does that) into the target file.

Further make documentation

There is much more to know about make, and the better you understand it, the more effectively you can use WorkingWiki. WorkingWiki relies on GNU Make, which has some extra features that are also very useful. Probably the best online reference is the GNU make manual, which is also available via the info make command on a computer that has it installed.

Makefiles in WorkingWiki

In WorkingWiki, you provide source files using the <source-file> syntax in a wiki page (at least in simple cases). For example, to provide a Python script you can write:

<source-file filename="pythagorean.py">
import math;
print math.sqrt(3*3+4*4);
</source-file>

Then assuming it has access to the relevant make rules, you could request a target file using the <project-file> tag, on the same page:

<project-file filename="pythagorean.py.out"/>

This will cause WorkingWiki to save the source file to a working directory, run make pythagorean.py.out, and then display the target file.

All the source files on the page (more generally, all the source files in the project) are assembled in the working directory before any make operation. All output files are generated using make. This includes the display versions of source files (for instance, a .tex file is displayed by making it into a HTML target, and a .eps file is displayed by making it into a .png target) as well as <project-file> tags.

The above example of a Python script and its output file would not work with just those two files, because WorkingWiki does not provide a make rule declaring how to run Python to process that script. You must also provide a makefile that includes that rule.

How to add a makefile to a project

A makefile is just another source file to WorkingWiki, so the above Python project can be completed by adding that file to the page:

<source-file filename="Makefile">
%.py.out : %.py
	python $< > $@
</source-file>

That file will then be written into the working directory along with the Python code, and then when WorkingWiki runs make pythagorean.py.out, make will find the makefile and use that rule to process the source code. This is enough to give you the answer:

pythagorean.py.out
log
5.0

Using multiple makefiles

As your project develops, you can add more make rules and other make directives to that makefile by adding more lines within its source-file tag.

If you prefer to place your makefile code in multiple files, so that you can put each part of the makefile code near the files it affects, make needs to be able to find those files. It automatically reads any file called Makefile or makefile in the working directory (or GNUmakefile - see below), so you can do this by providing a Makefile with an include directive telling it to read in the other files: for instance,

<source-file filename="Makefile">
include Makefile.*
</source-file>

if all your other makefiles' names begin with "Makefile.".

Note: The lalashan wikis take care of that for you.

The lalashan cluster's site makefiles provide an include command to read in all source files with the .mk extension. So on these wikis you can simply add source files python.mk, how-to-make-my-tex-file.mk, etc., and they will be read by make automatically.

Examples

Some useful WorkingWiki makefile recipes

Use pdflatex on your .tex files

In a plain WorkingWiki project, it uses latex to make PDF files from LaTeX inputs. Some projects need to use pdflatex instead. This is controlled by the DEFAULT_LATEX makefile variable, and you can tell it to use a different latex program by assigning that variable.

On non-lalashan wikis, place the following line anywhere in your project's Makefile (create a Makefile if needed):

<source-file filename="Makefile">
# ...
DEFAULT_LATEX = pdflatex
</source-file>
On lalashan wikis, you can alternatively do it with a .mk file:
<source-file filename="default_latex.mk">
DEFAULT_LATEX = pdflatex
</source-file>

See WorkingWiki/Using LaTeX in WorkingWiki for more detail about this and the rest of the LaTeX makefile features.

Isolate your background jobs

While you're loading a page, WorkingWiki enforces a short time limit for each make command. To make targets that take longer, you need to make them in background make jobs.

It's helpful to isolate parts of the makefile that instruct it to do long jobs, so they won't be used when WorkingWiki is making a job with a short time limit, because that would be futile and would leave you waiting a long time for your page to load as it tries and fails to run a long command. WorkingWiki makes this isolation possible by setting a variable called FOREGROUND_ONLY, only when it is running time-limited "foreground" jobs. You can use this to protect yourself from long jobs using this pattern:

<source-file filename="Makefile">
harmless-target : harmless-dependencies
	short commands

ifndef FOREGROUND_ONLY
dangerous-target : dangerous-dependencies
	time consuming commands
endif
</source-file>

Placing slow commands within that ifndef...endif pair ensures that they won't be run while you are loading a page, but will be run in background jobs and when the project is exported (see below). The reason for the inverted logic (ifndef FOREGROUND_ONLY rather than ifdef BACKGROUND_ONLY) is so that it will do the right thing when you export the project to your own computer and run make commands at the command line, where no special variables will be set.

Connect multiple projects together

Each project corresponds to a working directory on the server where that project's files are processed. To connect one project to another - for instance, to have one project for statistical analysis code and another project for a manuscript including the results of the analysis - you must use make commands to point one project's processing commands to the location of the other project's files.

Let's say you want files from project A to be used in project B. First you must register project A as a "prerequisite project" on the project management page for project B. Open that page by clicking on the name of project B in the "projects" box on the sidebar of a page that uses files in project B. The list of prerequisite projects in toward the bottom of the project management page. There you will enter the name of project A, and a variable name that will represent it in make. Let's say the variable name is PROJECT_DIR_A.

Then you can write make rules in project B that have dependencies in project A:

manuscript-related-target : $(PROJECT_DIR_A)/analysis-result
	commands to make the target

There is some inconvenience involved in the fact that latex doesn't understand these variables. See WorkingWiki/Using_LaTeX_in_WorkingWiki#Spanning_multiple_projects for more on this. Multi-project example demonstrates a pattern for making a figure in one project and using it in a manuscript in another project.

Reading the WorkingWiki makefiles

Here are links to the sitewide makefiles, which are used in all projects except those that opt out of using them.

To do intermediate-level makefile programming, or troubleshoot problems in your projects, it may be important to look into these makefiles and learn how they work.

These links are listed in the order they're included by make.

Non-lalashan sites may provide other features in their site/makefile-before and site/makefile-after.

To do intermediate-level makefile programming, or troubleshoot problems in your projects, it may be important to look into these makefiles and learn how they work.

The automatic GNUmakefile

In a non-wiki setting, when you run a command like make target-file, make uses whatever makefile it finds in the current directory to provide its rules. In the wikis, on the other hand, WorkingWiki wants it to use the WorkingWiki rules as well, which are stored in a central directory. To get this to happen, WorkingWiki intercepts the command flow of make by inserting a file called GNUmakefile into the project's working directory.

This is a special filename, which causes make to use this WW-generated file as its makefile, even if the project also has a makefile or Makefile. This file does the following:

  • define some variables that tell make about the project's source files and the other projects it is connected to
  • read in WorkingWiki's makefile-before and the other "before" makefiles
  • read in the project's makefile or Makefile
  • read in WorkingWiki's makefile-after and the other "after" makefiles.
  • It also contains some makefile code for use when the project is exported for use outside the wikis.

You can generally ignore this, but it's worth knowing about because

  • you might wonder why that file is there and what to do with it
  • you sometimes need to edit it a little when you export a project (see Making things outside the wikis, below)
  • you might want to import an existing project that has a file called GNUmakefile
  • you might want to use a project that will work better without the WorkingWiki-specific makefiles.

In the last two cases, you want those projects to opt out of the GNUmakefile feature.

How to opt out and use only your project's makefiles

There's a check box at the bottom of the Special:ManageProject page to control whether the project uses the WorkingWiki makefiles. If you turn off that option, WorkingWiki will not add a GNUmakefile to the project, and make jobs in that project will use only the project's makefiles, without supplementation from the WorkingWiki makefiles.

To set or unset that option:

  • Click on the project's name in the 'projects' section of a wiki page's sidebar. This opens the Special:ManageProject page for that project.
  • Scroll to the bottom of the page, and set the check box labeled "Use WorkingWiki's supplemental make rules" to either checked or unchecked.
  • Click the "submit" button to the right of the checkbox.

Making things outside the wikis

An important reason for the use of make as the heart of WorkingWiki is that it can be used for reproducible research: making it as easy as possible for other researchers to repeat, verify, and build on our work. To facilitate reproducibility, it's very important to be able to take the source code out of a wiki and run it on someone else's computer.

There are several ways to export projects' files from the wikis for use outside:

Exporting tar files

You can download a copy of a project's files using the "export source files" or "export working directory" links from the ManageProject page.

This gives you a .tar.gz file containing either the source files, or all the files, from the project, plus its prerequisite projects, and the WorkingWiki resources directory (where the makefiles and other utilities are).

When you do it this way, the main project directory (the one you asked to export) should arrive with its makefiles correctly configured so that you can simply go into that directory and run commands like make target-file-xyz. Running make in the other (prerequisite) project directories from the command line may or may not work, because it may need variable definitions that are set by WW in the main project's makefiles.

Using the git interface

There is an experimental WorkingWiki Git interface that is able to "clone" a WorkingWiki project using a command like
git clone workingwiki::http://site-name/wiki-name:project-name

This gives you a directory containing the project's source files, plus

  • git data containing the project's full history, and
  • a copy of WorkingWiki's resources directory, including the makefiles.

With a simple project (one with no prerequisite projects), you should be able to go into that directory and run straightforward make commands.

In the git case, if the project has prerequisites, currently you have to clone each of them separately, and then edit your offline copy of the main project's GNUmakefile to give it the correct locations of those project directories. For example: if a project has project "Trillium" as a prerequisite, and your clone of Trillium is located at ~/trillium, then you need to locate a line like

export PROJECT_DIR_Trillium = ../Trillium

and change it to

export PROJECT_DIR_Trillium = ~/trillium

The Git interface is not currently (fall 2013) able to push changes back into the wiki, but this is planned for the future.

Projects may also require specialized software to be installed - R libraries, LaTeX packages, compilers, etc. - so you may also need to install those things in order to get an exported project to run.

Related links

Here are some outside resources about writing makefiles: