Background Jobs

From projects

Some of our working wikis are now able to run background jobs, meaning start a process to make a given target, and come back and get the results later, without waiting for it to finish before the current wiki page shows in your browser.

Launching a background job

There are two ways to submit a background job:

• First is by using the "background make" button that is added to the ManageProject page for your project. Fill in the name of a target, the same way you would for a regular make operation, then click the "make in background" button.
• The second way to do it is with a project-file tag with the special attribute:

make="background":<project-file filename="XXYY" make="background"/>

Where this tag is placed in your wiki page, it will appear as a link which starts off a background job to make the target.

Resist the temptation to refresh the ManageProject page generated by this link; doing so will submit another copy of your job!

When you submit a background job, you will see a notice that a background job was created, identified by a unique number. There will also be a box showing the status of the job. While the job is running, you can kill it, and once it's finished (successfully or not), you can browse the background job's working files, discard the job's files, or merge the working files into the working directories of the projects that are included.

Here is what happens when a background job is created to make a target in a given project. First the system copies the working directories for the project in question and any dependency projects to a separate working directory tree. Then it calls on make to make the target in the new, separate project directory. When make is done, you have the option to merge the results into the main working directory for the project, and once you do that, the results will be visible in your project files on the wiki.

The background job's working directory tree is made by calling cp --recursive --preserve, which copies all the files in the project working directories, with their modification times intact. Then the projects sync all their source files into the background working directories. This only updates the source files if they've changed on their wiki pages since the last time something was made in the project (or in one that depends on it). So in general, the state of the background working directories should be the same as in the main working directories, and the background make will not make anything unless it's out of date.

The background make process runs in an only slightly different environment from WorkingWiki's standard foreground make processes. The main differences are that there is no time limit on background jobs, and that the environment variable FOREGROUND_ONLY is set in foreground processes and not in background processes. This allows you to mark some makefile rules to apply only in background jobs (or only in foreground jobs): more on this later.

Notes:

• Background make links and buttons are disabled during preview sessions. Save your work before starting off a background job.
• You can get the Wiki to send you an email when a background job finishes. For this, go to "My Preferences" (top menu on every wiki page), then in the "misc" section tick Email me when my WorkingWiki background jobs finish, and save.

How to retrieve your results

When a job's make process is finished, whether it succeeded or failed, you can merge its output into the main WorkingWiki project directories. Once you do that, the project files updated by the background job can be seen in wiki pages using regular project-file tags or the GetProjectFile interface.

The merge process is done using cp --recursive --update, which updates only the files that are newer in the background job's directory than in the main directory or don't exist in the main directory.

The merge operation destroys the copied directory and removes the job from the list of jobs.

Background jobs design pattern

The background jobs feature is intended for projects in which certain steps require substantial computing time. Normally WorkingWiki uses make to keep all project files up to date whenever they're displayed in a wiki page, and if a step in the dependency chain requires running a long simulation or data-processing step, it will time out, and users don't want to wait a long time for their page to appear anyway.

Using background jobs, you can isolate the intensive steps so that they don't run unless you explicitly ask for them to be run, and when they do run they don't get in the way of using the wiki. Here's one way to do it:

• Mark the intensive step(s) in your makefile to run only in background jobs using FOREGROUND_ONLY, like this:

normal-target : source-files
	commands that run quickly
 
ifndef FOREGROUND_ONLY
intensive-target : normal-files
	commands that run slowly
endif

• Include project files generated by the intensive process, since they won't be remade unless you create a background job to make them.
• Create a link to run the intensive process, like this:

  <project-file filename="intensive-target" project="project name" make="background" linktext="Click here!"/>

  When that target's dependencies have been modified, using the link causes the intensive target file to be remade in a background job.