Workflows¶
Although it would be possible to write analysis scripts using just Nipype Interfaces, and this may provide some advantages over directly making command-line calls, the main benefits of Nipype are the workflows.
A workflow controls the setup and the execution of individual interfaces. Let's assume you want to run multiple interfaces in a specific order, where some have to wait for others to finish while others can be executed in parallel. The nice thing about a nipype workflow is, that the workflow will take care of input and output of each interface and arrange the execution of each interface in the most efficient way.
A workflow therefore consists of multiple Nodes, each representing a specific Interface and directed connection between those nodes. Those connections specify which output of which node should be used as an input for another node. To better understand why this is so great, let's look at an example.
Interfaces vs. Workflows¶
Interfaces are the building blocks that solve well-defined tasks. We solve more complex tasks by combining interfaces with workflows:
| Interfaces | Workflows |
|---|---|
| Wrap *unitary* tasks | Wrap *meta*-tasks
|
| Keep track of the inputs and outputs, and check their expected types | Do not have inputs/outputs, but expose them from the interfaces wrapped inside |
| Do not cache results (unless you use [interface caching](advanced_interfaces_caching.ipynb)) | Cache results |
| Run by a nipype plugin | Run by a nipype plugin |
Preparation¶
Before we can start, let's first load some helper functions:
import numpy as np
import nibabel as nb
import matplotlib.pyplot as plt
# Let's create a short helper function to plot 3D NIfTI images
def plot_slice(fname):
# Load the image
img = nb.load(fname)
data = img.get_data()
# Cut in the middle of the brain
cut = int(data.shape[-1]/2) + 10
# Plot the data
plt.imshow(np.rot90(data[..., cut]), cmap="gray")
plt.gca().set_axis_off()
Example 1 - Command-line execution¶
Let's take a look at a small preprocessing analysis where we would like to perform the following steps of processing:
- Skullstrip an image to obtain a mask
- Smooth the original image
- Mask the smoothed image
This could all very well be done with the following shell script:
%%bash
ANAT_NAME=sub-01_ses-test_T1w
ANAT=/data/ds000114/sub-01/ses-test/anat/${ANAT_NAME}
bet ${ANAT} /output/${ANAT_NAME}_brain -m -f 0.3
fslmaths ${ANAT} -s 2 /output/${ANAT_NAME}_smooth
fslmaths /output/${ANAT_NAME}_smooth -mas /output/${ANAT_NAME}_brain_mask /output/${ANAT_NAME}_smooth_mask
This is simple and straightforward. We can see that this does exactly what we wanted by plotting the four steps of processing.
f = plt.figure(figsize=(12, 4))
for i, img in enumerate(["T1w", "T1w_smooth",
"T1w_brain_mask", "T1w_smooth_mask"]):
f.add_subplot(1, 4, i + 1)
if i == 0:
plot_slice("/data/ds000114/sub-01/ses-test/anat/sub-01_ses-test_%s.nii.gz" % img)
else:
plot_slice("/output/sub-01_ses-test_%s.nii.gz" % img)
plt.title(img)
Example 2 - Interface execution¶
Now let's see what this would look like if we used Nipype, but only the Interfaces functionality. It's simple enough to write a basic procedural script, this time in Python, to do the same thing as above:
from nipype.interfaces import fsl
# Skullstrip process
skullstrip = fsl.BET(
in_file="/data/ds000114/sub-01/ses-test/anat/sub-01_ses-test_T1w.nii.gz",
out_file="/output/sub-01_T1w_brain.nii.gz",
mask=True)
skullstrip.run()
# Smoothing process
smooth = fsl.IsotropicSmooth(
in_file="/data/ds000114/sub-01/ses-test/anat/sub-01_ses-test_T1w.nii.gz",
out_file="/output/sub-01_T1w_smooth.nii.gz",
fwhm=4)
smooth.run()
# Masking process
mask = fsl.ApplyMask(
in_file="/output/sub-01_T1w_smooth.nii.gz",
out_file="/output/sub-01_T1w_smooth_mask.nii.gz",
mask_file="/output/sub-01_T1w_brain_mask.nii.gz")
mask.run()
f = plt.figure(figsize=(12, 4))
for i, img in enumerate(["T1w", "T1w_smooth",
"T1w_brain_mask", "T1w_smooth_mask"]):
f.add_subplot(1, 4, i + 1)
if i == 0:
plot_slice("/data/ds000114/sub-01/ses-test/anat/sub-01_ses-test_%s.nii.gz" % img)
else:
plot_slice("/output/sub-01_%s.nii.gz" % img)
plt.title(img)
This is more verbose, although it does have its advantages. There's the automated input validation we saw previously, some of the options are named more meaningfully, and you don't need to remember, for example, that fslmaths' smoothing kernel is set in sigma instead of FWHM -- Nipype does that conversion behind the scenes.
Can't we optimize that a bit?¶
As we can see above, the inputs for the mask routine in_file and mask_file are actually the output of skullstrip and smooth. We therefore somehow want to connect them. This can be accomplished by saving the executed routines under a given object and then using the output of those objects as input for other routines.
from nipype.interfaces import fsl
# Skullstrip process
skullstrip = fsl.BET(
in_file="/data/ds000114/sub-01/ses-test/anat/sub-01_ses-test_T1w.nii.gz", mask=True)
bet_result = skullstrip.run() # skullstrip object
# Smooth process
smooth = fsl.IsotropicSmooth(
in_file="/data/ds000114/sub-01/ses-test/anat/sub-01_ses-test_T1w.nii.gz", fwhm=4)
smooth_result = smooth.run() # smooth object
# Mask process
mask = fsl.ApplyMask(in_file=smooth_result.outputs.out_file,
mask_file=bet_result.outputs.mask_file)
mask_result = mask.run()
f = plt.figure(figsize=(12, 4))
for i, img in enumerate([skullstrip.inputs.in_file, smooth_result.outputs.out_file,
bet_result.outputs.mask_file, mask_result.outputs.out_file]):
f.add_subplot(1, 4, i + 1)
plot_slice(img)
plt.title(img.split('/')[-1].split('.')[0].split('test_')[-1])
Here we didn't need to name the intermediate files; Nipype did that behind the scenes, and then we passed the result object (which knows those names) onto the next step in the processing stream. This is somewhat more concise than the example above, but it's still a procedural script. And the dependency relationship between the stages of processing is not particularly obvious. To address these issues, and to provide solutions to problems we might not know we have yet, Nipype offers Workflows.
Example 3 - Workflow execution¶
What we've implicitly done above is to encode our processing stream as a directed acyclic graphs: each stage of processing is a node in this graph, and some nodes are unidirectionally dependent on others. In this case, there is one input file and several output files, but there are no cycles -- there's a clear line of directionality to the processing. What the Node and Workflow classes do is make these relationships more explicit.
The basic architecture is that the Node provides a light wrapper around an Interface. It exposes the inputs and outputs of the Interface as its own, but it adds some additional functionality that allows you to connect Nodes into a Workflow.
Let's rewrite the above script with these tools:
# Import Node and Workflow object and FSL interface
from nipype import Node, Workflow
from nipype.interfaces import fsl
# For reasons that will later become clear, it's important to
# pass filenames to Nodes as absolute paths
from os.path import abspath
in_file = abspath("/data/ds000114/sub-01/ses-test/anat/sub-01_ses-test_T1w.nii.gz")
# Skullstrip process
skullstrip = Node(fsl.BET(in_file=in_file, mask=True), name="skullstrip")
# Smooth process
smooth = Node(fsl.IsotropicSmooth(in_file=in_file, fwhm=4), name="smooth")
# Mask process
mask = Node(fsl.ApplyMask(), name="mask")
This looks mostly similar to what we did above, but we've left out the two crucial inputs to the ApplyMask step. We'll set those up by defining a Workflow object and then making connections among the Nodes.
# Initiation of a workflow
wf = Workflow(name="smoothflow", base_dir="/output/working_dir")
The Workflow object has a method called connect that is going to do most of the work here. This routine also checks if inputs and outputs are actually provided by the nodes that are being connected.
There are two different ways to call connect:
connect(source, "source_output", dest, "dest_input")
connect([(source, dest, [("source_output1", "dest_input1"),
("source_output2", "dest_input2")
])
])
With the first approach, you can establish one connection at a time. With the second you can establish multiple connects between two nodes at once. In either case, you're providing it with four pieces of information to define the connection:
- The source node object
- The name of the output field from the source node
- The destination node object
- The name of the input field from the destination node
We'll illustrate each method in the following cell:
# First the "simple", but more restricted method
wf.connect(skullstrip, "mask_file", mask, "mask_file")
# Now the more complicated method
wf.connect([(smooth, mask, [("out_file", "in_file")])])
Now the workflow is complete!
Above, we mentioned that the workflow can be thought of as a directed acyclic graph. In fact, that's literally how it's represented behind the scenes, and we can use that to explore the workflow visually:
wf.write_graph("workflow_graph.dot")
from IPython.display import Image
Image(filename="/output/working_dir/smoothflow/workflow_graph.png")
This representation makes the dependency structure of the workflow obvious. (By the way, the names of the nodes in this graph are the names we gave our Node objects above, so pick something meaningful for those!)
Certain graph types also allow you to further inspect the individual connections between the nodes. For example:
wf.write_graph(graph2use='flat')
from IPython.display import Image
Image(filename="/output/working_dir/smoothflow/graph_detailed.png")
Here you see very clearly, that the output mask_file of the skullstrip node is used as the input mask_file of the mask node. For more information on graph visualization, see the Graph Visualization section.
But let's come back to our example. At this point, all we've done is define the workflow. We haven't executed any code yet. Much like Interface objects, the Workflow object has a run method that we can call so that it executes. Let's do that and then examine the results.
# Specify the base directory for the working directory
wf.base_dir = "/output/working_dir"
# Execute the workflow
wf.run()
The specification of base_dir is very important (and is why we needed to use absolute paths above) because otherwise all the outputs would be saved somewhere in the temporary files. Unlike interfaces, which by default spit out results to the local directly, the Workflow engine executes things off in its own directory hierarchy.
Let's take a look at the resulting images to convince ourselves we've done the same thing as before:
f = plt.figure(figsize=(12, 4))
for i, img in enumerate(["/data/ds000114/sub-01/ses-test/anat/sub-01_ses-test_T1w.nii.gz",
"/output/working_dir/smoothflow/smooth/sub-01_ses-test_T1w_smooth.nii.gz",
"/output/working_dir/smoothflow/skullstrip/sub-01_ses-test_T1w_brain_mask.nii.gz",
"/output/working_dir/smoothflow/mask/sub-01_ses-test_T1w_smooth_masked.nii.gz"]):
f.add_subplot(1, 4, i + 1)
plot_slice(img)
Perfect!
Let's also have a closer look at the working directory:
!tree /output/working_dir/smoothflow/ -I '*js|*json|*html|*pklz|_report'
As you can see, the name of the working directory is the name we gave the workflow base_dir. And the name of the folder within is the name of the workflow object smoothflow. Each node of the workflow has its' own subfolder in the smoothflow folder. And each of those subfolders contains the output of the node as well as some additional files.
The #1 gotcha of nipype Workflows¶
Nipype workflows are just DAGs (Directed Acyclic Graphs) that the runner Plugin takes in and uses to compose an ordered list of nodes for execution. As a matter of fact, running a workflow will return a graph object. That's why you often see something like <networkx.classes.digraph.DiGraph at 0x7f83542f1550> at the end of execution stream when running a workflow.
The principal implication is that Workflows don't have inputs and outputs, you can just access them through the Node decoration.
In practical terms, this has one clear consequence: from the resulting object of the workflow execution, you don't generally have access to the value of the outputs of the interfaces. This is particularly true for Plugins with an asynchronous execution.
A workflow inside a workflow¶
When you start writing full-fledged analysis workflows, things can get quite complicated. Some aspects of neuroimaging analysis can be thought of as a coherent step at a level more abstract than the execution of a single command line binary. For instance, in the standard FEAT script in FSL, several calls are made in the process of using susan to perform nonlinear smoothing on an image. In Nipype, you can write nested workflows, where a sub-workflow can take the place of a Node in a given script.
Let's use the prepackaged susan workflow that ships with Nipype to replace our Gaussian filtering node and demonstrate how this works.
from nipype.workflows.fmri.fsl import create_susan_smooth
Calling this function will return a pre-written Workflow object:
susan = create_susan_smooth(separate_masks=False)
Let's display the graph to see what happens here.
susan.write_graph("susan_workflow.dot")
from IPython.display import Image
Image(filename="susan_workflow.png")
We see that the workflow has an inputnode and an outputnode. While not strictly necessary, this is standard practice for workflows (especially those that are intended to be used as nested workflows in the context of a longer analysis graph) and makes it more clear how to connect inputs and outputs from this workflow.
Let's take a look at what those inputs and outputs are. Like Nodes, Workflows have inputs and outputs attributes that take a second sub-attribute corresponding to the specific node we want to make connections to.
print("Inputs:\n", susan.inputs.inputnode)
print("Outputs:\n", susan.outputs.outputnode)
Note that inputnode and outputnode are just conventions, and the Workflow object exposes connections to all of its component nodes:
susan.inputs
Let's see how we would write a new workflow that uses this nested smoothing step.
The susan workflow actually expects to receive and output a list of files (it's intended to be executed on each of several runs of fMRI data). We'll cover exactly how that works in later tutorials, but for the moment we need to add an additional Function node to deal with the fact that susan is outputting a list. We can use a simple lambda function to do this:
from nipype import Function
extract_func = lambda list_out: list_out[0]
list_extract = Node(Function(input_names=["list_out"],
output_names=["out_file"],
function=extract_func),
name="list_extract")
Now let's create a new workflow susanflow that contains the susan workflow as a sub-node. To be sure, let's also recreate the skullstrip and the mask node from the examples above.
# Initiate workflow with name and base directory
wf2 = Workflow(name="susanflow", base_dir="/output/working_dir")
# Create new skullstrip and mask nodes
skullstrip2 = Node(fsl.BET(in_file=in_file, mask=True), name="skullstrip")
mask2 = Node(fsl.ApplyMask(), name="mask")
# Connect the nodes to each other and to the susan workflow
wf2.connect([(skullstrip2, mask2, [("mask_file", "mask_file")]),
(skullstrip2, susan, [("mask_file", "inputnode.mask_file")]),
(susan, list_extract, [("outputnode.smoothed_files",
"list_out")]),
(list_extract, mask2, [("out_file", "in_file")])
])
# Specify the remaining input variables for the susan workflow
susan.inputs.inputnode.in_files = abspath(
"/data/ds000114/sub-01/ses-test/anat/sub-01_ses-test_T1w.nii.gz")
susan.inputs.inputnode.fwhm = 4
First, let's see what this new processing graph looks like.
wf2.write_graph(dotfilename='/output/working_dir/full_susanflow.dot', graph2use='colored')
from IPython.display import Image
Image(filename="/output/working_dir/full_susanflow.png")
We can see how there is a nested smoothing workflow (blue) in the place of our previous smooth node. This provides a very detailed view, but what if you just wanted to give a higher-level summary of the processing steps? After all, that is the purpose of encapsulating smaller streams in a nested workflow. That, fortunately, is an option when writing out the graph:
wf2.write_graph(dotfilename='/output/working_dir/full_susanflow_toplevel.dot', graph2use='orig')
from IPython.display import Image
Image(filename="/output/working_dir/full_susanflow_toplevel.png")
That's much more manageable. Now let's execute the workflow
wf2.run()
As a final step, let's look at the input and the output. It's exactly what we wanted.
f = plt.figure(figsize=(12, 4))
for i, e in enumerate([["/data/ds000114/sub-01/ses-test/anat/sub-01_ses-test_T1w.nii.gz", 'input'],
["/output/working_dir//susanflow/mask/sub-01_ses-test_T1w_smooth_masked.nii.gz",
'output']]):
f.add_subplot(1, 2, i + 1)
plot_slice(e[0])
plt.title(e[1])
So, why are workflows so great?¶
So far, we've seen that you can build up rather complex analysis workflows. But at the moment, it's not been made clear why this is worth the extra trouble from writing a simple procedural script. To demonstrate the first added benefit of the Nipype, let's just rerun the susanflow workflow from above and measure the execution times.
%time wf2.run()
That happened quickly! Workflows (actually this is handled by the Node code) are smart and know if their inputs have changed from the last time they are run. If they have not, they don't recompute; they just turn around and pass out the resulting files from the previous run. This is done on a node-by-node basis, also.
Let's go back to the first workflow example. What happened if we just tweak one thing:
wf.inputs.smooth.fwhm = 1
wf.run()
By changing an input value of the smooth node, this node will be re-executed. This triggers a cascade such that any file depending on the smooth node (in this case, the mask node, also recompute). However, the skullstrip node hasn't changed since the first time it ran, so it just coughed up its original files.
That's one of the main benefits of using Workflows: efficient recomputing.
Another benefit of Workflows is parallel execution, which is covered under Plugins and Distributed Computing. With Nipype it is very easy to up a workflow to an extremely parallel cluster computing environment.
In this case, that just means that the skullstrip and smooth Nodes execute together, but when you scale up to Workflows with many subjects and many runs per subject, each can run together, such that (in the case of unlimited computing resources), you could process 50 subjects with 10 runs of functional data in essentially the time it would take to process a single run.
To emphasize the contribution of Nipype here, you can write and test your workflow on one subject computing on your local CPU, where it is easier to debug. Then, with the change of a single function parameter, you can scale your processing up to a 1000+ node SGE cluster.
Exercise 1¶
Create a workflow that connects three nodes for:
- skipping the first 3 dummy scans using
fsl.ExtractROI - applying motion correction using
fsl.MCFLIRT(register to the mean volume, use NIFTI as output type) - correcting for slice wise acquisition using
fsl.SliceTimer(assumed that slices were acquired with interleaved order and time repetition was 2.5, use NIFTI as output type)
# write your solution here
# importing Node and Workflow
from nipype import Workflow, Node
# importing all interfaces
from nipype.interfaces.fsl import ExtractROI, MCFLIRT, SliceTimer
Defining all nodes
# extracting all time levels but not the first four
extract = Node(ExtractROI(t_min=4, t_size=-1, output_type='NIFTI'),
name="extract")
# using MCFLIRT for motion correction to the mean volume
mcflirt = Node(MCFLIRT(mean_vol=True,
output_type='NIFTI'),
name="mcflirt")
# correcting for slice wise acquisition (acquired with interleaved order and time repetition was 2.5)
slicetimer = Node(SliceTimer(interleaved=True,
output_type='NIFTI',
time_repetition=2.5),
name="slicetimer")
Creating a workflow
# Initiation of a workflow
wf_ex1 = Workflow(name="exercise1", base_dir="/output/working_dir")
# connect nodes with each other
wf_ex1.connect([(extract, mcflirt, [('roi_file', 'in_file')]),
(mcflirt, slicetimer, [('out_file', 'in_file')])])
# providing a input file for the first extract node
extract.inputs.in_file = "/data/ds000114/sub-01/ses-test/func/sub-01_ses-test_task-fingerfootlips_bold.nii.gz"
Exercise 2¶
Visualize and run the workflow
# write your solution here
We learnt 2 methods of plotting graphs:
wf_ex1.write_graph("workflow_graph.dot")
from IPython.display import Image
Image(filename="/output/working_dir/exercise1/workflow_graph.png")
And more detailed graph:
wf_ex1.write_graph(graph2use='flat')
from IPython.display import Image
Image(filename="/output/working_dir/exercise1/graph_detailed.png")
if everything works good, we're ready to run the workflow:
wf_ex1.run()
we can now check the output:
! ls -lh /output/working_dir/exercise1