Nipype Showcase

What's all the hype about Nipype? Is it really that good? Short answer: Yes!

Long answer: ... well, let's consider a very simple fMRI preprocessing workflow that just performs:

  1. slice time correction
  2. motion correction
  3. smoothing

Preparing the preprocessing workflow

First, we need to import the main Nipype tools: Node and Workflow

In [ ]:
from nipype import Node, Workflow

Now, we can import the interfaces that we want to use for the preprocessing.

In [ ]:
from nipype.interfaces.fsl import SliceTimer, MCFLIRT, Smooth

Next, we will put the three interfaces into a node and define the specific input parameters.

In [ ]:
# Initiate a node to correct for slice wise acquisition
slicetimer = Node(SliceTimer(index_dir=False,
                             interleaved=True,
                             time_repetition=2.5),
                  name="slicetimer")
In [ ]:
# Initiate a node to correct for motion
mcflirt = Node(MCFLIRT(mean_vol=True,
                       save_plots=True),
               name="mcflirt")
In [ ]:
# Initiate a node to smooth functional images
smooth = Node(Smooth(fwhm=4), name="smooth")

After creating the nodes, we can now create the preprocessing workflow.

In [ ]:
preproc01 = Workflow(name='preproc01', base_dir='.')

Now, we can put all the nodes into this preprocessing workflow. We specify the data flow / execution flow of the workflow by connecting the corresponding nodes to each other.

In [ ]:
preproc01.connect([(slicetimer, mcflirt, [('slice_time_corrected_file', 'in_file')]),
                   (mcflirt, smooth, [('out_file', 'in_file')])])

To better understand what we did we can write out the workflow graph and visualize it directly in this notebook.

In [ ]:
preproc01.write_graph(graph2use='orig')

180514-09:05:47,919 workflow INFO:
	 Generated workflow graph: /home/neuro/nipype_tutorial/notebooks/preproc01/graph.png (graph2use=orig, simple_form=True).
Out[ ]:
'/home/neuro/nipype_tutorial/notebooks/preproc01/graph.png'
In [ ]:
# Visualize graph
from IPython.display import Image
Image(filename="preproc01/graph_detailed.png")
Out[ ]:

Run the workflow on one functional image

Now, that we've created a workflow, let's run it on a functional image.

For this, we first need to specify the input file of the very first node, i.e. the slicetimer node.

In [ ]:
slicetimer.inputs.in_file = '/data/ds000114/sub-01/ses-test/func/sub-01_ses-test_task-fingerfootlips_bold.nii.gz'

To show off Nipype's parallelization power, let's run the workflow in parallel, on 5 processors and let's show the execution time:

In [ ]:
%time preproc01.run('MultiProc', plugin_args={'n_procs': 5})

180514-09:05:47,994 workflow INFO:
	 Workflow preproc01 settings: ['check', 'execution', 'logging', 'monitoring']
180514-09:05:48,0 workflow INFO:
	 Running in parallel.
180514-09:05:48,4 workflow INFO:
	 [MultiProc] Running 0 tasks, and 1 jobs ready. Free memory (GB): 53.94/53.94, Free processors: 5/5.
180514-09:05:48,55 workflow INFO:
	 [Node] Setting-up "preproc01.slicetimer" in "/home/neuro/nipype_tutorial/notebooks/preproc01/slicetimer".
180514-09:05:48,65 workflow INFO:
	 [Node] Running "slicetimer" ("nipype.interfaces.fsl.preprocess.SliceTimer"), a CommandLine Interface with command:
slicetimer --in=/data/ds000114/sub-01/ses-test/func/sub-01_ses-test_task-fingerfootlips_bold.nii.gz --odd --out=/home/neuro/nipype_tutorial/notebooks/preproc01/slicetimer/sub-01_ses-test_task-fingerfootlips_bold_st.nii.gz --repeat=2.500000
180514-09:05:50,6 workflow INFO:
	 [MultiProc] Running 1 tasks, and 0 jobs ready. Free memory (GB): 53.74/53.94, Free processors: 4/5.
                     Currently running:
                       * preproc01.slicetimer
180514-09:05:56,323 workflow INFO:
	 [Node] Finished "preproc01.slicetimer".
180514-09:05:58,15 workflow INFO:
	 [Job 0] Completed (preproc01.slicetimer).
180514-09:05:58,19 workflow INFO:
	 [MultiProc] Running 0 tasks, and 1 jobs ready. Free memory (GB): 53.94/53.94, Free processors: 5/5.
180514-09:05:58,56 workflow INFO:
	 [Node] Setting-up "preproc01.mcflirt" in "/home/neuro/nipype_tutorial/notebooks/preproc01/mcflirt".
180514-09:05:58,65 workflow INFO:
	 [Node] Running "mcflirt" ("nipype.interfaces.fsl.preprocess.MCFLIRT"), a CommandLine Interface with command:
mcflirt -in /home/neuro/nipype_tutorial/notebooks/preproc01/slicetimer/sub-01_ses-test_task-fingerfootlips_bold_st.nii.gz -meanvol -out /home/neuro/nipype_tutorial/notebooks/preproc01/mcflirt/sub-01_ses-test_task-fingerfootlips_bold_st_mcf.nii.gz -plots
180514-09:06:00,16 workflow INFO:
	 [MultiProc] Running 1 tasks, and 0 jobs ready. Free memory (GB): 53.74/53.94, Free processors: 4/5.
                     Currently running:
                       * preproc01.mcflirt
180514-09:07:07,646 workflow INFO:
	 [Node] Finished "preproc01.mcflirt".
180514-09:07:08,87 workflow INFO:
	 [Job 1] Completed (preproc01.mcflirt).
180514-09:07:08,94 workflow INFO:
	 [MultiProc] Running 0 tasks, and 1 jobs ready. Free memory (GB): 53.94/53.94, Free processors: 5/5.
180514-09:07:08,131 workflow INFO:
	 [Node] Setting-up "preproc01.smooth" in "/home/neuro/nipype_tutorial/notebooks/preproc01/smooth".
180514-09:07:08,140 workflow INFO:
	 [Node] Running "smooth" ("nipype.interfaces.fsl.utils.Smooth"), a CommandLine Interface with command:
fslmaths /home/neuro/nipype_tutorial/notebooks/preproc01/mcflirt/sub-01_ses-test_task-fingerfootlips_bold_st_mcf.nii.gz -kernel gauss 1.699 -fmean sub-01_ses-test_task-fingerfootlips_bold_st_mcf_smooth.nii.gz
180514-09:07:10,88 workflow INFO:
	 [MultiProc] Running 1 tasks, and 0 jobs ready. Free memory (GB): 53.74/53.94, Free processors: 4/5.
                     Currently running:
                       * preproc01.smooth
180514-09:07:14,863 workflow INFO:
	 [Node] Finished "preproc01.smooth".
180514-09:07:16,95 workflow INFO:
	 [Job 2] Completed (preproc01.smooth).
180514-09:07:16,100 workflow INFO:
	 [MultiProc] Running 0 tasks, and 0 jobs ready. Free memory (GB): 53.94/53.94, Free processors: 5/5.
CPU times: user 1.42 s, sys: 58.4 ms, total: 1.48 s
Wall time: 1min 30s
Out[ ]:
<networkx.classes.digraph.DiGraph at 0x7f15415c9160>

Conclusion

Nice, the whole execution took ~2min. But wait... The parallelization didn't really help.

That's true, but because there was no possibility to run the workflow in parallel. Each node depends on the output of the previous node.

Results of preproc01

So, what did we get? Let's look at the output folder preproc01:

In [ ]:
!tree preproc01 -I '*js|*json|*pklz|_report|*.dot|*html'

preproc01
├── graph_detailed.png
├── graph.png
├── mcflirt
│   ├── command.txt
│   └── sub-01_ses-test_task-fingerfootlips_bold_st_mcf.nii.gz
├── slicetimer
│   ├── command.txt
│   └── sub-01_ses-test_task-fingerfootlips_bold_st.nii.gz
└── smooth
    ├── command.txt
    └── sub-01_ses-test_task-fingerfootlips_bold_st_mcf_smooth.nii.gz

3 directories, 8 files

Rerunning of a workflow

Now, for fun. Let's run the workflow again, but let's change the fwhm value of the Gaussian smoothing kernel to 2.

In [ ]:
smooth.inputs.fwhm = 2

And let's run the workflow again.

In [ ]:
%time preproc01.run('MultiProc', plugin_args={'n_procs': 5})

180514-09:07:18,760 workflow INFO:
	 Workflow preproc01 settings: ['check', 'execution', 'logging', 'monitoring']
180514-09:07:18,766 workflow INFO:
	 Running in parallel.
180514-09:07:18,769 workflow INFO:
	 [MultiProc] Running 0 tasks, and 1 jobs ready. Free memory (GB): 53.94/53.94, Free processors: 5/5.
180514-09:07:18,813 workflow INFO:
	 [Job 0] Cached (preproc01.slicetimer).
180514-09:07:20,812 workflow INFO:
	 [Job 1] Cached (preproc01.mcflirt).
180514-09:07:22,811 workflow INFO:
	 [Node] Outdated cache found for "preproc01.smooth".
180514-09:07:22,816 workflow INFO:
	 [Node] Setting-up "preproc01.smooth" in "/home/neuro/nipype_tutorial/notebooks/preproc01/smooth".
180514-09:07:22,822 workflow INFO:
	 [Node] Outdated cache found for "preproc01.smooth".
180514-09:07:22,845 workflow INFO:
	 [Node] Running "smooth" ("nipype.interfaces.fsl.utils.Smooth"), a CommandLine Interface with command:
fslmaths /home/neuro/nipype_tutorial/notebooks/preproc01/mcflirt/sub-01_ses-test_task-fingerfootlips_bold_st_mcf.nii.gz -kernel gauss 0.849 -fmean sub-01_ses-test_task-fingerfootlips_bold_st_mcf_smooth.nii.gz
180514-09:07:24,775 workflow INFO:
	 [MultiProc] Running 1 tasks, and 0 jobs ready. Free memory (GB): 53.74/53.94, Free processors: 4/5.
                     Currently running:
                       * preproc01.smooth
180514-09:07:28,808 workflow INFO:
	 [Node] Finished "preproc01.smooth".
180514-09:07:30,782 workflow INFO:
	 [Job 2] Completed (preproc01.smooth).
180514-09:07:30,789 workflow INFO:
	 [MultiProc] Running 0 tasks, and 0 jobs ready. Free memory (GB): 53.94/53.94, Free processors: 5/5.
CPU times: user 269 ms, sys: 48.4 ms, total: 318 ms
Wall time: 14 s
Out[ ]:
<networkx.classes.digraph.DiGraph at 0x7f15415c90b8>

Conclusion

Interesting, now it only took ~15s to execute the whole workflow again. What happened?

As you can see from the log above, Nipype didn't execute the two nodes slicetimer and mclfirt again. This, because their input values didn't change from the last execution. The preproc01 workflow therefore only had to rerun the node smooth.

Running a workflow in parallel

Ok, ok... Rerunning a workflow again is faster. That's nice and all, but I want more. You spoke of parallel execution!

We saw that the preproc01 workflow takes about ~2min to execute completely. So, if we would run the workflow on five functional images, it should take about ~10min total. This, of course, assuming the execution will be done sequentially. Now, let's see how long it takes if we run it in parallel.

In [ ]:
# First, let's copy/clone 'preproc01'
preproc02 = preproc01.clone('preproc02')
preproc03 = preproc01.clone('preproc03')
preproc04 = preproc01.clone('preproc04')
preproc05 = preproc01.clone('preproc05')

We now have five different preprocessing workflows. If we want to run them in parallel, we can put them all in another workflow.

In [ ]:
metaflow = Workflow(name='metaflow', base_dir='.')
In [ ]:
# Now we can add the five preproc workflows to the bigger metaflow
metaflow.add_nodes([preproc01, preproc02, preproc03,
                    preproc04, preproc05])

Note: We now have a workflow (metaflow), that contains five other workflows (preproc0?), each of them containing three nodes.

To better understand this, let's visualize this metaflow.

In [ ]:
# As before, let's write the graph of the workflow
metaflow.write_graph(graph2use='flat')

180514-09:07:33,114 workflow INFO:
	 Generated workflow graph: /home/neuro/nipype_tutorial/notebooks/metaflow/graph.png (graph2use=flat, simple_form=True).
Out[ ]:
'/home/neuro/nipype_tutorial/notebooks/metaflow/graph.png'
In [ ]:
# And visualize the graph
from IPython.display import Image
Image(filename="metaflow/graph_detailed.png")
Out[ ]:

Ah... so now we can see that the metaflow has potential for parallelization. So let's put it to test

In [ ]:
%time metaflow.run('MultiProc', plugin_args={'n_procs': 5})

180514-09:07:33,160 workflow INFO:
	 Workflow metaflow settings: ['check', 'execution', 'logging', 'monitoring']
180514-09:07:33,171 workflow INFO:
	 Running in parallel.
180514-09:07:33,174 workflow INFO:
	 [MultiProc] Running 0 tasks, and 5 jobs ready. Free memory (GB): 53.94/53.94, Free processors: 5/5.
180514-09:07:33,223 workflow INFO:
	 [Node] Setting-up "metaflow.preproc01.slicetimer" in "/home/neuro/nipype_tutorial/notebooks/metaflow/preproc01/slicetimer".180514-09:07:33,225 workflow INFO:
	 [Node] Setting-up "metaflow.preproc02.slicetimer" in "/home/neuro/nipype_tutorial/notebooks/metaflow/preproc02/slicetimer".

180514-09:07:33,227 workflow INFO:
	 [Node] Setting-up "metaflow.preproc03.slicetimer" in "/home/neuro/nipype_tutorial/notebooks/metaflow/preproc03/slicetimer".
180514-09:07:33,229 workflow INFO:
	 [Node] Setting-up "metaflow.preproc04.slicetimer" in "/home/neuro/nipype_tutorial/notebooks/metaflow/preproc04/slicetimer".180514-09:07:33,231 workflow INFO:
	 [Node] Setting-up "metaflow.preproc05.slicetimer" in "/home/neuro/nipype_tutorial/notebooks/metaflow/preproc05/slicetimer".
180514-09:07:33,232 workflow INFO:
	 [Node] Running "slicetimer" ("nipype.interfaces.fsl.preprocess.SliceTimer"), a CommandLine Interface with command:
slicetimer --in=/data/ds000114/sub-01/ses-test/func/sub-01_ses-test_task-fingerfootlips_bold.nii.gz --odd --out=/home/neuro/nipype_tutorial/notebooks/metaflow/preproc01/slicetimer/sub-01_ses-test_task-fingerfootlips_bold_st.nii.gz --repeat=2.500000
180514-09:07:33,234 workflow INFO:
	 [Node] Running "slicetimer" ("nipype.interfaces.fsl.preprocess.SliceTimer"), a CommandLine Interface with command:
slicetimer --in=/data/ds000114/sub-01/ses-test/func/sub-01_ses-test_task-fingerfootlips_bold.nii.gz --odd --out=/home/neuro/nipype_tutorial/notebooks/metaflow/preproc02/slicetimer/sub-01_ses-test_task-fingerfootlips_bold_st.nii.gz --repeat=2.500000180514-09:07:33,237 workflow INFO:
	 [Node] Running "slicetimer" ("nipype.interfaces.fsl.preprocess.SliceTimer"), a CommandLine Interface with command:
slicetimer --in=/data/ds000114/sub-01/ses-test/func/sub-01_ses-test_task-fingerfootlips_bold.nii.gz --odd --out=/home/neuro/nipype_tutorial/notebooks/metaflow/preproc03/slicetimer/sub-01_ses-test_task-fingerfootlips_bold_st.nii.gz --repeat=2.500000


180514-09:07:33,239 workflow INFO:
	 [Node] Running "slicetimer" ("nipype.interfaces.fsl.preprocess.SliceTimer"), a CommandLine Interface with command:
slicetimer --in=/data/ds000114/sub-01/ses-test/func/sub-01_ses-test_task-fingerfootlips_bold.nii.gz --odd --out=/home/neuro/nipype_tutorial/notebooks/metaflow/preproc04/slicetimer/sub-01_ses-test_task-fingerfootlips_bold_st.nii.gz --repeat=2.500000180514-09:07:33,244 workflow INFO:
	 [Node] Running "slicetimer" ("nipype.interfaces.fsl.preprocess.SliceTimer"), a CommandLine Interface with command:
slicetimer --in=/data/ds000114/sub-01/ses-test/func/sub-01_ses-test_task-fingerfootlips_bold.nii.gz --odd --out=/home/neuro/nipype_tutorial/notebooks/metaflow/preproc05/slicetimer/sub-01_ses-test_task-fingerfootlips_bold_st.nii.gz --repeat=2.500000

180514-09:07:35,177 workflow INFO:
	 [MultiProc] Running 5 tasks, and 0 jobs ready. Free memory (GB): 52.94/53.94, Free processors: 0/5.
                     Currently running:
                       * metaflow.preproc05.slicetimer
                       * metaflow.preproc04.slicetimer
                       * metaflow.preproc03.slicetimer
                       * metaflow.preproc02.slicetimer
                       * metaflow.preproc01.slicetimer
180514-09:07:41,370 workflow INFO:
	 [Node] Finished "metaflow.preproc04.slicetimer".180514-09:07:41,362 workflow INFO:
	 [Node] Finished "metaflow.preproc03.slicetimer".

180514-09:07:41,403 workflow INFO:
	 [Node] Finished "metaflow.preproc02.slicetimer".
180514-09:07:43,184 workflow INFO:
	 [Job 3] Completed (metaflow.preproc02.slicetimer).
180514-09:07:43,185 workflow INFO:
	 [Job 6] Completed (metaflow.preproc03.slicetimer).
180514-09:07:43,187 workflow INFO:
	 [Job 9] Completed (metaflow.preproc04.slicetimer).
180514-09:07:43,190 workflow INFO:
	 [MultiProc] Running 2 tasks, and 3 jobs ready. Free memory (GB): 53.54/53.94, Free processors: 3/5.
                     Currently running:
                       * metaflow.preproc05.slicetimer
                       * metaflow.preproc01.slicetimer
180514-09:07:43,229 workflow INFO:
	 [Node] Setting-up "metaflow.preproc02.mcflirt" in "/home/neuro/nipype_tutorial/notebooks/metaflow/preproc02/mcflirt".
180514-09:07:43,231 workflow INFO:
	 [Node] Setting-up "metaflow.preproc03.mcflirt" in "/home/neuro/nipype_tutorial/notebooks/metaflow/preproc03/mcflirt".
180514-09:07:43,235 workflow INFO:
	 [Node] Setting-up "metaflow.preproc04.mcflirt" in "/home/neuro/nipype_tutorial/notebooks/metaflow/preproc04/mcflirt".180514-09:07:43,234 workflow INFO:
	 [Node] Running "mcflirt" ("nipype.interfaces.fsl.preprocess.MCFLIRT"), a CommandLine Interface with command:
mcflirt -in /home/neuro/nipype_tutorial/notebooks/metaflow/preproc02/slicetimer/sub-01_ses-test_task-fingerfootlips_bold_st.nii.gz -meanvol -out /home/neuro/nipype_tutorial/notebooks/metaflow/preproc02/mcflirt/sub-01_ses-test_task-fingerfootlips_bold_st_mcf.nii.gz -plots
180514-09:07:43,237 workflow INFO:
	 [Node] Running "mcflirt" ("nipype.interfaces.fsl.preprocess.MCFLIRT"), a CommandLine Interface with command:
mcflirt -in /home/neuro/nipype_tutorial/notebooks/metaflow/preproc03/slicetimer/sub-01_ses-test_task-fingerfootlips_bold_st.nii.gz -meanvol -out /home/neuro/nipype_tutorial/notebooks/metaflow/preproc03/mcflirt/sub-01_ses-test_task-fingerfootlips_bold_st_mcf.nii.gz -plots
180514-09:07:43,241 workflow INFO:
	 [Node] Running "mcflirt" ("nipype.interfaces.fsl.preprocess.MCFLIRT"), a CommandLine Interface with command:
mcflirt -in /home/neuro/nipype_tutorial/notebooks/metaflow/preproc04/slicetimer/sub-01_ses-test_task-fingerfootlips_bold_st.nii.gz -meanvol -out /home/neuro/nipype_tutorial/notebooks/metaflow/preproc04/mcflirt/sub-01_ses-test_task-fingerfootlips_bold_st_mcf.nii.gz -plots

180514-09:07:44,339 workflow INFO:
	 [Node] Finished "metaflow.preproc01.slicetimer".
180514-09:07:44,648 workflow INFO:
	 [Node] Finished "metaflow.preproc05.slicetimer".
180514-09:07:45,186 workflow INFO:
	 [Job 0] Completed (metaflow.preproc01.slicetimer).
180514-09:07:45,187 workflow INFO:
	 [Job 12] Completed (metaflow.preproc05.slicetimer).
180514-09:07:45,190 workflow INFO:
	 [MultiProc] Running 3 tasks, and 2 jobs ready. Free memory (GB): 53.34/53.94, Free processors: 2/5.
                     Currently running:
                       * metaflow.preproc04.mcflirt
                       * metaflow.preproc03.mcflirt
                       * metaflow.preproc02.mcflirt
180514-09:07:45,233 workflow INFO:
	 [Node] Setting-up "metaflow.preproc01.mcflirt" in "/home/neuro/nipype_tutorial/notebooks/metaflow/preproc01/mcflirt".
180514-09:07:45,236 workflow INFO:
	 [Node] Setting-up "metaflow.preproc05.mcflirt" in "/home/neuro/nipype_tutorial/notebooks/metaflow/preproc05/mcflirt".180514-09:07:45,241 workflow INFO:
	 [Node] Running "mcflirt" ("nipype.interfaces.fsl.preprocess.MCFLIRT"), a CommandLine Interface with command:
mcflirt -in /home/neuro/nipype_tutorial/notebooks/metaflow/preproc01/slicetimer/sub-01_ses-test_task-fingerfootlips_bold_st.nii.gz -meanvol -out /home/neuro/nipype_tutorial/notebooks/metaflow/preproc01/mcflirt/sub-01_ses-test_task-fingerfootlips_bold_st_mcf.nii.gz -plots
180514-09:07:45,247 workflow INFO:
	 [Node] Running "mcflirt" ("nipype.interfaces.fsl.preprocess.MCFLIRT"), a CommandLine Interface with command:
mcflirt -in /home/neuro/nipype_tutorial/notebooks/metaflow/preproc05/slicetimer/sub-01_ses-test_task-fingerfootlips_bold_st.nii.gz -meanvol -out /home/neuro/nipype_tutorial/notebooks/metaflow/preproc05/mcflirt/sub-01_ses-test_task-fingerfootlips_bold_st_mcf.nii.gz -plots

180514-09:07:47,188 workflow INFO:
	 [MultiProc] Running 5 tasks, and 0 jobs ready. Free memory (GB): 52.94/53.94, Free processors: 0/5.
                     Currently running:
                       * metaflow.preproc05.mcflirt
                       * metaflow.preproc01.mcflirt
                       * metaflow.preproc04.mcflirt
                       * metaflow.preproc03.mcflirt
                       * metaflow.preproc02.mcflirt
180514-09:09:20,304 workflow INFO:
	 [Node] Finished "metaflow.preproc03.mcflirt".
180514-09:09:21,125 workflow INFO:
	 [Node] Finished "metaflow.preproc04.mcflirt".
180514-09:09:21,278 workflow INFO:
	 [Job 7] Completed (metaflow.preproc03.mcflirt).
180514-09:09:21,280 workflow INFO:
	 [Job 10] Completed (metaflow.preproc04.mcflirt).
180514-09:09:21,283 workflow INFO:
	 [MultiProc] Running 3 tasks, and 2 jobs ready. Free memory (GB): 53.34/53.94, Free processors: 2/5.
                     Currently running:
                       * metaflow.preproc05.mcflirt
                       * metaflow.preproc01.mcflirt
                       * metaflow.preproc02.mcflirt
180514-09:09:21,325 workflow INFO:
	 [Node] Setting-up "metaflow.preproc03.smooth" in "/home/neuro/nipype_tutorial/notebooks/metaflow/preproc03/smooth".
180514-09:09:21,327 workflow INFO:
	 [Node] Setting-up "metaflow.preproc04.smooth" in "/home/neuro/nipype_tutorial/notebooks/metaflow/preproc04/smooth".
180514-09:09:21,332 workflow INFO:
	 [Node] Running "smooth" ("nipype.interfaces.fsl.utils.Smooth"), a CommandLine Interface with command:
fslmaths /home/neuro/nipype_tutorial/notebooks/metaflow/preproc03/mcflirt/sub-01_ses-test_task-fingerfootlips_bold_st_mcf.nii.gz -kernel gauss 0.849 -fmean sub-01_ses-test_task-fingerfootlips_bold_st_mcf_smooth.nii.gz
180514-09:09:21,340 workflow INFO:
	 [Node] Running "smooth" ("nipype.interfaces.fsl.utils.Smooth"), a CommandLine Interface with command:
fslmaths /home/neuro/nipype_tutorial/notebooks/metaflow/preproc04/mcflirt/sub-01_ses-test_task-fingerfootlips_bold_st_mcf.nii.gz -kernel gauss 0.849 -fmean sub-01_ses-test_task-fingerfootlips_bold_st_mcf_smooth.nii.gz
180514-09:09:23,281 workflow INFO:
	 [MultiProc] Running 5 tasks, and 0 jobs ready. Free memory (GB): 52.94/53.94, Free processors: 0/5.
                     Currently running:
                       * metaflow.preproc04.smooth
                       * metaflow.preproc03.smooth
                       * metaflow.preproc05.mcflirt
                       * metaflow.preproc01.mcflirt
                       * metaflow.preproc02.mcflirt
180514-09:09:25,764 workflow INFO:
	 [Node] Finished "metaflow.preproc01.mcflirt".
180514-09:09:27,284 workflow INFO:
	 [Job 1] Completed (metaflow.preproc01.mcflirt).
180514-09:09:27,286 workflow INFO:
	 [MultiProc] Running 4 tasks, and 1 jobs ready. Free memory (GB): 53.14/53.94, Free processors: 1/5.
                     Currently running:
                       * metaflow.preproc04.smooth
                       * metaflow.preproc03.smooth
                       * metaflow.preproc05.mcflirt
                       * metaflow.preproc02.mcflirt
180514-09:09:27,326 workflow INFO:
	 [Node] Setting-up "metaflow.preproc01.smooth" in "/home/neuro/nipype_tutorial/notebooks/metaflow/preproc01/smooth".
180514-09:09:27,332 workflow INFO:
	 [Node] Running "smooth" ("nipype.interfaces.fsl.utils.Smooth"), a CommandLine Interface with command:
fslmaths /home/neuro/nipype_tutorial/notebooks/metaflow/preproc01/mcflirt/sub-01_ses-test_task-fingerfootlips_bold_st_mcf.nii.gz -kernel gauss 0.849 -fmean sub-01_ses-test_task-fingerfootlips_bold_st_mcf_smooth.nii.gz
180514-09:09:27,502 workflow INFO:
	 [Node] Finished "metaflow.preproc05.mcflirt".
180514-09:09:28,189 workflow INFO:
	 [Node] Finished "metaflow.preproc04.smooth".
180514-09:09:28,902 workflow INFO:
	 [Node] Finished "metaflow.preproc02.mcflirt".
180514-09:09:29,286 workflow INFO:
	 [Job 4] Completed (metaflow.preproc02.mcflirt).
180514-09:09:29,288 workflow INFO:
	 [Job 13] Completed (metaflow.preproc05.mcflirt).
180514-09:09:29,290 workflow INFO:
	 [Job 11] Completed (metaflow.preproc04.smooth).
180514-09:09:29,292 workflow INFO:
	 [MultiProc] Running 2 tasks, and 2 jobs ready. Free memory (GB): 53.54/53.94, Free processors: 3/5.
                     Currently running:
                       * metaflow.preproc01.smooth
                       * metaflow.preproc03.smooth
180514-09:09:29,328 workflow INFO:
	 [Node] Setting-up "metaflow.preproc02.smooth" in "/home/neuro/nipype_tutorial/notebooks/metaflow/preproc02/smooth".180514-09:09:29,329 workflow INFO:
	 [Node] Setting-up "metaflow.preproc05.smooth" in "/home/neuro/nipype_tutorial/notebooks/metaflow/preproc05/smooth".

180514-09:09:29,335 workflow INFO:
	 [Node] Running "smooth" ("nipype.interfaces.fsl.utils.Smooth"), a CommandLine Interface with command:
fslmaths /home/neuro/nipype_tutorial/notebooks/metaflow/preproc02/mcflirt/sub-01_ses-test_task-fingerfootlips_bold_st_mcf.nii.gz -kernel gauss 0.849 -fmean sub-01_ses-test_task-fingerfootlips_bold_st_mcf_smooth.nii.gz
180514-09:09:29,338 workflow INFO:
	 [Node] Running "smooth" ("nipype.interfaces.fsl.utils.Smooth"), a CommandLine Interface with command:
fslmaths /home/neuro/nipype_tutorial/notebooks/metaflow/preproc05/mcflirt/sub-01_ses-test_task-fingerfootlips_bold_st_mcf.nii.gz -kernel gauss 0.849 -fmean sub-01_ses-test_task-fingerfootlips_bold_st_mcf_smooth.nii.gz
180514-09:09:29,733 workflow INFO:
	 [Node] Finished "metaflow.preproc03.smooth".
180514-09:09:31,288 workflow INFO:
	 [Job 8] Completed (metaflow.preproc03.smooth).
180514-09:09:31,290 workflow INFO:
	 [MultiProc] Running 3 tasks, and 0 jobs ready. Free memory (GB): 53.34/53.94, Free processors: 2/5.
                     Currently running:
                       * metaflow.preproc05.smooth
                       * metaflow.preproc02.smooth
                       * metaflow.preproc01.smooth
180514-09:09:34,284 workflow INFO:
	 [Node] Finished "metaflow.preproc01.smooth".
180514-09:09:35,291 workflow INFO:
	 [Job 2] Completed (metaflow.preproc01.smooth).
180514-09:09:35,293 workflow INFO:
	 [MultiProc] Running 2 tasks, and 0 jobs ready. Free memory (GB): 53.54/53.94, Free processors: 3/5.
                     Currently running:
                       * metaflow.preproc05.smooth
                       * metaflow.preproc02.smooth
180514-09:09:35,830 workflow INFO:
	 [Node] Finished "metaflow.preproc05.smooth".
180514-09:09:35,863 workflow INFO:
	 [Node] Finished "metaflow.preproc02.smooth".
180514-09:09:37,294 workflow INFO:
	 [Job 5] Completed (metaflow.preproc02.smooth).
180514-09:09:37,296 workflow INFO:
	 [Job 14] Completed (metaflow.preproc05.smooth).
180514-09:09:37,300 workflow INFO:
	 [MultiProc] Running 0 tasks, and 0 jobs ready. Free memory (GB): 53.94/53.94, Free processors: 5/5.
CPU times: user 633 ms, sys: 117 ms, total: 750 ms
Wall time: 2min 6s
Out[ ]:
<networkx.classes.digraph.DiGraph at 0x7f15415c9198>

This time we can see that Nipype uses all available processors.

And if all went well, the total execution time should still be around ~2min.

That's why Nipype is so amazing. The days of opening multiple SPMs, FSLs, AFNIs etc. are past!

Results of metaflow

In [ ]:
!tree metaflow -I '*js|*json|*pklz|_report|*.dot|*html'

metaflow
├── graph_detailed.png
├── graph.png
├── preproc01
│   ├── mcflirt
│   │   ├── command.txt
│   │   └── sub-01_ses-test_task-fingerfootlips_bold_st_mcf.nii.gz
│   ├── slicetimer
│   │   ├── command.txt
│   │   └── sub-01_ses-test_task-fingerfootlips_bold_st.nii.gz
│   └── smooth
│       ├── command.txt
│       └── sub-01_ses-test_task-fingerfootlips_bold_st_mcf_smooth.nii.gz
├── preproc02
│   ├── mcflirt
│   │   ├── command.txt
│   │   └── sub-01_ses-test_task-fingerfootlips_bold_st_mcf.nii.gz
│   ├── slicetimer
│   │   ├── command.txt
│   │   └── sub-01_ses-test_task-fingerfootlips_bold_st.nii.gz
│   └── smooth
│       ├── command.txt
│       └── sub-01_ses-test_task-fingerfootlips_bold_st_mcf_smooth.nii.gz
├── preproc03
│   ├── mcflirt
│   │   ├── command.txt
│   │   └── sub-01_ses-test_task-fingerfootlips_bold_st_mcf.nii.gz
│   ├── slicetimer
│   │   ├── command.txt
│   │   └── sub-01_ses-test_task-fingerfootlips_bold_st.nii.gz
│   └── smooth
│       ├── command.txt
│       └── sub-01_ses-test_task-fingerfootlips_bold_st_mcf_smooth.nii.gz
├── preproc04
│   ├── mcflirt
│   │   ├── command.txt
│   │   └── sub-01_ses-test_task-fingerfootlips_bold_st_mcf.nii.gz
│   ├── slicetimer
│   │   ├── command.txt
│   │   └── sub-01_ses-test_task-fingerfootlips_bold_st.nii.gz
│   └── smooth
│       ├── command.txt
│       └── sub-01_ses-test_task-fingerfootlips_bold_st_mcf_smooth.nii.gz
└── preproc05
    ├── mcflirt
    │   ├── command.txt
    │   └── sub-01_ses-test_task-fingerfootlips_bold_st_mcf.nii.gz
    ├── slicetimer
    │   ├── command.txt
    │   └── sub-01_ses-test_task-fingerfootlips_bold_st.nii.gz
    └── smooth
        ├── command.txt
        └── sub-01_ses-test_task-fingerfootlips_bold_st_mcf_smooth.nii.gz

20 directories, 32 files

Home | github | Nipype