-------------------------------------
PIPELINE ENGINE
Version 1.2 Mohana Ramaratnam
Email: mohanakannan9@gmail.com
Neuroinformatics Research Group (http://nrg.wustl.edu),
Washington University in St Louis

Date: October 22, 2010
-------------------------------------

WHATS NEW:

* Validation module which performs ImageSession protocol check validation. 

* Transfer and AutoRun pipelines now invoke all pipelines which have been marked as run automatically on archival. 

Contains validation pipeline

INSTALLATION:

0. Set an environmental variable JAVA_HOME pointing to the location of jdk 1.4+
1. Unzip the file pipeline.zip. This will create a folder pipeline, (say <PIPELINE_HOME>).
2. cd <PIPELINE_HOME>
3. Edit the file pipeline.config
4. On unix based system, launch setup.sh <admin email id> <smtp server>
   On windows, launch setup.bat <admin email id> <smtp server>

DOCUMENTATION:

Open <PIPELINE_HOME>/doc/index.html

DEMO/SAMPLE:

<PIPELINE_HOME>/sample_pipelines folder contains a pipeline descriptor Demo_Loop.xml which is a process which has 4 steps.
 Each of the step demostrate different features and usability of the Pipeline Engine. The feature being demonstrated is described in the attribute called "description" of the steps. Each step is identified by its id. For a given step, this id must be unique to the pipeline.


As the pipeline descriptor is XML and XPATH based, one can use the features of XPATH to set the parameters to a resource. One can also use custom extensions to the XPATH functions.

In the sample pipeline the following features are demonstrated:

1. How to create a loop (see <loop> element on line 8 of the demo pipeline) and perform a task repeatedly for each value of the loop (see  Step id=0). How to execute a task in a specific folder.
2. How to use gotoStepId and precondition attributes. Only if the condition is true the step is executed otherwise the step is skipped. When the engine encounters a gotoStepId, the next step to execute is the step identified by the gotoStepId attribute. If no gotoStepId is used, the next step is the next one in sequence.
3. How to use AwaitApprovalToProceed.  (see Step id=2)
4. How to send a custom email notification (see Step id=2)
5. How to use continueOnFailure attribute (see Step id=3)
6. How to send all values of a loop to a custom/wrapper script (see Step id=3a)
7. How to send a custom email with an attachment (See Step id=4)
8. How to use custom extensions to XPATH functions. See definition of parameter2 variable.

In order to launch the Demo_Loop pipeline, perform the following steps:

LAUNCHING A PIPELINE:

1. Modify <PIPELINE_HOME>/sample_pipelines/Parameters.xml to enter an SMTP host and your email id.
2. Modify <PIPELINE_HOME>/pipeline.config file, (if not already done) to set the properties
	PIPELINE_EMAIL_ID
	PIPELINE_SMTP_HOST
3. From a command prompt, invoke:
	<PIPELINE_HOME>/bin/PipelineRunner[.bat] -pipeline <PIPELINE_HOME>/sample_pipelines/Demo_Loop.xml -parameterFile <PIPELINE_HOME>/sample_pipelines/Parameters.xml  -config <PIPELINE_HOME>/pipeline.config -notify <YOUR EMAIL ID HERE>
   The effect of the above step will be creation of  <PIPELINE_HOME>/sample_demo folder and subfolders Loop1, Loop2 and Loop3 within sample_demo folder and an email notification will be sent to you.
   The err and the log files for this execution will be in the <PIPELINE_HOME>/sample_pipelines/Demo.err and Demo.log respectively.


MORE WAYS TO LAUNCH A PIPELINE

a) Delete the folder  <PIPELINE_HOME>/sample_demo
   Modify  <PIPELINE_HOME>/sample_pipelines/Demo_Loop.xml to set the value of the parameter i to 2
   From a command prompt, invoke:
	<PIPELINE_HOME>/bin/PipelineRunner[.bat] -pipeline <PIPELINE_HOME>/sample_pipelines/Demo_Loop.xml -parameterFile <PIPELINE_HOME>/sample_pipelines/Parameters.xml  -config <PIPELINE_HOME>/pipeline.config -notify <YOUR EMAIL ID HERE>

   After this step completes, an email will be sent to you stating that the pipeline is awaiting some action. In order to restart the pipeline where it left off, perform the following step:

b) From the command prompt invoke:
	<PIPELINE_HOME>/bin/PipelineRunner[.bat] -pipeline <PIPELINE_HOME>/sample_pipelines/Demo_Loop.xml -parameterFile <PIPELINE_HOME>/sample_pipelines/Parameters.xml  -config <PIPELINE_HOME>/pipeline.config -notify <YOUR EMAIL ID HERE> -startAt 3

   After this step completes, an email will be sent to you with a file attached



Steps 3,a) and b) above do not interact with XNAT, if you want the engine to update XNAT workflow status, invoke <PIPELINE_HOME>/bin/XnatPipelineLauncher[.bat] with appropriate parameters.






