Pegasus Integration

This page is dedicated to documenting how Pegasus will be integrated into the current job management system

  • Add pegasus as a job and data tool in DB and copied globusjobsubmit job tool and gsiftp data tool as basis for the tool code.
  • Added pegasus as a restricted site that uses the pegasus tools. This should ensure that only pegasus jobs match to the pegasus site.
  • (data) copy.pm will use the rc-client tool from pegasus to register the local data to pegasus. It will set rdata.path to the registered code. The code will most likely be the rdataid to keep it unique. It will automatically set status=4 to show that its ready.
  • (job) submit.pm will use the template-builder.pl code (borrowed from the job wrapper) to convert the generated config file to a pegasus workflow. It will take a template file that will sit on the job submission VM (grid-submit).
    • The template builder converts the registered pegasus data into a workflow that pegasus understands.
    • The site.execpath will be used to define the prefix path where the temporary pegasus workflow will be stored on the VM.
      • Note: this is misusing the execpath slightly, but it should be ok.
    • The site.apppath is set to a valid path on the VM and is used to store the templates for making the workflows.
    • The template can contain 1 additional # value beyond the typical template where the output should be stored. This additional value is replaced with expid/runid/jobid/results.tar.gz because pegasus needs it to be defined at the beginning. So build-workflow.pl with replace the # value with the appropriate output on the fly.
  • (job) submit.pm will submit to pegasus-plan with the submit option and record the pegasus id.
  • (job) status.pm will use the pegasus id to track job status.

Status

  • May 18 - asked Yan to add a Gi*d application to the development version of GISolve so that I can link it to the pegasus site. This will allow for testing of Gi*d application and the integration of Pegasus at the same time. DONE