GISolve2: Grid Job Submission Service

Purpose

This project is to establish a VM for Grid job submission and management. Based on this VM, we can form a pool of Grid job management VM instances for all the jobs submitted in GISolve2.

Team

  • Anand
  • Eric
  • Yan

Progress

2009
Week 10/18 - 10/25
  • Anand and Yan need to summarize previous work that Anand, Yan, and Yingjie have done for Grid job submission
    • parameter-sweeping job support: db design and db access API
    • OSG parameter-sweeping job submission utility for submitting ABM jobs
    • TeraGrid parameter-sweeping job submission utility for submitting ABM jobs
  • Discussion on the project plan
Week 10/26 - 10/30

Week 11/26 - 11/30
  • IP: 130.126.3.66 (grid-submit.cigi.uiuc.edu)
  • DB: Mysql or postgres? (mysql)
  • Anand said we will be putting the DB on greatlakes, I am waiting on a username and DB to be installed on that machine with the appropriate permissions for the GISolve2 project (if that is not ready in time I will create a temporary version for my own development purposes) (Yan - finished by Tuesday)
Week 12/14 - 12/18
Week 1/4 - 1/8
  • Updated UML diagram

  • Continued work on DB manipulation scripts
  • Starting to work on a workflow for node side scripts to handle application execution
Week 1/11-1/15
  • After discussion with Yan I added 3 components to the DB to include VM job submission and to account for restricted application installation on VM and Teragrid. The UML diagram from the previous week was updated and posted to reflect these changes including a restriction table to describe where the applications are installed on the sites.
  • I updated the DB to reflect the new UML diagram
  • I will begin working on the code for match-making and other components to use the new information
Week 1/18-1/22
  • Added data and job tools in the DB to abstract interface to sites. This will enable GISolve to flexibly adapt to new resources. Currently we are targeting OSG, Teragrid, and VM's.
  • I also added a system designed in Perl to manage each of the tool interfaces. Currently I have a test connector that directly executes a script. Adding more connects will only involve adding an entry in the DB, copying the shell connector code and inserting the specific code to interface with the tool. The features that each tool interface supports are the following:
    • Job - submit, status, cancel
    • Data - copy, status, cancel, delete
  • I updated the management system to incorporate the above changes within the DB infrastructure and the tracking system which maintains the status of jobs and data.
  • Waiting Need a service from GISolve that returns a local-data id which contains the filelocation where the output of each job should be stored
  • Waiting Yan, Anand, and I will have a meeting next week to discuss the node execution aspects of the job submission system. I have begun working on the core pieces of this system, but many decisions need to be made.
  • Currently it is unclear how data will be tied to experiments for the user, but that is technically outside the scope of job management.
  • Updated UML diagram

Week 2/1-2/5
  • I had a few individual meetings with Yan and Anand to discuss current progress with the job management system and future components. The primary missing component was node side job execution. After some discussion I have reached an initial design that is described below.
  • Began working on node side job execution script. I finished a wrapper script that reads in a configuration file that performs the following steps: Check application and data paths, make execution path, parse parameters and template file, run the application, tar up results if successful otherwise tar up std{out,err} on error
    1. Currently the script does not deploy applications (assumes they are installed) or works with MPI jobs
    2. I feel this current script is sufficiently developed to start testing once a job submission tool is agreed upon for OSG and Teragrid.
  • The wrapper script reads in a configuration file that will be dynamically generated for each job to keep the script generic and only need to change a single configuration file.
  • I finished work on the direct execution connector, it now executes an application directly.
  • I also cleaned up some of the code and added some data to the DB such as datapath and apppath for the site. Eventually execpath will most likely be added, but I am still evaluating.
  • Waiting Need to decide on a job submission tool for OSG and Teragrid. Current candidates include: Condor-G, Pegasus, ??
  • Waiting Service API for VM interaction
  • Waiting Need a service from GISolve that returns a local-data id which contains the filelocation where the output of each job should be stored
  • Waiting Data server to copy data to/from using gsiftp preferably
Week 2/15-2/21
  • I discussed this work and my Pegasus experience at the previous group meeting (feb 16).
  • Last week I was experimenting with Pegasus. It was decided at the group meeting that I focus my efforts on the current system - which is what I have been working on since.
  • I have a test application fully working on everest.
    • The application is already installed so the application deployment process is skipped
    • The job management system copies a configuration file and all the required input data via globus-url-copy
    • It then executes a job wrapper and points to the generated configuration file for each job
      • The wrapper sets up the environment, executes the application, and tar's up the results and output
    • The results tarball is copied back to GISolve
  • NOTE: currently the results are copied locally to a temporary directory, because I am still waiting on the service which will return a local-data id that contains the file location where the output should actually be stored.
  • I will begin work to submit a job to a Teragrid resource.
Week 2/22-2/28
  • Yan's task list:
    1. Change the data tables that interface with JM. Update ER diagram's left part. By 02/26 noon.

DB Integration ER Diagram

  1. Create experiment table entries and invoke JM for job submission. By 02/26 evening.
    • Coding and initial testing on gisolve portal is done for JM integration
  2. Write a Web service for JM to get a destination path where the output tarball of a job should be copied to. The service returns a uri, and it is gsiftp: for now. CHANGE The service should return a local-data id. * Change (Yan Liu): the output path for an experiment will be a field in the experiment table. JM should copy the output tarball for each job to this path. * The gsiftp path for above change is now set to gsiftp:everest.cigi.uiuc.edu/cigi/data-grid/data/gisolve/${expid}
  3. Proxy management utility: a utility tool deployed on grid-submit machine to get a job proxy by using GISolve's TeraGrid certificate. No need to input password; supports gateway attributes and osg VO attributes
  • Integration task list that includes all:
  1. DMS go-through
  2. pRPL MPI application go-through
  • Eric's status and task list:
  1. DMS on abe is working using globus-job-submit on jobmanager-pbs.
  2. Currently the template file is assumed to be on the site if the application is on the restriction list.
  3. Once the service to get a local-data id for the results tarball is up I will alter my code to use the service. Dropped under new assumption
  4. Once the new data tables are updated I will change my code to use the new tables.
  5. I will change the job submission database according to discussions I had with yan today (Feb 25).
  6. A new column in the experiment table will list the local_result_dir for that experiment (a gsiftp uri). This entry will be populated by Yan and Eric will use this entry as the base directory to store the output of runs and jobs for this experiment.
Week 03/01 - 03/07
  • Finish integration of GISolve portal and JM
  • JM: MPI job management
  • Eric's status:
    1. I altered the test DB to match Yan's new DB diagram.
      • Except the new entry in the experiment table, the wiki says it should be “local_results”dir” the diagram says it should be “outputlocation” and our discussion said it should be “resultsuri”. I added the entry as resultsuri from my notes, but can alter it later if necessary.
      • I assume that the resultsuri in experiment is the base directory where I will place the results tarball (This structure was given to me by Anand in a discussion last week):
        • {resultsuri}/{expid}/{runid}/{jobid}/results.tar.gz
    2. Clarification (Yan Liu): Please use “outputlocation” as specificed in ER diagram and db table. Thanks for pointing out the naming confusion. DONE
  • Problem encountered when using https urls in globus-url-copy
globus-url-copy -cd https://gisolve2.cigi.uiuc.edu:8443 /home/userspace/yanliu/sample gsiftp://gridftp-abe.ncsa.teragrid.org/u/ac/gisolve/jobsub/datapath/2/sample
error: [globus_l_gass_copy_gass_setup_callback]:  url: https://gisolve2.cigi.uiuc.edu:8443/home/userspace/yanliu/sample request was DENIED, for reason: 400, Bad Request
  • Please make sure that JM follows TeraGrid policy: input and output data must be copied to site-specific scratch directory. Otherwise, home directory of gisolve community account will be overfilled soon.
    • This is easily achieved by setting the site's datauri to the scratch directory of the site.
  • Question to discuss: if user deletes an experiment, JM has problem handling that. A solution might be that we do not really delete the experiment entry in DB. Instead, we mark it as 'deleted'. For Web server, it does not show 'deleted' jobs; JM does not need to change anything. However, JM needs to make sure that a DB exception does not crash any JM daemon. Update: added 'dirty' field to table 'experiment' for marking deleted entries; code changed accordingly
    • This is not just a problem with job management, the local data is also linked to experiment (byExp) so a deletion of experiments will result in DB corruption in the local data table. Previous discussions stated that the experiments would not be deleted (e.g. they are permanent), but the runs/jobs/rdata can be deleted. The database and subsequent code was built off this assumption. If this is a necessary feature then we can redesign the system to accommodate the new requirement.
  • Status update (Eric)
    • The daemon is configured to automatically trigger the job management system in a reasonably intelligent way that shouldn't overwhelm the sites and should be fairly responsive for GISolve
    • I have ran through 2 test experiments which returned the correct DMS results from abe.
    • Once the daemon is running on grid-submit the job management system should be available for Yan to trigger. Daemon is running and should be completely working
Week 03/08 - 03/12
  • Eric's status update
    • I have a working MPI launch using globus-job-run on Abe. It will require a few modifications to the current implementation of the job management system:
      • Add pre,launch,post strings that are site specific to support MPI startup, launch, and teardown DONE
      • Add rsl string to launch an MPI instance which is site specific DONE
      • Add MPI bit for application to indicate it is an MPI application
      • MISSING Calculate the number of nodes to request based on the user input
      • MISSING Calculate the walltime based on the user input
  • Future feature
    • If a job fails, then mark any data that the job depended on as not on the site. This will reduce likelihood of errors due to site side data purging.
    • A cron should be setup to check for the existence of data on the remote side periodically.
    • The data (if not used within X days) should be removed from a remote site.
Week 03/29 - 4/2
  • Job management code is uploaded into svn
  • grid-submit
    • To turn on job management on grid-submit.cigi.uiuc.edu as root type the following command: (although it should start automatically)
      • /etc/init.d/jobmanagement start
  • Data purge instructions
    • If data is purged on a site then type in the following commands into mysql
      • update rdata set status=0 where siteid=<site id with purged data>;
    • The previous command will mark all data ready to be transferred from GISolve to the site. Its not a perfect solution, but will transfer the data back keeping the database consistent.
Week 4/26 - 4/30
Week 5/3 - 5/7
  • Began looking into Pegasus ⇐⇒ Job manager integration.
  • Use Pegasus as a restricted site in the job manager and only have “workflow ready applications” be able to map to it.
  • Must support job submit,status,cancel.
  • Job submit will need to incorporate an application to workflow matcher. In addition we will have to create some sort of a workflow template such that the actual parameters can be inserted into the workflow before submission.
  • Job status will have to parse the pegasus-status command. At first a simple line count will tell us if the job is complete or not, but eventually we will have to parse the jobstate.log file to get any errors that may come up.
  • Job cancel should be easily supported.
  • Data copy will be supported by using the tc-client tool and “registering” any data that needs to be copied and then pegasus will handle the actual data copying.
  • I will abstract out as much interaction as possible with the communication between Pegasus and JM such that if Pegasus will need to be deployed on another VM that the overhead is reduced.
  • TODO Will need to get if copying a data entry in pegasus as type=executable will enable us to copy a binary file and execute it within a wrapper script on Pegasus. If this is possible then we can wrap up all of the GI(*)d jobs and such that they will feasibly without changing the code. Although this is a hack to workaround the shortcomings of Pegasus.
    • Pegasus will not copy data if type=executable, only if type=data
    • Solutions around this problem are to mark programs as data and have a special flag in the wrapper tell them that they are executable and to chmod +x each application.
  • Yan said to wait until there are development VM's available for job-management to incorporate and test Pegasus. So I am waiting for the dev-VM's before trying some of these ideas.
Week 5/17 - 5/22
  • Pegasus integration continues.
  • Ideas for pegasus integration into job manager will be documented on this wiki page pegasusintegration