CIGI's Coordinated Usage of OSG Computing Resources

High Level Idea: Our goal in the project is to steadily increase the resource usage on the Open Science Grid machines. The usage will mainly come from Agent Based Modeling (ABM) experiments and other embarrassingly parallel application (e.g. viewshed analysis). The plan to sustain a steady rate of job submissions to OSG thereby increasing the visibility of CIGI VO on OSG.

Preliminary Tasks:

  1. Anand will provide help and work on improving scripts so that job submission to OSG and management will become easier.
    • Currently we have scripts that can be used to define ABM experiments. These scripts create the necessary submit files, invoke necessary commands to execute the jobs remotely, and stage-in and stage-out relevant application data and results respectively.
    • A tightly coupling of these scripts with GISolve will be useful.
  2. Wenwu will continue exploring OSG and begin submitting jobs in a small scale to selected OSG sites.
  3. Wenwu will also think about experiments that will be interesting to study and together with Anand develop a plan to scale up the runs.

Quantitative Goals:

  1. Start submitting some jobs daily to a selected OSG site (End of Aug)
  2. Able to submit atleast 1 and possibly 2 applications to atleast 5 OSG sites (Mid Sept)
  3. Submit a small number jobs daily to atleast 5 selected OSG sites (End Sept)
  4. Have a sustained usage of atleast 500 hours/week as reported to Gratia across OSG (End Sept)
  5. Scale Up (Goal undecided at this time)!

Current Progress

Viewshed analysis jobs

We are currently working on submitted Wenwu's viewshed analysis code. We have 3 dataset that includes landscape information of 3 counties. There are ~15000 blocks (points) associated with each dataset and the requirement is to find the coverage of each individual point. So we will have ~15000 independent jobs per dataset with each job taking 30-60 mins.

  • Aug 17-21: Wenwu installed some OSG clients and submitted OSG jobs on his VM.
  • Aug 24: Anand has developed submit script that will be able to submit/manage a large number of jobs
    • This script heavily leveraged the scripts developed for ABM.
    • We are trying to debug errors we are encountering.
  • Aug 26: Anand completed a small runs of park county data at Fermilab
    • Each job completed quickly ~3-5 mins.
    • There were in total 142 jobs
  • Aug 27: We decided to use a complete DEM that contains all 3 county and rerun the previous experiment
    • Wenwu thinks it will take ~50 mins for each jobs.
    • Jobs are running on 4 OSG sites.
  • Sept 7: Anand ran 142 jobs on fullscale DEM, each job took ~30 mins. Results varified by Wenwu
  • Sept 9: New centroid data now available.
  • Oct 4: All runs on park county dataset have been completed. 15090 jobs overall on two sites FNAL and UFlorida
  • Oct 6: A new set of data is available and we have begun processing
  • Oct 8: 5000 data points processed
  • Oct 9: Over the past month CIGI's usage of OSG has improved dramatically. We have begun showing up on Gratia graphs. We are the top 20 usage among all VOs (both last week and last month). If we do not consider the top VOs like cms, atlas, cdf, des, ligo then we are among the top 10 in terms of number of jobs and processing hours.

Using scripts to submit jobs - Example

 ./create_vs_experiment.pl -e whole/parcelparkw.txt -o experiments/expt1 -id 1 -start 1 -end 500 -of 1
 ./submit_jobs.pl -p experiments/expt1 -s job_vs_new.sh -o /cigi/data-grid/data/cigi/vs_params.pl -nodir -d /cigi/data-grid/data/cigi/viewshed/production