ModflowOnAzure + CyberGIS integration

In this page, we'll describe our efforts on integrating ModflowOnAzure application which is an Azure application written by Yong Liu with our CyberGIS infrastructure.

Week 3 : Jun 26 - July 2

  • The program is now running without any problem. The high-level view of the steps to run the program is as follows:
  1. Deploy the Modflow program on Windows Azure.
  2. Put the URL to the program on Azure, the Account name and the Access Key into the Java Program.
  3. Run the Java program, it'll upload the Model file and Setup file onto the cloud and puts a Message into the queue.
  4. The ModflowOnAzure program sees that message in the queue and runs the program.
  5. The results will be put on the storage by the ModflowOnAzure program. I should just get the URL to it and let the user know it.
  • I have tried my best to run UNIX version of Modflow on a Linux machine. But unfortunately could not! These are my main efforts:
  1. Check if it runs with the icc compiler on both Abe and Trestles. No! The error file is available here.
  2. Check if it runs with the gcc compiler on both Abe, Trestles and my desktop computer (running Centos 5.6). No! (These computers are are 64-bit).
  3. Check if it runs on a 32-bit machine. Created a 32-bit Ubuntu Virtual Machine first, it can not be run but the error is different now :
    • At line 127 of file zonbud.f (unit=5, file = 'stdin') Fortran runtime error: end of file
    • At line 101 of file hydfmt.f (unit=60, fiel = 'fort.60') Fortran runtime error: end of file
  4. I didn't trust to Ubuntu fortran! Installed a 32-bit CentOS and the problem occured again!
  5. I also used a compiler option named -fsimple-precision-constant as Kieth said in his email, but it didn't help.

Week 2 : Jun 19 - Jun 25

What should the program do? (View from file DropboxMonitor.cs in WorkerRole1)

  1. DropboxMonitor has some important const strings : CONTAINERNAME = “modflow”, IOQUEUENAME = “modflow”
  2. It also creates a CloudBlobClient _blobStorage;, a CloudQueueClient _queueStorage;, a CloudBlobContainer container; and a CloudQueue ioQueue; object.
  3. In its constructor, DropboxMonitor class :
    • instantiates the cloudstorage manger (dropbox CloudStorage is not related to Azure),
    • reads storage account configuration settings.
    • It then creates blob container using CONTAINERNAME by running _blobStorage = storageAccount.CreateCloudBlobClient(); statement
    • Getting the reference to the container by calling container = _blobStorage.GetContainerReference(CONTAINERNAME); is done next
    • After creating the container, it configures it for public access by setting the access type and permissions
    • It then creates input job message queue to communicate with worker role by running _queueStorage = storageAccount.CreateCloudQueueClient();
    • It gets references to ioQueue by calling CloudQueue ioQueue = _queueStorage.GetQueueReference(IOQUEUENAME);
  4. NOT UNDERSTOOD: Who calls the run() function?
  5. run() function, basically has a while(true) infinite loop which does the following:
    • It generates a unique string named uniqueFileId.
    • It generates these two strings : string modelUploadName = projectName + uniqueFileId + ”.zip”; string setupUploadName = “RunSetup” + uniqueFileId + ”.txt”;
    • It uploads the files to blob :
    • Then it queues a message to prcess the data :
    • After sending the message to ioQueue, the program starts a thread for named outputThread and pass monitorOutput(uniqueFileId) function to it. And bloks on a while(!outputThread.IsAlive);
  6. MonitorOutput(id) function :
    • In an infinite loop here, the program sleeps for 10 seconds.
    • It then gets a blob named Out_{jobId}.zip from the container and Fetch its attributes.
    • Checks if the job has finished by looking at a Metadata of this Blob named “RunDuration”. If this Metadata is not null then the program downloads this blob to a file.
  7. The program will finish by downloading the output file.

What should the program do? (View from file Default.aspx.cs in WebRole1)

  1. _Defualt class has some important const strings : BLOBNAME = “modflow”, IOQUEUENAME = “modflow”, JOBQUEUENAME = “job”
  2. It also creates a CloudBlobClient _blobStorage; object and a CloudQueueClient _queueStorage; object
  3. As the first part of the program, it calls a function named InitializeStorage();
  4. The InitializeStorage(); function initializes storage by performing these steps:
    • NOT UNDERSTOOD: Read storage account configuration settings
    • NOT UNDERSTOOD: Read account configuration settings
    • It then creates blob container using BLOBNAME by running _blobStorage = storageAccount.CreateCloudBlobClient(); statement
    • Getting the reference to the container by calling _blobStorage.GetContainerReference(BLOBNAME); is done next
    • After creating the container, it configures it for public access by setting the access type and permissions
    • It then creates input and ouput queue to communicate with worker role by running _queueStorage = storageAccount.CreateCloudQueueClient();
    • It gets references to inputQueue and outputQueue by calling CloudQueue inputQueue = _queueStorage.GetQueueReference(IOQUEUENAME); and CloudQueue outputQueue = _queueStorage.getQueueReference(JOBQUEUENAME);
  5. After Initialization, clicking on the Run button by user calls the Run() method which works as follows:
    • It generates a unique string named uniqueFileId.
    • It generates these two strings : string modelUploadName = projectName + uniqueFileId + ”.zip”; string setupUploadName = “RunSetup” + uniqueFileId + ”.txt”;
    • It uploads the files to blob :
    • Then it queues a message to prcess the data :
    • NOT UNDERSTOOD: After adding this message to the IO queue, it starts a timer named Timer1. Timer1 has a Timer1_OnTick() function which
  6. Timer1_OnTick function:

Week 1 : Jun 12 - Jun 18

How to connect to Windows Azure using Java from Linux?

1st effort : jAzure API

2nd effort : WindowsAzure4j

3rd effort : jCloud API

  • It seems like a very live project. I just started to explore it. I don't know if we should use it or not.