HPC - High Performance Computing - Revision history

Laser at 12:42, 6 April 2012

2012-04-06T12:42:14Z

Cclark: /* Using PBS */

2012-04-05T18:15:42Z

‎Using PBS

Cclark: /* Using PBS */

2012-04-05T17:56:33Z

‎Using PBS

Cclark: /* Using PBS */

2012-04-05T17:43:08Z

‎Using PBS

Cclark: /* Terminology */

2012-04-05T17:29:36Z

‎Terminology

Cclark at 17:27, 5 April 2012

2012-04-05T17:27:23Z

Cclark: /* Using PBS */

2012-04-05T17:01:56Z

‎Using PBS

Cclark: /* Terminology */

2012-04-05T17:01:24Z

‎Terminology

Cclark: /* Terminology */

2012-04-05T16:32:44Z

‎Terminology

Cclark: /* Terminology */

2012-04-05T16:32:21Z

‎Terminology

@@ Line 1: / Line 1: @@
 =Using a Scheduler=
 ==Motivation==
-When developing code, it is usually sufficient to test that your code is working by running "small" configurations one at a time and checking the output. Once a code has been shown to work, we typically want to run several "large" configurations. From the terminal, this consists of configuring the code an running the executable. After the first configuration finishes, we run the executable with a different configuration. Once this finishes, we do the same until we have run all configurations. The problem here is that we are only running one instance of our code at a time, and we have to manually start the next configuration after the last completes. This means that if the last configuration completes at 2:00 AM, the next configuration will not be started until we check on the progress later in the mourning.
+When developing code, it is usually sufficient to test that your code is working by running "small" configurations one at a time and checking the output. Once a code has been shown to work, we typically want to run several "large" configurations. From the terminal, this consists of configuring the code an running the executable. After the first configuration finishes, we run the executable with a different configuration. Once this finishes, we do the same until we have run all configurations. The problem here is that we are only running one instance of our code at a time, and we have to manually start the next configuration after the last completes. This means that if the last configuration completes at 2:00 AM, the next configuration will not be started until we check on the progress later in the morning.
 We could open multiple terminals and run multiple instances of the executable for different configurations. We would need to be careful not to run too many instances simultaneously though, because we could end up requiring more resources than are available. For example, if running 1 instance of our code requires 1 GB of RAM, then running 5 instances will required 5 GB. If our computer only has 4 GB of RAM, running 5 instances of our code will fail. What we would like is a way to automatically run multiple jobs, one after the other, and run multiple instances of our code at once without running too many. Enter the scheduler.

@@ Line 50: / Line 50: @@
   Job id                    Name             User            Time Use S Queue
   ------------------------- ---------------- --------------- -------- - -----
-.localhost              simple_sleep.sh  cclark                 0 R test
+.localhost              simple_sleep.sh  laser                  0 R test

← Older revision		Revision as of 17:56, 5 April 2012
Line 35:		Line 35:

	Running this command will submit simple.sh to the scheduler to be ran. This command will print one line, which gives the job's ID. To check on a job's status, we run <code>qstat</code> (as in queue status).		Running this command will submit simple.sh to the scheduler to be ran. This command will print one line, which gives the job's ID. To check on a job's status, we run <code>qstat</code> (as in queue status).
		+
		+	# qstat
		+
		+	This will print out a list of jobs that are currently in the queue. Jobs marked as '''R''' are currently running, jobs marked as '''Q''' are waiting to run. The simple.sh job will not take very long to run, so it will not show up in the <code>qstat</code> output unless other jobs are already running. We can submit a script that will take 10 seconds to run, which should give us enough time to see it in the queue.
		+
		+	# cat simple_sleep.sh
		+	sleep 10
		+	hostname
		+
		+	Submitting this job and then calling <code>qstat</code> immediately should allow us to see the job in the queue.
		+
		+	# qsub simple_sleep.sh
		+	# qstat
		+	Job id Name User Time Use S Queue
		+	------------------------- ---------------- --------------- -------- - -----
		+	453.localhost simple_sleep.sh cclark 0 R test

← Older revision		Revision as of 17:43, 5 April 2012
Line 29:		Line 29:
	# cat simple.sh		# cat simple.sh
	hostname		hostname
		+
		+	To '''submit''' this '''job''' to the '''scheduler''', we use the <code>qsub</code> command (as in queue submit)
		+
		+	# qsub simple.sh
		+
		+	Running this command will submit simple.sh to the scheduler to be ran. This command will print one line, which gives the job's ID. To check on a job's status, we run <code>qstat</code> (as in queue status).

@@ Line 20: / Line 20: @@
 ; Submit
 : The user ''submits'' a job to the scheduler. The scheduler places this in the queue of jobs to run.
 ==Using PBS==

@@ Line 25: / Line 25: @@
 The most common scheduler used it PBS (Portable Batch System). PBS is an opensource scheduler that can work with an open source resource manager called Torque to manage a job queue. Other scheduling systems exist as well. To submit a job using PBS, we must first write a submit script. This is usually just a '''bash''' script. The simplest submit script contains just the executable to run.
-  # simple.sh
+  # cat simple.sh
   hostname

@@ Line 9: / Line 9: @@
 ==Terminology==
-A user ''submits'' ''jobs'' to the scheduler. The scheduler adds these jobs to a ''queue'', and runs them as old jobs finish. The scheduler can also work with a ''resource manager'' to determine what resources (CPUs, memory, etc) are available, and to run jobs on the available resources. On a cluster, the resource manager and scheduler will work together and take care of running jobs on different computers in the cluster. However, with the advent of multi-core processors, a single computer can run multiple jobs at one time. (in the old days, when one computer had one processor with one core, running two instances of a simulation took twice as long as running one instance, so it took the same amount of time as running them separately, one after the other.)
+A user ''submits'' ''jobs'' to the scheduler. The scheduler adds these jobs to a ''queue'', and runs them as old jobs finish. In order to submit a job, you must write a ''submit script''. This is a simple shell
 ; Job
@@ Line 18: / Line 21: @@
 : The user ''submits'' a job to the scheduler. The scheduler places this in the queue of jobs to run.
-==
+==Using PBS==