GProc

Getting Started with GProc

At its most basic level, GProc needs at least a single "master" node and a single "slave" node; for testing purposes, these can even be the same system. The master, started with the gproc m command (see below), accepts connections from slave nodes (started with gproc s). The user then runs a gproc e <nodes> <program> command on the master; this sends a command over a Unix Domain Socket to the listening GProc master process, which then instructs the appropriate slave nodes to execute the specified program.

Commands run by the user are copied over to the slave nodes, along with any required libraries and any additional files specified at the command line. Files that are transferred to nodes are stored in a replicated root under a certain directory of the node (default /tmp/xproc). This keeps gproc's operations from interfering with the rest of the node. For example, to run /bin/date, the binary would be copied to /tmp/xproc/bin/date and any necessary libraries would go to /tmp/xproc/lib/. The current working directory is also created on the nodes and the remote process starts in it. The mount is a private mount and disappears when the remote process and all of its children exit.

Installing GProc

GProc is a Go program. If you have Go 1 installed on your computer, you can simply run "go get bitbucket.org/dfritz/gproc-neu" to get GProc. This will build the GProc binary and place it at $GOPATH/bin/gproc.

A basic example

Let's assume you have a small cluster, with a head node and 4 slave nodes, in a configuration somewhat like this:

	192.168.0.254	master
	192.168.0.1	n1
	192.168.0.2	n2
	192.168.0.3	n3
	192.168.0.4	n4

In this case, "master" will run the GProc master process and n1-n4 will run the slave processes. Below is an example of how to start and use GProc on this simple cluster. Note that the use of the names "hostname" and "hostbase" are intentional, not placeholders--see the section on GProc's Forth syntax for more information.

	# Start the master
	master$ sudo rm -f /tmp/g; gproc m
	# (I typically leave the master process in the foreground)
	# In a new window, first copy over the gproc executable to every node:
	master$ for i in n1 n2 n3 n4; do scp -o /path/to/gproc root@$i:; done
	# Start gproc on the slaves
	master$ for i in n1 n2 n3 n4; do ssh root@$i './gproc -myId="hostname hostbase" -myAddress="hostname" -parent="master" s &'; done
	# Now run a command
	master$ gproc e . /bin/hostname
	n1
	n2
	n3
	n4

	# If we wanted to copy a file to every node's /root/, for example, we'd do this:
	master$ gproc -f="/path/to/file" e . cp /tmp/xproc/path/to/file /root/

When you are finished with gproc, you should be able to simply kill the master process; this will cause the slave processes on the various nodes to exit automatically.

Node specification syntax (BNF)

We use a special syntax for describing which nodes a given GProc command should affect. In the examples above, we used "." to specify all nodes. However, you might only want to run on half the cluster, or your locale might specify multiple levels of nodes; our syntax allows for that, as the BNF below shows.

	<nodes> ::= <nodeset> | <nodeset> "/" <nodeset> | <nodeset> "," <nodes>

	<nodeset> ::= <node> | <node> "-" <node>
	<node> ::= "." | <number>

Examples:

	1-80		# Specifies nodes 1 through 80, inclusive
	.		# Specifies all available nodes (flat network) or all first-level nodes (tree)
	1/.		# Specifies all second-level nodes under the first level-1 node
			# In the "kane" config, this would select the first 20 nodes
	./.		# All nodes, all levels. Note that this example is 2 levels, but there is no limit on depth.

GProc's Forth syntax

GProc uses a simple but powerful postfix command set (much like Forth) to specify the several of the command line options, such as a node's ID, its network address, and most importantly its parent in the GProc hierarchy. The commands are:

	*	- Multiple the top two stack elements
	+	- Add the top two stack elements
	-	- Subtract the top two stack elements (tos[1] - tos[0])
	/	- Divide the top two stack elements (tos[1] - tos[0])
	%	- Mod the top two stack elements (tos[1] % tos[0])
	roundup	- Round up tos[1] to the next highest integer defined by tos[0]
	strcat	- Concatenate the top two stack elements (tos[1],tos[0])
	ifelse	- if tos[0] == 0, replace tos[0] with tos[1], else tos[2]
	hostname- Push the os.Hostname field onto the stack
	hostbase- Strip the leading characters in the range a-z and . from tos[0], leaving only numbers
	dup	- Duplicate tos[0]
	swap	- Swap tos[0] with tos[1]
	
	Note: tos[0] is the top of the stack, tos[1] is the second elements, and so on.

The most important use for this syntax is specifying the myParent parameter, which tells each node who its immediate parent is. In the simple example above, the parent for every slave node is just the master node, so the specification is just the string "master". However, in our large cluster, we have 520 nodes--far too high of a load for the head node to handle efficiently. Instead, we want a two-level tree, with every twentieth node acting as an intermediate. This way, the head node only has to send traffic to 26 nodes, which will then forward it on to the 19 nodes under each of them.

The myParent flag for our large cluster, KANE, is shown below. The head node's IP is 172.16.255.254. Individual nodes are numbered kn1 through kn520.

-myParent="172.16.255.254 kn hostname hostbase 20 roundup strcat hostname hostbase 20 % ifelse"

Although this looks complicated, it can be broken down without too much trouble, starting at the right-hand side. "ifelse" means that if the value on the top of the stack is 0 (if tos[0] == 0), the value at tos[2] will be returned; otherwise, tos[1] will be returned. "hostname hostbase 20 %" takes the base of the node's hostname (e.g. "kn15" becomes "15") and takes it modulo 20. "kn hostname hostbase 20 roundup strcat" takes the base of the node's hostname (e.g. "15") and rounds it up to the nearest 20, then catenates that number with the string "kn" (thus our example "15" becomes "kn15"). The 172.16.255.254 is simply the IP of our head node. When it is time for the "ifelse" statement to be evaluated, the stack would look something like this (in the case of node kn15):

	15
	kn20
	172.16.255.254

When "ifelse" is evaluated, it sees that tos[0] is not 0 (meaning that the node number was not divisible by 20), so the current node (kn15) is on the second level of the tree and its parent is kn20. In the case of a node like kn40, the stack would look like this:

	0
	kn40
	172.16.255.254

Since tos[0] is 0, this node is in the first level of the tree; its parent is the head node, 172.16.255.254.

Command line options

There are several important command line options for use with gproc, the most important of which are a set of one-letter options which set the role for gproc. The basic roles for gproc are:

		  gproc [switches] m
		  gproc [switches] s
		  gproc [switches] e <nodes> <command>
		  gproc [switches] i
"gproc m" starts the master process and should be executed on the front-end node. "gproc s" starts the slave process and should be run on every node you wish to control. "gproc e" is used to actually run a command on the specified nodes. "gproc i" provides information about the first level of nodes (support for deeper levels will be added eventually). The slave role is the only role which will typically need additional options (specified below) in normal use, because the slave processes must be told who their immediate parent is in the GProc tree and by what name they should identify themselves. Typically, "myAddress" will be set to "hostname", and "myId" will be set to "hostname hostbase" or simply "hostname". The "myParent" option can vary, but it should always be safe to simply set it to the IP of the node running "gproc m". There are a number of switches which can modify the behavior of gproc; some of the most important ones are described here. Some only make sense in certain modes; each switch's appropriate mode(s) can be found in parentheses after the description. The default value for the option is listed as well.

Other options can be found in the source code but in general, those listed above are sufficient for all operations.