19. Distributed Processing This section discusses the use of qremote to support distributed processing or running tools limited to specific hosts.

All one has to do is to specify the list of hosts that may be used to execute specific tools.

_Servers_[service]	To specify that a tool is be distributed or run on specific hosts, one sets a qvrs associative array variable called _Servers_ with the index being the name of the tool, to the list of hosts which may be used to run the tool. For example: set _Servers_[cc] host1 host2 host3 indicates that cc commands are to be distributed to host1, host2, and host3. When qef is processing its input, it will modify the definition of any _T_service for which a _Servers_[service] is defined to indicate to mimk that the service is to be distributed. mimk in turn will transform the commands so indicated to insert "qremote DRSservice" before the tool's name. So if the recipe was: _T_cc -c ... echo.c ... mimk will actually execute: qremote -DRScc cc -c ... echo.c ... Note: The "cc" in the recipe will be replaced by the value of _T_cc if it was defined. When qremote is invoked with a Sservice flag, it retrieves the value of _Servers_[service], checks which hosts in the list are available (using qhost) and then uses qdsrv to select the least recently used host (see below) in the list. The argument command (i.e., "cc -c ...") is then run on the selected host. Note: The D flag indicates that the command is to be run in the current directory, albeit from the remote host. The R flag indicates that any recursive calls to qremote should be ignored. Recursive calls might arise if a tool, (e.g., purify) runs another tool (e.g., cc) both of which have _Servers_[] settings.
Limitations	The target hosts must be able to access the directory on the current host in which qef is being run. They must have an installed Q-Tree and qhost must be running on it. The cost of running qremote, rsh, cush and the overhead of the messages to the qdsrv and qhost servers is not trivial. The savings in running a process on another host will have to offset these overheads, thus its use should be limited to expensive processes.
Where to set _Servers_	The _Servers_[service] can be set in any of the qvrs files, include the traits file for the host. It can be modified as required in the more local qvrs file (e.g., qeffile), but should probably be limited to platform specific files as there are no checks (as yet) to ensure that the target platform is compatible with the current host.
Least Recently Used Host	qremote needs to pick a host from the list named by the _Servers_[] list. It first sends messages to the target host's qhost server which also returns the host's Q-Tree directory and the user's home directory. This determines which hosts are available. qremote then sends the list to qdsrv which maintains an internal list of hosts and when they were last selected. qdsrv then returns the name of the least recently selected host. It might be preferable to select a host based on its current load, however, that is difficult to determine on some systems. Furthermore the return on investment is minimal as it is likely that one is cycling through the list of server hosts and the least-recently-selected scheme does that effectively.
MAXMKPROCS	This qvrs variable specifies the maximum number of parallel processes to be run by mimk. If using the distributed processing facility, this variable could be increased to take advantage of the available servers. For example, one could set MAXMKPROCS to three times the number of servers available as in: set MAXMKPROCS @(expr 3 * @_Servers_[cc]~l)

cook30.qh - 1.4 - 03/11/06