nc

Section: LOCAL USER COMMANDS (1L)
Updated: August 5, 2004
Index

NAME

nc - distribute jobs from a build across a network of machines

SYNPOSIS

nc [-r] [-p directory] [-j maxjobs] [-t type] buildcmd
nc -i|-c [-r] [-f|-m] [-p directory]
nc -s [-b] [-q] compilecmd

DESCRIPTION

nc is a tool to distribute build commands executed from a build tool (such as make or scons) across a network of machines sharing a common clock and filesystem. nc relies on the build tool itself to parallelize the commands to be executed as part of the build. When executed as nc make or nc scons, nc will set up a shared memory segment, contact a set of potential remote build machines and begin tracking load averages on the working remote machines. It will then invoke the build tool (along with whatever command line options the buildcmd includes) with some environment variables set that allow the build script used by the build tool to know that an nc build is in progress; this way the same build script can be used for both nc and non-nc builds. The build script should arrange to prefix any build command to be distributed with the expansion of the $NC environment variable. For example, in Makefiles:


     CC := $(NC) $(CC)

or in Sconstructs:

     import os
     env = Environment()
     try:
         env['NC'] = os.environ['NC']
     except KeyError:
         pass
        
     env.Prepend(CC="$( $NC $)")

to allow for the distribution of invocations of the C compiler. Further, build commands can also use the $NCQUIET and $NCBIG variables, which can be used to cause nc to not echo command lines or to attempt to use only NCFASTHOST hosts respectively. nc is known to work well with GNU Make and SCons.

OPTIONS

-r: relaunch nc on one of the available build machines rather than running locally
-p: use the specified directory (which must exist) rather than the current working directory as the shared memory key and for keeping build status
-j: limit the maximum number of jobs run in parallel to maxjobs (can only be used to lower the maximum from the .ncrc configuration file)
-t: set the machine type of the build master (see SYNTAX/NCTRANS below)
-i: print build information about a running or the last completed build
-c: clean up shared memory and/or build status information
-f: get information or clean only the build status file (for a completed build)
-m: get information or clean only the shared memory (for an active build)
-s: run nc as a slave (normally only invoked from within the build tool)
-b: run this job as a big job
-q: run this job quietly

EXAMPLES

nc make
nc -i
nc -c

EXIT STATUS

nc always returns the exit status of the invoked build tool, if the build tool was successfully invoked, otherwise, it returns:

1: the command line arguments are invalid
2: the configuration file was invalid
3: there was a shared memory problem
4: there was a network timeout
5: there was an unclassifiable error
6: there was some kind of resource error
7: nc appeared to be running already in this directory
8: there was some kind of filesystem or path resolution error

SYNTAX

The nc configuration file will be looked for in the following places: ./.ncrc, ~/.ncrc and /etc/ncrc. Only the first file found will be used. An nc configuration file is a text file made up of single line commands. Within a configuration file, '#' is a comment character, anything up to the next newline is part of the comment.

The available configuration file commands are as follows:

NCDEFCPU cpus: cpus is the default number of cpus for any following NCHOST or NCFASTHOST, until the next NCDEFCPU command. The default value of cpus is 1.
NCDEFJOB jobs: jobs is the default number of jobs for any following NCHOST or NCFASTHOST, until the next NCDEFCPU command. The default value of jobs is 2.
NCHOST host [jobs [cpus [type]]]: specifies that the given host is available to be used for remote jobs. jobs and cpus specify the number of jobs to run on that host and the number of cpus to assume that host has. type specifies the type of machine that this host is--if unspecified, it will be determined from the result of a uname command.
NCFASTHOST host [jobs [cpus [type]]]: specifies that the given host is available to be used for remote jobs, and is a fast host capable of handling big jobs. jobs and cpus specify the number of jobs to run on that host and the number of cpus to assume that host has. type specifies the type of machine that this host is--if unspecified, it will be determined from the result of a uname command.
NCMARGIN margin: specifies the minimum required delta between the number of available job slots and the maximum number of jobs that will be run in parallel. It is unwise to run a build with no spare job slots, as then nc will be unable to try to avoid more heavily loaded machines. The default value of margin is 2.
NCMAXJOBS maxjobs: specifies the maximum number of jobs to run in parallel. The actual number chosen will be the lesser of 100, maxjobs, and the number of available job slots less margin. NCMAXJOBS can be overridden at nc invocation time by the -j parameter, but it can only be lowered there.
NCVAR varname: specifies the name of an environment variable that must be sent to the remote machine running any job. The value of each variable is collected at the time the job is being launched, in case some environment variables change value during the course of a build. Generally, you will probably want to export at least the PATH environment variable.
NCTYPE type: declares a typename to be used in NCHOST, NCFASTHOST and NCTRANS commands. If no types are declared, then transforming build commands to support heterogenous builds is not enabled. It is recommended that the typenames used be the same as running a uname -s command--if so, then normally NCHOST and NCFASTHOST declarations need not specify a typename, as nc will discover the types automatically.
NCTRANS type1 type2 pattern1 pattern2: declares a translation to be used to convert commands from a form suitable for a type1 host to a type2 host. The idea is to convert command lines generated on the build master (the host on which nc was originally invoked, or the host picked by nc -r) into a suitable command for the host on which the command is to be executed. The patterns are just plain strings--no regexp support is provided currently. NOTE: this is the least tested feature of nc--it is believed to work but has not been extensively used.

FILES

/etc/ncrc
~/.ncrc
./.ncrc
.ncstat
.ncmaster

AUTHOR

Steven Ellis <ellis@brouhaha.com>

This document was created by man2html, using the manual pages.
Time: 19:16:09 GMT, August 05, 2004

nc