Chapter 9: Qsg: The Principal QEF Script Generator

What qsg is, how it works, command-line options, the scripting language and the library.

Although any arbitrary shell pipeline can be used as a script generator, the primary ones used are qsg and qefdirs. The qefdirs utility is used for multiple directories, and qsg is used almost all other times. qsg uses a library of pre-written scripts which address most of the common tasks performed in a software project. However, since a majority of qeffiles contain a qsg script after the Begin line, some knowledge of the qsg scripting language will be necessary. Chapter 7.5 introduced the basic facilities used in qeffiles. This chapter covers qsg and its associated tools and files in much more detail, although, full detail will require consulting the x-qsg database and the various qsg manual pages.

qsg is several things at once. First and foremost, it is a script generator used to create mimk or make files or shell scripts. Beyond that, qsg is a small programming language that can be used for a variety of applications such as this document, xfig diagrams, the vci interface, and small programming exercises such as Scientific American puzzles and other such problems.

Using qsg on simple files is probably the best way to learn it, as in:

    % cat > tmp
    if @argvv
	return No arguments specified
    fi
    >Arguments: @argv
    ^D
    % qsg -f tmp
    No arguments specified
    % qsg -f tmp hi there
    Arguments: hi there

    % qsg -c hanoi 5
    1:	move ring 1 from this one to that one
    2:	move ring 2 from this one to over there
    3:	move ring 1 from that one to over there
    4:	move ring 3 from this one to that one
    ...
    % qsg -c commands example.c
    #include mimkvars.qh
    #include	deps.qh
    
    #define _Main_ example
    Objects:V:	example.o
    example.o:P: _S_(example.c) _Touch_(cc)
    	_T_cc -c _F_cc _F_cc[example.c] _F_cc_c _F_cc_c[example.c] \
	    _Optimize_(example.c) _Threads_(example.c) \
	    _CcFlags_(example.c) _D_cc _D_cc[example.c] \
	    _InclFlags_() _A_(example.c)
    ...
    ...

Every directory in which a construction occurs must have an associated file called qeffile (or qeffile2). The part of that file that follows the Begin line, is usually a qsg script. Everything that applies to standard qsg scripts applies also to the script portion of a qeffile.

The most common qeffile looks something like this:

    Begin


    commands @argv 

qsg M

The M argument specifies that the basenames of the directory's source files -- stored in srclist._ -- are to be assigned to the qsg variable argv, and qef pre-processor #srcmap directives for the sources are output (see M(x-qsg)) commands is a qsg library routine. The variable argv is the list of arguments to the qsg script. For the qsg script commands, each argument is processed according to its suffix. This script compiles each source file in the current directory into an executable. Scripts to build a library or to construct a program from all the files in the current directory are almost as simple.

Note:

Chapter 7.5: The Construction Script presents an extensive discussion of the use of qsg in construction scripts and describes the 10 most commonly used scripts such as commands, program, and finstall.

The qsg components of the QEF product consist of the following tools and files:

ant.qsl	Script to generate ant input files.
cvs.qsl	vci library to interface to CVS.
dtree.qsl	Scripts to process non-product Q-Tree tools, support, and testing processes used within the Q-Tree itself. For a list of functions use: x-qsg L \*dtree_qsl.
html.qsl	HTML functions as used to create this guide. For a list of functions use: x-html L.
misc.qsl	User contributed scripts provided as examples or starting points for creating your own scripts. Not supported. For a list of functions use: x-qsg L \*misc_qsl.
p4.qsl	vci library to interface to perforce.
qmsdev.qsl	Contains a script that generates a Microsoft Visual Studio compatible project file. Script is invoked by qmsdev.
rcs.qsl	vci library to interface to RCS.
rcsupd.qsl	Scripts to manage updates from remote development site. Not supported.
sccs.qsl	vci library to interface to SCCS.
vci.qsl	Subroutines used by the other vci libraries.
xfig.qsl	Scripts to generate xfig objects.
xppt.qsg	Scripts to generate presentation slides via xfig for those people who cannot or will not use power point.

To get a listing of the scripts provided by a particular library use:

    % qsgdump -L -llib -l-N

l-N

    % qsgdump -S -llib script

X

S

    % x-qsg
    x-qsg : qsg and its support library eXplanatory D.B.
    
	   qsg-intro : introduction to qsg
	   qsg-tools : qsg tools and files
	       qsg-M : that qsg -M flag
       qsg-variables : variables in qsg
	        argv : arguments to a qsg script
 <Q>/lib/qsg/std.qsl : the standard qsg library
	 qsg-sfxmaps : qsg mapping of suffix to script controls
	 qsg-scripts : list of standard qsglib scripts
	   qsg-input : qsg input conventions and the @ escape
	  qsg-syntax : qsg keywords and syntax
	   tilde-ops : the qsg postfix  list operators
       qsg-functions : list of all qsg-functions
	 qsg-opcodes : list of qsg opcodes and their interpretation
    
    See also: qsg qsg(1) qsgdump qsgcomp qvrs x-qsg x_db

    % x-qsg qsg-M
    qsg-M : that qsg -M flag
    
    As has been mentioned in qsg-intro, about 50% of the Begin
    lines are:
    
    	Begin qsg -M

    or its equivalent:

	Begin
    
    ...

• Qsg Keywords e.g., if, set
• Qsg Functions e.g., exists, readword
• Qsg Tilde Operators e.g., @Lr
• Qsg Standard Scripts e.g., commands, finstall

The synopsis of the qsg command is:

    qsg [qsg-options] [--] [script-options]

    % qsg -x
    qsg [-xVMmqrST] [-C[mp]] [-a file] [-f script]
	[-o outfile] [-l lib] [[-[-c]] args]
    
    the qef script generator interpreter
    
    -x		display this explanation
    -V		output version information
    -M		equivalent to `-m -a srclist._' -- see -r
    ...

Typically, qsg is called with one of the following options:

M	construct the product from the list of source files (srclist._)
f <file>	interpret the named file
c <command>	execute the specified command

The qsg M command suffices for the majority of software constructions. Using the list of source files (usually the sls produced srclist._), and the directory's qeffile as the input script, it builds the construction in the current directory.

The qsg -f <file> option is usually used to run a script that is not contained in a script library, as in:

    % qsg -f file -- arguments

The qsg -c <command> option is usually used directly from the command line, as in:

    % qsg -c hanoi 3 # run 3 ring tower of hanoi

c

-

To run a qsg library script, you simply need to know its name and the flags and arguments it accepts. The result of running a library script is almost always another script to another process. That is, scripts are both the input to and output from qsg. Running the qsg script to create a library or program, for example, does not directly create the library or program. Rather, it creates a mimk script to accomplish this. To actually create the program or library, you must interpret the generated script using the appropriate back-end (e.g. mimk). This processing, that is the invocation of qsg to create the script, and mimk to interpret it, is all managed by qef.

For a list of the scripts available in the qsg library, use the command:

    % x-qsg -L \*scripts

For a detailed description of any particular script, use the command:

    % x-qsg <scriptname>

The standard qsg library consists of seventy five scripts designed to handle the most common tasks involved in software production. We recommend that you do not modify any of these scripts, but we encourage you to examine them for useful constructs. To get the source for a specific library module, use:

    % qsgdump -S script

Some of the most important library scripts are described in Chapter 7.5: The Construction Script.

A project might need to extend the range of facilities offered by std.qsl to deal with languages and/or file types not currently handled. While you could add new scripts to the standard qsg library, we recommend that you create new a new project or organization specific qsg library instead. This is best done as follows:

1. create a directory called qsglib (or qsl) in the top directory. This could be done within the project's tree or in a supplementary project created to hold organization specific QEF files -- see Chapter 12.11.
2. create the required scripts in that directory with *.qsg suffixes. Avoid using any names already in use in std.qsl, other than qsl_init.qsg which is required in most qsg libraries. A qsl_init script might be required to map suffixes to processing scripts -- see qsl_init(x-qsg).
3. create the following qeffile in the new directory:

       set Suffixes qsg
       Begin qsg -M -l-n
       mkqsglib -nname -v @argv

   Note:

   The l-n suppresses the normal inclusion of @QsgLibs named libraries. The name should be a meaningful name that does not clash with any *.qsl file in <qtree-root>/lib/qsg. The v, which causes a version string to be added to the library, is optional.

4. Add the following line to the qeffile in the top directory, assuming that it is a qefdirs script:

       qsglib B

   The `B' specifies that the install operation is to be done if the Install or Local constructions are requested. The line must appear before any other directory that uses the new library.
5. Add the following lines to the top level tree.vrs file:

       addpath QsgLibPath @_DestDir_/lib/qsg @ProdDir/lib/qsg
       append  QsgLibs  name

   If ProdDir is not been defined, elide the ProdDir directory. The name is the one used in step 3.

A qsg script has the following structure:

    summary <flag_spec> [-] <argument_spec>
    <explanatory text>
    endsummary

    # zero or more procs
    proc <name> <flags> [-] <arguments>
    ...
    endproc

    <script> 

Both the summary and the procedure definition(s) are optional. The summary defines the flags and their types accepted by the script. The summary/endsummary keywords can also bracket a text description, displayed by qsgdump -X. The proc keyword begins a procedure. Procedures are local to the defining script. The arguments specify the name of the procedure and the flags, if any. The procedure terminates with endproc. The [] in the synopsis is literal (i.e., not an optional `'). A `' is used to terminate flags if the first argument begins with a `'. See *defs-kws(x-qsg) for more information on the summary and proc keywords.

The actual script consists of keywords, variables, and arguments.

Variables are lists of words. A variable's value may be incorporated into an argument list by placing the @ character before it: @L is the value of the variable L. Literal spaces, tabs, and `@'s can be embedded in an argument list by preceding it with an `@'. The @ character can also be used to escape newlines to split long lines.

Any list can be modified by a post-fix operator (e.g., x/sfxs/, which selects those members of the list with the suffix "sfxs".). The tilde operators may be compounded by concatenation, and are evaluated left to right, with each operator operating on the list created by the preceding operator. See Appendix F: QSG Listops.

summary [-ab] [-p parm] [-l list]+ [-o[opt]] [-] @
	arguments ...
short optional description

-a		a boolean flag -- variable aFlag contains value
-b		another flag
-p parm		value assigned to pParm -- only one allowed
		second instance will raise fatal error
		This description should include the default.
-l list+	values assigned to lParm -- multiple instances
		allowed
-o[opt]		optional value -- oOpt0 is zero if -o not
		specified, one if it is.  Optional value assigned
		to oOpt1
args ...	strictly for documentation -- no interpretation
		all arguments after flags assigned to argv
		If first argument begins with `' precede with `'

This is a short description of the script.  When compiled, this
description can be extracted using qsgdump.  qsgdump can also
produce an x_db entry for inclusion in a x_db database.

The summary text is optional, but the flags list is required if
script has any flags.  The [-] is required.  The other `['s and
`]'s, other than those in -o[opt] are optional.
endsummary

# example proc
proc num [-rl] [-] ...
	# if ! -r return decimal form
	if ! @rFlag
		returnval @argv
	fi
	# @(libraries) yields list of loaded libraries
	# @List=/str/ yields elements in List equal to str
	# @Listv yields 1 if List is empty
	if @(libraries)=/dtree/v  @# test if dtree lib loaded
		qsglib dtree  @# load dtree library
	fi
	# if -r return roman numeral; lower case if -l
	# @<roman ...> is call to library script roman
	# @(flags l) is call to built-in flags function which
	#  will yield -l if @lFlag != 0 (i.e., -l was specified)
	returnval @<roman @(flags l) @argv>
endproc

cset pParm default @# set pParm to default but only if
		# not already set.
>output this line and the number of arguments: @argvl
<\1>Output argument list, splitting over multiple lines@
	if necessary with \ escaped newlines and 1 tab indent@
	for subsequent lines.  Arguments are: @argv

# output roman numerals for 1 to 20
set I 1
repeat
	>@I: @(num -rl @I)
until @[I @(expr @I + 1)] > 20
# @[I list] assigns list to I
# Note: @[/list] sets global variable named by first
#	element to the balance of the list
# @(expr ...) evaluates argument arithmetic expression

# output roman numerals for arbitrary list
for I in 4 49 94 56
	>@I: @<roman @I>
endfor

# can just name library script or proc as first keyword
hanoi 3  @# run towers of hanoi for three rings
# For hanoi source use: qsgdump -S hanoi
# For a similarly recursive merge sort see: qsgdump -S sort

# list files in current directory by suffix
set list @<readlist -p ls>      @# get list of files
for sfx in @(uniq @liste)      @# for each unique suffix
	<1>@sfx:@	@listx/@sfx/   @# list *.sfx files
endfor
if @[nosfx @list!m/*.*/]n
	<1>None:@       @nosfx  @# list files without a suffix
fi

Note:

For discussion of use of postfix operator see faq7(x-qmisc).

Local variable names are one of more alphanumerics or underbars, the first of which cannot be a digit (i.e., the same as C modulo the `$' allowed by some C compilers).

Global variable names can be any normal printable unsigned ASCII character greater than a space (i.e., bang to tilde), however use of `(' or `)' will make it awkward to retrieve the value using the literal name.

First time switch names can also be any sequence of printable characters excluding white space (one can use escaped spaces and tabs if really necessary).

The local variables are local to the current instance of the script or proc. When a script or proc is invoked, the flag processing then will set any specified flag's variable. The simple X flag's variable (e.g., XFlag) is set to 0 or 1 depending on whether or not the flag was specified. The arguments to a parameter flag P are assigned to PParm and OOpt is set to any optional parameter values. The argv variable (which is always defined) is set to the arguments of the call after the flags and options. All other local variables are set to the empty list.

Global variables are, obviously, global and are accessible to any script or proc, as are first time switches.

Note:

Given the global nature of global variables and first time switches, the function @(scriptnum) is often used as part of such a variable's name when access to such a variable should be limited to the current script, its procs and possibly subsequent or recursive calls to the same script.

Global variables are set or modified using gset, gappend, gpop, and gprepend. The latter two functions are often used to implement stacks. In addition to the keywords, the special expansion @[/List], as in:

    ... @[/ Example example value]

First-time-switches are used to prevent processes from being done a second time such as the processing for a source file or the output of the include of a header file. A first time switch can be set using the keyword set1st or the test and set function 1stset as in:

    if @(1stset switch-name)
	# switch-name was not previously set but it is now
	# ... first time only code
    else # 1st time switch was set previously
	# 2nd and subsequent time code
    fi

This section briefly discusses some of the principles that should be followed in developing new scripts that will conform to the normal style and form of the standard scripts.

    % qsgdump -S qsl_init

    % qsgdump -X script

    #target file description

    #item file1 file2 ...

Ptargets

    % x-qsg -L \*dtree_qsl \*misc_qsl

    % qsgdump -llib script

This text is written in qsg using the html library, which will be replaceable by an XML library in the future. This permits the use of variables such as @QSG which has been set to "@<tool qsg>" and will be translated into:

    <a href="appendix-D.html#qsg"><i>qsg</i></a>

    chapter -inum @(flags pn) Title ... @# p and n flags
		# are previous and next links.  Note '@#' used to comment
		# in an argument list as '#' is treated literally.
    section title ...
    label title ...

    % qsg -c hanoi 3 # run 3 ring tower of hanoi

t

    <<
    ...
    The text is just entered using the ...
    ><label Command In Middle of Output
    More text, some in @<b bold> ...
    ...
    >>

    gset PageMap @(call readlist @(findfile page.map))
    if @(option PrintVersion)
	qsg2html @argv
    else
	qsg2html -d @{_DestDir_}/pages @argv
    fi

    c080.html:P: _S_(c080.qh) _DestDir_/lib/qsg/guide.qsl \
		<qtree>/lib/qsg/html.qsl
	    qsg -lguide -o c080.html -c _S_(c080.qh) \
		-p chapter-08.html -n chapter-10.html
    
    #rmobj c080.html

Yet another qsg script is used to invoke netscape remote commands to produce the PostScript versions of the pages. The PostScript files are then read by still another qsg script to modify the clipping box, insert page headers and footers that incorporate the section titles and page numbers and produce the page number mappings to be used in the table of contents and keyword index, also produced by qsg scripts.

Finally qsg scripts are used to create the installation packages and book production forms.

Hence a single qef command will install the web pages, produce the book form, create the distribution package, and do the Chapter 10: file system integrity check via the simple application of a few qsg scripts.

For your edification, the frequently referenced hanoi source is listed below for those who have not as yet installed the Q-Tree:

% qsgdump -S hanoi
summary [-] n [from to spare]
run the towers of hanoi for N rings

N		number of rings to be moved
from		name of origin tower
to		name of target tower
spare		name of spare tower

Well why not?  If names of towers not provided,
default names used.

This script is provided for illustration and testing.
endsummary

proc hanoi [-]
	if @argv0
		nset n @argv0 - 1
		hanoi @n @argv1 @argv3 @argv2
		# retrieve, increment and reset global Move#, and
		#	output the ring move
		>@[/Move# @(expr @(g Move#)+1)]: move ring@
			@argv0 from @argv1 to @argv2
		hanoi @n @argv3 @argv2 @argv1
	fi
endproc

if @argvv
	fatal @(script): ring count not specified
fi
if !@(check @argv0)
	fatal @(script): @argv0: @(chk_msg);@
		1st argument must be numeric
fi
if @[N @(chk_value)] < 0
	fatal Tell me my dear @{Logname}, just what does@
		a negative number of rings look like?
fi
drop1st argv
if @argvv  @# if no arguments left
	set argv this@ one that@ one over@ there
elif @argvl != 3
	fatal arg count - must be zero or three towers specified
fi
gset Move# 0	
hanoi @N @argv

qsg is a remarkably useful little tool and it is fun to use.