Chapter 10: The File System Integrity Check (FSIC) System

The File System Integrity Check (FSIC) package is a combination of files, programs and a qsg script (see fsic(x-qsg)) that aid in the maintenance of databases within the fsic directory of source, object, and installation trees.

When qef is run in the FSIC directory in a source tree, it compares the list of existing files against a list of files that are supposed to exist and reports any discrepancies such as missing files, new files, or files without a version system administration file.

Similar types of discrepancies are also reported when qef is run in the FSIC directory of an object tree. In this case, the list of existing files and directories for both the object and destination trees are checked against a database that defines the set of files that should exist and the exceptions or extensions to that set for specific configurations.

Typically the source tree check is run manually by the project administrator on a regular basis to ensure the completeness and correctness of the source file system. The object tree check should be run automatically as part of the build process and whenever the database is modified.

Given the volatility and size of modern software projects, we believe the application of the FSIC package is essential to preserve and protect the health of a software project.

This chapter serves to introduce the FSIC system. The definitive documentation is contained in the x_db database x-qfsic, as will be described later, and the manual sections fsic(1), sfsic(1), and bfsic(1).

    % x-qfsic
    x-qfsic : file system integrity package eXplanatory D.B.
    
    fsic-intro : Introduction to the FSIC package
    Src : procedure to check the source tree
    Post : procedure to check the object and installed trees
    Setup : procedure to set up Post files
    fsic-programs : the supporting FSIC programs and files
    fsic-files : files of the fsic directory
    fsic/qeffile : fsic qeffile script and its contents
    Src-lists : lists of source files created by Src process
    Src-files : lists of files possibly produced by Src process
    Post-files : lists of files possibly produced by Post process
    sfsic-errors : list of possible errors reported by sfsic
    bfsic-errors : list of possible errors reported by bfsic
    bfsic-types : types used in bfsic

    /root.vrs -tgz -w,s.cons

The type/pattern action lines specify what actions to take for files with specified types. For example:

    &Sgs Ss -e07 missing g-file for S type file

    % dirsetup fsic
    - create QefAdm/fsic/deviates.fl
    - create QefAdm/fsic/except/example.fl
    - create QefAdm/fsic/except/read.me; mode=0444
    - create QefAdm/fsic/files.fl
    - create QefAdm/fsic/inst.fl
    - create QefAdm/fsic/objs.fl
    - create QefAdm/fsic/qeffile; mode=0444
    - create QefAdm/fsic/read.me; mode=0444
    - create QefAdm/treedirs/read.me; mode=0444
    - create QefAdm/treedirs/treedirs.lst

The read.me files are just for documentation. Brief descriptions of the other files follow:

FSIC qeffile script -- see fsic/qeffile(x-qfsic).
Generates script to invoke various FSIC checks as listed by:

    % qef -Ptargets
    Src:	check source directory (source tree only)
    Checks:	check s-files vs. g-files (source tree only)
    Post:	check object and installation trees
		(object tree only)
    Setup:	build config files if required
    Note:	Only allowed in a subdirectory called fsic.

list of all known and regular source files -- see files.fl(x-qfsic).
This file is a list of all the files of the source tree including those that have a version system administration file but are obsolete. The lines in the file consist of the pathname relative to the source root directory and a type which is one of:

S	registered S-file -- should be both a g-file and a version system administration file.
U	registered unadministered file -- should be a g-file but no version system file
T	registered unadministered temporary file -- a known temporary file that is to be preserved but is not part of a source distribution
O	registered old file -- has a version file but no text as the file is deemed obsolete or old.

list of all known installed tree paths -- see inst.fl(x-qfsic).
This should be a sorted list of the superset (see note) of files and directories to be found in the Destination directory. Directories must have a trailing `/'.

Note:

The {inst,objs}.fl lists should list all the files to be found on all platforms for all configurations, within reason. The exceptions to these lists are specified in the individual system's exception file. However, in some instances (particularly on windows) files may exist on only one platform, thus it is much easier to not name such files in the master list, but just name it in the exception list. Otherwise such a file would have to be named in every other exception list as a legitimately missing file.

list of all object tree paths -- see objs.fl(x-qfsic).
This should be a sorted list of the superset (see above note) of files and directories to be found in the object tree. Directories must have a trailing `/'.

exception list -- see <sysnm>.fl(x-qfsic).
This file lists the exceptions to the inst.fl and objs.fl lists for the <sysnm> system. In addition to the system name, exceptions are bound to a key specified by @FSIC_Key (defaults to `a') which is used to specify the configurations to which the exception is applied. The exceptions are specified using one of the following keys: the UPPER case keys are applied to the installation directory, whereas the lower case version applies to the object tree:

E/e	File or directory is should exist (i.e., is extra) in the installed/object tree. The file should exist in specific tree, but should not be named in the corresponding inst.fl or objs.fl list.
N/n	Named file or directory and its contents should not to be found in installed/object tree.

Note:

The syntax and contents of this file and the deviations file are described by the command: % bfsic -X -I ilist -- list of existing installed files to be checked ...

list of normal deviations -- see deviates.fl(x-qfsic).
This file lists normal deviations, that is files that can normally exist, but deviate from the FSIC database. For example, the <qtree>/data/traits directory will contain host.tab files which cannot be registered in the inst.fl list due to the range of possible names. The entries in this file consist of a pathname (in which the file name can be a pattern) and one or more of the following types:

O/o	The Installed/Object file or directory is optional.
R/r	The Installed/Object file is required -- used to override an [NnOo]-type exception.

As is the case with the exception list, the UPPER case types refer to the Destination tree, whereas the lower case version refers to the object tree.

list of qeffile and *.vrs files used by treedirs -- see treedirs.lst(x-qfsic).
This file is actually created by the Src construction. It can also be created using treedirs L.

    # run rls in the source root to write file list to ,tmp
    % cush -d ../.. -o. ,tmp rls -sf

    # if using CVS, list of CVS Repository put into ,tmp2

    # run sfsic to check list against files.fl
    % cush -o ,s.errors sfsic -Ffiles.fl -S ,tmp

The list of files being edited and lists of the source files and temporary files are useful, but the most important product of the process is the ,s.errors file. This file will contain lines consisting of a filename, a tab, `#' followed by an error code, a brief description of the error, and the types for the file as assigned by the files.fl file and the sfsic.cf processing. The following are example lines:

    missing	# e07 missing g-file for S type file (type=Ss)
    newfile	# e05 new g-file (type=gz)

    % x-qfsic e05
    e05 : new g-file
    
    Explanation:
    The named file exists, but is not any recognizable type thus
    is assumed to be a g-file, but it is not named in files.fl.
    
    To Fix:
    If the file should not exist, it should be removed:

	% rm path

    Otherwise, if the file should be administered, a version file
    should be created and then registered as `S' type file in
    files.fl:

	% vci new path
	% sfsic -CS path

    Otherwise, the file should be registered using:

	% sfsic -CU path # if file is source but unadministered
	# or
	% sfsic -CT path # if file is not source but is preserved

    See also: sfsic-errors

Note:

When getting started there will probably be a large number of errors. However, two sfsic features can help. sfsic can read the files to be processed from stdin and the se-code option can be used to select the error codes to be processed, hence one can process all the e05 codes at once using: % sfsic -f -CS -se05 < ,s.errors

The error messages are much the same as those for the source tree, specifically a file name, a tab and a hash, an `i' or `o' followed by an error code, and a short description of the error. For example:

    missing	# i08 missing inst file (type=L)
    newinst	# i05 new inst file (type=F)
    missing	# o08 missing objs file (type=l)
    newobj	# o05 new objs file (type=f)

    % x-qfsic bfsic-errors

    path Nab

    # add newtypes and remove oldtypes for file ...
    % bfsic -A<newtypes> -R<oldtypes> files ...

    # add type L for ,p.error named files with error code i08
    % bfsic -si08 -AL < ,p.errors

    man/cat1/*.1 O

    data/qremote/*  O

    data/qremote/README  R

Initially the exception file will not exist. The command

    % qef Setup