Section 7. Test and Checkout Procedure

: Testing the Covered tool for general "goodness", which is required for release, is accomplished with a suite of C and Verilog diagnostics. These diagnostics are located in the "diag" directory within the main Covered directory. These suite of tests are run in a regression manner; that is, each diagnostic is self-checking and run in serial order. The C regression is run from the "c" directory while the Verilog diagnostic regression is run from the "regress" directory. The C regression test is used to test out specific functions within the code that might otherwise not be adequately tested/testable in the full system. The Verilog diagnostic suite is used to verify that Covered works correctly as a whole. The following subsections describe the testing methodology used by both test suites.

: The C regression test suite consists of single file C code that includes the necessary header files from the "src" directory and specifies the necessary source files from the "src" directory in the linking phase. Each C diagnostic must contain a "main()" routine, and it must print the keyword "PASSED" if the diagnostic was considered successful or print the keyword "FAILED" if the diagnostic was not considered successful. This output must be sent to standard output as well as the regression output file "regress.log". After all diagnostics are run, the output file is grep'ed for the keyword "PASSED". The number of diagnostics finishing the PASS message are compared against the total number of diagnostics. The results of which are output to standard output.

: The Verilog regression suite consists of five directories: regress, verilog, cdd, err, and rpt. The regress directory is the directory where all Verilog regressions are run from. In this directory is the main Makefile and supporting *.cfg files. Each *.cfg file is named after its corresponding Verilog diagnostic file and in it contains all of the options to be passed to Covered on the score command-line. The *.cfg file is passed to Covered using the score command's '-f' option.

: The verilog directory contains all of the Verilog diagnostic files, Verilog include files and Verilog library files necessary to run regression. Additionally, two extra files "Makefile" (Makefile in charge of running the Verilog diagnostics and verifying their output) and "check_test" (Perl script used to perform verification of test passing) exist to handle diagnostic running and output checking. All output from a regression run is also placed into this directory.

: The cdd directory contains a generated/scored CDD file for each Verilog diagnostic. The CDD files in this directory are CDD files determined to be good by the test writer. When a diagnostic is run in which the output is deemed to be good, the *.cdd file from the diagnostic run is copied to this directory. All CDD files in this directory are used to compare the final CDD output from a diagnostic run to determine if the generated CDD file is correct. Compares are performed via the "diff" Unix command.

: The err directory contains error output generated from running Covered in such a way that produces an error. It is used as a means of testing Covered's error detection capabilities in a regression environment.

: The rpt directory contains a generated module (*.rptM) and instance (*.rptI) report for each Verilog diagnostic. Like the cdd directory, the rpt directory contains generated reports that were deemed to be correct by the diagnostic writer. When diagnostic is run in which the output is deemed to be good, the *.rptI and *.rptM files from the diagnostic run are copied to this directory. All report files in this directory are used to compare the final report outputs from a diagnostic run to determine if the generated report files are correct. Compares are performed via the "diff" Unix command.

: When a Verilog regression is run (from the "regress" directory), all diagnostics are run through the score and report commands (only a handful of diagnostics are run through the merge command also) in the "verilog" directory. The generated CDD, module report and instance report files are placed into the "verilog" directory. When these output files are generated for a diagnostic, the check_test script is run for that diagnostic to compare the new outputs with the known good outputs. If an output file is not found to differ from the golden version, the newly generated output file is removed from the verilog directory. If an output file is found to differ from the golden version, the output file is not removed from the verilog directory (this makes it easier to identify which diagnostics failed after regression). If all output files match, the keyword "PASSED" is sent to standard output and the passed counter in the "regress.output" file is incremented by one. If at least one output file does not match, the keyword "FAILED" is sent to standard output and the failed counter in the "regress.output" is incremented by one. If a diagnostic is considered a failure, regression continues to run until all diagnostics have been tested. After all diagnostics have been run, the contents of the "regress.output" file is output to standard output to indicate the number of passing and failing diangostics.

: The reason for having two directories for regression testing relies on the feature under test. Verilog diagnostics are condensed DUTs which only contain the required code for testing a particular syntax of the Verilog language to verify that Covered is able to correctly parse the code and generate the appropriate coverage results for that feature. All Verilog diagnostics are accompanied by a text file that is used for comparison purposes. The Makefile, after simulating the Verilog file, creating the dumpfile, generating the CDD and generating a verbose report based on the CDD will compare the generated report to the text file by performing a UNIX "diff" command. If the results of the "diff" are no differences between the two files, the Makefile will assume that the diagnostic has successfully passed and output the keyword "PASSED" to the output result file. If the results of "diff" show that there are differences between the two files, the Makefile will assume failure and output the keyword "FAILED" to the output result file.

: C diagnostics exist to test certain functions of Covered rather than the entire tool itself. Many times it is impossible/impractical to create Verilog diagnostics that exercise certain functions within Covered to completion. In these cases, it is often easier to write more specialized tests that can more quickly manipulate inputs to functions and verify that all output values are correct. Examples of C diagnostics that currently exist in the C regression directory include tests of bitwise operators, mathematical functions, etc.

: It is suggested that if functions can be adequately tested at a system level, that it be done so using Verilog diagnostics as these will get the most testing out of the entire tool. However, if functions are best tested in seclusion, it is suggested that the C testing environment be used.

Go To Section...