Panorama OO-Test

Basic Concepts

Panorama OO-Test is the leading testing tool on the market and has been integrated into Panorama software engineering environment. Panorama OO-Test is makefile-drive. It helps user to plan, run, and measure the testing effort. It provides accurate test coverage results down to the segment or condition level and offers many advanced testing features, including complexity analysis, frequency analysis, test case minimization, and test case-code correspondence analysis to make testing much more efficient. Panorama/OO-Test provides unexpected program termination location to assist user to solve the fatal run-time errors. The user friendly interface and the graphic representation of the testing data and program logic, make Panorama/OO-Test easy to use and understand.

Problems and solutions in software testing

Testing consumes at least half of the labor expended to produce a working program. The goal of software testing is bug discovery. The cost of testing may include: the cost of detecting them, the cost of correcting them, the cost of designing tests that discover them, and the cost of running those tests. It is obvious that to facilitate the testing is a major way to save the cost of and reduce the time during the software development cycle.

Software testing is an extremely creative and intellectually challenging task. It is probably true that the creativity required in testing a large program exceeds the creativity required in designing that program. Software should be thoroughly tested before its release to ensure both its reliability and its functionality. Unfortunately, it is too complicated and time consuming for testing engineers to cover all the situations of a large program. Realistically, for example, each part of the software can be tested respectively, with some paths being tested specifically. In this way, significant improvements in overall software quality can also be achieved.

On average, without coverage analysis tools, only around 50% of the source is actually tested before release.

With almost half of the source code untested, a lot of bugs are released together with the software. In this way, the quality, performance and the functionality of the software can not be assured. Besides, it is unknown for testing engineers and testing management when the testing should be finished, or how much further testing is required for a given program. The problems can be solved by introducing the concept of test coverage.

Test coverage is a measure of testing process from both a developer and a user perspective. The measure relates directly to the development, integration and operational test stages of product development, in particular, unit, functional, system and acceptance tests. Developers can apply the measure in unit test to obtain a measure of thoroughness of structural tests. System testers can apply the measure to gain a user-view of the thoroughness of functional tests, and to determine the amount of implementation in the operational environment.

Since various test coverage results may offer different measures concerning the structural and functional tests of the software, Panorama OO-Test provides the most comprehensive test coverage analysis capabilities to satisfy divers coverage analysis requirements.

Another problem in software testing is the inefficiency of the test cases and actually the inefficiency of testing. Many test cases merely cover the same logic parts of the program which have already been explored by the previous test cases run. In addition, it is sometimes unknown for testing engineers and testing management which test cases should be selected to test a modified logical unit, or which and how many test cases will be needed to test a revised program. To improve the test efficiency, the tools for analyzing each test case's scope of effect is required. Furthermore, the tools for minimizing the number of the test cases and, in the mean time, to assure the same test coverage result is of great importance.

Fortunately, OO-Test also provides these functions to reduce both the time and budget for the software development and testing.

Often, it is important to test some critical paths. The design of the test cases requires the through understanding of the control flow of the logic. For complicated paths, it is sometimes difficult to design test cases to cover all the logical parts along the paths. The tools for path testing is especially useful for the testing engineers. OO-Test provides the path/control flow analysis function which helps the users understand the logical control flow of a function and allows you to create the right test cases for more thorough testing.

Run-time fatal errors are often difficult to locate and may cause disastrous results. In order to fix the errors, programmers are keen to find out where and why the run-time errors occurred. OO-Test can clearly mark the latest termination location on the source, thus enable the programmers to fix the bugs quickly.

Sometimes, it is also important to determine the execution frequencies of different modules and segments in order to trace the program execution and find out the potential logic problems. With OO-Test, either the accumulated execution frequency or the last-run execution frequency of each segment and function can be clearly marked on the J-Flow diagram or provided in test coverage reports.

Although it is not likely that the cyclomatic complexity will eventually become a true bug predictor, and there are some weaknesses in the complexity metric and in its theoretical foundation, still cyclomatic complexity provides some useful rules of thumb in measuring the structural complexity of the coded modules and in improving the program modulation theory, etc. OO-Test provides McCabe's cyclomatic complexity data and graphically show the complexity in J-Flow diagram.

OO-Test also provides the Test Complexity (J-complexity) which is defined according to the efforts required for achieving different test coverage results. J-complexity is useful for professionals in planning the test activities and in measuring the true structural complexity.

Graphic representation of the testing results is an important aspects of the usability of the testing tools. Also important to the usability is the various aspects of the user interface including the on-line help function, the Graphic User Interface, the availability of the command line operation, and so on.

OO-Test is makefile-driven. It automatically generates the input bin file from the makefile, The input bin file is similar to a makefile. It lists the source files you want to examine. You can include the CPP directives -D, -U, -H, and -I in the input bin. You can also nest input bin files by listing the .hsi files required.

In all these fields aforementioned, Panorama OO-Test has offered comprehensive solutions and, in some respects, unique contributions to the software testing.

Features and Benefits of Panorama OO-Test

Panorama OO-Test offers object-oriented multi-level coverage analysis functions from low cost to high efficient. OO-Test provides the most comprehensive test coverage analysis capabilities to meet different testing requirements:

Some of the unique features will be introduced below.

SC0, the basic segment test coverage, is better than the statement test coverage A set of test cases satisfies the statement test coverage if all statements of the program have been executed at least once. It is unclear whether a decision statement should be considered executed if only the condition part has been checked but never satisfied. In the following example, there is a decision statement in which the condition part and the execution part of a decision statement are written in the same line (line 11):

testco1.GIF - 4.13 K

The condition of line 11 is never satisfied, the execution part (segment) has never been executed. It is difficult for statement test coverage tools to report the actual coverage of the above example, whereas Panorama OO-Test can clearly point out an unexecuted segment (the execution part of the decision statement) with its basic segment test coverage analysis function.

SC1, the standard segment test coverage, is better than the branch test coverage

A test based on execution of each branch does not enforce the execution of each statement. So, the branch test coverage can not be used to check dead code. For example, the following code contains an unexecuted lines (lines 10). It is difficult for the branch test tools to report the unexecuted code, whereas SC1 function of Panorama OO-Test can clearly highlight it in the J-Diagram. testco2.GIF - 3.37 K

File-scope and Program-scope for test coverage results

A class with in-line member functions can be defined in a header file that is included in several source files. Each of these files, after preprocessing, contains a copy of the class with all its in-line functions. So in each of these files, the class can have different test coverage through its instances in that file. Therefore, the test coverage of a class defined in a header file is usually calculated in file-scope. But for considering the reusability and the reliability of a class, users may often want to know the total test coverage of a project with the coverage of classes calculated in program-scope. That is, consider all these different copies of the class as one.

Panorama OO-Test provides class test coverage in both file and program scopes. Users can view the overall program-scope results generated by the reporter of OO-Test and by other tools of Panorama C/C++ when working with OO-Test. They can also view the detailed file-scope results on the J-Flow of OO-Test. For helping users to more accurately analyze test cases, both file- and program-scope analysis are provided by TCA of OO-Test.

For more information about the calculation of test coverage in File scope and Program-scope, please refer to the Test Coverage item in Glossary.

Test Case and Code Correspondence Analysis

Panorama OO-Test provides the Test Case and Code correspondence analysis function. It means that for a given test case, Panorama OO-Test identifies clearly what classes/ functions/segments can be tested by the test case; for a specific class/functions/segment, it determines which test cases can be adopted to test the class/function/segment. This function is very useful in unit re-testing. After a module-level modification, only the related test cases, which can be identified by this function, should be run again. In this way, the unit testing will be greatly speeded up.

Test Case Minimization

Panorama OO-Test also provides the function of Test Case Minimization which can be used to achieve significant reduction in time and cost for the system-level software re-testing. As we pointed out earlier in this chapter, many test cases duplicate the test coverage results as the previous ones, resulting in the waste of resources. By setting certain test coverage objectives, Panorama OO-Test can automatically extract a minimal set of the test cases from the original test cases and ensure the same test coverage results. After the system-level modification, only the desired minimal set of test cases should be run. This is extremely important for the testing of the large programs.

Unexpected Termination Locating

For unexpected program terminations,Panorama OO-Test can determine and graphically point out the location in the source code where the fatal run-time error occurred. Obviously this is a very useful function for programmers and testing engineers.

Path Analysis

In addition, OO-Test provides the path/control flow analysis function which helps the users understand the logical control flow of a function and allows you to create the right test cases for more thorough testing.

Complexity Analysis

OO-Test provides two kinds of complexity analyses which provide flexibility in software complexity measurement.

Cyclomatic complexity: This measure may be used to determine the structural complexity of a coded module. The use of this measure is designed to limit the complexity of a module, thereby promoting understandability of the module. For a typical module, ten represents a maximum ideal complexity. If the calculated complexity number is too large, modularization of the code is required.

Test Complexity (J-Complexity): Test Complexity focuses on software testing. It provides the precise measurement of the efforts required for achieving various levels of test coverage results.

Graphic Representation of Test Results and Program Logic

Another major feature of Panorama OO-Test is the graphic representation of the test data and the program logic. The testing results can be shown in bar graphs, charts, tables as well as on J-Flow diagrams. More than 40 different reports will be provided in Panorama OO-Test to document the various test coverage and complexity analysis results. As a versatile graphic notation as well as graphic interface, J-Flow can be used to show the program logic together with the test results, including the execution frequency of the functions and the segments, the segment/condition coverage, and with the unexecuted parts highlighted. In this way, it is easy to view the results and locate the problems in the source code.

Panorama OO-Test also provides the cross-reference function between J-Flow and the source code. Click on a statement of the source code, the corresponding segment on the J-Flow will be automatically located. Similarly, click on a segment on the J-Flow, the first line of the corresponding segment in the source will be located. It is valuable for programmers and testers to go through the program logic and locate the source code to be modified quickly.

User-Friendly Interface

Panorama OO-Test allows the specification of a project with only a makefile. It also provides the graphic interface for creating and running the test scripts. Graphic interface and the command line mode can be provided to meet different operation requirements. The powerful on-line help with dynamic cross-reference allows the users to access the required information easily and quickly.

Operations

This part covers:

Activating Panorama OO-Test
Preparing and running the test cases
J-Flow Diagram
Generating the J-Flow diagram
Understanding the J-Flow diagram
Invoking the program's source
Diagram-Source cross-references
Invoking the header files for reviewing
Setting tags
Searching for strings
Printing various parts in J-Flow, SOURCE or HEADER windows
TCA
How to activate TCA
Test cases and file/class/function/segment correspondence
Test cases to file/class/function/segment correspondence
File/class/function/segment to test cases correspondence
File scope and program scope
Case-to-Code situation recording and recovering
Edit the test cases highlighted by code-to-case correspondence
Save the accumulated test cases
File/Class/Function, Segment and Source cross-references
Efficiency
Minimization
Setting tags
Searching for strings
Print
Exiting Panorama OO-Test

Activating Panorama OO-Test

test.gif - 0.35 KTo invoke OO-Test from the Panorama C/C++ MAIN menu:

  1. Make sure that you have created or loaded a database for your project. (See "Creating and Loading a Database").

  2. Click on the OO-Test button on the tool bar of the MAIN menu. Alternatively, you may click the TOOLS menu, then click OO-Test option. The OO-Test menu bar will pop up.
    Tst_bar.gif - 3.74 K

    In the menu bar, the FOLLOW SUBSET check button is set by default. You can unselect it by clicking on the button.

    If you select FOLLOW SUBSET, OO-Test will test only the files you have selected in the Panorama C/C++ MAIN Menu. (See "SUBSET pull-down menu" .) If you unselect FOLLOW SUBSET, OO-Test will show the analysis results of all the files in the project.

Preparing and running the test cases

To acquire the dynamic analysis data of the program being tested, a series of test cases should be prepared and applied to the program.

To prepare the test cases:

  1. Click the left mouse button at SCRIPT on the OO-Test menu bar, the SPECIFY .HSI FILE NAME dialog box pops up:

    Tst_hsi1.gif - 3.56 K

    In this dialog box, the default .hsi file name has been loaded into the .HSI NAME field. Click OK to select this default .hsi file. The PANORAMA /OO-TEST SCRIPT window pops up.

    If you intend to select another .hsi file for testing, just click BROWSE. A SELECT .HSI FILE window will pop up for you to select a new .hsi file

    Tst_hsi2.gif - 7.55 K

    In the SELECT .HSI FILE dialog box, select the proper directory, click the right .hsi file name, the file name clicked will appear in the SELECTION field. Click OK, the file name will be loaded into the previous SPECIFY .HSI FILE NAME dialog box. Click OK on the SPECIFY .HSI FILE NAME dialog box to select the .hsi file. The PANORAME C++ /OO-TEST SCRIPT window pops up.

    Script.gif - 5.66 K

  2. Select SHELL: click the left mouse button at SHELL on the manu bar, then select the desired shell among Bourne shell, C shell and K shell.
  3. Define the SEPERATOR: click the left mouse button at SEPERATOR on the menu bar, the SEPERATOR DEFINITION dialog box pops up.

    You may type in the required symbol into the DEFINE SEPERATOR field, then click the left mouse button at ADD button, the symbol will be loaded into the SEPERATOR TABLE and will be identified as a separator when it appears in the test case script file. An empty line can also be defined as a separator. To define the empty line, click the left mouse button at the EMPTY_LINE box above the DEFINE SEPERATOR field, then click the left mouse button at ADD. To delete a separator from the SEPERATOR TABLE, click the left mouse button at the separator to highlight it, then type RETURN.

    After defining the separator, click CLOSE to shut down the box.

  4. Click the left mouse button at PROPERTY to select the mode of edition: Overwrite or Addition.

    Overwrite: When new test cases are loaded or typed into the OO-Test Script window, the old test cases previously loaded or typed will be overwritten.

    Append: New test cases loaded or typed into the OO-Test Script window will be appended to the end of the original test cases available in the OO-Test Script widow.

  5. Prepare the test cases:

    If the test cases have previously been saved in a script file, you can input the test cases by clicking the left mouse button at the FILE menu on the menu bar, then click the left mouse button at the LOAD option. A LOAD TEST CASE SCRIPT dialog box will pop up.

    Load_tcs.gif - 3.77 K

    In this box, the current directory is shown in the DIRECTORY field. You can type the test case script file name into the FILE NAME field, then click the left mouse button at OK.

    Alternatively, you may click the left mouse button at BROWSE. The SELECT TEST SCRIPT FILE dialog box pops up.

    Slec_tsf.gif - 8.79 K

    In the SELECT TEST SCRIPT FILE dialog box, the FILTER field shows the current directory, and the DIRECTORIES and FILES fields display, respectively, the subdirectories and files under the current directory.

    To change to the directory where the test case script file is to be searched, you can type the desired directory into the FILTER field. Alternatively, you can change directory by means of the DIRECTORIES field. Click the left mouse button at .. to change to the parent directory, click the left mouse button on the name of the subdirectory to change to it.

    Click the left mouse button at the desired file name in the FILES field, the file name will appear in the SELECTION field. You may directly type the file name into the SELECTION field too. Click OK, the script file name selected will appear in the FILE NAME field of the LOAD TEST CASE SCRIPT box. Click the left mouse button at OK of the LOAD TEST CASE SCRIPT box, the test cases in the script file specified will be loaded into the OO-TEST SCRIPT box.

    NOTE: If you wish to list all the files in a directory, the correct FILTER format is the full directory specification followed by "*".

    If the test cases are not available, click the left mouse button at EDIT on the menu bar of the OO-TEST SCRIPT box to activate the SCRIPT EDIT box.

  6. On the SCRIPT EDIT box, specify the executable file name of the software to be tested in the PROGRAM NAME field, then enter each test case as follows:

    Scripted.gif - 7.52 K
    1. Input the value of the test case into the ARGUMENT field.
    2. Input the command, if any, to be executed before the execution of the case into the COMMAND BEFORE EXECUTION field. Multiple commands can be separated by semicolons.
    3. Input the command, if any, to be executed after the execution of the case into the COMMAND AFTER EXECUTION field. Multiple commands can be separated by semicolons.
    4. Input the comments, if any, into the COMMENTS field.
    5. Click the left mouse button at APPLY, the test case will be sent to the OO-TEST SCRIPT box.
    6. Click the left mouse button at CLEAR button, then start to input another test case.
    7. After finishing the input of the test cases, close the SCRIPT EDIT box.

    Although one script file may contain test cases defined for testing more than one executable files, we recommend that the test cases contained in the same script file are defined for the same executable file.

  7. You may edit the test cases on the OO-TEST SCRIPT box.

  8. To save the screen of test cases, click the left mouse button at the FILE menu on the menu bar, then click the left mouse button at SAVE option. A SAVE TEST CASE SCRIPT dialog box pops up.

    Savescri.gif - 4.00 K
    In this box, the current directory is shown in the DIRECTORY field. You can type the test case script file name into the FILE NAME field, then click the left mouse button at OK.

    If you intend to use an existing script file to store the test cases, instead of typing the file name into the FILE NAME field, you may click the left mouse button at BROWSE. The SELECT TEST SCRIPT FILE dialog box pops up.

    Slec_tsf.gif - 8.79 K

    In the SELECT TEST SCRIPT FILE box, the FILTER field shows the current directory, and the DIRECTORIES and FILES fields display, respectively, the subdirectories and files under the current directory.

    To change to the directory where the script file containing the test cases will be stored, you can type the desired directory into the FILTER field. Alternatively, you can change directory by means of the DIRECTORIES field. Click the left mouse button at .. to change to the parent directory, click the left mouse button on the name of the subdirectory to change to it.

    Click the left mouse button at the desired file name in the FILES field, the file name will appear in the SELECTION field. You may directly type the file name into the SELECTION field too. Click OK, the script file name selected will appear in the FILE NAME field of the SAVE TEST CASE SCRIPT box. Click the left mouse button at OK of the SAVE TEST CASE SCRIPT box, the test cases will be saved into the script file specified.

    NOTE: If you wish to list all the files in a directory, the correct FILTER format is the full directory specification followed by "*".

To run the test cases:

  1. Click the left mouse button at RUN on the menu bar, the SPECIFY TEST DATABASE box pops up with the current directory shown in the DIRECTORY field.

    Testdb.gif - 3.90 K
    The DIRECTORY field contains the path where the test data base file is to be created. You should specify the proper directory in this field to store the test data base files for the programs tested.

    The test data base file (.tdbs file) records all the test cases which have been executed during the test run. Every time a test data base file is specified, the test cases executed will be appended at the end of the .tdbs file. It is recommended that each .tdbs file should be used to record the test cases for a specific input bin.

    You may type the name of the test data base file into the .tdbs FILE NAME field. Alternatively, you may select an existing database file.

    Click the left mouse button at BROWSE, the SELECT TEST DATABASE dialog box pops up.

    Selc_tb.gif - 7.54 K

    In the SELECT TEST DATABASE box, the FILTER field shows the current directory, and the DIRECTORIES and FILES fields display, respectively, the subdirectories and files under the current directory.

    To change to the directory where the test database will be stored, you can type the desired directory into the FILTER field. Alternatively, you can change directory by means of the DIRECTORIES field. Click the left mouse button at .. to change to the parent directory, click the left mouse button on the name of the subdirectory to change to it.

    Click the left mouse button at the desired file name in the FILES field, the file name will appear in the SELECTION field. You may directly type the file name into the SELECTION field too. Click OK, the file name selected will appear in the .tdbs NAME field of the SPECIFY TEST DATABASE box. Click the left mouse button at RUN of the SPECIFY TEST DATABASE box, an OOT_CMD window pops up displaying the information of the test run. After the execution of the test cases, you may select to retain or close the OOT_CMD window.

    NOTE: If you wish to list all the files in a directory, the correct FILTER format is the full directory specification followed by "*".

  2. After the execution of the test cases, activate the FILE pull down menu on the menu bar. You may click the left mouse button at PRINT to print out the script file, or click the left mouse button at CLEAR to clear the content of the script file, or exit the OO-TEST SCRIPT box.

J-Flow Diagram

J-Flow diagram shows path complexity by stripping the logic down to basic.This schematic of the decisions, loops, and switches show precisely where the code is complex, and help simplify the complexity of the code and improve the program maintainability and reliability.

The test coverage results and execution frequencies can be shown on the J-Flow diagrams. Unexecuted nodes and segments will be highlighted on the J-Flow. Unexpected termination location will be clearly marked on the diagram too. Each part of J-Flow is linked to the corresponding part of the source code. It is easy for users to view the test results and locate the problems.

Generating the J-Flow diagram

Click the left mouse button at the J-FLOW button on the OO-Test menu bar, a J-FLOW OPTIONS box pops up.

Tst_flow.gif - 8.16 K

J-FLOW FORMAT of the J-FLOW OPTIONS box defines the display mode of the test coverage. Click the left mouse button on one of the radio buttons to select the required display mode of control logic flow and the testing results.

J-FLOW SCOPE of the J-FLOW OPTIONS box defines whether the header files will be included in the J-Flow diagram. Many classes are defined in .h files. If the header files are included, how many member functions are included in each of these classes can be shown in J-Flow.

EXECUTION FREQUENCY of the J-FLOW OPTIONS box defines whether the last run execution frequency or the accumulated execution frequency will be displayed on the J-Flow. It should be emphasized that this selection is critical. If LAST RUN ONLY is selected, all the test coverage data as well as the untested elements highlighted in J-Flow will based on the test results of the last-run only. If ACCUMULATED RUN is selected, all the test coverage data as well as the untested elements highlighted in J-Flow will based on the accumulated data of all the test runs.

TEST COVERAGE SUMMARY FOR of the J-FLOW OPTIONS box defines if a test coverage summary will be provided at the beginning of the program, and/or at the beginning of each file, and/or at the beginning of each function. Make proper selections on the check buttons.

When PROGRAM button in the TEST COVERAGE SUMMARY FOR field is selected, CLASS SCOPE specifies how the test coverage data in the program summary is calculated.

When FILE scope is selected, class of the same name in different files are considered as different classes.

When PROGRAM scope is selected, all the classes with the same class name are considered to be the same class, no matter in which file they are located. Once a class is tested, all the classes with the same class name in other files will be considered tested.

Note: In case classes with the same class name are defined in different files and possess different structures, then the FILE scope should be selected.

Each J-Flow generated is based on the source files of one input bin. The default input bin file name is shown in the INPUT BIN field.

If the selections are made, click the left mouse button at OK button, the J-Flow diagram will be generated. The optional program level test coverage summary is shown below:

J_flow1.gif - 16.49 K

Understanding the J-Flow diagram

The J-Flow diagram contains, at the beginning, the optional test coverage summary, the histogram of the test coverage results and the histogram of the untested elements for the whole program to be tested. If it is specified in the J-FLOW OPTIONS window, similar test coverage summaries will also appear in front of each file and/or function.

J_flow2.gif - 12.25 K

In the J-Flow diagram, the first column from the left lists the source file line numbers of the corresponding logical parts in the diagram. The second line from the left lists optionally the header indexes corresponding to the source code of .h files included. Each function in the diagram begins with a function head enclosed in a double-line rectangular. The execution frequency data, whether last-run or accumulated, are listed to the right of the header rectangular, or to the right of the statements, or to the right of the specifications of nodes or segments. The unexpected program termination location during the last-run will be clearly marked in red. On the top of the window, displays the file name of the first line in the window.

J_flow3.gif - 13.98 K

Various test coverage results are marked on the J-Flow diagram. The execution frequency data marked to the right of the file heads show the test coverage of the functions, either last-run or accumulated. Since block coverage equals to the visible segment coverage, the block coverage is clearly shown as the execution frequency of each segment marked. Loop boundary coverage is indicated by the execution frequency data of loop boundary invisible segments. As to condition-decision coverage, each condition or decision is marked in blue if the condition or decision has been satisfied: F means that the condition or decision is tested but not satisfied; T means that the condition or decision is tested and satisfied. Two small blocks means that the condition or decision has never been tested.

Standard segment test coverage is calculated based on the execution frequency data of both visible and invisible segments. A class is considered tested if one of its member functions is executed.

Besides, untested elements will be highlighted in black blocks in J-Flow diagram and the execution frequency will be zero. It should be emphasized that the selection of Execution Frequency in the J-FLOW OPTIONS dialog box is critical. If LAST RUN ONLY is selected, all the test coverage data as well as the untested elements highlighted in J-Flow will based on the test results of the last-run only. If ACCUMULATED RUN is selected, all the test coverage data as well as the untested elements highlighted in J-Flow will based on the accumulated data of all the test runs.

For understanding the notations of the J-Flow diagrams, please refer to the attached J-Flow graph cell standard in Appendix.

Click the left mouse button at HELP on top right of the window will activate the on-line help function.

Invoking the program's source

Click the left mouse button at SOURCE on the menu bar of the J-FLOW window, the SOURCE window pops up.

J_flow4.gif - 21.77 K

In this window, the source code of the input bin .cc files are listed consecutively. At the top of the window is the file name of the first line. The first column from the left lists the line numbers in the source files. At the beginning of each file’s source code, the line number starts at 000001. The key words corresponding to those in the J-Flow are highlighted in red for easy identification.

Click the left mouse button at HELP on top right of the window activates the on-line help function.

Diagram-Source cross-references

J-Flow diagram is logically linked to the source file. Click the left mouse button on the J-Flow diagram, the line clicked will be highlighted in white. Also the corresponding line in SOURCE window will be highlighted in green. Similarly, click the left mouse button on the source code, the line clicked will be highlighted in white, and the corresponding part in J-Flow will be highlighted in green too. Since invisible segments do not have the corresponding code in the source files, click on the invisible segments in the J-Flow will not invoke this function.

J_flow5.gif - 24.49 K

For large programs with a lot of files, this cross reference function is very useful and effective for locating the problems or for program understanding, and without any limitations to the number of the source files, to the size of the source files, or to the locations of the source files.

Invoking the header files for reviewing

Another useful function provided is to open the header files for reviewing. Double click the left mouse button at one #include directive in either the J-Flow or the SOURCE window will activate a HEADER window displaying the header file in the #include directive.

J_header.gif - 8.66 K

Only one HEADER window can be activated at the same time. In the HEADER window, two header files can be opened simultaneously. Click the left mouse button at PREVIOUS on the top menu bar of the HEADER window will switch to another header file opened.

Click the left mouse button at HELP on top right of the window will activate the on-line help function.

Setting tags

In order to help users review the J-Flow diagram, to go through the source code in both .cc and .h files, tag setting function is provided in J-FLOW, SOURCE and HEADER windows. Users can attach tags to different positions of interest in J-Flow or source code where further examination may be needed.

  1. To attach tags, first click the left mouse button at TAG to pull down the menu.

  2. Then click the left mouse button at SET on the pull-down menu, an OO-TEST SET JUMP ID dialog box pops up.

    J_tag.gif - 2.90 K

    In the box, the top row number in the current window is provided. This number is the unique identification of this row in a window, and it is not the line number in a .cc or .h file. There may be more than one file of source code shown in the SOURCE window, and the line number in each file starts at 1. So the source line number in a SOURCE window may not be unique. Whereas in each window, the row number starts at zero with the increment of one for each succeeding row.

  3. You may adopt the row number as the identification for each tag, or you can type in any character combination to mark the tag.

  4. Click the left mouse button at OK, the tag is attached to the row.

In the same way, you can attach tags to other locations in the diagram or file.

To jump to a location marked with a tag:

  1. Click the left mouse button at TAG to pull down the menu.

  2. Click the left mouse button at JUMP. A list of tag identifications appears.

  3. Click the left mouse button at a tag will jump to its attached location.

To clear a tag:

Close a window will delete the attached tag list for the window. Click PREVIOUS on the HEADER window to switch to another header file will also delete the attached tag list for the header file.

Searching for strings

In J-FLOW, SOURCE and HEADER windows, another useful function provided is the FIND string function.

  1. Click the left mouse button at FILE pull down menu, then click FIND option, a FIND dialog box pops up.

    J_find.gif - 3.46 K

  2. Type the string to be searched into SEARCH FOR field.

  3. Click the SEARCH DOWN or SEARCH UP radio button.

  4. Click the left mouse button at NEXT, the first match to the string in the specified direction will be highlighted.

  5. Click NEXT again, another match will be identified.

  6. Click Clear, the SEARCH FOR field will be cleared.

  7. Click CANCEL will close the FIND dialog box.

Printing various parts in J-Flow, SOURCE or HEADER windows

The whole or part of J-Flow diagram or the code in .cc or .h files can be printed out.

  1. Determine the scope to be printed: Click the left mouse button at one location on the diagram or source code, then click the right mouse button at another location on the same diagram or source code, the diagram or code between the two points clicked will be highlighted.

    Click the right mouse button at other locations will change the size of the covered area.

    Click the left mouse button again will discard the selection.

    You do not need to highlight the contents if the contents of the whole window is to be printed or if the line numbers are to be specified on the J-FLOW PRINT dialog box.

  2. Click FILE on the menu bar.

  3. Click PRINT. A PRINT dialog box pops up.

    Jfl_prt.gif - 8.54 K

  4. Select TO PRINTER or TO FILE.

    If TO PRINTER is selected, you may type the printer ID into this field. If the TO PRINTER field is left blank when it is selected, system default printer will be used.

    If TO FILE is selected, a file name with full path is expected.

    You may click BROWSE to select a file and load the filename into the TO FILE field.

    By clicking BROWSE, a BROWSE dialog box pops up:

    Browse.gif - 8.20 K

    In this dialog box, you may select a proper directory in the DIRECTORIES field, then click the filename, the filename will appear in the SELECTION field. Click OK, this filename will be loaded into the TO FILE field of the PRINT dialog box.

  5. Select ASCII or POSTSCRIPT.

    This selection determines the format of the printout or the format in the specified file.

  6. Select ALL or HIGHLIGHTED or LINE.

    When ALL is selected, the contents of the whole window will be printed out or be saved into the specified file.

    When HIGHLIGHTED is selected, only the highlighted part will be printed out or be saved into the specified file.

    To highlight a part of content for printing, click the left mouse button at the beginning, then click the right mouse button at the end of the part to be printed.
    When LINE is selected, the range of lines should be specified to the right of the selection:
    FROM: The window line number of the first line to be printed.
    TO: The window line number of the last line to be printed.
  7. If POSTSCRIPT is selected, the following selections should be made:
    1. Select SINGLE PAGE or DOUBLE PAGE format.

    2. Select paper format: USLETTER, A4 or PAPER.

      If PAPER is selected, the paper size in inches is to be specified in the WIDTH and HEIGHT fields to the right of the paper format fields.

    3. Select margins: TOP, BOTTOM, LEFT and RIGHT.
  8. Click OK.

    The specified contents will be printed or be saved in the file.

Test Coverage Analysis (TCA)

Panorama OO-Test provides the most comprehensive test coverage analysis capabilities on the market to satisfy various coverage analysis requirements:

Besides, TCA offers you the unique test case and code correspondence analysis capability and the unique Test Case Minimization capability for C/C++ language.

TCA can determine each scope of test case of effect, that is, which class or function or segment can be tested by a given test case. Also, TCA can determine which test cases can be used to test a specified class or function or segment. This function is very useful for evaluating the efficiencies of the test cases and for retesting the specified classes or functions or segments after modification.

With test case minimization function, TCA automatically extracts a minimal set of test cases from the original test cases based respectively on the contributions of the test cases to the class coverage, function coverage, branch coverage, block coverage, segment coverage, etc. With this function, great amount of time and money can be saved for system level retesting.

How to activate TCA

To start TCA:

  1. Click the left mouse button at TCA on the Panorama OO-Test menu bar.A TEST COVERAGE ANALYSIS dialog box pops up.

    Tca_1.gif - 4.03 K
  2. In the TEST COVERAGE ANALYSIS dialog box, select proper .hsi file in the proper directory, then click OK. A Test Coverage Analysis window pops up. All the test cases applied to the program are recorded internally and listed in this window.

    Tca_2.gif - 8.41 K

Test cases and file/class/function/segment correspondence

Click the CORRESPONDENCES on the menu bar, then click FILE-CLASS-FUNCTION to activate the FILE-CLASS-FUNCTION window. In this window, lists the files, classes and functions contained in the user program being tested. The files, classes and functions are properly arranged in a hierarchical tree to show their interrelations.

Similarly, click the CORRESPONDENCES on the menu bar, then click SEGMENT option to activate the SEGMENT window. In the SEGMENT window, the J-Flow diagram lists all the segments contained in the user program being tested.

Properly arrange the three windows so that the contents in these windows can be seen simultaneously.

Test cases to file/class/function/segment correspondence:

Click the SELECTION pull-down menu, select SINGLE. When SINGLE is selected, each time only one test case in the TEST COVERAGE ANALYSIS window can be clicked to highlight.

Click one test case in the TEST COVERAGE ANALYSIS window, the test case will be highlighted in blue. Meanwhile, the corresponding files/classes/functions which can be tested by this test case will turn red in the FILE-CLASS-FUNCTION window, and the segments which can be tested by this test case will also turn red in the SEGMENT window.

Tca_3.gif - 8.99 K

Tca_4.gif - 10.47 K

Tca_5.gif - 10.93 K

Click the SELECTION pull-down menu, select MULTIPLE. When MULTIPLE is selected, more than one test case can be clicked to highlight at the same time.

Click on the test cases in the TEST COVERAGE ANALYSIS window, the test cases clicked will be highlighted in blue. Meanwhile, the corresponding files/classes/functions which can be tested by these test cases will turn red in the FILE-CLASS-FUNCTION window, and the segments which can be tested by these test cases will also turn red in the SEGMENT window.

File/class/function/segment to test cases correspondence:

Click on a file or a class or a function in the FILE-CLASS-FUNCTION window, this file/ class/function will be highlighted in blue. In the meantime, the corresponding test cases which can be used to test this specified file/class/function will turn to red in the TEST COVERAGE ANALYSIS window. In the SEGMENT window, the corresponding file/class/function name will be highlighted too.

Tca_6.gif - 8.41 K

Tca_7.gif - 9.41 K

Tca_8.gif - 9.65 K

Click the left mouse button on a segment in the SEGMENT window, this segment will be highlighted in blue. Meanwhile, the test cases which can be used to test this segment will turn to red in the TEST COVERAGE ANALYSIS window. In the FILE-CLASS-FUNCTION window, the function name, in which the specified segment exists, will be highlighted too.

File scope and program scope

In FILE/CLASS/FUNCTION window and SEGMENT window, OO-Test provides a choice of FILE SCOPE or PROGRAM SCOPE.

When FILE SCOPE is selected and a class in the FILE/CLASS/FUNCTION window or SEGMENT window is clicked, only the test cases that can be adopted to test the clicked class in the current file will be highlighted.

When PROGRAM SCOPE is selected, all the classes with the same class name are considered to be the same class, no matter in which file they are located. Once a class is clicked, all the classes with the same class name in other files will be considered clicked and are highlighted in blue. In this case, test cases which can test any one of these classes will be highlighted in red.

Similarly, if an in-line function of a class is clicked and turned to blue, other in-line functions with the same function name located in other classes of other files will also be considered clicked and be highlighted in blue. Test cases which can test any one of these functions will be highlighted in red.

Note: In case classes with the same class name are defined in different files and possess different structures, then the FILE SCOPE should be selected.

Case-to-Code situation recording and recovering

Sometimes it may be important to record a special group of test cases into a file so that the situation generated by this group of test cases can be restored later. OO-Test provides another useful function to save the group of test cases selected in the TEST COVERAGE ANALYSIS window.

In TEST COVERAGE ANALYSIS window, select one or more test cases, the files/classes/ functions/segments which can be tested will be highlighted in red. To save this screen, click the left mouse button on SELECTION menu, then click SAVE, a SAVE TEST CASE SELECTION dialog box pops up.

Tca_sav1.gif - 3.55 K

You may type the directory and file name into the DIRECTORY and FILE NAME fields. The file specified is used for recording the test cases. Or you may click BROWSE to invoke a SELECT TEST SCRIPT FILE dialog box to choose an existing script file to keep the test cases.

Tca_sav2.gif - 7.99 K

In the SELECT TEST SCRIPT FILE window, click the proper file name, then click OK. The file name selected will be loaded into the FILE NAME field of the SAVE TEST CASE SELECTION dialog box. Click OK button in the SAVE TEST CASE SELECTION dialog box, the test cases selected will be saved in the specified file.

To restore the group of test cases saved, click the SELECTION button on the TEST COVERAGE ANALYSIS window, then click LOAD option. The LOAD TEST CASE SELECTION dialog box pops up.

Tca_lod1.gif - 3.45 K

Again you may type the directory and file name into the DIRECTORY and FILE NAME fields respectively. Or you may click BROWSE to invoke the SELECT TEST SCRIPT FILE dialog box to load a file name into the LOAD TEST CASE SELECTION dialog box. Click OK, the test cases recorded in the specified file will be restored in the TEST COVERAGE ANALYSIS window with the selected test cases being highlighted in blue. If the FILE/CLASS/FUNCTION and the SEGMENT windows are opened when the test cases are loaded in, the files, classes, functions and segments which can be tested by those highlighted test cases will be shown in red.

Edit the test cases highlighted by code-to-case correspondence

When a file/class/function/segment in the FILE-CLASS-FUNCTION or SEGMENT window is clicked, the test cases, which can be used to test the code clicked, will be highlighted in red. These red-highlighted test cases can be listed in a separated window, edited and saved. The file thus saved can be loaded into SCRIPT window for retesting the source code, or can be edited and printed out.

To edit these red-highlighted test cases, click LIST on the FILE pull-down menu of the TEST CASE ANALYSIS window. The LIST HIGHLIGHTED TEST CASES edit window pops up.

Tca_edt1.gif - 8.21 K

The test cases highlighted in red will be listed in this window. You can edit these test cases, then save the test cases in a file for reloading or retesting.

You may set PROPERTY in the top bar to APPEND, then click other part of the source code in FILE-CLASS-FUNCTION or SEGMENT window to highlight some test cases in red, then click LIST on FILE pull-down menu to append more test cases to the LIST window.

Save the accumulated test cases

The test cases in TEST COVERAGE ANALYSIS window can be saved in a editable, printable and executable file. To save the test cases into a customized file, select SAVE in the FILE pull-down menu. The TCA SAVE dialog box pops up.

Tca_sav3.gif - 3.31 K

In this dialog box, specify the directory and file name, then click OK.

Alternatively, you may click BROWSE to open a SELECT TEST SCRIPT FILE dialog box. In the SELECT TEST SCRIPT FILE dialog box, select a directory and click the proper filename, the file name with full path will appear in SELECTION field. Click OK to load the path and file name into the DIRECTORY and FILE NAME fields of the TCA SAVE dialog box. Click OK to save the test cases.

The file thus saved can be reloaded in the SCRIPT window to retesting the program. Or this file can be edited and printed out.

File/Class/Function, Segment and Source cross-references

Click the left mouse button of SOURCE on the SEGMENT window menu bar, a SOURCE window pops up in which the source code of the program being tested are listed.

The FILE/CLASS/FUNCTION, SEGMENT and SOURCE windows are closely linked to each other. Click one element in a window, the corresponding elements in other two windows will be highlighted. In this way, it is easy for new users of Panorama C/C++ to find the corresponding locations of the source elements in the FILE/CLASS/FUNCTION and SEGMENT windows, and to facilitate their understanding of the FILE/CLASS/FUNCTION and SEGMENT windows.

In both SOURCE and SEGMENT windows, the HEADER window can be invoked. For example, double click the left mouse button at one #include directive in the SOURCE window, the header file of the #include directive will be shown in a HEADER window opened. Double click a class definition in SEGMENT window will also open the header file in which the class is defined.

Two header files can be opened simultaneously for your reference. Press PREVIOUS to shift between the two header files.

Efficiency

Test cases have different test efficiency. Adopting the highly efficient test cases results in great savings of test effort. To derive the efficiencies of the test cases:

  1. You should apply the test cases to the program in the SCRIPT window of OO-TEST first.

  2. Invoke TCA by clicking the button on OO-TEST menu bar.

  3. Click EFFICIENCY on the TEST COVERAGE ANALYSIS window.

    The TEST CASE EFFICIENCY ANALYSIS OPTIONS dialog box pops up.

    Tca_effi.gif - 5.44 K

  4. In this box, make the following selections:
    1. Select which test coverage analysis level is required. The default is four levels.
    2. Select PROGRAM scope or FILE scope.

      When PROGRAM scope is selected, all the classes with the same class name are considered as one class, no matter which files these classes are located. A test case, which tests one of these classes of the same class name, will be considered to have tested all of these classes.

      If FILE scope is selected, the classes of the same class name in different files are considered as different classes. One of these classes is tested does not automatically mean that other classes are tested by the same test case.

    3. Select all the test cases or part of the test cases specified.

      When ALL is selected, the efficiency data will be given for all the test cases listed in TEST COVERAGE ANALYSIS window.

      When SELECTED is clicked, the efficiency data will be given for those test cases clicked and highlighted in blue.

  5. Click OK.

    The EFFICIENCY ANALYSIS RESULT window will be generated.

    Tca_eff2.gif - 9.04 K

    In this window, lists the current date and time, the number of test cases included, the program scope or file scope selected, the test database file name and the input bin file name.

    Also listed is the test cases' serial numbers and their test coverage results of different levels specified.

    You can save the results in a file by clicking SAVE in the FILE pull-down menu, specifying the path and file name, clicking OK.

    You can also print the results by clicking PRINT in the FILE pull-down menu, specifying the target printer, the print format and the paper format and size, and clicking OK.

    If large number of test cases are listed, a tag setting function provided will facilitate your inspection of the test cases' efficiency results.

Test Case Minimization

In a large set of test cases, many test cases merely duplicate the test coverage results obtained by the previous test cases run. A lot of test efforts thus are wasted.

OO-TEST provides the test case minimization capability based on the test case efficiency analysis.

To acquire the minimized test cases:

  1. You have to apply the test cases to the program in the SCRIPT window of OO-TEST first.
  2. Invoke TCA by clicking the button on OO-TEST menu bar.
  3. Click MINIMIZATION on the menu bar of Test Coverage Analysis window.

    The TEST CASE MINIMIZATION OPTIONS dialog box pops up.

    Tca_min1.gif - 5.38 K
  4. In this dialog box, make the following selections:
    1. Select one of the four test coverage analysis levels.
    2. Select PROGRAM scope or FILE scope.

      When PROGRAM scope is selected, all the classes with the same class name are considered as one class, no matter which files these classes are located. A test case, which tests one of these classes of the same class name, will be considered to have tested all of these classes.

      If FILE scope is selected, the classes of the same class name in different files are considered as different classes. One of these classes is tested does not automatically mean that other classes are tested by the same test case.

    3. Select all the test cases or part of the test cases specified.

      When ALL is selected, the efficiency data will be given for all the test cases listed in TEST COVERAGE ANALYSIS window.

      When SELECTED is clicked, the efficiency data will be given for those test cases clicked and highlighted in blue in TEST COVERAGE ANALYSIS window.

  5. Click OK.

    The MINIMIZATION RESULT window pops up.

    Tca_min2.gif - 8.20 K

    In this window, lists the current data and time, the number of test cases before minimization, the number of test cases after minimization, the actual test coverage reached, the test coverage level selected, and the file scope or program scope selected.

    Also listed in this window is the minimized set of test cases obtained.

    The minimized set of test cases can be saved in a playback script file and be reloaded in the SCRIPT window of OO-TEST for other test runs automatically after code modification. To save the minimized set, click SAVE in the FILE menu. The MINIMIZATION RESULT SAVE dialog box pops up. Specify the directory and file name, then click OK. The minimized test cases will be saved.

    The minimized test case set can also be printed out. To print the minimized set of test cases, click PRINT on the FILE menu, the PRINT dialog box pops up. Specify the target printer, the print format and the paper format and size, then click OK.

    In this window, the string FIND function is provided. It is useful for large set of minimized test cases.

    If the number of minimized test cases is large, a tag setting function provided will facilitate your inspection of the test cases.

Setting tags

You can set tags in many windows. Tags can be attached at different locations of interest. You can even write a few comments as a tag which is helpful for your later reviewing.

To attach a tag:

  1. Click the left mouse button at TAG to pull down the menu.
  2. Click the left mouse button at SET on the pull-down menu, an OO-TEST SET TAG dialog box pops up.

    Tca_tag1.gif - 2.38 K

    In the box, the top row number in the current window is provided. This number is the unique identification of this row in a window, and it is not the line number in a .cc or .h file. There may be more than one file shown in the FILE-CLASS-FUNCTION window or SEGMENT window, and the line number in each file starts at 1. So the line number in the FILE-CLASS-FUNCTION window or in the SEGMENT window may not be unique. Whereas in each window, the row number starts at zero with the increment of one for each succeeding row.

  3. You may adopt the row number as the identification for each tag, or you can type in any character combination to mark the tag.
  4. Click the left mouse button at OK, the tag is attached to the row.

In the same way, you can attach tags to other locations in the windows.

To jump to a location marked with a tag:

  1. Click the left mouse button at TAG to pull down the menu.
  2. Click the left mouse button at JUMP. A list of tag identifications appears.
  3. Click the left mouse button at a tag will jump to its attached location.

To clear a tag:

  1. Click the left mouse button at TAG to pull down the menu.
  2. Click the left mouse button at CLEAR. A list of tag identifications appears.
  3. Click the left mouse button at a tag will erase it from the list.

Click CLEAR ALL will remove all the tags set.

Close a window will delete the attached tag list for the window.

Searching for strings

In many windows, another useful function provided is the FIND string function.

To use FIND function:

  1. Click the left mouse button at FILE pull down menu, then click FIND option, a FIND dialog box pops up.

    Tca_find.gif - 3.17 K
  2. Type the string to be searched into SEARCH FOR field.
  3. Click the SEARCH DOWN or SEARCH UP radio button.
  4. Click the left mouse button at NEXT, the first match to the string in the specified direction will be highlighted.
  5. Click NEXT again, another match will be identified.
  6. Click CLEAR, the SEARCH FOR field will be cleared.
  7. Click CANCEL will close the FIND dialog box.

Print

The contents in TEST COVERAGE ANALYSIS, FILE-CLASS-FUNCTION, SEGMENT, SOURCE and HEADER windows can be printed directly or can be saved into a file in either ASCII or PostScript format.

To use the print function:

  1. Click FILE on the menu bar.

  2. Click PRINT. A PRINT dialog box pops up:

    Tca_pnt1.gif - 9.52 K

  3. Select TO PRINTER or TO FILE.

    If TO PRINTER is selected, you may type the printer ID into this field. If the TO PRINTER field is left blank when it is selected, system default printer will be used.

    If TO FILE is selected, a file name with full path is expected.

    You may click BROWSE to select a file and load the filename into the TO FILE field.

    By clicking BROWSE, a BROWSE dialog box pops up:

    Browse.gif - 8.20 K

    In this dialog box, you may select a proper directory in the DIRECTORIES field, then click the filename, the filename will appear in the SELECTION field. Click OK, this filename will be loaded into the TO FILE field of the PRINT dialog box.

  4. Select ASCII or POSTSCRIPT.

    This selection determines the format of the printout or the format in the specified file.

  5. Select ALL or HIGHLIGHTED or LINE.

    When ALL is selected, the contents of the whole window will be printed out or be saved into the specified file.

    When HIGHLIGHTED is selected, only the highlighted part will be printed out or be saved into the specified file.

    When LINE is selected, the range of lines should be specified to the right of the selection:

    FROM: The window line number of the first line to be printed.
    TO: The window line number of the last line to be printed.

  6. If POSTSCRIPT is selected, the following selections should be made:
    1. Select SINGLE PAGE or DOUBLE PAGE format.

    2. Select paper format: USLETTER, A4 or PAPER.

      If PAPER is selected, the paper size in inches is to be specified in the WIDTH and HEIGHT fields to the right of the paper format fields.

    3. Select margins: TOP, BOTTOM, LEFT and RIGHT.
  7. Click OK.

    The specified contents will be printed or be saved in the file.

Exiting Panorama OO-Test

To quit OO-Test:

Click QUIT on the OO-Test menu bar.


back to Table of Contents