High Integrity Unit Testing with advanced DUnit V9.3 code.
DUnit.
DUnit is an excellent aid to creating and maintaining code to meet
design criteria. It provides a very convenient framework to test pieces
of code as they are written and as they are modified later. Automated
testing after each code change gives confidence that code will perform
as designed when deployed in production code. During development a
change in one piece of code may pass it's associated unit tests but
still cause failures elsewhere in the project. Project wide automated
test suites help catch such errors early, while the developer is
focused on the recently altered code and so significantly reduces
debugging time and effort. Although DUnit has been closely allied with
"Xtreme" techniques it is equally valid as a tool for individuals or
disparate groups and quality assurance testers.
Empty Test Detection
Memory Leak Detection
Handling legitimate leaks in test suites
Memory Leak Detection at Shutdown
Showing TestCase Run-Time Applied Properties
Displaying Overridden GUI Settings
Finding TestCases containing armed Run-Time Properties
Displaying Allowed Leak Size in Run-Time Properties
Halt Repeated tests on first failure
Design Goals
Summary of New Features
Downloads
Terms used
Empty Tests.
Previous versions of DUnit can convey a false sense of completeness
when unit tests are left empty or fail to call Check() within an
individual test case's test procedure.
Completely empty tests only failed when compiled with "Optimization ON"
and passed otherwise. Developers frequently compile with "Optimization
OFF" and so can easily overlook empty tests while a test suite is being
developed. All tests containing at least one valid Delphi statement
will always pass, regardless of the compiler optimization state.
Instances have been encountered where complex test cases contain calls
to Check() but the execution path bypasses the Check() statement. Such
tests always pass, regardless of errors in the code under test, giving
an optimistic impression of code correctness.
To detect all empty tests, regardless of compiler optimization and test
cases which fail to call Check() select the GUI menu option as shown
below.
In the rare circumstance where it is inappropriate to call Check() from
within a test case, the global GUI setting can be overridden to prevant
false test failures. This is achieved by placing the following
statement within the individual test case's test procedure.
FailsOnNoChecksExecuted := False;
Empty test failures take precedence over memory leak failures.
Back to Top
Memory Leak Detection.
The effect of memory leaks in production code varies from no ill effects to shutdown disasters.
Production code can be executed by memory and resource leak detection
tools such as CodeWatch and MemProof but it's a tedious and non
automated task. The code sequence leading to leaks and hence their
discovery may never be exercised in the tester's environment. Whilst
individual test cases can be also executed under the same tools, the
tedium of repeatedly running more than a few tests becomes unrealistic.
Test scenarios may create objects, strings, exceptions and populated
dynamic arrays, perform some function and finally call Check() to
report success or failure. With DUnit's new leak detection enabled if
the test case passes but memory is leaked the test case now fails, The
error message shows the imbalance for, Setup, test procedure or
TearDown. DUnit reports memory "imbalance" i.e. where memory is
consumed or retuned to the memory manager so the developer can make
informed decisions about leak locations. Test cases do not fail where
all memory used during the test is returned to the memory manager by
the end of TearDown. Failures can optionally be ignored for SetUp and
TearDown, however many unit tests are structured such that TearDown
does the cleanup so this option may change in later versions.
Leak detection is globally enabled from the GUI under the "Options" main menu.
Below we see the result of running Leak Detection on code which was previously thought to be free of memory leaks.
Deeper investigation can reveal however that leaks may be legitimate in the context of unit testing and not problems in the application code under test.
Executing test suites a second time without closing DUnit generally recovers assigned strings memory.
This is particularly noticeable where strings declared and assigned in test cases are not cleared in TearDown.
The string' ref count may not decrement to zero until DUnit closes.
The screen shot below shows the dramatic effect or re-executing the test suite a second time without closing the test project.
Back to Top
Handling legitimate leaks in test suites.
Unit test Setup and Teardown itself can legitimately contain code
which does not release memory back to the memory manager. To prevent
failure of a test case containing legitimate memory retention the
developer can choose between several options. The first offers the
least protection against real code leaks and simply prevents failure on
any leak size.
Adding the following code to the test procedure or alternatively to
SetUp if all tests in the suite need to be prevented from failing.
FailsOnMemoryLeak := False;
If the legitimate leak is of known size the developer can add the following code to the pest procedure.
AllowedMemoryLeakSize := xxx; // where xxx is in bytes, as reported but the executed test.
In a multi threaded environment FastMM does not guarantee to always
allocate a predictable size of memory and may allocate a larger block
than requested. Other situations arise too where string lenghts may be
differ from day to day, for instance when a long date format is used.
Consequently it may be advisable to allocate several discrete memory
leak sizes.
The developer can the add the following code to the test procedure.
SetAllowedLeakArray([XX, YY, ZZ]); //to a maximum of three values, where XX,YYand ZZ are in bytes, as reported by the executed test.
The user can also globally ignore leaks caused solely by code in Setup
and Teardown and just remain armed to fail on leaks in the test
procedure.
The above GUI setting can be overridden at the test case level by setting:-
IgnoreSetUpTearDownLeaks := True; //False as appropriate.
Lastly, memory recovered by a test that was leaked by the prior test
will show as a negative leak. Currently there is a run-time property to
force a failure if this occurs. This feature will probably disappear in
future versions. Ideally tests should not be written to rely on any
other tests having been executed. Having said that, in a QA test
environment this is precisely what happens. Real leak sizes might
therefore be misleadingly reported on some tests when run in succession
and be accurate when run in isolation. Keep that in mind while
debugging code.
Back to Top
Leak Detection at Shutdown.
Finally, FastMM provides a leak report on shutdown which has a more
detailed explanation of what leaked but applies to the whole test
executable. It is of most value when it can be targeted at specific
test cases that are run in isolation then DUnit is shut down.
Leak detection on shutdown is invoked at the GUI level regardless of other leak detection settings.
FastMM's report looks like this:-
Notes.
Leak reporting on shutdown is automatically available when the code is
compiled under BDS2006 and the Unit Tests are executed from the IDE.
For Delphi5..Delphi2005 the following applies.
As shipped, FastMM4options.inc does not define
"ManualLeakReportingControl". To invoke a leak report on shutdown
ManualLeakReportingControl needs to be defined either of 2 ways.
1 In Delphi, Project - Options - Conditional defines, before the project is compiled, or
2 In FastMM4options.inc by removing the . in the line containing {.$define ManualLeakReportingControl}
Leak reporting as described is only available when the code is executed from the IDE.
To obtain leak reporting on shutdown outside Delphi's IDE follow the instructions in FastMM4options.inc.
Back to Top
TestCase Run-Time Applied Properties Popup Menu.
If you have read this far you may have noticed, that code affecting the
new modes of test failure can be placed within Setup, (affecting all
tests within a test class) or within individual test procedures. This
code obviously has no effect until the test actually runs. Nor does the
code have any effect if an ordinary failure occurs because the latter
takes priority in being reported.
This web page title mentions "High Integrity Unit Testing". So what are
the implications of overriding at the test procedure level a GUI
command to fail a test. Such overrides could go undetected, buried deep
in test code and silently mask potential failures. Consequently, a
Popup Menu has been added, which reports the actual properties that
applied when the test completed execution.
To Access the Run-Time Applied Properties:-
1 Select an individual test case in the Test Hierarchy.
2 Right click and select TestCase Properties at the top of the popup menu. -or-
2 Press Crtl+Shift+T.
The basic popup menu is as shown below.
Back to Top
Displaying Overridden GUI Settings.
The purpose of overriding a global GUI setting such as FailsOnNoChecksExecuted is to prevent an inappropriate failure.
A good example of this situation occurs in DUnit's own UnitTests
project and so is illustrated below. Not all tests need to call Check()
because the test might not have a checked result. Consequently when a
test legitimately does not call Check() we place a GUI override
statement in the test procedure.
FailsOnNoChecksExecuted := False;
Subsequently, when the test is executed the value of a new Results View
value "Overrides" is incremented. This serves to indicate, regardless
of the GUI option "Warn if Fail Test Overridden" setting.
To assist developers in locating Overridden GUI settings an Options
menu item is provide, "Warn if Fail Test Overridden". When checked and
underlaying test procedure passes the result is shown in Yellow instead
of Green.
Selecting the Run-Time Applied Properties shows that the GUI command to
"Fail TestCase if no checks executed" has been overridden at the test
case level and is highlighted in Yellow.
It is important to be aware, that in this scenario, Yellow signifies that a potential failure has been masked.
The reverse situation where a test case is armed to fail but the GUI is
not requesting a failure does not produce a Yellow "Caution". If a
failure results, when the test suite is run from the GUI the
developer's attention will be grabbed by the familar Magenta color
against test case and the Failure count increments.
Finding all TestCases that contain assertive Run-Time Properties
The following non persisted menu option is provided to highlight test
cases, that are armed at the test case level to fail when any of the
following are set true. "FailsOnNoChecksExecuted", "FailsOnMemoryLeak",
"FailsOnMemoryRecovery" or IgnoreSetUpTearDownLeaks.
The same function also highlights test cases containing a non zero
"AllowedMemoryLeakSize", regardless of whether "FailsOnMemoryLeak" or
"FailsOnMemoryRecovery" are set true.
Note. After selecting either "Warn if Fail Test Overridden" or "Show
TestCases with RunTime Properties" it is necessary to re-run the unit
tests to display the results.
Running the DUnit "UnitTests" under Win32 with the above selection produces the following result.
Back to Top
Display of AllowedLeakSize in Run-Time Properties
When "Show TestCases with RunTime Properties" is set, test cases with a
non zero AllowedLeakSize value are highlighted in yellow after
executing the test suite.
Test cases containing a range of allowed leak sizes established using
the SetAllowedLeakArray statement are similarly highlighted.
The allowed leak size programmed by the developer is displayed as shown below.
When more than one allowed leak size has been set using the
SetAllowedLeakArray statement the "intelligent" display shows the value
which allowed the test to pass or the lowest value in the set if the
test failed.
Back to Top
Halt Repeated Tests on first failure.
DUnit included several Test Decorator classes, which can provide group
SetUp and TearDown procedures and the ability to run counted multiple
tests. The latter has been expanded to allow a counted test to halt at
the first failure, rather than continuing to execute N tests, with the
subsequent N failure reports filling up the Failures View.
Design Goals.
The overriding goal in designing the new high integrity additions has
been to ensure that existing test suites do not break. This aim has
been achieved. Note that no attempt has been made to invoke the new
class of "Soft Failure" when the tests are run via TextTestRunner.
Empty Test failures behave exactly as before until the new GUI option "Fail TestCase in no checks executed" is selected.
For D5..D2005 the option to include FastMM is made by a global
conditional define. Leak checking Options are disabled until FASTMM is
defined.
Similarly post execption leak reporting is disabled until
ManualLeakReportingControl is added to the project conditional defines
and then corresponding "Report memory leak type on Shutdown" is
selected.
Finally the DUnit test program requires additional code in the uses
section of the project.dpr to access FastMM. Until FastMM is linked
into your DUnit test suite and compiled the Options menu items relating
to memory leak detection will remain disabled.
Back to Top
Summary of Additional Features.
- GUI selectable Memory Leak Detection at the test case level
- GUI selectable Memory Leak display after shutdown
- GUI selectable detection of test cases without effective call to Check()
- Addition of Test Decorator to allow a failure to halt repeated test loops.
- Retention of Results View data when individual test cases are clicked.
- Run-Time Properties Menu property names are "copied-to-clipboard" when clicked to help with developer usage.
- When Compiled as Dot Net test suite Leak Detection is excluded.
Back to Top
Download Links.
DUnit code as described above is available for download from Sourceforge
Full DUnit code is available at SourceForge.org
FastMM now provides Delphi with an advanced memory manager that
coupled with the new DUnit code provides automated memory leak
detection for every test case. BDS2006 already incorporates FastMM
memory manager so Borland/CodeGear have recognized and endorsed it's
validity as a memory manager.
Regardless of whether you make use of the upgraded DUnit code, the
addition of FastMM to your DUnit test suites and applications may
improve the execution time for reasons best spelled out in the FastMM
documentation. For D6 users, you will need to add additional code to
your .dpr as shown in the UnitTests.dpr from the HIDUnit.zip file.
FastMM can be downloaded from sourceforge.net/projects/fastmm
Back to Top
Terms Used.
Within the context of this document a test case is the code which DUnit executes that the user has written deriving from TTestCase.
A specific test case may be referred to as TestCase, particularly in menus.
A test case always entails a test procedure and optionally include an associated SetUp and TearDown procedure.
A test suite is the collection of test cases associated with a specific derivation of TTestcase.
Similarly test suites are a collection of individual test suites.
Run may sometimes be used in place of test procedure that executes between SetUp and TearDown.
The notion of Run-Time Properties has been introduced to DUnit to describe property values which may be applied to test cases at execution time.
Run time properties have no effect on the outcome of a test if the test
encounters an exception or normal test failure. However they are
examined by DUnit after the test has executed to optionally raise a Post Execution Failure.
Run time properties may be turned on by GUI options, in which case they
are applied by DUnit immediately before SetUp is called. They may also
be applied by the user anywhere within Setup or Run. However users
cannot expect run time properties to behave as required if they are
altered by user code within TearDown.
Soft Failure is sometimes used to describe the new class of failures selectively enabled by run time properties.
Back to Top
Page last updated 13-Feb-2007
