Run tests used to verify TestSummary.

Run the exception handling tests used to verify TestSummary.

Generate a simulated report containing a hundred tests.

Generate a summary of the report

A test summary has certain properties in common with a test report including whether all tests succeeded.

A test summary contains some properties not available in a test report as well as generating more human-readable output for other properties. Here, determine time and memory resources used in running tests including mean values of time and memory resources.

A report summary has a much smaller memory footprint than a test report due to not storing all of the original tests and their specifications.

Generate a two reports each populated with 100 toy tests.

Combine the reports

Create a summary of the combined report and name it "Combined"

The difference between the respective memory footprints of summaries and reports grows as a test suite scales given that summaries more or less stay at a constant size. Summaries therefore become increasingly advantageous as a system scales.

Compare the application of reports and summaries.

First use TestFile to locate the core unit tests of TestSummary, and then generate both a report and summary of these tests.

Extract some core statistical properties

A summary natively possesses some additional properties.

Now, generate a report and summary of the core unit tests for TestSummaryObject.

Generate a combined summary from the summaries just generated

Compare summaries across selected properties.

Although the combined test summary does not contain the original tests, it nonetheless retains sufficient information for descriptive statistics to be precisely reconstructed.

Confirm that these statistics have been precisely computed by using the data from the original tests stored in corresponding reports. To do this collect up into a list the CPUTimeUsed values for the tests in each report before running the statistics on this list.

Hence even though summaries are an extremely lightweight version of the reports from which they are originate, core information and distributional insights can still be extracted from this compact form--an outcome with significant scaling implications. Naturally, in order to identify tests that have failed or to perform new analyses, a report that does store all tests is required.

Tests are not stored in summaries and hence both reports and summaries together provide a complete testing regime.

When generating the following summary, tests are first run to create an interim test report whose creation involves automatically displaying a progress report:

Disable the automatic display of a progress report by setting the ProgressReporting option-value inherited by an interim TestReportapplication.

Run all tests used to assure the PacletAssurance framework.

Because a test summary can be generated from a variety of sources, it is not always immediately apparent if all the summarized tests were performed during the same invocation. This may be important to establish to ensure that all tests pass under like conditions or without any unexpected interactions. For example, consider a generated summary, summA.

Subsequently another summary, summB, is generated.

Combine both summaries.

All of the tests in this combined summary were however, not conducted in the same invocation

Generate the same summary in a single statement.

Here, all of the tests in the combined summary were conducted in the same invocation.

By including all of the original tests, a TestReport is able to identify any tests that failed.

In contrast, a TestSummaryObject, while being able to efficiently detect failures …

… is unable to identify the tests that failed along with any accompanying details.

It is therefore best practice to use both TestReport and TestSummary in concert; the former to establish a complete record of unit-testing performed and the latter to efficiently measure incremental unit-testing progress as a system evolves. Prior to final deployment, a full TestReport should always be generated.

The property "MemoryUsed" aggregates the memory used in all tests and hence is a measure of memory requirements to the extent tests are run in parallel. Typically however, tests are run sequentially with memory recycled after each test meaning that "MaxMemoryUsed" is likely to be more indicative of memory resources needed.

TestSummary when applied to a file runs TestReport on the file before summarizing the resulting report. Because some of the properties of a report will vary from run to run, slightly different summaries may result despite using the same source file. First create a summary from an existing report.

Now create a summary directly from the source file itself.

Some properties are identical.

Other properties can differ marginally such as, for example, TimeElapsed.

Test reports automatically remove duplicates

Test summaries however, do not retain the original tests so duplicates cannot be similarly removed.

TestSummary operates on the same types of input as TestReport except that it also supports operating on symbols, the names of symbols and or particular test files associated with symbols. In particular, TestSummary also operates on a file but require it to be wrapped in File (to disambiguate when names of symbols are used).