|
|
|
@ -2,10 +2,10 @@ |
|
|
|
<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN" |
|
|
|
"http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" [ |
|
|
|
<!ENTITY legal SYSTEM "legal.xml"> |
|
|
|
<!ENTITY appversion "1.5.1"> |
|
|
|
<!ENTITY version "1.5.1"> |
|
|
|
<!ENTITY appversion "1.5.4"> |
|
|
|
<!ENTITY version "1.5.4"> |
|
|
|
<!ENTITY manrevision "1.0"> |
|
|
|
<!ENTITY date "Feb 2004"> |
|
|
|
<!ENTITY date "November 2015"> |
|
|
|
<!ENTITY app "<application>DejaGnu</application>"> |
|
|
|
<!ENTITY appname "DejaGnu"> |
|
|
|
<!ENTITY dj "DejaGnu"> |
|
|
|
@ -42,82 +42,41 @@ |
|
|
|
<!-- </legalnotice> --> |
|
|
|
<revhistory> |
|
|
|
<revision> |
|
|
|
<revnumber>0.6.2</revnumber> |
|
|
|
<date>2002-07-16</date> |
|
|
|
<authorinitials>rob</authorinitials> |
|
|
|
<revremark>Add new tutorial as a new sect1.</revremark> |
|
|
|
</revision> |
|
|
|
<revision> |
|
|
|
<revnumber>0.6.1</revnumber> |
|
|
|
<date>2001-02-16</date> |
|
|
|
<authorinitials>rob</authorinitials> |
|
|
|
<revremark>Add info on the new dejagnu.h file.</revremark> |
|
|
|
</revision> |
|
|
|
<revision> |
|
|
|
<revnumber>0.6</revnumber> |
|
|
|
<date>2001-02-16</date> |
|
|
|
<authorinitials>rob</authorinitials> |
|
|
|
<revremark>Updated for new release.</revremark> |
|
|
|
</revision> |
|
|
|
<revision> |
|
|
|
<revnumber>0.5</revnumber> |
|
|
|
<date>2000-01-24</date> |
|
|
|
<authorinitials>rob</authorinitials> |
|
|
|
<revremark>Initial version after conversion to DocBook.</revremark> |
|
|
|
<revnumber>1.5.3</revnumber> |
|
|
|
<date>2015-11-12</date> |
|
|
|
<authorinitials>bje</authorinitials> |
|
|
|
<revremark>Overhaul the manual.</revremark> |
|
|
|
</revision> |
|
|
|
</revhistory> |
|
|
|
|
|
|
|
</articleinfo> |
|
|
|
|
|
|
|
<sect1 id="preface"> |
|
|
|
<title>Abstract</title> |
|
|
|
|
|
|
|
<para>This document describes the functionality of &dj;, the |
|
|
|
testing framework of the GNU project. &dj; is written in |
|
|
|
<productname>Expect</productname>, which uses |
|
|
|
<productname>Tcl</productname> as a command |
|
|
|
language. <productname>Expect</productname> acts as a very |
|
|
|
programmable shell. As with other Unix command shells, you can |
|
|
|
run any program, but once the program is started, your test script |
|
|
|
has programmable control over its input and output. This does not |
|
|
|
just apply to the programs under test; <command>expect</command> |
|
|
|
can also run any auxiliary program, such as |
|
|
|
<command>diff</command> or <command>sh</command>, with full |
|
|
|
control over its input and output.</para> |
|
|
|
|
|
|
|
<para>&dj; itself is merely a framework for the creation of |
|
|
|
testsuites. Testsuites are distributed with each |
|
|
|
application.</para> |
|
|
|
|
|
|
|
</sect1> |
|
|
|
|
|
|
|
<sect1 id="overview" xreflabel="Overview"> |
|
|
|
<title>Overview</title> |
|
|
|
<sect2 id="whatis" xreflabel="What is &dj; ?"> |
|
|
|
<title>What is &dj; ?</title> |
|
|
|
<sect1 id="intro" xreflabel="Introduction"> |
|
|
|
<title>Introduction</title> |
|
|
|
<sect2 id="whatis" xreflabel="What is &dj;?"> |
|
|
|
<title>What is &dj;?</title> |
|
|
|
|
|
|
|
<para><productname>&dj;</productname> is a framework for |
|
|
|
testing other programs. Its purpose is to provide a single |
|
|
|
front end for all tests. Think of it as a custom library of |
|
|
|
Tcl procedures crafted to support writing a test harness. A |
|
|
|
<emphasis>Test Harness</emphasis> is the testing |
|
|
|
<para><productname>&dj;</productname> is a framework for testing |
|
|
|
other programs, providing a single front-end for all |
|
|
|
tests. You can think of it as a custom library of Tcl |
|
|
|
procedures crafted to help with writing a test harness. A |
|
|
|
<emphasis>test harness</emphasis> is the testing |
|
|
|
infrastructure that is created to support a specific program |
|
|
|
or tool. Each program can have multiple testsuites, all |
|
|
|
supported by a single test harness. &dj; is written in |
|
|
|
<productname>Expect</productname>, which in turn uses |
|
|
|
<productname>Tcl</productname> -- Tool command |
|
|
|
language. There is more information on Tcl at the <ulink |
|
|
|
url="http://www.tcl.tk">Tcl/Tk</ulink> web site and the |
|
|
|
Expect web site is at <ulink |
|
|
|
url="http://expect.nist.gov">NIST</ulink>.</para> |
|
|
|
<productname>Tcl</productname>, the Tool command |
|
|
|
language. There is more information on Tcl at |
|
|
|
the <ulink url="http://www.tcl.tk">Tcl/Tk</ulink> web site and |
|
|
|
the Expect web site is |
|
|
|
at <ulink url="http://expect.nist.gov">NIST</ulink>.</para> |
|
|
|
|
|
|
|
<para>Julia Menapace first coined the term ``&dj;'' to describe |
|
|
|
an earlier testing framework at Cygnus Support she had written |
|
|
|
for <command>GDB</command>. When we replaced it with the |
|
|
|
Expect-based framework, it was like &dj; all over again. |
|
|
|
More importantly, it was also named after my daughter, Deja |
|
|
|
Snow Savoye, who was a toddler during &dj;'s |
|
|
|
beginnings.</para> |
|
|
|
<para>Julia Menapace first coined the term ``&dj;'' to describe an |
|
|
|
earlier testing framework she wrote at Cygnus Support for |
|
|
|
testing GDB. When we replaced it with the Expect-based |
|
|
|
framework, it was like &dj; all over again. More importantly, |
|
|
|
it was also named after my daughter, Deja Snow Savoye, who was |
|
|
|
a toddler during &dj;'s beginnings.</para> |
|
|
|
|
|
|
|
<para>&dj; offers several advantages for testing:</para> |
|
|
|
|
|
|
|
@ -167,70 +126,22 @@ |
|
|
|
<sect2 id="new" xreflabel="Release Notes"> |
|
|
|
<title>New In This Release</title> |
|
|
|
|
|
|
|
<para>This release has a number of substantial changes over version |
|
|
|
1.3. The most visible change is that the version of Expect and Tcl |
|
|
|
included in the release are up-to-date with the current stable net |
|
|
|
releases. The biggest change is years of modifications to the |
|
|
|
target configuration system, used for cross testing. While this |
|
|
|
greatly improved cross testing, it has made that subsystem very |
|
|
|
complicated. The goal is to have this entirely rewritten using |
|
|
|
<productname>iTcl</productname> by the next release. Other changes |
|
|
|
are:</para> |
|
|
|
|
|
|
|
<itemizedlist> |
|
|
|
<listitem> |
|
|
|
<para>More built-in support for building target binaries |
|
|
|
with the correct linker flags. Currently this only works with |
|
|
|
<productname>GCC</productname> as the cross compiler, |
|
|
|
preferably with a target supported by |
|
|
|
<xref linkend="libgloss"/>. |
|
|
|
</para> |
|
|
|
<para>A completely new manual.</para> |
|
|
|
</listitem> |
|
|
|
|
|
|
|
<listitem><para>Lots of little bug fixes from years of heavy |
|
|
|
use at Cygnus Solutions.</para></listitem> |
|
|
|
|
|
|
|
<listitem><para>&dj; now uses |
|
|
|
<productname>Automake</productname> for Makefile |
|
|
|
configuration.</para></listitem> |
|
|
|
|
|
|
|
<listitem><para>Updated documentation, now in DocBook XML. |
|
|
|
</para></listitem> |
|
|
|
|
|
|
|
<listitem><para>Windows support. There is beta level support for |
|
|
|
Windows that is still a work in progress. This requires the |
|
|
|
<ulink url="http://www.cygwin.com/">Cygwin</ulink> POSIX |
|
|
|
subsystem for Windows.</para></listitem> |
|
|
|
|
|
|
|
</itemizedlist> |
|
|
|
|
|
|
|
<sect3 id="cygwin" xreflabel="Windows Support"> |
|
|
|
<title>Windows Support</title> |
|
|
|
|
|
|
|
<para>To use &dj; on Windows, you need to first install the |
|
|
|
<ulink url="http://www.cygwin.com/">Cygwin</ulink> |
|
|
|
release. This works as of the B20.1 release. Cygwin is a POSIX |
|
|
|
system for Windows. This covers both utility programs and a library |
|
|
|
that adds POSIX system calls to Windows. Among them is pseudo tty |
|
|
|
support for Windows that emulates the POSIX pty standard. The |
|
|
|
latest Cygwin is always available from <ulink |
|
|
|
url="http://www.cygwin.com/">this location</ulink>. This |
|
|
|
works well enough to run <emphasis>"make check"</emphasis> of |
|
|
|
the GNU development tree on Windows after a native build. But the |
|
|
|
nature of ptys on Windows is still evolving. Your mileage may |
|
|
|
vary.</para> |
|
|
|
|
|
|
|
</sect3> |
|
|
|
|
|
|
|
</sect2> |
|
|
|
|
|
|
|
<sect2 id="designgoals" xreflabel="Design Goals"> |
|
|
|
<title>Design Goals</title> |
|
|
|
|
|
|
|
<para>&dj; grew out of the internal needs of Cygnus Solutions, |
|
|
|
the company formerly known as Cygnus Support. Cygnus maintained |
|
|
|
and enhanced a variety of free programs in many different |
|
|
|
environments and we needed a testing tool that:</para> |
|
|
|
<para>&dj; grew out of the internal needs of Cygnus Solutions |
|
|
|
(formerly Cygnus Support). Cygnus maintained and enhanced a |
|
|
|
variety of free programs in many different environments and needed |
|
|
|
a testing tool that:</para> |
|
|
|
|
|
|
|
<itemizedlist mark="bullet"> |
|
|
|
<listitem><para>was useful to developers while fixing |
|
|
|
@ -239,17 +150,17 @@ |
|
|
|
release process;</para></listitem> |
|
|
|
<listitem><para>was portable among a variety of host |
|
|
|
computers;</para></listitem> |
|
|
|
<listitem><para>supported cross-development |
|
|
|
testing;</para></listitem> |
|
|
|
<listitem><para>permitted testing interactive programs, like |
|
|
|
<command>GDB</command>; and </para></listitem> |
|
|
|
<listitem><para>permitted testing batch oriented programs, like |
|
|
|
<command>GCC</command>.</para></listitem> |
|
|
|
<listitem><para>supported a cross-development environment; |
|
|
|
</para></listitem> |
|
|
|
<listitem><para>permitted testing of interactive programs |
|
|
|
like <command>GDB</command>; and </para></listitem> |
|
|
|
<listitem><para>permitted testing of batch-oriented programs |
|
|
|
like <command>GCC</command>.</para></listitem> |
|
|
|
</itemizedlist> |
|
|
|
|
|
|
|
<para>Some of the requirements proved challenging. For example, |
|
|
|
interactive programs do not lend themselves very well to automated testing. |
|
|
|
But all the requirements are important: for instance, it is imperative to |
|
|
|
But all the requirements are important. For instance, it is imperative to |
|
|
|
make sure that <command>GDB</command> works as well when cross-debugging |
|
|
|
as it does in a native configuration. </para> |
|
|
|
|
|
|
|
@ -260,28 +171,29 @@ |
|
|
|
communication interfaces vary from a serial line to Ethernet. |
|
|
|
&dj; was designed with a modular communication setup, so that |
|
|
|
each kind of communication can be added as required and supported |
|
|
|
thereafter. Once a communication procedure is coded, any test can |
|
|
|
thereafter. Once a communication procedure is written, any test can |
|
|
|
use it. Currently &dj; can use <command>rsh</command>, |
|
|
|
<command>rlogin</command>, <command>telnet</command>, |
|
|
|
<command>tip</command>, <command>kermit</command> and |
|
|
|
<command>mondfe</command> for remote communications.</para> |
|
|
|
<command>tip</command>, and <command>kermit</command> for remote |
|
|
|
communications.</para> |
|
|
|
|
|
|
|
</sect2> |
|
|
|
|
|
|
|
<sect2 id="posix" xreflabel="A POSIX Conforming Test Framework"> |
|
|
|
<title>A POSIX conforming test framework</title> |
|
|
|
<title>A POSIX compliant test framework</title> |
|
|
|
|
|
|
|
<para>&dj; conforms to the POSIX 1003.3 standard for test |
|
|
|
frameworks. Rob Savoye was a member of that committee.</para> |
|
|
|
|
|
|
|
<para>The POSIX standard 1003.3 defines what a testing framework needs to |
|
|
|
provide, in order to permit the creation of POSIX conformance test |
|
|
|
suites. This standard is primarily oriented to running POSIX conformance |
|
|
|
tests, but its requirements also support testing of features not related |
|
|
|
to POSIX conformance. POSIX 1003.3 does not specify a particular testing |
|
|
|
framework, but at this time there is only one other POSIX conforming test |
|
|
|
framework: TET. TET was created by Unisoft for a consortium comprised of |
|
|
|
X/Open, Unix International and the Open Software Foundation.</para> |
|
|
|
<para>POSIX standard 1003.3 defines what a testing framework |
|
|
|
needs to provide to create a POSIX compliant testsuite. This |
|
|
|
standard is primarily oriented to checking POSIX conformance, |
|
|
|
but its requirements also support testing of features not |
|
|
|
related to POSIX conformance. POSIX 1003.3 does not specify a |
|
|
|
particular testing framework, but at this time there is only one |
|
|
|
other POSIX conforming test framework. TET was created by |
|
|
|
Unisoft for a consortium comprised of X/Open, Unix International |
|
|
|
and the Open Software Foundation.</para> |
|
|
|
|
|
|
|
<para>The POSIX documentation refers to <firstterm>assertions</firstterm>. |
|
|
|
An assertion is a description of behavior. For example, if a standard |
|
|
|
@ -291,16 +203,16 @@ |
|
|
|
that the standard being tested is never 1003.3; the standard being tested |
|
|
|
is some other standard, for which the assertions were written.</para> |
|
|
|
|
|
|
|
<para>As there is no testsuite to test testing frameworks for POSIX |
|
|
|
1003.3 conformance, verifying conformance to this standard is done by |
|
|
|
repeatedly reading the standard and experimenting. One of the main |
|
|
|
things 1003.3 does specify is the set of allowed output messages and |
|
|
|
their definitions. Four messages are supported for a required feature of |
|
|
|
POSIX conforming systems and a fifth for a conditional feature. &dj; |
|
|
|
supports the use of all five output messages. In this sense a testsuite |
|
|
|
that uses exactly these messages can be considered POSIX conforming. |
|
|
|
These definitions specify the output of a test |
|
|
|
case:</para> |
|
|
|
<para>As there is no testsuite to verify that testing frameworks |
|
|
|
are POSIX 1003.3 compliant, this is done by repeatedly reading |
|
|
|
the standard and experimenting. One of the main things POSIX |
|
|
|
1003.3 does specify is the set of allowed output messages and |
|
|
|
their definitions. Four messages are supported for a required |
|
|
|
feature of POSIX conforming systems and a fifth for a |
|
|
|
conditional feature. &dj; supports all five output messages. In |
|
|
|
this sense a testsuite that uses exactly these messages can be |
|
|
|
considered POSIX compliant. These definitions specify the |
|
|
|
output of a test case:</para> |
|
|
|
|
|
|
|
<variablelist> |
|
|
|
<varlistentry> |
|
|
|
@ -309,6 +221,17 @@ |
|
|
|
the assertion is true.</para></listitem> |
|
|
|
</varlistentry> |
|
|
|
|
|
|
|
<varlistentry> |
|
|
|
<term>FAIL</term> |
|
|
|
<listitem><para>A test has not succeeded -- the assertion is |
|
|
|
false. The <emphasis>FAIL</emphasis> message is based on |
|
|
|
this test case only. Other messages are used to indicate a |
|
|
|
failure of the framework. As with <emphasis>PASS</emphasis>, |
|
|
|
POSIX tests must return |
|
|
|
<emphasis>FAIL</emphasis> rather than <emphasis>XFAIL</emphasis> even |
|
|
|
if a failure was expected.</para></listitem> |
|
|
|
</varlistentry> |
|
|
|
|
|
|
|
<varlistentry> |
|
|
|
<term>XFAIL</term> |
|
|
|
<listitem><para>POSIX 1003.3 does not incorporate the notion of |
|
|
|
@ -319,22 +242,11 @@ |
|
|
|
<emphasis>XPASS</emphasis> is also used.</para></listitem> |
|
|
|
</varlistentry> |
|
|
|
|
|
|
|
<varlistentry> |
|
|
|
<term>FAIL</term> |
|
|
|
<listitem><para>A test has produced the bug it was intended to |
|
|
|
capture. That is, it has demonstrated that the assertion is false. |
|
|
|
The <emphasis>FAIL</emphasis> message is based on the test case only. |
|
|
|
Other messages are used to indicate a failure of the framework. As |
|
|
|
with <emphasis>PASS</emphasis>, POSIX tests must return |
|
|
|
<emphasis>FAIL</emphasis> rather than <emphasis>XFAIL</emphasis> even |
|
|
|
if a failure was expected.</para></listitem> |
|
|
|
</varlistentry> |
|
|
|
|
|
|
|
<varlistentry> |
|
|
|
<term>UNRESOLVED</term> |
|
|
|
<listitem><para>A test produced indeterminate results. Usually, this |
|
|
|
means the test executed in an unexpected fashion; this outcome |
|
|
|
requires that a human being go over results, to determine if the test |
|
|
|
means the test executed in an unexpected fashion. This outcome |
|
|
|
requires a human to go over results to determine if the test |
|
|
|
should have passed or failed. This message is also used for any test |
|
|
|
that requires human intervention because it is beyond the abilities |
|
|
|
of the testing framework. Any unresolved test should resolved to |
|
|
|
@ -357,7 +269,7 @@ |
|
|
|
</variablelist> |
|
|
|
|
|
|
|
<itemizedlist> |
|
|
|
<listitem><para>A test's execution is |
|
|
|
<listitem><para>Execution of a test is |
|
|
|
interrupted.</para></listitem> |
|
|
|
|
|
|
|
<listitem><para>A test does not produce a clear |
|
|
|
@ -367,12 +279,12 @@ |
|
|
|
<emphasis>WARNING</emphasis> messages. Any |
|
|
|
<emphasis>WARNING</emphasis> or <emphasis>ERROR</emphasis> |
|
|
|
messages can invalidate the output of the test. This |
|
|
|
usually requires a human being to examine the output to |
|
|
|
determine what really happened---and to improve the test |
|
|
|
usually requires a human to examine the output to |
|
|
|
determine what really happened -- and to improve the test |
|
|
|
case.</para></listitem> |
|
|
|
|
|
|
|
<listitem><para>A test depends on a previous test, which |
|
|
|
fails.</para></listitem> |
|
|
|
has failed.</para></listitem> |
|
|
|
|
|
|
|
<listitem><para>The test was set up |
|
|
|
incorrectly.</para></listitem> |
|
|
|
@ -381,15 +293,12 @@ |
|
|
|
<variablelist> |
|
|
|
<varlistentry> |
|
|
|
<term>UNTESTED</term> |
|
|
|
<listitem><para>A test was not run. This is a place-holder, used |
|
|
|
when there is no real test case yet.</para></listitem> |
|
|
|
<listitem><para>A test was not run. This is a placeholder |
|
|
|
used when there is no real test case |
|
|
|
yet.</para></listitem> |
|
|
|
</varlistentry> |
|
|
|
</variablelist> |
|
|
|
|
|
|
|
<para>The only remaining output message left is intended to test |
|
|
|
features that are specified by the applicable POSIX standard as |
|
|
|
conditional:</para> |
|
|
|
|
|
|
|
<variablelist> |
|
|
|
<varlistentry> |
|
|
|
<term>UNSUPPORTED</term> |
|
|
|
@ -413,6 +322,25 @@ |
|
|
|
as described in the <emphasis>UNRESOLVED</emphasis> section |
|
|
|
above.</para> |
|
|
|
</sect2> |
|
|
|
|
|
|
|
<sect2 id="installation" xreflabel="Installation"> |
|
|
|
<title>Installation</title> |
|
|
|
|
|
|
|
<para>Refer to the <filename>INSTALL</filename> in the source |
|
|
|
distribution for detailed installation instructions. Note that |
|
|
|
there is no compilation step as with many other GNU packages, as |
|
|
|
&dj; consists of interpreted code only.</para> |
|
|
|
|
|
|
|
<para>Save for its own small testsuite, the &dj; distribution does |
|
|
|
not include any testsuites. Testsuites for the various GNU |
|
|
|
development tools are included with those packages. After |
|
|
|
configuring the top-level &dj; directory, unpack and configure |
|
|
|
the test directories for the tools you want to test; then, in |
|
|
|
each test directory, run <emphasis>make check</emphasis> to |
|
|
|
build auxiliary programs required by some of the tests, and run |
|
|
|
the test suites.</para> |
|
|
|
</sect2> |
|
|
|
|
|
|
|
</sect1> |
|
|
|
|
|
|
|
<!-- include the user manual --> |
|
|
|
|