Test::More(3p) Perl Programmers Reference Guide Test::More(3p)NAMETest::More - yet another framework for writing test scripts
SYNOPSIS
use Test::More tests => $Num_Tests;
# or
use Test::Moreqw(no_plan);
# or
use Test::More skip_all => $reason;
BEGIN { use_ok( 'Some::Module' ); }
require_ok( 'Some::Module' );
# Various ways to say "ok"
ok($this eq $that, $test_name);
is ($this, $that, $test_name);
isnt($this, $that, $test_name);
# Rather than print STDERR "# here's what went wrong\n"
diag("here's what went wrong");
like ($this, qr/that/, $test_name);
unlike($this, qr/that/, $test_name);
cmp_ok($this, '==', $that, $test_name);
is_deeply($complex_structure1, $complex_structure2, $test_name);
SKIP: {
skip $why, $how_many unless $have_some_feature;
ok( foo(), $test_name );
is( foo(42), 23, $test_name );
};
TODO: {
local $TODO = $why;
ok( foo(), $test_name );
is( foo(42), 23, $test_name );
};
can_ok($module, @methods);
isa_ok($object, $class);
pass($test_name);
fail($test_name);
BAIL_OUT($why);
perl v5.8.8 2005-02-05 1
Test::More(3p) Perl Programmers Reference Guide Test::More(3p)
# UNIMPLEMENTED!!!
my @status = Test::More::status;
DESCRIPTION
STOP! If you're just getting started writing tests, have a
look at Test::Simple first. This is a drop in replacement
for Test::Simple which you can switch to once you get the
hang of basic testing.
The purpose of this module is to provide a wide range of
testing utilities. Various ways to say "ok" with better
diagnostics, facilities to skip tests, test future features
and compare complicated data structures. While you can do
almost anything with a simple "ok()" function, it doesn't
provide good diagnostic output.
I love it when a plan comes together
Before anything else, you need a testing plan. This basi-
cally declares how many tests your script is going to run to
protect against premature failure.
The preferred way to do this is to declare a plan when you
"use Test::More".
use Test::More tests => $Num_Tests;
There are rare cases when you will not know beforehand how
many tests your script is going to run. In this case, you
can declare that you have no plan. (Try to avoid using this
as it weakens your test.)
use Test::Moreqw(no_plan);
NOTE: using no_plan requires a Test::Harness upgrade else it
will think everything has failed. See "CAVEATS and NOTES").
In some cases, you'll want to completely skip an entire
testing script.
use Test::More skip_all => $skip_reason;
Your script will declare a skip with the reason why you
skipped and exit immediately with a zero (success). See
Test::Harness for details.
If you want to control what functions Test::More will
export, you have to use the 'import' option. For example,
to import everything but 'fail', you'd do:
use Test::More tests => 23, import => ['!fail'];
perl v5.8.8 2005-02-05 2
Test::More(3p) Perl Programmers Reference Guide Test::More(3p)
Alternatively, you can use the plan() function. Useful for
when you have to calculate the number of tests.
use Test::More;
plan tests => keys %Stuff * 3;
or for deciding between running the tests at all:
use Test::More;
if( $^O eq 'MacOS' ) {
plan skip_all => 'Test irrelevant on MacOS';
}
else {
plan tests => 42;
}
Test names
By convention, each test is assigned a number in order.
This is largely done automatically for you. However, it's
often very useful to assign a name to each test. Which
would you rather see:
ok 4
not ok 5
ok 6
or
ok 4 - basic multi-variable
not ok 5 - simple exponential
ok 6 - force == mass * acceleration
The later gives you some idea of what failed. It also makes
it easier to find the test in your script, simply search for
"simple exponential".
All test functions take a name argument. It's optional, but
highly suggested that you use it.
I'm ok, you're not ok.
The basic purpose of this module is to print out either "ok
#" or "not ok #" depending on if a given test succeeded or
failed. Everything else is just gravy.
All of the following print "ok" or "not ok" depending on if
the test succeeded or failed. They all also return true or
false, respectively.
ok
ok($this eq $that, $test_name);
perl v5.8.8 2005-02-05 3
Test::More(3p) Perl Programmers Reference Guide Test::More(3p)
This simply evaluates any expression ("$this eq $that"
is just a simple example) and uses that to determine if
the test succeeded or failed. A true expression passes,
a false one fails. Very simple.
For example:
ok( $exp{9} == 81, 'simple exponential' );
ok( Film->can('db_Main'), 'set_db()' );
ok( $p->tests == 4, 'saw tests' );
ok( !grep !defined $_, @items, 'items populated' );
(Mnemonic: "This is ok.")
$test_name is a very short description of the test that
will be printed out. It makes it very easy to find a
test in your script when it fails and gives others an
idea of your intentions. $test_name is optional, but we
very strongly encourage its use.
Should an ok() fail, it will produce some diagnostics:
not ok 18 - sufficient mucus
# Failed test 'sufficient mucus'
# in foo.t at line 42.
This is actually Test::Simple's ok() routine.
is
isnt
is ( $this, $that, $test_name );
isnt( $this, $that, $test_name );
Similar to ok(), is() and isnt() compare their two argu-
ments with "eq" and "ne" respectively and use the result
of that to determine if the test succeeded or failed.
So these:
# Is the ultimate answer 42?
is( ultimate_answer(), 42, "Meaning of Life" );
# $foo isn't empty
isnt( $foo, '', "Got some foo" );
are similar to these:
ok( ultimate_answer() eq 42, "Meaning of Life" );
ok( $foo ne '', "Got some foo" );
(Mnemonic: "This is that." "This isn't that.")
So why use these? They produce better diagnostics on
perl v5.8.8 2005-02-05 4
Test::More(3p) Perl Programmers Reference Guide Test::More(3p)
failure. ok() cannot know what you are testing for
(beyond the name), but is() and isnt() know what the
test was and why it failed. For example this test:
my $foo = 'waffle'; my $bar = 'yarblokos';
is( $foo, $bar, 'Is foo the same as bar?' );
Will produce something like this:
not ok 17 - Is foo the same as bar?
# Failed test 'Is foo the same as bar?'
# in foo.t at line 139.
# got: 'waffle'
# expected: 'yarblokos'
So you can figure out what went wrong without rerunning
the test.
You are encouraged to use is() and isnt() over ok()
where possible, however do not be tempted to use them to
find out if something is true or false!
# XXX BAD!
is( exists $brooklyn{tree}, 1, 'A tree grows in Brooklyn' );
This does not check if "exists $brooklyn{tree}" is true,
it checks if it returns 1. Very different. Similar
caveats exist for false and 0. In these cases, use ok().
ok( exists $brooklyn{tree}, 'A tree grows in Brooklyn' );
For those grammatical pedants out there, there's an
"isn't()" function which is an alias of isnt().
like
like( $this, qr/that/, $test_name );
Similar to ok(), like() matches $this against the regex
"qr/that/".
So this:
like($this, qr/that/, 'this is like that');
is similar to:
ok( $this =~ /that/, 'this is like that');
(Mnemonic "This is like that".)
The second argument is a regular expression. It may be
given as a regex reference (i.e. "qr//") or (for better
perl v5.8.8 2005-02-05 5
Test::More(3p) Perl Programmers Reference Guide Test::More(3p)
compatibility with older perls) as a string that looks
like a regex (alternative delimiters are currently not
supported):
like( $this, '/that/', 'this is like that' );
Regex options may be placed on the end ('/that/i').
Its advantages over ok() are similar to that of is() and
isnt(). Better diagnostics on failure.
unlike
unlike( $this, qr/that/, $test_name );
Works exactly as like(), only it checks if $this does
not match the given pattern.
cmp_ok
cmp_ok( $this, $op, $that, $test_name );
Halfway between ok() and is() lies cmp_ok(). This
allows you to compare two arguments using any binary
perl operator.
# ok( $this eq $that );
cmp_ok( $this, 'eq', $that, 'this eq that' );
# ok( $this == $that );
cmp_ok( $this, '==', $that, 'this == that' );
# ok( $this && $that );
cmp_ok( $this, '&&', $that, 'this && that' );
...etc...
Its advantage over ok() is when the test fails you'll
know what $this and $that were:
not ok 1
# Failed test in foo.t at line 12.
# '23'
# &&
# undef
It's also useful in those cases where you are comparing
numbers and is()'s use of "eq" will interfere:
cmp_ok( $big_hairy_number, '==', $another_big_hairy_number );
can_ok
can_ok($module, @methods);
can_ok($object, @methods);
perl v5.8.8 2005-02-05 6
Test::More(3p) Perl Programmers Reference Guide Test::More(3p)
Checks to make sure the $module or $object can do these
@methods (works with functions, too).
can_ok('Foo', qw(this that whatever));
is almost exactly like saying:
ok( Foo->can('this') &&
Foo->can('that') &&
Foo->can('whatever')
);
only without all the typing and with a better interface.
Handy for quickly testing an interface.
No matter how many @methods you check, a single can_ok()
call counts as one test. If you desire otherwise, use:
foreach my $meth (@methods) {
can_ok('Foo', $meth);
}
isa_ok
isa_ok($object, $class, $object_name);
isa_ok($ref, $type, $ref_name);
Checks to see if the given "$object->isa($class)". Also
checks to make sure the object was defined in the first
place. Handy for this sort of thing:
my $obj = Some::Module->new;
isa_ok( $obj, 'Some::Module' );
where you'd otherwise have to write
my $obj = Some::Module->new;
ok( defined $obj && $obj->isa('Some::Module') );
to safeguard against your test script blowing up.
It works on references, too:
isa_ok( $array_ref, 'ARRAY' );
The diagnostics of this test normally just refer to 'the
object'. If you'd like them to be more specific, you
can supply an $object_name (for example 'Test custo-
mer').
pass
fail
perl v5.8.8 2005-02-05 7
Test::More(3p) Perl Programmers Reference Guide Test::More(3p)
pass($test_name);
fail($test_name);
Sometimes you just want to say that the tests have
passed. Usually the case is you've got some complicated
condition that is difficult to wedge into an ok(). In
this case, you can simply use pass() (to declare the
test ok) or fail (for not ok). They are synonyms for
ok(1) and ok(0).
Use these very, very, very sparingly.
Module tests
You usually want to test if the module you're testing loads
ok, rather than just vomiting if its load fails. For such
purposes we have "use_ok" and "require_ok".
use_ok
BEGIN { use_ok($module); }
BEGIN { use_ok($module, @imports); }
These simply use the given $module and test to make sure
the load happened ok. It's recommended that you run
use_ok() inside a BEGIN block so its functions are
exported at compile-time and prototypes are properly
honored.
If @imports are given, they are passed through to the
use. So this:
BEGIN { use_ok('Some::Module', qw(foo bar)) }
is like doing this:
use Some::Module qw(foo bar);
Version numbers can be checked like so:
# Just like "use Some::Module 1.02"
BEGIN { use_ok('Some::Module', 1.02) }
Don't try to do this:
BEGIN {
use_ok('Some::Module');
...some code that depends on the use...
...happening at compile time...
}
because the notion of "compile-time" is relative.
perl v5.8.8 2005-02-05 8
Test::More(3p) Perl Programmers Reference Guide Test::More(3p)
Instead, you want:
BEGIN { use_ok('Some::Module') }
BEGIN { ...some code that depends on the use... }
require_ok
require_ok($module);
require_ok($file);
Like use_ok(), except it requires the $module or $file.
Complex data structures
Not everything is a simple eq check or regex. There are
times you need to see if two data structures are equivalent.
For these instances Test::More provides a handful of useful
functions.
NOTE I'm not quite sure what will happen with filehandles.
is_deeply
is_deeply( $this, $that, $test_name );
Similar to is(), except that if $this and $that are
references, it does a deep comparison walking each data
structure to see if they are equivalent. If the two
structures are different, it will display the place
where they start differing.
is_deeply() compares the dereferenced values of refer-
ences, the references themselves (except for their type)
are ignored. This means aspects such as blessing and
ties are not considered "different".
is_deeply() current has very limited handling of func-
tion reference and globs. It merely checks if they have
the same referent. This may improve in the future.
Test::Differences and Test::Deep provide more in-depth
functionality along these lines.
Diagnostics
If you pick the right test function, you'll usually get a
good idea of what went wrong when it failed. But sometimes
it doesn't work out that way. So here we have ways for you
to write your own diagnostic messages which are safer than
just "print STDERR".
diag
diag(@diagnostic_message);
perl v5.8.8 2005-02-05 9
Test::More(3p) Perl Programmers Reference Guide Test::More(3p)
Prints a diagnostic message which is guaranteed not to
interfere with test output. Like "print"
@diagnostic_message is simply concatenated together.
Handy for this sort of thing:
ok( grep(/foo/, @users), "There's a foo user" ) or
diag("Since there's no foo, check that /etc/bar is set up right");
which would produce:
not ok 42 - There's a foo user
# Failed test 'There's a foo user'
# in foo.t at line 52.
# Since there's no foo, check that /etc/bar is set up right.
You might remember "ok() or diag()" with the mnemonic
"open() or die()".
NOTE The exact formatting of the diagnostic output is
still changing, but it is guaranteed that whatever you
throw at it it won't interfere with the test.
Conditional tests
Sometimes running a test under certain conditions will cause
the test script to die. A certain function or method isn't
implemented (such as fork() on MacOS), some resource isn't
available (like a net connection) or a module isn't avail-
able. In these cases it's necessary to skip tests, or
declare that they are supposed to fail but will work in the
future (a todo test).
For more details on the mechanics of skip and todo tests see
Test::Harness.
The way Test::More handles this is with a named block.
Basically, a block of tests which can be skipped over or
made todo. It's best if I just show you...
SKIP: BLOCK
SKIP: {
skip $why, $how_many if $condition;
...normal testing code goes here...
}
This declares a block of tests that might be skipped,
$how_many tests there are, $why and under what $condi-
tion to skip them. An example is the easiest way to
illustrate:
perl v5.8.8 2005-02-05 10
Test::More(3p) Perl Programmers Reference Guide Test::More(3p)
SKIP: {
eval { require HTML::Lint };
skip "HTML::Lint not installed", 2 if $@;
my $lint = new HTML::Lint;
isa_ok( $lint, "HTML::Lint" );
$lint->parse( $html );
is( $lint->errors, 0, "No errors found in HTML" );
}
If the user does not have HTML::Lint installed, the
whole block of code won't be run at all. Test::More
will output special ok's which Test::Harness interprets
as skipped, but passing, tests.
It's important that $how_many accurately reflects the
number of tests in the SKIP block so the # of tests run
will match up with your plan. If your plan is "no_plan"
$how_many is optional and will default to 1.
It's perfectly safe to nest SKIP blocks. Each SKIP
block must have the label "SKIP", or Test::More can't
work its magic.
You don't skip tests which are failing because there's a
bug in your program, or for which you don't yet have
code written. For that you use TODO. Read on.
TODO: BLOCK
TODO: {
local $TODO = $why if $condition;
...normal testing code goes here...
}
Declares a block of tests you expect to fail and $why.
Perhaps it's because you haven't fixed a bug or haven't
finished a new feature:
TODO: {
local $TODO = "URI::Geller not finished";
my $card = "Eight of clubs";
is( URI::Geller->your_card, $card, 'Is THIS your card?' );
my $spoon;
URI::Geller->bend_spoon;
is( $spoon, 'bent', "Spoon bending, that's original" );
}
perl v5.8.8 2005-02-05 11
Test::More(3p) Perl Programmers Reference Guide Test::More(3p)
With a todo block, the tests inside are expected to
fail. Test::More will run the tests normally, but print
out special flags indicating they are "todo".
Test::Harness will interpret failures as being ok.
Should anything succeed, it will report it as an unex-
pected success. You then know the thing you had todo is
done and can remove the TODO flag.
The nice part about todo tests, as opposed to simply
commenting out a block of tests, is it's like having a
programmatic todo list. You know how much work is left
to be done, you're aware of what bugs there are, and
you'll know immediately when they're fixed.
Once a todo test starts succeeding, simply move it out-
side the block. When the block is empty, delete it.
NOTE: TODO tests require a Test::Harness upgrade else it
will treat it as a normal failure. See "CAVEATS and
NOTES").
todo_skip
TODO: {
todo_skip $why, $how_many if $condition;
...normal testing code...
}
With todo tests, it's best to have the tests actually
run. That way you'll know when they start passing.
Sometimes this isn't possible. Often a failing test will
cause the whole program to die or hang, even inside an
"eval BLOCK" with and using "alarm". In these extreme
cases you have no choice but to skip over the broken
tests entirely.
The syntax and behavior is similar to a "SKIP: BLOCK"
except the tests will be marked as failing but todo.
Test::Harness will interpret them as passing.
When do I use SKIP vs. TODO?
If it's something the user might not be able to do, use
SKIP. This includes optional modules that aren't
installed, running under an OS that doesn't have some
feature (like fork() or symlinks), or maybe you need an
Internet connection and one isn't available.
If it's something the programmer hasn't done yet, use
TODO. This is for any code you haven't written yet, or
bugs you have yet to fix, but want to put tests in your
testing script (always a good idea).
perl v5.8.8 2005-02-05 12
Test::More(3p) Perl Programmers Reference Guide Test::More(3p)
Test control
BAIL_OUT
BAIL_OUT($reason);
Incidates to the harness that things are going so badly
all testing should terminate. This includes the running
any additional test scripts.
This is typically used when testing cannot continue such
as a critical module failing to compile or a necessary
external utility not being available such as a database
connection failing.
The test will exit with 255.
Discouraged comparison functions
The use of the following functions is discouraged as they
are not actually testing functions and produce no diagnos-
tics to help figure out what went wrong. They were written
before is_deeply() existed because I couldn't figure out how
to display a useful diff of two arbitrary data structures.
These functions are usually used inside an ok().
ok( eq_array(\@this, \@that) );
"is_deeply()" can do that better and with diagnostics.
is_deeply( \@this, \@that );
They may be deprecated in future versions.
eq_array
my $is_eq = eq_array(\@this, \@that);
Checks if two arrays are equivalent. This is a deep
check, so multi-level structures are handled correctly.
eq_hash
my $is_eq = eq_hash(\%this, \%that);
Determines if the two hashes contain the same keys and
values. This is a deep check.
eq_set
my $is_eq = eq_set(\@this, \@that);
Similar to eq_array(), except the order of the elements
is not important. This is a deep check, but the
irrelevancy of order only applies to the top level.
perl v5.8.8 2005-02-05 13
Test::More(3p) Perl Programmers Reference Guide Test::More(3p)
ok( eq_set(\@this, \@that) );
Is better written:
is_deeply( [sort @this], [sort @that] );
NOTE By historical accident, this is not a true set com-
parison. While the order of elements does not matter,
duplicate elements do.
NOTE eq_set() does not know how to deal with references
at the top level. The following is an example of a com-
parison which might not work:
eq_set([\1, \2], [\2, \1]);
Test::Deep contains much better set comparison func-
tions.
Extending and Embedding Test::More
Sometimes the Test::More interface isn't quite enough. For-
tunately, Test::More is built on top of Test::Builder which
provides a single, unified backend for any test library to
use. This means two test libraries which both use
Test::Builder can be used together in the same program.
If you simply want to do a little tweaking of how the tests
behave, you can access the underlying Test::Builder object
like so:
builder
my $test_builder = Test::More->builder;
Returns the Test::Builder object underlying Test::More
for you to play with.
EXIT CODES
If all your tests passed, Test::Builder will exit with zero
(which is normal). If anything failed it will exit with how
many failed. If you run less (or more) tests than you
planned, the missing (or extras) will be considered
failures. If no tests were ever run Test::Builder will
throw a warning and exit with 255. If the test died, even
after having successfully completed all its tests, it will
still be considered a failure and will exit with 255.
So the exit codes are...
0 all tests successful
255 test died or all passed but wrong # of tests run
any other number how many failed (including missing or extras)
perl v5.8.8 2005-02-05 14
Test::More(3p) Perl Programmers Reference Guide Test::More(3p)
If you fail more than 254 tests, it will be reported as 254.
NOTE This behavior may go away in future versions.
CAVEATS and NOTES
Backwards compatibility
Test::More works with Perls as old as 5.004_05.
Overloaded objects
String overloaded objects are compared as strings (or in
cmp_ok()'s case, strings or numbers as appropriate to
the comparison op). This prevents Test::More from
piercing an object's interface allowing better blackbox
testing. So if a function starts returning overloaded
objects instead of bare strings your tests won't notice
the difference. This is good.
However, it does mean that functions like is_deeply()
cannot be used to test the internals of string over-
loaded objects. In this case I would suggest Test::Deep
which contains more flexible testing functions for com-
plex data structures.
Threads
Test::More will only be aware of threads if "use
threads" has been done before Test::More is loaded.
This is ok:
use threads;
use Test::More;
This may cause problems:
use Test::More
use threads;
Test::Harness upgrade
no_plan and todo depend on new Test::Harness features
and fixes. If you're going to distribute tests that use
no_plan or todo your end-users will have to upgrade
Test::Harness to the latest one on CPAN. If you avoid
no_plan and TODO tests, the stock Test::Harness will
work fine.
Installing Test::More should also upgrade Test::Harness.
HISTORY
This is a case of convergent evolution with Joshua
Pritikin's Test module. I was largely unaware of its
existence when I'd first written my own ok() routines. This
module exists because I can't figure out how to easily wedge
test names into Test's interface (along with a few other
perl v5.8.8 2005-02-05 15
Test::More(3p) Perl Programmers Reference Guide Test::More(3p)
problems).
The goal here is to have a testing utility that's simple to
learn, quick to use and difficult to trip yourself up with
while still providing more flexibility than the existing
Test.pm. As such, the names of the most common routines are
kept tiny, special cases and magic side-effects are kept to
a minimum. WYSIWYG.
SEE ALSO
Test::Simple if all this confuses you and you just want to
write some tests. You can upgrade to Test::More later (it's
forward compatible).
Test is the old testing module. Its main benefit is that it
has been distributed with Perl since 5.004_05.
Test::Harness for details on how your test results are
interpreted by Perl.
Test::Differences for more ways to test complex data struc-
tures. And it plays well with Test::More.
Test::Class is like XUnit but more perlish.
Test::Deep gives you more powerful complex data structure
testing.
Test::Unit is XUnit style testing.
Test::Inline shows the idea of embedded testing.
Bundle::Test installs a whole bunch of useful test modules.
AUTHORS
Michael G Schwern <schwern@pobox.com> with much inspiration
from Joshua Pritikin's Test module and lots of help from
Barrie Slaymaker, Tony Bowden, blackstar.co.uk, chromatic,
Fergal Daly and the perl-qa gang.
BUGS
See http://rt.cpan.org to report and view bugs.
COPYRIGHT
Copyright 2001, 2002, 2004 by Michael G Schwern
<schwern@pobox.com>.
This program is free software; you can redistribute it
and/or modify it under the same terms as Perl itself.
See http://www.perl.com/perl/misc/Artistic.html
perl v5.8.8 2005-02-05 16
