This tutorial explains step by step with a concrete example, how to use Uttk in order to test command line programs. More precisely, it describes how to write a good test file and how to perform common tests (diff standard output/error, program exit status, etc...). After reading this tutorial, you will have a good understanding of what you can do with the Cmd strategy.
This part will teach you how to write a very simple test file to check a command line program. You will see that Uttk is very powerful, especially with that kind of programs.
For this tutorial, we will use a simple program written in Ruby. It takes a word as argument, and write it to the standard output in its plural form. The goal is to show you how to write a test suite for this program in different ways. So, you will learn how to write a Uttk test file to:
We are now ready to begin! The following is the program to test:
$ cat pluralizer.rb
#! /usr/bin/env ruby
def main()
if (ARGV.length != 1)
exit 1
end
puts ARGV[0] + "s"
end
main
Don't forget to add the execution flag to the program file:
$ chmod u+x pluralizer.rb
You can try this program this way:
$ ./pluralizer.rb bird birds
Now, here is the Uttk test file, at the moment it is very simple:
$ cat check.yml
---
Test the pluralizer program: !S::Cmd
command: ./pluralizer.rb
args:
- "bird"
You can see that the test file is in Yaml format, which is the default format. However, there are some tips you should know in order to not to waste your time with stupid stuff due to using that format (for example, the indentation is strict). For further details about how to use Yaml please refer to the Yaml's cookbook.
Ok, now, let's examine the file:
We first give a name to our test directly followed by the strategy used to performed it. Then, we specify the attribute values of the test. In that case, the name of the command to start and the argument to submit to this command.
This test file doesn't check anything at the moment but we are now able to call it with Uttk !
In order to test your program, you only have to execute Uttk with your test file in argument.
$ uttk check.yml
---
root:
contents:
- Test the pluralizer program:
status: PASS
status: PASS
You can notice that the output is very short and looks like more or less the input. Actually, the output is Yaml compliant. If you want a more details about the result of your test, you can look at the log.yml file:
$ cat log.yml
---
root: !S::Suite
contents:
- Test the pluralizer program: !S::Cmd
command: ./pluralizer.rb
args: [bird]
running: ./pluralizer.rb bird
status: PASS
status: PASS
Note, that there are also log.xml and log.html which contain your test result respectively in XML and HTML.
If you want, no log file to be created and only to see the Yaml output in your terminal you can run:
$ uttk -F Yaml check.yml
---
root: !S::Suite
contents:
- Test the pluralizer program: !S::Cmd
command: ./pluralizer.rb
args: [bird]
running: ./pluralizer.rb bird
status: PASS
status: PASS
Finally, we can notice that the output is very similar to the input. So there is a new Strategy called Suite. This one has been automatically created. In fact, you can call Uttk with several input files as argument, and thus all strategy mentioned in those files are bunched together in one strategy suite called root. This point will be detailed later at step three.
For further information about, how to tune the output of Uttk, see:
$ uttk --filter-help
filters:
The --filter (-F) option take a pipeline description which
is a little yaml document.
The syntax is basic:
Observer ::= Class
| <filename>
| Class ":" Observer
| "[" Observers "]"
;
Observers ::= Observer
| Observer "," Observers
;
Class ::= FilterClass | DumperClass ;
FilterClass ::= Buffer
| Compact
| Default
| DefaultColor
| Id
| JustStatus
| KeepSkipBased
| NodeCut
| RPathFilter
| Saver
| TextFilter
;
DumperClass ::= Basic
| BasicColor
| Html
| Mail
| Path
| Xml
| Yaml
;
Examples:
Yaml # Use the Yaml output format on the stdout
Yaml: foo # Use Yaml on file foo
Xml: foo, Yaml: bar # Use Xml on file foo and Yaml on file bar
Xml: [ foo, bar] # Use Xml on files foo and bar
Yaml, Xml: log.xml # Use Yaml on stdout and Xml on log.xml
JustStatus: Yaml # Use the filter JustStatus and display
# its output with Yaml
JustStatus: {Yaml, Xml: log.xml}
# Use Yaml on stdout and Xml on log.xml but trough the
# filter JustStatus.
Yaml: log.yml, JustStatus: Yaml # This one is usefull
You can make your own compositions and save them, see the
documentation of the Filters module.
Warning:
Xml: foo, Xml: bar # Will create only one Xml dumper since
# this is a hash with twice the same key.
and you can use the following command to see the list of available filters:
$ uttk -F
Uttk::Logger classes hierarchy:
/Backend/
/Dumper/
BasicColor
Yaml
Html
Path
Xml
Basic
Mail
/Filter/
Id
Compact
DefaultColor
RPathFilter
JustStatus
Default
KeepSkipBased
NodeCut
TextFilter
Buffer
Saver
/PathFilter/
RemoveTypes
ColorStatus
Every strategy of Uttk embed an online reference documentation about the use of its attribute. To get some help about the Cmd strategy, do the following:
$ uttk -H Cmd Uttk::Strategies::Cmd: name: test name [String] (invisible). strategy: the strategy class [Class] (mandatory, invisible). wclass: a class to control weights [Class]. weight: a sort of coefficient. See wclass. fatal: if the test fail all the suite fail. timeout: a duration (in seconds) before failure. symbols: some user defined symbols [Hash]. pre_assertion: ruby code to assert environment condition which must return true/false [StringProcTrueClassFalseClass] (invisible). post_assertion: ruby code to assert environment condition which must return true/false [StringProcTrueClassFalseClass] (invisible). input: the input reference (invisible). output: the output reference (invisible). error: the error reference (invisible). stream_class: the stream class (< Streams::Stream) [Class]. verbose_print: print even if the test pass (invisible). quiet_print: quiet even if the test fails (invisible). command: the command to execute [StringPathnameArray] (mandatory). dir: the directory where to launch the command [StringPathname]. args: the arguments for the command [ArrayStringNumeric]. exit: the exit status reference [Integer]. env: environment variables [Hash].
To obtain the list of all available Uttk strategies, do the following:
$ uttk --strategy-list
Uttk::Strategies classes hierarchy:
/Strategy/
JUnit
Clean
Sleep
Fail
Pass
Jump
Make
Configure
/Composite/
PackageCollection
/Collection/
Iterate
Pool
Suite
Package
/Proxy/
Compile
RMatch
Import
Test
Threshold
SubCmd
Bootstrap
/IOBased/
Block
Assert
/CmdBase/
Cmd
SignalCmd
Abort
RUnit
Command
Checkout
SqlQuery
Error
KillAll
HostSelector
Authors
Stub
In this step, we will really check our program. We must check the output depending on the input. Moreover, the exit status must be correct too.
This step is very easy if you have correctly followed the previous one. In fact, we only need two new lines in our Yaml test file.
$ cat check.yml
---
Test the pluralizer program: !S::Cmd
command: ./pluralizer.rb
args:
- "bird"
output: "birds\n"
exit: 0
Here, we specify the expected output ("birds\n") for the given argument ("bird"). We also specify the exit status the command must return.
$ uttk check.yml
---
root:
contents:
- Test the pluralizer program:
status: PASS
status: PASS
We can see that our program pass the test. If you aren't convinced, you can try to change the expected output. If we replace it with a wrong value, we see that Uttk shows us an error:
$ cat bad_check.yml
---
Test the pluralizer program: !S::Cmd
command: ./pluralizer.rb
args:
- "bird"
output: "wrong value\n"
exit: 0
$ uttk bad_check.yml
---
root:
contents:
- Test the pluralizer program:
status: FAIL
status: FAIL
--- |
/-----------------------------------------------------------------------------.
| *** Some tests failed *** |
| The last status was not PASS but something like FAIL(42%) (just FAIL stands |
| for FAIL(0%)). This means that 42% of tests *pass* but 100% was expected. |
| To investigate results these different outputs are available: |
| - the standard output is the shortest one (like a progression bar). |
| - log.html contains an HTML/JavaScript page nice to display big outputs. |
| - log.yml contains more information in the very readable YAML format. |
| - log.xml contains the output as an XML document. |
`-----------------------------------------------------------------------------/
$ cat log.yml
---
root: !S::Suite
contents:
- Test the pluralizer program: !S::Cmd
command: ./pluralizer.rb
args: [bird]
exit: 0
running: ./pluralizer.rb bird
output_status: FAILED
my_output: |
birds
ref_output: |
wrong value
my_exit: 0
status: FAIL
reason: output is different
status: FAIL
Ok, that's all for this step !
This step is very important because it deals with the Suite Strategy, which allows you to write more than one test in a single file. We have seen, at the end of the first step, that a Suite strategy had been created automatically. Indeed, you can run Uttk with several input files as argument. See the following example.
$ uttk check.yml check.yml
---
root:
contents:
- Test the pluralizer program:
status: PASS
- Test the pluralizer program:
status: PASS
status: PASS
Now, we are able to check our program with many test files. However, creating a new test file for each test is not very handy ! So, we can create ourselves a Suite strategy which will simplify our work. The following test file shows how to write it in a naive way.
$ cat check.yml
---
Test the pluralizer program: !S::Suite
contents:
- Test the bird word: !S::Cmd
command: ./pluralizer.rb
args:
- "bird"
output: "birds\n"
exit: 0
- Test the ant word: !S::Cmd
command: ./pluralizer.rb
args: "ant"
output: "ants\n"
exit: 0
In this case, we must specify that the main test is a test suite by using the Suite strategy. Thus, we have to specify the contents of the Suite, using its contents attribute, followed by each test we want to run. Be careful ! Remember that Yaml syntax is strict, you must respect the indentation (no tabulations but only white spaces) and the colon after each test name.
As you can see, there are some lines in common in each test that composed the test suite. Indeed, each one is composed of the same Strategy, with the same exit code, and so on. Uttk allows you to factor these attributes using the attribute attributes of the Suite strategy.
$ cat check.yml
---
Test the pluralizer program: !S::Suite
attributes: !S::Cmd
command: ./pluralizer.rb
exit: 0
contents:
- Test the bird word:
args: "bird"
output: "birds\n"
- Test the ant word:
args: "ant"
output: "ants\n"
Here, we precise all common attributes for each test in the contents.
It is very common to create a program which reads input data from the standard input. With Uttk it is very simple to check this kind of program. First of all, we need that our pluralizer.rb program handles the standard input. So, we modify it this way:
$ cat pluralizer.rb
#! /usr/bin/env ruby
def main()
if (ARGV.length == 0)
stream = STDIN.gets.chomp
puts stream + "s"
else
puts ARGV[0] + "s"
end
end
main
Thus, if we don't give any argument to it, it reads on the standard input. So, if we execute our program like that:
$ echo 'bird' | ./pluralizer.rb bird | ./pluralizer.rb
How to check that behavior? It's very simple. Let's see how to write that in your Yaml file.
$ cat check.yml
---
Test the pluralizer program: !S::Suite
attributes: !S::Cmd
command: ./pluralizer.rb
exit: 0
contents:
- Test the bird word:
input: "bird"
output: "birds\n"
- Test the ant word:
args: "ant"
output: "ants\n"
We have introduced the attribute input. The contents of this attribute is sent to the standard input of our program.
When the input contents of your test is big you may want to store it in a separate file instead of writing it directly in your test suite file. The attribute input allows you to do so.
$ cat bird.txt bird
$ cat check.yml
---
Test the pluralizer program: !S::Suite
attributes: !S::Cmd
command: ./pluralizer.rb
exit: 0
contents:
- Test the bird word:
input: !path bird.txt
output: "birds\n"
- Test the ant word:
args: "ant"
output: "ants\n"
In order to precise the given file name, you just need to warn Uttk that you are giving a path name by writing !path before the file name.
In this part, we detail how to check the standard error output. In order to do that we will first falsify our program.
$ cat pluralizer.rb
#! /usr/bin/env ruby
def main()
if (ARGV.length == 0)
word = STDIN.gets.chomp
else
word = ARGV[0]
end
if word =~ /\d/
STDERR.puts "wrong word!"
else
puts word + "s"
end
end
main
Ok, let's sum up the tested program. If we run it without argument, the word to pluralized is picked up from the standard input. Otherwise, if we give at least one argument, the word is the first argument. After, the program verifies if the word is correct by testing its composition: if it contains at least one digit, the word is wrong and so it writes on the error output the string bad word !. In order to test that behavior, we need to modify the Yaml test file by adding a new attribute. See the following.
$ cat check.yml
---
Test the pluralizer program: !S::Suite
attributes: !S::Cmd
command: ./pluralizer.rb
exit: 0
contents:
- Test the bird word:
input: !path bird.txt
output: "birds\n"
- Test the ant word:
args: "ant"
output: "ants\n"
- Test a bad word:
args: "b1rd"
output: ""
error: "wrong word!\n"
In fact, in the first test of the contents, we check two things: the output must be empty and the error output must be exactly equal to the string "wrong word!\\n".
Moreover, if we want to return a special exit code in this case, we only have to precise it by adding to the test the exit: attribute with the expected value, as we have seen in the second step. The default attribute exit: 0 will be overrided if you specify it again.
Our patched program:
$ cat pluralizer.rb
#! /usr/bin/env ruby
def main()
if (ARGV.length == 0)
word = STDIN.gets.chomp
else
word = ARGV[0]
end
if word =~ /\d/
STDERR.puts "wrong word!"
exit 1
else
puts word + "s"
end
end
main
and the corresponding test suite:
$ cat check.yml
---
Test the pluralizer program: !S::Suite
attributes: !S::Cmd
command: ./pluralizer.rb
exit: 0
contents:
- Test the bird word:
input: !path bird.txt
output: "birds\n"
- Test the ant word:
args: "ant"
output: "ants\n"
- Test a bad word:
args: "b1rd"
output: ""
error: "wrong word!\n"
exit: 1
and the result:
$ uttk check.yml
---
root:
contents:
- Test the pluralizer program:
contents:
- Test the bird word:
status: PASS
- Test the ant word:
status: PASS
- Test a bad word:
status: PASS
status: PASS
status: PASS
This part presents the use of the attribute weight common to all the strategies. This attribute has been introduced in order to balance the importance of each test. The weight influence the final grade of the global test suite. Look at the example below:
$ cat check.yml
---
Test the pluralizer program: !S::Suite
attributes: !S::Cmd
command: ./pluralizer.rb
exit: 0
contents:
- Test the bird word:
input: !path bird.txt
output: "birds\n"
- Test the ant word:
args: "ant"
output: "ants\n"
- Test a bad word:
args: "b1rd"
output: ""
error: "wrong word!\n"
exit: 1
- Test the trap word:
weight: -1
args: "child"
output: "childs\n"
The third test introduce the attribute weight, with a value of -1. This means that the result status will be inverted. In this case, if the output is "childs\\n" when the argument of the command is child, it means that the test is wrong ! So, we have the result:
$ uttk check.yml
---
root:
contents:
- Test the pluralizer program:
contents:
- Test the bird word:
status: PASS
- Test the ant word:
status: PASS
- Test a bad word:
status: PASS
- Test the trap word:
status: PASS
status: FAIL(75%)
status: FAIL(75%)
--- |
/-----------------------------------------------------------------------------.
| *** Some tests failed *** |
| The last status was not PASS but something like FAIL(42%) (just FAIL stands |
| for FAIL(0%)). This means that 42% of tests *pass* but 100% was expected. |
| To investigate results these different outputs are available: |
| - the standard output is the shortest one (like a progression bar). |
| - log.html contains an HTML/JavaScript page nice to display big outputs. |
| - log.yml contains more information in the very readable YAML format. |
| - log.xml contains the output as an XML document. |
`-----------------------------------------------------------------------------/
As you can see, the status of the third test is PASS, but because of the negative value of the weight, the result of the test is in fact wrong. As a consequence, we have two good tests and one fail, so the final status is FAIL (the tested program have passed only 75% of all tests).
At last, there is another use of this new attribute. As its name let guess it, it can be use for balance each test with a weight. Let's see the following.
$ cat check.yml
---
Test the pluralizer program: !S::Suite
attributes: !S::Cmd
command: ./pluralizer.rb
exit: 0
contents:
- Test the bird word:
input: !path bird.txt
output: "birds\n"
- Test the ant word:
args: "ant"
output: "ants\n"
- Test a bad word:
weight: 2
args: "b1rd"
output: ""
error: "wrong word!\n"
exit: 1
- Test the trap word:
weight: -1
args: "child"
output: "childs\n"
The result is:
$ uttk check.yml
---
root:
contents:
- Test the pluralizer program:
contents:
- Test the bird word:
status: PASS
- Test the ant word:
status: PASS
- Test a bad word:
status: PASS
- Test the trap word:
status: PASS
status: FAIL(80%)
status: FAIL(80%)
--- |
/-----------------------------------------------------------------------------.
| *** Some tests failed *** |
| The last status was not PASS but something like FAIL(42%) (just FAIL stands |
| for FAIL(0%)). This means that 42% of tests *pass* but 100% was expected. |
| To investigate results these different outputs are available: |
| - the standard output is the shortest one (like a progression bar). |
| - log.html contains an HTML/JavaScript page nice to display big outputs. |
| - log.yml contains more information in the very readable YAML format. |
| - log.xml contains the output as an XML document. |
`-----------------------------------------------------------------------------/
We have balanced the first test with a weight of 2, that's why the final percentage is 80%.
Uttk also implements a scoped environment variables. We call these "environment variables": symbols. They are all kept in a hash which may have A Sub Hashes That Represent Sub Scopes.
For instance, we can use the variable my_var instead of hard writing the value of the weight in the Yaml file. In fact, the symbol written between << and >> are expanded.
$ cat check.yml
---
Test the pluralizer program: !S::Suite
attributes: !S::Cmd
command: ./pluralizer.rb
exit: 0
contents:
- Test the bird word:
input: !path bird.txt
output: "birds\n"
- Test the ant word:
args: "ant"
output: "ants\n"
- Test a bad word:
weight: <<my_var>>
args: "b1rd"
output: ""
error: "wrong word!\n"
exit: 1
- Test the trap word:
weight: -1
args: "child"
output: "childs\n"
Obviously, we need to define this variable first. Some of these symbols are defined by the strategies themselves, but you can define you own to factor common information in your test file. Indeed, every strategies have the attribute symbols which takes a hash of symbols as argument. But you can also define some symbols from the command line by using the -S option. See the following example:
$ uttk -S 'my_var: 2' check.yml
---
root:
contents:
- Test the pluralizer program:
contents:
- Test the bird word:
status: PASS
- Test the ant word:
status: PASS
- Test a bad word:
status: PASS
- Test the trap word:
status: PASS
status: FAIL(80%)
status: FAIL(80%)
--- |
/-----------------------------------------------------------------------------.
| *** Some tests failed *** |
| The last status was not PASS but something like FAIL(42%) (just FAIL stands |
| for FAIL(0%)). This means that 42% of tests *pass* but 100% was expected. |
| To investigate results these different outputs are available: |
| - the standard output is the shortest one (like a progression bar). |
| - log.html contains an HTML/JavaScript page nice to display big outputs. |
| - log.yml contains more information in the very readable YAML format. |
| - log.xml contains the output as an XML document. |
`-----------------------------------------------------------------------------/
It is often needed to run only a sub part of a big test suite. To address this issue, Uttk allow you to select the test you want to run using a technology similar to XPath.
Using the last check.yml of the step 6, the following command select only the test: "Test the bird word" and "Test the ant word"
$ uttk --rpath '/////(bird|ant)' check.yml
---
root:
contents:
- Test the pluralizer program:
contents:
- Test the bird word:
status: PASS
- Test the ant word:
status: PASS
- Test a bad word:
status: SKIP(100%)
- Test the trap word:
status: SKIP(100%)
status: FAIL(66%)
status: FAIL(66%)
--- |
/-----------------------------------------------------------------------------.
| *** Some tests failed *** |
| The last status was not PASS but something like FAIL(42%) (just FAIL stands |
| for FAIL(0%)). This means that 42% of tests *pass* but 100% was expected. |
| To investigate results these different outputs are available: |
| - the standard output is the shortest one (like a progression bar). |
| - log.html contains an HTML/JavaScript page nice to display big outputs. |
| - log.yml contains more information in the very readable YAML format. |
| - log.xml contains the output as an XML document. |
`-----------------------------------------------------------------------------/
You can check that all other tests are skipped. Some of you may wish to not see all the skipped test. In order to learn how to do so, follow the tutorial at [[Simple log filtering]].