package alcotest

  1. Overview
  2. Docs
Alcotest is a lightweight and colourful test framework

Install

Dune Dependency

github.com Readme Changelog ISC License Edit opam file Versions (41)

Authors

  1. T
    Thomas Gazagnaire

Maintainers

  1. T
    thomas@gazagnaire.org

Sources

alcotest-lwt-1.1.0.tbz
sha256=212827a49abf4008581c0da53e7ec78a9d639b415380dcb1fdeeb23f3ff083e2
sha512=c47d04b3c7100af703b470b93ff9fe9fe22790415370b6d5972736f46a5c83901717d67caf0c4115d01020d3078dc7f3063838578174921cab352546dad00148

Description

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Published: 03 Apr 2020

README

Alcotest is a lightweight and colourful test framework.

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Examples

A simple example (taken from examples/simple.ml):

(* Build with `ocamlbuild -pkg alcotest simple.byte` *)

(* A module with functions to test *)
module To_test = struct
  let lowercase = String.lowercase_ascii
  let capitalize = String.capitalize_ascii
  let str_concat = String.concat ""
  let list_concat = List.append
end

(* The tests *)
let test_lowercase () =
  Alcotest.(check string) "same string" "hello!" (To_test.lowercase "hELLO!")

let test_capitalize () =
  Alcotest.(check string) "same string" "World." (To_test.capitalize "world.")

let test_str_concat () =
  Alcotest.(check string) "same string" "foobar" (To_test.str_concat ["foo"; "bar"])

let test_list_concat () =
  Alcotest.(check (list int)) "same lists" [1; 2; 3] (To_test.list_concat [1] [2; 3])

(* Run it *)
let () =
  let open Alcotest in
  run "Utils" [
      "string-case", [
          test_case "Lower case"     `Quick test_lowercase;
          test_case "Capitalization" `Quick test_capitalize;
        ];
      "string-concat", [ test_case "String mashing" `Quick test_str_concat  ];
      "list-concat",   [ test_case "List mashing"   `Slow  test_list_concat ];
    ]

The result is a self-contained binary which displays the test results. Use ./simple.byte --help to see the runtime options.

$ ./simple.native
Testing Utils.
[OK]       string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
Test Successful in 0.001s. 4 tests run.

Selecting tests to execute

You can filter which tests to run by supplying a regular expression matching the names of the tests to execute, or by passing a regular expression and a comma-separated list of test numbers (or ranges of test numbers, e.g. 2,4..9):

$ ./simple.native test '.*concat*'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[SKIP]     string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 2 tests run.

$ ./simple.native test 'string-case' '1..3'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[SKIP]     string-concat          0   String mashing.
[SKIP]     list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 1 test run.

Note that you cannot filter by test case name (i.e. Lower case or Capitalization), you must filter by test name & number instead. Test names may contain only alphanumeric characters, spaces, hyphens and underscores.

See the examples folder for more examples.

Quick and Slow tests

In general you should use `Quick tests: tests that are ran on any invocations of the test suite. You should only use `Slow tests for stress tests that are ran only on occasion (typically before a release or after a major change). These slow tests can be suppressed by passing the -q flag on the command line, e.g.:

$ ./test.exe -q # run only the quick tests
$ ./test.exe    # run quick and slow tests

Passing custom options to the tests

In most cases, the base tests are unit -> unit functions. However, it is also possible to pass an extra option to all the test functions by using 'a -> unit, where 'a is the type of the extra parameter.

In order to do this, you need to specify how this extra parameter is read on the command-line, by providing a Cmdliner term for command-line arguments which explains how to parse and serialize values of type 'a (note: do not use positional arguments, only optional arguments are supported).

For instance:

let test_nice i = Alcotest.(check int) "Is it a nice integer?" i 42

let int =
  let doc = "What is your prefered number?" in
  Cmdliner.Arg.(required & opt (some int) None & info ["n"] ~doc ~docv:"NUM")

let () =
  Alcotest.run_with_args "foo" int [
    "all", ["nice", `Quick, test_nice]
  ]

Will generate test.exe such that:

$ test.exe test
test.exe: required option -n is missing

$ test.exe test -n 42
Testing foo.
[OK]                all          0   int.

Lwt

Alcotest provides an Alcotest_lwt module that you could use to wrap Lwt test cases. The basic idea is that instead of providing a test function in the form unit -> unit, you provide one with the type unit -> unit Lwt.t and alcotest-lwt calls Lwt_main.run for you.

However, there are a couple of extra features:

  • If an async exception occurs, it will cancel your test case for you and fail it (rather than exiting the process).

  • You get given a switch, which will be turned off when the test case finishes (or fails). You can use that to free up any resources.

For instance:

let free () = print_endline "freeing all resources"; Lwt.return ()

let test_lwt switch () =
  Lwt_switch.add_hook (Some switch) free;
  Lwt.async (fun () -> failwith "All is broken");
  Lwt_unix.sleep 10.

let () =
  Lwt_main.run @@ Alcotest_lwt.run "foo" [
    "all", [
      Alcotest_lwt.test_case "one" `Quick test_lwt
    ]
  ]

Will generate:

$ test.exe
Testing foo.
[ERROR]             all          0   one.
-- all.000 [one.] Failed --
in _build/_tests/all.000.output:
freeing all resources
[failure] All is broken

Screenshots

The following screenshots demonstrate the HTML testing output from the odoc project.

All tests passed Some tests failed Failed test with custom diffing
ok err diff

Comparison with other testing frameworks

The README is pretty clear about that:

Alcotest is the only testing framework using colors!

More seriously, Alcotest is similar to ounit but it fixes a few of the problems found in that library:

  • Alcotest has a nicer output, it is easier to see what failed and what succeeded and to read the log outputs of the failed tests;

  • Alcotest uses combinators to define pretty-printers and comparators between the things to test.

Other nice tools doing different kind of testing also exist:

  • qcheck qcheck does random generation and property testing (e.g. Quick Check)

  • crowbar and bun are similar to qcheck, but use compiler-directed randomness, e.g. it takes advantage of the AFL support the OCaml compiler.

  • ppx_inline_tests allows to write tests in the same file as your source-code; they will be run only in a special mode of compilation.

Dependencies (8)

  1. stdlib-shims
  2. re >= "1.7.2"
  3. uuidm
  4. cmdliner >= "1.0.3" & < "1.1.0"
  5. astring
  6. fmt >= "0.8.6"
  7. ocaml >= "4.03.0"
  8. dune >= "2.0"

Dev Dependencies (1)

  1. odoc with-doc

  1. ahrocksdb
  2. albatross >= "1.5.0"
  3. alcotest-async < "1.0.0" | = "1.1.0"
  4. alcotest-lwt < "1.0.0" | = "1.1.0"
  5. alg_structs_qcheck
  6. ambient-context
  7. ambient-context-eio
  8. angstrom >= "0.7.0"
  9. ansi >= "0.6.0"
  10. anycache >= "0.7.4"
  11. anycache-async
  12. anycache-lwt
  13. archetype >= "1.4.2"
  14. archi
  15. arp
  16. arp-mirage
  17. arrakis
  18. art
  19. asak >= "0.2"
  20. asli >= "0.2.0"
  21. asn1-combinators >= "0.2.2"
  22. atd >= "2.3.3"
  23. atdgen >= "2.10.0"
  24. atdpy
  25. atdts
  26. base32
  27. base64 >= "2.1.2" & < "3.2.0" | >= "3.4.0"
  28. bastet
  29. bastet_async
  30. bastet_lwt
  31. bech32
  32. bechamel >= "0.5.0"
  33. bigarray-overlap
  34. bigstring >= "0.3"
  35. bigstring-unix >= "0.3"
  36. bigstringaf
  37. bitlib
  38. blake2
  39. bloomf
  40. bls12-381 < "0.4.1" | >= "3.0.0" & < "18.0"
  41. bls12-381-hash
  42. bls12-381-js >= "0.4.2"
  43. bls12-381-js-gen >= "0.4.2"
  44. bls12-381-legacy
  45. bls12-381-signature
  46. bls12-381-unix
  47. blurhash
  48. builder-web < "0.2.0"
  49. bulletml
  50. bytebuffer
  51. ca-certs
  52. ca-certs-nss
  53. cactus
  54. caldav
  55. calendar >= "3.0.0"
  56. callipyge
  57. camlix
  58. camlkit
  59. camlkit-base
  60. capnp-rpc < "1.2.3"
  61. capnp-rpc-lwt < "0.3"
  62. capnp-rpc-mirage >= "0.9.0"
  63. capnp-rpc-unix >= "0.9.0" & < "1.2.3"
  64. carray
  65. carton
  66. cborl
  67. ccss >= "1.6"
  68. cf-lwt
  69. chacha
  70. channel
  71. charrua-client
  72. charrua-client-lwt
  73. charrua-client-mirage < "0.11.0"
  74. checkseum >= "0.0.3"
  75. cid
  76. clarity-lang
  77. class_group_vdf
  78. cohttp >= "0.17.0"
  79. cohttp-curl-async
  80. cohttp-curl-lwt
  81. cohttp-eio >= "6.0.0~beta2"
  82. colombe >= "0.2.0"
  83. color
  84. conan
  85. conan-cli
  86. conan-database
  87. conan-lwt
  88. conan-unix
  89. conduit = "3.0.0"
  90. conex < "0.10.0"
  91. conex-mirage-crypto
  92. conex-nocrypto
  93. cookie
  94. cow >= "2.2.0"
  95. css
  96. css-parser
  97. cstruct >= "3.3.0"
  98. cstruct-sexp
  99. ctypes-zarith
  100. cuid
  101. curly
  102. current_incr
  103. cwe_checker
  104. data-encoding
  105. datakit >= "0.12.0"
  106. datakit-bridge-github >= "0.12.0"
  107. datakit-ci
  108. datakit-client-git >= "0.12.0"
  109. decompress >= "0.8" & < "1.5.3"
  110. depyt
  111. digestif >= "0.8.1"
  112. dispatch >= "0.4.1"
  113. dkim
  114. dkim-bin
  115. dkim-mirage
  116. dns >= "4.0.0"
  117. dns-cli
  118. dns-client >= "4.6.0"
  119. dns-forward < "0.9.0"
  120. dns-forward-lwt-unix
  121. dns-resolver
  122. dns-server
  123. dns-tsig
  124. dnssd
  125. dnssec
  126. docfd >= "2.2.0"
  127. dog < "0.2.1"
  128. domain-name
  129. dot-merlin-reader >= "5.3~5.3preview"
  130. dream
  131. dream-pure
  132. duff
  133. dune-release >= "1.0.0"
  134. duration >= "0.1.1"
  135. emile
  136. encore
  137. eqaf >= "0.5"
  138. equinoxe
  139. equinoxe-cohttp
  140. equinoxe-hlc
  141. eris
  142. eris-lwt
  143. ezgzip
  144. ezjsonm >= "0.4.2" & < "1.3.0"
  145. ezjsonm-lwt < "1.3.0"
  146. FPauth
  147. FPauth-core
  148. FPauth-responses
  149. FPauth-strategies
  150. faraday != "0.2.0"
  151. farfadet
  152. fat-filesystem >= "0.12.0"
  153. ff
  154. ff-pbt
  155. fiat-p256
  156. flex-array
  157. fsevents-lwt
  158. functoria >= "2.2.0"
  159. functoria-runtime >= "2.2.0" & != "3.0.1" & < "4.0.0~beta1"
  160. geojson
  161. geoml >= "0.1.1"
  162. git = "1.4.10" | = "1.5.0" | >= "1.5.2" & != "1.10.0"
  163. git-mirage < "3.0.0"
  164. git-split
  165. git-unix >= "1.10.0" & != "2.1.0"
  166. git_split
  167. gitlab-unix
  168. glicko2
  169. gmap >= "0.3.0"
  170. gobba
  171. gpt
  172. graphql
  173. graphql-async
  174. graphql-cohttp >= "0.13.0"
  175. graphql-lwt
  176. graphql_parser != "0.11.0"
  177. graphql_ppx >= "0.7.1"
  178. h1_parser
  179. h2
  180. hacl
  181. hacl-star >= "0.6.0" & < "0.7.2"
  182. hacl_func
  183. hacl_x25519 >= "0.2.0"
  184. highlexer
  185. hkdf
  186. hockmd
  187. html_of_jsx
  188. http
  189. http-multipart-formdata < "2.0.0"
  190. httpaf >= "0.2.0"
  191. httpun
  192. httpun-ws
  193. hvsock
  194. icalendar >= "0.1.4"
  195. imagelib >= "20200929"
  196. index
  197. inferno >= "20220603"
  198. influxdb-async
  199. influxdb-lwt
  200. inquire < "0.2.0"
  201. interval-map
  202. iomux
  203. irmin < "0.8.0" | >= "0.9.6" & != "0.11.1" & < "1.0.0" | >= "2.0.0" & != "2.3.0"
  204. irmin-bench >= "2.7.0"
  205. irmin-chunk < "1.3.0" | >= "2.3.0"
  206. irmin-cli
  207. irmin-containers
  208. irmin-fs < "1.3.0" | >= "2.3.0"
  209. irmin-git < "2.0.0" | >= "2.3.0"
  210. irmin-http < "2.0.0"
  211. irmin-mem < "1.3.0"
  212. irmin-pack >= "2.4.0" & != "2.6.1"
  213. irmin-pack-tools
  214. irmin-test >= "2.2.0" & < "3.0.0"
  215. irmin-tezos
  216. irmin-tezos-utils
  217. irmin-unix >= "1.0.0" & < "1.3.3" | >= "2.4.0" & != "2.6.1"
  218. irmin-watcher
  219. jekyll-format
  220. jerboa
  221. jitsu
  222. jose
  223. json-data-encoding >= "0.9"
  224. json_decoder
  225. jsonxt
  226. junit_alcotest
  227. jwto
  228. kdf
  229. ke >= "0.2"
  230. kkmarkdown
  231. lambda-runtime
  232. lambda_streams
  233. lambda_streams_async
  234. lambdapi >= "2.0.0"
  235. lambdoc >= "1.0-beta4"
  236. ledgerwallet-tezos >= "0.2.1" & < "0.4.0"
  237. letters
  238. lmdb >= "1.0"
  239. logical
  240. logtk >= "1.6"
  241. lp
  242. lp-glpk
  243. lp-glpk-js
  244. lp-gurobi
  245. lru
  246. lt-code
  247. luv
  248. mbr-format >= "1.0.0"
  249. mdx >= "1.6.0"
  250. mec
  251. mechaml >= "1.0.0"
  252. merge-queues >= "0.2.0"
  253. merge-ropes >= "0.2.0"
  254. merlin >= "4.17.1-414" & < "5.0-502" | >= "5.2.1-502"
  255. merlin-lib >= "4.17.1-414" & < "5.0-502" | >= "5.2.1-502"
  256. metrics
  257. minicaml = "0.3.1" | >= "0.4"
  258. mirage >= "4.0.0~beta1"
  259. mirage-block-partition
  260. mirage-block-ramdisk >= "0.3"
  261. mirage-channel >= "4.0.0"
  262. mirage-channel-lwt
  263. mirage-crypto-ec
  264. mirage-flow >= "1.0.2" & < "1.2.0"
  265. mirage-flow-unix
  266. mirage-fs-mem
  267. mirage-fs-unix >= "1.2.0"
  268. mirage-kv >= "2.0.0"
  269. mirage-kv-mem
  270. mirage-kv-unix
  271. mirage-logs >= "0.3.0"
  272. mirage-nat
  273. mirage-net-unix >= "2.3.0"
  274. mirage-runtime >= "4.0.0~beta1" & < "4.5.0"
  275. mirage-tc
  276. mjson
  277. mmdb
  278. mnd
  279. monocypher
  280. mrmime >= "0.2.0"
  281. mrt-format
  282. msgpck >= "1.6"
  283. mssql >= "2.0.3"
  284. multibase
  285. multihash
  286. multihash-digestif
  287. multipart-form-data
  288. multipart_form
  289. multipart_form-eio
  290. multipart_form-lwt
  291. named-pipe
  292. nanoid
  293. nbd >= "4.0.3"
  294. nbd-tool
  295. nloge
  296. nocoiner
  297. non_empty_list
  298. OCADml >= "0.6.0"
  299. obatcher
  300. ocaml-index >= "1.1"
  301. ocaml-r >= "0.5.0"
  302. ocaml-version >= "3.1.0"
  303. ocamlformat >= "0.13.0" & != "0.19.0~4.13preview" & < "0.25.1"
  304. ocamlformat-rpc < "removed"
  305. ocamline
  306. ocluster < "0.3.0"
  307. odoc >= "1.4.0" & < "2.1.0"
  308. ohex
  309. oidc
  310. opam-0install
  311. opam-0install-cudf >= "0.5.0"
  312. opam-file-format >= "2.1.1"
  313. opentelemetry >= "0.6"
  314. opentelemetry-client-cohttp-lwt >= "0.6"
  315. opentelemetry-client-ocurl >= "0.6"
  316. opentelemetry-cohttp-lwt >= "0.6"
  317. opentelemetry-lwt >= "0.6"
  318. opium >= "0.15.0"
  319. opium-graphql
  320. opium-testing
  321. opium_kernel
  322. orewa
  323. ortac-core
  324. osx-acl
  325. osx-attr
  326. osx-cf
  327. osx-fsevents
  328. osx-membership
  329. osx-mount
  330. osx-xattr
  331. otoggl
  332. owl >= "0.6.0" & != "0.9.0" & != "1.0.0"
  333. owl-base < "0.5.0"
  334. owl-ode >= "0.1.0" & != "0.2.0"
  335. owl-symbolic
  336. passmaker
  337. patch
  338. pbkdf
  339. pecu >= "0.2"
  340. pf-qubes
  341. pg_query >= "0.9.6"
  342. pgx >= "1.0"
  343. pgx_unix >= "1.0"
  344. pgx_value_core
  345. pgx_value_ptime < "2.2"
  346. phylogenetics
  347. piaf
  348. polyglot
  349. polynomial
  350. ppx_blob >= "0.3.0"
  351. ppx_deriving_cmdliner
  352. ppx_deriving_rpc
  353. ppx_deriving_yaml
  354. ppx_graphql >= "0.2.0"
  355. ppx_inline_alcotest
  356. ppx_protocol_conv >= "5.0.0"
  357. ppx_protocol_conv_json >= "5.0.0"
  358. ppx_protocol_conv_jsonm >= "5.0.0"
  359. ppx_protocol_conv_msgpack >= "5.0.0"
  360. ppx_protocol_conv_xml_light >= "5.0.0"
  361. ppx_protocol_conv_xmlm
  362. ppx_protocol_conv_yaml >= "5.0.0"
  363. ppx_repr < "0.4.0"
  364. ppx_subliner
  365. ppx_units
  366. ppx_yojson >= "1.1.0"
  367. pratter
  368. prc
  369. preface
  370. pretty_expressive
  371. prettym
  372. proc-smaps
  373. producer < "0.2.0"
  374. progress < "0.2.0"
  375. prom
  376. prometheus < "1.2"
  377. prometheus-app
  378. protocell
  379. protocol-9p >= "0.3" & < "0.11.0" | >= "0.11.2"
  380. protocol-9p-unix
  381. psq
  382. qcheck >= "0.18" & < "0.22"
  383. qcheck-alcotest < "0.22"
  384. quickjs
  385. radis
  386. randii
  387. reason-standard
  388. red-black-tree
  389. reparse >= "2.0.0" & < "3.0.0"
  390. reparse-unix < "2.1.0"
  391. resp
  392. resp-unix >= "0.10.0"
  393. rfc1951 < "1.0.0"
  394. routes < "2.0.0"
  395. rpc >= "7.1.0"
  396. rpclib >= "7.1.0"
  397. rpclib-async
  398. rpclib-lwt >= "7.1.0"
  399. rpmfile < "0.3.0"
  400. rpmfile-eio
  401. rpmfile-unix
  402. rubytt
  403. SZXX >= "4.0.0"
  404. salsa20
  405. salsa20-core
  406. sanddb >= "0.2"
  407. scaml >= "1.5.0"
  408. scrypt-kdf
  409. secp256k1 >= "0.4.1"
  410. secp256k1-internal
  411. semver >= "0.2.1"
  412. sendmail
  413. sendmail-lwt
  414. sendmail-miou-unix
  415. sendmail-mirage
  416. sendmsg
  417. server-reason-react
  418. session-cookie
  419. session-cookie-async
  420. session-cookie-lwt
  421. sherlodoc
  422. slug
  423. sodium-fmt
  424. spin >= "0.6.0"
  425. squirrel
  426. ssh-agent
  427. ssl >= "0.6.0"
  428. stramon-lib
  429. styled-ppx
  430. swapfs
  431. syslog-rfc5424
  432. tcpip >= "2.4.2" & < "4.0.0" | >= "5.0.1" & < "7.0.0"
  433. tdigest < "2.1.0"
  434. term-indexing
  435. term-tools
  436. terminal_size >= "0.1.1"
  437. terminus
  438. terminus-cohttp
  439. terminus-hlc
  440. terml
  441. textrazor
  442. tezos-base-test-helpers < "13.0"
  443. tezos-bls12-381-polynomial
  444. tezos-client-base < "12.0"
  445. tezos-crypto >= "8.0" & < "9.0"
  446. tezos-lmdb
  447. tezos-plompiler = "0.1.3"
  448. tezos-plonk = "0.1.3"
  449. tezos-signer-backends >= "8.0" & < "13.0"
  450. tezos-stdlib >= "8.0" & < "12.0"
  451. tezos-test-helpers < "12.0"
  452. tftp
  453. timedesc
  454. timere
  455. tls >= "0.12.0"
  456. toc
  457. topojson
  458. topojsone
  459. transept
  460. twostep
  461. type_eq
  462. type_id
  463. typebeat
  464. typeid >= "1.0.1"
  465. tyre >= "0.4"
  466. tyxml >= "4.0.0"
  467. tyxml-jsx
  468. tyxml-ppx >= "4.3.0"
  469. tyxml-syntax
  470. uecc
  471. ulid
  472. universal-portal
  473. unix-dirent
  474. unix-errno >= "0.3.0"
  475. unix-fcntl >= "0.3.0"
  476. unix-sys-resource
  477. unix-sys-stat
  478. unix-time
  479. unstrctrd
  480. user-agent-parser
  481. uspf
  482. uspf-lwt
  483. uspf-mirage
  484. uspf-unix
  485. utop >= "2.13.0"
  486. validate
  487. validator
  488. vercel
  489. vpnkit
  490. wcwidth
  491. websocketaf
  492. x509 >= "0.7.0"
  493. xapi-rrd >= "1.8.2"
  494. xapi-stdext-date
  495. xapi-stdext-encodings
  496. xapi-stdext-std >= "4.16.0"
  497. yaml < "3.2.0"
  498. yaml-sexp
  499. yocaml < "2.0.0"
  500. yocaml_syndication = "2.0.0"
  501. yocaml_yaml < "2.0.0"
  502. yojson >= "1.6.0"
  503. yojson-five
  504. yuscii >= "0.3.0"
  505. zar
  506. zed >= "3.2.2"
  507. zlist < "0.4.0"

Conflicts

None

OCaml

Innovation. Community. Security.

GitHub Discord Twitter Peertube RSS
About
Changelog Releases Industrial Users Academic Users Why OCaml
Ecosystem
Packages Community Events OCaml Planet Jobs
Resources
Install OCaml Get Started Platform Tools Language Manual Standard Library API Books Exercises Papers OCaml Playground Logo
Policies
Carbon Footprint Governance Privacy Code of Conduct