gauge is a library for doing performance measurement and analysis. It
is actually a fork of criterion package but with less
dependencies. So, this documentation can also be used for criterion
package with little or no changes.
Get started
$ stack new mylib rio --resolver lts-13.8
$ cd mylib
Enable benchmarking by including the following lines in the
package.yaml file:
benchmarks:
reverse-bench:
main: reverse.hs
source-dirs: bench
dependencies:
- mylib
- gauge
ghc-options:
- -O2
Create an src/Reverse.hs:
{-# LANGUAGE NoImplicitPrelude #-}
module Reverse
( myReverse
) where
import RIO
myReverse :: [a] -> [a]
myReverse [] = []
myReverse (x:xs) = myReverse xs ++ [x]
Benchmarking Pure code
Create bench/reverse.hs:
import Reverse
import Gauge
main :: IO ()
main =
defaultMain $ map group [5, 100]
where
group size =
bgroup (show size)
[ bench "reverse" $ whnf reverse list
, bench "myReverse" $ whnf myReverse list
]
where
list = [1..size :: Int]
And now run in a terminal:
$ stack bench --file-watch --fast
Running 1 benchmarks...
Benchmark reverse-bench: RUNNING...
benchmarked 5/reverse
time 20.21 ns (20.02 ns .. 20.40 ns)
0.999 R² (0.998 R² .. 1.000 R²)
mean 20.28 ns (20.18 ns .. 20.59 ns)
std dev 534.5 ps (225.4 ps .. 1.115 ns)
variance introduced by outliers: 11% (moderately inflated)
benchmarked 5/myReverse
time 50.47 ns (50.37 ns .. 50.60 ns)
1.000 R² (1.000 R² .. 1.000 R²)
mean 49.91 ns (49.77 ns .. 50.03 ns)
std dev 423.5 ps (359.4 ps .. 541.1 ps)
benchmarked 100/reverse
time 235.7 ns (231.2 ns .. 242.0 ns)
0.998 R² (0.997 R² .. 1.000 R²)
mean 236.2 ns (234.9 ns .. 238.6 ns)
std dev 5.908 ns (3.911 ns .. 9.003 ns)
variance introduced by outliers: 10% (moderately inflated)
benchmarked 100/myReverse
time 865.1 ns (855.7 ns .. 876.4 ns)
0.999 R² (0.998 R² .. 1.000 R²)
mean 866.3 ns (861.8 ns .. 873.1 ns)
std dev 18.73 ns (12.72 ns .. 27.74 ns)
Benchmark reverse-bench: FINISH
Our function myReverse performs badly as compared to the reverse
function exposed from RIO module.
Benchmarking IO actions
Create src/Download.hs:
{-# LANGUAGE OverloadedStrings #-}
module Download where
import Network.HTTP.Simple
import RIO
import RIO.ByteString.Lazy
import RIO.Process
conduitDownload :: IO ()
conduitDownload = do
response <- httpLBS "https://google.com"
writeFileBinary "conduit.txt" (toStrict $ getResponseBody response)
wgetDownload :: IO ()
wgetDownload = do
runSimpleApp $ do
void $
proc
"wget"
["-O", "wget.txt", "--quiet", "https://google.com"]
readProcessStdout_
Change the benchmarking code to:
import Download
import Gauge
main :: IO ()
main = defaultMain [downloadBenchmark]
where
downloadBenchmark =
bgroup
"Download"
[ bench "conduit" $ nfIO conduitDownload
, bench "wget" $ nfIO wgetDownload
]
And this is the benchmarking output you get:
benchmarking Download/conduit ... took 9.380 s, total 56 iterations
benchmarked Download/conduit
time 182.4 ms (156.4 ms .. 232.4 ms)
0.920 R² (0.833 R² .. 1.000 R²)
mean 165.4 ms (158.4 ms .. 191.3 ms)
std dev 20.15 ms (4.015 ms .. 33.56 ms)
variance introduced by outliers: 38% (moderately inflated)
benchmarking Download/wget ... took 13.88 s, total 56 iterations
benchmarked Download/wget
time 240.8 ms (211.7 ms .. 264.9 ms)
0.982 R² (0.958 R² .. 1.000 R²)
mean 260.1 ms (245.7 ms .. 299.1 ms)
std dev 40.06 ms (7.484 ms .. 66.67 ms)
variance introduced by outliers: 48% (moderately inflated)
Let's play with vector
{-# LANGUAGE NoImplicitPrelude #-}
module Reverse
( myReverse
, betterReverse
, vectorReverse
, uvectorReverse
, svectorReverse
) where
import RIO
import qualified RIO.Vector as V
import qualified RIO.Vector.Boxed as VB
import qualified RIO.Vector.Storable as VS
import qualified RIO.Vector.Unboxed as VU
import qualified Data.Vector.Generic.Mutable as VM
myReverse :: [a] -> [a]
myReverse [] = []
myReverse (x:xs) = myReverse xs ++ [x]
betterReverse :: [a] -> [a]
betterReverse =
loop []
where
loop res [] = res
loop res (x:xs) = loop (x:res) xs
vectorReverseGeneric
:: V.Vector v a
=> [a]
-> v a
vectorReverseGeneric input = V.create $ do
let len = length input
v <- VM.new len
let loop [] idx = assert (idx == -1) (return v)
loop (x:xs) idx = do
VM.unsafeWrite v idx x
loop xs (idx - 1)
loop input (len - 1)
{-# INLINEABLE vectorReverseGeneric #-}
vectorReverse :: [a] -> [a]
vectorReverse = VB.toList . vectorReverseGeneric
{-# INLINE vectorReverse #-}
svectorReverse :: VS.Storable a => [a] -> [a]
svectorReverse = VS.toList . vectorReverseGeneric
{-# INLINE svectorReverse #-}
uvectorReverse :: VU.Unbox a => [a] -> [a]
uvectorReverse = VU.toList . vectorReverseGeneric
{-# INLINE uvectorReverse #-}
And bench/reverse.hs
import Lib
import Gauge
main :: IO ()
main =
defaultMain $ map group [5, 100, 10000, 1000000]
where
group size =
bgroup (show size)
[ bench "reverse" $ whnf reverse list
, bench "myReverse" $ whnf myReverse list
, bench "betterReverse" $ whnf betterReverse list
, bench "vectorReverse" $ whnf vectorReverse list
, bench "svectorReverse" $ whnf svectorReverse list
, bench "uvectorReverse" $ whnf uvectorReverse list
]
where
list = [1..size :: Int]
Results:
Benchmark reverse-bench: RUNNING...
benchmarking 5/reverse
time 23.28 ns (21.61 ns .. 25.46 ns)
0.967 R² (0.939 R² .. 0.999 R²)
mean 22.36 ns (21.83 ns .. 23.78 ns)
std dev 2.680 ns (1.041 ns .. 5.029 ns)
variance introduced by outliers: 94% (severely inflated)
benchmarking 5/myReverse
time 44.75 ns (44.26 ns .. 45.20 ns)
0.999 R² (0.999 R² .. 1.000 R²)
mean 44.44 ns (43.98 ns .. 44.86 ns)
std dev 1.413 ns (1.185 ns .. 1.744 ns)
variance introduced by outliers: 50% (severely inflated)
benchmarking 5/betterReverse
time 22.67 ns (22.21 ns .. 23.28 ns)
0.997 R² (0.994 R² .. 0.999 R²)
mean 22.75 ns (22.47 ns .. 23.08 ns)
std dev 1.055 ns (836.9 ps .. 1.428 ns)
variance introduced by outliers: 70% (severely inflated)
benchmarking 5/vectorReverse
time 424.6 ns (413.9 ns .. 437.0 ns)
0.996 R² (0.994 R² .. 0.999 R²)
mean 416.7 ns (411.3 ns .. 423.0 ns)
std dev 20.06 ns (14.97 ns .. 27.91 ns)
variance introduced by outliers: 66% (severely inflated)
benchmarking 5/svectorReverse
time 69.79 ns (69.14 ns .. 70.36 ns)
0.999 R² (0.999 R² .. 1.000 R²)
mean 69.76 ns (69.20 ns .. 70.47 ns)
std dev 2.173 ns (1.807 ns .. 2.647 ns)
variance introduced by outliers: 49% (moderately inflated)
benchmarking 5/uvectorReverse
time 69.79 ns (68.58 ns .. 71.32 ns)
0.998 R² (0.995 R² .. 0.999 R²)
mean 69.26 ns (68.52 ns .. 70.17 ns)
std dev 2.928 ns (2.210 ns .. 4.326 ns)
variance introduced by outliers: 64% (severely inflated)
benchmarking 100/reverse
time 440.4 ns (437.1 ns .. 443.8 ns)
0.999 R² (0.998 R² .. 1.000 R²)
mean 448.7 ns (442.5 ns .. 471.6 ns)
std dev 34.27 ns (13.84 ns .. 79.16 ns)
variance introduced by outliers: 83% (severely inflated)
benchmarking 100/myReverse
time 1.020 μs (997.1 ns .. 1.053 μs)
0.996 R² (0.992 R² .. 0.999 R²)
mean 1.020 μs (1.008 μs .. 1.039 μs)
std dev 52.24 ns (38.18 ns .. 74.27 ns)
variance introduced by outliers: 67% (severely inflated)
benchmarking 100/betterReverse
time 432.5 ns (429.5 ns .. 436.6 ns)
0.999 R² (0.999 R² .. 1.000 R²)
mean 433.0 ns (429.5 ns .. 438.0 ns)
std dev 13.61 ns (11.12 ns .. 17.58 ns)
variance introduced by outliers: 45% (moderately inflated)
benchmarking 100/vectorReverse
time 3.304 μs (3.276 μs .. 3.344 μs)
0.998 R² (0.996 R² .. 0.999 R²)
mean 3.383 μs (3.306 μs .. 3.548 μs)
std dev 354.1 ns (106.1 ns .. 609.7 ns)
variance introduced by outliers: 89% (severely inflated)
benchmarking 100/svectorReverse
time 695.4 ns (682.1 ns .. 708.0 ns)
0.998 R² (0.997 R² .. 0.999 R²)
mean 693.5 ns (686.7 ns .. 702.2 ns)
std dev 24.88 ns (21.48 ns .. 30.24 ns)
variance introduced by outliers: 51% (severely inflated)
benchmarking 100/uvectorReverse
time 677.0 ns (669.2 ns .. 684.9 ns)
0.999 R² (0.997 R² .. 0.999 R²)
mean 676.3 ns (669.5 ns .. 684.9 ns)
std dev 26.76 ns (20.91 ns .. 39.11 ns)
variance introduced by outliers: 56% (severely inflated)
benchmarking 10000/reverse
time 83.32 μs (81.00 μs .. 86.26 μs)
0.992 R² (0.985 R² .. 0.998 R²)
mean 84.04 μs (82.26 μs .. 87.09 μs)
std dev 7.867 μs (4.523 μs .. 14.30 μs)
variance introduced by outliers: 80% (severely inflated)
benchmarking 10000/myReverse
time 1.021 ms (975.4 μs .. 1.085 ms)
0.968 R² (0.939 R² .. 0.993 R²)
mean 1.019 ms (989.0 μs .. 1.067 ms)
std dev 125.8 μs (81.76 μs .. 210.2 μs)
variance introduced by outliers: 80% (severely inflated)
benchmarking 10000/betterReverse
time 80.50 μs (79.97 μs .. 81.07 μs)
1.000 R² (0.999 R² .. 1.000 R²)
mean 80.94 μs (80.37 μs .. 81.84 μs)
std dev 2.240 μs (1.670 μs .. 3.303 μs)
variance introduced by outliers: 25% (moderately inflated)
benchmarking 10000/vectorReverse
time 490.6 μs (467.7 μs .. 516.9 μs)
0.974 R² (0.953 R² .. 0.995 R²)
mean 487.4 μs (467.1 μs .. 526.0 μs)
std dev 88.08 μs (53.06 μs .. 140.0 μs)
variance introduced by outliers: 91% (severely inflated)
benchmarking 10000/svectorReverse
time 82.30 μs (81.49 μs .. 83.28 μs)
0.999 R² (0.998 R² .. 0.999 R²)
mean 84.65 μs (83.64 μs .. 85.73 μs)
std dev 3.463 μs (2.856 μs .. 4.741 μs)
variance introduced by outliers: 43% (moderately inflated)
benchmarking 10000/uvectorReverse
time 82.09 μs (80.97 μs .. 83.18 μs)
0.998 R² (0.995 R² .. 0.999 R²)
mean 83.02 μs (81.81 μs .. 85.07 μs)
std dev 5.168 μs (3.065 μs .. 8.483 μs)
variance introduced by outliers: 64% (severely inflated)
benchmarking 1000000/reverse
time 60.50 ms (56.80 ms .. 64.33 ms)
0.990 R² (0.975 R² .. 0.997 R²)
mean 60.70 ms (58.23 ms .. 63.52 ms)
std dev 4.870 ms (3.493 ms .. 6.662 ms)
variance introduced by outliers: 24% (moderately inflated)
benchmarking 1000000/myReverse
time 155.0 ms (142.2 ms .. 166.9 ms)
0.991 R² (0.958 R² .. 1.000 R²)
mean 156.2 ms (152.1 ms .. 165.9 ms)
std dev 8.219 ms (1.470 ms .. 12.12 ms)
variance introduced by outliers: 13% (moderately inflated)
benchmarking 1000000/betterReverse
time 61.28 ms (56.93 ms .. 64.75 ms)
0.990 R² (0.981 R² .. 0.997 R²)
mean 60.47 ms (58.42 ms .. 63.90 ms)
std dev 4.605 ms (2.755 ms .. 6.999 ms)
variance introduced by outliers: 24% (moderately inflated)
benchmarking 1000000/vectorReverse
time 59.07 ms (55.77 ms .. 62.18 ms)
0.994 R² (0.989 R² .. 0.998 R²)
mean 56.09 ms (53.91 ms .. 58.04 ms)
std dev 3.582 ms (2.581 ms .. 5.108 ms)
variance introduced by outliers: 16% (moderately inflated)
benchmarking 1000000/svectorReverse
time 12.78 ms (12.16 ms .. 13.55 ms)
0.955 R² (0.881 R² .. 0.996 R²)
mean 13.49 ms (13.09 ms .. 14.58 ms)
std dev 1.558 ms (663.3 μs .. 2.871 ms)
variance introduced by outliers: 56% (severely inflated)
benchmarking 1000000/uvectorReverse
time 12.04 ms (11.72 ms .. 12.31 ms)
0.997 R² (0.993 R² .. 0.999 R²)
mean 12.22 ms (12.07 ms .. 12.37 ms)
std dev 388.4 μs (306.0 μs .. 490.7 μs)
variance introduced by outliers: 10% (moderately inflated)
Question What does this report tell you about the performance?