Comparing Type Error Messages Across Languages

Update (12pm): Made chosen implementation of a language more clear, as error message quality varies across implementations.

The purpose of this post is to survey the landscape of error messages from various language implementations with respect to a simple type mismatch. I wanted to take a look at the many ways compilers and interpreters respond to:

1 + "1"

This is a rather trivial example, and that’s part of the point. I didn’t want to bring in polymorphism, higher-kinds, or any nesting of said features. I did want to higlight what languages would catch this at compile-time, what languages would catch this at run-time, and what languages would just coerce. In all cases, I wanted to see how clearly the nature of the error was expressed.

Some languages fail to catch this error, so I gave them the following test as a second chance:

1 < "1"

Good Error Messages

A good error message signals clearly and succinctly what went wrong. The design of good error messages is demanding on system designers. It requires that:

Here’s an example of what not to do:

With that out of the way, let’s take an overview via language categories and their representatives:

Functional. Compile-time Type Check

Functional programming languages tend to provide the most feature-rich type systems. Notably, only one language below fails test (1) above. That same language passes test (2).

Agda and Coq get special tests because they don’t package ints and strings as primitive types, as far as I can tell. They certainly would pass tests (1) and (2) above, though.

Functional, Run-time Type Check

These languages give users the option of leveraging type checking before running their programs. For example, check out Typed Clojure at Strange Loop 2014. All of these languages catch the type mismatch above.

Object-oriented, Compile-time Type Check

Object-oriented languages tend to discard type-safety in favor of more familiar 1 models of computation. Two of the five languages below fail type test (1). All of them pass type test (2).

Object-oriented, Run-time Type Check

This set of languages amounts for most of the code written today. Three of these seven fail type test (1). Two of them fail even type test (2). Python (mypy), Ruby (rtc), PHP (hack), and Lua (typed Lua) all appeared within the past year to support optional, compile-time type checking.

Imperative

These languages operate close to the model of hardware and tend to have poor support for abstraction.

Javascript

Powers your browser, for better or worse. Fortunately, all of the alternatives fare better than vanilla JS in the type test.

SQL

Finally, why not? How do these SQL implementations handle a variation on the type test?

Other

I’ve included a few applied-math-oriented languages below. Octave is a matrix/vector DSL of sorts. R is a statistics-focused programming environment.

Wrap Up

We’ve seen a variety of error messages and language styles. Some languages caught errors at compile-time, others at run-time. Some languages lacked a REPL. Some languages did not pass test (1). Some, even failed test (2). Some languages required some boilerplate code to run these tests. Some languages required entirely different tests.

This experiment was greatly facilitated by nix. Often times, it was just a matter of:

$ nix-env -i <language>

… to get up and running!

An interesting follow up would be to see how languages supporting polymorphism with compile-time type checking express type errors involving polymorphism.

Towards what end did I write this post? Curiosity! Programming languages are interesting in and of themselves to me. I hope that my curiosity has given you some entertaining reading!

If nothing else, this post demonstrates the variety of programming languages that exist. The world is rich with means to program computers. I encourage you to explore it!

The Catalogue

Agda ↵

data Bool : Set where
  true : Bool
  false : Bool

data Nat : Set where
  zero : Nat
  succ : Nat -> Nat

not : Bool -> Bool
not true = zero
not false = true
/home/allele/langs/x.agda:23,12-16
Nat !=< Bool of type Set
when checking that the expression zero has type Bool

ATS ↵

val _ = 1 + "1"
$ atscc x.dats
/home/allele/x.dats: 9(line=1, offs=9) -- 16(line=1, offs=16): error(3): the symbol [+] cannot be resolved: there is no match.
exit(ATS): uncaught exception: ATS_2d0_2e2_2e11_2src_2ats_error_2esats__FatalErrorException(1027)

Coq ↵

Inductive day : Type :=
  | monday : day
  | tuesday : day
  | wednesday : day
  | thursday : day
  | friday : day
  | saturday : day
  | sunday : day.

Inductive bool : Type :=
  | true : bool
  | false : bool.

Definition negb (b:bool) : bool :=
  match b with
    | true => monday
    | false => true
  end.
Toplevel input, characters 64-70:
Error: In environment
b : bool
The term "monday" has type "day" while it is expected to have type
"bool".

Fsharp ↵

1 + "1";;

  1 + "1";;
  ----^^^

/home/allele/stdin(1,5): error FS0001: The type 'string' does not match the type 'int'

Haskell ↵

1 + "1"

    No instance for (Num [Char]) arising from a use of ‘+’
    In the expression: 1 + "1"
    In an equation for ‘it’: it = 1 + "1"

Idris ↵

1 + "1"
Can't resolve type class Num String

Ocaml ↵

1 + "1";;
Error: This expression has type string but an expression was expected of type
         int

Rust ↵

fn main() {
  return 1 + "1";
}
$ rustc x.rs
x.rs:2:14: 2:17 error: mismatched types: expected `<generic integer #0>`, found `&'static str` (expected integral variable, found &-ptr)
x.rs:2   return 1 + "1";
                    ^~~
x.rs:2:10: 2:17 error: mismatched types: expected `()`, found `<generic integer #0>` (expected (), found integral variable)
x.rs:2   return 1 + "1";
                ^~~~~~~
error: aborting due to 2 previous errors

Scala ↵

1 + "1"
"11"
1 < "1"
error: overloaded method value < with alternatives:
  (x: Double)Boolean <and>
  (x: Float)Boolean <and>
  (x: Long)Boolean <and>
  (x: Int)Boolean <and>
  (x: Char)Boolean <and>
  (x: Short)Boolean <and>
  (x: Byte)Boolean
 cannot be applied to (String)
              1 < "a"

Standard ML (NJ) ↵

1 + "1";
stdIn:1.2-1.9 Error: operator and operand don't agree [literal]
  operator domain: int * int
  operand:         int * string
  in expression:
    1 + "1"

Swift ↵

1 + "1"

error: type 'Int' does not conform to protocol 'ExtendedGraphemeClusterLiteralConvertible'

Clojure ↵

(+ 1 "1")

ClassCastException java.lang.String cannot be cast to java.lang.Number  clojure.lang.Numbers.add (Numbers.java:126)

Common Lisp (SBCL) ↵

(+ 1 "1")

debugger invoked on a SIMPLE-type-errors in thread
#<THREAD "main thread" RUNNING {1002D6D1F3}>:
  Argument Y is not a NUMBER: "1"

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [ABORT] Exit debugger, returning to top level.

(SB-KERNEL:TWO-ARG-+ 1 "1")

Emacs Lisp ↵

(+ 1 "1")

1 Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p "1")
  2   +(1 "1")
  eval((+ 1 "1") nil)
  eval-last-sexp-1(nil)
  eval-last-sexp(nil)
  funcall-interactively(eval-last-sexp nil)
  call-interactively(eval-last-sexp nil nil)
  command-execute(eval-last-sexp)

Elixir ↵

1 + "1"
$ elixirc x.ex

x.ex:1: warning: this expression will fail with a 'badarith' exception

== Compilation error on file x.ex ==
** (ArithmeticError) bad argument in arithmetic expression
    x.ex:1: (file)
    (elixir) src/elixir_lexical.erl:17: :elixir_lexical.run/3
    (elixir) lib/kernel/parallel_compiler.ex:95: anonymous fn/4 in Kernel.ParallelCompiler.spawn_compilers/8

Erlang ↵

1 + "1".
** exception error: an error occurred when evaluating an arithmetic expression
     in operator  +/2
        called as 1 + "1"

Racket ↵

(+ 1 "1")
; +: contract violation
;   expected: number?
;   given: "1"
;   argument position: 2nd
; [,bt for context]

C# ↵

1 + "1"
"11"
1 < "1"
(1,2): error CS0019: Operator `<' cannot be applied to operands of type `int' and `string'

C++ ↵

int main()
{
  return 1 + "1";
}
x.c:3:10: error: cannot initialize return object of type 'int' with an rvalue
      of type 'const char *'
  return 1 + "1";
         ^~~~~~~
1 error generated.

D ↵

int main() {
  return 1 + "1";
}
$ dmd x.d
x.d(2): Error: incompatible types for ((1) + ("1")): 'int' and 'string'

Go ↵

package main

func main () {
  return 1 + "1";
}
$ go build x.go
./x.go:4: cannot convert "1" to type int
./x.go:4: invalid operation: 1 + "1" (mismatched types int and string)
./x.go:4: too many arguments to return

Java ↵

class App
{
  public static void main()
  {
    return 1 < "1";  // gladly accepts 1 + "1"
  }
}
$ javac x.java
x.java:5: error: incompatible types: unexpected return value
    return 1 < "1";
             ^
1 error

Lua ↵

print(1 + "1")
2
print(1 < "1")
stdin:1: attempt to compare number with string
stack traceback:
	stdin:1: in main chunk
	[C]: in ?

Perl ↵

print 1 + "1"
print 1 < "1"
$ perl x.pl
2

Hack, PHP ↵

<?hh
function f(int $x): int {return $x+"cat";}

echo f(1);
echo f("cat");
$ hhvm x.php
1
Fatal error: Argument 1 passed to f() must be an instance of int, string given in /home/allele/x.php on line 2
<?php
function f($x) {return $x+"cat";}

echo f(1);
echo f("cat");
$ php x.php
10

Python ↵

1 + "1"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'int' and 'str'

Ruby ↵

1 + "1"
TypeError: String can't be coerced into Fixnum
	from (irb):1:in `+'
	from (irb):1
	from /usr/bin/irb:12:in `<main>'

Ada ↵

with Ada.Text_IO; use Ada.Text_IO;
procedure X is
begin
  Put_Line(1 + "1");
end X;
$ gnatmake x.adb
gcc-4.6 -c x.adb
x.adb:4:16: expected type universal integer
x.adb:4:16: found a string type
gnatmake: "x.adb" compilation error

C ↵

int main()
{
  return 1 + "1";
}
$ clang x.c
x.c:3:10: warning: incompatible pointer to integer conversion returning
      'char *' from a function with result type 'int' [-Wint-conversion]
  return 1 + "1";
         ^~~~~~~
1 warning generated.

Fortran ↵

      program hello
        print *, 1 + "1"
      end program hello
$ gfortran x.f
x.f:2.16:

        print *, 1 + "1"
                1
Error: Operands of binary numeric operator '+' at (1) are INTEGER(4)/CHARACTER(1)

Javascript ↵

1 + "1"
'11'

Dart ↵

int main() {
  return 1 + "1";
}
$ dart2js x.dart
Warning: 'String' is not assignable to 'num'.
  return 1 + "1";
             ^^^
../lib/core/num.dart:83:16:
Info: This is the method declaration.
  num operator +(num other);

Elm ↵

main = 1 + "1"
$ elm x.elm
[1 of 1] Compiling Main                ( x.elm )
Type error on line 1, column 8 to 15:
A number must be an Int or Float.
        1 + "1"

   Expected Type: number
     Actual Type: String

Purescript ↵

1 + "1"

Error at  line 1, column 1:
Error in declaration it
Error in expression "1":
Expr does not have type Prim.Number

Typescript ↵

function a(x: string): void {}

a(1);

Supplied parameters do not match any signature of call target: Could not apply 'string' to argument 1 which is of type 'number'.

MySQL ↵

> select abs(-1);
+---------+
| abs(-1) |
+---------+
|       1 |
+---------+
1 row in set (0.00 sec)

> select abs("cat");
+------------+
| abs("cat") |
+------------+
|          0 |
+------------+
1 row in set, 1 warning (0.00 sec)

sqlite ↵

select abs("cat");
0.0

Octave ↵

1 + "1"
> 50

R ↵

1 + "1"
Error in "1" + "1" : non-numeric argument to binary operator

  1. Object-orientation is familiar at this point because it is so widely taught. It is uncommon to find programming courses, academic or otherwise, that teach alternate models of expression.↩