The Go error printing catastrophe - The CockroachDB errors library, part 3/

Go 1.13’s standard library has adopted Dave Cheney‘s main contribution to error handling from 2015: the idea that Go error objects are structured as linked lists. Alas, this adoption created a giant gap for Go developers: it has become difficult, almost impossible, to make printing error objects useful.

This is what I call the “Go error printing catastrophe,” and below we will see precisely what it is about.

Reminder: why and how Go errors are linked lists

Go’s errors API, as of v1.13, looks like this:

// error is a pre-defined type.
type error interface {
   // Error returns an error's short message string.
   // This is used e.g. when formatting an error with %s/%v.
   Error() string
}

// wrapper can be implemented by additional error
// “layers”, to decorate an error. This interface
// is not pre-defined in the language but should be
// implemented by API-conformant error decorators.
//
// This is the interface that powers the error identification
// facilities errors.Is() and errors.As().
type wrapper interface {
   // Unwrap accesses the next layer in the error object.
   // This used to be called “Cause” in Dave Cheney's
   // pkg/errors library.
   Unwrap() error
}

With this API, code in the Go ecosystem can construct error objects in two ways:

they can construct “leaf” errors, with terminal error constructors like fmt.Errorf() or errors.New().
they can “wrap” an error with a layer of decoration, for example using errors.Wrap() to prefix an error message, errors.WithStack() to attach a stack trace, or fmt.Errorf() with the %w verb (new in v1.13) as in: err = fmt.Errorf("some context: %w", err)
wrapper types declare their ability to be “peeled out” by implementing an Unwrap() method. This is checked and used by Go’s standard library, in particular errors.Is() which can identify whether an error “is” of a particular kind by looking at all the intermediate layers.

The linked list abstraction makes it possible to attach decorations to any error object, using decoration types coming from unrelated Go packages. By defining the relationship between layers as just error, there is no package dependency and no difficulty with import cycles. It also makes it possible to decouple the development of decorators/wrappers across different projects, while retaining interoperability.

For more on this topic, read this previous article in the series: The Go standard error APIs.

Reminder: Go’s formatting facilities

The most commonly used, and most versatile printing facility in the Go library is the standard fmt package. It contains, between other things, logic to format abitrary Go values to strings, to files, to buffers and to the console.

For example, fmt.Println(v) prints the value of v to the terminal.

Most of the printing functions in fmt share underlying logic, which powers the more powerful Printf() API. Printf() uses a format string and a variable argument list, and displays the arguments according to the format. This is directly derived from the similar standard API in C.

What Go’s fmt can do and C’s stdio cannot is print arbitrary data types.

This is achived using a mix of pre-defined logic to handle Go’s own base types, some intelligence to recursively print struct types, pointers and array types, and a mechanism for custom types to inject logic inside fmt: four interfaces which fmt tries to use on the values passed to Print-like functions.

The fmt.Formatter interface defines a Format(...) method, which a type can implement to completely override the formatting.
If fmt.Formatter is not present, then fmt also recognizes the predefined error interface. In that case, it calls the Error() method and prints just that.
If neither fmt.Formatter nor error are available, then depending on the formatting verb being used, a type may either implement fmt.Stringer (a String() method) or fmt.GoStringer (a GoString() method) to drive a simpler, less flexible formatting output.

Only when neither of these interfaces are implemented does fmt fall back to using its predefined logic.

For more on this topic, read this previous article in the series: Go’s formatting API.

Reminder: simple printing of errors

fmt detects error arguments and calls the Error() method automatically. This works well, even for wrapped errors: Error() is called on the outermost wrapper, the head of the linked list. That wrapper’s implementation of Error() can thus override the error of the layers “behind” it, in its tail.

For example:

errors.New("world").Error() returns world.
errors.Wrap(errors.New("world"), "hello") returns hello: world.
ditto for fmt.Errorf("hello %w", errors.New("world")), which also constructs a wrapper.

This way, we automatically get the natural “longer”, “fully equipped” Error() result when passing an error to print to fmt.

All seems well—and was well ever since Go v1.0—but what of verbose printing?

Reminder: Go’s verbose printing mode

When the %+v formatting verb is used, the internal fmt logic engages “verbose” mode to display the corresponding value in the argument list. By default, verbose mode triggers e.g. the display of field names in struct types.

For example:

s := struct { a int }{123}
fmt.Printf("%v\n", s)  // prints {123}
fmt.Printf("%+v\n", s) // verbose mode: prints {a:123}

The definition of “verbose mode” puts a new abstraction on the table: the ability for some data to be invisible in the common case, but become visible upon request.

This is useful during e.g. “printf debugging” or even simply event logging, as the expert user looking at the debugging or logging output gets access to more information than is otherwise displayed in the program’s regular output.

Custom types can only customize the output of verbose mode in fmt by implementing the fmt.Formatter interface. Only that interface’s Format() method gets information about whether verbose mode is requested or not; the other interfaces recognized by fmt are not powerful enough.

In particular, neither the error interface nor the implicit wrapper interface provide a way for error types to customize the way they are displayed in fmt. As of Go 1.16, fmt.Formatter is all there is.

Desirability of verbose printing of Go errors

The Go ecosystem already has built demand for a separate verbose mode to print error objects, in addition to the simple mode available by calling the Error() method.

For example, Dave Cheney’s pkg/errors package (source) and CockroachDB’s errors library (source) both automatically embed stack traces in error objects. This stack trace is not reported in the output of Error() and thus not included when printing the error objects in simple mode. When/if a program encounters an error and finds itself unable to handle it satisfactorily, the programmer can reach out to verbose mode with %+v to see the stack trace. This assists with understanding where the error comes from and where it has been in the program.

Additionally, a program may choose to use error wrappers to embed control information in their program that is not part of the error message, for example special numeric codes that indicate what to do during error handling in caller functions. The caller functions can use the standard API errors.As() to extract this data from the error linked list / chain.

What if the programmer in the process of troubleshooting a bug wanted to visualize this information, which is not included in the output of Error()? Again, reporting this information as part of “verbose mode” seems a natural choice.

Unfortunately, achieving that turns out to be surprisingly hard.

Fundamental shortcoming 1: loss of customization in wrappers

Let’s experiment and design our own error type, with some hidden information that only shows up in verbose mode. We could do this as follows:

type myError struct {
   msg string // public message
   code int // hidden code
}

// Error implements the error interface.
func (e *myError) Error() string { return e.msg }

// Format implements the fmt.Formatter interface.
func (e *myError) Format(s fmt.State, verb rune) {
   if verb == 'v' && s.Flag('+') {
      // Verbose mode.
      fmt.Fprintf(s, "(code: %d) %s", e.code, e.msg)
   } else {
      fmt.Fprint(s, e.msg)
   }
}

What does this give us: when we print an instance of *myError with %v, we get the text of msg; with %+v we get the same but prefixed by (code: NNN) and the value of the code field.

The astute reader may notice that this code appear incomplete, as it does not handle e.g. the %q formatting verb. This is not directly relevant in this section, so let us just ignore it for a moment.

Besides this last point, the code seems to work fine?

Alas!

Try the following code:

err := &myError{"hello", 123}
err = fmt.Errorf("wazaa: %w", err)
fmt.Println(err)         // simple mode: prints just "wazaa: hello"
fmt.Printf("%+v\n", err) // verbose: prints... what?

We would like the code in this example to print zawaa: (code: 213) hello. Unfortunately, it does not: the error type returned by fmt.Errorf does not forward the fmt.Formatter interface, and so our customization in myError is lost when wrapping with fmt.Errorf!

In other words, it is not sufficient to define an Unwrap() method to create a well-formed error wrapper types in “standard Go”; one must also implement a suitable Format() method to forward any possible formatting customization via fmt.Formatter in the wrapped error.

This presents two major problems:

it is a clear “abstraction tax”: one must implement the Format() method, even if the custom wrapper does not need to customize formatting itself, lest the fmt.Formatter interface become useless for all the participants.
this mandate was not documented in the Go library. Folk simply do not realize nor know about this. In fact, a cursory examination reveals that many custom error wrapper types in the Go ecosystem do not implement Format() and thus break format customizations in their “tail”.

Fundamental shortcoming 2: difficulty of forwarding `fmt.Formatter`

What if we were willing to pay the abstraction tax, and agree that all wrapper error types would also implement fmt.Formatter. How would one go about this?

As a supporting example, let us try a very simple wrapper which does not have any special feature:

type myWrapper struct {
   cause error // tail of linked list
}

// Error implements the error interface.
func (e *myWrapper) Error() string { return e.cause.Error() }

// Unwrap implements the unwrap interface.
func (e *myWrapper) Unwrap() error { return e.cause }

Then we can start implementing fmt.Formatter for it. At the very least, it should distinguish verbose and non-verbose mode.

But what if we’re not sure whether the error cause actually implements fmt.Formatter? Maybe it does not. So, for minimum surprise, we need to do “the same as fmt”. The best way to achieve that is to call the fmt logic itself:

// Format implements the fmt.Formatter interface.
func (e *myWrapper) Format(s fmt.State, verb rune) {
   if verb == 'v' && s.Flag('+') {
      // Verbose mode. Make fmt ask the cause
      // to print itself verbosely.
      fmt.Fprintf(s, "%+v", e.cause)
   } else {
      // Simple mode. Make fmt ask the cause
      // to print itself simply.
      fmt.Fprint(s, e.cause)
   }
}

This is a cumbersome ritual just to ensure e.cause gets printed as it wants to!

Moreover, what if e.cause wanted to know more about the original format? What if it had some custom logic to display something else when used with %#v? or %#+v? Or %q?

Unfortunately, there is no standard API in fmt to properly forward all the state to a recursive call. As of Go 1.15, the minimum amount of code that exactly forwards all the formatting state to the error cause without printing anything else is the following:

// Format implements the fmt.Formatter interface.
func (e *myWrapper) Format(s fmt.State, verb rune) {
    var f strings.Builder
    f.WriteByte('%')
    if s.Flag('+') {
        f.WriteByte('+')
    }
    if s.Flag('-') {
        f.WriteByte('-')
    }
    if s.Flag('#') {
        f.WriteByte('#')
    }
    if s.Flag(' ') {
        f.WriteByte(' ')
    }
    if s.Flag('0') {
        f.WriteByte('0')
    }
    if w, wp := s.Width(); wp {
        f.WriteString(strconv.Itoa(w))
    }
    if p, pp := s.Precision(); pp {
        f.WriteByte('.')
        f.WriteString(strconv.Itoa(p))
    }
    f.WriteRune(verb)
    fmt.Fprintf(f.String(), e.cause)
}

This looks extremely inconvenient and error-prone.

Even Dave Cheney’s pkg/error package does not do this properly; it only implements Format() in wrappers as follows:

func (w *withMessage) Format(s fmt.State, verb rune) {
    switch verb {
    case 'v':
        if s.Flag('+') {
            fmt.Fprintf(s, "%+v\n", w.Cause())
            io.WriteString(s, w.msg)
            return
        }
        fallthrough
    case 's', 'q':
        io.WriteString(s, w.Error())
    }
}

This code is incorrect for the verb %q, completely omits the other formatting flags (%#v, etc.) and does not recognize any verb other than v, s or q (for example, not x, which fmt does otherwise support for error objects without fmt.Formatter).

As far as I was able to explore in the Go ecosystem, very few custom error wrapper types achieve a correct implementation of Format().

The fact that it is so hard to implement an adequate custom Format(), and the fact there is no pre-defined (nor recommended) mechanism to forward a Format() call in fmt, is a fundamental limitation of the Go’s standard library.

(Promo pitch: a copy of the correct code above is available as a reusable fmtfwd.MakeFormat() function in my go-fmtfwd package. However, it is not a panacea. Read on.)

Fundamental shortcoming 3: problems non-repairable without API change

Go’s team praises itself for building a language that maximally preserves backward compatibility. Additions to the standard library are made by introducing or substituting features, but in a way that does not affect the semantics of existing code.

In that context, what could the Go developers do to “repair” the problems identified above, without breaking existing error code, but also without requiring existing packages to add “missing” glue code, like the missing Format() forwarders?

It turns out, there is not much that can be done within the fmt package directly.

At a high level, the impossible mission is to ensure that all the details in an error chain would be printed in verbose mode, taking into consideration customizations in Format() methods.

Because not every errors in a chain provide a Format() method, the fmt code would need to iterate itself using the Unwrap() methods. Then on each layer it would need to print… something. But what exactly?

It could not call Error(), because Error() on a wrapper will itself recurse and pick up string pieces from further layers in the chain.
It could not call Format() naively, because Format() on a wrapper already (as per the current ecosystem) applies formatting recursively on the error cause.

Since the first argument to fmt.Formatter‘s Format() is passed as an interface type, fmt.State, perhaps the fmt library could inject a special instance of State that could “separate” direct prints inside the current error layer, from those performed recursively from further layers.

For example, take an example Format() implemented like this:

// Format implements the fmt.Formatter interface.
func (e *myWrapper) Format(s fmt.State, verb rune) {
   if verb == 'v' && s.Flag('+') {
      // Verbose mode. Make fmt ask the cause
      // to print itself verbosely.
      fmt.Fprintf(s, "(code %d) %+v", e.code, e.cause)
   } else {
      // Simple mode. Make fmt ask the cause
      // to print itself simply.
      fmt.Fprint(s, e.cause)
   }
}

Given this code, the fmt package could inject a fmt.State whose implementation records all the calls to Fprintf and Fprint with the fmt.State as first argument. Simple strings and non-error values could be passed through, and every time an error value is seen it could be “ignored” (I am hand-waving) so that the outer fmt loop could then go to the next layer without creating duplicate output.

(Note: this is just an idea.)

The issue with this idea, as well as any idea that tries to make fmt.State aware of how it is being used from “within” the Format() method, is that it cannot work for packages that implement as follows:

func (w *withMessage) Format(s fmt.State, verb rune) {
    switch verb {
        // ...
    case 's', 'q':
        io.WriteString(s, w.Error())
    }
}

(This example is from pkg/errors.)

Notice how this implementation, like many others in the Go ecosystem, foils our idea: some prints use the io.Writer sub-interface of fmt.State and pass the .Error() string to it directly. There is no way to reliably detect from “within” fmt.State when a wrapper’s Format() is printing the next layer of error, and thus catch that to do something else.

And so the integration of “errors as linked lists” in the Go ecosystem collided with the fmt.Formatter abstraction and created a tar pit where everyone in the community got stuck, and where the Go standard library cannot help anyone with magic inside fmt.

Maybe to the rescue: pre-1.13 `xerrors`

In the work leading to Go 1.13, a task force was formed in 2017 to study the adoption of “errors as linked lists” and basically take over Dave Cheney’s work in pkg/errors.

This is how a group composed of Jonathan Amsterdam, Russ Cox, Marcel van Lohuizen and Damien Neil started to develop the xerrors package (source), to serve as prototype and investigation grounds for new abstractions.

This work guided the authors towards several proposals:

Marcel van Lohuizen: Error Printing — Draft Design (August 2018)
Jonathan Amsterdam, Russ Cox, Marcel van Lohuizen, Damien Neil: Proposal: Go 2 Error Inspection (January 2019)

Their work was primarily focused on the semantics of Unwrap() and the creation of the new APIs errors.Is() and errors.As() to reliably identify and extract information from error objects.

Marcel van Lohuizen focused a little more on the printing aspects of error handling, and designed the following proposal:

in addition to fmt.Formatter, error, fmt.Stringer, and fmt.GoStringer, the fmt package would be taught about a new interface: errors.Formatter.
the new interface would be implemented by error wrapper and leaf types.

the proposed interface worked as follows:

package errors

type Formatter {
     error

     // FormatError can be implemented to customize the formatting
     // of errors, instead of fmt.Formatter's Format.
     //
     // It has access to an errors.Printer (see below)
     // to actually produce output.
     //
     // In the common case, the code in FormatError details
     // the current layer and returns the next error layer
     // to print, or `nil` to indicate the tail of the
     // linked list has been reached.
     //
     // Optionally, the code for a wrapper's FormatError
     // can take over formatting of both itself *and all
     // subsequent layers* by producing its custom
     // representation for all and then returning `nil`,
     // even though its Unwrap() method is still used
     // by errors.Is() to iterate through the tail.
     FormatError(p Printer) (next error)
}

type Printer interface {
    Print(...)  // can be used to output stuff
    Printf(...) // can be used to output stuff

    // Detail is a “magic” predicate which both indicates whether
    // verbose mode is requested via %+v, and also starts indenting
    // the output performed by subsequent Print()/Printf() calls in
    // the interface, so that the details are visually “pushed to
    // the right”.
    Detail() bool
}

An example use could look like this:

// FormatError implements the errors.Formatter interface.
func (e *myWrapper) FormatError(p errors.Printer) {
    p.Print("always")
    if p.Detail() {
       p.Printf("hidden: ", e.code)
    }
    return e.cause
}

With this code, we would get the following behavior:

err := errors.New("hello")
err = &myWrapper{cause: err, code: 123}
err = &myWrapper{cause: err, code: 456}

fmt.Println(err) // simple mode: prints "always: always: hello"

fmt.Printf("%+v\n", err)
// prints:
//
//   always:
//      hidden: 456
//   always:
//      hidden: 123
//   hello

(Notice some idiosyncraties: the error was printed from outermost/head to innermost/tail, and there were automatic colons—:—inserted after each prefix, before the detail).

The way the fmt code would have been modified to use the new interface was thus:

detect whether a Format() method was available. If it was, that would be called and nothing else.
otherwise, if the object being printed was an error, it would iterate over it: call FormatError() if present and iterate using its return value as input for the next iteration.
The iteration would stop when FormatError() was not present on the error object, or nil was returned.
If at the end of iteration there was a Format() or Error() method still available to call, that would be called to “finish” the formatting.

What the xerror prototype brought to the table was the ability to focus on formatting just one layer of wrappers, without wondering about how to properly forward a Format() call to further layers.

This was thus an attempt at solving the second fundamental limitation identified above.

Alas, in no way did it solve the first fundamental limitation: if a wrapper layer did not implement FormatError(), then the fmt code would simply stop trying at that level, and any FormatError() or Format() customizations further in the error chain would remain forgotten.

Moreover, lots of folk did not like the choice to print the errors “front to back”: when troubleshooting error details, developers find it important to display the “innermost” (tail) of the linked list first, and the “outermost” (head) later. The xerrors implementation did not permit that.

Finally, all this discussion is moot anyway: the xerror printing abstractions, including errors.Formatter, errors.Printer and the corresponding fmt changes, were not selected for inclusion in Go 1.13. As of Go 1.16, any further work in this direction has been postponed until further notice.

Strategic blunder: breaking compatibility with `pkg/errors`

With more than 50.000 community projets depending on Dave Cheney’s pkg/errors, that package had become the de facto extension, able to provide a basic library of error wrappers an an example to follow for error printing customizations, albeit imperfect.

There was even an ecosystem of extensions, relying on the basic linked list abstraction, using a method called Cause() to accept the next level in a chain.

The Go team could have accepted that approach, and could have drawn the line at “all error wrappers must implement Format in a way that looks like what pkg/errors does.”. Then errors.Is() / errors.As() could have opted into pkg/errors‘s Cause() abstraction.

Alas, the Go team chose for a different method name: Unwrap(). Therefore, it has become impossible to reuse pkg/errors with the new generation of error packages developed after Go 1.13 was released.

So not only did 1.13 introduce fundamental limitations; it also prevented the Go community from continuing to use pkg/errors reliably.

In summary: the Go error printing catastrophe

In 2019, Go 1.13 has adopted Dave Cheney’s 2015 recommendation to treat error objects as linked lists. Accordingly, the Unwrap() method was standardized and the errors package was augmented with Is() and As() functions that can reliably extract information from errors structured in this way.

Unfortunately, the fmt package did not learn how to print this new shape of errors, and it has become impossible to customize the display of error objects reliably.

This is because as in previous versions, fmt only knows about Format(), Error() and String(), and only consider these methods on the tip, or “head” of an error chain.

If a package defines a custom wrapper error type but forgets to define a custom Format() method, any further Format() method in the “tail” of the linked list is ignored by fmt and customizations are lost.

Additionally, only the Format() method can provide different implementations for “verbose” and “simple” formatting (%+v / %+v). In practice, it turns nearly impossible to reliably implement Format() for a wrapper error in a way that recursively calls further customizations in the tail of the error chain.

So in short, the customization of error printing has become error-prone and essentially unreliable in Go 1.13. The other package that had some logic that folk had some agreement on, Dave Cheney’s pkg/errors, was made incompatible by abandoning its key Cause() interface in Go 1.13. The only further attempt by the Go team to repair this situation inside the Go standard library, the xerrors project, did not actually solve these problems successfully, had major new flaws and was ultimately abandoned as unsatisfactory.

And this is how we, programmers, are left with no solution.

This is the Go error printing catastrophe, and it remains in Go 1.16.

Next steps

The CockroachDB errors library has spent significant effort in the area of error printing. Although it does not fill all the gaps, it does make the situation much less painful.

The next article in this series explains further.

References

So what do you think? Did I miss something? Is any part unclear? Leave your comments below.

Comments

The Go error printing catastrophe The CockroachDB errors library, part 3/