My Go executable files are still large What's new in 2021 and Go 1.16

Introduction

Two years ago, my article “Why are my Go executable files so large?” showed how to utilize D3 and a tree map visualization to explore the size of executable files produced by the Go compiler.

A few things have changed since, and so an update is in order.

Tooling updates

As presented the first time around, we are using a data pipeline that looks as follows:

1. build a Go executable.
2. use go tool nm -size and apply c++filt on the output.
3. transform the output into a tree using a custom-designed Python script tab2dict.py.
4. transform the tree into a valid input for the D3 tree map visualization using a custom-designed Python script simplify.py.

The source code for the Python scripts is public on GitHub: https://github.com/knz/go-binsize-viz

Since 2019, the Go and C++ compilers produce a larger diversity of symbols; the regular expressions used in tab2dict.py have been adjusted accordingly.

Separately, there was a usability shortcoming in the original implementation: if a Go package contained both some source files (e.g. sql/create.go) and sub-packages (e.g. sql/sem/tree/eval.go), the “own” size of the package and that of its sub-packages were appearing side-by-side in the visualization, instead of “inside” each other. This was confusing because the (human) explorer naturally expects a hierarchical view between these two values.

This shortcoming has also been corrected.

Surprising finding: “dark” file usage

The sum of the sizes reported by go tool nm does not add up to the final size of the Go executable.

For example, in the CockroachDB 20.2.7 binary:

• the file occuppies 211694984 bytes (202MiB) on disk;
• however, the sum of symbol sizes adds up to 118928245 bytes (113MB).
• there is a gap of 92766739 bytes (88MiB) missing, or ~44% unaccounted for.

At first I suspected that this size was occupied by the symbol table itself, or the debugging information. To check this, we can use strip to remove the symbol table and debugging info and observe the difference. Alas:

• the stripped executable size is 190680384 bytes (182MiB) on disk;
• so there is still a gap of ~68MiB, or ~34% non-symtable data that is unaccounted for.

At this time, I do not have an explanation for this “dark” file usage.

(The word “dark” here reflects the idea that the bytes are not enlightened by the symbol table. It is also inspired from the concept of dark silicon invented/discovered around 2011. It is not a statement of moral value about the nature of these bytes.)

We can see how this dark file usage has evolved throughout the growth of CockroachDB:

In this table:

• “Exec. size” is the raw size of the executable file, in bytes.
• “Stripped” is the size of the executable after the strip command was applied; i.e. after the symbol table and debugging information are removed.
• “Sum nm -size” is the sum of the advertised sizes of the entries in the symbol table.
• “Symtable sz.” is the estimated size of the symbol table itself, as deducted by taking the difference between the first two sizes. We can see that the v1.0.7 to v2.0.7 executables, as well as v20.1.0, were released pre-stripped.
• “Dark bytes” is the gap between the raw file size and the combined sum of the advertised symbol sizes and the symbol table’s size and debugging information, in bytes.
• “% dark bytes” is the percentage of the dark bytes relative to the raw file size.

We can see that the dark size percentage was lower than 20% prior to CockroachDB v19.1, and has then been oscillating around 33% of the file size until v21.1. With the upcoming v21.1 release, using Go 1.15, the dark size is reduced to 15% again.

The evolution of pclntab

Transition from Go 1.15 to 1.16

Using the source code for CockroachDB v21.1-alpha-geb1aa69bc4, we can produce custom builds across Linux and FreeBSD, with both the 1.15 and 1.16 compilers.

What do we see here?

From Go 1.15 to 1.16, the overall size of the executable file has decreased. This reflects improvements in the Go toolchain.

Meanwhile, the bytes previously occupied by pclntab are now part of the dark bytes, which are not in the symbol table.

So much data! For what exactly?

An interesting way to think about the results above is that we now have two parts of a Go executable file that do not really contribute to making a program “work”:

• Optional parts, which can be deleted via strip:
  • the symbol table itself,
  • the debugging information.
• Parts which are neither code nor data; we’ll call them “Go internal data”:
  • The pclntab, when generated. This is needed to generate stack traces upon errors and other debugging-related runtime features of Go programs.
  • The “dark bytes”, which is byte usage in the raw executable file not accounted for in the symbol table.

We can derive the numbers from the tables above:

Summary and conclusions

In our original analysis in 2019, we looked at the output of go tool nm -size and drew a tree map representation for it. This helped us detect an anomaly, in the size of a special data structure called runtime.pclntab, which was growing excessively large.

In files generated by a Go 1.16 compiler, this data structure appears to be absent from the symbol table. Was it removed?

This year, we revisited the analysis and discovered that the symbol table is not complete. There are many bytes in the binary executable that are not accounted for, neither by the announced size of objects in the symbol table, nor by the size of the symbol table itself.

We can call this the “dark file usage” of Go binaries. It is “dark” not because it is “bad” or “unknown” but because it is not enlightened by the symbol table.

In fact, the removal of the advertised bytes for pclntab in Go 1.16 was accompanied by a corresponding increase to the “dark” bytes. The data is still there, it is just not accounted for. As explained by Russ Cox from the Go team, it is not accounted for because it does not need to be.

These bytes of non-code, non-data objects have been made necessary by internal algorithmic choices by the Go team, which are not made in the same way in other languages, and we can thus say they are “Go internal bytes”.

They are nearly as large as the code+data bytes in the file: for every 6 bytes of code+data, there are 5 accompanying “Go internal” bytes (approximately, in the latest version).

The proportion of these “Go internal bytes” relative to program code+data has grown in CockroachDB over the course of Go versions up to Go 1.13, and then reduced in Go 1.15, and has grown again in Go 1.16. However, meanwhile, improvements in the Go toolchain have reduced the absolute size of compiled code+data to a larger extent, so that the corresponding small increase in the size of Go internal bytes is not reflected in total file size.

Separately and independently, the Go compiler produces extensive debugging information in the DWARF format. This data, together with the symbol tables, grows in proportion to the number and size of items in the compiled code. It is optional and can be removed (“stripped”) without change in functionality. However, it is also fairly large and has grown recently in absolute size: more than 50% of total executable size in the latest versions. Moreover, Go application deployments in the wild do not often strip this data out.

So in conclusion, if we combine the Go internal bytes (both “dark” and non-“dark”) with the optional symbol tables and debugging information, we see a growing proportion of executable files encoded as non-code, non-data, currently at 70%, for the CockroachDB executable.