Learn Go Series (#7) - Strings, Runes, and Unicode

What will I learn

• What a Go string really is -- an immutable, read-only slice of bytes holding UTF-8;
• Why len(s) counts bytes, indexing gives you a byte, and how that bites you with non-ASCII text;
• The difference between a byte and a rune, and why for range over a string yields runes;
• Converting between string, []byte and []rune, and when each conversion is the right one;
• The strings, unicode/utf8 and unicode packages for the operations you actually need;
• strings.Builder, the efficient way to assemble a string in a loop.

Requirements

• A working modern computer running macOS, Windows or Ubuntu;
• An installed Go distribution (1.27 or newer, from go.dev/dl) -- tested against Go 1.27;
• Episodes 1-6, especially slices (episode 5) and the byte/rune aliases from episode 2;
• The ambition to learn Go programming.

Difficulty

• Beginner

Curriculum (of the Learn Go Series):

• Learn Go Series (#1) - Introduction to Go
• Learn Go Series (#2) - Variables, Types, Constants, and iota
• Learn Go Series (#3) - Functions and First-Class Functions
• Learn Go Series (#4) - Control Flow: if, switch, and the Only Loop
• Learn Go Series (#5) - Arrays, Slices, and Slice Internals
• Learn Go Series (#6) - Maps and Sets
• Learn Go Series (#7) - Strings, Runes, and Unicode (this post)

Learn Go Series (#7) - Strings, Runes, and Unicode

Text looks simple until it is not, and Go makes a deliberate choice that saves you from a whole category of Unicode bugs -- but only once you understand it. A Go string is not a sequence of characters; it is an immutable sequence of bytes that usually holds UTF-8-encoded text. That one fact explains why len sometimes gives a surprising answer, why indexing a string gives you a number, and why looping over one has two very different meanings. Get this one mental model right and text handling in Go becomes clear and fast; get it wrong and you will fight the same confusing bug over and over.

Here is the thing to hold on to, because it is the thread through the whole episode: most languages hide the encoding and pretend a string is "characters". That is comfortable right up until you paste in an emoji, a Chinese name, or a café with an accented e, and suddenly your length is wrong and your substring cuts a character in half. Go refuses to pretend. It shows you the bytes, and hands you a clean, explicit way to decode them into characters (runes) exactly when you ask. Coming from Python -- which is where most of my audience comes from -- that feels like extra work at first. By the end I hope it reads as honesty: no hidden magic, no quiet mangling. Having said that, let us clear last episode's exercises and then get precise about text.

Solutions to Episode 6 Exercises

Exercise 1 -- invert a map. Swap keys and values into a new map:

package main

import (
    "fmt"
    "sort"
)

func invert(m map[string]int) map[int]string {
    out := make(map[int]string, len(m))
    for k, v := range m {
        out[v] = k
    }
    return out
}

func main() {
    inv := invert(map[string]int{"a": 1, "b": 2, "c": 3})
    keys := make([]int, 0, len(inv))
    for k := range inv {
        keys = append(keys, k)
    }
    sort.Ints(keys)
    for _, k := range keys {
        fmt.Printf("%d -> %s\n", k, inv[k])
    }
}

The insight to note in a comment: this only works cleanly if the original values are unique. If two keys shared a value, the second one written would clobber the first in the inverted map, and you would silently lose an entry. Uniqueness is a precondition, not a detail.

Exercise 2 -- set intersection. Build a set from the first list, keep members of the second (skipping duplicates):

package main

import (
    "fmt"
    "sort"
)

func intersect(a, b []string) []string {
    set := make(map[string]struct{}, len(a))
    for _, w := range a {
        set[w] = struct{}{}
    }
    var out []string
    added := make(map[string]struct{})
    for _, w := range b {
        if _, ok := set[w]; ok {
            if _, dup := added[w]; !dup {
                out = append(out, w)
                added[w] = struct{}{}
            }
        }
    }
    sort.Strings(out)
    return out
}

func main() {
    fmt.Println(intersect([]string{"go", "rust", "zig"}, []string{"zig", "go", "c", "go"}))
}

This is the set idiom (map[string]struct{}) plus the collect-and-sort pattern from last episode, working together. The second added set guards against the duplicate "go" in the second list producing two entries in the output.

Exercise 3 -- group by length. A map of slices, keyed by len:

package main

import (
    "fmt"
    "sort"
)

func main() {
    words := []string{"go", "rust", "zig", "c", "ada", "java"}
    byLen := make(map[int][]string)
    for _, w := range words {
        byLen[len(w)] = append(byLen[len(w)], w)
    }
    lengths := make([]int, 0, len(byLen))
    for l := range byLen {
        lengths = append(lengths, l)
    }
    sort.Ints(lengths)
    for _, l := range lengths {
        fmt.Printf("%d: %v\n", l, byLen[l])
    }
}

The one line that matters is byLen[len(w)] = append(byLen[len(w)], w): appending to a possibly-nil slice is safe (episode 5), so grouping into buckets needs no first-sight special case. Now, text.

A string is bytes

A Go string is an immutable, read-only slice of bytes. len reports the number of bytes, not characters, and indexing with s[i] gives you the byte at that position -- a uint8, not a character. For plain ASCII the two happen to line up (every ASCII character is exactly one byte), which is precisely why the confusion is so easy to fall into: everything works fine in your tests until the first non-ASCII input arrives. The moment you have a multi-byte character the difference shows:

package main

import "fmt"

func main() {
    s := "héllo" // the é is two bytes in UTF-8 (0xC3 0xA9)

    fmt.Println("byte length:", len(s))            // 6, not 5
    fmt.Printf("s[0] = %d (%c)\n", s[0], s[0])     // 104 (h) -- a byte
    fmt.Printf("s[1] = %d\n", s[1])                // 195 -- the FIRST byte of é, not 'é'

    // s[0] = 'H' // ERROR: cannot assign to s[0] -- strings are immutable
}

len("héllo") is 6 because é takes two bytes. And s[1] is 195, the first byte of the two that encode é -- indexing a string is a byte operation, full stop. The commented line shows the other half of the rule: strings are immutable, so you cannot assign to s[i]. This immutability is not an arbitrary restriction, by the way -- it is what makes strings cheap to pass around and safe to share, because nobody can change a string out from under you (the compiler can even store string constants in read-only memory). When you genuinely need to edit text, you convert to []byte or []rune first, which we will do below.

Nota bene: UTF-8 is a variable-width encoding. ASCII characters take one byte, most Latin-with-accents and Cyrillic and Greek take two, most of the common CJK (Chinese, Japanese, Korean) characters take three, and things like emoji take four. So there is no fixed "bytes per character" number you can multiply by -- which is exactly why len counting bytes and indexing returning bytes surprises people who expect a character count.

for range yields runes

A rune is a Unicode code point -- Go's name for "a character" -- and it is stored in an int32 (in fact rune is just an alias for int32, exactly like byte is an alias for uint8, as we saw back in episode 2). The magic is that for range over a string does not walk bytes; it decodes the UTF-8 and hands you each rune along with the byte index where it started:

package main

import "fmt"

func main() {
    s := "héllo"
    for i, r := range s { // i is the byte index, r is the decoded rune
        fmt.Printf("byte %d: %c (code point %d)\n", i, r, r)
    }
    // the byte index jumps 0,1,3,4,5 -- é occupies bytes 1 and 2
}

So a string has two natural iterations, and this is the whole crux of the episode: index it with s[i] to walk bytes, or for range it to walk runes. The byte index in the range loop even jumps from 1 to 3, skipping the second byte of é, which is your visible proof that Go decoded a two-byte character as one rune. That i is genuinely useful, too -- because it is the byte offset, you can slice the original string at those boundaries (s[i:]) and know you are cutting on a character edge, not through the middle of one.

One more thing worth knowing about ranging: if the string contains bytes that are not valid UTF-8, the range loop does not crash. It yields the Unicode replacement character (U+FFFD, the little ?-in-a-diamond you have surely seen on a broken web page) for each bad byte and keeps going. Go chose "degrade gracefully" over "panic on bad input", which is the right call for text that came from the outside world.

Counting characters correctly

Because len counts bytes, counting actual characters needs utf8.RuneCountInString from the unicode/utf8 package. And when you want to index by character position rather than byte, convert the string to a []rune, which gives you one slot per code point:

package main

import (
    "fmt"
    "unicode/utf8"
)

func main() {
    s := "héllo, 世界"

    fmt.Println("bytes:", len(s))                        // 14
    fmt.Println("runes:", utf8.RuneCountInString(s))     // 9

    runes := []rune(s)          // one element per code point
    fmt.Printf("8th char: %c\n", runes[7]) // 世
    fmt.Printf("char count via slice: %d\n", len(runes))
}

Look at the two numbers: 14 bytes but 9 characters, because the two CJK characters cost three bytes each and the é costs two. []rune(s) allocates a slice with one entry per character, so runes[7] is genuinely the eighth character regardless of how many bytes came before it, and len(runes) agrees with RuneCountInString. For the common "how many characters" and "character at position N" questions on international text, these are the correct tools -- len(s) and s[i] would give you byte answers, and on non-ASCII text those are almost never what a human means by "length" or "the Nth character".

Converting between string, []byte and []rune

Three conversions cover almost everything. []byte(s) copies the bytes into a mutable slice (for editing, or for APIs that take bytes). []rune(s) gives one element per character (for character-level editing). And string(...) turns either back into a string. Because strings are immutable, every one of these conversions copies -- there is no way to get a mutable view onto a string's memory, and that is by design:

package main

import "fmt"

func main() {
    s := "hello"

    b := []byte(s) // a mutable copy of the bytes
    b[0] = 'H'
    fmt.Println(string(b)) // "Hello"
    fmt.Println(s)         // "hello" -- the original is untouched

    r := []rune("café")
    r[3] = 'X'
    fmt.Println(string(r)) // "cafX"
}

Editing b does not touch s, because []byte(s) made a copy -- immutability preserved. The rule of thumb for which conversion: use []byte when you are dealing with ASCII or raw bytes (reading a file, writing to the network, hashing), and []rune when you need to manipulate characters of arbitrary Unicode (reversing text, indexing by character, editing a specific letter). Do be aware these conversions allocate and copy, so in a hot loop you avoid converting back and forth needlessly -- convert once, do your work, convert back once. For a great many programs the cost is irrelevant, but it is the kind of thing that quietly matters when you are processing megabytes of text.

The strings package: the operations you actually need

You will rarely walk bytes by hand, because the strings package has the everyday operations: trimming, casing, splitting, joining, searching, replacing. They are UTF-8-aware where it matters and return new strings (immutability again -- nothing mutates the input):

package main

import (
    "fmt"
    "strings"
)

func main() {
    s := "  Go, Rust, Zig  "
    s = strings.TrimSpace(s)

    fmt.Println(strings.ToUpper(s))                 // GO, RUST, ZIG
    fmt.Println(strings.Contains(s, "Rust"))        // true
    fmt.Println(strings.Split(s, ", "))             // [Go Rust Zig]
    fmt.Println(strings.ReplaceAll(s, ", ", " | ")) // Go | Rust | Zig
    fmt.Println(strings.HasPrefix(s, "Go"))         // true
    fmt.Println(strings.Index(s, "Zig"))            // 9 (byte offset)
}

Split returns a []string, Join does the reverse, Fields (which we used last episode to split on runs of whitespace) is the whitespace-aware cousin, and Contains/HasPrefix/HasSuffix/Index cover searching. Learn to reach for this package first -- almost every string task you can think of has a clean one-liner here, and reaching for it in stead of hand-rolling a byte loop will save you both effort and bugs. (One gotcha to file away: Index returns a byte offset, not a rune offset, consistent with everything else we have seen -- indexing is always about bytes.)

strings.Builder: assembling a string efficiently

Because strings are immutable, building one with += in a loop reallocates every single time -- Go has to make a whole new string on each concatenation, copying everything accumulated so far. For a few pieces that is completely fine and you should not think twice. For many pieces it becomes quadratic and wasteful. strings.Builder accumulates into a growable buffer (the same amortised-doubling growth we saw with slices in episode 5) and produces the final string once, at the end. It even satisfies the io.Writer interface, so fmt.Fprintf can write straight into it:

package main

import (
    "fmt"
    "strings"
)

func main() {
    var b strings.Builder
    for i := 1; i <= 3; i++ {
        fmt.Fprintf(&b, "line %d\n", i) // Builder is an io.Writer
    }
    fmt.Print(b.String())
}

Notice we pass &b (a pointer) to Fprintf -- the Builder must not be copied after you start writing to it, and passing a pointer is how you avoid that. For a handful of concatenations, + is perfectly clear and idiomatic; do not reach for a Builder to glue two strings together. But for building up a large string in a loop -- a report, generated code, a big response body -- strings.Builder is the right tool and sidesteps the repeated allocations that plain += would rack up. That io.Writer satisfaction is a small preview of something big: in a couple of episodes we will see how Go's interfaces let a Builder, a file, and a network connection all be written to with the exact same code. ;-)

The unicode package: classifying and transforming runes

Once you are iterating runes, you often want to ask questions about each one -- is it a letter, a digit, whitespace, punctuation? -- or transform it, like upper-casing. That is what the unicode package is for, and it operates on a single rune at a time, which pairs perfectly with a for range loop:

package main

import (
    "fmt"
    "unicode"
)

func main() {
    s := "Go 1.27! 世界"
    var letters, digits, spaces, other int
    for _, r := range s {
        switch {
        case unicode.IsLetter(r):
            letters++
        case unicode.IsDigit(r):
            digits++
        case unicode.IsSpace(r):
            spaces++
        default:
            other++
        }
    }
    fmt.Printf("letters=%d digits=%d spaces=%d other=%d\n",
        letters, digits, spaces, other)
    fmt.Printf("%c -> %c\n", 'a', unicode.ToUpper('a')) // a -> A
}

The important detail is that unicode.IsLetter counts 世 and 界 as letters too, not just the ASCII a-z -- it is genuinely Unicode-aware, consulting the same character tables your operating system uses. That is a world apart from the naive r >= 'a' && r <= 'z' check, which quietly fails on every language that is not English. If you find yourself writing character-classification logic by hand, stop and check unicode first; the function you want almost certainly exists (IsLetter, IsDigit, IsNumber, IsSpace, IsPunct, IsUpper, ToUpper, ToLower, ToTitle, and quit some more).

The same idea in Python

Python 3 strings are sequences of Unicode code points, so len("héllo") is 5 and s[1] is "é" -- the character view, not the byte view. Go chose the byte view (with rune decoding on demand) for zero-copy efficiency and explicit control over encoding. It is a genuine philosophical split, and neither side is simply "right":

s = "héllo"
print(len(s))          # 5 -- characters, not bytes
print(s[1])            # "é" -- a character

b = s.encode("utf-8")  # bytes, like Go's []byte(s)
print(len(b))          # 6

for ch in s:           # iterates characters directly
    print(ch, end=" ")

Python hides the bytes and shows you characters, which is wonderful for getting text handling right without thinking about encoding at all -- until you do need the bytes (writing to a socket, hashing, reading a binary file) and have to .encode() back down. Go shows you bytes and lets you decode to runes when you ask, which is a touch more work up front but means the encoding is never a mystery and never copies behind your back. Once you know which view you are in -- bytes with len and indexing, runes with for range and []rune -- Go's model is precise and fast, and it will never quietly mangle a multi-byte character on you. That predictability is the whole point.

Exercises

1. Reverse by runes. Write reverse(s string) string that reverses a string correctly for Unicode by converting to []rune, reversing the slice, and converting back. Test it on "héllo" and confirm the é survives intact (a naive byte reverse would corrupt it into invalid UTF-8).

2. Count vowels, Unicode-safe. Write a function that counts the vowels in a string using for range (so it iterates runes), comparing each rune against the set aeiou (upper and lower). Explain in a comment why ranging is safer here than indexing with s[i].

3. A CSV field splitter. Given a line like "go,rust,,zig", use strings.Split on the comma to produce the fields, then print each field's index and value, showing that the empty field between the two commas comes through as an empty string (not skipped).

What we learned

• A Go string is an immutable, read-only slice of bytes holding UTF-8; len(s) counts bytes and s[i] is a byte, so both surprise you on non-ASCII text;
• A rune is a Unicode code point stored in an int32; for range over a string decodes UTF-8 and yields runes with their starting byte index, and it degrades gracefully to U+FFFD on invalid bytes;
• Count characters with utf8.RuneCountInString, and index by character via []rune(s) -- not len and s[i], which are byte operations;
• []byte(s), []rune(s) and string(...) convert between the views and always copy (strings are immutable), so editing the copy never touches the original;
• The strings package has the everyday operations (trim, case, split, join, search, replace) as clean one-liners returning new strings, and the unicode package classifies and transforms individual runes in a genuinely Unicode-aware way;
• strings.Builder assembles a string in a growable buffer -- the efficient choice for building large strings in a loop, and it is an io.Writer.

Next episode we start building our own types in earnest: structs, the methods you attach to them, and the single most important early decision -- value receivers versus pointer receivers, and what it means for mutation and copying. Bring the editor, there is real code to write. See you there.

Veel plezier met tekst, en tot de volgende! ;-)

@scipio