cyberwitchery lab
← log

contextual-encoder after done

five releases and a bunch of breaking changes after we called it done.
cargo add contextual-encoder

in june i introduced contextual-encoder and said i considered it done. no new contexts unless the need arises. and i stayed true to that. still, since then it has had five more releases, 0.7.0 to 0.11.0, with a bunch of breaking changes between them. none of them added a context, and yet they were all necessary.

the june post made the argument that you should encode output for where it goes instead of scrubbing input. i still think that.

what the work since then made clear is how much “where it goes” matters. a sink is just not a string with some special characters, and i can’t just eyeball it and say “looks right”.

context needs boundary

the simplest example was the css url report. for_css_url left parentheses alone. inside an unquoted url(...), an attacker-controlled ) closes the url token, and the rest of the input can become a declaration. we had an encoder for the context, but its rules did not enforce the right boundaries.

the more interesting failures were at the edges of encoded values. a css hex escape can consume up to six hex digits. if the encoder ends its output with \3c and the caller appends a, the parser reads \3ca, a different character. the encoder now terminates that final escape with a separator space. looking ahead inside the input was not enough, because the next character could come from the template.

template literals had the same problem with ${. we first escaped a trailing $, so a { written after the value could not open an interpolation. that still left the other edge: trusted text ending in $, followed by an untrusted value starting with {7*6}.

// before: trusted "total: $" followed by encoded "{7*6}"
`total: ${7*6}` // evaluates to "total: 42"

// after: the leading brace is escaped
`total: $\x7b7*6}` // evaluates to "total: ${7*6}"

that fix, and the latest xml comment and cdata fixes, are in the very latest release.

i think this is the useful class of bugs to remember: the encoder only sees its input, but its output participates in a larger grammar. testing two encoded writes next to each other also misses cases where one side is raw trusted template text. nothing is ever as easy as it seems.

sometimes there is more than one parser

inside a <script> element, the html tokenizer gets to the text before the javascript engine does. escaping / stopped a literal </script>, but an input containing <!--<script> could still put the html tokenizer into its double-escaped script state. the template’s real closing </script> then failed to close the element. the script encoders now escape < too.

xhtml adds another variation: its xml parser resolves character references before javascript sees them. a literal &#96; in untrusted input can arrive at the javascript parser as a backtick. for_js_template therefore has to escape & as well.

not every failure was an injection. xml 1.1 normalizes raw NEL and line separator characters to line feeds, so preserving them requires character references. for_rust_char could escape every character correctly and still produce an invalid literal from ab1. these are different failures, but they all require checking what the consumer actually accepts and reads.

as ever: fuzz it

this all made me break down and write a differential fuzzing harness in a separate crate. it generates inputs, encodes them, inserts the output into a small document or literal, and runs an independent parser over it.

the checks depend on the sink. html5ever checks for unexpected elements and attributes. cssparser checks that quoted css values remain one string token. quick-xml reads xml content, attributes, comments and cdata. serde_json parses json strings, and the boa javascript engine evaluates string and template literals.

the inputs combine a fixed regression corpus, fragments such as </script>, ${ and ]]>, and random unicode scalar values. fixed seeds make a failure reproducible. this is a small deterministic fuzzing run, with parser results as the oracle. comparing bytes against another encoder is useful for conformance, but two encoders can agree on an output that the sink reads incorrectly.

there is a limitation here i want to call out, though: the current harness mostly wraps a value directly in its delimiters. it does not yet vary trusted text on both sides, and its parser coverage does not span every encoder. the boundary sweeps recorded in the recent fixes utilize exactly that surrounding text.

so the property i want to check is that the whole composed sink keeps its structure and reads back the intended value. the fuzzing is useful when the oracle and the surrounding template fulfill that contract.

the harness stays outside the library’s workspace and has its own lockfile. its parser dependencies can be large and contain unsafe code without entering the published crate’s dependency graph or its unsafe budget. it’ll not be published on crates.io, because why would i do that?

not every useful crate needs to be published.

takeaway

the fixes themselves were largely written by our agents, and two of them started with outside reports from hey-jj (thanks, kind stranger!). i didn’t find them by staring at the code and thinking real hard.

as is often the case, a bit of fuzzing replaces a lot of thinking, and does a better job. the bitter lesson of application security, i suppose.

it’s still done, in the june-announcement sense. the list of contexts isn’t growing. we’re just making the existing invariants reliable.


  1. also reported by hey-jj. for_rust_char_checked now returns None unless the input is exactly one unicode scalar value. the original function documents that precondition.↩︎