contextual-encoder after done
five releases and a bunch of breaking changes after we called it
done.cargo add contextual-encoder
in june i introduced contextual-encoder and said i considered it done. no new contexts unless the need arises. and i stayed true to that. still, since then it has had five more releases, 0.7.0 to 0.11.0, with a bunch of breaking changes between them. none of them added a context, and yet they were all necessary.
the june post made the argument that you should encode output for where it goes instead of scrubbing input. i still think that.
what the work since then made clear is how much “where it goes” matters. a sink is just not a string with some special characters, and i can’t just eyeball it and say “looks right”.
context needs boundary
the simplest example was the css
url report. for_css_url
left parentheses alone. inside an unquoted url(...), an
attacker-controlled ) closes the url token, and the rest of
the input can become a declaration. we had an encoder for the context,
but its rules did not enforce the right boundaries.
the more interesting failures were at the edges of encoded values. a
css hex escape can consume up to six hex digits. if the encoder ends its
output with \3c and the caller appends a, the
parser reads \3ca, a different character. the encoder now
terminates that final escape with a separator space. looking ahead
inside the input was not enough, because the next character could come
from the template.
template literals had the same problem with ${. we first
escaped a trailing $, so a { written after the
value could not open an interpolation. that still left the other edge:
trusted text ending in $, followed by an untrusted value
starting with {7*6}.
// before: trusted "total: $" followed by encoded "{7*6}"
`total: ${7*6}` // evaluates to "total: 42"
// after: the leading brace is escaped
`total: $\x7b7*6}` // evaluates to "total: ${7*6}"that fix, and the latest xml comment and cdata fixes, are in the very latest release.
i think this is the useful class of bugs to remember: the encoder only sees its input, but its output participates in a larger grammar. testing two encoded writes next to each other also misses cases where one side is raw trusted template text. nothing is ever as easy as it seems.
sometimes there is more than one parser
inside a <script> element, the html tokenizer gets
to the text before the javascript engine does. escaping /
stopped a literal </script>, but an input containing
<!--<script> could still put the html tokenizer
into its double-escaped script state. the template’s real closing
</script> then failed to close the element. the
script encoders now escape < too.
xhtml adds another variation: its xml parser resolves character
references before javascript sees them. a literal `
in untrusted input can arrive at the javascript parser as a backtick. for_js_template
therefore has to escape & as well.
not every failure was an injection. xml 1.1 normalizes raw NEL and
line separator characters to line feeds, so preserving them requires
character references. for_rust_char
could escape every character correctly and still produce an invalid
literal from ab1. these are different
failures, but they all require checking what the consumer actually
accepts and reads.
as ever: fuzz it
this all made me break down and write a differential fuzzing harness in a separate crate. it generates inputs, encodes them, inserts the output into a small document or literal, and runs an independent parser over it.
the checks depend on the sink. html5ever checks for
unexpected elements and attributes. cssparser checks that
quoted css values remain one string token. quick-xml reads
xml content, attributes, comments and cdata. serde_json
parses json strings, and the boa javascript engine evaluates string and
template literals.
the inputs combine a fixed regression corpus, fragments such as
</script>, ${ and ]]>,
and random unicode scalar values. fixed seeds make a failure
reproducible. this is a small deterministic fuzzing run, with parser
results as the oracle. comparing bytes against another encoder is useful
for conformance, but two encoders can agree on an output that the sink
reads incorrectly.
there is a limitation here i want to call out, though: the current harness mostly wraps a value directly in its delimiters. it does not yet vary trusted text on both sides, and its parser coverage does not span every encoder. the boundary sweeps recorded in the recent fixes utilize exactly that surrounding text.
so the property i want to check is that the whole composed sink keeps its structure and reads back the intended value. the fuzzing is useful when the oracle and the surrounding template fulfill that contract.
the harness stays outside the library’s workspace and has its own lockfile. its parser dependencies can be large and contain unsafe code without entering the published crate’s dependency graph or its unsafe budget. it’ll not be published on crates.io, because why would i do that?
not every useful crate needs to be published.
takeaway
the fixes themselves were largely written by our agents, and two of them started with outside reports from hey-jj (thanks, kind stranger!). i didn’t find them by staring at the code and thinking real hard.
as is often the case, a bit of fuzzing replaces a lot of thinking, and does a better job. the bitter lesson of application security, i suppose.
it’s still done, in the june-announcement sense. the list of contexts isn’t growing. we’re just making the existing invariants reliable.
also reported by hey-jj.
for_rust_char_checkednow returnsNoneunless the input is exactly one unicode scalar value. the original function documents that precondition.↩︎