ENCODING

Why verification depends on exact representation, even when two messages appear to say the same thing.

A digital signature algorithm does not receive a document, a credential or a transaction in the way a person understands those objects. It receives a sequence of bytes, usually through a hash function, and produces a result bound to that sequence.

This distinction sounds elementary. In implementation work, it is one of the most persistent sources of failed verification and unsafe protocol design.

Consider these two JSON objects:

{"amount":100,"currency":"UYU"}

{"currency":"UYU","amount":100}

Most JSON applications interpret them as the same mapping. Their property order differs, but their apparent meaning does not. Their UTF-8 representations are nevertheless different. The SHA-256 digest of the first is:

8cead320a7772b16a00c2845b4772c3391c8ae1bb55170dee92175a4fbc9cefc

The digest of the second is:

8ad324c77dc614b62b093b8bf4b77e7e2bb61261fdb96f518e43a1545ba0e3c2

The hash function has not misunderstood the data. It has done exactly what it was designed to do: detect that the byte sequences are different.

If one system signs the first representation and another reconstructs the second before verification, the signature fails. Choosing a stronger algorithm will not help. The disagreement exists before the cryptographic primitive begins.

One data model, many representations

A structured object normally passes through several layers:

1. an application data model;
2. a serialization such as JSON or CBOR;
3. a protocol structure that selects protected fields;
4. the exact byte string supplied to the signature operation.

Confusion arises when these layers are treated as interchangeable.

At the data-model level, a map may not assign meaning to property order. At the serialization level, order, whitespace and number formatting produce different bytes. At the protocol level, some fields may be protected while others remain outside the signature. At the application level, a valid signature may still be attached to a value that is expired, addressed to another audience or unsuitable for the requested operation.

The signature only answers a precise cryptographic question: does this signature validate over these bytes with this key and algorithm?

Everything included in “these bytes” must be defined by the protocol.

Three ways to obtain the signed bytes

Applied protocols generally follow one of three strategies.

Preserve the original representation

The signer protects the exact bytes being transported, and the verifier uses those same bytes without parsing and reconstructing them first.

This is conceptually simple and avoids canonicalization. It also means that any harmless-looking transformation invalidates the signature. Changing whitespace, line endings or character escapes creates a different message.

The strategy works when the transport preserves the payload exactly and when the application can retain the original byte sequence through verification.

Define a canonical representation

The protocol defines one deterministic serialization for every permitted data object. Signer and verifier independently produce that representation and therefore obtain the same bytes.

Canonicalization is attractive when applications need to process or display structured data in different forms. It is also protocol logic. Every rule for strings, numbers, map ordering and unsupported values must match across implementations.

A partially specified canonicalization procedure is worse than none. It creates confidence that two systems will agree while leaving exceptional values to platform defaults.

Construct a separate signing structure

The protocol builds a dedicated object containing the payload and the context that must be protected. That structure is serialized under defined rules and becomes the input to the signature.

COSE follows this pattern with its Sig_structure. The object includes a context string, protected attributes, external authenticated data and the payload. The final signature is calculated over the CBOR encoding of that structure, not simply over the application payload as it happens to appear on screen.

This approach makes an important fact visible: the signed bytes may not be identical to the transmitted payload. They may include protocol context that is reconstructed during verification.

JWS preserves bytes by encoding them

JSON Web Signature does not generally sign a parsed JSON object. In its common compact form, JWS constructs the signing input as:

BASE64URL(UTF8(protected header)) || "." || BASE64URL(payload)

The base64url encoding preserves the byte sequence of the protected header and payload inside an ASCII representation. A verifier reconstructs that precise signing input and verifies the signature against it.

If the payload happens to contain JSON, parsing it and serializing it again before verification is a mistake. The new serialization may express the same values while producing different payload bytes.

JWS therefore avoids the need to canonicalize the payload for its own signature calculation. It does not make equivalent JSON serializations interchangeable. It protects the particular serialization carried by the JWS.

This distinction matters with detached payloads as well. If the payload travels through another channel, the verifier must possess the exact byte sequence expected by the JWS construction. “The same object” is not precise enough.

Canonical JSON is a specific protocol

When a design genuinely needs different parties to derive the same byte representation from JSON data, it can adopt a defined canonicalization scheme.

RFC 8785, the JSON Canonicalization Scheme, defines such a procedure. It constrains input to the I-JSON model, rejects duplicate property names, removes whitespace between tokens, specifies serialization of primitive values, sorts object properties recursively and emits UTF-8.

Numbers deserve particular attention. Values that appear equivalent to an application may be emitted differently by serializers, rounded when converted to IEEE 754 double precision or rejected when they fall outside the permitted model. A system that handles large integers cannot assume every JSON implementation will preserve them as numbers.

Unicode introduces another subtle boundary. RFC 8785 does not normalize Unicode strings. Components must preserve string data as received. Two strings can look identical to a reader while using different Unicode code-point sequences. For example, an accented character may be represented as one precomposed code point or as a base character followed by a combining mark.

Canonicalization cannot be added as an informal instruction to “sort the fields.” It must be named, versioned and tested as part of the protocol.

Duplicate names create parser disagreements

RFC 8259 says that names within a JSON object should be unique. It also notes that software behaves unpredictably when they are not. Some parsers retain the last value, some retain the first, some report every occurrence and others reject the object.

Consider a signed payload containing two role properties with different values. One component may validate the signature over the received bytes and then expose the first value to its authorization logic. Another may expose the last. Both can agree that the signature is mathematically valid while disagreeing about what the signed message authorizes.

For security protocols, duplicate names should be rejected before application decisions are made. A signature does not repair ambiguity in the parser.

The same principle applies to permissive number syntax, invalid Unicode, unknown critical fields and alternative date representations. Strict input profiles reduce the semantic space that a signed byte sequence can occupy.

CBOR is compact, not automatically unique

CBOR is designed for compact representation and implementation in constrained environments. Its data model can also have more than one valid encoding for the same value.

An integer may be encoded with more bytes than necessary. A collection can use a definite or indefinite length. Map entries may appear in different orders. A floating-point value may have several representations that preserve its value.

RFC 8949 defines preferred serialization and core deterministic encoding requirements. These include using the shortest available representations and deterministic map ordering. A protocol can adopt these rules and add further restrictions required by its own data model.

Variation-tolerant decoding and deterministic encoding serve different purposes. A general decoder may accept several valid forms for interoperability. A protocol that produces bytes for hashing or signing may require exactly one form. Implementations should not assume that because a decoder accepts two encodings, those encodings are cryptographically interchangeable.

COSE signs a reconstructed structure

COSE makes the relationship between protocol context and signed bytes explicit. For a signature, the Sig_structure can contain:

[
  context,
  body_protected,
  sign_protected,
  external_aad,
  payload
]

The fields depend on the COSE structure in use. The protected header maps are carried as encoded byte strings, the external authenticated data is supplied by the application, and the full payload participates even when it is transported separately.

Signer and verifier must build the same structure and encode it according to the applicable COSE restrictions. A mismatch in the context string, protected header bytes, external data or payload produces a different ToBeSigned value.

This is not incidental formatting. Each component prevents a class of substitution. The context distinguishes signature structures. Protected attributes bind cryptographic parameters. External authenticated data can bind information supplied outside the COSE object. The payload binds the application content.

A library that reports “invalid signature” may therefore be describing a construction mismatch rather than a problem with the public key or algorithm.

Hashing does not solve representation

It is sometimes proposed that systems avoid these issues by signing a hash instead of the original data. Most signature schemes already incorporate hashing, directly or through their specified construction.

Hashing moves a byte sequence into a fixed-size value. It does not decide which byte sequence represents the application object. Two equivalent-looking serializations still produce different digests, as the opening example shows.

The representation must be settled first. Only then can hashing and signing provide stable results.

Verification is more than a valid signature

A robust verification path separates cryptographic validity from application acceptance.

It should:

1. retain the received bytes needed by the signature format;
2. parse the outer structure under strict limits;
3. reject ambiguous or disallowed representations;
4. reconstruct the protocol-defined signing input exactly;
5. select the key and algorithm under explicit policy;
6. verify the signature;
7. validate the signed content, including type, purpose, audience, time and required fields;
8. make the application decision only after those checks succeed.

The order can vary with the protocol, but unverified content must not acquire authority during parsing. Key identifiers and algorithm declarations are inputs to policy, not instructions that a verifier should trust without restriction.

Successful signature verification means that the protected byte sequence has not changed relative to the signing operation and that the corresponding private key produced the signature, subject to the properties of the algorithm and key management. It does not prove that every relevant fact was included, that the signer was authorized for the action or that two applications interpret every field identically.

Make the byte boundary observable

Cryptographic tooling often displays the decoded object and the final signature while hiding the exact signing input. That makes the most important implementation boundary difficult to inspect.

A useful diagnostic tool should expose:

• the received payload bytes;
• the decoded data model;
• the protected fields;
• the constructed signing structure;
• the final byte sequence or digest passed to verification;
• the interpretation used by application policy.

Test vectors should include intermediate values, not only a finished token and an expected pass result. Negative vectors are equally important: reordered maps, duplicate names, altered Unicode sequences, non-preferred CBOR encodings and mismatched external data.

When two implementations disagree, comparing their final signature bytes is too late. The first divergence usually appears in serialization or protocol construction.

The boundary worth documenting

Applied cryptography begins before the algorithm call.

A protocol must define how application meaning becomes structured data, how that data becomes bytes and which contextual bytes join it inside the signature input. Verification must travel through the same definitions in reverse without silently changing representation or interpretation.

The signature protects the bytes it was given. The engineering task is to ensure that those bytes express exactly one intended statement.

---

References

1. RFC 8259: The JavaScript Object Notation Data Interchange Format
2. RFC 7515: JSON Web Signature
3. RFC 8785: JSON Canonicalization Scheme
4. RFC 8949: Concise Binary Object Representation
5. RFC 9052: CBOR Object Signing and Encryption, Structures and Process
6. RFC 9053: CBOR Object Signing and Encryption, Initial Algorithms