Text Compression with Web Standards

snippet

by FND

An ill-advised return to the rabbit hole of personal web apps and client-side secrets left me with a series of usability challenges. Naturally, those could only be addressed by fumbling with byte streams.

For environmental reasons *scowls at mobile operating systems*, I felt the need to encode non-trivial amounts of state in the URL.1 That inevitably led to a desire to at least keep the URL as short as possible by smooshing redundant bits.

Fortunately, browsers now expose their compression routines to JavaScript. These APIs are designed around the assumption of network traffic, so we have to deal with streams, which isn’t always intuitive.

All we want here is a function that accepts a string and returns a compressed byte representation:

/**
 * @param {string} txt
 * @returns {Promise<Uint8Array>}
 */
function compress(txt) {
    let stream = txt2stream(txt).
        pipeThrough(new TextEncoderStream()).
        pipeThrough(new CompressionStream("gzip"));
    return stream2bytes(stream);
}

In our case, we can use toBase64 (where supported2) to convert those bytes to text again so we can stuff that into the URL.

Now we just need to implement those conversion helpers. Let’s start with turning our string into a stream so we can start piping data through the compressor:

/** @param {string} txt */
function txt2stream(txt) {
    return new ReadableStream({
        start(controller) {
            controller.enqueue(txt);
            controller.close();
        },
    });
}

Kinda awkward, but tests are green. Eventually we’ll be able to just use ReadableStream.from instead.

Next we want to turn streamed data into a byte array:

/**
 * @param {ReadableStream} stream
 * @returns {Promise<Uint8Array>}
 */
async function stream2bytes(stream) {
    let reader = stream.getReader();
    let res, chunk;
    while (chunk = await reader.read()) {
        if (chunk.done) {
            break;
        }

        res = res ? combine(res, chunk.value) : chunk.value;
    }
    return res ?? new Uint8Array();
}

/**
 * @param {Uint8Array} first
 * @param {Uint8Array} second
 */
function combine(first, second) {
    let res = new Uint8Array(first.length + second.length);
    res.set(first);
    res.set(second, first.length);
    return res;
}

That seems even more cumbersome, but I guess we’ll have to live with it? Well, with those aforemention network assumptions in mind, we can apply one weird trick (thanks, foxy Jake):

async function stream2bytes(stream) {
    let blob = await new Response(stream).blob();
    return new Uint8Array(await blob.arrayBuffer());
}

In fact, it gets even simpler, so we might as well inline it:

function compress(txt) {
    let stream = txt2stream(txt).
        pipeThrough(new TextEncoderStream()).
        pipeThrough(new CompressionStream("gzip"));
    return new Response(stream).bytes();
}

Resolving Base64-encoded data from the URL is left as an exercise for the reader.