HTTP gets a QUERY method so complex searches can stop pretending to be POST

"Idempotent" may be jargon, but the term performs an important job in HTTP as a hall pass that gives reverse proxies and gateways the go-ahead to cache complex query responses and automatically retry failed requests.

HTTP has long allowed automatic retries for idempotent methods, but complex queries are often sent using POST, which intermediaries cannot safely assume is retryable. Developers have worked around that limitation for decades. 

The Internet Engineering Task Force (IETF) has published a new HTTP request method, QUERY (RFC 10008), joining familiar methods including GET, POST, PUT, and PATCH.

In development since 2021, the QUERY request method provides a way for an HTTP client to make an idempotent request to an HTTP server. An idempotent request has the same intended effect whether it is sent once or multiple times (so retrying it should not charge a user's credit card again).

The specification defines QUERY as safe and idempotent, and the solution was surprisingly simple. 

"Thanks to QUERY, we finally have functional HTTP caching for complex requests. Proxies, CDNs, and browsers can now cache requests with a body. This is huge for performance," writes developer Elie Treport in a recent blog post. 

Query operations have traditionally used the GET method. HTTP defines GET as safe and idempotent, as it made no changes to the server itself, but GET becomes awkward when the query data is too large or complex for a URI.

The idea behind GET was to load all the necessary query parameters onto the form's URL, resulting in a very long string that the browser sent to the server. Many search services still work this way, appending query parameters to the URL after a question mark. Those URLs can expose sensitive data through browser histories, server logs, and bookmarks.

Festooned with nested filters, sort rules, date ranges, and other database flotsam, GET URLs can become unwieldy. HTTP recommends support for URIs of at least 8,000 octets, but there is no universal maximum, so an oversized URL may be rejected by any of the systems it passes through.

Long query strings are also a pain to read and debug.

Faced with these Franken-URLs, many developers turned to POST instead. POST can carry query data in the request body, but its semantics do not tell intermediaries that the operation is safe and idempotent. The method is more commonly associated with operations such as submitting form data, uploading a PDF, or creating and modifying resources.

For a basic HTML form, switching <form method="GET"> to <form method="POST"> moves the encoded form data from the URL into the request body.

Crafty developers soon began placing complex query payloads, often encoded as JSON, in POST request bodies and having their applications process the responses. Many APIs adopted the pattern; GraphQL, for example, commonly uses POST for queries, although it also supports GET.

It was a hack, though. This was not what POST was designed for. HTTP defines POST as neither safe nor idempotent by default. A POST request could change the server's state.

And this is why internet networking software treats POST far more delicately than GET. Intermediaries generally cannot reuse POST responses for later POST requests or automatically retry POST without knowing the operation is idempotent. If the message fails, the browser or gateway will not resend it, necessitating intervention from either the user or the app.

QUERY's long road to adoption

So basically, QUERY combines request content similar to POST with explicitly safe and idempotent query semantics. No longer will the query URL need to be appended beyond recognition, but, like POST, apps and browsers will get a dedicated space to put their complex query data.

Unlike POST, QUERY is a read-only operation, and hence receives the IETF's blessing as idempotent, freeing up HTTP clients and intermediaries to cache and resend QUERY requests after a connection failure. A QUERY request does not ask or expect the server to change the state of the target resource. It's just there to ask a question.

Cloudflare and Akamai engineers co-wrote RFC 10008. Both companies provide edge caching for large clients. German internet engineering firm greenbytes also contributed.

As a new standard, QUERY still has limited support. The HTML forms standard still only understands GET and POST for ordinary form submission, so it will need to be updated even before the browsers get on board. The good news is that the Web Hypertext Application Technology Working Group is already on the case.

However, a whole ecosystem of network software still doesn't understand QUERY and may reject requests using an unfamiliar method. Reverse proxies, load balancers, content delivery networks, API gateways, firewalls, and web frameworks will all need to be updated.

"The pattern one would expect is the same seen with other HTTP methods and headers that became standard over the last twenty years: first server-side adoption and dev tools, then consolidation in frameworks, and finally, more slowly, native browser support and JavaScript APIs like fetch()," wrote open source developer Daniele Teti in a blog post. 

Teti noted that Node.js is adding support in its HTTP module, and the Go programming language is ahead of the game because it can already send custom HTTP methods. Elsewhere, the PHP framework Laravel is already ingesting QUERY.

But, as with IPv6, the IETF faces an uphill battle to get the HTTP ecosystem on the same page. ®