Overlong

In the course of the excursion which resulted in reference 1, I got into the coding for Unicode characters in UTF-8, as described at reference 2, and the problems arising from something called overlong encoding.

UTF for Unicode Transformation Format, UTF-8 being the most widely used iteration.

For reasons which I do not yet understand, it seemed to take me a long time – perhaps as much for what passes these days as a couple of working days – to get to a reasonable understanding of the matter, enough to satisfy my curiosity. Perhaps I would have done better had I resorted to Gemini rather earlier than I did.

The hexadecimals of reference 4 are perhaps a good place to start, which use a single, printable and displayable character – 0 thru 9 and A thru F – to code for the sixteen combinations of four binary digits or bits. So no spaces or other funny characters to confuse matters. Hexadecimals can vary in length from one character up and there are lots of ways of marking them, including: 

• U+ for Unicode, as in the snap above

• 0x in C and related programming languages

• % in URLs, to mark hex pairs, covering the second batch of 128 ASCII characters. ASCII being first off the blocks on this one. As in ‘https://fr.wikipedia.org/wiki/F%C3%A9licit%C3%A9_Herzog’. Sorting this one out is left as an exercise for the reader.

They are also a very convenient way of getting bits and bit streams down on paper, streams which are a bit indigestible in their raw state.

For present purposes, we assume that our data is organised as a stream of eight bit bytes and that we do not need to worry about how the computer knows where in a stream of bits the byte boundaries are. A lower layer matter with which we need not concern ourselves.

Thus armed, I got stuck into the figure above.

Which expresses a neat solution to the problem of coping with a very large character set without the space overheads of fixed length character encoding – in the English speaking world, you can do a lot with 100 characters, each of which can be coded in just one eight bit byte, while meeting all reasonable requirements might mean 1,000,000 characters – and without putting too much load on encoding and decoding processes. And remaining reasonably robust under the stress of entry, transmission and exit.

A one byte character has seven data bits and the coding coincided with the ANSII standard coding. There is too much stuff out there to be meddling with it.

A two byte character has eleven data bits giving 2,048 more codes, with four bits being lost to multi-byte overheads.

A three byte character has sixteen data bits giving 65,536 more codes. Another three bits being lost to multi-byte overheads.

A four byte character has twenty one data bits giving 1,114,112 more codes. Another three bits being lost to multi-byte overheads.

The important idea, which I read, but did not really grasp, was that all these bits encode a number, a code point in Unicode. With the least significant digit last, far right in most diagrams. Bits which might or might not include leading zeros, which last do not affect the value of the number - usually, but not always, with a leading zero in the most significant slot.

One consequence of this is that there is more than one way to encode a short number. One just adds some leading zeros and jumps from two bytes to three bytes, or whatever. 

Lots of old or sloppy computer programs fail to take proper account of this, a failure which can lead to obscure but real vulnerabilities. And there is a lot of legacy code out there.

This is known in the trade as the problem of overlong encoding, for which I started at reference 5. Which pretty much says it all, but I failed to compute. So I had a go at reference 6, which seemed much better at first. But I was making heavy weather of it all.

Part of the confusion was the use of hex codes and I found it hard to keep track of whether the codes were for the UTF-8 encoded bytes or the binary numbers that were being coded, numbers which map in a trivial way onto Unicode code points – with U+00E9 or No.233, the accented letter ‘é’, being an example of such a code point. Which is coded by UTF-8 into two bytes as 0xC3A9.

Eventually, I printed off reference 3 and armed with that, things started to move a bit faster. At which point I started to use Gemini to confirm my growing understanding of the matter.

It was interesting to find out about how the past has determined the future. The engineers had to build on what they had got, what was out there in computers and on the Internet: starting with a clean sheet was not an option. One was stuck with somehow bodging the past into the future – and I dare say this is the root cause of many of our security problems today.

I associate to finding much the same thing when attending a course about lower level communications protocols, back in my days in IT

In the olden days the Internet was a just tool to move (mainly scientific) information about efficiently, and no-one gave much thought to the possibility that hacking into all this might be of interest to the bad guys. But if one was starting over, one would not start where they did!

 Which understanding I have attempted to encapsulate in the figure above.

Character count

One often wants to know the character count of a string of bytes. In the olden days, this was always the same as the byte count and was easy. Now it depends where one is on the processing chain from the outside world to the inside world.

A further complication is that the character count is often very close to the byte count and careless programming may go undetected for some time. Which may include the inclusion of legacy code from before the days when this was an issue.

So not exactly a flaw in the code that hackers can exploit in the ordinary way, but it is a weakness which can lead to trouble.

Problems caused by overlong coding

It seems that something hackers like to do is introduce the ‘/’ character (U+002F) for which, as is shown in the snap above, there are four codings, three overlong. This character is important when a hacker is trying to escape from some container into the wider file structure of its host. Testing for this character is much more fiddly when you have four versions to check for.

Reference 5 tells of various ways in which this might cause problems. Quoting with light editing:

• In C and its friends, designed before UTF-8 was invented, in any situation where UTF-8 strings are processed using functions designed for ASCII, there is a possibility that over-long characters can create a security weakness. The only solution is to remove all over-long characters from the input, using a processor that is fully aware of the subtleties of UTF-8 encoding

• In Java, problems like the above are far less likely to arise because Unicode strings and characters are first-class language elements. But not impossible if, for example, we are using an XML parser, that exposes the low-level byte stream for pre-processing

• As we have seen, over-long UTF-8 encoding can be a problem in any development scenario where we need to sanitize text input, and we use methods that are appropriate for ASCII strings, rather than Unicode. Any language is potentially at risk, but languages that treat text strings as byte arrays are most vulnerable: even in a programming language that has native Unicode support, we still have to be careful about the use of external libraries, particular those written in C.

The mitigation is careful programming, with careful quality control and testing.

Reference 6 comes from the world of pentesting, aka penetration testers, that is to say people you hire to find out how easy it is to break into your system. In my limited experience of these matters, is was quite disturbing how many elementary errors the first round of such testing uncovers. Things like not changing the low-level – but all powerful – passwords set during manufacture of servers. Quality control and testing notwithstanding.

‘… However, if the parser accepts and decodes invalid overlong UTF-8 sequences after filtering, it becomes dangerous. In this case, malicious actors can exploit overlong UTF-8 sequences to perform various attacks such as path traversals, cross-site scripting (XSS), SQL injection, or command injection. These attacks often rely on the ability to encode characters in unexpected ways, allowing attackers to bypass security checks that rely on standard encoding forms…’.

The author goes on to talk about buffer overflows and ‘consuming the beginning of concatenated application data’. She concludes this section by observing that these weaknesses, while real enough, would only be one small part of a hacking exploit. But goes on to qualify this with the observation that one needs to be careful about the use of legacy code, which might be hidden under some innocuous looking blanket or other, perhaps provided by some entirely respectable provider, like Microsoft or Oracle. 

Turning to history, she provides chapter and verse in the form of references to a secure database of such things, for example reference 8. From which, for example, I am referred on to an IBM database about such things, which you are invited to join, to find that my personal email does not cut it, despite claiming affiliation to the Home Office. You clearly need something a bit more respectable.

But I am allowed into the parent site, at reference 9, where I get the scenic snap above and, a bit further down:

'Acquisition on a War Footing: The Department of War is working to deliver its most substantial transformation to its acquisition system in decades. MITRE’s Beyond Barriers Summit connected Industry, government, and policy leaders to help clear the way for innovation'.

I dare say with a bit of poking around, one could find out a bit more about the services on offer.

Plenty of other sites out there covering this sort of thing, most of them not very coy about sharing. See, for example, references 10 and 11. A bit on the geeky side for me, but there is the odd chink of light.

Other matters

One

One could get quite philosophical about the way that you can only give meaning to strings of bits by associating them with formatting rules and conventions. And the same string of bits can mean quite different things, depending on what rules and conventions you do attach to them.

Sometimes a string of bits can be reverse engineered, without much prior knowledge. The sort of thing that code breakers get up to. Sometimes you can help this process along, if, for example, you are sending out a repeating bit stream coding for ‘π’ or something like that, in an attempt to tell aliens out there that there is intelligent life here on earth.

Two

Regular search turned up the standard that specifies URI, ‘RFC 3986: STD 66: Uniform resource identifier: Generic Syntax’, but this is a dense document, written for professionals. One of the authors was a chap called Berners-Lee. But it did not turn up any tutorial material of the sort I wanted. Gemini, however, could oblige – and in response to a supplementary, gave me a long story about why regular search failed. Something of a puff for his own approach to the matter!

I learned that this specification included query capability – which I had noticed but not registered in Blogger queries. For a recent example, see reference 2 in reference 7 below.

Conclusions

Another example of Gemini doing well in a dialogue when I am doing my homework and asking the right questions. A bit like having a tutor all to oneself.

Things did speed up a bit when I downloaded the table at reference 3 into a spreadsheet and printed it off. Much easier having that piece on paper to refer to.

But it is still a bit of a puzzle why it all took so long. Maybe it was all too much for older working memory.

Along the way, I came to suspect that the motivation for hacking is not entirely evil, even if the results are evil. It is a fascinating business, a challenge and a puzzle, and there are plenty of people out there who like puzzles, without much regard for the consequences.

References

Reference 1: https://psmv6.blogspot.com/2026/09/test-report.html.  

Reference 2: https://en.wikipedia.org/wiki/UTF-8. The source of the opening snap.

Reference 3: https://www.ibm.com/docs/en/aix/7.2.0?topic=adapters-ascii-decimal-hexadecimal-octal-binary-conversion-table.  

Reference 4: https://en.wikipedia.org/wiki/Hexadecimal. 

Reference 5: https://kevinboone.me/overlong.html. 

Reference 6: https://herolab.usd.de/the-security-risks-of-overlong-utf-8-encodings/.  

Reference 7: https://psmv6.blogspot.com/2026/09/screwfix-monday.html. 

Reference 8: https://www.cve.org/CVERecord?id=CVE-2000-0884. 

Reference 9: https://www.mitre.org/.  

Reference 10: https://blackhat.com/presentations/bh-usa-09/VELANAVA/BHUSA09-VelaNava-FavoriteXSS-SLIDES.pdf. 

Reference 11: http://sirdarckcat.blogspot.com/. Flagged as insecure by Edge, but still visible.

Group search key: aigoodsk.