Hotlinking, in the context I want to discuss here, is the act of using a resource on your website by linking to it on someone else’s website. This might be any resource: a script, an image, anything that is referenced by URL.

It’s a bit of an anti-social practice, to be honest. Essentially, you’re offloading the responsibility for the bandwidth of serving that resource to someone else, but it’s your site and your users who get the benefit of that. That’s not all that nice.

Now, if the “other person’s website” is a CDN— that is, a site deliberately set up in order to serve resources to someone else — then that’s different. There are many CDNs, and using resources served from them is not a bad thing. That’s not what I’m talking about. But if you’re including something direct from someone else’s not-a-CDN site, then… what, if anything, should the owner of that site do about it?

I’ve got a fairly popular, small, piece of JavaScript: sorttable.js, which makes an HTML table be sortable by clicking on the headers. It’s existed for a long time now (the very first version was written twenty years ago!) and I get an email about it once a week or so from people looking to customise how it works or ask questions about how to do a thing they want. It’s open source, and I encourage people to use it; it’s deliberately designed to be simple¹, because the target audience is really people who aren’t hugely experienced with web development and who can add sortability to their HTML tables with a couple of lines of code.

The instructions for sorttable are pretty clear: download the library, then put it in your web space and include it. However, some sites skip that first step, and instead just link directly to the copy on my website with a <script> element. Having looked at my bandwidth usage recently, this happens quite a lot², and on some quite high-profile sites. I’m not going to name and shame anyone³, but I’d quite like to encourage people to not do that, if there’s a way to do it. So I’ve been thinking about ways that I might discourage hotlinking the script directly, while doing so in a reasonable and humane fashion. I’m also interested in suggestions: hit me up on Mastodon at @sil@mastodon.social or Twitter⁴ as @sil.

Move the script to a different URL

This is the obvious thing to do: I move the script and update my page to link to the new location, so anyone coming to my page to get the script will be wholly unaffected and unaware I did it. I do not want to do this, for two big reasons: it’s kicking the can down the road, and it’s unfriendly.

It’s can-kicking because it doesn’t actually solve the problem: if I do nothing else to discourage the practice of hotlinking, then a few years from now I’ll have people hotlinking to the new location and I’ll have to do it again. OK, that’s not exactly a lot of work, but it’s still not a great answer.

But more importantly, it’s unfriendly. If I do that, I’ll be deliberately breaking everyone who’s hotlinking the script. You might think that they deserve it, but it’s not actually them who feel the effect; it’s their users. And their users didn’t do it. One of the big motives behind the web’s general underlying principle of “don’t break the web” is that it’s not reasonable to punish a site’s users for the bad actions of the site’s creators. This applies to browsers, to libraries, to websites, the whole lot. I would like to find a less harsh method than this.

Move the script to a different dynamicURL

That is: do the above, but link to a URL which changes automatically every month or every minute or something. The reason that I don’t want to do this (apart from the unfriendly one from above, which still applies even though this fixes the can-kicking) is that this requires server collusion; I’d need to make my main page be dynamic in some way, so that links to the script also update along with the script name change. This involves faffery with cron jobs, or turning the existing static HTML page into a server-generated page, both of which are annoying. I know how to do this, but it feels like an inelegant solution; this isn’t really a technical problem, it’s a social one, where developers are doing an anti-social thing. Attempting to solve social problems with technical measures is pretty much always a bad idea, and so it is in this case.

Contact the highest-profile site developers about it

I’m leaning in this direction. I’m OK with smaller sites hotlinking (well, I’m not really, but I’m prepared to handwave it; I made the script and made it easy to use exactly to help people, and if a small part of that general donation to the universe includes me providing bandwidth for it, then I can live with that). The issue here is that it’s not always easy to tell who those heavy-bandwidth-consuming sites are. It relies on the referrer being provided, which it isn’t always. It’s also a bit more work on my part, because I would want to send an email saying “hey, Site X developers, you’re hotlinking my script as you can see on page sitex.example.com/sometable.html and it would be nice if you didn’t do that”, but I have no good way of identifying those pages; the document referrer isn’t always that specific. If I send an email saying “you’re hotlinking my script somewhere, who knows where, please don’t do that” then the site developers are quite likely to put this request at the very bottom of their list, and I don’t blame them.

Move the script and maliciously break the old one

This is: I move the script somewhere else and update my links, and then I change the previous URL to be the same script but it does something like barf a complaint into the console log, or (in extreme cases based on suggestions I’ve had) pops up an alert box or does something equally obnoxious. Obviously, I don’t wanna do this.

Legal-ish things

That is: contact the highest profile users, but instead of being conciliatory, be threatening. “You’re hotlinking this, stop doing it, or pay the Hotlink Licence Fee which is one cent per user per day” or similar. I think the people who suggest this sort of thing (and the previous malicious approach) must have had another website do something terrible to them in a previous life or something and now are out for revenge. I liked John Wick as much as the next poorly-socialised revenge-fantasy tech nerd, but he’s not a good model for collaborative software development, y’know?

Put the page (or whole site) behind a CDN

I could put the site behind Cloudflare (or perhaps a better, less troubling CDN) and then not worry about it; it’s not my bandwidth then, it’s theirs, and they’re fine with it. This used to be the case, but recently I moved web hosts⁵ and stepped away from Cloudflare in so doing. While this would work… it feels like giving up, a bit. I’m not actually solving the problem, I’m just giving it to someone else who is OK with it.

Live with it

This isn’t overrunning my bandwidth allocation or anything. I’m not actually affected by this. My complaint isn’t important; it’s more a sort of distaste for the process. I’d like to make this better, rather than ignoring it, even if ignoring it doesn’t mean much, as long as I’m not put to more inconvenience by fixing it. We want things to be better, after all, not simply tolerable.

So… what do you think, gentle reader? What would you do about it? Answers on a postcard.

1. and will stay simple; I’d rather sorttable were simple and relatively bulletproof than comprehensive and complicated. This also explains why it’s not written in very “modern” JS style; the best assurance I have that it works in old browsers that are hard to test in now is that it DID work in them and I haven’t changed it much ↩
2. in the last two weeks I’ve had about 200,000 hits on sorttable.js from sites that hotlink it, which ain’t nothin’ ↩
3. yet, at least, so don’t ask ↩
4. if you must ↩
5. to the excellent Mythic Beasts, who are way better than the previous hosts ↩

In 1701, Asano Naganori, a feudal lord in Japan, was summoned to the shogun’s court in Edo, the town now called Tokyo. He was a provincial chieftain, and knew little about court etiquette, and the etiquette master of the court, Kira Kozuke-no-Suke, took offence. It’s not exactly clear why; it’s suggested that Asano didn’t bribe Kira sufficiently or at all, or that Kira felt that Asano should have shown more deference. Whatever the reasoning, Kira ridiculed Asano in the shogun’s presence, and Asano defended his honour by attacking Kira with a dagger.

Baring steel in the shogun’s castle was a grievous offence, and the shogun commanded Asano to atone through suicide. Asano obeyed, faithful to his overlord. The shogun further commanded that Asano’s retainers, over 300 samurai, were to be dispossessed and made leaderless, and forbade those retainers from taking revenge on Kira so as to prevent an escalating cycle of bloodshed. The leader of those samurai offered to divide Asano’s wealth between all of them, but this was a test. Those who took him up on the offer were paid and told to leave. Forty-seven refused this offer, knowing it to be honourless, and those remaining 47 reported to the shogun that they disavowed any loyalty to their dead lord. The shogun made them rōnin, masterless samurai, and required that they disperse. Before they did, they swore a secret oath among themselves that one day they would return and avenge their master. Then each went their separate ways. These 47 rōnin immersed themselves into the population, seemingly forgoing any desire for revenge, and acting without honour to indicate that they no longer followed their code. The shogun sent spies to monitor the actions of the rōnin, to ensure that their unworthy behaviour was not a trick, but their dishonour continued for a month, two, three. For a year and a half each acted dissolutely, appallingly; drunkards and criminals all, as their swords went to rust and their reputations the same.

A year and a half later, the forty-seven rōnin gathered together again. They subdued or killed and wounded Kira’s guards, they found a secret passage hidden behind a scroll, and in the hidden courtyard they found Kira and demanded that he die by suicide to satisfy their lord’s honour. When the etiquette master refused, the rōnin cut off Kira’s head and laid it on Asano’s grave. Then they came to the shogun, surrounded by a public in awe of their actions, and confessed. The shogun considered having them executed as criminals but instead required that they too die by suicide, and the rōnin obeyed. They were buried, all except one who was not present and who lived on, in front of the tomb of their master. The tombs are a place to be visited even today, and the story of the 47 rōnin is a famous one both inside and outside Japan.

You might think: why have I been told this story? Well, there were 47 of them. 47 is a good number. It’s the atomic number of silver, which is interesting stuff; the most electrically conductive metal. (During World War II, the Manhattan Project couldn’t get enough copper for the miles of wiring they needed because it was going elsewhere for the war effort, so they took all the silver out of Fort Knox and melted it down to make wire instead.) It’s strictly non-palindromic, which means that it’s not only not a palindrome, it remains not a palindrome in any base smaller than itself. And it’s how old I am today.

Yes! It’s my birthday! Hooray!

I have had a good birthday this year. The family and I had delightful Greek dinner at Mythos in the Arcadian, and then yesterday a bunch of us went to the pub and had an absolute whale of an afternoon and evening, during which I became heartily intoxicated and wore a bag on my head like Lord Farrow, among other things. And I got a picture of the Solvay Conference from Bruce.

This year is shaping up well; I have some interesting projects coming up, including one will-be-public thing that I’ve been working on and which I’ll be revealing more about in due course, a much-delayed family thing is very near its end (finally!), and in general it’s just gotta be better than the ongoing car crash that the last few years have been. Fingers crossed; ask me again in twelve months, anyway. I’ve been writing these little posts for 21 years now (last year has more links) and there have been ups and downs, but this year I feel quite hopeful about the future for the first time in a while. This is good news. Happy birthday, me.

So, Dungeons & Dragons: Honour Among Thieves, which I have just watched. I have some thoughts. Spoilers from here on out!

Up front I shall say: that was OK. Not amazing, but not bad either. It could have been cringy, or worthy, and it was not. It struck a reasonable balance between being overly puffed up with a sense of epic self-importance (which it avoided) and being campy and ridiculous and mocking all of us who are invested in the game (which it also avoided). So, a tentative thumbs-up, I suppose. That’s the headline review.

But there is more to be considered in the movie. I do rather like that for those of us who play D&D, pretty much everything in the film was recognisable as an actual rules-compliant thing, without making a big deal about it. I’m sure there are rules lawyers quibbling about the detail (“blah blah wildshape into an owlbear”, “blah blah if she can cast time stop why does she need some crap adventurers to help”) but that’s all fine. It’s a film, not a rulebook.

I liked how Honour Among Thieves is recognisably using canon from an existing D&D land, Faerûn, but without making it important; someone who doesn’t know this stuff will happily pass over the names of Szass Tam or Neverwinter or Elminster or Mordenkainen as irrelevant world-building, but that’s in there for those of us who know those names. It’s the good sort of fanservice; the sort that doesn’t ruin things if you’re not a fan.

(Side notes: Simon is an Aumar? And more importantly, he’s Simon the Sorcerer? Is that a sly reference to theSimon the Sorcerer? Nice, if so. Also, I’m sure there are one billion little references that I didn’t catch but might on a second or third or tenth viewing, and also sure that there are one billion web pages categorising them all in exhaustive detail. I liked the different forms of Bigby’s Hand. But what happened to the random in the gelatinous cube?)

And Chris Pine is nowhere near as funny as he thinks he is. Admittedly, he’s playing a character, and obviously Edgin the character’s vibe is that Edgin is not as funny as he thinks he is, but even given that, it felt like half the jokes were delivered badly and flatly. Marvel films get the comedy right; this film seemed a bit mocking of the concept, and didn’t work for me at all.

I was a bit disappointed in the story in Honour Among Thieves, though. The characters are shallow, as is the tale; there’s barely any emotional investment in any of it. We’re supposed, presumably, to identify with Simon’s struggles to attune to the helmet and root for him, or with the unexpectedness of Holga’s death and Edgin and Kira’s emotions, but… I didn’t. None of it was developed enough; none of it made me feel for the characters and empathise with them. (OK, small tear at Holga’s death scene. But I’m easily emotionally manipulated by films. No problem with that.) Similarly, I was a bit annoyed at how flat and undeveloped the characters were at first; the paladin Xenk delivering the line about Edgin re-becoming a Harper with zero gravitas, and the return of the money to the people being nowhere near as epically presented as it could have been.

But then I started thinking, and I realised… this is a D&D campaign!

That’s not a complaint at all. The film is very much like an actual D&D game! When playing, we do all strive for epic moves and fail to deliver them with the gravitas that a film would, because we’re not pro actors. NPCs do give up the info you want after unrealistically brief persuasion, because we want to get through that quick and we rolled an 18. The plans are half-baked but with brilliant ideas (the portal painting was great). That’s D&D! For real!

You know how when someone else is describing a fun #dnd game and the story doesn’t resonate all that strongly with you? This is partially because the person telling you is generally not an expert storyteller, but mostly because you weren’t there. You didn’t experience it happening, so you missed the good bits. The jokes, the small epic moments, the drama, the bombast.

That’s what D&D: Honour Among Thieves is. It’s someone telling you about their D&D campaign.

It’s possible to rise above this, if you want to and you’re really good. Dragonlance is someone telling you about their D&D campaign, for example. Critical Role can pull off the epic and the tragic and the hilarious in ways that fall flat when others try (because they’re all very good actors with infinite charisma). But I think it’s OK to not necessarily try for that. Our games are fun, even when not as dramatic or funny as films. Honour Among Thieves is the same.

I don’t know if there’s a market for more. I don’t know how many people want to hear a secondhand story about someone else’s D&D campaign that cost $150m. This is why I only gave it a tentative thumbs-up. But… I believe that the film-makers’ attempt to make Honour Among Thieves be like actual D&D is deliberate, and I admire that.

This game of ours is epic and silly and amateurish and glorious all at once, and I’m happy with that. And with a film that reflects it.

A few days ago I had an interesting maths thought which I dropped on Mastodon:

Disappointingly, I couldn’t think of an interesting way to solve it which wasn’t “have a computer check all the possibilities”, so I had a computer check all the possibilities.

I’m not publishing the script, because it’s not very good; in particular, I’m sure this is the sort of thing that someone could make run in a few seconds if they were clever. I wasn’t feeling clever, so I brute-forced it and just left it to run for a few hours.

These are numeric pangrams, I suppose you might call them. A pangram is a sentence which uses all the letters in the alphabet at least once: there are a bunch of famous English examples, starting with “The quick brown fox jumps over a lazy dog” and getting less and less comprehensible as they get shorter. Vexing quizzes show up a lot. Anyway, after I thought up the term “numeric pangrams” I checked to see if anybody else had already done so and of course Greg Ross at Futility Closet did sixteen years ago. His examples are the digits from 1-9, though (his biggest example is 4 × 1963 = 7852), and my 0-9 suggestion allows for much larger total values.

Anyway, the best equations I could come up with which use all the digits from 0-9 were:

• 42 × 8 × 91 = 30576
• 46 × 715 = 32890
• 4 × 9127 = 36508
• 63 × 927 = 58401
• 7 × 9403 = 65821

This is the sort of thing that Henry Dudeney would have cleverly done by hand. Python is the way forward, of course.

What’s-a matter you? Hey!
Gotta no respect?
What-a you t’ink you do, why you look-a so sad?
It’s-a not so bad, it’s-a nice-a place

(If you just want to know about how to make your own Matter device in software and don’t want to read a whole story to get to that point, then first of all that’s, I mean, that’s fine, it’s not a problem, you go ahead, I’m not hurt at all, definitely not, and secondly skip down to Part 4 or check out the Github repo. But, I mean, really? You’re that busy? You might wanna find a way to chill out a bit. Have a cup of tea. Enjoy life. It’s better, I promise.)

Part the first: the kingdom of the blind

I’ve got new window blinds in my flat. They’re pretty cool; ivory and purple, we’re all very pleased. They’re from Hillarys, who don’t get a link because although the actual blinds are great, their customer service is… less so, so I’m loath to recommend them. But one of the neat things about them is that they’re electrically operated. There’s a remote control; press the up or down button and they go up and down. Some people would be like a fascinated small child with this technology and spend half of the first day just pressing the buttons and giggling, but not me, no no.

One of the questions I was asked when speccing them out was: do you want them to be controllable by Alexa? And I thought: well, yes, I would like that. Being able to command my blinds to open or close is one more step on the quest to be a James Bond villain. All I need after that is a monorail and a big picture on the wall which revolves to show my world domination map and I’m golden.

But… I do not like or trust the IoT industry. I’m not entirely against the whole concept — I mean, I have an Amazon Echo, after all. It gets used for cooking timers, my shopping list, and music¹, but it doesn’t get used for IoT stuff because I don’t have any. This is, basically, because the whole IoT industry is full of grifters and thieves and I do not trust them with my data or with still being around in two years. The whole idea that I speak to a thing in my flat, that thing routes my command up to some server run by an IoT vendor, and then the IoT vendor’s server speaks to another thing in my flat to make it do work… this is a stupid plan. It’s the vendor forcibly inserting themselves as a middle man for literally no reason other than to exploit me later. Because they’ll get bored or bought in a few years and they’ll shut down the servers, with a sorry-not-sorry email about how it’s “no longer viable to support” a thing I paid for and therefore they’re turning it into a brick, but I can buy a replacement for the low low cost of… anyway, I’m not doing it. No. I don’t want some random vendor’s server to punch a hole into my internal network, I don’t want my stuff to stop working if they turn their servers off or they go down, and the whole concept is hugely irritating.

(You might be thinking: why do you have an Echo at all, then? That’s got all the same problems! And you’re not wrong. But I trust Amazon to still be around for a while. Trusting them with my data is another thing, of course, but I’m already on board with that… although I entirely sympathise with people who choose to not do so either from distrust or from objections to their crappy employment or sales practices.)

Anyway, I looked up these blinds and the Alexa integration they do comes via a company called Somfy. Their APIs seem semi-reasonably documented and I’ve heard nothing specifically bad about them as a company… but I still don’t like the idea. If I were Matthew Garrett then I would probably find joy in reverse-engineering whatever the protocol is and making it work for me, but I’m not as clever as he is². And they’ll get bored or bought: I’m not sure I trust them to keep their servers running for years and years. Maybe I’d be OK with a thing that required an internet connection to someone else’s server and only let me fiddle with that to the extent that I am given permission for³, if I’d only expect to keep that thing for a short time, but these are window blinds. How often do you change your window blinds? I expect these will still be here in twenty years! Do I expect these servers to also be there that long? Hell no. So, I’m not doing that.

Part the second: keep government local

I do actually have one “IoT” device in my flat, though. It’s a remotely-controllable wall socket. It’s from a company called LocalBytes, and it’s basically a “smart plug”: it’s a cylinder that plugs into a wall socket, and has another socket on the other side of it, rather like one of those mini gangplug cube things.

It contains a little microcontroller and wifi chip, and it runs software called Tasmota. And it’s entirely locally controlled; you get a Tasmota app (or talk the documented protocol yourself from code) which can send the plug a command to turn on and off (and also a bunch more complex stuff such as turning on at a specific time), and it involves no internet servers at all. I can’t get screwed by you turning off your servers (or failing to secure them properly) if there aren’t any⁴. Now, I would not recommend these Tasmota devices to a normal person; the app is unreliable and pretends it can’t find the device some of the time, and the configuration API is obscure and weird. But I am, as has been said by kings and queens, not a normal person. I’m a computer bloke. So I am OK with this, and I’d be OK with something similar to control my blinds; something that runs locally and accepts commands from Alexa and then talks to the blinds to open and close them.

Now, the bit that actually talks to the blinds, I haven’t started working on yet. As far as I can tell from reading, the remote control works on a standard “smart home” frequency of 433MHz, and there are loads of little dongles and boards that plug into USB or Raspberry Pi GPIO pins which can talk that. I’ll get to that eventually; once I have a working thing, making it talk 433MHz is step 2. Step 1 is to make something that I control and which Alexa can talk to, but which doesn’t require servers on the internet to make it work. This rules out writing a custom Alexa skill; I can do that, and have, but you can’t write a skill which makes the actual Echo in my flat do network connections. The connection comes from Amazon’s servers, which means I’d have to put my little device on the internet, which I don’t want to do. The concern of the servers going away doesn’t apply here — if Amazon’s servers go away, my Echo stops working anyway and all of this is moot — but I do not want to punch a hole into my internal network from outside, and I shouldn’t have to. This is one thing in my house talking to another thing in my house.

This problem used to be unsolvable. And then, just like in the beginning of all things, someone invented Matter.

Part the third: does it Matter?

It is, obviously, stupid that every device invents its own protocol to talk to it, and none of it’s documented, and none of it’s standardised, and everything gets reinvented from scratch by programmers who clearly have their minds on lunchtime. So, equally obviously, the way to solve this is to have all the biggest tech companies in the world collaboratively agree on a standard way to do things.

I can hear you laughing from here.

This has, equally equally obviously, resulted in a disastrous nine-way collision of corporate speak, absurd specifications, paywalls for things, requirements to sign up as a “partner” before you’re told anything, and documentation by press release. But to my surprise it has actually resulted in something! The Matter specifications have basically everybody on board — Amazon, Apple, Google, most of the big players in the device world — and have finally started to actually appear in real shipping products rather than only in conference talks. The Amazon Echo supports Matter back to the 3rd gen Dot (there’s a list of Matter-supporting Echo devices here). If you’ve got a Matter-supporting thingy, then it’ll have the Matter logo on it:

and then you can say “Alexa, discover devices” and your Echo will search and then say “I have found a Matter device!”. You then use the Alexa app on your phone to pair it, either by scanning a QR code or typing in a code, both of which should be on the device itself.

Now, Matter is a big corporate spec and wants to deal with all sorts of complicated and detailed edge cases. In particular, there is a standard problem with an IoT device in your house, which is that you can’t talk to it until it’s on the wifi, but you can’t put it on the wifi without talking to it. This normally involves the device pretending to be a wifi access point, and you connecting to it with a mobile app you have to install⁵, but Matter attempts to improve this; a Matter device can potentially exchange data over wifi, over Bluetooth, over some extra network thing called Thread that I don’t understand, over ethernet, the works. A lot of the setup and spec for Matter involves dealing with all this.

I, personally, for myself, for this device, do not care about this. If I were making a device that would be sold to real people, I’d implement all this stuff. But since it’s just me, I’m OK with requiring that my window blind device is already on my wifi, and getting it that way is my problem, before I try detecting it with Alexa.

So, I need a way to make a Matter device; it only has to deal with wifi. The Matter specification⁶ describes how to talk Matter⁷, but there’s a lot of detail in there. Surely someone has already implemented this stuff?

Well, they have… but it’s not great. The @project-chip/connectedhomeip Github repository, named back in the days when Matter was still called “project CHIP” before the Branding People got their hands on it, is the reference implementation. And it’s a nightmare. It’s all in C, it’s huge (I checked out the repo and all its submodules and ran out of disk space; to get the repo and all sub-repos and then build it, you’d better have 30GB or so free), compiling it all is a massive headache, it contains code for all sorts of different single-board computers to make them Matter devices, and I couldn’t make head nor tail of it. I’m sure you need all this if you’re looking to build a device and ship millions of them to people in Best Buy and Currys, but I ain’t. Normally, people write stuff like this in Python, but in looking around all I could find was a Python library designed to work with Home Assistant, which is not what I wanted (I want to make a device, not a controller) and which required the CHIPSDK anyway, the big complicated nightmare thing mentioned above. I resigned myself to having to write a very noddy implementation of enough of Matter’s pairing and communications stuff in Python to get a device up and running, and bitched about having to do this on social media. And then Alan Bell, hero of the revolution, said: have you tried matter.js?⁸

I had not. So I gave it a look.

Part the fourth: yes it Matters!

Matter.js is an implementation of the Matter suite of protocols in JavaScript. It does not build on the huge complicated Project CHIPSDK; it’s its own thing. And it just works.

If you want to build a Matter device in JavaScript, take a look at the Matter.js Node examples and they work fine. You can do that right now.

I did something a little tiny bit different. The Matter.js stuff is not actually JS; it’s TypeScript. I don’t like TypeScript⁹ and so I wanted a plain JS thing. So I did a simple reimplementation of the simplest Matter.js example in plain JS, and made it even simpler; the original correctly lets you pass in a bunch of details on the command line to configure it, but I’m not bothered about that and wanted to keep it as simple as possible. So, the simplest possible “virtual” Matter device implemented in JavaScript. It’s a lightbulb; Alexa, and Matter, do support multiple types of devices (including window blind controllers!) but Matter.js only does lightbulbs and sockets at the moment. I’m probably going to build and contribute the window blind controller device type, if someone else doesn’t get there first. Anyway, let’s get this code running! Use is pretty easy for a developer:

$gitclonehttps://github.com/stuartlangridge/simple-js-matter-device.git
Cloninginto'simple-js-matter-device'...
remote:Enumeratingobjects:11,done.
remote:Countingobjects:100%(11/11),done.
remote:Compressingobjects:100%(10/10),done.
remote:Total11(delta0),reused7(delta0),pack-reused0
Receivingobjects:100%(11/11),25.16KiB|1.48MiB/s,done.
$cdsimple-js-matter-device
$npminstall

added90packages,andaudited91packagesin5s

5packagesarelookingforfunding
run`npmfund`fordetails

found0vulnerabilities
$nodesil.mjs2023-10-0611:38:24.231INFODevicenode-matter
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
█▄▄▄▄▄██▀▄▀▄█▄▄▄▄▄█
████▄▀▄▀▄████
██▄▄▄███▀▀▄██▄▄▄██
█▄▄▄▄▄▄▄██▀▄█▄▄▄▄▄▄▄█
█▄█▄▄▄▄██▄▀███▀█
██▀█▀▄█▄█▀▀▄█▀█▀█
██▄█▄▄█▄▄█▄▄▀▀▄█▄▀▄▀█
█▄▄▄▄▄█▄█▄█▄▄▄██
████▄▄█▄▄▀▄▀██
██▄▄▄██▀█▄█▀█▀█
█▄▄▄▄▄▄▄█▄███▄██▄██▄█▄█
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀

2023-10-0611:38:24.330INFODeviceQRCodeURL:https://project-chip.github.io/connectedhomeip/qrcode.html?data=MT:Y.K90-Q000KA0648G00
2023-10-0611:38:24.330INFODeviceManualpairingcode:34970112332

We’ve git cloned the repository, done npm install to install the dependencies, and run it. That starts up our virtual device, and also prints the QR code and manual pairing code used for pairing. Job done; your device now exists.

Next, we teach Alexa about it. Exercise your most commanding voice and say “Alexa, discover new devices!” Your Echo will tell you it’s doing it, and then after a few seconds of searching, tell you that it’s found one Matter device and send you off to the devices section of the Alexa app on your phone to connect it up; I also got a notification on my phone telling me the same thing.

Hit that notification and you end up on the Devices screen, where it’ll tell you to “Connect your Matter Light Bulb”:

(For some reason, my device seems to present as two separate devices; a light bulb and an unnamed “Matter device”. I don’t know why. It seems that you can pick either.)

Choose to connect to this Matter device, and you get a screen called “Control your Matter device with Alexa”, which is exactly what we want to do:

When asked if your device has a Matter logo, say yes (it would do if it were a real Matter device, of course):

Then you have to scan the QR code. Usefully, the device has printed out a QR code in the Terminal where you ran it, above; scan that. If that didn’t work for some reason, it has also printed a URL for that QR code which you can follow to show the QR in the browser, and if that doesn’t work either, you can enter the numeric code it printed (the “manual pairing code) instead:

Alexa will then claim to be “Looking for your device” (and the stuff in the Terminal will print a bunch of logging things about “Pase server: Received pairing request from udp://192.168.1.103:5541” and whatnot):

and then, all else being good, you’ll get a screen telling you your light is connected! It’s called “Kryogenix Light” in this screen, and “Stuart Light” elsewhere; you can see those strings in sil.mjs and can customise them to your heart’s content.

You should now have a “Lights” section in the Devices screen of your Alexa app. You can use this to turn the “light” on and off from the Alexa app, or you can say “Alexa, turn Stuart Light on” to do it by voice. When you do that, the Terminal should print !!!!!!!!!!!!!!!! GOT COMMAND true (or false), which means it’s calling the onOffListener we defined in sil.mjs. You can customise this to do whatever you’d like!

And that’s it working. That’s a software device, which you can pair with Alexa and customise how you choose; it doesn’t require you to write an Alexa skill, and the device does not need to be accessible from the internet. That’s just what I want.

Now I suppose I have to make it do something useful…

1. less so music than before, after the bloody shysters at Amazon decided to take away the free music playing from Amazon Prime and make people pay extra for it ↩
2. I make up for it by being, like, twice his size, though ↩
3. honestly, this is the big issue. I react very badly to stuff where I ought to be able to do a thing and I’m not permitted to do so because they don’t want me to or can’t be bothered to implement it, and won’t let me do it myself. This is one of the big things that makes me move away from iOS every few years ↩
4. insert picture here of that bloke tapping his temple knowingly ↩
5. you should not need a mobile app to connect to a device pretending to be an access point. You should be able to do this from the web browser on your phone. But you can’t, as I have laboriously discussed before in these pages and am still annoyed about ↩
6. as usual with corporate technology things, the Matter specification is difficult to find; one of these things where you need to sign up, or be a “partner”, or pay money, or something. But it is obtainable. The Alexa Matter docs link to a page on the Matter website which lets you fill in an email address and then emails you a link to a PDF of the spec in all its 895 page glory ↩
7. super reassuringly, Matter seems to use all standard protocols! device discovery is mdns! They even have example commands for avahi-publish-service to show you how to do it, right there in the spec! ↩
8. There is an unfortunate naming collision here. There already was a Matter.js: it’s a 2d physics engine for the web, which I’ve used before. But I can’t see what else the people implementing the Matter spec in JavaScript could have called their thing. Only so many words in the world, I guess ↩
9. feel free to email your long diatribe about why TypeScript is the future to IDontCare@Whatever.Bored ↩

Discord have changed the way bots work quite a few times. Recently, though, they built a system that lets you create and register “slash commands” — commands that you can type into the Discord chat and which do things, like /hello— and which are powered by “webhooks”. That is: when someone uses your command, it sends an HTTP request to a URL of your choice, and your URL then responds, and that process powers what your users see in Discord. Importantly, this means that operating a Discord bot does not require a long-running server process. You don’t need to host it somewhere where you worry about the bot process crashing, how you’re going to recover from that, all that stuff. No daemon required. In fact, you can make a complete Discord bot in one single PHP file. You don’t even need any PHP libraries. One file, which you upload to your completely-standard shared hosting webspace, the same way you might upload any other simple PHP thing. Here’s some notes on how I did that.

The Discord documentation is pretty annoying and difficult to follow; all the stuff you need is in there, somewhere, but it’s often hard to find where, and there’s very little that explains why a thing is the way it is. It’s tough to grasp the “Zen” of how Discord wants you to work with their stuff. But in essence, you’ll need to create a Discord app: follow their instructions to do that. Then, we’ll write our small PHP file, and upload it; finally, fill in the URL of that PHP file as the “interactive endpoint URL” in your newly-created app’s general information page in the Discord developer admin. You can then add the bot to a server by visiting the URL from the “URL generator” for your app in the Discord dev admin.

The PHP file will get sent blocks of JSON, which describe what a user is doing — a command they’ve typed, parameters to that command, or whatever — and respond with something which is shown to the user — the text of a message which is your bot’s reply, or a command to alter the text of a previous message, or add a clickable button to it, and the like. I won’t go into detail about all the things you can do here (if that would be interesting, let me know and maybe I’ll write a followup or two), but the basic structure of your bot needs to be that it authenticates the incoming request from Discord, it interprets that request, and it responds to that request.

Authentication first. When you create your app, you get a client_public_key value, a big long string of hex digits that will look like c78c32c3c7871369fa67 or whatever. Your PHP file needs to know this value somehow. (How you do that is up to you; think of this like a MySQL username and password, and handle this the same way you do those.) Then, every request that comes in will have two important HTTP headers: X-Signature-ED25519 and X-Signature-Timestamp. You use a combination of these (which provide a signature for the incoming request) and your public key to check whether the request is legitimate. There are PHP libraries to do this, but fortunately we don’t need them; PHP has the relevant signature verification stuff built in, these days. So, to read the content of the incoming post and verify the signature on it:

/* read the incoming request data */$postData = file_get_contents('php://input');/* get the signature and timestamp header values */$signature = isset($_SERVER['HTTP_X_SIGNATURE_ED25519']) ?     $_SERVER['HTTP_X_SIGNATURE_ED25519'] : "";$timestamp = isset($_SERVER['HTTP_X_SIGNATURE_TIMESTAMP']) ?     $_SERVER['HTTP_X_SIGNATURE_TIMESTAMP'] : "";/* check the signature */$sigok = sodium_crypto_sign_verify_detached(    hex2bin($signature),     $timestamp . $postData,    hex2bin($client_public_key));/* If signature is not OK, reject the request */if (!$sigok) {    http_response_code(401);    die();}

We need to correctly reject invalidly signed requests, because Discord will check that we do — they will occasionally send test requests with bad signatures to confirm that you’re doing the check. (They do this when you first add the URL to the Discord admin screens; if it won’t let you save the settings, then it’s because Discord thinks your URL returned the wrong thing. This is annoying, because you have no idea why Discord didn’t like it; best bet is to add lots of error_log() logging of inputs and outputs to your PHP file and inspect the results carefully.)

Next, interpret the incoming request and do things with it. The only thing we have to respond to here is a ping message; Discord will send them as part of their irregular testing, and expects to get back a correctly-formatted pong message.

$data = json_decode($postData);if ($data->type == 1) { // this is a ping message    echo json_encode(array('type' => 1)); // response: pong    die();}

The magic numbers there (1 for a ping, 1 for a pong) are both defined in the Discord docs (incoming values being the “Interaction Type” field and outgoing values the “Interaction Callback Type”.)

After that, the world’s approximately your oyster. You check the incoming type field for the type of incoming thing this is — a slash command, a button click in a message, whatever — and respond appropriately. This is all stuff for future posts if there’s interest, but the docs (in particular the “receiving and responding and “message components” sections) have all the detail. For your bot to provide a slash command, you have to register it first, which is a faff; I wrote a little Python script to do that. You only have to do it once. The script looks approximately like this; you’ll need your APP_ID and your BOT_TOKEN from the Discord dashboard.

importrequests,jsonMY_COMMAND={"name":'doit',"description":'Do the thing',"type":1}discord_endpoint=_f"https://discord.com/api/v10/applications/{APP_ID}/commands"requests.request("PUT",discord_endpoint,json=[MY_COMMAND],headers={"Authorization":f"Bot {BOT_TOKEN}","User-Agent":'mybotname (myboturl, 1.0.0)',})

Once you’ve done that, you can use /doit in a channel with your bot in, and your PHP bot URL will receive the incoming request for you to process.

There’s a YouTube channel called Clickspring, run by an Australian bloke called Chris who is a machinist: a mechanical engineer with a lathe and a mill and all manner of little tools. I am not a machinist — at school I was fairly inept at what we called CDT, for Craft Design and Technology, and what Americans much more prosaically call “shop class”. My dad was, though, or an engineer at least. Although Chris builds clocks and beautiful brass mechanisms, and my dad built aeroplanes. Heavy engineering. All my engineering is software, which actual engineers don’t think is engineering at all, and most of the time I don’t either.

You can romanticise it: claim that software development isn’t craft, it’s art. And there is a measure of truth in this. It’s like writing, which is the other thing I spend a lot of time doing for money; that’s an art, too.

If you’re doing it right, at least.

Most of the writing that’s done, though, isn’t art. And most of the software development isn’t, either. Or most of the engineering. For every one person creating beauty in prose or code or steel, there are fifty just there doing the job with no emotional investment in what they’re doing at all. Honestly, that’s probably a good thing, and not a complaint. While I might like the theoretical idea of a world where everything is hand made by someone who cares, I don’t think that you should have to care in order to get paid. The people who are paying you don’t care, so you shouldn’t have to either.

It’s nice if you can swing it so you get both, though.

The problem is that it’s not possible to instruct someone to give a damn. You can’t regulate the UK government into giving a damn about people who fled a war to come here to find their dream of being a nurse, you can’t regulate Apple bosses into giving a damn about the open web, you can’t regulate CEOs into giving a damn about their employees or governments about their citizens or landlords about their tenants. That’s not what regulation is for; people who give a damn largely don’t need regulation because they want to do the right thing. They might need a little steering into knowing what the right thing is, but that’s not the same.

No, regulation is there as a reluctant compromise: since you can’t make people care, the best you can do is in some rough and ready fashion make them behave in a similar way to the way they would if they did. Of course, this is why the most insidious kind of response is not an attempt to evade responsibility but an attack on the system of regulation itself. Call judges saboteurs or protesters criminals or insurgents patriots. And why the most heinous betrayal is one done in the name of the very thing you’re destroying. Claim to represent the will of the people while hurting those people. Claim to be defending the law while hiding violence and murder behind a badge. Claim privacy as a shield for surveillance or for exclusion. We all sorta thought that the system could protect us, that those with the power could be trusted to use it at least a little responsibly. And the last year has been one more in a succession of years demonstrating just how wrong that is. This and no other is the root from which a tyrant springs; when he first appears he is a protector.

The worst thing about it is that the urge to protect other people is not only real but the best thing about ourselves. When it’s actually real. Look after others, especially those who need it, and look after yourself, because you’re one of the people who needs it.

Chris from Clickspring polishes things to a high shine using tin, which surprised me. I thought bringing out the beauty in something needed a soft cloth but no, it’s done with metal. Some things, like silver, are basically shiny with almost no effort; there’s a reason people have prized silver things since before we could even write down why, and it’s not just because you could find lumps of it lying around the place with no need to build a smelting furnace. Silver looks good, and makes you look good in turn. Tin is useful, and it helps polish other things to a high shine.

Today’s my 48th birthday. A highly composite number. The ways Torah wisdom is acquired. And somewhere between silver and tin. That sounds OK to me.

I’ve recently switched back from vscode to Sublime Text, which means that after all the time I spent training my fingers to type “code somefile.txt” instead of “subl somefile.txt” I now need to undo all that conditioning and go back to subl again. So I thought, hey, maybe I should dump a little shell script called code in my bin folder which admonished me in some amusing way, thus Pavlov-ing myself into learning to do it right.

And then I thought, hey, what’d be cool is if I had that Matrix-esque “raining code” effect in the Terminal and then it was superimposed with a box saying “STOPTYPING code ANDUSE subl INSTEAD”, like the “SYSTEMERROR” message at the end of the first movie.

And then I thought: someone’s already done this, right? And they have; it is called cmatrix. But I don’t like cmatrix because it doesn’t do the colours right; the text just sorta stops rather than fading away like the movie does, and it feels unreal and too sharp for me. Now, don’t get me wrong, I understand why this is; terminals support a full proper range of colour these days, but writing a program which gets released to actual people and which can deal with the bewildering array of terminal settings out there is a miserable waste of everyone’s time. But I’m not writing this for anyone else; it only has to work in my terminal (in true works on my machine fashion). And this will give me a chance to noodle about with Python terminal libraries such as blessed to make something interesting. Hence, matrix24.py:

It’s a bodge all round, and it still doesn’t look right, and Jess pointed out that making something cool happen when I make a mistake is the opposite of conditioning, but I got to fiddle about with a new library for a bit, so that was fun. Can I do something productive now?

(title from a classic post about the Matrix which still makes me laugh even after all these years, although it is very unfair to Keanu Reeves who is a cool bloke and should be emulated in his approach to life)

Many years ago (2012!) I was invited to be part of "The Pastry Box Project", which described itself thus:

Each year, The Pastry Box Project gathers 30 people who are each influential in their field and asks them to share thoughts regarding what they do. Those thoughts are then published every day throughout the year at a rate of one per day, starting January 1st and ending December 31st.

It was interesting. Sadly, it's dropped off the web (as has its curator, Alex Duloz, as far as I can tell), but thankfully the Wayback Machine comes to the rescue once again.¹ I was quietly proud of some of the things I wrote there (and I was recently asked for a reference to a thing I said which the questioner couldn't find, which is what made me realise that the site's not around any more), so I thought I'd republish the stuff I wrote there, here, for ease of finding. This was all written in 2012, and the world has moved on in a few ways since then, a dozen years ago at time of writing, but... I think I'd still stand by most of this stuff. The posts are still at archive.org and you can get to and read other people's posts from there too, some of which are really good and worth your time. But here are mine, so I don't lose them again.

Tuesday, 18 December 2012

My daughter’s got a smartphone, because, well, everyone has. It has GPS on it, because, well, every one does. What this means is that she will never understand the concept of being lost.

Think about that for a second. She won’t ever even know what it means to be lost.

Every argument I have in the pub now goes for about ten minutes before someone says, right, we’ve spent long enough arguing now, someone look up the correct answer on Wikipedia. My daughter won’t ever understand the concept of not having a bit of information available, of being confused about a matter of fact.

A while back, it was decreed that telephone directories are not subject to copyright, that a list of phone numbers is “information alone without a minimum of original creativity” and therefore held no right of ownership.

What instant access to information has provided us is a world where all the simple matters of fact are now yours; free for the asking. Putting data on the internet is not a skill; it is drudgery, a mechanical task for robots. Ask yourself: why do you buy technical books? It’s not for the information inside: there is no tech book anywhere which actually reveals something which isn’t on the web already. It’s about the voice; about the way it’s written; about how interesting it is. And that is a skill. Matters of fact are not interesting — they’re useful, right enough, but not interesting. Making those facts available to everyone frees up authors, creators, makers to do authorial creative things. You don’t have to spend all your time collating stuff any more: now you can be Leonardo da Vinci all the time. Be beautiful. Appreciate the people who do things well, rather than just those who manage to do things at all. Prefer those people who make you laugh, or make you think, or make you throw your laptop out of a window with annoyance: who give you a strong reaction to their writing, or their speaking, or their work. Because information wanting to be free is what creates a world of creators. Next time someone wants to build a wall around their little garden, ask yourself: is what you’re paying for, with your time or your money or your personal information, something creative and wonderful? Or are they just mechanically collating information? I hope to spend 2013 enjoying the work of people who do something more than that.

Wednesday, 31 October 2012

Not everyone who works with technology loves technology. No, really, it’s true! Most of the people out there building stuff with web tech don’t attend conferences, don’t talk about WebGL in the pub, don’t write a blog with CSS3 “experiments” in it, don’t like what they do. It’s a job: come in at 9, go home at 5, don’t think about HTML outside those hours. Apparently 90% of the stuff in the universe is “dark matter”: undetectable, doesn’t interact with other matter, can’t be seen even with a really big telescope. Our “dark matter developers”, who aren’t part of the community, who barely even know that the community exists… how are we to help them? You can write all the A List Apart articles you like but dark matter developers don’t read it. And so everyone’s intranet is horrid and Internet-Explorer-specific and so the IE team have to maintain backwards compatibility with that and that hurts the web. What can we do to reach this huge group of people? Everyone’s written a book about web technologies, and books help, but books are dying. We want to get the word out about all the amazing things that are now possible to everyone: do we know how? Do we even have to care? The theory is that this stuff will “trickle down”, but that doesn’t work for economics: I’m not sure it works for @-moz-keyframes either.

Monday, 8 October 2012

The web moves really fast. How many times have you googled for a tutorial on or an example of something and found that the results, written six months or a year or two years ago, no longer work? The syntax has changed, or there’s a better way now, or it never worked right to begin with. You’ll hear people bemoaning this: trying to stop the web moving so quickly in order that knowledge about it doesn’t go out of date. But that ship’s sailed. This is the world we’ve built: it moves fast, and we have to just hat up and deal with it. So, how? How can we make sure that old and wrong advice doesn’t get found? It’s a difficult question, and I don’t think anyone’s seriously trying to answer it. We should try and think of a way.

Tuesday, 18 September 2012

Software isn’t always a solution to problems. If you’re a developer, everything generally looks like a nail: a nail which is solved by making a new bit of code. I’ve got half-finished mobile apps done for tracking my running with GPS, for telling me when to switch between running and walking, and… I’m still fat, because I’m writing software instead of going running. One of the big ideas behind computers was to automate repetitive and boring tasks, certainly, which means that it should work like this: identify a thing that needs doing, do it for a while, think “hm, a computer could do this more easily”, write a bit of software to do it. However, there’s too much premature optimisation going on, so it actually looks like this: identify a thing that needs doing, think “hm, I’m sure a computer would be able to do this more easily”, write a bit of software to do it. See the difference? If the software never gets finished, then in the first approach the thing still gets done. Don’t always reach for the keyboard: sometimes it’s better to reach for Post-It notes, or your running shoes.

Saturday, 18 August 2012

Changing the world is within your grasp.

This is not necessarily a good thing.

If you go around and talk to normal people, it becomes clear that, weirdly, they don’t ever imagine how to get ten million dollars. They don’t think about new ways to redesign a saucepan or the buttons in their car. They don’t contemplate why sending a parcel is slow and how it could be a slicker process. They don’t think about ways to change the world.

I find it hard to talk to someone who doesn’t think like that.

To an engineer, the world is a toy box full of sub-optimized and feature-poor toys, as Scott Adams once put it. To a designer, the world is full of bad design. And to both, it is not only possible but at a high level obvious how to (a) fix it (b) for everyone (c) and make a few million out of doing so.

At first, this seems a blessing: you can see how the world could be better! And make it happen!

Then it’s a curse. Those normal people I mentioned? Short of winning the lottery or Great Uncle Brewster dying, there’s no possibility of becoming a multi-millionaire, and so they’re not thinking about it. Doors that have a handle on them but say “Push” are not a source of distress. Wrong kerning in signs is not like sandpaper on their nerves.

The curse of being able to change the world is… the frustration that you have so far failed to do so.

Perhaps there is a Zen thing here. Some people have managed it. Maybe you have. So the world is better, and that’s a good thing all by itself, right?

Friday, 27 July 2012

The best systems are built by people who can accept that no-one will ever know how hard it was to do, and who therefore don’t seek validation by explaining to everyone how hard it was to do.

Tuesday, 12 June 2012

The most poisonous idea in the world is when you’re told that something which achieved success through lots of hard work actually got there just because it was excellent.

Friday, 18 May 2012

Ever notice how the things you slave over and work crushingly hard on get less attention, sometimes, than the amusing things you threw together in a couple of evenings?

I can't decide whether this is a good thing or not.

Thursday, 5 April 2012

It's OK to not want to build websites for everybody and every browser. Making something which is super-dynamic in Chrome 18 and also works excellently in w3m is jolly hard work, and a lot of the time you might well be justified in thinking it's not worth it. If your site stats, or your belief, or your prediction of the market's direction, or your favourite pundit tell you that the best use of your time is to only support browsers with querySelector, or only support browsers with JavaScript, or only support WebKit, or only support iOS Safari, then that's a reasonable decision to make; don't let anyone else tell you what your relationship with your users and customers and clients is, because you know better than them.

Just don't confuse what you're doing with supporting "the web". State your assumptions up front. Own your decisions, and be prepared to back them up, for your project. If you're building something which doesn't work in IE6, that requires JavaScript, that requires mobile WebKit, that requires Opera Mobile, then you are letting some people down. That's OK; you've decided to do that. But your view's no more valid than theirs, for a project you didn't build. Make your decisions, and state what the axioms you worked from were, and then everyone else can judge whether what you care about is what they care about. Just don't push your view as being what everyone else should do, and we'll all be fine.

Sunday, 18 March 2012

Publish and be damned, said the Duke of Wellington; these days, in between starting wars in France and being sick of everyone repeating the jokes about his name from Blackadder, he’d probably say that we should publish or be damned. If you’re anything like me, you’ve got folders full of little experiments that you never got around to finishing or that didn’t pan out. Put ’em up somewhere. These things are useful.

Twitter, autobiographies, collections of letters from authors, all these have shown us that the minutiae can be as fascinating as carefully curated and sieved and measured writings, and who knows what you’ll inspire the next person to do from the germ of one of your ideas?

Monday, 27 February 2012

There's a lot to think about when you're building something on the web. Is it accessible? How do I handle translations of the text? Is the design OK on a 320px-wide screen? On a 2320px-wide screen? Does it work in IE8? In Android 4.0? In Opera Mini? Have I minimized the number of HTTP requests my page requires? Is my JavaScript minified? Are my images responsive? Is Google Analytics hooked up properly? AdSense? Am I handling Unicode text properly? Avoiding CSRF? XSS? Have I encoded my videos correctly? Crushed my pngs? Made a print stylesheet?

We've come a long way since:

<HEADER>
<TITLE>The World Wide Web project</TITLE>
<NEXTID N="55">
</HEADER>
<BODY>
<H1>World Wide Web</H1>The WorldWideWeb (W3) is a wide-area<A
NAME=0 HREF="WhatIs.html">
hypermedia</A> information retrieval
initiative aiming to give universal
access to a large universe of documents.

Look at http://html5boilerplate.com/—a base level page which helps you to cover some (nowhere near all) of the above list of things to care about (and the rest of the things you need to care about too, which are the other 90% of the list). A year in development, 900 sets of changes and evolutions from the initial version, seven separate files. That's not over-engineering; that's what you need to know to build things these days.

The important point is: one of the skills in our game is knowing what you don't need to do right now but still leaving the door open for you to do it later. If you become the next Facebook then you will have to care about all these things; initially you may not. You don't have to build them all on day one: that is over-engineering. But you, designer, developer, translator, evangelist, web person, do have to understand what they all mean. And you do have to be able to layer them on later without having to tear everything up and start again. Feel guilty that you're not addressing all this stuff in the first release if necessary, but you should feel a lot guiltier if you didn't think of some of it.

Wednesday, 18 January 2012

Don't be creative. Be a creator. No one ever looks back and wishes that they'd given the world less stuff.

1. Also, the writing is all archived at Github!↩

I recently discovered that there's an old software edition of the Oxford English Dictionary (the second edition) on archive.org for download. Not sure how legal this is, mind, but I thought it would be useful to get it running on my Ubuntu machine. So here's how I did that.

Firstly, download the file; that will give you a file called Oxford English Dictionary (Second Edition).iso, which is a CD image. We want to unpack that, and usefully there is 7zip in the Ubuntu archives which knows how to unpack ISO files.¹ So, unpack the ISO with 7z x "Oxford English Dictionary (Second Edition).iso". That will give you two more files: OED2.DAT and SETUP.EXE. The .DAT file is, I think, all the dictionary entries in some sort of binary format (and is 600MB, so be sure you have the space for it). You can then run wine SETUP.EXE, which will install the software using wine, and that's all good.² Choose a folder to install it in (I chose the same folder that SETUP.EXE is in, at which point it will create an OED subfolder in there and unpack a bunch of files into it, including OED.EXE).

That's the easy part. However, it won't quite work yet. You can see this by running wine OED/OED.EXE. It should start up OK, and then complain that there's no CDROM.

This is because it expects there to be a CDROM drive with the OED2.DAT file on it. We can set one up, though; we tell Wine to pretend that there's a CD drive connected, and what's on it. Run winecfg, and in the Drives tab, press Add… to add a new drive. I chose D: (which is a common Windows drive letter for a CD drive), and OK. Select your newly added D: drive and set the Path to be the folder where OED2.DAT is (which is wherever you unpacked the ISO file). Then say Show Advanced and change the drive Type to CD-ROM to tell Wine that you want this new drive to appear to be a CD. Say OK.

Now, when you wine OED/OED.EXE again, it should start up fine! Hooray, we're done! Except…

…that's not good. The app runs, but it looks like it's having font issues. (In particular, you can select and copy the text, even though it looks like a bunch of little squares, and if you paste that text into somewhere else it's real text! So this is some sort of font display problem.)

Fortunately, the OED app does actually come with the fonts it needs. Unfortunately, it seems to unpack them to somewhere (C:\WINDOWS\SYSTEM)³ that Wine doesn't appear to actually look at. What we need to do is to install those font files so Linux knows about them. You could click them all to install them, but there's a quicker way; copy them, from where the installer puts them, into our own font folder.

To do this...

• first make a new folder to put them in: mkdir ~/.local/share/fonts/oed.
• Then find out where the installer put the font files, as a real path on our Linux filesystem: winepath -u "C:/WINDOWS/SYSTEM". Let's say that that ends up being /home/you/.wine/dosdevices/c:/windows/system
• Copy the TTF files from that folder (remembering to change the first path to the one that winepath output just now): cp /home/you/.wine/dosdevices/c:/windows/system/*.TTF ~/.local/share/fonts/oed
• And tell the font system that we've added a bunch of new fonts: fc-cache

And now it all ought to work! Run wine OED/OED.EXE one last time…

1. and using 7zip is much easier than mounting the ISO file as a loopback thing ↩
2. There's a Microsoft Word macro that it offers to install; I didn't want that, and I have no idea whether it works ↩
3. which we can find out from OED/INSTALL.LOG↩

I nerd-sniped myself.

Internet: I am disappointed that you have not provided me with one of those meme-maker websites for a Family Fortunes board. Now I will have to make one.

See, I put word to thought and that was my mistake. So I spent, like, half my Saturday doing this instead of what I should be doing.

I got to use a couple of interesting techniques with it that I hadn't done before, now that nice new web APIs exist for them: sharing an image, for example, which involves creating a Blob (thank you Thomas Steiner), doing the equivalent of the venerable "yellow fade technique" with the web animations API, and yoinking all the information from a <form> conveniently by reading new FormData(form) in form.oninput. This is all cool stuff; the web's such a nice environment to program for.

A number of people have pointed out that this is not emulating the particular style of Family Fortunes board that they prefer, to which our survey says: feel free to make your own. This is the one I grew up with; those of you of more venerable years who prefer Max Bygraves or Bob Monkhouse to Les Dennis will simply have to struggle on under the burden. (Let us not speak of the appalling colourful modern video wall, which is just literally no fun at all and looks like a pub quiz machine.) Also, it turns out that Quite A Lot Of People have Opinions on what the exact shape of the X's should be, and no two of these opinions are the same. All good clean fun, yes.

Anyway, I can't think that this is ever likely to be all that important, but if you ever need a way to mock up a Family Fortunes¹ screen from the 1980s with your own custom answers and scores, we asked 100 people and they all said that kryogenix.org/code/family-fortunes is right there waiting patiently for you. Enjoy.

1. I think Americans call it Family Feud? ↩

Here's the tl;dr: if you make web apps in or for the UK, the CMA, the UK tech regulator, want to hear from you about their proposals before August 29th 2024, which is only a week away. Read their list of remedies to anticompetitive behaviour between web browsers and platforms, and email your thoughts to browsersandcloud@cma.gov.uk before the deadline. They really do want to hear from you, confidentally if you want, and your voice is useful here; you don't need to have some formally written legal opinion here. They want to hear from actual web devs and companies. Email them.

Now let's look at what the CMA have written in a little more detail. (This is the "tl" bit, although hopefully you will choose to "r".) They have been conducting a "Market Investigation Reference", which is regulator code for "talk to loads of people to see if there's a problem and then decide what to do about that", and the one we care about is about web browsers. I have, as part of Open Web Advocacy, been part of those consultations a number of times, and they've always been very willing to listen, and they do seem to identify a bunch of problems with browser diversity that I personally also think are problems. You know what we're talking about here: all browsers on iOS are required to be Safari's WebKit and not their own engine; Google have a tight grip on a whole bunch of stuff; browser diversity is a good thing and there's not enough of it in the world and this looks to be from deliberate attempts to act like monopolies by some players. These are the sorts of issues that CMA are concerned about (and they have published quite a few working papers explaining their issues in detail which you can read). What we're looking at today is their proposed list of fixes for these problems, which they call "remedies". At OWA we have also done this, of course, and you should read the OWA blog post about the CMA's remedies paper. But the first important point is, to be clear, that a whole bunch of these remedies being proposed by the CMA are good. This is not a complaint that it's all bad or that it's toothless, not at all. They're going to stop the #AppleBrowserBan and require that other browsers are allowed to use their own engines on iOS as browser apps and in in-app browsing, they're going to require both Apple and Google to grant other browsers' access to the same APIs that their own browsers can get at, they've got suggestions in place for how users can choose which browser they use to get past the problem of the "default hotseat" where you get one browser when you buy a phone and never think to change it, they're suggesting that Google open access to WebAPK minting to everyone. All of these help demonopolise the UK market. This is all good.

But there are some places where their remedies don't really go far enough, and this is the stuff where it would be good for you, my web-engaged reader, to send them an email with your thoughts one way or the other. Part of making the web as capable as a platform-specific app is that web sites can be presented like a platform-specific app. This is (part of) what a PWA is, which most of you reading this will already know about. But releasing your app as a PWA, while it has a bunch of good characteristics for you (no reviews needed, instant updates, full control, cross-platform development, no tithing of money required to the people who sold the phone it's on) also has some downsides. In particular, it's difficult to get people to "install" a PWA, especially on iOS where you have to tell your users to go digging around in the share menu. And this is a fairly big area where the CMA could have proposed remedies, and have so far not chosen to. The first problem here is that iOS Safari doesn't support any sort of install prompt: as mentioned, there's the "add to home screen" entry hidden in the share menu. There's an API for this, everyone else implements it, iOS Safari doesn't. Maybe the API's got problems and needs fixing? That seems fine; engage with the web standards process to get it fixed! But there's been no sign of doing that in any useful way.

The second and related issue is that although the CMA's remedies state that browsers can use their own engine rather than having to be mere wrappers around the platform's web view, they do not say that when a browser installs a web app, that that web app will also use that browser's engine. That is: if there were a port of, say, Microsoft Edge to iOS, then Edge would be able to use its own engine, which is Microsoft's port of Blink. That Edge browser can implement install prompts how it likes, because it's using its own engine. But there's no guarantee in the CMA remedies that the PWA that gets installed will then be opened up in Edge. Calling the "install this PWA as an app" API provided by the platform might add it as a PWA in the platform maker's browser -- iOS Safari in this example. This would be rubbish. It means that the installed app might not even work; how will it know your passwords or cookies, etc; this can't be what's intended. But the remedies do not explicitly state this requirement, and so it's quite possible that platform owners will therefore use this as another way to push back against PWAs to make them less of a competitor to their own app stores. I would like to be able to say that platform owners wouldn't do that, that they wouldn't deliberately make things worse in an effort at malicious compliance, but after the debacle earlier this year of Apple dropping PWA support entirely and then only backing off on that after public outcry, we can't assume that there will be a good-faith attempt to improve PWA support (either by implementation, or by engaging wholeheartedly with the public web standards process), and so the remedies need to spell this out in more detail. This should be easy enough if I'm right and the CMA's intent is that this should be done, and your voice adding to that is likely to encourage them.

The worry about malicious compliance hampering web apps being a proper competitor to platform-specific apps also extends to another thing missing in the remedies: that access to hardware and software platform APIs for other browsers isn't required to be "which APIs there are", but "which APIs the existing browser elects to use". That is: if you write a native platform-specific app, it can talk to various hardware-ish things; bluetooth, USB, NFC, whichever. Therefore, you ought to be able, if you're a browser, to also have those APIs, in enough detail that you can then offer (mediated, secure) access to those services to your users, the PWAs and websites that people run with you. But the remedies do not ensure that this is the case; they ensure that there is a "requirement for Apple to grant equivalent access to APIs used by WebKit and Safari to browsers using alternative browser engines." What this means is that if Safari doesn't use a thing, no other browser can use it either. So it's not possible to make the browser you build better than Safari at this; Apple get to set the ceiling of what the web can do on the platform, and the ceiling is defined as "whatever their browser can do". That's... not very competitive, is it? No. If you agree with that, then you should also write to the CMA about it. They would like to hear about actual examples where this sort of thing harms UK businesses, of course, and if that's you definitely write in, but it's also worth giving your opinion if you are a UK developer who works in this area, delivering things via the web to your users (or if you want to do that but are currently prevented).

OK. Discussion over: go and write to the CMA with your thoughts on their remedies. Even if you think what they've proposed is perfect, you should still send them a message saying that; one thing that they and all government agencies tend to bemoan is that they only hear from people with lots of skin in the game, and generally only from people who are opposed, not people who are supportive. That means that they get a skewed view of what the web developer community actually think, and this is a chance for us to unskew that a bit, together. You can request that the CMA keep your name, business, or submission confidential, so you don't have to worry about giving away secrets or your participation, and you need only comment on stuff which is relevant to you; you do not need a comprehensive position paper on the whole thing! The address to email is browsersandcloud@cma.gov.uk, the list of remedies is Working Paper 7, and the deadline is Thursday 29th August.

If you want to hear more about this, then I am speaking about OWA, how it happened, what we've done, and how you can be involved at State of the Browser 2024 on Saturday 14th September (just under a month from now!) in the Barbican in London. I'm told that there are less than 30 in-person tickets left, although there are online streaming tickets available too, so get in quick if you want to hear me and a whole bunch of other speakers!

(Late breaking update: Bruce has also written about this and you should read that too!)

Recently I have been most baffled by how on my iPhone, my iCloud backup was over 5GB (and therefore would not back up into the free space that I have) despite how I had turned off most of the apps that might want to be included in the backup.

There are many, many, many posts on the internet from people having this problem, and there are a few common things which come up. The first one is that iMessage is included in the backup, and this includes any images or videos you've received or sent. The second is that your photos are included. So if you're thinking "hey there's hardly anything on my phone, why is the iCloud backup so big" but you've got 2 years worth of chats with a zillion people full of videos... that's why.

I, however, had tried all that and I still couldn't get the backup size down. I even spoke to Apple suport directly about it, three times; each of the people I spoke to had helpful suggestions, but it was also equally clear that each of them was fishing around in the dark, because all the "usual" tricks and traps they knew about which caused this were things that I'd already dealt with or disabled. It all ended up inconclusive, and I still didn't have a backup.

Today, in desperation, I decided to try backing up the phone to my Linux machine so I could poke about in the backup to see if I could tell what was using all the space. Now, iPhones can be backed up to Macs (not surprisingly) and Windows, but there's no official provision for doing so on Linux, sadly. However, there is libimobiledevice, a set of command line tools. I used them to back up my phone to my desktop as follows:

1. plug the phone in
2. idevicebackup2 cloud off # disable iCloud backup
3. idevicebackup2 -i encryption on # enter a password
4. idevicebackup2 backup ./ # back up phone into current dir

Once I'd done that, I had a folder named for the UDID of my phone, filled with encrypted data. Fortunately, there is a Python library called iOSbackup which knows how to read and decrypt these backups, so I used it to write myself a little equivalent of the du utility to see which folders in the backup might be unexpectedly using a lot more storage than I expected.

And in fact there were a whole bunch of folders called something like Library/WebClips/(long string).webclip/ which were using tons of storage, some over a gigabyte. I immediately thought: hey, I bet they're Home Screen web apps. I use a lot of these -- if there's a PWA for a service, I'll use it rather than a platform-specific app. We set up Open Web Advocacy for a reason, after all. So this made me jump to a (what turned out to be correct) conclusion from a standing start. Each of these Library/WebClips/blah.webclip folders contains an ApplicationManifest file, and you can get iOSbackup to disgorge its decrypted content; it's a "binary plist" (which Python knows how to read) and with that I could see which Home Screen web apps were taking up space with this little script:

fromiOSbackupimportiOSbackupimportplistlibimportosimportjsonUDID="ENTER YOUR UDID HERE (the backup folder name)"PASSWORD="BACKUP ENCRYPTION PASSWORD"FOLDER="2024-09-17"# folder you put the backup inb=iOSbackup(udid=UDID,cleartextpassword=PASSWORD,backuproot=FOLDER)# https://stackoverflow.com/a/53567149/1418014 thanks!defformatSize(sizeInBytes,decimalNum=1,isUnitWithI=False,sizeUnitSeperator=""):"""format size to human readable string"""# K=kilo, M=mega, G=giga, T=tera, P=peta, # E=exa, Z=zetta, Y=yottasizeUnitList=['','K','M','G','T','P','E','Z']largestUnit='Y'ifisUnitWithI:sizeUnitListWithI=[]forcurIdx,eachUnitinenumerate(sizeUnitList):unitWithI=eachUnitifcurIdx>=1:unitWithI+='i'sizeUnitListWithI.append(unitWithI)sizeUnitList=sizeUnitListWithIlargestUnit+='i'suffix="B"decimalFormat="."+str(decimalNum)+"f"# ".1f"finalFormat=("%"+decimalFormat+sizeUnitSeperator+"%s%s")# "%.1f%s%s"sizeNum=sizeInBytesforsizeUnitinsizeUnitList:ifabs(sizeNum)<1024.0:returnfinalFormat%(sizeNum,sizeUnit,suffix)sizeNum/=1024.0returnfinalFormat%(sizeNum,largestUnit,suffix)webapp_sizes={}forfileinb.getBackupFilesList():ifnotfile["name"].startswith("Library/WebClips/"):continuewebclip_folder=file["name"].split("/")[2]ifwebclip_foldernotinwebapp_sizes:webapp_sizes[webclip_folder]={"size":0,"name":None}# work out where this file is in the backupbackup_file_loc=os.path.join(FOLDER,UDID,file["backupFile"][:2],file["backupFile"])try:# technically this is accumulating the encrypted# size of the file, not the decrypted. But it's finebf_size=os.stat(backup_file_loc).st_sizewebapp_sizes[webclip_folder]["size"]+=bf_sizeexceptFileNotFoundError:continueiffile["name"].endswith("/ApplicationManifest"):# decrypt it to a temp location# you should be doing this with tempfiledec=b.getFileDecryptedCopy(relativePath=file["name"],targetFolder="/tmp",targetName="iosdec")withopen("/tmp/iosdec",mode="rb")asfp:data=fp.read()am=plistlib.loads(data)# go looking for the first one which looks like JSONforiteminam["$objects"]:iftype(item)isnotstr:continueelifitem.startswith("https://"):webapp_name=itemelse:try:manifest_json=json.loads(item)except:continuewebapp_name=manifest_json.get("name",manifest_json.get("short_name","?"))webapp_sizes[webclip_folder]["name"]=webapp_namebreakforvinwebapp_sizes.values():ifv["size"]<50ornotv["name"]:continueprint(f"{v['name']}: {formatSize(v['size'])}")

and this helpfully printed a list which looked like this (but longer; I've kept a few around to give you the flavour):

https://elk.zone/: 133.1MB
https://squoosh.app/: 1.1MB
https://www.kryogenix.org/farmbound/: 11.5MB
https://nerdlegame.com/: 220.9MB
https://twitter.com/: 1.2GB
Phanpy: 1.6GB
https://www.nytimes.com/games/: 376.2MB

So... aha. Twitter and Phanpy can, I suppose, be excused since they are presumably caching every post ever, but I can delete those and re-add (and not bother re-adding Twitter) to get some of that back. Wordle, you are the weakest link, goodbye, and also I don't need elk.zone any more now I'm using Phanpy.

I removed a bunch of these. Then I told the iCloud Backup to run again. And now my backup size is 800MB, not 5GB. Hooray!

To be clear, this is not at all a Safari problem. Safari is absolutely doing the right thing here; well done Safari team. Web apps are apps; they should be included in my phone backup, 100%. The bug here is in the iCloud Backup Settings App List, which lists all the apps that are taking up space in the backup but does not list Home Screen web apps. This sucks, and it's a bug, and it should be fixed. Show me PWAs in the backup list, especially ones taking up a gigabyte of space in it. I have filed the bug at feedbackassistant.apple.com although I've never heard back from any of the others I've filed there so I don't really know what the process is.

OK, now off to add Phanpy again.

In a land far away, there were two brothers, two plumbers. To preserve their anonymity, we'll call them... Mario and Luigi. Their mother, a kind and friendly woman, and their father, a man with (by the laws of averages and genetics) a truly gargantuan moustache, raised them both to be kind and friendly (and moustachioed) in their turn. There was enough work in the town to keep both the plumbers busy, and they each grew through apprentice to journeyman to experience and everyone liked them. They both cared about the job, about their clients, and they each did good work, always going the extra mile, doing more than was necessarily asked for, putting in an extra hour to tighten that pipe or fit a better S-bend or clean up the poor workmanship of lesser craftsmen and cowboys. They were happy. Even their rivalry for each job was good-humoured, a friendly source of amusement to them and to the town. Sometimes people would flip a coin to choose which to ring, having no way to choose between them, and Mario would laugh and suggest that he should have two-headed coins made, or Luigi would laugh and say that that ought to make it his turn next.

But there came a time of downturn, when the people of the town had to hold tighter to their purses, and fewer called out for plumbers. And Luigi, after much thought, decided to take a job with Bowser's, the big plumbing conglomerate from the city. He was worried: the big company were often slapdash or inexperienced in their work, and discourteous or evasive to their clients, and more interested in bottom lines than hot water lines. But they paid extremely well, and they had the latest tools, and there was security in having a contract and a title and a boss. Besides, Bowser's worked for so many more people that Luigi's own skills could only help that many more. Maybe he could even teach them something about quality, and craftsmanship, and care. He suggested to Mario that they both joined, and Mario thought hard about it, and eventually decided not to, though it was a close-run thing. Both the brothers shook hands on it, respecting one another's decision, although in the silence of their hearts each was a little disappointed in the other.

Luigi did well at Bowser's. He was right about the latest tools, and about the pay, and about the security. And he was partially right about teaching the big company something about quality. His work was often better than his colleagues, sometimes through expertise but most often because he tried harder: he loved the work, and wanted to do well, and was kind and friendly when he could be. But sometimes, try though he might, the time wasn't there, or the parts weren't in the van, and these things were not his fault; someone else at the big company had cut corners on their job and that forced Luigi to cut corners on his and make people sad and angry, or put in more time to fix it than he would have spent doing it all correctly himself in the first place. He pushed hard inside the company to fix these things, and he had some successes; a policy was written suggesting that employees work harder to improve customer happiness, and many customers across the land were made a little happier as a result. Luigi won an award. He trained some apprentices, and many of his little ways of making people happier and the job better were adopted into the company training scheme. One time he went home after another argument with his boss about the things that were not adopted, and that night he looked enviously out of their window at his brother's house across the street, thinking that it would be a fine thing to not have a boss who stopped you from doing things right.

Mario did well working for himself. The time of downturn ended and things began to pick up again, maybe not quite to where they had been but nearly there for all that, and the phone calls and messages came in once more. Everyone was pleased to see him, and although he maybe took a little longer than the men from the big company, his work was never slapdash, always taking the time to do it right. And he had less money, but he really didn't mind, or begrudge it; he had enough to get by, and he loved the work, and wanted to do well, and was kind and friendly. He did envy his brother's toolbox, though, all the latest gear while Mario himself made do with things a little older, a little rustier, but they were all good quality tools that he understood, and the work was as good and better. In November one year a very expensive plumber's inspection camera was stolen from his brother's van, and Mario thought that it would have been great to have such a thing and maybe he would have taken better care, and then he felt guilty about thinking that of his brother. He felt guiltier still when on Christmas morning he opened the box from Luigi to find an expensive inspection camera in it. But then his brother winked at him and put a finger to his lips, and all was well between them again. One time Mario was up to his waist in the drain outside a house, raindrops rattling on his hat and cursing the god who invented backflow, when he saw his brother drive past all unknowing in his modern van, windows wound up and singing along with the radio, and he looked enviously after the van's lights in the storm, thinking that it would be a fine thing to have just a notch more comfort and influence and two fewer wet knees.

I have a problematic relationship with keys.

Well, that's not true. I have a problematic relationship with key rings. For some reason, my pockets are a violently hostile environment for things I put in them. I don't really understand why this is, but it's true. Keyrings bend out of shape; the concentric rings separate, and my actual keys fall off of them. People have expressed scepticism about this in the past, and they've been wrong and I've been right. The last time I complained about this, I thought I'd come up with a solution where I bought a keyring which was a tiny padlock. It lasted three days before a bolt sheared off. You can see the whole thing on posts made to twitter.

At that point most people would give up, or be sad, or just live with split rings continually letting them down. But most people don't have a dad who is a king of engineering.

My dad made me this.

It's a keyring. It's a solid block of brass with the middle cut out, so it looks like a very shallow "U", or like three sides of a long rectangle. There's a hole drilled in each of the short ends, and a long bolt is threaded through those holes. On one end of the bolt is a nut, tight against the outside of the "U", and the bolt protrudes out about an inch where there's another, locking, nut. All the keys are hung from the bolt. To add a new key, I undo the locking nut on the end, undo the tight nut, pull the bolt out, hang another key from it, and then do everything back up. It's brilliant. I've not had a single problem with it.

Those of you carefully studying the picture will notice that there is writing on the brass "U". (And will also notice that I've blacked out the details of the actual keys, because you can cut a key based on a picture, and I'm not stupid.) That's engraving, which mentions my website, so if I lose my keys (which I am really careful to not do¹) then whoever finds them can get in touch with me to tell me that happened but my address is not on the keys, so a nefarious finder gets less benefit from it.

I love my keyring. It's the best. I do not know why more keyrings are not like this. It works just like a normal keyring (I have non-key things on mine, such as a USB stick and a tiny flashlight, but they would go on a regular split-ring keyring as well), but it doesn't just fail all the time like normal ones do. I surely can't be the only person who experiences this? Anyway, I don't mind, 'cos I have the solution. Cheers, dad. Maybe I should make this a product or something.

1. the historical version of checking your pockets, as a man, was to feel for "spectacles, testicles, wallet, and watch" -- this was actually a ribald mnemonic for how to cross yourself as a Catholic, but this modern day man checks his pockets for keys, wallet, and phone in the same way to check they're not lost ↩

The sum of the digits of the square of 49 (2401) is the square root of 49.

49 is the first square where the digits are squares. In this case, 4 and 9 are square numbers.

It seems that 49 is an age of squares.

I find myself increasingly OK with this.

It is interesting, although really quite disillusioning, reading through the series of posts I've done on my birthday (follow the links back from last year for 22 years now) and seeing the trend in recent years. The world has got more worrying. A lot more. I hate this.

So I've tried to find joy where I can get it. I'll give you an example. I looked up the number 49 for this, the same as I always do, and apparently 49 is the smallest discriminant of a totally real cubic field. Now, I'm sure that there are maths people out there who understand this. But to me, it is impossible to read this without putting on a mock Valley bro accent: yeah brah, this field is todally real, yah? And that made me smile. Bill dropped a birthday message on me at half seven this morning. Yesterday, I remembered a throwaway joke line that popey dropped on me in 2015 and I literally laughed out loud in my own living room at the thought of it. Two days ago Jess and I went to see Murder on the Orient Express on stage at the Alexandra Theatre. Tonight I've been for beers with Bruce and Matt. My D&D group are about to confront Harazos to let them perform a ritual they've made up. Everyone in my team at Barnardo's was really nice about my birthday. None of this helps fix the world crashing down around us, but all of it certainly makes it easier to cope with, because all of it is a little bit of joy, of niceness, in the midst of unjoy and unniceness. Thank you, all of you.

This has been not a great year. There's been ups and there's been downs. I'm still a lot better off than almost everyone -- I'm not under threat, I'm not more under threat than I was before, I can afford to do what I like as long as what I like isn't too extravagant. But it has nonetheless been tough. Maybe 2025 will be better, although to be honest I doubt it. And I have this sense of regret, not about the world getting worse, but that the idea that the world might get better has basically gone away. When I was younger, the thing was that the world would get better, year on year, as time passed. And I am 100% sure that this was a thing that I got to think because I was privileged to not be being screwed over by the world. But I still had that thought, that things were OK today and would be slightly more than OK tomorrow, and at some point that went away. Things probably won't be better tomorrow. If they're the same, that probably in itself is a victory. This isn't really my tragedy; it's for people younger than me, who haven't experienced this change, who assume that things getting steadily more grey and sad and worse is just the way the world is; that's just normal.

So I look for little bits of joy. Hopefully you have some little bits of joy yourself, as I do. Happy birthday to you, even if it's not your birthday. Eat some cake.

The latest thing circulating around people still blogging is the Blog Questions Challenge; Jon did it (and asked if I was) and so have Jeremy and Ethan and a bunch of others, so clearly it is time I should get on board, fractionally late as ever.¹

Why did you start blogging in the first place?

Some other people I admired were doing it. I think the person I was most influenced by to start doing it was Simon Willison, who is also still at it², but a whole bunch of people got on board at around that same time, back in the early days when you be a medium-sized fish in a small pool just by participating. Mark Pilgrim springs to mind as well -- that's a good example of having influence, when the "standard format" of permalinks got sort of hashed out collectively to be /2025/02/03/blog-questions-challenge, which a lot of places still adhere to (although it feels faintly quaint, these days).

Interestingly, a lot of the early posts on this site are short two-sentence half-paragraph things, throwaway thoughts, and that all got sucked up by social media... but social media hadn't been invented, back in 2002.

Also interestingly: the second post on this here blog³ was bitching at Mozilla about the Firefox release schedule. Nothing new under the sun.⁴

What platform are you using to manage your blog and why did you choose it? Have you blogged on other platforms before?

Cor. When it started, this site was being run by Castalian, which was basically "classic ASP but Python instead of VBScript", a thing I built. This is because I was using ASP at work on Windows machines, so that was the model for "dynamic web pages" that I understood, but I wasn't on Windows⁵ and so I built it myself. No idea if it still works and I very much doubt it since it's old enough to buy all the drinks these days.

After that it was Movable Type for a bit and then, because I'd discovered the idea of funky caching⁶ it was Vellum, that model (a) in Python and (b) written by me. Then for a while it was "Thort", which was based on CouchDB⁷, and then it was WordPress, and then in 2014 I switched from WP to a static build based on Pelican, which it still is to this day. Crikey, that was over ten years ago!⁸ I like static site generators: I even wrote 10 Popular Static Site Generators a few years ago for WebsiteSetup which I think is still pretty good.

How do you write your posts? For example, in a local editing tool, or in a panel/dashboard that’s part of your blog?

In my text editor, which is Sublime Text. The static setup is here on my machine; I write a post, I type make kryogenix, and it runs a whole little series of scripts which invoke Pelican to build the static HTML for the blog, do a few things that I've added (such as add footnote handling⁹, make og:image links and images¹⁰, and sort of handle webmentions but that's broken at the moment) and then copy it up to my actual website (via git) to be published.

It's all a bit lashed together, to be honest, but this whole website is like that. It is something like an ancient city, such as London or Rome; what this site is mostly built on is the ruins of the previous history of the city. Sometimes the older bits poke through because they're still actually OK, or they never got updated; sometimes they've been replaced with the new shiny. You should see the .htaccess file, which operates a bewildering set of redirects through about six different generations of URLs so all the old links still work.¹¹

When do you feel most inspired to write?

When the muse seizes me. Sometimes that's a lot; sometimes not. I do quite a lot of paid writing as part of my various day jobs for others, and quite a lot of creative writing as part of running a play-by-post D&D campaign, and that sucks up a reasonable amount of the writing energy, but there are things which just demand going on the website. Normally these days it's things where I want them to be a reference of some kind -- maybe of a useful tech thing, or some important thought, or something interesting -- for myself or for others.

Alternatively you might think the answer is "while in the pub, which leads to making random notes in an email to myself from my phone and then writing a blog post when I get home" and while this is not true, it's not not true either. I do not want to do a histogram of posting times from this site because I am worried that I will find that the majority are at, like, 11.15pm.

Do you publish immediately after writing, or do you let it simmer a bit as a draft?

Always post immediately. I have discovered about myself that, for semi-ephemeral stuff like posts here or projects that I do for fun, that I need to get them done as part of that initial burst of inspiration and energy. If I don't get it done, then my enthusiasm will fade and they will linger half-finished for ever and never get completed. I don't necessarily like this, but I've learned to live with it. If I think of an idea for a post and write a note about it and then don't do it, when I rediscover the note a week later it will not seem anything like as compelling. So posts are mostly written as one long stream-of-consciousness to capitalise on the burning of the creative fire before it gets doused by time or work or everything going on in the world. Carpe diem, I guess.¹²

What’s your favourite post on your blog?

Maybe It's Cold Outside, or Monkey Island 2, for about the fifth time, or Charles Paget Wade and the Underthing for writing, although each of them have little burrs in the wording that I want to polish when I re-read them. The series of birthday posts have been going on since the beginning, one every year, which probably wins for consistency. For technical stuff, maybe Some thoughts on soonsnap and little big details (now sadly defunct) or The thing and the whole of the thing: on DRM in HTML. I like my own writing, mostly. Arrogant, I know.

Any future plans for your blog? Maybe a redesign, a move to another platform, or adding a new feature?

Not really at the moment, but, as above, these things tend to arrive in a blizzard of excitement and implementation and then linger forever once done. But right now... it all seems to work OK. Ask me when I get back from the pub.

Next?

Well, I should probably point back at some of the people who inspired me to do this or other things and keep doing so to this day. So Simon, Remy, and Bruce, perhaps!

1. In my defence, it was my birthday. ↩
2. although no longer at simon.incutio.com -- what even was Incutio? ↩
3. I resisted the word "blog" for a long time, calling it a "weblog", and the activity being "weblogging", because "blog" is such an ugly word. Like most of the fights I was picking in the mid 2000s, this also seems faintly antiquated and passé now. Sic transit gloria mundi and all that. ↩
4. or "nihil sub sole novum", since we're doing Latin quotes today ↩
5. and Windows's relationship with Python has always been a bit unsteady, although it's better these days now that Microsoft are prepared to acknowledge that other people can have ideas ↩
6. you write the pages in an online form, but then a server process builds a static HTML version of them; the advanced version of this where pages were only built on request was called "funky caching" back then ↩
7. if a disinterested observer were to consider this progression, they might unfairly but accurately conclude that whatever this site runs on is basically a half-arsed system I built based on the latest thing I'm interested in, mightn't they? ↩
8. tempus fugit. OK, I'll stop now. ↩
9. like this! ↩
10. an idea I stole shamelessly from Zach Leatherman↩
11. Outgoing links are made to continue to work via unrot.link from the excellent Remy Sharp↩
12. I was lying about not doing this any more, obviously ↩

Everyone's got a newsletter these days (like everyone's got a podcast). In general, I think this is OK: instead of going through a middleman publisher, have a direct connection from you to the people who want to read what you say, so that that audience can't be taken away from you.

On the other hand, I don't actually like newsletters. I don't really like giving my email address to random people¹, and frankly an email app is not a great way to read long-form text! There are many apps which are a lot better at this.

There is a solution to this and the solution is called RSS. Andy Bell explains RSS and this is exactly how I read newsletters. If I want to read someone's newsletter and it's on Substack, or ghost.io, or buttondown.email, what I actually do is subscribe to their newsletter but what I'm actually subscribing to is their RSS feed. This sections off newsletter stuff into a completely separate app that I can catch up on when I've got the time, it means that the newsletter owner (or the site they're using) can't decide to "upsell" me on other stuff they do that I'm not interested in, and it's a better, nicer reading experience than my mail app.²

I use NetNewsWire on my iOS phone, but there are a bunch of other newsreader apps for every platform and you should choose whichever one you want. Andy lists a bunch, above.

The question, of course, then becomes: how do you find the RSS feed for a thing you want to read?³ Well, it turns out... you don't have to.

When you want to subscribe to a newsletter, you literally just put the web address of the newsletter itself into your RSS reader, and that reader will take care of finding the feed and subscribing to it, for you. It's magic. Hooray! I've tested this with substack, with ghost.io, with buttondown.email, and it works with all of them. You don't need to do anything.

If that doesn't work, then there is one neat alternative you can try, though. Kill The Newsletter will give you an email address for any site you name, and provide the incoming emails to that as an RSS feed. So, if you've found a newsletter which doesn't exist on the web (boo hiss!) and doesn't provide an RSS feed, then you go to KTN, it gives you some randomly-generated email address, you subscribe to the intransigent newsletter with that email address, and then you can subscribe to the resultant feed in your RSS reader. It's dead handy.

If you run a newsletter and it doesn't have an RSS feed and you want it to have, then have a look at whatever newsletter software you use; it will almost certainly provide a way to create one, and you might have to tick a box. (You might also want to complain to the software creators that that box wasn't ticked by default.) If you've got an RSS feed for the newsletter that you write, but putting your site's address into an RSS reader doesn't find that RSS feed, then what you need is RSS autodiscovery, which is the "magic" alluded to above; you add a line to your site's HTML in the <head> section which reads <link rel="alternate" type="application/rss+xml" title="RSS" href="https://URL/of/your/feed"> and then it'll work.

I like this. Read newsletters at my pace, in my choice of app, on my terms. More of that sort of thing.

1. despite how it's my business to do so and it's right there on the front page of the website, I know, I know ↩
2. Is all of this doable in my mail client? Sure. I could set up filters, put newsletters into their own folders/labels, etc. But that's working around a problem rather than solving it ↩
3. I suggested to Andy that he ought to write this post explaining how to do this and then realised that I should do it myself and stop being such a lazy snipe, so here it is ↩