Egregoros

@jonny @elebertus

I suspect testing has the same properties as translation. It’s moderately easy to build machine-translation systems that are kind-of okay. A mechanical dictionary is a reasonable approximation. If something goes through your post, looks up each word in an English-French dictionary (for example) and outputs the resulting text, it won’t be correct, but it will be vaguely comprehensible. If you build a dictionary of bigrams or trigrams (sequences of 2-3 words) this gets a bit better because now collocations are more likely to be translated correctly. It won’t be as good as a professional translator, but it will more or less look like the target language. Add more statistical modelling and you will get better up to a point. But there’s a cliff where you can’t improve without actually understanding the content. No amount of statistical modelling will let you accurately translate the things that are statistical outliers and the extrinsic knowledge necessary means that you can’t infer a correct translation from the text alone without understanding its context.

Tests have a similar property. Good tests convey the intention, but the intention is not part of the code and so can’t be inferred from it. Good tests cover the things that the test author knows are corner cases, but these can’t be inferred from the code either (a few can, if the language has explicit error-handling constructs) because they’re a property of the input data.

In both cases, LLMs try to compensate for the lack of understanding by having a lot of examples of similar things in their input. If the thing you’re translating is similar to a load of other things, you may not need to understand it to translate it correctly because the first dozen (or hundred, thousand, or whatever scale you need) people to translate something like that did the hard work and you can reuse it. If the thing you’re testing is similar to a load of other things that already exist, someone else may have done the hard work of identifying the common failure modes and expressing intent.

But commonly LLM-generated tests end up testing that the code does what the code does. And that’s not useful. If you want that, just use fuzzing in a harness that tests trace equivalence between two versions of the program (for the same sequence of inputs, do they generate the same output?). That is useful for no-functionality-change-intended patches (typically things that improve performance or simplify unnecessary complexity), but most changes to the codebase are there because you want the behaviour to change. Good tests will fail if you changed something that was part of an API contract but will not fail if you added new behaviour, but tests based on the code will change.

This isn’t limited to LLMs. Some of the LLVM tests are just ‘run this command, the output should look like this’. People typically reject these in review now because long and painful experience showed us that it was hard to refactor when a change broke a test and the change author couldn’t tell if the difference in output came from something we actually cared about or just something that happened to be part of the old version’s output. But humans can, at least, tell the difference in the tests because they understand what it is that they intend with the change that introduces the test.

@harrysintonen I'd have a different recommendation for the vendor: Stop trying to pretend disappearing messages are a thing.

Signal has backups. Revocation from old backups is a very hard problem that they don't even try to store.

With the old backup model, each day got a completely new snapshot of all messages and media. If any participant in a chat has backups turned on and doesn't clean out their old backups, disappearing messages are recoverable at an arbitrary point in the future.

The newer backup is similar, each day generates a new snapshot of all messages, it's just that they reference media that are backed up separately.

And that's assuming everyone is using the official client. But any user using a different client may simply choose not to delete them.

I have one chat where I set deleting messages to try to encourage people to write discussions up elsewhere, I wouldn't use it as a security or privacy feature and I think it's quite misleading that Signal pretends that it is either.

Every time #Signal says that they will pull out of a country if some law is passed, they remind everyone that they are a single point of failure. How many times have you ever heard that email will pull out of a country if some surveillance law is passed?

EDIT: Yes, of course this has the usual flurry of people saying 'Don't like Signal? Use {thing that is worse than Signal and no one who cares about security takes seriously}'. Please just stop. I am using Signal because I have evaluated alternatives and Signal is the least bad. That doesn't mean I'm going to stop criticising it and it doesn't mean I'm going to use something worse.

@bagder

AI powered code analyzers are significantly better at finding security flaws and mistakes in source code than any traditional code analyzers did in the past

I’m not sure this follows from what you’ve said in the rest of the post. Static analysers and fuzzers also made it very easy for people to find vulnerabilities and typically found a lot when they were deployed for the first time. And both were a lot cheaper to run than something like Mythos.

They aren’t finding as many vulnerabilities now because projects that are critical for security are integrating them into their CI flows.

And this is what always happens with some new technique: valgrind, Coverity, sanitisers, fuzzers, and so on: they’re released, they find a load of bugs that existing techniques failed to find, people fix them, they get integrated into regular CI runs, and the kinds of bugs that those tools find never make it into the tree.

Syskaller, for example, has found a lot more bugs in the Linux kernel than any Anthropic tools. And that’s just one fuzzing tool.

Why does our industry keep looking at things, claiming it's doing them, and doing the exact opposite of what the original idea was? A few examples:

Alan Kay (who coined the term) defined the key idea of object orientation as late bounding, so we ended up with a load of things that use rigid nominal type systems to tightly couple components, marketed as 'object oriented'.

The Agile Manifesto's core idea was 'people over process'. I've lost count of the number of times I've seen places claim they're using 'the agile methodology' because they have sprints, standups, and other processes taken from Agile.

The Zero Trust paper said, at its core, 'assume endpoints are compromised, design your systems so that an endpoint compromise doesn't automatically give control over everything', yet almost everything I've seen branding itself as Zero Trust has been of the form 'run some over-privileged thing on the endpoints to increase their attack surface, then if that thing reports that the endpoint isn't compromised allow it to do a load of things it shouldn't be allowed to do'.

The recent post criticising Free Software advocates for advocating user-modifiable software and then being annoyed at LLMs annoys me and the reason is best illustrated by this analogy:

Public-transport advocates spend years advocating for a connected public-transport infrastructure, where it’s easy to take a small combination of busses, metros, trams, and trains to get from anywhere to anywhere. The network would be efficient and operated as a non-profit-making public good, making individual movement cheap (or, ideally, free). They work with municipalities to build out some of this infrastructure, persuade national governments to invest in the longer routes, and so on.

Someone comes along with a massive subsidy for a handful of private taxi companies to hire a bunch of drivers and give free (paid for by investors) ride to everyone. The drivers are immigrants who don’t speak the language very well, which is great for the taxi companies because they are easy to exploit (they are, in fact, underpaid and put in dangerous situations routinely). The owners of the taxis are pocketing a load of investor money for every ride though.

When you get in one of these taxis, there’s a 90% chance they’ll take you where you want, a 9% chance they’ll take you somewhere nearby, and a 1% chance they’ll just drop you off in a dangerous part of town. A bunch of people are mugged and a few more murdered as a result of this, but the companies aren’t liable. The investors behind this tell everyone ‘don’t bother learning to drive, there’s no point, our taxis will take you anywhere, for much less money!’. At the same time, ridership on existing public transport drops off, leading to calls to cut its funding and there are mass redundancies for bus drivers and so on. The taxis are all diesel and heavily polluting, leading to worse air quality everywhere they go. To make sure that they can pick people up easily, the ones not actively giving rides are constantly circulating, placing huge strain on road infrastructure and further increasing pollution.

And then someone says to those public-transport advocates: ‘this is what you wanted, why are you unhappy just because it’s not delivered in the way you imagined?’

@alice

I now live somewhere civilised, where someone saw my partner and I riding a tandem and felt the need to roll down the window and shout ‘you two are so fucking cute!’ as he drove past, but before I moved here:

I have long wavy hair and often wear long black coats, so look quite feminine from behind. The number of people who would roll down their window to catcall as they went past, then see the front of me and realise that they had just catcalled a boy was… a lot. And this was especially fun when they had other people in the car, because you can bet that was a cat full of homophobes.

@feld @phnt @giacomo

The FSF was not a large enough organisation to deter infringers in the case I was directly involved with and they do have a track record of going after infringers. And, unfortunately, copyright assignment meant that they were the only people who could go after infringers.

What might work is a CLA, or even license clause, that allows multiple parties to go after infringers (have standing to sue as designated agents of the copyright holder) and requires that they split the proceeds with the copyright holders on a fixed ratio, with legal fees coming out of their part. That would incentivise other groups to chase down any instance of infringement in hope of a payout.

I’d be very nervous of putting something like this in a license as complex as any of the FSF’s though, because it would also incentivise chasing individuals who accidentally infringed and would immediately settle because they wouldn’t be able to afford to go to court.

One of the problems with using copyright law to protect end users (the FSF’s strategy) is that only the copyright holder has standing to sue, but they are not the ones harmed by violation. If I release a GPL’d program, and you buy something containing a derived work that doesn’t respect the license and (for example) contains some easily fixed bug that you cannot fix due to this violation, you are harmed but I am the one who has standing to sue.

But rather than do any of this, what I want is to build systems where end users can easily modify and extend them and create an ecosystem where closed proprietary products can’t compete because users expect and actively exercise the rights to modify and redistribute software. Requiring users to understand a complex license before they exercise the rights that are the thing that differentiates Free Software from non-Free software provides a barrier to their exercising these rights, which composes with any technical barriers.