Egregoros

@SecureOwl This is the second post I’ve seen today about clotheslines, so I have a question that people in this thread might be able to answer:

How do you keep the spiders off?

If we hang up clothes in the garden (and we have tried with both a simple retractable line and a rotating thingy on a pole), they get covered in cobwebs in about ten minutes. Spiders love them, clothes hanging on a line are a perfect place for a web that catches insects flying across the garden, but the anchor points need to be strong so they put a load of web on the clothes. And then they all need washing again when they come in.

RE: https://infosec.exchange/@catsalad/116944318046104194

It’s fascinating seeing notifications for boosts here, where something you wrote a year or two ago will suddenly be found by someone, who boosts it, and then a little flurry of other people do.

It really makes you realise how much recency bias there is in most other systems, including most web forum software. Sometimes that’s useful (I don’t want obsolete solutions to be highlighted in preference to the ones that actually work), but it also makes writing there seem ephemeral. Here, a conversation can start up after two years of silence when someone comes up with something new to contribute. And that’s incredible.

@hfalcke

LLMs will stay and people will use it.

There's a big assumption here, that comes from the previous one:

It is not that AI does not work, as often said here, it does work for many applications.

That's not the question. The first-order question is: do they deliver more value than they cost. And the answers to that are skewed by the fact that the current services are heavily subsidised and customers are paying 10-50% of the minimum viable price. So, if people actually have to pay the real price, how many of those use cases still exist?

The follow-on to that is, given that increasing the price by a factor of 2-10x will significantly reduce the demand, what will the resulting price be? Inference costs won't change much, but the price to users is the inference costs plus the training costs amortised across users (plus profit). Fewer users means that the training costs are amortised across fewer users and so the per-user costs go up. So how many use cases are still economically feasible if the prices go up by a factor of 20x?

But that's just the first order. The second order is: how affordable is it when you factor in the cost of externalities. Current LLMs are trained by doing something that, in any other context, would be a clear-cut case of large-scale copyright infringement. If LLM vendors had to actually pay for licenses for the things that they are ingesting in training, the cost of training would go up by at least a factor of 10x.

So there's a huge assumption in 'LLMs will stay'. The companies that train them do so on the assumption that they'll eventually be able to recoup that (huge) investment. If you don't train new ones, they become stale quickly. To be useful, they need to include recent events. For software, an LLM that doesn't know about new APIs and language features and does know about deprecated / removed ones is useless. For other uses, the shifts may be slower or faster. But training isn't a one-shot cost as a result, you're constantly needing to do retraining. And that's what needs the massive datacentres full of GPUs.

So, if you want to claim LLMs are here to stay, you need to answer a simple question: Who is going to pay for the training? If no one is, then how will LLMs remain useful?

The entire industry is spending enormous amounts of money for tiny returns. How do you expect this to shift to actually making money? Note that the numbers in this page look much better than they are, because all of that $516 B in revenue for NVIDIA is spending from other companies (which must be passed on to customers for anyone else to become profitable).

The Free Software movement never escaped from its origins: the early ‘80s MIT AI Lab. Two things were true in this environment:

• Back when most computers had tens of KiBs of RAM and 1 MiB was a huge amount, programs were simple. Very few programs were so complex that one person could not completely understand them.
• The AI Lab was full of some of the most talented programmers in the world.

This meant that the only obstacles for these people being able to fix bugs and add features to any program were access to the source code and the legal rights to modify it. Once you have those, any program was understandable by that group and they could modify it however they wished.

For the next 40 years, the FSF focused on these two things. The world around them changed. These two prerequisites were never enough for most people (what do 90% of computer users do if you give them even a modest 10,000 line C codebase and tell them they can change it however they like?) and now they aren’t enough even for competent programmers.

When Linus says ‘fork it’ to folks who don’t want LLM-extruded code in their kernel, he knows full well that it is almost impossible to fork a 40 MLoC C (and Rust now) codebase that averages more than one CVE per day and have something useful.

The Free Software movement is struggling now because it obsessed over licenses, which was never a path that would succeed, and ignored the hard problems:

• How do you design environments that enable end users to modify their software?
• How do you engineer software so that it is cheap and easy for a random user to maintain a fork that meets their specific needs?
• How do you foster communities where people want to share improvements, so forks don’t proliferate even when it’s easy?
• How do you create an environment where everyone sees the benefits of user-modifiable code to such a degree that trying to sell anything that doesn’t come with these rights is commercially impossible?

Instead of tackling any of these problems, they created more complex and restrictive GPL variants. And well-paid lawyers found loopholes in them that allowed corporations to keep doing what they wanted (and even pick licenses like AGPLv3 to control ecosystems, because they give the copyright owners so many more rights than everyone else that it’s hard for anyone else to compete). They said ‘don’t worry about the complexity of the licenses, you only need to understand the legal details if you’re creating and distributing derived works’ while completely forgetting that making it possible for anyone to create and distribute modified versions of the programs was the entire point of the Free Software movement.

@zwol

I also like this but voted no. Cambridge had a research architecture like this and I really liked it when I first saw it: architecturally, you have an unbounded (bounded by memory) set of hyperthreads. Microarchitecturally, you have a finite number and all runnable threads are pulled in when they can be. You can build a system with no register rename or speculation if you do this, which means it can get enormous (GPU-like) aggregate throughput.

The problems come when you want to provide scheduling policies. Think about how you’d implement a priority-inheriting mutex on such a system and you realise quite how complex a hardware-software interface you need. I’m willing to believe it’s possible but it’s a multi-year well-funded prototype to come up with something that actually addresses the hard problems.

When I was a small child, Anchor Butter ran an ad campaign with a jingle that went:

Anchor Butter, from green green grass.

And, as a very witty 8-year-old, I completed this with ‘… tastes like it comes from out of your arse!’ And we sang this in the playground.

Today, several decades later, whenever I see the Anchor Butter logo, that plays in a loop in my head and there is no possible way that they can remove that association and there is no way I would ever buy their product because I do not want that association on my toast.

The moral of this story is: if you are marketing a consumer product, before you launch a campaign, ask a focus group of children of all ages to make fun of your slogan. You may cause serious damage to your brand otherwise.

One of the things that was drummed into my head repeatedly working with the late Ross Anderson was that most security problems are usability problems. It doesn't matter how secure your cryptosysetm is if people send unencrypted messages because they can't understand how to do key exchange. It doesn't matter how secure and fine-grained your sandboxing is if it has a user-prompted privilege elevation mechanism and users are trained to just hit 'approve' every time a dialog pops up.

This is why #CHERIoT focused so heavily on building a usable programmer model from the start. Compartments communicate with other compartments by calling functions. Compartments share objects by passing pointers to them. Compartments have a mechanism to expose type-safe opaque types and that's surfaced directly in the source languages. There is no look-aside policy that you have to read to know whether a compartment exercises some right: it's all explicit in the source code.

WebAssembly makes it easy to do the same kind of isolation, but not the sharing. A C pointer in WebAssembly is a 32-bit offset into a memory. If you want an object that's shared between two WebAssembly sandboxes, that's a totally different kind of pointer. Early CHERI work tried this and we gave up because the cognitive load and porting effort of having two kinds of pointer was too painful.

#cheri

@jonny @elebertus

I suspect testing has the same properties as translation. It’s moderately easy to build machine-translation systems that are kind-of okay. A mechanical dictionary is a reasonable approximation. If something goes through your post, looks up each word in an English-French dictionary (for example) and outputs the resulting text, it won’t be correct, but it will be vaguely comprehensible. If you build a dictionary of bigrams or trigrams (sequences of 2-3 words) this gets a bit better because now collocations are more likely to be translated correctly. It won’t be as good as a professional translator, but it will more or less look like the target language. Add more statistical modelling and you will get better up to a point. But there’s a cliff where you can’t improve without actually understanding the content. No amount of statistical modelling will let you accurately translate the things that are statistical outliers and the extrinsic knowledge necessary means that you can’t infer a correct translation from the text alone without understanding its context.

Tests have a similar property. Good tests convey the intention, but the intention is not part of the code and so can’t be inferred from it. Good tests cover the things that the test author knows are corner cases, but these can’t be inferred from the code either (a few can, if the language has explicit error-handling constructs) because they’re a property of the input data.

In both cases, LLMs try to compensate for the lack of understanding by having a lot of examples of similar things in their input. If the thing you’re translating is similar to a load of other things, you may not need to understand it to translate it correctly because the first dozen (or hundred, thousand, or whatever scale you need) people to translate something like that did the hard work and you can reuse it. If the thing you’re testing is similar to a load of other things that already exist, someone else may have done the hard work of identifying the common failure modes and expressing intent.

But commonly LLM-generated tests end up testing that the code does what the code does. And that’s not useful. If you want that, just use fuzzing in a harness that tests trace equivalence between two versions of the program (for the same sequence of inputs, do they generate the same output?). That is useful for no-functionality-change-intended patches (typically things that improve performance or simplify unnecessary complexity), but most changes to the codebase are there because you want the behaviour to change. Good tests will fail if you changed something that was part of an API contract but will not fail if you added new behaviour, but tests based on the code will change.

This isn’t limited to LLMs. Some of the LLVM tests are just ‘run this command, the output should look like this’. People typically reject these in review now because long and painful experience showed us that it was hard to refactor when a change broke a test and the change author couldn’t tell if the difference in output came from something we actually cared about or just something that happened to be part of the old version’s output. But humans can, at least, tell the difference in the tests because they understand what it is that they intend with the change that introduces the test.