Like if I type “I have two appl…” for example, often it will suggest “apple” singular instead of plural. Just a small example, but it is really bad at predicting which variant of a word should come after the previous
Like if I type “I have two appl…” for example, often it will suggest “apple” singular instead of plural. Just a small example, but it is really bad at predicting which variant of a word should come after the previous
Phones don’t use LLM for predictive text. The algos are a lot less complex on phones.
I guess, the real question is: Could we be using (simplistic) LLMs on a phone for predictive text?
There’s some LLMs that can be run offline and which maybe wouldn’t use enormous amounts of battery. But I don’t know how good the quality of those is…
You can run an LLM on a phone (tried it myself once, with llama.cpp), but even on the simplest model I could find it was doing maybe one word every few seconds while using up 100% of the CPU. The quality is terrible, and your battery wouldn’t last an hour.
Does the AI processing have to be performed locally or constantly active?
The kind of local/offline LLMs that would work on your phone would not be very good quality. There’s been amazing progress in quantization of LLMs to get them working on weaker GPUs with lower VRAM and CPUs, so maybe it’ll occur, but I’m not an expert.
I also don’t foresee them linking it up to a cloud-based LLM as that’d be a shit load of queries and extremely expensive.
That was my next question, thanks!
Didn’t think of battery use, makes sense
Openhermes 2.5 Mistral 7b competes with LLMs that require 10x the resources. You could try it out on your phone.
I think apple has pitched this for a future iPhone, yes.
They’ll probably have to offload that to a server farm in real time. That’s not gonna be easy.
I guess… why not… but the db is probably huge, like in the hundreds of GB (maybe even TB… who knows), can’t run that offline.
A pre trained model isn’t going to learn how you type the more you use it. Though with Microsoft owning SwiftKey, I imagine they will try it soon
iOS 17 uses a small gpt-2 based model for predictive text.
Hm, that’s interesting 👍.
The algorithms are the same. The models are different, being trained on a smaller data set.
No, the algorithms are not the same. Phones don’t use transformer models for text prediction, they use Markov chain-based approaches. Also, retraining of transformer models for individualized completion would be too expensive, whereas it’s basically free with Markov approaches. Where do you get these ideas?
Perhaps, I’m not a dev, especially not an iOS or an Android one.