Working LLaMA and Gemma LLMs with C++ and Python
These days, “cellular AI” is a fast-growing pattern. Smartphones change into extra highly effective, and huge fashions change into extra environment friendly. Some prospects could need to wait till new options are added by telephone producers, however can we use the newest AI fashions on our personal? Certainly, we are able to, and the outcomes are enjoyable. On this article, I’ll present how one can run LLaMA and Gemma giant language fashions on an Android telephone, and we’ll see the way it works. As common in all my assessments, all fashions will run regionally, and no cloud APIs or funds are wanted.
Let’s get into it!
Termux
The primary part of our take a look at is Termux, a full-fledged Linux terminal made as an Android utility. It’s free, and it doesn’t require root entry; all Linux parts are operating solely in a Termux folder. Termux may be downloaded from Google Play, however on the time of penning this textual content, that model was fairly previous, and the “pkg replace” command in Termux didn’t work anymore. A more recent model is offered as an APK on the F-Droid web site; it really works nicely, and I had no issues with it.
When Termux is put in on the telephone, we are able to run it and see an ordinary Linux command-line interface:
In principle, we are able to enter all instructions straight on the telephone, however typing on the tiny keyboard is inconvenient. A a lot better approach is to put in SSH; this may be carried out through the use of “pkg set up”:
pkg replace
pkg improve
pkg set up openssh
After that, we are able to begin the SSH daemon in Termux by operating the sshd command. We additionally have to get the consumer identify and set the SSH password:
sshd
whoami
#> u0_a461
passwd
#> Enter new password
...
Now, we are able to hook up with a telephone with any SSH consumer:
ssh -p 8022 u0_a461@192.168.100.101
Right here, 8022 is a default Termux SSH port, “u0_a461” is a consumer identify that we get from a “whoami” command, and “192.168.100.101” is the IP…
