Off-loading LLM work to my gaming PC

I build private RAG systems, which means I frequently run LLM inferences on my local machine. When docked, my M2 MacBook fits the bill just fine. However, sometimes I work on the go with no power supply, and running LLMs locally drains my battery too fast. At the same time, I have a gaming PC at home constantly plugged in. What if I could offload the LLM labor to that?

My gaming PC

Create a VPN for myself

First of all, how do I access my PC at home over internet safely? I need a virtual private network (VPN) for that. Not the kind of VPN that unlocks geo-restricted Netflix content offered by a for-profit company; that kind of service usually isolates clients (computer/phone/tablet/…) from each other, so you can’t connect to your device from another, even if both are logged in to the same account. Instead, I need something that allows me to discover my other computers as if we are on the same local area network (LAN).

Luckily, I’ve already figured that part out: When setting up my Raspberry Pi, I’ve created a VPN with ZeroTier One. I only had to add my Windows PC into it.

On my gaming PC, under PowerShell, I installed ZeroTier One via Windows Package Manager (winget):

winget install "ZeroTier One"

In the GUI, I set it up and give it the IP address 10.0.0.5, so SSH'ing into it would be:

ssh -i ~/.ssh/lan_rsa 'Ming-yeung Lee@10.0.0.5'

It threw me off when I realized that Windows 11 duly took my full name as the account name. This is the name I entered when the setup interface asked me “Who’s going to use this device”. Somehow, I had assumed that Windows would escape the whitespace and the hyphen under the hood, just like Ubuntu does. Now that they are still there, I will have to keep quoting the username@address part in my SSH commands. On a second thought, this makes sense to me -- An average Windows user has virtually zero incentive to care about command-line ergonomics because they don't need to, but that's not true for an average Ubuntu user, at least not yet.

Enable SSH on Windows

Here’s another caveat that caught me off-guard: Don’t sign in to Windows with a Microsoft account. OpenSSH Server doesn’t work with them yet (source). I also have recently removed password from my Microsoft account (how), so this is even more infeasible.

If you had, convert it to a local account. When choosing a user name, use something simple (without spaces and case-insensitive; see above), so that it’s more compatible and easier to type in when remote-logging in to this system. If you don’t like the user name appearing so cryptic, you change its “display name” from the Control Panel (source).

Enable SSH server by following this tutorial. Remember to make the OpenSSH Server service automatically start upon boot. In PowerShell, this is (source):

# Set the sshd service to be started automatically
Get-Service -Name sshd | Set-Service -StartupType Automatic
# Now start the sshd service
Start-Service sshd

Let’s set up the SSH key.

In PowerShell, make sshd service automatically start up, and then start it (source):

# By default the ssh-agent service is disabled. Configure it to start automatically.
# Make sure you're running as an Administrator.
Get-Service ssh-agent | Set-Service -StartupType Automatic

# Start the service
Start-Service ssh-agent

On your MacBook, you can now copy the SSH key file to it. However, the usual ssh-copy-id can't handle a Windows destination. Instead, manually transfer the content of ~/.ssh/lan_rsa.pub (on my MacBook) to the file C:\ProgramData\ssh\administrators_authorized_keys (on the Windows PC).

The Microsoft-flavored OpenSSH Server somehow defaults to the legacy Command Prompt (cmd.exe) when you log in, while PowerShell has become the default since Windows 10 build 14971. I would like some consistency: To use PowerShell as the default shell over SSH:

New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -PropertyType String -Force

Don’t use WSL as your login shell

If you have installed Windows Subsystem of Linux (WSL), you may be tempted to use it as the default shell over SSH, like this:

New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value "C:\Program Files\WSL\wsl.exe" -PropertyType String -Force

This is a bad idea, because you can’t escape from WSL to PowerShell, which you’ll find yourself needing it quite often.

Instead, let’s live with one extra step of going from PowerShell to WSL:

& 'C:\Program Files\WSL\wsl.exe'

If you have already installed tmux in WSL, you can even do:

& 'C:\Program Files\WSL\wsl.exe' tmux new -As win

Save this as a wsl.bat file, so that it's easier to activate in the future.

Install a LLM server

I’ve been using LM Studio, but it doesn’t have a CLI interface, and it’s a pain to switch between different models (ref). With Ollama gaining popularity, I see that many LLM application frameworks now have first-party support for interfacing with Ollama today, so it’s a good time to make the switch.

I prefer to install things with CLI commands, so that — in case I re-install the operating system — I can re-install everything in batch. To install Ollama via winget (source):

winget install --id=Ollama.Ollama -e

Next up, I made Ollama listen on the IP address over my VPN (how).

Finally, on my MacBook, for each project I wrote that expects a local LLM inference server, I rewired it to use Ollama instead. For example, here’s how to do it with LlamaIndex:

-Settings.llm = OpenAILike(
-    api_base="http://localhost:1234/v1",
-    model="zephyr beta 7B q5_k_m gguf", # This could be arbitrary
+Settings.llm = Ollama(
+    base_url="http://10.147.20.237:11434",
+    model="zephyr:7b-beta",

Now everything should work as expected!

Wake-on-LAN (WOL)

With great power comes great [electricity bill]. — Uncle Ben

Consuming more power sitting idle than a MacBook does running full load, my gaming PC is configured to go to sleep with 10 mins of inactivity. How do I wake it up when I need it while I’m not home?

I installed a small tool to my Raspberry Pi, which is always on and connected to my personal VPN:

environment.systemPackages = with pkgs; [
+    wakeonlan

Then, I noted down the MAC address of my gaming PC. From now on, whenever I need to wake it up remotely, I only have to run this command from my MacBook:

ssh -i ~/.ssh/lan_rsa 'user@10.0.0.3' 'wakeonlan MA:CA:DD:RE:SS:OF:GA:MI:NG:PC'

where 10.0.0.3 is the IP address of my Raspberry Pi over the VPN. As always, I save this as an alias in my macOS shell, so that I can wake the gaming PC up just by typing wakepcup.

Summary

In this post, I wrote down how I set up my gaming PC as a LLM inference server. The less-known bumpers I encountered include:

• OpenSSH Server on Windows doesn’t work with Microsoft accounts.
• The usual ssh-copy-id script can't handle a Windows destination.
• Once you specified WSL as your login shell over SSH, you can’t escape back to PowerShell.

Shared also in this article are a few tricks to make lives easier. They include keeping display name and account username separate, as well as remotely waking up your power-hungry PC.

I hope this post helps someone in the future to avoid some time traps.