Turning a Proxmox VM into a Headless GPU-Accelerated Linux Workstation

How I built a remotely accessible Linux desktop with GPU passthrough, Sunshine, Moonlight and WireGuard - and worked through software rendering, headless NVIDIA displays and broken NVENC support along the way.

By Aleksander Kowalczuk 5 October 2026

In this article

Why I built this

I wanted a Linux workstation that I could access from my laptop both at home and away from home, without dual booting and without dedicating another physical machine to it.

I already had a Proxmox server running 24/7 and a GTX 1050 available in it, so the obvious idea was to turn one of its VMs into a proper GPU-accelerated remote desktop.

The target setup was simple:

Laptop
  ↓
Moonlight
  ↓
Sunshine
  ↓
Linux Mint VM
  ↓
GTX 1050

At home, I wanted it to behave like a local workstation over LAN.

Away from home, I wanted to use exactly the same setup through WireGuard without exposing Sunshine directly to the Internet.

Getting there turned out to be slightly more interesting than expected.

The hardware and VM

Proxmox host

Proxmox VE
Intel Xeon host
32 GB RAM
NVIDIA GeForce GTX 1050

The GTX 1050 was previously being used by one of my Jellyfin containers, so moving it to the workstation VM meant giving that VM exclusive ownership of the GPU.

The GTX 1050 is also the only physical GPU in the host, so once passed through, the Proxmox server itself effectively becomes headless. (not that this matters)

Guest

Linux Mint 22.3 Cinnamon
8 vCPU
8 GB RAM
128 GB virtual disk
Q35
OVMF / UEFI
VirtIO networking
NVIDIA GTX 1050 PCIe passthrough

GPU passthrough

The card exposes two PCI functions:

04:00.0 NVIDIA Corporation GP107 [GeForce GTX 1050]
04:00.1 NVIDIA Corporation GP107GL High Definition Audio Controller

Both functions were in the same isolated IOMMU group, which made passthrough straightforward.

The host was configured with IOMMU enabled:

intel_iommu=on iommu=pt

After binding the GPU to VFIO, the host showed:

Kernel driver in use: vfio-pci

Inside the Mint VM, the NVIDIA driver detected the card correctly:

nvidia-smi

Output:

NVIDIA GeForce GTX 1050
Driver Version: 580.178.04

At that point, PCI passthrough itself was working.

But that did not mean the desktop was actually using the GPU.

The first trap: having a GPU does not mean using a GPU

The VM still had the virtual display adapter created by Proxmox.

Xorg exposed it as:

Virtual-1

The desktop worked.

The NVIDIA card was visible.

Sunshine could start.

At first glance, everything looked fine.

Then I checked the actual OpenGL renderer:

DISPLAY=:0 glxinfo -B

The result was:

OpenGL vendor string: Mesa
OpenGL renderer string: llvmpipe (LLVM 20.1.2, 256 bits)

So the Cinnamon desktop was being rendered entirely in software.

The GTX 1050 existed inside the VM, but Xorg was not actually using it to render the desktop.

This is an important distinction with GPU passthrough:

Seeing a GPU in lspci or nvidia-smi does not mean your desktop or applications are actually using it.

Choosing Sunshine and Moonlight

I initially looked at more traditional remote desktop solutions, but I wanted something free, low-latency and suitable for a GPU-accelerated desktop.

Sunshine and Moonlight were a much better fit.

The architecture became:

Linux Mint VM
      │
      ├── Sunshine
      │
      └── GTX 1050
             │
             ▼
          Moonlight
             │
             ▼
           Laptop

Sunshine runs on the VM and captures the desktop.

Moonlight acts as the client.

Because Sunshine can use hardware video encoders, the GTX 1050 could theoretically handle both desktop rendering and H.264 encoding.

That was the goal.

Sunshine worked... technically

Sunshine installed correctly and exposed its web interface at:

https://192.168.1.197:47990

Moonlight found the host and pairing worked.

But starting a stream resulted in:

No video received from host

Sunshine was logging errors such as:

Failed to import DMA-BUF
Could not convert image

The problem was that Sunshine was effectively trying to capture the framebuffer from the virtual Proxmox graphics adapter while performing encoding work on the NVIDIA GPU.

That created an awkward cross-GPU pipeline:

Virtual Proxmox GPU
        ↓
      capture
        ↓
GTX 1050 encoder

The DMA-BUF import between those devices was failing.

Switching Sunshine from KMS to X11

I switched Sunshine to X11 capture:

capture = x11

That immediately improved things.

The desktop became streamable.

But now Sunshine had another problem.

It attempted to initialize NVENC and failed:

Trying encoder [nvenc]
...
Encoder [nvenc] failed
...
Trying encoder [software]
...
Found H.264 encoder: libx264 [software]

Another error appeared during the NVIDIA path:

RGBA_to_NV12 failed: cudaErrorNoKernelImageForDevice

So the stream worked, but only because Sunshine fell back to software H.264 encoding.

At that point the pipeline looked like:

Cinnamon
   ↓
software rendering
   ↓
Sunshine
   ↓
libx264 on CPU
   ↓
Moonlight

Which defeated most of the reason for passing a GPU through in the first place.

The Pascal / NVENC compatibility issue

The GTX 1050 is based on NVIDIA Pascal.

The current Sunshine build I was using had problems with the NVENC path on this GPU and driver combination.

I tested an older Sunshine release:

Sunshine 2025.924.154138

Suddenly the encoder detection succeeded:

Found H.264 encoder: h264_nvenc [nvenc]
Found HEVC encoder: hevc_nvenc [nvenc]

That was the first real breakthrough.

To make sure Sunshine was not simply reporting NVENC while still doing work on the CPU, I checked actual encoder utilization:

nvidia-smi dmon -s u -d 1

During streaming, the encoder column showed activity around:

enc 13%
enc 15%

So NVENC was genuinely being used.

At this point the GTX 1050 was finally doing hardware video encoding.

Running an older Sunshine build is not ideal because newer releases can contain security fixes.

For that reason, I do not expose Sunshine directly to the public Internet.

Remote access goes through WireGuard instead.

Fixing the actual desktop renderer

NVENC was working, but Cinnamon was still being rendered through LLVMpipe.

The next step was to make NVIDIA the actual Xorg device.

I generated an Xorg configuration using the proprietary NVIDIA driver.

The relevant part looked like this:

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "NVIDIA GeForce GTX 1050"
    BusID          "PCI:1:0:0"
EndSection

The screen was initially configured as headless:

Section "Screen"
    Identifier     "Screen0"
    Device         "Device0"
    Monitor        "Monitor0"
    DefaultDepth    24

    Option         "AllowEmptyInitialConfiguration" "True"
    Option         "UseDisplayDevice" "None"

    SubSection     "Display"
        Virtual     1920 1200
        Depth       24
    EndSubSection
EndSection

After rebooting:

DISPLAY=:0 glxinfo -B

finally returned:

OpenGL vendor string: NVIDIA Corporation
OpenGL renderer string: NVIDIA GeForce GTX 1050/PCIe/SSE2

Before:

Mesa
llvmpipe

After:

NVIDIA Corporation
NVIDIA GeForce GTX 1050/PCIe/SSE2

The Cinnamon desktop was now actually GPU accelerated.

nvidia-smi also showed Xorg using GPU memory:

/usr/lib/xorg/Xorg    45 MiB

The next trap: a headless NVIDIA screen is not necessarily capturable

There was one more issue.

With:

Option "UseDisplayDevice" "None"

Xorg had a framebuffer size:

1920x1200

but there was no actual RandR output.

Running:

DISPLAY=:0 xrandr --current

showed the screen dimensions, but no connected monitor.

Sunshine then failed to initialize capture:

Unable to initialize capture method
Video failed to find working encoder

The encoder was not actually the problem.

The problem was that Sunshine's X11 capture path needed an actual display output.

Creating a fake NVIDIA display

The solution was to make the NVIDIA driver believe a display was connected.

I replaced:

Option "UseDisplayDevice" "None"

with:

Option "ConnectedMonitor" "DFP-0"
Option "UseDisplayDevice" "DFP-0"

I also added:

Option "ModeValidation" "DFP-0: NoEdidModes, NoMaxPClkCheck, NoHorizSyncCheck, NoVertRefreshCheck"

and:

Option "MetaModes" "DFP-0: 1920x1200_60 +0+0"

After rebooting:

DISPLAY=:0 xrandr --current

showed:

DVI-D-0 connected primary

even though nothing was physically connected to the GPU.

The NVIDIA driver was now exposing a real RandR output.

I could then set:

DISPLAY=:0 xrandr --output DVI-D-0 --mode 1920x1200 --rate 59.95

The important part is that no physical dummy HDMI adapter was required.

The display exists entirely in software.

The final graphics pipeline

At this point the full graphics pipeline finally looked correct:

┌────────────────────────────┐
│ Linux Mint / Cinnamon      │
│ 1920×1200 headless desktop │
└─────────────┬──────────────┘
              │
              ▼
┌────────────────────────────┐
│ NVIDIA Xorg                │
│ GTX 1050 rendering         │
└─────────────┬──────────────┘
              │
              │ X11 capture
              ▼
┌────────────────────────────┐
│ Sunshine                   │
│ H.264 NVENC                │
└─────────────┬──────────────┘
              │
              ▼
┌────────────────────────────┐
│ Moonlight                  │
│ LAN / WireGuard            │
└────────────────────────────┘

The GTX 1050 now handles both:

rendering the Linux desktop
encoding the Moonlight stream

Making Sunshine persistent

The working Sunshine version was an AppImage, so I created a systemd user service for it.

[Unit]
Description=Sunshine Pascal NVENC
After=graphical-session.target network-online.target
Wants=network-online.target

[Service]
Type=simple
Environment=DISPLAY=:0
Environment=XAUTHORITY=%h/.Xauthority
ExecStartPre=/bin/sleep 5
ExecStart=%h/.local/bin/sunshine-pascal.AppImage
Restart=on-failure
RestartSec=3

[Install]
WantedBy=default.target

Then:

systemctl --user daemon-reload
systemctl --user enable --now sunshine-pascal.service

The short startup delay gives LightDM and Xorg enough time to create display :0 before Sunshine starts.

The exact service configuration may vary depending on the Linux distribution and display manager.

Remote access through WireGuard

I did not want to expose Sunshine ports directly to the Internet.

Instead, remote access looks like this:

Laptop
  │
  │ WireGuard tunnel
  ▼
Home network
  │
  ▼
192.168.1.197
  │
  ▼
Sunshine

Moonlight simply connects to the VM's private LAN address:

192.168.1.197

When I am at home, that traffic goes directly over LAN.

When I am away, WireGuard routes the same private subnet to my laptop.

No Sunshine port forwarding is required.

I tested it by putting the laptop on my phone's hotspot and connecting through WireGuard.

The stream worked exactly the same way.

The final bug was... an outdated Moonlight client

After solving GPU passthrough, Xorg, headless displays, DMA-BUF and NVENC, I still had one strange problem.

Moonlight worked perfectly from my desktop.

On my Linux laptop, it showed:

Unable to connect to the specified PC

The network itself was completely fine.

ping -c 4 192.168.1.197

worked.

The Sunshine control ports were reachable:

nc -vz 192.168.1.197 47989
nc -vz 192.168.1.197 47990

Both succeeded.

Sunshine worked from another machine.

The solution was significantly less interesting than everything before it:

I updated Moonlight on the laptop.

After the update, it connected immediately.

So after debugging PCI passthrough, CUDA, Xorg, RandR and NVENC, the final connectivity issue was simply an outdated client.

What I learned

PCI passthrough is only the beginning

Getting a GPU into a VM is not the same as actually using it.

nvidia-smi confirmed that the VM could see the GTX 1050, but only glxinfo exposed the fact that Cinnamon was still running through LLVMpipe.

Always verify the renderer

This command ended up being one of the most useful checks during the entire setup:

DISPLAY=:0 glxinfo -B

If it says:

llvmpipe

your GPU passthrough may be technically working while your desktop remains completely software-rendered.

Headless GPUs still need a display abstraction

A GPU can render without a physical monitor.

That does not mean every screen-capture application can operate without an X11/RandR output.

Creating a fake NVIDIA display solved that problem without requiring a physical dummy plug.

Hardware encoding should be measured

I did not want to trust configuration files alone.

nvidia-smi dmon -s u

made it easy to confirm that the NVENC engine was genuinely active during streaming.

Debug the stack layer by layer

The final setup spans multiple independent layers:

networking
↓
Moonlight connection
↓
Sunshine
↓
screen capture
↓
Xorg
↓
OpenGL renderer
↓
video encoder
↓
GPU driver
↓
PCI passthrough

A failure at one layer can easily look like a problem somewhere else.

For example, an error that initially looked like a Sunshine networking problem was actually caused by the graphics capture pipeline.

Conclusion

The final result is effectively a small private cloud workstation.

At home, Moonlight connects directly over LAN.

Away from home, WireGuard gives me access to the exact same machine without exposing Sunshine to the public Internet.

The GTX 1050 handles both desktop rendering and H.264 encoding, while Proxmox keeps the entire workstation virtualized alongside the rest of my homelab.

It took considerably more troubleshooting than simply installing Sunshine, but that is also what made the project useful.

I now have a much clearer understanding of where PCI passthrough ends, where the Linux graphics stack begins, and what remote desktop software actually needs from a headless GPU.

And yes — after all of that, the final problem really was an outdated Moonlight client.