KVM Virtualization and SteamOS - A guide.

[ArmchairCommando]

This is a guide for installing and configuring KVM in SteamOS. If you have the right hardware, you can use KVM to create a Windows virtual machine and pass your video card through to it. This allows you to run all of your favorite non-linux compatible games on the same machine as SteamOS. Under this configuration, in-home streaming might work really well as well (haven't been able to test). I borrowed heavily from the following guide for ArchLinux:
https://bbs.archlinux.org/viewtopic.php?id=162768

What you will need:
1. Intel or AMD CPU with full support for virtualization extentions. I have an i7-2600
2. A motherboard that supports these features and has the option to enable them in the bios. These are kind of hard to find because a lot of vendors do not mention it. I have a Gigabyte Z77-D3H.
3. Some Linux knowledge, or failing that, high level google-fu.
4. An AMD video card. I have an HD7970. Supposedly you can use Xen and pass through an nVidia card, but that will be left as an exercise for the reader.
5. Maybe a little luck.

Note: If you continue past this point and something bad happens to your computer, its not my fault.

SteamOS Prep Work: Below are my notes for the prep work I did in SteamOS before I started. It is mainly convenience stuff for myself, but the important piece is installing the Debian repository. Bolded stuff is Linux commands, italics are file contents. ===================================================================
passwd to change the desktop password.

Set up the debian repositories without risking the SteamOS ones.
http://steamcommunity.com/groups/steamuniverse/discussions/1/648814396114274132/
sudo nano /etc/apt/sources.list
deb http://http.debian.net/debian wheezy main contrib non-free
deb-src http://http.debian.net/debian wheezy main contrib non-free
sudo nano /etc/apt/preferences
Package: *
Pin: release l=SteamOS
Pin-Priority: 900
Package: *
Pin: release l=Debian
Pin-Priority:-10
sudo apt-get update
Installed some convenience stuff:
installed gedit
installed chromium-browser
installed libreoffice

Install KVM:
sudo apt-get install -t wheezy qemu-kvm libvirt-bin virtinst bridge-utils: installs all of the tools and plugins KVM needs.
virt-manager:
had to install: sudo apt-get install python-glade2,sudo apt-get install libgconf2-4 before virt-manager could figure itself out.
test that you can launch it. should launch from terminal using sudo virt-manager

Note: sound had to be reconfigured. had to switch from HDMI sound on video card to the USB headphones.

Opt into the SteamOS beta after all install functions are completed.
==================================================================
KVM Configuration: Below are my notes from preparing my machine for KVM and some of the configuration work for KVM itself.
==================================================================
Get into your motherboard BIOS and enable anything and everything relating to virtualization.
egrep -c '(vmx|svm)' /proc/cpuinfo
You should get a number as the output. 0 means your CPU does not support virtualization, 1 or more as the output means you have virtualization support. My hardware gives 8 as the output.

This guide was the source of most of my information.
https://bbs.archlinux.org/viewtopic.php?id=162768

LSPCI: run this command to list out all of the PCI devices in your machine. Look for your video card and it's HDMI audio component. This is a part of my output.
01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Tahiti XT [Radeon HD 7970]
01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Tahiti XT HDMI Audio [Radeon HD 7970 Series]

lspci -n: this dumps the PCI bus address or ID or something. record the ones that correspond to the information from the output of LSPCI.
01:00.0 0300: 1002:6798
01:00.1 0403: 1002:aaa0

sudo su: to get permanent root, this just saves some time.
echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf
echo "blacklist fglrx" >> /etc/modprobe.d/blacklist.conf
prevents the HD7970 drivers from being loaded at startup. Should make it so SteamOS can use the integrated graphics instead.
***had an issue here, this did not work as expected. The FGLRX driver ignored this and kept grabbing the video card. Only way i could prevent this was to set my integrated graphics to be initialized first in the BIOS.
nano /etc/default/grub: opens the grub configuration file for editing.
Append "intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1" to /etc/defaut/grub (GRUB_CMDLINE_LINUX="" to GRUB_CMDLINE_LINUX="intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1").
update-grub: updates grub.

I had to enable the SteamOS beta in order to make the integrated graphics work. This is a point of no return. Appearently once you are in the beta the best way to get out is to reinstall you machine....

Note: I had to reboot it once before enabling the integrated graphics.
After you successfully reboot with the IGFX displaying, run sudo modprobe -r fglrx. It should not give any output. If it does, something is wrong.
Note: My processor comes with an Intel HD Graphics 2000. It sucks. The SteamOS big picture mode runs pretty badly with it.

echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf
Does something...dont remember what, but its probably important.

create a file using "nano /usr/bin/vfio-bind"
paste this into it and save:

#!/bin/bash

modprobe vfio-pci

for dev in "$@"; do
vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
device=$(cat /sys/bus/pci/devices/$dev/device)
if [ -e /sys/bus/pci/devices/$dev/driver ]; then
echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
fi
echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
done
Run this command:
chmod 755 /usr/bin/vfio-bind
I think this makes the file we just created executable.

Then run the script, using your lspci -n results. It appears that you need to add "0000:" to the front. This will allow your graphics card and HDMI sound to be passed through to a VM, or something.

vfio-bind 0000:01:00.0 0000:01:00.1

If it doesnt work and you had an intel CPU, run:
modprobe -r kvm_intel
modprobe kvm_intel emulate_invalid_guest_state=0
Not sure what it does, but it helped in my case.
==================================================================
VM Configuration: and finally, some basic instructions for using virt-manager to configure a VM with VGA passthrough.
==================================================================11. Sudo virt-manager
2. Click Edit
3. Select Connection Details
4. Select Storage Tab
1.Hit the blue plus to create a new storage pool
2. give it a name, and choose the type. i chose "disk:physical disk device" because I had an SSD set aside for this.
3.Launch disk utility to find the path to the device you want and set the source path to that.
4.Click finish.
5. New storage pool will be created. Create a new volume in it back on the Storage tab and allocate all of it. Make sure autostart is checked.
6. Go to the virtual netowkrs and interfaces tabs and make sure the default items are set to start at boot.
7. Go back to the main VMM screen.
8. Right click on your localhost entry and select new.
1. give the VM a name and choose local install media. Click next
2. Select a Windows install disk. Set the OS type and version. note: Windows 7 works for Windows 8
3.Give the VM RAM and CPUs. Dont give it everything, and if you have hyperthreading allocate at most half of your total number of threads.
4. Click select managed or other existing source and use the volume we just created. You can also create a new disk image on the hard drive. click next.
5. Click customize before install and click finish.
6. Bunch of stuff in here. not going into all of the details. If performance isnt very good chances are you will be able to change something here that will help. To get to it, you can right click on your VM, click open, and then lick the little lightbulb next to the computer screen. Note: use virtio whenever possible. It offers really good performance with little virtualization overhead.
7. In the VM hardware config screen, click add hardware.
8. Click on PCI Host Device.
9. Scroll until you find your video card. Select it and click finish. If you get an error, there is something wrong. A reboot *might* fix it. Repeat to add the HDMI sound piece as well. Apply your changes.
10. Boot up the VM by hitting the play button.
11. Install windows normally. You should be able to download anything you need because of the virtual network interface. Download the latest AMD drivers for your card. You should be able to install them normally, just like as if it was a real machine.
12. Reboot, download Steam, and a benchmark program to install if you wish.
==================================================================

I hope this guide is useful to someone. Sorry about the formatting. I plan on testing streaming between the VM and SteamOS soon. I just got into the streaming beta and haven't had time to try it. Please do not expect this to be an overly stable solution. I have had to force shutdown my machine several times while running this. The performance is great though. Using the Valley benchmark from Unigine under Windows 7, I scored 1180 with my setup as a physical machine. Running Windows 8.1 as a virtual machine under SteamOS, I scored 1219. This was using the same resolution and graphical settings.

By the way, if there are any Linux pros that want to point out anything stupid/wrong/dangerous in my guide, please do so. I will update it as i have time. I am a Windows pro, so the only way I was able to do with was with much Googling

[King Dude]

Nice work. I'll have to check this out for myself.

[leeps]

Contains more instances of "not sure what it does", "or something", "I think" than I'd like to read in a tutorial.

[Vladimir Harkonnen]

As i understand it, you can't use your fast videocard for steamos anymore.

[Shark]

Originally posted by Vladimir Harkonnen:

As i understand it, you can't use your fast videocard for steamos anymore.

Depends, you can only use your primary graphics card for SteamOS if you do this for afaik.

[TooMeeK]

Thank You. Not tested yet, but been using KVM for years. Probably this will be useful since I'm running 16GB of RAM.

[ArmchairCommando]

Originally posted by Vladimir Harkonnen:

As i understand it, you can't use your fast videocard for steamos anymore.

That is absolutely correct, you must have two video cards in order to make this work. You cannot pass integrated graphics to a virtual machine. It is possible to make with work with and nVidia card and an AMD card, and maybe with two AMD cards, but it would require more work in the terminal.

Originally posted by leeps:

Contains more instances of "not sure what it does", "or something", "I think" than I'd like to read in a tutorial.

A fair criticism.


Notes from today's testing - IMPORTANT NOTICE
Rebooting the virtual machine may cause your entire system to lock up and you will have to manually reboot. If this happens during, say, a reboot caused by Windows Update, your VM might be totally screwed. System Restore might be able to save it though. This link - http://blog.ktz.me/?p=219 - suggests that ejecting the video card, and probably any other PCI passthrough devices before a reboot, may stop this from happening. Did not get to test streaming.

[ArmchairCommando]

UPDATE:
The reboot fix I mentioned in my previous post does indeed work. Also, I was able to test streaming. It was a little slow; probably a combination of high resolution and a junk graphics card in SteamOS doing the decoding. It was running at about 10fps, forgot to check the proper stats with F6.

[MC Mjolnir]

Originally posted by Armchair_Commando:

Originally posted by Vladimir Harkonnen:

As i understand it, you can't use your fast videocard for steamos anymore.

That is absolutely correct, you must have two video cards in order to make this work. You cannot pass integrated graphics to a virtual machine. It is possible to make with work with and nVidia card and an AMD card, and maybe with two AMD cards, but it would require more work in the terminal.

Can you not just run the host machine on the integrated graphics? I plan to run 2 VMs on a headless linux system, and passthrough the ethernet ports to stream to other devices, so I won't even need a gui on the host OS.

[ArmchairCommando]

You should be able to run the host headless, but integrated graphics would still be assigned to it. To the best of my knowldge, the integrated graphics on Intel processors cannot be used in pass through. I dont know about AMD APUs...those might work. You would just connect a monitor to the video card you pass to the VM. If your motherboard has multiple USB controllers, you can pass those through to the VM as well.

[MC Mjolnir]

Did you have any trouble getting the HDMI audio to work?

[wh1sper_123]

just curious, why would you stream if you made passtrough GPU VM? put hdmi cable in and use tv remote to switch between the 2 systems. with local on pc streaming... you're better of with wine

streaming takes double toll on your setup as you need to copy stream whole time between the server and client, where same physical machine gets hammered twice. and you already split your machine in half

also, with kvm it depends also which driver you used for net card in VM. if cpu/board doesn't support VirtIO beside VTx it might just be other drivers perform better than default.

[MC Mjolnir]

Wine is good, but it still doesn't have 100% coverage, it's always playing catch-up and it usually has to be configured for each indivual game. Once set up, a Windows VM with a passed-through card should play all games without issue. Switching the input on the monitor only changes the display, you will still have to either reconnect your input devices, buy a kvm switch or have a seperate set of peripherals. This method means you're just an alt+tab away from your other programs.

Performance wise, with the latest version of streaming they've added HW encoding for Nvidia cards, and will soon add it for AMD cards too, and most processors have a dedicated h.264 decoder. You can also overprvision CPUs, my experience with openstack has demonstrated that you can overprovision an Intel CPU 3:1 without issue, so a single windows guest shouldn't be too much of a stretch.

[wh1sper_123]

Originally posted by MC Mjolnir ART:

Wine is good, but it still doesn't have 100% coverage, it's always playing catch-up and it usually has to be configured for each indivual game. Once set up, a Windows VM with a passed-through card should play all games without issue. Switching the input on the monitor only changes the display, you will still have to either reconnect your input devices, buy a kvm switch or have a seperate set of peripherals. This method means you're just an alt+tab away from your other programs.

Performance wise, with the latest version of streaming they've added HW encoding for Nvidia cards, and will soon add it for AMD cards too, and most processors have a dedicated h.264 decoder. You can also overprvision CPUs, my experience with openstack has demonstrated that you can overprovision an Intel CPU 3:1 without issue, so a single windows guest shouldn't be too much of a stretch.

2 things,
passtrough VM is not what i questioned. implementation seems its done in a more or less most efficient way one can do it. i questioned streaming, because no matter how much there is hw enc/dec, you still need to send data from virtual machine and receive it on hypervisor. and that amount of data is not really small task, especially when you consider you already split machine in 2. well, depends on resolution, 1080p would kill it on amount

just try copying large file over fast network and watch utilization. now imagine that in this setup you're doing both and more since one machine is suffering some virtualization penalty

also, i forgot one other thing. bottleneck in games now is not gpu, it is cpu and since it was cut by virtualization, halving, streaming... games will never run as well as it would in wine. even when you run on 2 machines lag is noticeable.

main question was formed on occams razor principle. if you already have complete machine, isn't it easier and better if you simply connect it directly?

[ArmchairCommando]

I cant really answer the HDMI audio question, as I do not have anything that uses it. However, I would assume that it works. The HDMI audio device is detected by Windows just fine.

So far, my Windows VM has been able to run every game I've thrown at it without issue and at pretty much native speed. Streaming has been a mixed bag. Streaming at 1280x720 usually gives gives a steady 60fps. 1920x1080 generally gives around 30fps, and 2560x1440 between 10-20fps. I am not sure if the low speeds are a consequence of CPU overworking or bandwidth contraints (or something else entirely). I have been trying to minimize the amount of compression done on the stream, as the virtual network connection should be more than capable of dealing with the bandwidth needed for uncompressed video. The fact that graphically simple games like Risk of Rain run perfectly even at 2560x1440, while games like Skyrim chug, leads me to think it might the CPU stream processing bottlenecking the stream. I added a cheap Nvidia card to my rig, so hopefully I will get to test HW encoding on both sides to see how it works.

Generally speaking, I don't use the streaming. I switch inputs on my monitor and switch my keyboard over to the VM. Its pretty easy to do if you pass through a USB controller and buy a hub.

Edit: I don't think the virtualization penalty is as much as you think. The VirtIO drivers have performance that is pretty close to native at this point. I think you are correct about the CPU being the bottle neck here, but I think HW decode would help a bit with this (with VM graphics performance suffering some in place of the CPU). As for Occam's Razor, what fun is that? :)

[wh1sper_123]

Originally posted by Armchair_Commando:

I cant really answer the HDMI audio question, as I do not have anything that uses it. However, I would assume that it works. The HDMI audio device is detected by Windows just fine.

So far, my Windows VM has been able to run every game I've thrown at it without issue and at pretty much native speed. Streaming has been a mixed bag. Streaming at 1280x720 usually gives gives a steady 60fps. 1920x1080 generally gives around 30fps, and 2560x1440 between 10-20fps. I am not sure if the low speeds are a consequence of CPU overworking or bandwidth contraints (or something else entirely). I have been trying to minimize the amount of compression done on the stream, as the virtual network connection should be more than capable of dealing with the bandwidth needed for uncompressed video. The fact that graphically simple games like Risk of Rain run perfectly even at 2560x1440, while games like Skyrim chug, leads me to think it might the CPU stream processing bottlenecking the stream. I added a cheap Nvidia card to my rig, so hopefully I will get to test HW encoding on both sides to see how it works.

Generally speaking, I don't use the streaming. I switch inputs on my monitor and switch my keyboard over to the VM. Its pretty easy to do if you pass through a USB controller and buy a hub.

Edit: I don't think the virtualization penalty is as much as you think. The VirtIO drivers have performance that is pretty close to native at this point. I think you are correct about the CPU being the bottle neck here, but I think HW decode would help a bit with this (with VM graphics performance suffering some in place of the CPU). As for Occam's Razor, what fun is that? :)

hmmm, the occams razor argument response... hats off, sir. you couldn't make a better one ;) god only knows how many times i did something that didn't really make sense just for fun of it. in the end i could easily buy windows whenever i wanted, but i play those in wine... nah, i just can't stand windows too much, wine is still better option for me

my job is more or less coding and maintaining servers with shitload of VMs), so... i'm used to take birtualization from optimal side. never tried it for fun

what might help you on windows side (at least i didn't notice you mentioning installing virtio drivers on client. having virtio on hw still requires you installing them)
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Virtualization/chap-Virtualization-KVM_Para_virtualized_Drivers.html