SLi with two Nvidia GTX 660 Ti

[xor1337]

So with vulkan and SLi, you get a blank grey screen.
With SLi disabled I get 99% usage out of 1 card, and 0% usage from the other.
I thought the beauty of Vulkan/DX12 was that you could get usuage from more than one card without SLi.

Does your Vulkan renderer even look for secondary GPU resources?

I think that should be a huge priority, since its so easy to throw in extra cards, maybe even last generation's card when you upgrade.

I'm sure you are just trying to get things to run with a single card, but just think of the possiblities of using Vulkan to get at a second, third, or fourth GPU. No longer bounded by AFR with SLi, we could get some serious (ha pun intended as always) rendering done with multiple cards.

[AlenL]

Well, you are right that we are primarily focused on single cards for now. Thing is that even that is still not stable and performant enough. Now, SLI, unfortunately, it's not as easy as it seems. There are serious problems as for why multi-card rendering will not be as fast as you'd like. The reason is similar to why an 8-core CPU is not 8x faster than a 1-core CPU. Even with AFR, there are problems with algorithms that do iterative calculations based on previous frame. Same-frame rendering is even harder due to intra-frame dependencies between pixels (e.g. rippling/refraction effects) and auxiliary buffers and effects e.g. shadow buffer, godrays, SSAO, that sample pixels across the frame make it extremely complicated and robs it of most of performance you'd expect.
It will happen eventually, but it will take a lot of time, and it won't be as fast as you'd like. Sorry. :(

[xor1337]

SFR. Create two cameras each with a perspective frustum of exactly 1/2 the normal width. Have each render indipendantly on each of the two GPUs and then stitch the frame together in the end? It isn't double the speed I'm sure, but will be a bump in FPS and should allow for inter frame calculations etc
I used to do open GL 2.1 back in college, which there has been a whole world of change since then but you are the GPU guy. I know you can do it. I have faith.
I mean that low level card access is available with Vulkan right? I don't care if you had to basiclly double the engine/main ram useage. x64 is the future/present right? and games rarely use more than 2GB these days which I think is a total waste. Why do I have 18GB of ram if you arn't willing to break out of the chains of consoles and make some good stuff.

I guess this is kinda a side point. For some reason PC games aren't shy about requireing a 700 or 900 level Nvidia GPU these days. They also arn't shy about eating up 30-50 GB of harddrive space for a game install. No developer will seem to take the plung and start requiring more ram. I mean lets just make 4GB or even 6GB a system requirement, and lets optimize the games engines to take advantage of that extra breathing room. I'm sure once a AAA developer requires 8GB for some new game, there will be like 2 days of shock, and then everybody will follow suit. I mean ram is cheapest part of a system these days. Sorry for the rant, just my 2 cents.

[Migz - DH]

Цитата допису xor1337:

No developer will seem to take the plung and start requiring more ram. I mean lets just make 4GB or even 6GB a system requirement, and lets optimize the games engines to take advantage of that extra breathing room. I'm sure once a AAA developer requires 8GB for some new game, there will be like 2 days of shock, and then everybody will follow suit. I mean ram is cheapest part of a system these days. Sorry for the rant, just my 2 cents.

There are actually a fair number of games which require 4GB of System RAM.
Example:
http://store.steampowered.com/app/289130/
Memory: 4 GB RAM

--------------------------------------------------------------------

There are even at least two I can think of right now which require 8GB.
Examples:
http://store.steampowered.com/app/386070/
Memory: 8 GB RAM

Star Citizen also requires at least 8GB.
https://robertsspaceindustries.com/

[xor1337]

Thats awesome, I've played Planetary Annihilation, I can see why. The battles are just huge. I've never watched my memory usage during play, but I still doubt that it uses more than 3GB of ram, reguardless of the quoted system requrements. The thing about these few games, are that they are never coming to consoles. Also they are farily niche games, not likely to get much press reaction no matter what they do. Star Citizen is likely to be massive and amazing, if it doesn't end up vaporware. It will use PC potential to the max and could never work on the constraints of a console.
Seriously though, the PS4 VR pack? Oculus recommends a 980 for VR, and those boneheads at Sony think that they can pull off VR with the PS4? sounds like a low framerate nausea inducing machine. I have the DK2 and my SLi 660 Ti's and it can barely handle VR rendering. That sony thing is likely going to kill the VR industry, since most people's VR experiance would be on a crappy console, and they will get sick and hate it. Much like how terrible Sony's 3d TVs were. Oh well, this is enough off topic rambling, thanks for the post Migz

[Migz - DH]

Цитата допису xor1337:

Thats awesome, I've played Planetary Annihilation, I can see why. The battles are just huge. I've never watched my memory usage during play, but I still doubt that it uses more than 3GB of ram, reguardless of the quoted system requrements.

You should, PA can take all of the System RAM you can throw at it and still ask for more. For example, 8GB isn't enough System RAM to play in a planetary system with just two large planets. You have to play on smaller planets, or the virtual memory gets used.

[AlenL]

Цитата допису xor1337:

SFR. Create two cameras each with a perspective frustum of exactly 1/2 the normal width. Have each render indipendantly on each of the two GPUs and then stitch the frame together in the end? It isn't double the speed I'm sure, but will be a bump in FPS and should allow for inter frame calculations etc

It doesn't work for intra-frame effects. E.g. when you see jammers activate in Talos, the "jamming field" displaces the graphics behind it. Which means that you have to read some other pixel to the side, to be able to calculate what's in this pixel. If those two pixels are on different GPUs it either doesn't work, or imposes a huge performance penalty (if you sync frame content between the GPUs), killing all the potential gains. Some effects like bloom and godrays are even worse (i.e. the offset is bigger than with the "ripple" used in jammers).
TL;DR: SLI is nice in theory, but will never work nearly as good as people would want it to. :(
[/quote]

[xor1337]

Awesome, I love that this thread is bringing in some great discussion. AlenL, I'm sure you know about this stuff and I'm glad to hear you thoughts. So godrays, bloom, jamming field effects cause SLi problems then right? I've read that its the "deferred renderer" type stuff that breaks SLi. Anyway, under DX11 this game is great with SLi, I get really good framerates. Sans SLi I get around 50-80 FPS with all the graphics settings I like. Its totally playable but there are some prettty significant studders, and this game is already kind of motion sickness inducing and that makes it way worse. Anyway I play with a 120Hz monitor (which I recommend) and I prefer sustained 75-90 FPS with Vsync on for playing, and if I can get it, the full 120Hz with vsync on. With SLi I get closer to 95-120 FPS with this game, and the studders are almost entirely gone. So under DX11 and SLi this game plays awesome. There are some SLi compatability bits in the driver making all this happen, so there is probably some sharing of Framebuffer between the cards to make this possible.
I'm computer architechture we have techniques we use for making CPU's "superscaler" meaning using the instruction pipeline correctly to get closer to 1 Instruction finished per cycle, instead of each instruction going though the pipeline independantly. These techniques can and should be applied to a graphics pipline, and I think could be setup to take the previously rendered frame from one GPU and allow the second GPU to save those renering effects till the end of the pipeline and then grab that frame right at the end, do the effect, and then finish the frame, and then swap buffers. Vulkan should give you the ability to setup your own pipelines and try some of these techniques. I mean branch predicion I'm sure seemed crazy back in the 80's, but its old news now. Lets think outside the box on multi gpu rendering, Vulkan might allow us the freedom to make it better.

[AlenL]

Are you sure it's not using AFR when it is working so well for you? We have support for AFR in general - where supported by the drivers (it still requires the app to deal with it, even under DX11). It's not ideal, but it's definitely better than SFR. SFR is just a bad idea. Older games, like Serious Sam classic might be able to show speedup with SFR, but for modern effects, it's too difficult.

[Migz - DH]

GTX 570s in SLI here using the 364.47 driver. NVIDIA Control Panel says "NVIDIA recommended (custom)" (which I don't remember seeing the custom part for a game before). Whichever AFR version the driver is using, it works pretty well.

[xor1337]

Yes, AlenL, the DX11 SLi I was talking about in post #7 is using the Nvidia default SLi setting of AFR. The only time I was talking about SFR, was a suggestion of how Multi GPU could work with Vulkan, setting up two cameras each with half the normal view frustum, therefore cutting down the workload per GPU per frame, thereby causing FPS to increase. Still though I love this discussion, any other Vulkan developers out there have any ideas? I'm tempted to have a look at the Vulkan api and see what kind of multi gpu access I could get at using my limited knowledge.

[AlenL]

Yeah, that thing with two cameras doesn't work well for refraction and for post-processing effects. Hence AFR is preferred.
Also note that microstutter is not an SLI-specific problem. It can appear on single cards as well. The cause of it is inability of the game to tell actual frame times or to schedule frames for specific times. This problem is in all APIs and there's still no solution offered. I asked for that to be included in Vulkan, but it wasn't even properly considered. On Linux, we have support for a specific Nvidia extension which, when running on some Nvidia cards, allow us to schedule frames and that gets rid of microstutter. Pitty it's not universally available.

[xor1337]

I see, yeah I hadn't thought about refraction. Yeah, there are some problems with a multi gpu setup, but I'm sure we can figure a way around them. Right around 1999-2000 when multi core cpus started to come on the scene, all game engines were single threaded, and for a few years people said game engine calculations couldn't be multithreaded, everything is so linear in games, but now everything is multi threaded. I think there must be a way to do SFR or AFR better in a game engine. CPU speeds have reletively stalled out for like 8 years, but the number of cpu cores has been increasing. I feel like GPUs could likely do the same thing, and then multi gpu cards and systems might be the best option for going forward.

[AlenL]

Same thing as with CPU multithreading - it's far from ideal. You should look into the difference between latency and bandwidth. On CPU side, you either do data-parallel which scales great for some things like physics, collision and rendering, but you get a lot of linear code in the AI side, or you do task-parallel, but then you are just increasing bandwidth, but keeping the same (or even a bit increased) latency.
On the GPU side, SFR and AFR are equivalent - SFR would reduce latency (which is ultimately what you are looking for), but it's complex and often imbalanced, and doesn't work for some effects (keeps the linear part, which is even worse here, since cross-bus copying is super-expensive on GPU, unlike on CPU). AFR works better, as it increases bandwidth much more easily but it also increases latency which is bad.
It's a rock and the hard place situation. :P

[xor1337]

yeah, AlenL you are right about latency and bandwidth. Bandwidth is great for cuda or compute and less latency is really what we want games. SFR is of course best for latency because you both cards are working on a single frame. AFR can help push through the old fps pipeline, but mostly increace bandwidth and only slightly help studdering. SLi mirrors the ram between the cards so there shouldn't be too much cross bus writing, and a frame buffer is almost nothing in terms of size. I think we can make cool effects that work well with SFR, or even AFR, we just need to focus on that. I mean, I for one would say screw all effects that can't play nice with multi gpu. That isn't really moving forward though. I think saying "we can't" because this effect needs "so and so" isn't helping. There must be a way. Graphics rendering has been using tricks to make stuff look good without actually calculating everything since its inception. Everything in graphics is some kind of trick unless you are talking about ray tracing or the new global illumination tricks. Lets think voxels, lets think shared rendering piplines, lets work outside the box. Lets work on some tricks that focus on SFR or AFR. I think I need to go back to grad school and work on this problem. Or at least someone should. Game programers have gotten soft these days. We have high level API's that do everything for us, and if you stick to using a prebuilt engine, its ridiculus what little code it takes to make a game these days. I used to write open GL engines back in school and even that was pretty high level. Now with vulkan we all get access that only the driver devs at AMD or NVIDIA had. We CAN do this. We have the Powa... :)