Investigating Game Lag

[jdlech]

Granted, I always play huge galaxies that end up with lots of empires. And I also play with up to 40 mods engaged - usually around 35 to 38 mods.
But late game lag gets to be terrible, and my system is no slacker.
I use a i7-6700 @ 4.09MHz (slightly overclocked) on a ROG Maximus VIII. The problem, as I see it, is that one processor out of 8 is maxed out running the game while the other 7 are hardly utilized at all.
Processor #4 is always at 100% while the others are anywhere from 45% to as little as 4% utilized. When I pause the game, Processor #4 utilization drops to about 65% on average, variation between 58% to 71%.

I have 32MB of DDR-4 RAM on that board, and it's not even 50% used even with other programs running. So it's not the memory that is causing this. Nor is it the memory modules since I have recently tested them extensively.
My boot drive is a 4 drive RAID using my motherboards hardware RAID controller (not managed by windows), using 4 SATA III SSDs, each with a read bandwdth of 560MB/s and a write bandwidth of 420. With a RAID read speed of 1920MB/s, and less than 30% utilization, I can't see that it's my drive causing any problems.
My graphics card is the vaunted Asus STRIX GTX 980Ti with 6GB RAM, and it's using the official drivers from Asus. The drivers are exceptionally stable and there is absolutely no problems with any other game, program, or setting. Card GPU utilization is 45% at most (usually around 22%) and memory utilization is 38% at most. So it's not the GPU, the card, its memory, or the graphics driver.

I tried to delve deeper by using MS Windbg - Microsofts debugging tool (and such an unfortunate name for any program)- to see how many threads Stellaris is using. No dice. Stellaris does not accommodate MS debugging. Not that I expect them to, but it is a bit disappointing.

So, we've pretty much eliminated everything except the program itself. I wonder what Paradox plans to do about it?

[GeneralVeers]

Originally posted by jdlech:

I wonder what Paradox plans to do about it?

Nothing. Welcome to modern gaming.

Most game developers are going to give players the option to trade performance for size, and play a huge galaxy at a lower tick rate. The upper limit on the game settings is often going to be something no computer except a six-thousand-dollar Alienware rig can handle. Insert reference to "But can it run Crysis?" here.

Your computer may be good, but you bumped into its upper limit, and you're going to have to turn the galaxy size down a notch.

[EleventhStar]

Originally posted by GeneralVeers:

Your computer may be good, but you bumped into its upper limit, and you're going to have to turn the galaxy size down a notch.

More like you have a Ferrari, and while there is no speed limit the road you are on has holes in it and a ton of bends.

PDX sometimes fixes a few holes, but they keep adding more bends and the machinery to make bends also creates holes.

[GeneralVeers]

Originally posted by EleventhStar:

Originally posted by GeneralVeers:

Your computer may be good, but you bumped into its upper limit, and you're going to have to turn the galaxy size down a notch.

More like you have a Ferrari, and while there is no speed limit the road you are on has holes on it and a ton of bends.

More like, you snagged yourself a shiny new Lexus RCF and decided to enter the Indy 500.

Yeah, the RC is mean, but an Indy car will force-feed it a whole lotta dust. The only way an RCF gets on an Indy track is if it's the pace car.

[jdlech]

Yes, I love my digital ferrari. In fact, I play Cities Skylines with a mod that increases the upper limit for traffic and spawned sims. I play it at maximum settings for everything. In other words, I regularly play at levels that would make most systems beg for mercy. And you know what? An 800K city maxed out with 64K vehicles has barely any noticeable lag.
But Stellaris kludges along like I'm on a 25MHz-386sx. It's terrible, pathetic, the worst I've ever seen since my 486.

But the real point of all that "boasting" is to prove that it's not hardware or drivers. Paradox... I'm addressing you directly. IT'S.... YOUR.... GAME.
Paradox needs to learn how to load balance processors.

[EleventhStar]

Originally posted by GeneralVeers:

More like, you snagged yourself a shiny new Lexus RCF and decided to enter the Indy 500.

Yeah, the RC is mean, but an Indy car will force-feed it a whole lotta dust. The only way an RCF gets on an Indy track is if it's the pace car.

that implies there are good quality pcs that can max out their hardware on this game. which i've yet to see an example off.

[TwoTonTuna]

https://steamcommunity.com/games/281990/announcements/detail/1635299113660403804

In addition, to give us the best chance of improving some of the pain points you’ve shared with us, we have assigned some of our team members to focus solely on trying to improve performance and AI. It is very important to us that 2.6 does not compound any of the current issues with the game, and that we can take the time we need to address some of the issues remaining from 2.2. It’s important to remember, however, that working on these kinds of issues is not a sprint, but a marathon – it's something that is constantly being worked on over longer periods of time.

If you want to read more about performance, and how we work to maintain it over time, we shared some more information on this topic in Dev Diary #149.

While we have been unable to give concrete information or specifics related to these issues, we can say that it is very important to us. With that said, it's important for us that you know that your feedback is not being ignored, even if we have no news to share.

Tips for improving gameplay performance:
1. Remove your mods and install the Stellaris Immortal mod from the workshop. It's a work-in-progress that overhauls the game, simplifying mechanics so that the game runs smoother.
https://steamcommunity.com/sharedfiles/filedetails/?id=1891758612&searchtext=immortal

2. If you want to play vanilla, reduce the number of habitable planets (0.5) and play on a smaller galaxy with fewer empires. Set your game so that you play against gestalt consciousness empires (Machine, Hive Mind Empires.) I ran a test before and found out that the late-game/end-game runs so much smoother on a game populated solely by gestalt consciousnesses.

[Vyndicu]

To OP.

I recommend several steps to help mitigate late-game lag.

First, on your own world's make sure that all empty jobs are few in number or close those jobs. Make sure that if you are going to create more jobs that you have enough idle population to take created jobs.

Unfortunately you can't stop AI from creating unnecessary lag from empty jobs short of going full genocide on the galaxy.

Play as a gestalt yourself or limit trade collection to the core sector. It is well known that trade pathfinding is calculated every single day.

Play on a smaller Galaxy or tune world density to a lower number.

That all you really can do without modifications and there are a few focused on addressing the late-game lag.

P.S. I feel your pain as late-game can end up running slow as 8 seconds on day-to-day tick. Despite having a decent PC as well (AMD 1700x(stock speed), 32 GB, Vega 64).

[jdlech]

I could drop an I7-7700K under the hood,. Which is the maximum my MB can take. But without overclocking, that's only 20% better. I could maybe double that overclocking it. I would have to replace the whole MB to get better. Sure, I could spend thousands on a new MB/CPU/memory upgrade. But seriously. What I have is already well above average and handles everything else I throw at it more than adequately. This is the only game that gives my system problems. And it's the worst I've seen since I was a poor college kid in the 90s.

[GeneralVeers]

Originally posted by jdlech:

Yes, I love my digital ferrari. In fact, I play Cities Skylines with a mod that increases the upper limit for traffic and spawned sims. I play it at maximum settings for everything. In other words, I regularly play at levels that would make most systems beg for mercy. And you know what? An 800K city maxed out with 64K vehicles has barely any noticeable lag.
But Stellaris kludges along like I'm on a 25MHz-386sx. It's terrible, pathetic, the worst I've ever seen since my 486.

This is meaningless to me. It tells me nothing about how much processing power is actually needed to run an 800k city without lagging. You're asking me to compare apples to hammers.

What I do know is, it's incredibly easy to add a menu item saying "40,000 stars" in the game setup menu, and it's almost as easy to program the galaxy generator to produce a galaxy with 40,000 stars. The fact that no computer on this planet can run that without catching fire is irrelevant. If they had programmed this game to be able to make a 40,000 star galaxy, a bunch of players would be dumb enough to try, and a lot of them would complain when it didn't work. That's what you're doing.

Originally posted by jdlech:

But the real point of all that "boasting" is to prove that it's not hardware or drivers.

Yes it is. Your computer isn't broken; it's just not up to an arbitrarily large task. There is a value X (where X is the number of stars in a galaxy, that your computer simply cannot run.

Originally posted by jdlech:

Paradox needs to learn how to load balance processors.

They won't. Multithreading and load balancing are a pain in the ass, and I've got a whole bunch of games that don't do it.

[jdlech]

Some great tips so far. Thanks to everyone responding. If you have any more tips, I'm sure there's a lot of people who would like to know them.
Again, thanks for sharing. You might salvage this game yet. (though I'm really addicted to huge galaxies, know what I mean?).

[GeneralVeers]

Originally posted by jdlech:

(though I'm really addicted to huge galaxies, know what I mean?).

Admit it. You want to try the 40,000-star galaxy.........

[Vyndicu]

Oh I almost forget. Avoid xeno-compatibily perk like the black plague.

Higher number of sub-species there are in the galaxy the worse lag can be. Not sure why.

[Sabaithal]

Its well known that CPU is the primary issue with this game.

Specifically, multi-core CPUs. You see, unlike every sensible company that develops modern PC games, PDX decided to use an engine that is 7+ years old. Said engine is very poorly optimized for multi-threading. That is why you experience lag.

Doesn't matter how good your PCs specs are. Unless you have a CPU with very high clock speed you will see lag unfortunately.

Again, this is a well known issue that has caused a lot of people to stop playing. PDX should have done something about this first and foremost BEFORE prioritizing the release of another DLC, but they didn't. Interpret that how you will.

[TwoTonTuna]

Originally posted by Sabaithal:

Again, this is a well known issue that has caused a lot of people to stop playing. PDX should have done something about this first and foremost BEFORE prioritizing the release of another DLC, but they didn't. Interpret that how you will.

That's may be why Federations is coming out Q1 2020 instead of Q4 2019. They normally release a new DLC this time of the year, but they pushed back the next release to polish the game up. Hopefully they can optimize the game better to make the late and end-game feasible once more.

[Cookiesnm1lk]

okay so protip, the lion's share of the lag in stellaris is caused by the ai constantly micromanaging its pops for maximum efficiency. changing their positions EVERY. SINGLE. DAY. and when you have this check for thousands of pops every second.... well the game doesn't like that. HOWEVER there's a band aid for this. https://steamcommunity.com/sharedfiles/filedetails/?id=1882139456 with THIS mod here you can tell it to only switch the pops around once every 2 months. and WOW DOES IT HELP. seriously.... grab this. and all you guys who responded, GRAB THIS MOD.