This reply is more in reply to your mod- comment ... I hit the 1000char limit... cut pasting here...
Will respond to the above in a minute.


As you found out it's more then 40 lines to do it right everywhere but it's not that big a deal in and of itself. I went though that process with v1.07b of the game. I successfully changed all the 32k limits to 65k and 262's to 512k, and got the saving process to work fine so long as you used my assembly (as I set my own version number for testing and code triggered accordingly). Like you said the problem comes in once you want compatibility. (you can find the details on pdx forums)

Yes in most places in thier serialize and de-serialize they 'assume' a set size based on s.Version of the game, instead of just storing a 4 byte header storing the number of objects that are stored so it know how far to read. In at least some places they do it cause they're are refferences elsewhere that link to specific index# in what they are reading back, which is why they can't do what I suggest in some cases, in others, like 'trees', there is no reason other then consistency\simplicity as to why it's done.

See unlimited trees for an example of storing the custom data and changing the array size BEFORE their own deserialize kicks in. Basically you detour their method to your own in on-created (not onlevelloaded)... change the array size, run basically their code to load the first part (in your case 16384), then your own to load the rest. You still have a compatilibiy issue though if someone loads a map without your mod vehc mod though, as unlike trees there are refferences all over the place to vehicle indexs that will be higher then exist without your mod running. As for the saves... you store you extra data in the crp file with the map (so you don't need the file name, nor do you screw with the root datastream), when you save your bytes, you save to a sort of seperate path with in the file like /veryuniquename/yourmodname; when you load you load back from the same path.

I think I sort of touched on most your points in the first post, but some additional thoughts relating to vehicle limit extention specifically. prepare for a wall of text.


1. Are you sure your getting 'everwhere' where the 16384 limit is applied? because any routine that assumes that limit will need to be detoured to apply you new limits. Just as a random example (and forgive me if I'm wasting your time cause you know this), AircraftAi.CanSpawnAt()
spins it's way looping though the m_vehicle array and assumes a 16k max for breaking out. There is no way to change that other then detouring AircraftAi.CanSpawnAt with you identical own version with your own new limit. Well there is another way... raw editing\patching the assembly... but I'm assuming we're talking about making a public mod and not some jacked up version of the assemby I did with 1.07 just to see if 'it could be done without performance issues'.

I count up to 137 places where you might need to do that. (it's probably a little less, but there are at least 137 ref's directly to m_vehicles, and vast majority have loops like that). Also... even if you do detour them all... it could break other mod's who have their own AI's and assume 16k... though smart moders will size their stuff to m_vehicles.m_size instead of hard coding... but I digress. ;)

1a.. Also and off hand I forget.. but watch out for things like 'xxx_grid or xxx_updated array sizes too that need to be equally scaled or sometimes at first things will 'work' but your actual 'new instances' that > 16k may not actually show up the way they are supposed to. Again this is just from memory for when I did all this for segments\nodes\lanes\electric\water\sewage\paths\ etc. Vehcs might be a little different.

1b. Also watch out for things like the infoview related code that may not show instances that are higher than the base limit as their data arrays are matched in size to the counts but it's not always obvious because they store multiple things in a large array. This is just from memory when I did roads... in like the first 32k of a 192k array was segments, then next 32k was nodes, the next something else...etc.. so I had to adjust those arrays and the associated and related code to match the new figures. What i mean here is like make sure if you spawn a vechicle # 17000 and it's a garbage truck.. it actually shows up on the map and in the info-view as there AND purple. It's little sh1t like that that was the hard stuff to track down when I did this stuff.

1.c Sometimes..and this you probably noticed, but incase you didn't ... sometimes (actually most times when it comes to 'simulationstep or 'updates' etc type functions) there are either coroutines to process only so many entries per-frame... or code that only specifically runs on specific frame #'s. As I recall you can see this best in some of the buildings related code where in certain code it only updates as I recall 128 buildings per certain frameindex# and if you follow the math it works out to eventually 32k (their limit)... so when I jacked stuff up to 65k I had to change there formula to still only run on the frames they wanted (I was keeping sh1t simple) but now I needed to do 256 buildings per pass. It actually would have been a far harder to do if I didn't use power of 2 as the new limit, as lots of their bitshifting code revolves around that.

omg I'm so randomly rambling right now... sorry so much floating around in my head from may when I did all this (and got it all working), and then a few days later 1.1 came out and I was like... ♥♥♥♥ I'm not doing all this again 739 places changing just numbers, a dozen'ish places where formuals needed changing a dozen places where save\load routines needed to be modified (and that was made easy based on my own assigned version number# bump). Again that was just raw editing the assembly with reflixIL - the purpose was just to let me and like 3 other people play around - not for public distribution.


2. You shouldn't need to go the painful reflection route if you go the detour route though as m_vehicles is public.

3. In addition to detouring during the simulation load (before map load), you may want to look into hi-jacking or overriding the onbeforedeserialze methods or the onAfterdeserialze ones as well... that is if you have fix-up things before or after the data is loaded.

'unlmited trees mod'
It's funny back in april\map I never actually increased the trees in my test version because when I looked at it because a) I didn't have the personal need, and b) it seemed the easist to screw with because almost nothing other then Tree Manager touched or ref'd them. It's actually kind of funny, markabo put out the original and then didn't want to maintain it so some guys saw my 1.07b experiment and contacted me about taking it over and maintaining it. It's only then where I really saw a way of doing a real mod for other things (besides trees) as possible given the donated detours code. Go look my source for that on github in the ad_updates branch(for latest) to see how to get the timing right for serialize\deserialize. Maybe there is something specific with vehicles that creates and issue but I doubt it and if nothing else it might provide some insight and help to you. The source may not be 'pretty' and very broken out class wise like some mods but it's fully functional.

The problems are three fold as I see it with doing this in general either for just for Vehicles or a massive coordinated project that does them all (you almost have to do some in conjunction because some are interdenpent sometimes) like netsegs,netnodes,lanes, paths, electric,water, sewage, vehicles, buildings, zonedblocks, ...and I'm probably forgetting one or two others that have to be done if you touch the others.

a) Ton of work.
You have to basically detour a third of the dll...90% of that is just to change _somenumber_ to _someothernumber_ in while loops or in a few places function calls and a few others in calls to the randomizer. 5% are where you have to change some logic to match\work with your new numbers. 5% is where you have make changes in seraliziation\deserialization.

This is all "doable", a sh1t ton of work, but it can be done, particularly if split up, though it's sort of messy to split it up because various functions touch on multiple limits but again it's possible.

b) Maintence nightmare.
The real problem comes in maintaining it. Every time C\O puts out anything but tiny hotfix you're basically going to have to export their new dll to a project and scan for all differences against and export of the last dll. This assumes the decompiler tool also has not changed over that time so that you're getting consistant output, oh what joy's. Each noted change (or most of them anyway) then have to be evaluated for any corresponding changes needed now in all you're detoured versions of the same functions.

That is a lot of work, and semi-mind numbing work at that.

c) Compatibility.
So let say you got this monster unlimiter mod, it works and everything. No matter what you do the map will be unplayable without with the mod running; additionally if people want to make a compatiable map they'll need to disable the mod before creating or at least before saving the game the first time.. That is probably ok and comes with the territory.

Then there is the larger issue ... we've had to detour several hundreds of routines... some other mod(s) will of course want to detour something we've detoured and will step on us causing all sorts of odd problems, some obvious at load... some not obvious till things are ♥♥♥♥♥♥ up bad. So net-net it's a sort of becomes a "use this mod and almost nothing else" sort of mod, or at least nothing else really cool ;). I can't think of a way around that, at least in many cases.

B - Is my biggest problem and why I never moved forward doing this for the public at large, I mean if it was my day-job I might but it's not, and .net is just hobby for me anyway I'm not a programmer by trade I'm a network guy that has always liked to dabble. :)


Oh and as for current mapfile name I think you can pull that from <SavePanel>.m_lastLoadedName ,,, you might have to tack on the .crp yourself though but floating around in those related types I recall being able to find the package name\id and then being able to get the full path from the package manager. You could always look up some of the original autosave mods if they're still out there and see where they pull it from.

.... hopefully at least some of that made some sense.


What I'd like to see and I know some of this isn't good code practice is one of the the following.

1) C\O change all those instances where MAX_WHATEVER is (where the compiler is inserting the number for perfomance reasons), changed to TheManager.m_SomeVarMax or this.m_SomeVarMax which is set to the right number based on MAX_WHATEVER in Awake() or in the constructors. Now I know that's going to result in literally millions of more lookups each full pass. I'd like to know the preformance differnce though. I thought about doing that test, but when I was going at it... I had a choice... test that route that might pollute my findings or... just test 'their way' with bigger arrays, since I only wanted to do everything once I went the second route. But I am curious just how bad the impact is to changing all those lines to referance a var... if I understand things right in most cases it would be on-stack, so I'm thinking the hit may not be super bad, but I know there will be an impact - the question is, is it noticable on the low end of their system specs? It would be a pain in the ass for CO to do that, but really they only have to do it once, and I'd guess at least 3/4 of it could be a simple search replace.

Doing that removes 90% of the maintence problem, and removes the need for about an equal percentage of the amount of function that have to be detour'd. Ie a modding person doesn't have to touch all those places where just changing a raw number is needed, it just has to be done in the manager for the base number and some of those grid\update sizes and then everything else related and of course the save\load stuff, but it saves....so much unneccessary work.

2) They could just go through and increase the maxes on their own doubling the 32k ones to 65k, and everything related x2 where needed like I did. It's a more work then #1, but not that much more and then they don't really have to touch their save\load routines other then version > x = do larger number of times. Also practically speaking unless you go 81 titles... 65k on the segment stuff and 1/2 mil on the related lanes as pathfinding ... should generally be enough for everyone. I could even get close to using 65k on a 25 title map, (granted it was pre-tunnels) and while not literally maxing everything for road space I had a ridculous amount of use of segments\nods\lanes and filled everything in the 45k range so I think 65k will do. Now 81... that's a whole other ball game and I'm not sure the game itself is designe to scale that well for 81, plus there is so much other stuff that has to be changed for that...it's a whole other ball game.

Anyway the down side to just doubling things for everyone is a) everyone now uses at least some* more ram (lots of those arrays are filled with empty objects 'used' or not) and takes the small hit of cycling through the larger arrays (space used or not). Base file sizes will grow by at least 50% at a minimum. I found on an i7-950 using in the mid 40's of the larger limits < 5% cpu difference and about 200-300mb more of ram...dependent on how much you actually use the new sizes. Obviously there is the base increases.. and then 'the more you build the more it takes' factor as the arrays get filled with real\filled out objects and more and more the loops must dig past the 'iscreated' flag during their work. They could for instance bump the array's as mentioned and still keep current limits... unless you check a 'use higher limits - with big red warning' check box that ups the CheckLimit() checks to use new higher ones, and flags the map metadata to indicate this ( so if people go to load a big-map save it they can warn them if they don't have that enabled on their system).
The problem there is apparently they already have issues on the low-end side of specs with the base game so I don't know if they are willing to do this, and it's not something they can do (not easily anyway) just for a paid for expansion where they could up the requirements.

3) Rearchitech a whole bunch of things so that users can actually select at map creation time what limits they want to play with, original, extended, unsupported-but-possible. This would burn way too much time I think for C\O to consider during 1.x.x timeframe of this game for way to little reward, though for version 2.x they should do it.


BTW why they artifically limit stuff in the map editor is beyond me I get why they might want to do (max - small reserve for actual game) like they sort of do with tree's 250k vs 262k, but it seems there is no fundimental reason they do it with other things and seems a stupid choice to me, but it's also possible I'm missing something as to 'why' they do that.

Ok.. I've said way too much, but basically that's my overall head dump on these issues, just getting it out there all at once and out of the way, I know most of it isn't vehicle specific, too bad. Happy to talk specifics though. :)

Wow, thanks so much for your experiences and thoughts! I only spent a couple days fiddling with it because I was curious and hoped to finish a mod (the main mod I've been working on is a massive undertaking; basically re-engineers buildingAIs to be mod-friendly and multi-mod friendly). I'll take another look at your code at some point here; I skimmed it briefly, but yeah, must've missed where you handled the pre/post loading matters.

It sounds to me that because vehicles are so deeply engrained in the code, and they rarely are the big issue anyhow, it'd be best if I just let them be for the moment. I might look at network limits though ;)

I'll send you a friend invite; maybe you'd like to work with me on my building overhaul? Or perhaps we could work together to fix the growable limits instead? (12x12m is pretty laughable, imho). Either way, I think we could help eachother when we have questions or problems :)

Well your better off trying vehicles first probably netsegments\nodes\lanes\and pathing are interdepentent and sort of need to be done as a package I think,are like 739 ref's vs 139... just saying that's actually what I got working first before doing buildings and zones and then electric and water... which need to go hand and hand with buildings, else you get no water\electric to your buildings >32k for example.

I'm always willing to help where I can, if I know something or can point in the right direction I will. Sadly, at the moment, I wouldn't know where to start with modding the building sizes or the logic around the ai's for growable. Never looked in that area, that stuff along with all the 'magic' asset modders do to actually create their assets and mark them up right for the game is greek to me, actually I sort of understand some of it..but still... it's greek to me.


So specifically look at the override for OnCreated() https://github.com/Knighth/TreeUnlimiter/blob/ad_updates/Unlimiter/Loader.cs#L21
Irnore my stupid comments..which are old and make no sense to anyone but me anyway..

It calls Mod.setup.. which does the detouring of all the functions we need to replace.
*The key here is, this is before the map loads and deserialize calls start getting called.

Mod.Setup will redirect TreeManager.Data.Deserialize to --> My replacement one.
https://github.com/Knighth/TreeUnlimiter/blob/ad_updates/Unlimiter/LimitTreeManager.cs#L838

Which 97% of which is cut and past from C\O source with my changes to first.. EnsureInit() -> which will go change the TreeManager array size and the "trees_updated" array size scaled to match. Then do C\O's usual work for the first 262k trees.... then check if the mod is enabled go call our own custom deserialzer -->
https://github.com/Knighth/TreeUnlimiter/blob/ad_updates/Unlimiter/LimitTreeManager.cs#L712
Who loads the rest out of bytes stored in "mabako/unlimiter" ..if they're there.


We highjack TreeManager.Data.Serialize the same way.