Left 4 Dead 2

Left 4 Dead 2

View Stats:
Tsuey Jun 14, 2019 @ 12:59am
FIXED -- Broken Finales :: the hunt for an impossible fix
WARNING: I'm about to get technical and a little theoretical.
WARNING: This thread is a constant work in progress.

This thread's focus is solely the rare bug of escape vehicles instantly arriving. I sincerely ask that should you post here, to remain on topic, ideally to post your own latest experiences (with video if possible i.e. ShadowPlay) of the bug.

If you have any sources or corrections to make, please do not hesitate!

This dumb bug is going to be fixed in 2019 even if it kills me.

UPDATE: This bug has still not been fixed. It can be "predicted" if the new Versus ladders fail to load, and may also result in Mutations failing to run -- on extreme rarity, and exclusively on Official Dedicated.

HOW CAN YOU HELP?

Typing "status" into console will give the server's IP and list of players which will look like this:

udp/ip : 162.254.194.172:27103 [ public same ]
# 1852 1 "Tsuey" STEAM_1:1:62897574 00:26 93 0 active 30000

Given the details from Lux, it's apparent that the User ID #'s (here mine is 1852) are more important than the server IP, as they at least give some record of how long the server has been running.

This information will be incredibly useful. If your escape vehicle arrives instantly, please, "status".

CONTEXT

Over the last couple months I've been working on custom-tailored map fixes for Valve with Rayman1103, who has successfully submitted map fixes in the past. This isn't a pursuit to rid every exploit from the game. All shortcuts and teamwork-based Infected tricks are being ignored. If you think something should or shouldn't be fixed, I'm eager to hear what you have to say, so please add me and let's discuss it. I've already heard from many of you, and appreciate all the feedback.

However, this thread is not the place to discuss any of those other bugs or exploits.

TABLE OF CONTENTS

The intrigue is in the bug's rarity. The goal is to understand why and ideally reproduce it. Getting it fixed is actually at the bottom of the priority list.
  1. HISTORICAL PRECEDENCE
  2. DEAD CENTER 4 ANALYSIS
  3. Goben's comment about VPK'ing an empty script
  4. Lux's Vscript leak explanation and SourceMod plugin
Last edited by Tsuey; Apr 14, 2022 @ 7:07pm
Originally posted by Tsuey:
In conclusion...

Valve fixed this bug today on April 14, 2022:

- Fixed a long-standing issue where scripts would randomly fail to load, leading to a variety of game-breaking bugs including missing crescendo events and finales that end instantly. Thanks to the community members who helped target specific server instances for investigation.

Yesterday, at 9:14 AM Seattle time, we made Valve aware of an actively cursed server:

hostname: Valve Left4Dead 2 US West Server (srcds101-tuk2.117.216) version : 2.2.2.4 8491 secure (unknown) udp/ip : 208.78.166.228:27230 [ public same ] os : Linux Dedicated map : c5m1_waterfront players : 8 humans, 0 bots (8 max) (not hibernating) (reserved 186000000233bca)

At the time, only the Infected ladders added with TLS on Parish 1 failed to spawn. Parish 2 was just fine -- but, if scripts were to break on that map, it'd have caused Commons to not spawn during the tower event.

By 11:06 AM, our team confirmed several instances of the failure on that server, most of them while playing RocketDude.

While instantly arriving escape vehicles is one symptom, quicker means of identifying the bug are if RocketDude unloads after map transition, or if the No Mercy 1 fans[github.com] don't spin -- since that was fixed with script for TLS.

Also -- for the first time -- it was observed that carnival games such as the Dark Carnival 4 Whacker don't work (the heads didn't pop up).

Note that in RocketDude on Parish 5, the escape chatter and sequence will always start immediately -- 100% unrelated to the script failure bug, and exclusive to RocketDude.

At 11:57 AM, Valve responded and was looking at log data by 12:54 PM.

We updated them on RocketDude unloading across map transition. Atypical script terminations was their first sign of proof of the issue, but nothing stood out as obvious.

At 1:48 PM we caught a live one on Dead Center 4, where the escape sequence (and infinite Tanks) started as soon as Survivors descended the elevator. Even though we poured all gascans, we couldn't escape since only script unlocks the car.

By 1:51 PM, Valve saw long streams of script errors where "DirectorScript" and "MapScript" can't be found.

This was the first live proof of the bug Valve had ever documented.

The bugged finale persists forever if the map isn't reloaded. If the map is reloaded, it's back to randomness if it happens again. We had well over 5 bugged finales in a row.

Until 2:38 PM, we idled on the server to ensure it wasn't taken by anyone else.

At 3:14 PM, it was determined the cause is a combination of a large number of script operations running combined with a slowdown in the server. This code has existed since L4D2 shipped, so it has always been there -- the extra scripts added with TLS just increased the likelihood.

At 5:29 PM, the bug was fixed:

So the summary is: deep in the bowels of the Squirrel VM is a counter that tracks significant script operations. If that counter goes over 100k for a single script call, it's assumed that is a runaway script and a callback is triggered that can handle the issue. Our callback checks a timer to see how long the call was running, and greater than 0.1 seconds then it aborts the script, otherwise it lets it continue.

The issue was, that counter only gets reset when the callback runs - rather than after every script call. So every 100k operations TOTAL the callback runs (in RD it's about every forty seconds). In most cases, the script call that happened to be active at the time probably took about 0.0001 seconds so everything continues like normal. But if there's any lag going on at the time, the timer might trip and the script is aborted, and then all bets are off.

Now, the counter gets reset after every script call, so the callback will only trip on a legitimate runaway script - which in reality would probably lag out the server and kick players before it managed to kill the script and continue playing in a broken state anyway.

Unrelated and to be fixed later is a race condition issue[github.com] that causes Tank spawns on maps like No Mercy 1, Dead Center 1, or Sacrifice 1 (before scripted boxcar Tank). Those happen because the Tank spawns before the script that restricts that spawn has time to run. The scripts still run in these cases, except they're too late.
< >
Showing 1-15 of 81 comments
Tsuey Jun 14, 2019 @ 12:59am 
HISTORICAL PRECEDENCE

This was never a problem in L4D1 because Vscripts weren't a thing then. L4D1 finales were kept very simple and were handled solely by a trigger_finale entity. When it comes to L4D2, Vscripts offer amazing possibilities except for the unfortunate fact that over-reliance in them has likely caused this bug, illustrated by the fact that all L4D1 maps in L4D2 now use Custom _finale.nut Vscripts[github.com] to do all the work instead. When this bug occurs, it's most likely because the Vscript read fails, resulting in a finale with 0 stages that is effectively equivalent to the finale being completed.

I'm going to chronicle every historic public record of this bug right here to illustrate how unacceptably prevalent it has been, in my pursuit of actually fixing it, largely thanks to ArchiveTeam[www.archiveteam.org] and their efforts preserving SPUF.

Several Valve developers have commented on this glitch, such as Chet Faliszek[en.wikipedia.org]. I will be extracting all of them here, per discovered thread.

Valve developers have done the best job in the time they have investigating this bug, but honestly it's brutally clear they had no clue about how specific it is. I don't think this is a server performance problem but I'm also unsure. But it is definitely a Vscript problem. And only trigger_finale Vscripts.

The problem is its apparent impossible reproducibility.

CHRONOLOGICAL ORDER

Nov 11, 2012 -- Space-Man Gympy -- 24 posts
L4D2 Finale glitch (Steam Community)

Jan 8, 2013 -- Buick72 -- 10 pages
Broken finales (SPUF ArchiveTeam)[web.archive.org]

Chet Faliszek:

If you experience this issue - please go into the console and run the status command and from there post the ip address of the server here.

Mar 19, 2013 -- sg6970 -- 9 pages
Broken Finale's Part II List Your IP's (SPUF ArchiveTeam)[web.archive.org]

Chet Faliszek:

Actually, we need the server ip. You can get that by typing
status
in the console. You can just cut-n-paste that complete output here.
List what you were playing (vs/coop), what happened - did you go straight to the finale, restart, etc.

Jun 5, 2019 -- scrubbles -- 7 posts
What's with all the bugged finales recently? (Steam Community)
Last edited by Tsuey; Jun 14, 2019 @ 5:59am
Tsuey Jun 14, 2019 @ 12:59am 
DEAD CENTER 4 ANALYSIS

The following code excerpts are from both the decompiled map[raw.githubusercontent.com] and its c1m4_atrium.nut Vscript[github.com] and c1m4_atrium_finale.nut Vscript[github.com].

Dead Center 4 is an exceptional case because its escape vehicle doesn't activate instantly like other maps. No matter how many hours you hold off the hordes, it never will as it's handled differently than others. So I'm getting that one out of the way first.

If you noclipped out into the atrium without riding the elevator, you'd notice that there is a manual button to start the event originally meant for development testing (and later used in cool tricks for pre-gathering cans):

"id" "387548" "origin" "-4451.92 -3207.61 2" "VersusTravelCompletion" "0.1" "type" "4" "targetname" "trigger_finale" "skin" "0" "model" "models/props_mall/mall_display_03.mdl" "angles" "0 270 0" "classname" "trigger_finale"

The c1m4_atrium.nut Vscript[github.com] doesn't do anything notable beyond setting it so Single Players only need to pour 8 cans. It is executed by logic_auto as soon as the map loads via:

entity { "id" "775527" "origin" "-4472 -3492 9" "targetname" "relay_begin_director_script" "StartDisabled" "0" "spawnflags" "2" "classname" "logic_relay" connections { "OnTrigger" "@directorBeginScriptc1m4_atrium0-1" } }

Note already how this is the only Vscript that's explicitly called within the map file. The "_finale.nut" Vscript is never referenced by name at all.

Its biggest significance is to know that it's not the Vscript that's bugging out... but the fact it also never fails, is of extreme value.

In fact this line within that Vscript:

EntFire( "progress_display", "SetTotalItems", NumCansNeeded )

Is extremely fascinating because whenever the instant escape vehicle bug occurs, you're still fully capable of pouring all the cans. It's just that the escape vehicle will never activate after all of them are poured.

I have never seen or heard of any instances of this Vscript falling to execute. If you've seen Official Dedicated server instances of having no can counter whatsoever, let me know, because I'm adamant it's impossible.

So, we can presume that it's not a generic Vscript read or precaching flaw but rather a problem with the trigger_finale entity itself.

When you reach the bottom of the elevator:

"classname" "func_elevator" connections { "OnReachedBottom" "relay_force_finale_startTrigger0-1" "OnReachedBottom" "goal_elevatorKill0-1" "OnReachedBottom" "relay_elevator_bottomTrigger0-1" "OnReachedBottom" "door_elevatorOpen0-1" "OnReachedBottom" "button_elev_3rdfloorKill0-1" "OnReachedBottom" "relay_elevator_bottom_scavengeTrigger0-1" }

The logic_relay "relay_force_finale_start" is Triggered, which does this:

entity { "targetname" "relay_force_finale_start" "classname" "logic_relay" connections { "OnTrigger" "trigger_finaleForceFinaleStart0-1" } }

So, that mall display mentioned earlier? It's effectively automatically pressing that button for you, when you reach the bottom of the elevator. Every map where you don't manually press a button to start a finale has one of these, in various forms.

With c1m4_atrium_finale.nut Vscript[github.com] now finally running, the Director detects when gas cans are poured through its "OnTeamScored" output that's only relevant for Scavenge mode:

entity { "id" "528412" "origin" "-4496.35 -3296 168" "targetname" "@director" "classname" "info_director" connections { "OnTeamScored" "branch_gascan_usedSetValue101" "OnTeamScored" "@directorrunscriptcodeDirectorScript.MapScript.LocalScript.GasCanPoured()0-1" "OnScavengeTimerExpired" "relay_scvng_timer_expireTrigger0-1" "OnScavengeRoundStart" "relay_scavenge_roundstartTrigger0-1" "OnScavengeOvertimeStart" "relay_scvng_overtime_startTrigger0-1" "OnScavengeIntensityChanged" "case_scvng_intensityInValue0-1" } }

This line in particular:

@directorrunscriptcodeDirectorScript.MapScript.LocalScript.GasCanPoured()

Executes this Vscript:

function GasCanPoured() { GasCansPoured++ DelayPoured++ Msg(" Poured: " + GasCansPoured + "\n") if ( GasCansPoured == NumCansNeeded ) { Msg(" needed: " + NumCansNeeded + "\n") EntFire( "relay_car_ready", "trigger" ) } EvalGasCansPouredOrTouched() }

So, remember how Dead Center 4's escape vehicle will never activate? it's because this "relay_car_ready" is supposed to do that. But because c1m4_atrium_finale.nut Vscript[github.com] isn't running at all, that will never happen.

And since c1m4_atrium_finale.nut Vscript[github.com] is demonstrably not running, this isn't a generic Vscript issue but rather one solely with regard to trigger_finale.

And I'm betting that it's a failure of Vscript filename reconciliation, because this type of behavior isn't present anywhere else with Vscripts, and all those other Vscripts work just fine:

Finale scripts
The <map name>_finale.nut script is automatically loaded when a finale map starts. The script is triggered when the trigger_finale is used, either manually by the player, or with the ForceFinaleStart input.
To do: trigger_finale also has an option to specify a finale script, but it doesn't seem to work
Last edited by Tsuey; Jun 14, 2019 @ 1:58am
Tsuey Jun 14, 2019 @ 1:00am 
L4D1 MAP ANALYSIS

All L4D1 finales are identical with regard to this issue. Frequency of occurrence is solely based around anecdote, because functionally they're not different at all.

Most L4D2 finales are also identical. The exceptions are Parish[raw.githubusercontent.com] and Cold Stream[raw.githubusercontent.com] as the only 2 finales where it's impossible for this bug to occur. It's not because they're gauntlets. It's because they don't have a "_finale.nut" to mess up.

These are the finale types available with trigger_finale:

Finale Type <choices> (New with Left 4 Dead 2)
Specifies which style of finale to trigger
0 : Standard
1 : Gauntlet
2 : Custom
4 : Scavenge

Script File <string> (New with Left 4 Dead 2)
To do: add description

And this is Parish:

"id" "1395479" "origin" "-11530 6088.74 533.43" "VersusTravelCompletion" "0.2" "UseDelay" "30" "type" "1" "targetname" "finale" "StartDisabled" "0" "model" "models/props_unique/generator_switch_01_outline.mdl" "FirstUseDelay" "0" "angles" "0 90 0" "classname" "trigger_finale" connections

Parish and Cold Streams are both "type 1" Gauntlets.

Every single other finale is covered by either the DEAD CENTER 4 ANALYSIS with regard to Scavenge, or is "type 2" for Custom that then calls upon Vscripts.

Given that all L4D1 maps are identical, I have been using Blood Harvest 5[github.com] for all my tests.

Blood Harvest 5's trigger_finale:

entity { "id" "238321" "origin" "6799 1350 274" "VersusTravelCompletion" "0.33" "UseDelay" "9" "type" "2" "targetname" "radio" "StartDisabled" "1" "spawnflags" "0" "model" "models/props/terror/hamradio.mdl" "disableshadows" "1" "angles" "0 270 0" "classname" "trigger_finale" connections { "FinaleEscapeStarted" "new_apc_arrive_relayTrigger0-1" "FinaleStart" "radio_game_event_preKill0-1" "EscapeVehicleLeaving" "//camera_truck_relayTrigger0-1" "UseStart" "radio_game_eventKill0-1" "UseStart_" "soldier_radio_onourway_lcsStart0-1" "FinalePause" "orator_farm_radioSpeakResponseConceptfarm_radio_intransit 0-1" "UseStart" "orator_farm_radioSpeakResponseConceptfarm_radio_button2 WhoDidIt:!Activator0-1" "FinalePause_" "soldier_radio_intransit_relayTrigger0-1" } }

It does the following:
  • It's "type 2" Custom. This can be changed to "type 0" Standard to effect the same behavior as in L4D1. This would completely resolve the bug, because this bug never happened in L4D1. But Valve made it into a Vscript for some good reason, so demolishing their asset this way is not a proper fix.
  • There's 2 identical radio models. The 1st is a func_button that on press initiates the "Good there's somebody alive out there!" chatter. This Enables the actual trigger_finale radio model (which starts Disabled), required for the 2nd press to confirm and start the finale after a delay of 9 seconds.
  • "camera_truck_relay" is commented out because it's a L4D1 Beta camera angle that overlooks the escape route. Most Beta stuff really was meant to never see the light of day, because the final version is actually good.
  • "FinaleEscapeStarted" Triggers "new_apc_arrive_relay" which is the escape vehicle.

I made a very simple modification to that trigger_finale that's identical to the above except for one key difference:

connections { "FinaleEscapeStarted" "!selfKill0-1" }

When the finale finishes, the radio model deletes itself.

My goal is to prove that a trigger_finale with no stages read from Vscript results in the instantly-arriving escape vehicle and "infinite panic" that players experience.

Blood Harvest 4[raw.githubusercontent.com]'s barn is a convenient place to inject a custom finale for alternative tests, of which I've been through dozens:

https://steamcommunity.com/sharedfiles/filedetails/?id=1769488402

There, too -- on a map that's never had a finale -- I was able to replicate the bug. Just, it's unfortunate this is absolutely meaningless because of all the modding required.
Last edited by Tsuey; Jun 14, 2019 @ 2:59am
Tsuey Jun 14, 2019 @ 1:00am 
CHAT LOG EXCERPTS
Tsuey Jun 14, 2019 @ 1:00am 
FIX THEORIES
Tsuey Jun 14, 2019 @ 1:00am 
REPLICATION

Almost none. Because it's a nightmare.

This needs some kind of master of hacky reproduction
Last edited by Tsuey; Jun 14, 2019 @ 3:47am
Tsuey Jun 14, 2019 @ 1:00am 
RESERVED
Tsuey Jun 14, 2019 @ 1:00am 
RESERVED
Tsuey Jun 14, 2019 @ 1:00am 
RESERVED
Tsuey Jun 14, 2019 @ 1:00am 
RESERVED
Bob the Boomer Jun 14, 2019 @ 1:49am 
good over view, hope you continue this and we see a good outcome
Everice Jun 14, 2019 @ 2:25am 
So what are you going to do to make sure vscript reads them properly?
This looks interesting to me
Tsuey Jun 14, 2019 @ 3:01am 
Originally posted by Everice:
So what are you going to do to make sure vscript reads them properly?
This looks interesting to me

That's a great question because you know what? I have no clue yet. I'm in the process of bothering literally everyone about this bug until I find someone who is smarter than me about it.

That's the brutal truth.

This thread is outreach for finding someone who knows their stuff. It's do or die time, because I'm sick of how old and un-addressed this bug is.
Last edited by Tsuey; Jun 14, 2019 @ 3:04am
JAiZ Jun 14, 2019 @ 3:13am 
Originally posted by Tsuey:
Originally posted by Everice:
So what are you going to do to make sure vscript reads them properly?
This looks interesting to me

That's a great question because you know what? I have no clue yet. I'm in the process of bothering literally everyone about this bug until I find someone who is smarter than me about it.

That's the brutal truth.

This thread is outreach for finding someone who knows their stuff. It's do or die time, because I'm sick of how old and un-addressed this bug is.
It's also worth noting that Tsuey has gotten in contact with people who have experience with the existing bugs in the game, notably people who have played a lot and have encountered such bugs often. I'll be keeping logs and sharing all details possible with Tsuey when I run into bugs in the future, as well as try to replicate them, in order to accurately pinpoint what is causing it in the first place. That will allow Tsuey to start making precise fixes. It's essentially "late stage beta testing" for l4d2
Last edited by JAiZ; Jun 14, 2019 @ 3:14am
$pirit Jun 14, 2019 @ 3:43am 
Please don't fix this game, its perfect how it is.exploits glitches and all.
< >
Showing 1-15 of 81 comments
Per page: 1530 50

Date Posted: Jun 14, 2019 @ 12:59am
Posts: 81