Be sure that the game is using the highest refresh rate supported by your monitor. You want something like "1920 x 1080 x 60 Hz", and not "1920 x 1080 x 24 Hz". We've patched the game to select the highest refresh rate by default, but there was a bug causing the game to select the lowest refresh rate. It's best to just check your display settings in the game.
Also, try removing any USB input devices that are plugged into your PC. The Cave supports XBox360 gamepads, but some players have found that other peripherals (such as joysticks, wheels, and non-xbox gamepads) can cause the framerate to decrease over a few minutes. We aren't sure why this is happening yet, so while we're sorting that out, please just unplug those devices while playing The Cave.
Finally, The Cave's support FAQ is here: http://www.doublefine.com/forums/viewthread/8420/
Hope that helps!