I think you just have to press unsubscribe.

Simple enough

Btw, if you want to just temporarily disable, you can do that in the Extras menu.

Lets say I want to play a survival map with a couple friends and I have a gameplay-changing mod installed. Do all the clients that connect need the same mod as well?

Yes, and they all have to run the moddable version as well. This is one area that we would like to automate, but haven't yet decided on what is the best approach to do it.