Any chance we could get API support for Stable Diffusion webui and/or KoboldAI?

[NaN 𝒆]

For those of us who prefer to play on their own hardware, this could bring great benefits in terms of flexibility and performance.

For Stable Diffusion, the game's built-in implementation recommends ~8Gb (iirc) of VRAM; on the other hand, generating an image via sd-webui (with the xformers optimization) only consumes about ~4Gb of VRAM on my machine.

On the text generation front, the differences can be even more drastic. With recent developments (and a bit of hacky tinkering) allowing one to load textgen models in 8-bit or 4-bit, rather than the usual 16-bit, providing immense VRAM savings. Notably, this is done with minimal loss to output quality.

With this in mind, one should be able to run Stable Diffusion along with a 6B/7B textgen model (in 4-bit) with a little under 12Gb of VRAM. In comparison, when using the built-in implementations, I hit my GPU's 12Gb limit running the 1.3B and SD at 448x448.

Sorry if this comes off as a little rude. It's certainly not my intention; I just suck at writing sometimes :p

[AI Roguelite: AI-generated RPG]

No problem; it depends how much time it will take me to move over to the new stuff. I was not aware of 8-bit and 4-bit models; can you link to the github and/or other resources/tutorials on how to use them?

[NaN 𝒆]

Personally, I used this guide[hackmd.io] (minus the bits regarding TavernAI) to set up the 4-bit fork of KoboldAI. You might find more information here, although that guide pertains to text-generation-webui rather than KoboldAI.

The guides are still a little complicated, and require a bit of computer know-how that not every user might have. Fortunately, the API between the 4-bit fork and the original KoboldAI should remain the same. Because of this, my idea was to have the game support the KoboldAI API as an option for text generation, while leaving the actual installation and such of KoboldAI up to the user.

The game would just need to know the IP and port of the KoboldAI server that the user intends to use. Usually, this would be 'localhost:5000' if the user is running it on their own machine, but one could even run the server on a separate machine on their LAN.

If by any chance you already have an installation of KoboldAI on your computer, you might find the API documentation helpful. The API docs can be accessed by appending '/api' to the KoboldAI server url. (ex. 'localhost:5000/api')