Ollama/Custom OpenAI API URL Request

[HuMan34b]

It would be awesome if we could have other backend servers that we could connect the large language model directly from. It could possibly come from either a local host server or a custom OpenAI URL. It would be somewhat similar to what Oobabooga offers, but for Ollama, the model name would have to be set (such as setting the name to 'llama3' or something similar).

[MASTER virgin]

This. Having the ability to connect to a custom model proxy a la SillyTavern would make this a 10/10 game. Especially if Claude 3 support is in the cards.

[User]

I think the game developer consciously doesn't add this because it would compete with his own cloud offering. I've asked for this years ago, and the response just wasn't there.

[Dementia5 認知症]

Why not make use of Kobold CPP, or OpenRouter as a cloud based model?

[AI Roguelite: AI-generated RPG]

Currently there are two ways to hook it to a proxy; either Kobold API or Oobabooga API. There is no limitation to how you use these. You could even write a script that uses it to provide your own OpenAI or Gemini API key. However I probably won't make this an official option, in case it creates a trend where only the heaviest users using tons of tokens remain on paid cloud.

Kobold CPP should work out of the box via Kobold API.

I will look into whether it makes sense to support Ollama's API. Ideally there should only be 2-3 standardized API's which the open source community agrees to use, rather than a separate one for every model hosting service

Master virgin if you could provide some more info about how SillyTavern does it, I could look into that as well

[Dementia5 認知症]

Originally posted by AI Roguelite: AI-generated RPG:

Kobold CPP should work out of the box via Kobold API.

It does.

[User]

Originally posted by AI Roguelite: AI-generated RPG:

Currently there are two ways to hook it to a proxy; either Kobold API or Oobabooga API. There is no limitation to how you use these. You could even write a script that uses it to provide your own OpenAI or Gemini API key. However I probably won't make this an official option, in case it creates a trend where only the heaviest users using tons of tokens remain on paid cloud.

Kobold CPP should work out of the box via Kobold API.

I will look into whether it makes sense to support Ollama's API. Ideally there should only be 2-3 standardized API's which the open source community agrees to use, rather than a separate one for every model hosting service

Master virgin if you could provide some more info about how SillyTavern does it, I could look into that as well

It should be trivial for you to add - you just allow users to provide a BASE URL for the OpenAI api, see https://github.com/openai/openai-python?tab=readme-ov-file#configuring-the-http-client, basically the default is https://api.openai.com/v1, you should just make it configurable so people can replace that URL base with any other URL. And of course then the OpenAI key will actually be the key used for the proxy. Additionally (and it should be quite easy too) adding ability to specify custom model name for when using the OpenAI API should be nice.

[AI Roguelite: AI-generated RPG]

It sounds like the "base URL" is the same as what already exists in the oobabooga api (aka openai api) option isn't it?

Then, you could specify the model name by manually adding the key/value pair to the moddable json in model-params\oobabooga-common-params.txt (note if you add it under "default" it may be overwritten by updates, so you may want to create a mod first and add it there).

Adding a key to use on the auth header isn't supported yet, partly for reasons I mentioned above, but might be in the future.

[MASTER virgin]

Originally posted by AI Roguelite: AI-generated RPG:

Master virgin if you could provide some more info about how SillyTavern does it, I could look into that as well

Hey mate. Sorry for the late reply. Don't know much about the specifics, but it basically lets you connect to any API supporting OpenAI's API formula, like GPT and Claude and so on.
Here's a link to it's documentation that goes into more detail:
https :// docs.sillytavern.app/usage/api-connections/openai/

Also, don't wanna be all demanding or anything, but is implementing support for API keys for OAI, Anthropic, et al in the works?

[AI Roguelite: AI-generated RPG]

Originally posted by MASTER virgin:

Originally posted by AI Roguelite: AI-generated RPG:

Master virgin if you could provide some more info about how SillyTavern does it, I could look into that as well

Hey mate. Sorry for the late reply. Don't know much about the specifics, but it basically lets you connect to any API supporting OpenAI's API formula, like GPT and Claude and so on.
Here's a link to it's documentation that goes into more detail:
https :// docs.sillytavern.app/usage/api-connections/openai/

Also, don't wanna be all demanding or anything, but is implementing support for API keys for OAI, Anthropic, et al in the works?

Is SillyTavern's option any different from the Kobold API and Openai API options which already exist in AI Roguelite? The only difference I know of is that AI Roguelite doesn't provide an easy way for you to use your own API key (eg openai API key) at the moment, but everything else (including locally run LLMs or whatever server you want to connect to) should work fine.

[Dementia5 認知症]

Is OpenRouter an option?

[AI Roguelite: AI-generated RPG]

Originally posted by Dementia5 認知症:

Is OpenRouter an option?

I will think about it, but for now assume there's no plan to support it

[dan]

Has this changed yet?

[AI Roguelite: AI-generated RPG]

Yes, OpenRouter has been supported since December 2024: https://store.steampowered.com/news/app/1889620/view/523076978666898101

[tecmacrun]

Soooo is the Ollama API supported? I am interested in the game but i dont want to redo my LLM server setup and i could not find a comprehensive list of the supported APIs

[AI Roguelite: AI-generated RPG]

Originally posted by tecmacrun:

Soooo is the Ollama API supported? I am interested in the game but i dont want to redo my LLM server setup and i could not find a comprehensive list of the supported APIs

According to someone on the Discord:

the developer of AI Roguelite did not add support for the Ollama API. But the devs behind Ollama have added OpenAI API-request support to Ollama.
AI Roguelite Supports OpenAI API-requests too:
Instead of:
Ollama API ↔ AI Roguelite
Its:
Ollama ↔ OpenAI API ↔ AI Roguelite