Steam Controller - Consumable Bug

[Salsgore]

So I've been playing Witcher 3 recently with the steam controller. I have it set up where the right track pad works as a mouse and that witcher always considers the game to be in "Controller Mode".

After a while of playing though I noticed something strange. Basically, If my last input wasn't the track pad the game would read my right consumables, but if i was moving around with my the analog stick and the track pad ot would cause the consumables to flicker between "sets".

From what I can tell it seems that the game considers a set of consumables as for the controller and mouse inputs. I'm wondering if there is some way to stop this from happening. It tends to cause meto drink the wrong potion and just upping my toxicity accidentally.