steven.l.starr Dec 15, 2013 @ 2:58pm
NextGen Steam and SteamOS Architecture
I'm a former core Gtk and Qt developer about a year ago I ended my contribution's and started developing an alternative toolkit.

I would like to share my thoughts and ideas in regards to the development of NextGen Steam and SteamOS.

Out of fear of this post being deleted by a mod for what ever reason I am only going to share a minimal feature set and hardware requirements plus supported platforms, until others post are made to this thread.

The toolkit its self is a full stack, equivalent to Qt5 minus the Web Browser module.

Graphics can be rendered in OpenGL 1,2,3,4+ or ES 1,2,3+
OpenVG, Nvidia Path Rendering Ext or Anti-Grain Geometry can be used to draw widgets depending on if your using an Intel, Nvidia or Ati GPU.

The toolkit is completely modular, to the point that you can compile and use a single class if that is all that's needed without having to compile an entire module set or the toolkit its self.

The toolkit also has no out side dependence's things like Sqlite or PostgreSQL or Png, WebP, Ptex, Theora, WebM are incorporated into the toolkit its self.

The toolkit uses premake as its build tool but premake is also incorporated.

The toolkit is BSD licensed and contains no code that uses viral copy left licenses (GPL, LGPL)

Supports: Windows XP, Vista, 7, 8, Linux, Mac OS X, FreeBSD
iOS, Android, Blackberry QNX.

x86, x86_64 and Arm

Minimum system requirements: 75 MHz CPU and 8 MB RAM
Last edited by steven.l.starr; Dec 15, 2013 @ 3:00pm
Showing 1-15 of 25 comments
< >
Zyro Dec 15, 2013 @ 11:10pm 
As far as I can see, you neither put a question nor proposed anything.
I think you should dare to.

Just one thought: Incorporating all this stuff could impose a nightmare of scurity support.
I think modular software is here for a reason. But that's just my two euro cents...
LOLCAT Dec 16, 2013 @ 1:54am 
Valve has its own GUI system they've been working on since the beginning of Steam. I don't think they'd just switch to anything and abandon their work.
steven.l.starr Dec 16, 2013 @ 2:03am 
Originally posted by Zyro:
As far as I can see, you neither put a question nor proposed anything.
I think you should dare to.

I was going to save my question or proposal until after I shared my current work and ideas. There fairly lengthy so I didn't want to write that all down or be out right rejected until after I shared.

Originally posted by Zyro:
Just one thought: Incorporating all this stuff could impose a nightmare of security support.
I think modular software is here for a reason. But that's just my two euro cents...

When I say "Incorporated" in terms to thirdpart library's Example(Sqlite), that's in reference to
there build systems. Typically with OSS you will have multiple build systems (Autotools, CMake, Scons, QMake, Premake, ...) in regards to that I just rebuild those packages build scripts to use premake instead.

Here are some example commands to generate project files.

Linux: ./premake4 gmake
Windows: premake4 vs2013
Mac: ./premake4 xcode4
...

Here is an example premake script this one builds the Cache Module
solution "Hydra"
configurations {"StaticRelease", "StaticDebug", "SharedRelease", "SharedDebug"}

project "Cache"
language "C++"

includedirs {"AbstractCache", "AccessExpirationDecorator",
"AccessExpireCache", "AccessExpireLRUCache", "AccessExpireStrategy",
"ExpireCache", "ExpireLRUCache", "ExpireStrategy", "KeyValueArgs",
"LRUCache", "LRUStrategy", "StrategyCollection", "UniqueAccessExpireCache",
"UniqueAccessExpireLRUCache", "UniqueAccessExpireStrategy", "UniqueExpireCache",
"UniqueExpireLRUCache", "UniqueExpireStrategy", "ValidArgs"}

files { "include/**.hpp", "src/**.cpp" }

configuration "StaticRelease"
kind "StaticLib"
defines { "NDEBUG" }
flags { "Optimize" }

configuration "StaticDebug"
kind "StaticLib"
defines { "DEBUG" }
flags { "Symbols" }

configuration "SharedRelease"
kind "SharedLib"
defines { "NDEBUG" }
flags { "Optimize" }

configuration "SharedDebug"
kind "SharedLib"
defines { "DEBUG" }
flags { "Symbols" }


However I have been doing some heavy re-structuring of how some of these thirdparty src directory's are layed out. This isn't a security issues, and in many cases i've actually improved src code maintenance and manageability, in some cases I have been able to expose hard to track bugs do to errors in those particular packages build systems.

Lastly much of my motivation comes from friends of mine who are in fact authors or maintainers of many of the thirdparty packages I am using.

I have been a Linux developer for a very long time(16 years) longer than most current developers, I helped in development of many OSS projects during there initial development (Gtk/Gnome then Kde/Qt) ...

http://qt-project.org/member/2183

I have held a position as an editor for sites like..
KDE-Look.org
KDE-Apps.org
KDE-Files.org
Qt-Apps.org
Qt-Prop.org
MeeGo-Central.org
Maemo-Apps.org
GNOME-Look.org
Xfce-look.org
...
Since 2004.

See here for proof of being and editor since 2004.
http://kde-look.org/usermanager/search.php?username=comtux

So this isn't something that I am or will be going at alone. FreeBSD in particular needs this toolkit.

Thanks for that first post. I am going to go ahead and prep for my next post in regards to NextGen Steam and SteamOS Architecture
Last edited by steven.l.starr; Dec 16, 2013 @ 2:09am
steven.l.starr Dec 16, 2013 @ 2:32am 
Originally posted by LOLCAT:
Valve has its own GUI system they've been working on since the beginning of Steam. I don't think they'd just switch to anything and abandon their work.

Are you referring to VGUI? I believe Steam has 2 ui library's one is legacy.

But regardless this is completely different this replaces there use of Gtk there are huge disadvantages to Gtk(I might go into this later). And steams gui library VGUI? is fully compatible regardless, and more than likely already employs most of the same tech. The gui(widget) side is agnostic and works via Path Rendering Example(Nvidia Path Rendering Ext) and from the modifications I have seen to SteamOS current code base(Compositor) there already using this on supported hardware.

But as I said the toolkit is modular at the class level you don't have to use anything you don't need. Its designed to be a SDK that you use to build custom SDK's. And uses OSS library's that Valve is already using. It also free's Valve from its responsibility of GPL and LGPL licensing requirements. As everything is BSD licensed or Equivalent(Boost, MIT, Zlib/Png, ...)

Also note that from what I have seen in the Steamworks SDK, future development of Steam will be very much Community orientated. Right now I could rewrite the Steam Client using the Steamworks SDK with the exception being I don't see any support for actually Installing Games but everything else appears to be there.

If Valve created a REST or XML-RPC api for Steamworks that would be even better, and eliminate all or most thirdpart dependence's.
Last edited by steven.l.starr; Dec 16, 2013 @ 2:58am
steven.l.starr Dec 16, 2013 @ 3:16am 
Before I get trolled by fans of RMS and the GPL/LGPL licenses for this comment ...

It also free's Valve from its responsibility of GPL and LGPL licensing requirements. As everything is BSD licensed or Equivalent(Boost, MIT, Zlib/Png, ...)

Let me explane something to you. Not long ago the FSF sued a FOSS developer for mixing GPL2 and GPL3 code and refusing to re-licenses his code base. When that happen'd it spit our community into two sides, those that want the GPL in user-space and those that didn't.

Currently user-space is being re-written to remove all possible GPL/LGPL code, at current assessments it should be a year or two before all GNU code is completely replaced, everything from GCC to Glibc, Bintuils, Coreutils, ... is being replaced. The only GPL code that will remain is the Linux Kernel.

Right now there are a handful of experimental linux based distros using these new userspace components, and there working on testing the 16,000+ packages you would find in a typical linux distros package management repo.

FreeBSD, NetBSD, ... developers are all working with Linux devs to accomplish this huge undertaking. And also purging GNU code from there operating systems also.

Several large commercial corps have been funding the move in addition.
Zyro Dec 16, 2013 @ 3:41am 
Originally posted by steven.l.starr:
Before I get trolled by fans of RMS and the GPL/LGPL licenses for this comment ...

It also free's Valve from its responsibility of GPL and LGPL licensing requirements. As everything is BSD licensed or Equivalent(Boost, MIT, Zlib/Png, ...)

Let me explane something to you.
[...]

Yeah, this seems suitable to not end up in licence wars. Not.
steven.l.starr Dec 16, 2013 @ 3:44am 
Originally posted by LOLCAT:
Valve has its own GUI system they've been working on since the beginning of Steam. I don't think they'd just switch to anything and abandon their work.

I just wanted to touch on one more thing in regards.

Everything about this SDK is Current and Future Standards compliant. Which mean that regardless of what Valve uses now, not only is Valve but Microsoft, Mac, Linux and every other Software Application, Game and Hardware platform will eventually be using the same tech and features in my Hydra Toolkit.

The point wasn't to re-invent anything its called NextGen because that's exactly what it is a NextGen toolkit based on tech that everyone is moving to or will be using in the future regardless of what they use now. All I am doing is getting a huge jump on the industry as I don't have legacy code bases I have to support. Others like Windows, Mac, Linux do, and are doing the same-thing but on an incremental bases.

steven.l.starr Dec 16, 2013 @ 3:47am 
Originally posted by Zyro:
Originally posted by steven.l.starr:
Before I get trolled by fans of RMS and the GPL/LGPL licenses for this comment ...



Let me explane something to you.
[...]

Yeah, this seems suitable to not end up in licence wars. Not.

Doesn't make any difference the FSF should have know better, and in doing so they lost 90% of there developers. GCC has what 3 full time developers where as LLVM has hundreds and is backed by Apple, AMD, Nvidia, Intel, and ....

Disney, Pixar, Sony are all using the BSD license for there contributions, in fact last I heard BSD Licences code development is beginning to surpass that of GPL in the Linux Kernel.

Last edited by steven.l.starr; Dec 16, 2013 @ 3:50am
Zyro Dec 16, 2013 @ 3:52am 
Originally posted by steven.l.starr:
Originally posted by Zyro:

Yeah, this seems suitable to not end up in licence wars. Not.

Doesn't make any difference the FSF should have know better, and in doing so they lost 90% of there developers. GCC has what 3 full time developers where as LLVM has hundreds and is backed by Apple, AMD, Nvidia, Intel, and ....

Disney, Pixar, Sony are all using the BSD license for there contributions, in fact last I heard BSD Licences code development is beginning to surpass that of GPL in the Linux Kernel.

Ah, cool, now you're _really_ getting to make this thread not being about a licence war!
steven.l.starr Dec 16, 2013 @ 4:20am 
Originally posted by Zyro:
Originally posted by steven.l.starr:

Doesn't make any difference the FSF should have know better, and in doing so they lost 90% of there developers. GCC has what 3 full time developers where as LLVM has hundreds and is backed by Apple, AMD, Nvidia, Intel, and ....

Disney, Pixar, Sony are all using the BSD license for there contributions, in fact last I heard BSD Licences code development is beginning to surpass that of GPL in the Linux Kernel.

Ah, cool, now you're _really_ getting to make this thread not being about a licence war!

Maybe but, I am mild in comparison to others especially Rob Landley. My defense mechanism
kicks in when I talk licenses, its a learned behavior.

But if you prefer a particular license you might be happy to know that the BSD license is only a requirement for internal and contributed code.

I added an additional clause that allows you to fork and re-license the code base GPL or MIT, .... or whatever. Originally I was going to not licenses the toolkit at all but my legal representation informed me not licensing isn't considered valid by us law.

Anyways I want to talk about software development and not licenses just got distracted there for a min. Once I get my kids off to school I will do that.
Zyro Dec 16, 2013 @ 4:27am 
Originally posted by steven.l.starr:
Anyways I want to talk about software development and not licenses just got distracted there for a min. Once I get my kids off to school I will do that.

I'd consider that reasonable. You know, in Linux world you can alway have a licence, distro or desktop environment war (and probably others), but it will distract you from what you want to get done.
steven.l.starr Dec 16, 2013 @ 8:19am 
I should probable start out by talking about the Graphics Framework, as that's what most people here will be able to directly relate to if they have ever done any type of game development.

If your familiar with X(GLX) or Wayland and how OpenGL works with them. What they basically do is provide a container for a GL rendering surface, with input events.

Some History
--------------------------------------
X has some serious problems at its current state, these issues could be fixed and rather easily
but you would endup breaking compatibility with every know Linux Gui application and you would still have the burden of having to deal with a 26+ year old code base that wasn't very clean and organized(Easy to Read).

Some of my first Gui related experiments, came in the form of trying to fix a problem I was having when trying to combine Qt4 and Qt5 with Ogre3D(3D Graphics Rendering Engine for Games) <-- Torchlight uses it. It turned out that Qt's abstraction layer over thirdparty supported librarys and features(QObject, OpenGL, ...) was causing a huge reduction in performance, and some compex GL Context related conflicts. Also the way Qt is designed it forces you to adapt your code base to there arch design <-- See comment by Leadwerks author.

Also Qt5 had removed the Xlib back-end and replaced it with XCB, I don't know if they were aware or if they just didn't care but XCB doesn't support hardware accelerated graphics with the Ati and Nvidia binary drivers it offloads that to Xlib instead. So instead of having Xlib as an abstraction over the X-Server you now have XCB and Xlib sitting in the pipeline. This is highly inefficient. Note Gtk does this also.

After what I call the "Phoronix Event" were I basically made national Linux news, pointing out the problems with XCB and Qt5, it was apparent that many in the Linux community, didn't understand or care about the issues, and those who did only cared about there targeted hardware Wayland(Intel), Mer(Arm), none of them particularly cared about Nvidia or Ati. Situations is a little different now at least in regards to Mer.





In my frustration I started experimenting with my own display server. Using SFML as a testing bed I created a very simple server that used zeromq as a rudimentary IPC server.

And I started testing out how to send rendering commands via a client. My first test left me in absolute shock, what I had done was sent a rendering call in the form of XML/JSON that looked like ...

<?xml version="1.0"?>
<Widget id="MainWindow">
<Widget id="PushButton1">Button</Widget>
</Widget>

{
"#MainWindow": {
"background-color": "#000000ff",
"border-width": "1",
"border-style": "dotted",
"border-color": "#00000000",
"padding": "5",
"x": "0",
"y": "0",
"r": "0",
"width": "100",
"height": "50"
}
}

To the server via 4 different clients(C, C++, Python and Lua) not only did it work BUT I was getting a framerate of over 300FPS.

Following test involved using Skia, Magick++, Cairo, QPainter, AGG, ShivaVG) via the client and passing the buffer over the wire.

Later I tried using SDL, Glut, GLFW, Plane X/GLX based servers and in all cases performance was just around 300FPS.

At this point a few things occurred to me.

1. I want better hardware to really test a theory I had
2. I need a hardware accelerated solution for 2D graphics
3. I had just created a Client/Server application that supported both XML and JSON but also
didn't care what Vector or Raster based Graphics Library or Programming Language I was using.

I had to hold off for a couple of months, to save for a new system, what I purchased was a
current generation MB that had onboard mSata, a 60GB mSata SSD card, Intel i7 cpu and a Nvidia GTX 660 card.

After installing Kubuntu on this new system, I knew right away that SSD ROCKS, Kubuntu was booting instantly to the desktop.

At this point I really wanted to test these new GPU's out, I started with Intel and used OpenVG, and re-implemented the Display Server I was experimenting with earlier. Performance was amazing.

It took me awhile, but eventually after a conversation with a Nvidia employee I was made aware of the Nvidia Path Rendering Extension. This is when the real shock kicked in, my test were showing that I was rendering at over 3000FPS

As a reference X renders at I think a fixed(60FPS) and QtOpenGL reenders at 130FPS last I checked. Cairo and OpenGL last I checked was around 30FPS(Note: Path Rendering).

For those that don't know what Path Rendering is see here https://developer.mozilla.org/en-US/docs/Web/SVG/Tutorial/Paths and here https://developer.nvidia.com/nv-path-rendering
OpenVG and most Software based Vetor Graphics librarys support path's.

For some interactive path rendering examples you can play with see http://paperjs.org/examples/


In short Path Rendering isn't new but Hardware Accelerated Path Rendering on Linux is, no one is using it for there Gui Librarys.

Here is a short list of what the Graphics Stack currently looks like.

OpenGL 1,2,3,4+
OpenGL ES 1,2,3+
GLSL
OpenVG
Nvidia Path Rendering
AGG Vector Graphics
Bullet 2D and 3D Physics
Freetype for Fonts
Image Codec:
Png
Jpeg
OpenEXR
Tiff
WebP

This stuff works out of the box on all tested platforms, see my first post on what those might be.

Also keep in mind the Performance I mention here and that the Minimum system requirements: 75 MHz CPU and 8 MB RAM.


My hands are tired I am going to take a break I fill fill in the gaps shortly.
steven.l.starr Dec 16, 2013 @ 3:37pm 
I didnt't go into very much technical detail about the graphics stacks for a reason. Most of what I have been doing is researching and testing how alot of core components and features are developed in linux so the community at large can develop from scratch on there own at will because the next thing I am going to tell you is going to sound kinda tinfoil'sh and crazy.

After PRISM, and all the new information we have learned about the NSA the one thing you don't want to do is trust the US Government to respect your rights and privacy. I think we can all agree on that at least I hope we can.

Before you read any further go here http://en.wikipedia.org/wiki/Hugh_Shelton and ask your self if you would trust him NOT to violate your privacy? When you have your answer read on.





That is General Henry Hugh Shelton(US Army) he is the chairman of Redhat. Redhats biggest customer is ........ "The NSA"

Redhat also controls various key components of your typical Linux distribution, in particular the Gtk toolkit which Valve uses for Steam and SteamOS. Out of the top 5 Gtk developers three work for Redhat, one works for Google and one works for Novell.

All 11 of the Gtk team members work for a huge corp, and only one is a full time developer,
Lead programmer Matthias Clasen(Redhat)

And every Linux user know the NSA has already tried to plant a backdoor in linux they vary well could have succeed as ...

An intentionally planted exploit is almost impossible to detect if you know what your doing, I could show you a simple networking example that featured an "Off by One" bug that would ruin your day. And unless you were looking for it would never see it.

Also Gtk GObject is just plane horrible, there are a thousand and one reasons why you don't try to shoehorn OOP ontop of plane C.

The solution at-least in regards to a graphics stack is to implement everything your self and live by this rule ... "Never touch anything you can't control, audit, maintain"


Thankfully everything I will be presenting, any decent programmer will be able to control, audit, maintain, and developing a Display Server/Widget Toolkit/ ... in a couple of days will be an fairly easy task. The hardest part to software development is always trying to figure out how to do something you have never done before. Once done you can normal do it again blind folded.

Another concern in regards to graphics programming is patents, luckily we are using primarily OpenGL and OpenVG and both being standardized, come with some pretty hefty patent protection, against trolls.

Same goes with our Core classes, the core framework is actually a merged and re-factored fork of Poco C++ and Boost. So in time many of the classes and features it contains will more than likely become apart of the C++ standard.

Those with experience with Boost will be thinking "oowwww Boost is kinda heavy" to those I will point to my above original post. And also mention we are designing a new Linux based distro from scratch that only contains our selected library's.

And Gtk, Qt, Wx, Fltk won't be apart of this new design.
Last edited by steven.l.starr; Dec 16, 2013 @ 3:44pm
Oerthling Dec 16, 2013 @ 4:07pm 
Bold claims.
Got me curious. I googled "hydra toolkit" - couldn't find anything (or rather there are many hydra links that are not this toolkit).

Is there a public URL?
steven.l.starr Dec 16, 2013 @ 4:18pm 
Originally posted by Oerthling:
Bold claims.
Got me curious. I googled "hydra toolkit" - couldn't find anything (or rather there are many hydra links that are not this toolkit).

Is there a public URL?

I am sure many were told that post PRISM.

No its not publicly released I am still working on it. But if you want to see some of my earlyer work you can see here. There not directly related but are lol.

https://bbs.archlinux.org/viewtopic.php?id=125088&p=1
http://en.sfml-dev.org/forums/index.php?topic=6651.0
http://stackoverflow.com/questions/6639640/want-to-learn-graphics-using-skia-on-ubuntu
http://www.ogre3d.org/forums/viewtopic.php?f=1&t=79384
https://github.com/zester/Quantum

If you would like to see more I might be able to dig more up I have alot out there.
Last edited by steven.l.starr; Dec 16, 2013 @ 4:20pm
Showing 1-15 of 25 comments
< >
Per page: 15 30 50