This is really a good thing please don't call it SpyWare and panic!

[strask]

news: In Alpha 19, and presumably going forward, the game will include software from GameSparks to collect statistics about things like

• When players pick up broken glass, how often do they just throw it out again?
  
• Do people get too many feathers? Is the hypothesis that overabundance of feathers makes eggs more rare than they should be, due to players not wanting to even look in the nests because too many feathers?
  
• Blah blah blah more questions that they could make the game better especially loot balance if they knew the answers.

The Gamesparks software, as far as I can figure out from a verbal description on the livestream, (followed by actual research, of course) basically just sends your gameplay decisions up to the GameSparks cloud for storage. Then once there is data to look at, the Fun Pimps can sit around with a thing called Nimbus (which is open source software and may be found at nimbusproject.org ) and run custom data science queries on it.

It's pretty cool stuff.

Oh also, GameSparks-the-company also sells some products and features that support microtransactions. And some stuff that looks like actual spyware. And you know, I bet about a third of the things they sell could be reasonably criticized without even needing to think much about it.

In this case, though, I'm pretty sure that the fun pimps are just using GameSparks for the https://www.gamesparks.com/blog/gamesparks-game-data-service/ bucket to hold the data in, and a virtual machine to run Nimbus on and suck up all the data. All the analysis is inside fun pimps, in my rose-colored-glasses imagination.

So, presuming that's the case, I do not object in any way. I also really really don't want any community panic over this. And I'm sure at least once some joker is going to post "I decompiled the game and the spyware is evil bad stuff and also please use my referral code over on some random other game community" or whatever.

So, I would like to volunteer my time to the community, in parallel with whomever else decides to independently do it, to dissect the analytics traffic between my machine and the GameSparks servers, to find (hopefully reassuring, but otherwise at least useful) evidence about what exactly gets sent.

I run the game on linux so I can also use strace(1) to peek at every system call, it's a super great tool and I hope every linux user knows it exists. :)

So down below this first post, we can talk about "what's up with spyware" and speculate about what we think the funpimps would do or how this is implemented or what our concerns are with this whole addition to the game. Meanwhile up here, I'll update the post a bit as I think of easy tips for fellow analytics inspection enthusiasts, and when the experimentals phase begins and we can grab our own copies of A19, then we'll also presumably discuss findings as they are made.

Just basically doing this so that the inevitable controversy at least has a structured, orderly beginning. :) And keep in mind my presumption is that everything is totally cool. Cheers all.

Update: Prime (one of the fun pimps devs) was asked during a June 14/15 (depending on time zone) livestream on twitch, about the analytics. He said the best person to ask would be present on the stream on June 17. UPDATE UPDATE: According to Roland, Programmers will be answering questions on June 24, not June 17. Roland speculates Prime may have mixed up the two dates.

[SylenThunder]

Yes, the new GameSparks addition is just tracking gameplay data. Things like what you're getting from looting, things you craft, what you buy and sell at the trader.

GameSparks collects and creates data charts for the Pimps to get better idea of how we play. Which will in turn help them balance aspects of the game better.

[strask]

Thanks SylenThunder, that was my understanding as well. And, since it's not too much work, collecting reassurring evidence is a goal of mine. :)

[JimmyIowa]

Originally posted by strask:

And, since it's not too much work, collecting reassurring evidence is a goal of mine. :)

As someone who is passably knowledgeable with ip traffic analysis and security in general, I can tell you that the contents of the packets are almost certainly compressed. This means that they will show as binary gunk in your analyzer. Which means that even with a packet sniffer you probably won't know what data is being sent without some serious work. You can tell what thread is sending data, to what remote addresses, whether it was a push or pull operation, (did your machine send it or did the remote machine request information), what ports it is using, the amount of data, and maybe a few other details, but that's about it.

The main thing to look for is not the ip traffic so much as what it accesses in your computer. If gamesparks accesses only memory and files from the game then there's no issue. If gamesparks tries to access memory outside the game, files not game related, change internet access, the registry, or spawn other processes that don't have a command line, then you look deeper.

[strask]

Originally posted by JimmyIowa:

Originally posted by strask:

And, since it's not too much work, collecting reassurring evidence is a goal of mine. :)

As someone who is passably knowledgeable with ip traffic analysis and security in general, I can tell you that the contents of the packets are almost certainly compressed. This means that they will show as binary gunk in your analyzer. Which means that even with a packet sniffer you probably won't know what data is being sent without some serious work. You can tell what thread is sending data, to what remote addresses, whether it was a push or pull operation, (did your machine send it or did the remote machine request information), what ports it is using, the amount of data, and maybe a few other details, but that's about it.

The main thing to look for is not the ip traffic so much as what it accesses in your computer. If gamesparks accesses only memory and files from the game then there's no issue. If gamesparks tries to access memory outside the game, files not game related, change internet access, the registry, or spawn other processes that don't have a command line, then you look deeper.

Oh totally, but before even going out the ip interface the game has to send the data to the GameSparks dll, and that interface is where I'm really going to be gathering my info I think. I would guess by your phrasage that your level of knowledge is comparable to mine, although mine may have been more in depth at one time I am guessing yours is more recent as I've been on a break, so speak, for some years. Of course that "mine may have been more in depth" assumption is probably because you were simplifying for your audience, like I did, and we thought each other simple due to our simplifications. Heh. :)

[strask]

Additional thought. I need to look into the available 7dtd mod loaders now, because that's actually probably one of the best inspection methods. :)

[JimmyIowa]

Originally posted by strask:

Oh totally, but before even going out the ip interface the game has to send the data to the GameSparks dll, and that interface is where I'm really going to be gathering my info I think.

a) that sort of misses the major point that gamesparks dll can potentially gather data on its own without it being sent by the game. I already know that 7dtd.exe doesn't do any of the dangerous stuff (like reading private files) I listed above. Therefore that data won't come from the game.

b) how exactly do you plan on sniffing the interface between two processes at runtime?

[strask]

Originally posted by JimmyIowa:

b) how exactly do you plan

JimmyIowa, I appreciate your concern, or whatever feeling is motivating you to challenge the feasibility of this project. However, debating the feasibility of the project was not the purpose with which I posted the thread.

The purpose of this thread is twofold, 1) To discuss, as is generally relevant to us all as gamers, the topic of spyware included in games, and 2) to report anything I learn from whatever investigation I do, once the experimental version of alpha 19 becomes available to me in roughly 13 days.

I have not planned my approach in great detail at this moment, as I'm only just now finishing the introductory section of the GameSparks API documentation. And I don't intend to talk out my ass about things I haven't learned yet. I am merely confidant that with the tools at my disposal it shouldn't be too very difficult to characterize what the software is doing, given that I have almost two weeks to study and write a bit of code if needed before the target software is in my hands. I'm even happy to open my own analytics account with GameSparks, drop in my credentials instead of those of the fun pimps, send all the data to a container I pay for (for a single play session, might be affordable... haven't looked at price list yet) and just read it. I don't know, is the point. But I have an interest in pursuing the project, so I'll just do so and share what I learn.

In the meantime, while I am very open to suggestions, information, pointers, whatever, I think the most valuable use of my time is to just do it, rather than talk about how I intend to do the thing I would be otherwise doing if I weren't talking about it. But I'll catch you all up once I have accomplished something, or determined that no such accomplishment is possible. :)

PS just to be clear, any information I obtain and present here will of course, unlike the panic brokers, be accompanied with detailed, step by step accounts of how I obtained the information, such that anyone with a little time could reproduce and check for themselves.

[JimmyIowa]

Originally posted by strask:

JimmyIowa, I appreciate your concern, or whatever feeling is motivating you to challenge the feasibility of this project. However, debating the feasibility of the project was not the purpose with which I posted the thread.

I wasn't debating the feasibility of it. I do this sort of stuff for games all the time. I was letting you know how to accomplish it , and letting you know what probably is a dead end (ip traffic analysis).

Please relax. I am not "challenging" anything. I'm not contesting anything. I already said I have only a passing familiarity with ip traffic analysis. I am basically just an educated layman at security. I am a computer professional, but my field is algorithms and coding theory, not security. Please please please relax and don't think of this as a competition. Please? :)

The thing motivating me to respond is that I do, in fact, check the security behavior of most games that I run regularly. You basically posted about something I do, and have an interest in. For example, I could give you information on what EAC does in your machine. :) You asked about someone else who could independently work on this in parallel. But then get defensive and testy at a person who can, and ask why they are even responding?

My advice for the first step, if you were new at this, which you aren't, but if you were - don't bother with IP traffic analysis. Use some basic tools to watch what machine resources are touched by the process of interest. Files handles, spawned threads, etc. This will tell you much more about what a process is doing in your machine than looking at a stack trace would. The stack trace won't be very helpful unless you have symbols for the processes involved .

Originally posted by strask:

In the meantime, while I am very open to suggestions, information, pointers, whatever, I think the most valuable use of my time is to just do it, rather than talk about how I intend to do the thing I would be otherwise doing if I weren't talking about it.

Okay in your OP you were asking about other people to work together in parallel. And you also wanted to open up a discussion. Now you don't want to discuss it, and apparently don't want someone who knows a thing or two to help in parallel.

Would you please sit back, take a deep breath, and not feel threatened that I know a tiny bit about desktop security?

[strask]

Originally posted by JimmyIowa:

I wasn't debating the feasibility of it.

My apologies, then, for the misunderstanding on that particular point. :)

Originally posted by JimmyIowa:

I do this sort of stuff for games all the time. I was letting you know how to accomplish it , and letting you know what probably is a dead end (ip traffic analysis).

Excellent, more people should do that! And yes, on the ip side it isn't a good place to work on it. Which is why I never talked about IP in my OP. Before going out over the network, the data shuffles around in memory. That is also part of the "traffic" as it were, getting from 7dtd.exe to whatever the endpoint is at GameSparks.

Originally posted by JimmyIowa:

Please relax. I am not "challenging" anything. I'm not contesting anything. I already said I have only a passing familiarity with ip traffic analysis. I am basically just an educated layman at security. I am a computer professional, but my field is algorithms and coding theory, not security. Please please please relax and don't think of this as a competition. Please? :)

Request(s) granted with enthusiasm. I am not unrelaxed, and I sincerely appreciate that you want me to be relaxed. :) I wish the same for you. <3

I am not currently a computer professional, but I was for many years, and my speciality was security. Reverse engineering malware is a specialized sub-topic that, while I've never really gotten too deeply into, was an area of expertise for many of my friends and peers, and I picked up a few tricks by osmosis.

Originally posted by JimmyIowa:

The thing motivating me to respond is that I do, in fact, check the security behavior of most games that I run regularly. For example, I could give you information on what EAC does in your machine. :) You asked about someone else who could independently do this. But then get defensive and testy at a person who can.

I am sorry if you thought that I was defensive or testy. My internet-writing-tone can sometimes come off that way, but I think it is just because I use more words than most people, because I hate not being precise. And, I think for some people, they go on more at length when angry or upset, whereas for me I just try to be super super clear so nobody could possibly take the wrong literal meaning from my words. But then people read between the lines, and there are too many lines to read between, and they suspect there must be more emotion behind all that text than there really is. Or at least that's the story in my head. :)

Originally posted by JimmyIowa:

My advice for the first step, if you were new at this, which you aren't, but if you were - don't bother with IP traffic analysis. Use some basic tools to watch what machine resources are touched by the process of interest. Files handles, spawned threads, etc. This will tell you much more about what a process is doing in your machine than looking at a stack trace would. The stack trace won't be very helpful unless you have symbols for the processes involved .

Well said.

Originally posted by JimmyIowa:

Okay in your OP you were asking about other people to work together in parallel. And you also wanted to opern up a discussion. Now you don't want to discuss it, and apparently don't want someone who knows a thing or two to help in parallel.

I think this may be the root of our misunderstanding. When I said "in parallel with whomever else decides to independently do it", what I meant was "You may wish to do something like this yourself all alone, then afterwards we can compare notes!"

It was, unfortunately, not an offer to collaborate because, while collaboration is something I need more of in my life, I'm not currently able to show up with any reliability in a team setting. Right now I just have to do my own thing.

Originally posted by JimmyIowa:

Would you please sit back, take a deep breath, and not feel threatened that I know a tiny bit about desktop security?

I once again apologize that I made you think I felt that way.

[JimmyIowa]

Originally posted by strask:

Cheers then. Glad we came to a better understanding.

I will, indeed, quite certainly be making my own observations about what sparky does. Because this is one of the most played games in my gaming history. I go to bed with this game on pause. So if it is doing something questionable, I will want to find it. Plus...I enjoy security as a hobby.

Good luck in your investigations. I hope we find nothing questionable, because I hope it is doing nothing questionable.

[strask]

Originally posted by JimmyIowa:

Originally posted by strask:

Cheers then. Glad we came to a better understanding.

I will, indeed, quite certainly be making my own observations about what sparky does. Because this is one of the most played games in my gaming history. I go to bed with this game on pause. So if it is doing something questionable, I will want to find it. Plus...I enjoy it as a hobby. Good luck in your investigations. I hope we find nothing questionable, because I hope it is doing nothing questionable.

Agreed and celebrated in all respects. :)

[NWRMidnight]

I don't like the idea of them using the data to balance the game for 2 reasons:

1. It will cause the game to be balanced towards 1 or 2 game play styles only, leaving all other's out in the cold.
2. It won't be accurate because of the RNG factor where some people get way to much of something, where other's never get it, or rarely see it.

[Panda]

I didn't watch the dev stream but did they mention what the performance hit of this is going to be? Would it be possible to opt out? Thanks.

[SylenThunder]

Originally posted by Panda:

I didn't watch the dev stream but did they mention what the performance hit of this is going to be? Would it be possible to opt out? Thanks.

I haven't noticed any performance hit from it at all. I am uncertain as to whether there will be an option to opt-out of it. And I haven't watched the stream yet either so I don't know if they said what their plans are for leaving it it. Maybe it will only be in for the experimental test phase, and will not be in Stable. Or maybe it will be in the client until full release.

[Triggerhappy]

Is there a way to tell what players have done through the creative menu? That would be a good inclusion to consider while making interpretations on the data. I did a decent amount of testing on crafting/looting using the creative menu when a18 exp came out (and found some item stat issues that I hope are fixed in a19), and I imagine that sort of play would be difficult to distinguish from normal play without additional considerations.