Real world performance testing by Shane and Krash has indicated that the primary performance bottleneck is the server to client packet limits, and not the ghost duplicate detection. In the current implementation, one packet is sent per client remote procedure call, which occurs whenever an object is moved, added, deleted, cloaked, resized, etc..
Fortunately, this issue is relatively easy to fix by clustering updates into larger packets. Around 250 bytes can be packed in a packet, which would reduce the number of packets required to about 5% of current. Loading a 2000 piece building at 24 packets per second could take up to 80 seconds. With basic clustering, this time is reduced to about 4 seconds.
I ran some practical upper limit tests on SI buildings as well. It appears that the game does various book keeping operations on every object that exists in the game, independent of any scripts. On my relatively modern hardware, the time spent computing those objects starts producing unplayable levels of lag around 25,000 pieces.
I have generated and loaded a 300x300 maze (44735 pieces) incrementally to measure that lag. Pretty final result image is attached. Framerates were about 0.1 fps.
Edit: Attached the maze save file if anyone else wants to play around with it.
Line 54 - %this.setDatablock(%payload); - does not properly tell the client which datablock to switch to.
To see datablock changes, change this line to:
%this.setDatablock(ServerConnection.getObject(%payload + 1));
Just updated the auto-updating client file from the original post to reflect this change.
The script is glitched. When you run ACCM, ServerConnection.getobject(0) doesn't return -1, thus having the server send you the SI ghosts causing unneeded lag. However, this glitch could be caused by Krash's quick installer, ACCM or Windows Vista.
I think that loaded pieces are somehow different than pieces newly places (In ECM at least, I don't know about other mods). I was using the editor gun in my server, and any pieces that I had loaded would not have the editor ghost appear while being edited. I could edit them, though that is hard to do without the editor ghost.
After applying edits, it would leave a copy of itself that had no collision, which would go away right when I placed a new piece. None of these problems occurred when I edited newly placed pieces. I haven't really had the time to try to fix this, so I'm not sure if it is ECM or SI.
The script is glitched. When you run ACCM, ServerConnection.getobject(0) doesn't return -1, thus having the server send you the SI ghosts causing unneeded lag. However, this glitch could be caused by Krash's quick installer, ACCM or Windows Vista.
I know we havn't been best of friends but I got to give you two some respect considering you solved a problem that MostLikely and I couldn't even solve working directly with GarageGames.
I've updated the first post to indicate recent changes.
I've experimented a bit with various spatial partitioning schemes to reduce the lookup time for an object based on transform/scale/datablock, and as of the 0.6 SI client, I've come up with something extremely viable. Loading times of large buildings still suck, but at least now, the duplicate detection code doesn't produce lagspikes. Additionally, there are none of the false positive/false negative issues that were common with the 0.1 client series. In fact, the client is very pleasant to use, even with thousands of pieces coming and going.
I'm still testing out some of my notification conglomeration ideas, but so far they don't look very promising. It might be possible to boost transfer times by 50-100%, but it still ends up slower than ghost transmission by a few times. Once I finish these conglomeration experiments, I'll consider some sort of public release.
There are still some semi-serious bugs that need to be addressed found during testing today. It ends up that there are some false positive issues with the ECM editor tool for v-ghosted objects. I'm going to add a "protect" notification that allows temporarily overriding the duplicate removal code, since this could potentially cause problems for any deployable/tool that causes overlap.
Well, I'm going to try packing two bytes into one. There are only 15 characters used in the transfer, so each can be encoded as 4 bits. It will probably make the Huffman coding less efficient, but worth a shot.
Alternatively, since only 15 characters are used, I could replace them with values that correspond to the shortest bit sequences in the code table -- I just don't know where in the Torque source that is located.
Using the high frequency Huffman codes to encode payloads seems to have a pretty measurable improvement. A test building load time was reduced from 15 seconds to 11 seconds, and deletion time was reduced from 7 seconds to 5 seconds. Overall, roughly a 30% decrease in transfer time. Not a whole lot, but better than nothing.
I also made some interesting observations for very large buildings -- the game doesn't send all of the ghosts for a building when it is loaded, rather just the ones in sensor range. Roughly 2/3rds of the pieces that were queued to be ghosted weren't sent until I approached them. It goes to show how essential out of order processing of ghosts and script generated notifications is.
I have some ideas for tag label display on the client side, but those still need work...
I've fixed the last major show stopper bugs in the new client series. The 0.7 client and 0.5 server are now in the update system. With this new client, you REALLY need to make sure that it's been downloaded and loaded by the auto-updater before you join a server.
Expect frequent minor updates in the near future as I add a few new notification types.
Colors! I've posted a new server and client script that, in addition to a few bug fixes, have support for tag labels on SI generated objects. I noticed that \c0 to \c5 all produced the same white text, and I decided that I might as well use this redundancy to add a few more colors. \c5 is still white, but \c0 to \c4 are red, orange, green, and blue. The color characters that were processed by the tag display before are the same as before.
You can see a rainbow tagged solar panel in the attached screen shot. Keep in mind that these extra colors only appear if the object is an SI generated virtual ghost. You'll just have the bland white colors for non-SI objects.
Oh... and you'll need a client patched with Linker's raycast container reselection for this to work. At the moment, you'll need to patch your executable yourself. I'll release a patched one in a short while.
Comments
Fortunately, this issue is relatively easy to fix by clustering updates into larger packets. Around 250 bytes can be packed in a packet, which would reduce the number of packets required to about 5% of current. Loading a 2000 piece building at 24 packets per second could take up to 80 seconds. With basic clustering, this time is reduced to about 4 seconds.
Expect an update in the near future.
I have generated and loaded a 300x300 maze (44735 pieces) incrementally to measure that lag. Pretty final result image is attached. Framerates were about 0.1 fps.
Edit: Attached the maze save file if anyone else wants to play around with it.
To see datablock changes, change this line to:
%this.setDatablock(ServerConnection.getObject(%payload + 1));
Just updated the auto-updating client file from the original post to reflect this change.
Thanks, Krash.
After applying edits, it would leave a copy of itself that had no collision, which would go away right when I placed a new piece. None of these problems occurred when I edited newly placed pieces. I haven't really had the time to try to fix this, so I'm not sure if it is ECM or SI.
It's probably Windows Vista.
Very clever Linker and Electricutioner.
Good job guys. :clap:
I know we havn't been best of friends but I got to give you two some respect considering you solved a problem that MostLikely and I couldn't even solve working directly with GarageGames.
I've experimented a bit with various spatial partitioning schemes to reduce the lookup time for an object based on transform/scale/datablock, and as of the 0.6 SI client, I've come up with something extremely viable. Loading times of large buildings still suck, but at least now, the duplicate detection code doesn't produce lagspikes. Additionally, there are none of the false positive/false negative issues that were common with the 0.1 client series. In fact, the client is very pleasant to use, even with thousands of pieces coming and going.
I'm still testing out some of my notification conglomeration ideas, but so far they don't look very promising. It might be possible to boost transfer times by 50-100%, but it still ends up slower than ghost transmission by a few times. Once I finish these conglomeration experiments, I'll consider some sort of public release.
14ish bytes for a transform instead of 26ish bytes...
though you have to binarify numbers manually, which might be somewhat painful, and then base64ify it.. which might also be painful...
Though you can probably defer the work to a dll or something...
...
or maybe perhaps poke a dso so it pushes the value as a float and read it back as an int...
...
or maybe what i say is totally pointless since all the strings are huffman coded before being sent...
...
for some reason i think this has been discussed before...
Alternatively, since only 15 characters are used, I could replace them with values that correspond to the shortest bit sequences in the code table -- I just don't know where in the Torque source that is located.
Looks like it might make some difference.
I also made some interesting observations for very large buildings -- the game doesn't send all of the ghosts for a building when it is loaded, rather just the ones in sensor range. Roughly 2/3rds of the pieces that were queued to be ghosted weren't sent until I approached them. It goes to show how essential out of order processing of ghosts and script generated notifications is.
I have some ideas for tag label display on the client side, but those still need work...
Expect frequent minor updates in the near future as I add a few new notification types.
You can see a rainbow tagged solar panel in the attached screen shot. Keep in mind that these extra colors only appear if the object is an SI generated virtual ghost. You'll just have the bland white colors for non-SI objects.
Oh... and you'll need a client patched with Linker's raycast container reselection for this to work. At the moment, you'll need to patch your executable yourself. I'll release a patched one in a short while.
No other changes have been made.