Realism settings calibration

[dwayne2005]

This is my NBA Live historic rosters project, which derived player ratings from a spreadsheet/database of basketball statistics for every season. By default, the .dbf's are in NBA Live 2003 format but can be extracted to .csv using Open Office and converted to formats for NBA Live 2000, 2001, 2004, 2005 and 06--if you can follow my lousy instructions. But you also have the side effect of various appearance randomisations, missing faces and even worse, some black heads on white players (mainly in instances where players had similar names, eg. Magic Johnson and Marques Johnson which both result in a cyberface gen of MAJOHNS, a minor thing I shouldn't have accounted for when making the files). The files for converting over are in the .bkr BK Replace Em format, an excellent small utility I included here with the authors original installer.

http://rapidshare.com/files/326999899/HistoricRostersR2m7f.exe

Note: Some appearance stuff is put back over players that appeared in the original files. For Live 06, I also included one that inserts data to fit the Classic Teams mod. Teams will be more complete graphically with seasons around that of the original files (2004-05 here). Also, keep in mind all files are simple 0-99 equations. They don't become 'real' until you fine tune the ranges (done by applying .bkr files on the players.csv), players will be uncomfortable playing by default with very inconsistent ranges between players attributes (most notably speed).

But there is another little project I have inside this that is for settings calibration (in the Open Office .ods format). It takes the player stats files and tries to adjust the realism settings. It will work for any .dbf used, including the original files so is not made just for my project. I am not a programmer, and not experienced in spreadsheets, so it can be a bit awkward to understand I guess. Feel free to better it if you know how and think it worthwhile.

The first thing you do is take ConvertPLAYERSTATS[2005].bkr and apply it to the games player stats files (BK ReplaceEm/Ecobyte Replace Text should be easy enough to understand, but you may need to set my .reg file to set a few basic options which may give weird outputs first). This will alter the text format for insertion into the spreadsheet. You copy & paste all of the humans team data first from the player stats file. And then the computer stats below it. The team totals should be calculated under each input.



The next thing you do is you scroll back up to the top of the .ods file where you will see two team lines. These contain the teams real life stats for the relevant season. The lines can be found in the DataTEAMS[ALL].csv. You copy and paste them relative to the order they appear below (Human team first/CPU second). It is better to leave this process until after doing the above, because you are more likely to make mistakes and insert them around the wrong way as I have learned.



The next step you have to do is add the realism settings that you played that game with. I have included some default realism settings in the RECORDS tab of the spreadsheet so if you played them at the games defaults you can copy and paste each section from the two tabs. It should be obvious what it does. It tries to calculate a multiplication value of the setting that was played at to guess what the setting should perhaps be.



The settings just above it are 'weightings'. For fouls for instance, the increase is split up to 4 ways between the 4 foul categories. Inserting a number determines what weight that value should have. All with 1 mean they all get a 25% share. All but one set to 0 and that one gets 100% of the adjustment.

I never really bother with any except the 'speed format' weightings. There are 4 options for game speed adjustment there. The first one reduces game speed. The second one reduces (if played too fast; increases if played too slow) human FG shooting. The third one increases (if played too fast...) CPU block ratings. And finally, the fourth one is an option for quarter length. If the 4th one is set, what it does is it distributes it's share to either increasing or decreasing quarter length. All of the other correctors could be switched off (set to 0) and just have this one set which is just a way of saying this is the quarter length you should play at. I like to have the block setting set but the main problem is if it goes above 100 then its share is discarded. I've included a second spreadsheet for values up to 255 which can only be set in the hex. (At this point, I'm not sure what settings greater than 100 do!)

In the RECORDS tab, there is an immediate black column with the computed adjusted settings. You MUST copy those to Game 0 through Open Office's paste special/text+numbers. Copying those allows you to copy them back into the played at settings column in the main part of the spreadsheet with ease. You can store your game stats below that section if you desire to, but has served no real purpose to me.



This is the hex output.



You copy -- the cell A70 for the single game or A76 for averages (by default, game 0 doesn't count to the average so you have to play 3 games before the average gets set) -- the hex to the settings.dat file using the freeware hex editor XVI32 (provided in the authors original file) at the address beginning here using the Edit/Overwrite String option and save the file, which inserts the realism settings so you don't have to do it manually each time:



EVERY rating is (should be) calculated ratioed to FGA's. So if you play a short game, then the right amount of rebounds/steals/blocks/etc should be set. But in truth, I've only ever played 48 minute games and I'm not sure it all works for games less than that. It DOESN'T set the results to the real life game averages, but measures them up against the FGA's. Only the game speed calculations try to match FGA's to real life game averages (it takes at least two games to 'close the gap'), so between the two they should calculate the appropriate adjustments. But there are several instances where settings are going to be double counted and in the end averages will be more fine tuned than game to game.

There is another major problem with this spreadsheet. It assumes, for instance, that 30 defensive rebounds is associated with the current rating, let's say 30. Let's say in fact that should be 24, then it's going to be multiplied by about 0.8. If the value, however, was at 60, then the reduction will be double for the same ratio. As the number gets higher, the amount the value will fluctuate from game to game will be higher and higher/plus and negative... because it associates the present number with its rating. The only way around it would be to associate a number with its own rating, but that is biased, variable and kind of stupid. Average values may in the end find the proper balance from game to game. But you can stop the rating from going too high or too low by adjusting the functions (Insert/Function in Open Office) in the D columns here under CORRECT:



The functions look like: IF(ROUND(C55*B55)>255;255;IF(ROUND(C55*B55)<1;1;ROUND(C55*B55)))

Where values greater than 255 equal 255 and less than 1 equals 1 (needed to keep the formula calculating). If you had a problem with 3 point shooting, for instance, being too high, let's say you zoned in the rating to between 40-60 you'd change it to: IF(ROUND(C55*B55)>60;60;IF(ROUND(C55*B55)<40;40;ROUND(C55*B55)))

IN CONCLUSION those are my ideas, they are amateur but thought I'd share. I think it takes a lot of patience and a lot of calibrating to get real results. (It should also work for NBA Live 2004, as unlike Live 06, it exports player stats files; can't seem to find such an option for Live 06...)

I have provided an example ranged 2005 season for NBA Live 2005, but the main problem is I've made adjustments and the realism settings are going to get an overhaul once I play enough games to get a good average. I have changed the range values for blocks to 0-10 from 45-55, it makes blocks a lot harder and my realism settings for CPU blocks, offensive rebounds and fouls are going beyond 100 for the first games...

I forgot one thing...


This section can be ignored. It just determines at what percentage 'real' the individual players played at. To do this, it needs the simulated season stats at the end of the 2003 PLAYERS.dbf files. It then copies those stats to the lines in the players statistics. By default, ConvertPLAYERSTATS[2005].bkr features the stats for players from the 1979-80 season. To get it from the other seasons, you need to convert my original Live 2003 PLAYERS.dbf files to .csv then use ConvertDBFtoBKR[2005].bkr on it. Then rename that PLAYERS.csv to ConvertPLAYERSTATS[2005] <insert season>.bkr and it is your new ConvertPLAYERSTATS[2005].bkr. You'd need to make sure the lines are fully cleared at the end of the team stats above it otherwise it just couples with it seasons you don't do this with, so you usually get nonsense results no matter which season you play without this procedure as that data tends to be left over in the spreadsheet.

[Marek]

Very, very interesting, I must say...But it's too complicated for me besides my english is not so good to understand everything Anyway, you did a very ambitious project. Congrats!

[dwayne2005]

Thanks.

I'm not sure anyone who is good with English understands my English, either.

[dwayne2005]

Some improvements:

http://rapidshare.com/files/326999899/HistoricRostersR2m7f.exe

EDIT: A slight change in the way the pace is calculated for the human penalty. There are 4 main ways of accounting for game pace. Both the 'game speed' and the quarter length options are based on aggregate field goal attempts between both sides. However, for a while I switched it to just the human side for Human FG shooting and CPU block penalties. It makes sense... if you shot x amount of shots too many, you should be penalized. But it ignores the reasons why that team shot x amount more than the CPU, which is defensive (steals, blocks) which are accounted for elsewhere in the spreadsheet. I am convinced now all game speed adjustments should be based on the aggregate FGA's.



What the new two rows at the bottom of the team output aren't just inserted to look pretty, though having the averages directly lined up is a great thing. The blue DATA line under the player stats is meant to either be the team total (1-CPU/1-HUMAN) or a team reversal of the equations for one team + a game aggregate for the other (for games with two CPU's). It determines this based on how many CPU players have been specified. This results in the human values not being altered and pools the CPU values. It allows me to do it without quadrupling the already overlong main formulas with 'if' commands. However, it still has 1 shortfall: it still calibrates to one team or the other at the top of the file, hence the blue DATA line beneath that which sums up the fields (even percentages, but since I don't use percentages but shot attempts at misses for my formulas, no point in correcting it). You can copy and paste with paste special/text+numbers the data to the corresponding data team that has the game aggregates.

The thing is, I don't believe you can even get accurate CPU vs CPU games. In my trials, I accidentally screwed up the rating ranges and players were heavily distorted in the ratings, but it seemed to require a lot more game speed than sim conditions where AI isn't running into another AI, and then it just looks stupid. I just wanted to add it to round things off a bit, it may work, it may not. Plus, I was able to use it to verify all my equations using it by relating it to averages to reverse the equations, and yes, they seem to be 100% correct in ratio to offensive FGA's, defensive FGA's (steals/blocks) and offensive/defensive FG misses (rebounds).



There were problems with my game speed calculations. They were done to per minute values. What this meant is if you played 6 minute games, rather than try to find sim conditions for a 100% game, it'd try to make it a realistic 50%. At some point I had changed it without realizing it so overtime games wouldn't be 'decelerated'. I fixed it and then decided I wanted both after playing another overtime game. If acceleration is turned to on (1), it'll try to sim as real as possible 48 minute game stats in lower game time (and will slow down for overtimes, so it's recommended to switch off for OT games or simply discount them); if it's turned to off (0), no matter how long you play it will try to keep realistic to the game length. The medium range shooting option is another important one. By default, I put in a 3 point attempt corrector. While short range shots were adjusted based on field goals -3 pointers, mid-range was based on aggregate FG%. So what it meant is that you could shoot 3 pointers all game long. If you scored 30% of them, your long range shooting will not adjust much if at all. But your mid range shot will sky rocket, because it tries to match than FG% to real life FG%. Though unrealistic, hopefully it encourages players to shoot less from the outside. Now you can switch that to off and have it compute the mid range shots the best way stats can tell, regardless of how many 3 point shots you launch. The third option for deciding which team to calculate the free throw rating to is a space filler.