Jump to content

Research toward 32 team Season Stats extractor


Dusto

Recommended Posts

**Updated **  Attached file that now does a csv export if anyone wants to test it out  


 


Working from cxrom's original notes here: http://tecmobowl.org/topic/7850-removing-teams-from-the-menu/?p=66596


  I went through today noting the updated locations of the team stats.  I'm trying to see if I can code a prototype tool and maybe you more experienced guys can chime in or use some of this for the Madison tool.


 


Each team starts exactly one addition of D0 apart (or 202). The Seahawks and Cardinals are the ones that were saved in non standard spots. This tool so far is using the battery save file or .SAV.  These locations also hold true for the 2013 rom, although I haven't QA'd them yet individually to make certain


Here are the new locations:



Bills 07A0
Dolphins 0870
Pats 0940
Jet 0A10
Bengals 0AEO
Brown 0BB0
Ravens 0C80
Steelers 0D50
Colts 0E20
Texans 0EF0
Jags 0FC0
Titans 1090
Broncos 1160
Chiefs 1230
Raiders 1300
Chargers 13D0
Redskins 14A0
Giants 1570
Eagles 1640
Cowboys 1710
Bears 17E0
Lions 18B0
Packers 1980
Vikings 1A50
Bucs 1B20
Saints 1BF0
Falcons 1CC0
Panthers 1D90
49ers 1E60
Rams 1F30
Seahawks 0250
Cards 0320

statsExtractor3a.html

Edited by Dusto
Link to comment
Share on other sites

Right now the direction I'm going for stat collection is this:


1. Loop Through Teams


2. For each team parse:  2 Passers 9 stats each, 10 Position Players 12 stats each, 11 Defenders 3 stats each


3  Each of the above sets of players ends up in an Array collection


4. Push these lists into a Player Array List


 


So right now, to get Bills QB1's pass attempts


i'd log out   passersArray[0].passAtt


Hopefully by the end it will be structured something like player.passer.ejManual.stats.passAtt


 


I've attached a screen shot of sort of the rough organization of data i've colected.


 


QB , and the RB,WR,TE group are done for data collection.


Just need to write some functions up to do defense.  I'll add kickers, punters and other data once the stats I think most people focus on for leagues is done. Once I get to that point I'll post some source if that's cool.  Maybe get a little collaboration going or some feedback.

post-11854-0-27426000-1387065548_thumb.p

post-11854-0-29213800-1387065810_thumb.p

Edited by Dusto
Link to comment
Share on other sites

I've got the stats extraction in a good place.  Still need to do a few conversions on stats that are stored oddly like yards + yardsmultiplier*256 , etc.


My other main goal right now is coming up with a regex to extract player names from the rom.   I'm consistently grabbing the easy ones like johnDoe, but names like c.j.SPILLER  or names clumped together oddly like these guys


leeSMITHpericWOODgandyLEVITRE


kraigURBIKwcordyGLENNyerikPEARS


 


are proving tough for me as sort of a regex novice.


 


Posting the source here tonight for what I have so far if anyone wants to take it and run with it or make some suggestions. This is all first draft , no refactoring or clean up. No special features and using the console to output.   If you save it to a file as .html you should be able to run it in Chrome.  Right click and Inspect Element and view the console to see what it's putting out.  Still pretty novice at Javascript too so don't poke fun :)


 


edited to remove code for now


Edited by Dusto
Link to comment
Share on other sites

Alright.  I've got the regex done, at least as far as the initial testing. A special middle finger to names like b de la PUENTE and willTA.UFO.OU  for making me give them their own special rules :)



([a-z]+[A-Z]+|([a-z][.][a-z][.]) [A-Z]+|[a-z][.] ([A-Z]+)[.]([A-Z]+)|([a-z]+[A-Z]+[.][A-Z]+)|[a-z][.] [A-Z]+|[a-z]. [a-z]+ [a-z]+ [A-Z]+|[a-z].[a-z]+ [A-Z]+)
Link to comment
Share on other sites

End of Day Effort:


File 1 is  NES Cart


File 2 is .SAV file   


At the point where i raw stats and names in a giant associate array.


It doesn't output anything just yet.   All the info is still in the console:      In Google Chrome just right click, inspect element and then click on Console.  You'll see a big list of players that can be expanded.   I'm not going to put too much work into output format yet until I get the raw stats converted and hopefully get some feedback. I know everyone's at tournaments this week that usually chimes in on this stuff.  Anyway here goes:


firstPassStatsExtractor.html

Edited by Dusto
Link to comment
Share on other sites

Looks good man. If you get stuck on anything brookstonfowler who helped write stats code for dynastyphile, which is used for league stats,  could probably help along with a few others like tecmoturd, and avgtsbplayer. I have a few routines I wrote for extracting player names and stats as well. But sometimes its best to struggle through it on your own to learn. 


 


Its always good to have another alternative and another person who can program tools like this. 

Link to comment
Share on other sites

the simple method i used to split first name and last name  which works for names that aren't totally jacked up was this: 


 


 


This is a snippet of the java code I used.

 

 

      for (int i = start; i < next - 1 + teamflag; i++) {

         int compare = romData;

         char c = (char) romData;

         temp = Character.toString©;

 

         if (compare > 96) {

            firstName = firstName + temp;

         } else {

            lastName = lastName + temp;

         }

      }

 

Basically they key thing here is that all lower case letters should have a hex value > than 96  and all upper case letters as well as a period should have one <96. As most names are stored like


 


joeMONTANA or j.MONTANA


Link to comment
Share on other sites

Thanks guys.  Yeah one of the big problems with names I'm having with the updated carts is there are weird seperator values in between some and none between others.   Eric Wood was one example.  This is not exact but there's a string like  johnDOEpericWOOD


Since the names aren't like X bytes for each , I had to use regex to try and pull them.  But since the violating letter is of the same type(lower case a-z) as the valid letters I can't really think of a way to eliminate it without making it cart specific.   Right now this works for every 32 team cart with minor spelling errors.    So he's listed as pericWood.  If anyone has any ideas outside of just case by case error correction let me know.  I guess I could always release a 2013 & 2014 specific thing. Though at that point I might as well just story the player name array in the script rather than extract it.


 


One line of thought I haven't explored yet is that if the names are diplaying correctly on the NES there might be some sort of pointer in the player attributes section to the Individual's name start and end location somewhere.  I'm sure it's in the hex notes on one of the pages if it exists.


Edited by Dusto
Link to comment
Share on other sites

Just to make sure I understand the docs right.  Would the name pointer values be in this clump somewhere?
3010-3CD0: Player Attributes and Data     

 

I'm reading through the original rom at the moment for an example to try and match up to qbBills stats and name.

Link to comment
Share on other sites

Here is an example from the 28 team rom.


 


The pointer to the names starts at 0x48 & 0x49 in the rom where we have the two bytes CA 86. Now we reverse these to get 0x86CA. Since the Names are in the first bank we subtract 0x8000 from this to get 0x6CA. Since we have an 0x10 INES header at the beginning of every rom file we add this offset to get the final location in the file. We do this and get 0x6CA + 0x10= 0x6DA.


 


If we go down to 0x6DA we see a value of 00, or we can think of it as the seperation character between names, right before the value of 71. 71 is the q in qbBills. 


 


The start of the next name we would get from the next two bytes after 0x48&0x49 or   0x4A and 0x4B. Going through the conversion process again we get 0x6E2 which is the separation character before the f in frankREICH.


 


 


For the 32 team rom,  I believe the name pointers start at 0x52.   The NFC WEst name pointers start at 0x3EB0 with the first separator for the actual NFC WEST names starting at  0x3F9FA. 


Link to comment
Share on other sites

I've got a 32 team names extractor written.  I did way too much work converting strings and swapping I think.  When I get some more time to read I want to try to read the whole thing as a Unit8Array and see if I can do bit shifting instead.  Seems like it would be a lot less looping.


 


I'm running into an issue with the general formula for extracting NFC West names the correct way.    The formula seems to be something like , swap the bytes and add 0x30000.  But something is different than the original names.  With the same routine I'm getting so I'm figuring out both the length and the starting address incorrectly by a byte or 2 somehow.


  1. 840: "colinKAEP"
  2. 841"RNICKcol"
  3. 842"MCCOY!frank"
  4. 843"OREIbruceMILL"
  5. 844"R2kendallHUN

 


 


Anyway in case anyone's interested here is the javascript and html for the 28 team name piece I actually got working


nesOnlyNamesExtract.html

Edited by Dusto
Link to comment
Share on other sites

figured out my mistake. I forgot to offset by 0x10 for the NES header on the second list of names.  should have been swap + 0x30000 +0x10.  


 


Def wouldn't mind checking other ideas averagetbsplayer :)


Edited by Dusto
Link to comment
Share on other sites

Today's progress.  Working 28 and 32 Team Names Extractor with the checkbox the TecmoMadison guys used. 


My todo list:


-Write 28 team only stats extractor piece (should just be a shorter max length hopefully)


-Combine the newest Team names extractor  with the first pass stats extractor stuff


-Add output type buttons and templates: csv, json, etc;


-Code cleanup (make things into reusable code instead of 32 and 28 team code duplication)


nameExtractorv2.html

Edited by Dusto
Link to comment
Share on other sites

Also, for what it's worth, this ParseSaveState API call can handle parsing the ROM's team/player data and return it in JSON form for you. Obviously, it's an API call that requires some network usage, but it makes it so writing a JavaScript parser isn't a necessity. It's probably still worth have a JavaScript-only parser, though, in order to avoid that network usage. I can write a season data parser API function, too, on the TecmoMadison site and expose it. I'll have to first write up the XML schema, though.


Link to comment
Share on other sites

trying to figure out some easy conversions for the season stats now that I have them all collected.  Have the QBs done so far:


These numbers were derived from jstouts post here: http://tecmobowl.org/topic/7850-removing-teams-from-the-menu/#entry66596


 


attempts =  attempts + ((TDs%4)*256)        


completions = completions + ((Ints%4)*256)


passing TDs =  passTD/4


Interceptions = ints / 4


passYards = passYards + ((passModifier/8)*256)


rushAttempts = rushAttempts


rushingYards = rushYards + ((passModifier % 8 )*256)


 


For those who haven't done much in the way of operators in programing.  % is modulus.   It gives you the remainder when dividing 1 number into another.    4 modulus 2 is 0 for example. 4 modulus 3 is 1 etc.


Secondly  /  is division.    However the when dealing with integers, it sort of acts like the oposite of modulus.    10 / 2  is 5   10 /3  is 3.  Not 3.333 or 3 1/3 as you'd see on a calculator.


 


As a proof of concept so far lets do Peyton Manning. This is the data currently in each of the above categories


name: "peyton MANNING"

passAtt: 93    =  93 + (92%4)*256        Since 4 divides evenly into 92,  we end up with 93 + 0*256  or 93  completions

passComp: 64   64 + (8%4)*256        Again 4 divides evenly into 8 so  63+0*256   or  63  attempts

passTd: 92        92/4      giving us 23 Touchdowns

passInt: 8          8/4   giving us 2 interceptions

passYds: 104    104+(48/8 )*256    this gives us 104 + 6*256   or 1640 passing yards

rushAtt: 0          this is straight forward   0 rushing attempts

rushYds: 0         0 + (48 % 8 )*256    since 8 divides evenly into 48  we get  0 + 0*256  or 0 rushing yards

passMod: 48     this is just in her for the purpose of showing you where the modifier comes from

rushTd: 0         0/4  = 0  in integer division.

 

 

Checking the math against the cart (see screenshot)     And good stuff.     Also attached the version that collects all 32 team stats.  this doesn't have the conversions I just posted.  I need to do these for the other positions as well and then hopefully tomorrow or Saturday I'll be able to wrap things up.

post-11854-0-22518400-1387501577_thumb.p

statsExtractor2.html

Edited by Dusto
Link to comment
Share on other sites

Also, for what it's worth, this ParseSaveState API call can handle parsing the ROM's team/player data and return it in JSON form for you. Obviously, it's an API call that requires some network usage, but it makes it so writing a JavaScript parser isn't a necessity. It's probably still worth have a JavaScript-only parser, though, in order to avoid that network usage. I can write a season data parser API function, too, on the TecmoMadison site and expose it. I'll have to first write up the XML schema, though.

Thanks man.  I need to catch up with you soon.  Your tool actually was my inspiration.  And I saw you've had the 32 team season thing on your todo list for a  while.  Just figured I'd help out.    I think now that it's this far along, you can grab what you want from anything I've posted here.  I've mainly just been sticking to HTML and javascript so that anyone who felt like it could just load it up in Chrome or Firefox with no extra tools.     I'm hoping the way I wrote it with associative arrays will lend itself to JSON format.    If not I may end up going back and doing it differently.

Link to comment
Share on other sites

New today:  Stats are now converted to match the cart.   After selecting save state it exports to a csv.    If anyone likes this sort of thing feel free to test it out now and let me know if you see any glaring mistakes in the stats vs your team data.  Attaching this to the first post in the thread as well.


 


 


Edit, Quick Fix for rushing yards calculation. Used the wrong number for modifier


statsExtractor3.html

Edited by Dusto
Link to comment
Share on other sites

  • 2 weeks later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...