Elo Ratings in Competitive Melee

PEEF! said:

Lists lists we love lists.

This is Melee nobody really cares.

Click to expand...

This isn't a list.
And sorry, but this is the kind of attitude that has perpetually held back the Smash community from being really legit imo. Becoming more legit is the only way this sinking ship will stay afloat.

Nintendude1189 said:

The MBR, or some other committee, will "approve" tournaments to be rated. An approved tournament will use the official MBR ruleset and Tio, and send a completed Tio file to this committee, which will then go through the bracket and update the results (either manually or with a script). Once the system passes its "incubation period," seeding via rating will become another requirement to be approved. Ratings will be updated chronologically and before each weekend so that updated ratings can be used for those tournaments. There should be some kind of database where TOs can go in and quickly get the ratings of all the entrants.

Click to expand...

The program I'm working on can already parse a completed Tio file automatically and update Elos accordingly. Additionally it stores its data in plaintext so if you wanted an online database, just upload the text file and throw it in the program's directory.

I'm making a lot of progress on this, pretty much only the GUI is left to do, and that's coming along well enough. The biggest problem is making sure people use the same name/tag at each tournament. I could implement a system for aliases, but I think there's far too much variation in what TOs name their players in Tio for that to be viable.

Cobalt said:

I'm making a lot of progress on this, pretty much only the GUI is left to do, and that's coming along well enough. The biggest problem is making sure people use the same name/tag at each tournament. I could implement a system for aliases, but I think there's far too much variation in what TOs name their players in Tio for that to be viable.

Click to expand...

So does your program fetch real names or gamertags to update ratings? Also, how does it handle unranked players? I think that if real names are used there won't be an issue. People will just have to remember exactly what they used as their real name last time (like Mike instead of Michael). Perhaps if the system is unable to find a match for a player, it can give a "did you mean _____?" as a way of catching typos. There's probably a better way to do it. Does anyone know how the chess community manages this issue?

So it goes through the Tio file getting its list of players. At each player, it attempts to match Tio's "Nickname" field to a player already in the database, by their handle/gamertag/nickname. If it doesn't find a match, it attempts to match by the "Name" field to a corresponding real name field in my program. Unranked players are given a base Elo you can specify in the program, which currently defaults to 1000.

I don't think I can implement a "Did you mean?" feature right now; I don't really know how I would handle the heuristics for it. The solution I would suggest would be having the committee you mentioned earlier go through the Tio file first, updating names themselves to ensure that everything matches up. Additionally, impose a requirement on all "approved" tournaments that for players to be ranked, they must use their [real name/full desired tag], and TOs must put those in accurately. The committee would be there to catch typos, basically.

I fully support this.

As to Names/Tags, you may want to take both into account, as some may have the same name / tag.

What if you had players use their nicknames for the bracket, but instead of their real names in the "Name" thing each person can be given some sort of ID #. When TOs register people, they'll just type in their nickname and ask them for their ID #. New players will have to be given ID #s, but this way there won't be any overly specific errors (like the difference between Bones and Bones0 or John or Johnathan).

Cobalt said:

So it goes through the Tio file getting its list of players. At each player, it attempts to match Tio's "Nickname" field to a player already in the database, by their handle/gamertag/nickname. If it doesn't find a match, it attempts to match by the "Name" field to a corresponding real name field in my program. Unranked players are given a base Elo you can specify in the program, which currently defaults to 1000.

Click to expand...

I'd suggest searching for real names first, because I think the likelihood of having inaccurate or multiple entries of the same real name is more unlikely. I actually know of a few duplicate gamertags used by this community, and I think gamertags for amateurs are occasionally entered incorrectly because they sometimes have numbers and other oddities. In event of multiple of the same real name in the database, or if the system can't find a match, then go by gamertag.

edit: I think the ID# thing is a great idea.

Yeah, going by name first is actually smarter. Changing that will only take one line of code, too, so consider it done. As for ID numbers, it would ensure uniqueness if we could avoid giving duplicate IDs to players, which would be hard.

The "safest" way to avoid duplication is to just check both name and tag, but that relies on people giving the same information every tournament, and it's understandable many people wouldn't want their full names available in an online database held by a committee of Internet strangers. Anyway, all of these options are open and I could implement any of them pretty easily.

Cobalt said:

Yeah, going by name first is actually smarter. Changing that will only take one line of code, too, so consider it done. As for ID numbers, it would ensure uniqueness if we could avoid giving duplicate IDs to players, which would be hard.

The "safest" way to avoid duplication is to just check both name and tag, but that relies on people giving the same information every tournament, and it's understandable many people wouldn't want their full names available in an online database held by a committee of Internet strangers. Anyway, all of these options are open and I could implement any of them pretty easily.

Click to expand...

The only issue I can see with the ID# is I don't think Tio has a field for it. However, it's probably fine to use ID#s in the "real name" field. I think that strings of 7 random digits should be generated for all players. If someone is new and does not have an ID#, 0000000 can be entered. If the system encounters a 0000000, it will generate a random ID# and then loop through all existing IDs. If it encounters a duplicate (very unlikely - 1 in 10 million), it will generate another new one and try again. What's good about designating 0000000 for new entries is it is easy to identify typos. If a player is #1234567 and a TO accidentally puts 1234566, the system will know that it is a typo because if it was a new entry it would have been 0000000.

ID#s can also be used schematically as a way to categorize players easily. For example, the first digit of all North American players can be 1, for Europeans 2, South America 3, etc. Yes, players relocate, but it's transcontinental relocation would only affect a few players and it'd be interesting to know where a player originally came from anyway. Also while you are coding, it might be wise to take into account the possibility of a single entity having a rating for multiple games. I believe in Tio you can specify what game you are playing so this should be simple. This would allow Brawl to jump on ship if they like the work we are doing. It'd also be a big help to the SSB64 community.

edit:
I was just thinking a cool numbering scheme, at least for the US, would be 1XXYYZZZZ, where 1 means from North America, XX is a country code, YY is a state code, and ZZZZ is your player number. So, 123321234 would mean I'm from North America (1), United States (23, assigning numbers alphabetically), New York (32), and then I am number 1234. This is getting kind of ambitious but it would be really neat.

Who was I talking to at GSG2 who was trying to start a melee ranking/points thing? There was a special setup for it and everything.

I've been thinking about this ever since I joined the smash scene. I thought it was pretty wierd there wasn't a rating system.

I'm all for this idea, keep developing it and fleshing it out I think it will in fact provide incentive for player progress and also add something positive to competitive smash as a whole.

The ID # idea is great, and using it as a regional fingerprinting tool is also a wonderful idea. Also if that was done, we can quickly use it to generate region specific information/statistics. It's a great way to consolidate extra information into the system.

Nintendude1189 said:

This isn't a list.
And sorry, but this is the kind of attitude that has perpetually held back the Smash community from being really legit imo. Becoming more legit is the only way this sinking ship will stay afloat.

Click to expand...

The ship isn't sinking. It is kept aflot because we like getting together and drinking and playing this damn game with old friends.. Everyone wants to have a number next to their name including me, but that isn't what will save Melee.

I don't "unsupport" it, it's certainly cool. The problem is that alot of smashers won't care, and that will be harmful to a good rating system.

I am NOT for this idea if it will chase away noobs, some socal d00d refused to play in tourney because he wanted to hold onto his newly attained BOTTOM ranking and not risk losing


sounds fun otherwise

PEEF! said:

The ship isn't sinking. It is kept aflot because we like getting together and drinking and playing this damn game with old friends.. Everyone wants to have a number next to their name including me, but that isn't what will save Melee.

I don't "unsupport" it, it's certainly cool. The problem is that alot of smashers won't care, and that will be harmful to a good rating system.

Click to expand...

I don't think smashers not caring will be a big issue. In fact, I think the system will punish people who don't care. People who don't get rated at tournaments will get bottom seeds at large tournaments, hindering their ability to progress really far. If it's the fault of the TOs for being too lazy to get approved tournaments, then the local community should realize the harm this is causing the players and force the TO to step it up.

People are only lazy and apathetic if you let them be.

Nintendude1189 said:

People are only lazy and apathetic if you let them be.

Click to expand...

I can vouch for that one.

National ranking sounds alright. That's a solid step but I think the tournament scene needs something more that will help attract new players. The game itself is nearly flawless but it's aging (Nintendo doesn't want you playing melee, they don't make money that way) and it's definitely hard to break into the tourney scene for new players.

Just some updates, the trial version of my program is almost ready for release. The GUI is basically done, now I just have to break it every way I can and fix the problems. Optimistically I can release it for use/widespread testing tomorrow.

edit: I do still need a name though, haha.

why not get somebody to do something like rajam's brawl rankings list?

MK26 said:

why not get somebody to do something like rajam's brawl rankings list?

Click to expand...

Because what we are going for here is a ratings scheme and not a rankings scheme. The Brawl list is an attempt at absolute ranking. That's why it takes into account tournament size, prize money, etc. In order to get really high on that list you'll need to place highly in a bunch of tournaments. With Elo, you do not need to win tournaments or enter big events to become a highly rated player. You simply have to beat strong opponents.

This is in response to Brookman too. Getting rated and getting ranked are not the same thing. Ratings are appealing to new players. Rankings, not necessarily so.

Nintendude1189 said:

I don't think smashers not caring will be a big issue. In fact, I think the system will punish people who don't care. People who don't get rated at tournaments will get bottom seeds at large tournaments, hindering their ability to progress really far. If it's the fault of the TOs for being too lazy to get approved tournaments, then the local community should realize the harm this is causing the players and force the TO to step it up.

People are only lazy and apathetic if you let them be.

Click to expand...

This is the attitude that sucks. "IF YOU WANT TO HAVE FUN YOU GET BOTTOM SEED". This is so dumb. Mango is now going to be last seed and tryhards like...Rubyiris or something will be ranked number one in the world. (Nothing against Rubyiris.)

TOS BE LESS LAZY WHEN WORKING FOR FREE. PLAYERS PLAY LIKE THERE IS ACTUALLY REAL MONEY ON THE LINE. OR ELSE.

Nah. Get some of the midwest spirit bro, where the biggest win is a drink with the guys, and the tryhard matches are low-tier ironmans.

PEEF! said:

This is the attitude that sucks. "IF YOU WANT TO HAVE FUN YOU GET BOTTOM SEED". This is so dumb. Mango is now going to be last seed and tryhards like...Rubyiris or something will be ranked number one in the world. (Nothing against Rubyiris.)

TOS BE LESS LAZY WHEN WORKING FOR FREE. PLAYERS PLAY LIKE THERE IS ACTUALLY REAL MONEY ON THE LINE. OR ELSE.

Nah. Get some of the midwest spirit bro, where the biggest win is a drink with the guys, and the tryhard matches are low-tier ironmans.

Click to expand...

Right. This community is terrible when it comes to bias. Time to fix that. Sure, Mango will be bottom seed at a national if he's unranked. Then he'll whoop everyone and come out of the tournament with a much stronger rating. The system corrects itself rapidly.

This is a great idea and it should be implemented in every competitive scene. Are only tournament matches rated in this system?

Cobalt said:

Just some updates, the trial version of my program is almost ready for release. The GUI is basically done, now I just have to break it every way I can and fix the problems. Optimistically I can release it for use/widespread testing tomorrow.

edit: I do still need a name though, haha.

Click to expand...

name it after yourself, Cobalt: ELO Ratings Module

"Yo dawg we need to send those matches in, fire up Cobalt"

"Nice to meet you. Good luck. Btw are you THE Cobalt? The guy behind that revolutionary rating system?"

etc :D

So who's going to bite the bullet and create a database of all players?

CHAiN-ACE said:

So who's going to bite the bullet and create a database of all players?

Click to expand...

The program should be able to do that automatically, as people enter tournaments. A preliminary set of ratings / players can be obtained as long as there exists some Tio files from recent tournaments.

Will players even start as bottom seed, or somewhere in the middle, like in chess?

Although it is discouraging seeing your ranking drop, it's less distance for players who are already good at this game.

CloneHat said:

Will players even start as bottom seed, or somewhere in the middle, like in chess?

Although it is discouraging seeing your ranking drop, it's less distance for players who are already good at this game.

Click to expand...

Well as I said before I think we can use some recent tournaments from this past year to generate some preliminary ratings for established players. There would also be a period where tournaments are rated, but tournaments aren't seeded by rating, to allow the system to settle and correct itself a little bit. I think that unrated players should enter "somewhere in the middle" as you said, but I don't actually know what the standard is for initial rating. Is it 1000 or something else?

It's 1000 or 1500 I believe.

This is so good really, keep working and I totally can see this being used at all tournies.

Zoler's right, but it's 1500, just to clarify.

Yeah, chess uses 1500 base. Some video games implement the system for a competitive ladder and use a different base. Guild Wars uses base 1000, and was originally k=30 but I think is now k=5. League of Legends uses base 1200 and k~=35, from what I can tell.

http://www.teamliquid.net/ uses ELO for Starcraft: Broodwar and Starcraft 2

I don't know the base but I'd guess they use 1700 or something lol. No idea.

1500 would probably be the best

WoW arena used to start at 1500 then they changed to start at zero when they realized that meant people had to play more. And more play = more money

The GuildWars ladder is a little more complex in that different events get different k-factors (automatic tournaments are weighted more than regular matches, while "snowball-tournaments" are weighted less). The k-factor at the moment is 15 for regular matches, but has previously been 30 and then 5. Maybe it's worth a thought to change k-factors based on the event, but it probably should not be considered until we have a working system up and running.
The starting point is 1000, by the way. A guild is considered to be a "top" guild when they reach a rating of 1200.

What if we start at some ridiculous number (5000?) and the k-factor is the number of entrants in a given tournament?

Yeah, I used to play Guild Wars back with the original 30 and 5, when ATs where like 50 or something stupidly large? I don't remember that much. I was in a top 20 guild shortly before I quit lol, but that was a long time ago.

I'm in top 30 right now, we don't play really much but as far as ATs go, I don't know the exact number but above 70 would be the case, looking as how much our ratings change with a guild rated aprox. the same as us.
SnowballATs are only played at Christmas, but the K value there was like 2 or 3, we beat guild ranked number 7 and got +1 rating so...

InferiorityComplex said:

What if we start at some ridiculous number (5000?) and the k-factor is the number of entrants in a given tournament?

Click to expand...

I'd say starting at between 1200 and 1500 would be the best, the K-factor should be a fixed number or the number of entrants divided by 8, 9 or 10. That way, big Nationals count around 30-40, Locals from 3 to 10 and Regionals in between that range, which seems fairly balanced.

edit: Just as I made this post I realized that it doesn't make any sense to determine the K-Factor different at different Events, because in Melee, it's really only about who you play, not when or where.

Alright, here's a preliminary release of my program. Keep in mind that right now I don't yet endorse it to be usable for tournament Elo management, since I'm not confident enough it'll work. Consider this a beta. I'll post my own thread for this once I'm confident that it will work properly and as advertised for large tournaments. As far as I can tell, it does all work properly, but I want testing and feedback before I give it my seal of approval.

Reported bugs:

Zivilyn Bane said:

Ok I found a bug.

I am going through the Springfield tournament monthlies, starting with January. I added every singles entrant from January, February, and March. Then went through and started putting results in. Names that I had already entered have been disappearing from the list. Like, I put that LinkMO beat Jake. Then did the rest of the matches for round one. Then when I went to put that Jake lost again in R2, Jake's name had disappeared off the list.

Click to expand...

Not entirely sure what's up with this one. Once I get some data from this I should be able to fix it.

Zoler said:

ok, so I made a tio file with just a bracket, no pools, but it's not working. Did I miss something?

Nothing happens when I browse the tio file and press update.

Click to expand...

This problem occurs in version 0.0.2 if you load a Tio file and some match has a player 1 bye vs. a player 2 actual person. Try 0.0.2a.

What you need: The program was tested on the latest version of Java. It should be compatible with Java 5 (1.5) and later, but as of now I provide no guarantees it will work except on the latest update of Java 6, which is compatible with all major platforms (Windows, Mac, etc.)

Download: EloCalc version 0.0.2a

Install and run: Download the .zip file from the above link. Extract the .zip into its own folder (by default, Windows should unzip to a folder called EloCalc_[version_number]). There will be two files: EloCalc.jar and players.elo. Double-click EloCalc.jar to run the program. Do not edit players.elo; that file contains the data for players used by the program and any edits could corrupt it, rendering your data lost or unusable.

How to use:


Above is a picture of the program's interface. In the top left you can see the list of players, giving their tag and Elo rating. You can sort the list by clicking on either column header. The "Player" column will sort alphabetically by tags, and the "Elo" column will sort by Elo rating. The "Remove player" button removes the currently-selected player from the list.

Below that is how to add new players to the list. Fill in the player's name, tag/handle, and a base Elo rating if they have one. If you do not provide an Elo, the program will use the "Base value" you can see under the "Update by Tio File" section. Press the "Add player" button and the new player will be added to the list.

On the right side, in the section labeled "Update Matches" is a box labeled "K-Value." This is the k-value used by the Elo system for all matches the program simulates, and represents the maximum number of points a player can win or lose in any single match. Players rated close to each other will win or lose a similar number of points. However, if players are rated far apart, the higher-ranked player is at risk of losing far more points for losing, and gaining far fewer points for winning, and vice versa.

You can represent a single match in the "Single Match" section. From the drop-down boxes, select the players in the match, and choose which of them is the winner by clicking the appropriate radio button. Press "Update" to update the Elo ratings for these players as if the match had occurred. For example, if Dr. PeePee played SilentSpectre and won, select Dr. PeePee and SilentSpectre from the drop down lists, click Dr. PeePee's radio button, and press Update. Both players' Elos will change accordingly, and the changes will appear in the blank area above the Save button.

Below this section, you can provide the program a Tio file and have it update Elo ratings for every player involved in the tournament. Click the "Browse" button and select the appropriate Tio file, then hit "Update" below Browse. Any players not in the database will be added with a base Elo value determined by the box here, and then EloCalc will step through the tournament updating matches chronologically.

Press the Save button in the bottom-right to save the current data to the players.elo file mentioned earlier. The next time you open EloCalc, all of the players with their appropriate Elos will be automatically loaded, as well as the Base Elo and k-value you selected.

The About button in the top-right provides some information on the development of the program.

Known Issues:
Tio files with pools do not currently work. The fix should be pretty simple, but it will take a bit of time.
The drop-down boxes for updating single matches have odd visual behavior, such as their arrows disappearing and defaulting to the first value in the database after every change to the player list. Additionally, the boxes are ordered in terms of what order players were added.
The name sucks and I need a new one.

Usage Notes (IMPORTANT):
When using a Tio file, please ensure that the Name and Nickname fields of the Tio file exactly match the Name and Handle fields you used if you added any players with the Add player button of this program. Otherwise, the program may not be able to identify that two players are actually the same person.
Ideally, you could do nothing but add Tio files to use this program. Soon, when support for pools is added, this should be entirely doable. Using the single match updater is for people who may like to rate MMs (perhaps with a lower k-value than tournament matches) etc., or for people who do not have a Tio file for their tournament.
Do not use "odd" characters in player names. Particularly, avoid using commas for sure. Most other non-alphanumeric characters should work, but I only guarantee that alphanumeric characters, period, and space do.

Future Features:
Support for unique player ID numbers, as done in Tio, perhaps by the conventions mentioned in this thread.
Pools support, as already mentioned.
I'm also thinking of adding a match history list for each player, but that likely goes beyond the scope of the program as an Elo manager, and so it's unlikely that I'll add it.

If you find any bugs, let me know and I'll try to fix them. Say what you were doing, post a screenshot if you can, and I may ask for the contents of your players.elo file.

That's incredible, Cobalt. I can't thank you enough for putting all of this together. I'll try running the program as soon as I put a dent in my large pile of work.

so when can i play you in tournament and make your rank drop mike?

btw, i know this is on the melee forums, but will there be separate databases potentially for brawl and 64? I imagine there may be some programming things needed so that the scores for each game don't overlap each other for people that have interest in multiple games. that's a long term goal obviously.

The database for the program right now is just the players.elo file I mentioned, which is a specially-formatted text file. In fact, if you opened it up, you could probably figure out the format really easily. You could very quickly swap between a Melee, Brawl, and 64 database by maintaining three separate database files, one for each game, and then just renaming the one you want to work with players.elo. It's not a very convenient system for most people, but if we're going to have a managing "committee" of some kind, I don't think it would be much of an issue.

Cobalt said:

Alright, here's a preliminary release of my program. Keep in mind that right now I don't yet endorse it to be usable for tournament Elo management, since I'm not confident enough it'll work. Consider this a beta. I'll post my own thread for this once I'm confident that it will work properly and as advertised for large tournaments. As far as I can tell, it does all work properly, but I want testing and feedback before I give it my seal of approval.

What you need: The program was tested on the latest version of Java. It should be compatible with Java 5 (1.5) and later, but as of now I provide no guarantees it will work except on the latest update of Java 6, which is compatible with all major platforms (Windows, Mac, etc.)

Download: EloCalc version 0.0.2

Install and run: Download the .zip file from the above link. Extract the .zip into its own folder (by default, Windows should unzip to a folder called EloCalc_0.0.2). There will be two files: EloCalc.jar and players.elo. Double-click EloCalc.jar to run the program. Do not edit players.elo; that file contains the data for players used by the program and any edits could corrupt it, rendering your data lost or unusable.

How to use:

Above is a picture of the program's interface. In the top left you can see the list of players, giving their tag and Elo rating. You can sort the list by clicking on either column header. The "Player" column will sort alphabetically by tags, and the "Elo" column will sort by Elo rating. The "Remove player" button removes the currently-selected player from the list.

Below that is how to add new players to the list. Fill in the player's name, tag/handle, and a base Elo rating if they have one. If you do not provide an Elo, the program will use the "Base value" you can see under the "Update by Tio File" section. Press the "Add player" button and the new player will be added to the list.

On the right side, in the section labeled "Update Matches" is a box labeled "K-Value." This is the k-value used by the Elo system for all matches the program simulates, and represents the maximum number of points a player can win or lose in any single match. Players rated close to each other will win or lose a similar number of points. However, if players are rated far apart, the higher-ranked player is at risk of losing far more points for losing, and gaining far fewer points for winning, and vice versa.

You can represent a single match in the "Single Match" section. From the drop-down boxes, select the players in the match, and choose which of them is the winner by clicking the appropriate radio button. Press "Update" to update the Elo ratings for these players as if the match had occurred. For example, if Dr. PeePee played SilentSpectre and won, select Dr. PeePee and SilentSpectre from the drop down lists, click Dr. PeePee's radio button, and press Update. Both players' Elos will change accordingly, and the changes will appear in the blank area above the Save button.

Below this section, you can provide the program a Tio file and have it update Elo ratings for every player involved in the tournament. Click the "Browse" button and select the appropriate Tio file, then hit "Update" below Browse. Any players not in the database will be added with a base Elo value determined by the box here, and then EloCalc will step through the tournament updating matches chronologically.

Press the Save button in the bottom-right to save the current data to the players.elo file mentioned earlier. The next time you open EloCalc, all of the players with their appropriate Elos will be automatically loaded, as well as the Base Elo and k-value you selected.

The About button in the top-right provides some information on the development of the program.

Known Issues:
Tio files with pools do not currently work. The fix should be pretty simple, but it will take a bit of time.
The drop-down boxes for updating single matches have odd visual behavior, such as their arrows disappearing and defaulting to the first value in the database after every change to the player list. Additionally, the boxes are ordered in terms of what order players were added.
The name sucks and I need a new one.

Usage Notes (IMPORTANT):
When using a Tio file, please ensure that the Name and Nickname fields of the Tio file exactly match the Name and Handle fields you used if you added any players with the Add player button of this program. Otherwise, the program may not be able to identify that two players are actually the same person.
Ideally, you could do nothing but add Tio files to use this program. Soon, when support for pools is added, this should be entirely doable. Using the single match updater is for people who may like to rate MMs (perhaps with a lower k-value than tournament matches) etc., or for people who do not have a Tio file for their tournament.
Do not use "odd" characters in player names. Particularly, avoid using commas for sure. Most other non-alphanumeric characters should work, but I only guarantee that alphanumeric characters, period, and space do.

Future Features:
Support for unique player ID numbers, as done in Tio, perhaps by the conventions mentioned in this thread.
Pools support, as already mentioned.
I'm also thinking of adding a match history list for each player, but that likely goes beyond the scope of the program as an Elo manager, and so it's unlikely that I'll add it.

If you find any bugs, let me know and I'll try to fix them. Say what you were doing, post a screenshot if you can, and I may ask for the contents of your players.elo file.

Click to expand...

Ok I found a bug.

I am going through the Springfield tournament monthlies, starting with January. I added every singles entrant from January, February, and March. Then went through and started putting results in. Names that I had already entered have been disappearing from the list. Like, I put that LinkMO beat Jake. Then did the rest of the matches for round one. Then when I went to put that Jake lost again in R2, Jake's name had disappeared off the list.