For the last few years, the Baltimore Orioles have been looked at as overachievers, in part because the preseason projection systems have, in the eyes on some Orioles fans, slighted the team. The one system that seems to draw the ire of Orioles fans the most is PECOTA. This season, PECOTA has projected the Orioles to win 78 games. This has upset many Orioles fans and it has caused lots of discussion throughout message boards and other social media outlets.
(You can discuss this on the BSL Board here.)
I have a theory that a big reason PECOTA underestimates the Orioles is because it does not properly measure defense. When talking about the Orioles, many of the sabermetric sites and analysts point to the pitching staff. The staff isn’t overwhelming by any means. They have been outperforming their peripherals. This isn’t a high strikeout team. They let up homers at a higher than average rate. The walk rate is good but nothing special. All in all, it’s a very average staff but it’s a staff that looks much better because of the Orioles defense. This past week, on Sports Tonight with Rob and Mike, Rob Neyer from FoxSports joined us. Neyer has a deserved reputation as a leading sabermetric analyst. We had a discussion with him about this and he said that there could be merit to it but that he felt more research needed to be done to see if we could find some kind of correlation over a larger sample size. He talked about BABIP and said that any system should factor that into their equation.
So, this got me thinking and I decided to try and do some kind of research to find out if there is any hint that my theory could be correct. I joined the Baseball Prospectus website today in hopes to find all of the PECOTA standings over the last several years and do some comparisons. Unfortunately, I wasn’t able to find anything on their site for PECOTA except for the 2015 projected standings. I am new to the site and its possible I am not looking at the right spot. So, I went to Google and tried to find out as much as I could. I was able to find some PECOTA standings from 2009-2014. In some years, I was able to get the whole list and in others, only a portion of the list. The sample size isn’t as large as I would have liked and not all of the lists were the final PECOTA projections but I worked with what I could find.
My baseline was a 10 win differential in either direction. There were plenty of teams that I could have gone with that would have been 7-9 wins but I just used 10 since it’s a nice big round number. I also looked at 4 stats to test this theory out. Those 4 stats were BABIP, UZR, UZR/150 and Defensive Efficiency. UZR and UZR/150 are stats used on FanGraphs and DE is a BP stat. With BABIP, UZR and UZR/150, I took the league averages and judged the teams based off of that. With DE, I merely used the team’s ranking throughout MLB. With that, teams ranks 1-14 were the top half of the league and 15-30 was the bottom half. I got the league averages of BABIP, UZR and UZR/150 from FanGraphs.com. League average BABIP was usually 293, give or take. UZR was around 2, give or take and UZR/150 was around 0, give or take over the 2009-2014 time periods.
Since 2009, there were 60 teams(that I found with the research I had to work with) that fit the profile of PECOTA being off by 10 or more games. Amongst those 60 teams, 29 of them were +10 or more and the other 31 were -10 or more.
My findings were not that surprising to me. Of the 29 teams that were in the +10 wins or more category, 69%(20/29) of them had a better than league average team BABIP, 59%(17/29) of them had a better than league average UZR and 72%(21/29) of them had a better than league average UZR/150. Also, 72%(21/29) also ranked in the top half of the league in DE. So, basically, if you played good defense, that was part of the reason you exceeded the expectations of PECOTA.
When looking at the 31 teams that were -10 wins or more, 74%(23/31) had a higher than league average BABIP, 65% had a higher than average UZR and 65% also had a higher than league average UZR/150. To go along with those stats, 87% of those teams had a DE in the bottom half of the league. This basically worked out the way you would expect in my opinion. If you play poor defense, you are more likely to under-perform based on the PECOTA projections.
Now, there were exceptions. For example, in 2014, the Rays were better than league average in all of these categories, yet they fell 12 wins shy of their PECOTA projections. All in all, I found 7 teams that across the board were either all pluses or all negatives in these categories and yet, the opposite of my theory ended up being true. That is 12%.
There are further things to consider here. Did I provide the proper stats? Is it fair to use UZR and UZR/150 to judge what Baseball Prospectus predicts wins off of? Is this enough of a sample size to go off of? Is this enough proof to show that defense is what is effecting these numbers? Could I have found similar numbers had I looked at 4 random stats, whether it be sabermetric stats or old school stats? I am not sure what the answer to these questions are. I went into this with a theory and I used one of the stats that Mr. Neyer said we should look at and I used 3 other stats that the sabermetric community uses to measure defense. I feel I was fair here and I feel I found enough of sample size to at least be able to use as the beginning of the defense of my theory.
Lastly, this isn’t an article where I am trying to slam PECOTA. I feel PECOTA is an excellent tool and one that does a good job when evaluating individual performance and, for a lot of teams, I feel it is good at projecting team wins(I found many where PECOTA was very accurate). However, what got all of this started was how they seem to be slighting the Orioles this year. I feel this system doesn’t do the Orioles justice because of how these guys fit together as a team. They may be right for the individual players but as a team, I feel they just can’t properly measure defense and, to a lesser degree, how Showalter and Duquette handle the roster.
We will see what happens in 2015. I think the best part of this roster is the defense. If the main guys can stay healthy, I think this team has a chance to have 4 or 5 legit Gold Glove candidates on the roster and this team’s defense will continue to make the pitching staff look much better than it is and that alone will effect what the projection systems seem to think
NOTE: After I was done writing this, I received an email from BP providing me with the PECOTA standings from the beginning. I may go back and look more into things and try to get a larger sample size but the new baby isn’t going to allow all of that extra time right now.
NOTE 2: We also recommend listening to our BSL Colleague (and BP contributor) Jeff Long’s appearance on BP’s Effectively Wild podcast where there was further discussion of the O’s PECOTA projection.
Rob has interviewed guests from outlets such as ESPN, Sports Illustrated, NBC Sports, CBS Sports, FOX Sports, Baseball Prospectus, Athlon, Sporting News, MLB Network, Brooks Baseball, Baseball Info Solutions, The Boston Globe, The Washington Post, Sports on Earth, Grantland, NFL Network, FanGraphs, Football Outsiders, ProFootballFocus, etc. etc. The Baltimore native lives in Perry Hall with his Wife Lindsay, and two young sons. He has appeared as a guest on 105.7 The Fan, Q1370, and WNST 1570. Co-Host of The Warehouse: https://anchor.fm/the-warehouse Co-Host of Sports Tonight: https://anchor.fm/sports-tonight