Arcane Seal(ed)
Abstract
The Sloppy Lab recently conducted a survey on player format preference. In this survey, we presented players with a series of choices between pairs of popular format. We share the results of that survey here.
In addition, the lab ran an experiment comparing the relative competitiveness of playing fields in a Sealed Alliance event versus a Traditional Sealed event. The results of that experiment are detailed below.
Method
Player Survey
Our survey on format preference focused on premier level organized play events as defined by Ghost Galaxy. Premier events include Vault Tours, National Championships, World Championships, and events held at the yearly KeyForge Celebration.
Briefly, the formats in question are defined as follow:
- Sealed Alliance - Players open a pool of (typically) three decks. They construct an Alliance using the pods in their pool to play with in the event.
- Traditional Sealed - Player open a pool of (typically) three decks, and select one deck to play with in the event.
- Alliance - Players bring one Alliance deck with them to use in the event.
- Archon - Players bring one Archon deck with them to use in the event.
For each possible pair of formats, we asked players which they would prefer to see offered at premier events if they could only have one of the two.
Playing Field Experiment
At the lab, we wanted to answer the following question: which sealed variant - Alliance or Traditional - offered players a more even playing field?
To approach this question we needed a way to estimate the strength of decks - both Archon and Alliance - on a large scale. For this we turned to the SAS ratings provided by Decks of KeyForge (DoK). In addition to giving decks an overall score, DoK will report scores for each house (pod) in a deck. This allowed us to swap and recombine pods, estimating the strength of potential Alliances very efficiently.
Our process was as follow:
- Download SAS details for 5,000 semi-random Call of the Archons (CotA) decks from DoK.
- Shuffle the decks and split them into pools wth 3 decks each. This yielded 1,666 3-deck pools, leaving 2 unused decks that were discarded.
- From each pool, determine the "strongest" Traditional Sealed deck (i.e. single, unmodified deck), and the "strongest" possible valid Alliance.
- Compare the resulting fields for Traditional Sealed and Sealed Alliance.
Results
Survey Results
At the time of this writing, our survey received 145 responses.
Given the choice between Traditional Sealed and Sealed Alliance, players tended to prefer Traditional Sealed to Sealed Alliance with 62.1% of respondents in favor of Traditional Sealed.
Archon proved to be a slightly heavier favorite over Sealed Alliance, with nearly 70% of players opting to bring their own single deck.
An overwhelming majority - 85% - would rather see Sealed Alliance than Alliance as their premier event offering.
In what turned out to be the tightest contest between any two formats in our survey, Archon beat out Traditional sealed by a mere 7.6 percentage points.
Traditional Sealed was preferred to Alliance at a rate of roughly 4 to 1.
Archon has the honor of being in both our closest and most lopsided polls. Nine times as many players wished to see Archon as their format for premier events over Alliance when asked to pick between the two.
In addition to format preference, we wanted to get a little information about the players responding to our survey. Nearly 70% attended a premier level event in 2023.
We were also curious how long respondents had been playing the game. Roughly 70% claim to have started before the game turned 2 sets old.
Finally, we gave players a chance to lets us know on which continent their LGS can be found.
To that one person playing in Antarctica - stay sloppy <3.
Playing Field Experiment
The histograms below illustrate the frequency with which our considered formats produced decks in a given SAS range.
Our first chart shows all formats together, with Single Deck Sealed (blue) included for comparison. As expected, Alliance (green) generates the strongest decks on average with a mean value of 73.7, roughly 5 points higher than Traditional Sealed (orange).
More surprising, however, our numbers for Alliance also show a higher variance (and therefore standard deviation) when compared with Traditional Sealed. What's more, the difference in variance is statistically significant.
Within the parameters of this experiment, we would conclude that Traditional Sealed offers competitors a more even playing field than Sealed Alliance by a slim margin and using weaker decks.
That said, while (statistically) significant, the difference in competitiveness between these two sealed variants may not be meaningful in practice.
Aside: We want to extend a special thanks to Tobin Lopes from the WarCast Reforged for being our head statistician in this exercise. Our sloppy selves can't even spell "T-test". Thanks, Tobin!
Finally, we include charts isolating each of the formats depicted above.
Some Particulars
Why CotA for your experiment? - Our lab felt that SAS is most accurate for CotA, and that there are fewest complicating factors for estimating pod strength using this set. For example, SAS scores for Winds of Exchange decks are weighted toward the house of the deck's token, making it unusable for our experiment. Mass Mutation poses a similar difficulty with tracking where enhancement bonus icons are placed.
What matters most is that we are able to reasonably model the typical strength distribution between houses in a set. Running this experiment again using a set with significant outlier houses (for example, Brobnar in Worlds Collide is rated very poorly) may yield different results. However, we felt that while this may happen from time to time in practice, having a wide range in general house strength was not something the game designers would aim for.
Why did you say you used semi-random decks? - We didn't have a great way to get truly random decks from DoK in bulk. So, we sorted them by date from earliest to latest, and exported as many as DoK would allow (5k).
SAS scores for a house will shift as pods are swapped around! - This is true and our hope is that these effects would be minimal and largely average out across the field. To reiterate, we only needed to model relative strengths between houses to estimate differences in deck strength.
Author's Note
Three years after the World Championship that never was, we're seeing premier events up and running again for the game we love. I couldn't be happier about that. 2023 was a tremendous year for KeyForge, minor hiccups not withstanding. My hope for 2024 is that we can look back and say it was the start of meaningful growth - of rebuilding - at the retail level.
I didn't take our survey but I do agree with the results. Alliance is fun and has its audience, it has earned a place in the pantheon of KeyForge formats. However, for premier events, I do not believe the flavors of Alliance we have tasted so far should get top billing alongside Archon and ahead of Traditional Sealed.
The first job of a premier event circuit is to be a marketing campaign for the game. Whether or not you or I ever personally attend a Vault Tour, these are the events we're going to read about online. The decks that win these events are the ones we'll talk about with our playgroups. Our locals who do go to these events will come home hyped and excited to jam the same formats they got to play there. Let's showcase the game on its grandest stage in the same way it's being played at our local stores and kitchen tables.
Two other small things.
First - players, as you head to events this season and get to spend time with Ghost Galaxy employees, whatever your feelings on formats, be sure to talk to them. The folks at GG have been nothing but exceptional every time I've had a chance to interact with them. They're great people who love the game and will listen - grab the opportunity in person if you can.
Second - to the GG peeps who find their way here, my ask of you is this: be mindful not to mistake general enthusiasm for Sealed as an endorsement for Sealed Alliance specifically. I believe that many hunger for some sort of sealed, but would prefer that be traditional sealed. Likewise, be mindful not to confuse competitive concerns over Single Deck Sealed with Pick One Sealed.
Thanks for reading, folks. I'll see you in the lab - and I can't think of a better way to close my note on sealed than how Ghost Galaxy opened theirs(1):
Opening a new KeyForge deck and exploring the intricacies of its unique mix of cards is one of the most exciting parts of the game. It’s a moment of discovery that is at the heart of the KeyForge experience, and it was a critical part of designer Richard Garfield’s vision for the game.
Stay sloppy!