🪄 spells ✨

spells is a python package that tutors up blazing-fast and extensible analysis of the public data sets provided by 17Lands and exiles the annoying and slow parts of your workflow. Spells exposes one first-class function, summon, which summons a Polars DataFrame to the battlefield.

$ spells add DSK
  🪄 spells ✨ [data home]=/home/joel/.local/share/spells/
🪄 add ✨ Downloading draft dataset from 17Lands.com 100% [......................................................................] 250466473 / 250466473   🪄 add ✨ Unzipping and transforming to parquet (this might take a few minutes)...   🪄 add ✨ Wrote file /home/joel/.local/share/spells/external/DSK/DSKPremierDraftdraft.parquet   🪄 clean ✨ No local cache found for set DSK   🪄 add ✨ Fetching card data from mtgjson.com and writing card file   🪄 add ✨ Wrote file /home/joel/.local/share/spells/external/DSK/DSK_card.parquet   🪄 add ✨ Calculating set context   🪄 add ✨ Wrote file /home/joel/.local/share/spells/external/DSK/DSKPremierDraftcontext.parquet   🪄 add ✨ Downloading game dataset from 17Lands.com 100% [........................................................................] 77145600 / 77145600   🪄 add ✨ Unzipping and transforming to parquet (this might take a few minutes)...   🪄 add ✨ Wrote file /home/joel/.local/share/spells/external/DSK/DSKPremierDraftgame.parquet   🪄 clean ✨ Removed 1 files from local cache for set DSK   🪄 clean ✨ Removed local cache dir /home/joel/.local/share/spells/cache/DSK

In [1]: from spells import summon
In [2]: %time summon('DSK') CPU times: user 20.3 s, sys: 7.9 s, total: 28.2 s Wall time: 7.55 s Out[2]: shape: (286, 14) ┌────────────────────────────┬───────┬──────────┬──────────┬───┬────────┬──────────┬─────────┬──────────┐ │ name                       ┆ color ┆ rarity   ┆ numseen ┆ … ┆ numoh ┆ ohwr    ┆ numgih ┆ gih_wr   │ │ ---                        ┆ ---   ┆ ---      ┆ ---      ┆   ┆ ---    ┆ ---      ┆ ---     ┆ ---      │ │ str                        ┆ str   ┆ str      ┆ i64      ┆   ┆ i64    ┆ f64      ┆ i64     ┆ f64      │ ╞════════════════════════════╪═══════╪══════════╪══════════╪═══╪════════╪══════════╪═════════╪══════════╡ │ Abandoned Campground       ┆       ┆ common   ┆ 178750   ┆ … ┆ 21350  ┆ 0.559672 ┆ 49376   ┆ 0.547594 │ │ Abhorrent Oculus           ┆ U     ┆ mythic   ┆ 6676     ┆ … ┆ 4255   ┆ 0.564042 ┆ 11287   ┆ 0.593337 │ │ Acrobatic Cheerleader      ┆ W     ┆ common   ┆ 308475   ┆ … ┆ 34177  ┆ 0.541709 ┆ 74443   ┆ 0.532152 │ │ Altanak, the Thrice-Called ┆ G     ┆ uncommon ┆ 76981    ┆ … ┆ 13393  ┆ 0.513925 ┆ 34525   ┆ 0.539175 │ │ Anthropede                 ┆ G     ┆ common   ┆ 365380   ┆ … ┆ 8075   ┆ 0.479876 ┆ 20189   ┆ 0.502353 │ │ …                          ┆ …     ┆ …        ┆ …        ┆ … ┆ …      ┆ …        ┆ …       ┆ …        │ │ Wildfire Wickerfolk        ┆ GR    ┆ uncommon ┆ 98040    ┆ … ┆ 18654  ┆ 0.592366 ┆ 42251   ┆ 0.588696 │ │ Winter's Intervention      ┆ B     ┆ common   ┆ 318565   ┆ … ┆ 27552  ┆ 0.537638 ┆ 66921   ┆ 0.548453 │ │ Winter, Misanthropic Guide ┆ BGR   ┆ rare     ┆ 52091    ┆ … ┆ 1261   ┆ 0.462331 ┆ 3183    ┆ 0.479422 │ │ Withering Torment          ┆ B     ┆ uncommon ┆ 76237    ┆ … ┆ 15901  ┆ 0.511729 ┆ 39323   ┆ 0.542024 │ │ Zimone, All-Questioning    ┆ GU    ┆ rare     ┆ 20450    ┆ … ┆ 9510   ┆ 0.654574 ┆ 23576   ┆ 0.616686 │ └────────────────────────────┴───────┴──────────┴──────────┴───┴────────┴──────────┴─────────┴──────────┘
In [3]: %time spells.summon('DSK') CPU times: user 16.3 ms, sys: 66.2 ms, total: 82.5 ms Wall time: 80.8 ms

Coverting to pandas DataFrame is as simple as invoking the chained call summon(...).to_pandas().

Spells is not affiliated with 17Lands. Please review the Usage Guidelines for 17lands data before using Spells, and consider supporting their patreon. Spells is free and open-source; please consider contributing and feel free to make use of the source code under the terms of the MIT license.

spells

• Uses Polars for high-performance, multi-threaded aggregations of large datasets
• Uses Polars to power an expressive query language for specifying custom extensions
• Analyzes larger-than-memory datasets using Polars streaming mode
• Converts csv datasets to parquet for 10x faster calculations and 20x smaller file sizes
• Supports calculating the standard aggregations and measures out of the box with no arguments (ALSA, GIH WR, etc)
• Caches aggregate DataFrames in the local file system automatically for instantaneous reproduction of previous analysis
• Manages grouping and filtering by built-in and custom columns at the row level
• Provides 124 explicitly specified, enumerated, documented column definitions
• Can aggregate over multiple sets at once, even all of them, if you want.
• Supports "Deck Color Data" aggregations with built-in column definitions.
• Lets you feed card metrics back in to column definitions to support scientific workflows like MLE
• Provides a CLI tool spells [add|refresh|clean|remove|info] [SET] to download and manage external files
• Downloads and manages public datasets from 17Lands
• Retrieves and models booster configuration and card data from MTGJSON
• Is fully typed, linted, and statically analyzed for support of advanced IDE features
• Provides optional enums for all base columns and built-in extensions, as well as for custom extension parameters
• Uses Polars expressions to support second-stage aggregations and beyond like game-weighted z-scores with one call to summon
• Works on MacOS, Linux, and Windows
• Provides example notebooks to kickstart your exploration

summon

summon takes five optional arguments, allowing a fully declarative specification of your desired analysis. Basic functionality not provided by this api can often be managed by simple chained calls using the polars API, e.g. sorting and post-agg filtering. - columns specifies the desired output columns

>>> spells.summon('DSK', columns=["numgp", "pctgp", "gpwr", "gpwr_z"])     shape: (286, 5)     ┌────────────────────────────┬────────┬──────────┬──────────┬───────────┐     │ name                       ┆ numgp ┆ pctgp   ┆ gpwr    ┆ gpwr_z   │     │ ---                        ┆ ---    ┆ ---      ┆ ---      ┆ ---       │     │ str                        ┆ i64    ┆ f64      ┆ f64      ┆ f64       │     ╞════════════════════════════╪════════╪══════════╪══════════╪═══════════╡     │ Abandoned Campground       ┆ 114632 ┆ 0.643404 ┆ 0.546444 ┆ 0.12494   │     │ Abhorrent Oculus           ┆ 26046  ┆ 0.908476 ┆ 0.561852 ┆ 1.245212  │     │ Acrobatic Cheerleader      ┆ 188674 ┆ 0.705265 ┆ 0.541474 ┆ -0.236464 │     │ Altanak, the Thrice-Called ┆ 87285  ┆ 0.798662 ┆ 0.538695 ┆ -0.438489 │     │ Anthropede                 ┆ 50634  ┆ 0.214676 ┆ 0.515444 ┆ -2.129016 │     │ …                          ┆ …      ┆ …        ┆ …        ┆ …         │     │ Wildfire Wickerfolk        ┆ 106557 ┆ 0.725806 ┆ 0.565331 ┆ 1.498173  │     │ Winter's Intervention      ┆ 157534 ┆ 0.616868 ┆ 0.531758 ┆ -0.942854 │     │ Winter, Misanthropic Guide ┆ 7794   ┆ 0.197207 ┆ 0.479985 ┆ -4.70721  │     │ Withering Torment          ┆ 92468  ┆ 0.875387 ┆ 0.525858 ┆ -1.371877 │     │ Zimone, All-Questioning    ┆ 54687  ┆ 0.844378 ┆ 0.560974 ┆ 1.181387  │     └────────────────────────────┴────────┴──────────┴──────────┴───────────┘

- group_by specifies the grouping by one or more columns. By default, group by card names, but optionally group by any of a large set of fundamental and derived columns, including card attributes and your own custom extension.

>>> summon('BLB', columns=["numwon", "numgames", "gamewr", "deckmanavalueavg"], groupby=["maincolors"], filterspec={"numcolors": 2})     shape: (10, 5)     ┌─────────────┬─────────┬───────────┬──────────┬─────────────────────┐     │ maincolors ┆ numwon ┆ numgames ┆ gamewr  ┆ deckmanavalue_avg │     │ ---         ┆ ---     ┆ ---       ┆ ---      ┆ ---                 │     │ str         ┆ u32     ┆ u32       ┆ f64      ┆ f64                 │     ╞═════════════╪═════════╪═══════════╪══════════╪═════════════════════╡     │ BG          ┆ 85022   ┆ 152863    ┆ 0.556197 ┆ 2.862305            │     │ BR          ┆ 45900   ┆ 81966     ┆ 0.559988 ┆ 2.76198             │     │ RG          ┆ 34641   ┆ 64428     ┆ 0.53767  ┆ 2.852182            │     │ UB          ┆ 30922   ┆ 57698     ┆ 0.535928 ┆ 3.10409             │     │ UG          ┆ 59879   ┆ 109145    ┆ 0.548619 ┆ 2.861026            │     │ UR          ┆ 19638   ┆ 38679     ┆ 0.507717 ┆ 2.908215            │     │ WB          ┆ 59480   ┆ 107443    ┆ 0.553596 ┆ 2.9217              │     │ WG          ┆ 76134   ┆ 136832    ┆ 0.556405 ┆ 2.721064            │     │ WR          ┆ 49712   ┆ 91224     ┆ 0.544944 ┆ 2.5222              │     │ WU          ┆ 16483   ┆ 31450     ┆ 0.524102 ┆ 2.930967            │     └─────────────┴─────────┴───────────┴──────────┴─────────────────────┘

- filter_spec specifies a row-level filter for the dataset, using an intuitive custom query formulation

>>> from spells import ColName     >>> spells.summon('BLB', columns=[ColName.GAMEWR], groupby=[ColName.PLAYERCOHORT], filterspec={'lhs': ColName.NUM_MULLIGANS, 'op': '>', 'rhs': 0})     shape: (4, 2)     ┌───────────────┬──────────┐     │ playercohort ┆ gamewr  │     │ ---           ┆ ---      │     │ str           ┆ f64      │     ╞═══════════════╪══════════╡     │ Bottom        ┆ 0.33233  │     │ Middle        ┆ 0.405346 │     │ Other         ┆ 0.406151 │     │ Top           ┆ 0.475763 │     └───────────────┴──────────┘

- extensions allows for the specification of arbitrarily complex derived columns and aggregations, including custom columns built on top of custom columns.

>>> import polars as pl     >>> from spells import ColSpec, ColType      >>> ext = {     ...     'deq_base': ColSpec(     ...         col_type=ColType.AGG,     ...         expr=(pl.col('gpwrexcess') + 0.03  (1 - pl.col('ata')/14).pow(2))  pl.col('pct_gp'),     ...     )     ... }     >>> spells.summon('DSK', columns=['deqbase'], groupby=["name", "color", "rarity"], filterspec={'playercohort': 'Top'}, extensions=ext)     ...     .filter(pl.col('deqbase').isfinite())     ...     .filter(pl.col('rarity').is_in(['common', 'uncommon'])     ...     .sort('deq_base', descending=True)     ...     .head(10)     shape: (10, 4)     ┌──────────────────────────┬──────────┬──────────┬───────┐     │ name                     ┆ deq_base ┆ rarity   ┆ color │     │ ---                      ┆ ---      ┆ ---      ┆ ---   │     │ str                      ┆ f64      ┆ str      ┆ str   │     ╞══════════════════════════╪══════════╪══════════╪═══════╡     │ Sheltered by Ghosts      ┆ 0.03945  ┆ uncommon ┆ W     │     │ Optimistic Scavenger     ┆ 0.036131 ┆ uncommon ┆ W     │     │ Midnight Mayhem          ┆ 0.034278 ┆ uncommon ┆ RW    │     │ Splitskin Doll           ┆ 0.03423  ┆ uncommon ┆ W     │     │ Fear of Isolation        ┆ 0.033901 ┆ uncommon ┆ U     │     │ Floodpits Drowner        ┆ 0.033198 ┆ uncommon ┆ U     │     │ Gremlin Tamer            ┆ 0.032048 ┆ uncommon ┆ UW    │     │ Arabella, Abandoned Doll ┆ 0.032008 ┆ uncommon ┆ RW    │     │ Unnerving Grasp          ┆ 0.030278 ┆ uncommon ┆ U     │     │ Oblivious Bookworm       ┆ 0.028605 ┆ uncommon ┆ GU    │     └──────────────────────────┴──────────┴──────────┴───────┘

- card_context takes a name-indexed DataFrame or name-keyed dict and allows the construction of column definitions based on the results.

>>> deq = spells.summon('DSK', columns=['deqbase'], filterspec={'player_cohort': 'Top'}, extensions=[ext])     >>> ext = {      ...     'pickeddeqbase': ColSpec(     ...         coltype=ColType.PICKSUM,     ...         expr=lambda name, cardcontext: cardcontext[name]['deq_base']     ...     ),     ...     'pickeddeqbase_avg', ColSpec(     ...         col_type=ColType.AGG,     ...         expr=pl.col('pickeddeqbase') / pl.col('num_taken')     ...     ),     ... }     >>> spells.summon('DSK', columns=['pickeddeqbaseavg'], groupby=['playercohort'], extensions=ext, cardcontext=deq)     shape: (4, 2)     ┌───────────────┬─────────────────────┐     │ playercohort ┆ pickeddeqbaseavg │     │ ---           ┆ ---                 │     │ str           ┆ f64                 │     ╞═══════════════╪═════════════════════╡     │ Bottom        ┆ 0.004826            │     │ Middle        ┆ 0.00532             │     │ Other         ┆ 0.004895            │     │ Top           ┆ 0.005659            │     └───────────────┴─────────────────────┘

Installation

Spells is available on PyPI as spells-mtg, and can be installed using pip or any package manager:

pip install spells-mtg

Spells is still in development and could benefit from many new features and improvements. As such, you might rather clone this repository and install locally. It is set up to use pdm, but it's just a regular old python package and you can install with your normal workflow.

If you are new to Python, I recommend using a package manager like poetry, pdm or uv to create a virtual environment and manage your project.

Once Spells is installed, check out the notebooks under examples for ideas on getting started.

Why did you make this? Who is it for?

Earlier this year I developed a card quality metric called DEq, short for "Estimated Draft Equity", which is designed to estimate the average value of selecting a card in draft relative to a comparable baseline, in order to improve on commonly-used metrics like GIH WR, which has a number of major and minor problems when interpreted as a card quality metric. DEq depends on the daily drops from 17Lands.com and won't be replaced by this static kind of analysis.

While the modeling underpinning DEq remains sound, the estimation of the value depends on several parameters which should be inferred statistically, particularly the value of a pick and the pool bias estimate, and that process has been, let's say, somewhat less sound. In order to provide more scientific estimates of the parameters, and to continue on with deeper research, I felt the need to build a python library to enable quicker iteration and concise, portable declarations of analysis.

That need compounded with a feeling that the barrier to entry to working with these datasets is too high and that a tool like this would benefit the community. So that's what this is. It is for data-curious beginning programmers and professional developers and scientists. I hope you find it useful.

If you're interested in the fruits of my DEq research, or in checking my work, keep an eye on my deq repository.

Performance

Spells provides several features to optimize performance.

Parquet Transformation

The most significant optimization used by Spells is the simplest: the csv files are scanned and streamed to Parquet files by Polars. This allows 10x faster compute times with 20x less storage space and lower memory usage compared to csv. Yes, it's twenty times smaller and ten times faster!

Query Optimization

Spells is built on top of Polars, a modern, well-supported DataFrame engine written for performance in Rust that enables declarative query plans and lazy evaluation, allowing for automatic performance optimization in the execution of the query plan. Spells selects only the necessary columns for your analysis, recursively traversing the dependency tree.

Local Caching

Spells caches the results of expensive aggregations in the local file system as parquet files, which by default are found under the data/local path from the execution directory, which can be configured using the environment variable SPELLSPROJECTDIR. Query plans which request the same set of first-stage aggregations (sums over base rows) will attempt to locate the aggregate data in the cache before calculating. This guarantees that a repeated call to summon returns instantaneously.

Memory Usage

One of my goals in creating Spells was to eliminate issues with memory pressure by exclusively using the map-reduce paradigm and a technology that supports partitioned/streaming aggregation of larget-than-memory datasets. By default, Polars loads the entire dataset in memory, but the API exposes a parameter streaming which I have exposed as use_streaming and defaulted to True in Spells. Further testing is needed to determine the performance impacts, so you could try turning it off if you have expansive virtual memory. My 16 GB MacBook Air is fine using 60 GB of memory, but my 32 GB homelab is not.

When refreshing a given set's data files from 17Lands using the provided cli, the cache for that set is automatically cleared. The spells CLI gives additional tools for managing the local and external caches.

Documentation

In order to give a valid specification for more complex queries, it's important to understand a bit about what Spells is doing under the hood.

Basic Concepts

Let's briefly review the structure of the underlying data. Spells supports aggregations on two of the three large data sets provided by 17Lands, which are identified as "views" within Spells. First there is draft, which is the source for information about draft picks. The row model is single draft picks with pack and pool context. Unlike game, there are two different paradigms for aggregating over card names.

First, one can group by the value of the "pick" column and sum numerical column values. This is how ATA is calculated. In Spells, we tag columns to be summed in this way as picksum columns. For example, "takenat" is equivalent to "picknumber", but whereas the latter is available for grouping, "takenat" is summed over groups.

Second, certain columns are pivoted horizontally within the raw data and suffixed with card names, for example "packcardSavor". In Spells we tag such columns as name_sum, and group by non-name columns and sum before unpivoting. The columns are identified by their prefix only and Spells handles the mapping.

A standard way to aggregate information in non-namesum columns over names is to multiply that column over the pivoted column. For example, to calculate the namesum column "lastseen", used in ALSA, we multiply "packcard" by a modified version of "pick_number".

In the game file, the row model represents games, and primarily uses namesum aggregations for the familiar columns, such as "numgih", from which win rates are derived. For groupings that do not use card names or card attributes (to recreate the "deck color data" page, for example), one can also specify game_sum columns which aggregate simply over rows.

Aggregate View

Once aggregation columns, filters and groupings are determined at the row level for each of the required base views, Spells asks Polars to sum over groups and unpivot as needed to produce the "base aggregate view", which fixes the row model (pre-card attributes) to the provided base groupings. This base aggregate view is cached by default to the local file system, keyed by the manifest, which is a function of the specification provided by the user.

Next, card attributes are calculated and joined to the base aggregate view by name, and an additional grouping is performed if requested by the user to produce the aggregate view.

A final extension and selection stage is applied to the aggregate view, which is where weighted averages like GIH WR are calculated. Polars expression language enables aggregations to be represented as expressions and broadcast back to the row level, enabling Spells to support arbitrary chains of aggregation and extension at the aggregate view level. For example, one could calculate the mean of a metric over groups by archetype, regress a metric by a function of that mean, then calculate the mean of that regressed metric, all expressed declaratively as column expressions and simply specified by name in the summon api call.

So that's it, that's what Spells does from a high level. summon will hand off a Polars DataFrame which can be cast to pandas, sorted, filtered, used to be generate plots or whatever you like. If a task can be handled as easily via a chained call or outside library, it should stay that way, but if you have a request for features specific to the structure of limited data that could be handled in a general way, please reach out! In particular I am interested in scientific workflows like maximum likelihood estimation, but I haven't yet considered how to build it into Spells.

CLI

Spells includes a command-line interface spells to manage your external data files and local cache. Spells will download files to an appropriate file location on your system, typically ~/.local/share/spells on Unix-like platforms and C:\Users\{Username}\AppData\Local\Spells on Windows, or to a location specified by the environment variable SPELLSDATAHOME. To use spells, make sure Spells is installed in your environment using pip or a package manager, and type spells help into your shell, or dive in with spells add DSK or your favorite set. If Spells is installed globally using pipx, any local version of Spells will be able to read the managed files.

API

Summon

from spells import summon
summon(     set_code: list[str] | str,     columns: list[str] | None = None,     group_by: list[str] | None = None,     filter_spec: dict | None = None,     extensions: dict[str, ColSpec] | None = None,     card_context: pl.DataFrame | dict[str, dict[str, Any] | None = None,     set_context: pl.DataFrame | dict[str, Any] | None = None,     read_cache: bool = True,     write_cache: bool = True,     use_streaming: bool = True,     logtoconsole: int = logging.ERROR, ) -> polars.DataFrame

parameters

• set_code: a set code or list of set codes among those that you have added using spells add.

You can use "expansion" as a group_by to separate results from multiple sets, or you can aggregate them together.

• columns: a list of string or ColName values to select as non-grouped columns. Valid ColTypes are PICKSUM, NAMESUM, GAME_SUM, and AGG. Min/Max/Unique

aggregations of non-numeric (or numeric) data types are not supported. If None, use a set of columns modeled on the commonly used values on 17Lands.com/card_data.

• groupby: a list of string or ColName values to display as grouped columns. Valid ColTypes are GROUPBY and CARD_ATTR. By default, group by "name" (card name). For contextual card attrs, include

in group_by, even when grouping by name.

• filterspec: a dictionary specifying a filter, using a small number of paradigms. Columns used must be in each base view ("draft" and "game") that the columns and groupby columns depend on, so

AGG and CARDATTR columns are not valid. Functions of card attributes in the base views can be filtered on using cardcontext, see the documentation for expr for details. NAME_SUM columns are also not supported. Derived columns are supported. No filter is applied by default. Yes, I should rewrite it to use the mongo query language. The specification is best understood with examples:

- {'playercohort': 'Top'} "playercohort" value equals "Top". - {'lhs': 'playercohort', 'op': 'in', 'rhs': ['Top', 'Middle']} "playercohort" value is either "Top" or "Middle". Supported values for op are <, <=, >, >=, !=, =, in and nin. - {'$and': [{'lhs': 'draft_date', 'op': '>', 'rhs': datetime.date(2024, 10, 7)}, {'rank': 'Mythic'}]} Drafts after October 7 by Mythic-ranked players. Supported values for query construction keys are $and, $or, and $not.

• extensions: a dict of spells.columns.ColSpec objects, keyed by name, which are appended to the definitions built-in columns described below.

• cardcontext: Typically a Polars DataFrame containing a "name" column with one row for each card name in the set, such that any usages of cardcontext[name][key] in column specs reference the column key. Typically this will be the output of a call to summon requesting cards metrics like GP_WR. Can also be a dictionary having the necessary form for the same access pattern.

• set_context: Typically, a dict of abitrary values to use in column definitions, for example, you could provide the quick draft release date and have a column that depended on that. You can also provide a one-row dataframe and access the column values.

• readcache/writecache: Use the local file system to cache and retrieve aggregations to minimize expensive reads of the large datasets. You shouldn't need to touch these arguments unless you are debugging.

• 'logtoconsole': Set to logging.INFO to see useful messages on the progress of your aggregation, or logging.WARNING to see warning messages about potentially invalid column definitions.

Enums

from spells import ColName, ColType

Recommended to import ColName for any usage of summon, and to import ColType when defining custom extensions.

ColSpec

from spells import ColSpec
ColSpec(     col_type: ColType,     expr: pl.Expr | Callable[..., pl.Expr] | None = None,     version: str | None = None )

Used to define extensions in summon

parameters

• coltype: one of the ColType enum values, FILTERONLY, GROUPBY, PICKSUM, NAMESUM, GAMESUM, CARDATTR, and AGG. See documentation for summon for usage. All columns except CARDATTR

and AGG must be derivable at the individual row level on one or both base views. CARD_ATTR must be derivable at the individual row level from the card file. AGG can depend on any column present after summing over groups, and can include polars Expression aggregations. Arbitrarily long chains of aggregate dependencies are supported.

• expr: A polars expression or function returning a polars expression giving the derivation of the column value at the first level where it is defined. The name is inferred from the dictionary key in the extensions argument, you do not need to specify an alias.

- For NAME_SUM columns, expr must be a function of name which will result in a list of expressions mapped over all card names. - PICK_SUM columns can also be functions on name, in which case the value will be a function of the value of the PICK field. - AGG columns that depend on NAME_SUM columns reference the prefix (cdef.name) only, since the unpivot has occured prior to selection. - AGG columns must not be functions, since they may be applied to the aggregation of several sets' data. (And they shouldn't need this anyway) - The possible arguments to expr, in addition to name when appropriate, are as follows: - names: An array of all card names in the canonical order. - cardcontext: A dictionary keyed by card name which contains card dict objects with all CARDATTR values, including custom extensions and metric columns passed by the card_context argument to summon. See example notebooks for more details. - setcontext: A dictionary with arbitrary fields provided via the setcontext argument. Has two built-in attributes, picksperpack (e.g. 13 or 14), and releasetime, which is the minimum value of the drafttime field.

• version: When defining a column using a python function, as opposed to Polars expressions, add a unique version number so that the unique hashed signature of the column specification can be derived

for caching purposes, since Polars cannot generate a serialization natively. When changing the definition, be sure to increment the version value. Otherwise you do not need to use this parameter.

Columns

A table of all included columns. Columns can be referenced by enum or by string value in arguments and filter specs. The string value is always the lowercase version of the enum attribute.

| ColName | Name | View | ColType | Description | Type | | --------------------------- | ---------------------------- | ------------- | ------------- | --------------- | --------------- | | NAME | "name" | | GROUPBY | Special handling, don't use in filterspec | String | | EXPANSION | "expansion" | DRAFT, GAME | GROUP_BY | Dataset Column | String | | EVENTTYPE | "eventtype" | DRAFT, GAME | GROUP_BY | Dataset Column | String | | DRAFTID | "draftid" | DRAFT, GAME | FILTER_ONLY | Dataset column | String | | DRAFTTIME | "drafttime" | DRAFT, GAME | FILTER_ONLY | Dataset column | String | | DRAFTDATE | "draftdate" | DRAFT, GAME | GROUP_BY | | datetime.date | | FORMATDAY | "formatday" | DRAFT, GAME | GROUP_BY | 1 for release day, 2, 3, etc. | Int | | DRAFTDAYOFWEEK | "draftdayofweek | DRAFT, GAME | GROUP_BY | 1-7 (Mon-Sun) | Int | | DRAFTHOUR | "drafthour" | DRAFT, GAME | GROUP_BY | 0-23 | Int | | DRAFTWEEK | "draftweek" | DRAFT, GAME | GROUP_BY | 1-53 | Int | | FORMATWEEK | "formatweek" | DRAFT, GAME | GROUPBY | 1 for FORMATDAY 1 - 7, etc. | Int | | RANK | "rank" | DRAFT, GAME | GROUP_BY | Dataset column | String | | USERNGAMESBUCKET | "userngamesbucket" | DRAFT, GAME | GROUP_BY | Dataset Column | Int | | USERGAMEWINRATEBUCKET | "usergamewinratebucket | DRAFT, GAME | GROUP_BY | Dataset Column | Float | | PLAYERCOHORT | "playercohort" | DRAFT, GAME | GROUPBY | In-sample version of "Top", "Middle", "Bottom", etc based on USERGAMEWINRATE_BUCKET. Thresholds are 49% and 57% and 100 games played. | String | | EVENTMATCHWINS | "eventmatchwins | DRAFT | GROUP_BY | Dataset Column | Int | | EVENTMATCHWINSSUM | "eventmatchwinssum | DRAFT | PICK_SUM | | Int | | EVENTMATCHLOSSES | "eventmatchlosses | DRAFT | GROUP_BY | Dataset Column | Int | | EVENTMATCHLOSSESSUM | "eventmatchlossessum" | DRAFT | PICK_SUM | | Int | | EVENTMATCHES | "eventmatches" | DRAFT | GROUP_BY | | Int | | EVENTMATCHESSUM | "eventmatchessum" | DRAFT | PICK_SUM | | Int | | ISTROPHY | "istrophy" | DRAFT | GROUP_BY | 3 Match Wins if "Traditional", 7 if Premier | Boolean | | ISTROPHYSUM | "istrophysum" | DRAFT | PICK_SUM | | Int | | PACKNUMBER | "packnumber | DRAFT | FILTER_ONLY | Dataset Column | Int | | PACKNUM | "packnum" | DRAFT | GROUP_BY | 1-indexed | Int | | PICKNUMBER | "picknumber" | DRAFT | FILTER_ONLY | Dataset Column | Int | | PICKNUM | "picknum" | DRAFT | GROUP_BY | 1-indexed | Int | | PICKINDEX | "pickindex" | DRAFT | GROUP_BY | 0-indexed, through 39/42/45 depending on set | Int | | TAKENAT | "takenat | DRAFT | PICKSUM | Summable alias of PICKNUM | Int | | NUMDRAFTS | "numdrafts" | DRAFT | PICK_SUM | | Int | | NUMTAKEN | "numtaken" | DRAFT | PICK_SUM | Sum 1 over rows | Int | | PICK | "pick" | DRAFT | FILTER_ONLY | Dataset Column, joined as "name" | String | | PICKMAINDECKRATE | "pickmaindeckrate" | DRAFT | PICK_SUM | Dataset Column | Float | | PICKSIDEBOARDINRATE | "picksideboardinrate | DRAFT | PICK_SUM | Dataset Column | Float | | PACKCARD | "packcard | DRAFT | NAME_SUM | Dataset Column | Int | | LASTSEEN | "lastseen" | DRAFT | NAMESUM | PACKCARD times min(8, PICK_NUM), add 8 to give last pick num seen when summed | Int | | NUMSEEN | "numseen" | DRAFT | NAMESUM | PACKCARD for PICK_NUM less than 9 | Int | | POOL | "pool" | DRAFT | NAME_SUM | Dataset Column | Int | | GAMETIME | "gametime" | GAME | FILTER_ONLY | Dataset Column | String | | GAMEDATE | "gamedate" | GAME | GROUP_BY | | datetime.date | | GAMEDAYOFWEEK | "gamedayofweek | GAME | GROUP_BY | 1-7 (Mon-Sun) | Int | | GAMEHOUR | "gamehour" | GAME | GROUP_BY | 0-23 | Int | | GAMEWEEK | "gameweek" | GAME | GROUP_BY | 1-53 | Int | | BUILDINDEX | "buildindex" | GAME | GROUP_BY | Dataset Column | Int | | MATCHNUMBER | "matchnumber" | GAME | GROUP_BY | Dataset Column | Int | | GAMENUMBER | "gamenumber" | GAME | GROUP_BY | Dataset Column | Int | | NUMEVENTS | "numevents" | GAME | GAME_SUM | | Int | | NUMMATCHES | "nummatches" | GAME | GAME_SUM | | Int | | NUMGAMES | "numgames" | GAME | GAME_SUM | | Int | | OPPRANK | "opprank" | GAME | GROUP_BY | Dataset Column (tends to be blank) | String | | MAINCOLORS | "maincolors" | GAME | GROUP_BY | Dataset Column | String | | NUMCOLORS | "numcolors" | GAME | GROUPBY | len(MAINCOLORS) | Int | | SPLASHCOLORS | "splashcolors" | GAME | GROUP_BY | Dataset Column | String | | HASSPLASH | "hassplash" | GAME | GROUP_BY | | Boolean | | ONPLAY | "onplay" | GAME | GROUP_BY | Dataset Column | Boolean | | NUMONPLAY | "numonplay" | GAME | GAME_SUM | | Int | | NUMMULLIGANS | "nummulligans" | GAME | GROUP_BY | Dataset Column | Boolean | | NUMMULLIGANSSUM | "nummulliganssum" | GAME | GAME_SUM | | Int | | OPPNUMMULLIGANS | "oppnummulligans" | GAME | GROUP_BY | Dataset Column | Boolean | | OPPNUMMULLIGANSSUM | "oppnummulliganssum" | GAME | GAME_SUM | | Int | | OPPCOLORS | "oppcolors" | GAME | GROUP_BY | Dataset Column | Boolean | | NUMTURNS | "numturns" | GAME | GROUP_BY | Dataset Column | Int | | NUMTURNSSUM | "numturnssum" | GAME | GROUP_BY | | Int | | WON | "won" | GAME | GROUP_BY | Dataset Column | Boolean | | NUMWON | "numwon" | GAME | GAME_SUM | | Int | | OPENINGHAND | "openinghand" | GAME | NAME_SUM | | Int | | WONOPENINGHAND | "wonopeninghand" | GAME | NAMESUM | WON * OPENINGHAND| Int | | DRAWN | "drawn" | GAME | NAME_SUM | | Int | | WONDRAWN | "wondrawn" | GAME | NAME_SUM | WON * DRAWN| Int | | TUTORED | "tutored" | GAME | NAME_SUM | | Int | | WONTUTORED | "wontutored" | GAME | NAME_SUM | WON * TUTORED| Int | | DECK | "deck" | GAME | NAME_SUM | | Int | | WONDECK | "wondeck" | GAME | NAME_SUM | WON * DECK| Int | | SIDEBOARD | "sideboard" | GAME | NAME_SUM | | Int | | WONSIDEBOARD | "wonsideboard" | GAME | NAME_SUM | WON * SIDEBOARD| Int | | NUMGNS | '"numns" | GAME | NAMESUM | max(DECK - TUTORED - DRAWN - OPENINGHAND) | Int | | WONNUMGNS | "wonnumgms" | GAME | NAME_SUM | | Int | | SETCODE | "setcode" | CARD | CARD_ATTR | | String | | COLOR | "color" | CARD | CARD_ATTR | | String | | RARITY | "rarity" | CARD | CARD_ATTR | | String | | COLORIDENTITY | "coloridentity" | CARD | CARD_ATTR | | String | | CARDTYPE | "cardtype" | CARD | CARD_ATTR | | String | | SUBTYPE | "subtype" | CARD | CARD_ATTR | | String | | MANAVALUE | "manavalue" | CARD | CARD_ATTR | | Float | | DECKMANAVALUE | "deckmanavalue" | | NAMESUM | DECK * MANAVALUE | Float | | DECKLANDS | "decklands" | | NAME_SUM | Number of lands in deck | Float | | DECKSPELLS | "deckspells" | | NAME_SUM | Number of spells in deck | Float | | MANACOST | "manacost" | CARD | CARD_ATTR | | String | | POWER | "power" | CARD | CARD_ATTR | | Float | | TOUGHNESS | "toughness" | CARD | CARD_ATTR | | Float | | ISBONUSSHEET | "isbonussheet" | CARD | CARDATTR | SETCODE != EXPANSION | Boolean | | ISDFC | "isdfc" | CARD | CARD_ATTR | Includes split cards | Boolean | | ORACLETEXT | "oracletext" | CARD | CARD_ATTR | | String | | CARDJSON | "cardjson" | CARD | CARD_ATTR | The full dump of the mtgjson entry for the card as printed in the draft booster | String | | PICKEDMATCHWR | "pickedmatchwr" | | AGG | EVENTMATCHWINS / EVENT_MATCHES | Float | | TROPHYRATE | "trophyrate" | | AGG || Float | | GAMEWR | "gamewr" | | AGG | NUMWON / NUMGAMES | Float | | ALSA | "alsa" | | AGG | LASTSEEN / NUMSEEN | Float | | ATA | "ata" | | AGG | PICKEDAT / NUMPICKED | Float | | NUMGP | "numgp" | | AGG | DECK | Int | | PCTGP | "pctgp" | | AGG | DECK / (DECK + SIDEBOARD) | Float | | GPWR | "gpwr" | | AGG | WON_DECK / DECK | Float | | NUMOH | "numoh" | | AGG || Int | | OHWR | "ohwr" | | AGG || Float | | NUMGIH | "numgih" | | AGG |OPENING_HAND + DRAWN| Int | | NUMGIHWON | "numgihwon" | | AGG | WONOPENINGHAND + WON_DRAWN | Int | | GIHWR | "gihwr" | | AGG | NUMGIHWON / NUM_GIH | Float | | GNSWR | "gnsWr" | | AGG | WONNUMGNS / NUM_GNS | Float | | IWD | "iwd" | | AGG | GIHWR - GNSWR | Float | | NUMINPOOL | "numinpool" | | AGG | DECK + SIDEBOARD| Int | | INPOOLWR | "inpoolwr" | | AGG || Float | | DECKTOTAL | "decktotal" | | AGG | Sum DECK over all rows and broadcast back to row level| Int | | WONDECKTOTAL | "wondecktotal" | | AGG || Int | | GPWRMEAN | "gpwrmean" | | AGG | WONDECKTOTAL / DECK_TOTAL | Float | | GPWREXCESS | "gpwrexcess" | | AGG | GPWR - GPWR_MEAN | Float | | GPWRVAR | "gpwrvar" | | AGG | Game-weighted Variance | Float | | GPWRSTDEV | "gpwrstdev" | | AGG | Sqrt of GPWRVAR | Float | | GPWRZ | "gpwrz" | | AGG | GPWREXCESS / GPWRSTDEV | Float | | GIHTOTAL | "gihtotal" | | AGG | Sum NUM_GIH over all rows and broadcast back to row level| Float | | WONGIHTOTAL | "wongihtotal" | | AGG | | Float | | GIHWRMEAN | "gihwrmean" | | AGG | WONGIHTOTAL / GIH_TOTAL | Float | | GIHWREXCESS | "gihwrexcess" | | AGG | GIHWR - GIHWR_MEAN | Float | | GIHWRVAR | "gihwrvar" | | AGG | Game-weighted Variance | Float | | GIHWRSTDEV | "gihwrstdev" | | AGG | Sqrt of GIHWRVAR | Float | | GIHWRZ | "gihwrz" | | AGG |GIHWREXCESS / GIHWRSTDEV | Float | | DECKMANAVALUEAVG | "deckmanavalueavg" | | AGG | DECKMANAVALUE / DECK_SPELLS | Float | | DECKLANDSAVG | "decklandsavg" | | AGG | DECKLANDS / NUMGAMES | Float | | DECKSPELLSAVG | "deckspellsavg" | | AGG | DECKSPELLS / NUMGAMES` | Float |

Roadmap to 1.0

• [ ] Support Traditional and Premier datasets (currently only Premier is supported)
• [ ] Enable configuration using $XDGCONFIGHOME/cfg.toml
• [ ] Enhanced profiling
• [ ] Optimized caching strategy
• [ ] Organize and analyze daily downloads from 17Lands (not a scraper!)
• [ ] Helper functions to generate second-order analysis by card name
• [ ] Helper functions for common plotting paradigms
• [ ] Scientific workflows: regression, MLE, etc