Roman Parker
Intern, Wolfram Research
Teacher forcing networks are extremely effective for generating text, especially if that text contains specific repeating patterns. This makes it ideal for creating text based on Magic: The Gathering cards, which contain several easily-distinguishable patterns. Since there are over 20,000 different cards, there is a significant amount of data for the network to train on. In addition, the Japanese poem style of Haiku has a specific short form. This is good for analysis by this AI, demonstrating that it works on different kinds of text.

Magic: The Gathering Card Generation

In this section, a teacher forcing network is created to generate the text of the Magic: The Gathering cards. All cards have the following five parts, in this order:
◼
  • Card name (what the card is called)
    • ◼
  • Card type (basic information about when the card can be used and what rules it follows)
    • ◼
  • Mana cost (how easy is it to play the card)
    • ◼
  • Rules text (what the card does)
    • ◼
  • Set symbol (when the card was printed)
    • This makes it relatively easy for an AI to create cards that follow this basic pattern. Cards also often share specific words in their rules text which instruct the player on how to treat the cards, such as "flying" or "vigilance," which makes it easier for the network to create these cards.
    For this program, a teacher forcing network will be defined with two GatedRecurrentLayers as the main part of the network. Unlike most networks for text training, which take an input of characters and try to predict the next character, a teacher forcing network takes the input of characters and tries to predict the string of characters of the same length of the input, starting from the second character of the input. For example, if the input is "I like to code in the Wolfram Languag" and the "correct" character is "e," the network will attempt to generate " like to code in the Wolfram Language," rather than just the "e." This helps the network understand the context of the character better than if it simply had to generate a single character.
    Import the plaintext of all Magic: The Gathering cards printed up to the year 2018:
    In[]:=
    corpus=
    ;
    To create the training data, a “sliding window” of length 500 is created. Starting from the first character of the corpus, the first 500 characters become the input portion of the first piece of training data, and the 501st character is the output. Then, for the next piece of data, the 2nd through 501st character are the input and the 502nd is the output. This process is done until the end of the dataset is reached, creating the training data. This works nicely with the teacher forcing networks since every piece of training data is identical to the “correct answer” of the previous piece of training data.
    Create the training data:
    In[]:=
    trainingData=Module[{b=Characters[corpus]},Thread[Table[StringJoin[b[[i;;i+500]]]b[[i+501]],{i,Length[b]-501}]]];
    Display a random data point in the training data:
    In[]:=
    trainingData[[RandomInteger[{1,Length[trainingData]}]]]
    Out[]=
    ntil end of turn.US-C​Ravenous Trap2BBInstant -- TrapIf an opponent had three or more cards put into their graveyard from anywhere this turn, you may pay {0} rather than pay this spell's mana cost.Exile all cards from target player's graveyard.ZEN-U​Ravenous Vampire3BBCreature -- Vampire3/3FlyingAt the beginning of your upkeep, you may sacrifice a nonartifact creature. If you do, put a +1/+1 counter on Ravenous Vampire. If you don't, tap Ravenous Vampire.MI-U​Raving Dead4BCreature
    Find all characters used in the dataset:
    In[]:=
    characters=Union[Flatten[Characters[corpus]]]
    Out[]=
    !,&,*,(,),_,-,+,{,[,},],.,,,;,",?,',/,:,, ,€,ˆ,ž,²,½,0,1,2,3,4,5,6,7,8,9,a,â,A,Â,b,B,c,C,d,D,e,E,f,F,g,G,h,H,i,I,Ï,j,J,k,K,l,L,m,M,n,N,o,O,p,P,q,Q,r,R,s,S,t,T,u,U,v,V,w,W,x,X,y,Y,z,Z,¢
    Find length of card list:
    In[]:=
    n=Length[characters];
    Define the NetChain of the network:
    In[]:=
    predict=NetChain[{UnitVectorLayer[n],GatedRecurrentLayer[128],GatedRecurrentLayer[128],NetMapOperator[LinearLayer[n]],SoftmaxLayer[]}]
    Out[]=
    NetChain
    uniniti
    aliz
    ed
    Input port:
    vector of n indices (range: 1..93)
    Output port:
    matrix (size: n×93)
    Number of layers:
    5
    
    Create the teacher forcing network based on the example in tutorial/NeuralNetworksSequenceLearning:
    In[]:=
    teacherForcingNet=NetGraph[<|"predict"predict,"rest"SequenceRestLayer[],"most"SequenceMostLayer[],"loss"CrossEntropyLossLayer["Index"]|>,{NetPort["Input"]"most""predict"NetPort["loss","Input"],NetPort["Input"]"rest"NetPort["loss","Target"]},"Input"NetEncoder[{"Characters",characters}]]
    Out[]=
    NetGraph
    uniniti
    aliz
    ed
    Input port:
    string
    Loss port:
    real
    Number of layers:
    4
    
    Train the network (takes a long time):
    result=NetTrain[teacherForcingNet,<|"Input"Keys[trainingData],"Output"Values[trainingData]|>,All,ValidationSetScaled[0.1],TargetDevice"GPU",TrainingStoppingCriterion"Loss",MaxTrainingRounds3]
    Extract network:
    In[]:=
    trained=NetExtract[result["TrainedNet"],"predict"]
    Out[]=
    NetChain
    Input port:
    vector of n indices (range: 1..93)
    Output port:
    matrix (size: n×93)
    Number of layers:
    5
    
    Save a prediction net for generating text and potential future training:
    In[]:=
    predictNet=NetJoin[NetTake[trained,3,"Input"Automatic],{SequenceLastLayer[],NetExtract[trained,{4,"Net"}],SoftmaxLayer[]},"Input"NetEncoder[{"Characters",characters}],"Output"NetDecoder[{"Class",characters}]]
    Out[]=
    NetChain
    Input port:
    string
    Output port:
    class
    Number of layers:
    6
    
    Define code to generate text and generate example text:
    In[]:=
    generateSample[net_,start_,len_]:=Block[{obj=NetStateObject[net]},StringJoin@NestList[obj[#,"RandomSample"]&,start,len]]
    In[]:=
    generate[net_,start_,len_]:=StringJoin@NestList[NetStateObject[net],start,len]
    Generate Magic: The Gathering cards from the network:
    In[]:=
    generateSample[predictNet,"",500]
    Out[]=
    {1}{W}, {T}: Discard a card." (Tap X is your control since the beginning of remove attach opponent.)Phaebling Myr Responephere enters the battlefield with three ores.AKH-C​Gruul GangLandYou may have Jace's Shambler gets +1/+0 until end of turn.WWK-C​Drown Loby1WEnchantment -- AuraEnchant creatureEnchanted creature has "{2}{U}, {T}: Put a +1/+1 counter on this creature.)M15-U​Lifeblood Totem2UUCreature tokehs without flying and If an opponent controls a land of each basic thir comb
    A few of the interesting cards the network created after training for about 24 hours:
    ◼
  • Regenge Myr
    4
    Artifact Creature -- Scarecrow
    1/4
    {2}, Pay 1 life: Creatures you control get +2/+2 until end of turn.
    EMN-U
    • ◼
  • Earth Squire
    2
    Artifact
    Tap two untapped permanents you control: Destroy target creature.
    LGN-U
    • ◼
  • Pursi
    4W
    Instant
    Creatures you control get +1/+1 and gain lifelink until end of turn.
    DOM-U
    • ◼
  • Descack Mamchais
    4BBGG
    Legendary Artifact
    {3}{B}, {T}: Put a creature card from a graveyard onto the battlefield with three +1/+1 counters on it.
    RTR-M
    • ◼
  • Llaitic Sergecling
    1U
    Creature -- Bird
    2/2
    Flying
    Whenever a creature deals damage to you, draw a card.
    ISD-R
    • ◼
  • Workspiant Archer
    3W
    Creature -- Griffin
    1/4
    Flying
    Pay 2 life: Regenerate target creature.
    HOU-C
    • ◼
  • Spire Cull
    2BB
    Enchantment
    White creatures you control get +1/+1.
    LE-U
    • ◼
  • Smolderous Farm
    2GG
    Enchantment
    Whenever you cast a spell, you may draw a card.
    MRD-U
    • ◼
  • Honee Temple
    Land
    {T}: Add {C}.
    {T}: You may prevent all damage that would be dealt to target permanent you control this turn. If you do, draw a card.
    IN-R
    • ◼
  • Ojutai Distorvo
    G
    Enchantment
    Sacrifice a Mountain: Creatures you control gain indestructible until end of turn.
    RAV-U, C13-U
    • ◼
  • Parading Stronghold
    2W
    Creature -- Elf
    2/1
    As an additional cost to cast this spell, sacrifice a creature.
    When this creature enters the battlefield, you gain 10 life.
    Morph {5}{G}{G} (You may cast this card face down as a 2/2 creature for {3}. Turn it face up any time for its morph cost.)
    ONS-R
    • As can be seen in the 500 characters of generated cards above, many of the cards don’t make much sense or contain gibberish. However, almost all of the cards follow the correct textual patterns, demonstrating that the teacher forcing network was able to pick up on the five parts. Additionally, approximately 10% of the cards, besides their unusual names, make perfect sense and seem to be human-designed. This shows that the program is capable of generating text with a well-defined pattern and structure to it. The same network is next used to generate Haiku, which has a less obvious pattern in its structure.

    Haiku Data

    Haiku is a Japanese short poem form with no rhyme or stress requirements, but the first and third lines have only five syllables, while the second has seven. Due to the strict syllabic requirements and short length, the poems are often difficult to write. A challenge this poses for a teacher forcing network is that, in English, there is no obvious divider between syllables in text, so it should be difficult for the program to create technically correct poems. Additionally, the varied topics in this dataset mean that there are few semantic patterns for the AI to pick up on to create grammatically and linguistically cohesive text.
    Import the data:
    Find the characters used:
    Like on the previous dataset, the training data input uses a sliding window, but the “window” size is only 70 characters instead of 200. This is approximately the mean length of the poems.
    Create the training data from all possible continuous length-70 strings in the corpus:
    Find how many characters are in the haiku dataset:
    Create the NetChain:
    Define the network:
    Train the network:
    Extract the network:
    Save the prediction net:
    Generate haiku:
    A few of the interesting poems this program created:
    I had given up
    before you been on account
    recreated will
    ​
    There’s nothing better
    and we not crush up in Memes
    are written boogies
    ​
    Please have your very
    current trust my line
    a brand of umbrellas
    ​
    I never wanted
    crossed spring words to the same
    long place is that bless
    ​
    I just gotta grow
    your sentence finishes times
    I already know
    ​
    I think I wonder
    now and be called time after
    that this is some type
    ​
    I don’t even know
    why you can do the husbands
    oh meeting and straight

    Concluding remarks

    Impressively, this AI is quite adept at creating haiku which follow the 5-7-5 syllable pattern, even without actually knowing where the syllable breaks are located. However, the poems often lack a narrative structure or flow, suggesting that either the AI didn’t train enough to pick it up or the haiku dataset contained too many different topics for it to figure out what theme to follow in each haiku.
    This demonstrates that it is possible to use the same AI, only modifying the training of the network, to create two very different types of text. It is likely possible to model any kind of text consisting of many repeated patterns using this network, making it extremely useful.

    Acknowledgment

    I’d like to thank Siria Sadeddin (WRI) and Mads Bahrami (WRI) for their help in improving the code and the essay.