The project’s goal is attempting to apply a technique based on neural nets that has been already and successfully applied with images: style transfer.
When dealing with text, this means paraphrasing a sentence in a target style without changing its meaning. Examples of style transfer include: formal/informal, modern/ancient, spoken/written with the stylistic signature of a particular author.
When dealing with text, this means paraphrasing a sentence in a target style without changing its meaning. Examples of style transfer include: formal/informal, modern/ancient, spoken/written with the stylistic signature of a particular author.
Background
Background
“I wish I could paint like Van Gogh.”
“I wish my writing could be as engaging as Conan Doyle’s.”
Truly fulfilling these kinds of wishes would require a genie. Most of the people would answer that is just a matter of natural aptitude. Yet, recent deep learning advances have created an interesting tool called style transfer. If we have a digital image that we took with our camera, in a few lines of Wolfram Language code we could produce a (digital) painting with the style of the famous Dutch artist, print it on canvas, and show it off on a wall.
But what about the text? Is deep learning capable of doing the same trick with prose? In the realm of Natural Language Processing (NLP), the term “style transfer” usually means paraphrasing a sentence by changing its style – which can include formality, the depth of vocabulary, and other features – while preserving its meaning.
A very close analogy is text translation: we want to take an input sentence in language A and transform all its features but the meaning. The new features should comply to the grammar and vocabulary of language B. We can say that stylistic paraphrasing (i.e., style transfer with text) is a very affine task to translation: the target language is identical the input language.
Unfortunately, neural algorithms for text translation require a training dataset that consists of parallel samples. The network must know how to align each sentence of the input language to those of the target one. If our aim is to apply the style of an author to a sentence originally written in another style, there are no parallel dataset we can build. We must find another approach.
“I wish my writing could be as engaging as Conan Doyle’s.”
Truly fulfilling these kinds of wishes would require a genie. Most of the people would answer that is just a matter of natural aptitude. Yet, recent deep learning advances have created an interesting tool called style transfer. If we have a digital image that we took with our camera, in a few lines of Wolfram Language code we could produce a (digital) painting with the style of the famous Dutch artist, print it on canvas, and show it off on a wall.
But what about the text? Is deep learning capable of doing the same trick with prose? In the realm of Natural Language Processing (NLP), the term “style transfer” usually means paraphrasing a sentence by changing its style – which can include formality, the depth of vocabulary, and other features – while preserving its meaning.
A very close analogy is text translation: we want to take an input sentence in language A and transform all its features but the meaning. The new features should comply to the grammar and vocabulary of language B. We can say that stylistic paraphrasing (i.e., style transfer with text) is a very affine task to translation: the target language is identical the input language.
Unfortunately, neural algorithms for text translation require a training dataset that consists of parallel samples. The network must know how to align each sentence of the input language to those of the target one. If our aim is to apply the style of an author to a sentence originally written in another style, there are no parallel dataset we can build. We must find another approach.
Possible models for non-parallel corpora
Possible models for non-parallel corpora
Investigating the best deep learning approach for NLP tasks is a relatively new field compared to computer vision. Yet, the literature on style transfer with text is quite extensive. The starting point of this project was a paper by Fu and coworkers: they recognized the limitation of the lack of parallel corpora and propose two approaches both based on the encoder–decoder architecture that was originally introduced with the “Sequence-to-sequence” (or briefly Seq2Seq) model by Sutskever.
1
.A model with an encoder combined with multiple decoders
2
.An encoder–decoder model with style embeddings
Our attempt was to implement the first architecture whose scheme is the following :
The important features of the model are:
◼
We have two encoder-decoder pieces, each of which is trained with the “Teacher Forcing” approach: the training model at time (t+1) receives as input the ground truth output y(t). We we’ll have to train the two decoders where the “Target” style is passed as the initial input.
◼
The classifier on the left is also important. It’s trained along the other part of the network with an adversarial loss against the encoder: the encoder tries to produce a “content representation” of an input which is style independent, while the classifier tries as much as possible to distinguish the encoded input.
◼
The adversarial network is therefore composed by two parts that correspond to the two loss functions that we have to minimize during training:
In[]:=
ℒ
1
Θ
class
M
∑
i=1
l
i
x
i
Θ
encoder
Θ
class
ℒ
2
Θ
encoder
N
∑
j=1
M
∑
i=1
x
i
Θ
encoder
Θ
class
where M is the number of input tokens and N is the number of styles. The style flag is indicated by ,whilethedifferentΘ indicate the sets of parameters.
l
i
A multi-decoder approach based on the Seq2Seq model
A multi-decoder approach based on the Seq2Seq model
The Seq2Seq model was originally developed for neural translation, and therefore it’s suited for text corpora that are parallel by nature. The tricky detail that Seq2Seq has to deal with is that a translated sentence can have (and usually has) a different length than the input sentence.
Let’s now see how our multi-decoder model is built piece by piece and what is the purpose of each of them.
Let’s now see how our multi-decoder model is built piece by piece and what is the purpose of each of them.
◼
Encoder, which takes a sentence and a word-embeddings vector. In our case, it’s a fixed-length vector.
◼
Decoder, which is responsible of generating, token by token, the output sequence. It’s based on a Recurrent Neural Net as it was in the original Seq2Seq. Our multi-decoder model is in fact a “two-decoder” model since we aim at changing the style of a sentence from author A to author B. As we said, this is very similar to language translation.
◼
Classifier, whose task is taking an element of the so-called latent space – the space produced by the encoder – and assigning it a label corresponding to the author’s style. This piece is crucial: the parameters of the classifier are adjusted against the encoder’s ones with the “adversarial trick” we described above.
◼
Teacher Forcing architecture, which puts everything together for the training step. The peculiarity of this approach is that, when training, we have to provide the inputs as well as the targets.
◼
Style index is a simple Boolean flag that plays the role of a binary switch and picks which decoder has to be adjusted for each input–target pair of sentences.
Encoder
Encoder
Decoder
Decoder
The task of the decoder is to take an encoded sentence and “rebuild” it token by token. Hence, we first need a vocabulary from which the decoder can reconstruct the output sentence. We can build such dictionary starting from our input sentences.
For example, say that our input sentences are the following:
For example, say that our input sentences are the following:
sentences={"\"Must we not all die, dearest?\" I asked with a sad smile.","We were surely sufficiently degraded.","The youth was an early victim.","I found the granaries of Rome well stored with grain, and particularly with\nIndian corn; this product requiring less art in its preparation for food, I\nselected as my principal support.","Should we, any of us, survive the coming summer?"};
We can then build the vocabulary as follows. Although we could build our tokenizer – or import and use one from well-known libraries available online – we can grab the TextTokenize function which is available in the Wolfram Language:
<<NaturalLanguageProcessing`ClearAll[tokenizer];tokenizer[___,sentences_]:=Flatten@NaturalLanguageProcessing`TextTokenize[sentences]
$vocabWithSpecTokens=Join[Union@@KeyValueMap[tokenizer/*Union,sentences],{StartOfString,EndOfString}];$vocab=$vocabWithSpecTokens〚;;-3〛;
where we added the two special tokens StartOfString (SOS) and EndOfString (EOS). These special tokens shouldn’t be part of the actual vocabulary: that’s why we removed from the $vocab variable.
Since we have used the word embeddings from ELMo (see the encoder above), for the decoder we need to build word embeddings from our vocabulary.
This is ELMo’s embedding layer
wordembedding=NetTake[elmo,{All,"embedding"}];
And these are the word embeddings of our vocabulary
wordEmbeddingWeightMatrix=NumericArray[Join[wordembedding[TextElement/@List/@$vocab][[All,1,All]],ConstantArray[0,{2,NetExtract[encoderToVector,"Output"]}]],"Real32"]
Finally, the decoder:
decoder=NetInitialize@NetGraph[<|"wordEmbedding"NetInsertSharedArrays[EmbeddingLayer["Weights"->wordEmbeddingWeightMatrix],"WordEmbedding/"],"RNN"->GatedRecurrentLayer[NetExtract[encoderToVector,"Output"]],"classif"->NetMapOperator[NetInsertSharedArrays[LinearLayer[Length[$vocab]],"WordEmbedding/"]],"softmax"->SoftmaxLayer[]|>,{NetPort["Input"]"wordEmbedding""RNN""classif""softmax",NetPort["State"]NetPort["RNN","State"]},"Input"NetEncoder[{"Class",$vocabWithSpecTokens,"Index","Dimensions""Varying"}],"Output"->NetDecoder[{"Class",$vocabWithSpecTokens,"InputDepth"2}]]
Teacher Forcing
Teacher Forcing
Are we grasping the authors’ styles?
Are we grasping the authors’ styles?
Training the net: single-style inputs
Training the net: single-style inputs
We first train the network as an auto-encoder by providing only input and target samples of the same author.Since there are two distinct purposes of this net – auto-encoding and classifying – we use the option TrainingUpdateSchedule to specify which parts of the net should be trained simultaneously. The syntax means “any of or, but don’t mix the two groups. The StyleIndex flag set to True corresponds to the 1st author (we picked Mary Shelley), while False will switch to the 2nd author (Robert Louis Stevenson).
{|,|}
part
1
part
2
part
3
part
4
part
1,2
part
3,4
net=NetTrain[NetInitialize@teacherForcingNet,<|"Input"trainingInputsA,"Target"trainingTargetsA,"StyleIndex"ConstantArray[True,Length@trainingInputsA]|>,All,ValidationSetScaled[.2],LearningRate0.001,LossFunction{"LossClassifStyle"Scaled[1],"LossAutoEncoder"Scaled[1]},TrainingUpdateSchedule{"encoder"|"decoder1"|"decoder2","classifier"|"decoder1"|"decoder2"},TargetDevice"GPU",WorkingPrecision"Mixed",MaxTrainingRounds10,PerformanceGoalAutomatic]
We can then build a token-by-token predictor out of the trained net. It suffices taking the encoder plus the correct decoder – that is, the one that was effectively trained since we supplied single-style inputs. In this example, we have to take the first decoder:
trainedNet=net["TrainedNet"];
trainedEncoderDecoder1=NetTake[trainedNet,"decoder1"]
And the predictor
predictor=NetGraph[{trainedEncoderDecoder1,SequenceLastLayer[]},{12},"Output"->NetDecoder[{"Class",$vocab}]]
where the output port is a post-processor that will convert the output, a vector of probabilities, into the most probable token from our vocabulary.
Additional information
Additional information
The full notebook with the additional sections on data collection and pre-processing, together with the details on how to train the Teacher Forcing network can be found in the Notebook Archive.
Acknowledgements
Acknowledgements
I want to thank my mentor, Jesse Galef, for his constant support and valuable hints on how to make (apparently naive) stuff work. And for his optimism throughout the school.
I also owe a huge thank you to Jérôme Louradour (Wolfram Research) for his tireless help in setting up the neural net architecture and clarifying a thousand times the smallest details of it.
I also owe a huge thank you to Jérôme Louradour (Wolfram Research) for his tireless help in setting up the neural net architecture and clarifying a thousand times the smallest details of it.
References
References
1 Fu et al., “Style Transfer in Text: Exploration and Evaluation”, arXiv:1711.06861 (2017)
2 Sutskever, Vinyals, and Le, “Sequence to Sequence Learning with Neural Networks”, Advances in neural information processing systems, 3104-3112 (2014)
3 Peters et al., “Deep contextualized word representations”, Proceedings of NAACL (2018). Also available at arXiv:1802.05365