Stylistic Transfer in Natural Language Generation Systems Using Recurrent Neural Networks
11 Feb 2018
- This workshop paper explores the problem of style transfer in natural language generation (NLG).
- One possible manifestation would be rewriting technical articles in an easy-to-understate manner.
- Identifying relevant stylistic cues and using them to control text generation in NLG systems.
- Absence of a large amount of training data.
- Using Recurrent Neural Networks (RNNs) to disentangle the style from semantic content.
- Autoencoder model with two components - one for learning style and another for learning content.
- This allows for “style” component to be replaced while keeping the “content” component same, resulting in a style transfer.
- One way to think about this is - the encoder generates a 100-dimensional vector. In this, the first 50 entries, correspond to the “style” component and remaining to the “content” component.
- The proposal is that the loss function should be modified to include a cross-covariance term for ensuring disentanglement.
- I think one way of doing this is to have two loss functions:
- The first loss function ensures that the input sentence is decoded properly into the target sentence. This loss is computed for each sentence.
- The second loss ensures that the first 50 entries across all the encoded represenations are are correlated. This loss operates at the batch level.
- The total loss is the weighted sum of these 2 losses.
- Soundness - is the generated text entailed with the input sentence.
- Coherence - free of grammatical errors, proper word usage etc.
- Effectiveness - how effective was the style transfer
- Since some of the metrics are subjective, human evaluators also need to be employed.