NaNoGenMo 2019 - GPT-2 Edition
My goal this year was to take some random text, character, and location generators and build a basic recursive quest engine.
Let's unpack that. All I want for now is:
- locations - a small network of places, with the ability for characters to travel between them
- characters - a recurring finite set of characters who travel between the locations and interact when they're in the same place
- stretch goal - interact when they're passing one another!
- recursive - the idea is that we have a world... it has, say, cities, within cities are places (inns, bars, markets), within those may be rooms... or maybe our recursion is deep enough at two levels
Eh, maybe that's even too much. I work a super lot and in eight days of November so far all I've done is written those bullets. But let's see what's out there...
So there are some good generators out there that sort of start to do what I want. I may just daisy chain them together... listed a few below. TextWorld world generations is almost exactly what I want, it's just single-character. Hmmm.
My repo is here: https://github.com/chipmonkey/ChipMonkeyNaNoGenMo2019 And my NaNoGenMo repo is here: https://github.com/NaNoGenMo/2019/issues/76
hit this bug: https://github.com/openai/gpt-2/issues/178
git clone https://github.com/openai/gpt-2.git && cd gpt-2 docker build --tag gpt-2 -f Dockerfile.cpu . docker run -it gpt-2 bash export PYTHONIOENCODING=UTF-8
Need to run this on AWS or something:
2019-11-09 20:43:18.919955: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2019-11-09 20:43:23.418088: W tensorflow/core/framework/allocator.cc:122] Allocation of 154389504 exceeds 10% of system memory. 2019-11-09 20:43:23.722155: W tensorflow/core/framework/allocator.cc:122] Allocation of 154389504 exceeds 10% of system memory. 2019-11-09 20:43:44.915623: W tensorflow/core/framework/allocator.cc:122] Allocation of 18137088 exceeds 10% of system memory. 2019-11-09 20:43:45.006514: W tensorflow/core/framework/allocator.cc:122] Allocation of 18210816 exceeds 10% of system memory. 2019-11-09 20:43:45.102333: W tensorflow/core/framework/allocator.cc:122] Allocation of 18284544 exceeds 10% of system memory.
Although it did actually spit out text, so for now I'm working inside this docker.
git clone https://github.com/facebookresearch/ParlAI.git cd ParlAI; python setup.py develop
pip install textworld
Textworld is really straighforward and has a nice generator... the example from the README.md is:
tw-make custom --world-size 5 --nb-objects 10 --quest-length 5 --seed 1234 --output tw_games/custom_game.ulx
tw-make custom --world-size 15 --nb-objects 20 --quest-length 20 --seed 1729 --output tw_games/game2.ulx
Also, it's a quick modification to the sample "play_gym.py" to the environment https://textworld.readthedocs.io/en/latest/textworld.envs.wrappers.html Just open an outfile
outfile = open('textworld_out.txt', 'w+') and drop the environment to it via text
So that a playthrough drops a nice quest line with some descriptive information.
After that, it's a fairly easy change to https://github.com/openai/gpt-2/blob/master/src/interactive_conditional_samples.py to read from an input file rather than the command line. Point that at the output from TextWorld, and GPT-2 will embellish a story based on each line of the TextWorld adventure.
GPT-2 is all over the place. Need to dig in to focus on specific datasets; retrain on gutenberg-fantasy-english. Also, TextWorld seems broken. Can't quite figure out how to win even a very simple quest which makes me wonder if it's generating winnable quest trees. Lastly, still need to add multiple people.
But for now, we have writing!
Ok, here's what happened. Work got busy. HAHA, no, it's always busy. Anyway.
What follows here is basically the README.md from the github code here:
Borrowing heavily from these open source products
- Run finetune_gpt2.py to train GPT2 on Grimm's fairy tales
- Run make_a_novel.py
make_a_novel.py generates a story with TextWorld and walks through a play run (that part isn't fully automated yet). It then takes the narrative from the playthrough and passes it through gpt2 to add lots of words, which are hopefully embellishments on the story thus generating a novel.
It's not very coherent yet, but ya know.
As a shortcut, if you have a gpt-2 model, and want to use the example text you should be able to just do:
import make_a_novel make_a_novel.makethenovel()
which will load a model available to gpt-2-simple and generate text from a pre-played TextWorld output storyguide.
The result of doing that, at least one example, is my NaNoGenMo 2019 submission: NaNoGenMo2019 Final Text