After countless days of research and ideas, I finally have a clear goal. I’m so excited about this project because I’m sure I myself will use it quite a lot.
The first thing I need to do is to deepfake the characters in the book. And I found this amazing paper which was published only a few months ago.
First Order Motion Model for Image Animation generates a video sequence so that an object in a source image is animated according to the motion of a driving video.
Pretty impressive right? You could either transfer the motion or just simply replace the animated image video back to the source motion video itself.
There is a pre-trained model released for human faces to animate any human face image. The model recognizes the key points in human faces and thus learn the motion of those points during training. Interestingly, it not only works on actual human faces but also some non-human faces like paintings and other species. This would be fantastic to animate people from the TV series and even video game characters that are not humans!
The output video is always 256×256 pixels since the model is trained on 256×256 videos, for higher solution output, ESRGAN can be used for upsampling the video.
Here are the results:
(I noticed that the face in the source video can’t move around too much and too fast, but for most of the case, the result is fantastic.)