sketchnet

sketchnet - processing code generator

can we teach a computer to draw pictures with code. We use Processing and java/jruby code paired with the pictures it makes to train the system. The model will then be able to take unseen images generate working code for it.

For more info and results, see the blog post: http://www.jtoy.net/projects/sketchnet/

As far as we know, this should be the first working model to generate working usable code.

functions we are interested in modeling

• sin/cos
• shapes: ellipse/rect/etc
• line
• bezier

references

• https://github.com/karpathy/neuraltalk andrew karpathy neuraltalk first implementation
• https://github.com/karpathy/neuraltalk2 rewrite #2
• https://github.com/tensorflow/models/tree/master/im2txt/im2txt im2txt tensorflow version
• http://blog.otoro.net/2015/12/28/recurrent-net-dreams-up-fake-chinese-characters-in-vector-format-with-tensorflow/
• https://arxiv.org/abs/1609.06647 Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
• https://cl.ly/2w1d3e1F1P3S NeuralTalk on Embedded System and GPU-accelerated RNN
• https://cl.ly/2P2Z2y2e2N0j Deep Visual-Semantic Alignments for Generating Image Descriptions
• https://arxiv.org/pdf/1611.01989.pdf deepcoder paper
• https://arxiv.org/abs/1510.07211 On End-to-End Program Generation from User Intention by Deep Neural Networks
• http://memorability.csail.mit.edu/index.html lamem site
• http://people.csail.mit.edu/khosla/papers/iccv2015_khosla.pdf lamem paper
• https://arxiv.org/abs/1506.06726 skip thought vectors
• https://arxiv.org/pdf/1405.0312.pdf mscoco paper
• https://arxiv.org/pdf/1602.02410.pdf Exploring the Limits of Language Modeling
• https://people.csail.mit.edu/rinard/paper/popl16.pdf Automatic Patch Generation by Learning Correct Code
• https://arxiv.org/abs/1705.07962 pix2code: Generating Code from a Graphical User Interface Screenshot
• https://arxiv.org/pdf/1609.04938.pdf What You Get Is What You See: A Visual Markup Decompiler
• https://arxiv.org/pdf/1511.07275.pdf LEARNING SIMPLE ALGORITHMS FROM EXAMPLES