Misc

Image to Haiku on Raspberry Pi 5 (Talking LLaMA)

This project, for On Device Deep Learning, involved taking three transformer based models and making them more suitable for deployment on a small device, in our case, the Raspberry Pi 5. For each component, there was an element of model compression. Our ViT was trained from scratch and pruned on CIFAR-100, and used to determine a top five classes from the input image to our camera. For the purpose of a more extended project, this can definitely be improved to give more specific and granular results, that would work better in the context of a camera box. Our LLaMA model, which generated a haiku based on 5 input classes from CIFAR-100, was finetuned on a haiku dataset and pruned using LoRA. Lastly, our text to speech model used HiFi GAN with tacotron, which was pruned aggresively, with a pruning of 30% for the decoder and 20% for the encoder. By ablating pruning of different layers in the text to speech pipeline, only specific portions were pruned from the final model.

To the side is a demo of the setup, with an example output haiku. This takes around 85 seconds to run on the Raspberry Pi 5, with 5W for 0.18s for the ViT CIFAR-100 classification model, 7W over 16.16s for the LLaMA model, and 6.25W over 69s on the text to speech model. This causes the majority of time spent to be in the text to speech, which is pruned agressively enough that is already difficult to understand.

Kayak Game

This is a puzzle game I have been making in Godot when I have free time or inspiration for a new puzzle. This has been an idea I came up with years ago but have just recently been designing into a playable game. The objective of this game is simple. You have a certain number of time, displayed in the top left corner, and a distribution of people, kayaks, and houses around a lake. In order to move a kayak, you need a person to paddle it. There are variants such as two person kayaks that can transport multiple people, and other quirks like "walking trails", but the simple premise is you need a person to paddle each kayak. The objective is to get every kayak to the house of its corresponding color before the number of moves runs out. As you may be able to see by the demo, I am not a particularly talented visual artist, so this serves currently as a proof of concept for the mechanics of the game. Eventually, I plan to flesh out the design and mechanics, as well as the artwork with help from those who have those skills. As a move along in the proces of designing this game, I may create a more in depth dev log of the progress and my thought process in designing these puzzles.