Piano-Drone: How I 'Hacked' a Drone With a Piano

A Horrible, Expensive, Time-Consuming Idea is Born

The idea was pretty simple. Take a drone, make a MIDI interface which sends commands over a network using PyARDrone based on what key is pressed, record the video off the drone using OpenCV, and record piano keypresses using PyGame, and sync those two together to make a movie, that said movie's soundtrack being, you guessed it, the piano track!

Yea. Simple.

Technologies Used:

Obviously, this won't cover absolutely EVERYTHING, just the stuff I used personally.

• Python 3: The programming language used for this project.
• Pygame & Pygame.midi: Used to make the GUI, AND used to get the piano key presses.
• PyArDrone: Used for sending drone commands over a network. HUGE shoutout to the author of this open-source library. Without this, the whole project would have been impossible.
• OpenCV (cv2): Manages video recording and processing from the drone's camera.
• MIDIUtil: Used for creating the midi file of the keyboard presses.
• Ubuntu: Used to actually connect, and send, commands to the drone. Windows firewall rules are terribnle, so I just dualbooted my pc for the sake of this project. Work Smarter, not Harder. (Fun Fact: I am actually full time on Fedora Linux for about a year now. It's FOSS!)

Let's put it together now:

Architecture:

How it all worked:

1. Initialization:

   Piano.py kicks off the show by initializing Pygame and Pygame.midi to handle graphical display and MIDI input. It sets up the MIDI input/output devices and loads visual assets representing key states. Meanwhile, Drone.py initializes the AR.Drone object, sets up video recording parameters, and prepares threading for handling video streams.

2. MIDI Input Handling:

   As your fingers waltz across the MIDI keyboard, playing beautiful symphonies that would make Mozart jealous, Piano.py detects these graceful (or clumsy) movements. Each key corresponds to a specific drone command (e.g., takeoff, move forward, rotate). The script updates the visual display to reflect the current state of each key, giving you real-time feedback that your drone is both listening and watching. At the same time, MIDIUtil is recording all of your beautiful keystrokes, ensuring they can be played back later.

3. Drone Command Execution:

   Piano.py sends interpreted commands to Drone.py, which then processes these commands, translating them into actions using the PyArDrone library. For example, pressing the "Takeoff" key sends a takeoff command to the drone, while pressing directional keys moves the drone accordingly. I did this by setting up a dictionary of all the keypresses' MIDI identifier commands, as well as their "value", or how much to tell the drone to do them. Like, rotate left a small amount, or go up a large amount.

4. Video Recording:

   When you decide to capture your performance, Drone.py starts recording video from the drone's camera using OpenCV. Video recording runs on a separate thread to ensure it doesn't interfere with real-time drone control. Recorded footage is saved with timestamped filenames for easy organization—perfect for sharing your aerial concerts on social media. I made the video recording function on the daemon, which in retrospect, was probably not a good idea.

5. Emergency Controls:

   An emergency key can be pressed multiple times to trigger an immediate drone shutdown, killing all rotors immediately. This key was more for the protection of everything else around the drone. Not the drone itself.