Artificial (?) Antidancer
A digital companion for the antidance improviser, a stick figure that follows, remixes and counterpoints the movements of a person.
Background – Recap of previous homeworks
In the three homeworks for the class, I explored and studied the processing of tracked skeletons and how they, as multi-dimensional data, could be used as the source of generative (anti)choreography.
In TSP Dances, I explored the idea of distance (or difference) between poses in a mathematical way, to see to what extent that mathematical distance corresponded to visual similarity. This exercise used a brute-force approach to order a given set of poses so that the resulting sequence would have the least amount of travelling distance. It confirmed that there is a relationship between the abstract distance and the visual similarity.
With GA Choreographic Counterpoint (and its previous study of an appropriate fitness function) I decided to expand on the previous idea, using a Genetic Algorithm to re-order a previously shuffled set of poses that corresponded to a movement sequence. The hypothesis was that there were arrangements of poses that would have a similar “movement flow” as the original sequence, and that the Genetic Algorithm would be useful to find them. The experiment turned out to be positive in a way, as some fluidity and movement recombination was achieved by the algorithm. However, there were still significant jumps between adjacent poses.
With kNN Dances (Dijkstra Dances) and its previous experiment kNN Dances, I wanted to explore the kNN (k-Nearest Neighbors) concept in a bigger set of poses: all the poses exist as points (vectors) in a 75-dimensional space, and the distance between them correspond to their similarity. Poses that were executed contiguously, for example as two successive frames, would be clearly similar and close. However, searching for more than 2 neighbors (k>2), would force the system to find similarities between poses that were executed at different moments in time. I like to visualize this network (or graph) as “turning off” the dimension of time, and as connecting (and finding / creating) paths inside a history of poses/movements.
Once that I could create a graph consisting of the nearest neighbors of all the poses, in the first experiments I tried to traverse that graph in a random way, either jumping to any neighbors, or weighting that jump according to the distance between poses. The algorithms worked, but the visual result was not what I was looking for as there was a lot of flickering. I decided that I could include another topic for the class in order to make the traversal more “deliberate” / “directed”: Using Dijkstra’s algorithm, I would be able to find the shortest path between two random poses, and theoretically that path would be a smooth (and using only the existing “materials”) transition between them. The experiment worked in a convincingly way, and that made me decide to push the idea further.
For this project I will develop the kNN – Dijkstra’s dances idea in the following ways:
- It will be live tracking a person
- It will show in screen both the tracked skeleton and the artificial skeleton, probably with different colors.
- It will base the “target poses” of its graph traversal in the/a pose(s) of the person. It could capture a live pose, look for its nearest neighbor in the existing graph of poses, and move towards it.
- It will also have some “creative freedom”, e.g. it could move to very distant poses in the space at certain intervals.
- It will add and process new poses from the person. It could randomly add them at a certain sampling rate, and/or it could only add them when they are above a given threshold of distance with the current poses.
- This previous point raises something interesting, that is the Pose Representation – this new version, instead of storing the poses as collections of three-dimensional points in space, should store them in a “normalized” way, i.e. as a collection of three-dimensional unit vectors representing the directions (angles) to go from one joint to the other. This will allow for comparison between poses of different people and or in different positions in the space.
- For the display of the artificial stick figure, it will be required that the poses-as-vectors are “converted” back to points in space. For this it would be interesting to use the magnitude of the segments of the person.
- A challenge that arose when working with Dijkstra’s dances, was that depending on the k, the graph of the kNN could be non-completely connected (sorry if I’m not using the right term) – therefore it was impossible to arrive from one pose to the other. What I did to solve it for that particular data set, was to find the minimum k that would make sure that there were paths between all the poses. Maybe this should be automatized, iteratively increasing the k when Dijkstra can’t find a path between chosen poses.
- I will need to find an optimal size for the pool of poses. The addition of new poses could be worked out in a queue. In Dijkstra dances I used 900 poses (~30 seconds) and everything worked well, I wonder at what size the distances and kNN calculation, as well as the Dijkstra algorithm, will start to lag the execution of the program.
- I will probably port everything to Processing so that I can run it live with the Kinect and with “Java Virtual Machine” full power (?). The data structures will be interesting to solve…