Color Forecast

Pimkie Color Forecast is a framework of applications that harvest the most trending colors on certain locations. It was a joint venture with Happiness Brussels, namely Patrick Glorieux, Thibault Jadoul+Nicolas Felten from Everything is fun and Bliss Interactive.

Usually, in my projects, I am most interested in coming up with the visual outcomes of certain approaches, accounting if the approach itself is of relevant interest, intellectual honesty and conceptual elegance. For the Color Forecast I developed the color clustering software, being framed within a straightforward concept (without known applications in the fashion context), delivered on a beautifully designed website, iPhone app, and presented with joyful short videos. Having a project that results in something so conceptually balanced and with great design quality, it is a great pleasure to contribute with the low-level intricacies, that in spite of being under the hood, just make it work.

The process is broadly described in the following video. Albeit the effort of Happiness Brussels in publicizing the existence of these technical intricacies, or how it is called, technology, I feel the necessity of describing my contribution, explaining the devoted effort and clarifying the overall simplicity of the approach.

Frame differencing

The cameras are meant to be installed pointing at pedestrian streets, where generally the only things that move are people and no vehicles are expected. To get rid of the backgrounds that are of no importance in detecting the colors that people are wearing, the application only captures moving pixels through frame differencing. These moving pixels are stored during fixed periods of time (for example, 3 minutes, after when the pixels are discarded and a new set of temporal moving pixels starts being recorded again).

3D histogram

Since the application has to be running on 24/7, special attention had to be taken on the data structures used, in order to assure performance and avoid memory leaks. The temporal histogram, for example, is stored on a 255 × 256 × 256 matrix, so instead of storing a pixel, which coordinates are of no interest in this simple approach, only a specific entrance of the matrix is incremented in order count the number of occurrences of a certain color which RGB values [0-255] are expressed in the matrix coordinates.

K-means

Each time the capture period is over, a K-means clustering runs over a copy of the histogram on a separate thread. Typically the number of clusters ranges from 10 to 20, and can be setup together with other important variables (camera resolution, framerate, threshold for frame differencing, accumulation period, name of the city and database scripts). It is important to notice that this K-means clustering is only interested in grouping colors, therefore not accounting for the amount of occurrences of a color, but only if it occurred or not.

Seeds

Since this K-means is meant for Pimkie, the seeds for the clusters are fixated at white, black, and all the equally distant pure hues that are left for the K-2 clusters.

Color distance

The K-means needs a distance measure between colors. I tried several approaches for this, from a RGB Manhattan distance to improve performance, to an HSB distance and passing through a perceptual euclidean distance measure ( √(0.26R + 0.70G + 0.04B), referred in Gijenij et al. “A Perceptual Comparison of Distance Measures for Color Constancy Algorithms“, ECCV 2008). After several experiments, the distance measure that presented the best clustering results, from of course my empirical subjective observation, was just the straight forward euclidean distance between R, G and B.

Averaging colors

The K-means algorithm also requires a procedure to compute the mean color of a cluster, iteratively adapting the cluster’s centroid during the K-means routine. Again, the approach that presented the best results for this was just the average of R, G, B.

Representative color

After having the colors clustered in K groups, it is time to compute the representative color for each cluster. This mean color is not just the simple mean of colors, but a weighted average of colors by their number of occurrences (because if in a group of reds, there are much more pinks than maroons, then the representative color should be more pink than maroon).

Color perception

Now that we have these K groups of colors, each with a specified amount of occurrences, how to say that one group is more important, or perceptible than another? After-all we are interested in finding the most trending color of the moment. In my point of view, and without academic formalisms, if everybody is grey, dull or dark, and a magnificent woman parades with a yellow scarf, I will say that the most perceptible color in the scenery is yellow. So, after running an image segmentation algorithm (a simple region growing one) over a database of about one thousand images, the empirical measure that I came up with to describe how vibrant a color can be is: (saturation × brightness)² × number_of_occurrences. So the color clusters are sorted not only by how many occurrences of moving pixels they represent, but also weighted by a quadratic importance of saturation and brightness of their representative color.

Normalization

After each K-means concluded, its results are submitted to a database, in the format of: K colors sorted by order of importance; the corresponding absolute number of occurrences; and the normalized (by maximum) number of occurrences, because color perception is dependent of the temporal accumulation context and should not be assumed that the yellow scarf woman that passed today in front of the camera (more yellow moving pixels) is less or more important than that same yellow scarf woman that passed yesterday farther from the camera.

Ongoing project

For now, colors are being clustered for Milan, Antwerp and Paris and might extend to other cities. Feedback is being absorbed in order to improve the computational approach accuracy and corresponding representation of the most trendy colors on specific streets.

Celebrating colors

And as this project is all about colors, I share an old visual essay that I did about the nature and harmony of colors (in CMYK, though).