A Visual Representation Of Music 2f3z2n

A visual representation of music Guillermo Cornejo Suarez

Fabian Melendez Bolaos

Escuela de Ingenieria Electrica. Universidad de Costa Rica. PRIS-LAB collaborator Email: [email protected]

Escuela de Ingenieria Electrica. Universidad de Costa Rica. PRIS-LAB collaborator Email: [email protected]

Abstract—We attempt to create an intelligent bioinspired system capable to identify different structures in a song, in order to create a visual representation of music, that establishes a relation between sound and color, based in physical properties of waves and generates an aesthetic result in which tone, pitch and volume keeps a relation with color, texture and shape, respectively.

July, 2014

can result in an extremely powerful form of expression. A visual representation of music tool could be applied in several fields like musical therapy for disabled kids, so as in new music learning techniques; synesthetic, stimulus or perception studies, and in future it could help in bio-inspired systems mixing visual and audio data towards a better human-computer interaction.

I. I NTRODUCTION To find a reliable visual representation of music is a cuttingedge investigation field, that depends of transdisciplinary work from neuroscience, engineering, computer science, psychology, music and visual arts. Investigation in this area could be fundamental to clarify some complex phenomena, like synesthesia. Synesthesia is a phenomenon that two senses resonate such as hearing and vision, sense of touch and vision [1]. There is evidence that demonstrates matches between music and colors that are mediated by emotional associations. Usually faster music in the major mode is related to color choices that are more saturated, lighter, and yellower whereas slower, minor music is related the opposite pattern (choices that were desaturated, darker, and bluer). [3] Color-Music investigations are closely related to colorhearing synesthesia. As Cytowic explained that ability of the color-heard is very rare [1]. He thought that the color-heardperson will exist one person in 25,000 people. However, it is reported by N.Nagata et al [2] that color-heard sense is a general sense that everyone has. So in a way as Palmer et al citePalmer concluded, there is a common sense for relating color and music. There is another concept that relates to the concept of music and color relationship, Color music is a concept, according to Poast [4], that attempts to create a notation system composed of painted colors and shapes to provoke musical responses: it is a complex representation of musical composition ideas in a visually fixed form. The system is based on the idea that color sensations can trigger correlations to musical sound in a performer who is sensitive to visual experience. By combining visual and aural stimuli, the system and its use

A. Related work In two papers, Kawanobe et al [8] first investigates the corresponding affect between music and color in order to realize the media conversion between music and image according to the feeling. This research derived in the creation of an algorithm for media conversion based on the feelings and perceptions of a small group of persons, and possibly not a representative sample of the population, and using other valuable psychological elements to describe music, such as words, but not directly computable information. Husain et al [6] proposed a framework for visualising music mood using texture image. The goal was to allow finding new music. This framework could be use to establish a relation between music and sound. Shimotomai et al [11] proposed a mapping model from chord to color. Based in questionnaire applied to common people, they created a linear, statistical model for chord-color mapping, the result was a table relating a small number of chords with a color in HLS color coordinates.

II. A V ISUAL REPRESENTATION OF MUSIC BASED IN PHYSICAL RELATIONS

Many works on the field propose to establish a relation between music and color based on perceptions or psychological elements, resulting in complex tables or algorithms. We propose to follow a relation based on physical correspondence between the sound and light wave. Perez, Gilaber [9] achieved a simple logarithmic relation between the frequency of sound (related to tone of music) and light wavelength, given in the equation:

λc = 72.135 ln

340 f

+ 577.76

(1)

The use of texture to visualize the music mood were proposed by Husain et al [?], but uses emotional perceptions to establish relations as cheerful and elegant to smooth textures or rough texture to aggressive genre such as rock. Even so, it could be use as a base to find a relation between pitch and texture, and a more mathematical treatment of the common elements of rough or smooth music could drive to a more appropriate relation for our intentions. Smith, Williams [5] proposed a representation of the volume as relative size of the displayed spheres. Each spheres radius is restricted to the range [0, 127], this corresponds to the minimum and maximum volume values for a tone.

to create fractals, specifically the Julia set fractals, from which the Mandelbrot fractals are a subset. Basically it is a way of studying complex plane numbers behavior towards an iterative function, and comparing the module of the resulting complex-surface with certain value, which gives us a defined area in a complex plane.n In order to represent the color, and shapes of our music visualization, we have taken three values of colors (now in RGB notation), from which we will produce the images. For this, we decided to modificate the normal Julia Set Fractals approach towards an algorithm that studies the resulting complex-plane numbers module with more than one value, obtaining different shapes and areas (that are painted with the three different colors). The movement of the fractals in order to create moving shapes, is made by changing a constant in the complex iterative function in the Julia Set, in a pattern of a rhodonea curve ( which is a sinusoid curve plotted in polar coordinates). One of the generated frames could be appreciated in image 2

Fig. 1: Block diagram for creating the proposed visual representation of music Image 1 shows the block diagram to create the proposed visual representation. The process starts with the sound capture. The image display must occur in less than human reaction time, around 100 ms, so the process only records a minimum number of samples. It also stores around the last minute to perform an analysis. This consist in a frequency domain transform, and then a search for the minimum window capable to store the smallest constant frequency present in the song, in other words, it looks for the minimum window capable to store the smallest executable note, it could be a sixty-fourth note at 270 beats per second in a specially fast piece. Note that the process of creating a visual representation could be understand as a sampling of music, using the window determined above as sampling period. In order to respect the NyquistShannon sampling theorem, the number of frames per second of display must double the sample frequency, i.e. each frame must be show for less than half the window determined above. Finally, for each window we determine a corresponding color using the relation found by Perez, Gilabert [9], shown in equation 1, and translate this wavelength information to a RGB color space. As the concept of color-music embraces, the representation of music involves the mixture of color, texture, patterns and shapes. To make an abstract, yet mathematical approach towards creating a visual representation of music, we decided

Fig. 2: One of the generated frames. Color is obtained by analysing the principal frequencies of sound and shape is created using fractals III. F UTURE WORK In order to improve the current implementation more statistical methods should be considered in the analysis of the song, detecting even more complex features as pitch. Also simple features as the amplitude should be part of the algorithm. A current ime in having a real time algorithm for live music is the processing time for the images, that

needs to be faster, or it could be implemented with color light sets. R EFERENCES [1] R.E.Cytowic, The man who tasted shapes, Soushisya, Tokyo,2002. [2] N.Nagata and Shokuchi, Mapping extraction of Color-heard-person, Japan, pp.97-104,2002-MUS-4717, 2002. [3] Stephen E. Palmer, Karen B. Schlossa, Zoe Xua, and Lilia R. Prado. Musiccolor associations are mediated by emotions. Proceedings of the national academy of sciences. Vol 110 no. 22, 8836-8841 - May 13, 2013. [4] Michael Poast & Leonardo June. Color Music: Visual Color Notation for Musical Expression. The MIT Press, Vol. 33, No. 3: 215-221, 2000. [5] Sean M. Smith & Glen N. Williams, Ph.D. A Visualization of music. Proceedings of the 8th IEEE Visualization ’97 Conference, 2007. [6] Adzira Husain, Mohd Fairuz Shiratuddin, Kok Wai Wong. ”A Proposed Framework for Visualising Music Mood using Texture Image”. 3rd International Conference on Research and Innovation in Information Systems, 2013. [7] Kazuhiro Yamawaki & Hisao Shiiuka.”Correlation of Synesthesia and Common Recognition Concerning Music and Color”. IEEE International Conference on Systems, Man and Cybernetics, 2004. [8] Makoto Kawanobe Masashi & Kameda Makoto Miyahara”Corresponding Affect between Music and Color”. In proceeding of: Systems, Man and Cybernetics. IEEE International Conference on, Volume: 5, 2003 [9] Perez, J., Gilabert, E. (2010). Color y msica: Relaciones fsicas entre tonos de color y notas musicales. ptica pura y aplicada. 43 (4) p267-274. [10] Makoto Kawanobe & Masashi Kameda Development of an Algorithm for Media Conversion between Music and Color Combination based on Impressions. 6th - International Conference on Information, Communications & Signal Processing, 2007. [11] Takayuki Shimotomai, Takashi Omori, Eriko Aiba, Takashi X. Fujisawa, & Noriko Nagata. ”Mapping Model from Chord to Color”. t 6th International Conference on Soft Computing and Intelligent Systems (SCIS) and 13th International Symposium on Advanced Intelligent Systems (ISIS), 2012 Kobe, Japan, 2012.