Category: Research

I wanted to know a little more about the history of AI generated music, mainly as I wanted to get more background, and I wanted to separate the work that I’m going to algorithmic generations. Algorithmic generation is software that uses rules or formula to generate the results. A good example is Verbasizer (@taoski, no date), created for David Bowie to generate song lyrics from randomised words. It doesn’t use AI, but a randomised number generator to put lyrics together. 

The first academic texts on music that used neural networks that were published are from Peter Todd (1988), “A sequential network design for musical applications” and Lewis (1988), “Creation by refinement: A creativity paradigm for gradient descent learning networks”. While I can get access to the paper that Lewis authored, I can only see the paper from Todd as a reference cited quite widely, with no available sources online. It’s unfortunate that I cannot get access to the paper from Todd as it is one of the first examples of the use of an RNN to generate music. 

The next major advancements in music AI were in 2002. This was the use of LSTM for music. In their paper Eck and Schmidhuber used LSTMs (with RNNs) to find the temporal structure in Blues music. 

Also in 2002 was the use of spectrograms as the input for a generative neural network (Marolt, 2002). While using spectrograms as data to train a music model doesn’t sound amazing. It was a speculative leap. The idea of using images, rather than just sound, to train a neural network for music opened the field of study. And allowed advances in image processing to be applied to music recognition and categorisation. During this time is when GPUs transistor count and acceleration of maths functions on a GPU really came into their own with parallel processing which made GPUs outperform CPUs (Merit, 2023)

To help visualise these papers over time, I’ve created a handy timeline (Figure 1):

As you can see from the timeline the technology around AI has increased dramatically in the last few years. 

For ST4RT+, most of the technology that I’m working with is based on the 2002 paper by Eck & Schmidhuber. Their use of RNN and LSTM is the closest in terms of structure and concept to the ST4RT+ model. While the technology I’m using was first developed in the early 2000’s, later technology focused on generating and analysing waveform data rather than representational (MIDI) data. My reasoning for using representational data was so that I didn’t need to spend a lot of money in processing to create the artefacts. Plus, it has the added benefit that it allows me more creative freedom to choose instruments that work with the melodies that ST4RT+ generates. This allows me to be the “Music Producer” in the collaboration with ST4RT+ rather than have the model generate full music sequences. 

Timeline Bibliography

Lewis and Todd papers from the 80’s:

Todd, P. (1988) “A sequential network design for musical applications,” in Proceedings of the 1988 connectionist models summer school, 1988, pp. 76-84. 

Lewis, J. P.  (1988) “Creation by refinement: a creativity paradigm for gradient descent learning networks.” IEEE 1988 International Conference on Neural Networks (1988): 229-233 vol.2.

The first time someone used LSTMs for music:

Eck, D., & Schmidhuber, J. (2002). “Finding temporal structure in music: blues improvisation with LSTM recurrent networks”. Proceedings of the 12th IEEE Workshop on Neural Networks for Signal Processing, 747-756. 

The first time someone processed spectrograms with neural networks:

Marolt, M., Kavčič, A., & Privosnik, M. (2002). “Neural Networks for Note Onset Detection in Piano Music” in International Computer Music Conference (ICMC).

The first time someone built a music genre classifier with neural networks — based on Hinton’s deep belief networks for unsupervised pre-training:

Lee, H., Pham, P.T., Largman, Y., & Ng, A. (2009). “Unsupervised feature learning for audio classification using convolutional deep belief networks” in Neural Information Processing Systems (NIPS).

Hinton, G.E., Osindero, S., & Teh, Y.W. (2006). “A Fast Learning Algorithm for Deep Belief Nets” in Neural Computation, 18, 1527-1554. 

The first time someone built an end-to-end music classifier:

Dieleman, S., & Schrauwen, B. (2014). End-to-end learning for music audio. 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 6964-6968.

A study run at Pandora Radio showing the potential of end-to-end learning at scale:

Pons, J., Nieto, O., Prockup, M., Schmidt, E.M., Ehmann, A.F., & Serra, X. (2017). End-to-end Learning for Music Audio Tagging at Scale. ArXiv, abs/1711.02520. 

Humphrey and Bello did some work on chord recognition and wrote the deep learning for music manifesto:

Humphrey, E.J., & Bello, J.P. (2012). “Rethinking Automatic Chord Recognition with Convolutional Neural Networks” in 2012 11th International Conference on Machine Learning and Applications, 2, 357-362. 

Humphrey, E.J., Bello, J.P., & LeCun, Y. (2012). “Moving Beyond Feature Design: Deep Architectures and Automatic Feature Learning in Music Informatics” in International Society for Music Information Retrieval Conference.

Discussion on how to improve current architectures:

Choi, K., Fazekas, G., & Sandler, M.B. (2016). “Automatic Tagging Using Deep Convolutional Neural Networks” in International Society for Music Information Retrieval Conference. 

Pons, J., Lidy, T., & Serra, X. (2016). Experimenting with musically motivated convolutional neural networks. 2016 14th International Workshop on Content-Based Multimedia Indexing (CBMI), 1-6.

Lee, J., Park, J., Kim, K.L., & Nam, J. (2017). Sample-level Deep Convolutional Neural Networks for Music Auto-tagging Using Raw Waveforms. ArXiv, abs/1703.01789.

Some modern generative models for algorithmic composition (GANs and VAEs, basically):

Yang, L., Chou, S., & Yang, Y. (2017). MidiNet: A Convolutional Generative Adversarial Network for Symbolic-Domain Music Generation. ArXiv, abs/1703.10847.

Roberts, A., Engel, J., Raffel, C., Hawthorne, C., & Eck, D. (2018). A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music. ArXiv, abs/1803.05428.

And some works directly synthesizing music audio (waveGAN and Wavenet, basically):

Donahue, C., McAuley, J., & Puckette, M. (2018). Synthesizing Audio with Generative Adversarial Networks. ArXiv, abs/1802.04208.

Oord, A.V., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A.W., & Kavukcuoglu, K. (2016). WaveNet: A Generative Model for Raw Audio. Speech Synthesis Workshop.

Dieleman, S., Oord, A.V., & Simonyan, K. (2018). The challenge of realistic music generation: modelling raw audio at scale. ArXiv, abs/1806.10474.

Engel, J., Resnick, C., Roberts, A., Dieleman, S., Norouzi, M., Eck, D., & Simonyan, K. (2017). Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders. ArXiv, abs/1704.01279. 

Bibliography

Merrit, R (2023) “Why GPUs Are Great for AI”. Available at: https://blogs.nvidia.com/blog/why-gpus-are-great-for-ai/ (Accessed: 20 January 2025)

@taoski (no date) “Verbasizer”. Available at: https://verbasizer.com (Accessed: 20 January 2025)

To better prepare for this project I researched various types of Machine Learning (ML) models. This post is a distillation of my research into the models and why I have chosen to use RNNs (Recurrent Neural Networks) and VAEs (Variational Autoencoders).

Recurrent Neural Networks

Are a type of neural network that processes sequences of data, in the case of music, a time series. The difference between an RNN and other types of neural networks is that RNNs have “hidden memory” of any previous inputs. This is because they have loops that allow information to pass between one step in a sequence and the next sequence.

Because of the way that RNNs loop it means that they are good at time-series prediction. The issue with RNNs is that they aren’t so good with long-term dependencies. In the case of music, repeated motifs, may well disappear from longer generated passages because of the looping limitation. To combat this RNNs can be used in conjunction with LSTM (Long Short-Term Memory) for the ML to “remember” previous outputs for longer giving the music a more pleasing structure. 

For this project I chose to use an RNN as it is a good model to both generate sequences and create based on previous input.

Variational Auto Encoders

Variational Autoencoders (VAEs) work differently and are designed to generate new data. A VAE has two main parts. An Encoder that compresses the input data into a smaller, abstract representation of the data, called a latent space. And a Decoder, which tries to reconstruct the input data from the compressed representation (Figure 2). 

By encoding the data into a probability distribution rather than a single point in the latent space, you can introduce some randomness by sampling the learnt distribution in the latent space. 

VAEs are usually trained to balance what is called Reconstructive Loss, where we can measure how well the VAE can reconstruct the original data from the latent space. And KL Divergence, which ensures that the latent space is organised and smooth so that any new samples it creates produce good results. 

I chose to use a VAE as it is a true generative model. It can also handle interpolations and variations by sampling points near the original point in the sampled latent space.

Why both?

Well, VAEs are generative but aren’t good a time-series (notes over time). I could use Transformer-based architectures to do this but RNNs have the advantage for music models for keeping longer term structures in the music when combined with LSTM. 

Bibliography

IRCAM (2022) “IRCAM Tutorials / Rave and nn~” [Online]. Available at: https://www.youtube.com/watch?v=o09BSf9zP-0 (Accessed: 1 November 2024).

For the Final Major Project (FMP), I have decided to combine my other passion, technology, with my music practice.

Introducing ST4RT+! An AI version of me, to help me co-create with an AI for composition.

To do this I have done a bit of a dive into Machine Learning (ML) models with a view to being able to do this project without having to train on an exceptionally large dataset. This is important for a few reasons: I do not want to spend most of my time creating training data (I want to be able to feasibly do the project in the time given), and I also want to make sure the the model data is ethical.

So, to find a model that has these attributes, I looked first at Recurrent Neural Networks (RNN) and Variational AutoEncoders (VAE). Both can utilise “latent space” in a pre-trained model. Latent space is where you create a smaller model within an existing model that trains on a much smaller set of data (Figure 1).

For an RNN Model there are two different types, Lookback and Attention, that were both created to give the model the ability to create long term structure in the music it produces. The Lookback RNN can recognize patterns that occur over a 1-2 bar range, using a technique called LSTM (Long Short-Term Memory).

To learn a longer pattern phrase, we use an encoder/decoder during every output step for each new note. This is how the Attention RNN model works. Simply put, with every new note added to the sequence from the model, we look at the last number of steps to evaluate what the next note should be. 

My work from here is to play with some of the code and produce a proof of concept of both a RNN and a VAE-based model and see if the results are acceptable after training both types of models using my input in the latent space to create a model that sounds more like me.

From there I should have the tools to use the AI model as a way of bouncing off as a creative partner, and a form of co-creation, which is the aim of my project.

I was attempting to use only JavaScript for most of the melody generation, but on further research, I may have to use Python for the RNN but should still be able to use JavaScript for the VAE model. This is due to the complexities of having to either the lookback or attention models which require complex vector math that isn’t as efficient in JavaScript. 

Bibliography

Dinculescu, M., Engel, J., Roberts, A. (2019) MidiMe: Personalizing a MusicVAE model with user data. [Online]. Google Research. Available at https://research.google/pubs/midime-personalizing-a-musicvae-model-with-user-data/ (Accessed: 10 October 2024)

Waite, E. (2016) Generating Long -Term Structure in Songs and Stories. [Online]. Google TensorFlow. Available at: https://magenta.tensorflow.org/2016/07/15/lookback-rnn-attention-rnn (Accessed: 12 October 2024)