MusicExampleSource - maximum length of audio

keyser84keyser84 MemberPosts:9Contributor II
edited November 2018 inHelp

I want to cluster large audio data (>2h) using RapidMiner GUI.

I use MusicExampleSource (from ValueSeries Plugin) for reading the audio, but obviously it can handle only very short audio samples.
When trying to load a ca. 2 min. WAV sample (44100Hz, 16-Bit, Mono --> should be approx. 5 MB in memory) I get an OutOfMemoryError (Java heap space).

What goes wrong? Why is MusicExampleSource allocating tons of memory?
How can I read in larger audio samples?

Answers

  • IngoRMIngoRM Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, Community Manager, RMResearcher, Member, University ProfessorPosts:1,751RM Founder
    Hi,

    (44100Hz, 16-Bit, Mono --> should be approx. 5 MB in memory)
    No. 441000 Hz * 120 s * 8 byte are approximately 40 Mb. As long as you do not change the internal representation format, internally RapidMiner uses double values (hence the 8 byte). Plus overhead you will end up more likely with something like 100 or 200 Mb if you are lucky. And that's for a single audio sample only. The next thing is the question which features are calculated (if any at all) which will usually also drastically increase memory usage.

    In general, it is hard to work on larger audio samples (we ourself often work with a few seconds only) but on a larger system you should be able to process a complete song - at least for a simple feature extraction tree.

    Cheers,
    Ingo
  • keyser84keyser84 MemberPosts:9Contributor II
    Thanks for reply. Just noticed that I've calculated for 1 min. (assuming one sample takes 2 Bytes).

    But ca. 20 MB (for 1 min., double) plus overhead aren't enough, too.
    RapidMiner consumes about 1.5 GB reading only a 1 min. sample. I've tested with the MusicExampleSource node alone.
    1.5 GB is far beyond what should be allocated.

    Is there a way to do windowing without reading in the complete sample (using RapidMiner GUI)? Otherwise I would have to write my own loader in Java.
Sign InorRegisterto comment.