Massive Music Libraries

[Audio_ELF]

Not sure how much use it would be... But check out collectorz.com for a (fully relational) music database which (IIrC) can link to music files for playback... Not sure what playback applications you could make it link to though...

 

Eloise

 

[PeterSt]

I just tried it.

 

I don't like the web application, but that will be personal, I'm sure.

 

Let it loose on a folder of 600 albums (which it all read in - so much for a 100 CD trial :-) - but now I indeed can't add anymore) ...

 

A fairly complete chaos came from it. Unuseable further. It will depend on some expected structure, which exactly is the most difficult to make transparent (I know, haha).

 

 

 

 

 

 

 

 

 

 

 

 

[my attachments seem to cease showing up here, but they are at the bottom. Notice that the last 3 ending at a should be at the top. In the mean time by now my keyboard adopted another mapping, and I see a relation to Collectorz (question mark)

 

Main Coverart is recognized, but the remainder is nowhere to be seen.

 

 

 

 

Search facilities look good (great) to me, though results seem to be album oriented always (no tracks for result). Thus, search for "Love" and albums pop up with "Love" in track names. This will show nicely highlighted once the tracks are examined, but I see no way to show the love tracks. I also see no way to search in album names only, so, unuseable in the end.

 

You can create youw own filters in a fairly intuitive way, including exclusions ("not") - (like filters on email).

 

From all the sort/filter results, immediate statistics are available.

 

Scanning of the 600 album folder took several minutes, but as far as I can tell it accesses some internet source to find some general data like date of production and genre (useful).

 

Playback goes via associations, and whether the m3u can be changed to something else ... I guess so.

 

Coverart pictures are not rendered to their shown size (looking quite awful).

 

Although it seems to try ("Discs"), I found no functional support for multi volume albums (all end up as separates).

 

I found no means to multi-select anything, so no mass applications seem possible (like removing a bunch of albums from the list).

 

An album can be "linked" to a central database, so the data comes from there. Now it gets really messy because now multi volumes are recognized, and they mix up with the other structure which it created from the scan. Additionally icons for FLAC (which were correctly taken from the physical music data, now turn into something else (possibly indicating *that* it originates from somewhere else now).

 

Ok, I'll leave it at this. Can't perform many of the functions because I reached the 100 CD trial limit (not so smart, so *if* you try, scan less than 100 albums !).

 

I'd say superb and fast search facilities (looks like to remain fast with many more albums), but personally I wouldn't use those much.

Coverart support is very poor, and ugly in the mean time.

I see no integration with *maintaining* a collection. This also means, no support for that, while is really is about that for 100% (try it, at having a couple of thousands of albums).

There's nothing of the kind of support for sharing music data, unless sharing your database with someone else to just look at. But ok, this is nothing to expect of course.

At this moment, for me it would be totally unusable because it makes a mess of my structure. It will do that from yours just the same, unless you'd comply to some rules (which I don't see but probably will be written somewhere, but which I won't ever comply to anyway -> how to do that).

 

May more gadgets are in there, like being able to scan the barcode from a CD cover, and obtain the data. Probably it may find your own CD somewhere in your own collection on disc (didn't try that).

 

Peter

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

[vortecjr]

Audiozorro recommended having several computers to split up the data. That seemed reasonable, but why not run several instances of the player instead? That is exactly what I just tried here for the first time and this is how I did it. When Andrew at Vortexbox setup the Vortexbox-Player section of the web interface he incorporated a way to allow independent playback to several devices. This is actually several instances of the player running. So I set up two players one named Vortexbox-Player Jazz and one named Vortexbox-Player Classical. I also set both to output to the same device:) I then manually edited there configuration files to point them to the corresponding data. This can be done up to five times with the current version of the OS and took me 3 minute to setup. BTW I use an itouch for control and I can swap between the two instances in the app:) This may not help you directly, but it may inspire you or others to find a solution as it did for me.

 

Jesus R

www.sonore.us

 

[Paul R]

I was referring to the original posters question of how to build a music system that will manage 500,000 tracks or more.

 

In that context, and that context only, there are several pieces to the puzzle:

 

The music engine - the part that actually picks up the data file (wherever or however it is stored) and sends it to a DAC, via whatever media. This is where Pure Music, Amarra, Bryston, Quicktime, and XXHighEnd fit today. None of those products manages the music library. (XXHighEnd uses file organization on the disk, no?)

 

The library engine - this part organizes the files on the disk, presents the metadata, organizes playlists, handles remote access, etc. iTunes and J.River Media center are examples of this. JRMC stretches to include a music engine as well, but as a separate, if well integrated component. iTunes uses Quicktime. Within its data storage limits, iTunes is, in my opinion, the best library manager out there.

 

The display engine - this part shows the user a view into the library, allows updates, insertions, replacements, and deletes to the library, allows the user to control the music engine, and allows the user to configure the display, library, and music engine parts as the user desires and to the limit each part is configurable.

 

The part that is failing for the original poster is the "library engine" - as iTunes uses SQLlite for this, and 500,000 records is approaching or perhaps even exceeding the limitations of SQL lite.

 

Yes, it is possible to replace the SQLLite engine in iTunes and thus relieve the constraints, but Apple won't be happy with that, and moreover, you run the risk of breaking many other parts of a Mac system. (Media can be inserted into many kinds of documents, and much of that relies on the same library manager that iTunes uses.)

 

Replacing the library manager with a custom written monolithic application is the "ideal" solution, and in many ways, fits with Audiophile thinking. It has some limitations, none of which, except cost, would come into play in the situation the OP posited. Also, the display manager would be integrated with a monolithic system, almost by definition.

 

Replacing the library manger with a COTS (Commercial Off The Shelf) database system would easily manage the size and response time requirements - far exceed the stated need of a half million records in fact. Physical file storage can be a disk based system, or it can a more sophisticated storage system that accounts for other issues, like backup. Since all the COTS software one would choose for this task is multi-user, the addition of multiple users is a freebie.

 

The cost for using COTS software is almost always lower, even given the need to create a separate display engine for it.

 

Apple, with iCloud, is doing something very much like this.

 

I like the ideas you have for a global system, and it would be a very fun task to build it up. As you say, it would have some legal ramifications. You should start a thread to discuss it Peter. It's probably a bit off topic for the OP and I expect we have hijacked this thread quite enough.

 

-Paul

 

 

[Paul R]

That is an excellent practical solution Jesus!

-Paul

 

 

[vortecjr]

Thanks! I have another idea, but not sure if people will like it. When I had Windows machines I hated to see the player scan the library each time I started it. I also didn't want that processing taking place while I was listening to music. So I deleted the library and no more scanning:) I would then use Windows Explorer to locate and initiate playback. It works like a champ every time. What can be faster and easier to browse a large collection of files. Of course I maintain a nice folder structure to help the process. Also when you get to the artist and album level it shows the cover art on the folder icon...lol

 

Jesus R

www.sonore.us

 

 

 

[Paul R]

But it won't work quite as well when you get 500,000 tracks out there, because the file system itself will start to get slow.

 

Then you get into the concept to adding a level of indirection, perhaps you start storing them on different drives - the JAZZ drive, the ROCK drive the LONGHAIR drive, etc.

 

It is a classical problem in computing, and you are intuitively solving it the right way.

 

-Paul

 

 

[blownsi]

Just realized someone had already made my point... about running multiple databases instead of multiple computers.

 

 

[Notung]

Hi Sanjay,

 

J. River MC supports 500.000 tracks. I know a person in Spain with 300.000 classical tracks in a 12TB NAs working without any problem. He has a Pentium Core Duo with 4GB or RAM.

Sonata is not a classical database it's a music server software that it's based on J. River MC player and GUI. SonataDB is the classical database.

You can visit J River Forum for more info on their database specs.

Regards

 

[ciamara]

J River / Sonata sound like great options, as do some of the Linux propositions. However, I'm currently focused on an OS X solution (without Parallels etc.).

 

It seems to me that random read/write to the database file (and the XML file) is the culprit. I generally like to keep the library files and the underlying music content together in a single folder, but I'd be willing to put the database files on a super fast SSD array and the music on a large array of spinning discs... and I may try this to see what happens. I just hate the idea of having to deal with backing up files from two separately locations and making sure they are in sync with one another. Curious to know what others do here, as there is certainly more than one way to skin a cat!

 

[PeterSt]

and the XML file

 

[ciamara]

[EDIT) Just realized I'm repeating myself, but ...

 

It is possible to lock the XML file so that iTunes does not write to it. The XML file does not appear to be required by iTunes itself. Rather, other applications read from the XML file (e.g. the DJ software, Traktor appears to). When locking the XML file, iTunes performance does appear to improve a little.

 

I am in the middle of a massive tagging expedition for multiple libraries, so I don't want to start moving library files around now for fear of linking the wrong files. But once this exercise is complete, I will see what happens if the library files are moved to an SSD array and the music left on a fast spinning disk RAID DAS. And I will report back with findings.

 

I'm hopeful (perhaps overly optimistic) that OS 10.7 Lion will maybe address this problem with a rewrite to iTunes. Certainly there must be enough consumers out there that are experiencing performance issues with large libraries ...

 

[1audio]

I looked into this quickly since SQLlite is used a lot of places. MPD also uses SQLlite to maintain a database of the available tracks. SQLlite's limitation that matters is below. I'm not a database guy and would not know how this limitation translates into a music management system. I am very interested in a solution for this problem (the practical issue of listening to 500,000 tracks or more is a separate issue).

 

Per the website http://www.sqlite.org/whentouse.html :

Situations Where Another RDBMS May Work Better

 

Very large datasets

 

With the default page size of 1024 bytes, an SQLite database is limited in size to 2 terabytes (241 bytes). And even if it could handle larger databases, SQLite stores the entire database in a single disk file and many filesystems limit the maximum size of files to something less than this. So if you are contemplating databases of this magnitude, you would do well to consider using a client/server database engine that spreads its content across multiple disk files, and perhaps across multiple volumes.

 

[PeterSt]

With the default page size of 1024 bytes, an SQLite database is limited in size to 2 terabytes (241 bytes).

 

An obviously unintended "typo" : 2^41 bytes ...

 

[mercer]

Hey, the new guy here!

 

This massive music library has been a problem for me for the last couple of years. Once I converted all my music to digital, I now sit on 650GB worth of music. I believe we can all agree the iTunes is simply incapable of effectively handling a collection this size.

 

I have, in the past, tried to 'split' my library into several directories - but in the end this is just not a practical solution. All my music is 'filed' as artist - album - song : with the only exception of 'Soundtracks and Compilations'. Just going through the collection to re-organize it would be a huge undertaking.

 

I believe it is time for me to create my own solution to this mess. I was thinking of a mySQL db and a web front end .... I have about 7 weeks off work this summer and I plan on spending some time on this every day until I have a solution. Any comments or suggestions are welcome.

 

Friggin' iTunes!

 

// Steve

 

 

[jhwalker]

I have over 500GB of music (45k tracks) and see *no* issues with iTunes - running on only 2GB of RAM on a 5-year-old MacBook Pro. Currently using 1% CPU and 155MB memory with this collection.

 

What kind of problem are you having?

 

[mercer]

I am running my library off an external drive and I find that my machine (both at home and at work - newest latest greatest) slow down noticeably when iTunes is running.

 

Not sure what the issue could be, but any suggestions are welcomed!

 

// Steve

 

[Paul R]

 

My full iTunes library is quite a bit larger than than 650gb and I have no troubles with it at all.

 

MySQL is a fairly decent choice, though to be honest, if your database is under 2tb, I would use the free version of Oracle or better yet, DB/2.

 

Or actually, the better solution is to not use a database at all, but use indexed files or the like.

 

-Paul

 

 

[mercer]

Should have mentioned both run Windows ....

 

[audiozorro]

I have about 7 weeks off work this summer and I plan on spending some time on this every day until I have a solution. Any comments or suggestions are welcome. [End Quote]

 

Please don’t spend your 7 weeks off work doing this. A 650GB music library is not a massive music library.

 

JRMC supports 500,000 tracks. Assuming 15 music tracks on each CD that is roughly 33,000 albums. And assuming each album CD is not full and only 400MB that is roughly 13.2TB or 20 times your modest 650GB music library.

 

iTunes should be able to handle 1,600 albums with no problem. If not it’s time to switch to a capable player.

 

And it's perfectly acceptable to use multiple music players and accept the fact that not all music players sound the same or are as capable in managing large music libraries.