Rebuild patch db selectively

Most people only need to rebuild the DB when a new library is added. Even with approximately 3,000 patches, it only takes my rusty, old 2010 Mac Pro 5,1 about 10 seconds to do a complete rebuild. Are you having issues where you need to rebuild the DB more frequently or that it's taking too much time complete?

@karlfranz

I have Unified all my O2 libraries and libraries from some other synths as Breaktweaker and Kontakt. I am using some scripts I have develped to automate the process and it requires some trial and error 😀. Hence the reason to rebuild more often than normally. At the moment I have almost 9.200 .unify files in 42 libraries. At the moment it takes 2 minute and 15 sec to rebuild the db. I am totally aware that this not is an important feature so it is not a big deal. 😊 

Optimized database rebuilding is coming in Unify v1.2.x

@getdunne

👍

Posted by: @thsve

@karlfranz

I have Unified all my O2 libraries and libraries from some other synths as Breaktweaker and Kontakt. I am using some scripts I have develped to automate the process and it requires some trial and error 😀. Hence the reason to rebuild more often than normally. At the moment I have almost 9.200 .unify files in 42 libraries. At the moment it takes 2 minute and 15 sec to rebuild the db. I am totally aware that this not is an important feature so it is not a big deal. 😊 

Sounds like you've been experimenting with optimizing the database to better serve your needs just like I have. I wonder how many of us are out there. 🙂 

@karlfranz, @thsve

I've been talking with John about the idea of somehow allowing users to import/export/share databases, or indexing schemes/rules, etc. It's not obvious how this ought to be done, but I'm open to suggestions.

@getdunne

Just tested the Rebuild in 1.2. If I just click rebuild with "All Libraries" selected, it looks like the same behaviour as in earlier version(i.e takes the same amount of time). If I do the same with a specific library choosen, UNIFY becomes unresponsive for quite a while and my machine hit almost 20% CPU. No indication in UNIFY what is happening.

I've run both the test above when the Db was freshly renewed and no changes was made.

@thsve This is unexpected, and does not match the behavior on my Win10 machine at all.

What kind of drive do you keep Unify's main data folder on?

@getdunne

Internal Samsung SSD 860 QVO 2 Gb.

@thsve

Fascinating, because my PC also has Samsung SSDs, although mine are the older EVO 840s.

Roughly how many libraries/patches are in your database, and what kinds of rebuild-times are you experiencing?

2036 .upf-files and 9360 .unify-files, so quite a lot 😎 .

Rebuild All libraries (no option click) ca 1m 25s

Rebuild, PlugInGuru O2 Kolours for Keyscape selected (no option click) ca 1:25 

I tested rebuilding several times choosing different libraries and the time is the same every time.

In the performance monitor cpu is around 20% and almost no disk-activity at all.

@thsve

Thank you for the details. 1:25 is tolerable time, but should be faster if no files have changed.

Let's keep an eye on this. I'll be tied up updating the Unify manual for the next week or so, but I'd like to pursue this with you after that.

@getdunne

This is not important as it does not affect the basic functionality. As far as I am concerned, you can put this aside for now. I suspect you will have plenty to do anyway for a while now😅.

My Dell Precision Workstation computer has 3 SSD drives. One for the program files, one for samples and patches, and one for projects. 

I have 32 libraries in Unify. Using the stand alone version, it takes 2:40 to do a full rebuild. However, it takes only 15 seconds to do a conditional rebuild, whether I have a library selected or not.

@getdunne
One suggestion. The tool tip on the lightning bolt says to press the Option key and click to do a full rebuild. I happen to know the equivalent on a PC, but some people may not. Perhaps you can conditionally change the message based on whether it is running on a PC or Mac, or include both in the message, i.e. Option/Alt.

-- Martin

Posted by: @mschiff

The tool tip on the lightning bolt says to press the Option key and click to do a full rebuild. I happen to know the equivalent on a PC, but some people may not. Perhaps you can conditionally change the message based on whether it is running on a PC or Mac, or include both in the message, i.e. Option/Alt.

Thanks for spotting that, Martin! That's a bug, which I'll fix ASAP. I just checked the code and realized I got it backwards, so PC and Mac users will be equally confused!

@getdunne

Been there, done that. (got the logic backwards). 😉

-- Martin

@getdunne

Hi,

Sorry to bother you again with this, but I can't really drop it 🤗 . According to others' posts in this thread, it seems that this is a problem in my machine only(?) and I am anxious to find the cause. I turn to you who know the mechanism behind this functionality and wonder if you have any troubleshooting tips?

With me, there is no difference in processing time, regardless of whether I select "All Libraries" and just click or alt + click. In addition, it takes the same time if I choose an individual library and do the same. The only difference is that when a specific library is selected, the entire UNIFY freezes (unresponsive) until it is ready, while when "All Libraries" is selected, you see the text "Building patch database ..".

I have checked the disk and it is ok. I have created a totally new presets.db.

Any ideas?

Again, if you are busy with other pressing issues, just skip this. I'll survive 🤗 .

@thsve

I had also experienced cases where the "Building patch database..." text did not appear, but couldn't nail it down. I can now confirm your observation that this happens when a library has been selected before rebuilding. This should NOT happen, but the program logic which governs this is quite intricate and hard to get exactly right.

I added some details about the difference between Full and Incremental rebuilds to the Unify manual at https://pluginguru.net/unify/manual/doku.php?id=playing-patches&s []=sidebar#about_the_patch_database.

Whether a Full or Incremental rebuild is performed is determined only by whether or not you hold down the Option/Alt key while clicking the lightning-bolt icon; the current library selection has no bearing on this. Unify attempts to re-select the same library and patch after rebuilding, as had been selected before rebuilding, but this should have little effect on the total time.

How long these operations take is affected by many things, and each user's results are likely to vary. I timed one Full rebuild at 1:21 (one minute 21 seconds), then immediately triggered a second Full rebuild and it took only 0:36. This is due to most of the required data still being in RAM (due to the way modern operating systems work). I then did an Incremental rebuild and it took only 6 seconds, probably because all the required data were still in RAM. But these are unusual cases; normally you would rebuild so infrequently that none of the required data will remain in RAM, and hence the timing will be dominated by disk-read operations--the so-called "worst case".

All that said, I'm quite surprised that Full and Incremental rebuilds are taking the same amount of time on your system, because Incremental rebuilds should involve much less disk I/O. Let's keep discussing and try to narrow this down.

@getdunne

Thank you for taking your the time to do this. First, I had misunderstood "Incremental", I thought it was per library. So we can put that problem aside for me. But what differs from mine machine to yours is that it does not matter when and how I run the db-update. It takes the same amount of time regardless of whether there are patches changed or not or if I run directly after an update without any changes. It takes about 1.5 minutes regardless. It is not the time itself that is an issue for me but that my machine obviously behaving differently than expected.

I can very well understand the complexity of the code because I have a bit of my own programming background. I've even been inspired by you and John and started looking at the JUCE-framework. Let's see if I eventually can put something together. I am still early on the learning curve 😎 .

Keep up the good work. I am very impressed of what you have achieve so far!

@thsve

You have over 11,000 patch files. It seems reasonable that just checking the dates on that many could take a minute and a half. One test you could try is to move some library folders out of the main Libraries folder and rebuild a few times after that. (The very first rebuild will find a lot of changes, so don't go by that. See what happens when rebuilding after no file changes. If the time taken varies roughly linearly with the total number of .upf/.unify files, you'll have your answer.)

@getdunne

Have tested after removing a number of libraries and then I get a big difference between full and incremental update 😀 . Obviously, this difference decreases with increasing number of patches. In my case (about 11000 patches) there is almost no difference at all. At some point, if you are still thinking about changing the update functionality, at least I would think it would be good to be able select and update a specific library. But like I said before, this is not important in any way. At least now I know that it does not seem to be my environment that depends.

@thsve

Very interesting, thank you for checking. I'll keep these results in mind for future updates.

Similar problem here - I am not seeing the "in progress" message at alll now.  Just a period of unresponsiveness.

Just to add to the discussuion, and following comments on the livestream on Sat, I thought I would report that on my machine a "quick" rebuild (i.e. no alt/option pressed) takes just over 7 minutes, when there have been no changes to any library, and nothing else is active on my machine.  I have about 35k patches to check, as I have most of the Unified libraries that exist, but surely that is the point of Unify?  My machine is a 3.4 GHz i7 with 32 GB RAM and the libraries are on a fast SSD.  This means that installing a new library is a tedious job - as others have said, go and make a coffee while it installs.  If there was a way to just check the patches of any one library, or a new one only, that could save time?