Notifications
Clear all

Generating Unfiy patches from fxp and aupreset files

17 Posts
2 Users
1 Reactions
147 Views
kevleyski
(@kevleyski)
Honorable Member
Joined: 3 weeks ago
Posts: 13
Topic starter  

Hi, loving Unify on my Mac where I can now switch things to VST and share across platforms and DAWs and actually find the patches again 🙂

I’m keen to put some code on GitHub that will listen for changes to my instrument patch folders and auto update Unity patches, hoping to hack up a tool to convert folders to guru zip files from them for sharing on this forum

Can you guide me into the open source Unify patch format please so I read and write the parameters for each patch file

Kev


   
Quote
(@getdunne)
Illustrious Member Admin
Joined: 5 years ago
Posts: 4487
 

Hi @kevleyski. The patch format is described at https://github.com/pluginguru/unify-batch/tree/main/Documents

You may find the whole https://github.com/pluginguru/unify-batch repository worth a look. Just be aware: converting plug-in preset files to corresponding "unified" patches is NOT simple. It generally involves some fairly deep reverse-engineering, and the process will be different for different plug-ins. In that GitHub repo, you'll find several examples of how I've done this.


   
ReplyQuote
kevleyski
(@kevleyski)
Honorable Member
Joined: 3 weeks ago
Posts: 13
Topic starter  

Cool, I got things working thanks for showing me where to look

If useful to others I've converted a few things to build for macOS, and added a U-he Diva converter (basically a quick Zebralette rehash) I recon we can refactor this to a "Uhe Unfied" maybe, anyhow I managed to grow my database by 8,000 patches in an afternoon! Thanks

https://github.com/kevleyski/unify-batch

(PR coming)

This post was modified 2 weeks ago 2 times by kevleyski

   
ReplyQuote
(@getdunne)
Illustrious Member Admin
Joined: 5 years ago
Posts: 4487
 

@kevleyski

This is excellent progress, Kev. Our forum software doesn't allow new users to post links. I just upgraded your account, so you should be able to go back and edit your comment above (if you wish), to put the link back in.

Any refactoring you can achieve would be most welcome. As you can probably guess, I hack these projects together on a very ad-hoc basis. Any effort to bring order to the chaos would be very helpful.


   
kevleyski reacted
ReplyQuote
kevleyski
(@kevleyski)
Honorable Member
Joined: 3 weeks ago
Posts: 13
Topic starter  

G’day Shane

Native Instruments Kontakt nki files have be a little stumped when batch transforming them to Unity files

Clearly Kontakt VST3 instrument can be saved and restored successfully in Unify but bundling only the nki file as the binary base64 isn’t the whole story when I hex compare the diff between my patched .unify file and the one Unify made it (only part of it)

I tried saving Komplete Control instead and that seemed similar, i can us the same header but if I only take the nki change it can’t find the sample files it seems

Just wondering what I’m missing, eg is Unify saving a rack rather than just nki preset maybe?


   
ReplyQuote
(@getdunne)
Illustrious Member Admin
Joined: 5 years ago
Posts: 4487
 

Posted by: @kevleyski

Just wondering what I’m missing, eg is Unify saving a rack rather than just nki preset maybe?

Unify saves blobs of binary "state" data received from the plug-in. These are completely opaque--there are no standards--every plug-in is different. Unify knows nothing at all about how individual plug-ins manage their own presets. In a few cases I have been able to reverse-engineer state-blob and preset-file formats well enough to write programs that generate them by processing preset files (as you've seen in my GitHub repo), but this tends to be a laborious, error-prone process.

Don't waste your time trying to reverse-engineer presets for Kontakt versions later than 5.4. They switched to proprietary encryption after that. GUI automation tools such as AutoHotKey for Windows are the best approach we know of for unifying Kontakt and Reaktor libraries.

 


   
ReplyQuote
kevleyski
(@kevleyski)
Honorable Member
Joined: 3 weeks ago
Posts: 13
Topic starter  

@getdunne ah totally makes sense, so just persisting what’s given back (presumably all DAWs are like this too), actually I thought that might have been the case but hoping there was a trick 🙂

I noticed a small change to NI plug-in results in a completely different nki binary - that’s not great for comparison, effects processing decisions etc so I think long term this won’t be good for NI and maybe they will rethink this decision and correct that to be a checksum instead

At very least provide a mechanism to unlock if you have the (likely just AES) key to something I purchased outright I’m not renting them through subscription etc so this DRM is not fair on any of their users

… but no worries will automate instead 

This post was modified 2 weeks ago 2 times by kevleyski

   
ReplyQuote
(@getdunne)
Illustrious Member Admin
Joined: 5 years ago
Posts: 4487
 

Posted by: @kevleyski

...this won’t be good for NI and maybe they will rethink this decision and correct that to be a checksum instead

Unlikely. NI is run by private equity now, and it's been years since they cared much about anything.

 


   
ReplyQuote
kevleyski
(@kevleyski)
Honorable Member
Joined: 3 weeks ago
Posts: 13
Topic starter  

@getdunne yeah double edged sword give folks a reason to _not_ buy Kontakt instruments for example future DAWs suggesting and providing changes to plugins, they will either walled garden themselves in and hope they win on numbers or secure the plug-in instead like everyone else does

Maybe Steinberg is thinking about this too 


   
ReplyQuote
kevleyski
(@kevleyski)
Honorable Member
Joined: 3 weeks ago
Posts: 13
Topic starter  

lol, adding Serum batch converter for the win! I got 11,000 Unify files from my existing collection, at some point I’ll hit all permutations and will need a Unify for Unify.

As an idea a simple threshold “it’s basically the same” button to identify near duplicates in the database and tag/change their library/category would be a great feature in Unfiy!

Vital didn’t seem to work but hey I have the sources for that so I’ll add a converter for their blob (let me know if you already did)

As a thought we could separate that part and add an instrument patch to blob converter C++ class? That way anyone could maybe init a new Unify patch later by drag and dropping the original instrument patch onto Unify plug-in itself? That would be neat

(well, the already friendly plugins anyway and we can call out to plug-in vendors for those that need a solution which might well be you upload send _them_ your preset to be converted online and you download the unify files/guru bank, hopefully for free but not necessarily)

- the community can likely help here too, I’m kind of hoping/assuming some are actually already here, Unify is very cool and a great use case for their work and makes for a great paid and free preset shop format too IMO)

This post was modified 2 weeks ago 3 times by kevleyski

   
ReplyQuote
(@getdunne)
Illustrious Member Admin
Joined: 5 years ago
Posts: 4487
 

Posted by: @kevleyski

lol, adding Serum batch converter for the win! I got 11,000 Unify files from my existing collection, at some point I’ll hit all permutations and will need a Unify for Unify.

Unify's patch browser isn't really up to the task of wrangling tens of thousands of patches. If you're interested, I can tell you how to create a separate patch-browser app which can read Unify's patch database and send it OSC remote-control commands to open selected patches.

As an idea a simple threshold “it’s basically the same” button to identify near duplicates in the database and tag/change their library/category would be a great feature in Unify!

Unfortunately not at all "simple" because if the only differences are within third-party plug-in state-blobs, the code would need to know how to parse every state-blob from every plug-in ever made (or ever to be made in future)--not gonna go there. Quickly checking for state-blobs that are simply not identical (e.g. using checksums) would flag every patch in a unified collection as "basically the same", so not very useful.

Vital didn’t seem to work but hey I have the sources for that so I’ll add a converter for their blob (let me know if you already did)

I don't recall having built any auto-unifier code for Vital. I had a quick look and didn't find any. However I checked using the test() code in the GForce XML Unifiers project, and the state-blob payload for the Vital VST3 appears to be JSON, wrapped in the binary "FbCh" (chunk) format used by VST2 plug-ins. The preset files themselves are just JSON, but I have code to do the FbCh wrapping, and the GForce Xml Unifiers project has code to wrap again for VST3.

 

As a thought we could separate that part and add an instrument patch to blob converter C++ class? That way anyone could maybe init a new Unify patch later by drag and dropping the original instrument patch onto Unify plug-in itself? That would be neat

See my comments above about parsing state-blobs. Same issues apply, hence not practical. This is why I always build auto-unifier code as separate programs.

(well, the already friendly plugins anyway and we can call out to plug-in vendors for those that need a solution which might well be you upload send _them_ your preset to be converted online and you download the unify files/guru bank, hopefully for free but not necessarily)

- the community can likely help here too, I’m kind of hoping/assuming some are actually already here, Unify is very cool and a great use case for their work and makes for a great paid and free preset shop format too IMO)

What I would prefer is to define a generic, standard "please load one of your own presets" state-blob that can be accepted by any plug-in. They payload would basically be a sequence of one or more character-strings, to identify a preset according to its name, and optionally the names of any containing folders in a hierarchy. If we could get plug-in vendors to sign on to that (i.e., make their plug-ins accept such state-blobs), unified patch libraries would be trivial to make (requiring almost no reverse-engineering), and the patch files would be tiny.

Good luck getting vendors to agree to that (or indeed anything) but hey, a man can dream...

 


   
ReplyQuote
kevleyski
(@kevleyski)
Honorable Member
Joined: 3 weeks ago
Posts: 13
Topic starter  

@getdunne yeah my theory is make it their idea rather than ours, a collective easy standard way to describe patches for any plugin

An extension of what VST kind of already is. If params were just a key value store and VST provided that store that would have been good design

Samples go here

License Key goes here which would unlock the samples but can also act as checksum to allow the KVS to load (that protects the patch from sharing or at least watermark trace it back to order)

The instrument preset/snapshot class to serialise the blob won’t work in all cases but would in many and vendors might be more open than we think to provide assistance 

But yeah Unify as the shareable metadata and mappings to macros and so on (as you have it - it’s great) is the first step. I’ll ask Apple about it anyway

But yeah dreamin

This post was modified 2 weeks ago by kevleyski

   
ReplyQuote
(@getdunne)
Illustrious Member Admin
Joined: 5 years ago
Posts: 4487
 

Posted by: @kevleyski

If params were just a key value store and VST provided that store that would have been good design

The VST SDK does support a standard key-value format for presets. So do the SDKs for VST3 and Apple Audio Units. The problem is that non-trivial plug-ins require more than just key-value pairs, so basically nobody uses this approach anymore.

Please note I edited my earlier response above, to clarify the format of Vital's state-blob.


   
ReplyQuote
kevleyski
(@kevleyski)
Honorable Member
Joined: 3 weeks ago
Posts: 13
Topic starter  

Cool yeah it’s a bit LibreOffice ambitious to think everyone would just support the open standard for all params I guess.
Some of this will be efficiency of splitting multiple key values vs ease of serialising sub blobs etc I bet there is a huge amount of crossover and wheel reinvention though 

(just trying to find some incentive) definitely a agreed shared way to search patches based on the plugins you own across say Logic library, Komplete, Audio Labs, ADSR online or Unify and get same aggregate results (maybe with audition too) would be magic

At the same time Unify isn’t a really the way to find/discover patches at all, it’s just a way to bring them together - so yeah going in with the plug-in loaded or the OSC method that grabs from external is a good way forward

I’ll look at the OSC method

This post was modified 2 weeks ago 4 times by kevleyski

   
ReplyQuote
(@getdunne)
Illustrious Member Admin
Joined: 5 years ago
Posts: 4487
 

Posted by: @kevleyski

I’ll look at the OSC method

See the page page on OSC Support in Unify in the manual, and note that the patch database is the file Presets.db which lives in the main Libraries folder. It's a sqlite3 database, for which there are many language bindings, e.g. SqliteCpp.

 


   
ReplyQuote
kevleyski
(@kevleyski)
Honorable Member
Joined: 3 weeks ago
Posts: 13
Topic starter  

@getdunne Ah sqlite3 my old friend, have ported that to a few devices in my time

Hmm yeah I think I grabbed the wrong XML, it did work after all, Vital JUCE and just uses the same MemoryBlock write JSON as string to fiie - I did spot the caveat about the free plugins distro, not entirely sure why that's important as there are a bazillion copies in the wild and GPLv3 derived but I'll have to keep my guru private I guess... that said whats check into my fork does work - you'll get 239 Vital presets that work as a Unify layer you can add to Unify from the database if you run it yourselves

This post was modified 2 weeks ago by kevleyski

   
ReplyQuote
(@getdunne)
Illustrious Member Admin
Joined: 5 years ago
Posts: 4487
 

Posted by: @kevleyski

I think I grabbed the wrong XML...

I made that mistake myself at first. The Unify patch.xml file used in that project (made by stripping the binary header/trailer from a .unify patch file) must be imported into the code using the JUCE Projucer on the project .jucer file. Just re-running the old IDE project will reuse the previous XML. You could rewrite the code to load Unify patch.xml from disk at runtime; I haven't done this because the present method works well enough, and my goal is to achieve a result, not publish tutorial code examples.

 


   
ReplyQuote
Share: