Clicks and Pops, Pianoteq v8.1.1 continued here after FB users group post

@dolmensdude's original questions on Facebook:

Pianoteq v8.1.1 not playing nice anymore with Unify. Lots of clicks and pops, Pianoteq thinks it's CPU overloading, but it's limited to only Pianoteq. Used to even be able to use multi-core in Pianoteq inside of Unify, but even if turning that off... still goes south.

 

Pianoteq features largely in my live stuff, inside of Unify.

 

It's only this, and happening on both my Apple Silicon and Intel i9 maxed out machine. Might be for a couple of weeks...

Unify creates a separate thread for each layer. This is a bit intricate (click on the word "thread" for a Wikipedia article which covers the basics), but basically, it's a program construct which allows the operating system to spread the execution load across multiple CPU cores. Each core can execute more than one thread (layer), and a given thread may bounce around from one core to another over time, but unless a program or plug-in declares more than one thread, its execution cannot be split across two or more cores at the same time.

Prior to Apple's introduction of "Apple Silicon", all Mac and Windows computers were built with Intel processors which had multiple identical cores. These are called Symmetric Multi-Processing or SMP machines, and Unify runs brilliantly on SMP processors.

Apple's newer processors feature a mix of "efficiency" and "performance" cores (aka E and P cores), which are not identical. The E cores are slower, but consume very little power; the P cores are just the opposite. The MacOS operating system tries to run as many threads as possible on E cores (which in the latest M-series CPUs are still quite formidable), and as a result, provide excellent performance AND surprising low power consumption. Not to be outdone, Intel's latest "Alder Lake" CPU designs also feature a mix of E and P cores, and along with Apple's M-series chips, are referred to as Asymmetric Multi-Processing or AMP machines.

Thread scheduling on AMP systems is complicated. Each program must provide much more information to the operating system than just a rough "priority" number for each thread. Threads which must synchronize well with specific hardware (such as the audio I/O system) must declare that fact to the operating system, along with details about how much time they require to run, and how often.

Unify is built on a cross-platform code base called JUCE, developed by a small group of extremely clever people in the UK, and also used by a great many other plug-in developers. It allows me, and all of these other developers, to write code once, and have it work more or less identically on Mac and Windows (and in some cases other platforms). JUCE is basically a "thick layer" of code between my code and the operating system, which includes all the necessary platform-specific intricacies necessary to, e.g., render graphics on the screen, schedule threads, and so on.

And there's the rub. JUCE's thread-management scheme was built for SMP machines, and does not allow me to specify the intricate details needed to achieve top performance on AMP machines. I have been waiting for over a year for them to update the JUCE framework to take care of this, and I'm reluctant to try to do it myself, because it's difficult, requires a lot of detailed knowledge I don't have, and ultimately is what I pay them for. The JUCE team, as brilliantly capable as they are, are still only human, and this is only one of a great many things on their plate. So we wait. Other developers have "bitten the bullet" and devised their own thread-scheduling solutions, and I may have to do the same.

What we have seen on Apple's M1 Macs is that Unify's layer-threads get assigned to E cores, and only occasionally get bounced to a P core. This is why you observe "spiking" on Unify's CPU meters. Since the E cores are slower, they "max out" sooner; when this happens, you'll see a big spike, often accompanied by an audio glitch because the E cores aren't fast enough to process all the layers before the audio hardware needs its next buffer-full of samples. Unfortunately, the effect turns out to be worst on the top-of-the-line M1 Pro and Max CPU's, because they have fewer E cores than the cheaper ones do (2 instead of 4). Based on @dolmensdude's report, it appears Intel's new Alder Lake CPUs suffer from similar effects.

All I can say is, this situation is just as frustrating for me as it is for you. Eventually, we will have a solution, but in the meantime it's embarrassing. Please bear with us.

@getdunne brilliant!

@getdunne Logic now lets us choose to only use P cores as a preference. Is there any way for the JUCE team to do the same? p core affinity, even lock is available to Apple internally, but I would have no insight to that being a private API or not. Since this affects all JUCE-based code, perhaps Apple would be willing to to allow others to follow suite?

nearly all of my findings are with Unify standalone, but I will try later to load unify as a plugin in Logic, where the setting for p-only cores is on, curious if that would apply to plugins “under/within” Logic as well (recognizing these would be separate codespaces but wondering if they inherit the p-only flag, which would be nice 🙂

My workflow is hunt and sound design in unify, save there and then load those into MainStage for all live work. Haven’t even gotten to that on this Mac Studio, but it’s not going out for any live shows anyway. 

I live in France so six or more hours ahead if you.

Posted by: @dolmensdude

↑

Logic now lets us choose to only use P cores as a preference. Is there any way for the JUCE team to do the same?

Of course there is, but it's very intricate. That's exactly what I and many other developers are begging the JUCE team for.

Since this affects all JUCE-based code, perhaps Apple would be willing to to allow others to follow suite?

Clearly, you've never dealt with Apple as a developer. They don't care.

nearly all of my findings are with Unify standalone, but I will try later to load unify as a plugin in Logic, where the setting for p-only cores is on, curious if that would apply to plugins “under/within” Logic as well (recognizing these would be separate codespaces but wondering if they inherit the p-only flag, which would be nice 🙂

Ordinary plug-ins, which don't try to create their own subsidiary threads, work perfectly in Logic, Mainstage, etc. The problem is how to "inherit the p-only flag" (it's nowhere near as simple as a "flag") for new threads, in a clean, platform-independent way, which is maintained indefinitely into the future.

I'm ex-Microsoft. I wore my Microsoft hat through D.C. airport the day after they walloped us with anti-trust charges.

There's usually a way to escalate, and it's always around finding "their interest, the thing that makes them want to act themselves on something"...

Not wanting to get into a blame game, or a why the heck can't the JUCE team get this resolved, when others find a way?  In my inbox today, Vienna Ensemble Pro *finally* went AS. Downloading now. I'll see how far they got with this... They were my "in-between layer of choice" before Unify, which tackles the SMP stuff *and* is quite a bit more which is why I made the move about a year ago to switch out VSL and use Unify, even on patches that didn't really benefit from anything else in Unify...  Done some crazy good stuff with Unify. No doubt.

Today's further testings. Unify in Rosetta mode plugs along like it always did. Didn't try everything, but no clicks and pops was nice.  Hosting inside of Logic Pro in P-only mode fell flat within seconds, even the sine wave synth cracked and popped.

Back to how to motivate Apple. Go to Tim. Of course he won't respond. There's a lot of glowing apple logo's on stages everywhere. JUCE versions of plugins surely too. All we're asking is that they document/share the P-only scheme so that JUCE can replicate that. Now *obviously*, Apple ain't interested in cross-platform concerns, so they shouldn't be wearing that on their sleeve when talking with them. 

Every Unify on Mac customer should also make a feature request/bug report stating this issue as well, same as for all JUCE guys.

Then it's in their interests, but not before then. All they do is prioritize feature requests and bug fixes into releases. If it ain't on a list, it ain't happening. It was that way at Mickeysoft too.

I *want*, heck, need you guys to be successful. You'll probably need to leverage us all to help move this forward, if in fact there is no other way than for JUCE team, while saying they want to stay platform neutral/independent then so be it, but regardless of how architecturally superior Unify on top of JUCE may be, it can't be clicking and popping into FOH.

I'm guessing, but seeing as I'm watching processes on the E cores and well, Logic offers P-Core affinity as a basic option - *they know* that's where these apps need to run, and that's why they offer this option.  It just makes sense.  Stating the obvious, the JUCE folks simply gotta understand that they need to develop a fix, and blaming Apple won't fix anything. 

I'm still here to help, as I think you guys are doing a great job on Unify. Really. I mean that from the bottom of my soul. "Losing" the AS hosts??? That's not a win for you guys, or us... so let me/us know who we need to lean into to get you out of these chains!!!

in the mean time... does putting out an Intel-only binary for the plug-in Unify help? The standalone app in rosetta mode is a little better yes?

I need the option to load Intel versions of the plugin of Unify in Logic and MainStage, or has in your testing, they shown the same results (whereas the standalone in Rosetta in a single try here was clearly better - probably 10 minutes of playing, including loading things like East West Play, the Spitfire Pro library and Pianoteq.

VEP was okay in a quick test. Not great, so not sure they've actually solved the problem, but I barely had a click or pop at 128 sample latency. In VEP, standard is 2 additional buffers, I took that down to zero and 1. Zero clicked now and then... 1 didn't...

MainStage shows only the P-cores when selecting how many cores allotted to MainStage.  There isn't even any kind of option that would allow the E-cores to be involved.

This is on the JUCE team it seems. Clearly a quandary. Sigh. Please, let me know who to "lean on". I will.

Posted by: @dolmensdude

This is on the JUCE team it seems. Clearly a quandary. Sigh. Please, let me know who to "lean on". I will.

I suggested the following text to another user, to post at the JUCE Forum:

I’m a long-time user of Unify, a JUCE-based combination host/plug-in from PlugInGuru. It runs poorly on my Macbook M1 Max, with a lot of CPU spiking. I have already complained to the developers, who have told me this only happens on the Pro/Max versions of the M1 chip, and requested that I report this directly to the JUCE development team. I see that this phenomenon, which affects other multi-threaded JUCE apps/plug-ins, has been discussed extensively on the JUCE Forum for a year, but there is still no official solution. I urge you to address this as soon as possible.

He dutifully posted here: https://forum.juce.com/t/mac-m1-thread-priority-audio-workgroups/52188/15

As you can see, there was no response. The JUCE team are past masters at ignoring people, and while I find it frustrating, I also fully understand. They are a small, poorly funded group, with a ridiculous amount on their plates. We're asking them to do exactly what I've been refusing to do: dig deep into new and still-evolving platform API's, devise and implement a reliable way to use those APIs to achieve certain results from platform-independent client code, and commit to maintaining that solution forever. They claimed to have a solution "still in testing" over a year ago. I'm prepared to believe them, despite the fact that they haven't yet committed it to the develop branch of their GitHub repo.

Your experience with Microsoft may indeed yield helpful insights, but understand there is a vast difference of scale. MS has over 100,000 developers and an R&D budget surpassing 27 billion dollars; the JUCE team has three or four guys. Holler and plead all you want--heaven knows I've tried. In the meantime, I'm biting the bullet and starting to figure this out myself. Maybe I can come up with a hack that will hold until the JUCE group finally deliver, at which point I can throw my work away, and rework Unify yet again to accommodate their new approach.

I've read thru some of the threads... yeah, my personal opinion is they are indeed, dragging their heels. 

In the mean time, I fired up Vienna Ensemble Pro Server (the Rosetta-specifically named version, not clicking enable Rosetta via Get Info). Loaded Unify plugin there, and have been loading up Unify like crazy. It's holding up. No clicks and pops.

I don't know if this is a workable workaround for all kinds of reasons including iLok. And I'm not sure how knobs and such, and MIDI learns would work thru an additional layer...  but for the audio bits??? It's holding up so far.  

Will walk away for a while and try another extended play through.

I posted up there today... 

I want them, or anyone else building on the JUCE platform that have a viable workaround in the absence of real support via JUCE, to peer-support you.

You make a great product, and it really does and enables some really, really cool stuff. Just needs to be usable in the AS world.

I tried to say I'm not complaining, but rather pointing folks at an issue and *encouraging* all to solve that issue.

I have no idea what the response will be, if any, and if there'll be any "blowback" or not.

Just need them to prioritize, is it's *their* platform, and *their interests too*.

Posted by: @dolmensdude

↑

I posted up there today... 

I want them, or anyone else building on the JUCE platform that have a viable workaround in the absence of real support via JUCE, to peer-support you.

Thank you, but I will not hold my breath. The VAST majority of JUCE users do not use multiple threads for audio processing, so they can't help. Most of those that do are unlikely to want to share their code (which provides competitive advantage for them).

The exception is "wavesequencer". He and I have been in touch, and he has tried to help, but his code requires JUCE 7 and I use JUCE 6. I tried back-porting, but it worked very poorly, so I have now started updating Unify to JUCE 7. This is a difficult process because they make many "breaking changes", and I have to port over about half a dozen custom hacks I have made to the JUCE files, or key Unify features will break. I'm about half-way through this process.

I saw his posts. I think his posts indicated he had a kludge for AU, but not VST.  Yeah, that's a thing.

Dunno if it's a v6 or v7 thing.  Surge? that was a JUCE based synth you guys say needs a non-JUCE version to work?  Well, my v6 download of that from their site is working fine, well, without clicks and pops.

Even the sine wave synth clicks and pops...  

If you want to have me try and x86-only non-fat binary (sounds like a Starbucks order now), I can see how that does...  

I'll be on that board again in a few days if no response whatsoever. That's not a fair response to you or your customers - and I'll continue to "encourage" until they enable you.

@dolmensdude

Surge is single-threaded, like most plug-ins. The problem is Unify itself, which is multi-threaded. Surge works fine in Unify (no need for a "non-JUCE version"), but there may be glitches on AMP systems, for the reasons I explained.

Go gently on the JUCE forum. Most people there are other developers, mostly indies, who have their own problems to worry about. The JUCE devs themselves are a tiny, under-funded group, with a gigantic number of issues on their to-do list. It's not their job to fix Unify. When they have a robust, well-tested threading solution which works on all known platforms, they'll publish it.

I get it. You deserve a response from them, that's all. Let's see what they do. I'm hopeful and will stay in "encourage" modus.