After a lot of back and forth with VW of America and Colonial VW in Medford MA, they did replace some parts of the heating system. Dealing with VW corporate was actually a pretty nice experience, and they certainly helped moved things along. Colonial VW is another story. They made a lot of pretty odd mistakes throughout the process, though the tech that actually worked on my car was a big EV advocate, and seemed quite knowledgable.
I am waiting for the final list of what was done, but it looks like these two parts were the main issue:
Element - 5QE-819-013-P
https://parts.vw.com/p/Heater/64431585/5QE819013P.html
Auxillary Water Pump - 5G0-965-567
https://parts.vw.com/p/Volkswagen__/Aux-pump-Engine-Auxiliary-Water-Pump/52645875/5G0965567.html
It hasn't been as cold here as it was when I originally encountered the loud noise, but it has come close. And on those days it was freezing cold the heat was a good deal quieter than before.
I will update this thread if anything else changes.