Thanks for the file, Shadowgin.
I hope I was able to set the scene as you would have, most of the things were switched on or off.
Anyway, the problem with Photoshop is, that it can’t handle two Color Profiles at the same time. It is either linear or gamma. In the moment you merge the RGB to the Buffer files it becomes problematic. The rendering in C4D is always 32bit/channel. In the render settings I got, the RGB content is 8bit/channel sRGB and the Buffer is 16bit/channel linear. Not something Photoshop handles nicely. You can have only one or the other, in hi end comp work we need mostly both, image values and data values. Hence my suggestion to do it in format that serves both well (see below). Another problem with Photoshop since ever is the premultiplied material handling, and in a way, this is what happens here.
In a nutshell, think of the edge of an mask as a gradient. With 8bit/channel you have a maximum of 256 steps to provide a definition to that, but it is gamma based, so not really a similar step by step. With 16 bit/channel that step problem vanish, but combined with a linear version of the same information is like a bow and a straight line, it doesn’t match. It becomes even more of a problem if the edge of an object is already mixed with another, a mask can’t separate the mix anymore, as it is a single value with no separation of object a or b. If the mask is used for color correction of a single object for example, then the mixed values of that area become a huge problem. The only way, is straight alpha, more about below. The problem with object to object edges is, that if you Anti Alias them, they mix to a certain degree. If you shrink now one mask and the other one as well, to avoid the mixed material, you end up with a gap between both. Either way you have a problem, with that render at once and avoid straight alpha. Straight Alpha can have both objects. Here is the take system a great solution, BTW, e.g., for Matte Objects.
The checkboxes for Multi-Pass>shadow correction might help a little as well, test this case wise.
As well as well Anti Alias> Consider Multi-Pass
====
My tip for absolute professional work would be always to stay in 32bit/channel and keep it as long as possible. If the file gets too big (PSD needs to be below 2GB) go to PSB, which has shown for me over the last decade to be a more stable format than PSD. (My image work is typically min 16K wide, files are between 1 and 20 GB). Render each part as Straight Alpha. I guess that might be not suitable to do it for many parts in a complex scene. But straight alpha gives you zero problems with seams and is considered industry standard in VFX feature film work, as pixel will be delivered full and color correction is possible for each part very precisely. If you like to dive deeper into this theme, Pre-Multiplied is the key word to look up in VFX standard books.
Going with RGB and Buffer the 32bit/channel route will allows for a more clean conversion to gamma or integer material, and the seam will get noticeable smaller. Oversampling as discussed later might help here. As a side note, pretty all my comp work is based on edge-glow or edge reflection. In a nutshell, I find the edge in a new mask, and blur it inwards the object. Then I take the BG/FG with the given mask/comp and blur this in an copy. Then I apply the edge mask with the blur copy as Light-Wrap on top of it, adjust the transparency and get a very natural handling of the seam. CG results are mostly un-natural. (Not my idea, but a typical technique in VFX) This would eliminate even more the problem.
====
A more practical way would be to render and save in 32bit/channel. (I write per channel, as some people add 8bit four times and call it 32 bit, as in “.hdr/radiance” which is nothing that I ever would suggest for print work for example, it is the least format to deliver color!)
Render it all in one file and in Photoshop go then to 16 bit/channel and Adobe RGB. I would avoid sRGB during production as much as possible, it is a delivery format in my book; It has the smallest color gamut possible.
The only downside, going with the full provided bit-depth, is that values in the scene should be well controlled, as Anti Alias is not really equipped that goes way beyond 100% for one object and not the other. If you need a super white background, perhaps do it in post (Ps) later, which gives you more control—needless to say that the Background Object has only limited influence on opaque objects anyway.
Another way, if the objects have such an strong contrast, you render with a Compositing Tag>Matte Object on.
====
The quick and dirty way would beset the 100% luminance value to 0% for one render and back to 100% for the other render, then pick what makes the most sense. I had no images for all these tests. I rendered each possible variation, as I do not just answer from memory, and r17 and Ps might have changed something, so, if all of that is not helping, please contact the support. http://www.maxon.net/support/support-questions.html
Quick and dirty part 2 would be oversampling, but with 6K already that sounds not reasonable. (E.g., overample to ~24K and scale down, after setting the colorspace for both correctly.)
My best wishes