Craig,
Thanks for the file, as usual, files are providing information to share the needed information.
Please find in your Cineversity-Private Message a link with a download.
The file was small (504x600 pixels), just 8bit/channel and the “transparent” part was premultiplied. You can see the background in it, but Photoshop shows an premultiplied image. This information is then used twice on the image in C4D, doubles the mask.
That is not nice, but it certainly only added to the problem. There was just too much blue-spill from the sky in the image. If you open the image in Photoshop and use a Image>Calculation on it, e.g., green and blue channel, multiply or difference, pretty much any blend-mode shows that the mask has a lot of problems, even outside of the tree.
To just save the alpha and grow the values back to opaque, for a proper masking, wouldn’t work here at all.
The blue spill needed to be selected and tinted (or masked out, I tinted it, see folder)
After some adjustments in the mask (levels, dodge and burn, as well brush work) the alpha was stored in the psd file.
I took a small sample of the green, average blurred it and used it as background, to avoid fringing. A merge/option visible gave me the resulting layer.
There is more about textures and use of images, see link below [signature] , that I do not repeat all of that again
I hope the tree is working now, as much as it can—with the low res.
To have transparence in the layer is normally suggested to be avoided, it will bite, soon or later.
All the best