I assume you are talking of the actual borders of the warped images, not
the contents of the images, right?
Indeed PTGui doesn't antialias the alpha channel (and hence the edges of
the image). The reason is a bit technical but it comes down to problems
that would otherwise occur due to the negative coefficients in
interpolators such as Lanczos.
In normal panoramas the blender makes the edges invisible, but if you do
those non blended 'photomontages' the image borders remain visible. The
only solution I have would be to stitch at twice (or more) the desired
output size, and then reduce the size in Photoshop afterwards. This will
anti alias the edges.
> I'm using PTGui of the v.7 flavor to "flat stitch" photos and give
> them the joiner look. If I set the lens properties correctly and keep
> the optimization from looking for corrections, I get just displacement
> and rotation. It stitches (and layers) a really nice image, if I want
> to crop it.
> The problem I'm having is that, regardless of the interpolator I
> choose, there are still jaggies at the rotated image edges.
> Is there a way to antialias the edges of the rotated images?