Cam bearings always look worn on LS engines with some miles. If the camshaft wasn't all scored up and you can't feel any huge ridges on the bearing, they are fine. Most guys will say if you look at the cam bearings, they will look bad, so just don't look at them and slap the camshaft in there anyway lol.
If the cam retainer plate in new and the o-ring and pump are new, then I guess it could be leaking out oil pressure from the VLOM area if you deleted that too. Or maybe the new oil pump or o-ring is bad out of the box.