Yep, the valve arrangement is different, and you cannot use a 318 poly cam in a 318 wedge; the cam will physically fit in the block, but the lobes will be in the wrong place.
But THE BLOCK IS ESSENTIALLY THE SAME, with the same number of head bolts, located in the same place, and you can - and it has been done - swap "LA" wedge heads on an "A" poly block, and vice versa. You then have to use the corresponding intake manifold, cam, pushrods, rockers, pistons and exhaust that go with the heads.
You can believe it or not, I really don't care, but IT HAS BEEN DONE.
Understanding that what became the 318 poly was a Plymouth-designed engine, note that none of the foregoing applies to the Chrysler or Dodge poly engines; there are no wedge heads to fit either, although you can convert a Dodge poly into a Dodge hemi, and a Chrysler poly to a Chrysler hemi. (And you need to be aware that 55 and some 56 Plymouths used Dodge poly engines, as did some 57 De Sotos.)
You may think you know more about Mopar engines than I do, but you don't.