Turns out its coming through some uncaulked holes in the gunwall, behind the rub rail, in the port stern corner. On days when the winds are from the north the swells splash enough water up to the gunwall and the water trickles in.
I have found that its not unusual to find uncaulked holes in the gunwall behind the rub rail. My last two Whalers both suffered from it, with the last one having a total of 27 uncaulked holes. They use drill tip screws that don't work well and result in many stripped out screw holes which they don't seem to thiink need to be caulked.