To remove points off your licence do you send the licence to dvla swansea? And how does it take for them to return it?

Yes, you will need to fill in a D1 form (application for a new licence which you can get from the post office) and send this along with your photocard and paper counterpart to the following address: DVLA, Swansea, SA99 1BU

There's a fee of £20 which you will need to pay as well.

Provided the points have now expired, you will get your new licence within 3 weeks. But you can continue driving while waiting for DVLA to send back your licence as long as you don't have any restrictions imposed on you.