There is a special button in Xero for Overpayment by a customer:
Within the bank rec, go to the transaction and choose Create. Then click the "add details" at the bottom right of the blue box for the transaction.
Under Section 2 you'll see you have a section called Received As and a drop down - in there select Overpayment and the screen turns pink. Fill in the details and it attaches the overpayment to the customer's account so you don't lose track of it.
Then to record the cheque being paid back to them if you go into your Sales Invoices and look at Awaiting Payment you should see the Overpayment there as a credit on the account. If you click into it you can record the cheque payment at the bottom of the screen just as you can when paying a bill.