It used to be simple, if a publisher sends you a glowing letter telling you how wonderful your work is, how they'd be honoured to publish you, and how you just need to send them a cheque for hundreds of pounds for 'expenses' - they are a vanity publisher - avoid.
A vanity publisher appeals to your vanity, to the burning desire to see your work in print, and hopes you'll skip over the small print. A 'real' publisher will be offering you money, in the form of an advance, because they think your book is marketable and they'll be able to make a profit out of selling it. A vanity publisher has taken their maximum money off you, and doesn't need to try and help you sell the books, it's of no or little interest to them.
So where does self-publishing come into it? If you decide to self-publish - to get your own printed or e-copies out there by yourself, you are acting as your own publisher. The money you spend is about getting something physical back - either e-copies or an e-platform, or paper copies. It's then clearly up to you to market, promote and sell those copies, in order to earn back your expenses, and perhaps a little profit.
What has happened to the vanity publishers now, is they have noticed the rise in self-publishing and decided to get in on the act. They are offering an apparently transparent deal, physical or e-copies, perhaps with a bit of editing thrown in, but at a marked-up price on what is out there.
If you have decided to self-publish - you should visit this invaluable site run by Johnathon Clifford, and see if the good deal you think you're getting is in fact a scam to exploit the hopeful author.
I accepted the invitation by Austin Macauley to have my first novel published. After paying over 5k Australian, 3 years and complete inaction on their part, despite the fact my book was #1 on their Sci-Fi list over that time, there was no action by them. Sales were few and cheques too inadequate for Australian banks to honour.
Communication was via cut and paste, no questions were answered, and I was compelled to conclude that they are con-artists.
Stay away. Publish via KDP - at least that's free.