CentOS

My VPS provider has disabled my service, because they say that my server is sending SYN FLOOD attacks.
The server has been running quite happily for 3 years. CSF/LFD are loaded.

I now have only console access to the system, to try and 'clean it up'

How would I start this? How can I find a rogue process? I don't even know where to start here, so need some help!

I guess ss -ipt and look for high sends with low receives. The nethogs program could display the top used processes (network wise). Use ss -ntap
and look at the State field. Personally I'd start with ps and look for "strange" processes.

Although if you have been compromised you don't know if you can trust any of the tools on your machine.

Also I'd expect something like:
kernel: possible SYN flooding on port X.
to be logged.

I think that's more likely to be seen if you are the target of a syn flood attack rather than being the one doing it.

If your system has been compromised the only way to 'clean it up' properly and know that it is safe is to create a fresh OS install and copy your applications and data across. There are so many backdoor apps that hackers can use that it is very hard to trust your machine once it has been compromised.

You could try an online penetration test such as https://pentest-tools.com/network-vulne ... ne-openvas to see what it finds.