That's ridiculous. If you use even 1/4 of that 4GB swap on a system with only 5GB RAM then you'll be swapping so much that the system will grind to a halt.
Besides, the way you have your system set up makes it very tricky to change it now. The majority of your 128GB disk is set up as a single linux filesystem, not using LVM. To expand your current swap partition you would need to reduce the size of the filesystem on /dev/xvda1 - which if it's using xfs (the default in CentOS 7, is impossible without a reinstallation/reformat). If it's using ext4 then the filesystem can be shrunk but you will need to boot from alternate boot media to do so as you cannot shrink a mounted ext4 filesystem and you cannot umount your / filesystem while the system is running. Once the filesystem is resized to be smaller than it is now then you would need to use fdisk to delete the current /dev/xvda1 partition and then recreate it starting on the exact same sector number that it starts on now but with a smaller ending sector. It is extremely important to do those steps in the right order - first resize the filesystem and then resize the partition. If you do that in the opposite order then you will damage the filesystem. Then you'd need to use swapoff to stop using the current swap partition and also delete and redefine that too, then mkswap it to make it usable. If the swap partition is referenced in /etc/fstab by uuid or label then that would need to be changed before you rebooted.
My suggestion was based on looking at the numbers you posted which show that you are actually using a little more than 1GB RAM. Then there is nearly 3.5GB used for filesystem cache and that's where the majority of your memory is being used. That's something that can be tuned and the swappiness parameter is the way to do that. The file /usr/share/doc/kernel-doc-3.10.0/Documentation/sysctl/vm.txt (part of the kernel-doc package) has this to say about it:
This control is used to define how aggressive the kernel will swap
memory pages. Higher values will increase agressiveness, lower values
decrease the amount of swap. A value of 0 instructs the kernel not to
initiate swap until the amount of free and file-backed pages is less
than the high water mark in a zone.
The default value is 60
In my tests I don't think 60 is the default on RHEL/CentOS kernels, it appears to be 30. You can reduce how much swapping goes on by reducing the size of the number and in my tests, 3 appeared to be the magic number where it stopped using so much. Yours may be different.