[SOLVED] Create local repo

[D8TA]

I am looking at creating a local repo which will allow all my CentOS 6 and 7 servers to access and get patches and updates. I have been following a few different examples from various websites and I have the DVD ready to go but I would like to clone an existing mirror for all the updates. Most of the examples are referencing the rsync but the problem I have is that port isn't open on our firewall so I am trying to figure out if this can be done using http or https, maybe even FTP but SFTP should be open I think.

This was one example and then you set it up in cron to run and update accordingly.

Code: Select all

rsync -avz --exclude='repo*' rsync://mirror.cisp.com/CentOS/7/updates/x86_64/ /var/www/html/repos/centos/7/updates/x86_64/

I cannot get outside using rsync unless I can change the port. I know we allow 80 and 443 so that would be ideal.

Still learning all this but this would be a great win to prevent all the clients from having to go out to the Internets for updates if I can just have one server get them and then the others use this one to pull patches and updates.

[jlehtone]

I cannot get outside using rsync unless I can change the port. I know we allow 80 and 443 so that would be ideal.

You have to know more.

You have a process in your machine that tries to contact a (server)process in remote machine.
Telling the rsync to use some port will not help unless there is rsyncd at that port in the server.

Does your firewall filter which remote ports can be accessed? (An egress rule. Outbound connection filtering.)
What ports does the remote firewall allow access to? (Their ingress rule.)
What processes do listen in the remote machine for those accessible ports? This is a key point.

If you can use only protocol X (http, ftp, ssh, rsync, ...) then you have to use a client that supports X.

[D8TA]

Is it possible to sync my local repository from a mirror using wget or curl?

Looks like there is a reposync that I could use to sync a mirror to my local repository?

[D8TA]

Anyone?

[avij]

Yes, you can use wget and curl to mirror the repositories, but using rsync would be much much safer.

[pjsr2]

You can create and maintain a local repo with reposync. Reposync will use the URLs you have configured in your /etc/yum.repos.d/*.repo files.
Reposync is downloading files in parallel, so it is quite fast.

To create local repositories that can be served through your local web server (assuming you have a web server running with /var/www/html as the document root):

Code: Select all

sudo yum install createrepo yum-utils
sudo mkdir -p /var/www/html/repos/{base,centosplus,extras,updates}

Synchronize the directories for the repositories. Thew --newest-only option puts only the latest version of each package in the repos.

Code: Select all

sudo reposync --gpgcheck --plugins --repoid=base \
  --newest-only --delete --downloadcomps --download-metadata \
  --download_path=/var/www/html/repos/
sudo reposync --gpgcheck --plugins --repoid=centosplus \
  --newest-only --delete --downloadcomps --download-metadata \
  --download_path=/var/www/html/repos/
sudo reposync --gpgcheck --plugins --repoid=extras \
  --newest-only --delete --downloadcomps --download-metadata \
  --download_path=/var/www/html/repos/
sudo reposync --gpgcheck --plugins --repoid=updates \
  --newest-only --delete --downloadcomps --download-metadata \
  --download_path=/var/www/html/repos/

Create (or update) new repodata for the local repositories. Note the option "-g comps.xml" to update the package group information.

Code: Select all

sudo createrepo /var/www/html/repos/base/ -g comps.xml
sudo createrepo /var/www/html/repos/centosplus/
sudo createrepo /var/www/html/repos/extras/
sudo createrepo /var/www/html/repos/updates/

Add a new /etc/yum.repos.d/internal-repos.repo file on your clients to point to your local repository server. Give it the following contents, replacing myreposerver.mydomain with the name of your server:

Code: Select all

# File:/etc/yum.repos.d/internal-repos.repo
[internal-base]
name=CentOS Base
baseurl=http://myreposerver.mydomain/repos/base/
gpgcheck=0
enabled=1
[internal-centosplus]
name=CentOS CentOSPlus
baseurl=http://myreposerver.mydomain/repos/centosplus/
gpgcheck=0
enabled=1
[internal-extras]
name=CentOS Extras
baseurl=http://myreposerver.mydomain/repos/extras/
gpgcheck=0
enabled=1
[internal-updates]
name=CentOS Updates
baseurl=http://myreposerver.mydomain/repos/updates/
gpgcheck=0
enabled=1

You will have to synchronize your local repositories every now and then to get in updates and security patches. This is conveniently done by creating a cron job that executes the reposync and createrepo commands listed above at regular intervals.

[D8TA]

You can create and maintain a local repo with reposync. Reposync will use the URLs you have configured in your /etc/yum.repos.d/*.repo files.
Reposync is downloading files in parallel, so it is quite fast.

To create local repositories that can be served through your local web server (assuming you have a web server running with /var/www/html as the document root):

Code: Select all

sudo yum install createrepo yum-utils
sudo mkdir -p /var/www/html/repos/{base,centosplus,extras,updates}

Synchronize the directories for the repositories. Thew --newest-only option puts only the latest version of each package in the repos.

Code: Select all

sudo reposync --gpgcheck --plugins --repoid=base \
  --newest-only --delete --downloadcomps --download-metadata \
  --download_path=/var/www/html/repos/
sudo reposync --gpgcheck --plugins --repoid=centosplus \
  --newest-only --delete --downloadcomps --download-metadata \
  --download_path=/var/www/html/repos/
sudo reposync --gpgcheck --plugins --repoid=extras \
  --newest-only --delete --downloadcomps --download-metadata \
  --download_path=/var/www/html/repos/
sudo reposync --gpgcheck --plugins --repoid=updates \
  --newest-only --delete --downloadcomps --download-metadata \
  --download_path=/var/www/html/repos/

Create (or update) new repodata for the local repositories. Note the option "-g comps.xml" to update the package group information.

Code: Select all

sudo createrepo /var/www/html/repos/base/ -g comps.xml
sudo createrepo /var/www/html/repos/centosplus/
sudo createrepo /var/www/html/repos/extras/
sudo createrepo /var/www/html/repos/updates/

Add a new /etc/yum.repos.d/internal-repos.repo file on your clients to point to your local repository server. Give it the following contents, replacing myreposerver.mydomain with the name of your server:

Code: Select all

# File:/etc/yum.repos.d/internal-repos.repo
[internal-base]
name=CentOS Base
baseurl=http://myreposerver.mydomain/repos/base/
gpgcheck=0
enabled=1
[internal-centosplus]
name=CentOS CentOSPlus
baseurl=http://myreposerver.mydomain/repos/centosplus/
gpgcheck=0
enabled=1
[internal-extras]
name=CentOS Extras
baseurl=http://myreposerver.mydomain/repos/extras/
gpgcheck=0
enabled=1
[internal-updates]
name=CentOS Updates
baseurl=http://myreposerver.mydomain/repos/updates/
gpgcheck=0
enabled=1

You will have to synchronize your local repositories every now and then to get in updates and security patches. This is conveniently done by creating a cron job that executes the reposync and createrepo commands listed above at regular intervals.

Thanks pjsr2!! Great information and I'll give this a go today. Greatly appreciate your time in spelling this all out for a newbie like me.

[D8TA]

I have another question. Is there a way, well I know there is a way so let me rephrase. I need to use specific mirrors so I can work with our Networking guys to allow request to these and travel through our BlueCoat. They said I can provide them a list of several mirrors but they need the URLs and also to have the mirror check timeout in like a minute or something.

Apparently, I was running the

Code: Select all

reposync --gpgcheck --plugins --repoid=base \
  --newest-only --delete --downloadcomps --download-metadata \
  --download_path=/var/www/html/repos/

and the BlueCoat has the files sent to the AV device for security scanning. The fastestmirror plugin would then just try a different mirror and the BlueCoat queue started backing up causing some Internet traffic issues for people inside going to outbound addresses. So, I was going to provide 5 or so URLs for http mirrors from this list

Code: Select all

https://www.centos.org/download/mirrors/

so they can allow the traffic and then if I can change the timeout to something like a minute before trying another mirror. I am assuming I just need to disable the fastestmirror plugin to start with but not real certain with all this stuff yet.

[TrevorH]

Disable fastestmirror, it's useless.

Look up the mirrors closest to you https://www.centos.org/download/mirrors/ and pick them from there.

[pjsr2]

You can change the repository to use in the /etc/yum.repos.d/*.repo files. Comment out the mirrorlist line and un-comment the baseurl line while changing to the url of your preferred repository. You can specify multiple urls. See man yum.conf and look for "baseurl"

yum has a default timeout of 30 s before trying a different repo from the mirror list . If this is to short for your BlueCoat, add a larger value for "timeout" in /etc/yum.conf. Again, see man yum.conf for more information.

My understanding is that when you clean the yum catch with yum clean all the mirrorlist will be retrieved again and all mirrors in the list of mirrors will be tested first time you use yum. When your firewall settings only allow access to a subset of the mirrors, only reachable mirrors will be added in the cached list of mirrors in /var/cache/yum/x86_64/17/timedhosts.txt and only these will be used next time.
When you use the fastestmirror plugin, adjust the exclude or include lines in /etc/yum/pluginconf.d/fastestmirror.conf