mkdir: cannot create directory Error

[taliez]

hi all,

encountered an error after developer turned over the server (root account)

details below

[root@server ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_server-lv_root 18G 14G 2.6G 85% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/sda1 477M 55M 397M 13% /boot
/dev/mapper/vg_fileserver-lv_fileserver 128G 45G 84G 35% /app
[root@serverg ~]# mkdir testFolder
mkdir: cannot create directory `testFolder': No space left on device
[root@server ~]# mount

[root@server ~]# more /etc/redhat-release
Red Hat Enterprise Linux Server release 6.5 (Santiago)
[root@server ~]# uname -a
Linux server 3.8.13-16.2.1.el6uek.x86_64 #1 SMP Thu Nov 7 17:01:44 PST 2013 x86_64 x86_64 x86_64 GNU/Linux
[root@server ~]# uname -r
3.8.13-16.2.1.el6uek.x86_64
[root@server ~]#


thanks all in advance

[TrevorH]

Several points there:

Red Hat Enterprise Linux Server release 6.5 (Santiago)

RHEL not CentOS

Linux server 3.8.13-16.2.1.el6uek.x86_64 #1 SMP Thu Nov 7 17:01:44 PST 2013 x86_64 x86_64 x86_64 GNU/Linux

RHEL with an Oracle kernel

As for the source of your problem, most likely you have run out of inodes - use df -i to see. If you have then you'll need to backup the data, reformat the filesystem and restore.

[taliez]

Several points there:

Red Hat Enterprise Linux Server release 6.5 (Santiago)

RHEL not CentOS

Linux server 3.8.13-16.2.1.el6uek.x86_64 #1 SMP Thu Nov 7 17:01:44 PST 2013 x86_64 x86_64 x86_64 GNU/Linux

RHEL with an Oracle kernel

As for the source of your problem, most likely you have run out of inodes - use df -i to see. If you have then you'll need to backup the data, reformat the filesystem and restore.

thank you very much sir for the FAST reply

your were right it ran out of inodes

Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/vg_server-lv_root 1.1M 1.1M 0 100% /
tmpfs 999K 1 999K 1% /dev/shm
/dev/sda1 126K 44 125K 1% /boot
/dev/mapper/vg_fileserver-lv_fileserver 8.2M 323K 7.9M 4% /app
[root@server ~]#

any common reason of running out of INODE?

and how can we prove to the developer that it runs NOW on ORACLE kernel?

will this help? on the justification?

[root@server ~]# cd /etc/yum.repos.d/
[root@server yum.repos.d]# ll
total 12
-rw-r--r--. 1 root root 2542 Jan 13 2015 public-yum-ol6.repo
-rw-r--r--. 1 root root 4562 Nov 8 2014 public-yum-ol6.repo.1
[root@server yum.repos.d]#


thanks sir TrevorH

[TrevorH]

Usual cause of running out of inodes is thousands (millions) of small files, usually in /tmp or /var/tmp but could be anywhere on the filesytem really.

and how can we prove to the developer that it runs NOW on ORACLE kernel?

Your uname -a output shows it in the version number: 3.8.13-16.2.1.el6uek.x86_64. CentOS 6 uses a 2.6.32-xxx kernel and always will. The 3.8 means it's a non-standard kernel and the "uek" portion of that is Oracle's "unbreakable enterprise kernel" (sic).

[taliez]

Usual cause of running out of inodes is thousands (millions) of small files, usually in /tmp or /var/tmp but could be anywhere on the filesytem really.

and how can we prove to the developer that it runs NOW on ORACLE kernel?

Your uname -a output shows it in the version number: 3.8.13-16.2.1.el6uek.x86_64. CentOS 6 uses a 2.6.32-xxx kernel and always will. The 3.8 means it's a non-standard kernel and the "uek" portion of that is Oracle's "unbreakable enterprise kernel" (sic).

just a NOTE sir this is ORACLE Enterprise Linux BTW,forgot to mention

last question .. how can we prevent issues like this in the near future?

cleaned up some files on the /tmp and /var/tmp and free up some but still 100%

[root@pri-fsprd-apg ~]# df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/vg_server-lv_root 1.1M 1.1M 3.0K 100% / <<<<<<<===============
tmpfs 999K 1 999K 1% /dev/shm
/dev/sda1 126K 44 125K 1% /boot
/dev/mapper/vg_fileserver-lv_fileserver 8.2M 323K 7.9M 4% /app

[root@server ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_server-lv_root 18G 14G 2.8G 84% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/sda1 477M 55M 397M 13% /boot
/dev/mapper/vg_fileserver-lv_fileserver 128G 45G 84G 35% /app
[root@server ~]#

thanks