(Deleted) files and disk space isn’t grownig


Some times happen when you remove files locked by some aps (example weblogic, less app) you remove files but this action don’t free up space on the disk. One option is reboot because file descriptor create in /proc/pid/fd/XX symbolic link .
I want show you second option where you don’t need reboot machine .

 A file descriptor is a
capability.  You create the file descriptor with the "open" call and
request read or write permissions.  Later, when doing a read or write
operation, the kernel uses the file descriptor as an index into a
data structure that indicates what operations are allowed. 

E.G.: In temp I’ll create 1000 mb file test.tmp .When will still open in “Less” i delete file test.tmp

procpidfd

Preparing  example problem

Step one checking free space

 df -m /tmp
Filesystem     1M-blocks  Used Available Use% Mounted on
tmpfs               4850   130      4721   3% /tmp

Step two create 1000 mb file

dd if=/dev/urandom of=/tmp/test.tmp bs=1M count=1000
df -m /tmp/
Filesystem     1M-blocks  Used Available Use% Mounted on
tmpfs               4850  1030      3821  22% /tmp

Step three :use less for open file descriptor in read mode on this file .

less /tmp/test.tmp
Filesystem     1M-blocrks  Used Available Use% Mounted on
tmpfs               4850  1030      3821  22% /tmp
ps aux | grep less                                                                                                                                 ⏎
...
dzaczek   9149  1.0  0.0 117040  2284 pts/3    S+   19:14   0:07 less /tmp/test.tmp
...

Step four: remove test.tmp

rm -f /tmp/test.tmp

Checking fisk space :

df -m /tmp/
Filesystem     1M-blocks  Used Available Use% Mounted on
tmpfs               4850  1030      3821  22% /tmp

W removed file but use of diks is still 22%

So we need find this process:

ps uax | awk '/less/ {print $2}'
9149
 


/tmp ❯❯❯ lsof -p 9149
.....
less    9149 dzaczek    0u   CHR  136,3        0t0       6 /dev/pts/3
less    9149 dzaczek    1u   CHR  136,3        0t0       6 /dev/pts/3
less    9149 dzaczek    2u   CHR  136,3        0t0       6 /dev/pts/3
less    9149 dzaczek    3r   CHR    5,0        0t0      34 /dev/tty
less    9149 dzaczek    4r   REG   0,40 1048576000 1268546 /tmp/test.tmp (deleted)

look line 7  4r- opened for reading /tmp/test.tmp

we can check what is in File Decriptor /proc/PID/fd

[root@Euzebiusz ~]# ls -halst /proc/9149/fd/
total 0
0 lrwx------. 1 dzaczek dzaczek 64 Jan 19 19:18 0 ->/dev/pts/3
0 lrwx------. 1 dzaczek dzaczek 64 Jan 19 19:18 1 -> /dev/pts/3
0 lr-x------. 1 dzaczek dzaczek 64 Jan 19 19:18 3 -> /dev/tty
0 lr-x------. 1 dzaczek dzaczek 64 Jan 19 19:18 4 -> '/tmp/test.tmp (deleted)'
0 dr-x------. 2 dzaczek dzaczek  0 Jan 19 19:15 .
0 lrwx------. 1 dzaczek dzaczek 64 Jan 19 19:15 2 -> /dev/pts/3
0 dr-xr-xr-x. 9 dzaczek dzaczek  0 Jan 19 19:15 ..

'This is a subdirectory containing one entry for each file which the process has open, named by its file descriptor, and which is a symbolic link to the actual file. Thus, 0 is standard input, 1 standard output, 2 standard error, etc. '

A lite script we can find all deleted files locked by File descriptor

[root@Euzebiusz ~]# find /proc/*/fd -ls 2> /dev/null | grep deleted
  1288189      0 lr-x------   2  dzaczek  dzaczek        64 Jan 19 19:18 /proc/9149/fd/4 -> /tmp/test.tmp\ (deleted)

 

We can see there file is deleted but fd stil locked disk area .
I found 2 nice options for resolve this problem.
First shrink this file to size 0 block use *truncate
or use gdb

Solutions:

First examplefind-procfd-ls-2

truncate -s 0 /proc/9149/fd/4
[root@Euzebiusz ~]# truncate  -s 0 /proc/9149/fd/4
[root@Euzebiusz ~]# df -m /tmp
Filesystem     1M-blocks  Used Available Use% Mounted on
tmpfs               4850    30      4821   1% /tmp
[root@Euzebiusz ~]#

Free disk space is recovered.

or create small one line script for shrink all 'deleted' files

find /proc/*/fd -ls 2> /dev/null | awk '/deleted/ {print $11}' | xargs -p -n 1 truncate -s 0

Second option

use debugger for close file descriptor
i need create new file pid id changed so i need find pid

 find /proc/*/fd -ls 2> /dev/null | awk '/test.tmp/ {print $11}'
/proc/11596/fd/4

pid is 1156 file descriptor number 4


....
~ # ❯❯❯ gdb -p 11596
#send to process
(gdb) p close(4)
$1 = 0
(gdb)

[root@Euzebiusz ~]# df -m /tmp
Filesystem     1M-blocks  Used Available Use% Mounted on
tmpfs               4850    30      4821   1% /tmp
[root@Euzebiusz ~]#

Remember this is not only one way resolve this issue you can truncate by sent  :> to file descriptor or others not familiar method.

Important links and bibliography

Advertisements

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s