bash fork error (Resource temporarily unavailable) does not stop, and keeps showing up every time I try to kill/reboot

I mistakenly used a limited server as an iperf server for 5000 parallel connections. (limit is 1024 processes) Now every time I log in, I see this:

-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: Resource temporarily unavailable

Then, I try to kill them, but when I do ps, I get this:

-bash-4.1$ ps
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: Resource temporarily unavailable

Same happens when I do a killall or similar things. I have even tried to reboot the system but again this is what I get after reboot:

-bash-4.1$ sudo reboot
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: Resource temporarily unavailable
-bash-4.1$ 

So Basically I cannot do anything. all the commands get this error :/ I can, however, do "exit".

This is an off-site server that I do not have physical access to, so I cannot turn it off/on physically.

Any ideas how I can fix this problem? I highly appreciate any help.

Answers


Given that you can login, you may want to try using exec to execute all your commands. After executing exec you will have to log in again, since exec will kill your shell (by replacing it with the command you run).

exec won't take up an extra process slot because it will replaces the running shell with the program to run. Thus, it should be able to bypass the ulimit restriction.


I had the same issue recently. In my case the reason was there was code that was executing under my ownership and consumed almost all the resources leaving nothing for my commands. Here's what I did, "exec top" to identify the PID thats consuming maximum resources "exec kill -9 " killing the PID identified by above command.

After killing the PID, everything came back to normal and I was able to login back.


Need Your Help

Angular filter works but causes "10 $digest iterations reached"

angularjs underscore.js

I receive data from my back end server structured like this:

ASP.NET MVC: How to Route Search Term with . (Period) at the end

.net asp.net-mvc model-view-controller routing period

I get a 404 response from .Net MVC when I try to make a request where my search term ends with a . (period). This is the route that I'm using: