Conserve mode activated due to high memory usage fortigate как исправить

3 REPLIES 3

We had an issue where our Fortigate was using “Conserve Mode” due to high memory usage. All outbound traffic was halted as a result. FW was running at about 90% at the time. Looking into this further we found multiple “wad” processes running.

After killing these instances the memory dropped down to ~50%.

Reading further into the WAD process it is used for ” cache, explicit-proxy and wan optimization.” Which we are only using SD-WAN with Performance SLA for outbound traffic load balancing & failover.

• Has anyone ran into this issue before?

• Any advise as to what we can do so this doesn’t happen again? We still need SD-WAN enabled.

I also found this Fortinet Forum post with people also seeing similar issues currently. https://forum.fortinet.com/tm.aspx?m=100000&mpage=2

“The system has entered conserve mode”

“Fortigate has reached connection limit for n seconds”

That is status field from the “Alert message control”  on System Dashboard. that status indicates the critical level from FortiGate device if it has entered conserve mode.

This problem happens when shared memory goes over 80%, to exit this conserve mode you have to wait (or kill some of the processes) until the memory goes under 70%. A FortiGate goes into the conserve mode state as a self protection measure when a memory shortage appears on the system. When entering conserve mode the FortiGate activates protection measures in order to recover memory space. When enough memory is recovered, the system is leaving/exiting the conserve mode state and releases the protection measures.

Antivirus fail-open is a safeguard feature that determines the behavior of the FortiGate AntiVirus system, when it becomes overloaded with high traffic.

to mitigate this you have more type of options:

# set av-failopen { off | on-shot | pass | idledrop}

Below we will describe what all of them do:

a. Off – if the FortiGate enters conserve mode, the FortiGate will stop accepting new AV sessions, but will continue to process currently active sessions

b. One-shot – if the FortiGate enters conserve mode, all new connections will bypass the AV system, but currently sessions will continue to be processed. This is the same as the “pass” options, but it will NOT turn off once the condition causing the av-failopen has stopped

c. Idle-drop – will drop connection based on the clients that has the most opened connection

d. Pass – this is the default option

Please keep in mind that with one-shot and pass option, NO content filtering of the traffic is done. The data stream could contain malicious content.

Below are some commands to troubleshoot when the system enters conserve mode:

a. Check if the system is in Conserve Mode:

# diagnose hardware sysinfo shm
SHM counter: 67
SHM allocated: 1556480
SHM total: 101220352
conservemode: 0     [conservemode 0 means not in conserve mode, 1 means on conserve mode, 2 means on kernel conserve mode]
shm last entered: n/a
system last entered: n/a
SHM FS total: 106827776
SHM FS free: 105205760
SHM FS avail: 105205760
SHM FS alloc: 1622016

b. Check if there any errors on the interfaces:

#diagnose hardware deviceinfo nic

So, If this problem occurs somehow we need to reduce shared memory usage on Fortigate, some optimization which I have try to improve performance on this box is:

1. Session timer optimizations
# config system global
set tcp-halfclose-timer 30         [ default 120 s ]
set tcp-halfopen-timer 30          [ default 60 s ]
set tcp-timewait-timer 0           [ default 120 s ]
set udp-idle-timer 60                [ default 120 s ]
end

# config system session-ttl
set default 300                  [ default 300 ]

config port
edit 0
set protocol 17
set timeout 10
set end-port 53
set start-port 53
end
end

2. Reduce the FortiGuard services for the cache
# config system fortiguard
set webfilter-cache-ttl 500 [ default 3600 ]
set antispam-cache-ttl 500 [ default 1800 ]
end

3. DNS cache optimization
# config system dns
set dns-cache-limit 300     [ default: 5000 ]
end

Optional:
3a. Disable options for DNS forwarding:
# config system dns
unset fwdintf
end

3b. Delete dns-udp session-helper:
# config system session-helper
delete 14     [ 14 is references number for dns-udp ]
end

4. Reduce memory caching in some features (Explicit proxy, FortiGuard Antispam/Webfiltering)
on FortiOS 5.0: System > Config > Features [ enable/disable ]

5. Turn off all non mandatory features such as Logging, archiving, data leak prevention, IPS
Display CPU/Memory usage:
# get system performance top <delay> <max_lines>
or
# diag sys top <delay> <max_lines>

And to kill process:
# diagnose sys kill 9

Restart any applications:
# diagnose test application <application> <option>

Restart IPS engine:
# diagnose test application ipsengine 99

6. Turn off DHCP-server services
# config sys dhcp server
delete (reference number for dhcp-server)
end

7. Reduce the maximum file size for antivirus scanning
On FortiOS 5.0: Go to Policy > Proxy Options > Common Options > Change: Amount (bytes)
On FortiOS 4.0: Go to Firewall > Policy > Protocol Options > reduce the file size threshold

If your firewall still entered conserve mode instead of still more RAM, replace with the higher model 😀

CONSERVE MODE ACTIVATED DUE TO HIGH MEMORY USAGE

FORTIGATE 60F KEEPS GOING INTO CONSERVE MODE : R/FORTINET – REDDIT

FORTIGATE PERFORMANCE ISSUES – BRUDERER RESEARCH GMBH

HOW TO AVOID FORTIGATE ENTERING CONSERVE MODE – GEMBUL …

TROUBLESHOOTING TIP: HOW TO DO INITIAL TROUBLESHOO

FORTIGATE VERSION 7.0.0-2(3) CONSERVE MODE WAD – YOUTUBE

FORTIGATE 201F ENTERS MEMORY CONSERVE MODE : R/FORTINET – REDDIT

IPS ENGINE 5.00239 HIGH MEMORY UTILIZATION, CONSERVE MODE

FORTIGATE CONSERVE MODE – HOW TO STOP IT AND WHAT IT MEANS

“THE SYSTEM HAS ENTERED CONSERVE MODE” FORTIGATE

TECHNICAL TIP: CONSERVE MODE CHANGES – FORTINET …

CONSERVE MODE | FORTIGATE / FORTIOS 7.2.3

TECHNICAL TIP: FREE UP MEMORY TO AVOID CONSERVE MODE

CONSERVE MODE | FORTIGATE / FORTIOS 6.2.12

HTTPSD MEMORY LEAK IN FORTIOS 6.2.7 #62 – GITHUB

FORTIGATE CONSERVE MODE : FORTINET – REDDIT

FORTIOS 7.0.0 CONSERVE MODE : R/FORTINET – REDDIT

TECHNICAL TIP: CONFIGURATION FILE SAVE MODE FOR CO … – FORTINET

WHAT IS CONSERVE MODE IN FORTIGATE AND ACTION TO BE TAKEN TO …

PREVENT FORTIGATE ENTERING CONSERVE MODE BY REDUCING MEMORY …

WHAT IS ‘CONSERVE’ MODE ON FORTIGATE? – FORTIANSWERS

FORTIGATE FIREWALL TROUBLESHOOTING : BECOME EXPERT IN 30 MINUTES.

TECHNICAL TIP: HOW CONSERVE MODE IS TRIGGERED – FORTINET