During a recent visit to one of our clients we received complaints about random account lockouts throughout the organization. People were frequently locked for unknown reason. I was asked to help investigate this issue.
We started by enabling audit logging in their Default Domain Controller policy. We’ve enabled these settings:
- Audit Policy : Audit account logon events (succes/failure)
- Advanced Audit Configuration-Logon/Logoff : Audit Account Lockout
Next we applied a filter to the Domain Controllers security logfiles. EventID 4740 will show when a lockout has happened. It didn’t take long before we actually saw the machine responsible. By examining the event you can determine the Caller Computer Name. That will be the machine on which you will have to focus.
Now we knew it was one of the Microsoft AD FS servers locking out the users. That raised the obvious questions like: why? and how? Fortunately AD FS can provide you with detailed logging which first needs to be enabled. In the Local Group Policy Editor enable Audit Application Generated (Succes and Failures). This will return a world of information.
And sure enough, after waiting a while we noticed lockouts again. We also noticed something we didn’t expect. In the eventlog we first noticed the URL that was accessed during the lockout. But we also noticed the location it was accessed from. We noticed IP’s from countries all over the world. Brute Force attack in progress!
Now we knew what was happening we decided to leverage Citrix ADC (NetScaler) to help mitigate. Denying traffic on Geo IP Location would certainly stop this attack. I started by creating 2 String Maps with the following purpose:
- SM_GEOIP_IP_WHITELIST – Purpose : Whitelist IP’s that will not be blocked
- SM_GEOIP_URL_WHITELIST – Purpose : List URLs that will be excluded from Geo IP blocking
Next we imported the builtin GeoIP Location database already present on NetScaler.
After importing the database we need to set the locationParameter with the matchWilcardto any parameter. Do so with the following command:
1 |
set locationParameter -matchWildcardtoany YES |
With show locationparameter the functionality of the GeoIP Database can be tested, as with nsmap -d -t and entering an IP address. The last command will actually show the notation we need further ahead.
Then an Advanced Policy Expression was created to actually make the IP whitelist function:
1 2 3 4 5 6 7 8 9 |
add policy expression PE_GEOIP_IP_WHITELIST "client.IP.SRC.TYPECAST_TEXT_T.IS_STRINGMAP_KEY(\"SM_GEOIP_IP_WHITELIST\") || (client.IP.SRC.SUBNET(16) + \"/16\").IS_STRINGMAP_KEY(\"SM_GEOIP_IP_WHITELIST\") || (client.IP.SRC.SUBNET(24) + \"/24\").IS_STRINGMAP_KEY(\"SM_GEOIP_IP_WHITELIST\") || (client.IP.SRC.SUBNET(25) + \"/25\").IS_STRINGMAP_KEY(\"SM_GEOIP_IP_WHITELIST\") || (client.IP.SRC.SUBNET(26) + \"/26\").IS_STRINGMAP_KEY(\"SM_GEOIP_IP_WHITELIST\") || (client.IP.SRC.SUBNET(27) + \"/27\").IS_STRINGMAP_KEY(\"SM_GEOIP_IP_WHITELIST\") || (client.IP.SRC.SUBNET(28) + \"/28\").IS_STRINGMAP_KEY(\"SM_GEOIP_IP_WHITELIST\") || (client.IP.SRC.SUBNET(29) + \"/29\").IS_STRINGMAP_KEY(\"SM_GEOIP_IP_WHITELIST\") || (client.IP.SRC.SUBNET(30) + \"/30\").IS_STRINGMAP_KEY(\"SM_GEOIP_IP_WHITELIST\")" -comment "GEOIP Allowed IPs from Blocked Countries" |
And another one to define the countries that we would like to allow:
1 2 3 4 |
add policy expression PE_GEOIP_ALLOWED_COUNTRIES "(client.IP.SRC.MATCHES_LOCATION(\"*.NL.*.*.*.*\") || client.IP.SRC.MATCHES_LOCATION(\"*.UK.*.*.*.*\") || client.IP.SRC.MATCHES_LOCATION(\"*.DE.*.*.*.*\") || client.IP.SRC.MATCHES_LOCATION(\"*.BE.*.*.*.*\")).NOT" -comment "GEOIP Allowed Countries" |
Finally we need a Responder policy that will return some useful information about why a user is blocked. If you prefer you can ofcourse just drop traffic. It would also be nice if the Responder Policy did some logging in the ns.log when it actually kicks in. You can then easily search your NetScaler logfiles if the Geo IP block did it’s work. To make it work first you need to enable User Configurable Log Messages in both the Syslog as nslog parameters.
1 2 |
set audit syslogParams -dateFormat DDMMYYYY -userDefinedAuditlog YES set audit nslogParams -dateFormat DDMMYYYY -userDefinedAuditlog YES |
After that a Audit Message Action can be created that can later be bound in the Responder Policy:
1 |
add audit messageaction AMA_GEOIP_BLOCK INFORMATIONAL "\"GeoIP_Block: \" + \" \"+ client.IP.SRC + \" \" + \" \" + http.REQ.HOSTNAME.SERVER + http.REQ.URL" -logtoNewnslog YES |
Now it’s time to create the Responder Action and Policy
1 2 |
add responder action RSA_GEOIP_BLOCK respondwith q{"HTTP/1.1 418 Not Authorized\r\nContent-Type: text/html\r\n\r\n<HTML><BODY><H1>Blocked</H1><br>Your IP address: " + client.IP.SRC + "<br>Requested url: " + HTTP.REQ.HOSTNAME + HTTP.REQ.URL + "<br>Is blocked due to GEOIP violation.<br> Please contact your administrator</BODY></HTML>"} add responder policy RSP_GEOIP_BLOCK "(PE_GEOIP_ALLOWED_COUNTRIES || PE_GEOIP_IP_WHITELIST || PE_GEOIP_URL_WHITELIST).NOT" RSA_GEOIP_BLOCK -logAction AMA_GEOIP_BLOCK |
And to make it all work bind the Responder Policy to the desired Load Balancing vServer. In my case I bound it to the Content Switching vServer so all resources would be protected.
1 |
bind cs vserver VS_CS_SSL -policyName RSP_GEOIP_BLOCK -priority 1 -gotoPriorityExpression END -type REQUEST |
After a while the Brute Force Attack stopped and we haven’t seen it again.