New to the community? Start here

[Defender Pro] Enhancement to Defender 404 detection

0

Hi,
I am struggling a bit to describe the suggestion, so I will tell you what I am seeing and then my proposed solution to see if it can spark a discussion around what can be done (hopefully).

I regularly get 404 hits to files that obviously (to me and probably any real person) should be banned immediately (various attempts for php files in the root directory or other locations). I am not sure how Defender triggers a 404 hit, but couldn’t we have some optionally enhanced checking when its of a specific filetype (ie PHP/CNF/configurable) possibly in some configurable folders (or even the opposite where we exclude folders) such that Defender then does real (!file_exists(xyz.php)) checking and blocks according to the enhanced settings.

The standard WP 404’s tend to be cache files missing/pages moved etc and not so much PHP files that no longer exist.

Anyway I hope that helps start the discussion of the feature suggestion

Regards

Mark

Thu, 14 May 2020 10:27:04
More Defender Pro discussions
Defender Pro
  • Adam
    • Support Gorilla

    Hi Mark

    I hope you’re well today!

    Defender just detects 404’s as WP would do but while WP would just serve 404 page, Defender logs it along with IP and, if set to, puts a lockout on the IP. There’s not that much more into it actually as all “404 thing” is basically a web/HTTP protocol related rather than checking if file exists.

    I admit I got some doubts here.

    A “file_exists()” check based on the URL/path of the request – that would also be quite unreliable because a given file might not really physically exist when URL suggests. As an example:

    SmartCrawl creates sitemap thats available at “domain.com/sitemap.xml” and that would suggest that the “sitemap.xml” file is in root folder of your WP installation, but it is not. So let’s say that due to some unexpected temporary issue on site the sitemap is, temporarily, unavailable. A Google Crawler tries to visit it, hits 404, Defender does “file_exists()” and it returns “false” (because the file doesn’t really exist where URL suggests it should be) and suddenly Google Crawler gets blocked and cannot index the site anymore.

    Similar could be with “robots.txt” and many other files, including php files.

    However, Defender already has some options that partially match what you suggested: black- and white-lists for “files & folders” and for “filetypes and extensions”, which you can specify on “Defender Pro -> Firewall -> 404 Detection” page.

    Then, I’m thinking that maybe a bit different approach could be more handy. For example use the kind of “enhanced settings” that you describe to, instead, e.g. be able to set “immediate ban” for certain types of files, then e.g. ban only after set number of 404 hits for others and e.g. redirect to defined page instead of the ban for some others?

    What do you think about it?

    Best regards,
    Adam

  • Mark
    • The Crimson Coder

    Hi Adam,

    Yes that’s why I was a bit vague, I can see the need for enhancement but not quite sure what (it niggles the back of my head).

    One thing though I would think this check should happen AFTER WP has already 404’d it (ie its not there according to WP) and again in combination with a series of “enhanced settings” so for example PHP files in the root/sensitive/other/configurable directory. Obviously it would need a series of configuration options (probably including a series of whitelists to stop any false positives that the defaults do not take care of).

    I see a lot of tries for files (/w.php /2012.php etc. that should obviously be immediately banned) as well as some other folders. But TBH the root folder is the key and you very rarely see stuff added to the root folder.

    Regards

    Mark

  • Predrag Dubajic
    • Support

    Hi Mark,

    I gave this a couple of reads and it more and more looks to me that what Adam is suggesting above, with the Blacklist and Whitelist option, is what you are after because blacklisting the file will trigger the block after the first 404 report.

    For example, if you add something like /blockme.php as blacklist rule and someone visits that URL it will be blocked right away and on the second try it will show Defender block message.
    However, the rest of the files will follow the default blocking options, so if you would visit /dontblockme.php it will not block you right away and it will only apply the block if the file is visited the number of times that it was set in Defender.

    Is that the workflow that you’re after or you’re looking to have some further control on that?

    Best regards,
    Predrag

  • Mark
    • The Crimson Coder

    Predrag,

    Well that’s one option, but it was more along the lines of:

    We set a folder (or series of folders) and have an added “risk” score (I suppose you would say) for php (or whatever you configure). Where you can say IF a hit occurs here for a file of this type rather than wait for the 20 404’s in 300 second,. Immediately ban this IP (or possibly have a different rate setting).

    But yes having a “blockme” button/config option so that future hits would be automatically blocked would be good as well.

    Regards

    Mark

  • Adam
    • Support Gorilla

    Hi Mark

    Thanks for response!

    I gave it another thought and I think it all probably comes down to something slightly different. What I’m thinking is:

    – we already have blacklists and whitelists for IPs
    – we already have option to blacklist and whitelists for specific files and folder
    – and for file types and file extensions

    But we only got a common “threshold” while it would be good to be able to specify different “thresholds” for different “things”. So for example:

    – I’d set a “default” threshold for temporary 600 seconds ban if a given 404 was hit 20 times within 300 seconds

    – but then I’d be able to also set temporary ban for 1200 seconds if a 404 was hit for e.g. /wp-content/plugins/ folder (and that could be a “partial sting” – so would use e.g. regex or similar matching) 5 times in 600 seconds

    – and also permanent ban if a 404 was hit 3 times in a day for e.g. any .php file

    These are just examples but my point is that it seems to all come down to me to have an option to set default lockout threshold and then set additional lockout rules (temporary/permanent, number of hits, timeout) depending on file/folder/filetype/file extension/partial URL

    Do you think we are “getting somewhere” with it? :slight_smile:

    Best regards,
    Adam

  • Mark
    • The Crimson Coder

    HI Adam,

    That “ruleset/regex” approach also gives me another idea.
    If they were importable/exportable you could set up an area to distribute the rules for people with various combinations of plugins (ie if you use woocommerce install these rules etc.). This way we potentially only have the rules we need for the modules being used (as well as a generic set for general attacks).

    Mark

  • Adam
    • Support Gorilla

    Hi Mark

    If they were importable/exportable you could set up an area to distribute the rules for people with various combinations of plugins (ie if you use woocommerce install these rules etc.).

    That’s actually something we are already looking into – ways to export/import settings so they could be distributed across sites, though it will work with Hub 2.0 and while part in Defender is already ready as far as I’m aware, it still awaits implementation in Hub.

    As for what we “worked out” about the 404 enhancement, I’ve passed the summary of it to our developers for further consideration.

    Best regards,
    Adam