New to the community? Start here

[SmartCrawl Pro] Deactivate indexing of custom post type, blocks template after activation

2

Hi, i have seen that when i enable smartcrawler, he makes all archives and all post types indexable, in my opinion it would be better to put all of them as non-indexable by disabling all options.

very often it also enables wordpress blocks and builder templates to index. And this is not good, if by chance a link is put somewhere it would be indexed.

i hope i explained myself.

Thu, 19 May 2022 7:06:07
More SmartCrawl Pro discussions
SmartCrawl Pro
  • Adam
    • Support Gorilla

    Hi Edoardo

    I hope you’re well today!

    It’s not quite that SmartCrawl “enables indexing”. By default they’d be indexable anyway unless specifically denied. Allowing/disallowing indexing is actually a “negative selection” thing – whatever is not specifically set to not be indexed, is allowed to be indexed.

    I’m referring to search engines here.

    Point is, regardless of whether SmartCrawl (or any other plugin or theme feature) adds “index” tag or not, a page (post, archive, template – as long as it’s publicly visible) is allowed to be indexed.

    Only specifically adding “noindex” would “switch off” indexing (or blocking it via robots.txt or blocking access to given URL entirely).

    So in other words – SmartCrawl is in fact following/”legitimizing” default state anyway.

    ———————

    However, I think it’s a good point and some of the types of content could very well be “blocked” from indexing right from the start.

    I suppose content like “archives” and similar – that would be debatable. If we make it “indexing disabled” by default we may also find out soon that some other Members ask us to make the “indexing enabled” by default – as it depends on the needs and specific site.

    But things like templates and other “technical” post types – that could be blocked right away. There’s no point in indexing templates and with builders used more and more often, it becomes an issue.

    —–

    So to sum it up, here’s what I’m thinking:

    What about keeping things like archives as they are but then by default setting “noindex” for all the post types that are “different” than some essential and most popular ones like, for example:

    post, page, product, portfolio, event and so on?

    Basically – all post types except these specific ones – would be set to not indexing by default. What do you think? What post types would you add to such list?

    Best regards,
    Adam

    • Edoardo
      • The Bug Hunter

      sure all clear.

      yes it is true with archives it is questionable, but you could do some kind of onboard setup that asks what to index and what not to index.

      in my opinion are absolutely avoid indexing post types like templates, wordpress blocks, and all variants related to various builders, also post types related to form plugins like gravity forms, entries and others, they are not always public but it can happen.

      for example even upon creation of a new post type it could be detected and a notification appear asking whether to index it or not.

      it is very wearisome every time to have to turn off all the various settings even of social etc. for every thing that should not have anything active(including schema and other)

  • Adam
    • Support Gorilla

    Hi Edoardo

    Thank you for your feedback!

    Sounds good to me :)

    Some points are already flying your way and I’ve passed your idea to our SmartCrawl Team.

    Let’s keep this thread open and see if other Members will also vote for this idea and/or have something to add to it.

    Best regards,
    Adam

    • Adam
      • Support Gorilla

      Hi Edoardo

      Actually, such a tool is already built-in into WordPress. On “Settings -> Reading” page there’s “Search engine visibility” checkbox that does exactly that – tells search engines to not index site at all if it’s checked.

      Such a button in SmartCrawl would be just duplication of a feature that already exists in WP core.

      What we discussed earlier – settings “noindex” “in bulk” for certain content types and/or settings to “noindex” by default – that’d be slightly different thing and, as mentioned previously, I’ve passed that idea to our SmartCrawl Team already. I don’t have any ETA on this at the moment but I”ve forwarded it to them so “fingers crossed” ;)

      Best regards,
      Adam

    • Adam
      • Support Gorilla

      Hi Edoardo

      Hm… wouldn’t that be a bit of overkill? I’m not saying it would, just thinking out loud :)

      I mean, how many custom post types would you usually have registered – other than “hidden” ones used by various plugins for their internal purposes, which should probably be better just “left alone”? It would probably be tempting (I’m saying that based on behaviors that I often see on various sites of our Members) to just “try all of them” not even realizing what actually gets disabled…

      My point is: I can how such feature could possibly be useful but I’m also afraid that, all in all, it could possibly create more confusion and, sometimes, actual “mess”… But that’s my thought only, I suppose I may be on a wrong track here :)

      What do you think?

      Best regards,
      Adam