I've had problems with Yandex (and Baidu) in the past, they both hit harder than google. Given that I had 0% interest in the Russian/Chinese market where I was, I blocked them. I tried using robots.txt, but they didn't play well.
You might find this[^] discussion helpful I went down the URLRewrite route, it worked pertty well, but needed IIS7 plus the module installed.
Thanks for the article, I need to know that stuff anyway. Google is actually pretty courteous crawling at a rate of about 2 minutes each.
I think my strategy is wrong here. Perhaps for crawlers and bots, I should make them wait for the whole process to complete on a single thread. But on human users, I can put data writes on background threads so there experience is faster.
I prefer to have such plugins downloadable by nuget rather than build into language/framework. As you won't always need that and can keep things more simple. Too many contros means you will never know about them so won't use them anyway
"My software never has bugs. It just develops random features."
I'm sending mail to newly register Users in html format(Asp.net C#). It works fine but sometimes Some Unwanted html code(that dose not exist in my webpage) showing after mail body even mail content is correct. if anyone know the solution, plz help.
Here is that unwanted code that User getting.
With all do respect, how are we suppose to know where to look.
You should add a watch to your message body and run the program in debug, then make a copy of the body text, and send it to yourself. Then inspect both message body content and compare the two.
Then inspect the message content of the other email with the extra text and make a comparison as well. If the extra html only appears after relaying through the other mail server, then that mail server is injecting the html.