A couple days ago I needed to call a remote web api call in my AuthorizeAttribute sometimes but as mentioned in this (Is it possible to use async/await in MVC 4 AuthorizeAttribute?) StackOverflow question (and other forums) it isn’t supported but is in the newer .net core. Unfortunately the project I needed this on was traditional MVC so I was left still finding a way .
Firstly you might want to know why would you want to implement a dynamic robots.txt? Isn't it better to just put a static file in the web app? Yes it is but when you running your application on certain platforms like web apps in Azure when you using deployment slots. I found this handy link which explains deployment slots quite nicely , but basically what happens is when you switch 2 slots Azure will do a virtual IP switch so traffic from your slots will now be switched (not the files from each slot).