Up coming, it is best to choose how you want the crawler to handle various Internet technologies. You can find an ongoing debate with regard to the intelligence of search engine crawlers. It isn't really solely very clear Should they be full-blown headless browsers or just glorified curl scripts https://www.youtube.com/watch?v=EVeTP0aV8gY