Recently picked up "Python" and decided to try my hand at a "n00b" script to dump the contents of "Robots.txt" file residing on a webserver.
# Ask for Protocol and store it in protocol
protocol = input('Enter HTTP or HTTPS: ')
# Ask for URL or IP and store it in domain
domain = input('Enter URL or IP: ')
robots = "/robots.txt"
from urllib.request import Request, urlopen
print('Checking "Robots.txt" for:')
print(domain)
print()
from urllib.error import URLError, HTTPError
req = Request(protocol+"://"+domain+robots)
try:
response = urlopen(req)
except HTTPError as e:
print('The server couldn\'t fulfill the request.')
print('Error code: ', e.code)
except URLError as e:
print('We failed to reach the server.')
print('Reason: ', e.reason)
else:
print('Contents of "Robots.txt" is as follows.')
print()
with urlopen((protocol+"://"+domain+robots)) as stream:
print(stream.read().decode("utf-8"))
#Written by commandrine.
#Last updated on 22 Jun 2017.
Subscribe to:
Posts (Atom)
VMware Workstation Pro is now free for personal use!!!
VMware Workstation Pro is now free for personal use!!! However, it was not straight forward to install on Ubuntu as I encountered error mes...
-
This annoying message popped up after I ran the update in avast! in Ubuntu yesterday. avast! crashes every time I attempt to launch it after...
-
I decided to install a digital certificate for my Gmail account. This is simple and free to set up. Apply for a free certificate from Comod...