python script hang over the screen method in cloud

[firoj26]

I have an issue with running python scripts on linode cloud server. I create a scrapping script to scrape data by check in and check out so that i run the script using screen method. It run over one day very smoothly but today it hang. No response. When i call getRequest() method then i print the proxy number and send request with proxy and timeout=10.

def getRequest(self,url):
        self.proxieName = random.choice(self.proxyDict.keys())
        print getYellow("Proxy: ")+getCyan(self.proxieName)
        self.proxie = self.proxyDict[self.proxieName]
        try:
            self.getRqst=requests.get(url,headers=self.nrmlHeaders,proxies=self.proxie, timeout=10)
        except Exception, e:
            print e
            time.sleep(10)
            return self.getRequest(url)
        else:
            if self.getRqst.status_code==200:
                self.rqstErr = 0
                return self.getRqst
            else:
                self.rqstErr+=1
                print getYellow("Hi Dear, Don't worry. May be Server down. Exception in Url: ")+url
                print getRed("Error!!! ")+getBlue("I get Status Code: ")+str(self.getRqst.status_code)
                print getGreen("I am sleeping now for 10 second")
                time.sleep(10)
                return self.getRequest(url)

Here is the image snapshot link https://www.dropbox.com/s/yh2bs7z1mxcgkcd/script%20error.png?dl=0

after getting the content i parse them and save information into a CSV file and send another request.

And i see also that after 01:40 of 6th July 2015, the csv file alos not updated. So it's hanging.

[Gribouillis]

This function should be rewritten in a non recursive style. If the connection times out every time, it will call itself indefinitely until it reaches the recursion limit. Did you change the recursion limit ?

It is very easy to write this function with a while True loop.

[firoj26]

Thank you, But after trying the request in line 6,

self.getRqst=requests.get(url,headers=self.nrmlHeaders,proxies=self.proxie, timeout=10)

it doesn't show any message in Exception Block in line 8 or line 17

print getYellow("Hi Dear, Don't worry. May be Server down. Exception in Url: ")+url

[Gribouillis]

You probably need to find some documentation about the timeout parameter in requests.get(), or perhaps you could try this solution with eventlet.Timeout().

[firoj26]

can i use post request with eventlet.Timeout()