cannot access web page with urllib/urllib2

[netvigator]

I am trying to post to this page and get the result:

https://writerep.house.gov/writerep/welcome.shtml

I tried this:

import urllib
import urllib2

sURL = 'https://writerep.house.gov/writerep/welcome.shtml'
dInfo = { "state" : 'WAWashington', "zipcode" : '98103', "zipext" : '' }

sSendData = urllib.urlencode( dInfo )
resp = urllib2.urlopen( sURL, sSendData )

I get an error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.5/urllib2.py", line 124, in urlopen
    return _opener.open(url, data)
  File "/usr/lib/python2.5/urllib2.py", line 387, in open
    response = meth(req, response)
  File "/usr/lib/python2.5/urllib2.py", line 498, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.5/urllib2.py", line 425, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.5/urllib2.py", line 360, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.5/urllib2.py", line 506, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 405: Method Not Allowed

I am using Python 2.5.2 at the moment.

I would appreciate any tips to get this working.

Thanks,

[woooee]

Try mechanize and see if that is any easier http://wwwsearch.sourceforge.net/mechanize

[Tech B]

I had a snippet on jut this. It may be cookie bad cookie handling, or lack there of. Check this out: http://www.daniweb.com/code/snippet278635.html

[netvigator]

Hey Tech-B,

I tried the snippet, but I got the same error.

import urllib, urllib2, cookielib
cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
sUserAgent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
opener.addheaders.append( ( 'User-agent', sUserAgent ) )
sURL = 'https://writerep.house.gov/writerep/welcome.shtml'
opener.addheaders.append( ( 'Referer', sURL ) )
dInfo = { "state" : 'WAWashington ', "zipcode" : '98103', "zipext" : '' }
sPostData = urllib.urlencode( dInfo )
resp = opener.open( sURL, sPostData )

I do not think the web page wants the data coming back at the end of the URL, rather it should come back in the headers?

I downloaded mechanize and I could try that, but I still think the problem is sending the data to the web server in the format it wants.

If I get no other suggestions, I will try mechanize.

Thanks,

[Tech B]

After actually visiting the site and taking a look at the source for the page, I got it working. I'll comment in the code where you went wrong.

import urllib, urllib2, cookielib
cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
sUserAgent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
opener.addheaders.append( ( 'User-agent', sUserAgent ) )

#The source of the page says to POST your data to the following adress
sURL = 'https://writerep.house.gov/htbin/wrep_findrep'
opener.addheaders.append( ( 'Referer', sURL ) )

#the zip field is named zip, not zipcode. You also have to submit the info, also found in the source of the page
dInfo = { "state" : 'WAWashington ', "zip" : '98103', "submit": "Contact My Represenrative"}
sPostData = urllib.urlencode( dInfo )
resp = opener.open( sURL, sPostData )

#I write everything to an html page, that way I can open it in the
#browser and debug from there. This step isn't really needed.
f = open("f.html","w")

#resp is now a file like handle, so you have to read or readlines.
f.write(resp.read())
f.close()

[netvigator]

Tech B,

It works for me. Excellent!

I will also take a look at the source and zero in on what I overlooked.

Thanks,

[Tech B]

Glad I could help :).
Could you mark the thread solved please.