Search DaniWeb - urllib2

urllib in python 3.1 15 Years Ago by Lingson hi, i've tried to search any examples to understand more about how to use the urllib in python 3.1, but all the tutorials are for python 2.x i need to do just a simple thing like getting the text (as string) and manipulate it. the code in python 2.x would be like: [CODE] import urllib url = 'http://www.pythonchallenge.com/pc/def/… Re: urllib in python 3.1 15 Years Ago by bumsfeld … urllib.request.urlopen() # instead of Python2's urllib.urlopen() or urllib2.urlopen() import urllib.request fp = urllib.request.urlopen("http… Urllib suddenly stopped working 16 Years Ago by trihaitran I just reinstalled OS X 10.5 Leopard on my laptop and restored my data/applications from a Time Machine backup. When I use urllib.urlopen() I get the following error: [code] >>> import urllib >>> u=urllib.urlopen(aUrl) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/… Re: urllib in python 3.1 15 Years Ago by vegaseat As Scru mentioned, let's hope the authors of the website were good guys and included the type of encoding. You can modify Henri's code ... [code=python]# get the code of a given URL as html text string # Python3 uses urllib.request.urlopen() # get the encoding used first # tested with Python 3.1 with the Editra IDE import urllib.request … Re: urllib tuts.. 16 Years Ago by mn_kthompson …[URL="http://www.velocityreviews.com/forums/t326690-urllib-urllib2-what-is-the-difference-.html"]http://www.velocityreviews.…com/forums/t326690-urllib-urllib2-what-is-the-difference-.html[/URL] It was … for me to understand the difference between urllib and urllib2 and what each can do. It's short,… Re: urllib in python 3.1 15 Years Ago by willygstyle Yeah good point, thank you for pointing this out I'm new to python in general and with not much documentation on 3.1 it's easy to assume things. I never actually looked at the read close enough to notice the b'. Yah at least I only have a few programs running with urllib atm, better to find out now than later. urllib problems 16 Years Ago by Shadow14l Whenever I open a page with urllib or urllib2 (file = urllib.urlopen(urllinkhere)) and when I print it, I … you need more details. The source is simply: [CODE]f = urllib2.urlopen(anyurlhere) print f.read()[/CODE] Thanks for any/all… urllib tuts.. 16 Years Ago by revenge2 Does anyone know of any good urllib tutorials other than [url]http://www.voidspace.org.uk/python/articles/urllib2.shtml#introduction[/url] & [url]http://effbot.org/librarybook/urllib.htm[/url]. Something which a little beginner friendly:| with good explanations. -Thanks urllib.request.urlopen (python 3) 16 Years Ago by scru I noticed a weird thing happening when i use this function. If the page i open is in latin-1 encoding, the bytes returned by this function would have some weird junk characters inserted in various places. However, if i use urlretrieve to fetch the page to disk, there is no junk in the resulting file. Ideas? EDIT: I decoded the bytes returned by … python 3.1 urllib request 15 Years Ago by yoni0505 I'm trying to learn how to send forms using python 3.1 and it doesn't go well. I've made a simple site with a POST form. it got user and pass fields, and it will save the data into a text file. (for debugging) The site is working, I can type in anything, press submit and it will be saved. I want to do this action with python, send the form.… Data Scraping Using Urllib With Multiple Option Select Param 13 Years Ago by blahman I am running into some issue with scraping data. If I hardcode value for key "lbo race" in the code below it is able to scrape the data but if I try to set key "lbo race" to a variable which is being read in it doesn't seem to scrape the data correctly. I tried to put a time to slow it down but that doesn't seem to be the issue.… Re: Data Scraping Using Urllib With Multiple Option Select Param 13 Years Ago by blahman So it seems to be a problem with the response.read() any idea why it would only read part of the html data? It does it fine if I just do one request at a time. I also put some of the code into a function just incase so the variables are reset: Code blocks are created by indenting at least 4 spaces ... and can span multiple lines … Correct Concept with Urllib 15 Years Ago by flebber Hi I am continuing on with one of my first projects in python. This module for my larger project will be to download files(legal freely published by sporting assosciations). I am not sure if my concepts are right in how best to execute this, I have read the urllib docs on python.org and the syntax itself seems okay. For example the first site… Help with Python urllib module 13 Years Ago by VulcanDesign Hi all, I'm trying to use Python's urllib to get a Facebook profile page. I get the following error: [CODE]IOError: [Errno socket error] [Errno 10035] A non-blocking socket operation could not be completed immediately[/CODE] Here's my code: [CODE] import urllib member_profile_text = urllib.urlopen('http://www.facebook.com/profile.php?id=… Help with urllib 11 Years Ago by giancan Dear friends, I need help understanding my error with urllib. I wrote the following code to check the availability of updates for a software I wrote try: datasource = urllib.urlopen("URL to FILE.txt") while 1: line = datasource.readline() if line == "": break if (line.find("<"… cannot access web page with urllib/urllib2 14 Years Ago by netvigator …zipext" : '' } sSendData = urllib.urlencode( dInfo ) resp = urllib2.urlopen( sURL, sSendData ) [/code] I get an error: [CODE] …(*args) File "/usr/lib/python2.5/urllib2.py", line 506, in http_error_default raise…(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 405: Method Not Allowed [/… Re: cannot access web page with urllib/urllib2 14 Years Ago by netvigator … got the same error. [code] import urllib, urllib2, cookielib cj = cookielib.CookieJar() opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj)) sUserAgent = 'Mozilla/4.0 (compatible… Re: cannot access web page with urllib/urllib2 14 Years Ago by Tech B … where you went wrong. [code] import urllib, urllib2, cookielib cj = cookielib.CookieJar() opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj)) sUserAgent = 'Mozilla/4.0 (compatible… Question about urllib 14 Years Ago by Jplusplus …} url = "http://mysite.hostoi.com/Register.html" req = urllib2.Request(url,data,headers) resp = urllib2.urlopen(req) return resp [/CODE] problem with urllib 14 Years Ago by debasishgang7 hey all, i'm trying to get all content of a webpage using python and trying to put them all into a file. Here is my code: [CODE]import urllib data = urllib.urlopen('http://www.site.net/index.html').read() #print data //if i ignore the rest part of this code the code is just working awesome...its printing the content of index.html f =… Re: problem with urllib 14 Years Ago by snippsat [B]read()[/B] read the whole file into a string,so no need [B]for lines in data[/B] [B]with open([/B]) close the file object auto. So you can write it like this. [CODE]import urllib data = urllib.urlopen('http://www.site.net/index.html').read() with open('somefile.html', 'w') as f: f.write(data) [/CODE] any suggestion for tutorial to create web crawler with python urllib? 10 Years Ago by Niloofar24 Hi again. I want to create a robot or spider or crawler with python urllib. Still couldn't find any good tutorial. Any suggestion?! Re: urllib in python 3.1 15 Years Ago by Lingson ugh.. the solution was so easy? *lol* one more question then: is it safe to say that everytime i want to convert bytearrays to string i just need to use the .decode('utf8')? (you are right with what you said about python 3 use more bytearrays than string. until now whenever that happen, i just try to use other way to overcome the problem since i … Re: urllib in python 3.1 15 Years Ago by vegaseat UTF8 is pretty much the defacto standard for decoding in Python. However there may be other standards on the internet, depending on the country and its native language. In this particular case ( [url]http://www.python.org[/url] ), you can find the encoding used in this line: [code] <meta http-equiv="content-type" content="… Re: urllib in python 3.1 15 Years Ago by scru [QUOTE=Lingson;955374]ugh.. the solution was so easy? *lol* one more question then: is it safe to say that everytime i want to convert bytearrays to string i just need to use the .decode('utf8')?...some other stuff...[/QUOTE] No, please don't do that. That is okay if the page you are reading is encoded in ascii or utf-8. But if it is … Re: urllib in python 3.1 15 Years Ago by Lingson ok.. i get the point.. huh, i guess there are no such thing as beginners luck in python :-p Re: urllib in python 3.1 15 Years Ago by Lingson amazing.. thanks a lot guys.. really appreciate all your help.. Re: urllib in python 3.1 15 Years Ago by willygstyle Not sure if this is a proper method or not but I havn't had any problems so far using.... [CODE=python] x = repr(request.urlopen(req).read()) print(x) [/CODE] Re: urllib in python 3.1 15 Years Ago by scru [QUOTE=willygstyle;957406]Not sure if this is a proper method or not but I havn't had any problems so far using.... [CODE=python] x = repr(request.urlopen(req).read()) print(x) [/CODE][/QUOTE] You may want to take a second look at what that actually does... Re: urllib in python 3.1 15 Years Ago by willygstyle It changes the returned request into a string? Seems to work so far.