Member Avatar for gunbuster363

I want to crawl my gf's xanga's post into my computer for better reading
but it require me to login before viewing the post

I am wondering ,can python crawl this password protected webpage?
I already have the id and password, because that is my id.

the login webpage, for example, is like this:

after entering the password and login, the page is :
http://gunbuster363.xanga.com/archives/2004/12/


Thanks all!

  • 2 Contributors
  • 5 Replies
  • 699 Views
  • 4 Days Discussion Span
  • Latest Post Latest Post by gunbuster363

Dani AI

Generated

Python can emulate a browser well enough to log into an ASP.NET (ASPX) page and then fetch protected content — the key is reproducing the exact HTTP exchange the site expects. was right to point toward the “form-post + cookies” approach, and ’s diagnosis (inspecting the browser’s network traffic and replicating the POST) is exactly the practical method that works.

Typical, reliable workflow

  • Start a session and GET the login page so the server issues initial cookies and any hidden inputs.
  • Parse required hidden fields and tokens from the returned HTML (examples common on ASP.NET sites: VIEWSTATE, EVENTVALIDATION, etc.).
  • Build the form payload including those hidden fields plus the credentials, then POST to the form’s action URL exactly as the browser does.
  • Follow redirects in the same session, then GET the protected page and verify expected content (not a login page).

Common pitfalls and debugging tips

  • Missing hidden fields or anti-CSRF tokens will usually return the login page again. Capture the full browser POST (Network tab or Live HTTP Headers) and replicate headers that matter (Host, Referer, User-Agent, Content-Type).
  • Some sites post to a different host or rely on JavaScript to compute a token; in those cases a headless browser (Selenium or Playwright) is required.
  • Watch for CAPTCHAs, rate limits, or IP-based protections — those are not solvable by simple form-posts.
  • Verify cookies, response codes, and redirects after the POST. Logging request/response headers and response.url will show whether the session succeeded.

Minimal pattern (replace placeholder names and URLs)

import requests
from bs4 import BeautifulSoup

s = requests.Session()
login_page = s.get('LOGIN_PAGE_URL')
soup = BeautifulSoup(login_page.text, 'html.parser')
payload = {
  'username_field': 'MYUSER',
  'password_field': 'MYPASS',
  'hidden_token_name': soup.find('input', {'name':'hidden_token_name'})['value'],
}
s.post('FORM_ACTION_URL', data=payload)
protected = s.get('PROTECTED_PAGE_URL')
print(protected.status_code, protected.url)

Respect account ownership and site TOS/privacy when crawling. The thread’s solution path (inspect the real browser POST, capture hidden fields, replay with a session) is the robust approach that resolves the issue described by .

Recommended Answers

All 5 Replies

Member Avatar for Beat_Slayer

The concept is the same as in this Tech B snippet.

Give it a try,

Happy coding.

Member Avatar for gunbuster363

I think I cannot login properly.
Can anybody help me out?

this is my code

import urllib, urllib2, cookielib

#cookie storage
cj = cookielib.CookieJar()
#create an opener
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
#Add useragent, sites don't like to interact programs.
opener.addheaders.append(('User-agent', 'Mozilla/4.0'))
opener.addheaders.append( ('Referer', 'http://www.hellboundhackers.org/index.php') )

#encode the login data. This will vary from site to site.
#View the sites source code
#Example###############################################
#<form id='loginform' method='post' action='index.php'>
#<div style="text-align: center;">
#Username<br />
#<input type='text' name='user_name' class='textbox' style='width:100px' /><br />
#Password<br />
#<input type='password' name='user_pass' class='textbox' style='width:100px' /><br />
#<input type='checkbox' name='remember_me' value='y' />Remember Me<br /><br />
#<input type='submit' name='login' value='Login' class='button' /><br />

login_data = urllib.urlencode({'XangaHeader$txtSigninUsername' : 'ZZZZZZZZZ',
                               'XangaHeader$txtSigninPassword' : 'ZZZZZZZZZ',
                               'signin' : 'Sign In'
                               })

resp = opener.open('http://holly-hahaha.xanga.com/?nextdate=10/21/2005', login_data)
#you are now logged in and can access "members only" content.
#when your all done be sure to close it

print resp.read()

resp.close()

and this the part of the website for login

<form id="SigninForm" class="Form1" method="post" action="">
<input name="IsPostBack" type="hidden" id="IsPostBack" />
<ul class="list details-only">
<li class="item item-1 item-odd">
<div class="details">
<h4 class="itemtitle"><label for="XangaHeader_txtSigninUsername">Username</label></h4>
<div class="itembody">
<input name="XangaHeader$txtSigninUsername" type="text" id="XangaHeader_txtSigninUsername" maxlength="100" onmouseover="this.className='over';" onmouseout="this.className='';" onfocus="this.className='over';" onblur="this.className='';" tabindex="1" />
</div>
</div>
</li>
<li class="item item-2 item-even">
<div class="details">
<h4 class="itemtitle"><label for="XangaHeader_txtSigninPassword">Password</label></h4>
<div class="itembody">
<input name="XangaHeader$txtSigninPassword" type="password" id="XangaHeader_txtSigninPassword" maxlength="16" onkeypress="return SigninOnEnter(event);" onmouseover="this.className='over';" onmouseout="this.className='';" onfocus="this.className='over';" onblur="this.className='';" tabindex="2" />
<a id="signin" href="javascript: SigninSubmit();" tabindex="3">Sign In</a>
</div>
</div>
</li>
</ul>

I know I am wrong because what I got from the python shell is the page that require me to login

Member Avatar for Beat_Slayer

I think your line 28 should have the login address page, and not the page you are trying to read later.

Member Avatar for gunbuster363

I think your line 28 should have the login address page, and not the page you are trying to read later.

I don't know.
The fact is, if I is not login-ed, it always show the login page
which is ""

I have tried to used this page to login,

import urllib, urllib2, cookielib

#cookie storage
cj = cookielib.CookieJar()
#create an opener
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
#Add useragent, sites don't like to interact programs.
opener.addheaders.append(('User-agent', 'Mozilla/4.0'))
opener.addheaders.append( ('Referer', 'http://www.hellboundhackers.org/index.php') )

#encode the login data. This will vary from site to site.
#View the sites source code
#Example###############################################
#<form id='loginform' method='post' action='index.php'>
#<div style="text-align: center;">
#Username<br />
#<input type='text' name='user_name' class='textbox' style='width:100px' /><br />
#Password<br />
#<input type='password' name='user_pass' class='textbox' style='width:100px' /><br />
#<input type='checkbox' name='remember_me' value='y' />Remember Me<br /><br />
#<input type='submit' name='login' value='Login' class='button' /><br />

login_data = urllib.urlencode({'XangaHeader$txtSigninUsername' : 'ZZZZZZZZZ',
                               'XangaHeader$txtSigninPassword' : 'ZZZZZZZZZ',
                               'signin' : 'Sign In'
                               })

resp = opener.open('', login_data)
#you are now logged in and can access "members only" content.
#when your all done be sure to close it

page = urllib2.urlopen("http://holly-hahaha.xanga.com/?nextdate=10/21/2005")
print page.read()
resp.close()

But the webpage source code I get is not what I expect,
I expect there would be some Chinese text,
but there are no a single Chinese character.
And it obviously is the login page, because I see:

<head>
<title>Xanga - Signin Lock</title>

Edited by gunbuster363 because: n/a
Member Avatar for gunbuster363

Problem solved.
I found out that, no matter what technology/method the website used to submit the username and password, what they sends are fixed.
We can use some tools in browser to check the package/header sent, and thus we can discover the name of the variable. For example, I used a add-on in firefox named Live http headers. Then for each communication between browser and server, we know what they send. For example, in xanga, when we press login, what they send to the server is :

POST /front.aspx HTTP/1.1
Host: hk.xanga.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:) Gecko/20100625 Firefox/3.6.6 GTB7.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Proxy-Connection: keep-alive
Referer: http://hk.xanga.com/
Cookie: __gads=ID=3b1be96f92f256c2:T=1279503979:S=ALNI_MZ-Hq7T3QNrg91hsHqHkJcnuhZ21w; FFSkp=1044,646,16; __qca=P0-1691570154-1279504135090; __utma=259717779.1902645459.1279504131.1279504131.1279504131.1; __utmb=259717779.2.10.1279504131; __utmc=259717779; __utmz=259717779.1279504135.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
Content-Type: application/x-www-form-urlencoded
Content-Length: 101
IsPostBack=true&XangaHeader%24txtSigninUsername=AAAAAAAA&XangaHeader%24txtSigninPassword=AAAAAAAA


Thus, we can send 3 information to the server thru python, namely "IsPostBack"=true, "XangaHeader%24txtSigninUsername"/"XangaHeader$txtSigninUsername"=username, "XangaHeader%24txtSigninUsername"/"XangaHeader$txtSigninUsername"=password

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.