Member Avatar for gunbuster363

I know the password.

I want to crawl all the post of my gf's xanga down to my comp, for better reading.
But my gf have set "require friend login" to see her post.
So I need to login my id of xanga before reading her post.

What I want to ask is,
can we write a program that know how to get pass the password and crawl the webpage?
I know the password, because that is my id.


Thanks All.

  • 2 Contributors
  • 4 Replies
  • 115 Views
  • 1 Day Discussion Span
  • Latest Post Latest Post by masijade

Dani AI

Generated

If you already control the account, crawling the posts is doable — the task is reproducing the browser login flow and keeping the session cookies. noted the page calls SigninSubmit() and pointed you toward inspecting the JavaScript: that is the right first step. Before coding, use the browser DevTools Network panel to capture exactly what the page sends and what the server returns.

Practical checklist

  • In DevTools -> Network, perform the sign-in and locate the login request. Record method, URL, request headers, form fields (including hidden CSRF tokens) and the Set-Cookie response headers.
  • If you see a form POST, your crawler must GET the login page first (to fetch any hidden tokens), then POST the correct fields and preserve cookies. If the site uses an AJAX/XHR endpoint, mimic that request including the same headers (e.g., X-Requested-With) and tokens.
  • If the JS computes values client-side or a CAPTCHA is present, use a headless browser/automation (Selenium) instead of trying to reimplement the JS.

Example approach in Java
Use a modern HTTP client with a cookie store, POST the form data, then reuse the same client to GET protected pages and hand the returned HTML to a parser such as jsoup.

import java.net.*;
import java.net.http.*;

CookieManager cookieManager = new CookieManager();
cookieManager.setCookiePolicy(CookiePolicy.ACCEPT_ALL);
HttpClient client = HttpClient.newBuilder()
    .cookieHandler(cookieManager)
    .followRedirects(HttpClient.Redirect.ALWAYS)
    .build();

// POST login form (application/x-www-form-urlencoded)
String form = "username=YOU&password=YOUR_PASS";
HttpRequest login = HttpRequest.newBuilder()
    .uri(URI.create("https://example.com/login"))
    .header("Content-Type", "application/x-www-form-urlencoded")
    .POST(HttpRequest.BodyPublishers.ofString(form))
    .build();

HttpResponse<String> resp = client.send(login, HttpResponse.BodyHandlers.ofString());
String protectedHtml = client.send(HttpRequest.newBuilder(URI.create("https://example.com/protected")).GET().build(),
    HttpResponse.BodyHandlers.ofString()).body();

Further reading and tools: MDN on HTTP methods and cookies (POST, Cookies), Java 11 HttpClient docs (HttpClient), jsoup parser (jsoup) and Selenium for JS-heavy flows (Selenium docs). Also respect the site’s terms of service, throttle requests, and store credentials securely.

Recommended Answers

All 4 Replies

Member Avatar for masijade

HttpUrlConnection or HttpsUrlConnection. It is possible to use Cookies and post requests and authentication with both of them. Search for some tutorials concerning them.

Member Avatar for gunbuster363

I think the sign in button is in javascript
I am stucked in here

<a id="signin" href="javascript: SigninSubmit();" tabindex="3">Sign In</a>

Member Avatar for masijade

So look at what that function does.

It's probably no more than a post request. And, if you don't know what I mean by that, then you need to study HTML and Http in general before you continue here.

Edited by masijade because: n/a
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.