Member Avatar for gunbuster363

Hi, I am new to here. I am doing with my honours project in University, it is about data mining. Before I can do data mining, first I need data.

I want to extract data from a site : www.tripadvisor.com
Should I write my web crawler with python? I don't know python but I've seen people using this to do it.

I don't need the extracted hyperlinks(Although I need to extract them in the process), I only want to extract the words(String) within the pages. Can python to that? Are there any help with me writing that?


Thank you very much.


Raymond

Edited by gunbuster363 because: n/a
  • 4 Contributors
  • 6 Replies
  • 477 Views
  • 1 Month Discussion Span
  • Latest Post Latest Post by gunbuster363

Dani AI

Generated

A concise, practical plan for extracting visible text and following pagination on a TripAdvisor‑style site, building on the pointers from , and and noting that later succeeded with an HTML parser. Focus on a reliable crawl loop, clear scope, polite behavior, and robust extraction so the crawler keeps working as the site changes.

First, decide scope and traversal rules: allowed domains/path patterns, start URL(s), depth or page limits, and whether to sample or exhaust reviews. Check robots.txt and the site Terms of Service and identify an honest User‑Agent string and contact. Implement politeness (fixed delay or token bucket), concurrency limits, retry/backoff on 429/5xx, and persistent visited state so interrupted runs can resume.

For extraction and pagination: fetch the page, parse the DOM, remove non‑visible nodes (scripts, styles, nav, headers/footers), then select the main content block (heuristics: largest text block or container with many paragraph nodes). Save both raw HTML and cleaned text plus URL/timestamp. Detect the next page by (in order) rel="next", anchor text like "next", or a numeric pagination pattern; fall back to filtering anchors by likely keywords and URL structure. If content is loaded client‑side, add a renderer or target the underlying AJAX/API endpoints rather than scraping rendered HTML.

Example pseudocode (high level):

seed = ["start_url"]
visited = set()
queue = deque(seed)
while queue:
    url = queue.popleft()
    if url in visited: continue
    html = fetch(url, headers={"User-Agent":"project/1.0 (contact@example.com)"})
    dom = parse_html(html)
    text = extract_visible_text(dom)
    store(url, text, html, crawl_time)
    next_link = find_next_link(dom)
    if next_link and next_link not in visited:
        queue.append(next_link)
    visited.add(url)
    sleep(polite_delay)

Troubleshooting tips: handle charset via HTTP/meta, persist raw HTML for debugging, dedupe with content hashing, log HTTP status and response sizes, and be prepared for CAPTCHAs or login walls. When in doubt about scale or legality, prefer official APIs or request permission.

Recommended Answers

All 6 Replies

Member Avatar for jlm699

You have a lot of options. If you're looking to purely look at html you can use urllib2, or if you'd rather have the module parse out all the elements for you and give you purely the text data you'd be better off using beautifulsoup. Search this forum to find plenty of examples of using both.

Member Avatar for gunbuster363

bump

Member Avatar for willygstyle

Yes it can grab specific links, but it can't do it without instructions. You are going to have to tell it where to crawl and why. I'm not sure what you mean by that link is encrypted, but to me it looks like it could have valuable information that you could use in your logic. Such as "showuserreviews", "place", "hongkong". Possibly extract all the links then look for the ones with these kinds of keywords and use that to decide where to go next.

Member Avatar for somnia

Try Scrapy.

It's a very simple (though quite powerful) web crawling and screen scraping framework for Python. It's also pretty well documented, and has a growing community.

Member Avatar for gunbuster363

I've already be able to use BeautifulSoup to write it.
Thank you all!

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.