Member Avatar for metabo_man

this is probably one of the best places to ask such questions. so i do it now.

first of - i have to explain something; I have to grab some data out of a phpBB in order to do some field reseach. I need the data out of a forum that is runned by a user community. I need the data to analyze the discussions.


to give an example - let us take this forum here. How can i grab all the data out of this forum - and get it local and then after wards put it in a local database - of a phpBB-forum - is this possible"?!"?


Nothing harmeful - nothing bad - nothing serious and dangerous. But the issue is.
i have to get the data - so what?

I need the data in a allmost full and complete formate. So i need all the data like

username .-
forum
thread
topic
text of the posting and so on and so on.

how to do that?

i need some kind of a grabbing tool - can i do it with that kind of tool. How do i sove the storing-issue into the local mysql-database.

Well you see that is a tricky work - and i am pretty sure taht i am getting help here. So for any and all help i am very very thankful

#many many thanks in advance

metabo_man [IMG]http://images.devshed.com/fds/smilies/smile.gif[/IMG]
- a Ethno-reseracher

  • 3 Contributors
  • 4 Replies
  • 168 Views
  • 19 Hours Discussion Span
  • Latest Post Latest Post by metabo_man
Member Avatar for Puckdropper

Ok, first you get a pencil and a piece of paper... Oh you want it done automatically.

If it's your forum, you can simply download the database and do your data mining in that. If it's not your forum, you may find a friendly admin that will help you.

Member Avatar for metabo_man

hi


thanks for the reply.


Ok, first you get a pencil and a piece of paper... Oh you want it done automatically.

yes - but agreed

web automation can be a difficult task;at least it depends on countermeasures implemented. Some will ban accesses based on the user-agent you send, some will use HTTP_REFERER values to restrict 'bots, some will use session-based authorization schemes to keep 'bots from interacting with the site.

but for the automation task we can look at WWW::Mechanize as it encapsulates many of the low-level web automation tools provided by perl.

If it's your forum, you can simply download the database and do your data mining in that. If it's not your forum, you may find a friendly admin that will help you.

yes right - but i need to do it on the fly. I have no time to ask. And to do for some research i want to have some kind of data that is gained without long and controversial debates....

metabo

Member Avatar for iamthwee

to give an example - let us take this forum here. How can i grab all the data out of this forum

Hmm, I'm not sure about how you would grab all the information from this forum, but I can see a way how to do it for individual threads.

All I would do is click the view source code button on your internet browser, then copy that info to a text file.

Then you can just write your own parser (in any language for that matter not just php) to extract the relevant info.

username .-
forum
thread
topic
text of the posting and so on and so on.

Member Avatar for metabo_man

hello


many thanks that is of big help. I do now want to grab all the informations but only that of certain individual threads.


Hmm, I'm not sure about how you would grab all the information from this forum, but I can see a way how to do it for individual threads.

All I would do is click the view source code button on your internet browser, then copy that info to a text file.

Then you can just write your own parser (in any language for that matter not just php) to extract the relevant info.

that sounds interesting - i will look how to solve it. I will dive into the thing and return here - in case of having more questions or ideas or just think that i need more help.

best regards
metabo

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.