Member Avatar for Nadeem_2

Hi folks

I want to universal website crawler using PHP, so my crawler will work on any given site.
By using my web application, user will input any site, will provide input, what he needs to get from given site and will click on Start button. Then my web application will begin to get data from source website.
I am using iframe for this purpose , load page in iframe and using jquery I get class and tags name of spacific area from user. But when I load external website like ebay or amazon etc it does not work, as these site are restricted.
Is there any way to resolve this issue, so I can load any site in iFrame. If there is any alternative to what I want to achieve.

I am actually inspired with mozenda, a software developed in .NET, http://www.mozenda.com/video01-overview/. They load a site in browser control and almost doing same thing.
Please help me on this!!

Thank you

  • 2 Contributors
  • 1 Reply
  • 408 Views
  • 4 Hours Discussion Span
  • Latest Post Latest Post by cereal
Member Avatar for cereal

This happens because of the X-Frame-Options header, if the remote server uses a directive like this:

Header always append X-Frame-Options SAMEORIGIN

The browser will check and won't allow the iframes, unless you don't use a local script through Greasemonkey or Tampermonkey. It means your users should install it on their browsers to bypass that kind of block.

Otherwise you could contact some of these website to be allowed, they can create a directive like this:

Header always append X-Frame-Options ALLOWFROM http://yourwebsite.tld/

DOCS:
* https://developer.mozilla.org/en-US/docs/HTTP/X-Frame-Options
* http://tools.ietf.org/html/rfc7034

Edited by cereal
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.