Please join us at the new JavaWorld Q&A Forums. Your existing login will work there. The discussions here are now read-only.


JavaWorld Talkback >> 959482

Pages: 1
JavaWorld
addict


Reged: 06/20/03
Posts: 482
Create intelligent Web spiders
      #12999 - 10/31/04 12:53 AM

Create intelligent Web spiders

Post Extras: Print Post   Remind Me!   Notify Moderator  
Anonymous
Unregistered




Re: Create intelligent Web spiders [Re: JavaWorld]
      #13029 - 11/01/04 07:14 AM

I can think of an improvment by checking the stream content type instead of the file extention. It should be "text/html" and then you can use HttpURLConnection or HttpsURLConnection for getting the html content.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Anonymous
Unregistered




Re: Create intelligent Web spiders [Re: JavaWorld]
      #13034 - 11/01/04 09:58 AM

Thanks for the great code! However, I miss the separation between Model, View and Controller. If somebody wants to reuse this code in a slightly different context, he will have to do lots of copy paste!

Post Extras: Print Post   Remind Me!   Notify Moderator  
Anonymous
Unregistered




Re: Create intelligent Web spiders [Re: JavaWorld]
      #13232 - 11/11/04 08:25 AM

Nice article. Code is interesting to play with. Thank you.

Post Extras: Print Post   Remind Me!   Notify Moderator  
Mark Pendergast
Unregistered




Re: Create intelligent Web spiders [Re: Anonymous]
      #13727 - 11/30/04 02:16 PM

Thanks. That's a great suggestion. Takes some of the guess work out.

Post Extras: Print Post   Remind Me!   Notify Moderator  
Anonymous
Unregistered




Re: Create intelligent Web spiders [Re: Mark Pendergast]
      #25759 - 01/09/06 09:00 PM

Hi

Problem is that what if you just needed .html files and not say jsp files. Both return text/html as the contenttype

Nick


Post Extras: Print Post   Remind Me!   Notify Moderator  
Pages: 1



Extra information
0 registered and 1 anonymous users are browsing this forum.

Moderator:   

Print Topic

Forum Permissions
      You cannot start new topics
      You cannot reply to topics
      HTML is disabled
      UBBCode is enabled

Rating:
Topic views: 5808

Rate this topic

Jump to

Contact us JavaWorld

Powered by UBB.threads™ 6.5.5