Click here to Skip to main content
15,892,697 members

Get HTML of URL and parse it so the website can be viewed offline

dsagner asked:

Open original thread
Hello all,

today I had some business with html parsing. The requested result was: using the java.net.URL class get all html content from http://www.google.com/ and set up a file which can be used to view the website offline. The greatest problem turned out to be "fetching" the html elements attributes like src from an <img></img> tag, href from an
tag etc. So far I have got to the src attribute by using regular expressions and BufferedReader/Writer classes. A code sample:
Java
URL google = new URL("http://www.google.com/");
        BufferedReader in = new BufferedReader(new InputStreamReader(google
                         .openStream()));
        BufferedWriter wr;
        String s = null;
        Pattern p;
        
        p = Pattern.compile(".*<img[^>]*src=\"([^\"]*)",Pattern.CASE_INSENSITIVE);
        Matcher m;
        try {
            wr = new BufferedWriter(new FileWriter("D:/HTMLFile.txt"));
            while ((s = in.readLine()) != null) {
                    m = p.matcher(s);
                    wr.write(s);
                while(m.find()) {
                    System.out.println(m.group(1));
                }
            }
            in.close();
        } catch (IOException ex) {
            Logger.getLogger(JavaNetworking.class.getName()).log(Level.SEVERE, null, ex);
        }

For this particular URL the output is: "/textinputassistant/tia.png"

What I wanted to ask, is can someone give a better example on how to do this? I read on various forums that regex + java is a hidious monster, sort of speak. I have an algorithm in mind that could lighten stuff up for an experienced programmer, unlike me :)...here it is.
- read all html from the URL
- copy to a string variable
- search in string for "<img"
- when "<img"> - copy to new string variable
- search for "src" or "href" attribute
- extract the attributes value (System.out.println("..") will do just fine for now)
I see this is an idiot-proof problem since I think that this could work out just fine like this, but still I think it's better to ask for an oppinion from a community made of waaay bigger professionals :)
Tags: Java, HTML, Eclipse, Parsing, Netbeans

Plain Text
ASM
ASP
ASP.NET
BASIC
BAT
C#
C++
COBOL
CoffeeScript
CSS
Dart
dbase
F#
FORTRAN
HTML
Java
Javascript
Kotlin
Lua
MIDL
MSIL
ObjectiveC
Pascal
PERL
PHP
PowerShell
Python
Razor
Ruby
Scala
Shell
SLN
SQL
Swift
T4
Terminal
TypeScript
VB
VBScript
XML
YAML

Preview



When answering a question please:
  1. Read the question carefully.
  2. Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
  3. If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
  4. Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.
Let's work to help developers, not make them feel stupid.
Please note that all posts will be submitted under the http://www.codeproject.com/info/cpol10.aspx.



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900