12_Web_as_Corpus
abc
abc
11
0.0 (0)
R. P.
R. P.
This flashcard set covers advanced university-level topics in computer science, focusing on the web as a corpus. It delves into text processing, language identification, error detection, and content extraction, highlighting techniques like shingling for duplicate removal and language models for cleaning web data. Researchers and students in natural language processing and web data analysis will find this flashcard set particularly useful for understanding how to handle and analyze web-based text data effectively.
Cartes-fiches
11
Utilisateurs
2
Langue
Anglais
Catégorie
Informatique
Niveau
Université
Créé / Mis à jour
07.02.2018 / 11.03.2018
Pros and Cons using the Web as a Corpus
Commandes clavier:
= tourner,
= avant/arrière,
= faire défiler
Approaches to the Web as a linguistic corpus
Commandes clavier:
= tourner,
= avant/arrière,
= faire défiler
- Use hit counts provided by a commercial search engine as frequency estimates
- hits("the most torpid heart") = 4
- hits("the most obdurate heart“) = 1 240
- Build corpora through search engine queries
- Linguistic crawls of the Web
Googleology is bad Science
Commandes clavier:
= tourner,
= avant/arrière,
= faire défiler
- This information changes rapidly over time
- Estimates might be unreliable
- Impossible to reproduce
- Only very simple queries are possible (no regular expressions, no lemmatisation or POS tagging of the documents and queries)
- Constraints on the numbers of queries per day
- The pages retrieved depend on the search engine ranking
Cleaning up the Web
Commandes clavier:
= tourner,
= avant/arrière,
= faire défiler
- Web spam filtering
- HTML file pre-processing
- Content extraction
- Duplicate removal
- Content cleaning and filtering: smileys, spelling errors, etc.
- Language identification
HTML file pre-processing
Commandes clavier:
= tourner,
= avant/arrière,
= faire défiler
- HTML File cleaning
- Boilerplate removal
- Boilerplate = "canned or automatically generated text blocks including navigation bars, page headers, link lists, disclaimers and copyright statements, as well as the ubiquitous advertisements"
- Methods:
- Use of HTML structure
- Use of HTML tags
- Use of language models
Content extraction
Commandes clavier:
= tourner,
= avant/arrière,
= faire défiler
- Remove irrelevant parts like: Copyright information, Links to other pages within the same site, Menu and navigation items, Advertisements
- NCLEARER-System
- Two separate character n-gram language models:
- clean text, trained on manually cleaned web pages
- dirty text, trained on uncleaned web pages
- Compute the probability of a text segment based on each language model
- Applied to each text segment in turn:
- boilerplate: the probability calculated by the dirty model is higher than the probability calculated by the clean model
- Two separate character n-gram language models:
Duplicate removal
Commandes clavier:
= tourner,
= avant/arrière,
= faire défiler
- Simplest case:
- perfect duplicates, i.e. the same Web page is available from different URLs
- The online documentation for popular programs.
- Documents stored in several mirror sites.
- Near-duplicates:
- Mostly the same content
- Different layout and design
- Detection:
- Shingling algorithm (Broder et al., 1997)
- Given a positive integer k and a sequence of terms in a document d, define the k-shingles of d to be the set of all consecutive sequences of k terms in d.
- compute shingle fingerprint of each document
- compute super-shingles by shingling over the shingles
- only compare documents that share a super-shingle
- Simplified version of the algorithm (Baroni & Ueyama, 2006)
- Remove all function words in the documents
- Take fingerprints of a fixed number s of randomly selected n-grams (s = 25, n= 5). These n-grams are called shingles
- For each pair of documents count the number of shared n-grams
- Discard one of the two documents for pairs sharing more than t n-grams (t = 2)
- Partial duplicates:
- Quotations
- Copy and paste of text from another Web pages
- Mashups
Language identification
Commandes clavier:
= tourner,
= avant/arrière,
= faire défiler
- Motivation:
- For language-specific processing (morphological analysis, POS tagging, etc.) it is necessary to know the language
- For Web corpus building, pages written in a language which is not the target language have to be filtered out
- Language-specific features
- Closed-class words e.g. determiners, conjunctions, prepositions
- English: the, and, of, ...
- Spanish: de, les, los, el, la, y, ...
- Word endings
- English: -ious, -ism, ...
- French: -eux, -isme, ...
- Closed-class words e.g. determiners, conjunctions, prepositions
- Simple method for language identification: compare the frequency of usage of common words in a text to their frequencies in a large reference corpus
Web corpus building through search engine queries
Commercial search engines have limited search functionalities which make them ill-suited for most research and scientific purposes --> BootCat Method
Commandes clavier:
= tourner,
= avant/arrière,
= faire défiler
Spelling Error Detection and Correction (including Error Types)
Commandes clavier:
= tourner,
= avant/arrière,
= faire défiler
- Types of spelling errors:
- Non-word spelling errors
- occured instead of occurred
- ater instead of after, later, alter, water, ate
- Word conflation or splitting
- ofthe, understandhme
- sp ent, th ebook
- Malapropisms: real-word spelling errors in open-class words
- diary – dairy
- there – their – they're
- Non-word spelling errors
- Interactive spell-checkers:
- return multiple possible corrections and leave the final choice to the user
- Automatic error correction:
- no human intervention is required for applications such as text-to-speech synthesis or text recognition
- Research Problems:
- Non-word error detection
- n-gram analysis
- Dictionary lookup
- Non-word error detection
- Context-dependent word correction
- Isolated-word error correction
- Minimum Edit Distance algorithm
- minimum number of editing operations (i.e., insertions, deletions, and substitutions) needed to transform one string into another
- Minimum Edit Distance algorithm
Conclusion on Web as Corpus
- More data is often better... and there is a lot of Web data
- But this also leads to performance problems
- People from Google or Bing are usually good prepared for this
- The Web is free (if you are able to crawl it) but Web search queries aren't
- Many other uses of Web data:
- Distinguish cognates (verwandte Wörter) from false friends
- Named Entity Gazetteer (geographisches Lexikon) building
- Word Sense Corpus acquisition
- Relation extraction
-
- 1 / 11
-
Cartes-fiches
Pros and Cons using the Web as a Corpus
Approaches to the Web as a linguistic corpus
- Use hit counts provided by a commercial search engine as frequency estimates
- hits("the most torpid heart") = 4
- hits("the most obdurate heart“) = 1 240
- Build corpora through search engine queries
- Linguistic crawls of the Web
Googleology is bad Science
- This information changes rapidly over time
- Estimates might be unreliable
- Impossible to reproduce
- Only very simple queries are possible (no regular expressions, no lemmatisation or POS tagging of the documents and queries)
- Constraints on the numbers of queries per day
- The pages retrieved depend on the search engine ranking
Cleaning up the Web
- Web spam filtering
- HTML file pre-processing
- Content extraction
- Duplicate removal
- Content cleaning and filtering: smileys, spelling errors, etc.
- Language identification
HTML file pre-processing
- HTML File cleaning
- Boilerplate removal
- Boilerplate = "canned or automatically generated text blocks including navigation bars, page headers, link lists, disclaimers and copyright statements, as well as the ubiquitous advertisements"
- Methods:
- Use of HTML structure
- Use of HTML tags
- Use of language models
Content extraction
- Remove irrelevant parts like: Copyright information, Links to other pages within the same site, Menu and navigation items, Advertisements
- NCLEARER-System
- Two separate character n-gram language models:
- clean text, trained on manually cleaned web pages
- dirty text, trained on uncleaned web pages
- Compute the probability of a text segment based on each language model
- Applied to each text segment in turn:
- boilerplate: the probability calculated by the dirty model is higher than the probability calculated by the clean model
- Two separate character n-gram language models:
Duplicate removal
- Simplest case:
- perfect duplicates, i.e. the same Web page is available from different URLs
- The online documentation for popular programs.
- Documents stored in several mirror sites.
- Near-duplicates:
- Mostly the same content
- Different layout and design
- Detection:
- Shingling algorithm (Broder et al., 1997)
- Given a positive integer k and a sequence of terms in a document d, define the k-shingles of d to be the set of all consecutive sequences of k terms in d.
- compute shingle fingerprint of each document
- compute super-shingles by shingling over the shingles
- only compare documents that share a super-shingle
- Simplified version of the algorithm (Baroni & Ueyama, 2006)
- Remove all function words in the documents
- Take fingerprints of a fixed number s of randomly selected n-grams (s = 25, n= 5). These n-grams are called shingles
- For each pair of documents count the number of shared n-grams
- Discard one of the two documents for pairs sharing more than t n-grams (t = 2)
- Partial duplicates:
- Quotations
- Copy and paste of text from another Web pages
- Mashups
Language identification
- Motivation:
- For language-specific processing (morphological analysis, POS tagging, etc.) it is necessary to know the language
- For Web corpus building, pages written in a language which is not the target language have to be filtered out
- Language-specific features
- Closed-class words e.g. determiners, conjunctions, prepositions
- English: the, and, of, ...
- Spanish: de, les, los, el, la, y, ...
- Word endings
- English: -ious, -ism, ...
- French: -eux, -isme, ...
- Closed-class words e.g. determiners, conjunctions, prepositions
- Simple method for language identification: compare the frequency of usage of common words in a text to their frequencies in a large reference corpus
Web corpus building through search engine queries
Commercial search engines have limited search functionalities which make them ill-suited for most research and scientific purposes --> BootCat Method
Spelling Error Detection and Correction (including Error Types)
- Types of spelling errors:
- Non-word spelling errors
- occured instead of occurred
- ater instead of after, later, alter, water, ate
- Word conflation or splitting
- ofthe, understandhme
- sp ent, th ebook
- Malapropisms: real-word spelling errors in open-class words
- diary – dairy
- there – their – they're
- Non-word spelling errors
- Interactive spell-checkers:
- return multiple possible corrections and leave the final choice to the user
- Automatic error correction:
- no human intervention is required for applications such as text-to-speech synthesis or text recognition
- Research Problems:
- Non-word error detection
- n-gram analysis
- Dictionary lookup
- Non-word error detection
- Context-dependent word correction
- Isolated-word error correction
- Minimum Edit Distance algorithm
- minimum number of editing operations (i.e., insertions, deletions, and substitutions) needed to transform one string into another
- Minimum Edit Distance algorithm
Conclusion on Web as Corpus
- More data is often better... and there is a lot of Web data
- But this also leads to performance problems
- People from Google or Bing are usually good prepared for this
- The Web is free (if you are able to crawl it) but Web search queries aren't
- Many other uses of Web data:
- Distinguish cognates (verwandte Wörter) from false friends
- Named Entity Gazetteer (geographisches Lexikon) building
- Word Sense Corpus acquisition
- Relation extraction
