Web数据挖掘

INTRODUCTION

1.1CrawlingandIndexing6

1.2TopicDirectories7

1.3ClusteringandClassification8

1.4HyperlinkAnalysis9

1.5ResourceDiscoveryandVerticalPortals11

1.6Structuredvs.UnstructuredDataMining11

1.7BibliographicNotes13

PARTⅠINFRASTRUCTURE

2CRAWLINGTHEWEB

2.1HTMLandHTTPBasics18

2.2CrawlingBasics19

2.3EngineeringLarge-ScaleCrawlers21

2.3.1DNSCaching,Prefetching,andResolution22

2.3.2MultipleConcurrentFetches23

2.3.3LinkExtractionandNormalization25

2.3.4RobotExclusion26

2.3.5EliminatingAlready-VisitedURLs26

2.3.6SpiderTraps28

2.3.7AvoidingRepeatedExpansionofLinksonDuplicatePages29

2.3.8LoadMonitorandManager29

2.3.9Per-ServerWork-Queues30

2.3.10TextRepository31

2.3.11RefreshingCrawledPages33

2.4PuttingTogetheraCrawler35

2.4.1DesignoftheCoreComponents35

2.4.2CaseStudy:Usingw3c-libwww40

2.5BibliographicNotes40

3WEBSEARCHANDINFORMATIONRETRIEVAL

3.1BooleanQueriesandtheInvertedIndex45

3.1.1StopwordsandStemming48

3.1.2BatchIndexingandUpdates49

3.1.3IndexCompressionTechniques51

3.2RelevanceRanking53

3.2.1RecallandPrecision53

3.2.2

4.1.1PartitioningApproaches81

4.1.2GeometricEmbeddingApproaches82

4.1.3GenerativeModelsandProbabilisticApproaches83

4.2Bottom-UpandTop-DownPartitioningParadigms84

4.2.1AgglomerativeClustering84

4.2.2Thek-MeansAlgorithm87

4.3ClusteringandVisualizationviaEmbeddings89

4.3.1Self-OrganizingMaps(SOMs)90

4.3.2MultidimensionalScaling(MDS)andFastMap91

4.3.3ProjectionsandSubspaces94

4.3.4LatentSemanticIndexing(LSI)96

4.4ProbabilisticApproachestoClnstermg99

4.4.1GenerativeDistributionsforDocuments101

4.4.2MixtureModelsandExpectationMaximization(EM)103

4.4.3MultipleCauseMixtureModel(MCMM)108

4.4.4AspectModelsandProbabilisticLSI109

4.4.5ModelandFeatureSelection112

4.5CollaborativeFiltering115

4.5.1ProbabilisticModels115

4.5.2CombiningContent-BasedandCollaborativeFeatures117

4.6BibliographicNotes121

5SUPERVISEDLEARNING

5.1TheSupervisedLearningScenario126

5.2OverviewofClassificationStrategies128

5.3EvaluatingTextClassifiers129

5.3.1Benchmarks130

5.3.2MeasuresofAccuracy131

5.4NearestNeighborLearners133

5.4.1ProsandCons134

5.4.2IsTFIDFAppropriate?135

5.5FeatureSelection136

6SEMISUPERVISEDLEARNING

6.1ExpectationMaximization178

6.1.1ExperimentalResults179

6.1.2ReducingtheBeliefinUnlabeledDocuments181

6.1.3ModelingLabelsUsingManyMixtureComponents183

6.2LabelingHypertextGraphs184

6.2.1AbsorbingFeaturesfromNeighboringPages185

6.2.2ARelaxationLabelingAlgorithm188

6.2.3AMetricGraph-LabelingProblem193

6.3Co-training195

6.4BibliographicNotes198

PARTⅢAPPLICATIONS

7SOCIALNETWORKANALYSIS

7.1SocialSciencesandBibliometry205

7.1.1Prestige205

7.1.2Centrality206

7.1.3Co-citation207

7.2PageRankandHITS209

7.2.1PageRank209

7.2.2HITS212

7.2.3StochasticHITSandOtherVariants216

7.3ShortcomingsoftheCoarse-GrainedGraphModel219

7.3.1ArtifactsofWebAuthorship219

7.3.2TopicContaminationandDrift223

7.4EnhancedModelsandTechniques225

7.4.1AvoidingTwo-PartyNepotism225

7.4.2OutlierElimination226

7.4.3ExploitingAnchorText227

7.4.4ExploitingDocumentMarkupStructure228

7.5EvaluationofTopicDistillation235

7.5.1HITSandRelatedAlgorithms235

7.5.2EffectofExploitingOtherHypertextFeatures238

7.6MeasuringandModelingtheWeb243

8.4DiscoveringCommunities284

8.4.1BipartiteCoresasCommunities284

8.4.2NetworkFlow/Cut-BasedNotionsofCommunities285

8.5BibliographicNotes288

9THEFUTUREOFWEBMINING

9.1InformationExtraction290

9.2NaturalLanguageProcessing295

9.2.1LexicalNetworksandOntologies296

9.2.2Part-of-SpeechandSenseTagging297

9.2.3ParsingandKnowledgeRepresentation299

9.3QuestionAnswering302

9.4Profiles,Personalization,andCollaboration305

References307

Index327

　　本书是Web挖掘与搜索引擎领域的经典著作，自出版以来深受好评，已经被斯坦福、普林斯顿、卡内基梅隆等世界名校采用为教材。书中首先介绍了Web爬行和搜索等许多基础性的问题，并以此为基础，深入阐述了解决Web挖掘各种难题所涉及的机器学习技术，提出了机器学习在系统获取、存储和分析数据中的许多应用，并探讨了这些应用的优劣和发展前景。全书分析透彻，富于前瞻性，为构建Web挖掘创新性应用奠定了理论和实践基础，既适用于信息检索和机器学习领域的研究人员和高校师生，也是广大Web开发人员的优秀参考书。　　本书是信息检索领域的名著，深入讲解了从大量非结构化Web数据中提取和产生知识的技术。书中首先论述了Web的基础(包括Web信息采集机制、Web标引机制以及基于关键字或基于相似性搜索机制)，然后系统地描述了Web挖掘的基础知识，着重介绍基于超文本的机器学习和数据挖掘方法，如聚类、协同过滤、监督学习、半监督学习，最后讲述了这些基本原理在Web挖掘中的应用。本书为读者提供了坚实的技术背景和最新的知识。　　本书是从事数据挖掘学术研究和开发的专业人员理想的参考书，同时也适合作为高等院校计算机及相关专业研究生的教材。【作者简介】　　SoumenChakrabarti，Web搜索与挖掘领域的知名专家，ACMTransactionsontheWeb副主编。加州大学伯克利分校博士，目前是印度理工学院计算机科学与工程系副教授。曾经供职于IBMAlmaden研究中心，从事超文本数据库和数据挖掘方面的工作。他有丰富的实际项目开发经验，开发了多个Web挖掘系统，并获得了多项美国专利。