a0001 硕士论文 Data Acquisition in Hadoop System.pdf

《a0001 硕士论文 Data Acquisition in Hadoop System》.pdfFaculty of Science and TechnologyMASTER’STHESISStudy program/Specialization:Master puter ScienceSpringsemester, 2010Open/ Restricted accessWriter: Baodong Jia…………………………………………(Writer’s signature)Faculty supervisor:Professor. PhD. Chunming Rong; PhD Candidate. . Tomasz Wiktor WlodarczykExternal supervisor(s):Titelof thesis: Data Acquisition in HadoopSystemCredits (ECTS):30Key words:Hadoop;Data Acquisition;Chukwa;Performance;Real-time data;: …………………+ enclosure: …………Stavanger, ………………..Date/yearContentsContentsiAbstract11 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Problem Description . . . . . . . . . . . . . . . . . . . . . . . . . . Aim of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . Thesis Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . .42 Technologies DataStorm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hadoop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chukwa . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .73 Data How to Acquire Data for Hadoop? . . . . . . . . . . . . . . . . . . . Architecture of Data Acquisition Solution using Chukwa . . . . . . Why Chukwa? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Solution Without Chukwa . . . . . . . . . . . . . . . . . . . Solution With Chukwa . . . . . . . . . . . . . . . . . . . . . Acquire Data From Di?erent Data Sources . . . . . . . . . . . . . . Streaming Data from Twitter . . . . . . . . . . . . . . . . . Historical Data from Service Provider of Statoil . . . . . . .154 Implementation Environment . . . . . . . . . . . . . . . . . . . . . Hadoop Setup . . . . . . . . . . . . . . . . . . . . . . . . . .164.