Procesiranje podataka na Hadoopu

master's thesis

• Title (english): Spark and In-memory data processing in Hadoop
• Author: Jurica Potočnjak
• Mentor: Dijana Oreški (mentor); MBZ: 318615
• Committee member: Kornelije Rabuzin; MBZ: 246472
• Granter: University of Zagreb; Faculty of Organization and Informatics; (Department of Information Systems Development); Varaždin
• Defense date and country: 2016-09-23, Croatia
• Scientific / art field, discipline and subdiscipline: SOCIAL SCIENCES; Information and Communication Sciences; Organization and Informatics

Abstract

U teorijskom djelu rada je opisan Apache Hadoop, tehnologija za pohranu i obradu velikih količina podataka. Prikazan je Hadoop te njegov ekosistem preko Clouderine distiribucije Hadoopa. Rad se temelji na prikazu obrade, streaminga, spremanja podataka u realnom vremenu na Hadoop preko servisa Hadoop-ovog ekosistema kao štu su Spark, Flume, Kafka, Solr. Prikazani su primjeri real time streaminga podataka pomoću Flume alata na HDFS, Kafke na Vertica bazu podataka. U praktičnom djelu rada je prikazana vizualizacija u realnom vremenu kroz Solr pomoću Flume servisa te Morphline parsera. Na kraju je prikazan primjer streaminga IPTV JSON logova u realnom vremenu kroz FLume, Spark streaming, Spark SQL u Hive bazu podataka na Hadoopu.

• Keywords: skladištenje podataka; obrada podataka; hadoop; veliki podaci
• Language: croatian
• URN:NBN: urn:nbn:hr:211:172233
• Study programme: Title: Database and Knowledge Base Studies; Study programme type: university; Study level: graduate; Academic / professional title: magistar/magistra informatike (magistar/magistra informatike)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Access restricted to students and staff of home institution
• Created on: 2017-01-26 13:52:35