Data Science for Big Data Application and Analytics MOOC

Overview

This section is meant to give an overview of the python tools needed for doing for this course. These are really powerful tools which every data scientist who wishes to use python must know.

Python for X+Informatics: IntroductionPython Packages and Tools

• Canopy IDE and IPython
• NumPy
• Matplotlib
• SciPy

​Virtual Box Appliance

• Virtual Box Appliance with all packages set+up
• Tutorial to use the virtual box appliance
• You can manually setup the necessary stuff or use the virtual box appliance

Python for X-Informatics: Introduction to Canopy

Download Canopy

• https://www.enthought.com/store/
• Versions
  • Express
  • Basic
  • Professional

Getting Necessary Packages

NumPy is a python package which can be used to represent data in a vectorized format. A number of other packages are built on top of numpy. These packages use the data structures and operations used in numpy.The express version of canopy (the one recommended for this class) comes with its own version of NumPy.

Matplotlib is a popular plotting package and is used for creating graphs and plots for data visualization. It can be used with ipython is an interactive manner and proves to be very useful. It has interactive features like zooming, panning. It also supports all the popular graphic formats like JPEG, GIF, TIFF, etc and can be used to plot, modify, images. If you are familiar with MATLAB, you can do almost all kinds of plotting possible with MATLAB using matplotlib.

Scipy is a powerful python package built on top of numpy. It has a number of algorithms pertaining to different fields like calculus, stats,signal processing, image processing, etc.

X-Informatics: Radar Informatics (with application to glaciology)
Jerome E. Mitchell
School of Informatics and Computing
Indiana University Bloomington

Motivation

• CHANGING (DISAPPEARING OUTLET GLACIERS)
• DATA ANALYTICS INVOLVES IMAGE PROCESSING AFTER DETAILED PROCESSING OF RADAR DATA
• EXAMPLE OF LARGE (PETABYTE TODAY) GEOLOCATED DATASET

Outline

• Background – Remote Sensing
• Background – Global Climate Change
• Ice Sheet Science
• Radar Overview
• Radar Basics
• Radar Informatics

Overview

Geoffrey gives a short introduction to the course covering it’s content and structure. He presents the X-Informatics fields (defined values of X) and the Rallying cry of course: Use Clouds running Data Analytics Collaboratively processing Big Data to solve problems in X-Informatics (or e-X). The courses is set up as a MOOC divided into units that vary in length but are typically around an hour and those are further subdivided into 5-15 minute lessons.

The course covers a mix of applications (the X in X-Informatics) and technologies needed to support the field electronically i.e. to process the application data. The introduction ends with a discussion of course content at highest level. The course starts with a longish Motivation unit summarizing clouds and data science, then units describing applications (X = Physics, e-Commerce, Web Search and Text mining, Health, Sensors and Remote Sensing). These are interspersed with discussions of infrastructure (clouds) and data analytics (algorithms like clustering and collaborative filtering used in applications)

The course uses either Python or Java and there are Side MOOCs discussing Python and Java tracks.

Big Data Ecosystem in One Sentence

The Course: Overview

The Approach

The Course Structure

The Topics of Big Data & X-Informatics

0: Motivation

3 Units: X-Informatics Introduction: What is Big Data, Data Analytics and X-Informatics? PartsI-III

​

Python for Big Data and X-Informatics: NumPy, SciPy, MatPlotlib

​

4 Units: X-Informatics Physics Use Case, Discovery of Higgs Particle; Counting Events and Basic Statistics Parts I-IV

Using Plotviz Software for Displaying Point Distributions in 3D

3 Units: X-Informatics Case Study: e-Commerce and Life Style Informatics: Recommender Systems Parts I-III

2 Units: X-Informatics Technologies: K-Nearest Neighbor Algorithms and Clustering

Parallel Computing: Overview of Basic Principles with Familiar Examples

3 Units: X-Informatics Cloud Technology Parts I-III

2 Units: X-Informatics Web Search and Text Mining Parts I and II

X-Informatics Technologies

X-Informatics: Health Informatics

X-Informatics: Sensors

X-Informatics: Radar Informatics (with application to glaciology)

Overview

Geoffrey motivates the study of X-informatics by describing data science and clouds. He starts with striking examples of the data deluge with examples from research, business and the consumer. The growing number of jobs in data science is highlighted. He describes industry trend in both clouds and big data.

He introduces the cloud computing model developed at amazing speed by industry. The 4 paradigms of scientific research are described with growing importance of data oriented version. He covers 3 major X-informatics areas: Physics, e-Commerce and Web Search followed by a broad discussion of cloud applications. Parallel computing in general and particular features of MapReduce are described. He comments on a data science education and the benefits of using MOOC’s.

Introduction

Data Deluge

Jobs

Industry Trends

Computing Model: Industry Adopted Clouds Which Are Attractive for Data Analytics

Research Model: 4th Paradigm; From Theory to Data Driven Science?

Data Science Process

Physics Informatics Looking for Higgs Particle with Large Hadron Collider LHC

Recommender Systems

Web Search Information Retrieval

Cloud Applications in Research

Parallel Computing and MapReduce

​

Data Science Education

Conclusions

Overview

Geoffrey starts with X-Informatics and its rallying cry. The growing number of jobs in data science is highlighted. This unit offers a look at the phenomenon described as the Data Deluge starting with its broad features. Then he discusses data science and the famous DIKW (Data to Information to Knowledge to Wisdom) pipeline. Then more detail is given on the flood of data from Internet and Industry applications with eBay and General Electric discussed in most detail.

X-Informatics and Its Motto

Jobs

Data Deluge: General Structure

Data Science Process

Data Deluge: Internet

Data Deluge: Business

Overview

Geoffrey continues the discussion of the data deluge with a focus on scientific research. He takes a first peek at data from the Large Hadron Collider considered later as physics Informatics and gives some biology examples. He discusses the implication of data for the scientific method which is changing with the data-intensive methodology joining observation, theory and simulation as basic methods.

We discuss the long tail of sciences; many users with individually modest data adding up to a lot. The last lesson emphasizes how everyday devices -- the Internet of Things -- are ???

Data Deluge: Science & Research

Data Deluge: Implications for Scientific Method

Data Deluge: Long Tail of Science

Data Deluge: Internet of Things

Overview

Geoffrey starts with X-Informatics and its rallying cry. The growing number of jobs in data Geoffrey gives an initial technical overview of cloud computing as pioneered by companies like Amazon, Google and Microsoft with new centers holding up to a million servers. The benefits of Clouds in terms of power consumption and the environment are also touched upon, followed by a list of the most critical features of Cloud computing with a comparison to supercomputing.

He discusses features of the data deluge with a salutary example where more data did better than more thought. Examples are given of end to end systems to process big data. He introduces data science and one part of it -- data analytics -- the large algorithms that crunch the big data to give big wisdom. There are many ways to describe data science and several are discussed to give a good composite picture of this emerging field.

Clouds

Features of Data Deluge

Processing Big Data

Data Science Process

Data Analytics

http://cra.org/ccc/docs/nitrdsymposium/pdfs/keyes.pdf My Note: Page not found!

Overview

This section is meant to give an overview of the python tools needed for doing for this course. These are really powerful tools which every data scientist who wishes to use python must know.

Python for X+Informatics: Introduction

Python Packages and Tools

• Canopy IDE and IPython
• NumPy
• Matplotlib
• SciPy

​Virtual Box Appliance

• Virtual Box Appliance with all packages set+up
• Tutorial to use the virtual box appliance
• You can manually setup the necessary stuff or use the virtual box appliance

Python for X-Informatics: Introduction to Canopy

Download Canopy

• https://www.enthought.com/store/
• Versions
  • Express
  • Basic
  • Professional

Getting Necessary Packages

NumPy is a python package which can be used to represent data in a vectorized format. A number of other packages are built on top of numpy. These packages use the data structures and operations used in numpy.The express version of canopy (the one recommended for this class) comes with its own version of NumPy.

Matplotlib is a popular plotting package and is used for creating graphs and plots for data visualization. It can be used with ipython is an interactive manner and proves to be very useful. It has interactive features like zooming, panning. It also supports all the popular graphic formats like JPEG, GIF, TIFF, etc and can be used to plot, modify, images. If you are familiar with MATLAB, you can do almost all kinds of plotting possible with MATLAB using matplotlib.

Scipy is a powerful python package built on top of numpy. It has a number of algorithms pertaining to different fields like calculus, stats,signal processing, image processing, etc.

Overview

In this short unit Geoffrey describes the LHC accelerator at CERN and evidence found by the experiments ATLAS suggesting existence of a Higgs Boson. The huge number of authors on a paper, remarks on histograms and Feynman diagrams is followed by an accelerator picture gallery.

Physics Informaics: Looking for Higgs Particle and Counting Errors Introduction

Physics Informatics: Looking for Higgs Particle Experiments

Physics Informatics: Accelerator Picture Gallery of Big Science

Overview

This unit is devoted to Python experiments with Geoffrey looking at histograms of Higgs Boson production with various forms of shape of signal and various background and with various event totals.

Event Counting

Examples of Event Counting: With Python Examples of Signal Plus Background

Overview

Geoffrey introduces random variables and some simple principles of statistics and explains why they are relevant to Physics counting experiments. The unit introduces Gaussian (normal) distributions and explains why they seen so often in natural phenomena. Several Python illustrations are given.

Fundamental Idea: Random Variables

Physics and Random Variables

Statistics of Events with Normal Distributions

Overview

Geoffrey discusses Random Numbers with their Generators and Seeds. It introduces Binomial and Poisson Distribution. Monte-Carlo and accept-reject methods are discussed. The Central Limit Theorem concludes discussion. Python examples and Physics applications are given.

Random Numbers: Generators and Seeds

Random Numbers: Binomial Distribution

Random Numbers: Accept-Reject

Random Numbers: Monte Carlo Method

Random Numbers: Poisson Distribution

Random Numbers: Central Limit Theorem

Overview

Geoffrey introduces Plotviz, a data visualization tool developed at Indiana University to display 2 and 3 dimensional data. The motivation is that the human eye is very good at pattern recognition and can ‘’see’’ structure in data. Although most Big data is higher dimensional than 3, all can be transformed by dimension reduction techniques to 3D. He gives several examples to show how the software can be used and what kind of data can be visualized. This includes individual plots and the manipulation of multiple synchronized plots. Finally, he describes the download and software dependency of Plotviz.

Data Sets

• http://salsahpc.indiana.edu/
• http://salsahpc.indiana.edu/pviz3/
• http://salsahpc.indiana.edu/pviz3/#downloads
• http://salsahpc.indiana.edu/pviz3/da...iz3_sample.zip
• http://salsahpc.indiana.edu/pviz3/#samples
• http://salsahpc.indiana.edu/pviz3/da...iz3_sample.zip
• http://salsahpc.indiana.edu/pviz3/#screenshots My Note: Main Data Source and Documentation (PDF Papers)
  • http://salsahpc.indiana.edu/pviz3/data/cube.txt
  • http://salsahpc.indiana.edu/pviz3/data/Oil.out.txt
  • http://salsahpc.indiana.edu/pviz3/data/trading.txt
  • http://salsahpc.indiana.edu/pviz3/da..._property.pviz XML to Excel
  • http://salsahpc.indiana.edu/pviz3/da...lclusters.pviz XML to Excel
• http://salsabiology.blogspot.com/p/introduction.html

Motivation and Introduction to Use

Example of Use I: Cube and Structured Dataset

Example of Use II: Proteomics and Synchronized Rotation

Example of Use III: More Features and Larger Proteomics Sample

Example of Use IV: Tools and Examples

Example of Use V: Final Examples

Overview

Geoffrey introduces Recommender systems as an optimization technology used in a variety of applications and contexts online. They operate in the background of such widely recognized sites as Amazon, eBay, Monster and Netflix where everything is a recommendation. This involves a symbiotic relationship between vendor and buyer whereby the buyer provides the vendor with information about their preferences, while the vendor then offers recommendations tailored to match their needs, to the benefit of both.

There follows an exploration of the Kaggle competition site, other recommender systems and Netflix, as well as competitions held to improve the success of the Netflix recommender system. Finally attention is paid to models that are used to compare how changes to the systems affect their overall performance. Geoffrey muses how the humble ranking has become such a dominant driver of the world’s economy.

Recommender Systems as an Optimization Problem

Recommender Systems Introduction

Kaggle Competitions

http://www.kaggle.com

Examples of Recommender Systems

Netflix on Recommender Systems

Overview

Geoffrey continues the discussion of recommender systems and their use in e-commerce. More examples are given from Google News, Retail stores and in depth Yahoo! covering the multi-faceted criteria used in deciding recommendations on web sites. Then the formulation of recommendations in terms of points in a space or bag is given. Here bags of item properties, user properties, rankings and users are useful. Then we go into detail on basic principles behind recommender systems: user-based collaborative filtering, which uses similarities in user rankings to predict their interests, and the Pearson correlation, used to statistically quantify correlations between users viewed as points in a space of items.

Recap of Recommender Systems

Examples of Recommender Systems

Case Study of Yahoo Recommender Systems

Vector Space Formulation of Recommender Systems

User-based nearest-neighbor collaborative filtering

Overview

Geoffrey introduces Recommender systems as an optimization technology used in a Geoffrey moves on to item-based collaborative filtering where items are viewed as points in a space of users. The Cosine Similarity is introduced, the difference between implicit and explicit ratings and the k Nearest Neighbors algorithm. General features like the curse of dimensionality in high dimensions are discussed

Item-based Collaborative Filtering

Technologies: k-Nearest Neighbors and High Dimensional Spaces

Overview

Geoffrey discusses a simple Python k Nearest Neighbor code and its application to an artificial data set in 3 dimensions. Results are visualized in Matplotlib in 2D and with Plotviz in 3D. The concept of training and testing sets are introduced with training set prelabelled.

Python K-Nearest Neighbor Algorithms

Visualization

Testing K Nearest-Neighbor Algorithms

Overview

Geoffrey uses example of recommender system to discuss clustering. The details of methods are not discussed but k-means based clustering methods are used and their results examined in Plotviz. The original labelling is compared to clustering results and extension to 28 clusters given. General issues in clustering are discussed including local optima, the use of annealing to avoid this and value of heuristic algorithms.

Kmeans Clustering

Clustering of Recommender System Example  

Clustering Into More Than Three Clusters

Local Optima in Clustering

Clustering in General

Heuristics

Overview

Geoffrey describes the central role of Parallel computing in Clouds and Big Data which is decomposed into lots of ‘’Little data’’ running in individual cores. Many examples are given and it is stressed that issues in parallel computing are seen in day to day life for communication, synchronization, load balancing and decomposition.

Decomposition

Parallel Processing in Society

Parallel Processing for Hadrian’s Wall

Overview

Geoffrey discusses Cyberinfrastructure for e-moreorlessanything or moreorlessanything- Informatics and the basics of cloud computing. This includes virtualization and the important ‘’as a Service’’ components and we go through several different definitions of cloud computing.

Gartner’s Technology Landscape includes hype cycle and priority matrix and covers clouds and Big Data. Geoffrey reviews the nature of 48 technologies in 2012 emerging technology hype cycle. Gartner has specific predictions for cloud computing growth areas. The unit concludes with two simple examples of the value of clouds for enterprise applications.

Cyberinfrastructure for e-more or less anything or more or less anything-Informatics

What is Cloud Computing: Introduction

Gartner’s Technology Landscape

Simple Examples

Overview

Geoffrey covers different views as to nature of Cloud Computing. This IaaS (Infrastructure as a Service) discussion is followed by PaaS and SaaS (Platform and Software as a Service). The unit discusses features introduced in Grid computing and features introduced by clouds. The unit concludes with the treatment of data in the cloud from an architecture perspective.

What is Cloud Computing In More Detail

As a Service and Platform Model: IaaS PaaS SaaS

Platform as a Service

Data in the Cloud

Overview

Geoffrey opens up with a discussion of Cloud (Data Center) Architectures with physical setup, Green Computing issues and software models. A discussion of Cloud Industry stakeholders is followed by applications on the cloud including data intensive problems and comparison with high performance computing. Remarks on Security, Fault Tolerance and Synchronicity issues in cloud follow. The Big Data Processing from an application perspective with commercial examples including eBay concludes unit.

Cloud (Data Center) Architectures

Cloud Industry Players

Cloud Applications

Security

Comments on Fault Tolerance and Synchronicity Constraints

Big Data Processing From Application Perspective Technology Discussed Earlier

Overview

Geoffrey starts this unit with an overview of data mining and noting our study of classification, clustering and exploration methods. We examine the problem to be solved in web and text search and note the relevance of history with libraries, catalogs and concordances. An overview of web search is given describing the continued evolution of search engines and the relation to the field of Information Retrieval.

The importance of recall, precision and diversity is discussed. The important Bag of Words model is introduced and both Boolean queries and the more general fuzzy indices. The important vector space model and revisiting the Cosinne Similarity as a distance in this bag follows. The basic TF-IDF approach is discussed. Relevance is discussed with a probabilistic model while the distinction between Bayesian and frequency views of probability distribution completes this unit.

Data Mining Survey

DIKW

Web Search Solution in General Starting with History

Information Retrieval (Web Search) Technology

Boolean Query

Fuzzy Index

Vector Space Model

Probabilistic Models

Frequency Versus Bayes

​

Overview

Geoffrey starts with an overview of the different steps (data analytics) in web search and then goes key steps in detail starting with document preparation. An inverted index is described and then how it is prepared for web search. The Boolean and Vector Space approach to query processing follow.

This is followed by Link Structure Analysis including Hubs, Authorities and PageRank. The application of PageRank ideas as reputation outside web search is covered. The web graph structure, crawling it and issues in web advertising and search follow. The use of clustering and topic models completes unit.

Data Analytics for Web Search

Document Preparation

Inverted Index

Index Construction

Query Structure and Processing

Link Structure Analysis including PageRank

​

Summary-Issues

Crawling the Web

Web Advertising and Search

Clustering and Topic Models

Overview

Geoffrey uses Python to Calculate PageRank from Web Linkage Matrix showing several different formulations of the basic matrix equations to finding leading eigenvector. The unit is concluded by a calculation of PageRank for general web pages by extracting the secret from Google.

PageRank in Python

Overview

Geoffrey uses the K-means Python code in SciPy package to show real code for clustering. After a simple example we generate 4 clusters of distinct centers and various choice for sizes using Matplotlib tor visualization. We show results can sometimes be incorrect and sometimes make different choices among comparable solutions. We discuss the ‘’hill’’ between different solutions and rationale for running K-means many times and choosing best answer.

Kmeans in Python

Analysis of 4 Artificial Clusters

Overview

Geoffrey’s introduction to MapReduce describes the basic architecture and a homely example. The discussion of advanced topics includes extension to Iterative MapReduce from Indiana University called Twister and a generalized Map Collective model. Some measurements of parallel performance are given.

MapReduce

Overview

Geoffrey modifies the SciPy K-means code to support a MapReduce execution style and runs it in this short unit. This illustrates the key ideas of mappers and reducers. With appropriate runtime this code would run in parallel but here the ‘’parallel’’ maps run sequentially. Geoffrey stresses that this simple 2 map version can be generalized to scalable parallelism.

MapReduce Kmeans in Python

Overview

Geoffrey starts by discussing general aspects of Big Data and Health including data sizes, different areas including genomics, EBI, radiology and the Quantified Self movement. We survey an April 2013 McKinsey report on the Big Data revolution in US health care; a Microsoft report in this area and a European Union report on how Big Data will allow patient centered care in the future. Some remarks on Cloud computing and Health focus on security and privacy issues. The final topic is Genomics, Proteomics and Information Visualization.

Big Data and Health

McKinsey Report on the big data revolution in US healthcare

Microsoft Report on Big Data in Health

EU Report on Redesigning Health in Europe for 2020

Clouds and Health

Genomics, Proteomics and Information Visualization

Overview

Geoffrey starts with the Internet of Things giving examples like monitors of machine operation, QR codes, surveillance cameras, scientific sensors, drones and self driving cars and more generally transportation systems. Sensor clouds control these many small distributed devices. More detail is given for radar data gathered by sensors; ubiquitous or smart cities and homes including U-Korea; and finally the smart electric grid.

Internet of Things

Sensor Clouds

Radar Data Gathered by Sensors

Ubiquitous/Smart Cites

U-Korea.
U.=.Ubiquitous.
See:
http://www.academia.edu/1731078/
Korean_Ubiquitous_Society_Visions

Smart Grid

Overview

The changing global climate is suspected to have long-term effects on much of the world’s inhabitants. Among the various effects, the rising sea level will directly affect many people living in low-lying coastal regions. While the ocean-s thermal expansion has been the dominant contributor to rises in sea level, the potential contribution of discharges from the polar ice sheets in Greenland and Antarctica may provide a more significant threat due to the unpredictable response to the changing climate. The Radar-Informatics unit provides a glimpse in the processes fueling global climate change and explains what methods are used for ice data acquisitions and analysis.

X-Informatics: Radar Informatics (with application to glaciology)
Jerome E. Mitchell
School of Informatics and Computing
Indiana University Bloomington

Motivation

• CHANGING (DISAPPEARING OUTLET GLACIERS)
• DATA ANALYTICS INVOLVES IMAGE PROCESSING AFTER DETAILED PROCESSING OF RADAR DATA
• EXAMPLE OF LARGE (PETABYTE TODAY) GEOLOCATED DATASET

Outline

• Background – Remote Sensing
• Background – Global Climate Change
• Ice Sheet Science
• Radar Overview
• Radar Basics
• Radar Informatics