Table of contents
  1. Story
    1. Slides
      1. Slide 1 Data Science for Conservation International's Big Ecosystem Data
      2. Slide 2 Overview
      3. Slide 3 TeamNetwork.Org: Web Page
      4. Slide 4 Data Query and Download: Web Page
      5. Slide 5 Data Query and Download: Files
      6. Slide 6 Earth Insights from Big Data: MindTouch Knowledge Base
      7. Slide 7 Earth Insights from Big Data: Spotfire Cover Page Lidar Data
      8. Slide 8 Earth Insights from Big Data: Spotfire CI Biodiversity Hot Spot Lines
      9. Slide 9 Earth Insights from Big Data: Spotfire CI Biodiversity Hot Spot Polygons
      10. Slide 10 Earth Insights from Big Data: Spotfire Ground Truth
      11. Slide 11 Earth Insights from Big Data: Spotfire BBS
      12. Slide 12 Earth Insights from Big Data: Spotfire Rwanda Terrestrial Vertebrate 2013
      13. Slide 13 Earth Insights from Big Data: Spotfire AV-20150522043510_4688
      14. Slide 14 Conclusions and Recommendations
  2. Spotfire Dashboard
  3. Research Notes
  4. Slides
    1. Slide 1 Deriving Earth Insights from Big Data
    2. Slide 2 Today's Speakers
    3. Slide 3 Big Data is Everywhere...and Growing
    4. Slide 4 HP Vertica: No Linits, no compromises
    5. Slide 5 HO HAVen: No Linits to Future Success
    6. Slide 6 Leading Organizations Driving Analytic Use Cases
    7. Slide 7 Conservation International - conservation.org
    8. Slide 8 We rely on tropical forests...
    9. Slide 9 Tropical Ecology Assessment and Monitoring (TEAM) Network
    10. Slide 10 TEAM Network
    11. Slide 11 TEAM Protocols
    12. Slide 12 How TEAM works
    13. Slide 13 TEAM IT System - Pre HP Earth Insights Partnership
    14. Slide 14 Big Data Challenges
    15. Slide 15 HP Earth Insights with Conservation International
    16. Slide 16 HP Living Progress and Earth Insights
    17. Slide 17 HP Earth Insights with CI
    18. Slide 18 Beyond Big Data-Additional Challenges
    19. Slide 19 Wildlife Picture Index (WPI) Analytics System
    20. Slide 20 WOI Functional Architecture
    21. Slide 21 WPI Interface 1
    22. Slide 22 WOI Interface 2
    23. Slide 23 WPI Interface 3
    24. Slide 24 Generating new analysis and insights
    25. Slide 25 Beyond benefiting conservation
    26. Slide 26 Vertica Big Data Conference 2014
    27. Slide 27 Evaluate HP Vertica 7 Today!
    28. Slide 28 Thank you!
  5. Deriving Earth Insights from Big Data
  6. Finding Meaning in a Deluge of Data
  7. NEXT

Earth Insights from Big Data

Last modified
Table of contents
  1. Story
    1. Slides
      1. Slide 1 Data Science for Conservation International's Big Ecosystem Data
      2. Slide 2 Overview
      3. Slide 3 TeamNetwork.Org: Web Page
      4. Slide 4 Data Query and Download: Web Page
      5. Slide 5 Data Query and Download: Files
      6. Slide 6 Earth Insights from Big Data: MindTouch Knowledge Base
      7. Slide 7 Earth Insights from Big Data: Spotfire Cover Page Lidar Data
      8. Slide 8 Earth Insights from Big Data: Spotfire CI Biodiversity Hot Spot Lines
      9. Slide 9 Earth Insights from Big Data: Spotfire CI Biodiversity Hot Spot Polygons
      10. Slide 10 Earth Insights from Big Data: Spotfire Ground Truth
      11. Slide 11 Earth Insights from Big Data: Spotfire BBS
      12. Slide 12 Earth Insights from Big Data: Spotfire Rwanda Terrestrial Vertebrate 2013
      13. Slide 13 Earth Insights from Big Data: Spotfire AV-20150522043510_4688
      14. Slide 14 Conclusions and Recommendations
  2. Spotfire Dashboard
  3. Research Notes
  4. Slides
    1. Slide 1 Deriving Earth Insights from Big Data
    2. Slide 2 Today's Speakers
    3. Slide 3 Big Data is Everywhere...and Growing
    4. Slide 4 HP Vertica: No Linits, no compromises
    5. Slide 5 HO HAVen: No Linits to Future Success
    6. Slide 6 Leading Organizations Driving Analytic Use Cases
    7. Slide 7 Conservation International - conservation.org
    8. Slide 8 We rely on tropical forests...
    9. Slide 9 Tropical Ecology Assessment and Monitoring (TEAM) Network
    10. Slide 10 TEAM Network
    11. Slide 11 TEAM Protocols
    12. Slide 12 How TEAM works
    13. Slide 13 TEAM IT System - Pre HP Earth Insights Partnership
    14. Slide 14 Big Data Challenges
    15. Slide 15 HP Earth Insights with Conservation International
    16. Slide 16 HP Living Progress and Earth Insights
    17. Slide 17 HP Earth Insights with CI
    18. Slide 18 Beyond Big Data-Additional Challenges
    19. Slide 19 Wildlife Picture Index (WPI) Analytics System
    20. Slide 20 WOI Functional Architecture
    21. Slide 21 WPI Interface 1
    22. Slide 22 WOI Interface 2
    23. Slide 23 WPI Interface 3
    24. Slide 24 Generating new analysis and insights
    25. Slide 25 Beyond benefiting conservation
    26. Slide 26 Vertica Big Data Conference 2014
    27. Slide 27 Evaluate HP Vertica 7 Today!
    28. Slide 28 Thank you!
  5. Deriving Earth Insights from Big Data
  6. Finding Meaning in a Deluge of Data
  7. NEXT

  1. Story
    1. Slides
      1. Slide 1 Data Science for Conservation International's Big Ecosystem Data
      2. Slide 2 Overview
      3. Slide 3 TeamNetwork.Org: Web Page
      4. Slide 4 Data Query and Download: Web Page
      5. Slide 5 Data Query and Download: Files
      6. Slide 6 Earth Insights from Big Data: MindTouch Knowledge Base
      7. Slide 7 Earth Insights from Big Data: Spotfire Cover Page Lidar Data
      8. Slide 8 Earth Insights from Big Data: Spotfire CI Biodiversity Hot Spot Lines
      9. Slide 9 Earth Insights from Big Data: Spotfire CI Biodiversity Hot Spot Polygons
      10. Slide 10 Earth Insights from Big Data: Spotfire Ground Truth
      11. Slide 11 Earth Insights from Big Data: Spotfire BBS
      12. Slide 12 Earth Insights from Big Data: Spotfire Rwanda Terrestrial Vertebrate 2013
      13. Slide 13 Earth Insights from Big Data: Spotfire AV-20150522043510_4688
      14. Slide 14 Conclusions and Recommendations
  2. Spotfire Dashboard
  3. Research Notes
  4. Slides
    1. Slide 1 Deriving Earth Insights from Big Data
    2. Slide 2 Today's Speakers
    3. Slide 3 Big Data is Everywhere...and Growing
    4. Slide 4 HP Vertica: No Linits, no compromises
    5. Slide 5 HO HAVen: No Linits to Future Success
    6. Slide 6 Leading Organizations Driving Analytic Use Cases
    7. Slide 7 Conservation International - conservation.org
    8. Slide 8 We rely on tropical forests...
    9. Slide 9 Tropical Ecology Assessment and Monitoring (TEAM) Network
    10. Slide 10 TEAM Network
    11. Slide 11 TEAM Protocols
    12. Slide 12 How TEAM works
    13. Slide 13 TEAM IT System - Pre HP Earth Insights Partnership
    14. Slide 14 Big Data Challenges
    15. Slide 15 HP Earth Insights with Conservation International
    16. Slide 16 HP Living Progress and Earth Insights
    17. Slide 17 HP Earth Insights with CI
    18. Slide 18 Beyond Big Data-Additional Challenges
    19. Slide 19 Wildlife Picture Index (WPI) Analytics System
    20. Slide 20 WOI Functional Architecture
    21. Slide 21 WPI Interface 1
    22. Slide 22 WOI Interface 2
    23. Slide 23 WPI Interface 3
    24. Slide 24 Generating new analysis and insights
    25. Slide 25 Beyond benefiting conservation
    26. Slide 26 Vertica Big Data Conference 2014
    27. Slide 27 Evaluate HP Vertica 7 Today!
    28. Slide 28 Thank you!
  5. Deriving Earth Insights from Big Data
  6. Finding Meaning in a Deluge of Data
  7. NEXT

Story

Data Science for Conservation International's Big Ecosystem Data

Conservation International and HP Vertica teamed for an excellent online presentation (see Slides below) on the deluge of data from the Tropical Ecology Assessment and Monitoring (TEAM) Network. The TEAM is described in a Vimeo video (see 23:52 - 26:18). The presentation also includes a reference (at 16:41).to an article on The Popularity of Data Analysis Software.

So I mined for these data sets (see Research Notes) to write a Data Science Story.

The results are shown in the Slides and Spotfire Dashboard below.

The Conclusions and Recommendations are:

  • I Data Mined for the best datasets to visualize in Spotfire that come from Vertica and the TEAM Site.
  • The data sets, and their metadata, required extraction from ZIP files and some reformatting for input to Spotfire.
  • I built a MindTouch Knowledge Base and a Spotfire Dashboard to analyze 12 data sets in CSV, Excel, and Shape file formats.
  • The results are shown in 7 Spotfire Tabs with dynamically linked adjacent visualizations.
  • I need Conservation International Subject Matter Expertise to help interpret the Spatial Data and suggestions for more data sets for Data Science.

Slides

Slides

Slide 3 TeamNetwork.Org: Web Page

http://www.teamnetwork.org/

BrandNiemann05252015Slide3.PNG

Slide 4 Data Query and Download: Web Page

http://www.teamnetwork.org/data/query

BrandNiemann05252015Slide4.PNG

Slide 5 Data Query and Download: Files

BrandNiemann05252015Slide5.PNG

Slide 6 Earth Insights from Big Data: MindTouch Knowledge Base

Earth Insights from Big Data

BrandNiemann05252015Slide6.PNG

Slide 7 Earth Insights from Big Data: Spotfire Cover Page Lidar Data

Web Player

BrandNiemann05252015Slide7.PNG

Slide 8 Earth Insights from Big Data: Spotfire CI Biodiversity Hot Spot Lines

Web Player

BrandNiemann05252015Slide8.PNG

Slide 9 Earth Insights from Big Data: Spotfire CI Biodiversity Hot Spot Polygons

Web Player

BrandNiemann05252015Slide9.PNG

Slide 10 Earth Insights from Big Data: Spotfire Ground Truth

Web Player

BrandNiemann05252015Slide10.PNG

Slide 11 Earth Insights from Big Data: Spotfire BBS

Web Player

BrandNiemann05252015Slide11.PNG

Slide 12 Earth Insights from Big Data: Spotfire Rwanda Terrestrial Vertebrate 2013

Web Player

BrandNiemann05252015Slide12.PNG

Slide 13 Earth Insights from Big Data: Spotfire AV-20150522043510_4688

Web Player

BrandNiemann05252015Slide13.PNG

Slide 14 Conclusions and Recommendations

BrandNiemann05252015Slide14.PNG

Spotfire Dashboard

For Internet Explorer Users and Those Wanting Full Screen Display Use: Web Player Get Spotfire for iPad App

Error: Embedded data could not be displayed. Use Google Chrome

Research Notes

I am looking for the best datasets to visualize in Spotfire that come from Vertica and the TEAM Site.

For example I see at: http://www.teamnetwork.org/products/gis

GIS Datasets I could download by agreeing to the Agreement (which I did) and others that say:

Supplementary GIS Data (for select sites)

TEAM Network Members only

Your account does not currently have Network Member privileges.

So should I request permission to work with those?

Riwanda: http://www.teamnetwork.org:28080/gridsphere/gridsphere?cid=download Got a few examples of spreadsheets

Lidar: http://www.teamnetwork.org/data/lidar Got a few examples of images and PDF Metadata

Camera Trap: http://www.teamnetwork.org/camera-trap-downloads Got an example of JPEGs

I tried: http://www.teamnetwork.org/gridsphere/gridsphere?cid=search

And got a popup message that said the file was 132 MB and an email would be sent to download it. I got the 10 MB ZIP download file just now and it looks useful.

Did I miss anything here and are there any other data download sites like TEAM?

http://www.conservation.org/Pages/default.aspx

efegraus@conservation.org

Video: 16:41 http://r4stats.com/articles/popularity

Tropical Ecology Assessment and Monitoring (TEAM) Network

http://blog.conservation.org/2012/07/finding-meaning-in-a-deluge-of-data/

http://www.teamnetwork.org/

http://www.teamnetwork.org/data/query

http://www.teamnetwork.org/products/gis

http://www.teamnetwork.org/data/lidar

http://www.teamnetwork.org:28080/gridsphere/gridsphere?cid=download

http://www.teamnetwork.org/gridsphere/gridsphere?cid=download

http://www.teamnetwork.org/endangered_species_request

http://www.teamnetwork.org/camera-trap-downloads

Slides

Slide 1 Deriving Earth Insights from Big Data

Deriving Earth Insights from Big Data on Vimeo28.png

Slide 2 Today's Speakers

Deriving Earth Insights from Big Data on Vimeo1.png

Slide 3 Big Data is Everywhere...and Growing

Deriving Earth Insights from Big Data on Vimeo2.png

Slide 4 HP Vertica: No Linits, no compromises

Deriving Earth Insights from Big Data on Vimeo3.png

Slide 5 HO HAVen: No Linits to Future Success

Deriving Earth Insights from Big Data on Vimeo4.png

Slide 6 Leading Organizations Driving Analytic Use Cases

Deriving Earth Insights from Big Data on Vimeo5.png

Slide 7 Conservation International - conservation.org

Deriving Earth Insights from Big Data on Vimeo6.png

Slide 8 We rely on tropical forests...

Deriving Earth Insights from Big Data on Vimeo7.png

Slide 9 Tropical Ecology Assessment and Monitoring (TEAM) Network

Deriving Earth Insights from Big Data on Vimeo8.png

Slide 10 TEAM Network

Deriving Earth Insights from Big Data on Vimeo9.png

Slide 11 TEAM Protocols

Deriving Earth Insights from Big Data on Vimeo10.png

Slide 12 How TEAM works

Deriving Earth Insights from Big Data on Vimeo11.png

Slide 13 TEAM IT System - Pre HP Earth Insights Partnership

Deriving Earth Insights from Big Data on Vimeo12.png

Slide 14 Big Data Challenges

Deriving Earth Insights from Big Data on Vimeo13.png

Slide 15 HP Earth Insights with Conservation International

Deriving Earth Insights from Big Data on Vimeo14.png

Slide 16 HP Living Progress and Earth Insights

Deriving Earth Insights from Big Data on Vimeo15.png

Slide 17 HP Earth Insights with CI

My Note: There is a video from 23:52 - 26:18

Deriving Earth Insights from Big Data on Vimeo16.png

Slide 18 Beyond Big Data-Additional Challenges

Deriving Earth Insights from Big Data on Vimeo17.png

Slide 19 Wildlife Picture Index (WPI) Analytics System

Deriving Earth Insights from Big Data on Vimeo18.png

Slide 20 WOI Functional Architecture

Deriving Earth Insights from Big Data on Vimeo19.png

Slide 21 WPI Interface 1

Deriving Earth Insights from Big Data on Vimeo20.png

Slide 22 WOI Interface 2

Deriving Earth Insights from Big Data on Vimeo21.png

Slide 23 WPI Interface 3

Deriving Earth Insights from Big Data on Vimeo22.png

Slide 24 Generating new analysis and insights

Deriving Earth Insights from Big Data on Vimeo23.png

Slide 25 Beyond benefiting conservation

Deriving Earth Insights from Big Data on Vimeo24.png

Slide 26 Vertica Big Data Conference 2014

Deriving Earth Insights from Big Data on Vimeo25.png

Slide 27 Evaluate HP Vertica 7 Today!

Deriving Earth Insights from Big Data on Vimeo26.png

Slide 28 Thank you!

Deriving Earth Insights from Big Data on Vimeo27.png

Deriving Earth Insights from Big Data

Source: http://vimeo.com/93397800

from 

learn about an innovative collaboration that applies HP’s technology and expertise to the research of the Tropical Ecology Assessment and Monitoring Network (TEAM), a partnership led by Conservation International (CI), the Wildlife Conservation Society, and the Smithsonian Institution.

The initiative leverages HP’s big data solutions – powered by the HP Vertica Analytics Platform -- to help TEAM scientists dramatically improve the accuracy and speed of data collection and analysis. What took a team of experts weeks, months, or more to analyze can now be done by a single person in hours. HP Earth Insights has created a first-of-its-kind, early warning system for threatened species by generating usable species trends and delivering near real-time data analytics.

Finding Meaning in a Deluge of Data

Source: http://blog.conservation.org/2012/07...eluge-of-data/

The Tropical Ecology Assessment and Monitoring (TEAM) Network was the cover story for this month’s issue of the journal BioScience. Today on the blog, Eric Fegraus explains why initiatives like TEAM are so ground-breaking.

TEAM scientists collecting data in Tanzania.

TEAM scientists collecting data in Tanzania. (© Benjamin Drummond)

In today’s globalized, technology-driven world, we are inundated with information on a daily basis. Faced with a deluge of data emanating from traditional news outlets, advertising and our social media networks, we must constantly sift through this mountain of data and evaluate which information is the most accurate, important and relevant to our lives.

Within the scientific community, things aren’t much different. Technology has now enabled us to collect near real-time ecological information on scales ranging from a 1-hectare plot to an entire continent via space-, air- and ground-based sensors. Global, regional and national monitoring networks and observing systems can now use these technologies to capture information from more geographies and time periods than ever before.

It wasn’t always this way. Back in 2001, while commuting between graduate school and an ecological think tank where I was working as an intern, I often reflected on how relatively little environmental and ecological data was available. The limited data that did exist was often incompatible, isolated and collected using non-standardized methods. Furthermore, taking the next step — integrating natural resource information with socioeconomic data in order to inform conservation and sustainable development efforts — was mostly done only in small-scale projects.

A decade later, and after spending six years at CI working on TEAM, it’s exciting to see how much has changed. It’s becoming clear that massive quantities of scientific data — known in the information technology (IT) world as “big data” — is essential to accurately measure, monitor and quantify our natural systems.

Governments are increasingly recognizing the need to integrate data collected from many agencies, and private companies like Esri are recognizing the value of big data to help make informed decisions. Global monitoring networks like TEAM, the soon to be implemented Vital Signs monitoring system and a host of other multidisciplinary monitoring and observing networks are now busy collecting this data.

These monitoring networks and systems will serve as data providers and synthetic engines to process this ever-expanding volume of environmental and socioeconomic data and share it with the right people. They are also demonstrating that we can use these standardized datasets to address some of the most pressing environmental challenges, such as how our natural resources will respond to climate change or the impact local land-use decisions will have on ecosystem services.

Finally, a critical component for any network with large quantities of data is a cyberinfrastructure — or robust IT systems and tools —that can facilitate the operations of the network but also to manage, analyze and distribute the large quantities of data. (Click on the infographic below to learn more.)

How TEAM works

The recent BioScience paper highlights our experience building a cyberinfrastructure for TEAM, which was originally created by CI and is now a partnership among CI, the Missouri Botanical Garden, the Smithsonian Institution and the Wildlife Conservation Society.

TEAM’s global network of scientists is collecting and distributing near real-time data on trends in biodiversity, climate, land-cover change and ecosystem services. An important goal of the paper was to provide “lessons learned” for other major scientific monitoring and observing networks. In addition, these experiences are very relevant to REDD+ projects and to national governments, as they implement new national resource monitoring regimes and improve upon existing ones.

My experience working with TEAM has been incredibly exciting so far. Technology changes so fast, and the future will yield even more data as we take advantage of the prevalence of mobile devices. Social networks and citizen science efforts will continue the data deluge and — importantly — encourage an open data approach amongst a new generation of young scientists to share data freely.

Eric Fegraus

Eric Fegraus

Ultimately we look forward to the time when no matter the size of the project, from small projects run by students and faculty to global monitoring networks, the data they collect will become part of a public resource, accessible and helpful to all.

Eric Fegraus is the director of information systems for the TEAM Network.

NEXT

Page statistics
2780 view(s) and 19 edit(s)
Social share
Share this page?

Tags

This page has no custom tags.
This page has no classifications.

Comments

You must to post a comment.

Attachments