Table of contents
  1. Story
    1. Introduction
    2. Some Background
      1. Explanation of Data Science Publication
      2. Explanation of Spotfire Visualizations
    3. Question
    4. Workflow
    5. ​Data Sets
      1. Polar Data Catalogue
        1. Polar Data Catalogue Help Manual
          1. Part I: Submit Data and Metadata
          2. Part II: Search for Data
          3. Part III: Approvers Guide
          4. Part IV: Frequently Asked Questions
      2. Introduction to BCO-DMO
        1. Data Access Tutorial 2014
          1. How to Submit Data
          2. Step 1: Complete Metadata Forms
          3. Step 2: Prepare Data Files
          4. Step 3: Email BCO-DMO
        2. Data access: TEXT-BASED SEARCH scenario 1: You have a general idea of what you are looking for
        3. Data access: MAP BROWSE scenario 2: You are interested in data from a particular geographic region
        4. Data access: MAP KEYWORD SEARCH scenario 3: You are interested in data of a particular type from a particular geographic area
        5. Data access: MAP SEMANTIC SEARCH scenario 4: You have an idea what you are looking for, but you do not know the Program, Project, or Deployment name
        6. Glossary of Terms
        7. Acknowledgments
        8. Follow BCO-DMO
      3. Polar Hub: A Global Hub for Polar Data Discovery
        1. Step 1. Create
        2. Step 2. Mine
        3. Step 3. Discovery
      4. Index of /pub/requests/DVPC/
        1. readme.txt
    6. Some Results and Conclusions
      1. BCO Program
      2. Beaufort Sea Ice Thickness 2012
      3. Beaufort Sea Ice Thickness 2013
      4. Canadian Index of Ice and Snow
    7. Feedback
  2. Slides
    1. Slide 1 Data Science Publication for NSF Polar Cyberinfrastructure
    2. Slide 2 Preface
    3. Slide 3 Overview
    4. Slide 4 Data Science for Business: Data Mining Process
    5. Slide 5 Data Science for NSF Polar Cyberinfrastructure: Knowledge Base
    6. Slide 6 Possible Data Sets
    7. Slide 7 Open Science Codefest: NASA/NSF/NSIDC Data Sets
    8. Slide 8 Polar Data Catalogue: Home Page
    9. Slide 9 Polar Data Catalogue: Collections
    10. Slide 10 Polar Data Catalogue: Search
    11. Slide 11 Polar Data Catalogue: Canadian Lake Ice Database
    12. Slide 12 Polar Data Catalogue: Sea Ice Thickness in Southern Beaufort Sea
    13. Slide 13 Polar Data Catalogue: Spreadsheet
    14. Slide 14 BCO-DMO
    15. Slide 15 Data Access Tutorial 2014 OCB PI Summer Workshop
    16. Slide 16 BCO-DMO Datasets
    17. Slide 17 BCO-DMO MapServer Geospatial Interface
    18. Slide 18 Polar Hub: A Global Hub for Polar Data Discovery
    19. Slide 19 The AMRC at University of Wisconsin-Madison
    20. Slide 20 Data Science for NSF Polar Cyberinfrastructure: Spreadsheet Knowledge Base
    21. Slide 21 Data Science for NSF Polar Cyberinfrastructure: Spotfire Cover Page
    22. Slide 22: Data Science for NSF Polar Cyberinfrastructure: Spotfire Visualizations
  3. Spotfire Dashboard
  4. Workshop Information
    1. Call for Remote Participation
    2. Federal Big Data Working Group Meetup
    3. ​Create a Session
    4. Details of the Workshop
      1. Invite Email to Participants
      2. Logistics Email
    5. Call for Participation: NSF DataViz Hackathon for Polar CyberInfrastructure: NYC, November 3-4, 2014
  5. Research Notes
    1. Qlik Hackathon
      1. FAQs
      2. Qlik Branch
    2. ESIP Preservation and Stewardship Committee Telecon 2014-10-17:
  6. NSF Polar Cyberinfrastructure
    1. Abstract
    2. Location
    3. Sessions
      1. Creation of session dependent repos
      2. Importance of seeing Apache way in a Hackathon with DataVis
      3. Build a Visualization, Tell a Story
      4. Ice Core (and Ice Core Archive) Visualization 
      5. Exploring Data with Time as a 4th Dimension
      6. Change over time in 2D
      7. ​Data Science Publication for NSF Polar Cyberinfrastructure
      8. Integrating data discovery, analysis and visualization into polar cyberinfrastructure -- the PolarHub solution
      9. "Higher Dimensional" Polar Data Visualization
      10. Design an outline for adding analysis and vis algorithms from the Polar domain to Apache Open Climate Workbench data
      11. NASA Near Real-time Polar Imagery Services
      12. Open-source Polar Data workflows with Tangelo-Hub
      13. Polar Data Analytics as a Service (PDAaaS)
      14. ETL 101 - Bringing Polar data analytics one step closer to the Polar Scientist
      15. GISCube, Open Source Web-based geoprocessing and visualization application
      16. Run Distributed Release Audit Tool (DRAT) on all codefest generated code and report out on license statistics
      17. Antarctic Meteorology Research Center (AMRC) datasets
      18. Crawl and prepare NSF ACADIS, NASA AMD and NSIDC Arctic Data Explorer datasets Part 2
    4. Agenda
      1. Day 1
      2. Day 2
    5. Tweets
    6. IRC
    7. Hackathon Participants
    8. Organizing Committee
  7. NEXT

Data Science for NSF Polar Cyberinfrastructure

Last modified
Table of contents
  1. Story
    1. Introduction
    2. Some Background
      1. Explanation of Data Science Publication
      2. Explanation of Spotfire Visualizations
    3. Question
    4. Workflow
    5. ​Data Sets
      1. Polar Data Catalogue
        1. Polar Data Catalogue Help Manual
          1. Part I: Submit Data and Metadata
          2. Part II: Search for Data
          3. Part III: Approvers Guide
          4. Part IV: Frequently Asked Questions
      2. Introduction to BCO-DMO
        1. Data Access Tutorial 2014
          1. How to Submit Data
          2. Step 1: Complete Metadata Forms
          3. Step 2: Prepare Data Files
          4. Step 3: Email BCO-DMO
        2. Data access: TEXT-BASED SEARCH scenario 1: You have a general idea of what you are looking for
        3. Data access: MAP BROWSE scenario 2: You are interested in data from a particular geographic region
        4. Data access: MAP KEYWORD SEARCH scenario 3: You are interested in data of a particular type from a particular geographic area
        5. Data access: MAP SEMANTIC SEARCH scenario 4: You have an idea what you are looking for, but you do not know the Program, Project, or Deployment name
        6. Glossary of Terms
        7. Acknowledgments
        8. Follow BCO-DMO
      3. Polar Hub: A Global Hub for Polar Data Discovery
        1. Step 1. Create
        2. Step 2. Mine
        3. Step 3. Discovery
      4. Index of /pub/requests/DVPC/
        1. readme.txt
    6. Some Results and Conclusions
      1. BCO Program
      2. Beaufort Sea Ice Thickness 2012
      3. Beaufort Sea Ice Thickness 2013
      4. Canadian Index of Ice and Snow
    7. Feedback
  2. Slides
    1. Slide 1 Data Science Publication for NSF Polar Cyberinfrastructure
    2. Slide 2 Preface
    3. Slide 3 Overview
    4. Slide 4 Data Science for Business: Data Mining Process
    5. Slide 5 Data Science for NSF Polar Cyberinfrastructure: Knowledge Base
    6. Slide 6 Possible Data Sets
    7. Slide 7 Open Science Codefest: NASA/NSF/NSIDC Data Sets
    8. Slide 8 Polar Data Catalogue: Home Page
    9. Slide 9 Polar Data Catalogue: Collections
    10. Slide 10 Polar Data Catalogue: Search
    11. Slide 11 Polar Data Catalogue: Canadian Lake Ice Database
    12. Slide 12 Polar Data Catalogue: Sea Ice Thickness in Southern Beaufort Sea
    13. Slide 13 Polar Data Catalogue: Spreadsheet
    14. Slide 14 BCO-DMO
    15. Slide 15 Data Access Tutorial 2014 OCB PI Summer Workshop
    16. Slide 16 BCO-DMO Datasets
    17. Slide 17 BCO-DMO MapServer Geospatial Interface
    18. Slide 18 Polar Hub: A Global Hub for Polar Data Discovery
    19. Slide 19 The AMRC at University of Wisconsin-Madison
    20. Slide 20 Data Science for NSF Polar Cyberinfrastructure: Spreadsheet Knowledge Base
    21. Slide 21 Data Science for NSF Polar Cyberinfrastructure: Spotfire Cover Page
    22. Slide 22: Data Science for NSF Polar Cyberinfrastructure: Spotfire Visualizations
  3. Spotfire Dashboard
  4. Workshop Information
    1. Call for Remote Participation
    2. Federal Big Data Working Group Meetup
    3. ​Create a Session
    4. Details of the Workshop
      1. Invite Email to Participants
      2. Logistics Email
    5. Call for Participation: NSF DataViz Hackathon for Polar CyberInfrastructure: NYC, November 3-4, 2014
  5. Research Notes
    1. Qlik Hackathon
      1. FAQs
      2. Qlik Branch
    2. ESIP Preservation and Stewardship Committee Telecon 2014-10-17:
  6. NSF Polar Cyberinfrastructure
    1. Abstract
    2. Location
    3. Sessions
      1. Creation of session dependent repos
      2. Importance of seeing Apache way in a Hackathon with DataVis
      3. Build a Visualization, Tell a Story
      4. Ice Core (and Ice Core Archive) Visualization 
      5. Exploring Data with Time as a 4th Dimension
      6. Change over time in 2D
      7. ​Data Science Publication for NSF Polar Cyberinfrastructure
      8. Integrating data discovery, analysis and visualization into polar cyberinfrastructure -- the PolarHub solution
      9. "Higher Dimensional" Polar Data Visualization
      10. Design an outline for adding analysis and vis algorithms from the Polar domain to Apache Open Climate Workbench data
      11. NASA Near Real-time Polar Imagery Services
      12. Open-source Polar Data workflows with Tangelo-Hub
      13. Polar Data Analytics as a Service (PDAaaS)
      14. ETL 101 - Bringing Polar data analytics one step closer to the Polar Scientist
      15. GISCube, Open Source Web-based geoprocessing and visualization application
      16. Run Distributed Release Audit Tool (DRAT) on all codefest generated code and report out on license statistics
      17. Antarctic Meteorology Research Center (AMRC) datasets
      18. Crawl and prepare NSF ACADIS, NASA AMD and NSIDC Arctic Data Explorer datasets Part 2
    4. Agenda
      1. Day 1
      2. Day 2
    5. Tweets
    6. IRC
    7. Hackathon Participants
    8. Organizing Committee
  7. NEXT

  1. Story
    1. Introduction
    2. Some Background
      1. Explanation of Data Science Publication
      2. Explanation of Spotfire Visualizations
    3. Question
    4. Workflow
    5. ​Data Sets
      1. Polar Data Catalogue
        1. Polar Data Catalogue Help Manual
          1. Part I: Submit Data and Metadata
          2. Part II: Search for Data
          3. Part III: Approvers Guide
          4. Part IV: Frequently Asked Questions
      2. Introduction to BCO-DMO
        1. Data Access Tutorial 2014
          1. How to Submit Data
          2. Step 1: Complete Metadata Forms
          3. Step 2: Prepare Data Files
          4. Step 3: Email BCO-DMO
        2. Data access: TEXT-BASED SEARCH scenario 1: You have a general idea of what you are looking for
        3. Data access: MAP BROWSE scenario 2: You are interested in data from a particular geographic region
        4. Data access: MAP KEYWORD SEARCH scenario 3: You are interested in data of a particular type from a particular geographic area
        5. Data access: MAP SEMANTIC SEARCH scenario 4: You have an idea what you are looking for, but you do not know the Program, Project, or Deployment name
        6. Glossary of Terms
        7. Acknowledgments
        8. Follow BCO-DMO
      3. Polar Hub: A Global Hub for Polar Data Discovery
        1. Step 1. Create
        2. Step 2. Mine
        3. Step 3. Discovery
      4. Index of /pub/requests/DVPC/
        1. readme.txt
    6. Some Results and Conclusions
      1. BCO Program
      2. Beaufort Sea Ice Thickness 2012
      3. Beaufort Sea Ice Thickness 2013
      4. Canadian Index of Ice and Snow
    7. Feedback
  2. Slides
    1. Slide 1 Data Science Publication for NSF Polar Cyberinfrastructure
    2. Slide 2 Preface
    3. Slide 3 Overview
    4. Slide 4 Data Science for Business: Data Mining Process
    5. Slide 5 Data Science for NSF Polar Cyberinfrastructure: Knowledge Base
    6. Slide 6 Possible Data Sets
    7. Slide 7 Open Science Codefest: NASA/NSF/NSIDC Data Sets
    8. Slide 8 Polar Data Catalogue: Home Page
    9. Slide 9 Polar Data Catalogue: Collections
    10. Slide 10 Polar Data Catalogue: Search
    11. Slide 11 Polar Data Catalogue: Canadian Lake Ice Database
    12. Slide 12 Polar Data Catalogue: Sea Ice Thickness in Southern Beaufort Sea
    13. Slide 13 Polar Data Catalogue: Spreadsheet
    14. Slide 14 BCO-DMO
    15. Slide 15 Data Access Tutorial 2014 OCB PI Summer Workshop
    16. Slide 16 BCO-DMO Datasets
    17. Slide 17 BCO-DMO MapServer Geospatial Interface
    18. Slide 18 Polar Hub: A Global Hub for Polar Data Discovery
    19. Slide 19 The AMRC at University of Wisconsin-Madison
    20. Slide 20 Data Science for NSF Polar Cyberinfrastructure: Spreadsheet Knowledge Base
    21. Slide 21 Data Science for NSF Polar Cyberinfrastructure: Spotfire Cover Page
    22. Slide 22: Data Science for NSF Polar Cyberinfrastructure: Spotfire Visualizations
  3. Spotfire Dashboard
  4. Workshop Information
    1. Call for Remote Participation
    2. Federal Big Data Working Group Meetup
    3. ​Create a Session
    4. Details of the Workshop
      1. Invite Email to Participants
      2. Logistics Email
    5. Call for Participation: NSF DataViz Hackathon for Polar CyberInfrastructure: NYC, November 3-4, 2014
  5. Research Notes
    1. Qlik Hackathon
      1. FAQs
      2. Qlik Branch
    2. ESIP Preservation and Stewardship Committee Telecon 2014-10-17:
  6. NSF Polar Cyberinfrastructure
    1. Abstract
    2. Location
    3. Sessions
      1. Creation of session dependent repos
      2. Importance of seeing Apache way in a Hackathon with DataVis
      3. Build a Visualization, Tell a Story
      4. Ice Core (and Ice Core Archive) Visualization 
      5. Exploring Data with Time as a 4th Dimension
      6. Change over time in 2D
      7. ​Data Science Publication for NSF Polar Cyberinfrastructure
      8. Integrating data discovery, analysis and visualization into polar cyberinfrastructure -- the PolarHub solution
      9. "Higher Dimensional" Polar Data Visualization
      10. Design an outline for adding analysis and vis algorithms from the Polar domain to Apache Open Climate Workbench data
      11. NASA Near Real-time Polar Imagery Services
      12. Open-source Polar Data workflows with Tangelo-Hub
      13. Polar Data Analytics as a Service (PDAaaS)
      14. ETL 101 - Bringing Polar data analytics one step closer to the Polar Scientist
      15. GISCube, Open Source Web-based geoprocessing and visualization application
      16. Run Distributed Release Audit Tool (DRAT) on all codefest generated code and report out on license statistics
      17. Antarctic Meteorology Research Center (AMRC) datasets
      18. Crawl and prepare NSF ACADIS, NASA AMD and NSIDC Arctic Data Explorer datasets Part 2
    4. Agenda
      1. Day 1
      2. Day 2
    5. Tweets
    6. IRC
    7. Hackathon Participants
    8. Organizing Committee
  7. NEXT

Story

Data Science Publication for NSF Polar Cyberinfrastructure

Introduction

Please see Workshop Information and NSF Polar Cyberinfrastructure Web Site below.

The results and demo will be the Slides and Spotfire Dashboard below.

My goal is to see if I can integrate and federate these multiple data sources and produce linked small multiples and semantically-enabled, faceted search by the use of the search categories that I and others have constructed iin the datasets.

It should also be pointed out that I am using MindTouch and Spotfire in place of GitHub, but contributing the results to GitHub.

Some Background

Explanation of Data Science Publication

I think that NIH’s Associate Director for Data Science, Dr. Phil Bourne, really invented the data science publication with his journal (PLOS Computational Biology) and was hired to change the data culture to that at NIH: http://semanticommunity.info/Data_Science/Data_Culture_at_the_NIH#Story

He views data science publications as the building blocks for a “data commons” which is like a sandbox where researchers can come and play with others scientific data with their own tools or tools that are part of the “data commons”

His best explanation of a data science publication I have found is in his slides at:http://semanticommunity.info/Data_Science/Data_Science_for_RDA#Best_Practices_for_Data:_A_Biologists_View

Especially the one called: The Knowledge and Data Cycle

I am doing essentially the same thing by putting the content in MindTouch (a state-of-the-art Wiki) with structure and embedding the Spotfire file in MindTouch with a link to the Web Player version

For example, the NSF/NSB Indicators Digest 2014 text and graphics are in MindTouch with well-defined URLs in a searchable index which is integrated with the data tables for the graphics which are integrated by topic in multiple adjacent visualizations as Tufte suggests so the user can more easily compare trends, etc.

See:http://semanticommunity.info/Data_Science/Data_Science_for_Big_Data_Analytics#Data_Science_Data_Publication_for_National_Science_Board

More importantly, the Spotfire visualizations are more than static graphs, they are dynamically linked to one another and the underlying data, and also to the metadata and story in MindTouch.

We have done Data Science Publications for many senior government leaders:
http://semanticommunity.info/Data_Science/NSF_Funding_for_BIG_DATA_and_Data_Science/NSF_Grant_Proposal_Guide#Conclusion

Another example: Data Science (publication) for the NOAA Chief Data Officer

http://semanticommunity.info/Data_Science/Data_Science_for_the_NOAA_Chief_Data_Officer

for our November 3rd Meetup: http://www.meetup.com/Federal-Big-Data-Working-Group/events/213175262/

Explanation of Spotfire Visualizations

Spotfire
TIBCO Spotfire designs, develops and distributes in-memory analytics software for next generation business intelligence.

TIBCO Spotfire® Ranked Highest “Current Offering” in Forrester Wave for Agile BI 2014
Source: http://spotfire.tibco.com/

Complimentary Subscription: http://spotfire.tibco.com/tsc/donate
What I have as a Journalist, Professor, and Non-Profit Organization

Free One Month Trial: https://spotfire.cloud.tibco.com/tsc/#!/tryspotfire 
Cloud Personal, Cloud Work Group, and Cloud Enterprise

For the NSF/NSB Indicators, there a live link for the Spotfire analysis:
https://spotfire.cloud.tibco.com/public/ViewAnalysis.aspx?file=/users/bniemann/Public/NSBIndicators2014-Spotfire&waid=eaccd3aaab73f89cda578-26211723b2ccba

You can download my Spotfire File (and/or my Excel Spreadsheets) and use it in your own Spotfire Client or another tool of your choosing so it is open in that sense.

Question

Thanks, I checked out the Spotfire description for the NOAA meetup. Looks really neat! Did it build the wiki/website for that automatically?

No, there is considerable science and art involved in building the knowledge base (in MindTouch) and spreadsheet (in Excel) first, which then makes the Spotfire (data browser) application easier to “storify” the results.

These tools make it easier, but still require data science, statistics, visualization, data journalism talents and experience.

This is what I did in two phases for the NOAA work over a period of about a month and will do for the Polar Data in the next week for a demo and more after that to prepare for another EarthCube Meetup we have been planning for some time now:

http://semanticommunity.info/Data_Science/EarthCube_Data_Science_Publications

Workflow

Some prep work is already underway (if you scour the Open Science Codefest site you will find some) to prepare some datasets of relevance to the Polar community. We will provide some of this prepared data to interested parties ahead of the workshop in the next few weeks in case folks want to start hacking early. We will tweet under the hash tag: #nsfpolardatavis

I followed the Cross-Industry Data Mining Standard by first 1 Business Understanding (of the Hackathon), 2 Data Understanding (by mining the Sessions), 3 Data Preparation, 4 Modeling, 5 Evaluation, and 6 Deployment (Demo). The documentation will be in the form of the Data Science Publication for NSF Polar Cyberinfrastructure.

​Data Sets

The Polar Data Catalogue has lots of CTD data (at least 51 cruises) that can be downloaded for free (also other data available, too), if it would fit your needs.  Just go to https://polardata.ca, click on the map (for the PDC Geospatial Search), then select the Polar Data Catalogue collection.  You can do a keyword search for CTD or do a map search in a particular location.  May want to click the “[ ] View downloadable datasets only” check box, so that you only see results with data available for download.

I'd like to add that BCO-DMO has data that may be of potential use for the workshop. Mark suggested "CTD casts from multiple ocean cruises", which BCO-DMO can provide along with any associated biological and chemical data from complete and ongoing projects. My Note: Excellent Tutorial (PDF) should be in-line

This session will introduce the research progress of ASU researchers on a service-oriented Polar Cyberinfrastructure that integrates data search, intelligent online analysis and visualization to support polar sciences. The core component of this cyberinfrastructure portal is PolarHub, which has the ability to conduct large-scale web crawling to discover distributed geospatial data in the format of OGC web services. We are also working to integrate multi-dimensional visualization techniques to support more intuitive data presentation and analysis. The link to the work is here: http://polar.geodacenter.org/polarhub/

Building off of NCEAS/open-science-codefest#26, continue data prep and crawl of AMD, ACADIS and ADE with goal of preparing some of the data for (GeoViz; science focused viz, etc.) My Note: See Presentation Slides

Open Science Codefest: NASA/NSF/NSIDC Data Sets

Participants would use real world data science tools like Tika (http://tika.apache.org/), Nutch (http://nutch.apache.org/), Solr (http://lucene.apache.org/solr/) and OODT (http://oodt.apache.org/) to crawl and prepare the datasets of interesting Polar parameters for Visualization experts to then hack on during a 2 day NSF visualization hackathon in NYC in November. Be part of doing something real, contributing to Apache projects (and getting the merit and potentially becoming a committer and PMC member yourself) and also contributing to NSF and NASA goals!

The AMRC at University of Wisconsin-Madison studies the weather in Antarctic in two ways; Automatic Weather Station (AWS) data and satellite composite imagery. For this workshop, a dataset has been prepared that contains five formats of AWS data and two formats of infrared satellite composites for one month May, 2014. This dataset can be found via our ftp site (ftp://amrc.ssec.wisc.edu/pub/requests/DVPC/ My Note: See if can read Text files in Spotfire.

Polar Data Catalogue

​Source: https://polardata.ca/

The Polar Data Catalogue is a database of metadata and data that describes, indexes, and provides access to diverse data sets generated by Arctic and Antarctic researchers. The metadata records follow ISO 19115 and Federal Geographic Data Committee (FGDC) standard formats to provide exchange with other data centres. The records cover a wide range of disciplines from natural sciences and policy, to health and social sciences. The PDC Geospatial Search tool is available to the public and researchers alike and allows searching data using a mapping interface and other parameters. 

Please click on the PDC Search map below to start searching for datasets or sign in to the PDC Input application to contribute metadata or data to the Polar Data Catalogue. 

The PDC Lite Search is also available for users with limited Internet speed.

Polar Data Catalogue Help Manual

Source: https://polardata.ca/pdcinput/public...elpManual.ccin

The Polar Data Catalogue (PDC) is the metadata repository for the ArcticNet Network of Centres of Excellence, the Government of Canada Program for the International Polar Year (IPY), the Northern Contaminants Program (NCP), and other related Canadian and international research programs. Registered users enter metadata and data which are searchable by the public and other researchers.
The PDC Geospatial Search is available to researchers looking for polar research projects and data in their study area and to people who are simply curious about Arctic and Antarctic data. Through an easy-to use mapping interface, the PDC Geospatial Search allows easy and quick retrieval of spatial data in the Arctic and Antarctic regions.
The PDC provides an interface not only to search for data but also to upload and share data with the general public. To upload data, contributors should be researchers with our partner organizations and registered users of our website. The uploaded metadata and data will not be available for public search until our metadata and data Approvers inspect your files and approve them.
If you have questions or concerns, please contact:
Julie Friddell at PDC
519-888-4567 x32689
pdc@uwaterloo.ca
Part I: Submit Data and Metadata
This section guides you through the process of adding new metadata and data records into the PDC using the online metadata/data entry system. The PDC also provides the interface to manage all of your metadata and data records. The three functional components are "Submit Metadata," "Submit Data," and "My Metadata." In the following section, each part is described in detail.
Metadata-Entry-System.png
To add or edit data and/or metadata, "data owners" must be registered. First-time users should go to the Polar Data Catalogue home page and click the "Register for PDC link" in the left navigation panel.
 
Metadata_Register.png
 
 Submit Metadata

To produce a good metadata record, always try to gather as many details as possible about the resource that you want to describe. The most important fields that may not be waived while compiling a standard metadata record are the following: Title, Date of Creation and Publication, Abstract, Language used for documenting data and metadata, Topic Category, Scale, Maintenance and Update Frequency, and Metadata Author. After the metadata records are created, the data owners submit the metadata record, and a research group 'approver' will check the quality of the metadata. Once approved, the metadata record is available to be searched in the geospatial search tool. The process is shown in the flowchart above. 

For detailed instructions about metadata entry, please refer to the "PDC Best Practices Document - Complete Guide," found under the "Help" link on the PDC's homepage. 

Please refer to the "Descriptions of required information in Metadata Input Form" for the definition of each field.

Descriptions of required information in Metadata Input Form 

The Metadata Input Form Template (available in several formats under "Help" in the left navigation menu) allows you to completely fill in the form offline and then to copy into the online form. All fields of the online form need to be completed for your record to be saved. 
  1. Metadata Input Form - Fillable PDF file
  2. Metadata Input Form - Fillable MS Word Document (will download when selected)
  3. Metadata Input Form - .txt format

Completed Metadata Input Form Example 

After you login, you will see several menus related to metadata management on the left panel.

1. Log in (or register for first time users) to PDC using your registered email address.
  • [Note for students: Log in (or register for first time users) using your supervisor's email address. The email address is the main link to the laboratory's records and thus remains the most reliable and constant login name for years to come.]
 
Metadata_signin.png

 

2. Agree to the terms of the PDC
 

Metadata-Agree.png


3. Creating a New Metadata Record using the web based Metadata Editor.
  • The "Submit Metadata" link in the left navigation menu on the "My Metadata" page can be used to create new metadata.
 
submit_metadata.png


4. Complete the Metadata Input Form. The fields marked with a red asterisk are mandatory. 
 
Metadata_entry_form.png


5. Validate and save metadata record: 

To save the metadata, click on the "Validate and Save" button at the bottom of the page. A screen indicating "Record created successfully" will appear if no errors were present in the metadata. Once validated, the record status will be marked as "Saved" in the "My Metadata" page, but the record will not be available online until approval (see below). By clicking "Update," you can make changes to a record.
 
Metadata-Validate.png


If there are errors, you will be returned to the form. Errors will be shown in red to indicate the field(s) that must be modified.
 
Metadata-Validate2.png


You will be taken to the "Record Created Successfully" page once the metadata is validated.


6. Submit your metadata to the project Approver: 

Once you are satisfied with the metadata, click "Submit" at the bottom of the metadata record page or choose "Submit Metadata" on the "Record Created Successfully" page. The "My Metadata" page will be shown. 

An automatic email will be sent to your e-mail (or the email used to log in) and your research program's Approver (i.e., Scott Tomlinson for Northern Contaminants Program). This approval process is to further ensure good quality, standardization, and accuracy of metadata found in the PDC. 

The Approver can approve, edit (for minor errors), or send back (for major changes) the record. An automatic notification email will be sent to the submitter once the record is "Sent Back" or "Approved." Once approved, the record will be available to the public, through the PDC's Geospatial Search application. If the metadata is sent back by the approver, you will receive an email explaining why. The metadata can be edited on the "My Metadata" page and submitted once again for approval.
 
 Update Metadata and Enter Similar Metadata Entry

Once submitted, metadata cannot be updated by the submitter; the Approver alone can do this. Once the metadata record is approved, however, you have the ability to update the metadata. 

The "Update" button at the bottom of the desired record can be selected to modify an approved record. For instance, additional years and geographic coordinates can be added to update the temporal and spatial coverage of a study. While the record is waiting for re-approval, only the old record will be available online. 

To add similar metadata to an existing entry, go to "My Metadata" in the left navigation menu, and click the CCIN Reference Number of the metadata entry that your new entry will be based on. A summary of the selected data will appear. Click, "Similar Entry" at the end of the page. You can then add a similar metadata entry. 
 
Metadata-Update.png
 
 My Metadata

"My Metadata" on the left navigation panel allows for metadata management. Once clicked, a table containing a list of your metadata will be shown on the right side of the page. The table has six columns. 

The CCIN reference number is assigned automatically to the metadata record. It is a unique identification number for each individual record. 

Metadata can have different statuses, which include SAVED (metadata has been validated and can be edited prior to submission), SUBMITTED (metadata has been submitted for review and approval), APPROVED (metadata has been approved and is now publicly visible in the PDC Search website), and SENT BACK (metadata has been returned to the creator for editing). RESAVED, RESUBMITTED, and REAPPROVED status are present if the metadata has been previously approved and updated. 
 
Metadata.png
 
 Submit Data

1. Select "Submit Data" from the left menu. Click on the metadata file for which you will upload corresponding data, and the following screen will appear. Upload the appropriate data files, and choose "Submit File" to continue. 
 
Metadata-Submit.png

2. You will be notified when data is submitted successfully, and you will have the option to upload additional data files.
 
Metadata-Submit2.png


3. You can then view all of your uploaded data to date by choosing the "My Data" link from the left menu.

 
Part II: Search for Data
This section guides you through the process of searching for data using user-defined criteria. Anyone can search for data using our PDC interface.
1. Go to http://www.polardata.ca, and click on the map to start your search.
welcome_to_pdc.png
2. The search user interface is divided into two panels.
  • The left panel shows the map of the poles in polar stereographic projection. You can use the tab on the upper left corner of the left panel to switch between the Arctic and Antarctic. Several tools are provided to explore the map, including zoom in, zoom out, pan and full extent. The longitude and latitude of the mouse location is shown in the lower right corner of the map.
  • The right panel provides the interface to do the search and show the search results.
 
PDC-Search
 
3. Choose one of the two data collections. "Polar Data Catalogue" includes various research datasets, while the RADARSAT Polar Science Dataset includes RADARSAT-1 imagery. You do not need to register to download most of the data in our system or to search for RADARSAT imagery. However, registration is required to download RADARSAT-1 imagery. Agree to the Terms of Use to continue.
 Polar Data Catalogue Search
 GIS Viewer
 RADARSAT Polar Science Dataset Search
 PDCLite Low-Bandwidth Search
Part III: Approvers Guide
The Approvers check the quality of the metadata before it is made available to the public. The Approver has a "My Metadata Approvals" page that shows all the records that were submitted to him/her as well as the records he/she previously approved.
1.To approve a submitted record, the Approver needs to click on the desired record title in the "My Metadata Approvals" page and check the record for errors (see attached Checklist of main items to review).
Approvers_Guide1.png
 
2. Once reviewed, the choices at the bottom of the page are:
  • Approve: Metadata record is approved and will be searchable online shortly.
  • Update: Approver may make small changes and save record again.
  • Send Back: Send the record back to submitter if significant changes are required.
Approvers_Guide2.png
 
3. Once approved, the record will be available online.
4. The Approver can later edit the records he/she previously approved by clicking the "Update" button at the bottom of the page. The record will no longer be available to the public and will not be so until it is approved again. The Approver can also send back a previously approved record to its creator if important changes are needed (by clicking "Send Back" and providing comments or questions for the submitter).
 Checklist for Metadata Approvers of the Polar Data Catalogue

The following points summarize the main items that should be verified before approving a metadata record:

1. Make sure there is a contact email in the record. If there is no available link to data, this email address should also be in the field "Link to data" and should be the email of the data producer. 

2. Check content for description of variables sampled. The important variables should be selected as keywords (even if they are present in other fields such as Abstract, Purpose, Study Site, etc.). 

3. Add keywords as needed such as keywords that are overarching, especially when only sector-specific keywords have been used. For example, studies examining certain bird species may have the species' names as keywords but should also include the more general term "bird" as a keyword. 

4. Make sure that files containing potentially sensitive information are highlighted. Scan these records for: 
  • a) the release of confidential information such as names of individuals, and
  • b) possible compromises to individuals and communities.

It may be suggested that large geographic regions be identified rather than specific communities, to avoid divulging potentially sensitive information.

5. Check the format for latitude and longitude (all Canadian longitudes should be negative and latitudes in the Northern Hemisphere should be positive). For study sites that are stations and not bounding areas, check that the user has simply repeated the same latitude and longitude, and not entered zero.

6. Ensure a consistent format for the field "How Data Should be Cited." For example, use the publication citation, or if no publication is yet available, the authors' names followed by "unpublished data" should be used.

7. If you judge that significant changes are required to a record, you can send back the record to the creator and request these changes.

8. Once the record is approved, please then verify in the Polar Data Catalogue: (a) that the title appears in a search (using for example a Keyword and/or a polygon on the map), (b) that it has the correct position on the map (using the "Show" button), and (c) that the metadata appear when clicking on the title. 


CCIN will conduct periodic quality checking of records, and you may receive a list of previously approved records that require your attention. You can either make the noted corrections (i.e. for simple changes, such as an incorrect sign on the coordinates) or send it back to the creator for larger changes, if deemed necessary.
Part IV: Frequently Asked Questions

A:User Interface

 1. Where can I change my login information?
Once you are logged into the PDC, click "Edit Profile" in the menu on the left side.
 
 2. What if I forget my password?
Click the "Forgot Password" link below the "Sign In" button.
 
 3. How can I report an error I experienced in PDC?
To help us resolve the issue, send us an email using the "Report Error" link in the menu. Include your email address so we know whom to contact when the problem is resolved. Please provide a concise description of the error that occurred, including: 
1. Where in the website did this problem occur? Give the URL or the internet address of the page you were on. 
2. What you were doing? Give the steps preceding the error, including the last steps that produced the error. 
3. What you expected to happen. 
4. What actually happened. Include the text of the error. 
5. Provide the date and approximate time of the error.
 

B. Metadata and Data Entry

 1. What is Metadata? Why metadata is important?
Metadata is the description of a dataset, providing information about the data. It is important for data searching and data sharing.
 
 2. What information is contained in the metadata form?
This is described in the "Descriptions of required information in Metadata Input Form" document on the PDC Help page.
 
 3. What processes should I follow to submit metadata?
1. Log in and click "Submit Metadata" in the left navigation panel 
2. Enter metadata 
3. Submit metadata
4. Submit data and wait for approval
 
 4. Where do I go to view my metadata records?
Anytime you would like to see your list of records, log in to your account in the PDC and click "My Metadata" in the menu.
 
 5. How can I update my metadata?
Go to "My Metadata" in the default page, and select your metadata entry. You can update it if the status for that metadata record is "saved" or "approved". If the status is "submitted", you must wait until the record is approved or sent back by the Approver.
 
 6. How can I make a copy of one of my existing records so that I don't have to type some of the same information all over again?
Choose the record you wish to copy from your list of metadata, under "My Metadata." Scroll to the bottom of the record and click on the "Similar Entry" button.
 
 7. Why do I receive emails from the PDC?
Users who create metadata are considered data owners. Data owners receive an email when their metadata records are approved or rejected. Approvers receive emails when metadata is submitted for approval.
 
 8. What information will the Approver verify?
Please go to "Part III: Approvers Guide" of this Help Manual and look for "Checklist for Metadata Approvers of the Polar Data Catalogue."
 
 9. What happens after my metadata is reviewed by the approver?
Once reviewed, each metadata record will be given a status of approved, update, or sent back.
  • a) Approved: Metadata record is approved and will be searchable online shortly.
  • b) Update: Approver may make small changes and save the record again.
  • c) Sent back: If significant changes are required, the record will be sent back to the submitter.
 
 10. What happens if the approver does not approve my metadata?
An email will be sent to you, including details for improving your metadata.
 
 11. How soon can I find my approved metadata in the search tool?
If an approved record is updated and re-approved, the search tool updates and provides the new metadata online immediately.

C. Quality Assurance

 1. What is the point of having metadata approved?
In order for metadata to be found using the geospatial search tool, a quality assurance step has been introduced to ensure metadata is documented without error. Users create metadata and submit it to research group Approvers. Approvers may approve, update or send the record back to the User for major revisions.
 
 2. Do I have research group Approver access?
If you see the "My Approvals" choice in the menu, you have Approver access.
 
 3. What is an Approver? Who are the Approvers? What do Approvers see?
Approvers are appointed by the PDC to review and approve submitted metadata and data records. Approvers see a "My Metadata Approvals" and "My Data Approvals" tab, in addition to the options available to the average user. 
 
 4. What buttons do Approvers see?
The button descriptions are:
  • Approve: Metadata record is approved and will be searchable shortly.
  • Update: Approver may make small changes and save record again.
  • Send Back: The record will be sent back to the submitter, with suggestions for improvements.
  • Cancel: Returns to "My Approvals" page.

Introduction to BCO-DMO

Source: http://www.bco-dmo.org/

The Biological and Chemical Oceanography Data Management Office (BCO-DMO) staff members work with investigators to serve data online from research projects funded by the Biological and Chemical Oceanography Sections and the Division of Polar Programs Antarctic Organisms & Ecosystems Program at the U.S. National Science Foundation.

As of November 14, 2013 we have migrated our metadata database from ColdFusion to Drupal, a public domain content management system. Some of the pages will be slightly modified from what you are used to, but for the most part, pages should look familiar. If you have any trouble with the new website, or have any comments or suggestions, please just let us know.

See 10 Tables in Spreadsheet and OBC Tutorial (Should this 4.2 MB PDF be in-line? Yes! See below)

Data Access Tutorial 2014

Source: http://www.bco-dmo.org/files/bcodmo/OCB-Tutorial.pdf (4.2 MB PDF)

Excerpts

OCB PI Summer Workshop

How to Submit Data

BCO-DMO works with investigators to publish data from research projects funded by the NSF Geosciences Directorate (GEO) Division of Ocean Sciences (OCE) Biological and Chemical Oceanography Sections and the Division of Polar Programs (PLR) Antarctic Sciences (ANT) Organisms and Ecosystems Program. Once published, BCO-DMO also fulfills NSF requirements for long-term preservation by submitting data to the appropriate national data center for archive (e.g., NOAA’s National Oceanographic Data Center).

To start the process, please visit our “How to Get Started” page: http://www.bco-dmo.org/how-get-started

Here you will find the necessary forms to contribute metadata related to programs, projects, deployments, and datasets. These forms are available in Rich Text Format (.rtf), making them accessible using any word processing application:

Program: A large coordinated research effort, usually comprising multiple projects generating several datasets. (e.g., OCB, U.S. GEOTRACES, SEES-OA)

Project: A project usually encompasses one or more datasets, and may be part of a larger program (e.g. CoFeMUG is a project affiliated with U.S. GEOTRACES), or could be (and most are) unaffiliated with any program.

Deployment: A specific cruise, mooring, laboratory, or some other unique event of data collection and/or analysis.

Dataset: A collection of values representing scientific observations, measurements, or derivations.

Step 1: Complete Metadata Forms

Use the forms described above to submit new information to the Office.

Step 2: Prepare Data Files

We accept data in any format, however comma or tab-delimited (preferred), or Excel spreadsheets are most common. If the dataset is too large to contribute as an email attachment, contact our office via info@bco-dmo.org for instructions on the best way to contribute your data.

Step 3: Email BCO-DMO

Once completed, metadata forms and data files can be emailed as attachments to the office along with a brief communication explaining the nature of the submission, by addressing the email to: info@bco-dmo.org

You will receive a response from the Office confirming your submission and informing you of the staff assigned to your dataset. BCO-DMO staff members will work with you to publish your data and metadata.
Additional Information:

Also available on the “How To Get Started” page are useful links to additional information on the NSF OCE data policy, submitting data in spreadsheets, data archives, and more. Feel free to contact the Office with any questions or concerns you may have regarding your data submission.

Data access: TEXT-BASED SEARCH scenario 1: You have a general idea of what you are looking for

Go to: http://bco-dmo.org

The DATABASE links in the left navigation area provide an idea of how the data are organized. Data sets are grouped by Program (largest collection), Project (smaller in scope than a Program), Deployment (cruise, mooring, and many other examples), Dataset (a logical collection of data).

One could start with any of those navigation links and search to find the data of interest. If you know you are interested in data from an OCB project (for example, “CoFeMUG”) you may start at the top level “Programs”.

Select “Ocean Carbon and Biogeochemistry” from the Program list. URL: http://www.bco-dmo.org/program/2015

Scroll down and expand the Projects section below the “More Information” heading.

Select “Cobalt, Iron and Micro-organisms from the Upwelling zone to the Gyre” (CoFeMUG) from the list of project names.

The browser displays the project description page. URL: http://www.bco-dmo.org/project/2067

Scroll down that page and expand the Datasets section.

Select the data set of interest from the list. For this example, choose “nutrients and metals”. URL: http://www.bco-dmo.org/dataset/3233

From the dataset display page, expand the Deployments section to see the cruises during which the nutrients and metal data were collected.

These data were collected on one cruise, KN192-05. If data were collected on multiple deployments, all deployment names would be listed here (e.g. cruises, moorings, floats, gliders, etc.). To see more information about the cruise, including a description and other available data, one could click on the deployment name.

However, we want the data …

For this dataset there are two choices: and At this point, please click the button to access these data online.

Browser displays URL: http://data.bco-dmo.org/jg/serv/BCO/...s_metals.html0 in another window or browser tab depending on the browser configuration.

Click on the blue cruise ID at level 0 to expand the data display. Then, click on a station number to see the data for that station.

Explanation of buttons in the data system:

Directory displays (returns to) the Data Directory listing for this cruise
Documentation displays the supporting documentation for this dataset
Data Display returns to data display from documentation display
Download & Other Operations options for download, sub-setting and reformatting of data
Level 0 returns to level 0
Next Level expands the data to the next level of detail
Flat Listing displays one record per line of the current level of data


Now that we’ve seen the data, let’s look at it on a map. Return to the previous browser window (or tab) with the display of dataset metadata. URL: http://www.bco-dmo.org/dataset/3233

Click the button to launch the MapServer GIS for this dataset. The browser opens a new window or tab, and the display should look similar to this for URL: http://mapservice.bco-dmo.org/mapser...datasetId=3233

Some things to notice on the map shown on the previous page:

  • The KN192-05 cruise track, with the map zoomed in to the area of the cruise.
  • Up top, the MapServer is in “BROWSE map” mode.
  • Because we launched the MapServer from the text-based data discovery system, the only “Available deployment” (in the right-most panel directly above the map) is KN192-05, the only deployment contributing data to this dataset.
  • The “Map options” button (upper right-hand corner of the map panel) provides options for changing the map display (e.g. base map, projection, etc.) and printing the map. Note the small button to maximize the map window above the “Map options” button.

Once again, there are many ways to access additional information from this display:

  • Clicking the + box to the left of the KN192-05 cruise ID in the “Available deployments” or “Visible deployments” panels, displays metadata about that cruise including a link to the cruise report if one is available.
  • In the “Visible deployments” panel (in the top-right corner of the window), right click on the cruise ID to display available options for that cruise.

Down below (right) in the “Datasets” panel:

  • Clicking on the + symbol in the green circle, or anywhere on the row with the dataset name (nutrients and metals), requests that this dataset be ‘mapped’.

Try it … small colored dots will appear on the map, and the Datasets panel on the right has shifted to a display of “Mapped datasets”, of which there is only one at the moment. The single mapped dataset is listed, with a color-coded dot, the dataset name (nutrients and metals), a number (28) that indicates the number of sampling locations included in this dataset, from the selected cruise (KN192-05).

Once again, we have several ways to get more information: Right-clicking anywhere on the text in the row with the dataset name pops up a menu with several options:

  • “View/export mapped dataset” shows a tabular listing of the sampling locations, with options for data export;
  • “View mapped dataset on-line” opens the full dataset online;
  • “Choose a color” allows you to change the color of the dots on the map;
  • “MapServer link to mapped dataset” gives you the ability to save/bookmark the MapServer link to this dataset (for later access)
  • “Remove mapped dataset” removes this dataset from the map (removes the dots). Note that unchecking the box to the left of the dataset name temporarily removes the dataset. Checking the box again displays them.

The line underneath the dataset name shows more options/information for each dataset (different options are available depending on the data type).

The MapServer also provides the ability to generate ‘quick view’ plots of the data at a sampling location. On the map, select the dot for station 1 (the western-most point indicated by the red arrow in the following figure), to bring up a dialog box that offers: links to the data from the database and a way to select variables from the dataset to generate a ‘quick view’ X-Y plot. (If the dialog box ever gets in the way, just move it aside.)

Choose ‘Co_tot for the X-axis and ‘depth’ for the Y-axis. Click “view and get data” at the bottom of the box to generate the plot on the next page of total Cobalt concentration vs. depth.

Notice once again, several tabs at the top make it easy to view a tabular list of the data in the graph or download the data from the “File output tab”. Close the graph.

(Note: this dataset has the option to plot data from multiple stations, as indicated by the line stating “Click here to create an aggregate graph by sta” under the dataset name in the “Mapped datasets” panel. This function will be demonstrated in part 3 of this tutorial.)

Data access: MAP BROWSE scenario 2: You are interested in data from a particular geographic region

Go to: http://bcodmo.org/ the BCO-DMO home page

At the bottom of the DATABASE column on the left, click on the GEOSPATIAL ACCESS map.

The MapServer system map showing all the deployments from the BCO-DMO database opens in a new window or tab.

A common way to use the MapServer GIS is to define a region of interest on the map.

For example: you are interested in phytoplankton blooms in the North Atlantic.

Etc.

Data access: MAP KEYWORD SEARCH scenario 3: You are interested in data of a particular type from a particular geographic area

The MapServer has a KEYWORD search that is useful for finding certain types of data.

To test out the KEYWORD search function,

1. Click “Start over” in the top tool bar above the search panels (to clear the map and restart with a fresh map), and
2. click the “KEYWORD search” menu item toward the upper left-hand corner of the browser.

Enter something of interest in the “Keyword search string” text entry box.

For example, type in “pigments”.

If you are interested in pigments from only a certain geographic area, you can zoom-in to that area on the map, then choose “Yes” next to “Restrict results to map?”

You may also specify a date range in the minimum and maximum date fields.

For now, let’s just search for “pigments”. Click the “Run Search” button. The results appear in the datasets panel. Select “chlorophyll_phaeophytin”.

The map updates to show that these data have been reported from two deployments: OC404-01 and OC404-04. At this point, you can choose a different search result if this was not what you were looking for, or you close the results window. Let’s close the search results. Zoom-in to the cruise tracks (if needed).

The cruise tracks are displayed on the map, and the datasets available from both cruises are listed in the “Datasets” panel. They are grouped by dataset by default. Choose “deployment” from the “Group by” drop-down menu to change the grouping. (This is more useful for cruises that have many datasets.)

The datasets have been discovered and at this point one can proceed to map the datasets, view the data, or make quick view plots.

Data access: MAP SEMANTIC SEARCH scenario 4: You have an idea what you are looking for, but you do not know the Program, Project, or Deployment name

The MapServer ADVANCED search is a semantically-enabled, faceted search. It provides access to the same data as the “BROWSE map” and “KEYWORD search” modes, but the use of the search categories allows the user to construct their own hierarchical filter/search.

A beta version of this interface was released in May 2012, and is available as the ADVANCED search option (still as a beta version) from the MapServer.

Try it out …

The elements of the ADVANCED (semantic) search interface are similar to the MapServer layout you have seen in the previous scenarios in this tutorial.

Etc.

Glossary of Terms

BCO-DMO Biological and Chemical Oceanography Data Management Office: http://bcodmo.org/

GIS Geospatial Information System; a map system to display spatial data

MapServer Open Source software for publishing spatial data and providing interactive mapping applications via the Web: http://mapserver.org/

OCB Ocean Carbon and Biogeochemistry research program: http://www.us-ocb.org/

US GLOBEC GLOBal Ocean ECosystems Dynamics research program: http://www.usglobec.org/

US JGOFS United States Joint Global Ocean Flux Study http://usjgofs.whoi.edu/

Acknowledgments

Acknowledgments The MapServer interface is a custom implementation of the Open Source MapServer software developed at the University of Minnesota. The BCO-DMO MapServer system is the result of collaboration between programmer Charlton Galvarino (Second Creek Consulting, Columbia, SC) and BCO-DMO staff members. In particular, BCO-DMO marine biology data specialist, Dicky Allison has led this development effort. Funding for this work is provided by the National Science Foundation Division of Ocean Sciences and Division of Polar Programs.

Follow BCO-DMO

Email: info@bco-dmo.org
Twitter: @BCODMO
Youtube: https://www.youtube.com/user/BCODMO

Polar Hub: A Global Hub for Polar Data Discovery

Source: http://polar.geodacenter.org/polarhu...w/AboutUs.html

My Comment: I am did not readily find data with this like for the others

Creating the hub for polar data services!

PolarHub conducts large-scale web crawling for automatically collection of distributed web services to enhance the accessibility of polar data. A new technique named'Geobridge' is developed to enable this discovery .

PolarHub has discovered 1000+ OGC WMS, WFS, WPS, CSW services!

Step 1. Create

PolarHub allows authorized users to create new crawling tasks.

Step 2. Mine

PolarHub allows you to monitor real-time status of a crawling task. Locations and types of polar data services can be visualized on-the-fly.

Step 3. Discovery

PolarHub bridges the gaps between polar researchers and polar data services to accelerate science discovery.

This project is led by Dr. Wenwen Li at the School of Geographical Sciences and Urban Planning, at Arizona State University.

 
Vidit Bhatia is a graduate student at the Computer Science department at ASU.
 
Dr. Miaomiao Song is postdoctoral scholar at the School of Geographical Sciences and Urban Planning at ASU.

Contact Us: Prof. Wenwen Li | wenwen@asu.edu | (480) 727- 5987 | 975 S. Myrtle Ave., COOR Hall, POBOX 875302, Tempe, AZ 85287-5302

Index of /pub/requests/DVPC/

Source: ftp://amrc.ssec.wisc.edu/pub/requests/DVPC/

Name Size Date Modified
[parent directory]    
Ant_IR_area/   9/30/14, 8:18:00 PM
Ant_IR_netCDF/   9/30/14, 8:37:00 PM
AWS_dat_MAY_2014/   9/30/14, 3:23:00 PM
AWS_q10_MAY_2014/   9/30/14, 3:24:00 PM
AWS_q1h_MAY_2014/   9/30/14, 3:25:00 PM
AWS_q3h_MAY_2014/   9/30/14, 3:27:00 PM
AWS_r_MAY_2014/   9/30/14, 3:22:00 PM
readme.txt 4.6 kB 10/6/14, 6:37:00 PM
 
readme.txt

Notice on the use of Antarctic Meteorological Research Center data sets
 
The Antarctic Meteorological Research Center (AMRC) collects, archives and provides Antarctic meteorological observational data to the community and public for research, logistic, and educational activities. The AMRC requests acknowledgement for use of the data in any published work. See http://amrc.ssec.wisc.edu/acknowledgement.html for details on how to acknowledge AMRC data, displays or information. If the AMRC data are critical to the work, co-authorship may be appropriate.  Please contact the AMRC in such a case.        

AMRC Contact Information:
Address: 947 Atmospheric, Oceanic and Space Sciences Building
1225 West Dayton Street
Madison, Wisconsin, USA 53706
Telephone: +1 (608) 265-4816
Fax: +1 (608) 263-6738
E-mail: amrc@ssec.wisc.edu
Web: http://amrc.ssec.wisc.edu/
FTP: ftp://amrc.ssec.wisc.edu/
McIDAS ADDE:  Group AMRC and ARCHIVE on http://aws.ssec.wisc.edu
RAMADDA: https://amrc.ssec.wisc.edu/repository/
Facebook: http://www.facebook.com/AMRCAWS
Twitter: http://twitter.com/antmet
Google+: https://plus.google.com/115034961929...93/posts?hl=en
YouTube: http://www.youtube.com/user/AMRCantmet
Wikipedia: http://en.wikipedia.org/wiki/Antarct...esearch_Center
http://en.wikipedia.org/wiki/Antarct...ations_Project

Updated: 29 September 2012

Created by C. Costanza October 2014

Dataset for Data Visulatization of Polar Cyberinfrastructure in Novemeber 2014. All data is May, 2014. This set includes .r .dat .q10 .q1h .q3h AWS data and both area and netCDF formatted hourly Antarctic infrared composites

Data Formats:

*For all AWS formats 444.0 is missing data*

.dat
These files contain 3 hourly AWS data that had been quality controlled using our interactive IDL programs.The file names for the 3-hourly data are constructed using the station's ARGOS identifier, the month, and the year.
(IIIMMYYh.dat)

Column 1 - temperature (C)
Column 2 - pressure (mb)
Column 3 - wind speed (m/s)
Column 4 - wind direction
Column 5 - relative humidity (%)
Column 6 - delta-T (C)

.q10, .q1h, .q3h
These files contain 10 minute, 1 hourly, and 3 hourly AWS data that has been quality controlled using our interactive IDL programs. The files are labelled with a three letter code to indicate the station, the year, the month, and the frequency of the data.
(IIIYYYYMMq10.txt for 10 minute data)
(IIIYYYYMMq1h.txt for 1 hourly data)
(IIIYYYYMMq3h.txt for 3 hourly data)

Column 1 - year
Column 2 - julian day
Column 3 - month
Column 4 - day
Column 5 - ten minute observation time
Column 6 - temperature (C)
Column 7 - pressure (mb)
Column 8 - wind speed (m/s)
Column 9 - wind direction
Column 10- relative humidity (%)
Column 11- delta-T (C)

.r
These files contain 10 minute raw AWS data.The filenames of the data are set up as follows:  the first five digits are the ARGOS ID # of the unit, the next two are the month, followed by two more digits indicating the year.
(IIIIIMMYY.r)
 
Column 1 - julian day
Column 2 - ten minute interval marker
Column 3 - temperature (C)
Column 4 - pressure (mb)
Column 5 - wind speed (m/s)
Column 6 - wind direction
Column 7 - relative humidity (%)
Column 8 - potential temperature (K)

AREA
This is unique McIDAS format used to display our satellite images. For more information visit this webpage: 
http://www.ssec.wisc.edu/mcidas/doc/...formats-1.html

There are hourly Antarctic infrared composities available in AREA and netCDF

Some Results and Conclusions

My goal is to see if I can integrate and federate these multiple data sources and produce linked small multiples and semantically-enabled, faceted search by the use of the search categories that I and others have constructed iin the datasets.

So I have found used metadata and data from the following:

I was unable to find data at the Polar Hub - maybe some more data mining is needed by me.

I also mined the Hackathon Sessions for data sets, but was not able to find any except the AMRC-FPT - maybe it is too soon - but I found an excellent spreadsheet of Hackthon Resources that I visualized in Spotfire.

So I have the following so far in Spotfire:

  • Cover Page - Knowledge Base analytics and linked small multiples and semantically-enabled, faceted search.
  • Hackathon Resources - spreadsheet and linked small multiples and semantically-enabled, faceted search.
  • Polar Data Calalogue - spreadsheet and linked small multiples and semantically-enabled, faceted search.
  • AMRC-FTP - spreadsheet and linked small multiples
  • BCO Dataset - .spreadsheet and linked small multiples and semantically-enabled, faceted search.
  • BCO Program - 9 spreadsheets

This has helped me to identify individual datasets for the Polar Data Catalog and BCO-DMO that I can use in Spotfire for time series, correlation, and geospatial visualizations next.

For Internet Explorer Users and Those Wanting Full Screen Display Use: Web Player Get Spotfire for iPad App

BCO Program

NSFPolarCyberinfrastructure-Spotfire-BCO Program.png

Beaufort Sea Ice Thickness 2012

NSFPolarCyberinfrastructure-Spotfire-Beaufort Sea Ice Thickness 2012.png

Beaufort Sea Ice Thickness 2013

NSFPolarCyberinfrastructure-Spotfire-Beaufort Sea Ice Thickness 2013.png

Canadian Index of Ice and Snow

NSFPolarCyberinfrastructure-Spotfire-Canadian Index of Ice and Snow.png

MORE TO FOLLOW

Feedback

Brand Niemann, 
Chris Mattmann replied to your Tweet!

Brand Niemann @bniemannsr 

#nsfpolardatavis My Demo is the following: 4 Visualization Example 3: Spotfire Canadian Index of Ice and Snow pic.twitter.com/BfgAbPbRgi - 04 Nov 

Chris Mattmann @chrismattmann 
Follow 
@bniemannsr This is super cool I will point people to the remote session today also (did so yesterday) #nsfpolardatavis

04:54 PM - 04 Nov 14

Brand Niemann, 
Your Tweet got retweeted! 
 
Brand Niemann 

@bniemannsr 

#nsfpolardatavis Thank your for an excellent event! We are planning a Meetup on this in 2015: meetup.com/Federal-Big-Da… 

06:56 AM - 04 Nov 14 


Retweeted by 
Chris Mattmann @chrismattmann 
To 4522 followers.

Brand Niemann, 
Your Tweet got favorited! 
     
Brand Niemann 

@bniemannsr 

#nsfpolardatavis Thank your for an excellent event! We are planning a Meetup on this in 2015: meetup.com/Federal-Big-Da…         
06:56 AM - 04 Nov 14 
Favorited by 
Chris Mattmann @chrismattmann 
rocket scientist guy @NASAJPL tweets are my own #opensource #apache #bigdata and all that jazz.

Brand Niemann, 
You have a new follower on Twitter. 
           
Chris Mattmann 

@chrismattmann 

rocket scientist guy @NASAJPL tweets are my own #opensource #apache #bigdata and all that jazz. 

Pasadena, CA • http://sunset.usc.edu/~mattmann/

Followed by ESIP Federation and Chris C. Kemp.

Brand Niemann, 
Your Tweet got retweeted! 
           
Brand Niemann 

@bniemannsr 

#nsfpolardatavis My Demo is the following: 4 Visualization Example 2: Spotfire AMRC FTP Correction pic.twitter.com/giIGzBbR3E         
07:33 AM - 04 Nov 14 
Retweeted by Chris Mattmann @chrismattmann 
To 4521 followers.

Brand Niemann, 
Your Tweet got favorited! 
     
Brand Niemann 

@bniemannsr 

#nsfpolardatavis My Demo is the following: 4 Visualization Example 2: Spotfire AMRC FTP Correction pic.twitter.com/giIGzBbR3E         
07:33 AM - 04 Nov 14 
Favorited by Chris Mattmann @chrismattmann 
rocket scientist guy @NASAJPL tweets are my own #opensource #apache #bigdata and all that jazz.

Brand Niemann, 
Chris Mattmann replied to your Tweet! 
          
Brand Niemann @bniemannsr 

#nsfpolardatavis I have some Spotfire Visualizations now: semanticommunity.info/Data_Science/D… - 03 Nov 
Chris Mattmann @chrismattmann 
@bniemannsr This looks great!

02:49 AM - 04 Nov 14

Slides

Slides

Slide 1 Data Science Publication for NSF Polar Cyberinfrastructure

http://semanticommunity.info/
http://www.meetup.com/Federal-Big-Data-Working-Group/
http://semanticommunity.info/Data_Science/Federal_Big_Data_Working_Group_Meetup

BrandNiemann11042014Slide1.PNG

Slide 2 Preface

BrandNiemann11042014Slide2.PNG

Slide 3 Overview

BrandNiemann11042014Slide3.PNG

Slide 4 Data Science for Business: Data Mining Process

BrandNiemann11042014Slide4.PNG

Slide 5 Data Science for NSF Polar Cyberinfrastructure: Knowledge Base

Data Science for NSF Polar Cyberinfrastructure

BrandNiemann11042014Slide5.PNG

Slide 8 Polar Data Catalogue: Home Page

https://polardata.ca/

BrandNiemann11042014Slide8.PNG

Slide 9 Polar Data Catalogue: Collections

https://polardata.ca/pdcsearch/

BrandNiemann11042014Slide9.PNG

Slide 10 Polar Data Catalogue: Search

https://polardata.ca/pdcsearch/

BrandNiemann11042014Slide10.PNG

Slide 11 Polar Data Catalogue: Canadian Lake Ice Database

https://polardata.ca/pdcsearch/PDC_M..._Ver2003_1.zip

BrandNiemann11042014Slide11.PNG

Slide 12 Polar Data Catalogue: Sea Ice Thickness in Southern Beaufort Sea

https://polardata.ca/pdcsearch/PDC_M...kness_2012.zip

BrandNiemann11042014Slide12.PNG

Slide 13 Polar Data Catalogue: Spreadsheet

http://semanticommunity.info/@api/deki/files/31201/NSFPolarCI.xlsx?origin=mt-web

BrandNiemann11042014Slide13.PNG

Slide 14 BCO-DMO

http://www.bco-dmo.org/

BrandNiemann11042014Slide14.PNG

Slide 15 Data Access Tutorial 2014 OCB PI Summer Workshop

http://www.bco-dmo.org/files/bcodmo/OCB-Tutorial.pdf

BrandNiemann11042014Slide15.PNG

Slide 16 BCO-DMO Datasets

http://www.bco-dmo.org/datasets

BrandNiemann11042014Slide16.PNG

Slide 17 BCO-DMO MapServer Geospatial Interface

http://mapservice.bco-dmo.org/mapser...s-ol/index.php

BrandNiemann11042014Slide17.PNG

Slide 18 Polar Hub: A Global Hub for Polar Data Discovery

http://polar.geodacenter.org/polarhub/

BrandNiemann11042014Slide18.PNG

Slide 19 The AMRC at University of Wisconsin-Madison

ftp://amrc.ssec.wisc.edu/pub/requests/DVPC/

BrandNiemann11042014Slide19.PNG

Slide 20 Data Science for NSF Polar Cyberinfrastructure: Spreadsheet Knowledge Base

http://semanticommunity.info/@api/deki/files/31201/NSFPolarCI.xlsx?origin=mt-web

BrandNiemann11042014Slide20.PNG

Slide 21 Data Science for NSF Polar Cyberinfrastructure: Spotfire Cover Page

Web Player

BrandNiemann11042014Slide21.PNG

Slide 22: Data Science for NSF Polar Cyberinfrastructure: Spotfire Visualizations

Web Player

BrandNiemann11042014Slide22.PNG

Spotfire Dashboard

For Internet Explorer Users and Those Wanting Full Screen Display Use: Web Player Get Spotfire for iPad App

Error: Embedded data could not be displayed. Use Google Chrome

Workshop Information

Call for Remote Participation

The NSF DataViz Hackathon for Polar CyberInfrastructure will bring together Polar researchers, Cyber Infrastructure experts, Data Visualization experts, and members of the community interested in connecting technology, science and communication.

The Hackathon website is located at:

http://nsf-polar-cyberinfrastructure.github.io/datavis-hackathon/

The Hackathon will take place tomorrow, Monday, November 3, 2014 and Tuesday November 4, 2014, beginning at 9am ET and located at:

The Orozco Room Parsons - The New School 66 W 12th St, 7th floor (Between Fifth Avenue and Sixth Avenue) New York, NY 10011

Though onsite attendance is closed, we are offering remote participation in the meeting via our Github repository:

https://github.com/NSF-Polar-Cyberinfrastructure/datavis-hackathon

You can participate in the meeting by reviewing, commenting and providing your feedback on our current sessions, located at:

https://github.com/NSF-Polar-Cyberinfrastructure/datavis-hackathon/issues

Please note that a Github account is required to participate.

We look forward to your remote participation and to the hackathon and its results!

Federal Big Data Working Group Meetup

Our Federal Big Data Working Group Meetup will be participating in this, but not directly because we have our own Meetup on November 3rd in Washington, DC.

We will be posting our work at:
>http://semanticommunity.info/Data_Science/Data_Science_for_NSF_Polar_Cyberinfrastructure

​Create a Session

That sounds awesome and you should definitely be able to remotely participate. Please consider creating a session on our Github that simply points back to your wiki page below.

To create a session:

1. Register your (or use your existing) Github account 2. Head over to: 
https://github.com/NSF-Polar-Cyberin...ckathon/issues
3. Create a new issue with the session title and description.
(here you can put something like:
Data Science for NSF Polar Cyberinfrastructure

And the description from your wiki page

Once you do that, I’ll go ahead and add the proposed session and data science labels to it and it will automatically show on the website! :)

Thanks and really appreciate this!

Details of the Workshop

The invite email to participants is here:

https://github.com/chrismattmann/NSF...ics/invite.txt

Here is the logistics email: https://github.com/chrismattmann/NSF...info1email.txt

Here is our website: http://nsf-polar-cyberinfrastructure.github.io/datavis-hackathon/

and our Github issue tracker where folks are proposing ³hackathon² sessions around data, code, or a visualization (or all three):

https://github.com/NSF-Polar-Cyberinfrastructure/datavis-hackathon/issues

Please let us know if you can attend the meeting. Did you get my prior email about confirming your participation in the workshop? I¹d be happy to answer any and all questions you may have. Sorry for the delay in replying!

Cheers, Chris

Chris Mattmann, Ph.D.

Chief Architect

Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA

Office: 168-519, Mailstop: 168-527

Email: chris.a.mattmann@nasa.gov

WWW:  http://sunset.usc.edu/~mattmann/

Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA

Invite Email to Participants

Source: https://github.com/chrismattmann/NSF...ics/invite.txt

Dear Interested Participant,

You have been selected to attend the NSF DataViz Hackathon for Polar CyberInfrastructure after expressing interest in the event.

Congratulations and thank you for your interest!

We are in the process of putting up a website right now, via Github.You can find the website here:

http://nsf-polar-cyberinfrastructure...vis-hackathon/

The Github Source for the website is here:

https://github.com/NSF-Polar-Cyberin...vis-hackathon/

In the coming week you will see more and more information pop up on that website, for now please find the following logistical information:

1. The workshop will take place in New York City, New York on Monday November 3, 2014 and Tuesday November 4, 2014.

2. The workshop will take an "unconference"² and "hackathon" style to it, much like the recent Open Science Codefest (also sponsored by NSF). See here: http://nceas.github.io/open-science-codefest/

In this vein, we are asking participants to help create "sessions" as Github issues (please register for Github as we will be tracking all of our conference information there). Each issue that you create here: https://github.com/NSF-Polar-Cyberin...vis-hackathon/ will become a "session" on the website  for the event. Use the issues as means for data preparation, for discussing and collaborating ahead of time for the event amongst your peers, and for planning on what data, visualizations, code and other fun stuff you will be working on at the hackathon. We will spend some time in the beginning of the meeting going through all of the proposed hackathon sessions, and organizing and potentially collating similar ones, and pairing them down to a manageable set to work on. Note that if you are proposing sessions that involve specific data, we expect you to show up to the meeting with a data brick or with some example to share/host with others you are working with at the meeting.

3. The meeting place will be: 11/3 and 11/4 (9am - 4:30pm both days) 

Parsons New School Orozco Room, Alvin Johnson/J.M. Kaplan Hall 66 West 12th Street, Room A712, New York, NY 10011

Some images of the meeting location are here:
http://www.newschool.edu/leadership/...school-murals/ 
http://www.newschool.edu/ofb/busines....aspx?id=51095

4. If you requested travel participation we will email you separately to confirm your request and to work details out with you.  Costs will be reimbursed for Flights up to $680 dollars per participant and hotel costs up to $285/night for 2 nights in NYC area. We are working with NSF to obtain a limited amount of additional travel support and we will keep everyone posted on that (that requested it).

5. Please note that our goal is to create amazing visualizations of data relevant to the Polar community and to the Polar CyberInfrastructure community.  So, any data prep that can be done ahead of time is most valuable. Please use Github to coordinate and the existing shared repository. We will be collecting the code and visualizations generated during the workshop and asking participants to license their code and visualizations under the Apache 2 Software License (³ALv2²). You can find a copy of the ALv2 here: http://www.apache.org/licenses/LICENSE-2.0.html This will allow the most reuse of the output of the workshop and the ability for folks to reproduce the work and to share the visualizations and data generated. This workshop includes invites (your peers) that are Cyber Infrastructure, Polar Scientists, Data Visualization experts, data processing experts, Remote Sensing and GIS experts, Urban Planning, and a host of other talents that I¹m sure we don¹t even know about yet.  Please begin collaborating on Github and working together so that we can make this an awesome event and generate some real value to the NSF and to the community. The work you perform here will have a large impact on community.

6. Some prep work is already underway (if you scour the Open Science Codefest site you will find some) to prepare some datasets of relevance to the Polar community. We will provide some of this prepared data to interested parties ahead of the workshop in the next few weeks in case folks want to start hacking early. We will tweet under the hash tag:

#nsfpolardatavis

So please feel free to start chatting and getting the buzz going!

Thank you all and apologies for the delay in getting this out. We expect to have frequent updates coming in the following weeks and look forward to working with you all to make this an amazing event!

Cheers, 
Chris Mattmann 
(on behalf of the organizing committee)

Logistics Email

Source: https://github.com/chrismattmann/NSF...info1email.txt

Dear NSF DataViz Hackathon for Polar CyberInfrastructure Participants,

Thank you for your patience as the meeting organization has really been coming together!

We have just emailed today information to those that requested travel support for the meeting - so if you did request support and haven¹t received our contact, please do let us know.

The meeting is now 18 days away and will be here before you know it!

The logistics are really coming together on the website: http://nsf-polar-cyberinfrastructure...vis-hackathon/

New information available:

1. Suggested hotel logistics (use information to book your hotel to get New School¹s rate) 
2. Parking information and information on how to get to New school 
3. A draft agenda (that will be updated likely a few times over the next week or so) 
4. Proposed sessions are coming in (more on this later!) and there is information on the website for how you should submit a session.

We are asking everyone on this list to *please confirm no later than Monday October 20th, 2014* your participation in this meeting. You can do so simply by sending an email to nsfdatavis@gmail.com with the subject line:

[NSF DataViz Hackathon] Confirm participation of <Your First Name Your Last Name> where <Your First Name Your Last Name> if you were me would be Chris Mattmann and the full subject line would be:

[NSF DataViz Hackathon] Confirm participation of Chris Mattmann

We would like to publish a list of participants on the website by Monday and we require your confirmation before publishing that list.

Speaking of sessions, here¹s how you can get involved!

0. Register for Github and get an account if you don¹t already have one: https://github.com/join

1. If you have data, code, or a visualization (or some subset of those three), please first read the sessions already present. They are visible on the website under the heading ³Sessions², which you can find from the left hand navigation bar (click on the 3 stacked line bars in the upper left of the page to get the menu, then click on Sessions). If your idea falls under one of the existing sessions, please start commenting and interacting there. The organizers and leads will appreciate your contributions and ideas. We¹ll use this both leading up to; during; and after the workshop to stay in touch If you don¹t see a session there, please see these instructions for submitting a new session: https://github.com/NSF-Polar-Cyberin...hackathon/wiki


2. Begin collaborating with your other participants in the proposed session. Expose your data to them. Discuss what you are going to visualize.  Talk about the technologies and tools. All of these things will be your interaction points and will help to ensure there is a clear and well defined goal when you guys hit the ground running at the hackathon in a few weeks in NYC. If you aren¹t sure what to do, comment that.  If you need one of the other participants to check your code out, link it to them. Some of the existing sessions are already talking about and sharing data as well.

3. Get social - use Twitter and Hash Tag #nsfpolardatavis to start tweeting to your network to get people excited and commenting about the event.  Talk about it on Facebook or Instagram or Google Plus. Jump into IRC on irc://irc.freenode.net/#nsfpolardatavis and start meeting some of the folks you are going to be hacking and data-vizzing with.  Instructions for using IRC are here: https://meta.wikimedia.org/wiki/IRC/Instructions

4. Show up to the workshop ready to get a little technical, but to also provide key science input into the meeting. This hackathon was envisioned as a meeting of the minds between CI people and Polar people, and is a meant as one of the natural follow ons from last year¹s very successful NSF Polar CyberInfrastructure workshop at the Polar Geospatial Center. In addition, we are drawing and borrowing from experience at the NSF funded Open Science Codefest and other recent hackathon activities that we have been organized in. The more you put in the more we all will collectively get out of this.

Finally please keep watching the website and comment if there is any question that you aren¹t sure of, or if you simply want to reach out. Github is our first place, or you may also email any of the organizing committee, myself, and/or use one of the many other communication mediums (Twitter, IRC, etc.)

Thank you and we will be providing regular information updates as they are available. Thank you all and looking forward to a successful workshop!

Cheers, 
Chris 
(on behalf of the organizing committee)

Call for Participation: NSF DataViz Hackathon for Polar CyberInfrastructure: NYC, November 3-4, 2014

The U.S. National Science Foundation is sponsoring a workshop related to data visualization in the Polar Sciences. The workshop will focus on bridging the cyberinfrastructure/data visualization and polar communities and it is scheduled to be held in New York City at the Parsons New School for Design November 3-4, 2014.

Improving the use and the value of existing datasets over the Polar Regions is crucial to promote science and support new discoveries.

Ultimately, collaborations between data visualization experts and Polar scientists will foster a greater understanding of the spatio-temporal dynamics at play in Polar Regions and their implications to society.

The participants will be motivated by several public Polar datasets that will made available before the starting date of the workshop.

One of the expected outcomes of the workshop is to produce high impact novel prototypes and data visualizations that will be made available under open source licenses. Releasing the prototypes will allow the NSF to make longer-term investments in technologies and visualizations that can be adopted by the community. The workshop will also increase cross agency collaboration between NSF, NASA, NOAA and other Polar-related agencies. The organizing committee is composed by both cyberinfrastructure and Polar experts, including participation from academia, industry, federally funded research and development centers, and from the broader open source community.

The workshop will:

(1) recommend several sets of open source software for data and metadata processing; scientific workflow management; data curation; and data dissemination;

(2) identify some relevant

Polar data visualization techniques and assess the needs and challenges of visualizing Polar datasets;

(3) package, deliver, and

make available the outcomes of the workshop via a public website; and;

(4) provide input to the NSF Polar CyberInfrastructure program officer through a final report.

Travel support is available for around twenty-five workshop participants that will be selected meritocratically based on interest and based on recommendations from the community.

An organizing committee for the meeting is being formed, with current membership listed below:

* Dr. Chris Mattmann, University of Southern California & Jet Propulsion Laboratory, California Institute of Technology

* Dr. Annie Bryant Burgess, University of Southern California

* Dr. Suzanne Carbotte, Columbia University

* Dr. Bruce Caron, New Media Research Institute

* Dr. Patrick Driscoll, Aalborg University

* Mr. Christopher Goranson, Parsons Institute for Information Mapping

* Mr. Aaron Hill, Parsons New School for Public Engagement

* Dr. Daniel Katz, National Science Foundation

* Dr. Martin Lehmanm, Aalborg University

* Dr. Alan Maceachren, Penn State University

* Dr. Jonathan Pundsack, University of Minnesota Polar Geospatial Center

* Dr. Marco Tedesco, National Science Foundation

* Mr. Joel Towers, Parsons New School

* Dr. Saskia Van Manen, Open University

* Dr. Alexander Lex, Harvard

Datasets for the workshop will be made publicly available via the Cloud to workshop participants around two-weeks prior to the start of the workshop. The workshop format will consist of a 2-day series of invited speakers in Polar Sciences and CyberInfrastructure and Data Visualization to motivate the start of the art and challenges to the community. Four interactive "hackathons" will provide the opportunity for workshop participants to break off into teams and develop novel data visualizations on the provided Cloud datasets. Hackathon results will be shared and disseminated during the workshop read out and will be made available to the community under permissive open source licenses (e.g., the Apache License, version 2).

Please reserve the dates in your calendars and we welcome your inquiries about the workshop. Please send email to the workshop committee at the following address and we will respond promptly to your inquiries.

nsfdatavis@gmail.com

Research Notes

Qlik Hackathon

Source: http://go.qlik.com/Customer-Conference-Hackathon.html

One app could change everything. Will it be yours? Qlik and the United Nations invite you to turn your ideas into apps that will have a social good impact and make a difference in people’s lives at the Qlik World Conference 2014.

24 hours before the first keynote, join developers, technologists and Qlik experts from around the world to collaborate and engage with data to design innovative solutions for global United Nations challenges. Leveraging Qlik Sense you’ll be challenged to use your skills to achieve new insights and/or visualizations that best address identified challenges. The results of this event could be extremely beneficial in impacting the ability of United Nations programs to effectively improve peace and security, socio-economic development, human rights, humanitarian affairs, and international law.

The Hackathon takes place on Monday, November 17 in Orlando Florida at the Qlik World Conference 2014. Developers are invited to create something extraordinary and useful based on unique United Nations and social media datasets as well as Qlik Sense APIs. Every Hackathon participant will have hands-on access to Qlik Sense and its new capabilities in web integration, with ‘live’ enablement and support from internal Qlik experts who will be moderating the event.

Join to network, learn about new technologies, refine your skills, collaborate with other app enthusiasts, and “Change Our World”. This is your chance to make a difference for a cause that matters, "become the ultimate hacker for good", while you compete for great prizes.

FAQs

Is there a cost to participate in the Hackathon?

No, as long as you are a full Qlik World Conference 2014 registrant, you may apply to participate in the Hackathon.

I’ve already registered for Qlik World Conference 2014, how do I register for the Hackathon?

You simply need to log back into your Qlik World Conference account and add the Hackathon to your registration.

I work for a partner/vendor. May I participate?

Yes, as long as you are registered for the conference as an attendee or as an exhibitor.

Do I need a teammate?

Nope, don’t worry if you don’t have a buddy to team up with. We actually prefer it that way. Participants will be assigned into groups of two during the welcome breakfast session at the beginning of the Hackathon.

If the Hackathon fills up, can I still join?

The Hackathon has a 50 participant limit. We will be creating a short waiting list for developers who register after we reach our limit. If we get cancellations we will begin reaching out to those on the waiting list to invite them to participate.

Where is the Hackathon?

The Hackathon will be held at the Rosen Shingle Creek in Orlando, FL on Monday, November 17, 2014 from 8:00 AM to 5:45 PM. Please pick up your Qlik World Conference 2014 badge before coming to the Hackathon.

When will I learn more about the challenges?

You and your team will hack on big problems to support a worthy cause. Teams will be presented with several challenges benefiting the United Nations, a Qlik “Change Our World” grant recipient and partner. The Qlik Hackathon challenges will remain secret until the day of the event. However, attendees will be provided with materials beforehand to help prepare for the event. Developers must use web skills, data and the Qlik Sense APIs to achieve new insights and/or visualizations that best address the challenges.

May I promote my participation in the Hackathon?

Of course – it is encouraged! This event is about learning and sharing knowledge. Use #QLIK14 to let everyone know what you are up to.

Will there be prizes?

Yes. Teams with the best apps can win a wide range of prizes. We’re co-organizing the Hackathon with some wonderful companies including Attivio, HP Vertica and Twitter who have agreed to provide some great prizes for the challenges.

What is the judging process?

Results from the challenges will be presented during the Qlik World Conference 2014 to a United Nations representative and panel of judges. The teams that create the three best applications, as declared by the judging panel, will be awarded prizes.

What should I bring?

  • A laptop suitable for coding with your preferred development tools installed
  • Great ideas and a hunger to hack

What will Qlik provide?

  • An amazing start to Qlik World Conference 2014!
  • Sample data
  • Qlik Sense APIs
  • Food and beverages
  • Whiteboards, power strips, flip charts, wireless internet access
  • A Hackathon t-shirt that will be the envy of all your friends!

Do I need web development experience or experience with certain types of apps/coding to participate?

This is a highly technical event focusing on web developers and individuals with strong web and JavaScript skills. Attendees should have a high level of experience in developing websites or web applications using Javascript (and Javascript APIs and libraries), HTML and CSS with bonus points for database experience. Qlik Sense offers a variety of great new JavaScript APIs that developers can leverage on the web. If you’re technical with experience in design and UI/UX or a QlikView Developer, you are invited to participate in the Qlik Hackathon, however it is highly recommended that your buddy and teammate is an experienced and strong web developer. Prior experience with Qlik Sense and its extensible platform will be advantageous.

Qlik Branch

Qlik Branch is a collaborative workspace and open exchange. It’s a place for developers to innovate and take advantage of the capabilities of Qlik products. Here we can all feel free to share anything from APIs and extensions to web mash-ups and KML data (and everything in between). Beyond simply condensing these projects to one central location, Qlik Branch aims to make collaboration between developers easier. Join the community today to gain insight, collaborate on projects, pull from libraries, and share. Qlik Branch (beta) – http://branch.qlik.com – open up a whole new world of integration and extensibility.

ESIP Preservation and Stewardship Committee Telecon 2014-10-17:

https://docs.google.com/document/d/1...h.6x05e5y46s6n

Agenda:
* Preparations for half-day workshop on citing dynamic data at ESIP winter meeting
* Picking a direction for the PCCS work - to generalize or restrict to certain categories of data
* How to move forward with NOAA’s Data Stewardship Maturity Matrix work
* Next round of comment on Citation guidelines for publishers, editors and reviewers
* Paper status (if time)

Dynamic working group in RDA??

Need a person knowledgeable of the particular data set to attend the meeting.

Only need to know the types of data sets used, what the users want and not the science.

The emphasis is on the dynamic data set - how often changes occur and the types.

Have a server with dynamic things.

The person there should be able to have some authority to make changes to their system.

Between now and the winter meeting - data set owners (for the workshop) should look at the PCCS and see if it would work for their data and make notes about what would not.

Work through a list of the processes for dynamic data for 3-4 data sets

Suggestions so far: 
NCAR Research Data Archive
IEDA
BCO-DMO My Note: See Below
Polar Data Catalogue
USGS National Water Information System (NWIS)
NASA Last Satellite (space related - Anne)
Set from Natalie (mentioned on call, see below for details).

The ESIP workshop will be customized to the community needs, so please review the materials below and either reply to the list with comments or attend the call to take part in the discussion.

To help you think about these issues, below are a few resources.  The minutes from last month’s call where we discussed the workshop, the program from the previous workshop in London, and two white papers from the workshop leads. 

Resources:

Last month’s telecom minutes for a discussion about the workshop

https://docs.google.com/document/d/1rzA84s4-HtAzMpO_oVohvELwY6gfHJG5xiybsjoXx4k/edit?usp=sharing

Example of previous workshop

https://rd-alliance.org/sites/default/files/RDA_WG-DC_Workshop_London_Programme.pdf

White papers as background:

Providing Concise Data Citations to Enable Reproducibility Across Different eScience Disciplines

https://drive.google.com/file/d/0B4k6w8oONy3XcTBkdHRoR3pwYWM/view?usp=sharing

Scalable Data Citation in Dynamic, Large Databases: Model and Reference Implementation

https://drive.google.com/file/d/0B4k6w8oONy3XMjJkU3lvUGxWaUk/view?usp=sharing

The Polar Data Catalogue has lots of CTD data (at least 51 cruises) that can be downloaded for free (also other data available, too), if it would fit your needs.  Just go to https://polardata.ca, click on the map (for the PDC Geospatial Search), then select the Polar Data Catalogue collection.  You can do a keyword search for CTD or do a map search in a particular location.  May want to click the “[ ] View downloadable datasets only” check box, so that you only see results with data available for download.

I'd like to add that BCO-DMO has data that may be of potential use for the workshop. Mark suggested "CTD casts from multiple ocean cruises", which BCO-DMO can provide along with any associated biological and chemical data from complete and ongoing projects.

NSF Polar Cyberinfrastructure

Source: http://nsf-polar-cyberinfrastructure...vis-hackathon/

DataVis Hackathon

November 3 - 4, 2014
#nsfpolardatavis
 nsfpolardatavis G+ Event
Please share your photos and comments on our G+ Event

Abstract

This workshop will focus on the bridging of the cyberinfrastructure/data visualization and polar communities and it is scheduled to be held in New York City at the Parsons New School for Design in November 2014. Improving the use and the value of existing data sets over the polar regions is crucial to promote science and support new discoveries. Ultimately, collaborations between data visualization experts and polar scientists will foster the understanding of the variability of the polar regions at different timescales, with implicit benefit for the society. The participants will be motivated by several public Polar datasets that will be acquired and made available before the starting date of the workshop. One of the expected outcomes is to produce high impact and novel prototypes and data visualizations that will be made available under open source licenses. Releasing the prototypes will allow the NSF to make longer-term investments in technologies and visualizations that can be adopted by the community. The workshop will also increase cross agency collaboration between NSF, NASA, NOAA and other Arctic/Polar related agencies. The organizing committee is composed by both cyberinfrastructure and polar experts, including participation from academia, industry, the federally funded research and development centers, and from the broader open source community. The workshop will: (1) recommend several sets of open source software for data and metadata processing; scientific workflow management; data curation; and data dissemination; (2) identify some relevant Polar data visualization techniques and assess the needs and challenges of visualizing Polar datasets; (3) package, deliver, and make available the outcomes of the workshop via a public website; and (4) provide input to the NSF Polar CyberInfrastructure program officer through a final report.

Location

Parsons - The New School
66 W 12th St, New York, NY 10011

Sessions

My Note: I created a new session and minded the other sessions for data sets for our Data Science Publication

This is a parent issue which tracks the creation and maintenance of data and output relating to every independent session. 
Justification is so that

  • sessions are individual... with individual outcomes
  • people wishing to check out the outcomes of an individual repos can merely clone the repos they are interested in
  • we can see what worked and what did not
  • people can document notes on the repos specific wiki. This will reduce clutter on a session-by-session basis.
I'm recently found the is cool Bitergia (http://bitergia.com/) to see as a tool that we can analyze data to evaluate community efforts by getting, providing and analyzing Software Development metrics and information about FLOSS projects and communities for managing and improving them . After hosting a hackathon, we expect to see the marvel of growing these session ideas carried away to perform as active communities. We will be able to find the meritocracy, code review activities and active mentorship among the communities. This will also help to understand community interests based on the topics.

Even though "a picture is a thousand words," some things are "easy" to write about but hard to show in a picture. But often that one graphic is more powerful than 10,00 words. This session would try to take advantage of the full breadth of workshop attendees by pulling together a full story/visualization of an "issue" that has been difficult to visualize/explain in polar science. A target audience is proposed as the "newspaper reading public."

I would suggest starting with something that has been well written up in one or more sources so that we can focus on the visualization side of the narrative. Something else to keep in mind will be data available to create any visualizations. Some thoughts on what to focus on would be:
-Recent papers on the collapse of Pine Island / Thwaites Glaciers
-Arctic vs Antarctic sea ice processes
-Weather patterns' role in disintegrating a weakened sea ice pack (or not)
-Other suggestions?

A few links to get started...
• "These Simple Tips Will Make Your Science Visualizations Rock:" http://io9.com/these-simple-tips-will-make-your-science-visualizations-1633922235
• "Data visualization: A view of every Points of View column:"http://blogs.nature.com/methagora/2013/07/data-visualization-points-of-view.html

 
This seems as a good as place as any (but applicable to many sessions): NSIDC Developer Soren Scott put together a list of visualization tools which might be helpful for Hackathon Attendees -https://docs.google.com/spreadsheets/d/12DCw2YSa9wUeG24DjCEGXoZQTYyiFgPK0HNQV-qeswg/edit#gid=0 
 
My Note: I included this spreadsheet.
 
The national climate data center hosts an ice core archive (http://www.ncdc.noaa.gov/data-access/paleoclimatology-data/datasets/ice-core), with an interactive map AND google earth map that allows you see information on the core and then link back to the data either via text file or download from FTP. While I think this is already an incredibly useful service, I imagine an archive that is more user friendly, perhaps something that looks like: (source http://htcexperiments.org/2008/09/) with cylinders instead of buildings. Cylinder height could correspond to core depth or dating, and when you click on the cylinder (or cylinder layer) you retrieve associated information, a visual representation of the selected core, available attributes, and links to raw data and publications that came from the core. Cylinders could be grouped by region, or even arranged in the shape of the region so you can quickly select the core archive in the region you are most interested in.
There has been some work has been done on ice core visualization (http://earthobservatory.nasa.gov/Features/Paleoclimatology_IceCores/), but from what I have tried to discover this seems limited to variable plots that allow you to scroll through time. While these are useful, I think there could be more exciting way to visualize individual core data. This could be as simple as a cylinder with a color ramp that corresponds to a certain variable (I'll try to make a series of sample visualizations before the hackathon). I also think cutaways (like- http://svs.gsfc.nasa.gov/cgi-bin/details.cgi?aid=20110) would be interesting to visualize location and core depth.

Imagine this not-so-hypothetical scenario: It's exciting - I'm a researcher and have found some ocean model output to help understand changes to the ice sheet I've seen. The problem is I'm used to working with elevation change and glacier albedo, so this data cube is a new type of data for me and I'm not sure how to interact with it easily. What is a research to do?

This session will come up with a solution to this problem - how to explore and interact with a netcdf "cube" or data where each variable (zonal velocity, meridional velocity, potential temperature) has 4 dimensions (x, y, z, and time). I'm thinking something which lets the user select which variables/dimensions to show, sliders to go through the data, editable color bars, etc.

Soren Scott at NSIDC suggested potentially monkeying around with WebWorkers? Or other web-examples like:
• http://tools.pacificclimate.org/dataportal/downscaled_gcms/map/
• http://www.globalcarbonatlas.org/?q=flux_maps
• http://earth.nullschool.net/#current/wind/surface/level/orthographic=-90.00,0.00,330
Toni Rosati at NSIDC suggested that Vapor (https://www.vapor.ucar.edu/) might be a good start, too.

I also have and example dataset ready to go (it's MIT-GCM output). ~2.7GB in .nc - once I know where to put it / how to do so...

In addition, when more developed, something like Toolmatch could help solve this sort of issue in the future: http://toolmatch.esipfed.org/#

Change over time in 2D

data science proposed session by allenpope

Looking at changes to the poles is central to a lot of polar science, but scientists often still have a hard time visualizing their data through time, especially in 2D formats appropriate for (for example) journal articles (we will decide on the best audience, very impt for visualization). Sure, we can use small multiples, but that often isn't enough. Even in seemingly simple timeseries of raster imagery, it is often difficult for scientists to publish/share their timeseries. Rates of change (implicit to many timeseries) can be difficult to visualize as well.

This session will identify such a challenging (and important) dataset and put together a good visualization for it. Possible datasets to use could be a Modis or Landsat series, surface temperatures, sea ice concentration data, historical Nimbus data, etc.

A few links to get started...
--"These Simple Tips Will Make Your Science Visualizations Rock:" http://io9.com/these-simple-tips-will-make-your-science-visualizations-1633922235
--"Visualizing time-oriented data—A systematic view:"http://www.sciencedirect.com/science/article/pii/S0097849307000611
--"Data visualization: A view of every Points of View column:"http://blogs.nature.com/methagora/2013/07/data-visualization-points-of-view.html

I saw that you were thinking about print visualizations in particular. But if you were also considering interactive visualizations, FlowingData recently had an tutorial on linked small multiples with some examples. http://flowingdata.com/2014/10/15/linked-small-multiples/ 

My Note: Spotfire does this

Integrating data discovery, analysis and visualization into polar cyberinfrastructure -- the PolarHub solution

data science proposed session by liwwchina 

This session will introduce the research progress of ASU researchers on a service-oriented Polar Cyberinfrastructure that integrates data search, intelligent online analysis and visualization to support polar sciences. The core component of this cyberinfrastructure portal is PolarHub, which has the ability to conduct large-scale web crawling to discover distributed geospatial data in the format of OGC web services. We are also working to integrate multi-dimensional visualization techniques to support more intuitive data presentation and analysis.

The link to the work is here: http://polar.geodacenter.org/polarhub/

Just in case other polar people are confused - there is another PolarHub (http://thepolarhub.org/) - but that one is focused on polar & climate education & outreach.

"Higher Dimensional" Polar Data Visualization

data science proposed session by alb0

Data visualization need not stop at a 2d image, especially since the data themselves are geospatial. 

I have been 3D printing various polar datasets, but have met many challenges along the way which may prevent others from embracing 3D printing's potential.  These include having to grapple with obtuse architectural modeling software, adjusting the resolution and scale of the data to suit the printer, physical limitations of the printer itself, changing the format of the data to suit that of the printer...

I would like to discuss:
How to make 3D printing easy for scientists to use
If 3D printing can be used for polar research purposes beyond data visualization. What tools would we need for this to happen?
Can we make interesting and accurate models for people to download and print on their own?

Along the similar lines, I think it is possible to create an interesting "installation" of a these datasets using light or sound. This could be very effective and fun to make for data that are not inherently visual.

If there is an interest, there is a lot of potential for creating visualizations that go beyond the computer screen.

i have added some data and code to a repository in my account called "3d greenland" since i couldn't seem to get pull request to work. more to follow

we can definitely print arctic bathymetry and i think it'd be really cool to overlay geopolitical data... i hadn't thought of that before!

Design an outline for adding analysis and vis algorithms from the Polar domain to Apache Open Climate Workbench data

data science proposed session by lewismc

Apache Open Climate Workbench (OCW) is an effort to develop software that performs climate model evaluation using model outputs from a variety of different sources (the Earth System Grid Federation, the Coordinated Regional Downscaling Experiment, the U.S. National Climate Assessment and the North American Regional Climate Change Assessment Program) and temporal/spatial scales with remote sensing data from NASA, NOAA and other agencies. The toolkit includes capabilities for rebinning, metrics computation and visualization.

We recently integrated some cutting edge algorithms for searching for graph-based search for the identification and chatacterization of Mesoscale Convective Complexes (MCC).

I would REALLY like to

  • abstract some of the identification, characterization and search functionality present within the MCCSearch codebase
  • find a suitable area of relevant Polar-oriented science, of which there is a Polar expert in this field present at the hackathon
  • hack the hell out of the codebase in the form of a polarviz branch which would make ways to improving the codebase as well as expending the Apache OCW community in the process. The continued success of OCW relies heavily on engagement with leading scientists... this issue is an effort to advance towards that goal.

NASA Near Real-time Polar Imagery Services

data science proposed session by jschmaltz

NASA's Global Imagery Browse Services (GIBShttp://earthdata.nasa.gov/gibs) was developed to provide highly responsive, scalable, and expandable imagery services using Open Geospatial Consortium (OGC) Web Map Tile Service (WMTS) standards.

Currently, there are more than a dozen MODIS imagery products available in polar stereographic projections for each pole, including four daily one kilometer 11 micron thermal infrared band images during all seasons. Imagery back to mid-2013 is currently available and reprocessing of imagery from the entire MODIS record is underway and community input is being solicited on recommendations for additional imagery layers from MODIS and other NASA instruments.

Here is the GIBS web examples page in GitHub:

https://github.com/nasa-gibs/gibs-web-examples

And some live examples:

https://earthdata.nasa.gov/labs/gibs/examples/leaflet/arctic-epsg3413.html
https://earthdata.nasa.gov/labs/gibs/examples/leaflet/time.html

Examples of the ability to retrieve imagery via scripts:

https://wiki.earthdata.nasa.gov/display/GIBS/GIBS+API+for+Developers#GIBSAPIforDevelopers-Script-levelAccessviaGDAL

I would like to give a brief demo of the imagery layers and access methods.

For a full list of currently available products in GIBS, see here:
https://wiki.earthdata.nasa.gov/display/GIBS/GIBS+Available+Imagery+Products

Open-source Polar Data workflows with Tangelo-Hub

data science demo proposed session by curtislisle

In this session, I will demonstrate Tangelo, a rapid prototyping web application framework with a small learning curve (http://tangelo.kitware.com/ and https://tangelo.readthedocs.org/en/v0.6.1/) useful for creating and hosting dataset visualizations through public websites.

After a short introduction to Tangelo, I will focus particularly on how to process polar datasets in Tangelo-Hub, a web-hosted, datascience workflow processing and analysis application funded by NSF's Biology division (https://github.com/tangelo-hub and http://www.arborworkflows.com).

Tangelo-Hub allows users to create and run multi-step workflows interactively through a web interface. Quick visualizations and dataset management are built in. Individual steps in a workflow can be implemented in the Python or R languages.

An example of a tangelo-hub workflow is shown below as it is viewed through a browser session:

screen shot 2014-09-18 at 10 04 14 pm

During this session, I will present a use case performed prior to the hackathon on one of the Polar Datasets using Tangelo-Hub. Participants will then be able to get hands on experience with a Tangelo-Hub instance to process datasets themselves during the session.

Polar Data Analytics as a Service (PDAaaS)

data science proposed session by lewismc

Based on the outcome and success of ISSUE-14 which is ETL 101 - Bringing Polar data analytics one step closer to the Polar Scientist, I would really like to explore the concept of moving towards a PDAaaS-type model for Polar data analysis.

Questions for this track

  • what is the outcome of ISSUE-14 with regards to addressing the use cases gathered there.
  • what is a good format for dissemination of Polar data?
  • Is the current persistence format satisfactory?
  • What are the barriers to data analysis based on 3 above
  • what tools can we ship with data in order to move towards a PDAaaS paradigm

ETL 101 - Bringing Polar data analytics one step closer to the Polar Scientist

data science proposed session by lewismc

I would like this session to focus on

  • Gathering and assessing some data analysis tasks... are there some that present low hanging fruit which we can focus on?
  • determine whether we can address those tasks with the data sets provided at the PCI workshop
  • invoke discussion around the design of a suitable data model which would move towards satisfying the tasks
  • build ETL pipelines using Apache Gora as a data modeling framework and object mapper to evaluate which persistence technologies best address the data analysis tasks identified in 1.

GISCube, Open Source Web-based geoprocessing and visualization application

data sciencedemo proposed session by MBoustani

​GISCube is an open source web-based GIS application that supports variety of geospatial data format (such as Shapefile, GeoTIFF and netCDF).

In this session I will demo GISCube for processing as well as visualization some polar data.

Source:
https://github.com/MBoustani/GISCube
https://github.com/MBoustani/Geothon

Run Distributed Release Audit Tool (DRAT) on all codefest generated code and report out on license statistics

data science proposed session by chrismattmann 

DRAT (https://github.com/chrismattmann/drat/) is a release audit tool that takes Apache RAT and turns it into a Map Reduce style system for large and heterogeneous code bases where RAT falls flat on its face.

Antarctic Meteorology Research Center (AMRC) datasets

data science proposed session by ccostanza

The AMRC at University of Wisconsin-Madison studies the weather in Antarctic in two ways; Automatic Weather Station (AWS) data and satellite composite imagery. For this workshop, a dataset has been prepared that contains five formats of AWS data and two formats of infrared satellite composites for one month May, 2014. This dataset can be found via our ftp site (ftp://amrc.ssec.wisc.edu/pub/requests/DVPC/)

The data is all from the same time period, but different time scales. Also, the infrared satellite images are in area and netcdf format. I added more to the readme file (ftp://amrc.ssec.wisc.edu/pub/requests/DVPC/readme.txt) to explain the different formats.

In terms of visualization, it would be great if we could display the satellite images in the database and create meteorograms real-time. For example, http://amrc.ssec.wisc.edu/data/view-data.php?action=view_image&product=surface/awsmeteorograms/8928.GIF

Crawl and prepare NSF ACADIS, NASA AMD and NSIDC Arctic Data Explorer datasets Part 2

data science proposed session by chrismattmann

Building off of NCEAS/open-science-codefest#26, continue data prep and crawl of AMD, ACADIS and ADE with goal of preparing some of the data for (GeoViz; science focused viz, etc.)

Participants would use real world data science tools like Tika (http://tika.apache.org/), Nutch (http://nutch.apache.org/), Solr (http://lucene.apache.org/solr/) and OODT (http://oodt.apache.org/) to crawl and prepare the datasets of interesting Polar parameters for Visualization experts to then hack on during a 2 day NSF visualization hackathon in NYC in November. Be part of doing something real, contributing to Apache projects (and getting the merit and potentially becoming a committer and PMC member yourself) and also contributing to NSF and NASA goals!

Agenda

Day 1

Time

Activity 

Lead

8:00 am ET

Registration opens - coffee/snacks

All

9:00 am ET

Introduction to the NSF DataViz Hackathon for Polar CyberInfrastructure

Chris Mattmann

9:20 am ET

NSF Polar CyberInfrastructure Program

Dr. Marco Tedesco

9:20 am ET Opening Keynote: Data Viz and Immersive Interfaces Dr. Curtis Lisle

10:00 am ET

Session Leaders Lightning Talks / Synthesis

Session Leaders/Proposers

10:30 am ET

Coffee Break

All

11:00 am ET

First Hack - sessions break up and begin hacking

Session Leaders/Participants

12:30 pm ET

Lunch - provided

All

2:00 pm ET

Second Hack - continue hacking (feel free to switch from prior sessions or stay/maintain)

Session Leaders/Participants

  Design Feedback from Parsons Faculty Parsons Faculty led by Aaron Hill

4:30 pm ET

Coffee Break

All

4:45 pm ET

Report Out - Day 1 - goals for next day

Session Leaders/Participants

 

5:30 pm ET

Dinner:

Vapiano, 113 University Pl, New York, NY 10003.

All

Day 2

Time

Activity

Lead

8:00 am ET

Registration opens - coffee/snacks

All

9:00 am ET

Day 2 Keynote: DataViz and Science Keynote

Mr. Jer Thorp

9:30 am ET

Review of Goals from Day 1 - outcomes

Session Leaders

10:00 am ET

Coffee break

All

10:39 am ET

Third Hack - sessions break up and begin hacking

Session Leaders/Participants
  Design Feedback from Parsons Faculty Parsons Faculty led by Aaron Hill

12:30 pm ET

Lunch - provided

All

2:00 pm ET

Fourth Hack - continue hacking (feel free to switch from prior sessions or stay/maintain)

Session Leaders/Participants

4:00 pm ET

Coffee Break

All

4:30 pm ET

Report Out - Day 2 - Closing Remarks  

All / Chris Mattmann and Organizing Committee


Tweets

#nsfpolardatavis

IRC

IRC generously provided by Freenode.net.
Join #nsfpolardatavis on irc.freenode.net.

Hackathon Participants

Indrani Das, Lamont-Doherty Earth Observing Laboratory, Columbia University

S. McKenzie, Skiles University of California Los Angeles (UCLA)

David Reagan, Indiana University

Lewis John, McGibbney NASA Jet Propulsion Laboratory

Jeffrey Schmaltz, NASA Goddard Space Flight Center

Ryan Boller, NASA Goddard Space Flight Center

Tyler Palsulich, New York University (NYU)

Chalalai Chaihirunkarn, Carnegie Mellon University

WenWen Li, Arizona State Unviersity

Jeremiah Dabney, University of Alaska, Fairbanks

Matthew Savoie, National Snow and Ice Data Center (NSIDC)/University of Colorado - Boulder

Justin Paul-Peters, Indiana University

John Morton, Lamont-Doherty Earth Observing Laboratory, Columbia University

Suzanne Carbotte, Lamont-Doherty Earth Observing Laboratory, Columbia University

Christopher Sweeney, US Coast Guard Academy

Carol Costanza, University of Wisconsin-Madison

Aaron Presnall, Jefferson Institute

Stephen Diggs, Scripps/Univeristy of California San Diego

Kanchana Welagedara, Computer Society of Sri Lanka/Apache Software Foundation

Zoran Hrncic, Jefferson Institute

Mia Bennett, University of California Los Angeles (UCLA)

Allen Pope, National Snow and Ice Data Center (NSIDC)/University of Colorado - Boulder

Maziyar Boustani, NASA Jet Propulsion Laboratory

Scott Pearse, National Center for Atmospheric Research (NCAR)

Rachel Obbard, Dartmouth University

Laura Kehrl, University of Washington APL Polar Science Center

Martin Lehmann, University of Aalborg

Saskia van Manen, Open University, UK

Jesse Johnson, University of Montana

Curtis Lisle, KnowledgeVis, Inc.

Yuan Ho, Unidata/UCAR

Bruce Caron, New Media Studio/University of California Santa Barbara

Geetha Ratnam, Scripps/Univeristy of California San Diego

Eric Nienhouse, National Center for Atmospheric Research (NCAR)

Christine Laney, University of Texas El Paso (UTEP)

Alex Boghosian, Lamont-Doherty Earth Observing Laboratory, Columbia University

Paul Ramirez, NASA Jet Propulsion Laboratory

Justin Fields, Rails Dog

Zhong Liu, George Mason University

NEXT

 

 

Page statistics
4080 view(s) and 84 edit(s)
Social share
Share this page?

Tags

This page has no custom tags.
This page has no classifications.

Comments

You must to post a comment.

Attachments